RoboLab

[Website] | Paper (coming soon)

RoboLab is a task-based evaluation benchmark for robot manipulation policies built on NVIDIA Isaac Lab. It provides 100+ manipulation tasks with automated success detection, a server-client policy architecture, and multi-environment parallel evaluation — designed for reproducible, large-scale benchmarking of generalist robot policies in simulation.

Key Features

• RoboLab-120: An initial set of 120 brand new benchmark tasks spanning pick-and-place, stacking, rearrangement, tool use, and more — each with language instructions and automated success/failure detection via composable predicates.
• Bring your own robot: Tasks are not tied to a specific robot embodiment, so you can plug in any robot compatible with IsaacLab!
• Rich Asset Libraries: See a list of objects, scenes, and curated backgrounds — everything you need to create new scenes and new tasks for your own evaluation needs.
• AI-Enabled Workflows: Generate new scenes and tasks in minutes using natural language with the /robolab-scenegen and /robolab-taskgen Claude Code skills.
• Multi-Environment Parallel Evaluation: Run multiple episodes in parallel across environments with vectorized conditionals and per-environment termination.
• Server-Client Policy Architecture: Policy models run as standalone servers; RoboLab connects via lightweight inference clients (OpenPI, GR00T, and more).

Getting Started

Requires Isaac Lab and uv. See Requirements for versions and hardware.

Installation

git clone https://github.com/NVlabs/RoboLab.git
cd robolab
uv venv --python 3.11
source .venv/bin/activate
uv pip install "setuptools<81"
uv sync

Verify installation:

python scripts/check_registered_envs.py

Run without a policy

# Run an empty episode with random actions
python examples/demo/run_empty.py --headless

# Playback recorded demonstration data
python examples/demo/run_recorded.py --headless

Run with a policy

RoboLab uses a server-client architecture: your model runs as a standalone server, and RoboLab connects to it via a lightweight inference client. To quickly test RoboLab, try Pi0-5 via OpenPI.

1. Start your policy server in a separate terminal.
2. Run evaluation: bash python examples/policy/run_eval.py --policy pi05 --task BananaInBowlTask --num-envs 12 --headless
3. Analyze results: bash python analysis/read_results.py output/<your_run_folder>

Common CLI Options

# Run on specific tasks (these two are good for sanity checking)
python examples/policy/run_eval.py --policy pi05 --task BananaInBowlTask RubiksCubeAndBananaTask

# Run on a tag of tasks
python examples/policy/run_eval.py --policy pi05 --tag semantics

# Run 12 parallel episodes per task
python examples/policy/run_eval.py --policy pi05 --headless --num-envs 12

# Enable subtask progress tracking
python examples/policy/run_eval.py --policy pi05 --headless --enable-subtask

# Resume a previous run (skips completed episodes)
python examples/policy/run_eval.py --policy pi05 --output-folder-name my_previous_run

Documentation

Full documentation is at docs/README.md, covering:

• Objects, Scenes, Tasks — Creating and managing assets and benchmark tasks
• Robots, Cameras, Lighting, Backgrounds — Configuring simulation parameters
• Environment Registration — Combining tasks with robot/observation/action configs
• Inference Clients — Built-in policy clients and server setup (OpenPI, GR00T)
• Evaluating a New Policy — Implementing your own inference client
• Running Environments — CLI reference and evaluation workflows
• Analysis and Results — Summarizing, comparing, and auditing results
• Subtask Checking, Conditionals, Event Tracking — Advanced task features

Example Tasks

"Make sure all the white mugs are upright so that the opening is facing upwards."

"Put all plastic bottles away in the bin."

"Put the orange measuring cup and the blue measuring cup outside of the plate."

See the full Benchmark Task Library for all 120 tasks.

Requirements

• Disk space: ~8 GB (assets account for ~7 GB)
• GPU: NVIDIA RTX GPU required. Recommend 48GB+ VRAM. See Isaac Lab's hardware requirements for recommended GPUs and VRAM.
• Speed: 30 GPU hours / 100 tasks, 1.4 it/s (assuming ~200ms inference step)

License

The RoboLab framework is released under CC-BY-NC-4.0.

Citation

% Coming soon
@article{robolab2026,
  title={RoboLab: A Task-Based Evaluation Benchmark for Robot Manipulation},
  year={2026}
}