MolmoAct2: Action Reasoning Models for Real-world Deployment

MolmoAct2 is Ai2's open family of action reasoning models for robot control and real-world deployment. It builds on the Molmo2-ER embodied-reasoning vision-language backbone, adds robot state and action modeling, and connects the VLM to a flow-matching continuous action expert for closed-loop manipulation. The release includes base checkpoints for continued training, fine-tuned robot policies for evaluation and deployment, and the datasets used to build MolmoAct2 and Molmo2-ER.

Updates

• [2026/08/22] 🔥 We fixed some finetuning issues in LeRobot's MolmoAct2 training scripts: huggingface/lerobot.
• [2026/06/13] 🔥 We have released pre-training and post-training code and full experimental details for MolmoAct2, get started Here.
• [2026/06/10] 🔥 We have setup zero-shot evaluation for MolmoAct2 (DROID and Bimanual YAM) on Maniskill simulation, get started Here.
• [2026/05/28] 🔥 MolmoAct2 has been fully integrated into Huggingface, LeRobot official repo at MolmoAct2.
• [2026/05/19] 🔥 We've also released MolmoAct2-Cortex evaluation rollouts on YAM bimanual setups (useful for failure annotation and reward model training) at Policy Rollouts.
• [2026/05/17] 🔥 We have released FastAPI inference servers for MolmoAct2 using DROID and YAM setups at Inference Servers (implemented by Jie Wang).
• [2026/05/14] 🔥 We have released MolmoAct2 lerobot workflow for fine-tuning and inference. Check it out.
• [2025/05/06] 🔥 Detail implementation and setup for Franka, SO-100/101, and bimanual YAM have been released at Real-world Deployment.
• [2026/05/05] 🔥 MolmoAct2 has been released!

Intel XPU Support

MolmoAct2 inference has been validated on Intel XPU (Intel GPUs) and runs without any code changes to this repository. Install a PyTorch build with Intel XPU support and select the xpu device at runtime.

1. Models

Base Models

We provide base checkpoints at every training stage for continued MolmoAct2 training and robot fine-tuning. These are foundation checkpoints rather than one-size-fits-all deployment policies.

Finetuned Models

We also provide fine-tuned checkpoints for common robot platforms and benchmarks. These models are intended to run directly in their target setting, or to serve as a stronger starting point for closely related robots. As with any robot policy, performance depends on hardware, cameras, calibration, action conventions, and language/task distribution.

2. Datasets

Note that all of the robotics datasets for pre-training and post-training are in LeRobot v3.0 format, paired with extra language annotations.

3. LeRobot Integration

MolmoAct2 is integrated into LeRobot as a policy implementation, so users can train, evaluate, and deploy MolmoAct2 with standard LeRobot datasets and workflows. This repository includes the LeRobot integration as a Git submodule at lerobot/, pinned to the branch allenai/lerobot:molmoact2-policy.

For training, although all of our experiments start from the base checkpoint allenai/MolmoAct2, we recommend starting from the fine-tuned checkpoints listed in the Finetuned Models section above if your embodiment is similar to Bimanual YAM, DROID Franka, or SO-100/SO-101, as they can provide better initialization and downstream performance. For generic use, use the base checkpoint.

After cloning this repository, initialize the submodule from the repo root:

git submodule update --init --recursive
cd lerobot

For training, evaluation, and deployment instructions, see the MolmoAct2 LeRobot documentation at docs/source/molmoact2.mdx. To reproduce the original LIBERO benchmark results exactly with the v0.5.1 evaluation stack, use the pinned inference branch allenai/lerobot:molmoact2-hf-inference with instructions in MolmoAct2 README.

We also open-source the original MolmoAct2 experiment scripts under experiments/. These cover training and evaluation replication, depth annotation, Hugging Face checkpoint conversion, and fine-tuning on new LeRobot datasets. See experiments/README.md for setup and commands.

4. Real-world Deployment

[!WARNING] Disclaimer: Out-of-the-box deployment is intended for simple tasks within the training task distribution (e.g., Pick-and-Place, opening, closing and etc). Performance has only been empirically verified on the SO-100 and Franka DROID embodiments. Results on other embodiments and tasks are not guaranteed.

MolmoAct2 supports out-of-the-box deployment on three robot embodiments:

• SO-100
• Bimanual YAMs
• Franka DROID setup

SO-100/101 Setup

For the best performance, we recommend using an SO-100 with the standard wrist configuration and a third-person camera. Here is an open implementation by Irene Grace. Code

Bimanual YAM Setup

For the best performance, please build your Bimanual YAM setup following the reference design below:

All required components can be purchased using this Bimanual YAM parts list.

Implementation code for setting up, data collection, and inference for Bimanual YAM is here

Standardize evaluation implementation for zero-shot by Cortex AI here

Franka Setup

For the Franka setup, we recommend following the official DROID implementation for best results.

5. Inference Servers

This repository ships two FastAPI inference servers under examples/, one per fine-tuned checkpoint. Each server exposes the same /act wire protocol — json_numpy-encoded request/response — but with an embodiment-specific schema (camera count, state dimension, normalisation tag).

1. Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh
exec $SHELL          # reload PATH so the `uv` binary is picked up
uv --version

2. Create the project environment

The pinned dependencies (CUDA-12.8 PyTorch wheels, transformers, fastapi, json-numpy, …) live in pyproject.toml. CUDA 12.8 supports Blackwell GPUs, including RTX 50-series cards; it requires NVIDIA driver 570.26 or newer on Linux and 570.65 or newer on Windows. The CUDA toolkit itself does not need to be installed separately because the PyTorch wheels include the CUDA runtime.

From the repo root:

uv sync                  # creates .venv/ and installs all deps
uv run python -c "import torch, torchvision; print(torch.__version__, torchvision.__version__, torch.version.cuda, torch.cuda.is_available(), torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0), torch.cuda.get_arch_list())"
# expected: torch 2.11.0+cu128, torchvision 0.26.0+cu128, CUDA 12.8, and your GPU

The project supports Python 3.11 and 3.12; uv selects a compatible interpreter and downloads one if needed. Re-run uv sync after pulling new commits.

For an RTX 50-series GPU, confirm that the final architecture list contains sm_120. If inference fails with `