LingBot-VA: Causal World Modeling for Robot Control

https://github.com/user-attachments/assets/cec7b7a6-953b-4fa4-8f1a-47efc1fce547

Table of Contents

• News
• Model Download
• Quick Start
• Installation
• attn_mode Configuration
• Deploying LingBot-VA for Inference
  • Evaluation on RoboTwin-2.0
  • Evaluation on LIBERO
  • Run Image to Video-Action Generation
• Post-Training LingBot-VA
  • Data Preparation
  • Custom Dataset Preparation
  • Training
• Performance
• Simulation Evaluation
• Real-world Deployment
• License
• Citation
• Acknowledgments

---

💫 Meet LingBot-VA! We've built an AR diffusion framework for simultaneous world modeling and action! 🤖✨

LingBot-VA has focused on: - Autoregressive Video-Action World Modeling: Architecturally unifies visual dynamics prediction and action inference within a single interleaved sequence while maintaining their conceptual distinction. - High-efficiency Execution: A dual-stream mixture-of-transformers(MoT) architecture with Asynchronous Execution and KV Cache. - Long-Horizon Performance and Generalization: High improvements in sample efficiency, long-horizon success rates, and generalization to novel scenes.

🚀 News

• [2026-04-24] Weights for post-train on LIBERO-LONG released! (IMPORTANT: Ensure that va_libero_cfg.action_snr_shift, va_libero_cfg.used_action_channel_ids and va_libero_cfg.norm_stat in wan_va/configs/va_libero_cfg.py are synchronized with the latest version of the repository.)
• [2026-04-08] Post-training and inference code for the LIBERO dataset is now available!
• [2026-02-17] Post-training code and dataset released! Support fine-tuning LingBot-VA on custom robotic manipulation datasets.
• [2026-01-29] Weights and code for shared backbone released! Please stay tuned for our separated version!

---

📦 Model Download

• Pretrained Checkpoints for Post-Training

Model Name	Huggingface Repository	ModelScope Repository	Description
lingbot-va-base	🤗 robbyant/lingbot-va-base	🤖 Robbyant/lingbot-va-base	LingBot-VA w/ shared backbone
lingbot-va-posttrain-robotwin	🤗 robbyant/lingbot-va-posttrain-robotwin	🤖 Robbyant/lingbot-va-posttrain-robotwin	LingBot-VA-Posttrain-Robotwin w/ shared backbone
lingbot-va-posttrain-libero-long	🤗 robbyant/lingbot-va-posttrain-libero-long	🤖 Robbyant/lingbot-va-posttrain-libero-long	LingBot-VA-Posttrain-LIBERO-LONG w/ shared backbone

• Post-Training Dataset

Dataset Name	Huggingface Repository	ModelScope Repository	Description
robotwin-clean-and-aug-lerobot	🤗 robbyant/robotwin-clean-and-aug-lerobot	🤖 Robbyant/robotwin-clean-and-aug-lerobot	Cleaned & augmented RoboTwin dataset in LeRobot format for post-training
libero-long-lerobot	🤗 robbyant/libero-long-lerobot	🤖 Robbyant/libero-long-lerobot	LIBERO-Long dataset in LeRobot format for post-training
---

🛠️ Quick Start

Installation

Requirements • Python == 3.10.16 • Pytorch == 2.9.0 • CUDA 12.6

pip install torch==2.9.0 torchvision==0.24.0 torchaudio==2.9.0 --index-url https://download.pytorch.org/whl/cu126
pip install websockets einops diffusers==0.36.0 transformers==4.55.2 accelerate msgpack opencv-python matplotlib ftfy easydict
pip install flash-attn --no-build-isolation

⚠️ Important: attn_mode Configuration

You MUST change the attn_mode setting depending on whether you are training or running inference. Since LingBot-VA is loaded via from_pretrained, this parameter is read from the model folder's transformer/config.json. You need to manually edit this file before launching.

Mode	attn_mode value	Notes
Training	"flex"	Required for training. Will not work for inference.
Inference / Evaluation	"torch" or "flashattn"	Required for inference. "flex" will cause errors at eval time.

How to change: Open <your-model-path>/transformer/config.json, find the "attn_mode" field, and set it to the appropriate value.

---

Deploying LingBot-VA for Inference

LingBot-VA supports both standalone execution and Server-Client architecture which separates the model environment from simulation. By isolating dependencies, the design avoids package clashes and supports distributed inference on GPUs, clusters, and other devices.

Evaluation on RoboTwin-2.0

Preparing the Environment

You can follow the official instructions from the original RoboTwin-2.0 repository:
https://robotwin-platform.github.io/doc/usage/robotwin-install.html

In summary:

1. Install Vulkan dependencies: bash sudo apt install libvulkan1 mesa-vulkan-drivers vulkan-tools

2. Clone the RoboTwin repository: bash git clone https://github.com/RoboTwin-Platform/RoboTwin.git && cd RoboTwin && git checkout 2eeec322

3. Modify script/requirements.txt with the following content: txt transforms3d==0.4.2 sapien==3.0.0b1 scipy==1.10.1 mplib==0.2.1 gymnasium==0.29.1 trimesh==4.4.3 open3d==0.18.0 imageio==2.34.2 pydantic zarr openai huggingface_hub==0.36.2 h5py # For Description Generation azure==4.0.0 azure-ai-inference pyglet<2 wandb moviepy imageio termcolor av matplotlib ffmpeg

4. Modify line 8 of script/_install.sh: bash pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable" --no-build-isolation

5. Install dependencies: bash bash script/_install.sh

6. Download assets: bash bash script/_download_assets.sh

Deploying the Inference Server

# single GPU
bash evaluation/robotwin/launch_server.sh

# multi-GPU
bash evaluation/robotwin/launch_server_multigpus.sh

Executing the Inference Client

# single GPU
task_name="adjust_bottle";
save_root="results/";
bash evaluation/robotwin/launch_client.sh ${save_root} ${task_name}

# multi-GPU
save_root="results/"
task_group_id=0;
bash evaluation/robotwin/launch_client_multigpus.sh ${save_root} ${task_group_id}

Related experiments results will be save in /path/to/your/RoboTwin/${save_root}. Please note that an eval_result folder is also generated. This is a native output from RoboTwin and is identical to the contents in the results folder; it can be safely ignored. It is important to note that the inference server and client must be deployed on the same machine. For launching multi-GPU client, we padded the original 50 tasks to 56 via duplication and partitioned them into 7 groups to align with the 8-GPU configuration of our inference node. You can specify the task_group_id (0-6) to select a particular group for inference. For detailed grouping configurations, please refer to evaluation/robotwin/launch_client_multigpus.sh.

GPU Memory Requirements: Approximately 24GB VRAM for single-GPU RoboTwin evaluation with offload mode enabled (VAE and text_encoder offloaded to CPU).

Evaluation on LIBERO

Follow the official instructions to install LIBERO, then launch the server and client:

# server
bash evaluation/libero/launch_server.sh

# client
bash evaluation/libero/launch_client.sh

Run Image to Video-Action Generation

We also provide a script for image to video-action generation:

NGPU=1 CONFIG_NAME='robotwin_i2av' bash script/run_launch_va_server_sync.sh

GPU Memory Requirements: Approximately 18GB VRAM for single-GPU i2av inference with offload mode enabled (VAE and text_encoder offloaded to CPU).

Post-Training LingBot-VA

We support post-training (fine-tuning) LingBot-VA on custom robotic manipulation datasets. The training pipeline uses FSDP for distributed training and integrates with LeRobot dataset format.

Additional Dependencies

On top of the base installation, post-training requires:

pip install lerobot==0.3.3 scipy wandb --no-deps

Data Preparation

Download the post-training dataset from HuggingFace:

huggingface-cli download --repo-type dataset robbyant/robotwin-clean-and-aug-lerobot --local-dir /path/to/your/dataset

Custom Dataset Preparation

If you want to fine-tune LingBot-VA on your own robotic manipulation data, follow these steps:

Example Dataset

We provide a converted example dataset based on data from Issue #29. This dataset has been converted into the expected format and is fully supported for training. You can download it to understand the required data structure:

• Download: Example Dataset

This example can serve as a reference for converting your own robotic manipulation data into the proper format.

Data Pipeline Overview

When preparing your custom dataset, the data goes through the following processing pipeline:

1. Raw Data → Convert to LeRobot format (with metadata and video files)
2. Add Action Segmentation → Add action_config to episodes.jsonl
3. Extract Latents → Process videos through VAE according to video specifications
4. Dataset Loading → Load processed data with proper action dimensions for training

The final data should conform to these specifications:

Action Format: - Output dimension: 30 dimensions, structured as follows: - Left arm EEF (end-effector): 7 dimensions - Right arm EEF (end-effector): 7 dimensions - Left arm joints: 7 dimensions - Right arm joints: 7 dimensions - Left arm gripper: 1 dimension - Right arm gripper: 1 dimension - In your dataset class loader, map your robot's action dimensions to this standard 30-dimensional format. Missing dimensions are padded with 0.

Video Format: - During VAE latent extraction, resize videos to ~256 × 256 pixels and downsample to 5-15 fps as a reference (adjust based on your task requirements).

Implementation Steps

Step 1: Convert your data to LeRobot format

Follow the official LeRobot dataset documentation to convert your raw data (e.g., HDF5, video files, etc.) into the standard LeRobot dataset format. Ensure that each episode contains the required observation videos, actions, and metadata.

Step 2: Add action_config field to episodes.jsonl

After converting to LeRobot format, you need to modify the meta/episodes.jsonl file to add an action_config field to each line. This field describes the temporal segmentation and natural language description of the robot's actions within each episode.

Each line in episodes.jsonl should follow this format:

{
  "episode_index": 0,
  "tasks": ["task description"],
  "length": 450,
  "action_config": [
    {
      "start_frame": 0,
      "end_frame": 450,
      "action_text": "Natural language description of the robot action in this segment.",
    }
  ]
}

• start_frame / end_frame: The frame range (0-indexed) of the action segment within the episode.
• action_text: A natural language description of what the robot does in this segment.

For episodes with a single continuous action, start_frame should be 0 and end_frame should equal the episode length. You can also define multiple segments per episode if your data contains sequential sub-tasks.

Step 3: Extract video latents with Wan2.2 VAE

LingBot-VA operates on video latent representations rather than raw pixels. You need to extract the latent