🤖 ABot-PhysWorld

AMAP CV Lab

ABot-PhysWorld is a physically consistent, action-controllable video world model for robotic manipulation, built on a 14-billion-parameter Diffusion Transformer. It integrates physics-aware training, memory-efficient preference optimization, and precise spatial action injection to generate realistic and physically plausible robot-object interactions — even in zero-shot settings.

🗞️ News

• [2026-04] 🏆 1st Place on WorldArena Leaderboard! ABot-PhysWorld achieves the top rank on the WorldArena benchmark.
• [2026-04] 🥈 2nd Place on GigaBrain Challenge CVPR 2026 – World Model Track! ABot-PhysWorld secures the runner-up position in the CVPR 2026 GigaBrain Challenge World Model Track.
• [2026-04] 🎮 A2V code released! Action-to-Video training and inference via VACE parallel context blocks. See training/README_A2V.md and inference/README_A2V.md.
• [2026-04] 🧪 DPO training released! Direct Preference Optimization pipeline for physics-aware alignment with LoRA. See training/README_DPO.md.
• [2026-03] 🎉 Training code released! Full-parameter SFT training scripts for fine-tuning on custom robot manipulation datasets. See training/.
• [2026-03] 📦 SFT training data released! The v1 SFT training dataset is available on ModelScope.
• [2026-03] 🔬 Benchmark released! EZS-Bench evaluation toolkit and data are open-sourced. See EZS-Bench/.
• [2026-03] 🚀 Inference code released! Generate robot manipulation videos with the pre-trained model. See inference/.

🏆 Competition Results

WorldArena Leaderboard – 🥇 1st Place

👆 Click the image to view the live leaderboard on HuggingFace

GigaBrain Challenge CVPR 2026 – World Model Track – 🥈 2nd Place

👆 Click the image to view the live leaderboard on HuggingFace

Table of Contents

• 📚 Key Contributions
• 🚀 EZS-Bench
• 📊 Evaluation
• 🖼️ Qualitative Results
• 🛠️ Usage
• 🏋️ Training
• 🎮 A2V (Action-to-Video)
• 🧪 DPO Training
• 📜 Citing

📚 Key Contributions

1. Industrial-Grade Data Pipeline
   Curated ~3M real-world manipulation clips from five datasets (AgiBot, RoboCoin, RoboMind, Galaxea, OXE) with motion, semantic, and action consistency filtering, plus hierarchical sampling for balanced generalization.

   

2. Physics-Aware DPO Training
   Introduces a decoupled VLM-based discriminator: Qwen3-VL generates task-specific physics checklists, Gemini 3 Pro scores videos via Chain-of-Thought; combined with LoRA-augmented DPO on a 14B DiT to enforce physical plausibility.

   

3. Parallel Context Blocks for Action Control
   Enables precise action-conditioned generation by residually injecting spatial action maps into cloned DiT blocks, preserving physical priors while supporting cross-embodiment control.

   

4. EZSbench – First True Zero-Shot Benchmark
   Fully training-independent evaluation covering unseen robot, scene, and task combinations, with dual-model scoring to eliminate self-evaluation bias.

   

---

🚀 EZS-Bench

Embodied-ZeroShot Benchmark for Physically Consistent Video Generation 🤖✨

EZS-Bench is a zero-shot evaluation benchmark designed to rigorously assess physically plausible video generation in robotic manipulation. It evaluates models on physical consistency, action controllability, and cross-embodiment generalization—with no training-test data overlap. 🔍🔬

✨ Key Features

✅ True Zero-Shot Evaluation
Unseen combinations of:
- 🤖 Robot morphologies (e.g., single-arm, bimanual, custom kinematics)
- 🌍 Scenes & backgrounds
- 🎯 Manipulation tasks (pick-and-place, wiping, assembly, etc.)

🎨 Dual-Source Data Construction
- 🧬 Synthetic branch: Text-to-image generation with controlled variation
- 🖼️ Real-world editing: VLM-driven scene augmentation preserving physical interactions

🧠 Physics-Aware Evaluation
- Dynamic physical checklists generated by VLMs (e.g., "Does the gripper penetrate the object?", "Is gravity respected?")
- 30–50% negative questions to prevent guessing 🚫
- Decoupled scorer architecture to eliminate self-evaluation bias ⚖️

📊 Comprehensive Metrics
Evaluates:
- Physical fidelity (penetration, contact, deformation) 💥
- Temporal coherence 🕒
- Spatial alignment & trajectory consistency 🎯

📦 Getting Started

Download evaluation data from ModelScope:

git lfs install
git clone https://www.modelscope.cn/datasets/amap_cvlab/EZS-Bench_data.git

Install and run the evaluation toolkit:

cd EZS-Bench
pip install -e .

# Full evaluation (Video Quality + Domain Score)
torchrun --standalone --nproc_per_node=4 evaluate_ezsbench.py \
    --data_file /path/to/EZS-Bench_data/video_prompt_question_196_ezs0.jsonl \
    --method_name "YourMethod" \
    --method_dir /path/to/generated_videos \
    --output_dir ./results

The VLM judge model (Qwen2.5-VL-72B-Instruct, ~150 GB) is automatically downloaded on first run.

🔗 See EZS-Bench/README.md for full documentation.

---

📊 Evaluation

We evaluate ABot-PhysWorld on three key aspects:
- Physical Consistency (via PBench and EZSbench)
- Zero-Shot Generalization (via EZSbench)
- Action-Conditioned Controllability (via custom A2V benchmark)

📈 Summary of Advancements 🎉🎉

Capability	Benchmark	Ours	Best Baseline	Gain
Physical Fidelity	PBench (Domain Score)	0.9306	0.8644 (Wan2.5)	+6.62%
Zero-Shot Generalization	EZSbench (Domain Score)	0.8366	0.7951 (WoW)	+4.15%
Action Control	Trajectory Consistency	0.8522	0.8157 (Enerverse)	+3.65%

✅ ABot-PhysWorld establishes a new standard for physically grounded, controllable, and generalizable world models in robotic manipulation.

---

🖼️ Qualitative Results

Selected representative zero-shot generation results demonstrating ABot-PhysWorld's strong generalization and physical plausibility.

🎯 Zero-Shot Capabilities

🔧 Scene 1: Deformable Object – Dual-Arm Towel Folding

• Task: Fold a towel using dual robotic arms
• Challenge: Complex cloth dynamics and bimanual coordination
• Ours:
  ✅ Physically realistic deformation
  ✅ Smooth, collision-free arm motion
  ✅ Natural folding sequence with consistent contact

🥤 Scene 2: Fine Manipulation – Diverse Object Handling

• Task: Stack cups, build blocks, place a knife
• Challenge: Varying shapes, weights, and friction
• Ours:
  ✅ Accurate grasp pose prediction
  ✅ Adaptive gripper control
  ✅ Stable pick-and-place without slippage or penetration

🚪 Scene 3: Articulated Object – Opening a Cabinet Door

• Task: Open a hinged cabinet or door
• Challenge: Enforce rotational constraints and correct force direction
• Ours:
  ✅ Proper handle grasping
  ✅ Realistic hinge rotation
  ✅ Motion follows physical pivot axis

🫗 Scene 4: Fluid Interaction – Pouring Water

• Task: Pour water from a cup into a bowl using dual arms
• Challenge: Bimanual coordination, tilt control, liquid dynamics
• Ours:
  ✅ Collision-free trajectory planning
  ✅ Accurate pour timing and angle
  ✅ Visual consistency in fluid transfer (simulated proxy)

🧽 Scene 5: Cleaning Task – Wiping a Stain

Note: The Gemini watermark (bottom-right) indicates the initial frame generated by Gemini (ensuring it is completely unseen); all other frames are generated by ABot-PhysWorld.

• Task: Wipe a stain off a table
• Challenge: Maintain contact, uniform pressure, full coverage
• Ours:
  ✅ Continuous tool-surface contact
  ✅ Systematic wiping motion
  ✅ Gradual removal of the stain in video output

🍓 Scene 6: Multi-Scene Generalization – Fruit Sorting

Note: The Gemini watermark (bottom-right) indicates the initial frame generated by Gemini (ensuring it is completely unseen); all other frames are generated by ABot-PhysWorld.

• Task: Place fruits into a plate across diverse scenes
• Challenge: Background, lighting, and fruit variation
• Ours:
  ✅ Robust object recognition under domain shifts
  ✅ Consistent performance across unseen environments
  ✅ Fast and stable manipulation regardless of setup

🔍 Pbench Results Demonstration

We conducted systematic qualitative comparative experiments on the PAI-Bench benchmark dataset. Below are the generated results from several typical scenarios.

Task	Baselines	Ours
Grasping	Frequent penetration, floatation	✅ Firm contact, no violation
Long-horizon Planning	Inconsistent state transitions	✅ Coherent multi-step reasoning
Rigid-body Dynamics	Unphysical deformations	✅ Preserved geometry and mass behavior
Conta