github.com/World-In-World/world-in-world @main

World-in-World is a unified closed-loop benchmark and toolkit for evaluating visual world models (WMs) by their embodied utility rather than only image or video appearance. World-in-World provides: (1) a unified online planning strategy that works with different WMs, (2) a unified action API that adapts to text, viewpoint, and low‑level controls, and (3) a task suite covering Active Recognition (AR), Active Embodied QA (A‑EQA), Image‑Goal Navigation (IGNav), and Robotic Manipulation.

📰 News

• 2025-10-22: Preprint released on arXiv. Landing page and repository initialized.
• 2025-11-10: Add post‑training instructions and data collection instructions in data collection section.
• 2026-02-09: Add Manipulation task instructions and environment setup.
• 2026-04-01: Add OpenPI as proposer in the manipulation task.
• 2026-04-03: Now supports LIBERO as the backend in the manipulation task. Thanks @DehuiWang01!

✨ Overview

In this work, we propose World-in-World, which wraps generative World models In a closed-loop World interface to measure their practical utility for embodied agents. We test whether generated worlds actually enhance embodied reasoning and task performance—for example, helping an agent perceive the environment, plan and execute actions, and re-plan based on new observations within such a closed loop. Establishing this evaluation framework is essential for tracking genuine progress across the rapidly expanding landscape of visual world models and embodied AI.

🚧 Repository Status

The release will follow the to‑do list below and will be updated continuously.

Under construction - Full documentation and tutorials for environment setup and task evaluation. - [X] AR, IGNav, AEQA - [X] Manipulation - [X] WM post‑training instructions - [ ] Instructions to add a new WM to World‑in‑World

🚀 Getting Started

1) Documentation structure

• 01_setup_env.md: Environment setup for all environments used in the repo.
• 02_evaluation_datasets.md: Datasets used for evaluation.
• 03_run_commands.md: How to deploy servers and run evaluation scripts.
• 04_post_training.md: Post‑training configurations, data collection instructions, and checkpoints for different WMs.
• 05_add_new_WM.md: How to add a new WM to World‑in‑World.
• 09_WM_server_design.md: Design details of the WM server.

2) Checklist for running an evaluation

For any task, complete the following steps in order.

1. Set up environments.
2. AR, IGNav, AEQA: set up Habitat‑sim as described in 01_setup_env.md: Environment for Habitat‑sim.
3. Manipulation: set up wow-manip as described in 01_setup_env.md: Environment for WIW-Manipulation
4. Download scene datasets.
5. AR: download MP3D as described in 02_evaluation_datasets.md: Common Steps.
6. IGNav, AEQA: download HM3D as described in 02_evaluation_datasets.md: Common Steps.
7. Manipulation: already exists in downstream/world-in-world-manip/wiw_manip/envs/vlm
8. Download evaluation episodes.
9. AR: see 02_evaluation_datasets.md: Download AR evaluation episodes.
10. IGNav: see 02_evaluation_datasets.md: Download IGNav evaluation episodes.
11. AEQA: see 02_evaluation_datasets.md: Download AEQA evaluation episodes.
12. Manipulation: already exists in downstream/world-in-world-manip/data
13. Deploy policies (VLM policy, heuristic policy, diffusion policy).
14. AR: deploy VLM policy as in 03_run_commands.md: VLM Deployment. If you use a heuristic policy, you can skip the VLM step.
15. IGNav: deploy VLM policy as in 03_run_commands.md: VLM Deployment. If you use a heuristic policy, you can skip the VLM step.
16. AEQA: deploy VLM policy as in 03_run_commands.md: VLM Deployment.
17. Manipulation:
    • VLM policy: deploy the VLM policy as described in 03_run_commands.md: VLM Deployment.
    • Diffusion policy: after configuring the ckpt paths in 01_setup_env.md: Configure the required ckpt files for 3D-Diffuser-Actor, no additional deployment is needed.
18. Deploy other task‑related models if needed.
19. AR: deploy the SAM2 server as in 03_run_commands.md: SAM2 Deployment.
20. IGNav: no extra task models.
21. AEQA: deploy the Grounding SAM2 server as in 03_run_commands.md: Grounding SAM2 Deployment.
22. Manipulation: no extra task models.
23. Deploy the WM server.
24. AR, IGNav, AEQA: see 03_run_commands.md: World Model Deployment and WMs for Habitat‑sim Tasks.
25. Manipulation: see 03_run_commands.md: World Model Deployment and WMs for Manipulation Tasks.
26. Run the evaluation script.
27. AR, IGNav, AEQA: see 03_run_commands.md: Run the Evaluation Scripts.
28. Manipulation: see 03_run_commands.md: Manip pattern.
29. Accumulate results.
30. AR, IGNav, AEQA: see 03_run_commands.md: Get evaluation results.
31. Manipulation: see 03_run_commands.md: Get evaluation results.

After the first run, the environment and datasets are in place. For later runs, you usually only repeat steps 4–8. If you encounter any issue, please feel free to open an issue or contact us.

P.S. if u have any question about the server deployment, you can also refer to 03_run_commands.md: Common questions and 09_WM_server_design.md: WM Server Design Details for troubleshooting.

🏆 Submit Custom Results to the Leaderboard

To submit new results to the leaderboard:

1. Update this repository. Fork/clone this repo, add your modifications (e.g., custom model inference script) and instructions for how we can reproduce the results, then open a pull request for review.

2. Update the website leaderboard. Fork/clone the website repo: https://github.com/World-In-World/World-In-World.github.io Edit subpages/leaderboard.html: https://github.com/World-In-World/World-In-World.github.io/blob/main/subpages/leaderboard.html Then open a pull request.

We will review the submission and, once verified, we will merge the changes and update the leaderboard accordingly.

📝 Citation

If you find this work useful, please cite:

@misc{zhang2025worldinworld,
  title        = {World-in-World: World Models in a Closed-Loop World},
  author       = {Zhang, Jiahan and Jiang, Muqing and Dai, Nanru and Lu, Taiming and Uzunoglu, Arda and Zhang, Shunchi and Wei, Yana and Wang, Jiahao and Patel, Vishal M. and Liang, Paul Pu and Khashabi, Daniel and Peng, Cheng and Chellappa, Rama and Shu, Tianmin and Yuille, Alan and Du, Yilun and Chen, Jieneng},
  year         = {2025},
  eprint       = {2510.18135},
  archivePrefix= {arXiv},
}