SAGE: Scalable Agentic 3D Scene Generation for Embodied AI

Hongchi Xia, Xuan Li, Zhaoshuo Li, Qianli Ma, Jiashu Xu, Ming-Yu Liu, Yin Cui, Tsung-Yi Lin, Wei-Chiu Ma, Shenlong Wang, Shuran Song, Fangyin Wei

NVIDIA, University of Illinois Urbana-Champaign, Cornell University, Stanford University

Intro

SAGE is an agentic framework that, given a user-specified embodied task, understands the intent and automatically generates simulation-ready environments at scale. Here we release both 3D scene and action generation code, as well as the agentic-generated SAGE-10k dataset to foster further research.

SAGE-10k Dataset

SAGE-10k is a large-scale interactive indoor scene dataset featuring realistic layouts, generated by the agentic-driven pipeline introduced in "SAGE: Scalable Agentic 3D Scene Generation for Embodied AI". The dataset contains 10,000 diverse scenes spanning 50 room types and styles, along with 565K uniquely generated 3D objects.

Directory Structure

The repository is organized into the following main components:

• client/ Contains the client-side implementation and scripts. This is the primary entry point for users to initiate scene generation, control the pipeline, and interface with NVIDIA Isaac Sim.

• server/ Hosts the core backend logic. This includes integrations with Foundation Models (LLMs, VLMs), 3D asset generation (TRELLIS), material synthesis, and the scene layout solvers.

• IsaacLab/
  Integration with NVIDIA Isaac Lab, providing the simulation environment for robot learning and physical interaction tasks.

  Note: This directory contains modifications to the original code and remains subject to the original BSD-3-Clause License.

• M2T2/
  Integration with M2T2, utilized for generating contact-rich manipulation data and handling complex robot-object interactions.

  Note: This directory contains modifications to the original code and remains subject to the original NVIDIA License.

• matfuse-sd/
  Integration with MatFuse material generation engine, used to generate high-quality textures and materials for 3D objects and scenes.

  Note: This directory contains modifications to the original code and remains subject to the original MIT License.

• robomimic/
  Integration with robomimic, a framework for robot policy learning from demonstration, used for training policies on generated data.

  Note: This directory contains modifications to the original code and remains subject to the original MIT License.

Getting Started

To use this repository, you will need to set up both the server (backend) and the client (frontend/interface). Please refer to the respective README files for detailed instructions.

1. Server Setup

Read the Server Documentation - Setup instructions for the backend infrastructure. - Hosting details for VLM (Qwen), LLM (GPT), and 3D generation models (TRELLIS). - Guides for running augmentation pipelines.

2. Client Setup

Read the Client Documentation - Installation of the Python environment and dependencies. - Instructions for installing and linking NVIDIA Isaac Sim. - Scripts for running scene generation, robot task generation, and visualization.

Usage Workflow

1. Start the Backend: Ensure all model servers (LLM, VLM, TRELLIS) are running as described in the Server README.
2. Configure the Client: Set up your key.json and environment variables in the Client directory.
3. Run Generation: Use the scripts in client/scripts/ to generate scenes (e.g., generate_from_room_desc.sh) or robot data.

Citation

If you find our work useful in your research, please consider citing:

@article{xia2026sage,
  title={SAGE: Scalable Agentic 3D Scene Generation for Embodied AI},
  author={Xia, Hongchi and Li, Xuan and Li, Zhaoshuo and Ma, Qianli and Xu, Jiashu and Liu, Ming-Yu and Cui, Yin and Lin, Tsung-Yi and Ma, Wei-Chiu and Wang, Shenlong and Song, Shuran and Wei, Fangyin},
  journal={arXiv preprint arXiv:2602.10116},
  year={2026}
}

Acknowledgements

We gratefully acknowledge the authors of the following projects for their foundational work and open-source contributions. This repository builds upon and adapts components from:

Modifications have been made to the following components, which remain subject to their original licenses:

Additionally, our implementation of the MCP client, server. and robotics sim draws inspiration and references from: