SpatialWorld

Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

Project Page | Paper | Data

Abstract

Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world. However, existing benchmarks predominantly rely on passive evaluation (e.g., static VQA) or simulator-specific pipelines, failing to assess general interactive spatial understanding. We introduce SpatialWorld, a unified benchmark for evaluating interactive spatial understanding of multimodal agents in complex real-world tasks.

Integrating eight heterogeneous simulation backends under a shared, simulator-agnostic protocol, SpatialWorld features 760 human-annotated tasks across diverse domains (e.g., household routines, travel, social collaboration). Agents must solve tasks under vision-only partial observability, actively gathering egocentric visual evidence and expressing decisions via a unified, text-based action interface native to MLLMs. For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier. Evaluating 15 advanced agents reveals that robust spatial task solving remains challenging: GPT-5 achieves an average TSR of only 17.4%, while the leading open-source model, Qwen-3.5-397B-A17B, reaches 14.1%.

760 Tasks	8 Backends	5 Categories	15 Models
Human-annotated	Simulation environments	Around all daily life	Evaluated MLLM agents

Benchmark Overview

SpatialWorld unifies diverse 3D backends under a standardized observation-action interface. Agents receive only a natural-language instruction and egocentric RGB observations, express decisions through a unified text-based action interface, and are evaluated with human-validated terminal-state verifiers.

SpatialWorld unifies eight 3D simulation backends under a shared closed-loop evaluation protocol.

Eight Simulation Backends

Backend	Description	Tasks
AI2-THOR	Near-photorealistic indoor scenes with rich object affordances	311
ProcTHOR	Procedurally generated indoor layouts for layout generalization	127
VirtualHome	Daily activity scripts in home environments	38
CARLA	Urban traffic simulation for outdoor navigation	80
EmbodiedCity	Large-scale city navigation with realistic dynamics	53
Multi-AI2THOR	Multi-agent social collaboration in shared indoor scenes	29
Multi-ProcTHOR	Coordinated multi-agent tasks in procedural scenes	17
3D Games	Block 3D, Maze 3D, Snake, Rubik's Cube, etc.	105

Complexity levels (parallel, not hierarchical): Navigation - reach targets via exploration; Interaction - object-level state changes; Hybrid - long-horizon navigation plus multi-step manipulation.

Key Findings

• Far from reliable 3D task solving. Across the full benchmark, GPT-5 reaches only 17.4% average TSR, while the strongest open-source model, Qwen-3.5-397B-A17B, reaches 14.1%.
• Physical tasks remain especially hard. GPT-5 achieves 14.4% Physical Overall TSR, followed by Qwen-3.5-397B-A17B at 12.2%.
• Success is not efficiency. Models with comparable TSR can differ substantially in step efficiency (SE), indicating redundant exploration and task-dependent shortcuts.
• Domain-specific strengths. GPT-5 leads Daily (14.9%), Travel (6.8%), and Social Collaboration (34.8%); Qwen-3.5-397B-A17B ties GPT-5 in Work & Study (16.9%) and leads physical entertainment; Gemini-3.1-Pro leads digital 3D games (39.0% TSR).
• Indoor vs. outdoor domain shift. GPT-5 (14.1%) and Qwen-3.5-397B-A17B (13.7%) lead indoors; Gemini-3-Flash (9.0%) and GPT-5 (8.3%) lead outdoors.
• Vision-only closed-loop evaluation. Agents actively explore under partial observability using only egocentric RGB and a text-based action interface.

Experimental Results

Full TSR / SE tables and task examples are on the project page. Highlights below:

Model	Physical Overall TSR	Digital 3D Games TSR	Notable Strength
GPT-5	14.4	36.4	Best Daily, Travel, and Social Collaboration scores
Qwen-3.5-397B-A17B	12.2	26.0	Ties GPT-5 in Work & Study; leads physical entertainment
Gemini-3.1-Pro	9.2	39.0	Best digital 3D games performance
Kimi-K2.5	9.2	31.0	Competitive social and game performance

Indoor and outdoor physical domains — overall TSR with environment-level bars for the top-five models in each domain.

Scenario Distribution (click to expand)

Environment	Daily	Work	Entertain.	Travel	Social	Total
AI2-THOR	219	41	40	11	0	311
ProcTHOR	92	10	23	2	0	127
VirtualHome	27	8	3	0	0	38
CARLA	0	0	0	80	0	80
EmbodiedCity	12	0	2	39	0	53
Multi-AI2THOR	0	0	0	0	29	29
Multi-ProcTHOR	0	0	0	0	17	17
3D Games	0	0	105	0	0	105
Total	350	59	173	132	46	760

Multi-Agent Social Collaboration Profile (click to expand)

About This Repository

SpatialWorld is a unified benchmark and toolkit for evaluating interactive spatial reasoning of multimodal agents across AI2-THOR, ProcTHOR, CARLA, VirtualHome, EmbodiedCity, and game environments.

The repository keeps one default config per environment under configs/ for standard evaluation. Model-specific and historical experiment configs are kept separately under experiments/configs/, while experiment CSV files are kept under experiments/csv/.

Task Classification

Per-task scenario labels and complexity types are maintained in task_classification_detail.csv at the repository root. Each row maps a task_id to:

Column	Description
environment	Simulation backend (ai2thor, procthor, carla, etc.)
task_id	Task identifier used in benchmark CSV files
instruction	Natural-language task instruction
category	Scenario category: Daily, Work, Entertain, Travel, or Social
task_type	Complexity type: Navigation, Interaction, or Hybrid

Task JSON files under data/ intentionally omit legacy metadata fields (Evaluation_Type, Category, Level); use the classification CSV as the single source of truth for category breakdowns.

Getting Started

See Install for environment setup, then Running Benchmarks to launch evaluation.

Install

Install uv once if it is not already available:

curl -LsSf https://astral.sh/uv/install.sh | sh

Use the matching environment directory for the backend you want to run:

cd envs/ai2thor            # or: procthor / virtualhome / carla / embodiedcity / game
uv sync
source .venv/bin/activate

There is also a top-level grouped environment definition:

cd envs
uv sync --group ai2thor

The Python dependencies only prepare the client code. CARLA, VirtualHome, and EmbodiedCity also require their simulator runtime, dataset, or Unreal/Unity executable to be configured as described below.

Project Structure

SpatialWorld/
|-- actions/                # Unified action space and parsers
|-- configs/                # Default configs, one per environment
|   |-- ai2thor/dual/       # Default multi-agent AI2-THOR config
|   |-- procthor/dual/      # Default multi-agent ProcTHOR config
|   `-- embodiedcity/       # EmbodiedCity runtime templates
|-- data/                   # Task definitions, JSON data, game levels, assets
|-- envs/                   # uv environment definitions and env wrappers
|-- evaluation/             # Evaluation metrics and verifiers
|-- experiments/
|   |-- configs/            # Model-specific and historical configs
|   `-- csv/                # Experiment CSV files
|-- game/                   # Game environments and data generation scripts
|-- mllm_base_agent/        # Agent runner, LLM provider, prompts, env wrappers
|   `-- dual_agent/
|       |-- ai2thor/        # Multi-agent AI2-THOR implementation
|       `-- procthor/       # Multi-agent ProcTHOR implementation
|-- scripts/                # Environment CLI entry points
|   |-- ai2thor/            # `work/run_task.py` and `run_benchmark.py`
|   |-- procthor/           # `work/run_task.py` and `run_benchmark.py`
|   |-- carla/              # `work/run_task.py` and `run_benchmark.py`
|   |-- virtualhome/        # `work/run_task.py` and `run_benchmark.py`
|   |-- embodiedcity/       # `work/run_task.py`
|   `-- game/               # `run_benchmark.py`
|-- spatialworld/           # Package modules for shared runtime utilities
`-- tests/

Configuration files are grouped by environment. Dual-agent default configs are kept under configs/<environment>/dual/; model-specific experiment configs and CSV trackers stay under experiments/.

Standard Evaluation Files

Environment	CSV	Default Config
AI2-THOR	experiments/csv/ai2thor/Spatial-Annotation-ai2thor-gpt-5.csv	experiments/configs/ai2thor/config_close_gpt-5.yaml
Multi-AI2-THOR	experiments/csv/ai2thor/dual/Spatial-Annotation-ai2thor-Gemini-2.5-pro.csv	experiments/configs/ai2thor/dual/config_close_Gemini-2.5-pro.yaml
ProcTHOR	experiments/csv/procthor/Spatial-Annotation-procthor-gpt-5.csv	experiments/configs/procthor/config_close_gpt-5.yaml
Multi-ProcTHOR	experiments/csv/procthor/dual/Spatial-Annotation-procthor-Gpt-5p4.csv	experiments/configs/procthor/dual/config_close_Gpt-5p4.yaml
CARLA	experiments/csv/carla/Spatial-Annotation-carla-gpt-5.csv	experiments/configs/carla/config_close_gpt-5.yaml
VirtualHome	experiments/csv/virtualhome/Spatial-Annotation-virtualhome-gpt-5.csv	experiments/configs/virtualhome/config_close_gpt-5.yaml
EmbodiedCity	experiments/csv/embodiedcity/Spatial-Annotation-embodiedcity-gpt-5.4.csv	experiments/configs/embodiedcity/vln-agent-config-gpt54.yaml
Game	experiments/csv/game/*.csv	configs/game/*_config.py

Before evaluation, fill the target config's model block (provider, model_name, base_url, api_key, temperature/token options) for your VLM endpoint. To reproduce a model-specific experiment, use the matching config from experiments/configs/<environment>/.

Simulator Backends

AI2-THOR

AI2-THOR does not require a separate simulator executable. Installing the ai2thor dependency is enough; the package manages the Unity build used by ai2thor.controller.Controller.

cd envs/ai2thor
uv sync
cd ../..

Key config file: experiments/configs/ai2thor/config_close_gpt-5.yaml.

Important fields include render size (env.width, env.height), camera FOV, navigation grid size, visibility distance, platform/headless rendering, and movement/rotation action settings.

Configure your VLM credentials before running. The OpenAI-compatible provider uses model.vlm.api_key when set, otherwise it reads OPENAI_API_KEY; base_url can also be supplied through OPENAI_BASE_URL when omitted from the config. Do not commit real API keys to the default config.

Smoke test:

envs/ai2thor/.venv/bin/python -u -m scripts.ai2thor.work.run_task \
  --config experiments/configs/ai2thor/config_close_gpt-5.yaml \
  --scene FloorPlan1 \
  --tasks open_fridge

For machines without a display server, add --headless to the smoke-test command. This sets env.platform to CloudRendering for the run.

ProcTHOR

ProcTHOR uses AI2-THOR for rendering plus the ProcTHOR-10K house dataset. Install the simulator dependency and prior through the ProcTHOR environment.

cd envs/procthor
uv sync
source .venv/bin/activate

Dataset reference: https://github.com/allenai/procthor-10k.

The code searches for the dataset in this order:

1. PROCTHOR_DATASET_DIR
2. procthor/datasets/procthor-10k
3. the local prior cache

To create a local dataset copy, either let prior download it on first run or run:

python -c "import prior; dataset = prior.load_dataset('procthor-10k'); print(dataset.keys())"

If you keep the dataset outside the repo, point the scripts to it:

export PROCTHOR_DATASET_DIR=/data/procthor-10k

Key config file: experiments/configs/procthor/config_close_gpt-5.yaml.

Smoke test:

python -m scripts.procthor.work.run_task \
  --config experiments/configs/procthor/config_close_gpt-5.yaml \
  --scene-index 0 \
  --headless

Multi-agent ProcTHOR uses experiments/configs/procthor/dual/config_close_gpt-5.yaml.

VirtualHome

VirtualHome requires the Python package and a Unity simulator executab