SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Models (RSS 2025)

A spatial-enhanced vision-language-action model trained on 1.1 Million real robot episodes. 🤗 purely huggingFace-based, concise code with efficient performance.

Delin Qu*^1,2, HaomingSong*^1,3, Qizhi Chen*^1,4, Dong Wang†¹, Yuanqi Yao¹, X. Ye¹, Y. Ding¹, Z. Wang¹, Jiayuan Gu⁵, Bin Zhao†¹, Xuelong Li^1,6 Shanghai AI Laboratory¹, Fudan University², Shanghai Jiao Tong University³, Zhejiang University⁴, ShanghaiTech⁵, TeleAI⁶

[📄Paper] [🔥Project Page] [📖 Document] [🚀 Quick Start] [🤗 Model Zoo] [✅ Performance] [🙋 FAQs]

[🔥Pre-train] [🚀 Fine-tune] [🎄Custom Dataset]

News 🚀🚀🚀

• 2025/01/29: We release the SpatialVLA 1.0. SpatialVLA achieves state-of-the-art performance across a diverse range of evaluations and shows significantly faster inference speed with fewer tokens per action.
• 2025/02/06: We release the SimplerEnv evaluation code for SpatialVLA. Please refer to DelinQu/SimplerEnv-OpenVLA, and make sure transformers >= 4.47.0.
• 2025/03/16: Simplify the code structure and fix the dependencies conflict in issue #19.

[!NOTE] 🔥 An advanced version of SpatialVLA is under development! It leverages lerobot to simplify and accelerate data loading, supports multi-view and state inputs, and features a more streamlined code structure with enhanced performance! Please check out the lerobot-branch

Documents

🚀 Quick Start

[!TIP] During the runtime process, a large amount of data is cached in the CPU content. To better manage and allocate content, we have replaced the memory management tool library with tcmalloc.

For users with sudo privileges, you can install tcmalloc using sudo apt-get install google-perftools and find the libtcmalloc.so.4 library in /usr/lib/x86_64-linux-gnu or /usr/lib.

For users without sudo privileges, you can download the suitable version for your operating system from official repo and install it manually.

This step is not necessary and can be skipped based on your individual memory requirements.

SpatialVLA relies solely on HuggingFace Transformers 🤗, making deployment extremely easy. If your environment supports transformers >= 4.47.0, you can directly use the following code to load the model and perform inference. (requires 8.5GB of GPU memory).

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

model_name_or_path="IPEC-COMMUNITY/spatialvla-4b-224-pt"
processor = AutoProcessor.from_pretrained(model_name_or_path, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name_or_path, trust_remote_code=True, torch_dtype=torch.bfloat16).eval().cuda()

image = Image.open("example.png").convert("RGB")
prompt = "What action should the robot take to pick the cup?"
inputs = processor(images=[image], text=prompt, return_tensors="pt")
generation_outputs = model.predict_action(inputs)

actions = processor.decode_actions(generation_outputs, unnorm_key="bridge_orig/1.0.0")
print(actions)

If you want to use the model for fine-tuning or pre-training, you need to install the required packages and download the model from the Hugging Face model hub. The VLM backbone of SpatialVLA is PaLiGemma2, which requires transformers >= 4.47.0. Hence, create a Python environment with Python >= 3.10.

git clone git@github.com:SpatialVLA/SpatialVLA.git --depth 1
conda create -n spatialvla python=3.10
conda activate spatialvla

Install packages from requirements.txt file. Note that we use a customised dlimp to support seed setting for reproducibility. If you catch any problems, please manually install the dlimp form the dlimp_custom.

pip install -r requirements.txt

🌟 Pre-train from Scratch

SpatialVLA is pre-trained with 1.1 Million real-robot demonstrations from the OXE and RH20T dataset on a cluster of 64 A100 GPUs for abut 10 days, using a batch size of 2048. You can pre-train the model from scratch using the following command. Before running the script, please download the Open X-Embodiment dataset and RH20T dataset (optional). Please also filter the dataset by following the instructions in the moojink/rlds_dataset_builder and spatialvla/rh20t to filter the dataset or convert it to the RLDS format.

# download paligemma2 and zoe depth
bash scripts/hf_download.sh

# torchrun
bash scripts/spatialvla_4b_pretrain/torchrun_pretrain.sh

# or in a slurm cluster
bash scripts/spatialvla_4b_pretrain/slurm_pretrain.sh

🌟 Fine-tune from SpatialVLA

Most of our fine-tuning experiments are conducted using LoRA on 4 or 8 A100 GPUs. You can use the following scripts for full-parameter or LoRA fine-tuning. For real-world experiments with small datasets, we prefer using LoRA for fine-tuning.

# full fine-tuning
bash scripts/spatialvla_4b_finetune/finetune_full.sh

# LoRA fine-tuning
bash scripts/spatialvla_4b_finetune/finetune_lora.sh

🌟 SimplerEnv Benchmark

We release the SimplerEnv evaluation code for SpatialVLA based on DelinQu/SimplerEnv-OpenVLA. Please install the simpler_env environment by following DelinQu/SimplerEnv-OpenVLA and make sure transformers >= 4.47.0. Please refer to the Please refer to the Model Zoo for the model and dataset settings. After install all the dependencies, you can perform the evaluation by:

# under the project dir of SimplerEnv-OpenVLA/
bash scripts/run_spatialvla.sh

Note: Similar to most papers, e.g., HPT and TraceVLA, we omitted the Open Top Drawer and Place Apple from our evaluation, since the vast majority of policies achieved scores approaching 0 on this task.

🎄 Use Custom Datasets

To train on a custom dataset that is not part of OXE, we recommend converting it into the RLDS format, as this format directly aligns with our framework.

Once the dataset is converted, you’ll need to modify the following files: - data/oxe/mixtures.py: Define a new mixture for your dataset in the OXE_NAMED_MIXTURES dictionary. - data/oxe/configs.py: Add a new configuration specifying your dataset’s observation and action spaces to the OXE_DATASET_CONFIGS dictionary. - data/oxe/transforms.py: Define a new dataset transform function for your dataset, and add it to the OXE_STANDARDIZATION_TRANSFORMS registry at the bottom of the file.

🤗 Model Zoo

Model Name	Backbone	VLA Model	Note
SpatialVLA-4B-224-pt	google/paligemma2-3b-pt-224	spatialvla-4b-224-pt	pretrained on openx and rh20t, TABLE I and II zero-shot, Fig.5 and 7
SpatialVLA-4B-mix-224-pt	spatialvla-4b-224-pt	spatialvla-4b-mix-224-pt	fine-tuning on the fractal and bridge mixture dataset, Fig.5 and 7
spatialvla-4b-224-sft-bridge	spatialvla-4b-224-pt	spatialvla-4b-224-sft-bridge	fine-tuning on the bridge dataset, testing on simple-env widowx-robot, TABLE I fine-tuning
spatialvla-4b-224-sft-bridge	spatialvla-4b-224-pt	spatialvla-4b-224-sft-fractal	fine-tuning on the fractal dataset, testing on simple-env google-robot, TABLE II ine-tuning

✅ Performance in Simulation and Real-world

[!NOTE] SimplerEnv evaluation on Google Robot tasks.

Model	Visual Matching	Variant Aggregation
Pick Coke Can	Move Near	Open/Close Drawer	#Average	Pick Coke Can	Move Near	Open/Close Drawer	#Average
RT-1 (Begin)	2.7%	5.0%	13.9%	6.8%	2.2%	4.0%	6.9%	4.2%
RT-1 (15%)	71.0%	35.4%	56.5%	60.2%	81.3%	44.6%	26.7%	56.2%
RT-1 (Converged)	85.7%	44.2%	73.0%	74.6%	89.8%	50.0%	32.3%	63.3%
HPT	56.0%	60.0%	24.0%	46.0%	--	--	31.0%	45.0%
TraceVLA	28.0%	53.7%	57.0%	42.0%	60.0%	56.4%	29.4%	39.6%
RT-1-X	56.7%	31.7%	59.7%	53.4%	49.0%	32.3%	35.3%	64.3%
RT-2-X	78.7%	77.9%	25.0%	60.7%	82.3%	79.2%	--	--
Octo-Base	17.0%	4.2%	22.7%	16.8%	0.6%	3.1%	1.1%	1.1%
OpenVLA	16.3%	46.2%	35.6%	27.7%	54.5%	47.7%	17.7%	39.8%
RoboVLM (zero-shot)	72.7%	66.3%	26.8%	56.3%	68.3%	56.0%	8.5%	46.3%
RoboVLM (fine-tuning)	77.3%	61.7%	43.5%	63.4%	75.6%	60.0%	10.6%	51.3%
SpatialVLA (zero-shot)	81.0%	69.6%	59.3%	71.9%	89.5%	71.7%	36.2%	68.8%
SpatialVLA (fine-tuning)	86.0%	77.9%	57.4%	75.1%	88.0%	72.7%	41.8%	70.7%

[!NOTE] SimplerEnv evaluation on WidowX Robot tasks.

Model	Put Spoon on Towel	Put Carrot on Plate	Stack Green Block on Yellow Block