OpenEAI Platform: Open-source Embodied Artificial Intelligence Hardware-Software Unified Platform

This repository provides the official (open-source) implementation for our paper: OpenEAI-Platform, a reproducible hardware–software stack for real-world embodied manipulation. OpenEAI-Platform aims to lower the barrier for building, reproducing, and scaling real-world Embodied AI systems by releasing the complete pipeline—from robot hardware designs and low-level control to data collection, dataset processing, two-stage VLA training, and deployment interfaces.

This repository contains OpenEAI-VLA, an end-to-end vision-language-action (VLA) policy trained with a two-stage recipe (large-scale pretraining + task-specific fine-tuning). For OpenEAI-Arm, please refer to the OpenEAI-Arm repository.

Key Features

• Fully open-source end-to-end pipeline: hardware designs, control stack, dataset processing, training, and deployment.
• Reproducible two-stage training recipe: large-scale pretraining on public robot datasets + task-specific fine-tuning with a small number of demonstrations (optionally mixed with multimodal data).
• Cross-dataset alignment: dataset adapters for unifying heterogeneous state/action conventions across data sources.
• Real-time deployment: a standard robot-client / policy-server interface for streaming observations and returning action chunks.
• Extensible data collection: multiple demonstration pipelines (e.g., kinesthetic teaching, teleoperation, VR, UMI-style collection).

Installation

You can install the required packages using pip. We recommand you to use Python 3.10 or higher installed.

conda create -n openeai python=3.10 -y
conda activate openeai
pip install -r requirements.txt
pip install -e .

Dataset Preparation

Put demand pretraining and fine-tuning datasets in the data/ directory. Currently, we provide download scripts at data_utils/vla_data_utils. We provide a processed version at OpenEAI/OpenEAI-Dataset which has a total size of ~3.12TB.

Besides, you need to prepare some base models. These models may include

• t5-v1_1-xl
• siglip-so400m-patch14-384
• paligemma-3b-pt-224
• Qwen2.5-VL-3B-Instruct
• Qwen3-VL-4B-Instruct

where Qwen3-VL-4B-Instruct is necessary for our current OpenEAI-VLA training. You can download these models from huggingface. For example, to download Qwen3-VL-4B-Instruct, you can run:

huggingface-cli download Qwen/Qwen3-VL-4B-Instruct --repo-type model

Data Postprocessing

We provide data postprocessing scripts in data_utils/ to convert raw datasets into the required HDF5 format. You can run the following command to convert downloaded pretrained dataset or your own dataset to our standard format:

cd data_utils
bash run.sh <dataset_name>

Training

Pretrain

You can directly download our pretrained model at huggingface repository. Or, we provide a script for you to use your own dataset:

First, modify the pretraining configuration file in config/pretrain_openeai.json for your own dataset paths at data.data_root.

Then, you can start pretraining with single-node multi-GPU or multi-node multi-GPU. For example, to start pretraining with 8 GPUs on 4 nodes, you can run:

bash scripts/pretrain.sh <task_name>

Fine-tune

Similar to pretraining, modify the fine-tuning configuration file in config/sft_openeai_multimodal.json for your own dataset paths at data.data_root and data.multimodal_root.

Then, you can start fine-tuning with single-node multi-GPU or multi-node multi-GPU. For example, to start fine-tuning with 8 GPUs on a single node with zero2, you can run:

bash scripts/sft.sh <task_name>

Multi-node Multi-GPU Training Notes

Note: For multi-node multi-GPU training, ensure that your environment is set up for high-speed inter-node communication using InfiniBand (IB). You can verify this by checking the training logs. If you see NCCL INFO NET/IB in the logs and the training speed is around 1.5s/iteration, then your setup is correct. If you see NET/SOCKET, please reconfigure your environment to use IB by installing the necessary packages.

You can also run simple tests by running the following script:

bash scripts/test_ib.sh

Inference

To run inference with a trained model, you can use the provided FastAPI server. First, ensure you have a trained model checkpoint, then, modify CKPT_PATH in openeai/infer.py to point to your checkpoint path.

Then, start the inference server by running:

python openeai/infer.py 

Then you can send inference requests to the server. Local inference example is provided in OpenEAI-Arm/ros2/src/examples/inference.py.

Release Notes

Some resources (e.g., CAD packages, manufacturing files, pretrained checkpoints, and large-scale configs) will be released soon due to large uploading volume. Please check the module READMEs and the repository releases page for updates.

Citation

If you use this project in your research, please cite our paper:

@inproceedings{openeai_platform,
  title   = {OpenEAI-Platform: Open-source Embodied Artificial Intelligence Hardware-Software Unified Platform},
  author  = {Jinyuan Zhang, Luoyi Fan, Leiyu Wang, Yeqiang Wang, Yichen Zhu, Cewu Lu, Nanyang Ye},
  year    = {2026}
}

License

Licensed under the BSD-3-Clause License. See LICENSE for details.

Contact / Issues

If you have questions, suggestions, or would like to contribute:

• Please open an issue or submit a pull request.
• Send an e-mail to ynylincoln@sjtu.edu.cn