An Anatomy of Vision-Language-Action Models

From Modules to Milestones and Challenges

This repository accompanies the survey paper:

An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
Chao Xu, Suyu Zhang, Yang Liu, Baigui Sun, et al.

We organize the rapidly growing Vision-Language-Action (VLA) literature around core challenges, rather than architectures or tasks, following the structure of Section 4 of the paper.

🔍 What This Repository Provides

• A challenge-centric taxonomy of VLA research
• Fine-grained sub-challenge decomposition with representative papers
• A living, continuously updated survey website

🧭 Survey Roadmap

This survey follows a natural learning trajectory:

1. Basic Modules – What components make up a VLA system?
2. Milestones & Evolution – How did VLA models evolve over time?
3. Core Challenges – What are the fundamental bottlenecks today?
4. Applications – Where are VLA models deployed?

This repository focuses primarily on (3) Core Challenges.

🚧 Core Challenges in Vision-Language-Action Models

We identify five fundamental challenges, each further decomposed into sub-challenges.

⭐Starred papers are submitted by authors who contacted us. We warmly welcome submissions and encourage researchers to share their latest results.

1️⃣ Multi-Modal Alignment and Physical World Modeling

(Sec. 4.1)

4.1.1 The GAP between Semantics Perception and Physical Interaction

• JALA[2026-02-25]: Learn a predictive action embedding aligned with both inverse dynamics and real actions. [paper] [website]
• VLANeXt[2026-02-24]: Systematically dissects design choices along foundational components perception essentials and action modelling perspectives to distill a practical recipe for building strong VLA models. [paper] [website]
• SimVLA[2026-02-23]: Strictly decouple perception from control using a standard vision language backbone and a lightweight action head. [paper] [website]
• CAG[2026-02-19]: Propose Counterfactual Action Guidance as a simple yet effective dual-branch inference scheme that explicitly regularizes language conditioning in VLAs. [paper] [website]
• VISTA[2026-02-04]: Align action prediction with visual input via preference optimization on a track-following surrogate task and transfer enhanced alignment through latent-space distillation. [paper] [website]
• TaF-VLA[2026-01-28]: Grounds high-dimensional tactile observations in physical interaction forces via a tactile-force adapter using contrastive learning. [paper] [website]
• TwinBrainVLA[2026-01-20]: Synergizes a frozen generalist VLM with a trainable specialist VLM via Asymmetric Mixture-of-Transformers. [paper] [website]
• CLAP[2026-01-07]: Aligns visual latent space from videos with executable robot action space using contrastive learning. [paper] [website]
• DreamTacVLA[2025-12-29]: Utilize Hierarchical Spatial Alignment loss to fuse multi-scale sensory streams and employ a tactile world model to predict future contact dynamics. [paper] [website]
• Point-VLA[2025-12-22]: Augments language instructions with explicit visual cues like bounding boxes to resolve referential ambiguity enabling precise object-level grounding. [paper] [website]
• TwinAligner[2025-12-22]: Uses SDF reconstruction and editable 3DGS rendering for pixel level alignment while ensuring dynamic consistency by identifying rigid physics. [paper] [website]
• PhysBrain[2025-12-19]: Train an egocentric-aware embodied brain on the E2E-3M dataset to bridge vision language models with physical intelligence and enable sample-efficient VLA fine-tuning. [paper] [website]
• mimic-video[2025-12-19]: Leverage pretrained video model latents to capture physical dynamics missing from static vision language backbones. [paper] [website]
• DexGrasp-VLA[2025-12-13]: Develop an Arm-Hand Feature Enhancement module to explicitly capture distinct latent features of arm and hand movements. [paper] [website]
• InternVLA-M1⭐[2025-10-15]: Employs a two-stage pipeline combining spatial grounding pre-training with spatially guided action post-training to bridge instructions and actions. [paper] [website]
• WALL-OSS⭐[2025-09-08]: Employ a tightly coupled Mixture-of-Experts architecture with static routing to align action and vision-language features. [paper] [website]
• ACT-LLM[2025]: Formulates raw per- ceptual inputs into structured language representation. [paper] [website]
• Humanoid-VLA[2025]: Pre-training language-action. [paper] [website]
• Orion[2025]: Introduces a high-level VLM planner with a separate low-level motion controller. [paper] [website]
• Gemini RObotics[2025]: Introduces a high-level VLM planner with a separate low-level motion controller. [paper] [website]
• KnowledgeVLA[2025]: Introduces a high-level VLM planner with a separate low-level motion controller. [paper] [website]
• Beyond Sight[2025]: Incorporating additional modalities. [paper] [website]
• TouchVLA[2025]: Incorporating additional modalities. [paper] [website]
• TLA[2025]: Introduces tactile perception. [paper] [website]
• OmniVTLA[2025]: Constructs a se mantically aligned tactile encoder. [paper] [website]
• Tactile-VLA[2025]: Ranging from deep fusion across the full pipeline. [paper] [website]
• ForceVLA[2025]: Modular mixture-of-experts (MoE) fusion. [paper] [website]
• MultiGen[2025]: Uses multimodal generation for simulated multi-modal data. [paper] [website]
• Grounding MLLMs[2024]: Fine-tuning a pretrained VLM to directly output action tokens. [paper] [website]
• OpenVLA[2024]: Fine-tuning a pretrained VLM to directly output action tokens. [paper] [website]
• CLIP-RT[2024]: Extends CLIP-style vision-language alignment. [paper] [website]
• LIV[2023]: Introdues a contrastive framework on robot-control data to construct a joint vision-language embedding space. [paper] [website]
• Look-Leap[2023]: Structured action-plan generation from visual inputs. [paper] [website]
• RT-2[2023]: Fine-tuning a pretrained VLM to directly output action tokens. [paper] [website]
• Prompt-a-Robot-to-Walk[2023]: Fine-tuning a pretrained VLM to directly output action tokens. [paper] [website]
• VoxPoser[2023]: Generate intermediate programs and 3D affordance maps as strong intermediate representations by LLM. [paper] [website]
• RH-20T[2023]: Incorporating additional modalities. [paper] [website]
• OTTER[2021]: Introdues a text-aware feature extraction which preserves semantics aligned with task descriptions. [paper] [website]

4.1.2 From 2D Images to Spatial Temporal Representations

• IVRA[2026-01-22]: Inject affinity hints from the frozen vision encoder into language model layers to restore spatial structure without retraining. [paper] [website]
• ActiveVLA[2026-01-13]: Projects 3D inputs onto multi-view 2D representations to identify critical regions and uses active view selection with 3D zoom-in for precise manipulation. [paper] [website]
• Real2Edit2Real[2025-12-22]: Reconstruct metric scale geometry from RGB observations and use depth as a control interface for video generation. [paper] [website]
• GeoPredict[2025-12-18]: Constructs a predictive 3D Gaussian geometry module to forecast workspace geometry with track guided refinement along future keypoint trajectories. [paper] [website]
• HiF-VLA[2025-12-10]: Leverage low-dimensional motion vectors for hindsight foresight and action fusion in a unified latent space. [paper] [website]
• Lumo-1[2025-12-10]: Build a vision language action model with spatial action tokenization and structured reasoning. [paper] [website]
• GLaD[2025-12-10]: Distill geometric features from a frozen VGGT teacher into LLM hidden states for visual tokens to fuse 3D priors. [paper] [website]
• VLA-4D[2025-11-21]: Embed 3D positions and 1D time into visual features and extend actions with temporal variables for coherent manipulation. [paper] [website]
• CronusVLA⭐[2025-10-30]: Proposes a two-stage framework with single-frame pretraining and multi-frame post-training using feature chunking to aggregate temporal information. [paper] [website]
• WALL-OSS⭐[2025-09-08]: Leverage embodied VQA and discrete action priors to strengthen spatial reasoning and progress modeling. [paper] [website]
• RoboFlamingo-Plus[2025]: Fuses preprocessed depth maps with RGB features. [paper] [website]
• PointVLA[2025]: integrates point cloud inputs into pretrained VLA models to improve spatial reasoning without modifying the backbone. [paper] [website]
• GeoVLA[2025]: unify 2D and 3D modalities. [paper] [website]
• FP3[2025]: Uses a point-cloud–centric pipeline reconstruction. [paper] [website]
• SoFar[2025]: Constructs semantic 3D scene graphs by integrating VLM-recognized objects and point-cloud orientation cues. [paper] [website]
• Weakly-Supervised 3D[2025]: Leverages CLIP’s 2D–text alignment for weakly supervised 3D semantic transfer. [paper] [website]
• LLM-3DP[2025]: Fuses 2D semantic features via back-projection with point-cloud geometry for unified semantic-geometric representation. [paper] [website]
• ARM4R[2025]: Learns space–time coupling by predicting the evolution of 3D point trajectories. [paper] [website]
• SpatialVLA[2025]: Uses positional encoding and adaptive spatial