SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos

SurgMotion is a video-native foundation model that shifts the learning paradigm from pixel-level reconstruction to latent motion prediction, with technical innovations tailored to surgical videos, built on top of V-JEPA 2.

Model Overview

Key innovations: - Latent motion prediction — shifts from pixel-level reconstruction to abstract motion forecasting in latent space - Flow-Guided Latent Prediction — a novel objective that prevents feature collapse in homogeneous surgical tissue regions - Pre-trained on SurgMotion-15M — the largest multi-modal surgical video dataset to date (15M frames, 3,658 hours, 13+ anatomical regions)

Model Variants

Architecture

1. Video Encoder (ViT) — processes 64-frame surgical video clips into spatiotemporal token sequences
2. Latent Predictor — predicts masked region representations in latent space guided by optical flow
3. Probing Head — lightweight temporal classifier for downstream phase recognition

Performance Highlights

SurgMotion achieves SOTA on all representative surgical tasks (workflow, action, segmentation, triplet, skill, depth estimation). For detailed results, see our paper and project page.

Quick Start

• Setup:
• Environment Installation
• Data Preparation
• Usage:
• Run Foundation Probing
• Extend:
• Add a New Dataset
• Add a New Foundation Model

Project Structure

SurgMotion/
├── src/                        # V-JEPA2 core: ViT, VideoMAE, datasets, masks
├── evals/                      # Evaluation entry points & foundation phase probing
│   ├── main.py                 # Single-task entry: python -m evals.main --fname <yaml>
│   └── foundation_phase_probing/
│       ├── eval.py             # Probing evaluation logic
│       ├── models.py           # Probing head definitions
│       └── modelcustom/        # Per-model adapters (SurgMotion, DINOv3, SurgVLP, …)
├── configs/
│   └── foundation_model_probing/
│       ├── surgmotion/         # YAML configs per dataset
│       ├── dinov3/
│       ├── endofm/
│       ├── …                   # 15 model families supported
│       └── videomaev2/
├── data_process/               # End-to-end dataset preprocessing scripts
│   ├── autolaparo_prepare.py
│   ├── cholect80_prepare.py
│   ├── egosurgery_prepare.py
│   ├── m2cai2016_prepare.py
│   ├── ophnet_prepare.py
│   ├── pitvis_prepare.py
│   ├── pmlr50_prepare.py
│   ├── polypdiag_prepare.py
│   └── surgicalactions160_prepare.py
├── ckpts/                      # Store all the foundation models
├── scripts/                    # Batch probing & environment setup shells
├── foundation_models/          # Third-party model implementations (git submodules)
├── data/                       # Data directory
├── setup.py                    # pip install -e .
└── requirements.txt            # All dependencies (excluding EndoMamba)

Environment Installation

Main Environment (Recommended)

conda create -n SurgMotion python=3.12 -y
conda activate SurgMotion

# Install PyTorch matching your CUDA version first:
# https://pytorch.org/get-started/locally/

pip install -e .

EndoMamba (Separate Environment)

EndoMamba requires its own Conda env with custom CUDA extensions. Do not mix with the main environment.

bash scripts/srun_endomamba_complie.sh   # Creates env + compiles extensions
conda activate endomamba                 # Use only for EndoMamba configs

Dependency Files

EndoMamba has its own isolated environment managed by scripts/srun_endomamba_complie.sh.

Data Preparation

All preprocessing scripts under data_process/ now follow a unified end-to-end pipeline and support one-command execution:

python data_process/<dataset>_prepare.py --step all

Common step options:

--step all|frames|metadata|clips
--window_size 64
--stride 1
--fps 1
--no_padding

Typical outputs: - clip_infos/*.txt — per-case frame path lists - {train,val,test}_metadata.csv — frame-level metadata for dense clip generation - clips_<window_size>f/{train,val,test}_dense_<window_size>f_detailed.csv - clips_<window_size>f/clip_dense_<window_size>f_info/{train,val,test}/*.txt

Frame-level metadata schema:

Dense Sampling Strategy

We use an online workflow recognition setting: - A clip is a sliding temporal window. - The last frame in the window is the target frame to predict. - Previous frames in the window are temporal context. - Neighboring windows overlap.

For window_size=64, stride=1: - clip 1: frames [0, ..., 63] - clip 2: frames [1, ..., 64] - clip 3: frames [2, ..., 65]

Padding at video start: - If the early timeline does not have enough preceding frames, we pad the window by repeating the current window's last frame. - This behavior is enabled by default; use --no_padding to disable.

Clip Labeling Rule

For phase recognition: - Frames are sampled at 1 fps. - Clip label = label of the clip's last frame.

Example: - frames 0-40: Phase 0 - frames 41-63: Phase 1 - clip [0, ..., 63] label is Phase 1.

Notes on Performance Gaps

If reproduced results are lower than expected, dense sampling mismatch is one possible source, but not the only one. We also recommend checking: - longer training schedules (e.g., 2 / 4 / 8 epochs) - class balancing / class weighting strategy

Class weighting can strongly affect surgical long-tail performance. See implementation in evals/foundation_phase_probing/eval.py.

Supported Datasets

Most datasets already provide extracted frames in data/Surge_Frames/.... The pipelines read annotations and frames; frame extraction from videos (--step frames) is optional and only needed if you have raw mp4 files.

All annotation and frame paths above are relative to the data/ directory (e.g., data/Landscopy/cholec80/phase_annotations).

Pipeline behavior

The scripts are built around frame-based clip CSVs. Depending on whether you already have extracted frames or only raw videos, use the path that matches your data.

1) Frames-first input (recommended default)

You already have image sequences under --frames_root (e.g. data/Surge_Frames/...).

Typical command: python data_process/<dataset>_prepare.py --step all with correct --frames_root / annotation paths. No --videos_dir needed.

2) Videos-first input (optional extraction)

You only have .mp4 files and need JPEG/PNG frames under --frames_root first.

Typical two-stage flow:

python data_process/<dataset>_prepare.py --step frames --videos_dir /path/to/mp4s  # if supported
python data_process/<dataset>_prepare.py --step all

3) How --step all treats frame extraction

Quick reference: flags vs. input type

Example: Prepare Cholec80

python data_process/cholect80_prepare.py \
    --frames_root data/Surge_Frames/Cholec80/frames \
    --annot_dir data/Landscopy/cholec80/phase_annotations \
    --output_dir data/Surge_Frames/Cholec80 \
    --step all \
    --debug

Example: Prepare SurgicalActions160 (with extraction)

python data_process/surgicalactions160_prepare.py \
    --src_root data/Landscopy/SurgicalActions160 \
    --fps 1 \
    --step all

Run Foundation Probing

Supported Foundation Models

Model Preparation

1. Download Model Weights from their corresponding repos.
2. Put the Model under the ckpts folder:

# For SurgMotion
SurgMotion/ckpts 

# For Other Foundation Models
SurgMotion/ckpts/ckpts_foundation 

1. Make sure the model path align with the corresponding adapters.py:

# For Other Foundation Models
SurgMotion/evals/foundation_phase_probing/modelcustom/adapters

Single Task

```bash

SurgMotion

python -m evals.main \ --fname configs/foundation_model_probing/surgmotion/AutoLaparo/surgmotion_vitl_64f_autolaparo.yaml \ --devices cuda:0

Dinov3

python -m evals.main \ --fname configs/foundat