Continuous 3D Perception Model with Persistent State

---

Official implementation of Continuous 3D Perception Model with Persistent State, CVPR 2025 (Oral)

QianqianWang*, Yifei Zhang*, Aleksander Holynski, Alexei A Efros, Angjoo Kanazawa

(*: equal contribution)

Table of Contents

• TODO
• Get Started
• Installation
• Checkpoints
• Inference
• Datasets
• Evaluation
• Datasets
• Evaluation Scripts
• Training and Fine-tuning
• Acknowledgements
• Citation

TODO

• [x] Release multi-view stereo results of DL3DV dataset.
• [ ] Online demo integrated with WebCam

Getting Started

Installation

1. Clone CUT3R.

git clone https://github.com/CUT3R/CUT3R.git
cd CUT3R

1. Create the environment.

conda create -n cut3r python=3.11 cmake=3.14.0
conda activate cut3r
conda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia  # use the correct version of cuda for your system
pip install -r requirements.txt
# issues with pytorch dataloader, see https://github.com/pytorch/pytorch/issues/99625
conda install 'llvm-openmp<16'
# for training logging
pip install git+https://github.com/nerfstudio-project/gsplat.git
# for evaluation
pip install evo
pip install open3d

1. Compile the cuda kernels for RoPE (as in CroCo v2).

cd src/croco/models/curope/
python setup.py build_ext --inplace
cd ../../../../

Download Checkpoints

We currently provide checkpoints on Google Drive:

Modelname	Training resolutions	#Views	Head
cut3r_224_linear_4.pth	224x224	16	Linear
cut3r_512_dpt_4_64.pth	512x384, 512x336, 512x288, 512x256, 512x160, 384x512, 336x512, 288x512, 256x512, 160x512	4-64	DPT

cut3r_224_linear_4.pth is our intermediate checkpoint and cut3r_512_dpt_4_64.pth is our final checkpoint.

To download the weights, run the following commands:

cd src
# for 224 linear ckpt
gdown --fuzzy https://drive.google.com/file/d/11dAgFkWHpaOHsR6iuitlB_v4NFFBrWjy/view?usp=drive_link 
# for 512 dpt ckpt
gdown --fuzzy https://drive.google.com/file/d/1Asz-ZB3FfpzZYwunhQvNPZEUA8XUNAYD/view?usp=drive_link
cd ..

Inference

To run the inference code, you can use the following command:

# the following script will run inference offline and visualize the output with viser on port 8080
python demo.py --model_path MODEL_PATH --seq_path SEQ_PATH --size SIZE --vis_threshold VIS_THRESHOLD --output_dir OUT_DIR  # input can be a folder or a video
# Example:
#     python demo.py --model_path src/cut3r_512_dpt_4_64.pth --size 512 \
#         --seq_path examples/001 --vis_threshold 1.5 --output_dir tmp
#
#     python demo.py --model_path src/cut3r_224_linear_4.pth --size 224 \
#         --seq_path examples/001 --vis_threshold 1.5 --output_dir tmp

# the following script will run inference with global alignment and visualize the output with viser on port 8080
python demo_ga.py --model_path MODEL_PATH --seq_path SEQ_PATH --size SIZE --vis_threshold VIS_THRESHOLD --output_dir OUT_DIR

Output results will be saved to output_dir.

Currently, we accelerate the feedforward process by processing inputs in parallel within the encoder, which results in linear memory consumption as the number of frames increases.

Datasets

Our training data includes 32 datasets listed below. We provide processing scripts for all of them. Please download the datasets from their official sources, and refer to preprocess.md for processing scripts and more information about the datasets.

• ARKitScenes
• BlendedMVS
• CO3Dv2
• MegaDepth
• ScanNet++
• ScanNet
• WayMo Open dataset
• WildRGB-D
• Map-free
• TartanAir
• UnrealStereo4K
• Virtual KITTI 2
• 3D Ken Burns
• BEDLAM
• COP3D
• DL3DV
• Dynamic Replica
• EDEN
• Hypersim
• IRS
• Matterport3D
• MVImgNet
• MVS-Synth
• OmniObject3D
• PointOdyssey
• RealEstate10K
• SmartPortraits
• Spring
• Synscapes
• UASOL
• UrbanSyn
• HOI4D

Evaluation

Datasets

Please follow MonST3R and Spann3R to prepare Sintel, Bonn, KITTI, NYU-v2, TUM-dynamics, ScanNet, 7scenes and Neural-RGBD datasets.

The datasets should be organized as follows:

data/
├── 7scenes
├── bonn
├── kitti
├── neural_rgbd
├── nyu-v2
├── scannetv2
├── sintel
└── tum

Evaluation Scripts

Please refer to the eval.md for more details.

Training and Fine-tuning

Please refer to the train.md for more details.

Acknowledgements

Our code is based on the following awesome repositories:

• DUSt3R
• MonST3R
• Spann3R
• Viser

We thank the authors for releasing their code!

Citation

If you find our work useful, please cite:

@article{wang2025continuous,
  title={Continuous 3D Perception Model with Persistent State},
  author={Wang, Qianqian and Zhang, Yifei and Holynski, Aleksander and Efros, Alexei A and Kanazawa, Angjoo},
  journal={arXiv preprint arXiv:2501.12387},
  year={2025}
}