MCPcopy Index your code
hub / github.com/Ar-Ray-code/lingbot-depth-trt

github.com/Ar-Ray-code/lingbot-depth-trt @main

Chat with this repo
repository ↗ · DeepWiki ↗ · + Follow
118 symbols 293 edges 7 files 2 documented · 2%
What it actually does AI analysis from the code graph — generated when you open this
loading…
README

lingbot-depth-trt

TensorRT conversion tools and an RGB-D camera live demo for LingBot-Depth.

LingBot-Depth is not vendored in this repository. Please refer to and cite the upstream project.

Setup

  • Ubuntu 26.04 (NVIDIA driver version : 595.71.05)
  • RTX 3060 (12GB RAM)
  • uv 0.11.22 (x86_64-unknown-linux-gnu)
  • RealSense D405 / Orbbec Gemini 335
cd python
uv venv --python 3.12
uv sync

RealSense

RealSense support is optional. Install the realsense extra when using Intel RealSense:

cd python
uv sync --extra realsense

Orbbec

Orbbec support is optional. Install the orbbec extra when using Orbbec Gemini 335:

cd python
uv sync --extra orbbec

To install both camera backends:

cd python
uv sync --extra realsense --extra orbbec

If you are installing from requirements.txt instead of uv sync, install pyrealsense2 and/or pyorbbecsdk2 separately.

If you want to use a local LingBot-Depth checkout instead of the GitHub dependency:

cd python
uv pip install -e ../../lingbot-depth

Download Model

Download the pretrained model from Hugging Face:

cd python
mkdir -p ../output/models
uv run hf download \
  robbyant/lingbot-depth-pretrain-vitl-14-v0.5 \
  model.pt \
  --local-dir ../output/models

The model page is:

https://huggingface.co/robbyant/lingbot-depth-pretrain-vitl-14-v0.5/blob/main/model.pt

Convert to TensorRT

Convert the model to a fixed-shape TensorRT engine. The default shape is 640 x 480.

cd python
uv run python ../tools/export_trt.py \
  --model ../output/models/model.pt \
  --precision fp16 \
  --num-tokens 1200 \
  --work-dir ../output/trt_nt1200_fp16

The generated TensorRT engine is written to:

output/trt_nt1200_fp16/lingbot_depth_nt1200.engine

To build another fixed input size, pass --width and --height.

You may pass --capture /path/to/capture for smoke validation with real RGB-D data. The capture directory should contain rgb.png and raw_depth.png. It is optional and is not required for conversion.

Demo

RealSense

Requires the realsense extra.

List RealSense and Orbbec devices:

cd python
uv run python scripts/live_demo.py --list-devices

Run RealSense with an OpenCV display window:

cd python
uv run python scripts/live_demo.py \
  --model ../output/trt_nt1200_fp16/lingbot_depth_nt1200.engine \
  --camera realsense \
  --realsense auto \
  --show-display

Run with a viser point cloud viewer:

cd python
uv run python scripts/live_demo.py \
  --model ../output/trt_nt1200_fp16/lingbot_depth_nt1200.engine \
  --camera realsense \
  --realsense auto \
  --show-viser

Orbbec

Requires the orbbec extra.

Run Orbbec Gemini 335 with an OpenCV display window:

cd python
uv run python scripts/live_demo.py \
  --model ../output/trt_nt1200_fp16/lingbot_depth_nt1200.engine \
  --camera orbbec \
  --orbbec auto \
  --show-display

Run with a viser point cloud viewer:

cd python
uv run python scripts/live_demo.py \
  --model ../output/trt_nt1200_fp16/lingbot_depth_nt1200.engine \
  --camera orbbec \
  --orbbec auto \
  --show-viser

Open http://localhost:8080 in a browser. In the default --viser-mode both mode, this is a comparison dashboard with two independent viser viewers. Raw and refined point clouds each have their own coordinate axes, and camera movement in either viewer is synchronized to the other. The raw/refined viewer ports are printed at startup; with the default dashboard port they are normally 8081 and 8082. Use --viser-mode raw or --viser-mode refined to start with only one point cloud.

Click a visible point to inspect the matching pixel in both viewers. The Selection field shows x/y/z coordinates from the axis origin and distance in meters. Axis size adjusts the coordinate frame size and is synchronized between the two viewers in both mode.

Orbbec RGB-D IMU Tracking

Run browser-only tracking with raw depth vs TensorRT-filtered depth. The default tracking engine is the smaller nt100 engine:

output/trt_nt100_fp16/lingbot_depth_nt100.engine

Start the viser viewer:

cd python
uv run python scripts/orbbec_tracking_demo.py \
  --orbbec auto \
  --metrics-csv ../output/orbbec_tracking_metrics.csv

Open the printed URL, normally http://localhost:8090. The viewer shows raw and filtered point clouds, their camera trajectories, an IMU orientation frame, and a 10 cm XYZ grid over a 2 m workspace.

Citation

@article{lingbot-depth2026,
  title={Masked Depth Modeling for Spatial Perception},
  author={Tan, Bin and Sun, Changjiang and Qin, Xiage and Adai, Hanat and Fu, Zelin and Zhou, Tianxiang and Zhang, Han and Xu, Yinghao and Zhu, Xing and Shen, Yujun and Xue, Nan},
  journal={arXiv preprint arXiv:2601.17895},
  year={2026}
}
@article{oquab2023dinov2,
  title={DINOv2: Learning Robust Visual Features without Supervision},
  author={Oquab, Maxime and Darcet, Timothée and Moutakanni, Theo and Vo, Huy and Szafraniec, Marc and Khalidov, Vasil and Fernandez, Pierre and Haziza, Daniel and Massa, Francisco and El-Nouby, Alaaeldin and others},
  journal={Transactions on Machine Learning Research},
  year={2024}
}

Core symbols most depended-on inside this repo

update
called by 8
python/scripts/viser_pointcloud_viewer.py
get_port
called by 6
python/scripts/viser_pointcloud_viewer.py
_reshape_color
called by 6
python/scripts/camera/orbbec.py
_selector
called by 4
python/scripts/camera/__init__.py
_ob
called by 4
python/scripts/camera/orbbec.py
summarize_depth
called by 3
tools/export_trt.py
display_host
called by 3
python/scripts/viser_pointcloud_viewer.py
trt_dtype_to_torch
called by 3
python/scripts/live_demo.py

Shape

Method 54
Function 49
Class 15

Languages

Python100%

Modules by API surface

python/scripts/viser_pointcloud_viewer.py42 symbols
tools/export_trt.py19 symbols
python/scripts/camera/orbbec.py17 symbols
python/scripts/live_demo.py13 symbols
python/scripts/camera/realsense.py12 symbols
python/scripts/camera/base.py10 symbols
python/scripts/camera/__init__.py5 symbols

For agents

$ claude mcp add lingbot-depth-trt \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact