T2LDM

This repo is the official project repository of the paper A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation. - [ arXiv ], [ CVPR ], For the the deeper theoretical insights, the more efficient framework design, and the more comprehensive experimental extensions, please refer to [ T2LDM++ paper ]. - Our paper has been accepted by CVPR 2026! - Released model weights are temporarily as the model structure of T2LDM may be adjusted later. - The code wiil further be updated for nuScenes, KITTI360 and SemanticKITTI!

• We strongly recommend to use the extension vision of T2LDM. The extension version, [ T2LDM++ ], achieves (Box, BEV, Camera)-to-Lidar !

Overall Framework

Citation

If you find our paper useful to your research, please cite our work as an acknowledgment.

@inproceedings{qu2026self,
  title={A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation},
  author={Qu, Wentao and Mei, Guofeng and Wu, Yang and Gong, Yongshun and Huang, Xiaoshui and Xiao, Liang},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={9434--9444},
  year={2026}
}

Codebase

T2LDM is built upon the codebase of R2DM and Text2LiDAR. We further refine and optimize this codebase with the hope of contributing to the LiDAR generation community.

1. Simple Dataset Form

The original dataset follows a TensorFlow-style design, which is not convenient for reading and debugging. T2LDM follows a PyTorch-style implementation and introduces a validation dataset(ConditionalX0) to facilitate monitoring intermediate generation results.

  # If you want to create your dataset, please check:
    data/new_dataset/new_dataset.py

2. Supporting Multi-GPU Generation

We rewrote the generation code to support generation samples using multiple GPUs.

  # Please check:
    generation_mgpus_{task}_{dataset}.py

3. Friendly Generation Result

The original code only produces the BEV PNGs from generation Range Maps.

Our code directly transfer RMs to PCs for results.

For Text-to-LiDAR, the code saves texts and PCs.

For Semantic-to-LiDAR, the code saves semantics, GT PCs and generated colorful PCs.

  # Please check:
    utils/lidar.py

4. Flexible Network Framework

The network framework of T2LDM is flexible. We can easily adjust the framework only though changing the network list.

  # Please check:
    models/T2LDM.py

  def get_encoder_deocder_gn():
    ...

  def get_encoder_deocder_dn():
    ...

5. Supporting T5 Text Encoder

We add [ T5 ] on T2LDM.

  # Please check: 
    models/T5/T5.py

  # Change the config file: utils/config_text_nuScenes.py
    clip_mode: None # closing the CLIP Text Encoder
    T5_mode: "base" # small(512), base(768), large(1024)

6. Provide an example of PC to RM conversion

An example of PC to RM conversion is provided to help you understand the conversion process.

 # Please check:
   utils/lidar.py
   python lidar.py

Adding Some Tricks from Papers

1) [ JIT(Back to Basics: Let Denoising Generative Models Denoise) ] 2) [ Gated Attention(Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free, NeurIPS 2025, Best Paper) ] 3) [ FreeU(FreeU: Free Lunch in Diffusion U-Net, CVPR 2024, Oral) ] 4) [ ScaleLong(ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection, NeurIPS 2023) ]

Overview

• Installation
• Data Preparation
• Model Zoo
• Quick Start

Installation

Requirements

The following environment is recommended for running T2LDM (four NVIDIA 3090 GPUs or eight NVIDIA 4090 GPUs):

If you only want to generate some LiDAR results (64 Steps or 1024 Steps, the GPU memory < 1G on BS=1), the Single 3090(24G)/3060(12G) GPU is enough! - Ubuntu: 18.04 and above - gcc/g++: 11.4 and above - CUDA: 12.1 - PyTorch: 2.1.0 - python: 3.10

Environment

Using environments.yaml (based on conda command)

  cd envs
  conda env create -f environment.yaml

  # If you want to conduct sparse-to-dense/dense-to-sparse experiments.
  cd ../pointops
  python setup.py install

Using requirements.txt (based on pip command)

  conda create -n t2ldm python=3.10 -y
  conda activate t2ldm

  cd envs

  pip install -r requirements.txt

  pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121
  pip install ema-pytorch==0.4.8 kornia==0.7.0 accelerate==0.22.0

  # If you want to conduct sparse-to-dense/dense-to-sparse experiments.
  cd ../pointops
  python setup.py install

Data Preparation

nuScenes

• 1. Download the official nuScenes (or Baidu Disk(code:1111)) dataset (with Lidar Segmentation) and organize the downloaded files as follows: bash /root/dataset/rsd_data/nuscenes/v1.0-trainval │── samples │── sweeps │── lidarseg ... │── v1.0-trainval │── v1.0-test

1. T2LnuScenes (34149 Text-LiDAR Pairs)

• 1. Download the pkl files (nuscenes_infos_10sweeps_train.pkl and nuscenes_infos_10sweeps_val.pkl) from the Huggingface project of HuggingFace. # Running data/nuScenes/descriptor.py to generate nuscenes_infos_10sweeps_description.pkl. ROOT_PATH = "Your the path of pkl files" python descriptor.py
• 1. Of course, you also can download generate nuscenes_infos_10sweeps_description.pkl from HuggingFace. This size is about 9.33GB.

  Meanwhile, you can only download point cloud files of nuScenes (LiDAR_TOP.zip): HuggingFace.

  Please put LiDAR_TOP.zip on nuScenes/v1.0-trainval/samples/LiDAR_TOP.zip, and unzip it.

  This means that this is allowed you to retrain T2LDM using generate nuscenes_infos_10sweeps_description.pkl + point cloud files of nuScenes.

2. T2LnuScenes++ (107816 Text-LiDAR Pairs)

Please check data/nuScenes/description_plus_class.txt and data/nuScenes/description_plus_text.txt to know the related information for T2LnuScenes++.

• 1. Generating the generate nuscenes_infos_10sweeps_description.pkl + point cloud files of nuScenes. # Running data/nuScenes/descriptor_plus.py to generate text.pkl. ROOT_PATH = "Your the path of pkl files" python descriptor_plus.py
• 1. Of course, you also can download text.pkl from HuggingFace.
• 1. If you want to train sparse-to-dense/dense-to-sparse models or generate the samples, please download LIDAR_TOP_DOWNSAMPLING.zip. Then, put it on nuScenes/v1.0-trainval/samples/LIDAR_TOP_DOWNSAMPLING.zip and unzip it.

KITTI360

• 1. Dowload the official KITTI360 (Raw Velodyne Scans (119G)) and organize the download files as follows:

  /root/dataset/KITTI360/data_3d_raw
  │── 2013_05_28_drive_0000_sync
  │── 2013_05_28_drive_0002_sync
  │── 2013_05_28_drive_0003_sync
  ...
  │── 2013_05_28_drive_0009_sync
  │── 2013_05_28_drive_0010_sync

• 1. Of course, you also download the KITTI360 address on HuggingFace.

SemanticKITTI

• 1. Dowload the official [SemanticKITTI (https://semantic-kitti.org/dataset.html) and organize the download files as follows:

  /root/dataset/SemanticKITTI/dataset/sequences
  │── 00
  │── 01
  │── 02
  ...
  │── 20
  │── 21

Model Zoo

We create a Huggingface project (QWTforHuggingFace/T2LDM) for HuggingFace. Please download something from Huggingface.

I am very sorry for no space of my Google Cloud Disk. Please download something by the Baidu Disk or HuggingFace.

As I rewrote the code of T2LDM, I have to retrain T2LDM. However, I currently don't have eight 4090 NVIDIA GPUs, so T2LDM is retrained on four 3090 NVIDIA GPUs.

nuScenes

Model	Task	Samples	checkpoint
Frozen SCRG on 10W Steps	Unconditional Generation	HuggingFace	HuggingFace
Full Training SCRG on 40W Steps	Unconditional Generation	Baidu Disk, Google Cloud Disk, HuggingFace	Baidu Disk, Google Disk, HuggingFace
Frozen SCRG on 10W Steps	Text-guied Generation	HuggingFace	HuggingFace
Full Training SCRG on 40W Steps	Text-guided Generation	HuggingFace	HuggingFace
40W Steps	Semantic-to-LiDAR Generation	HuggingFace	HuggingFace
40W Steps	Sparse-to-Dense Generation	HuggingFace	HuggingFace
40W Steps	Dense-to-Sparse Generation	HuggingFace	HuggingFace

Some results from T2LDM with the Full Training SCRG checkpoint on nuScenes.

KITTI360

Model	Task	Samples	checkpoint
Frozen SCRG on 10W Steps	Unconditional Generation	HuggingFace	HuggingFace
Full Training SCRG on 40W Steps	Unconditional Generation	HuggingFace	HuggingFace

Some results from T2LDM with the Full Training SCRG checkpoint on KITTI360 (Sorry, I am too lazy to draw. Please see samples).

SemanticKITTI

Model	Task	Samples	checkpoint
Frozen SCRG on 10W Steps	Unconditional Generation	--	--
Full Training SCRG on 40W Steps	Unconditional Generation	HuggingFace	[H