Lidar AI Solution

This is a highly optimized solution for self-driving 3D-lidar repository. It does a great job of speeding up sparse convolution/CenterPoint/BEVFusion/OSD/Conversion.

Pipeline overview

GetStart

$ git clone --recursive https://github.com/NVIDIA-AI-IOT/Lidar_AI_Solution
$ cd Lidar_AI_Solution

• For each specific task please refer to the readme in the sub-folder.

3D Sparse Convolution

A tiny inference engine for 3d sparse convolutional networks using int8/fp16. - Tiny Engine: Tiny Lidar-Backbone inference engine independent of TensorRT. - Flexible: Build execution graph from ONNX. - Easy To Use: Simple interface and onnx export solution. - High Fidelity: Low accuracy drop on nuScenes validation. - Low Memory: 422MB@SCN FP16, 426MB@SCN INT8. - Compact: Based on the CUDA kernels and independent of cutlass.

CUDA BEVFusion

CUDA & TensorRT solution for BEVFusion inference, including: - Camera Encoder: ResNet50 and finetuned BEV pooling with TensorRT and onnx export solution. - Lidar Encoder: Tiny Lidar-Backbone inference independent of TensorRT and onnx export solution. - Feature Fusion: Camera & Lidar feature fuser with TensorRT and onnx export solution. - Pre/Postprocess: Interval precomputing, lidar voxelization, feature decoder with CUDA kernels. - Easy To Use: Preparation, inference, evaluation all in one to reproduce torch Impl accuracy. - PTQ: Quantization solutions for mmdet3d/spconv, Easy to understand.

CUDA CenterPoint

CUDA & TensorRT solution for CenterPoint inference, including: - Preprocess: Voxelization with CUDA kernel - Encoder: 3D backbone with NV spconv-scn and onnx export solution. - Neck & Header: RPN & CenterHead with TensorRT and onnx export solution. - Postprocess: Decode & NMS with CUDA kernel - Easy To Use: Preparation, inference, evaluation all in one to reproduce torch Impl accuracy. - QAT: Quantization solutions for traveller59/spconv, Easy to understand.

CUDA PointPillars

CUDA & TensorRT solution for pointpillars inference, including: - Preprocess: Voxelization & Feature Extending with CUDA kernel - Detector: 2.5D backbone with TensorRT and onnx export solution. - Postprocess: Parse bounding box, class type and direction - Easy To Use: Preparation, inference, evaluation all in one to reproduce torch Impl accuracy.

CUDA-V2XFusion

Training and inference solutions for V2XFusion. - Easy To Use: Provides easily reproducible solutions for training, quantization, and ONNX export. - Quantification friendly：PointPillars based backbone with pre-normalization which can reduce quantization error. - Feature Fusion: Camera & Lidar feature fuser and onnx export solution. - PTQ: Quantization solutions for V2XFusion, easy to understand. - Sparsity: 4:2 structural sparsity support. - Deepstream sample: Sample inference using CUDA, TensorRT/Triton in NVIDIA DeepStream SDK 7.0.

cuOSD(CUDA On-Screen Display Library)

Draw all elements using a single CUDA kernel. - Line: Plotting lines by interpolation(Nearest or Linear). - RotateBox: Supports drawn with different border colors and fill colors. - Circle: Supports drawn with different border colors and fill colors. - Rectangle: Supports drawn with different border colors and fill colors. - Text: Supports stb_truetype and pango-cairo backends, allowing fonts to be read via TTF or using font-family. - Arrow: Combination of arrows by 3 lines. - Point: Plotting points by interpolation(Nearest or Linear). - Clock: Time plotting based on text support

cuPCL(CUDA Point Cloud Library)

Provide several GPU accelerated Point Cloud operations with high accuracy and high performance at the same time: cuICP, cuFilter, cuSegmentation, cuOctree, cuCluster, cuNDT, Voxelization(incoming). - cuICP: CUDA accelerated iterative corresponding point vertex cloud(point-to-point) registration implementation. - cuFilter: Support CUDA accelerated features: PassThrough and VoxelGrid. - cuSegmentation: Support CUDA accelerated features: RandomSampleConsensus with a plane model. - cuOctree: Support CUDA accelerated features: Approximate Nearest Search and Radius Search. - cuCluster: Support CUDA accelerated features: Cluster based on the distance among points. - cuNDT: CUDA accelerated 3D Normal Distribution Transform registration implementation for point cloud data.

YUVToRGB(CUDA Conversion)

YUV to RGB conversion. Combine Resize/Padding/Conversion/Normalization into a single kernel function. - Most of the time, it can be bit-aligned with OpenCV. - It will give an exact result when the scaling factor is a rational number. - Better performance is usually achieved when the stride can divide by 4. - Supported Input Format: - NV12BlockLinear - NV12PitchLinear - YUV422Packed_YUYV - Supported Interpolation methods: - Nearest - Bilinear - Supported Output Data Type: - Uint8 - Float32 - Float16 - Supported Output Layout: - CHW_RGB/BGR - HWC_RGB/BGR - CHW16/32/4/RGB/BGR for DLA input - Supported Features: - Resize - Padding - Conversion - Normalization

ROI Conversion (ROIs To Continuous Tensor Conversion)

Combine Resize/Padding/Conversion/Normalization into a single kernel function. - Most of the time, it can be bit-aligned with OpenCV. - It will give an exact result when the scaling factor is a rational number. - Better performance is usually achieved when the stride can divide by 4. - Supported Input Format: - NV12BlockLinear - NV12PitchLinear - YUV422Packed_YUYV - Supported Interpolation methods: - Nearest - Bilinear - Supported Output Data Type: - Uint8 - Float32 - Float16 - Supported Output Layout: - CHW_RGB/BGR - HWC_RGB/BGR - CHW16/32/4/RGB/BGR for DLA input - Gray - Supported Features: - Resize - Padding - Conversion - Normalization

Thanks

This project makes use of a number of awesome open source libraries, including:

• stb_image for PNG and JPEG support
• pybind11 for seamless C++ / Python interop
• and others! See the dependencies folder.

Many thanks to the authors of these brilliant projects!