github.com/dlp3d-ai/dlp3d.ai @v2.1.0

<img src="https://github.com/dlp3d-ai/dlp3d.ai/raw/v2.1.0/docs/_static/banner.jpg"/>

SIGGRAPH Asia 2025 (Real-Time Live!), Hong Kong

English | 中文

Overview

Digital Life Project 2 (DLP3D) is an open-source real-time framework that brings Large Language Models (LLMs) to life through expressive 3D avatars. Users converse naturally by voice, while characters respond on demand with unified audio, whole-body animation, and physics simulation directly in the browser. Characters are fully customizable in both appearance (3D models) and personality (character prompts) and readily adaptable to any LLM or text-to-speech (TTS) service.

<a href="https://www.youtube.com/watch?v=cNEMW6IHKl8">
<img src="https://github.com/dlp3d-ai/dlp3d.ai/raw/v2.1.0/docs/_static/poster_small.jpg" alt="Watch the video">
[Click to Watch the Demo Video]
</a>

Content

This organization contains the following key repositories:

• dlp3d.ai: the main entry point, start here!
• orchestrator: coordinates and synchronizes all components.
• web_backend: manages the backend web services.
• speech2motion: converts speech into body animation.
• audio2face: generates facial animation from audio.
• MotionDataViewer: visualizes and inspects motion data.

Get Started

• Option 1: Check out Quick Start to deploy your own DLP3D locally! You’ll need to provide your own API keys, but you’ll have complete freedom to modify, extend, and integrate the system into anything you like.

• Option 2: Visit https://dlp3d-ai.github.io/dlp3d.ai/ to enjoy a ready-to-play hosted service — entirely free, including complimentary API access to selected providers (recommended for users in mainland China for the smoothest experience)!

License and Copyrights

While DLP3D itself is distributed under the MIT License, we remind users to comply with the licenses and terms of any third-party or community assets (e.g., 3D models) included or referenced in the project. Certain materials may feature characters from Genshin Impact by HoYoverse (MiHoYo). The 3D models used are from HoYoverse’s official release and are utilized in accordance with their fan content policy, which permits non-commercial, personal, and educational use. These materials are presented solely for illustrative and academic purposes and do not imply any official affiliation or endorsement by HoYoverse.

Citations

Digital Life Project 2 (SIGGRAPH Asia 2025)

@inproceedings{dlp3d,
    author    = {Cai, Zhongang and Ren, Daxuan and Gao, Yang and Wei, Yukun and Zhou, Tongxi and Lin, Zhengyu and Jang, Huimuk and Zeng, Haoyang and Loy, Chen Change and Liu, Ziwei and Yang, Lei},
    title     = {Digital Life Project 2: Open-source Autonomous 3D Characters on the Web},
    booktitle = {SIGGRAPH Asia 2025 Real-Time Live!},
    year      = {2025},
    pages     = {3},
    isbn      = {9798400721359},
    publisher = {Association for Computing Machinery},
    address   = {New York, NY, USA},
    url       = {https://doi.org/10.1145/3757375.3774342},
    doi       = {10.1145/3757375.3774342},
    abstract  = {Digital Life Project 2 (DLP2) presents an open-source real-time framework that brings Large Language Models (LLMs) to life through expressive 3D avatars. Users converse naturally by voice, while characters respond on demand with unified audio, whole-body animation, and physics simulation directly in the browser. At its core are: (1) an agentic orchestration of large and small LLMs that governs character behavior, supported by a memory system tracking emotional states and evolving relationships to enable context-dependent reactions; (2) a hybrid real-time pipeline that segments long LLM responses, performs parallel motion retrieval and audio-motion synchronization, and streams efficiently through a custom Protocol Buffers structure for low-latency playback of voice, motion, and expression; and (3) robust mechanisms for user interruption handling, adaptive buffering, and fault tolerance. Characters are fully customizable in both appearance (3D models) and personality (character prompts) and readily adaptable to any LLM or text-to-speech (TTS) service. DLP2 demonstrates how LLMs can be embodied in responsive 3D characters, offering a practical blueprint for real-time, emotionally adaptive digital interactions on the web.},
    articleno = {3},
    numpages  = {2},
    location  = {Hong Kong Convention and Exhibition Centre, Hong Kong, Hong Kong},
    series    = {SA '25}
}

Digital Life Project (CVPR 2024) [Homepage]

@InProceedings{dlp,
    author    = {Cai, Zhongang and Jiang, Jianping and Qing, Zhongfei and Guo, Xinying and Zhang, Mingyuan and Lin, Zhengyu and Mei, Haiyi and Wei, Chen and Wang, Ruisi and Yin, Wanqi and Pan, Liang and Fan, Xiangyu and Du, Han and Gao, Peng and Yang, Zhitao and Gao, Yang and Li, Jiaqi and Ren, Tianxiang and Wei, Yukun and Wang, Xiaogang and Loy, Chen Change and Yang, Lei and Liu, Ziwei},
    title     = {Digital Life Project: Autonomous 3D Characters with Social Intelligence},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {582-592}
}