General Digital Human System

Project Overview

The General Digital Human System is an intelligent interaction platform built on cutting-edge AI technologies, integrating speech recognition, speech synthesis, natural language processing, and digital avatar rendering. The system supports real-time voice dialogue, knowledge Q&A, and emotional synchronization, making it suitable for various scenarios such as intelligent customer service, education training, and digital exhibitions.

precautions

Due to the use of Microsoft's real-time speech synthesis and virtual image services in this project, the required cost is relatively high. Please be cautious when using them.

Project Highlights

• High-Performance Rendering
• 30+ FPS stable frame rate
• Audio-video latency under 200ms

• 1080P HD output support

• Intelligent Dialogue

• Real-time knowledge base retrieval
• Multi-turn conversation memory

• Emotion analysis and sync

• System Stability

• Distributed architecture
• Automatic fault recovery
• Complete logging and monitoring

Features

• 🎭 Digital Avatar Rendering
• High-quality rendering based on Microsoft Avatar
• Multiple avatar style switching

• Facial expression and voice emotion sync

• 🗣️ Intelligent Voice Interaction

• Real-time speech recognition and synthesis
• Multiple voice options (Standard/Dialect)

• Emotional voice synthesis

• 📚 Knowledge Base Q&A

• Support multiple document formats (PDF, Word, TXT)
• Vector storage and semantic retrieval

• Real-time knowledge base updates

• 🤖 Intelligent Dialogue

• GPT-based natural language understanding
• Context memory and multi-turn dialogue
• Emotion recognition and response

System Architecture

+------------------+     +------------------+     +------------------+
|                  |     |                  |     |                  |
|  Web Frontend    |     |  FastAPI Backend |     |  Azure Services  |
|  (HTML/JS/CSS)   |<--->|  (Python)        |<--->|  (Speech/Avatar) |
|                  |     |                  |     |                  |
+------------------+     +------------------+     +------------------+
         ^                       ^                        ^
         |                       |                        |
         v                       v                        v
+------------------+     +------------------+     +------------------+
|                  |     |                  |     |                  |
|  WebRTC          |     |  Vector Database |     |  OpenAI/Cohere  |
|  (A/V Transfer)  |     |  (ChromaDB)      |     |  (AI Models)    |
|                  |     |                  |     |                  |
+------------------+     +------------------+     +------------------+

Technology Stack

Frontend

• HTML5 + CSS3 + JavaScript
• WebRTC real-time A/V transmission
• Azure Cognitive Services SDK
• Responsive design

Backend

• Python FastAPI framework
• LangChain LLM framework
• ChromaDB vector database
• Redis cache

AI Models & Services

• Azure Cognitive Services (Speech & Avatar)
• OpenAI GPT (Dialogue)
• Cohere (Text Vectorization)

Installation

Requirements

• Python 3.10+
• Docker (for Redis and Turnserver)

Setup Steps

1. Clone the repository

git clone [repository_url]
cd general_digital_human_system

1. Install dependencies

pip install -r requirements.txt

1. Start required services

# Ensure Docker service is running

# Check and remove existing containers (if needed)
docker rm -f redis-server turnserver_c

# Create Docker network
docker network create digital-human-network

# Start Redis server (with password)
docker run -d --name redis-server \
  -p 6379:6379 \
  redis:latest \
  --requirepass your_redis_password

# Start Turnserver (for WebRTC)
docker run -d \
  -p 3478:3478 \
  -p 3478:3478/udp \
  --name turnserver_c \
  coturn/coturn

1. Configure environment variables

cp .env.example .env

# Edit .env file with required configurations:

# Network proxy (if needed)
HTTP_PROXY=http://127.0.0.1:7890
HTTPS_PROXY=http://127.0.0.1:7890

# Azure service configuration
SUBSCRIPTION_KEY=your_azure_subscription_key
COGNITIVE_SERVICE_REGION=your_region

# OpenAI configuration
OPENAI_API_KEY=your_openai_api_key

# Cohere configuration
COHERE_API_KEY=your_cohere_api_key

# Redis configuration
REDIS_URL=redis://:your_redis_password@localhost:6379

# Search functionality
SERPAPI_API_KEY=your_serpapi_key

1. Start the service

python main.py

1. Access the system

Open browser and visit http://localhost:8000

Usage Guide

1. Knowledge Base Management
2. Click "Upload Document" button to upload knowledge files
3. System automatically processes document vectorization

4. Select knowledge base from dropdown menu

5. Voice Interaction

6. Click microphone icon to start voice input
7. Switch between text and voice input

8. Select different voice options

9. Avatar Switching

10. Select different avatars at the bottom
11. Support real-time avatar switching

Future Plans

1. Multi-modal Interaction
2. Add gesture recognition
3. Support image recognition

4. 3D scene interaction

5. Personalization

6. Custom avatar creation
7. Voice customization

8. Knowledge base deep training

9. Scenario Expansion

10. Metaverse social interaction
11. Virtual broadcasting

12. Intelligent education

13. Technical Upgrades

14. Support more LLMs
15. Optimize rendering performance
16. Enhance multi-turn dialogue

Docker Container Management

Common Commands

# View all container status
docker ps -a

# View container logs
docker logs redis-server
docker logs turnserver_c

# Restart containers
docker restart redis-server
docker restart turnserver_c

# Stop and remove all containers
docker stop $(docker ps -aq)
docker rm $(docker ps -aq)

# Clean unused images and containers
docker system prune -a

Troubleshooting

1. Redis Connection Issues
2. Check Redis container status: docker ps | grep redis-server
3. Verify Redis password: docker exec -it redis-server redis-cli -a your_redis_password ping

4. View Redis logs: docker logs redis-server

5. Turnserver Connection Issues

6. Check port availability: netstat -an | findstr "3478"
7. View Turnserver logs: docker logs turnserver_c

8. Ensure firewall allows UDP/3478 port

9. Container Network Issues

10. Check network list: docker network ls
11. Check network details: docker network inspect digital-human-network
12. Rebuild network: bash docker network rm digital-human-network docker network create digital-human-network

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Contributing

Welcome to submit Issues and Pull Requests. Before submitting a PR, please ensure:

1. Code follows project style guidelines
2. Add necessary test cases
3. Update relevant documentation
4. All Docker-related changes are tested

Acknowledgments

Thanks to the following open-source projects:

• FastAPI
• LangChain
• ChromaDB
• Azure Cognitive Services SDK