Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

Chetwin Low ^{* 1} , Weimin Wang ^{* † 1} , Calder Katyal ²

^* Equal contribution, ^† Project Lead

¹ Character AI, ² Yale University

---

🎥 Video Demo

🆕 Ovi 1.1 10-Second Demo

Ovi 1.1 – 10-second temporally consistent video generation (960 × 960 resolution)

🎬 Original 5-Second Demo

---

🆕 Ovi 1.1 Update (10 November 2025)

• Key Feature: Enables temporal-consistent 10-second video generation at 960 × 960 resolution
• Training Improvements:
• Trained natively on 960×960 resolution videos
• Dataset includes 100% more videos for greater diversity
• Prompt Format Update:
• Audio descriptions should now be written as
  Audio: ...
  instead of using
  <AUDCAP> ... <ENDAUDCAP>

🌟 Key Features

Ovi is a veo-3-like, video + audio generation model that simultaneously generates both video and audio content from text or text + image inputs. - 🎬 Video+Audio Generation: Generate synchronized video and audio content simultaneously - 🎵 High-Quality Audio Branch: We designed and pretrained our 5B audio branch from scratch using our high quality in-house audio datasets - 📝 Flexible Input: Supports text-only or text+image conditioning - ⏱️ 10-second (or 5-second) Videos: Generates 10-second or 5-second videos at 24 FPS, resolution of 960x960p, at various aspect ratios (9:16, 16:9, 1:1, etc) - 🔧 ComfyUI Integration: ComfyUI support is now available via ComfyUI-WanVideoWrapper, related PR. - 🎬 Create videos now on wavespeed.ai: https://wavespeed.ai/models/character-ai/ovi/image-to-video & https://wavespeed.ai/models/character-ai/ovi/text-to-video - 🎬 Create videos now on HuggingFace: https://huggingface.co/spaces/akhaliq/Ovi

🎯 10-second examples

Click the ⛶ button on any video to view full screen.

🎯 5-second examples

Click the ⛶ button on any video to view full screen.

---

📋 Todo List

• [x] Release research paper and website for demos
• [x] Checkpoint of 11B model
• [x] Inference Codes
• [x] Text or Text+Image as input
• [x] Gradio application code
• [x] Multi-GPU inference with or without the support of sequence parallel
• [x] fp8 weights and improved memory efficiency (credits to @rkfg)
• [x] qint8 quantization thanks to @gluttony-10
• [ ] Improve efficiency of Sequence Parallel implementation
• [ ] Implement Sharded inference with FSDP
• [x] Video creation example prompts and format
• [x] Finetune model with higher resolution data, and RL for performance improvement.
• [x] Longer video generation (10s)
• [ ] Reference voice condition
• [ ] Distilled model for faster inference
• [ ] Training scripts

---

🎨 An Easy Way to Create

We provide example prompts to help you get started with Ovi: - Text-to-Audio-Video (T2AV) 10s: example_prompts/gpt_examples_t2v.csv - Image-to-Audio-Video (I2AV) 10s: example_prompts/gpt_examples_i2v.csv - Text-to-Audio-Video (T2AV): example_prompts/gpt_examples_t2v.csv - Image-to-Audio-Video (I2AV): example_prompts/gpt_examples_i2v.csv

📝 Prompt Format

Our prompts use special tags to control speech and audio: - Speech: <S>Your speech content here<E> - Text enclosed in these tags will be converted to speech - Audio Description: Audio: YOUR AUDIO DESCRIPTION - Describes the audio or sound effects present in the video at the end of prompt!

---

📦 Installation

Step-by-Step Installation

# Clone the repository
git clone https://github.com/character-ai/Ovi.git

cd Ovi

# Create and activate virtual environment
virtualenv ovi-env
source ovi-env/bin/activate

# Install PyTorch first
pip install torch==2.6.0 torchvision torchaudio

# Install other dependencies
pip install -r requirements.txt

# Install Flash Attention
pip install flash_attn --no-build-isolation

Alternative Flash Attention Installation (Optional)

If the above flash_attn installation fails, you can try the Flash Attention 3 method:

git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/hopper
python setup.py install
cd ../..  # Return to Ovi directory

Download Weights

To download our main Ovi checkpoint, as well as T5 and vae decoder from Wan, and audio vae from MMAudio

# Default is downloaded to ./ckpts, and the inference yaml is set to ./ckpts so no change required
# Default installs all versions of Ovi models, 720x720_5s, 960x960_5s, 960x960_10s
python3 download_weights.py
# For qint8 also ues python3 download_weights.py

OR

# Optional can specific --output-dir to download to a specific directory
# but if a custom directory is used, the inference yaml has to be updated with the custom directory
python3 download_weights.py --output-dir <custom_dir>

# Optional can specific --models to download selective versions of Ovi instead of all of them
# but if a custom directory is used, the inference yaml has to be updated with the custom directory
python3 download_weights.py --models 960x960_10s # ["720x720_5s", "960x960_5s", "960x960_10s"]

# Additionally, if you only have ~ 24Gb of GPU vram, please download the fp8 quantized version of the model, and follow the following instructions in sections below to run with fp8
wget -O "./ckpts/Ovi/model_fp8_e4m3fn.safetensors" "https://huggingface.co/rkfg/Ovi-fp8_quantized/resolve/main/model_fp8_e4m3fn.safetensors"

🚀 Run Examples

⚙️ Configure Ovi

Ovi's behavior and output can be customized by modifying ovi/configs/inference/inference_fusion.yaml configuration file. The following parameters control generation quality, video resolution, and how text, image, and audio inputs are balanced:

# Output and Model Configuration
model_name: "960x960_10s" # ["720x720_5s", "960x960_5s", "960x960_10s"]
output_dir: "/path/to/save/your/videos"                    # Directory to save generated videos
ckpt_dir: "/path/to/your/ckpts/dir"                        # Path to model checkpoints

# Generation Quality Settings
sample_steps: 50                             # Number of denoising steps. Lower (30-40) = faster generation
solver_name: "unipc"                     # Sampling algorithm for denoising process
shift: 5.0                               # Timestep shift factor for sampling scheduler
seed: 100                                # Random seed for reproducible results

# Guidance Strength Control
audio_guidance_scale: 3.0                # Strength of audio conditioning. Higher = better audio-text sync
video_guidance_scale: 4.0                # Strength of video conditioning. Higher = better video-text adherence
slg_layer: 11                            # Layer for applying SLG (Skip Layer Guidance) technique - feel free to try different layers!

# Multi-GPU and Performance
sp_size: 1                               # Sequence parallelism size. Set equal to number of GPUs used
cpu_offload: False                       # CPU offload, will largely reduce peak GPU VRAM but increase end to end runtime by ~20 seconds
fp8: False                               # load fp8 version of model, will have quality degradation and will not have speed up in inference time as it still uses bf16 matmuls, but can be paired with cpu_offload=True, to run model with 24Gb of GPU vram

# Input Configuration
text_prompt: "/path/to/csv" or "your prompt here"          # Text prompt OR path to CSV/TSV file with prompts
mode: ['i2v', 't2v', 't2i2v']                          # Generate t2v, i2v or t2i2v; if t2i2v, it will use flux krea to generate starting image and then will follow with i2v
video_frame_height_width: [704, 1280]    # Video dimensions [height, width] for T2V mode only
each_example_n_times: 1                  # Number of times to generate each prompt

# Quality Control (Negative Prompts)
video_negative_prompt: "jitter, bad hands, blur, distortion"  # Artifacts to avoid in video
audio_negative_prompt: "robotic, muffled, echo, distorted"    # Artifacts to avoid in audio

🎬 Running Inference

Single GPU (Simple Setup)

python3 inference.py --config-file ovi/configs/inference/inference_fusion.yaml

Use this for single GPU setups. The text_prompt can be a single string or path to a CSV file.

Multi-GPU (Parallel Processing)

torchrun --nnodes 1 --nproc_per_node 8 inference.py --config-file ovi/configs/inference/inference_fusion.yaml

Use this to run samples in parallel across multiple GPUs for faster processing.

Memory & Performance Requirements

Below are approximate GPU memory requirements for different configurations. Sequence parallel implementation will be optimized in the future. All End-to-End time calculated based on a 121 frame, 720x720 video, using 50 denoising steps. Minimum GPU vram requirement to run our model is 32Gb, fp8 parameters is currently supported, reducing peak VRAM usage to 24Gb with slight quality degradation.

Sequence Parallel Size	FlashAttention-3 Enabled	CPU Offload	With Image Gen Model	Peak VRAM Required	End-to-End Time
1	Yes	No	No	~80 GB	~83s
1	No	No	No	~80 GB	~96s
1	Yes	Yes	No	~80 GB	~105s
1	No	Yes	No	~32 GB	~118s
1	Yes	Yes	Yes	~32 GB	~140s
4	Yes	No	No	~80 G