audio.cpp

audio.cpp is a high-performance C++ audio inference framework built on top of ggml, designed to make modern local audio models practical, portable, and fast.

Tired of juggling a dozen Conda environments, hundreds of Python packages, and dependency conflicts just to try a few audio models? audio.cpp gives those paths a shared native runtime instead.

[!IMPORTANT] CUDA performance headline: multiple TTS paths already run 1.8x-5.0x faster than their Python reference paths while cutting end-to-end latency by 45%-80%. VibeVoice 1.5B: generates a 93.9-minute podcast in 18.2 minutes with 10 diffusion steps and without quantization, running about 5.15x faster than real time.

It is built for real end-to-end execution rather than one-off model demos: the same runtime powers TTS, voice cloning, voice conversion, ASR, diarization, VAD, source separation, alignment, codec-style models, and higher-level workflows through a common framework surface.

Highlights:

• Parity. Strong parity tooling against Python reference paths.
• Performance. Performance-focused execution, reusable sessions, and batch-style offline inference. Optimized for CUDA.
• Portability. A portable native stack centered on ggml, with CLI and server entry points instead of Python-only deployment paths.
• Pipelines. Experimental JSON pipeline support for higher-level multi-step workflows.
• Audio Utilities. Built-in denoise, enhancement, resampling, and STFT/ISTFT utilities for real production-style task paths.

The goal of the framework is to provide highly optimized, reusable building blocks for audio-related models, so new model integrations can be brought up faster, shared components can be improved once and benefit many families, and real end-to-end inference paths can stay efficient, maintainable, and portable.

[!TIP] Contribution focus: the most helpful contributions right now are improvements to the UI, API server, and pipeline/workflow subsystems. These areas make the existing model surface easier to use, serve, compose, and validate. See CONTRIBUTING.md for more details.

New model PRs: before starting a new model port, please check the supported model table because several families are already implemented or under testing. If you do add a model, follow the validation style in PR #19: include exact build/run commands, model paths or package ids, generated outputs, parity or path-test results, and relevant performance or memory notes.

News

[!IMPORTANT] 2026-07-08: Four new ASR families are now released in the framework: Higgs Audio STT, Hviske ASR, Nemotron ASR, and VibeVoice ASR. Initial model-specific streaming support also lands for VoxCPM2 TTS, Nemotron ASR, and Higgs Audio STT, with server SSE configuration and request examples for streaming speech generation and transcription.

[!IMPORTANT] 2026-07-03: Conv1DTransp module CUDA optimization: VibeVoice reaches 5.15x realtime on 93.9-minute long-form generation. Overall, VibeVoice inference time was reduced by 73.17%, PocketTTS by 35.32%, Chatterbox by 33.56%, Qwen3-TTS by 30.60%, HeartMuLa by 17.03%, and VoxCPM2 by 14.7% compared with the previous release.

[!IMPORTANT] 2026-07-02: Music generation and source separation expanded in the released framework surface: ACE-Step 1.5 Turbo/Base, HeartMuLa, Stable Audio 3 Small Music/SFX and Medium, Mel-Band RoFormer, and HTDemucs are now available through the normal audio.cpp CLI/framework paths.

• 2026-07-02: VibeVoice 7B joins the 1.5B model, and full fine-tune adapters — language-model LoRA plus fine-tuned diffusion head and acoustic/semantic connectors — can now be merged at load time through --load-option vibevoice.lora.
• 2026-06-30: VibeVoice 1.5B is now released in the framework, bringing long-form, multi-speaker dialogue TTS into the normal audio.cpp model surface.
• 2026-06-30: More detailed usage documentation is now available in docs/usage.md, covering model setup, CLI usage, server usage, and common workflows.
• 2026-06-26: The speech intelligence side grew with released Citrinet ASR, MarbleNet VAD, and Sortformer diarization paths.
• 2026-06-25: The first release wave landed with TTS, voice cloning, voice conversion, alignment, VAD, codec, and multilingual generation support across Chatterbox, MioCodec, MioTTS, OmniVoice, PocketTTS, Qwen3, SeedVC, Silero VAD, Vevo2, and VoxCPM2.

Supported Models

Current model status in the framework:

• released: The model is fully wired into the broader framework surface and ready for normal use.
• testing: The model is implemented and working in this repo, and is still being validated, polished, or promoted into the broader released surface.
• optimization: The model is end-to-end working, but still needs more optimization work before it should be treated like a released or testing-level path.

Family	Task	Supported language(s)	Supported variant(s) in this repo	Release status
ace_step	music generation, music editing	50+ langs	ACE-Step 1.5 Turbo and Base with acestep-5Hz-lm-1.7B	released
chatterbox	TTS, voice cloning	ar, da, de, el, en, es, fi, fr, hi, it, ko, ms, nl, no, pl, pt, sv, sw, tr	Chatterbox with 0.5B backbone	released
citrinet_asr	ASR	en	Citrinet-256	released
heartmula	music generation	zh, en, ja, ko, es	HeartMuLa-oss-3B with HeartCodec-oss	released
higgs_audio_stt	ASR	en	Higgs Audio v3 STT	released
htdemucs	source separation	lang agnostic	HTDemucs, HTDemucs_ft	released
hviske_asr	ASR	da	Hviske v5.3	released
marblenet_vad	VAD	lang agnostic	MarbleNet VAD	released
mel_band_roformer	vocal separation	lang agnostic	Mel-Band RoFormer MLX vocal separation variants	released
miocodec	audio codec, voice conversion backend	lang agnostic	MioCodec v2, 25 Hz, 44.1 kHz	released
miotts	TTS, voice cloning	en, ja	MioTTS-1.7B	released
omnivoice	TTS, voice cloning, voice design	646+ langs	OmniVoice, Qwen3-0.6B based	released
pocket_tts	TTS, voice cloning	en, de, it, pt, es	PocketTTS-100M	released
nemotron_asr	ASR	100+ ASR prompt codes incl. auto	Nemotron 3.5 ASR Streaming 0.6B	released
qwen3_asr	ASR	zh, en, yue, ar, de, fr, es, pt, id, it, ko, ru, th, vi, ja, tr, hi, ms, nl, sv, da, fi, pl, cs, fil, fa, el, ro, hu, mk	Qwen3-ASR-0.6B	released
qwen3_forced_aligner	forced alignment	zh, yue, en, de, es, fr, it, pt, ru, ko, ja	Qwen3-ForcedAligner-0.6B	released
qwen3_tts	TTS, voice cloning, voice design	zh, en, fr, de, it, ja, ko, pt, ru, es	Qwen3-TTS-12Hz-0.6B-Base, Qwen3-TTS-12Hz-1.7B-Base, Qwen3-TTS-12Hz-1.7B-CustomVoice, Qwen3-TTS-12Hz-1.7B-VoiceDesign	released
seed_vc	voice conversion	lang agnostic	SeedVC XLS-R + HiFT, SeedVC Whisper-small + BigVGAN	released
silero_vad	VAD	lang agnostic	Silero VAD	released
sortformer_diar	diarization	en	Sortformer-4spk-v1	released
stable_audio	music generation, sound generation, audio editing	en	Stable Audio 3 Small Music, Stable Audio 3 Small SFX, Stable Audio 3 Medium	released
vevo2	TTS, singing generation, voice conversion, singing conversion, editing	en, zh	Vevo2 with Qwen2.5-0.5B AR model	released
vibevoice	TTS, multi-speaker dialogue TTS	en, zh	VibeVoice-1.5B, VibeVoice-7B	released
vibevoice_asr	ASR	auto	VibeVoice ASR	released
voxcpm2	TTS, voice cloning, voice design	ar, da, de, el, en, es, fi, fr, he, hi, id, it, ja, km, ko, lo, ms, my, nl, no, pl, pt, ru, sv, sw, th, tl, tr, vi, zh	VoxCPM2-2B, 48 kHz	released
higgs_tts	TTS, voice cloning, expressive speech	100+ languages	Higgs Audio v3 TTS 4B	testing
index_tts2	TTS, voice cloning, expressive speech	zh, en	IndexTTS-2	testing
irodori_tts	TTS, voice cloning, voice design	ja	Irodori-TTS-500M-v3, Irodori-TTS-600M-v3-VoiceDesign	testing
kokoro_tts	TTS	en-us, en-gb	Kokoro-82M	testing
moss_tts	TTS, voice cloning	zh, yue, en, ar, cs, da, nl, fi, fr, de, el, he, hi, hu, it, ja, ko, mk, ms, fa, pl, pt, ro, ru, es, sw, sv, tl, th, tr, vi	MOSS-TTS-Local	testing
supertonic	TTS	en	Supertonic 3	testing

PocketTTS language selection is a model-load option. When the model path points at the PocketTTS root, the loader uses english unless you pass --load-option language=<name>. Kyutai's normal non-English PocketTTS releases are smaller distilled language models intended for the fast PocketTTS path. The _24l variants are larger 24-layer, undistilled preview models that can sound better but are slower. Kyutai currently publishes French only as french_24l, not as a normal distilled french language directory, so French is not listed as a normal PocketTTS language here.

Docker

Docker CPU and CUDA images are available for both CLI and server use. See Docker.md for build commands and working Docker examples.

Build

Linux Build

On Linux, use a normal CMake build directory such as build/.

For single-config generators, the default build type is RelWithDebInfo.

That default configure is a CPU build unless you enable an accelerator backend explicitly.

Use GCC 13 or newer for Linux builds.

Native ggml CPU optimization is enabled by default for local performance. If your compiler or assembler rejects a generated CPU instruction such as vpdpbusd, reconfigure with -DENGINE_ENABLE_NATIVE_CPU=OFF to build portable CPU kernels.

Common Linux configure examples:

CPU-only:

cmake -S . -B build

CUDA:

cmake -S . -B build -DENGINE_ENABLE_CUDA=ON

Vulkan:

cmake -S . -B build -DENGINE_ENABLE_VULKAN=ON

Portable CPU-kernel fallback:

cmake -S . -B build -DENGINE_ENABLE_NATIVE_CPU=OFF

Build the CLI and server from the configured tree:

cmake --build build -j$(nproc) --target audiocpp_cli --target audiocpp_server

If your machine is memory-constrained, use a smaller -j value, for example -j4.

If you use an environment manager or custom toolchain, activate it before running the commands above.

The optional Linux helper script wraps the same CMake flow and uses aligned build directory names:

• build/linux-cuda-release
• build/linux-vulkan-release
• build/linux-cpu-release

Examples:

scripts/build_linux.sh --backend cuda --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend vulkan --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cpu --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cuda --native-cpu OFF --target audiocpp_cli --target audiocpp_server

Use --build-dir <dir> only when you intentionally want a custom output directory.

Windows Build

The recommended native Windows build is command-line only:

• Visual Studio Build Tools 2022 or newer with the C++ desktop workload
• MSVC x64 compiler, Windows SDK, CMake, Ninja, and MSVC OpenMP components
• Official NVIDIA CUDA Toolkit installed under C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\... for CUDA builds

Use MSVC cl.exe as the compiler. For CUDA builds, cl.exe is also used as the CUDA host compiler. Native Windows nvcc does not support clang-cl as its host compiler, and the Visual Studio IDE is not required.

From PowerShell:

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1

CPU-only:

.\scripts\build_windows.ps1 -Preset windows-cpu-release -Target audiocpp_cli

From cmd.exe:

scripts\build_windows.cmd

If GNU Make is available on Windows:

make -f Makefile.windows cpu JOBS=16
make -f Makefile.windows cuda JOBS=16
make -f Makefile.windows cuda NATIVE_CPU=OFF JOBS=16

The Windows script configures build/windows-cuda-release by default and builds audiocpp_cli. CUDA presets enable CUDA, CUDA graphs, OpenMP, Ninja, /utf-8, /EHsc, MSVC OpenMP SIMD support with /openmp:experimental, and native CPU optimization by default. The CPU preset uses the same MSVC/Ninja/OpenMP setup without requiring CUDA. CUDA presets auto-detect the local GPU CUDA architecture when nvidia-smi is available. Pass -NativeCpu OFF or NATIVE_CPU=OFF to use portable CPU kernels.

For Windows prebuilt release zips and CPU compatibility profiles, see docs/windows_build.md.

Useful variants:

.\scripts\build_windows.ps1 -Target audiocpp_server -Jobs 16
.\scripts\build_windows.ps1 -Preset windows-cpu-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Preset windows-cuda-debug -Target audiocpp_cli
.\scripts\build_windows.ps1 -NativeCpu OFF -Target audiocpp_cli
.\scripts\build_windows.ps1 -ConfigureOnly
.\scripts\build_windows.ps1 -CudaArchitectures 120a-real

If multiple Build Tools installations are present, pass the one you want explicitly:

.\scripts\build_windows.ps1 -VsInstall "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools"

The built CLI is written to the selected preset directory:

build\windows-cpu-release\bin\audiocpp_cli.exe
build\windows-cuda-release\bin\audiocpp_cli.exe

Metal Build

On macOS, use the Metal helper script to build against ggml's Metal backend. It requires Xcode or the Xcode Command Line Tools with the Metal compiler available through xcrun.

```bash scripts/buil