github.com/LqNoob/Neural-Codec-and-Speech-Language-Models @main

If you like our project, please give us a star ⭐ on GitHub for the latest update.

[](https://github.com/sindresorhus/awesome) [](https://github.com/LqNoob/Neural-Codec-and-Speech-Language-Models/blob/main/LICENSE) [](https://gitHub.com/LqNoob/Neural-Codec-and-Speech-Language-Models/) [](https://github.com/LqNoob/Neural-Codec-and-Speech-Language-Models) [](https://github.com/LqNoob/Neural-Codec-and-Speech-Language-Models/issues?q=is%3Aopen+is%3Aissue) [](https://github.com/LqNoob/Neural-Codec-and-Speech-Language-Models/issues?q=is%3Aissue+is%3Aclosed) [](https://github.com/LqNoob/Neural-Codec-and-Speech-Language-Models/graphs/contributors/)

• Awesome Codec, TTS & Speech LM
• Neural Codec Models
• Zero-Shot Text-to-Speech Synthesizers
• Speech Language Models
  • End-to-End (Paradigms) Models
  • Cascaded (Paradigms) Models
• Benchmark
• Survey
• Music Generation
• Some Interesting Models
• Speech DataSet
• Some Interesting knowledge
• Blog & Courses
• Minor Points of Concern
• Reference

Awesome Codec, TTS & Speech LM

1. Acoustic Tokens: Acoustic tokens focuses on speech compression and reconstruction, which rely on encoder-decoder architectures with residual vector quantization (RVQ). Specifically, these models quantify speech features (which are downsampled from raw wavforms by one encoder) into a series of discrete tokens, then use one decoder to upsample these discrete tokens into the speech, calculating the reconstruction loss against the original signal. By this approach, we can get discrete acoustic tokens with impressive compression rates and high-fidelity acoustic information, making it more suitable for tasks such as speech synthesis and emotion analysis. (requires maintaining reconstruction ability and a low bitrate)
2. Semantic Tokens: Semantic tokens involve applying clustering algorithms such as K-means to extract features from self-supervised learning models, using the cluster indices as discrete representations. And it is prediction-based modeling, these models are trained for representation learning by predicting future frames in an autoregressive manner or by using surrounding frames to predict masked frames. This approach tends to prioritize capturing linguistic information within speech, making it particularly useful for recognition and understanding tasks.
3. Speech Large Language Models: These models are trained on top of speech and acoustic tokens in a language modeling approach. They demonstrate proficiency in tasks on speech understanding and speech generation. (From speech-trident)

Neural Codec Models

• [2025/12] PURE Codec: Progressive Unfolding of Residual Entropy for Speech Codec Learning [paper][code]
• [2025/11] BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction [paper][code][demo]
• [2025/10] UniTok-Audio: A Unified Audio Generation Framework Via Universal Discrete Token [paper][code][demo] :heavy_check_mark:
• [2025/10] Low Resource Audio Codec Challenge Track1: Transparency Codec [paper][demo]
• [2025/10] PhoenixCodec: Taming Neural Speech Coding for Extreme Low-Resource Scenarios [paper][demo]
• [2025/10] SpecTokenizer: A Lightweight Streaming Codec in the Compressed Spectrum Domain [paper]
• [2025/10] Speaking Clearly: A Simplified Whisper-Based Codec for Low-Bitrate Speech Coding [paper][code][demo] :heavy_check_mark:
• [2025/10] SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization [paper][code][demo] :heavy_check_mark:
• [2025/10] MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding [paper][code][demo] :heavy_check_mark:
• [2025/10] U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation [paper][code][demo] :heavy_check_mark:
• [2025/10] LDCodec: A high quality neural audio codec with low-complexity decoder [paper]
• [2025/10] FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates [paper][code][demo] :heavy_check_mark:
• [2025/10] LongCat-Audio-Codec: An Audio Tokenizer and Detokenizer Solution Designed for Speech Large Language Models [paper][code] :heavy_check_mark:
• [2025/10] BridgeCode: A Dual Speech Representation Paradigm for Autoregressive Zero-Shot Text-to-Speech Synthesis [paper][demo]
• [2025/10] Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates [paper][code]
• [2025/09] Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis [paper][code]
• [2025/09] MBCodec:Thorough disentangle for high-fidelity audio compression [paper]
• [2025/09] FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation [paper][code] :heavy_check_mark:
• [2025/09] MSR-Codec: A Low-Bitrate Multi-Stream Residual Codec for High-Fidelity Speech Generation with Information Disentanglement [paper]
• [2025/09] FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs [paper][code] :heavy_check_mark:
• [2025/09] CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio [paper][code] :heavy_check_mark:
• [2025/09] DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners [paper][demo]
• [2025/09] Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding [paper][demo]
• [2025/08] Exploring Disentangled Neural Speech Codecs from Self-Supervised Representations [paper]
• [2025/08] DualSpeechLM: Towards Unified Speech Understanding and Generation via Dual Speech Token Modeling with Large Language Models [paper][code][demo] :heavy_check_mark:
• [2025/08] NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference [paper][code][demo] :heavy_check_mark:
• [2025/08] SpectroStream: A Versatile Neural Codec for General Audio [paper]
• [2025/08] SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec [paper][demo]
• [2025/07] HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling [paper][code]
• [2025/06] XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in LowBitrate Speech Codecs [paper][code]
• [2025/06] DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding [paper]
• [2025/06] CodecSlime: Temporal Redundancy Compression of Neural Speech Codec via Dynamic Frame Rate [paper][demo]
• [2025/06] USAD: Universal Speech and Audio Representation via Distillation [paper][HF]
• [2025/06] Towards Bitrate-Efficient and Noise-Robust Speech Coding with Variable Bitrate RVQ [paper][code][demo] :heavy_check_mark:
• [2025/06] LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization [paper]
• [2025/06] TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling [paper][code][demo] :heavy_check_mark:
• [2025/05] MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation [paper][code] :heavy_check_mark:
• [2025/05] SwitchCodec: A High-Fidelity Nerual Audio Codec With Sparse Quantization [paper]
• [2025/05] DS-Codec: Dual-Stage Training with Mirror-to-NonMirror Architecture Switching for Speech Codec [paper][demo]
• [2025/05] Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate [paper]
• [2025/05] PAST: Phonetic-Acoustic Speech Tokenizer [paper][code][demo] Code Comming Soon
• [2025/05] Universal Semantic Disentangled Privacy-preserving Speech Representation Learning [paper][demo]
• [2025/05] Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding [paper][code][demo] :heavy_check_mark:
• [2025/05] Toward a Sparse and Interpretable Audio Codec [paper][[co