github.com/Tencent/AngelSlim @v0.5.0

English | 简体中文

<img alt="AngelSlim" src="https://github.com/Tencent/AngelSlim/raw/v0.5.0/docs/source/assets/logos/angelslim_logo.png" width=55%>

A more accessible, comprehensive, and efficient toolkit for large model compression.

      ✒️ <a href="https://arxiv.org/abs/2602.21233">TechnicalReport</a>&nbsp&nbsp | &nbsp&nbsp 📖 <a href="https://angelslim.readthedocs.io/">Documentation</a>&nbsp&nbsp | &nbsp&nbsp🤗 <a href="https://huggingface.co/AngelSlim">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp🤖 <a href="https://modelscope.cn/organization/AngelSlim">ModelScope</a>










      💬 <a href="https://github.com/Tencent/AngelSlim/raw/v0.5.0/docs/source/assets/angel_slim_wechat.png">WeChat</a> | &nbsp&nbsp🫨 <a href="https://discord.com/invite/dHVNeuNdFt">Discord</a>

📣Latest News

• [26/06/04] We have released Stem, a sparse attention algorithm that accelerates the Prefill stage of long-context LLMs by dynamically selecting top-k key blocks for block-sparse attention, significantly reducing latency while preserving generation quality. [Docs]
• [26/06/01] We have released DFlare, a block-diffusion speculative decoding framework with layer-wise fusion that achieves up to 5.52× end-to-end speedup. [Docs]
• [26/05/27] We have released D-Cut, an adaptive verification depth pruning technique for speculative decoding. [Docs]
• [26/05/20] We support Distillation for full-precision HuggingFace models and quantized QAT-style models, as detailed in the distillation documentation.
• [26/05/08] We have released STQ1_0 kernel for 1.25-bit model and given a PR to llama.cpp PR #22836 ! If you have any questions or suggestions for STQ_0, welcome to comment under the PR !🔥🔥🔥
• [26/04/29] We have released 2-bit and 1.25-bit versions of Tencent Hy-MT1.5-1.8B Translation Model: Hy-MT1.5-1.8B-2bit and Hy-MT1.5-1.8B-1.25bit. Additionally, we have make an offline translation demo for you to try out. We invite you to give it a spin! 🔥🔥🔥
• [26/04/23] We now support FP8-Static quantization for Hy3-preview (MoE A20B).
• [26/03/25] We have released DAQ, the quantization algorithm that preserves the knowledge acquired while the update of parameters is relatively small during post-training training.[Paper] | [Docs]
• [26/02/09] We have released HY-1.8B-2Bit, 2bit on-device large language model,[Huggingface].
• [26/01/13] We have released v0.3. We support the training and deployment of Eagle3 for all-scale LLMs/VLMs/Audio models, as detailed in the guidance documentation. And We released Sherry, the hardware-efficient 1.25 bit quantization algorithm [Paper] | [Code]🔥🔥🔥

Previous News

• [25/11/05] We have released v0.2. Quantization support for new models, such as GLM-4.6, Qwen3-VL and Qwen3-Omni, open-sources the Eagle3 speculative decoding training framework, and updates the Diffusion model quantization tools.
• [25/09/30] We have released SpecExit, the reasoning early-exit algorithm: [Paper] | [Docs] | [vLLM Code]
• [25/09/26] We have released TEQUILA, the ternary quantization algorithm [Paper] | [Code]
• [25/09/24] We now support the PTQ quantization of NVFP4 for the Qwen3 series models. We also opensource Qwen3-32B-NVFP4 and Qwen3-235B-A22B-NVFP4 weights.
• [25/09/01] We now support ​FP8 quantization​ of the Hunyuan-MT-7B translation model. And enabled ​Torch inference and Benchmark evaluation​ for Eagle3. And implemented support for ​quantization and Cache​ for FLUX. And support ​quantization​ for the Seed-OSS.
• [25/08/06] We now support quantization for Hunyuan 0.5B/1.8B/4B/7B and multimodal model Qwen2.5VL 3B/7B/32B/72B, including FP8/INT4 algorithms, and quantization for DeepSeek-R1/V3 and Kimi-K2, including FP8-Static and W4A8-FP8 algorithms. We also opensource Hunyuan 1.8B/4B/7B series Eagle3 model weight.
• [25/07/04] We now support quantization for Hunyuan/Qwen2.5/Qwen3/DeepSeek-R1-Distill-Qwen and other models, including INT8/FP8/INT4 algorithms. We also opensource Qwen3 series Eagle3 model weight.

🌟Key Features

• Highly Integrated: This toolkit integrates mainstream compression algorithms into a unified framework, offering developers one-click access with exceptional ease of use.
• Continuous Innovation: Beyond integrating widely-used industry algorithms, we are continuously researching better compression algorithms, which will be gradually open-sourced in the future.
• Performance-Driven: We continuously optimize end-to-end performance in model compression workflows and algorithm deployment, such as enabling quantization of models like Qwen3-235B and DeepSeek-R1 on a single GPU.

💼Technical Overview