DeepSeek v4 Flash experimental support

This is a fork of llama.cpp that implements DSv4 support, with generated GGUF that aims to target MacBooks with just 128GB of RAM using 2bit quantization of routed experts.

Disclaimer: * This code was written with heavy help from GPT 5.5 and the official DeepSeek v4 Flash as reference. * The model quantized in this way behaves very very well in the chat, frontier-model vibes, but it was not extensively tested. * The code runs both with CPU and Metal backends. With Metal is faster.

Download the GGUF from: https://huggingface.co/antirez/deepseek-v4-gguf

Then to test it run with (but for production you may want to tune context, disable thinking for faster replies and so forth):

llama-cli \
    -m DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat.gguf \
    -cnv

llama.cpp

Manifesto / ggml / ops

LLM inference in C/C++

Recent API changes

• Changelog for libllama API
• Changelog for llama-server REST API

Hot topics

• Hugging Face cache migration: models downloaded with -hf are now stored in the standard Hugging Face cache directory, enabling sharing with other HF tools.
• guide : using the new WebUI of llama.cpp
• guide : running gpt-oss with llama.cpp
• [FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗
• Support for the gpt-oss model with native MXFP4 format has been added | PR | Collaboration with NVIDIA | Comment
• Multimodal support arrived in llama-server: #12898 | documentation
• VS Code extension for FIM completions: https://github.com/ggml-org/llama.vscode
• Vim/Neovim plugin for FIM completions: https://github.com/ggml-org/llama.vim
• Hugging Face Inference Endpoints now support GGUF out of the box! https://github.com/ggml-org/llama.cpp/discussions/9669
• Hugging Face GGUF editor: discussion | tool

Quick start

Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine:

• Install llama.cpp using brew, nix or winget
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed, you'll need a model to work with. Head to the Obtaining and quantizing models section to learn more.

Example command:

# Use a local model file
llama-cli -m my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

# Launch OpenAI-compatible API server
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

Description

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is the main playground for developing new features for the ggml library.

Models

Typically finetunes of the base models below are supported as well.

Instructions for adding support for new models: HOWTO-add-model.md

Text-only

• [X] LLaMA 🦙
• [x] LLaMA 2 🦙🦙
• [x] LLaMA 3 🦙🦙🦙
• [X] Mistral 7B
• [x] Mixtral MoE
• [x] DBRX
• [x] Jamba
• [X] Falcon
• [X] Chinese LLaMA / Alpaca and Chinese LLaMA-2 / Alpaca-2
• [X] Vigogne (French)
• [X] BERT
• [X] Koala
• [X] Baichuan 1 & 2 + derivations
• [X] Aquila 1 & 2
• [X] Starcoder models
• [X] Refact
• [X] MPT
• [X] Bloom
• [x] Yi models
• [X] StableLM models
• [x] Deepseek models
• [x] Qwen models
• [x] PLaMo-13B
• [x] Phi models
• [x] PhiMoE
• [x] GPT-2
• [x] Orion 14B
• [x] InternLM2
• [x] CodeShell
• [x] Gemma
• [x] Mamba
• [x] Grok-1
• [x] Xverse
• [x] Command-R models
• [x] SEA-LION
• [x] GritLM-7B + GritLM-8x7B
• [x] OLMo
• [x] OLMo 2
• [x] OLMoE
• [x] Granite models
• [x] GPT-NeoX + Pythia
• [x] Snowflake-Arctic MoE
• [x] Smaug
• [x] Poro 34B
• [x] Bitnet b1.58 models
• [x] Flan T5
• [x] Open Elm models
• [x] ChatGLM3-6b + ChatGLM4-9b + GLMEdge-1.5b + GLMEdge-4b
• [x] GLM-4-0414
• [x] SmolLM
• [x] EXAONE-3.0-7.8B-Instruct
• [x] FalconMamba Models
• [x] Jais
• [x] Bielik-11B-v2.3
• [x] RWKV-7
• [x] RWKV-6
• [x] QRWKV-6
• [x] GigaChat-20B-A3B
• [X] Trillion-7B-preview
• [x] Ling models
• [x] LFM2 models
• [x] Hunyuan models
• [x] BailingMoeV2 (Ring/Ling 2.0) models

Multimodal

• [x] LLaVA 1.5 models, LLaVA 1.6 models
• [x] BakLLaVA
• [x] Obsidian
• [x] ShareGPT4V
• [x] MobileVLM 1.7B/3B models
• [x] Yi-VL
• [x] Mini CPM
• [x] Moondream
• [x] Bunny
• [x] GLM-EDGE
• [x] Qwen2-VL
• [x] LFM2-VL

Bindings

• Python: ddh0/easy-llama
• Python: abetlen/llama-cpp-python
• Go: go-skynet/go-llama.cpp
• Node.js: withcatai/node-llama-cpp
• JS/TS (llama.cpp server client): lgrammel/modelfusion
• JS/TS (Programmable Prompt Engine CLI): offline-ai/cli
• JavaScript/Wasm (works in browser): tangledgroup/llama-cpp-wasm
• Typescript/Wasm (nicer API, available on npm): ngxson/wllama
• Ruby: yoshoku/llama_cpp.rb
• Rust (more features): edgenai/llama_cpp-rs
• Rust (nicer API): mdrokz/rust-llama.cpp
• Rust (more direct bindings): utilityai/llama-cpp-rs
• Rust (automated build from crates.io): ShelbyJenkins/llm_client
• C#/.NET: SciSharp/LLamaSharp
• C#/VB.NET (more features - community license): LM-Kit.NET
• Scala 3: donderom/llm4s
• Clojure: phronmophobic/llama.clj
• React Native: mybigday/llama.rn
• Java: kherud/java-llama.cpp
• Java: QuasarByte/llama-cpp-jna
• Zig: deins/llama.cpp.zig
• Flutter/Dart: netdur/llama_cpp_dart
• Flutter: xuegao-tzx/Fllama
• PHP (API bindings and features built on top of llama.cpp): distantmagic/resonance (more info)
• Guile Scheme: guile_llama_cpp
• Swift srgtuszy/llama-cpp-swift
• Swift ShenghaiWang/SwiftLlama
• Delphi Embarcadero/llama-cpp-delphi
• Go (no CGo needed): hybridgroup/yzma
• Android: llama.android

UIs

(to have a project listed here, it should clearly state that it depends on llama.cpp)

• AI Sublime Text plugin (MIT)
• [BonzAI