🦙 llama-cpp-rs

Safe Rust bindings to llama.cpp, tracking upstream closely.

Crate	Description	crates.io
llama-cpp-4	Safe high-level API
llama-cpp-sys-4	Raw bindgen bindings

llama.cpp version: 4fc4ec5 (b9859) (Jun 2026) — includes TurboQuant (PR #21038), MTP / multi-token-prediction speculative decoding (PR #22673), and upstream next-n embedding hooks used by MTP (llama_set_embeddings_nextn).

---

Using the library

[dependencies]
llama-cpp-4 = "0.4.0"

Import the common types with the prelude:

use llama_cpp_4::prelude::*;

Core types are also at the crate root (llama_cpp_4::LlamaModel, …). See llama-cpp-4/README.md for the full API guide and prelude on docs.rs for runnable examples.

---

Examples

Package name	Directory	Description
simple	examples/simple/	Single-turn text completion from CLI or Hugging Face
chat	examples/chat/	Interactive multi-turn chat REPL
embeddings	examples/embeddings/	Batch embedding with cosine similarity
split-model-example	examples/split_model/	Load sharded / split GGUF files
openai-server	examples/server/	OpenAI-compatible HTTP server — chat, completions, embeddings, tools, files (mtmd), tokenize
mtmd	examples/mtmd/	Multimodal (vision / audio) inference (requires --features mtmd)
quantize	examples/quantize/	Quantize a GGUF model with full typed API
turbo-quant	examples/turbo-quant/	TurboQuant demo — compare attn rotation on/off
incremental-chat	examples/incremental-chat/	Chat with incremental prefill — processes tokens while you type
mtp	examples/mtp/	MTP speculative decoding via MtpSession (--predict, --p-min, draft loop)

---

Quick start

git clone --recursive https://github.com/eugenehp/llama-cpp-rs
cd llama-cpp-rs

Interactive chat

cargo run -p chat -- \
    hf-model bartowski/Llama-3.2-3B-Instruct-GGUF Llama-3.2-3B-Instruct-Q4_K_M.gguf

OpenAI-compatible server

# Starts on http://127.0.0.1:8080
cargo run -p openai-server -- \
    hf-model bartowski/Llama-3.2-3B-Instruct-GGUF Llama-3.2-3B-Instruct-Q4_K_M.gguf

Full REST API reference: examples/server/README.md.

Method	Path	Description
GET	/health, /v1/health	Liveness (no auth)
GET	/v1/models	Loaded model metadata
POST	/v1/chat/completions, /chat/completions	Chat · streaming · tools
POST	/v1/completions, /completions	Raw completion · streaming
POST	/v1/embeddings, /embeddings	L2-normalised embeddings
POST	/tokenize, /detokenize	llama.cpp-compatible token helpers
POST/GET/DELETE	/v1/files/...	File store for multimodal (--features mtmd, --mmproj)

Legacy paths without /v1 mirror upstream llama-server. Not implemented here (use upstream server instead): /v1/responses, /v1/messages, /rerank, /slots, /props.

Using prebuilt native libraries (skip CMake compile)

llama-cpp-sys-4 can consume precompiled llama/ggml libraries via env vars. This is useful for CI pipelines that publish native artifacts once and reuse them in downstream repos (for example, speeding up a separate app build).

# Directory containing prebuilt libs in one of:
#   <dir>, <dir>/lib, <dir>/lib64, <dir>/bin
export LLAMA_PREBUILT_DIR=/path/to/prebuilt

# Optional: force dynamic linking mode for prebuilt artifacts.
# Defaults to the crate's normal link mode for the active feature set.
# export LLAMA_PREBUILT_SHARED=1

cargo build -p your-app --features "q1,vulkan"

Notes: - q1 compatibility is determined by the prebuilt artifact itself — publish separate artifacts per feature/backend tuple (q1+vulkan, q1+metal, ...). - build.rs still generates Rust bindings, but skips the expensive CMake compile when LLAMA_PREBUILT_DIR is set.

Backend feature coverage (practical targets): - metal → macOS (Apple Silicon and Intel Macs) - vulkan → Linux/Windows (cross-vendor desktop GPUs) - webgpu → Linux/Windows (experimental; requires Dawn/WebGPU-native stack) - cuda → Linux/Windows with NVIDIA CUDA toolkit (experimental in CI) - hip → Linux ROCm/HIP environments (experimental in CI)

Prebuilt Feature Benchmark Results

The prebuilt feature flag provides automatic prebuilt artifact management. Benchmark results (Apple Silicon M2, macOS 14.4):

Configuration	Build Type	Time	Improvement
Base (Static)	Debug	11.99s	Baseline
Base + prebuilt	Debug	11.01s	8% faster
Dynamic Linking	Debug	26.80s	-123% (slower)
Dynamic + prebuilt	Debug	27.47s	-129% (slower)
Base (Static)	Release	26.01s	Baseline
Dynamic Linking	Release	26.79s	-3% (slower)

Key Insights: - ✅ Static linking + prebuilt: 8% faster debug builds (11.99s → 11.01s) - ✅ Release builds: Minimal difference between static/dynamic - ✅ Development workflow: Prebuilt feature provides best iteration speed - 🚀 CI/CD potential: When fully implemented with artifact caching, expect 50-80% speedups for complex builds

Usage:

# Enable prebuilt feature for faster development
cargo build --features prebuilt

# Combine with other features
cargo build --features "prebuilt,vulkan"

# Release builds (prebuilt provides minimal benefit)
cargo build --release --features prebuilt

Implementation Status: - ✅ Feature flag infrastructure complete - ✅ Automatic feature detection and configuration - ✅ Safe fallback to local compilation - ✅ Automatic download from GitHub releases into target/llama-prebuilt-cache/

When the prebuilt feature is enabled, build.rs will: 1. Resolve the matching release asset for your target and backend (cpu, vulkan, blas, metal) 2. Download it from GitHub releases (tag defaults to v{CARGO_PKG_VERSION}) 3. Cache extracted libraries under target/llama-prebuilt-cache/ 4. Fall back gracefully to local compilation if no asset is available

Environment overrides:

Variable	Description
LLAMA_PREBUILT_DIR	Use a local directory (skips download)
LLAMA_PREBUILT_TAG	Release tag to download (default: crate version, e.g. v0.4.0)
LLAMA_PREBUILT_REPO	GitHub owner/repo (default: eugenehp/llama-cpp-rs)
LLAMA_PREBUILT_URL	Full URL override for the tarball
LLAMA_PREBUILT_OFF	Set to 1 to disable auto-download
LLAMA_PREBUILT_SHARED	Force shared/dynamic linking when using LLAMA_PREBUILT_DIR

Manual prefetch:

./scripts/fetch-prebuilt.sh
cargo build --features prebuilt

• opencl → Linux/Windows with OpenCL SDK/runtime (experimental in CI)
• blas → CPU acceleration (Linux/macOS/Windows)

# Chat completion (max_completion_tokens is also accepted)
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello!"}], "max_tokens":128}'

# Streaming
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Count to 5"}], "stream":true}'

# Embeddings
curl http://127.0.0.1:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": ["Hello world", "Bonjour le monde"]}'

# Tokenize / detokenize (llama.cpp server-compatible)
curl http://127.0.0.1:8080/tokenize \
  -H "Content-Type: application/json" \
  -d '{"content":"Hello","add_special":false}'

With --api-key, pass Authorization: Bearer <key> on every route except /health and /v1/health.

Text generation (library)

use llama_cpp_4::prelude::*;
use std::num::NonZeroU32;

fn main() -> anyhow::Result<()> {
    let backend = LlamaBackend::init()?;
    let model = LlamaModel::load_from_file(
        &backend,
        "model.gguf",
        &LlamaModelParams::default(),
    )?;

    let mut ctx = model.new_context(
        &backend,
        LlamaContextParams::default().with_n_ctx(NonZeroU32::new(2048)),
    )?;

    let tokens = model.str_to_token("Hello, world!", AddBos::Always)?;
    let mut batch = LlamaBatch::new(512, 1);
    for (i, &tok) in tokens.iter().enumerate() {
        batch.add(tok, i as i32, &[0], i == tokens.len() - 1)?;
    }
    ctx.decode(&mut batch)?;

    let sampler = LlamaSampler::chain_simple([LlamaSampler::greedy()]);
    let token = sampler.sample(&ctx, 0);
    let piece = model.token_to_bytes(token, Special::Plaintext)?;
    println!("{}", String::from_utf8_lossy(&piece));
    Ok(())
}

---

Quantization

The llama_cpp_4::quantize module provides a fully typed Rust API for all quantization options.

use llama_cpp_4::prelude::*;
use llama_cpp_4::quantize::TensorTypeOverride;

// Basic — quantize to Q4_K_M
let params = QuantizeParams::new(LlamaFtype::MostlyQ4KM)
    .with_nthread(8)
    .with_quantize_output_tensor(true);

llama_cpp_4::model_quantize("model-f16.gguf", "model-q4km.gguf", &params).unwrap();

// Advanced — keep output tensor in F16, prune layers 28-31
let params = QuantizeParams::new(LlamaFtype::MostlyQ5KM)
    .with_tensor_type_override(TensorTypeOverride::new("output", GgmlType::F16).unwrap())
    .with_pruned_layers(28..=31);

llama_cpp_4::model_quantize("model-f16.gguf", "model-q5km-pruned.gguf", &params).unwrap();

From the CLI:

# List all available quantization types
cargo run -p quantize -- --list-types

# Quantize with auto output name
cargo run -p quantize -- model-f16.gguf Q4_K_M

# Override a specific tensor type
cargo run -p quantize -- --tensor-type output=F16 model-f16.gguf Q5_K_M

# Dry-run: show size without writing
cargo run -p quantize -- --dry-run model-f16.gguf Q4_K_M

---

TurboQuant — attention rotation

TurboQuant (llama.cpp PR #21038) applies a Hadamard rotation to the Q, K, and V tensors before they are stored in the KV cache.

Why it matters

Attention activations have large outlier values on some dimensions that make quantization hard. The rotation spreads these outliers evenly so the KV cache can be stored in aggressive formats (Q4_0, Q5_0) with drastically less quality loss:

KV cache type	Without TurboQuant	With TurboQuant	VRAM vs F16
F16 (baseline)	—	—	100%
Q8_0	+0.003 PPL	+0.003 PPL	53%
Q5_1	+61.70 PPL	+0.44 PPL	37%
Q5_0	+17.28 PPL	+0.55 PPL	34%
Q4_1	+212.5 PPL	+8.65 PPL	31%
Q4_0	+62.02 PPL	+32.6 PPL	28%

PPL delta vs F16 baseline on Qwen3 0.6B BF16 — source: llama.cpp PR #21038.

Measured KV-cache space savings

Numbers below come from a benchmark run against Qwen2.5-0.5B-Instruct (24 layers, 2 KV heads, 64 head-dim), obtained by calling ggml_row_size() directly against the compiled GGML library in this repo's build tree.

Model : Qwen2.5-0.5B-Instruct  (24 layers, 2 KV heads, 64 head-dim)

Config                 B/row  B/elem     KV @2K      KV @32K  Saved@32K  Ratio
--------------------  ------  ------  ---------  ----------  ---------  -----
F16  (baseline)          128  2.0000   24.00 MB   384.00 MB      —       1.00x
Q8_0 + TurboQuant         68  1.0625   12.75 MB   204.00 MB  180.0 MB   1.88x
Q5_1 + TurboQuant         48  0.7500    9.00 MB   144.00 MB  240.0 MB   2.67x
Q5_0 + TurboQuant         44  0.6875    8.25 MB   132.00 MB  252.0 MB   2.91x  ← sweet spot
Q4_1 + TurboQuant         40  0.6250    7.50 MB   120.00 MB  264.0 MB   3.20x
Q4_0 + TurboQuant         36  0.5625    6.75 MB   108.00 MB  276.0 MB   3.56x

The ratios are pure GGML block geometry and scale identically to larger models — for a 7B model (32 layers, 8 KV heads, 128 head-dim) multiply every MB figure by ~85×; the ratios and % savings are the same.

Sweet spot: Q5_0 + TurboQuant

• 2.91× smaller KV cache than vanilla F16 (saves 252 MB per 32 K context window on the 0.5B model, ~21 GB on a 70B model at 32 K ctx)
• Only +0.55 PPL delta — essentially indistinguishable from F16 in practice
• The same Q5_0 without TurboQuant gives +17.28 PPL (noticeably wrong output)
• Q8_0 is the conservative zero-risk choice (1.88×, near-zero PPL cost)
• Q4_0 gives maximum compression (3.56×) at the price of measurable but tolerable quality loss with rotation on

Key properties

• Enabled automatically for any model whose head dimension is a power of two (covers essentially all modern transformers).
• No GGUF changes required — it is a runtime transform of the KV cache only.
• Reversible — the rotation is applied before storing and reversed before