DwarfStar

DwarfStar is a small native inference engine optimized first for DeepSeek V4 Flash, with support for DeepSeek V4 PRO on very high-memory machines. It is intentionally narrow: not a generic GGUF runner, not a wrapper around another runtime: it is completely self-contained. Other than running the model in a correct and fast way, the project goal is to provide DeepSeek specific loading, prompt rendering, tool calling, KV state handling (RAM and on-disk), server API and integrated coding agent, all ready to work with coding agents or with the provided CLI interface. There are also tools for GGUF and imatrix generation, and for quality and speed testing.

We support the following backends: * Metal is our primary target. Starting from MacBooks with 96GB of RAM (or less, using SSD streaming). * NVIDIA CUDA / DGX Spark, CUDA with special care for the DGX Spark. * Strix Halo (ROCm), systems like the Framework Desktop and other systems based on the same GPU and unified RAM design.

This project would not exist without llama.cpp and GGML, make sure to read the acknowledgements section, a big thank you to Georgi Gerganov and all the other contributors.

Note that DeepSeek v4 is not our only target. Right now Flash and PRO are the perfect fit because of capabilities, size, KV cache efficiency. If tomorrow a better open weight model is released for the 128GB size, we could switch, the same for other important size classes like 512GB of RAM. The project is stictly opportunistic depending on what open weight models exist in a given moment. If a new model will be supported, the old one may be removed completely and no longer supported, unless there is some kind of overlap of abilities.

Motivations

• Very capable open weight models finally exist. DeepSeek v4 Flash feels quasi-frontier. The PRO is even better. Both resist 2 bit quantization very well.
• Very capable computers like MacBooks, the DGX Spark now exist.
• DeepSeek v4 kv cache design makes it pratical to run very big contexts. Other vendors are using this approach.
• This few hundred billions models are strictly better than smaller (even if dense) models, regardless of what benchmarks say.

That said, a few important things about this project:

• The local inference landscape contains many excellent projects, but new models are released continuously, and the attention immediately gets captured by the next model to implement. This project takes a deliberately narrow bet: one model at a time, official-vector validation (logits obtained with the official implementation), long-context tests, and enough agent integration to know if it really works. The exact model may change as the landscape evolves, but the constraint remains: local inference credible on high end personal machines or Mac Studios, starting from 96/128GB of memory.
• This software is developed with strong assistance from GPT 5.5 and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without llama.cpp and GGML, largely written by hand.
• This implementation is based on the idea that compressed KV caches like the one of DeepSeek v4 and the fast SSD disks of modern MacBooks should change our idea that KV cache belongs to RAM. The KV cache is actually a first-class disk citizen. Fast SSD disks also changed the inference game from the point of view of "model needs to fit RAM": while having more RAM the the model size is still preferred, SSD streaming allows to turn the available amount of RAM from a hard cutoff (can I run this model or not?) to continuous spectrum of speed levels.
• Our vision is that local inference should be a set of three things working well together, out of the box: A) inference engine with HTTP API + B) GGUF specially crafted to run well under a given engine and given assumptions + C) testing and validation with coding agents implementations. D) Purpose built agents for specific models and execution environments. DwarfStar only runs with the GGUF files provided. It gets tested against officially obtained logits at different context sizes. This project exists because we wanted to make one local model feel finished end to end, not just runnable. However this is beta quality code, so probably we are not still there, especially since recently we introduced large new features: distributed inference, SSD streaming, and other minor improvements.
• The optimized graph path targets Metal on macOS and CUDA on Linux. The CPU path is only for correctness checks and model/tokenizer diagnostics. For CPU-only Linux builds, use make cpu; it builds the normal ./ds4 and ./ds4-server binaries without CUDA or Metal. On macOS, warning: current macOS versions have a bug in the virtual memory implementation that will crash the kernel if you try to run the CPU code. Remember? Software sucks. It was not possible to fix the CPU inference to avoid crashing, since each time you have to restart the computer, which is not funny. Help us, if you have the guts.

Acknowledgements to llama.cpp and GGML

ds4.c does not link against GGML, but it exists thanks to the path opened by the llama.cpp project and the kernels, quantization formats, GGUF ecosystem, and hard-won engineering knowledge developed there. We are thankful and indebted to llama.cpp and its contributors. Their implementation, kernels, tests, and design choices were an essential reference while building this DeepSeek V4 specific inference path. Some source-level pieces are retained or adapted here under the MIT license: GGUF quant layouts and tables, CPU quant/dot logic, and certain kernels. For this reason, and because we are genuinely grateful, we keep the GGML authors copyright notice in our LICENSE file.

Status

The code and GGUF files are to be considered of beta quality because inference and model serving is a complicated matter and all this exists only for a few days. It will take months to reach a more stable form. However, we try to keep the project in a usable state, and we are making progress. If you have issues, make sure to use --trace to log the sessions, and open issues including the full trace.

The ds4-agent is alpha quality, the project was later added.

More Documentation

If you are looking for very specific things, we have other sub-README files. Otherwise for normal usage keep reading the next sections.

• CONTRIBUTING.md: correctness and speed regression testing guide for contributors. Read this before sending a pull request.
• gguf-tools/README.md: offline GGUF generation, imatrix collection, quantization tooling, and quality checks.
• gguf-tools/imatrix/README.md: how the routed-MoE imatrix is collected and used.
• gguf-tools/imatrix/dataset/README.md: how the calibration prompt corpus is generated.
• gguf-tools/quality-testing/README.md: how local GGUFs are scored against official DeepSeek V4 Flash/PRO continuations.
• dir-steering/README.md: directional steering data, vector generation, and usage.
• speed-bench/README.md: benchmark commands, charts, and CSV generation.
• tests/test-vectors/README.md: official continuation vectors used for regression checks.

Model Weights

This implementation only works with the DeepSeek V4 Flash and PRO GGUFs published for this project. It is not a general GGUF loader, and arbitrary DeepSeek/GGUF files will not have the tensor layout, quantization mix, metadata, or optional MTP state expected by the engine. The 2 bit quantizations provided here are not a joke: they behave well, work under coding agents, call tools in a reliable way. The 2 bit quants use a very asymmetrical quantization: only the routed MoE experts are quantized, up/gate at IQ2_XXS, down at Q2_K. They are the majority of all the model space: the other components (shared experts, projections, routing) are left untouched to guarantee quality.

Download one main model. Prefer the imatrix versions.

./download_model.sh q2-imatrix   # 96/128 GB RAM machines, imatrix-tuned q2
./download_model.sh q2-q4-imatrix  # 96/128 GB RAM machines, q2 with last 6 layers q4
./download_model.sh q4-imatrix   # >= 256 GB RAM machines, imatrix-tuned q4
./download_model.sh pro-q2-imatrix  # 512 GB RAM machines, PRO q2 imatrix quant

For the full PRO Q4 distributed run, download one half on each machine:

./download_model.sh pro-q4-layers00-30      # first half of PRO Q4 split
./download_model.sh pro-q4-layers31-output  # second half of PRO Q4 split

The script downloads from https://huggingface.co/antirez/deepseek-v4-gguf, stores files under ./gguf/, resumes partial downloads with curl -C -, and updates ./ds4flash.gguf to point at the selected main model. The pro-q4-layers00-30, pro-q4-layers31-output, and pro-q4-split targets download distributed PRO Q4 pieces and do not update ./ds4flash.gguf. Authentication is optional for public downloads, but --token TOKEN, HF_TOKEN, or the local Hugging Face token cache are used when present.

If you want to regenerate GGUF files or collect a new imatrix, see gguf-tools/README.md. Those tools are meant for offline model-building work and can take a long time on the full DeepSeek V4 Flash weights. Flash GGUF generation is supported by the local tools. PRO GGUF production currently still depends on the external llama.cpp-based workflow; native tooling can be added later.

./download_model.sh mtp fetches the optional speculative decoding support GGUF for Flash. It can be used with q2-imatrix, q2-q4-imatrix, and q4-imatrix, but must be enabled explicitly with --mtp. The current MTP/speculative decoding path is still experimental: it is correctness-gated and currently provides at most a slight speedup, not a meaningful generation-speed win.

Then build:

make                  # macOS Metal
make cuda-spark       # Linux CUDA, DGX Spark / GB10
make cuda-generic     # Linux CUDA, other local CUDA GPUs
make cpu              # CPU-only diagnostics build

./ds4flash.gguf is the default model path used by both binaries. Pass -m to select another supported GGUF from ./gguf/. Run ./ds4 --help and ./ds4-server --help for the full flag list.

Speed

These are single-run Metal CLI numbers with --ctx 32768, --nothink, greedy decoding, and -n 256. The short prompt is a normal small Italian story prompt. The long prompts exercise chunked prefill plus long-context decode. Q4 requires the larger-memory machine class, so M3 Max Q4 numbers are N/A.

Machine	Quant	Prompt	Prefill	Generation
MacBook Pro M3 Max, 128 GB	q2	short	58.52 t/s	26.68 t/s
MacBook Pro M3 Max, 128 GB	q2	11709 tokens	250.11 t/s	21.47 t/s
MacBook Pro M3 Max, 128 GB	q4	short	N/A	N/A
MacBook Pro M3 Max, 128 GB	q4	long	N/A	N/A
MacBook Pro M5 Max, 128 GB	q2	short	87.25 t/s	34.27 t/s
MacBook Pro M5 Max, 128 GB	q2	11707 tokens	463.44 t/s	25.90 t/s
Mac Studio M3 Ultra, 512 GB	q2	short	84.43 t/s	36.86 t/s
Mac Studio M3 Ultra, 512 GB	q2	11709 tokens	468.03 t/s	27.39 t/s
Mac Studio M3 Ultra, 512 GB	q4	short	78.95 t/s	35.50 t/s
Mac Studio M3 Ultra, 512 GB	q4	12018 tokens	448.82 t/s	26.62 t/s
Mac Studio M3 Ultra, 512 GB	PRO q2	32768 tokens	138.82 t/s	9.56 t/s
DGX Spark GB10, 128 GB	q2	7047 tokens	343.81 t/s	13.75 t/s

Running models larger than RAM

The normal Metal path tries to make the model resident in GPU-addressable memory. This is the fastest path and should remain your default when the model fits. When it does not fit, DwarfStar also has a Metal-only SSD streaming capacity mode. In this mode the non-routed model weights stay resident, while routed MoE experts are kept in an in-memory cache and loaded from the GGUF file on cache misses.

Streaming is not as fast as fitting the full model in RAM. It still needs memory for non-routed weights, KV cache, graph scratch, activations, and the routed expert cache. It is useful because routed experts dominate model size and modern Mac SSDs are fast enough to make cache misses tolerable. Long prefills can still be fast; generation is more sensitive to cache misses because every new token routes through experts again.

Start with the automatic cache budget:

./ds4 -m ./ds4flash.gguf --ssd-streaming

If startup reports that the expert cache is too large, or if you want to reserve more memory for context, set the routed expert cache explicitly:

./ds4 -m ./ds4flash.gguf --ssd-streaming --ssd-streaming-cache-experts 32GB

The 32GB value is a memory budget for complete routed experts, not a generic byte cache. DwarfStar converts it to the number of full experts that fit for the current GGUF. Non-routed weights, KV cache, graph scratch, and activations need additional memory. Only the automatic cache budget does the subtraction for you: it takes 80% of the Metal recommended working set, subtracts non-routed weights, then uses the rest for routed experts. Leave the hot expert preload enabled for normal use; use --ssd-streaming-cold and --ssd-streaming-preload-experts N only for measurements.

Practical SSD streaming examples

On 64GB MacBooks, start with the 2-bit Flash GGUF and a moderate expert cache:

```sh ./download_mo