DwarfStar is a small native inference engine optimized first for DeepSeek V4 Flash. It also supports GLM 5.2 and, on very high-memory machines, DeepSeek V4 PRO. It is self-contained and deliberately narrow, not a general GGUF runner. Model loading, prompt rendering, tool calls, KV state, the HTTP server, and the coding agent are built and tested together. The repository also includes tools and data for GGUF, imatrix, quality, and speed.

Supported backends:

• Metal, the primary target, on Macs with 96 GB or more. Smaller machines can use SSD streaming.
• NVIDIA CUDA, including multi-GPU systems and DGX Spark.
• ROCm on Strix Halo systems such as the Framework Desktop.

This project would not exist without llama.cpp and GGML, make sure to read the acknowledgements section, a big thank you to Georgi Gerganov and all the other contributors.

Model support is intentionally opportunistic. The project follows the best open weights for useful local machine sizes, especially 128 GB laptops and 512 GB workstations. A model may be removed when a better replacement arrives.

So, what can I do with this software?

• You can run a very capable models in your consumer hardware, a MacBook, a DGX Spark, or a Strix Halo for example. Even if you have not enough RAM, with SSD streaming, you can run it at a decent speed.
• Using the CUDA multi-GPU support and with ds4-server micro batching of decoding and generation, you can turn a server with old-ish CUDA cards (Ada Lovelace architecture), no longer supported for new models by vLLM, into a multi-user LLM server for your company. We tested this setup with 8xL40S NVIDIA cards and multiple sessions with very good results. 120 t/s aggreated generation, 2000 t/s prefill.
• Using two MacBook M5 Max / M3 Ultra RDMA, you can run 4 bit DeepSeek Flash or GLM 5.2 with tensor parallelism.
• You can also use pipeline paralellism to glue together multiple systems to sum their RAM and run larger models.

Motivations

• Capable open-weight models now fit on high-end personal machines.
• DeepSeek V4 Flash and PRO, GLM 5.2, tolerate aggressive routed-expert quantization.
• Compressed KV caches and fast local SSDs make long contexts practical.
• The idea of an inference system specialized for a few models.

AI full disclosure

• This software is developed with strong assistance from GPT 5.5, 5.6, Claude Fable and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without llama.cpp and GGML, largely written by hand.

Acknowledgements to llama.cpp and GGML

ds4.c does not link against GGML, but it exists thanks to the path opened by the llama.cpp project and the kernels, quantization formats, GGUF ecosystem, and hard-won engineering knowledge developed there. We are thankful and indebted to llama.cpp and its contributors. Their implementation, kernels, tests, and design choices were an essential reference while building this DeepSeek V4 specific inference path. Some source-level pieces are retained or adapted here under the MIT license: GGUF quant layouts and tables, CPU quant/dot logic, and certain kernels. For this reason, and because we are genuinely grateful, we keep the GGML authors copyright notice in our LICENSE file.

Status

The software is currently very fast changing. Consider it beta quality. Before each release, a big QA run is executed, however instabilities are definitely possible.

How to use this project?

I (Salvatore) believe that the way projects should be shipped and used changed because of AI. The main differences today are:

1. With AI, users can modify the software in significant ways with low efforts, costs, and even lacking deep domain knowledge about the task they want to accomplish. For instance, a DwarfStar user with a specific hardware setup can ask a coding agent to improve the inference speed of this software for the specific hardware setup, asking the model to reach the maximum prefill and generation speed without impacting correctness, and also asking to do a deep QA pass.
2. Similiarly, because of "1", software may be shipped in a different way than before. It must be more a working template for the biggest use cases, without trying to cover every possible setup. If DwarfStar showcases a few good implementations of tensor parallel execution, the code will work as a rail for implementing the same feature in specific conditions, for a new model, and so forth.

So, while this project attempts to be usable for the featured models and the most common hardware setups, I ask you, if you have access to coding agents, to consider using coding agents as an interface to discover the project, make modifications, create personalized setups. This way you can likely do more than what we ship, and certain things that are not documented or implemented, and that you require, are potentially very easy to achieve.

More Documentation

If you are looking for very specific things, we have other sub-README files. Otherwise for normal usage keep reading the next sections.

• CONTRIBUTING.md: correctness and speed regression testing guide for contributors. Read this before sending a pull request.
• QA_BEFORE_RELEASES.md: the complete release test matrix, including the remote Metal, CUDA, and ROCm machines.
• gguf-tools/README.md: offline GGUF generation, imatrix collection, quantization tooling, and quality checks.
• gguf-tools/imatrix/README.md: how the routed-MoE imatrix is collected and used.
• gguf-tools/imatrix/dataset/README.md: how the calibration prompt corpus is generated.
• gguf-tools/quality-testing/README.md: how local GGUFs are scored against official DeepSeek V4 Flash/PRO continuations.
• dir-steering/README.md: directional steering data, vector generation, and usage.
• speed-bench/README.md: benchmark commands, charts, and CSV generation.
• tests/test-vectors/README.md: official continuation vectors used for regression checks.

Model Weights

This implementation only works with the DeepSeek V4 and GLM 5.2 GGUFs listed below. It is not a general GGUF loader, and arbitrary GGUF files will not have the tensor layout, quantization mix, metadata, or optional MTP state expected by the engine. The 2 bit quantizations provided here are verified to be actually high quality: they behave well, work under coding agents, call tools in a reliable way.

The 2 bit quants use a very asymmetrical quantization: only the routed MoE experts are quantized, up/gate at IQ2_XXS, down at Q2_K. They are the majority of all the model space: the other components (shared experts, projections, routing) are left untouched to guarantee quality.

Download one main model. Prefer the imatrix versions.

./download_model.sh ds4f-q2      # 96/128 GB RAM machines
./download_model.sh ds4f-q2-q4   # q2 with the last 6 expert layers at q4
./download_model.sh ds4f-q4      # >= 256 GB RAM machines
./download_model.sh ds4f-mxfp4   # native MXFP4 experts, about 156 GB
./download_model.sh pro-q2-imatrix  # 512 GB RAM machines, PRO 0813 q2 imatrix

The MXFP4 GGUF preserves DeepSeek's released MXFP4 routed-expert weights rather than requantizing them. It runs on Metal and CUDA; Blackwell CUDA devices use native FP4 matrix instructions and FP4 activations for batched expert work. Decode and other CUDA devices use Q8 activations.

For the full PRO Q4 distributed run, download one half on each machine:

./download_model.sh pro-q4-layers00-30      # first half of PRO Q4 split
./download_model.sh pro-q4-layers31-output  # second half of PRO Q4 split

The script downloads from https://huggingface.co/antirez/deepseek-v4-gguf, stores files under ./gguf/, resumes partial downloads with curl -C -, and updates ./ds4flash.gguf to point at the selected main model. The pro-q4-layers00-30, pro-q4-layers31-output, and pro-q4-split targets download distributed PRO Q4 pieces and do not update ./ds4flash.gguf. Authentication is optional for public downloads, but --token TOKEN, HF_TOKEN, or the local Hugging Face token cache are used when present.

If you want to regenerate GGUF files or collect a new imatrix, see gguf-tools/README.md. Those tools are meant for offline model-building work and can take a long time on the full DeepSeek weights. Flash and PRO GGUF generation are supported by the local tools. PRO conversion uses a compatible published PRO GGUF as its metadata, tensor-layout, and output type template.

GLM 5.2 support is limited to the GGUF files tested by this branch:

./download_model.sh glm-unsloth-q4  # Unsloth UD-Q4_K_XL, 11 shards
./download_model.sh glm-antirez-iq2xxs  # antirez routed IQ2_XXS single-file GGUF
./download_model.sh glm-antirez-q2  # antirez routed Q2_K single-file GGUF
./download_model.sh glm-antirez-q4  # antirez routed Q4_K single-file GGUF

The supported GLM layout keeps dense/model-control tensors in the existing Q8/F32 paths and supports routed expert gate/up tensors in Q2_K, Q4_K, or Q5_K; routed expert down tensors are supported in Q2_K, Q4_K, Q5_K, or Q6_K. Other GLM GGUF quant layouts should be treated as unsupported until they are added deliberately and scored against the official 100-case fixture.

These formats do not all support the same execution modes. The Q4 files work for normal Metal and CUDA inference. Two-Mac tensor parallelism currently requires an ownership-aware IQ2_XXS or Q2_K routed layout; a routed Q4 GLM must be rejected before evaluation.

GLM's MTP block is part of the main GGUF; it does not use the separate Flash MTP file. Ordinary decode remains the default. --glm-mtp enables experimental greedy speculation. --glm-mtp-timing also enables it and prints acceptance and timing counters:

./ds4 -m gguf/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf \
  --glm-mtp-timing --temp 0

GLM inference uses the Metal, CUDA, or ROCm graph backend. Directional steering, --power below 100, an explicit --prefill-chunk, and the external --mtp file are not supported for GLM yet.

Then build:

make                  # macOS Metal
make cuda-spark       # Linux CUDA, DGX Spark / GB10
make cuda-generic     # Linux CUDA, other local CUDA GPUs
make strix-halo       # Linux ROCm, AMD Strix Halo
make cpu              # CPU-only diagnostics build

./ds4flash.gguf is the default model path used by both binaries. Pass -m to select another supported GGUF from ./gguf/. Run ./ds4 --help and ./ds4-server --help for the full flag list.

DSpark Speculative Decoding

DSpark is an auxiliary draft model released by DeepSeek for DeepSeek V4 Flash. It reads hidden states from the main model and proposes up to five future tokens. DwarfStar checks those proposals with the main Flash model and commits only the accepted prefix. The main model remains authoritative; a rejected or low-confidence suffix falls back to ordinary target decoding.

The possible gain is faster generation: when several proposed tokens are accepted, one target verification pass advances the stream by several tokens. It does not accelerate prefill, and the draft and verification work is not free. Predictable continuations, especially code, tend to benefit most; low-yield prompts can be no faster or even slower. DSpark is therefore still experimental and explicitly opt-in.

Accepted proposals keep the state produced by the batched target verifier instead of running the same tokens through one-token decode again. Both paths execute the same inference graph, but floating-point operations are grouped in a different order. A long greedy DSpark run may therefore diverge from a run without DSpark after an otherwise valid accepted block. This is not a reduced precision or approximate-model mode; use ordinary decoding, --quality, or --dspark-strict when byte-for-byte reproducibility with one-token decode is required.

The DSpark checkpoint for Flash 0731 is packaged here as a separate support GGUF of about 5.6 GiB. It is not a standalone model. Download it once:

./download_model.sh ds4f-dspark

The support file can be used with the 0731 Flash ds4f-q2, ds4f-q2-q4, and ds4f-q4 models listed above. It is checkpoint-specific and must not be paired with an older Flash model. For now DeepSeek V4 PRO is not supported. On Metal, the main model may be resident or use --ssd-streaming; the support model still adds its own weights and runtime state to the memory requirement. DSpark replaces the legacy one-stage MTP support model for that run rather than stacking with it.

Run it with the normal sampling defaults:

./ds4 -m ds4flash.gguf \
  --mtp gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
  --dspark

--mtp supplies the support GGUF, while --dspark selects the DSpark runtime. The default confidence threshold is 0.6 on Metal and 0.7 on CUDA and ROCm. It prunes suffixes that are unlikely to repay their verification cost. --dspark-confidence 0 forces fixed five-token blocks and is intended for diagnostics.

At a non-zero temperature, ordinary --dspark uses opportunistic sampling. Tokens evaluated normally are sampled with the requested temperature, top-p, top-k, and min-p. DFlash then proposes a temperature-zero suffix. Every draft token that matches the target's temperature-zero continuation is committed directly, even though the r