DwarfStar is a small native inference engine optimized first for DeepSeek V4 Flash (including the experimental vision model). It also supports GLM 5.2 and 5.3, GLM 5.3 Flash, and, on very high-memory machines, DeepSeek V4 PRO. It is self-contained and deliberately narrow, not a general GGUF runner. Model loading, prompt rendering, tool calls, KV state, the HTTP server, and the coding agent are built and tested together. The repository also includes tools and data for GGUF, imatrix, quality, and speed.

Supported backends:

• Metal, the primary target, on Macs with 96 GB or more. Smaller machines can use SSD streaming.
• NVIDIA CUDA, including multi-GPU systems and DGX Spark.
• ROCm on Strix Halo systems such as the Framework Desktop.

This project would not exist without llama.cpp and GGML, make sure to read the acknowledgements section, a big thank you to Georgi Gerganov and all the other contributors.

Model support is intentionally opportunistic. The project follows the best open weights for useful local machine sizes, especially 128 GB laptops and 512 GB workstations. A model may be removed when a better replacement arrives.

The project has first class support for SSD streaming of weights, so it is possible to run models bigger than RAM while often still getting decent performances, and even running very large models (like the full GLM 5.3 or DeepSeek v4 PRO) on systems with just 128GB of RAM at a slower speed, but fast enough for QA-style chats.

So, what can I do with this software?

• You can run a very capable models in your consumer hardware, a MacBook, a DGX Spark, or a Strix Halo for example. Even if you have not enough RAM, with SSD streaming, you can run it at a decent speed.
• Using the CUDA multi-GPU support and with ds4-server micro batching of decoding and generation, you can turn a server with old-ish CUDA cards (Ada Lovelace architecture), no longer supported for new models by vLLM, into a multi-user LLM server for your company. We tested this setup with 8xL40S NVIDIA cards and multiple sessions with very good results. 120 t/s aggreated generation, 2000 t/s prefill.
• Using two MacBook M5 Max / M3 Ultra RDMA, you can run 4 bit DeepSeek Flash, GLM 5.2, or GLM 5.3 Flash with tensor parallelism.
• You can also use pipeline paralellism to glue together multiple systems to sum their RAM and run larger models.

Motivations

• Capable open-weight models now fit on high-end personal machines.
• DeepSeek V4 Flash and PRO, GLM 5.2, tolerate aggressive routed-expert quantization.
• Compressed KV caches and fast local SSDs make long contexts practical.
• The idea of an inference system specialized for a few models.

AI full disclosure

• This software is developed with strong assistance from GPT 5.5, 5.6, Claude Fable and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without llama.cpp and GGML, largely written by hand.

Acknowledgements to llama.cpp and GGML

ds4.c does not link against GGML, but it exists thanks to the path opened by the llama.cpp project and the kernels, quantization formats, GGUF ecosystem, and hard-won engineering knowledge developed there. We are thankful and indebted to llama.cpp and its contributors. Their implementation, kernels, tests, and design choices were an essential reference while building this DeepSeek V4 specific inference path. Some source-level pieces are retained or adapted here under the MIT license: GGUF quant layouts and tables, CPU quant/dot logic, and certain kernels. For this reason, and because we are genuinely grateful, we keep the GGML authors copyright notice in our LICENSE file.

Status

The software is currently very fast changing. Consider it beta quality. Before each release, a big QA run is executed, however instabilities are definitely possible.

How to use this project?

I (Salvatore) believe that the way projects should be shipped and used changed because of AI. The main differences today are:

1. With AI, users can modify the software in significant ways with low efforts, costs, and even lacking deep domain knowledge about the task they want to accomplish. For instance, a DwarfStar user with a specific hardware setup can ask a coding agent to improve the inference speed of this software for the specific hardware setup, asking the model to reach the maximum prefill and generation speed without impacting correctness, and also asking to do a deep QA pass.
2. Similiarly, because of "1", software may be shipped in a different way than before. It must be more a working template for the biggest use cases, without trying to cover every possible setup. If DwarfStar showcases a few good implementations of tensor parallel execution, the code will work as a rail for implementing the same feature in specific conditions, for a new model, and so forth.

So, while this project attempts to be usable for the featured models and the most common hardware setups, I ask you, if you have access to coding agents, to consider using coding agents as an interface to discover the project, make modifications, create personalized setups. This way you can likely do more than what we ship, and certain things that are not documented or implemented, and that you require, are potentially very easy to achieve.

More Documentation

If you are looking for very specific things, we have other sub-README files. Otherwise for normal usage keep reading the next sections.

• CONTRIBUTING.md: correctness and speed regression testing guide for contributors. Read this before sending a pull request.
• QA_BEFORE_RELEASES.md: the complete release test matrix, including the remote Metal, CUDA, and ROCm machines.
• gguf-tools/README.md: offline GGUF generation, imatrix collection, quantization tooling, and quality checks.
• gguf-tools/imatrix/README.md: how the routed-MoE imatrix is collected and used.
• gguf-tools/imatrix/dataset/README.md: how the calibration prompt corpus is generated.
• gguf-tools/quality-testing/README.md: how local GGUFs are scored against official DeepSeek V4 Flash/PRO continuations.
• dir-steering/README.md: directional steering data, vector generation, and usage.
• speed-bench/README.md: benchmark commands, charts, and CSV generation.
• tests/test-vectors/README.md: official continuation vectors used for regression checks.

Model Weights

This implementation only works with the DeepSeek V4 and GLM GGUFs listed below. It is not a general GGUF loader, and arbitrary GGUF files will not have the tensor layout, quantization mix, metadata, or optional MTP state expected by the engine. The 2 bit quantizations provided here are verified to be actually high quality: they behave well, work under coding agents, call tools in a reliable way.

The 2 bit quants use a very asymmetrical quantization: only the routed MoE experts are quantized, up/gate at IQ2_XXS, down at Q2_K. They are the majority of all the model space: the other components (shared experts, projections, routing) are left untouched to guarantee quality.

Download one main model. Prefer the imatrix versions.

./download_model.sh ds4f-q2      # 96/128 GB RAM machines
./download_model.sh ds4f-q2-q4   # q2 with the last 6 expert layers at q4
./download_model.sh ds4f-q4      # >= 256 GB RAM machines
./download_model.sh ds4f-mxfp4   # native MXFP4 experts, about 156 GB
./download_model.sh pro-q2-imatrix  # 512 GB RAM machines, PRO 0813 q2 imatrix

The MXFP4 GGUF preserves DeepSeek's released MXFP4 routed-expert weights rather than requantizing them. It runs on Metal and CUDA; Blackwell CUDA devices use native FP4 matrix instructions and FP4 activations for batched expert work. Decode and other CUDA devices use Q8 activations.

For the full PRO Q4 distributed run, download one half on each machine:

./download_model.sh pro-q4-layers00-30      # first half of PRO Q4 split
./download_model.sh pro-q4-layers31-output  # second half of PRO Q4 split

The script stores files under ./gguf/ and updates ./ds4flash.gguf to point at the selected main model. DeepSeek files come from antirez/deepseek-v4-gguf; GLM targets use the repository named in the script's help. Smaller files resume with curl -C -, while large files use the official Hugging Face downloader. The pro-q4-layers00-30, pro-q4-layers31-output, and pro-q4-split targets download distributed PRO Q4 pieces and do not update ./ds4flash.gguf. Authentication is optional for public downloads, but --token TOKEN, HF_TOKEN, or the local Hugging Face token cache are used when present.

If you want to regenerate GGUF files or collect a new imatrix, see gguf-tools/README.md. Those tools are meant for offline model-building work and can take a long time on the full DeepSeek weights. Flash and PRO GGUF generation are supported by the local tools. PRO conversion uses a compatible published PRO GGUF as its metadata, tensor-layout, and output type template.

GLM 5.2 support is limited to the GGUF files tested by this branch:

./download_model.sh glm-unsloth-q4  # Unsloth UD-Q4_K_XL, 11 shards
./download_model.sh glm-antirez-iq2xxs  # antirez routed IQ2_XXS single-file GGUF
./download_model.sh glm-antirez-q2  # antirez routed Q2_K single-file GGUF
./download_model.sh glm-antirez-q4  # antirez routed Q4_K single-file GGUF

GLM 5.3 Flash has its own graph, artifacts, and run instructions in the GLM 5.3 Flash section below.

The full GLM 5.3 Q2 model is about 197 GiB. It can run resident on a 256 GB machine, or with SSD streaming on a smaller system:

./download_model.sh glm53-full-q2
./ds4 -m gguf/GLM-5.3-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf --ssd-streaming

The supported GLM 5.2 layout keeps dense/model-control tensors in the existing Q8/F32 paths and supports routed expert gate/up tensors in Q2_K, Q4_K, or Q5_K; routed expert down tensors are supported in Q2_K, Q4_K, Q5_K, or Q6_K. Other GLM GGUF quant layouts should be treated as unsupported until they are added deliberately and scored against the official 100-case fixture.

These GLM 5.2 formats do not all support the same execution modes. The Q4 files work for normal Metal and CUDA inference. Two-Mac tensor parallelism for GLM 5.2 currently requires an ownership-aware IQ2_XXS or Q2_K routed layout. GLM 5.3 has its own ownership-aware Q4 path.

GLM's MTP block is part of the main GGUF; it does not use the separate Flash MTP file. Ordinary decode remains the default. --mtp enables experimental greedy speculation. --mtp-timing also enables it and prints acceptance and timing counters:

./ds4 -m gguf/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf \
  --mtp-timing --temp 0

GLM 5.2 uses the Metal, CUDA, or ROCm graph backend. GLM 5.3 is validated on Metal, with Q2 also validated on CUDA. Directional steering, --power below 100, an explicit --prefill-chunk, and an external --mtp-model file are not supported for GLM yet.

Then build:

make                  # macOS Metal
make cuda-spark       # Linux CUDA, DGX Spark / GB10
make cuda-generic     # Linux CUDA, other local CUDA GPUs
make strix-halo       # Linux ROCm, AMD Strix Halo
make cpu              # CPU-only diagnostics build

For ROCm packages, GTT configuration and the reproducible ROCm 10.0 container build, see DS4 on Strix Halo.

./ds4flash.gguf is the default model path used by both binaries. Pass -m to select another supported GGUF from ./gguf/. Run ./ds4 --help and ./ds4-server --help for the full flag list.

DeepSeek V4 Flash Vision Experimental

Vision-Exp is a separate DeepSeek checkpoint, not the 0731 text model. It uses a matching language GGUF and a 0.9 GiB vision encoder. The Q2 model is the recommended version for 96 and 128 GB systems:

./download_model.sh ds4f-vision-q2
./ds4 --vision gguf/DeepSeek-V4-Flash-Vision-Encoder.gguf

Use /read image.png in the interactive CLI. The same --vision option gives ds4-agent its view_image tool and enables image input through ds4-server. PNG and JPEG are supported. The Q2 path is validated on Metal, single-GPU CUDA including DGX Spark, and ROCm. Larger ds4f-vision-q2-q4 and ds4f-vision-mxfp4 downloads are also available.

Vision-Exp has its own DSpark checkpoint. Do not use the 0731 support model:

./download_model.sh ds4f-vision-dspark
./ds4 --vision gguf/DeepSeek-V4-Flash-Vision-Encoder.gguf --dspark \
  --mtp-model gguf/DeepSeek-V4-Flash-Vision-Exp-DSpark-support.gguf

GLM 5.3 Flash

GLM 5.3 Flash uses a separate graph for its recurrent KDA layers, sparse DSA layers, hyper-connections, and built-in MTP block. The release GGUFs were made from the official FP8 checkpoint:

./download_model.sh glm53-q2  # about 90 GiB
./download_model.sh glm53-q4  # about 178 GiB
./download_model.sh glm53-fp8 # about 305 GiB; packaged weights only

The Q2 file uses imatrix-guided IQ2_XXS gate/up experts and Q2_K down experts. It runs resident on a 128 GB M3 Max or M5 Max, and on one DGX Spark. The Q4 file is the higher-quality control. Run it across two 128 GB Macs, or use SSD streaming on one Mac. The FP8 file preserves the released text weights without requantization; DwarfStar cannot execute that artifact yet.

On one 128 GB Mac:

```sh ./ds4 -m gguf/GLM-5.3-Flash-Q2.