github.com/Luce-Org/lucebox-hub @main

Local LLM inference server built for speed. Custom kernels, speculative prefill & decoding.

Each optimization in our engine is for specific model family and hardware target.

---

Inference Engine Optimizations

Each one is self-contained with setup instructions and benchmark notes.

  

  

---

Supported Models & Drafters

All speedups measured vs vendored llama.cpp (-fa 1, matching KV quant). Combined = geometric mean √(TTFT × decode) where both phases benched; otherwise the single-phase speedup. Drafters published on huggingface.co/Lucebox.

| Model | Speedup | |-------|:-------:| | Qwen 3.5-0.8B (Megakernel) | **~2×** | | Qwen 3.6-27B + PFlash | **~5.6×** | | Qwen 3.6-27B + DDTree | **4.84×** | | Laguna-XS-2.1 33B + PFlash | **8.2×** @256K | | Laguna-XS-2.1 33B + DFlash | **1.7×** @256K | | Qwen 3.6-27B HIP | **~2.6×** | | Gemma-4-26B-A4B | **1.31×** |

| Drafter | Phase | |---------|:-----:| | [`Qwen3.6-27B`](https://huggingface.co/Lucebox/Qwen3.6-27B-DFlash-GGUF) | decode | | [`gemma-4-26B-A4B`](https://huggingface.co/Lucebox/gemma-4-26B-A4B-it-DFlash-GGUF) | decode | | [`gemma-4-31B`](https://huggingface.co/Lucebox/gemma-4-31B-it-DFlash-GGUF) | decode | | [`Qwen3-0.6B`](https://huggingface.co/Qwen/Qwen3-0.6B) | prefill |

Tested Machines (GPU/APU)

Reference target: RTX 3090 (Ampere sm_86) — all headline numbers. Other NVIDIA archs auto-detected by CMake / setup.py; AMD HIP backend separate (Strix Halo section).

	Arch	GPU	Min CUDA / ROCm	Status	Bench
	Ampere sm_86	RTX 3090, A-series	CUDA 12.0	✅ reference	megakernel · dflash
	Blackwell sm_120	RTX 5090	CUDA 12.8	✅ 205 tok/s, 4.84×	↗
	Blackwell sm_121	DGX Spark / GB10	CUDA 12.9	✅ megakernel NVFP4	↗
	Turing sm_75	RTX 2080 Ti	CUDA 12.0	✅ 53 tok/s DFlash	↗
	Ada sm_89	RTX 40xx	CUDA 12.0	🟡 community WSL2 bench	↗
—	Blackwell sm_110	Jetson AGX Thor	CUDA 13.0	🟡 builds, unbenched	—
	Volta sm_70 / Pascal sm_61	V100, P40	CUDA 12.0	🟡 fallback paths, unbenched	—
	RDNA3.5 gfx1151	Ryzen AI MAX+ 395 / Strix Halo	ROCm 6+	✅ 37 tok/s HIP	↗
	RDNA3 gfx1100	Radeon RX 7900 XTX	ROCm 6+	✅ 50 tok/s HIP	↗
—	RDNA4 gfx1201	Radeon AI PRO R9700	ROCm 6.4+	✅ 55 tok/s HIP	↗

server/ (DFlash) builds with CMake 3.18+ and vendors the required ggml sources directly; only Block-Sparse-Attention remains a git submodule. No PyTorch is needed for server/. optimizations/megakernel/ is the only component requiring PyTorch 2.0+ (CUDAExtension links against torch C++ libs). Power-tune: sudo nvidia-smi -pl 220 (3090 sweet spot, re-sweep for other cards).

Quick Start On Harnesses

harness/ contains RTX 3090 client launchers and regression tests for Lucebox server compatibility. Run Lucebox inside Claude Code, Codex, OpenCode, Hermes, Pi, OpenClaw, or Open WebUI, or check if a server change still works with those clients.

| Client | Launcher | |--------|----------| | Claude Code | [`run_claude_code.sh`](harness/clients/run_claude_code.sh) | | Codex | [`run_codex.sh`](harness/clients/run_codex.sh) | | OpenCode | [`run_opencode.sh`](harness/clients/run_opencode.sh) | | Hermes | [`run_hermes.sh`](harness/clients/run_hermes.sh) | | Pi | [`run_pi.sh`](harness/clients/run_pi.sh) | | OpenClaw | [`run_openclaw.sh`](harness/clients/run_openclaw.sh) | | Open WebUI | [`run_openwebui.sh`](harness/clients/run_openwebui.sh) |

All launchers spawn the native C++ HTTP server (dflash_server). Override defaults via env vars:

DFLASH_SERVER_BIN=server/build/dflash_server \
DFLASH_TARGET=server/models/Qwen3.6-27B-Q4_K_M.gguf \
DFLASH_DRAFT=server/models/draft/dflash-draft-3.6-q4_k_m.gguf \
MAX_CTX=32768 BUDGET=22 VERIFY_MODE=ddtree \
harness/clients/run_codex.sh

For no-draft targets such as Gemma, set only DFLASH_TARGET or pass DRAFT=none; the harness will not attach the default Qwen draft to a custom target.

Launcher scripts install missing real-client CLIs automatically under .harness-work/. To preinstall them yourself:

python3 harness/client_test_runner.py install --clients codex,hermes,openwebui

For direct TPS/TTFT numbers against a running server:

python3 harness/client_test_runner.py bench \
  --url http://127.0.0.1:8000 \
  --suite he,agent \
  --n-sample 3

Quick Start With Docker

Prebuilt images on GHCR track main. No CUDA toolkit or build needed. Pull the image, mount weights and serve. OpenAI-compatible API on :8000.

| GPU | Image tag | |-----|-----------| | NVIDIA (CUDA 12+) | `:cuda12` | | AMD (ROCm 6+) | `:rocm` | Drop a GGUF model target into `server/models/` first, then `:8000/v1/chat/completions`. Full tutorial in the [Docker blog](https://lucebox.com/blog/docker).

Install and run:

# 1. Pull the image for your GPU
docker pull ghcr.io/luce-org/lucebox-hub:cuda12   # NVIDIA
docker pull ghcr.io/luce-org/lucebox-hub:rocm     # AMD

# 2. Download a target model into server/models/ and the DFlash draft
#    into server/models/draft/ (the entrypoint only auto-discovers the
#    draft there; without it the server runs slower, target-only)
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf \
  --local-dir server/models/
hf download Lucebox/Qwen3.6-27B-DFlash-GGUF dflash-draft-3.6-q4_k_m.gguf \
  --local-dir server/models/draft/

# 3a. NVIDIA (CUDA 12+)
docker run --rm --gpus all -p 8000:8080 \
  -v "$PWD/server/models:/opt/lucebox-hub/server/models" \
  ghcr.io/luce-org/lucebox-hub:cuda12

# 3b. AMD (ROCm 6+, Strix Halo / RX 7900)
docker run --rm --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render --security-opt seccomp=unconfined \
  -p 8000:8080 -v "$PWD/server/models:/opt/lucebox-hub/server/models" \
  ghcr.io/luce-org/lucebox-hub:rocm

Then hit :8000/v1/chat/completions (OpenAI-compatible).

Run the Server

Default: Qwen 3.6-27B Q4_K_M target + Lucebox Q4_K_M DFlash drafter on RTX 3090. DDTree budget=22, TQ3_0 KV cache, full attention. OpenAI-compatible HTTP on :8000.

# build (CUDA 12+, CMake 3.18+)
git clone --recurse-submodules https://github.com/Luce-Org/lucebox-hub && cd lucebox-hub
cmake -B server/build -S server -DCMAKE_BUILD_TYPE=Release
cmake --build server/build --target dflash_server -j

# default weights (~18 GB)
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir server/models/
hf download Lucebox/Qwen3.6-27B-DFlash-GGUF dflash-draft-3.6-q4_k_m.gguf --local-dir server/models/draft/

# run (TQ3_0 KV auto-enabled; set =0 to disable)
DFLASH27B_KV_TQ3=1 \
./server/build/dflash_server server/models/Qwen3.6-27B-Q4_K_M.gguf \
  --draft server/models/draft/dflash-draft-3.6-q4_k_m.gguf \
  --ddtree --ddtree-budget 22 --port 8000

Making requests

For the fastest, deterministic responses send temperature: 0 (greedy decoding gives the highest spec-decode acceptance):

curl :8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "dflash",
  "messages": [{"role": "user", "content": "Write quicksort in Python."}],
  "temperature": 0
}'

Requests that omit temperature use the model card's sampling (Qwen3.6: temperature: 1.0, top_p: 0.95, top_k: 20).

Server flags

Core

Flag	Default	Effect
--draft <path>	—	DFlash draft GGUF, required for speculative decode
--port N	8000	HTTP port
--host H	127.0.0.1	Bind address
--max-ctx N	auto-fit	KV cache size; oversizing slows prefill (FA stride over unused KV)
--max-tokens N	model-card	Generation cap
--model-name S	filename	OpenAI model field
--chat-template-file <path>	autodetect	Override Jinja template

Decode (DFlash + DDTree)

Flag	Default	Effect
--ddtree	off (chain)	Enable tree verify
--ddtree-budget N	22	Tree size. 22 on 3090 (default), 40 on 5090, re-sweep on GB10
--fa-window N	0 / 2048 (full attention)	Sliding FA window. Leave at 0: a finite window breaks tool calls (the full-attention layers lose the system prompt/tools).
--draft-residency {auto,persistent,request-scoped}	auto	When draft weights are evicted from VRAM. request-scoped parks/frees them after each request's draft work (frees VRAM for the target on tight GPUs); persistent keeps them resident across requests; auto preserves current behavior while honoring the low-VRAM / --lazy-draft hint. Reported at /props.runtime.draft_residency.
--lazy-draft	off	Legacy alias for --draft-residency=request-scoped (defer draft load until first request, release after)

GPU draft top-K & verify-argmax (DFlash)

The draft-token top-K extraction and the per-step verify argmax used to run on the CPU, each requiring a full vocab × n_tokens logits copy from device to host (D2H) every speculation step. These two env flags move both onto the GPU, reading the logits in place on the device buffer and skipping the bulk D2H. Both are on by default in the server (the test_dflash harness defaults DFLASH_GPU_VERIFY_ARGMAX to off, see the table below) and take effect on both CUDA and HIP/ROCm builds: the draft top-K uses a custom device kernel (geometric_draft_topk_cuda.cu, the same source compiled directly for HIP) and the verify argmax reads an in-graph ggml_argmax node, so neither depends on a CUDA-only path. Each path validates its result and falls back to the legacy CPU computation automatically on any failure (e.g. an out-of-range index), so disabling them is only needed for debugging or A/B comparison.

Env	Default	Effect
DFLASH_GPU_DRAFT_TOPK=1	1 (on)	Compute the draft model's top-K vocab indices (K in 1–8) and log-sum-exp directly on the logits device buffer. =0 forces the legacy CPU top-K (full-voc