github.com/Luce-Org/lucebox-hub @main

Speculative inference for heterogeneous machines and consumer GPUs.

Custom kernels, speculative prefill and decoding, tuned for each model and hardware target.

Inference Engine Optimizations

Supported Models and Drafters

Model links open the exact weights used by the measured setup. Drafter links open the published quant, or the source checkpoint when conversion is required.

Tested Machines (GPU/APU)

The engine is not tied to one reference card. NVIDIA architectures are selected by CMake; HIP builds should target the device's exact gfx architecture.

	Architecture	Hardware	Runtime	Details
	RDNA4 gfx1201	Radeon AI PRO R9700	ROCm 7.2	Qwen 3.8 R9700 quick start
	RDNA3.5 gfx1151	Ryzen AI MAX+ 395 / Strix Halo	ROCm 7.2	DeepSeek V4 Strix profile
	RDNA3 gfx1100	Radeon RX 7900 XT / XTX	ROCm 6+	DeepSeek V4 dual AMD profile
	Ampere sm_86	RTX 3090	CUDA 12+	Qwen 3.8 NVLink result and Megakernel results
	Blackwell sm_120	RTX 5090	CUDA 12.8+	Qwen 3.8 single-GPU result
	Blackwell sm_121	DGX Spark / GB10	CUDA 12.9	Qwen 3.5 NVFP4 results
	Ada sm_89	RTX 4090	CUDA 12+	Linux and WSL2 community runs
	Turing sm_75	RTX 2080 Ti	CUDA 12.0	DFlash results
	Volta sm_70, Pascal sm_61	V100, P40	CUDA 12.0	CUDA quick start
Not pictured	Blackwell sm_110	Jetson AGX Thor	CUDA 13.0	Thor quick start

Single-device results

Heterogeneous and parallel results

These runs use different prompts, quantizations, and inference policies. They show which configurations work; they are not a cross-hardware ranking.

Recommended Setups

See Recommended server setups for the model and hardware matrix, including single-GPU and mixed-GPU profiles.

Client Harnesses

harness/ runs Lucebox through popular coding clients and checks server compatibility.

| Client | Launcher | |--------|----------| | Claude Code | [`run_claude_code.sh`](harness/clients/run_claude_code.sh) | | Codex | [`run_codex.sh`](harness/clients/run_codex.sh) | | OpenCode | [`run_opencode.sh`](harness/clients/run_opencode.sh) | | Hermes | [`run_hermes.sh`](harness/clients/run_hermes.sh) | | Pi | [`run_pi.sh`](harness/clients/run_pi.sh) | | OpenClaw | [`run_openclaw.sh`](harness/clients/run_openclaw.sh) | | Open WebUI | [`run_openwebui.sh`](harness/clients/run_openwebui.sh) |

Set the server binary and model paths, then run a launcher:

DFLASH_SERVER_BIN=server/build/dflash_server \
DFLASH_TARGET=server/models/Qwen3.8-27B-UD-IQ4_XS.gguf \
DFLASH_DRAFT=server/models/draft/qwen38-dflash2-q8_0.gguf \
MAX_CTX=32768 \
harness/clients/run_codex.sh

See the harness guide for setup, no-draft targets, and benchmarks.

Quick Start With Docker

Prebuilt images on GHCR track main. Mount the weights and serve the OpenAI-compatible API on :8000.

| GPU | Image tag | |-----|-----------| | NVIDIA (CUDA 12+) | `:cuda12` | | AMD (ROCm 6+) | `:rocm` | Put the target in `server/models/` and its matching drafter in `server/models/draft/`.

Run the image for your GPU:

# NVIDIA
docker run --rm --gpus all -p 8000:8080 \
  -v "$PWD/server/models:/opt/lucebox-hub/server/models" \
  ghcr.io/luce-org/lucebox-hub:cuda12

# AMD
docker run --rm --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render --security-opt seccomp=unconfined \
  -p 8000:8080 -v "$PWD/server/models:/opt/lucebox-hub/server/models" \
  ghcr.io/luce-org/lucebox-hub:rocm

Run the Server

This quick start runs the R9700 profile above. The complete flag reference is in the server guide.

# build (ROCm 7.2+, RDNA4)
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
cd lucebox
cmake -S server -B server/build-hip -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
  -DDFLASH27B_GPU_BACKEND=hip \
  -DDFLASH27B_HIP_ARCHITECTURES=gfx1201 \
  -DGGML_HIP_MMQ_MFMA=ON \
  -DGGML_HIP_NO_VMM=ON
cmake --build server/build-hip --target dflash_server -j"$(nproc)"

# target and DFlash2 drafter
mkdir -p models
huggingface-cli download unsloth/Qwen3.8-27B-GGUF \
  Qwen3.8-27B-UD-IQ4_XS.gguf --local-dir models
huggingface-cli download incoai/Qwen3.8-27B-DFlash2 --local-dir models/dflash2
python server/scripts/convert_dflash_to_gguf.py \
  models/dflash2/model.safetensors models/qwen38-dflash2-f16.gguf
python server/scripts/quantize_dflash_draft.py \
  models/qwen38-dflash2-f16.gguf models/qwen38-dflash2-q8_0.gguf --scheme q8_0

# launch the measured profile
./server/build-hip/dflash_server models/Qwen3.8-27B-UD-IQ4_XS.gguf \
  --draft models/qwen38-dflash2-q8_0.gguf \
  --draft-block-size 16 --max-ctx 131072 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --port 8216

curl -s http://127.0.0.1:8216/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Write a Python LRU cache."}],
       "max_tokens":256,"temperature":0}'

Documentation