Bonsai Demo

Website  |  GitHub  |  Discord

HF Collections: Bonsai 27B · Bonsai (1-bit) · Ternary-Bonsai

Whitepapers: Bonsai 27B · 1-bit Bonsai 8B · Ternary-Bonsai 8B

---

Using this demo repository you can run Bonsai (1-bit) and Ternary-Bonsai language models locally on Mac (Metal), Linux/Windows (CUDA, Vulkan, ROCm), or CPU.

🌱 New: Bonsai 27B

The family's newest and largest generation, and its first vision-language models (Bonsai 27B collection):

• Vision: send photos, screenshots, and PDFs; ask about them (see VISION.md).
• Agentic tool calling: native OpenAI-style tool_calls with full round-trips, plus MCP servers in both demo UIs (see TOOLS.md).
• Thinking: a reasoning model; pick the reasoning effort per chat in the UI or budget it per request.
• Long context: 256k+ token conversations.
• Tiny footprint: the 1-bit Bonsai-27B packs to ~1.125 bits per weight: it fits on a modern iPhone without memory offloading. Ternary-Bonsai-27B (~1.7 bits per weight, packed into 2-bit for fast accelerated kernels) is the higher-quality option and this demo's default.

Quick Start below gets you there in two commands: ./setup.sh downloads Ternary-Bonsai-27B by default, then ./scripts/start_llama_server.sh gives you chat, vision, and tools at http://localhost:8080.

Quick Start

Setting things up with an AI coding agent? Point it at AGENTS.md, a guide written for agents (hardware-specific knobs, defaults, and what to ask the user).

macOS / Linux

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo

# (Optional) Choose a model size: 27B (default), 8B, 4B, or 1.7B
export BONSAI_MODEL=27B

# Set your HuggingFace token (only required for 27B while its repos are private)
export BONSAI_TOKEN="hf_your_token_here"

# One command does everything: installs deps, downloads models + binaries
./setup.sh

Windows (PowerShell)

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo

# (Optional) Choose a model size: 27B (default), 8B, 4B, or 1.7B
$env:BONSAI_MODEL = "27B"

# Set your HuggingFace token (only required for 27B while its repos are private)
$env:BONSAI_TOKEN = "hf_your_token_here"

# Run setup
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\setup.ps1

Switching families and sizes

You can switch between the Ternary (default) and 1-bit families, and different model sizes instantly:

# run Ternary-Bonsai 4B
BONSAI_FAMILY=ternary BONSAI_MODEL=4B ./scripts/download_models.sh
BONSAI_FAMILY=ternary BONSAI_MODEL=4B ./scripts/run_llama.sh -p "Hello!"

for Windows:

$env:BONSAI_FAMILY="ternary"; $env:BONSAI_MODEL="4B"
.\setup.ps1
.\scripts\run_llama.ps1 -p "Hello!"

---

Speed Benchmarks

See community-benchmarks/ for results on different hardware and templates to submit your own.

Models

Two model families are available, each in sizes 27B, 8B, 4B, and 1.7B. The 27B models are vision-language models: they accept images as well as text; all 27B repos are gathered in the Bonsai 27B HF collection.

Both formats are landing in mainline llama.cpp: Q1_0 (1-bit) is fully merged upstream, and Q2_0 (ternary) now runs on mainline CPU and Metal, with Vulkan in review. Details and mainline-compatible files: binary status and ternary status below.

Bonsai (1-bit)

Available in GGUF (llama.cpp) and MLX 1-bit formats.

Model	Format	HuggingFace Repo
Bonsai-27B	GGUF	prism-ml/Bonsai-27B-gguf
Bonsai-27B	MLX	prism-ml/Bonsai-27B-mlx-1bit
Bonsai-8B	GGUF	prism-ml/Bonsai-8B-gguf
Bonsai-8B	MLX	prism-ml/Bonsai-8B-mlx-1bit
Bonsai-4B	GGUF	prism-ml/Bonsai-4B-gguf
Bonsai-4B	MLX	prism-ml/Bonsai-4B-mlx-1bit
Bonsai-1.7B	GGUF	prism-ml/Bonsai-1.7B-gguf
Bonsai-1.7B	MLX	prism-ml/Bonsai-1.7B-mlx-1bit

Set BONSAI_MODEL to choose which size to download and run (default: 27B).

Ternary-Bonsai

Available in GGUF (llama.cpp) and MLX 2-bit formats.

Model	Format	HuggingFace Repo
Ternary-Bonsai-27B	GGUF	prism-ml/Ternary-Bonsai-27B-gguf
Ternary-Bonsai-27B	MLX (2-bit)	prism-ml/Ternary-Bonsai-27B-mlx-2bit
Ternary-Bonsai-8B	GGUF	prism-ml/Ternary-Bonsai-8B-gguf
Ternary-Bonsai-8B	MLX (2-bit)	prism-ml/Ternary-Bonsai-8B-mlx-2bit
Ternary-Bonsai-4B	GGUF	prism-ml/Ternary-Bonsai-4B-gguf
Ternary-Bonsai-4B	MLX (2-bit)	prism-ml/Ternary-Bonsai-4B-mlx-2bit
Ternary-Bonsai-1.7B	GGUF	prism-ml/Ternary-Bonsai-1.7B-gguf
Ternary-Bonsai-1.7B	MLX (2-bit)	prism-ml/Ternary-Bonsai-1.7B-mlx-2bit

This is the default family. Set BONSAI_FAMILY=bonsai to use the 1-bit Bonsai family instead.

Environment variables

Both variables are optional. If you set neither, the default is Ternary-Bonsai-27B: that's what plain ./setup.sh downloads and runs. They're read by setup.sh, setup.ps1, download_models.sh, and every run_* / start_* script (Linux, macOS, and Windows).

Variable	Default	Valid values	Purpose
BONSAI_FAMILY	ternary	ternary, bonsai, all	Model family. ternary = Ternary-Bonsai; bonsai = 1-bit Bonsai. all expands to both families (setup/download only).
BONSAI_MODEL	27B	27B, 8B, 4B, 1.7B, all	Model size. all expands to all four sizes (setup/download only).
BONSAI_TOKEN	—	HF read-only token	Only needed for the 27B models while their repos are private (removed at launch).

all is only valid for setup.sh / setup.ps1 / download_models.sh — the run/server scripts need a concrete family/size.

Combine them freely:

./setup.sh                                                  # Ternary-Bonsai-27B (default)
BONSAI_MODEL=1.7B ./setup.sh                                # Ternary-Bonsai-1.7B
BONSAI_FAMILY=bonsai ./setup.sh                             # Bonsai-27B (1-bit)
BONSAI_FAMILY=bonsai BONSAI_MODEL=4B ./setup.sh             # Bonsai-4B
BONSAI_MODEL=all ./setup.sh                                 # All 4 Ternary-Bonsai sizes
BONSAI_FAMILY=all BONSAI_MODEL=all ./setup.sh               # Full matrix (8 downloads)

Upstream Status for Binary

Q1_0 is supported out of the box in upstream llama.cpp across many backends: CPU (generic, NEON, and optimized x86), Metal, CUDA, and Vulkan.

Runtime	Status
llama.cpp (CPU, Metal, CUDA, Vulkan)	✅ Merged upstream, works out of the box
MLX (1-bit)	⏳ Pending upstream: mlx#3161; until it merges, use PrismML-Eng/mlx (branch prism, built automatically by setup.sh)

Upstream Status for Ternary

Ternary support is in the middle of migrating into mainline llama.cpp: backends are landing one by one, so today it is a mix of mainline and our fork. The practical consequence first: we currently ship three ternary GGUF variants, and each one needs to run in the right place.

File	Format	Runs on
*-Q2_0.gguf	Group size 128. The format this demo uses, compatible with our fork. Once the llama.cpp migration completes, these files will be deprecated and replaced by the PQ2_0 ggufs	This demo / the fork binaries. Will not load on mainline (same type id, different block size)
*-Q2_0_g64.gguf	Group size 64 (2.25 bpw). The official llama.cpp format; these will be renamed to plain Q2_0, replacing the current ones	Mainline llama.cpp (CPU and Metal so far)
*-PQ2_0.gguf	Not supported yet. Planned as the fork format going forward: the same format as the current group-128 Q2_0, just under its own ggml type id so it can coexist with the upstream Q2_0	Nothing yet (fork support planned)

Backend-by-backend migration status:

Backend	Status	Where
CPU (ARM NEON + generic scalar)	✅ Merged in mainline llama.cpp	ggml-org/llama.cpp#24448
Metal	✅ Merged in mainline llama.cpp	ggml-org/llama.cpp#25419
Vulkan	🔄 In progress upstream (separate PR, not ours)	ggml-org/llama.cpp#25430
CUDA	🔄 In review upstream	ggml-org/llama.cpp#25707
x86 (AVX-512-VNNI)	⏳ Pending	TBD

CPU and Metal now run Q2_0 on mainline llama.cpp, no fork needed (use a recent ggml-org/llama.cpp build with the *-Q2_0_g64.gguf files). For CUDA and the other backends, use this demo: it ships the fork pre-built binaries, so everything works out of the box with the group-128 *-Q2_0.gguf files it downloads. MLX 2-bit is supported in stock MLX, no fork needed.

To run the smaller ternary models directly on stock ggml-org/llama.cpp (CPU or Metal), use the group-64 files:

Model	Repo	File (mainline-compatible)
1.7B	prism-ml/Ternary-Bonsai-1.7B-gguf	Ternary-Bonsai-1.7B-Q2_0_g64.gguf
4B	prism-ml/Ternary-Bonsai-4B-gguf	Ternary-Bonsai-4B-Q2_0_g64.gguf
8B	prism-ml/Ternary-Bonsai-8B-gguf	Ternary-Bonsai-8B-Q2_0_g64.gguf

hf download prism-ml/Ternary-Bonsai-1.7B-gguf Ternary-Bonsai-1.7B-Q2_0_g64.gguf --local-dir models
hf download prism-ml/Ternary-Bonsai-4B-gguf  Ternary-Bonsai-4B-Q2_0_g64.gguf  --local-dir models
hf download prism-ml/Ternary-Bonsai-8B-gguf  Ternary-Bonsai-8B-Q2_0_g64.gguf  --local-dir models

What setup.sh Does

The setup script handles everything for you, even on a fresh machine:

1. Checks/installs system deps: Xcode CLT on macOS, build-essential on Linux
2. Installs uv: fast Python package manager (user-local, not global)
3. Creates a Python venv and runs uv sync — installs cmake, ninja, huggingface-cli from pyproject.toml
4. Downloads models from HuggingFace (needs BONSAI_TOKEN for 27B while its repos are private)
5. Downloads pre-built binaries from GitHub Release (or builds from source if you prefer)
6. Builds MLX from source (macOS only): clones our fork, builds it into the venv, installs the ML stack (mlx-lm, torch, transformers)
7. Installs Open WebUI into the venv for the agentic demo (skip with BONSAI_OPENWEBUI=0)
8. Builds the code-interpreter venv (.venv-jupyter): Jupyter + matplotlib / pandas / numpy / scipy / sympy / yfinance for the Open WebUI code interpreter (skip with BONSAI_CODE_INTERPRETER=0)

Re-running setup.sh is safe — it skips already-completed steps.

---

Running the Model

All run scripts respect `BONSAI_MO