live-bench

A LAN-only live benchmarking platform for streaming local-model evaluation results onto a presentation-grade dashboard. Built to answer one question on camera: how close are local models to replacing the cloud?

Watch the video — M5 Max vs M4 Max, Qwen 3.5 / 3.6 vs Gemma 4, GGUF vs MLX, with every chart in this repo running live.

Why this exists

Anthropic's API went down mid-recording for the launch video. That's the whole pitch: when your agent stack depends on a single provider, every outage is your outage. The way out is private, cheap, fast, performant models running on your own hardware — but you only know when "the time has arrived" if you've been measuring along the way.

This repo is the harness that does the measuring. M4 Max and M5 Max sit side-by-side, both pushing datapoints into the same dashboard over the LAN, so the bars fill in live as each model finishes a prompt. Five takeaways from the bake-off:

• MLX wins on Apple Silicon, full stop. Qwen 3.5 MLX hits ~118 tok/s vs ~50 tok/s on GGUF — almost 2× the decode speed for the same model. If you're on a Mac and you're not on MLX, you're leaving a free 2× on the table.
• M5 Max is 15–50% faster than M4 Max on wall-clock, with the largest gap on prefill — exactly the metric that dominates as prompts get longer.
• Wall-clock is the only number that matters. Tokens/sec is a vanity metric; the time you actually sit and wait end-to-end is what determines whether a model is usable inside an agent.
• Context is the real ceiling, not intelligence. Both Qwen 3.5 and Gemma 4 give correct GraphWalks answers up to 16K, but wall-clock falls off a cliff: 30+ seconds at 16K, unusable above 32K. Advertised context windows are not usable context windows on local hardware.
• Local models can already do agentic work — up to ~8–16K tokens of context. The viable pattern today is micro-agents and sub-agent processes that hand narrow tasks to a fast local model, not running the whole stack locally.

The platform itself is model-agnostic — anything that emits (group, metric, value) tuples can drive a chart. YAML files in benchmarks/ define every chart; a tiny live-bench CLI POSTs datapoints from any host on the LAN; a Vue 3 frontend animates the bars in real time over SSE. Three apps in apps/{backend,cli,frontend}, one SQLite file, no external services.

Prerequisites

Optional:

Skip the manual checklist — run /install from Claude Code in this repo and it walks every prerequisite, dependency, and optional model pull interactively.

Quick start

60-second start

Local-only, no models required. Copy-paste in order:

git clone <this repo> && cd live-bench
just install-frontend          # bun install for the Vue frontend
just install-cli               # `live-bench` on your PATH (uv tool, editable)
just mock                      # boots backend + frontend, opens fake demo data

just mock boots both services, prints the dashboard URL (http://127.0.0.1:5787/), waits 8 seconds, then streams fake benchmark data with animated bars and clickable prompt/response modals. Open the URL it prints.

Full platform (real or fake data)

just serve   # boots backend (:8787) + frontend (:5787) together via mprocs

Open http://127.0.0.1:5787/, pick a benchmark from the landing dropdown, then in another terminal emit a point against the built-in mock session:

just emit mock-test q1 model_a 142.7
just complete mock-test q1

Discover what's available with just list (every session) and just show <session_id> (config + current values for that session). Without mprocs, run just backend and just frontend in two terminals instead.

Cross-device (LAN)

Both services bind 0.0.0.0 so the dashboard and ingest API are reachable from any device on the network. The CLI reads LIVE_BENCH_SERVER to know where to POST datapoints — default http://127.0.0.1:8787 works for same-machine; for LAN you point benchmark machines at the dashboard host:

$ just lan-ip
LAN IP:    192.168.1.42
Dashboard: http://192.168.1.42:5787
Backend:   http://192.168.1.42:8787

On the benchmark machine, set:
  LIVE_BENCH_SERVER=http://192.168.1.42:8787

Three ways to point the CLI at a remote dashboard:

# 1. Persist via .env (cp .env.sample .env, then edit)
LIVE_BENCH_SERVER=http://192.168.1.42:8787

# 2. Inline, one-off
LIVE_BENCH_SERVER=http://192.168.1.42:8787 just bench-qg3-smoke-m5

# 3. Per-call --server flag
live-bench --server http://192.168.1.42:8787 emit ...

Data flow: benchmark script → live-bench emit → CLI POSTs over HTTP → backend stores in SQLite + assigns seq + broadcasts via SSE → frontend's session store does newer-wins on client_ts → animated bar transitions over 1s.

Ports

Both recipes free their port before booting, so re-running never fails with EADDRINUSE. just ports shows what's holding each port; just free-ports clears both.

Project structure

live-bench/
├── apps/
│   ├── backend/                — FastAPI + sqlite3 + sse-starlette (uv)
│   │   └── src/live_bench_server/
│   │       ├── main.py         — app factory, lifespan, YAML reconcile
│   │       ├── http_ingest.py  — POST /ingest
│   │       ├── sse_stream.py   — GET /stream/:sessionId
│   │       ├── rest.py         — /benchmarks, /reset, /healthz
│   │       ├── store.py        — repository over SQLite
│   │       ├── broker.py       — in-proc pub/sub fanout
│   │       └── models.py       — Pydantic schemas
│   ├── cli/                    — `live-bench` Click app (uv tool)
│   │   └── src/live_bench_cli/commands/   — emit, stream, complete, reset, ...
│   ├── frontend/               — Vue 3 + Pinia + Zod + hand-rolled SVG charts
│   │   └── src/
│   │       ├── pages/          — LandingPage.vue, BenchmarkPage.vue
│   │       ├── components/     — GroupedBarChart, AnimatedBar, NavBar, ...
│   │       ├── stores/         — session.ts (newer-wins on client_ts)
│   │       └── styles/         — themes.css (10 themes, obsidian default)
│   └── benchmarks/             — uv single-file benchmark scripts
│       ├── mock-test/          — fake data for UI smoke tests
│       ├── qwen35-vs-gemma4/   — 4-way raw inference (GGUF + MLX)
│       ├── qwen36-vs-qwen35-vs-gemma4/  — 6-way raw inference (3-way models)
│       ├── context-scaling{,-3way}/     — GraphWalks 200 → 32K tokens
│       └── pi-coding-agent{,-3way}/     — agentic coding tasks
├── benchmarks/                 — YAML chart definitions (loaded on backend boot)
├── specs/                      — design docs, including init_live_bench.md
├── .claude/commands/           — /install, /prime, /build, /commit-push, /plan-w-team
├── images/hero.svg             — README hero (architecture overview)
├── CLAUDE.md                   — project rules for AI agents
├── justfile                    — every recipe documented below
└── .env.sample                 — LIVE_BENCH_SERVER template

CLI commands

Every command is wrapped by a just recipe. just with no args lists everything.

Reading benchmarks

Writing/updating benchmarks (any session)

⚠️ reset is destructive across the whole session — every chart, every hardware tag (*_m4, *_m5), every metric. See CLAUDE.md for the rule that keeps real-inference recipes additive.

Live benchmarks

All benchmark recipes are additive (M4 and M5 emit to the same session over the LAN). Each pair of *-m4 / *-m5 recipes runs on its respective Mac. Smoke variants run a single prompt for sanity-checking before kicking off a long run. Reset recipes wipe one session at a time.

qwen35-vs-gemma4 — 4-way raw inference

Qwen 3.5 vs Gemma 4 across GGUF and MLX. 6 charts: prefill / decode / wall × M4 / M5.

qwen36-vs-qwen35-vs-gemma4 — 6-way raw inference

Adds the new Qwen 3.6 35B-A3B family. 8 charts: prefill / decode / wall / memory × M4 / M5.

Prereq on each benchmark machine:

ollama pull qwen3.6:35b-a3b
ollama pull qwen3.6:35b-a3b-nvfp4

context-scaling-3way — GraphWalks context scaling

Qwen 3.6 MLX vs Qwen 3.5 MLX vs Gemma 4 MLX across 200 → 32K tokens. 8 charts: prefill / decode / wall / accuracy × M4 / M5.

pi-coding-agent-3way — agentic coding

5 models × 6 coding tasks. 8 charts: wall / score / tokens / tools × M4 / M5.

Combined runners

Architecture & design docs

• specs/init_live_bench.md — the source-of-truth spec: data model, wire protocol, animation state machine, theming, transport trade-offs.
• **[specs/qwen36-trio-benchmarks.md](specs/qwen36