github.com/Hal0ai/hal0 @v0.9.3

Open-source home AI inference platform

hal0.dev · Install · Docs · Roadmap · Discord

hal0 turns a Linux box — ideally a Ryzen AI Max+ 395 with 128 GB of unified memory — into a polished, OpenAI-compatible inference appliance. Hardware-aware slots, prewired chat UI, signed self-update. One command installs the lot.

Every inference workload runs as its own podman container under a hal0-slot@<name>.service unit. hal0-api on :8080 is the sole control plane — it owns slot state machines, dispatches OpenAI-compatible /v1/* requests to the right slot port, and serves the dashboard. No shared inference daemon; no extra process to babysit.

curl -fsSL https://hal0.dev/install.sh | bash

Status: v0.9.0 — first public beta. Container-runtime era, declarative config. Each slot (agent, utility, embed, rerank, stt, tts, img, vision, NPU trio) runs as a dedicated podman container (hal0-slot@<name>.service). Slot definitions live in /etc/hal0/slots/<name>.toml; backend profiles in /etc/hal0/profiles.toml; the model catalog is registry.toml — the single source of truth for every HuggingFace coordinate and SHA-256 digest. The launch command for every slot is resolved from a single source (deduped, with per-flag provenance), Stacks apply declarative model/slot layouts atomically, and profiles share the same portable export/import envelope. hal0 is Strix Halo native, not Strix-Halo-only: experimental CUDA + multi-GPU pinning ride alongside the ROCm/Vulkan defaults. Companion services (Open WebUI, ComfyUI, Hermes, Hindsight) are managed from one Services dashboard page with mDNS discovery; the dashboard itself has a redesigned fixed-band layout with a live telemetry header. Seed profiles are virtual (code-defined, self-healing on upgrade) with dedicated embed/rerank lanes and a model×profile×slot MTP decision; models carry a preferred runtime profile, interrupted pulls resume, and voice runs end-to-end (NPU STT + Kokoro or GPU Qwen3-TTS). The one-liner writes the first-run sentinel only — no model picks, no downloads; run hal0 setup anytime to configure models, extensions, and NPU interactively. See PLAN.md §1 for what ships now and the path to v1.0.

Screenshots

Dashboard — unified-memory hero, at-a-glance health, and live slot rows.

Slots — the inference engine: per-slot models, devices, and live telemetry.

Model registry — catalogue with quant chips and pull status.	Image generation — ComfyUI queue and GPU gauges.	Memory — Hindsight facts and graph extraction.
Services — Open WebUI, ComfyUI, Hermes, Hindsight, n8n with mDNS.	Stacks — declarative slot + profile + model bundles, applied atomically.	Settings — full hal0.toml parity with an Advanced section.

_{More in the docs.}

Why hal0

Strix Halo native, but not Strix-Halo-only. The probe is UMA-aware on Strix Halo and falls back to portable parsers (/proc/cpuinfo, /proc/meminfo, lspci) on every other host. The platform field on /api/hardware resolves to one of strix-halo, wsl2, lxc, proxmox-kvm, kvm, bare-metal-{nvidia,amd,intel}-gpu, or bare-metal-cpu-only, and the dashboard only labels memory as "unified" when it actually is. The slot-fit warnings size against the real unified pool, not a BAR carve-out. The XDNA NPU is a first-class device — opt-in even at Pro/Max tier, packing three model roles into one process via the FLM trio (see below). 128 GB Ryzen AI Max+ 395 is the reference deployment — not a hopeful port.

One control plane, six modalities, dedicated containers. Each slot runs in isolation — chat, embed, rerank, STT, TTS, and image generation each get their own podman container with its own image, flags, port, and lifecycle. Concurrent workloads don't share a process pool; they share only the GPU through the arbiter. Slot state is persisted to /var/lib/hal0/slots/<name>/state.json and streamed via SSE. Chat + embed alone sustains ~258 tok/s on Strix Halo at the reference ROCm FP4 loadout.

NPU multi-role via the FLM trio. On Strix Halo with FastFlowLM installed, a single FLM process inside the hal0-toolbox-flm container hosts chat + transcription + embedding concurrently on the one AMDXDNA hardware context — ~2 GB NPU memory, gemma3:1b at 40 tok/s + Whisper-V3-Turbo + Embedding-Gemma all coresident. hal0 exposes this as three slots (npu, stt-npu, embed-npu); the npu.toml [npu] table toggles ASR and embed. The host-side FLM .deb is installed for device-sanity probes only — inference runs in the container. See ADR-0009 (FLM trio NPU packing) for why.

Dispatcher with single-flight + OmniRouter tool-calling. Registry-aware routing across local slots and external upstreams (OpenRouter, Anthropic, OpenAI, custom). The client-side OmniRouter loop provides 8 OpenAI tool-calling tools (image generation/editing, TTS, transcription, vision, embed, rerank, route_to_chat) dispatched through hal0 from any chat slot whose model carries the tool-calling label. The set is filtered per request — a tool only appears in the prompt if its target slot is enabled and its model carries the required labels.

Reliability bar. Atomic env file writes. Schema-validated TOML. Structured error envelopes ({"error":{"code":"slot.not_ready",...}}). Cosign-verified self-update with one-flag rollback. Per-type LRU concurrency with active-inference protection — a serving slot cannot be evicted out from under a streaming request.

What's in the box

• OpenAI-compatible /v1/* API — chat, completions, embeddings, rerank, audio transcriptions, audio speech, image generations, image edits, models. Drop-in for any OpenAI SDK; point your client at http://localhost:8080/v1 and go.
• Slots — each named target in capabilities.toml carries a type (llm | embedding | reranking | transcription | tts | image), a device (gpu-rocm | gpu-vulkan | cpu | npu | img), a model, plus enabled and optional default. Eight seeded slots (agent, utility, embed, rerank, stt, tts, img, vision) plus NPU slots (npu, stt-npu, embed-npu) when FastFlowLM is installed. User-added slots via hal0 slot create NAME --type TYPE --model MODEL. Slots refer to a profile in /etc/hal0/profiles.toml that pins the container image + flag bundle for that backend.
• Container runtime — each slot runs as its own podman container under hal0-slot@<name>.service. hal0-api on :8080 is the control plane; slot containers bind loopback ports (8081–8099 + fixed seeds). No shared inference daemon. See docs/concepts/architecture.mdx.
• hal0 setup TUI — after install (or anytime), hal0 setup walks you through storage, Extensions (Apps / Agents), Main and Agent model selection, and NPU opt-in, then downloads models with live progress. Non-interactive: hal0 setup --auto applies recommended defaults without prompting. Other flags: --storage-dir PATH, --no-pull (seed slots without downloading models), --no-extensions. Set HAL0_SKIP_SETUP=1 to skip first-run setup entirely during install.
• Hardware-aware probe — detects GPU / NPU / unified memory, writes /etc/hal0/hardware.json, surfaces VRAM/RAM fit warnings inline in the slot form and during hal0 setup.
• Dispatcher — registry-aware routing, cold-cache prefetch, upstream fallback (OpenRouter, Anthropic, OpenAI, custom OpenAI-shaped endpoints). Mix local + remote per-model in one config.
• Dashboard — React 18 + Vite UI for slot/model management, hardware-aware configuration, live logs, system health, and a built-in chat page (with popout window + reasoning toggle). SSE-backed status + log tail. Journal panel streams per-slot container logs via journald. Dark by default. Live telemetry — Power & Thermal (GPU clock/temp/power) and Per-Slot Throughput — is on by default, and the logs/events surface is unified across channels with real source/slot attribution. The slots page splits into Inference | Image Gen tabs: the Image-Gen tab operates the ComfyUI container (live GTT/RAM gauges, queue depth, model inventory) with a gated inference ⇄ generation iGPU switchover behind a blast-radius confirm.
• Extensions — selectable Apps (Open WebUI) and Agents (Hermes, Pi) that hal0 setup auto-wires into the platform.
• Companion-service management — /api/services is the declarative source of truth for OpenWebUI, Hermes, Hindsight, ComfyUI, and n8n: audit-logged systemd lifecycle actions (start/stop/restart/enable/ disable), mDNS .local advertisement, and a dedicated dashboard Services page.
• Stacks — declarative, runtime-switchable model/slot layouts as a replacement for install-time bundle tiers. A StackConfig is planned into a change set, applied atomically (with rollback) and converged against the live slot set; drift is detected by content hash. Stacks export/import via a checksummed .hal0stack.json envelope, and a live config can be snapshotted back into a Stack.
• OpenWebUI prewired — chat at :3001, zero config. The installer writes openwebui.env pointing at the local hal0 API.
• OmniRouter (8 tools) — generate_image, edit_image, text_to_speech, transcribe_audio, analyze_image (vision), embed_text, rerank_documents, route_to_chat. Dispatched client-side from chat slots; dynamically filtered per request.
• Image generation, day one — POST /v1/images/generations via ComfyUI in the img slot container (ROCm). hal0 setup lets you select image-gen as part of the initial configuration.
• Atomic self-update with rollback — hal0 update --channel stable|nightly. Cosign-verified tarballs swap a /usr/lib/hal0/current symlink; --rollback reverts.
• One-line install — curl -fsSL https://hal0.dev/install.sh | bash writes the first-run sentinel and wiring automatically — no model picks, no download, no interaction needed. Run hal0 setup afterward to choose models and configure apps and agents. (--models-dir=PATH or HAL0_MODELS_DIR=PATH redirects model pulls off /var/lib/hal0/models). The bootstrap fetches the release manifest, sha256-verifies the tarball, cosign-verifies the signature against the workflow OIDC identity, then hands off to installer/install.sh.
• One-line Proxmox VE install — on a Proxmox host, bash -c "$(curl -fsSL https://raw.githubusercontent.com/Hal0ai/hal0/main/scripts/proxmox-ve/hal0.sh)" creates an unprivileged Debian 13 LXC and runs the standard bootstrap inside it. --advanced opens whiptail prompts; every parameter has an env-var override (CTID, RAM_MB, STORAGE, …). Hardware-agnostic — Strix Halo passthrough still requires the privileged-LXC recipe. See scripts/proxmox-ve/README.md.

Bundled agents

hal0 ships two MCP servers and one bundled agent app. The MCP servers (/mcp/admin for slot / model / capability / config / hardware / log admin and /mcp/memory for Hindsight-backed long-term memory) are reachable by any MCP-speaking client — Claude Code, future RAG services, external scripts. The bundled agent is single-pick at install: pi-coder (CLI shape, installed from Hal0ai/pi-mono fork via @earendil-works/pi-coding-agent on npm) or Hermes-Agent (service shape, installed via the hal0-owned hermes wrapper — hal0-hermes is kept as a back-compat symlink; connects to hal0-api via HAL0_INFERENCE_BASE=http://127.0.0.1:8080). Select one during hal0 setup (Extensions step) or any time via hal0 agent install <name>; swap atomically with --switch. Capital-D destructive MCP calls (model_pull, slot_delete, config_write, etc.) gate through a header bell + inbox modal in th