franken_whisper

Agent-first Rust ASR stack with a real in-process pure-Rust Whisper engine (no FFI, no Python, no subprocess — and on CPU it beats whisper.cpp 2.33× on tiny.en, at parity on large-v3-turbo), adaptive Bayesian backend routing, real-time NDJSON streaming, DTW word timestamps, and SQLite-backed run history.

Install in one line

curl -fsSL "https://raw.githubusercontent.com/Dicklesworthstone/franken_whisper/main/install.sh?$(date +%s)" | bash

_{SHA-256-verified prebuilt binaries for Linux (x86_64 / aarch64), macOS (Intel / Apple Silicon), and WSL — proxy-aware, airgap-capable (--offline), reversible (--uninstall). Windows users: grab windows_amd64.zip from the latest release. All flags: Installation.}

v0.2.0 — the native engine is real, and it is fast. The in-process pure-Rust Whisper engine (built on FrankenTorch kernels, #![forbid(unsafe_code)] in-crate) now transcribes 2.33× faster than whisper.cpp on tiny.en and at parity on large-v3-turbo — CPU vs CPU, same machine (Apple M4 Pro, interleaved runs), while using about 18% less user CPU on the large model (53.8 s vs 65.4 s). Native-vs-whisper.cpp conformance on the reference fixture: WER 0.0000.

Model	franken_whisper native (CPU)	whisper.cpp (CPU)
tiny.en (11 s clip)	475 ms	1,105 ms	2.33× faster
large-v3-turbo (11 s clip)	9.73 s	9.59 s	parity

The Problem

Speech-to-text pipelines are fragmented. To get production-quality transcription you need whisper.cpp for speed, insanely-fast-whisper for GPU batching, and whisper-diarization for speaker identification. Each has its own CLI, output format, error handling, and deployment story. Orchestrating them from scripts means parsing inconsistent stdout, handling timeouts manually, killing zombie subprocesses, and losing all run history when something crashes.

Agent workflows make the problem worse. Modern LLM agents need structured, streaming, machine-readable output, not human-oriented terminal decorations that break the moment they touch jq, pipes, or SSH.

The Solution

franken_whisper is a single Rust binary that wraps every major Whisper backend behind a unified, agent-first interface — and ships its own engine:

• A real in-process Whisper engine, in pure Rust. ggml model parsing, log-mel frontend, encoder/decoder transformer inference on FrankenTorch CPU kernels, greedy decoding with whisper.cpp's full timestamp-rule suite, and cross-attention DTW word timestamps. No FFI, no Python, no subprocess — drop a ggml model file in place and transcribe.
• Adaptive Bayesian backend routing. Each auto request runs a formal decision contract with an explicit loss matrix, per-backend Beta posteriors, Brier-scored calibration, and deterministic fallback when the model is mis-calibrated.
• Real-time NDJSON streaming. Every pipeline stage emits sequenced, timestamped events on stable schema v1.0.0. No fragile regex; agents parse JSON.
• Durable run history. Every transcription persists to SQLite with full event logs, replay envelopes, and JSONL export/import, even when the process crashes mid-run.
• Cooperative cancellation. Ctrl+C propagates through the whole pipeline via cancellation tokens. Subprocesses get killed, transactions roll back via savepoints, finalizers run within a bounded budget, exit code 130.
• Zero-dependency audio decode. MP3, AAC, FLAC, WAV, OGG, Vorbis, and ALAC decode natively via symphonia. ffmpeg is a fallback for video and exotic codecs, and is auto-provisioned on Linux x86_64 if missing.
• Native engine rollout governance. In-process Rust replacements for the bridge adapters ship behind a 5-stage rollout (Shadow → Validated → Fallback → Primary → Sole) with conformance gating at every promotion.
• TTY audio transport. Compressed audio (mu-law + zlib + base64) over PTY links with handshake, integrity hashes, deterministic retransmission, and an adaptive bitrate controller. Transcript-streaming control frames (protocol v2) carry speculation events end-to-end over the same link.
• Word-level timestamps. First-class support via whisper.cpp's word-timestamp pipeline, with the cancellation token threaded into the inner extraction loop.

Why franken_whisper?

Quick Example

# Transcribe an audio file. MP3 / FLAC / OGG / AAC decoded natively, no ffmpeg needed.
franken_whisper transcribe --input meeting.mp3 --json

# Transcribe a video file. Audio extracted automatically via ffmpeg fallback.
franken_whisper transcribe --input presentation.mp4 --json

# Stream real-time pipeline events for agents
franken_whisper robot run --input meeting.mp3 --backend auto

# Speculative streaming: fast partial transcripts with quality-model corrections
franken_whisper robot run --input meeting.mp3 --speculative \
  --fast-model tiny.en --quality-model large-v3

# Speaker diarization with pyannote
franken_whisper transcribe --input meeting.mp3 --diarize --hf-token "$HF_TOKEN" --json

# TinyDiarize: whisper.cpp's built-in speaker-turn detection (no HF token needed)
franken_whisper transcribe --input meeting.mp3 --tiny-diarize --json

# Discover available backends and their capabilities
franken_whisper robot backends

# System health check (backends, ffmpeg, database, resources)
franken_whisper robot health

# Query run history
franken_whisper runs --limit 10 --format json

# Export run history to a portable JSONL snapshot (full or incremental)
franken_whisper sync export-jsonl --output ./snapshot

# Compressed audio over a text-only channel (PTY, SSH, serial)
franken_whisper tty-audio encode --input audio.wav > frames.ndjson
cat frames.ndjson | franken_whisper tty-audio decode --output restored.wav

Design Philosophy

Agent-First, Human-Optional

The primary interface is robot. Every command in robot mode emits sequenced, timestamped NDJSON on stdout with schema version 1.0.0. Human-friendly output (transcribe without --json) is the exception. Robot mode never mixes decorative stderr into the data stream; structured run_error envelopes replace human prose even for argument-parsing failures.

Deterministic by Default

Given identical inputs and parameters, franken_whisper produces identical outputs. The TTY retransmit loop, replay envelopes, replay packs, and conformance harness all enforce determinism. Random elements (UUIDs, wall-clock timestamps) are quarantined to metadata fields; they never enter computational output.

Fail Loud, Recover Gracefully

Every error has a structured code (FW-IO, FW-CMD-TIMEOUT, FW-BACKEND-UNAVAILABLE, FW-STAGE-TIMEOUT, …) and propagates through the NDJSON event stream. Cancellation tokens let in-flight work checkpoint and clean up instead of dying mid-write. In-progress SQLite transactions roll back via savepoints; subprocesses get SIGKILL; temp directories are removed by registered finalizers under a bounded budget.

Composition Over Configuration

The pipeline is composed dynamically per request. The 10 canonical stages (Ingest, Normalize, VAD, Source Separate, Backend, Accelerate, Align, Punctuate, Diarize, Persist) are skipped when unnecessary, budgeted independently, and profiled automatically. The PipelineBuilder validates ordering at build time; runtime never hits "stage X requires stage Y" errors.

No Unsafe Code

The entire crate uses #![forbid(unsafe_code)]. forbid is stricter than deny: it cannot be overridden per item. Memory safety is enforced by the compiler, not by code review.

Zero External Dependencies for Common Audio

franken_whisper decodes MP3, AAC, FLAC, WAV, OGG (Vorbis), and ALAC entirely in-process via symphonia; no ffmpeg, no Python, no PATH lookup. ffmpeg is only invoked as a fallback for video files, exotic codecs (AC3, DTS, Opus-in-MKV), and live microphone capture. When ffmpeg is needed and missing on Linux x86_64, it is downloaded once into the per-user state directory.

Adaptive Controllers With Conservative Fallbacks

Every adaptive controller in the system (the backend router, the speculative window controller, the budget tuner, the TTY adaptive bitrate controller, the correction tracker) ships with an explicit alien-artifact engineering contract: state space, action space, loss matrix, posterior, calibration metric, deterministic fallback, and an evidence ledger. When models drift, the system falls back to a deterministic policy rather than confidently making bad decisions.

The Whisper Ecosystem Landscape

          +--------------------------------------------------------------+
          |            INFERENCE ENGINES (run the model)                 |
          |                                                              |
          | whisper.cpp           (C++, CPU/Metal/CUDA, ~47k stars)      |
          | faster-whisper        (Python/CTranslate2, ~14k stars)       |
          | OpenAI Whisper        (Python/PyTorch, ~95k stars)           |
          +--------------------------------------------------------------+
                                         |
          +------------------------------v-------------------------------+
          |     ENHANCED PIPELINES (add features on top)                 |
          |                                                              |
          | WhisperX               (faster-whisper + wav2vec2 + pyannote)|
          | whisper-diarization    (Whisper + Demucs + NeMo TitaNet)     |
          | insanely-fast-whisper  (HuggingFace Transformers, max GPU)   |
          | whisper-timestamped    (DTW word timestamps)                 |
          +--------------------------------------------------------------+
                                         |
          +------------------------------v-------------------------------+
          | ORCHESTRATION (manage engines/pipelines)                     |
          |                                                              |
          | > franken_whisper <    (Rust, Bayesian routing,              |
          |                         10-stage pipeline, conformance       |
          |                         gating, native engine rollout,       |
          |                         evidence-based decisions)            |
          +--------------------------------------------------------------+

Most tools occupy one level. franken_whisper is the orchestration layer: it wraps inference engines and enhanced pipelines behind a unified interface, then adds capabilities none of them provide individually.

How franken_whisper Compares

Orchestration & Architecture

Persistence & Observability

| Capability | whisper.cpp | faster-whisper | WhisperX | WhisperLive | **franken_whisper