franken_ocr

A pure-Rust, memory-safe, CPU-only OCR engine that runs exactly one model, Baidu Unlimited-OCR, with no general ML framework, no Python, no CUDA, no FFI at inference, and no GPU. It parses document images into Markdown, JSON, or a versioned NDJSON event stream, on a single static binary that fits in about 5 MB.

Quick Install

curl -fsSL https://raw.githubusercontent.com/Dicklesworthstone/franken_ocr/main/install.sh | bash

The installer detects your platform, downloads the right prebuilt binary from the v0.2.0 release, verifies it by SHA256, and puts focr on your PATH. Then focr pull fetches the weights once and you run offline forever after.

TL;DR

The problem. Baidu Unlimited-OCR is a strong document-parsing model: Markdown, tables, LaTeX, reading order, many pages in one pass. The official stack is Python plus CUDA. Most machines that need OCR (laptops, CI runners, agent hosts, edge boxes) have no usable GPU, and a Python plus CUDA dependency is heavy to ship and awkward to embed.

The solution. franken_ocr is a library plus a single-binary CLI (focr) that runs this one model on CPU with nothing but a Rust binary. It transforms the bf16 checkpoint into a custom int8 format and runs it through kernels written for this model's exact shapes. On a real page measured against the Baidu reference, the end-to-end character-error-rate is 0.0094; the decode matched the reference to within a single token, and on one token it beat the reference. That is a measured result on the 6.67 GB model, not a target.

Why focr?

Quick Example

# 1. Fetch the int8 weights once (~3.9 GB) into the local cache, verified by SHA256.
focr pull

# 2. OCR a page into Markdown (the human default).
focr ocr page.png

# 3. Same page as structured JSON (markdown + every span's bounding boxes).
focr ocr page.png --json

# 4. Write the result to a file — format follows the extension (.md or .json).
focr ocr page.png -o page.md
focr ocr page.png -o page.json            # structured JSON with bounding boxes

# 5. Stream NDJSON pipeline events for an agent (run_start ... run_complete; full event set via `focr robot schema`).
focr ocr page.png --robot

# 6. Prove the int8 kernel on THIS CPU is bit-identical to the scalar oracle.
focr robot selftest

# 6. (optional) Convert your own bf16 safetensors into the int8 .focrq format.
focr convert model.safetensors -o unlimited-ocr.focrq --quant int8

After step 1 the weights live in ~/.cache/franken_ocr/models and every later command runs fully offline.

Design Philosophy

One model, every dimension fixed. A general ML framework pays a generality tax on every operation: dynamic dtype dispatch, arbitrary shapes, autograd bookkeeping, broadcast machinery, a device abstraction. franken_ocr runs exactly one model whose every dimension is known at compile time (hidden 1280, 10 heads, head_dim 128, 64 experts, top-6 routing, MoE intermediate 896, R-SWA window 128, vocab 129280). That buys shape-specialized kernels with no runtime shape branching in the hot loop.

Offline at inference. The only network step is focr pull, which runs ahead of time. There is no Python, no CUDA, no FFI, and no GPU in the inference path. The async runtime that orchestrates I/O and cancellation is an owned internal detail; the library API is synchronous and blocking, so there is no async plumbing to thread through your code.

Correctness before speed (always). A parity gate comes first and a faster kernel that drifts the OCR output is reverted, no source landed, and recorded in the negative-evidence ledger. Speed is shipped on top of parity, never instead of it. The int8 expert/FFN quantization is validated against the f32 path; the vision tower, projector, embeddings, MoE router, and all norms stay high precision.

Runtime ISA dispatch, one binary per arch. There is no per-CPU-feature variant to choose. One x86_64 binary covers AVX2 / AVX-VNNI / AVX-512-VNNI; one aarch64 binary covers NEON / SDOT / SMMLA. At load, CPU feature detection picks the fastest available int8 kernel tier. On a Threadripper 5995WX (a Zen 3 part whose ceiling is AVX2), dispatch correctly selects AVX2.

Bounded generated-token KV. Every decoder attention layer is R-SWA (Reference Sliding Window Attention). Each generated token attends to all reference tokens (visual plus prompt prefix, kept as a frozen, never-evicted global KV) plus only the previous 128 generated tokens through a ring-buffer KV cache. Generated-token KV memory stays constant instead of growing with output length. That, not arbitrary input resolution, is what "Unlimited" means.

How franken_ocr Compares

franken_ocr is the only one of these built for a single model on CPU.

When to use franken_ocr: - You need this model's output and your host has no usable GPU. - You want to embed OCR in a Rust program with no Python or FFI. - You want a single binary you can drop on a CI runner, an agent host, or an edge box.

When franken_ocr might not be ideal: - You need a model zoo or a generic inference runtime. franken_ocr runs exactly one model, by design. - You need the OmniDocBench leader; Unlimited-OCR is strong but not the top of the board.

Installation

Quick install (recommended)

curl -fsSL https://raw.githubusercontent.com/Dicklesworthstone/franken_ocr/main/install.sh | bash

The script detects your OS and CPU architecture, downloads the matching binary from the v0.2.0 release, verifies the SHA256 sidecar, and installs focr. Under WSL it proceeds as Linux. Under native Git-Bash, MSYS, or Cygwin it points you at the PowerShell installer below and exits cleanly.

On native Windows, install from PowerShell:

irm https://raw.githubusercontent.com/Dicklesworthstone/franken_ocr/main/install.ps1 | iex

This downloads the focr-x86_64-pc-windows-msvc.exe binary, verifies it by SHA256, and puts focr on your PATH.

Manual binary download

Release binaries are raw executables, not tar.gz archives. Each one is a single portable file that dispatches the ISA tier at runtime, so there is one binary per architecture (no per-CPU-feature variant). Sizes are roughly 4.7 to 5.9 MB. Linux binaries are gnu (glibc), not musl.

Each asset has a <asset>.sha256 sidecar in the standard "<hex> <asset>" format. Download the binary and its sidecar from the release base URL, verify, then install. Example for Apple Silicon:

base=https://github.com/Dicklesworthstone/franken_ocr/releases/download/v0.2.0
asset=focr-aarch64-apple-darwin-neon-sdot-i8mm

curl -fsSLO "$base/$asset"
curl -fsSLO "$base/$asset.sha256"

# macOS: shasum -a 256 -c   |   Linux: sha256sum -c
shasum -a 256 -c "$asset.sha256"

chmod +x "$asset"
mv "$asset" /usr/local/bin/focr

On Linux, swap the asset name and use sha256sum -c "$asset.sha256".

From source (advanced; not a one-liner)

franken_ocr requires the nightly Rust toolchain pinned in rust-toolchain.toml. cargo build --release builds both the focr and franken_ocr binaries from one shared entrypoint.

The catch: franken_ocr path-depends on sibling repositories that are not published on crates.io (asupersync, patched to /dp/asupersync; ../frankentorch; ../frankensqlite). A fresh-clone cargo build or a cargo install --git will fail to resolve those dependencies. There is no working cargo install from crates.io. Prebuilt binaries are the supported path; build from source only if you have those sibling repositories laid out as the workspace expects.

cargo build --release
# produces target/release/focr and target/release/franken_ocr (identical behavior)

Quick Start

1. Install the binary with the curl one-liner, or download and verify a release asset by hand.
2. Fetch the weights once: focr pull. This downloads about 3.9 GB of int8 weights plus tokenizer.json into ~/.cache/franken_ocr/models, verifying every byte by SHA256 against a committed manifest.
3. Verify your CPU kernel (optional but reassuring): focr robot selftest. Exit 0 means the dispatched int8 GEMM is bit-identical to the scalar oracle on this host.
4. OCR a page: focr ocr page.png for Markdown, add --json for structured output (markdown + bounding boxes), -o out.md / -o out.json to write a file, or --robot for an NDJSON event stream.
5. Batch many pages in one process (model loaded once): focr ocr-batch page1.png page2.png page3.png --json.

Command Reference

Both focr ocr and focr robot run accept the same image plus inference-tuning flags. The default crop mode is gundam (dynamic-resolution tiling); the alternative is base.

focr ocr <image>

Parse a document image into Markdown (default), JSON, or an NDJSON stream.

focr ocr page.png                          # Markdown to stdout
focr ocr page.png --json                   # structured JSON to stdout
focr ocr page.png -o page.md               # write Markdown to a file
focr ocr page.png -o page.json             # write structured JSON (markdown + boxes) to a file
focr ocr page.png --robot                  # NDJSON pipeline events
focr ocr page.png --crop-mode base         # disable dynamic-resolution tiling
focr ocr page.png --max-length 4096 --temperature 0.0
focr ocr page.png --model /path/to/unlimited-ocr.focrq

Output (-o/--output FILE). Writes the result to a file instead of stdout; the format follows the extension — .json emits structured JSON, any other extension (e.g. .md) emits Markdown. --json forces JSON regardless of extension. The structured JSON carries the rendered markdown plus a layout array — one {label, boxes} entry per grounded span, where each box is [x1, y1, x2, y2] in source-image pixels (a PDF nests these under a per-page pages array). This is the same shape --json prints to stdout.

Tuning flags: --base-size and --image-size (preprocessing resolution), --crop-mode (gundam or base), --max-length (decode token cap), --temperature (sampling), --no-repeat-ngram and --ngram-window (the sliding no-repeat n-gram decode guard).

focr ocr-batch <images...>

OCR many images in one process, loading the model once and reusing it across all pages.

```bash focr ocr-batch a.png b.png c.png --json # one load, many pages focr ocr-batch *.png --f