github.com/Geekgineer/needle-rs @v0.1.0

<a href="https://needle-rs.pages.dev"><b>→ Live demo</b></a>







<a href="https://github.com/Geekgineer/needle-rs/actions/workflows/ci.yml"><img src="https://github.com/Geekgineer/needle-rs/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI"/></a>
<a href="https://github.com/Geekgineer/needle-rs/actions/workflows/release.yml"><img src="https://github.com/Geekgineer/needle-rs/actions/workflows/release.yml/badge.svg" alt="Release"/></a>
<a href="https://github.com/Geekgineer/needle-rs/actions/workflows/wasm-demo.yml"><img src="https://github.com/Geekgineer/needle-rs/actions/workflows/wasm-demo.yml/badge.svg?branch=main" alt="Demo"/></a>
<a href="#parity"><img src="https://img.shields.io/badge/parity-560%2F560%20token--exact-brightgreen?style=flat-square" alt="Parity 560/560"/></a>
<a href="https://crates.io/crates/needle-rs"><img src="https://img.shields.io/crates/v/needle-rs?style=flat-square&color=CE422B" alt="crates.io"/></a>
<a href="https://www.npmjs.com/package/needle-rs"><img src="https://img.shields.io/npm/v/needle-rs?style=flat-square&color=CE422B" alt="npm"/></a>
<a href="https://pypi.org/project/needle-rs/"><img src="https://img.shields.io/pypi/v/needle-rs?style=flat-square&color=CE422B" alt="PyPI"/></a>
<a href="https://github.com/Geekgineer/needle-rs/raw/v0.1.0/LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" alt="MIT"/></a>







<a href="#quick-start">Quick start</a> &nbsp;·&nbsp;
<a href="#how-it-works">How it works</a> &nbsp;·&nbsp;
<a href="#parity">Parity</a> &nbsp;·&nbsp;
<a href="https://github.com/cactus-compute/needle">Upstream model</a>

A pure-Rust + WebAssembly runtime for Needle by Cactus Compute — a 26M-parameter transformer that maps (query, tool list) → JSON call in one forward pass. Deploys to browsers, edge workers, CLIs, Python, and no_std embedded targets. No server, no API key.

Tool calling usually means either a paid API call or hundreds of megabytes on disk. needle-rs ships the whole agent in 23 MB.

Same answer to "did the user ask for a flight booking?" — at a fraction of the footprint.

npm install needle-rs

import init, { NeedleWasm } from "needle-rs";

await init();
const engine = NeedleWasm.load(weights, vocab);

engine.run(
  "Book a flight from London to JFK tomorrow",
  toolsJson
);
// → {"name":"book_flight","arguments":{...}}

cargo add needle-infer

use needle_infer::NeedleEngine;

let engine = NeedleEngine::load(
    "needle.safetensors",
    "vocab.txt"
)?;
let result = engine.run(query, tools_json);
println!("{}", result.text);

pip install needle-rs

from needle_rs import NeedleEngine

engine = NeedleEngine.load(
    "needle.safetensors",
    "vocab.txt",
)
result = engine.run(query, tools_json)
print(result)
# → [{"name":"book_flight","arguments":{...}}]

Get the weights

huggingface-cli download Abdalrahman/needle-rs-safetensors \
  needle.safetensors vocab.txt --local-dir weights/

Or load directly from a URL in the browser — no install step.

Cactus's official engine targets mobile and NPUs with hand-tuned ARM SIMD. needle-rs targets everywhere else. The two stacks are complementary.

Needle is a 26M-parameter encoder-decoder transformer with a small twist: it's trained to do exactly one thing — emit a function-call JSON object from a query and a tool list. That focus is why a model this small works at all.

Architecture deep-dive: ARCHITECTURE.md.

A common failure mode for from-scratch model reimplementations is silent drift — outputs that look right but diverge at the third decimal place, with rare and untraceable downstream bugs. needle-rs rejects that. The Rust engine is required to produce the exact same token ID sequence as the Python/JAX reference on every input, at every decode step.

The test suite generates 560 inference examples by running the Python model on a diverse input set: five tool-name conventions _{(snake_case, camelCase, PascalCase, UPPER_SNAKE_CASE, kebab-case)}, parameter counts from 0 to 8, tool arrays from 1 to 20 entries, and a spread of natural-language query phrasings. For each example we capture the Python model's complete output token sequence. The Rust engine is then run on every example and required to produce the identical sequence.

560 / 560 pass. Not approximately — same argmax decision at every step.

Token-exact parity is checked on every CI run. Any change that drifts gets caught before merge. The reference vectors are committed to the repo, so the parity contract is version-pinned and reproducible without re-running Python:

• Reference generator: tools/gen_e2e_vectors.py
• Reference data: tests/e2e_vectors.json
• Rust parity test: crates/needle-infer/tests/e2e_parity.rs

Plus 55 unit tests on the constrained decoder covering edge cases the parity suite doesn't reach _{(empty tool arrays, parameter-less tools, name-collision under snake_case normalization, max-length queries)}.

JavaScript / TypeScript

engine.run(query, tools)                              // → string
engine.run_stream(query, tools, (id, piece) => {})    // per-token callback → final string
engine.run_batch([{ query, tools }, ...])             // → string[]
engine.encode_contrastive(text)                       // → Float32Array | null
engine.retrieve_tools(query, descriptionsJson, topK)  // semantic tool routing

Rust

engine.run(query, tools_json);
engine.run_stream(query, tools_json, |_id, piece| print!("{piece}"));
engine.run_batch(&[(q1, t1), (q2, t2)]);
engine.encode_contrastive(text);            // → Option<Vec<f32>>
engine.retrieve_tools(query, descs, k);     // → Vec<(usize, f32)>

C (and anything with FFI)

#include "needle.h"

NeedleHandle h  = needle_load("needle.safetensors", "vocab.txt");
const char *out = needle_run(h, query, tools_json);
printf("%s\n", out);
needle_free_str((char *)out);
needle_free(h);

Full header: crates/needle-c/include/needle.h. Null-safe throughout; errors via thread-local needle_last_error().

Intel i7-1185G7 (Tiger Lake, 4-core), Linux, release build, median of 5 runs.

Apple Silicon NEON path is implemented but unbenchmarked — M-series numbers welcome via PR.

Footprint, stripped release:

Full methodology and raw numbers: BENCHMARKS.md.