MCPcopy Index your code
hub / github.com/ericcurtin/inferrs

github.com/ericcurtin/inferrs @0.0.20260424164259

Chat with this repo
repository ↗ · DeepWiki ↗ · release 0.0.20260424164259 ↗ · + Follow
3,623 symbols 13,889 edges 238 files 894 documented · 25% ● updated 2mo ago0.0.20260424164259 · 2026-04-24★ 46015 open issues
What it actually does AI analysis from the code graph — generated when you open this
loading…
README

inferrs

A TurboQuant LLM inference server.

Why inferrs?

Most LLM serving stacks force a trade-off between features and resource usage. inferrs targets both:

inferrs vLLM llama.cpp
Language Rust Python/C++ C/C++
Streaming (SSE) ✓ ✓ ✓
KV cache management TurboQuant, Per-context alloc, PagedAttention PagedAttention Per-context alloc
Memory friendly ✓ — lightweight ✗ — claims most GPU memory ✓ — lightweight
Binary footprint Single binary Python environment + deps Single binary

Features

  • OpenAI-compatible API — /v1/completions, /v1/chat/completions, /v1/models, /health
  • Anthropic-compatible API — /v1/messages (streaming and non-streaming)
  • Ollama-compatible API — /api/generate, /api/chat, /api/tags, /api/ps, /api/show, /api/version
  • Hardware backends — CUDA, ROCm, Metal, Hexagon, OpenVino, MUSA, CANN, Vulkan and CPU

Quick start

Install

macOS / Linux

brew tap ericcurtin/inferrs
brew install inferrs

Windows

scoop bucket add inferrs https://github.com/ericcurtin/scoop-inferrs
scoop install inferrs

Run

inferrs run google/gemma-4-E2B-it

Serve

Serve a specific model vLLM-style

inferrs serve --paged-attention google/gemma-4-E2B-it

Serve a specific model llama.cpp-style

inferrs serve google/gemma-4-E2B-it

Serve models ollama-style

inferrs serve

Extension points exported contracts — how you extend this code

EncoderParam (Interface)
Helper functions to create the various objects on the compute command encoder on a single line. Prevents getting wrong s [10 …
inferrs-kernels/candle-metal-kernels/src/utils.rs
Module (Interface)
Defining a module with forward method using a single argument. [17 implementers]
candle-core/src/lib.rs
SimpleBackend (Interface)
(no doc) [14 implementers]
candle-nn/src/var_builder.rs
Benchmark (Interface)
(no doc) [9 implementers]
candle-nn/examples/cpu_benchmarks.rs
FromF32 (Interface)
Trait for types that can be produced from a dequantized f32 value. [3 implementers]
inferrs-models/src/turbo_quant.rs
TokenSender (Interface)
(no doc) [2 implementers]
inferrs/src/engine.rs
BenchDevice (Interface)
(no doc) [1 implementers]
candle-nn/benches/benchmarks/mod.rs
BenchDevice (Interface)
(no doc) [1 implementers]
candle-core/benches/benchmarks/mod.rs

Core symbols most depended-on inside this repo

map
called by 670
candle-core/src/cpu_backend/utils.rs
clone
called by 453
candle-nn/src/var_builder.rs
reshape
called by 323
candle-core/src/tensor.rs
len
called by 295
candle-core/src/strided_index.rs
len
called by 280
candle-nn/src/sequential.rs
to_dtype
called by 280
candle-core/src/tensor.rs
dims
called by 280
candle-core/src/shape.rs
add
called by 271
candle-core/src/tensor.rs

Shape

Method 1,592
Function 1,393
Class 480
Enum 92
Interface 62
Struct 4

Languages

Rust99%
Go1%

Modules by API surface

inferrs/src/server.rs174 symbols
candle-core/src/tensor.rs174 symbols
inferrs-kernels/candle-metal-kernels/src/tests.rs108 symbols
inferrs/src/tokenizer.rs102 symbols
inferrs-models/src/models/qwen3_5.rs89 symbols
candle-core/src/cpu_backend/mod.rs83 symbols
candle-core/src/cuda_backend/mod.rs82 symbols
inferrs-models/src/models/gemma4.rs70 symbols
candle-core/src/op.rs64 symbols
inferrs/src/engine.rs63 symbols
inferrs-models/src/models/mod.rs60 symbols
candle-core/src/dummy_cuda_backend.rs56 symbols

For agents

$ claude mcp add inferrs \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact

Ask about this repo answers extend the page