InferCore

InferCore is an open-source AI Inference Control Plane: a decision layer that sits above model serving and data plane systems.

It provides intelligent routing, cost-aware decisions, fallback and degrade orchestration, AI-native SLO signals, multi-tenant policy enforcement, and observability / scaling-signal exports.

InferCore is not a model server. It does not run token generation and does not replace vLLM, Triton, Ray Serve, or KServe.

Website: infercore.dev — product overview, blog, and links to this repository.

Design goals

InferCore fills the missing system decision layer in typical serving stacks:

System boundaries

InferCore owns

• Request ingress and normalization
• Routing decisions
• Tenant and policy enforcement
• RAG orchestration (optional): retrieval + rerank after routing, then merge into the model payload (request_type: rag; file-backed knowledge bases in v1.5)
• Fallback and reliability orchestration
• Cost estimation and budget-based routing
• AI SLO signal generation
• Metrics / traces / events export

InferCore does not own

• Model inference execution
• CUDA/GPU kernel optimization
• Training and MLOps pipelines
• Executing autoscaling (it can export signals for HPA/KEDA/custom autoscalers)
• Advanced dashboards and long-term analytics stores

Architecture

Client / SDK
    ↓
[ InferCore Gateway ]
    ↓
[ Request Normalizer ]
    ↓
[ Policy Engine ] ← tenant / budget / priority / guardrails
    ↓
┌──────────────────────────────────────────────────────────────────────────┐
│ Reliability · overload admission                                         │
│   • reliability.overload.queue_limit + action: reject (503) | degrade    │
│   • Degrade: skip cost optimization, annotate response; router sees      │
│     overload state (queue depth / degrade) for routing hints             │
└──────────────────────────────────────────────────────────────────────────┘
    ↓
[ Router Engine ] ← route rules / health-aware backend choice / cost-aware pick
    ↓
┌──────────────────────────────────────────────────────────────────────────┐
│ Optional RAG (request_type=rag, when configured)                         │
│   retrieve (KB) → rerank (e.g. noop) → merge into input.*                │
└──────────────────────────────────────────────────────────────────────────┘
    ↓
┌──────────────────────────────────────────────────────────────────────────┐
│ Reliability · execution & fallback                                       │
│   • ExecuteWithFallback: primary backend then fallback_rules chain       │
│   • Triggers: timeout, backend_unhealthy, backend_error                  │
│   • Per-backend timeout_ms; bounded by gateway request_timeout_ms        │
│   • Streaming: stream_fallback_enabled when options.stream               │
└──────────────────────────────────────────────────────────────────────────┘
    ↓
[ Execution Adapter Layer ] ← OpenAI-compatible (e.g. OpenAI, vLLM)  / Gemini / Mock / …
    ↓
Inference Backends

Parallel outputs:
- Metrics exporter (Prometheus text on /metrics)
- Trace / event exporters (configurable telemetry backends)
- In-memory AI SLO engine (bounded store; snapshots on responses)
- Scaling signals (/status.scaling_signals + infercore_scaling_* gauges)

Request lifecycle

1. Client sends an AI request to InferCore (POST /infer) — same endpoint for inference, RAG, and agent (preview).
2. Gateway parses tenant, task type, priority, and payload.
3. Policy engine evaluates quota, budget, priority, and guardrails.
4. Overload admission (reliability.overload) may reject or degrade before routing.
5. Router selects a backend using rules, health, overload state, and optional cost optimization.
6. If request_type is rag and knowledge_bases are configured, InferCore runs retrieve then rerank, and merges retrieved text into the payload (e.g. input.retrieved_context) before the model call.
7. Execution adapter invokes the selected backend.
8. On timeout or classified failure, reliability rules may trigger fallback or degrade behavior.
9. InferCore records TTFT/TPOT/latency/fallback (where available) and exports metrics, events, and traces as configured.

Differentiators

1. Cost-aware routing — Not only “can this run?” but “should this use the expensive model?” within budget and compatibility constraints.
2. AI-native SLO — TTFT, TPOT, completion latency, fallback markers; rolling hints exposed for operators (/status, Prometheus).
3. Reliability orchestration — Per-backend timeouts, configurable fallback chains, overload reject/degrade, health-aware routing.
4. Multi-tenant policy — Tenant classes, priorities, per-request budget estimates, per-tenant RPS windows (in-memory per replica).
5. Scaling signals — Queue depth, timeout-spike hints, TTFT degradation ratio, recent fallback rate, optional max_concurrency hints from config — intended for HPA/KEDA or custom autoscalers (per-replica unless you add shared state).

Repository layout

This repository includes:

• YAML configuration examples and OpenAPI draft (api/openapi.yaml)
• Go implementation of the control plane (policy, routing, reliability, RAG retrieval, adapters); infer pipeline orchestration in internal/inferexec
• Documentation under docs/ (architecture, observability, load testing, streaming, retrieval adapters, offline tooling, KB registry roadmap)

v1.0.0: AI Request model, ledger, RAG, CLI

• AI Request — Optional fields on POST /infer: request_type (inference | rag | agent), pipeline_ref (defaults: inference/basic:v1, rag/basic:v1, agent/basic:v0), and context (RAG: query, knowledge_base; agent: tool hints for policy).
• Request ledger — Enable with ledger.enabled and driver: memory, sqlite (path), or postgres (dsn). Persists full normalized AIRequest as ai_request_json (plus input/context JSON), policy_snapshot after routing, and per-step records for audit and replay.
• RAG (supported) — Set request_type: rag and list knowledge_bases with type: file (local demo), http (JSON microservice), opensearch / elasticsearch, or meilisearch. See docs/retrieval-adapters.md. Execution order: policy → overload admission → route → retrieve → rerank (rag.rerank.type, default noop) → chat completion. User text via input.text and/or context.query; chunks merge into input.retrieved_context.
• Agent (request model only; no runtime) — request_type: agent is accepted for agent-ready ingress and policy (tenant limits such as max_steps, max_tool_calls, allowed_tools when features.agent_enabled is true). There is no tool loop or agent executor in this release: the gateway returns 501 agent_not_implemented. Do not describe InferCore as a full “agent platform” yet.
• CLI (same binary):

infercore serve                    # HTTP gateway (default when no subcommand)
infercore trace <request_id>       # dump ledger request + steps JSON
infercore replay <request_id> --mode exact|current   # single response JSON
infercore replay id1 id2 --mode current              # batch: NDJSON lines (or --ids-file)
infercore ledger export-eval <request_id>... -o items.json   # eval dataset from ai_request_json
infercore eval run --dataset examples/queries.json --pipeline inference/basic:v1
# With server API key auth (or env INFERCORE_API_KEY):
infercore eval run --dataset examples/queries.json --api-key "$INFERCORE_API_KEY"

Replay and ledger commands use the CLI and internal/replay; the HTTP gateway does not expose a replay API (see docs/offline-tooling.md).

Replay semantics (infercore replay)

• exact — Uses the policy snapshot in the ledger to force the primary backend and disables fallback. RAG requests still re-run retrieve + rerank; if knowledge-base files or retrieval config changed, results may differ from the original online call. Routing does not use live health probes in this mode.
• current — Re-evaluates policy and router. The replay harness treats all backends as healthy (see internal/replay); production /infer uses cached health probes for routing, so routes can differ from a current replay on the same machine.

Telemetry

When telemetry.tracing_enabled is true, OTLP/log exporters receive infer_request (whole handler) and per-step infer_step spans (step_type, backend, result).

Configuration checklists (minimal vs production)

Use this as a rough guide—not every deployment fits two buckets. Items reflect keys in configs/infercore.example.yaml.

Minimal viable config (conceptually): one backend + routing.default_backend pointing to it + valid YAML for telemetry exporter and reliability overload action. Everything else is layered behavior.

Quickstart

Prerequisites

• Go 1.22+

Run

go run ./cmd/infercore

The service starts on :8080.

Use a custom config file:

INFERCORE_CONFIG=./configs/infercore.example.yaml go run ./cmd/infercore

Secrets in YAML: placeholders like ${OPENAI_API_KEY} are expanded at load time from the process environment (os.Getenv). Export the variable before starting, or prefix the command: OPENAI_API_KEY=sk-... infercore serve --config=.... Unset variables become empty strings. Only ${VAR} is supported (ASCII identifier after VAR).

Make targets

make help     # list targets
make all      # fmt, vet, test
make test
make build    # writes bin/infercore
make run      # CONFIG=... optional (default configs/infercore.example.yaml)

Docker

make docker-build   # image tag: infercore:local
docker run --rm -p 8080:8080 infercore:local

Mount your own config:

docker run --rm -p 8080:8080 \
  -v "$(pwd)/configs/my.yaml:/app/config.yaml:ro" \
  -e INFERCORE_CONFIG=/app/config.yaml \
  -e OPENAI_API_KEY=... \
  infercore:local

(If the YAML uses ${OPENAI_API_KEY}, pass it with -e so expansion works inside the container. Otherwise, skip -e OPENAI_API_KEY=... )

CI

GitHub Actions workflow (on push/pull_request to main or master): .github/workflows/ci.yml runs go vet and go test ./....

Try APIs

Health:

curl -s http://localhost:8080/health

Status:

curl -s http://localhost:8080/status

Infer (plain inference):

curl -s -X POST http://localhost:8080/infer \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "team-a",
    "task_type": "simple",
    "priority": "normal",
    "input": {"text": "Summarize this article"},
    "options": {"stream": false, "max_tokens": 256}
  }'

RAG (requires knowledge_bases in config, e.g. pointing at examples/kb):

```bash curl -s -X POST http://localhost:8080/infer \