Merge states with identical incoming or outgoing transition structure (`ab | ac | ad` -> `a(b|c|d)`, and the mirror case). Upstream renamed this `Mer

Open an existing layout (panics if a file is missing or undersized).

Consume a token of `ty`, or report `message`.

Load a quantized weight using the appropriate naming convention. For `Fp8Dequanted`, requires `quant_ctx` (absmax_k, quantize_k, stream) to runtime-q

Render the complete grammar to EBNF text — one rule per line.

Roll back the matcher by `n` accepted tokens / strings. Each prior `accept_token`/`accept_string` is one unit; a stop token is a zero-byte unit whose

Apply the router pre-normalization (Gemma-4 only) and return the pointer that should be fed into the gate GEMV. If the MoE has no router_pre_norm weig

Subset-construction NFA -> DFA. Returns `Err` if the result would exceed `max_num_states`.

───────────────────────── upload helpers ─────────────────────────

Visit an expression by id.

Read the per-model watchdog tunables. Returns the historical-default `WatchdogParams` until `set_watchdog_params` runs — so unit tests and any pre-boo

POST to /v1/chat/completions. Returns (http_status, parsed_body).

Add a rule with a body, deriving a unique name from `name_hint`.

Append `element` to the result, inserting a separating space when the result is non-empty — port of `AddEBNFSegment`.

Advance a set of states — see [`Fsm::advance`].

Reassembled arguments for `idx`: concatenated live fragments, else the buffered `ToolCallDelta`, else the complete `ToolCall`'s arguments.

`current_ == end_`: at or past the terminator.

Whether XML-style emission applies at the current nesting depth (root or first level).

Index into `edge_aux_data`, or `-1` if not a repeat reference.

Batched embedding: gather N rows from embedding table in one launch. Replaces N individual D2D copies with a single kernel. `token_ids_dev` must poin

Cache stride in elements (for FP8/BF16 kernels that need explicit stride). Same for all layers (element count is dtype-independent).

Compile an EBNF grammar against [`tok`].

── memory ────────────────────────────────────────────────────────────

Dense BF16 GEMM: C = A @ B^T. A: [M, K] row-major (activations) B: [N, K] row-major (weights, HuggingFace layout) C: [M, N] row-major (output) Kerne

Dequantize an NVFP4 weight to BF16 on CPU, then upload to GPU. Used at load time when projections are NVFP4-quantized on disk but need BF16 format fo

Phase 6.3 — Sliding-window allocation helper (decode path). Ensures `seq.physical_block_for(abs_block_idx)` is `Some` after this call returns. With H

Bulk broadcast: send an array of u32 tokens from rank 0 to all ranks. Uses a single NCCL broadcast instead of per-token broadcasts. Per-token broadca

The associated optimized grammar.

Resolve a content token's id by its decoded bytes.

Load one MoE expert projection, dispatching by the on-disk format so the V4 loader handles every DeepSeek-V4-Flash checkpoint variant. The nvidia chec

Resolve the first existing tensor among `candidates`, returning an F32 device pointer (BF16 tensors are widened). Errors if none exist or the element

Sequence the four Q/K/V/O loads via a loader-supplied closure. The closure receives `(name, full_out, full_in, kind)` and returns the loader's represe

Snapshot `n_elements` BF16 values starting at `ptr + byte_offset` to `<ATLAS_GDN_DUMP>/gdnsub_step0_L{layer_idx}_{stage}.bin`. Skips when `ATLAS_GDN_D

Mix several values into `seed` (incremental `HashCombine`).

Batched sigmoid blend: output += sigmoid(dot(normed, gate_weight)) * shared_out. Grid: (num_tokens, 1, 1) Block: (256, 1, 1)

The next contextual separator, advancing the `is_first` state. Port of `NextSeparator`.

First transition target, or [`NO_NEXT_STATE`].

F76 (2026-04-29): pre-parse `tool_calls[*].function.arguments` from OpenAI's wire format (JSON-encoded string) into the JSON value the model's chat te

Decode `expr_id` into an [`OwnedExpr`], pre-decoding tag dispatch.

Parse a BNF/EBNF grammar string into a [`GrammarData`]. `root_rule_name` names the start rule (xgrammar's default is `"root"`). Returns a [`ParseErro

Pre-dequant NVFP4 → FP8 E4M3 for zero-overhead prefill GEMMs. Reads B_packed[N, K/2] + B_scale[N, K/GROUP_SIZE] + scale2 and produces B_fp8[N, K] on

Referenced rule id, or `-1` if this is not a rule reference.

Sample a token from logits on the GPU. `logits_ptr` points to `[vocab_size]` BF16 values on device. Reads BF16, expands to FP32, then samples with fu

Register a snapshot slot as belonging to a session.

Scale in-place embeddings by config.embed_scale. The residual stream is always BF16, so this dispatches `embed_scale::bf16_scale_inplace`.

FP32 [gate, beta] for GDN (num_v_heads * 2 floats).

Turbo2 data section bytes (2-bit packed: 4 values per byte).

Turbo8 data section bytes (FP8 E4M3: 1 byte per element).

Merge the sets containing `a` and `b` (union by size). # Panics Panics if either element is absent.

Insert `mask` under `key`. Returns `true` if it was stored, `false` if rejected (already present, or larger than the whole budget). Port of `RuleLevel

Add a `TagDispatch` expr. Tag strings and stop/exclude string lists are interned as `ByteString`/`Choices` sub-exprs first, matching the CSR layout de

Pure stop-string accumulator + hold-back algorithm. Returns the bytes that should be forwarded to the client this delta; the remainder (≤ `buffer_len`

Build EP routing table from flattened gate indices and weights. Given the top-K expert assignments for M tokens, partitions them into local vs remote

Build a precompiled minijinja Environment from the chat template. Leaks the template string to 'static — acceptable since one ChatTokenizer lives for

Check the NCCL communicator for asynchronous errors. Returns `true` if the communicator is healthy (no async errors). If an async error is detected,

Reject every token (clear all bits).

F62 (2026-04-27): commit a verify pass to the canonical SSM state. `num_accepted ∈ [0, k]`: full accept → copy `h_state` → checkpoint; partial → copy

Compile the builtin "any JSON value" grammar. Port of `GrammarCompiler::CompileBuiltinJSONGrammar`.

DEBUG (env-gated): PER-LAYER fingerprint of h_state + conv_state for a pool slot, used to prove restore/recompute state divergence. States are FP32 (`

Load a dense weight, converting FP32 → BF16 if needed. Used for norm weights that may be FP32 in some checkpoints (e.g. Qwen FP8 `linear_attn.norm.wei

Compute effective attention scale: override if set, else `1/sqrt(head_dim)`.

Epsilon closure (in place); `state_set` is *not* cleared.

Find citations in `content` and return an annotation for each reference site. Returns an empty vec when nothing matched.

Scan `content` for http(s) URLs and emit OpenAI-compatible `url_citation` annotations. The extractor handles three shapes: - Markdown links `[title](

GPU matmul: `C[M,N] = A[M,K] × B[K,N]` using dense_gemm_bf16 kernel. Allocate GPU memory, falling back to managed (UVM) if device alloc fails. Uses a

Get fixed-address intermediate h_state for K=2/3/4 verify. `token_idx` is 0..3 (which token in the verify pass).