Remove the last `count` input positions — the rollback used by the matcher to undo speculatively-consumed tokens. Advancing `count` bytes then `pop_la

Build a QuantizedWeight from the store using the standard NVFP4 naming. `weight_scale_2` is a single FP32 scalar — extracted from GPU via D2H copy.

Update the predictor-score record for `slot`.

Fused RMS norm + residual save: normed = rms_norm(input), residual = input. Eliminates a separate D2D copy by writing the raw input to the residual b

SSM projection output size: Q + K + V + Z concatenated.

`return_token_ids`: stamp the exact sampled token IDs onto this chunk's first choice. No-op when `ids` is empty, so requests that did not opt in keep

Add a `(cache_key -> rule_name)` mapping in the layer-correct cache. Port of `AddCache`.

Add a body-less rule (`body_expr_id = -1`). The body must be set later with [`GrammarBuilder::update_rule_body`]. Useful when the rule id is needed to

Apply the disk-ref obligation reported by a `prefix_cache.insert*` call. The cache returns the disk_block_ids on which it newly took an ownership ref

Forward pass: gate → top-K routing → batched expert FFN → blend. All expert dispatch stays on device — zero D2H synchronization. 9 kernel launches pe

Number of bytes a single group occupies on disk (== group_stride).

Whether the given expert ID is local to this EP rank.

Number of expressions.

Number of SSM (linear attention) layers.

Merge states linked by removable epsilon transitions. `a --eps--> b` is collapsible when either (1) `a` has no other outgoing edge, or (2) `b` has no

Compile the built-in JSON grammar (any valid JSON).

Dequantize FP8 E4M3 block-scaled weight → BF16, entirely on the GPU. Block-scaled FP8 (e.g. `quant_method: "fp8"` with `weight_block_size: [128, 128]

Detect the weight quantization variant from the weight store. Dispatch order matches vLLM / TRT-LLM / SGLang: 1. **Config-declared scheme** (`config.

Debug: read back FP32 GPU tensor and compute L2 norm + first 4 values. Used by the DeepSeek-V4 multi-seq decode diagnostic path (post/comb-attn hologr

Build a [`LexError`] at the current source position.

Whether this value is a JSON object.

Replay all operations captured in the graph on `stream`.

Append a new row from a slice; returns the new row's index.

Add `element` as its own singleton set. Returns `true` if inserted, `false` if it already existed.

Add a rule-reference edge for `rule_id`.

Drop every cached entry. Port of `RuleLevelCache::ClearCache`.

Compile a grammar for Hermes-format tool calls. Hermes format: `<tool_call>{"name":"fn","arguments":{...}}</tool_call>` Builds raw structural tag JS

Compile an already-optimized grammar against `tokenizer_info`. Port of `GrammarCompilerSub::MultiThreadCompileGrammar`. The grammar passed in MUST al

Compile a grammar for Qwen3-Coder XML tool calls. Format: `<tool_call>\n<function=name>\n<parameter=key>\nvalue\n</parameter>\n</function>\n</tool_ca

A deep copy.

Look up the cached mask for `key`, marking it most-recently-used on a hit. Port of `RuleLevelCache::GetCache`.

True iff the loader should override the inherited path.

Per-kernel launch geometry. Add a new arm when a rewritten kernel lands so the harness can A/B old vs new on identical inputs.

Load a BF16 tensor from an open safetensors file handle.

Fill the XGrammar bitmask for the current matcher position and clone it into an owned `Vec<i32>` the caller can pass into MTP draft sampling. Returns

Iterator over all weight names.

Extract a DevicePtr from the weight store by name.

Append `new_data` as a fresh row; returns its row index.

Build a QuantizedWeight from Sehyo/compressed-tensors NVFP4 naming convention. Sehyo quantization uses: weight_packed, weight_scale, weight_global_sc

Return slot indices in eviction-preference order (most-evictable first), excluding `pinned` slots.

Look up every kernel the per-layer forward needs. Fails loudly if any are missing — better to surface that at startup than silently mid-decode.

Decode a `TagDispatch` expression. Panics if `expr_id` is not a tag dispatch (matches the C++ `XGRAMMAR_DCHECK`).

Take the IDs buffered since the last emit when the request opted into `return_token_ids`; otherwise return empty (the builder no-ops on empty, so the

The vocabulary size. Port of the vendored `vocab_size`.

Accept a sampled token and advance the grammar state. Returns `true` if the token was accepted by the grammar. Returns `false` if the token violates

GPU-side argmax over BF16 logits. Finds the index of the maximum value, writes a single u32 to `out`. Kernel: `argmax_bf16(logits, out, n)` Grid: (1

Create a new conversation with optional initial items and metadata. Returns the generated id.

Load a weight and ensure it's FP32 on GPU, regardless of source dtype. Used for SSM gate parameters (A_log, dt_bias) where BF16 precision causes expo

Dequantize FP8 E4M3 + per-tensor scale → BF16, returning a DenseWeight. Allocates a new GPU buffer for the result. Use `dequant_fp8_to_bf16_into` to

Resolve a `__device__` symbol in a loaded PTX module to its device pointer + byte length. Required for drivers that read/write device globals without

Whether all auto-watchdogs are disabled at runtime. `false` by default; flipped only when `ATLAS_DISABLE_WATCHDOGS=1`/`true`.

Free a previously allocated block (decrements ref, frees if count hits 0).

Find an unused rule name starting from `name_hint`, appending an integer suffix (`_1`, `_2`, …) on collision. The first probed suffix is read from `n

FNV-1a-ish stable hash for the first `count` tokens — used to key snapshots independently of the radix tree (allows the same prefix hash to be reprodu

Run all registered hint injectors on a tool response. Returns the first matching hint (specific injectors first, generic last).

True if the root rule has completed at the current input position — i.e. the stop token may be emitted now.

True for the symmetric turbo dtypes whose cache contents are stored in the WHT-rotated basis (the write path applies `wht_bf16_inplace` before quantiz

Number of rows.

`mask` where token id `id` is a boundary iff `id` is in `boundary_ids`.