Record a match. `winner` ∈ {"a", "b", "draw"}. Returns the new (rating_a, rating_b).

Trust-on-first-use record. Returns the (possibly pre-existing) pin. If the repo was already pinned, returns the existing pin without overwrit

Register a dataset by name.

Adapt a preset to the run it is about to be used for. Returns a *new* config plus human-readable notes describing every change, so the caller

Resolve ``module.path:fn_name`` into a callable. Lazy-imports the named module via :mod:`importlib`, fetches the referenced attribute, and ve

Return ``trl.<name>``, falling back to its ``trl.experimental`` home. The public namespace is tried FIRST, so any ``trl`` that still exports the

Score one model ``output`` against a mini-benchmark ``answer``. MCQ letter answers are scored by extraction + exact letter match; every other

POST ``payload`` to PostHog if telemetry is enabled, else no-op. Returns ``True`` on a 2xx response, ``False`` on any failure or skip. NEVER

Sign a payload (in-toto JSON bytes) with the chosen backend. ``ed25519`` (v0.71.2 #179) produces a real detached signature over ``payload`` u

Shape AND dtype for ONE decoder layer, read from the shard header. The dtype is read per tensor rather than taken from ``index.dtype``: an

Return ``(sft_delta, wall_clock_secs)`` for an N-step LoRA probe. When ``model`` is supplied (and torch + peft + the model load succeed) this

Canonicalise a conversion direction against the closed allowlist.

Validate and normalise a GRPO variant name. Returns the canonical (lower-cased) name on success. Raises ``ValueError`` with an actionable mes

Raise ValueError if ``name`` is invalid for registry use.

Normalise + validate an RA-DIT stage name. Returns the canonical (lowercase) form. Mirrors v0.51.0 ``validate_hub_name`` / v0.61.0 ``validate

Validate and normalise a rollout backend name.

Validate and normalise a ULD strategy name. Returns the canonical (lower-cased) name. Raises ``ValueError`` on any failure.

Apply the ``--replay*`` flags, then RE-VALIDATE. Re-validation is the point: a CLI override must clear the same cross-validators as YAML, or

Resolve the branches directory with env-var override + containment.

Pure tensor kernel: divergence between student and teacher logits. Both logits are ``(batch, seq, vocab)``. Temperature softens the distribut

Helper to create a JSONL file with sample data.

Read base_model_name_or_path from adapter_config.json.

Mirror a vision processor's nested-tokenizer token surface onto itself. HF vision processors (Idefics3/SmolVLM, LLaVA, Qwen2-VL, ...) keep the te

Print a friendly red error and raise ``typer.Exit(code)``.

Valid JSON that parses to an object or array. A bare scalar (``"42"``, ``"true"``) is valid JSON yet is not the structured object/array the f

Create a mock model with given config attributes.

Helper: return an EmbeddingTrainerWrapper with trainer pre-injected.

Helper: return a PretrainTrainerWrapper with trainer pre-injected.

Split a ``judge_model`` URL into ``(provider, model, api_base)``. Examples: ``ollama://llama3.1`` -> ("ollama", "llama3.1", None) ``h

Return ``(bf16, fp16)`` flags for TrainingArguments. - When ``tcfg.auto_mixed_precision`` is True: query GPU compute capability and

Run a single training with modified parameters.

Write a two-tensor LoRA adapter, optionally through torch.compile's prefix. Module-level so #351's checkpoint tests below build the file the same

Write a synthetic PEFT LoRA adapter; return its arrays.

Write a 2-row JSONL eval task file.

Fold per-step scalar scores into a single reward. ``mode`` is one of ``AGGREGATE_MODES``. Empty ``scores`` → 0.0. Non-finite per-step values

Align student token positions to teacher token positions. Returns ``list[list[int]]`` — for each student position, the (sorted) list of teach

Resolve ``(model, base_model)`` after an optional non-HF hub prefetch. v0.53.10 #152 — shared CLI helper for chat / serve / infer / merge / e

Compute the per-batch loss tensor for a GRPO variant (v0.53.11 #123). This is the live kernel that lifts the v0.50.0 ``NotImplementedError``

Atomically write ``data`` (bytes) to ``output_path`` under cwd containment. Binary sibling of :func:`atomic_write_text` (v0.71.3 #181 — used by t

Corpus-level BLEU (single reference per prediction). Returns score in [0.0, 1.0]. Empty corpus returns 0.0. `smooth=True` (default) uses Che

Build a ``(B, 1, S, S)`` additive attention mask from segment IDs. Args: seq_pos_ids: integer array of shape ``(B, S)`` where entry

Render chat messages into a prompt string (#332). THE single prompt builder shared by ``soup serve``'s transformers and vLLM backends, so the

Validate inputs and return a frozen ``CmaesPlan``. - ``adapters`` must contain ≥2 unique paths, all under cwd. - ``eval_suite`` must be a rea

The prompt actually sent to the model for one mini-benchmark item. Adds ``_MCQ_ANSWER_CUE`` for multiple-choice items and returns ``question``

Build a low-level MCP :class:`Server` that dispatches to ``specs``. Each tool result is a plain ``dict`` from the handler; it is sanitized (C

Decide whether to proceed with another iteration. The decision composes three checks in this order: 1. Run-cap (daily) — fast rejection when

OK / MINOR / MAJOR — the single source of truth for the verdict. The verdict asks whether MORE canaries look memorized than chance alone expl

OK if rate <= 1%, MINOR if <= 5%, MAJOR otherwise. Boundary semantics chosen so the boundary value lands in the MORE SEVERE bucket (i.e. exac

Map ``score in [0, 1]`` to OK / MINOR / MAJOR verdict.

Compare the suite's scorer mix to the taxonomy's recommended set. The task category is validated against v0.54.0 :data:`TASK_CATEGORIES` so t

Compute Elo ratings from all judgments.

Mean pairwise variance across a small RM ensemble. Input is a sequence of per-RM score lists; every inner list must be the same length (one s

Count URL domains across ``rows`` and return the top-N by share. Returns a tuple of ``(domain, share)`` where ``share`` is the domain's fract

Fit per-item Rasch difficulty from a flat list of {item_id, correct}. The closed-form estimate when ability is centred at 0 is ``β̂_i = -log(

Return the process-wide tool-output buffer.

Load and validate the manifest from a ``.can`` file. Refuses paths outside the current working directory so an ``inspect`` invocation cannot

v0.53.6 #105 / v0.53.7 — instantiate live upstream callbacks. Validates ``names`` against :func:`validate_trainer_plugin_list`, then lazy-imp

Read a design JSON back into :class:`EvalDesign`. Cwd-contained.

Resolve a template name to its YAML body. Returns None if unknown. Order: 1. YAML file in this directory (per ``manifest.json``). 2.

Stable string key for the cache. Hashed for filesystem safety.

Signed task-vector combine over the intersection of tensor names. Per adapter ``i`` and tensor ``k`` the effective factor coefficient is :fun

Run a single mSPRT decision step. Returns ``MsprtVerdict`` with one of: - ``continue``: keep collecting samples - ``reject_h0``: differen

Parse a budget string (``60s`` / ``5m`` / ``2h`` / bare seconds).

Parse a spec dict with optional explicit ``kind`` override.

Pick the fastest candidate whose score >= min_score. Tie-break: first-encountered (stable, matches v0.28.0 kernel_picker). Raises ValueError

Post ``body`` as a comment on the GitHub PR named by ``target``. Uses ``gh api`` (no PyGithub dependency) with the body sent over JSON stdin

``{'max_prompt_length': N}`` iff this trl's config still accepts it. Returns an empty dict on a trl that removed the field, which is the whole

Read a JSONL of {prompt, output} rows, strip shared prefix, write. Returns a :class:`PrunePromptReport` summarising the pass. Output JSONL co

Read the last ``limit`` audit records (newest last).

Record a new checkpoint eval.

Render up to ``n`` rows with per-token trained/masked colouring. Routes to the REAL collator path per format: RAFT rows go through ``utils.ra

Render the Modal app stub for ``config_yaml`` (v0.71.18 #16). ``config_yaml`` is base64-embedded as data (no interpolation). ``gpu`` is valid

Resolve the block **count** from exactly one of ratio / explicit count. ``drop_layers = round(drop_ratio * num_layers)`` when a ratio is given.

Return the active endpoint for ``hub`` after env-override + validation. Looks up the per-hub env var (e.g. ``MODELSCOPE_ENDPOINT``) — if set, run

The store/compute dtype for a streamed run, chosen from the CARD. Until v0.73.x this was the literal ``"bfloat16" if on_cuda else "float32"``

Entry point with friendly error handling.

Run the live Llama-4 expert delinearization described by ``plan``. Live since v0.71.21 (#97). ``num_experts`` defaults to the value read from

Run the full chat-template compat report over a sample of ``raw_rows``. ``train_on_responses_only``/``train_on_messages_with_train_field`` pick

Run every task in ``suite`` and return an aggregate verdict. ``generate_fn`` accepts a prompt and returns the model output. Injecting this ke

Score forgetting from per-task base + adapter accuracies. ``tolerance`` is the per-task accuracy drop considered "no regression"; defaults to

Return a copy of the latest `limit` records (None = all).

Return the streaming architecture family, or raise naming the allowlist. Mirrors ``utils/shrink.arch_family_of_config`` — an allowlist, not a

Convert document boundaries into per-token 1-indexed segment IDs. Example: ``[0, 3, 5, 9]`` (3 docs of length 3, 2, 4) → ``[1, 1, 1, 2, 2, 3,

Validate a base model name against the bundled probe catalogue. Case-insensitive: returns the canonical entry that matches.

Schema-time gate for ``long_context_grpo=True``. Rejects on: - non-GRPO task (long_context_grpo only meaningful for RL rollouts); - mlx b