Look up a cached result. Returns None on miss.

Add an entry to the trace.

Execute all pipeline steps on the input data. Applies each transformation step in sequence. If any step raises an exception, the pipe

Return the sum of *a* and *b*.

Load and return the headers.json contents.

Execute a single task: prepare workspace -> send to agent -> verify. This runs locally (no Docker) for simplicity. The agent adapter handles

Find an element matching the given criteria.

Create summary.json and manifest.sha256 inside *results_dir*. Returns the path to the manifest file.

Compute aggregate statistics for a full benchmark run. Parameters ---------- all_results: All task results across all runs. t

Scan *tasks_root* for task directories and return matching tasks. The tasks directory is organized as:: tasks/<domain>/<task-id>/task.to

Best-effort parse of pytest text output when JSON is unavailable.

Compute consistency metrics from variant test results. Args: variant_results: Maps variant_id -> list of boolean pass/fail results.

Create a minimal task.toml in the given directory.

Convenience helper that builds a ``TraceEntry`` with the current timestamp.

Create and start the container. Returns the container id.

Import and return the adapter class by spec name.

Load a module from the workspace by filename.

Compute a difficulty-weighted score where harder tasks count more. Parameters ---------- results: Task results from one or more r

Compute SHA-256 hashes of all files in *results_dir*.

Compute dimension scores and a weighted composite. Parameters ---------- results: Task results from one or more runs. metrics

Helper to find an employee entry by name.

Return estimated cost in USD for the given token counts. Falls back to the utils.cost module, then to zero if model is unknown.

Linearly map *raw* from [min_val, max_val] to [0, 100]. Values outside the range are clamped.

Verify that every file in *results_dir* matches the stored manifest.

Heuristic: task IDs contain a dash followed by digits (e.g. file-001, cal-012).

Strip examples, reduce to bullet points of requirements only.

Return a copy of the recorded entries.

Store a result in the cache.

Recursively find and load benchmark result JSON files. Discovers three kinds of files: - summary.json / leaderboard.json (legacy format with

Block until *tokens* are available. Returns False on timeout.

Identify non-dominated points on the cost-performance plane. Each entry in *results* must have ``"cost"`` (lower is better) and ``"score"`` (

Compute statistics for a single task across multiple runs. Parameters ---------- task_id: The task identifier. results:

Validate model-tier consistency. Checks that: 1. The tier is a valid canonical tier 2. The model appears in the specified tier in models.

Comprehensive pre-flight validation for a benchmark run. Checks weight profile, skills mode, model tier, and run count.

Return True when *value* has the conventional task-id shape.

Reverse the order of numbered requirements while keeping everything else.

Parse the JSON report produced by pytest-json-report. Falls back to basic stdout analysis when the JSON file is unavailable.

Compute a deterministic hash of task content for cache keying.

Calculate skills gain metrics from the 3-condition comparison. Parameters ---------- pass_rate_vanilla: Pass rate with no skills

Discover plugin adapters via the ``claw_bench.adapters`` entry points. Each entry point may reference either: * a :class:`ClawAdapter` subcl

Extract text under a ### heading.

Get a database connection.

Import Stack class from workspace/stack.py.

Fork *repo*, create a branch, commit results, and open a PR. Uses the ``gh`` CLI tool for GitHub operations. Returns the URL of the created p

Check that a weight profile exists and sums to 1.0.

Find all issues for a given component.

Locate the tasks directory, preferring an explicit path and CWD.

Locate the tasks/ directory relative to the project.

Execute the task inside a Docker container.

Compute the estimated cost in USD for a given model and token counts.

Compute a weighted score given task results and a weight profile. Each result dict must have keys: 'level' (str) and 'score' (float 0-1). Ret

Return a frozen ``Metrics`` snapshot.

Generate deterministic instruction variants for robustness testing. Creates variants by applying systematic transformations: - terse: Minimal

Load an identity by *claw_id*, or ``None`` if not found.

Read task.toml from *task_dir* and return a validated TaskConfig. Raises ``FileNotFoundError`` if task.toml is missing and ``pydantic.Validat

Execute the pytest verifier for a task and return structured results. The verifier script is expected at ``<task_dir>/verifier/test_output.py``.