Token usage statistics for a single model call.

Result of parsing LLM output.

Result from a chat completion call. ``content`` is the textual response. The provider's full message object is preserved in ``message`` for c

A multi-endpoint client that retries across endpoints on failure. Each ``create()`` round-robins over a pool of underlying :class:`ChatComple

Multimodal rubric-based scoring agent (v3_mm). Produces two independent signals: - PROCESS REWARD (Steps 0-7): fine-grained rubric score

Configuration for the multimodal rubric verification agent.

Held-out evaluation on WebTailBench, scored with MMRubricAgent.

Task-only critical-point classification produced by the rubric agent. Stored on ``data_point.verification`` under the key ``"rubric_critical_

WebSurferSystem that communicates with either a local or hosted WebSurfer model to perform web-based tasks. If websurfer_client_cfg i

Manages browser instance, context, and page lifecycle.

Observation from the environment: screenshot and/or page state.

A single data generation run output.

Binary outcome verification from the rubric pipeline.

Result from the multimodal rubric verification pipeline.

Metadata from majority voting in rubric scoring.

Observations before an action.

Wrapper around the OpenAI Python SDK (``openai.AsyncOpenAI``).

A single step: observations grouped around an action. Computed, never serialized.

Defines a task for the agent to solve.

Result from the unified task verification classification (Step 10). Classifies a task along two axes before execution: - Task Ambiguity (

Fake ChatCompletionClient whose create() always raises ``exc``. Exposes the minimal surface GracefulRetryClient.create() touches: ``endpoint`

Minimal stand-in for ``MMRubricAgent`` (Steps 0–8). Returns a fixed rubric + outcome verdict (with the new CP-classification fields populated

Stand-in for ``VerifierAgent`` — Steps 9a / 9b / 10. Mirrors the contract documented on ``VerifierAgent.verify`` — takes the rubric + outcome

The agent's decision and action.

Wrapper for Azure ML managed-online endpoints with a custom ``/score`` API.

Wrapper around Azure OpenAI Chat Completions.

Wrapper around the Azure OpenAI Responses API (codex / o3-pro / gpt-5).

Configuration for a single critical point type.

Metadata about a data generation run.

Represents a function call with arguments

Token usage from an LLM call.

File handler that serialises structured webeval events to JSONL.

Response from model call

Result of solving a task.

Variant that targets the Responses API (codex / gpt-5 family).

Solver output containing the sequence of observation-action steps.

Per-step structured data extracted from SolverLog.steps.

Steps 9–10 of the multimodal rubric pipeline. Consumes the scored rubric + outcome verification produced by :class:`MMRubricAgent` and produc

Configuration for :class:`VerifierAgent` (Steps 9a/9b/10).

Loads the WebVoyager dataset, stores it locally, and evaluates predictions using the GAIA evaluator.

Trimmed Agent base — holds config, no run-context plumbing.

Base configuration for agents.

Snapshot of agent's internal state at decision time.

All systems should implement this interface.

Base benchmark class. Responsible for: - Downloading/loading data into data/<benchmark_name>/. - Storing examples in self.examples (e

Base interface for chat completion clients.

Namespace for the two ChatCompletionClient construction helpers. The wrappers in this module already carry the metadata callers historically

Base class providing serialization/deserialization via pydantic.

Judge confidence levels, with human-readable descriptions. Each member carries a short value (``"HIGH"``, etc.) and a longer ``description``

Verifier agent that classifies a task's critical-point type. Two usage patterns: 1. **Via RunContext** (``run``): reads the task from the Da

Configuration for ``CriticalPointAgent``.

Loaded configuration for all critical point types. Typically deserialized from ``critical_point_types.yaml``.

Image wrapper for handling screenshots and images.

Image wrapper for handling screenshots and images

Score for a single screenshot against task key points.

Structured log event emitted by :class:`LogHandler` to record token usage.

A sequence of LLM calls.

A single LLM call: request/response pair.