Iterates over types as ``(name, dict)`` pairs for template use.

Call an LLM client and return the text content. Wrapper-agnostic: handles both webeval-native ``CreateResult`` (whose ``.content`` is already

Returns the set of type names.

Encode *image* as a base64-encoded JPEG string. ``quality`` defaults to 95 (near-lossless). ``subsampling=0`` is pinned so chroma is not subs

Return True if ``model_name`` matches ``model_name_filter``. ``model_name_filter`` may be a single string or list. ``"*"`` matches ev

Convert a PIL image to base64 string.

Wait for the page to reach a specific load state.

Alias for :func:`create_completion_client_from_env` taking a dict.

Read the taxonomy markdown file.

Coerce an ``azure_ad_token_provider`` config value into a callable. Accepts ``None`` (default → DefaultAzureCredential), ``"DEFAULT"`` (same)

Convert *task_data* + *candidate* Trajectory into a :class:`DataPoint`. Parameters ---------- task_data : dict Must contain ``"qu

Return observations of a specific type from an event stream.

Split the markdown into a dict mapping category number → full text. Each value includes the ``## N. …`` heading through to (but not including

Escape ``$`` → ``$$`` so *text* is safe inside a ``string.Template``.

Extract model name from URL for consistent naming.

Parse the minimum step number from a ``step_numbers`` string. Handles formats: ``"5"``, ``"5-7"``, ``"5,8,12"``, ``"8,5"``, ``"3-7,12"``.

Returns a string hash of the benchmark evaluation configuration suitable to be a path. This is used to route results with different p

Return ``(name, definition)`` for a single subcategory code like ``'6.3'``. Parses the ``- **N.M Name** — Definition...`` bullet from the taxonom

Capture a screenshot of the current page. Args: page (Page): The Playwright page object. path (str, optional

Extract structured per-step data for the current agent format. Current agents (GPT54Agent, ComputerAgent) log content as ``{"action":

Dynamically load a system class based on the system name.

Convert an :class:`LLMMessage` to an OpenAI chat-completion dict.

Run the agent with a user message.

Convert a DataPoint into the dict expected by _generate_reply.

Full rubric verification pipeline. This is the direct port of the original _generate_reply() from rubric_agent_v3_mm.py, adapted to w

Wrap the raw rubric dict into two VerificationResult objects.

Helper function to calculate statistics for a group of trajectories.

Classify a task into a critical-point type for rubric shaping. See module docstring for context.

Unified task verification classification across ambiguity and validity.

Extract the application name(s) from environment_config.apps.

Extract the starting URL, preferring task config over solver log events.

Returns scores where the answer is not empty, None, or <no_answer> and score is not None.

Returns scores where reasoning does not contain any error string and score is not None.

Return metadata for a harness analysis code (9.1 or 9.2). Used exclusively by ``_detect_fine_grained_grounding_errors``. Raises ``KeyError``

Helper to load an answer from disk if it exists.

Parse the TSV(s) into :attr:`examples`. Reads the ``precomputed_rubric`` column when present (the ``WebTailBench-v1-rubrics.tsv`` fil

Turn an endpoint-config dict, dict-list, file path, or directory into a :class:`webeval.oai_clients.ChatCompletionClient`. Plain dicts are ha

Create annotated wide-view and zoom-in images for grounding checks. Returns: (wide_view, zoom_marked, zoom_unmarked) —

Extract a crop centred on (*x*, *y*), clamped to image bounds.

Draw concentric circles on a copy of *image* at (*x*, *y*). Returns a new image; the original is not modified.

Build LLM clients from their *_client_config if absent. Called from ``__init__`` and from :meth:`initialize` so that retry workflows

Get current screenshot and scale it for the model.

Approximate image token cost (OpenAI high-detail vision pricing).