Shared helper for screenshot lookup and user-message collection.

Safely handle new pages in single tab mode.

Decode an image URI (only data URLs are supported) into an :class:`AGImage`.

Check whether two images are perceptually identical via dhash. Returns ``(is_match, hamming_distance)``. *is_match* is ``True`` when

Initialize BrowserBase connection, defaults to chromium.

Initialize persistent browser with data directory.

Initialize regular browser according to the specified channel.

Check if a given path is an Azure blob URL.

Load the canonical critical-point taxonomy from YAML.

Load all screenshots in chronological order with strict 1-to-1 verification.

Make a model call using OpenAI client

Ensure screenshots follow ``screenshot_{N}.png`` 1-based naming. Creates symlinks when originals use a different convention and returns the l

Replace the static action-space list in category 6 with Template vars. The .md has a concrete list of actions; the prompt needs ``$action_spa

Resolve the ``--eval-config`` dir for the live test from env vars. ``GracefulRetryClient.from_path`` lists ``*.json`` directly under its ``--

Verify one trajectory. Returns a result dict (also writes scores JSON).

Set up features common to all browser types.

Filter summary table to only rows whose error code starts with categories in [start, end]. Keeps the header + separator rows.

Validate the parsed YAML and coerce into a dataclass.

Verify the parameters of the function call

Close the browser and clean up resources.

Compute aggregate metrics (e.g. average score). Must produce a dictionary. You can override this method to compute custom me

Compute aggregate metrics for multiple runs.

Count the number of WebSurferEvent log entries in web_surfer.log files and return detailed statistics about the number of steps per trajector

Construct a chat completion client from a config dict. ``env`` is the same dict shape used elsewhere in webeval (with ``CHAT_COMPLETION_PROVI

Use LLM to detect if the answer is a refusal to complete the task.

Detect refusals in parallel for all answer texts.

Fetch the WebTailBench TSVs from Hugging Face into ``data_dir``. Always pulls ``WebTailBench-v1-rubrics.tsv`` (the file with the ``pr

Return the first observation, or None if empty.

Run the multimodal rubric pipeline on a candidate Trajectory. Converts the trajectory to a DataPoint, invokes ``MMRubricAgent._genera

Evaluate a single example

Implement the evaluator logic here

Returns a string hash of the benchmark execution configuration suitable to be a path. This is used to route results with different pa

Extract timing statistics for WebSurfer actions from log content. Returns: tuple: (avg_time_between_actions_seconds, session_dur

Return all blockquote content (``> ...`` lines) from a single category. Useful for nuance notes and clarifications that appear after the sub-

Extract the last error from a core.log file content. Returns: tuple: (error_type, full_error_description) or (None, None) if no

Extract score from JSON content of score files. Returns: dict or int or None: For '0.8-5-3.json' files, returns a dict with score an

Return the summary decision table filtered to categories [start, end]. Raw markdown, NOT Template-escaped.

Extract and parse a YAML block from an LLM response. Searches for a fenced ``yaml`` code block, then falls back to unfenced and raw patterns.

Returns scores that are not empty, not captcha, and not error, and score is not None.

Returns scores that are not empty, and not error, and score is not None.

A trajectory dir contains web_surfer.log AND a *_final_answer.json.

* Recursively gathers elements matching specified roles from both regular DOM and Shadow DOM * @param {Array} roles - Array of role selectors to

Return an answer for the question. Should use save_answer_to_disk to save the answer.

Resolve a function-call template name to its actual prompt string. Uses the in-repo ``fara.qwen_helpers.fncall_prompt.NousFnCallPrompt`` temp

Return taxonomy text for task-classification prompts (categories 7-8). Returns ``$$``-escaped text ready to embed in Template strings.

Adapted from https://github.com/MinorJerry/WebVoyager/blob/main/evaluation/auto_eval.py Evaluates the candidate answer by calling GPT

Copy scoring fields from the model response onto the original rubric. Validates lexical overlap of criterion strings using CRITERION_SIMILARI