Helper to create a callback wrapper that handles both single results and lists.

Attempt to recover a closed page by reloading it. Args: page: The Playwright page object to recover timeout_secs: Timeout fo

Move vLLM-only sampling params into ``extra_body`` so the OpenAI SDK accepts them.

Raise ``ValueError`` if *result* is missing or mis-typed fields.

Returns the smallest integer greater than or equal to 'number' that is divisible by 'factor'.

Close the browser and the page. Should be called when the agent is no longer needed.

Compute aggregate metrics (e.g. average score). Must produce a dictionary. Computes: - mean_score: average over all - mean_score

Compute only refusal detection metrics.

Construct a client from a config dict. The dict must contain ``CHAT_COMPLETION_PROVIDER`` ("openai", "azure", "trapi", "azure_ml", or "gracef

Download or load the dataset into data/<benchmark_name>/.

Evaluate the benchmark. Args: benchmark_name (str): Benchmark name. system_name (str): System name. benchmark_dir (s

Evaluate a single example. Args: example_data (tuple): (ex, output_dir, redo_eval). Returns: tuple: (qid, score, durati

Return the taxonomy text for categories *start* through *end*. The result is raw markdown (NOT Template-escaped). Call ``escape_for_template(

Extract final answer data from _final_answer.json content. Returns: dict: Contains is_aborted, final_answer, num_screenshots, and to

Extract the prediction from the response.

Return only the ``- **N.M …**`` bullet lines for a single category. Useful when the prompt provides its own framing/preamble and only needs t

Returns scores where reasoning does not contain 'captcha' (case-insensitive) and score is not None.

Returns the largest integer less than or equal to 'number' that is divisible by 'factor'.

Format step summaries into the ``State N / Action N`` text format.

Get the current page URL.

Return ``(taxonomy_text, summary_table)`` for the failure-analysis prompt. Categories 1-6. Category 6 is post-processed to contain ``$action

Return the last observation, or None if empty.

Loads the previously downloaded dataset from self.data_dir into self.examples.

Load or download the benchmark data. Args: benchmark_name (str): Benchmark name. benchmark_dir (str): Directory for benchmar

Normalize a URL by unquoting the path, removing any trailing slash, and sorting query parameters. Args: url (str): The URL to normal

Remove the screenshot from the message content.

Returns the closest integer to 'number' that is divisible by 'factor'.

Run the benchmark. Args: benchmark_name (str): Benchmark name. system_name (str): System name. parallel (int): Numbe

Run a single task in a separate process. Args: question_data (tuple): (question_id, example_data, output_dir). Returns:

Pick the right user-simulator policy block. Args: enabled: ``MMRubricAgentConfig.user_simulator_enabled`` value. for_outcome: ``T

Start Xvfb virtual display server.

Build formatted error-code definitions for FINE_GRAINED_GROUNDING_PROMPT. Pulls definitions from ``error_taxonomy_analysis.md`` at import time.

Call an LLM client and return the text content.

Decide whether the trajectory crossed the irreversible-action boundary in violation of the user's permissions. This is a focused, sin

Step 10: Delegates to :func:`task_classification.classify_task`. Returns the ``TaskAgentResult`` as a plain dict so it can be stored

Step 9b: Trajectory-informed task verification. Uses the same ambiguity / validity axes as Step 10 (``CHECK_VALID_TASK_PROMPT``), but

Clean MLflow keys by replacing unsupported characters, Names may only contain alphanumerics, underscores (_), dashes (-), periods (.), spaces ( ), col

Compute ``first_failure_step`` and ``first_failure_summary`` from the LLM's ``failure_points`` list. Priority: first failure of any k

Build a bearer-token provider with sensible Azure credential priority. Order: AzureCliCredential → ManagedIdentityCredential → DefaultAzureCr

Visually verify that coordinate-bearing actions land on the intended target. For each action whose tool definition includes ``x`` and ``y``

Programmatically detect 6.1 (Invalid invocation) and 6.2 (Hallucinated action) errors by comparing each step's action name and argumen

Download model from HuggingFace Hub if not already present.

Extract the configured starting URL/page from task.environment_config.

Return the full summary decision table (header + all rows).

Step 9a: Failure Point Analysis — identify all failure points in the trajectory. The first (earliest) point of failure is computed pr

Return action names whose parameters include ``x`` and ``y``. Inlined from agento_next's ``computer_agent.tools.get_coordinate_actions`` so t