Set callback to be called when captcha status changes.

``load_dataset()`` should consume a TSV placed directly in ``data_dir`` without hitting Hugging Face.

Backwards-compat alias for callers that still use ``ClientWrapper.from_config``.

The webeval.utils factory delegates to the new oai_clients module.

End-to-end: load the checked-in example trajectory via ``Trajectory.from_folder`` and convert it with ``create_datapoint``. Intentionally pas

Task_data without ``init_url`` should default to an empty string rather than crashing — WebTailBench TSV does not ship init_url.

The error-taxonomy loader must produce a non-empty taxonomy block plus the summary table the failure-point prompt expects, and must be able to

The example trajectory satisfies the discovery predicate (``web_surfer.log`` + ``*_final_answer.json``).

Real trajectory must load, parse events, and resolve screenshot paths relative to the trajectory directory.

``Trajectory.from_folder`` swallows exceptions and returns None; an empty dir has no web_surfer.log, so the call must not raise.

Regression: a pool that only ever raises AuthenticationError must make create() *terminate* (raising the last error), not spin forever. Pre-f

A pool that only raises NotFoundError must terminate once every endpoint is blocklisted (next_client() raises), never looping forever.

Live HF download: must pull ``WebTailBench-v1-rubrics.tsv`` and every loaded example must carry a ``precomputed_rubric``.

The WebTailBench TSV loader must pull ``id``, ``task_summary``, ``init_url``, and JSON-decode ``precomputed_rubric``.

``WebTailBench-v1-rubrics.tsv`` carries a JSON ``precomputed_rubric`` column. The loader must parse it and stash it on each example so the Uni

Passing a pre-built ChatCompletionClient should satisfy config validation.

MMRubricAgentConfig needs at least one of the two clients or a config dict — construction should assert when both are missing.

Stub the OpenAI client and verify .create() returns a CreateResult with text content and updated usage.

End-to-end: rubric column → example dict → DataPoint.task.metadata. This is the contract the Universal Verifier relies on: ``MMRubricAgent._e

``repr(Trajectory)`` reports the counts debugging scripts rely on.

The fixture dir must contain the file set the current webeval pipeline produces — ``verify_trajectories.find_trajectory_dirs`` keys off ``web_

The ``webeval.rubric_agent`` package should import cleanly and every exported class should resolve to a module inside this package (i.e. no ac

``_run_one`` against a copy of the example trajectory must: * Find the task in ``_GLOBAL_TASKS``. * Load the Trajectory + convert to DataPoin

If the trajectory id isn't in ``_GLOBAL_TASKS`` the runner must fail soft with ``status='no_task_data'``, not raise.

Actually call the real MMRubricAgent against the example trajectory. Skips unless env vars below are set: * ``FARA_VERIFY_LIVE_TEST=1`` (opt-

The rubric agent emits raw OpenAI dicts. The wrappers' ``_to_oai_messages`` must pass dicts through unchanged so no adapter is needed.

Verify that total_max_points and total_earned_points correctly account for conditional criteria. Rules: - Non-conditional criteria: Always co