Preview tasks without running them.

Print a human-readable evaluation report.

Print results

Print copying summary to console

Print grading summary

Process user input, especially files. Returns formatted initial user message content list and updated task description.

Process LLM response

Quick completion for simple tasks like compression. Uses minimal parameters for fast response. Args: pro

Rerank chunks based on relevance to the query. Only filters out irrelevant chunks (below threshold), keeps all relevant ones

Restore all hidden db files in a folder. NOTE: This function is kept for backward compatibility but is no longer needed since prepar

Run multiple tasks in sequence. Args: data_dir: Directory containing task data tasks: List of task dictionaries resul

Main entry point for running benchmarks with Hydra.

Run the full benchmark evaluation process

Execute the main end-to-end task

Run inference on multiple tasks in parallel using threading

Run a single task and save results to a unique run folder. Args: data_dir: Directory containing task data task: Task dictiona

Run inference for a single benchmark task with pass@k support Args: task: BenchmarkTask object Returns:

Save batch execution summary to a file in the result folder. This file contains: - Execution statistics - List of tasks that nee

Save the final report (after validation) to the run folder.

Save the initial report (before validation) to the run folder.

Save RAG queries summary to a human-readable txt file. This file shows all RAG queries organized by agent and turn, making it easy t

Save evaluation results to JSONL file

Convert objects to JSON-serializable format

Set the allowed database path for RAG operations. When set, only this path (or paths that resolve to it) can be used for RAG operations.

Set the path for saving retrieval results.

* Sort the rows in a HTML Table * * @param Table The Table DOM object * @param col The zero-based column number by which to sort * @param dir

Start a new sub-agent session

Summarize each chunk to extract key information. Args: query: The original search query chunks: List

Update original task files with grading results

Use BrowseComp judge to verify if the predicted answer is correct.

Verify the answer for a given dataset. Returns a tuple of (result, judge_type).

Use HLE-style LLM judge to verify if the predicted answer is correct. Returns the evaluation result as a string: "CORRECT", "INCORRECT", or "

Use LLM to verify if the predicted answer is correct. Expects the LLM to choose between A (correct), B or C (incorrect).

Use XBench-DeepResearch judge to verify if the predicted answer is correct.

Converts WAV files to markdown via extraction of metadata (if `exiftool` is installed), and speech transcription (if `speech_recognition` is

Initialize Progressive Memory. Args: max_tokens: Maximum tokens allowed in memory reserved_tokens: T

Initialize compressor with LLM client. Args: llm_client: LLM client for compression

Initialize ToolManager. :param server_configs: List returned by create_server_parameters()

Initialize from environment variables

Initialize Enhanced VQA client from environment variables.

Initialize the reranker. Args: api_key: OpenAI API key (default: from OPENAI_API_KEY env var) base_u

Initialize the summarizer. Args: api_key: OpenAI API key (default: from OPENAI_API_KEY env var) base

Initialize the post-processor. Args: api_key: OpenAI API key base_url: OpenAI API base URL

Initialize RAG Tool. Args: api_key: OpenAI API key base_url: OpenAI API base URL embeddi

Initialize from environment variables

Initialize the evaluation runner with config and metrics.

Initialize benchmark evaluator Args: data_dir: Path to benchmark data directory benchmark_name: Name of the

Initialize generic evaluator Args: data_dir: Path to benchmark data directory benchmark_name: Name of the be

Initialize the copier Args: gaia_text_103_data_path: Path to GAIA-Text-103 standardized data file output_dir

Initialize the grader Args: extraction_dir: Directory containing extracted GAIA-Text-103 tasks