Extract text from message content

Find all run directories in the target path

Fix common parameter name mistakes made by LLM. Args: tool_name: Name of the tool being called arguments: Or

Format token usage statistics, return summary_lines for format_final_summary and log string

Generate agent-specific objective prompts based on agent type. Different agent types have different objectives: - main: Task-solving age

Generate a failure experience summary for context compression. This is the core of the context management mechanism. When a task att

Generate the MCP (Model Context Protocol) system prompt for LLM. Creates a structured prompt that instructs the LLM on how to use available

Get main agent message history

Get main agent message list

Extract the query string from tool call arguments based on tool_name. Supports search_and_browse, google_search, sogou_search, scrap

Get paths to conversion scripts Returns: Tuple of (oai_script_path, non_oai_script_path)

Parse tool calls from LLM response content. Supports multiple formats: - OpenAI Response API format (dict with 'output' containing funct

Parse MCP tool call

Perform a search and return organic results and search parameters.

Run inference for a single benchmark task with pass@k support Args: task: BenchmarkTask object Returns:

Safely parse a JSON string with multiple fallbacks. Parsing strategy: 1. Try standard json.loads() 2. If it fails, try json_repair t

Set the task logger for structured logging.

Parse system prompt and cache the tool_name → server_name mapping. Should be called once when system prompt is available. Args:

Check if a tool result should trigger a rollback. Args: tool_name: Name of the tool result: The result value

Send show_error event and signal stream end. Args: error: The error message to display

Serialize the TaskLog to a JSON string. Converts the dataclass to a dictionary, handles non-JSON-serializable objects (like

Update message history with tool calls data (llm client specific)

Create an LLM client based on the provider specified in configuration. This factory function automatically selects and instantiates the appr

Extracts ZIP files to a temporary directory and processes each file according to its extension. Returns a combined result of all processed fi

Initialize the benchmark run Args: cfg: Hydra configuration object

Apply cache control to the last user message and system message (if applicable)

Create LLM client

Send message to OpenAI API. :param system_prompt: System prompt string. :param messages_history: Message history list.

Simple language detection based on character analysis. Returns a language description suitable for the summarize prompt.

Display summary statistics

Display summary statistics

Lazy load pipeline components on first request.

Extract run number from directory path for sorting

Extract task-relevant information directly from an audio file based on the task description. Args: audio_path: Path to the audio fil

Extract task-relevant information directly from an image based on the task description. Args: image_path: Path to the image file

Extract task-relevant information directly from a video based on the task description. Args: video_path: Path to the video file

Format scrape/webpage results in a card layout.

Format google_search results in a beautiful card layout.

Format sogou_search results in a beautiful card layout.

Convert <think> tags to readable markdown format.

Determine the appropriate audio file extension from URL or content type. Args: url: The URL of the audio file content_type:

Returns a template for the summary data structure.

Check if the URL is a banned URL. :param url: The URL to check :return: True if it's a banned URL, False otherwise

Check if the URL is a HuggingFace dataset or space URL. :param url: The URL to check :return: True if it's a HuggingFace dataset or space

Check if the URL is a HuggingFace dataset or space URL. :param url: The URL to check :return: True if it's a HuggingFace dataset or space

Check if the URL is a Hugging Face dataset or space URL. :param url: The URL to check :return: True if it's a HuggingFace dat

Create a cached async function for fetching sub-agent tool definitions. This factory function returns an async closure that lazily fetches a

Load benchmark data and configuration

Load GAIA-specific data and configuration

Load task IDs from GAIA-Text-103 dataset

Load JSON file

Patch input handler to remove boxed format requirement.

Patch output formatter to disable boxed format check. In demo mode, we don't require \boxed{} format, so we patch the format_final_summa

Patch summarize prompt generation for better user experience.

Patch system prompt generation to include custom identity.

Reset token usage counter to zero. Returns: A new TokenUsage dict with all counters set to zero.

Save analysis results to a log file in the target directory

Update difficulty level statistics for a task

Helper method to update log file with evaluation result

Update summary statistics with data from a single run

Update cumulative token usage

Update cumulative token usage

Validate file size based on media category. Returns: Tuple of (is_valid, error_message)

Verify the answer for a given dataset. Args: benchmark_name: Name of the benchmark dataset question: The question being answ

Asynchronous main function.

Analyze a single run directory and return statistics and task results Returns: Tuple[TaskStats, Dict[str, bool]]: Statistics and

Analyze a single run directory and return statistics (GAIA-specific) Returns: Tuple[GAIATaskStats, Dict[str, bool]]: Statistics a

Apply monkey patches to customize prompts for demo mode. Patches applied: 1. `generate_mcp_system_prompt` - Prepends custom identity pro

Batch process multiple files Args: input_paths: List of input paths output_dir: Output directory Returns: Dicti

Calculate average scores from multiple runs - automatically detect pass_at_k value

Calculate DeepSearchQA-specific metrics from results file. Following the official Google DeepSearchQA evaluation metrics: 1. Fully Correc

Calculate metrics from individual task log files (for in-progress runs). Returns: Dict with metrics or None if no completed tasks fo

Calculate mean and standard deviation of a list of values

Check if dependencies are installed