Parse system prompt to extract tool_name → server_name mapping. Parses patterns like: ## Server name: tool-python ### Tool n

Send POST request with retry and exponential backoff. Returns response object if success, otherwise None.

Prepare task description and file path for the agent Args: task: BenchmarkTask object Returns: Tupl

Print results

Print copying summary to console

Print grading summary

Process user input and associated files. Extracts content from the task file (if provided) and appends it to the task description in a f

Process LLM response

Process a single log file, extract message history and convert to ChatML format Args: log_file_path: Log file path output_di

Process a single JSON file Args: json_file_path: Path to JSON file output_dir: Output directory Returns: True i

Main entry point for running benchmarks with Hydra.

Run the full benchmark evaluation process

Execute the main end-to-end task. Args: task_description: Description of the task to execute task_file_name:

Run inference on multiple tasks in parallel using multiprocessing

Run a sub-agent to handle a subtask. Args: sub_agent_name: Name of the sub-agent to run task_description: De

Safely decode URL-encoded strings, only decoding characters that won't alter URL semantics. Preserve the following encodings (because decodi

Save ChatML format messages to files Args: chatml_data: Dictionary containing message history output_dir: Output directory

Save evaluation results to JSONL file

Scrape content from a URL and save to a temporary file. Need to read the content from the temporary file. Args: url (str): The URL

Fallback scraping method using Python's httpx library directly. Args: url (str): The URL to scrape content from custom_heade

Convert objects to JSON-serializable format

Show DeepSearchQA-specific metrics for all runs. Following Google DeepSearchQA official metrics: 1. Fully Correct 2. Fully Incorrect

Show help information

Start a new sub-agent session

Send start_of_workflow event. Args: user_input: The initial user input for the workflow Returns: Th

Optimized event stream generator that directly outputs structured events, no longer wrapped as SSE strings.

Test the complete tool block regex used in streaming.

Test edge cases and potential bugs.

Test the partial tool regex for streaming.

Test the main tool call regex pattern.

Update original task files with grading results

Use BrowseComp judge (English version) to verify if the predicted answer is correct. Expects the LLM to return A (correct) or B (incorrect).

Use BrowseComp judge (Chinese version) to verify if the predicted answer is correct. Expects the LLM to return A (correct) or B (incorrect).

Use DeepSearchQA-specific judge to verify if the predicted answer is correct. Uses the official DeepSearchQA evaluation prompt with JSON outp

Wrapper with retry logic for NOT_ATTEMPTED results. Args: benchmark_name: Name of the benchmark dataset question: The questi

Use HLE-style LLM judge to verify if the predicted answer is correct. Returns the evaluation result as a string: "CORRECT", "INCORRECT", or "

Use LLM to verify if the predicted answer is correct. Expects the LLM to choose between A (correct), B or C (incorrect).

Use XBench-DeepSearch judge to verify if the predicted answer is correct.

Initialize analyzer Args: json_file_path: Path to the JSON file

Initialize the stream handler. Args: stream_queue: Optional async queue for sending stream messages.

Initialize the answer generator. Args: llm_client: The LLM client for API calls output_formatter: Formatter

Initialize the orchestrator. Args: main_agent_tool_manager: Tool manager for main agent sub_agent_tool_manag

Initialize the tool executor. Args: main_agent_tool_manager: Tool manager for main agent sub_agent_tool_mana

Initialize benchmark evaluator Args: data_dir: Path to benchmark data directory benchmark_name: Name of the

Initialize generic evaluator Args: data_dir: Path to benchmark data directory benchmark_name: Name of the be

Initialize the copier Args: gaia_text_103_data_path: Path to GAIA-Text-103 standardized data file output_dir

Initialize the grader Args: extraction_dir: Directory containing extracted GAIA-Text-103 tasks

Initialize ToolManager. :param server_configs: List returned by create_server_parameters()

Validate info_level after initialization

Create LLM client

Send message to Anthropic API. :param system_prompt: System prompt string. :param messages_history: Message history list.

Encode audio file to base64 and determine format.

Ensure the tool_id is valid and arrays have enough elements

Fix common backslash escape issues in JSON strings. This handles cases where backslashes in string values are not properly escaped. Comm

Format response for logging

Get audio duration in seconds. Tries to use wave (for .wav), then falls back to mutagen (for mp3, etc).

Sync wrapper for run_single_task to be used in threads

Save message history to task log.

Worker function to run a single task in a separate process. This function is called by ProcessPoolExecutor and must be at module level.

Create a task plan by adding one or more task items. CRITICAL: Before starting to work on ANY task, you MUST first create a complete task pl

Answer the question based on the given audio information. Args: audio_path_or_url: The path of the audio file locally or its URL. Pa

Transcribe audio file to text and return the transcription. Args: audio_path_or_url: The path of the audio file locally or its URL. P

Transcribe audio file to text and return the transcription. Args: audio_path_or_url: The path of the audio file locally or its URL. P

Calculate average turns per completed task

Mark one or more tasks as completed in your plan. Use this after finishing a task to track your progress and maintain an accurate view o

Calculate completion rate percentage

Detect if a string contains Chinese characters or Chinese punctuation Args: text (str): The string to detect Returns: b