Extract task information from a task file

Recursively extract text from a shape, including nested GROUP shapes. Args: shape: The shape to extract text fro

Extract tool call information from LLM response

Extract trace ID from an exception. Looks for trace ID in: 1. Response headers (eagleeye-traceid) 2. Error message body (traceId

Extract word count requirement from query. Looks for patterns like: - "1500-2000字" - "2000字左右" - "不少于1000字" - "1500 to 2

Filter google search result content based on environment variables. Args: result_content: The JSON string result from google search

Filter google search result content based on environment variables. Args: result_content: The JSON string result from google search

Find where an insight appears in the conversation history. Args: insight: The golden insight text insight_index: Index o

Find all task JSON files in the extraction directory

Format token usage statistics, return summary_lines for format_final_summary and log string

Generate an incremental summary report of evaluation results.

Generate a prompt to validate if the report meets all query requirements. This prompt instructs the agent to use Python code execution tool t

Generates a summary of benchmark results by reading log files from a directory, calculating total and average trace data, both overall and gr

Get color code based on log level for better visual distinction

Get the default evaluations output directory.

Collect current configuration environment variable information for logging

Determine the category of a file based on its extension.

Read model_name from the LLM config file. Args: llm_config: Name of the LLM config (e.g., 'qwen3_235b' for conf/llm/qwen3_235b.y

Args: Returns:

Process scrape result and truncate if too long to support more conversation turns.

Get parameters for the specified server

Grade all tasks with concurrent processing

Grade a single task using GAIA-Text-103 evaluator

Guess the MIME type based on the file extension.

Guess the MIME type based on the file extension.

Check if a shape is a group

Check if a shape is a picture

Check if a report file is valid (exists and has sufficient content). A report is considered invalid if: - It doesn't exist - It'

Check if a shape is a table

Load documents from a long_context.json file and split into chunks. Uses SQLite cache if available and valid. If a .

Load .env file from various possible locations

Load and parse the execution log JSON file.

Load documents directly from a pre-built SQLite database. This skips the JSON file entirely and loads embeddings from the db.

Load golden insights from JSON file.

Load benchmark tasks from metadata.jsonl Args: limit: Maximum number of tasks to load (None for all) Returns:

Load tasks from a JSONL file.

Args:

Args:

Main entry point with Hydra config.

Main function

Main function

Compute the majority-vote prediction for a list of candidate predictions. Tie-breaking rules (deterministic): 1) Highest frequency win

Make HTTP request to Serper API with retry logic.

Parse tool call information from the JSON log file.

Send POST request with retry and exponential backoff. Returns response object if success, otherwise None.

Prepare a folder to use a specific context size. IMPORTANT: This function NO LONGER renames files to support concurrent execution. I

Prepare task description and file path for the agent Args: task: BenchmarkTask object Returns: Tupl