Get embeddings for all chunks one by one with rate limiting protection. Args: chunks: List of chunks to get embeddings fo

Get MD5 hash of file for cache validation.

Get prompt for heavy compression

Get image information string.

Detect image media type.

Get prompt for light compression

Get appropriate max_tokens for the given model.

Returns a template for the summary data structure.

Get video information string.

Check if the URL is a HuggingFace dataset or space URL. :param url: The URL to check :return: True if it's a HuggingFace dataset or space

Check if the URL is a Hugging Face dataset or space URL. :param url: The URL to check :return: True if it's a HuggingFace dat

Load benchmark data and configuration

Load GAIA-specific data and configuration

Load existing evaluation results from a case directory.

Load task IDs from GAIA-Text-103 dataset

Load user document filenames from the dataset directory.

Log final task statistics

Check if a required title matches a citation using multi-layer matching.

Merge successful and new verification results.

Parses the evaluation XML response into a structured dictionary. Args: response (str): The raw LLM output in XML-like fo

Parse the rankings from LLM response.

Parse the summaries from LLM response.

Refine confidence based on verification consistency. Args: initial_confidence: Initial confidence from primary transcrip

Refine confidence score based on multi-turn consistency.

Refine confidence based on verification consistency. Args: initial_confidence: Initial confidence from primary analysis

Reset token usage counter - implemented by concrete classes

Sync wrapper for run_single_task to be used in threads

Save analysis results to a log file in the target directory

Save cache metadata.

Save chunks and embeddings to cache.

Split a document into chunks using the configured method. Args: text: The document text to split doc

Split a document into chunks based on character count. Fallback method when tiktoken is not available. Args:

Split a document into chunks based on token count. This ensures consistent information density across different languages.

Send end_of_workflow event

Send start_of_workflow event

Transcribe audio using Google's speech recognition Args: local_path: Path to the audio file Returns: Transcription as a

Extract valid partial content from truncated JSON with "results" or "claims" arrays.

Update difficulty level statistics for a task

Helper method to update log file with evaluation result

Update summary statistics with data from a single run

Update cumulative token usage

Update cumulative token usage

For Qwen-Audio, may need to upload to OSS or encode as data URL

For providers that need upload or encoding

Validate configuration

Validate configuration

Validate input parameters. Args: video_path: Path to video file or URL question: Optional question a

Re-analyze key aspects for verification. Strategy: 1. Identify 3 key aspects to verify (actions, objects, temporal sequence)

Re-analyze key segments for verification. Strategy: 1. Identify 2-3 key segments (start, middle, end or keyword-rich parts)

Verify claims against PDF content using GPT-5 responses API.

Verify transcription/answer with web search. Args: transcript: Transcribed text answer: Answer to verify (co

Verify answer with web search. Searches for information related to the answer to provide external validation.

Verify video content with web search. Args: content: Video description or answer to verify Returns:

Analyze a single run directory and return statistics

Analyze a single run directory and return statistics (GAIA-specific)

Args: Returns:

Calculate average scores from multiple runs - automatically detect pass_at_k value

Check if this item can be deleted

Clean markdown content by removing images and normalizing whitespace.

Copy GAIA-Text-103 tasks from GAIA validation logs Args: gaia_logs_dir: Directory containing GAIA validation logs

Call LLM to generate response, supports tool calls - unified implementation Note: APIConnectionSkipError, APIRateLimitError,

Create a unique folder for this task run. Directory structure: result/<run_batch>/<dataset>/<context_size>/<model>/<task_number>/ Example

Traverse all values in the data structure: - If it's a string starting with http, apply urllib.parse.unquote - If it's a list, recursivel

Detect the pass_at_k value used in the results directory

Returns: List of (context_size, model) tuples

Discover subdirectories inside results_dir that potentially contain a 'benchmark_results.jsonl'. We don't strictly require the subdir name to

End the current sub-agent session

Evaluate overall quality of a research report.

Evaluate pass@k accuracy (verification already done in run_single_task)

Main evaluation function. Args: execution_log_path: Path to execution_log.json golden_insights_path: Path to golden insi

Expose sub-agents as tools

Extract conversation turns from execution log. Supports multiple log formats: 1. New format: main_agent_message_history.message_hist

Args: Returns:

Extract document titles from RAG result text.

Args: Returns:

Args: Returns:

Extract key phrases from an insight for matching.

Public method to extract audio metadata. Args: audio_path: Path to audio file Returns:

Args: Returns: