Evaluation result.

The result of converting a document to text.

Main class for checking benchmark progress

Information about a document chunk.

A custom version of markdownify's MarkdownConverter. Changes include: - Altering the default heading style to use '#', '##', etc. - Remo

Evaluation configuration.

LLM client for evaluation API calls.

A single search result from RAG.

Exception raised when an API connection error occurs and the current task should be skipped. This exception is NOT retried - it immediat

Exception raised when rate limiting is detected.

Exception raised when API request times out.

Generic benchmark evaluation result structure

Evaluates whether the report cites all required documents.

Enhanced Vision Question Answering client supporting multiple VLLM providers. Provides: 1. Single-turn VQA with confidence scoring

Information recall evaluator using ternary scoring (1.0 / 0.5 / 0.0).

Class to maintain a persistent Playwright MCP session.

Generic benchmark task data structure

Custom formatter that adds colors for better developer visualization

Main class to run a benchmark

Loads and manages source documents and long-context data.

Enhanced Audio Understanding Client with multi-turn verification and multi-provider support

Enhanced Video Understanding Client with multi-turn verification and multi-provider support

Raised when an API call fails during evaluation.

Overall evaluation result.

Configuration for file content extraction limits.

Information about a single file.

Structured representation of folder contents.

Summary statistics across all runs

Statistics for a single task

Grader for GAIA-Text-103 tasks using LLM judgement

Copy GAIA-Text-103 tasks from GAIA validation logs

Generic benchmark evaluator for JSONL format

Manages global progress tracking for batch evaluation.

Result of grading a single task

Represents a match between a golden insight and content in the log.

A single item in the memory

Rerank retrieved chunks using LLM-based relevance scoring. This class uses an LLM to evaluate the relevance of each retrieved chunk

Summarize retrieved chunks to extract key information. This class uses an LLM to extract the most relevant information from each chunk,

RAG Tool for semantic search over long context documents. Uses chunk-based processing to handle long documents and SQLite to cache e

Record detailed information of task execution steps

Summary statistics across all runs

Statistics for a single task

we unify openai and anthropic format. there are four usage types: - input/output tokens - cache write/read tokens openai: - cache

Base class for all evaluators.

Abstract base class for benchmark evaluators

Priority levels for memory content

Main class for checking GAIA benchmark progress

Record technical details of LLM calls

Helper class to compress memory content using LLM.

Progressive Memory Manager with priority-based compression. When context limit is reached: 1. First, compress lower-priority items (

Combined post-processor that applies both reranking and summarization. This is the main entry point for RAG post-processing, combining

Result of reranking a single chunk.

Result of summarizing a single chunk.

Record detailed information of tool calls

this enables other kinds of tool manager.