Get performance summary information

Get parameters for specified server

Get spans statistical summary

Get step logs summary statistics

Get current cumulative token usage - OpenAI implementation

Get current cumulative Token usage - Anthropic implementation

Get current cumulative token usage - OpenAI implementation

Get current cumulative token usage - OpenAI Response API implementation

Get current cumulative token usage - OpenAI implementation

Get current cumulative token usage - OpenAI implementation

Create a new handoff span. The span will not be started automatically, you should either do `with handoff_span() ...` or call `span.start()` + `sp

Install dependencies (recommended to use uv)

Load configuration from YAML file

Load queries and model results data

Main function

Main function

Main function.

Parse command line arguments.

Fix escape issues in JSON strings, supports smart syntax fixing Uses 5 progressive parsing strategies: 1. Direct parsing - return direct

Print information about cache usage

Print formatted evaluation results

处理单个目录 Args: json_dir: JSON 文件目录路径 output_base_dir: 输出基础目录 timeout: 超时时间 pattern: 文件匹配模式 Re

Process user input, especially files. Returns formatted initial user message content list and updated task description.

处理单个 JSON 文件 Args: json_file_path: JSON 文件路径 output_dir: 输出目录 timeout: 超时时间（秒） Returns: boo

Processes span-data for performance profile. This hydrates `TaskTracer.trace_data`.

Add a processor that will receive all traces and spans.

Execute the complete generation pipeline. Returns: List of final generated tasks

Execute the main end-to-end task.

Save results to JSON file

Score a single report using pointwise evaluation - processes each dimension in parallel

This function uses requests to scrape a website. Args: url: The URL of the website to scrape.

This function uses SERPER for scraping a website. Args: url: The URL of the website to scrape.

Select queries based on configuration Args: query_selection_config: Query selection configuration

Enable or disable tracing globally.

Set the list of trace processors. This will replace the current list of processors.

Sets up a custom trace processor for the benchmark and returns it. Ensures that only one instance of the processor is created and used global

Setup logging configuration

Configure logging.

Asynchrono us main function.

Asynchrono us main function.

Get cache size

Start the span. Args: mark_as_current: If true, the span will be marked as the current span.

Test various API endpoints

Cleanup when object is destroyed

Initialize analyzer Args: json_file_path: Path to the JSON file

Args: api_key: The API key for the "Authorization" header. Defaults to `os.environ["OPENAI_API_KEY"]` if not prov

Args: exporter: The exporter to use. max_queue_size: The maximum number of spans to store in the queue. After this, w