Some implementation notes: 1. Using Queue, which is thread-safe. 2. Using a background thread to export spans, to minimize any performance iss

LLM API client wrapping OpenAI SDK calls.

Load research domains from file.

Configuration parameters for Deep Research task generation.

Represents an Agent Span in the trace. Includes name, handoffs, tools, and output type.

Filter tasks that require deep search.

Generate expert personas in parallel.

A no-op trace that will not be recorded.

Class to maintain a persistent Playwright MCP session.

Assess baseline answer quality, filter tasks needing search augmentation.

Generate Deep Research tasks based on expert backgrounds.

A trace that will be recorded by the tracing library.

API Client

Manager for multiple cache instances used by evaluators

Prints the traces and spans to the console.

Represents a Custom Span in the trace. Includes name and data property bag.

FanoutExporter is a tracing exporter that sends trace data to multiple exporters. It allows you to fan out the trace data to different destin

File-based cache for storing evaluation data persistently

A tracing processor that stores completed traces in memory.

Represents a Function Span in the trace. Includes input, output and MCP data (if applicable).

Represents a Generation Span in the trace. Includes input, output, model, model configuration, and usage.

Represents a Guardrail Span in the trace. Includes name and triggered status.

Represents a Handoff Span in the trace. Includes source and destination agents.

Represents an MCP List Tools Span in the trace. Includes server and result.

NoopExporter does nothing. This replaces default BackendExporter.

OtelExporter is a tracing exporter that sends trace data to an OpenTelemetry endpoint. It uses the `http-json` format for exporting traces.

Deep Research task generation pipeline orchestrator.

Pointwise evaluator for Deep Research Arena Inherits from BaseEvaluator and adds specific functionality for: - Query-specific dimens

Represents a Response Span in the trace. Includes response and input.

A simple processor that stores finished spans in memory. This is thread-safe and suitable for tests or basic usage.

Represents a Speech Group Span in the trace.

Represents a Speech Span in the trace. Includes input, output, model, model configuration, and first content timestamp.

Record detailed information of task execution steps

Forwards all calls to a list of TracingProcessors, in order of registration.

Only use primitive types, datatime, Path etc.

we unify openai and anthropic format. there are four usage types: - input/output tokens - cache write/read tokens openai: - cache

Class for analyzing trace JSON files, facilitating reading and accessing important information Supports two tool call formats: 1. Legacy

Represents a Transcription Span in the trace. Includes input, output, model, and model configuration.

Core methods for pointwise evaluation

Manages the current span and trace in the context.

Represents span data in the trace.

Test suite for Audio MCP Server functionality.

Test suite for Python MCP Server functionality.

Test suite for Reading MCP Server functionality.

Test suite for Reasoning MCP Server functionality.

Test suite for Searching MCP Server functionality.

Test suite for Vision MCP Server functionality with dual-model support.

this enables other kinds of tool manager.

a long-lived sandbox that keeps mcp stdio session alive. vs ToolManager: - ToolManager automatically open/close mcp session in calls to `get_a

A trace is the root level object that tracing creates. It represents a logical "workflow".

Interface for creating traces and spans.

Exports traces and spans. For example, could log them or send them to a backend.

Interface for processing spans.