Get E2B API credentials, skip test if not available.

Get model price information

Get OpenAI API credentials, skip test if not available.

Get vision API credentials, skip test if neither is available.

Handles the logic for oai_tool_thinking.

Handle context limit retry logic when processing summary Returns: str: final_answer_text - LLM generated summary text, e

Initialize all prompt templates

Check if the URL is a Hugging Face dataset or space URL. :param url: The URL to check :return: True if it's a HuggingFace dat

```proto // KeyValue is a key-value pair that is used to store Span attributes, Link // attributes, etc. message KeyValue { str

Legacy simple escape method: mainly handles special cases of code_block field

Load cache from file

Load JSON file

Load model results from directory

Load pricing information from Hydra configuration

Load queries from JSONL file

Print statistics.

Reset token usage counter - implemented by specific implementation class

Save final task data.

Determine if the response indicates a retryable error. :param response: The HTTP response object. :return: True if the response indicates

Convert a span export dictionary to the OpenTelemetry format. :param span_export: The span export dictionary. :return: The span in OpenTe

assume using default trace provider, which generates trace_id in the format: input = f"trace_{uuid.uuid4().hex}"

Strategy 1: Try direct parsing, no fixing needed if successful

Strategy 4: Try legacy simple method

Strategy 2: Use line start pattern for parsing

Strategy 3: Use negative lookbehind pattern for parsing

Update cumulative token usage - OpenAI implementation

Update cumulative token usage - Anthropic implementation

Update cumulative token usage - OpenAI implementation

Update cumulative token usage - OpenAI Response API implementation

Update cumulative token usage - OpenAI implementation

Add a processor to the list of processors. Each processor will receive all traces/spans.

Assess baseline answer quality, filter tasks needing search augmentation. Args: tasks_with_baseline: Task list with base

Calculate final hierarchical weighted scores

Check if dependencies are installed

Close client connection

Generate criteria for all dimensions for a specific query

Create a new trace.

The default exporter, which exports traces and spans to the backend in batches.

The default processor, which exports traces and spans to the backend in batches.

Record the environment in the trace. This hydrdates `TaskTracer.env_info`.

Smart escaping and fixing: different processing based on key type Escaping strategy: - Basic escaping: double quotes, newlines and other

Evaluate all queries across all models

从 JSON 文件中提取 user_query 和 response Args: json_file_path: JSON 文件路径 Returns: Tuple[str, str]: (user_query, r

Extract tool call information - implemented by subclass

Filter tasks that require deep search. Args: tasks: List of tasks to filter Returns: Filtered task

Fix common errors in JSON strings

Fix common syntax errors in Python code

Fix common syntax errors in Shell commands

Forces an immediate flush of all queued spans.

Format final summary information, including answer and token statistics

Format token usage statistics and cost estimation, return summary_lines for format_final_summary and log string - implemented by subclass

Generate a new group identifier.

Generate a new span ID.

Generate a new trace ID.

Generate no-search baseline answers for tasks. Args: tasks: Task list Returns: Task list with basel

Generate specific criteria for a dimension

Generate hierarchical weights for all dimensions

Generate query-specific evaluation dimensions

Generate response

Returns the stored traces for a specific task and clears them.

Get basic task information

Get message list for specified session

Get sizes of all caches Returns: Dictionary mapping cache names to their sizes

Get execution summary information