A critic for performing exact equality comparisons between expected and actual values. This critic evaluates whether the expected and actual

Value schema for input parameters and outputs.

A suite for evaluating AI model performance on specific tasks or scenarios. EvalSuite manages a collection of EvalCases, each representing a

Represents an expected tool call identified by tool name (string). Use this for: - Tools loaded from MCP servers (local stdio or remote

A FastAPI-like interface for building MCP servers. The app collects tools and configuration, then lazily creates the server and transpor

OAuth 2.1 Resource Server with discovery metadata support. This class implements the MCP server's role as an OAuth 2.1 Resource Server, valid

Singleton class that holds all tools for a given worker

Mock EvaluationResult for testing.

A parameter that can be passed to a tool.

Defines the rubric for evaluating an AI model's performance on a task. Attributes: fail_threshold: The minimum score required to pas

A critic for evaluating the similarity between two strings. This critic uses a specified similarity metric to compare the expected and actua

The requirements for a tool to run.

A minimal internal registry that stores tools in MCP-style descriptors: { "name": "...", "description": "...",

Data structure that holds tool information while stored in the Catalog

Metadata for a tool once it's been materialized.

HTML formatter for evaluation results. Produces a styled HTML document with colors matching the terminal output. Security Note: All use

The context for a tool invocation. This type is transport-agnostic and contains only authorization, secret, and metadata information needed b

The output of a tool.

The specification of a toolkit.

Markdown formatter for evaluation results. Produces a well-structured Markdown document with tables and collapsible sections.

The specification of a tool.

The inputs that a tool accepts.

Represents the result of an evaluation case. Attributes: score: The normalized evaluation score (0.0-1.0). passed: Whether t

Represents a tool call with its name and arguments. Attributes: name: The name of the tool. args: A dictionary containing th

Primary context interface for MCP tools. This class provides both runtime capabilities (logging, resources, prompts, etc.) and tool-specific

Manages resources for the MCP server.

Configuration entry for a single authorization server. Each authorization server that can issue valid tokens for this MCP server (Resource Se

Plain text formatter for evaluation results. Produces output similar to pytest's format with simple ASCII formatting.

MCP Server with middleware and context support. This server provides: - Middleware chain for extensible request processing - Context

Represents a single evaluation case within an EvalSuite. Attributes: name: A descriptive name for this evaluation case. syst

Tracks CLI command execution for usage analytics.

Represents an expected tool call for a Python tool (registered via ToolCatalog). Use this for Python functions decorated with @tool. At

JSON formatter for evaluation results. Produces a structured JSON document containing all evaluation data, suitable for programmatic pro

JWKS-based JWT token validator for simple, explicit token validation. This validator fetches public keys from a JWKS endpoint and validates J

User information extracted from validated access token. This represents the authenticated resource owner (end-user) making requests to the MC

Result of running a single case in capture mode. Attributes: case_name: The name of the evaluation case. user_message: The u

The output of a tool invocation.

Error from an upstream service/API during tool execution. This class handles all upstream failures except rate limiting. The status_code

Plain text formatter for capture results.

A critic for evaluating numeric values within a specified range. This critic performs a "fuzzy" comparison of numeric values, where values c

Context passed through the middleware chain. Contains the message being processed and metadata about the request.

Manages the lifecycle of MCP tasks. Args: max_retention: Maximum TTL ceiling in milliseconds. None = no ceiling (clients may

A requirement for a tool to run.

Server-related settings.

Markdown formatter for capture results.

A captured tool call from the model during capture mode. Attributes: name: The name of the tool that was called. args: The a

The fully-qualified name of a tool.

A specific model to run evaluations against. This is the expanded form used by the runner - one ModelSpec per (provider, model, api_key) comb

Container for metadata about a tool. - classification: What type of service does this tool interface with? (for discovery/boosting) - be

Raised when there is an error in the definition/signature of a tool.

RFC 6750 ``insufficient_scope`` error. Subclasses ``Exception`` directly (NOT ``AuthenticationError``): 400-class auth errors are the 401 pat

ASGI middleware that validates Bearer tokens on every HTTP request. Validates tokens per MCP specification: - Checks Authorization header for

Stdio transport implementation for stdio communication. This transport uses stdin/stdout for MCP communication, suitable for command-lin

Raised while importing / loading a toolkit package (e.g. missing dependency, SyntaxError in module top-level code).

HTML formatter for capture results.

JSON formatter for capture results.

A list of content objects that represent the unstructured result of the tool call.

Icon metadata (MCP 2025-11-25).

MCP server session handling a single client connection. Manages: - Session state and lifecycle - Client capabilities - Request/r

HTTP transport with SSE streaming support for MCP. Responsibilities - Parse HTTP requests into JSON-RPC messages and enqueue them on the

Token is invalid (signature, audience, issuer, etc.).

Main MCP settings container.

How a tool can be executed (MCP 2025-11-25 task augmentation).

The name and version of a tool.

Result of running an EvalSuite in capture mode. Attributes: suite_name: The name of the evaluation suite. model: The model u

Requested entity not found.

Error in server operations.

A requirement for authorization to use a tool.

A case that runs against multiple tracks for comparison. Shared context (messages) is defined once, while each track has its own expected too

The error that occurred during the tool invocation.

What effects does the tool have? Arcade's data model for tool behavior. When using MCP, Behavior is projected to MCP annotations: - read

Mock Typer context for testing.

Task not found or context mismatch (same error for both -- no info leak).

Configuration for a single provider from CLI input. Parsed from --use-provider flag values like: - "openai" -> provider=OPENAI, models=[] (us

Error in session management

The request to call (invoke) a tool.

Manages named tracks, each with its own isolated tool registry. Tracks enable comparative evaluations where the same cases are run against di

Manages HTTP streaming sessions with optional resumability. This class abstracts session management, event storage, and request handling for

Mock EvaluationResult for testing.

A critic that has no effect on the evaluation results and does not actually evaluate. If a critic is not found for an evaluation case's fiel

A simple typed dictionary.

Raised when there is an error in the schema of a tool's input parameter.

Rate limit error from an upstream service. Special case of UpstreamError that includes retry_after_ms information.