Create a minimal TrajectoryEvaluation for filter testing.

Execute the full three-stage pipeline. Parameters ---------- task : TaskDescription The task being evaluated.

Return only the evaluations that pass the filter criteria. This method also sets ``passed_threshold`` on each evaluation.

Compute Krippendorff's alpha for reliability estimation. Parameters ---------- ratings : np.ndarray Shape (n_raters, n_items). Us

Completion token budget for trajectory evaluation (long JSON).

Return the first JSON object or array substring from ``text``. Strips common markdown fences (``json ... ```) then scans for balanced ``{...}

Completion token budget for rubric generation.

Compute a reward for each step in the trajectory. Returns ------- list[float] Per-step reward values, ordered by

Load config from a JSON file, with env var override for api_key.

Generate DPO pairs from a batch of evaluations. Trajectories are compared pairwise. The higher-scoring one becomes "chosen" and the l

Save config to a JSON file. By default ``llm.api_key`` is omitted so keys are not committed accidentally.

Override sensitive fields from environment variables. Returns a new config.

Chat completion with exponential backoff on transient errors.

Score a trajectory against the given rubric. Parameters ---------- trajectory : Trajectory The agent's multi-step

Stage 1: Generate a dynamic rubric for the task.

Generate a response and parse it into ``response_model``. Implementations should handle retries and JSON extraction internally.

Load trajectories from a JSONL file.

Convert parsed LLM output into the canonical evaluation model.

Post-generation sanity checks; returns rubric with task_id corrected if needed.

Release any held resources (connection pools, etc.).

Stage 2 (batch): Evaluate multiple trajectories concurrently.

Export DPO pairs in HuggingFace-compatible JSONL format. Each line contains: - chosen: trajectory_id of the preferred trajectory - reject

Stage 3: Apply survival-of-the-fittest filtering.

Render trajectory steps as readable text for the LLM prompt.

Build a pipeline from a configuration object.

Generate a dynamic rubric for the given task. Parameters ---------- task : TaskDescription The agentic task to cr

Load evaluations from a JSONL file.

Write evaluations to a JSONL file.

Write trajectories to a JSONL file.

Convert evaluations into reward values. Parameters ---------- evaluations : list[TrajectoryEvaluation] Evaluated

Parameters ---------- recency_decay : float Exponential decay factor for step weighting. 0.0 = uniform weights.

Aggregate step-level scores into global scores. Returns ------- tuple[dict[str, float], float] (per-dimension glo

Stage 2: Evaluate a single trajectory against a rubric.

Evaluate multiple trajectories concurrently with bounded parallelism.

Evaluate multiple trajectories against the same rubric. Default implementation is sequential; subclasses may override with concurrent

Run evaluation N times and compute inter-rater reliability. Parameters ---------- evaluator : TrajectoryEvaluatorBase The evaluat

Load config from a YAML file (requires ``pyyaml``).

Generate a plain-text response.