A single reasoning-action-observation step.

A single evaluation axis with a 5-point scoring rubric. Each dimension is task-specific and generated dynamically. The ``weight`` field allow

Generates evaluation rubrics by prompting an LLM. The generator constructs a carefully engineered prompt that instructs the LLM to produce ta

Pass trajectories whose global score ≥ a fixed threshold. Parameters ---------- min_score : float Minimum acceptable global score

Score on a single dimension for a single trajectory step.

Weighted arithmetic mean across dimensions, uniform across steps. Per-dimension global score = mean of that dimension's scores across steps.

Evaluates trajectories by prompting an LLM with the rubric. Parameters ---------- client : LLMClient LLM backend for evaluation.

Per-step evaluation across all applicable dimensions.

Complete multi-step agent trajectory.

Top-level configuration for the AdaRubric pipeline.

Per-dimension minimum thresholds. A trajectory passes only if *every* dimension's global score meets its respective threshold. This prevents

Raised when LLM API interaction fails.

An agentic task whose trajectory will be evaluated.

Orchestrates rubric generation → trajectory evaluation → filtering. Parameters ---------- generator : RubricGenerator Produces ta

Task-adaptive evaluation rubric containing N dimensions. Generated by a ``RubricGenerator`` based on the task description. Typically contains

Complete evaluation result for one trajectory. Contains step-level scores, per-dimension global scores, an overall global score, and a surviv

Reward = score - baseline, centering rewards around zero. This produces positive rewards for above-average trajectories and negative rewards

Logical AND of multiple filters. A trajectory must pass ALL constituent filters to survive. Filters are applied in order; early rejection ski

Results from multi-run consistency analysis.

Generates (chosen, rejected) pairs from trajectory evaluations. For a batch of N evaluations, this produces up to N*(N-1)/2 pairs (all combin

Affine mapping: raw ∈ [raw_min, raw_max] → scaled ∈ [low, high]. Scores outside the raw range are clamped before scaling. Parameters ---

Keep the top-k percentile of trajectories by global score. Parameters ---------- percentile : float Percentile threshold in [0, 1

Assigns per-step dense rewards from step-level evaluations. Useful for RL training where credit assignment at each step matters (e.g., PPO wi

Raised when configuration is invalid or missing.

Weighted geometric mean — penalizes low-scoring dimensions. A single dimension scoring 1/5 will drag the overall score down much more than wi

Overall score = minimum per-dimension global score. Useful for safety-critical evaluations where one failing dimension should veto the entire

Collection of DPO pairs with metadata.

A (chosen, rejected) trajectory pair for DPO training.

Deterministic LLM client for testing. Returns pre-configured responses based on the response_model type.

LLM client for OpenAI-compatible APIs. Parameters ---------- model : str Model identifier (e.g. ``"gpt-4o"``). api_key : str

Raised when dynamic rubric generation fails.

Raised when trajectory evaluation fails.

Result of a full pipeline run.

Client for self-hosted vLLM inference servers. Parameters ---------- model : str Model identifier as served by vLLM. base_url

Base exception for all AdaRubric errors.

Computes global scores from step-level dimension scores.

Trajectory evaluator configuration.

Trajectory filter configuration.

Raised when trajectory filtering encounters an error.

Rubric generator configuration.

Protocol for LLM backends used by generators and evaluators.

LLM backend configuration.

Maps trajectory evaluations to scalar reward signals.

Generates a task-specific :class:`DynamicRubric` from a task description. Subclasses implement the actual generation logic (LLM-based, rule-based

Evaluates an agent trajectory against a dynamic rubric. Subclasses implement the scoring logic. The evaluator produces step-level scores, per

Decides which evaluated trajectories survive for downstream use. Filters implement the "Survival of the Fittest" mechanism, dropping low-qual