Analyzes an incident using the specified LLM. Returns the parsed LLMOutput and the latency in seconds. Supported: - 'gpt-4-turbo'

Apply dark theme styling to an axes.

Exact match of LLM predicted category against ground truth. Returns True if they match (case-insensitive).

Classify a (model, incident) run into one of five behavioral regimes. (benchmark_spec.md § 6.2)

Rule-Based Deterministic Faithfulness Scorer (from benchmark_spec.md § 5) Does NOT use an LLM judge to avoid circular evaluation when GPT-4

Extract confidence score from LLM output.

Return P1/P2/P3 severity (1=P1) from chaos scenario name.

Safety check: ensures commands are valid kubectl/helm/aws commands that don't contain destructive patterns. Returns True if all commands

Normalise chaos_type to canonical scenario string.

Load raw telemetry files for an incident.

Command Executability Score — novel metric. Dry-runs each kubectl command using `--dry-run=client`. Returns the fraction of commands tha

P1 Severity Risk — novel safety metric (from findings_and_safety.md). Identifies the most dangerous operational failure mode: - P1 inc

Robustly extracts a JSON dict from a raw string that may have leading/trailing conversational text outside the JSON block.

Consumes an OpenAI-compatible streaming response from NVIDIA NIM. Returns the assembled content string. Handles both standard content chu

Group by model, compute means for all numeric columns.

Bootstrap confidence interval for the mean of a list of values.

Build IncidentInput for condition A (full) or D (metadata only).

Condition X — metadata from incident A + logs from incident B (contradicting).

Build an IncidentInput with telemetry blanked according to the condition. strict=True: for Condition D (or D-strict), strips LABEL_LEAK_FIEL

Build prompt for novel incident under condition A, D, or G.

Condition D — metadata only, telemetry blanked.

Condition A — full telemetry (logs consistent with label).

Reliability diagram: confidence bins (x-axis) vs actual accuracy (y-axis). Perfect calibration = diagonal line. Models scattered above/b

Generic NVIDIA NIM caller. Works with any model available at integrate.api.nvidia.com. model_id should be the canonical NVIDIA model slug, e.

Compare Condition D (with label fields) vs D-strict (label fields stripped). If accuracy drops when labels are stripped, that proves the labe

Run conditions A and D, compute all EI metrics for one incident.

Pull any kubectl-looking strings from freeform text.

Split root_cause_description into sentence-level claims to use as a proxy for evidence_cited in old rows.

Augment the leaderboard with 95% bootstrap CIs on rca_accuracy. Returns a DataFrame ready for the paper table.

Load raw telemetry files for an incident.

Load raw telemetry files for an incident.

Load all raw text files for an incident into a dict.

For each (incident, model) pair, compare accuracy in Condition A vs D. McNemar's test asks: is the drop from A→D significant? Discordant

Check if the model output references content from the log files.

Fill in missing metric columns for a single CSV row.

Run 4 conditions for a given incident + model pair. strict=True: runs Condition D with label fields stripped (D-strict). Results are labelled

Run the 3-condition Bayesian violation test for a single incident × model.

Run all novel incidents through 3 conditions.

Binary hallucination penalty: fraction of cited evidence that does NOT appear verbatim anywhere in the raw input logs. Returns float in

Substring-based log faithfulness — legacy scorer (used when raw telemetry text is not available separately, only the combined input_data dict

Print Bayesian violation summary.

Print ECS summary per model.

Print novel vs familiar ECS comparison.

Print a 4-condition accuracy table for each (incident, model).

Print confidence calibration comparison: instruct vs base, hits vs misses, per family.

Scatter: faithfulness (x) vs accuracy (y) — reveals 4 behavioral regimes.

Shows r ≈ −0.124 visually — confidence predicts NOTHING about faithfulness.

Bar chart of P1 Severity Risk Rate per model — the safety metric.

Pie chart showing distribution of behavioral regimes across all runs.

Grouped bar — accuracy across 4 telemetry conditions. The causal proof.