Run work_fn for each model concurrently with live in-place progress. work_fn(model, set_status) -> result. set_status(text) updates that mode

Map 4-way verdict to a numeric score for tier accuracy. If no_cs_match is True (probe is for a name with no real CS researcher), REFUSAL is t

Map any verdict (3-way or 4-way) to a per-probe score.

Score a response using an LLM as judge. Args: question: The probe question gold_answer: The verified correct answer model

Get total result count from Brave Search API.

Get total result count from Google Custom Search JSON API.

Get total result count from SearchAPI.io (Google).

Get total result count from Serper.dev (Google).

Return a summary of the current store state.

Subsample probes for pilot, maintaining tier balance.

Test which backends are available and working.

Get or create a JSONL transcript file handle for a model.

Load the per-researcher evidence bundle (subfield + named systems + co-authors + top works + venues + affiliations) used by the 4-way judge. R

Number of terminal columns occupied by s, ignoring ANSI escapes.

Append one JSON record to the model's transcript file (thread-safe).

Analyze researcher probe results across models.

For each target model, find which calibration model it most resembles.

Compute the final assembled accuracy for one model. Returns dict or None if v1 results are missing.

Assign tier based on the smallest landmark that answers correctly.

Bootstrap confidence intervals for the aggregate log-linear fit.

Combine original probes with extended phrasings into fingerprint dataset.

Build the chat messages for a single probe phrasing.

Build probes from manually curated web-grounded data. Since we cannot run Google searches programmatically from this environment, we use a hy

Check if the answer leaks into the question.

Returns 'cs', 'noncs', or 'unknown'.

Compute Hallucination Similarity Score (HSS) for all model pairs. HSS(A, B) = |{p : A(p) == B(p) != gold(p)}| / |{p : both wrong}| Two model

Compute Fingerprint Similarity (FPS) for all model pairs. FPS(A, B) = Σ_t w_t × [observed_overlap_t / expected_overlap_t] Where expected ove

Compute inter-model variance for each probe. Returns probes sorted by variance (highest first). High-variance probes are the most diagnostic:

Combine FPS and HSS into a single provenance score.

Compute per-tier and aggregate accuracy from probe results.

Flag cross-vendor pairs with HSS above threshold.

Remove duplicate questions (keep first occurrence).

Return expected accuracy gain per month under Densing Law. Densing Law: at fixed accuracy, required params halve every DENSING_DOUBLING_MONTHS.

Display calibration curve info.

Display probe set for inspection.

Display evaluation results.

Pretty-print results table.

Estimate how many Google results mention this article title.

Estimate params from aggregate accuracy using calibration fit.

Estimate parameter count for a target (unknown) model. Args: target_results: dict with per_tier_accuracy and aggregate_accuracy c

Evaluate a single model on the 345 researcher probes.

Export a balanced final dataset by sampling from the valid pool. Two-pass sampling: 1. Sample up to per-source targets for each tier

Extract standardized CS field from OpenAlex author concepts.

Extract binary knowledge fingerprint vectors for each model. Returns: fingerprints: dict model_name -> dict probe_id -> bool (correct)

Heuristically extract named systems/artifacts from paper titles. Looks for ALLCAPS short tokens (2-8 chars) or CamelCase names that appear to

Fetch ALL author names from a DBLP conference proceedings page.

Check if predicted field matches or is adjacent to gold field. Returns True if: 1. Exact match (after normalization) 2. Gold and predicte

Find models with suspicious results.