Create judge function using OpenRouter.

Create a function that queries the model.

Compute mean per-tier penalized score for a model. For researcher probes (T3-T7): use data/results_v2/<model>.json scores (CORRECT_STRONG

Analyze MoE models: total vs active params.

Return pairs dict with metrics.

Patch failed probes in the existing v2 result file.

Recompute mean per-tier penalized accuracy under a given penalty.

Figure 1: Aggregate accuracy vs log(params) with calibration fit.

Two-panel figure: (a) per-family consecutive-pair HSS with family bands; (b) Jaccard vs HSS scatter for the whole pairwise universe.

Plot distribution of FPS scores, highlighting known derived vs independent pairs.

Plot pairwise fingerprint similarity heatmap.

Figure 3: LOO-CV predicted vs actual parameter counts.

Plot per-tier overlap ratios for notable pairs.

Figure 4: Frontier model parameter estimates with CIs.

Figure 5: Per-tier accuracy heatmap for all models.

Figure 2: Per-tier accuracy sigmoids.

How does estimation quality degrade with fewer probes per tier?

Query a landmark model and return the cleaned response.

Query a model via OpenRouter or local Ollama (OpenAI-compatible API). Models with "ollama/" prefix are routed to the local Ollama server. Bot

Query OpenRouter or local Ollama. Returns dict with response, attempts, error.

Query OpenRouter with aggressive 429 handling.

Query local Ollama model.

Query a local Ollama model.

Query OpenAlex for a single author. Returns dict or None.

Query OpenAlex for one author. Returns dict or None.

Query OpenAlex for authors in a citation range (filter CS client-side).

Query OpenRouter directly via httpx (no rate limiter).

Query OpenRouter model with retries.

Refit calibration on open models with known params at the given lambda.

Re-evaluate a single model.

Calibrate all pending probes with high parallelism.

Run Chinese probes against a single model.

Run full calibration pipeline. Args: calibration_data: list of dicts, each with: - model_name: str - params_billi

Leave-one-out cross-validation.

Run search counts on all valid probes in the store.

Run a single probe against a model and score the result.

Run researcher probes and collect detailed response analysis.

Run a single probe (all 3 phrasings) against a model. Returns a result dict with raw responses and scoring.

Select the top-N most diagnostic probes, balanced across T5-T7.

Remove <think>...</think> blocks from thinking model output.