Run-to-run (within-prompt) stddev, pooled across prompts. The headline ± must reflect measurement *noise* — how much a single prompt's number wobbles between identical reps — NOT how much different prompts differ from each other. Pooling every raw sample across prompts would fold the
(runs, key)
| 586 | |
| 587 | |
| 588 | def pooled_within_stddev(runs, key): |
| 589 | """Run-to-run (within-prompt) stddev, pooled across prompts. |
| 590 | |
| 591 | The headline ± must reflect measurement *noise* — how much a single prompt's |
| 592 | number wobbles between identical reps — NOT how much different prompts differ |
| 593 | from each other. Pooling every raw sample across prompts would fold the |
| 594 | (large, real, repeatable) prompt-to-prompt spread into the stddev and badly |
| 595 | overstate the noise. We instead pool only the per-prompt deviations: |
| 596 | |
| 597 | sqrt( Σ_prompt Σ_rep (x - prompt_mean)^2 / Σ_prompt (n_rep - 1) ) |
| 598 | |
| 599 | i.e. the standard pooled within-group standard deviation. Prompt-to-prompt |
| 600 | spread stays visible via the min–max line and the per-prompt table. |
| 601 | """ |
| 602 | num, den = 0.0, 0 |
| 603 | for r in runs: |
| 604 | s = [x for x in r.get("samples", {}).get(key, []) if x is not None] |
| 605 | if len(s) > 1: |
| 606 | m = statistics.mean(s) |
| 607 | num += sum((x - m) ** 2 for x in s) |
| 608 | den += len(s) - 1 |
| 609 | return (num / den) ** 0.5 if den > 0 else 0.0 |
| 610 | |
| 611 | |
| 612 | def main(): |