(gt, ans)
| 32 | } |
| 33 | |
| 34 | const e2ePrompt = (gt, ans) => `You are scoring whether an AI coding agent correctly answered a code-flow question about a repository. Judge ONLY against the verified ground truth. Do NOT use any tools. |
| 35 | |
| 36 | QUESTION: ${gt.question} |
| 37 | |
| 38 | VERIFIED GROUND TRUTH (the actual call path + files): |
| 39 | ${gt.truth} |
| 40 | |
| 41 | AGENT'S ANSWER: |
| 42 | ${ans || '(empty)'} |
| 43 | |
| 44 | Score how correct the agent's answer is vs the ground truth. A "pass" means it identifies the core mechanism and the major hops with the right files/symbols and makes no materially wrong claim. "partial" = right area but misses major hops or has notable errors. "fail" = wrong layer, fabricated, or misses the mechanism. |
| 45 | Output ONLY minified JSON, no prose, no code fences: |
| 46 | {"verdict":"pass|partial|fail","score":<0-100>,"missedHops":["..."],"wrongClaims":["..."],"note":"<=20 words"}`; |
| 47 | |
| 48 | const fidPrompt = (gt, ans) => `You are scoring the FIDELITY of a machine-synthesized code-exploration answer against verified ground truth. The synthesized answer claims to trace a flow and cite file:line locations. Do NOT use any tools. |
| 49 |
no outgoing calls
no test coverage detected