* Shared planted-bug eval runner. * Gives the agent concise bug-finding instructions (not the full QA workflow), * then scores the report with an LLM outcome judge.
(fixture: string, groundTruthFile: string, label: string)
| 50 | * then scores the report with an LLM outcome judge. |
| 51 | */ |
| 52 | async function runPlantedBugEval(fixture: string, groundTruthFile: string, label: string) { |
| 53 | // Each test gets its own isolated working directory to prevent cross-contamination |
| 54 | // (agents reading previous tests' reports and hallucinating those bugs) |
| 55 | const testWorkDir = fs.mkdtempSync(path.join(os.tmpdir(), `skill-e2e-${label}-`)); |
| 56 | setupBrowseShims(testWorkDir); |
| 57 | const reportDir = path.join(testWorkDir, 'reports'); |
| 58 | fs.mkdirSync(path.join(reportDir, 'screenshots'), { recursive: true }); |
| 59 | const reportPath = path.join(reportDir, 'qa-report.md'); |
| 60 | |
| 61 | // Direct bug-finding with browse. Keep prompt concise — no reading long SKILL.md docs. |
| 62 | // "Write early, update later" pattern ensures report exists even if agent hits max turns. |
| 63 | const targetUrl = `${testServer.url}/${fixture}`; |
| 64 | const result = await runSkillTest({ |
| 65 | prompt: `Find bugs on this page: ${targetUrl} |
| 66 | |
| 67 | Browser binary: B="${browseBin}" |
| 68 | |
| 69 | PHASE 1 — Quick scan (5 commands max): |
| 70 | $B goto ${targetUrl} |
| 71 | $B console --errors |
| 72 | $B snapshot -i |
| 73 | $B snapshot -c |
| 74 | $B accessibility |
| 75 | |
| 76 | PHASE 2 — Write initial report to ${reportPath}: |
| 77 | Write every bug you found so far. Format each as: |
| 78 | - Category: functional / visual / accessibility / console |
| 79 | - Severity: high / medium / low |
| 80 | - Evidence: what you observed |
| 81 | |
| 82 | PHASE 3 — Interactive testing (targeted — max 15 commands): |
| 83 | - Test email: type "user@" (no domain) and blur — does it validate? |
| 84 | - Test quantity: clear the field entirely — check the total display |
| 85 | - Test credit card: type a 25-character string — check for overflow |
| 86 | - Submit the form with zip code empty — does it require zip? |
| 87 | - Submit a valid form and run $B console --errors |
| 88 | - After finding more bugs, UPDATE ${reportPath} with new findings |
| 89 | |
| 90 | PHASE 4 — Finalize report: |
| 91 | - UPDATE ${reportPath} with ALL bugs found across all phases |
| 92 | - Include console errors, form validation issues, visual overflow, missing attributes |
| 93 | |
| 94 | CRITICAL RULES: |
| 95 | - ONLY test the page at ${targetUrl} — do not navigate to other sites |
| 96 | - Write the report file in PHASE 2 before doing interactive testing |
| 97 | - The report MUST exist at ${reportPath} when you finish`, |
| 98 | workingDirectory: testWorkDir, |
| 99 | maxTurns: 50, |
| 100 | timeout: 300_000, |
| 101 | testName: `qa-${label}`, |
| 102 | runId, |
| 103 | model: 'claude-opus-4-7', |
| 104 | }); |
| 105 | |
| 106 | logCost(`/qa ${label}`, result); |
| 107 | |
| 108 | // Phase 1: browse mechanics. Accept error_max_turns — agent may have written |
| 109 | // a partial report before running out of turns. What matters is detection rate. |
no test coverage detected