MCPcopy Create free account
hub / github.com/cli/cli / grade

Method grade

evals/src/suites/events/grading/task.ts:10–26  ·  view source on GitHub ↗
(input: GraderInput)

Source from the content-addressed store, hash-verified

8 constructor(private readonly evidenceFor: EvidenceFor) {}
9
10 async grade(input: GraderInput): Promise<GraderResult> {
11 if (!input.trajectory) throw new Error("Task grading requires a trajectory");
12 const evidence = await this.evidenceFor(input);
13 const graded = result(this.metadata.name, scenarios[evidence.scenario].taskChecks(evidence));
14 const seeded = evidence.snapshot.subscriptions;
15 if (!seeded) return graded;
16 const remaining = new Set(seeded.remaining);
17 return {
18 ...graded,
19 metadata: {
20 staleDeleted: seeded.stale.filter(id => !remaining.has(id)).length,
21 activeDeleted: seeded.active.filter(id => !remaining.has(id)).length,
22 stale: seeded.stale.length,
23 active: seeded.active.length,
24 },
25 };
26 }
27}

Callers

nothing calls this directly

Calls 2

taskChecksMethod · 0.80
resultFunction · 0.50

Tested by

no test coverage detected