run-evals
Run Evals
Section titled “Run Evals”HARD GATE — Define evals before implementation. Code graders = runnable
verify:commands; model graders = explicit rubric with pass/fail criteria.
Process
Section titled “Process”-
Name the capability under test (one sentence).
-
Write
specs/EVALS-<feature>.mdwith:- Capability evals (does it do the job?)
- Regression evals (did we break anything?)
-
Assign grader type per eval:
code(shell verify) ormodel(rubric). -
Assign strictness tier per eval (graduated promotion — e45s37):
Tier Meaning Promotion rule EXPERIMENTALNew eval, may flake Not gating USUALLY_PASSESStable in dev; ≥2/3 recent runs pass Blocks BUILD only when combined with ALWAYS_PASSES suite ALWAYS_PASSESZero tolerance; required for release Any single failure blocks BUILD and merge Promote:
EXPERIMENTAL → USUALLY_PASSESafter 3 consecutive passes;USUALLY_PASSES → ALWAYS_PASSESafter 5 consecutive passes with zero flakes documented inspecs/state.yaml. -
Run evals; log results table with pass@k (e.g. 3/3 runs) and tier per eval.
-
Block BUILD phase until all
ALWAYS_PASSESevals pass at agreed k.USUALLY_PASSESfailures warn;EXPERIMENTALfailures log only.
Artefact
Section titled “Artefact”specs/verifications/eNNsYY-eval-report.md — see REFERENCE.md for template. Eval reports are stored alongside verification evidence in specs/verifications/, keyed by story ID for traceability.
Verify
Section titled “Verify”→ verify: find specs/verifications -name "*-eval-report.md" | wc -l | awk '{if($1>0) print "OK: "$1" eval reports"; else print "MISSING"}'
<!– story: e02s01 –>