Skip to content

run-evals

HARD GATE — Define evals before implementation. Code graders = runnable verify: commands; model graders = explicit rubric with pass/fail criteria.

  1. Name the capability under test (one sentence).

  2. Write specs/EVALS-<feature>.md with:

    • Capability evals (does it do the job?)
    • Regression evals (did we break anything?)
  3. Assign grader type per eval: code (shell verify) or model (rubric).

  4. Assign strictness tier per eval (graduated promotion — e45s37):

    Tier Meaning Promotion rule
    EXPERIMENTAL New eval, may flake Not gating
    USUALLY_PASSES Stable in dev; ≥2/3 recent runs pass Blocks BUILD only when combined with ALWAYS_PASSES suite
    ALWAYS_PASSES Zero tolerance; required for release Any single failure blocks BUILD and merge

    Promote: EXPERIMENTAL → USUALLY_PASSES after 3 consecutive passes; USUALLY_PASSES → ALWAYS_PASSES after 5 consecutive passes with zero flakes documented in specs/state.yaml.

  5. Run evals; log results table with pass@k (e.g. 3/3 runs) and tier per eval.

  6. Block BUILD phase until all ALWAYS_PASSES evals pass at agreed k. USUALLY_PASSES failures warn; EXPERIMENTAL failures log only.

specs/verifications/eNNsYY-eval-report.md — see REFERENCE.md for template. Eval reports are stored alongside verification evidence in specs/verifications/, keyed by story ID for traceability.

→ verify: find specs/verifications -name "*-eval-report.md" | wc -l | awk '{if($1>0) print "OK: "$1" eval reports"; else print "MISSING"}'

<!– story: e02s01 –>