/evaluate-run - grade a factory run, feed the lessons back

You ran /factory (or did a complex task by hand) and it converged on a green PR. Green is not the
same as good: it can pass CI while bending a convention, mocking a test into meaninglessness,
cutting a corner you didn't notice, or repeating a mistake you already corrected. This skill judges
the run cold - from a compressed digest + the real diff, not the agent's own self-narration - and
turns what it finds into durable improvements so the next run is better.

Execute directly - do not enter plan mode. Read-only w.r.t. the code under evaluation; the only
things it may write (on your yes) are rules/guards/memories via the routing in the last step.

Step 1 - Resolve the target and squash it

The argument is the worktree (default: the current one) or an explicit transcript.jsonl. Produce a
compact digest of the session (how you export and compress a transcript depends on your harness).

The digest is compact markdown - prompts (with corrections flagged ⚠️), the diff, and the
ordered tool timeline - compressing a multi-hundred-KB transcript to a few KB. Read it. Also pull the
real diff for the evaluator to check against (git diff origin/<default-branch>...HEAD in that worktree, plus any
uncommitted/untracked). The digest tells you what happened; the diff is ground truth for whether it
was done right.

> Evaluating the current chat is fine, but the digest won't include the in-flight turn. For a clean
> read, evaluate a run that has come to rest (converged, or stopped).

Step 2 - Judge cold, in a fresh evaluator (a separate context)

Spawn a fresh subagent as the evaluator and hand it the digest + the diff. A cold
context is the point - the doer rationalizes its own choices; the evaluator must not inherit them. For a
thorough pass, fan out one evaluator per rubric group in parallel and merge; for a quick read, one
evaluator covering all groups is enough. Instruct the evaluator to open the actual changed files and
verify claims against the code and your conventions docs - never grade from the digest's summary alone.

The rubric (score each: ✅ pass / ⚠️ warn / ❌ fail, with file:line or digest evidence)

• Followed the loop. triage → (/design + go/no-go, if non-trivial) → implement → review-loop →

(e2e-verify, if the change has a runtime/UI surface) → review-loop → /babysit-pr → converge. Which steps ran, which were skipped, and was each skip justified by the triage - a localized fix skipping design is correct, a UI change skipping e2e is not. Did skipping anything matter (e.g. a cross-cutting change built without a design, called a green CI "done" without a clean re-review)?

• Honored the conventions (your conventions docs): type

safety (no as/any/!/@ts-ignore), minimal surgical change at the right layer, searched-before- creating (no near-duplicate of an existing util), comments-default-to-none, the perf/batching rule on hot paths, project-specific conventions. Flag anything the guards/reviewers didn't catch.

• Tests are real. DB-backed where your rules require it (no mocking the thing under test into a

tautology; exercises the change against real dependencies); the test actually exercises the change and would fail without it - not a tautology or a snapshot of fixtures.

• Judgment: ask vs proceed. Did it stop for a genuine decision and not stop for trivia? Over-ask

(interrogated a clear ask) or under-ask (guessed on a real fork and built the wrong thing)?

• Corrections handled. Identify corrections semantically - the digest pre-tags the obvious

ones ⚠️, but that's a regex hint; you read the prompts, so also count the calmly-phrased redirects it missed ("use X instead of Y", "make it Z"). For each: did the run (a) fix the instance and (b) capture it durably (a rule/guard or a memory)? Did it repeat a mistake it was already corrected on? An uncaptured correction is a process failure even if the instance was fixed.

• Verification was honest. It actually ran your gate (format/lint/typecheck/test) and reported

truthfully - no "done" without evidence, no claimed-passing that the diff contradicts.

• Efficiency. Thrash, re-reading the same file, redundant tool calls, or a subagent that could have

parallelized - cheap signal, low weight.

The evaluator returns: a per-dimension score with evidence, the top 3-5 issues ranked by severity,
and for each a proposed durable fix tagged RULE (mechanizable/code convention) or PROCESS
(how-you-work). Bias toward specific, evidenced findings over vague ones; "this looked fine" is not a
finding.

Step 3 - Present the report and route the lessons

Merge the evaluator output into one report for the user: the scorecard, the ranked issues with
evidence, and the proposed durable fixes. Then route each accepted lesson (act only on the user's
yes, exactly as /add-rule requires):

• RULE findings → hand to /add-rule: a conventions-doc rule + a lint guard when

mechanizable. A convention the run broke that a guard could have caught is the highest-value output of the whole exercise - it converts a one-time miss into a permanent floor.

• PROCESS findings → a persistent memory (with the why + how-to-apply), and the root

conventions doc if it's repo-wide.

End with the one number that matters over time: how many corrections/new-rules this run produced. The
whole thesis is that this trends down run over run as the environment absorbs the lessons - track it,
call it out, and note whether this run beat the last.

Guardrails

• Cold and adversarial. Judge from the diff + digest, not the agent's self-praise. If the evaluator

finds nothing, say so plainly - don't manufacture issues to look thorough. A genuinely clean run is the goal, and it should happen more as the system learns.

• Read-only on the code under review. The only writes are the routed rules/guards/memories, on the

user's yes.

• Don't re-litigate green CI. CI already proved it compiles and passes. Your job is the quality CI

can't see: conventions, test integrity, judgment, and whether corrections stuck.