- test-designer and release-reviewer keep a named model (Opus when none, never Haiku)
- verifier checks every T-id and N-id of a "## Test plan" section
- manager guidance: record each verdict with `ticket` before resume or rewrite
Already running on the homelab install (applied from ems-dev-tooling patches/model-router-0.5.0-pipeline.patch).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bdqr7PiaHcBnzPituu2yTC
The full suite run can time out files that pass alone (an overloaded
shared host, a suite-wide deadline in the runner): MAINT-28 T307 was
judged on 30 TIMEOUTs, T325 on a file that passes alone in 230 s.
- With testFileCommand set, every file the full run reports as TIMEOUT
or FAIL (testIssuePattern, default ^(TIMEOUT|FAIL)\s+(\S+); lines are
read as the run streams) is rerun alone in the clone,
testRerunJobs at a time, testFileTimeoutSec each, at most
testRerunMax files.
- A file that still fails or times out alone runs again on the base,
in a git worktree next to the clone (<clone>-verifier-base), which is
removed and pruned afterwards. Without a worktree, base stays "not
run".
- The verifier gets a table (file, full run, alone on head, on base,
meaning) and is told: passes alone does not count; fails on base too
is not caused by the change; name every file it counts.
- Only file names of letters, digits and _ . - / reach the shell; no
".." parts, no leading "-".
- 12 new tests; each of the 23 new rules was also broken on purpose and
caught.
Version 0.4.9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
A dev that waits on its own background work (a 35-45 min test suite)
ends several turns before it hands back. Every turn end posted a new
"verifying on Opus..." line and started the verifier on an unfinished
tree (MAINT-28 T307: 5 lines; T276/T278 judged an uncommitted tree).
- After each turn end of a tracked agent, its loop status decides:
completed, failed, killed, or already dropped by the engine is the
hand-back; waiting, idle, running or pending is an interim stop.
- An interim stop posts nothing and starts no verifier. The router
checks the status again every 15 s, up to 24 h, and runs the round
once the run has ended. A newer turn end takes the watch over.
- The hand-back uses the last turn's model and end reason.
- 8 new tests, including the ticket's acceptance (3 interim stops: one
line, one verdict, both after the hand-back); each new rule was also
broken on purpose and caught, apart from one redundant guard.
Version 0.4.7.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
An Opus-run dev got no verification and no message, so silence meant
both "still verifying" and "will never verify" (SAFETY-135 waited
overnight, SAFETY-126 8 h). Verdicts were also labelled with the
routing choice, not the model the run used.
- Every dev hand-back posts one line at once:
"[model-router] Tn verifying on Opus..." (a verdict follows), or
"[model-router] Tn no verification: dev ran on Opus. Go straight to
review." A run that ends with an error or names no clone says so.
- Each round is decided and labelled by the model the run reported
(its end usage), the routing choice only as fallback. A difference
is shown as "(routed to X, ran on Y)", also on later rounds.
- 8 new tests; each new rule was also broken on purpose and caught,
apart from one equivalent change.
Version 0.4.6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
The verdict reader matched from the first { to the last } in the whole
reply, so a fenced JSON verdict followed by prose containing a } failed
to parse, counted as FAIL and triggered a rewrite (SAFETY-129, T192).
- Read a ```json fenced block first, then the first balanced {...} that
parses and has a PASS or FAIL verdict. Braces inside strings are
ignored.
- An unreadable reply is re-asked once for the JSON only. Still
unreadable: NOT VERIFIED, "verdict unreadable", handed back to the
manager with the start of the first reply. No fail counted, no
resume, no rewrite.
- 10 new tests; each new rule was also broken on purpose and caught.
Version 0.4.5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo