- test-designer and release-reviewer keep a named model (Opus when none, never Haiku)
- verifier checks every T-id and N-id of a "## Test plan" section
- manager guidance: record each verdict with `ticket` before resume or rewrite
Already running on the homelab install (applied from ems-dev-tooling patches/model-router-0.5.0-pipeline.patch).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bdqr7PiaHcBnzPituu2yTC
The full suite run can time out files that pass alone (an overloaded
shared host, a suite-wide deadline in the runner): MAINT-28 T307 was
judged on 30 TIMEOUTs, T325 on a file that passes alone in 230 s.
- With testFileCommand set, every file the full run reports as TIMEOUT
or FAIL (testIssuePattern, default ^(TIMEOUT|FAIL)\s+(\S+); lines are
read as the run streams) is rerun alone in the clone,
testRerunJobs at a time, testFileTimeoutSec each, at most
testRerunMax files.
- A file that still fails or times out alone runs again on the base,
in a git worktree next to the clone (<clone>-verifier-base), which is
removed and pruned afterwards. Without a worktree, base stays "not
run".
- The verifier gets a table (file, full run, alone on head, on base,
meaning) and is told: passes alone does not count; fails on base too
is not caused by the change; name every file it counts.
- Only file names of letters, digits and _ . - / reach the shell; no
".." parts, no leading "-".
- 12 new tests; each of the 23 new rules was also broken on purpose and
caught.
Version 0.4.9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
Resuming an agent (SendMessage to a finished agent) raises agent.spawn
again, with resume.agentId set. The router treated every resume as a
new task: new Tn, fresh routing, the brief replaced by the resume
message, the failure count and round base reset. Hence T322's
"routed to Sonnet, ran on Opus" for an Opus agent that was only
resumed (#4), and verifiers that judged later rounds against the wrong
text (#6).
- A resume passes through untouched; its message is kept as a later
instruction to that agent. Task, routing, failures and rounds stay.
- A SendMessage to a tracked agent (by id or by the name its Agent call
gave it) that was delivered is kept as a later instruction too.
- The verifier prompt lists later instructions after the brief and
judges scope added there as in scope.
- [router-rewrite Tn]: the stored rewrite is used up only once its
agent has started (a failed start kept nothing before: T332 was then
routed afresh, to Haiku). With nothing stored the call is refused
instead of starting an agent with no brief. The task keeps Tn.
- The routing label compares with the model the engine resolved at
start, not the one the router asked for.
- 11 new tests; each new rule was also broken on purpose and caught.
Version 0.4.8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
A dev that waits on its own background work (a 35-45 min test suite)
ends several turns before it hands back. Every turn end posted a new
"verifying on Opus..." line and started the verifier on an unfinished
tree (MAINT-28 T307: 5 lines; T276/T278 judged an uncommitted tree).
- After each turn end of a tracked agent, its loop status decides:
completed, failed, killed, or already dropped by the engine is the
hand-back; waiting, idle, running or pending is an interim stop.
- An interim stop posts nothing and starts no verifier. The router
checks the status again every 15 s, up to 24 h, and runs the round
once the run has ended. A newer turn end takes the watch over.
- The hand-back uses the last turn's model and end reason.
- 8 new tests, including the ticket's acceptance (3 interim stops: one
line, one verdict, both after the hand-back); each new rule was also
broken on purpose and caught, apart from one redundant guard.
Version 0.4.7.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
An Opus-run dev got no verification and no message, so silence meant
both "still verifying" and "will never verify" (SAFETY-135 waited
overnight, SAFETY-126 8 h). Verdicts were also labelled with the
routing choice, not the model the run used.
- Every dev hand-back posts one line at once:
"[model-router] Tn verifying on Opus..." (a verdict follows), or
"[model-router] Tn no verification: dev ran on Opus. Go straight to
review." A run that ends with an error or names no clone says so.
- Each round is decided and labelled by the model the run reported
(its end usage), the routing choice only as fallback. A difference
is shown as "(routed to X, ran on Y)", also on later rounds.
- 8 new tests; each new rule was also broken on purpose and caught,
apart from one equivalent change.
Version 0.4.6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
The verdict reader matched from the first { to the last } in the whole
reply, so a fenced JSON verdict followed by prose containing a } failed
to parse, counted as FAIL and triggered a rewrite (SAFETY-129, T192).
- Read a ```json fenced block first, then the first balanced {...} that
parses and has a PASS or FAIL verdict. Braces inside strings are
ignored.
- An unreadable reply is re-asked once for the JSON only. Still
unreadable: NOT VERIFIED, "verdict unreadable", handed back to the
manager with the start of the first reply. No fail counted, no
resume, no rewrite.
- 10 new tests; each new rule was also broken on purpose and caught.
Version 0.4.5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo