6 Commits
Author SHA1 Message Date
adminandClaude Opus 5.5 c88884c54a Fix #7: non-dev roles (test-design) are never treated as a dev (0.5.1)
A test-design brief (Pipeline-Role: test-design, Clone: ... (Role: test-design))
that mentioned the dev clone was read as a dev in that clone: it got a T number
and a "no verification: dev ran on Opus" line, also after a resume.

- Role comes from Pipeline-Role:, then (Role: ...) on the Clone: line, then
  the folder name, which must end in -dev, -review or -test. Any other role,
  or a Clone: line the router cannot place, is a non-dev role.
- A non-dev Pipeline-Role never falls back to other clone paths in the brief.
- Non-dev roles and the 0.5.0 gate types (test-designer, release-reviewer):
  named model kept, never Haiku, Opus when none; no hand-back line, no
  verification, no savings record. A gate agent type is never a dev, whatever
  clone paths its brief names.
- T numbers for dev agents only; other agents get A1, A2, ...
- /router log shows the role as written (test-design, test-designer).
- 0.5.0 test updated: a -design clone is now a non-dev role, not "no role".

102 tests; 16 mutations of the new rules, all caught.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
2026-11-17 21:47:25 +01:00
admin 3cd69762be Merge pull request '0.5.0: pipeline support for ems-dev-tooling' (#8) from pipeline-0.5.0 into main
Reviewed-on: #8
2026-11-17 21:29:23 +01:00
glenn schrooyenandClaude Opus 5.5 86bb9d79cc 0.5.0: pipeline support for ems-dev-tooling
- test-designer and release-reviewer keep a named model (Opus when none, never Haiku)
- verifier checks every T-id and N-id of a "## Test plan" section
- manager guidance: record each verdict with `ticket` before resume or rewrite

Already running on the homelab install (applied from ems-dev-tooling patches/model-router-0.5.0-pipeline.patch).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bdqr7PiaHcBnzPituu2yTC
2026-11-17 21:28:36 +01:00
adminandClaude Opus 5.5 d5c762b339 Fix #5: rerun reported files alone, and on base, before judging
The full suite run can time out files that pass alone (an overloaded
shared host, a suite-wide deadline in the runner): MAINT-28 T307 was
judged on 30 TIMEOUTs, T325 on a file that passes alone in 230 s.

- With testFileCommand set, every file the full run reports as TIMEOUT
  or FAIL (testIssuePattern, default ^(TIMEOUT|FAIL)\s+(\S+); lines are
  read as the run streams) is rerun alone in the clone,
  testRerunJobs at a time, testFileTimeoutSec each, at most
  testRerunMax files.
- A file that still fails or times out alone runs again on the base,
  in a git worktree next to the clone (<clone>-verifier-base), which is
  removed and pruned afterwards. Without a worktree, base stays "not
  run".
- The verifier gets a table (file, full run, alone on head, on base,
  meaning) and is told: passes alone does not count; fails on base too
  is not caused by the change; name every file it counts.
- Only file names of letters, digits and _ . - / reach the shell; no
  ".." parts, no leading "-".
- 12 new tests; each of the 23 new rules was also broken on purpose and
  caught.

Version 0.4.9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
2026-11-14 00:09:18 +00:00
adminandClaude Opus 5.5 fd75c183ba Fix #4, #6: resumes keep their task; later instructions reach the verifier
Resuming an agent (SendMessage to a finished agent) raises agent.spawn
again, with resume.agentId set. The router treated every resume as a
new task: new Tn, fresh routing, the brief replaced by the resume
message, the failure count and round base reset. Hence T322's
"routed to Sonnet, ran on Opus" for an Opus agent that was only
resumed (#4), and verifiers that judged later rounds against the wrong
text (#6).

- A resume passes through untouched; its message is kept as a later
  instruction to that agent. Task, routing, failures and rounds stay.
- A SendMessage to a tracked agent (by id or by the name its Agent call
  gave it) that was delivered is kept as a later instruction too.
- The verifier prompt lists later instructions after the brief and
  judges scope added there as in scope.
- [router-rewrite Tn]: the stored rewrite is used up only once its
  agent has started (a failed start kept nothing before: T332 was then
  routed afresh, to Haiku). With nothing stored the call is refused
  instead of starting an agent with no brief. The task keeps Tn.
- The routing label compares with the model the engine resolved at
  start, not the one the router asked for.
- 11 new tests; each new rule was also broken on purpose and caught.

Version 0.4.8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
2026-11-13 23:41:48 +00:00
adminandClaude Opus 5.5 fc41ad21aa Fix TOOL-12 follow-up: no line or verifier on interim stops
A dev that waits on its own background work (a 35-45 min test suite)
ends several turns before it hands back. Every turn end posted a new
"verifying on Opus..." line and started the verifier on an unfinished
tree (MAINT-28 T307: 5 lines; T276/T278 judged an uncommitted tree).

- After each turn end of a tracked agent, its loop status decides:
  completed, failed, killed, or already dropped by the engine is the
  hand-back; waiting, idle, running or pending is an interim stop.
- An interim stop posts nothing and starts no verifier. The router
  checks the status again every 15 s, up to 24 h, and runs the round
  once the run has ended. A newer turn end takes the watch over.
- The hand-back uses the last turn's model and end reason.
- 8 new tests, including the ticket's acceptance (3 interim stops: one
  line, one verdict, both after the hand-back); each new rule was also
  broken on purpose and caught, apart from one redundant guard.

Version 0.4.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
2026-11-12 21:36:04 +00:00
6 changed files with 1070 additions and 53 deletions
+31 -1
View File
@@ -1,6 +1,6 @@
{
"name": "model-router",
"version": "0.4.6",
"version": "0.5.1",
"description": "Routes each subagent to Haiku, Sonnet or Opus by how well-defined its brief is, verifies dev work below Opus in its own clone on a different model, sends failures back to the same agent then a stronger one, and estimates the savings against Opus.",
"author": {
"name": "Glenn Schrooyen"
@@ -18,6 +18,36 @@
"description": "How long the test command may run in the clone. A run that times out is reported as NOT VERIFIED, never as a FAIL.",
"default": 1800
},
"testFileCommand": {
"type": "string",
"title": "Command that runs one test file",
"description": "Run in the clone, with {file} replaced by a file the full run reported as TIMEOUT or FAIL. Each such file is rerun alone, and on the base branch if it still fails. Empty: no reruns.",
"default": ""
},
"testFileTimeoutSec": {
"type": "number",
"title": "Time limit per rerun file (seconds)",
"description": "A file still running after this counts as timing out alone.",
"default": 900
},
"testRerunJobs": {
"type": "number",
"title": "Reruns at a time",
"description": "How many reported files are rerun in parallel.",
"default": 6
},
"testRerunMax": {
"type": "number",
"title": "Most files to rerun",
"description": "Reported files beyond this are listed as not rerun.",
"default": 40
},
"testIssuePattern": {
"type": "string",
"title": "Pattern of a reported file line",
"description": "Regular expression: group 1 is TIMEOUT or FAIL, group 2 the file. Empty: ^(TIMEOUT|FAIL)\\s+(\\S+)",
"default": ""
},
"defaultBaseBranch": {
"type": "string",
"title": "Base branch when the brief names none",
+13 -5
View File
@@ -6,22 +6,25 @@ A Claude Code plugin that picks a model for every subagent, checks dev work on a
| Step | Behaviour |
|---|---|
| Route | Dev agents run on **Sonnet** when the brief decides the approach (Expected / acceptance criteria), on **Opus** when the design is still open. Review and test agents keep the model they are given (Opus when none). |
| Route | Dev agents run on **Sonnet** when the brief decides the approach (Expected / acceptance criteria), on **Opus** when the design is still open. Review and test agents keep the model they are given (Opus when none). Other non-dev agents (a `test-design` role, or the `test-designer` / `release-reviewer` agent types) do too, but never Haiku. None of them get a hand-back line or verification. |
| Rerun reported files | With `testFileCommand` set, every file the full run reports as `TIMEOUT` or `FAIL` is rerun alone in the clone. A file that still fails or times out alone runs again on the base branch, in a worktree next to the clone (`<clone>-verifier-base`, removed afterwards). The verifier gets a table: a file that passes alone does not count, one that also fails on base is not caused by the change. |
| Later instructions | Every message sent to a dev agent after its brief (a SendMessage, or the message that resumes it) reaches the verifier as part of the task, so scope added later is judged as in scope. A resumed agent keeps its task number, routing and failure count. |
| Verify | When a dev agent below Opus finishes a round, the router diffs its clone against the merge base, runs the test command in that clone, and has **Opus** judge the change. |
| Hand-back line | Every dev hand-back gets one line at once: `[model-router] Tn verifying on Opus...` (a verdict follows), or `[model-router] Tn no verification: dev ran on Opus. Go straight to review.` Decided by the model the run actually used, not only the routing choice; a difference is shown as `(routed to X, ran on Y)`. |
| Hand-back line | Only dev agents get `T` numbers and lines; other agents are logged as `A1`, `A2`, ... Every dev hand-back gets one line at once: `[model-router] Tn verifying on Opus...` (a verdict follows), or `[model-router] Tn no verification: dev ran on Opus. Go straight to review.` Only the real hand-back counts: an interim stop while the dev waits on its own background work (e.g. a long test suite) posts nothing and starts no verifier. Decided by the model the run actually used, not only the routing choice; a difference is shown as `(routed to X, ran on Y)`. |
| On FAIL | 1st: resume the same agent with the issues. 2nd: a new agent one model up, in the same clone. After that: handed back to the manager. |
| Not verified | A test timeout, a test command that cannot run, a missing clone or merge base, or a verdict that cannot be read even after asking once more: reported as NOT VERIFIED, never as a FAIL. |
| Savings | `/router savings` prices every run on the model it ran on and compares it with the same tokens on Opus 5.5. |
## Briefs
Every dev, review and test brief carries one line:
Every pipeline brief carries these lines:
```
Pipeline-Role: dev
Clone: /home/homelab/projects/goodwe/clones/<TICKET>-dev/
```
Add `(Role: dev|review|test)` when the path does not end in the role. Tickets with extra parts (`OPT-11-P4`) are fine. Name the base branch (`Base branch: release/1.0`).
The role comes from `Pipeline-Role:` first, then `(Role: ...)` on the `Clone:` line, then the folder name (which must end in `-dev`, `-review` or `-test`). Any other role (`test-design`, ...) is never a dev, whatever other clones the brief mentions; neither is a `test-designer` or `release-reviewer` agent. With a `## Test plan` section, the verifier checks that every T-id and N-id is covered. Tickets with extra parts (`OPT-11-P4`) are fine. Name the base branch (`Base branch: release/1.0`).
## Install
@@ -38,6 +41,11 @@ Or for development: `claude --plugin-dir /path/to/model-router`.
|---|---|---|
| `testCommand` | empty | Run in the agent's clone; output goes to the verifier |
| `testTimeoutSec` | 1800 | Wrapped in GNU `timeout --verbose`; a timeout is NOT VERIFIED |
| `testFileCommand` | empty | Runs one file, `{file}` replaced by a reported file name; only names of letters, digits, `_ . - /` are run |
| `testFileTimeoutSec` | 900 | Limit per rerun file |
| `testRerunJobs` | 6 | Reruns at a time |
| `testRerunMax` | 40 | Most files rerun per round |
| `testIssuePattern` | `^(TIMEOUT\|FAIL)\s+(\S+)` | Group 1 the kind, group 2 the file |
| `defaultBaseBranch` | `release/1.0` | After the PR lookup and the brief's own base branch |
| `prBaseCommand` | empty | Optional command that prints the PR's base branch |
| `maxRewrites` | 2 | Failed verdicts before handing back |
@@ -59,7 +67,7 @@ claude plugin validate .
claude plugin test .
```
62 tests. Every routing, verification, cost and timeout rule was also checked by breaking it on purpose (66 mutations in v0.4.4, plus 13 for the verdict reader in v0.4.5 and 12 for the hand-back line in v0.4.6; all caught by a test).
102 tests. Every routing, verification, cost and timeout rule was also checked by breaking it on purpose (66 mutations in v0.4.4, plus 13 for the verdict reader in v0.4.5 12 for the hand-back line in v0.4.6 10 for interim stops in v0.4.7 14 for resumes, later instructions and rewrites in v0.4.8 23 for file reruns in v0.4.9 and 16 for non-dev roles and gate agent types in v0.5.1; all caught by a test).
## Known limits
+2 -1
View File
@@ -150,7 +150,8 @@ export const usd = (n: number) => (Math.abs(n) < 0.01 && n !== 0 ? `$${n.toFixed
export const savingsReport = (costs: Readonly<Record<string, TaskCost>>, limit: number, only?: string): string => {
const ids = Object.keys(costs)
.filter(id => !only || id === only)
.sort((a, b) => Number(b.slice(1)) - Number(a.slice(1)))
// dev tasks (T) first, then other agents (A); newest first within each
.sort((a, b) => b.charAt(0).localeCompare(a.charAt(0)) || Number(b.slice(1)) - Number(a.slice(1)))
if (ids.length === 0) return only ? `model-router: no costs recorded for ${only}.` : 'savings: no agent runs recorded yet.'
const row = (id: string, t: TaskCost) => {
+265 -31
View File
@@ -16,14 +16,26 @@ import {
clip,
decide,
onFail,
parseBrief,
briefFor,
parsePatterns,
parseVerdict,
handBackLine,
ranTier,
turnEndKind,
WATCH_EVERY_MS,
WATCH_FOR_MS,
safetyHit,
skipReason,
validTimeout,
fileCommand,
isSafeFile,
issueRegex,
parseIssueLine,
positiveInt,
rerunTable,
type FileKind,
type FileRun,
type RerunRow,
verifierFor,
type Role,
type Tier,
@@ -36,9 +48,15 @@ type AgentRec = {
taskId: string
/** The model the last round actually ran on (the routing choice until a round reports one). */
tier: Tier
/** The model the router chose at spawn; tier can differ when the run used another. */
/** The model the engine resolved at spawn; tier can differ when the run used another. */
routed?: Tier
/** The name the Agent call gave it, which SendMessage may address it by. */
name?: string
/** Later instructions to this agent (resumes, SendMessage), oldest first: part of its task. */
followUps?: string[]
role?: Role
/** The role as the brief wrote it, for non-dev roles (test-design). */
roleName?: string
clone?: string
baseBranch?: string
critical: boolean
@@ -67,6 +85,7 @@ type Outcome =
| 'verdict-unreadable'
| 'test-agent'
| 'review-agent'
| 'non-dev-agent'
| 'no-clone'
| 'opus-unverified'
| 'agent-ended'
@@ -78,6 +97,8 @@ type LogEntry = {
task: string
tier: Tier
role?: Role
/** The role as the brief wrote it, for non-dev roles (test-design). */
roleName?: string
reason: string
round?: number
verifier?: Tier
@@ -87,6 +108,8 @@ type LogEntry = {
const S = {
counter: 'counter',
// other agents (review, test, test-design, no role) count apart, as A1, A2, ...
counterOther: 'counter-other',
pending: 'pending',
agents: 'agents',
log: 'log',
@@ -119,6 +142,9 @@ When unsure between sonnet and opus, pick sonnet: a failed verification escalate
const VERIFIER_SYSTEM = `You are an adversarial verifier. Another model wrote the change below. You did not.
Judge only whether the change correctly and completely does the task, without breaking anything nearby.
Look for: requirements or acceptance criteria missed, wrong logic, unhandled edge cases, tests that cannot fail, tests weakened or deleted, safety checks bypassed, unrelated changes.
Scope added in "Later instructions" was asked for after the brief: judge it as in scope, not as unrequested work.
When a table of files rerun one by one is given: a file that passes alone does not count against the change (the full run was overloaded or flaky); one that also fails or times out on base is not caused by this change: mention it, do not fail for it. Name every file you count against the change in issues.
When the task has a "## Test plan" section with T-ids and N-ids: every T-id must be implemented by a test that can fail, and every N-id asserted somewhere, unless the developer says why not. A missing or untestable T-id/N-id is a FAIL issue naming the id. A test that cannot fail on the old code where the plan says "red on base" is a FAIL issue.
Do not fail a change for style alone. A later PR review and mutation testing follow you; you are the cheap first gate.
Reply with JSON only:
@@ -126,11 +152,13 @@ Reply with JSON only:
const ORCHESTRATOR_NOTE = `## Model router (model-router mod)
The model-router mod picks each subagent's model and verifies dev work before it reaches a PR.
- Every dev, review and test brief carries one line \`Clone: <path>\` (add \`(Role: dev|review|test)\` if the path does not end in the role). The router reads the role from that line only; a brief naming several clones without it runs on Opus, unverified.
- Every pipeline brief carries \`Pipeline-Role: <role>\` and one line \`Clone: <path>\` (add \`(Role: <role>)\` if the folder name does not end in -dev, -review or -test). The role comes from Pipeline-Role, then (Role: ...), then the folder name. Only dev briefs are routed and verified; any other role (test-design, ...) keeps the model you name, gets no "[model-router]" line and an A number, not a T number. A brief naming several clones and no Clone: line runs on Opus, unverified.
- Dev agents are routed: Sonnet when the brief decides the approach (Expected / acceptance criteria), Opus when the design is still open. Naming a model on a dev brief has no effect. Give each dev brief its clone path and base branch.
- Review and test agents keep the model you name (Opus when you name none).
- Every dev hand-back gets one "[model-router] Tn" line at once. "verifying on Opus...": wait for the verdict before advancing the PR. "no verification: dev ran on Opus. Go straight to review.": no verdict will come. Lines are decided by the model the dev actually ran on.
- On "[model-router] ... FAIL ... resume agent", resume that same agent with SendMessage, passing the issues. On "start a new agent ... [router-rewrite Tn]", stop the old agent and call Agent with exactly that prompt. On "handed back", stop and show the issues to the person. Do not fix rejected work yourself.`
- Review and test agents keep the model you name (Opus when you name none); other non-dev agents (test-design, test-designer, release-reviewer) too, but never Haiku.
- Every dev hand-back gets one "[model-router] Tn" line at once; an interim stop while the dev waits on its own background work gets none. "verifying on Opus...": wait for the verdict before advancing the PR. "no verification: dev ran on Opus. Go straight to review.": no verdict will come. Lines are decided by the model the dev actually ran on.
- On "[model-router] ... FAIL ... resume agent", resume that same agent with SendMessage, passing the issues. On "start a new agent ... [router-rewrite Tn]", stop the old agent and call Agent with exactly that prompt. On "handed back", stop and show the issues to the person.
- Scope or decisions added after the brief: send them to the dev agent with SendMessage. The verifier sees every later message to that agent and treats it as in scope. A resumed agent keeps its task number. Do not fix rejected work yourself.
- With ems-dev-tooling: record every verdict with \`ticket\` first (PASS: \`ticket advance <T> --to review --sha <sha>\`; FAIL: \`ticket fail <T> --stage router\`), then resume or rewrite. pipeline-guard refuses a resume while the ticket is not back in development.`
// ---------- store helpers (top-level, as the engine requires for `$`) ----------
@@ -165,11 +193,13 @@ async function putIn<T>($: EngineInterface, key: string, id: string, value: T |
await updateIn<T>($, key, id, () => value)
}
async function nextId($: EngineInterface) {
/** T numbers are for dev agents only; every other agent gets an A number. */
async function nextId($: EngineInterface, isDev: boolean) {
const key = isDev ? S.counter : S.counterOther
return serial(async () => {
const n = Number((await $.store.get(S.counter)) ?? 0) + 1
await $.store.set(S.counter, n)
return `T${n}`
const n = Number((await $.store.get(key)) ?? 0) + 1
await $.store.set(key, n)
return `${isDev ? 'T' : 'A'}${n}`
})
}
@@ -251,18 +281,37 @@ async function chargeOverhead($: EngineInterface, taskId: string, alias: Tier, u
await updateIn<TaskCost>($, S.costs, taskId, old => addOverhead(old ?? emptyTask(taskId), modelId, u, prices))
}
type TestRun = { status: 'ran' | 'timeout' | 'error'; code: number | null; tail: string; seconds: number }
type TestRun = {
status: 'ran' | 'timeout' | 'error'
code: number | null
tail: string
seconds: number
/** Files the run reported as TIMEOUT or FAIL, in order, each once. */
issues: { kind: FileKind; file: string }[]
}
/**
* Runs the test command in the clone, streamed: `$.process.run` stops at ten
* minutes, a stream has no limit of its own. GNU `timeout` does the kill.
* A timeout or a run that could not start is no verdict on the code.
*/
async function runTests($: EngineInterface, cmd: string, cwd: string, timeoutSec: number): Promise<TestRun> {
async function runTests($: EngineInterface, cmd: string, cwd: string, timeoutSec: number, issueRe?: RegExp): Promise<TestRun> {
const started = await $.clock.now()
const seconds = async () => Math.round(((await $.clock.now()) - started) / 1000)
let out = ''
let errTail = ''
// reported files are read line by line as the run streams, not from the kept tail
const issues: TestRun['issues'] = []
const partial: Record<string, string> = { stdout: '', stderr: '' }
const readLines = (stream: string, text: string, isLast = false) => {
if (!issueRe) return
const lines = ((partial[stream] ?? '') + text).split('\n')
partial[stream] = isLast ? '' : lines.pop() ?? ''
for (const line of lines) {
const hit = parseIssueLine(line, issueRe)
if (hit && !issues.some(i => i.file === hit.file)) issues.push(hit)
}
}
try {
const stream = $.process.spawn({
argv: ['timeout', '--verbose', '--kill-after=30', String(Math.max(1, Math.round(timeoutSec))), 'sh', '-c', cmd],
@@ -271,7 +320,10 @@ async function runTests($: EngineInterface, cmd: string, cwd: string, timeoutSec
for await (const piece of stream) {
out = (out + piece.text).slice(-65_536)
if (piece.stream === 'stderr') errTail = (errTail + piece.text).slice(-65_536)
readLines(piece.stream, piece.text)
}
readLines('stdout', '', true)
readLines('stderr', '', true)
// GNU timeout --verbose says so on stderr when, and only when, it fires.
// Read from the joined stderr, so a line split across pieces still counts.
const isFired = errTail.includes('timeout: sending signal')
@@ -280,17 +332,81 @@ async function runTests($: EngineInterface, cmd: string, cwd: string, timeoutSec
const took = await seconds()
// Only a timeout that fired is a timeout: a 137 without it is the code's own
// (e.g. out of memory) and goes to the verifier as a failed run.
if (isFired) return { status: 'timeout', code: end.code, tail, seconds: took }
if (end.code === 126 || end.code === 127) return { status: 'error', code: end.code, tail, seconds: took }
return { status: 'ran', code: end.code, tail, seconds: took }
if (isFired) return { status: 'timeout', code: end.code, tail, seconds: took, issues }
if (end.code === 126 || end.code === 127) return { status: 'error', code: end.code, tail, seconds: took, issues }
return { status: 'ran', code: end.code, tail, seconds: took, issues }
} catch (err) {
return { status: 'error', code: null, tail: String(err).slice(0, 500), seconds: await seconds() }
return { status: 'error', code: null, tail: String(err).slice(0, 500), seconds: await seconds(), issues }
}
}
/** The result of one file run alone. */
function fileRunOf(t: TestRun): FileRun {
if (t.status === 'timeout') return 'timeout'
if (t.status === 'error') return 'error'
return t.code === 0 ? 'pass' : 'fail'
}
/** Runs `work` over the items, at most `jobs` at a time, keeping their order. */
async function inPool<T, R>(items: readonly T[], jobs: number, work: (item: T) => Promise<R>): Promise<R[]> {
const out: R[] = new Array(items.length)
let next = 0
const lane = async () => {
while (next < items.length) {
const i = next++
out[i] = await work(items[i] as T)
}
}
await Promise.all(Array.from({ length: Math.max(1, Math.min(jobs, items.length)) }, lane))
return out
}
/**
* Reruns the files a full run reported, one by one in the clone; those that
* still fail or time out alone run again on the base, checked out next to the
* clone (`<clone>-verifier-base`, removed afterwards).
*/
async function rerunReported($: EngineInterface, rec: AgentRec, issues: TestRun['issues'], cfg: Settings): Promise<RerunRow[]> {
const cwd = rec.clone as string
const picked = issues.filter(i => isSafeFile(i.file)).slice(0, cfg.testRerunMax)
const rows: RerunRow[] = picked.map(i => ({ file: i.file, full: i.kind, head: 'not run', base: 'not run' }))
const runAll = (dir: string, which: RerunRow[]) =>
inPool(which, cfg.testRerunJobs, async r =>
fileRunOf(await runTests($, fileCommand(cfg.testFileCommand, r.file), dir, cfg.testFileTimeoutSec)),
)
$.ui.status(`router: ${rec.taskId} rerunning ${rows.length} reported file(s) alone`)
const head = await runAll(cwd, rows)
rows.forEach((r, i) => (r.head = head[i] ?? 'error'))
const onBase = rows.filter(r => r.head === 'fail' || r.head === 'timeout')
if (onBase.length === 0) return rows
const { base } = await firstBase($, rec, cfg.prBaseCommand, cfg.defaultBaseBranch)
if (!base) return rows
const tree = `${cwd.replace(/\/+$/, '')}-verifier-base`
await sh($, ['git', '-C', cwd, 'worktree', 'remove', '--force', tree], cwd)
await sh($, ['git', '-C', cwd, 'worktree', 'prune'], cwd)
const added = await sh($, ['git', '-C', cwd, 'worktree', 'add', '--detach', tree, base], cwd, 120_000)
if (!added.ok) return rows
try {
$.ui.status(`router: ${rec.taskId} rerunning ${onBase.length} file(s) on base`)
const results = await runAll(tree, onBase)
onBase.forEach((r, i) => (r.base = results[i] ?? 'error'))
} finally {
await sh($, ['git', '-C', cwd, 'worktree', 'remove', '--force', tree], cwd, 120_000)
await sh($, ['git', '-C', cwd, 'worktree', 'prune'], cwd)
}
return rows
}
type Settings = {
testCommand: string
testTimeoutSec: number
testFileCommand: string
testFileTimeoutSec: number
testRerunJobs: number
testRerunMax: number
testIssueRe: RegExp
prBaseCommand: string
defaultBaseBranch: string
maxFails: number
@@ -352,7 +468,7 @@ async function verifyRound($: EngineInterface, agentId: string, rec: AgentRec, c
let tests = ''
if (cfg.testCommand) {
$.ui.status(`router: ${rec.taskId} round ${round} running tests`)
const t = await runTests($, cfg.testCommand, cwd, cfg.testTimeoutSec)
const t = await runTests($, cfg.testCommand, cwd, cfg.testTimeoutSec, cfg.testIssueRe)
if (t.status !== 'ran') {
// No verdict on the code: no verifier call, no fail counted, no rewrite.
entry.outcome = t.status === 'timeout' ? 'tests-timeout' : 'tests-error'
@@ -369,6 +485,13 @@ async function verifyRound($: EngineInterface, agentId: string, rec: AgentRec, c
return
}
tests = `exit code ${t.code} after ${t.seconds} s\n${t.tail}`
if (cfg.testFileCommand && t.issues.length > 0) {
const rows = await rerunReported($, rec, t.issues, cfg)
const skipped = t.issues.length - rows.length
tests +=
`\n\n### Files the full run reported, rerun one by one\n${rerunTable(rows)}` +
(skipped > 0 ? `\n(${skipped} more reported file(s) not rerun: unsafe name or over testRerunMax.)` : '')
}
}
// 3. The verdict, from another model.
@@ -377,6 +500,11 @@ async function verifyRound($: EngineInterface, agentId: string, rec: AgentRec, c
$.ui.status(`router: ${rec.taskId} round ${round} verifying on ${verifier}`)
const question = [
`## Task\n${clip(rec.prompt, 12_000)}`,
rec.followUps?.length
? `## Later instructions to this agent (after the brief; part of the task, in scope)\n${rec.followUps
.map((f, i) => `${i + 1}. ${f}`)
.join('\n\n')}`
: '',
rec.lastIssues?.length
? `## Issues raised last round (check they are fixed)\n${rec.lastIssues.map(i => `- ${i}`).join('\n')}`
: '',
@@ -503,6 +631,7 @@ async function skipRound($: EngineInterface, agentId: string, rec: AgentRec, out
task: rec.prompt.split('\n')[0]?.slice(0, 60) ?? '',
tier: rec.tier,
role: rec.role,
roleName: rec.roleName,
reason: outcome,
outcome,
})
@@ -523,6 +652,71 @@ async function runRound($: EngineInterface, agentId: string, rec: AgentRec, ende
else await verifyRound($, agentId, round, cfg)
}
// ---------- later instructions (#6) ----------
const FOLLOW_UPS_KEPT = 12
/** Adds a later instruction to a tracked agent's record; the same text twice counts once. */
async function noteFollowUp($: EngineInterface, agentId: string, text: string) {
const t = text.trim()
if (!t) return
await updateIn<AgentRec>($, S.agents, agentId, old => {
if (!old) return undefined
const list = old.followUps ?? []
if (list.includes(clip(t, 4000))) return old
return { ...old, followUps: [...list, clip(t, 4000)].slice(-FOLLOW_UPS_KEPT) }
})
}
/** The tracked agent a SendMessage `to` names: its id, or the name its Agent call gave it. */
async function agentAddressed($: EngineInterface, to: string) {
const recs = await getMap<AgentRec>($, S.agents)
if (recs[to]) return to
return Object.keys(recs).find(id => recs[id]?.name === to)
}
// ---------- waiting for the real hand-back ----------
// The newest turn end per agent. A turn end that is only an interim stop (the
// agent waits on its own background work) must not post a line or start the
// verifier: only the run's end does. Module state: a reload drops the watches.
const lastTurn = new Map<string, { token: number; ended: string; reported?: string }>()
let tokens = 0
function pause($: EngineInterface, ms: number) {
return new Promise<void>(resolve => {
$.clock.after(ms, () => resolve())
})
}
async function statusOf($: EngineInterface, agentId: string) {
try {
return (await $.agent.list()).find(a => a.id === agentId)?.status
} catch {
return undefined
}
}
/**
* After a tracked agent's turn ends: runs the round once its run has ended,
* and not for an interim stop. A newer turn end of the same agent takes over.
*/
async function awaitHandBack($: EngineInterface, agentId: string, token: number, cfg: Settings) {
const started = await $.clock.now()
for (;;) {
if (lastTurn.get(agentId)?.token !== token) return // a newer turn end took over
if (turnEndKind(await statusOf($, agentId)) === 'handback') break
if ((await $.clock.now()) - started > WATCH_FOR_MS) return
await pause($, WATCH_EVERY_MS)
}
const turn = lastTurn.get(agentId)
if (!turn || turn.token !== token) return
lastTurn.delete(agentId)
// read the record now: an earlier round may have changed it meanwhile
const rec = (await getMap<AgentRec>($, S.agents))[agentId]
if (rec) await runRound($, agentId, rec, turn.ended, turn.reported, cfg)
}
// ---------- the hooks ----------
export const register: Register = (on, options) => {
@@ -531,6 +725,11 @@ export const register: Register = (on, options) => {
const cfg: Settings = {
testCommand: String(options.testCommand ?? '').trim(),
testTimeoutSec: validTimeout(options.testTimeoutSec),
testFileCommand: String(options.testFileCommand ?? '').trim(),
testFileTimeoutSec: positiveInt(options.testFileTimeoutSec, 900),
testRerunJobs: positiveInt(options.testRerunJobs, 6),
testRerunMax: positiveInt(options.testRerunMax, 40),
testIssueRe: issueRegex(String(options.testIssuePattern ?? '')),
prBaseCommand: String(options.prBaseCommand ?? '').trim(),
defaultBaseBranch: String(options.defaultBaseBranch ?? 'release/1.0').trim(),
maxFails: Number(options.maxRewrites ?? 2),
@@ -556,6 +755,14 @@ export const register: Register = (on, options) => {
on('agent.spawn', async ($, e, next) => {
if (e.fork || e.isTeammate || e.workflow) return next(e)
// A resume continues a known agent: same task, same routing, same counts.
// Its message is a later instruction the verifier must see (#4, #6).
if (e.resume) {
const resumed = await next(e)
if (resumed.deny === undefined) await noteFollowUp($, e.resume.agentId, e.prompt)
return resumed
}
let taskId: string
let prompt = e.prompt
let tier: Tier
@@ -577,14 +784,21 @@ export const register: Register = (on, options) => {
tier = pending.tier
fails = pending.fails
critical = pending.critical
reason = `new agent after ${fails} failed verdicts`
await putIn<Pending>($, S.pending, markId, undefined)
reason = `rewrite of ${markId} after ${fails} failed verdicts`
} else if (markId) {
// Starting it would hand an agent nothing but the marker.
return {
deny:
`model-router: no stored rewrite for ${markId} (already started, or cleared). ` +
'Start the agent with the full brief instead, or check /router.',
}
} else {
taskId = await nextId($)
const brief = parseBrief(e.prompt)
const brief = briefFor(e.prompt, e.subagentType)
taskId = await nextId($, brief.role === 'dev')
critical = safetyHit(`${e.description}\n${e.prompt}`, patterns) !== undefined
const d = decide({
role: brief.role,
roleName: brief.roleName,
named: asTier(e.model),
subagentType: e.subagentType,
prompt: e.prompt,
@@ -595,6 +809,7 @@ export const register: Register = (on, options) => {
isRouted =
brief.role !== 'review' &&
brief.role !== 'test' &&
brief.role !== 'other' &&
!(d.kind === 'fixed' && d.reason === 'model chosen by orchestrator')
if (d.kind === 'fixed') {
tier = d.tier
@@ -607,9 +822,14 @@ export const register: Register = (on, options) => {
}
}
const brief = parseBrief(prompt)
const brief = briefFor(prompt, e.subagentType)
const ran = await next({ ...e, prompt, model: tier })
if (ran.deny !== undefined || !ran.agentId) return ran
// the stored rewrite is used up only once its agent has started
if (markId && pending) await putIn<Pending>($, S.pending, markId, undefined)
// label against what the engine resolved, not what the router asked for
const resolved = asTier(ran.model) ?? tier
if (resolved !== tier) reason += ` (asked ${tier}, engine resolved ${resolved})`
// Learn what the alias resolves to, so router calls are priced on the real model.
if (ran.model && ran.model !== tier && ran.model.includes(tier)) await putIn($, S.aliases, tier, ran.model)
@@ -623,9 +843,11 @@ export const register: Register = (on, options) => {
if (brief.role)
await putIn<AgentRec>($, S.agents, ran.agentId, {
taskId,
tier,
routed: tier,
tier: resolved,
routed: resolved,
name: e.name,
role: brief.role,
roleName: brief.roleName,
clone: brief.clone,
baseBranch: brief.baseBranch,
critical,
@@ -639,13 +861,14 @@ export const register: Register = (on, options) => {
agentId: ran.agentId,
at: await $.clock.now(),
task: e.description,
tier,
tier: resolved,
role: brief.role,
roleName: brief.roleName,
reason,
outcome: 'spawned',
opusRequested,
})
$.ui.status(`router: ${taskId} ${brief.role ?? 'agent'} → ${tier}`)
$.ui.status(`router: ${taskId} ${brief.roleName ?? brief.role ?? 'agent'} → ${tier}`)
return ran
}).catch(($, e, next) => {
// Fail open: a router bug must never block delegation.
@@ -668,9 +891,20 @@ export const register: Register = (on, options) => {
}
const rec = (await getMap<AgentRec>($, S.agents))[agentId]
if (!rec) return out
// Verification takes minutes: run it after this event has gone on.
const reported = e.usage?.model
$.clock.after(0, () => void runRound($, agentId, rec, e.reason, reported, cfg))
// Only the run's end is a hand-back; wait for it after this event has gone on.
const token = ++tokens
lastTurn.set(agentId, { token, ended: e.reason, reported: e.usage?.model })
$.clock.after(0, () => void awaitHandBack($, agentId, token, cfg))
return out
})
// A message to a running agent is a later instruction too (#6).
on('tool.call', { tool: 'SendMessage' }, async ($, e, next) => {
const out = await next(e)
if (out.deny !== undefined || out.isError) return out
const to = typeof e.to === 'string' ? e.to : ''
const id = to ? await agentAddressed($, to) : undefined
if (id && typeof e.message === 'string') await noteFollowUp($, id, e.message)
return out
})
@@ -704,14 +938,14 @@ export const register: Register = (on, options) => {
.slice(-15)
.map(
l =>
`${l.id.padEnd(5)} ${(l.role ?? '-').padEnd(6)} ${l.tier.padEnd(6)} ${(l.verifier ? '→' + l.verifier : '').padEnd(8)} ${l.outcome.padEnd(17)} ${(l.task ?? '').slice(0, 40)}${l.opusRequested ? ' [opus requested]' : ''}`,
`${l.id.padEnd(5)} ${(l.roleName ?? l.role ?? '-').padEnd(6)} ${l.tier.padEnd(6)} ${(l.verifier ? '→' + l.verifier : '').padEnd(8)} ${l.outcome.padEnd(17)} ${(l.task ?? '').slice(0, 40)}${l.opusRequested ? ' [opus requested]' : ''}`,
)
.join('\n')
const pending = Object.keys(await getMap<Pending>($, S.pending))
return {
text:
`model-router - last ${Math.min(15, list.length)} of ${list.length}\n` +
`writers: ${count(l => (l.outcome === 'spawned' ? `${l.role ?? 'other'}:${l.tier}` : undefined))}\n` +
`writers: ${count(l => (l.outcome === 'spawned' ? `${l.roleName ?? l.role ?? 'none'}:${l.tier}` : undefined))}\n` +
`results: ${count(l => (l.outcome === 'spawned' ? undefined : l.outcome))}\n` +
`opus requested on dev briefs: ${list.filter(l => l.opusRequested).length}\n` +
(pending.length ? `pending rewrites: ${pending.join(', ')}\n` : '') +
+136 -13
View File
@@ -1,7 +1,8 @@
// Pure routing logic: no `$`, so every rule here is unit-tested.
export type Tier = 'haiku' | 'sonnet' | 'opus' | 'fable'
export type Role = 'dev' | 'review' | 'test'
/** `other`: any pipeline role the router does not know (test-design, ...): never a dev. */
export type Role = 'dev' | 'review' | 'test' | 'other'
export const TIERS: readonly Tier[] = ['haiku', 'sonnet', 'opus', 'fable']
@@ -31,6 +32,8 @@ export type Brief = {
clone?: string
ticket?: string
role?: Role
/** The role as the brief wrote it (`test-design`), when it is not dev, review or test. */
roleName?: string
baseBranch?: string
/** Several clone paths and no `Clone:` line: the role cannot be told. */
isAmbiguous?: boolean
@@ -38,40 +41,56 @@ export type Brief = {
// A ticket is LETTERS-digits plus any extra parts (OPT-11-P4); the role is the last part.
const TICKET = '[A-Za-z]+-\\d+(?:-[A-Za-z0-9]+)*?'
const CLONE_PATH = new RegExp(`(\\S*?\\/clones\\/(${TICKET})-(dev|review|test))\\/?`)
// The role must end the folder name: EV-05-design and EV-05-test-design are no -test clone.
const CLONE_PATH = new RegExp(`(\\S*?\\/clones\\/(${TICKET})-(dev|review|test))(?![A-Za-z0-9_-])\\/?`)
const CLONE_PATHS = new RegExp(CLONE_PATH.source, 'g')
// The explicit line every brief carries: `Clone: <path>` with an optional `(Role: dev)`.
const CLONE_LINE = /^[ \t>*-]*Clone:[ \t]*`?([^\s`]+?)\/?`?[ \t]*(?:\(Role:[ \t]*(dev|review|test)\))?[ \t]*$/im
const CLONE_LINE = /^[ \t>*-]*Clone:[ \t]*`?([^\s`]+?)\/?`?[ \t]*(?:\(Role:[ \t]*([A-Za-z][\w-]*)\))?[ \t]*$/im
// The pipeline's own header, which decides over everything else.
const PIPELINE_ROLE = /^[ \t>*-]*Pipeline-Role:[ \t]*`?([A-Za-z][\w-]*)`?[ \t]*$/im
const BASE =
/\b(?:base(?:\s+branch)?|target(?:\s+branch)?|merge\s+into|PR\s+(?:against|into))\s*[:=]?\s*`?((?:release|main|master|develop)[\w./-]*)/i
/** A role as written in a brief: dev, review and test as they are, anything else `other`. */
export const asRole = (written: string | undefined): { role?: Role; roleName?: string } => {
if (!written) return {}
const w = written.toLowerCase()
if (w === 'dev' || w === 'review' || w === 'test') return { role: w }
return { role: 'other', roleName: w }
}
const trimPath = (p: string) => p.replace(/^[`'"(]+/, '').replace(/[`'")]+$/, '').replace(/\/+$/, '')
/**
* Reads the clone path, ticket, role and base branch out of a brief.
*
* The `Clone:` line decides when present: its `(Role: ...)`, else the role in
* its path. Without one, a brief that names one clone is read from that path;
* one that names several is ambiguous and gets no role.
* The role: `Pipeline-Role:` first, then `(Role: ...)` on the `Clone:` line,
* then the role in the clone's folder name. A role other than dev, review or
* test is `other`, never a dev. Without a `Clone:` line, a brief that names
* one clone is read from that path; several are ambiguous. A brief whose
* header names a non-dev role is never read from other paths it mentions.
*/
export const parseBrief = (prompt: string): Brief => {
const b = prompt.match(BASE)
// a sentence's full stop or a trailing slash is no part of the branch
const baseBranch = b?.[1]?.replace(/[./-]+$/, '')
const header = asRole(prompt.match(PIPELINE_ROLE)?.[1])
const line = prompt.match(CLONE_LINE)
if (line?.[1]) {
const clone = trimPath(line[1])
const fromPath = clone.match(CLONE_PATH)
const role = (line[2] ?? fromPath?.[3]) as Role | undefined
return { clone, ticket: fromPath?.[2], role, baseBranch }
const stated = header.role ? header : line[2] ? asRole(line[2]) : asRole(fromPath?.[3])
// A pipeline agent with a clone the router cannot place is no dev: keep its model.
return { clone, ticket: fromPath?.[2], role: 'other', ...stated, baseBranch }
}
if (header.role && header.role !== 'dev') return { ...header, baseBranch }
const paths = [...new Set([...prompt.matchAll(CLONE_PATHS)].map(m => trimPath(m[1] ?? '')))]
if (paths.length > 1) return { baseBranch, isAmbiguous: true }
const only = paths[0]?.match(CLONE_PATH)
if (!only) return { baseBranch }
return { clone: trimPath(only[1] ?? ''), ticket: only[2], role: only[3] as Role, baseBranch }
if (!only) return { ...header, baseBranch }
return { clone: trimPath(only[1] ?? ''), ticket: only[2], ...(header.role ? header : asRole(only[3])), baseBranch }
}
// ---------- routing ----------
@@ -108,6 +127,21 @@ export const heuristic = (subagentType: string | undefined, prompt: string, role
return undefined
}
/** Agent types that gate rather than write: never routed down, never verified. */
export const KEEP_MODEL_TYPES = /^(test-designer|release-reviewer)$/i
/**
* The brief as the router treats it: a gate agent type (test-designer,
* release-reviewer) is never a dev, nor an unplaced agent, whatever paths
* its brief names. Review and test roles stay as they are.
*/
export const briefFor = (prompt: string, subagentType?: string): Brief => {
const b = parseBrief(prompt)
if (!subagentType || !KEEP_MODEL_TYPES.test(subagentType)) return b
if (b.role === 'review' || b.role === 'test') return b
return { ...b, role: 'other', roleName: b.role === 'other' ? b.roleName : subagentType.toLowerCase(), isAmbiguous: undefined }
}
export type Decision =
| { kind: 'fixed'; tier: Tier; reason: string; opusRequested?: boolean }
| { kind: 'classify'; opusRequested?: boolean }
@@ -126,15 +160,25 @@ export const decide = (args: {
prompt: string
respectExplicit: boolean
isAmbiguous?: boolean
/** The role as written, for an `other` role (test-design). */
roleName?: string
}): Decision => {
const { role, named, subagentType, prompt, respectExplicit, isAmbiguous } = args
const who = args.roleName ?? role
// Could be a test or review agent: never risk a weaker model on it.
if (isAmbiguous)
return { kind: 'fixed', tier: named && named !== 'haiku' && named !== 'sonnet' ? named : 'opus', reason: 'several clone paths and no Clone: line' }
if (role === 'review' || role === 'test')
return named
? { kind: 'fixed', tier: named, reason: `named model kept for ${role} agent` }
: { kind: 'fixed', tier: 'opus', reason: `${role} agent default` }
? { kind: 'fixed', tier: named, reason: `named model kept for ${who} agent` }
: { kind: 'fixed', tier: 'opus', reason: `${who} agent default` }
// Pipeline gates (test design, release review, any other non-dev role) judge, they do not
// write code: they keep a named model, never Haiku, and run on Opus when none is named.
const gate = role === 'other' ? (args.roleName ?? subagentType ?? 'other') : !role && subagentType && KEEP_MODEL_TYPES.test(subagentType) ? subagentType : undefined
if (gate)
return named && named !== 'haiku'
? { kind: 'fixed', tier: named, reason: `named model kept for ${gate}` }
: { kind: 'fixed', tier: 'opus', reason: `${gate} default` }
if (!role && respectExplicit && named) return { kind: 'fixed', tier: named, reason: 'model chosen by orchestrator' }
const opusRequested = role === 'dev' && (named === 'opus' || named === 'fable')
const h = heuristic(subagentType, prompt, role)
@@ -143,13 +187,14 @@ export const decide = (args: {
// ---------- verification ----------
export type Skip = 'test-agent' | 'review-agent' | 'no-clone' | 'opus-unverified' | 'agent-ended'
export type Skip = 'test-agent' | 'review-agent' | 'non-dev-agent' | 'no-clone' | 'opus-unverified' | 'agent-ended'
/** Why an agent run is not verified, or undefined when it is. */
export const skipReason = (rec: { role?: Role; clone?: string; tier: Tier }, ended: string): Skip | undefined => {
if (ended !== 'answer') return 'agent-ended'
if (rec.role === 'test') return 'test-agent'
if (rec.role === 'review') return 'review-agent'
if (rec.role === 'other') return 'non-dev-agent'
if (!rec.clone) return 'no-clone'
if (rank(rec.tier) >= rank('opus')) return 'opus-unverified'
return undefined
@@ -298,3 +343,81 @@ export const handBackLine = (args: {
if (skip === 'no-clone') return `${head} no verification: the brief names no clone path.${note}`
return undefined
}
// ---------- interim stops vs the hand-back (TOOL-12 follow-up) ----------
/** An agent's loop status, as `$.agent.list()` reports it. */
export type LoopStatus = 'pending' | 'running' | 'waiting' | 'idle' | 'completed' | 'failed' | 'killed'
/**
* What a turn's end means for the agent, read from its status afterwards.
* - `handback`: its run ended (completed, failed, killed), or the engine
* already dropped it from the list: the report went back to its spawner.
* - `interim`: it is held on its own background work (`waiting`), between
* turns (`idle`), or already in its next turn (`running`, `pending`).
*/
export const turnEndKind = (status: LoopStatus | undefined): 'handback' | 'interim' =>
status === undefined || status === 'completed' || status === 'failed' || status === 'killed' ? 'handback' : 'interim'
/** How often an interim agent's status is checked again, and for how long at most. */
export const WATCH_EVERY_MS = 15_000
export const WATCH_FOR_MS = 24 * 60 * 60 * 1000
// ---------- rerunning the files a full run reported (#5) ----------
export type FileKind = 'TIMEOUT' | 'FAIL'
export type FileRun = 'pass' | 'fail' | 'timeout' | 'error' | 'not run'
/** How the runner marks a file; group 1 the kind, group 2 the file. */
export const DEFAULT_ISSUE_PATTERN = '^(TIMEOUT|FAIL)\\s+(\\S+)'
/** A compiled issue pattern, or the default when the setting is empty or broken. */
export const issueRegex = (pattern: string): RegExp => {
try {
return new RegExp(pattern.trim() || DEFAULT_ISSUE_PATTERN)
} catch {
return new RegExp(DEFAULT_ISSUE_PATTERN)
}
}
/** One output line the runner wrote about a file, or undefined. */
export const parseIssueLine = (line: string, re: RegExp): { kind: FileKind; file: string } | undefined => {
const m = line.trim().match(re)
const kind = m?.[1]?.toUpperCase()
const file = m?.[2]
if (!file || (kind !== 'TIMEOUT' && kind !== 'FAIL')) return undefined
return { kind, file }
}
/**
* Whether a file name from the runner's output may go into a shell command:
* letters, digits, `_ . - /` only, no `..` part, not starting with `-`.
*/
export const isSafeFile = (file: string) =>
/^[A-Za-z0-9_./-]+$/.test(file) && !file.startsWith('-') && !file.split('/').includes('..')
export const fileCommand = (cmd: string, file: string) => cmd.split('{file}').join(file)
export type RerunRow = { file: string; full: FileKind; head: FileRun; base: FileRun }
/** What a rerun row means for the change under review. */
export const rerunVerdict = (r: RerunRow): string => {
if (r.head === 'pass') return 'passes alone: does not count against the change'
if (r.head === 'error' || r.head === 'not run') return 'could not be rerun: judge from the full run'
if (r.base === 'fail' || r.base === 'timeout') return `also ${r.base === 'fail' ? 'fails' : 'times out'} on base: not caused by this change`
if (r.base === 'pass') return `${r.head === 'fail' ? 'fails' : 'times out'} alone, passes on base: counts against the change`
return `${r.head === 'fail' ? 'fails' : 'times out'} alone; base not checked`
}
export const rerunTable = (rows: readonly RerunRow[]): string =>
[
'| file | full run | alone on head | on base | meaning |',
'|---|---|---|---|---|',
...rows.map(r => `| ${r.file} | ${r.full} | ${r.head} | ${r.base} | ${rerunVerdict(r)} |`),
].join('\n')
/** A positive whole number from a setting, else the fallback. */
export const positiveInt = (raw: unknown, fallback: number): number => {
const n = Number(raw)
return Number.isFinite(n) && n >= 1 ? Math.round(n) : fallback
}
+623 -2
View File
@@ -12,7 +12,13 @@ import {
validTimeout,
verifierFor,
handBackLine,
isSafeFile,
issueRegex,
parseIssueLine,
rerunVerdict,
fileCommand,
ranTier,
turnEndKind,
} from '../hooks/router'
const DEV = '/home/homelab/projects/goodwe/clones/SAFETY-95-dev'
@@ -60,6 +66,46 @@ test('fix 2: several clones and no Clone: line gives no role, and routes to opus
expect(parseBrief(`Work in ${DEV}/. Commit in ${DEV}.`).role).toBe('dev')
})
// ---------- non-dev roles (#7, EV-05) ----------
const EV05_DEV = '/home/homelab/projects/goodwe/clones/EV-05-dev'
const EV05_DESIGN = '/home/homelab/projects/goodwe/clones/EV-05-design'
const designBrief = (devClone = EV05_DEV) =>
`Pipeline-Role: test-design\nClone: ${EV05_DESIGN} (Role: test-design)\n\n` +
`Design the EV-05 tests. The developer works in ${devClone}/; do not touch it.`
test('#7: a test-design brief is a non-dev role, even when it names the dev clone', async () => {
expect(parseBrief(designBrief())).toMatchObject({ clone: EV05_DESIGN, role: 'other', roleName: 'test-design' })
// the header alone: other paths in the brief are never read for a role
const b = parseBrief(`Pipeline-Role: test-design\nThe developer works in ${EV05_DEV}/.`)
expect(b.role).toBe('other')
expect(b.clone).toBe(undefined)
})
test('#7: Pipeline-Role decides over (Role: ...) and the path', async () => {
expect(parseBrief(`Pipeline-Role: review\nClone: ${EV05_DEV} (Role: dev)`).role).toBe('review')
expect(parseBrief(`Pipeline-Role: dev\nClone: ${EV05_DESIGN}`).role).toBe('dev')
expect(parseBrief(`Pipeline-Role: Dev\nWork in ${EV05_DEV}/.`).role).toBe('dev')
// without the header, (Role: ...) still wins over the path
expect(parseBrief(`Clone: ${EV05_DEV} (Role: test-design)`)).toMatchObject({ role: 'other', roleName: 'test-design' })
})
test('#7: the role must end the folder name', async () => {
expect(parseBrief('Work in /x/clones/EV-05-test-design/.').role).toBe(undefined)
expect(parseBrief('Work in /x/clones/EV-05-dev-notes/.').role).toBe(undefined)
expect(parseBrief('Work in /x/clones/EV-05-dev/.').role).toBe('dev')
// a Clone: line the router cannot place keeps its model, unverified
expect(parseBrief('Clone: /x/clones/EV-05-test-design').role).toBe('other')
})
test('#7: other roles keep the named model, opus when none is named, and are never verified', async () => {
expect(decide({ role: 'other', roleName: 'test-design', named: 'sonnet', prompt: 'x', respectExplicit: false }))
.toEqual({ kind: 'fixed', tier: 'sonnet', reason: 'named model kept for test-design' })
expect(decide({ role: 'other', prompt: 'x', respectExplicit: false })).toMatchObject({ kind: 'fixed', tier: 'opus' })
expect(decide({ role: 'other', roleName: 'test-design', named: 'haiku', prompt: 'x', respectExplicit: false })).toMatchObject({ tier: 'opus' })
expect(skipReason({ role: 'other', clone: EV05_DESIGN, tier: 'sonnet' }, 'answer')).toBe('non-dev-agent')
})
// ---------- who picks the writer (C4) ----------
test('C4: review and test agents keep the model they are given', async () => {
@@ -218,6 +264,29 @@ type World = {
handBacks: string[]
/** Every message in the order it was sent. */
all: string[]
/** Resumes that reached the spawn hook. */
resumed: { agentId: string; prompt: string; model?: string }[]
/** How many coming new spawns the engine refuses. */
denySpawns: number
/** The model the engine resolves for new spawns, whatever was asked. */
engineModel?: string
verifierPrompts: string[]
verifierSystems: string[]
/** SendMessage calls fail (nothing delivered). */
sendFails?: boolean
/** Pieces the full test run prints to stdout. */
fullRunOutput: string[]
/** How a file run alone ends, keyed `head:<file>` or `base:<file>`; absent: pass. */
fileOutcomes: Record<string, 'pass' | 'fail' | 'timeout'>
/** Files run alone, as `head:<file>` or `base:<file>`. */
fileRuns: string[]
/** git worktree commands run. */
worktree: string[]
worktreeFails?: boolean
running: number
maxParallel: number
/** Each agent's loop status as $.agent.list() reports it; absent means completed. */
status: Record<string, string>
diffs: string[]
diffBases: string[]
testCwd: string[]
@@ -246,6 +315,8 @@ function world(on: On, w: World) {
on('model.complete', async (_$, e) => {
if (e.system?.startsWith('You route')) return { value: { isAnswered: true as const, text: 'sonnet', usage: USAGE } }
w.verifier.push(e.model)
w.verifierPrompts.push(e.prompt)
w.verifierSystems.push(e.system ?? '')
return { value: { isAnswered: true as const, text: w.verdicts.shift() ?? '{"verdict":"PASS"}', usage: USAGE } }
})
on('process.run', async (_$, e) => {
@@ -258,15 +329,35 @@ function world(on: On, w: World) {
w.diffBases.push(e.argv[2] ?? '')
return ok(w.diffs.shift() ?? '')
}
if (a.includes(' worktree ')) {
w.worktree.push(a)
if (a.includes(' worktree add ') && w.worktreeFails) return ok('', 128)
}
return ok('')
})
on('process.spawn', async function* (_$, e) {
const cmd = e.argv[e.argv.length - 1] ?? ''
if (cmd.startsWith('runone ')) {
// one file alone: `runone <file>`, in the clone or in the base worktree
const file = cmd.slice('runone '.length)
const where = String(e.cwd ?? '').endsWith('-verifier-base') ? 'base' : 'head'
w.fileRuns.push(`${where}:${file}`)
w.running++
w.maxParallel = Math.max(w.maxParallel, w.running)
for (let i = 0; i < 5; i++) await Promise.resolve() // let other reruns start
w.running--
const outcome = w.fileOutcomes[`${where}:${file}`] ?? 'pass'
if (outcome === 'timeout') yield { stream: 'stderr' as const, text: "timeout: sending signal TERM to command 'sh'\n" }
return { value: { code: outcome === 'pass' ? 0 : outcome === 'timeout' ? 124 : 1, signal: null } }
}
w.testCwd.push(String(e.cwd ?? ''))
w.testArgv.push([...e.argv])
const end = w.testEnds.shift() ?? 0
const fired = w.testFired.shift() ?? false
if (end === 'throw') throw new Error('docker: not found')
yield { stream: 'stdout' as const, text: end === 0 ? '67 passed\n' : 'test_modes_p4 ... still running\n' }
// the full run's report comes last, so a last line without a newline stays last
for (const piece of w.fullRunOutput) yield { stream: 'stdout' as const, text: piece }
if (fired) yield { stream: 'stderr' as const, text: "timeout: sending signal TERM to command 'sh'\n" }
if (w.testSplitLine) {
yield { stream: 'stderr' as const, text: 'timeout: sending sig' }
@@ -276,10 +367,27 @@ function world(on: On, w: World) {
return { value: { code: end, signal: null } }
})
on('agent.spawn', async (_$, e) => {
if (e.resume) {
w.resumed.push({ agentId: e.resume.agentId, prompt: e.prompt, model: e.model })
return { model: 'claude-opus-5-5', agentId: e.resume.agentId }
}
if (w.denySpawns > 0) {
w.denySpawns--
return { deny: 'spawn failed' }
}
w.spawned.push({ model: e.model, prompt: e.prompt })
return { model: e.model ?? 'inherit', agentId: `agent-${w.spawned.length}` }
return { model: w.engineModel ?? e.model ?? 'inherit', agentId: `agent-${w.spawned.length}` }
})
on('tool.call', { tool: 'SendMessage' }, async () =>
(w.sendFails ? { result: 'No agent by that name is running', isError: true } : { result: { success: true } }) as never,
)
on('turn.complete', async (_$, e) => ({ text: e.answer }))
on('agent.list', async () => ({
value: w.spawned
.map((_, i) => `agent-${i + 1}`)
.filter(id => w.status[id] !== 'gone') // 'gone': the engine dropped it
.map(id => ({ id, description: '', type: 'general-purpose', status: (w.status[id] ?? 'completed') as never })),
}))
on('prompt.submit', async (_$, e) => {
w.all.push(e.text)
if (/^\[model-router\] T\d+ (verifying on|no verification:)/.test(e.text)) w.handBacks.push(e.text)
@@ -289,7 +397,7 @@ function world(on: On, w: World) {
return clock
}
const fresh = (): World => ({ spawned: [], verifier: [], verdicts: [], told: [], handBacks: [], all: [], diffs: [], diffBases: [], testCwd: [], testArgv: [], testEnds: [], testFired: [], snap: 0 })
const fresh = (): World => ({ spawned: [], verifier: [], verdicts: [], told: [], handBacks: [], all: [], resumed: [], denySpawns: 0, verifierPrompts: [], verifierSystems: [], fullRunOutput: [], fileOutcomes: {}, fileRuns: [], worktree: [], running: 0, maxParallel: 0, status: {}, diffs: [], diffBases: [], testCwd: [], testArgv: [], testEnds: [], testFired: [], snap: 0 })
async function finish($: Engine, clock: { advance: (ms: number) => Promise<void> }, agentId: string, ranModel?: string, reason = 'answer') {
const usage = ranModel
@@ -643,3 +751,516 @@ test('TOOL-12: review and test agents get no hand-back line', async ($, on) => {
await finish($, clock, r.agentId as string, 'claude-opus-5-5')
expect(w.all).toEqual([])
})
// ---------- interim stops (TOOL-12 follow-up) ----------
/** A turn end of the agent while it is still alive: its status says so. */
async function interimStop($: Engine, clock: { advance: (ms: number) => Promise<void> }, w: World, agentId: string, status = 'waiting') {
w.status[agentId] = status
await ($.turn as unknown as { complete: (e: unknown) => Promise<unknown> }).complete({
answer: 'Waiting for the test suite.',
durationMs: 1,
isAborted: false,
turnId: 't',
agentId,
reason: 'answer',
usage: { model: 'claude-sonnet-5-5', input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 },
})
await clock.advance(1)
}
test('follow-up acceptance: 3 interim stops give exactly one verifying line and one verdict, both after the hand-back', async ($, on) => {
const w = fresh()
w.diffs = ['diff']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'MAINT-28 dev' } as never)
const id = r.agentId as string
for (let i = 0; i < 3; i++) {
await interimStop($, clock, w, id)
await clock.advance(40 * 60 * 1000) // the test suite runs 35-45 min
expect(w.all).toEqual([])
expect(w.verifier).toEqual([])
expect(w.diffBases).toEqual([])
}
w.status[id] = 'completed'
await finish($, clock, id, 'claude-sonnet-5-5')
await clock.advance(60_000)
expect(w.handBacks).toEqual(['[model-router] T1 verifying on Opus...'])
expect(w.verifier).toEqual(['opus'])
expect(w.told.length).toBe(1)
expect(w.all[1]).toContain('PASS from verifier opus')
})
test('follow-up: an interim stop of an Opus dev posts nothing; its hand-back posts the skip line once', async ($, on) => {
const w = fresh()
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: `SAFETY-135: decide how the staleness guard should recover. Clone: ${DEV}/`, description: 'dev' } as never)
const id = r.agentId as string
await interimStop($, clock, w, id)
await interimStop($, clock, w, id, 'idle')
await clock.advance(10 * 60 * 1000)
expect(w.all).toEqual([])
w.status[id] = 'completed'
await finish($, clock, id, 'claude-opus-5-5')
await clock.advance(60_000)
expect(w.all).toEqual(['[model-router] T1 no verification: dev ran on Opus. Go straight to review.'])
})
test('follow-up: a status that turns ended a moment after the turn is still caught', async ($, on) => {
const w = fresh()
w.diffs = ['diff']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
const id = r.agentId as string
w.status[id] = 'running' // not yet updated when the turn ends
await finish($, clock, id, 'claude-sonnet-5-5')
expect(w.all).toEqual([])
w.status[id] = 'completed'
await clock.advance(16_000)
expect(w.handBacks).toEqual(['[model-router] T1 verifying on Opus...'])
expect(w.verifier).toEqual(['opus'])
})
test('follow-up: an agent the engine already dropped counts as handed back', async ($, on) => {
const w = fresh()
w.diffs = ['diff']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
const id = r.agentId as string
w.status[id] = 'gone'
await finish($, clock, id, 'claude-sonnet-5-5')
await clock.advance(1)
expect(w.handBacks.length).toBe(1)
})
test('follow-up: a failed or killed run is a hand-back too', async () => {
for (const s of ['completed', 'failed', 'killed', undefined] as const) expect(turnEndKind(s)).toBe('handback')
for (const s of ['waiting', 'idle', 'running', 'pending'] as const) expect(turnEndKind(s)).toBe('interim')
})
test('follow-up: the hand-back uses the last turn: its model and how it ended', async ($, on) => {
const w = fresh()
w.diffs = ['diff']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
const id = r.agentId as string
await interimStop($, clock, w, id) // reported Sonnet
w.status[id] = 'completed'
await finish($, clock, id, 'claude-opus-5-5') // the last turn ran on Opus
await clock.advance(60_000)
expect(w.all).toEqual(['[model-router] T1 no verification: dev ran on Opus. Go straight to review. (routed to Sonnet, ran on Opus)'])
})
test('follow-up: each agent is judged by its own status, not by that of another agent', async ($, on) => {
const w = fresh()
w.diffs = ['diff']
const clock = world(on, w)
await $.agent.spawn({ prompt: devBrief(), description: 'finished dev' } as never) // agent-1: completed
const r = await $.agent.spawn({ prompt: devBrief(), description: 'busy dev' } as never)
await interimStop($, clock, w, r.agentId as string) // agent-2: waiting
await clock.advance(60_000)
expect(w.all).toEqual([])
expect(w.verifier).toEqual([])
})
// ---------- resumes, follow-ups and rewrites (#4, #6) ----------
const sendMessage = ($: Engine, to: string, message: string) =>
$.tool.call({ tool: 'SendMessage', to, message } as never)
const resume = ($: Engine, agentId: string, prompt: string) =>
$.agent.spawn({ prompt, description: 'resume', resume: { agentId }, model: 'sonnet' } as never)
test('#4: a resume keeps its task, routing and counts: no new Tn, no new routing', async ($, on) => {
const w = fresh()
w.diffs = ['d1', 'd2']
w.verdicts = ['{"verdict":"FAIL","issues":["x"]}', '{"verdict":"FAIL","issues":["y"]}']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'clamp' } as never)
const id = r.agentId as string
await finish($, clock, id, 'claude-sonnet-5-5')
expect(w.told[0]).toContain(`Resume agent ${id}`)
await resume($, id, 'Fix the issues: x')
expect(w.resumed[0]?.model).toBe('sonnet') // passed through untouched
expect(w.spawned.length).toBe(1)
await finish($, clock, id, 'claude-sonnet-5-5')
expect(w.handBacks[1]).toBe('[model-router] T1 verifying on Opus...')
expect(w.diffBases[1]).toBe('snap1') // round 2 diffs from the last verified round
expect(w.told[1]).toContain('round 2')
expect(w.told[1]).toContain('[router-rewrite T1]') // second fail counted: escalates
})
test('#4: a resumed Opus dev is labelled as Opus, with no "routed to" note', async ($, on) => {
const w = fresh()
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: `SAFETY-135: decide how the guard should recover. Clone: ${DEV}/`, description: 'dev' } as never)
const id = r.agentId as string
await finish($, clock, id, 'claude-opus-5-5')
await resume($, id, 'Docs-only round: update the README.')
await finish($, clock, id, 'claude-opus-5-5')
expect(w.handBacks).toEqual([
'[model-router] T1 no verification: dev ran on Opus. Go straight to review.',
'[model-router] T1 no verification: dev ran on Opus. Go straight to review.',
])
})
test('#4: the label compares with what the engine resolved at start, not what the router asked', async ($, on) => {
const w = fresh()
w.engineModel = 'claude-opus-5-5' // e.g. the agent type sets its own model
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'clamp' } as never)
expect(w.spawned[0]?.model).toBe('sonnet')
await finish($, clock, r.agentId as string, 'claude-opus-5-5')
expect(w.handBacks).toEqual(['[model-router] T1 no verification: dev ran on Opus. Go straight to review.'])
const log = (await $.command.run({ command: 'router', args: '' } as never)).text ?? ''
expect(log).toContain('opus')
})
test('#4: a router rewrite keeps the task number and its model', async ($, on) => {
const w = fresh()
w.diffs = ['d1', 'd2']
w.verdicts = ['{"verdict":"FAIL","issues":["x"]}', '{"verdict":"FAIL","issues":["y"]}']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'clamp' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
const rw = await $.agent.spawn({ prompt: '[router-rewrite T1]', description: 'rewrite', model: 'opus' } as never)
expect(w.spawned[1]?.model).toBe('opus')
await finish($, clock, rw.agentId as string, 'claude-opus-5-5')
expect(w.handBacks[2]).toBe('[model-router] T1 no verification: dev ran on Opus. Go straight to review.')
})
test('#4: a rewrite whose first start fails can be started again', async ($, on) => {
const w = fresh()
w.diffs = ['d1', 'd2']
w.verdicts = ['{"verdict":"FAIL","issues":["x"]}', '{"verdict":"FAIL","issues":["y"]}']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'clamp' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
w.denySpawns = 1
const first = await $.agent.spawn({ prompt: '[router-rewrite T1]', description: 'rewrite' } as never)
expect(first.deny).toBe('spawn failed')
const second = await $.agent.spawn({ prompt: '[router-rewrite T1]', description: 'rewrite' } as never)
expect(second.deny).toBe(undefined)
expect(w.spawned[1]?.model).toBe('opus')
expect(w.spawned[1]?.prompt).toContain('Previous attempts rejected')
})
test('#4: a rewrite marker with nothing stored is refused, not started without a brief', async ($, on) => {
const w = fresh()
world(on, w)
const r = await $.agent.spawn({ prompt: '[router-rewrite T325]', description: 'rewrite', model: 'opus' } as never)
expect(r.deny).toContain('no stored rewrite for T325')
expect(w.spawned).toEqual([])
})
test('#6: a resume message reaches the verifier as a later instruction, in scope', async ($, on) => {
const w = fresh()
w.diffs = ['d1', 'd2']
w.verdicts = ['{"verdict":"FAIL","issues":["x"]}']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'MAINT-24' } as never)
const id = r.agentId as string
await finish($, clock, id, 'claude-sonnet-5-5')
await resume($, id, 'Owner: add MAINT-25 to this PR as well.')
await finish($, clock, id, 'claude-sonnet-5-5')
const p = w.verifierPrompts[1] ?? ''
expect(p).toContain('## Later instructions to this agent')
expect(p).toContain('1. Owner: add MAINT-25 to this PR as well.')
expect(p).toContain('SAFETY-95: clamp the integrator') // the original brief stays
})
test('#6: SendMessage to a running agent, by id or by name, is a later instruction too', async ($, on) => {
const w = fresh()
w.diffs = ['d1']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'MAINT-28', name: 'maint28-dev' } as never)
const id = r.agentId as string
await sendMessage($, id, 'Owner decided: cap is 1.')
await sendMessage($, 'maint28-dev', 'Also cover the empty list.')
await sendMessage($, 'maint28-dev', 'Also cover the empty list.') // the same text counts once
await sendMessage($, 'someone-else', 'Not for this agent.')
await finish($, clock, id, 'claude-sonnet-5-5')
const p = w.verifierPrompts[0] ?? ''
expect(p).toContain('1. Owner decided: cap is 1.')
expect(p).toContain('2. Also cover the empty list.')
expect(p).not.toContain('3.')
expect(p).not.toContain('Not for this agent.')
})
test('#6: without later instructions the section is left out', async ($, on) => {
const w = fresh()
w.diffs = ['d1']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'clamp' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
expect(w.verifierPrompts[0] ?? '').not.toContain('Later instructions')
})
test('#4: a rewrite that started is used up: the same marker again is refused', async ($, on) => {
const w = fresh()
w.diffs = ['d1', 'd2']
w.verdicts = ['{"verdict":"FAIL","issues":["x"]}', '{"verdict":"FAIL","issues":["y"]}']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'clamp' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
await $.agent.spawn({ prompt: '[router-rewrite T1]', description: 'rewrite' } as never)
const again = await $.agent.spawn({ prompt: '[router-rewrite T1]', description: 'rewrite' } as never)
expect(again.deny).toContain('no stored rewrite for T1')
expect(w.spawned.length).toBe(2)
})
test('#6: a SendMessage that was not delivered is not a later instruction', async ($, on) => {
const w = fresh()
w.diffs = ['d1']
w.sendFails = true
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'clamp' } as never)
await sendMessage($, r.agentId as string, 'Never arrived.')
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
expect(w.verifierPrompts[0] ?? '').not.toContain('Never arrived.')
})
// ---------- reruns of reported files (#5) ----------
const RERUN = { testCommand: 'make t', testFileCommand: 'runone {file}' }
const T307_OUTPUT = [
'PASS test_a.py 12s\nTIMEOUT test_tariff24_weather.py 230s\nFAIL test_loop.py\nTIME',
'OUT test_slow.py 1500s\n', // a line split across pieces
'--- test_tariff24_weather.py (TIMEOUT)\n<tail>\n77 pass, 1 fail, 2 timeout (limit 1500 s)\n',
]
test('#5: reported lines are read, also when split across pieces; one entry per file', async () => {
const re = issueRegex('')
expect(parseIssueLine('TIMEOUT test_tariff24_weather.py 230s', re)).toEqual({ kind: 'TIMEOUT', file: 'test_tariff24_weather.py' })
expect(parseIssueLine('FAIL test_loop.py', re)).toEqual({ kind: 'FAIL', file: 'test_loop.py' })
expect(parseIssueLine('--- test_tariff24_weather.py (TIMEOUT)', re)).toBe(undefined)
expect(parseIssueLine('77 pass, 0 fail, 30 timeout (limit 1500 s)', re)).toBe(undefined)
expect(parseIssueLine('PASS test_a.py', re)).toBe(undefined)
// a custom pattern, and a broken one falls back to the default
expect(parseIssueLine('[timeout] x.py', issueRegex('^\\[(timeout|fail)\\]\\s+(\\S+)'))?.file).toBe('x.py')
expect(parseIssueLine('TIMEOUT y.py', issueRegex('(unclosed'))?.file).toBe('y.py')
})
test('#5: only safe file names reach the shell', async () => {
for (const ok of ['test_a.py', 'tests/test_b.py', 'a-b.c_d.py']) expect(isSafeFile(ok)).toBe(true)
for (const bad of ['a;rm -rf ~', '$(id).py', '../x.py', 'a/../b.py', '-rf', 'a b.py', 'x`y`.py', "a'.py"]) expect(isSafeFile(bad)).toBe(false)
expect(fileCommand('docker run img python3 {file} && echo {file}', 'test_a.py')).toBe('docker run img python3 test_a.py && echo test_a.py')
})
test('#5: what each rerun outcome means', async () => {
expect(rerunVerdict({ file: 'f', full: 'TIMEOUT', head: 'pass', base: 'not run' })).toContain('does not count')
expect(rerunVerdict({ file: 'f', full: 'FAIL', head: 'fail', base: 'fail' })).toContain('not caused by this change')
expect(rerunVerdict({ file: 'f', full: 'TIMEOUT', head: 'timeout', base: 'timeout' })).toContain('also times out on base')
expect(rerunVerdict({ file: 'f', full: 'FAIL', head: 'fail', base: 'pass' })).toContain('counts against the change')
expect(rerunVerdict({ file: 'f', full: 'FAIL', head: 'fail', base: 'not run' })).toContain('base not checked')
expect(rerunVerdict({ file: 'f', full: 'FAIL', head: 'error', base: 'not run' })).toContain('could not be rerun')
})
test('#5 acceptance: timed-out and failed files are rerun alone, failures again on base, and the verifier gets the table', { options: RERUN }, async ($, on) => {
const w = fresh()
w.diffs = ['diff']
w.testEnds = [1]
w.fullRunOutput = T307_OUTPUT
w.fileOutcomes = { 'head:test_loop.py': 'fail', 'base:test_loop.py': 'fail', 'head:test_slow.py': 'timeout', 'base:test_slow.py': 'pass' }
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'MAINT-28' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
expect([...w.fileRuns].sort()).toEqual([
'base:test_loop.py',
'base:test_slow.py',
'head:test_loop.py',
'head:test_slow.py',
'head:test_tariff24_weather.py',
])
const p = w.verifierPrompts[0] ?? ''
expect(p).toContain('### Files the full run reported, rerun one by one')
expect(p).toContain('| test_tariff24_weather.py | TIMEOUT | pass | not run | passes alone: does not count against the change |')
expect(p).toContain('| test_loop.py | FAIL | fail | fail | also fails on base: not caused by this change |')
expect(p).toContain('| test_slow.py | TIMEOUT | timeout | pass | times out alone, passes on base: counts against the change |')
// the base worktree sits next to the clone, and is removed and pruned afterwards
expect(w.worktree).toContain(`git -C ${DEV} worktree add --detach ${DEV}-verifier-base mergebase`)
expect(w.worktree.at(-2)).toBe(`git -C ${DEV} worktree remove --force ${DEV}-verifier-base`)
expect(w.worktree.at(-1)).toBe(`git -C ${DEV} worktree prune`)
})
test('#5: when every reported file passes alone, no base worktree is made', { options: RERUN }, async ($, on) => {
const w = fresh()
w.diffs = ['diff']
w.testEnds = [1]
w.fullRunOutput = ['TIMEOUT test_tariff24_weather.py 230s\n']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
expect(w.fileRuns).toEqual(['head:test_tariff24_weather.py'])
expect(w.worktree.filter(c => c.includes(' add '))).toEqual([])
})
test('#5: a base worktree that cannot be made leaves base as not run', { options: RERUN }, async ($, on) => {
const w = fresh()
w.diffs = ['diff']
w.testEnds = [1]
w.worktreeFails = true
w.fullRunOutput = ['FAIL test_loop.py\n']
w.fileOutcomes = { 'head:test_loop.py': 'fail' }
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
expect(w.fileRuns).toEqual(['head:test_loop.py'])
expect(w.verifierPrompts[0]).toContain('| test_loop.py | FAIL | fail | not run | fails alone; base not checked |')
})
test('#5: unsafe names and files past testRerunMax are not rerun, and said so', { options: { ...RERUN, testRerunMax: 1 } }, async ($, on) => {
const w = fresh()
w.diffs = ['diff']
w.testEnds = [1]
w.fullRunOutput = ['FAIL a;rm.py\nTIMEOUT test_a.py 1s\nTIMEOUT test_b.py 1s\n']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
expect(w.fileRuns).toEqual(['head:test_a.py'])
expect(w.verifierPrompts[0]).toContain('(2 more reported file(s) not rerun')
})
test('#5: without testFileCommand nothing is rerun', { options: { testCommand: 'make t' } }, async ($, on) => {
const w = fresh()
w.diffs = ['diff']
w.testEnds = [1]
w.fullRunOutput = T307_OUTPUT
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
expect(w.fileRuns).toEqual([])
expect(w.verifierPrompts[0] ?? '').not.toContain('rerun one by one')
})
test('#5: reruns run at most testRerunJobs at a time', { options: { ...RERUN, testRerunJobs: 2 } }, async ($, on) => {
const w = fresh()
w.diffs = ['diff']
w.testEnds = [1]
w.fullRunOutput = ['TIMEOUT t1.py 1s\nTIMEOUT t2.py 1s\nTIMEOUT t3.py 1s\nTIMEOUT t4.py 1s\nTIMEOUT t5.py 1s\n']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
expect(w.fileRuns.length).toBe(5)
expect(w.maxParallel).toBe(2)
})
test('#5: a custom pattern that also matches other lines still keeps only TIMEOUT and FAIL', async () => {
const re = issueRegex('^(\\w+)\\s+(\\S+)')
expect(parseIssueLine('PASS test_a.py', re)).toBe(undefined)
expect(parseIssueLine('timeout test_b.py', re)).toEqual({ kind: 'TIMEOUT', file: 'test_b.py' })
})
test('#5: a last line without a newline is read; a file reported twice is rerun once', { options: RERUN }, async ($, on) => {
const w = fresh()
w.diffs = ['diff']
w.testEnds = [1]
w.fullRunOutput = ['FAIL test_twice.py\nFAIL test_twice.py\n', 'TIMEOUT test_last.py 1500s']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
expect([...w.fileRuns].sort()).toEqual(['head:test_last.py', 'head:test_twice.py'])
})
test('#5: the verifier is told how to read the rerun table', { options: RERUN }, async ($, on) => {
const w = fresh()
w.diffs = ['diff']
w.testEnds = [1]
w.fullRunOutput = ['TIMEOUT test_a.py 1s\n']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
const sys = w.verifierSystems[0] ?? ''
expect(sys).toContain('When a table of files rerun one by one is given')
expect(sys).toContain('a file that passes alone does not count against the change')
expect(sys).toContain('not caused by this change')
expect(sys).toContain('Name every file you count against the change')
})
// ---------- ems-dev-tooling (test design, test-plan check) ----------
test('pipeline: a test designer keeps Opus; a named Sonnet is kept, Haiku never', async () => {
expect(decide({ subagentType: 'test-designer', prompt: 'Write the test plan', respectExplicit: false })).toEqual({ kind: 'fixed', tier: 'opus', reason: 'test-designer default' })
expect(decide({ subagentType: 'test-designer', named: 'sonnet', prompt: 'x', respectExplicit: false }).kind).toBe('fixed')
expect(decide({ subagentType: 'test-designer', named: 'haiku', prompt: 'find x', respectExplicit: false })).toEqual({ kind: 'fixed', tier: 'opus', reason: 'test-designer default' })
// a design clone path is no dev clone: a non-dev role, so never verified as dev work (0.5.1: was no role)
expect(parseBrief('Clone: /home/homelab/projects/goodwe/clones/EV-15-design').role).toBe('other')
})
test('pipeline: the verifier is told to check every T-id and N-id of the test plan', async ($, on) => {
const w = fresh()
w.diffs = ['diff --git a/x.py']
const clock = world(on, w)
const r = await $.agent.spawn({ prompt: devBrief('\n## Test plan\n| T1 | boundary | batteries above house | no alarm | yes |'), description: 'ev15' } as never)
await finish($, clock, r.agentId as string)
expect(w.verifierSystems[0]).toContain('every T-id must be implemented')
expect(w.verifierPrompts[0]).toContain('## Test plan')
})
// ---------- non-dev roles end to end (#7, EV-05) ----------
test('#7 acceptance: the test designer gets no line, also after a resume; the real dev gets T1 and a verdict', async ($, on) => {
const w = fresh()
w.diffs = ['diff --git a/ev.py']
const clock = world(on, w)
// the mock git answers in DEV only: the designer names that clone, so a misread would diff it
const d = await $.agent.spawn({ prompt: designBrief(DEV), description: 'EV-05 test design', model: 'opus' } as never)
expect(w.spawned[0]?.model).toBe('opus')
await finish($, clock, d.agentId as string, 'claude-opus-5-5')
await resume($, d.agentId as string, 'Add a case for a missing meter.')
await finish($, clock, d.agentId as string, 'claude-opus-5-5')
expect(w.all).toEqual([])
expect(w.verifier).toEqual([])
const dev = await $.agent.spawn({
prompt: `Pipeline-Role: dev\nClone: ${DEV} (Role: dev)\nBase branch: release/1.0.\nEV-05: add the meter staleness check as specified.`,
description: 'EV-05 dev',
} as never)
expect(w.spawned[1]?.model).toBe('sonnet')
await finish($, clock, dev.agentId as string, 'claude-sonnet-5-5')
expect(w.all[0]).toBe('[model-router] T1 verifying on Opus...')
expect(w.all[1]).toContain('PASS from verifier opus')
expect(w.all.length).toBe(2)
})
test('#7: a test-designer agent type that only names the dev clone is no dev either', async ($, on) => {
const w = fresh()
w.diffs = ['diff']
const clock = world(on, w)
const d = await $.agent.spawn({ prompt: `Write the tests for the work in ${DEV}/.`, description: 'design', subagentType: 'test-designer', model: 'haiku' } as never)
expect(w.spawned[0]?.model).toBe('opus') // never Haiku
await finish($, clock, d.agentId as string, 'claude-opus-5-5')
expect(w.all).toEqual([])
expect(w.verifier).toEqual([])
const log = (await $.command.run({ command: 'router', args: '' } as never)).text ?? ''
expect(log).toContain('A1')
expect(log).toContain('A1 test-designer opus')
expect(log).not.toMatch(/\bother\b/)
})
test('#7: an unnamed test designer runs on opus, unverified', async ($, on) => {
const w = fresh()
w.diffs = ['diff']
const clock = world(on, w)
const d = await $.agent.spawn({ prompt: designBrief(), description: 'EV-05 test design' } as never)
expect(w.spawned[0]?.model).toBe('opus')
await finish($, clock, d.agentId as string, 'claude-sonnet-5-5')
expect(w.all).toEqual([])
expect(w.verifier).toEqual([])
})
test('#7: a test designer is no saving and gets an A number; the log shows its role', async ($, on) => {
const w = fresh()
const clock = world(on, w)
const d = await $.agent.spawn({ prompt: designBrief(), description: 'EV-05 test design', model: 'opus' } as never)
await finish($, clock, d.agentId as string, 'claude-opus-5-5')
const savings = (await $.command.run({ command: 'router', args: 'savings' } as never)).text ?? ''
expect(savings).toBe('savings: no agent runs recorded yet.')
const log = (await $.command.run({ command: 'router', args: '' } as never)).text ?? ''
expect(log).toContain('A1')
expect(log).toContain('test-design')
expect(log).not.toContain('T1')
})