Fix TOOL-12: one line on every dev hand-back, by the model that ran

An Opus-run dev got no verification and no message, so silence meant
both "still verifying" and "will never verify" (SAFETY-135 waited
overnight, SAFETY-126 8 h). Verdicts were also labelled with the
routing choice, not the model the run used.

- Every dev hand-back posts one line at once:
  "[model-router] Tn verifying on Opus..." (a verdict follows), or
  "[model-router] Tn no verification: dev ran on Opus. Go straight to
  review." A run that ends with an error or names no clone says so.
- Each round is decided and labelled by the model the run reported
  (its end usage), the routing choice only as fallback. A difference
  is shown as "(routed to X, ran on Y)", also on later rounds.
- 8 new tests; each new rule was also broken on purpose and caught,
  apart from one equivalent change.

Version 0.4.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "model-router",
-  "version": "0.4.5",
+  "version": "0.4.6",
   "description": "Routes each subagent to Haiku, Sonnet or Opus by how well-defined its brief is, verifies dev work below Opus in its own clone on a different model, sends failures back to the same agent then a stronger one, and estimates the savings against Opus.",
   "author": {
     "name": "Glenn Schrooyen"

README.md

@@ -8,6 +8,7 @@ A Claude Code plugin that picks a model for every subagent, checks dev work on a
 |---|---|
 | Route | Dev agents run on **Sonnet** when the brief decides the approach (Expected / acceptance criteria), on **Opus** when the design is still open. Review and test agents keep the model they are given (Opus when none). |
 | Verify | When a dev agent below Opus finishes a round, the router diffs its clone against the merge base, runs the test command in that clone, and has **Opus** judge the change. |
+| Hand-back line | Every dev hand-back gets one line at once: `[model-router] Tn verifying on Opus...` (a verdict follows), or `[model-router] Tn no verification: dev ran on Opus. Go straight to review.` Decided by the model the run actually used, not only the routing choice; a difference is shown as `(routed to X, ran on Y)`. |
 | On FAIL | 1st: resume the same agent with the issues. 2nd: a new agent one model up, in the same clone. After that: handed back to the manager. |
 | Not verified | A test timeout, a test command that cannot run, a missing clone or merge base, or a verdict that cannot be read even after asking once more: reported as NOT VERIFIED, never as a FAIL. |
 | Savings | `/router savings` prices every run on the model it ran on and compares it with the same tokens on Opus 5.5. |
@@ -58,7 +59,7 @@ claude plugin validate .
 claude plugin test .
 ```
 
-54 tests. Every routing, verification, cost and timeout rule was also checked by breaking it on purpose (66 mutations in v0.4.4, plus 13 for the verdict reader in v0.4.5; all caught by a test).
+62 tests. Every routing, verification, cost and timeout rule was also checked by breaking it on purpose (66 mutations in v0.4.4, plus 13 for the verdict reader in v0.4.5 and 12 for the hand-back line in v0.4.6; all caught by a test).
 
 ## Known limits
 

hooks/register.ts

@@ -19,6 +19,8 @@ import {
   parseBrief,
   parsePatterns,
   parseVerdict,
+  handBackLine,
+  ranTier,
   safetyHit,
   skipReason,
   validTimeout,
@@ -32,7 +34,10 @@ import {
 /** One agent the router started, kept from spawn to its last verified round. */
 type AgentRec = {
   taskId: string
+  /** The model the last round actually ran on (the routing choice until a round reports one). */
   tier: Tier
+  /** The model the router chose at spawn; tier can differ when the run used another. */
+  routed?: Tier
   role?: Role
   clone?: string
   baseBranch?: string
@@ -124,7 +129,7 @@ The model-router mod picks each subagent's model and verifies dev work before it
 - Every dev, review and test brief carries one line \`Clone: <path>\` (add \`(Role: dev|review|test)\` if the path does not end in the role). The router reads the role from that line only; a brief naming several clones without it runs on Opus, unverified.
 - Dev agents are routed: Sonnet when the brief decides the approach (Expected / acceptance criteria), Opus when the design is still open. Naming a model on a dev brief has no effect. Give each dev brief its clone path and base branch.
 - Review and test agents keep the model you name (Opus when you name none).
-- When a dev agent below Opus finishes a round, the router verifies its clone's diff on Opus and posts a "[model-router]" verdict. Wait for that verdict before opening or advancing the PR.
+- Every dev hand-back gets one "[model-router] Tn" line at once. "verifying on Opus...": wait for the verdict before advancing the PR. "no verification: dev ran on Opus. Go straight to review.": no verdict will come. Lines are decided by the model the dev actually ran on.
 - On "[model-router] ... FAIL ... resume agent", resume that same agent with SendMessage, passing the issues. On "start a new agent ... [router-rewrite Tn]", stop the old agent and call Agent with exactly that prompt. On "handed back", stop and show the issues to the person. Do not fix rejected work yourself.`
 
 // ---------- store helpers (top-level, as the engine requires for `$`) ----------
@@ -501,14 +506,21 @@ async function skipRound($: EngineInterface, agentId: string, rec: AgentRec, out
     reason: outcome,
     outcome,
   })
-  if (rec.role === 'dev' && outcome === 'no-clone')
-    await tell($, `[model-router] ${rec.taskId} (agent ${agentId}): NOT VERIFIED - the brief names no clone path.`)
 }
 
-async function runRound($: EngineInterface, agentId: string, rec: AgentRec, ended: string, cfg: Settings) {
+/**
-  const skip = skipReason(rec, ended)
+ * One finished run of a tracked agent. Decides on the model that actually ran,
-  if (skip) await skipRound($, agentId, rec, skip)
+ * says at once what happens next for a dev, then verifies or logs the skip.
-  else await verifyRound($, agentId, rec, cfg)
+ */
+async function runRound($: EngineInterface, agentId: string, rec: AgentRec, ended: string, reported: string | undefined, cfg: Settings) {
+  const routed = rec.routed ?? rec.tier
+  const ran = ranTier(reported, routed)
+  const round: AgentRec = { ...rec, routed, tier: ran.tier }
+  const skip = skipReason(round, ended)
+  const line = handBackLine({ taskId: rec.taskId, role: rec.role, ran: ran.tier, routed, skip, ended })
+  if (line) await tell($, line)
+  if (skip) await skipRound($, agentId, round, skip)
+  else await verifyRound($, agentId, round, cfg)
 }
 
 // ---------- the hooks ----------
@@ -612,6 +624,7 @@ export const register: Register = (on, options) => {
       await putIn<AgentRec>($, S.agents, ran.agentId, {
         taskId,
         tier,
+        routed: tier,
         role: brief.role,
         clone: brief.clone,
         baseBranch: brief.baseBranch,
@@ -656,7 +669,8 @@ export const register: Register = (on, options) => {
     const rec = (await getMap<AgentRec>($, S.agents))[agentId]
     if (!rec) return out
     // Verification takes minutes: run it after this event has gone on.
-    $.clock.after(0, () => void runRound($, agentId, rec, e.reason, cfg))
+    const reported = e.usage?.model
+    $.clock.after(0, () => void runRound($, agentId, rec, e.reason, reported, cfg))
     return out
   })
 

hooks/router.ts

@@ -262,3 +262,39 @@ export function validTimeout(raw: unknown): number {
   return Number.isFinite(n) && n >= 1 ? Math.round(n) : 1800
 }
 
+
+// ---------- the hand-back line (TOOL-12) ----------
+
+const LABEL: Record<Tier, string> = { haiku: 'Haiku', sonnet: 'Sonnet', opus: 'Opus', fable: 'Fable' }
+
+/**
+ * The model a run actually used, from the model id its end reports; the
+ * routing choice only when the run reported none.
+ */
+export const ranTier = (reportedModel: string | undefined, routed: Tier): { tier: Tier; isMismatch: boolean } => {
+  const seen = asTier(reportedModel)
+  return seen ? { tier: seen, isMismatch: seen !== routed } : { tier: routed, isMismatch: false }
+}
+
+/**
+ * The one line every dev hand-back gets at once, so silence never stands for
+ * both "still verifying" and "will never verify". Undefined: no line (not a dev).
+ */
+export const handBackLine = (args: {
+  taskId: string
+  role?: Role
+  ran: Tier
+  routed: Tier
+  skip?: Skip
+  ended: string
+}): string | undefined => {
+  const { taskId, role, ran, routed, skip, ended } = args
+  if (role !== 'dev') return undefined
+  const note = ran !== routed ? ` (routed to ${LABEL[routed]}, ran on ${LABEL[ran]})` : ''
+  const head = `[model-router] ${taskId}`
+  if (!skip) return `${head} verifying on ${LABEL[verifierFor(ran)]}...${note}`
+  if (skip === 'opus-unverified') return `${head} no verification: dev ran on ${LABEL[ran]}. Go straight to review.${note}`
+  if (skip === 'agent-ended') return `${head} no verification: the dev run ended without an answer (${ended}).${note}`
+  if (skip === 'no-clone') return `${head} no verification: the brief names no clone path.${note}`
+  return undefined
+}

tests/router.test.ts

@@ -11,6 +11,8 @@ import {
   skipReason,
   validTimeout,
   verifierFor,
+  handBackLine,
+  ranTier,
 } from '../hooks/router'
 
 const DEV = '/home/homelab/projects/goodwe/clones/SAFETY-95-dev'
@@ -180,6 +182,31 @@ test('TOOL-02: garbage, prose and odd verdicts are unreadable, never a FAIL', as
   expect(long.isReadable ? '' : long.preview).toBe('x'.repeat(300))
 })
 
+// ---------- the hand-back line (TOOL-12) ----------
+
+test('TOOL-12: the model that ran wins over the routing choice', async () => {
+  expect(ranTier('claude-opus-5-5', 'sonnet')).toEqual({ tier: 'opus', isMismatch: true })
+  expect(ranTier('claude-sonnet-5-5', 'opus')).toEqual({ tier: 'sonnet', isMismatch: true })
+  expect(ranTier('claude-sonnet-5-5', 'sonnet')).toEqual({ tier: 'sonnet', isMismatch: false })
+  expect(ranTier(undefined, 'sonnet')).toEqual({ tier: 'sonnet', isMismatch: false })
+  expect(ranTier('some-other-model', 'sonnet')).toEqual({ tier: 'sonnet', isMismatch: false })
+})
+
+test('TOOL-12: the exact lines', async () => {
+  expect(handBackLine({ taskId: 'T7', role: 'dev', ran: 'sonnet', routed: 'sonnet', ended: 'answer' })).toBe('[model-router] T7 verifying on Opus...')
+  expect(handBackLine({ taskId: 'T7', role: 'dev', ran: 'opus', routed: 'opus', skip: 'opus-unverified', ended: 'answer' })).toBe(
+    '[model-router] T7 no verification: dev ran on Opus. Go straight to review.',
+  )
+  expect(handBackLine({ taskId: 'T7', role: 'dev', ran: 'opus', routed: 'sonnet', skip: 'opus-unverified', ended: 'answer' })).toBe(
+    '[model-router] T7 no verification: dev ran on Opus. Go straight to review. (routed to Sonnet, ran on Opus)',
+  )
+  expect(handBackLine({ taskId: 'T7', role: 'dev', ran: 'sonnet', routed: 'sonnet', skip: 'agent-ended', ended: 'error' })).toContain('ended without an answer (error)')
+  expect(handBackLine({ taskId: 'T7', role: 'dev', ran: 'sonnet', routed: 'sonnet', skip: 'no-clone', ended: 'answer' })).toContain('no clone path')
+  expect(handBackLine({ taskId: 'T7', role: 'review', ran: 'opus', routed: 'opus', skip: 'review-agent', ended: 'answer' })).toBe(undefined)
+  expect(handBackLine({ taskId: 'T7', role: 'test', ran: 'opus', routed: 'opus', skip: 'test-agent', ended: 'answer' })).toBe(undefined)
+  expect(handBackLine({ taskId: 'T7', ran: 'sonnet', routed: 'sonnet', ended: 'answer' })).toBe(undefined)
+})
+
 // ---------- the hooks, with the engine mocked beneath ----------
 
 type World = {
@@ -187,6 +214,10 @@ type World = {
   verifier: string[]
   verdicts: string[]
   told: string[]
+  /** TOOL-12 hand-back lines, kept apart so `told` holds the verdicts. */
+  handBacks: string[]
+  /** Every message in the order it was sent. */
+  all: string[]
   diffs: string[]
   diffBases: string[]
   testCwd: string[]
@@ -250,16 +281,21 @@ function world(on: On, w: World) {
   })
   on('turn.complete', async (_$, e) => ({ text: e.answer }))
   on('prompt.submit', async (_$, e) => {
-    w.told.push(e.text)
+    w.all.push(e.text)
+    if (/^\[model-router\] T\d+ (verifying on|no verification:)/.test(e.text)) w.handBacks.push(e.text)
+    else w.told.push(e.text)
     return { text: e.text }
   })
   return clock
 }
 
-const fresh = (): World => ({ spawned: [], verifier: [], verdicts: [], told: [], diffs: [], diffBases: [], testCwd: [], testArgv: [], testEnds: [], testFired: [], snap: 0 })
+const fresh = (): World => ({ spawned: [], verifier: [], verdicts: [], told: [], handBacks: [], all: [], diffs: [], diffBases: [], testCwd: [], testArgv: [], testEnds: [], testFired: [], snap: 0 })
 
-async function finish($: Engine, clock: { advance: (ms: number) => Promise<void> }, agentId: string) {
+async function finish($: Engine, clock: { advance: (ms: number) => Promise<void> }, agentId: string, ranModel?: string, reason = 'answer') {
-  await ($.turn as unknown as { complete: (e: unknown) => Promise<unknown> }).complete({ answer: 'done', durationMs: 1, isAborted: false, turnId: 't', agentId, reason: 'answer' })
+  const usage = ranModel
+    ? { usage: { model: ranModel, input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 } }
+    : {}
+  await ($.turn as unknown as { complete: (e: unknown) => Promise<unknown> }).complete({ answer: 'done', durationMs: 1, isAborted: false, turnId: 't', agentId, reason, ...usage })
   await clock.advance(1)
 }
 
@@ -535,3 +571,75 @@ test('TOOL-02: unreadable twice is handed back, no fail counted, no rewrite', as
   expect(w.told[1]).toContain('Resume agent')
   expect(w.told[1]).not.toContain('[router-rewrite')
 })
+
+// ---------- TOOL-12 end to end ----------
+
+const opusBrief = () =>
+  `SAFETY-135: decide how the staleness guard should recover. Clone: ${DEV}/\nBase branch: release/1.0.`
+
+test('TOOL-12 acceptance 1: an Opus-routed dev gets the skip line at hand-back, and no verdict later', async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: opusBrief(), description: 'SAFETY-135 dev' } as never)
+  expect(w.spawned[0]?.model).toBe('opus')
+  await finish($, clock, r.agentId as string, 'claude-opus-5-5')
+  await clock.advance(60_000)
+  expect(w.all).toEqual(['[model-router] T1 no verification: dev ran on Opus. Go straight to review.'])
+  expect(w.verifier).toEqual([])
+})
+
+test('TOOL-12 acceptance 2: a Sonnet-routed dev gets "verifying", then the verdict', async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'clamp' } as never)
+  expect(w.spawned[0]?.model).toBe('sonnet')
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  expect(w.all[0]).toBe('[model-router] T1 verifying on Opus...')
+  expect(w.all[1]).toContain('PASS from verifier opus')
+  expect(w.all.length).toBe(2)
+})
+
+test('TOOL-12 acceptance 3a: routed Sonnet but ran on Opus is reported and handled as Opus', async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'clamp' } as never)
+  await finish($, clock, r.agentId as string, 'claude-opus-5-5')
+  expect(w.all).toEqual(['[model-router] T1 no verification: dev ran on Opus. Go straight to review. (routed to Sonnet, ran on Opus)'])
+  expect(w.verifier).toEqual([])
+})
+
+test('TOOL-12 acceptance 3b: routed Opus but ran on Sonnet is verified, and labelled Sonnet', async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  w.verdicts = ['{"verdict":"FAIL","issues":["x"]}']
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: opusBrief(), description: 'SAFETY-135 dev' } as never)
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  expect(w.all[0]).toBe('[model-router] T1 verifying on Opus... (routed to Opus, ran on Sonnet)')
+  expect(w.verifier).toEqual(['opus'])
+  expect(w.told[0]).toContain('(agent agent-1, sonnet, round 1)')
+  // the resumed round still reports against the original routing choice
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  expect(w.handBacks[1]).toBe('[model-router] T1 verifying on Opus... (routed to Opus, ran on Sonnet)')
+})
+
+test('TOOL-12: a dev run that ends with an error says so, no verification', async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'clamp' } as never)
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5', 'error')
+  expect(w.all).toEqual(['[model-router] T1 no verification: the dev run ended without an answer (error).'])
+  expect(w.verifier).toEqual([])
+})
+
+test('TOOL-12: review and test agents get no hand-back line', async ($, on) => {
+  const w = fresh()
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: `Review. Clone: ${REVIEW}/`, description: 'review', model: 'opus' } as never)
+  await finish($, clock, r.agentId as string, 'claude-opus-5-5')
+  expect(w.all).toEqual([])
+})