Fix #5 : rerun reported files alone, and on base, before judging

The full suite run can time out files that pass alone (an overloaded shared host, a suite-wide deadline in the runner): MAINT-28 T307 was judged on 30 TIMEOUTs, T325 on a file that passes alone in 230 s. - With testFileCommand set, every file the full run reports as TIMEOUT or FAIL (testIssuePattern, default ^(TIMEOUT|FAIL)\s+(\S+); lines are read as the run streams) is rerun alone in the clone, testRerunJobs at a time, testFileTimeoutSec each, at most testRerunMax files. - A file that still fails or times out alone runs again on the base, in a git worktree next to the clone (<clone>-verifier-base), which is removed and pruned afterwards. Without a worktree, base stays "not run". - The verifier gets a table (file, full run, alone on head, on base, meaning) and is told: passes alone does not count; fails on base too is not caused by the change; name every file it counts. - Only file names of letters, digits and _ . - / reach the shell; no ".." parts, no leading "-". - 12 new tests; each of the 23 new rules was also broken on purpose and caught. Version 0.4.9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MdqPWvuDpbpRvDEdURoAwo
2026-11-14 00:09:18 +00:00
5 changed files with 408 additions and 10 deletions
@@ -1,6 +1,6 @@
 {
  "name": "model-router",
-  "version": "0.4.8",
+  "version": "0.4.9",
  "description": "Routes each subagent to Haiku, Sonnet or Opus by how well-defined its brief is, verifies dev work below Opus in its own clone on a different model, sends failures back to the same agent then a stronger one, and estimates the savings against Opus.",
  "author": {
    "name": "Glenn Schrooyen"
@@ -18,6 +18,36 @@
      "description": "How long the test command may run in the clone. A run that times out is reported as NOT VERIFIED, never as a FAIL.",
      "default": 1800
    },
+    "testFileCommand": {
+      "type": "string",
+      "title": "Command that runs one test file",
+      "description": "Run in the clone, with {file} replaced by a file the full run reported as TIMEOUT or FAIL. Each such file is rerun alone, and on the base branch if it still fails. Empty: no reruns.",
+      "default": ""
+    },
+    "testFileTimeoutSec": {
+      "type": "number",
+      "title": "Time limit per rerun file (seconds)",
+      "description": "A file still running after this counts as timing out alone.",
+      "default": 900
+    },
+    "testRerunJobs": {
+      "type": "number",
+      "title": "Reruns at a time",
+      "description": "How many reported files are rerun in parallel.",
+      "default": 6
+    },
+    "testRerunMax": {
+      "type": "number",
+      "title": "Most files to rerun",
+      "description": "Reported files beyond this are listed as not rerun.",
+      "default": 40
+    },
+    "testIssuePattern": {
+      "type": "string",
+      "title": "Pattern of a reported file line",
+      "description": "Regular expression: group 1 is TIMEOUT or FAIL, group 2 the file. Empty: ^(TIMEOUT|FAIL)\\s+(\\S+)",
+      "default": ""
+    },
    "defaultBaseBranch": {
      "type": "string",
      "title": "Base branch when the brief names none",
@@ -7,6 +7,7 @@ A Claude Code plugin that picks a model for every subagent, checks dev work on a
 | Step | Behaviour |
 |---|---|
 | Route | Dev agents run on **Sonnet** when the brief decides the approach (Expected / acceptance criteria), on **Opus** when the design is still open. Review and test agents keep the model they are given (Opus when none). |
+| Rerun reported files | With `testFileCommand` set, every file the full run reports as `TIMEOUT` or `FAIL` is rerun alone in the clone. A file that still fails or times out alone runs again on the base branch, in a worktree next to the clone (`<clone>-verifier-base`, removed afterwards). The verifier gets a table: a file that passes alone does not count, one that also fails on base is not caused by the change. |
 | Later instructions | Every message sent to a dev agent after its brief (a SendMessage, or the message that resumes it) reaches the verifier as part of the task, so scope added later is judged as in scope. A resumed agent keeps its task number, routing and failure count. |
 | Verify | When a dev agent below Opus finishes a round, the router diffs its clone against the merge base, runs the test command in that clone, and has **Opus** judge the change. |
 | Hand-back line | Every dev hand-back gets one line at once: `[model-router] Tn verifying on Opus...` (a verdict follows), or `[model-router] Tn no verification: dev ran on Opus. Go straight to review.` Only the real hand-back counts: an interim stop while the dev waits on its own background work (e.g. a long test suite) posts nothing and starts no verifier. Decided by the model the run actually used, not only the routing choice; a difference is shown as `(routed to X, ran on Y)`. |
@@ -39,6 +40,11 @@ Or for development: `claude --plugin-dir /path/to/model-router`.
 |---|---|---|
 | `testCommand` | empty | Run in the agent's clone; output goes to the verifier |
 | `testTimeoutSec` | 1800 | Wrapped in GNU `timeout --verbose`; a timeout is NOT VERIFIED |
+| `testFileCommand` | empty | Runs one file, `{file}` replaced by a reported file name; only names of letters, digits, `_ . - /` are run |
+| `testFileTimeoutSec` | 900 | Limit per rerun file |
+| `testRerunJobs` | 6 | Reruns at a time |
+| `testRerunMax` | 40 | Most files rerun per round |
+| `testIssuePattern` | `^(TIMEOUT\|FAIL)\s+(\S+)` | Group 1 the kind, group 2 the file |
 | `defaultBaseBranch` | `release/1.0` | After the PR lookup and the brief's own base branch |
 | `prBaseCommand` | empty | Optional command that prints the PR's base branch |
 | `maxRewrites` | 2 | Failed verdicts before handing back |
@@ -60,7 +66,7 @@ claude plugin validate .
 claude plugin test .
 ```

-80 tests. Every routing, verification, cost and timeout rule was also checked by breaking it on purpose (66 mutations in v0.4.4, plus 13 for the verdict reader in v0.4.5 12 for the hand-back line in v0.4.6 10 for interim stops in v0.4.7 and 14 for resumes, later instructions and rewrites in v0.4.8; all caught by a test).
+92 tests. Every routing, verification, cost and timeout rule was also checked by breaking it on purpose (66 mutations in v0.4.4, plus 13 for the verdict reader in v0.4.5 12 for the hand-back line in v0.4.6 10 for interim stops in v0.4.7 14 for resumes, later instructions and rewrites in v0.4.8 and 23 for file reruns in v0.4.9; all caught by a test).

 ## Known limits

@@ -27,6 +27,15 @@ import {
  safetyHit,
  skipReason,
  validTimeout,
+  fileCommand,
+  isSafeFile,
+  issueRegex,
+  parseIssueLine,
+  positiveInt,
+  rerunTable,
+  type FileKind,
+  type FileRun,
+  type RerunRow,
  verifierFor,
  type Role,
  type Tier,
@@ -127,6 +136,7 @@ const VERIFIER_SYSTEM = `You are an adversarial verifier. Another model wrote th
 Judge only whether the change correctly and completely does the task, without breaking anything nearby.
 Look for: requirements or acceptance criteria missed, wrong logic, unhandled edge cases, tests that cannot fail, tests weakened or deleted, safety checks bypassed, unrelated changes.
 Scope added in "Later instructions" was asked for after the brief: judge it as in scope, not as unrequested work.
+When a table of files rerun one by one is given: a file that passes alone does not count against the change (the full run was overloaded or flaky); one that also fails or times out on base is not caused by this change: mention it, do not fail for it. Name every file you count against the change in issues.
 Do not fail a change for style alone. A later PR review and mutation testing follow you; you are the cheap first gate.

 Reply with JSON only:
@@ -260,18 +270,37 @@ async function chargeOverhead($: EngineInterface, taskId: string, alias: Tier, u
  await updateIn<TaskCost>($, S.costs, taskId, old => addOverhead(old ?? emptyTask(taskId), modelId, u, prices))
 }

-type TestRun = { status: 'ran' | 'timeout' | 'error'; code: number | null; tail: string; seconds: number }
+type TestRun = {
+  status: 'ran' | 'timeout' | 'error'
+  code: number | null
+  tail: string
+  seconds: number
+  /** Files the run reported as TIMEOUT or FAIL, in order, each once. */
+  issues: { kind: FileKind; file: string }[]
+}

 /**
 * Runs the test command in the clone, streamed: `$.process.run` stops at ten
 * minutes, a stream has no limit of its own. GNU `timeout` does the kill.
 * A timeout or a run that could not start is no verdict on the code.
 */
-async function runTests($: EngineInterface, cmd: string, cwd: string, timeoutSec: number): Promise<TestRun> {
+async function runTests($: EngineInterface, cmd: string, cwd: string, timeoutSec: number, issueRe?: RegExp): Promise<TestRun> {
  const started = await $.clock.now()
  const seconds = async () => Math.round(((await $.clock.now()) - started) / 1000)
  let out = ''
  let errTail = ''
+  // reported files are read line by line as the run streams, not from the kept tail
+  const issues: TestRun['issues'] = []
+  const partial: Record<string, string> = { stdout: '', stderr: '' }
+  const readLines = (stream: string, text: string, isLast = false) => {
+    if (!issueRe) return
+    const lines = ((partial[stream] ?? '') + text).split('\n')
+    partial[stream] = isLast ? '' : lines.pop() ?? ''
+    for (const line of lines) {
+      const hit = parseIssueLine(line, issueRe)
+      if (hit && !issues.some(i => i.file === hit.file)) issues.push(hit)
+    }
+  }
  try {
    const stream = $.process.spawn({
      argv: ['timeout', '--verbose', '--kill-after=30', String(Math.max(1, Math.round(timeoutSec))), 'sh', '-c', cmd],
@@ -280,7 +309,10 @@ async function runTests($: EngineInterface, cmd: string, cwd: string, timeoutSec
    for await (const piece of stream) {
      out = (out + piece.text).slice(-65_536)
      if (piece.stream === 'stderr') errTail = (errTail + piece.text).slice(-65_536)
+      readLines(piece.stream, piece.text)
    }
+    readLines('stdout', '', true)
+    readLines('stderr', '', true)
    // GNU timeout --verbose says so on stderr when, and only when, it fires.
    // Read from the joined stderr, so a line split across pieces still counts.
    const isFired = errTail.includes('timeout: sending signal')
@@ -289,17 +321,81 @@ async function runTests($: EngineInterface, cmd: string, cwd: string, timeoutSec
    const took = await seconds()
    // Only a timeout that fired is a timeout: a 137 without it is the code's own
    // (e.g. out of memory) and goes to the verifier as a failed run.
-    if (isFired) return { status: 'timeout', code: end.code, tail, seconds: took }
-    if (end.code === 126 || end.code === 127) return { status: 'error', code: end.code, tail, seconds: took }
-    return { status: 'ran', code: end.code, tail, seconds: took }
+    if (isFired) return { status: 'timeout', code: end.code, tail, seconds: took, issues }
+    if (end.code === 126 || end.code === 127) return { status: 'error', code: end.code, tail, seconds: took, issues }
+    return { status: 'ran', code: end.code, tail, seconds: took, issues }
  } catch (err) {
-    return { status: 'error', code: null, tail: String(err).slice(0, 500), seconds: await seconds() }
+    return { status: 'error', code: null, tail: String(err).slice(0, 500), seconds: await seconds(), issues }
  }
 }

+/** The result of one file run alone. */
+function fileRunOf(t: TestRun): FileRun {
+  if (t.status === 'timeout') return 'timeout'
+  if (t.status === 'error') return 'error'
+  return t.code === 0 ? 'pass' : 'fail'
+}
+
+/** Runs `work` over the items, at most `jobs` at a time, keeping their order. */
+async function inPool<T, R>(items: readonly T[], jobs: number, work: (item: T) => Promise<R>): Promise<R[]> {
+  const out: R[] = new Array(items.length)
+  let next = 0
+  const lane = async () => {
+    while (next < items.length) {
+      const i = next++
+      out[i] = await work(items[i] as T)
+    }
+  }
+  await Promise.all(Array.from({ length: Math.max(1, Math.min(jobs, items.length)) }, lane))
+  return out
+}
+
+/**
+ * Reruns the files a full run reported, one by one in the clone; those that
+ * still fail or time out alone run again on the base, checked out next to the
+ * clone (`<clone>-verifier-base`, removed afterwards).
+ */
+async function rerunReported($: EngineInterface, rec: AgentRec, issues: TestRun['issues'], cfg: Settings): Promise<RerunRow[]> {
+  const cwd = rec.clone as string
+  const picked = issues.filter(i => isSafeFile(i.file)).slice(0, cfg.testRerunMax)
+  const rows: RerunRow[] = picked.map(i => ({ file: i.file, full: i.kind, head: 'not run', base: 'not run' }))
+  const runAll = (dir: string, which: RerunRow[]) =>
+    inPool(which, cfg.testRerunJobs, async r =>
+      fileRunOf(await runTests($, fileCommand(cfg.testFileCommand, r.file), dir, cfg.testFileTimeoutSec)),
+    )
+
+  $.ui.status(`router: ${rec.taskId} rerunning ${rows.length} reported file(s) alone`)
+  const head = await runAll(cwd, rows)
+  rows.forEach((r, i) => (r.head = head[i] ?? 'error'))
+
+  const onBase = rows.filter(r => r.head === 'fail' || r.head === 'timeout')
+  if (onBase.length === 0) return rows
+  const { base } = await firstBase($, rec, cfg.prBaseCommand, cfg.defaultBaseBranch)
+  if (!base) return rows
+  const tree = `${cwd.replace(/\/+$/, '')}-verifier-base`
+  await sh($, ['git', '-C', cwd, 'worktree', 'remove', '--force', tree], cwd)
+  await sh($, ['git', '-C', cwd, 'worktree', 'prune'], cwd)
+  const added = await sh($, ['git', '-C', cwd, 'worktree', 'add', '--detach', tree, base], cwd, 120_000)
+  if (!added.ok) return rows
+  try {
+    $.ui.status(`router: ${rec.taskId} rerunning ${onBase.length} file(s) on base`)
+    const results = await runAll(tree, onBase)
+    onBase.forEach((r, i) => (r.base = results[i] ?? 'error'))
+  } finally {
+    await sh($, ['git', '-C', cwd, 'worktree', 'remove', '--force', tree], cwd, 120_000)
+    await sh($, ['git', '-C', cwd, 'worktree', 'prune'], cwd)
+  }
+  return rows
+}
+
 type Settings = {
  testCommand: string
  testTimeoutSec: number
+  testFileCommand: string
+  testFileTimeoutSec: number
+  testRerunJobs: number
+  testRerunMax: number
+  testIssueRe: RegExp
  prBaseCommand: string
  defaultBaseBranch: string
  maxFails: number
@@ -361,7 +457,7 @@ async function verifyRound($: EngineInterface, agentId: string, rec: AgentRec, c
  let tests = ''
  if (cfg.testCommand) {
    $.ui.status(`router: ${rec.taskId} round ${round} running tests`)
-    const t = await runTests($, cfg.testCommand, cwd, cfg.testTimeoutSec)
+    const t = await runTests($, cfg.testCommand, cwd, cfg.testTimeoutSec, cfg.testIssueRe)
    if (t.status !== 'ran') {
      // No verdict on the code: no verifier call, no fail counted, no rewrite.
      entry.outcome = t.status === 'timeout' ? 'tests-timeout' : 'tests-error'
@@ -378,6 +474,13 @@ async function verifyRound($: EngineInterface, agentId: string, rec: AgentRec, c
      return
    }
    tests = `exit code ${t.code} after ${t.seconds} s\n${t.tail}`
+    if (cfg.testFileCommand && t.issues.length > 0) {
+      const rows = await rerunReported($, rec, t.issues, cfg)
+      const skipped = t.issues.length - rows.length
+      tests +=
+        `\n\n### Files the full run reported, rerun one by one\n${rerunTable(rows)}` +
+        (skipped > 0 ? `\n(${skipped} more reported file(s) not rerun: unsafe name or over testRerunMax.)` : '')
+    }
  }

  // 3. The verdict, from another model.
@@ -610,6 +713,11 @@ export const register: Register = (on, options) => {
  const cfg: Settings = {
    testCommand: String(options.testCommand ?? '').trim(),
    testTimeoutSec: validTimeout(options.testTimeoutSec),
+    testFileCommand: String(options.testFileCommand ?? '').trim(),
+    testFileTimeoutSec: positiveInt(options.testFileTimeoutSec, 900),
+    testRerunJobs: positiveInt(options.testRerunJobs, 6),
+    testRerunMax: positiveInt(options.testRerunMax, 40),
+    testIssueRe: issueRegex(String(options.testIssuePattern ?? '')),
    prBaseCommand: String(options.prBaseCommand ?? '').trim(),
    defaultBaseBranch: String(options.defaultBaseBranch ?? 'release/1.0').trim(),
    maxFails: Number(options.maxRewrites ?? 2),
@@ -317,3 +317,62 @@ export const turnEndKind = (status: LoopStatus | undefined): 'handback' | 'inter
 /** How often an interim agent's status is checked again, and for how long at most. */
 export const WATCH_EVERY_MS = 15_000
 export const WATCH_FOR_MS = 24 * 60 * 60 * 1000
+
+// ---------- rerunning the files a full run reported (#5) ----------
+
+export type FileKind = 'TIMEOUT' | 'FAIL'
+export type FileRun = 'pass' | 'fail' | 'timeout' | 'error' | 'not run'
+
+/** How the runner marks a file; group 1 the kind, group 2 the file. */
+export const DEFAULT_ISSUE_PATTERN = '^(TIMEOUT|FAIL)\\s+(\\S+)'
+
+/** A compiled issue pattern, or the default when the setting is empty or broken. */
+export const issueRegex = (pattern: string): RegExp => {
+  try {
+    return new RegExp(pattern.trim() || DEFAULT_ISSUE_PATTERN)
+  } catch {
+    return new RegExp(DEFAULT_ISSUE_PATTERN)
+  }
+}
+
+/** One output line the runner wrote about a file, or undefined. */
+export const parseIssueLine = (line: string, re: RegExp): { kind: FileKind; file: string } | undefined => {
+  const m = line.trim().match(re)
+  const kind = m?.[1]?.toUpperCase()
+  const file = m?.[2]
+  if (!file || (kind !== 'TIMEOUT' && kind !== 'FAIL')) return undefined
+  return { kind, file }
+}
+
+/**
+ * Whether a file name from the runner's output may go into a shell command:
+ * letters, digits, `_ . - /` only, no `..` part, not starting with `-`.
+ */
+export const isSafeFile = (file: string) =>
+  /^[A-Za-z0-9_./-]+$/.test(file) && !file.startsWith('-') && !file.split('/').includes('..')
+
+export const fileCommand = (cmd: string, file: string) => cmd.split('{file}').join(file)
+
+export type RerunRow = { file: string; full: FileKind; head: FileRun; base: FileRun }
+
+/** What a rerun row means for the change under review. */
+export const rerunVerdict = (r: RerunRow): string => {
+  if (r.head === 'pass') return 'passes alone: does not count against the change'
+  if (r.head === 'error' || r.head === 'not run') return 'could not be rerun: judge from the full run'
+  if (r.base === 'fail' || r.base === 'timeout') return `also ${r.base === 'fail' ? 'fails' : 'times out'} on base: not caused by this change`
+  if (r.base === 'pass') return `${r.head === 'fail' ? 'fails' : 'times out'} alone, passes on base: counts against the change`
+  return `${r.head === 'fail' ? 'fails' : 'times out'} alone; base not checked`
+}
+
+export const rerunTable = (rows: readonly RerunRow[]): string =>
+  [
+    '| file | full run | alone on head | on base | meaning |',
+    '|---|---|---|---|---|',
+    ...rows.map(r => `| ${r.file} | ${r.full} | ${r.head} | ${r.base} | ${rerunVerdict(r)} |`),
+  ].join('\n')
+
+/** A positive whole number from a setting, else the fallback. */
+export const positiveInt = (raw: unknown, fallback: number): number => {
+  const n = Number(raw)
+  return Number.isFinite(n) && n >= 1 ? Math.round(n) : fallback
+}
@@ -12,6 +12,11 @@ import {
  validTimeout,
  verifierFor,
  handBackLine,
+  isSafeFile,
+  issueRegex,
+  parseIssueLine,
+  rerunVerdict,
+  fileCommand,
  ranTier,
  turnEndKind,
 } from '../hooks/router'
@@ -226,8 +231,20 @@ type World = {
  /** The model the engine resolves for new spawns, whatever was asked. */
  engineModel?: string
  verifierPrompts: string[]
+  verifierSystems: string[]
  /** SendMessage calls fail (nothing delivered). */
  sendFails?: boolean
+  /** Pieces the full test run prints to stdout. */
+  fullRunOutput: string[]
+  /** How a file run alone ends, keyed `head:<file>` or `base:<file>`; absent: pass. */
+  fileOutcomes: Record<string, 'pass' | 'fail' | 'timeout'>
+  /** Files run alone, as `head:<file>` or `base:<file>`. */
+  fileRuns: string[]
+  /** git worktree commands run. */
+  worktree: string[]
+  worktreeFails?: boolean
+  running: number
+  maxParallel: number
  /** Each agent's loop status as $.agent.list() reports it; absent means completed. */
  status: Record<string, string>
  diffs: string[]
@@ -259,6 +276,7 @@ function world(on: On, w: World) {
    if (e.system?.startsWith('You route')) return { value: { isAnswered: true as const, text: 'sonnet', usage: USAGE } }
    w.verifier.push(e.model)
    w.verifierPrompts.push(e.prompt)
+    w.verifierSystems.push(e.system ?? '')
    return { value: { isAnswered: true as const, text: w.verdicts.shift() ?? '{"verdict":"PASS"}', usage: USAGE } }
  })
  on('process.run', async (_$, e) => {
@@ -271,15 +289,35 @@ function world(on: On, w: World) {
      w.diffBases.push(e.argv[2] ?? '')
      return ok(w.diffs.shift() ?? '')
    }
+    if (a.includes(' worktree ')) {
+      w.worktree.push(a)
+      if (a.includes(' worktree add ') && w.worktreeFails) return ok('', 128)
+    }
    return ok('')
  })
  on('process.spawn', async function* (_$, e) {
+    const cmd = e.argv[e.argv.length - 1] ?? ''
+    if (cmd.startsWith('runone ')) {
+      // one file alone: `runone <file>`, in the clone or in the base worktree
+      const file = cmd.slice('runone '.length)
+      const where = String(e.cwd ?? '').endsWith('-verifier-base') ? 'base' : 'head'
+      w.fileRuns.push(`${where}:${file}`)
+      w.running++
+      w.maxParallel = Math.max(w.maxParallel, w.running)
+      for (let i = 0; i < 5; i++) await Promise.resolve() // let other reruns start
+      w.running--
+      const outcome = w.fileOutcomes[`${where}:${file}`] ?? 'pass'
+      if (outcome === 'timeout') yield { stream: 'stderr' as const, text: "timeout: sending signal TERM to command 'sh'\n" }
+      return { value: { code: outcome === 'pass' ? 0 : outcome === 'timeout' ? 124 : 1, signal: null } }
+    }
    w.testCwd.push(String(e.cwd ?? ''))
    w.testArgv.push([...e.argv])
    const end = w.testEnds.shift() ?? 0
    const fired = w.testFired.shift() ?? false
    if (end === 'throw') throw new Error('docker: not found')
    yield { stream: 'stdout' as const, text: end === 0 ? '67 passed\n' : 'test_modes_p4 ... still running\n' }
+    // the full run's report comes last, so a last line without a newline stays last
+    for (const piece of w.fullRunOutput) yield { stream: 'stdout' as const, text: piece }
    if (fired) yield { stream: 'stderr' as const, text: "timeout: sending signal TERM to command 'sh'\n" }
    if (w.testSplitLine) {
      yield { stream: 'stderr' as const, text: 'timeout: sending sig' }
@@ -319,7 +357,7 @@ function world(on: On, w: World) {
  return clock
 }

-const fresh = (): World => ({ spawned: [], verifier: [], verdicts: [], told: [], handBacks: [], all: [], resumed: [], denySpawns: 0, verifierPrompts: [], status: {}, diffs: [], diffBases: [], testCwd: [], testArgv: [], testEnds: [], testFired: [], snap: 0 })
+const fresh = (): World => ({ spawned: [], verifier: [], verdicts: [], told: [], handBacks: [], all: [], resumed: [], denySpawns: 0, verifierPrompts: [], verifierSystems: [], fullRunOutput: [], fileOutcomes: {}, fileRuns: [], worktree: [], running: 0, maxParallel: 0, status: {}, diffs: [], diffBases: [], testCwd: [], testArgv: [], testEnds: [], testFired: [], snap: 0 })

 async function finish($: Engine, clock: { advance: (ms: number) => Promise<void> }, agentId: string, ranModel?: string, reason = 'answer') {
  const usage = ranModel
@@ -944,3 +982,160 @@ test('#6: a SendMessage that was not delivered is not a later instruction', asyn
  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
  expect(w.verifierPrompts[0] ?? '').not.toContain('Never arrived.')
 })
+
+// ---------- reruns of reported files (#5) ----------
+
+const RERUN = { testCommand: 'make t', testFileCommand: 'runone {file}' }
+const T307_OUTPUT = [
+  'PASS test_a.py 12s\nTIMEOUT test_tariff24_weather.py 230s\nFAIL test_loop.py\nTIME',
+  'OUT test_slow.py 1500s\n', // a line split across pieces
+  '--- test_tariff24_weather.py (TIMEOUT)\n<tail>\n77 pass, 1 fail, 2 timeout (limit 1500 s)\n',
+]
+
+test('#5: reported lines are read, also when split across pieces; one entry per file', async () => {
+  const re = issueRegex('')
+  expect(parseIssueLine('TIMEOUT test_tariff24_weather.py 230s', re)).toEqual({ kind: 'TIMEOUT', file: 'test_tariff24_weather.py' })
+  expect(parseIssueLine('FAIL test_loop.py', re)).toEqual({ kind: 'FAIL', file: 'test_loop.py' })
+  expect(parseIssueLine('--- test_tariff24_weather.py (TIMEOUT)', re)).toBe(undefined)
+  expect(parseIssueLine('77 pass, 0 fail, 30 timeout (limit 1500 s)', re)).toBe(undefined)
+  expect(parseIssueLine('PASS test_a.py', re)).toBe(undefined)
+  // a custom pattern, and a broken one falls back to the default
+  expect(parseIssueLine('[timeout] x.py', issueRegex('^\\[(timeout|fail)\\]\\s+(\\S+)'))?.file).toBe('x.py')
+  expect(parseIssueLine('TIMEOUT y.py', issueRegex('(unclosed'))?.file).toBe('y.py')
+})
+
+test('#5: only safe file names reach the shell', async () => {
+  for (const ok of ['test_a.py', 'tests/test_b.py', 'a-b.c_d.py']) expect(isSafeFile(ok)).toBe(true)
+  for (const bad of ['a;rm -rf ~', '$(id).py', '../x.py', 'a/../b.py', '-rf', 'a b.py', 'x`y`.py', "a'.py"]) expect(isSafeFile(bad)).toBe(false)
+  expect(fileCommand('docker run img python3 {file} && echo {file}', 'test_a.py')).toBe('docker run img python3 test_a.py && echo test_a.py')
+})
+
+test('#5: what each rerun outcome means', async () => {
+  expect(rerunVerdict({ file: 'f', full: 'TIMEOUT', head: 'pass', base: 'not run' })).toContain('does not count')
+  expect(rerunVerdict({ file: 'f', full: 'FAIL', head: 'fail', base: 'fail' })).toContain('not caused by this change')
+  expect(rerunVerdict({ file: 'f', full: 'TIMEOUT', head: 'timeout', base: 'timeout' })).toContain('also times out on base')
+  expect(rerunVerdict({ file: 'f', full: 'FAIL', head: 'fail', base: 'pass' })).toContain('counts against the change')
+  expect(rerunVerdict({ file: 'f', full: 'FAIL', head: 'fail', base: 'not run' })).toContain('base not checked')
+  expect(rerunVerdict({ file: 'f', full: 'FAIL', head: 'error', base: 'not run' })).toContain('could not be rerun')
+})
+
+test('#5 acceptance: timed-out and failed files are rerun alone, failures again on base, and the verifier gets the table', { options: RERUN }, async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  w.testEnds = [1]
+  w.fullRunOutput = T307_OUTPUT
+  w.fileOutcomes = { 'head:test_loop.py': 'fail', 'base:test_loop.py': 'fail', 'head:test_slow.py': 'timeout', 'base:test_slow.py': 'pass' }
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'MAINT-28' } as never)
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  expect([...w.fileRuns].sort()).toEqual([
+    'base:test_loop.py',
+    'base:test_slow.py',
+    'head:test_loop.py',
+    'head:test_slow.py',
+    'head:test_tariff24_weather.py',
+  ])
+  const p = w.verifierPrompts[0] ?? ''
+  expect(p).toContain('### Files the full run reported, rerun one by one')
+  expect(p).toContain('| test_tariff24_weather.py | TIMEOUT | pass | not run | passes alone: does not count against the change |')
+  expect(p).toContain('| test_loop.py | FAIL | fail | fail | also fails on base: not caused by this change |')
+  expect(p).toContain('| test_slow.py | TIMEOUT | timeout | pass | times out alone, passes on base: counts against the change |')
+  // the base worktree sits next to the clone, and is removed and pruned afterwards
+  expect(w.worktree).toContain(`git -C ${DEV} worktree add --detach ${DEV}-verifier-base mergebase`)
+  expect(w.worktree.at(-2)).toBe(`git -C ${DEV} worktree remove --force ${DEV}-verifier-base`)
+  expect(w.worktree.at(-1)).toBe(`git -C ${DEV} worktree prune`)
+})
+
+test('#5: when every reported file passes alone, no base worktree is made', { options: RERUN }, async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  w.testEnds = [1]
+  w.fullRunOutput = ['TIMEOUT test_tariff24_weather.py 230s\n']
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  expect(w.fileRuns).toEqual(['head:test_tariff24_weather.py'])
+  expect(w.worktree.filter(c => c.includes(' add '))).toEqual([])
+})
+
+test('#5: a base worktree that cannot be made leaves base as not run', { options: RERUN }, async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  w.testEnds = [1]
+  w.worktreeFails = true
+  w.fullRunOutput = ['FAIL test_loop.py\n']
+  w.fileOutcomes = { 'head:test_loop.py': 'fail' }
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  expect(w.fileRuns).toEqual(['head:test_loop.py'])
+  expect(w.verifierPrompts[0]).toContain('| test_loop.py | FAIL | fail | not run | fails alone; base not checked |')
+})
+
+test('#5: unsafe names and files past testRerunMax are not rerun, and said so', { options: { ...RERUN, testRerunMax: 1 } }, async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  w.testEnds = [1]
+  w.fullRunOutput = ['FAIL a;rm.py\nTIMEOUT test_a.py 1s\nTIMEOUT test_b.py 1s\n']
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  expect(w.fileRuns).toEqual(['head:test_a.py'])
+  expect(w.verifierPrompts[0]).toContain('(2 more reported file(s) not rerun')
+})
+
+test('#5: without testFileCommand nothing is rerun', { options: { testCommand: 'make t' } }, async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  w.testEnds = [1]
+  w.fullRunOutput = T307_OUTPUT
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  expect(w.fileRuns).toEqual([])
+  expect(w.verifierPrompts[0] ?? '').not.toContain('rerun one by one')
+})
+
+test('#5: reruns run at most testRerunJobs at a time', { options: { ...RERUN, testRerunJobs: 2 } }, async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  w.testEnds = [1]
+  w.fullRunOutput = ['TIMEOUT t1.py 1s\nTIMEOUT t2.py 1s\nTIMEOUT t3.py 1s\nTIMEOUT t4.py 1s\nTIMEOUT t5.py 1s\n']
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  expect(w.fileRuns.length).toBe(5)
+  expect(w.maxParallel).toBe(2)
+})
+
+test('#5: a custom pattern that also matches other lines still keeps only TIMEOUT and FAIL', async () => {
+  const re = issueRegex('^(\\w+)\\s+(\\S+)')
+  expect(parseIssueLine('PASS test_a.py', re)).toBe(undefined)
+  expect(parseIssueLine('timeout test_b.py', re)).toEqual({ kind: 'TIMEOUT', file: 'test_b.py' })
+})
+
+test('#5: a last line without a newline is read; a file reported twice is rerun once', { options: RERUN }, async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  w.testEnds = [1]
+  w.fullRunOutput = ['FAIL test_twice.py\nFAIL test_twice.py\n', 'TIMEOUT test_last.py 1500s']
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  expect([...w.fileRuns].sort()).toEqual(['head:test_last.py', 'head:test_twice.py'])
+})
+
+test('#5: the verifier is told how to read the rerun table', { options: RERUN }, async ($, on) => {
+  const w = fresh()
+  w.diffs = ['diff']
+  w.testEnds = [1]
+  w.fullRunOutput = ['TIMEOUT test_a.py 1s\n']
+  const clock = world(on, w)
+  const r = await $.agent.spawn({ prompt: devBrief(), description: 'dev' } as never)
+  await finish($, clock, r.agentId as string, 'claude-sonnet-5-5')
+  const sys = w.verifierSystems[0] ?? ''
+  expect(sys).toContain('When a table of files rerun one by one is given')
+  expect(sys).toContain('a file that passes alone does not count against the change')
+  expect(sys).toContain('not caused by this change')
+  expect(sys).toContain('Name every file you count against the change')
+})