Compare commits
2
Commits
pipeline-0.5.0
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c88884c54a | ||
|
|
3cd69762be |
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "model-router",
|
||||
"version": "0.5.0",
|
||||
"version": "0.5.1",
|
||||
"description": "Routes each subagent to Haiku, Sonnet or Opus by how well-defined its brief is, verifies dev work below Opus in its own clone on a different model, sends failures back to the same agent then a stronger one, and estimates the savings against Opus.",
|
||||
"author": {
|
||||
"name": "Glenn Schrooyen"
|
||||
|
||||
@@ -6,24 +6,25 @@ A Claude Code plugin that picks a model for every subagent, checks dev work on a
|
||||
|
||||
| Step | Behaviour |
|
||||
|---|---|
|
||||
| Route | Dev agents run on **Sonnet** when the brief decides the approach (Expected / acceptance criteria), on **Opus** when the design is still open. Review and test agents keep the model they are given (Opus when none). |
|
||||
| Route | Dev agents run on **Sonnet** when the brief decides the approach (Expected / acceptance criteria), on **Opus** when the design is still open. Review and test agents keep the model they are given (Opus when none). Other non-dev agents (a `test-design` role, or the `test-designer` / `release-reviewer` agent types) do too, but never Haiku. None of them get a hand-back line or verification. |
|
||||
| Rerun reported files | With `testFileCommand` set, every file the full run reports as `TIMEOUT` or `FAIL` is rerun alone in the clone. A file that still fails or times out alone runs again on the base branch, in a worktree next to the clone (`<clone>-verifier-base`, removed afterwards). The verifier gets a table: a file that passes alone does not count, one that also fails on base is not caused by the change. |
|
||||
| Later instructions | Every message sent to a dev agent after its brief (a SendMessage, or the message that resumes it) reaches the verifier as part of the task, so scope added later is judged as in scope. A resumed agent keeps its task number, routing and failure count. |
|
||||
| Verify | When a dev agent below Opus finishes a round, the router diffs its clone against the merge base, runs the test command in that clone, and has **Opus** judge the change. |
|
||||
| Hand-back line | Every dev hand-back gets one line at once: `[model-router] Tn verifying on Opus...` (a verdict follows), or `[model-router] Tn no verification: dev ran on Opus. Go straight to review.` Only the real hand-back counts: an interim stop while the dev waits on its own background work (e.g. a long test suite) posts nothing and starts no verifier. Decided by the model the run actually used, not only the routing choice; a difference is shown as `(routed to X, ran on Y)`. |
|
||||
| Hand-back line | Only dev agents get `T` numbers and lines; other agents are logged as `A1`, `A2`, ... Every dev hand-back gets one line at once: `[model-router] Tn verifying on Opus...` (a verdict follows), or `[model-router] Tn no verification: dev ran on Opus. Go straight to review.` Only the real hand-back counts: an interim stop while the dev waits on its own background work (e.g. a long test suite) posts nothing and starts no verifier. Decided by the model the run actually used, not only the routing choice; a difference is shown as `(routed to X, ran on Y)`. |
|
||||
| On FAIL | 1st: resume the same agent with the issues. 2nd: a new agent one model up, in the same clone. After that: handed back to the manager. |
|
||||
| Not verified | A test timeout, a test command that cannot run, a missing clone or merge base, or a verdict that cannot be read even after asking once more: reported as NOT VERIFIED, never as a FAIL. |
|
||||
| Savings | `/router savings` prices every run on the model it ran on and compares it with the same tokens on Opus 5.5. |
|
||||
|
||||
## Briefs
|
||||
|
||||
Every dev, review and test brief carries one line:
|
||||
Every pipeline brief carries these lines:
|
||||
|
||||
```
|
||||
Pipeline-Role: dev
|
||||
Clone: /home/homelab/projects/goodwe/clones/<TICKET>-dev/
|
||||
```
|
||||
|
||||
Add `(Role: dev|review|test)` when the path does not end in the role. Tickets with extra parts (`OPT-11-P4`) are fine. Name the base branch (`Base branch: release/1.0`).
|
||||
The role comes from `Pipeline-Role:` first, then `(Role: ...)` on the `Clone:` line, then the folder name (which must end in `-dev`, `-review` or `-test`). Any other role (`test-design`, ...) is never a dev, whatever other clones the brief mentions; neither is a `test-designer` or `release-reviewer` agent. With a `## Test plan` section, the verifier checks that every T-id and N-id is covered. Tickets with extra parts (`OPT-11-P4`) are fine. Name the base branch (`Base branch: release/1.0`).
|
||||
|
||||
## Install
|
||||
|
||||
@@ -66,7 +67,7 @@ claude plugin validate .
|
||||
claude plugin test .
|
||||
```
|
||||
|
||||
92 tests. Every routing, verification, cost and timeout rule was also checked by breaking it on purpose (66 mutations in v0.4.4, plus 13 for the verdict reader in v0.4.5 12 for the hand-back line in v0.4.6 10 for interim stops in v0.4.7 14 for resumes, later instructions and rewrites in v0.4.8 and 23 for file reruns in v0.4.9; all caught by a test).
|
||||
102 tests. Every routing, verification, cost and timeout rule was also checked by breaking it on purpose (66 mutations in v0.4.4, plus 13 for the verdict reader in v0.4.5 12 for the hand-back line in v0.4.6 10 for interim stops in v0.4.7 14 for resumes, later instructions and rewrites in v0.4.8 23 for file reruns in v0.4.9 and 16 for non-dev roles and gate agent types in v0.5.1; all caught by a test).
|
||||
|
||||
## Known limits
|
||||
|
||||
|
||||
+2
-1
@@ -150,7 +150,8 @@ export const usd = (n: number) => (Math.abs(n) < 0.01 && n !== 0 ? `$${n.toFixed
|
||||
export const savingsReport = (costs: Readonly<Record<string, TaskCost>>, limit: number, only?: string): string => {
|
||||
const ids = Object.keys(costs)
|
||||
.filter(id => !only || id === only)
|
||||
.sort((a, b) => Number(b.slice(1)) - Number(a.slice(1)))
|
||||
// dev tasks (T) first, then other agents (A); newest first within each
|
||||
.sort((a, b) => b.charAt(0).localeCompare(a.charAt(0)) || Number(b.slice(1)) - Number(a.slice(1)))
|
||||
if (ids.length === 0) return only ? `model-router: no costs recorded for ${only}.` : 'savings: no agent runs recorded yet.'
|
||||
|
||||
const row = (id: string, t: TaskCost) => {
|
||||
|
||||
+27
-13
@@ -16,7 +16,7 @@ import {
|
||||
clip,
|
||||
decide,
|
||||
onFail,
|
||||
parseBrief,
|
||||
briefFor,
|
||||
parsePatterns,
|
||||
parseVerdict,
|
||||
handBackLine,
|
||||
@@ -55,6 +55,8 @@ type AgentRec = {
|
||||
/** Later instructions to this agent (resumes, SendMessage), oldest first: part of its task. */
|
||||
followUps?: string[]
|
||||
role?: Role
|
||||
/** The role as the brief wrote it, for non-dev roles (test-design). */
|
||||
roleName?: string
|
||||
clone?: string
|
||||
baseBranch?: string
|
||||
critical: boolean
|
||||
@@ -83,6 +85,7 @@ type Outcome =
|
||||
| 'verdict-unreadable'
|
||||
| 'test-agent'
|
||||
| 'review-agent'
|
||||
| 'non-dev-agent'
|
||||
| 'no-clone'
|
||||
| 'opus-unverified'
|
||||
| 'agent-ended'
|
||||
@@ -94,6 +97,8 @@ type LogEntry = {
|
||||
task: string
|
||||
tier: Tier
|
||||
role?: Role
|
||||
/** The role as the brief wrote it, for non-dev roles (test-design). */
|
||||
roleName?: string
|
||||
reason: string
|
||||
round?: number
|
||||
verifier?: Tier
|
||||
@@ -103,6 +108,8 @@ type LogEntry = {
|
||||
|
||||
const S = {
|
||||
counter: 'counter',
|
||||
// other agents (review, test, test-design, no role) count apart, as A1, A2, ...
|
||||
counterOther: 'counter-other',
|
||||
pending: 'pending',
|
||||
agents: 'agents',
|
||||
log: 'log',
|
||||
@@ -145,9 +152,9 @@ Reply with JSON only:
|
||||
|
||||
const ORCHESTRATOR_NOTE = `## Model router (model-router mod)
|
||||
The model-router mod picks each subagent's model and verifies dev work before it reaches a PR.
|
||||
- Every dev, review and test brief carries one line \`Clone: <path>\` (add \`(Role: dev|review|test)\` if the path does not end in the role). The router reads the role from that line only; a brief naming several clones without it runs on Opus, unverified.
|
||||
- Every pipeline brief carries \`Pipeline-Role: <role>\` and one line \`Clone: <path>\` (add \`(Role: <role>)\` if the folder name does not end in -dev, -review or -test). The role comes from Pipeline-Role, then (Role: ...), then the folder name. Only dev briefs are routed and verified; any other role (test-design, ...) keeps the model you name, gets no "[model-router]" line and an A number, not a T number. A brief naming several clones and no Clone: line runs on Opus, unverified.
|
||||
- Dev agents are routed: Sonnet when the brief decides the approach (Expected / acceptance criteria), Opus when the design is still open. Naming a model on a dev brief has no effect. Give each dev brief its clone path and base branch.
|
||||
- Review and test agents keep the model you name (Opus when you name none).
|
||||
- Review and test agents keep the model you name (Opus when you name none); other non-dev agents (test-design, test-designer, release-reviewer) too, but never Haiku.
|
||||
- Every dev hand-back gets one "[model-router] Tn" line at once; an interim stop while the dev waits on its own background work gets none. "verifying on Opus...": wait for the verdict before advancing the PR. "no verification: dev ran on Opus. Go straight to review.": no verdict will come. Lines are decided by the model the dev actually ran on.
|
||||
- On "[model-router] ... FAIL ... resume agent", resume that same agent with SendMessage, passing the issues. On "start a new agent ... [router-rewrite Tn]", stop the old agent and call Agent with exactly that prompt. On "handed back", stop and show the issues to the person.
|
||||
- Scope or decisions added after the brief: send them to the dev agent with SendMessage. The verifier sees every later message to that agent and treats it as in scope. A resumed agent keeps its task number. Do not fix rejected work yourself.
|
||||
@@ -186,11 +193,13 @@ async function putIn<T>($: EngineInterface, key: string, id: string, value: T |
|
||||
await updateIn<T>($, key, id, () => value)
|
||||
}
|
||||
|
||||
async function nextId($: EngineInterface) {
|
||||
/** T numbers are for dev agents only; every other agent gets an A number. */
|
||||
async function nextId($: EngineInterface, isDev: boolean) {
|
||||
const key = isDev ? S.counter : S.counterOther
|
||||
return serial(async () => {
|
||||
const n = Number((await $.store.get(S.counter)) ?? 0) + 1
|
||||
await $.store.set(S.counter, n)
|
||||
return `T${n}`
|
||||
const n = Number((await $.store.get(key)) ?? 0) + 1
|
||||
await $.store.set(key, n)
|
||||
return `${isDev ? 'T' : 'A'}${n}`
|
||||
})
|
||||
}
|
||||
|
||||
@@ -622,6 +631,7 @@ async function skipRound($: EngineInterface, agentId: string, rec: AgentRec, out
|
||||
task: rec.prompt.split('\n')[0]?.slice(0, 60) ?? '',
|
||||
tier: rec.tier,
|
||||
role: rec.role,
|
||||
roleName: rec.roleName,
|
||||
reason: outcome,
|
||||
outcome,
|
||||
})
|
||||
@@ -783,11 +793,12 @@ export const register: Register = (on, options) => {
|
||||
'Start the agent with the full brief instead, or check /router.',
|
||||
}
|
||||
} else {
|
||||
taskId = await nextId($)
|
||||
const brief = parseBrief(e.prompt)
|
||||
const brief = briefFor(e.prompt, e.subagentType)
|
||||
taskId = await nextId($, brief.role === 'dev')
|
||||
critical = safetyHit(`${e.description}\n${e.prompt}`, patterns) !== undefined
|
||||
const d = decide({
|
||||
role: brief.role,
|
||||
roleName: brief.roleName,
|
||||
named: asTier(e.model),
|
||||
subagentType: e.subagentType,
|
||||
prompt: e.prompt,
|
||||
@@ -798,6 +809,7 @@ export const register: Register = (on, options) => {
|
||||
isRouted =
|
||||
brief.role !== 'review' &&
|
||||
brief.role !== 'test' &&
|
||||
brief.role !== 'other' &&
|
||||
!(d.kind === 'fixed' && d.reason === 'model chosen by orchestrator')
|
||||
if (d.kind === 'fixed') {
|
||||
tier = d.tier
|
||||
@@ -810,7 +822,7 @@ export const register: Register = (on, options) => {
|
||||
}
|
||||
}
|
||||
|
||||
const brief = parseBrief(prompt)
|
||||
const brief = briefFor(prompt, e.subagentType)
|
||||
const ran = await next({ ...e, prompt, model: tier })
|
||||
if (ran.deny !== undefined || !ran.agentId) return ran
|
||||
// the stored rewrite is used up only once its agent has started
|
||||
@@ -835,6 +847,7 @@ export const register: Register = (on, options) => {
|
||||
routed: resolved,
|
||||
name: e.name,
|
||||
role: brief.role,
|
||||
roleName: brief.roleName,
|
||||
clone: brief.clone,
|
||||
baseBranch: brief.baseBranch,
|
||||
critical,
|
||||
@@ -850,11 +863,12 @@ export const register: Register = (on, options) => {
|
||||
task: e.description,
|
||||
tier: resolved,
|
||||
role: brief.role,
|
||||
roleName: brief.roleName,
|
||||
reason,
|
||||
outcome: 'spawned',
|
||||
opusRequested,
|
||||
})
|
||||
$.ui.status(`router: ${taskId} ${brief.role ?? 'agent'} → ${tier}`)
|
||||
$.ui.status(`router: ${taskId} ${brief.roleName ?? brief.role ?? 'agent'} → ${tier}`)
|
||||
return ran
|
||||
}).catch(($, e, next) => {
|
||||
// Fail open: a router bug must never block delegation.
|
||||
@@ -924,14 +938,14 @@ export const register: Register = (on, options) => {
|
||||
.slice(-15)
|
||||
.map(
|
||||
l =>
|
||||
`${l.id.padEnd(5)} ${(l.role ?? '-').padEnd(6)} ${l.tier.padEnd(6)} ${(l.verifier ? '→' + l.verifier : '').padEnd(8)} ${l.outcome.padEnd(17)} ${(l.task ?? '').slice(0, 40)}${l.opusRequested ? ' [opus requested]' : ''}`,
|
||||
`${l.id.padEnd(5)} ${(l.roleName ?? l.role ?? '-').padEnd(6)} ${l.tier.padEnd(6)} ${(l.verifier ? '→' + l.verifier : '').padEnd(8)} ${l.outcome.padEnd(17)} ${(l.task ?? '').slice(0, 40)}${l.opusRequested ? ' [opus requested]' : ''}`,
|
||||
)
|
||||
.join('\n')
|
||||
const pending = Object.keys(await getMap<Pending>($, S.pending))
|
||||
return {
|
||||
text:
|
||||
`model-router - last ${Math.min(15, list.length)} of ${list.length}\n` +
|
||||
`writers: ${count(l => (l.outcome === 'spawned' ? `${l.role ?? 'other'}:${l.tier}` : undefined))}\n` +
|
||||
`writers: ${count(l => (l.outcome === 'spawned' ? `${l.roleName ?? l.role ?? 'none'}:${l.tier}` : undefined))}\n` +
|
||||
`results: ${count(l => (l.outcome === 'spawned' ? undefined : l.outcome))}\n` +
|
||||
`opus requested on dev briefs: ${list.filter(l => l.opusRequested).length}\n` +
|
||||
(pending.length ? `pending rewrites: ${pending.join(', ')}\n` : '') +
|
||||
|
||||
+54
-18
@@ -1,7 +1,8 @@
|
||||
// Pure routing logic: no `$`, so every rule here is unit-tested.
|
||||
|
||||
export type Tier = 'haiku' | 'sonnet' | 'opus' | 'fable'
|
||||
export type Role = 'dev' | 'review' | 'test'
|
||||
/** `other`: any pipeline role the router does not know (test-design, ...): never a dev. */
|
||||
export type Role = 'dev' | 'review' | 'test' | 'other'
|
||||
|
||||
export const TIERS: readonly Tier[] = ['haiku', 'sonnet', 'opus', 'fable']
|
||||
|
||||
@@ -31,6 +32,8 @@ export type Brief = {
|
||||
clone?: string
|
||||
ticket?: string
|
||||
role?: Role
|
||||
/** The role as the brief wrote it (`test-design`), when it is not dev, review or test. */
|
||||
roleName?: string
|
||||
baseBranch?: string
|
||||
/** Several clone paths and no `Clone:` line: the role cannot be told. */
|
||||
isAmbiguous?: boolean
|
||||
@@ -38,40 +41,56 @@ export type Brief = {
|
||||
|
||||
// A ticket is LETTERS-digits plus any extra parts (OPT-11-P4); the role is the last part.
|
||||
const TICKET = '[A-Za-z]+-\\d+(?:-[A-Za-z0-9]+)*?'
|
||||
const CLONE_PATH = new RegExp(`(\\S*?\\/clones\\/(${TICKET})-(dev|review|test))\\/?`)
|
||||
// The role must end the folder name: EV-05-design and EV-05-test-design are no -test clone.
|
||||
const CLONE_PATH = new RegExp(`(\\S*?\\/clones\\/(${TICKET})-(dev|review|test))(?![A-Za-z0-9_-])\\/?`)
|
||||
const CLONE_PATHS = new RegExp(CLONE_PATH.source, 'g')
|
||||
// The explicit line every brief carries: `Clone: <path>` with an optional `(Role: dev)`.
|
||||
const CLONE_LINE = /^[ \t>*-]*Clone:[ \t]*`?([^\s`]+?)\/?`?[ \t]*(?:\(Role:[ \t]*(dev|review|test)\))?[ \t]*$/im
|
||||
const CLONE_LINE = /^[ \t>*-]*Clone:[ \t]*`?([^\s`]+?)\/?`?[ \t]*(?:\(Role:[ \t]*([A-Za-z][\w-]*)\))?[ \t]*$/im
|
||||
// The pipeline's own header, which decides over everything else.
|
||||
const PIPELINE_ROLE = /^[ \t>*-]*Pipeline-Role:[ \t]*`?([A-Za-z][\w-]*)`?[ \t]*$/im
|
||||
const BASE =
|
||||
/\b(?:base(?:\s+branch)?|target(?:\s+branch)?|merge\s+into|PR\s+(?:against|into))\s*[:=]?\s*`?((?:release|main|master|develop)[\w./-]*)/i
|
||||
|
||||
/** A role as written in a brief: dev, review and test as they are, anything else `other`. */
|
||||
export const asRole = (written: string | undefined): { role?: Role; roleName?: string } => {
|
||||
if (!written) return {}
|
||||
const w = written.toLowerCase()
|
||||
if (w === 'dev' || w === 'review' || w === 'test') return { role: w }
|
||||
return { role: 'other', roleName: w }
|
||||
}
|
||||
|
||||
const trimPath = (p: string) => p.replace(/^[`'"(]+/, '').replace(/[`'")]+$/, '').replace(/\/+$/, '')
|
||||
|
||||
/**
|
||||
* Reads the clone path, ticket, role and base branch out of a brief.
|
||||
*
|
||||
* The `Clone:` line decides when present: its `(Role: ...)`, else the role in
|
||||
* its path. Without one, a brief that names one clone is read from that path;
|
||||
* one that names several is ambiguous and gets no role.
|
||||
* The role: `Pipeline-Role:` first, then `(Role: ...)` on the `Clone:` line,
|
||||
* then the role in the clone's folder name. A role other than dev, review or
|
||||
* test is `other`, never a dev. Without a `Clone:` line, a brief that names
|
||||
* one clone is read from that path; several are ambiguous. A brief whose
|
||||
* header names a non-dev role is never read from other paths it mentions.
|
||||
*/
|
||||
export const parseBrief = (prompt: string): Brief => {
|
||||
const b = prompt.match(BASE)
|
||||
// a sentence's full stop or a trailing slash is no part of the branch
|
||||
const baseBranch = b?.[1]?.replace(/[./-]+$/, '')
|
||||
const header = asRole(prompt.match(PIPELINE_ROLE)?.[1])
|
||||
|
||||
const line = prompt.match(CLONE_LINE)
|
||||
if (line?.[1]) {
|
||||
const clone = trimPath(line[1])
|
||||
const fromPath = clone.match(CLONE_PATH)
|
||||
const role = (line[2] ?? fromPath?.[3]) as Role | undefined
|
||||
return { clone, ticket: fromPath?.[2], role, baseBranch }
|
||||
const stated = header.role ? header : line[2] ? asRole(line[2]) : asRole(fromPath?.[3])
|
||||
// A pipeline agent with a clone the router cannot place is no dev: keep its model.
|
||||
return { clone, ticket: fromPath?.[2], role: 'other', ...stated, baseBranch }
|
||||
}
|
||||
if (header.role && header.role !== 'dev') return { ...header, baseBranch }
|
||||
|
||||
const paths = [...new Set([...prompt.matchAll(CLONE_PATHS)].map(m => trimPath(m[1] ?? '')))]
|
||||
if (paths.length > 1) return { baseBranch, isAmbiguous: true }
|
||||
const only = paths[0]?.match(CLONE_PATH)
|
||||
if (!only) return { baseBranch }
|
||||
return { clone: trimPath(only[1] ?? ''), ticket: only[2], role: only[3] as Role, baseBranch }
|
||||
if (!only) return { ...header, baseBranch }
|
||||
return { clone: trimPath(only[1] ?? ''), ticket: only[2], ...(header.role ? header : asRole(only[3])), baseBranch }
|
||||
}
|
||||
|
||||
// ---------- routing ----------
|
||||
@@ -111,6 +130,18 @@ export const heuristic = (subagentType: string | undefined, prompt: string, role
|
||||
/** Agent types that gate rather than write: never routed down, never verified. */
|
||||
export const KEEP_MODEL_TYPES = /^(test-designer|release-reviewer)$/i
|
||||
|
||||
/**
|
||||
* The brief as the router treats it: a gate agent type (test-designer,
|
||||
* release-reviewer) is never a dev, nor an unplaced agent, whatever paths
|
||||
* its brief names. Review and test roles stay as they are.
|
||||
*/
|
||||
export const briefFor = (prompt: string, subagentType?: string): Brief => {
|
||||
const b = parseBrief(prompt)
|
||||
if (!subagentType || !KEEP_MODEL_TYPES.test(subagentType)) return b
|
||||
if (b.role === 'review' || b.role === 'test') return b
|
||||
return { ...b, role: 'other', roleName: b.role === 'other' ? b.roleName : subagentType.toLowerCase(), isAmbiguous: undefined }
|
||||
}
|
||||
|
||||
export type Decision =
|
||||
| { kind: 'fixed'; tier: Tier; reason: string; opusRequested?: boolean }
|
||||
| { kind: 'classify'; opusRequested?: boolean }
|
||||
@@ -129,21 +160,25 @@ export const decide = (args: {
|
||||
prompt: string
|
||||
respectExplicit: boolean
|
||||
isAmbiguous?: boolean
|
||||
/** The role as written, for an `other` role (test-design). */
|
||||
roleName?: string
|
||||
}): Decision => {
|
||||
const { role, named, subagentType, prompt, respectExplicit, isAmbiguous } = args
|
||||
const who = args.roleName ?? role
|
||||
// Could be a test or review agent: never risk a weaker model on it.
|
||||
if (isAmbiguous)
|
||||
return { kind: 'fixed', tier: named && named !== 'haiku' && named !== 'sonnet' ? named : 'opus', reason: 'several clone paths and no Clone: line' }
|
||||
if (role === 'review' || role === 'test')
|
||||
return named
|
||||
? { kind: 'fixed', tier: named, reason: `named model kept for ${role} agent` }
|
||||
: { kind: 'fixed', tier: 'opus', reason: `${role} agent default` }
|
||||
// Pipeline gates without a clone role (test design, release review) judge, they do not write code:
|
||||
// like review and test agents they keep a named model, Opus when none.
|
||||
if (!role && subagentType && KEEP_MODEL_TYPES.test(subagentType))
|
||||
? { kind: 'fixed', tier: named, reason: `named model kept for ${who} agent` }
|
||||
: { kind: 'fixed', tier: 'opus', reason: `${who} agent default` }
|
||||
// Pipeline gates (test design, release review, any other non-dev role) judge, they do not
|
||||
// write code: they keep a named model, never Haiku, and run on Opus when none is named.
|
||||
const gate = role === 'other' ? (args.roleName ?? subagentType ?? 'other') : !role && subagentType && KEEP_MODEL_TYPES.test(subagentType) ? subagentType : undefined
|
||||
if (gate)
|
||||
return named && named !== 'haiku'
|
||||
? { kind: 'fixed', tier: named, reason: `named model kept for ${subagentType}` }
|
||||
: { kind: 'fixed', tier: 'opus', reason: `${subagentType} default` }
|
||||
? { kind: 'fixed', tier: named, reason: `named model kept for ${gate}` }
|
||||
: { kind: 'fixed', tier: 'opus', reason: `${gate} default` }
|
||||
if (!role && respectExplicit && named) return { kind: 'fixed', tier: named, reason: 'model chosen by orchestrator' }
|
||||
const opusRequested = role === 'dev' && (named === 'opus' || named === 'fable')
|
||||
const h = heuristic(subagentType, prompt, role)
|
||||
@@ -152,13 +187,14 @@ export const decide = (args: {
|
||||
|
||||
// ---------- verification ----------
|
||||
|
||||
export type Skip = 'test-agent' | 'review-agent' | 'no-clone' | 'opus-unverified' | 'agent-ended'
|
||||
export type Skip = 'test-agent' | 'review-agent' | 'non-dev-agent' | 'no-clone' | 'opus-unverified' | 'agent-ended'
|
||||
|
||||
/** Why an agent run is not verified, or undefined when it is. */
|
||||
export const skipReason = (rec: { role?: Role; clone?: string; tier: Tier }, ended: string): Skip | undefined => {
|
||||
if (ended !== 'answer') return 'agent-ended'
|
||||
if (rec.role === 'test') return 'test-agent'
|
||||
if (rec.role === 'review') return 'review-agent'
|
||||
if (rec.role === 'other') return 'non-dev-agent'
|
||||
if (!rec.clone) return 'no-clone'
|
||||
if (rank(rec.tier) >= rank('opus')) return 'opus-unverified'
|
||||
return undefined
|
||||
|
||||
+107
-2
@@ -66,6 +66,46 @@ test('fix 2: several clones and no Clone: line gives no role, and routes to opus
|
||||
expect(parseBrief(`Work in ${DEV}/. Commit in ${DEV}.`).role).toBe('dev')
|
||||
})
|
||||
|
||||
// ---------- non-dev roles (#7, EV-05) ----------
|
||||
|
||||
const EV05_DEV = '/home/homelab/projects/goodwe/clones/EV-05-dev'
|
||||
const EV05_DESIGN = '/home/homelab/projects/goodwe/clones/EV-05-design'
|
||||
const designBrief = (devClone = EV05_DEV) =>
|
||||
`Pipeline-Role: test-design\nClone: ${EV05_DESIGN} (Role: test-design)\n\n` +
|
||||
`Design the EV-05 tests. The developer works in ${devClone}/; do not touch it.`
|
||||
|
||||
test('#7: a test-design brief is a non-dev role, even when it names the dev clone', async () => {
|
||||
expect(parseBrief(designBrief())).toMatchObject({ clone: EV05_DESIGN, role: 'other', roleName: 'test-design' })
|
||||
// the header alone: other paths in the brief are never read for a role
|
||||
const b = parseBrief(`Pipeline-Role: test-design\nThe developer works in ${EV05_DEV}/.`)
|
||||
expect(b.role).toBe('other')
|
||||
expect(b.clone).toBe(undefined)
|
||||
})
|
||||
|
||||
test('#7: Pipeline-Role decides over (Role: ...) and the path', async () => {
|
||||
expect(parseBrief(`Pipeline-Role: review\nClone: ${EV05_DEV} (Role: dev)`).role).toBe('review')
|
||||
expect(parseBrief(`Pipeline-Role: dev\nClone: ${EV05_DESIGN}`).role).toBe('dev')
|
||||
expect(parseBrief(`Pipeline-Role: Dev\nWork in ${EV05_DEV}/.`).role).toBe('dev')
|
||||
// without the header, (Role: ...) still wins over the path
|
||||
expect(parseBrief(`Clone: ${EV05_DEV} (Role: test-design)`)).toMatchObject({ role: 'other', roleName: 'test-design' })
|
||||
})
|
||||
|
||||
test('#7: the role must end the folder name', async () => {
|
||||
expect(parseBrief('Work in /x/clones/EV-05-test-design/.').role).toBe(undefined)
|
||||
expect(parseBrief('Work in /x/clones/EV-05-dev-notes/.').role).toBe(undefined)
|
||||
expect(parseBrief('Work in /x/clones/EV-05-dev/.').role).toBe('dev')
|
||||
// a Clone: line the router cannot place keeps its model, unverified
|
||||
expect(parseBrief('Clone: /x/clones/EV-05-test-design').role).toBe('other')
|
||||
})
|
||||
|
||||
test('#7: other roles keep the named model, opus when none is named, and are never verified', async () => {
|
||||
expect(decide({ role: 'other', roleName: 'test-design', named: 'sonnet', prompt: 'x', respectExplicit: false }))
|
||||
.toEqual({ kind: 'fixed', tier: 'sonnet', reason: 'named model kept for test-design' })
|
||||
expect(decide({ role: 'other', prompt: 'x', respectExplicit: false })).toMatchObject({ kind: 'fixed', tier: 'opus' })
|
||||
expect(decide({ role: 'other', roleName: 'test-design', named: 'haiku', prompt: 'x', respectExplicit: false })).toMatchObject({ tier: 'opus' })
|
||||
expect(skipReason({ role: 'other', clone: EV05_DESIGN, tier: 'sonnet' }, 'answer')).toBe('non-dev-agent')
|
||||
})
|
||||
|
||||
// ---------- who picks the writer (C4) ----------
|
||||
|
||||
test('C4: review and test agents keep the model they are given', async () => {
|
||||
@@ -1146,8 +1186,8 @@ test('pipeline: a test designer keeps Opus; a named Sonnet is kept, Haiku never'
|
||||
expect(decide({ subagentType: 'test-designer', prompt: 'Write the test plan', respectExplicit: false })).toEqual({ kind: 'fixed', tier: 'opus', reason: 'test-designer default' })
|
||||
expect(decide({ subagentType: 'test-designer', named: 'sonnet', prompt: 'x', respectExplicit: false }).kind).toBe('fixed')
|
||||
expect(decide({ subagentType: 'test-designer', named: 'haiku', prompt: 'find x', respectExplicit: false })).toEqual({ kind: 'fixed', tier: 'opus', reason: 'test-designer default' })
|
||||
// a design clone path carries no role: no record, so never verified as dev work
|
||||
expect(parseBrief('Clone: /home/homelab/projects/goodwe/clones/EV-15-design').role).toBe(undefined)
|
||||
// a design clone path is no dev clone: a non-dev role, so never verified as dev work (0.5.1: was no role)
|
||||
expect(parseBrief('Clone: /home/homelab/projects/goodwe/clones/EV-15-design').role).toBe('other')
|
||||
})
|
||||
|
||||
test('pipeline: the verifier is told to check every T-id and N-id of the test plan', async ($, on) => {
|
||||
@@ -1159,3 +1199,68 @@ test('pipeline: the verifier is told to check every T-id and N-id of the test pl
|
||||
expect(w.verifierSystems[0]).toContain('every T-id must be implemented')
|
||||
expect(w.verifierPrompts[0]).toContain('## Test plan')
|
||||
})
|
||||
|
||||
// ---------- non-dev roles end to end (#7, EV-05) ----------
|
||||
|
||||
test('#7 acceptance: the test designer gets no line, also after a resume; the real dev gets T1 and a verdict', async ($, on) => {
|
||||
const w = fresh()
|
||||
w.diffs = ['diff --git a/ev.py']
|
||||
const clock = world(on, w)
|
||||
// the mock git answers in DEV only: the designer names that clone, so a misread would diff it
|
||||
const d = await $.agent.spawn({ prompt: designBrief(DEV), description: 'EV-05 test design', model: 'opus' } as never)
|
||||
expect(w.spawned[0]?.model).toBe('opus')
|
||||
await finish($, clock, d.agentId as string, 'claude-opus-5-5')
|
||||
await resume($, d.agentId as string, 'Add a case for a missing meter.')
|
||||
await finish($, clock, d.agentId as string, 'claude-opus-5-5')
|
||||
expect(w.all).toEqual([])
|
||||
expect(w.verifier).toEqual([])
|
||||
|
||||
const dev = await $.agent.spawn({
|
||||
prompt: `Pipeline-Role: dev\nClone: ${DEV} (Role: dev)\nBase branch: release/1.0.\nEV-05: add the meter staleness check as specified.`,
|
||||
description: 'EV-05 dev',
|
||||
} as never)
|
||||
expect(w.spawned[1]?.model).toBe('sonnet')
|
||||
await finish($, clock, dev.agentId as string, 'claude-sonnet-5-5')
|
||||
expect(w.all[0]).toBe('[model-router] T1 verifying on Opus...')
|
||||
expect(w.all[1]).toContain('PASS from verifier opus')
|
||||
expect(w.all.length).toBe(2)
|
||||
})
|
||||
|
||||
test('#7: a test-designer agent type that only names the dev clone is no dev either', async ($, on) => {
|
||||
const w = fresh()
|
||||
w.diffs = ['diff']
|
||||
const clock = world(on, w)
|
||||
const d = await $.agent.spawn({ prompt: `Write the tests for the work in ${DEV}/.`, description: 'design', subagentType: 'test-designer', model: 'haiku' } as never)
|
||||
expect(w.spawned[0]?.model).toBe('opus') // never Haiku
|
||||
await finish($, clock, d.agentId as string, 'claude-opus-5-5')
|
||||
expect(w.all).toEqual([])
|
||||
expect(w.verifier).toEqual([])
|
||||
const log = (await $.command.run({ command: 'router', args: '' } as never)).text ?? ''
|
||||
expect(log).toContain('A1')
|
||||
expect(log).toContain('A1 test-designer opus')
|
||||
expect(log).not.toMatch(/\bother\b/)
|
||||
})
|
||||
|
||||
test('#7: an unnamed test designer runs on opus, unverified', async ($, on) => {
|
||||
const w = fresh()
|
||||
w.diffs = ['diff']
|
||||
const clock = world(on, w)
|
||||
const d = await $.agent.spawn({ prompt: designBrief(), description: 'EV-05 test design' } as never)
|
||||
expect(w.spawned[0]?.model).toBe('opus')
|
||||
await finish($, clock, d.agentId as string, 'claude-sonnet-5-5')
|
||||
expect(w.all).toEqual([])
|
||||
expect(w.verifier).toEqual([])
|
||||
})
|
||||
|
||||
test('#7: a test designer is no saving and gets an A number; the log shows its role', async ($, on) => {
|
||||
const w = fresh()
|
||||
const clock = world(on, w)
|
||||
const d = await $.agent.spawn({ prompt: designBrief(), description: 'EV-05 test design', model: 'opus' } as never)
|
||||
await finish($, clock, d.agentId as string, 'claude-opus-5-5')
|
||||
const savings = (await $.command.run({ command: 'router', args: 'savings' } as never)).text ?? ''
|
||||
expect(savings).toBe('savings: no agent runs recorded yet.')
|
||||
const log = (await $.command.run({ command: 'router', args: '' } as never)).text ?? ''
|
||||
expect(log).toContain('A1')
|
||||
expect(log).toContain('test-design')
|
||||
expect(log).not.toContain('T1')
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user