Every piece of per-device state in the Controller was a scalar, and several of them are safety inputs — so a second inverter would have been graded against the first one's numbers.
what
what the scalar was silently asserting
dev_min/dev_max
one pair, read from one setpoint_entity. Inverter B's rating was never read at all, so its clamp, its freeze rail and the peak guard's achievable_w were inverter A's numbers.
soc
one reading into the maintenance FSM, SAFETY-24's pack floor and arbitrage's bounds. An average lets the emptier pack be driven below its own floor; a maximum parks the fuller one.
batt
one reading used as the loop's actual_w — see the measurement below.
sign_wd
oneSignWatchdog, whose own docstring says "one instance per actuator", latched under flat store keys. A trip caused by B latched A across restarts and one operator press cleared both.
last_write / last_beat
one deadline and one serialised hass.call(), so a hung write to A ate B's 30 s layer-1 budget.
app/inverter.py introduces an Inverter that owns its entities, its rating, its device range, its SignWatchdog, its last_write, its last_beat and namespaced store keys. The Controller holds a list of them and keeps only what is genuinely site-level: the P1 reading, the site integrator, the tariff quarter.
The measured finding ALLOC-01's review handed this card
compute()'s anti-windup test is abs(prev_w - actual_w) > saturation_w, and prev_w is the site target. Fed one inverter's power it compares a site number against a unit number, permanently trips the saturation freeze, and leaves a standing grid error. On admin/goodwe's two-master rig — tools/alloc_sim.py --splits, 2 kW load, 40 cycles, the shipped compute() closing the loop against plantsim.Plant with the firmware's own encoder on the wire — worst |delta| between splits at the meter:
gain
one inverter's power (before)
the site sum (after)
0.2
510 W
0 W
0.4
250 W
0 W
0.6
50 W
0 W
0.8 – 1.0
0 W
0 W
So actual_w became site level while the per-unit readings became per-instance. Those are opposite directions and getting them backwards reintroduces the divergence. test_alloc.py part 8 keeps the old stimulus — the defect is still reproducible there, on a plant that file models itself — and now grades the other row too; the same experiment run against the rig is the table above.
What I chose, and why
The store migration is graded, not merely performed.sign_tripped/sign_reason move to unit/<name>/… once, verified by reading back what was written, and the version marker is set only when nothing that carried state was lost. Three rules, each a case that would otherwise be a wrong answer: it never overwrites an existing namespaced value (so an operator's clear is not resurrected); the flat keys are left in the file (so a rollback still finds its latch, and the staleness then fails toward a spurious 0 W hold rather than a missing one); the version marker is written last. And Controller.__init__ compares the pre-migration flat latch against what unit A actually came up with — if the latch did not survive it is re-armed from the legacy value and shouted about, because a latch that silently fails to migrate is a disarmed safety guard on an install whose state file says it is protected. test_inverter.py part 2 breaks the migration and watches the guard fire.
No per-instance Tuning, and that is a deliberate deviation from the card's wording. The card lists Tuning among the things the object should own. A per-unit Tuning is a per-unit control law waiting to happen, and §6.2 / ALLOC-01's stability property is one integrator on P1 above the split. The one field of Tuning that genuinely is per-device — max_w, the supervised cap on what may be written to one setpoint entity — is on the object as Inverter.max_w, and the siteTuning takes its bound from sum(inv.max_w). At N=1 that is max_w, byte for byte. The argument is written where a reader looking for the missing Tuning will find it.
live_units() answers one question: can this controller put a number on this inverter right now. A unit whose every write has been rejected for a whole LINK_STALE_S window (= validate.FIRMWARE_INGEST_VETO_S, 30 s — the point at which the firmware takes that inverter to zero anyway) is dropped; the survivors carry its share, the action log names it, and the dropped unit is commanded 0 W rather than left standing on its last share. It is deliberately not "its sensors are missing" (that is inputs_ok, a site stop) and deliberately not "it has no setpoint_entity" — that last one is a configuration that cannot be delivered, and it reaches main.py's refusal branch, which is how that branch keeps a live trigger now that DEV-02 removed its old one. test_peakguard.py 10h-1b drives it.
Heartbeats: concurrent dispatch plus a per-write budget.WRITE_BUDGET_S = 4 s, and the inequality is graded rather than asserted — 4 + heartbeat_s's schema maximum of 25 < 30. send_heartbeats() was extracted from run_control() so the shipped dispatch is reachable by a check at all (see the mutation note below).
Site aggregates are named for the question they answer, with the reduction argued at the definition: site_max_w (sum), site_rating_w (sum of the per-unit min(max_w, |dev|) conjunctions), site_batt_w (sum), pack_floor_soc (minimum — a discharge permission must be graded against the emptiest pack).
Evidence
19 files green, ~4 min. test_inverter.py is new (82 checks). test_alloc.py part 8, test_peakguard.py 10g/10h, test_watchdog.py and test_clamp.py were re-aimed rather than relaxed — every one of them grades something stronger now than it did.
Eighteen mutations, each applied to the shipped module on a scratch copy, purged, and run through tools/mutation_run.sh; every one observed RED. The table is in test_inverter.py part 9.
⚠️Two of them survived the first time and I changed the checks, not the table. The serialised dispatch and the shared last_beat both passed while part 6 gathered its ownwrite_setpoint() calls and computed its owndue list — a check on the test file rather than on the add-on. That is this repository's own named failure mode, found once more here; send_heartbeats() exists because of it, and the episode is recorded in the file rather than quietly fixed.
Rig runs are read-only against admin/goodwe @ 0c31e6f; nothing there was modified and no hardware was touched.
What I am unsure about
arbitrage's single soc argument. I pass pack_floor_soc (the emptiest pack). That is right for the sell side and wrong for the buy side, which wants the fullest pack — one scalar cannot be conservative in both directions. It is named as a residual in the code rather than hidden, and at the shipped roster of one the two reductions are the same number. A per-pack charge ceiling belongs to CFG-04/SPIKE-04; if you would rather I refused arbitrage outright at N≥2 until then, say so and I will.
The maintenance FSM is still one object. Its keys (phase, last_completed, …) deliberately did not move into a namespace — re-homing a month-scale history under a unit name is a different decision from namespacing a latch. The claim is now filed on a named maint_unit rather than on whichever unit happened to be asked, and the FSM is graded on that pack's own SoC. Two packs recalibrating on two schedules needs a second FSM with its own namespaced history.
inputs_ok requires every roster unit's readings. At N≥2 one silent pack stops the whole site at P_SAFETY. I think that is right (§11.1 rule 1, and the alternative hands its share to a sibling while the site keeps regulating against a pack nobody read), but it is a policy choice and it is the one that most changes what a two-inverter site does on a bad day.
SAFETY-28 touches write_setpoint (~:409-450), which this card split into a site-level write_setpoint(value, *, force, unit=None) and a per-unit _write_unit(). Expect a merge there; whoever lands second merges release/1.0 and re-runs.
A hazard this card created, found by re-reading rather than by a failing check
Second commit. Before DEV-02 the heartbeat re-emitted self.target, so the stop button zeroing that one scalar was enough. It now re-emits inv.target — it must, or it would put the site number on every inverter, which is the ARCH-01 broadcast fault arriving once every heartbeat_s on a path with no allocator in it. So a stop that zeroed only self.target left each unit's last share standing: no cycle has run at that moment, nothing else clears it, and the very next heartbeat puts it back on the wire. The operator presses stop, watches the inverter go to zero, and sees it resume seconds later.
command_site_zero() moves both together — one method, because the two lines have to stay together, and shutdown(), the stop button and maint_abort all mean the same thing. test_inverter.py part 6 presses stop with 995/995 standing and then runs the shippedsend_heartbeats() before any cycle; the mutation that zeroes the site target alone takes it red (mutation 19).
I am reporting this rather than folding it silently into the first commit because it is the shape of defect this repository keeps finding — a refactor that moves what a path re-emits without moving what clears it — and the next person to touch the heartbeat should see that it was reachable.
Review round 2 — both blockers closed, and release/1.0 merged in (head 753f700)
release/1.0 @ f594543 (SAFETY-28 + the ALLOC-01 fix) is merged. 19/19 files green on the merged tree.
Blocker 1 — AC 4's arbiter-bound evidence was vacuous.second_inverter() defaulted B's rating and device range to A's LIVE_MAX_W, so the check that claimed "each unit's supervised bound is ITS OWN max_w" compared (-5000, 5000) against (-5000, 5000). Filing the per-unit device/supervisedbounds from self.inverters[0] — with actuator= left correct — passed all 19 files. Fixed at the helper rather than at one call site: B is 3000 W everywhere in the file, the bounds are asserted against literal numbers for supervisedanddevice, the resolved envelope is checked as well as the filed claim, and a PRECONDITION check fails if the two ratings are ever made equal again. The BOUND mutation joins the ACTUATOR one in the table and was observed RED (3 checks in test_peakguard.py).
Blocker 2 — the SAFETY-02 latch was silently disarmed on the rollback round trip. Migrate → downgrade → trip inside the downgrade window (old code writes only the flat key) → upgrade: rep.ran is False, the guard was skipped, and the controller came up unlatched while /data/state.json said sign_tripped: true. The three-step scenario is now a check, preconditions included. Three changes, four mutations each observed RED:
the operator's clear clears both spellings, which is what lets the guard run on every start instead of only the one that migrates;
a trip mirrors the flat pair — rule 2 only ever covered the latch present at migration time, so a downgrade after a post-migration trip found no latch at all. store.py's "the staleness fails toward a SPURIOUS latch, never a missing one" was untrue and is corrected;
the guard asks the store, not the roster, whether anything namespaced is latched, so a mirrored trip on B cannot re-arm a healthy A.
test_watchdog.py's "the flat key is NOT written" assertion is re-aimed (right about the read, wrong about the write). HANDOVER.md / DOCS.md no longer tell an operator the flat key "is never read again".
Merge cost.write_refusal() takes the inverter it is about — asking it with self.unit would authorise A's share on B's wire and would trip part 1's AST scan. Every fixture that calls write_setpoint() directly now arms a real allocation record through the shipped resolve(); test_clamp.py's two-unit fixture appends a real Inverter rather than assigning the now-derived c.units.
MUTATION.md gains a third failure mode beside stale bytecode (TEST-15) and masking (TEST-16): a check whose data cannot reach the failure it names, with this week's three instances and the rule — any check asserting "X differs from Y" must be run once with X and Y actually made different.
## The defect
Every piece of per-device state in the `Controller` was a **scalar**, and several of them are safety inputs — so a second inverter would have been graded against the first one's numbers.
| what | what the scalar was silently asserting |
|---|---|
| `dev_min`/`dev_max` | one pair, read from one `setpoint_entity`. **Inverter B's rating was never read at all**, so its clamp, its freeze rail and the peak guard's `achievable_w` were inverter A's numbers. |
| `soc` | one reading into the maintenance FSM, SAFETY-24's pack floor and arbitrage's bounds. An average lets the emptier pack be driven below its own floor; a maximum parks the fuller one. |
| `batt` | one reading used as the loop's `actual_w` — see the measurement below. |
| `sign_wd` | **one** `SignWatchdog`, whose own docstring says *"one instance per actuator"*, latched under **flat** store keys. A trip caused by B latched A across restarts and one operator press cleared both. |
| `last_write` / `last_beat` | one deadline and one serialised `hass.call()`, so a hung write to A ate B's 30 s layer-1 budget. |
`app/inverter.py` introduces an `Inverter` that owns its entities, its rating, its device range, its `SignWatchdog`, its `last_write`, its `last_beat` and **namespaced store keys**. The Controller holds a list of them and keeps only what is genuinely site-level: the P1 reading, the site integrator, the tariff quarter.
## The measured finding ALLOC-01's review handed this card
`compute()`'s anti-windup test is `abs(prev_w - actual_w) > saturation_w`, and `prev_w` is the **site** target. Fed one inverter's power it compares a site number against a unit number, permanently trips the saturation freeze, and leaves a standing grid error. On `admin/goodwe`'s two-master rig — `tools/alloc_sim.py --splits`, 2 kW load, 40 cycles, the shipped `compute()` closing the loop against `plantsim.Plant` with the firmware's own encoder on the wire — worst `|delta|` between splits at the meter:
| gain | one inverter's power (before) | the site sum (after) |
|---|---|---|
| 0.2 | **510 W** | 0 W |
| 0.4 | **250 W** | 0 W |
| 0.6 | **50 W** | 0 W |
| 0.8 – 1.0 | 0 W | 0 W |
So `actual_w` became **site level** while the per-unit readings became **per-instance**. Those are opposite directions and getting them backwards reintroduces the divergence. `test_alloc.py` part 8 keeps the old stimulus — the defect is still reproducible there, on a plant that file models itself — and now grades the other row too; the same experiment run against the rig is the table above.
## What I chose, and why
**The store migration is graded, not merely performed.** `sign_tripped`/`sign_reason` move to `unit/<name>/…` once, verified by **reading back** what was written, and the version marker is set only when nothing that carried state was lost. Three rules, each a case that would otherwise be a wrong answer: it never overwrites an existing namespaced value (so an operator's clear is not resurrected); the flat keys are left in the file (so a rollback still finds its latch, and the staleness then fails toward a *spurious* 0 W hold rather than a missing one); the version marker is written last. And `Controller.__init__` compares the pre-migration flat latch against what unit A actually came up with — if the latch did not survive it is **re-armed from the legacy value and shouted about**, because a latch that silently fails to migrate is a disarmed safety guard on an install whose state file says it is protected. `test_inverter.py` part 2 breaks the migration and watches the guard fire.
**No per-instance `Tuning`, and that is a deliberate deviation from the card's wording.** The card lists `Tuning` among the things the object should own. A per-unit `Tuning` is a per-unit control law waiting to happen, and §6.2 / ALLOC-01's stability property is **one** integrator on P1 above the split. The one field of `Tuning` that genuinely is per-device — `max_w`, the supervised cap on what may be written to one setpoint entity — is on the object as `Inverter.max_w`, and the **site** `Tuning` takes its bound from `sum(inv.max_w)`. At N=1 that is `max_w`, byte for byte. The argument is written where a reader looking for the missing `Tuning` will find it.
**`live_units()` answers one question: can this controller put a number on this inverter right now.** A unit whose *every* write has been rejected for a whole `LINK_STALE_S` window (= `validate.FIRMWARE_INGEST_VETO_S`, 30 s — the point at which the firmware takes that inverter to zero anyway) is dropped; the survivors carry its share, the action log names it, and the dropped unit is commanded **0 W** rather than left standing on its last share. It is deliberately **not** "its sensors are missing" (that is `inputs_ok`, a site stop) and deliberately **not** "it has no `setpoint_entity`" — that last one is a *configuration* that cannot be delivered, and it reaches main.py's refusal branch, which is how that branch keeps a live trigger now that DEV-02 removed its old one. `test_peakguard.py` 10h-1b drives it.
**Heartbeats: concurrent dispatch plus a per-write budget.** `WRITE_BUDGET_S = 4 s`, and the inequality is graded rather than asserted — `4 + heartbeat_s`'s schema maximum of `25 < 30`. `send_heartbeats()` was **extracted** from `run_control()` so the shipped dispatch is reachable by a check at all (see the mutation note below).
**Site aggregates are named for the question they answer**, with the reduction argued at the definition: `site_max_w` (sum), `site_rating_w` (sum of the per-unit `min(max_w, |dev|)` conjunctions), `site_batt_w` (sum), `pack_floor_soc` (**minimum** — a discharge permission must be graded against the emptiest pack).
## Evidence
* **19 files green**, ~4 min. `test_inverter.py` is new (82 checks). `test_alloc.py` part 8, `test_peakguard.py` 10g/10h, `test_watchdog.py` and `test_clamp.py` were **re-aimed rather than relaxed** — every one of them grades something stronger now than it did.
* **Eighteen mutations**, each applied to the shipped module on a scratch copy, purged, and run through `tools/mutation_run.sh`; every one observed RED. The table is in `test_inverter.py` part 9.
* ⚠️ **Two of them survived the first time and I changed the checks, not the table.** The serialised dispatch and the shared `last_beat` both passed while part 6 gathered its *own* `write_setpoint()` calls and computed its *own* `due` list — a check on the test file rather than on the add-on. That is this repository's own named failure mode, found once more here; `send_heartbeats()` exists because of it, and the episode is recorded in the file rather than quietly fixed.
* Rig runs are read-only against `admin/goodwe` @ `0c31e6f`; nothing there was modified and no hardware was touched.
## What I am unsure about
1. **`arbitrage`'s single `soc` argument.** I pass `pack_floor_soc` (the emptiest pack). That is right for the sell side and **wrong for the buy side**, which wants the fullest pack — one scalar cannot be conservative in both directions. It is named as a residual in the code rather than hidden, and at the shipped roster of one the two reductions are the same number. A per-pack charge ceiling belongs to CFG-04/SPIKE-04; if you would rather I refused arbitrage outright at N≥2 until then, say so and I will.
2. **The maintenance FSM is still one object.** Its keys (`phase`, `last_completed`, …) deliberately did **not** move into a namespace — re-homing a month-scale history under a unit name is a different decision from namespacing a latch. The claim is now filed on a named `maint_unit` rather than on whichever unit happened to be asked, and the FSM is graded on that pack's own SoC. Two packs recalibrating on two schedules needs a second FSM with its own namespaced history.
3. **`inputs_ok` requires every roster unit's readings.** At N≥2 one silent pack stops the whole site at P_SAFETY. I think that is right (§11.1 rule 1, and the alternative hands its share to a sibling while the site keeps regulating against a pack nobody read), but it is a policy choice and it is the one that most changes what a two-inverter site does on a bad day.
4. **`SAFETY-28` touches `write_setpoint`** (`~:409-450`), which this card split into a site-level `write_setpoint(value, *, force, unit=None)` and a per-unit `_write_unit()`. Expect a merge there; whoever lands second merges `release/1.0` and re-runs.
Base: `release/1.0` @ `3030f72`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
## A hazard this card created, found by re-reading rather than by a failing check
Second commit. Before DEV-02 the heartbeat re-emitted `self.target`, so the stop button zeroing that one scalar was enough. It now re-emits `inv.target` — it must, or it would put the **site** number on every inverter, which is the ARCH-01 broadcast fault arriving once every `heartbeat_s` on a path with no allocator in it. So a stop that zeroed only `self.target` left each unit's **last share** standing: no cycle has run at that moment, nothing else clears it, and the very next heartbeat puts it back on the wire. The operator presses stop, watches the inverter go to zero, and sees it resume seconds later.
`command_site_zero()` moves both together — one method, because the two lines have to stay together, and `shutdown()`, the stop button and `maint_abort` all mean the same thing. `test_inverter.py` part 6 presses stop with 995/995 standing and then runs the **shipped** `send_heartbeats()` before any cycle; the mutation that zeroes the site target alone takes it red (mutation 19).
I am reporting this rather than folding it silently into the first commit because it is the shape of defect this repository keeps finding — a refactor that moves *what* a path re-emits without moving *what* clears it — and the next person to touch the heartbeat should see that it was reachable.
---
## Review round 2 — both blockers closed, and `release/1.0` merged in (head `753f700`)
`release/1.0` @ `f594543` (SAFETY-28 + the ALLOC-01 fix) is merged. 19/19 files green on the merged tree.
**Blocker 1 — AC 4's arbiter-bound evidence was vacuous.** `second_inverter()` defaulted B's rating and device range to A's `LIVE_MAX_W`, so the check that claimed *"each unit's `supervised` bound is ITS OWN `max_w`"* compared `(-5000, 5000)` against `(-5000, 5000)`. Filing the per-unit `device`/`supervised` **bounds** from `self.inverters[0]` — with `actuator=` left correct — passed all 19 files. Fixed at the helper rather than at one call site: B is 3000 W everywhere in the file, the bounds are asserted against **literal** numbers for `supervised` **and** `device`, the **resolved** envelope is checked as well as the filed claim, and a **PRECONDITION check fails if the two ratings are ever made equal again**. The BOUND mutation joins the ACTUATOR one in the table and was observed RED (3 checks in `test_peakguard.py`).
**Blocker 2 — the SAFETY-02 latch was silently disarmed on the rollback round trip.** Migrate → downgrade → trip inside the downgrade window (old code writes only the flat key) → upgrade: `rep.ran` is False, the guard was skipped, and the controller came up unlatched while `/data/state.json` said `sign_tripped: true`. The three-step scenario is now a check, preconditions included. Three changes, four mutations each observed RED:
* the operator's clear clears **both** spellings, which is what lets the guard **run on every start** instead of only the one that migrates;
* **a trip mirrors the flat pair** — rule 2 only ever covered the latch present *at migration time*, so a downgrade after a post-migration trip found no latch at all. `store.py`'s "the staleness fails toward a SPURIOUS latch, never a missing one" was untrue and is corrected;
* the guard asks the **store**, not the roster, whether anything namespaced is latched, so a mirrored trip on B cannot re-arm a healthy A.
`test_watchdog.py`'s "the flat key is NOT written" assertion is re-aimed (right about the read, wrong about the write). `HANDOVER.md` / `DOCS.md` no longer tell an operator the flat key "is never read again".
**Merge cost.** `write_refusal()` takes the **inverter** it is about — asking it with `self.unit` would authorise A's share on B's wire and would trip part 1's AST scan. Every fixture that calls `write_setpoint()` directly now arms a real allocation record through the shipped `resolve()`; `test_clamp.py`'s two-unit fixture appends a real `Inverter` rather than assigning the now-derived `c.units`.
**`MUTATION.md` gains a third failure mode** beside stale bytecode (TEST-15) and masking (TEST-16): *a check whose data cannot reach the failure it names*, with this week's three instances and the rule — **any check asserting "X differs from Y" must be run once with X and Y actually made different**.
Every piece of per-device state in the Controller was a scalar, and several of
them are safety inputs. app/inverter.py introduces an `Inverter` that owns its
entities, its rating, its device range, its SignWatchdog, its last_write and
its last_beat, under store keys namespaced by unit name. The Controller holds a
list of them and keeps only what is genuinely site-level: the P1 reading, the
site integrator, and the tariff quarter.
WHAT THE SCALARS WERE SILENTLY ASSERTING
dev_min/dev_max one pair, read from one setpoint_entity. A second
inverter's rating was NEVER READ AT ALL, so its clamp, its
freeze rail and the peak guard's achievable_w were the
first one's numbers.
soc one reading into the maintenance FSM, SAFETY-24's pack
floor and arbitrage's bounds. An average lets the emptier
pack be driven below its own floor.
sign_wd ONE SignWatchdog, whose own docstring says "one instance
per actuator", latched under FLAT store keys - so a trip
caused by B latched A across restarts and one operator
press cleared both.
last_beat one deadline and one serialised hass.call(), so a hung
write to A ate B's 30 s layer-1 budget.
actual_w BECAME SITE LEVEL WHILE THE READINGS BECAME PER-INSTANCE, and those
are opposite directions on purpose. compute()'s anti-windup test is
`abs(prev_w - actual_w) > saturation_w` with prev_w the SITE target, so feeding
it one inverter's power compares a site number against a unit number, trips the
saturation freeze permanently and leaves a standing grid error. Measured on
admin/goodwe's two-master rig (tools/alloc_sim.py --splits, 2 kW load, 40
cycles), worst |delta| between splits at the meter:
gain one unit's power the site sum
0.2 510 W 0 W
0.4 250 W 0 W
0.6 50 W 0 W
That was ALLOC-01's disclosed finding; test_alloc.py part 8 kept its stimulus
and now grades the other row.
THE STORE MIGRATION IS GRADED, NOT MERELY PERFORMED. sign_tripped/sign_reason
move to unit/<name>/... once, verified by READING BACK what was written, and
the version marker is set only when nothing that carried state was lost. The
flat keys stay in the file so a rollback still finds its latch. And __init__
compares the pre-migration flat latch against what unit A actually came up
with: if the latch did not survive it is RE-ARMED from the legacy value and
shouted about. A latch that silently fails to migrate is a disarmed safety
guard on an install whose state file says it is protected - which is strictly
worse than never persisting it, because now there is evidence saying it is
fine. test_inverter.py part 2 breaks the migration and watches the guard fire.
HEARTBEATS ARE INDEPENDENT AND THE BUDGET IS DERIVED. Each inverter owns its
deadline, the refreshes go out concurrently, and each write is bounded by
WRITE_BUDGET_S = 4 s. The inequality is graded rather than asserted: 4 +
heartbeat_s's schema maximum of 25 < 30, validate.FIRMWARE_INGEST_VETO_S.
Verified by injecting a transport that accepts and never answers.
live_units() ANSWERS HONESTLY. A unit whose every write has been rejected for a
whole layer-1 window is not commandable; the allocator drops it, the survivors
carry its share, the action log names it, and the dropped unit is commanded
0 W rather than left standing on its last share. A roster entry with NO
setpoint_entity is deliberately not handled there - that is a configuration
that cannot be delivered, and it reaches main.py's refusal branch, which is how
that branch keeps a live trigger after this card removed its old one.
AT N=1 NOTHING MOVES. The removed scalars survive as read/write properties of
the sole inverter, so every existing replay runs unmodified - and they REFUSE
at N>=2 rather than answering for the first inverter, which is the tripwire on
a consumer that was never made per-instance.
EVIDENCE. 19 files green. test_inverter.py is new (82 checks). Eighteen
mutations were applied to the shipped modules on a purged tree via
tools/mutation_run.sh and each was observed RED. Two of them SURVIVED first
time and the checks were changed rather than the table: the serialised dispatch
and the shared last_beat both passed while part 6 gathered its own writes and
computed its own `due` list. send_heartbeats() was extracted from run_control()
so the shipped dispatch is reachable by a check at all.
INVERTER B STILL DOES NOT EXIST. build_roster() returns exactly one inverter
from the flat options and is the one function CFG-04 changes; every two-unit
figure here is architecture, and the second rating in the checks is marked a
hypothesis.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
A hazard THIS CARD CREATED, found by re-reading the operator paths rather than
by a failing check - so a check now exists for it.
Before DEV-02 the heartbeat re-emitted `self.target`, so the stop button
zeroing that one scalar was enough: the next refresh carried the zero. The
heartbeat now re-emits `inv.target` - it has to, or it would put the SITE
number on every inverter, which is the ARCH-01 broadcast fault arriving once
every `heartbeat_s` on a path with no allocator in it.
So a stop that zeroed only `self.target` left each unit's LAST SHARE standing.
No cycle has run yet at that moment, so nothing else would have cleared it, and
the very next heartbeat - up to `heartbeat_s` later - would put it back on the
wire. The operator presses stop, watches the inverter go to zero, and sees it
resume seconds later.
`command_site_zero()` moves both together and carries the reason. It is one
method rather than two lines at three call sites precisely because the two
lines have to stay together: shutdown(), the stop button and maint_abort all
mean the same thing, and 0 W is the one value that may legitimately be written
site-wide.
test_inverter.py part 6 presses stop with 995/995 standing and then runs the
SHIPPED `send_heartbeats()` before any cycle, which is the racing refresh. The
mutation that zeroes the site target alone takes it red.
19 files green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
HANDOVER.md told an operator to "delete `sign_tripped` from
/data/state.json" to clear a sign-disagreement trip by hand. After this card
that key is a FOSSIL: the live latch is `unit/<inverter>/sign_tripped`, and
the flat one is left in the file only so a rollback finds its latch and is
never read again. Following the old instruction would delete a key that clears
nothing, on the one path an operator reaches while staring at a FAILSAFE
banner - so the instruction would look like it had failed.
DOCS.md's §SAFETY-02 section named the file but not the key; it now names both,
and says why there is one per inverter.
No code change; the suite is unaffected and green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
PR #43 was rejected on two findings and had gone unmergeable behind SAFETY-28
and the ALLOC-01 fix. This is the merge commit; the conflict resolution and both
fixes are below, because a merge that also changes behaviour has to say so.
THE MERGE, AND WHAT IT COST
===========================
Two conflicts in main.py, both where SAFETY-28 and this card touched the same
method:
* `write_setpoint()` split into a site entry point plus a per-unit
`_write_unit()` here, while SAFETY-28 put its record gate in the old
single-unit body. `write_refusal()` now takes the INVERTER it is about.
Asking it with `self.unit` - as release/1.0 does - would authorise A's share
on B's wire, and would also trip test_inverter.py part 1's AST scan, which
goes red on any read of a Controller N=1 view inside Controller.
* the SAFETY-28 gate sits above the clamp and below the non-finite coercion,
exactly where release/1.0 put it, and is now asked about `inv`.
Fallout the merge produced in the checks, re-aimed rather than relaxed:
* every fixture that calls `write_setpoint()` DIRECTLY - the clamp probe, the
hung-heartbeat experiment and its serialised counterfactual, the split
deadline probe - now arms a real allocation record first. test_inverter.py
grows an `arm()` built through the shipped `resolve()`, the same shape
test_clamp.py's is: the per-unit envelopes are filed as real `supervised`
claims and the site target is their sum, so the water-fill has exactly one
answer and the fixture cannot quietly become its own allocator.
* test_clamp.py's two-unit failsafe fixture assigned `c.units`, which DEV-02
made a derived property with no setter. It appends a real `Inverter`.
BLOCKER 1 - AC 4's ARBITER-BOUND EVIDENCE WAS VACUOUS
=====================================================
test_peakguard.py claimed "each unit's `supervised` bound is ITS OWN `max_w` -
give inverter B a different rating and the claim filed on inverter B moves with
it", and compared B's bound against `_b2.max_w`. `second_inverter()` DEFAULTED
B's rating to `LIVE_MAX_W`, which is A's. IT COMPARED (-5000, 5000) AGAINST
(-5000, 5000): a number against itself, unable to observe the failure it named.
Reproduced: filing the per-unit `device`/`supervised` claims from
`self.inverters[0]`'s numbers while leaving `actuator=self.actuator_of(..., inv)`
alone left ALL 19 FILES GREEN. At a roster of two with A=5000/B=3000 that makes
inverter B's planned envelope (-5000, 5000) instead of (-3000, 3000) - a 2000 W
over-command the arbiter believes is supervised. It is a DIFFERENT defect from
the actuator variant the static routing pin catches, and only the actuator
variant was in the mutation table.
* `second_inverter()` gives B 3000 W - a hypothesis, marked as one - for every
fixture in the file, so no check there can compare B against A by accident.
* the claim is asserted against LITERAL numbers rather than against an
expression that can read the same object twice, for `supervised` AND for
`device` (whose `dev` had the same defect), plus the envelope the ARBITER
RESOLVED for B - a claim filed right and resolved wrong is the same defect
one layer down.
* a PRECONDITION check fails if the two ratings are ever made equal again.
* the BOUND mutation is in the table, beside the actuator one, and was
observed RED (3 checks in test_peakguard.py) via tools/mutation_run.sh.
⚠️ THIRD INSTANCE THIS SESSION of a check whose DATA cannot reach the failure it
names - ARB-03's call-site scan saw 3 of 8 appends, ALLOC-01's replay generator
saw 0 escapes where a widened one saw 39,991. MUTATION.md gains a section for it
beside the stale-bytecode and masking ones, and the rule is written at the line:
ANY CHECK ASSERTING "X DIFFERS FROM Y" MUST BE RUN ONCE WITH X AND Y ACTUALLY
MADE DIFFERENT.
BLOCKER 2 - THE LATCH WAS SILENTLY DISARMED BY THE ROLLBACK ROUND TRIP
======================================================================
store.py's rule 2 keeps the flat keys "so a rollback still finds its latch". The
operator path that rule contemplates is a ROUND TRIP, and this card lost the
latch on it:
1. install migrates -> schema_version=1, unit/inverter-a/... = False
2. operator DOWNGRADES; SAFETY-02 trips inside the downgrade window and the
old add-on writes only the flat `sign_tripped: true`
3. operator upgrades again -> `rep.ran` is False, so `_grade_store_migration()`
returned early, and `Inverter.__init__` read the namespaced key: False
Observed: the controller came up with `sign_wd.tripped = False` while
/data/state.json said `sign_tripped: true`. That is store.py's own header
sentence - "the file still says sign_tripped: true, the banner is green, and the
controller resumes driving the wrong way" - rebuilt by the version marker. A
firmware version change with a rollback path was performed on this site on
2026-09-02, so this is not hypothetical.
Three changes, each graded by a mutation observed RED:
* THE OPERATOR'S CLEAR CLEARS BOTH SPELLINGS. That is what removes the
"re-arm forever" objection the `rep.ran` gate existed to answer: with the
fossil cleared, `flat True + namespaced False` is unambiguously a
downgrade-window latch.
* A TRIP MIRRORS THE FLAT PAIR. Rule 2 only ever covered the latch present AT
MIGRATION TIME - nothing flat was written for a trip taken afterwards, so an
operator who downgraded found `sign_tripped: false` and came up DRIVING.
store.py claimed the staleness could only fail toward a SPURIOUS latch. It
could not; that sentence is corrected rather than left standing.
* THE GUARD RUNS ON EVERY START, and asks the STORE - not the roster - whether
anything namespaced is latched. A mirrored trip on B must not re-arm a
healthy A, and the store is the right place to ask because it holds every
`unit/*/sign_tripped` this install has written, including one belonging to a
unit that is not in today's roster.
Graded by the three-step scenario itself, run end to end in test_inverter.py
part 2, with all three preconditions asserted - including that the catching
start did NOT migrate anything, which is what used to skip the guard. Part 2's
idempotence check now PRESSES the button instead of hand-writing the store,
because the state it used to construct is not one the shipped clear path can
produce. Part 3 grades the mirror and the clear on a real roster of two.
test_watchdog.py's "the flat key is NOT written" assertion is re-aimed rather
than deleted: its argument was right about the READ and wrong about the WRITE.
Nothing at runtime reads that key, and the one thing that does ignores it
whenever any namespaced latch is set - which a sibling's trip always leaves.
What its absence actually cost was the rollback.
HANDOVER.md and DOCS.md said the flat key "is never read again", which was the
instruction an operator clearing a trip by hand would follow. Both corrected: it
is a maintained rollback view, it IS read on every start, and clearing a trip by
hand means clearing both spellings.
MUTATIONS THIS ROUND, each applied on a purged tree via tools/mutation_run.sh
and observed RED:
* per-unit `device`/`supervised` BOUNDS from `self.inverters[0]`, actuator
left correct -> test_peakguard.py 10g
* `_grade_store_migration()` gated on `rep.ran` -> test_inverter.py part 2
* the operator's clear leaving the flat keys set -> test_inverter.py part 2/3
* a trip not mirroring the flat pair -> test_inverter.py part 3
* the guard asking only `self.inverters[0]` -> test_inverter.py part 3
19/19 files green on the merged tree. No hardware was touched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The defect
Every piece of per-device state in the
Controllerwas a scalar, and several of them are safety inputs — so a second inverter would have been graded against the first one's numbers.dev_min/dev_maxsetpoint_entity. Inverter B's rating was never read at all, so its clamp, its freeze rail and the peak guard'sachievable_wwere inverter A's numbers.socbattactual_w— see the measurement below.sign_wdSignWatchdog, whose own docstring says "one instance per actuator", latched under flat store keys. A trip caused by B latched A across restarts and one operator press cleared both.last_write/last_beathass.call(), so a hung write to A ate B's 30 s layer-1 budget.app/inverter.pyintroduces anInverterthat owns its entities, its rating, its device range, itsSignWatchdog, itslast_write, itslast_beatand namespaced store keys. The Controller holds a list of them and keeps only what is genuinely site-level: the P1 reading, the site integrator, the tariff quarter.The measured finding ALLOC-01's review handed this card
compute()'s anti-windup test isabs(prev_w - actual_w) > saturation_w, andprev_wis the site target. Fed one inverter's power it compares a site number against a unit number, permanently trips the saturation freeze, and leaves a standing grid error. Onadmin/goodwe's two-master rig —tools/alloc_sim.py --splits, 2 kW load, 40 cycles, the shippedcompute()closing the loop againstplantsim.Plantwith the firmware's own encoder on the wire — worst|delta|between splits at the meter:So
actual_wbecame site level while the per-unit readings became per-instance. Those are opposite directions and getting them backwards reintroduces the divergence.test_alloc.pypart 8 keeps the old stimulus — the defect is still reproducible there, on a plant that file models itself — and now grades the other row too; the same experiment run against the rig is the table above.What I chose, and why
The store migration is graded, not merely performed.
sign_tripped/sign_reasonmove tounit/<name>/…once, verified by reading back what was written, and the version marker is set only when nothing that carried state was lost. Three rules, each a case that would otherwise be a wrong answer: it never overwrites an existing namespaced value (so an operator's clear is not resurrected); the flat keys are left in the file (so a rollback still finds its latch, and the staleness then fails toward a spurious 0 W hold rather than a missing one); the version marker is written last. AndController.__init__compares the pre-migration flat latch against what unit A actually came up with — if the latch did not survive it is re-armed from the legacy value and shouted about, because a latch that silently fails to migrate is a disarmed safety guard on an install whose state file says it is protected.test_inverter.pypart 2 breaks the migration and watches the guard fire.No per-instance
Tuning, and that is a deliberate deviation from the card's wording. The card listsTuningamong the things the object should own. A per-unitTuningis a per-unit control law waiting to happen, and §6.2 / ALLOC-01's stability property is one integrator on P1 above the split. The one field ofTuningthat genuinely is per-device —max_w, the supervised cap on what may be written to one setpoint entity — is on the object asInverter.max_w, and the siteTuningtakes its bound fromsum(inv.max_w). At N=1 that ismax_w, byte for byte. The argument is written where a reader looking for the missingTuningwill find it.live_units()answers one question: can this controller put a number on this inverter right now. A unit whose every write has been rejected for a wholeLINK_STALE_Swindow (=validate.FIRMWARE_INGEST_VETO_S, 30 s — the point at which the firmware takes that inverter to zero anyway) is dropped; the survivors carry its share, the action log names it, and the dropped unit is commanded 0 W rather than left standing on its last share. It is deliberately not "its sensors are missing" (that isinputs_ok, a site stop) and deliberately not "it has nosetpoint_entity" — that last one is a configuration that cannot be delivered, and it reaches main.py's refusal branch, which is how that branch keeps a live trigger now that DEV-02 removed its old one.test_peakguard.py10h-1b drives it.Heartbeats: concurrent dispatch plus a per-write budget.
WRITE_BUDGET_S = 4 s, and the inequality is graded rather than asserted —4 + heartbeat_s's schema maximum of25 < 30.send_heartbeats()was extracted fromrun_control()so the shipped dispatch is reachable by a check at all (see the mutation note below).Site aggregates are named for the question they answer, with the reduction argued at the definition:
site_max_w(sum),site_rating_w(sum of the per-unitmin(max_w, |dev|)conjunctions),site_batt_w(sum),pack_floor_soc(minimum — a discharge permission must be graded against the emptiest pack).Evidence
test_inverter.pyis new (82 checks).test_alloc.pypart 8,test_peakguard.py10g/10h,test_watchdog.pyandtest_clamp.pywere re-aimed rather than relaxed — every one of them grades something stronger now than it did.tools/mutation_run.sh; every one observed RED. The table is intest_inverter.pypart 9.last_beatboth passed while part 6 gathered its ownwrite_setpoint()calls and computed its ownduelist — a check on the test file rather than on the add-on. That is this repository's own named failure mode, found once more here;send_heartbeats()exists because of it, and the episode is recorded in the file rather than quietly fixed.admin/goodwe@0c31e6f; nothing there was modified and no hardware was touched.What I am unsure about
arbitrage's singlesocargument. I passpack_floor_soc(the emptiest pack). That is right for the sell side and wrong for the buy side, which wants the fullest pack — one scalar cannot be conservative in both directions. It is named as a residual in the code rather than hidden, and at the shipped roster of one the two reductions are the same number. A per-pack charge ceiling belongs to CFG-04/SPIKE-04; if you would rather I refused arbitrage outright at N≥2 until then, say so and I will.phase,last_completed, …) deliberately did not move into a namespace — re-homing a month-scale history under a unit name is a different decision from namespacing a latch. The claim is now filed on a namedmaint_unitrather than on whichever unit happened to be asked, and the FSM is graded on that pack's own SoC. Two packs recalibrating on two schedules needs a second FSM with its own namespaced history.inputs_okrequires every roster unit's readings. At N≥2 one silent pack stops the whole site at P_SAFETY. I think that is right (§11.1 rule 1, and the alternative hands its share to a sibling while the site keeps regulating against a pack nobody read), but it is a policy choice and it is the one that most changes what a two-inverter site does on a bad day.SAFETY-28toucheswrite_setpoint(~:409-450), which this card split into a site-levelwrite_setpoint(value, *, force, unit=None)and a per-unit_write_unit(). Expect a merge there; whoever lands second mergesrelease/1.0and re-runs.Base:
release/1.0@3030f72.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
A hazard this card created, found by re-reading rather than by a failing check
Second commit. Before DEV-02 the heartbeat re-emitted
self.target, so the stop button zeroing that one scalar was enough. It now re-emitsinv.target— it must, or it would put the site number on every inverter, which is the ARCH-01 broadcast fault arriving once everyheartbeat_son a path with no allocator in it. So a stop that zeroed onlyself.targetleft each unit's last share standing: no cycle has run at that moment, nothing else clears it, and the very next heartbeat puts it back on the wire. The operator presses stop, watches the inverter go to zero, and sees it resume seconds later.command_site_zero()moves both together — one method, because the two lines have to stay together, andshutdown(), the stop button andmaint_abortall mean the same thing.test_inverter.pypart 6 presses stop with 995/995 standing and then runs the shippedsend_heartbeats()before any cycle; the mutation that zeroes the site target alone takes it red (mutation 19).I am reporting this rather than folding it silently into the first commit because it is the shape of defect this repository keeps finding — a refactor that moves what a path re-emits without moving what clears it — and the next person to touch the heartbeat should see that it was reachable.
Review round 2 — both blockers closed, and
release/1.0merged in (head753f700)release/1.0@f594543(SAFETY-28 + the ALLOC-01 fix) is merged. 19/19 files green on the merged tree.Blocker 1 — AC 4's arbiter-bound evidence was vacuous.
second_inverter()defaulted B's rating and device range to A'sLIVE_MAX_W, so the check that claimed "each unit'ssupervisedbound is ITS OWNmax_w" compared(-5000, 5000)against(-5000, 5000). Filing the per-unitdevice/supervisedbounds fromself.inverters[0]— withactuator=left correct — passed all 19 files. Fixed at the helper rather than at one call site: B is 3000 W everywhere in the file, the bounds are asserted against literal numbers forsupervisedanddevice, the resolved envelope is checked as well as the filed claim, and a PRECONDITION check fails if the two ratings are ever made equal again. The BOUND mutation joins the ACTUATOR one in the table and was observed RED (3 checks intest_peakguard.py).Blocker 2 — the SAFETY-02 latch was silently disarmed on the rollback round trip. Migrate → downgrade → trip inside the downgrade window (old code writes only the flat key) → upgrade:
rep.ranis False, the guard was skipped, and the controller came up unlatched while/data/state.jsonsaidsign_tripped: true. The three-step scenario is now a check, preconditions included. Three changes, four mutations each observed RED:store.py's "the staleness fails toward a SPURIOUS latch, never a missing one" was untrue and is corrected;test_watchdog.py's "the flat key is NOT written" assertion is re-aimed (right about the read, wrong about the write).HANDOVER.md/DOCS.mdno longer tell an operator the flat key "is never read again".Merge cost.
write_refusal()takes the inverter it is about — asking it withself.unitwould authorise A's share on B's wire and would trip part 1's AST scan. Every fixture that callswrite_setpoint()directly now arms a real allocation record through the shippedresolve();test_clamp.py's two-unit fixture appends a realInverterrather than assigning the now-derivedc.units.MUTATION.mdgains a third failure mode beside stale bytecode (TEST-15) and masking (TEST-16): a check whose data cannot reach the failure it names, with this week's three instances and the rule — any check asserting "X differs from Y" must be run once with X and Y actually made different.Every piece of per-device state in the Controller was a scalar, and several of them are safety inputs. app/inverter.py introduces an `Inverter` that owns its entities, its rating, its device range, its SignWatchdog, its last_write and its last_beat, under store keys namespaced by unit name. The Controller holds a list of them and keeps only what is genuinely site-level: the P1 reading, the site integrator, and the tariff quarter. WHAT THE SCALARS WERE SILENTLY ASSERTING dev_min/dev_max one pair, read from one setpoint_entity. A second inverter's rating was NEVER READ AT ALL, so its clamp, its freeze rail and the peak guard's achievable_w were the first one's numbers. soc one reading into the maintenance FSM, SAFETY-24's pack floor and arbitrage's bounds. An average lets the emptier pack be driven below its own floor. sign_wd ONE SignWatchdog, whose own docstring says "one instance per actuator", latched under FLAT store keys - so a trip caused by B latched A across restarts and one operator press cleared both. last_beat one deadline and one serialised hass.call(), so a hung write to A ate B's 30 s layer-1 budget. actual_w BECAME SITE LEVEL WHILE THE READINGS BECAME PER-INSTANCE, and those are opposite directions on purpose. compute()'s anti-windup test is `abs(prev_w - actual_w) > saturation_w` with prev_w the SITE target, so feeding it one inverter's power compares a site number against a unit number, trips the saturation freeze permanently and leaves a standing grid error. Measured on admin/goodwe's two-master rig (tools/alloc_sim.py --splits, 2 kW load, 40 cycles), worst |delta| between splits at the meter: gain one unit's power the site sum 0.2 510 W 0 W 0.4 250 W 0 W 0.6 50 W 0 W That was ALLOC-01's disclosed finding; test_alloc.py part 8 kept its stimulus and now grades the other row. THE STORE MIGRATION IS GRADED, NOT MERELY PERFORMED. sign_tripped/sign_reason move to unit/<name>/... once, verified by READING BACK what was written, and the version marker is set only when nothing that carried state was lost. The flat keys stay in the file so a rollback still finds its latch. And __init__ compares the pre-migration flat latch against what unit A actually came up with: if the latch did not survive it is RE-ARMED from the legacy value and shouted about. A latch that silently fails to migrate is a disarmed safety guard on an install whose state file says it is protected - which is strictly worse than never persisting it, because now there is evidence saying it is fine. test_inverter.py part 2 breaks the migration and watches the guard fire. HEARTBEATS ARE INDEPENDENT AND THE BUDGET IS DERIVED. Each inverter owns its deadline, the refreshes go out concurrently, and each write is bounded by WRITE_BUDGET_S = 4 s. The inequality is graded rather than asserted: 4 + heartbeat_s's schema maximum of 25 < 30, validate.FIRMWARE_INGEST_VETO_S. Verified by injecting a transport that accepts and never answers. live_units() ANSWERS HONESTLY. A unit whose every write has been rejected for a whole layer-1 window is not commandable; the allocator drops it, the survivors carry its share, the action log names it, and the dropped unit is commanded 0 W rather than left standing on its last share. A roster entry with NO setpoint_entity is deliberately not handled there - that is a configuration that cannot be delivered, and it reaches main.py's refusal branch, which is how that branch keeps a live trigger after this card removed its old one. AT N=1 NOTHING MOVES. The removed scalars survive as read/write properties of the sole inverter, so every existing replay runs unmodified - and they REFUSE at N>=2 rather than answering for the first inverter, which is the tripwire on a consumer that was never made per-instance. EVIDENCE. 19 files green. test_inverter.py is new (82 checks). Eighteen mutations were applied to the shipped modules on a purged tree via tools/mutation_run.sh and each was observed RED. Two of them SURVIVED first time and the checks were changed rather than the table: the serialised dispatch and the shared last_beat both passed while part 6 gathered its own writes and computed its own `due` list. send_heartbeats() was extracted from run_control() so the shipped dispatch is reachable by a check at all. INVERTER B STILL DOES NOT EXIST. build_roster() returns exactly one inverter from the flat options and is the one function CFG-04 changes; every two-unit figure here is architecture, and the second rating in the checks is marked a hypothesis. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU