Files
goodwe-addon/goodwe_controller/DECISIONS/SAFETY-31-circulating-power.md
T
glenn schrooyenandClaude Opus 5 9099ceb1af
checks / unit (pull_request) Successful in 6m27s
SAFETY-31: the grid is 42, not 90, and the wrong number was in the failure text
Review reproduced every figure in both PRs except one, and rejected on it.

⚠️ "0 of 90 GAINS x SPLITS combinations produce opposing shares" - the SUBSTANCE
is right and review re-measured it, but the DENOMINATOR is invented. sitesim.py
declares GAINS with 7 entries and SPLITS with 6: the grid is 42. 90 is not even
divisible by 7, so no reading of "GAINS x SPLITS" reaches it. It is almost
certainly TEST-20`s RUN count (84 + 6), which is a different quantity that
happened to be nearby.

⚠️ AND THE WORST COPY OF IT WAS INSIDE THE `PRECONDITION FAILED` TEXT, so the
next engineer to see this scenario go red would have been handed an
unreproducible measurement as their diagnosis - the exact defect class this
campaign exists to correct, in the failure path where it does the most harm.

Corrected in all seven places. At the failure-text site the denominator is now
`len(GAINS) * len(SPLITS)`, computed rather than typed, so it cannot drift from
the grid again - the same fix test_translations.py already took.

Also N3: a missing blank line was going to render the S3 and M19 paragraphs as
one in DECISIONS.

Re-run after the change: `inject.py two_master_circulation` PASS in 35 s, norig,
no rig lock taken. Add-on suite 26 files, all pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-07 20:13:01 +02:00

26 KiB
Raw Blame History

SAFETY-31 — what "acted on" means for circulating power, and what it cannot mean

Status: DECIDED 2026-09-04. The trip stops the site: a SET 0 W claim at P_SAFETY on the site actuator, latched, persisted, and re-announced after a restart. This document records why that is an action rather than theatre, given that HW-01 established a wedged inverter cannot be commanded at all — and it records the numbers the threshold and the window were read off, because "establish the window from the rig, do not pick a round number" is an acceptance criterion and a number without its measurement is a guess.

Filed with SAFETY-31. Card 6a99e0022b6a6c8ca15dbe3e.


0. The two halves, and which one this card built

Half of this card was already shipped, and it is not the half below.

ALLOC-01, AC 7 — the COMMAND half SAFETY-31 — the MEASUREMENT half
what it is allocator.REFUSE_SIGN circulation.circulating_w() + CirculationWatchdog
what it sees the shares the allocator is about to hand out each inverter's own measured power (Inverter.batt, register 0x0518)
what it does refuses the allocation, names both units, returns no shares stops the site, latches, alarms durably
what it covers circulating power the allocator would have commanded circulating power nobody commanded
graded by test_alloc.py part 6 test_watchdog.py, the SAFETY-31 section

They are complements, not duplicates. The gate refuses an allocation before it is written and cannot see one millisecond past that. This card reads the hardware and does not care what was written. Neither replaces the other, and a sensor built on the allocation record would collapse the second into the first and detect nothing — see §2.

1. What "acted on" means

The trip files Claim.set("circulation", P_SAFETY, 0.0) on the SITE actuator. Every inverter is commanded 0 W and held there until an operator starts control again.

The objection, which is correct

HW-01 is on the record: on 2026-09-02 the comms subsystem failed and the power stage went on executing its last instruction for 6 h 25 m, through every command this add-on sent it. If the cause of the circulation is a unit that has stopped listening, a command is not a remedy, and shipping one as though it were is theatre.

The answer

A circulating loop needs two participants. In every cause on this card's list — a unit ignoring its share, a unit lagging it, a stale share still being executed, a maintenance or SAFETY-25 pin holding one unit while the loop drives the other, or HW-01's wedged power stage — exactly one participant is the unit that stopped listening, and the other one is this controller, faithfully commanding the sibling to a share computed to sum with a number the wedged unit is not actually producing.

Stopping the commanding half does two things a notification cannot:

  1. It halves the loop. The unit that is still listening stops being driven into the one that is not. That is real energy and real cycles off a real pack, and it needs no cooperation from the wedged unit to take effect.
  2. It makes the residue visible — and this is the more important half. Circulating power hides because it nets to zero at the meter. With this controller's contribution removed, whatever the wedged unit is still doing stops being cancelled and appears at P1 as a plain net flow, where SAFETY-02's sign watchdog, the peak guard and the capacity tariff can all see it. The fault stops being invisible.

Measured on the TEST-19 rig, and this is the number that makes the case: in the card's own scenario — A at −3000 W, B at +3000 W, house load 0 W — the meter reads +0 W and the shipped SignWatchdog's predicate is false on every one of 30 cycles, because it gates on abs(grid_w) > 300 and the gate never opens.

What it does not do, said plainly

It does not stop the wedged inverter. Nothing this add-on can send will. If the loop persists at the meter after this trip, the remaining actuator is a human at the isolator, and the notification says so in those words.

Why the claim is SITE-scoped

arbiter.CLAIM_SCOPE classifies circulation as SCOPE_SITE, beside loop, peak, arbitrage and safety. The neighbouring failsafe entry looks like it and is not: SAFETY-02 has site-wide evidence and a per-unit conclusion, because it compares the meter against the command this controller put on one actuator.

Circulating power has no such conclusion available. It is Σ|Bᵢ| − |ΣBᵢ| — defined over the whole roster, identically zero for any one unit — and there is no fact of the matter about which of two opposing inverters is the offender: a unit ignoring its share and a sibling faithfully executing a share computed to sum with it produce identical readings. Filing this per-unit would require inventing an attribution the measurement cannot support, and would park one inverter while the other went on feeding the loop — quite possibly parking the only one still listening.

The same argument is why the latch is flat in the store (circ_tripped, circ_reason) and not namespaced like SAFETY-02's. One site, one latch.

2. Why the sensor is MEASURED, and why the cheap version is worthless

The obvious cheap version computes the same formula over Allocation.shares. It would be simpler, would never be None, and would be worth nothing, for two independent reasons:

  1. It cannot fire. ALLOC-01's sign-coherence gate already refuses any allocation whose shares carry two signs. A record-derived sensor is therefore a sensor over a set that is guaranteed sign-coherent. It reads zero by construction, forever.
  2. It is MUTATION.md's third failure mode. It could only ever confirm that the allocator agrees with itself — a check whose data cannot reach the failure it names.

Verified rather than argued. On the rig, with inverter B wedged on a stale discharge while PV drives the site to charge:

  S = -2500 W   shares  a = -1250 W   b = -1250 W     <- record: ok, ONE sign, sums to S
  MEASURED      A = -1250 W   B = +1000 W             <- hardware
  sensor over the RECORD      :     0 W
  sensor over MEASURED power  :  2000 W      -> trips at 190 s

And the card's own headline scenario, driven on the same rig — A at −3000 W, B at +3000 W, house load 0 W:

  METER (P1)                  :    +0 W      <- the site looks IDLE
  sensor over MEASURED power  :  6000 W      -> trips at 100 s
  SAFETY-02 SignWatchdog      :  predicate false on 30 of 30 cycles

The same pair is asserted in test_watchdog.py with the precondition written as its own check and printed, per MUTATION.md's rule, and mutation M10 (the property rewritten to read self.allocation.shares) takes it red.

3. The numbers, and where they come from

Both were read off a curve produced on admin/goodwe's sim/ two-master rig, used read-only: the add-on's own compute(), its own allocator.plan() and its own circulating_w(), driving plantsim.Plant through the firmware's own encoder, with the sensor input taken from each unit's holding register 0x0518.

924 closed-loop runs in which nothing is faulty — every unit executes the share it was given — over both families of stimulus that can produce legitimate opposition (7 reversals × 5 write staggers, and 7 flap periods × 4 amplitudes), each swept across the whole range config.yaml accepts for the two settings that decide how fast the site can turn around: gain float(0.05,1.0) and slew_w int(50,5000).

⚠️ THE HARNESS BEHIND EVERY FIGURE ABOVE IS NOT COMMITTED, AND UNTIL 2026-09-07 THAT MADE THEM UNCHECKABLE

Read this before quoting any number in this section. The paragraph below said the campaign harness "is not committed here" and treated that as a packaging detail. It is not: a figure produced by a script that exists in nobody's tree cannot be re-run, cannot be reviewed, and cannot be defended when it is contradicted. It has now been contradicted twice.

sweep worst peak worst continuous trips
the 924-run campaign below — harness uncommitted 1700 W 505.0 s at 100 W; 5.0 s above 700 W 0 / 924
independent reviewer sweep, 2026-09-07, 48 honest reversals 2750 W 15.0 s above 700 W 0 / 48
admin/goodwe two_master_circulation, 840 runs — committed 1048 W 10.0 s at 100 W; 5.0 s above 700 W 0 / 840

This is not proof that anything was fabricated, and it should not be reported as one. The three swept different grids. What the third one adds is a measurement of why the spread is that wide — on one grid, changing nothing but the model:

plant / stagger worst legitimate peak
modelled plant (lag 10 s, ramp 800 W/s), staggers up to a whole cycle 1048 W
INSTANT (lag 0, unbounded ramp), staggers up to a whole cycle 2250 W
INSTANT, staggers strictly under a whole cycle 0 W

A factor of 2.1×, and a factor of infinity, between three defensible descriptions of the same rig. The plant model and the write-stagger set are terms of the number, and a worst case quoted without them is not a measurement. That is enough to explain the whole spread above without anybody having invented anything — and it is exactly what cannot be established when the instrument is missing.

What survives, and what does not.

  • The window argument survives, and is now committed. The claim that decides circulation_hold_s: 75 is "the worst continuous legitimate episode above 700 W is about one control cycle". All three sweeps agree on the shape of that (5.0 s, 15.0 s, 5.0 s — one to three cycles against a fifteen-cycle window), and all three trip zero times. two_master_circulation re-runs it on every pull request in admin/goodwe.
  • The threshold curve does NOT survive as stated. 505.0 s at 100 W is the number the "knee at 400 W" argument for circulation_w: 700 rests on, and the committed sweep measures 10.0 s at 100 W on its grid — 50× smaller. The curve still falls with the threshold (126 of 840 runs cross 100 W against 1 that crosses 700 W), so the knee is real in direction; its depth is not established by anything anybody can run. Do not quote 505.0 s.
  • Nothing about the guard is compromised. 0 trips in 924 + 48 + 840 non-faulty runs, from three independent grids.

What would settle it: the original harness, committed. Until then the committed sweep is the only figure in this section anybody can check, and it is labelled as its own grid's rather than as a reproduction.

Reproducing it

⚠️ THE COMMITTED VERSION IS admin/goodwe's sim/inject.py two_master_circulation (sim/sitesim.py:circulation_findings, runnable on its own as python3 sim/sitesim.py --circulation, norig, ~36 s, graded by checks.yml on every pull request there). It produces the fault with plantsim.Inverter.wedge() — a unit answering its bus and no longer steering its power stage, which is HW-01 — and sweeps the legitimate family with the serialised write path staggered up to a whole control cycle. The paragraph below describes the ORIGINAL campaign, which is the one that is not committed.

The rig lives in admin/goodwe (sim/) and was used read-only; the harness is not committed here because it imports that repo's plantsim/frame_check, which are not present in this one or in CI. What it does is small enough to restate: build Site([Inverter.from_spec("A", INVERTER_A), Inverter.from_spec("B", INVERTER_B)]), close the loop with the add-on's own compute() and allocator.plan(), put each share on its unit's wire through Inverter.command() (which goes out through the firmware's real encoder), and read the sensor from inv.reg(0x0518) — the same register the live add-on's batt_entity binds, so the input is the one the shipped code gets. Point it at a checkout with GOODWE_ADDON_REPO. ⚠️ MUTATION.md's note applies: those harnesses write __pycache__ into both trees, so purge both by hand before grading a mutation through one.

Every figure below was re-run against the final shipped CirculationConfig (min_w=700, hold_s=75) rather than against the draft values they were first measured at.

Worst CONTINUOUS legitimate opposition, by threshold

threshold worst continuous runs that ever cross it
100 W 505.0 s 281 / 924
200 W 445.0 s 174 / 924
250 W 40.0 s 132 / 924
300 W 10.0 s 115 / 924
400 W 5.0 s 74 / 924
700 W 5.0 s 33 / 924
1000 W 5.0 s 16 / 924
1700 W 0.0 s 0 / 924

**With the shipped min_w=700 W and hold_s=75 s, 0 of those 924 runs trip.

The knee is at 400 W and it is sharp.** Below ~250 W a flapping load holds the sensor above the threshold for minutes at a time, and no window shorter than the 900 s billing quarter could tell that from a fault. At 400 W and above the worst legitimate episode collapses to 5.0 s — one control cycle — and stays there. The worst peak on any legitimate run is 1700 W.

That 5.0 s is the serialised write path doing exactly what it should: on a reversing cycle inverter B's new number lands up to a whole control cycle after A's, both then slew through zero at 800 W/s behind a 10 s lag, and between the two zero crossings they genuinely oppose. Nothing is wrong. A guard that tripped on it would be the fourth false-alarm generator of the week, and this project's doctrine on crying wolf is explicit (SAFETY-15, SAFETY-19).

circulation_w: 700 — schema floor 400

700 is 1.75× the measured knee. The headroom is bought for one named reason: inverter B does not physically exist. Its ramp and lag in the rig are inverter A's, copied, carrying measured=False; SPIKE-04 measures the real ones. A slower or faster second unit moves the transient's shape, and the margin above the knee is what absorbs that without re-opening this card. Revisit when SPIKE-04 lands, not before.

The schema floor is the knee itself, 400 W, so a legally configured install cannot land in the 445–505 s regime — ⚠️ and those two figures are from the uncommitted campaign; the committed sweep measures 10.0 s at 100 W on its own grid. See the disclosure at the head of this section. The floor is still the right shape (fewer runs cross a higher threshold, measured 126/840 at 100 W against 1/840 at 700 W); its DEPTH is not established by anything re-runnable. What the threshold costs: a circulating loop smaller than 350 W per unit is not reported, which is deliberate — at that scale the round-trip loss is a few watts.

circulation_hold_s: 75 — schema range 30..900

Three bounds, and the value clears all three:

  • lower, from the rig — 5.0 s. The worst continuous legitimate opposition above 700 W across all 924 runs.
  • lower, from the firmware — 30 s (inverter.LINK_STALE_S). A unit still executing a stale share for less than the firmware's own veto window is a link fault, and the add-on already answers it: liveness drops the unit and the survivors carry its share, with the record saying so. Firing inside that window would raise a second, louder alarm about a state already being handled.
  • upper, from the bill — 900 s, the quarter that sets the capacity tariff.

75 s is fifteen control cycles: 15× the worst legitimate episode, 2.5× the firmware veto, and 12× inside the quarter. It is deliberately not 60 or 120 — a round number here would be a guess wearing a measurement's clothes, and a check in test_watchdog.py asserts it is none of the values a guess lands on.

4. Why the window cannot be defeated the way SAFETY-02's was

allocator.py records the measurement that makes SAFETY-02 blind to the broadcast fight: the sign disagreement flips on alternate ticks, so disagree_since is cleared between every pair of them and the 180 s hold never accumulates.

Circulating power cannot do that. It is a magnitude, non-negative by construction, and a sustained loop holds it high on every cycle rather than every other one. An oscillation that genuinely alternated between opposing and agreeing is not a circulating loop and should not be reported as one — which is exactly the flapping-load case in §3, and it is why the window clears.

5. SAFETY-34: the latch is not the alarm

persistent_notification is in memory. Home Assistant restarts and the card is gone while the fault runs on — which is precisely how the 2026-09-02 outage went unseen for 6 h 25 m with the detector correct throughout.

So the durable half and the visible half are joined, and the join is Controller._circ_announced: a per-process flag, deliberately not persisted. The latch in /data/state.json says "this site is tripped"; the flag says "this process has put that in front of a human". A restart clears the flag and keeps the latch, so the notification is re-raised on the first cycle after every start — which is the first cycle after a Home Assistant restart, because the add-on restarts with it.

One code path serves both the fresh trip and the restart, on purpose: two paths would be two chances for the restart one to rot unnoticed, and the restart one is the case nobody exercises by hand. Mutations M13 (flag initialised from the latch) and M18 (the call removed) both take it red.

6. Mutation ledger

Every run on a freshly copied tree with no __pycache__, mounted :ro into the CI image — the structural immunity MUTATION.md records for CI and for SAFETY-25 — so no stale .pyc could exist to be read. Byte deltas recorded; three are +0 B, the risky delta, and are marked.

# file mutation Δ result
M1 circulation.py sum(abs(v)) → sum(v) in the sensor −5 B RED — the 6000 W, 2000 W and three-unit checks
M2 circulation.py N≤1 returns None instead of 0.0 +1 B RED — AC 5's nine-reading sweep
M3 circulation.py a missing reading at N≥2 returns 0.0 −1 B RED — "health from an absence"
M4 circulation.py drop max(0.0, ...) — EQUIVALENT, not run as a claim. For a same-sign roster the two sums accumulate the same magnitudes in the same order and are bit-identical; 700 000 random rosters produced no negative. Argued at the line and in the check's own text.
M5 circulation.py threshold <= → < −1 B RED — "exactly 700 W forever does not trip"
M6 circulation.py window < → <= +1 B RED — "trips at 75 s, not before"
M7 circulation.py the window removed entirely −85 B RED — both slew-transient checks
M8 circulation.py None no longer clears the window +34 B RED — the stale-reading gap check
M9 circulation.py the latch removed −123 B RED — "restored as tripped stays tripped"
M10 main.py the sensor derived from self.allocation.shares +60 B RED — AC 2's headline check. This is the card.
M11 main.py the circulation claim never filed −235 B RED — test_watchdog.py and test_peakguard.py 10f/10g
M12 main.py the latch not persisted to the store −112 B RED — all three durability checks
M13 main.py _circ_announced initialised from the latch +21 B RED — SAFETY-34's re-raise
M14 main.py the operator clear leaves _circ_announced set −45 B RED — "the next trip would raise nothing"
M15 arbiter.py circulation scoped SCOPE_UNIT +0 B RED — test_arbiter.py ×3 and test_peakguard.py ×3
M16 config.yaml schema floor 400 → 100 +0 B RED — the measured-knee check
M17 circulation.py default 700 → 500 +0 B RED — the CFG-03 pin and the not-a-round-number check
M18 main.py await self.check_circulating() removed −39 B RED — all four notification checks
M19 main.py publish round(x or 0.0) instead of None −68 B survivor on the first battery — GAP CLOSED, then RED
M20 main.py pos, neg = opposing(...) → ([], []) — the attribution clause deleted from the alarm −102 B SURVIVED ALL 26 FILES on the shipped battery — GAP CLOSED, then RED on §6a
M21 main.py pos and neg exchanged — the alarm names both packs, both the wrong way round +0 B RED — §6a ×3 (the mirrored run, and "only its own sentence")
M22 main.py opposing(..., min_w=cfg.min_w / 2.0) → min_w=0.0 −25 B RED — §6a's resting-pack check

And four in admin/goodwe, graded through python3 sim/sitesim.py --circulation rather than through the injector — deliberately, because inject.py is the file that takes the rig lock, and "run the injector to grade a mutation" is not a sentence anybody should have to type. Byte deltas over plantsim.py + sitesim.py together.

# file mutation Δ result
S1 plantsim.py wedge() no longer stops the latch reaching the power stage −20 B RED — the opposing-signs precondition and "the watchdog did not trip"
S2 sitesim.py the sensor computed over the allocation record instead of register 0x0518 +0 B RED ×3 — this is M10 restated on the rig
S3 sitesim.py the campaign run on plant=INSTANT −51 B RED — the plant is a term of the figures; it was a SURVIVOR until that check was added, and finding out why produced §3's 1048/2250/0 W table
S4 sitesim.py the write stagger removed from the legitimate campaign −19 B RED — "the legitimate campaign produced no opposition at all"

S3 is recorded as a survivor because it was one, and it is the most useful result in this table. The scenario first shipped with a vacuity check whose text claimed plant=INSTANT would make the sweep measure 0 W. S3 was run and passed: INSTANT with a stagger of exactly one control cycle samples the plant before the second unit has stepped, so it produces opposition anyway — and a larger worst peak (2250 W) than the modelled plant does (1048 W). The claim in the check's own text was wrong. It was corrected against the measurement, the plant was pinned as an explicit term, and S3 is red now for a stated reason rather than by accident. §3's spread table is that measurement.

M19 is recorded as a survivor because it was one. The first battery found no check grading what publish() does when a per-unit reading is missing, so turning "I do not know" into "0 W" on the one surface an operator actually watches went unnoticed — the sensor's own None rule, defeated one layer out. A check was added and the mutation re-run; it is red now. The run that found it was purged, so it is a result and not an artefact.

M20 IS RECORDED AS A SURVIVOR BECAUSE IT WAS ONE, AND IT IS THE MORE UNCOMFORTABLE OF THE TWO. The commit message for this card claimed "18 mutations, all RED", and M1–M19 above have no entry for the notification's attribution clause. Verification on 2026-09-07 ran it and it survived the whole 26-file suite: the alarm still fires, still stops the site, still says the meter cannot see it — and no longer says which two packs or which way round. circulation.py's own docstring calls that sentence the first thing somebody woken by this alarm needs and "impossible to reconstruct from the number alone".

It survived for one reason, and it is not reachability: M18 (the whole call removed) is red, so check_circulating() is exercised. The three checks on that notification asserted METER CANNOT SEE, CONTROL HAS BEEN STOPPED and isolator — and never an inverter name. Nothing graded the attribution.

test_watchdog.py §6a closes it, and closes it on a fixture that can tell the three failures apart. It runs a roster of ["garage", "shed", "cellar"] — operator names, none of them UNIT_A, for test_maintenance.py §11's reason: on a fixture whose names are the module's own constants a check cannot tell the roster from the constant. It runs the site twice, mirrored, so the sentence is not symmetric and a swap (M21, +0 B) is a different red from a deletion. And it carries a third pack resting at 5 W, so the floor that stops a pack in standby being named as a participant in a 6 kW loop (M22) is graded too.

7. What this card does NOT close

  • Inverter B's ramp and lag are placeholders. Every transient number in §3 was produced with inverter A's parameters copied onto B (measured=False). SPIKE-04 measures the real ones and is the revisit trigger for the 1.75× margin. Do not present any figure here as a measurement of inverter B.
  • A wedged inverter still cannot be stopped by this add-on. §1 says so; the notification says so; there is no software fix and this card does not pretend to one. It can now be produced on demand: plantsim.Inverter.wedge().
  • The depth of the threshold curve is not established by anything committed. §3's disclosure. The window is; the knee's magnitude is not.
  • A loop below ~350 W per unit is not reported. Deliberate, §3.
  • The saturation freeze fires under a wedged unit and leaves a standing grid error. Observed on the rig while producing §2's numbers: with B wedged, the loop's anti-windup test compares a site target against a measured site power that cannot track it, freezes the integrator and leaves the meter parked at −3750 W. Nothing alarms on that today — a freeze is an ordinary control action by design. It is a real finding, it is not in scope here, and it wants a card of its own.