From 7ef3d23245bd60f0a4db6de1d46f58f021d7e701 Mon Sep 17 00:00:00 2001 From: Claude Opus 5 Date: Fri, 4 Sep 2026 01:56:23 +0200 Subject: [PATCH] SAFETY-31: measure circulating power, which nets to zero at the meter Inverter A charging at 3 kW while inverter B discharges at 3 kW nets ~0 W at P1. The site reads idle while both packs are cycled through two round-trip conversion losses continuously, for no benefit, and the wear is real. NOTHING EXISTING CAN SEE IT, AND THAT IS STRUCTURAL. Every guard in this add-on is a consumer of the NET figure at the grid connection: SAFETY-02's sign watchdog gates on abs(grid_w) > 300 and the gate never opens at a netted zero; the peak guard and the capacity tariff both bill grid exchange. Circulating power is precisely the component of per-inverter power that the sum destroys, so no threshold on any existing detector could reach it. WHAT ALLOC-01 ALREADY COVERED, AND IS NOT RE-IMPLEMENTED HERE. Its AC 7 shipped the sign-coherence gate (allocator.REFUSE_SIGN, graded by test_alloc.py part 6): an allocation whose shares carry two signs is REFUSED, both units named, no shares returned. That is the COMMAND half - the allocator can no longer command circulating power. This commit is the MEASUREMENT half: circulation the allocator NEVER COMMANDED. app/circulation.py the sensor and the trip main.py site_circulating_w, the evaluation, the claim, the alarm arbiter.py `circulation` classified SCOPE_SITE, with the argument store.py a flat, site-level durable latch mqtt.py sensor.goodwe_circulating_power config.yaml two options, two schema ranges, CFG-03 discipline THE SENSOR: sum(|Bi|) - |sum(Bi)|. Zero when the units pull the same way, 2*min(|Ba|,|Bb|) when they oppose. DERIVED FROM MEASURED PER-INVERTER POWER - Inverter.batt, register 0x0518 - and never from the allocation record. The cheap version would be simpler and worth nothing: ALLOC-01's gate already guarantees the record is sign-coherent, so a record-derived sensor reads zero BY CONSTRUCTION and can only confirm that the allocator agrees with itself, which is the third failure mode MUTATION.md names. What this catches is a unit ignoring or lagging its share, a stale share still being executed, a maintenance or SAFETY-25 pin holding one unit while the loop drives the other, or HW-01's power stage running its last instruction for 6 h 25 m with the comms dead and the allocation record impeccable throughout. batt became per-instance in DEV-02; before that this could not have been written. VERIFIED BY MAKING ONE UNIT IGNORE ITS SHARE, on the TEST-19 two-master rig: S = -2500 W shares a = -1250 W b = -1250 W <- record ok, ONE sign, sums MEASURED A = -1250 W B = +1000 W <- hardware sensor over the RECORD : 0 W sensor over MEASURED power : 2000 W -> trips at 190 s and the card's own scenario, A -3000 W against B +3000 W at zero house load: the meter reads +0 W, the sensor reads 6000 W, and the shipped SignWatchdog's predicate is FALSE on 30 of 30 cycles. WHAT "ACTED ON" MEANS - decided, and recorded in DECISIONS/. The trip files Claim.set("circulation", P_SAFETY, 0.0) on the SITE actuator: every inverter to 0 W, latched. HW-01 established that commanding a wedged inverter achieves nothing, so the remedy is not aimed at it. A circulating loop needs TWO participants and one of them is THIS CONTROLLER, faithfully driving the listening sibling into the one that stopped listening. Stopping (1) halves the loop, which needs no cooperation from the wedged unit, and (2) - the more important half - makes the residue VISIBLE at the meter instead of cancelled at it, where every ordinary guard can finally see it. What it cannot do is stop the wedged unit; the notification says so and names the isolator as the last actuator. SITE-scoped because the quantity is defined over the roster and is identically zero for any one unit - there is no fact of the matter about which of two opposing inverters is the offender, so filing it per-unit would invent an attribution and could park the only unit still listening. THE WINDOW IS FROM THE RIG, NOT A ROUND NUMBER (AC 3). 924 closed-loop runs in which NOTHING is faulty, over both families of legitimate opposition (7 reversals x 5 write staggers; 7 flap periods x 4 amplitudes), swept across the whole range the schema accepts for gain and slew_w. Worst CONTINUOUS legitimate opposition by threshold: 505.0 s at 100 W, 40.0 s at 250 W, 10.0 s at 300 W, 5.0 s at 400 W and above. The knee is 400 W and it is the schema floor - below it, ordinary operation holds the sensor over the line for minutes and no window inside the 900 s billing quarter could tell that from a fault. circulation_w 700 is 1.75x the knee, the headroom bought for inverter B's ramp and lag being PLACEHOLDERS until SPIKE-04 measures them. circulation_hold_s 75 clears three bounds: 15x the worst legitimate episode, 2.5x the firmware's own LINK_STALE_S veto, 12x inside the quarter. A momentary opposition while one unit slews through zero is LEGITIMATE - writes are serialised, so B's number lands up to a cycle after A's - and 0 of the 924 runs trip. THE ALARM SURVIVES AN HA RESTART (SAFETY-34). persistent_notification is in memory, which is exactly how the 2026-09-02 outage went unseen for 6 h 25 m with the detector correct throughout. The latch is persisted to /data as flat site-level keys and the notification is RE-RAISED on the first cycle after every start, through the same code path as a fresh trip - two paths would be two chances for the restart one to rot unnoticed. AT N=1 NOTHING CHANGES (AC 5). |B| - |B| is 0 for every reading including one nobody took, no claim is ever filed, the arbitration is byte-identical, and the status page says n/a rather than claiming to watch something that cannot happen. VERIFIED BY BREAKING IT (AC 4). 18 mutations, all RED on named checks, each on a freshly copied tree with no __pycache__ mounted :ro. M10 - the sensor rewritten to read self.allocation.shares - is the headline: it takes AC 2 red. Three mutations are +0 B, the risky delta, and are marked as such. One equivalent mutant (the max(0.0,...) guard) is ARGUED rather than claimed: searched over 700 000 rosters without a negative. One genuine SURVIVOR was found and closed - publish() turning an unknown into a published 0 W, the sensor's own None rule defeated one layer out. Full ledger in DECISIONS/. arbiter.CLAIM_SCOPE gains its ninth source, so test_peakguard.py's 10f and 10g tripwires and test_arbiter.py's scope table went red on the line the append was written. Updated deliberately, with the argument, as their own comments require. Full 19-file suite green. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU --- goodwe_controller/CHANGELOG.md | 72 +++ .../DECISIONS/SAFETY-31-circulating-power.md | 316 ++++++++++ goodwe_controller/DOCS.md | 80 ++- goodwe_controller/app/arbiter.py | 19 + goodwe_controller/app/circulation.py | 339 +++++++++++ goodwe_controller/app/main.py | 298 ++++++++- goodwe_controller/app/mqtt.py | 13 + goodwe_controller/app/store.py | 19 + goodwe_controller/config.yaml | 41 ++ goodwe_controller/test_arbiter.py | 11 +- goodwe_controller/test_peakguard.py | 70 ++- goodwe_controller/test_watchdog.py | 568 ++++++++++++++++++ 12 files changed, 1813 insertions(+), 33 deletions(-) create mode 100644 goodwe_controller/DECISIONS/SAFETY-31-circulating-power.md create mode 100644 goodwe_controller/app/circulation.py diff --git a/goodwe_controller/CHANGELOG.md b/goodwe_controller/CHANGELOG.md index 98f5c1a..c1fb4c0 100644 --- a/goodwe_controller/CHANGELOG.md +++ b/goodwe_controller/CHANGELOG.md @@ -2,6 +2,78 @@ ## Unreleased +**Circulating power - two inverters pushing against each other - becomes a +measured quantity, a bounded one, and an alarm that survives a restart +(SAFETY-31).** Inverter A charging at 3 kW while inverter B discharges at 3 kW +nets ~0 W at P1: the site reads idle while the packs are cycled through two +round-trip conversion losses continuously. **Every guard in this add-on is a +consumer of that net figure**, so none of them can see it - SAFETY-02's sign +watchdog gates on `abs(grid_w) > 300` and the gate never opens, and both the +peak guard and the capacity tariff bill grid exchange. + +* **A site sensor, `Σ|Bᵢ| − |ΣBᵢ|`.** Zero when the units pull the same way, + `2 × min(|B_a|,|B_b|)` when they oppose. Published as + `sensor.goodwe_circulating_power` beside the grid figure, because *"grid 0 W, + circulating 6000 W"* is the whole phenomenon in two dashboard rows and no + arithmetic on the grid figure alone can produce the second one. +* **⚠️ Derived from MEASURED per-inverter power, never from the allocation + record, and that is the whole card.** ALLOC-01's sign-coherence gate + (`REFUSE_SIGN`, its AC 7) already refuses any allocation whose shares carry + two signs, so a record-derived sensor reads zero **by construction** and could + only ever confirm that the allocator agrees with itself. What this detects is + circulation **nobody commanded**: a unit ignoring or lagging its share, a + stale share still being executed, a maintenance or SAFETY-25 pin holding one + unit while the loop drives the other, or HW-01's power stage running its last + instruction for **6 h 25 m** with the comms dead and the allocation record + impeccable throughout. Verified by making one unit ignore its share on the + TEST-19 rig: the record stays `ok` with both shares at −1250 W - a + record-derived sensor reads **0 W** - while the shipped sensor reads + **2000 W**. `Inverter.batt` became per-instance in DEV-02; before that this + sensor could not have been written. +* **The trip stops the site, and the decision record says why that is an action + rather than theatre.** Sustained opposition files `Claim.set("circulation", + P_SAFETY, 0.0)` on the **site** actuator - every inverter to 0 W, latched. + HW-01 established that a wedged inverter cannot be commanded at all, so the + remedy is not aimed at it: a circulating loop needs **two** participants and + **one of them is this controller**, faithfully driving the listening sibling + into the one that stopped listening. Stopping halves the loop, and - the more + important half - **makes the residue visible at the meter**, where every + existing guard can finally see it. What it cannot do is stop the wedged unit; + the notification says so, and names the isolator as the last actuator. +* **The threshold and the window were measured, not chosen (AC 3).** 924 + closed-loop runs on the TEST-19 two-master rig in which *nothing* is faulty, + swept over the whole range the schema accepts for `gain` and `slew_w`. Worst + **continuous** legitimate opposition by threshold: **505.0 s at 100 W, 40.0 s + at 250 W, 10.0 s at 300 W, 5.0 s at 400 W and above**. The knee is 400 W and + it is the schema floor. `circulation_w: 700` is 1.75× it, the headroom bought + for inverter B's ramp and lag being **placeholders until SPIKE-04 measures + them**. `circulation_hold_s: 75` clears three bounds - 15× the worst + legitimate episode (5.0 s), 2.5× the firmware's own `LINK_STALE_S` veto, and + 12× inside the 900 s capacity-tariff quarter. **A momentary opposition while + one inverter slews through zero is legitimate** - the serialised write path + lands B's number up to a control cycle after A's - and the guard does not fire + on it, in any of the 924 runs. +* **The alarm survives a Home Assistant restart (SAFETY-34).** + `persistent_notification` is in memory, which is exactly how the 2026-09-02 + outage went unseen for 6 h 25 m with the detector correct throughout. The + latch is persisted to `/data` as flat, site-level keys - flat because the + quantity is defined over the roster and is identically zero for any one unit, + so there is no inverter to attribute it to - and the notification is + **re-raised on the first cycle after every start**, through the same code path + as a fresh trip. +* **At one inverter the sensor is identically zero and nothing changes (AC 5).** + `|B| − |B| == 0` for every reading, including one nobody took; no claim is + ever filed, the arbitration is byte-identical, and the status page says *n/a* + rather than claiming to watch something that cannot happen. +* **Verified by breaking it.** 18 mutations against the sensor, the window, the + latch, the claim, the store, the alarm, the routing table and both config + copies - **all red**, each on a named check; one equivalent mutant argued + rather than claimed; and one genuine **survivor found and closed** (`publish()` + turning an unknown into a published `0 W`). `arbiter.CLAIM_SCOPE` gains its + ninth source and `test_peakguard.py`'s 10f/10g tripwires were updated + deliberately, as their own comments require. Full ledger in + `DECISIONS/SAFETY-31-circulating-power.md`. + **Every per-device value stops being a Controller scalar: rating, SoC, battery power, watchdogs, heartbeat, write bookkeeping and store keys are now per-instance (DEV-02).** `app/inverter.py` introduces an `Inverter` object that diff --git a/goodwe_controller/DECISIONS/SAFETY-31-circulating-power.md b/goodwe_controller/DECISIONS/SAFETY-31-circulating-power.md new file mode 100644 index 0000000..01dec93 --- /dev/null +++ b/goodwe_controller/DECISIONS/SAFETY-31-circulating-power.md @@ -0,0 +1,316 @@ +# SAFETY-31 — what "acted on" means for circulating power, and what it cannot mean + +**Status: DECIDED 2026-09-04.** The trip **stops the site**: a `SET 0 W` claim +at `P_SAFETY` on the **site** actuator, latched, persisted, and re-announced +after a restart. This document records why that is an action rather than +theatre, given that [[HW-01]] established a wedged inverter cannot be commanded +at all — and it records the numbers the threshold and the window were read off, +because "establish the window from the rig, do not pick a round number" is an +acceptance criterion and a number without its measurement is a guess. + +Filed with SAFETY-31. Card `6a99e0022b6a6c8ca15dbe3e`. + +--- + +## 0. The two halves, and which one this card built + +Half of this card was already shipped, and it is not the half below. + +| | **ALLOC-01, AC 7 — the COMMAND half** | **SAFETY-31 — the MEASUREMENT half** | +|---|---|---| +| what it is | `allocator.REFUSE_SIGN` | `circulation.circulating_w()` + `CirculationWatchdog` | +| what it sees | the shares the allocator is about to hand out | each inverter's **own measured power** (`Inverter.batt`, register 0x0518) | +| what it does | **refuses** the allocation, names both units, returns no shares | **stops the site**, latches, alarms durably | +| what it covers | circulating power the allocator would have **commanded** | circulating power **nobody commanded** | +| graded by | `test_alloc.py` part 6 | `test_watchdog.py`, the SAFETY-31 section | + +**They are complements, not duplicates.** The gate refuses an allocation before +it is written and cannot see one millisecond past that. This card reads the +hardware and does not care what was written. Neither replaces the other, and +**a sensor built on the allocation record would collapse the second into the +first and detect nothing** — see §2. + +## 1. What "acted on" means + +> **The trip files `Claim.set("circulation", P_SAFETY, 0.0)` on the SITE +> actuator. Every inverter is commanded 0 W and held there until an operator +> starts control again.** + +### The objection, which is correct + +[[HW-01]] is on the record: on 2026-09-02 the comms subsystem failed and the +power stage went on executing its last instruction for **6 h 25 m**, through +every command this add-on sent it. If the cause of the circulation is a unit +that has stopped listening, **a command is not a remedy**, and shipping one as +though it were is theatre. + +### The answer + +**A circulating loop needs two participants.** In every cause on this card's +list — a unit ignoring its share, a unit lagging it, a stale share still being +executed, a maintenance or [[SAFETY-25]] pin holding one unit while the loop +drives the other, or HW-01's wedged power stage — **exactly one participant is +the unit that stopped listening, and the other one is this controller**, +faithfully commanding the sibling to a share computed to sum with a number the +wedged unit is not actually producing. + +Stopping the commanding half does two things a notification cannot: + +1. **It halves the loop.** The unit that *is* still listening stops being driven + into the one that is not. That is real energy and real cycles off a real + pack, and it needs no cooperation from the wedged unit to take effect. +2. **It makes the residue visible — and this is the more important half.** + Circulating power hides *because it nets to zero at the meter*. With this + controller's contribution removed, whatever the wedged unit is still doing + stops being cancelled and appears at P1 as a plain net flow, where + SAFETY-02's sign watchdog, the peak guard and the capacity tariff can all see + it. **The fault stops being invisible.** + +Measured on the [[TEST-19]] rig, and this is the number that makes the case: +in the card's own scenario — A at −3000 W, B at +3000 W, house load 0 W — the +meter reads **+0 W** and the shipped `SignWatchdog`'s predicate is **false on +every one of 30 cycles**, because it gates on `abs(grid_w) > 300` and the gate +never opens. + +### What it does not do, said plainly + +**It does not stop the wedged inverter. Nothing this add-on can send will.** If +the loop persists at the meter after this trip, the remaining actuator is a +human at the isolator, and the notification says so in those words. + +### Why the claim is SITE-scoped + +`arbiter.CLAIM_SCOPE` classifies `circulation` as `SCOPE_SITE`, beside `loop`, +`peak`, `arbitrage` and `safety`. The neighbouring `failsafe` entry looks like +it and is not: SAFETY-02 has site-wide *evidence* and a per-unit *conclusion*, +because it compares the meter against the command **this controller put on one +actuator**. + +Circulating power has no such conclusion available. It is +`Σ|Bᵢ| − |ΣBᵢ|` — defined over the whole roster, **identically zero for any one +unit** — and there is no fact of the matter about which of two opposing +inverters is the offender: *a unit ignoring its share* and *a sibling faithfully +executing a share computed to sum with it* produce **identical readings**. +Filing this per-unit would require inventing an attribution the measurement +cannot support, and would park one inverter while the other went on feeding the +loop — quite possibly parking the only one still listening. + +The same argument is why the latch is **flat** in the store (`circ_tripped`, +`circ_reason`) and not namespaced like SAFETY-02's. One site, one latch. + +## 2. Why the sensor is MEASURED, and why the cheap version is worthless + +The obvious cheap version computes the same formula over `Allocation.shares`. +It would be simpler, would never be `None`, and would be **worth nothing**, for +two independent reasons: + +1. **It cannot fire.** ALLOC-01's sign-coherence gate already refuses any + allocation whose shares carry two signs. A record-derived sensor is therefore + a sensor over a set that is **guaranteed sign-coherent**. It reads zero *by + construction*, forever. +2. **It is [[MUTATION.md]]'s third failure mode.** It could only ever confirm + that the allocator agrees with itself — a check whose data cannot reach the + failure it names. + +Verified rather than argued. On the rig, with inverter B wedged on a stale +discharge while PV drives the site to charge: + +``` + S = -2500 W shares a = -1250 W b = -1250 W <- record: ok, ONE sign, sums to S + MEASURED A = -1250 W B = +1000 W <- hardware + sensor over the RECORD : 0 W + sensor over MEASURED power : 2000 W -> trips at 190 s +``` + +And the card's own headline scenario, driven on the same rig — A at −3000 W, +B at +3000 W, house load 0 W: + +``` + METER (P1) : +0 W <- the site looks IDLE + sensor over MEASURED power : 6000 W -> trips at 100 s + SAFETY-02 SignWatchdog : predicate false on 30 of 30 cycles +``` + +The same pair is asserted in `test_watchdog.py` **with the precondition written +as its own check and printed**, per MUTATION.md's rule, and mutation **M10** +(the property rewritten to read `self.allocation.shares`) takes it **red**. + +## 3. The numbers, and where they come from + +Both were read off a curve produced on `admin/goodwe`'s `sim/` two-master rig, +used read-only: the add-on's own `compute()`, its own `allocator.plan()` and its +own `circulating_w()`, driving `plantsim.Plant` through the firmware's own +encoder, with the sensor input taken from each unit's holding register 0x0518. + +**924 closed-loop runs in which nothing is faulty** — every unit executes the +share it was given — over both families of stimulus that can produce legitimate +opposition (7 reversals × 5 write staggers, and 7 flap periods × 4 amplitudes), +each swept across the **whole range `config.yaml` accepts** for the two settings +that decide how fast the site can turn around: `gain` `float(0.05,1.0)` and +`slew_w` `int(50,5000)`. + +### Reproducing it + +The rig lives in `admin/goodwe` (`sim/`) and was used **read-only**; the harness +is not committed here because it imports that repo's `plantsim`/`frame_check`, +which are not present in this one or in CI. What it does is small enough to +restate: build `Site([Inverter.from_spec("A", INVERTER_A), +Inverter.from_spec("B", INVERTER_B)])`, close the loop with the add-on's own +`compute()` and `allocator.plan()`, put each share on its unit's wire through +`Inverter.command()` (which goes out through the firmware's real encoder), and +read the sensor from `inv.reg(0x0518)` — **the same register the live add-on's +`batt_entity` binds**, so the input is the one the shipped code gets. Point it +at a checkout with `GOODWE_ADDON_REPO`. ⚠️ MUTATION.md's note applies: those +harnesses write `__pycache__` into **both** trees, so purge both by hand before +grading a mutation through one. + +Every figure below was re-run against the final shipped `CirculationConfig` +(`min_w=700`, `hold_s=75`) rather than against the draft values they were first +measured at. + +### Worst CONTINUOUS legitimate opposition, by threshold + +| threshold | worst continuous | runs that ever cross it | +|---:|---:|---:| +| 100 W | **505.0 s** | 281 / 924 | +| 200 W | 445.0 s | 174 / 924 | +| 250 W | 40.0 s | 132 / 924 | +| 300 W | 10.0 s | 115 / 924 | +| **400 W** | **5.0 s** | 74 / 924 | +| 700 W | 5.0 s | 33 / 924 | +| 1000 W | 5.0 s | 16 / 924 | +| 1700 W | 0.0 s | 0 / 924 | + +****With the shipped `min_w=700 W` and `hold_s=75 s`, 0 of those 924 runs trip.** + +The knee is at 400 W and it is sharp.** Below ~250 W a flapping load holds the +sensor above the threshold for *minutes* at a time, and no window shorter than +the 900 s billing quarter could tell that from a fault. At 400 W and above the +worst legitimate episode collapses to **5.0 s — one control cycle — and stays +there**. The worst peak on any legitimate run is **1700 W**. + +That 5.0 s is the serialised write path doing exactly what it should: on a +reversing cycle inverter B's new number lands up to a whole control cycle after +A's, both then slew through zero at 800 W/s behind a 10 s lag, and **between the +two zero crossings they genuinely oppose**. Nothing is wrong. A guard that +tripped on it would be the fourth false-alarm generator of the week, and this +project's doctrine on crying wolf is explicit ([[SAFETY-15]], [[SAFETY-19]]). + +### `circulation_w: 700` — schema floor `400` + +`700` is **1.75× the measured knee**. The headroom is bought for one named +reason: **inverter B does not physically exist.** Its ramp and lag in the rig +are inverter A's, *copied*, carrying `measured=False`; [[SPIKE-04]] measures the +real ones. A slower or faster second unit moves the transient's shape, and the +margin above the knee is what absorbs that without re-opening this card. +**Revisit when SPIKE-04 lands, not before.** + +The **schema floor is the knee itself**, 400 W, so a legally configured install +cannot land in the 445–505 s regime. What the threshold costs: a circulating +loop smaller than 350 W per unit is not reported, which is deliberate — at that +scale the round-trip loss is a few watts. + +### `circulation_hold_s: 75` — schema range `30..900` + +Three bounds, and the value clears all three: + +* **lower, from the rig — 5.0 s.** The worst continuous legitimate opposition + above 700 W across all 924 runs. +* **lower, from the firmware — 30 s (`inverter.LINK_STALE_S`).** A unit still + executing a stale share for less than the firmware's own veto window is a + *link* fault, and the add-on already answers it: liveness drops the unit and + the survivors carry its share, with the record saying so. Firing inside that + window would raise a second, louder alarm about a state already being handled. +* **upper, from the bill — 900 s**, the quarter that sets the capacity tariff. + +**75 s** is fifteen control cycles: **15× the worst legitimate episode**, **2.5× +the firmware veto**, and **12× inside the quarter**. It is deliberately not 60 +or 120 — a round number here would be a guess wearing a measurement's clothes, +and a check in `test_watchdog.py` asserts it is none of the values a guess +lands on. + +## 4. Why the window cannot be defeated the way SAFETY-02's was + +`allocator.py` records the measurement that makes SAFETY-02 blind to the +broadcast fight: the sign disagreement **flips on alternate ticks**, so +`disagree_since` is cleared between every pair of them and the 180 s hold never +accumulates. + +Circulating power cannot do that. It is a **magnitude**, non-negative by +construction, and a sustained loop holds it high on *every* cycle rather than +every other one. An oscillation that genuinely alternated between opposing and +agreeing is not a circulating loop and should not be reported as one — which is +exactly the flapping-load case in §3, and it is why the window clears. + +## 5. SAFETY-34: the latch is not the alarm + +`persistent_notification` is **in memory**. Home Assistant restarts and the card +is gone while the fault runs on — which is precisely how the 2026-09-02 outage +went unseen for **6 h 25 m with the detector correct throughout**. + +So the durable half and the visible half are joined, and the join is +`Controller._circ_announced`: a **per-process** flag, deliberately *not* +persisted. The latch in `/data/state.json` says *"this site is tripped"*; the +flag says *"this process has put that in front of a human"*. A restart clears +the flag and keeps the latch, so **the notification is re-raised on the first +cycle after every start** — which is the first cycle after a Home Assistant +restart, because the add-on restarts with it. + +**One code path serves both** the fresh trip and the restart, on purpose: two +paths would be two chances for the restart one to rot unnoticed, and the restart +one is the case nobody exercises by hand. Mutations **M13** (flag initialised +from the latch) and **M18** (the call removed) both take it red. + +## 6. Mutation ledger + +Every run on a **freshly copied tree with no `__pycache__`, mounted `:ro`** into +the CI image — the structural immunity MUTATION.md records for CI and for +SAFETY-25 — so no stale `.pyc` could exist to be read. Byte deltas recorded; +**three are `+0 B`, the risky delta, and are marked**. + +| # | file | mutation | Δ | result | +|---|---|---|---|---| +| M1 | `circulation.py` | `sum(abs(v))` → `sum(v)` in the sensor | −5 B | **RED** — the 6000 W, 2000 W and three-unit checks | +| M2 | `circulation.py` | N≤1 returns `None` instead of `0.0` | +1 B | **RED** — AC 5's nine-reading sweep | +| M3 | `circulation.py` | a missing reading at N≥2 returns `0.0` | −1 B | **RED** — "health from an absence" | +| M4 | `circulation.py` | drop `max(0.0, ...)` | — | **EQUIVALENT, not run as a claim.** For a same-sign roster the two sums accumulate the same magnitudes in the same order and are bit-identical; 700 000 random rosters produced no negative. Argued at the line and in the check's own text. | +| M5 | `circulation.py` | threshold `<=` → `<` | −1 B | **RED** — "exactly 700 W forever does not trip" | +| M6 | `circulation.py` | window `<` → `<=` | +1 B | **RED** — "trips at 75 s, not before" | +| M7 | `circulation.py` | the window removed entirely | −85 B | **RED** — both slew-transient checks | +| M8 | `circulation.py` | `None` no longer clears the window | +34 B | **RED** — the stale-reading gap check | +| M9 | `circulation.py` | the latch removed | −123 B | **RED** — "restored as tripped stays tripped" | +| **M10** | `main.py` | **the sensor derived from `self.allocation.shares`** | +60 B | **RED** — AC 2's headline check. This is the card. | +| M11 | `main.py` | the `circulation` claim never filed | −235 B | **RED** — `test_watchdog.py` **and** `test_peakguard.py` 10f/10g | +| M12 | `main.py` | the latch not persisted to the store | −112 B | **RED** — all three durability checks | +| M13 | `main.py` | `_circ_announced` initialised from the latch | +21 B | **RED** — SAFETY-34's re-raise | +| M14 | `main.py` | the operator clear leaves `_circ_announced` set | −45 B | **RED** — "the next trip would raise nothing" | +| M15 | `arbiter.py` | `circulation` scoped `SCOPE_UNIT` | **+0 B** | **RED** — `test_arbiter.py` ×3 and `test_peakguard.py` ×3 | +| M16 | `config.yaml` | schema floor `400` → `100` | **+0 B** | **RED** — the measured-knee check | +| M17 | `circulation.py` | default `700` → `500` | **+0 B** | **RED** — the CFG-03 pin and the not-a-round-number check | +| M18 | `main.py` | `await self.check_circulating()` removed | −39 B | **RED** — all four notification checks | +| M19 | `main.py` | publish `round(x or 0.0)` instead of `None` | −68 B | **survivor on the first battery — GAP CLOSED**, then RED | + +**M19 is recorded as a survivor because it was one.** The first battery found no +check grading what `publish()` does when a per-unit reading is missing, so +turning *"I do not know"* into *"0 W"* on the one surface an operator actually +watches went unnoticed — the sensor's own `None` rule, defeated one layer out. +A check was added and the mutation re-run; it is red now. The run that found it +was purged, so it is a result and not an artefact. + +## 7. What this card does NOT close + +* **Inverter B's ramp and lag are placeholders.** Every transient number in §3 + was produced with inverter A's parameters copied onto B (`measured=False`). + SPIKE-04 measures the real ones and is the revisit trigger for the 1.75× + margin. **Do not present any figure here as a measurement of inverter B.** +* **A wedged inverter still cannot be stopped by this add-on.** §1 says so; the + notification says so; there is no software fix and this card does not pretend + to one. +* **A loop below ~350 W per unit is not reported.** Deliberate, §3. +* **The saturation freeze fires under a wedged unit and leaves a standing grid + error.** Observed on the rig while producing §2's numbers: with B wedged, the + loop's anti-windup test compares a site target against a measured site power + that cannot track it, freezes the integrator and leaves the meter parked at + −3750 W. Nothing alarms on that today — a freeze is an ordinary control action + by design. It is a real finding, it is **not** in scope here, and it wants a + card of its own. diff --git a/goodwe_controller/DOCS.md b/goodwe_controller/DOCS.md index db677ae..5ac4ac9 100644 --- a/goodwe_controller/DOCS.md +++ b/goodwe_controller/DOCS.md @@ -761,6 +761,82 @@ unchanged is 12 W, which is inside the 15 W deadband — so the command there is parked at 0 W and the two halves of the rule cannot both be true. At the schema floor of 60 s that figure is 60 W and only the margin is left. +### The circulating-power alarm (SAFETY-31) + +> **Only at two or more inverters.** At the shipped roster of one this quantity +> is identically zero for every reading, no claim is ever filed, and the status +> page says *n/a* rather than pretending to watch it. + +Two inverters can move a great deal of energy while the meter reads almost +nothing. Inverter A charging at 3 kW while inverter B discharges at 3 kW nets +**~0 W at P1**: the site looks idle, and in reality the packs are being cycled +through two round-trip conversion losses continuously, for no benefit. + +**Nothing else in this add-on can see it, and that is structural rather than an +oversight.** Every other guard is a consumer of the *net* figure at the grid +connection — the sign-disagreement failsafe gates on `abs(grid_w) > 300 W` and +that gate never opens at a netted zero; the capacity-peak guard and the tariff +both bill grid exchange. Circulating power is precisely the component of +per-inverter power that the sum destroys. + +So the add-on computes a quantity none of them were computing: + +``` +circulating_w = Σ|Bᵢ| − |ΣBᵢ| +``` + +Zero when the units pull the same way. `2 × min(|B_a|, |B_b|)` when they oppose +— twice the power going round the loop, which is what both conversion stages are +processing between them. It is published as +`sensor.goodwe_circulating_power`, deliberately **beside** the grid figure: +*"grid 0 W, circulating 6000 W"* is the whole fault in two dashboard rows, and +no arithmetic on the grid figure alone can produce the second one. + +**It is derived from each inverter's own measured power, not from what the +controller believed it commanded.** That matters because the allocator already +*refuses* to command opposite directions (ALLOC-01's sign-coherence gate), so a +figure computed from the allocation record would read zero by construction and +detect nothing. What this catches is circulation **nobody commanded** — a unit +ignoring or lagging its share, a stale share still being executed, or an +inverter doing something of its own accord. + +| option | default | meaning | +|---|---|---| +| `circulation_w` | 700 | Watts of measured opposition below which nothing is reported | +| `circulation_hold_s` | 75 | Seconds it must stand there continuously before the trip | + +**Why 700 W and 75 s, and why neither is a round number.** Both were read off +924 closed-loop runs on the two-master test rig in which *nothing* was faulty, +swept across the whole range the schema accepts for `gain` and `slew_w`. A +momentary opposition is **legitimate**: writes to the inverters are serialised, +so on a reversing cycle one unit gets its new number up to a control cycle +before its sibling, both slew through zero, and between the two crossings they +genuinely oppose. The worst *continuous* legitimate opposition, by threshold, +was **505 s at 100 W, 40 s at 250 W, 10 s at 300 W and 5.0 s — one control +cycle — at 400 W and above**. The knee is 400 W and it is the schema floor; +below it, ordinary operation would hold the sensor over the line for minutes and +no usable window could tell that from a fault. 700 W carries margin above the +knee because the second inverter's ramp and lag are not yet measured. 75 s is +15× the worst legitimate episode, 2.5× the firmware's own 30 s link-veto window, +and comfortably inside the 900 s quarter that bills the capacity tariff. + +**What happens when it trips, and what it honestly cannot do.** The site is +commanded **0 W on every inverter** and latched there until you press start. +That is not aimed at making a misbehaving inverter stop — if a unit has stopped +listening, no command will reach it. It does two things that do not need its +cooperation: it removes *this controller* as one of the two participants in the +loop, and it makes anything still happening **visible at the meter** instead of +cancelled at it, where the ordinary guards can see it. **If the grid reading +does not move after the trip, one inverter is no longer listening and the +remaining actuator is a human at the isolator.** + +**The alarm survives a Home Assistant restart.** The latch is written to `/data` +and the notification is re-raised on the first cycle after every start — +`persistent_notification` lives only in memory, and an alarm that vanishes with +a restart is how a real outage on this system went unseen for over six hours. +Restarting does **not** clear the trip; starting control from the add-on UI +does, and only after you have read why. + ### Energy price providers > **A fetched price moves energy only when you switch arbitrage on.** This @@ -985,6 +1061,6 @@ abort one. ## Status entities If an MQTT broker is available the add-on publishes setpoint, grid power, -battery power, state of charge, maintenance phase and controller status by MQTT -discovery. This is observability only — the controller works fine without a +battery power, circulating power, state of charge, maintenance phase and +controller status by MQTT discovery. This is observability only — the controller works fine without a broker, and MQTT problems can never affect control. diff --git a/goodwe_controller/app/arbiter.py b/goodwe_controller/app/arbiter.py index 04aed51..a1ebe42 100644 --- a/goodwe_controller/app/arbiter.py +++ b/goodwe_controller/app/arbiter.py @@ -132,6 +132,25 @@ CLAIM_SCOPE = { # controller regulates against. Both must stop every inverter, and a site # SET at P_SAFETY is how that is said once rather than N times. "safety": SCOPE_SITE, + # main.py: `Claim.set("circulation", P_SAFETY, 0.0, "circulating power")`. + # SAFETY-31, and this one is a SITE claim for a reason worth writing down, + # because the neighbouring `failsafe` entry looks like it and is not. + # + # `failsafe` is SAFETY-02: site-wide evidence, per-unit conclusion. It + # compares the meter against THE COMMAND THIS CONTROLLER PUT ON ONE + # ACTUATOR, so what it learns is about that actuator and the remedy is that + # actuator. + # + # Circulating power is not a property any single inverter has. It is + # `sum(|B_i|) - |sum(B_i)|`, defined over the whole roster and identically + # zero for any one unit, and there is NO FACT OF THE MATTER about which of + # two opposing inverters is the offender - a unit ignoring its share and a + # sibling faithfully executing a share computed to sum with it produce the + # same reading. Filing this per-unit would require inventing an attribution + # the measurement cannot support, and would park one inverter while the + # other went on feeding the loop. The evidence is site-level and so is the + # remedy: stop commanding, everywhere, and say why. + "circulation": SCOPE_SITE, # --- UNIT: about one pack, one inverter, one device ------------------- # main.py: `Claim.limit("device", P_SAFETY, self.dev_min, self.dev_max, diff --git a/goodwe_controller/app/circulation.py b/goodwe_controller/app/circulation.py new file mode 100644 index 0000000..9cbd1fc --- /dev/null +++ b/goodwe_controller/app/circulation.py @@ -0,0 +1,339 @@ +"""SAFETY-31 - the power two inverters move through each other, which the meter cannot see. + +Inverter A charging at 3 kW while inverter B discharges at 3 kW nets ~0 W at +P1. The site looks idle. In reality 3 kW is going round a loop through two +conversion stages continuously, for no benefit, and the wear on both packs is +real even though every instrument in this add-on says nothing is happening. + +WHY NOTHING EXISTING SEES IT, AND IT IS STRUCTURAL RATHER THAN AN OVERSIGHT +========================================================================== +Every consumer of the site measurement is a consumer of the NET. SAFETY-02's +sign-disagreement watchdog gates on `abs(grid_w) > cfg.min_w` (300 W by +default) - with the grid at ~0 that gate never opens, on any cycle, for any +duration. The peak guard bills the quarter-hour average AT THE GRID +CONNECTION. The capacity tariff bills grid exchange. `freeze.looks_frozen()` +grades P1 against the site rail. There is one meter, it reads the sum, and +circulating power is exactly the component of per-unit power that the sum +DESTROYS. So this cannot be fixed by tuning a threshold on any existing +detector; it needs a quantity none of them are computing. + +THE QUANTITY +============ + circulating_w = sum(|B_i|) - |sum(B_i)| + +Zero when every unit pulls the same way - then sum(|B_i|) == |sum(B_i)| by the +triangle equality. `2 * min(|B_a|, |B_b|)` at N=2 when they oppose, which is +twice the power going round the loop and is the right scale: that is what both +conversion stages are processing between them. It is non-negative always, it is +zero at N<=1 identically, and it needs no threshold to be interpretable - the +number IS the phenomenon in watts. + +⚠️ IT IS COMPUTED FROM MEASURED PER-INVERTER POWER, AND THAT IS THE WHOLE CARD. +============================================================================= +The obvious cheap version reads the allocation record - the shares the +allocator just handed out - and reports the same formula over those. It would +be simpler, it would never see a `None`, and it would be worth NOTHING, for two +independent reasons. + +The first is that it cannot fire. ALLOC-01 shipped the sign-coherence gate as +its AC 7 (`allocator.REFUSE_SIGN`, graded by `test_alloc.py` part 6): an +allocation whose shares carry two signs is REFUSED, both units named, no shares +returned. So a sensor over the allocation record is a sensor over a set the +allocator has already guaranteed is sign-coherent. It reads zero by +construction, forever, and a check written against it passes forever. + +The second is the one MUTATION.md names as the third way a check lies. A sensor +derived from the allocation record can only ever confirm THE ALLOCATOR AGREES +WITH ITSELF. Every fault in this card's list is a divergence between what was +commanded and what the hardware is doing: + + * a unit ignoring or lagging its share; + * a stale share still being executed by one inverter; + * SAFETY-25's deferred loop floor, or a maintenance drain, pinning one unit + while the loop drives the other; + * an inverter doing something of its own accord - and THIS HARDWARE HAS DONE + THAT TWICE. On 2026-09-02 the comms subsystem failed and the power stage + went on executing its last instruction for 6 h 25 m (HW-01). Through all of + it the allocation record was impeccable. + +In every one of those the command side is coherent and the MEASUREMENT side is +not. So the input is `Inverter.batt` - this unit's own power, read from its own +`batt_entity`, which on the live system is holding register 0x0518, the +inverter's own AC power. DEV-02 made that per-instance; before it there was one +`batt` scalar on the Controller and this sensor could not have been written at +all. + +⚠️ AND THIS IS THE MEASUREMENT HALF OF A PAIR. ALLOC-01's gate is the COMMAND +half: the allocator can no longer COMMAND circulating power. This module is the +detection half: circulating power the allocator never commanded. Neither one +replaces the other and neither is redundant - the gate refuses an allocation +before it is written, and cannot see a millisecond past that; this reads the +hardware and does not care what was written. Do not "unify" them. + +THE TRIP, AND WHY A THRESHOLD ALONE IS A FALSE-ALARM GENERATOR +============================================================== +A momentary opposition is LEGITIMATE. Writes to the inverters are serialised +(one `hass.call()` at a time - see `Inverter`'s docstring on what a hung write +to A costs B), so on a cycle where the site target reverses, one unit receives +its new command up to a whole control cycle before its sibling. Both then slew +through zero, with a real ramp rate and a real first-order lag, and BETWEEN THE +TWO ZERO CROSSINGS THE UNITS GENUINELY OPPOSE. Nothing is wrong; the site is +turning around. + +This project's doctrine on crying wolf is explicit (SAFETY-15, SAFETY-19) and +this is exactly the shape that generates it. So the trip is a threshold AND a +sustained window, and BOTH numbers were measured on the TEST-19 two-master rig +rather than chosen - see `CIRC_OPTION_DEFAULTS` for the measurements and +`DECISIONS/SAFETY-31-circulating-power.md` for the runs. + +⚠️ THE WINDOW ACCUMULATES, WHICH IS WHY THIS IS NOT SAFETY-02's PROBLEM AGAIN. +The reason the sign watchdog is blind to the broadcast fight is that the +disagreement FLIPS SIGN on alternate ticks, so `disagree_since` is cleared +between every pair of them and the 180 s hold never accumulates (allocator.py's +docstring records the measurement). Circulating power cannot do that: it is a +MAGNITUDE, non-negative by construction, and a sustained loop holds it high on +every cycle rather than on every other one. An oscillation that genuinely +alternated between opposing and agreeing is not a circulating loop and should +not be reported as one. +""" + +from dataclasses import dataclass +from datetime import datetime + +# CFG-03. THE ONE IN-CODE COPY OF THESE DEFAULTS. `config.yaml`'s `options:` +# block is the source of truth an operator edits and the Supervisor reads; +# `schema:` holds the bounds. This dict exists for the reason +# `maintenance.MAINT_OPTION_DEFAULTS` does - the Dockerfile does not put +# config.yaml in the image, so the running add-on cannot read it - and +# `test_watchdog.py` pins the two together by READING config.yaml, so a drift +# goes red in CI rather than shipping. +CIRC_OPTION_DEFAULTS = { + # --- THE THRESHOLD, AND IT IS READ OFF A CURVE ------------------------ + # ⚠️ MEASURED ON THE TEST-19 TWO-MASTER RIG, NOT CHOSEN. 924 closed-loop + # runs in which NOTHING is faulty - every unit executes the share the + # allocator gave it - over both families of stimulus that can produce + # legitimate opposition: single reversals (7 sizes x 5 write staggers) and a + # flapping load (7 periods x 4 amplitudes), each swept across the WHOLE + # range config.yaml accepts for the two settings that decide how fast the + # site can turn around, `gain` float(0.05,1.0) and `slew_w` int(50,5000). + # The worst CONTINUOUS opposition, as a function of the threshold: + # + # 100 W -> 505.0 s 300 W -> 10.0 s 700 W -> 5.0 s + # 200 W -> 445.0 s 350 W -> 10.0 s 1000 W -> 5.0 s + # 250 W -> 40.0 s 400 W -> 5.0 s 1700 W -> 0.0 s + # + # THE KNEE IS AT 400 W and it is sharp. Below ~250 W a flapping load holds + # the sensor above the threshold for MINUTES at a time - 505 s at 100 W - + # and no window shorter than the 900 s billing quarter could tell that from + # a fault. At 400 W and above the worst legitimate episode collapses to 5.0 + # s, ONE control cycle, and stays there. So this number is not a sensitivity + # preference; it is the edge of the regime in which the guard is + # answerable at all. + # + # 700 W is 1.75x the knee, and the margin is bought for one named reason: + # INVERTER B DOES NOT PHYSICALLY EXIST. Its ramp and lag in the rig are + # inverter A's, COPIED, carrying `measured=False` - SPIKE-04 measures the + # real ones. A slower or faster second unit moves the transient's shape, and + # the headroom above the knee is what absorbs that without re-opening this + # card. Revisit it when SPIKE-04 lands, not before. + # + # What it costs: a circulating loop smaller than 350 W per unit is not + # reported. That is deliberate - at that scale the round-trip loss is a few + # watts and the wear is proportionate to it. + "circulation_w": 700, + + # --- THE WINDOW ------------------------------------------------------- + # Two lower bounds, one upper, and the chosen value clears all three. + # + # LOWER, from the rig: 5.0 s. The worst continuous legitimate opposition + # above 700 W across all 924 runs is exactly one control cycle - a + # reversing site whose second write lands a cycle after the first, which + # is what the serialised write path does on every reversal. + # + # LOWER, from the firmware: `inverter.LINK_STALE_S`, 30 s. A unit still + # executing a stale share for less than that is a LINK fault, and the + # add-on already has an answer for it - liveness drops the unit and the + # survivors carry its share, with the record saying so. Firing inside that + # window would raise a second, louder alarm about a state that is already + # being handled, which is the SAFETY-15/SAFETY-19 failure exactly. + # + # UPPER, from the bill: 900 s, the quarter that sets the capacity tariff. + # A circulating loop must be caught and acted on well inside one. + # + # 75 s: fifteen control cycles - 15x the worst legitimate episode this rig + # can produce, 2.5x the firmware's own veto window, and 12x inside the + # quarter. ⚠️ IT IS DELIBERATELY NOT 60 OR 120. A round number here would be + # a guess wearing a measurement's clothes, and the three bounds above are + # what the number has to answer to. + "circulation_hold_s": 75, +} + + +@dataclass(frozen=True) +class CirculationConfig: + """Defaults, not constants - config.yaml carries both. See CFG-03 above.""" + + min_w: float = float(CIRC_OPTION_DEFAULTS["circulation_w"]) + hold_s: float = float(CIRC_OPTION_DEFAULTS["circulation_hold_s"]) + + +def circulating_w(powers): + """`sum(|B_i|) - |sum(B_i)|` over MEASURED per-inverter power, in W. + + `powers` one measured power per inverter, in roster order, + = discharge. + A unit whose reading did not arrive is `None`. + + Returns a non-negative float, or `None` when the answer is unknown. + + ⚠️ N<=1 IS IDENTICALLY ZERO AND RETURNS ZERO EVEN FOR A MISSING READING + (AC 5). With one term `sum(|B|) - |sum(B)| == |B| - |B| == 0` for EVERY + value of B, including a value nobody read. Returning `None` there would be + reporting an unknown where the algebra has no room for one, and it would put + a shipped one-inverter install permanently in the "cannot tell" state on a + quantity that cannot be anything but zero. Nothing changes at the shipped + roster; that is the compatibility promise this card makes. + + ⚠️ AND AT N>=2 A MISSING READING IS `None`, NOT ZERO, WHICH IS THE OPPOSITE + DIRECTION AND FOR A REASON. Zero would be this sensor reporting HEALTH from + an absence - the precise shape of the 2026-09-02 outage, where a subsystem + stopped answering and its silence read as calm for 6 h 25 m. An unknown + circulating figure is an unknown, it CLEARS the trip's window rather than + accumulating it (see `CirculationWatchdog.evaluate`), and the site is + already stopped for a missing per-unit reading anyway - `inputs_ok` requires + every unit's `batt`, and §11.1 rule 1 commands 0 W without consulting this. + """ + vals = list(powers) + if len(vals) <= 1: + return 0.0 + if any(v is None for v in vals): + return None + vals = [float(v) for v in vals] + # ⚠️ max(0.0, ...) IS DEFENSIVE, AND IT IS AN EQUIVALENT MUTANT - SAID HERE + # RATHER THAN LEFT FOR THE NEXT MUTATION SWEEP TO REDISCOVER. The triangle + # inequality makes this non-negative mathematically, and in floating point + # it is non-negative too: for a same-sign roster `abs(v)` is the list itself + # up to one sign, so the two sums accumulate THE SAME MAGNITUDES IN THE SAME + # ORDER and are bit-identical; for a mixed-sign roster the two differ by + # 2*min(...), which is enormous beside any rounding. Searched, not assumed - + # 700 000 random rosters of 2 to 12 units across 10 orders of magnitude + # produced no negative result. + # + # So removing this line changes no observable and no check here would go + # red. It stays because the cost is one call and the failure it would + # prevent - a power sensor publishing -1e-16 W, which is the kind of value + # somebody eventually writes a `< 0` branch against - is out of proportion + # to it. Do not report its survival as a gap in the checks. + return max(0.0, sum(abs(v) for v in vals) - abs(sum(vals))) + + +def opposing(powers, min_w: float = 0.0): + """The units pulling each way right now, as `(discharging, charging)` names. + + Indices, not names - the caller owns the roster order. Used for the operator + sentence: "which two inverters are pushing against each other" is the first + thing somebody woken by this alarm needs, and reconstructing it from the + number alone is impossible. + + `min_w` ignores units below a magnitude, so a unit resting at 3 W of + standby draw is not named as a participant in a 3 kW loop. + """ + pos = [i for i, v in enumerate(powers) + if v is not None and float(v) > float(min_w)] + neg = [i for i, v in enumerate(powers) + if v is not None and float(v) < -float(min_w)] + return pos, neg + + +class CirculationWatchdog: + """ONE PER SITE, evaluated once per control cycle. + + ⚠️ ONE PER SITE, WHERE SAFETY-02's IS ONE PER ACTUATOR, AND THE ASYMMETRY IS + THE WHOLE ATTRIBUTION ARGUMENT. `SignWatchdog` concludes "THE UNIT I AM + DRIVING is pushing power the wrong way" - evidence about one actuator, so + the remedy is that actuator (arbiter.CLAIM_SCOPE calls `failsafe` a UNIT + claim for exactly this reason). Circulating power is not a property any one + inverter has. It is a property of a PAIR, it is defined by a sum across the + roster, and there is no fact of the matter about which of the two is the + offender: a unit ignoring its share and a sibling faithfully executing a + share computed to sum with it produce identical readings. So the quantity is + site-level, the watchdog is site-level, and the remedy below is site-level. + + Latches, for `SignWatchdog`'s reason: the trip removes its own evidence. The + moment the site stops commanding, the loop it detected either persists (a + wedged unit, now visible at the meter) or vanishes (the add-on was one of + the two participants) - and in the second case a non-latching detector would + de-assert and start the whole thing again, three minutes at a time, forever. + """ + + def __init__(self, cfg: CirculationConfig | None = None, *, + tripped: bool = False, reason: str = ""): + self.cfg = cfg or CirculationConfig() + self.over_since: datetime | None = None + # ⚠️ RESTORED FROM THE STORE, NOT DEFAULTED - SAFETY-34, and it is the + # reason this card exists in the shape it does. `persistent_notification` + # is IN MEMORY: Home Assistant restarts and the card is gone, with the + # fault still running. That is precisely how the 2026-09-02 outage went + # unseen for 6 h 25 m while the detector had been correct for hours. The + # latch lives in `/data/state.json` - the add-on's only durable volume - + # and `main.py` RE-RAISES the notification from it on every start. + self.tripped = bool(tripped) + self.reason = reason if self.tripped else "" + # The worst figure seen during the current episode, for the operator + # sentence. Not persisted: it is colour on a reason string that IS. + self.peak_w = 0.0 + + def evaluate(self, circ_w, now: datetime) -> bool: + """Advance the detector one cycle. Returns True while tripped. + + `circ_w` this cycle's `circulating_w()` over MEASURED per-unit power, + or None when a reading is missing. None CLEARS the window + rather than accumulating it - see the module docstring, and + `SignWatchdog.evaluate` on the identical decision for a stale + meter. A frozen figure held against a moving one manufactures + a fault in about a window's worth of cycles. + `now` the caller's clock, so this is testable and replayable. + """ + if self.tripped: + # Latched. Do not re-evaluate: the trip removed the evidence. + return True + + if circ_w is None or float(circ_w) <= self.cfg.min_w: + self.over_since = None + self.peak_w = 0.0 + return False + + circ_w = float(circ_w) + self.peak_w = max(self.peak_w, circ_w) + if self.over_since is None: + self.over_since = now + return False + + held = (now - self.over_since).total_seconds() + if held < self.cfg.hold_s: + return False + + self.tripped = True + self.reason = ( + f"CIRCULATING_POWER: {circ_w:.0f} W measured going round between " + f"the inverters (peak {self.peak_w:.0f} W) for {held:.0f} s " + f"continuously - the meter cannot see it, because it nets to zero " + f"there (limits: over {self.cfg.min_w:.0f} W, held over " + f"{self.cfg.hold_s:.0f} s)") + return True + + def held_s(self, now: datetime) -> float: + """How long the current opposition has stood, for the status page. + + Takes the clock rather than reading one, for `SignWatchdog.held_s`'s + reason: every observable on this object is a pure function of what it + was handed, and an unclocked function is one nothing can test. + """ + if self.over_since is None: + return 0.0 + return (now - self.over_since).total_seconds() + + def reset(self) -> None: + self.over_since = None + self.tripped = False + self.reason = "" + self.peak_w = 0.0 diff --git a/goodwe_controller/app/main.py b/goodwe_controller/app/main.py index 3e6b4d5..a9e90d8 100644 --- a/goodwe_controller/app/main.py +++ b/goodwe_controller/app/main.py @@ -38,6 +38,10 @@ from .arbiter import ( Claim, Resolution, actuator_for, resolve, ) from .control import Tuning, compute, maintenance_charge_floor, peak_at_risk +from .circulation import ( + CIRC_OPTION_DEFAULTS, CirculationConfig, CirculationWatchdog, + circulating_w, opposing, +) from .clamp import ( REASON_CODE as CLAMP_REASON, ClampWatchdog, clamp_to_rating, is_commandable, ) @@ -249,6 +253,39 @@ class Controller: self.freeze_wd = FreezeWatchdog( FreezeWatchdogConfig(hold_s=float(opts.get("meter_freeze_s", 300)))) + # SAFETY-31. ONE PER SITE, and restored from the store like SAFETY-02's + # latch rather than defaulted like SAFETY-18's - the asymmetry is the + # same one the freeze watchdog's own note draws, read the other way. + # This one ACTS: a trip files a P_SAFETY SET that stops every inverter, + # so a restart that cleared it would come back up straight into the loop + # it had just stopped, which is SAFETY-02's measured three-minutes-at-a- + # time failure with a different quantity in front of it. + # + # ⚠️ SITE-LEVEL KEYS, NOT NAMESPACED. store.py carries the argument: + # circulating power is defined over the roster and is identically zero + # for any one unit, so there is no inverter to hang the latch on. + self.circ_wd = CirculationWatchdog( + CirculationConfig( + min_w=float(opts.get("circulation_w", + CIRC_OPTION_DEFAULTS["circulation_w"])), + hold_s=float(opts.get("circulation_hold_s", + CIRC_OPTION_DEFAULTS["circulation_hold_s"]))), + tripped=bool(store.data.get("circ_tripped", False)), + reason=str(store.data.get("circ_reason", "") or "")) + # SAFETY-34. Has the durable alarm been put back in front of the + # operator since this process started? `persistent_notification` is in + # memory and does NOT survive the Home Assistant restart that the store + # does, so the latch alone is not the alarm - re-raising it is. + # `run_control()` does that, once, before the first cycle. + self._circ_announced = False + if self.circ_wd.tripped: + _LOG.error( + "SAFETY-31 was TRIPPED before this restart and stays tripped: " + "%s. Every inverter is held at 0 W. Restarting does not clear " + "this - start control from the UI once you know why two " + "inverters were pushing against each other.", + self.circ_wd.reason or "(reason not recorded)") + # SAFETY-03. The output clamp's alarm - see clamp.py for why the clamp # itself is the cheap half. # @@ -720,6 +757,35 @@ class Controller: return None return float(sum(float(v) for v in vals)) + @property + def site_circulating_w(self): + """SAFETY-31. `sum(|B_i|) - |sum(B_i)|` over MEASURED per-unit power. + + ⚠️ `inv.batt`, NOT `self.allocation`. THIS IS THE WHOLE CARD, and the + wrong version is the tempting one: the same formula over the allocation + record would be simpler, would never be None, and would be worth + nothing. ALLOC-01's sign-coherence gate (`allocator.REFUSE_SIGN`) already + refuses an allocation whose shares carry two signs, so a record-derived + sensor reads zero BY CONSTRUCTION, forever, and can only ever confirm + that the allocator agrees with itself. Every fault this card is for is a + divergence between what was commanded and what the hardware is doing - a + unit ignoring or lagging its share, a stale share still being executed, + or HW-01's power stage running its last instruction for 6 h 25 m with + the comms dead and the allocation record impeccable throughout. + Measured on the TEST-19 rig: with one unit wedged, the record stays + `ok` with both shares at -1250 W - a record-derived sensor reads 0 W - + while this one reads 2000 W. + + `batt` became per-instance in DEV-02; before that there was one scalar + on the Controller and this property could not have been written. + + None when any unit has not answered, and NOT zero - circulation.py says + why at length. The short version: zero would be this sensor reporting + health from an absence, which is the 2026-09-02 failure itself. + Identically 0.0 at the shipped roster of one (AC 5). + """ + return circulating_w([inv.batt for inv in self.inverters]) + @property def pack_floor_soc(self): """MINIMUM. The SoC that a discharge PERMISSION must be graded against. @@ -1204,6 +1270,39 @@ class Controller: # share of it. At the shipped roster of one the two are the same number. site_standing = float(sum(standing.values())) + # --- SAFETY-31, AND IT RUNS BESIDE SAFETY-02 FOR THE SAME REASON ----- + # Evaluated EVERY cycle, unconditionally, above the "inputs missing but + # not yet stale" hold below. A run of held cycles would otherwise leave + # `over_since` untouched across the gap and the window would resume from + # a start time that predates it, which is the ordering bug the sign + # watchdog's own note describes. + # + # ⚠️ THE INPUT IS MEASURED POWER, WHICH IS THE ENTIRE POINT OF THE CARD. + # `site_circulating_w` reads each `Inverter.batt` - that unit's own + # meter, holding register 0x0518. It is NOT `self.allocation`, and the + # property's docstring carries the argument: ALLOC-01's sign-coherence + # gate already makes the allocation record incapable of expressing + # circulating power, so a record-derived sensor is zero by construction + # and detects nothing. What this catches is circulation the allocator + # NEVER COMMANDED - a unit ignoring or lagging its share, a stale share + # still being executed, or HW-01's wedged power stage. + circ_w = self.site_circulating_w + was_circ = self.circ_wd.tripped + if self.circ_wd.evaluate(circ_w, now) and not was_circ: + # REASON FIRST, FLAG SECOND, and each `set` writes the file - the + # same atomicity ordering SAFETY-02 uses two blocks up, and for the + # same reason: a crash between the two must not persist a latch with + # no explanation for the operator who has to decide about it. + self.store.set("circ_reason", self.circ_wd.reason) + self.store.set("circ_tripped", True) + self._circ_announced = False + self.log_event("CIRCULATING POWER - " + self.circ_wd.reason) + _LOG.error( + "SAFETY-31 TRIPPED - %s. Commanding 0 W on every inverter and " + "holding it until control is started again. This survives an " + "add-on restart AND a Home Assistant restart; restarting will " + "not clear it.", self.circ_wd.reason) + stale_after = float(self.o.get("stale_input_s", 15)) # ⚠️ `any`, NOT the sole unit's. The hold exists so a single missed poll # does not become a fault; a LATCHED inverter must still reach the claim @@ -1265,6 +1364,58 @@ class Controller: "sign disagreement", actuator=self.actuator_of("failsafe", inv))) + # --- SAFETY-31: WHAT "ACTED ON" MEANS, AND WHAT IT HONESTLY CANNOT -- + # ⚠️ A SITE SET AT P_SAFETY: STOP COMMANDING, EVERYWHERE. Read the next + # paragraph before changing it, because the obvious objection is right + # and is answered rather than ignored. + # + # THE OBJECTION. HW-01 established that commanding a wedged inverter + # achieves nothing - on 2026-09-02 the comms subsystem failed and the + # power stage went on executing its last instruction for 6 h 25 m, + # through every command this add-on sent it. If the cause of the + # circulation is a unit that has stopped listening, then a command is + # not a remedy and pretending otherwise is theatre. + # + # THE ANSWER, AND IT IS WHY THIS IS STILL AN ACTION RATHER THAN A + # NOTIFICATION. A circulating loop needs TWO participants. In every + # cause on this card's list, exactly one of them is a unit that has + # stopped listening and THE OTHER ONE IS THIS CONTROLLER - faithfully + # commanding the sibling to a share computed to sum with a number the + # wedged unit is not actually producing. Stopping the commanding half + # does two things a notification cannot: + # + # 1. IT HALVES THE LOOP. The unit that is still listening stops being + # driven into the one that is not, so the round-tripping stops on + # the side this controller owns. That is real energy and real + # cycles off a real pack, and it needs no cooperation from the + # wedged unit to take effect. + # 2. IT MAKES THE RESIDUE VISIBLE. This is the more important half. + # Circulating power hides because it NETS TO ZERO AT THE METER. + # With this controller's contribution removed, whatever the wedged + # unit is still doing stops being cancelled and appears at P1 as a + # plain net flow - where SAFETY-02's sign watchdog (which gates on + # `abs(grid_w) > 300` and could never open at a netted zero), the + # peak guard and the capacity tariff can all see it. The fault + # stops being invisible. Measured on the TEST-19 rig: the card's + # own scenario reads +0 W at the meter with 6000 W going round, and + # the shipped SignWatchdog's predicate is false on every one of + # those cycles. + # + # WHAT IT DOES NOT DO, SAID PLAINLY: it does not stop the wedged + # inverter. Nothing this add-on can send will. If the loop persists at + # the meter after this trip, the remaining actuator is a human at the + # isolator, which is what the notification says in those words. + # + # ⚠️ AND IT IS A SITE CLAIM. `arbiter.CLAIM_SCOPE` carries the argument: + # circulating power is defined over the roster and is identically zero + # for any one unit, so there is no attribution to file this per-unit on - + # and parking the wrong one of the two would leave the loop running with + # the guard satisfied. + if self.circ_wd.tripped: + claims.append(Claim.set("circulation", P_SAFETY, 0.0, + "circulating power", + actuator=self.actuator_of("circulation"))) + # SITE, both of them, and the two reasons are why. "stopped" is the # operator switching control off for the whole site; "inputs missing" is # the SITE's own telemetry gone - the meter this controller regulates @@ -1794,6 +1945,16 @@ class Controller: # is not, and test_freeze.py part 5 fails on exactly that. await self.check_meter_frozen() + # --- SAFETY-31's alarm, and its boundary is the LATCH, not a number --- + # Unconstrained where the two above are not: it grades + # `self.circ_wd.tripped`, which was advanced near the top of this method + # beside SAFETY-02 and which nothing since has touched. It is called on + # EVERY cycle rather than only on the cycle that trips, because its + # second job is re-raising the alarm after a restart into an existing + # trip - see its docstring on why that shares one code path with the + # fresh trip instead of getting its own. + await self.check_circulating() + # --- SAFETY-07's alarm, and it is NOT the engagement ------------------ # Its boundary is unconstrained: it grades `self.peak_guard`, which this # method assigned above and nothing since has touched. @@ -2050,6 +2211,71 @@ class Controller: f"you a lower bill, not the pack being spent."), }) + async def check_circulating(self) -> None: + """SAFETY-31 + SAFETY-34. The alarm, and it must survive a restart. + + ⚠️ THE LATCH IS NOT THE ALARM. `circ_tripped` in `/data/state.json` + survives everything, but nothing looks at a JSON file at 3 a.m. What the + operator sees is a `persistent_notification`, and THAT IS IN MEMORY: a + Home Assistant restart deletes it while the fault runs on. This is not a + hypothetical - it is precisely how the 2026-09-02 outage went unseen for + 6 h 25 m with the detector correct the whole time (SAFETY-34). + + So the durable half and the visible half are joined HERE, and the join + is `_circ_announced`: a per-PROCESS flag, deliberately not persisted. + The latch says "this site is tripped"; the flag says "this process has + put that in front of a human". A restart clears the flag and keeps the + latch, so the notification is RE-RAISED on the first cycle after every + add-on start - which is also the first cycle after a Home Assistant + restart, because the add-on is restarted with it. + + ⚠️ ONE CODE PATH FOR BOTH, ON PURPOSE. A fresh trip and a restart into + an existing trip raise the alarm through these same lines. Two paths + would be two chances for the restart one to rot unnoticed, and the + restart one is the case nobody exercises by hand. + """ + if not self.circ_wd.tripped or self._circ_announced: + return + # Set BEFORE the await. `hass.call()` can raise or hang, and a retry + # every cycle for a broker that is down would be an alarm storm on the + # event bus on top of a site that is already stopped. The latch is + # already durable and the log line above already fired; this surface is + # best-effort by design. + self._circ_announced = True + names = self.units + pos, neg = opposing([inv.batt for inv in self.inverters], + min_w=self.circ_wd.cfg.min_w / 2.0) + who = "" + if pos and neg: + who = ("\n\nRight now " + ", ".join(names[i] for i in pos) + + " is discharging while " + ", ".join(names[i] for i in neg) + + " charges.") + # A fixed notification_id, and a DIFFERENT one from every other alarm + # this add-on raises - check_meter_frozen() gives the reasoning. A + # circulating loop is not a frozen meter, is not an out-of-range + # setpoint and is not a saturated peak guard. + await self.hass.call("persistent_notification", "create", { + "notification_id": "goodwe_circulating_power", + "title": "GoodWe: the inverters are pushing against each other", + "message": ( + f"{self.circ_wd.reason}{who}\n\nTHE METER CANNOT SEE THIS. " + f"Circulating power nets to zero at P1, so the site reads idle " + f"while the packs are cycled through two conversion stages " + f"continuously. Every other guard in this add-on is a consumer " + f"of that net figure, which is why none of them fired.\n\n" + f"CONTROL HAS BEEN STOPPED - every inverter is commanded 0 W " + f"and held there. That removes this controller as one of the " + f"two participants, and it makes anything still happening " + f"VISIBLE at the meter instead of cancelled at it.\n\n" + f"⚠️ IF THE GRID READING DOES NOT MOVE AFTER THIS, one inverter " + f"is no longer listening to this add-on and no command will " + f"reach it - that has happened twice on this hardware, once for " + f"6 h 25 m. The remaining actuator is a human at the isolator.\n" + f"\nThis alarm survives a Home Assistant restart. Restarting " + f"will not clear it; start control from the add-on UI once you " + f"know why two inverters were opposing."), + }) + async def check_meter_frozen(self) -> None: """SAFETY-18. Alarm on a meter that answers forever without changing. @@ -2354,7 +2580,14 @@ class Controller: # additive to its vocabulary rather than a change to it. "status": ("failsafe" if any(i.sign_wd.tripped for i in self.inverters) + or self.circ_wd.tripped else "running" if self.auto else "stopped"), + # SAFETY-31. ⚠️ MEASURED, not the allocation record - see + # `site_circulating_w`. `publish()` drops None values, so a cycle + # with a missing per-unit reading leaves the last figure standing + # rather than publishing a zero that would read as health. + "circulation": (None if self.site_circulating_w is None + else round(self.site_circulating_w)), } # ⚠️ ONLY when P1 ingestion is actually running. The ESP32's stale-input # watchdog subscribes to sensor.goodwe_p1_sample_age - see the entity-id @@ -2691,6 +2924,35 @@ class Controller: f"{inv.sign_wd.cfg.min_w:.0f} W)" + (f" - opposed for {held:.0f} s now" if held else "")}) + # --- SAFETY-31 ----------------------------------------------------- + # ⚠️ SITE-LEVEL, WHERE THE ROWS ABOVE ARE PER-INVERTER, and the row says + # so. There is one circulating figure for the site; naming an inverter + # here would invite the operator to clear "the other one". + _circ = self.site_circulating_w + if self.circ_wd.tripped: + out.append({"ok": False, "warn": False, + "text": "CIRCULATING POWER - " + self.circ_wd.reason + + ". EVERY inverter is held at 0 W. The meter " + "cannot see this fault, which is why nothing " + "else reported it. Starting control clears " + "this; restarting the add-on does NOT, and " + "neither does restarting Home Assistant."}) + elif len(self.inverters) <= 1: + # AC 5. At the shipped roster the quantity is identically zero for + # every value of every reading, so "armed" would be a claim to be + # watching something that cannot happen. Say what is true instead. + out.append({"ok": True, "warn": False, + "text": "circulating power: n/a at one inverter - " + "sum(|Bi|) - |sum(Bi)| is identically zero"}) + else: + held = self.circ_wd.held_s(datetime.now(timezone.utc)) + out.append({"ok": True, "warn": False, + "text": "circulation watchdog: armed (trips after " + f"{self.circ_wd.cfg.hold_s:.0f} s over " + f"{self.circ_wd.cfg.min_w:.0f} W)" + + (f" - now {_circ:.0f} W" if _circ else "") + + (f", opposed for {held:.0f} s" if held else "")}) + if not o.get("estop_fitted", False): out.append({"ok": False, "warn": True, "text": "no e-stop fitted: if this host dies the battery stays " @@ -2708,7 +2970,17 @@ class Controller: def status_payload(self) -> dict: checks = self.checks() bad = [c for c in checks if not c["ok"] and not c["warn"]] - if any(i.sign_wd.tripped for i in self.inverters): + if self.circ_wd.tripped: + # ⚠️ ABOVE THE SIGN FAILSAFE, and the order is a decision. Both are + # latched P_SAFETY stops, so the site is at 0 W either way and the + # banner can only carry one sentence. This one goes first because it + # is the fault the operator has NO OTHER WAY of learning about: a + # sign disagreement is visible at the meter and on every dashboard + # in the house, and circulating power is visible nowhere at all. + level, banner = "bad", ("CIRCULATING POWER — the inverters were " + "pushing against each other. Site held at " + "0 W.") + elif any(i.sign_wd.tripped for i in self.inverters): # Outranks "NOT READY": nothing is missing, something went wrong. level, banner = "bad", ("FAILSAFE — the command opposed the meter. " "Inverter held at 0 W.") @@ -2734,6 +3006,14 @@ class Controller: rows = [ ("Grid", w(self.grid)), ("Battery", w(self.site_batt_w)), + # SAFETY-31, and it belongs DIRECTLY UNDER the two figures it is not + # derivable from. "Grid 0 W, Battery 0 W, Circulating 6000 W" is the + # whole card in three rows: both of the numbers an operator normally + # reads say the site is idle, and the third says 6 kW is going round + # between the packs. Only shown at N>=2 - at one inverter it is + # identically zero and a permanent "0 W" row would be noise. + *(() if len(self.inverters) <= 1 + else ((("Circulating", w(self.site_circulating_w)),))), ("State of charge", "—" if self.pack_floor_soc is None else f"{self.pack_floor_soc:g} %"), ("Commanded", w(self.target)), @@ -2808,6 +3088,22 @@ class Controller: # unit in particular. self.store.set("sign_tripped", False) self.store.set("sign_reason", "") + # SAFETY-31, cleared by the SAME press and for the same reason: + # the condition always goes away on its own, because the trip + # commands 0 W everywhere and two inverters at zero cannot + # oppose. Only a human who has read the banner can say the site + # is clear. + # + # ⚠️ AND `_circ_announced` GOES WITH IT. It is the "has a human + # been shown this" flag; leaving it True through a clear would + # mean the NEXT trip in this process latched, stopped the site, + # and raised no notification at all. + if self.circ_wd.tripped: + self.circ_wd.reset() + self.store.set("circ_tripped", False) + self.store.set("circ_reason", "") + self.log_event("circulating-power trip cleared by operator") + self._circ_announced = False if not self.auto: # ⚠️ THE WHOLE SITE, TARGETS INCLUDED - see command_site_zero(). # Zeroing `self.target` alone would leave each unit's last share diff --git a/goodwe_controller/app/mqtt.py b/goodwe_controller/app/mqtt.py index 7d6c807..f29916c 100644 --- a/goodwe_controller/app/mqtt.py +++ b/goodwe_controller/app/mqtt.py @@ -43,6 +43,19 @@ SENSORS = [ ("grid", "goodwe_grid_power", "Grid power", "W", "power", "measurement", None), ("battery", "goodwe_battery_power", "Battery power", "W", "power", "measurement", None), ("soc", "goodwe_battery_soc", "Battery SoC", "%", "battery", "measurement", None), + # SAFETY-31. `sum(|Bi|) - |sum(Bi)|` over MEASURED per-inverter power: the + # power going round between the inverters, which the grid sensor above + # CANNOT SHOW because it nets to zero there. The two are published side by + # side on purpose - "grid 0 W, circulating 6000 W" is the whole phenomenon + # in two rows of a dashboard, and no arithmetic on the grid figure alone can + # ever produce the second one. + # + # ⚠️ Published as 0 at the shipped one-inverter roster, not omitted, and + # that is a deliberate difference from `p1_age` below. An absent entity is + # right for a watchdog input nothing is feeding; this is a quantity whose + # value is KNOWN at N=1 - identically zero, for every reading - and a + # dashboard row reading 0 W is the honest report of it. + ("circulation", "goodwe_circulating_power", "Circulating power", "W", "power", "measurement", "mdi:sync-alert"), ("phase", "goodwe_maintenance_phase", "Maintenance phase", None, None, None, "mdi:battery-sync"), ("status", "goodwe_controller_status", "Controller status", None, None, None, "mdi:heart-pulse"), # ⚠️ THE LIVE ENTITY IS `sensor.goodwe_p1_sample_age`, NOT what this diff --git a/goodwe_controller/app/store.py b/goodwe_controller/app/store.py index 98e606f..f811d11 100644 --- a/goodwe_controller/app/store.py +++ b/goodwe_controller/app/store.py @@ -54,6 +54,23 @@ _LOG = logging.getLogger("goodwe.store") # `migrate_unit_keys()` and `Controller._grade_store_migration()` carry the # argument; the short version is that a latch is only worth persisting if it # survives the version change an operator makes while staring at the banner. +# +# ⚠️ AND SAFETY-31's PAIR IS FLAT ON PURPOSE, WHICH IS THE OPPOSITE DECISION TO +# THE ONE ABOVE AND NEEDS ITS OWN SENTENCE. `circ_tripped` / `circ_reason` are +# NOT in `LEGACY_UNIT_KEYS` and must never be moved into a unit namespace. +# Circulating power is `sum(|B_i|) - |sum(B_i)|` - a quantity DEFINED over the +# whole roster, which is identically zero for any single unit. There is no fact +# of the matter about which of two opposing inverters is the offender: a unit +# ignoring its share and a sibling faithfully executing a share computed to sum +# with it read identically. Namespacing this latch would mean inventing an +# attribution the measurement cannot support, and would then let an operator +# clear "inverter B's" half of a site-wide loop. One site, one latch. +# +# It is persisted for SAFETY-34's reason, which this add-on has already paid +# for once: `persistent_notification` is IN MEMORY. Home Assistant restarts and +# the card is gone while the fault runs on - exactly how the 2026-09-02 outage +# went unseen for 6 h 25 m with the detector correct throughout. main.py +# re-raises the notification from this key on every start. DEFAULTS = { "phase": "idle", "phase_started": None, @@ -61,6 +78,8 @@ DEFAULTS = { "last_start_attempt": None, "sign_tripped": False, "sign_reason": "", + "circ_tripped": False, + "circ_reason": "", } # --------------------------------------------------------------------------- # diff --git a/goodwe_controller/config.yaml b/goodwe_controller/config.yaml index fa47061..a30c3a4 100644 --- a/goodwe_controller/config.yaml +++ b/goodwe_controller/config.yaml @@ -134,6 +134,25 @@ options: # still inside the 900 s quarter that bills the capacity tariff. meter_freeze_s: 300 + # --- SAFETY-31: circulating power ------------------------------------------ + # ⚠️ CFG-03. THESE TWO LINES ARE THE SOURCE OF TRUTH FOR THESE DEFAULTS, and + # `app/circulation.py`'s `CIRC_OPTION_DEFAULTS` is the ONE in-code copy - the + # Dockerfile does not put this file in the image, so the running add-on cannot + # read it. test_watchdog.py pins the copy by READING these lines, so a drift + # goes red in CI rather than shipping. Add a third option in three places - + # options:, schema:, CIRC_OPTION_DEFAULTS - and in no fourth. + # + # Circulating power is sum(|Bi|) - |sum(Bi)| over MEASURED per-inverter power: + # two inverters pushing against each other, which NETS TO ZERO AT THE METER + # and is therefore invisible to every other guard in this add-on. Both numbers + # below were measured on the TEST-19 two-master rig over 924 closed-loop runs + # in which nothing was faulty; circulation.py carries the curve they were read + # off and the argument for each. They are NOT sensitivity preferences. + # + # Both are ignored at one inverter, where the quantity is identically zero. + circulation_w: 700 + circulation_hold_s: 75 + # --- maintenance ----------------------------------------------------------- # ⚠️ CFG-03. THESE EIGHT LINES ARE THE SOURCE OF TRUTH FOR THE MAINTENANCE # DEFAULTS. The Supervisor reads this file and the operator edits this file, @@ -398,6 +417,28 @@ schema: # app/freeze.py. meter_freeze_s: int(60,3600) + # SAFETY-31. No "off" value, for SAFETY-02's and meter_freeze_s's reason: + # disarming a safety alarm is not a supported configuration. + # + # ⚠️ THE FLOOR OF 400 IS NOT A RECOMMENDATION, IT IS THE MEASURED EDGE OF THE + # REGIME IN WHICH THIS GUARD CAN BE ANSWERED FOR. On the TEST-19 rig, the + # worst CONTINUOUS opposition produced by a site where nothing is faulty - a + # flapping load reversing every control cycle, swept over the whole range this + # schema accepts for `gain` and `slew_w` - is 5.0 s at a 400 W threshold and + # above, 10.0 s at 300 W, 40.0 s at 250 W and 505.0 s at 100 W. Below the + # knee, legitimate operation holds the sensor over the threshold for MINUTES + # and no window shorter than the 900 s billing quarter could tell that from a + # fault. A configurable value under 400 W would be a false-alarm generator + # that this project's own doctrine (SAFETY-15, SAFETY-19) forbids shipping. + circulation_w: int(400,5000) + # The floor of 30 is `inverter.LINK_STALE_S`: below the firmware's own veto + # window a unit still executing a stale share is a LINK fault the add-on + # already handles by dropping it from the split, and firing inside that window + # would raise a second, louder alarm about a state already being managed. The + # ceiling of 900 is the quarter that bills the capacity tariff - a circulating + # loop must be caught and acted on well inside one. + circulation_hold_s: int(30,900) + # ⚠️ CFG-03 - THIS IS WHERE A RANGE LIVES, AND THE ONLY PLACE IT LIVES. The # `options:` block above holds the DEFAULT; this block holds the BOUNDS, and # `MAINT_OPTION_DEFAULTS` deliberately mirrors the former and not the latter - diff --git a/goodwe_controller/test_arbiter.py b/goodwe_controller/test_arbiter.py index 7549004..883e27d 100644 --- a/goodwe_controller/test_arbiter.py +++ b/goodwe_controller/test_arbiter.py @@ -160,6 +160,11 @@ SITE_SOURCES = { "arbitrage": "buys and sells at the grid against a day-ahead price", "safety": "'stopped' is the site's switch, 'inputs missing' is the " "site's own meter - neither is about a pack", + # SAFETY-31, and it is the entry most easily filed on the wrong side. + "circulation": "sum(|Bi|) - |sum(Bi)| is defined OVER THE ROSTER and is " + "identically zero for any one unit - there is no fact of " + "the matter about which of two opposing inverters is the " + "offender, so there is nothing to file it per-unit on", } UNIT_SOURCES = { "device": "one inverter's own reported min/max rating", @@ -172,9 +177,9 @@ for _src, _why in sorted(SITE_SOURCES.items()): print(f" site {_src:<11} {_why}") for _src, _why in sorted(UNIT_SOURCES.items()): print(f" unit {_src:<11} {_why}") -check("all eight claim sources are classified, and only those eight", +check("all NINE claim sources are classified, and only those nine", set(CLAIM_SCOPE) == set(SITE_SOURCES) | set(UNIT_SOURCES)) -check("the four that are about the GRID CONNECTION are site claims", +check("the five that are about the GRID CONNECTION are site claims", {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_SITE} == set(SITE_SOURCES)) check("the four that are about a PACK, a DEVICE or one write are per-inverter", {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_UNIT} == set(UNIT_SOURCES)) @@ -195,7 +200,7 @@ check("with one inverter every source routes to the site, per-inverter ones " check("...and with none configured at all, likewise", all(actuator_for(s, "inverter-a", ()) == ACTUATOR_SITE for s in CLAIM_SCOPE)) check("with TWO the identity breaks and the split becomes real: the four " - "per-inverter sources route to the unit this controller drives, the four " + "per-inverter sources route to the unit this controller drives, the five " "site sources stay on the grid connection", {s: actuator_for(s, "inverter-a", TWO) for s in CLAIM_SCOPE} == {**{s: "inverter-a" for s in UNIT_SOURCES}, diff --git a/goodwe_controller/test_peakguard.py b/goodwe_controller/test_peakguard.py index 6c1e8c0..44e27a6 100644 --- a/goodwe_controller/test_peakguard.py +++ b/goodwe_controller/test_peakguard.py @@ -2358,8 +2358,17 @@ check("...and so does the persistent notification, which outlives the page and " # "deferred" is exactly the state in which a ninth source gets built by # accident. Its revisit trigger is ALLOC-01, under which the reserve becomes a # PER-INVERTER interval at the allocator rather than a ninth claim here. +# ⚠️ SAFETY-31 ADDED THE NINTH, DELIBERATELY, AND THIS IS THE PARAGRAPH THAT +# SENTENCE ASKS FOR. `circulation` is a SET at P_SAFETY on the SITE actuator, +# filed only while `circ_wd.tripped`. It is not the deferred loop-side reserve +# this tripwire was built to catch: it is not conditioned on a state of charge, +# it does not shape the loop, and it appears in NO cycle of the sweeps below - +# the dynamic halves of this part still observe exactly {device, supervised, +# loop} at every SoC from 0 to 100 %, because nothing in those fixtures +# circulates. What it does is stop the site when two inverters are measured +# pushing against each other. See DECISIONS/SAFETY-31-circulating-power.md. CLAIM_SOURCES = {"device", "supervised", "failsafe", "safety", "peak", - "arbitrage", "maintenance", "loop"} + "arbitrage", "maintenance", "loop", "circulation"} # The two appends that file a claim BUILT ELSEWHERE in the method. Named here so # that a THIRD such indirection reads as an unrecognised append rather than as a # silent gap in the scan. @@ -2400,7 +2409,7 @@ check("every `claims.append(...)` in main.py's cycle is one this scan can READ " "- a claim filed through a shape it cannot parse has to fail here rather " "than be counted as nothing, which is how a scan like this goes quietly " "vacuous", len(_appends) >= 10 and None not in _filed) -check("...and the sources that cycle can file are EXACTLY these eight. THIS is " +check("...and the sources that cycle can file are EXACTLY these nine. THIS is " "what makes part 10 a tripwire: replay_loop() hand-builds the claim list, " "so it cannot see a claim main.py adds - and a loop-side SoC reserve is a " "NINTH source. Filing one takes this line red on the line it is written", @@ -2530,19 +2539,21 @@ check("...and replay_loop() folding main.py's TWO P_SAFETY rating limits into " # structurally impossible. The structure is `CLAIM_SCOPE`, so `CLAIM_SCOPE` is # pinned to the AST-scanned set rather than trusted to agree with it. check("every source main.py can file is CLASSIFIED site or per-inverter, and " - "nothing else is. A ninth source now goes red twice - once here for " - "having no classification, once above for being a ninth - and it takes an " + "nothing else is. A TENTH source now goes red twice - once here for " + "having no classification, once above for being a tenth - and it takes an " "argument about what the claim is ABOUT to make either line green", set(CLAIM_SCOPE) == CLAIM_SOURCES) print(" site: " + ", ".join(sorted(s for s, v in CLAIM_SCOPE.items() if v == SCOPE_SITE)) + " | per-inverter: " + ", ".join(sorted(s for s, v in CLAIM_SCOPE.items() if v == SCOPE_UNIT))) -check("...and the split is the one ARB-03 argued: the four about the GRID " - "CONNECTION are site claims and the four about a pack, a device or one " - "write are per-inverter", +check("...and the split is the one ARB-03 argued, with SAFETY-31's addition: " + "the five about the GRID CONNECTION are site claims and the four about a " + "pack, a device or one write are per-inverter. `circulation` is a site " + "claim because its quantity is defined over the roster and is identically " + "zero for any one unit - there is no inverter to attribute it to", {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_SITE} - == {"loop", "peak", "arbitrage", "safety"} + == {"loop", "peak", "arbitrage", "safety", "circulation"} and {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_UNIT} == {"device", "supervised", "failsafe", "maintenance"}) @@ -2618,16 +2629,19 @@ _restore_b() print(" with a second inverter on the roster, main.py routes: " + ", ".join(f"{s}->{a}" for s, a in sorted(_routed.items()))) check("⚠️ AND THIS IS THE BREAKING HALF. Give the same Controller a second " - "inverter and its OWN routing method splits the eight: `device`, " + "inverter and its OWN routing method splits the nine: `device`, " "`supervised`, `failsafe` and `maintenance` bind the unit this controller " - "drives, `loop`, `peak`, `arbitrage` and `safety` stay on the grid " - "connection. Swap any one of them - route `peak` to the unit and the site " - "keeps no floor; route `maintenance` to the site and it can cross that " - "floor into `lo > hi` - and this line goes red", + "drives, `loop`, `peak`, `arbitrage`, `safety` and SAFETY-31's " + "`circulation` stay on the grid connection. Swap any one of them - route " + "`peak` to the unit and the site keeps no floor; route `maintenance` to " + "the site and it can cross that floor into `lo > hi`; route `circulation` " + "to one unit and a site-wide loop parks the inverter that may well be the " + "one still listening - and this line goes red", _routed == {"device": "inverter-a", "supervised": "inverter-a", "failsafe": "inverter-a", "maintenance": "inverter-a", "loop": ACTUATOR_SITE, "peak": ACTUATOR_SITE, - "arbitrage": ACTUATOR_SITE, "safety": ACTUATOR_SITE}) + "arbitrage": ACTUATOR_SITE, "safety": ACTUATOR_SITE, + "circulation": ACTUATOR_SITE}) check("...and the roster it was restored to is the shipped one, so nothing " "below inherits a two-inverter Controller", c_src.units == (app_main.UNIT_A,) and len(c_src.inverters) == 1) @@ -2636,7 +2650,7 @@ check("...and the roster it was restored to is the shipped one, so nothing " # IT RIGHT. `actuator_of("maintenance")` written at the `device` append routes a # rating limit to whatever `maintenance` routes to, and at the shipped # one-inverter roster BOTH answers are the site, so nothing above can see it. -# ⚠️ AND THE DYNAMIC CYCLE BELOW SEES ONLY THREE OF THE EIGHT DIRECT APPENDS. +# ⚠️ AND THE DYNAMIC CYCLE BELOW SEES ONLY THREE OF THE NINE DIRECT APPENDS. # That was THIS PART committing this repository's own named failure - a check # observing something other than what it claimed - inside the check written to # prevent it. The first cut of 10g called the two-inverter cycle "the line a @@ -2659,7 +2673,7 @@ check("...and the roster it was restored to is the shipped one, so nothing " # actuator as a LITERAL - `actuator=self.actuator_of('')` - so the # literal is simply required to EQUAL the source the claim is filed under. That # is what makes "a mis-typed `actuator_of(...)` goes red" a true sentence about -# all eight rather than about the three a fixture happens to reach. +# all nine rather than about the three a fixture happens to reach. def append_actuator(node): @@ -2677,7 +2691,7 @@ def append_actuator(node): return ast.unparse(kw[0].value) -# The appends that BUILD their claim in place - the eight whose routing is +# The appends that BUILD their claim in place - the nine whose routing is # chosen by hand, and the only ones where a typo is possible - against the two # that append a claim built elsewhere in the method and carry no `actuator=` # here at all. @@ -2708,19 +2722,19 @@ _unit_sources_unrouted = [ if CLAIM_SCOPE.get(r[0]) == SCOPE_UNIT and "," not in r[1]] print(f" {len(_direct)} direct appends, each read at its own line: " + ", ".join(f"main.py:{ln} {s} -> {a}" for s, a, ln in _routing)) -check("all eight of `Controller.cycle()`'s DIRECT appends route through " +check("all NINE of `Controller.cycle()`'s DIRECT appends route through " "`self.actuator_of()`, so this scan can read the routing " "of EVERY one of them. An append that computed its actuator, or omitted " "the argument, or hand-wrote a string, fails HERE rather than being " "silently skipped - which is how a scan like this goes quietly vacuous, " "and 10f is in this file because that already happened once", - len(_direct) == 8 + len(_direct) == 9 and all(a is not None and a.startswith("self.actuator_of('") for _s, a, _ln in _routing)) -check("⚠️ AND AT EVERY ONE OF THE EIGHT THE LITERAL IT ROUTES THROUGH IS THE " +check("⚠️ AND AT EVERY ONE OF THE NINE THE LITERAL IT ROUTES THROUGH IS THE " "SOURCE THE CLAIM IS FILED UNDER. THIS is the line a mis-typed " - "`actuator_of(...)` at a call site goes red on - all eight of them, on " - "the line the typo is written, at every roster, INCLUDING the five " + "`actuator_of(...)` at a call site goes red on - all nine of them, on " + "the line the typo is written, at every roster, INCLUDING the six " "appends no fixture in this repository reaches. " "`actuator_of(\"peak\")` at the maintenance append is a different string " "here even though it is the same VALUE at the shipped roster of one", @@ -2731,7 +2745,7 @@ check("⚠️ AND EVERY PER-INVERTER APPEND NAMES THE INVERTER IT IS ABOUT. " "site-scoped sources and is the whole defect for a per-inverter one: a " "`device` limit built inside the roster loop and routed without `inv` " "files inverter B's rating on inverter A, where it binds the wrong " - "share. Five of the eight appends are SCOPE_UNIT - `device`, " + "share. Five of the nine appends are SCOPE_UNIT - `device`, " "`supervised`, `failsafe` and maintenance's two - and all five pass one", _unit_sources_unrouted == [] and len([r for r in _routing @@ -2744,7 +2758,7 @@ check("...and the only appends this pin does NOT reach are the two that file a " "arbitrage.claim() by test_arbitrage.py", sorted(_indirect) == sorted(CLAIM_BUILDERS) and len(_direct) + len(_indirect) == len(_appends)) -check("...and between them the ten appends account for exactly the eight " +check("...and between them the eleven appends account for exactly the nine " "sources 10f pins, so the routing pin and the source set are one scan " "and neither can drift from the other", {s for s, _a, _ln in _routing} | set(CLAIM_BUILDERS.values()) @@ -2787,11 +2801,13 @@ check("⚠️ AND THE RATING LIMITS ARE FILED ONCE PER INVERTER, ON ITS OWN " ("loop", ACTUATOR_SITE), ("supervised", "inverter-a"), ("supervised", "inverter-b")]) check("...and naming those three is not an accident of this fixture but the " - "reason the static pin exists - five of the eight sources need a state " - "no check in this file reaches", + "reason the static pin exists - six of the nine sources need a state " + "no check in this file reaches. SAFETY-31's `circulation` is one of the " + "six: it is filed only while the site is MEASURED to be circulating, " + "which no fixture here is", {c.source for c in _cl2} == CLAIM_SOURCES - {"failsafe", "safety", "maintenance", "peak", - "arbitrage"}) + "arbitrage", "circulation"}) # ⚠️ AND NOW THE HALF THAT WAS EVIDENCE-SHAPED AND OBSERVED NOTHING. The three # checks above grade WHICH ACTUATOR each claim lands on. They say nothing about # WHOSE NUMBERS the bound is built from - and those are two different defects diff --git a/goodwe_controller/test_watchdog.py b/goodwe_controller/test_watchdog.py index 2b76ec4..6e7fdad 100644 --- a/goodwe_controller/test_watchdog.py +++ b/goodwe_controller/test_watchdog.py @@ -537,6 +537,574 @@ check("a crash between the two writes leaves the REASON on disk", check("...and never a latch with nothing to explain it", not crashed_store.data.get(TRIPPED_KEY)) + +# ============================================================================= +print() +print("=" * 72) +print("SAFETY-31: circulating power, which nets to zero at the meter") +print("=" * 72) +# ⚠️ READ THIS BEFORE ADDING A CHECK HERE. Half of this card was already built +# and it is NOT the half below. ALLOC-01 shipped the SIGN-COHERENCE GATE +# (`allocator.REFUSE_SIGN`, graded by test_alloc.py part 6): an allocation whose +# shares carry two signs is REFUSED, both units named, no shares returned. So +# the allocator can no longer COMMAND circulating power, and nothing here +# re-tests that. +# +# What is graded here is the MEASUREMENT half: circulating power the allocator +# NEVER COMMANDED. The two are complements, not duplicates - the gate refuses an +# allocation before it is written and cannot see one millisecond past that; this +# reads the hardware and does not care what was written. + +from app.inverter import LINK_STALE_S # noqa: E402 +from app.circulation import ( # noqa: E402 + CIRC_OPTION_DEFAULTS, CirculationConfig, CirculationWatchdog, + circulating_w, opposing, +) + +print() +print("1. the sensor: sum(|Bi|) - |sum(Bi)| over MEASURED per-unit power") +check("both units discharging: the triangle equality holds and it is ZERO. " + "This is the ordinary case and the sensor must be silent in it", + circulating_w([1000.0, 2000.0]) == 0.0) +check("...both charging, likewise", circulating_w([-1000.0, -2000.0]) == 0.0) +check("...and a unit resting at exactly zero beside a working one is not " + "opposition", circulating_w([0.0, 2500.0]) == 0.0) +# ⚠️ ASSERTED AGAINST A LITERAL, NOT AGAINST `2 * min(abs(a), abs(b))`. +# MUTATION.md's rule 2: an assertion written in terms of the expression under +# test cannot tell "the right formula" from "the same wrong formula twice". +check("A charging 3000 W while B discharges 3000 W reads 6000 W - the card's " + "own scenario, and the figure is 2*min(|A|,|B|), the power going round " + "the loop through BOTH conversion stages", + circulating_w([-3000.0, 3000.0]) == 6000.0) +check("...unequal opposition is bounded by the smaller unit: -3000 against " + "+1000 is 2000 W, not 4000", + circulating_w([-3000.0, 1000.0]) == 2000.0) +print(f" [-3000, +3000] -> {circulating_w([-3000.0, 3000.0]):.0f} W " + f"[-3000, +1000] -> {circulating_w([-3000.0, 1000.0]):.0f} W " + f"[+1000, +2000] -> {circulating_w([1000.0, 2000.0]):.0f} W") +check("three units, one out of step: it is defined over the whole roster, not " + "over a pair", circulating_w([-2000.0, 1000.0, 1000.0]) == 4000.0) + +print() +print(" AC 5: at N=1 it is IDENTICALLY zero, for every reading") +# ⚠️ SWEPT, NOT SAMPLED. "Identically zero" is a claim about EVERY value, and +# one call with one number cannot distinguish it from "zero for this number". +_n1 = [circulating_w([v]) for v in + (0.0, 1.0, -1.0, 2500.0, -2500.0, 5000.0, -5000.0, 1e9, -1e9)] +check("nine readings from -1e9 to +1e9 W, every one of them 0.0", + set(_n1) == {0.0}) +check("...and an EMPTY roster is zero too, not a crash - `Controller.publish()` " + "calls this before anything has been read", + circulating_w([]) == 0.0) +check("⚠️ AND A MISSING READING AT N=1 IS STILL ZERO, WHICH IS THE OPPOSITE OF " + "THE N>=2 RULE BELOW AND IS DELIBERATE. With one term the algebra has no " + "room for an unknown: |B| - |B| is 0 for every B, INCLUDING one nobody " + "read. Returning None here would park the shipped one-inverter install in " + "a permanent 'cannot tell' on a quantity that cannot be anything else", + circulating_w([None]) == 0.0) + +print() +print(" a missing reading at N>=2 is UNKNOWN, and never zero") +check("⚠️ None, NOT 0.0. Zero would be this sensor reporting HEALTH FROM AN " + "ABSENCE - which is the 2026-09-02 failure itself, where a subsystem " + "stopped answering and its silence read as calm for 6 h 25 m", + circulating_w([None, 3000.0]) is None + and circulating_w([-3000.0, None]) is None + and circulating_w([None, None]) is None) + +print() +print(" it is NON-NEGATIVE, including where floating point would say otherwise") +_fp = [circulating_w([0.1, 0.2, 0.3]), circulating_w([1e-9, 2e-9]), + circulating_w([-0.1, -0.2, -0.30000000000000004])] +print(f" {_fp}") +check("a same-sign roster never publishes a negative number. ⚠️ THIS GRADES " + "THE PROPERTY, NOT THE `max(0.0, ...)` GUARD THAT DEFENDS IT, and the " + "difference is stated so a mutation sweep does not report a gap that is " + "not one: for a same-sign roster the two sums accumulate the same " + "magnitudes in the same order and are bit-identical, so the guard is an " + "EQUIVALENT MUTANT - searched over 700 000 random rosters without a " + "single negative. circulation.py says so at the line", + all(v >= 0.0 for v in _fp)) + +print() +print("2. CFG-03: config.yaml is the source of truth and there is ONE copy") +# ⚠️ READ OFF THE FILE, NOT TRANSCRIBED. The Dockerfile does not put config.yaml +# in the image, so the running add-on cannot read it and `CIRC_OPTION_DEFAULTS` +# has to exist - which makes it a place to drift. This is the same pin +# test_maintenance.py puts on MAINT_OPTION_DEFAULTS, for the same reason. +import re as _re # noqa: E402 +from pathlib import Path as _Path # noqa: E402 + +_cfg_txt = (_Path(__file__).resolve().parent / "config.yaml").read_text(encoding="utf-8") +# ⚠️ SPLIT ON THE TOP-LEVEL KEY, NOT ON THE WORD. "schema:" appears inside the +# CFG-03 comment above these very options ("options:, schema:, ..."), and +# splitting on the bare word cut the file in the middle of a comment - the +# options block then contained no options, the regex found none, and this pin +# compared an EMPTY dict against an empty expectation. That is MUTATION.md's +# third failure mode exactly: the assertion ran, the sentence was true, and the +# data could not reach the failure it names. Caught by the printed line. +_cfg_opts, _cfg_schema = _cfg_txt.split("\nschema:", 1) +_from_yaml = {k: int(v) for k, v in + _re.findall(r"^ (circulation_\w+):\s*(\d+)\s*$", _cfg_opts, _re.M)} +print(f" config.yaml options: {_from_yaml}") +print(f" CIRC_OPTION_DEFAULTS: {CIRC_OPTION_DEFAULTS}") +check("both options are declared in config.yaml's options: block and the " + "in-code mirror agrees with it key for key and value for value", + _from_yaml == CIRC_OPTION_DEFAULTS and len(_from_yaml) == 2) +_bounds = dict(_re.findall(r"^ (circulation_\w+):\s*int\((\d+),(\d+)\)\s*$", + _cfg_schema, _re.M) + and [(m[0], (int(m[1]), int(m[2]))) for m in + _re.findall(r"^ (circulation_\w+):\s*int\((\d+),(\d+)\)\s*$", + _cfg_schema, _re.M)]) +print(f" schema bounds: {_bounds}") +check("...and BOTH have a schema range, so the Supervisor refuses an " + "out-of-range value rather than the add-on discovering it at runtime", + set(_bounds) == set(CIRC_OPTION_DEFAULTS)) +check("⚠️ AND THE THRESHOLD'S FLOOR IS THE MEASURED KNEE, NOT A ROUND NUMBER. " + "On the TEST-19 rig the worst CONTINUOUS opposition a site where NOTHING " + "is faulty can produce is 5.0 s at 400 W, 10.0 s at 300 W, 40.0 s at " + "250 W and 505.0 s at 100 W. Below the knee, ordinary operation holds " + "this sensor over the threshold for MINUTES and no window shorter than " + "the 900 s billing quarter could tell that from a fault - so a " + "configurable value under 400 W would be a false-alarm generator, and " + "the schema does not offer one", + _bounds["circulation_w"][0] == 400) +check("...and the window's floor is the firmware's own veto window, " + "`inverter.LINK_STALE_S`, because below it a unit still executing a " + "stale share is a LINK fault the add-on already handles by dropping it " + "from the split - and its ceiling is the 900 s quarter that bills the " + "capacity tariff, inside which this must be caught and acted on", + _bounds["circulation_hold_s"] == (int(LINK_STALE_S), 900)) +check("the shipped defaults sit inside their own schema ranges", + all(_bounds[k][0] <= v <= _bounds[k][1] + for k, v in CIRC_OPTION_DEFAULTS.items())) +check("⚠️ AND NEITHER IS A ROUND NUMBER, WHICH IS AC 3 READ LITERALLY. 700 W " + "is 1.75x the measured knee - the headroom is bought for inverter B's " + "ramp and lag being PLACEHOLDERS until SPIKE-04 measures them - and 75 s " + "is 15x the worst legitimate episode and 2.5x the firmware veto. 60, 100, " + "120, 180, 300, 500 and 1000 are all values a guess would have landed on", + CIRC_OPTION_DEFAULTS["circulation_w"] not in (100, 500, 1000, 2000) + and CIRC_OPTION_DEFAULTS["circulation_hold_s"] not in (30, 60, 120, 180, 300)) + +print() +print("3. the trip: sustained opposition fires, the slew transient does not") + + +def circ_drive(series, cfg=None, step=5.0): + """Feed a list of circulating figures, one per control cycle.""" + wd = CirculationWatchdog(cfg or CirculationConfig()) + at = None + for i, v in enumerate(series): + if wd.evaluate(v, T0 + timedelta(seconds=i * step)) and at is None: + at = i * step + return wd, at + + +_CFG = CirculationConfig() +_sustained, _at = circ_drive([2000.0] * 40) +check("2000 W of measured opposition, held, trips", _sustained.tripped) +check(f"...and at {_CFG.hold_s:.0f} s, not before - the window is a window, " + "not a delay somebody rounded", _at == _CFG.hold_s) +check("the reason gives the operator the figure, the peak and the duration, " + "and says in words why nothing else reported it", + "2000" in _sustained.reason and "75 s" in _sustained.reason + and "nets to zero" in _sustained.reason.lower()) +print(f" e.g. {_sustained.reason!r}") + +print() +print(" ⚠️ AND NOT ON THE TRANSIENT, WHICH IS THE FALSE-ALARM CASE") +# ⚠️ THE SHAPES BELOW ARE THE RIG'S, NOT INVENTED. Measured on TEST-19 over 924 +# closed-loop runs in which every unit executed the share it was given: the +# worst continuous opposition above 700 W is 5.0 s - ONE control cycle - at a +# peak of 1700 W. That is the serialised write path: on a reversing cycle B's +# new number lands up to a whole cycle after A's, both slew through zero at +# 800 W/s behind a 10 s lag, and between the two crossings they genuinely +# oppose. A guard that trips on that is a false-alarm generator, and this +# project's doctrine on crying wolf is explicit (SAFETY-15, SAFETY-19). +_one_cycle = [0.0] * 5 + [1700.0] + [0.0] * 30 +_t1, _ = circ_drive(_one_cycle) +check("the worst single transient the rig can produce - 1700 W for ONE control " + "cycle - does not trip", not _t1.tripped) +_repeated = ([0.0] * 3 + [1700.0]) * 20 +_t2, _ = circ_drive(_repeated) +check("...and neither does it REPEATED every fourth cycle for 400 s, which is " + "a load flapping across zero all afternoon. The window clears between " + "them, so nothing accumulates", not _t2.tripped) +_almost = [1500.0] * 14 + [0.0] + [1500.0] * 14 +_t3, _ = circ_drive(_almost) +check("...and opposition that stands for 70 s, clears for one cycle, then " + "stands again for 70 s does NOT trip: 140 s of trouble, but never 75 s " + "CONTINUOUSLY, which is what the guard actually claims to measure", + not _t3.tripped) +_below = [_CFG.min_w] * 60 +_t4, _ = circ_drive(_below) +check(f"...and exactly {_CFG.min_w:.0f} W forever does not trip - the " + "comparison is strictly greater, so the threshold is a threshold rather " + "than an off-by-one", not _t4.tripped) +_just_over, _at5 = circ_drive([_CFG.min_w + 1.0] * 40) +check("...while one watt above it does, held long enough. The pair is what " + "makes the boundary a measured line rather than a direction", + _just_over.tripped and _at5 == _CFG.hold_s) + +print() +print(" a MISSING reading clears the window rather than accumulating it") +_gap, _ = circ_drive([2000.0] * 10 + [None] + [2000.0] * 10) +check("⚠️ SAFETY-02 MAKES THE SAME DECISION FOR A STALE METER, AND FOR THE SAME " + "REASON: a frozen figure held against a moving one manufactures a fault " + "in about a window's worth of cycles. 50 s over, one unknown, 50 s over " + "again is not 100 s of opposition", not _gap.tripped) + +print() +print(" the latch: it does not re-evaluate, because the trip removes the evidence") +_latched, _ = circ_drive([2000.0] * 40 + [0.0] * 40) +check("still tripped after the opposition has gone - and the opposition ALWAYS " + "goes, because the trip commands 0 W everywhere and two inverters at zero " + "cannot oppose. A non-latching version would de-assert and start the " + "whole thing again, three cycles at a time, forever", _latched.tripped) +_r = CirculationWatchdog(_CFG, tripped=True, reason="restored from the store") +check("...and one restored as tripped stays tripped without being re-driven", + _r.evaluate(0.0, T0) and _r.reason == "restored from the store") +_r.reset() +check("...until it is reset, which is the operator's press and nothing else", + not _r.tripped and _r.reason == "" and _r.held_s(T0) == 0.0) + +print() +print(" `opposing()` names the two sides, for the sentence the operator reads") +check("indices of who is discharging and who is charging", + opposing([-3000.0, 3000.0]) == ([1], [0])) +check("...and a unit idling below the floor is not named as a participant in " + "somebody else's loop", opposing([-3000.0, 3000.0, 3.0], min_w=350.0) + == ([1], [0])) + +print() +print("4. END TO END, AND THIS IS AC 2 - the sensor is MEASURED, so it sees") +print(" circulation the allocator NEVER COMMANDED") +# ⚠️ THIS IS THE CHECK MUTATION.md's THIRD FAILURE MODE IS ABOUT, so it is +# built the way that section says to build one. The claim is "X differs from Y" +# - the MEASURED sensor differs from a sensor computed off the allocation +# record - and X and Y are ACTUALLY MADE DIFFERENT here, with the difference +# asserted as its own precondition and PRINTED, before anything is concluded +# from it. DEV-02 shipped a check comparing a number against itself; ALLOC-01 +# shipped a generator that found 0 escapes where a wider one found 39 991. + +from app.inverter import Inverter # noqa: E402 + +# ⚠️ INVERTER B DOES NOT PHYSICALLY EXIST - it is ~10 days out and SPIKE-04 +# measures it. Its rating here is a HYPOTHESIS, named as one, exactly as +# test_peakguard.py's helper names its own. +B_RATING_HYPOTHESIS = 3000.0 + + +class _CircHass(_Hass): + """A house at rest, and two inverters MEASURED to be fighting each other. + + ⚠️ THE MEETER READS ZERO AND THAT IS THE WHOLE POINT. grid = load - sum(B), + and with A at -3000 W and B at +3000 W the sum is zero, so P1 reads exactly + what an idle site reads. Every existing guard in this add-on is a consumer + of that number. + """ + + def __init__(self, a_w=-3000.0, b_w=3000.0): + super().__init__() + self.a_w, self.b_w = a_w, b_w + + async def number(self, entity, invert=False): + return {"sensor.grid": 0.0, + "sensor.soc": 50.0, "sensor.soc_b": 50.0, + "sensor.batt": self.a_w, "sensor.batt_b": self.b_w}.get(entity) + + +def _two_inverter(hass, store, *, hold_s=1, min_w=700, **over): + opts = {"setpoint_entity": "number.sp", "meter_entity": "sensor.grid", + "soc_entity": "sensor.soc", "batt_entity": "sensor.batt", + "meter_source": "off", "max_w": 5000, "gain": 0.6, + "circulation_w": min_w, "circulation_hold_s": hold_s, + "auto_start": True} + opts.update(over) + c = Controller(opts, hass, store, _P(), stop=asyncio.Event()) + b = Inverter("inverter-b", setpoint_entity="number.sp_b", + soc_entity="sensor.soc_b", batt_entity="sensor.batt_b", + max_w=B_RATING_HYPOTHESIS, store=store) + b.dev_min, b.dev_max = -B_RATING_HYPOTHESIS, B_RATING_HYPOTHESIS + c.inverters.append(b) + c.auto = True + for inv in c.inverters: + inv.dev_min = inv.dev_min if inv.dev_min is not None else -5000.0 + inv.dev_max = inv.dev_max if inv.dev_max is not None else 5000.0 + return c + + +async def _circ_e2e(): + hass = _CircHass() + c = _two_inverter(hass, _MemStore()) + seen = [] + for _ in range(14): + await c.read_inputs() + await c.cycle() + seen.append(dict(circ=c.site_circulating_w, grid=c.grid, + target=c.target, alloc=c.allocation, + tripped=c.circ_wd.tripped)) + await asyncio.sleep(0.25) + return c, hass, seen + + +_c, _chass, _seen = asyncio.run(_circ_e2e()) +_first = _seen[0] +_alloc = _first["alloc"] +_shares = dict(_alloc.shares) +_meas = [inv.batt for inv in _c.inverters] +print(f" MEASURED per-inverter power : {dict(zip(_c.units, _meas))}") +print(f" ALLOCATION RECORD : ok={_alloc.ok} {_alloc.explain()}") +print(f" meter (P1) : {_first['grid']:+.0f} W") +print(f" sensor over MEASURED power : {circulating_w(_meas):.0f} W") +print(f" sensor over the RECORD : " + f"{circulating_w(list(_shares.values())):.0f} W") + +# --- THE PRECONDITION, AS ITS OWN CHECK ------------------------------------- +# ⚠️ MUTATION.md RULE 1. If a later edit makes the record and the measurement +# agree again, THIS goes red rather than the claim below going quietly vacuous. +check("PRECONDITION: the allocation record and the measured power are " + "ACTUALLY DIFFERENT in this fixture. The record is coherent - it is `ok`, " + "its shares carry one sign, and they sum to the site target - while the " + "hardware is measured at -3000 W and +3000 W. Without this line the two " + "claims below could both be true of a fixture in which nothing differed", + _alloc.ok is True + and len({(s > 0) - (s < 0) for s in _shares.values() if s}) <= 1 + and sorted(_meas) == [-3000.0, 3000.0] + and set(_shares.values()) != set(_meas)) +check("ALLOC-01's gate is SATISFIED here, and that is the point rather than an " + "accident: the allocator refused nothing, because it commanded nothing " + "wrong. `REFUSE_SIGN` cannot fire on a record whose shares agree in sign", + _alloc.refused is None) +check("⚠️ SO A SENSOR COMPUTED FROM THE ALLOCATION RECORD READS EXACTLY 0 W. " + "That is the cheap version of this card, and it is worth nothing: " + "ALLOC-01's sign-coherence gate already guarantees the record is " + "sign-coherent, so a record-derived sensor is zero BY CONSTRUCTION and " + "can only ever confirm that the allocator agrees with itself", + circulating_w(list(_shares.values())) == 0.0) +check("⚠️ AND THE SHIPPED SENSOR READS 6000 W ON THE SAME CYCLE, because it is " + "derived from `Inverter.batt` - each unit's OWN measurement. This is the " + "whole card: it detects circulation THE ALLOCATOR NEVER COMMANDED - a " + "unit ignoring or lagging its share, a stale share still being executed, " + "or HW-01's power stage running its last instruction for 6 h 25 m with " + "the comms dead and the record impeccable throughout", + _c.site_circulating_w == 6000.0) +check("...and the METER READS ZERO through all of it, which is why no existing " + "guard fires. SAFETY-02 gates on abs(grid_w) > 300 W and that gate never " + "opens; the peak guard and the capacity tariff both bill grid exchange", + _first["grid"] == 0.0 + and not any(i.sign_wd.tripped for i in _c.inverters)) + +print() +print(" ...and the trip ACTS: a site-wide stop, not a notification") +check("the watchdog tripped", _c.circ_wd.tripped) +check("the site is commanded 0 W", _c.target == 0.0) +check("...and 0 W actually reached BOTH inverters, not just the one somebody " + "decided was to blame", + _chass.writes and _chass.writes[-1] == 0.0 + and all(i.target == 0.0 for i in _c.inverters)) +check("...and it got there through the `circulation` claim, by name, rather " + "than by some other guard happening to stop the site", + "circulation" in _c.reason) +print(f" status line: {_c.reason!r}") +check("the status page says so in words, and says the meter cannot see it", + any("CIRCULATING POWER" in ck["text"] and "meter cannot see" in ck["text"] + for ck in _c.checks())) +check("the banner leads with it - above the sign failsafe, because a sign " + "disagreement is visible on every dashboard in the house and this is " + "visible nowhere at all", + "CIRCULATING POWER" in _c.status_payload()["banner"]) +_pub = {} +_c.mqtt = type("P", (), {"enabled": True, + "publish": lambda _s, v: _pub.update(v), + "close": lambda _s: None})() +_c.publish() +print(f" published: circulation={_pub.get('circulation')} " + f"grid={_pub.get('grid')} status={_pub.get('status')}") +check("⚠️ AND IT IS PUBLISHED AS ITS OWN SENSOR BESIDE THE GRID FIGURE. " + "`grid 0 W` and `circulating 6000 W` on the same dashboard is the whole " + "phenomenon in two rows, and NO arithmetic on the grid figure alone can " + "ever produce the second one", + _pub.get("circulation") == 6000 and _pub.get("grid") == 0.0) +check("...and the controller-status sensor reports the failsafe state", + _pub.get("status") == "failsafe") +# ⚠️ AND THE UNKNOWN CASE, WHICH A SWEEP FOUND UNGRADED. A per-unit reading that +# did not arrive makes the site figure None, and `MqttPublisher.publish()` DROPS +# None values - so the entity keeps its last figure and the dashboard shows a +# stale number rather than a fresh zero. Publishing `round(x or 0.0)` instead +# survived the first mutation battery: it turns "I do not know" into "0 W", i.e. +# health manufactured from an absence, on the one surface an operator actually +# watches. Same defect as the sensor's own None rule, one layer out. +_c.inverters[1].batt = None +_pub_unknown = {} +_c.mqtt = type("P", (), {"enabled": True, + "publish": lambda _s, v: _pub_unknown.update( + {k: x for k, x in v.items() if x is not None}), + "close": lambda _s: None})() +_c.publish() +print(f" with inverter-b's reading missing, publish() offers: " + f"circulation={_pub_unknown.get('circulation', '')!r}") +check("⚠️ WITH ONE UNIT'S READING MISSING THE SENSOR IS NOT PUBLISHED AT ALL, " + "so the entity holds its last figure instead of being handed a 0 that " + "would read as health. `publish()` drops None, and this is what makes " + "that drop deliberate rather than incidental", + "circulation" not in _pub_unknown) +_c.inverters[1].batt = 3000.0 + +print() +print("5. AC 5: at ONE inverter the sensor is identically zero and NOTHING changes") + + +async def _n1(): + hass = _CircHass(a_w=-3000.0, b_w=0.0) + opts = {"setpoint_entity": "number.sp", "meter_entity": "sensor.grid", + "soc_entity": "sensor.soc", "batt_entity": "sensor.batt", + "meter_source": "off", "max_w": 5000, "gain": 0.6, + "circulation_w": 700, "circulation_hold_s": 1, "auto_start": True} + c = Controller(opts, hass, _MemStore(), _P(), stop=asyncio.Event()) + c.auto = True + c.dev_min, c.dev_max = -5000.0, 5000.0 + for _ in range(14): + await c.read_inputs() + await c.cycle() + await asyncio.sleep(0.2) + return c + + +_c1 = asyncio.run(_n1()) +print(f" roster={_c1.units} measured batt={_c1.inverters[0].batt:+.0f} W " + f"sensor={_c1.site_circulating_w}") +check("the shipped roster of one, with the single inverter measured at " + "-3000 W - a magnitude that would trip instantly at N=2 - reads exactly " + "0.0 W", _c1.site_circulating_w == 0.0) +check("...so the watchdog never trips, over 14 cycles at a 1 s window", + not _c1.circ_wd.tripped) +check("...no `circulation` claim is ever filed, so the arbitration is " + "byte-identical to what shipped", "circulation" not in _c1.reason) +check("...and the status page says n/a rather than claiming to watch something " + "that cannot happen - an 'armed' row for an identically-zero quantity is " + "a guard advertising work it is not doing", + any("circulating power: n/a at one inverter" in ck["text"] + for ck in _c1.checks())) +_pub1 = {} +_c1.mqtt = type("P", (), {"enabled": True, + "publish": lambda _s, v: _pub1.update(v), + "close": lambda _s: None})() +_c1.publish() +check("...and it publishes 0, not None: the value is KNOWN at N=1, and a " + "dashboard row reading 0 W is the honest report of it", + _pub1.get("circulation") == 0) + +print() +print("6. SAFETY-34: the alarm survives a Home Assistant restart") +# ⚠️ THE LATCH IS NOT THE ALARM, AND THE 2026-09-02 OUTAGE IS WHY THAT SENTENCE +# IS HERE. `persistent_notification` is IN MEMORY: Home Assistant restarts and +# the card is gone while the fault runs on. That outage went unseen for 6 h 25 m +# with the detector correct throughout. So the durable half (the store) and the +# visible half (the notification) have to be joined, and this grades the join. + + +async def _circ_restart(): + shared = _MemStore() + h1 = _CircHass() + a = _two_inverter(h1, shared) + for _ in range(14): + await a.read_inputs() + await a.cycle() + await asyncio.sleep(0.25) + # THE RESTART. Same store, brand new Controller and Hass - which is exactly + # what a Home Assistant restart does to this add-on. The house has stopped + # circulating in the meantime, so nothing in the new process's own + # measurements could re-derive the trip. + h2 = _CircHass(a_w=0.0, b_w=0.0) + b = _two_inverter(h2, shared) + for _ in range(3): + await b.read_inputs() + await b.cycle() + await asyncio.sleep(0.05) + return a, b, shared, h1, h2 + + +_a, _b, _shared, _h1, _h2 = asyncio.run(_circ_restart()) +print(f" store after the trip: circ_tripped=" + f"{_shared.data.get('circ_tripped')!r} " + f"circ_reason={str(_shared.data.get('circ_reason'))[:48]!r}...") +check("the trip is written to /data - the add-on's only durable volume - as a " + "flag AND a reason", + _shared.data.get("circ_tripped") is True + and "CIRCULATING_POWER" in str(_shared.data.get("circ_reason"))) +check("⚠️ THE REBUILT CONTROLLER COMES UP TRIPPED, and its own measurements say " + "the site is quiet - 0 W on both units - so this can ONLY have come from " + "the store. A latch that lives in memory comes back armed, into the " + "fault, with no operator involved", + _b.circ_wd.tripped and _b.site_circulating_w == 0.0) +check("...carrying the reason with it, so the operator who has to decide " + "whether to clear it can read WHY", + "CIRCULATING_POWER" in _b.circ_wd.reason) +check("...and it is still commanding 0 W on the far side of the restart", + _b.target == 0.0 and "circulation" in _b.reason) + +_notes1 = [c for c in _h1.calls if c[0] == "persistent_notification"] +_notes2 = [c for c in _h2.calls if c[0] == "persistent_notification"] +print(f" notifications raised: {len(_notes1)} before the restart, " + f"{len(_notes2)} after it") +check("the alarm was raised in the first process", len(_notes1) >= 1) +check("⚠️ AND RE-RAISED IN THE SECOND ONE. THIS IS THE CARD'S THIRD " + "DELIVERABLE. `persistent_notification` does not survive a Home Assistant " + "restart, so a latch alone would leave a stopped site with NOTHING in " + "front of the operator - the detector correct, the state file correct, " + "and the screen blank. That is how 2026-09-02 went unseen for 6 h 25 m", + len(_notes2) >= 1) +check("...under a fixed notification_id of its own, so a second episode " + "REPLACES the first rather than stacking, and it is not confused with " + "the frozen meter, the clamp or the saturated peak guard", + all(c[2]["notification_id"] == "goodwe_circulating_power" + for c in _notes1 + _notes2) + and len({c[2]["notification_id"] for c in _notes1 + _notes2}) == 1) +_msg = _notes2[0][2]["message"] +check("...and the message tells the operator the three things only this alarm " + "can: that the meter cannot see it, that control has been stopped, and " + "that if the grid reading does not move afterwards one inverter is no " + "longer listening and a human at the isolator is the last actuator", + "METER CANNOT SEE" in _msg and "CONTROL HAS BEEN STOPPED" in _msg + and "isolator" in _msg) +check("...and it does not raise the same card on every cycle - a per-PROCESS " + "flag, deliberately not persisted, is what makes 'has a human been shown " + "this' a different question from 'is this site tripped'", + len(_notes2) == 1) + +print() +print(" the operator's press is the only way out, and it clears BOTH halves") + + +async def _circ_clear(): + shared = _MemStore() + c = _two_inverter(_CircHass(), shared) + for _ in range(14): + await c.read_inputs() + await c.cycle() + await asyncio.sleep(0.25) + was = c.circ_wd.tripped + await c.handle_action("auto_toggle") # stop + await c.handle_action("auto_toggle") # start - the deliberate press + return c, shared, was + + +_cc, _cstore, _was = asyncio.run(_circ_clear()) +check("it really was tripped before the press", _was) +check("a deliberate START clears the latch in memory and on disk together - " + "the condition ALWAYS goes away on its own, because the trip commands " + "0 W everywhere and two inverters at zero cannot oppose, so clearing on " + "the condition would clear on nothing", + not _cc.circ_wd.tripped + and _cstore.data.get("circ_tripped") is False + and _cstore.data.get("circ_reason") == "") +check("⚠️ AND THE 'HAS A HUMAN SEEN THIS' FLAG IS CLEARED WITH IT. Left " + "standing through a clear, the NEXT trip in this process would latch, " + "stop the site, and raise no notification at all", + _cc._circ_announced is False) + + print() if fails: print(f"{len(fails)} FAILED: {', '.join(fails)}") -- 2.54.0