diff --git a/goodwe_controller/CHANGELOG.md b/goodwe_controller/CHANGELOG.md index 98f5c1a..c1fb4c0 100644 --- a/goodwe_controller/CHANGELOG.md +++ b/goodwe_controller/CHANGELOG.md @@ -2,6 +2,78 @@ ## Unreleased +**Circulating power - two inverters pushing against each other - becomes a +measured quantity, a bounded one, and an alarm that survives a restart +(SAFETY-31).** Inverter A charging at 3 kW while inverter B discharges at 3 kW +nets ~0 W at P1: the site reads idle while the packs are cycled through two +round-trip conversion losses continuously. **Every guard in this add-on is a +consumer of that net figure**, so none of them can see it - SAFETY-02's sign +watchdog gates on `abs(grid_w) > 300` and the gate never opens, and both the +peak guard and the capacity tariff bill grid exchange. + +* **A site sensor, `Σ|Bᵢ| − |ΣBᵢ|`.** Zero when the units pull the same way, + `2 × min(|B_a|,|B_b|)` when they oppose. Published as + `sensor.goodwe_circulating_power` beside the grid figure, because *"grid 0 W, + circulating 6000 W"* is the whole phenomenon in two dashboard rows and no + arithmetic on the grid figure alone can produce the second one. +* **⚠️ Derived from MEASURED per-inverter power, never from the allocation + record, and that is the whole card.** ALLOC-01's sign-coherence gate + (`REFUSE_SIGN`, its AC 7) already refuses any allocation whose shares carry + two signs, so a record-derived sensor reads zero **by construction** and could + only ever confirm that the allocator agrees with itself. What this detects is + circulation **nobody commanded**: a unit ignoring or lagging its share, a + stale share still being executed, a maintenance or SAFETY-25 pin holding one + unit while the loop drives the other, or HW-01's power stage running its last + instruction for **6 h 25 m** with the comms dead and the allocation record + impeccable throughout. Verified by making one unit ignore its share on the + TEST-19 rig: the record stays `ok` with both shares at −1250 W - a + record-derived sensor reads **0 W** - while the shipped sensor reads + **2000 W**. `Inverter.batt` became per-instance in DEV-02; before that this + sensor could not have been written. +* **The trip stops the site, and the decision record says why that is an action + rather than theatre.** Sustained opposition files `Claim.set("circulation", + P_SAFETY, 0.0)` on the **site** actuator - every inverter to 0 W, latched. + HW-01 established that a wedged inverter cannot be commanded at all, so the + remedy is not aimed at it: a circulating loop needs **two** participants and + **one of them is this controller**, faithfully driving the listening sibling + into the one that stopped listening. Stopping halves the loop, and - the more + important half - **makes the residue visible at the meter**, where every + existing guard can finally see it. What it cannot do is stop the wedged unit; + the notification says so, and names the isolator as the last actuator. +* **The threshold and the window were measured, not chosen (AC 3).** 924 + closed-loop runs on the TEST-19 two-master rig in which *nothing* is faulty, + swept over the whole range the schema accepts for `gain` and `slew_w`. Worst + **continuous** legitimate opposition by threshold: **505.0 s at 100 W, 40.0 s + at 250 W, 10.0 s at 300 W, 5.0 s at 400 W and above**. The knee is 400 W and + it is the schema floor. `circulation_w: 700` is 1.75× it, the headroom bought + for inverter B's ramp and lag being **placeholders until SPIKE-04 measures + them**. `circulation_hold_s: 75` clears three bounds - 15× the worst + legitimate episode (5.0 s), 2.5× the firmware's own `LINK_STALE_S` veto, and + 12× inside the 900 s capacity-tariff quarter. **A momentary opposition while + one inverter slews through zero is legitimate** - the serialised write path + lands B's number up to a control cycle after A's - and the guard does not fire + on it, in any of the 924 runs. +* **The alarm survives a Home Assistant restart (SAFETY-34).** + `persistent_notification` is in memory, which is exactly how the 2026-09-02 + outage went unseen for 6 h 25 m with the detector correct throughout. The + latch is persisted to `/data` as flat, site-level keys - flat because the + quantity is defined over the roster and is identically zero for any one unit, + so there is no inverter to attribute it to - and the notification is + **re-raised on the first cycle after every start**, through the same code path + as a fresh trip. +* **At one inverter the sensor is identically zero and nothing changes (AC 5).** + `|B| − |B| == 0` for every reading, including one nobody took; no claim is + ever filed, the arbitration is byte-identical, and the status page says *n/a* + rather than claiming to watch something that cannot happen. +* **Verified by breaking it.** 18 mutations against the sensor, the window, the + latch, the claim, the store, the alarm, the routing table and both config + copies - **all red**, each on a named check; one equivalent mutant argued + rather than claimed; and one genuine **survivor found and closed** (`publish()` + turning an unknown into a published `0 W`). `arbiter.CLAIM_SCOPE` gains its + ninth source and `test_peakguard.py`'s 10f/10g tripwires were updated + deliberately, as their own comments require. Full ledger in + `DECISIONS/SAFETY-31-circulating-power.md`. + **Every per-device value stops being a Controller scalar: rating, SoC, battery power, watchdogs, heartbeat, write bookkeeping and store keys are now per-instance (DEV-02).** `app/inverter.py` introduces an `Inverter` object that diff --git a/goodwe_controller/DECISIONS/SAFETY-31-circulating-power.md b/goodwe_controller/DECISIONS/SAFETY-31-circulating-power.md new file mode 100644 index 0000000..01dec93 --- /dev/null +++ b/goodwe_controller/DECISIONS/SAFETY-31-circulating-power.md @@ -0,0 +1,316 @@ +# SAFETY-31 — what "acted on" means for circulating power, and what it cannot mean + +**Status: DECIDED 2026-09-04.** The trip **stops the site**: a `SET 0 W` claim +at `P_SAFETY` on the **site** actuator, latched, persisted, and re-announced +after a restart. This document records why that is an action rather than +theatre, given that [[HW-01]] established a wedged inverter cannot be commanded +at all — and it records the numbers the threshold and the window were read off, +because "establish the window from the rig, do not pick a round number" is an +acceptance criterion and a number without its measurement is a guess. + +Filed with SAFETY-31. Card `6a99e0022b6a6c8ca15dbe3e`. + +--- + +## 0. The two halves, and which one this card built + +Half of this card was already shipped, and it is not the half below. + +| | **ALLOC-01, AC 7 — the COMMAND half** | **SAFETY-31 — the MEASUREMENT half** | +|---|---|---| +| what it is | `allocator.REFUSE_SIGN` | `circulation.circulating_w()` + `CirculationWatchdog` | +| what it sees | the shares the allocator is about to hand out | each inverter's **own measured power** (`Inverter.batt`, register 0x0518) | +| what it does | **refuses** the allocation, names both units, returns no shares | **stops the site**, latches, alarms durably | +| what it covers | circulating power the allocator would have **commanded** | circulating power **nobody commanded** | +| graded by | `test_alloc.py` part 6 | `test_watchdog.py`, the SAFETY-31 section | + +**They are complements, not duplicates.** The gate refuses an allocation before +it is written and cannot see one millisecond past that. This card reads the +hardware and does not care what was written. Neither replaces the other, and +**a sensor built on the allocation record would collapse the second into the +first and detect nothing** — see §2. + +## 1. What "acted on" means + +> **The trip files `Claim.set("circulation", P_SAFETY, 0.0)` on the SITE +> actuator. Every inverter is commanded 0 W and held there until an operator +> starts control again.** + +### The objection, which is correct + +[[HW-01]] is on the record: on 2026-09-02 the comms subsystem failed and the +power stage went on executing its last instruction for **6 h 25 m**, through +every command this add-on sent it. If the cause of the circulation is a unit +that has stopped listening, **a command is not a remedy**, and shipping one as +though it were is theatre. + +### The answer + +**A circulating loop needs two participants.** In every cause on this card's +list — a unit ignoring its share, a unit lagging it, a stale share still being +executed, a maintenance or [[SAFETY-25]] pin holding one unit while the loop +drives the other, or HW-01's wedged power stage — **exactly one participant is +the unit that stopped listening, and the other one is this controller**, +faithfully commanding the sibling to a share computed to sum with a number the +wedged unit is not actually producing. + +Stopping the commanding half does two things a notification cannot: + +1. **It halves the loop.** The unit that *is* still listening stops being driven + into the one that is not. That is real energy and real cycles off a real + pack, and it needs no cooperation from the wedged unit to take effect. +2. **It makes the residue visible — and this is the more important half.** + Circulating power hides *because it nets to zero at the meter*. With this + controller's contribution removed, whatever the wedged unit is still doing + stops being cancelled and appears at P1 as a plain net flow, where + SAFETY-02's sign watchdog, the peak guard and the capacity tariff can all see + it. **The fault stops being invisible.** + +Measured on the [[TEST-19]] rig, and this is the number that makes the case: +in the card's own scenario — A at −3000 W, B at +3000 W, house load 0 W — the +meter reads **+0 W** and the shipped `SignWatchdog`'s predicate is **false on +every one of 30 cycles**, because it gates on `abs(grid_w) > 300` and the gate +never opens. + +### What it does not do, said plainly + +**It does not stop the wedged inverter. Nothing this add-on can send will.** If +the loop persists at the meter after this trip, the remaining actuator is a +human at the isolator, and the notification says so in those words. + +### Why the claim is SITE-scoped + +`arbiter.CLAIM_SCOPE` classifies `circulation` as `SCOPE_SITE`, beside `loop`, +`peak`, `arbitrage` and `safety`. The neighbouring `failsafe` entry looks like +it and is not: SAFETY-02 has site-wide *evidence* and a per-unit *conclusion*, +because it compares the meter against the command **this controller put on one +actuator**. + +Circulating power has no such conclusion available. It is +`Σ|Bᵢ| − |ΣBᵢ|` — defined over the whole roster, **identically zero for any one +unit** — and there is no fact of the matter about which of two opposing +inverters is the offender: *a unit ignoring its share* and *a sibling faithfully +executing a share computed to sum with it* produce **identical readings**. +Filing this per-unit would require inventing an attribution the measurement +cannot support, and would park one inverter while the other went on feeding the +loop — quite possibly parking the only one still listening. + +The same argument is why the latch is **flat** in the store (`circ_tripped`, +`circ_reason`) and not namespaced like SAFETY-02's. One site, one latch. + +## 2. Why the sensor is MEASURED, and why the cheap version is worthless + +The obvious cheap version computes the same formula over `Allocation.shares`. +It would be simpler, would never be `None`, and would be **worth nothing**, for +two independent reasons: + +1. **It cannot fire.** ALLOC-01's sign-coherence gate already refuses any + allocation whose shares carry two signs. A record-derived sensor is therefore + a sensor over a set that is **guaranteed sign-coherent**. It reads zero *by + construction*, forever. +2. **It is [[MUTATION.md]]'s third failure mode.** It could only ever confirm + that the allocator agrees with itself — a check whose data cannot reach the + failure it names. + +Verified rather than argued. On the rig, with inverter B wedged on a stale +discharge while PV drives the site to charge: + +``` + S = -2500 W shares a = -1250 W b = -1250 W <- record: ok, ONE sign, sums to S + MEASURED A = -1250 W B = +1000 W <- hardware + sensor over the RECORD : 0 W + sensor over MEASURED power : 2000 W -> trips at 190 s +``` + +And the card's own headline scenario, driven on the same rig — A at −3000 W, +B at +3000 W, house load 0 W: + +``` + METER (P1) : +0 W <- the site looks IDLE + sensor over MEASURED power : 6000 W -> trips at 100 s + SAFETY-02 SignWatchdog : predicate false on 30 of 30 cycles +``` + +The same pair is asserted in `test_watchdog.py` **with the precondition written +as its own check and printed**, per MUTATION.md's rule, and mutation **M10** +(the property rewritten to read `self.allocation.shares`) takes it **red**. + +## 3. The numbers, and where they come from + +Both were read off a curve produced on `admin/goodwe`'s `sim/` two-master rig, +used read-only: the add-on's own `compute()`, its own `allocator.plan()` and its +own `circulating_w()`, driving `plantsim.Plant` through the firmware's own +encoder, with the sensor input taken from each unit's holding register 0x0518. + +**924 closed-loop runs in which nothing is faulty** — every unit executes the +share it was given — over both families of stimulus that can produce legitimate +opposition (7 reversals × 5 write staggers, and 7 flap periods × 4 amplitudes), +each swept across the **whole range `config.yaml` accepts** for the two settings +that decide how fast the site can turn around: `gain` `float(0.05,1.0)` and +`slew_w` `int(50,5000)`. + +### Reproducing it + +The rig lives in `admin/goodwe` (`sim/`) and was used **read-only**; the harness +is not committed here because it imports that repo's `plantsim`/`frame_check`, +which are not present in this one or in CI. What it does is small enough to +restate: build `Site([Inverter.from_spec("A", INVERTER_A), +Inverter.from_spec("B", INVERTER_B)])`, close the loop with the add-on's own +`compute()` and `allocator.plan()`, put each share on its unit's wire through +`Inverter.command()` (which goes out through the firmware's real encoder), and +read the sensor from `inv.reg(0x0518)` — **the same register the live add-on's +`batt_entity` binds**, so the input is the one the shipped code gets. Point it +at a checkout with `GOODWE_ADDON_REPO`. ⚠️ MUTATION.md's note applies: those +harnesses write `__pycache__` into **both** trees, so purge both by hand before +grading a mutation through one. + +Every figure below was re-run against the final shipped `CirculationConfig` +(`min_w=700`, `hold_s=75`) rather than against the draft values they were first +measured at. + +### Worst CONTINUOUS legitimate opposition, by threshold + +| threshold | worst continuous | runs that ever cross it | +|---:|---:|---:| +| 100 W | **505.0 s** | 281 / 924 | +| 200 W | 445.0 s | 174 / 924 | +| 250 W | 40.0 s | 132 / 924 | +| 300 W | 10.0 s | 115 / 924 | +| **400 W** | **5.0 s** | 74 / 924 | +| 700 W | 5.0 s | 33 / 924 | +| 1000 W | 5.0 s | 16 / 924 | +| 1700 W | 0.0 s | 0 / 924 | + +****With the shipped `min_w=700 W` and `hold_s=75 s`, 0 of those 924 runs trip.** + +The knee is at 400 W and it is sharp.** Below ~250 W a flapping load holds the +sensor above the threshold for *minutes* at a time, and no window shorter than +the 900 s billing quarter could tell that from a fault. At 400 W and above the +worst legitimate episode collapses to **5.0 s — one control cycle — and stays +there**. The worst peak on any legitimate run is **1700 W**. + +That 5.0 s is the serialised write path doing exactly what it should: on a +reversing cycle inverter B's new number lands up to a whole control cycle after +A's, both then slew through zero at 800 W/s behind a 10 s lag, and **between the +two zero crossings they genuinely oppose**. Nothing is wrong. A guard that +tripped on it would be the fourth false-alarm generator of the week, and this +project's doctrine on crying wolf is explicit ([[SAFETY-15]], [[SAFETY-19]]). + +### `circulation_w: 700` — schema floor `400` + +`700` is **1.75× the measured knee**. The headroom is bought for one named +reason: **inverter B does not physically exist.** Its ramp and lag in the rig +are inverter A's, *copied*, carrying `measured=False`; [[SPIKE-04]] measures the +real ones. A slower or faster second unit moves the transient's shape, and the +margin above the knee is what absorbs that without re-opening this card. +**Revisit when SPIKE-04 lands, not before.** + +The **schema floor is the knee itself**, 400 W, so a legally configured install +cannot land in the 445–505 s regime. What the threshold costs: a circulating +loop smaller than 350 W per unit is not reported, which is deliberate — at that +scale the round-trip loss is a few watts. + +### `circulation_hold_s: 75` — schema range `30..900` + +Three bounds, and the value clears all three: + +* **lower, from the rig — 5.0 s.** The worst continuous legitimate opposition + above 700 W across all 924 runs. +* **lower, from the firmware — 30 s (`inverter.LINK_STALE_S`).** A unit still + executing a stale share for less than the firmware's own veto window is a + *link* fault, and the add-on already answers it: liveness drops the unit and + the survivors carry its share, with the record saying so. Firing inside that + window would raise a second, louder alarm about a state already being handled. +* **upper, from the bill — 900 s**, the quarter that sets the capacity tariff. + +**75 s** is fifteen control cycles: **15× the worst legitimate episode**, **2.5× +the firmware veto**, and **12× inside the quarter**. It is deliberately not 60 +or 120 — a round number here would be a guess wearing a measurement's clothes, +and a check in `test_watchdog.py` asserts it is none of the values a guess +lands on. + +## 4. Why the window cannot be defeated the way SAFETY-02's was + +`allocator.py` records the measurement that makes SAFETY-02 blind to the +broadcast fight: the sign disagreement **flips on alternate ticks**, so +`disagree_since` is cleared between every pair of them and the 180 s hold never +accumulates. + +Circulating power cannot do that. It is a **magnitude**, non-negative by +construction, and a sustained loop holds it high on *every* cycle rather than +every other one. An oscillation that genuinely alternated between opposing and +agreeing is not a circulating loop and should not be reported as one — which is +exactly the flapping-load case in §3, and it is why the window clears. + +## 5. SAFETY-34: the latch is not the alarm + +`persistent_notification` is **in memory**. Home Assistant restarts and the card +is gone while the fault runs on — which is precisely how the 2026-09-02 outage +went unseen for **6 h 25 m with the detector correct throughout**. + +So the durable half and the visible half are joined, and the join is +`Controller._circ_announced`: a **per-process** flag, deliberately *not* +persisted. The latch in `/data/state.json` says *"this site is tripped"*; the +flag says *"this process has put that in front of a human"*. A restart clears +the flag and keeps the latch, so **the notification is re-raised on the first +cycle after every start** — which is the first cycle after a Home Assistant +restart, because the add-on restarts with it. + +**One code path serves both** the fresh trip and the restart, on purpose: two +paths would be two chances for the restart one to rot unnoticed, and the restart +one is the case nobody exercises by hand. Mutations **M13** (flag initialised +from the latch) and **M18** (the call removed) both take it red. + +## 6. Mutation ledger + +Every run on a **freshly copied tree with no `__pycache__`, mounted `:ro`** into +the CI image — the structural immunity MUTATION.md records for CI and for +SAFETY-25 — so no stale `.pyc` could exist to be read. Byte deltas recorded; +**three are `+0 B`, the risky delta, and are marked**. + +| # | file | mutation | Δ | result | +|---|---|---|---|---| +| M1 | `circulation.py` | `sum(abs(v))` → `sum(v)` in the sensor | −5 B | **RED** — the 6000 W, 2000 W and three-unit checks | +| M2 | `circulation.py` | N≤1 returns `None` instead of `0.0` | +1 B | **RED** — AC 5's nine-reading sweep | +| M3 | `circulation.py` | a missing reading at N≥2 returns `0.0` | −1 B | **RED** — "health from an absence" | +| M4 | `circulation.py` | drop `max(0.0, ...)` | — | **EQUIVALENT, not run as a claim.** For a same-sign roster the two sums accumulate the same magnitudes in the same order and are bit-identical; 700 000 random rosters produced no negative. Argued at the line and in the check's own text. | +| M5 | `circulation.py` | threshold `<=` → `<` | −1 B | **RED** — "exactly 700 W forever does not trip" | +| M6 | `circulation.py` | window `<` → `<=` | +1 B | **RED** — "trips at 75 s, not before" | +| M7 | `circulation.py` | the window removed entirely | −85 B | **RED** — both slew-transient checks | +| M8 | `circulation.py` | `None` no longer clears the window | +34 B | **RED** — the stale-reading gap check | +| M9 | `circulation.py` | the latch removed | −123 B | **RED** — "restored as tripped stays tripped" | +| **M10** | `main.py` | **the sensor derived from `self.allocation.shares`** | +60 B | **RED** — AC 2's headline check. This is the card. | +| M11 | `main.py` | the `circulation` claim never filed | −235 B | **RED** — `test_watchdog.py` **and** `test_peakguard.py` 10f/10g | +| M12 | `main.py` | the latch not persisted to the store | −112 B | **RED** — all three durability checks | +| M13 | `main.py` | `_circ_announced` initialised from the latch | +21 B | **RED** — SAFETY-34's re-raise | +| M14 | `main.py` | the operator clear leaves `_circ_announced` set | −45 B | **RED** — "the next trip would raise nothing" | +| M15 | `arbiter.py` | `circulation` scoped `SCOPE_UNIT` | **+0 B** | **RED** — `test_arbiter.py` ×3 and `test_peakguard.py` ×3 | +| M16 | `config.yaml` | schema floor `400` → `100` | **+0 B** | **RED** — the measured-knee check | +| M17 | `circulation.py` | default `700` → `500` | **+0 B** | **RED** — the CFG-03 pin and the not-a-round-number check | +| M18 | `main.py` | `await self.check_circulating()` removed | −39 B | **RED** — all four notification checks | +| M19 | `main.py` | publish `round(x or 0.0)` instead of `None` | −68 B | **survivor on the first battery — GAP CLOSED**, then RED | + +**M19 is recorded as a survivor because it was one.** The first battery found no +check grading what `publish()` does when a per-unit reading is missing, so +turning *"I do not know"* into *"0 W"* on the one surface an operator actually +watches went unnoticed — the sensor's own `None` rule, defeated one layer out. +A check was added and the mutation re-run; it is red now. The run that found it +was purged, so it is a result and not an artefact. + +## 7. What this card does NOT close + +* **Inverter B's ramp and lag are placeholders.** Every transient number in §3 + was produced with inverter A's parameters copied onto B (`measured=False`). + SPIKE-04 measures the real ones and is the revisit trigger for the 1.75× + margin. **Do not present any figure here as a measurement of inverter B.** +* **A wedged inverter still cannot be stopped by this add-on.** §1 says so; the + notification says so; there is no software fix and this card does not pretend + to one. +* **A loop below ~350 W per unit is not reported.** Deliberate, §3. +* **The saturation freeze fires under a wedged unit and leaves a standing grid + error.** Observed on the rig while producing §2's numbers: with B wedged, the + loop's anti-windup test compares a site target against a measured site power + that cannot track it, freezes the integrator and leaves the meter parked at + −3750 W. Nothing alarms on that today — a freeze is an ordinary control action + by design. It is a real finding, it is **not** in scope here, and it wants a + card of its own. diff --git a/goodwe_controller/DOCS.md b/goodwe_controller/DOCS.md index db677ae..5ac4ac9 100644 --- a/goodwe_controller/DOCS.md +++ b/goodwe_controller/DOCS.md @@ -761,6 +761,82 @@ unchanged is 12 W, which is inside the 15 W deadband — so the command there is parked at 0 W and the two halves of the rule cannot both be true. At the schema floor of 60 s that figure is 60 W and only the margin is left. +### The circulating-power alarm (SAFETY-31) + +> **Only at two or more inverters.** At the shipped roster of one this quantity +> is identically zero for every reading, no claim is ever filed, and the status +> page says *n/a* rather than pretending to watch it. + +Two inverters can move a great deal of energy while the meter reads almost +nothing. Inverter A charging at 3 kW while inverter B discharges at 3 kW nets +**~0 W at P1**: the site looks idle, and in reality the packs are being cycled +through two round-trip conversion losses continuously, for no benefit. + +**Nothing else in this add-on can see it, and that is structural rather than an +oversight.** Every other guard is a consumer of the *net* figure at the grid +connection — the sign-disagreement failsafe gates on `abs(grid_w) > 300 W` and +that gate never opens at a netted zero; the capacity-peak guard and the tariff +both bill grid exchange. Circulating power is precisely the component of +per-inverter power that the sum destroys. + +So the add-on computes a quantity none of them were computing: + +``` +circulating_w = Σ|Bᵢ| − |ΣBᵢ| +``` + +Zero when the units pull the same way. `2 × min(|B_a|, |B_b|)` when they oppose +— twice the power going round the loop, which is what both conversion stages are +processing between them. It is published as +`sensor.goodwe_circulating_power`, deliberately **beside** the grid figure: +*"grid 0 W, circulating 6000 W"* is the whole fault in two dashboard rows, and +no arithmetic on the grid figure alone can produce the second one. + +**It is derived from each inverter's own measured power, not from what the +controller believed it commanded.** That matters because the allocator already +*refuses* to command opposite directions (ALLOC-01's sign-coherence gate), so a +figure computed from the allocation record would read zero by construction and +detect nothing. What this catches is circulation **nobody commanded** — a unit +ignoring or lagging its share, a stale share still being executed, or an +inverter doing something of its own accord. + +| option | default | meaning | +|---|---|---| +| `circulation_w` | 700 | Watts of measured opposition below which nothing is reported | +| `circulation_hold_s` | 75 | Seconds it must stand there continuously before the trip | + +**Why 700 W and 75 s, and why neither is a round number.** Both were read off +924 closed-loop runs on the two-master test rig in which *nothing* was faulty, +swept across the whole range the schema accepts for `gain` and `slew_w`. A +momentary opposition is **legitimate**: writes to the inverters are serialised, +so on a reversing cycle one unit gets its new number up to a control cycle +before its sibling, both slew through zero, and between the two crossings they +genuinely oppose. The worst *continuous* legitimate opposition, by threshold, +was **505 s at 100 W, 40 s at 250 W, 10 s at 300 W and 5.0 s — one control +cycle — at 400 W and above**. The knee is 400 W and it is the schema floor; +below it, ordinary operation would hold the sensor over the line for minutes and +no usable window could tell that from a fault. 700 W carries margin above the +knee because the second inverter's ramp and lag are not yet measured. 75 s is +15× the worst legitimate episode, 2.5× the firmware's own 30 s link-veto window, +and comfortably inside the 900 s quarter that bills the capacity tariff. + +**What happens when it trips, and what it honestly cannot do.** The site is +commanded **0 W on every inverter** and latched there until you press start. +That is not aimed at making a misbehaving inverter stop — if a unit has stopped +listening, no command will reach it. It does two things that do not need its +cooperation: it removes *this controller* as one of the two participants in the +loop, and it makes anything still happening **visible at the meter** instead of +cancelled at it, where the ordinary guards can see it. **If the grid reading +does not move after the trip, one inverter is no longer listening and the +remaining actuator is a human at the isolator.** + +**The alarm survives a Home Assistant restart.** The latch is written to `/data` +and the notification is re-raised on the first cycle after every start — +`persistent_notification` lives only in memory, and an alarm that vanishes with +a restart is how a real outage on this system went unseen for over six hours. +Restarting does **not** clear the trip; starting control from the add-on UI +does, and only after you have read why. + ### Energy price providers > **A fetched price moves energy only when you switch arbitrage on.** This @@ -985,6 +1061,6 @@ abort one. ## Status entities If an MQTT broker is available the add-on publishes setpoint, grid power, -battery power, state of charge, maintenance phase and controller status by MQTT -discovery. This is observability only — the controller works fine without a +battery power, circulating power, state of charge, maintenance phase and +controller status by MQTT discovery. This is observability only — the controller works fine without a broker, and MQTT problems can never affect control. diff --git a/goodwe_controller/app/arbiter.py b/goodwe_controller/app/arbiter.py index 04aed51..a1ebe42 100644 --- a/goodwe_controller/app/arbiter.py +++ b/goodwe_controller/app/arbiter.py @@ -132,6 +132,25 @@ CLAIM_SCOPE = { # controller regulates against. Both must stop every inverter, and a site # SET at P_SAFETY is how that is said once rather than N times. "safety": SCOPE_SITE, + # main.py: `Claim.set("circulation", P_SAFETY, 0.0, "circulating power")`. + # SAFETY-31, and this one is a SITE claim for a reason worth writing down, + # because the neighbouring `failsafe` entry looks like it and is not. + # + # `failsafe` is SAFETY-02: site-wide evidence, per-unit conclusion. It + # compares the meter against THE COMMAND THIS CONTROLLER PUT ON ONE + # ACTUATOR, so what it learns is about that actuator and the remedy is that + # actuator. + # + # Circulating power is not a property any single inverter has. It is + # `sum(|B_i|) - |sum(B_i)|`, defined over the whole roster and identically + # zero for any one unit, and there is NO FACT OF THE MATTER about which of + # two opposing inverters is the offender - a unit ignoring its share and a + # sibling faithfully executing a share computed to sum with it produce the + # same reading. Filing this per-unit would require inventing an attribution + # the measurement cannot support, and would park one inverter while the + # other went on feeding the loop. The evidence is site-level and so is the + # remedy: stop commanding, everywhere, and say why. + "circulation": SCOPE_SITE, # --- UNIT: about one pack, one inverter, one device ------------------- # main.py: `Claim.limit("device", P_SAFETY, self.dev_min, self.dev_max, diff --git a/goodwe_controller/app/circulation.py b/goodwe_controller/app/circulation.py new file mode 100644 index 0000000..9cbd1fc --- /dev/null +++ b/goodwe_controller/app/circulation.py @@ -0,0 +1,339 @@ +"""SAFETY-31 - the power two inverters move through each other, which the meter cannot see. + +Inverter A charging at 3 kW while inverter B discharges at 3 kW nets ~0 W at +P1. The site looks idle. In reality 3 kW is going round a loop through two +conversion stages continuously, for no benefit, and the wear on both packs is +real even though every instrument in this add-on says nothing is happening. + +WHY NOTHING EXISTING SEES IT, AND IT IS STRUCTURAL RATHER THAN AN OVERSIGHT +========================================================================== +Every consumer of the site measurement is a consumer of the NET. SAFETY-02's +sign-disagreement watchdog gates on `abs(grid_w) > cfg.min_w` (300 W by +default) - with the grid at ~0 that gate never opens, on any cycle, for any +duration. The peak guard bills the quarter-hour average AT THE GRID +CONNECTION. The capacity tariff bills grid exchange. `freeze.looks_frozen()` +grades P1 against the site rail. There is one meter, it reads the sum, and +circulating power is exactly the component of per-unit power that the sum +DESTROYS. So this cannot be fixed by tuning a threshold on any existing +detector; it needs a quantity none of them are computing. + +THE QUANTITY +============ + circulating_w = sum(|B_i|) - |sum(B_i)| + +Zero when every unit pulls the same way - then sum(|B_i|) == |sum(B_i)| by the +triangle equality. `2 * min(|B_a|, |B_b|)` at N=2 when they oppose, which is +twice the power going round the loop and is the right scale: that is what both +conversion stages are processing between them. It is non-negative always, it is +zero at N<=1 identically, and it needs no threshold to be interpretable - the +number IS the phenomenon in watts. + +⚠️ IT IS COMPUTED FROM MEASURED PER-INVERTER POWER, AND THAT IS THE WHOLE CARD. +============================================================================= +The obvious cheap version reads the allocation record - the shares the +allocator just handed out - and reports the same formula over those. It would +be simpler, it would never see a `None`, and it would be worth NOTHING, for two +independent reasons. + +The first is that it cannot fire. ALLOC-01 shipped the sign-coherence gate as +its AC 7 (`allocator.REFUSE_SIGN`, graded by `test_alloc.py` part 6): an +allocation whose shares carry two signs is REFUSED, both units named, no shares +returned. So a sensor over the allocation record is a sensor over a set the +allocator has already guaranteed is sign-coherent. It reads zero by +construction, forever, and a check written against it passes forever. + +The second is the one MUTATION.md names as the third way a check lies. A sensor +derived from the allocation record can only ever confirm THE ALLOCATOR AGREES +WITH ITSELF. Every fault in this card's list is a divergence between what was +commanded and what the hardware is doing: + + * a unit ignoring or lagging its share; + * a stale share still being executed by one inverter; + * SAFETY-25's deferred loop floor, or a maintenance drain, pinning one unit + while the loop drives the other; + * an inverter doing something of its own accord - and THIS HARDWARE HAS DONE + THAT TWICE. On 2026-09-02 the comms subsystem failed and the power stage + went on executing its last instruction for 6 h 25 m (HW-01). Through all of + it the allocation record was impeccable. + +In every one of those the command side is coherent and the MEASUREMENT side is +not. So the input is `Inverter.batt` - this unit's own power, read from its own +`batt_entity`, which on the live system is holding register 0x0518, the +inverter's own AC power. DEV-02 made that per-instance; before it there was one +`batt` scalar on the Controller and this sensor could not have been written at +all. + +⚠️ AND THIS IS THE MEASUREMENT HALF OF A PAIR. ALLOC-01's gate is the COMMAND +half: the allocator can no longer COMMAND circulating power. This module is the +detection half: circulating power the allocator never commanded. Neither one +replaces the other and neither is redundant - the gate refuses an allocation +before it is written, and cannot see a millisecond past that; this reads the +hardware and does not care what was written. Do not "unify" them. + +THE TRIP, AND WHY A THRESHOLD ALONE IS A FALSE-ALARM GENERATOR +============================================================== +A momentary opposition is LEGITIMATE. Writes to the inverters are serialised +(one `hass.call()` at a time - see `Inverter`'s docstring on what a hung write +to A costs B), so on a cycle where the site target reverses, one unit receives +its new command up to a whole control cycle before its sibling. Both then slew +through zero, with a real ramp rate and a real first-order lag, and BETWEEN THE +TWO ZERO CROSSINGS THE UNITS GENUINELY OPPOSE. Nothing is wrong; the site is +turning around. + +This project's doctrine on crying wolf is explicit (SAFETY-15, SAFETY-19) and +this is exactly the shape that generates it. So the trip is a threshold AND a +sustained window, and BOTH numbers were measured on the TEST-19 two-master rig +rather than chosen - see `CIRC_OPTION_DEFAULTS` for the measurements and +`DECISIONS/SAFETY-31-circulating-power.md` for the runs. + +⚠️ THE WINDOW ACCUMULATES, WHICH IS WHY THIS IS NOT SAFETY-02's PROBLEM AGAIN. +The reason the sign watchdog is blind to the broadcast fight is that the +disagreement FLIPS SIGN on alternate ticks, so `disagree_since` is cleared +between every pair of them and the 180 s hold never accumulates (allocator.py's +docstring records the measurement). Circulating power cannot do that: it is a +MAGNITUDE, non-negative by construction, and a sustained loop holds it high on +every cycle rather than on every other one. An oscillation that genuinely +alternated between opposing and agreeing is not a circulating loop and should +not be reported as one. +""" + +from dataclasses import dataclass +from datetime import datetime + +# CFG-03. THE ONE IN-CODE COPY OF THESE DEFAULTS. `config.yaml`'s `options:` +# block is the source of truth an operator edits and the Supervisor reads; +# `schema:` holds the bounds. This dict exists for the reason +# `maintenance.MAINT_OPTION_DEFAULTS` does - the Dockerfile does not put +# config.yaml in the image, so the running add-on cannot read it - and +# `test_watchdog.py` pins the two together by READING config.yaml, so a drift +# goes red in CI rather than shipping. +CIRC_OPTION_DEFAULTS = { + # --- THE THRESHOLD, AND IT IS READ OFF A CURVE ------------------------ + # ⚠️ MEASURED ON THE TEST-19 TWO-MASTER RIG, NOT CHOSEN. 924 closed-loop + # runs in which NOTHING is faulty - every unit executes the share the + # allocator gave it - over both families of stimulus that can produce + # legitimate opposition: single reversals (7 sizes x 5 write staggers) and a + # flapping load (7 periods x 4 amplitudes), each swept across the WHOLE + # range config.yaml accepts for the two settings that decide how fast the + # site can turn around, `gain` float(0.05,1.0) and `slew_w` int(50,5000). + # The worst CONTINUOUS opposition, as a function of the threshold: + # + # 100 W -> 505.0 s 300 W -> 10.0 s 700 W -> 5.0 s + # 200 W -> 445.0 s 350 W -> 10.0 s 1000 W -> 5.0 s + # 250 W -> 40.0 s 400 W -> 5.0 s 1700 W -> 0.0 s + # + # THE KNEE IS AT 400 W and it is sharp. Below ~250 W a flapping load holds + # the sensor above the threshold for MINUTES at a time - 505 s at 100 W - + # and no window shorter than the 900 s billing quarter could tell that from + # a fault. At 400 W and above the worst legitimate episode collapses to 5.0 + # s, ONE control cycle, and stays there. So this number is not a sensitivity + # preference; it is the edge of the regime in which the guard is + # answerable at all. + # + # 700 W is 1.75x the knee, and the margin is bought for one named reason: + # INVERTER B DOES NOT PHYSICALLY EXIST. Its ramp and lag in the rig are + # inverter A's, COPIED, carrying `measured=False` - SPIKE-04 measures the + # real ones. A slower or faster second unit moves the transient's shape, and + # the headroom above the knee is what absorbs that without re-opening this + # card. Revisit it when SPIKE-04 lands, not before. + # + # What it costs: a circulating loop smaller than 350 W per unit is not + # reported. That is deliberate - at that scale the round-trip loss is a few + # watts and the wear is proportionate to it. + "circulation_w": 700, + + # --- THE WINDOW ------------------------------------------------------- + # Two lower bounds, one upper, and the chosen value clears all three. + # + # LOWER, from the rig: 5.0 s. The worst continuous legitimate opposition + # above 700 W across all 924 runs is exactly one control cycle - a + # reversing site whose second write lands a cycle after the first, which + # is what the serialised write path does on every reversal. + # + # LOWER, from the firmware: `inverter.LINK_STALE_S`, 30 s. A unit still + # executing a stale share for less than that is a LINK fault, and the + # add-on already has an answer for it - liveness drops the unit and the + # survivors carry its share, with the record saying so. Firing inside that + # window would raise a second, louder alarm about a state that is already + # being handled, which is the SAFETY-15/SAFETY-19 failure exactly. + # + # UPPER, from the bill: 900 s, the quarter that sets the capacity tariff. + # A circulating loop must be caught and acted on well inside one. + # + # 75 s: fifteen control cycles - 15x the worst legitimate episode this rig + # can produce, 2.5x the firmware's own veto window, and 12x inside the + # quarter. ⚠️ IT IS DELIBERATELY NOT 60 OR 120. A round number here would be + # a guess wearing a measurement's clothes, and the three bounds above are + # what the number has to answer to. + "circulation_hold_s": 75, +} + + +@dataclass(frozen=True) +class CirculationConfig: + """Defaults, not constants - config.yaml carries both. See CFG-03 above.""" + + min_w: float = float(CIRC_OPTION_DEFAULTS["circulation_w"]) + hold_s: float = float(CIRC_OPTION_DEFAULTS["circulation_hold_s"]) + + +def circulating_w(powers): + """`sum(|B_i|) - |sum(B_i)|` over MEASURED per-inverter power, in W. + + `powers` one measured power per inverter, in roster order, + = discharge. + A unit whose reading did not arrive is `None`. + + Returns a non-negative float, or `None` when the answer is unknown. + + ⚠️ N<=1 IS IDENTICALLY ZERO AND RETURNS ZERO EVEN FOR A MISSING READING + (AC 5). With one term `sum(|B|) - |sum(B)| == |B| - |B| == 0` for EVERY + value of B, including a value nobody read. Returning `None` there would be + reporting an unknown where the algebra has no room for one, and it would put + a shipped one-inverter install permanently in the "cannot tell" state on a + quantity that cannot be anything but zero. Nothing changes at the shipped + roster; that is the compatibility promise this card makes. + + ⚠️ AND AT N>=2 A MISSING READING IS `None`, NOT ZERO, WHICH IS THE OPPOSITE + DIRECTION AND FOR A REASON. Zero would be this sensor reporting HEALTH from + an absence - the precise shape of the 2026-09-02 outage, where a subsystem + stopped answering and its silence read as calm for 6 h 25 m. An unknown + circulating figure is an unknown, it CLEARS the trip's window rather than + accumulating it (see `CirculationWatchdog.evaluate`), and the site is + already stopped for a missing per-unit reading anyway - `inputs_ok` requires + every unit's `batt`, and §11.1 rule 1 commands 0 W without consulting this. + """ + vals = list(powers) + if len(vals) <= 1: + return 0.0 + if any(v is None for v in vals): + return None + vals = [float(v) for v in vals] + # ⚠️ max(0.0, ...) IS DEFENSIVE, AND IT IS AN EQUIVALENT MUTANT - SAID HERE + # RATHER THAN LEFT FOR THE NEXT MUTATION SWEEP TO REDISCOVER. The triangle + # inequality makes this non-negative mathematically, and in floating point + # it is non-negative too: for a same-sign roster `abs(v)` is the list itself + # up to one sign, so the two sums accumulate THE SAME MAGNITUDES IN THE SAME + # ORDER and are bit-identical; for a mixed-sign roster the two differ by + # 2*min(...), which is enormous beside any rounding. Searched, not assumed - + # 700 000 random rosters of 2 to 12 units across 10 orders of magnitude + # produced no negative result. + # + # So removing this line changes no observable and no check here would go + # red. It stays because the cost is one call and the failure it would + # prevent - a power sensor publishing -1e-16 W, which is the kind of value + # somebody eventually writes a `< 0` branch against - is out of proportion + # to it. Do not report its survival as a gap in the checks. + return max(0.0, sum(abs(v) for v in vals) - abs(sum(vals))) + + +def opposing(powers, min_w: float = 0.0): + """The units pulling each way right now, as `(discharging, charging)` names. + + Indices, not names - the caller owns the roster order. Used for the operator + sentence: "which two inverters are pushing against each other" is the first + thing somebody woken by this alarm needs, and reconstructing it from the + number alone is impossible. + + `min_w` ignores units below a magnitude, so a unit resting at 3 W of + standby draw is not named as a participant in a 3 kW loop. + """ + pos = [i for i, v in enumerate(powers) + if v is not None and float(v) > float(min_w)] + neg = [i for i, v in enumerate(powers) + if v is not None and float(v) < -float(min_w)] + return pos, neg + + +class CirculationWatchdog: + """ONE PER SITE, evaluated once per control cycle. + + ⚠️ ONE PER SITE, WHERE SAFETY-02's IS ONE PER ACTUATOR, AND THE ASYMMETRY IS + THE WHOLE ATTRIBUTION ARGUMENT. `SignWatchdog` concludes "THE UNIT I AM + DRIVING is pushing power the wrong way" - evidence about one actuator, so + the remedy is that actuator (arbiter.CLAIM_SCOPE calls `failsafe` a UNIT + claim for exactly this reason). Circulating power is not a property any one + inverter has. It is a property of a PAIR, it is defined by a sum across the + roster, and there is no fact of the matter about which of the two is the + offender: a unit ignoring its share and a sibling faithfully executing a + share computed to sum with it produce identical readings. So the quantity is + site-level, the watchdog is site-level, and the remedy below is site-level. + + Latches, for `SignWatchdog`'s reason: the trip removes its own evidence. The + moment the site stops commanding, the loop it detected either persists (a + wedged unit, now visible at the meter) or vanishes (the add-on was one of + the two participants) - and in the second case a non-latching detector would + de-assert and start the whole thing again, three minutes at a time, forever. + """ + + def __init__(self, cfg: CirculationConfig | None = None, *, + tripped: bool = False, reason: str = ""): + self.cfg = cfg or CirculationConfig() + self.over_since: datetime | None = None + # ⚠️ RESTORED FROM THE STORE, NOT DEFAULTED - SAFETY-34, and it is the + # reason this card exists in the shape it does. `persistent_notification` + # is IN MEMORY: Home Assistant restarts and the card is gone, with the + # fault still running. That is precisely how the 2026-09-02 outage went + # unseen for 6 h 25 m while the detector had been correct for hours. The + # latch lives in `/data/state.json` - the add-on's only durable volume - + # and `main.py` RE-RAISES the notification from it on every start. + self.tripped = bool(tripped) + self.reason = reason if self.tripped else "" + # The worst figure seen during the current episode, for the operator + # sentence. Not persisted: it is colour on a reason string that IS. + self.peak_w = 0.0 + + def evaluate(self, circ_w, now: datetime) -> bool: + """Advance the detector one cycle. Returns True while tripped. + + `circ_w` this cycle's `circulating_w()` over MEASURED per-unit power, + or None when a reading is missing. None CLEARS the window + rather than accumulating it - see the module docstring, and + `SignWatchdog.evaluate` on the identical decision for a stale + meter. A frozen figure held against a moving one manufactures + a fault in about a window's worth of cycles. + `now` the caller's clock, so this is testable and replayable. + """ + if self.tripped: + # Latched. Do not re-evaluate: the trip removed the evidence. + return True + + if circ_w is None or float(circ_w) <= self.cfg.min_w: + self.over_since = None + self.peak_w = 0.0 + return False + + circ_w = float(circ_w) + self.peak_w = max(self.peak_w, circ_w) + if self.over_since is None: + self.over_since = now + return False + + held = (now - self.over_since).total_seconds() + if held < self.cfg.hold_s: + return False + + self.tripped = True + self.reason = ( + f"CIRCULATING_POWER: {circ_w:.0f} W measured going round between " + f"the inverters (peak {self.peak_w:.0f} W) for {held:.0f} s " + f"continuously - the meter cannot see it, because it nets to zero " + f"there (limits: over {self.cfg.min_w:.0f} W, held over " + f"{self.cfg.hold_s:.0f} s)") + return True + + def held_s(self, now: datetime) -> float: + """How long the current opposition has stood, for the status page. + + Takes the clock rather than reading one, for `SignWatchdog.held_s`'s + reason: every observable on this object is a pure function of what it + was handed, and an unclocked function is one nothing can test. + """ + if self.over_since is None: + return 0.0 + return (now - self.over_since).total_seconds() + + def reset(self) -> None: + self.over_since = None + self.tripped = False + self.reason = "" + self.peak_w = 0.0 diff --git a/goodwe_controller/app/main.py b/goodwe_controller/app/main.py index 3e6b4d5..a9e90d8 100644 --- a/goodwe_controller/app/main.py +++ b/goodwe_controller/app/main.py @@ -38,6 +38,10 @@ from .arbiter import ( Claim, Resolution, actuator_for, resolve, ) from .control import Tuning, compute, maintenance_charge_floor, peak_at_risk +from .circulation import ( + CIRC_OPTION_DEFAULTS, CirculationConfig, CirculationWatchdog, + circulating_w, opposing, +) from .clamp import ( REASON_CODE as CLAMP_REASON, ClampWatchdog, clamp_to_rating, is_commandable, ) @@ -249,6 +253,39 @@ class Controller: self.freeze_wd = FreezeWatchdog( FreezeWatchdogConfig(hold_s=float(opts.get("meter_freeze_s", 300)))) + # SAFETY-31. ONE PER SITE, and restored from the store like SAFETY-02's + # latch rather than defaulted like SAFETY-18's - the asymmetry is the + # same one the freeze watchdog's own note draws, read the other way. + # This one ACTS: a trip files a P_SAFETY SET that stops every inverter, + # so a restart that cleared it would come back up straight into the loop + # it had just stopped, which is SAFETY-02's measured three-minutes-at-a- + # time failure with a different quantity in front of it. + # + # ⚠️ SITE-LEVEL KEYS, NOT NAMESPACED. store.py carries the argument: + # circulating power is defined over the roster and is identically zero + # for any one unit, so there is no inverter to hang the latch on. + self.circ_wd = CirculationWatchdog( + CirculationConfig( + min_w=float(opts.get("circulation_w", + CIRC_OPTION_DEFAULTS["circulation_w"])), + hold_s=float(opts.get("circulation_hold_s", + CIRC_OPTION_DEFAULTS["circulation_hold_s"]))), + tripped=bool(store.data.get("circ_tripped", False)), + reason=str(store.data.get("circ_reason", "") or "")) + # SAFETY-34. Has the durable alarm been put back in front of the + # operator since this process started? `persistent_notification` is in + # memory and does NOT survive the Home Assistant restart that the store + # does, so the latch alone is not the alarm - re-raising it is. + # `run_control()` does that, once, before the first cycle. + self._circ_announced = False + if self.circ_wd.tripped: + _LOG.error( + "SAFETY-31 was TRIPPED before this restart and stays tripped: " + "%s. Every inverter is held at 0 W. Restarting does not clear " + "this - start control from the UI once you know why two " + "inverters were pushing against each other.", + self.circ_wd.reason or "(reason not recorded)") + # SAFETY-03. The output clamp's alarm - see clamp.py for why the clamp # itself is the cheap half. # @@ -720,6 +757,35 @@ class Controller: return None return float(sum(float(v) for v in vals)) + @property + def site_circulating_w(self): + """SAFETY-31. `sum(|B_i|) - |sum(B_i)|` over MEASURED per-unit power. + + ⚠️ `inv.batt`, NOT `self.allocation`. THIS IS THE WHOLE CARD, and the + wrong version is the tempting one: the same formula over the allocation + record would be simpler, would never be None, and would be worth + nothing. ALLOC-01's sign-coherence gate (`allocator.REFUSE_SIGN`) already + refuses an allocation whose shares carry two signs, so a record-derived + sensor reads zero BY CONSTRUCTION, forever, and can only ever confirm + that the allocator agrees with itself. Every fault this card is for is a + divergence between what was commanded and what the hardware is doing - a + unit ignoring or lagging its share, a stale share still being executed, + or HW-01's power stage running its last instruction for 6 h 25 m with + the comms dead and the allocation record impeccable throughout. + Measured on the TEST-19 rig: with one unit wedged, the record stays + `ok` with both shares at -1250 W - a record-derived sensor reads 0 W - + while this one reads 2000 W. + + `batt` became per-instance in DEV-02; before that there was one scalar + on the Controller and this property could not have been written. + + None when any unit has not answered, and NOT zero - circulation.py says + why at length. The short version: zero would be this sensor reporting + health from an absence, which is the 2026-09-02 failure itself. + Identically 0.0 at the shipped roster of one (AC 5). + """ + return circulating_w([inv.batt for inv in self.inverters]) + @property def pack_floor_soc(self): """MINIMUM. The SoC that a discharge PERMISSION must be graded against. @@ -1204,6 +1270,39 @@ class Controller: # share of it. At the shipped roster of one the two are the same number. site_standing = float(sum(standing.values())) + # --- SAFETY-31, AND IT RUNS BESIDE SAFETY-02 FOR THE SAME REASON ----- + # Evaluated EVERY cycle, unconditionally, above the "inputs missing but + # not yet stale" hold below. A run of held cycles would otherwise leave + # `over_since` untouched across the gap and the window would resume from + # a start time that predates it, which is the ordering bug the sign + # watchdog's own note describes. + # + # ⚠️ THE INPUT IS MEASURED POWER, WHICH IS THE ENTIRE POINT OF THE CARD. + # `site_circulating_w` reads each `Inverter.batt` - that unit's own + # meter, holding register 0x0518. It is NOT `self.allocation`, and the + # property's docstring carries the argument: ALLOC-01's sign-coherence + # gate already makes the allocation record incapable of expressing + # circulating power, so a record-derived sensor is zero by construction + # and detects nothing. What this catches is circulation the allocator + # NEVER COMMANDED - a unit ignoring or lagging its share, a stale share + # still being executed, or HW-01's wedged power stage. + circ_w = self.site_circulating_w + was_circ = self.circ_wd.tripped + if self.circ_wd.evaluate(circ_w, now) and not was_circ: + # REASON FIRST, FLAG SECOND, and each `set` writes the file - the + # same atomicity ordering SAFETY-02 uses two blocks up, and for the + # same reason: a crash between the two must not persist a latch with + # no explanation for the operator who has to decide about it. + self.store.set("circ_reason", self.circ_wd.reason) + self.store.set("circ_tripped", True) + self._circ_announced = False + self.log_event("CIRCULATING POWER - " + self.circ_wd.reason) + _LOG.error( + "SAFETY-31 TRIPPED - %s. Commanding 0 W on every inverter and " + "holding it until control is started again. This survives an " + "add-on restart AND a Home Assistant restart; restarting will " + "not clear it.", self.circ_wd.reason) + stale_after = float(self.o.get("stale_input_s", 15)) # ⚠️ `any`, NOT the sole unit's. The hold exists so a single missed poll # does not become a fault; a LATCHED inverter must still reach the claim @@ -1265,6 +1364,58 @@ class Controller: "sign disagreement", actuator=self.actuator_of("failsafe", inv))) + # --- SAFETY-31: WHAT "ACTED ON" MEANS, AND WHAT IT HONESTLY CANNOT -- + # ⚠️ A SITE SET AT P_SAFETY: STOP COMMANDING, EVERYWHERE. Read the next + # paragraph before changing it, because the obvious objection is right + # and is answered rather than ignored. + # + # THE OBJECTION. HW-01 established that commanding a wedged inverter + # achieves nothing - on 2026-09-02 the comms subsystem failed and the + # power stage went on executing its last instruction for 6 h 25 m, + # through every command this add-on sent it. If the cause of the + # circulation is a unit that has stopped listening, then a command is + # not a remedy and pretending otherwise is theatre. + # + # THE ANSWER, AND IT IS WHY THIS IS STILL AN ACTION RATHER THAN A + # NOTIFICATION. A circulating loop needs TWO participants. In every + # cause on this card's list, exactly one of them is a unit that has + # stopped listening and THE OTHER ONE IS THIS CONTROLLER - faithfully + # commanding the sibling to a share computed to sum with a number the + # wedged unit is not actually producing. Stopping the commanding half + # does two things a notification cannot: + # + # 1. IT HALVES THE LOOP. The unit that is still listening stops being + # driven into the one that is not, so the round-tripping stops on + # the side this controller owns. That is real energy and real + # cycles off a real pack, and it needs no cooperation from the + # wedged unit to take effect. + # 2. IT MAKES THE RESIDUE VISIBLE. This is the more important half. + # Circulating power hides because it NETS TO ZERO AT THE METER. + # With this controller's contribution removed, whatever the wedged + # unit is still doing stops being cancelled and appears at P1 as a + # plain net flow - where SAFETY-02's sign watchdog (which gates on + # `abs(grid_w) > 300` and could never open at a netted zero), the + # peak guard and the capacity tariff can all see it. The fault + # stops being invisible. Measured on the TEST-19 rig: the card's + # own scenario reads +0 W at the meter with 6000 W going round, and + # the shipped SignWatchdog's predicate is false on every one of + # those cycles. + # + # WHAT IT DOES NOT DO, SAID PLAINLY: it does not stop the wedged + # inverter. Nothing this add-on can send will. If the loop persists at + # the meter after this trip, the remaining actuator is a human at the + # isolator, which is what the notification says in those words. + # + # ⚠️ AND IT IS A SITE CLAIM. `arbiter.CLAIM_SCOPE` carries the argument: + # circulating power is defined over the roster and is identically zero + # for any one unit, so there is no attribution to file this per-unit on - + # and parking the wrong one of the two would leave the loop running with + # the guard satisfied. + if self.circ_wd.tripped: + claims.append(Claim.set("circulation", P_SAFETY, 0.0, + "circulating power", + actuator=self.actuator_of("circulation"))) + # SITE, both of them, and the two reasons are why. "stopped" is the # operator switching control off for the whole site; "inputs missing" is # the SITE's own telemetry gone - the meter this controller regulates @@ -1794,6 +1945,16 @@ class Controller: # is not, and test_freeze.py part 5 fails on exactly that. await self.check_meter_frozen() + # --- SAFETY-31's alarm, and its boundary is the LATCH, not a number --- + # Unconstrained where the two above are not: it grades + # `self.circ_wd.tripped`, which was advanced near the top of this method + # beside SAFETY-02 and which nothing since has touched. It is called on + # EVERY cycle rather than only on the cycle that trips, because its + # second job is re-raising the alarm after a restart into an existing + # trip - see its docstring on why that shares one code path with the + # fresh trip instead of getting its own. + await self.check_circulating() + # --- SAFETY-07's alarm, and it is NOT the engagement ------------------ # Its boundary is unconstrained: it grades `self.peak_guard`, which this # method assigned above and nothing since has touched. @@ -2050,6 +2211,71 @@ class Controller: f"you a lower bill, not the pack being spent."), }) + async def check_circulating(self) -> None: + """SAFETY-31 + SAFETY-34. The alarm, and it must survive a restart. + + ⚠️ THE LATCH IS NOT THE ALARM. `circ_tripped` in `/data/state.json` + survives everything, but nothing looks at a JSON file at 3 a.m. What the + operator sees is a `persistent_notification`, and THAT IS IN MEMORY: a + Home Assistant restart deletes it while the fault runs on. This is not a + hypothetical - it is precisely how the 2026-09-02 outage went unseen for + 6 h 25 m with the detector correct the whole time (SAFETY-34). + + So the durable half and the visible half are joined HERE, and the join + is `_circ_announced`: a per-PROCESS flag, deliberately not persisted. + The latch says "this site is tripped"; the flag says "this process has + put that in front of a human". A restart clears the flag and keeps the + latch, so the notification is RE-RAISED on the first cycle after every + add-on start - which is also the first cycle after a Home Assistant + restart, because the add-on is restarted with it. + + ⚠️ ONE CODE PATH FOR BOTH, ON PURPOSE. A fresh trip and a restart into + an existing trip raise the alarm through these same lines. Two paths + would be two chances for the restart one to rot unnoticed, and the + restart one is the case nobody exercises by hand. + """ + if not self.circ_wd.tripped or self._circ_announced: + return + # Set BEFORE the await. `hass.call()` can raise or hang, and a retry + # every cycle for a broker that is down would be an alarm storm on the + # event bus on top of a site that is already stopped. The latch is + # already durable and the log line above already fired; this surface is + # best-effort by design. + self._circ_announced = True + names = self.units + pos, neg = opposing([inv.batt for inv in self.inverters], + min_w=self.circ_wd.cfg.min_w / 2.0) + who = "" + if pos and neg: + who = ("\n\nRight now " + ", ".join(names[i] for i in pos) + + " is discharging while " + ", ".join(names[i] for i in neg) + + " charges.") + # A fixed notification_id, and a DIFFERENT one from every other alarm + # this add-on raises - check_meter_frozen() gives the reasoning. A + # circulating loop is not a frozen meter, is not an out-of-range + # setpoint and is not a saturated peak guard. + await self.hass.call("persistent_notification", "create", { + "notification_id": "goodwe_circulating_power", + "title": "GoodWe: the inverters are pushing against each other", + "message": ( + f"{self.circ_wd.reason}{who}\n\nTHE METER CANNOT SEE THIS. " + f"Circulating power nets to zero at P1, so the site reads idle " + f"while the packs are cycled through two conversion stages " + f"continuously. Every other guard in this add-on is a consumer " + f"of that net figure, which is why none of them fired.\n\n" + f"CONTROL HAS BEEN STOPPED - every inverter is commanded 0 W " + f"and held there. That removes this controller as one of the " + f"two participants, and it makes anything still happening " + f"VISIBLE at the meter instead of cancelled at it.\n\n" + f"⚠️ IF THE GRID READING DOES NOT MOVE AFTER THIS, one inverter " + f"is no longer listening to this add-on and no command will " + f"reach it - that has happened twice on this hardware, once for " + f"6 h 25 m. The remaining actuator is a human at the isolator.\n" + f"\nThis alarm survives a Home Assistant restart. Restarting " + f"will not clear it; start control from the add-on UI once you " + f"know why two inverters were opposing."), + }) + async def check_meter_frozen(self) -> None: """SAFETY-18. Alarm on a meter that answers forever without changing. @@ -2354,7 +2580,14 @@ class Controller: # additive to its vocabulary rather than a change to it. "status": ("failsafe" if any(i.sign_wd.tripped for i in self.inverters) + or self.circ_wd.tripped else "running" if self.auto else "stopped"), + # SAFETY-31. ⚠️ MEASURED, not the allocation record - see + # `site_circulating_w`. `publish()` drops None values, so a cycle + # with a missing per-unit reading leaves the last figure standing + # rather than publishing a zero that would read as health. + "circulation": (None if self.site_circulating_w is None + else round(self.site_circulating_w)), } # ⚠️ ONLY when P1 ingestion is actually running. The ESP32's stale-input # watchdog subscribes to sensor.goodwe_p1_sample_age - see the entity-id @@ -2691,6 +2924,35 @@ class Controller: f"{inv.sign_wd.cfg.min_w:.0f} W)" + (f" - opposed for {held:.0f} s now" if held else "")}) + # --- SAFETY-31 ----------------------------------------------------- + # ⚠️ SITE-LEVEL, WHERE THE ROWS ABOVE ARE PER-INVERTER, and the row says + # so. There is one circulating figure for the site; naming an inverter + # here would invite the operator to clear "the other one". + _circ = self.site_circulating_w + if self.circ_wd.tripped: + out.append({"ok": False, "warn": False, + "text": "CIRCULATING POWER - " + self.circ_wd.reason + + ". EVERY inverter is held at 0 W. The meter " + "cannot see this fault, which is why nothing " + "else reported it. Starting control clears " + "this; restarting the add-on does NOT, and " + "neither does restarting Home Assistant."}) + elif len(self.inverters) <= 1: + # AC 5. At the shipped roster the quantity is identically zero for + # every value of every reading, so "armed" would be a claim to be + # watching something that cannot happen. Say what is true instead. + out.append({"ok": True, "warn": False, + "text": "circulating power: n/a at one inverter - " + "sum(|Bi|) - |sum(Bi)| is identically zero"}) + else: + held = self.circ_wd.held_s(datetime.now(timezone.utc)) + out.append({"ok": True, "warn": False, + "text": "circulation watchdog: armed (trips after " + f"{self.circ_wd.cfg.hold_s:.0f} s over " + f"{self.circ_wd.cfg.min_w:.0f} W)" + + (f" - now {_circ:.0f} W" if _circ else "") + + (f", opposed for {held:.0f} s" if held else "")}) + if not o.get("estop_fitted", False): out.append({"ok": False, "warn": True, "text": "no e-stop fitted: if this host dies the battery stays " @@ -2708,7 +2970,17 @@ class Controller: def status_payload(self) -> dict: checks = self.checks() bad = [c for c in checks if not c["ok"] and not c["warn"]] - if any(i.sign_wd.tripped for i in self.inverters): + if self.circ_wd.tripped: + # ⚠️ ABOVE THE SIGN FAILSAFE, and the order is a decision. Both are + # latched P_SAFETY stops, so the site is at 0 W either way and the + # banner can only carry one sentence. This one goes first because it + # is the fault the operator has NO OTHER WAY of learning about: a + # sign disagreement is visible at the meter and on every dashboard + # in the house, and circulating power is visible nowhere at all. + level, banner = "bad", ("CIRCULATING POWER — the inverters were " + "pushing against each other. Site held at " + "0 W.") + elif any(i.sign_wd.tripped for i in self.inverters): # Outranks "NOT READY": nothing is missing, something went wrong. level, banner = "bad", ("FAILSAFE — the command opposed the meter. " "Inverter held at 0 W.") @@ -2734,6 +3006,14 @@ class Controller: rows = [ ("Grid", w(self.grid)), ("Battery", w(self.site_batt_w)), + # SAFETY-31, and it belongs DIRECTLY UNDER the two figures it is not + # derivable from. "Grid 0 W, Battery 0 W, Circulating 6000 W" is the + # whole card in three rows: both of the numbers an operator normally + # reads say the site is idle, and the third says 6 kW is going round + # between the packs. Only shown at N>=2 - at one inverter it is + # identically zero and a permanent "0 W" row would be noise. + *(() if len(self.inverters) <= 1 + else ((("Circulating", w(self.site_circulating_w)),))), ("State of charge", "—" if self.pack_floor_soc is None else f"{self.pack_floor_soc:g} %"), ("Commanded", w(self.target)), @@ -2808,6 +3088,22 @@ class Controller: # unit in particular. self.store.set("sign_tripped", False) self.store.set("sign_reason", "") + # SAFETY-31, cleared by the SAME press and for the same reason: + # the condition always goes away on its own, because the trip + # commands 0 W everywhere and two inverters at zero cannot + # oppose. Only a human who has read the banner can say the site + # is clear. + # + # ⚠️ AND `_circ_announced` GOES WITH IT. It is the "has a human + # been shown this" flag; leaving it True through a clear would + # mean the NEXT trip in this process latched, stopped the site, + # and raised no notification at all. + if self.circ_wd.tripped: + self.circ_wd.reset() + self.store.set("circ_tripped", False) + self.store.set("circ_reason", "") + self.log_event("circulating-power trip cleared by operator") + self._circ_announced = False if not self.auto: # ⚠️ THE WHOLE SITE, TARGETS INCLUDED - see command_site_zero(). # Zeroing `self.target` alone would leave each unit's last share diff --git a/goodwe_controller/app/mqtt.py b/goodwe_controller/app/mqtt.py index 7d6c807..f29916c 100644 --- a/goodwe_controller/app/mqtt.py +++ b/goodwe_controller/app/mqtt.py @@ -43,6 +43,19 @@ SENSORS = [ ("grid", "goodwe_grid_power", "Grid power", "W", "power", "measurement", None), ("battery", "goodwe_battery_power", "Battery power", "W", "power", "measurement", None), ("soc", "goodwe_battery_soc", "Battery SoC", "%", "battery", "measurement", None), + # SAFETY-31. `sum(|Bi|) - |sum(Bi)|` over MEASURED per-inverter power: the + # power going round between the inverters, which the grid sensor above + # CANNOT SHOW because it nets to zero there. The two are published side by + # side on purpose - "grid 0 W, circulating 6000 W" is the whole phenomenon + # in two rows of a dashboard, and no arithmetic on the grid figure alone can + # ever produce the second one. + # + # ⚠️ Published as 0 at the shipped one-inverter roster, not omitted, and + # that is a deliberate difference from `p1_age` below. An absent entity is + # right for a watchdog input nothing is feeding; this is a quantity whose + # value is KNOWN at N=1 - identically zero, for every reading - and a + # dashboard row reading 0 W is the honest report of it. + ("circulation", "goodwe_circulating_power", "Circulating power", "W", "power", "measurement", "mdi:sync-alert"), ("phase", "goodwe_maintenance_phase", "Maintenance phase", None, None, None, "mdi:battery-sync"), ("status", "goodwe_controller_status", "Controller status", None, None, None, "mdi:heart-pulse"), # ⚠️ THE LIVE ENTITY IS `sensor.goodwe_p1_sample_age`, NOT what this diff --git a/goodwe_controller/app/store.py b/goodwe_controller/app/store.py index 98e606f..f811d11 100644 --- a/goodwe_controller/app/store.py +++ b/goodwe_controller/app/store.py @@ -54,6 +54,23 @@ _LOG = logging.getLogger("goodwe.store") # `migrate_unit_keys()` and `Controller._grade_store_migration()` carry the # argument; the short version is that a latch is only worth persisting if it # survives the version change an operator makes while staring at the banner. +# +# ⚠️ AND SAFETY-31's PAIR IS FLAT ON PURPOSE, WHICH IS THE OPPOSITE DECISION TO +# THE ONE ABOVE AND NEEDS ITS OWN SENTENCE. `circ_tripped` / `circ_reason` are +# NOT in `LEGACY_UNIT_KEYS` and must never be moved into a unit namespace. +# Circulating power is `sum(|B_i|) - |sum(B_i)|` - a quantity DEFINED over the +# whole roster, which is identically zero for any single unit. There is no fact +# of the matter about which of two opposing inverters is the offender: a unit +# ignoring its share and a sibling faithfully executing a share computed to sum +# with it read identically. Namespacing this latch would mean inventing an +# attribution the measurement cannot support, and would then let an operator +# clear "inverter B's" half of a site-wide loop. One site, one latch. +# +# It is persisted for SAFETY-34's reason, which this add-on has already paid +# for once: `persistent_notification` is IN MEMORY. Home Assistant restarts and +# the card is gone while the fault runs on - exactly how the 2026-09-02 outage +# went unseen for 6 h 25 m with the detector correct throughout. main.py +# re-raises the notification from this key on every start. DEFAULTS = { "phase": "idle", "phase_started": None, @@ -61,6 +78,8 @@ DEFAULTS = { "last_start_attempt": None, "sign_tripped": False, "sign_reason": "", + "circ_tripped": False, + "circ_reason": "", } # --------------------------------------------------------------------------- # diff --git a/goodwe_controller/config.yaml b/goodwe_controller/config.yaml index fa47061..a30c3a4 100644 --- a/goodwe_controller/config.yaml +++ b/goodwe_controller/config.yaml @@ -134,6 +134,25 @@ options: # still inside the 900 s quarter that bills the capacity tariff. meter_freeze_s: 300 + # --- SAFETY-31: circulating power ------------------------------------------ + # ⚠️ CFG-03. THESE TWO LINES ARE THE SOURCE OF TRUTH FOR THESE DEFAULTS, and + # `app/circulation.py`'s `CIRC_OPTION_DEFAULTS` is the ONE in-code copy - the + # Dockerfile does not put this file in the image, so the running add-on cannot + # read it. test_watchdog.py pins the copy by READING these lines, so a drift + # goes red in CI rather than shipping. Add a third option in three places - + # options:, schema:, CIRC_OPTION_DEFAULTS - and in no fourth. + # + # Circulating power is sum(|Bi|) - |sum(Bi)| over MEASURED per-inverter power: + # two inverters pushing against each other, which NETS TO ZERO AT THE METER + # and is therefore invisible to every other guard in this add-on. Both numbers + # below were measured on the TEST-19 two-master rig over 924 closed-loop runs + # in which nothing was faulty; circulation.py carries the curve they were read + # off and the argument for each. They are NOT sensitivity preferences. + # + # Both are ignored at one inverter, where the quantity is identically zero. + circulation_w: 700 + circulation_hold_s: 75 + # --- maintenance ----------------------------------------------------------- # ⚠️ CFG-03. THESE EIGHT LINES ARE THE SOURCE OF TRUTH FOR THE MAINTENANCE # DEFAULTS. The Supervisor reads this file and the operator edits this file, @@ -398,6 +417,28 @@ schema: # app/freeze.py. meter_freeze_s: int(60,3600) + # SAFETY-31. No "off" value, for SAFETY-02's and meter_freeze_s's reason: + # disarming a safety alarm is not a supported configuration. + # + # ⚠️ THE FLOOR OF 400 IS NOT A RECOMMENDATION, IT IS THE MEASURED EDGE OF THE + # REGIME IN WHICH THIS GUARD CAN BE ANSWERED FOR. On the TEST-19 rig, the + # worst CONTINUOUS opposition produced by a site where nothing is faulty - a + # flapping load reversing every control cycle, swept over the whole range this + # schema accepts for `gain` and `slew_w` - is 5.0 s at a 400 W threshold and + # above, 10.0 s at 300 W, 40.0 s at 250 W and 505.0 s at 100 W. Below the + # knee, legitimate operation holds the sensor over the threshold for MINUTES + # and no window shorter than the 900 s billing quarter could tell that from a + # fault. A configurable value under 400 W would be a false-alarm generator + # that this project's own doctrine (SAFETY-15, SAFETY-19) forbids shipping. + circulation_w: int(400,5000) + # The floor of 30 is `inverter.LINK_STALE_S`: below the firmware's own veto + # window a unit still executing a stale share is a LINK fault the add-on + # already handles by dropping it from the split, and firing inside that window + # would raise a second, louder alarm about a state already being managed. The + # ceiling of 900 is the quarter that bills the capacity tariff - a circulating + # loop must be caught and acted on well inside one. + circulation_hold_s: int(30,900) + # ⚠️ CFG-03 - THIS IS WHERE A RANGE LIVES, AND THE ONLY PLACE IT LIVES. The # `options:` block above holds the DEFAULT; this block holds the BOUNDS, and # `MAINT_OPTION_DEFAULTS` deliberately mirrors the former and not the latter - diff --git a/goodwe_controller/test_arbiter.py b/goodwe_controller/test_arbiter.py index 7549004..883e27d 100644 --- a/goodwe_controller/test_arbiter.py +++ b/goodwe_controller/test_arbiter.py @@ -160,6 +160,11 @@ SITE_SOURCES = { "arbitrage": "buys and sells at the grid against a day-ahead price", "safety": "'stopped' is the site's switch, 'inputs missing' is the " "site's own meter - neither is about a pack", + # SAFETY-31, and it is the entry most easily filed on the wrong side. + "circulation": "sum(|Bi|) - |sum(Bi)| is defined OVER THE ROSTER and is " + "identically zero for any one unit - there is no fact of " + "the matter about which of two opposing inverters is the " + "offender, so there is nothing to file it per-unit on", } UNIT_SOURCES = { "device": "one inverter's own reported min/max rating", @@ -172,9 +177,9 @@ for _src, _why in sorted(SITE_SOURCES.items()): print(f" site {_src:<11} {_why}") for _src, _why in sorted(UNIT_SOURCES.items()): print(f" unit {_src:<11} {_why}") -check("all eight claim sources are classified, and only those eight", +check("all NINE claim sources are classified, and only those nine", set(CLAIM_SCOPE) == set(SITE_SOURCES) | set(UNIT_SOURCES)) -check("the four that are about the GRID CONNECTION are site claims", +check("the five that are about the GRID CONNECTION are site claims", {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_SITE} == set(SITE_SOURCES)) check("the four that are about a PACK, a DEVICE or one write are per-inverter", {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_UNIT} == set(UNIT_SOURCES)) @@ -195,7 +200,7 @@ check("with one inverter every source routes to the site, per-inverter ones " check("...and with none configured at all, likewise", all(actuator_for(s, "inverter-a", ()) == ACTUATOR_SITE for s in CLAIM_SCOPE)) check("with TWO the identity breaks and the split becomes real: the four " - "per-inverter sources route to the unit this controller drives, the four " + "per-inverter sources route to the unit this controller drives, the five " "site sources stay on the grid connection", {s: actuator_for(s, "inverter-a", TWO) for s in CLAIM_SCOPE} == {**{s: "inverter-a" for s in UNIT_SOURCES}, diff --git a/goodwe_controller/test_peakguard.py b/goodwe_controller/test_peakguard.py index 6c1e8c0..44e27a6 100644 --- a/goodwe_controller/test_peakguard.py +++ b/goodwe_controller/test_peakguard.py @@ -2358,8 +2358,17 @@ check("...and so does the persistent notification, which outlives the page and " # "deferred" is exactly the state in which a ninth source gets built by # accident. Its revisit trigger is ALLOC-01, under which the reserve becomes a # PER-INVERTER interval at the allocator rather than a ninth claim here. +# ⚠️ SAFETY-31 ADDED THE NINTH, DELIBERATELY, AND THIS IS THE PARAGRAPH THAT +# SENTENCE ASKS FOR. `circulation` is a SET at P_SAFETY on the SITE actuator, +# filed only while `circ_wd.tripped`. It is not the deferred loop-side reserve +# this tripwire was built to catch: it is not conditioned on a state of charge, +# it does not shape the loop, and it appears in NO cycle of the sweeps below - +# the dynamic halves of this part still observe exactly {device, supervised, +# loop} at every SoC from 0 to 100 %, because nothing in those fixtures +# circulates. What it does is stop the site when two inverters are measured +# pushing against each other. See DECISIONS/SAFETY-31-circulating-power.md. CLAIM_SOURCES = {"device", "supervised", "failsafe", "safety", "peak", - "arbitrage", "maintenance", "loop"} + "arbitrage", "maintenance", "loop", "circulation"} # The two appends that file a claim BUILT ELSEWHERE in the method. Named here so # that a THIRD such indirection reads as an unrecognised append rather than as a # silent gap in the scan. @@ -2400,7 +2409,7 @@ check("every `claims.append(...)` in main.py's cycle is one this scan can READ " "- a claim filed through a shape it cannot parse has to fail here rather " "than be counted as nothing, which is how a scan like this goes quietly " "vacuous", len(_appends) >= 10 and None not in _filed) -check("...and the sources that cycle can file are EXACTLY these eight. THIS is " +check("...and the sources that cycle can file are EXACTLY these nine. THIS is " "what makes part 10 a tripwire: replay_loop() hand-builds the claim list, " "so it cannot see a claim main.py adds - and a loop-side SoC reserve is a " "NINTH source. Filing one takes this line red on the line it is written", @@ -2530,19 +2539,21 @@ check("...and replay_loop() folding main.py's TWO P_SAFETY rating limits into " # structurally impossible. The structure is `CLAIM_SCOPE`, so `CLAIM_SCOPE` is # pinned to the AST-scanned set rather than trusted to agree with it. check("every source main.py can file is CLASSIFIED site or per-inverter, and " - "nothing else is. A ninth source now goes red twice - once here for " - "having no classification, once above for being a ninth - and it takes an " + "nothing else is. A TENTH source now goes red twice - once here for " + "having no classification, once above for being a tenth - and it takes an " "argument about what the claim is ABOUT to make either line green", set(CLAIM_SCOPE) == CLAIM_SOURCES) print(" site: " + ", ".join(sorted(s for s, v in CLAIM_SCOPE.items() if v == SCOPE_SITE)) + " | per-inverter: " + ", ".join(sorted(s for s, v in CLAIM_SCOPE.items() if v == SCOPE_UNIT))) -check("...and the split is the one ARB-03 argued: the four about the GRID " - "CONNECTION are site claims and the four about a pack, a device or one " - "write are per-inverter", +check("...and the split is the one ARB-03 argued, with SAFETY-31's addition: " + "the five about the GRID CONNECTION are site claims and the four about a " + "pack, a device or one write are per-inverter. `circulation` is a site " + "claim because its quantity is defined over the roster and is identically " + "zero for any one unit - there is no inverter to attribute it to", {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_SITE} - == {"loop", "peak", "arbitrage", "safety"} + == {"loop", "peak", "arbitrage", "safety", "circulation"} and {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_UNIT} == {"device", "supervised", "failsafe", "maintenance"}) @@ -2618,16 +2629,19 @@ _restore_b() print(" with a second inverter on the roster, main.py routes: " + ", ".join(f"{s}->{a}" for s, a in sorted(_routed.items()))) check("⚠️ AND THIS IS THE BREAKING HALF. Give the same Controller a second " - "inverter and its OWN routing method splits the eight: `device`, " + "inverter and its OWN routing method splits the nine: `device`, " "`supervised`, `failsafe` and `maintenance` bind the unit this controller " - "drives, `loop`, `peak`, `arbitrage` and `safety` stay on the grid " - "connection. Swap any one of them - route `peak` to the unit and the site " - "keeps no floor; route `maintenance` to the site and it can cross that " - "floor into `lo > hi` - and this line goes red", + "drives, `loop`, `peak`, `arbitrage`, `safety` and SAFETY-31's " + "`circulation` stay on the grid connection. Swap any one of them - route " + "`peak` to the unit and the site keeps no floor; route `maintenance` to " + "the site and it can cross that floor into `lo > hi`; route `circulation` " + "to one unit and a site-wide loop parks the inverter that may well be the " + "one still listening - and this line goes red", _routed == {"device": "inverter-a", "supervised": "inverter-a", "failsafe": "inverter-a", "maintenance": "inverter-a", "loop": ACTUATOR_SITE, "peak": ACTUATOR_SITE, - "arbitrage": ACTUATOR_SITE, "safety": ACTUATOR_SITE}) + "arbitrage": ACTUATOR_SITE, "safety": ACTUATOR_SITE, + "circulation": ACTUATOR_SITE}) check("...and the roster it was restored to is the shipped one, so nothing " "below inherits a two-inverter Controller", c_src.units == (app_main.UNIT_A,) and len(c_src.inverters) == 1) @@ -2636,7 +2650,7 @@ check("...and the roster it was restored to is the shipped one, so nothing " # IT RIGHT. `actuator_of("maintenance")` written at the `device` append routes a # rating limit to whatever `maintenance` routes to, and at the shipped # one-inverter roster BOTH answers are the site, so nothing above can see it. -# ⚠️ AND THE DYNAMIC CYCLE BELOW SEES ONLY THREE OF THE EIGHT DIRECT APPENDS. +# ⚠️ AND THE DYNAMIC CYCLE BELOW SEES ONLY THREE OF THE NINE DIRECT APPENDS. # That was THIS PART committing this repository's own named failure - a check # observing something other than what it claimed - inside the check written to # prevent it. The first cut of 10g called the two-inverter cycle "the line a @@ -2659,7 +2673,7 @@ check("...and the roster it was restored to is the shipped one, so nothing " # actuator as a LITERAL - `actuator=self.actuator_of('')` - so the # literal is simply required to EQUAL the source the claim is filed under. That # is what makes "a mis-typed `actuator_of(...)` goes red" a true sentence about -# all eight rather than about the three a fixture happens to reach. +# all nine rather than about the three a fixture happens to reach. def append_actuator(node): @@ -2677,7 +2691,7 @@ def append_actuator(node): return ast.unparse(kw[0].value) -# The appends that BUILD their claim in place - the eight whose routing is +# The appends that BUILD their claim in place - the nine whose routing is # chosen by hand, and the only ones where a typo is possible - against the two # that append a claim built elsewhere in the method and carry no `actuator=` # here at all. @@ -2708,19 +2722,19 @@ _unit_sources_unrouted = [ if CLAIM_SCOPE.get(r[0]) == SCOPE_UNIT and "," not in r[1]] print(f" {len(_direct)} direct appends, each read at its own line: " + ", ".join(f"main.py:{ln} {s} -> {a}" for s, a, ln in _routing)) -check("all eight of `Controller.cycle()`'s DIRECT appends route through " +check("all NINE of `Controller.cycle()`'s DIRECT appends route through " "`self.actuator_of()`, so this scan can read the routing " "of EVERY one of them. An append that computed its actuator, or omitted " "the argument, or hand-wrote a string, fails HERE rather than being " "silently skipped - which is how a scan like this goes quietly vacuous, " "and 10f is in this file because that already happened once", - len(_direct) == 8 + len(_direct) == 9 and all(a is not None and a.startswith("self.actuator_of('") for _s, a, _ln in _routing)) -check("⚠️ AND AT EVERY ONE OF THE EIGHT THE LITERAL IT ROUTES THROUGH IS THE " +check("⚠️ AND AT EVERY ONE OF THE NINE THE LITERAL IT ROUTES THROUGH IS THE " "SOURCE THE CLAIM IS FILED UNDER. THIS is the line a mis-typed " - "`actuator_of(...)` at a call site goes red on - all eight of them, on " - "the line the typo is written, at every roster, INCLUDING the five " + "`actuator_of(...)` at a call site goes red on - all nine of them, on " + "the line the typo is written, at every roster, INCLUDING the six " "appends no fixture in this repository reaches. " "`actuator_of(\"peak\")` at the maintenance append is a different string " "here even though it is the same VALUE at the shipped roster of one", @@ -2731,7 +2745,7 @@ check("⚠️ AND EVERY PER-INVERTER APPEND NAMES THE INVERTER IT IS ABOUT. " "site-scoped sources and is the whole defect for a per-inverter one: a " "`device` limit built inside the roster loop and routed without `inv` " "files inverter B's rating on inverter A, where it binds the wrong " - "share. Five of the eight appends are SCOPE_UNIT - `device`, " + "share. Five of the nine appends are SCOPE_UNIT - `device`, " "`supervised`, `failsafe` and maintenance's two - and all five pass one", _unit_sources_unrouted == [] and len([r for r in _routing @@ -2744,7 +2758,7 @@ check("...and the only appends this pin does NOT reach are the two that file a " "arbitrage.claim() by test_arbitrage.py", sorted(_indirect) == sorted(CLAIM_BUILDERS) and len(_direct) + len(_indirect) == len(_appends)) -check("...and between them the ten appends account for exactly the eight " +check("...and between them the eleven appends account for exactly the nine " "sources 10f pins, so the routing pin and the source set are one scan " "and neither can drift from the other", {s for s, _a, _ln in _routing} | set(CLAIM_BUILDERS.values()) @@ -2787,11 +2801,13 @@ check("⚠️ AND THE RATING LIMITS ARE FILED ONCE PER INVERTER, ON ITS OWN " ("loop", ACTUATOR_SITE), ("supervised", "inverter-a"), ("supervised", "inverter-b")]) check("...and naming those three is not an accident of this fixture but the " - "reason the static pin exists - five of the eight sources need a state " - "no check in this file reaches", + "reason the static pin exists - six of the nine sources need a state " + "no check in this file reaches. SAFETY-31's `circulation` is one of the " + "six: it is filed only while the site is MEASURED to be circulating, " + "which no fixture here is", {c.source for c in _cl2} == CLAIM_SOURCES - {"failsafe", "safety", "maintenance", "peak", - "arbitrage"}) + "arbitrage", "circulation"}) # ⚠️ AND NOW THE HALF THAT WAS EVIDENCE-SHAPED AND OBSERVED NOTHING. The three # checks above grade WHICH ACTUATOR each claim lands on. They say nothing about # WHOSE NUMBERS the bound is built from - and those are two different defects diff --git a/goodwe_controller/test_watchdog.py b/goodwe_controller/test_watchdog.py index 2b76ec4..6e7fdad 100644 --- a/goodwe_controller/test_watchdog.py +++ b/goodwe_controller/test_watchdog.py @@ -537,6 +537,574 @@ check("a crash between the two writes leaves the REASON on disk", check("...and never a latch with nothing to explain it", not crashed_store.data.get(TRIPPED_KEY)) + +# ============================================================================= +print() +print("=" * 72) +print("SAFETY-31: circulating power, which nets to zero at the meter") +print("=" * 72) +# ⚠️ READ THIS BEFORE ADDING A CHECK HERE. Half of this card was already built +# and it is NOT the half below. ALLOC-01 shipped the SIGN-COHERENCE GATE +# (`allocator.REFUSE_SIGN`, graded by test_alloc.py part 6): an allocation whose +# shares carry two signs is REFUSED, both units named, no shares returned. So +# the allocator can no longer COMMAND circulating power, and nothing here +# re-tests that. +# +# What is graded here is the MEASUREMENT half: circulating power the allocator +# NEVER COMMANDED. The two are complements, not duplicates - the gate refuses an +# allocation before it is written and cannot see one millisecond past that; this +# reads the hardware and does not care what was written. + +from app.inverter import LINK_STALE_S # noqa: E402 +from app.circulation import ( # noqa: E402 + CIRC_OPTION_DEFAULTS, CirculationConfig, CirculationWatchdog, + circulating_w, opposing, +) + +print() +print("1. the sensor: sum(|Bi|) - |sum(Bi)| over MEASURED per-unit power") +check("both units discharging: the triangle equality holds and it is ZERO. " + "This is the ordinary case and the sensor must be silent in it", + circulating_w([1000.0, 2000.0]) == 0.0) +check("...both charging, likewise", circulating_w([-1000.0, -2000.0]) == 0.0) +check("...and a unit resting at exactly zero beside a working one is not " + "opposition", circulating_w([0.0, 2500.0]) == 0.0) +# ⚠️ ASSERTED AGAINST A LITERAL, NOT AGAINST `2 * min(abs(a), abs(b))`. +# MUTATION.md's rule 2: an assertion written in terms of the expression under +# test cannot tell "the right formula" from "the same wrong formula twice". +check("A charging 3000 W while B discharges 3000 W reads 6000 W - the card's " + "own scenario, and the figure is 2*min(|A|,|B|), the power going round " + "the loop through BOTH conversion stages", + circulating_w([-3000.0, 3000.0]) == 6000.0) +check("...unequal opposition is bounded by the smaller unit: -3000 against " + "+1000 is 2000 W, not 4000", + circulating_w([-3000.0, 1000.0]) == 2000.0) +print(f" [-3000, +3000] -> {circulating_w([-3000.0, 3000.0]):.0f} W " + f"[-3000, +1000] -> {circulating_w([-3000.0, 1000.0]):.0f} W " + f"[+1000, +2000] -> {circulating_w([1000.0, 2000.0]):.0f} W") +check("three units, one out of step: it is defined over the whole roster, not " + "over a pair", circulating_w([-2000.0, 1000.0, 1000.0]) == 4000.0) + +print() +print(" AC 5: at N=1 it is IDENTICALLY zero, for every reading") +# ⚠️ SWEPT, NOT SAMPLED. "Identically zero" is a claim about EVERY value, and +# one call with one number cannot distinguish it from "zero for this number". +_n1 = [circulating_w([v]) for v in + (0.0, 1.0, -1.0, 2500.0, -2500.0, 5000.0, -5000.0, 1e9, -1e9)] +check("nine readings from -1e9 to +1e9 W, every one of them 0.0", + set(_n1) == {0.0}) +check("...and an EMPTY roster is zero too, not a crash - `Controller.publish()` " + "calls this before anything has been read", + circulating_w([]) == 0.0) +check("⚠️ AND A MISSING READING AT N=1 IS STILL ZERO, WHICH IS THE OPPOSITE OF " + "THE N>=2 RULE BELOW AND IS DELIBERATE. With one term the algebra has no " + "room for an unknown: |B| - |B| is 0 for every B, INCLUDING one nobody " + "read. Returning None here would park the shipped one-inverter install in " + "a permanent 'cannot tell' on a quantity that cannot be anything else", + circulating_w([None]) == 0.0) + +print() +print(" a missing reading at N>=2 is UNKNOWN, and never zero") +check("⚠️ None, NOT 0.0. Zero would be this sensor reporting HEALTH FROM AN " + "ABSENCE - which is the 2026-09-02 failure itself, where a subsystem " + "stopped answering and its silence read as calm for 6 h 25 m", + circulating_w([None, 3000.0]) is None + and circulating_w([-3000.0, None]) is None + and circulating_w([None, None]) is None) + +print() +print(" it is NON-NEGATIVE, including where floating point would say otherwise") +_fp = [circulating_w([0.1, 0.2, 0.3]), circulating_w([1e-9, 2e-9]), + circulating_w([-0.1, -0.2, -0.30000000000000004])] +print(f" {_fp}") +check("a same-sign roster never publishes a negative number. ⚠️ THIS GRADES " + "THE PROPERTY, NOT THE `max(0.0, ...)` GUARD THAT DEFENDS IT, and the " + "difference is stated so a mutation sweep does not report a gap that is " + "not one: for a same-sign roster the two sums accumulate the same " + "magnitudes in the same order and are bit-identical, so the guard is an " + "EQUIVALENT MUTANT - searched over 700 000 random rosters without a " + "single negative. circulation.py says so at the line", + all(v >= 0.0 for v in _fp)) + +print() +print("2. CFG-03: config.yaml is the source of truth and there is ONE copy") +# ⚠️ READ OFF THE FILE, NOT TRANSCRIBED. The Dockerfile does not put config.yaml +# in the image, so the running add-on cannot read it and `CIRC_OPTION_DEFAULTS` +# has to exist - which makes it a place to drift. This is the same pin +# test_maintenance.py puts on MAINT_OPTION_DEFAULTS, for the same reason. +import re as _re # noqa: E402 +from pathlib import Path as _Path # noqa: E402 + +_cfg_txt = (_Path(__file__).resolve().parent / "config.yaml").read_text(encoding="utf-8") +# ⚠️ SPLIT ON THE TOP-LEVEL KEY, NOT ON THE WORD. "schema:" appears inside the +# CFG-03 comment above these very options ("options:, schema:, ..."), and +# splitting on the bare word cut the file in the middle of a comment - the +# options block then contained no options, the regex found none, and this pin +# compared an EMPTY dict against an empty expectation. That is MUTATION.md's +# third failure mode exactly: the assertion ran, the sentence was true, and the +# data could not reach the failure it names. Caught by the printed line. +_cfg_opts, _cfg_schema = _cfg_txt.split("\nschema:", 1) +_from_yaml = {k: int(v) for k, v in + _re.findall(r"^ (circulation_\w+):\s*(\d+)\s*$", _cfg_opts, _re.M)} +print(f" config.yaml options: {_from_yaml}") +print(f" CIRC_OPTION_DEFAULTS: {CIRC_OPTION_DEFAULTS}") +check("both options are declared in config.yaml's options: block and the " + "in-code mirror agrees with it key for key and value for value", + _from_yaml == CIRC_OPTION_DEFAULTS and len(_from_yaml) == 2) +_bounds = dict(_re.findall(r"^ (circulation_\w+):\s*int\((\d+),(\d+)\)\s*$", + _cfg_schema, _re.M) + and [(m[0], (int(m[1]), int(m[2]))) for m in + _re.findall(r"^ (circulation_\w+):\s*int\((\d+),(\d+)\)\s*$", + _cfg_schema, _re.M)]) +print(f" schema bounds: {_bounds}") +check("...and BOTH have a schema range, so the Supervisor refuses an " + "out-of-range value rather than the add-on discovering it at runtime", + set(_bounds) == set(CIRC_OPTION_DEFAULTS)) +check("⚠️ AND THE THRESHOLD'S FLOOR IS THE MEASURED KNEE, NOT A ROUND NUMBER. " + "On the TEST-19 rig the worst CONTINUOUS opposition a site where NOTHING " + "is faulty can produce is 5.0 s at 400 W, 10.0 s at 300 W, 40.0 s at " + "250 W and 505.0 s at 100 W. Below the knee, ordinary operation holds " + "this sensor over the threshold for MINUTES and no window shorter than " + "the 900 s billing quarter could tell that from a fault - so a " + "configurable value under 400 W would be a false-alarm generator, and " + "the schema does not offer one", + _bounds["circulation_w"][0] == 400) +check("...and the window's floor is the firmware's own veto window, " + "`inverter.LINK_STALE_S`, because below it a unit still executing a " + "stale share is a LINK fault the add-on already handles by dropping it " + "from the split - and its ceiling is the 900 s quarter that bills the " + "capacity tariff, inside which this must be caught and acted on", + _bounds["circulation_hold_s"] == (int(LINK_STALE_S), 900)) +check("the shipped defaults sit inside their own schema ranges", + all(_bounds[k][0] <= v <= _bounds[k][1] + for k, v in CIRC_OPTION_DEFAULTS.items())) +check("⚠️ AND NEITHER IS A ROUND NUMBER, WHICH IS AC 3 READ LITERALLY. 700 W " + "is 1.75x the measured knee - the headroom is bought for inverter B's " + "ramp and lag being PLACEHOLDERS until SPIKE-04 measures them - and 75 s " + "is 15x the worst legitimate episode and 2.5x the firmware veto. 60, 100, " + "120, 180, 300, 500 and 1000 are all values a guess would have landed on", + CIRC_OPTION_DEFAULTS["circulation_w"] not in (100, 500, 1000, 2000) + and CIRC_OPTION_DEFAULTS["circulation_hold_s"] not in (30, 60, 120, 180, 300)) + +print() +print("3. the trip: sustained opposition fires, the slew transient does not") + + +def circ_drive(series, cfg=None, step=5.0): + """Feed a list of circulating figures, one per control cycle.""" + wd = CirculationWatchdog(cfg or CirculationConfig()) + at = None + for i, v in enumerate(series): + if wd.evaluate(v, T0 + timedelta(seconds=i * step)) and at is None: + at = i * step + return wd, at + + +_CFG = CirculationConfig() +_sustained, _at = circ_drive([2000.0] * 40) +check("2000 W of measured opposition, held, trips", _sustained.tripped) +check(f"...and at {_CFG.hold_s:.0f} s, not before - the window is a window, " + "not a delay somebody rounded", _at == _CFG.hold_s) +check("the reason gives the operator the figure, the peak and the duration, " + "and says in words why nothing else reported it", + "2000" in _sustained.reason and "75 s" in _sustained.reason + and "nets to zero" in _sustained.reason.lower()) +print(f" e.g. {_sustained.reason!r}") + +print() +print(" ⚠️ AND NOT ON THE TRANSIENT, WHICH IS THE FALSE-ALARM CASE") +# ⚠️ THE SHAPES BELOW ARE THE RIG'S, NOT INVENTED. Measured on TEST-19 over 924 +# closed-loop runs in which every unit executed the share it was given: the +# worst continuous opposition above 700 W is 5.0 s - ONE control cycle - at a +# peak of 1700 W. That is the serialised write path: on a reversing cycle B's +# new number lands up to a whole cycle after A's, both slew through zero at +# 800 W/s behind a 10 s lag, and between the two crossings they genuinely +# oppose. A guard that trips on that is a false-alarm generator, and this +# project's doctrine on crying wolf is explicit (SAFETY-15, SAFETY-19). +_one_cycle = [0.0] * 5 + [1700.0] + [0.0] * 30 +_t1, _ = circ_drive(_one_cycle) +check("the worst single transient the rig can produce - 1700 W for ONE control " + "cycle - does not trip", not _t1.tripped) +_repeated = ([0.0] * 3 + [1700.0]) * 20 +_t2, _ = circ_drive(_repeated) +check("...and neither does it REPEATED every fourth cycle for 400 s, which is " + "a load flapping across zero all afternoon. The window clears between " + "them, so nothing accumulates", not _t2.tripped) +_almost = [1500.0] * 14 + [0.0] + [1500.0] * 14 +_t3, _ = circ_drive(_almost) +check("...and opposition that stands for 70 s, clears for one cycle, then " + "stands again for 70 s does NOT trip: 140 s of trouble, but never 75 s " + "CONTINUOUSLY, which is what the guard actually claims to measure", + not _t3.tripped) +_below = [_CFG.min_w] * 60 +_t4, _ = circ_drive(_below) +check(f"...and exactly {_CFG.min_w:.0f} W forever does not trip - the " + "comparison is strictly greater, so the threshold is a threshold rather " + "than an off-by-one", not _t4.tripped) +_just_over, _at5 = circ_drive([_CFG.min_w + 1.0] * 40) +check("...while one watt above it does, held long enough. The pair is what " + "makes the boundary a measured line rather than a direction", + _just_over.tripped and _at5 == _CFG.hold_s) + +print() +print(" a MISSING reading clears the window rather than accumulating it") +_gap, _ = circ_drive([2000.0] * 10 + [None] + [2000.0] * 10) +check("⚠️ SAFETY-02 MAKES THE SAME DECISION FOR A STALE METER, AND FOR THE SAME " + "REASON: a frozen figure held against a moving one manufactures a fault " + "in about a window's worth of cycles. 50 s over, one unknown, 50 s over " + "again is not 100 s of opposition", not _gap.tripped) + +print() +print(" the latch: it does not re-evaluate, because the trip removes the evidence") +_latched, _ = circ_drive([2000.0] * 40 + [0.0] * 40) +check("still tripped after the opposition has gone - and the opposition ALWAYS " + "goes, because the trip commands 0 W everywhere and two inverters at zero " + "cannot oppose. A non-latching version would de-assert and start the " + "whole thing again, three cycles at a time, forever", _latched.tripped) +_r = CirculationWatchdog(_CFG, tripped=True, reason="restored from the store") +check("...and one restored as tripped stays tripped without being re-driven", + _r.evaluate(0.0, T0) and _r.reason == "restored from the store") +_r.reset() +check("...until it is reset, which is the operator's press and nothing else", + not _r.tripped and _r.reason == "" and _r.held_s(T0) == 0.0) + +print() +print(" `opposing()` names the two sides, for the sentence the operator reads") +check("indices of who is discharging and who is charging", + opposing([-3000.0, 3000.0]) == ([1], [0])) +check("...and a unit idling below the floor is not named as a participant in " + "somebody else's loop", opposing([-3000.0, 3000.0, 3.0], min_w=350.0) + == ([1], [0])) + +print() +print("4. END TO END, AND THIS IS AC 2 - the sensor is MEASURED, so it sees") +print(" circulation the allocator NEVER COMMANDED") +# ⚠️ THIS IS THE CHECK MUTATION.md's THIRD FAILURE MODE IS ABOUT, so it is +# built the way that section says to build one. The claim is "X differs from Y" +# - the MEASURED sensor differs from a sensor computed off the allocation +# record - and X and Y are ACTUALLY MADE DIFFERENT here, with the difference +# asserted as its own precondition and PRINTED, before anything is concluded +# from it. DEV-02 shipped a check comparing a number against itself; ALLOC-01 +# shipped a generator that found 0 escapes where a wider one found 39 991. + +from app.inverter import Inverter # noqa: E402 + +# ⚠️ INVERTER B DOES NOT PHYSICALLY EXIST - it is ~10 days out and SPIKE-04 +# measures it. Its rating here is a HYPOTHESIS, named as one, exactly as +# test_peakguard.py's helper names its own. +B_RATING_HYPOTHESIS = 3000.0 + + +class _CircHass(_Hass): + """A house at rest, and two inverters MEASURED to be fighting each other. + + ⚠️ THE MEETER READS ZERO AND THAT IS THE WHOLE POINT. grid = load - sum(B), + and with A at -3000 W and B at +3000 W the sum is zero, so P1 reads exactly + what an idle site reads. Every existing guard in this add-on is a consumer + of that number. + """ + + def __init__(self, a_w=-3000.0, b_w=3000.0): + super().__init__() + self.a_w, self.b_w = a_w, b_w + + async def number(self, entity, invert=False): + return {"sensor.grid": 0.0, + "sensor.soc": 50.0, "sensor.soc_b": 50.0, + "sensor.batt": self.a_w, "sensor.batt_b": self.b_w}.get(entity) + + +def _two_inverter(hass, store, *, hold_s=1, min_w=700, **over): + opts = {"setpoint_entity": "number.sp", "meter_entity": "sensor.grid", + "soc_entity": "sensor.soc", "batt_entity": "sensor.batt", + "meter_source": "off", "max_w": 5000, "gain": 0.6, + "circulation_w": min_w, "circulation_hold_s": hold_s, + "auto_start": True} + opts.update(over) + c = Controller(opts, hass, store, _P(), stop=asyncio.Event()) + b = Inverter("inverter-b", setpoint_entity="number.sp_b", + soc_entity="sensor.soc_b", batt_entity="sensor.batt_b", + max_w=B_RATING_HYPOTHESIS, store=store) + b.dev_min, b.dev_max = -B_RATING_HYPOTHESIS, B_RATING_HYPOTHESIS + c.inverters.append(b) + c.auto = True + for inv in c.inverters: + inv.dev_min = inv.dev_min if inv.dev_min is not None else -5000.0 + inv.dev_max = inv.dev_max if inv.dev_max is not None else 5000.0 + return c + + +async def _circ_e2e(): + hass = _CircHass() + c = _two_inverter(hass, _MemStore()) + seen = [] + for _ in range(14): + await c.read_inputs() + await c.cycle() + seen.append(dict(circ=c.site_circulating_w, grid=c.grid, + target=c.target, alloc=c.allocation, + tripped=c.circ_wd.tripped)) + await asyncio.sleep(0.25) + return c, hass, seen + + +_c, _chass, _seen = asyncio.run(_circ_e2e()) +_first = _seen[0] +_alloc = _first["alloc"] +_shares = dict(_alloc.shares) +_meas = [inv.batt for inv in _c.inverters] +print(f" MEASURED per-inverter power : {dict(zip(_c.units, _meas))}") +print(f" ALLOCATION RECORD : ok={_alloc.ok} {_alloc.explain()}") +print(f" meter (P1) : {_first['grid']:+.0f} W") +print(f" sensor over MEASURED power : {circulating_w(_meas):.0f} W") +print(f" sensor over the RECORD : " + f"{circulating_w(list(_shares.values())):.0f} W") + +# --- THE PRECONDITION, AS ITS OWN CHECK ------------------------------------- +# ⚠️ MUTATION.md RULE 1. If a later edit makes the record and the measurement +# agree again, THIS goes red rather than the claim below going quietly vacuous. +check("PRECONDITION: the allocation record and the measured power are " + "ACTUALLY DIFFERENT in this fixture. The record is coherent - it is `ok`, " + "its shares carry one sign, and they sum to the site target - while the " + "hardware is measured at -3000 W and +3000 W. Without this line the two " + "claims below could both be true of a fixture in which nothing differed", + _alloc.ok is True + and len({(s > 0) - (s < 0) for s in _shares.values() if s}) <= 1 + and sorted(_meas) == [-3000.0, 3000.0] + and set(_shares.values()) != set(_meas)) +check("ALLOC-01's gate is SATISFIED here, and that is the point rather than an " + "accident: the allocator refused nothing, because it commanded nothing " + "wrong. `REFUSE_SIGN` cannot fire on a record whose shares agree in sign", + _alloc.refused is None) +check("⚠️ SO A SENSOR COMPUTED FROM THE ALLOCATION RECORD READS EXACTLY 0 W. " + "That is the cheap version of this card, and it is worth nothing: " + "ALLOC-01's sign-coherence gate already guarantees the record is " + "sign-coherent, so a record-derived sensor is zero BY CONSTRUCTION and " + "can only ever confirm that the allocator agrees with itself", + circulating_w(list(_shares.values())) == 0.0) +check("⚠️ AND THE SHIPPED SENSOR READS 6000 W ON THE SAME CYCLE, because it is " + "derived from `Inverter.batt` - each unit's OWN measurement. This is the " + "whole card: it detects circulation THE ALLOCATOR NEVER COMMANDED - a " + "unit ignoring or lagging its share, a stale share still being executed, " + "or HW-01's power stage running its last instruction for 6 h 25 m with " + "the comms dead and the record impeccable throughout", + _c.site_circulating_w == 6000.0) +check("...and the METER READS ZERO through all of it, which is why no existing " + "guard fires. SAFETY-02 gates on abs(grid_w) > 300 W and that gate never " + "opens; the peak guard and the capacity tariff both bill grid exchange", + _first["grid"] == 0.0 + and not any(i.sign_wd.tripped for i in _c.inverters)) + +print() +print(" ...and the trip ACTS: a site-wide stop, not a notification") +check("the watchdog tripped", _c.circ_wd.tripped) +check("the site is commanded 0 W", _c.target == 0.0) +check("...and 0 W actually reached BOTH inverters, not just the one somebody " + "decided was to blame", + _chass.writes and _chass.writes[-1] == 0.0 + and all(i.target == 0.0 for i in _c.inverters)) +check("...and it got there through the `circulation` claim, by name, rather " + "than by some other guard happening to stop the site", + "circulation" in _c.reason) +print(f" status line: {_c.reason!r}") +check("the status page says so in words, and says the meter cannot see it", + any("CIRCULATING POWER" in ck["text"] and "meter cannot see" in ck["text"] + for ck in _c.checks())) +check("the banner leads with it - above the sign failsafe, because a sign " + "disagreement is visible on every dashboard in the house and this is " + "visible nowhere at all", + "CIRCULATING POWER" in _c.status_payload()["banner"]) +_pub = {} +_c.mqtt = type("P", (), {"enabled": True, + "publish": lambda _s, v: _pub.update(v), + "close": lambda _s: None})() +_c.publish() +print(f" published: circulation={_pub.get('circulation')} " + f"grid={_pub.get('grid')} status={_pub.get('status')}") +check("⚠️ AND IT IS PUBLISHED AS ITS OWN SENSOR BESIDE THE GRID FIGURE. " + "`grid 0 W` and `circulating 6000 W` on the same dashboard is the whole " + "phenomenon in two rows, and NO arithmetic on the grid figure alone can " + "ever produce the second one", + _pub.get("circulation") == 6000 and _pub.get("grid") == 0.0) +check("...and the controller-status sensor reports the failsafe state", + _pub.get("status") == "failsafe") +# ⚠️ AND THE UNKNOWN CASE, WHICH A SWEEP FOUND UNGRADED. A per-unit reading that +# did not arrive makes the site figure None, and `MqttPublisher.publish()` DROPS +# None values - so the entity keeps its last figure and the dashboard shows a +# stale number rather than a fresh zero. Publishing `round(x or 0.0)` instead +# survived the first mutation battery: it turns "I do not know" into "0 W", i.e. +# health manufactured from an absence, on the one surface an operator actually +# watches. Same defect as the sensor's own None rule, one layer out. +_c.inverters[1].batt = None +_pub_unknown = {} +_c.mqtt = type("P", (), {"enabled": True, + "publish": lambda _s, v: _pub_unknown.update( + {k: x for k, x in v.items() if x is not None}), + "close": lambda _s: None})() +_c.publish() +print(f" with inverter-b's reading missing, publish() offers: " + f"circulation={_pub_unknown.get('circulation', '')!r}") +check("⚠️ WITH ONE UNIT'S READING MISSING THE SENSOR IS NOT PUBLISHED AT ALL, " + "so the entity holds its last figure instead of being handed a 0 that " + "would read as health. `publish()` drops None, and this is what makes " + "that drop deliberate rather than incidental", + "circulation" not in _pub_unknown) +_c.inverters[1].batt = 3000.0 + +print() +print("5. AC 5: at ONE inverter the sensor is identically zero and NOTHING changes") + + +async def _n1(): + hass = _CircHass(a_w=-3000.0, b_w=0.0) + opts = {"setpoint_entity": "number.sp", "meter_entity": "sensor.grid", + "soc_entity": "sensor.soc", "batt_entity": "sensor.batt", + "meter_source": "off", "max_w": 5000, "gain": 0.6, + "circulation_w": 700, "circulation_hold_s": 1, "auto_start": True} + c = Controller(opts, hass, _MemStore(), _P(), stop=asyncio.Event()) + c.auto = True + c.dev_min, c.dev_max = -5000.0, 5000.0 + for _ in range(14): + await c.read_inputs() + await c.cycle() + await asyncio.sleep(0.2) + return c + + +_c1 = asyncio.run(_n1()) +print(f" roster={_c1.units} measured batt={_c1.inverters[0].batt:+.0f} W " + f"sensor={_c1.site_circulating_w}") +check("the shipped roster of one, with the single inverter measured at " + "-3000 W - a magnitude that would trip instantly at N=2 - reads exactly " + "0.0 W", _c1.site_circulating_w == 0.0) +check("...so the watchdog never trips, over 14 cycles at a 1 s window", + not _c1.circ_wd.tripped) +check("...no `circulation` claim is ever filed, so the arbitration is " + "byte-identical to what shipped", "circulation" not in _c1.reason) +check("...and the status page says n/a rather than claiming to watch something " + "that cannot happen - an 'armed' row for an identically-zero quantity is " + "a guard advertising work it is not doing", + any("circulating power: n/a at one inverter" in ck["text"] + for ck in _c1.checks())) +_pub1 = {} +_c1.mqtt = type("P", (), {"enabled": True, + "publish": lambda _s, v: _pub1.update(v), + "close": lambda _s: None})() +_c1.publish() +check("...and it publishes 0, not None: the value is KNOWN at N=1, and a " + "dashboard row reading 0 W is the honest report of it", + _pub1.get("circulation") == 0) + +print() +print("6. SAFETY-34: the alarm survives a Home Assistant restart") +# ⚠️ THE LATCH IS NOT THE ALARM, AND THE 2026-09-02 OUTAGE IS WHY THAT SENTENCE +# IS HERE. `persistent_notification` is IN MEMORY: Home Assistant restarts and +# the card is gone while the fault runs on. That outage went unseen for 6 h 25 m +# with the detector correct throughout. So the durable half (the store) and the +# visible half (the notification) have to be joined, and this grades the join. + + +async def _circ_restart(): + shared = _MemStore() + h1 = _CircHass() + a = _two_inverter(h1, shared) + for _ in range(14): + await a.read_inputs() + await a.cycle() + await asyncio.sleep(0.25) + # THE RESTART. Same store, brand new Controller and Hass - which is exactly + # what a Home Assistant restart does to this add-on. The house has stopped + # circulating in the meantime, so nothing in the new process's own + # measurements could re-derive the trip. + h2 = _CircHass(a_w=0.0, b_w=0.0) + b = _two_inverter(h2, shared) + for _ in range(3): + await b.read_inputs() + await b.cycle() + await asyncio.sleep(0.05) + return a, b, shared, h1, h2 + + +_a, _b, _shared, _h1, _h2 = asyncio.run(_circ_restart()) +print(f" store after the trip: circ_tripped=" + f"{_shared.data.get('circ_tripped')!r} " + f"circ_reason={str(_shared.data.get('circ_reason'))[:48]!r}...") +check("the trip is written to /data - the add-on's only durable volume - as a " + "flag AND a reason", + _shared.data.get("circ_tripped") is True + and "CIRCULATING_POWER" in str(_shared.data.get("circ_reason"))) +check("⚠️ THE REBUILT CONTROLLER COMES UP TRIPPED, and its own measurements say " + "the site is quiet - 0 W on both units - so this can ONLY have come from " + "the store. A latch that lives in memory comes back armed, into the " + "fault, with no operator involved", + _b.circ_wd.tripped and _b.site_circulating_w == 0.0) +check("...carrying the reason with it, so the operator who has to decide " + "whether to clear it can read WHY", + "CIRCULATING_POWER" in _b.circ_wd.reason) +check("...and it is still commanding 0 W on the far side of the restart", + _b.target == 0.0 and "circulation" in _b.reason) + +_notes1 = [c for c in _h1.calls if c[0] == "persistent_notification"] +_notes2 = [c for c in _h2.calls if c[0] == "persistent_notification"] +print(f" notifications raised: {len(_notes1)} before the restart, " + f"{len(_notes2)} after it") +check("the alarm was raised in the first process", len(_notes1) >= 1) +check("⚠️ AND RE-RAISED IN THE SECOND ONE. THIS IS THE CARD'S THIRD " + "DELIVERABLE. `persistent_notification` does not survive a Home Assistant " + "restart, so a latch alone would leave a stopped site with NOTHING in " + "front of the operator - the detector correct, the state file correct, " + "and the screen blank. That is how 2026-09-02 went unseen for 6 h 25 m", + len(_notes2) >= 1) +check("...under a fixed notification_id of its own, so a second episode " + "REPLACES the first rather than stacking, and it is not confused with " + "the frozen meter, the clamp or the saturated peak guard", + all(c[2]["notification_id"] == "goodwe_circulating_power" + for c in _notes1 + _notes2) + and len({c[2]["notification_id"] for c in _notes1 + _notes2}) == 1) +_msg = _notes2[0][2]["message"] +check("...and the message tells the operator the three things only this alarm " + "can: that the meter cannot see it, that control has been stopped, and " + "that if the grid reading does not move afterwards one inverter is no " + "longer listening and a human at the isolator is the last actuator", + "METER CANNOT SEE" in _msg and "CONTROL HAS BEEN STOPPED" in _msg + and "isolator" in _msg) +check("...and it does not raise the same card on every cycle - a per-PROCESS " + "flag, deliberately not persisted, is what makes 'has a human been shown " + "this' a different question from 'is this site tripped'", + len(_notes2) == 1) + +print() +print(" the operator's press is the only way out, and it clears BOTH halves") + + +async def _circ_clear(): + shared = _MemStore() + c = _two_inverter(_CircHass(), shared) + for _ in range(14): + await c.read_inputs() + await c.cycle() + await asyncio.sleep(0.25) + was = c.circ_wd.tripped + await c.handle_action("auto_toggle") # stop + await c.handle_action("auto_toggle") # start - the deliberate press + return c, shared, was + + +_cc, _cstore, _was = asyncio.run(_circ_clear()) +check("it really was tripped before the press", _was) +check("a deliberate START clears the latch in memory and on disk together - " + "the condition ALWAYS goes away on its own, because the trip commands " + "0 W everywhere and two inverters at zero cannot oppose, so clearing on " + "the condition would clear on nothing", + not _cc.circ_wd.tripped + and _cstore.data.get("circ_tripped") is False + and _cstore.data.get("circ_reason") == "") +check("⚠️ AND THE 'HAS A HUMAN SEEN THIS' FLAG IS CLEARED WITH IT. Left " + "standing through a clear, the NEXT trip in this process would latch, " + "stop the site, and raise no notification at all", + _cc._circ_announced is False) + + print() if fails: print(f"{len(fails)} FAILED: {', '.join(fails)}")