diff --git a/goodwe_controller/CHANGELOG.md b/goodwe_controller/CHANGELOG.md index 612931c..afc7f31 100644 --- a/goodwe_controller/CHANGELOG.md +++ b/goodwe_controller/CHANGELOG.md @@ -2,6 +2,89 @@ ## Unreleased +**At two inverters the sign watchdog stops comparing one unit against the whole +site meter, and becomes two checks instead of one (SAFETY-30).** +`disagrees(commanded_w, grid_w, ...)` compared the command **this controller put +on ONE actuator** against **what the whole site** is exchanging with the network. +At a roster of one those are two views of the same thing - §6.2's identity - and +that is what makes the check sound and what earned it the best record in the +safety table. At two they are not, and it fails **in both directions**: inverter +A commanded −1500 W while the meter reads +1500 W is evidence of nothing when +inverter B is discharging 3000 W exactly as told, and - worse - a unit genuinely +driving the wrong way is **MASKED** by a sibling that covers for it, because the +site figure nets. No threshold recovers that; the summation destroys the +evidence. + +* **PER UNIT: each inverter's own share against its OWN measured power** + (`Inverter.batt`, holding register 0x0518, per-instance since DEV-02). + Conclusion: *the actuator I am driving is not doing what I told it* - evidence + about one device, so the remedy stays the per-unit `failsafe` SET that + `arbiter.CLAIM_SCOPE` has classified `SCOPE_UNIT` since it was written. That + entry read *"the evidence is site-wide and the remedy is not"*; after this card + the first half is simply true. Verified by making one unit reverse while its + sibling covers it at the meter: the shipped comparison is false on **0 of 14** + cycles while the new one convicts that unit on every one, and only that unit is + parked. +* **SITE: the SUM of the standing commands against the meter**, as a new + `sign_site` SET at `P_SAFETY` on the site actuator, latched under flat + `site_sign_tripped` / `site_sign_reason`. This is the fault the 2026-08-20 + capture contains and **no per-unit check can see it**, because every inverter + is faithfully executing its share - what is wrong is the number they were + given. It stops every inverter and the banner names the site rather than an + inverter, so nobody is sent to swap the wrong box. +* **⚠️ Different questions from SAFETY-31, and the PR and the decision record + both say which.** `circulation.py` compares the units **against each other** + and catches two inverters pushing *through* each other, which nets to zero at + the meter and which no single command is evidence about. This compares each + unit against **its own command**. They overlap in exactly one scenario - a unit + reversing while its sibling pulls the other way - and `test_watchdog.py` + asserts on that fixture that both fire and what each concludes. It is also not + the saturation freeze: that is a magnitude test (`|command − measured| > + 500 W`) that shapes the loop and fires on a merely saturated unit; this is a + direction test that latches and parks one. +* **⚠️ All four thresholds RE-DERIVED at two inverters - one carried with a + justification, one refused.** 1890 closed-loop runs in which nothing is faulty, + over the whole range `config.yaml` accepts for `gain` and `slew_w`, across load + reversals with write staggers and flapping loads, **run at both rosters so the + N=1 arm is the control.** `unit_sign_disagree_w: 250` is `saturation_w / 2`, so + the smallest divergence this can see is exactly the gap at which the control + law itself already says the inverter is not following; `unit_sign_disagree_s: + 85` is 8.5× the worst legitimate opposition those runs produce (10.0 s), 2.8× + the firmware's own 30 s link veto and under half the site hold. + `site_sign_disagree_w: 300` is **carried** - its argument is the meter's noise + floor and the deadband, and the roster touches neither. **`site_sign_disagree_s` + is NOT: 180 s would have shipped a false trip.** The worst legitimate crossing + at 300 W is 110 s with one inverter and **215 s with two**, because the command + must travel a rail that is the roster sum at a `slew_w` that is not - reachable + at `gain: 0.05`, `slew_w: 50` and a 2.2 kW load reversal, all schema-legal. + 325 s is 1.51× that, inside the 900 s billing quarter and inside every episode + of the fault day. See `DECISIONS/SAFETY-30-per-unit-sign.md`. +* **⚠️ At ONE inverter this changes nothing, and "nothing" is asserted rather + than assumed.** The two questions are the same sentence at a roster of one, so + only the detector that shipped is armed: same comparison against the meter, + same 300 W / 180 s, same reason string byte for byte, same namespaced latch and + flat rollback mirror, same `failsafe` claim, same banner, same single status + row - and the site detector's window is never so much as opened. + `unit_sign_disagree_*` and `site_sign_disagree_*` do nothing on a + one-inverter install. +* **The threshold WIRING is pinned, not just the threshold values.** Review + found that `arm(self.sign_cfg if one else self.unit_sign_cfg, ...)` could be + collapsed to `arm(self.sign_cfg, ...)` — arming every unit with the + meter-facing pair instead of the re-derived per-unit one — with all 19 test + files still green, because nothing at a roster of two read `sign_wd.cfg` and + the trips in the fixture happen under either pair. `test_watchdog.py` part 3 + now asserts the pair and the instrument on **every** detector against + literals, and the fixture's three threshold pairs were made pairwise distinct + in both coordinates so that a hold delivered from the wrong pair cannot + compare equal to the right one. M14–M16 in + `DECISIONS/SAFETY-30-per-unit-sign.md` §5. +* **Known and recorded rather than fixed:** the site comparison is defeated by a + square-wave load whose half period is about one control cycle, which holds it + opposed for ~1000 s at *both* rosters. The N=1 control arm is what says this is + a property of comparing a lagged command against a meter and has nothing to do + with the second inverter, so it is not acted on here - it wants an alternation + test, not a number, and a card of its own. + **An operator can declare a second inverter, and the shape was measured against the Supervisor rather than assumed (CFG-04).** DEV-02 made rating, SoC, watchdogs, heartbeat and store keys per-instance, so the code no longer assumed diff --git a/goodwe_controller/DECISIONS/SAFETY-30-per-unit-sign.md b/goodwe_controller/DECISIONS/SAFETY-30-per-unit-sign.md new file mode 100644 index 0000000..2b8ffd7 --- /dev/null +++ b/goodwe_controller/DECISIONS/SAFETY-30-per-unit-sign.md @@ -0,0 +1,474 @@ +# SAFETY-30 — what the sign watchdog compares once there is more than one inverter, and the numbers it compares with + +**Status: DECIDED 2026-09-04.** The one comparison becomes **two**, asked of the +evidence that can answer each: **each unit's standing command against that +unit's own measured power** (register 0x0518, per-instance since [[DEV-02]]), +and **the sum of the standing commands against the meter**. At a roster of one +nothing changes at all — the two questions are the same sentence there, and the +detector that ships is the only one armed. + +This document records the argument for that split, the campaign the four numbers +were read off, **a number that was carried and a number that refused to be**, and +the mutation ledger. Filed with SAFETY-30, card `6a9a1828e844f22fabf7fe9c`. + +--- + +## 0. The half that was already built, and the half this card is + +[[DEV-02]] shipped the **per-instance** half and nothing here re-tests it: each +`Inverter` owns its own `SignWatchdog`, the latch is namespaced under +`unit//sign_tripped`, the flat pre-DEV-02 pair is maintained as a rollback +view, and a trip on B no longer latches A. `test_inverter.py` part 3 grades that. + +What was left is **the comparison itself**, and it was one line: + +```python +inv.sign_wd.evaluate(stood, self.grid, self.following, now) # every unit +``` + +`stood` is the command **this controller put on THIS actuator**. `self.grid` is +what the **whole site** is exchanging with the network. At one inverter those are +two views of one thing — §6.2's identity — which is what makes the check sound +and what earned its record. At two they are not, and it fails in **both** +directions: + +| | what happens | why no threshold fixes it | +|---|---|---| +| **false trip** | A commanded −1500 W while the meter reads +1500 W, because B is discharging 3000 W exactly as told | the meter is answering for a unit A's command says nothing about | +| **missed fault** | B commanded +1500 W and **measured** at −1500 W, beside A at +1500 W — the meter sits where a healthy site leaves it | the summation **destroys** the evidence | + +NOTES.md calls SAFETY-02 the highest-value check in the safety table: 93 % +agreement while healthy against 4 % during runaway, four observed episodes, zero +false positives. **That record was earned at N=1 on the site comparison.** The +first row above spends it; the second row is the guard being absent when it +matters. + +## 1. The two questions, and the two remedies + +> **PER UNIT** — this unit's standing command against **its own** measured power. +> Conclusion: *the actuator I am driving is not doing what I told it.* Evidence +> about one device, so the remedy is that device: the `failsafe` SET, which +> `arbiter.CLAIM_SCOPE` has classified `SCOPE_UNIT` since it was written. +> +> **SITE** — the **sum** of the standing commands against the meter. Conclusion: +> *the site as a whole is driving the wrong way* — an inverted gain, an +> integrator wound past the rail, a strategy on the wrong side of the meter. +> That is the 2026-08-20 fault day, it is a property of the **connection**, and +> the remedy is a new SITE SET, `sign_site`, that stops every inverter. + +### Why `failsafe` stays per-unit and `sign_site` is new + +`CLAIM_SCOPE`'s entry for `failsafe` used to read *"the evidence is site-wide and +the remedy is not"* — a site measurement attributed to one actuator on the +strength of the N=1 identity. After this card the first half of that sentence is +simply true: the evidence **is** per-unit. The site-wide half did not disappear; +it moved to a claim that is site-scoped for the reason `circulation` is, one +entry above it: **the offending number is the site target, which no single +inverter owns.** Parking one unit on it would leave the rest of the roster +pushing the same way with the guard satisfied. + +### At N=1, exactly one detector is armed + +`Controller._arm_sign_watchdogs()` is the only place the roster decides anything +here, and it is a pure function of `len(self.inverters)`: + +* **one inverter** — the unit detector is armed against **the meter**, at the + shipped `sign_disagree_w` / `sign_disagree_s`. The site detector is + **constructed and never evaluated**. Same comparison, same thresholds, same + reason string byte for byte, same namespaced keys and flat mirror, same + `failsafe` claim, same banner, same single status row. `test_watchdog.py` + part 7 asserts each of those, including that the site detector's window was + never so much as opened. +* **two or more** — each unit against its own 0x0518 power under the per-unit + pair, and the site detector against the meter under the site pair. + +Arming both at N=1 would be two latches and two banners for one fault, and a +false-positive record measured on **one** comparison spent twice. + +## 2. What this is NOT — the two neighbours, named + +Two guards that overlap without saying so is how a false-alarm generator gets +built, so both are written down. + +### It is not [[SAFETY-31]] + +| | **SAFETY-31 — `circulation.py`** | **SAFETY-30 — the per-unit half** | +|---|---|---| +| compares | the units **against each other**: `Σ\|Bᵢ\| − \|ΣBᵢ\|` over measured power | each unit against **its own command** | +| catches | two inverters pushing **through** each other; nets to zero at the meter | a unit **not doing what it was told**; that unit is faulty or wedged | +| attributes | nothing — the quantity is defined over the roster and is identically zero for one unit | **the unit**, by name, and parks it | +| scope | SITE claim, flat latch | UNIT claim, namespaced latch | + +They genuinely overlap in **one** scenario — a unit reversing while its sibling +pulls the other way — and `test_watchdog.py` part 4 asserts exactly that, on the +same fixture, with both readings printed: circulating power is over its threshold +**and** SAFETY-30 names B. Neither is the other with a different threshold. Take +two units faithfully executing an allocation with two signs: circulating power is +the whole loop and every per-unit sign check reads perfect agreement. Take one +unit doing the opposite of its share while its sibling covers: this reads the +whole disagreement and circulating power can be zero. + +### It is not the saturation freeze + +`compute()`'s anti-windup test is `|prev_w − actual_w| > saturation_w` (500 W). It +concludes *the inverter cannot follow*, it fires on a merely **saturated** unit as +readily as on a reversed one, it **stops the integrator winding** — an ordinary +control action taken every day — and it neither latches nor parks anything. + +This asks for **opposition**: a sign, on both sides of a threshold, held +continuously. A saturated-but-obedient unit cannot produce it. And the two meet +at a number rather than overlapping in a band nobody drew — see §3. + +## 3. The numbers, and where they come from + +Read off `admin/goodwe`'s `sim/` two-master rig (TEST-19), used **read-only** and +run container-free. Real and imported, never transcribed: the control law +(`control.compute`, `Tuning`), the allocator (`arbiter.allocate_record` → +`allocator.plan`), the sign predicate (`watchdog.disagrees`), the wire +(`frame_check.setpoint_atomic`, the firmware's own encoder), the plant +(`plantsim.Plant`), and the per-unit reading, taken from **holding register +0x0518** — the register the live `batt_entity` binds. + +**1890 closed-loop runs in which NOTHING is faulty** — every unit executes the +share the allocator gave it — over **6 gains × 5 slews**, the whole range +`config.yaml` accepts for `gain` `float(0.05,1.0)` and `slew_w` `int(50,5000)`, +across both families of stimulus that can produce legitimate opposition: + +* **1050 load reversals** — 7 sizes (±300 W … ±4000 W) × 5 write staggers, the + serialised write path landing unit B's number up to four cycles after A's; +* **840 flapping loads** — 7 periods (10 s … 600 s) × 4 amplitudes. + +Every run is executed **twice**: once at a roster of two and once at a roster of +one on the same stimulus. **The N=1 arm is the control**, and it is the reason +this document can separate *"a property of the check that shipped"* from *"a +property of the second inverter"* instead of asserting one and hoping. + +Supervised caps are the shipped `max_w` (2000 W per unit), so the site rail is +2000 W at N=1 and 4000 W at N=2 — `Controller.site_max_w`, the roster sum. + +### Reproducing it + +The harness is **not committed here**, for the reason SAFETY-31's was not: it +imports `plantsim` and `frame_check`, which live in the other repository and are +not present in CI. What it does is small enough to restate: build two +`plantsim.Inverter`s from `INVERTER_A` / `INVERTER_B`, close the loop with the +add-on's own `compute()` and `arbiter.allocate_record()`, put each share on its +own wire through `Inverter.command()`, advance `Site.settle()` one control cycle, +and record `(standing command, inv.reg(0x0518))` per unit and +`(Σ standing, site.grid)` for the site. Then, for each candidate threshold, the +**longest run of consecutive cycles** on which the shipped `disagrees()` is true — +not a count of true samples, because the watchdog's window is cleared by any +disagreeing sample. Point it at a checkout with `GOODWE_ADDON_REPO`. +⚠️ MUTATION.md's note applies: those harnesses write `__pycache__` into **both** +trees. + +### A. PER-UNIT — a unit's standing command vs its own 0x0518 power, N=2 + +| threshold | worst continuous, reversals | worst continuous, flapping | runs crossing (of 1890) | +|---:|---:|---:|---:| +| 50 W | 25.0 s | 20.0 s | 962 | +| 100 W | 20.0 s | 15.0 s | 511 | +| 150 W | 15.0 s | 15.0 s | 379 | +| 200 W | 10.0 s | 15.0 s | 258 | +| **250 W** | **10.0 s** | **10.0 s** | **194** | +| 300 W | 10.0 s | 10.0 s | 124 | +| 500 W | 5.0 s | 5.0 s | 37 | +| 700 W | 5.0 s | 5.0 s | 16 | +| 1000 W | 0.0 s | 0.0 s | 0 | + +Largest opposed magnitude anywhere in the 1890 runs: **875 W**. + +**The curve is flat, and that is the finding that decided the shape of this +pair.** Unlike the site comparison — where the meter is driven by the LOAD and a +flapping load can hold it opposed for minutes — a unit's own power **chases its +own command** through a bounded ramp and a first-order lag, so the opposition can +only last as long as the crossing does. Dropping the threshold from 300 W to +50 W buys 15 s of extra transient and nothing else. **So the work is done by the +window, and the threshold is put where the control law already draws its line.** + +#### `unit_sign_disagree_w: 250` — schema `int(50,5000)` + +`saturation_w / 2`. The check gates on **both** sides being over `min_w`, and the +two are **opposed**, so the smallest command-versus-power gap it can ever see is +`2 × min_w`. At 250 W that is exactly the 500 W at which `compute()`'s own +anti-windup test already concludes the inverter is not following. **This watchdog +therefore cannot fire on a divergence the shipped control law itself calls +ordinary tracking**, and the two guards meet at a number. + +Rejected: a threshold above the observed 875 W maximum, which would gate nothing +legitimate in — and would also be blind to a unit reversing a 900 W share. + +#### `unit_sign_disagree_s: 85` — schema `int(30,1800)` + +Three bounds, and the value clears all three. + +* **lower, from the rig — 10.0 s.** Two control cycles, the worst continuous + legitimate opposition above 250 W across all 1890 runs. It is a reversing site + whose second write lands a cycle after the first, both units then slewing + through zero behind a 10 s lag. Nothing is wrong. +* **lower, from the firmware — 30 s (`inverter.LINK_STALE_S`).** A unit not + executing its share for less than the firmware's own veto window is a *link* + fault the add-on already answers by dropping the unit from the split. Firing + inside that window would raise a second, louder alarm about a state already + being handled — the SAFETY-15/SAFETY-19 failure exactly. +* **upper, from SAFETY-02 itself — 180 s, the site hold.** A unit that is not + doing what it was told must not be parked *later* than the site would be + parked for the same watts, and this conclusion needs nothing to net out: it is + complete on the first cycle it appears. + +**85 s** is seventeen control cycles: **8.5× the worst legitimate episode**, +**2.8× the firmware veto**, and **under half** the site hold. Deliberately not +60, 75, 90, 120 or 180. + +### B / C. SITE — Σ standing commands vs the meter, at BOTH rosters + +**Load-reversal family** (1050 runs each), worst continuous legitimate opposition: + +| threshold | **N=2** | **N=1 (control)** | ratio | +|---:|---:|---:|---:| +| 100 W | 335.0 s | 175.0 s | 1.91× | +| 200 W | 260.0 s | 120.0 s | 2.17× | +| **300 W** | **215.0 s** | **110.0 s** | **1.95×** | +| 500 W | 155.0 s | 90.0 s | 1.72× | +| 700 W | 130.0 s | 70.0 s | 1.86× | +| 1000 W | 85.0 s | 35.0 s | 2.43× | + +**The second inverter roughly DOUBLES the worst legitimate crossing, and at the +shipped 300 W threshold it takes it from 110 s — comfortably inside the 180 s +hold — to 215 s, which is outside it.** + +The mechanism is not mysterious and is worth stating because it predicts the +ratio: the transient the 180 s hold exists to reject is the loop crossing zero +after a load reversal, and the command has to travel the whole **site rail** at +`slew_w` per cycle. At N≥2 that rail is `Controller.site_max_w`, the roster sum, +while `slew_w` is unchanged. The worst case is the corner of the config space +where that takes longest — `gain: 0.05`, `slew_w: 50`, both **schema floors** — +against a ±2200 W load reversal, which is an oven and a heat pump. + +#### `site_sign_disagree_w: 300` — CARRIED, and here is the justification + +The argument behind 300 W is about **noise**: *"near zero the sign is noise, and +near zero is where a healthy loop LIVES"* — the deadband holds the command still +while the meter wanders either side of `target_grid_w`. `deadband_w` is 15 W and +there is one P1 whatever is behind it. **Nothing in that argument depends on the +roster**, and the campaign agrees: the fraction of samples gated in at 300 W is +within a few percent at both rosters. Carried, with a measurement behind the +carry rather than an assumption. + +#### `site_sign_disagree_s: 325` — NOT carried, and that is the point of the card + +180 s is **inside** the 215 s the two-inverter campaign measures, at a +configuration the schema accepts. Carrying it would have shipped a false trip on +the check with the best record in the safety table — which is the exact failure +this card was opened to prevent, arriving from the other direction. + +**325 s** is 1.51× the measured worst. It is well inside the **900 s** quarter +that bills the capacity tariff, and it is inside **every** episode the fault day +contains: 2026-08-20's four clusters run 16, 30, 77 and 77 minutes, and its +opposed runs reach 1750 s. What it costs is stated rather than hidden — about two +and a half extra minutes of wrong-direction **site** power before the stop — +and it is bought to remove a false trip a legal configuration produces. +Deliberately not 300 or 360. + +**Rejected alternative:** keep 180 s and raise the site threshold to 1000 W, +where the worst is 85 s. It gives a better margin and a faster stop, and it is +wrong: it would be raising a **noise** gate to reject a **transient**, which is +what the window is for, and it would leave the site check blind to a 900 W +whole-site reversal at a 4 kW site. + +### A limitation this campaign found, which SAFETY-30 does not introduce and does not fix + +The **flapping-load** family behaves completely differently for the SITE +comparison, and the control arm is what makes it interpretable: + +| threshold | **N=2** | **N=1 (control)** | +|---:|---:|---:| +| 300 W | 980.0 s | 1000.0 s | +| 700 W | 975.0 s | 1000.0 s | +| 1000 W | 950.0 s | 1000.0 s | + +A square wave whose half period is about one control cycle opposes the command on +**every** cycle: the command chases the previous half-cycle's meter while the +meter has already reversed. **No window inside the billing quarter survives it and +no threshold up to 1000 W does either — at EITHER roster.** The N=1 column is +this document's whole reason for existing: this is a property of comparing a +lagged command against a meter, and it has nothing to do with the second +inverter. + +It is not acted on here, in order of weight: it is **identical at the roster this +add-on ships**, so acting on it would be changing a live safety path on evidence +that has nothing to do with this card; a ~6 kW square wave at 0.1 Hz sustained +for a quarter of an hour is not a house (the captured healthy day's longest +opposed run is **30 s**); and the fix is not a number — it wants the detector to +notice that the disagreement is **alternating**, which is a different check. +**It is a card, and this section is its evidence.** + +### D. The fault arm — because a survey of healthy data is not a derivation + +210 runs with inverter B executing the **negation** of its share throughout, +sibling faithful, meter left to net: + +| threshold | longest run the per-unit check detects | runs the SITE check also sees | +|---:|---:|---:| +| 250 W | 890.0 s | 210 / 210 | +| 300 W | 890.0 s | 180 / 210 | +| 700 W | 885.0 s | 112 / 210 | +| 1000 W | 885.0 s | 60 / 210 | +| 1500 W | 880.0 s | 45 / 210 | +| 2000 W | 0.0 s | 39 / 210 | + +At the shipped 250 W the fault is detected continuously for the whole run in +every one of the 210 — an 85 s hold has ten times the evidence it needs. The +right-hand column is the honest half: on a *reversing* stimulus the site +comparison often does see something too, because the reversal is large enough to +move the meter. **The case it cannot see is the steady one**, and that is the +fixture `test_watchdog.py` part 4 runs: the shipped comparison is false on +**0 of 14** cycles while the per-unit check convicts B on every one. + +## 4. Inverter B does not exist + +Every two-inverter figure in §3 was produced with **inverter A's ramp and lag +copied onto B** (`plantsim.INVERTER_B`, `measured=False`); its 3000 W rating, +where this repository's fixtures use one, is a **hypothesis marked as one**. +[[SPIKE-04]] measures the real ones and is the revisit trigger for all four +numbers — not a reason to move any of them now. + +Which figures depend on it, stated so nobody has to work it out: + +* **`unit_sign_disagree_s: 85`** — most. Its lower bound is the 10.0 s crossing, + which is a direct function of B's ramp and lag. A slower second unit lengthens + it; the 8.5× is what buys headroom for that. +* **`site_sign_disagree_s: 325`** — yes, but less directly. Its 215 s is + dominated by the **rail** and by `slew_w`, which are configuration, not + hardware; B's dynamics move it at the margin. +* **`unit_sign_disagree_w: 250`** — **no.** It is `saturation_w / 2`, read off + the add-on's own control law. The rig only confirms that nothing legitimate + sits over it for long. +* **`site_sign_disagree_w: 300`** — **no.** It is the meter's noise floor and the + deadband, neither of which is a property of any inverter. + +## 5. Mutation ledger + +Every run on a **freshly copied tree of its own, no `__pycache__`, mounted +`:ro`** into the CI image — the structural immunity MUTATION.md records. ⚠️ The +first battery used ONE shared directory rebuilt between mutations and reported a +**RED baseline**; a container mounted it mid-copy and read a missing +`config.yaml` as a failing check. That whole run was discarded: MUTATION.md's +rule is that a run which could have been racing is not a result, and a battery +whose baseline is not green establishes nothing. The harness now copies a +directory per mutation and **refuses to report anything if the baseline is not +green**. + +Byte deltas measured, not estimated; the `+0 B` ones are marked, since a +size-preserving edit is what TEST-15 turned into a false survivor once already. + +| # | file | mutation | Δ | result | +|---|---|---|---|---| +| M1 | `main.py` | the per-unit comparison put back to `self.grid` at every roster | −30 B | **RED** — `test_watchdog.py`. This is the card | +| M2 | `main.py` | the per-unit comparison forced to `inv.batt` at N=1 too | −31 B | **RED** — AC 8 | +| M3 | `main.py` | the site half never evaluated | −6 B | **RED** — AC 3 | +| **M4** | `main.py` | **the site half armed at N=1 as well** | **+0 B** | **SURVIVOR on the first battery — GAP CLOSED**, then RED. See below | +| M5 | `main.py` | the `sign_site` claim never filed | −6 B | **RED** — `test_watchdog.py`. ⚠️ `test_peakguard.py` stayed **GREEN**, correctly: 10f's AST scan pins which sources CAN be filed and the append is still written; whether it is REACHED is a dynamic question no fixture there gets to | +| M6 | `main.py` | the site latch not persisted to the store | −58 B | **RED** — the flat-keys and restart checks | +| M7 | `main.py` | the operator clear leaves the site latch set | −42 B | **RED** — the clear check | +| M8 | `watchdog.py` | `arm()` clears `disagree_since` | +35 B | **RED** — "re-armed on every one of 38 cycles still trips" | +| M9 | `watchdog.py` | the reason hard-codes "a meter reading" again | +1 B | **RED** — `test_watchdog.py`. `test_inverter.py` stayed **GREEN**; its part-3 row reads the noun off `sign_wd.against` rather than out of the reason string, so the two checks bite in different places | +| M10 | `watchdog.py` | `unit_sign_disagree_w` 250 → 300 | **+0 B** | **RED** — the `saturation_w / 2` pin and the CFG-03 pin | +| M11 | `watchdog.py` | `site_sign_disagree_s` 325 → 180, i.e. **the number carried** | **+0 B** | **RED** — the measured-215 s check and the CFG-03 pin | +| M12 | `arbiter.py` | `sign_site` scoped `SCOPE_UNIT` | **+0 B** | **RED** — `test_arbiter.py` **and** `test_peakguard.py` | +| M13 | `validate.py` | the `unit_sign_disagree_w` rule removed | −126 B | **RED** — `test_validate.py` | +| **M14** | `main.py` | **`arm()` handed `self.sign_cfg` at EVERY roster** — the WIRING, not the values | −31 B | **SURVIVOR at review — GAP CLOSED**, then RED. See below | +| M15 | `main.py` | per-unit `arm()` given the right `min_w` and the SHIPPED hold — a HOLD-ONLY mis-wire | +78 B | **RED** — the same wiring pin | +| M16 | `main.py` | per-unit `arm()` given the right `min_w` and the SITE hold — hold-only again, and behaviour-preserving | +83 B | **RED** — the wiring pin, and the ONLY red in `test_watchdog.py` | + +### M4 is recorded as a survivor because it was one + +The check it should have gone red on read: + +```python +not _c1s.site_sign_wd.tripped and _c1s.site_sign_wd.disagree_since is None +``` + +Arming the site detector at a roster of one left that **green**, and the reason +is worth reading because the check looked airtight. In the N=1 fixture the unit +detector trips first at its 1 s hold and parks the inverter; the site's standing +command is then **0 W**, so the site predicate is false, its window is never +opened, and `disagree_since is None` is true *whether or not the detector was +evaluated at all*. **The assertion ran, the sentence was true, and the data +could not reach the failure it named** — MUTATION.md's third failure mode, +inside the check written to prevent it. + +Replaced with a **call counter installed on the object** before the run: twelve +cycles, zero calls, printed. A counter can only be zero for one reason. The +mutation was re-run against it and is red; the run that found it was discarded +along with its tree, so this is a result and not an artefact. + +### M14: the VALUES were pinned and the WIRING was not + +M10 and M11 mutate the numbers in `watchdog.SIGN_OPTION_DEFAULTS` and both are +red through the config pins. Neither of them touches the line that DELIVERS +those numbers to a detector: + +```python +inv.sign_wd.arm(self.sign_cfg if one else self.unit_sign_cfg, + AGAINST_METER if one else AGAINST_OWN_POWER) +``` + +Drop the conditional — `arm(self.sign_cfg, ...)` — and every unit is armed with +SAFETY-02's meter-facing pair instead of the re-derived per-unit one. At review +that left **all 19 files green**. `grep -rn "sign_wd.cfg" test_*.py` found +exactly one threshold assertion, and it is §7's N=1 arm — `(300.0, 1.0)` — +which the mutation PRESERVES. Nothing anywhere read `inv.sign_wd.cfg` at a +roster of two. + +The behavioural checks in part 3 cannot close it and it is worth saying why: B +is reversed at ~2000 W and every hold in that fixture is short, so B trips under +either pair. **A fault big enough and long enough to trip under both thresholds +is evidence about neither.** + +What the survivor costs in the field is the whole product of §3: at +`config.yaml`'s own defaults the per-unit check would run at `sign_disagree_s` +**180 s instead of 85 s** — 2.1× slower to park a reversed inverter — and its +gate would move from 250 W to 300 W, so a unit reversing a share between the two +becomes invisible, while `SIGN_OPTION_DEFAULTS`, `config.yaml` and §3 all go on +describing a pair the running code does not use. With CI green. + +**Closed by a wiring pin in `test_watchdog.py` part 3**, on the real +two-inverter Controller: every `inv.sign_wd.cfg` equals the per-unit pair, +`site_sign_wd.cfg` equals the site pair, and each detector's `against` is its +own instrument — all graded **against literals**, since `== self.unit_sign_cfg` +cannot tell *the per-unit pair* from *whatever this Controller was handed*. + +⚠️ **And the fixture had to be changed for that pin to mean anything.** +`_sign_site` set `unit_sign_disagree_s: 1` **and** `site_sign_disagree_s: 1` +**and** `sign_disagree_s: 1`, with two of the three magnitude gates at 300 W. A +bare equality there discriminates on `min_w` alone: a mis-wiring that delivered +the right magnitude with the wrong hold would compare EQUAL to the pair it was +supposed to have. That is this document's own §5 lesson and MUTATION.md's third +failure mode — a check comparing two things that happen to be equal cannot see +the difference it names — reappearing inside the fix for it. The three pairs are +now pairwise distinct **in both coordinates** (250 W / 1 s, 320 W / 2 s, +300 W / 3 s), a PRECONDITION check asserts that distinctness so a later edit +collapsing them goes red rather than quietly vacuous, and both pairs are +**printed**. M15 and M16 exist to show the hold half actually bites: right +magnitude, wrong hold, red both times — and M16, whose wrong hold still trips +inside the drive, is red on the wiring pin and on **nothing else** in that file. + +## 6. What this card does NOT close + +* **The flapping-load blind spot in the SITE comparison.** §3, at length. Present + identically at the shipped roster of one; wants an alternation test, not a + number; a card of its own. +* **A unit misbehaving while it is NOT loop-driven.** Gate 1 (`following`) is + kept for the per-unit comparison, deliberately: the whole no-false-positive + argument is built on it, and the saturation freeze already grades that window. + So between the start of a maintenance cycle and its end, a unit reversing + against its own command is the freeze's to notice. Widening it is a card with a + rig campaign behind it, not an edit. +* **A sign inversion downstream of this add-on** — in the firmware, on the wire, + or in a future driver. Unchanged: our view stays self-consistent and both + comparisons stay silent. That is the witness's problem and it always was. +* **A false-alarm interaction with SAFETY-31 that nobody has to think about.** + They overlap in one scenario and both fire in it. That is correct — they + conclude different things — but an operator will see two alarms for one event. + Whether the banner should say so in one sentence is a UI card. diff --git a/goodwe_controller/DOCS.md b/goodwe_controller/DOCS.md index 99547f3..4315368 100644 --- a/goodwe_controller/DOCS.md +++ b/goodwe_controller/DOCS.md @@ -363,6 +363,10 @@ behind it would trip the firmware watchdog on a system that is working fine. | `auto_start` | false | Start controlling on boot (only after commissioning) | | `sign_disagree_w` | 300 | Both the command and the meter must exceed this before a sign difference counts. Must stay **below `max_w`** or the watchdog can never fire, and the add-on refuses to start | | `sign_disagree_s` | 180 | How long they must stay opposed, continuously, before the failsafe trips | +| `unit_sign_disagree_w` | 250 | **Two or more inverters only.** Both a unit's own share and its own measured power must exceed this before their sign difference counts. It is `saturation_w / 2`, so the smallest divergence it can see is exactly the gap at which the loop already calls the inverter "not following". Must stay **below `max_w`** or it can never fire, and the add-on refuses to start | +| `unit_sign_disagree_s` | 85 | **Two or more inverters only.** How long one unit's command and its own power must stay opposed before that unit is parked | +| `site_sign_disagree_w` | 300 | **Two or more inverters only.** Threshold for the site-wide check - the sum of the commands against the meter. The same 300 W as `sign_disagree_w`, for the same reason: near zero the sign is noise | +| `site_sign_disagree_s` | 325 | **Two or more inverters only.** How long the site's total command and the meter must stay opposed before every inverter is parked. Longer than `sign_disagree_s` because the command has to cross zero over a rail that is the sum of the roster, at a `slew_w` that is not - measured at 110 s with one inverter and 215 s with two | #### The sign-disagreement failsafe @@ -374,6 +378,43 @@ there. The banner reads `FAILSAFE`, the commissioning list says which two numbers disagreed and for how long, and the `Controller status` entity reports `failsafe`. +#### ⚠️ At two or more inverters this is TWO checks, not one (SAFETY-30) + +The paragraph above describes what a **one-inverter** install does, and it is +unchanged: there is one meter, one actuator, and comparing the command against +the meter is comparing a thing with a view of itself. + +With a second inverter that stops being true. The meter answers for **both** +units, so a command put on one of them can be opposed to it with nothing wrong +(inverter A charging while inverter B discharges more), and - the worse +direction - a unit genuinely driving the wrong way can be **hidden**, because its +sibling's watts arrive at the meter and the sum looks healthy. So the check +splits in two, and they are different faults: + +- **Per inverter** — that unit's own share against **its own measured power** + (the `batt_entity` reading, register 0x0518). This catches *"this inverter is + not doing what it was told"*. Only that inverter is parked; the site keeps + running on the one that works. Thresholds: `unit_sign_disagree_w` / + `unit_sign_disagree_s`. +- **Site-wide** — the **sum** of what every inverter has standing, against the + meter. This catches *"the site as a whole is pushing the wrong way"*, which no + per-inverter check can see because every unit may be faithfully executing its + share of a number that is itself wrong. **Every** inverter is parked and the + banner names the site rather than an inverter. Thresholds: + `site_sign_disagree_w` / `site_sign_disagree_s`, latched under flat + `site_sign_tripped` in `/data/state.json`. + +Both latch, both are cleared by the same *stop, then start*, and both survive a +restart. **On a one-inverter install neither pair does anything at all** - the +two questions are the same question there, and only the check described above is +armed. + +This is **not** the circulating-power alarm further down: that one compares the +inverters *against each other* and catches them pushing power through each other, +which nets to zero at the meter and which no command is evidence about. It is +also not the saturation freeze, which is a *magnitude* test that shapes the loop +rather than a *direction* test that parks an inverter. + **It does not clear itself, and restarting the add-on does not clear it either.** Pressing *Start / stop control* twice — stop, then start — is the only thing that does. That is deliberate on both counts: diff --git a/goodwe_controller/app/arbiter.py b/goodwe_controller/app/arbiter.py index a1ebe42..abcbc5b 100644 --- a/goodwe_controller/app/arbiter.py +++ b/goodwe_controller/app/arbiter.py @@ -151,6 +151,30 @@ CLAIM_SCOPE = { # other went on feeding the loop. The evidence is site-level and so is the # remedy: stop commanding, everywhere, and say why. "circulation": SCOPE_SITE, + # main.py: `Claim.set("sign_site", P_SAFETY, 0.0, "site sign disagreement")`. + # SAFETY-30, and it is the twin of `failsafe` below rather than a duplicate + # of it - the two exist BECAUSE the same words describe two different + # measurements once there is more than one inverter. + # + # `failsafe` grades ONE unit's standing command against THAT UNIT'S OWN + # measured power (register 0x0518). Its conclusion is "the actuator I am + # driving is not doing what I told it", which is a fact about one device, + # so it parks one device. + # + # This one grades the SUM of the standing commands against P1. Its + # conclusion is "the site as a whole is driving the wrong way" - an inverted + # gain, an integrator wound past the rail, a strategy on the wrong side of + # the meter - and there is nothing per-unit about it: the quantity is a sum + # over the roster and the offending number is the site target, which no + # single inverter owns. Parking one unit on that evidence would leave the + # rest of the site pushing the same way with the guard satisfied. + # + # ⚠️ AT N=1 IT IS NEVER FILED, and that is the identity rather than a + # special case: with one inverter the sum of the commands IS that unit's + # command and `failsafe` already carries exactly this trip, under the keys, + # the banner and the claim an install has had since SAFETY-02 shipped. + # `Controller.cycle()` does not even arm the site detector there. + "sign_site": SCOPE_SITE, # --- UNIT: about one pack, one inverter, one device ------------------- # main.py: `Claim.limit("device", P_SAFETY, self.dev_min, self.dev_max, @@ -164,13 +188,22 @@ CLAIM_SCOPE = { # same reason, so it is scoped like `device`. "supervised": SCOPE_UNIT, # main.py: `Claim.set("failsafe", P_SAFETY, 0.0, "sign disagreement")`. - # ⚠️ THE EVIDENCE IS SITE-WIDE AND THE REMEDY IS NOT, and that is why this - # entry needs its reasoning written down. SAFETY-02 compares P1 - a site - # measurement - against `standing`, THE COMMAND THIS CONTROLLER PUT ON THIS - # ACTUATOR. What it concludes is "the unit I am driving is pushing power the - # wrong way", and what it does is park that unit. At N>=2 the same evidence - # cannot be attributed to a sibling nobody here commands, so a site-wide - # remedy would be this watchdog stopping an inverter it never graded. + # ⚠️ THE EVIDENCE IS PER-UNIT AND SO IS THE REMEDY - AND SINCE SAFETY-30 THE + # FIRST HALF OF THAT SENTENCE IS TRUE. It used to read "the evidence is + # site-wide and the remedy is not", because the watchdog compared THIS + # unit's standing command against P1: a site measurement, attributed to one + # actuator on the strength of §6.2's N=1 identity. That identity does not + # survive a second inverter - at N>=2 the meter answers for a sibling this + # unit's command says nothing about, in both directions (a false trip when + # the sibling covers the meter, and a missed one when the sibling MASKS a + # unit that really is reversed). + # + # So at N>=2 the comparison behind this claim is THIS unit's command against + # THIS unit's own measured power - register 0x0518, per-instance since + # DEV-02 - and the scope that was already right for the wrong reason is now + # right for the right one. The site-wide half moved to `sign_site` above. + # At N=1 the two are the same comparison and this claim is the one filed, + # exactly as it shipped. "failsafe": SCOPE_UNIT, # main.py: `Claim.set("maintenance", P_MAINTENANCE, ...)` when the cycle owns # the setpoint, and `Claim.limit("maintenance", P_MAINT_SHAPE, hi=...)` when diff --git a/goodwe_controller/app/main.py b/goodwe_controller/app/main.py index 334af98..1e18ad9 100644 --- a/goodwe_controller/app/main.py +++ b/goodwe_controller/app/main.py @@ -64,7 +64,10 @@ from .tariff import PeakStore, PeakTracker from .store import migrate_unit_keys from .validate import (ConfigRefused, refuse, validate_startup_config, validate_unit_ranges) -from .watchdog import SignWatchdogConfig, loop_is_following +from .watchdog import ( + AGAINST_METER, AGAINST_OWN_POWER, SIGN_OPTION_DEFAULTS, + SignWatchdog, SignWatchdogConfig, loop_is_following, +) from . import web OPTIONS_PATH = "/data/options.json" @@ -173,10 +176,42 @@ class Controller: # first or every existing tripped install comes up ARMED, into the # fault, with a green banner - which is the exact failure the latch was # persisted to prevent, rebuilt by a key rename. - sign_cfg = SignWatchdogConfig( + # SAFETY-02's pair, and since SAFETY-30 it is THE SITE COMPARISON's + # pair - which at the shipped roster of one is the same thing under a + # more careful name. It is what `Inverter.sign_wd` is armed with at N=1, + # byte for byte, and what the site detector is armed with at N>=2. + self.sign_cfg = sign_cfg = SignWatchdogConfig( min_w=float(opts.get("sign_disagree_w", 300)), hold_s=float(opts.get("sign_disagree_s", 180)), ) + # SAFETY-30. THE PER-UNIT COMPARISON'S OWN PAIR, RE-DERIVED AT TWO + # INVERTERS AND NOT CARRIED ACROSS. `sign_disagree_w` / `sign_disagree_s` + # were measured on two SINGLE-INVERTER bus captures against a command + # opposed to THE METER; this comparison is a command against ONE UNIT'S + # OWN POWER, where the magnitudes, the transient and the worst + # legitimate episode are all different quantities. watchdog.py's + # SIGN_OPTION_DEFAULTS carries the campaign that produced these, and + # DECISIONS/SAFETY-30-per-unit-sign.md carries the tables. + self.unit_sign_cfg = SignWatchdogConfig( + min_w=float(opts.get("unit_sign_disagree_w", + SIGN_OPTION_DEFAULTS["unit_sign_disagree_w"])), + hold_s=float(opts.get("unit_sign_disagree_s", + SIGN_OPTION_DEFAULTS["unit_sign_disagree_s"])), + ) + # SAFETY-30. THE SITE COMPARISON'S OWN PAIR, USED ONLY AT N>=2. The + # threshold is the shipped 300 W - carried, because the argument behind + # it is about the meter's noise floor and the deadband, neither of which + # the roster touches - and the WINDOW is not, because the transient it + # rejects is the loop crossing zero at `slew_w` per cycle across a rail + # that IS the roster sum. Measured at both rosters on the same stimuli: + # 110 s at one inverter, 215 s at two, against a 180 s hold. See + # watchdog.SIGN_OPTION_DEFAULTS. + self.site_sign_cfg = SignWatchdogConfig( + min_w=float(opts.get("site_sign_disagree_w", + SIGN_OPTION_DEFAULTS["site_sign_disagree_w"])), + hold_s=float(opts.get("site_sign_disagree_s", + SIGN_OPTION_DEFAULTS["site_sign_disagree_s"])), + ) self.store_migration = migrate_unit_keys(store, UNIT_A) self.inverters = build_roster(opts, store, sign_cfg) # ⚠️ CFG-04. SAY WHICH SHAPE IS IN FORCE, BECAUSE THE OTHER ONE'S KEYS @@ -261,6 +296,45 @@ class Controller: "opposed the meter.", inv.name, inv.sign_wd.reason or "(reason not recorded)") + # --- SAFETY-30: THE SITE HALF OF THE SIGN CHECK ---------------------- + # ⚠️ ONE PER SITE, BESIDE THE ONE PER INVERTER, AND ARMED ONLY AT N>=2. + # The per-unit watchdogs above ask "is THIS actuator doing what I told + # it", graded on that unit's own measured power. This one asks "is the + # SITE driving the wrong way", graded on the SUM of the standing + # commands against P1 - the comparison SAFETY-02 shipped, re-scoped to + # the quantity it was always actually about. + # + # ⚠️ AT A ROSTER OF ONE IT IS CONSTRUCTED AND NEVER EVALUATED, and that + # is the whole compatibility promise of this card rather than an + # oversight. With one inverter the sum of the commands IS that + # inverter's command, so arming this would be arming a duplicate of + # `inverters[0].sign_wd`: two latches for one fault, two banners, and a + # false-positive record measured on ONE comparison being spent twice. + # `cycle()` is where that is enforced and says so at the line. + # + # Restored from the store like SAFETY-02's and SAFETY-31's latches, for + # the reason store.py gives: "restart the add-on" is the first thing an + # operator looking at a FAILSAFE banner tries, and a latch held only in + # memory brings the controller back armed, into the fault, unattended. + # FLAT keys, not namespaced - store.py carries that argument too. + self.site_sign_wd = SignWatchdog( + self.site_sign_cfg, + tripped=bool(store.data.get("site_sign_tripped", False)), + reason=str(store.data.get("site_sign_reason", "") or "")) + if self.site_sign_wd.tripped: + _LOG.error( + "SAFETY-30 (site) was TRIPPED before this restart and stays " + "tripped: %s. Every inverter is held at 0 W. Restarting does " + "not clear this - start control from the UI once you know why " + "the site's total command opposed the meter.", + self.site_sign_wd.reason or "(reason not recorded)") + # And the per-unit detectors are told which question they are asking, + # here as well as in `cycle()`. `checks()` and the status page read + # `inv.sign_wd.cfg` and can be called before the first cycle has run, so + # a roster of two would otherwise advertise the site thresholds on a + # detector that is about to be graded against a unit's own power. + self._arm_sign_watchdogs() + # SAFETY-18. The frozen-but-fresh meter detector - see freeze.py for why # it is a conjunction and not an `unchanged_s` threshold, and for the # measurements behind the default. @@ -712,6 +786,45 @@ class Controller: def sign_wd(self): return self._only().sign_wd + # -- SAFETY-30: which comparison each sign detector is making ------------- + def _arm_sign_watchdogs(self) -> None: + """Tell every sign detector which of SAFETY-30's two questions it asks. + + ⚠️ A PURE FUNCTION OF `len(self.inverters)`, AND THE ONLY PLACE THE + ROSTER DECIDES ANYTHING ABOUT THIS CHECK. Written as one method rather + than as a conditional at the two call sites for `loop_is_following()`'s + reason: the arrangement that does not keep two things in step is the one + where they are written down twice. + + AT ONE INVERTER - §6.2's identity, and the whole of AC 8. The single + unit's detector is armed against THE METER with the shipped + `sign_disagree_w` / `sign_disagree_s`, which is what it has always been + armed against, and the site detector is not armed at all. A one-inverter + install therefore makes the same comparison, at the same thresholds, + writes the same reason string, latches under the same keys and files the + same `failsafe` claim as it did before this card. + + AT TWO OR MORE - the identity is gone and the meter stops being a view + of any one actuator. Each unit's detector is armed against ITS OWN + MEASURED POWER (`Inverter.batt`, register 0x0518) under the re-derived + per-unit pair, and the site detector is armed against the meter under + the site pair. Two questions, two remedies, two latches. + + ⚠️ CALLED AGAIN AT THE TOP OF EVERY `cycle()`, not only from + `__init__`. The roster is re-read every cycle for liveness and for the + allocation, and this repository's two-inverter fixtures append an + `Inverter` AFTER the Controller is constructed - a detector frozen at + construction would then be graded against the meter at a roster of two, + which is the defect this card exists to remove, wearing a fixture's + name. `arm()` touches no timer and no latch, so calling it every cycle + cannot disturb an episode in progress. + """ + one = len(self.inverters) <= 1 + for inv in self.inverters: + inv.sign_wd.arm(self.sign_cfg if one else self.unit_sign_cfg, + AGAINST_METER if one else AGAINST_OWN_POWER) + self.site_sign_wd.arm(self.site_sign_cfg, AGAINST_METER) + # -- the site aggregates, each one named for the question it answers ------ # ⚠️ THESE ARE NOT "THE SCALARS WITH A SUM IN FRONT". Each one picks a # DIFFERENT reduction, because the questions are different, and getting the @@ -1239,17 +1352,44 @@ class Controller: # case where the last write was REJECTED - the inverter is then still # latched on `last_write`, which is exactly what `last_write` holds). # ⚠️ ONE WATCHDOG PER INVERTER, EVALUATED ON THAT INVERTER'S OWN - # STANDING COMMAND. The meter is site-wide - there is one P1 - but what - # the watchdog concludes is "THE UNIT I AM DRIVING is pushing power the - # wrong way", and that evidence cannot be pinned on a sibling this - # controller commanded differently. `inv.target` is that unit's share - # from the last resolution, which at N=1 is the site target itself. + # STANDING COMMAND. What the watchdog concludes is "THE UNIT I AM + # DRIVING is pushing power the wrong way", and that evidence cannot be + # pinned on a sibling this controller commanded differently. + # `inv.target` is that unit's share from the last resolution, which at + # N=1 is the site target itself. + # + # ⚠️ SAFETY-30: AND WHAT IT IS COMPARED AGAINST IS DECIDED BY THE ROSTER. + # This block used to hand every unit's detector `self.grid`. There is + # ONE P1, so at N>=2 that compares a number this controller put on ONE + # actuator against a measurement that answers for ALL of them, and it is + # wrong in both directions: + # + # FALSE TRIP inverter A commanded -1500 W while the meter reads + # +1500 W is evidence of nothing when inverter B is + # discharging 3000 W exactly as instructed. Nothing is + # faulty and the highest-value check in the safety table + # fires - on a record of zero false positives. + # MISSED the converse, and it is worse. A unit commanded +1500 W + # and MEASURED at -1500 W, beside a sibling at +1500 W, + # leaves P1 exactly where a healthy site leaves it. The + # sum destroys the evidence; no threshold recovers it. + # + # So at N>=2 each unit is graded against ITS OWN measured power - + # `inv.batt`, register 0x0518, per-instance since DEV-02 - which is the + # only instrument that can answer "is THIS actuator doing what I told + # it". At N=1 §6.2's identity holds, the meter IS the view of that one + # unit, and the comparison stays the meter, byte for byte. The site's + # own question has not gone away; it is asked below, once, of the sum. + # `_arm_sign_watchdogs()` states the choice and its docstring argues it. + self._arm_sign_watchdogs() + n_units = len(self.inverters) standing = {} for inv in self.inverters: was_tripped = inv.sign_wd.tripped stood = inv.last_write if inv.last_write is not None else inv.target standing[inv.name] = stood - if inv.sign_wd.evaluate(stood, self.grid, self.following, now) \ + against = self.grid if n_units <= 1 else inv.batt + if inv.sign_wd.evaluate(stood, against, self.following, now) \ and not was_tripped: # ⚠️ REASON FIRST, FLAG SECOND, and each `set` writes the file. # The ordering is the whole atomicity story: a crash between the @@ -1290,6 +1430,50 @@ class Controller: # share of it. At the shipped roster of one the two are the same number. site_standing = float(sum(standing.values())) + # --- SAFETY-30: THE SITE HALF OF THE SIGN CHECK ---------------------- + # The comparison SAFETY-02 shipped, asked of the quantity it was always + # about: the TOTAL this controller has standing on the wire, against the + # meter that total is answerable for. What it catches is a fault no + # per-unit check can see, because every unit is faithfully executing its + # share of a site target that is itself pointing the wrong way - an + # inverted gain, an integrator wound past the rail, a strategy on the + # wrong side of the meter. That is the 2026-08-20 fault day, and it is + # the reason the site question survives rather than being replaced. + # + # ⚠️ ONLY AT N>=2, AND THIS BRANCH IS AC 8. At one inverter the sum of + # the commands IS that inverter's command and the loop above has already + # made exactly this comparison, at exactly these thresholds, on the + # detector whose latch, store keys, claim and banner an install has had + # since SAFETY-02 shipped. Arming this as well would be two latches for + # one fault, two banners, and the same false-positive record spent + # twice. A shipped install does not reach this block at all. + if n_units > 1: + was_site = self.site_sign_wd.tripped + if self.site_sign_wd.evaluate(site_standing, self.grid, + self.following, now) and not was_site: + # REASON FIRST, FLAG SECOND - the same atomicity ordering the + # per-unit trip uses above and SAFETY-31 uses below: a crash + # between the two must not persist a latch with no explanation + # for the operator who has to decide about clearing it. + # + # ⚠️ AND THE KEYS ARE FLAT, NOT NAMESPACED, AND NOT THE + # PRE-DEV-02 PAIR EITHER. `sign_tripped` / `sign_reason` are the + # ROLLBACK VIEW of a UNIT latch (store.py rule 2) and mirroring + # a site trip onto them would hand a downgraded add-on - which + # drives exactly one actuator - a latch taken on evidence about + # a roster it cannot have. `site_sign_*` is its own pair, for + # the same reason `circ_*` is. + self.store.set("site_sign_reason", self.site_sign_wd.reason) + self.store.set("site_sign_tripped", True) + self.log_event("FAILSAFE site: " + self.site_sign_wd.reason) + _LOG.error("SAFETY-30 TRIPPED at the SITE - %s. That is the " + "SUM of every standing command opposing the meter, " + "which no single inverter is answerable for. " + "Commanding 0 W on every inverter and holding it " + "until control is started again. This survives an " + "add-on restart; restarting will not clear it.", + self.site_sign_wd.reason) + # --- SAFETY-31, AND IT RUNS BESIDE SAFETY-02 FOR THE SAME REASON ----- # Evaluated EVERY cycle, unconditionally, above the "inputs missing but # not yet stale" hold below. A run of held cycles would otherwise leave @@ -1329,8 +1513,14 @@ class Controller: # list below, because its `failsafe` SET is what parks it. One tripped # unit therefore takes the whole roster past the hold, which is the # conservative direction: every other unit's claims are re-filed too. + # ⚠️ AND SAFETY-30's SITE LATCH TAKES THE ROSTER PAST THE HOLD TOO, for + # the same reason: its `sign_site` SET is what parks every inverter, and + # a latched site that returned early here would leave each unit's last + # share standing for the heartbeat to re-emit - i.e. the trip would stop + # acting the moment a poll was missed. if not self.inputs_ok and not any(i.sign_wd.tripped - for i in self.inverters): + for i in self.inverters) \ + and not (n_units > 1 and self.site_sign_wd.tripped): bad_for = (now - self.inputs_bad_since).total_seconds() if self.inputs_bad_since else 0.0 if bad_for < stale_after: # A single missed poll is not a fault. Hold, keep the heartbeat @@ -1384,6 +1574,30 @@ class Controller: "sign disagreement", actuator=self.actuator_of("failsafe", inv))) + # --- SAFETY-30: THE SITE SIGN TRIP, AND IT IS A SITE SET ------------ + # ⚠️ THE TWIN OF THE LOOP ABOVE, NOT A DUPLICATE OF IT. That loop files + # one `failsafe` per unit on the evidence of THAT unit's own measured + # power, and parks that unit. This files ONE claim on the SITE, because + # what tripped is the SUM of every standing command against the meter - + # a fact about the grid connection, whose offending number is the site + # target and which no single inverter owns. Parking one unit on it would + # leave the rest of the roster pushing the same way with the guard + # satisfied. See arbiter.CLAIM_SCOPE, which classifies `sign_site` and + # `failsafe` side by side and says why they differ. + # + # ⚠️ AND THE `n_units > 1` GUARD IS NOT REDUNDANT, EVEN THOUGH `cycle()` + # never arms this detector at a roster of one. The latch is RESTORED + # FROM THE STORE, so an install that ran with two inverters, tripped, + # and was then reconfigured back to one would come up with + # `site_sign_tripped: true` and would otherwise file a claim on evidence + # about a roster it no longer has. Guarded, that install comes up + # driving - which is right: the fault it latched on is not a statement + # about the site it is now. + if n_units > 1 and self.site_sign_wd.tripped: + claims.append(Claim.set("sign_site", P_SAFETY, 0.0, + "site sign disagreement", + actuator=self.actuator_of("sign_site"))) + # --- SAFETY-31: WHAT "ACTED ON" MEANS, AND WHAT IT HONESTLY CANNOT -- # ⚠️ A SITE SET AT P_SAFETY: STOP COMMANDING, EVERYWHERE. Read the next # paragraph before changing it, because the obvious objection is right @@ -2608,8 +2822,15 @@ class Controller: # here would be a surface to migrate later. This sensor is already # discovered, already on every dashboard, and "failsafe" is # additive to its vocabulary rather than a change to it. + # SAFETY-30 adds the site sign latch to the same vocabulary, for + # the reason the note above gives: "failsafe" is additive to a + # sensor that already exists and is already on every dashboard, and + # a P_SAFETY stop that did not reach it would be a site at 0 W + # reporting "running". "status": ("failsafe" if any(i.sign_wd.tripped for i in self.inverters) + or (self.site_sign_wd.tripped + and len(self.inverters) > 1) or self.circ_wd.tripped else "running" if self.auto else "stopped"), # SAFETY-31. ⚠️ MEASURED, not the allocation record - see @@ -2936,24 +3157,70 @@ class Controller: # and a trip on B leaves A commanding. A single row would have to pick # one of them to describe, and the honest answer at a roster of two is # both. At a roster of one the wording is unchanged. + # ⚠️ AND SINCE SAFETY-30 THE ROW SAYS WHAT THE UNIT WAS COMPARED + # AGAINST, because at a roster of two that is no longer the meter and a + # row that still said "the command opposed the meter" would send an + # operator to look at P1 for a fault that is visible only on one + # inverter's own power sensor. `sign_wd.against` is the same string the + # trip sentence uses, so the two cannot drift apart. for inv in self.inverters: tag = "" if len(self.inverters) == 1 else f" [{inv.name}]" + # ⚠️ ADDED ONLY WHEN THERE IS SOMETHING TO DISAMBIGUATE. At the + # shipped roster of one the two sentences below are the ones that + # have always been on the page, word for word - AC 8 is about what + # an operator sees as much as about what the detector computes, and + # "opposed a meter reading" would be a gratuitous change to a line + # somebody has learned to recognise. At N>=2 the noun is load- + # bearing: the row would otherwise send them to look at P1 for a + # fault that is visible only on one inverter's own power sensor. + vs = ("" if len(self.inverters) == 1 + else f" against {inv.sign_wd.against}") if inv.sign_wd.tripped: + what = ("the meter" if len(self.inverters) == 1 + else inv.sign_wd.against) out.append({"ok": False, "warn": False, "text": f"SIGN-DISAGREEMENT FAILSAFE{tag} - " + inv.sign_wd.reason - + ". The inverter is held at 0 W. Find out why the " - "command opposed the meter before starting control " + + f". The inverter is held at 0 W. Find out why the " + f"command opposed {what} before starting control " "again; starting clears this. Restarting the " "add-on does NOT clear it."}) else: held = inv.sign_wd.held_s(datetime.now(timezone.utc)) out.append({"ok": True, "warn": False, - "text": f"sign watchdog{tag}: armed (trips after " + "text": f"sign watchdog{tag}: armed{vs} (trips after " f"{inv.sign_wd.cfg.hold_s:.0f} s opposed, both over " f"{inv.sign_wd.cfg.min_w:.0f} W)" + (f" - opposed for {held:.0f} s now" if held else "")}) + # --- SAFETY-30: the SITE row ----------------------------------------- + # ⚠️ ONLY AT N>=2, and the absence at N=1 is the honest report rather + # than a missing row. With one inverter the site comparison and the row + # above are the same sentence about the same actuator; printing it twice + # would advertise two guards where there is one, which is how an + # operator learns to skim the list. + if len(self.inverters) > 1: + if self.site_sign_wd.tripped: + out.append({"ok": False, "warn": False, + "text": "SITE SIGN-DISAGREEMENT FAILSAFE - " + + self.site_sign_wd.reason + + ". EVERY inverter is held at 0 W. This is the " + "SUM of the commands against the meter, so it is " + "not one inverter's fault to find - look at the " + "loop, the integrator and whatever won the " + "arbitration. Starting control clears this; " + "restarting the add-on does NOT."}) + else: + _sheld = self.site_sign_wd.held_s(datetime.now(timezone.utc)) + out.append({"ok": True, "warn": False, + "text": f"site sign watchdog: armed against the sum of " + f"the commands (trips after " + f"{self.site_sign_wd.cfg.hold_s:.0f} s opposed, " + f"both over " + f"{self.site_sign_wd.cfg.min_w:.0f} W)" + + (f" - opposed for {_sheld:.0f} s now" + if _sheld else "")}) + # --- SAFETY-31 ----------------------------------------------------- # ⚠️ SITE-LEVEL, WHERE THE ROWS ABOVE ARE PER-INVERTER, and the row says # so. There is one circulating figure for the site; naming an inverter @@ -3010,10 +3277,28 @@ class Controller: level, banner = "bad", ("CIRCULATING POWER — the inverters were " "pushing against each other. Site held at " "0 W.") + elif self.site_sign_wd.tripped and len(self.inverters) > 1: + # SAFETY-30's site half, ABOVE the per-unit rows below and below + # circulating power above, and the ordering is the same argument + # both times: the banner carries one sentence, so it carries the one + # that says the widest true thing. A site sign trip means EVERY + # inverter is parked and the fault is in what the site was asking + # for, not in one device - naming a unit here would send an operator + # to swap the wrong box. + level, banner = "bad", ("FAILSAFE — the site's total command " + "opposed the meter. Every inverter held at " + "0 W.") elif any(i.sign_wd.tripped for i in self.inverters): # Outranks "NOT READY": nothing is missing, something went wrong. - level, banner = "bad", ("FAILSAFE — the command opposed the meter. " - "Inverter held at 0 W.") + # ⚠️ THE SENTENCE IS UNCHANGED AT A ROSTER OF ONE, where "the meter" + # is what the unit was graded against. At N>=2 it is that unit's own + # power, so the banner says so - see checks() on why the noun is + # load-bearing rather than cosmetic. + level, banner = "bad", ( + "FAILSAFE — the command opposed the meter. " + "Inverter held at 0 W." if len(self.inverters) == 1 else + "FAILSAFE — an inverter's command opposed its own measured " + "power. That inverter held at 0 W.") elif bad: # ⚠️ `banner` IF THE ROW OFFERS ONE, `text` OTHERWISE. A row may # need to say more to an operator reading the checks list than fits @@ -3118,6 +3403,24 @@ class Controller: # unit in particular. self.store.set("sign_tripped", False) self.store.set("sign_reason", "") + # SAFETY-30's SITE latch, cleared by the SAME press and for the + # same reason the per-unit ones are: the condition always goes + # away on its own, because the trip commands 0 W everywhere and + # a site commanding zero cannot oppose anything. Only a human + # who has read the banner can say the site is clear. + # + # ⚠️ UNCONDITIONAL, LIKE THE FLAT PAIR ABOVE, AND NOT GUARDED ON + # THE ROSTER. An install that ran with two inverters, tripped, + # and was then reconfigured back to one would otherwise carry + # `site_sign_tripped: true` in its file forever, with no roster + # that can reach the code which clears it - and would re-arm the + # moment a second inverter was configured again. + if self.site_sign_wd.tripped: + self.log_event("site sign-disagreement failsafe cleared " + "by operator") + self.site_sign_wd.reset() + self.store.set("site_sign_tripped", False) + self.store.set("site_sign_reason", "") # SAFETY-31, cleared by the SAME press and for the same reason: # the condition always goes away on its own, because the trip # commands 0 W everywhere and two inverters at zero cannot diff --git a/goodwe_controller/app/store.py b/goodwe_controller/app/store.py index f811d11..282b938 100644 --- a/goodwe_controller/app/store.py +++ b/goodwe_controller/app/store.py @@ -71,6 +71,24 @@ _LOG = logging.getLogger("goodwe.store") # the card is gone while the fault runs on - exactly how the 2026-09-02 outage # went unseen for 6 h 25 m with the detector correct throughout. main.py # re-raises the notification from this key on every start. +# +# ⚠️ AND SAFETY-30's PAIR IS FLAT FOR THE SAME REASON SAFETY-31's IS, WHICH IS +# NOT THE SAME REASON THE ONES ABOVE ARE NAMESPACED. `site_sign_tripped` / +# `site_sign_reason` hold the SITE sign-disagreement latch: the SUM of every +# standing command opposed the meter, continuously. That is a statement about +# the grid connection - the offending number is the site target, which no single +# inverter owns - so there is no unit to hang it on, and hanging it on one would +# let an operator clear "inverter B's half" of a fault that has no halves. +# `unit//sign_tripped` above stays what it always was and now means what +# it says: THAT unit's own command against THAT unit's own measured power. +# +# ⚠️ AND THEY ARE NOT IN `LEGACY_UNIT_KEYS`, MUST NEVER BE, AND ARE NOT WRITTEN +# AT A ROSTER OF ONE AT ALL. At N=1 the site comparison and the unit comparison +# are the same sentence and `Controller.cycle()` arms only the per-unit +# detector, so a one-inverter install's state file is byte-identical to what it +# was before this card - these two keys sit at their defaults and nothing reads +# them. A rollback therefore finds exactly the flat `sign_tripped` mirror it +# always found. DEFAULTS = { "phase": "idle", "phase_started": None, @@ -80,6 +98,8 @@ DEFAULTS = { "sign_reason": "", "circ_tripped": False, "circ_reason": "", + "site_sign_tripped": False, + "site_sign_reason": "", } # --------------------------------------------------------------------------- # diff --git a/goodwe_controller/app/validate.py b/goodwe_controller/app/validate.py index 13d2e9a..0cb22b2 100644 --- a/goodwe_controller/app/validate.py +++ b/goodwe_controller/app/validate.py @@ -74,7 +74,8 @@ NOT READY banner already names the problem, (b) does not apply. was overruled. That firmware file's own comment says a lower bound on this "belongs in CFG-01, not here" - so here it is. - 5. `sign_disagree_w` at or above `max_w`. [(a)] + 5. `sign_disagree_w` - or SAFETY-30's `unit_sign_disagree_w` - at or above + `max_w`. [(a)] The arbiter's `supervised` limit bounds every command to +-max_w, and SAFETY-02's predicate needs `abs(command) > sign_disagree_w`. Set the threshold at or above the ceiling and the command side of the comparison @@ -234,6 +235,7 @@ from .p1 import is_enabled, validate_meter_options from .peakguard import PEAK_OPTION_DEFAULTS from .price import PRICE_OPTION_DEFAULTS, PROVIDERS from .static_tariff import ProfileInvalid, parse_static_profile +from .watchdog import SIGN_OPTION_DEFAULTS _LOG = logging.getLogger("goodwe.config") @@ -741,6 +743,38 @@ def validate_startup_config(opts: dict, dev_min=None, dev_max=None) -> list[str] f"sign_disagree_w below {weakest:.0f} W (300 W is the measured " "default), or raise max_w.") + # ⚠️ AND THE SAME RULE FOR SAFETY-30's PER-UNIT THRESHOLD, WHICH IS A + # SEPARATE COMPARISON AND THEREFORE A SEPARATE WAY TO DISARM. It grades a + # unit's own SHARE against that unit's own measured power, and a share is + # bounded by the same `supervised` claim at +-max_w - so a threshold at or + # above the ceiling makes the command side unsatisfiable by construction, + # exactly as above. Written as its own branch rather than folded into the + # one above because the two produce different sentences and an operator + # needs to be told WHICH watchdog they have disarmed. + # + # ⚠️ REFUSED EVEN AT A ROSTER OF ONE, WHERE THIS CHECK IS NOT ARMED. A + # config that cannot work is refused when it is WRITTEN, not on the day a + # second inverter is added - which is the day nobody is watching the startup + # log. This is `meter_max_age_s`'s rule read the same way: the failure it + # prevents is silent. + unit_sign_w = _number(opts, "unit_sign_disagree_w", + float(SIGN_OPTION_DEFAULTS["unit_sign_disagree_w"])) + if unit_sign_w is None: + problems.append(f"unit_sign_disagree_w " + f"{opts.get('unit_sign_disagree_w')!r} is not a number") + elif max_w is not None and max_w > 0 and unit_sign_w >= max_w: + problems.append( + f"unit_sign_disagree_w is {unit_sign_w:.0f} W but max_w is " + f"{max_w:.0f} W, so the SAFETY-30 per-inverter sign-disagreement " + "watchdog can never fire: no inverter is ever commanded past max_w, " + f"and the watchdog needs a command over {unit_sign_w:.0f} W. At two " + "inverters that is the guard which catches a unit doing the " + "OPPOSITE of what it was told while its sibling covers for it at " + "the meter - the case the site-level watchdog is structurally " + f"blind to. Lower unit_sign_disagree_w below {max_w:.0f} W " + f"({SIGN_OPTION_DEFAULTS['unit_sign_disagree_w']:.0f} W is the " + "measured default), or raise max_w.") + # --- 6. a maintenance charge the ceiling can actually accommodate -------- # ⚠️ Strictly `>`, matching the arbiter exactly. The shaping is # `Claim.limit(hi=-charge_w)` against `supervised`'s `lo=-max_w`, so diff --git a/goodwe_controller/app/watchdog.py b/goodwe_controller/app/watchdog.py index d2ff923..0ea7afa 100644 --- a/goodwe_controller/app/watchdog.py +++ b/goodwe_controller/app/watchdog.py @@ -164,11 +164,106 @@ saturation freeze's question (|command - the battery's actual power| > 500 W, NOTES.md §4.7 row 4), and the two checks are complementary rather than overlapping. Neither substitutes for the other. +⚠️ THAT PARAGRAPH IS ABOUT THE COMMAND-VERSUS-METER COMPARISON, AND SINCE +SAFETY-30 THAT IS NOT THE ONLY ONE THIS MODULE DRIVES. At a roster of two or +more each unit is ALSO graded against its own measured power, which is exactly +the comparison the paragraph says nothing here makes - see the SAFETY-30 section +below, which says what that does and does not take away from the freeze. At the +shipped roster of one the paragraph stands unchanged, word for word: the site +comparison is the only detector armed and the freeze is still the whole answer. + Nor does it see a sign inversion introduced DOWNSTREAM of this add-on - in the firmware, on the wire, or in a future driver. Our view stays self-consistent and this stays silent. That is a wire-versus-intent problem and it belongs to the witness, which is the only vantage point that sees both. +------------------------------------------------------------------------------ +SAFETY-30 - AT TWO INVERTERS THIS IS TWO QUESTIONS, NOT ONE + +Everything above was written at a roster of ONE, and at N=1 it is right for a +reason that stops being true at N=2. `commanded_w` is what THIS controller put +on THIS actuator. `grid_w` is what the WHOLE SITE is exchanging with the +network. At one inverter those are two views of one thing - §6.2's identity - +which is what makes comparing them sound. At two they are not: + + * A FALSE TRIP THE COMPARISON CANNOT AVOID. "Inverter A commanded -1500 W + while the meter reads +1500 W" is evidence of nothing at N=2: inverter B may + be discharging 3000 W and doing exactly what it was told. No fault anywhere, + and the check NOTES.md calls the highest-value entry in the safety table - + 93 % agreement while healthy against 4 % during runaway, four episodes, zero + false positives - fires. + * THE CONVERSE, WHICH IS WORSE. A unit genuinely driving the wrong way is + MASKED, because the site figure NETS. Commanded +1500 W, measured -1500 W, + beside a sibling measured at +1500 W: the meter sits exactly where a healthy + site leaves it. This is not a threshold that is set too high - the + information is destroyed by the summation, and no tuning recovers it. + +So the one comparison becomes two, each asked of the evidence that can answer +it, and they are DIFFERENT FAULTS rather than one fault at two scales: + + PER UNIT this unit's standing command against ITS OWN measured power - + `Inverter.batt`, which on the live system is holding register + 0x0518, this inverter's own AC power, per-instance since DEV-02. + Conclusion: "THE ACTUATOR I AM DRIVING IS NOT DOING WHAT I TOLD + IT". The evidence is about one unit, so the remedy is that unit: + the `failsafe` SET, which arbiter.CLAIM_SCOPE has called a UNIT + claim from the day it was written, for this exact reason. + SITE the SUM of the standing commands against the meter. Conclusion: + "THE SITE AS A WHOLE IS DRIVING THE WRONG WAY" - an inverted gain, + an integrator wound past the rail, a strategy pushing the wrong + side of the meter. That is the fault the 2026-08-20 capture + contains and it is a property of the CONNECTION, so the remedy is + the connection: a SITE SET that stops every inverter. + +⚠️ AND AT N=1 THERE IS EXACTLY ONE ARMED DETECTOR, WHICH IS THE ONE THAT +SHIPPED. The two questions above are the SAME SENTENCE at a roster of one: the +sum of one command is that command, and "the site" and "the unit" name one +actuator. Arming both would be arming a duplicate - two latches, two banners and +two chances to spend a false-positive record that was earned by ONE comparison +on two captured bus days. `Controller.cycle()` therefore hands the single unit's +watchdog THE METER, under the shipped `sign_disagree_w` / `sign_disagree_s`, +exactly as before, and never arms the site one. `arm()` below is where that +choice is written down. Nothing about a one-inverter install changes - not the +comparison, not the thresholds, not the store keys, not the claim, not the +banner. + +⚠️ WHAT THE PER-UNIT COMPARISON IS NOT. Two guards that overlap without saying +so is how a false-alarm generator gets built, so both neighbours are named: + + * IT IS NOT SAFETY-31. `circulation.py` compares the units AGAINST EACH OTHER + - `sum(|B_i|) - |sum(B_i)|` over MEASURED power - and catches two inverters + pushing THROUGH each other, which nets to zero at the meter and about which + no single command is evidence at all. This compares each unit against ITS + OWN COMMAND. Take two units faithfully executing an allocation that ALLOC-01 + would refuse: circulating power is the whole loop, and every per-unit sign + check here reads perfect agreement. Take one unit doing the exact opposite + of its share while its sibling covers for it: this reads the whole + disagreement, and circulating power can be zero, because the two measured + powers may well agree in sign. Neither is the other with a different + threshold. + * IT IS NOT THE SATURATION FREEZE, AND THE OVERLAP IS REAL AND BOUNDED. The + freeze asks `|command - measured| > saturation_w` (500 W) and concludes "the + inverter cannot follow"; it then stops the integrator winding, which is a + control action taken every day, on a unit that is merely SATURATED as + readily as on one that is reversed, and it neither latches nor parks + anything. This asks for OPPOSITION - a sign, on both sides of a threshold, + held continuously - which a saturated-but-obedient unit cannot produce at + all, and it LATCHES and commands that unit to zero. The freeze is a + magnitude test that shapes the loop; this is a direction test that stops an + actuator. + +⚠️ GATE 1 STILL APPLIES TO BOTH, AND THAT IS A DELIBERATE CHOICE WITH A COST. +The per-unit comparison could have been armed whenever ANY non-zero command is +standing - a unit should do what it is told whoever told it - and that would +catch a wedged inverter during a maintenance charge. It is not, because the +whole no-false-positive argument above is built on `following`, because the +freeze already grades that window, and because this card's own warning is that +SAFETY-02's record was earned at N=1 and must not be spent on a check that fires +when nothing is wrong. What it costs is written here rather than discovered: +between the start of a maintenance cycle and its end, a unit reversing against +its own command is the freeze's to notice, not this module's. Widening it is a +card, with a rig campaign behind it, not an edit. + ------------------------------------------------------------------------------ STATE, TRIP AND LATCH @@ -198,6 +293,178 @@ from dataclasses import dataclass from datetime import datetime +# SAFETY-30. CFG-03: THE ONE IN-CODE COPY OF THE FOUR MULTI-INVERTER DEFAULTS. +# `config.yaml`'s `options:` block is the source of truth an operator edits and +# the Supervisor reads, and `schema:` holds the bounds. This dict exists for the +# reason `circulation.CIRC_OPTION_DEFAULTS` and `maintenance.MAINT_OPTION_DEFAULTS` +# do - the Dockerfile does not put config.yaml in the image, so the running +# add-on cannot read it - and `test_watchdog.py` pins the two together by READING +# config.yaml, so a drift goes red in CI rather than shipping. +# +# ⚠️ EVERY KEY HERE IS DEAD AT A ROSTER OF ONE, and that is the point rather +# than a caveat. `Controller._arm_sign_watchdogs()` arms the single unit's +# detector against the meter with the SHIPPED `sign_disagree_w` / +# `sign_disagree_s`, which are read in main.py exactly where they always were +# and are not in this dict. Nothing below is consulted until a second inverter +# exists. +SIGN_OPTION_DEFAULTS = { + # --- THE MAGNITUDE GATE, AND IT IS NOT SAFETY-02's 300 W BY ACCIDENT --- + # ⚠️ DERIVED FROM `control.Tuning.saturation_w`, WHICH IS 500 W, AND THE + # DERIVATION IS THE POINT. This check gates on BOTH sides being over + # `min_w`, so the smallest divergence it can ever see between a command and + # the power that command produced is `2 * min_w` - the two are OPPOSED, so + # the gap is the sum of the magnitudes. At 250 W that is exactly 500 W: the + # gap at which `compute()`'s own anti-windup test already concludes THE + # INVERTER IS NOT FOLLOWING and freezes the integrator. So this watchdog + # cannot fire on a divergence the shipped control law itself calls ordinary + # tracking, and the two guards meet at a number rather than overlapping in a + # band nobody has drawn. + # + # MEASURED, on 1890 closed-loop two-inverter runs in which NOTHING is faulty + # (every unit executes the share the allocator gave it), swept over the + # WHOLE range config.yaml accepts for `gain` float(0.05,1.0) and `slew_w` + # int(50,5000), across both families of stimulus that can produce legitimate + # opposition. Worst CONTINUOUS opposition between a unit's standing command + # and its own 0x0518 power, as a function of this threshold: + # + # 50 W -> 25.0 s 250 W -> 10.0 s 700 W -> 5.0 s + # 150 W -> 15.0 s 300 W -> 10.0 s 1000 W -> 0.0 s + # 200 W -> 15.0 s 500 W -> 5.0 s + # + # ⚠️ AND THE CURVE IS FLAT, WHICH IS THE FINDING THAT DECIDED THE SHAPE OF + # THIS PAIR. Unlike the site comparison - where the meter is driven by the + # LOAD and a flapping load holds it opposed for minutes - a unit's own power + # CHASES ITS OWN COMMAND through a bounded ramp and a first-order lag, so the + # opposition can only last as long as the crossing does. Dropping the + # threshold from 300 W to 50 W buys 15 s of extra transient and nothing else. + # The work is therefore done by the WINDOW below, and this number is set + # where the control law already draws its line rather than being tuned. + # + # The largest opposed magnitude seen anywhere in those 1890 runs is 875 W; + # a threshold above that would gate NOTHING legitimate in at all, and is + # rejected because it would also be blind to a unit reversing a 900 W share. + "unit_sign_disagree_w": 250, + + # --- THE WINDOW ------------------------------------------------------- + # Three bounds, and the value clears all three. + # + # LOWER, from the rig: 10.0 s - two control cycles, the worst CONTINUOUS + # legitimate opposition above 250 W across all 1890 runs. It is a + # reversing site whose second write lands a cycle after the first, both + # units then slewing through zero behind a 10 s lag; nothing is wrong. + # + # LOWER, from the firmware: `inverter.LINK_STALE_S`, 30 s. A unit not + # executing its share for less than the firmware's own veto window is a + # LINK fault, and the add-on already answers it - liveness drops the unit + # and the survivors carry its share, with the record saying so. Firing + # inside that window would raise a second, louder alarm about a state + # already being handled, which is the SAFETY-15/SAFETY-19 failure exactly. + # + # UPPER, from SAFETY-02 itself: 180 s, the SITE hold. A unit that is not + # doing what it was told must not be parked LATER than the site would be + # parked for the same watts, and unlike the site comparison this one needs + # nothing to net out before it can conclude - the evidence is complete on + # the first cycle it appears. + # + # 85 s: seventeen control cycles - 8.5x the worst legitimate episode this + # rig can produce, 2.8x the firmware's own veto window, and under HALF the + # site hold. ⚠️ IT IS DELIBERATELY NOT 60, 75, 90, 120 OR 180. A round + # number here would be a guess wearing a measurement's clothes, and the + # three bounds above are what the number has to answer to. + # + # ⚠️ AND INVERTER B DOES NOT PHYSICALLY EXIST. Its ramp and lag in the rig + # are inverter A's, COPIED, carrying `measured=False` - SPIKE-04 measures + # the real ones. A slower second unit lengthens every crossing above, which + # is what the 8.5x buys headroom for. SPIKE-04 is the revisit trigger for + # both numbers here, not a reason to move them now. + "unit_sign_disagree_s": 85, + + # --- THE SITE COMPARISON AT N>=2, AND IT IS NOT THE SHIPPED PAIR ------- + # ⚠️ THE THRESHOLD IS CARRIED AND THE WINDOW IS NOT, WHICH IS A FINDING + # RATHER THAN A COMPROMISE. Only one of the two arguments behind 300 W / + # 180 s depends on how many inverters the site has: + # + # THE THRESHOLD is about NOISE. "Near zero the sign is noise, and near + # zero is where a healthy loop LIVES" - the deadband holds the command + # still while the meter wanders either side of `target_grid_w`. The + # deadband is 15 W and P1 is one meter whatever the roster, so nothing + # about that argument moves. 300 W, carried - and the two-inverter + # campaign says the carry is safe rather than merely convenient: in the + # load-reversal family the site figure ever goes opposed AND over 300 W in + # 994 of 1050 runs at two inverters and 955 of 1050 at one. The gate opens + # about as often either way; what changed is how LONG it stays open, which + # is the window's problem and not the threshold's. + # + # THE WINDOW is about a TRANSIENT, and the transient scales with the site. + # The measured worst legitimate case is a load reversal that the loop + # crosses zero slowly: the command has to travel the whole rail at + # `slew_w` per cycle, and at N>=2 the rail is `Controller.site_max_w` - the + # ROSTER SUM - while `slew_w` is unchanged. So the crossing takes about + # twice as long at two inverters as at one, and that is exactly what the + # rig measures. + # + # 1890 closed-loop runs in which nothing is faulty, the SAME campaign the + # per-unit pair above is read off, run at BOTH rosters as its own control. + # Worst CONTINUOUS opposition between the standing command(s) and the meter, + # LOAD-REVERSAL family, at a 300 W threshold: + # + # N=1 (the shipped comparison) 110.0 s - inside 180 s, 1.6x + # N=2 (the sum, same stimuli) 215.0 s - OVER 180 s + # + # ⚠️ SO CARRYING 180 s WOULD HAVE SHIPPED A FALSE TRIP, and it is a config a + # schema-legal install can reach: `gain` 0.05 and `slew_w` 50 are both the + # floor of what config.yaml accepts, and a +-2200 W load reversal is an oven + # and a heat pump. That is the number this card exists to not assume. + # + # 325 s: 1.51x the measured worst, well inside the 900 s quarter that bills + # the capacity tariff, and still inside every episode the fault day + # contains - 2026-08-20's four clusters run 16, 30, 77 and 77 minutes and + # its opposed runs reach 1750 s, so a 325 s window fires in all of them. + # What it costs is stated rather than hidden: about two and a half extra + # minutes of wrong-direction SITE power before the stop, bought to remove a + # false trip that a legal configuration produces. ⚠️ DELIBERATELY NOT 300 OR + # 360; the measurement is what it answers to. + # + # ⚠️ AND IT IS NOT THE PER-UNIT WINDOW EITHER, IN THE OTHER DIRECTION. The + # per-unit comparison is 85 s because a unit's own power CHASES ITS OWN + # COMMAND and the transient is two cycles; the site comparison is 325 s + # because the meter is driven by the LOAD and the transient is the loop + # crossing zero. Same predicate, two instruments, two windows, both measured. + "site_sign_disagree_w": 300, + "site_sign_disagree_s": 325, +} + +# ⚠️ A LIMIT OF THE SITE COMPARISON THAT SAFETY-30 DID NOT INTRODUCE AND DOES +# NOT FIX, RECORDED HERE BECAUSE THE CAMPAIGN FOUND IT AND SILENCE WOULD BE +# WORSE. The rig's other stimulus family is a FLAPPING LOAD, and a square wave +# whose half period is about one control cycle opposes the command on EVERY +# cycle: the command chases the previous half-cycle's meter while the meter has +# already reversed. Across the 840 flapping runs the worst continuous opposition +# at a 300 W threshold is 980 s at two inverters - and 1000 s at ONE, which is +# the whole point. The N=1 arm is the control, and it says this is a property of +# comparing a lagged command against a meter, not a property of the second +# inverter. No window inside the 900 s billing quarter survives it, and no +# threshold up to 1000 W does either. +# +# It is not acted on here for three reasons, in order of weight: it is identical +# at the roster this add-on actually ships, so acting on it would be changing a +# live safety path on evidence that has nothing to do with this card; a +# ~6 kW square wave at 0.1 Hz sustained for a quarter of an hour is not a house +# (the captured healthy day's longest opposed run is 30 s); and the fix is not a +# number - it wants the detector to notice that the disagreement is ALTERNATING, +# which is a different check. It is a card, and this comment is its evidence. + +# SAFETY-30. THE TWO NOUNS THE TRIP SENTENCE CAN NAME, WRITTEN ONCE. They are +# constants rather than literals at the call sites because `main.py` arms the +# detectors and `test_watchdog.py` grades what they say, and a string typed in +# two places is a string that drifts in one of them - the same argument +# `store.unit_key()` makes about a key name. `AGAINST_METER` is the wording that +# has always shipped and must not change: it is what a one-inverter install's +# `reason` reads, in the store, in the action log and on the status page. +AGAINST_METER = "a meter reading" +AGAINST_OWN_POWER = "its own measured power" + + @dataclass(frozen=True) class SignWatchdogConfig: """Defaults, not constants - config.yaml carries both (SAFETY-02 AC 5).""" @@ -206,7 +473,7 @@ class SignWatchdogConfig: hold_s: float = 180.0 -def disagrees(commanded_w, grid_w, following: bool, +def disagrees(commanded_w, against_w, following: bool, cfg: SignWatchdogConfig) -> bool: """The predicate, and the ONLY place it is written down. @@ -215,12 +482,23 @@ def disagrees(commanded_w, grid_w, following: bool, inline. Two copies of a safety predicate is one copy plus a future disagreement about what the check means, and the copy that drifts is always the one in the harness that grades it. + + ⚠️ `against_w` WAS CALLED `grid_w`, AND THE RENAME IS THE WHOLE OF SAFETY-30 + IN ONE PARAMETER. The arithmetic is unchanged - opposed signs, neither side + near zero, and the loop actually driving - but WHAT the command is opposed + TO is now a property of the caller rather than a fact about this function. + The site comparison hands it the meter; each unit's own comparison hands it + that unit's measured power (register 0x0518). One predicate, two callers, + for `loop_is_following()`'s reason: a safety predicate written down twice is + one copy plus a future disagreement about what the check means. The + parameter is positional at every call site in both repositories, so the + rename moves no caller. """ return (following - and commanded_w is not None and grid_w is not None - and commanded_w * grid_w < 0 + and commanded_w is not None and against_w is not None + and commanded_w * against_w < 0 and abs(commanded_w) > cfg.min_w - and abs(grid_w) > cfg.min_w) + and abs(against_w) > cfg.min_w) def loop_is_following(resolution, loop_asked) -> bool: @@ -272,7 +550,17 @@ def loop_is_following(resolution, loop_asked) -> bool: class SignWatchdog: - """One instance per actuator, evaluated once per control cycle. + """One instance per COMPARISON, evaluated once per control cycle. + + ⚠️ SINCE SAFETY-30 THAT IS NOT THE SAME SENTENCE AS "one per actuator", + which is what this docstring used to say. There is one instance per + inverter - `Inverter.sign_wd`, latched under that unit's own store keys - + AND one for the site, `Controller.site_sign_wd`, latched under flat ones. + The two ask different questions of different evidence and reach different + remedies; `arm()` below is where an instance is told which of them it is. + At a roster of one they would be the same question, so only the per-unit + instance is ever armed there - see the SAFETY-30 section of the module + docstring. ⚠️ NOT SHARED ACROSS CADENCES. `disagree_since` is a wall-clock start time, so driving one instance from two loops at two rates would have each one @@ -282,8 +570,14 @@ class SignWatchdog: """ def __init__(self, cfg: SignWatchdogConfig | None = None, *, - tripped: bool = False, reason: str = ""): + tripped: bool = False, reason: str = "", + against: str = AGAINST_METER): self.cfg = cfg or SignWatchdogConfig() + # SAFETY-30. WHAT THE COMMAND IS BEING COMPARED AGAINST, as the operator + # reads it in the trip sentence. The default is the meter, which is what + # every trip this add-on has ever taken was compared against and is what + # keeps a one-inverter install's `reason` string byte-identical. + self.against = str(against) self.disagree_since: datetime | None = None # ⚠️ RESTORED FROM THE STORE, NOT DEFAULTED. A latch that lives only in # memory is not a latch: a container restart clears it, and the add-on @@ -292,8 +586,31 @@ class SignWatchdog: self.tripped = bool(tripped) self.reason = reason if self.tripped else "" + # -- which of SAFETY-30's two questions this instance is asking ---------- + def arm(self, cfg: SignWatchdogConfig, against: str) -> None: + """Set the thresholds and the subject of the comparison. Nothing else. + + ⚠️ IT IS CONFIGURATION, NOT STATE, AND IT DELIBERATELY DOES NOT TOUCH + `disagree_since`, `tripped` OR `reason`. A detector that cleared its own + window when it was told what it was comparing against would lose the + episode it is in the middle of, every cycle, and would never accumulate + a hold at all - the check would be permanently green and nothing would + say so. + + ⚠️ AND IT IS RE-STATED EVERY CYCLE RATHER THAN FROZEN AT CONSTRUCTION. + The answer is a property of the ROSTER and of nothing else, and + `Controller.cycle()` already re-reads the roster every cycle for + liveness and for the allocation. Freezing it in `__init__` would also + make it wrong for any caller that appends an inverter afterwards, which + is exactly how a two-inverter fixture is built in this repository - and + a fixture that silently got the one-inverter comparison would grade + nothing while looking green. + """ + self.cfg = cfg + self.against = str(against) + # -- the whole check, and it is a pure function of its arguments ---------- - def evaluate(self, commanded_w, grid_w, following: bool, + def evaluate(self, commanded_w, against_w, following: bool, now: datetime) -> bool: """Advance the detector one cycle. Returns True while tripped. @@ -301,15 +618,23 @@ class SignWatchdog: after deadband, slew, quantisation and the arbiter. Not the optimizer's raw wish: the wire carries the emitted value, so anything else can disagree with the meter for a - reason that is not a fault. - `grid_w` net P1, + = importing. None when the meter is stale or - missing, which CLEARS the timer rather than accumulating - - a frozen figure held against a moving command would - otherwise manufacture a disagreement in about three - minutes, on top of SAFETY-01 having already commanded - zero for the same cause. + reason that is not a fault. For a per-unit instance it is + THAT UNIT'S share of it; for the site instance it is the + roster sum. + `against_w` what the command is graded against, and SAFETY-30 made + this a decision rather than a fact: net P1 for the SITE + comparison (+ = importing), and this unit's own measured + power for the PER-UNIT one (+ = discharging, register + 0x0518). Both carry the same sign convention as the + command, which is why one predicate serves both. None + when the reading is stale or missing, which CLEARS the + timer rather than accumulating - a frozen figure held + against a moving command would otherwise manufacture a + disagreement in about three minutes, on top of SAFETY-01 + having already commanded zero for the same cause. `following` did the grid-following loop actually win this cycle? See - gate 1 in the module docstring. + gate 1 in the module docstring, and the SAFETY-30 section + on why the per-unit comparison keeps it. `now` the caller's clock, so this stays testable and replayable. """ if self.tripped: @@ -317,7 +642,7 @@ class SignWatchdog: return True cfg = self.cfg - if not disagrees(commanded_w, grid_w, following, cfg): + if not disagrees(commanded_w, against_w, following, cfg): self.disagree_since = None return False @@ -330,9 +655,17 @@ class SignWatchdog: return False self.tripped = True + # ⚠️ `self.against` IS THE ONLY THING SAFETY-30 CHANGED IN THIS STRING, + # and it defaults to the wording that shipped - "a meter reading" - so a + # one-inverter install's `reason` is byte-identical to what an operator + # (and test_watchdog.py) has always read. A per-unit instance names what + # it actually compared instead, because "commanded +1500 W against a + # meter reading -1500 W" would be a false statement about which + # instrument produced the evidence, on the one line somebody woken by + # this alarm reads first. self.reason = ( - f"SIGN_DISAGREEMENT: commanded {commanded_w:+.0f} W against a meter " - f"reading {grid_w:+.0f} W for {held:.0f} s continuously " + f"SIGN_DISAGREEMENT: commanded {commanded_w:+.0f} W against " + f"{self.against} {against_w:+.0f} W for {held:.0f} s continuously " f"(limits: both over {cfg.min_w:.0f} W, held over {cfg.hold_s:.0f} s)" ) return True diff --git a/goodwe_controller/config.yaml b/goodwe_controller/config.yaml index ca05e76..e0a8ce3 100644 --- a/goodwe_controller/config.yaml +++ b/goodwe_controller/config.yaml @@ -167,6 +167,62 @@ options: sign_disagree_w: 300 sign_disagree_s: 180 + # --- SAFETY-30 per-inverter sign-disagreement watchdog --------------------- + # ⚠️ A DIFFERENT COMPARISON, NOT A SECOND COPY OF THE ONE ABOVE, AND IT IS + # ARMED ONLY WHEN THIS SITE HAS MORE THAN ONE INVERTER. The pair above grades + # the SITE's total command against P1 - one meter, one grid connection. This + # pair grades EACH inverter's own share against THAT inverter's own measured + # power (holding register 0x0518), and it catches the fault the meter cannot + # answer for once there are two units: a unit doing the opposite of what it + # was told, while its sibling covers for it and P1 reads exactly what a + # healthy site reads. + # + # At ONE inverter these two keys do nothing at all. §6.2's identity makes the + # site's total command that inverter's command, so the pair above is already + # asking this question of the same actuator, and arming a second detector + # would be two latches and two banners for one fault. + # + # Both defaults are measured on admin/goodwe's two-master rig, over 1890 + # closed-loop runs in which nothing is faulty, swept across the whole range + # this file accepts for `gain` and `slew_w`. 250 W is `saturation_w / 2`, so + # the smallest divergence this can see is exactly the gap at which the control + # law itself already says the inverter is not following; 85 s is 8.5x the + # worst legitimate opposition those runs produce (10.0 s), 2.8x the firmware's + # own 30 s link veto, and under half the 180 s the site check takes. See + # app/watchdog.py's SIGN_OPTION_DEFAULTS and + # DECISIONS/SAFETY-30-per-unit-sign.md. + # + # ⚠️ unit_sign_disagree_w must stay BELOW max_w, for sign_disagree_w's reason: + # a share is bounded to +-max_w, so a threshold at or above it can never fire. + # validate.py refuses that config. + unit_sign_disagree_w: 250 + unit_sign_disagree_s: 85 + + # --- SAFETY-30 SITE sign-disagreement watchdog ----------------------------- + # ⚠️ THE SAME COMPARISON `sign_disagree_*` MAKES, AT THE ROSTER WHERE IT STOPS + # BEING A STATEMENT ABOUT ONE INVERTER. At two or more units the SUM of the + # standing commands is graded against P1, and the trip stops the whole site - + # what it catches is the site being asked for the wrong thing, which every + # per-inverter check is blind to because each unit is faithfully executing its + # share. At ONE inverter these two keys do nothing: `sign_disagree_*` above is + # already asking this of the same actuator. + # + # The threshold is the shipped 300 W, CARRIED, and the reason is that the + # argument behind it does not depend on the roster: near zero the sign is + # noise, the deadband is 15 W, and there is one P1 whatever is behind it. + # + # ⚠️ THE WINDOW IS NOT CARRIED, AND THAT IS A MEASUREMENT. The transient 180 s + # exists to reject is the loop crossing zero after a load reversal, and the + # command has to travel the whole SITE rail at `slew_w` per cycle - a rail + # that is the roster sum while `slew_w` is not. On 1890 two-inverter runs in + # which nothing is faulty, the worst continuous legitimate opposition at 300 W + # is 110 s at one inverter and 215 s at two: carrying 180 s across would have + # shipped a false trip reachable at `gain: 0.05`, `slew_w: 50` and an ordinary + # 2.2 kW load reversal. 325 s is 1.51x that, inside the 900 s billing quarter, + # and still inside every episode of the 2026-08-20 fault day. + site_sign_disagree_w: 300 + site_sign_disagree_s: 325 + # --- SAFETY-18 frozen-but-fresh meter detector ----------------------------- # Seconds the meter may keep answering with an UNCHANGED measurement while the # command stands at the rail before the add-on raises an alarm. Both halves @@ -487,6 +543,33 @@ schema: sign_disagree_w: int(50,5000) sign_disagree_s: int(30,1800) + # SAFETY-30. No "off" value, for SAFETY-02's reason. + # + # ⚠️ THE THRESHOLD'S RANGE IS THE SAME AS THE SITE CHECK'S ON PURPOSE, AND THE + # WINDOW'S FLOOR IS NOT. On the two-inverter rig the per-unit comparison's + # worst continuous legitimate opposition is nearly FLAT in the threshold - + # 25.0 s at 50 W, 15.0 s at 150 W, 10.0 s at 250 W, 5.0 s at 500 W - because a + # unit's own power chases its own command rather than the load. So no value in + # this range puts the guard outside the regime it can be answered for, and the + # floor does not have to be a knee the way `circulation_w`'s does. + unit_sign_disagree_w: int(50,5000) + # The floor of 30 is `inverter.LINK_STALE_S`, the firmware's own veto window: + # below it, a unit not executing its share is a LINK fault the add-on already + # answers by dropping the unit from the split, and firing inside that window + # would be a second, louder alarm about a state that is already being handled. + unit_sign_disagree_s: int(30,1800) + + # SAFETY-30's SITE pair. Same ranges as `sign_disagree_*`, because it is the + # same comparison asked of the site total - and no "off" value, for the same + # reason. ⚠️ THERE IS DELIBERATELY NO validate.py RULE PAIRING + # site_sign_disagree_w WITH max_w, unlike the other two thresholds: the bound + # on the SITE command is `Controller.site_max_w`, the ROSTER SUM, and + # `validate_startup_config()` is handed options rather than a roster. A rule + # written against max_w would refuse a perfectly reachable two-inverter + # configuration the day CFG-04 lands. The schema range is the bound here. + site_sign_disagree_w: int(50,5000) + site_sign_disagree_s: int(30,1800) + # No "off" value, for SAFETY-02's reason: disarming a safety alarm is not a # supported configuration. ⚠️ THE FLOOR OF 60 IS NOT A RECOMMENDATION. At 300 s # the slowest a HEALTHY meter's 1 Wh register can tick and still look unchanged diff --git a/goodwe_controller/test_arbiter.py b/goodwe_controller/test_arbiter.py index 883e27d..d8826f3 100644 --- a/goodwe_controller/test_arbiter.py +++ b/goodwe_controller/test_arbiter.py @@ -165,21 +165,29 @@ SITE_SOURCES = { "identically zero for any one unit - there is no fact of " "the matter about which of two opposing inverters is the " "offender, so there is nothing to file it per-unit on", + # SAFETY-30, and it is the entry most easily confused with `failsafe` + # below, because the same three words describe both. + "sign_site": "the SUM of every standing command against the meter. The " + "offending number is the SITE target, which no single " + "inverter owns - parking one of them would leave the rest " + "pushing the same way with the guard satisfied", } UNIT_SOURCES = { "device": "one inverter's own reported min/max rating", "supervised": "max_w caps what may be WRITTEN to one setpoint entity", - "failsafe": "site-wide evidence, per-unit remedy: SAFETY-02 grades the " - "command THIS controller put on THIS actuator", + "failsafe": "per-unit evidence, per-unit remedy: since SAFETY-30 it " + "grades THIS unit's command against THIS unit's OWN " + "measured power (0x0518), so what it learns is about one " + "device. The site-wide half moved to `sign_site`", "maintenance": "the monthly recalibration is a pack procedure", } for _src, _why in sorted(SITE_SOURCES.items()): print(f" site {_src:<11} {_why}") for _src, _why in sorted(UNIT_SOURCES.items()): print(f" unit {_src:<11} {_why}") -check("all NINE claim sources are classified, and only those nine", +check("all TEN claim sources are classified, and only those ten", set(CLAIM_SCOPE) == set(SITE_SOURCES) | set(UNIT_SOURCES)) -check("the five that are about the GRID CONNECTION are site claims", +check("the six that are about the GRID CONNECTION are site claims", {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_SITE} == set(SITE_SOURCES)) check("the four that are about a PACK, a DEVICE or one write are per-inverter", {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_UNIT} == set(UNIT_SOURCES)) @@ -200,7 +208,7 @@ check("with one inverter every source routes to the site, per-inverter ones " check("...and with none configured at all, likewise", all(actuator_for(s, "inverter-a", ()) == ACTUATOR_SITE for s in CLAIM_SCOPE)) check("with TWO the identity breaks and the split becomes real: the four " - "per-inverter sources route to the unit this controller drives, the five " + "per-inverter sources route to the unit this controller drives, the six " "site sources stay on the grid connection", {s: actuator_for(s, "inverter-a", TWO) for s in CLAIM_SCOPE} == {**{s: "inverter-a" for s in UNIT_SOURCES}, diff --git a/goodwe_controller/test_inverter.py b/goodwe_controller/test_inverter.py index 4f6f125..db2f548 100644 --- a/goodwe_controller/test_inverter.py +++ b/goodwe_controller/test_inverter.py @@ -609,28 +609,40 @@ print("3. AC 3: a trip on B does not latch A, and it survives a restart") # inverters get a share of ONE site target, so an ordinary wrong-way loop trips # BOTH - correctly, because both really are pushing the wrong way. The state # this part is about is the one the flat store keys made unreachable: ONE -# inverter standing on a command that opposes the meter while the other does -# not. It is reached the way the field would reach it - inverter B's writes are -# being REJECTED, so `inv.last_write` stays where it was while A's tracks the -# loop - and `standing` is what SAFETY-02 grades, per unit. +# inverter standing on a command that opposes what it is graded against while +# the other does not. It is reached the way the field would reach it - inverter +# B's writes are being REJECTED, so `inv.last_write` stays where it was while +# A's tracks the loop - and `standing` is what SAFETY-02 grades, per unit. +# +# ⚠️ SAFETY-30 CHANGED WHAT "OPPOSES" MEANS HERE, AND THE FIXTURE HAD TO MOVE +# WITH IT - WHICH IS THIS CHECK DOING ITS JOB. It used to stand B on a -1500 W +# charge against a +2000 W METER and let the site figure convict it. At a roster +# of two that comparison is exactly the defect SAFETY-30 removes: the meter +# answers for BOTH inverters, so it is evidence about neither. B is now +# convicted by its OWN power sensor - standing on a -1500 W charge while +# `sensor.batt_b` measures it DISCHARGING at +1500 W, which is one inverter +# doing the opposite of what it was told and is a fact about that inverter +# alone. Inverter A's own reading stays at 0 W, so A is never gated in. # # ⚠️ AND THE REJECTION WINDOW IS WHY THIS RUNS IN THREE SECONDS AND NOT THIRTY. -# `sign_disagree_s` is 1 s here; `LINK_STALE_S` is 30 s, so B is still LIVE for -# the whole run and the trip is graded before liveness has anything to say. +# `unit_sign_disagree_s` is 1 s here; `LINK_STALE_S` is 30 s, so B is still LIVE +# for the whole run and the trip is graded before liveness has anything to say. def _tripping_site(): hass = Hass( {"sensor.grid": 2000.0, "sensor.soc_a": 50.0, "sensor.batt_a": 0.0, - "sensor.soc_b": 50.0, "sensor.batt_b": 0.0}, + "sensor.soc_b": 50.0, "sensor.batt_b": 1500.0}, {"number.sp_a": (-A_RATING, A_RATING, 1.0), "number.sp_b": (-B_RATING_HYPOTHESIS, B_RATING_HYPOTHESIS, 1.0)}, reject={"number.sp_b"}) store = MemStore() c, _ = built({"sign_disagree_w": 300, "sign_disagree_s": 1, + "unit_sign_disagree_w": 300, "unit_sign_disagree_s": 1, "auto_start": True}, store=store, hass=hass) c.inverters.append(inverter_b(store, sign_cfg=c.inverters[0].sign_wd.cfg)) c.auto = True - # B is standing on a charge command while the house imports 2000 W. Its - # writes are rejected, so this is what it goes on standing at. + # B is standing on a charge command while its own sensor says it is + # discharging. Its writes are rejected, so this is what it goes on + # standing at. c.inverters[1].last_write = -1500.0 return c, hass, store @@ -681,10 +693,12 @@ check("⚠️ ...AND THE TRIP IS MIRRORED ONTO THE PRE-DEV-02 FLAT PAIR, WHICH I # THE RESTART, which is the whole reason the latch is persisted at all. _h3b = Hass({"sensor.grid": 2000.0, "sensor.soc_a": 50.0, - "sensor.batt_a": 0.0, "sensor.soc_b": 50.0, "sensor.batt_b": 0.0}, + "sensor.batt_a": 0.0, "sensor.soc_b": 50.0, + "sensor.batt_b": 1500.0}, {"number.sp_a": (-A_RATING, A_RATING, 1.0), "number.sp_b": (-B_RATING_HYPOTHESIS, B_RATING_HYPOTHESIS, 1.0)}) _c3r, _ = built({"sign_disagree_w": 300, "sign_disagree_s": 1, + "unit_sign_disagree_w": 300, "unit_sign_disagree_s": 1, "auto_start": True}, store=_s3, hass=_h3b) _c3r.inverters.append(inverter_b(_s3, sign_cfg=_c3r.inverters[0].sign_wd.cfg)) @@ -703,12 +717,32 @@ check("a brand new Controller against the same store comes back with B " check("...and A goes on carrying the site target across the restart too", _c3r.allocation.shares.get("inverter-a") == _c3r.allocation.sum_w and _c3r.allocation.sum_w != 0) -check("...and the status page shows BOTH rows - one latched, one armed. A " - "single row would have to pick one inverter to describe, and 'the sign " - "watchdog is armed' printed over a latched sibling is the page lying", - len([r for r in _c3r.checks() if "sign watchdog" in r["text"]]) == 1 +check("...and the status page shows BOTH per-inverter rows - one latched, one " + "armed. A single row would have to pick one inverter to describe, and " + "'the sign watchdog is armed' printed over a latched sibling is the page " + "lying", + len([r for r in _c3r.checks() + if r["text"].startswith("sign watchdog [")]) == 1 and len([r for r in _c3r.checks() - if "SIGN-DISAGREEMENT FAILSAFE" in r["text"]]) == 1) + if "SIGN-DISAGREEMENT FAILSAFE [" in r["text"]]) == 1) +check("⚠️ AND SAFETY-30's SITE ROW IS THERE BESIDE THEM, EXACTLY ONCE. At two " + "inverters there are three sign questions on this page and they are three " + "rows: is A doing what it was told, is B, and is the SITE'S TOTAL " + "pointing the right way. The last one is not derivable from the other " + "two - both units can execute their shares perfectly while the number " + "they were given is itself wrong - so a page that showed only the " + "per-unit rows would be advertising two guards where the install has " + "three", + len([r for r in _c3r.checks() + if r["text"].startswith("site sign watchdog:")]) == 1) +check("...and the per-inverter rows now say WHAT the unit was compared " + "against, because at a roster of two it is no longer the meter. A row " + "still reading 'the command opposed the meter' would send an operator to " + "look at P1 for a fault that is visible only on one inverter's own power " + "sensor", + all("its own measured power" in r["text"] for r in _c3r.checks() + if r["text"].startswith(("sign watchdog [", + "SIGN-DISAGREEMENT FAILSAFE [")))) # The operator's clear, which is one button and has to reach every latch. run(_c3r.handle_action("auto_toggle")) # -> stopped @@ -728,8 +762,9 @@ check("...and the same press cleared the flat rollback view, so a downgrade " "- and neither does the next start, which now reads that key on EVERY " "boot rather than only on the one that migrates", _s3.data["sign_tripped"] is False and _s3.data["sign_reason"] == "") -_c3r2, _ = built({"sign_disagree_w": 300, "sign_disagree_s": 1}, store=_s3, - hass=_h3b) +_c3r2, _ = built({"sign_disagree_w": 300, "sign_disagree_s": 1, + "unit_sign_disagree_w": 300, "unit_sign_disagree_s": 1}, + store=_s3, hass=_h3b) check("⚠️ AND THE RESTART AFTER THAT CLEAR COMES UP UNLATCHED. This is the " "'re-arm forever' hazard the every-start guard would have if the clear " "did not reach both spellings, driven through the real button and a real " diff --git a/goodwe_controller/test_peakguard.py b/goodwe_controller/test_peakguard.py index 0faf418..4adb374 100644 --- a/goodwe_controller/test_peakguard.py +++ b/goodwe_controller/test_peakguard.py @@ -2410,8 +2410,19 @@ check("...and so does the persistent notification, which outlives the page and " # loop} at every SoC from 0 to 100 %, because nothing in those fixtures # circulates. What it does is stop the site when two inverters are measured # pushing against each other. See DECISIONS/SAFETY-31-circulating-power.md. +# ⚠️ SAFETY-30 ADDED THE TENTH, AND IT NEEDS ITS OWN PARAGRAPH FOR THE REASON +# `circulation` needed one. `sign_site` is a SET at P_SAFETY on the SITE +# actuator, filed only while `site_sign_wd.tripped` AND the roster holds more +# than one inverter. It is not a second copy of `failsafe`: that one grades ONE +# unit's command against THAT unit's own measured power and parks that unit, +# while this grades the SUM of the commands against the meter and stops the +# site. It appears in NO cycle of the sweeps below - the dynamic halves still +# observe exactly {device, supervised, loop} at every SoC, because the shipped +# roster is one inverter and this claim cannot be filed there at all. See +# DECISIONS/SAFETY-30-per-unit-sign.md. CLAIM_SOURCES = {"device", "supervised", "failsafe", "safety", "peak", - "arbitrage", "maintenance", "loop", "circulation"} + "arbitrage", "maintenance", "loop", "circulation", + "sign_site"} # The two appends that file a claim BUILT ELSEWHERE in the method. Named here so # that a THIRD such indirection reads as an unrecognised append rather than as a # silent gap in the scan. @@ -2590,13 +2601,17 @@ print(" site: " + ", ".join(sorted(s for s, v in CLAIM_SCOPE.items() if v == SCOPE_SITE)) + " | per-inverter: " + ", ".join(sorted(s for s, v in CLAIM_SCOPE.items() if v == SCOPE_UNIT))) -check("...and the split is the one ARB-03 argued, with SAFETY-31's addition: " - "the five about the GRID CONNECTION are site claims and the four about a " - "pack, a device or one write are per-inverter. `circulation` is a site " - "claim because its quantity is defined over the roster and is identically " - "zero for any one unit - there is no inverter to attribute it to", +check("...and the split is the one ARB-03 argued, with SAFETY-31's and " + "SAFETY-30's additions: the six about the GRID CONNECTION are site " + "claims and the four about a pack, a device or one write are " + "per-inverter. `circulation` is a site claim because its quantity is " + "defined over the roster and is identically zero for any one unit; " + "`sign_site` is one because the number that tripped it is the SITE " + "target, which no single inverter owns - while `failsafe` beside it is " + "per-inverter because since SAFETY-30 it grades one unit's command " + "against that unit's OWN measured power", {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_SITE} - == {"loop", "peak", "arbitrage", "safety", "circulation"} + == {"loop", "peak", "arbitrage", "safety", "circulation", "sign_site"} and {s for s, v in CLAIM_SCOPE.items() if v == SCOPE_UNIT} == {"device", "supervised", "failsafe", "maintenance"}) @@ -2672,19 +2687,21 @@ _restore_b() print(" with a second inverter on the roster, main.py routes: " + ", ".join(f"{s}->{a}" for s, a in sorted(_routed.items()))) check("⚠️ AND THIS IS THE BREAKING HALF. Give the same Controller a second " - "inverter and its OWN routing method splits the nine: `device`, " + "inverter and its OWN routing method splits the ten: `device`, " "`supervised`, `failsafe` and `maintenance` bind the unit this controller " - "drives, `loop`, `peak`, `arbitrage`, `safety` and SAFETY-31's " - "`circulation` stay on the grid connection. Swap any one of them - route " - "`peak` to the unit and the site keeps no floor; route `maintenance` to " - "the site and it can cross that floor into `lo > hi`; route `circulation` " - "to one unit and a site-wide loop parks the inverter that may well be the " - "one still listening - and this line goes red", + "drives, `loop`, `peak`, `arbitrage`, `safety`, SAFETY-31's `circulation` " + "and SAFETY-30's `sign_site` stay on the grid connection. Swap any one of " + "them - route `peak` to the unit and the site keeps no floor; route " + "`maintenance` to the site and it can cross that floor into `lo > hi`; " + "route `circulation` to one unit and a site-wide loop parks the inverter " + "that may well be the one still listening; route `sign_site` to one unit " + "and a SITE total pointing the wrong way parks one inverter while the " + "rest of the roster goes on pushing the same way - and this line goes red", _routed == {"device": "inverter-a", "supervised": "inverter-a", "failsafe": "inverter-a", "maintenance": "inverter-a", "loop": ACTUATOR_SITE, "peak": ACTUATOR_SITE, "arbitrage": ACTUATOR_SITE, "safety": ACTUATOR_SITE, - "circulation": ACTUATOR_SITE}) + "circulation": ACTUATOR_SITE, "sign_site": ACTUATOR_SITE}) check("...and the roster it was restored to is the shipped one, so nothing " "below inherits a two-inverter Controller", c_src.units == (app_main.UNIT_A,) and len(c_src.inverters) == 1) @@ -2716,7 +2733,7 @@ check("...and the roster it was restored to is the shipped one, so nothing " # actuator as a LITERAL - `actuator=self.actuator_of('')` - so the # literal is simply required to EQUAL the source the claim is filed under. That # is what makes "a mis-typed `actuator_of(...)` goes red" a true sentence about -# all nine rather than about the three a fixture happens to reach. +# all ten rather than about the three a fixture happens to reach. def append_actuator(node): @@ -2765,16 +2782,16 @@ _unit_sources_unrouted = [ if CLAIM_SCOPE.get(r[0]) == SCOPE_UNIT and "," not in r[1]] print(f" {len(_direct)} direct appends, each read at its own line: " + ", ".join(f"main.py:{ln} {s} -> {a}" for s, a, ln in _routing)) -check("all NINE of `Controller.cycle()`'s DIRECT appends route through " +check("all TEN of `Controller.cycle()`'s DIRECT appends route through " "`self.actuator_of()`, so this scan can read the routing " "of EVERY one of them. An append that computed its actuator, or omitted " "the argument, or hand-wrote a string, fails HERE rather than being " "silently skipped - which is how a scan like this goes quietly vacuous, " "and 10f is in this file because that already happened once", - len(_direct) == 9 + len(_direct) == 10 and all(a is not None and a.startswith("self.actuator_of('") for _s, a, _ln in _routing)) -check("⚠️ AND AT EVERY ONE OF THE NINE THE LITERAL IT ROUTES THROUGH IS THE " +check("⚠️ AND AT EVERY ONE OF THE TEN THE LITERAL IT ROUTES THROUGH IS THE " "SOURCE THE CLAIM IS FILED UNDER. THIS is the line a mis-typed " "`actuator_of(...)` at a call site goes red on - all nine of them, on " "the line the typo is written, at every roster, INCLUDING the six " @@ -2788,8 +2805,11 @@ check("⚠️ AND EVERY PER-INVERTER APPEND NAMES THE INVERTER IT IS ABOUT. " "site-scoped sources and is the whole defect for a per-inverter one: a " "`device` limit built inside the roster loop and routed without `inv` " "files inverter B's rating on inverter A, where it binds the wrong " - "share. Five of the nine appends are SCOPE_UNIT - `device`, " - "`supervised`, `failsafe` and maintenance's two - and all five pass one", + "share. Five of the ten appends are SCOPE_UNIT - `device`, " + "`supervised`, `failsafe` and maintenance's two - and all five pass one. " + "SAFETY-30's `sign_site` is deliberately NOT one of them: it is filed on " + "the site, beside `circulation`, because the number that tripped it is " + "the site target", _unit_sources_unrouted == [] and len([r for r in _routing if CLAIM_SCOPE.get(r[0]) == SCOPE_UNIT]) == 5) @@ -2844,13 +2864,15 @@ check("⚠️ AND THE RATING LIMITS ARE FILED ONCE PER INVERTER, ON ITS OWN " ("loop", ACTUATOR_SITE), ("supervised", "inverter-a"), ("supervised", "inverter-b")]) check("...and naming those three is not an accident of this fixture but the " - "reason the static pin exists - six of the nine sources need a state " + "reason the static pin exists - SEVEN of the ten sources need a state " "no check in this file reaches. SAFETY-31's `circulation` is one of the " - "six: it is filed only while the site is MEASURED to be circulating, " - "which no fixture here is", + "seven: it is filed only while the site is MEASURED to be circulating, " + "which no fixture here is. SAFETY-30's `sign_site` is another, and it " + "needs TWO states at once - a roster of two AND the site's total command " + "standing opposed to the meter for its whole window", {c.source for c in _cl2} == CLAIM_SOURCES - {"failsafe", "safety", "maintenance", "peak", - "arbitrage", "circulation"}) + "arbitrage", "circulation", "sign_site"}) # ⚠️ AND NOW THE HALF THAT WAS EVIDENCE-SHAPED AND OBSERVED NOTHING. The three # checks above grade WHICH ACTUATOR each claim lands on. They say nothing about # WHOSE NUMBERS the bound is built from - and those are two different defects diff --git a/goodwe_controller/test_validate.py b/goodwe_controller/test_validate.py index 535aaa6..ef0d60f 100644 --- a/goodwe_controller/test_validate.py +++ b/goodwe_controller/test_validate.py @@ -523,9 +523,26 @@ check("...and the message explains it would read as a trip, not a mistake", # check below. check("the smallest sane max_w still starts", validate_startup_config(base(max_w=1, sign_disagree_w=0.5, + unit_sign_disagree_w=0.5, maintenance_charge_w=1), -5000, 5000) == []) check("...but that same 1 W ceiling with the DEFAULT sign threshold is refused", why(validate_startup_config(base(max_w=1), -5000, 5000), "never fire")) +# ⚠️ SAFETY-30 ADDED A SECOND WAY TO DISARM A SIGN WATCHDOG BY CONFIGURATION, +# and the fixture above now has to name BOTH thresholds - which is the check +# doing its job rather than a bookkeeping edit. `unit_sign_disagree_w` bounds a +# unit's own SHARE, which the same `supervised` claim holds at +-max_w, so at or +# above the ceiling it is unsatisfiable by construction exactly as +# `sign_disagree_w` is. Refused even at the shipped roster of one, where this +# watchdog is not armed: a config that cannot work is refused when it is +# written, not on the day somebody adds a second inverter. +check("...and the per-inverter threshold has the same rule, named separately " + "so an operator is told WHICH watchdog they disarmed", + why(validate_startup_config(base(max_w=1, sign_disagree_w=0.5, + maintenance_charge_w=1), -5000, 5000), + "unit_sign_disagree_w")) +check("...and one watt under that ceiling is reachable, so accepted", + validate_startup_config(base(max_w=2000, unit_sign_disagree_w=1999), + -5000, 5000) == []) # --------------------------------------------------------------------------- # print("what must NOT be refused - a validator that fires on trivia gets deleted") diff --git a/goodwe_controller/test_watchdog.py b/goodwe_controller/test_watchdog.py index 6e7fdad..7ce60b8 100644 --- a/goodwe_controller/test_watchdog.py +++ b/goodwe_controller/test_watchdog.py @@ -401,10 +401,19 @@ print("controller and the first thing an operator will try") class _MemStore: - """A Store that keeps its dict across "restarts", like the real file does.""" + """A Store that keeps its dict across "restarts", like the real file does. + + ⚠️ SEEDED FROM `store.DEFAULTS`, NOT FROM A LITERAL. `Store.load()` merges + that dict over whatever is on disk, so a fake that hand-listed three keys + answered `None` where the real store answers a default - and a check reading + "this latch was never written" then could not tell an absent key from a + False one. It is also a place to drift: SAFETY-31 and SAFETY-30 both added + flat pairs, and neither would have appeared here. + """ def __init__(self): - self.data = {"phase": "idle", "sign_tripped": False, "sign_reason": ""} + from app.store import DEFAULTS as _SDEF + self.data = dict(_SDEF) def set(self, k, v): self.data[k] = v @@ -1105,6 +1114,701 @@ check("⚠️ AND THE 'HAS A HUMAN SEEN THIS' FLAG IS CLEARED WITH IT. Left " _cc._circ_announced is False) + +print() +print("=" * 72) +print("SAFETY-30: at two inverters the sign check is TWO questions, not one") +print("=" * 72) +# ⚠️ READ THIS BEFORE ADDING A CHECK HERE. Half of this card was already built +# and it is NOT the half below. DEV-02 shipped the PER-INSTANCE half: each +# `Inverter` owns its own `SignWatchdog`, latched under its own namespaced store +# keys, and a trip on B no longer latches A. test_inverter.py part 3 grades +# that; nothing here re-tests it. +# +# What is graded here is THE COMPARISON ITSELF. Before this card every unit's +# detector was handed `self.grid`. There is ONE P1, so at N>=2 that is a number +# this controller put on ONE actuator, graded against a measurement that answers +# for ALL of them - wrong in both directions: a FALSE trip when a sibling +# accounts for the meter, and a MISSED one when a sibling MASKS a unit that +# really is reversed. The fix is two comparisons, and they are two different +# faults: each unit against ITS OWN measured power, and the SUM of the standing +# commands against the meter. +# +# ⚠️ AND IT IS NOT SAFETY-31, WHICH IS THE SECTION ABOVE. `circulation.py` +# compares the units AGAINST EACH OTHER and catches two inverters pushing +# THROUGH each other, which nets to zero at the meter. This compares each unit +# against ITS OWN COMMAND and catches a unit not doing what it was told. They +# overlap in exactly one scenario - a unit reversing while its sibling pulls the +# other way - and part 4 below asserts what each one concludes there, because +# two guards that overlap without saying so is how a false-alarm generator gets +# built. + +from app.control import Tuning as _Tuning # noqa: E402 +from app.watchdog import ( # noqa: E402 + AGAINST_METER, AGAINST_OWN_POWER, SIGN_OPTION_DEFAULTS, disagrees, +) + +print() +print("1. ONE predicate, TWO second terms - and the trip sentence says which") +# ⚠️ THE ARITHMETIC IS UNCHANGED AND THAT IS THE POINT: `disagrees()` was not +# split in two. What moved is a PARAMETER NAME - `grid_w` became `against_w` - +# because what the command is opposed TO is now the caller's decision. A second +# copy of a safety predicate is one copy plus a future disagreement about what +# the check means (see `loop_is_following()`), so there is still exactly one. +_scfg = SignWatchdogConfig(min_w=300.0, hold_s=180.0) +_ucfg = SignWatchdogConfig( + min_w=float(SIGN_OPTION_DEFAULTS["unit_sign_disagree_w"]), + hold_s=float(SIGN_OPTION_DEFAULTS["unit_sign_disagree_s"])) +check("the same call that means 'commanded -1000 W into a +1500 W import' " + "means 'commanded -1000 W while measuring +1500 W of discharge' when the " + "caller hands it a unit's own power - one function, two questions", + disagrees(-1000.0, 1500.0, True, _scfg) is True) +check("...and every gate still binds on the second term whatever it is: a " + "250 W reading does not qualify against a 300 W gate", + disagrees(-1000.0, 250.0, True, _scfg) is False) +check("...nor does a unit sitting at zero, which is a REFUSED command and not " + "a reversed one - the distinction the module docstring makes about a " + "battery declining a discharge, now read of the unit's own sensor", + disagrees(1500.0, 0.0, True, _ucfg) is False) + +_wd_site = SignWatchdog(_scfg) +_wd_unit = SignWatchdog(_ucfg, against=AGAINST_OWN_POWER) +for _t in range(0, 400, 5): + _wd_site.evaluate(-1000.0, 1500.0, True, T0 + timedelta(seconds=_t)) + _wd_unit.evaluate(-1000.0, 1500.0, True, T0 + timedelta(seconds=_t)) +print(f" site: {_wd_site.reason!r}") +print(f" unit: {_wd_unit.reason!r}") +check("⚠️ THE SITE INSTANCE'S SENTENCE IS BYTE-IDENTICAL TO THE ONE THAT " + "SHIPPED. It is what an operator reads on the status page, what goes in " + "the action log and what is persisted to /data/state.json, so it is " + "asserted against the literal rather than against a format string", + _wd_site.reason == ( + "SIGN_DISAGREEMENT: commanded -1000 W against a meter reading " + "+1500 W for 180 s continuously (limits: both over 300 W, held over " + "180 s)")) +check("...and the per-unit instance names what it ACTUALLY compared instead. " + "'against a meter reading' there would be a false statement about which " + "instrument produced the evidence, on the one line somebody woken by " + "this alarm reads first", + AGAINST_OWN_POWER in _wd_unit.reason + and AGAINST_METER not in _wd_unit.reason + and "+1500 W" in _wd_unit.reason) +# ⚠️ `arm()` IS CONFIGURATION AND MUST NOT BE STATE. `Controller.cycle()` calls +# it on EVERY cycle, so a version that cleared `disagree_since` there would +# restart the window every cycle, never accumulate a hold, and leave the check +# permanently green with nothing saying so. Driven rather than argued: open a +# window, re-arm mid-episode, and see the window still standing. +_rearm = SignWatchdog(_ucfg, against=AGAINST_OWN_POWER) +_rearm.evaluate(-1000.0, 1500.0, True, T0) +_rearm.evaluate(-1000.0, 1500.0, True, T0 + timedelta(seconds=5)) +_rearm.arm(_ucfg, AGAINST_OWN_POWER) +check("re-arming mid-episode leaves the window exactly where it was", + _rearm.disagree_since == T0) +for _t in range(10, 200, 5): + _rearm.arm(_ucfg, AGAINST_OWN_POWER) + _rearm.evaluate(-1000.0, 1500.0, True, T0 + timedelta(seconds=_t)) +check("...so a detector re-armed on every one of 38 cycles still trips at its " + "85 s hold, which is what says `arm()` did not quietly disarm it", + _rearm.tripped) + +print() +print("2. CFG-03: config.yaml is the source of truth and there is ONE copy") +_sign_from_yaml = {k: int(v) for k, v in + _re.findall(r"^ ((?:unit|site)_sign_disagree_\w+):\s*(\d+)\s*$", + _cfg_opts, _re.M)} +print(f" config.yaml options: {_sign_from_yaml}") +print(f" SIGN_OPTION_DEFAULTS: {SIGN_OPTION_DEFAULTS}") +check("all four new options are declared in config.yaml's options: block and " + "the in-code mirror agrees key for key and value for value - the " + "Dockerfile does not put config.yaml in the image, so the running add-on " + "cannot read it and this is where the two can drift", + _sign_from_yaml == SIGN_OPTION_DEFAULTS and len(_sign_from_yaml) == 4) +_sign_bounds = {m[0]: (int(m[1]), int(m[2])) for m in + _re.findall(r"^ ((?:unit|site)_sign_disagree_\w+):" + r"\s*int\((\d+),(\d+)\)\s*$", + _cfg_schema, _re.M)} +print(f" schema bounds: {_sign_bounds}") +check("...and ALL FOUR have a schema range, so the Supervisor refuses an " + "out-of-range value rather than the add-on discovering it at runtime", + set(_sign_bounds) == set(SIGN_OPTION_DEFAULTS)) +check("the shipped defaults sit inside their own schema ranges", + all(_sign_bounds[k][0] <= v <= _sign_bounds[k][1] + for k, v in SIGN_OPTION_DEFAULTS.items())) +check("⚠️ THE THRESHOLD IS `saturation_w / 2`, WHICH IS WHERE THE CONTROL LAW " + "ITSELF ALREADY DRAWS THE LINE. This check gates on BOTH sides, and the " + "two are OPPOSED, so the smallest command-versus-power gap it can ever " + "see is 2 x min_w. At 250 W that is exactly the 500 W at which " + "`compute()`'s own anti-windup test concludes the inverter is not " + "following and freezes the integrator - so this watchdog cannot fire on " + "a divergence the shipped loop calls ordinary tracking, and the two " + "guards meet at a number instead of overlapping in a band nobody drew", + SIGN_OPTION_DEFAULTS["unit_sign_disagree_w"] * 2 + == _Tuning().saturation_w) +check("...and the window clears its three bounds: above the firmware's own " + "30 s link veto (`inverter.LINK_STALE_S` - inside it, a unit not " + "executing its share is a LINK fault the add-on already answers by " + "dropping it from the split), and UNDER HALF the 180 s the site check " + "takes, because the per-unit conclusion needs nothing to net out before " + "it is complete", + SIGN_OPTION_DEFAULTS["unit_sign_disagree_s"] > LINK_STALE_S + and SIGN_OPTION_DEFAULTS["unit_sign_disagree_s"] < 180 / 2) +check("⚠️ AND NEITHER IS A ROUND NUMBER. 250 W is read off `saturation_w`; " + "85 s is 8.5x the worst legitimate opposition 1890 two-inverter runs " + "produce (10.0 s) and 2.8x the firmware veto. 60, 75, 90, 100, 120, 180 " + "and 300 are all values a guess would have landed on", + SIGN_OPTION_DEFAULTS["unit_sign_disagree_s"] + not in (30, 60, 75, 90, 100, 120, 180, 300)) + +# ⚠️ AND THE SITE PAIR IS TWO DECISIONS, NOT ONE, WHICH IS WHY BOTH ARE PINNED. +# The THRESHOLD is carried from the shipped pair because its argument is about +# the meter's noise floor and the deadband - a property of ONE P1 and of +# `Tuning.deadband_w`, neither of which the roster touches. The WINDOW is NOT +# carried, because the transient it rejects is the loop crossing zero across a +# rail that IS the roster sum: measured at 110 s with one inverter and 215 s +# with two, on the same stimuli, against a 180 s hold. That is a carried number +# justified and a carried number refused, in the same pair. +check("the SITE threshold is the shipped one, carried - the noise argument " + "behind 300 W is about the meter and the deadband, and a second inverter " + "changes neither", + SIGN_OPTION_DEFAULTS["site_sign_disagree_w"] == 300) +check("⚠️ ...AND THE SITE WINDOW IS LONGER THAN THE SHIPPED 180 s, WHICH IS THE " + "MEASUREMENT REFUSING TO BE CARRIED. At two inverters the worst " + "legitimate opposition the rig produces is 215.0 s - reachable at " + "`gain: 0.05`, `slew_w: 50` and a 2.2 kW load reversal, all inside what " + "config.yaml accepts - so 180 s would have shipped a false trip on the " + "check with the best record in the safety table. 325 s is 1.51x that", + SIGN_OPTION_DEFAULTS["site_sign_disagree_s"] > 215.0 + and SIGN_OPTION_DEFAULTS["site_sign_disagree_s"] < 215.0 * 2) +check("...and it is still inside the 900 s quarter that bills the capacity " + "tariff, and inside every episode the 2026-08-20 fault day contains - " + "its four clusters run 16, 30, 77 and 77 minutes", + SIGN_OPTION_DEFAULTS["site_sign_disagree_s"] < 900 + and SIGN_OPTION_DEFAULTS["site_sign_disagree_s"] < 16 * 60) +check("...and it is deliberately not 300 or 360, the two round numbers next to " + "it", + SIGN_OPTION_DEFAULTS["site_sign_disagree_s"] not in (300, 360, 600, 900)) + +print() +print("3. AC 1: ONE unit drives the wrong way, and ONLY that unit trips") +# ⚠️ THE PLANT IS MODELLED AND SAYS SO. `_SignHass` is two inverters that obey +# INSTANTLY - no ramp, no lag - and one meter that is `load - sum(measured)`. +# That is deliberately not a claim about transients: the worst legitimate +# opposition a real ramp and lag can produce is measured on admin/goodwe's +# two-master rig, over 1890 closed-loop runs, and recorded in +# DECISIONS/SAFETY-30-per-unit-sign.md. What this fixture is for is the +# ATTRIBUTION - which detector concludes what, from which instrument - and for +# that an instant plant is the honest model, because it removes the one +# mechanism that could make a unit look reversed without being reversed. + + +class _SignHass(_Hass): + """Two inverters, one meter, and a chosen subset that does the OPPOSITE. + + `reversed_units` are roster indices whose measured power is the NEGATION of + what this controller told them - HW-01's wedged power stage, an inverted + driver, a unit executing a stale share backwards. Everything else obeys. + """ + + def __init__(self, load_w=2000.0, reversed_units=()): + super().__init__() + self.load_w = float(load_w) + self.reversed_units = set(reversed_units) + self.ctrl = None + + def measured(self): + if self.ctrl is None: + return [0.0, 0.0] + out = [] + for i, inv in enumerate(self.ctrl.inverters): + w = inv.last_write if inv.last_write is not None else inv.target + w = float(w or 0.0) + out.append(-w if i in self.reversed_units else w) + return out + + async def number(self, entity, invert=False): + m = self.measured() + [0.0, 0.0] + return {"sensor.grid": self.load_w - sum(m[:2]), + "sensor.soc": 50.0, "sensor.soc_b": 50.0, + "sensor.batt": m[0], "sensor.batt_b": m[1]}.get(entity) + + +def _sign_site(hass, store=None, **over): + """A two-inverter Controller, with SAFETY-31 deliberately given a long window. + + ⚠️ THE LONG `circulation_hold_s` IS ISOLATION, NOT A CLAIM THAT SAFETY-31 IS + SILENT HERE. A unit reversing while its sibling pulls the other way IS + circulating power and SAFETY-31 sees it - part 4 asserts exactly that, and + what each guard concludes about it. What must not happen is the SITE being + stopped by circulation while this section is trying to grade a PER-UNIT + attribution, because then every "only B is parked" check below would be + measuring the wrong guard. + + ⚠️ THE THREE THRESHOLD PAIRS ARE PAIRWISE DISTINCT IN BOTH COORDINATES, + AND THAT IS LOAD-BEARING RATHER THAN COSMETIC. This fixture is the only + place in the suite where `_arm_sign_watchdogs()` runs at a roster of two, so + it is the only place that can see WHICH pair reaches WHICH detector. The + first version of it set every hold to 1 s and two of the three magnitude + gates to 300 W; a detector armed with the shipped pair, or with the site + pair, or with the right magnitude and the wrong hold, then compared EQUAL + to the pair it was supposed to have, and the wiring check below would have + passed on three of those four arrangements. That is MUTATION.md's third + failure mode written into a fixture - a check comparing two things that + happen to be equal cannot see the difference it names - and the + precondition beside the check is what stops a later edit collapsing them + again without a red line. + + Anyone changing a hold here: the PER-UNIT hold is what part 3's trip timing + is read against, and the SITE hold must still expire inside `_drive()`'s + window (14 cycles x 0.2 s), which part 5 depends on. + """ + opts = {"setpoint_entity": "number.sp", "meter_entity": "sensor.grid", + "soc_entity": "sensor.soc", "batt_entity": "sensor.batt", + "meter_source": "off", "max_w": 2000, "gain": 0.6, + "sign_disagree_w": 300, "sign_disagree_s": 3, + "unit_sign_disagree_w": 250, "unit_sign_disagree_s": 1, + "site_sign_disagree_w": 320, "site_sign_disagree_s": 2, + "circulation_hold_s": 3600, "auto_start": True} + opts.update(over) + c = Controller(opts, hass, store if store is not None else _MemStore(), + _P(), stop=asyncio.Event()) + b = Inverter("inverter-b", setpoint_entity="number.sp_b", + soc_entity="sensor.soc_b", batt_entity="sensor.batt_b", + max_w=B_RATING_HYPOTHESIS, store=c.store) + b.dev_min, b.dev_max = -B_RATING_HYPOTHESIS, B_RATING_HYPOTHESIS + c.inverters.append(b) + c.auto = True + for inv in c.inverters: + if inv.dev_min is None: + inv.dev_min, inv.dev_max = -5000.0, 5000.0 + hass.ctrl = c + return c + + +async def _drive(c, n=14, dt=0.2, watch=None): + seen = [] + for _ in range(n): + await c.read_inputs() + await c.cycle() + if watch is not None: + seen.append(watch(c)) + await asyncio.sleep(dt) + return seen + + +async def _one_reversed(): + hass = _SignHass(load_w=2000.0, reversed_units={1}) + c = _sign_site(hass) + seen = await _drive(c, watch=lambda c: dict( + grid=c.grid, + stood=[i.last_write if i.last_write is not None else i.target + for i in c.inverters], + meas=[i.batt for i in c.inverters], + a=c.inverters[0].sign_wd.tripped, b=c.inverters[1].sign_wd.tripped, + site=c.site_sign_wd.tripped)) + return c, hass, seen + + +_c30, _h30, _s30 = asyncio.run(_one_reversed()) +_a30, _b30 = _c30.inverters + +# ⚠️ THE WIRING PIN FOR AC 5, AND IT IS A DIFFERENT CLAIM FROM EVERY +# BEHAVIOURAL CHECK IN THIS SECTION. The rest of part 3 grades what the +# detectors CONCLUDED; this grades WHICH THRESHOLDS THEY WERE HANDED, and the +# gap between the two was live. Replacing +# `arm(self.sign_cfg if one else self.unit_sign_cfg, ...)` with +# `arm(self.sign_cfg, ...)` in `Controller._arm_sign_watchdogs()` throws the +# re-derived per-unit pair away and arms every unit with SAFETY-02's +# meter-facing one - and left ALL 19 FILES GREEN. Nothing at a roster of two +# read `inv.sign_wd.cfg` at all; the one place that does is part 7's N=1 arm, +# which that edit PRESERVES. +# +# WHY THE TRIPS BELOW CANNOT CATCH IT, said here because "the suite is green" +# and "nothing was wrong" are the two readings this comment exists to separate. +# B is reversed at ~2000 W - far over 250 W and far over 300 W - and every hold +# in this fixture is short enough to expire inside `_drive()`. A fault that is +# big enough and long enough to trip under BOTH pairs is evidence about +# neither. Only an equality on the pair itself can tell them apart. +# +# WHAT THE SURVIVOR COSTS IN THE FIELD, at config.yaml's own defaults: the +# per-unit check runs at `sign_disagree_s` 180 s instead of the re-derived 85 s +# - 2.1x slower to park a reversed inverter - and its gate moves from 250 W to +# 300 W, so a unit reversing a share between the two becomes invisible, while +# watchdog.SIGN_OPTION_DEFAULTS, config.yaml and DECISIONS section 3 all go on +# describing a pair the running code does not use. +_UNIT_PAIR = (250.0, 1.0) # `unit_sign_disagree_w` / `_s` - one per unit +_SITE_PAIR = (320.0, 2.0) # `site_sign_disagree_w` / `_s` - one per site +_SHIPPED_PAIR = (300.0, 3.0) # `sign_disagree_w` / `_s` - N=1 only +_armed = {inv.name: (inv.sign_wd.cfg.min_w, inv.sign_wd.cfg.hold_s) + for inv in _c30.inverters} +_armed_site = (_c30.site_sign_wd.cfg.min_w, _c30.site_sign_wd.cfg.hold_s) +print(f" armed per unit: {_armed}") +print(f" armed site: {_armed_site} " + f"shipped pair, wanted by NOBODY here: {_SHIPPED_PAIR}") +check("PRECONDITION: the three pairs are pairwise distinct IN BOTH " + "COORDINATES, so an equality below can fail on the magnitude alone, on " + "the hold alone, or on both. This fixture originally gave all three the " + "same 1 s hold and two of them the same 300 W gate, and a mis-wiring " + "that delivered the right magnitude with the wrong hold would then have " + "compared EQUAL to the pair it was supposed to have", + len({_UNIT_PAIR, _SITE_PAIR, _SHIPPED_PAIR}) == 3 + and len({p[0] for p in (_UNIT_PAIR, _SITE_PAIR, _SHIPPED_PAIR)}) == 3 + and len({p[1] for p in (_UNIT_PAIR, _SITE_PAIR, _SHIPPED_PAIR)}) == 3) +check("PRECONDITION: and all three are pairs this Controller actually holds, " + "so the literals below are the fixture's own numbers and not three " + "constants that agree with nothing. In particular the shipped pair is " + "still constructed and still a reachable mis-wiring target", + (_c30.unit_sign_cfg.min_w, _c30.unit_sign_cfg.hold_s) == _UNIT_PAIR + and (_c30.site_sign_cfg.min_w, _c30.site_sign_cfg.hold_s) == _SITE_PAIR + and (_c30.sign_cfg.min_w, _c30.sign_cfg.hold_s) == _SHIPPED_PAIR) +check("⚠️ AC 5 WIRING. EVERY per-unit detector on this roster of two is armed " + "with the PER-UNIT pair - 250 W / 1 s here, `unit_sign_disagree_w` and " + "`unit_sign_disagree_s` in the field. Graded against LITERALS rather " + "than against `_c30.unit_sign_cfg`, which cannot tell 'the per-unit " + "pair' from 'whatever this Controller was handed'. This is the line that " + "goes red when `_arm_sign_watchdogs()` stops choosing between the two " + "pairs and hands every unit the site-facing one", + len(_armed) == 2 and all(p == _UNIT_PAIR for p in _armed.values())) +check("...and the SITE detector is armed with the SITE pair - 320 W / 2 s " + "here - which differs from the per-unit pair AND from the shipped pair " + "in both coordinates, so this equality is not carried by a number the " + "fixture happens to hold twice", + _armed_site == _SITE_PAIR) +check("...and each detector is pointed at its own INSTRUMENT as well as at its " + "own thresholds: the units at their own 0x0518 power, the site at the " + "meter. They are two arguments of one call and a pin on one is not a pin " + "on the other", + all(inv.sign_wd.against == AGAINST_OWN_POWER for inv in _c30.inverters) + and _c30.site_sign_wd.against == AGAINST_METER) + +_mid = _s30[3] +print(f" cycle 3: stood {[round(x) for x in _mid['stood']]} " + f"measured {[round(x) for x in _mid['meas']]} " + f"meter {_mid['grid']:+.0f} W") +check("PRECONDITION: the stimulus is not vacuous - inverter B really is doing " + "the OPPOSITE of what it was told, and inverter A really is obeying. " + "Without this line every claim below could be true of a fixture in which " + "nothing was reversed", + _mid["stood"][1] * _mid["meas"][1] < 0 + and abs(_mid["meas"][1]) > 250 + and _mid["stood"][0] * _mid["meas"][0] > 0) +check("⚠️ AC 1. THE REVERSED UNIT TRIPS AND ITS SIBLING DOES NOT. The evidence " + "is B's own 0x0518 power against B's own share, so it is a fact about B " + "and nothing about A - which is what makes parking one of them the right " + "remedy rather than an attribution nobody measured", + _b30.sign_wd.tripped and not _a30.sign_wd.tripped) +check("...and the reason names the instrument, so the operator is sent to the " + "right one", AGAINST_OWN_POWER in _b30.sign_wd.reason) +print(f" {_b30.sign_wd.reason!r}") +# ⚠️ READ OFF THE ALLOCATION RECORD, NOT OFF `Controller.reason`. At a roster +# of two `reason` is the SITE resolution's line - "loop -> 2000 W" - because +# B's failsafe is a UNIT claim resolved on B's own actuator and never competes +# for the site winner. That is the DEV-02 split working, and a check that +# grepped the site line for "failsafe" would go green only while the roster was +# one. The per-unit record is where the pin is stated, and it names its source. +_bplan = [pl for pl in (_c30.allocation.plans if _c30.allocation else ()) + if pl.unit == "inverter-b"] +print(f" allocation: {_c30.allocation.explain()}") +print(f" inverter-b plan: {_bplan[0].why if _bplan else '(none)'}") +check("...B is commanded 0 W by its OWN `failsafe` claim, named in the " + "allocation record, rather than by some other guard happening to stop it", + _b30.target == 0.0 and _bplan and "failsafe" in _bplan[0].why) +check("...and A IS STILL COMMANDING. One inverter in trouble stops that " + "inverter; the site goes on running on the one that works, which is what " + "a UNIT-scoped claim buys and what a site-wide stop would have thrown " + "away", _a30.target != 0.0) +check("...the latch is under B's OWN namespaced key, and A's is untouched", + _c30.store.data.get(unit_key("inverter-b", "sign_tripped")) is True + and _c30.store.data.get(TRIPPED_KEY) is False) + +print() +print("4. AC 2: and it is caught EVEN THOUGH THE METER SAYS EVERYTHING IS FINE") +# ⚠️ THIS IS THE CHECK THE WHOLE CARD IS FOR, so it is built the way MUTATION.md +# says to build one: the claim is "the new comparison sees what the old one +# structurally cannot", and the OLD comparison is RUN, on the same samples, and +# its blindness is asserted as its own precondition and PRINTED - not argued. +_old_view = [disagrees(s["stood"][1], s["grid"], True, _scfg) for s in _s30] +_site_view = [disagrees(sum(s["stood"]), s["grid"], True, _scfg) for s in _s30] +print(f" the SHIPPED comparison (B's command vs the meter) was true on " + f"{sum(_old_view)} of {len(_old_view)} cycles") +print(f" the SITE comparison (sum of commands vs the meter) on " + f"{sum(_site_view)} of {len(_site_view)}") +check("⚠️ THE COMPARISON THAT SHIPPED IS FALSE ON EVERY SINGLE CYCLE HERE. B is " + "commanded to discharge and the house is importing, so B's command and " + "the meter AGREE all the way through - the meter cannot tell that the " + "watts arriving are inverter A's while B does the opposite. This is not " + "a threshold set too high: the summation destroys the evidence and no " + "tuning recovers it", + not any(_old_view)) +check("...and the SITE comparison is blind for the same reason, which is why " + "it is kept as a SEPARATE question rather than offered as the fix", + not any(_site_view)) +check("...while the shipped SITE watchdog stayed correctly silent, so nothing " + "here is a site-level trip wearing a per-unit label", + not _c30.site_sign_wd.tripped + and _c30.store.data.get("site_sign_tripped") is not True) +check("⚠️ AND SAFETY-31 SEES SOMETHING TOO, WHICH IS NOT A DUPLICATE AND IS " + "SAID HERE RATHER THAN LEFT FOR SOMEBODY TO DISCOVER. A unit reversing " + "while its sibling pulls the other way IS power going round a loop, so " + "`circulating_w()` is over its threshold on this fixture - it is only " + "the long window this fixture gives it that keeps it from stopping the " + "site. The two guards CONCLUDE different things: circulating power is " + "defined over the roster and names no offender, while this one names B " + "and parks B. Neither is the other with a different threshold", + max(circulating_w(s["meas"]) for s in _s30) > _c30.circ_wd.cfg.min_w + and not _c30.circ_wd.tripped + and _c30.site_circulating_w == 0.0) + +print(f" circulating power over the episode: " + f"{[round(circulating_w(s['meas'])) for s in _s30]}") +check("...and it falls back to ZERO the moment B is parked, which is the " + "remedy working rather than the sensor going quiet: a unit held at 0 W " + "cannot be half of a loop. It is asserted on the EPISODE rather than on " + "the end state for exactly that reason", + _s30[-1]["meas"][1] == 0.0) + +print() +print("5. AC 3: the SITE question survives, separately, and catches whole-site") +print(" reversal - which no per-unit check can see") +# ⚠️ THE FAULT HERE IS THE ONE THE 2026-08-20 CAPTURE CONTAINS, and it is +# INVISIBLE to every per-unit comparison by construction: both inverters are +# doing EXACTLY what they were told. What is wrong is the number they were told. +# A negative gain is the minimal faithful model of it - an inverted polarity, a +# wound-up integrator, a strategy on the wrong side of the meter - and it is not +# a reachable configuration (`gain` is float(0.05,1.0)), which is the point: it +# stands in for a code regression, which is the case a check can hold the line on. + + +async def _site_reversed(): + hass = _SignHass(load_w=2000.0, reversed_units=()) + c = _sign_site(hass, gain=-0.6) + seen = await _drive(c, watch=lambda c: dict( + grid=c.grid, site=c.site_sign_wd.tripped, + stood=[i.last_write if i.last_write is not None else i.target + for i in c.inverters], + meas=[i.batt for i in c.inverters])) + return c, hass, seen + + +_c31, _h31, _s31 = asyncio.run(_site_reversed()) +# ⚠️ THE LAST SAMPLE BEFORE THE TRIP, NOT A FIXED INDEX. After the trip every +# command is 0 W by design, so a fixed index reads the REMEDY rather than the +# fault and the precondition below would be asserting that a stopped site is +# stopped. This is `Run.p2p_between`'s lesson from the rig, in a unit test: name +# the window rather than inferring it. +_pre31 = [x for x in _s31 if not x["site"]] +_mid31 = _pre31[-1] +print(f" last cycle before the trip: stood " + f"{[round(x) for x in _mid31['stood']]} " + f"measured {[round(x) for x in _mid31['meas']]} " + f"meter {_mid31['grid']:+.0f} W") +check("PRECONDITION: EVERY inverter is obeying - each unit's measured power " + "agrees in sign with its own share - and the site is still charging into " + "an importing house. Without this the site trip below could be a " + "per-unit fault in disguise", + all(s * m > 0 for s, m in zip(_mid31["stood"], _mid31["meas"])) + and sum(_mid31["stood"]) * _mid31["grid"] < 0) +check("⚠️ AC 3. THE SITE WATCHDOG TRIPS AND NOT ONE PER-UNIT WATCHDOG DOES. " + "That asymmetry is the card: the fault is in the number the site was " + "asking for, which no single inverter owns, and every unit is innocent " + "of it", + _c31.site_sign_wd.tripped + and not any(i.sign_wd.tripped for i in _c31.inverters)) +print(f" {_c31.site_sign_wd.reason!r}") +check("...and it stops the WHOLE site, through the `sign_site` claim by name " + "rather than by some other guard happening to stop it", + _c31.target == 0.0 and "sign_site" in _c31.reason + and all(i.target == 0.0 for i in _c31.inverters)) +check("...the latch is FLAT - `site_sign_tripped`, beside SAFETY-31's pair - " + "and NOT mirrored onto the pre-DEV-02 `sign_tripped`, which is the " + "rollback view of a UNIT latch. A downgraded add-on drives exactly one " + "actuator; handing it a latch taken on evidence about a roster it cannot " + "have would be a trip it can never explain", + _c31.store.data.get("site_sign_tripped") is True + and "SIGN_DISAGREEMENT" in str(_c31.store.data.get("site_sign_reason")) + and _c31.store.data.get("sign_tripped") is not True) +check("the status page carries a SITE row of its own, and the banner says the " + "site's TOTAL command opposed the meter rather than naming an inverter - " + "naming one would send an operator to swap the wrong box", + any("SITE SIGN-DISAGREEMENT FAILSAFE" in ck["text"] + for ck in _c31.checks()) + and "total command" in _c31.status_payload()["banner"]) +_pub30 = {} +_c31.mqtt = type("P", (), {"enabled": True, + "publish": lambda _s, v: _pub30.update(v), + "close": lambda _s: None})() +_c31.publish() +check("...and the controller-status sensor reports the failsafe state, so a " + "site parked at 0 W cannot publish 'running'", + _pub30.get("status") == "failsafe") + +# The latch, and the one way out of it. +_was31 = _c31.site_sign_wd.tripped +asyncio.run(_c31.handle_action("auto_toggle")) # stop +check("pressing STOP clears nothing - the condition always goes away on its " + "own, because the trip commands 0 W and a site at zero cannot oppose " + "anything", _c31.site_sign_wd.tripped and _was31) +asyncio.run(_c31.handle_action("auto_toggle")) # the deliberate start +check("pressing START clears the latch in memory and on disk together, which " + "is the same deliberate human act that clears every other latch here", + not _c31.site_sign_wd.tripped + and _c31.store.data.get("site_sign_tripped") is False + and _c31.store.data.get("site_sign_reason") == "") + + +async def _site_restart(store): + hass = _SignHass(load_w=2000.0) + c = _sign_site(hass, store=store, gain=-0.6) + await _drive(c, n=3, dt=0.05) + return c + + +_store31 = _MemStore() +_store31.data["site_sign_tripped"] = True +_store31.data["site_sign_reason"] = "SIGN_DISAGREEMENT: (from before the restart)" +_c31r = asyncio.run(_site_restart(_store31)) +check("⚠️ AND IT SURVIVES A RESTART, which is the only thing that makes it a " + "latch rather than a pause. 'Restart the add-on' is the first thing an " + "operator staring at a FAILSAFE banner tries, and a site that came back " + "armed would drive straight into the fault it had just stopped for", + _c31r.site_sign_wd.tripped and _c31r.target == 0.0 + and "sign_site" in _c31r.reason) + +print() +print("6. AC 4: two inverters commanded OPPOSITE SIGNS by a legitimate") +print(" allocation, and nothing trips") +# ⚠️ THIS IS THE FALSE TRIP THE CARD OPENS WITH, AND IT IS DRIVEN THROUGH THE +# SHIPPED PREDICATE RATHER THAN DESCRIBED. Inverter A charging 1500 W while +# inverter B discharges 1500 W into a 1500 W import is an ordinary two-inverter +# arrangement - a per-unit maintenance charge on A while the loop drives B, or +# SAFETY-25's floor pinning one unit. Nothing is faulty. Before this card the +# meter convicted A on it. +_A_CMD, _B_CMD, _METER = -1500.0, 1500.0, 1500.0 +_wd_a = SignWatchdog(_ucfg, against=AGAINST_OWN_POWER) +_wd_b = SignWatchdog(_ucfg, against=AGAINST_OWN_POWER) +_sitecfg = SignWatchdogConfig( + min_w=float(SIGN_OPTION_DEFAULTS["site_sign_disagree_w"]), + hold_s=float(SIGN_OPTION_DEFAULTS["site_sign_disagree_s"])) +_wd_s = SignWatchdog(_sitecfg) +for _t in range(0, 1200, 5): + _wd_a.evaluate(_A_CMD, _A_CMD, True, T0 + timedelta(seconds=_t)) + _wd_b.evaluate(_B_CMD, _B_CMD, True, T0 + timedelta(seconds=_t)) + _wd_s.evaluate(_A_CMD + _B_CMD, _METER, True, T0 + timedelta(seconds=_t)) +print(f" A commanded {_A_CMD:+.0f} W and measured {_A_CMD:+.0f} W; " + f"B {_B_CMD:+.0f} / {_B_CMD:+.0f}; meter {_METER:+.0f} W") +check("PRECONDITION: THE COMPARISON THAT SHIPPED CONVICTS INVERTER A HERE, on " + "every one of these samples. A is commanded to CHARGE while the meter " + "reads IMPORT - opposed, both far over 300 W, and it would have held for " + "20 minutes. That is the false trip, and it is asserted rather than " + "asserted about", + disagrees(_A_CMD, _METER, True, _scfg) is True) +check("⚠️ AC 4. NEITHER UNIT TRIPS UNDER THE NEW COMPARISON, over 20 minutes at " + "an 85 s hold. Each one is doing exactly what it was told, and its own " + "power sensor is the instrument that can say so", + not _wd_a.tripped and not _wd_b.tripped) +check("...and the SITE watchdog does not trip either, because the shares SUM " + "to zero and a 0 W total is not a wrong-direction command - the " + "magnitude gate that has always been there, now applied to the quantity " + "it belongs on", not _wd_s.tripped and _wd_s.disagree_since is None) +# ⚠️ AND THE COUNTERFACTUAL, BECAUSE "nothing tripped" IS ALSO WHAT A DISARMED +# GUARD REPORTS. Same site, same meter, but the shares no longer cancel: a NET +# -1600 W of charge into a +1500 W import is the site genuinely driving the +# wrong way, and it must trip. Without this line the check above passes +# identically against a site watchdog that was never wired to anything. +_wd_s_net = SignWatchdog(_sitecfg) +for _t in range(0, 600, 5): + _wd_s_net.evaluate(-1600.0, _METER, True, T0 + timedelta(seconds=_t)) +check("...and the site question is not thereby disarmed: the same site, " + "commanded a NET -1600 W into the same import, DOES trip", + _wd_s_net.tripped) + +print() +print("7. AC 8: at ONE inverter this card changes NOTHING, and 'nothing' is") +print(" asserted rather than assumed") + + +async def _n1_sign(): + hass = _SignHass(load_w=2000.0) + opts = {"setpoint_entity": "number.sp", "meter_entity": "sensor.grid", + "soc_entity": "sensor.soc", "batt_entity": "sensor.batt", + "meter_source": "off", "max_w": 2000, "gain": -0.6, + "sign_disagree_w": 300, "sign_disagree_s": 1, + "auto_start": True} + c = Controller(opts, hass, _MemStore(), _P(), stop=asyncio.Event()) + c.auto = True + c.dev_min, c.dev_max = -5000.0, 5000.0 + hass.ctrl = c + # ⚠️ A CALL COUNTER, NOT AN OBSERVATION OF THE DETECTOR'S STATE, AND A + # MUTATION IS WHY. The first version of the check below read + # `site_sign_wd.disagree_since is None` - and arming the site detector at + # N=1 as well left this suite GREEN, because the unit detector trips first + # at its 1 s hold, parks the inverter, and the site's standing command is + # then 0 W: the site predicate is false, so its window is never opened and + # "never evaluated" reads identically to "evaluated and found nothing". That + # is MUTATION.md's third failure mode - the assertion ran, the sentence was + # true, and the data could not reach the failure it names. A counter can + # only be zero for the one reason. + _n1_calls = [] + _orig_eval = c.site_sign_wd.evaluate + + def _spy(*a, **kw): + _n1_calls.append(a) + return _orig_eval(*a, **kw) + + c.site_sign_wd.evaluate = _spy + await _drive(c, n=12, dt=0.2) + return c, _n1_calls + + +_c1s, _n1_site_calls = asyncio.run(_n1_sign()) +_inv1 = _c1s.inverters[0] +print(f" roster={_c1s.units} armed against {_inv1.sign_wd.against!r} " + f"at {_inv1.sign_wd.cfg.min_w:.0f} W / {_inv1.sign_wd.cfg.hold_s:.0f} s") +check("the single unit is graded against THE METER, at the SHIPPED thresholds " + "- §6.2's identity, and the whole compatibility promise of this card. " + "The unit's own power is not consulted, and `unit_sign_disagree_*` does " + "nothing on a one-inverter install", + _inv1.sign_wd.against == AGAINST_METER + and (_inv1.sign_wd.cfg.min_w, _inv1.sign_wd.cfg.hold_s) == (300.0, 1.0)) +check("...it trips, on the same evidence and through the same UNIT `failsafe` " + "claim as before, commanding 0 W", + _inv1.sign_wd.tripped and _c1s.target == 0.0 + and "failsafe" in _c1s.reason and "sign_site" not in _c1s.reason) +check("...the reason sentence still says 'a meter reading', word for word - it " + "is what an operator reads, what the action log carries and what is " + "persisted, so it is graded on the literal", + "against a meter reading" in _inv1.sign_wd.reason) +check("...the latch is under the same namespaced key with the same flat " + "rollback mirror, and SAFETY-30's site pair is untouched at its default. " + "A one-inverter install's state file is byte-identical to what it was", + _c1s.store.data.get(TRIPPED_KEY) is True + and _c1s.store.data.get("sign_tripped") is True + and _c1s.store.data.get("site_sign_tripped") is False + and _c1s.store.data.get("site_sign_reason") == "") +print(f" site detector evaluated {len(_n1_site_calls)} times over " + f"12 cycles") +check("⚠️ ...AND THE SITE DETECTOR IS NEVER ARMED. Not 'did not trip', and not " + "'its window stayed closed' - NEVER CALLED, counted on the object over " + "twelve cycles. That is what says the second detector is ABSENT at a " + "roster of one rather than merely quiet, and arming it would be two " + "latches and two banners for one fault with a false-positive record " + "measured on ONE comparison spent twice", + _n1_site_calls == [] + and not _c1s.site_sign_wd.tripped + and _c1s.site_sign_wd.disagree_since is None) +check("...the banner is the sentence that shipped", + _c1s.status_payload()["banner"] + == "FAILSAFE — the command opposed the meter. Inverter held at 0 W.") +check("...the status page carries ONE sign row, with no unit tag and no site " + "row - an extra row for an identically-duplicated question is a page " + "advertising two guards where the install has one", + len([ck for ck in _c1s.checks() + if "sign watchdog" in ck["text"] or "SIGN-DISAGREEMENT" in ck["text"] + ]) == 1 + and not any("site sign" in ck["text"].lower() for ck in _c1s.checks())) +check("...and no `sign_site` claim can be filed at a roster of one, so the " + "arbitration is byte-identical to what shipped", + "sign_site" not in _c1s.reason) + print() if fails: print(f"{len(fails)} FAILED: {', '.join(fails)}")