"""SAFETY-31 - the power two inverters move through each other, which the meter cannot see. Inverter A charging at 3 kW while inverter B discharges at 3 kW nets ~0 W at P1. The site looks idle. In reality 3 kW is going round a loop through two conversion stages continuously, for no benefit, and the wear on both packs is real even though every instrument in this add-on says nothing is happening. WHY NOTHING EXISTING SEES IT, AND IT IS STRUCTURAL RATHER THAN AN OVERSIGHT ========================================================================== Every consumer of the site measurement is a consumer of the NET. SAFETY-02's sign-disagreement watchdog gates on `abs(grid_w) > cfg.min_w` (300 W by default) - with the grid at ~0 that gate never opens, on any cycle, for any duration. The peak guard bills the quarter-hour average AT THE GRID CONNECTION. The capacity tariff bills grid exchange. `freeze.looks_frozen()` grades P1 against the site rail. There is one meter, it reads the sum, and circulating power is exactly the component of per-unit power that the sum DESTROYS. So this cannot be fixed by tuning a threshold on any existing detector; it needs a quantity none of them are computing. THE QUANTITY ============ circulating_w = sum(|B_i|) - |sum(B_i)| Zero when every unit pulls the same way - then sum(|B_i|) == |sum(B_i)| by the triangle equality. `2 * min(|B_a|, |B_b|)` at N=2 when they oppose, which is twice the power going round the loop and is the right scale: that is what both conversion stages are processing between them. It is non-negative always, it is zero at N<=1 identically, and it needs no threshold to be interpretable - the number IS the phenomenon in watts. ⚠️ IT IS COMPUTED FROM MEASURED PER-INVERTER POWER, AND THAT IS THE WHOLE CARD. ============================================================================= The obvious cheap version reads the allocation record - the shares the allocator just handed out - and reports the same formula over those. It would be simpler, it would never see a `None`, and it would be worth NOTHING, for two independent reasons. The first is that it cannot fire. ALLOC-01 shipped the sign-coherence gate as its AC 7 (`allocator.REFUSE_SIGN`, graded by `test_alloc.py` part 6): an allocation whose shares carry two signs is REFUSED, both units named, no shares returned. So a sensor over the allocation record is a sensor over a set the allocator has already guaranteed is sign-coherent. It reads zero by construction, forever, and a check written against it passes forever. The second is the one MUTATION.md names as the third way a check lies. A sensor derived from the allocation record can only ever confirm THE ALLOCATOR AGREES WITH ITSELF. Every fault in this card's list is a divergence between what was commanded and what the hardware is doing: * a unit ignoring or lagging its share; * a stale share still being executed by one inverter; * SAFETY-25's deferred loop floor, or a maintenance drain, pinning one unit while the loop drives the other; * an inverter doing something of its own accord - and THIS HARDWARE HAS DONE THAT TWICE. On 2026-09-02 the comms subsystem failed and the power stage went on executing its last instruction for 6 h 25 m (HW-01). Through all of it the allocation record was impeccable. In every one of those the command side is coherent and the MEASUREMENT side is not. So the input is `Inverter.batt` - this unit's own power, read from its own `batt_entity`, which on the live system is holding register 0x0518, the inverter's own AC power. DEV-02 made that per-instance; before it there was one `batt` scalar on the Controller and this sensor could not have been written at all. ⚠️ AND THIS IS THE MEASUREMENT HALF OF A PAIR. ALLOC-01's gate is the COMMAND half: the allocator can no longer COMMAND circulating power. This module is the detection half: circulating power the allocator never commanded. Neither one replaces the other and neither is redundant - the gate refuses an allocation before it is written, and cannot see a millisecond past that; this reads the hardware and does not care what was written. Do not "unify" them. THE TRIP, AND WHY A THRESHOLD ALONE IS A FALSE-ALARM GENERATOR ============================================================== A momentary opposition is LEGITIMATE. Writes to the inverters are serialised (one `hass.call()` at a time - see `Inverter`'s docstring on what a hung write to A costs B), so on a cycle where the site target reverses, one unit receives its new command up to a whole control cycle before its sibling. Both then slew through zero, with a real ramp rate and a real first-order lag, and BETWEEN THE TWO ZERO CROSSINGS THE UNITS GENUINELY OPPOSE. Nothing is wrong; the site is turning around. This project's doctrine on crying wolf is explicit (SAFETY-15, SAFETY-19) and this is exactly the shape that generates it. So the trip is a threshold AND a sustained window, and BOTH numbers were measured on the TEST-19 two-master rig rather than chosen - see `CIRC_OPTION_DEFAULTS` for the measurements and `DECISIONS/SAFETY-31-circulating-power.md` for the runs. ⚠️ THE WINDOW ACCUMULATES, WHICH IS WHY THIS IS NOT SAFETY-02's PROBLEM AGAIN. The reason the sign watchdog is blind to the broadcast fight is that the disagreement FLIPS SIGN on alternate ticks, so `disagree_since` is cleared between every pair of them and the 180 s hold never accumulates (allocator.py's docstring records the measurement). Circulating power cannot do that: it is a MAGNITUDE, non-negative by construction, and a sustained loop holds it high on every cycle rather than on every other one. An oscillation that genuinely alternated between opposing and agreeing is not a circulating loop and should not be reported as one. """ from dataclasses import dataclass from datetime import datetime # CFG-03. THE ONE IN-CODE COPY OF THESE DEFAULTS. `config.yaml`'s `options:` # block is the source of truth an operator edits and the Supervisor reads; # `schema:` holds the bounds. This dict exists for the reason # `maintenance.MAINT_OPTION_DEFAULTS` does - the Dockerfile does not put # config.yaml in the image, so the running add-on cannot read it - and # `test_watchdog.py` pins the two together by READING config.yaml, so a drift # goes red in CI rather than shipping. CIRC_OPTION_DEFAULTS = { # --- THE THRESHOLD, AND IT IS READ OFF A CURVE ------------------------ # ⚠️ MEASURED ON THE TEST-19 TWO-MASTER RIG, NOT CHOSEN. 924 closed-loop # runs in which NOTHING is faulty - every unit executes the share the # allocator gave it - over both families of stimulus that can produce # legitimate opposition: single reversals (7 sizes x 5 write staggers) and a # flapping load (7 periods x 4 amplitudes), each swept across the WHOLE # range config.yaml accepts for the two settings that decide how fast the # site can turn around, `gain` float(0.05,1.0) and `slew_w` int(50,5000). # The worst CONTINUOUS opposition, as a function of the threshold: # # 100 W -> 505.0 s 300 W -> 10.0 s 700 W -> 5.0 s # 200 W -> 445.0 s 350 W -> 10.0 s 1000 W -> 5.0 s # 250 W -> 40.0 s 400 W -> 5.0 s 1700 W -> 0.0 s # # ⚠️ SAFETY-39 MOVED THAT SCHEMA FLOOR FROM 50 TO 0, AND THE CAMPAIGN ABOVE # WAS NOT RE-RUN. It does not have to be, and here is the argument rather # than the assurance. What these windows are about is how long the command # takes to CROSS ITS RAIL - the swept worst case is the SLOWEST crossing, # which is `slew_w` at its floor, 5000/50 = 100 cycles. `slew_w: 0` means # "follow the rail" and resolves to `SLEW_RAIL_FRAC * max_w`, which is # exactly 9 cycles at EVERY rail by construction. So the new value can never # be the slowest crossing at any `max_w`, and it is strictly inside the # envelope this campaign measured. An explicit number still spans 50..5000 # and is unchanged. If `SLEW_RAIL_FRAC` is ever loosened past 1/100, this # paragraph stops being true and the campaign has to be re-run. # # THE KNEE IS AT 400 W and it is sharp. Below ~250 W a flapping load holds # the sensor above the threshold for MINUTES at a time - 505 s at 100 W - # and no window shorter than the 900 s billing quarter could tell that from # a fault. At 400 W and above the worst legitimate episode collapses to 5.0 # s, ONE control cycle, and stays there. So this number is not a sensitivity # preference; it is the edge of the regime in which the guard is # answerable at all. # # 700 W is 1.75x the knee, and the margin is bought for one named reason: # INVERTER B DOES NOT PHYSICALLY EXIST. Its ramp and lag in the rig are # inverter A's, COPIED, carrying `measured=False` - SPIKE-04 measures the # real ones. A slower or faster second unit moves the transient's shape, and # the headroom above the knee is what absorbs that without re-opening this # card. Revisit it when SPIKE-04 lands, not before. # # What it costs: a circulating loop smaller than 350 W per unit is not # reported. That is deliberate - at that scale the round-trip loss is a few # watts and the wear is proportionate to it. "circulation_w": 700, # --- THE WINDOW ------------------------------------------------------- # Two lower bounds, one upper, and the chosen value clears all three. # # LOWER, from the rig: 5.0 s. The worst continuous legitimate opposition # above 700 W across all 924 runs is exactly one control cycle - a # reversing site whose second write lands a cycle after the first, which # is what the serialised write path does on every reversal. # # LOWER, from the firmware: `inverter.LINK_STALE_S`, 30 s. A unit still # executing a stale share for less than that is a LINK fault, and the # add-on already has an answer for it - liveness drops the unit and the # survivors carry its share, with the record saying so. Firing inside that # window would raise a second, louder alarm about a state that is already # being handled, which is the SAFETY-15/SAFETY-19 failure exactly. # # UPPER, from the bill: 900 s, the quarter that sets the capacity tariff. # A circulating loop must be caught and acted on well inside one. # # 75 s: fifteen control cycles - 15x the worst legitimate episode this rig # can produce, 2.5x the firmware's own veto window, and 12x inside the # quarter. ⚠️ IT IS DELIBERATELY NOT 60 OR 120. A round number here would be # a guess wearing a measurement's clothes, and the three bounds above are # what the number has to answer to. "circulation_hold_s": 75, } @dataclass(frozen=True) class CirculationConfig: """Defaults, not constants - config.yaml carries both. See CFG-03 above.""" min_w: float = float(CIRC_OPTION_DEFAULTS["circulation_w"]) hold_s: float = float(CIRC_OPTION_DEFAULTS["circulation_hold_s"]) def circulating_w(powers): """`sum(|B_i|) - |sum(B_i)|` over MEASURED per-inverter power, in W. `powers` one measured power per inverter, in roster order, + = discharge. A unit whose reading did not arrive is `None`. Returns a non-negative float, or `None` when the answer is unknown. ⚠️ N<=1 IS IDENTICALLY ZERO AND RETURNS ZERO EVEN FOR A MISSING READING (AC 5). With one term `sum(|B|) - |sum(B)| == |B| - |B| == 0` for EVERY value of B, including a value nobody read. Returning `None` there would be reporting an unknown where the algebra has no room for one, and it would put a shipped one-inverter install permanently in the "cannot tell" state on a quantity that cannot be anything but zero. Nothing changes at the shipped roster; that is the compatibility promise this card makes. ⚠️ AND AT N>=2 A MISSING READING IS `None`, NOT ZERO, WHICH IS THE OPPOSITE DIRECTION AND FOR A REASON. Zero would be this sensor reporting HEALTH from an absence - the precise shape of the 2026-09-02 outage, where a subsystem stopped answering and its silence read as calm for 6 h 25 m. An unknown circulating figure is an unknown, it CLEARS the trip's window rather than accumulating it (see `CirculationWatchdog.evaluate`), and the site is already stopped for a missing per-unit reading anyway - `inputs_ok` requires every unit's `batt`, and §11.1 rule 1 commands 0 W without consulting this. """ vals = list(powers) if len(vals) <= 1: return 0.0 if any(v is None for v in vals): return None vals = [float(v) for v in vals] # ⚠️ max(0.0, ...) IS DEFENSIVE, AND IT IS AN EQUIVALENT MUTANT - SAID HERE # RATHER THAN LEFT FOR THE NEXT MUTATION SWEEP TO REDISCOVER. The triangle # inequality makes this non-negative mathematically, and in floating point # it is non-negative too: for a same-sign roster `abs(v)` is the list itself # up to one sign, so the two sums accumulate THE SAME MAGNITUDES IN THE SAME # ORDER and are bit-identical; for a mixed-sign roster the two differ by # 2*min(...), which is enormous beside any rounding. Searched, not assumed - # 700 000 random rosters of 2 to 12 units across 10 orders of magnitude # produced no negative result. # # So removing this line changes no observable and no check here would go # red. It stays because the cost is one call and the failure it would # prevent - a power sensor publishing -1e-16 W, which is the kind of value # somebody eventually writes a `< 0` branch against - is out of proportion # to it. Do not report its survival as a gap in the checks. return max(0.0, sum(abs(v) for v in vals) - abs(sum(vals))) def opposing(powers, min_w: float = 0.0): """The units pulling each way right now, as `(discharging, charging)` names. Indices, not names - the caller owns the roster order. Used for the operator sentence: "which two inverters are pushing against each other" is the first thing somebody woken by this alarm needs, and reconstructing it from the number alone is impossible. `min_w` ignores units below a magnitude, so a unit resting at 3 W of standby draw is not named as a participant in a 3 kW loop. """ pos = [i for i, v in enumerate(powers) if v is not None and float(v) > float(min_w)] neg = [i for i, v in enumerate(powers) if v is not None and float(v) < -float(min_w)] return pos, neg class CirculationWatchdog: """ONE PER SITE, evaluated once per control cycle. ⚠️ ONE PER SITE, WHERE SAFETY-02's IS ONE PER ACTUATOR, AND THE ASYMMETRY IS THE WHOLE ATTRIBUTION ARGUMENT. `SignWatchdog` concludes "THE UNIT I AM DRIVING is pushing power the wrong way" - evidence about one actuator, so the remedy is that actuator (arbiter.CLAIM_SCOPE calls `failsafe` a UNIT claim for exactly this reason). Circulating power is not a property any one inverter has. It is a property of a PAIR, it is defined by a sum across the roster, and there is no fact of the matter about which of the two is the offender: a unit ignoring its share and a sibling faithfully executing a share computed to sum with it produce identical readings. So the quantity is site-level, the watchdog is site-level, and the remedy below is site-level. Latches, for `SignWatchdog`'s reason: the trip removes its own evidence. The moment the site stops commanding, the loop it detected either persists (a wedged unit, now visible at the meter) or vanishes (the add-on was one of the two participants) - and in the second case a non-latching detector would de-assert and start the whole thing again, three minutes at a time, forever. """ def __init__(self, cfg: CirculationConfig | None = None, *, tripped: bool = False, reason: str = ""): self.cfg = cfg or CirculationConfig() self.over_since: datetime | None = None # ⚠️ RESTORED FROM THE STORE, NOT DEFAULTED - SAFETY-34, and it is the # reason this card exists in the shape it does. `persistent_notification` # is IN MEMORY: Home Assistant restarts and the card is gone, with the # fault still running. That is precisely how the 2026-09-02 outage went # unseen for 6 h 25 m while the detector had been correct for hours. The # latch lives in `/data/state.json` - the add-on's only durable volume - # and `main.py` RE-RAISES the notification from it on every start. self.tripped = bool(tripped) self.reason = reason if self.tripped else "" # The worst figure seen during the current episode, for the operator # sentence. Not persisted: it is colour on a reason string that IS. self.peak_w = 0.0 def evaluate(self, circ_w, now: datetime) -> bool: """Advance the detector one cycle. Returns True while tripped. `circ_w` this cycle's `circulating_w()` over MEASURED per-unit power, or None when a reading is missing. None CLEARS the window rather than accumulating it - see the module docstring, and `SignWatchdog.evaluate` on the identical decision for a stale meter. A frozen figure held against a moving one manufactures a fault in about a window's worth of cycles. `now` the caller's clock, so this is testable and replayable. """ if self.tripped: # Latched. Do not re-evaluate: the trip removed the evidence. return True if circ_w is None or float(circ_w) <= self.cfg.min_w: self.over_since = None self.peak_w = 0.0 return False circ_w = float(circ_w) self.peak_w = max(self.peak_w, circ_w) if self.over_since is None: self.over_since = now return False held = (now - self.over_since).total_seconds() if held < self.cfg.hold_s: return False self.tripped = True # UX-10: plain sentences, every figure kept (test_messages.py part 6). self.reason = ( f"CIRCULATING_POWER: {circ_w:.0f} W measured going round between " f"the inverters for {held:.0f} s, peaking at {self.peak_w:.0f} W, " f"over the limits of {self.cfg.min_w:.0f} W and " f"{self.cfg.hold_s:.0f} s. The meter cannot see it, because it " f"nets to zero there") return True def held_s(self, now: datetime) -> float: """How long the current opposition has stood, for the status page. Takes the clock rather than reading one, for `SignWatchdog.held_s`'s reason: every observable on this object is a pure function of what it was handed, and an unclocked function is one nothing can test. """ if self.over_since is None: return 0.0 return (now - self.over_since).total_seconds() def reset(self) -> None: self.over_since = None self.tripped = False self.reason = "" self.peak_w = 0.0