adminandClaude Opus 5.5
e0b578756f
Part 6 checked the quoted reason alone, so title + reason + tails logged as 4+ sentences. It now also checks the whole line (at most 3 sentences, none over 30 words); 25 lines failed and are rewritten (one-sentence wrappers, two-sentence circulation and peak reasons, STALE_READING's fixed limits on a DEBUG line). The clamp lines for one inverter are pinned at 30 words with OUTPUT_CLAMP and the numbers kept. Home Assistant's bare '400: Bad Request' while it loads becomes one plain line; still WARNING, throttled, a failed write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
353 lines
19 KiB
Python
353 lines
19 KiB
Python
"""SAFETY-31 - the power two inverters move through each other, which the meter cannot see.
|
|
|
|
Inverter A charging at 3 kW while inverter B discharges at 3 kW nets ~0 W at
|
|
P1. The site looks idle. In reality 3 kW is going round a loop through two
|
|
conversion stages continuously, for no benefit, and the wear on both packs is
|
|
real even though every instrument in this add-on says nothing is happening.
|
|
|
|
WHY NOTHING EXISTING SEES IT, AND IT IS STRUCTURAL RATHER THAN AN OVERSIGHT
|
|
==========================================================================
|
|
Every consumer of the site measurement is a consumer of the NET. SAFETY-02's
|
|
sign-disagreement watchdog gates on `abs(grid_w) > cfg.min_w` (300 W by
|
|
default) - with the grid at ~0 that gate never opens, on any cycle, for any
|
|
duration. The peak guard bills the quarter-hour average AT THE GRID
|
|
CONNECTION. The capacity tariff bills grid exchange. `freeze.looks_frozen()`
|
|
grades P1 against the site rail. There is one meter, it reads the sum, and
|
|
circulating power is exactly the component of per-unit power that the sum
|
|
DESTROYS. So this cannot be fixed by tuning a threshold on any existing
|
|
detector; it needs a quantity none of them are computing.
|
|
|
|
THE QUANTITY
|
|
============
|
|
circulating_w = sum(|B_i|) - |sum(B_i)|
|
|
|
|
Zero when every unit pulls the same way - then sum(|B_i|) == |sum(B_i)| by the
|
|
triangle equality. `2 * min(|B_a|, |B_b|)` at N=2 when they oppose, which is
|
|
twice the power going round the loop and is the right scale: that is what both
|
|
conversion stages are processing between them. It is non-negative always, it is
|
|
zero at N<=1 identically, and it needs no threshold to be interpretable - the
|
|
number IS the phenomenon in watts.
|
|
|
|
⚠️ IT IS COMPUTED FROM MEASURED PER-INVERTER POWER, AND THAT IS THE WHOLE CARD.
|
|
=============================================================================
|
|
The obvious cheap version reads the allocation record - the shares the
|
|
allocator just handed out - and reports the same formula over those. It would
|
|
be simpler, it would never see a `None`, and it would be worth NOTHING, for two
|
|
independent reasons.
|
|
|
|
The first is that it cannot fire. ALLOC-01 shipped the sign-coherence gate as
|
|
its AC 7 (`allocator.REFUSE_SIGN`, graded by `test_alloc.py` part 6): an
|
|
allocation whose shares carry two signs is REFUSED, both units named, no shares
|
|
returned. So a sensor over the allocation record is a sensor over a set the
|
|
allocator has already guaranteed is sign-coherent. It reads zero by
|
|
construction, forever, and a check written against it passes forever.
|
|
|
|
The second is the one MUTATION.md names as the third way a check lies. A sensor
|
|
derived from the allocation record can only ever confirm THE ALLOCATOR AGREES
|
|
WITH ITSELF. Every fault in this card's list is a divergence between what was
|
|
commanded and what the hardware is doing:
|
|
|
|
* a unit ignoring or lagging its share;
|
|
* a stale share still being executed by one inverter;
|
|
* SAFETY-25's deferred loop floor, or a maintenance drain, pinning one unit
|
|
while the loop drives the other;
|
|
* an inverter doing something of its own accord - and THIS HARDWARE HAS DONE
|
|
THAT TWICE. On 2026-09-02 the comms subsystem failed and the power stage
|
|
went on executing its last instruction for 6 h 25 m (HW-01). Through all of
|
|
it the allocation record was impeccable.
|
|
|
|
In every one of those the command side is coherent and the MEASUREMENT side is
|
|
not. So the input is `Inverter.batt` - this unit's own power, read from its own
|
|
`batt_entity`, which on the live system is holding register 0x0518, the
|
|
inverter's own AC power. DEV-02 made that per-instance; before it there was one
|
|
`batt` scalar on the Controller and this sensor could not have been written at
|
|
all.
|
|
|
|
⚠️ AND THIS IS THE MEASUREMENT HALF OF A PAIR. ALLOC-01's gate is the COMMAND
|
|
half: the allocator can no longer COMMAND circulating power. This module is the
|
|
detection half: circulating power the allocator never commanded. Neither one
|
|
replaces the other and neither is redundant - the gate refuses an allocation
|
|
before it is written, and cannot see a millisecond past that; this reads the
|
|
hardware and does not care what was written. Do not "unify" them.
|
|
|
|
THE TRIP, AND WHY A THRESHOLD ALONE IS A FALSE-ALARM GENERATOR
|
|
==============================================================
|
|
A momentary opposition is LEGITIMATE. Writes to the inverters are serialised
|
|
(one `hass.call()` at a time - see `Inverter`'s docstring on what a hung write
|
|
to A costs B), so on a cycle where the site target reverses, one unit receives
|
|
its new command up to a whole control cycle before its sibling. Both then slew
|
|
through zero, with a real ramp rate and a real first-order lag, and BETWEEN THE
|
|
TWO ZERO CROSSINGS THE UNITS GENUINELY OPPOSE. Nothing is wrong; the site is
|
|
turning around.
|
|
|
|
This project's doctrine on crying wolf is explicit (SAFETY-15, SAFETY-19) and
|
|
this is exactly the shape that generates it. So the trip is a threshold AND a
|
|
sustained window, and BOTH numbers were measured on the TEST-19 two-master rig
|
|
rather than chosen - see `CIRC_OPTION_DEFAULTS` for the measurements and
|
|
`DECISIONS/SAFETY-31-circulating-power.md` for the runs.
|
|
|
|
⚠️ THE WINDOW ACCUMULATES, WHICH IS WHY THIS IS NOT SAFETY-02's PROBLEM AGAIN.
|
|
The reason the sign watchdog is blind to the broadcast fight is that the
|
|
disagreement FLIPS SIGN on alternate ticks, so `disagree_since` is cleared
|
|
between every pair of them and the 180 s hold never accumulates (allocator.py's
|
|
docstring records the measurement). Circulating power cannot do that: it is a
|
|
MAGNITUDE, non-negative by construction, and a sustained loop holds it high on
|
|
every cycle rather than on every other one. An oscillation that genuinely
|
|
alternated between opposing and agreeing is not a circulating loop and should
|
|
not be reported as one.
|
|
"""
|
|
|
|
from dataclasses import dataclass
|
|
from datetime import datetime
|
|
|
|
# CFG-03. THE ONE IN-CODE COPY OF THESE DEFAULTS. `config.yaml`'s `options:`
|
|
# block is the source of truth an operator edits and the Supervisor reads;
|
|
# `schema:` holds the bounds. This dict exists for the reason
|
|
# `maintenance.MAINT_OPTION_DEFAULTS` does - the Dockerfile does not put
|
|
# config.yaml in the image, so the running add-on cannot read it - and
|
|
# `test_watchdog.py` pins the two together by READING config.yaml, so a drift
|
|
# goes red in CI rather than shipping.
|
|
CIRC_OPTION_DEFAULTS = {
|
|
# --- THE THRESHOLD, AND IT IS READ OFF A CURVE ------------------------
|
|
# ⚠️ MEASURED ON THE TEST-19 TWO-MASTER RIG, NOT CHOSEN. 924 closed-loop
|
|
# runs in which NOTHING is faulty - every unit executes the share the
|
|
# allocator gave it - over both families of stimulus that can produce
|
|
# legitimate opposition: single reversals (7 sizes x 5 write staggers) and a
|
|
# flapping load (7 periods x 4 amplitudes), each swept across the WHOLE
|
|
# range config.yaml accepts for the two settings that decide how fast the
|
|
# site can turn around, `gain` float(0.05,1.0) and `slew_w` int(50,5000).
|
|
# The worst CONTINUOUS opposition, as a function of the threshold:
|
|
#
|
|
# 100 W -> 505.0 s 300 W -> 10.0 s 700 W -> 5.0 s
|
|
# 200 W -> 445.0 s 350 W -> 10.0 s 1000 W -> 5.0 s
|
|
# 250 W -> 40.0 s 400 W -> 5.0 s 1700 W -> 0.0 s
|
|
#
|
|
# ⚠️ SAFETY-39 MOVED THAT SCHEMA FLOOR FROM 50 TO 0, AND THE CAMPAIGN ABOVE
|
|
# WAS NOT RE-RUN. It does not have to be, and here is the argument rather
|
|
# than the assurance. What these windows are about is how long the command
|
|
# takes to CROSS ITS RAIL - the swept worst case is the SLOWEST crossing,
|
|
# which is `slew_w` at its floor, 5000/50 = 100 cycles. `slew_w: 0` means
|
|
# "follow the rail" and resolves to `SLEW_RAIL_FRAC * max_w`, which is
|
|
# exactly 9 cycles at EVERY rail by construction. So the new value can never
|
|
# be the slowest crossing at any `max_w`, and it is strictly inside the
|
|
# envelope this campaign measured. An explicit number still spans 50..5000
|
|
# and is unchanged. If `SLEW_RAIL_FRAC` is ever loosened past 1/100, this
|
|
# paragraph stops being true and the campaign has to be re-run.
|
|
#
|
|
# THE KNEE IS AT 400 W and it is sharp. Below ~250 W a flapping load holds
|
|
# the sensor above the threshold for MINUTES at a time - 505 s at 100 W -
|
|
# and no window shorter than the 900 s billing quarter could tell that from
|
|
# a fault. At 400 W and above the worst legitimate episode collapses to 5.0
|
|
# s, ONE control cycle, and stays there. So this number is not a sensitivity
|
|
# preference; it is the edge of the regime in which the guard is
|
|
# answerable at all.
|
|
#
|
|
# 700 W is 1.75x the knee, and the margin is bought for one named reason:
|
|
# INVERTER B DOES NOT PHYSICALLY EXIST. Its ramp and lag in the rig are
|
|
# inverter A's, COPIED, carrying `measured=False` - SPIKE-04 measures the
|
|
# real ones. A slower or faster second unit moves the transient's shape, and
|
|
# the headroom above the knee is what absorbs that without re-opening this
|
|
# card. Revisit it when SPIKE-04 lands, not before.
|
|
#
|
|
# What it costs: a circulating loop smaller than 350 W per unit is not
|
|
# reported. That is deliberate - at that scale the round-trip loss is a few
|
|
# watts and the wear is proportionate to it.
|
|
"circulation_w": 700,
|
|
|
|
# --- THE WINDOW -------------------------------------------------------
|
|
# Two lower bounds, one upper, and the chosen value clears all three.
|
|
#
|
|
# LOWER, from the rig: 5.0 s. The worst continuous legitimate opposition
|
|
# above 700 W across all 924 runs is exactly one control cycle - a
|
|
# reversing site whose second write lands a cycle after the first, which
|
|
# is what the serialised write path does on every reversal.
|
|
#
|
|
# LOWER, from the firmware: `inverter.LINK_STALE_S`, 30 s. A unit still
|
|
# executing a stale share for less than that is a LINK fault, and the
|
|
# add-on already has an answer for it - liveness drops the unit and the
|
|
# survivors carry its share, with the record saying so. Firing inside that
|
|
# window would raise a second, louder alarm about a state that is already
|
|
# being handled, which is the SAFETY-15/SAFETY-19 failure exactly.
|
|
#
|
|
# UPPER, from the bill: 900 s, the quarter that sets the capacity tariff.
|
|
# A circulating loop must be caught and acted on well inside one.
|
|
#
|
|
# 75 s: fifteen control cycles - 15x the worst legitimate episode this rig
|
|
# can produce, 2.5x the firmware's own veto window, and 12x inside the
|
|
# quarter. ⚠️ IT IS DELIBERATELY NOT 60 OR 120. A round number here would be
|
|
# a guess wearing a measurement's clothes, and the three bounds above are
|
|
# what the number has to answer to.
|
|
"circulation_hold_s": 75,
|
|
}
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class CirculationConfig:
|
|
"""Defaults, not constants - config.yaml carries both. See CFG-03 above."""
|
|
|
|
min_w: float = float(CIRC_OPTION_DEFAULTS["circulation_w"])
|
|
hold_s: float = float(CIRC_OPTION_DEFAULTS["circulation_hold_s"])
|
|
|
|
|
|
def circulating_w(powers):
|
|
"""`sum(|B_i|) - |sum(B_i)|` over MEASURED per-inverter power, in W.
|
|
|
|
`powers` one measured power per inverter, in roster order, + = discharge.
|
|
A unit whose reading did not arrive is `None`.
|
|
|
|
Returns a non-negative float, or `None` when the answer is unknown.
|
|
|
|
⚠️ N<=1 IS IDENTICALLY ZERO AND RETURNS ZERO EVEN FOR A MISSING READING
|
|
(AC 5). With one term `sum(|B|) - |sum(B)| == |B| - |B| == 0` for EVERY
|
|
value of B, including a value nobody read. Returning `None` there would be
|
|
reporting an unknown where the algebra has no room for one, and it would put
|
|
a shipped one-inverter install permanently in the "cannot tell" state on a
|
|
quantity that cannot be anything but zero. Nothing changes at the shipped
|
|
roster; that is the compatibility promise this card makes.
|
|
|
|
⚠️ AND AT N>=2 A MISSING READING IS `None`, NOT ZERO, WHICH IS THE OPPOSITE
|
|
DIRECTION AND FOR A REASON. Zero would be this sensor reporting HEALTH from
|
|
an absence - the precise shape of the 2026-09-02 outage, where a subsystem
|
|
stopped answering and its silence read as calm for 6 h 25 m. An unknown
|
|
circulating figure is an unknown, it CLEARS the trip's window rather than
|
|
accumulating it (see `CirculationWatchdog.evaluate`), and the site is
|
|
already stopped for a missing per-unit reading anyway - `inputs_ok` requires
|
|
every unit's `batt`, and §11.1 rule 1 commands 0 W without consulting this.
|
|
"""
|
|
vals = list(powers)
|
|
if len(vals) <= 1:
|
|
return 0.0
|
|
if any(v is None for v in vals):
|
|
return None
|
|
vals = [float(v) for v in vals]
|
|
# ⚠️ max(0.0, ...) IS DEFENSIVE, AND IT IS AN EQUIVALENT MUTANT - SAID HERE
|
|
# RATHER THAN LEFT FOR THE NEXT MUTATION SWEEP TO REDISCOVER. The triangle
|
|
# inequality makes this non-negative mathematically, and in floating point
|
|
# it is non-negative too: for a same-sign roster `abs(v)` is the list itself
|
|
# up to one sign, so the two sums accumulate THE SAME MAGNITUDES IN THE SAME
|
|
# ORDER and are bit-identical; for a mixed-sign roster the two differ by
|
|
# 2*min(...), which is enormous beside any rounding. Searched, not assumed -
|
|
# 700 000 random rosters of 2 to 12 units across 10 orders of magnitude
|
|
# produced no negative result.
|
|
#
|
|
# So removing this line changes no observable and no check here would go
|
|
# red. It stays because the cost is one call and the failure it would
|
|
# prevent - a power sensor publishing -1e-16 W, which is the kind of value
|
|
# somebody eventually writes a `< 0` branch against - is out of proportion
|
|
# to it. Do not report its survival as a gap in the checks.
|
|
return max(0.0, sum(abs(v) for v in vals) - abs(sum(vals)))
|
|
|
|
|
|
def opposing(powers, min_w: float = 0.0):
|
|
"""The units pulling each way right now, as `(discharging, charging)` names.
|
|
|
|
Indices, not names - the caller owns the roster order. Used for the operator
|
|
sentence: "which two inverters are pushing against each other" is the first
|
|
thing somebody woken by this alarm needs, and reconstructing it from the
|
|
number alone is impossible.
|
|
|
|
`min_w` ignores units below a magnitude, so a unit resting at 3 W of
|
|
standby draw is not named as a participant in a 3 kW loop.
|
|
"""
|
|
pos = [i for i, v in enumerate(powers)
|
|
if v is not None and float(v) > float(min_w)]
|
|
neg = [i for i, v in enumerate(powers)
|
|
if v is not None and float(v) < -float(min_w)]
|
|
return pos, neg
|
|
|
|
|
|
class CirculationWatchdog:
|
|
"""ONE PER SITE, evaluated once per control cycle.
|
|
|
|
⚠️ ONE PER SITE, WHERE SAFETY-02's IS ONE PER ACTUATOR, AND THE ASYMMETRY IS
|
|
THE WHOLE ATTRIBUTION ARGUMENT. `SignWatchdog` concludes "THE UNIT I AM
|
|
DRIVING is pushing power the wrong way" - evidence about one actuator, so
|
|
the remedy is that actuator (arbiter.CLAIM_SCOPE calls `failsafe` a UNIT
|
|
claim for exactly this reason). Circulating power is not a property any one
|
|
inverter has. It is a property of a PAIR, it is defined by a sum across the
|
|
roster, and there is no fact of the matter about which of the two is the
|
|
offender: a unit ignoring its share and a sibling faithfully executing a
|
|
share computed to sum with it produce identical readings. So the quantity is
|
|
site-level, the watchdog is site-level, and the remedy below is site-level.
|
|
|
|
Latches, for `SignWatchdog`'s reason: the trip removes its own evidence. The
|
|
moment the site stops commanding, the loop it detected either persists (a
|
|
wedged unit, now visible at the meter) or vanishes (the add-on was one of
|
|
the two participants) - and in the second case a non-latching detector would
|
|
de-assert and start the whole thing again, three minutes at a time, forever.
|
|
"""
|
|
|
|
def __init__(self, cfg: CirculationConfig | None = None, *,
|
|
tripped: bool = False, reason: str = ""):
|
|
self.cfg = cfg or CirculationConfig()
|
|
self.over_since: datetime | None = None
|
|
# ⚠️ RESTORED FROM THE STORE, NOT DEFAULTED - SAFETY-34, and it is the
|
|
# reason this card exists in the shape it does. `persistent_notification`
|
|
# is IN MEMORY: Home Assistant restarts and the card is gone, with the
|
|
# fault still running. That is precisely how the 2026-09-02 outage went
|
|
# unseen for 6 h 25 m while the detector had been correct for hours. The
|
|
# latch lives in `/data/state.json` - the add-on's only durable volume -
|
|
# and `main.py` RE-RAISES the notification from it on every start.
|
|
self.tripped = bool(tripped)
|
|
self.reason = reason if self.tripped else ""
|
|
# The worst figure seen during the current episode, for the operator
|
|
# sentence. Not persisted: it is colour on a reason string that IS.
|
|
self.peak_w = 0.0
|
|
|
|
def evaluate(self, circ_w, now: datetime) -> bool:
|
|
"""Advance the detector one cycle. Returns True while tripped.
|
|
|
|
`circ_w` this cycle's `circulating_w()` over MEASURED per-unit power,
|
|
or None when a reading is missing. None CLEARS the window
|
|
rather than accumulating it - see the module docstring, and
|
|
`SignWatchdog.evaluate` on the identical decision for a stale
|
|
meter. A frozen figure held against a moving one manufactures
|
|
a fault in about a window's worth of cycles.
|
|
`now` the caller's clock, so this is testable and replayable.
|
|
"""
|
|
if self.tripped:
|
|
# Latched. Do not re-evaluate: the trip removed the evidence.
|
|
return True
|
|
|
|
if circ_w is None or float(circ_w) <= self.cfg.min_w:
|
|
self.over_since = None
|
|
self.peak_w = 0.0
|
|
return False
|
|
|
|
circ_w = float(circ_w)
|
|
self.peak_w = max(self.peak_w, circ_w)
|
|
if self.over_since is None:
|
|
self.over_since = now
|
|
return False
|
|
|
|
held = (now - self.over_since).total_seconds()
|
|
if held < self.cfg.hold_s:
|
|
return False
|
|
|
|
self.tripped = True
|
|
# UX-10: plain sentences, every figure kept (test_messages.py part 6).
|
|
self.reason = (
|
|
f"CIRCULATING_POWER: {circ_w:.0f} W measured going round between "
|
|
f"the inverters for {held:.0f} s, peaking at {self.peak_w:.0f} W, "
|
|
f"over the limits of {self.cfg.min_w:.0f} W and "
|
|
f"{self.cfg.hold_s:.0f} s. The meter cannot see it, because it "
|
|
f"nets to zero there")
|
|
return True
|
|
|
|
def held_s(self, now: datetime) -> float:
|
|
"""How long the current opposition has stood, for the status page.
|
|
|
|
Takes the clock rather than reading one, for `SignWatchdog.held_s`'s
|
|
reason: every observable on this object is a pure function of what it
|
|
was handed, and an unclocked function is one nothing can test.
|
|
"""
|
|
if self.over_since is None:
|
|
return 0.0
|
|
return (now - self.over_since).total_seconds()
|
|
|
|
def reset(self) -> None:
|
|
self.over_since = None
|
|
self.tripped = False
|
|
self.reason = ""
|
|
self.peak_w = 0.0
|