glenn schrooyenandClaude Opus 5.5
29837dfaa7
- Every WARNING/ERROR line that reported a safety reason (direction, delivery, stuck meter, stale reading, peak guard, circulation, over-rating) now ends in the reason's code (messages.reason_code()). The reason itself, figures and limits included, is on a DEBUG line right after it and in the action log. - The reasons and state.json are unchanged, so an old state file loads and shows the same way: every saved reason opens with its code. - test_messages part 6 checks every WARNING+ line at 25 words as logged; nine non-safety families are listed with a reason and capped. The byte pin in test_watchdog now pins the meaning, in order. - The meter logs one WARNING per quiet episode, and INFO when back, so the red row's "the add-on log shows why" holds. The Why line past the hold names the missing reading. Action log: "25 engagements", "Shadow mode off: ...". No trigger, latch, threshold, log level, notification id, cadence, row colour or banner order changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LyR88rXETvwhTgx6yHE39p
701 lines
40 KiB
Python
701 lines
40 KiB
Python
"""SAFETY-02 - the sign-disagreement watchdog.
|
|
|
|
WHAT IT CATCHES, in one sentence: a grid-following loop that is pushing power in
|
|
the wrong direction, which every other guard in this system is blind to.
|
|
|
|
The stale-input watchdog (SAFETY-01) asks "is the meter still talking?". The
|
|
saturation freeze asks "can the inverter follow?". The output clamp asks "is the
|
|
number too big?". A loop can pass all three and still be charging the battery
|
|
out of the grid while the house imports - the meter is fresh, the inverter is
|
|
tracking beautifully, and the magnitude is inside the rating. The only thing
|
|
that separates that from healthy operation is the SIGN.
|
|
|
|
⚠️ THIS IS NOT A HYPOTHETICAL. On 2026-08-20 the vendor controller (Illusmart)
|
|
did exactly this four times in one day (NOTES.md §4.7). Both captures of that
|
|
bus are committed - `sim/scenarios/bus-2026-08-20.log.gz` and the healthy
|
|
`bus-2026-08-23.log.gz` in the infra repo - and `sim/replay_sign.py` drives this
|
|
module over both of them. Every number in this file was measured on those two
|
|
days, not chosen:
|
|
|
|
day samples gated in sign agreement
|
|
2026-08-23 4 212 / 17 280 93.4 % (healthy)
|
|
2026-08-20 7 182 / 17 280 56.4 % (the whole fault day)
|
|
cluster A 15:26-16:43 0.3 %
|
|
cluster B 17:11-18:28 0.9 %
|
|
cluster C 20:01-20:31 10.5 %
|
|
cluster D 20:55-21:11 0.0 %
|
|
|
|
⚠️ THOSE AGREEMENT FIGURES ARE THE WEAKEST EVIDENCE ON THIS PAGE, and an earlier
|
|
version of this comment leaned on them as though they were the strongest. The
|
|
clusters are found by looking for sustained opposed samples, so measuring "how
|
|
opposed is this cluster" over them is close to a tautology - and "the healthy day
|
|
contains no such cluster" is the same sentence as "the healthy day produces no
|
|
trip", not independent corroboration of it.
|
|
|
|
What actually carries the argument is two things neither the detector nor the
|
|
clustering assumes:
|
|
|
|
1. The RUN-LENGTH DISTRIBUTION. The longest unbroken opposed run on the healthy
|
|
day is 30 s. On the fault day there are 18 runs over 180 s and the longest is
|
|
1750 s. Nothing about that gap was chosen - it is what the two days measure.
|
|
2. The clusters land where a human independently wrote down onsets. NOTES.md
|
|
§4.7 records 15:01, 16:56, 19:57 and 20:23 from watching it happen, before
|
|
any of this existed, and cluster D coincides with the separately documented
|
|
inverter-off runaway of §4.7 item 2.
|
|
|
|
Be precise about (2), because the convenient overstatement is available and wrong:
|
|
the clustering does NOT reproduce four episodes. It yields THREE clusters over the
|
|
four documented onsets - C spans both 19:57 and 20:23 - plus a fourth for §4.7
|
|
item 2, which is not one of the four. Every trip does fall inside a documented
|
|
window, which is the claim worth making.
|
|
|
|
------------------------------------------------------------------------------
|
|
WHY IT LIVES HERE AND NOT IN THE FIRMWARE OR THE WITNESS
|
|
|
|
Three vantage points see three different things. The firmware knows what it
|
|
WROTE, the witness Pi knows what was ON THE WIRE and whether the inverter ACKed
|
|
it, and the add-on knows what it INTENDED and holds the authoritative meter.
|
|
|
|
This check is not command-versus-wire; it is command-versus-physics. It needs
|
|
the emitted setpoint and the P1 net figure in the same place, and it needs the
|
|
authority to command zero when they fight:
|
|
|
|
- the witness is read-only by charter and cannot command anything, so it could
|
|
raise the alarm but never take the trip action the AC requires;
|
|
- the firmware has the wire but not the meter. §5.1 makes P1 the only
|
|
authoritative grid figure and the board only ever receives its AGE
|
|
(`sensor.goodwe_p1_sample_age`), never its value. Putting the check there
|
|
would mean shipping the meter to the ESP32 for no other reason, and it would
|
|
be C++ per install rather than one generic guard;
|
|
- the add-on has both halves already, plus an arbiter that can outrank every
|
|
strategy with a P_SAFETY claim.
|
|
|
|
The cost of standing here is real and worth stating: this watchdog compares what
|
|
we MEANT to put on the wire against the meter. If the firmware or the driver
|
|
inverts the sign on the way out, the add-on's own view still agrees with itself
|
|
and this check stays silent. That gap is a wire-versus-intent problem and it
|
|
belongs to the witness, not here.
|
|
|
|
------------------------------------------------------------------------------
|
|
WHAT "DISAGREEMENT" MEANS, PRECISELY
|
|
|
|
disagree = following
|
|
and commanded_w * grid_w < 0
|
|
and abs(commanded_w) > min_w
|
|
and abs(grid_w) > min_w
|
|
|
|
Four gates, and each one exists to exclude a specific thing that is NOT a fault:
|
|
|
|
1. `following`. ⚠️ THE GATE THAT IS EASIEST TO FORGET AND MOST EXPENSIVE TO
|
|
OMIT. Sign agreement is a property of a GRID-FOLLOWING loop. It is not
|
|
expected of a strategy that deliberately commands against the meter -
|
|
maintenance is exactly that. A maintenance grid charge of 1500 W into a house
|
|
drawing 500 W puts +2000 W on the meter against a -1500 W command: opposed,
|
|
both far over any sane threshold, and sustained for HOURS. Armed
|
|
unconditionally this watchdog would trip on every single maintenance cycle,
|
|
which is not a tuning problem, it is the wrong question being asked. So the
|
|
caller says whether the grid-following loop actually won the arbitration this
|
|
cycle, and the timer only accumulates when it did.
|
|
|
|
⚠️ 1500 W, NOT 2500 W, AND THE REASON IS NOT COSMETIC. The example has to be
|
|
a charge the arbiter actually produces. When this was written the shipped
|
|
`maintenance_charge_w` was 2500 against a `max_w` of 2000, so the cheap-window
|
|
claim was `hi = -2500` against a supervised `lo = -2000`: lo > hi, which
|
|
arbiter.py resolves as a CONTRADICTION and 0 W, not as a -2500 W charge - a
|
|
default maintenance charge that silently never happened. MAINT-04 refused that
|
|
pair in validate.py and moved the shipped rate to 2000; CFG-03 gave the
|
|
default one source of truth so main.py and MaintConfig could not go on
|
|
answering 2500 to a dict that had never said so. The number here stays 1500
|
|
because the point of the example is a charge STRICTLY INSIDE the ceiling, not
|
|
one that happens to sit exactly on it.
|
|
|
|
2. The strict `< 0` product. "Opposed" and "neither side is zero" in one
|
|
expression.
|
|
|
|
3. The 300 W magnitude gate ON BOTH SIDES. Near zero the sign is noise, and near
|
|
zero is where a healthy loop LIVES - the deadband holds the command still
|
|
while the meter wanders either side of `target_grid_w`. On the healthy day
|
|
only 4 212 of 17 280 samples (24 %) are even gated in; the other 76 % are the
|
|
converged band where sign carries no information. This is also why the
|
|
deadband cannot produce a false trip: inside it the meter is by definition
|
|
under 300 W.
|
|
|
|
4. The 180 s continuity requirement. This is what excludes the legitimate
|
|
transient. A load step reverses the meter's sign instantly while the command
|
|
is still where the last cycle left it; slew limiting then needs a few cycles
|
|
to cross over. Measured on the healthy day, the LONGEST such crossing is 30 s
|
|
- and that is with a vendor controller far cruder than ours. 180 s is six
|
|
times that. It is not a fitted threshold: at 60 s the healthy day still
|
|
produces zero trips, and at 300 s the fault day still produces 13.
|
|
|
|
⚠️ A BATTERY THAT REFUSES A COMMAND CANNOT TRIP THIS, and that is structural
|
|
rather than lucky. At SoC 10 % in Standby, a 1350 W discharge command into a
|
|
1350 W import is DECLINED: the battery does nothing, so the meter stays at
|
|
+1350 W and the command stays at +1350 W. Same sign. Agreement. A refusal leaves
|
|
the meter where it was; only the inverter ACTING moves it, and acting in the
|
|
commanded direction moves it toward zero, not through it. The failure mode this
|
|
check keys on is the opposite of a refusal - it is the inverter obediently doing
|
|
a wrong thing.
|
|
|
|
⚠️ A FROZEN METER cannot trip it either, for the same reason read the other way:
|
|
a stuck reading of +800 W makes the loop command discharge and keep commanding
|
|
discharge. Command and meter converge on AGREEING. Freezes are SAFETY-12's and
|
|
SAFETY-01's problem, and this watchdog neither helps nor hinders there.
|
|
|
|
⚠️ ALL OF THAT DEPENDS ON ONE PROPERTY OF THE CONTROL LAW, so it is an invariant
|
|
and not a happy accident: the saturation freeze in control.py must keep letting
|
|
the command FALL, CROSS ZERO AND REVERSE - "may not wind further in the direction
|
|
it is already pushing", not "only corrections that shrink |i_w|". control.py
|
|
records the shrink-only encoding as tried and rejected because it pins the command
|
|
at its last value indefinitely; a pinned command against a meter that has moved on
|
|
is precisely a sustained sign disagreement, so under that encoding this watchdog
|
|
WOULD false-trip on a plain load step. If anyone ever revisits that freeze, this
|
|
module's no-false-positive argument is downstream of the answer.
|
|
|
|
------------------------------------------------------------------------------
|
|
WHAT IT DOES NOT CATCH
|
|
|
|
An inverter that ACTS opposite to what it was told. Tell it to discharge 2000 W
|
|
and have it charge instead, and the meter moves FURTHER the way it was already
|
|
going - so the command and the meter agree, all the way into the rail. This
|
|
watchdog compares the loop's output direction against the input state it was
|
|
regulating; it is blind to the plant lying about which way it went. That is the
|
|
saturation freeze's question (|command - the battery's actual power| > 500 W,
|
|
NOTES.md §4.7 row 4), and the two checks are complementary rather than
|
|
overlapping. Neither substitutes for the other.
|
|
|
|
⚠️ THAT PARAGRAPH IS ABOUT THE COMMAND-VERSUS-METER COMPARISON, AND SINCE
|
|
SAFETY-30 THAT IS NOT THE ONLY ONE THIS MODULE DRIVES. At a roster of two or
|
|
more each unit is ALSO graded against its own measured power, which is exactly
|
|
the comparison the paragraph says nothing here makes - see the SAFETY-30 section
|
|
below, which says what that does and does not take away from the freeze. At the
|
|
shipped roster of one the paragraph stands unchanged, word for word: the site
|
|
comparison is the only detector armed and the freeze is still the whole answer.
|
|
|
|
Nor does it see a sign inversion introduced DOWNSTREAM of this add-on - in the
|
|
firmware, on the wire, or in a future driver. Our view stays self-consistent and
|
|
this stays silent. That is a wire-versus-intent problem and it belongs to the
|
|
witness, which is the only vantage point that sees both.
|
|
|
|
------------------------------------------------------------------------------
|
|
SAFETY-30 - AT TWO INVERTERS THIS IS TWO QUESTIONS, NOT ONE
|
|
|
|
Everything above was written at a roster of ONE, and at N=1 it is right for a
|
|
reason that stops being true at N=2. `commanded_w` is what THIS controller put
|
|
on THIS actuator. `grid_w` is what the WHOLE SITE is exchanging with the
|
|
network. At one inverter those are two views of one thing - §6.2's identity -
|
|
which is what makes comparing them sound. At two they are not:
|
|
|
|
* A FALSE TRIP THE COMPARISON CANNOT AVOID. "Inverter A commanded -1500 W
|
|
while the meter reads +1500 W" is evidence of nothing at N=2: inverter B may
|
|
be discharging 3000 W and doing exactly what it was told. No fault anywhere,
|
|
and the check NOTES.md calls the highest-value entry in the safety table -
|
|
93 % agreement while healthy against 4 % during runaway, four episodes, zero
|
|
false positives - fires.
|
|
* THE CONVERSE, WHICH IS WORSE. A unit genuinely driving the wrong way is
|
|
MASKED, because the site figure NETS. Commanded +1500 W, measured -1500 W,
|
|
beside a sibling measured at +1500 W: the meter sits exactly where a healthy
|
|
site leaves it. This is not a threshold that is set too high - the
|
|
information is destroyed by the summation, and no tuning recovers it.
|
|
|
|
So the one comparison becomes two, each asked of the evidence that can answer
|
|
it, and they are DIFFERENT FAULTS rather than one fault at two scales:
|
|
|
|
PER UNIT this unit's standing command against ITS OWN measured power -
|
|
`Inverter.batt`, which on the live system is holding register
|
|
0x0518, this inverter's own AC power, per-instance since DEV-02.
|
|
Conclusion: "THE ACTUATOR I AM DRIVING IS NOT DOING WHAT I TOLD
|
|
IT". The evidence is about one unit, so the remedy is that unit:
|
|
the `failsafe` SET, which arbiter.CLAIM_SCOPE has called a UNIT
|
|
claim from the day it was written, for this exact reason.
|
|
SITE the SUM of the standing commands against the meter. Conclusion:
|
|
"THE SITE AS A WHOLE IS DRIVING THE WRONG WAY" - an inverted gain,
|
|
an integrator wound past the rail, a strategy pushing the wrong
|
|
side of the meter. That is the fault the 2026-08-20 capture
|
|
contains and it is a property of the CONNECTION, so the remedy is
|
|
the connection: a SITE SET that stops every inverter.
|
|
|
|
⚠️ AND AT N=1 THERE IS EXACTLY ONE ARMED DETECTOR, WHICH IS THE ONE THAT
|
|
SHIPPED. The two questions above are the SAME SENTENCE at a roster of one: the
|
|
sum of one command is that command, and "the site" and "the unit" name one
|
|
actuator. Arming both would be arming a duplicate - two latches, two banners and
|
|
two chances to spend a false-positive record that was earned by ONE comparison
|
|
on two captured bus days. `Controller.cycle()` therefore hands the single unit's
|
|
watchdog THE METER, under the shipped `sign_disagree_w` / `sign_disagree_s`,
|
|
exactly as before, and never arms the site one. `arm()` below is where that
|
|
choice is written down. Nothing about a one-inverter install changes - not the
|
|
comparison, not the thresholds, not the store keys, not the claim, not the
|
|
banner.
|
|
|
|
⚠️ WHAT THE PER-UNIT COMPARISON IS NOT. Two guards that overlap without saying
|
|
so is how a false-alarm generator gets built, so both neighbours are named:
|
|
|
|
* IT IS NOT SAFETY-31. `circulation.py` compares the units AGAINST EACH OTHER
|
|
- `sum(|B_i|) - |sum(B_i)|` over MEASURED power - and catches two inverters
|
|
pushing THROUGH each other, which nets to zero at the meter and about which
|
|
no single command is evidence at all. This compares each unit against ITS
|
|
OWN COMMAND. Take two units faithfully executing an allocation that ALLOC-01
|
|
would refuse: circulating power is the whole loop, and every per-unit sign
|
|
check here reads perfect agreement. Take one unit doing the exact opposite
|
|
of its share while its sibling covers for it: this reads the whole
|
|
disagreement, and circulating power can be zero, because the two measured
|
|
powers may well agree in sign. Neither is the other with a different
|
|
threshold.
|
|
* IT IS NOT THE SATURATION FREEZE, AND THE OVERLAP IS REAL AND BOUNDED. The
|
|
freeze asks `|command - measured| > saturation_w` (500 W) and concludes "the
|
|
inverter cannot follow"; it then stops the integrator winding, which is a
|
|
control action taken every day, on a unit that is merely SATURATED as
|
|
readily as on one that is reversed, and it neither latches nor parks
|
|
anything. This asks for OPPOSITION - a sign, on both sides of a threshold,
|
|
held continuously - which a saturated-but-obedient unit cannot produce at
|
|
all, and it LATCHES and commands that unit to zero. The freeze is a
|
|
magnitude test that shapes the loop; this is a direction test that stops an
|
|
actuator.
|
|
|
|
⚠️ GATE 1 STILL APPLIES TO BOTH, AND THAT IS A DELIBERATE CHOICE WITH A COST.
|
|
The per-unit comparison could have been armed whenever ANY non-zero command is
|
|
standing - a unit should do what it is told whoever told it - and that would
|
|
catch a wedged inverter during a maintenance charge. It is not, because the
|
|
whole no-false-positive argument above is built on `following`, because the
|
|
freeze already grades that window, and because this card's own warning is that
|
|
SAFETY-02's record was earned at N=1 and must not be spent on a check that fires
|
|
when nothing is wrong. What it costs is written here rather than discovered:
|
|
between the start of a maintenance cycle and its end, a unit reversing against
|
|
its own command is the freeze's to notice, not this module's. Widening it is a
|
|
card, with a rig campaign behind it, not an edit.
|
|
|
|
------------------------------------------------------------------------------
|
|
STATE, TRIP AND LATCH
|
|
|
|
One timestamp, not a sample counter. The cycle rate is not fixed - a meter
|
|
arrival, or `cycle_floor_s` elapsing, whichever comes first - so N samples is a
|
|
different wall-clock window on every path, and the AC is written in minutes.
|
|
|
|
The latch is not optional and not a nicety. Commanding 0 W removes the command
|
|
side of the comparison (|0| is not > 300), so the disagreement disappears the
|
|
instant the trip works. Un-latched, this watchdog would trip, clear itself, and
|
|
hand the actuator straight back to the loop that was driving the wrong way -
|
|
three minutes later, forever. `tripped` therefore stays true until an operator
|
|
`reset()`s it by starting control again, which is a deliberate human act after
|
|
looking at why.
|
|
|
|
⚠️ AND THE LATCH HAS TO OUTLIVE THE PROCESS. Held only in memory it is not a
|
|
latch at all - a container restart clears it and the add-on comes back armed,
|
|
into the same fault, unattended. Worse than merely losing state: "restart the
|
|
add-on" is the first thing an operator confronted with a FAILSAFE banner and a
|
|
0 W inverter will try, and under a supervisor that restarts a crashed add-on it
|
|
is a loop of three-minute wrong-direction bursts. `tripped` and `reason` are
|
|
therefore constructor arguments, restored from `Store` in `Controller.__init__`
|
|
and written back the moment the trip happens. See store.py.
|
|
"""
|
|
|
|
from dataclasses import dataclass
|
|
from datetime import datetime
|
|
|
|
|
|
# SAFETY-30. CFG-03: THE ONE IN-CODE COPY OF THE FOUR MULTI-INVERTER DEFAULTS.
|
|
# `config.yaml`'s `options:` block is the source of truth an operator edits and
|
|
# the Supervisor reads, and `schema:` holds the bounds. This dict exists for the
|
|
# reason `circulation.CIRC_OPTION_DEFAULTS` and `maintenance.MAINT_OPTION_DEFAULTS`
|
|
# do - the Dockerfile does not put config.yaml in the image, so the running
|
|
# add-on cannot read it - and `test_watchdog.py` pins the two together by READING
|
|
# config.yaml, so a drift goes red in CI rather than shipping.
|
|
#
|
|
# ⚠️ EVERY KEY HERE IS DEAD AT A ROSTER OF ONE, and that is the point rather
|
|
# than a caveat. `Controller._arm_sign_watchdogs()` arms the single unit's
|
|
# detector against the meter with the SHIPPED `sign_disagree_w` /
|
|
# `sign_disagree_s`, which are read in main.py exactly where they always were
|
|
# and are not in this dict. Nothing below is consulted until a second inverter
|
|
# exists.
|
|
SIGN_OPTION_DEFAULTS = {
|
|
# --- THE MAGNITUDE GATE, AND IT IS NOT SAFETY-02's 300 W BY ACCIDENT ---
|
|
# ⚠️ DERIVED FROM `control.Tuning.saturation_w`, WHICH IS 500 W, AND THE
|
|
# DERIVATION IS THE POINT. This check gates on BOTH sides being over
|
|
# `min_w`, so the smallest divergence it can ever see between a command and
|
|
# the power that command produced is `2 * min_w` - the two are OPPOSED, so
|
|
# the gap is the sum of the magnitudes. At 250 W that is exactly 500 W: the
|
|
# gap at which `compute()`'s own anti-windup test already concludes THE
|
|
# INVERTER IS NOT FOLLOWING and freezes the integrator. So this watchdog
|
|
# cannot fire on a divergence the shipped control law itself calls ordinary
|
|
# tracking, and the two guards meet at a number rather than overlapping in a
|
|
# band nobody has drawn.
|
|
#
|
|
# MEASURED, on 1890 closed-loop two-inverter runs in which NOTHING is faulty
|
|
# (every unit executes the share the allocator gave it), swept over the
|
|
# WHOLE range config.yaml accepts for `gain` float(0.05,1.0) and `slew_w`
|
|
# int(50,5000), across both families of stimulus that can produce legitimate
|
|
# opposition. Worst CONTINUOUS opposition between a unit's standing command
|
|
# and its own 0x0518 power, as a function of this threshold:
|
|
#
|
|
# 50 W -> 25.0 s 250 W -> 10.0 s 700 W -> 5.0 s
|
|
# 150 W -> 15.0 s 300 W -> 10.0 s 1000 W -> 0.0 s
|
|
# 200 W -> 15.0 s 500 W -> 5.0 s
|
|
#
|
|
# ⚠️ SAFETY-39 MOVED THAT SCHEMA FLOOR FROM 50 TO 0, AND THE CAMPAIGN ABOVE
|
|
# WAS NOT RE-RUN. It does not have to be, and here is the argument rather
|
|
# than the assurance. What these windows are about is how long the command
|
|
# takes to CROSS ITS RAIL - the swept worst case is the SLOWEST crossing,
|
|
# which is `slew_w` at its floor, 5000/50 = 100 cycles. `slew_w: 0` means
|
|
# "follow the rail" and resolves to `SLEW_RAIL_FRAC * max_w`, which is
|
|
# exactly 9 cycles at EVERY rail by construction. So the new value can never
|
|
# be the slowest crossing at any `max_w`, and it is strictly inside the
|
|
# envelope this campaign measured. An explicit number still spans 50..5000
|
|
# and is unchanged. If `SLEW_RAIL_FRAC` is ever loosened past 1/100, this
|
|
# paragraph stops being true and the campaign has to be re-run.
|
|
#
|
|
# ⚠️ AND THE CURVE IS FLAT, WHICH IS THE FINDING THAT DECIDED THE SHAPE OF
|
|
# THIS PAIR. Unlike the site comparison - where the meter is driven by the
|
|
# LOAD and a flapping load holds it opposed for minutes - a unit's own power
|
|
# CHASES ITS OWN COMMAND through a bounded ramp and a first-order lag, so the
|
|
# opposition can only last as long as the crossing does. Dropping the
|
|
# threshold from 300 W to 50 W buys 15 s of extra transient and nothing else.
|
|
# The work is therefore done by the WINDOW below, and this number is set
|
|
# where the control law already draws its line rather than being tuned.
|
|
#
|
|
# The largest opposed magnitude seen anywhere in those 1890 runs is 875 W;
|
|
# a threshold above that would gate NOTHING legitimate in at all, and is
|
|
# rejected because it would also be blind to a unit reversing a 900 W share.
|
|
"unit_sign_disagree_w": 250,
|
|
|
|
# --- THE WINDOW -------------------------------------------------------
|
|
# Three bounds, and the value clears all three.
|
|
#
|
|
# LOWER, from the rig: 10.0 s - two control cycles, the worst CONTINUOUS
|
|
# legitimate opposition above 250 W across all 1890 runs. It is a
|
|
# reversing site whose second write lands a cycle after the first, both
|
|
# units then slewing through zero behind a 10 s lag; nothing is wrong.
|
|
#
|
|
# LOWER, from the firmware: `inverter.LINK_STALE_S`, 30 s. A unit not
|
|
# executing its share for less than the firmware's own veto window is a
|
|
# LINK fault, and the add-on already answers it - liveness drops the unit
|
|
# and the survivors carry its share, with the record saying so. Firing
|
|
# inside that window would raise a second, louder alarm about a state
|
|
# already being handled, which is the SAFETY-15/SAFETY-19 failure exactly.
|
|
#
|
|
# UPPER, from SAFETY-02 itself: 180 s, the SITE hold. A unit that is not
|
|
# doing what it was told must not be parked LATER than the site would be
|
|
# parked for the same watts, and unlike the site comparison this one needs
|
|
# nothing to net out before it can conclude - the evidence is complete on
|
|
# the first cycle it appears.
|
|
#
|
|
# 85 s: seventeen control cycles - 8.5x the worst legitimate episode this
|
|
# rig can produce, 2.8x the firmware's own veto window, and under HALF the
|
|
# site hold. ⚠️ IT IS DELIBERATELY NOT 60, 75, 90, 120 OR 180. A round
|
|
# number here would be a guess wearing a measurement's clothes, and the
|
|
# three bounds above are what the number has to answer to.
|
|
#
|
|
# ⚠️ AND INVERTER B DOES NOT PHYSICALLY EXIST. Its ramp and lag in the rig
|
|
# are inverter A's, COPIED, carrying `measured=False` - SPIKE-04 measures
|
|
# the real ones. A slower second unit lengthens every crossing above, which
|
|
# is what the 8.5x buys headroom for. SPIKE-04 is the revisit trigger for
|
|
# both numbers here, not a reason to move them now.
|
|
"unit_sign_disagree_s": 85,
|
|
|
|
# --- THE SITE COMPARISON AT N>=2, AND IT IS NOT THE SHIPPED PAIR -------
|
|
# ⚠️ THE THRESHOLD IS CARRIED AND THE WINDOW IS NOT, WHICH IS A FINDING
|
|
# RATHER THAN A COMPROMISE. Only one of the two arguments behind 300 W /
|
|
# 180 s depends on how many inverters the site has:
|
|
#
|
|
# THE THRESHOLD is about NOISE. "Near zero the sign is noise, and near
|
|
# zero is where a healthy loop LIVES" - the deadband holds the command
|
|
# still while the meter wanders either side of `target_grid_w`. The
|
|
# deadband is 15 W and P1 is one meter whatever the roster, so nothing
|
|
# about that argument moves. 300 W, carried - and the two-inverter
|
|
# campaign says the carry is safe rather than merely convenient: in the
|
|
# load-reversal family the site figure ever goes opposed AND over 300 W in
|
|
# 994 of 1050 runs at two inverters and 955 of 1050 at one. The gate opens
|
|
# about as often either way; what changed is how LONG it stays open, which
|
|
# is the window's problem and not the threshold's.
|
|
#
|
|
# THE WINDOW is about a TRANSIENT, and the transient scales with the site.
|
|
# The measured worst legitimate case is a load reversal that the loop
|
|
# crosses zero slowly: the command has to travel the whole rail at
|
|
# `slew_w` per cycle, and at N>=2 the rail is `Controller.site_max_w` - the
|
|
# ROSTER SUM - while `slew_w` is unchanged. So the crossing takes about
|
|
# twice as long at two inverters as at one, and that is exactly what the
|
|
# rig measures.
|
|
#
|
|
# 1890 closed-loop runs in which nothing is faulty, the SAME campaign the
|
|
# per-unit pair above is read off, run at BOTH rosters as its own control.
|
|
# Worst CONTINUOUS opposition between the standing command(s) and the meter,
|
|
# LOAD-REVERSAL family, at a 300 W threshold:
|
|
#
|
|
# N=1 (the shipped comparison) 110.0 s - inside 180 s, 1.6x
|
|
# N=2 (the sum, same stimuli) 215.0 s - OVER 180 s
|
|
#
|
|
# ⚠️ SO CARRYING 180 s WOULD HAVE SHIPPED A FALSE TRIP, and it is a config a
|
|
# schema-legal install can reach: `gain` 0.05 and `slew_w` 50 are both the
|
|
# floor of what config.yaml accepts, and a +-2200 W load reversal is an oven
|
|
# and a heat pump. That is the number this card exists to not assume.
|
|
#
|
|
# 325 s: 1.51x the measured worst, well inside the 900 s quarter that bills
|
|
# the capacity tariff, and still inside every episode the fault day
|
|
# contains - 2026-08-20's four clusters run 16, 30, 77 and 77 minutes and
|
|
# its opposed runs reach 1750 s, so a 325 s window fires in all of them.
|
|
# What it costs is stated rather than hidden: about two and a half extra
|
|
# minutes of wrong-direction SITE power before the stop, bought to remove a
|
|
# false trip that a legal configuration produces. ⚠️ DELIBERATELY NOT 300 OR
|
|
# 360; the measurement is what it answers to.
|
|
#
|
|
# ⚠️ AND IT IS NOT THE PER-UNIT WINDOW EITHER, IN THE OTHER DIRECTION. The
|
|
# per-unit comparison is 85 s because a unit's own power CHASES ITS OWN
|
|
# COMMAND and the transient is two cycles; the site comparison is 325 s
|
|
# because the meter is driven by the LOAD and the transient is the loop
|
|
# crossing zero. Same predicate, two instruments, two windows, both measured.
|
|
"site_sign_disagree_w": 300,
|
|
"site_sign_disagree_s": 325,
|
|
}
|
|
|
|
# ⚠️ A LIMIT OF THE SITE COMPARISON THAT SAFETY-30 DID NOT INTRODUCE AND DOES
|
|
# NOT FIX, RECORDED HERE BECAUSE THE CAMPAIGN FOUND IT AND SILENCE WOULD BE
|
|
# WORSE. The rig's other stimulus family is a FLAPPING LOAD, and a square wave
|
|
# whose half period is about one control cycle opposes the command on EVERY
|
|
# cycle: the command chases the previous half-cycle's meter while the meter has
|
|
# already reversed. Across the 840 flapping runs the worst continuous opposition
|
|
# at a 300 W threshold is 980 s at two inverters - and 1000 s at ONE, which is
|
|
# the whole point. The N=1 arm is the control, and it says this is a property of
|
|
# comparing a lagged command against a meter, not a property of the second
|
|
# inverter. No window inside the 900 s billing quarter survives it, and no
|
|
# threshold up to 1000 W does either.
|
|
#
|
|
# It is not acted on here for three reasons, in order of weight: it is identical
|
|
# at the roster this add-on actually ships, so acting on it would be changing a
|
|
# live safety path on evidence that has nothing to do with this card; a
|
|
# ~6 kW square wave at 0.1 Hz sustained for a quarter of an hour is not a house
|
|
# (the captured healthy day's longest opposed run is 30 s); and the fix is not a
|
|
# number - it wants the detector to notice that the disagreement is ALTERNATING,
|
|
# which is a different check. It is a card, and this comment is its evidence.
|
|
|
|
# SAFETY-30. THE TWO NOUNS THE TRIP SENTENCE CAN NAME, WRITTEN ONCE. They are
|
|
# constants rather than literals at the call sites because `main.py` arms the
|
|
# detectors and `test_watchdog.py` grades what they say, and a string typed in
|
|
# two places is a string that drifts in one of them - the same argument
|
|
# `store.unit_key()` makes about a key name. `AGAINST_METER` is the wording that
|
|
# has always shipped and must not change: it is what a one-inverter install's
|
|
# `reason` reads, in the store, in the action log and on the status page.
|
|
AGAINST_METER = "a meter reading"
|
|
AGAINST_OWN_POWER = "its own measured power"
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class SignWatchdogConfig:
|
|
"""Defaults, not constants - config.yaml carries both (SAFETY-02 AC 5)."""
|
|
|
|
min_w: float = 300.0
|
|
hold_s: float = 180.0
|
|
|
|
|
|
def disagrees(commanded_w, against_w, following: bool,
|
|
cfg: SignWatchdogConfig) -> bool:
|
|
"""The predicate, and the ONLY place it is written down.
|
|
|
|
⚠️ Public because `sim/replay_sign.py` needs it too - to describe episodes, to
|
|
measure agreement, to sweep the clock-skew guard - and it had reimplemented it
|
|
inline. Two copies of a safety predicate is one copy plus a future
|
|
disagreement about what the check means, and the copy that drifts is always
|
|
the one in the harness that grades it.
|
|
|
|
⚠️ `against_w` WAS CALLED `grid_w`, AND THE RENAME IS THE WHOLE OF SAFETY-30
|
|
IN ONE PARAMETER. The arithmetic is unchanged - opposed signs, neither side
|
|
near zero, and the loop actually driving - but WHAT the command is opposed
|
|
TO is now a property of the caller rather than a fact about this function.
|
|
The site comparison hands it the meter; each unit's own comparison hands it
|
|
that unit's measured power (register 0x0518). One predicate, two callers,
|
|
for `loop_is_following()`'s reason: a safety predicate written down twice is
|
|
one copy plus a future disagreement about what the check means. The
|
|
parameter is positional at every call site in both repositories, so the
|
|
rename moves no caller.
|
|
"""
|
|
return (following
|
|
and commanded_w is not None and against_w is not None
|
|
and commanded_w * against_w < 0
|
|
and abs(commanded_w) > cfg.min_w
|
|
and abs(against_w) > cfg.min_w)
|
|
|
|
|
|
def loop_is_following(resolution, loop_asked) -> bool:
|
|
"""Did the grid-following loop actually choose the number going on the wire?
|
|
|
|
Gate 1 of the predicate above, and it lives here rather than inline in
|
|
`cycle()` for the reason `disagrees()` does: the test suite has to be able to
|
|
assert on THE SHIPPED EXPRESSION, not on a transcription of it that drifts
|
|
apart from it in silence.
|
|
|
|
⚠️ IT WAS A TRANSCRIPTION, AND THAT WAS MEASURED, NOT FEARED. The P_PEAK case
|
|
in test_watchdog.py exists to make whoever lands peak shaving come here and
|
|
change an expectation. With the expression copied into the test it did not do
|
|
that: editing the shipped gate alone to `winner in ("loop", "peak")` left every
|
|
suite green while the watchdog went blind for the whole of every peak event.
|
|
One definition, two callers, and that test now bites.
|
|
|
|
`resolution` the arbiter's Resolution. Duck-typed on `.winner` and
|
|
`.target_w` so this module still imports nothing from arbiter.
|
|
`loop_asked` what the grid-following loop claimed this cycle, or None if it
|
|
did not claim at all.
|
|
|
|
"The loop won" is NOT the question, and the difference is a whole class of
|
|
false trip. A LIMIT claim can survive the loop winning and still replace its
|
|
number: maintenance's cheap-window shaping is `hi = -maintenance_charge_w`, so
|
|
at 1500 W the loop can ask for +500 W of discharge, win the arbitration, and
|
|
have the emitted value come out at -1500 W of grid charge. Winner says "loop";
|
|
the wire carries maintenance's intent.
|
|
|
|
So the test is on the NUMBER: the loop is following only while the emitted
|
|
value points the way the loop asked AND is no larger than it asked - limits
|
|
reduced it, they did not redirect it. The magnitude half is load-bearing on
|
|
its own, because a `lo` limit can raise magnitude without flipping sign, and
|
|
that is not the loop following either. Zero on either side falls out as
|
|
not-following, which costs nothing: a 0 W command cannot be a
|
|
wrong-direction command.
|
|
|
|
⚠️ FORWARD HAZARD, and the claim block in cycle() already anticipates it.
|
|
`winner == "loop"` disarms the watchdog under ANY higher-priority SET claim.
|
|
Today only maintenance and the safety stops emit one and neither follows the
|
|
meter, so the gate costs nothing. A peak-shaving strategy claiming SET at
|
|
P_PEAK **is** grid-following in sign, and the moment it lands this goes blind
|
|
for the duration of every peak event. Widening it means asking "does the
|
|
winning strategy follow the meter", not adding another name to a comparison.
|
|
"""
|
|
return (resolution.winner == "loop" and loop_asked is not None
|
|
and resolution.target_w * loop_asked > 0
|
|
and abs(resolution.target_w) <= abs(loop_asked))
|
|
|
|
|
|
class SignWatchdog:
|
|
"""One instance per COMPARISON, evaluated once per control cycle.
|
|
|
|
⚠️ SINCE SAFETY-30 THAT IS NOT THE SAME SENTENCE AS "one per actuator",
|
|
which is what this docstring used to say. There is one instance per
|
|
inverter - `Inverter.sign_wd`, latched under that unit's own store keys -
|
|
AND one for the site, `Controller.site_sign_wd`, latched under flat ones.
|
|
The two ask different questions of different evidence and reach different
|
|
remedies; `arm()` below is where an instance is told which of them it is.
|
|
At a roster of one they would be the same question, so only the per-unit
|
|
instance is ever armed there - see the SAFETY-30 section of the module
|
|
docstring.
|
|
|
|
⚠️ NOT SHARED ACROSS CADENCES. `disagree_since` is a wall-clock start time,
|
|
so driving one instance from two loops at two rates would have each one
|
|
resetting the other's window. If a fast on-device path ever wants this
|
|
check, it gets its own instance - the same exactly-one-writer discipline the
|
|
arbiter applies to the actuator, applied to the detector's own state.
|
|
"""
|
|
|
|
def __init__(self, cfg: SignWatchdogConfig | None = None, *,
|
|
tripped: bool = False, reason: str = "",
|
|
against: str = AGAINST_METER):
|
|
self.cfg = cfg or SignWatchdogConfig()
|
|
# SAFETY-30. WHAT THE COMMAND IS BEING COMPARED AGAINST, as the operator
|
|
# reads it in the trip sentence. The default is the meter, which is what
|
|
# every trip this add-on has ever taken was compared against and is what
|
|
# keeps a one-inverter install's `reason` string byte-identical.
|
|
self.against = str(against)
|
|
self.disagree_since: datetime | None = None
|
|
# ⚠️ RESTORED FROM THE STORE, NOT DEFAULTED. A latch that lives only in
|
|
# memory is not a latch: a container restart clears it, and the add-on
|
|
# comes back armed, into the same fault, with no operator involved. See
|
|
# store.py's note on `sign_tripped` for the measured behaviour.
|
|
self.tripped = bool(tripped)
|
|
self.reason = reason if self.tripped else ""
|
|
|
|
# -- which of SAFETY-30's two questions this instance is asking ----------
|
|
def arm(self, cfg: SignWatchdogConfig, against: str) -> None:
|
|
"""Set the thresholds and the subject of the comparison. Nothing else.
|
|
|
|
⚠️ IT IS CONFIGURATION, NOT STATE, AND IT DELIBERATELY DOES NOT TOUCH
|
|
`disagree_since`, `tripped` OR `reason`. A detector that cleared its own
|
|
window when it was told what it was comparing against would lose the
|
|
episode it is in the middle of, every cycle, and would never accumulate
|
|
a hold at all - the check would be permanently green and nothing would
|
|
say so.
|
|
|
|
⚠️ AND IT IS RE-STATED EVERY CYCLE RATHER THAN FROZEN AT CONSTRUCTION.
|
|
The answer is a property of the ROSTER and of nothing else, and
|
|
`Controller.cycle()` already re-reads the roster every cycle for
|
|
liveness and for the allocation. Freezing it in `__init__` would also
|
|
make it wrong for any caller that appends an inverter afterwards, which
|
|
is exactly how a two-inverter fixture is built in this repository - and
|
|
a fixture that silently got the one-inverter comparison would grade
|
|
nothing while looking green.
|
|
"""
|
|
self.cfg = cfg
|
|
self.against = str(against)
|
|
|
|
# -- the whole check, and it is a pure function of its arguments ----------
|
|
def evaluate(self, commanded_w, against_w, following: bool,
|
|
now: datetime) -> bool:
|
|
"""Advance the detector one cycle. Returns True while tripped.
|
|
|
|
`commanded_w` the EMITTED setpoint - what is standing on the wire now,
|
|
after deadband, slew, quantisation and the arbiter. Not
|
|
the optimizer's raw wish: the wire carries the emitted
|
|
value, so anything else can disagree with the meter for a
|
|
reason that is not a fault. For a per-unit instance it is
|
|
THAT UNIT'S share of it; for the site instance it is the
|
|
roster sum.
|
|
`against_w` what the command is graded against, and SAFETY-30 made
|
|
this a decision rather than a fact: net P1 for the SITE
|
|
comparison (+ = importing), and this unit's own measured
|
|
power for the PER-UNIT one (+ = discharging, register
|
|
0x0518). Both carry the same sign convention as the
|
|
command, which is why one predicate serves both. None
|
|
when the reading is stale or missing, which CLEARS the
|
|
timer rather than accumulating - a frozen figure held
|
|
against a moving command would otherwise manufacture a
|
|
disagreement in about three minutes, on top of SAFETY-01
|
|
having already commanded zero for the same cause.
|
|
`following` did the grid-following loop actually win this cycle? See
|
|
gate 1 in the module docstring, and the SAFETY-30 section
|
|
on why the per-unit comparison keeps it.
|
|
`now` the caller's clock, so this stays testable and replayable.
|
|
"""
|
|
if self.tripped:
|
|
# Latched. Do not re-evaluate: the trip removed the evidence.
|
|
return True
|
|
|
|
cfg = self.cfg
|
|
if not disagrees(commanded_w, against_w, following, cfg):
|
|
self.disagree_since = None
|
|
return False
|
|
|
|
if self.disagree_since is None:
|
|
self.disagree_since = now
|
|
return False
|
|
|
|
held = (now - self.disagree_since).total_seconds()
|
|
if held < cfg.hold_s:
|
|
return False
|
|
|
|
self.tripped = True
|
|
# ⚠️ `self.against` IS THE ONLY THING SAFETY-30 CHANGED IN THIS STRING,
|
|
# and it defaults to the wording that shipped - "a meter reading" - so a
|
|
# one-inverter install's `reason` is byte-identical to what an operator
|
|
# has always read (test_watchdog.py pins its meaning). A per-unit
|
|
# instance names what it actually compared instead, because "commanded
|
|
# +1500 W against a meter reading -1500 W" would be a false statement
|
|
# about which instrument produced the evidence.
|
|
self.reason = (
|
|
f"SIGN_DISAGREEMENT: commanded {commanded_w:+.0f} W against "
|
|
f"{self.against} {against_w:+.0f} W for {held:.0f} s continuously "
|
|
f"(limits: both over {cfg.min_w:.0f} W, held over {cfg.hold_s:.0f} s)"
|
|
)
|
|
return True
|
|
|
|
def held_s(self, now: datetime) -> float:
|
|
"""How long the current disagreement has been standing, for the UI.
|
|
|
|
⚠️ Takes the clock rather than reading one. It was a property calling
|
|
`datetime.now()`, which quietly broke the property this module's header
|
|
advertises - every other observable here is a pure function of what it
|
|
was handed - and an unclocked function is one nothing can test, which is
|
|
exactly what happened to it.
|
|
"""
|
|
if self.disagree_since is None:
|
|
return 0.0
|
|
return (now - self.disagree_since).total_seconds()
|
|
|
|
def reset(self) -> None:
|
|
self.disagree_since = None
|
|
self.tripped = False
|
|
self.reason = ""
|