adminandClaude Opus 5
5d7e275df4
checks / unit (pull_request) Successful in 5m16s
`slew_w: 1000` was derived on 2026-08-22 against a clamp of 1500 W, where "capping how far the loop can jump ahead of the inverter" was true because the rail itself bounded the excursion. The live install went to max_w 5000 and the constant did not move, so the loop crosses its whole output range in 10 s against a plant NOTES.md §5.10 measured at ~18 s. Measured on the owner's system 2026-09-05 11:43-11:46: a 3.4 kW export drove the setpoint 380 -> 5000 W in five 2 s ticks, the battery to -5154 W past a 5000 VA rating, and the grid from 3.4 kW of EXPORT to 2.8 kW of IMPORT. Nothing was broken; the rail moved. REPRODUCED FIRST, and one half of it is not reproducible at all. On plantsim.Plant and on an independent in-repo model: the setpoint stepping a full 1000 W per tick to the rail (48 of 51 export steps), and the meter crossing from export into import. The -5154 W figure is NOT reproducible in either model, because both clamp the plant at its rating - so the card's AC "battery no longer exceeds max_w in simulation" is unfalsifiable and was rewritten rather than met. THE DELIVERABLE IS THE RELATIONSHIP. SLEW_RAIL_FRAC = LOOP_TICK_S / PLANT_SETTLE_S = 2 s / 18 s = 1/9 slew_w = 0 now means FOLLOW THE RAIL - SLEW_RAIL_FRAC * max_w - exactly as integrator_max_w = 0 already means follow max_w. It is the shipped default: 222 W at max_w 2000, 556 W at 5000, 1067 W at a ~9600 W two-inverter site. validate.py rule 10 refuses an explicit value faster than that, graded against sum(unit_caps(opts)) - the roster sum, i.e. Controller.site_max_w - so a second inverter moves the bound with no operator action. VERIFIED BY BREAKING IT: restoring slew_w 1000 at max_w 5000 is refused, and the refusal names the rail, the crossing time, the measurement and what to do. Equality, one watt of slack, the live interim 400 and slew_w 0 at every rating all still start. GAIN IS THE CONSTANT THAT IS STILL WRONG, and it is NOT changed here. Over 51 export steps the over-absorption is FLAT in slew (958 W at 1000, 1081 W at 250) and monotone in gain (987 W at 0.6, 876 at 0.3, 435 at 0.2): slew_w bounds the OUTPUT, while the integrator moves gain*error per cycle and reaches a 5000 W rail in under three ticks whatever the slew is. Two control-law constants moving in one release on a running house makes the next measurement uninterpretable. ⚠️ N=2: the site rail becomes sum(max_w) ~ 9600 W (inverter B FROM DATASHEET AND SERIAL, not measured, and sim/plantsim.py's INVERTER_B is A copied), and the enforced ratio produces 1067 W there automatically. But a growing rail LOOSENS this bound, and the >= 900 W symptom goes stale at N=2 the same way the constant did - both stated in the record rather than left to be discovered. Nothing reached the inverter. The live install still runs the interim slew_w 400 applied by hand on 2026-09-05, which rule 10 accepts unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
702 lines
41 KiB
Python
702 lines
41 KiB
Python
"""SAFETY-02 - the sign-disagreement watchdog.
|
|
|
|
WHAT IT CATCHES, in one sentence: a grid-following loop that is pushing power in
|
|
the wrong direction, which every other guard in this system is blind to.
|
|
|
|
The stale-input watchdog (SAFETY-01) asks "is the meter still talking?". The
|
|
saturation freeze asks "can the inverter follow?". The output clamp asks "is the
|
|
number too big?". A loop can pass all three and still be charging the battery
|
|
out of the grid while the house imports - the meter is fresh, the inverter is
|
|
tracking beautifully, and the magnitude is inside the rating. The only thing
|
|
that separates that from healthy operation is the SIGN.
|
|
|
|
⚠️ THIS IS NOT A HYPOTHETICAL. On 2026-08-20 the vendor controller (Illusmart)
|
|
did exactly this four times in one day (NOTES.md §4.7). Both captures of that
|
|
bus are committed - `sim/scenarios/bus-2026-08-20.log.gz` and the healthy
|
|
`bus-2026-08-23.log.gz` in the infra repo - and `sim/replay_sign.py` drives this
|
|
module over both of them. Every number in this file was measured on those two
|
|
days, not chosen:
|
|
|
|
day samples gated in sign agreement
|
|
2026-08-23 4 212 / 17 280 93.4 % (healthy)
|
|
2026-08-20 7 182 / 17 280 56.4 % (the whole fault day)
|
|
cluster A 15:26-16:43 0.3 %
|
|
cluster B 17:11-18:28 0.9 %
|
|
cluster C 20:01-20:31 10.5 %
|
|
cluster D 20:55-21:11 0.0 %
|
|
|
|
⚠️ THOSE AGREEMENT FIGURES ARE THE WEAKEST EVIDENCE ON THIS PAGE, and an earlier
|
|
version of this comment leaned on them as though they were the strongest. The
|
|
clusters are found by looking for sustained opposed samples, so measuring "how
|
|
opposed is this cluster" over them is close to a tautology - and "the healthy day
|
|
contains no such cluster" is the same sentence as "the healthy day produces no
|
|
trip", not independent corroboration of it.
|
|
|
|
What actually carries the argument is two things neither the detector nor the
|
|
clustering assumes:
|
|
|
|
1. The RUN-LENGTH DISTRIBUTION. The longest unbroken opposed run on the healthy
|
|
day is 30 s. On the fault day there are 18 runs over 180 s and the longest is
|
|
1750 s. Nothing about that gap was chosen - it is what the two days measure.
|
|
2. The clusters land where a human independently wrote down onsets. NOTES.md
|
|
§4.7 records 15:01, 16:56, 19:57 and 20:23 from watching it happen, before
|
|
any of this existed, and cluster D coincides with the separately documented
|
|
inverter-off runaway of §4.7 item 2.
|
|
|
|
Be precise about (2), because the convenient overstatement is available and wrong:
|
|
the clustering does NOT reproduce four episodes. It yields THREE clusters over the
|
|
four documented onsets - C spans both 19:57 and 20:23 - plus a fourth for §4.7
|
|
item 2, which is not one of the four. Every trip does fall inside a documented
|
|
window, which is the claim worth making.
|
|
|
|
------------------------------------------------------------------------------
|
|
WHY IT LIVES HERE AND NOT IN THE FIRMWARE OR THE WITNESS
|
|
|
|
Three vantage points see three different things. The firmware knows what it
|
|
WROTE, the witness Pi knows what was ON THE WIRE and whether the inverter ACKed
|
|
it, and the add-on knows what it INTENDED and holds the authoritative meter.
|
|
|
|
This check is not command-versus-wire; it is command-versus-physics. It needs
|
|
the emitted setpoint and the P1 net figure in the same place, and it needs the
|
|
authority to command zero when they fight:
|
|
|
|
- the witness is read-only by charter and cannot command anything, so it could
|
|
raise the alarm but never take the trip action the AC requires;
|
|
- the firmware has the wire but not the meter. §5.1 makes P1 the only
|
|
authoritative grid figure and the board only ever receives its AGE
|
|
(`sensor.goodwe_p1_sample_age`), never its value. Putting the check there
|
|
would mean shipping the meter to the ESP32 for no other reason, and it would
|
|
be C++ per install rather than one generic guard;
|
|
- the add-on has both halves already, plus an arbiter that can outrank every
|
|
strategy with a P_SAFETY claim.
|
|
|
|
The cost of standing here is real and worth stating: this watchdog compares what
|
|
we MEANT to put on the wire against the meter. If the firmware or the driver
|
|
inverts the sign on the way out, the add-on's own view still agrees with itself
|
|
and this check stays silent. That gap is a wire-versus-intent problem and it
|
|
belongs to the witness, not here.
|
|
|
|
------------------------------------------------------------------------------
|
|
WHAT "DISAGREEMENT" MEANS, PRECISELY
|
|
|
|
disagree = following
|
|
and commanded_w * grid_w < 0
|
|
and abs(commanded_w) > min_w
|
|
and abs(grid_w) > min_w
|
|
|
|
Four gates, and each one exists to exclude a specific thing that is NOT a fault:
|
|
|
|
1. `following`. ⚠️ THE GATE THAT IS EASIEST TO FORGET AND MOST EXPENSIVE TO
|
|
OMIT. Sign agreement is a property of a GRID-FOLLOWING loop. It is not
|
|
expected of a strategy that deliberately commands against the meter -
|
|
maintenance is exactly that. A maintenance grid charge of 1500 W into a house
|
|
drawing 500 W puts +2000 W on the meter against a -1500 W command: opposed,
|
|
both far over any sane threshold, and sustained for HOURS. Armed
|
|
unconditionally this watchdog would trip on every single maintenance cycle,
|
|
which is not a tuning problem, it is the wrong question being asked. So the
|
|
caller says whether the grid-following loop actually won the arbitration this
|
|
cycle, and the timer only accumulates when it did.
|
|
|
|
⚠️ 1500 W, NOT 2500 W, AND THE REASON IS NOT COSMETIC. The example has to be
|
|
a charge the arbiter actually produces. When this was written the shipped
|
|
`maintenance_charge_w` was 2500 against a `max_w` of 2000, so the cheap-window
|
|
claim was `hi = -2500` against a supervised `lo = -2000`: lo > hi, which
|
|
arbiter.py resolves as a CONTRADICTION and 0 W, not as a -2500 W charge - a
|
|
default maintenance charge that silently never happened. MAINT-04 refused that
|
|
pair in validate.py and moved the shipped rate to 2000; CFG-03 gave the
|
|
default one source of truth so main.py and MaintConfig could not go on
|
|
answering 2500 to a dict that had never said so. The number here stays 1500
|
|
because the point of the example is a charge STRICTLY INSIDE the ceiling, not
|
|
one that happens to sit exactly on it.
|
|
|
|
2. The strict `< 0` product. "Opposed" and "neither side is zero" in one
|
|
expression.
|
|
|
|
3. The 300 W magnitude gate ON BOTH SIDES. Near zero the sign is noise, and near
|
|
zero is where a healthy loop LIVES - the deadband holds the command still
|
|
while the meter wanders either side of `target_grid_w`. On the healthy day
|
|
only 4 212 of 17 280 samples (24 %) are even gated in; the other 76 % are the
|
|
converged band where sign carries no information. This is also why the
|
|
deadband cannot produce a false trip: inside it the meter is by definition
|
|
under 300 W.
|
|
|
|
4. The 180 s continuity requirement. This is what excludes the legitimate
|
|
transient. A load step reverses the meter's sign instantly while the command
|
|
is still where the last cycle left it; slew limiting then needs a few cycles
|
|
to cross over. Measured on the healthy day, the LONGEST such crossing is 30 s
|
|
- and that is with a vendor controller far cruder than ours. 180 s is six
|
|
times that. It is not a fitted threshold: at 60 s the healthy day still
|
|
produces zero trips, and at 300 s the fault day still produces 13.
|
|
|
|
⚠️ A BATTERY THAT REFUSES A COMMAND CANNOT TRIP THIS, and that is structural
|
|
rather than lucky. At SoC 10 % in Standby, a 1350 W discharge command into a
|
|
1350 W import is DECLINED: the battery does nothing, so the meter stays at
|
|
+1350 W and the command stays at +1350 W. Same sign. Agreement. A refusal leaves
|
|
the meter where it was; only the inverter ACTING moves it, and acting in the
|
|
commanded direction moves it toward zero, not through it. The failure mode this
|
|
check keys on is the opposite of a refusal - it is the inverter obediently doing
|
|
a wrong thing.
|
|
|
|
⚠️ A FROZEN METER cannot trip it either, for the same reason read the other way:
|
|
a stuck reading of +800 W makes the loop command discharge and keep commanding
|
|
discharge. Command and meter converge on AGREEING. Freezes are SAFETY-12's and
|
|
SAFETY-01's problem, and this watchdog neither helps nor hinders there.
|
|
|
|
⚠️ ALL OF THAT DEPENDS ON ONE PROPERTY OF THE CONTROL LAW, so it is an invariant
|
|
and not a happy accident: the saturation freeze in control.py must keep letting
|
|
the command FALL, CROSS ZERO AND REVERSE - "may not wind further in the direction
|
|
it is already pushing", not "only corrections that shrink |i_w|". control.py
|
|
records the shrink-only encoding as tried and rejected because it pins the command
|
|
at its last value indefinitely; a pinned command against a meter that has moved on
|
|
is precisely a sustained sign disagreement, so under that encoding this watchdog
|
|
WOULD false-trip on a plain load step. If anyone ever revisits that freeze, this
|
|
module's no-false-positive argument is downstream of the answer.
|
|
|
|
------------------------------------------------------------------------------
|
|
WHAT IT DOES NOT CATCH
|
|
|
|
An inverter that ACTS opposite to what it was told. Tell it to discharge 2000 W
|
|
and have it charge instead, and the meter moves FURTHER the way it was already
|
|
going - so the command and the meter agree, all the way into the rail. This
|
|
watchdog compares the loop's output direction against the input state it was
|
|
regulating; it is blind to the plant lying about which way it went. That is the
|
|
saturation freeze's question (|command - the battery's actual power| > 500 W,
|
|
NOTES.md §4.7 row 4), and the two checks are complementary rather than
|
|
overlapping. Neither substitutes for the other.
|
|
|
|
⚠️ THAT PARAGRAPH IS ABOUT THE COMMAND-VERSUS-METER COMPARISON, AND SINCE
|
|
SAFETY-30 THAT IS NOT THE ONLY ONE THIS MODULE DRIVES. At a roster of two or
|
|
more each unit is ALSO graded against its own measured power, which is exactly
|
|
the comparison the paragraph says nothing here makes - see the SAFETY-30 section
|
|
below, which says what that does and does not take away from the freeze. At the
|
|
shipped roster of one the paragraph stands unchanged, word for word: the site
|
|
comparison is the only detector armed and the freeze is still the whole answer.
|
|
|
|
Nor does it see a sign inversion introduced DOWNSTREAM of this add-on - in the
|
|
firmware, on the wire, or in a future driver. Our view stays self-consistent and
|
|
this stays silent. That is a wire-versus-intent problem and it belongs to the
|
|
witness, which is the only vantage point that sees both.
|
|
|
|
------------------------------------------------------------------------------
|
|
SAFETY-30 - AT TWO INVERTERS THIS IS TWO QUESTIONS, NOT ONE
|
|
|
|
Everything above was written at a roster of ONE, and at N=1 it is right for a
|
|
reason that stops being true at N=2. `commanded_w` is what THIS controller put
|
|
on THIS actuator. `grid_w` is what the WHOLE SITE is exchanging with the
|
|
network. At one inverter those are two views of one thing - §6.2's identity -
|
|
which is what makes comparing them sound. At two they are not:
|
|
|
|
* A FALSE TRIP THE COMPARISON CANNOT AVOID. "Inverter A commanded -1500 W
|
|
while the meter reads +1500 W" is evidence of nothing at N=2: inverter B may
|
|
be discharging 3000 W and doing exactly what it was told. No fault anywhere,
|
|
and the check NOTES.md calls the highest-value entry in the safety table -
|
|
93 % agreement while healthy against 4 % during runaway, four episodes, zero
|
|
false positives - fires.
|
|
* THE CONVERSE, WHICH IS WORSE. A unit genuinely driving the wrong way is
|
|
MASKED, because the site figure NETS. Commanded +1500 W, measured -1500 W,
|
|
beside a sibling measured at +1500 W: the meter sits exactly where a healthy
|
|
site leaves it. This is not a threshold that is set too high - the
|
|
information is destroyed by the summation, and no tuning recovers it.
|
|
|
|
So the one comparison becomes two, each asked of the evidence that can answer
|
|
it, and they are DIFFERENT FAULTS rather than one fault at two scales:
|
|
|
|
PER UNIT this unit's standing command against ITS OWN measured power -
|
|
`Inverter.batt`, which on the live system is holding register
|
|
0x0518, this inverter's own AC power, per-instance since DEV-02.
|
|
Conclusion: "THE ACTUATOR I AM DRIVING IS NOT DOING WHAT I TOLD
|
|
IT". The evidence is about one unit, so the remedy is that unit:
|
|
the `failsafe` SET, which arbiter.CLAIM_SCOPE has called a UNIT
|
|
claim from the day it was written, for this exact reason.
|
|
SITE the SUM of the standing commands against the meter. Conclusion:
|
|
"THE SITE AS A WHOLE IS DRIVING THE WRONG WAY" - an inverted gain,
|
|
an integrator wound past the rail, a strategy pushing the wrong
|
|
side of the meter. That is the fault the 2026-08-20 capture
|
|
contains and it is a property of the CONNECTION, so the remedy is
|
|
the connection: a SITE SET that stops every inverter.
|
|
|
|
⚠️ AND AT N=1 THERE IS EXACTLY ONE ARMED DETECTOR, WHICH IS THE ONE THAT
|
|
SHIPPED. The two questions above are the SAME SENTENCE at a roster of one: the
|
|
sum of one command is that command, and "the site" and "the unit" name one
|
|
actuator. Arming both would be arming a duplicate - two latches, two banners and
|
|
two chances to spend a false-positive record that was earned by ONE comparison
|
|
on two captured bus days. `Controller.cycle()` therefore hands the single unit's
|
|
watchdog THE METER, under the shipped `sign_disagree_w` / `sign_disagree_s`,
|
|
exactly as before, and never arms the site one. `arm()` below is where that
|
|
choice is written down. Nothing about a one-inverter install changes - not the
|
|
comparison, not the thresholds, not the store keys, not the claim, not the
|
|
banner.
|
|
|
|
⚠️ WHAT THE PER-UNIT COMPARISON IS NOT. Two guards that overlap without saying
|
|
so is how a false-alarm generator gets built, so both neighbours are named:
|
|
|
|
* IT IS NOT SAFETY-31. `circulation.py` compares the units AGAINST EACH OTHER
|
|
- `sum(|B_i|) - |sum(B_i)|` over MEASURED power - and catches two inverters
|
|
pushing THROUGH each other, which nets to zero at the meter and about which
|
|
no single command is evidence at all. This compares each unit against ITS
|
|
OWN COMMAND. Take two units faithfully executing an allocation that ALLOC-01
|
|
would refuse: circulating power is the whole loop, and every per-unit sign
|
|
check here reads perfect agreement. Take one unit doing the exact opposite
|
|
of its share while its sibling covers for it: this reads the whole
|
|
disagreement, and circulating power can be zero, because the two measured
|
|
powers may well agree in sign. Neither is the other with a different
|
|
threshold.
|
|
* IT IS NOT THE SATURATION FREEZE, AND THE OVERLAP IS REAL AND BOUNDED. The
|
|
freeze asks `|command - measured| > saturation_w` (500 W) and concludes "the
|
|
inverter cannot follow"; it then stops the integrator winding, which is a
|
|
control action taken every day, on a unit that is merely SATURATED as
|
|
readily as on one that is reversed, and it neither latches nor parks
|
|
anything. This asks for OPPOSITION - a sign, on both sides of a threshold,
|
|
held continuously - which a saturated-but-obedient unit cannot produce at
|
|
all, and it LATCHES and commands that unit to zero. The freeze is a
|
|
magnitude test that shapes the loop; this is a direction test that stops an
|
|
actuator.
|
|
|
|
⚠️ GATE 1 STILL APPLIES TO BOTH, AND THAT IS A DELIBERATE CHOICE WITH A COST.
|
|
The per-unit comparison could have been armed whenever ANY non-zero command is
|
|
standing - a unit should do what it is told whoever told it - and that would
|
|
catch a wedged inverter during a maintenance charge. It is not, because the
|
|
whole no-false-positive argument above is built on `following`, because the
|
|
freeze already grades that window, and because this card's own warning is that
|
|
SAFETY-02's record was earned at N=1 and must not be spent on a check that fires
|
|
when nothing is wrong. What it costs is written here rather than discovered:
|
|
between the start of a maintenance cycle and its end, a unit reversing against
|
|
its own command is the freeze's to notice, not this module's. Widening it is a
|
|
card, with a rig campaign behind it, not an edit.
|
|
|
|
------------------------------------------------------------------------------
|
|
STATE, TRIP AND LATCH
|
|
|
|
One timestamp, not a sample counter. The cycle rate is not fixed - a meter
|
|
arrival, or `cycle_floor_s` elapsing, whichever comes first - so N samples is a
|
|
different wall-clock window on every path, and the AC is written in minutes.
|
|
|
|
The latch is not optional and not a nicety. Commanding 0 W removes the command
|
|
side of the comparison (|0| is not > 300), so the disagreement disappears the
|
|
instant the trip works. Un-latched, this watchdog would trip, clear itself, and
|
|
hand the actuator straight back to the loop that was driving the wrong way -
|
|
three minutes later, forever. `tripped` therefore stays true until an operator
|
|
`reset()`s it by starting control again, which is a deliberate human act after
|
|
looking at why.
|
|
|
|
⚠️ AND THE LATCH HAS TO OUTLIVE THE PROCESS. Held only in memory it is not a
|
|
latch at all - a container restart clears it and the add-on comes back armed,
|
|
into the same fault, unattended. Worse than merely losing state: "restart the
|
|
add-on" is the first thing an operator confronted with a FAILSAFE banner and a
|
|
0 W inverter will try, and under a supervisor that restarts a crashed add-on it
|
|
is a loop of three-minute wrong-direction bursts. `tripped` and `reason` are
|
|
therefore constructor arguments, restored from `Store` in `Controller.__init__`
|
|
and written back the moment the trip happens. See store.py.
|
|
"""
|
|
|
|
from dataclasses import dataclass
|
|
from datetime import datetime
|
|
|
|
|
|
# SAFETY-30. CFG-03: THE ONE IN-CODE COPY OF THE FOUR MULTI-INVERTER DEFAULTS.
|
|
# `config.yaml`'s `options:` block is the source of truth an operator edits and
|
|
# the Supervisor reads, and `schema:` holds the bounds. This dict exists for the
|
|
# reason `circulation.CIRC_OPTION_DEFAULTS` and `maintenance.MAINT_OPTION_DEFAULTS`
|
|
# do - the Dockerfile does not put config.yaml in the image, so the running
|
|
# add-on cannot read it - and `test_watchdog.py` pins the two together by READING
|
|
# config.yaml, so a drift goes red in CI rather than shipping.
|
|
#
|
|
# ⚠️ EVERY KEY HERE IS DEAD AT A ROSTER OF ONE, and that is the point rather
|
|
# than a caveat. `Controller._arm_sign_watchdogs()` arms the single unit's
|
|
# detector against the meter with the SHIPPED `sign_disagree_w` /
|
|
# `sign_disagree_s`, which are read in main.py exactly where they always were
|
|
# and are not in this dict. Nothing below is consulted until a second inverter
|
|
# exists.
|
|
SIGN_OPTION_DEFAULTS = {
|
|
# --- THE MAGNITUDE GATE, AND IT IS NOT SAFETY-02's 300 W BY ACCIDENT ---
|
|
# ⚠️ DERIVED FROM `control.Tuning.saturation_w`, WHICH IS 500 W, AND THE
|
|
# DERIVATION IS THE POINT. This check gates on BOTH sides being over
|
|
# `min_w`, so the smallest divergence it can ever see between a command and
|
|
# the power that command produced is `2 * min_w` - the two are OPPOSED, so
|
|
# the gap is the sum of the magnitudes. At 250 W that is exactly 500 W: the
|
|
# gap at which `compute()`'s own anti-windup test already concludes THE
|
|
# INVERTER IS NOT FOLLOWING and freezes the integrator. So this watchdog
|
|
# cannot fire on a divergence the shipped control law itself calls ordinary
|
|
# tracking, and the two guards meet at a number rather than overlapping in a
|
|
# band nobody has drawn.
|
|
#
|
|
# MEASURED, on 1890 closed-loop two-inverter runs in which NOTHING is faulty
|
|
# (every unit executes the share the allocator gave it), swept over the
|
|
# WHOLE range config.yaml accepts for `gain` float(0.05,1.0) and `slew_w`
|
|
# int(50,5000), across both families of stimulus that can produce legitimate
|
|
# opposition. Worst CONTINUOUS opposition between a unit's standing command
|
|
# and its own 0x0518 power, as a function of this threshold:
|
|
#
|
|
# 50 W -> 25.0 s 250 W -> 10.0 s 700 W -> 5.0 s
|
|
# 150 W -> 15.0 s 300 W -> 10.0 s 1000 W -> 0.0 s
|
|
# 200 W -> 15.0 s 500 W -> 5.0 s
|
|
#
|
|
# ⚠️ SAFETY-39 MOVED THAT SCHEMA FLOOR FROM 50 TO 0, AND THE CAMPAIGN ABOVE
|
|
# WAS NOT RE-RUN. It does not have to be, and here is the argument rather
|
|
# than the assurance. What these windows are about is how long the command
|
|
# takes to CROSS ITS RAIL - the swept worst case is the SLOWEST crossing,
|
|
# which is `slew_w` at its floor, 5000/50 = 100 cycles. `slew_w: 0` means
|
|
# "follow the rail" and resolves to `SLEW_RAIL_FRAC * max_w`, which is
|
|
# exactly 9 cycles at EVERY rail by construction. So the new value can never
|
|
# be the slowest crossing at any `max_w`, and it is strictly inside the
|
|
# envelope this campaign measured. An explicit number still spans 50..5000
|
|
# and is unchanged. If `SLEW_RAIL_FRAC` is ever loosened past 1/100, this
|
|
# paragraph stops being true and the campaign has to be re-run.
|
|
#
|
|
# ⚠️ AND THE CURVE IS FLAT, WHICH IS THE FINDING THAT DECIDED THE SHAPE OF
|
|
# THIS PAIR. Unlike the site comparison - where the meter is driven by the
|
|
# LOAD and a flapping load holds it opposed for minutes - a unit's own power
|
|
# CHASES ITS OWN COMMAND through a bounded ramp and a first-order lag, so the
|
|
# opposition can only last as long as the crossing does. Dropping the
|
|
# threshold from 300 W to 50 W buys 15 s of extra transient and nothing else.
|
|
# The work is therefore done by the WINDOW below, and this number is set
|
|
# where the control law already draws its line rather than being tuned.
|
|
#
|
|
# The largest opposed magnitude seen anywhere in those 1890 runs is 875 W;
|
|
# a threshold above that would gate NOTHING legitimate in at all, and is
|
|
# rejected because it would also be blind to a unit reversing a 900 W share.
|
|
"unit_sign_disagree_w": 250,
|
|
|
|
# --- THE WINDOW -------------------------------------------------------
|
|
# Three bounds, and the value clears all three.
|
|
#
|
|
# LOWER, from the rig: 10.0 s - two control cycles, the worst CONTINUOUS
|
|
# legitimate opposition above 250 W across all 1890 runs. It is a
|
|
# reversing site whose second write lands a cycle after the first, both
|
|
# units then slewing through zero behind a 10 s lag; nothing is wrong.
|
|
#
|
|
# LOWER, from the firmware: `inverter.LINK_STALE_S`, 30 s. A unit not
|
|
# executing its share for less than the firmware's own veto window is a
|
|
# LINK fault, and the add-on already answers it - liveness drops the unit
|
|
# and the survivors carry its share, with the record saying so. Firing
|
|
# inside that window would raise a second, louder alarm about a state
|
|
# already being handled, which is the SAFETY-15/SAFETY-19 failure exactly.
|
|
#
|
|
# UPPER, from SAFETY-02 itself: 180 s, the SITE hold. A unit that is not
|
|
# doing what it was told must not be parked LATER than the site would be
|
|
# parked for the same watts, and unlike the site comparison this one needs
|
|
# nothing to net out before it can conclude - the evidence is complete on
|
|
# the first cycle it appears.
|
|
#
|
|
# 85 s: seventeen control cycles - 8.5x the worst legitimate episode this
|
|
# rig can produce, 2.8x the firmware's own veto window, and under HALF the
|
|
# site hold. ⚠️ IT IS DELIBERATELY NOT 60, 75, 90, 120 OR 180. A round
|
|
# number here would be a guess wearing a measurement's clothes, and the
|
|
# three bounds above are what the number has to answer to.
|
|
#
|
|
# ⚠️ AND INVERTER B DOES NOT PHYSICALLY EXIST. Its ramp and lag in the rig
|
|
# are inverter A's, COPIED, carrying `measured=False` - SPIKE-04 measures
|
|
# the real ones. A slower second unit lengthens every crossing above, which
|
|
# is what the 8.5x buys headroom for. SPIKE-04 is the revisit trigger for
|
|
# both numbers here, not a reason to move them now.
|
|
"unit_sign_disagree_s": 85,
|
|
|
|
# --- THE SITE COMPARISON AT N>=2, AND IT IS NOT THE SHIPPED PAIR -------
|
|
# ⚠️ THE THRESHOLD IS CARRIED AND THE WINDOW IS NOT, WHICH IS A FINDING
|
|
# RATHER THAN A COMPROMISE. Only one of the two arguments behind 300 W /
|
|
# 180 s depends on how many inverters the site has:
|
|
#
|
|
# THE THRESHOLD is about NOISE. "Near zero the sign is noise, and near
|
|
# zero is where a healthy loop LIVES" - the deadband holds the command
|
|
# still while the meter wanders either side of `target_grid_w`. The
|
|
# deadband is 15 W and P1 is one meter whatever the roster, so nothing
|
|
# about that argument moves. 300 W, carried - and the two-inverter
|
|
# campaign says the carry is safe rather than merely convenient: in the
|
|
# load-reversal family the site figure ever goes opposed AND over 300 W in
|
|
# 994 of 1050 runs at two inverters and 955 of 1050 at one. The gate opens
|
|
# about as often either way; what changed is how LONG it stays open, which
|
|
# is the window's problem and not the threshold's.
|
|
#
|
|
# THE WINDOW is about a TRANSIENT, and the transient scales with the site.
|
|
# The measured worst legitimate case is a load reversal that the loop
|
|
# crosses zero slowly: the command has to travel the whole rail at
|
|
# `slew_w` per cycle, and at N>=2 the rail is `Controller.site_max_w` - the
|
|
# ROSTER SUM - while `slew_w` is unchanged. So the crossing takes about
|
|
# twice as long at two inverters as at one, and that is exactly what the
|
|
# rig measures.
|
|
#
|
|
# 1890 closed-loop runs in which nothing is faulty, the SAME campaign the
|
|
# per-unit pair above is read off, run at BOTH rosters as its own control.
|
|
# Worst CONTINUOUS opposition between the standing command(s) and the meter,
|
|
# LOAD-REVERSAL family, at a 300 W threshold:
|
|
#
|
|
# N=1 (the shipped comparison) 110.0 s - inside 180 s, 1.6x
|
|
# N=2 (the sum, same stimuli) 215.0 s - OVER 180 s
|
|
#
|
|
# ⚠️ SO CARRYING 180 s WOULD HAVE SHIPPED A FALSE TRIP, and it is a config a
|
|
# schema-legal install can reach: `gain` 0.05 and `slew_w` 50 are both the
|
|
# floor of what config.yaml accepts, and a +-2200 W load reversal is an oven
|
|
# and a heat pump. That is the number this card exists to not assume.
|
|
#
|
|
# 325 s: 1.51x the measured worst, well inside the 900 s quarter that bills
|
|
# the capacity tariff, and still inside every episode the fault day
|
|
# contains - 2026-08-20's four clusters run 16, 30, 77 and 77 minutes and
|
|
# its opposed runs reach 1750 s, so a 325 s window fires in all of them.
|
|
# What it costs is stated rather than hidden: about two and a half extra
|
|
# minutes of wrong-direction SITE power before the stop, bought to remove a
|
|
# false trip that a legal configuration produces. ⚠️ DELIBERATELY NOT 300 OR
|
|
# 360; the measurement is what it answers to.
|
|
#
|
|
# ⚠️ AND IT IS NOT THE PER-UNIT WINDOW EITHER, IN THE OTHER DIRECTION. The
|
|
# per-unit comparison is 85 s because a unit's own power CHASES ITS OWN
|
|
# COMMAND and the transient is two cycles; the site comparison is 325 s
|
|
# because the meter is driven by the LOAD and the transient is the loop
|
|
# crossing zero. Same predicate, two instruments, two windows, both measured.
|
|
"site_sign_disagree_w": 300,
|
|
"site_sign_disagree_s": 325,
|
|
}
|
|
|
|
# ⚠️ A LIMIT OF THE SITE COMPARISON THAT SAFETY-30 DID NOT INTRODUCE AND DOES
|
|
# NOT FIX, RECORDED HERE BECAUSE THE CAMPAIGN FOUND IT AND SILENCE WOULD BE
|
|
# WORSE. The rig's other stimulus family is a FLAPPING LOAD, and a square wave
|
|
# whose half period is about one control cycle opposes the command on EVERY
|
|
# cycle: the command chases the previous half-cycle's meter while the meter has
|
|
# already reversed. Across the 840 flapping runs the worst continuous opposition
|
|
# at a 300 W threshold is 980 s at two inverters - and 1000 s at ONE, which is
|
|
# the whole point. The N=1 arm is the control, and it says this is a property of
|
|
# comparing a lagged command against a meter, not a property of the second
|
|
# inverter. No window inside the 900 s billing quarter survives it, and no
|
|
# threshold up to 1000 W does either.
|
|
#
|
|
# It is not acted on here for three reasons, in order of weight: it is identical
|
|
# at the roster this add-on actually ships, so acting on it would be changing a
|
|
# live safety path on evidence that has nothing to do with this card; a
|
|
# ~6 kW square wave at 0.1 Hz sustained for a quarter of an hour is not a house
|
|
# (the captured healthy day's longest opposed run is 30 s); and the fix is not a
|
|
# number - it wants the detector to notice that the disagreement is ALTERNATING,
|
|
# which is a different check. It is a card, and this comment is its evidence.
|
|
|
|
# SAFETY-30. THE TWO NOUNS THE TRIP SENTENCE CAN NAME, WRITTEN ONCE. They are
|
|
# constants rather than literals at the call sites because `main.py` arms the
|
|
# detectors and `test_watchdog.py` grades what they say, and a string typed in
|
|
# two places is a string that drifts in one of them - the same argument
|
|
# `store.unit_key()` makes about a key name. `AGAINST_METER` is the wording that
|
|
# has always shipped and must not change: it is what a one-inverter install's
|
|
# `reason` reads, in the store, in the action log and on the status page.
|
|
AGAINST_METER = "a meter reading"
|
|
AGAINST_OWN_POWER = "its own measured power"
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class SignWatchdogConfig:
|
|
"""Defaults, not constants - config.yaml carries both (SAFETY-02 AC 5)."""
|
|
|
|
min_w: float = 300.0
|
|
hold_s: float = 180.0
|
|
|
|
|
|
def disagrees(commanded_w, against_w, following: bool,
|
|
cfg: SignWatchdogConfig) -> bool:
|
|
"""The predicate, and the ONLY place it is written down.
|
|
|
|
⚠️ Public because `sim/replay_sign.py` needs it too - to describe episodes, to
|
|
measure agreement, to sweep the clock-skew guard - and it had reimplemented it
|
|
inline. Two copies of a safety predicate is one copy plus a future
|
|
disagreement about what the check means, and the copy that drifts is always
|
|
the one in the harness that grades it.
|
|
|
|
⚠️ `against_w` WAS CALLED `grid_w`, AND THE RENAME IS THE WHOLE OF SAFETY-30
|
|
IN ONE PARAMETER. The arithmetic is unchanged - opposed signs, neither side
|
|
near zero, and the loop actually driving - but WHAT the command is opposed
|
|
TO is now a property of the caller rather than a fact about this function.
|
|
The site comparison hands it the meter; each unit's own comparison hands it
|
|
that unit's measured power (register 0x0518). One predicate, two callers,
|
|
for `loop_is_following()`'s reason: a safety predicate written down twice is
|
|
one copy plus a future disagreement about what the check means. The
|
|
parameter is positional at every call site in both repositories, so the
|
|
rename moves no caller.
|
|
"""
|
|
return (following
|
|
and commanded_w is not None and against_w is not None
|
|
and commanded_w * against_w < 0
|
|
and abs(commanded_w) > cfg.min_w
|
|
and abs(against_w) > cfg.min_w)
|
|
|
|
|
|
def loop_is_following(resolution, loop_asked) -> bool:
|
|
"""Did the grid-following loop actually choose the number going on the wire?
|
|
|
|
Gate 1 of the predicate above, and it lives here rather than inline in
|
|
`cycle()` for the reason `disagrees()` does: the test suite has to be able to
|
|
assert on THE SHIPPED EXPRESSION, not on a transcription of it that drifts
|
|
apart from it in silence.
|
|
|
|
⚠️ IT WAS A TRANSCRIPTION, AND THAT WAS MEASURED, NOT FEARED. The P_PEAK case
|
|
in test_watchdog.py exists to make whoever lands peak shaving come here and
|
|
change an expectation. With the expression copied into the test it did not do
|
|
that: editing the shipped gate alone to `winner in ("loop", "peak")` left every
|
|
suite green while the watchdog went blind for the whole of every peak event.
|
|
One definition, two callers, and that test now bites.
|
|
|
|
`resolution` the arbiter's Resolution. Duck-typed on `.winner` and
|
|
`.target_w` so this module still imports nothing from arbiter.
|
|
`loop_asked` what the grid-following loop claimed this cycle, or None if it
|
|
did not claim at all.
|
|
|
|
"The loop won" is NOT the question, and the difference is a whole class of
|
|
false trip. A LIMIT claim can survive the loop winning and still replace its
|
|
number: maintenance's cheap-window shaping is `hi = -maintenance_charge_w`, so
|
|
at 1500 W the loop can ask for +500 W of discharge, win the arbitration, and
|
|
have the emitted value come out at -1500 W of grid charge. Winner says "loop";
|
|
the wire carries maintenance's intent.
|
|
|
|
So the test is on the NUMBER: the loop is following only while the emitted
|
|
value points the way the loop asked AND is no larger than it asked - limits
|
|
reduced it, they did not redirect it. The magnitude half is load-bearing on
|
|
its own, because a `lo` limit can raise magnitude without flipping sign, and
|
|
that is not the loop following either. Zero on either side falls out as
|
|
not-following, which costs nothing: a 0 W command cannot be a
|
|
wrong-direction command.
|
|
|
|
⚠️ FORWARD HAZARD, and the claim block in cycle() already anticipates it.
|
|
`winner == "loop"` disarms the watchdog under ANY higher-priority SET claim.
|
|
Today only maintenance and the safety stops emit one and neither follows the
|
|
meter, so the gate costs nothing. A peak-shaving strategy claiming SET at
|
|
P_PEAK **is** grid-following in sign, and the moment it lands this goes blind
|
|
for the duration of every peak event. Widening it means asking "does the
|
|
winning strategy follow the meter", not adding another name to a comparison.
|
|
"""
|
|
return (resolution.winner == "loop" and loop_asked is not None
|
|
and resolution.target_w * loop_asked > 0
|
|
and abs(resolution.target_w) <= abs(loop_asked))
|
|
|
|
|
|
class SignWatchdog:
|
|
"""One instance per COMPARISON, evaluated once per control cycle.
|
|
|
|
⚠️ SINCE SAFETY-30 THAT IS NOT THE SAME SENTENCE AS "one per actuator",
|
|
which is what this docstring used to say. There is one instance per
|
|
inverter - `Inverter.sign_wd`, latched under that unit's own store keys -
|
|
AND one for the site, `Controller.site_sign_wd`, latched under flat ones.
|
|
The two ask different questions of different evidence and reach different
|
|
remedies; `arm()` below is where an instance is told which of them it is.
|
|
At a roster of one they would be the same question, so only the per-unit
|
|
instance is ever armed there - see the SAFETY-30 section of the module
|
|
docstring.
|
|
|
|
⚠️ NOT SHARED ACROSS CADENCES. `disagree_since` is a wall-clock start time,
|
|
so driving one instance from two loops at two rates would have each one
|
|
resetting the other's window. If a fast on-device path ever wants this
|
|
check, it gets its own instance - the same exactly-one-writer discipline the
|
|
arbiter applies to the actuator, applied to the detector's own state.
|
|
"""
|
|
|
|
def __init__(self, cfg: SignWatchdogConfig | None = None, *,
|
|
tripped: bool = False, reason: str = "",
|
|
against: str = AGAINST_METER):
|
|
self.cfg = cfg or SignWatchdogConfig()
|
|
# SAFETY-30. WHAT THE COMMAND IS BEING COMPARED AGAINST, as the operator
|
|
# reads it in the trip sentence. The default is the meter, which is what
|
|
# every trip this add-on has ever taken was compared against and is what
|
|
# keeps a one-inverter install's `reason` string byte-identical.
|
|
self.against = str(against)
|
|
self.disagree_since: datetime | None = None
|
|
# ⚠️ RESTORED FROM THE STORE, NOT DEFAULTED. A latch that lives only in
|
|
# memory is not a latch: a container restart clears it, and the add-on
|
|
# comes back armed, into the same fault, with no operator involved. See
|
|
# store.py's note on `sign_tripped` for the measured behaviour.
|
|
self.tripped = bool(tripped)
|
|
self.reason = reason if self.tripped else ""
|
|
|
|
# -- which of SAFETY-30's two questions this instance is asking ----------
|
|
def arm(self, cfg: SignWatchdogConfig, against: str) -> None:
|
|
"""Set the thresholds and the subject of the comparison. Nothing else.
|
|
|
|
⚠️ IT IS CONFIGURATION, NOT STATE, AND IT DELIBERATELY DOES NOT TOUCH
|
|
`disagree_since`, `tripped` OR `reason`. A detector that cleared its own
|
|
window when it was told what it was comparing against would lose the
|
|
episode it is in the middle of, every cycle, and would never accumulate
|
|
a hold at all - the check would be permanently green and nothing would
|
|
say so.
|
|
|
|
⚠️ AND IT IS RE-STATED EVERY CYCLE RATHER THAN FROZEN AT CONSTRUCTION.
|
|
The answer is a property of the ROSTER and of nothing else, and
|
|
`Controller.cycle()` already re-reads the roster every cycle for
|
|
liveness and for the allocation. Freezing it in `__init__` would also
|
|
make it wrong for any caller that appends an inverter afterwards, which
|
|
is exactly how a two-inverter fixture is built in this repository - and
|
|
a fixture that silently got the one-inverter comparison would grade
|
|
nothing while looking green.
|
|
"""
|
|
self.cfg = cfg
|
|
self.against = str(against)
|
|
|
|
# -- the whole check, and it is a pure function of its arguments ----------
|
|
def evaluate(self, commanded_w, against_w, following: bool,
|
|
now: datetime) -> bool:
|
|
"""Advance the detector one cycle. Returns True while tripped.
|
|
|
|
`commanded_w` the EMITTED setpoint - what is standing on the wire now,
|
|
after deadband, slew, quantisation and the arbiter. Not
|
|
the optimizer's raw wish: the wire carries the emitted
|
|
value, so anything else can disagree with the meter for a
|
|
reason that is not a fault. For a per-unit instance it is
|
|
THAT UNIT'S share of it; for the site instance it is the
|
|
roster sum.
|
|
`against_w` what the command is graded against, and SAFETY-30 made
|
|
this a decision rather than a fact: net P1 for the SITE
|
|
comparison (+ = importing), and this unit's own measured
|
|
power for the PER-UNIT one (+ = discharging, register
|
|
0x0518). Both carry the same sign convention as the
|
|
command, which is why one predicate serves both. None
|
|
when the reading is stale or missing, which CLEARS the
|
|
timer rather than accumulating - a frozen figure held
|
|
against a moving command would otherwise manufacture a
|
|
disagreement in about three minutes, on top of SAFETY-01
|
|
having already commanded zero for the same cause.
|
|
`following` did the grid-following loop actually win this cycle? See
|
|
gate 1 in the module docstring, and the SAFETY-30 section
|
|
on why the per-unit comparison keeps it.
|
|
`now` the caller's clock, so this stays testable and replayable.
|
|
"""
|
|
if self.tripped:
|
|
# Latched. Do not re-evaluate: the trip removed the evidence.
|
|
return True
|
|
|
|
cfg = self.cfg
|
|
if not disagrees(commanded_w, against_w, following, cfg):
|
|
self.disagree_since = None
|
|
return False
|
|
|
|
if self.disagree_since is None:
|
|
self.disagree_since = now
|
|
return False
|
|
|
|
held = (now - self.disagree_since).total_seconds()
|
|
if held < cfg.hold_s:
|
|
return False
|
|
|
|
self.tripped = True
|
|
# ⚠️ `self.against` IS THE ONLY THING SAFETY-30 CHANGED IN THIS STRING,
|
|
# and it defaults to the wording that shipped - "a meter reading" - so a
|
|
# one-inverter install's `reason` is byte-identical to what an operator
|
|
# (and test_watchdog.py) has always read. A per-unit instance names what
|
|
# it actually compared instead, because "commanded +1500 W against a
|
|
# meter reading -1500 W" would be a false statement about which
|
|
# instrument produced the evidence, on the one line somebody woken by
|
|
# this alarm reads first.
|
|
self.reason = (
|
|
f"SIGN_DISAGREEMENT: commanded {commanded_w:+.0f} W against "
|
|
f"{self.against} {against_w:+.0f} W for {held:.0f} s continuously "
|
|
f"(limits: both over {cfg.min_w:.0f} W, held over {cfg.hold_s:.0f} s)"
|
|
)
|
|
return True
|
|
|
|
def held_s(self, now: datetime) -> float:
|
|
"""How long the current disagreement has been standing, for the UI.
|
|
|
|
⚠️ Takes the clock rather than reading one. It was a property calling
|
|
`datetime.now()`, which quietly broke the property this module's header
|
|
advertises - every other observable here is a pure function of what it
|
|
was handed - and an unclocked function is one nothing can test, which is
|
|
exactly what happened to it.
|
|
"""
|
|
if self.disagree_since is None:
|
|
return 0.0
|
|
return (now - self.disagree_since).total_seconds()
|
|
|
|
def reset(self) -> None:
|
|
self.disagree_since = None
|
|
self.tripped = False
|
|
self.reason = ""
|