admin/goodwe-addon
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SAFETY-02: a loop can be fresh, tracking and inside its rating and still push the wrong way

Browse Source 
The stale-input watchdog asks whether the meter is still talking. The saturation
freeze asks whether the inverter can follow. The output clamp asks whether the
number is too big. A loop can pass all three while charging the battery out of
the grid, because the only thing wrong is the sign.

On 2026-08-20 a commercial controller did exactly that four times in one day,
for up to 77 minutes at a stretch, reaching a commanded -14 768 W against a
±5000 W inverter. app/watchdog.py compares the emitted setpoint against the
authoritative P1 figure and, when they oppose each other with both over 300 W
for 180 s continuously, files a failsafe SET claim at P_SAFETY, commands 0 W and
latches there until an operator starts control again.

Validated against that day rather than against an argument: sim/replay_sign.py
in the infra repo drives this module over the committed captures of both
2026-08-20 and the healthy 2026-08-23 and reaches 18 trips on the fault day -
every one inside a window NOTES.md §4.7 accounts for - and 0 on the healthy day.
The four episodes do share one signature: 0.3 / 0.9 / 10.5 / 0.0 % sign
agreement inside them against 92.9 % healthy.

Three decisions the ticket could not make:

- It is DISARMED whenever the grid-following loop is not what is driving. Sign
  agreement is a property of a loop that follows the meter, and a maintenance
  grid charge opposes the meter by design - 2500 W of charge into a house
  drawing 500 W is +3000 W on the meter against a -2500 W command, for hours.
  The gate tests the emitted number, not the winning claim's name, because a
  limit claim can let the loop "win" and still replace its value.
- The trip LATCHES. Commanding 0 W removes the command side of the comparison,
  so the disagreement always clears the moment the trip works.
- sign_disagree_w at or above max_w is refused at startup. The arbiter never
  lets a command exceed max_w, so such a threshold leaves the watchdog incapable
  of firing while the status page still reports it armed.

What it does not catch, said out loud: an inverter that ACTS opposite to its
command moves the meter further the same way, leaving command and meter in
agreement. That is the saturation freeze's job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du77usMj8XNKNFZGmUiWDa
This commit is contained in:
glenn schrooyen
2026-08-28 21:51:35 +02:00
co-authored by Claude Opus 5
parent 35856a73ae
commit 9aed43990b
9 changed files with 755 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
goodwe_controller/app/watchdog.py
+220
View File
@@ -0,0 +1,220 @@
"""SAFETY-02 - the sign-disagreement watchdog.
WHAT IT CATCHES, in one sentence: a grid-following loop that is pushing power in
the wrong direction, which every other guard in this system is blind to.
The stale-input watchdog (SAFETY-01) asks "is the meter still talking?". The
saturation freeze asks "can the inverter follow?". The output clamp asks "is the
number too big?". A loop can pass all three and still be charging the battery
out of the grid while the house imports - the meter is fresh, the inverter is
tracking beautifully, and the magnitude is inside the rating. The only thing
that separates that from healthy operation is the SIGN.
⚠️ THIS IS NOT A HYPOTHETICAL. On 2026-08-20 the vendor controller (Illusmart)
did exactly this four times in one day (NOTES.md §4.7). Both captures of that
bus are committed - `sim/scenarios/bus-2026-08-20.log.gz` and the healthy
`bus-2026-08-23.log.gz` in the infra repo - and `sim/replay_sign.py` drives this
module over both of them. Every number in this file was measured on those two
days, not chosen:
day samples gated in sign agreement
2026-08-23 4 127 / 16 349 92.9 % (healthy)
2026-08-20 6 892 / 15 598 55.6 % (the whole fault day)
episode A 15:26-16:43 0.3 %
episode B 17:11-18:28 0.8 %
episode C 20:01-20:31 11.0 %
episode D 20:55-21:11 0.0 %
That is the signature, and it is the SAME signature in all four episodes: not a
noisier version of healthy, a near-total inversion.
------------------------------------------------------------------------------
WHY IT LIVES HERE AND NOT IN THE FIRMWARE OR THE WITNESS
Three vantage points see three different things. The firmware knows what it
WROTE, the witness Pi knows what was ON THE WIRE and whether the inverter ACKed
it, and the add-on knows what it INTENDED and holds the authoritative meter.
This check is not command-versus-wire; it is command-versus-physics. It needs
the emitted setpoint and the P1 net figure in the same place, and it needs the
authority to command zero when they fight:
- the witness is read-only by charter and cannot command anything, so it could
raise the alarm but never take the trip action the AC requires;
- the firmware has the wire but not the meter. §5.1 makes P1 the only
authoritative grid figure and the board only ever receives its AGE
(`sensor.goodwe_p1_sample_age`), never its value. Putting the check there
would mean shipping the meter to the ESP32 for no other reason, and it would
be C++ per install rather than one generic guard;
- the add-on has both halves already, plus an arbiter that can outrank every
strategy with a P_SAFETY claim.
The cost of standing here is real and worth stating: this watchdog compares what
we MEANT to put on the wire against the meter. If the firmware or the driver
inverts the sign on the way out, the add-on's own view still agrees with itself
and this check stays silent. That gap is a wire-versus-intent problem and it
belongs to the witness, not here.
------------------------------------------------------------------------------
WHAT "DISAGREEMENT" MEANS, PRECISELY
disagree = following
and commanded_w * grid_w < 0
and abs(commanded_w) > min_w
and abs(grid_w) > min_w
Four gates, and each one exists to exclude a specific thing that is NOT a fault:
1. `following`. ⚠️ THE GATE THAT IS EASIEST TO FORGET AND MOST EXPENSIVE TO
OMIT. Sign agreement is a property of a GRID-FOLLOWING loop. It is not
expected of a strategy that deliberately commands against the meter -
maintenance is exactly that. A maintenance grid charge of 2500 W into a house
drawing 500 W puts +3000 W on the meter against a -2500 W command: opposed,
both far over any sane threshold, and sustained for HOURS. Armed
unconditionally this watchdog would trip on every single maintenance cycle,
which is not a tuning problem, it is the wrong question being asked. So the
caller says whether the grid-following loop actually won the arbitration this
cycle, and the timer only accumulates when it did.
2. The strict `< 0` product. "Opposed" and "neither side is zero" in one
expression.
3. The 300 W magnitude gate ON BOTH SIDES. Near zero the sign is noise, and near
zero is where a healthy loop LIVES - the deadband holds the command still
while the meter wanders either side of `target_grid_w`. On the healthy day
only 4 127 of 16 349 samples (25 %) are even gated in; the other 75 % are the
converged band where sign carries no information. This is also why the
deadband cannot produce a false trip: inside it the meter is by definition
under 300 W.
4. The 180 s continuity requirement. This is what excludes the legitimate
transient. A load step reverses the meter's sign instantly while the command
is still where the last cycle left it; slew limiting then needs a few cycles
to cross over. Measured on the healthy day, the LONGEST such crossing is 30 s
- and that is with a vendor controller far cruder than ours. 180 s is six
times that. It is not a fitted threshold: at 60 s the healthy day still
produces zero trips, and at 300 s the fault day still produces 13.
⚠️ A BATTERY THAT REFUSES A COMMAND CANNOT TRIP THIS, and that is structural
rather than lucky. At SoC 10 % in Standby, a 1350 W discharge command into a
1350 W import is DECLINED: the battery does nothing, so the meter stays at
+1350 W and the command stays at +1350 W. Same sign. Agreement. A refusal leaves
the meter where it was; only the inverter ACTING moves it, and acting in the
commanded direction moves it toward zero, not through it. The failure mode this
check keys on is the opposite of a refusal - it is the inverter obediently doing
a wrong thing.
⚠️ A FROZEN METER cannot trip it either, for the same reason read the other way:
a stuck reading of +800 W makes the loop command discharge and keep commanding
discharge. Command and meter converge on AGREEING. Freezes are SAFETY-12's and
SAFETY-01's problem, and this watchdog neither helps nor hinders there.
------------------------------------------------------------------------------
STATE, TRIP AND LATCH
One timestamp, not a sample counter. The cycle rate is not fixed - a meter
arrival, or `cycle_floor_s` elapsing, whichever comes first - so N samples is a
different wall-clock window on every path, and the AC is written in minutes.
The latch is not optional and not a nicety. Commanding 0 W removes the command
side of the comparison (|0| is not > 300), so the disagreement disappears the
instant the trip works. Un-latched, this watchdog would trip, clear itself, and
hand the actuator straight back to the loop that was driving the wrong way -
three minutes later, forever. `tripped` therefore stays true until an operator
`reset()`s it by starting control again, which is a deliberate human act after
looking at why.
"""
from dataclasses import dataclass
from datetime import datetime
@dataclass(frozen=True)
class SignWatchdogConfig:
"""Defaults, not constants - config.yaml carries both (SAFETY-02 AC 5)."""
min_w: float = 300.0
hold_s: float = 180.0
class SignWatchdog:
"""One instance per actuator, evaluated once per control cycle.
⚠️ NOT SHARED ACROSS CADENCES. `disagree_since` is a wall-clock start time,
so driving one instance from two loops at two rates would have each one
resetting the other's window. If a fast on-device path ever wants this
check, it gets its own instance - the same exactly-one-writer discipline the
arbiter applies to the actuator, applied to the detector's own state.
"""
def __init__(self, cfg: SignWatchdogConfig | None = None):
self.cfg = cfg or SignWatchdogConfig()
self.disagree_since: datetime | None = None
self.tripped = False
self.reason = ""
# -- the whole check, and it is a pure function of its arguments ----------
def evaluate(self, commanded_w, grid_w, following: bool,
now: datetime) -> bool:
"""Advance the detector one cycle. Returns True while tripped.
`commanded_w` the EMITTED setpoint - what is standing on the wire now,
after deadband, slew, quantisation and the arbiter. Not
the optimizer's raw wish: the wire carries the emitted
value, so anything else can disagree with the meter for a
reason that is not a fault.
`grid_w` net P1, + = importing. None when the meter is stale or
missing, which CLEARS the timer rather than accumulating
- a frozen figure held against a moving command would
otherwise manufacture a disagreement in about three
minutes, on top of SAFETY-01 having already commanded
zero for the same cause.
`following` did the grid-following loop actually win this cycle? See
gate 1 in the module docstring.
`now` the caller's clock, so this stays testable and replayable.
"""
if self.tripped:
# Latched. Do not re-evaluate: the trip removed the evidence.
return True
cfg = self.cfg
disagree = (
following
and commanded_w is not None and grid_w is not None
and commanded_w * grid_w < 0
and abs(commanded_w) > cfg.min_w
and abs(grid_w) > cfg.min_w
)
if not disagree:
self.disagree_since = None
return False
if self.disagree_since is None:
self.disagree_since = now
return False
held = (now - self.disagree_since).total_seconds()
if held < cfg.hold_s:
return False
self.tripped = True
self.reason = (
f"SIGN_DISAGREEMENT: commanded {commanded_w:+.0f} W against a meter "
f"reading {grid_w:+.0f} W for {held:.0f} s continuously "
f"(limits: both over {cfg.min_w:.0f} W, held over {cfg.hold_s:.0f} s)"
)
return True
@property
def held_s(self) -> float:
"""How long the current disagreement has been standing, for the UI."""
if self.disagree_since is None:
return 0.0
return (datetime.now(self.disagree_since.tzinfo)
- self.disagree_since).total_seconds()
def reset(self) -> None:
self.disagree_since = None
self.tripped = False
self.reason = ""