goodwe-addon/goodwe_controller/app/watchdog.py

"""SAFETY-02 - the sign-disagreement watchdog.

WHAT IT CATCHES, in one sentence: a grid-following loop that is pushing power in
the wrong direction, which every other guard in this system is blind to.

The stale-input watchdog (SAFETY-01) asks "is the meter still talking?". The
saturation freeze asks "can the inverter follow?". The output clamp asks "is the
number too big?". A loop can pass all three and still be charging the battery
out of the grid while the house imports - the meter is fresh, the inverter is
tracking beautifully, and the magnitude is inside the rating. The only thing
that separates that from healthy operation is the SIGN.

⚠️ THIS IS NOT A HYPOTHETICAL. On 2026-08-20 the vendor controller (Illusmart)
did exactly this four times in one day (NOTES.md §4.7). Both captures of that
bus are committed - `sim/scenarios/bus-2026-08-20.log.gz` and the healthy
`bus-2026-08-23.log.gz` in the infra repo - and `sim/replay_sign.py` drives this
module over both of them. Every number in this file was measured on those two
days, not chosen:

    day          samples gated in    sign agreement
    2026-08-23   4 212 / 17 280      93.4 %   (healthy)
    2026-08-20   7 182 / 17 280      56.4 %   (the whole fault day)
      cluster A  15:26-16:43            0.3 %
      cluster B  17:11-18:28            0.9 %
      cluster C  20:01-20:31           10.5 %
      cluster D  20:55-21:11            0.0 %

⚠️ THOSE AGREEMENT FIGURES ARE THE WEAKEST EVIDENCE ON THIS PAGE, and an earlier
version of this comment leaned on them as though they were the strongest. The
clusters are found by looking for sustained opposed samples, so measuring "how
opposed is this cluster" over them is close to a tautology - and "the healthy day
contains no such cluster" is the same sentence as "the healthy day produces no
trip", not independent corroboration of it.

What actually carries the argument is two things neither the detector nor the
clustering assumes:

  1. The RUN-LENGTH DISTRIBUTION. The longest unbroken opposed run on the healthy
     day is 30 s. On the fault day there are 18 runs over 180 s and the longest is
     1750 s. Nothing about that gap was chosen - it is what the two days measure.
  2. The clusters land where a human independently wrote down onsets. NOTES.md
     §4.7 records 15:01, 16:56, 19:57 and 20:23 from watching it happen, before
     any of this existed, and cluster D coincides with the separately documented
     inverter-off runaway of §4.7 item 2.

Be precise about (2), because the convenient overstatement is available and wrong:
the clustering does NOT reproduce four episodes. It yields THREE clusters over the
four documented onsets - C spans both 19:57 and 20:23 - plus a fourth for §4.7
item 2, which is not one of the four. Every trip does fall inside a documented
window, which is the claim worth making.

------------------------------------------------------------------------------
WHY IT LIVES HERE AND NOT IN THE FIRMWARE OR THE WITNESS

Three vantage points see three different things. The firmware knows what it
WROTE, the witness Pi knows what was ON THE WIRE and whether the inverter ACKed
it, and the add-on knows what it INTENDED and holds the authoritative meter.

This check is not command-versus-wire; it is command-versus-physics. It needs
the emitted setpoint and the P1 net figure in the same place, and it needs the
authority to command zero when they fight:

  - the witness is read-only by charter and cannot command anything, so it could
    raise the alarm but never take the trip action the AC requires;
  - the firmware has the wire but not the meter. §5.1 makes P1 the only
    authoritative grid figure and the board only ever receives its AGE
    (`sensor.goodwe_p1_sample_age`), never its value. Putting the check there
    would mean shipping the meter to the ESP32 for no other reason, and it would
    be C++ per install rather than one generic guard;
  - the add-on has both halves already, plus an arbiter that can outrank every
    strategy with a P_SAFETY claim.

The cost of standing here is real and worth stating: this watchdog compares what
we MEANT to put on the wire against the meter. If the firmware or the driver
inverts the sign on the way out, the add-on's own view still agrees with itself
and this check stays silent. That gap is a wire-versus-intent problem and it
belongs to the witness, not here.

------------------------------------------------------------------------------
WHAT "DISAGREEMENT" MEANS, PRECISELY

    disagree = following
               and commanded_w * grid_w < 0
               and abs(commanded_w) > min_w
               and abs(grid_w)      > min_w

Four gates, and each one exists to exclude a specific thing that is NOT a fault:

1. `following`. ⚠️ THE GATE THAT IS EASIEST TO FORGET AND MOST EXPENSIVE TO
   OMIT. Sign agreement is a property of a GRID-FOLLOWING loop. It is not
   expected of a strategy that deliberately commands against the meter -
   maintenance is exactly that. A maintenance grid charge of 1500 W into a house
   drawing 500 W puts +2000 W on the meter against a -1500 W command: opposed,
   both far over any sane threshold, and sustained for HOURS. Armed
   unconditionally this watchdog would trip on every single maintenance cycle,
   which is not a tuning problem, it is the wrong question being asked. So the
   caller says whether the grid-following loop actually won the arbitration this
   cycle, and the timer only accumulates when it did.

   ⚠️ 1500 W, NOT THE DEFAULT 2500 W, AND THE REASON IS NOT COSMETIC. At the
   shipped defaults `maintenance_charge_w` is 2500 and `max_w` is 2000, so the
   cheap-window claim is `hi = -2500` against a supervised `lo = -2000`: lo > hi,
   which arbiter.py resolves as a CONTRADICTION and 0 W, not as a -2500 W charge.
   The example above is written at 1500 W because that is a combination the
   schema and the arbiter both actually produce. (The contradiction at the
   defaults is a real pre-existing bug - a default maintenance charge silently
   never happens in a cheap window - but it is maintenance's, not this module's,
   and it is filed separately. Do not "fix" it here.)

2. The strict `< 0` product. "Opposed" and "neither side is zero" in one
   expression.

3. The 300 W magnitude gate ON BOTH SIDES. Near zero the sign is noise, and near
   zero is where a healthy loop LIVES - the deadband holds the command still
   while the meter wanders either side of `target_grid_w`. On the healthy day
   only 4 212 of 17 280 samples (24 %) are even gated in; the other 76 % are the
   converged band where sign carries no information. This is also why the
   deadband cannot produce a false trip: inside it the meter is by definition
   under 300 W.

4. The 180 s continuity requirement. This is what excludes the legitimate
   transient. A load step reverses the meter's sign instantly while the command
   is still where the last cycle left it; slew limiting then needs a few cycles
   to cross over. Measured on the healthy day, the LONGEST such crossing is 30 s
   - and that is with a vendor controller far cruder than ours. 180 s is six
   times that. It is not a fitted threshold: at 60 s the healthy day still
   produces zero trips, and at 300 s the fault day still produces 13.

⚠️ A BATTERY THAT REFUSES A COMMAND CANNOT TRIP THIS, and that is structural
rather than lucky. At SoC 10 % in Standby, a 1350 W discharge command into a
1350 W import is DECLINED: the battery does nothing, so the meter stays at
+1350 W and the command stays at +1350 W. Same sign. Agreement. A refusal leaves
the meter where it was; only the inverter ACTING moves it, and acting in the
commanded direction moves it toward zero, not through it. The failure mode this
check keys on is the opposite of a refusal - it is the inverter obediently doing
a wrong thing.

⚠️ A FROZEN METER cannot trip it either, for the same reason read the other way:
a stuck reading of +800 W makes the loop command discharge and keep commanding
discharge. Command and meter converge on AGREEING. Freezes are SAFETY-12's and
SAFETY-01's problem, and this watchdog neither helps nor hinders there.

⚠️ ALL OF THAT DEPENDS ON ONE PROPERTY OF THE CONTROL LAW, so it is an invariant
and not a happy accident: the saturation freeze in control.py must keep letting
the command FALL, CROSS ZERO AND REVERSE - "may not wind further in the direction
it is already pushing", not "only corrections that shrink |i_w|". control.py
records the shrink-only encoding as tried and rejected because it pins the command
at its last value indefinitely; a pinned command against a meter that has moved on
is precisely a sustained sign disagreement, so under that encoding this watchdog
WOULD false-trip on a plain load step. If anyone ever revisits that freeze, this
module's no-false-positive argument is downstream of the answer.

------------------------------------------------------------------------------
WHAT IT DOES NOT CATCH

An inverter that ACTS opposite to what it was told. Tell it to discharge 2000 W
and have it charge instead, and the meter moves FURTHER the way it was already
going - so the command and the meter agree, all the way into the rail. This
watchdog compares the loop's output direction against the input state it was
regulating; it is blind to the plant lying about which way it went. That is the
saturation freeze's question (|command - the battery's actual power| > 500 W,
NOTES.md §4.7 row 4), and the two checks are complementary rather than
overlapping. Neither substitutes for the other.

Nor does it see a sign inversion introduced DOWNSTREAM of this add-on - in the
firmware, on the wire, or in a future driver. Our view stays self-consistent and
this stays silent. That is a wire-versus-intent problem and it belongs to the
witness, which is the only vantage point that sees both.

------------------------------------------------------------------------------
STATE, TRIP AND LATCH

One timestamp, not a sample counter. The cycle rate is not fixed - a meter
arrival, or `cycle_floor_s` elapsing, whichever comes first - so N samples is a
different wall-clock window on every path, and the AC is written in minutes.

The latch is not optional and not a nicety. Commanding 0 W removes the command
side of the comparison (|0| is not > 300), so the disagreement disappears the
instant the trip works. Un-latched, this watchdog would trip, clear itself, and
hand the actuator straight back to the loop that was driving the wrong way -
three minutes later, forever. `tripped` therefore stays true until an operator
`reset()`s it by starting control again, which is a deliberate human act after
looking at why.

⚠️ AND THE LATCH HAS TO OUTLIVE THE PROCESS. Held only in memory it is not a
latch at all - a container restart clears it and the add-on comes back armed,
into the same fault, unattended. Worse than merely losing state: "restart the
add-on" is the first thing an operator confronted with a FAILSAFE banner and a
0 W inverter will try, and under a supervisor that restarts a crashed add-on it
is a loop of three-minute wrong-direction bursts. `tripped` and `reason` are
therefore constructor arguments, restored from `Store` in `Controller.__init__`
and written back the moment the trip happens. See store.py.
"""

from dataclasses import dataclass
from datetime import datetime


@dataclass(frozen=True)
class SignWatchdogConfig:
    """Defaults, not constants - config.yaml carries both (SAFETY-02 AC 5)."""

    min_w: float = 300.0
    hold_s: float = 180.0


def disagrees(commanded_w, grid_w, following: bool,
              cfg: SignWatchdogConfig) -> bool:
    """The predicate, and the ONLY place it is written down.

    ⚠️ Public because `sim/replay_sign.py` needs it too - to describe episodes, to
    measure agreement, to sweep the clock-skew guard - and it had reimplemented it
    inline. Two copies of a safety predicate is one copy plus a future
    disagreement about what the check means, and the copy that drifts is always
    the one in the harness that grades it.
    """
    return (following
            and commanded_w is not None and grid_w is not None
            and commanded_w * grid_w < 0
            and abs(commanded_w) > cfg.min_w
            and abs(grid_w) > cfg.min_w)


def loop_is_following(resolution, loop_asked) -> bool:
    """Did the grid-following loop actually choose the number going on the wire?

    Gate 1 of the predicate above, and it lives here rather than inline in
    `cycle()` for the reason `disagrees()` does: the test suite has to be able to
    assert on THE SHIPPED EXPRESSION, not on a transcription of it that drifts
    apart from it in silence.

    ⚠️ IT WAS A TRANSCRIPTION, AND THAT WAS MEASURED, NOT FEARED. The P_PEAK case
    in test_watchdog.py exists to make whoever lands peak shaving come here and
    change an expectation. With the expression copied into the test it did not do
    that: editing the shipped gate alone to `winner in ("loop", "peak")` left every
    suite green while the watchdog went blind for the whole of every peak event.
    One definition, two callers, and that test now bites.

    `resolution`  the arbiter's Resolution. Duck-typed on `.winner` and
                  `.target_w` so this module still imports nothing from arbiter.
    `loop_asked`  what the grid-following loop claimed this cycle, or None if it
                  did not claim at all.

    "The loop won" is NOT the question, and the difference is a whole class of
    false trip. A LIMIT claim can survive the loop winning and still replace its
    number: maintenance's cheap-window shaping is `hi = -maintenance_charge_w`, so
    at 1500 W the loop can ask for +500 W of discharge, win the arbitration, and
    have the emitted value come out at -1500 W of grid charge. Winner says "loop";
    the wire carries maintenance's intent.

    So the test is on the NUMBER: the loop is following only while the emitted
    value points the way the loop asked AND is no larger than it asked - limits
    reduced it, they did not redirect it. The magnitude half is load-bearing on
    its own, because a `lo` limit can raise magnitude without flipping sign, and
    that is not the loop following either. Zero on either side falls out as
    not-following, which costs nothing: a 0 W command cannot be a
    wrong-direction command.

    ⚠️ FORWARD HAZARD, and the claim block in cycle() already anticipates it.
    `winner == "loop"` disarms the watchdog under ANY higher-priority SET claim.
    Today only maintenance and the safety stops emit one and neither follows the
    meter, so the gate costs nothing. A peak-shaving strategy claiming SET at
    P_PEAK **is** grid-following in sign, and the moment it lands this goes blind
    for the duration of every peak event. Widening it means asking "does the
    winning strategy follow the meter", not adding another name to a comparison.
    """
    return (resolution.winner == "loop" and loop_asked is not None
            and resolution.target_w * loop_asked > 0
            and abs(resolution.target_w) <= abs(loop_asked))


class SignWatchdog:
    """One instance per actuator, evaluated once per control cycle.

    ⚠️ NOT SHARED ACROSS CADENCES. `disagree_since` is a wall-clock start time,
    so driving one instance from two loops at two rates would have each one
    resetting the other's window. If a fast on-device path ever wants this
    check, it gets its own instance - the same exactly-one-writer discipline the
    arbiter applies to the actuator, applied to the detector's own state.
    """

    def __init__(self, cfg: SignWatchdogConfig | None = None, *,
                 tripped: bool = False, reason: str = ""):
        self.cfg = cfg or SignWatchdogConfig()
        self.disagree_since: datetime | None = None
        # ⚠️ RESTORED FROM THE STORE, NOT DEFAULTED. A latch that lives only in
        # memory is not a latch: a container restart clears it, and the add-on
        # comes back armed, into the same fault, with no operator involved. See
        # store.py's note on `sign_tripped` for the measured behaviour.
        self.tripped = bool(tripped)
        self.reason = reason if self.tripped else ""

    # -- the whole check, and it is a pure function of its arguments ----------
    def evaluate(self, commanded_w, grid_w, following: bool,
                 now: datetime) -> bool:
        """Advance the detector one cycle. Returns True while tripped.

        `commanded_w`  the EMITTED setpoint - what is standing on the wire now,
                       after deadband, slew, quantisation and the arbiter. Not
                       the optimizer's raw wish: the wire carries the emitted
                       value, so anything else can disagree with the meter for a
                       reason that is not a fault.
        `grid_w`       net P1, + = importing. None when the meter is stale or
                       missing, which CLEARS the timer rather than accumulating
                       - a frozen figure held against a moving command would
                       otherwise manufacture a disagreement in about three
                       minutes, on top of SAFETY-01 having already commanded
                       zero for the same cause.
        `following`    did the grid-following loop actually win this cycle? See
                       gate 1 in the module docstring.
        `now`          the caller's clock, so this stays testable and replayable.
        """
        if self.tripped:
            # Latched. Do not re-evaluate: the trip removed the evidence.
            return True

        cfg = self.cfg
        if not disagrees(commanded_w, grid_w, following, cfg):
            self.disagree_since = None
            return False

        if self.disagree_since is None:
            self.disagree_since = now
            return False

        held = (now - self.disagree_since).total_seconds()
        if held < cfg.hold_s:
            return False

        self.tripped = True
        self.reason = (
            f"SIGN_DISAGREEMENT: commanded {commanded_w:+.0f} W against a meter "
            f"reading {grid_w:+.0f} W for {held:.0f} s continuously "
            f"(limits: both over {cfg.min_w:.0f} W, held over {cfg.hold_s:.0f} s)"
        )
        return True

    def held_s(self, now: datetime) -> float:
        """How long the current disagreement has been standing, for the UI.

        ⚠️ Takes the clock rather than reading one. It was a property calling
        `datetime.now()`, which quietly broke the property this module's header
        advertises - every other observable here is a pure function of what it
        was handed - and an unclocked function is one nothing can test, which is
        exactly what happened to it.
        """
        if self.disagree_since is None:
            return 0.0
        return (now - self.disagree_since).total_seconds()

    def reset(self) -> None:
        self.disagree_since = None
        self.tripped = False
        self.reason = ""