SAFETY-02: a loop can be fresh, tracking and inside its rating and still push the wrong way

The stale-input watchdog asks whether the meter is still talking. The saturation
freeze asks whether the inverter can follow. The output clamp asks whether the
number is too big. A loop can pass all three while charging the battery out of
the grid, because the only thing wrong is the sign.

On 2026-08-20 a commercial controller did exactly that four times in one day,
for up to 77 minutes at a stretch, reaching a commanded -14 768 W against a
±5000 W inverter. app/watchdog.py compares the emitted setpoint against the
authoritative P1 figure and, when they oppose each other with both over 300 W
for 180 s continuously, files a failsafe SET claim at P_SAFETY, commands 0 W and
latches there until an operator starts control again.

Validated against that day rather than against an argument: sim/replay_sign.py
in the infra repo drives this module over the committed captures of both
2026-08-20 and the healthy 2026-08-23 and reaches 18 trips on the fault day -
every one inside a window NOTES.md §4.7 accounts for - and 0 on the healthy day.
The four episodes do share one signature: 0.3 / 0.9 / 10.5 / 0.0 % sign
agreement inside them against 92.9 % healthy.

Three decisions the ticket could not make:

- It is DISARMED whenever the grid-following loop is not what is driving. Sign
  agreement is a property of a loop that follows the meter, and a maintenance
  grid charge opposes the meter by design - 2500 W of charge into a house
  drawing 500 W is +3000 W on the meter against a -2500 W command, for hours.
  The gate tests the emitted number, not the winning claim's name, because a
  limit claim can let the loop "win" and still replace its value.
- The trip LATCHES. Commanding 0 W removes the command side of the comparison,
  so the disagreement always clears the moment the trip works.
- sign_disagree_w at or above max_w is refused at startup. The arbiter never
  lets a command exceed max_w, so such a threshold leaves the watchdog incapable
  of firing while the status page still reports it armed.

What it does not catch, said out loud: an inverter that ACTS opposite to its
command moves the meter further the same way, leaving command and meter in
agreement. That is the saturation freeze's job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du77usMj8XNKNFZGmUiWDa

goodwe_controller/app/main.py

@@ -41,6 +41,7 @@ from .maintenance import IDLE, MaintConfig, Maintenance
 from .mqtt import MqttPublisher
 from .p1 import P1Ingest, build_source, is_enabled
 from .validate import ConfigRefused, refuse, validate_startup_config
+from .watchdog import SignWatchdog, SignWatchdogConfig
 from . import web
 
 OPTIONS_PATH = "/data/options.json"
@@ -103,6 +104,12 @@ class Controller:
             store,
         )
 
+        # SAFETY-02. One instance, one actuator, one cadence - see watchdog.py.
+        self.sign_wd = SignWatchdog(SignWatchdogConfig(
+            min_w=float(opts.get("sign_disagree_w", 300)),
+            hold_s=float(opts.get("sign_disagree_s", 180)),
+        ))
+
         # P1 ingestion (TEL-01). `meter_source: off` keeps the original
         # single-entity meter_entity path, so an existing install is unchanged
         # until it opts in.
@@ -120,6 +127,11 @@ class Controller:
         self.sat_count = 0
         self.i_w = 0.0        # the loop's integrator, carried between cycles
         self.loop_w = None    # what the loop asked for last cycle, or None
+        # Did the grid-following loop actually win the LAST arbitration? That is
+        # the question SAFETY-02 has to ask about the command standing on the
+        # wire right now, so it is recorded when the arbitration happens rather
+        # than re-derived a cycle later. Starts False: nothing is on the wire.
+        self.following = False
         self.reason = "starting"
         self.grid = self.soc = self.batt = None
         self.peak_fc = None
@@ -214,8 +226,29 @@ class Controller:
         for msg in result.events:
             self.log_event(msg)
 
+        # --- SAFETY-02, and it runs BEFORE the early return below -------------
+        # ⚠️ ORDER MATTERS HERE. If this were evaluated after the "inputs missing
+        # but not yet stale" hold, a run of held cycles would leave
+        # `disagree_since` untouched across the gap and the timer would resume
+        # from a start time that predates it. It is evaluated every cycle,
+        # unconditionally, so a missing meter CLEARS the window - see watchdog.py.
+        #
+        # The value compared is the command STANDING ON THE WIRE, which is what
+        # this cycle's fresh meter reading is a response to. `last_write` is that
+        # value after the device clamp and confirmed accepted; `target` covers
+        # the case where nothing has been written yet (and, deliberately, the
+        # case where the last write was REJECTED - the inverter is then still
+        # latched on `last_write`, which is exactly what `last_write` holds).
+        was_tripped = self.sign_wd.tripped
+        standing = self.last_write if self.last_write is not None else self.target
+        if self.sign_wd.evaluate(standing, self.grid, self.following, now) \
+                and not was_tripped:
+            self.log_event("FAILSAFE " + self.sign_wd.reason)
+            _LOG.error("SAFETY-02 TRIPPED - %s. Commanding 0 W and holding it "
+                       "until control is started again.", self.sign_wd.reason)
+
         stale_after = float(self.o.get("stale_input_s", 15))
-        if not self.inputs_ok:
+        if not self.inputs_ok and not self.sign_wd.tripped:
             bad_for = (now - self.inputs_bad_since).total_seconds() if self.inputs_bad_since else 0.0
             if bad_for < stale_after:
                 # A single missed poll is not a fault. Hold, keep the heartbeat
@@ -235,6 +268,16 @@ class Controller:
         claims.append(Claim.limit("supervised", P_SAFETY, -max_w, max_w, "max_w"))
 
         # --- safety stops ----------------------------------------------------
+        # ⚠️ SAFETY-02 is claimed FIRST and unconditionally while latched. It is
+        # a SET at P_SAFETY, so it outranks maintenance, peak shaving and the
+        # loop alike - a controller pushing power the wrong way is wrong
+        # whatever it thinks it is doing. Re-filed EVERY cycle: claims are not
+        # sticky, and one filed once would lapse the next cycle and hand the
+        # actuator silently back to the loop that was driving the wrong way.
+        if self.sign_wd.tripped:
+            claims.append(Claim.set("failsafe", P_SAFETY, 0.0,
+                                    "sign disagreement"))
+
         if not self.auto:
             claims.append(Claim.set("safety", P_SAFETY, 0.0, "stopped"))
         elif not self.inputs_ok:
@@ -306,6 +349,28 @@ class Controller:
         resolution = resolve(claims)
         if resolution.contradiction:
             self.log_event("ARBITER CONTRADICTION - commanding 0 W, see the log")
+        # --- SAFETY-02's arming gate ------------------------------------------
+        # Was the number about to go on the wire chosen by the grid-following
+        # loop, or by a strategy that commands against the meter on purpose?
+        # Only the former is expected to agree in sign with P1. See gate 1 in
+        # watchdog.py - without this, every maintenance grid charge trips the
+        # failsafe within three minutes of starting.
+        #
+        # ⚠️ "THE LOOP WON" IS NOT ENOUGH, and the difference is a whole class of
+        # false trip. A LIMIT claim can survive the loop winning and still
+        # replace its number: maintenance's charge-only shaping is
+        # `hi = -charge_w`, so the loop can ask for +500 W of discharge, win the
+        # arbitration, and have the emitted value come out at -2500 W of grid
+        # charge. Winner says "loop"; the wire carries maintenance's intent. So
+        # the test is on the NUMBER, not on the name: the loop is following only
+        # while the emitted value points the way the loop asked and is no larger
+        # than it asked - i.e. limits reduced it, they did not redirect it.
+        # Zero on either side falls out as not-following, which costs nothing:
+        # a 0 W command cannot be a wrong-direction command.
+        asked = self.loop_w
+        self.following = (resolution.winner == "loop" and asked is not None
+                          and resolution.target_w * asked > 0
+                          and abs(resolution.target_w) <= abs(asked))
         self.target = resolution.target_w
         self.reason = resolution.explain()
         await self.write_setpoint(self.target)
@@ -439,7 +504,14 @@ class Controller:
             "battery": self.batt,
             "soc": self.soc,
             "phase": self.maint.phase,
-            "status": "running" if self.auto else "stopped",
+            # ⚠️ SAFETY-02's alarm surface, and it is deliberately an existing
+            # one. IF-01/IF-02 own `ems/event/failsafe` and the `failsafe active`
+            # binary sensor and neither is built yet; inventing a bespoke topic
+            # here would be a surface to migrate later. This sensor is already
+            # discovered, already on every dashboard, and "failsafe" is
+            # additive to its vocabulary rather than a change to it.
+            "status": ("failsafe" if self.sign_wd.tripped
+                       else "running" if self.auto else "stopped"),
         }
         # ⚠️ ONLY when P1 ingestion is actually running. The ESP32's stale-input
         # watchdog subscribes to sensor.goodwe_p1_sample_age - see the entity-id
@@ -521,6 +593,22 @@ class Controller:
             out.append({"ok": False, "warn": False,
                         "text": "last write to the inverter was REJECTED - check the log"})
 
+        # SAFETY-02. Reported in both states: a watchdog nobody can see is armed
+        # is a watchdog nobody trusts when it fires.
+        if self.sign_wd.tripped:
+            out.append({"ok": False, "warn": False,
+                        "text": "SIGN-DISAGREEMENT FAILSAFE - " + self.sign_wd.reason
+                                + ". The inverter is held at 0 W. Find out why the "
+                                  "command opposed the meter before starting control "
+                                  "again; starting clears this."})
+        else:
+            held = self.sign_wd.held_s
+            out.append({"ok": True, "warn": False,
+                        "text": f"sign watchdog: armed (trips after "
+                                f"{self.sign_wd.cfg.hold_s:.0f} s opposed, both over "
+                                f"{self.sign_wd.cfg.min_w:.0f} W)"
+                                + (f" - opposed for {held:.0f} s now" if held else "")})
+
         if not o.get("estop_fitted", False):
             out.append({"ok": False, "warn": True,
                         "text": "no e-stop fitted: if this host dies the battery stays "
@@ -538,7 +626,11 @@ class Controller:
     def status_payload(self) -> dict:
         checks = self.checks()
         bad = [c for c in checks if not c["ok"] and not c["warn"]]
-        if bad:
+        if self.sign_wd.tripped:
+            # Outranks "NOT READY": nothing is missing, something went wrong.
+            level, banner = "bad", ("FAILSAFE — the command opposed the meter. "
+                                    "Inverter held at 0 W.")
+        elif bad:
             level, banner = "bad", f"NOT READY — {bad[0]['text']}"
         elif not self.auto:
             level, banner = "warn", "Stopped — inverter commanded to 0 W"
@@ -572,6 +664,15 @@ class Controller:
             self.auto = not self.auto
             self.store.set("auto", self.auto)
             self.log_event("control started" if self.auto else "control stopped")
+            if self.auto and self.sign_wd.tripped:
+                # ⚠️ THE ONLY WAY OUT OF A SIGN-DISAGREEMENT TRIP, on purpose.
+                # It clears on a deliberate human start, not on the condition
+                # going away - the condition ALWAYS goes away, because the trip
+                # commands 0 W and a 0 W command cannot disagree with anything.
+                # An operator pressing start has, at minimum, looked at the
+                # banner that says why.
+                self.sign_wd.reset()
+                self.log_event("sign-disagreement failsafe cleared by operator")
             if not self.auto:
                 self.target = 0.0
                 await self.write_setpoint(0.0, force=True)