SAFETY-02 review: a latch that a restart clears is not a latch

BLOCKER. `auto` was persisted and `sign_wd.tripped` was not, so reconstructing
the Controller against the same store - which is what a container restart does -
brought it back ARMED, into the same fault, unattended: tripped=False, banner
back to NOT READY, and it resumed commanding -1000 then -2000 W before
re-tripping 180 s later. "Restart the add-on" is the first thing an operator
looking at a FAILSAFE banner and a 0 W inverter will try, and under a supervisor
that restarts a crashed add-on it is a loop of three-minute wrong-direction
bursts. DOCS.md's "it does not clear itself" was false across a restart.

`sign_tripped` and `sign_reason` are Store keys now, beside the maintenance
state and for the reason store.py's own docstring gives. Written the moment the
trip happens - reason first, flag second, so a crash between the two can never
persist a latch with no explanation - restored in Controller.__init__, cleared
only by the operator's start. The regression is covered: reverting the restore
fails seven checks in test_watchdog.py.

Review follow-ups, no behaviour change:

- The arming gate is tested against the REAL arbiter. It is the decision the
  ticket turns on and it was asserted in three comment blocks and verified by
  nothing - every case passed `following` in as a literal. Twelve claim
  configurations now derive it from resolve(), including the redirect case
  `winner == "loop"` alone would have got wrong and the `lo` limit that raises
  magnitude without flipping sign.
- The MQTT `status: "failsafe"` value is tested. The check that claimed to cover
  it restated the line above it and never touched the payload.
- The worked example described a config that does not ship: at the defaults
  maintenance_charge_w 2500 against max_w 2000 is an arbiter contradiction and
  0 W, not a -2500 W charge. Rewritten at 1500 W. The contradiction is a real
  pre-existing bug - a default cheap-window maintenance charge silently never
  happens - filed separately and deliberately not fixed here.
- Forward hazard recorded at the gate: `winner == "loop"` disarms the watchdog
  under any higher-priority SET, and a peak-shaving strategy at P_PEAK IS
  grid-following in sign, so it would blind this for every peak event.
- disagrees() is public so the replay harness stops carrying its own copy of the
  predicate it grades.
- held_s takes the clock instead of reading one.
- The two limits are now each stated in both places rather than one each.
- The no-false-positive argument is tied to control.py's freeze permitting the
  command to cross zero; under the shrink-only encoding control.py records as
  rejected, this would false-trip on a load step.
- Measured figures corrected: 93.4 % / 4212 of 17280, per-cluster
  0.3/0.9/10.5/0.0, and the near-circularity of those figures is now stated
  rather than leaned on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du77usMj8XNKNFZGmUiWDa
This commit is contained in:
glenn schrooyen
2026-08-28 22:33:15 +02:00
co-authored by Claude Opus 5
parent 2513ba1f12
commit cc4e88490b
8 changed files with 458 additions and 61 deletions
+63 -16
View File
@@ -105,10 +105,27 @@ class Controller:
)
# SAFETY-02. One instance, one actuator, one cadence - see watchdog.py.
self.sign_wd = SignWatchdog(SignWatchdogConfig(
min_w=float(opts.get("sign_disagree_w", 300)),
hold_s=float(opts.get("sign_disagree_s", 180)),
))
#
# ⚠️ RESTORED FROM THE STORE, and that is the difference between a latch
# and a pause. A trip held only in memory is cleared by the one action an
# operator staring at a FAILSAFE banner is most likely to take - restart
# the add-on - and the controller then comes back armed, into the fault,
# with nobody involved. Restored here, the banner and the 0 W survive the
# restart and only `auto_toggle` clears them.
self.sign_wd = SignWatchdog(
SignWatchdogConfig(
min_w=float(opts.get("sign_disagree_w", 300)),
hold_s=float(opts.get("sign_disagree_s", 180)),
),
tripped=bool(store.data.get("sign_tripped", False)),
reason=str(store.data.get("sign_reason", "") or ""),
)
if self.sign_wd.tripped:
_LOG.error("SAFETY-02 was TRIPPED before this restart and stays "
"tripped: %s. The inverter is held at 0 W. Restarting the "
"add-on does not clear this - start control from the UI "
"once you know why the command opposed the meter.",
self.sign_wd.reason or "(reason not recorded)")
# P1 ingestion (TEL-01). `meter_source: off` keeps the original
# single-entity meter_entity path, so an existing install is unchanged
@@ -243,9 +260,18 @@ class Controller:
standing = self.last_write if self.last_write is not None else self.target
if self.sign_wd.evaluate(standing, self.grid, self.following, now) \
and not was_tripped:
# ⚠️ REASON FIRST, FLAG SECOND, and each `set` writes the file. The
# ordering is the whole atomicity story: a crash between the two
# persists a reason nothing reads, which is harmless, where the other
# order persists a latch with no explanation for the operator who has
# to decide whether to clear it.
self.store.set("sign_reason", self.sign_wd.reason)
self.store.set("sign_tripped", True)
self.log_event("FAILSAFE " + self.sign_wd.reason)
_LOG.error("SAFETY-02 TRIPPED - %s. Commanding 0 W and holding it "
"until control is started again.", self.sign_wd.reason)
"until control is started again. This survives an add-on "
"restart; restarting will not clear it.",
self.sign_wd.reason)
stale_after = float(self.o.get("stale_input_s", 15))
if not self.inputs_ok and not self.sign_wd.tripped:
@@ -358,15 +384,33 @@ class Controller:
#
# ⚠️ "THE LOOP WON" IS NOT ENOUGH, and the difference is a whole class of
# false trip. A LIMIT claim can survive the loop winning and still
# replace its number: maintenance's charge-only shaping is
# `hi = -charge_w`, so the loop can ask for +500 W of discharge, win the
# arbitration, and have the emitted value come out at -2500 W of grid
# charge. Winner says "loop"; the wire carries maintenance's intent. So
# the test is on the NUMBER, not on the name: the loop is following only
# while the emitted value points the way the loop asked and is no larger
# than it asked - i.e. limits reduced it, they did not redirect it.
# Zero on either side falls out as not-following, which costs nothing:
# a 0 W command cannot be a wrong-direction command.
# replace its number: maintenance's cheap-window shaping is
# `hi = -maintenance_charge_w`, so with `maintenance_charge_w: 1500` the
# loop can ask for +500 W of discharge, win the arbitration, and have the
# emitted value come out at -1500 W of grid charge. Winner says "loop";
# the wire carries maintenance's intent. So the test is on the NUMBER,
# not on the name: the loop is following only while the emitted value
# points the way the loop asked and is no larger than it asked - i.e.
# limits reduced it, they did not redirect it. Zero on either side falls
# out as not-following, which costs nothing: a 0 W command cannot be a
# wrong-direction command. The `abs(...) <= abs(asked)` half is
# load-bearing on its own: a `lo` limit can raise the magnitude without
# flipping the sign, and that is not the loop following either.
#
# (1500 W because at the SHIPPED defaults - maintenance_charge_w 2500 vs
# max_w 2000 - that claim is `lo=-2000 > hi=-2500`, which the arbiter
# calls a contradiction and resolves to 0 W. Filed separately; the
# redirect hazard here is real at any charge rate inside max_w.)
#
# ⚠️ FORWARD HAZARD, AND IT IS NOT HYPOTHETICAL - the claim block above
# already anticipates it. `winner == "loop"` disarms this watchdog under
# ANY higher-priority SET claim. Today only maintenance and the safety
# stops emit one, and neither is grid-following, so the gate costs
# nothing. A peak-shaving strategy claiming SET at P_PEAK **is**
# grid-following in sign, and the moment it lands this watchdog goes
# blind for the duration of every peak event. Whoever adds it must widen
# the gate to "the winning strategy follows the meter" rather than
# "the winner is the loop", not just add another name to a comparison.
asked = self.loop_w
self.following = (resolution.winner == "loop" and asked is not None
and resolution.target_w * asked > 0
@@ -600,9 +644,10 @@ class Controller:
"text": "SIGN-DISAGREEMENT FAILSAFE - " + self.sign_wd.reason
+ ". The inverter is held at 0 W. Find out why the "
"command opposed the meter before starting control "
"again; starting clears this."})
"again; starting clears this. Restarting the "
"add-on does NOT clear it."})
else:
held = self.sign_wd.held_s
held = self.sign_wd.held_s(datetime.now(timezone.utc))
out.append({"ok": True, "warn": False,
"text": f"sign watchdog: armed (trips after "
f"{self.sign_wd.cfg.hold_s:.0f} s opposed, both over "
@@ -672,6 +717,8 @@ class Controller:
# An operator pressing start has, at minimum, looked at the
# banner that says why.
self.sign_wd.reset()
self.store.set("sign_tripped", False)
self.store.set("sign_reason", "")
self.log_event("sign-disagreement failsafe cleared by operator")
if not self.auto:
self.target = 0.0