SAFETY-02: a loop can be fresh, tracking and inside its rating and still push the wrong way

The stale-input watchdog asks whether the meter is still talking. The saturation
freeze asks whether the inverter can follow. The output clamp asks whether the
number is too big. A loop can pass all three while charging the battery out of
the grid, because the only thing wrong is the sign.

On 2026-08-20 a commercial controller did exactly that four times in one day,
for up to 77 minutes at a stretch, reaching a commanded -14 768 W against a
±5000 W inverter. app/watchdog.py compares the emitted setpoint against the
authoritative P1 figure and, when they oppose each other with both over 300 W
for 180 s continuously, files a failsafe SET claim at P_SAFETY, commands 0 W and
latches there until an operator starts control again.

Validated against that day rather than against an argument: sim/replay_sign.py
in the infra repo drives this module over the committed captures of both
2026-08-20 and the healthy 2026-08-23 and reaches 18 trips on the fault day -
every one inside a window NOTES.md §4.7 accounts for - and 0 on the healthy day.
The four episodes do share one signature: 0.3 / 0.9 / 10.5 / 0.0 % sign
agreement inside them against 92.9 % healthy.

Three decisions the ticket could not make:

- It is DISARMED whenever the grid-following loop is not what is driving. Sign
  agreement is a property of a loop that follows the meter, and a maintenance
  grid charge opposes the meter by design - 2500 W of charge into a house
  drawing 500 W is +3000 W on the meter against a -2500 W command, for hours.
  The gate tests the emitted number, not the winning claim's name, because a
  limit claim can let the loop "win" and still replace its value.
- The trip LATCHES. Commanding 0 W removes the command side of the comparison,
  so the disagreement always clears the moment the trip works.
- sign_disagree_w at or above max_w is refused at startup. The arbiter never
  lets a command exceed max_w, so such a threshold leaves the watchdog incapable
  of firing while the status page still reports it armed.

What it does not catch, said out loud: an inverter that ACTS opposite to its
command moves the meter further the same way, leaving command and meter in
agreement. That is the saturation freeze's job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du77usMj8XNKNFZGmUiWDa
This commit is contained in:
glenn schrooyen
2026-08-28 21:51:35 +02:00
co-authored by Claude Opus 5
parent 35856a73ae
commit 9aed43990b
9 changed files with 755 additions and 8 deletions
+51
View File
@@ -24,6 +24,9 @@ The inverter **holds its last command forever** — it has no meter-timeout. So:
commanding it.
- This add-on commands 0 W when its inputs go missing, when you stop control,
and when it shuts down.
- This add-on commands 0 W **and latches there** if what it is commanding and
what the meter reads point opposite ways for three minutes — see "The
sign-disagreement failsafe" below.
- The **optional RS485 e-stop** is the only thing that covers this machine
dying. Without it, a failed host leaves the battery latched at its last
command until someone intervenes.
@@ -278,6 +281,54 @@ behind it would trip the firmware watchdog on a system that is working fine.
| `cycle_floor_s` | 15 | The longest the loop may go without evaluating. A deadline on the gap between cycles, reset by every cycle — on a healthy meter it never elapses and changes nothing. Raise it only if your meter is slower than ~10 s; it must stay under `meter_max_age_s`, past which there is no reading to evaluate anyway |
| `stale_input_s` | 15 | How long inputs may be missing before commanding 0 W. In P1 mode this clock starts only *after* `meter_max_age_s` has already expired — the two stack, see "How long a dead meter takes to reach 0 W" |
| `auto_start` | false | Start controlling on boot (only after commissioning) |
| `sign_disagree_w` | 300 | Both the command and the meter must exceed this before a sign difference counts. Must stay **below `max_w`** or the watchdog can never fire, and the add-on refuses to start |
| `sign_disagree_s` | 180 | How long they must stay opposed, continuously, before the failsafe trips |
#### The sign-disagreement failsafe
If the setpoint this add-on is emitting and the grid figure it is regulating on
point **opposite ways** — charging the battery while the house imports, or
discharging it while the house exports — with both over `sign_disagree_w`, for
`sign_disagree_s` **continuously**, the add-on commands **0 W** and holds it
there. The banner reads `FAILSAFE`, the commissioning list says which two
numbers disagreed and for how long, and the `Controller status` entity reports
`failsafe`.
**It does not clear itself.** Pressing *Start / stop control* twice clears it.
That is deliberate: commanding 0 W removes the command side of the comparison,
so the disagreement always vanishes the instant the trip works, and a watchdog
that took that as proof the fault was over would hand the inverter straight back
to the loop that was driving the wrong way.
**What it is for.** This is a loop-integrity check: it asks whether the loop's
output direction still matches the input state it was regulating. It catches a
polarity error anywhere between the meter and the wire, and an integrator so far
wound up that unwinding takes minutes. It is the check that would have caught all
four episodes of 2026-08-20, when a commercial controller charged the battery out
of the grid, four times, for up to 77 minutes at a time.
**What it is not for.** It does not detect an inverter that acts opposite to what
it was told — that moves the meter *further* the same way, so command and meter
agree while everything runs away. That is the saturation freeze's job
(`saturation_w` / `saturation_cycles`), which compares the command against the
battery's actual power. The two checks are complementary and neither replaces the
other.
**Why it will not false-trip on ordinary operation.** A battery that refuses a
command (empty, full, standby) leaves the meter where it was, so command and
meter agree. A frozen meter drives the loop into agreeing with it. Around the
resting point both signals are inside the 300 W gate, so the deadband can never
produce one. A load step does flip the meter's sign before the loop catches up —
measured on a full captured day, the longest such crossing is 30 s, against a
180 s window. And the watchdog is **disarmed whenever the grid-following loop is
not what is driving**: a maintenance grid charge opposes the meter by design and
must not be mistaken for a fault.
Both numbers are measured rather than chosen, and are exposed only because the
ticket requires defaults rather than constants. On the captured healthy day sign
agreement is 92.9 %; on the four captured fault episodes it is 0.3 %, 0.9 %,
10.5 % and 0.0 %. There is no reason to move either value without a capture that
says so.
#### Saturation is counted in cycles, not seconds