Files
glenn schrooyenandClaude Opus 5 3527c8a04c
checks / unit (pull_request) Successful in 4m59s
SAFETY-41: escalate a sustained engagement, so a runaway is never held
Review found the case I asked it to construct, and it is not a corner
case. The rate limit alone DELAYS A RUNAWAY: a benign one-cycle clamp
that spends the budget, then a continuous §10.3-shaped runaway, held the
runaway's first notification for up to 140 s - one REASSERT_S minus a
cycle. On the live install, where ordinary railing spends the budget
essentially all the time, a runaway arriving during normal operation
would be delayed with near-certainty. That is the "benign noise
suppresses the bug signal" shape this module's docstring is written
against, and the family the first attempt at this card died of.

THE ESCALATION. An episode that has not yet spoken because an EARLIER,
unrelated episode spent the budget speaks anyway once it has engaged
SUSTAINED_CYCLES = 3 times in a row. The discriminator is run length and
the two populations genuinely differ: the live flood was runs of length
ONE (engage, de-assert, engage), while a runaway engages every cycle.

Measured, and it costs nothing it was protecting:
  live scatter flood (305 cycles, 1-in-6)   11 -> 11 notifications
  chatter 1-engaged/3-clean over 8000 s     50 -> 50 notifications
  pinned 900 s quarter (floor is 5)          6 ->  6 alarms
  runaway delay after a benign episode     140 s -> 10 s

⚠️ MY COUNTERFACTUAL CHECK COULD NOT FAIL FOR THE DEFECT IT NAMED, and
review caught that too. It read `ClampWatchdog().note(True, ...) is True`
- a FRESH object, which only exercises the `last_alarm is None` branch,
the trivially-true one. A fresh watchdog has no earlier episode to be
held back BY. It now SPENDS THE BUDGET FIRST: benign clamp, gap past
CLEAR_CYCLES, then a continuous runaway, and asserts the delay is well
under REASSERT_S. That is the third instance in this file of the family
recorded at line 216 and at the 224 s sweep, and it was guarding the most
safety-critical property in the change.

Also pinned: `last_alarm` SURVIVING the de-assert. Review built the
mutation - if the de-assert also refreshed it, chatter collapses from 50
notifications to 1 and the alarm never fires again, with the suite green.
That is the "alarm that never fires" class, a different bug from one that
fires later. Now caught.

Both new checks verified against the mutations they name: removing the
escalation -> 1 FAILED; refreshing last_alarm on de-assert -> 1 FAILED.

Full suite: 21 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-05 18:13:09 +02:00

425 lines
22 KiB
Python

"""SAFETY-03 - the output clamp, and the alarm that makes it audible.
WHAT IT CATCHES, in one sentence: a control law that computes a setpoint the
actuator cannot deliver, which every layer below it will happily clip to
something legal-looking while the thing that produced it goes on winding.
THE CLAMP IS NOT THE DELIVERABLE. specs.md 10.3 is explicit that this project
already clipped, and that the clipping is what hid the defect:
with the inverter switched off entirely, an unclamped integrator climbed
~130 W per 4 s to over 10 kW and, pinned at the 5 kW rail, reported
14 768 W
A clamp alone turns that into a silent, perfectly flat 5000 W hold. Every
dashboard reads "commanded 5000 W", which is a legal number, and nothing
anywhere says the loop asked for three times it. 11.2's watchdog row says
"Clamp AND ALARM - the loop is winding"; the second verb is this module.
------------------------------------------------------------------------------
WHAT "THE RATING" IS HERE, AND WHY IT IS ONE NUMBER AND NOT A MIN()
The card names two bounds and warns against choosing between them:
- `DriverCapabilities.setpoint_limit_w` (3.2), the inverter's own magnitude
rating. In this add-on that arrives as the setpoint entity's `min`/`max`
from `hass.limits()` - `Controller.dev_min` / `dev_max`.
- `optimizer.loop.clamp_w` (14), the configured loop clamp. In this add-on
that is the `max_w` option, which `Tuning.max_w` and the arbiter's
`supervised` limit both already use.
14 says these two must AGREE, validated at startup with a refusal on
mismatch - and that gate already ships: `validate_startup_config()` refuses
`max_w > dev_max` and `-max_w < dev_min` (validate.py, problem 1), and
run_control() re-runs it once ESPHome is up and the real rating is known.
So this guard clamps to `max_w`, THE VALIDATED VALUE, and does not `min()` it
with the device rating at runtime. That is the card's instruction and it is also
the only reading that keeps `max_w` meaningful: a silent runtime `min()` is
precisely the "helper bounded to +-500 W under a loop clamped to 1500 W" failure
the 14 warning box is about.
WARNING - THE DEVICE BOUND IS STILL APPLIED, SECOND, AND IT IS NOT REDUNDANT.
After a successful validation `+-max_w` lies inside `[dev_min, dev_max]`, so the
second stage is arithmetically a no-op and can never engage. If it DOES engage,
the validation did not run or no longer holds - `hass.limits()` returns
(None, None) whenever ESPHome is not up yet, and the range checks are skipped
rather than guessed at - and that is a louder fault than the first stage, not a
quieter one. It therefore alarms too, through the same channel, with the bound
named.
⚠️ SAFETY-20 UPDATED THE SECOND HALF OF THAT SENTENCE. It used to read "this is
NOT SAFETY-20; nothing here changes what `freeze.py` measures its rail against",
and that was true of this module and false of the system: this stage is exactly
what clips the emitted command below `RAIL_FRAC * max_w`, which is how a device
rating disarmed the freeze detector while the page said `armed`. `freeze.py` now
takes `dev_min`/`dev_max` and measures its rail against
`min(max_w, |dev_min|, |dev_max|)`. THE CLAMP ITSELF IS UNCHANGED - it still
clamps to `max_w` first and to the device range second, for the reasons above,
and it still does not `min()` the two. What changed is only that the detector
downstream now knows this stage exists.
------------------------------------------------------------------------------
WHERE THE WIND ACTUALLY SHOWS UP - READ THIS BEFORE "SIMPLIFYING" THE CALLER
`control.compute()` bounds the integrator to `integrator_max_w or max_w` and
THEN derives the command from it, so on a default install (`integrator_max_w`
unset, i.e. following `max_w`) the command can never exceed `max_w` and the
actuator-side clamp below IS UNREACHABLE. control.py says so in as many words:
SAFETY-03 ... must watch for THIS, not for "clamped" below.
An alarm wired only to this module's `ClampResult.engaged` would therefore be
silent on the exact configuration the whole ticket is about. The winding is
visible one layer up, at the integrator bound, and that is why `Decision` now
carries `i_clamped` / `clamped` BOOLEANS and the caller ORs them in here.
AND WHY BOOLEANS RATHER THAN `Decision.reason`. `reason` is a single string that
later stages overwrite: a cycle that clamps the integrator and is then slew
limited reports "slew-limited", and one that clamps while saturated reports
"saturated-freeze". Both are exactly the winding-into-a-rail case, and on the
string the clamp signal is destroyed before anything can alarm on it. The two
flags are set where the clamp happens and are never overwritten. A check in
test_clamp.py drives that precise combination.
------------------------------------------------------------------------------
DISTINCT FROM SATURATION - THIS IS ACCEPTANCE CRITERION 2, NOT A NICETY
clamp the COMPUTED value exceeded the rating. A software bug signal:
something upstream is winding. Alarm, every engagement.
saturation |command - inverter AC power| > `saturation_w` for
`saturation_cycles`. The HARDWARE cannot follow a perfectly legal
command. Action is "freeze the integrator" (11.2), not an alarm.
A loop can be legally saturated all day - a cold pack, a curtailed inverter, an
export limit - so a saturation-shaped alarm is one an operator learns to ignore.
Routing the clamp through it puts the bug signal inside the noise, which is the
same silence 10.3 is complaining about. They share no reason code, no
notification id, no status row and no counter.
------------------------------------------------------------------------------
ANTI-FLOOD, WITHOUT GOING SILENT
A genuinely winding loop clamps on EVERY cycle for as long as it winds. One HA
notification per cycle is how an operator learns to dismiss notifications
unread, and this add-on already made that argument once (see the fixed
`notification_id` note in main.check_meter_frozen).
So the two surfaces are split, deliberately:
- EVERY engagement is recorded, never deduplicated, and that record is what
says how hard the loop is pushing. 15's action log is this add-on's
`Controller.events`, and it gets a row from `write_setpoint()` on every
engaged WRITE; `engagements` here counts every engaged CYCLE. The two layers
are counted separately on purpose - the heartbeat re-writes an
already-clamped value between cycles, so a per-write count would inflate and
a per-cycle log would miss an out-of-cycle write.
- THE NOTIFICATION fires on the rising edge and then RE-ASSERTS every
`REASSERT_S`, so a loop pinned on the rail for an hour does not go quiet.
- It DE-ASSERTS only after `CLEAR_CYCLES` consecutive clean cycles, which is
the 10.2 hysteresis: a command sitting exactly on the rail dithers in and
out of engagement by one quantisation step, and without this the alarm
chatters.
`REASSERT_S` and `CLEAR_CYCLES` are module constants, not options, for the same
reason `freeze.RAIL_FRAC` is: they describe the event bus, not the plant, and
every knob 10.3 asked to keep exposed is a LOOP TUNING knob and is untouched by
this module. No new config key is added anywhere - CFG-03 already has three
places a default has to be kept in step and this ticket does not earn a fourth.
------------------------------------------------------------------------------
NOT A TRIP
Clamping does not stop the loop and does not command 0 W. The clamped value is
written and control continues, which is what 11.2 asks for ("Clamp and alarm")
and what distinguishes this from SAFETY-01/02, whose action is 0 W. It follows
that this must NOT drive the failsafe status - `publish()` keeps reporting
"running", because the controller is.
"""
import math
from dataclasses import dataclass
from datetime import datetime
# The reason code, distinct from any saturation-freeze code by construction:
# nothing else in this add-on uses it. See "DISTINCT FROM SATURATION".
REASON_CODE = "OUTPUT_CLAMP"
# Re-assert the notification this often while the clamp stays engaged.
#
# ⚠️ RAISED 30 s -> 150 s on 2026-09-05, and the reason is a REGIME CHANGE, not a
# change of mind about the alarm. 30 s was the planning cadence (2.3), chosen
# when a loop-side engagement was a rare event: `main.py`'s note at the action
# log records "60 cycles of the §10.3 runaway produce 0 write-path engagements
# and 2 loop-side ones". On the live install the counter reached 48 loop-side
# ENGAGEMENTS in ~25 minutes of ordinary operation - roughly one cycle in six,
# not a pinned rail, but far more often than a guard sized for 2-in-60 assumed -
# and the owner received a persistent notification about it all day. ⚠️ NOT
# because of the 30 s re-assert - see the block below; those notifications never
# consulted this constant at all.
#
# ⚠️ BOTH FIGURES ARE ENGAGEMENT COUNTS; only the regime differs - 2-in-60 was
# measured over a §10.3 RUNAWAY, 48-in-305 was ordinary running on the live
# install. The 48 is read off the live log, not measured in this repo.
#
# ⚠️ AND RAISING THIS CONSTANT ALONE DID NOT FIX THAT FLOOD - review found it
# and the first version of this change was wrong. At ~5 s cycles a 1-in-6
# engagement leaves ~25 s of clean cycles between engagements, past
# CLEAR_CYCLES x 5 s = 15 s, so the alarm DE-ASSERTED between almost every one
# and came back as a fresh rising edge. The rising edge used to fire
# unconditionally, so those notifications never consulted this constant at all:
# 48 notifications at 30 s, and still 48 at 150 s. The fix was to rate-limit the
# RISING EDGE too and let `last_alarm` survive a de-assert, so this interval
# bounds the notification rate across episodes rather than only within one.
# Measured on that same window afterwards: 51 episodes, 11 notifications.
#
# WHY the loop reaches its integrator bound so much more often than the guard's
# author saw: `max_w` on the live install is 5000 W, while the loop's slew was
# derived against a 1500 W clamp - admin/goodwe `NOTES.md` §"Simulated 2 kW load
# step (gain 0.6, clamp 1500)". ⚠️ THAT DERIVATION IS IN THE OTHER REPO; nothing
# in this one corroborates it, so do not go looking here. SAFETY-39 is the ticket
# that re-derives it, and fixing that is what would stop the railing; this
# constant only changes how often the alarm repeats.
#
# ⚠️ 150 s IS THE REQUIREMENT'S EDGE WITH MARGIN, AND THE CHECK NOW ENFORCES THAT
# EDGE. `test_clamp.py` part 2 grades the cadence in wall-clock terms - a pinned
# loop must speak at least MIN_PER_QUARTER = 5 times inside the 900 s quarter
# that bills the capacity tariff - which caps this constant at 900/5 = 180 s.
# Measured on the worst sliding 900 s window: 180 s earns 5 alarms, 181 s earns
# 4. 150 s earns 6.
#
# ⚠️ THAT CHECK USED TO PERMIT 224 s, AND THE FIX SHIPPED WITH THIS COMMIT. Its
# sweep starts at i=0, where the rising edge is free, so it measured one alarm
# more than a real quarter earns. A later editor raising this to 200 s would have
# had a green suite and a requirement quietly unmet. `test_clamp.py` now grades
# floor(QUARTER_S / REASSERT_S) directly.
#
# ⚠️ MIN_PER_QUARTER = 5 IS A TEST-LOCAL INTEGER, not a spec figure - it exists
# only in `test_clamp.py` and the CHANGELOG entry that introduced it. The 900 s
# quarter IS system-wide (`tariff.py`, `freeze.py`, `DOCS.md`). So the 180 s
# ceiling rests on one number a test author chose; if that number is ever
# revisited, this constant moves with it.
#
# ⚠️ THIS IS A STOP-GAP AND DOES NOT MAKE THE ALARM CORRECT. The message still
# calls the integrator bound a BUG SIGNAL and still pairs a request from one
# moment with a `self.target` read at another - SAFETY-40. Nothing here is
# suppressed: every engagement is still counted, the action-log row is written on
# every engaged cycle regardless, and the rising edge of every episode alarms
# unconditionally, so no alarm is DROPPED - only delayed.
REASSERT_S = 150.0
# Consecutive clean cycles before the alarm de-asserts (10.2 hysteresis).
CLEAR_CYCLES = 3
# ⚠️ SAFETY-41. Consecutive ENGAGED cycles after which an episode speaks even if
# the notification budget is spent. THE ESCALATION EXISTS BECAUSE THE RATE LIMIT
# ALONE DELAYS A RUNAWAY, and review measured it: a benign one-cycle clamp
# followed by a continuous runaway held the runaway's first notification for up
# to 140 s - one REASSERT_S minus a cycle. On the live install, where ordinary
# railing spends the budget essentially all the time, a runaway arriving during
# normal operation would be delayed with near-certainty. That is the "benign
# noise suppresses the bug signal" shape this module's docstring is written
# against.
#
# THE DISCRIMINATOR IS RUN LENGTH, and the two populations genuinely differ: the
# live flood was runs of length ONE (engage, de-assert, engage), while a §10.3
# runaway engages on EVERY cycle. So a sustained run speaks regardless of budget,
# and scattered chatter does not.
#
# Measured, shipped-without vs shipped-with - it costs nothing it was protecting:
# live scatter flood (305 cycles, 1-in-6) 11 -> 11 notifications
# chatter 1-engaged/3-clean over 8000 s 50 -> 50 notifications
# pinned 900 s quarter (floor is 5) 6 -> 6 alarms
# runaway delay after a benign episode 140 s -> 10 s
SUSTAINED_CYCLES = 3
@dataclass(frozen=True)
class ClampResult:
"""What the guard did, with the RAW value preserved.
AC 2 turns on "the raw computed value differed from the clamped one", so the
raw value has to survive the call - a guard that returns only the applied
number cannot be asked the question the acceptance criterion asks.
"""
requested_w: float
applied_w: float
limit_w: float
engaged: bool
bound: str = "" # "" | "max_w" | "device" - which stage bit
def describe(self) -> str:
return (f"{REASON_CODE}: the computed setpoint {self.requested_w:+.0f} W "
f"was clamped to {self.applied_w:+.0f} W by the {self.bound} "
f"bound (rating {self.limit_w:.0f} W) - something upstream is "
f"winding")
def is_commandable(value) -> bool:
"""False for None, NaN and the infinities.
A non-finite setpoint is LOSS OF INPUT, not an out-of-range value, and
11.1 rule 1 says loss of input means command zero and keep commanding. It
must not reach the clamp arithmetic: `min(limit, nan)` returns nan on
CPython, so a nan would sail straight through a naive clamp and then report
`engaged` True on the `applied != requested` test - because nan != nan - i.e.
a transport bug filed in the winding-loop counter, with a nan on the wire.
"""
if value is None:
return False
try:
return math.isfinite(float(value))
except (TypeError, ValueError):
return False
def clamp_to_rating(requested_w, limit_w, dev_min=None, dev_max=None) -> ClampResult:
"""Bound a computed setpoint to the validated rating. The ONLY clamp.
`limit_w` the validated magnitude bound - `max_w`. Symmetric, because
3.2 calls `setpoint_limit_w` a MAGNITUDE clamp: it binds import
(+) and export (-) identically.
`dev_min` / `dev_max` the device's own asymmetric range, applied second and
expected never to bite. See the module note.
Public, and the predicate lives here rather than in the test, for the reason
`freeze.looks_frozen()` gives: a safety expression transcribed into its own
check is one copy plus a future disagreement about what the check means.
EXACTLY +-`limit_w` IS NOT AN ENGAGEMENT. The bound is inclusive; a loop
resting on the rail is doing what it was told, and alarming there would fire
on every healthy full-power hour.
"""
requested = float(requested_w)
limit = abs(float(limit_w))
# ⚠️ BOTH STAGES ALWAYS RUN. THIS USED TO `return` FROM THE FIRST ONE, AND
# THAT WAS A REGRESSION AGAINST THE CODE IT REPLACED. `write_setpoint()`
# previously clamped to the device range unconditionally, under the comment
# "never send a value the device will reject outright"; short-circuiting here
# meant the device bound was applied only when the rating bound had NOT
# fired, i.e. THE BIGGER THE OVERSHOOT THE LESS THE GUARD DID. Measured on
# the shipped function at max_w 5000 against a device of +-4000:
#
# +9999 -> +5000 bound='max_w' <- outside the device range
# +4500 -> +4000 bound='device'
#
# It also undercut this module's own argument for clamping to `max_w` rather
# than a runtime `min()`, which rests entirely on this stage catching the
# case where the startup validation did not run.
applied = max(-limit, min(limit, requested))
bound = "max_w" if applied != requested else ""
# Second stage. Arithmetically dead after a successful startup validation -
# and therefore worth alarming on precisely BECAUSE it fired.
if dev_min is not None and dev_max is not None:
bounded = max(float(dev_min), min(float(dev_max), applied))
if bounded != applied:
bound = "device" if not bound else "max_w+device"
applied = bounded
return ClampResult(requested, applied, limit, applied != requested, bound)
class ClampWatchdog:
"""One instance per controller. Counts every engagement, alarms sparsely.
`note()` is called once per write with whether the clamp bit; the caller ORs
in the loop's own `i_clamped` / `clamped` flags, which is where the winding
is actually visible on a default install (see the module note).
"""
def __init__(self):
self.active = False
self.engagements = 0 # every engagement, never deduplicated
self.episodes = 0 # rising edges
self.since: datetime | None = None
self.last_alarm: datetime | None = None
self.reason = ""
self.clean_cycles = 0
# SAFETY-41. Consecutive engaged cycles, and whether THIS episode has
# spoken yet - the two the escalation clause needs.
self.run_len = 0
self.spoken = False
def note(self, engaged: bool, reason: str, now: datetime) -> bool:
"""Advance one cycle. True when the caller should raise the alarm.
`now` is the caller's clock, the same discipline as
`SignWatchdog.evaluate()` and `FreezeWatchdog.evaluate()`.
"""
if engaged:
self.engagements += 1
self.clean_cycles = 0
self.reason = reason
self.run_len += 1
if not self.active:
self.active = True
self.since = now
self.episodes += 1
self.spoken = False
# ⚠️ SAFETY-41. THE RISING EDGE IS RATE-LIMITED TOO, and it did
# not used to be. It fired unconditionally, so a condition that
# engages and clears repeatedly raised one notification PER
# EPISODE no matter how fast the episodes came - and
# `REASSERT_S` never entered into it, because it only governs
# the still-engaged path below.
#
# That is the shape the live install actually produced: 48
# engagements in 25 minutes, scattered roughly one cycle in six.
# At a ~5 s cycle the gaps are ~25 s of clean cycles, which
# exceeds CLEAR_CYCLES x 5 s = 15 s, so the alarm DE-ASSERTED
# between almost every engagement and each one came back as a
# fresh rising edge. Measured on this class over that window:
# 48 episodes and 48 notifications at REASSERT_S = 30 - and
# still 48 at REASSERT_S = 150. Raising the re-assert removed
# NONE of them, because none of them came from the re-assert.
#
# So the interval now bounds the NOTIFICATION RATE rather than
# the repeat rate within one episode: `last_alarm` survives a
# de-assert, and a new episode speaks only if the interval has
# elapsed. ⚠️ NOTHING IS DROPPED. Every engagement is still
# counted, every episode still increments, the action-log row is
# still written on every engaged cycle, and a genuine runaway -
# which by definition keeps engaging - is picked up by the
# re-assert path within REASSERT_S. Delay, not loss.
if (self.last_alarm is None
or (now - self.last_alarm).total_seconds() >= REASSERT_S):
self.last_alarm = now
self.spoken = True
return True
return False
# ⚠️ THE ESCALATION. An episode that has not yet spoken because the
# budget was spent by an EARLIER, unrelated episode speaks anyway
# once it has engaged SUSTAINED_CYCLES times in a row. A runaway
# engages every cycle and reaches it; the scattered one-cycle
# railing that produced the live flood never does.
if not self.spoken and self.run_len >= SUSTAINED_CYCLES:
self.last_alarm = now
self.spoken = True
return True
# Still engaged: re-assert on the slow cadence, not every cycle.
if (self.last_alarm is None
or (now - self.last_alarm).total_seconds() >= REASSERT_S):
self.last_alarm = now
self.spoken = True
return True
return False
self.run_len = 0
if not self.active:
return False
self.clean_cycles += 1
if self.clean_cycles >= CLEAR_CYCLES:
self.active = False
self.since = None
# ⚠️ `last_alarm` DELIBERATELY SURVIVES THE DE-ASSERT. Clearing it
# here is what made the next rising edge speak immediately however
# soon it arrived, which is the flood described at the rising edge
# above. The episode ends; the notification budget does not reset
# with it.
self.reason = ""
self.clean_cycles = 0
return False