adminandClaude Opus 5.5
7b723a6cd5
Rule 2 graded each battery against last cycle's command and graded charges too. On plantsim's battery (10 s lag) at the 2 s meter it warned at every drain or charge start and 15 times an hour under the peak guard at N=2; a charge tapering near full warned 11 times in 50 min (SAFETY-42 section 7). Now a discharge is graded against the least it stood at over SHORT_CYCLES / SHORT_SPAN_S. DOCS lists `mode-bound` under Loop reason. test_safety85 part 7: peak guard with a dead battery, lagging plant, taper; part 3: the bound ends with maintenance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
527 lines
26 KiB
Python
527 lines
26 KiB
Python
"""SAFETY-03 - the output clamp, and the alarm that makes it audible.
|
|
|
|
WHAT IT CATCHES, in one sentence: a control law that computes a setpoint the
|
|
actuator cannot deliver, which every layer below it will happily clip to
|
|
something legal-looking while the thing that produced it goes on winding.
|
|
|
|
THE CLAMP IS NOT THE DELIVERABLE. specs.md 10.3 is explicit that this project
|
|
already clipped, and that the clipping is what hid the defect:
|
|
|
|
with the inverter switched off entirely, an unclamped integrator climbed
|
|
~130 W per 4 s to over 10 kW and, pinned at the 5 kW rail, reported
|
|
14 768 W
|
|
|
|
A clamp alone turns that into a silent, perfectly flat 5000 W hold. Every
|
|
dashboard reads "commanded 5000 W", which is a legal number, and nothing
|
|
anywhere says the loop asked for three times it. 11.2's watchdog row says
|
|
"Clamp AND ALARM - the loop is winding"; the second verb is this module.
|
|
|
|
------------------------------------------------------------------------------
|
|
WHAT "THE RATING" IS HERE, AND WHY IT IS ONE NUMBER AND NOT A MIN()
|
|
|
|
The card names two bounds and warns against choosing between them:
|
|
|
|
- `DriverCapabilities.setpoint_limit_w` (3.2), the inverter's own magnitude
|
|
rating. In this add-on that arrives as the setpoint entity's `min`/`max`
|
|
from `hass.limits()` - `Controller.dev_min` / `dev_max`.
|
|
- `optimizer.loop.clamp_w` (14), the configured loop clamp. In this add-on
|
|
that is the `max_w` option, which `Tuning.max_w` and the arbiter's
|
|
`supervised` limit both already use.
|
|
|
|
14 says these two must AGREE, validated at startup with a refusal on
|
|
mismatch - and that gate already ships: `validate_startup_config()` refuses
|
|
`max_w > dev_max` and `-max_w < dev_min` (validate.py, problem 1), and
|
|
run_control() re-runs it once ESPHome is up and the real rating is known.
|
|
|
|
So this guard clamps to `max_w`, THE VALIDATED VALUE, and does not `min()` it
|
|
with the device rating at runtime. That is the card's instruction and it is also
|
|
the only reading that keeps `max_w` meaningful: a silent runtime `min()` is
|
|
precisely the "helper bounded to +-500 W under a loop clamped to 1500 W" failure
|
|
the 14 warning box is about.
|
|
|
|
WARNING - THE DEVICE BOUND IS STILL APPLIED, SECOND, AND IT IS NOT REDUNDANT.
|
|
After a successful validation `+-max_w` lies inside `[dev_min, dev_max]`, so the
|
|
second stage is arithmetically a no-op and can never engage. If it DOES engage,
|
|
the validation did not run or no longer holds - `hass.limits()` returns
|
|
(None, None) whenever ESPHome is not up yet, and the range checks are skipped
|
|
rather than guessed at - and that is a louder fault than the first stage, not a
|
|
quieter one. It therefore alarms too, through the same channel, with the bound
|
|
named.
|
|
|
|
⚠️ SAFETY-20 UPDATED THE SECOND HALF OF THAT SENTENCE. It used to read "this is
|
|
NOT SAFETY-20; nothing here changes what `freeze.py` measures its rail against",
|
|
and that was true of this module and false of the system: this stage is exactly
|
|
what clips the emitted command below `RAIL_FRAC * max_w`, which is how a device
|
|
rating disarmed the freeze detector while the page said `armed`. `freeze.py` now
|
|
takes `dev_min`/`dev_max` and measures its rail against
|
|
`min(max_w, |dev_min|, |dev_max|)`. THE CLAMP ITSELF IS UNCHANGED - it still
|
|
clamps to `max_w` first and to the device range second, for the reasons above,
|
|
and it still does not `min()` the two. What changed is only that the detector
|
|
downstream now knows this stage exists.
|
|
|
|
------------------------------------------------------------------------------
|
|
WHERE THE WIND ACTUALLY SHOWS UP - READ THIS BEFORE "SIMPLIFYING" THE CALLER
|
|
|
|
`control.compute()` bounds the integrator to `integrator_max_w or max_w` and
|
|
THEN derives the command from it, so on a default install (`integrator_max_w`
|
|
unset, i.e. following `max_w`) the command can never exceed `max_w` and the
|
|
actuator-side clamp below IS UNREACHABLE. control.py says so in as many words:
|
|
|
|
SAFETY-03 ... must watch for THIS, not for "clamped" below.
|
|
|
|
An alarm wired only to this module's `ClampResult.engaged` would therefore be
|
|
silent on the exact configuration the whole ticket is about. The winding is
|
|
visible one layer up, at the integrator bound, and that is why `Decision` now
|
|
carries `i_clamped` / `clamped` BOOLEANS and the caller ORs them in here.
|
|
They alarm in different words: see `loop_clamp_note()` (SAFETY-40).
|
|
|
|
AND WHY BOOLEANS RATHER THAN `Decision.reason`. `reason` is a single string that
|
|
later stages overwrite: a cycle that clamps the integrator and is then slew
|
|
limited reports "slew-limited", and one that clamps while saturated reports
|
|
"saturated-freeze". Both are exactly the winding-into-a-rail case, and on the
|
|
string the clamp signal is destroyed before anything can alarm on it. The two
|
|
flags are set where the clamp happens and are never overwritten. A check in
|
|
test_clamp.py drives that precise combination.
|
|
|
|
------------------------------------------------------------------------------
|
|
DISTINCT FROM SATURATION - THIS IS ACCEPTANCE CRITERION 2, NOT A NICETY
|
|
|
|
clamp the COMPUTED value exceeded the rating. A software bug signal:
|
|
something upstream is winding. Alarm, every engagement.
|
|
saturation |command - inverter AC power| > `saturation_w` for
|
|
`saturation_cycles`. The HARDWARE cannot follow a perfectly legal
|
|
command. Action is "freeze the integrator" (11.2), not an alarm.
|
|
|
|
A loop can be legally saturated all day - a cold pack, a curtailed inverter, an
|
|
export limit - so a saturation-shaped alarm is one an operator learns to ignore.
|
|
Routing the clamp through it puts the bug signal inside the noise, which is the
|
|
same silence 10.3 is complaining about. They share no reason code, no
|
|
notification id, no status row and no counter.
|
|
|
|
------------------------------------------------------------------------------
|
|
ANTI-FLOOD, WITHOUT GOING SILENT
|
|
|
|
A genuinely winding loop clamps on EVERY cycle for as long as it winds. One HA
|
|
notification per cycle is how an operator learns to dismiss notifications
|
|
unread, and this add-on already made that argument once (see the fixed
|
|
`notification_id` note in main.check_meter_frozen).
|
|
|
|
So the two surfaces are split, deliberately:
|
|
|
|
- EVERY engagement is recorded, never deduplicated, and that record is what
|
|
says how hard the loop is pushing. 15's action log is this add-on's
|
|
`Controller.events`, and it gets a row from `write_setpoint()` on every
|
|
engaged WRITE; `engagements` here counts every engaged CYCLE. The two layers
|
|
are counted separately on purpose - the heartbeat re-writes an
|
|
already-clamped value between cycles, so a per-write count would inflate and
|
|
a per-cycle log would miss an out-of-cycle write.
|
|
- THE NOTIFICATION fires on the rising edge and then RE-ASSERTS every
|
|
`REASSERT_S`, so a loop pinned on the rail for an hour does not go quiet.
|
|
- It DE-ASSERTS only after `CLEAR_CYCLES` consecutive clean cycles, which is
|
|
the 10.2 hysteresis: a command sitting exactly on the rail dithers in and
|
|
out of engagement by one quantisation step, and without this the alarm
|
|
chatters.
|
|
|
|
`REASSERT_S` and `CLEAR_CYCLES` are module constants, not options, for the same
|
|
reason `freeze.RAIL_FRAC` is: they describe the event bus, not the plant, and
|
|
every knob 10.3 asked to keep exposed is a LOOP TUNING knob and is untouched by
|
|
this module. No new config key is added anywhere - CFG-03 already has three
|
|
places a default has to be kept in step and this ticket does not earn a fourth.
|
|
|
|
------------------------------------------------------------------------------
|
|
NOT A TRIP
|
|
|
|
Clamping does not stop the loop and does not command 0 W. The clamped value is
|
|
written and control continues, which is what 11.2 asks for ("Clamp and alarm")
|
|
and what distinguishes this from SAFETY-01/02, whose action is 0 W. It follows
|
|
that this must NOT drive the failsafe status - `publish()` keeps reporting
|
|
"running", because the controller is.
|
|
"""
|
|
|
|
import math
|
|
from dataclasses import dataclass
|
|
from datetime import datetime
|
|
|
|
from . import messages
|
|
|
|
# The reason code, distinct from any saturation-freeze code by construction:
|
|
# nothing else in this add-on uses it. See "DISTINCT FROM SATURATION".
|
|
REASON_CODE = "OUTPUT_CLAMP"
|
|
|
|
# Re-assert the notification this often while the clamp stays engaged.
|
|
#
|
|
# ⚠️ RAISED 30 s -> 150 s on 2026-09-05, and the reason is a REGIME CHANGE, not a
|
|
# change of mind about the alarm. 30 s was the planning cadence (2.3), chosen
|
|
# when a loop-side engagement was a rare event: `main.py`'s note at the action
|
|
# log records "60 cycles of the §10.3 runaway produce 0 write-path engagements
|
|
# and 2 loop-side ones". On the live install the counter reached 48 loop-side
|
|
# ENGAGEMENTS in ~25 minutes of ordinary operation - roughly one cycle in six,
|
|
# not a pinned rail, but far more often than a guard sized for 2-in-60 assumed -
|
|
# and the owner received a persistent notification about it all day. ⚠️ NOT
|
|
# because of the 30 s re-assert - see the block below; those notifications never
|
|
# consulted this constant at all.
|
|
#
|
|
# ⚠️ BOTH FIGURES ARE ENGAGEMENT COUNTS; only the regime differs - 2-in-60 was
|
|
# measured over a §10.3 RUNAWAY, 48-in-305 was ordinary running on the live
|
|
# install. The 48 is read off the live log, not measured in this repo.
|
|
#
|
|
# ⚠️ AND RAISING THIS CONSTANT ALONE DID NOT FIX THAT FLOOD - review found it
|
|
# and the first version of this change was wrong. At ~5 s cycles a 1-in-6
|
|
# engagement leaves ~25 s of clean cycles between engagements, past
|
|
# CLEAR_CYCLES x 5 s = 15 s, so the alarm DE-ASSERTED between almost every one
|
|
# and came back as a fresh rising edge. The rising edge used to fire
|
|
# unconditionally, so those notifications never consulted this constant at all:
|
|
# 48 notifications at 30 s, and still 48 at 150 s. The fix was to rate-limit the
|
|
# RISING EDGE too and let `last_alarm` survive a de-assert, so this interval
|
|
# bounds the notification rate across episodes rather than only within one.
|
|
# Measured on that same window afterwards: 51 episodes, 11 notifications.
|
|
#
|
|
# WHY the loop reaches its integrator bound so much more often than the guard's
|
|
# author saw: `max_w` on the live install is 5000 W, while the loop's slew was
|
|
# derived against a 1500 W clamp - admin/goodwe `NOTES.md` §"Simulated 2 kW load
|
|
# step (gain 0.6, clamp 1500)". ⚠️ THAT DERIVATION IS IN THE OTHER REPO; nothing
|
|
# in this one corroborates it, so do not go looking here. SAFETY-39 is the ticket
|
|
# that re-derives it, and fixing that is what would stop the railing; this
|
|
# constant only changes how often the alarm repeats.
|
|
#
|
|
# ⚠️ 150 s IS THE REQUIREMENT'S EDGE WITH MARGIN, AND THE CHECK NOW ENFORCES THAT
|
|
# EDGE. `test_clamp.py` part 2 grades the cadence in wall-clock terms - a pinned
|
|
# loop must speak at least MIN_PER_QUARTER = 5 times inside the 900 s quarter
|
|
# that bills the capacity tariff - which caps this constant at 900/5 = 180 s.
|
|
# Measured on the worst sliding 900 s window: 180 s earns 5 alarms, 181 s earns
|
|
# 4. 150 s earns 6.
|
|
#
|
|
# ⚠️ THAT CHECK USED TO PERMIT 224 s, AND THE FIX SHIPPED WITH THIS COMMIT. Its
|
|
# sweep starts at i=0, where the rising edge is free, so it measured one alarm
|
|
# more than a real quarter earns. A later editor raising this to 200 s would have
|
|
# had a green suite and a requirement quietly unmet. `test_clamp.py` now grades
|
|
# floor(QUARTER_S / REASSERT_S) directly.
|
|
#
|
|
# ⚠️ MIN_PER_QUARTER = 5 IS A TEST-LOCAL INTEGER, not a spec figure - it exists
|
|
# only in `test_clamp.py` and the CHANGELOG entry that introduced it. The 900 s
|
|
# quarter IS system-wide (`tariff.py`, `freeze.py`, `DOCS.md`). So the 180 s
|
|
# ceiling rests on one number a test author chose; if that number is ever
|
|
# revisited, this constant moves with it.
|
|
#
|
|
# ⚠️ THIS WAS A STOP-GAP; the wording and the numbers were SAFETY-40's, fixed
|
|
# in `loop_clamp_note()` below. Nothing here is
|
|
# suppressed: every engagement is still counted, the action-log row is written on
|
|
# every engaged cycle regardless, and the rising edge of every episode alarms
|
|
# unconditionally, so no alarm is DROPPED - only delayed.
|
|
REASSERT_S = 150.0
|
|
|
|
# Consecutive clean cycles before the alarm de-asserts (10.2 hysteresis).
|
|
CLEAR_CYCLES = 3
|
|
|
|
# ⚠️ SAFETY-41. Consecutive ENGAGED cycles after which an episode speaks even if
|
|
# the notification budget is spent. THE ESCALATION EXISTS BECAUSE THE RATE LIMIT
|
|
# ALONE DELAYS A RUNAWAY, and review measured it: a benign one-cycle clamp
|
|
# followed by a continuous runaway held the runaway's first notification for up
|
|
# to 140 s - one REASSERT_S minus a cycle. On the live install, where ordinary
|
|
# railing spends the budget essentially all the time, a runaway arriving during
|
|
# normal operation would be delayed with near-certainty. That is the "benign
|
|
# noise suppresses the bug signal" shape this module's docstring is written
|
|
# against.
|
|
#
|
|
# THE DISCRIMINATOR IS RUN LENGTH, and the two populations genuinely differ: the
|
|
# live flood was runs of length ONE (engage, de-assert, engage), while a §10.3
|
|
# runaway engages on EVERY cycle. So a sustained run speaks regardless of budget,
|
|
# and scattered chatter does not.
|
|
#
|
|
# Measured, shipped-without vs shipped-with - it costs nothing it was protecting:
|
|
# live scatter flood (305 cycles, 1-in-6) 11 -> 11 notifications
|
|
# chatter 1-engaged/3-clean over 8000 s 50 -> 50 notifications
|
|
# pinned 900 s quarter (floor is 5) 6 -> 6 alarms
|
|
# runaway delay after a benign episode 140 s -> 10 s
|
|
SUSTAINED_CYCLES = 3
|
|
|
|
|
|
# UX-10: the reasons below are quoted by a WARNING or ERROR line, so they
|
|
# follow STYLE.md's rule for a quoted reason (test_messages.py part 6): no
|
|
# final full stop, the line adds it. The line around them says "software
|
|
# fault" for an over-rating, so the reason does not. Each is one sentence, so
|
|
# the logged line stays within 3 sentences and about 30 words.
|
|
|
|
|
|
def loop_clamp_note(d) -> str:
|
|
"""The loop-side alarm text for one `control.Decision`, or "".
|
|
|
|
SAFETY-40: the integrator bound is anti-windup doing its job and says so;
|
|
only a command over the rating is a software fault. Every number is the
|
|
decision's own, i.e. from the tick that clamped.
|
|
"""
|
|
if d.clamped:
|
|
return (f"{REASON_CODE}: computed {d.requested_w:+.0f} W, cut to the "
|
|
f"{d.clamped_w:+.0f} W rating, command now {d.target_w:+.0f} W")
|
|
if d.i_clamped:
|
|
return (f"{REASON_CODE}: the correction the add-on builds up reached "
|
|
f"its {d.i_bound_w:+.0f} W limit, command now "
|
|
f"{d.target_w:+.0f} W")
|
|
return ""
|
|
|
|
|
|
def shortfall_note(named) -> str:
|
|
"""SAFETY-85: the alarm text while another owner sets the batteries.
|
|
`named` is `[(inverter name or None, asked W, given W)]`, one per battery
|
|
that gave less than asked."""
|
|
return f"{REASON_CODE}: " + ", ".join(
|
|
f"{name or 'the battery'} gives {given:+.0f} W of the {asked:+.0f} W "
|
|
f"asked" for name, asked, given in named)
|
|
|
|
|
|
def delivery_short(asked_w, given_w, margin_w) -> bool:
|
|
"""SAFETY-85: did a battery asked to discharge give more than `margin_w`
|
|
less? A charge is never short: a full pack tapers (SAFETY-42 section 7)."""
|
|
asked, given = float(asked_w), float(given_w)
|
|
return asked > 0.0 and asked - given > abs(float(margin_w))
|
|
|
|
|
|
def write_clamp_note(named) -> str:
|
|
"""The write-side alarm text: `named` is `[(inverter name or None,
|
|
ClampResult)]` for every inverter whose write was cut back this cycle.
|
|
|
|
One inverter reads as `describe()`. Several read as one sentence of
|
|
asked/sent pairs, so the line stays one sentence however many inverters
|
|
there are. Which limit cut each one back is on its own action-log row
|
|
(`describe()`) and in the notification."""
|
|
if len(named) == 1:
|
|
name, r = named[0]
|
|
return f"{name}: {r.describe()}" if name else r.describe()
|
|
return f"{REASON_CODE}: " + ", ".join(
|
|
f"{name} asked {r.requested_w:+.0f} W and sent {r.applied_w:+.0f} W"
|
|
for name, r in named)
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class ClampResult:
|
|
"""What the guard did, with the RAW value preserved.
|
|
|
|
AC 2 turns on "the raw computed value differed from the clamped one", so the
|
|
raw value has to survive the call - a guard that returns only the applied
|
|
number cannot be asked the question the acceptance criterion asks.
|
|
"""
|
|
|
|
requested_w: float
|
|
applied_w: float
|
|
limit_w: float
|
|
engaged: bool
|
|
bound: str = "" # "" | "max_w" | "device" - which stage bit
|
|
|
|
def describe(self) -> str:
|
|
# The sent value IS the limit that bound, so the rating is not
|
|
# repeated: on "max_w" it equals the rating, and on "device" the
|
|
# rating was not what cut it back.
|
|
return (f"{REASON_CODE}: asked {self.requested_w:+.0f} W and sent "
|
|
f"{self.applied_w:+.0f} W, "
|
|
f"{messages.LIMIT_WORDS.get(self.bound, self.bound)}")
|
|
|
|
|
|
def is_commandable(value) -> bool:
|
|
"""False for None, NaN and the infinities.
|
|
|
|
A non-finite setpoint is LOSS OF INPUT, not an out-of-range value, and
|
|
11.1 rule 1 says loss of input means command zero and keep commanding. It
|
|
must not reach the clamp arithmetic: `min(limit, nan)` returns nan on
|
|
CPython, so a nan would sail straight through a naive clamp and then report
|
|
`engaged` True on the `applied != requested` test - because nan != nan - i.e.
|
|
a transport bug filed in the winding-loop counter, with a nan on the wire.
|
|
"""
|
|
if value is None:
|
|
return False
|
|
try:
|
|
return math.isfinite(float(value))
|
|
except (TypeError, ValueError):
|
|
return False
|
|
|
|
|
|
def clamp_to_rating(requested_w, limit_w, dev_min=None, dev_max=None) -> ClampResult:
|
|
"""Bound a computed setpoint to the validated rating. The ONLY clamp.
|
|
|
|
`limit_w` the validated magnitude bound - `max_w`. Symmetric, because
|
|
3.2 calls `setpoint_limit_w` a MAGNITUDE clamp: it binds import
|
|
(+) and export (-) identically.
|
|
`dev_min` / `dev_max` the device's own asymmetric range, applied second and
|
|
expected never to bite. See the module note.
|
|
|
|
Public, and the predicate lives here rather than in the test, for the reason
|
|
`freeze.looks_frozen()` gives: a safety expression transcribed into its own
|
|
check is one copy plus a future disagreement about what the check means.
|
|
|
|
EXACTLY +-`limit_w` IS NOT AN ENGAGEMENT. The bound is inclusive; a loop
|
|
resting on the rail is doing what it was told, and alarming there would fire
|
|
on every healthy full-power hour.
|
|
"""
|
|
requested = float(requested_w)
|
|
limit = abs(float(limit_w))
|
|
|
|
# ⚠️ BOTH STAGES ALWAYS RUN. THIS USED TO `return` FROM THE FIRST ONE, AND
|
|
# THAT WAS A REGRESSION AGAINST THE CODE IT REPLACED. `write_setpoint()`
|
|
# previously clamped to the device range unconditionally, under the comment
|
|
# "never send a value the device will reject outright"; short-circuiting here
|
|
# meant the device bound was applied only when the rating bound had NOT
|
|
# fired, i.e. THE BIGGER THE OVERSHOOT THE LESS THE GUARD DID. Measured on
|
|
# the shipped function at max_w 5000 against a device of +-4000:
|
|
#
|
|
# +9999 -> +5000 bound='max_w' <- outside the device range
|
|
# +4500 -> +4000 bound='device'
|
|
#
|
|
# It also undercut this module's own argument for clamping to `max_w` rather
|
|
# than a runtime `min()`, which rests entirely on this stage catching the
|
|
# case where the startup validation did not run.
|
|
applied = max(-limit, min(limit, requested))
|
|
bound = "max_w" if applied != requested else ""
|
|
|
|
# Second stage. Arithmetically dead after a successful startup validation -
|
|
# and therefore worth alarming on precisely BECAUSE it fired.
|
|
if dev_min is not None and dev_max is not None:
|
|
bounded = max(float(dev_min), min(float(dev_max), applied))
|
|
if bounded != applied:
|
|
bound = "device" if not bound else "max_w+device"
|
|
applied = bounded
|
|
|
|
return ClampResult(requested, applied, limit, applied != requested, bound)
|
|
|
|
|
|
class ClampWatchdog:
|
|
"""One instance per controller. Counts every engagement, alarms sparsely.
|
|
|
|
`note()` is called once per write with whether the clamp bit; the caller ORs
|
|
in the loop's own `i_clamped` / `clamped` flags, which is where the winding
|
|
is actually visible on a default install (see the module note).
|
|
"""
|
|
|
|
def __init__(self):
|
|
self.active = False
|
|
self.engagements = 0 # every engagement, never deduplicated
|
|
# UX-08: the same count split by alarm type, so the status page names
|
|
# each with its own alarm's title after the episode clears.
|
|
self.over_rating = 0 # bug=True: "Power request too high"
|
|
self.bound = 0 # bug=False: "Battery is not keeping up"
|
|
self.episodes = 0 # rising edges
|
|
# SAFETY-85: episodes with a bug=False engagement. The status row
|
|
# counts these; counting cycles read 861 on one rig run.
|
|
self.bound_episodes = 0
|
|
self._bound_counted = False
|
|
self.since: datetime | None = None
|
|
self.last_alarm: datetime | None = None
|
|
self.reason = ""
|
|
# UX-08: the numbers behind `reason`, for the notification - set under
|
|
# the same rule as `reason`, so both always describe the same tick.
|
|
self.facts = ()
|
|
self.clean_cycles = 0
|
|
# SAFETY-41. Consecutive engaged cycles, and whether THIS episode has
|
|
# spoken yet - the two the escalation clause needs.
|
|
self.run_len = 0
|
|
self.spoken = False
|
|
# SAFETY-40. Did this episode include a genuine over-rating (`bug`), and
|
|
# has THAT spoken? A bug has its own budget, so anti-windup notices can
|
|
# never delay it; with bug=True throughout, cadence is as before.
|
|
self.bug = False
|
|
self.bug_spoken = False
|
|
self.last_bug_alarm: datetime | None = None
|
|
|
|
def _due(self, last, now) -> bool:
|
|
return last is None or (now - last).total_seconds() >= REASSERT_S
|
|
|
|
def _speak(self, now, bug) -> bool:
|
|
self.last_alarm = now
|
|
self.spoken = True
|
|
if bug:
|
|
self.last_bug_alarm = now
|
|
self.bug_spoken = True
|
|
return True
|
|
|
|
def note(self, engaged: bool, reason: str, now: datetime,
|
|
bug: bool = True, facts=()) -> bool:
|
|
"""Advance one cycle. True when the caller should raise the alarm.
|
|
|
|
`now` is the caller's clock, the same discipline as
|
|
`SignWatchdog.evaluate()` and `FreezeWatchdog.evaluate()`. `bug` False
|
|
is the integrator bound (SAFETY-40): counted and notified like the
|
|
rest, but it never replaces a bug's text within an episode.
|
|
"""
|
|
if engaged:
|
|
self.engagements += 1
|
|
if bug:
|
|
self.over_rating += 1
|
|
else:
|
|
self.bound += 1
|
|
self.clean_cycles = 0
|
|
self.run_len += 1
|
|
rising = not self.active
|
|
if rising:
|
|
self.active = True
|
|
self.since = now
|
|
self.episodes += 1
|
|
self.spoken = False
|
|
self.bug = False
|
|
self.bug_spoken = False
|
|
self._bound_counted = False
|
|
if not bug and not self._bound_counted:
|
|
self._bound_counted = True
|
|
self.bound_episodes += 1
|
|
if bug or not self.bug:
|
|
self.reason = reason
|
|
self.facts = tuple(facts)
|
|
self.bug = self.bug or bug
|
|
if bug and not self.bug_spoken and self._due(self.last_bug_alarm, now):
|
|
return self._speak(now, True)
|
|
if rising:
|
|
# ⚠️ SAFETY-41. THE RISING EDGE IS RATE-LIMITED TOO, and it did
|
|
# not used to be. It fired unconditionally, so a condition that
|
|
# engages and clears repeatedly raised one notification PER
|
|
# EPISODE no matter how fast the episodes came - and
|
|
# `REASSERT_S` never entered into it, because it only governs
|
|
# the still-engaged path below.
|
|
#
|
|
# That is the shape the live install actually produced: 48
|
|
# engagements in 25 minutes, scattered roughly one cycle in six.
|
|
# At a ~5 s cycle the gaps are ~25 s of clean cycles, which
|
|
# exceeds CLEAR_CYCLES x 5 s = 15 s, so the alarm DE-ASSERTED
|
|
# between almost every engagement and each one came back as a
|
|
# fresh rising edge. Measured on this class over that window:
|
|
# 48 episodes and 48 notifications at REASSERT_S = 30 - and
|
|
# still 48 at REASSERT_S = 150. Raising the re-assert removed
|
|
# NONE of them, because none of them came from the re-assert.
|
|
#
|
|
# So the interval now bounds the NOTIFICATION RATE rather than
|
|
# the repeat rate within one episode: `last_alarm` survives a
|
|
# de-assert, and a new episode speaks only if the interval has
|
|
# elapsed. ⚠️ NOTHING IS DROPPED. Every engagement is still
|
|
# counted, every episode still increments, the action-log row is
|
|
# still written on every engaged cycle, and a genuine runaway -
|
|
# which by definition keeps engaging - is picked up by the
|
|
# re-assert path within REASSERT_S. Delay, not loss.
|
|
if self._due(self.last_alarm, now):
|
|
return self._speak(now, self.bug)
|
|
return False
|
|
# ⚠️ THE ESCALATION. An episode that has not yet spoken because the
|
|
# budget was spent by an EARLIER, unrelated episode speaks anyway
|
|
# once it has engaged SUSTAINED_CYCLES times in a row. A runaway
|
|
# engages every cycle and reaches it; the scattered one-cycle
|
|
# railing that produced the live flood never does.
|
|
if not self.spoken and self.run_len >= SUSTAINED_CYCLES:
|
|
return self._speak(now, self.bug)
|
|
# Still engaged: re-assert on the slow cadence, not every cycle.
|
|
if self._due(self.last_alarm, now):
|
|
return self._speak(now, self.bug)
|
|
return False
|
|
|
|
self.run_len = 0
|
|
if not self.active:
|
|
return False
|
|
self.clean_cycles += 1
|
|
if self.clean_cycles >= CLEAR_CYCLES:
|
|
self.active = False
|
|
self.since = None
|
|
# ⚠️ `last_alarm` DELIBERATELY SURVIVES THE DE-ASSERT. Clearing it
|
|
# here is what made the next rising edge speak immediately however
|
|
# soon it arrived, which is the flood described at the rising edge
|
|
# above. The episode ends; the notification budget does not reset
|
|
# with it.
|
|
self.reason = ""
|
|
self.facts = ()
|
|
self.bug = False
|
|
self.clean_cycles = 0
|
|
return False
|