SAFETY-41: escalate a sustained engagement, so a runaway is never held
checks / unit (pull_request) Successful in 4m59s

Review found the case I asked it to construct, and it is not a corner
case. The rate limit alone DELAYS A RUNAWAY: a benign one-cycle clamp
that spends the budget, then a continuous §10.3-shaped runaway, held the
runaway's first notification for up to 140 s - one REASSERT_S minus a
cycle. On the live install, where ordinary railing spends the budget
essentially all the time, a runaway arriving during normal operation
would be delayed with near-certainty. That is the "benign noise
suppresses the bug signal" shape this module's docstring is written
against, and the family the first attempt at this card died of.

THE ESCALATION. An episode that has not yet spoken because an EARLIER,
unrelated episode spent the budget speaks anyway once it has engaged
SUSTAINED_CYCLES = 3 times in a row. The discriminator is run length and
the two populations genuinely differ: the live flood was runs of length
ONE (engage, de-assert, engage), while a runaway engages every cycle.

Measured, and it costs nothing it was protecting:
  live scatter flood (305 cycles, 1-in-6)   11 -> 11 notifications
  chatter 1-engaged/3-clean over 8000 s     50 -> 50 notifications
  pinned 900 s quarter (floor is 5)          6 ->  6 alarms
  runaway delay after a benign episode     140 s -> 10 s

⚠️ MY COUNTERFACTUAL CHECK COULD NOT FAIL FOR THE DEFECT IT NAMED, and
review caught that too. It read `ClampWatchdog().note(True, ...) is True`
- a FRESH object, which only exercises the `last_alarm is None` branch,
the trivially-true one. A fresh watchdog has no earlier episode to be
held back BY. It now SPENDS THE BUDGET FIRST: benign clamp, gap past
CLEAR_CYCLES, then a continuous runaway, and asserts the delay is well
under REASSERT_S. That is the third instance in this file of the family
recorded at line 216 and at the 224 s sweep, and it was guarding the most
safety-critical property in the change.

Also pinned: `last_alarm` SURVIVING the de-assert. Review built the
mutation - if the de-assert also refreshed it, chatter collapses from 50
notifications to 1 and the alarm never fires again, with the suite green.
That is the "alarm that never fires" class, a different bug from one that
fires later. Now caught.

Both new checks verified against the mutations they name: removing the
escalation -> 1 FAILED; refreshing last_alarm on de-assert -> 1 FAILED.

Full suite: 21 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
This commit is contained in:
glenn schrooyen
2026-09-05 18:13:09 +02:00
co-authored by Claude Opus 5
parent 8790c4d38f
commit 3527c8a04c
2 changed files with 93 additions and 10 deletions
+50 -8
View File
@@ -78,7 +78,8 @@ import sys
from datetime import datetime, timedelta, timezone
from app.clamp import (
CLEAR_CYCLES, REASON_CODE, REASSERT_S, ClampWatchdog, clamp_to_rating,
CLEAR_CYCLES, REASON_CODE, REASSERT_S, SUSTAINED_CYCLES, ClampWatchdog,
clamp_to_rating,
is_commandable,
)
from app.allocator import Allocation, UnitPlan, plan
@@ -296,13 +297,54 @@ check(f"...and the rate honours the same interval the pinned case does: at most
check("...while EVERY episode and EVERY engagement is still counted - the "
"budget is on notifications, never on evidence",
scat.episodes == scat.engagements == SCATTER_CYCLES // SCATTER_ONE_IN + 1)
# ⚠️ THE COUNTERFACTUAL. A runaway that begins from quiet must NOT be delayed by
# a budget spent on an earlier, unrelated episode - so a cold watchdog speaks at
# once, and this is the half that would go red if the rate limit were applied
# without regard to how long it has been silent.
check("...and a rising edge from a COLD watchdog still fires immediately, so a "
"runaway beginning from quiet is never held back",
ClampWatchdog().note(True, "cold runaway", T0) is True)
# ⚠️ THE COUNTERFACTUAL, AND THE FIRST VERSION OF IT COULD NOT FAIL FOR THE
# DEFECT IT NAMED. It read `ClampWatchdog().note(True, ...) is True` - a FRESH
# object, so it only ever exercised the `last_alarm is None` branch, which is the
# trivially-true one. A fresh watchdog has no earlier episode to be held back BY.
# That is the third instance in this file of the family recorded at line 216 and
# at the 224 s sweep, and it was guarding the most safety-critical property in
# the change. Review found it.
#
# The stimulus must therefore SPEND THE BUDGET FIRST: a benign one-cycle clamp,
# a gap long enough to de-assert, then a continuous runaway. Without the
# escalation clause the runaway's first notification is held for up to
# REASSERT_S - measured at 140 s before it was added.
_esc = ClampWatchdog()
_esc.note(True, "benign", T0) # spends the budget
for _i in range(1, 1 + CLEAR_CYCLES + 1): # de-assert
_esc.note(False, "", T0 + timedelta(seconds=5.0 * _i))
_RUNAWAY_AT = 5.0 * (CLEAR_CYCLES + 2)
_esc_fires = [5.0 * (CLEAR_CYCLES + 2 + _j)
for _j in range(200)
if _esc.note(True, "runaway",
T0 + timedelta(seconds=5.0 * (CLEAR_CYCLES + 2 + _j)))]
_esc_delay = _esc_fires[0] - _RUNAWAY_AT if _esc_fires else None
print(f" budget spent by a benign episode, then a continuous runaway from "
f"t={_RUNAWAY_AT:.0f}s: first alarm at t={_esc_fires[0]:.0f}s "
f"(delay {_esc_delay:.0f}s)")
check(f"⚠️ a SUSTAINED runaway speaks even when an earlier, unrelated episode "
f"spent the budget - it is not held for REASSERT_S ({_esc_delay:.0f}s "
f"delay, escalation is {SUSTAINED_CYCLES} cycles)",
_esc_delay is not None and _esc_delay < REASSERT_S / 2)
check("...and it speaks at the escalation boundary rather than immediately, "
"which is what keeps the scattered flood suppressed",
_esc_delay >= 5.0 * (SUSTAINED_CYCLES - 1))
# ⚠️ `last_alarm` SURVIVING THE DE-ASSERT IS NOW LOAD-BEARING, AND NOTHING
# PINNED IT. Review built the mutation: if the de-assert path ALSO refreshed
# `last_alarm = now`, the chatter case collapses from 50 notifications to 1 - the
# alarm never fires again. That is the "alarm that never fires" class, which is a
# different bug from one that fires later, and the suite was green on it.
_starve = ClampWatchdog()
_starve_fires = sum(1 for _i in range(1600)
if _starve.note(_i % 4 == 0, "chatter",
T0 + timedelta(seconds=5.0 * _i)))
print(f" chatter 1-engaged/{CLEAR_CYCLES}-clean over 8000 s: "
f"{_starve.episodes} episodes, {_starve_fires} notification(s)")
check(f"a de-assert does not RESET the notification budget, so repeated chatter "
f"cannot starve the alarm to silence ({_starve_fires} notifications over "
f"8000 s, not 1)",
_starve_fires > 10)
# --- THE HYSTERESIS, PINNED AGAINST THE CHATTER DIRECTION --------------------
# ⚠️ REVIEW-FOUND. This used to issue `range(CLEAR_CYCLES - 1)` clean cycles,