Closes CI-06. Branched off release/1.0 @ 86eaef2. Test-only — no app/ file is touched, nothing is skipped, disabled or marked xfail, and no timeout was widened.test_peakguard.py still carries all of SAFETY-07 and SAFETY-25's forced-discharge SoC floor, and it gains a check.
1. Reproduced deliberately, before any theory
A genuine failing run of the unmodified file was captured under CPU contention (--cpus=0.5, run 7 of 8, tree rsync'd to a plain path and mounted :ro in goodwe-addon-ci, bytecode purged and verified at zero):
9 FAILED: the real cycle files the peak claim and the real arbiter binds the real loop with it,
...and the number written to the inverter is the guard's floor,
one cycle of the intent form is exactly the delivered form plus the command standing on the wire,
the REAL Controller reaches the floor-limited branch, so main.py hands decide() a state of charge at all,
...and the number written to the inverter is not a forced discharge,
...and the operator is TOLD THE PEAK WILL BE EXCEEDED, exactly once,
...once per episode, not once per cycle,
the status page carries exactly one peak-guard row and it is NOT ok,
...and it reaches the action log too
Three of those are named on the card. Instrumentation in that same run says exactly why:
And it was then reproduced on demand: eight runs of the unmodified file started at eight chosen wall-clock offsets 12 s apart, so their fixtures land at successively less of the quarter remaining.
start (epoch mod 900)
795
807
819
831
843
855
867
879
result
ok
ok
2 FAIL
1 FAIL
5 FAIL
ok
ok
ok
Starting the identical file twelve seconds later flips it from green to red. At 831 the single failing check is "one cycle of the intent form is exactly the delivered form plus the command standing on the wire" — mechanism 1 alone, isolated.
2. The cause — twice, and both times 900 / remaining applied to a wall-clock gap
(1) The cross-time projection
_proj3=PeakTracker.project(b3.p1.averager,datetime.now(timezone.utc))# a SECOND projection_intent=_shipped_decide(_proj3,batt_w=STANDING,**_kw)check("one cycle of the intent form is exactly the delivered form plus ...",near(_intent.floor_w-b3.peak_guard.floor_w,STANDING,1.5))# ...against the FIRST
b3.peak_guard.floor_w was decided inside cycle() from one projection; _intent from a second one taken after the cycle returned — three lines under a comment stating that both forms run "against the same frozen projection". They did not.
excess = 900 × (projected − ceiling) / remaining has time-derivative excess / remaining, so at a 1500 W floor the 1.5 W tolerance bought exactly remaining / 1000 seconds. Measured by bisection on the shipped decide():
remaining
880 s
500 s
200 s
100 s
30 s
10 s
budget
333 ms
225 ms
106 ms
56 ms
18 ms
6 ms
The gap between those two statements measured 12.7 – 46.3 ms over 28 instrumented runs (median 24.1 ms). So the check failed whenever remaining_s < gap_ms — a gap_ms / 900 chance per run, uniform in the quarter, and it is bigger on a busier machine. The captured failure sits at ratio = 1.017: 23.7 ms of budget against a 24.1 ms gap.
Fix:_recording_decide keeps the Projectionobject the real cycle graded; both forms are decided from it. The clock is out of the comparison, so the tolerance goes to the file's default 1e-6 Winstead of being widened. Measured on a real cycle: re-deciding the delivered form from that object reproduces the controller's own floor to 0.000000000000 W, and the identity holds to 0.000000000000 W.
It also gains a check: that re-decide now pins the test's decide() kwargs as main.py's actual call rather than an approximation of it.
(2) The fixture's own room
wired_over_limit() laid telegrams with range(0, span + 1, 5), so 0–5 s of the open block were uncovered by construction. Uncovered seconds are credited at the held value, and excess multiplies the loss back up by 900 / remaining:
Measured on the shipped helper, asking for a 1500 W floor, drift 30 ms:
remaining
600 s
100 s
30 s
12 s
8 s
2 s
as shipped
1499.8
1499.3
1497.9
612.7 (−59 %)
610.8 (−59 %)
NOT ENGAGED
with the fix
1499.8
1499.3
1497.9
1494.9
1492.4
1469.5
In the captured failure the first fixture delivered 800.8 W where 1500 W was asked, and 800 W does not bind a loop that already wants 1000 W — so c.reason read loop -> 1000 W, with no "peak" in it, and the write was 1000 W rather than the floor. The check was right; the fixture had silently changed the site being graded. The SAFETY-24 fixture a few lines later hit rem_helper = 1.0 and all four retry attempts came back not engaged — four attempts take ~40 ms and all land in the same dying quarter, which is why the existing retry cannot help.
Fix: one telegram stamped at now (which is what a real meter does and what the docstring already claimed), plus FIXTURE_ROOM_S — the helper waits out the tail of a quarter it cannot size a fixture in. 24 s is derived, not picked, in SAFETY-41's manner: requiring the delivered floor within 1 % of the floor asked for, at this file's smallest fixture and the measured 0.1 s worst-case lag,
The wait costs at most 24 s on ~2.6 % of runs. The alternative was a suite that silently graded a −59 % fixture. Independent confirmation of the constant: the captured failure is at 23.7 s.
3. Proof
(pass rates and the after-side of the targeted table are posted as a comment below — the campaigns are still running as this is opened.)
Full suite on the fixed tree: 21/21 files green.
4. Audit — the same shape elsewhere
Every other PeakTracker.project() in test_peakguard.py (21 of them) takes a fixed B + timedelta(...); line 1297 was the only one reading the real clock, and it is gone. The two remaining 1.5 W tolerances in the file both compare a value to the guard from the same cycle, so no clock crosses them.
Two other files on the required context carry a real wall-clock margin. Neither is touched here; both are reported with a measured number rather than a worry:
test_freeze.py part 3 — a 16 s observation window against a 10 s hold, alarm measured at 11.0 s on 16 polls. One poll of bottom margin. The file already records that the rig transport "failed 1 run in 3" on this section. Strongest remaining candidate.
test_loop.py — worst inter-cycle gap asserted <= FLOOR + 1.5 with FLOOR = 3.0; measured 3.0 s against 4.5 s. A 1.5 s scheduler stall in a 12 s window is red.
On the card's pointer to SAFETY-30: I could not find a "site hold 0.8 s inside a 2.8 s window" anywhere in the tree, and I am reporting that rather than inventing a match. What exists is unit_sign_disagree_s = 85 s, justified as 2.8× the 30 s firmware veto (a config-constant derivation in DECISIONS/SAFETY-30-per-unit-sign.md, not a test margin), and site_sign_disagree_s = 325 s against a measured 215 s worst legitimate opposition at N=2 — a 1.51× margin that the file argues for explicitly. Neither is a wall-clock race of the CI-06 kind.
Closes CI-06. Branched off `release/1.0` @ `86eaef2`. **Test-only — no `app/` file is touched, nothing is skipped, disabled or marked xfail, and no timeout was widened.** `test_peakguard.py` still carries all of SAFETY-07 and SAFETY-25's forced-discharge SoC floor, and it gains a check.
## 1. Reproduced deliberately, before any theory
A genuine failing run of the **unmodified** file was captured under CPU contention (`--cpus=0.5`, run 7 of 8, tree rsync'd to a plain path and mounted `:ro` in `goodwe-addon-ci`, bytecode purged and verified at zero):
```
9 FAILED: the real cycle files the peak claim and the real arbiter binds the real loop with it,
...and the number written to the inverter is the guard's floor,
one cycle of the intent form is exactly the delivered form plus the command standing on the wire,
the REAL Controller reaches the floor-limited branch, so main.py hands decide() a state of charge at all,
...and the number written to the inverter is not a forced discharge,
...and the operator is TOLD THE PEAK WILL BE EXCEEDED, exactly once,
...once per episode, not once per cycle,
the status page carries exactly one peak-guard row and it is NOT ok,
...and it reaches the action log too
```
Three of those are named on the card. Instrumentation in that same run says exactly why:
```
b3 remaining_at_cycle=23.7s gap=24.1ms budget=23.7ms ratio=1.017 <- (1) below
wired attempt=0 rem_helper=25.1 want=1500 got=800.8 eng=True <- (2) below
wired attempt=0 rem_helper=1.0 want=1500 got=None eng=False <- (2), retry
wired attempt=1 rem_helper=1.0 want=1500 got=None eng=False
wired attempt=2 rem_helper=1.0 want=1500 got=None eng=False
wired attempt=3 rem_helper=1.0 want=1500 got=None eng=False
```
And it was then reproduced **on demand**: eight runs of the unmodified file started at eight chosen wall-clock offsets 12 s apart, so their fixtures land at successively less of the quarter remaining.
| start (epoch mod 900) | 795 | 807 | **819** | **831** | **843** | 855 | 867 | 879 |
|---|---|---|---|---|---|---|---|---|
| result | ok | ok | **2 FAIL** | **1 FAIL** | **5 FAIL** | ok | ok | ok |
**Starting the identical file twelve seconds later flips it from green to red.** At 831 the single failing check is *"one cycle of the intent form is exactly the delivered form plus the command standing on the wire"* — mechanism 1 alone, isolated.
## 2. The cause — twice, and both times `900 / remaining` applied to a wall-clock gap
### (1) The cross-time projection
```python
_proj3 = PeakTracker.project(b3.p1.averager, datetime.now(timezone.utc)) # a SECOND projection
_intent = _shipped_decide(_proj3, batt_w=STANDING, **_kw)
check("one cycle of the intent form is exactly the delivered form plus ...",
near(_intent.floor_w - b3.peak_guard.floor_w, STANDING, 1.5)) # ...against the FIRST
```
`b3.peak_guard.floor_w` was decided inside `cycle()` from one projection; `_intent` from a second one taken after the cycle returned — three lines under a comment stating that both forms run *"against the same frozen projection"*. They did not.
`excess = 900 × (projected − ceiling) / remaining` has time-derivative `excess / remaining`, so at a 1500 W floor the 1.5 W tolerance bought exactly **`remaining / 1000` seconds**. Measured by bisection on the shipped `decide()`:
| remaining | 880 s | 500 s | 200 s | 100 s | 30 s | 10 s |
|---|---|---|---|---|---|---|
| budget | 333 ms | 225 ms | 106 ms | 56 ms | 18 ms | **6 ms** |
The gap between those two statements measured **12.7 – 46.3 ms** over 28 instrumented runs (median 24.1 ms). So the check failed whenever `remaining_s < gap_ms` — a **`gap_ms / 900`** chance per run, uniform in the quarter, and it is bigger on a busier machine. The captured failure sits at `ratio = 1.017`: 23.7 ms of budget against a 24.1 ms gap.
**Fix:** `_recording_decide` keeps the `Projection` **object** the real cycle graded; both forms are decided from it. The clock is out of the comparison, so the tolerance goes to the file's default `1e-6 W` **instead of being widened**. Measured on a real cycle: re-deciding the delivered form from that object reproduces the controller's own floor to `0.000000000000 W`, and the identity holds to `0.000000000000 W`.
It also **gains** a check: that re-decide now pins the test's `decide()` kwargs as main.py's actual call rather than an approximation of it.
### (2) The fixture's own room
`wired_over_limit()` laid telegrams with `range(0, span + 1, 5)`, so **0–5 s of the open block were uncovered by construction**. Uncovered seconds are credited at the held value, and `excess` multiplies the loss back up by `900 / remaining`:
```
floor ≈ want × (1 − uncovered/900) − net_w × uncovered / remaining
```
Measured on the shipped helper, asking for a 1500 W floor, drift 30 ms:
| remaining | 600 s | 100 s | 30 s | **12 s** | **8 s** | **2 s** |
|---|---|---|---|---|---|---|
| as shipped | 1499.8 | 1499.3 | 1497.9 | **612.7 (−59 %)** | **610.8 (−59 %)** | **NOT ENGAGED** |
| with the fix | 1499.8 | 1499.3 | 1497.9 | 1494.9 | 1492.4 | 1469.5 |
In the captured failure the first fixture delivered **800.8 W where 1500 W was asked**, and 800 W does not bind a loop that already wants 1000 W — so `c.reason` read `loop -> 1000 W`, with no `"peak"` in it, and the write was 1000 W rather than the floor. The check was right; **the fixture had silently changed the site being graded.** The SAFETY-24 fixture a few lines later hit `rem_helper = 1.0` and all four retry attempts came back not engaged — four attempts take ~40 ms and all land in the same dying quarter, which is why the existing retry cannot help.
**Fix:** one telegram stamped at `now` (which is what a real meter does and what the docstring already claimed), plus `FIXTURE_ROOM_S` — the helper waits out the tail of a quarter it cannot size a fixture in. **24 s is derived, not picked**, in SAFETY-41's manner: requiring the delivered floor within 1 % of the floor asked for, at this file's smallest fixture and the measured 0.1 s worst-case lag,
```
net_w × lag / (remaining × want) ≤ 0.01 − lag/900
remaining ≥ 3500 × 0.1 / (1500 × 0.00989) = 23.6 s
```
The wait costs at most 24 s on ~2.6 % of runs. The alternative was a suite that silently graded a −59 % fixture. Independent confirmation of the constant: the captured failure is at **23.7 s**.
## 3. Proof
*(pass rates and the after-side of the targeted table are posted as a comment below — the campaigns are still running as this is opened.)*
Full suite on the fixed tree: **21/21 files green**.
## 4. Audit — the same shape elsewhere
Every other `PeakTracker.project()` in `test_peakguard.py` (21 of them) takes a fixed `B + timedelta(...)`; line 1297 was the only one reading the real clock, and it is gone. The two remaining 1.5 W tolerances in the file both compare a value to the guard **from the same cycle**, so no clock crosses them.
Two other files on the required context carry a real wall-clock margin. Neither is touched here; both are reported with a measured number rather than a worry:
* **`test_freeze.py` part 3** — a **16 s** observation window against a **10 s** hold, alarm measured at **11.0 s** on 16 polls. One poll of bottom margin. The file already records that the rig transport *"failed 1 run in 3"* on this section. **Strongest remaining candidate.**
* **`test_loop.py`** — worst inter-cycle gap asserted `<= FLOOR + 1.5` with `FLOOR = 3.0`; measured **3.0 s against 4.5 s**. A 1.5 s scheduler stall in a 12 s window is red.
**On the card's pointer to SAFETY-30:** I could not find a "site hold 0.8 s inside a 2.8 s window" anywhere in the tree, and I am reporting that rather than inventing a match. What exists is `unit_sign_disagree_s = 85 s`, justified as **2.8× the 30 s firmware veto** (a config-constant derivation in `DECISIONS/SAFETY-30-per-unit-sign.md`, not a test margin), and `site_sign_disagree_s = 325 s` against a measured **215 s** worst legitimate opposition at N=2 — a **1.51×** margin that the file argues for explicitly. Neither is a wall-clock race of the CI-06 kind.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
All runs: tree rsync'd to a plain path, bytecode purged and verified at zero, mounted :ro in goodwe-addon-ci, python3 -B test_peakguard.py.
campaign
shipped file (86eaef2)
with this branch
4-way parallel loops, 12 runs each
48 / 48 pass
48 / 48 pass
instrumented loops (margins recorded)
20 / 20 pass
10 / 10 pass
--cpus=0.5 CPU-starved, 8 runs each
15 / 16 pass — ⚠️1 FAIL, 9 checks red
16 / 16 pass
targeted: 8 starts, 12 s apart, aimed at the tail of a quarter
5 / 8 pass — ⚠️3 FAIL
⭐8 / 8 pass
total
84 free-running + 8 targeted, 4 failing runs
82 free-running + 8 targeted, 0 failing runs
Plus the full suite on the fixed tree: 21/21 files green, and checks / unit (pull_request) is green on this PR.
The targeted table, side by side
start (epoch mod 900)
795
807
819
831
843
855
867
879
shipped
ok
ok
2 FAIL
1 FAIL
5 FAIL
ok
ok
ok
this branch
ok
ok
ok
ok
ok
ok
ok
ok
The margin, before and after
Before, the check's budget was remaining / 1000 seconds and the gap it had to fit in measured 12.7 – 46.3 ms over 28 runs (median 24.1 ms). The captured failure is at ratio = 1.017 — 23.7 ms of budget against a 24.1 ms gap.
After, there is no budget because there is no clock in the comparison. Ten instrumented runs, spread over the quarter from 119 s to 898 s of remaining:
Zero or one unit in the last place of a double, against a 1e-6 W tolerance — ten orders of magnitude of margin, and flat across the quarter, which is the property the old form did not have at any tolerance.
The fix is pinned against being undone
Putting the clock back — literally release/1.0's own line, _proj3 = PeakTracker.project(b3.p1.averager, datetime.now(timezone.utc)), with nothing else changed — is now caught deterministically:
mut run 1 rc=1 failing=1 :: FAIL the delivered form re-decided from the cycle's OWN projection...
mut run 2 rc=1 failing=1 :: FAIL the delivered form re-decided from the cycle's OWN projection...
mut run 3 rc=1 failing=1 :: FAIL the delivered form re-decided from the cycle's OWN projection...
mut run 4 rc=1 failing=1 :: FAIL the delivered form re-decided from the cycle's OWN projection...
4 of 4, at four different points in the quarter, one check each. That is not luck: the reintroduced error is ≈ 84 / remaining W, so it is ≥ 0.093 W anywhere in the quarter against a 1e-6 W tolerance, and cannot pass. The old 1.5 W form could not see this mutation ~97 % of the time.
⚠️ What I did NOT reproduce, stated plainly
The card's ~20 % figure did not reproduce as a random per-run rate on this machine. Free-running, the shipped file failed 1 run in 84 (~1.2 %). That is not a contradiction — it is the same law with a different constant. The failure condition is remaining_s < gap_ms, so the per-run rate is E[gap_ms] / 900, and gap is whatever the machine gives that stretch of Python: 12.7–46.3 ms measured here → 1.4–5 %. A 20 % rate corresponds to a ~180 ms gap, which is exactly what a sequential 21-file suite or a busier runner produces, and the full sequential suite is where this was first seen. The mechanism is the same and the fix removes the term entirely, but the honest statement of the rate is "proportional to the machine's scheduling jitter", not "one in five everywhere".
Which is why the targeted experiment is the load-bearing evidence rather than the pass counts: it does not depend on the rate at all. Same file, start twelve seconds later, green becomes red.
Cost
FIXTURE_ROOM_S waits out the tail of a quarter on ~2.6 % of runs, up to 24 s. Observed after-run wall times: 88–140 s against 86–107 s before, on a machine carrying 16 concurrent containers for these campaigns. Well inside the workflow's 70-minute budget.
## Proof — pass rates before and after
All runs: tree rsync'd to a plain path, bytecode purged and **verified at zero**, mounted `:ro` in `goodwe-addon-ci`, `python3 -B test_peakguard.py`.
| campaign | shipped file (`86eaef2`) | with this branch |
|---|---|---|
| 4-way parallel loops, 12 runs each | **48 / 48** pass | **48 / 48** pass |
| instrumented loops (margins recorded) | **20 / 20** pass | **10 / 10** pass |
| `--cpus=0.5` CPU-starved, 8 runs each | **15 / 16** pass — ⚠️ **1 FAIL, 9 checks red** | **16 / 16** pass |
| **targeted**: 8 starts, 12 s apart, aimed at the tail of a quarter | **5 / 8** pass — ⚠️ **3 FAIL** | ⭐ **8 / 8** pass |
| **total** | 84 free-running + 8 targeted, **4 failing runs** | 82 free-running + 8 targeted, **0 failing runs** |
Plus the full suite on the fixed tree: **21/21 files green**, and `checks / unit (pull_request)` is **green on this PR**.
### The targeted table, side by side
| start (epoch mod 900) | 795 | 807 | **819** | **831** | **843** | 855 | 867 | 879 |
|---|---|---|---|---|---|---|---|---|
| shipped | ok | ok | **2 FAIL** | **1 FAIL** | **5 FAIL** | ok | ok | ok |
| this branch | ok | ok | **ok** | **ok** | **ok** | ok | ok | ok |
### The margin, before and after
Before, the check's budget was `remaining / 1000` seconds and the gap it had to fit in measured **12.7 – 46.3 ms** over 28 runs (median 24.1 ms). The captured failure is at `ratio = 1.017` — 23.7 ms of budget against a 24.1 ms gap.
After, there is no budget because there is no clock in the comparison. Ten instrumented runs, spread over the quarter from 119 s to 898 s of remaining:
```
remaining_at_cycle=897.6s identity_err=-4.5e-13W real_vs_redecide=0W
remaining_at_cycle=779.1s identity_err=+4.5e-13W real_vs_redecide=0W
remaining_at_cycle=678.2s identity_err= 0W real_vs_redecide=0W
remaining_at_cycle=657.4s identity_err= 0W real_vs_redecide=0W
remaining_at_cycle=585.0s identity_err=+4.5e-13W real_vs_redecide=0W
remaining_at_cycle=551.6s identity_err=-4.5e-13W real_vs_redecide=0W
remaining_at_cycle=438.4s identity_err=-4.5e-13W real_vs_redecide=0W
remaining_at_cycle=327.8s identity_err= 0W real_vs_redecide=0W
remaining_at_cycle=221.9s identity_err= 0W real_vs_redecide=0W
remaining_at_cycle=119.4s identity_err= 0W real_vs_redecide=0W
```
Zero or one unit in the last place of a double, against a `1e-6 W` tolerance — **ten orders of magnitude of margin, and flat across the quarter**, which is the property the old form did not have at any tolerance.
### The fix is pinned against being undone
Putting the clock back — literally `release/1.0`'s own line, `_proj3 = PeakTracker.project(b3.p1.averager, datetime.now(timezone.utc))`, with nothing else changed — is now caught **deterministically**:
```
mut run 1 rc=1 failing=1 :: FAIL the delivered form re-decided from the cycle's OWN projection...
mut run 2 rc=1 failing=1 :: FAIL the delivered form re-decided from the cycle's OWN projection...
mut run 3 rc=1 failing=1 :: FAIL the delivered form re-decided from the cycle's OWN projection...
mut run 4 rc=1 failing=1 :: FAIL the delivered form re-decided from the cycle's OWN projection...
```
4 of 4, at four different points in the quarter, one check each. That is not luck: the reintroduced error is `≈ 84 / remaining` W, so it is **≥ 0.093 W anywhere in the quarter** against a `1e-6 W` tolerance, and cannot pass. The old 1.5 W form could not see this mutation ~97 % of the time.
## ⚠️ What I did NOT reproduce, stated plainly
**The card's ~20 % figure did not reproduce as a random per-run rate on this machine.** Free-running, the shipped file failed **1 run in 84** (~1.2 %). That is not a contradiction — it is the same law with a different constant. The failure condition is `remaining_s < gap_ms`, so the per-run rate is `E[gap_ms] / 900`, and `gap` is whatever the machine gives that stretch of Python: **12.7–46.3 ms measured here → 1.4–5 %.** A 20 % rate corresponds to a ~180 ms gap, which is exactly what a **sequential 21-file suite** or a busier runner produces, and the full sequential suite is where this was first seen. The mechanism is the same and the fix removes the term entirely, but the honest statement of the rate is *"proportional to the machine's scheduling jitter"*, not *"one in five everywhere"*.
Which is why the targeted experiment is the load-bearing evidence rather than the pass counts: it does not depend on the rate at all. **Same file, start twelve seconds later, green becomes red.**
### Cost
`FIXTURE_ROOM_S` waits out the tail of a quarter on ~2.6 % of runs, up to 24 s. Observed after-run wall times: 88–140 s against 86–107 s before, on a machine carrying 16 concurrent containers for these campaigns. Well inside the workflow's 70-minute budget.
`test_peakguard.py` is flaky on the required `checks / unit (pull_request)`
context. Reproduced, measured and removed - not widened, and nothing skipped:
the file still carries all of SAFETY-07 and SAFETY-25, and gains a check.
TWO MECHANISMS, both the control law's own gain `900 / remaining` applied to a
wall-clock gap this file never meant to have.
1. THE CROSS-TIME PROJECTION (the "one cycle of the intent form..." check).
`_proj3` was a SECOND `PeakTracker.project(b3.p1.averager, datetime.now())`
taken after the real cycle had already returned, and its result was compared
against a floor decided from the FIRST projection - while the comment three
lines above says both forms run "against the same frozen projection".
`excess = 900 x (projected - ceiling) / remaining` moves at `excess /
remaining` W/s, so at a 1500 W floor the 1.5 W tolerance bought exactly
`remaining / 1000` seconds: 333 ms with 880 s left in the quarter and 6 ms
with 10 s left (measured by bisection, probe in the PR). The gap between the
two statements measured 12.7-39.5 ms over 26 instrumented runs, so the check
was decided by WHERE IN THE QUARTER it ran, about `gap_ms / 900` of the time.
FIX: `_recording_decide` keeps the Projection OBJECT the real cycle graded.
Both forms are now decided from it, the clock is out of the comparison, and
the tolerance is EXACT rather than 1.5 W. Measured on a real cycle: the
delivered form re-decided from that object reproduces the controller's own
floor to 0.000000000000 W, and the identity holds to 0.000000000000 W.
It also gains a check - that re-decide now pins `_kw` as main.py's actual
call rather than an approximation of it.
2. THE FIXTURE'S OWN ROOM (the three `wired_over_limit` checks named on the
card: "the real cycle files the peak claim...", "...and the number written to
the inverter is the guard's floor", "the STATUS PAGE row for a floor-limited
guard..."). `range(0, span + 1, 5)` leaves 0-5 s of the open block uncovered
by construction, and `excess` multiplies an uncovered second back up by
`900 / remaining`:
floor ~ want x (1 - uncovered/900) - net_w x uncovered / remaining
Measured on this tree, shipped: at 12 s of remaining a fixture that asked for
a 1500 W floor delivered 613 W (-59 %), and at 2 s the guard did not engage
at all - which fails all three of those checks together. The four-attempt
retry cannot help, because four attempts take ~40 ms and all land in the same
dying quarter.
FIX: one telegram stamped at `now` (which is what a real meter does and what
the docstring already claimed), plus FIXTURE_ROOM_S - the helper waits out
the tail of a quarter it cannot size a fixture in. 24 s is DERIVED from the
1 % accuracy the helper promises, at this file's smallest fixture and the
measured 0.1 s worst-case lag, not picked:
net_w x lag / (remaining x want) <= 0.01 - lag/900
remaining >= 3500 x 0.1 / (1500 x 0.00989) = 23.6 s
Same probe after the fix: -0.3 % at 12 s, -0.5 % at 8 s, still engaged at 2 s.
NO TIMEOUT WAS WIDENED, no check skipped, no xfail. One tolerance was made
STRICTER (1.5 W -> exact) because removing the clock made exactness possible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes CI-06. Branched off
release/1.0@86eaef2. Test-only — noapp/file is touched, nothing is skipped, disabled or marked xfail, and no timeout was widened.test_peakguard.pystill carries all of SAFETY-07 and SAFETY-25's forced-discharge SoC floor, and it gains a check.1. Reproduced deliberately, before any theory
A genuine failing run of the unmodified file was captured under CPU contention (
--cpus=0.5, run 7 of 8, tree rsync'd to a plain path and mounted:roingoodwe-addon-ci, bytecode purged and verified at zero):Three of those are named on the card. Instrumentation in that same run says exactly why:
And it was then reproduced on demand: eight runs of the unmodified file started at eight chosen wall-clock offsets 12 s apart, so their fixtures land at successively less of the quarter remaining.
Starting the identical file twelve seconds later flips it from green to red. At 831 the single failing check is "one cycle of the intent form is exactly the delivered form plus the command standing on the wire" — mechanism 1 alone, isolated.
2. The cause — twice, and both times
900 / remainingapplied to a wall-clock gap(1) The cross-time projection
b3.peak_guard.floor_wwas decided insidecycle()from one projection;_intentfrom a second one taken after the cycle returned — three lines under a comment stating that both forms run "against the same frozen projection". They did not.excess = 900 × (projected − ceiling) / remaininghas time-derivativeexcess / remaining, so at a 1500 W floor the 1.5 W tolerance bought exactlyremaining / 1000seconds. Measured by bisection on the shippeddecide():The gap between those two statements measured 12.7 – 46.3 ms over 28 instrumented runs (median 24.1 ms). So the check failed whenever
remaining_s < gap_ms— agap_ms / 900chance per run, uniform in the quarter, and it is bigger on a busier machine. The captured failure sits atratio = 1.017: 23.7 ms of budget against a 24.1 ms gap.Fix:
_recording_decidekeeps theProjectionobject the real cycle graded; both forms are decided from it. The clock is out of the comparison, so the tolerance goes to the file's default1e-6 Winstead of being widened. Measured on a real cycle: re-deciding the delivered form from that object reproduces the controller's own floor to0.000000000000 W, and the identity holds to0.000000000000 W.It also gains a check: that re-decide now pins the test's
decide()kwargs as main.py's actual call rather than an approximation of it.(2) The fixture's own room
wired_over_limit()laid telegrams withrange(0, span + 1, 5), so 0–5 s of the open block were uncovered by construction. Uncovered seconds are credited at the held value, andexcessmultiplies the loss back up by900 / remaining:Measured on the shipped helper, asking for a 1500 W floor, drift 30 ms:
In the captured failure the first fixture delivered 800.8 W where 1500 W was asked, and 800 W does not bind a loop that already wants 1000 W — so
c.reasonreadloop -> 1000 W, with no"peak"in it, and the write was 1000 W rather than the floor. The check was right; the fixture had silently changed the site being graded. The SAFETY-24 fixture a few lines later hitrem_helper = 1.0and all four retry attempts came back not engaged — four attempts take ~40 ms and all land in the same dying quarter, which is why the existing retry cannot help.Fix: one telegram stamped at
now(which is what a real meter does and what the docstring already claimed), plusFIXTURE_ROOM_S— the helper waits out the tail of a quarter it cannot size a fixture in. 24 s is derived, not picked, in SAFETY-41's manner: requiring the delivered floor within 1 % of the floor asked for, at this file's smallest fixture and the measured 0.1 s worst-case lag,The wait costs at most 24 s on ~2.6 % of runs. The alternative was a suite that silently graded a −59 % fixture. Independent confirmation of the constant: the captured failure is at 23.7 s.
3. Proof
(pass rates and the after-side of the targeted table are posted as a comment below — the campaigns are still running as this is opened.)
Full suite on the fixed tree: 21/21 files green.
4. Audit — the same shape elsewhere
Every other
PeakTracker.project()intest_peakguard.py(21 of them) takes a fixedB + timedelta(...); line 1297 was the only one reading the real clock, and it is gone. The two remaining 1.5 W tolerances in the file both compare a value to the guard from the same cycle, so no clock crosses them.Two other files on the required context carry a real wall-clock margin. Neither is touched here; both are reported with a measured number rather than a worry:
test_freeze.pypart 3 — a 16 s observation window against a 10 s hold, alarm measured at 11.0 s on 16 polls. One poll of bottom margin. The file already records that the rig transport "failed 1 run in 3" on this section. Strongest remaining candidate.test_loop.py— worst inter-cycle gap asserted<= FLOOR + 1.5withFLOOR = 3.0; measured 3.0 s against 4.5 s. A 1.5 s scheduler stall in a 12 s window is red.On the card's pointer to SAFETY-30: I could not find a "site hold 0.8 s inside a 2.8 s window" anywhere in the tree, and I am reporting that rather than inventing a match. What exists is
unit_sign_disagree_s = 85 s, justified as 2.8× the 30 s firmware veto (a config-constant derivation inDECISIONS/SAFETY-30-per-unit-sign.md, not a test margin), andsite_sign_disagree_s = 325 sagainst a measured 215 s worst legitimate opposition at N=2 — a 1.51× margin that the file argues for explicitly. Neither is a wall-clock race of the CI-06 kind.🤖 Generated with Claude Code
https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
Proof — pass rates before and after
All runs: tree rsync'd to a plain path, bytecode purged and verified at zero, mounted
:roingoodwe-addon-ci,python3 -B test_peakguard.py.86eaef2)--cpus=0.5CPU-starved, 8 runs eachPlus the full suite on the fixed tree: 21/21 files green, and
checks / unit (pull_request)is green on this PR.The targeted table, side by side
The margin, before and after
Before, the check's budget was
remaining / 1000seconds and the gap it had to fit in measured 12.7 – 46.3 ms over 28 runs (median 24.1 ms). The captured failure is atratio = 1.017— 23.7 ms of budget against a 24.1 ms gap.After, there is no budget because there is no clock in the comparison. Ten instrumented runs, spread over the quarter from 119 s to 898 s of remaining:
Zero or one unit in the last place of a double, against a
1e-6 Wtolerance — ten orders of magnitude of margin, and flat across the quarter, which is the property the old form did not have at any tolerance.The fix is pinned against being undone
Putting the clock back — literally
release/1.0's own line,_proj3 = PeakTracker.project(b3.p1.averager, datetime.now(timezone.utc)), with nothing else changed — is now caught deterministically:4 of 4, at four different points in the quarter, one check each. That is not luck: the reintroduced error is
≈ 84 / remainingW, so it is ≥ 0.093 W anywhere in the quarter against a1e-6 Wtolerance, and cannot pass. The old 1.5 W form could not see this mutation ~97 % of the time.⚠️ What I did NOT reproduce, stated plainly
The card's ~20 % figure did not reproduce as a random per-run rate on this machine. Free-running, the shipped file failed 1 run in 84 (~1.2 %). That is not a contradiction — it is the same law with a different constant. The failure condition is
remaining_s < gap_ms, so the per-run rate isE[gap_ms] / 900, andgapis whatever the machine gives that stretch of Python: 12.7–46.3 ms measured here → 1.4–5 %. A 20 % rate corresponds to a ~180 ms gap, which is exactly what a sequential 21-file suite or a busier runner produces, and the full sequential suite is where this was first seen. The mechanism is the same and the fix removes the term entirely, but the honest statement of the rate is "proportional to the machine's scheduling jitter", not "one in five everywhere".Which is why the targeted experiment is the load-bearing evidence rather than the pass counts: it does not depend on the rate at all. Same file, start twelve seconds later, green becomes red.
Cost
FIXTURE_ROOM_Swaits out the tail of a quarter on ~2.6 % of runs, up to 24 s. Observed after-run wall times: 88–140 s against 86–107 s before, on a machine carrying 16 concurrent containers for these campaigns. Well inside the workflow's 70-minute budget.`test_peakguard.py` is flaky on the required `checks / unit (pull_request)` context. Reproduced, measured and removed - not widened, and nothing skipped: the file still carries all of SAFETY-07 and SAFETY-25, and gains a check. TWO MECHANISMS, both the control law's own gain `900 / remaining` applied to a wall-clock gap this file never meant to have. 1. THE CROSS-TIME PROJECTION (the "one cycle of the intent form..." check). `_proj3` was a SECOND `PeakTracker.project(b3.p1.averager, datetime.now())` taken after the real cycle had already returned, and its result was compared against a floor decided from the FIRST projection - while the comment three lines above says both forms run "against the same frozen projection". `excess = 900 x (projected - ceiling) / remaining` moves at `excess / remaining` W/s, so at a 1500 W floor the 1.5 W tolerance bought exactly `remaining / 1000` seconds: 333 ms with 880 s left in the quarter and 6 ms with 10 s left (measured by bisection, probe in the PR). The gap between the two statements measured 12.7-39.5 ms over 26 instrumented runs, so the check was decided by WHERE IN THE QUARTER it ran, about `gap_ms / 900` of the time. FIX: `_recording_decide` keeps the Projection OBJECT the real cycle graded. Both forms are now decided from it, the clock is out of the comparison, and the tolerance is EXACT rather than 1.5 W. Measured on a real cycle: the delivered form re-decided from that object reproduces the controller's own floor to 0.000000000000 W, and the identity holds to 0.000000000000 W. It also gains a check - that re-decide now pins `_kw` as main.py's actual call rather than an approximation of it. 2. THE FIXTURE'S OWN ROOM (the three `wired_over_limit` checks named on the card: "the real cycle files the peak claim...", "...and the number written to the inverter is the guard's floor", "the STATUS PAGE row for a floor-limited guard..."). `range(0, span + 1, 5)` leaves 0-5 s of the open block uncovered by construction, and `excess` multiplies an uncovered second back up by `900 / remaining`: floor ~ want x (1 - uncovered/900) - net_w x uncovered / remaining Measured on this tree, shipped: at 12 s of remaining a fixture that asked for a 1500 W floor delivered 613 W (-59 %), and at 2 s the guard did not engage at all - which fails all three of those checks together. The four-attempt retry cannot help, because four attempts take ~40 ms and all land in the same dying quarter. FIX: one telegram stamped at `now` (which is what a real meter does and what the docstring already claimed), plus FIXTURE_ROOM_S - the helper waits out the tail of a quarter it cannot size a fixture in. 24 s is DERIVED from the 1 % accuracy the helper promises, at this file's smallest fixture and the measured 0.1 s worst-case lag, not picked: net_w x lag / (remaining x want) <= 0.01 - lag/900 remaining >= 3500 x 0.1 / (1500 x 0.00989) = 23.6 s Same probe after the fix: -0.3 % at 12 s, -0.5 % at 8 s, still engaged at 2 s. NO TIMEOUT WAS WIDENED, no check skipped, no xfail. One tolerance was made STRICTER (1.5 W -> exact) because removing the clock made exactness possible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjUeb134e6926to9b270a66a5