DATA-01: thirty days of the real meter, and what AC 3 actually measures
checks / unit (pull_request) Successful in 3m36s
checks / unit (pull_request) Successful in 3m36s
TARIFF-01 and SAFETY-07 both merged with AC 3 - "monthly peak held under
the soft limit in >= 95 % of intervals over a 30-day simulation replay" -
honestly unmet, because two days of capture existed and it was believed
nothing longer could be produced. That belief was a true statement about
the past read as a claim about the future: HA's recorder still held the
whole month, and the reason nobody found out is that the 2026-08-25
export was a one-off script that left no artefact.
So the exporter is committed and runnable now.
scenarios/p1-net-30d-2026-08/ is 2026-08-01 to 2026-08-30, 485,841
samples, 30/30 days read, 0 failed, 4.13 MB gzipped, in the schema the
two committed days already fixed.
VERIFIED AGAINST THE DAYS THAT EXIST. `--verify` re-fetches 2026-08-20
and 2026-08-23 by this exact route and reproduces both committed files
BYTE FOR BYTE after decompression (865,024 B / 15,600 samples and
894,387 B / 16,351 samples). The gzip containers differ by design: this
exporter writes mtime 0 so its output is reproducible, where the one-off
wrote a wall clock.
A FAILED READ IS NOT AN OBSERVATION. A day whose read fails writes no
file, is recorded "read": "FAILED", and the run exits non-zero; the
entity is proved to exist through /api/states before any empty day is
believed, because a typo would otherwise answer 200 [] thirty times and
produce a consistent dataset of nothing. Nothing is interpolated, so the
month's 13 gaps (1291 s, 99.95 % covered) and its 3 `unavailable`
samples stay visible as holes.
SIZE: COMMITTED, DELIBERATELY, AND MEASURED. A git bundle of the whole
history goes from 1.58 MB (release/1.0 @ 86c43eb) to 5.28 MB - +3.70 MB,
x3.3 - and a Home Assistant supervisor clones that on every install.
Downsampling was not on the table: meter_max_age_s is 30 s, so any
coarser resample turns real coverage into modelled meter-gap and moves
the number being measured. "Outside the repo with a pointer" is what
already happened and is why this card exists. The files sit in a
SUBDIRECTORY so the non-recursive globs in test_tariff.py part 9 and
test_peakguard.py part 6 do not pick them up - CI keeps replaying the two
small days instead of growing ~50 s per PR.
AC 3, MEASURED OVER 30 DAYS (tools/replay_ac3.py, which must reproduce
test_peakguard part 6's 189/190 to be believed):
uncontrolled 2847/2879 = 98.89 % of quarters at or under 3500 W
guarded 2850/2879 = 98.99 %
month peak 5029.8 W -> 4107.3 W
Met on the ">= 95 % of intervals" reading - and the uncontrolled figure
is the finding, because a month with no controller at all passes it. Not
met on the other reading of the same sentence: 4107.3 W is still over the
soft limit. The guard pulls 31 of the 32 over-cap quarters down but lands
only 3 under the cap, because CEILING_MONTH_PEAK defends the running
month peak rather than the cap once one is set. Both suites' honesty
notes are corrected to point at the month instead of at its absence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VXjA3LWccN1rJsk3okyPC
This commit is contained in:
co-authored by
Claude Opus 5
parent
86c43eba24
commit
04292c4a08
@@ -35,16 +35,39 @@ NOT allowed to be tested at:
|
||||
10. THE WIRING, through the real `Controller` and its real status page.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
⚠️ AC 3 IS NOT MET, AND PART 9 DOES NOT PRETEND OTHERWISE.
|
||||
⚠️ PART 9 IS A TWO-DAY MEASUREMENT AND SAYS SO. THE 30-DAY ANSWER IS ELSEWHERE
|
||||
(DATA-01), AND IT DOES NOT LAND WHERE ANYONE EXPECTED.
|
||||
|
||||
AC 3 asks for "the monthly peak held under the soft limit in >= 95 % of
|
||||
intervals over a 30-day simulation replay". THERE IS NO 30-DAY DATASET. The
|
||||
infra repo's `sim/scenarios/` holds exactly two days of this house's real meter
|
||||
- 2026-08-20 (the Illusmart failure day) and 2026-08-23 (the healthy reference)
|
||||
- exported on 2026-08-25 because they were about to age out of HA's recorder,
|
||||
and nothing longer was ever captured. `sim/scenarios/README.md` lists the whole
|
||||
inventory; the rest is RS485 bus frames, OTA witness logs and a 578-row rollback
|
||||
poll, none of which carries grid power over calendar time.
|
||||
intervals over a 30-day simulation replay". When this file was written there
|
||||
were two days of this house's real meter - 2026-08-20 (the Illusmart failure
|
||||
day) and 2026-08-23 (the healthy reference) - exported on 2026-08-25 because
|
||||
they were about to age out of HA's recorder.
|
||||
|
||||
⚠️ AND THE SENTENCE THAT USED TO STAND HERE, "nothing longer was ever
|
||||
captured", WAS TRUE ABOUT THE PAST AND WAS READ AS A CLAIM ABOUT THE FUTURE.
|
||||
It was not one. HA's recorder still held the whole month; DATA-01 probed it on
|
||||
2026-08-31 (230 samples in a 20-minute window at 30 days back, 0 at 45) and
|
||||
exported thirty days the same day, into `scenarios/p1-net-30d-2026-08/` with
|
||||
`tools/export_p1_history.py` - which is committed now precisely because the
|
||||
2026-08-25 one-off was not, and its absence is what let the belief survive.
|
||||
|
||||
Over those thirty days the criterion measures 98.89 % UNCONTROLLED and 98.99 %
|
||||
guarded, so it is met - and that is a finding about the CRITERION as much as
|
||||
about the guard, because a month with no controller at all passes it. Read the
|
||||
other way - the monthly PEAK itself under the soft limit - it is not met: the
|
||||
guard takes August's peak from 5029.8 W to 4107.3 W, still over 3500 W. Both
|
||||
numbers, and why they part company, are in
|
||||
`scenarios/p1-net-30d-2026-08/README.md`.
|
||||
|
||||
PART 9 IS DELIBERATELY LEFT GRADING THE TWO COMMITTED DAYS. It is a per-PR
|
||||
regression check and the thirty days cost about fifty seconds of replay; they
|
||||
also live one directory down precisely so this file's glob does not pick them
|
||||
up. `tools/replay_ac3.py` is the thing that runs the month, and it is checked
|
||||
against part 6 of test_peakguard.py rather than trusted:
|
||||
|
||||
python3 tools/replay_ac3.py scenarios --expect 189/190
|
||||
python3 tools/replay_ac3.py scenarios/p1-net-30d-2026-08
|
||||
|
||||
So part 9 replays the two days that exist, reports the percentage over them, and
|
||||
states the sample size in days next to it. It is a MEASUREMENT, not a verdict:
|
||||
@@ -60,8 +83,8 @@ export) for the reason `scenarios/p1-age-2026-08-26.jsonl.gz` already is: the
|
||||
infra repo is private, CI's token is minted for this repository only, and a
|
||||
regression fixture that CI cannot see is not a regression fixture.
|
||||
|
||||
Closing AC 3 needs 30 days of capture. That is TEST-01's replay harness, and it
|
||||
cannot start before somebody starts recording.
|
||||
The recorder's window rolls: 2026-08-01 leaves it around 2026-09-15, which is
|
||||
why the committed copy is the artefact and not the API call.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
@@ -748,9 +771,11 @@ print()
|
||||
print(f" AC 3 MEASURED: {under}/{total_q} quarters = {PCT:.1f} % under the "
|
||||
f"{SHIPPED_CAP_W:.0f} W soft limit, over {replay_days:.1f} DAYS of "
|
||||
f"replay, UNCONTROLLED (no peak shaving was running on either day).")
|
||||
print(f" AC 3 ASKS FOR 30 DAYS. {replay_days:.1f} days exist. "
|
||||
f"AC 3 IS NOT MET and this run does not claim it is - see the module "
|
||||
f"docstring. The figure above is the BASELINE SAFETY-07 has to improve.")
|
||||
print(f" AC 3 ASKS FOR 30 DAYS and this run is {replay_days:.1f}, so it "
|
||||
f"does not settle it either way. The 30 days DO now exist - "
|
||||
f"scenarios/p1-net-30d-2026-08/, DATA-01 - and "
|
||||
f"tools/replay_ac3.py is what runs them; the figure above is the "
|
||||
f"BASELINE SAFETY-07 has to improve, on the two days CI replays.")
|
||||
check("the replay produced a measurable number over a stated sample size, "
|
||||
"rather than a criterion quietly redefined to fit the data",
|
||||
total_q > 100 and replay_days < 30)
|
||||
|
||||
Reference in New Issue
Block a user