DATA-01 - Export 30 days of P1 before the retention window rolls.
What this is
TARIFF-01 AC 3 and SAFETY-07 AC 3 both ask for "monthly peak held under the soft limit in >= 95 % of intervals over a 30-day simulation replay". Both merged with it honestly unmet, on the belief that only two days existed and nothing longer could be produced retroactively.
That belief was a true statement about the past read as a claim about the future. HA's recorder still held the whole month. Probed 2026-08-31: 230 samples in a 20-minute window at 30 days back, 0 at 45. Exported the same day, because the window rolls — 2026-08-01 leaves the recorder around 2026-09-15.
goodwe_controller/scenarios/p1-net-30d-2026-08/ — 2026-08-01 to 2026-08-30, 485,841 samples, 30/30 days read, 0 failed, 4.13 MB gzipped, in the schema the two committed days already fixed.
The check that matters: the exporter reproduces the days that exist
tools/export_p1_history.py --verify re-fetches 2026-08-20 and 2026-08-23 from the live recorder by the exact route used for the month, and diffs them against the committed files:
Byte for byte after decompression, both days. The gzip containers differ and that is deliberate: a gzip header carries an mtime and a stored filename, and this exporter writes mtime 0 so its own output is reproducible, where the 2026-08-25 one-off wrote a wall clock. Container equality would only assert that two files were written in the same second. The transform was also checked offline against the raw HA responses archived in the infra repo, with the same result.
The month's own copies of those two days are byte-identical to the committed ones, and re-reading 2026-08-11 from HA a second time reproduced it byte for byte too.
tools/replay_ac3.py is checked the same way rather than trusted: --expect 189/190 makes it reproduce test_peakguard.py part 6's published totals — the 5029.8 W billed quarter, the 3485.3 W guarded result, the 878 Wh of modelled discharge.
A failed read is not an observation
The distinction the whole dataset stands on:
a day whose read fails writes no file, is recorded "read": "FAILED" with the error, and the run exits non-zero. A missing day is unreadable-and-known, never a silent zero;
a day that reads back empty writes an empty file and is recorded "read": "OK", "samples": 0;
the entity is proved to exist through /api/statesbefore any empty day is believed — a typo in --entity would otherwise answer 200 [] thirty times and produce a beautifully consistent dataset of nothing;
HTTP errors, timeouts and truncated bodies are all retried and all treated as failures, never as silence.
On this run: 30 read, 0 failed, 0 empty.
Gaps are visible, not interpolated
Nothing is filled, carried forward or resampled. 13 gaps over 30 s, 1291 s total, 99.950 % of the month covered, every one listed in manifest.json (with per-day sha256 and counts) and by --report-gaps. Longest 263 s on 2026-08-11.
Three samples carried the state unavailable rather than a number. There is no float to write and the schema has no slot for a non-number, so they are absent from the series and recorded individually under non_numeric_states — and each sits inside a gap that is already listed, which is the honest rendering. The controller's own meter_max_age_s of 30 s is what makes a hole replay as no coverage rather than as an hour of whatever the last sample said.
Size: committed, deliberately, and measured
git bundle of the whole history, before and after:
A full clone more than triples, and a Home Assistant supervisor clones it on every install. Stated, not absorbed. Why this option and not the other two:
Downsampling is not available.meter_max_age_s ships at 30 s — the age past which the controller's ingest stops counting a held reading as coverage — so any coarser resample converts real coverage into modelled meter-gap and moves the very number AC 3 measures. At or under 30 s it saves almost nothing against a 5 s cadence.
"Outside the repo with a pointer" is what already happened, and it is why this card exists: the two days came from a private repo, the export script was thrown away, and three cards then deferred an AC. test_tariff.py states the rule verbatim — a regression fixture CI cannot see is not a regression fixture — and CI's token is minted for this repository only.
So: committed, in a SUBDIRECTORY, which is load-bearing. test_tariff.py part 9 and test_peakguard.py part 6 both glob scenarios/p1-net-*.jsonl.gz non-recursively. Beside the two days, these thirty files would add sixty full-day replays (~50 s) to every PR forever and would break check(... replay_days < 30). One directory down they are visible to CI and invisible to the globs.
AC 3, run, with the actual numbers
485841 samples, 2879 closed quarters (30.0 days' worth)
month 2026-08: peak 5029.8 W uncontrolled -> 4107.3 W guarded
TARIFF-01 AC 3 (UNCONTROLLED): 2847/2879 = 98.89 % at or under 3500 W
SAFETY-07 AC 3 (GUARDED): 2850/2879 = 98.99 %
guard cycles 143810, engaged 1542, blind 195, saturated 0, 8060 Wh
Three findings, not a pass:
Met on the ">= 95 % of intervals" reading — and the criterion turns out to be nearly insensitive. The uncontrolled month, with no controller running at all, already scores 98.89 %. A criterion a house passes by doing nothing does not separate a working guard from an absent one. The card said a result below 95 % would be a finding; a result above it for this reason is one too.
Not met on the other reading of the same sentence. "Monthly peak held under the soft limit", taken as the peak itself, fails: 5029.8 W -> 4107.3 W (-922.5 W, -18.3 %, real money on a capacity tariff) but still over the 3500 W limit. Both readings are printed; neither is hidden.
The guard stops defending the soft limit once the month's peak is set, by design.quarter_ceiling_w uses CEILING_MONTH_PEAK — the billed quantity is the month's maximum, so a quarter under an already-recorded peak costs nothing extra. The 32 over-cap quarters fall on four nights; 31 are pulled down but only 3 land under the cap, because the ceiling being defended by then is the running peak (~4107 W). The one untouched quarter (2026-08-18T08:15Z, 4039.7 W) is the same rule from the other side: the running peak was already 4833 W, so it added nothing to the bill. AC 3 asks the guard for something its own tariff logic deliberately does not optimise.
Also worth an eye: 195 blind engagements over 143,810 cycles, against 5 on the two committed days. It scales with the gaps.
Not in this PR
The suites' checks are unchanged — only their honesty notes, which said the 30 days did not exist. CI still replays the two small days. Nothing was written to Home Assistant: read-only history API, day at a time, 2 s between days, no recorder setting or restart touched. All 15 test files pass locally.
DATA-01 - Export 30 days of P1 before the retention window rolls.
## What this is
[[TARIFF-01]] AC 3 and [[SAFETY-07]] AC 3 both ask for *"monthly peak held under the soft limit in >= 95 % of intervals over a 30-day simulation replay"*. Both merged with it honestly unmet, on the belief that only two days existed and nothing longer could be produced retroactively.
**That belief was a true statement about the past read as a claim about the future.** HA's recorder still held the whole month. Probed 2026-08-31: 230 samples in a 20-minute window at 30 days back, 0 at 45. Exported the same day, because the window rolls — 2026-08-01 leaves the recorder around 2026-09-15.
`goodwe_controller/scenarios/p1-net-30d-2026-08/` — 2026-08-01 to 2026-08-30, **485,841 samples, 30/30 days read, 0 failed, 4.13 MB gzipped**, in the schema the two committed days already fixed.
## The check that matters: the exporter reproduces the days that exist
`tools/export_p1_history.py --verify` re-fetches 2026-08-20 and 2026-08-23 from the live recorder by the exact route used for the month, and diffs them against the committed files:
```
p1-net-2026-08-20.jsonl.gz IDENTICAL: 865024 bytes, 15600 samples
p1-net-2026-08-23.jsonl.gz IDENTICAL: 894387 bytes, 16351 samples
```
**Byte for byte after decompression, both days.** The gzip *containers* differ and that is deliberate: a gzip header carries an mtime and a stored filename, and this exporter writes mtime 0 so its own output is reproducible, where the 2026-08-25 one-off wrote a wall clock. Container equality would only assert that two files were written in the same second. The transform was also checked offline against the raw HA responses archived in the infra repo, with the same result.
The month's own copies of those two days are byte-identical to the committed ones, and re-reading 2026-08-11 from HA a second time reproduced it byte for byte too.
`tools/replay_ac3.py` is checked the same way rather than trusted: `--expect 189/190` makes it reproduce test_peakguard.py part 6's published totals — the 5029.8 W billed quarter, the 3485.3 W guarded result, the 878 Wh of modelled discharge.
## A failed read is not an observation
The distinction the whole dataset stands on:
* a day whose read fails writes **no file**, is recorded `"read": "FAILED"` with the error, and the run exits non-zero. A missing day is unreadable-and-known, never a silent zero;
* a day that reads back **empty** writes an empty file and is recorded `"read": "OK", "samples": 0`;
* the entity is proved to exist through `/api/states` **before any empty day is believed** — a typo in `--entity` would otherwise answer `200 []` thirty times and produce a beautifully consistent dataset of nothing;
* HTTP errors, timeouts and truncated bodies are all retried and all treated as failures, never as silence.
On this run: 30 read, 0 failed, 0 empty.
## Gaps are visible, not interpolated
Nothing is filled, carried forward or resampled. **13 gaps over 30 s, 1291 s total, 99.950 % of the month covered**, every one listed in `manifest.json` (with per-day sha256 and counts) and by `--report-gaps`. Longest 263 s on 2026-08-11.
Three samples carried the state `unavailable` rather than a number. There is no float to write and the schema has no slot for a non-number, so they are absent from the series and recorded individually under `non_numeric_states` — and each sits inside a gap that is already listed, which is the honest rendering. The controller's own `meter_max_age_s` of 30 s is what makes a hole replay as *no coverage* rather than as an hour of whatever the last sample said.
## Size: committed, deliberately, and measured
`git bundle` of the whole history, before and after:
```
release/1.0 @ 4601682 1.51 MB
DATA-01 5.20 MB +3.69 MB, x3.4
```
A full clone more than triples, and a Home Assistant supervisor clones it on every install. Stated, not absorbed. Why this option and not the other two:
* **Downsampling is not available.** `meter_max_age_s` ships at 30 s — the age past which the controller's ingest stops counting a held reading as coverage — so any coarser resample converts real coverage into modelled meter-gap and moves the very number AC 3 measures. At or under 30 s it saves almost nothing against a 5 s cadence.
* **"Outside the repo with a pointer" is what already happened**, and it is why this card exists: the two days came from a private repo, the export script was thrown away, and three cards then deferred an AC. test_tariff.py states the rule verbatim — *a regression fixture CI cannot see is not a regression fixture* — and CI's token is minted for this repository only.
* **So: committed, in a SUBDIRECTORY**, which is load-bearing. `test_tariff.py` part 9 and `test_peakguard.py` part 6 both glob `scenarios/p1-net-*.jsonl.gz` non-recursively. Beside the two days, these thirty files would add sixty full-day replays (~50 s) to every PR forever and would break `check(... replay_days < 30)`. One directory down they are visible to CI and invisible to the globs.
## AC 3, run, with the actual numbers
```
485841 samples, 2879 closed quarters (30.0 days' worth)
month 2026-08: peak 5029.8 W uncontrolled -> 4107.3 W guarded
TARIFF-01 AC 3 (UNCONTROLLED): 2847/2879 = 98.89 % at or under 3500 W
SAFETY-07 AC 3 (GUARDED): 2850/2879 = 98.99 %
guard cycles 143810, engaged 1542, blind 195, saturated 0, 8060 Wh
```
Three findings, not a pass:
1. **Met on the ">= 95 % of intervals" reading — and the criterion turns out to be nearly insensitive.** The *uncontrolled* month, with no controller running at all, already scores 98.89 %. A criterion a house passes by doing nothing does not separate a working guard from an absent one. The card said a result below 95 % would be a finding; a result above it for this reason is one too.
2. **Not met on the other reading of the same sentence.** "Monthly peak held under the soft limit", taken as the peak itself, fails: 5029.8 W -> 4107.3 W (-922.5 W, -18.3 %, real money on a capacity tariff) but still over the 3500 W limit. Both readings are printed; neither is hidden.
3. **The guard stops defending the soft limit once the month's peak is set, by design.** `quarter_ceiling_w` uses `CEILING_MONTH_PEAK` — the billed quantity is the month's maximum, so a quarter under an already-recorded peak costs nothing extra. The 32 over-cap quarters fall on four nights; **31 are pulled down** but only **3 land under the cap**, because the ceiling being defended by then is the running peak (~4107 W). The one untouched quarter (2026-08-18T08:15Z, 4039.7 W) is the same rule from the other side: the running peak was already 4833 W, so it added nothing to the bill. AC 3 asks the guard for something its own tariff logic deliberately does not optimise.
Also worth an eye: **195 blind engagements** over 143,810 cycles, against 5 on the two committed days. It scales with the gaps.
## Not in this PR
The suites' checks are unchanged — only their honesty notes, which said the 30 days did not exist. CI still replays the two small days. Nothing was written to Home Assistant: read-only history API, day at a time, 2 s between days, no recorder setting or restart touched. All 15 test files pass locally.
TARIFF-01 and SAFETY-07 both merged with AC 3 - "monthly peak held under
the soft limit in >= 95 % of intervals over a 30-day simulation replay" -
honestly unmet, because two days of capture existed and it was believed
nothing longer could be produced. That belief was a true statement about
the past read as a claim about the future: HA's recorder still held the
whole month, and the reason nobody found out is that the 2026-08-25
export was a one-off script that left no artefact.
So the exporter is committed and runnable now.
scenarios/p1-net-30d-2026-08/ is 2026-08-01 to 2026-08-30, 485,841
samples, 30/30 days read, 0 failed, 4.13 MB gzipped, in the schema the
two committed days already fixed.
VERIFIED AGAINST THE DAYS THAT EXIST. `--verify` re-fetches 2026-08-20
and 2026-08-23 by this exact route and reproduces both committed files
BYTE FOR BYTE after decompression (865,024 B / 15,600 samples and
894,387 B / 16,351 samples). The gzip containers differ by design: this
exporter writes mtime 0 so its output is reproducible, where the one-off
wrote a wall clock.
A FAILED READ IS NOT AN OBSERVATION. A day whose read fails writes no
file, is recorded "read": "FAILED", and the run exits non-zero; the
entity is proved to exist through /api/states before any empty day is
believed, because a typo would otherwise answer 200 [] thirty times and
produce a consistent dataset of nothing. Nothing is interpolated, so the
month's 13 gaps (1291 s, 99.95 % covered) and its 3 `unavailable`
samples stay visible as holes.
SIZE: COMMITTED, DELIBERATELY, AND MEASURED. A git bundle of the whole
history goes from 1.58 MB (release/1.0 @ 86c43eb) to 5.28 MB - +3.70 MB,
x3.3 - and a Home Assistant supervisor clones that on every install.
Downsampling was not on the table: meter_max_age_s is 30 s, so any
coarser resample turns real coverage into modelled meter-gap and moves
the number being measured. "Outside the repo with a pointer" is what
already happened and is why this card exists. The files sit in a
SUBDIRECTORY so the non-recursive globs in test_tariff.py part 9 and
test_peakguard.py part 6 do not pick them up - CI keeps replaying the two
small days instead of growing ~50 s per PR.
AC 3, MEASURED OVER 30 DAYS (tools/replay_ac3.py, which must reproduce
test_peakguard part 6's 189/190 to be believed):
uncontrolled 2847/2879 = 98.89 % of quarters at or under 3500 W
guarded 2850/2879 = 98.99 %
month peak 5029.8 W -> 4107.3 W
Met on the ">= 95 % of intervals" reading - and the uncontrolled figure
is the finding, because a month with no controller at all passes it. Not
met on the other reading of the same sentence: 4107.3 W is still over the
soft limit. The guard pulls 31 of the 32 over-cap quarters down but lands
only 3 under the cap, because CEILING_MONTH_PEAK defends the running
month peak rather than the cap once one is set. Both suites' honesty
notes are corrected to point at the month instead of at its absence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VXjA3LWccN1rJsk3okyPC
Reviewed and merging. The re-export argument holds, the bridge reproduces numbers I had already verified twice, and the finding about the criterion is the most valuable thing in this PR.
Own detached worktree at 04292c4, own probes, own replays. Full suite green: 15 files, all exit 0, under python -B with a purged __pycache__.
No run was cancelled. Runs 1154 (d57122ec), 1155 (0abf5586) and 1156 (04292c4) all report status=success with real start times, and the head's check is Successful in 3m36s. The two-push disclosure checks out: the first run finished green before the second existed, so nothing queued was killed.
1. The re-export - verified myself, and the argument is right
I did not read the --verify output; I decompressed both pairs and compared:
2026-08-20: root 865024 B / 15600 samples 30d 865024 B / 15600 samples identical=True
2026-08-23: root 894387 B / 16351 samples 30d 894387 B / 16351 samples identical=True
gzip container: root mtime 1788109077 30d mtime 0 raw bytes equal=False
Byte-for-byte after decompression, and the only difference is the mtime field. Your reasoning holds, and I would put it more strongly than you did: container equality would not be a stricter version of the claim, it would be a different claim - that two exports run six days apart were written in the same second. That is unsatisfiable and says nothing about the data. And mtime 0 is not merely defensible, it is an improvement on the one-off: it makes this exporter's output reproducible, so the next re-export can be compared at the container level too. The 2026-08-25 wall-clock mtime is the defect; this removes it.
2. CRLF - right, not a trap, and git cannot drift it
Reproducing the original exactly is the verification: normalise to LF and the byte-identity claim above becomes untestable, which throws away the point of the ticket. The only question is whether the CRLF can rot, and it cannot:
.gitattributes: *.gz binary
git check-attr on p1-net-2026-08-20.jsonl.gz -> text: unset, binary: set
Git treats these as binary, so it will never normalise the compressed payload and the manifest sha256 cannot drift. Downstream, the readers open with gzip.open(..., "rt"), whose universal-newline translation removes the \r before json.loads sees it. Inert where it is read, load-bearing where it is verified. All 30 manifest sha256s match the committed content - 0 mismatches.
3. The bridge - reproduced, and it lands on numbers I had already re-derived
Every one of those is a number I measured independently on my own harness when reviewing TARIFF-01 and SAFETY-07 - 5029.8, 3485.3, 189/190, 878 Wh - and 110 graded + 5 blind is exactly the 115 engaged (blind 5) my own replay reported. So the new harness is calibrated against figures arrived at twice, by different code. That is a real bridge, not a self-consistency check.
4. THE RULING YOU ASKED FOR: the criterion is insensitive, and it is worse than insensitive
Your claim holds. The arithmetic:
quarters in the month : 2879
over the cap, UNCONTROLLED : 32
over the cap, GUARDED : 29
the guard moves the metric by : 3 quarters = 0.10 pp
quarters a 95 % bar would allow to be over : 144
slack the uncontrolled month already had : 112 quarters
signal-to-threshold ratio : 3/144 = 0.021
A month with no controller at all passes at 98.89 %, and would still pass with 112 more bad quarters than it had. The guard contributes three. The criterion is about fifty times less sensitive than the effect it exists to detect.
And the sharper form, which I think is the real finding: the criterion is not measuring the quantity the guard optimises. The capacity tariff bills the month's single worst quarter, so the guard minimises a maximum; AC 3 counts intervals. An 18.3 % cut in the maximum (5029.8 -> 4107.3 W) is invisible to an interval count, because the maximum is one interval out of 2879. Reading A cannot separate a working guard from an absent one, by construction rather than by bad luck.
So: record AC 3 as met on the letter and uninformative on the substance, and do not let that pass be read as evidence the guard works. The evidence for that is reading B and the two-day replay, both of which live elsewhere and both of which I have verified. If the AC is ever rewritten it should be on the monthly peak. Saying this plainly is worth more than closing the criterion, and this PR says it in the README, in both suites' honesty notes and in the tool's own output rather than burying it.
5. The other two claims
"The guard stops defending the cap once the month peak is set" - by design, and economically correct.quarter_ceiling_w returns max(cap, observed peak), so once the peak stands at X the ceiling is X. That is right: the bill is already X, so holding a later quarter to 3500 W instead of X saves nothing and spends pack energy to do it. 31 of 32 over-cap quarters pulled down with only 3 landing under the cap is not a shortfall - it is the ratchet doing the only thing worth doing, and it is also the mechanical reason reading A barely moves. The untouched quarter under a running peak of 4833 W cost nothing, as stated.
195 blind engagements - expected at scale, not a finding. A gap blinds the remainder of its own 900 s quarter, so 13 gaps bound the count at 13 x 900 / 15 = 780 cycles; 195 sits well inside that, about 15 cycles per gap. The per-day rate is ~2.6x the two-day figure because the two committed days are cleaner than the month's average, not because anything changed. And blind resolves to a 0 W floor, one-directional, which cannot cost pack energy.
6. The failed-read discipline - produced, not read
This is where you said the ticket was most likely to go wrong, so I ran the failures rather than reading the docstring:
A typo'd entity (would return 200 + [] for all 30 days) -> exit 1, 0 files
B unreachable recorder -> exit 1, 0 files
C entity exists, a day's read fails -> exit 1, 0 .gz files
manifest: [('2026-08-29','FAILED',None), ('2026-08-30','FAILED',None)]
summary : days_read 0, days_failed 2, empty_days []
An empty response cannot look like a quiet hour. The three states are distinguishable twice over - by the manifest (FAILED + file: null versus OK + samples: 0 + an entry in empty_days) and by the presence or absence of the file. empty_days being [] while days_failed is 2 is that distinction working. And assert_entity_exists fires before any day is fetched, so probe A aborts before it can manufacture thirty consistent files of nothing.
7. Housekeeping, and one honest gap in my own measurement
Glob-invisible, confirmed:Path("scenarios").glob("p1-net-*.jsonl.gz") finds 2 files; the 30 live one directory down. CI-visible, glob-invisible, and the existing replay_days < 30 check is untouched.
Honesty notes: no check changed. I grepped both suites' diffs for any added, removed or altered check(, assert or sys.exit - none. Docstrings and prints only, and the correction is the right one: "nothing longer was ever captured" was true about the past and was being read as a claim about the future.
Downsampling correctly ruled out.meter_max_age_s is 30 s, so any coarser resample converts real coverage into modelled meter-gap and moves the very number AC 3 measures. Derived from a shipped constant, not a preference.
Size: I could not reproduce your exact figure, and am not claiming to have. My bundle of the base came to 1.58 MB, matching yours exactly, but my head-side bundle used --all and gave 5.46 MB (+3.88, x3.5) against your 5.28 (+3.70, x3.3); the difference is the extra branches my spec included, and a second attempt at single-ref bundles failed on this git. So I corroborate the base figure exactly and the direction and magnitude of the delta, and nothing more.
Two conventions (tools/ here, scenarios/fetch_epex_be.py in DATA-02): cosmetic. The suites glob p1-net-*.jsonl.gz, so a stray .py in scenarios/ cannot be picked up. Worth a line in a future tidy, not worth churning two merged cards.
I ran nothing against the rig, the live add-on, the inverter or firmware. The only live traffic was read-only GETs to HA's recorder and /api/states from the failure probes.
Nothing survives. Merging.
## Reviewed and merging. The re-export argument holds, the bridge reproduces numbers I had already verified twice, and the finding about the criterion is the most valuable thing in this PR.
Own detached worktree at `04292c4`, own probes, own replays. Full suite green: **15 files, all exit 0**, under `python -B` with a purged `__pycache__`.
**No run was cancelled.** Runs 1154 (`d57122ec`), 1155 (`0abf5586`) and 1156 (`04292c4`) all report `status=success` with real start times, and the head's check is `Successful in 3m36s`. The two-push disclosure checks out: the first run finished green before the second existed, so nothing queued was killed.
---
## 1. The re-export - verified myself, and the argument is right
I did not read the `--verify` output; I decompressed both pairs and compared:
```
2026-08-20: root 865024 B / 15600 samples 30d 865024 B / 15600 samples identical=True
2026-08-23: root 894387 B / 16351 samples 30d 894387 B / 16351 samples identical=True
gzip container: root mtime 1788109077 30d mtime 0 raw bytes equal=False
```
**Byte-for-byte after decompression, and the only difference is the mtime field.** Your reasoning holds, and I would put it more strongly than you did: container equality would not be a stricter version of the claim, it would be a *different* claim - that two exports run six days apart were written in the same second. That is unsatisfiable and says nothing about the data. And mtime 0 is not merely defensible, it is **an improvement on the one-off**: it makes this exporter's output reproducible, so the *next* re-export can be compared at the container level too. The 2026-08-25 wall-clock mtime is the defect; this removes it.
## 2. CRLF - right, not a trap, and git cannot drift it
Reproducing the original exactly *is* the verification: normalise to LF and the byte-identity claim above becomes untestable, which throws away the point of the ticket. The only question is whether the CRLF can rot, and it cannot:
```
.gitattributes: *.gz binary
git check-attr on p1-net-2026-08-20.jsonl.gz -> text: unset, binary: set
```
Git treats these as binary, so it will never normalise the compressed payload and the manifest sha256 cannot drift. Downstream, the readers open with `gzip.open(..., "rt")`, whose universal-newline translation removes the `\r` before `json.loads` sees it. Inert where it is read, load-bearing where it is verified. **All 30 manifest sha256s match the committed content - 0 mismatches.**
## 3. The bridge - reproduced, and it lands on numbers I had already re-derived
```
python3 tools/replay_ac3.py scenarios --expect 189/190
190 closed quarters, peak 5029.8 W uncontrolled -> 3485.3 W guarded
uncontrolled 189/190 = 99.47 %, guarded 190/190 = 100.00 %
engaged 110, blind 5, saturated 0, modelled discharge 878 Wh
reproduces test_peakguard.py part 6's published 189/190: yes
```
Every one of those is a number I measured independently on my own harness when reviewing TARIFF-01 and SAFETY-07 - 5029.8, 3485.3, 189/190, 878 Wh - and `110 graded + 5 blind` is exactly the `115 engaged (blind 5)` my own replay reported. So the new harness is calibrated against figures arrived at twice, by different code. That is a real bridge, not a self-consistency check.
The month reproduces exactly too:
```
2879 closed quarters (30.0 days), peak 5029.8 -> 4107.3 W guarded
uncontrolled 2847/2879 = 98.89 %, guarded 2850/2879 = 98.99 %
gaps over 30 s: 13, 1291 s total; blind 195, saturated 0
```
## 4. THE RULING YOU ASKED FOR: the criterion is insensitive, and it is worse than insensitive
Your claim holds. The arithmetic:
```
quarters in the month : 2879
over the cap, UNCONTROLLED : 32
over the cap, GUARDED : 29
the guard moves the metric by : 3 quarters = 0.10 pp
quarters a 95 % bar would allow to be over : 144
slack the uncontrolled month already had : 112 quarters
signal-to-threshold ratio : 3/144 = 0.021
```
**A month with no controller at all passes at 98.89 %, and would still pass with 112 more bad quarters than it had.** The guard contributes three. The criterion is about fifty times less sensitive than the effect it exists to detect.
And the sharper form, which I think is the real finding: **the criterion is not measuring the quantity the guard optimises.** The capacity tariff bills the month's single worst quarter, so the guard minimises a *maximum*; AC 3 counts *intervals*. An 18.3 % cut in the maximum (5029.8 -> 4107.3 W) is invisible to an interval count, because the maximum is one interval out of 2879. Reading A cannot separate a working guard from an absent one, by construction rather than by bad luck.
So: **record AC 3 as met on the letter and uninformative on the substance, and do not let that pass be read as evidence the guard works.** The evidence for that is reading B and the two-day replay, both of which live elsewhere and both of which I have verified. If the AC is ever rewritten it should be on the monthly peak. Saying this plainly is worth more than closing the criterion, and this PR says it in the README, in both suites' honesty notes and in the tool's own output rather than burying it.
## 5. The other two claims
**"The guard stops defending the cap once the month peak is set" - by design, and economically correct.** `quarter_ceiling_w` returns `max(cap, observed peak)`, so once the peak stands at X the ceiling is X. That is right: the bill is already X, so holding a later quarter to 3500 W instead of X saves nothing and spends pack energy to do it. **31 of 32 over-cap quarters pulled down with only 3 landing under the cap is not a shortfall** - it is the ratchet doing the only thing worth doing, and it is also the mechanical reason reading A barely moves. The untouched quarter under a running peak of 4833 W cost nothing, as stated.
**195 blind engagements - expected at scale, not a finding.** A gap blinds the remainder of its own 900 s quarter, so 13 gaps bound the count at `13 x 900 / 15 = 780` cycles; 195 sits well inside that, about 15 cycles per gap. The per-day rate is ~2.6x the two-day figure because the two committed days are cleaner than the month's average, not because anything changed. And blind resolves to a 0 W floor, one-directional, which cannot cost pack energy.
## 6. The failed-read discipline - produced, not read
This is where you said the ticket was most likely to go wrong, so I ran the failures rather than reading the docstring:
```
A typo'd entity (would return 200 + [] for all 30 days) -> exit 1, 0 files
B unreachable recorder -> exit 1, 0 files
C entity exists, a day's read fails -> exit 1, 0 .gz files
manifest: [('2026-08-29','FAILED',None), ('2026-08-30','FAILED',None)]
summary : days_read 0, days_failed 2, empty_days []
```
**An empty response cannot look like a quiet hour.** The three states are distinguishable twice over - by the manifest (`FAILED` + `file: null` versus `OK` + `samples: 0` + an entry in `empty_days`) and by the presence or absence of the file. `empty_days` being `[]` while `days_failed` is 2 is that distinction working. And `assert_entity_exists` fires before any day is fetched, so probe A aborts before it can manufacture thirty consistent files of nothing.
## 7. Housekeeping, and one honest gap in my own measurement
- **Glob-invisible, confirmed:** `Path("scenarios").glob("p1-net-*.jsonl.gz")` finds **2** files; the 30 live one directory down. CI-visible, glob-invisible, and the existing `replay_days < 30` check is untouched.
- **Honesty notes: no check changed.** I grepped both suites' diffs for any added, removed or altered `check(`, `assert` or `sys.exit` - **none**. Docstrings and prints only, and the correction is the right one: "nothing longer was ever captured" was true about the past and was being read as a claim about the future.
- **Downsampling correctly ruled out.** `meter_max_age_s` is 30 s, so any coarser resample converts real coverage into modelled meter-gap and moves the very number AC 3 measures. Derived from a shipped constant, not a preference.
- **Size: I could not reproduce your exact figure, and am not claiming to have.** My bundle of the base came to **1.58 MB, matching yours exactly**, but my head-side bundle used `--all` and gave 5.46 MB (+3.88, x3.5) against your 5.28 (+3.70, x3.3); the difference is the extra branches my spec included, and a second attempt at single-ref bundles failed on this git. So I corroborate the base figure exactly and the direction and magnitude of the delta, and nothing more.
- **Two conventions** (`tools/` here, `scenarios/fetch_epex_be.py` in DATA-02): cosmetic. The suites glob `p1-net-*.jsonl.gz`, so a stray `.py` in `scenarios/` cannot be picked up. Worth a line in a future tidy, not worth churning two merged cards.
- I ran nothing against the rig, the live add-on, the inverter or firmware. The only live traffic was read-only `GET`s to HA's recorder and `/api/states` from the failure probes.
Nothing survives. Merging.
admin
merged commit f6ecd5ff03 into release/1.02026-08-31 11:35:46 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
DATA-01 - Export 30 days of P1 before the retention window rolls.
What this is
TARIFF-01 AC 3 and SAFETY-07 AC 3 both ask for "monthly peak held under the soft limit in >= 95 % of intervals over a 30-day simulation replay". Both merged with it honestly unmet, on the belief that only two days existed and nothing longer could be produced retroactively.
That belief was a true statement about the past read as a claim about the future. HA's recorder still held the whole month. Probed 2026-08-31: 230 samples in a 20-minute window at 30 days back, 0 at 45. Exported the same day, because the window rolls — 2026-08-01 leaves the recorder around 2026-09-15.
goodwe_controller/scenarios/p1-net-30d-2026-08/— 2026-08-01 to 2026-08-30, 485,841 samples, 30/30 days read, 0 failed, 4.13 MB gzipped, in the schema the two committed days already fixed.The check that matters: the exporter reproduces the days that exist
tools/export_p1_history.py --verifyre-fetches 2026-08-20 and 2026-08-23 from the live recorder by the exact route used for the month, and diffs them against the committed files:Byte for byte after decompression, both days. The gzip containers differ and that is deliberate: a gzip header carries an mtime and a stored filename, and this exporter writes mtime 0 so its own output is reproducible, where the 2026-08-25 one-off wrote a wall clock. Container equality would only assert that two files were written in the same second. The transform was also checked offline against the raw HA responses archived in the infra repo, with the same result.
The month's own copies of those two days are byte-identical to the committed ones, and re-reading 2026-08-11 from HA a second time reproduced it byte for byte too.
tools/replay_ac3.pyis checked the same way rather than trusted:--expect 189/190makes it reproduce test_peakguard.py part 6's published totals — the 5029.8 W billed quarter, the 3485.3 W guarded result, the 878 Wh of modelled discharge.A failed read is not an observation
The distinction the whole dataset stands on:
"read": "FAILED"with the error, and the run exits non-zero. A missing day is unreadable-and-known, never a silent zero;"read": "OK", "samples": 0;/api/statesbefore any empty day is believed — a typo in--entitywould otherwise answer200 []thirty times and produce a beautifully consistent dataset of nothing;On this run: 30 read, 0 failed, 0 empty.
Gaps are visible, not interpolated
Nothing is filled, carried forward or resampled. 13 gaps over 30 s, 1291 s total, 99.950 % of the month covered, every one listed in
manifest.json(with per-day sha256 and counts) and by--report-gaps. Longest 263 s on 2026-08-11.Three samples carried the state
unavailablerather than a number. There is no float to write and the schema has no slot for a non-number, so they are absent from the series and recorded individually undernon_numeric_states— and each sits inside a gap that is already listed, which is the honest rendering. The controller's ownmeter_max_age_sof 30 s is what makes a hole replay as no coverage rather than as an hour of whatever the last sample said.Size: committed, deliberately, and measured
git bundleof the whole history, before and after:A full clone more than triples, and a Home Assistant supervisor clones it on every install. Stated, not absorbed. Why this option and not the other two:
meter_max_age_sships at 30 s — the age past which the controller's ingest stops counting a held reading as coverage — so any coarser resample converts real coverage into modelled meter-gap and moves the very number AC 3 measures. At or under 30 s it saves almost nothing against a 5 s cadence.test_tariff.pypart 9 andtest_peakguard.pypart 6 both globscenarios/p1-net-*.jsonl.gznon-recursively. Beside the two days, these thirty files would add sixty full-day replays (~50 s) to every PR forever and would breakcheck(... replay_days < 30). One directory down they are visible to CI and invisible to the globs.AC 3, run, with the actual numbers
Three findings, not a pass:
quarter_ceiling_wusesCEILING_MONTH_PEAK— the billed quantity is the month's maximum, so a quarter under an already-recorded peak costs nothing extra. The 32 over-cap quarters fall on four nights; 31 are pulled down but only 3 land under the cap, because the ceiling being defended by then is the running peak (~4107 W). The one untouched quarter (2026-08-18T08:15Z, 4039.7 W) is the same rule from the other side: the running peak was already 4833 W, so it added nothing to the bill. AC 3 asks the guard for something its own tariff logic deliberately does not optimise.Also worth an eye: 195 blind engagements over 143,810 cycles, against 5 on the two committed days. It scales with the gaps.
Not in this PR
The suites' checks are unchanged — only their honesty notes, which said the 30 days did not exist. CI still replays the two small days. Nothing was written to Home Assistant: read-only history API, day at a time, 2 s between days, no recorder setting or restart touched. All 15 test files pass locally.
0abf5586c1to04292c4a08Reviewed and merging. The re-export argument holds, the bridge reproduces numbers I had already verified twice, and the finding about the criterion is the most valuable thing in this PR.
Own detached worktree at
04292c4, own probes, own replays. Full suite green: 15 files, all exit 0, underpython -Bwith a purged__pycache__.No run was cancelled. Runs 1154 (
d57122ec), 1155 (0abf5586) and 1156 (04292c4) all reportstatus=successwith real start times, and the head's check isSuccessful in 3m36s. The two-push disclosure checks out: the first run finished green before the second existed, so nothing queued was killed.1. The re-export - verified myself, and the argument is right
I did not read the
--verifyoutput; I decompressed both pairs and compared:Byte-for-byte after decompression, and the only difference is the mtime field. Your reasoning holds, and I would put it more strongly than you did: container equality would not be a stricter version of the claim, it would be a different claim - that two exports run six days apart were written in the same second. That is unsatisfiable and says nothing about the data. And mtime 0 is not merely defensible, it is an improvement on the one-off: it makes this exporter's output reproducible, so the next re-export can be compared at the container level too. The 2026-08-25 wall-clock mtime is the defect; this removes it.
2. CRLF - right, not a trap, and git cannot drift it
Reproducing the original exactly is the verification: normalise to LF and the byte-identity claim above becomes untestable, which throws away the point of the ticket. The only question is whether the CRLF can rot, and it cannot:
Git treats these as binary, so it will never normalise the compressed payload and the manifest sha256 cannot drift. Downstream, the readers open with
gzip.open(..., "rt"), whose universal-newline translation removes the\rbeforejson.loadssees it. Inert where it is read, load-bearing where it is verified. All 30 manifest sha256s match the committed content - 0 mismatches.3. The bridge - reproduced, and it lands on numbers I had already re-derived
Every one of those is a number I measured independently on my own harness when reviewing TARIFF-01 and SAFETY-07 - 5029.8, 3485.3, 189/190, 878 Wh - and
110 graded + 5 blindis exactly the115 engaged (blind 5)my own replay reported. So the new harness is calibrated against figures arrived at twice, by different code. That is a real bridge, not a self-consistency check.The month reproduces exactly too:
4. THE RULING YOU ASKED FOR: the criterion is insensitive, and it is worse than insensitive
Your claim holds. The arithmetic:
A month with no controller at all passes at 98.89 %, and would still pass with 112 more bad quarters than it had. The guard contributes three. The criterion is about fifty times less sensitive than the effect it exists to detect.
And the sharper form, which I think is the real finding: the criterion is not measuring the quantity the guard optimises. The capacity tariff bills the month's single worst quarter, so the guard minimises a maximum; AC 3 counts intervals. An 18.3 % cut in the maximum (5029.8 -> 4107.3 W) is invisible to an interval count, because the maximum is one interval out of 2879. Reading A cannot separate a working guard from an absent one, by construction rather than by bad luck.
So: record AC 3 as met on the letter and uninformative on the substance, and do not let that pass be read as evidence the guard works. The evidence for that is reading B and the two-day replay, both of which live elsewhere and both of which I have verified. If the AC is ever rewritten it should be on the monthly peak. Saying this plainly is worth more than closing the criterion, and this PR says it in the README, in both suites' honesty notes and in the tool's own output rather than burying it.
5. The other two claims
"The guard stops defending the cap once the month peak is set" - by design, and economically correct.
quarter_ceiling_wreturnsmax(cap, observed peak), so once the peak stands at X the ceiling is X. That is right: the bill is already X, so holding a later quarter to 3500 W instead of X saves nothing and spends pack energy to do it. 31 of 32 over-cap quarters pulled down with only 3 landing under the cap is not a shortfall - it is the ratchet doing the only thing worth doing, and it is also the mechanical reason reading A barely moves. The untouched quarter under a running peak of 4833 W cost nothing, as stated.195 blind engagements - expected at scale, not a finding. A gap blinds the remainder of its own 900 s quarter, so 13 gaps bound the count at
13 x 900 / 15 = 780cycles; 195 sits well inside that, about 15 cycles per gap. The per-day rate is ~2.6x the two-day figure because the two committed days are cleaner than the month's average, not because anything changed. And blind resolves to a 0 W floor, one-directional, which cannot cost pack energy.6. The failed-read discipline - produced, not read
This is where you said the ticket was most likely to go wrong, so I ran the failures rather than reading the docstring:
An empty response cannot look like a quiet hour. The three states are distinguishable twice over - by the manifest (
FAILED+file: nullversusOK+samples: 0+ an entry inempty_days) and by the presence or absence of the file.empty_daysbeing[]whiledays_failedis 2 is that distinction working. Andassert_entity_existsfires before any day is fetched, so probe A aborts before it can manufacture thirty consistent files of nothing.7. Housekeeping, and one honest gap in my own measurement
Path("scenarios").glob("p1-net-*.jsonl.gz")finds 2 files; the 30 live one directory down. CI-visible, glob-invisible, and the existingreplay_days < 30check is untouched.check(,assertorsys.exit- none. Docstrings and prints only, and the correction is the right one: "nothing longer was ever captured" was true about the past and was being read as a claim about the future.meter_max_age_sis 30 s, so any coarser resample converts real coverage into modelled meter-gap and moves the very number AC 3 measures. Derived from a shipped constant, not a preference.--alland gave 5.46 MB (+3.88, x3.5) against your 5.28 (+3.70, x3.3); the difference is the extra branches my spec included, and a second attempt at single-ref bundles failed on this git. So I corroborate the base figure exactly and the direction and magnitude of the delta, and nothing more.tools/here,scenarios/fetch_epex_be.pyin DATA-02): cosmetic. The suites globp1-net-*.jsonl.gz, so a stray.pyinscenarios/cannot be picked up. Worth a line in a future tidy, not worth churning two merged cards.GETs to HA's recorder and/api/statesfrom the failure probes.Nothing survives. Merging.