Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 glenn schrooyenandClaude Opus 5
|
04292c4a08 | ||
 admin
|
86c43eba24 | ||
|
glenn schrooyenandClaude Opus 5
|
d57122ec74 | ||
|
admin
|
4601682815 | ||
|
admin
|
b7272169b1 | ||
|
glenn schrooyenandClaude Opus 5
|
2b8b8cb97f |
@@ -2,6 +2,121 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
**Thirty days of the real meter, and the acceptance criterion three cards
|
||||
deferred is now a number (DATA-01).** TARIFF-01 and SAFETY-07 both merged with
|
||||
AC 3 - *"monthly peak held under the soft limit in >= 95 % of intervals over a
|
||||
30-day simulation replay"* - honestly unmet, on the belief that only two days of
|
||||
capture existed and nothing longer could be produced. That belief came from a
|
||||
true statement about the past being read as one about the future: HA's recorder
|
||||
still held the whole month. `scenarios/p1-net-30d-2026-08/` is 2026-08-01 to
|
||||
2026-08-30, **485,841 samples, 30/30 days read, 0 failed, 4.13 MB gzipped**, in
|
||||
the same schema as the two days that were already there.
|
||||
|
||||
`tools/export_p1_history.py` is the exporter, committed and runnable - the
|
||||
2026-08-25 one-off was not, and that is exactly why the data was thought
|
||||
unobtainable. It reads day-at-a-time, retries, and **never lets a failed read
|
||||
look like an empty hour**: a day whose read fails writes no file, is recorded as
|
||||
`"read": "FAILED"`, and the run exits non-zero; the entity is proved to exist
|
||||
before any empty day is believed. Nothing is interpolated, so the month's 13
|
||||
gaps (1291 s, 99.95 % covered) and its 3 `unavailable` samples stay visible.
|
||||
`--verify` re-fetches the two committed days by the same route and reproduces
|
||||
them **byte for byte** after decompression.
|
||||
|
||||
`tools/replay_ac3.py` runs the month. Over 30 days the criterion measures
|
||||
**98.89 % uncontrolled and 98.99 % guarded**, so it is met on that reading - and
|
||||
the uncontrolled figure is the finding: a month with no controller at all passes
|
||||
it. On the other reading of the same sentence it is not met - the guard takes
|
||||
August's peak from **5029.8 W to 4107.3 W**, still over the 3500 W soft limit,
|
||||
because once a month peak is set `CEILING_MONTH_PEAK` defends that peak rather
|
||||
than the cap. Neither number is hidden; both are in that directory's README.
|
||||
|
||||
The size decision is committed and measured, not absorbed: a `git bundle` of
|
||||
the whole history goes from **1.58 MB to 5.28 MB** (+3.70 MB, x3.3), which a
|
||||
Home Assistant supervisor clones on every install. Downsampling was not
|
||||
available - `meter_max_age_s` is 30 s, so any coarser resample converts real
|
||||
coverage into modelled meter-gap and moves the number being measured.
|
||||
|
||||
The thirty files sit in a SUBDIRECTORY on purpose: `test_tariff.py` part 9 and
|
||||
`test_peakguard.py` part 6 glob `scenarios/p1-net-*.jsonl.gz` non-recursively,
|
||||
so CI keeps replaying the two small days and does not grow ~50 s per PR. Both
|
||||
files' honesty notes are corrected to point at the month.
|
||||
|
||||
**OPT-04's AC4 is met: twelve months of real EPEX BE day-ahead prices are now in
|
||||
the repository, and the strategy has been replayed against them (DATA-02).**
|
||||
AC4 asked for "net-positive across 12 months of replayed prices". OPT-04 shipped
|
||||
it as NOT MET and replayed a synthetic year instead, because `scenarios/` held
|
||||
only P1 meter captures. That was the right call at the time and the wrong
|
||||
diagnosis of why: the criterion was being carried alongside a 30-day *meter*
|
||||
criterion as though both needed waiting, and EPEX BE day-ahead is **public data,
|
||||
not a capture**. It needed a fetch, not a month.
|
||||
|
||||
`scenarios/epex-be-dayahead-2025-09-01_2026-09-01.json` — **8760 hourly points**,
|
||||
2025-09-01 to 2026-09-01 Brussels-local, in §9.3's canonical three-field shape.
|
||||
`scenarios/fetch_epex_be.py` produced it and `--verify` re-derives every claim it
|
||||
makes about itself without touching the network.
|
||||
|
||||
**Licensing was checked before anything was committed.** The source is
|
||||
Fraunhofer ISE's Energy-Charts API, which publishes the **BE** zone under
|
||||
**CC BY 4.0 from Bundesnetzagentur | SMARD.de** and other zones under
|
||||
private-use-only terms — so this zone is redistributable with attribution and the
|
||||
data is committed rather than left as a script plus a checksum. The fetcher
|
||||
**refuses to write a file** whose per-response `license_info` stops saying
|
||||
CC BY 4.0. Cross-checked against the upstream: SMARD's own `4996/DE` series
|
||||
matched this feed on all 96 quarter-hours of 2026-04-10, maximum difference
|
||||
0.00 EUR/MWh.
|
||||
|
||||
**The figures are raw wholesale and the file says so.** `"all_in": false`, with
|
||||
the markup/tax assumption carried in `provenance.all_in_assumption` as *numbers
|
||||
to apply* rather than numbers already applied — TARIFF-02's double-counting trap
|
||||
in data form. `test_arbitrage.py` section 7B applies it in one visible place and
|
||||
**asserts arithmetically** that its two additive terms cancel out of every daily
|
||||
spread, so the trade counts depend on the VAT multiplier alone and not on the
|
||||
guessed markup. The same replay is run on raw wholesale as well, so how much of
|
||||
the answer rests on the guess is reported (110 trades all-in, 88 raw).
|
||||
|
||||
**DST is handled by construction, not by a special case.** The fetcher iterates
|
||||
in UTC and renders through `price.brussels()`, so **2025-10-26 comes out 25
|
||||
points and 2026-03-29 comes out 23** — verified by counting per Brussels day
|
||||
rather than asserting 24, and by checking that October's 02:00 appears twice at
|
||||
`+02:00` and `+01:00` with two different prices while March's 02:00 does not
|
||||
appear at all. The window is bounded at *local* midnight: bounding it at 00:00
|
||||
UTC produced a 22-hour first day, and a 24-hour day silently missing from the
|
||||
end, both of which the point-and-gap accounting caught.
|
||||
|
||||
**The result, at the shipped default, whatever it is.** At
|
||||
`cycle_cost_threshold_eur` **0.10** widened by FCST-03's no-weather **1.5×** to a
|
||||
required 0.1500 EUR/kWh, the real year completes **110 round trips** and is
|
||||
**net-positive at +24.88 EUR** of benefit net of the cycle cost (2 kWh a trade).
|
||||
Unwidened it completes 173. **Nothing was tuned:** the shipped default and the
|
||||
2.5/1.5 factors are untouched, and every figure comes out of the same `decide()`.
|
||||
|
||||
**What the product owner's open decision now has under it.** The four
|
||||
independent reports that a static day/night spread of ~0.0840 does not clear
|
||||
0.10 **all stand** — a static-tariff site still never trades. What was missing
|
||||
was the other half: a *dynamic* site's daily spread clears the widened
|
||||
requirement on **121 of 365 days**, median daily all-in spread **0.1130**. The
|
||||
threshold at which a median day would clear is **0.0754**, and at that value the
|
||||
same year completes 156 trips for +35.42 EUR. Both numbers are reported; neither
|
||||
is applied.
|
||||
|
||||
**And the synthetic year stays, because the real one cannot grade §17.** Not one
|
||||
of the 365 real days has a negative *dearest* hour — 372 negative wholesale hours
|
||||
and 0 days where the strategy would have sold into one — so "0 discharges into a
|
||||
negative-price hour" over real prices is a count of something that could not
|
||||
happen, and deleting the guard would not change it. Section 7B asserts that
|
||||
degeneracy instead of quietly reporting the zero; section 7's generator is still
|
||||
the only thing here that puts a negative price where the strategy would sell.
|
||||
|
||||
**Size, measured rather than assumed.** Plain JSON: 893 KB in the working tree
|
||||
but **74.6 KB as a git object**, against 56.5 KB for a committed `.json.gz`. Git
|
||||
already compresses blobs, so gzipping saves 18 KB and costs a reviewer the
|
||||
ability to read the provenance block — including the licence claim they are
|
||||
meant to check — in the diff. Plain, and `load()` still accepts `.gz` for a
|
||||
future dataset that earns it.
|
||||
|
||||
**Nothing is wired into the add-on.** `generic_dayahead` remains the live fetch
|
||||
path; nothing under `app/` imports the script, and the dataset is for replay.
|
||||
|
||||
**The capacity-peak guard's forced discharge now has an EMS-side energy floor
|
||||
(SAFETY-24).** SAFETY-07 can command a discharge to hold a billed quarter down,
|
||||
and nothing below it consulted the state of charge — `soc_floor` belongs to the
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,551 @@
|
||||
#!/usr/bin/env python3
|
||||
"""DATA-02: fetch 12 months of EPEX BE day-ahead prices, in §9.3's shape.
|
||||
|
||||
python3 scenarios/fetch_epex_be.py --start 2025-09-01 --end 2026-09-01 \
|
||||
--out scenarios/epex-be-dayahead-2025-09-01_2026-09-01.json
|
||||
|
||||
Run from `goodwe_controller/`. Rewrites the output file from scratch; the
|
||||
committed dataset is exactly what this produces, and `--verify` re-derives every
|
||||
claim the file makes about itself without going to the network.
|
||||
|
||||
|
||||
WHAT THIS IS NOT
|
||||
================
|
||||
NOT wired into the add-on, and it must never be. `price.py`'s `GenericDayAhead`
|
||||
is the live path and it has its own egress check; this is a one-shot archival
|
||||
fetch that produces a file for REPLAY, run by hand by a developer. Nothing under
|
||||
`app/` imports it.
|
||||
|
||||
|
||||
THE SOURCE, AND WHY THIS ONE
|
||||
============================
|
||||
`https://api.energy-charts.info/price?bzn=BE` - Fraunhofer ISE's Energy-Charts
|
||||
API. No key, no registration.
|
||||
|
||||
⚠️ THE LICENCE WAS CHECKED BEFORE ANY DATA WAS COMMITTED, because for this
|
||||
source it is NOT uniform across bidding zones. The API's own OpenAPI document
|
||||
(`https://api.energy-charts.info/openapi.json`, /price) splits the zones in two:
|
||||
|
||||
"The data for the following bidding zones is licensed as CC BY 4.0 from
|
||||
Bundesnetzagentur | SMARD.de and is published without changes:
|
||||
AT, BE (Belgium), CH, CZ, DE-LU, ..."
|
||||
|
||||
"The data for the other bidding zones is for private and internal use only.
|
||||
The utilization of any data, whether in its raw or derived form, for
|
||||
external or commercial purposes is expressly prohibited. Should you require
|
||||
licensing for market-related data, please direct your inquiries to the
|
||||
original data providers, including but not limited to EPEX SPOT SE."
|
||||
|
||||
BE is in the FIRST list, so redistribution is permitted with attribution and the
|
||||
dataset is committed rather than left as a fetch script and a checksum. Every
|
||||
response also carries the licence per-response in `license_info`, and this
|
||||
script REFUSES to write a file if that field ever stops saying CC BY 4.0 - a
|
||||
licence checked once by a human and assumed forever is a licence that changes
|
||||
silently. Had BE been in the second list, the correct output of this card would
|
||||
have been this script and a checksum and no data.
|
||||
|
||||
Attribution, carried in the dataset's own `provenance` block and reproduced
|
||||
here: data CC BY 4.0, Bundesnetzagentur | SMARD.de, retrieved via Energy-Charts
|
||||
(Fraunhofer ISE), https://energy-charts.info, published without changes.
|
||||
|
||||
Cross-checked against the upstream on 2026-08-31: SMARD's own chart_data series
|
||||
`4996/DE` over 2026-04-10 matched this API's BE series on all 96 quarter-hours
|
||||
with a maximum absolute difference of 0.00 EUR/MWh. "Published without changes"
|
||||
is a claim worth checking once; `--verify` does not repeat it, because it is a
|
||||
claim about the SOURCE and not about the file.
|
||||
|
||||
|
||||
RAW WHOLESALE, AND THE ASSUMPTION THAT IS NOT BAKED IN
|
||||
======================================================
|
||||
⚠️ WHAT COMES BACK IS RAW WHOLESALE, EUR/MWh, DAY-AHEAD AUCTION CLEARING PRICE.
|
||||
It is NOT all-in: no supplier markup, no distribution or transmission tariff, no
|
||||
levies, no VAT. §9.3's canonical point is all-in, and TARIFF-02's whole trap is
|
||||
a figure that already contains a component being handed to something that adds
|
||||
it again.
|
||||
|
||||
So this file does NOT add one. `eur_per_kwh` in the output is the wholesale
|
||||
figure converted from EUR/MWh, `provenance.all_in` is `false` beside it, and the
|
||||
markup/tax assumption a consumer needs lives in `provenance.all_in_assumption`
|
||||
as NUMBERS TO APPLY rather than numbers already applied. test_arbitrage.py
|
||||
section 7 applies them where a reader can watch it happen.
|
||||
|
||||
The assumption's SHAPE is the part that is not arbitrary, and it decides which
|
||||
conclusions depend on the guessed numbers at all:
|
||||
|
||||
all_in(h) = (wholesale(h) + markup + network_and_levies) * (1 + vat)
|
||||
|
||||
* The two ADDITIVE terms cancel out of a spread: max - min is unchanged by
|
||||
them. So how often a day clears `cycle_cost_threshold_eur` does NOT depend
|
||||
on the markup guess - only on the VAT multiplier.
|
||||
* They do NOT cancel out of §17's negative-price guard, which compares ONE
|
||||
hour's all-in price against zero. A wholesale hour at -0.02 EUR/kWh is
|
||||
comfortably positive all-in once ~0.12 of fixed charges is added.
|
||||
|
||||
Both are asserted arithmetically in section 7 rather than believed, and the
|
||||
trade count is reported on the raw spread as well, so a reader can see exactly
|
||||
how much of the answer rests on the guess.
|
||||
|
||||
The numbers themselves (2026 Belgian residential dynamic contract, order of
|
||||
magnitude) are a STATED GUESS, labelled as one everywhere they appear. They are
|
||||
not a measurement and this card had nothing to measure them with.
|
||||
|
||||
|
||||
HOURS, QUARTER-HOURS, AND THE SEAM IN THE MIDDLE OF THE YEAR
|
||||
============================================================
|
||||
⚠️ THE SOURCE CHANGES RESOLUTION PART-WAY THROUGH ANY RECENT 12 MONTHS. EPEX
|
||||
day-ahead moved to a 15-minute market time unit in October 2025 and the feed
|
||||
follows: 2025-09-10 comes back as 24 hourly points, 2025-12-10 as 96
|
||||
quarter-hourly ones. An importer that assumed either one silently loses three
|
||||
quarters of the year, or reads a quarter as an hour.
|
||||
|
||||
The canonical format is hourly, so quarter-hours are aggregated: an hour's price
|
||||
is the unweighted mean of its four quarters. That is a STATED APPROXIMATION of
|
||||
the hourly product, not the hourly product, and it errs in the direction that
|
||||
matters here - averaging discards intra-hour variation, so the observed spread
|
||||
is a LOWER bound on what a 15-minute strategy would see.
|
||||
`provenance.resolution` records how many hours came from each source shape.
|
||||
|
||||
⚠️ AND A PARTIAL HOUR IS DROPPED, NOT AVERAGED OVER WHAT ARRIVED. An hour with
|
||||
three of four quarters is an hour nobody published a complete price for; the
|
||||
mean of three would be an invented number carrying no flag. Dropped hours are
|
||||
listed in `provenance.gaps` with their timestamps, because a missing hour in a
|
||||
feed is not a zero price, and a zero price is exactly the value that would make
|
||||
everything look free.
|
||||
|
||||
|
||||
SIZE: WHY THIS IS PLAIN JSON AND NOT A `.gz`, MEASURED
|
||||
======================================================
|
||||
The directory's existing captures are gzipped, so the default would have been to
|
||||
gzip this too. Measured on the committed file instead of assumed:
|
||||
|
||||
working tree, plain JSON 893 171 bytes
|
||||
what GIT actually stores for it (zlib, loose object) 74 590 bytes
|
||||
what git would store for a committed `.json.gz` 56 504 bytes
|
||||
|
||||
⚠️ GIT ALREADY COMPRESSES BLOBS, so gzipping buys 18 KB - 24 % of an object
|
||||
that is already 75 KB, in a repository whose three existing P1 captures cost
|
||||
27 KB, 136 KB and 140 KB. Eighteen kilobytes is not worth making the provenance
|
||||
block unreadable in a pull request, and the licence claim in that block is
|
||||
precisely the thing a reviewer of this card has to read and check. Plain JSON
|
||||
also greps, diffs line-by-line when a future refetch changes something, and is
|
||||
what `--verify` opens. `load()` still handles `.gz` for a future dataset large
|
||||
enough to want it; this one is not.
|
||||
|
||||
The P1 captures are gzipped for the opposite reason and it is not an
|
||||
inconsistency: they are hundreds of thousands of telegram lines that no human
|
||||
reads, and there is no metadata in them to review.
|
||||
|
||||
|
||||
TIME
|
||||
====
|
||||
Every instant is carried as UTC and rendered with `app.price.brussels()`, the
|
||||
module that already owns the EU DST rule without a tz database. There is no
|
||||
second answer to that question here and there must not be: the point of
|
||||
`brussels()` is that the 02:00-03:00 hour which happens twice in October is two
|
||||
instants, and a 24-slot local-hour importer collapses them.
|
||||
|
||||
So a year is 8760 hours only if it contains no transition, or contains both and
|
||||
they cancel - 25 in October, 23 in March. `--verify` therefore asserts the
|
||||
PER-DAY counts, not the total, because the total is exactly the number a broken
|
||||
importer also produces.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from collections import Counter, defaultdict
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from app.price import brussels # noqa: E402
|
||||
|
||||
API = "https://api.energy-charts.info/price"
|
||||
BZN = "BE"
|
||||
LICENCE_MUST_CONTAIN = "CC BY 4.0"
|
||||
UTC = timezone.utc
|
||||
|
||||
ATTRIBUTION = ("Data licensed CC BY 4.0 by Bundesnetzagentur | SMARD.de, "
|
||||
"retrieved via the Energy-Charts API (Fraunhofer ISE), "
|
||||
"https://energy-charts.info. Published without changes.")
|
||||
|
||||
# ⚠️ A GUESS, LABELLED AS ONE, AND APPLIED BY THE CONSUMER - NOT HERE.
|
||||
# Order-of-magnitude 2026 Belgian residential dynamic contract. Nothing in this
|
||||
# repository measured any part of it.
|
||||
ALL_IN_ASSUMPTION = {
|
||||
"form": "all_in = (wholesale + markup + network_and_levies) * (1 + vat)",
|
||||
"markup_eur_per_kwh": 0.020,
|
||||
"network_and_levies_eur_per_kwh": 0.100,
|
||||
"vat_fraction": 0.06,
|
||||
"status": "ASSUMPTION, NOT MEASURED. Stated so a consumer applies it "
|
||||
"deliberately; the committed eur_per_kwh has none of it applied.",
|
||||
"note": "The two additive terms cancel out of any spread, so an arbitrage "
|
||||
"trade count depends on vat_fraction only. They do NOT cancel out "
|
||||
"of the negative-price guard (17), which compares one hour's "
|
||||
"all-in price against zero.",
|
||||
}
|
||||
|
||||
|
||||
PACE_S = 3.0 # between chunks, unconditionally
|
||||
RETRIES = 6
|
||||
|
||||
|
||||
def _get(url: str) -> dict:
|
||||
"""One GET, backing off on 429 and 5xx.
|
||||
|
||||
⚠️ THIS IS A FREE PUBLIC SERVICE AND IT RATE-LIMITS. The first run of this
|
||||
script was refused with 429 on its fifth chunk. Backing off is the polite
|
||||
behaviour; it is also the correct one, because the alternative - catching
|
||||
the error and carrying on - would produce a file that is silently missing a
|
||||
month and says nothing about it. A refused request is not an empty month.
|
||||
"""
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
headers={"User-Agent": "goodwe-addon/DATA-02 (one-shot archival fetch)"})
|
||||
wait = 5.0
|
||||
for attempt in range(1, RETRIES + 1):
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=90) as r:
|
||||
return json.loads(r.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code not in (429, 500, 502, 503, 504) or attempt == RETRIES:
|
||||
raise
|
||||
print(f" HTTP {e.code}; backing off {wait:.0f}s "
|
||||
f"(attempt {attempt}/{RETRIES})", file=sys.stderr)
|
||||
time.sleep(wait)
|
||||
wait *= 2
|
||||
raise SystemExit(f"unreachable: {url}")
|
||||
|
||||
|
||||
def local_midnight_utc(day: str) -> datetime:
|
||||
"""The INSTANT at which Brussels midnight begins on `day` (YYYY-MM-DD).
|
||||
|
||||
⚠️ THE WINDOW IS LOCAL, NOT UTC, AND THE FIRST RUN OF THIS SCRIPT PROVED WHY.
|
||||
Bounding it at 00:00 UTC starts the year at 02:00 CEST, which makes the first
|
||||
Brussels day 22 hours long - and 22 is not one of the two lengths a
|
||||
transition produces, so `report()` refused the file. A dataset whose first
|
||||
day is short is a dataset whose "days per year" arithmetic is quietly wrong
|
||||
everywhere downstream.
|
||||
|
||||
Derived from `brussels()` by trying the only two offsets Belgium has and
|
||||
keeping the one that round-trips, rather than by restating the EU rule here.
|
||||
"""
|
||||
for hours in (1, 2):
|
||||
guess = datetime.fromisoformat(f"{day}T00:00:00").replace(
|
||||
tzinfo=timezone(timedelta(hours=hours))).astimezone(UTC)
|
||||
if brussels(guess).isoformat() == f"{day}T00:00:00+0{hours}:00":
|
||||
return guess
|
||||
raise SystemExit(f"no Brussels midnight for {day!r} - brussels() disagrees "
|
||||
f"with itself, which is a bug in price.py, not here.")
|
||||
|
||||
|
||||
def _months(start: datetime, end: datetime):
|
||||
"""[start, end) in month-sized chunks, so a bad chunk names its own window."""
|
||||
a = start
|
||||
while a < end:
|
||||
b = min(end, (a.replace(day=1) + timedelta(days=32)).replace(day=1))
|
||||
yield a, b
|
||||
a = b
|
||||
|
||||
|
||||
def fetch(start: datetime, end: datetime):
|
||||
"""Raw {unix_second: EUR/MWh} over [start, end), deduplicated.
|
||||
|
||||
⚠️ A `null` price is DROPPED, never coerced. The API returns null for an
|
||||
interval it holds no result for, and `or 0.0` is the version of this line
|
||||
that ships a year of free electricity.
|
||||
"""
|
||||
seen, licences, nulls = {}, Counter(), 0
|
||||
for a, b in _months(start, end):
|
||||
# ⚠️ THE API'S `end` IS AN INCLUSIVE LOCAL DAY, AND THE WINDOW'S IS AN
|
||||
# EXCLUSIVE INSTANT. Ask for the day containing the LAST hour of the
|
||||
# chunk, derived from the instant rather than by subtracting a calendar
|
||||
# day from the bound: `b - 1 day` drops the final local day whenever the
|
||||
# window ends at local midnight, which is exactly how this window ends.
|
||||
# The first run lost 2026-08-31 that way, silently - 8736 points instead
|
||||
# of 8760 - and the points-plus-gaps accounting in report() is what
|
||||
# noticed. A day nobody asked for is not a day with no prices.
|
||||
first = brussels(a).date().isoformat()
|
||||
last = brussels(b - timedelta(hours=1)).date().isoformat()
|
||||
url = f"{API}?bzn={BZN}&start={first}&end={last}"
|
||||
d = _get(url)
|
||||
lic = str(d.get("license_info", ""))
|
||||
licences[lic] += 1
|
||||
if LICENCE_MUST_CONTAIN not in lic:
|
||||
raise SystemExit(
|
||||
f"REFUSING TO WRITE. {url} came back under licence {lic!r}, "
|
||||
f"which does not say {LICENCE_MUST_CONTAIN!r}. The BE zone was "
|
||||
f"redistributable when this card ran; if that has changed, the "
|
||||
f"right output is this script and a checksum, not the data.")
|
||||
unit = str(d.get("unit", ""))
|
||||
if "MWh" not in unit:
|
||||
raise SystemExit(
|
||||
f"REFUSING TO WRITE. {url} reports unit {unit!r}, not EUR/MWh. "
|
||||
f"A silent unit change is a factor of 1000 on every price.")
|
||||
for t, p in zip(d["unix_seconds"], d["price"]):
|
||||
if p is None:
|
||||
nulls += 1
|
||||
continue
|
||||
if start.timestamp() <= t < end.timestamp():
|
||||
seen[int(t)] = float(p)
|
||||
print(f" {a.date()} .. {b.date()} {len(d['price']):5d} intervals",
|
||||
file=sys.stderr)
|
||||
time.sleep(PACE_S)
|
||||
return seen, licences, nulls
|
||||
|
||||
|
||||
def to_hourly(raw: dict):
|
||||
"""Hourly means keyed by UTC hour. Returns (hours, gaps, shape_counts).
|
||||
|
||||
An hour is emitted only when every interval the source used for it arrived:
|
||||
1 of 1 where the source was hourly, 4 of 4 where it was quarter-hourly.
|
||||
Anything else is a gap and is named as one.
|
||||
"""
|
||||
buckets = defaultdict(dict)
|
||||
for t, p in raw.items():
|
||||
buckets[t - (t % 3600)][t % 3600] = p
|
||||
|
||||
hours, gaps, shapes = {}, [], Counter()
|
||||
for h in sorted(buckets):
|
||||
got = buckets[h]
|
||||
if set(got) == {0}:
|
||||
shapes["hourly"] += 1
|
||||
hours[h] = got[0]
|
||||
elif set(got) == {0, 900, 1800, 2700}:
|
||||
shapes["quarter_hourly_mean"] += 1
|
||||
hours[h] = sum(got.values()) / 4.0
|
||||
else:
|
||||
gaps.append({
|
||||
"hour_utc": datetime.fromtimestamp(h, UTC).isoformat(),
|
||||
"intervals_present": sorted(got),
|
||||
"why": "incomplete hour - dropped rather than averaged over "
|
||||
"what arrived; a partial read is not an observation",
|
||||
})
|
||||
return hours, gaps, shapes
|
||||
|
||||
|
||||
def canonical(hours: dict) -> list:
|
||||
"""9.3 points. THREE FIELDS, rendered at the Brussels offset for the instant."""
|
||||
return [{"start": brussels(datetime.fromtimestamp(h, UTC)).isoformat(),
|
||||
"eur_per_kwh": round(hours[h] / 1000.0, 6) + 0.0,
|
||||
"provisional": False}
|
||||
for h in sorted(hours)]
|
||||
|
||||
|
||||
def expected_hours(start: datetime, end: datetime) -> int:
|
||||
"""Whole hours in the window. Derived, rather than 8760 typed in."""
|
||||
return int((end - start).total_seconds() // 3600)
|
||||
|
||||
|
||||
def per_local_day(points) -> Counter:
|
||||
"""Points per BRUSSELS calendar day - the count that exposes a DST bug."""
|
||||
n = Counter()
|
||||
for p in points:
|
||||
n[p["start"][:10]] += 1
|
||||
return n
|
||||
|
||||
|
||||
def digest(points) -> str:
|
||||
"""A checksum over the POINTS, so reformatting the envelope does not move it."""
|
||||
h = hashlib.sha256()
|
||||
for p in points:
|
||||
h.update(f"{p['start']}|{p['eur_per_kwh']:.6f}|{int(p['provisional'])}\n"
|
||||
.encode())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def load(path: str) -> dict:
|
||||
op = gzip.open if path.endswith(".gz") else open
|
||||
with op(path, "rt", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def report(doc, *, start=None, end=None) -> int:
|
||||
"""Re-derive every claim the file makes about itself. Returns failures."""
|
||||
pts = doc["prices"]
|
||||
prov = doc["provenance"]
|
||||
bad = 0
|
||||
|
||||
def ck(name, cond):
|
||||
nonlocal bad
|
||||
print(f" {'ok ' if cond else 'FAIL'} {name}")
|
||||
if not cond:
|
||||
bad += 1
|
||||
|
||||
ck(f"the checksum over the {len(pts)} points matches the recorded one "
|
||||
f"({prov['sha256_of_points'][:16]}...)",
|
||||
digest(pts) == prov["sha256_of_points"])
|
||||
ck("every point has exactly the three canonical fields and nothing else",
|
||||
all(set(p) == {"start", "eur_per_kwh", "provisional"} for p in pts))
|
||||
ck("every start carries a UTC offset, and it is +01:00 or +02:00 - a naive "
|
||||
"timestamp here is a wall clock plus an assumption",
|
||||
bool(pts) and all(p["start"][-6:] in ("+01:00", "+02:00") for p in pts))
|
||||
ck("the series is strictly increasing in time, so no instant is duplicated",
|
||||
all(datetime.fromisoformat(a["start"]) < datetime.fromisoformat(b["start"])
|
||||
for a, b in zip(pts, pts[1:])))
|
||||
ck("consecutive points are exactly one hour apart AS INSTANTS, so the two "
|
||||
"02:00s of the October day are two hours rather than one",
|
||||
all((datetime.fromisoformat(b["start"])
|
||||
- datetime.fromisoformat(a["start"])) == timedelta(hours=1)
|
||||
for a, b in zip(pts, pts[1:])))
|
||||
|
||||
days = per_local_day(pts)
|
||||
odd = {d: n for d, n in days.items() if n != 24}
|
||||
ck(f"exactly two Brussels days are not 24 hours long, and they are the "
|
||||
f"transitions: {sorted(odd.items())}",
|
||||
sorted(odd.values()) == [23, 25] and len(odd) == 2)
|
||||
spring = [d for d, n in odd.items() if n == 23]
|
||||
autumn = [d for d, n in odd.items() if n == 25]
|
||||
ck(f"the 23-hour day {spring} is in March and the 25-hour day {autumn} is "
|
||||
f"in October, which is the direction the EU rule actually runs",
|
||||
len(spring) == 1 and len(autumn) == 1
|
||||
and spring[0][5:7] == "03" and autumn[0][5:7] == "10")
|
||||
if autumn:
|
||||
twice = [p["start"] for p in pts
|
||||
if p["start"].startswith(autumn[0] + "T02")]
|
||||
ck(f"...and 02:00 on {autumn[0]} occurs TWICE at two different offsets "
|
||||
f"{twice} - exactly the case a 24-slot local-hour array collapses",
|
||||
len(twice) == 2 and twice[0][-6:] == "+02:00"
|
||||
and twice[1][-6:] == "+01:00")
|
||||
ck(f"...and the two carry DIFFERENT prices, so the 25th hour is a real "
|
||||
f"observation and not the same one written down twice",
|
||||
len({p["eur_per_kwh"] for p in pts
|
||||
if p["start"].startswith(autumn[0] + "T02")}) == 2)
|
||||
if spring:
|
||||
ck(f"...and 02:00 on {spring[0]} does not occur at all, because that "
|
||||
f"wall clock reading never happened in Belgium",
|
||||
not any(p["start"].startswith(spring[0] + "T02") for p in pts))
|
||||
|
||||
if start and end:
|
||||
want = expected_hours(start, end)
|
||||
ck(f"{len(pts)} points for a window of {want} whole hours, and the "
|
||||
f"{len(prov['gaps'])} recorded gaps account for the difference "
|
||||
f"exactly", len(pts) + len(prov["gaps"]) == want)
|
||||
|
||||
# ⚠️ A ZERO IS NOT AUTOMATICALLY A DEFECT HERE, AND SAYING IT WAS WOULD BE
|
||||
# THE WRONG LESSON. The Belgian market really does clear at 0.00 EUR/MWh in
|
||||
# solar-glut hours; this window contains such hours. What must be impossible
|
||||
# is a zero that arrived by COERCION - a dropped read turned into a free
|
||||
# hour. That is excluded structurally rather than by outlawing the value:
|
||||
# nulls are dropped, incomplete hours become gaps, and the accounting above
|
||||
# shows points + gaps covering every hour of the window, so every emitted
|
||||
# point stands for intervals the source actually published.
|
||||
zeros = [p["start"] for p in pts if p["eur_per_kwh"] == 0.0]
|
||||
ck(f"the {len(zeros)} hours priced at exactly zero match the count recorded "
|
||||
f"at build time, so no zero appeared between the fetch and the file",
|
||||
len(zeros) == prov["exact_zero_hours"])
|
||||
ck("...and none of them is a coerced read: every emitted hour is backed by "
|
||||
"published intervals, which is what the points-plus-gaps accounting "
|
||||
"above establishes",
|
||||
len(prov["gaps"]) == 0 or all("intervals_present" in g
|
||||
for g in prov["gaps"]))
|
||||
ck("the figures are declared RAW WHOLESALE, and the all-in assumption is "
|
||||
"recorded as an assumption rather than applied",
|
||||
prov["all_in"] is False
|
||||
and "ASSUMPTION" in prov["all_in_assumption"]["status"])
|
||||
return bad
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--start", default="2025-09-01")
|
||||
ap.add_argument("--end", default="2026-09-01",
|
||||
help="exclusive; both bounds are BRUSSELS-local midnights")
|
||||
ap.add_argument("--out", default=None)
|
||||
ap.add_argument("--verify", default=None,
|
||||
help="re-derive the claims of an existing file; no network")
|
||||
a = ap.parse_args()
|
||||
|
||||
if a.verify:
|
||||
doc = load(a.verify)
|
||||
print(f"verifying {a.verify}")
|
||||
s = datetime.fromisoformat(doc["provenance"]["window_utc"]["start"])
|
||||
e = datetime.fromisoformat(doc["provenance"]["window_utc"]["end"])
|
||||
bad = report(doc, start=s, end=e)
|
||||
print("OK" if not bad else f"{bad} FAILED")
|
||||
return 1 if bad else 0
|
||||
|
||||
start = local_midnight_utc(a.start)
|
||||
end = local_midnight_utc(a.end)
|
||||
out = a.out or f"scenarios/epex-be-dayahead-{a.start}_{a.end}.json"
|
||||
|
||||
print(f"fetching {BZN} day-ahead {start.date()} .. {end.date()}",
|
||||
file=sys.stderr)
|
||||
raw, licences, nulls = fetch(start, end)
|
||||
hours, gaps, shapes = to_hourly(raw)
|
||||
pts = canonical(hours)
|
||||
|
||||
doc = {
|
||||
"provenance": {
|
||||
"card": "DATA-02",
|
||||
"what": "EPEX BE day-ahead auction clearing prices, hourly",
|
||||
"source": f"{API}?bzn={BZN} (Energy-Charts API, Fraunhofer ISE)",
|
||||
"upstream": "Bundesnetzagentur | SMARD.de",
|
||||
"licence": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)",
|
||||
"licence_checked": "before committing anything; the API's OpenAPI "
|
||||
"document lists BE among the CC BY 4.0 zones and "
|
||||
"the remaining zones as private/internal use "
|
||||
"only, so redistribution of THIS zone is "
|
||||
"permitted with attribution",
|
||||
"licence_as_returned": sorted(licences),
|
||||
"attribution": ATTRIBUTION,
|
||||
"cross_check": "SMARD chart_data 4996/DE over 2026-04-10 matched "
|
||||
"this feed on all 96 quarter-hours, max abs "
|
||||
"difference 0.00 EUR/MWh (2026-08-31)",
|
||||
"retrieved_utc": datetime.now(UTC).replace(microsecond=0).isoformat(),
|
||||
"fetched_by": "scenarios/fetch_epex_be.py",
|
||||
"window_utc": {"start": start.isoformat(), "end": end.isoformat()},
|
||||
"unit": "EUR per kWh (source publishes EUR/MWh; divided by 1000)",
|
||||
"all_in": False,
|
||||
"all_in_note": "RAW WHOLESALE. No supplier markup, no network or "
|
||||
"transmission tariff, no levies, no VAT. 9.3's "
|
||||
"canonical point is all-in and this is NOT; the "
|
||||
"consumer applies the assumption below, visibly.",
|
||||
"all_in_assumption": ALL_IN_ASSUMPTION,
|
||||
"resolution": {
|
||||
"note": "EPEX day-ahead moved to a 15-minute market time unit "
|
||||
"in Oct 2025, part-way through this window.",
|
||||
"hours_from_hourly_source": shapes["hourly"],
|
||||
"hours_from_quarter_hourly_mean": shapes["quarter_hourly_mean"],
|
||||
"aggregation": "unweighted mean of the four quarters; a stated "
|
||||
"approximation of the hourly product, and a "
|
||||
"LOWER bound on the spread a 15-minute strategy "
|
||||
"would have seen",
|
||||
},
|
||||
"timezone": "starts rendered at the Brussels offset by "
|
||||
"app.price.brussels(); instants carried in UTC, so the "
|
||||
"transition days come out 23 and 25 points by "
|
||||
"construction rather than by a special case",
|
||||
"nulls_dropped": nulls,
|
||||
"gaps": gaps,
|
||||
"gaps_note": "hours the source did not publish completely. Dropped, "
|
||||
"never zero-filled: a missing hour is not a free hour.",
|
||||
"points": len(pts),
|
||||
"exact_zero_hours": sum(1 for p in pts if p["eur_per_kwh"] == 0.0),
|
||||
"negative_hours": sum(1 for p in pts if p["eur_per_kwh"] < 0),
|
||||
"zero_note": "REAL CLEARINGS AT 0.00 EUR/MWh, not dropped reads. "
|
||||
"Nulls are dropped and incomplete hours become gaps, "
|
||||
"so points + gaps == hours in the window and no point "
|
||||
"stands in for something that failed to arrive.",
|
||||
"sha256_of_points": digest(pts),
|
||||
},
|
||||
"prices": pts,
|
||||
}
|
||||
|
||||
with open(out, "w", encoding="utf-8", newline="\n") as f:
|
||||
json.dump(doc, f, indent=1, ensure_ascii=False)
|
||||
f.write("\n")
|
||||
print(f"wrote {out} {len(pts)} points {os.path.getsize(out)} bytes",
|
||||
file=sys.stderr)
|
||||
print(f"verifying {out}")
|
||||
return 1 if report(doc, start=start, end=end) else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,167 @@
|
||||
# 30 days of P1 net power — 2026-08-01 … 2026-08-30
|
||||
|
||||
`sensor.p1_meter_active_power` out of Home Assistant's recorder, one gzipped
|
||||
JSONL file per **local** day (Europe/Brussels), same schema as the two files one
|
||||
directory up:
|
||||
|
||||
{"t": "2026-08-22T22:00:00+00:00", "w": -25.0}
|
||||
|
||||
**485,841 samples, 30/30 days read, 0 days failed, 4.13 MB gzipped.**
|
||||
Produced by `tools/export_p1_history.py`; `manifest.json` here carries the
|
||||
per-day counts, sha256 of every file, and every gap.
|
||||
|
||||
python3 tools/export_p1_history.py --days 30 --end-date 2026-08-30 \
|
||||
--out-dir scenarios/p1-net-30d-2026-08
|
||||
|
||||
## Why this exists
|
||||
|
||||
[[TARIFF-01]] AC 3 and [[SAFETY-07]] AC 3 both ask for *"monthly peak held under
|
||||
the soft limit in >= 95 % of intervals over a 30-day simulation replay"*, and
|
||||
both were merged with the criterion honestly unmet because only two days of
|
||||
capture existed. `test_tariff.py`'s docstring still recorded the reason as
|
||||
"nothing longer was ever captured" — which was true about the past and was
|
||||
quietly read as a statement about the future. It was not one. Probed on
|
||||
2026-08-31, HA's recorder answered a 20-minute window with 230 samples at 30
|
||||
days back and 0 at 45, so the whole month was still there and was exported the
|
||||
same day. **The window rolls: 2026-08-01 leaves the recorder around 2026-09-15.**
|
||||
|
||||
## The size decision — COMMITTED, deliberately, at 4.13 MB
|
||||
|
||||
Three options were on the table. This one was taken and the cost is stated
|
||||
rather than absorbed, and it is measured rather than estimated - a
|
||||
`git bundle` of the whole history before and after this commit:
|
||||
|
||||
release/1.0 @ 86c43eb 1.58 MB
|
||||
DATA-01 5.28 MB +3.70 MB, x3.3
|
||||
|
||||
A full clone of this repository therefore more than triples, and a Home
|
||||
Assistant supervisor clones it on every add-on install and update. That is the
|
||||
price; the paragraphs below are why it was judged worth paying rather than
|
||||
avoided by making the data worse or making it disappear.
|
||||
|
||||
* **Downsampling is not available.** It is not a matter of taste: the shipped
|
||||
`meter_max_age_s` is **30 s**, which is the age past which the controller's
|
||||
own ingest stops counting a held reading as coverage. Any resample coarser
|
||||
than 30 s therefore converts real coverage into modelled meter-gap and changes
|
||||
the very number AC 3 measures. Resampling at or under 30 s saves almost
|
||||
nothing against a 5 s cadence.
|
||||
* **"Outside the repo with a pointer" is what already happened, and it is why
|
||||
this card exists.** The two committed days came from the infra repo's private
|
||||
`sim/scenarios/`, the one-off export script was thrown away, and three cards
|
||||
then deferred an acceptance criterion on the belief that the data could not be
|
||||
produced again. `test_tariff.py` states the general rule verbatim: *a
|
||||
regression fixture CI cannot see is not a regression fixture.* CI's token is
|
||||
minted for this repository only.
|
||||
* **So: committed** — but in a SUBDIRECTORY, which is load-bearing. Both
|
||||
`test_tariff.py` part 9 and `test_peakguard.py` part 6 do
|
||||
`(HERE / "scenarios").glob("p1-net-*.jsonl.gz")`, which does not recurse. Drop
|
||||
these thirty files beside the two and CI grows sixty full-day
|
||||
replays - about 50 s of pure compute - on every PR forever, `test_tariff.py`'s
|
||||
`check(... replay_days < 30)` starts failing, and both files' honesty notes
|
||||
become false. In `p1-net-30d-2026-08/` they are visible to CI, invisible to
|
||||
the globs, and the two committed days keep their job as the small fixtures the
|
||||
suites grade against.
|
||||
|
||||
They are also written once and never edited, so git stores one already-deflated
|
||||
blob per day with no delta churn.
|
||||
|
||||
## Gaps are visible, never interpolated
|
||||
|
||||
Nothing is filled, carried forward or resampled. A hole in the recorder is a
|
||||
hole here — consecutive `t` values further apart than the cadence — which is
|
||||
exactly what the controller's freshness rule already reads as *no coverage*, so
|
||||
a missing hour replays as a missing hour rather than as an hour of whatever the
|
||||
last sample said.
|
||||
|
||||
**13 gaps over 30 s, 1291 s in total: 99.950 % of the month is covered.**
|
||||
All of them are listed in `manifest.json` and by
|
||||
`python3 tools/replay_ac3.py <dir> --report-gaps`. The longest is 263 s on
|
||||
2026-08-11.
|
||||
|
||||
Three samples in the month carried the state `unavailable` rather than a number
|
||||
(2026-08-18 02:03:54Z and 04:10:41Z, 2026-08-25 01:52:02Z). There is no float to
|
||||
write and the committed schema has no slot for a non-number, so they are absent
|
||||
from the series and recorded individually in `manifest.json` under
|
||||
`non_numeric_states`. Each one sits inside a gap that is already listed, which
|
||||
is the honest rendering: the meter was not reporting a power then.
|
||||
|
||||
**A failed read is never an empty hour.** If a day's read fails, the exporter
|
||||
writes NO FILE, records `"read": "FAILED"` with the error, and exits non-zero —
|
||||
so a missing day is unreadable-and-known rather than silently zero. The entity
|
||||
is also proved to exist through `/api/states` before any day is trusted, because
|
||||
a typo would otherwise answer `200 []` thirty times and produce a beautifully
|
||||
consistent dataset of nothing. On this run: 30 read, 0 failed, 0 empty.
|
||||
|
||||
## AC 3, measured
|
||||
|
||||
cd goodwe_controller
|
||||
python3 tools/replay_ac3.py scenarios/p1-net-30d-2026-08
|
||||
|
||||
485841 samples, 2879 closed quarters (30.0 days' worth)
|
||||
month 2026-08: peak 5029.8 W uncontrolled -> 4107.3 W guarded
|
||||
|
||||
TARIFF-01 AC 3 (UNCONTROLLED): 2847/2879 = 98.89 % of quarters at or under 3500 W
|
||||
SAFETY-07 AC 3 (GUARDED): 2850/2879 = 98.99 %
|
||||
guard cycles 143810, engaged on the measurement 1542, blind 195,
|
||||
saturated 0, modelled discharge 8060 Wh
|
||||
|
||||
2879 rather than 2880 quarters: the last quarter of 2026-08-30 closes only when
|
||||
a telegram past its boundary arrives, and that telegram is 2026-08-31's.
|
||||
|
||||
### What the numbers actually say — three findings, not a pass
|
||||
|
||||
1. **AC 3 is met on the ">= 95 % of intervals" reading, and the criterion turns
|
||||
out to be nearly insensitive.** The UNCONTROLLED month — no controller, no
|
||||
peak shaving, nothing running — already scores 98.89 %. A criterion that a
|
||||
house passes by doing nothing does not discriminate between a working guard
|
||||
and an absent one. That is a finding about the acceptance criterion, and it
|
||||
is reported rather than filed as a win.
|
||||
2. **On the other reading of the same sentence, AC 3 is NOT met.** "Monthly peak
|
||||
held under the soft limit" taken as *the peak itself* fails: the guard pulls
|
||||
August's peak from **5029.8 W to 4107.3 W** (-922.5 W, -18.3 %), which is
|
||||
real money on a Belgian capacity tariff, but 4107.3 W is still over the
|
||||
3500 W soft limit. Both readings are printed by the tool; neither is hidden.
|
||||
3. **The guard stops defending the soft limit once the month's peak is set, and
|
||||
that is by design.** `quarter_ceiling_w` uses `CEILING_MONTH_PEAK`: the
|
||||
billed quantity is the month's maximum, so a quarter under an
|
||||
already-recorded peak costs nothing extra and the guard does not spend the
|
||||
battery on it. The month's 32 over-cap quarters fall on four nights
|
||||
(2026-08-17, -18, -20, -21); **31 of them are pulled down** — 7494 W of
|
||||
quarter-average removed in total, 8060 Wh of modelled discharge — but only
|
||||
**3 land under 3500 W**, because by then the ceiling being defended is the
|
||||
running month peak (~4107 W), not the soft limit. The single untouched
|
||||
quarter, 2026-08-18T08:15Z at 4039.7 W, is the same rule seen from the other
|
||||
side: the running peak was already 4833 W, so 4039.7 W added nothing to the
|
||||
bill and the guard correctly stayed out. AC 3's wording asks the guard for
|
||||
something its own tariff logic deliberately does not optimise.
|
||||
|
||||
Also worth an eye: **195 blind engagements** over 143,810 cycles — the guard
|
||||
acting on a projection it could not fully cover. On the two committed days that
|
||||
count was 5. It scales with the gaps above.
|
||||
|
||||
## Verification of the exporter
|
||||
|
||||
`tools/export_p1_history.py --verify` re-fetches a committed day by the exact
|
||||
route used here and diffs it against the file:
|
||||
|
||||
python3 tools/export_p1_history.py \
|
||||
--verify scenarios/p1-net-2026-08-20.jsonl.gz \
|
||||
scenarios/p1-net-2026-08-23.jsonl.gz
|
||||
|
||||
Both days come back **byte-identical after decompression** — 865,024 B /
|
||||
15,600 samples and 894,387 B / 16,351 samples — and the copies of those two days
|
||||
inside this directory are byte-identical to the committed ones too. The gzip
|
||||
CONTAINERS differ by design: a gzip header carries an mtime and a stored
|
||||
filename, and this exporter writes mtime 0 so its own output is reproducible,
|
||||
where the 2026-08-25 one-off wrote a wall-clock mtime. Content equality is the
|
||||
claim; container equality would only assert that two files were written at the
|
||||
same second.
|
||||
|
||||
`tools/replay_ac3.py` is checked the same way rather than trusted: pointed at
|
||||
`scenarios/` it must reproduce `test_peakguard.py` part 6's published totals,
|
||||
and `--expect` makes it a hard check.
|
||||
|
||||
python3 tools/replay_ac3.py scenarios --expect 189/190
|
||||
|
||||
reproduces 189/190 quarters, the 5029.8 W billed quarter, the 3485.3 W guarded
|
||||
result and the 878 Wh of modelled discharge that part 6 prints.
|
||||
@@ -0,0 +1,767 @@
|
||||
{
|
||||
"generated": "2026-08-31T08:42:40.614548+00:00",
|
||||
"generator": "tools/export_p1_history.py",
|
||||
"source": "http://192.168.2.158:8123/api/history/period",
|
||||
"entity": "sensor.p1_meter_active_power",
|
||||
"timezone": "Europe/Brussels",
|
||||
"gap_threshold_s": 30.0,
|
||||
"format": "{\"t\": <ISO-8601 UTC>, \"w\": <float watts>} per line, gzip",
|
||||
"days": [
|
||||
{
|
||||
"date": "2026-08-01",
|
||||
"window": [
|
||||
"2026-08-01T00:00:00+02:00",
|
||||
"2026-08-02T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1128060,
|
||||
"attempts": 1,
|
||||
"samples": 16511,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-07-31T22:00:00+00:00",
|
||||
"last": "2026-08-01T21:59:56.268499+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86396.3,
|
||||
"mean_cadence_s": 5.233,
|
||||
"file": "p1-net-2026-08-01.jsonl.gz",
|
||||
"bytes_gz": 140531,
|
||||
"bytes_raw": 913203,
|
||||
"sha256_raw": "120de2a0b7eb552281dac4bc1f8fd76d947ba72d1f5b426d9dcdd85ec103bd06",
|
||||
"sha256_gz": "e09bdee349dd3668a073b7fd77974e639c269ec37e73e81a9706cadde083f064"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-02",
|
||||
"window": [
|
||||
"2026-08-02T00:00:00+02:00",
|
||||
"2026-08-03T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1115387,
|
||||
"attempts": 1,
|
||||
"samples": 16303,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-01T22:00:00+00:00",
|
||||
"last": "2026-08-02T21:59:55.191360+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86395.2,
|
||||
"mean_cadence_s": 5.3,
|
||||
"file": "p1-net-2026-08-02.jsonl.gz",
|
||||
"bytes_gz": 141404,
|
||||
"bytes_raw": 903234,
|
||||
"sha256_raw": "9ab4891c0abf9d621fa528848fc3a874687843804768e8aeab6a88f7bb04b679",
|
||||
"sha256_gz": "94774139b3e19d04bcda3ac5ea988dd4d645e99004f90b917c6b0bf3fbac21cb"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-03",
|
||||
"window": [
|
||||
"2026-08-03T00:00:00+02:00",
|
||||
"2026-08-04T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1123668,
|
||||
"attempts": 1,
|
||||
"samples": 16424,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-02T22:00:00+00:00",
|
||||
"last": "2026-08-03T21:59:55.255808+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86395.3,
|
||||
"mean_cadence_s": 5.261,
|
||||
"file": "p1-net-2026-08-03.jsonl.gz",
|
||||
"bytes_gz": 144202,
|
||||
"bytes_raw": 909942,
|
||||
"sha256_raw": "13f8dd904a31b5b67fcfde04f6afc49c34c3a9504617278d79d87516fcd6441f",
|
||||
"sha256_gz": "e1bc0cb0e271cbc8d896aee97aa0f5c18e49e78905d1aaf137904a768047c7bc"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-04",
|
||||
"window": [
|
||||
"2026-08-04T00:00:00+02:00",
|
||||
"2026-08-05T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1108582,
|
||||
"attempts": 1,
|
||||
"samples": 16286,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-03T22:00:00+00:00",
|
||||
"last": "2026-08-04T21:59:58.729138+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.7,
|
||||
"mean_cadence_s": 5.305,
|
||||
"file": "p1-net-2026-08-04.jsonl.gz",
|
||||
"bytes_gz": 136505,
|
||||
"bytes_raw": 896650,
|
||||
"sha256_raw": "a65cd9d1ceda61e5a40043edb401c2154feea2b36bd3221710df808d778eabce",
|
||||
"sha256_gz": "c19b19ac890f4caad4140cb27d2fe6f3e972545f664127e5d9f457e711ad8c88"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-05",
|
||||
"window": [
|
||||
"2026-08-05T00:00:00+02:00",
|
||||
"2026-08-06T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1136556,
|
||||
"attempts": 1,
|
||||
"samples": 16655,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-04T22:00:00+00:00",
|
||||
"last": "2026-08-05T21:59:56.189152+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86396.2,
|
||||
"mean_cadence_s": 5.188,
|
||||
"file": "p1-net-2026-08-05.jsonl.gz",
|
||||
"bytes_gz": 142561,
|
||||
"bytes_raw": 919827,
|
||||
"sha256_raw": "dd327685c6f322bfaa49bc5bf228301e790e5a96fd7c23c487b845e44a962db4",
|
||||
"sha256_gz": "5ff7a1336c49ba38b8058820181dd8ed9c2b8db41b5a3a52412a5a3d8c515332"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-06",
|
||||
"window": [
|
||||
"2026-08-06T00:00:00+02:00",
|
||||
"2026-08-07T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1124419,
|
||||
"attempts": 1,
|
||||
"samples": 16412,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-05T22:00:00+00:00",
|
||||
"last": "2026-08-06T21:59:57.192711+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86397.2,
|
||||
"mean_cadence_s": 5.265,
|
||||
"file": "p1-net-2026-08-06.jsonl.gz",
|
||||
"bytes_gz": 142313,
|
||||
"bytes_raw": 910849,
|
||||
"sha256_raw": "7cd1fe8315e24dc6391693c49d6306b0c0cf9b6c3a478e457da921001cea930c",
|
||||
"sha256_gz": "9108f169d645b9da07d4ae310666a1365ea07275ff311cd60adad1b702edbf11"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-07",
|
||||
"window": [
|
||||
"2026-08-07T00:00:00+02:00",
|
||||
"2026-08-08T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1087531,
|
||||
"attempts": 1,
|
||||
"samples": 15952,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-06T22:00:00+00:00",
|
||||
"last": "2026-08-07T21:59:58.195053+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.2,
|
||||
"mean_cadence_s": 5.416,
|
||||
"file": "p1-net-2026-08-07.jsonl.gz",
|
||||
"bytes_gz": 135979,
|
||||
"bytes_raw": 879941,
|
||||
"sha256_raw": "e0f67a9214980e2fd4893208cb402165065dadce2105ce1581c6b1d49952f39a",
|
||||
"sha256_gz": "b70b84f3743d1d84055c911bcd5589cb8f0371e8916bde741308c5bb50702c29"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-08",
|
||||
"window": [
|
||||
"2026-08-08T00:00:00+02:00",
|
||||
"2026-08-09T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1058569,
|
||||
"attempts": 1,
|
||||
"samples": 15468,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-07T22:00:00+00:00",
|
||||
"last": "2026-08-08T21:59:52.805887+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86392.8,
|
||||
"mean_cadence_s": 5.586,
|
||||
"file": "p1-net-2026-08-08.jsonl.gz",
|
||||
"bytes_gz": 134025,
|
||||
"bytes_raw": 857271,
|
||||
"sha256_raw": "0be14ca7593b4c7d68570115cc26e24f6a26ab87fe3c168a8b942b167e8d4f68",
|
||||
"sha256_gz": "d93dcc8ca3eae03af13b9ae36acd5ab1eaa1e7ee93c237d386e8ded1c61f5906"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-09",
|
||||
"window": [
|
||||
"2026-08-09T00:00:00+02:00",
|
||||
"2026-08-10T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1059349,
|
||||
"attempts": 1,
|
||||
"samples": 15508,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-08T22:00:00+00:00",
|
||||
"last": "2026-08-09T21:59:58.178998+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.2,
|
||||
"mean_cadence_s": 5.572,
|
||||
"file": "p1-net-2026-08-09.jsonl.gz",
|
||||
"bytes_gz": 135697,
|
||||
"bytes_raw": 857531,
|
||||
"sha256_raw": "f4793799173c502fcee5f78e2d9b0dcc984b5d2d8bd2fc0db0cd0f8d403733b3",
|
||||
"sha256_gz": "2fea6c5c8b1452d72ad5aa3b0ae9755bb2ea08071514c08b1a3b1155ef457cf1"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-10",
|
||||
"window": [
|
||||
"2026-08-10T00:00:00+02:00",
|
||||
"2026-08-11T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1066721,
|
||||
"attempts": 1,
|
||||
"samples": 15594,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-09T22:00:00+00:00",
|
||||
"last": "2026-08-10T21:59:59.170056+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86399.2,
|
||||
"mean_cadence_s": 5.541,
|
||||
"file": "p1-net-2026-08-10.jsonl.gz",
|
||||
"bytes_gz": 136682,
|
||||
"bytes_raw": 863785,
|
||||
"sha256_raw": "ff5c3fa419505127a51acc56dfc13e8437e4ec2c13a11c8d8b2d6c3e20e2410e",
|
||||
"sha256_gz": "4fd30e495f4e6d1c8501d05c23f9891ac34364d86a9c55ac825d52b2ebb31023"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-11",
|
||||
"window": [
|
||||
"2026-08-11T00:00:00+02:00",
|
||||
"2026-08-12T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1056281,
|
||||
"attempts": 1,
|
||||
"samples": 15371,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-10T22:00:00+00:00",
|
||||
"last": "2026-08-11T21:59:58.608150+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-11T08:00:14.121653+00:00",
|
||||
"to": "2026-08-11T08:02:54.427007+00:00",
|
||||
"seconds": 160.305
|
||||
},
|
||||
{
|
||||
"from": "2026-08-11T17:04:18.517592+00:00",
|
||||
"to": "2026-08-11T17:08:41.759777+00:00",
|
||||
"seconds": 263.242
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86398.6,
|
||||
"mean_cadence_s": 5.621,
|
||||
"file": "p1-net-2026-08-11.jsonl.gz",
|
||||
"bytes_gz": 133711,
|
||||
"bytes_raw": 856244,
|
||||
"sha256_raw": "67c2a740019f8908405b96b3436cd953af27baead2957acc7c1dcf10b37ac94e",
|
||||
"sha256_gz": "4f8c553edc9051644bd7baa659f6f92a8b5f5ce12a0410c8b5235d1e6c3084c9"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-12",
|
||||
"window": [
|
||||
"2026-08-12T00:00:00+02:00",
|
||||
"2026-08-13T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1124945,
|
||||
"attempts": 1,
|
||||
"samples": 16432,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-11T22:00:00+00:00",
|
||||
"last": "2026-08-12T21:59:58.648966+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.6,
|
||||
"mean_cadence_s": 5.258,
|
||||
"file": "p1-net-2026-08-12.jsonl.gz",
|
||||
"bytes_gz": 136903,
|
||||
"bytes_raw": 911115,
|
||||
"sha256_raw": "73c8a1626640226ddffb32d564557d1cbe57a15367ea1ba3632934e15a1413dc",
|
||||
"sha256_gz": "acec87065b7de2c605db00504ec5a9b3c99584620dfb3b67194e2cf9464d6a5f"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-13",
|
||||
"window": [
|
||||
"2026-08-13T00:00:00+02:00",
|
||||
"2026-08-14T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1126742,
|
||||
"attempts": 1,
|
||||
"samples": 16473,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-12T22:00:00+00:00",
|
||||
"last": "2026-08-13T21:59:57.562043+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86397.6,
|
||||
"mean_cadence_s": 5.245,
|
||||
"file": "p1-net-2026-08-13.jsonl.gz",
|
||||
"bytes_gz": 136221,
|
||||
"bytes_raw": 912379,
|
||||
"sha256_raw": "1d9a16c529174bd8c74625a742c46d98de4acf7a045f83df8bdbac22fa86e8e7",
|
||||
"sha256_gz": "adfef36f629decf54c625f2dbfe6b4dab9441218c42557c90ac7f6d331943a59"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-14",
|
||||
"window": [
|
||||
"2026-08-14T00:00:00+02:00",
|
||||
"2026-08-15T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1124413,
|
||||
"attempts": 1,
|
||||
"samples": 16427,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-13T22:00:00+00:00",
|
||||
"last": "2026-08-14T21:59:55.611487+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86395.6,
|
||||
"mean_cadence_s": 5.26,
|
||||
"file": "p1-net-2026-08-14.jsonl.gz",
|
||||
"bytes_gz": 135570,
|
||||
"bytes_raw": 910648,
|
||||
"sha256_raw": "dc1417881e54fe215e9fcd4ed643d2482a277d61f8f27f25e2ff1e2cba29f7a8",
|
||||
"sha256_gz": "d4eff781cd16a83fd1be9e643fee5bb5d5897e7d45bb9865972a7304d92c84a4"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-15",
|
||||
"window": [
|
||||
"2026-08-15T00:00:00+02:00",
|
||||
"2026-08-16T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1124461,
|
||||
"attempts": 1,
|
||||
"samples": 16516,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-14T22:00:00+00:00",
|
||||
"last": "2026-08-15T21:59:56.581991+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86396.6,
|
||||
"mean_cadence_s": 5.231,
|
||||
"file": "p1-net-2026-08-15.jsonl.gz",
|
||||
"bytes_gz": 137670,
|
||||
"bytes_raw": 909539,
|
||||
"sha256_raw": "f07a242faed68313512e0363d914846522ff839315ad80b55689072ac74e8a69",
|
||||
"sha256_gz": "082314c6003c2208752acc4614b9809f06735d01e216b935ea94faf578c5598e"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-16",
|
||||
"window": [
|
||||
"2026-08-16T00:00:00+02:00",
|
||||
"2026-08-17T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1127871,
|
||||
"attempts": 1,
|
||||
"samples": 16552,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-15T22:00:00+00:00",
|
||||
"last": "2026-08-16T21:59:59.527710+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86399.5,
|
||||
"mean_cadence_s": 5.22,
|
||||
"file": "p1-net-2026-08-16.jsonl.gz",
|
||||
"bytes_gz": 138347,
|
||||
"bytes_raw": 912481,
|
||||
"sha256_raw": "a44e5d961600a539ddf824d5d211ef319833666b7598e5939a6cdf2cb1e3581f",
|
||||
"sha256_gz": "8f6511e52f5be9187f422021abc93a5fa8b698b4c785877d69d15dd6de106fd8"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-17",
|
||||
"window": [
|
||||
"2026-08-17T00:00:00+02:00",
|
||||
"2026-08-18T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1133721,
|
||||
"attempts": 1,
|
||||
"samples": 16627,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-16T22:00:00+00:00",
|
||||
"last": "2026-08-17T21:59:59.685699+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-17T17:27:00.599126+00:00",
|
||||
"to": "2026-08-17T17:28:20.389628+00:00",
|
||||
"seconds": 79.791
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86399.7,
|
||||
"mean_cadence_s": 5.197,
|
||||
"file": "p1-net-2026-08-17.jsonl.gz",
|
||||
"bytes_gz": 140214,
|
||||
"bytes_raw": 917356,
|
||||
"sha256_raw": "70cce79b3732f292f7f58f0b2655bdb4a00a20a6530f1f50dac62ef5f039788b",
|
||||
"sha256_gz": "e2f9b461c4db7902e1a3d30b439ebc11ebcd9618a72d12f10e5131f38fb437bf"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-18",
|
||||
"window": [
|
||||
"2026-08-18T00:00:00+02:00",
|
||||
"2026-08-19T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1074019,
|
||||
"attempts": 1,
|
||||
"samples": 15699,
|
||||
"non_numeric_states": [
|
||||
{
|
||||
"t": "2026-08-18T02:03:54.417106+00:00",
|
||||
"state": "unavailable"
|
||||
},
|
||||
{
|
||||
"t": "2026-08-18T04:10:41.786299+00:00",
|
||||
"state": "unavailable"
|
||||
}
|
||||
],
|
||||
"first": "2026-08-17T22:00:00+00:00",
|
||||
"last": "2026-08-18T21:59:54.501766+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-18T02:02:59.633845+00:00",
|
||||
"to": "2026-08-18T02:04:38.694366+00:00",
|
||||
"seconds": 99.061
|
||||
},
|
||||
{
|
||||
"from": "2026-08-18T04:09:54.292049+00:00",
|
||||
"to": "2026-08-18T04:11:38.500238+00:00",
|
||||
"seconds": 104.208
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86394.5,
|
||||
"mean_cadence_s": 5.504,
|
||||
"file": "p1-net-2026-08-18.jsonl.gz",
|
||||
"bytes_gz": 135894,
|
||||
"bytes_raw": 869570,
|
||||
"sha256_raw": "68cd44ac56099ee38f0bba4c32d7d17d33b92d28a99d27419f53c22374491ca6",
|
||||
"sha256_gz": "d6c97508f805aff040d2b4ed4e06c0e7373db2ea6227c7d79ef24c8c79074115"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-19",
|
||||
"window": [
|
||||
"2026-08-19T00:00:00+02:00",
|
||||
"2026-08-20T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 989720,
|
||||
"attempts": 1,
|
||||
"samples": 14549,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-18T22:00:00+00:00",
|
||||
"last": "2026-08-19T21:59:56.556610+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-19T01:33:29.599224+00:00",
|
||||
"to": "2026-08-19T01:34:01.653383+00:00",
|
||||
"seconds": 32.054
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86396.6,
|
||||
"mean_cadence_s": 5.939,
|
||||
"file": "p1-net-2026-08-19.jsonl.gz",
|
||||
"bytes_gz": 126345,
|
||||
"bytes_raw": 800369,
|
||||
"sha256_raw": "0f11ce0cee43cbd7bdb08dc48b202e58c442157ac08b2cda5b64bbab9bd56300",
|
||||
"sha256_gz": "13d61b7da903877c8ae7813b833cb5836ec04a1cf1efd0e1fa87e916bef95fd5"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-20",
|
||||
"window": [
|
||||
"2026-08-20T00:00:00+02:00",
|
||||
"2026-08-21T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1068038,
|
||||
"attempts": 1,
|
||||
"samples": 15600,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-19T22:00:00+00:00",
|
||||
"last": "2026-08-20T21:59:56.742604+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-20T02:11:56.607373+00:00",
|
||||
"to": "2026-08-20T02:12:38.840458+00:00",
|
||||
"seconds": 42.233
|
||||
},
|
||||
{
|
||||
"from": "2026-08-20T20:42:04.623102+00:00",
|
||||
"to": "2026-08-20T20:43:41.666543+00:00",
|
||||
"seconds": 97.043
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86396.7,
|
||||
"mean_cadence_s": 5.539,
|
||||
"file": "p1-net-2026-08-20.jsonl.gz",
|
||||
"bytes_gz": 139693,
|
||||
"bytes_raw": 865024,
|
||||
"sha256_raw": "2612153963951dd13fe310ad7fbb5bec2e195db56fd99fd963134c8badae25ba",
|
||||
"sha256_gz": "362f2ee3d9d7f23f9051ba78993c5af7443dabb3b5f16339efc673444720c892"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-21",
|
||||
"window": [
|
||||
"2026-08-21T00:00:00+02:00",
|
||||
"2026-08-22T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1118002,
|
||||
"attempts": 1,
|
||||
"samples": 16383,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-20T22:00:00+00:00",
|
||||
"last": "2026-08-21T21:59:57.771588+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86397.8,
|
||||
"mean_cadence_s": 5.274,
|
||||
"file": "p1-net-2026-08-21.jsonl.gz",
|
||||
"bytes_gz": 139105,
|
||||
"bytes_raw": 904809,
|
||||
"sha256_raw": "933d9a6899f59c128d0d9b84d6944cea63b1ad58dc57ac52e3e476f9c616f07b",
|
||||
"sha256_gz": "a563581a9a3338af173abb99a200db32881ddaf893d2458f5d8de59b5ec6041f"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-22",
|
||||
"window": [
|
||||
"2026-08-22T00:00:00+02:00",
|
||||
"2026-08-23T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1129094,
|
||||
"attempts": 1,
|
||||
"samples": 16520,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-21T22:00:00+00:00",
|
||||
"last": "2026-08-22T21:59:56.913786+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-21T23:05:20.035692+00:00",
|
||||
"to": "2026-08-21T23:07:51.147719+00:00",
|
||||
"seconds": 151.112
|
||||
},
|
||||
{
|
||||
"from": "2026-08-21T23:07:56.966604+00:00",
|
||||
"to": "2026-08-21T23:08:27.695101+00:00",
|
||||
"seconds": 30.728
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86396.9,
|
||||
"mean_cadence_s": 5.23,
|
||||
"file": "p1-net-2026-08-22.jsonl.gz",
|
||||
"bytes_gz": 141564,
|
||||
"bytes_raw": 914120,
|
||||
"sha256_raw": "39ccdf67c806344490f791648822cd2cd14d73ea7cc0b6e8572a1e15e8f74641",
|
||||
"sha256_gz": "5c46f1879a723f9986d521124200cab839af2a4722b76bc97c6f6f59d0df65a2"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-23",
|
||||
"window": [
|
||||
"2026-08-23T00:00:00+02:00",
|
||||
"2026-08-24T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1107164,
|
||||
"attempts": 1,
|
||||
"samples": 16351,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-22T22:00:00+00:00",
|
||||
"last": "2026-08-23T21:59:56.921115+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86396.9,
|
||||
"mean_cadence_s": 5.284,
|
||||
"file": "p1-net-2026-08-23.jsonl.gz",
|
||||
"bytes_gz": 136277,
|
||||
"bytes_raw": 894387,
|
||||
"sha256_raw": "ffb15367bf3c98eff8a5600e4858ac99778b597c03d0a4f6154e6971b8f98a44",
|
||||
"sha256_gz": "3fa76981758a468ba0aa68b50fe68d5bdb1ef38cd7faec2b6ffdb61c9ddb2db2"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-24",
|
||||
"window": [
|
||||
"2026-08-24T00:00:00+02:00",
|
||||
"2026-08-25T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1122175,
|
||||
"attempts": 1,
|
||||
"samples": 16584,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-23T22:00:00+00:00",
|
||||
"last": "2026-08-24T21:59:57.940474+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86397.9,
|
||||
"mean_cadence_s": 5.21,
|
||||
"file": "p1-net-2026-08-24.jsonl.gz",
|
||||
"bytes_gz": 136834,
|
||||
"bytes_raw": 906369,
|
||||
"sha256_raw": "30465d5ff51220201b4a79341e5da7bfc3d88290465b1c9d886ba94eee8dd234",
|
||||
"sha256_gz": "5103088e047bfcbd9b3f55521f431b01cd27de10c2642e00281a13faa762bdd9"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-25",
|
||||
"window": [
|
||||
"2026-08-25T00:00:00+02:00",
|
||||
"2026-08-26T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1110567,
|
||||
"attempts": 1,
|
||||
"samples": 16451,
|
||||
"non_numeric_states": [
|
||||
{
|
||||
"t": "2026-08-25T01:52:02.104225+00:00",
|
||||
"state": "unavailable"
|
||||
}
|
||||
],
|
||||
"first": "2026-08-24T22:00:00+00:00",
|
||||
"last": "2026-08-25T21:59:56.883744+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-25T01:51:25.942719+00:00",
|
||||
"to": "2026-08-25T01:52:53.203963+00:00",
|
||||
"seconds": 87.261
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86396.9,
|
||||
"mean_cadence_s": 5.252,
|
||||
"file": "p1-net-2026-08-25.jsonl.gz",
|
||||
"bytes_gz": 135890,
|
||||
"bytes_raw": 896416,
|
||||
"sha256_raw": "0e7c86b2b74f2108bc7f5f5d2de2627caaafe473ca4ae18515534dff10526e26",
|
||||
"sha256_gz": "2a5411c10d8d3b271f133c62c04bd722279b48b08d7376084fbad6cc6dd66d76"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-26",
|
||||
"window": [
|
||||
"2026-08-26T00:00:00+02:00",
|
||||
"2026-08-27T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1123610,
|
||||
"attempts": 1,
|
||||
"samples": 16542,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-25T22:00:00+00:00",
|
||||
"last": "2026-08-26T21:59:58.915457+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.9,
|
||||
"mean_cadence_s": 5.223,
|
||||
"file": "p1-net-2026-08-26.jsonl.gz",
|
||||
"bytes_gz": 139301,
|
||||
"bytes_raw": 908350,
|
||||
"sha256_raw": "88d72c1b9f51707d62e744f40402de4260a04617875b85c65d2df6079c6c84ca",
|
||||
"sha256_gz": "0661588fbc0c6c42825b67d483c8818c070405c1002424b3f1d61d8fdadb6370"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-27",
|
||||
"window": [
|
||||
"2026-08-27T00:00:00+02:00",
|
||||
"2026-08-28T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1122433,
|
||||
"attempts": 1,
|
||||
"samples": 16573,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-26T22:00:00+00:00",
|
||||
"last": "2026-08-27T21:59:58.538446+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-27T18:50:29.030586+00:00",
|
||||
"to": "2026-08-27T18:52:22.938470+00:00",
|
||||
"seconds": 113.908
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86398.5,
|
||||
"mean_cadence_s": 5.214,
|
||||
"file": "p1-net-2026-08-27.jsonl.gz",
|
||||
"bytes_gz": 138009,
|
||||
"bytes_raw": 906770,
|
||||
"sha256_raw": "528ab8ee10a46275d2e91d56ef2ed847e5932fae76a1752e6f2e9828dbfa9865",
|
||||
"sha256_gz": "5cacc8c5788f43d6e6e2f569445fa82d9edc5a63b1f082f9a1d6a3bb0b404260"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-28",
|
||||
"window": [
|
||||
"2026-08-28T00:00:00+02:00",
|
||||
"2026-08-29T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1114796,
|
||||
"attempts": 1,
|
||||
"samples": 16456,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-27T22:00:00+00:00",
|
||||
"last": "2026-08-28T21:59:58.515555+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.5,
|
||||
"mean_cadence_s": 5.251,
|
||||
"file": "p1-net-2026-08-28.jsonl.gz",
|
||||
"bytes_gz": 135917,
|
||||
"bytes_raw": 900654,
|
||||
"sha256_raw": "f89838bc9f61af074c17817e4a6eab037318f38fa3c65dad3edfa2106abbc9fb",
|
||||
"sha256_gz": "97090b605f67dcc1f84dea2c51d4832dfe1c2af497a4f4cfe50ecb51024d9b4c"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-29",
|
||||
"window": [
|
||||
"2026-08-29T00:00:00+02:00",
|
||||
"2026-08-30T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1092983,
|
||||
"attempts": 1,
|
||||
"samples": 16132,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-28T22:00:00+00:00",
|
||||
"last": "2026-08-29T21:59:57.574524+00:00",
|
||||
"gaps": [
|
||||
{
|
||||
"from": "2026-08-29T00:03:20.539844+00:00",
|
||||
"to": "2026-08-29T00:03:50.569581+00:00",
|
||||
"seconds": 30.03
|
||||
}
|
||||
],
|
||||
"covered_span_s": 86397.6,
|
||||
"mean_cadence_s": 5.356,
|
||||
"file": "p1-net-2026-08-29.jsonl.gz",
|
||||
"bytes_gz": 134210,
|
||||
"bytes_raw": 883053,
|
||||
"sha256_raw": "896d4c9d660be4faabc5a088323e246e257877d5ff6bd0ec084b785480d76e12",
|
||||
"sha256_gz": "b6f264cd3afba63209123a27490e7e1c3ae70a86731d7ea05ae67f88917f136f"
|
||||
},
|
||||
{
|
||||
"date": "2026-08-30",
|
||||
"window": [
|
||||
"2026-08-30T00:00:00+02:00",
|
||||
"2026-08-31T00:00:00+02:00"
|
||||
],
|
||||
"read": "OK",
|
||||
"http_bytes": 1118895,
|
||||
"attempts": 1,
|
||||
"samples": 16490,
|
||||
"non_numeric_states": [],
|
||||
"first": "2026-08-29T22:00:00+00:00",
|
||||
"last": "2026-08-30T21:59:58.533323+00:00",
|
||||
"gaps": [],
|
||||
"covered_span_s": 86398.5,
|
||||
"mean_cadence_s": 5.24,
|
||||
"file": "p1-net-2026-08-30.jsonl.gz",
|
||||
"bytes_gz": 138429,
|
||||
"bytes_raw": 904311,
|
||||
"sha256_raw": "e704818efb58bb6c505791bc9eab2e9476bf48ff8911d0e79b401fd35b529fa0",
|
||||
"sha256_gz": "9052fead2418bff3083bec3ae8ce64d6b9f6224b668f4bb510c79775f66c5b79"
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"days_requested": 30,
|
||||
"days_read": 30,
|
||||
"days_failed": 0,
|
||||
"samples": 485841,
|
||||
"bytes_gz": 4126003,
|
||||
"empty_days": []
|
||||
}
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -20,12 +20,15 @@ WHAT THIS FILE IS DEFENDING, in the order the card's acceptance criteria name it
|
||||
exist - static_tariff's required `peak_guard_active` interlock and
|
||||
arbiter.py's clause 2 - both driven end to end, plus the third state
|
||||
(the guard never ran) that neither of them can express on its own.
|
||||
AC4 NET-POSITIVE ACROSS 12 MONTHS OF REPLAYED PRICES. ⚠️ SEE SECTION 7. THIS
|
||||
REPOSITORY HOLDS NO PRICE HISTORY AT ALL - `scenarios/` carries two days
|
||||
of real METER data and nothing else - so the criterion as written cannot
|
||||
be met, and it is NOT redefined here into something that can be. What
|
||||
section 7 does instead is stated there in full, and what it cannot say is
|
||||
stated with it.
|
||||
AC4 NET-POSITIVE ACROSS 12 MONTHS OF REPLAYED PRICES. ⚠️ MET AS OF DATA-02,
|
||||
AND SECTIONS 7 AND 7B DO DIFFERENT HALVES OF IT. 7B replays twelve months
|
||||
of REAL EPEX BE day-ahead prices - `scenarios/epex-be-dayahead-*.json`,
|
||||
8760 hours, public data under CC BY 4.0 - through the shipped `decide()`,
|
||||
and reports the euro figure at the shipped threshold. Section 7 keeps the
|
||||
SYNTHETIC year, because the real one turns out to be unable to grade §17
|
||||
at all: not one of its 365 days has a negative DEAREST hour, so a count
|
||||
of zero negative-price discharges over it grades nothing. 7B asserts that
|
||||
degeneracy rather than quietly reporting the zero.
|
||||
|
||||
THREE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
|
||||
|
||||
@@ -378,7 +381,11 @@ check("BUYING in a negative-price hour is still permitted - §17 forbids "
|
||||
"refusing to be paid to charge",
|
||||
act(NEG_PLAN, NEG.index(min(NEG)), thresh=_shipped,
|
||||
unc=NO_WEATHER).action == CHARGE)
|
||||
# A zero price is the boundary and it is NOT negative.
|
||||
# A zero price is the boundary and it is NOT negative. ⚠️ KEEP THIS PROBE. It
|
||||
# pins the boundary from ABOVE - it is what catches a guard written `<= 0` - and
|
||||
# the SAFETY-26 block below pins it from BELOW. Neither replaces the other, and
|
||||
# the whole defect that card exists to close is that only one of the two sides
|
||||
# was ever stated.
|
||||
ZERO = list(POS)
|
||||
ZERO[DEAR_H] = 0.0
|
||||
check("a sell hour priced at exactly ZERO is not a negative one, so the guard "
|
||||
@@ -387,6 +394,119 @@ check("a sell hour priced at exactly ZERO is not a negative one, so the guard "
|
||||
act(plan_of(ZERO), DEAR_H, thresh=_shipped, unc=NO_WEATHER).action
|
||||
== DISCHARGE)
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# SAFETY-26. THE OTHER SIDE OF THE SAME BOUNDARY, AND WHY IT HAD TO BE ADDED.
|
||||
#
|
||||
# OPT-04's reviewer mutated `if price < 0` to `if price < -0.001` and EVERY
|
||||
# SUITE STAYED GREEN, with a -0.0005 EUR/kWh sell hour discharging. The shipped
|
||||
# code was right the whole time; the CHECKS could not tell correct from
|
||||
# nearly-correct, because the only probe near the boundary sat at 0.0000. A
|
||||
# probe above the line catches `<=`. It says nothing at all about a tolerance
|
||||
# invented below the line, and a tolerance below the line IS §17's regression
|
||||
# case walking through with the suite passing.
|
||||
#
|
||||
# ⚠️ AND -0.0005 IS NOT AN EXOTIC INPUT. Belgian day-ahead prints small negative
|
||||
# hours regularly; an all-in price a twentieth of a cent below zero is an
|
||||
# ordinary day, not a fuzz value. That is what makes this a live regression
|
||||
# rather than a theoretical one.
|
||||
#
|
||||
# ⚠️ SO THE PROBE IS A SWEEP, NOT A POINT, and the sweep is the discriminating
|
||||
# part. One probe at -0.0005 closes `-0.001` and nothing else; a mutation
|
||||
# reaching for `-0.01` would survive it, which is the identical defect one
|
||||
# decimal place further out. The sweep spans four orders of magnitude down to
|
||||
# the last place the canonical six-decimal format can even express, so there is
|
||||
# no tolerance a guard can invent that some member of it does not straddle.
|
||||
# `_ROUNDING` asserts the smallest member SURVIVES the format rather than being
|
||||
# rounded to zero on the way in - a probe that silently became 0.0 would be the
|
||||
# `0.0000` check again, wearing a different name.
|
||||
TINY_NEG = (-0.05, -0.005, -0.0005, -0.00005, -0.000001)
|
||||
_ROUNDING = [round(p, 6) for p in TINY_NEG]
|
||||
check(f"every probe price survives the canonical 6-decimal rounding as a "
|
||||
f"STRICTLY negative number {_ROUNDING} - one that rounded to 0.0 would "
|
||||
f"be the check above again under another name",
|
||||
all(r < 0 for r in _ROUNDING) and len(set(_ROUNDING)) == len(TINY_NEG))
|
||||
check("...and they span four orders of magnitude, so no single invented "
|
||||
"tolerance can sit below all of them and above none",
|
||||
abs(min(TINY_NEG)) / abs(max(TINY_NEG)) > 10000)
|
||||
|
||||
|
||||
def tiny_neg_plan(price):
|
||||
"""The POS day with its DEAREST hour set to a small negative print."""
|
||||
day = list(POS)
|
||||
day[DEAR_H] = price
|
||||
return plan_of(day), day
|
||||
|
||||
|
||||
# The stimulus has to survive the rungs ABOVE §17 or it grades nothing: the
|
||||
# hour must still be the dearest, and the spread must still clear the widened
|
||||
# threshold, at every probe. Asserted first, for all of them, as its own check.
|
||||
_wellformed = [
|
||||
(p, day.index(max(day)) == DEAR_H,
|
||||
round(max(day) - min(day), 6) > required_spread(_shipped, NO_WEATHER))
|
||||
for p, (_pl, day) in ((p, tiny_neg_plan(p)) for p in TINY_NEG)]
|
||||
check(f"at every probe the negative hour is still the DEAREST one and the "
|
||||
f"spread still clears the threshold, so §17 is the only rung that can "
|
||||
f"refuse {_wellformed}",
|
||||
all(dearest and clears for _p, dearest, clears in _wellformed))
|
||||
_swept = [(p, act(tiny_neg_plan(p)[0], DEAR_H, thresh=_shipped,
|
||||
unc=NO_WEATHER).action) for p in TINY_NEG]
|
||||
check(f"SAFETY-26: a sell hour that is negative BY ANY MARGIN AT ALL refuses - "
|
||||
f"down to -0.000001 EUR/kWh, so the guard is `< 0` and not `< -epsilon` "
|
||||
f"for any epsilon {_swept}",
|
||||
all(a is None for _p, a in _swept))
|
||||
check("...and each refusal is §17's, not some other rung that happens to fire "
|
||||
"at small numbers",
|
||||
all("§17" in act(tiny_neg_plan(p)[0], DEAR_H, thresh=_shipped,
|
||||
unc=NO_WEATHER).reason for p in TINY_NEG))
|
||||
# The mirror image, one place above zero. Without this the sweep above would
|
||||
# pass for a guard that refused EVERYTHING, which grades §17 by nothing.
|
||||
_tiny_pos = [(-p, act(tiny_neg_plan(-p)[0], DEAR_H, thresh=_shipped,
|
||||
unc=NO_WEATHER).action) for p in TINY_NEG]
|
||||
check(f"...while THE SAME MAGNITUDES with the sign flipped all DISCHARGE, so "
|
||||
f"the sweep is a boundary at exactly zero rather than a refusal of small "
|
||||
f"numbers - which a guard that refused everything would also have passed "
|
||||
f"{_tiny_pos}",
|
||||
all(a == DISCHARGE for _p, a in _tiny_pos))
|
||||
|
||||
# SAFETY-26 AC4 - THE LOOK AT THE OTHER BOUNDARIES, NAMED RATHER THAN FIXED.
|
||||
#
|
||||
# The card asks whether the same one-sided pinning exists elsewhere. Every
|
||||
# price/quantity boundary in arbitrage.py and static_tariff.py was given the
|
||||
# tolerance mutation that survived on `price < 0`, in both directions where
|
||||
# both exist, under `-B` with a recursive purge. Two survived, BOTH in
|
||||
# static_tariff.py - TARIFF-03's file, and deliberately NOT touched here:
|
||||
#
|
||||
# D `spread == 0.0` (the flat-tariff rung) -> `spread <= 0.001` SURVIVES.
|
||||
# Measured, not reasoned: with it loosened, a series priced 0.20000 /
|
||||
# 0.20050 - a REAL 0.0005 spread - flips from ALLOWED to REFUSED at
|
||||
# min_spread_eur_kwh=0.0, which `arbitrage()`'s own docstring documents as
|
||||
# a supported value ("pass 0.0 to mean any positive spread, explicitly").
|
||||
# At a shipped threshold the OUTCOME is unchanged, but the operator's
|
||||
# reason becomes the false sentence "the tariff is flat - every one of the
|
||||
# 2 firm hours costs 0.2000 EUR/kWh" about a tariff that is not flat.
|
||||
# Same shape as §17's: the rung is only ever probed with an EXACTLY flat
|
||||
# series, so nothing pins the tiny-but-real side.
|
||||
#
|
||||
# F `len(firm) < 2` (§8.3's firm-pair rung) -> `< 1` SURVIVES, at +0 BYTES,
|
||||
# which is the delta TEST-15 names as the risky one. It is not live on its
|
||||
# own: a single firm hour has a spread of 0.0, so the flat rung refuses it
|
||||
# anyway - but the reason degrades from §8.3's honest "only 1 of 3 hours
|
||||
# carry a published price" to "the tariff is flat". THE FINDING THAT
|
||||
# MATTERS IS THE INTERACTION: D and F MASK EACH OTHER. Neither is
|
||||
# individually live because the other catches it, which is exactly the
|
||||
# arrangement in which a later edit to either one opens a hole nothing
|
||||
# reports.
|
||||
#
|
||||
# Everything else in arbitrage.py is pinned on BOTH sides and was verified so:
|
||||
# `price < 0` (this card), `soc <= floor_pct` and `soc >= ceiling_pct` are each
|
||||
# red under a +5 and a -5 tolerance; static_tariff's `spread < need` is red
|
||||
# under +/-0.001 and additionally bisected to 1e-6 in section 1; `cheap_now`'s
|
||||
# `price < mean` is red under a +0.001 tolerance in test_static_tariff.py.
|
||||
#
|
||||
# Not fixed here on purpose. D and F are TARIFF-03's rungs, they need probes in
|
||||
# test_static_tariff.py rather than in this file, and the card is explicit that
|
||||
# its fourth criterion is a look and not a refactor.
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
@@ -669,34 +789,39 @@ check("...and it is still ARMED when arbitrage is idle and the loop wins, so "
|
||||
print()
|
||||
print("7. AC4 - 12 months of replayed prices, and what could NOT be measured")
|
||||
|
||||
# ⚠️ READ THIS BEFORE READING THE NUMBERS BELOW.
|
||||
# ⚠️ READ THIS BEFORE READING THE NUMBERS BELOW. THIS SECTION IS THE SYNTHETIC
|
||||
# YEAR, AND IT IS NOT AC4's EVIDENCE ANY MORE - SECTION 7B IS.
|
||||
#
|
||||
# The acceptance criterion is "net-positive across 12 months of replayed prices
|
||||
# in simulation". THIS REPOSITORY CONTAINS NO PRICE HISTORY. `scenarios/` holds
|
||||
# three gzipped captures and all three are P1 METER data - two days of net power
|
||||
# from the reference house plus one age capture. There is no EPEX archive, no
|
||||
# supplier export, and no CSV of a past year anywhere in the tree.
|
||||
# When OPT-04 shipped, this repository held no price history: `scenarios/`
|
||||
# carried three gzipped P1 METER captures and nothing else. AC4 was therefore
|
||||
# reported NOT MET rather than redefined into something passable, which is what
|
||||
# TARIFF-01 and SAFETY-07 had both been upheld for doing.
|
||||
#
|
||||
# TARIFF-01 and SAFETY-07 both hit an unmeetable criterion and both DECLINED to
|
||||
# redefine it into one they could pass, and both were upheld for it. So:
|
||||
# DATA-02 pointed out that the missing ingredient was never a capture. EPEX BE
|
||||
# day-ahead is PUBLIC DATA; the 30-day meter criterion it was being carried
|
||||
# beside genuinely needs waiting, and this one only ever needed a fetch. Section
|
||||
# 7B replays the real year and AC4 is met there.
|
||||
#
|
||||
# AC4 IS NOT MET, AND IT IS NOT RESTATED. What follows is not a replay of the
|
||||
# Belgian market; it is a replay of a SYNTHETIC year, and a euro figure from a
|
||||
# synthetic year grades the generator that produced it, not the strategy.
|
||||
# ⚠️ SO WHY IS THE SYNTHETIC YEAR STILL HERE? Because the real one cannot grade
|
||||
# §17, and 7B proves that rather than assuming it. Across all 365 real days, the
|
||||
# number whose DEAREST hour is priced below zero is ZERO - raw wholesale and
|
||||
# all-in alike - so "no discharges into a negative hour" over the real year is a
|
||||
# count of something that never had the chance to happen. Deleting the guard
|
||||
# would not change it. This section's generator drives a summer midday block
|
||||
# negative ON PURPOSE, and it is the only thing in this file that puts a
|
||||
# negative price where the strategy would otherwise sell.
|
||||
#
|
||||
# What the replay is FOR is the half of AC4 that does not need real prices. Net
|
||||
# positivity here is not an empirical outcome to be hoped for - it is a property
|
||||
# of the decision rule, and the replay's job is to demonstrate that the property
|
||||
# holds on every trade the strategy actually chose, over a year of varied days
|
||||
# including negative-price ones. Stated as an invariant:
|
||||
# What this section establishes is the half of AC4 that does not need real
|
||||
# prices, and it is a property of the decision rule rather than an outcome to be
|
||||
# hoped for:
|
||||
#
|
||||
# every executed round trip has (sell - buy) >= required >= threshold,
|
||||
# so its benefit net of the threshold is >= 0, on every day, always.
|
||||
#
|
||||
# That is asserted per trade below, over 365 days. What it does NOT establish -
|
||||
# and no synthetic series could - is how OFTEN a real Belgian year clears the
|
||||
# threshold, which is the actual open question and the reason the shipped
|
||||
# default is a product decision rather than a number this card picked.
|
||||
# That is asserted per trade below, over 365 varied days including negative ones.
|
||||
# HOW OFTEN a real Belgian year clears the threshold is the other question, it
|
||||
# is the one the product owner's open decision turns on, and section 7B is the
|
||||
# only place in this file that answers it.
|
||||
|
||||
def synth_day(n):
|
||||
"""A synthetic all-in day-ahead day. Deterministic, seeded by day number.
|
||||
@@ -803,12 +928,256 @@ check("the shipped threshold is strictly more selective than the loose one on "
|
||||
"the same year - the parameter does what it says, at whatever value the "
|
||||
"owner sets it to",
|
||||
len(_sh_trades) < len(_trades))
|
||||
check("AC4 IS REPORTED AS NOT MET rather than redefined: this file says so in "
|
||||
"section 7's preamble, and there is no price history in the repository "
|
||||
"to meet it with",
|
||||
not any(f.endswith((".csv", ".json.gz"))
|
||||
and "price" in f.lower()
|
||||
for f in os.listdir(os.path.join(HERE, "scenarios"))))
|
||||
check("...and the shipped default was NOT moved to make any of this look "
|
||||
"better: app/arbitrage.py reads the threshold through price_option() and "
|
||||
"names no number of its own",
|
||||
"0.10" not in open(os.path.join(HERE, "app", "arbitrage.py"),
|
||||
encoding="utf-8").read().split('"""', 2)[2])
|
||||
|
||||
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("7B. AC4 - twelve months of REAL EPEX BE prices, replayed")
|
||||
|
||||
# THE DATASET, AND WHAT IT IS NOT.
|
||||
#
|
||||
# `scenarios/epex-be-dayahead-2025-09-01_2026-09-01.json` - 8760 hourly Belgian
|
||||
# day-ahead auction prices, 2025-09-01 to 2026-09-01 Brussels-local, fetched by
|
||||
# `scenarios/fetch_epex_be.py` from Fraunhofer ISE's Energy-Charts API, which
|
||||
# publishes the BE zone under CC BY 4.0 from Bundesnetzagentur | SMARD.de. The
|
||||
# licence was checked BEFORE the data was committed, because the same API serves
|
||||
# other zones under private-use-only terms; the script refuses to write a file
|
||||
# whose per-response `license_info` stops saying CC BY 4.0. Provenance,
|
||||
# attribution and the DST accounting live in the file's own `provenance` block
|
||||
# and `fetch_epex_be.py --verify` re-derives every claim in it.
|
||||
#
|
||||
# ⚠️ THE FIGURES IN THAT FILE ARE RAW WHOLESALE, NOT ALL-IN, AND THIS SECTION IS
|
||||
# WHERE THE DIFFERENCE IS MADE VISIBLE. §9.3's canonical point is all-in; the
|
||||
# committed file says `"all_in": false` and carries the markup/tax assumption as
|
||||
# NUMBERS TO APPLY rather than numbers already applied - TARIFF-02's
|
||||
# double-counting trap, avoided by never baking the guess in. `_all_in()` below
|
||||
# applies it in one place where a reader can watch it happen, and both replays
|
||||
# are run, so exactly how much of the answer rests on the guess is reported
|
||||
# rather than hidden.
|
||||
|
||||
import json # noqa: E402
|
||||
from collections import defaultdict # noqa: E402
|
||||
|
||||
_DS_PATH = os.path.join(HERE, "scenarios",
|
||||
"epex-be-dayahead-2025-09-01_2026-09-01.json")
|
||||
_DS = derived(lambda: json.load(open(_DS_PATH, encoding="utf-8")))
|
||||
_ASSUME = _DS["provenance"]["all_in_assumption"]
|
||||
|
||||
|
||||
def _all_in(wholesale):
|
||||
"""The stated assumption, applied HERE and nowhere in the committed data.
|
||||
|
||||
all_in = (wholesale + markup + network_and_levies) * (1 + vat), with the
|
||||
coefficients read from the dataset's own provenance block rather than
|
||||
retyped - a second copy is a second thing to disagree with the file.
|
||||
"""
|
||||
return round((wholesale + _ASSUME["markup_eur_per_kwh"]
|
||||
+ _ASSUME["network_and_levies_eur_per_kwh"])
|
||||
* (1.0 + _ASSUME["vat_fraction"]), 6)
|
||||
|
||||
|
||||
def _days(price_of):
|
||||
"""The dataset as Brussels calendar days of `PricePoint`s, in order.
|
||||
|
||||
⚠️ GROUPED BY THE LOCAL DATE IN THE `start` STRING, NOT BY SLICING 24 AT A
|
||||
TIME. Two of these days are not 24 hours long, which is the whole DST check,
|
||||
and a fixed stride would put October's 25th hour into the next day and
|
||||
misalign every day after it.
|
||||
"""
|
||||
out = defaultdict(list)
|
||||
for p in _DS["prices"]:
|
||||
out[p["start"][:10]].append(
|
||||
PricePoint(start=datetime.fromisoformat(p["start"]),
|
||||
eur_per_kwh=price_of(p["eur_per_kwh"]),
|
||||
provisional=bool(p["provisional"])))
|
||||
return [out[k] for k in sorted(out)]
|
||||
|
||||
|
||||
_AI_DAYS = derived(lambda: _days(_all_in))
|
||||
_RAW_DAYS = derived(lambda: _days(lambda w: w))
|
||||
|
||||
# ---- the dataset is not degenerate, asserted BEFORE anything is graded ------
|
||||
_lens = [len(d) for d in _AI_DAYS]
|
||||
check(f"the replay has {len(_AI_DAYS)} Brussels days and {sum(_lens)} hourly "
|
||||
f"points, which is a full year rather than a fortnight repeated",
|
||||
len(_AI_DAYS) == 365 and sum(_lens) == 8760)
|
||||
check(f"...and the day lengths are {sorted(set(_lens))}, so BOTH DST days are "
|
||||
f"in it: an importer that produced 24 everywhere would have lost an hour "
|
||||
f"in October and invented one in March",
|
||||
sorted(set(_lens)) == [23, 24, 25]
|
||||
and _lens.count(23) == 1 and _lens.count(25) == 1)
|
||||
_dst_day = [d for d in _AI_DAYS if len(d) == 25][0]
|
||||
check("...and the 25-hour day's two 02:00 points are two different INSTANTS at "
|
||||
"two different offsets, which is what `at()` has to tell apart",
|
||||
len({p.start for p in _dst_day}) == 25
|
||||
and {p.start.utcoffset() for p in _dst_day}
|
||||
== {timedelta(hours=1), timedelta(hours=2)})
|
||||
|
||||
_ai_spreads = sorted(round(max(p.eur_per_kwh for p in d)
|
||||
- min(p.eur_per_kwh for p in d), 6)
|
||||
for d in _AI_DAYS)
|
||||
_raw_spreads = sorted(round(max(p.eur_per_kwh for p in d)
|
||||
- min(p.eur_per_kwh for p in d), 6)
|
||||
for d in _RAW_DAYS)
|
||||
_req_shipped = required_spread(_shipped, NO_WEATHER)
|
||||
_clears = [s for s in _ai_spreads if s > _req_shipped]
|
||||
check(f"...and the year's daily spreads STRADDLE the shipped requirement of "
|
||||
f"{_req_shipped:.4f} EUR/kWh - {len(_clears)} of {len(_ai_spreads)} days "
|
||||
f"clear it - so the threshold is grading something. A year in which every "
|
||||
f"day cleared, or none did, would grade nothing at any value",
|
||||
0 < len(_clears) < len(_ai_spreads))
|
||||
check(f"...and the spreads are not one repeated number: median "
|
||||
f"{_ai_spreads[len(_ai_spreads) // 2]:.4f}, min {_ai_spreads[0]:.4f}, "
|
||||
f"max {_ai_spreads[-1]:.4f}",
|
||||
len(set(_ai_spreads)) > 300 and _ai_spreads[-1] > 4 * _ai_spreads[0])
|
||||
|
||||
# ---- the assumption's arithmetic, asserted rather than asserted-about -------
|
||||
check("the ADDITIVE half of the all-in assumption cancels out of every daily "
|
||||
"spread, so the trade counts below do not depend on the markup guess at "
|
||||
"all - only on the VAT multiplier",
|
||||
all(near(a, r * (1.0 + _ASSUME["vat_fraction"]), 1e-6)
|
||||
for a, r in zip(_ai_spreads, _raw_spreads)))
|
||||
|
||||
# ---- ⚠️ THE DEGENERACY THAT KEEPS SECTION 7 ALIVE --------------------------
|
||||
_raw_neg_h = sum(1 for d in _RAW_DAYS for p in d if p.eur_per_kwh < 0)
|
||||
_ai_neg_h = sum(1 for d in _AI_DAYS for p in d if p.eur_per_kwh < 0)
|
||||
_raw_neg_dear = sum(1 for d in _RAW_DAYS if max(p.eur_per_kwh for p in d) < 0)
|
||||
_ai_neg_dear = sum(1 for d in _AI_DAYS if max(p.eur_per_kwh for p in d) < 0)
|
||||
check(f"the real year DOES contain negative hours - {_raw_neg_h} of them at "
|
||||
f"wholesale, {_ai_neg_h} still negative once the assumed markup and "
|
||||
f"charges are added - so §17 is a live case on this market and not a "
|
||||
f"hypothetical", _raw_neg_h > 100 and _ai_neg_h > 0)
|
||||
check(f"BUT NOT ONE OF THE 365 DAYS HAS A NEGATIVE DEAREST HOUR "
|
||||
f"({_raw_neg_dear} raw, {_ai_neg_dear} all-in), and the guard is on the "
|
||||
f"hour being SOLD INTO. So a count of zero negative-price discharges over "
|
||||
f"this year would be a count of something that could not happen, and "
|
||||
f"section 7's synthetic year stays because it is the only thing here that "
|
||||
f"puts a negative price where the strategy would sell",
|
||||
_raw_neg_dear == 0 and _ai_neg_dear == 0)
|
||||
|
||||
|
||||
def replay_real(days, threshold, unc, *, soc0=50.0, floor=20.0, ceiling=95.0):
|
||||
"""The shipped `decide()` over the real year. Returns the ledger, nothing else.
|
||||
|
||||
One `TariffPlan` per Brussels day, so "the cheapest hour" means the cheapest
|
||||
hour of that day - the same shape as section 7's replay, and the same shape
|
||||
a site with a day-ahead feed actually sees.
|
||||
"""
|
||||
soc = soc0
|
||||
trades, holdings, bad, neg_sells = [], [], [], 0
|
||||
for day in days:
|
||||
plan = TariffPlan(
|
||||
PriceRead.series(day, PriceProduct.DAY_AHEAD, "generic_dayahead",
|
||||
day[0].start), None, "provider")
|
||||
open_buy = None
|
||||
for p in day:
|
||||
d = decide(plan, p.start + timedelta(minutes=30),
|
||||
peak_guard_active=False, soc=soc,
|
||||
min_spread_eur_kwh=threshold, uncertainty=unc,
|
||||
charge_w=MAX_W, discharge_w=MAX_W,
|
||||
soc_floor_pct=floor, soc_ceiling_pct=ceiling)
|
||||
if d.action is None:
|
||||
continue
|
||||
if d.spread_eur_kwh is None or d.required_eur_kwh is None:
|
||||
bad.append(p.start)
|
||||
if d.action == CHARGE:
|
||||
soc = min(ceiling, soc + STEP_PCT)
|
||||
open_buy = (p.eur_per_kwh, d)
|
||||
else:
|
||||
if p.eur_per_kwh < 0:
|
||||
neg_sells += 1
|
||||
soc = max(floor, soc - STEP_PCT)
|
||||
if open_buy is not None:
|
||||
trades.append((open_buy[0], p.eur_per_kwh, d))
|
||||
open_buy = None
|
||||
else:
|
||||
holdings.append(p.start)
|
||||
return trades, holdings, bad, neg_sells
|
||||
|
||||
|
||||
# ⚠️ AT THE SHIPPED THRESHOLD, UNTOUCHED, AND WIDENED BY FCST-03's NO-WEATHER
|
||||
# FACTOR - the harsher of the two operating states, so the figure reported is
|
||||
# the conservative one.
|
||||
_r_trades, _r_held, _r_bad, _r_neg = derived(
|
||||
lambda: replay_real(_AI_DAYS, _shipped, NO_WEATHER)) or (None,) * 4
|
||||
_r_kwh = MAX_W / 1000.0
|
||||
_r_net = round(sum((s - b - _shipped) * _r_kwh for b, s, _ in _r_trades), 2)
|
||||
|
||||
check(f"the real year traded at the SHIPPED threshold: {len(_r_trades)} "
|
||||
f"completed round trips over 365 days of Belgian day-ahead prices - so "
|
||||
f"the criterion is being graded on trades that happened, not on an empty "
|
||||
f"ledger that is net-positive by vacuity", len(_r_trades) > 0)
|
||||
check("...and every one of them carries the spread and the widened requirement "
|
||||
"it cleared, so the ledger is auditable rather than a count", _r_bad == [])
|
||||
check(f"...and it did NOT trade every day ({len(_r_trades)} round trips over "
|
||||
f"365 days), so the threshold selected rather than waved everything "
|
||||
f"through", len(_r_trades) < 365)
|
||||
_r_losses = [(b, s) for b, s, _ in _r_trades if s - b < _shipped]
|
||||
check(f"AC4: every completed round trip on the REAL year cleared the shipped "
|
||||
f"{_shipped} EUR/kWh - {len(_r_losses)} exceptions of {len(_r_trades)}",
|
||||
_r_losses == [])
|
||||
_r_under = [d.spread_eur_kwh for _b, _s, d in _r_trades
|
||||
if d.spread_eur_kwh < d.required_eur_kwh]
|
||||
check(f"...and each cleared its own WIDENED requirement of "
|
||||
f"{_req_shipped:.4f} EUR/kWh, which is the stricter statement §8.3 asks "
|
||||
f"for", _r_under == [])
|
||||
check(f"AC4 IS MET: twelve months of REAL replayed EPEX BE prices are "
|
||||
f"NET-POSITIVE at the shipped default - {_r_net:+.2f} EUR of benefit net "
|
||||
f"of the cycle cost over {len(_r_trades)} round trips of {_r_kwh:.1f} kWh "
|
||||
f"each, accumulated from the trades the strategy actually chose",
|
||||
_r_net > 0)
|
||||
|
||||
# ---- what the owner's open decision actually turns on -----------------------
|
||||
# ⚠️ REPORTED, NOT SETTLED. `cycle_cost_threshold_eur` and FCST-03's 2.5/1.5 are
|
||||
# the product owner's decision. DATA-02 supplies the input and changes nothing:
|
||||
# every number below comes out of the same untouched `decide()`.
|
||||
_r_calm = derived(lambda: replay_real(_AI_DAYS, _shipped, frozenset())[0])
|
||||
_r_raw = derived(lambda: replay_real(_RAW_DAYS, _shipped, NO_WEATHER)[0])
|
||||
# The threshold at which the MEDIAN day of this year clears its widened
|
||||
# requirement. Derived from the observed spreads, not chosen.
|
||||
_median_spread = _ai_spreads[len(_ai_spreads) // 2]
|
||||
_suggested = round(_median_spread / widen_factor(NO_WEATHER), 4)
|
||||
_r_sugg, _, _, _ = derived(
|
||||
lambda: replay_real(_AI_DAYS, _suggested, NO_WEATHER)) or (None,) * 4
|
||||
_sugg_net = round(sum((s - b - _suggested) * _r_kwh for b, s, _ in _r_sugg), 2)
|
||||
print(f" [report] SHIPPED {_shipped} EUR/kWh, widened {widen_factor(NO_WEATHER)}x "
|
||||
f"to {_req_shipped:.4f}: {len(_r_trades)} round trips, {_r_net:+.2f} EUR "
|
||||
f"over the year. Unwidened (weather present): {len(_r_calm)} trips. On "
|
||||
f"RAW wholesale instead of the assumed all-in: {len(_r_raw)} trips, so "
|
||||
f"the markup/VAT guess moves the count by "
|
||||
f"{len(_r_trades) - len(_r_raw):+d}.")
|
||||
print(f" [report] the year's MEDIAN daily all-in spread is "
|
||||
f"{_median_spread:.4f} EUR/kWh, so the threshold at which a median day "
|
||||
f"clears its widened requirement is {_suggested} - and at that value the "
|
||||
f"same year completes {len(_r_sugg)} round trips for {_sugg_net:+.2f} "
|
||||
f"EUR. THE SHIPPED DEFAULT IS NOT CHANGED BY THIS CARD; both numbers are "
|
||||
f"reported so the owner can decide between them.")
|
||||
print(f" [report] and the four-developer finding stands unchanged: a STATIC "
|
||||
f"day/night spread of ~0.0840 EUR/kWh still does not clear "
|
||||
f"{_req_shipped:.4f}, so a static-tariff site never trades at the shipped "
|
||||
f"default. What this year adds is that a DYNAMIC site does - on "
|
||||
f"{len(_clears)} of its 365 days the spread was there to be taken.")
|
||||
check("the loose threshold is strictly less selective than the shipped one on "
|
||||
"the SAME real year - the parameter does what it says at whatever value "
|
||||
"the owner sets, and neither run tuned anything",
|
||||
len(_r_sugg) > len(_r_trades) > 0)
|
||||
check(f"§17 held over the real year too ({_r_neg} discharges into a negative "
|
||||
f"hour) - but see the degeneracy check above: on this data that zero is "
|
||||
f"structural, and section 7's synthetic year is what actually grades the "
|
||||
f"guard", _r_neg == 0)
|
||||
check("the dataset carries its own provenance, states it is NOT all-in, and "
|
||||
"keeps the markup/tax figures as an assumption rather than applying "
|
||||
"them - the TARIFF-02 trap, in data form",
|
||||
_DS["provenance"]["all_in"] is False
|
||||
and "ASSUMPTION" in _ASSUME["status"]
|
||||
and _DS["provenance"]["licence"].startswith("CC BY 4.0"))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
@@ -810,9 +810,16 @@ print()
|
||||
print(f" BASELINE {totals['u_under']}/{totals['u_q']} = {U_PCT:.1f} % of "
|
||||
f"quarters under {CAP_W:.0f} W, uncontrolled")
|
||||
print(f" GUARDED {totals['g_under']}/{totals['g_q']} = {G_PCT:.1f} %")
|
||||
print(f" Two days, not thirty. AC 3 is TARIFF-01's and is still not met - "
|
||||
f"see test_tariff.py part 9. What this compares is the same quarters with "
|
||||
f"and without the guard, which is what two days can support.")
|
||||
print(f" Two days, not thirty - by choice: the thirty exist now "
|
||||
f"(scenarios/p1-net-30d-2026-08/, DATA-01) and live one directory down so "
|
||||
f"this glob does not pick them up and put ~50 s of replay on every PR. "
|
||||
f"tools/replay_ac3.py runs the month and must reproduce the totals above "
|
||||
f"to be believed: `python3 tools/replay_ac3.py scenarios --expect "
|
||||
f"{totals['u_under']}/{totals['u_q']}`. Over the thirty days the guard "
|
||||
f"takes August's peak from 5029.8 W to 4107.3 W - still over the "
|
||||
f"{CAP_W:.0f} W soft limit - and moves the interval count from 98.89 % to "
|
||||
f"98.99 %, because once a month peak is set the ceiling it defends is "
|
||||
f"that peak and not the cap. See that directory's README.")
|
||||
check("the guard improves on the uncontrolled baseline rather than merely "
|
||||
"matching it", G_PCT > U_PCT)
|
||||
check("...and it does not degrade a single quarter anywhere in the two days",
|
||||
|
||||
@@ -35,16 +35,39 @@ NOT allowed to be tested at:
|
||||
10. THE WIRING, through the real `Controller` and its real status page.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
⚠️ AC 3 IS NOT MET, AND PART 9 DOES NOT PRETEND OTHERWISE.
|
||||
⚠️ PART 9 IS A TWO-DAY MEASUREMENT AND SAYS SO. THE 30-DAY ANSWER IS ELSEWHERE
|
||||
(DATA-01), AND IT DOES NOT LAND WHERE ANYONE EXPECTED.
|
||||
|
||||
AC 3 asks for "the monthly peak held under the soft limit in >= 95 % of
|
||||
intervals over a 30-day simulation replay". THERE IS NO 30-DAY DATASET. The
|
||||
infra repo's `sim/scenarios/` holds exactly two days of this house's real meter
|
||||
- 2026-08-20 (the Illusmart failure day) and 2026-08-23 (the healthy reference)
|
||||
- exported on 2026-08-25 because they were about to age out of HA's recorder,
|
||||
and nothing longer was ever captured. `sim/scenarios/README.md` lists the whole
|
||||
inventory; the rest is RS485 bus frames, OTA witness logs and a 578-row rollback
|
||||
poll, none of which carries grid power over calendar time.
|
||||
intervals over a 30-day simulation replay". When this file was written there
|
||||
were two days of this house's real meter - 2026-08-20 (the Illusmart failure
|
||||
day) and 2026-08-23 (the healthy reference) - exported on 2026-08-25 because
|
||||
they were about to age out of HA's recorder.
|
||||
|
||||
⚠️ AND THE SENTENCE THAT USED TO STAND HERE, "nothing longer was ever
|
||||
captured", WAS TRUE ABOUT THE PAST AND WAS READ AS A CLAIM ABOUT THE FUTURE.
|
||||
It was not one. HA's recorder still held the whole month; DATA-01 probed it on
|
||||
2026-08-31 (230 samples in a 20-minute window at 30 days back, 0 at 45) and
|
||||
exported thirty days the same day, into `scenarios/p1-net-30d-2026-08/` with
|
||||
`tools/export_p1_history.py` - which is committed now precisely because the
|
||||
2026-08-25 one-off was not, and its absence is what let the belief survive.
|
||||
|
||||
Over those thirty days the criterion measures 98.89 % UNCONTROLLED and 98.99 %
|
||||
guarded, so it is met - and that is a finding about the CRITERION as much as
|
||||
about the guard, because a month with no controller at all passes it. Read the
|
||||
other way - the monthly PEAK itself under the soft limit - it is not met: the
|
||||
guard takes August's peak from 5029.8 W to 4107.3 W, still over 3500 W. Both
|
||||
numbers, and why they part company, are in
|
||||
`scenarios/p1-net-30d-2026-08/README.md`.
|
||||
|
||||
PART 9 IS DELIBERATELY LEFT GRADING THE TWO COMMITTED DAYS. It is a per-PR
|
||||
regression check and the thirty days cost about fifty seconds of replay; they
|
||||
also live one directory down precisely so this file's glob does not pick them
|
||||
up. `tools/replay_ac3.py` is the thing that runs the month, and it is checked
|
||||
against part 6 of test_peakguard.py rather than trusted:
|
||||
|
||||
python3 tools/replay_ac3.py scenarios --expect 189/190
|
||||
python3 tools/replay_ac3.py scenarios/p1-net-30d-2026-08
|
||||
|
||||
So part 9 replays the two days that exist, reports the percentage over them, and
|
||||
states the sample size in days next to it. It is a MEASUREMENT, not a verdict:
|
||||
@@ -60,8 +83,8 @@ export) for the reason `scenarios/p1-age-2026-08-26.jsonl.gz` already is: the
|
||||
infra repo is private, CI's token is minted for this repository only, and a
|
||||
regression fixture that CI cannot see is not a regression fixture.
|
||||
|
||||
Closing AC 3 needs 30 days of capture. That is TEST-01's replay harness, and it
|
||||
cannot start before somebody starts recording.
|
||||
The recorder's window rolls: 2026-08-01 leaves it around 2026-09-15, which is
|
||||
why the committed copy is the artefact and not the API call.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
@@ -748,9 +771,11 @@ print()
|
||||
print(f" AC 3 MEASURED: {under}/{total_q} quarters = {PCT:.1f} % under the "
|
||||
f"{SHIPPED_CAP_W:.0f} W soft limit, over {replay_days:.1f} DAYS of "
|
||||
f"replay, UNCONTROLLED (no peak shaving was running on either day).")
|
||||
print(f" AC 3 ASKS FOR 30 DAYS. {replay_days:.1f} days exist. "
|
||||
f"AC 3 IS NOT MET and this run does not claim it is - see the module "
|
||||
f"docstring. The figure above is the BASELINE SAFETY-07 has to improve.")
|
||||
print(f" AC 3 ASKS FOR 30 DAYS and this run is {replay_days:.1f}, so it "
|
||||
f"does not settle it either way. The 30 days DO now exist - "
|
||||
f"scenarios/p1-net-30d-2026-08/, DATA-01 - and "
|
||||
f"tools/replay_ac3.py is what runs them; the figure above is the "
|
||||
f"BASELINE SAFETY-07 has to improve, on the two days CI replays.")
|
||||
check("the replay produced a measurable number over a stated sample size, "
|
||||
"rather than a criterion quietly redefined to fit the data",
|
||||
total_q > 100 and replay_days < 30)
|
||||
|
||||
@@ -0,0 +1,455 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export a P1 power signal out of Home Assistant's recorder into the committed
|
||||
scenario format - one gzipped JSON object per line, one file per LOCAL day.
|
||||
|
||||
WHY THIS FILE EXISTS AT ALL
|
||||
===========================
|
||||
`scenarios/p1-net-2026-08-20.jsonl.gz` and `p1-net-2026-08-23.jsonl.gz` were
|
||||
produced once, by hand, and the script was thrown away. The consequence was not
|
||||
theoretical: TARIFF-01's and SAFETY-07's AC 3 ("monthly peak held under the soft
|
||||
limit in >= 95 % of intervals over a 30-day simulation replay") were both merged
|
||||
UNMET on the belief that thirty days could not be produced retroactively - a
|
||||
belief that was simply wrong, and that survived because nobody could re-run the
|
||||
export to find out. So the export is a committed, runnable artefact now, and the
|
||||
next person does not have to re-derive it under time pressure while a retention
|
||||
window rolls.
|
||||
|
||||
python3 tools/export_p1_history.py --days 30 --out-dir /some/where
|
||||
|
||||
THE TWO RULES THIS FILE IS BUILT AROUND
|
||||
=======================================
|
||||
1. A FAILED READ IS NOT AN OBSERVATION. An HTTP error, a timeout, a truncated
|
||||
body, a non-JSON body - none of those are "the meter reported nothing that
|
||||
hour". They are "we do not know what the meter reported that hour", and the
|
||||
two must never end up looking the same in the output. So:
|
||||
|
||||
* every request is retried with backoff;
|
||||
* a day whose read ultimately fails writes NO FILE and is recorded in the
|
||||
manifest as `"read": "FAILED"` with the error text, and the process exits
|
||||
non-zero;
|
||||
* a day that reads back successfully but EMPTY writes an empty file and is
|
||||
recorded as `"read": "OK", "samples": 0`. Empty-and-known and
|
||||
unknown are therefore distinguishable both in the manifest and by the
|
||||
presence or absence of the file;
|
||||
* before any day is fetched, the entity is proved to exist through
|
||||
/api/states/<entity>. Without that check a typo in --entity would return
|
||||
200 + `[]` for all thirty days and produce a beautifully consistent
|
||||
dataset of nothing.
|
||||
|
||||
2. GAPS ARE VISIBLE, NEVER INTERPOLATED. Nothing here carries a value forward,
|
||||
fills, resamples or averages. A recorder outage or an HA restart shows up as
|
||||
what it is: consecutive `t` values further apart than the meter's cadence.
|
||||
The controller's own ingest already reads that correctly - `meter_max_age_s`
|
||||
(30 s shipped) is exactly the age past which a held reading stops counting as
|
||||
coverage - so an hour with no telegrams replays as an hour of NO COVERAGE
|
||||
rather than as an hour of whatever the last sample said. The manifest lists
|
||||
every gap over --gap-s so a reader does not have to diff timestamps to find
|
||||
them.
|
||||
|
||||
DAY AT A TIME, AND WHY NOT SMALLER
|
||||
==================================
|
||||
One request per local day. That is gentle on a live HA (~1 MB of JSON per day)
|
||||
and it is also the only chunking that reproduces the committed files: the
|
||||
history API prepends the state as it stood AT the period start
|
||||
(`include_start_time_state`), stamped with the start instant. Splitting a day
|
||||
into six-hour chunks would therefore inject three synthetic boundary samples per
|
||||
day that no meter ever sent. Days are the natural boundary because the committed
|
||||
files are local days.
|
||||
|
||||
VERIFYING THE EXPORTER
|
||||
======================
|
||||
python3 tools/export_p1_history.py --verify scenarios/p1-net-2026-08-20.jsonl.gz
|
||||
|
||||
re-fetches that day by this exact route and compares the DECOMPRESSED bytes with
|
||||
the committed file. Byte equality of the .gz containers is not expected and not
|
||||
required: a gzip header carries an mtime, and this file writes mtime 0 so its
|
||||
own output is reproducible. Content equality is the claim that matters.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
try: # py3.9+; the add-on image has it
|
||||
from zoneinfo import ZoneInfo
|
||||
except ImportError: # pragma: no cover
|
||||
ZoneInfo = None # type: ignore
|
||||
|
||||
DEFAULT_URL = "http://192.168.2.158:8123"
|
||||
DEFAULT_ENTITY = "sensor.p1_meter_active_power"
|
||||
DEFAULT_TZ = "Europe/Brussels"
|
||||
# The shipped meter_max_age_s. A hole longer than this is a hole the controller
|
||||
# itself would refuse to see through, which makes it the honest gap threshold.
|
||||
DEFAULT_GAP_S = 30.0
|
||||
|
||||
|
||||
class ReadFailed(RuntimeError):
|
||||
"""The read did not happen. NOT the same thing as an empty result."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# transport
|
||||
|
||||
|
||||
def _get(url: str, token: str, timeout: float):
|
||||
req = urllib.request.Request(url, headers={
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Content-Type": "application/json",
|
||||
})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
if resp.status != 200:
|
||||
raise ReadFailed(f"HTTP {resp.status}")
|
||||
return resp.read()
|
||||
|
||||
|
||||
def fetch_json(url: str, token: str, *, timeout: float, retries: int,
|
||||
backoff: float, log=print):
|
||||
"""GET and parse, or raise ReadFailed. Never returns a partial answer."""
|
||||
last = None
|
||||
for attempt in range(1, retries + 1):
|
||||
try:
|
||||
raw = _get(url, token, timeout)
|
||||
except urllib.error.HTTPError as exc:
|
||||
last = f"HTTP {exc.code} {exc.reason}"
|
||||
except Exception as exc: # URLError, timeout, ...
|
||||
last = f"{type(exc).__name__}: {exc}"
|
||||
else:
|
||||
try:
|
||||
return json.loads(raw.decode("utf-8")), len(raw), attempt
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
# A truncated body looks exactly like this. It is a failure.
|
||||
last = f"unparseable body ({len(raw)} bytes): {exc}"
|
||||
if attempt < retries:
|
||||
wait = backoff * attempt
|
||||
log(f" read failed ({last}); retry {attempt + 1}/{retries} "
|
||||
f"in {wait:.0f} s")
|
||||
time.sleep(wait)
|
||||
raise ReadFailed(last or "unknown")
|
||||
|
||||
|
||||
def assert_entity_exists(base: str, entity: str, token: str, *, timeout: float,
|
||||
retries: int, backoff: float, log=print):
|
||||
"""Prove the entity is real BEFORE trusting any empty day.
|
||||
|
||||
Without this, `--entity sensor.typo` answers 200 + [] for every day and the
|
||||
exporter cheerfully writes thirty empty files.
|
||||
"""
|
||||
url = f"{base}/api/states/{urllib.parse.quote(entity)}"
|
||||
doc, _, _ = fetch_json(url, token, timeout=timeout, retries=retries,
|
||||
backoff=backoff, log=log)
|
||||
if not isinstance(doc, dict) or doc.get("entity_id") != entity:
|
||||
raise ReadFailed(f"{entity} is not a known entity")
|
||||
return doc
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# one day
|
||||
|
||||
|
||||
def day_bounds(day: datetime, tz):
|
||||
start = datetime(day.year, day.month, day.day, tzinfo=tz)
|
||||
return start, start + timedelta(days=1)
|
||||
|
||||
|
||||
def history_url(base: str, entity: str, start: datetime, end: datetime) -> str:
|
||||
q = urllib.parse.urlencode({
|
||||
"filter_entity_id": entity,
|
||||
"end_time": end.isoformat(),
|
||||
"minimal_response": "",
|
||||
})
|
||||
return (f"{base}/api/history/period/"
|
||||
f"{urllib.parse.quote(start.isoformat())}?{q}")
|
||||
|
||||
|
||||
def rows_from(doc, entity: str):
|
||||
"""HA's shape -> [(iso_t, state_str)], plus the states that are not numbers.
|
||||
|
||||
`minimal_response` returns the first record in full and the rest as
|
||||
{state, last_changed}. An entity that went unavailable reports the string
|
||||
"unavailable" (or "unknown") where a number belongs. Those are dropped from
|
||||
the numeric series - there is no float to write and the committed schema has
|
||||
no slot for a non-number - but every one is returned here so it lands in the
|
||||
manifest. Dropping them is what MAKES the outage visible: the series simply
|
||||
has no samples across it, which is the same thing the meter falling silent
|
||||
looks like, and is what the controller's freshness rule already handles.
|
||||
"""
|
||||
if not isinstance(doc, list):
|
||||
raise ReadFailed(f"expected a list, got {type(doc).__name__}")
|
||||
if not doc:
|
||||
return [], []
|
||||
series = doc[0]
|
||||
if not isinstance(series, list):
|
||||
raise ReadFailed("history series is not a list")
|
||||
good, bad = [], []
|
||||
for rec in series:
|
||||
eid = rec.get("entity_id")
|
||||
if eid is not None and eid != entity:
|
||||
raise ReadFailed(f"unexpected entity {eid!r} in the response")
|
||||
t = rec.get("last_changed") or rec.get("last_updated")
|
||||
s = rec.get("state")
|
||||
if t is None:
|
||||
raise ReadFailed("a record carries no timestamp")
|
||||
try:
|
||||
w = float(s)
|
||||
except (TypeError, ValueError):
|
||||
bad.append({"t": t, "state": s})
|
||||
continue
|
||||
good.append((t, w))
|
||||
return good, bad
|
||||
|
||||
|
||||
def render(rows, *, newline: str) -> bytes:
|
||||
"""The committed format, exactly: `{"t": <iso>, "w": <float>}` per line.
|
||||
|
||||
json.dumps' default separators (", " / ": ") and repr of a float are what
|
||||
the two committed files contain; so is the CRLF, an artefact of the original
|
||||
one-off having been run on Windows in text mode. It is reproduced rather
|
||||
than tidied because reproducing the committed bytes is this exporter's
|
||||
verification, and gzip.open(..., 'rt') strips it either way.
|
||||
"""
|
||||
out = bytearray()
|
||||
for t, w in rows:
|
||||
out += json.dumps({"t": t, "w": w}).encode("utf-8")
|
||||
out += newline.encode("ascii")
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def gz(payload: bytes) -> bytes:
|
||||
"""gzip level 9, mtime 0 - so the same input always gives the same bytes."""
|
||||
import io
|
||||
buf = io.BytesIO()
|
||||
with gzip.GzipFile(fileobj=buf, mode="wb", compresslevel=9, mtime=0) as f:
|
||||
f.write(payload)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
def gaps_in(rows, gap_s: float):
|
||||
out = []
|
||||
prev = None
|
||||
for t, _ in rows:
|
||||
cur = datetime.fromisoformat(t)
|
||||
if prev is not None:
|
||||
d = (cur - prev).total_seconds()
|
||||
if d > gap_s:
|
||||
out.append({"from": prev.isoformat(), "to": cur.isoformat(),
|
||||
"seconds": round(d, 3)})
|
||||
prev = cur
|
||||
return out
|
||||
|
||||
|
||||
def export_day(base, entity, token, day, tz, *, timeout, retries, backoff,
|
||||
gap_s, newline, log=print):
|
||||
"""Read one local day. Raises ReadFailed rather than returning a guess."""
|
||||
start, end = day_bounds(day, tz)
|
||||
doc, nbytes, attempts = fetch_json(
|
||||
history_url(base, entity, start, end), token,
|
||||
timeout=timeout, retries=retries, backoff=backoff, log=log)
|
||||
rows, bad = rows_from(doc, entity)
|
||||
entry = {
|
||||
"date": day.date().isoformat(),
|
||||
"window": [start.isoformat(), end.isoformat()],
|
||||
"read": "OK",
|
||||
"http_bytes": nbytes,
|
||||
"attempts": attempts,
|
||||
"samples": len(rows),
|
||||
"non_numeric_states": bad,
|
||||
"first": rows[0][0] if rows else None,
|
||||
"last": rows[-1][0] if rows else None,
|
||||
"gaps": gaps_in(rows, gap_s),
|
||||
}
|
||||
if rows:
|
||||
span = (datetime.fromisoformat(rows[-1][0])
|
||||
- datetime.fromisoformat(rows[0][0])).total_seconds()
|
||||
entry["covered_span_s"] = round(span, 1)
|
||||
entry["mean_cadence_s"] = round(span / max(1, len(rows) - 1), 3)
|
||||
return rows, entry
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# modes
|
||||
|
||||
|
||||
def do_export(args, token, tz, log=print):
|
||||
base = args.url.rstrip("/")
|
||||
out_dir = Path(args.out_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
newline = "\r\n" if args.newline == "crlf" else "\n"
|
||||
|
||||
log(f" proving {args.entity} exists before trusting any empty day")
|
||||
st = assert_entity_exists(base, args.entity, token, timeout=args.timeout,
|
||||
retries=args.retries, backoff=args.backoff,
|
||||
log=log)
|
||||
log(f" {args.entity} = {st.get('state')} "
|
||||
f"{st.get('attributes', {}).get('unit_of_measurement', '')}")
|
||||
|
||||
if args.end_date:
|
||||
last = datetime.fromisoformat(args.end_date).replace(tzinfo=tz)
|
||||
else: # yesterday: today is still being logged
|
||||
last = datetime.now(tz).replace(hour=0, minute=0, second=0,
|
||||
microsecond=0) - timedelta(days=1)
|
||||
days = [last - timedelta(days=i) for i in range(args.days - 1, -1, -1)]
|
||||
|
||||
manifest = {
|
||||
"generated": datetime.now(timezone.utc).isoformat(),
|
||||
"generator": "tools/export_p1_history.py",
|
||||
"source": f"{base}/api/history/period",
|
||||
"entity": args.entity,
|
||||
"timezone": args.tz,
|
||||
"gap_threshold_s": args.gap_s,
|
||||
"format": '{"t": <ISO-8601 UTC>, "w": <float watts>} per line, gzip',
|
||||
"days": [],
|
||||
}
|
||||
failures = []
|
||||
for day in days:
|
||||
name = f"{args.prefix}{day.date().isoformat()}.jsonl.gz"
|
||||
log(f" {name}")
|
||||
try:
|
||||
rows, entry = export_day(
|
||||
base, args.entity, token, day, tz, timeout=args.timeout,
|
||||
retries=args.retries, backoff=args.backoff, gap_s=args.gap_s,
|
||||
newline=newline, log=log)
|
||||
except ReadFailed as exc:
|
||||
# NO FILE. A missing file is the honest record of a read that did
|
||||
# not happen; an empty one would claim the meter was silent.
|
||||
entry = {"date": day.date().isoformat(), "read": "FAILED",
|
||||
"error": str(exc), "file": None}
|
||||
manifest["days"].append(entry)
|
||||
failures.append(entry)
|
||||
log(f" READ FAILED: {exc} - no file written")
|
||||
continue
|
||||
payload = render(rows, newline=newline)
|
||||
blob = gz(payload)
|
||||
(out_dir / name).write_bytes(blob)
|
||||
entry["file"] = name
|
||||
entry["bytes_gz"] = len(blob)
|
||||
entry["bytes_raw"] = len(payload)
|
||||
entry["sha256_raw"] = hashlib.sha256(payload).hexdigest()
|
||||
entry["sha256_gz"] = hashlib.sha256(blob).hexdigest()
|
||||
manifest["days"].append(entry)
|
||||
ng = len(entry.get("gaps", []))
|
||||
log(f" {entry['samples']} samples, {len(blob)} B gz, "
|
||||
f"{ng} gap(s) over {args.gap_s:g} s"
|
||||
+ (f", {len(entry['non_numeric_states'])} non-numeric state(s)"
|
||||
if entry["non_numeric_states"] else ""))
|
||||
if day is not days[-1]:
|
||||
time.sleep(args.delay)
|
||||
|
||||
ok = [d for d in manifest["days"] if d["read"] == "OK"]
|
||||
manifest["summary"] = {
|
||||
"days_requested": len(days),
|
||||
"days_read": len(ok),
|
||||
"days_failed": len(failures),
|
||||
"samples": sum(d["samples"] for d in ok),
|
||||
"bytes_gz": sum(d["bytes_gz"] for d in ok),
|
||||
"empty_days": [d["date"] for d in ok if d["samples"] == 0],
|
||||
}
|
||||
# newline="\n" explicitly: write_text() on Windows emits CRLF, and
|
||||
# the manifest would then differ from a Linux run of the same export
|
||||
# for no reason visible in a diff.
|
||||
with open(args.manifest or (out_dir / "manifest.json"), "w",
|
||||
encoding="utf-8", newline="\n") as fh:
|
||||
fh.write(json.dumps(manifest, indent=2) + "\n")
|
||||
log(f" {manifest['summary']['days_read']}/{len(days)} days, "
|
||||
f"{manifest['summary']['samples']} samples, "
|
||||
f"{manifest['summary']['bytes_gz'] / 1e6:.2f} MB gz")
|
||||
if failures:
|
||||
log(f" {len(failures)} DAY(S) NOT READ - the dataset is incomplete "
|
||||
f"and says so: " + ", ".join(d["date"] for d in failures))
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
def do_verify(args, token, tz, log=print):
|
||||
"""Re-export a committed day by this route and diff it against the file."""
|
||||
base = args.url.rstrip("/")
|
||||
newline = "\r\n" if args.newline == "crlf" else "\n"
|
||||
rc = 0
|
||||
for target in args.verify:
|
||||
path = Path(target)
|
||||
stamp = path.name.split(".")[0].rsplit("-", 3)[-3:]
|
||||
day = datetime(int(stamp[0]), int(stamp[1]), int(stamp[2]), tzinfo=tz)
|
||||
log(f" {path.name} (local day {day.date().isoformat()})")
|
||||
committed = gzip.decompress(path.read_bytes())
|
||||
rows, entry = export_day(base, args.entity, token, day, tz,
|
||||
timeout=args.timeout, retries=args.retries,
|
||||
backoff=args.backoff, gap_s=args.gap_s,
|
||||
newline=newline, log=log)
|
||||
fresh = render(rows, newline=newline)
|
||||
if fresh == committed:
|
||||
log(f" IDENTICAL: {len(fresh)} bytes, {entry['samples']} "
|
||||
f"samples, sha256 {hashlib.sha256(fresh).hexdigest()[:16]}")
|
||||
continue
|
||||
rc = 1
|
||||
cl = committed.split(newline.encode())
|
||||
fl = fresh.split(newline.encode())
|
||||
log(f" DIFFERS: committed {len(committed)} B / {len(cl)} lines, "
|
||||
f"re-export {len(fresh)} B / {len(fl)} lines")
|
||||
shown = 0
|
||||
for i in range(max(len(cl), len(fl))):
|
||||
a = cl[i] if i < len(cl) else b"<absent>"
|
||||
b = fl[i] if i < len(fl) else b"<absent>"
|
||||
if a != b:
|
||||
log(f" line {i}: committed {a!r}")
|
||||
log(f" re-export {b!r}")
|
||||
shown += 1
|
||||
if shown >= args.max_diff:
|
||||
log(f" ... stopping after {shown} differing lines")
|
||||
break
|
||||
return rc
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
p = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||
p.add_argument("--url", default=DEFAULT_URL)
|
||||
p.add_argument("--entity", default=DEFAULT_ENTITY)
|
||||
p.add_argument("--token", help="bearer token; else $HASS_TOKEN")
|
||||
p.add_argument("--tz", default=DEFAULT_TZ,
|
||||
help="local timezone that defines a day boundary")
|
||||
p.add_argument("--days", type=int, default=30)
|
||||
p.add_argument("--end-date", help="last local day to export (YYYY-MM-DD); "
|
||||
"default yesterday")
|
||||
p.add_argument("--out-dir", default=".")
|
||||
p.add_argument("--prefix", default="p1-net-")
|
||||
p.add_argument("--manifest", help="path for the manifest JSON")
|
||||
p.add_argument("--gap-s", type=float, default=DEFAULT_GAP_S)
|
||||
p.add_argument("--newline", choices=("crlf", "lf"), default="crlf",
|
||||
help="line terminator; crlf reproduces the committed files")
|
||||
p.add_argument("--timeout", type=float, default=120.0)
|
||||
p.add_argument("--retries", type=int, default=4)
|
||||
p.add_argument("--backoff", type=float, default=5.0)
|
||||
p.add_argument("--delay", type=float, default=2.0,
|
||||
help="pause between days - this is a live HA")
|
||||
p.add_argument("--verify", nargs="+", metavar="FILE",
|
||||
help="re-export these committed days and diff them")
|
||||
p.add_argument("--max-diff", type=int, default=20)
|
||||
args = p.parse_args(argv)
|
||||
|
||||
token = args.token or __import__("os").environ.get("HASS_TOKEN")
|
||||
if not token:
|
||||
print("no token: pass --token or set HASS_TOKEN", file=sys.stderr)
|
||||
return 2
|
||||
if ZoneInfo is None:
|
||||
print("zoneinfo unavailable", file=sys.stderr)
|
||||
return 2
|
||||
tz = ZoneInfo(args.tz)
|
||||
try:
|
||||
if args.verify:
|
||||
return do_verify(args, token, tz)
|
||||
return do_export(args, token, tz)
|
||||
except ReadFailed as exc:
|
||||
print(f"read failed: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,303 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run TARIFF-01's and SAFETY-07's AC 3 over a directory of `p1-net-*.jsonl.gz`.
|
||||
|
||||
AC 3, both cards, same words: *"monthly peak held under the soft limit in
|
||||
>= 95 % of intervals over a 30-day simulation replay."* Both cards merged with
|
||||
it UNMET because two days of capture existed. DATA-01 exported thirty. This is
|
||||
the thing that turns those thirty days into the number.
|
||||
|
||||
cd goodwe_controller
|
||||
python3 tools/replay_ac3.py scenarios/p1-net-30d-2026-08
|
||||
|
||||
WHAT IT MEASURES
|
||||
================
|
||||
Two runs over the same telegrams, one file at a time in date order into a
|
||||
SHARED peak store, so the 15-minute blocks accumulate into a real MONTH rather
|
||||
than into 30 disconnected days:
|
||||
|
||||
UNCONTROLLED the recorded meter as it happened. Nothing was shaving peaks on
|
||||
this house during August 2026, so this is not a simulation of
|
||||
anything - it is what the month did. It is TARIFF-01's AC 3.
|
||||
GUARDED the same month with `peakguard.decide()` in the loop and the
|
||||
battery it asks for modelled, closed-loop: the meter the guard
|
||||
sees is the recorded net MINUS the discharge it obtained. That
|
||||
is SAFETY-07's AC 3.
|
||||
|
||||
Both numbers are `quarters <= peak_cap_w` over `quarters closed`, with the cap
|
||||
read out of config.yaml rather than typed. Nothing here exits non-zero on 95 %:
|
||||
a measurement below the criterion is a finding about the house and the guard,
|
||||
and this tool's job is to report it, not to be graded by it.
|
||||
|
||||
FAITHFULNESS TO THE SUITE
|
||||
=========================
|
||||
The replay loop is `test_peakguard.py` part 6's, with `pack=None` (SoC pinned
|
||||
above the floor, so SAFETY-24's energy floor is not the thing deciding the
|
||||
answer). It is not imported from there because that file runs its whole suite on
|
||||
import. The duplication is checked instead of asserted: pointed at `scenarios/`
|
||||
- the two committed days - this tool must reproduce part 6's published totals,
|
||||
and `--expect` makes that a hard check.
|
||||
|
||||
python3 tools/replay_ac3.py scenarios --expect 189/190
|
||||
|
||||
GAPS
|
||||
====
|
||||
Nothing is interpolated. A hole in the export is a hole here: the shipped
|
||||
`meter_max_age_s` is what decides whether a held reading still counts as
|
||||
coverage, so an hour with no telegrams closes no quarters and contributes to
|
||||
neither numerator nor denominator. `--report-gaps` lists them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
from app.p1 import P1Ingest, make_sample, split_signed # noqa: E402
|
||||
from app.peakguard import decide # noqa: E402
|
||||
from app.tariff import PeakStore, PeakTracker, month_key # noqa: E402
|
||||
|
||||
|
||||
def read_option(key: str):
|
||||
"""The shipped default for `key`, out of config.yaml.
|
||||
|
||||
⚠️ `peak_cap_w:` appears TWICE in config.yaml - once under `options:` with
|
||||
the default, once under `schema:` as `int(500,15000)`. The FIRST match is
|
||||
the options block, which is the number an install actually lands on; taking
|
||||
the last would try to float() the schema line. Same rule as the suites.
|
||||
"""
|
||||
for line in (HERE / "config.yaml").read_text(encoding="utf-8").splitlines():
|
||||
if line.strip().startswith(f"{key}:"):
|
||||
try:
|
||||
return float(line.split(":", 1)[1].strip())
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def read_schema_max(key: str):
|
||||
for line in (HERE / "config.yaml").read_text(encoding="utf-8").splitlines():
|
||||
s = line.strip()
|
||||
if s.startswith(f"{key}:") and "(" in s and ")" in s:
|
||||
try:
|
||||
return float(s[s.index("(") + 1:s.index(")")].split(",")[-1])
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def sample(t: datetime, net_w: float):
|
||||
"""One telegram through the SHIPPED sign derivation - the suites' helper."""
|
||||
imp, exp = split_signed(net_w)
|
||||
return make_sample("ha_signed", imp, exp, phases=1,
|
||||
ingest_ts=t, ingest_mono=t.timestamp())
|
||||
|
||||
|
||||
def rows(path):
|
||||
"""Every sample in one file, in order, with the gaps left as gaps."""
|
||||
with gzip.open(path, "rt", encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
row = json.loads(line)
|
||||
yield datetime.fromisoformat(row["t"]), float(row["w"])
|
||||
|
||||
|
||||
def replay(paths, *, guarded: bool, cap_w, max_w, max_hold_s, cycle_s,
|
||||
soc, soc_floor, tmp: Path, gap_s: float):
|
||||
"""test_peakguard.py part 6's loop, over an arbitrary number of days.
|
||||
|
||||
⚠️ ONE STORE, AND AN INGEST THAT IS RESET ONLY AT A REAL DISCONTINUITY.
|
||||
Neither half of that is a detail.
|
||||
|
||||
The STORE is shared across the files because "monthly peak" is what AC 3 is
|
||||
about and a per-file store cannot express it.
|
||||
|
||||
The INGEST is carried across a file boundary when the next file's first
|
||||
telegram follows the previous file's last one within `gap_s`, and rebuilt
|
||||
when it does not. Both failure modes are real and were measured here:
|
||||
|
||||
* carried across a genuine hole, one averager manufactures quarters out of
|
||||
nothing - streamed straight through the three-day gap between the two
|
||||
committed days this reported 285/286 where the suite reports 189/190;
|
||||
* rebuilt at every file, each day's LAST quarter (21:45-22:00 UTC) never
|
||||
closes, because it closes only when a telegram past the boundary
|
||||
arrives and that telegram is in the next file. That is why the suite's
|
||||
two days give 95 quarters each rather than 96, and it silently discards
|
||||
one quarter per day - 30 of them over a month.
|
||||
|
||||
So contiguous days are joined and non-contiguous ones are not, which gives
|
||||
189/190 on the committed pair and a full 96 quarters a day on a run of
|
||||
consecutive days.
|
||||
"""
|
||||
store = PeakStore(str(tmp / f"ac3-{'g' if guarded else 'u'}.db"))
|
||||
tracker = PeakTracker(store, cap_w=cap_w)
|
||||
engaged = blind = saturated = cycles = n = joins = 0
|
||||
wh = 0.0
|
||||
gaps = []
|
||||
months = set()
|
||||
ing = None
|
||||
batt = 0.0 # delivered discharge, W (positive)
|
||||
next_cycle = None
|
||||
prev_t = None
|
||||
for path in paths:
|
||||
for i, (t, w) in enumerate(rows(path)):
|
||||
if i == 0:
|
||||
contiguous = (prev_t is not None
|
||||
and 0 <= (t - prev_t).total_seconds() <= gap_s)
|
||||
if contiguous:
|
||||
joins += 1
|
||||
else:
|
||||
if ing is not None:
|
||||
months |= {month_key(b.start) for b in ing.blocks}
|
||||
ing = P1Ingest(phases=1, max_age_s=max_hold_s)
|
||||
batt = 0.0
|
||||
next_cycle = None
|
||||
prev_t = None
|
||||
if prev_t is not None:
|
||||
dt = (t - prev_t).total_seconds()
|
||||
if dt > gap_s:
|
||||
gaps.append((prev_t, t, dt))
|
||||
wh += batt * dt / 3600.0
|
||||
prev_t = t
|
||||
n += 1
|
||||
ing.submit(sample(t, w - batt))
|
||||
tracker.sync(ing)
|
||||
if next_cycle is not None and t < next_cycle:
|
||||
continue
|
||||
next_cycle = t + timedelta(seconds=cycle_s)
|
||||
cycles += 1
|
||||
d = decide(PeakTracker.project(ing.averager, t), batt_w=batt,
|
||||
cap_w=cap_w, reading=tracker.peak(t),
|
||||
achievable_w=max_w, max_hold_s=max_hold_s, soc=soc,
|
||||
soc_floor=soc_floor, frozen=False)
|
||||
if d.engaged:
|
||||
blind += 1 if d.blind else 0
|
||||
engaged += 0 if d.blind else 1
|
||||
saturated += 1 if d.saturated else 0
|
||||
if not guarded:
|
||||
continue
|
||||
engaged_floor = d.engaged and d.floor_w is not None
|
||||
batt = max(0.0, d.floor_w) if engaged_floor else 0.0
|
||||
if ing is not None:
|
||||
months |= {month_key(b.start) for b in ing.blocks}
|
||||
out = {m: dict(store.blocks_in(m)) for m in sorted(months)}
|
||||
return {"months": out, "samples": n, "cycles": cycles, "joins": joins,
|
||||
"engaged": engaged, "blind": blind, "saturated": saturated,
|
||||
"wh": wh, "gaps": gaps}
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
p = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||
p.add_argument("directory", help="a directory of p1-net-*.jsonl.gz")
|
||||
p.add_argument("--glob", default="p1-net-*.jsonl.gz")
|
||||
p.add_argument("--expect", help="UNDER/TOTAL the uncontrolled run must "
|
||||
"reproduce, e.g. 189/190")
|
||||
p.add_argument("--report-gaps", action="store_true")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
paths = sorted(Path(args.directory).glob(args.glob))
|
||||
if not paths:
|
||||
print(f"no {args.glob} under {args.directory}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
CAP_W = read_option("peak_cap_w")
|
||||
MAX_HOLD_S = read_option("meter_max_age_s")
|
||||
CYCLE_S = read_option("cycle_floor_s")
|
||||
SOC_FLOOR = read_option("peak_soc_floor")
|
||||
LIVE_MAX_W = read_schema_max("max_w")
|
||||
if None in (CAP_W, MAX_HOLD_S, CYCLE_S, SOC_FLOOR, LIVE_MAX_W):
|
||||
print("could not read the shipped options out of config.yaml",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
# The suites' fixture SoC: comfortably above the floor, so SAFETY-24's
|
||||
# energy floor is not what decides an AC 3 that is not about it.
|
||||
SOC_OK = SOC_FLOOR + 30.0
|
||||
|
||||
days = len(paths)
|
||||
span_first = span_last = None
|
||||
print(f"AC 3 - {days} file(s) from {args.directory}")
|
||||
print(f" shipped: peak_cap_w {CAP_W:.0f} W, meter_max_age_s "
|
||||
f"{MAX_HOLD_S:.0f} s, cycle_floor_s {CYCLE_S:.0f} s, "
|
||||
f"max_w schema ceiling {LIVE_MAX_W:.0f} W")
|
||||
|
||||
tmp = Path(tempfile.mkdtemp(prefix="goodwe-ac3-"))
|
||||
kw = dict(cap_w=CAP_W, max_w=LIVE_MAX_W, max_hold_s=MAX_HOLD_S,
|
||||
cycle_s=CYCLE_S, soc=SOC_OK, soc_floor=SOC_FLOOR, tmp=tmp,
|
||||
gap_s=MAX_HOLD_S)
|
||||
u = replay(paths, guarded=False, **kw)
|
||||
g = replay(paths, guarded=True, **kw)
|
||||
|
||||
def flat(run):
|
||||
return [w for m in run["months"].values() for w in m.values()]
|
||||
|
||||
uq, gq = flat(u), flat(g)
|
||||
u_under = sum(1 for w in uq if w <= CAP_W)
|
||||
g_under = sum(1 for w in gq if w <= CAP_W)
|
||||
u_pct = 100.0 * u_under / max(len(uq), 1)
|
||||
g_pct = 100.0 * g_under / max(len(gq), 1)
|
||||
|
||||
print(f" {u['samples']} samples, {len(uq)} closed quarters "
|
||||
f"({len(uq) / 96.0:.1f} days' worth), {days - u['joins']} contiguous "
|
||||
f"run(s) of days")
|
||||
for m in sorted(u["months"]):
|
||||
mu = u["months"][m]
|
||||
mg = g["months"].get(m, {})
|
||||
print(f" month {m}: {len(mu)} quarters, peak "
|
||||
f"{max(mu.values()):.1f} W uncontrolled -> "
|
||||
f"{max(mg.values()) if mg else 0.0:.1f} W guarded")
|
||||
print()
|
||||
print(f" TARIFF-01 AC 3 (UNCONTROLLED, what the month did): "
|
||||
f"{u_under}/{len(uq)} = {u_pct:.2f} % of quarters at or under "
|
||||
f"{CAP_W:.0f} W")
|
||||
print(f" SAFETY-07 AC 3 (GUARDED, peakguard.decide in the loop): "
|
||||
f"{g_under}/{len(gq)} = {g_pct:.2f} %")
|
||||
print(f" guard cycles {g['cycles']}, engaged on the measurement "
|
||||
f"{g['engaged']}, blind {g['blind']}, saturated {g['saturated']}, "
|
||||
f"modelled discharge {g['wh']:.0f} Wh")
|
||||
# AC 3'S WORDS SUPPORT TWO READINGS AND BOTH ARE REPORTED, because they
|
||||
# disagree on this data and picking the flattering one is the failure mode
|
||||
# this whole card exists to correct.
|
||||
u_peak = max(uq) if uq else 0.0
|
||||
g_peak = max(gq) if gq else 0.0
|
||||
print(f" reading A - PERCENTAGE OF INTERVALS under the cap (the reading "
|
||||
f"test_tariff.py part 9 already uses): uncontrolled {u_pct:.2f} %, "
|
||||
f"guarded {g_pct:.2f} %")
|
||||
print(f" reading B - THE MONTHLY PEAK ITSELF under the cap: "
|
||||
f"{u_peak:.1f} W uncontrolled -> {g_peak:.1f} W guarded, against a "
|
||||
f"{CAP_W:.0f} W soft limit "
|
||||
f"({'under' if g_peak <= CAP_W else 'STILL OVER'})")
|
||||
print(f" verdict at the 95 % criterion: uncontrolled "
|
||||
f"{'MEETS' if u_pct >= 95.0 else 'DOES NOT MEET'}, guarded "
|
||||
f"{'MEETS' if g_pct >= 95.0 else 'DOES NOT MEET'} "
|
||||
f"({len(uq) / 96.0:.1f} days of replay)")
|
||||
|
||||
if u["gaps"]:
|
||||
tot = sum(d for _, _, d in u["gaps"])
|
||||
print(f" gaps over {MAX_HOLD_S:.0f} s: {len(u['gaps'])}, "
|
||||
f"{tot:.0f} s total - NOT interpolated; the quarters they fall in "
|
||||
f"simply carry less coverage")
|
||||
if args.report_gaps:
|
||||
for a, b, d in u["gaps"]:
|
||||
print(f" {a.isoformat()} -> {b.isoformat()} {d:.0f} s")
|
||||
|
||||
if args.expect:
|
||||
want_u, want_t = (int(x) for x in args.expect.split("/"))
|
||||
ok = (u_under, len(uq)) == (want_u, want_t)
|
||||
print(f" reproduces test_peakguard.py part 6's published "
|
||||
f"{want_u}/{want_t}: {'yes' if ok else 'NO'} "
|
||||
f"(got {u_under}/{len(uq)})")
|
||||
return 0 if ok else 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user