#!/usr/bin/env python3 """DATA-02: fetch 12 months of EPEX BE day-ahead prices, in §9.3's shape. python3 scenarios/fetch_epex_be.py --start 2025-09-01 --end 2026-09-01 \ --out scenarios/epex-be-dayahead-2025-09-01_2026-09-01.json Run from `goodwe_controller/`. Rewrites the output file from scratch; the committed dataset is exactly what this produces, and `--verify` re-derives every claim the file makes about itself without going to the network. WHAT THIS IS NOT ================ NOT wired into the add-on, and it must never be. `price.py`'s `GenericDayAhead` is the live path and it has its own egress check; this is a one-shot archival fetch that produces a file for REPLAY, run by hand by a developer. Nothing under `app/` imports it. THE SOURCE, AND WHY THIS ONE ============================ `https://api.energy-charts.info/price?bzn=BE` - Fraunhofer ISE's Energy-Charts API. No key, no registration. ⚠️ THE LICENCE WAS CHECKED BEFORE ANY DATA WAS COMMITTED, because for this source it is NOT uniform across bidding zones. The API's own OpenAPI document (`https://api.energy-charts.info/openapi.json`, /price) splits the zones in two: "The data for the following bidding zones is licensed as CC BY 4.0 from Bundesnetzagentur | SMARD.de and is published without changes: AT, BE (Belgium), CH, CZ, DE-LU, ..." "The data for the other bidding zones is for private and internal use only. The utilization of any data, whether in its raw or derived form, for external or commercial purposes is expressly prohibited. Should you require licensing for market-related data, please direct your inquiries to the original data providers, including but not limited to EPEX SPOT SE." BE is in the FIRST list, so redistribution is permitted with attribution and the dataset is committed rather than left as a fetch script and a checksum. Every response also carries the licence per-response in `license_info`, and this script REFUSES to write a file if that field ever stops saying CC BY 4.0 - a licence checked once by a human and assumed forever is a licence that changes silently. Had BE been in the second list, the correct output of this card would have been this script and a checksum and no data. Attribution, carried in the dataset's own `provenance` block and reproduced here: data CC BY 4.0, Bundesnetzagentur | SMARD.de, retrieved via Energy-Charts (Fraunhofer ISE), https://energy-charts.info, published without changes. Cross-checked against the upstream on 2026-08-31: SMARD's own chart_data series `4996/DE` over 2026-04-10 matched this API's BE series on all 96 quarter-hours with a maximum absolute difference of 0.00 EUR/MWh. "Published without changes" is a claim worth checking once; `--verify` does not repeat it, because it is a claim about the SOURCE and not about the file. RAW WHOLESALE, AND THE ASSUMPTION THAT IS NOT BAKED IN ====================================================== ⚠️ WHAT COMES BACK IS RAW WHOLESALE, EUR/MWh, DAY-AHEAD AUCTION CLEARING PRICE. It is NOT all-in: no supplier markup, no distribution or transmission tariff, no levies, no VAT. §9.3's canonical point is all-in, and TARIFF-02's whole trap is a figure that already contains a component being handed to something that adds it again. So this file does NOT add one. `eur_per_kwh` in the output is the wholesale figure converted from EUR/MWh, `provenance.all_in` is `false` beside it, and the markup/tax assumption a consumer needs lives in `provenance.all_in_assumption` as NUMBERS TO APPLY rather than numbers already applied. test_arbitrage.py section 7 applies them where a reader can watch it happen. The assumption's SHAPE is the part that is not arbitrary, and it decides which conclusions depend on the guessed numbers at all: all_in(h) = (wholesale(h) + markup + network_and_levies) * (1 + vat) * The two ADDITIVE terms cancel out of a spread: max - min is unchanged by them. So how often a day clears `cycle_cost_threshold_eur` does NOT depend on the markup guess - only on the VAT multiplier. * They do NOT cancel out of §17's negative-price guard, which compares ONE hour's all-in price against zero. A wholesale hour at -0.02 EUR/kWh is comfortably positive all-in once ~0.12 of fixed charges is added. Both are asserted arithmetically in section 7 rather than believed, and the trade count is reported on the raw spread as well, so a reader can see exactly how much of the answer rests on the guess. The numbers themselves (2026 Belgian residential dynamic contract, order of magnitude) are a STATED GUESS, labelled as one everywhere they appear. They are not a measurement and this card had nothing to measure them with. HOURS, QUARTER-HOURS, AND THE SEAM IN THE MIDDLE OF THE YEAR ============================================================ ⚠️ THE SOURCE CHANGES RESOLUTION PART-WAY THROUGH ANY RECENT 12 MONTHS. EPEX day-ahead moved to a 15-minute market time unit in October 2025 and the feed follows: 2025-09-10 comes back as 24 hourly points, 2025-12-10 as 96 quarter-hourly ones. An importer that assumed either one silently loses three quarters of the year, or reads a quarter as an hour. The canonical format is hourly, so quarter-hours are aggregated: an hour's price is the unweighted mean of its four quarters. That is a STATED APPROXIMATION of the hourly product, not the hourly product, and it errs in the direction that matters here - averaging discards intra-hour variation, so the observed spread is a LOWER bound on what a 15-minute strategy would see. `provenance.resolution` records how many hours came from each source shape. ⚠️ AND A PARTIAL HOUR IS DROPPED, NOT AVERAGED OVER WHAT ARRIVED. An hour with three of four quarters is an hour nobody published a complete price for; the mean of three would be an invented number carrying no flag. Dropped hours are listed in `provenance.gaps` with their timestamps, because a missing hour in a feed is not a zero price, and a zero price is exactly the value that would make everything look free. SIZE: WHY THIS IS PLAIN JSON AND NOT A `.gz`, MEASURED ====================================================== The directory's existing captures are gzipped, so the default would have been to gzip this too. Measured on the committed file instead of assumed: working tree, plain JSON 893 171 bytes what GIT actually stores for it (zlib, loose object) 74 590 bytes what git would store for a committed `.json.gz` 56 504 bytes ⚠️ GIT ALREADY COMPRESSES BLOBS, so gzipping buys 18 KB - 24 % of an object that is already 75 KB, in a repository whose three existing P1 captures cost 27 KB, 136 KB and 140 KB. Eighteen kilobytes is not worth making the provenance block unreadable in a pull request, and the licence claim in that block is precisely the thing a reviewer of this card has to read and check. Plain JSON also greps, diffs line-by-line when a future refetch changes something, and is what `--verify` opens. `load()` still handles `.gz` for a future dataset large enough to want it; this one is not. The P1 captures are gzipped for the opposite reason and it is not an inconsistency: they are hundreds of thousands of telegram lines that no human reads, and there is no metadata in them to review. TIME ==== Every instant is carried as UTC and rendered with `app.price.brussels()`, the module that already owns the EU DST rule without a tz database. There is no second answer to that question here and there must not be: the point of `brussels()` is that the 02:00-03:00 hour which happens twice in October is two instants, and a 24-slot local-hour importer collapses them. So a year is 8760 hours only if it contains no transition, or contains both and they cancel - 25 in October, 23 in March. `--verify` therefore asserts the PER-DAY counts, not the total, because the total is exactly the number a broken importer also produces. """ import argparse import gzip import hashlib import json import os import sys import time import urllib.error import urllib.request from collections import Counter, defaultdict from datetime import datetime, timedelta, timezone sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) from app.price import brussels # noqa: E402 API = "https://api.energy-charts.info/price" BZN = "BE" LICENCE_MUST_CONTAIN = "CC BY 4.0" UTC = timezone.utc ATTRIBUTION = ("Data licensed CC BY 4.0 by Bundesnetzagentur | SMARD.de, " "retrieved via the Energy-Charts API (Fraunhofer ISE), " "https://energy-charts.info. Published without changes.") # ⚠️ A GUESS, LABELLED AS ONE, AND APPLIED BY THE CONSUMER - NOT HERE. # Order-of-magnitude 2026 Belgian residential dynamic contract. Nothing in this # repository measured any part of it. ALL_IN_ASSUMPTION = { "form": "all_in = (wholesale + markup + network_and_levies) * (1 + vat)", "markup_eur_per_kwh": 0.020, "network_and_levies_eur_per_kwh": 0.100, "vat_fraction": 0.06, "status": "ASSUMPTION, NOT MEASURED. Stated so a consumer applies it " "deliberately; the committed eur_per_kwh has none of it applied.", "note": "The two additive terms cancel out of any spread, so an arbitrage " "trade count depends on vat_fraction only. They do NOT cancel out " "of the negative-price guard (17), which compares one hour's " "all-in price against zero.", } PACE_S = 3.0 # between chunks, unconditionally RETRIES = 6 def _get(url: str) -> dict: """One GET, backing off on 429 and 5xx. ⚠️ THIS IS A FREE PUBLIC SERVICE AND IT RATE-LIMITS. The first run of this script was refused with 429 on its fifth chunk. Backing off is the polite behaviour; it is also the correct one, because the alternative - catching the error and carrying on - would produce a file that is silently missing a month and says nothing about it. A refused request is not an empty month. """ req = urllib.request.Request( url, headers={"User-Agent": "goodwe-addon/DATA-02 (one-shot archival fetch)"}) wait = 5.0 for attempt in range(1, RETRIES + 1): try: with urllib.request.urlopen(req, timeout=90) as r: return json.loads(r.read().decode("utf-8")) except urllib.error.HTTPError as e: if e.code not in (429, 500, 502, 503, 504) or attempt == RETRIES: raise print(f" HTTP {e.code}; backing off {wait:.0f}s " f"(attempt {attempt}/{RETRIES})", file=sys.stderr) time.sleep(wait) wait *= 2 raise SystemExit(f"unreachable: {url}") def local_midnight_utc(day: str) -> datetime: """The INSTANT at which Brussels midnight begins on `day` (YYYY-MM-DD). ⚠️ THE WINDOW IS LOCAL, NOT UTC, AND THE FIRST RUN OF THIS SCRIPT PROVED WHY. Bounding it at 00:00 UTC starts the year at 02:00 CEST, which makes the first Brussels day 22 hours long - and 22 is not one of the two lengths a transition produces, so `report()` refused the file. A dataset whose first day is short is a dataset whose "days per year" arithmetic is quietly wrong everywhere downstream. Derived from `brussels()` by trying the only two offsets Belgium has and keeping the one that round-trips, rather than by restating the EU rule here. """ for hours in (1, 2): guess = datetime.fromisoformat(f"{day}T00:00:00").replace( tzinfo=timezone(timedelta(hours=hours))).astimezone(UTC) if brussels(guess).isoformat() == f"{day}T00:00:00+0{hours}:00": return guess raise SystemExit(f"no Brussels midnight for {day!r} - brussels() disagrees " f"with itself, which is a bug in price.py, not here.") def _months(start: datetime, end: datetime): """[start, end) in month-sized chunks, so a bad chunk names its own window.""" a = start while a < end: b = min(end, (a.replace(day=1) + timedelta(days=32)).replace(day=1)) yield a, b a = b def fetch(start: datetime, end: datetime): """Raw {unix_second: EUR/MWh} over [start, end), deduplicated. ⚠️ A `null` price is DROPPED, never coerced. The API returns null for an interval it holds no result for, and `or 0.0` is the version of this line that ships a year of free electricity. """ seen, licences, nulls = {}, Counter(), 0 for a, b in _months(start, end): # ⚠️ THE API'S `end` IS AN INCLUSIVE LOCAL DAY, AND THE WINDOW'S IS AN # EXCLUSIVE INSTANT. Ask for the day containing the LAST hour of the # chunk, derived from the instant rather than by subtracting a calendar # day from the bound: `b - 1 day` drops the final local day whenever the # window ends at local midnight, which is exactly how this window ends. # The first run lost 2026-08-31 that way, silently - 8736 points instead # of 8760 - and the points-plus-gaps accounting in report() is what # noticed. A day nobody asked for is not a day with no prices. first = brussels(a).date().isoformat() last = brussels(b - timedelta(hours=1)).date().isoformat() url = f"{API}?bzn={BZN}&start={first}&end={last}" d = _get(url) lic = str(d.get("license_info", "")) licences[lic] += 1 if LICENCE_MUST_CONTAIN not in lic: raise SystemExit( f"REFUSING TO WRITE. {url} came back under licence {lic!r}, " f"which does not say {LICENCE_MUST_CONTAIN!r}. The BE zone was " f"redistributable when this card ran; if that has changed, the " f"right output is this script and a checksum, not the data.") unit = str(d.get("unit", "")) if "MWh" not in unit: raise SystemExit( f"REFUSING TO WRITE. {url} reports unit {unit!r}, not EUR/MWh. " f"A silent unit change is a factor of 1000 on every price.") for t, p in zip(d["unix_seconds"], d["price"]): if p is None: nulls += 1 continue if start.timestamp() <= t < end.timestamp(): seen[int(t)] = float(p) print(f" {a.date()} .. {b.date()} {len(d['price']):5d} intervals", file=sys.stderr) time.sleep(PACE_S) return seen, licences, nulls def to_hourly(raw: dict): """Hourly means keyed by UTC hour. Returns (hours, gaps, shape_counts). An hour is emitted only when every interval the source used for it arrived: 1 of 1 where the source was hourly, 4 of 4 where it was quarter-hourly. Anything else is a gap and is named as one. """ buckets = defaultdict(dict) for t, p in raw.items(): buckets[t - (t % 3600)][t % 3600] = p hours, gaps, shapes = {}, [], Counter() for h in sorted(buckets): got = buckets[h] if set(got) == {0}: shapes["hourly"] += 1 hours[h] = got[0] elif set(got) == {0, 900, 1800, 2700}: shapes["quarter_hourly_mean"] += 1 hours[h] = sum(got.values()) / 4.0 else: gaps.append({ "hour_utc": datetime.fromtimestamp(h, UTC).isoformat(), "intervals_present": sorted(got), "why": "incomplete hour - dropped rather than averaged over " "what arrived; a partial read is not an observation", }) return hours, gaps, shapes def canonical(hours: dict) -> list: """9.3 points. THREE FIELDS, rendered at the Brussels offset for the instant.""" return [{"start": brussels(datetime.fromtimestamp(h, UTC)).isoformat(), "eur_per_kwh": round(hours[h] / 1000.0, 6) + 0.0, "provisional": False} for h in sorted(hours)] def expected_hours(start: datetime, end: datetime) -> int: """Whole hours in the window. Derived, rather than 8760 typed in.""" return int((end - start).total_seconds() // 3600) def per_local_day(points) -> Counter: """Points per BRUSSELS calendar day - the count that exposes a DST bug.""" n = Counter() for p in points: n[p["start"][:10]] += 1 return n def digest(points) -> str: """A checksum over the POINTS, so reformatting the envelope does not move it.""" h = hashlib.sha256() for p in points: h.update(f"{p['start']}|{p['eur_per_kwh']:.6f}|{int(p['provisional'])}\n" .encode()) return h.hexdigest() def load(path: str) -> dict: op = gzip.open if path.endswith(".gz") else open with op(path, "rt", encoding="utf-8") as f: return json.load(f) def report(doc, *, start=None, end=None) -> int: """Re-derive every claim the file makes about itself. Returns failures.""" pts = doc["prices"] prov = doc["provenance"] bad = 0 def ck(name, cond): nonlocal bad print(f" {'ok ' if cond else 'FAIL'} {name}") if not cond: bad += 1 ck(f"the checksum over the {len(pts)} points matches the recorded one " f"({prov['sha256_of_points'][:16]}...)", digest(pts) == prov["sha256_of_points"]) ck("every point has exactly the three canonical fields and nothing else", all(set(p) == {"start", "eur_per_kwh", "provisional"} for p in pts)) ck("every start carries a UTC offset, and it is +01:00 or +02:00 - a naive " "timestamp here is a wall clock plus an assumption", bool(pts) and all(p["start"][-6:] in ("+01:00", "+02:00") for p in pts)) ck("the series is strictly increasing in time, so no instant is duplicated", all(datetime.fromisoformat(a["start"]) < datetime.fromisoformat(b["start"]) for a, b in zip(pts, pts[1:]))) ck("consecutive points are exactly one hour apart AS INSTANTS, so the two " "02:00s of the October day are two hours rather than one", all((datetime.fromisoformat(b["start"]) - datetime.fromisoformat(a["start"])) == timedelta(hours=1) for a, b in zip(pts, pts[1:]))) days = per_local_day(pts) odd = {d: n for d, n in days.items() if n != 24} ck(f"exactly two Brussels days are not 24 hours long, and they are the " f"transitions: {sorted(odd.items())}", sorted(odd.values()) == [23, 25] and len(odd) == 2) spring = [d for d, n in odd.items() if n == 23] autumn = [d for d, n in odd.items() if n == 25] ck(f"the 23-hour day {spring} is in March and the 25-hour day {autumn} is " f"in October, which is the direction the EU rule actually runs", len(spring) == 1 and len(autumn) == 1 and spring[0][5:7] == "03" and autumn[0][5:7] == "10") if autumn: twice = [p["start"] for p in pts if p["start"].startswith(autumn[0] + "T02")] ck(f"...and 02:00 on {autumn[0]} occurs TWICE at two different offsets " f"{twice} - exactly the case a 24-slot local-hour array collapses", len(twice) == 2 and twice[0][-6:] == "+02:00" and twice[1][-6:] == "+01:00") ck(f"...and the two carry DIFFERENT prices, so the 25th hour is a real " f"observation and not the same one written down twice", len({p["eur_per_kwh"] for p in pts if p["start"].startswith(autumn[0] + "T02")}) == 2) if spring: ck(f"...and 02:00 on {spring[0]} does not occur at all, because that " f"wall clock reading never happened in Belgium", not any(p["start"].startswith(spring[0] + "T02") for p in pts)) if start and end: want = expected_hours(start, end) ck(f"{len(pts)} points for a window of {want} whole hours, and the " f"{len(prov['gaps'])} recorded gaps account for the difference " f"exactly", len(pts) + len(prov["gaps"]) == want) # ⚠️ A ZERO IS NOT AUTOMATICALLY A DEFECT HERE, AND SAYING IT WAS WOULD BE # THE WRONG LESSON. The Belgian market really does clear at 0.00 EUR/MWh in # solar-glut hours; this window contains such hours. What must be impossible # is a zero that arrived by COERCION - a dropped read turned into a free # hour. That is excluded structurally rather than by outlawing the value: # nulls are dropped, incomplete hours become gaps, and the accounting above # shows points + gaps covering every hour of the window, so every emitted # point stands for intervals the source actually published. zeros = [p["start"] for p in pts if p["eur_per_kwh"] == 0.0] ck(f"the {len(zeros)} hours priced at exactly zero match the count recorded " f"at build time, so no zero appeared between the fetch and the file", len(zeros) == prov["exact_zero_hours"]) ck("...and none of them is a coerced read: every emitted hour is backed by " "published intervals, which is what the points-plus-gaps accounting " "above establishes", len(prov["gaps"]) == 0 or all("intervals_present" in g for g in prov["gaps"])) ck("the figures are declared RAW WHOLESALE, and the all-in assumption is " "recorded as an assumption rather than applied", prov["all_in"] is False and "ASSUMPTION" in prov["all_in_assumption"]["status"]) return bad def main() -> int: ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap.add_argument("--start", default="2025-09-01") ap.add_argument("--end", default="2026-09-01", help="exclusive; both bounds are BRUSSELS-local midnights") ap.add_argument("--out", default=None) ap.add_argument("--verify", default=None, help="re-derive the claims of an existing file; no network") a = ap.parse_args() if a.verify: doc = load(a.verify) print(f"verifying {a.verify}") s = datetime.fromisoformat(doc["provenance"]["window_utc"]["start"]) e = datetime.fromisoformat(doc["provenance"]["window_utc"]["end"]) bad = report(doc, start=s, end=e) print("OK" if not bad else f"{bad} FAILED") return 1 if bad else 0 start = local_midnight_utc(a.start) end = local_midnight_utc(a.end) out = a.out or f"scenarios/epex-be-dayahead-{a.start}_{a.end}.json" print(f"fetching {BZN} day-ahead {start.date()} .. {end.date()}", file=sys.stderr) raw, licences, nulls = fetch(start, end) hours, gaps, shapes = to_hourly(raw) pts = canonical(hours) doc = { "provenance": { "card": "DATA-02", "what": "EPEX BE day-ahead auction clearing prices, hourly", "source": f"{API}?bzn={BZN} (Energy-Charts API, Fraunhofer ISE)", "upstream": "Bundesnetzagentur | SMARD.de", "licence": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)", "licence_checked": "before committing anything; the API's OpenAPI " "document lists BE among the CC BY 4.0 zones and " "the remaining zones as private/internal use " "only, so redistribution of THIS zone is " "permitted with attribution", "licence_as_returned": sorted(licences), "attribution": ATTRIBUTION, "cross_check": "SMARD chart_data 4996/DE over 2026-04-10 matched " "this feed on all 96 quarter-hours, max abs " "difference 0.00 EUR/MWh (2026-08-31)", "retrieved_utc": datetime.now(UTC).replace(microsecond=0).isoformat(), "fetched_by": "scenarios/fetch_epex_be.py", "window_utc": {"start": start.isoformat(), "end": end.isoformat()}, "unit": "EUR per kWh (source publishes EUR/MWh; divided by 1000)", "all_in": False, "all_in_note": "RAW WHOLESALE. No supplier markup, no network or " "transmission tariff, no levies, no VAT. 9.3's " "canonical point is all-in and this is NOT; the " "consumer applies the assumption below, visibly.", "all_in_assumption": ALL_IN_ASSUMPTION, "resolution": { "note": "EPEX day-ahead moved to a 15-minute market time unit " "in Oct 2025, part-way through this window.", "hours_from_hourly_source": shapes["hourly"], "hours_from_quarter_hourly_mean": shapes["quarter_hourly_mean"], "aggregation": "unweighted mean of the four quarters; a stated " "approximation of the hourly product, and a " "LOWER bound on the spread a 15-minute strategy " "would have seen", }, "timezone": "starts rendered at the Brussels offset by " "app.price.brussels(); instants carried in UTC, so the " "transition days come out 23 and 25 points by " "construction rather than by a special case", "nulls_dropped": nulls, "gaps": gaps, "gaps_note": "hours the source did not publish completely. Dropped, " "never zero-filled: a missing hour is not a free hour.", "points": len(pts), "exact_zero_hours": sum(1 for p in pts if p["eur_per_kwh"] == 0.0), "negative_hours": sum(1 for p in pts if p["eur_per_kwh"] < 0), "zero_note": "REAL CLEARINGS AT 0.00 EUR/MWh, not dropped reads. " "Nulls are dropped and incomplete hours become gaps, " "so points + gaps == hours in the window and no point " "stands in for something that failed to arrive.", "sha256_of_points": digest(pts), }, "prices": pts, } with open(out, "w", encoding="utf-8", newline="\n") as f: json.dump(doc, f, indent=1, ensure_ascii=False) f.write("\n") print(f"wrote {out} {len(pts)} points {os.path.getsize(out)} bytes", file=sys.stderr) print(f"verifying {out}") return 1 if report(doc, start=start, end=end) else 0 if __name__ == "__main__": raise SystemExit(main())