admin/goodwe-addon
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

DATA-02: a real Belgian price year, so AC4 stops being unmeetable
checks / unit (pull_request) Successful in 3m36s
Details

Browse Source 
OPT-04 reported AC4 - "net-positive across 12 months of replayed prices" -
as NOT MET and replayed a synthetic year, because scenarios/ held only P1
meter captures. Right call, wrong diagnosis: the criterion was being carried
beside a 30-day METER criterion as though both needed waiting. EPEX BE
day-ahead is public data, not a capture. It needed a fetch.

8760 hourly points, 2025-09-01 to 2026-09-01 Brussels-local, canonical three
fields. Licence checked BEFORE committing: Energy-Charts publishes the BE
zone under CC BY 4.0 from Bundesnetzagentur | SMARD.de and other zones under
private-use-only terms, so this one is redistributable with attribution. The
fetcher refuses to write a file whose per-response license_info stops saying
so. Cross-checked against SMARD's own 4996/DE: 96 of 96 quarter-hours, max
difference 0.00 EUR/MWh.

Raw wholesale, and the file says so. all_in: false, with the markup/tax
assumption carried as numbers to APPLY rather than numbers applied -
TARIFF-02's trap in data form. Section 7B applies it in one visible place and
asserts arithmetically that its additive terms cancel out of every daily
spread, so the trade count turns on the VAT multiplier alone; the raw replay
is run too, so how much rests on the guess is reported (110 vs 88).

DST by construction: iterate in UTC, render through price.brussels().
2025-10-26 is 25 points, 2026-03-29 is 23, verified by counting per Brussels
day rather than asserting 24, and by checking October's 02:00 appears twice
at two offsets with two prices while March's never appears. Both bugs the
point-and-gap accounting caught were real: a UTC-bounded window made the
first day 22 hours, and an inclusive-end off-by-one dropped the last day
silently. A day nobody asked for is not a day with no prices.

THE RESULT, AT THE SHIPPED DEFAULT, WHATEVER IT IS. At 0.10 widened 1.5x to
0.1500: 110 round trips, +24.88 EUR net of the cycle cost. Unwidened, 173.
Nothing tuned - the threshold and the 2.5/1.5 factors are untouched.

And what the owner's decision now has under it: the four reports that a
static ~0.0840 spread does not clear 0.10 all STAND. What was missing is that
a dynamic site clears the widened requirement on 121 of 365 days, median
all-in spread 0.1130, so a median day would clear at 0.0754 - 156 trips,
+35.42 EUR. Reported, not applied.

The synthetic year stays, because the real one cannot grade section 17: not
one of 365 days has a negative DEAREST hour, so a zero there counts something
that could not happen. 7B asserts that degeneracy rather than reporting the
zero.

Size measured, not assumed: 893 KB plain but 74.6 KB as a git object against
56.5 KB gzipped. Git already compresses; 18 KB is not worth making the
provenance block - the licence claim a reviewer must check - unreadable in a
diff.

Nothing wired into the add-on; generic_dayahead is still the live path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VXjA3LWccN1rJsk3okyPC
This commit is contained in:
glenn schrooyen
2026-08-31 11:01:42 +02:00
co-authored by Claude Opus 5
parent 4601682815
commit d57122ec74
4 changed files with 44760 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
goodwe_controller/test_arbitrage.py
+284 -32
View File
@@ -20,12 +20,15 @@ WHAT THIS FILE IS DEFENDING, in the order the card's acceptance criteria name it
exist - static_tariff's required `peak_guard_active` interlock and
arbiter.py's clause 2 - both driven end to end, plus the third state
(the guard never ran) that neither of them can express on its own.
AC4 NET-POSITIVE ACROSS 12 MONTHS OF REPLAYED PRICES. ⚠️ SEE SECTION 7. THIS
REPOSITORY HOLDS NO PRICE HISTORY AT ALL - `scenarios/` carries two days
of real METER data and nothing else - so the criterion as written cannot
be met, and it is NOT redefined here into something that can be. What
section 7 does instead is stated there in full, and what it cannot say is
stated with it.
AC4 NET-POSITIVE ACROSS 12 MONTHS OF REPLAYED PRICES. ⚠️ MET AS OF DATA-02,
AND SECTIONS 7 AND 7B DO DIFFERENT HALVES OF IT. 7B replays twelve months
of REAL EPEX BE day-ahead prices - `scenarios/epex-be-dayahead-*.json`,
8760 hours, public data under CC BY 4.0 - through the shipped `decide()`,
and reports the euro figure at the shipped threshold. Section 7 keeps the
SYNTHETIC year, because the real one turns out to be unable to grade §17
at all: not one of its 365 days has a negative DEAREST hour, so a count
of zero negative-price discharges over it grades nothing. 7B asserts that
degeneracy rather than quietly reporting the zero.
THREE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
@@ -786,34 +789,39 @@ check("...and it is still ARMED when arbitrage is idle and the loop wins, so "
print()
print("7. AC4 - 12 months of replayed prices, and what could NOT be measured")
# ⚠️ READ THIS BEFORE READING THE NUMBERS BELOW.
# ⚠️ READ THIS BEFORE READING THE NUMBERS BELOW. THIS SECTION IS THE SYNTHETIC
# YEAR, AND IT IS NOT AC4's EVIDENCE ANY MORE - SECTION 7B IS.
#
# The acceptance criterion is "net-positive across 12 months of replayed prices
# in simulation". THIS REPOSITORY CONTAINS NO PRICE HISTORY. `scenarios/` holds
# three gzipped captures and all three are P1 METER data - two days of net power
# from the reference house plus one age capture. There is no EPEX archive, no
# supplier export, and no CSV of a past year anywhere in the tree.
# When OPT-04 shipped, this repository held no price history: `scenarios/`
# carried three gzipped P1 METER captures and nothing else. AC4 was therefore
# reported NOT MET rather than redefined into something passable, which is what
# TARIFF-01 and SAFETY-07 had both been upheld for doing.
#
# TARIFF-01 and SAFETY-07 both hit an unmeetable criterion and both DECLINED to
# redefine it into one they could pass, and both were upheld for it. So:
# DATA-02 pointed out that the missing ingredient was never a capture. EPEX BE
# day-ahead is PUBLIC DATA; the 30-day meter criterion it was being carried
# beside genuinely needs waiting, and this one only ever needed a fetch. Section
# 7B replays the real year and AC4 is met there.
#
# AC4 IS NOT MET, AND IT IS NOT RESTATED. What follows is not a replay of the
# Belgian market; it is a replay of a SYNTHETIC year, and a euro figure from a
# synthetic year grades the generator that produced it, not the strategy.
# ⚠️ SO WHY IS THE SYNTHETIC YEAR STILL HERE? Because the real one cannot grade
# §17, and 7B proves that rather than assuming it. Across all 365 real days, the
# number whose DEAREST hour is priced below zero is ZERO - raw wholesale and
# all-in alike - so "no discharges into a negative hour" over the real year is a
# count of something that never had the chance to happen. Deleting the guard
# would not change it. This section's generator drives a summer midday block
# negative ON PURPOSE, and it is the only thing in this file that puts a
# negative price where the strategy would otherwise sell.
#
# What the replay is FOR is the half of AC4 that does not need real prices. Net
# positivity here is not an empirical outcome to be hoped for - it is a property
# of the decision rule, and the replay's job is to demonstrate that the property
# holds on every trade the strategy actually chose, over a year of varied days
# including negative-price ones. Stated as an invariant:
# What this section establishes is the half of AC4 that does not need real
# prices, and it is a property of the decision rule rather than an outcome to be
# hoped for:
#
# every executed round trip has (sell - buy) >= required >= threshold,
# so its benefit net of the threshold is >= 0, on every day, always.
#
# That is asserted per trade below, over 365 days. What it does NOT establish -
# and no synthetic series could - is how OFTEN a real Belgian year clears the
# threshold, which is the actual open question and the reason the shipped
# default is a product decision rather than a number this card picked.
# That is asserted per trade below, over 365 varied days including negative ones.
# HOW OFTEN a real Belgian year clears the threshold is the other question, it
# is the one the product owner's open decision turns on, and section 7B is the
# only place in this file that answers it.
def synth_day(n):
"""A synthetic all-in day-ahead day. Deterministic, seeded by day number.
@@ -920,12 +928,256 @@ check("the shipped threshold is strictly more selective than the loose one on "
"the same year - the parameter does what it says, at whatever value the "
"owner sets it to",
len(_sh_trades) < len(_trades))
check("AC4 IS REPORTED AS NOT MET rather than redefined: this file says so in "
"section 7's preamble, and there is no price history in the repository "
"to meet it with",
not any(f.endswith((".csv", ".json.gz"))
and "price" in f.lower()
for f in os.listdir(os.path.join(HERE, "scenarios"))))
check("...and the shipped default was NOT moved to make any of this look "
"better: app/arbitrage.py reads the threshold through price_option() and "
"names no number of its own",
"0.10" not in open(os.path.join(HERE, "app", "arbitrage.py"),
encoding="utf-8").read().split('"""', 2)[2])
# --------------------------------------------------------------------------- #
print()
print("7B. AC4 - twelve months of REAL EPEX BE prices, replayed")
# THE DATASET, AND WHAT IT IS NOT.
#
# `scenarios/epex-be-dayahead-2025-09-01_2026-09-01.json` - 8760 hourly Belgian
# day-ahead auction prices, 2025-09-01 to 2026-09-01 Brussels-local, fetched by
# `scenarios/fetch_epex_be.py` from Fraunhofer ISE's Energy-Charts API, which
# publishes the BE zone under CC BY 4.0 from Bundesnetzagentur | SMARD.de. The
# licence was checked BEFORE the data was committed, because the same API serves
# other zones under private-use-only terms; the script refuses to write a file
# whose per-response `license_info` stops saying CC BY 4.0. Provenance,
# attribution and the DST accounting live in the file's own `provenance` block
# and `fetch_epex_be.py --verify` re-derives every claim in it.
#
# ⚠️ THE FIGURES IN THAT FILE ARE RAW WHOLESALE, NOT ALL-IN, AND THIS SECTION IS
# WHERE THE DIFFERENCE IS MADE VISIBLE. §9.3's canonical point is all-in; the
# committed file says `"all_in": false` and carries the markup/tax assumption as
# NUMBERS TO APPLY rather than numbers already applied - TARIFF-02's
# double-counting trap, avoided by never baking the guess in. `_all_in()` below
# applies it in one place where a reader can watch it happen, and both replays
# are run, so exactly how much of the answer rests on the guess is reported
# rather than hidden.
import json # noqa: E402
from collections import defaultdict # noqa: E402
_DS_PATH = os.path.join(HERE, "scenarios",
"epex-be-dayahead-2025-09-01_2026-09-01.json")
_DS = derived(lambda: json.load(open(_DS_PATH, encoding="utf-8")))
_ASSUME = _DS["provenance"]["all_in_assumption"]
def _all_in(wholesale):
"""The stated assumption, applied HERE and nowhere in the committed data.
all_in = (wholesale + markup + network_and_levies) * (1 + vat), with the
coefficients read from the dataset's own provenance block rather than
retyped - a second copy is a second thing to disagree with the file.
"""
return round((wholesale + _ASSUME["markup_eur_per_kwh"]
+ _ASSUME["network_and_levies_eur_per_kwh"])
* (1.0 + _ASSUME["vat_fraction"]), 6)
def _days(price_of):
"""The dataset as Brussels calendar days of `PricePoint`s, in order.
⚠️ GROUPED BY THE LOCAL DATE IN THE `start` STRING, NOT BY SLICING 24 AT A
TIME. Two of these days are not 24 hours long, which is the whole DST check,
and a fixed stride would put October's 25th hour into the next day and
misalign every day after it.
"""
out = defaultdict(list)
for p in _DS["prices"]:
out[p["start"][:10]].append(
PricePoint(start=datetime.fromisoformat(p["start"]),
eur_per_kwh=price_of(p["eur_per_kwh"]),
provisional=bool(p["provisional"])))
return [out[k] for k in sorted(out)]
_AI_DAYS = derived(lambda: _days(_all_in))
_RAW_DAYS = derived(lambda: _days(lambda w: w))
# ---- the dataset is not degenerate, asserted BEFORE anything is graded ------
_lens = [len(d) for d in _AI_DAYS]
check(f"the replay has {len(_AI_DAYS)} Brussels days and {sum(_lens)} hourly "
f"points, which is a full year rather than a fortnight repeated",
len(_AI_DAYS) == 365 and sum(_lens) == 8760)
check(f"...and the day lengths are {sorted(set(_lens))}, so BOTH DST days are "
f"in it: an importer that produced 24 everywhere would have lost an hour "
f"in October and invented one in March",
sorted(set(_lens)) == [23, 24, 25]
and _lens.count(23) == 1 and _lens.count(25) == 1)
_dst_day = [d for d in _AI_DAYS if len(d) == 25][0]
check("...and the 25-hour day's two 02:00 points are two different INSTANTS at "
"two different offsets, which is what `at()` has to tell apart",
len({p.start for p in _dst_day}) == 25
and {p.start.utcoffset() for p in _dst_day}
== {timedelta(hours=1), timedelta(hours=2)})
_ai_spreads = sorted(round(max(p.eur_per_kwh for p in d)
- min(p.eur_per_kwh for p in d), 6)
for d in _AI_DAYS)
_raw_spreads = sorted(round(max(p.eur_per_kwh for p in d)
- min(p.eur_per_kwh for p in d), 6)
for d in _RAW_DAYS)
_req_shipped = required_spread(_shipped, NO_WEATHER)
_clears = [s for s in _ai_spreads if s > _req_shipped]
check(f"...and the year's daily spreads STRADDLE the shipped requirement of "
f"{_req_shipped:.4f} EUR/kWh - {len(_clears)} of {len(_ai_spreads)} days "
f"clear it - so the threshold is grading something. A year in which every "
f"day cleared, or none did, would grade nothing at any value",
0 < len(_clears) < len(_ai_spreads))
check(f"...and the spreads are not one repeated number: median "
f"{_ai_spreads[len(_ai_spreads) // 2]:.4f}, min {_ai_spreads[0]:.4f}, "
f"max {_ai_spreads[-1]:.4f}",
len(set(_ai_spreads)) > 300 and _ai_spreads[-1] > 4 * _ai_spreads[0])
# ---- the assumption's arithmetic, asserted rather than asserted-about -------
check("the ADDITIVE half of the all-in assumption cancels out of every daily "
"spread, so the trade counts below do not depend on the markup guess at "
"all - only on the VAT multiplier",
all(near(a, r * (1.0 + _ASSUME["vat_fraction"]), 1e-6)
for a, r in zip(_ai_spreads, _raw_spreads)))
# ---- ⚠️ THE DEGENERACY THAT KEEPS SECTION 7 ALIVE --------------------------
_raw_neg_h = sum(1 for d in _RAW_DAYS for p in d if p.eur_per_kwh < 0)
_ai_neg_h = sum(1 for d in _AI_DAYS for p in d if p.eur_per_kwh < 0)
_raw_neg_dear = sum(1 for d in _RAW_DAYS if max(p.eur_per_kwh for p in d) < 0)
_ai_neg_dear = sum(1 for d in _AI_DAYS if max(p.eur_per_kwh for p in d) < 0)
check(f"the real year DOES contain negative hours - {_raw_neg_h} of them at "
f"wholesale, {_ai_neg_h} still negative once the assumed markup and "
f"charges are added - so §17 is a live case on this market and not a "
f"hypothetical", _raw_neg_h > 100 and _ai_neg_h > 0)
check(f"BUT NOT ONE OF THE 365 DAYS HAS A NEGATIVE DEAREST HOUR "
f"({_raw_neg_dear} raw, {_ai_neg_dear} all-in), and the guard is on the "
f"hour being SOLD INTO. So a count of zero negative-price discharges over "
f"this year would be a count of something that could not happen, and "
f"section 7's synthetic year stays because it is the only thing here that "
f"puts a negative price where the strategy would sell",
_raw_neg_dear == 0 and _ai_neg_dear == 0)
def replay_real(days, threshold, unc, *, soc0=50.0, floor=20.0, ceiling=95.0):
"""The shipped `decide()` over the real year. Returns the ledger, nothing else.
One `TariffPlan` per Brussels day, so "the cheapest hour" means the cheapest
hour of that day - the same shape as section 7's replay, and the same shape
a site with a day-ahead feed actually sees.
"""
soc = soc0
trades, holdings, bad, neg_sells = [], [], [], 0
for day in days:
plan = TariffPlan(
PriceRead.series(day, PriceProduct.DAY_AHEAD, "generic_dayahead",
day[0].start), None, "provider")
open_buy = None
for p in day:
d = decide(plan, p.start + timedelta(minutes=30),
peak_guard_active=False, soc=soc,
min_spread_eur_kwh=threshold, uncertainty=unc,
charge_w=MAX_W, discharge_w=MAX_W,
soc_floor_pct=floor, soc_ceiling_pct=ceiling)
if d.action is None:
continue
if d.spread_eur_kwh is None or d.required_eur_kwh is None:
bad.append(p.start)
if d.action == CHARGE:
soc = min(ceiling, soc + STEP_PCT)
open_buy = (p.eur_per_kwh, d)
else:
if p.eur_per_kwh < 0:
neg_sells += 1
soc = max(floor, soc - STEP_PCT)
if open_buy is not None:
trades.append((open_buy[0], p.eur_per_kwh, d))
open_buy = None
else:
holdings.append(p.start)
return trades, holdings, bad, neg_sells
# ⚠️ AT THE SHIPPED THRESHOLD, UNTOUCHED, AND WIDENED BY FCST-03's NO-WEATHER
# FACTOR - the harsher of the two operating states, so the figure reported is
# the conservative one.
_r_trades, _r_held, _r_bad, _r_neg = derived(
lambda: replay_real(_AI_DAYS, _shipped, NO_WEATHER)) or (None,) * 4
_r_kwh = MAX_W / 1000.0
_r_net = round(sum((s - b - _shipped) * _r_kwh for b, s, _ in _r_trades), 2)
check(f"the real year traded at the SHIPPED threshold: {len(_r_trades)} "
f"completed round trips over 365 days of Belgian day-ahead prices - so "
f"the criterion is being graded on trades that happened, not on an empty "
f"ledger that is net-positive by vacuity", len(_r_trades) > 0)
check("...and every one of them carries the spread and the widened requirement "
"it cleared, so the ledger is auditable rather than a count", _r_bad == [])
check(f"...and it did NOT trade every day ({len(_r_trades)} round trips over "
f"365 days), so the threshold selected rather than waved everything "
f"through", len(_r_trades) < 365)
_r_losses = [(b, s) for b, s, _ in _r_trades if s - b < _shipped]
check(f"AC4: every completed round trip on the REAL year cleared the shipped "
f"{_shipped} EUR/kWh - {len(_r_losses)} exceptions of {len(_r_trades)}",
_r_losses == [])
_r_under = [d.spread_eur_kwh for _b, _s, d in _r_trades
if d.spread_eur_kwh < d.required_eur_kwh]
check(f"...and each cleared its own WIDENED requirement of "
f"{_req_shipped:.4f} EUR/kWh, which is the stricter statement §8.3 asks "
f"for", _r_under == [])
check(f"AC4 IS MET: twelve months of REAL replayed EPEX BE prices are "
f"NET-POSITIVE at the shipped default - {_r_net:+.2f} EUR of benefit net "
f"of the cycle cost over {len(_r_trades)} round trips of {_r_kwh:.1f} kWh "
f"each, accumulated from the trades the strategy actually chose",
_r_net > 0)
# ---- what the owner's open decision actually turns on -----------------------
# ⚠️ REPORTED, NOT SETTLED. `cycle_cost_threshold_eur` and FCST-03's 2.5/1.5 are
# the product owner's decision. DATA-02 supplies the input and changes nothing:
# every number below comes out of the same untouched `decide()`.
_r_calm = derived(lambda: replay_real(_AI_DAYS, _shipped, frozenset())[0])
_r_raw = derived(lambda: replay_real(_RAW_DAYS, _shipped, NO_WEATHER)[0])
# The threshold at which the MEDIAN day of this year clears its widened
# requirement. Derived from the observed spreads, not chosen.
_median_spread = _ai_spreads[len(_ai_spreads) // 2]
_suggested = round(_median_spread / widen_factor(NO_WEATHER), 4)
_r_sugg, _, _, _ = derived(
lambda: replay_real(_AI_DAYS, _suggested, NO_WEATHER)) or (None,) * 4
_sugg_net = round(sum((s - b - _suggested) * _r_kwh for b, s, _ in _r_sugg), 2)
print(f" [report] SHIPPED {_shipped} EUR/kWh, widened {widen_factor(NO_WEATHER)}x "
f"to {_req_shipped:.4f}: {len(_r_trades)} round trips, {_r_net:+.2f} EUR "
f"over the year. Unwidened (weather present): {len(_r_calm)} trips. On "
f"RAW wholesale instead of the assumed all-in: {len(_r_raw)} trips, so "
f"the markup/VAT guess moves the count by "
f"{len(_r_trades) - len(_r_raw):+d}.")
print(f" [report] the year's MEDIAN daily all-in spread is "
f"{_median_spread:.4f} EUR/kWh, so the threshold at which a median day "
f"clears its widened requirement is {_suggested} - and at that value the "
f"same year completes {len(_r_sugg)} round trips for {_sugg_net:+.2f} "
f"EUR. THE SHIPPED DEFAULT IS NOT CHANGED BY THIS CARD; both numbers are "
f"reported so the owner can decide between them.")
print(f" [report] and the four-developer finding stands unchanged: a STATIC "
f"day/night spread of ~0.0840 EUR/kWh still does not clear "
f"{_req_shipped:.4f}, so a static-tariff site never trades at the shipped "
f"default. What this year adds is that a DYNAMIC site does - on "
f"{len(_clears)} of its 365 days the spread was there to be taken.")
check("the loose threshold is strictly less selective than the shipped one on "
"the SAME real year - the parameter does what it says at whatever value "
"the owner sets, and neither run tuned anything",
len(_r_sugg) > len(_r_trades) > 0)
check(f"§17 held over the real year too ({_r_neg} discharges into a negative "
f"hour) - but see the degeneracy check above: on this data that zero is "
f"structural, and section 7's synthetic year is what actually grades the "
f"guard", _r_neg == 0)
check("the dataset carries its own provenance, states it is NOT all-in, and "
"keeps the markup/tax figures as an assumption rather than applying "
"them - the TARIFF-02 trap, in data form",
_DS["provenance"]["all_in"] is False
and "ASSUMPTION" in _ASSUME["status"]
and _DS["provenance"]["licence"].startswith("CC BY 4.0"))
# --------------------------------------------------------------------------- #