admin/goodwe-addon
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

TARIFF-09: ask the day-ahead feed for the horizon it was handed
checks / unit (pull_request) Successful in 7m10s
Details

Browse Source 
`generic_dayahead.fetch()` accepted `since` and `until` and requested the
bare `?bzn=BE`, which answers ONE Brussels day. Everything past today was
then filled from yesterday's profile by FCST-04, flagged provisional, and
§8.3 drops provisional hours from the trade set -- so the optimiser never
saw tomorrow as firm, at any hour of any day, and nothing reported it.

Measured on 4257e38 at 2026-09-10T09:00+02:00, 48 h horizon, live feed,
yesterday seeded through the shipped read path:

    60 firm points (15 whole hours) + 33 carried hours of 48 requested
    carried: 24 h on 2026-09-11, 9 h on 2026-09-12
    arbitrage(0.10): buy 11:45 and sell 19:45 -- BOTH on 2026-09-10
    firm local days = [2026-09-10]

The ceiling was structural: with a one-day window, at hour H a 48 h horizon
is at most 24-H firm and 24+H carried. At 23:00 that is 1 and 47, and the
remaining firm spread (0.0464 EUR/kWh) no longer clears the default 0.10
threshold, so arbitrage is refused outright. FCST-04's fill covers the
window until the ~13:00-14:00 CET auction; it was never meant to cover
every hour of every day.

The data exists and the endpoint serves it: verified 2026-09-10T07:45:49Z,
`?bzn=BE&start=2026-09-09&end=2026-09-10` returns 192 quarter-hours across
two local days. Committed unmodified beside TARIFF-08's captures, with the
URL, retrieval time and sha256 recorded and RE-DERIVED by the replay.

THE RANGE IS DERIVED ON THE BRUSSELS CALENDAR, not the UTC one. 22:30 UTC
in summer is already tomorrow in Belgium, so a UTC-derived start asks for
the day the controller has just left and drops the day it is standing in,
for the last one or two hours of every day. That is `_bdate()`'s lesson one
layer up with a worse consequence: `_bdate()` misfiles an hour, this never
fetches it. `until` is exclusive, so local midnight belongs to the day
before it; an inverted window collapses to one day rather than a backwards
range. Both ends are graded and both are driven red by mutation.

After, same code path and window, a feed serving both days: 39 firm hours
and 9 carried (was 15 and 33), firm across TWO Brussels days, and the
permitted cycle buys on one day and sells on the next.

`price_feed_url` IS THE EXACT URL AND THE PLUGIN APPENDS NOTHING. `start`/
`end` are this feed's spelling and the provider declares a second host
whose parameters are different, so merging would produce a request nobody
typed and the short horizon anyway. The cost is real, so it is not silent:
a WARNING on the shipped logger at fetch time names the consequence and the
start..end that would otherwise have been asked for. `dated_feed_url()`
refuses to append to a base that already carries a query string, so the
blind-append shape cannot come back by edit.
DECISIONS/TARIFF-09-the-range-and-the-override.md argues the alternative.

Before publication nothing regresses and it is graded, not assumed: at
09:46 CEST the two-day request came back byte-identical to the one-day
body, and that stays a `series` of what exists with FCST-04 filling the
rest. A `no_data` there would switch a working site's arbitrage off every
morning -- the mutation that adds that rule is red by name.

The plugin contract is still NOT enforced: three shipped plugins point at
endpoints nobody has verified, and a rule obliging them to pass a range
would be satisfied by an invented parameter name -- TARIFF-08's defect
moved from the response to the request. Which plugins consume the range is
pinned as a literal instead.

11 mutations, all red, no survivors; each applied to a fresh copy of the
tree rather than reverted, through tools/mutation_run.sh. One is +0 B (a
single price digit inside the committed capture) and the re-derived sha256
catches it. MUTATION.md habit 20 records why a fixture could never have
reached this: a captured body is the answer to a URL, and no property of a
body says which URL was asked for. All 29 suites green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
This commit is contained in:
Claude Opus 5
2026-09-10 10:35:12 +02:00
parent 4257e38e5a
commit 17b7b447c4
13 changed files with 980 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
goodwe_controller/test_price.py
+277 -6
View File
@@ -52,14 +52,16 @@ import json
import logging
import os
import sys
from datetime import datetime, timedelta, timezone
from datetime import date, datetime, timedelta, timezone
from textwrap import dedent
from app.price import (
ALL_IN, Absence, BANNED_IN_PLUGIN_CODE, Component, FrameworkOption,
MAX_PLAUSIBLE_EUR_KWH, code_leaks,
PRICE_OPTION_DEFAULTS, PROVIDERS, PluginOptions, PricePoint, PriceProduct,
PriceProvider, PriceRead, ProviderCapabilities, RawPoint, SourceUnavailable,
arbitrage_allowed, assemble, brussels, brussels_offset, check_egress,
arbitrage_allowed, assemble, brussels, brussels_date_range,
brussels_offset, check_egress, dated_feed_url,
price_option,
export_price_eur_kwh, load_prices, plan_is_provisional, register,
volatility_regime,
@@ -500,14 +502,14 @@ _REAL = {n: _capture(n) for n in _CAPTURES["files"]}
_bad_sha = sorted(n for n, b in _REAL.items()
if hashlib.sha256(b).hexdigest()
!= _CAPTURES["files"][n]["sha256"])
check(f"both captured bodies are the bytes the provenance file recorded "
f"{sorted(_REAL)}",
len(_REAL) == 2 and _bad_sha == [])
check(f"all three captured bodies are the bytes the provenance file "
f"recorded {sorted(_REAL)}",
len(_REAL) == 3 and _bad_sha == [])
# The document really is columnar, derived from the bytes rather than asserted:
# if it ever grows a `data` key this check says so before any parser does.
_shapes = {n: sorted(json.loads(b.decode("utf-8"))) for n, b in _REAL.items()}
check(f"...and both are COLUMNAR - parallel arrays, no 'data' key "
check(f"...and every one is COLUMNAR - parallel arrays, no 'data' key "
f"{sorted({tuple(v) for v in _shapes.values()})}",
all(v == ["deprecated", "license_info", "price", "unit", "unix_seconds"]
for v in _shapes.values()))
@@ -707,6 +709,275 @@ check(f"the networked plugins are {_networked}; {_replayed} is replayed from a "
"totalenergies_dynamic", "eneco_eddy"]))
# --------------------------------------------------------------------------- #
print()
print("2c. TARIFF-09 - the RANGE the caller asked for, on the Brussels "
"calendar")
#
# ⚠️ WHAT WAS WRONG, AND WHY NO CHECK IN THIS FILE COULD SEE IT.
# `generic_dayahead.fetch()` accepted `since` and `until` and requested the
# bare `https://api.energy-charts.info/price?bzn=BE`, which answers ONE
# Brussels day. Everything past today was then filled from yesterday's profile
# by FCST-04, flagged provisional, and §8.3 dropped all of it from the trade
# set - so the optimiser never saw tomorrow as firm, at any hour of any day.
# Measured on the merged tree at 2026-09-10T09:00+02:00 over a 48 h horizon:
# 15 firm whole hours and 33 carried, both ends of the only permitted cycle
# inside ONE calendar day; and at 23:00, 1 firm hour and 47 carried, where the
# firm spread no longer clears the round-trip cost at all.
#
# The reason section 2b could not catch it is that section 2b fed the parser
# BYTES and never looked at the REQUEST. `canned()` has always recorded the URL
# it was asked for and nothing asserted on it - so the transport knew the
# plugin was ignoring its arguments and no check ever asked. The checks below
# assert on the URL, and the two-day body they replay is a capture of what that
# URL really answers.
_TD = "energy-charts-be-price-2026-09-09_2026-09-10-twoday.json"
# ⚠️ NON-DEGENERACY FIRST (habit 1): the two-day capture must really cover TWO
# Brussels days and the one-day capture ONE, or "the range reached the feed"
# is graded against two fixtures that are the same shape.
_bdays = {n: sorted({brussels(datetime.fromtimestamp(t, UTC)).date().isoformat()
for t in json.loads(b.decode("utf-8"))["unix_seconds"]})
for n, b in _REAL.items()}
check(f"the captures span the local-day counts this section needs - "
f"{ {n: len(v) for n, v in _bdays.items()} } - or the range check has "
f"nothing to distinguish {_bdays[_TD]} from {_bdays[_QH]}",
len(_bdays[_TD]) == 2 and len(_bdays[_QH]) == 1
and len(_bdays[_H1]) == 1)
_npts = {n: len(json.loads(b.decode("utf-8"))["price"])
for n, b in _REAL.items()}
check(f"...and the two-day body carries 192 quarter-hours where the one-day "
f"body carries 96, which is the counterfactual this card is built on "
f"{_npts}",
len(json.loads(_REAL[_TD].decode("utf-8"))["price"]) == 192
and len(json.loads(_REAL[_QH].decode("utf-8"))["price"]) == 96)
# --- the URL the SHIPPED plugin asks for ----------------------------------- #
def asked_url(since, until, extra=None):
"""The URL `generic_dayahead` really requests for a window. Behaviour,
not source text: it comes back off the injected transport."""
t = canned(_REAL[_TD])
o = opts(price_provider="generic_dayahead", **(extra or {}))
run(load_prices(o, since, until, transport=t))
return t.calls[0][0]
# The expectation is built HERE, from the window, by different arithmetic than
# price.py uses: a literal string with the two dates spelled out.
_w0 = datetime.fromisoformat("2026-09-09T09:17:43+02:00")
check(f"a 48 h horizon from 09:17 local asks the feed for the three Brussels "
f"days it touches -> {asked_url(_w0, _w0 + timedelta(hours=48))!r}",
asked_url(_w0, _w0 + timedelta(hours=48))
== "https://api.energy-charts.info/price?bzn=BE"
"&start=2026-09-09&end=2026-09-11")
check("...and the shipped default no longer asks the bare URL that answers a "
"single day",
asked_url(_w0, _w0 + timedelta(hours=48))
!= "https://api.energy-charts.info/price?bzn=BE")
# ⚠️ THE BRUSSELS/UTC BOUNDARY, WHICH IS THE TRAP. `since`/`until` are
# INSTANTS and the API takes DATES. At 22:30 UTC in September Belgium is
# already on the NEXT calendar day, so a range derived from `since.date()` -
# or from `since.astimezone(utc).date()`, which is the same thing - asks for
# the day the controller has just left and drops the day it is standing in.
# This is `_bdate()`'s lesson one layer up: price_history.py MISFILES an hour
# when it gets this wrong; here the hours are never fetched at all.
_edge = datetime.fromisoformat("2026-09-10T22:30:00+00:00")
check(f"the precondition: {_edge.isoformat()} really is a different day in "
f"Brussels ({brussels(_edge).date().isoformat()}) than in UTC "
f"({_edge.date().isoformat()}), or the boundary check grades nothing",
brussels(_edge).date().isoformat() != _edge.date().isoformat())
check(f"...so the range starts on the BRUSSELS day, 2026-09-11, and not on "
f"the UTC day 2026-09-10 -> "
f"{asked_url(_edge, _edge + timedelta(hours=12))!r}",
asked_url(_edge, _edge + timedelta(hours=12))
== "https://api.energy-charts.info/price?bzn=BE"
"&start=2026-09-11&end=2026-09-11")
# ...and the SAME trap on the other end. `until` at 22:30 UTC is already
# 00:30 of the next Brussels day, so a UTC-derived END stops one day short and
# the last hours of the horizon are never fetched. Both ends of the range
# derivation are graded, or a mutation only has to get one of them right.
_edge2_a = datetime.fromisoformat("2026-09-09T12:00:00+00:00")
_edge2_b = datetime.fromisoformat("2026-09-10T22:30:00+00:00")
check(f"the precondition: {_edge2_b.isoformat()} is "
f"{brussels(_edge2_b).date().isoformat()} in Brussels and "
f"{_edge2_b.date().isoformat()} in UTC, so the two answers differ",
brussels(_edge2_b).date().isoformat() != _edge2_b.date().isoformat())
check(f"...so the range ENDS on the Brussels day 2026-09-11, not on the UTC "
f"day 2026-09-10 -> {asked_url(_edge2_a, _edge2_b)!r}",
asked_url(_edge2_a, _edge2_b)
== "https://api.energy-charts.info/price?bzn=BE"
"&start=2026-09-09&end=2026-09-11")
# ⚠️ `until` IS EXCLUSIVE, so local midnight belongs to the day BEFORE it. A
# 24 h horizon from local midnight is ONE day, not two.
_mid = datetime.fromisoformat("2026-09-09T00:00:00+02:00")
check(f"a horizon ending exactly at local midnight asks for one day, because "
f"`until` is exclusive -> {asked_url(_mid, _mid + timedelta(hours=24))!r}",
asked_url(_mid, _mid + timedelta(hours=24))
== "https://api.energy-charts.info/price?bzn=BE"
"&start=2026-09-09&end=2026-09-09")
check(f"...while one second past midnight is two days, so the rule is the "
f"exclusive END and not a fixed subtraction -> "
f"{asked_url(_mid, _mid + timedelta(hours=24, seconds=1))!r}",
asked_url(_mid, _mid + timedelta(hours=24, seconds=1))
== "https://api.energy-charts.info/price?bzn=BE"
"&start=2026-09-09&end=2026-09-10")
check(f"an empty or inverted window collapses to a single day rather than "
f"asking for a backwards range no feed has a defined answer for -> "
f"{asked_url(_mid, _mid - timedelta(hours=5))!r}",
asked_url(_mid, _mid - timedelta(hours=5))
== "https://api.energy-charts.info/price?bzn=BE"
"&start=2026-09-09&end=2026-09-09")
# The helper on its own, over the DST transitions, where a fixed-offset
# assumption and a correct one disagree.
_mar = datetime.fromisoformat("2026-03-29T00:30:00+00:00") # 02:30 CEST
_bdr = derived(lambda: brussels_date_range(_mar, _mar + timedelta(hours=1)))
check(f"brussels_date_range() is right across the spring transition, where "
f"00:30 UTC is already 02:30 local -> {_bdr}",
_bdr == (date(2026, 3, 29), date(2026, 3, 29)))
# --- the two-day body, end to end through the SHIPPED parser --------------- #
_two = load("generic_dayahead", transport=canned(_REAL[_TD]))
check(f"the shipped generic_dayahead parses the real two-day body "
f"{_two.words}", _two.kind == "series")
def _series_days(read):
return derived(lambda: sorted({brussels(p.start).date().isoformat()
for p in read.points}))
_two_days = _series_days(_two)
check(f"...and the series it produces spans TWO Brussels local days "
f"{_two_days}, where the one-day body's spans one "
f"{_series_days(_REPLAY[_QH])}",
_two_days == ["2026-09-09", "2026-09-10"]
and len(_two_days) == 2)
check(f"...one point per published interval, 192 of them, no decimation and "
f"no day dropped ({derived(lambda: len(_two.points))})",
derived(lambda: len(_two.points)) == 192)
# --- the SHORT ANSWER, which is normal and must stay a series -------------- #
#
# ⚠️ ASKING FOR TOMORROW BEFORE THE ~13:00-14:00 CET AUCTION RETURNS FEWER DAYS
# THAN REQUESTED, AND THAT IS NOT AN ERROR. Measured: at 2026-09-10T07:46:55Z
# the request `?bzn=BE&start=2026-09-10&end=2026-09-11` came back BYTE-IDENTICAL
# to the bare-URL capture - one day, 96 quarter-hours. A `no_data` there would
# switch a working site's arbitrage off every morning, so the requirement is
# that the short body stays a `series` of what EXISTS and FCST-04's fill covers
# the rest with flagged, provisional hours. The identity is re-derived from the
# bytes rather than restated.
_prepub = _CAPTURES["files"][_QH]["also_returned_verbatim_by"]
check(f"the pre-publication range request {_prepub['url']!r} returned the "
f"SAME BYTES as the bare default URL - one day where two were asked for",
hashlib.sha256(_REAL[_QH]).hexdigest()
== _CAPTURES["files"][_QH]["sha256"]
and "start=2026-09-10&end=2026-09-11" in _prepub["url"])
check(f"...and the shipped parser answers that short body with a SERIES of "
f"what exists, never no_data - {_REPLAY[_QH].words}",
_REPLAY[_QH].kind == "series"
and derived(lambda: len(_REPLAY[_QH].points)) == 96)
# ⚠️ THE COUNTS ARE TAKEN THROUGH `derived()` AND COMPARED ONLY WHEN BOTH ARE
# NUMBERS. A mutation that turns the short answer into `no_data` makes
# `.points` raise `Absence` by design, and `_Raised < _Raised` is a TypeError -
# which takes the file down with a traceback instead of reporting a FAIL. A
# check whose failure mode is a crash has no name attached (habit 5's shape,
# with an exception in place of a hang).
_n_short = derived(lambda: len(_REPLAY[_QH].points))
_n_two = derived(lambda: len(_two.points))
check(f"...which is strictly fewer points than the two-day body "
f"({_n_short} < {_n_two}), so 'short' is a real property of this "
f"stimulus and not a name for it",
isinstance(_n_short, int) and isinstance(_n_two, int)
and _n_short < _n_two)
# --- price_feed_url: the exact URL, and the cost said out loud ------------- #
#
# ⚠️ THE DECISION. An override is used VERBATIM and this plugin appends
# nothing to it. Merging our `start`/`end` into an operator's query string
# would mean re-serialising a URL they wrote, and `start`/`end` are
# energy-charts' spelling - and this provider also DECLARES a SECOND host,
# `web-api.tp.entsoe.eu`, whose date parameters are not those and are not
# verified here, so appending our vocabulary there yields two ignored
# parameters and the short horizon anyway. The cost is
# that an overriding site silently gets the horizon this card exists to fix,
# so it is NOT silent: a warning names the consequence at fetch time.
_ovr = "https://api.energy-charts.info/price?bzn=BE&somethingelse=1"
check(f"an override is requested EXACTLY as given, with nothing appended -> "
f"{asked_url(_w0, _w0 + timedelta(hours=48), {'price_feed_url': _ovr})!r}",
asked_url(_w0, _w0 + timedelta(hours=48),
{"price_feed_url": _ovr}) == _ovr)
_obuf = io.StringIO()
_oh = logging.StreamHandler(_obuf)
logging.getLogger("goodwe.price").addHandler(_oh)
asked_url(_w0, _w0 + timedelta(hours=48), {"price_feed_url": _ovr})
_ovr_log = _obuf.getvalue()
_obuf2 = io.StringIO()
_oh2 = logging.StreamHandler(_obuf2)
logging.getLogger("goodwe.price").removeHandler(_oh)
logging.getLogger("goodwe.price").addHandler(_oh2)
asked_url(_w0, _w0 + timedelta(hours=48))
_dflt_log = _obuf2.getvalue()
logging.getLogger("goodwe.price").removeHandler(_oh2)
check(f"...and the operator is TOLD what it costs them, at fetch time, naming "
f"the horizon consequence rather than only the substitution",
"price_feed_url" in _ovr_log and _ovr in _ovr_log
and "provisional" in _ovr_log and "2026-09-11" in _ovr_log)
# ⚠️ THE CONTROL. Without it, "warns about everything" passes the line above.
check("...while the DEFAULT path says nothing, so the warning is about the "
"override and not about fetching",
"price_feed_url is set" not in _dflt_log)
# The builder refuses to append to a base that already has a query string,
# which is the "URL nobody intended" this decision is meant to make
# unreachable. Driven, not read off the source.
check(f"dated_feed_url() refuses a base that already carries a query string, "
f"rather than producing a second '?' - "
f"{_reason(lambda: dated_feed_url('https://h/p?a=1', b='2'))!r}",
raises(lambda: dated_feed_url("https://h/p?a=1", b="2"),
SourceUnavailable))
check("...and a clean base still builds, so the refusal is the existing query "
"string and not the builder",
dated_feed_url("https://h/p", b="2") == "https://h/p?b=2")
# --- the plugin contract: who consumes the range, pinned ------------------- #
#
# ⚠️ THIS IS A SOURCE-SHAPE PIN AND IS NAMED AS ONE (habit 4). The claim is
# about the CODE - which plugins reference the range arguments they are handed
# - which is the same kind of claim `code_leaks()` makes, and it is not
# evidence that any feed parses. `generic_dayahead`'s BEHAVIOUR is graded above
# by the URL it really requests.
#
# It exists because `fetch(self, opts, since, until, transport)` promises every
# plugin the range and the promise is NOT enforced. Enforcing it would oblige
# the three plugins whose endpoints nobody has verified (TARIFF-07) to invent a
# date-parameter vocabulary - TARIFF-08's defect moved from the response to the
# request. So the split is pinned instead, and a sixth supplier has to change
# this line rather than inherit the silence.
def _uses_range(cls):
src = dedent(inspect.getsource(cls.fetch))
tree = ast.parse(src)
names = {n.id for n in ast.walk(tree) if isinstance(n, ast.Name)}
return bool(names & {"since", "until"})
_uses = sorted(pid for pid, cls in PROVIDERS.items() if _uses_range(cls))
check(f"the plugins that CONSUME their `since`/`until` are {_uses}; the rest "
f"take the range and ignore it, which is what TARIFF-09 fixed for "
f"generic_dayahead and what TARIFF-07 has to decide for the others",
_uses == ["generic_dayahead"])
check("...and `manual_csv` is honestly among the ones that do not: it reads a "
"whole file and the fill windows it downstream, so a range parameter "
"would have nowhere to go",
"manual_csv" in sorted(set(PROVIDERS) - set(_uses)))
# --------------------------------------------------------------------------- #
print()
print("3. all-in: energy + markup + taxes, added exactly once (AC1, trap 1)")