admin/goodwe-addon
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

TARIFF-08: the day-ahead feed is read in the shape it actually publishes
checks / unit (pull_request) Failing after 3s
Details

Browse Source 
`generic_dayahead` could not read one byte of `api.energy-charts.info`, and
never had been able to. `_points_from_json()` asks for `doc["data"]`, a list of
`{start, price_eur_mwh}` objects; that document does not exist. Fetched live on
2026-09-10 the feed returns COLUMNAR parallel arrays and no `data` key at all -
`unix_seconds` (96 UTC epoch seconds, 900 s apart), `price` (96 floats),
`unit: 'EUR / MWh'`. Driving the shipped parser against that body:
`SourceUnavailable: the feed carried no 'data' list`. It is the only shipped
provider whose endpoint anybody has reached, so dynamic pricing did not work.

AND §11.4 MADE IT SILENT, which is why it survived to release: a parse error
becomes no_data, becomes the static profile, becomes arbitrage off - the
CORRECT degradation, saying nothing about the cause. A site would have sat on
the fallback indefinitely.

AND THE SUITE COULD NOT SEE IT, because every fixture was built from the
parser's own assumption. test_price.py, test_price_history.py and
test_static_tariff.py each built `json.dumps({"data": [...]})`, a shape invented
here and published by nobody, so the checks graded the parser against the
parser's guess - a closed loop no coverage inside it could ever have escaped.
MUTATION.md habit 19 records it as the fourteenth instance of habit 4 and the
first where the fixture and the code under test shared a SHAPE, not a number.

WHAT CHANGED

* `_points_from_columnar_json()` - a SECOND function, not a sniff. A parser that
  decides which document it received is one that can decide wrong, and a
  wrong-but-parseable response then becomes a price. Feeding the row shape to
  `generic_dayahead` is now a graded no_data.
* Unequal arrays are refused in both directions, with the two lengths in the
  reason. `zip()` stops at the shorter list silently, and a truncated `price`
  array would have produced a real series missing the evening peak - traded on
  as complete, where no data at all switches arbitrage off.
* A unit that is not EUR/MWh is refused. The plugin converts with a fixed
  0.001, and `MAX_PLAUSIBLE_EUR_KWH` only bounds prices that are too LARGE.
* `unix_seconds` are read as UTC instants explicitly; §9.3's Brussels rendering
  stays in `assemble()` rather than being copied into the plugin, where a
  mutation could not tell it from a no-op (M4 proved exactly that on a first
  draft, and the line was removed rather than fenced).
* A published null is dropped and logged with a count; an all-null body is
  no_data named by this parser rather than by the boundary validator.

FIXTURES

Two unmodified response bodies under `scenarios/`, captured by hand on
2026-09-10: quarter-hourly (2026-09-10, 96 points at 900 s) and hourly
(2025-09-10, 24 at 3600 s - the same endpoint before Belgium's 15-minute MTU).
`energy-charts-be-price-captures.json` records the URL, retrieval time, unit,
licence-as-returned and a sha256 of each, and the replay RE-DERIVES the
checksums, so a fixture edited to make a check pass stops being a capture. Data
CC BY 4.0, Bundesnetzagentur | SMARD.de, via Energy-Charts (Fraunhofer ISE),
published without changes - the same BE-zone licence check DATA-02 established.

The surviving invented fixtures are renamed `synthetic_row_feed` at every call
site. `engie_dynamic`, `totalenergies_dynamic` and `eneco_eddy` are untouched
and have no capture: their URLs are unverified and withdrawing them is
TARIFF-07's open decision. Nothing should read their green checks as evidence
that those feeds parse.

MUTATIONS - 12, all red, no survivors. Purged tree, mounted `:ro`, `python3 -B`
(TEST-15). M1 drop the equal-length refusal; M2 make it one-sided; M3 drop the
unit check; M4 `assemble()` stops rendering at the Brussels offset; M5 read
`unix_seconds` as a naive local clock; M6 zero-fill a null instead of dropping
it; M7 drop the all-null refusal; M8 sniff for the row shape; M9 drop the
deprecated warning; M10 drop the dropped-nulls warning; M11 edit one price in
the committed capture; M12 point `generic_dayahead` back at the row parser -
red across all three price suites.

END TO END, LIVE, 2026-09-10: `resolve_tariff()` returns a `TariffPlan`,
`origin='provider'`, 96 quarter-hourly points, spacing [900] s, no decimation,
first point 2026-09-10T00:00:00+02:00 at 0.30608 EUR/kWh all-in. FCST-04's H1
fix was never theoretical. All 29 test files green.

STILL OPEN: `generic_dayahead` asks for no date range, so the default URL
returns one published day - a full 48 h horizon is reached only through §8.3's
carry-forward fill.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
This commit is contained in:
admin
2026-09-10 08:57:32 +02:00
co-authored by Claude Opus 5
parent 5059e28860
commit 6eba18968c
10 changed files with 773 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
goodwe_controller/test_price.py
+350 -18
View File
@@ -45,6 +45,7 @@ Read MUTATION.md before you report a survivor.
import ast
import asyncio
import hashlib
import inspect
import io
import json
@@ -148,6 +149,7 @@ def derived(fn):
UTC = timezone.utc
HERE = os.path.dirname(os.path.abspath(__file__))
# ⚠️ NOT the shipped 0.03 / 0.06. See this module's docstring: 0.06 is exactly
# 2 x 0.03, so under the shipped pair "the framework added the markup twice and
@@ -283,10 +285,31 @@ RETAIL_EUR_KWH = [0.1341, 0.0872, 0.2610] # EUR/kWh, markup already inside
EDDY_EUR_MWH = [141.0, 96.2, 268.4] # EUR/MWh, all-in
def json_feed(key, values, provisional_at=1):
"""A feed body. `provisional_at` makes ONE point provisional and the rest
not, so `plan_is_provisional` is graded against a mixed series - an
all-true or all-false fixture would return the same answer under any bug."""
def synthetic_row_feed(key, values, provisional_at=1):
"""An INVENTED row-per-point body. `provisional_at` makes ONE point
provisional and the rest not, so `plan_is_provisional` is graded against a
mixed series - an all-true or all-false fixture would return the same
answer under any bug.
⚠️ THE NAME SAYS `synthetic` BECAUSE TARIFF-08 IS WHAT HAPPENS WHEN IT DOES
NOT. This shape - `{"data": [{"start": ..., "price_eur_mwh": ...}]}` - was
invented by whoever wrote `_points_from_json()`, and until 2026-09-10 it
was the ONLY shape any price check in this repository had ever fed a
parser, `generic_dayahead` included. The public EPEX feed publishes
parallel arrays and no `data` key at all, so every live fetch that plugin
ever made raised `SourceUnavailable` while the suite stayed green: the
checks were grading the parser against the parser's own assumption. §11.4
then turned the failure into a quiet fall back to the static profile, so a
site would have sat on it forever.
The three plugins still fed from here - `engie_dynamic`,
`totalenergies_dynamic`, `eneco_eddy` - have NO captured body, because
their endpoints are unverified (TARIFF-07 is the open decision on whether
they are withdrawn). This fixture therefore grades that their DECLARATIONS
and the framework arithmetic are right; it grades NOTHING about what those
feeds publish, and it must not be read as evidence that they parse.
`generic_dayahead` is replayed from real bytes instead - section 2b.
"""
return json.dumps({"data": [
{"start": (T0 + timedelta(hours=i)).isoformat(),
key: v,
@@ -294,6 +317,37 @@ def json_feed(key, values, provisional_at=1):
for i, v in enumerate(values)]}).encode()
def columnar_feed(values, provisional_at=None, unit="EUR / MWh",
step_min=60, start=None, prices=None):
"""A body in the shape `api.energy-charts.info` really publishes.
Parallel arrays and no per-point object; `unix_seconds` are UTC epoch
seconds. `prices` overrides the price column outright, which is how the
unequal-length case below is built without hand-writing JSON.
⚠️ THERE IS NO PER-POINT `provisional` IN THIS SHAPE, so `provisional_at`
exists only to be REFUSED - the columnar document cannot carry the flag and
the plugin supplies it. It is a keyword the callers do not pass; the
parameter is here so that a future caller who reaches for it gets a
TypeError rather than a silently ignored argument.
"""
if provisional_at is not None:
raise TypeError(
"the columnar document has no per-point provisional flag; "
"`generic_dayahead` declares it in fetch(). See "
"_points_from_columnar_json().")
t0 = start if start is not None else T0
secs = [int((t0 + timedelta(minutes=step_min * i)).timestamp())
for i in range(len(values))]
doc = {"unix_seconds": secs,
"price": list(values) if prices is None else list(prices),
"unit": unit,
"license_info": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) "
"from Bundesnetzagentur | SMARD.de",
"deprecated": False}
return json.dumps(doc).encode()
CSV_BODY = ("start,eur_per_kwh,provisional\n"
"2026-01-15T00:00:00+01:00,0.1834,false\n"
"2026-01-15T01:00:00+01:00,-0.0210,true\n"
@@ -302,14 +356,18 @@ CSV_BODY = ("start,eur_per_kwh,provisional\n"
# The fixture, the transport that serves it, and - crucially - the ENERGY
# component of each point, computed here so the all-in expectation below is
# arithmetic this file did itself.
# ⚠️ `generic_dayahead` IS FED THE COLUMNAR SHAPE AND THE OTHER THREE ARE NOT,
# and the asymmetry is the honest one rather than an oversight: the columnar
# document is what the one endpoint anybody has ever reached actually returns.
FIXTURES = {
"generic_dayahead": (json_feed("price_eur_mwh", EPEX_EUR_MWH),
"generic_dayahead": (columnar_feed(EPEX_EUR_MWH),
[v / 1000 for v in EPEX_EUR_MWH]),
"engie_dynamic": (json_feed("price_eur_kwh", RETAIL_EUR_KWH),
"engie_dynamic": (synthetic_row_feed("price_eur_kwh", RETAIL_EUR_KWH),
list(RETAIL_EUR_KWH)),
"totalenergies_dynamic": (json_feed("price_eur_kwh", RETAIL_EUR_KWH),
"totalenergies_dynamic": (synthetic_row_feed("price_eur_kwh",
RETAIL_EUR_KWH),
list(RETAIL_EUR_KWH)),
"eneco_eddy": (json_feed("price_eur_mwh", EDDY_EUR_MWH),
"eneco_eddy": (synthetic_row_feed("price_eur_mwh", EDDY_EUR_MWH),
[v / 1000 for v in EDDY_EUR_MWH]),
"manual_csv": (None, [0.1834, -0.0210, 0.3115]),
}
@@ -396,6 +454,259 @@ check("the whole envelope round-trips through json.dumps",
for r in READS.values())) is True)
# --------------------------------------------------------------------------- #
print()
print("2b. TARIFF-08 - the SHIPPED parser, replayed against REAL captured "
"bodies")
#
# ⚠️ WHY THIS SECTION EXISTS, WHICH IS THE MORE IMPORTANT HALF OF THE TICKET.
# Until 2026-09-10 every price fixture in this repository - here, in
# test_price_history.py and in test_static_tariff.py - was a
# `{"data": [{"start": ..., "price_eur_mwh": ...}]}` document invented by
# whoever wrote `_points_from_json()`. `api.energy-charts.info` has never
# published that shape. So the suite validated the parser against the parser's
# own assumption: a closed loop that cannot notice the assumption is wrong, and
# it stayed green while `generic_dayahead` raised `SourceUnavailable` on every
# live fetch it ever made. §11.4 then made that silent - a parse error becomes
# `no_data` becomes the static profile with arbitrage off, which is the CORRECT
# degradation and says nothing at all about the cause.
#
# The only thing that breaks the loop is bytes nobody here wrote. Both fixtures
# below are unmodified response bodies, captured by hand with curl on
# 2026-09-10; `scenarios/energy-charts-be-price-captures.json` records the URL,
# the retrieval time, the licence as returned and a sha256 of each, and the
# checksums are RE-DERIVED here so a fixture edited to make a check pass stops
# being a capture and starts being a failure.
#
# ⚠️ THE OTHER THREE NETWORKED PLUGINS HAVE NO CAPTURE AND THIS FILE DOES NOT
# PRETEND OTHERWISE. `engie_dynamic`, `totalenergies_dynamic` and `eneco_eddy`
# point at endpoints nobody has verified; TARIFF-07 is the open decision on
# whether they are withdrawn. Their fixtures are named `synthetic_row_feed` so
# no reader can mistake them for evidence that those feeds parse, and the check
# below states, derived from the shipped `PROVIDERS` registry, exactly which
# plugins are replayed from real bytes and which are not.
_CAPTURES = json.load(open(
os.path.join(HERE, "scenarios", "energy-charts-be-price-captures.json"),
encoding="utf-8"))
def _capture(name):
with open(os.path.join(HERE, "scenarios", name), "rb") as fh:
return fh.read()
_REAL = {n: _capture(n) for n in _CAPTURES["files"]}
_bad_sha = sorted(n for n, b in _REAL.items()
if hashlib.sha256(b).hexdigest()
!= _CAPTURES["files"][n]["sha256"])
check(f"both captured bodies are the bytes the provenance file recorded "
f"{sorted(_REAL)}",
len(_REAL) == 2 and _bad_sha == [])
# The document really is columnar, derived from the bytes rather than asserted:
# if it ever grows a `data` key this check says so before any parser does.
_shapes = {n: sorted(json.loads(b.decode("utf-8"))) for n, b in _REAL.items()}
check(f"...and both are COLUMNAR - parallel arrays, no 'data' key "
f"{sorted({tuple(v) for v in _shapes.values()})}",
all(v == ["deprecated", "license_info", "price", "unit", "unix_seconds"]
for v in _shapes.values()))
check("...which is precisely the shape the shipped parser used to demand a "
"'data' list from",
all("data" not in v for v in _shapes.values()))
# ⚠️ CC BY 4.0 IS WHY THESE BYTES MAY BE IN THE REPOSITORY AT ALL. DATA-02
# established that the BE zone is redistributable with attribution; the feed
# restates the licence per response, so a capture whose own licence line has
# changed is a capture that must not stay committed.
_lic = {n: json.loads(b.decode("utf-8"))["license_info"] for n, b in _REAL.items()}
check(f"...and each carries the CC BY 4.0 line the provenance block cites",
all("CC BY 4.0" in v for v in _lic.values())
and all(_lic[n] == _CAPTURES["files"][n]["licence_as_returned"]
for n in _REAL))
_QH = "energy-charts-be-price-2026-09-10-quarterhourly.json"
_H1 = "energy-charts-be-price-2025-09-10-hourly.json"
# ⚠️ THE TWO GRANULARITIES ARE NOT DECORATION. Belgium moved to a 15-minute MTU
# on 2025-10-01 and the same endpoint answers 96 points at 900 s for a day after
# it and 24 at 3600 s for a day before it. A fixture at one spacing cannot grade
# whether anything assumes the other, so both are replayed and the spacings are
# DERIVED from the bodies rather than typed.
_spacing = {}
for _n, _b in _REAL.items():
_u = json.loads(_b.decode("utf-8"))["unix_seconds"]
_spacing[_n] = sorted({_u[i + 1] - _u[i] for i in range(len(_u) - 1)})
check(f"the two captures really differ in granularity - {_spacing[_QH]} s over "
f"{len(json.loads(_REAL[_QH])['price'])} points and {_spacing[_H1]} s "
f"over {len(json.loads(_REAL[_H1])['price'])} points - or replaying both "
f"grades nothing more than replaying one",
_spacing[_QH] == [900] and _spacing[_H1] == [3600])
_REPLAY = {n: load("generic_dayahead", transport=canned(b))
for n, b in _REAL.items()}
_not_series = sorted(n for n, r in _REPLAY.items() if r.kind != "series")
check(f"the SHIPPED generic_dayahead parses both real bodies "
f"{[r.words for r in _REPLAY.values()]}",
_not_series == [])
# The point count is the source's own interval count, taken from the document.
_counts = derived(lambda: {n: len(r.points) for n, r in _REPLAY.items()})
_want_counts = {n: len(json.loads(b.decode("utf-8"))["price"])
for n, b in _REAL.items()}
check(f"...emitting one point per published interval, no decimation "
f"{_counts}", _counts == _want_counts)
check(f"...which for the quarter-hourly body is 96 points and not 24 - the "
f"15-minute MTU is the shape of the real feed, not a hypothetical",
_want_counts[_QH] == 96 and _want_counts[_H1] == 24)
# §9.3: unix_seconds are UTC instants, RENDERED at the Brussels offset. Both
# captured days are inside CEST, so the correct first start is local midnight at
# +02:00. ⚠️ A NAIVE READING - `datetime.fromtimestamp(t)` with no tzinfo - is
# what this grades: it would produce the host's local wall clock, and on a UTC
# host that is 22:00 of the previous day. The expectation is computed here from
# the raw epoch second by different arithmetic than price.py uses.
_first_raw = {n: json.loads(b.decode("utf-8"))["unix_seconds"][0]
for n, b in _REAL.items()}
_first_utc = {n: datetime(1970, 1, 1, tzinfo=UTC) + timedelta(seconds=t)
for n, t in _first_raw.items()}
_first_got = derived(lambda: {n: r.points[0].start
for n, r in _REPLAY.items()})
check(f"every start is the UTC instant the feed published "
f"{ {n: v.isoformat() for n, v in _first_utc.items()} }",
isinstance(_first_got, dict)
and all(_first_got[n] == _first_utc[n] for n in _REAL))
check(f"...rendered at the Brussels offset, so a September day starts at "
f"00:00+02:00 and NOT at the 22:00 a naive local reading gives "
f"{ {n: v.isoformat() for n, v in (_first_got if isinstance(_first_got, dict) else {}).items()} }",
isinstance(_first_got, dict)
and all(v.utcoffset() == timedelta(hours=2) for v in _first_got.values())
and all((v.hour, v.minute) == (0, 0) for v in _first_got.values()))
# EUR/MWh -> all-in EUR/kWh, recomputed here from the raw column.
_allin_want = {n: [round(v / 1000 + MARKUP + TAXES, 6)
for v in json.loads(b.decode("utf-8"))["price"]]
for n, b in _REAL.items()}
_allin_got = derived(lambda: {n: [p.eur_per_kwh for p in r.points]
for n, r in _REPLAY.items()})
check(f"...and every price is the published EUR/MWh figure over 1000 plus "
f"markup and taxes (first: {[v[0] for v in _allin_want.values()]})",
_allin_got == _allin_want)
# ⚠️ NON-DEGENERACY. If the captured prices happened to be all equal, the
# arithmetic check above would pass under a parser that emitted the same number
# for every interval.
_raw_spread = {n: round(max(v) - min(v), 6)
for n, v in ((k, json.loads(b.decode("utf-8"))["price"])
for k, b in _REAL.items())}
check(f"...on bodies whose prices actually vary, or that check grades a "
f"constant {_raw_spread}",
all(v > 1.0 for v in _raw_spread.values()))
# --- the refusals, driven from the real body by breaking one thing ---------- #
def _edited(name, **fields):
"""The captured body with one field replaced. Everything else is real."""
doc = json.loads(_REAL[name].decode("utf-8"))
doc.update(fields)
return json.dumps(doc).encode()
_full = json.loads(_REAL[_QH].decode("utf-8"))
_short = load("generic_dayahead",
transport=canned(_edited(_QH, price=_full["price"][:-4])))
check(f"a `price` array 4 entries short of `unix_seconds` is no_data, NOT a "
f"series truncated to {len(_full['price']) - 4} points - the missing "
f"tail of a Belgian day is the evening peak, and a short series is "
f"traded on where no data at all switches arbitrage off (§11.4)",
_short.kind == "no_data")
check("...and the reason names the two lengths, so an operator can see it was "
"a length mismatch and not an outage",
_short.kind == "no_data"
and str(len(_full["unix_seconds"])) in _short.reason
and str(len(_full["price"]) - 4) in _short.reason)
# The other direction, so the check is not a one-sided `len(price) < len(time)`.
_long = load("generic_dayahead",
transport=canned(_edited(_QH, price=_full["price"] + [1.0])))
check("...and a `price` array one entry LONGER is refused too",
_long.kind == "no_data")
# ⚠️ THE CONTROL. Without it, "refuses everything" passes both checks above.
check("...while the untouched body is still a series, so the refusal is the "
"length and not the editing",
load("generic_dayahead",
transport=canned(_edited(_QH))).kind == "series")
_unit = load("generic_dayahead", transport=canned(_edited(_QH,
unit="EUR / kWh")))
check("a feed that changes its unit to EUR/kWh is no_data, not every price "
"silently divided by 1000 - MAX_PLAUSIBLE_EUR_KWH only bounds prices "
"that are too LARGE, so nothing downstream could catch it",
_unit.kind == "no_data")
_nulls = _full["price"][:]
for _i in (3, 17):
_nulls[_i] = None
_nulled = load("generic_dayahead", transport=canned(_edited(_QH,
price=_nulls)))
check(f"a published null is DROPPED, leaving a hole rather than a zero - a "
f"missing quarter is not a free quarter",
_nulled.kind == "series"
and derived(lambda: len(_nulled.points)) == len(_full["price"]) - 2)
check("...and the dropped instants are the ones the feed left null, not the "
"first two",
derived(lambda: sorted(
int(p.start.timestamp()) for p in _nulled.points))
== sorted(t for i, t in enumerate(_full["unix_seconds"])
if i not in (3, 17)))
_allnull = load("generic_dayahead",
transport=canned(_edited(_QH,
price=[None] * len(_full["price"]))))
check("...but a body with a price for nothing at all is no_data, not an empty "
"series", _allnull.kind == "no_data")
# ⚠️ WHICH RULE REFUSED IT, not merely that something did. `_validate_raw()`
# would also refuse an empty list, one layer further out and with a sentence
# that says nothing about the feed - so asserting only `no_data` here would
# grade the same under a parser that had lost this refusal entirely.
check(f"...refused by the COLUMNAR parser, naming what arrived, rather than "
f"reaching the boundary validator as a generic empty list - "
f"{_allnull.reason!r}",
_allnull.kind == "no_data"
and f"{len(_full['price'])} instants" in _allnull.reason
and "a price for none of them" in _allnull.reason)
# ⚠️ THE TWO WARNINGS ARE GRADED, NOT ASSUMED. §11.4 is exactly what makes a
# price defect quiet - a parse failure becomes no_data becomes the static
# profile, correctly and silently - so an unverified diagnostic is the first
# thing here to rot. Both are captured off the shipped logger.
_wbuf = io.StringIO()
_wh = logging.StreamHandler(_wbuf)
logging.getLogger("goodwe.price").addHandler(_wh)
load("generic_dayahead", transport=canned(_edited(_QH, price=_nulls)))
_dep = load("generic_dayahead", transport=canned(_edited(_QH,
deprecated=True)))
logging.getLogger("goodwe.price").removeHandler(_wh)
_warn = _wbuf.getvalue()
_n_null = len([v for v in _nulls if v is None])
check(f"the dropped nulls are reported with a count ({_n_null} of "
f"{len(_nulls)}), so a series with holes in it is visible to an operator "
f"rather than merely short",
f"{_n_null} of {len(_nulls)}" in _warn)
check("...and a feed announcing deprecated=true says so WHILE IT STILL WORKS - "
"the day it stops, §11.4 makes its death look exactly like a quiet one",
"deprecated" in _warn and _dep.kind == "series")
# Which plugins are replayed from real bytes, derived from the registry so a
# sixth supplier cannot be added without this line noticing.
_networked = sorted(pid for pid, cls in PROVIDERS.items()
if cls.capabilities().egress_hosts)
_replayed = ["generic_dayahead"]
check(f"the networked plugins are {_networked}; {_replayed} is replayed from a "
f"captured real body and {sorted(set(_networked) - set(_replayed))} have "
f"no capture because their endpoints are unverified (TARIFF-07)",
_networked == sorted(["generic_dayahead", "engie_dynamic",
"totalenergies_dynamic", "eneco_eddy"]))
# --------------------------------------------------------------------------- #
print()
print("3. all-in: energy + markup + taxes, added exactly once (AC1, trap 1)")
@@ -444,12 +755,29 @@ check(f"a negative day-ahead price passes through, all-in {_neg}",
_neg == [round(EPEX_EUR_MWH[1] / 1000 + MARKUP + TAXES, 6)])
# provisional is per point and comes from the feed, not from the series.
for pid, read in sorted(READS.items()):
#
# ⚠️ `generic_dayahead` IS EXCLUDED FROM THE "MIXED" REQUIREMENT, AND THE
# EXCLUSION IS THE POINT RATHER THAN A GAP. The columnar document the real feed
# publishes has no per-point provisional field - it is a settled auction result
# - so the plugin declares `False` for all of them. Demanding a mixed series
# from it would mean feeding it an invented body carrying an invented flag,
# which is exactly the closed loop TARIFF-08 exists to remove. It is graded on
# what its real body can grade instead, immediately below.
for pid in sorted(set(READS) - {"generic_dayahead"}):
read = READS[pid]
flags = derived(lambda r=read: [p.provisional for p in r.points])
check(f"{pid}: provisional is per point and mixed {flags}",
isinstance(flags, list) and any(flags) and not all(flags))
check(f"...so plan_is_provisional is derived from those points",
derived(lambda r=read: plan_is_provisional(r)) is True)
_gen_flags = derived(lambda: [p.provisional
for p in READS["generic_dayahead"].points])
check(f"generic_dayahead's settled auction comes out settled at every point "
f"{_gen_flags}",
isinstance(_gen_flags, list) and len(_gen_flags) == len(EPEX_EUR_MWH)
and not any(_gen_flags))
check("...so plan_is_provisional is False for it, derived from the same points",
derived(lambda: plan_is_provisional(READS["generic_dayahead"])) is False)
# ...and the same function says False when nothing is provisional. Without this
# arm, "always True" would pass every check above.
_settled = PriceRead.series(
@@ -759,7 +1087,7 @@ class _NewSupplier(PriceProvider):
_sixth = load("_test_sixth_supplier",
transport=canned(json_feed("price_eur_mwh", EPEX_EUR_MWH)))
transport=canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH)))
check("a plugin invented outside price.py resolves through the registry",
_sixth.kind == "series" and _sixth.provider == "_test_sixth_supplier")
_sixth_regime = derived(lambda: _sixth.regime)
@@ -894,7 +1222,14 @@ _failures = {
"the source is unreachable": load("generic_dayahead", transport=_boom),
"the body is not JSON": load("generic_dayahead", transport=canned(b"<html>")),
"the feed is empty": load("generic_dayahead",
transport=canned(b'{"data": []}')),
transport=canned(columnar_feed([]))),
# ⚠️ THE ROW SHAPE IS A FAILURE FOR THIS PLUGIN, NOT A SECOND DIALECT. If
# `_points_from_columnar_json()` ever grows a sniff for `data`, this case
# goes from no_data to a series and TARIFF-08's scope rule is broken by a
# check rather than by a review.
"the body is the OTHER shape": load(
"generic_dayahead",
transport=canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH))),
"the provider id is unknown": load("no_such_supplier"),
"no provider is configured": load("off"),
"the csv is missing": load("manual_csv",
@@ -1035,11 +1370,8 @@ check("the offset boundary is 01:00 UTC, not local midnight",
# A feed that publishes the duplicated hour must survive it. The 25-hour day is
# 25 strictly increasing instants; a plugin keyed on local hours would emit 24
# and be rejected by the boundary validator instead of silently mispricing.
_dst_feed = json.dumps({"data": [
{"start": (datetime(2026, 10, 24, 22, tzinfo=UTC)
+ timedelta(hours=i)).isoformat(),
"price_eur_mwh": 50.0 + i, "provisional": i == 24}
for i in range(25)]}).encode()
_dst_feed = columnar_feed([50.0 + i for i in range(25)],
start=datetime(2026, 10, 24, 22, tzinfo=UTC))
_dst_read = load("generic_dayahead", transport=canned(_dst_feed))
_dst_n = derived(lambda: len(_dst_read.points))
check(f"a 25-point autumn feed is accepted whole ({_dst_n} points)",
@@ -1164,7 +1496,7 @@ check("a userinfo prefix cannot smuggle a declared host past the check",
check("an operator-supplied URL to another supplier's host is refused",
load("generic_dayahead",
{"price_feed_url": "https://api.eneco.be/eddy/prices"},
transport=canned(json_feed("price_eur_mwh", EPEX_EUR_MWH))
transport=canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH))
).kind == "no_data")
_declared = {pid: cls.capabilities().egress_hosts
for pid, cls in PROVIDERS.items()}
@@ -1176,7 +1508,7 @@ check("the timeout the plugin passes to the transport is the configured one, "
"not a hardcoded default",
(lambda t: (load("generic_dayahead", {"price_timeout_s": 7},
transport=t), t.calls[0][1] == 7.0)[1])(
canned(json_feed("price_eur_mwh", EPEX_EUR_MWH))))
canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH))))
# --------------------------------------------------------------------------- #