TARIFF-08: the day-ahead feed is read in the shape it actually publishes, and the fixtures stop agreeing with the parser #76
@@ -15,6 +15,7 @@ if it has, it needs 0.10.0. Whoever deploys decides.
|
||||
| _this branch_ | FCST-04 - the carry-forward fill reaches the plan, and the day it carries survives a restart |
|
||||
| _this branch_ | FCST-05 - one byte stops stopping the add-on, and the write path gets the read path's standard |
|
||||
| _this branch_ | FCST-06 - "cannot persist" stops meaning "has no /data" and starts meaning what it says |
|
||||
| _this branch_ | TARIFF-08 - the day-ahead feed is read in the shape it actually publishes, and the fixtures stop agreeing with the parser |
|
||||
|
||||
### FCST-06 - the second way to be a store that cannot persist
|
||||
|
||||
@@ -72,6 +73,91 @@ EISDIR) and a volume that moves out from under an open store (`mkstemp`,
|
||||
ENOENT) - because the rule is enforced on the outcome and one cause would grade
|
||||
a coincidence. See `DECISIONS/FCST-06-cannot-persist-is-one-question.md`.
|
||||
|
||||
### TARIFF-08 - no network price provider had ever parsed a live response
|
||||
|
||||
**`generic_dayahead` could not read one byte of `api.energy-charts.info`, and
|
||||
had never been able to.** `_points_from_json()` asks for `doc["data"]`, a list
|
||||
of `{start, price_eur_mwh}` objects. That document does not exist. Fetched live
|
||||
on 2026-09-10, the feed returns **columnar** parallel arrays and no `data` key
|
||||
at all:
|
||||
|
||||
```
|
||||
top-level keys: ['deprecated', 'license_info', 'price', 'unit', 'unix_seconds']
|
||||
unix_seconds: 96 UTC epoch seconds, first 1788991200, spacing 900 s
|
||||
price: 96 floats, first 216.08
|
||||
unit: 'EUR / MWh'
|
||||
```
|
||||
|
||||
Driving the shipped parser against that body: `SourceUnavailable: the feed
|
||||
carried no 'data' list`. Since `generic_dayahead` is the only shipped provider
|
||||
whose endpoint anybody has reached, **dynamic pricing did not work at all**.
|
||||
|
||||
⚠️ **AND §11.4 MADE THAT SILENT, WHICH IS WHY IT LASTED.** A parse error becomes
|
||||
`no_data`, becomes the static profile, becomes arbitrage off - the *correct*
|
||||
degradation, and it says nothing whatsoever about the cause. A site would have
|
||||
sat on the fallback indefinitely while the status page reported only that the
|
||||
feed was down.
|
||||
|
||||
⚠️ **THE SUITE COULD NOT SEE IT BECAUSE EVERY FIXTURE WAS BUILT FROM THE
|
||||
PARSER'S OWN ASSUMPTION.** `test_price.py`, `test_price_history.py` and
|
||||
`test_static_tariff.py` each built `json.dumps({"data": [{"start": ...,
|
||||
"price_eur_mwh": ...}]})` - a shape invented here, published by nobody. The
|
||||
checks graded the parser against the parser's guess: a closed loop that cannot
|
||||
notice the guess is wrong, and no amount of coverage inside it would have. The
|
||||
loop is broken by **bytes nobody in this repository wrote**: two unmodified
|
||||
response bodies are committed under `scenarios/`, one quarter-hourly
|
||||
(2026-09-10, 96 points at 900 s) and one hourly (2025-09-10, 24 points at
|
||||
3600 s - the same endpoint before Belgium's 15-minute MTU), with URL, retrieval
|
||||
time, licence-as-returned and a sha256 recorded beside them in
|
||||
`energy-charts-be-price-captures.json` and **re-derived** by the replay, so a
|
||||
fixture edited to make a check pass stops being a capture. The remaining
|
||||
invented fixtures are renamed `synthetic_row_feed` at every call site.
|
||||
|
||||
**The columnar document is read by its own function, `_points_from_columnar_json()`.**
|
||||
Not by teaching the row parser to sniff for both: a parser that decides which
|
||||
document it received is a parser that can decide wrong, and a wrong-but-
|
||||
parseable response then becomes a price. Two shapes, two functions, and each
|
||||
plugin names the one its feed publishes. Feeding the row shape to
|
||||
`generic_dayahead` is now a graded `no_data`, so the sniff cannot come back
|
||||
quietly.
|
||||
|
||||
⚠️ **UNEQUAL ARRAYS ARE REFUSED RATHER THAN ZIPPED.** `zip()` stops at the
|
||||
shorter list and says nothing, so a truncated `price` array would have produced
|
||||
a **real price series with its tail missing** - and the tail of a Belgian day is
|
||||
the evening peak. That is worse than no series: `no_data` switches arbitrage
|
||||
off, whereas a short series is traded on as though complete, and nothing
|
||||
downstream can tell them apart. Refused in both directions, with the two lengths
|
||||
in the reason.
|
||||
|
||||
⚠️ **A UNIT CHANGE IS REFUSED FOR A REASON `_validate_raw()` CANNOT COVER.** The
|
||||
plugin converts with a fixed `0.001`; if the feed ever published EUR/kWh under
|
||||
the same keys, every price would come out a **thousand times too small** and
|
||||
`MAX_PLAUSIBLE_EUR_KWH` would not fire, because it only bounds prices that are
|
||||
too large. DATA-02's fetcher already refused on this field; the shipped path
|
||||
does now too.
|
||||
|
||||
**`unix_seconds` are UTC instants, said so explicitly**
|
||||
(`datetime.fromtimestamp(t, timezone.utc)`); §9.3's rendering at the Brussels
|
||||
offset stays where it already lived, in `assemble()`, rather than being copied
|
||||
into the plugin where a mutation could not tell it from a no-op. A published
|
||||
`null` is dropped and logged with a count - a missing quarter is not a free
|
||||
quarter - while an all-null body is `no_data` named by this parser rather than
|
||||
by the boundary validator.
|
||||
|
||||
**End to end against the live feed, 2026-09-10:** `resolve_tariff()` returns a
|
||||
`TariffPlan` with `origin='provider'`, **96 quarter-hourly points**, spacing
|
||||
`[900]` s and no decimation, first point `2026-09-10T00:00:00+02:00` at 0.30608
|
||||
EUR/kWh all-in. FCST-04's H1 fix was never theoretical - 900 s is the shape of
|
||||
the real feed.
|
||||
|
||||
⚠️ **WHAT THIS DOES NOT FIX.** `engie_dynamic`, `totalenergies_dynamic` and
|
||||
`eneco_eddy` are untouched: their URLs are unverified, no real body can be
|
||||
captured for them, and withdrawing them is TARIFF-07's open decision. Their
|
||||
fixtures are now named `synthetic_row_feed` so nothing reads them as evidence
|
||||
that those feeds parse. And `generic_dayahead` still ignores `since`/`until`
|
||||
and asks for no date range, so the default URL returns one published day - a
|
||||
full 48 h horizon is reached only through §8.3's carry-forward fill.
|
||||
|
||||
### FCST-05 - what FCST-04's own contract still did not hold
|
||||
|
||||
**A single non-UTF-8 byte in `/data/price_history.json` stopped the add-on from
|
||||
|
||||
@@ -1395,6 +1395,27 @@ flagged as high-volatility, which widens arbitrage thresholds once the
|
||||
optimiser lands. That comes from the plugin's own declaration, not from its
|
||||
name.
|
||||
|
||||
**`generic_dayahead` and the shape of the public feed.** It reads Fraunhofer
|
||||
ISE's Energy-Charts API (`https://api.energy-charts.info/price?bzn=BE`, no key
|
||||
and no registration), which publishes a *columnar* document — parallel
|
||||
`unix_seconds` and `price` arrays, in EUR/MWh, with the unit and the licence
|
||||
stated in the response. Two things follow that are worth knowing before you
|
||||
point `price_feed_url` somewhere else:
|
||||
|
||||
* The timestamps are **UTC instants**. They are rendered at the Brussels offset
|
||||
that applies to each one, so the hour that happens twice every October is two
|
||||
points rather than one overwritten one.
|
||||
* Belgium moved to a **15-minute market time unit** on 2025-10-01, and the feed
|
||||
followed. A published day is **96 points, not 24**. Nothing in the add-on
|
||||
assumes either spacing, and nothing aggregates the quarters away.
|
||||
|
||||
If the arrays ever come back different lengths, or the unit stops saying
|
||||
EUR/MWh, the fetch is refused outright rather than trusted in part: a price
|
||||
series missing its last hours is a series missing the evening peak, and it
|
||||
would be traded on as though it were complete. A refused fetch is the §11.4
|
||||
row — static profile, provisional plan, no arbitrage — and the reason is in the
|
||||
log.
|
||||
|
||||
**`manual_csv` — the offline fallback.** It needs no internet, no supplier
|
||||
account and no HTTP stack at all, and it is the supported way to run this
|
||||
add-on price-aware on a site with no outbound connectivity. Write a file and
|
||||
|
||||
@@ -695,6 +695,45 @@ Cheap habits fall out of it, and all of them are now used in this repository:
|
||||
that two of it cannot silently overwrite each other's work. A fixed path
|
||||
in a tool that edits source is a shared mutable global with a filename.
|
||||
|
||||
19. ⚠️ **A FIXTURE IN A SHAPE NOBODY PUBLISHES GRADES THE PARSER AGAINST THE
|
||||
PARSER — FOURTEENTH INSTANCE OF HABIT 4, AND THE FIRST WHERE THE FIXTURE
|
||||
AND THE CODE UNDER TEST SHARED A *SHAPE* RATHER THAN A NUMBER.** TARIFF-08.
|
||||
`price.py`'s `_points_from_json()` read `doc["data"]` as a list of
|
||||
`{start, price_eur_mwh}` objects. `api.energy-charts.info` publishes
|
||||
parallel `unix_seconds` / `price` arrays and has no `data` key at all, so
|
||||
`generic_dayahead` raised `SourceUnavailable` on **every live fetch it ever
|
||||
made** — and the suite was green, because all three price suites built
|
||||
their bodies with `json.dumps({"data": [...]})`. The fixture was derived
|
||||
from the parser's guess, so no mutation of the parser and no amount of
|
||||
coverage inside that loop could ever have reached the failure. §11.4 then
|
||||
turned it into a *correct* silent degradation: parse error -> `no_data` ->
|
||||
static profile -> arbitrage off, with nothing naming the cause.
|
||||
|
||||
Habit 2 with a network on the end of it: **the reference has to be data the
|
||||
code under test did not shape.** For a parser that means BYTES, captured
|
||||
from the source and committed unmodified, with the URL, the retrieval time
|
||||
and a checksum beside them - `scenarios/energy-charts-be-price-*.json` and
|
||||
its `-captures.json` provenance, re-derived by the replay so an edited
|
||||
fixture stops being a capture. Two granularities, because the same endpoint
|
||||
answers hourly before 2025-10-01 and quarter-hourly after it and a fixture
|
||||
at one spacing cannot grade an assumption about the other.
|
||||
|
||||
⚠️ **AND THE SURVIVING SYNTHETIC FIXTURES ARE NAMED FOR WHAT THEY ARE.**
|
||||
Three shipped plugins point at endpoints nobody has verified (TARIFF-07),
|
||||
so no capture exists for them; their fixture builder is called
|
||||
`synthetic_row_feed` at every call site, because the next reader's default
|
||||
assumption will otherwise be that a green check means the feed parses.
|
||||
|
||||
⚠️ **ONE OF THIS CARD'S OWN NEW LINES WAS DECORATION AND THE BATTERY SAID
|
||||
SO.** The columnar parser first rendered its instants with `brussels()`
|
||||
before returning them. `assemble()` already renders every plugin's points,
|
||||
so removing the parser's call changed nothing observable - **M4 came back a
|
||||
survivor on a `:ro`, purged tree**, which is not the direction stale
|
||||
bytecode fabricates. The line was removed rather than fenced with a check:
|
||||
a second copy of a rule that already has an owner is indistinguishable from
|
||||
a no-op, and the mutation that proves the rendering is real belongs on
|
||||
`assemble()`, where it is now aimed.
|
||||
|
||||
And the mutation-table consequence: **a variant is not covered because a
|
||||
near-miss of it is.** DEV-02's table listed the *actuator* variant of the
|
||||
per-unit claim defect (`device`/`supervised` filed without `inv`), which the
|
||||
|
||||
@@ -979,9 +979,166 @@ def _points_from_json(body, price_key, scale, provisional_default):
|
||||
return out
|
||||
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# TARIFF-08: the COLUMNAR document, which is what the public EPEX feed really
|
||||
# publishes.
|
||||
#
|
||||
# ⚠️ THIS IS A SECOND SHAPE, READ BY A SECOND FUNCTION ON PURPOSE.
|
||||
# `_points_from_json()` above reads a row-per-point document: `{"data": [{...},
|
||||
# {...}]}`. `https://api.energy-charts.info/price?bzn=BE` does not publish that
|
||||
# and never did - it publishes PARALLEL ARRAYS:
|
||||
#
|
||||
# {"unix_seconds": [1788991200, 1788992100, ...], <- UTC epoch seconds
|
||||
# "price": [216.08, 209.90, ...], <- one per instant
|
||||
# "unit": "EUR / MWh", "license_info": "...", "deprecated": false}
|
||||
#
|
||||
# The shipped parser was pointed at that URL and asked for `doc["data"]`, so
|
||||
# every fetch ended `SourceUnavailable: the feed carried no 'data' list` ->
|
||||
# §11.4 -> static profile, arbitrage off, and nothing in the status page said
|
||||
# why. Verified live 2026-09-10 against both fixtures under `scenarios/`.
|
||||
#
|
||||
# ⚠️ AND IT IS NOT ONE FUNCTION THAT SNIFFS FOR BOTH. A parser that decides
|
||||
# which document it got is a parser that can decide wrong, and a wrong-but-
|
||||
# parseable response then becomes a price. Two shapes, two functions, and each
|
||||
# plugin names the one its own feed publishes.
|
||||
#
|
||||
# The columnar shape was already known in this repository: DATA-02's
|
||||
# `scenarios/fetch_epex_be.py` has read `zip(d["unix_seconds"], d["price"])`
|
||||
# since 2026-08-31. Only the SHIPPED path never learned it, because every
|
||||
# fixture in the suite was built from the invented row shape - the parser was
|
||||
# graded against its own assumption. See test_price.py section 2b.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
COLUMNAR_TIME_KEY = "unix_seconds"
|
||||
COLUMNAR_PRICE_KEY = "price"
|
||||
COLUMNAR_UNIT_KEY = "unit"
|
||||
|
||||
|
||||
def _columnar_unit(text) -> str:
|
||||
"""`"EUR / MWh"` -> `"eur/mwh"`. Spacing and case are not information."""
|
||||
return "".join(str(text).split()).lower()
|
||||
|
||||
|
||||
def _points_from_columnar_json(body, expect_unit, scale,
|
||||
provisional_default):
|
||||
"""Parse a COLUMNAR price document: parallel `unix_seconds` / `price`.
|
||||
|
||||
Defects this refuses, each of which has a way of becoming a number rather
|
||||
than an error:
|
||||
|
||||
* ⚠️ UNEQUAL ARRAYS ARE `SourceUnavailable`, NOT A SHORT SERIES. `zip()`
|
||||
stops at the shorter array and says nothing, so a `price` list that
|
||||
arrived truncated would produce a REAL price series with its tail
|
||||
missing - and the tail of a Belgian day is the evening peak. A series
|
||||
that is missing the expensive hours is worse than no series: §11.4's
|
||||
no-data row switches arbitrage OFF, whereas a truncated series is
|
||||
traded on. Nothing downstream can tell the two apart, so the refusal
|
||||
has to happen here.
|
||||
* ⚠️ A UNIT THAT IS NOT THE DECLARED ONE IS REFUSED. `scale` is the
|
||||
caller's standing assumption about the unit; if the feed ever
|
||||
publishes EUR/kWh under the same keys, every price silently comes out
|
||||
a THOUSAND times too small. `_validate_raw()` cannot catch that -
|
||||
`MAX_PLAUSIBLE_EUR_KWH` only bounds prices that are too LARGE.
|
||||
DATA-02's fetcher refuses on the same field for the same reason.
|
||||
* ⚠️ `unix_seconds` ARE UTC INSTANTS, SAID SO EXPLICITLY.
|
||||
`datetime.fromtimestamp(t, timezone.utc)`. Without that second
|
||||
argument CPython returns a NAIVE datetime in the HOST's zone, and a
|
||||
naive datetime is a wall clock reading plus an assumption about whose
|
||||
clock it was. `brussels()` and `_validate_raw()` both refuse one, so
|
||||
the mistake surfaces as a refused fetch rather than as a mispriced
|
||||
day - but they refuse ALL naive datetimes and can say nothing about
|
||||
whether this one was the right instant. That part is decided here.
|
||||
|
||||
§9.3's rendering AT THE BRUSSELS OFFSET is `assemble()`'s, once, for
|
||||
every plugin - `brussels(p.start)` there is the single line that does
|
||||
it, and calling it here as well would be a second copy of a rule that
|
||||
already has an owner, indistinguishable from a no-op under mutation.
|
||||
A plugin's job is the right INSTANT; the framework's is the rendering.
|
||||
|
||||
A `null` price is DROPPED and logged, not fatal, and that is a different
|
||||
judgement from the one above: an unequal pair of arrays means the pairing
|
||||
of instant to price is unreliable, whereas a null means the source stated
|
||||
it published no price for that one instant. A dropped instant leaves a
|
||||
hole, and `_Series.at()` already answers a hole with `Absence` - §11.4 for
|
||||
that hour. `scenarios/fetch_epex_be.py` dropped 0 nulls over the 12 months
|
||||
to 2026-08-31, so this is a shape the source allows rather than one it uses.
|
||||
"""
|
||||
try:
|
||||
text = body.decode("utf-8") if isinstance(body, (bytes, bytearray)) \
|
||||
else str(body)
|
||||
doc = json.loads(text)
|
||||
except (ValueError, UnicodeDecodeError) as e:
|
||||
raise SourceUnavailable(f"the feed did not parse as JSON: {e}")
|
||||
if not isinstance(doc, dict):
|
||||
raise SourceUnavailable(
|
||||
f"the feed is a {type(doc).__name__}, not a columnar object with "
|
||||
f"{COLUMNAR_TIME_KEY!r} and {COLUMNAR_PRICE_KEY!r} arrays")
|
||||
times = doc.get(COLUMNAR_TIME_KEY)
|
||||
prices = doc.get(COLUMNAR_PRICE_KEY)
|
||||
for key, col in ((COLUMNAR_TIME_KEY, times), (COLUMNAR_PRICE_KEY, prices)):
|
||||
if not isinstance(col, list):
|
||||
raise SourceUnavailable(
|
||||
f"the feed carried no {key!r} array; this provider reads the "
|
||||
f"columnar shape (parallel {COLUMNAR_TIME_KEY}/"
|
||||
f"{COLUMNAR_PRICE_KEY} arrays), and a row-per-point document "
|
||||
f"is a different feed, not this one with a field missing")
|
||||
if len(times) != len(prices):
|
||||
raise SourceUnavailable(
|
||||
f"{COLUMNAR_TIME_KEY} has {len(times)} entries and "
|
||||
f"{COLUMNAR_PRICE_KEY} has {len(prices)}; refusing to zip them. "
|
||||
f"The short array would truncate the series silently, and a price "
|
||||
f"series missing its last hours is one missing the evening peak - "
|
||||
f"traded on as if complete, where no data at all would have "
|
||||
f"switched arbitrage off (§11.4)")
|
||||
unit = _columnar_unit(doc.get(COLUMNAR_UNIT_KEY, ""))
|
||||
if unit != _columnar_unit(expect_unit):
|
||||
raise SourceUnavailable(
|
||||
f"the feed reports {doc.get(COLUMNAR_UNIT_KEY)!r}, not "
|
||||
f"{expect_unit!r}. This provider converts with a fixed factor of "
|
||||
f"{scale}, so a changed unit is a factor of 1000 on every price "
|
||||
f"with nothing downstream able to notice it")
|
||||
if doc.get("deprecated"):
|
||||
# Not a refusal: today's prices are still prices. But §11.4 makes a
|
||||
# dead feed look exactly like a quiet one, so the warning that the
|
||||
# source is going away has to be emitted while it still works.
|
||||
_LOG.warning("the price feed reports deprecated=%r; it still answers, "
|
||||
"but this endpoint is scheduled to stop",
|
||||
doc.get("deprecated"))
|
||||
out = []
|
||||
dropped = []
|
||||
for i, (t, v) in enumerate(zip(times, prices)):
|
||||
if v is None:
|
||||
dropped.append(i)
|
||||
continue
|
||||
if isinstance(t, bool) or not isinstance(t, (int, float)) or \
|
||||
not math.isfinite(t):
|
||||
raise SourceUnavailable(
|
||||
f"{COLUMNAR_TIME_KEY}[{i}] is {t!r}, not epoch seconds")
|
||||
if isinstance(v, bool) or not isinstance(v, (int, float)) or \
|
||||
not math.isfinite(v):
|
||||
raise SourceUnavailable(
|
||||
f"{COLUMNAR_PRICE_KEY}[{i}] is {v!r}, which is not a price")
|
||||
start = datetime.fromtimestamp(float(t), timezone.utc)
|
||||
out.append(RawPoint(start=start, eur_per_kwh=float(v) * scale,
|
||||
provisional=bool(provisional_default)))
|
||||
if dropped:
|
||||
_LOG.warning("the feed published no price for %d of %d instants "
|
||||
"(index %s); those instants are left OUT of the series, "
|
||||
"never zero-filled - a missing hour is not a free hour",
|
||||
len(dropped), len(times),
|
||||
", ".join(str(i) for i in dropped[:5])
|
||||
+ (", ..." if len(dropped) > 5 else ""))
|
||||
if not out:
|
||||
raise SourceUnavailable(
|
||||
f"the feed carried {len(times)} instants and a price for none of "
|
||||
f"them")
|
||||
return out
|
||||
|
||||
|
||||
@register
|
||||
class GenericDayAhead(PriceProvider):
|
||||
"""Public EPEX BE day-ahead feed, in EUR/MWh.
|
||||
"""Public EPEX BE day-ahead feed, COLUMNAR, in EUR/MWh.
|
||||
|
||||
SOURCES: energy
|
||||
The framework ADDS: markup, taxes
|
||||
@@ -990,6 +1147,19 @@ class GenericDayAhead(PriceProvider):
|
||||
is the wholesale clearing price and nothing else, so both user-entered
|
||||
components have to be added to it - and because this plugin cannot read
|
||||
either number, it cannot be the one that adds them twice.
|
||||
|
||||
⚠️ IT READS THE COLUMNAR SHAPE, AND SAYS SO IN ITS OWN `fetch()`. Until
|
||||
TARIFF-08 it called `_points_from_json()` and asked
|
||||
`api.energy-charts.info` for a `"data"` list it has never published, so
|
||||
every fetch this provider ever made ended in `SourceUnavailable` and §11.4
|
||||
turned that into a silent fall back to the static profile. See
|
||||
`_points_from_columnar_json()` for the shape and for why there are two
|
||||
parsers rather than one that guesses.
|
||||
|
||||
⚠️ THE FEED IS QUARTER-HOURLY SINCE BELGIUM'S 15-MINUTE MTU (2025-10-01):
|
||||
96 points a day at 900 s, not 24 at 3600 s. Nothing here assumes either -
|
||||
the instants come from the document - and both granularities are replayed
|
||||
from real captured bodies in test_price.py section 2b.
|
||||
"""
|
||||
|
||||
id = "generic_dayahead"
|
||||
@@ -1006,7 +1176,9 @@ class GenericDayAhead(PriceProvider):
|
||||
"https://api.energy-charts.info/price?bzn=BE"
|
||||
check_egress(url, self.capabilities())
|
||||
body = await transport(url, float(opts.get("price_timeout_s")))
|
||||
return _points_from_json(body, "price_eur_mwh", 0.001, False)
|
||||
return _points_from_columnar_json(body, expect_unit="EUR / MWh",
|
||||
scale=0.001,
|
||||
provisional_default=False)
|
||||
|
||||
|
||||
@register
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"license_info":"CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de","unix_seconds":[1757455200,1757458800,1757462400,1757466000,1757469600,1757473200,1757476800,1757480400,1757484000,1757487600,1757491200,1757494800,1757498400,1757502000,1757505600,1757509200,1757512800,1757516400,1757520000,1757523600,1757527200,1757530800,1757534400,1757538000],"price":[73.0,61.31,64.32,62.49,67.0,64.48,94.52,149.26,132.43,91.61,84.8,79.61,69.23,66.75,68.91,69.31,80.29,66.91,105.2,161.05,116.73,84.2,64.1,42.41],"unit":"EUR / MWh","deprecated":false}
|
||||
@@ -0,0 +1 @@
|
||||
{"license_info":"CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de","unix_seconds":[1788991200,1788992100,1788993000,1788993900,1788994800,1788995700,1788996600,1788997500,1788998400,1788999300,1789000200,1789001100,1789002000,1789002900,1789003800,1789004700,1789005600,1789006500,1789007400,1789008300,1789009200,1789010100,1789011000,1789011900,1789012800,1789013700,1789014600,1789015500,1789016400,1789017300,1789018200,1789019100,1789020000,1789020900,1789021800,1789022700,1789023600,1789024500,1789025400,1789026300,1789027200,1789028100,1789029000,1789029900,1789030800,1789031700,1789032600,1789033500,1789034400,1789035300,1789036200,1789037100,1789038000,1789038900,1789039800,1789040700,1789041600,1789042500,1789043400,1789044300,1789045200,1789046100,1789047000,1789047900,1789048800,1789049700,1789050600,1789051500,1789052400,1789053300,1789054200,1789055100,1789056000,1789056900,1789057800,1789058700,1789059600,1789060500,1789061400,1789062300,1789063200,1789064100,1789065000,1789065900,1789066800,1789067700,1789068600,1789069500,1789070400,1789071300,1789072200,1789073100,1789074000,1789074900,1789075800,1789076700],"price":[216.08,216.08,214.92,200.0,223.37,201.43,196.32,190.0,199.53,194.83,180.53,172.23,179.88,175.77,167.1,173.89,163.29,165.99,171.41,178.73,170.27,177.37,187.71,201.07,193.62,207.61,218.27,226.77,227.17,234.05,236.99,229.94,248.93,227.5,216.08,195.67,215.99,195.01,177.1,155.09,174.91,152.71,139.59,123.83,140.77,123.19,114.96,94.5,102.77,96.63,94.57,116.34,114.63,126.82,129.89,123.95,105.0,119.11,121.0,136.79,123.41,135.01,135.0,156.68,146.88,152.16,162.02,181.78,154.76,184.28,194.3,215.76,197.23,234.15,253.03,296.58,345.32,389.16,482.99,559.0,551.11,446.41,358.97,295.9,271.39,268.06,241.43,238.01,239.7,226.41,223.05,211.56,218.21,210.61,197.78,171.79],"unit":"EUR / MWh","deprecated":false}
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"card": "TARIFF-08",
|
||||
"what": "UNMODIFIED response bodies from the public EPEX BE day-ahead feed, captured so the SHIPPED parser can be replayed against what the source really publishes",
|
||||
"why": "every price fixture in the suite was built from a `{\"data\": [{\"start\": ..., \"price_eur_mwh\": ...}]}` document that this feed has never published. The suite therefore graded `_points_from_json` against its own assumption - a closed loop that could not notice the assumption was wrong - and §11.4 turned the resulting failure into a silent fall back to the static profile. A captured real body is what breaks that loop, so these files are bytes, not a reconstruction: nothing here is pretty-printed, re-keyed or trimmed.",
|
||||
"source": "Energy-Charts API (Fraunhofer ISE), api.energy-charts.info",
|
||||
"upstream": "Bundesnetzagentur | SMARD.de",
|
||||
"licence": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)",
|
||||
"licence_note": "the BE bidding zone is in the API's CC BY 4.0 list, so redistribution is permitted with attribution. DATA-02 established this against the API's own OpenAPI document before committing `epex-be-dayahead-2025-09-01_2026-09-01.json`; both bodies below carry the same `license_info` string, recorded per file and asserted by the replay test.",
|
||||
"attribution": "Data licensed CC BY 4.0 by Bundesnetzagentur | SMARD.de, retrieved via the Energy-Charts API (Fraunhofer ISE), https://energy-charts.info. Published without changes.",
|
||||
"retrieved_utc": "2026-09-10T06:35:00+00:00",
|
||||
"retrieved_with": "curl, by hand; no code in `app/` fetches these",
|
||||
"shape": "COLUMNAR: parallel `unix_seconds` (UTC epoch seconds) and `price` arrays, plus `unit`, `license_info` and `deprecated`. There is no `data` key and no per-point object.",
|
||||
"replayed_by": "test_price.py section 2b",
|
||||
"files": {
|
||||
"energy-charts-be-price-2026-09-10-quarterhourly.json": {
|
||||
"url": "https://api.energy-charts.info/price?bzn=BE",
|
||||
"note": "the SHIPPED default URL of `generic_dayahead`, verbatim and with no parameters, so the replay drives the bytes the add-on's own request produces",
|
||||
"covers_local_day": "2026-09-10 (Europe/Brussels)",
|
||||
"granularity_s": 900,
|
||||
"intervals": 96,
|
||||
"bytes": 1877,
|
||||
"sha256": "176309d0020ad208320797385ef68b6a98ac248218539416cddf6579dd060487",
|
||||
"unit_as_returned": "EUR / MWh",
|
||||
"licence_as_returned": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de",
|
||||
"deprecated_as_returned": false,
|
||||
"top_level_keys": [
|
||||
"deprecated",
|
||||
"license_info",
|
||||
"price",
|
||||
"unit",
|
||||
"unix_seconds"
|
||||
]
|
||||
},
|
||||
"energy-charts-be-price-2025-09-10-hourly.json": {
|
||||
"url": "https://api.energy-charts.info/price?bzn=BE&start=2025-09-10&end=2025-09-10",
|
||||
"note": "a day BEFORE Belgium's 15-minute MTU (2025-10-01), so the same endpoint answers HOURLY. Kept because the parser must not assume either spacing, and a fixture at only one granularity cannot grade that",
|
||||
"covers_local_day": "2025-09-10 (Europe/Brussels)",
|
||||
"granularity_s": 3600,
|
||||
"intervals": 24,
|
||||
"bytes": 572,
|
||||
"sha256": "03fe41690810df224611f4016899f856457f67a3c12c7654da17b1331ff2eff4",
|
||||
"unit_as_returned": "EUR / MWh",
|
||||
"licence_as_returned": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de",
|
||||
"deprecated_as_returned": false,
|
||||
"top_level_keys": [
|
||||
"deprecated",
|
||||
"license_info",
|
||||
"price",
|
||||
"unit",
|
||||
"unix_seconds"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
+350
-18
@@ -45,6 +45,7 @@ Read MUTATION.md before you report a survivor.
|
||||
|
||||
import ast
|
||||
import asyncio
|
||||
import hashlib
|
||||
import inspect
|
||||
import io
|
||||
import json
|
||||
@@ -148,6 +149,7 @@ def derived(fn):
|
||||
|
||||
|
||||
UTC = timezone.utc
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
# ⚠️ NOT the shipped 0.03 / 0.06. See this module's docstring: 0.06 is exactly
|
||||
# 2 x 0.03, so under the shipped pair "the framework added the markup twice and
|
||||
@@ -283,10 +285,31 @@ RETAIL_EUR_KWH = [0.1341, 0.0872, 0.2610] # EUR/kWh, markup already inside
|
||||
EDDY_EUR_MWH = [141.0, 96.2, 268.4] # EUR/MWh, all-in
|
||||
|
||||
|
||||
def json_feed(key, values, provisional_at=1):
|
||||
"""A feed body. `provisional_at` makes ONE point provisional and the rest
|
||||
not, so `plan_is_provisional` is graded against a mixed series - an
|
||||
all-true or all-false fixture would return the same answer under any bug."""
|
||||
def synthetic_row_feed(key, values, provisional_at=1):
|
||||
"""An INVENTED row-per-point body. `provisional_at` makes ONE point
|
||||
provisional and the rest not, so `plan_is_provisional` is graded against a
|
||||
mixed series - an all-true or all-false fixture would return the same
|
||||
answer under any bug.
|
||||
|
||||
⚠️ THE NAME SAYS `synthetic` BECAUSE TARIFF-08 IS WHAT HAPPENS WHEN IT DOES
|
||||
NOT. This shape - `{"data": [{"start": ..., "price_eur_mwh": ...}]}` - was
|
||||
invented by whoever wrote `_points_from_json()`, and until 2026-09-10 it
|
||||
was the ONLY shape any price check in this repository had ever fed a
|
||||
parser, `generic_dayahead` included. The public EPEX feed publishes
|
||||
parallel arrays and no `data` key at all, so every live fetch that plugin
|
||||
ever made raised `SourceUnavailable` while the suite stayed green: the
|
||||
checks were grading the parser against the parser's own assumption. §11.4
|
||||
then turned the failure into a quiet fall back to the static profile, so a
|
||||
site would have sat on it forever.
|
||||
|
||||
The three plugins still fed from here - `engie_dynamic`,
|
||||
`totalenergies_dynamic`, `eneco_eddy` - have NO captured body, because
|
||||
their endpoints are unverified (TARIFF-07 is the open decision on whether
|
||||
they are withdrawn). This fixture therefore grades that their DECLARATIONS
|
||||
and the framework arithmetic are right; it grades NOTHING about what those
|
||||
feeds publish, and it must not be read as evidence that they parse.
|
||||
`generic_dayahead` is replayed from real bytes instead - section 2b.
|
||||
"""
|
||||
return json.dumps({"data": [
|
||||
{"start": (T0 + timedelta(hours=i)).isoformat(),
|
||||
key: v,
|
||||
@@ -294,6 +317,37 @@ def json_feed(key, values, provisional_at=1):
|
||||
for i, v in enumerate(values)]}).encode()
|
||||
|
||||
|
||||
def columnar_feed(values, provisional_at=None, unit="EUR / MWh",
|
||||
step_min=60, start=None, prices=None):
|
||||
"""A body in the shape `api.energy-charts.info` really publishes.
|
||||
|
||||
Parallel arrays and no per-point object; `unix_seconds` are UTC epoch
|
||||
seconds. `prices` overrides the price column outright, which is how the
|
||||
unequal-length case below is built without hand-writing JSON.
|
||||
|
||||
⚠️ THERE IS NO PER-POINT `provisional` IN THIS SHAPE, so `provisional_at`
|
||||
exists only to be REFUSED - the columnar document cannot carry the flag and
|
||||
the plugin supplies it. It is a keyword the callers do not pass; the
|
||||
parameter is here so that a future caller who reaches for it gets a
|
||||
TypeError rather than a silently ignored argument.
|
||||
"""
|
||||
if provisional_at is not None:
|
||||
raise TypeError(
|
||||
"the columnar document has no per-point provisional flag; "
|
||||
"`generic_dayahead` declares it in fetch(). See "
|
||||
"_points_from_columnar_json().")
|
||||
t0 = start if start is not None else T0
|
||||
secs = [int((t0 + timedelta(minutes=step_min * i)).timestamp())
|
||||
for i in range(len(values))]
|
||||
doc = {"unix_seconds": secs,
|
||||
"price": list(values) if prices is None else list(prices),
|
||||
"unit": unit,
|
||||
"license_info": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) "
|
||||
"from Bundesnetzagentur | SMARD.de",
|
||||
"deprecated": False}
|
||||
return json.dumps(doc).encode()
|
||||
|
||||
|
||||
CSV_BODY = ("start,eur_per_kwh,provisional\n"
|
||||
"2026-01-15T00:00:00+01:00,0.1834,false\n"
|
||||
"2026-01-15T01:00:00+01:00,-0.0210,true\n"
|
||||
@@ -302,14 +356,18 @@ CSV_BODY = ("start,eur_per_kwh,provisional\n"
|
||||
# The fixture, the transport that serves it, and - crucially - the ENERGY
|
||||
# component of each point, computed here so the all-in expectation below is
|
||||
# arithmetic this file did itself.
|
||||
# ⚠️ `generic_dayahead` IS FED THE COLUMNAR SHAPE AND THE OTHER THREE ARE NOT,
|
||||
# and the asymmetry is the honest one rather than an oversight: the columnar
|
||||
# document is what the one endpoint anybody has ever reached actually returns.
|
||||
FIXTURES = {
|
||||
"generic_dayahead": (json_feed("price_eur_mwh", EPEX_EUR_MWH),
|
||||
"generic_dayahead": (columnar_feed(EPEX_EUR_MWH),
|
||||
[v / 1000 for v in EPEX_EUR_MWH]),
|
||||
"engie_dynamic": (json_feed("price_eur_kwh", RETAIL_EUR_KWH),
|
||||
"engie_dynamic": (synthetic_row_feed("price_eur_kwh", RETAIL_EUR_KWH),
|
||||
list(RETAIL_EUR_KWH)),
|
||||
"totalenergies_dynamic": (json_feed("price_eur_kwh", RETAIL_EUR_KWH),
|
||||
"totalenergies_dynamic": (synthetic_row_feed("price_eur_kwh",
|
||||
RETAIL_EUR_KWH),
|
||||
list(RETAIL_EUR_KWH)),
|
||||
"eneco_eddy": (json_feed("price_eur_mwh", EDDY_EUR_MWH),
|
||||
"eneco_eddy": (synthetic_row_feed("price_eur_mwh", EDDY_EUR_MWH),
|
||||
[v / 1000 for v in EDDY_EUR_MWH]),
|
||||
"manual_csv": (None, [0.1834, -0.0210, 0.3115]),
|
||||
}
|
||||
@@ -396,6 +454,259 @@ check("the whole envelope round-trips through json.dumps",
|
||||
for r in READS.values())) is True)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("2b. TARIFF-08 - the SHIPPED parser, replayed against REAL captured "
|
||||
"bodies")
|
||||
#
|
||||
# ⚠️ WHY THIS SECTION EXISTS, WHICH IS THE MORE IMPORTANT HALF OF THE TICKET.
|
||||
# Until 2026-09-10 every price fixture in this repository - here, in
|
||||
# test_price_history.py and in test_static_tariff.py - was a
|
||||
# `{"data": [{"start": ..., "price_eur_mwh": ...}]}` document invented by
|
||||
# whoever wrote `_points_from_json()`. `api.energy-charts.info` has never
|
||||
# published that shape. So the suite validated the parser against the parser's
|
||||
# own assumption: a closed loop that cannot notice the assumption is wrong, and
|
||||
# it stayed green while `generic_dayahead` raised `SourceUnavailable` on every
|
||||
# live fetch it ever made. §11.4 then made that silent - a parse error becomes
|
||||
# `no_data` becomes the static profile with arbitrage off, which is the CORRECT
|
||||
# degradation and says nothing at all about the cause.
|
||||
#
|
||||
# The only thing that breaks the loop is bytes nobody here wrote. Both fixtures
|
||||
# below are unmodified response bodies, captured by hand with curl on
|
||||
# 2026-09-10; `scenarios/energy-charts-be-price-captures.json` records the URL,
|
||||
# the retrieval time, the licence as returned and a sha256 of each, and the
|
||||
# checksums are RE-DERIVED here so a fixture edited to make a check pass stops
|
||||
# being a capture and starts being a failure.
|
||||
#
|
||||
# ⚠️ THE OTHER THREE NETWORKED PLUGINS HAVE NO CAPTURE AND THIS FILE DOES NOT
|
||||
# PRETEND OTHERWISE. `engie_dynamic`, `totalenergies_dynamic` and `eneco_eddy`
|
||||
# point at endpoints nobody has verified; TARIFF-07 is the open decision on
|
||||
# whether they are withdrawn. Their fixtures are named `synthetic_row_feed` so
|
||||
# no reader can mistake them for evidence that those feeds parse, and the check
|
||||
# below states, derived from the shipped `PROVIDERS` registry, exactly which
|
||||
# plugins are replayed from real bytes and which are not.
|
||||
|
||||
_CAPTURES = json.load(open(
|
||||
os.path.join(HERE, "scenarios", "energy-charts-be-price-captures.json"),
|
||||
encoding="utf-8"))
|
||||
|
||||
|
||||
def _capture(name):
|
||||
with open(os.path.join(HERE, "scenarios", name), "rb") as fh:
|
||||
return fh.read()
|
||||
|
||||
|
||||
_REAL = {n: _capture(n) for n in _CAPTURES["files"]}
|
||||
_bad_sha = sorted(n for n, b in _REAL.items()
|
||||
if hashlib.sha256(b).hexdigest()
|
||||
!= _CAPTURES["files"][n]["sha256"])
|
||||
check(f"both captured bodies are the bytes the provenance file recorded "
|
||||
f"{sorted(_REAL)}",
|
||||
len(_REAL) == 2 and _bad_sha == [])
|
||||
|
||||
# The document really is columnar, derived from the bytes rather than asserted:
|
||||
# if it ever grows a `data` key this check says so before any parser does.
|
||||
_shapes = {n: sorted(json.loads(b.decode("utf-8"))) for n, b in _REAL.items()}
|
||||
check(f"...and both are COLUMNAR - parallel arrays, no 'data' key "
|
||||
f"{sorted({tuple(v) for v in _shapes.values()})}",
|
||||
all(v == ["deprecated", "license_info", "price", "unit", "unix_seconds"]
|
||||
for v in _shapes.values()))
|
||||
check("...which is precisely the shape the shipped parser used to demand a "
|
||||
"'data' list from",
|
||||
all("data" not in v for v in _shapes.values()))
|
||||
|
||||
# ⚠️ CC BY 4.0 IS WHY THESE BYTES MAY BE IN THE REPOSITORY AT ALL. DATA-02
|
||||
# established that the BE zone is redistributable with attribution; the feed
|
||||
# restates the licence per response, so a capture whose own licence line has
|
||||
# changed is a capture that must not stay committed.
|
||||
_lic = {n: json.loads(b.decode("utf-8"))["license_info"] for n, b in _REAL.items()}
|
||||
check(f"...and each carries the CC BY 4.0 line the provenance block cites",
|
||||
all("CC BY 4.0" in v for v in _lic.values())
|
||||
and all(_lic[n] == _CAPTURES["files"][n]["licence_as_returned"]
|
||||
for n in _REAL))
|
||||
|
||||
_QH = "energy-charts-be-price-2026-09-10-quarterhourly.json"
|
||||
_H1 = "energy-charts-be-price-2025-09-10-hourly.json"
|
||||
|
||||
# ⚠️ THE TWO GRANULARITIES ARE NOT DECORATION. Belgium moved to a 15-minute MTU
|
||||
# on 2025-10-01 and the same endpoint answers 96 points at 900 s for a day after
|
||||
# it and 24 at 3600 s for a day before it. A fixture at one spacing cannot grade
|
||||
# whether anything assumes the other, so both are replayed and the spacings are
|
||||
# DERIVED from the bodies rather than typed.
|
||||
_spacing = {}
|
||||
for _n, _b in _REAL.items():
|
||||
_u = json.loads(_b.decode("utf-8"))["unix_seconds"]
|
||||
_spacing[_n] = sorted({_u[i + 1] - _u[i] for i in range(len(_u) - 1)})
|
||||
check(f"the two captures really differ in granularity - {_spacing[_QH]} s over "
|
||||
f"{len(json.loads(_REAL[_QH])['price'])} points and {_spacing[_H1]} s "
|
||||
f"over {len(json.loads(_REAL[_H1])['price'])} points - or replaying both "
|
||||
f"grades nothing more than replaying one",
|
||||
_spacing[_QH] == [900] and _spacing[_H1] == [3600])
|
||||
|
||||
_REPLAY = {n: load("generic_dayahead", transport=canned(b))
|
||||
for n, b in _REAL.items()}
|
||||
_not_series = sorted(n for n, r in _REPLAY.items() if r.kind != "series")
|
||||
check(f"the SHIPPED generic_dayahead parses both real bodies "
|
||||
f"{[r.words for r in _REPLAY.values()]}",
|
||||
_not_series == [])
|
||||
|
||||
# The point count is the source's own interval count, taken from the document.
|
||||
_counts = derived(lambda: {n: len(r.points) for n, r in _REPLAY.items()})
|
||||
_want_counts = {n: len(json.loads(b.decode("utf-8"))["price"])
|
||||
for n, b in _REAL.items()}
|
||||
check(f"...emitting one point per published interval, no decimation "
|
||||
f"{_counts}", _counts == _want_counts)
|
||||
check(f"...which for the quarter-hourly body is 96 points and not 24 - the "
|
||||
f"15-minute MTU is the shape of the real feed, not a hypothetical",
|
||||
_want_counts[_QH] == 96 and _want_counts[_H1] == 24)
|
||||
|
||||
# §9.3: unix_seconds are UTC instants, RENDERED at the Brussels offset. Both
|
||||
# captured days are inside CEST, so the correct first start is local midnight at
|
||||
# +02:00. ⚠️ A NAIVE READING - `datetime.fromtimestamp(t)` with no tzinfo - is
|
||||
# what this grades: it would produce the host's local wall clock, and on a UTC
|
||||
# host that is 22:00 of the previous day. The expectation is computed here from
|
||||
# the raw epoch second by different arithmetic than price.py uses.
|
||||
_first_raw = {n: json.loads(b.decode("utf-8"))["unix_seconds"][0]
|
||||
for n, b in _REAL.items()}
|
||||
_first_utc = {n: datetime(1970, 1, 1, tzinfo=UTC) + timedelta(seconds=t)
|
||||
for n, t in _first_raw.items()}
|
||||
_first_got = derived(lambda: {n: r.points[0].start
|
||||
for n, r in _REPLAY.items()})
|
||||
check(f"every start is the UTC instant the feed published "
|
||||
f"{ {n: v.isoformat() for n, v in _first_utc.items()} }",
|
||||
isinstance(_first_got, dict)
|
||||
and all(_first_got[n] == _first_utc[n] for n in _REAL))
|
||||
check(f"...rendered at the Brussels offset, so a September day starts at "
|
||||
f"00:00+02:00 and NOT at the 22:00 a naive local reading gives "
|
||||
f"{ {n: v.isoformat() for n, v in (_first_got if isinstance(_first_got, dict) else {}).items()} }",
|
||||
isinstance(_first_got, dict)
|
||||
and all(v.utcoffset() == timedelta(hours=2) for v in _first_got.values())
|
||||
and all((v.hour, v.minute) == (0, 0) for v in _first_got.values()))
|
||||
|
||||
# EUR/MWh -> all-in EUR/kWh, recomputed here from the raw column.
|
||||
_allin_want = {n: [round(v / 1000 + MARKUP + TAXES, 6)
|
||||
for v in json.loads(b.decode("utf-8"))["price"]]
|
||||
for n, b in _REAL.items()}
|
||||
_allin_got = derived(lambda: {n: [p.eur_per_kwh for p in r.points]
|
||||
for n, r in _REPLAY.items()})
|
||||
check(f"...and every price is the published EUR/MWh figure over 1000 plus "
|
||||
f"markup and taxes (first: {[v[0] for v in _allin_want.values()]})",
|
||||
_allin_got == _allin_want)
|
||||
# ⚠️ NON-DEGENERACY. If the captured prices happened to be all equal, the
|
||||
# arithmetic check above would pass under a parser that emitted the same number
|
||||
# for every interval.
|
||||
_raw_spread = {n: round(max(v) - min(v), 6)
|
||||
for n, v in ((k, json.loads(b.decode("utf-8"))["price"])
|
||||
for k, b in _REAL.items())}
|
||||
check(f"...on bodies whose prices actually vary, or that check grades a "
|
||||
f"constant {_raw_spread}",
|
||||
all(v > 1.0 for v in _raw_spread.values()))
|
||||
|
||||
|
||||
# --- the refusals, driven from the real body by breaking one thing ---------- #
|
||||
|
||||
def _edited(name, **fields):
|
||||
"""The captured body with one field replaced. Everything else is real."""
|
||||
doc = json.loads(_REAL[name].decode("utf-8"))
|
||||
doc.update(fields)
|
||||
return json.dumps(doc).encode()
|
||||
|
||||
|
||||
_full = json.loads(_REAL[_QH].decode("utf-8"))
|
||||
_short = load("generic_dayahead",
|
||||
transport=canned(_edited(_QH, price=_full["price"][:-4])))
|
||||
check(f"a `price` array 4 entries short of `unix_seconds` is no_data, NOT a "
|
||||
f"series truncated to {len(_full['price']) - 4} points - the missing "
|
||||
f"tail of a Belgian day is the evening peak, and a short series is "
|
||||
f"traded on where no data at all switches arbitrage off (§11.4)",
|
||||
_short.kind == "no_data")
|
||||
check("...and the reason names the two lengths, so an operator can see it was "
|
||||
"a length mismatch and not an outage",
|
||||
_short.kind == "no_data"
|
||||
and str(len(_full["unix_seconds"])) in _short.reason
|
||||
and str(len(_full["price"]) - 4) in _short.reason)
|
||||
# The other direction, so the check is not a one-sided `len(price) < len(time)`.
|
||||
_long = load("generic_dayahead",
|
||||
transport=canned(_edited(_QH, price=_full["price"] + [1.0])))
|
||||
check("...and a `price` array one entry LONGER is refused too",
|
||||
_long.kind == "no_data")
|
||||
# ⚠️ THE CONTROL. Without it, "refuses everything" passes both checks above.
|
||||
check("...while the untouched body is still a series, so the refusal is the "
|
||||
"length and not the editing",
|
||||
load("generic_dayahead",
|
||||
transport=canned(_edited(_QH))).kind == "series")
|
||||
|
||||
_unit = load("generic_dayahead", transport=canned(_edited(_QH,
|
||||
unit="EUR / kWh")))
|
||||
check("a feed that changes its unit to EUR/kWh is no_data, not every price "
|
||||
"silently divided by 1000 - MAX_PLAUSIBLE_EUR_KWH only bounds prices "
|
||||
"that are too LARGE, so nothing downstream could catch it",
|
||||
_unit.kind == "no_data")
|
||||
|
||||
_nulls = _full["price"][:]
|
||||
for _i in (3, 17):
|
||||
_nulls[_i] = None
|
||||
_nulled = load("generic_dayahead", transport=canned(_edited(_QH,
|
||||
price=_nulls)))
|
||||
check(f"a published null is DROPPED, leaving a hole rather than a zero - a "
|
||||
f"missing quarter is not a free quarter",
|
||||
_nulled.kind == "series"
|
||||
and derived(lambda: len(_nulled.points)) == len(_full["price"]) - 2)
|
||||
check("...and the dropped instants are the ones the feed left null, not the "
|
||||
"first two",
|
||||
derived(lambda: sorted(
|
||||
int(p.start.timestamp()) for p in _nulled.points))
|
||||
== sorted(t for i, t in enumerate(_full["unix_seconds"])
|
||||
if i not in (3, 17)))
|
||||
_allnull = load("generic_dayahead",
|
||||
transport=canned(_edited(_QH,
|
||||
price=[None] * len(_full["price"]))))
|
||||
check("...but a body with a price for nothing at all is no_data, not an empty "
|
||||
"series", _allnull.kind == "no_data")
|
||||
# ⚠️ WHICH RULE REFUSED IT, not merely that something did. `_validate_raw()`
|
||||
# would also refuse an empty list, one layer further out and with a sentence
|
||||
# that says nothing about the feed - so asserting only `no_data` here would
|
||||
# grade the same under a parser that had lost this refusal entirely.
|
||||
check(f"...refused by the COLUMNAR parser, naming what arrived, rather than "
|
||||
f"reaching the boundary validator as a generic empty list - "
|
||||
f"{_allnull.reason!r}",
|
||||
_allnull.kind == "no_data"
|
||||
and f"{len(_full['price'])} instants" in _allnull.reason
|
||||
and "a price for none of them" in _allnull.reason)
|
||||
|
||||
# ⚠️ THE TWO WARNINGS ARE GRADED, NOT ASSUMED. §11.4 is exactly what makes a
|
||||
# price defect quiet - a parse failure becomes no_data becomes the static
|
||||
# profile, correctly and silently - so an unverified diagnostic is the first
|
||||
# thing here to rot. Both are captured off the shipped logger.
|
||||
_wbuf = io.StringIO()
|
||||
_wh = logging.StreamHandler(_wbuf)
|
||||
logging.getLogger("goodwe.price").addHandler(_wh)
|
||||
load("generic_dayahead", transport=canned(_edited(_QH, price=_nulls)))
|
||||
_dep = load("generic_dayahead", transport=canned(_edited(_QH,
|
||||
deprecated=True)))
|
||||
logging.getLogger("goodwe.price").removeHandler(_wh)
|
||||
_warn = _wbuf.getvalue()
|
||||
_n_null = len([v for v in _nulls if v is None])
|
||||
check(f"the dropped nulls are reported with a count ({_n_null} of "
|
||||
f"{len(_nulls)}), so a series with holes in it is visible to an operator "
|
||||
f"rather than merely short",
|
||||
f"{_n_null} of {len(_nulls)}" in _warn)
|
||||
check("...and a feed announcing deprecated=true says so WHILE IT STILL WORKS - "
|
||||
"the day it stops, §11.4 makes its death look exactly like a quiet one",
|
||||
"deprecated" in _warn and _dep.kind == "series")
|
||||
|
||||
# Which plugins are replayed from real bytes, derived from the registry so a
|
||||
# sixth supplier cannot be added without this line noticing.
|
||||
_networked = sorted(pid for pid, cls in PROVIDERS.items()
|
||||
if cls.capabilities().egress_hosts)
|
||||
_replayed = ["generic_dayahead"]
|
||||
check(f"the networked plugins are {_networked}; {_replayed} is replayed from a "
|
||||
f"captured real body and {sorted(set(_networked) - set(_replayed))} have "
|
||||
f"no capture because their endpoints are unverified (TARIFF-07)",
|
||||
_networked == sorted(["generic_dayahead", "engie_dynamic",
|
||||
"totalenergies_dynamic", "eneco_eddy"]))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
print()
|
||||
print("3. all-in: energy + markup + taxes, added exactly once (AC1, trap 1)")
|
||||
@@ -444,12 +755,29 @@ check(f"a negative day-ahead price passes through, all-in {_neg}",
|
||||
_neg == [round(EPEX_EUR_MWH[1] / 1000 + MARKUP + TAXES, 6)])
|
||||
|
||||
# provisional is per point and comes from the feed, not from the series.
|
||||
for pid, read in sorted(READS.items()):
|
||||
#
|
||||
# ⚠️ `generic_dayahead` IS EXCLUDED FROM THE "MIXED" REQUIREMENT, AND THE
|
||||
# EXCLUSION IS THE POINT RATHER THAN A GAP. The columnar document the real feed
|
||||
# publishes has no per-point provisional field - it is a settled auction result
|
||||
# - so the plugin declares `False` for all of them. Demanding a mixed series
|
||||
# from it would mean feeding it an invented body carrying an invented flag,
|
||||
# which is exactly the closed loop TARIFF-08 exists to remove. It is graded on
|
||||
# what its real body can grade instead, immediately below.
|
||||
for pid in sorted(set(READS) - {"generic_dayahead"}):
|
||||
read = READS[pid]
|
||||
flags = derived(lambda r=read: [p.provisional for p in r.points])
|
||||
check(f"{pid}: provisional is per point and mixed {flags}",
|
||||
isinstance(flags, list) and any(flags) and not all(flags))
|
||||
check(f"...so plan_is_provisional is derived from those points",
|
||||
derived(lambda r=read: plan_is_provisional(r)) is True)
|
||||
_gen_flags = derived(lambda: [p.provisional
|
||||
for p in READS["generic_dayahead"].points])
|
||||
check(f"generic_dayahead's settled auction comes out settled at every point "
|
||||
f"{_gen_flags}",
|
||||
isinstance(_gen_flags, list) and len(_gen_flags) == len(EPEX_EUR_MWH)
|
||||
and not any(_gen_flags))
|
||||
check("...so plan_is_provisional is False for it, derived from the same points",
|
||||
derived(lambda: plan_is_provisional(READS["generic_dayahead"])) is False)
|
||||
# ...and the same function says False when nothing is provisional. Without this
|
||||
# arm, "always True" would pass every check above.
|
||||
_settled = PriceRead.series(
|
||||
@@ -759,7 +1087,7 @@ class _NewSupplier(PriceProvider):
|
||||
|
||||
|
||||
_sixth = load("_test_sixth_supplier",
|
||||
transport=canned(json_feed("price_eur_mwh", EPEX_EUR_MWH)))
|
||||
transport=canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH)))
|
||||
check("a plugin invented outside price.py resolves through the registry",
|
||||
_sixth.kind == "series" and _sixth.provider == "_test_sixth_supplier")
|
||||
_sixth_regime = derived(lambda: _sixth.regime)
|
||||
@@ -894,7 +1222,14 @@ _failures = {
|
||||
"the source is unreachable": load("generic_dayahead", transport=_boom),
|
||||
"the body is not JSON": load("generic_dayahead", transport=canned(b"<html>")),
|
||||
"the feed is empty": load("generic_dayahead",
|
||||
transport=canned(b'{"data": []}')),
|
||||
transport=canned(columnar_feed([]))),
|
||||
# ⚠️ THE ROW SHAPE IS A FAILURE FOR THIS PLUGIN, NOT A SECOND DIALECT. If
|
||||
# `_points_from_columnar_json()` ever grows a sniff for `data`, this case
|
||||
# goes from no_data to a series and TARIFF-08's scope rule is broken by a
|
||||
# check rather than by a review.
|
||||
"the body is the OTHER shape": load(
|
||||
"generic_dayahead",
|
||||
transport=canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH))),
|
||||
"the provider id is unknown": load("no_such_supplier"),
|
||||
"no provider is configured": load("off"),
|
||||
"the csv is missing": load("manual_csv",
|
||||
@@ -1035,11 +1370,8 @@ check("the offset boundary is 01:00 UTC, not local midnight",
|
||||
# A feed that publishes the duplicated hour must survive it. The 25-hour day is
|
||||
# 25 strictly increasing instants; a plugin keyed on local hours would emit 24
|
||||
# and be rejected by the boundary validator instead of silently mispricing.
|
||||
_dst_feed = json.dumps({"data": [
|
||||
{"start": (datetime(2026, 10, 24, 22, tzinfo=UTC)
|
||||
+ timedelta(hours=i)).isoformat(),
|
||||
"price_eur_mwh": 50.0 + i, "provisional": i == 24}
|
||||
for i in range(25)]}).encode()
|
||||
_dst_feed = columnar_feed([50.0 + i for i in range(25)],
|
||||
start=datetime(2026, 10, 24, 22, tzinfo=UTC))
|
||||
_dst_read = load("generic_dayahead", transport=canned(_dst_feed))
|
||||
_dst_n = derived(lambda: len(_dst_read.points))
|
||||
check(f"a 25-point autumn feed is accepted whole ({_dst_n} points)",
|
||||
@@ -1164,7 +1496,7 @@ check("a userinfo prefix cannot smuggle a declared host past the check",
|
||||
check("an operator-supplied URL to another supplier's host is refused",
|
||||
load("generic_dayahead",
|
||||
{"price_feed_url": "https://api.eneco.be/eddy/prices"},
|
||||
transport=canned(json_feed("price_eur_mwh", EPEX_EUR_MWH))
|
||||
transport=canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH))
|
||||
).kind == "no_data")
|
||||
_declared = {pid: cls.capabilities().egress_hosts
|
||||
for pid, cls in PROVIDERS.items()}
|
||||
@@ -1176,7 +1508,7 @@ check("the timeout the plugin passes to the transport is the configured one, "
|
||||
"not a hardcoded default",
|
||||
(lambda t: (load("generic_dayahead", {"price_timeout_s": 7},
|
||||
transport=t), t.calls[0][1] == 7.0)[1])(
|
||||
canned(json_feed("price_eur_mwh", EPEX_EUR_MWH))))
|
||||
canned(synthetic_row_feed("price_eur_mwh", EPEX_EUR_MWH))))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
@@ -288,9 +288,25 @@ def allin(mwh):
|
||||
|
||||
|
||||
def feed_body(instants, table):
|
||||
return json.dumps({"data": [
|
||||
{"start": u.isoformat(), "price_eur_mwh": table[brussels(u).hour]}
|
||||
for u in instants]}).encode()
|
||||
"""A `generic_dayahead` body in the COLUMNAR shape the real feed publishes.
|
||||
|
||||
⚠️ TARIFF-08. This used to build `{"data": [{"start": ...,
|
||||
"price_eur_mwh": ...}]}` - a shape `api.energy-charts.info` has never
|
||||
published and that the shipped parser was nevertheless written to expect.
|
||||
Every price fixture in the repository fed that invented shape, so the
|
||||
suite graded the parser against the parser's own assumption while the live
|
||||
plugin raised `SourceUnavailable` on every fetch. Real captured bodies are
|
||||
replayed in test_price.py section 2b; the fixtures here are SYNTHETIC
|
||||
prices in the REAL shape, because this file is grading the fill and the
|
||||
history, not the parse.
|
||||
"""
|
||||
return json.dumps({
|
||||
"unix_seconds": [int(u.timestamp()) for u in instants],
|
||||
"price": [table[brussels(u).hour] for u in instants],
|
||||
"unit": "EUR / MWh",
|
||||
"license_info": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) "
|
||||
"from Bundesnetzagentur | SMARD.de",
|
||||
"deprecated": False}).encode()
|
||||
|
||||
|
||||
def transport_for(body):
|
||||
@@ -559,14 +575,24 @@ for _n in range(len(TODAY_H) * (60 // _QSTEP_MIN)):
|
||||
_qu = TODAY_H[0] + timedelta(minutes=_QSTEP_MIN * _n)
|
||||
# A spiky within-the-hour profile, so decimation is visible in the SPREAD
|
||||
# and not only in a count.
|
||||
_qrows.append({"start": _qu.isoformat(),
|
||||
"price_eur_mwh": 100.0 + 40.0 * (_n % 4) + brussels(_qu).hour})
|
||||
_qbody = json.dumps({"data": _qrows}).encode()
|
||||
_qrows.append((_qu,
|
||||
100.0 + 40.0 * (_n % 4) + brussels(_qu).hour))
|
||||
# ⚠️ COLUMNAR, like the real feed - see feed_body(). And 900 s spacing is not a
|
||||
# hypothetical here: Belgium moved to a 15-minute MTU on 2025-10-01 and
|
||||
# `api.energy-charts.info` has answered 96 points a day ever since, which is
|
||||
# exactly what test_price.py section 2b replays from a captured body.
|
||||
_qbody = json.dumps({
|
||||
"unix_seconds": [int(_u.timestamp()) for _u, _v in _qrows],
|
||||
"price": [_v for _u, _v in _qrows],
|
||||
"unit": "EUR / MWh",
|
||||
"license_info": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) "
|
||||
"from Bundesnetzagentur | SMARD.de",
|
||||
"deprecated": False}).encode()
|
||||
check(f"the fixture really is sub-hourly - {len(_qrows)} points across "
|
||||
f"{len(TODAY_H)} Brussels hours - or every check below passes for an "
|
||||
f"hourly feed too and grades nothing",
|
||||
len(_qrows) == 4 * len(TODAY_H)
|
||||
and len({_r["start"] for _r in _qrows}) == len(_qrows))
|
||||
and len({_r[0] for _r in _qrows}) == len(_qrows))
|
||||
|
||||
_q_before = run(resolve_tariff(CFG, NOW, _until, transport=transport_for(_qbody),
|
||||
now=NOW, history=NoHistory()))
|
||||
|
||||
@@ -778,9 +778,20 @@ check("a dead feed with NO static_profile has nothing to fall back to, and "
|
||||
_fb_none.origin == "none" and not _fb_none.has_prices)
|
||||
|
||||
# A live feed still wins over the profile.
|
||||
_body = json.dumps({"data": [
|
||||
{"start": (THU + timedelta(hours=i)).isoformat(), "price_eur_mwh": 40 + i}
|
||||
for i in range(24)]}).encode()
|
||||
# ⚠️ COLUMNAR (TARIFF-08). `api.energy-charts.info` publishes parallel
|
||||
# `unix_seconds`/`price` arrays and no `data` key; the `{"data": [...]}` body
|
||||
# this used to build was invented by the parser it was grading, which is how a
|
||||
# provider that could never parse a live response shipped green. Real captured
|
||||
# bodies are replayed in test_price.py section 2b - the prices here are
|
||||
# synthetic, the SHAPE is not.
|
||||
_body = json.dumps({
|
||||
"unix_seconds": [int((THU + timedelta(hours=i)).timestamp())
|
||||
for i in range(24)],
|
||||
"price": [40 + i for i in range(24)],
|
||||
"unit": "EUR / MWh",
|
||||
"license_info": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) "
|
||||
"from Bundesnetzagentur | SMARD.de",
|
||||
"deprecated": False}).encode()
|
||||
|
||||
|
||||
async def _live(url, timeout_s):
|
||||
|
||||
Reference in New Issue
Block a user