TARIFF-08: the day-ahead feed is read in the shape it actually publishes
checks / unit (pull_request) Failing after 3s

`generic_dayahead` could not read one byte of `api.energy-charts.info`, and
never had been able to. `_points_from_json()` asks for `doc["data"]`, a list of
`{start, price_eur_mwh}` objects; that document does not exist. Fetched live on
2026-09-10 the feed returns COLUMNAR parallel arrays and no `data` key at all -
`unix_seconds` (96 UTC epoch seconds, 900 s apart), `price` (96 floats),
`unit: 'EUR / MWh'`. Driving the shipped parser against that body:
`SourceUnavailable: the feed carried no 'data' list`. It is the only shipped
provider whose endpoint anybody has reached, so dynamic pricing did not work.

AND §11.4 MADE IT SILENT, which is why it survived to release: a parse error
becomes no_data, becomes the static profile, becomes arbitrage off - the
CORRECT degradation, saying nothing about the cause. A site would have sat on
the fallback indefinitely.

AND THE SUITE COULD NOT SEE IT, because every fixture was built from the
parser's own assumption. test_price.py, test_price_history.py and
test_static_tariff.py each built `json.dumps({"data": [...]})`, a shape invented
here and published by nobody, so the checks graded the parser against the
parser's guess - a closed loop no coverage inside it could ever have escaped.
MUTATION.md habit 19 records it as the fourteenth instance of habit 4 and the
first where the fixture and the code under test shared a SHAPE, not a number.

WHAT CHANGED

* `_points_from_columnar_json()` - a SECOND function, not a sniff. A parser that
  decides which document it received is one that can decide wrong, and a
  wrong-but-parseable response then becomes a price. Feeding the row shape to
  `generic_dayahead` is now a graded no_data.
* Unequal arrays are refused in both directions, with the two lengths in the
  reason. `zip()` stops at the shorter list silently, and a truncated `price`
  array would have produced a real series missing the evening peak - traded on
  as complete, where no data at all switches arbitrage off.
* A unit that is not EUR/MWh is refused. The plugin converts with a fixed
  0.001, and `MAX_PLAUSIBLE_EUR_KWH` only bounds prices that are too LARGE.
* `unix_seconds` are read as UTC instants explicitly; §9.3's Brussels rendering
  stays in `assemble()` rather than being copied into the plugin, where a
  mutation could not tell it from a no-op (M4 proved exactly that on a first
  draft, and the line was removed rather than fenced).
* A published null is dropped and logged with a count; an all-null body is
  no_data named by this parser rather than by the boundary validator.

FIXTURES

Two unmodified response bodies under `scenarios/`, captured by hand on
2026-09-10: quarter-hourly (2026-09-10, 96 points at 900 s) and hourly
(2025-09-10, 24 at 3600 s - the same endpoint before Belgium's 15-minute MTU).
`energy-charts-be-price-captures.json` records the URL, retrieval time, unit,
licence-as-returned and a sha256 of each, and the replay RE-DERIVES the
checksums, so a fixture edited to make a check pass stops being a capture. Data
CC BY 4.0, Bundesnetzagentur | SMARD.de, via Energy-Charts (Fraunhofer ISE),
published without changes - the same BE-zone licence check DATA-02 established.

The surviving invented fixtures are renamed `synthetic_row_feed` at every call
site. `engie_dynamic`, `totalenergies_dynamic` and `eneco_eddy` are untouched
and have no capture: their URLs are unverified and withdrawing them is
TARIFF-07's open decision. Nothing should read their green checks as evidence
that those feeds parse.

MUTATIONS - 12, all red, no survivors. Purged tree, mounted `:ro`, `python3 -B`
(TEST-15). M1 drop the equal-length refusal; M2 make it one-sided; M3 drop the
unit check; M4 `assemble()` stops rendering at the Brussels offset; M5 read
`unix_seconds` as a naive local clock; M6 zero-fill a null instead of dropping
it; M7 drop the all-null refusal; M8 sniff for the row shape; M9 drop the
deprecated warning; M10 drop the dropped-nulls warning; M11 edit one price in
the committed capture; M12 point `generic_dayahead` back at the row parser -
red across all three price suites.

END TO END, LIVE, 2026-09-10: `resolve_tariff()` returns a `TariffPlan`,
`origin='provider'`, 96 quarter-hourly points, spacing [900] s, no decimation,
first point 2026-09-10T00:00:00+02:00 at 0.30608 EUR/kWh all-in. FCST-04's H1
fix was never theoretical. All 29 test files green.

STILL OPEN: `generic_dayahead` asks for no date range, so the default URL
returns one published day - a full 48 h horizon is reached only through §8.3's
carry-forward fill.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
This commit is contained in:
2026-09-10 08:57:32 +02:00
co-authored by Claude Opus 5
parent 5059e28860
commit 6eba18968c
10 changed files with 773 additions and 30 deletions
+86
View File
@@ -14,6 +14,92 @@ if it has, it needs 0.10.0. Whoever deploys decides.
|---|---|
| _this branch_ | FCST-04 - the carry-forward fill reaches the plan, and the day it carries survives a restart |
| _this branch_ | FCST-05 - one byte stops stopping the add-on, and the write path gets the read path's standard |
| _this branch_ | TARIFF-08 - the day-ahead feed is read in the shape it actually publishes, and the fixtures stop agreeing with the parser |
### TARIFF-08 - no network price provider had ever parsed a live response
**`generic_dayahead` could not read one byte of `api.energy-charts.info`, and
had never been able to.** `_points_from_json()` asks for `doc["data"]`, a list
of `{start, price_eur_mwh}` objects. That document does not exist. Fetched live
on 2026-09-10, the feed returns **columnar** parallel arrays and no `data` key
at all:
```
top-level keys: ['deprecated', 'license_info', 'price', 'unit', 'unix_seconds']
unix_seconds: 96 UTC epoch seconds, first 1788991200, spacing 900 s
price: 96 floats, first 216.08
unit: 'EUR / MWh'
```
Driving the shipped parser against that body: `SourceUnavailable: the feed
carried no 'data' list`. Since `generic_dayahead` is the only shipped provider
whose endpoint anybody has reached, **dynamic pricing did not work at all**.
⚠️ **AND §11.4 MADE THAT SILENT, WHICH IS WHY IT LASTED.** A parse error becomes
`no_data`, becomes the static profile, becomes arbitrage off - the *correct*
degradation, and it says nothing whatsoever about the cause. A site would have
sat on the fallback indefinitely while the status page reported only that the
feed was down.
⚠️ **THE SUITE COULD NOT SEE IT BECAUSE EVERY FIXTURE WAS BUILT FROM THE
PARSER'S OWN ASSUMPTION.** `test_price.py`, `test_price_history.py` and
`test_static_tariff.py` each built `json.dumps({"data": [{"start": ...,
"price_eur_mwh": ...}]})` - a shape invented here, published by nobody. The
checks graded the parser against the parser's guess: a closed loop that cannot
notice the guess is wrong, and no amount of coverage inside it would have. The
loop is broken by **bytes nobody in this repository wrote**: two unmodified
response bodies are committed under `scenarios/`, one quarter-hourly
(2026-09-10, 96 points at 900 s) and one hourly (2025-09-10, 24 points at
3600 s - the same endpoint before Belgium's 15-minute MTU), with URL, retrieval
time, licence-as-returned and a sha256 recorded beside them in
`energy-charts-be-price-captures.json` and **re-derived** by the replay, so a
fixture edited to make a check pass stops being a capture. The remaining
invented fixtures are renamed `synthetic_row_feed` at every call site.
**The columnar document is read by its own function, `_points_from_columnar_json()`.**
Not by teaching the row parser to sniff for both: a parser that decides which
document it received is a parser that can decide wrong, and a wrong-but-
parseable response then becomes a price. Two shapes, two functions, and each
plugin names the one its feed publishes. Feeding the row shape to
`generic_dayahead` is now a graded `no_data`, so the sniff cannot come back
quietly.
⚠️ **UNEQUAL ARRAYS ARE REFUSED RATHER THAN ZIPPED.** `zip()` stops at the
shorter list and says nothing, so a truncated `price` array would have produced
a **real price series with its tail missing** - and the tail of a Belgian day is
the evening peak. That is worse than no series: `no_data` switches arbitrage
off, whereas a short series is traded on as though complete, and nothing
downstream can tell them apart. Refused in both directions, with the two lengths
in the reason.
⚠️ **A UNIT CHANGE IS REFUSED FOR A REASON `_validate_raw()` CANNOT COVER.** The
plugin converts with a fixed `0.001`; if the feed ever published EUR/kWh under
the same keys, every price would come out a **thousand times too small** and
`MAX_PLAUSIBLE_EUR_KWH` would not fire, because it only bounds prices that are
too large. DATA-02's fetcher already refused on this field; the shipped path
does now too.
**`unix_seconds` are UTC instants, said so explicitly**
(`datetime.fromtimestamp(t, timezone.utc)`); §9.3's rendering at the Brussels
offset stays where it already lived, in `assemble()`, rather than being copied
into the plugin where a mutation could not tell it from a no-op. A published
`null` is dropped and logged with a count - a missing quarter is not a free
quarter - while an all-null body is `no_data` named by this parser rather than
by the boundary validator.
**End to end against the live feed, 2026-09-10:** `resolve_tariff()` returns a
`TariffPlan` with `origin='provider'`, **96 quarter-hourly points**, spacing
`[900]` s and no decimation, first point `2026-09-10T00:00:00+02:00` at 0.30608
EUR/kWh all-in. FCST-04's H1 fix was never theoretical - 900 s is the shape of
the real feed.
⚠️ **WHAT THIS DOES NOT FIX.** `engie_dynamic`, `totalenergies_dynamic` and
`eneco_eddy` are untouched: their URLs are unverified, no real body can be
captured for them, and withdrawing them is TARIFF-07's open decision. Their
fixtures are now named `synthetic_row_feed` so nothing reads them as evidence
that those feeds parse. And `generic_dayahead` still ignores `since`/`until`
and asks for no date range, so the default URL returns one published day - a
full 48 h horizon is reached only through §8.3's carry-forward fill.
### FCST-05 - what FCST-04's own contract still did not hold