TARIFF-08: the day-ahead feed is read in the shape it actually publishes, and the fixtures stop agreeing with the parser #76

Merged
admin merged 2 commits from TARIFF-08 into release/1.0 2026-09-10 09:14:50 +02:00
Owner

The defect, reproduced before it was fixed

generic_dayahead could not read one byte of api.energy-charts.info, and never had been able to. _points_from_json() asks for doc["data"], a list of {start, price_eur_mwh} objects. That document does not exist. Fetched live on 2026-09-10, https://api.energy-charts.info/price?bzn=BE returns a columnar body:

top-level keys: ['deprecated', 'license_info', 'price', 'unit', 'unix_seconds']
has 'data' key: False
  unix_seconds: list of 96   first 1788991200   spacing 900 s
  price:        list of 96   first 216.08
  unit:         'EUR / MWh'

Driving the shipped parser against that body:

RAISED SourceUnavailable: the feed carried no 'data' list

It is the only shipped provider whose endpoint anybody has reached, so dynamic pricing did not work at all.

⚠️ And §11.4 made it silent, which is why it reached release. A parse error becomes no_data, becomes the static profile, becomes arbitrage off — the correct degradation, and it says nothing whatsoever about the cause. A site would have sat on the fallback indefinitely while the status page reported only that the feed was down.

Why the suite could not see it

Every price fixture in the repository was built from the parser's own assumption — test_price.py:290, test_price_history.py:291, test_static_tariff.py:781 all built json.dumps({"data": [{"start": ..., "price_eur_mwh": ...}]}), a shape invented here and published by nobody. The checks graded the parser against the parser's guess: a closed loop that cannot notice the guess is wrong, and no amount of coverage inside it would have.

The loop is broken by bytes nobody in this repository wrote. MUTATION.md habit 19 records this as the fourteenth instance of habit 4 — and the first where the fixture and the code under test shared a shape rather than a number.

What changed

  • _points_from_columnar_json() — a second function, not a sniff. Per the scope rule: a parser that decides which document it received is a parser that can decide wrong, and a wrong-but-parseable response then becomes a price. Feeding the row shape to generic_dayahead is now a graded no_data, so the sniff cannot come back quietly (M8).
  • Unequal arrays are refused, in both directions, with the two lengths in the reason. zip() stops at the shorter list and says nothing, so a truncated price array would have produced a real price series with its tail missing — and the tail of a Belgian day is the evening peak. That is worse than no series: no_data switches arbitrage off, whereas a short series is traded on as complete, and nothing downstream can tell them apart.
  • A unit that is not EUR/MWh is refused. The plugin converts with a fixed 0.001; a feed publishing EUR/kWh under the same keys would come out a thousand times too small, and MAX_PLAUSIBLE_EUR_KWH only bounds prices that are too large. DATA-02's fetcher already refused on this field.
  • unix_seconds are read as UTC instants explicitly (datetime.fromtimestamp(t, timezone.utc)). §9.3's rendering at the Brussels offset stays where it already lived — in assemble() — rather than being copied into the plugin. ⚠️ The first draft did copy it, and M4 came back a survivor on a purged, :ro tree: a second copy of a rule that already has an owner is indistinguishable from a no-op. The line was removed rather than fenced with a check, and the mutation that proves the rendering is real is now aimed at assemble().
  • A published null is dropped and logged with a count — a missing quarter is not a free quarter — while an all-null body is no_data named by this parser rather than by the boundary validator. The distinction from the length check is deliberate: unequal arrays mean the instant-to-price pairing is unreliable; a null means the source stated it published no price for that one instant, and _Series.at() already answers a hole with Absence.

The fixtures

Two unmodified response bodies, committed under scenarios/:

file URL shape
energy-charts-be-price-2026-09-10-quarterhourly.json .../price?bzn=BE (the shipped default URL, verbatim) 96 points, 900 s
energy-charts-be-price-2025-09-10-hourly.json .../price?bzn=BE&start=2025-09-10&end=2025-09-10 24 points, 3600 s

The hourly one is the same endpoint before Belgium's 15-minute MTU (2025-10-01) — a fixture at one spacing cannot grade an assumption about the other. energy-charts-be-price-captures.json records the URL, retrieval time, unit, licence-as-returned and a sha256 of each, and the replay re-derives the checksums, so a fixture edited to make a check pass stops being a capture (M11). Data CC BY 4.0, Bundesnetzagentur | SMARD.de, retrieved via Energy-Charts (Fraunhofer ISE), published without changes — the same BE-zone licence position DATA-02 established.

The surviving invented fixtures are renamed synthetic_row_feed at every call site.

⚠️ engie_dynamic, totalenergies_dynamic and eneco_eddy are untouched and have no capture. Their URLs are unverified and withdrawing them is TARIFF-07's open decision. A check derived from the shipped registry now states, in its own sentence, which networked plugins are replayed from real bytes and which are not, so nothing reads their green checks as evidence that those feeds parse.

Mutations — 12, all red, no survivors

Purged tree, mounted :ro, python3 -B (TEST-15).

# mutation red on
M1 drop the equal-length refusal entirely short / longer / reason-names-lengths
M2 one-sided length check (only a SHORT price refused) "...a price array one entry LONGER is refused too"
M3 drop the unit check "a feed that changes its unit to EUR/kWh is no_data"
M4 assemble() stops rendering at the Brussels offset the replay's 00:00+02:00 check, and the January-offset check
M5 read unix_seconds as a naive host-local clock the whole of section 2 — no plugin returns a series
M6 zero-fill a null instead of dropping the instant the null-drop pair, and the all-null refusal
M7 drop the all-null refusal (let the boundary validator catch it) "...refused by the COLUMNAR parser, naming what arrived"
M8 sniff: fall back to the row shape when unix_seconds is absent "every site-level failure returns no_data" (the row-shape case)
M9 drop the deprecated-feed warning "a feed announcing deprecated=true says so while it still works"
M10 drop the dropped-nulls warning "the dropped nulls are reported with a count (2 of 96)"
M11 edit one price in the committed capture "both captured bodies are the bytes the provenance file recorded"
M12 the shipped defect: point generic_dayahead back at the row parser red across test_price.py, test_price_history.py and test_static_tariff.py

End to end, against the live feed, 2026-09-10

type       : TariffPlan
origin     : provider
has_prices : True
product    : PriceProduct.DAY_AHEAD
points     : 96
spacing_s  : [900]
first      : 2026-09-10T00:00:00+02:00  0.30608 EUR/kWh all-in
last       : 2026-09-10T23:45:00+02:00  0.26179
arbitrage@0.10 : allowed, spread 0.4645 EUR/kWh

fetch -> parse -> assemble() -> TariffPlan works for the first time. FCST-04's H1 fix was never theoretical — 900 s is the shape of the real feed, and 96 points arrive with no decimation.

All 29 test files green.

Still open, named rather than fixed

generic_dayahead ignores since/until and asks for no date range, so the default URL returns one published day. A full 48 h horizon is reached only through §8.3's carry-forward fill. No config option was added.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU

## The defect, reproduced before it was fixed `generic_dayahead` could not read one byte of `api.energy-charts.info`, and never had been able to. `_points_from_json()` asks for `doc["data"]`, a list of `{start, price_eur_mwh}` objects. **That document does not exist.** Fetched live on 2026-09-10, `https://api.energy-charts.info/price?bzn=BE` returns a **columnar** body: ``` top-level keys: ['deprecated', 'license_info', 'price', 'unit', 'unix_seconds'] has 'data' key: False unix_seconds: list of 96 first 1788991200 spacing 900 s price: list of 96 first 216.08 unit: 'EUR / MWh' ``` Driving the shipped parser against that body: ``` RAISED SourceUnavailable: the feed carried no 'data' list ``` It is the only shipped provider whose endpoint anybody has reached, so **dynamic pricing did not work at all**. ⚠️ **And §11.4 made it silent, which is why it reached release.** A parse error becomes `no_data`, becomes the static profile, becomes arbitrage off — the *correct* degradation, and it says nothing whatsoever about the cause. A site would have sat on the fallback indefinitely while the status page reported only that the feed was down. ## Why the suite could not see it Every price fixture in the repository was built from the parser's own assumption — `test_price.py:290`, `test_price_history.py:291`, `test_static_tariff.py:781` all built `json.dumps({"data": [{"start": ..., "price_eur_mwh": ...}]})`, a shape invented here and published by nobody. The checks graded the parser against the parser's guess: a closed loop that cannot notice the guess is wrong, and no amount of coverage *inside* it would have. The loop is broken by bytes nobody in this repository wrote. `MUTATION.md` habit **19** records this as the fourteenth instance of habit 4 — and the first where the fixture and the code under test shared a **shape** rather than a number. ## What changed * **`_points_from_columnar_json()` — a second function, not a sniff.** Per the scope rule: a parser that decides which document it received is a parser that can decide wrong, and a wrong-but-parseable response then becomes a price. Feeding the row shape to `generic_dayahead` is now a **graded `no_data`**, so the sniff cannot come back quietly (M8). * **Unequal arrays are refused, in both directions, with the two lengths in the reason.** `zip()` stops at the shorter list and says nothing, so a truncated `price` array would have produced a **real price series with its tail missing** — and the tail of a Belgian day is the evening peak. That is worse than no series: `no_data` switches arbitrage off, whereas a short series is traded on as complete, and nothing downstream can tell them apart. * **A unit that is not EUR/MWh is refused.** The plugin converts with a fixed `0.001`; a feed publishing EUR/kWh under the same keys would come out a **thousand times too small**, and `MAX_PLAUSIBLE_EUR_KWH` only bounds prices that are too *large*. DATA-02's fetcher already refused on this field. * **`unix_seconds` are read as UTC instants explicitly** (`datetime.fromtimestamp(t, timezone.utc)`). §9.3's rendering at the Brussels offset stays where it already lived — in `assemble()` — rather than being copied into the plugin. ⚠️ The first draft *did* copy it, and **M4 came back a survivor on a purged, `:ro` tree**: a second copy of a rule that already has an owner is indistinguishable from a no-op. The line was removed rather than fenced with a check, and the mutation that proves the rendering is real is now aimed at `assemble()`. * **A published `null` is dropped and logged with a count** — a missing quarter is not a free quarter — while an all-null body is `no_data` named by *this* parser rather than by the boundary validator. The distinction from the length check is deliberate: unequal arrays mean the instant-to-price pairing is unreliable; a null means the source stated it published no price for that one instant, and `_Series.at()` already answers a hole with `Absence`. ## The fixtures Two **unmodified response bodies**, committed under `scenarios/`: | file | URL | shape | |---|---|---| | `energy-charts-be-price-2026-09-10-quarterhourly.json` | `.../price?bzn=BE` (the shipped default URL, verbatim) | 96 points, 900 s | | `energy-charts-be-price-2025-09-10-hourly.json` | `.../price?bzn=BE&start=2025-09-10&end=2025-09-10` | 24 points, 3600 s | The hourly one is the *same endpoint* before Belgium's 15-minute MTU (2025-10-01) — a fixture at one spacing cannot grade an assumption about the other. `energy-charts-be-price-captures.json` records the URL, retrieval time, unit, licence-as-returned and a sha256 of each, and the replay **re-derives** the checksums, so a fixture edited to make a check pass stops being a capture (M11). Data CC BY 4.0, Bundesnetzagentur | SMARD.de, retrieved via Energy-Charts (Fraunhofer ISE), published without changes — the same BE-zone licence position DATA-02 established. The surviving invented fixtures are renamed **`synthetic_row_feed`** at every call site. ⚠️ **`engie_dynamic`, `totalenergies_dynamic` and `eneco_eddy` are untouched and have no capture.** Their URLs are unverified and withdrawing them is TARIFF-07's open decision. A check derived from the shipped registry now states, in its own sentence, which networked plugins are replayed from real bytes and which are not, so nothing reads their green checks as evidence that those feeds parse. ## Mutations — 12, all red, no survivors Purged tree, mounted `:ro`, `python3 -B` (TEST-15). | # | mutation | red on | |---|---|---| | M1 | drop the equal-length refusal entirely | short / longer / reason-names-lengths | | M2 | one-sided length check (only a SHORT `price` refused) | "...a `price` array one entry LONGER is refused too" | | M3 | drop the unit check | "a feed that changes its unit to EUR/kWh is no_data" | | M4 | `assemble()` stops rendering at the Brussels offset | the replay's `00:00+02:00` check, and the January-offset check | | M5 | read `unix_seconds` as a naive host-local clock | the whole of section 2 — no plugin returns a series | | M6 | zero-fill a null instead of dropping the instant | the null-drop pair, and the all-null refusal | | M7 | drop the all-null refusal (let the boundary validator catch it) | "...refused by the COLUMNAR parser, naming what arrived" | | M8 | sniff: fall back to the row shape when `unix_seconds` is absent | "every site-level failure returns no_data" (the row-shape case) | | M9 | drop the deprecated-feed warning | "a feed announcing deprecated=true says so while it still works" | | M10 | drop the dropped-nulls warning | "the dropped nulls are reported with a count (2 of 96)" | | M11 | edit one price in the committed capture | "both captured bodies are the bytes the provenance file recorded" | | M12 | **the shipped defect**: point `generic_dayahead` back at the row parser | red across `test_price.py`, `test_price_history.py` **and** `test_static_tariff.py` | ## End to end, against the live feed, 2026-09-10 ``` type : TariffPlan origin : provider has_prices : True product : PriceProduct.DAY_AHEAD points : 96 spacing_s : [900] first : 2026-09-10T00:00:00+02:00 0.30608 EUR/kWh all-in last : 2026-09-10T23:45:00+02:00 0.26179 arbitrage@0.10 : allowed, spread 0.4645 EUR/kWh ``` `fetch -> parse -> assemble() -> TariffPlan` works for the first time. FCST-04's H1 fix was never theoretical — 900 s is the shape of the real feed, and 96 points arrive with no decimation. **All 29 test files green.** ## Still open, named rather than fixed `generic_dayahead` ignores `since`/`until` and asks for no date range, so the default URL returns **one published day**. A full 48 h horizon is reached only through §8.3's carry-forward fill. No config option was added. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
admin added 1 commit 2026-09-10 08:58:26 +02:00
`generic_dayahead` could not read one byte of `api.energy-charts.info`, and
never had been able to. `_points_from_json()` asks for `doc["data"]`, a list of
`{start, price_eur_mwh}` objects; that document does not exist. Fetched live on
2026-09-10 the feed returns COLUMNAR parallel arrays and no `data` key at all -
`unix_seconds` (96 UTC epoch seconds, 900 s apart), `price` (96 floats),
`unit: 'EUR / MWh'`. Driving the shipped parser against that body:
`SourceUnavailable: the feed carried no 'data' list`. It is the only shipped
provider whose endpoint anybody has reached, so dynamic pricing did not work.

AND §11.4 MADE IT SILENT, which is why it survived to release: a parse error
becomes no_data, becomes the static profile, becomes arbitrage off - the
CORRECT degradation, saying nothing about the cause. A site would have sat on
the fallback indefinitely.

AND THE SUITE COULD NOT SEE IT, because every fixture was built from the
parser's own assumption. test_price.py, test_price_history.py and
test_static_tariff.py each built `json.dumps({"data": [...]})`, a shape invented
here and published by nobody, so the checks graded the parser against the
parser's guess - a closed loop no coverage inside it could ever have escaped.
MUTATION.md habit 19 records it as the fourteenth instance of habit 4 and the
first where the fixture and the code under test shared a SHAPE, not a number.

WHAT CHANGED

* `_points_from_columnar_json()` - a SECOND function, not a sniff. A parser that
  decides which document it received is one that can decide wrong, and a
  wrong-but-parseable response then becomes a price. Feeding the row shape to
  `generic_dayahead` is now a graded no_data.
* Unequal arrays are refused in both directions, with the two lengths in the
  reason. `zip()` stops at the shorter list silently, and a truncated `price`
  array would have produced a real series missing the evening peak - traded on
  as complete, where no data at all switches arbitrage off.
* A unit that is not EUR/MWh is refused. The plugin converts with a fixed
  0.001, and `MAX_PLAUSIBLE_EUR_KWH` only bounds prices that are too LARGE.
* `unix_seconds` are read as UTC instants explicitly; §9.3's Brussels rendering
  stays in `assemble()` rather than being copied into the plugin, where a
  mutation could not tell it from a no-op (M4 proved exactly that on a first
  draft, and the line was removed rather than fenced).
* A published null is dropped and logged with a count; an all-null body is
  no_data named by this parser rather than by the boundary validator.

FIXTURES

Two unmodified response bodies under `scenarios/`, captured by hand on
2026-09-10: quarter-hourly (2026-09-10, 96 points at 900 s) and hourly
(2025-09-10, 24 at 3600 s - the same endpoint before Belgium's 15-minute MTU).
`energy-charts-be-price-captures.json` records the URL, retrieval time, unit,
licence-as-returned and a sha256 of each, and the replay RE-DERIVES the
checksums, so a fixture edited to make a check pass stops being a capture. Data
CC BY 4.0, Bundesnetzagentur | SMARD.de, via Energy-Charts (Fraunhofer ISE),
published without changes - the same BE-zone licence check DATA-02 established.

The surviving invented fixtures are renamed `synthetic_row_feed` at every call
site. `engie_dynamic`, `totalenergies_dynamic` and `eneco_eddy` are untouched
and have no capture: their URLs are unverified and withdrawing them is
TARIFF-07's open decision. Nothing should read their green checks as evidence
that those feeds parse.

MUTATIONS - 12, all red, no survivors. Purged tree, mounted `:ro`, `python3 -B`
(TEST-15). M1 drop the equal-length refusal; M2 make it one-sided; M3 drop the
unit check; M4 `assemble()` stops rendering at the Brussels offset; M5 read
`unix_seconds` as a naive local clock; M6 zero-fill a null instead of dropping
it; M7 drop the all-null refusal; M8 sniff for the row shape; M9 drop the
deprecated warning; M10 drop the dropped-nulls warning; M11 edit one price in
the committed capture; M12 point `generic_dayahead` back at the row parser -
red across all three price suites.

END TO END, LIVE, 2026-09-10: `resolve_tariff()` returns a `TariffPlan`,
`origin='provider'`, 96 quarter-hourly points, spacing [900] s, no decimation,
first point 2026-09-10T00:00:00+02:00 at 0.30608 EUR/kWh all-in. FCST-04's H1
fix was never theoretical. All 29 test files green.

STILL OPEN: `generic_dayahead` asks for no date range, so the default URL
returns one published day - a full 48 h horizon is reached only through §8.3's
carry-forward fill.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
admin added 1 commit 2026-09-10 09:06:41 +02:00
Merge release/1.0 (FCST-06) into TARIFF-08
checks / unit (pull_request) Successful in 7m10s
7a5281e97f
One conflict, in CHANGELOG.md's `## Unreleased` block, exactly as expected
from two branches landing beside each other: both `_this branch_` table rows
and both sections are kept, FCST-06's first. Nothing else conflicted -
FCST-06's `test_price_history.py` section 5d and this branch's columnar
`feed_body()` merged cleanly - and all 29 test files are green on the result.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
admin merged commit 4257e38e5a into release/1.0 2026-09-10 09:14:50 +02:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/goodwe-addon#76