TARIFF-09: ask the day-ahead feed for the horizon it was handed, on the Brussels calendar #77

Merged
admin merged 1 commits from TARIFF-09 into release/1.0 2026-09-10 10:44:16 +02:00
Owner

generic_dayahead.fetch() accepted since and until and requested the bare
?bzn=BE, which answers one Brussels day. Everything past today was then
filled from yesterday's profile by FCST-04, flagged provisional, and §8.3 drops
provisional hours from the trade set — so the optimiser never saw tomorrow as
firm, at any hour of any day
, and nothing reported it.

Before, measured

4257e38, 2026-09-10T09:00+02:00, 48 h horizon, live feed, yesterday seeded
through the shipped read path:

60 firm points (15 whole hours) + 33 carried hours of 48 requested  -> 31% firm, 69% carried
carried: 24 h on 2026-09-11, 9 h on 2026-09-12
arbitrage(0.10): allowed, spread 0.4645, buy 11:45 and sell 19:45 -- BOTH 2026-09-10
firm local days = [2026-09-10]

The ceiling was structural, not incidental: with a one-day window, at hour
H of the local day a 48 h horizon is at most 24−H hours firm and 24+H
carried
. At 23:00 that is 1 and 47, and the remaining firm spread
(0.0464 EUR/kWh) no longer clears the default 0.10 round-trip threshold — so
arbitrage is refused outright. FCST-04's fill covers the window until the
~13:00–14:00 CET auction; it was never meant to cover every hour of every day.

The data exists and the endpoint serves it. Verified 2026-09-10T07:45:49Z:
?bzn=BE&start=2026-09-09&end=2026-09-10 returns 192 quarter-hours across two
local days
. Committed unmodified beside TARIFF-08's captures, with URL,
retrieval time and sha256 recorded and re-derived by the replay.

After, same code path, same window, a feed serving both days

before after
firm hours of 48 15 39
carried, provisional 33 9
firm local days 1 2
permitted cycle buy and sell inside one calendar day buys on one day, sells on the next

At 23:00 on the same fixture: 1/47 → 25 firm / 23 carried, and arbitrage
goes from refused to allowed.

Yes — the trade set now contains hours from two days. Graded in
test_price_history.py section 1c, with the firm-local-days set asserted to
have two entries and the two feed states compared against each other rather
than against a refusal.

The Brussels/UTC boundary

since/until are instants; the API takes dates. brussels_date_range()
returns two inclusive dates:

  • first = brussels(since).date(). Not the UTC date. 22:30 UTC in summer is
    already tomorrow in Belgium, so a UTC-derived start asks for the day the
    controller has just left and drops the day it is standing in — for the last
    one or two hours of every day. _bdate()'s lesson one layer up, with a
    worse consequence: _bdate() misfiles an hour, this never fetches it.
  • last = brussels(until).date(), stepped back when until is exactly local
    midnight
    , because until is exclusive.
  • an inverted or empty window collapses to a single day rather than a backwards
    range.

Both ends are graded and both are driven red by mutation, including two that
swap brussels() for the UTC date.

price_feed_url: the exact URL, and the cost said out loud

An override is used verbatim; the plugin appends nothing. Rationale, in the
order it decided:

  1. start/end are this feed's spelling, and the provider declares a second
    host (web-api.tp.entsoe.eu) whose date parameters are different — so an
    override to the other declared host, a case the declaration itself
    anticipates, is exactly where merging appends parameters that host ignores
    and yields the short horizon anyway.
  2. Merging means re-serialising a URL somebody else wrote: reordered, deduped,
    re-escaped. "Override the feed" is what the option already says.
  3. Blind appending is off the table structurally: dated_feed_url() refuses to
    add a query string to a base that already has one.

⚠️ The cost is real, so it is not a footnote. A WARNING on the shipped
goodwe.price logger at fetch time names the consequence — one Brussels
day, the rest carried and flagged, §8.3 dropping them from the trade set — and
names the start..end it would have asked for. Graded, with a control that
the default path stays silent; a mutation that downgrades it to debug() takes
that check red. DECISIONS/TARIFF-09-the-range-and-the-override.md argues the
alternative rather than dismissing it. DOCS.md, config.yaml and both
translations say the same thing.

Before publication, nothing regresses — graded, not assumed

At 09:46 CEST the two-day request ?bzn=BE&start=2026-09-10&end=2026-09-11
came back byte-identical to the bare-URL capture: one day, 96 points. That
stays a series of what exists, with FCST-04 filling the rest and flagging it.
A no_data there would switch a working site's arbitrage off every morning —
and M10, the "defensive" rule that does exactly that, is red by name.

The identity is re-derived from the bytes by the replay, so no second fixture
was needed and none was invented.

The plugin contract

fetch(self, opts, since, until, transport) promises every plugin the range,
and until this card none of the four networked plugins used it. It is still
not enforced
, and the reason is TARIFF-08's: three shipped plugins point at
endpoints nobody has verified (TARIFF-07), so a rule obliging them to pass a
range would be satisfied by an invented parameter name — the
parser-graded-against-its-own-guess defect moved from the response to the
request, shipping as something that looks like a guarantee. Which plugins
consume the range is pinned as a literal instead, so a sixth supplier has to
decide rather than inherit the silence. My recommendation for TARIFF-07 is in
the DECISIONS file.

Mutations — 11, all red, no survivors

Each applied to a fresh cp -a of the clean worktree rather than reverted,
so there is no revert to verify (habit 18) and no shared scratch path two runs
could collide on. Each run through tools/mutation_run.sh, which purges and
fingerprints. Suites: test_price.py, test_price_history.py,
test_static_tariff.py, test_price_forecast.py.

# mutation Δ red on
M1 brussels(since).date() → since.astimezone(utc).date() — the Brussels/UTC start +15 B test_price.py, 4 named checks incl. "...the range starts on the BRUSSELS day, 2026-09-11, and not on the UTC day"
M2 brussels(until) → until.astimezone(utc) — the same trap on the END +15 B test_price.py, 2 named checks
M3 drop the exclusive-until midnight step-back −120 B test_price.py, "a horizon ending exactly at local midnight asks for one day"
M4 restore the shipped defect — ask the bare URL, ignore the range −173 B test_price.py (7), test_price_history.py (2)
M5 end=last → end=first (a one-day request) +1 B test_price.py (3), test_price_history.py (1)
M6 override merges instead of verbatim (override + "&start=…") +19 B "an override is requested EXACTLY as given, with nothing appended"
M7 override warning _LOG.warning → _LOG.debug −2 B "...and the operator is TOLD what it costs them, at fetch time"
M8 dated_feed_url() drops its query-string refusal −21 B "dated_feed_url() refuses a base that already carries a query string"
M9 drop the inverted-window clamp −42 B "an empty or inverted window collapses to a single day"
M10 add the rule this card forbids: refuse when the feed answers fewer days than requested +465 B test_price.py, 33 named checks incl. "...answers that short body with a SERIES of what exists, never no_data"
M11 one price digit changed inside the committed two-day capture +0 B "all three captured bodies are the bytes the provenance file recorded"

M11 is the risky +0 B delta and the re-derived sha256 catches it — an edited
fixture stops being a capture.

⚠️ One honest limit. M10 is red by name in test_price.py, but it takes
test_price_history.py down with a traceback at FCST-04's section 1, whose
_before read is unguarded (_NoData.points raises Absence by design). That
is a red, but an unnamed one. My own section 1c is guarded so it fails by name;
I did not retrofit guards onto another card's section, because that changes
what its checks report. Recorded in MUTATION.md habit 20 for whoever next
touches it.

MUTATION.md habit 20 also records the general lesson: a fixture grades the
response; nothing about the request is reachable from one.
canned() in
test_price.py had been recording (url, timeout_s) since TARIFF-02 and no
check had ever asserted on the first element — the harness knew the plugin was
ignoring its arguments and nothing asked it.

Not fixed here, deliberately

  • The other three networked plugins still ignore their range (TARIFF-07's
    decision; pinned, not enforced — see above).
  • manual_csv reads a whole file and is windowed downstream; it takes the range
    and has nowhere to put it. Stated in the pin rather than papered over.
  • No API rate limiting was risked: two live fetches were made, 66 s apart,
    and every check replays from committed bytes.

Checks

29 of 29 suites green, purged tree, :ro mount, PYTHONDONTWRITEBYTECODE=1.

Not deployed. No ha addons update, no writes to /addons/ or the
inverter, no supplier API touched. No version number picked — ## Unreleased
per PKG-07.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU

`generic_dayahead.fetch()` accepted `since` and `until` and requested the bare `?bzn=BE`, which answers **one Brussels day**. Everything past today was then filled from yesterday's profile by FCST-04, flagged provisional, and §8.3 drops provisional hours from the trade set — so **the optimiser never saw tomorrow as firm, at any hour of any day**, and nothing reported it. ## Before, measured `4257e38`, 2026-09-10T09:00+02:00, 48 h horizon, live feed, yesterday seeded through the shipped read path: ``` 60 firm points (15 whole hours) + 33 carried hours of 48 requested -> 31% firm, 69% carried carried: 24 h on 2026-09-11, 9 h on 2026-09-12 arbitrage(0.10): allowed, spread 0.4645, buy 11:45 and sell 19:45 -- BOTH 2026-09-10 firm local days = [2026-09-10] ``` The ceiling was **structural**, not incidental: with a one-day window, at hour *H* of the local day a 48 h horizon is at most **24−H hours firm and 24+H carried**. At 23:00 that is **1 and 47**, and the remaining firm spread (0.0464 EUR/kWh) no longer clears the default 0.10 round-trip threshold — so arbitrage is **refused outright**. FCST-04's fill covers the window until the ~13:00–14:00 CET auction; it was never meant to cover every hour of every day. **The data exists and the endpoint serves it.** Verified 2026-09-10T07:45:49Z: `?bzn=BE&start=2026-09-09&end=2026-09-10` returns **192 quarter-hours across two local days**. Committed unmodified beside TARIFF-08's captures, with URL, retrieval time and sha256 recorded and **re-derived by the replay**. ## After, same code path, same window, a feed serving both days | | before | after | |---|---|---| | firm hours of 48 | 15 | **39** | | carried, provisional | 33 | 9 | | firm local days | 1 | **2** | | permitted cycle | buy and sell inside one calendar day | **buys on one day, sells on the next** | At 23:00 on the same fixture: 1/47 → **25 firm / 23 carried**, and arbitrage goes from *refused* to *allowed*. **Yes — the trade set now contains hours from two days.** Graded in `test_price_history.py` section 1c, with the firm-local-days set asserted to have two entries and the two feed states compared against each other rather than against a refusal. ## The Brussels/UTC boundary `since`/`until` are **instants**; the API takes **dates**. `brussels_date_range()` returns two *inclusive* dates: * first = `brussels(since).date()`. **Not the UTC date.** 22:30 UTC in summer is already tomorrow in Belgium, so a UTC-derived start asks for the day the controller has just left and drops the day it is standing in — for the last one or two hours of *every* day. `_bdate()`'s lesson one layer up, with a worse consequence: `_bdate()` misfiles an hour, this never fetches it. * last = `brussels(until).date()`, **stepped back when `until` is exactly local midnight**, because `until` is exclusive. * an inverted or empty window collapses to a single day rather than a backwards range. Both ends are graded and both are driven red by mutation, including two that swap `brussels()` for the UTC date. ## `price_feed_url`: the exact URL, and the cost said out loud **An override is used verbatim; the plugin appends nothing.** Rationale, in the order it decided: 1. `start`/`end` are *this feed's* spelling, and the provider declares a second host (`web-api.tp.entsoe.eu`) whose date parameters are different — so an override to the *other declared host*, a case the declaration itself anticipates, is exactly where merging appends parameters that host ignores and yields the short horizon anyway. 2. Merging means re-serialising a URL somebody else wrote: reordered, deduped, re-escaped. "Override the feed" is what the option already says. 3. Blind appending is off the table structurally: `dated_feed_url()` refuses to add a query string to a base that already has one. ⚠️ **The cost is real, so it is not a footnote.** A `WARNING` on the shipped `goodwe.price` logger at **fetch time** names the consequence — one Brussels day, the rest carried and flagged, §8.3 dropping them from the trade set — and names the `start..end` it *would* have asked for. Graded, with a control that the default path stays silent; a mutation that downgrades it to `debug()` takes that check red. `DECISIONS/TARIFF-09-the-range-and-the-override.md` argues the alternative rather than dismissing it. DOCS.md, `config.yaml` and both translations say the same thing. ## Before publication, nothing regresses — graded, not assumed At 09:46 CEST the two-day request `?bzn=BE&start=2026-09-10&end=2026-09-11` came back **byte-identical** to the bare-URL capture: one day, 96 points. That stays a `series` of what exists, with FCST-04 filling the rest and flagging it. A `no_data` there would switch a working site's arbitrage off every morning — and **M10, the "defensive" rule that does exactly that, is red by name**. The identity is re-derived from the bytes by the replay, so no second fixture was needed and none was invented. ## The plugin contract `fetch(self, opts, since, until, transport)` promises every plugin the range, and until this card none of the four networked plugins used it. **It is still not enforced**, and the reason is TARIFF-08's: three shipped plugins point at endpoints nobody has verified (TARIFF-07), so a rule obliging them to pass a range would be satisfied by an **invented parameter name** — the parser-graded-against-its-own-guess defect moved from the response to the request, shipping as something that looks like a guarantee. Which plugins consume the range is **pinned as a literal** instead, so a sixth supplier has to decide rather than inherit the silence. My recommendation for TARIFF-07 is in the DECISIONS file. ## Mutations — 11, all red, no survivors Each applied to a **fresh `cp -a` of the clean worktree** rather than reverted, so there is no revert to verify (habit 18) and no shared scratch path two runs could collide on. Each run through `tools/mutation_run.sh`, which purges and fingerprints. Suites: `test_price.py`, `test_price_history.py`, `test_static_tariff.py`, `test_price_forecast.py`. | # | mutation | Δ | red on | |---|---|---|---| | M1 | `brussels(since).date()` → `since.astimezone(utc).date()` — **the Brussels/UTC start** | +15 B | `test_price.py`, 4 named checks incl. "...the range starts on the BRUSSELS day, 2026-09-11, and not on the UTC day" | | M2 | `brussels(until)` → `until.astimezone(utc)` — **the same trap on the END** | +15 B | `test_price.py`, 2 named checks | | M3 | drop the exclusive-`until` midnight step-back | −120 B | `test_price.py`, "a horizon ending exactly at local midnight asks for one day" | | M4 | **restore the shipped defect** — ask the bare URL, ignore the range | −173 B | `test_price.py` (7), `test_price_history.py` (2) | | M5 | `end=last` → `end=first` (a one-day request) | **+1 B** | `test_price.py` (3), `test_price_history.py` (1) | | M6 | override merges instead of verbatim (`override + "&start=…"`) | +19 B | "an override is requested EXACTLY as given, with nothing appended" | | M7 | override warning `_LOG.warning` → `_LOG.debug` | −2 B | "...and the operator is TOLD what it costs them, at fetch time" | | M8 | `dated_feed_url()` drops its query-string refusal | −21 B | "dated_feed_url() refuses a base that already carries a query string" | | M9 | drop the inverted-window clamp | −42 B | "an empty or inverted window collapses to a single day" | | M10 | **add** the rule this card forbids: refuse when the feed answers fewer days than requested | +465 B | `test_price.py`, 33 named checks incl. "...answers that short body with a SERIES of what exists, never no_data" | | M11 | one price digit changed inside the committed two-day capture | **+0 B** | "all three captured bodies are the bytes the provenance file recorded" | M11 is the risky `+0 B` delta and the re-derived sha256 catches it — an edited fixture stops being a capture. ⚠️ **One honest limit.** M10 is red *by name* in `test_price.py`, but it takes `test_price_history.py` down with a traceback at **FCST-04's section 1**, whose `_before` read is unguarded (`_NoData.points` raises `Absence` by design). That is a red, but an unnamed one. My own section 1c is guarded so it fails by name; I did **not** retrofit guards onto another card's section, because that changes what its checks report. Recorded in `MUTATION.md` habit 20 for whoever next touches it. `MUTATION.md` habit 20 also records the general lesson: **a fixture grades the response; nothing about the request is reachable from one.** `canned()` in `test_price.py` had been recording `(url, timeout_s)` since TARIFF-02 and no check had ever asserted on the first element — the harness knew the plugin was ignoring its arguments and nothing asked it. ## Not fixed here, deliberately * The other three networked plugins still ignore their range (TARIFF-07's decision; pinned, not enforced — see above). * `manual_csv` reads a whole file and is windowed downstream; it takes the range and has nowhere to put it. Stated in the pin rather than papered over. * No API rate limiting was risked: **two** live fetches were made, 66 s apart, and every check replays from committed bytes. ## Checks **29 of 29 suites green**, purged tree, `:ro` mount, `PYTHONDONTWRITEBYTECODE=1`. **Not deployed.** No `ha addons update`, no writes to `/addons/` or the inverter, no supplier API touched. No version number picked — `## Unreleased` per PKG-07. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
admin added 1 commit 2026-09-10 10:36:17 +02:00
TARIFF-09: ask the day-ahead feed for the horizon it was handed
checks / unit (pull_request) Successful in 7m10s
17b7b447c4
`generic_dayahead.fetch()` accepted `since` and `until` and requested the
bare `?bzn=BE`, which answers ONE Brussels day. Everything past today was
then filled from yesterday's profile by FCST-04, flagged provisional, and
§8.3 drops provisional hours from the trade set -- so the optimiser never
saw tomorrow as firm, at any hour of any day, and nothing reported it.

Measured on 4257e38 at 2026-09-10T09:00+02:00, 48 h horizon, live feed,
yesterday seeded through the shipped read path:

    60 firm points (15 whole hours) + 33 carried hours of 48 requested
    carried: 24 h on 2026-09-11, 9 h on 2026-09-12
    arbitrage(0.10): buy 11:45 and sell 19:45 -- BOTH on 2026-09-10
    firm local days = [2026-09-10]

The ceiling was structural: with a one-day window, at hour H a 48 h horizon
is at most 24-H firm and 24+H carried. At 23:00 that is 1 and 47, and the
remaining firm spread (0.0464 EUR/kWh) no longer clears the default 0.10
threshold, so arbitrage is refused outright. FCST-04's fill covers the
window until the ~13:00-14:00 CET auction; it was never meant to cover
every hour of every day.

The data exists and the endpoint serves it: verified 2026-09-10T07:45:49Z,
`?bzn=BE&start=2026-09-09&end=2026-09-10` returns 192 quarter-hours across
two local days. Committed unmodified beside TARIFF-08's captures, with the
URL, retrieval time and sha256 recorded and RE-DERIVED by the replay.

THE RANGE IS DERIVED ON THE BRUSSELS CALENDAR, not the UTC one. 22:30 UTC
in summer is already tomorrow in Belgium, so a UTC-derived start asks for
the day the controller has just left and drops the day it is standing in,
for the last one or two hours of every day. That is `_bdate()`'s lesson one
layer up with a worse consequence: `_bdate()` misfiles an hour, this never
fetches it. `until` is exclusive, so local midnight belongs to the day
before it; an inverted window collapses to one day rather than a backwards
range. Both ends are graded and both are driven red by mutation.

After, same code path and window, a feed serving both days: 39 firm hours
and 9 carried (was 15 and 33), firm across TWO Brussels days, and the
permitted cycle buys on one day and sells on the next.

`price_feed_url` IS THE EXACT URL AND THE PLUGIN APPENDS NOTHING. `start`/
`end` are this feed's spelling and the provider declares a second host
whose parameters are different, so merging would produce a request nobody
typed and the short horizon anyway. The cost is real, so it is not silent:
a WARNING on the shipped logger at fetch time names the consequence and the
start..end that would otherwise have been asked for. `dated_feed_url()`
refuses to append to a base that already carries a query string, so the
blind-append shape cannot come back by edit.
DECISIONS/TARIFF-09-the-range-and-the-override.md argues the alternative.

Before publication nothing regresses and it is graded, not assumed: at
09:46 CEST the two-day request came back byte-identical to the one-day
body, and that stays a `series` of what exists with FCST-04 filling the
rest. A `no_data` there would switch a working site's arbitrage off every
morning -- the mutation that adds that rule is red by name.

The plugin contract is still NOT enforced: three shipped plugins point at
endpoints nobody has verified, and a rule obliging them to pass a range
would be satisfied by an invented parameter name -- TARIFF-08's defect
moved from the response to the request. Which plugins consume the range is
pinned as a literal instead.

11 mutations, all red, no survivors; each applied to a fresh copy of the
tree rather than reverted, through tools/mutation_run.sh. One is +0 B (a
single price digit inside the committed capture) and the re-derived sha256
catches it. MUTATION.md habit 20 records why a fixture could never have
reached this: a captured body is the answer to a URL, and no property of a
body says which URL was asked for. All 29 suites green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
admin merged commit 88c59a9918 into release/1.0 2026-09-10 10:44:16 +02:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/goodwe-addon#77