Compare commits

..
17 Commits
Author SHA1 Message Date
Claude f5ea39784d Merge release/1.0 (TARIFF-10) into TARIFF-12
checks / unit (pull_request) Successful in 7m16s
CHANGELOG.md was the only conflict: both entries belong under the same
`## Unreleased` heading, TARIFF-10's first because it merged first. No
version number picked.

app/main.py auto-merged, and was re-read because of it. TARIFF-10 changed
WHICH installs run `run_tariff_refresh()` (`arbitrage.enabled(opts)` ->
`tariff_refresh_wanted(opts)`); this card changed what that task does on
a failed fetch. Nothing in the keep/replace reasoning changed, and the
two newly reachable configurations are now DRIVEN in test_arbitrage.py
8b rather than reasoned about:

  * a provider configured with trading DISARMED is the case the guard
    was designed for, and needs no second fixture because nothing on
    this path reads `arbitrage_enabled` at all - asserted over the AST's
    symbols and non-docstring constants, after two drafts of that check
    went red on a CORRECT tree for matching the word "arbitrage" in a
    docstring and in a log line;
  * `price_provider: off` + a `static_profile` resolves to the
    CONFIGURED product (`origin: "static"`), so `standing.from_feed` is
    false on every refresh and the guard never latches - driven twice in
    a row, because a guard that latched there would freeze a static
    site's plan at its first window for the life of the install.

Also: the kept-plan log line said "arbitrage stops with it", which after
TARIFF-10 is false on a site that fetches without trading; it now says
"arbitrage, where it is armed, stops with it". The keep decision is
strictly downstream of `resolve_tariff()`, so the §8.3 carry into
price_history - TARIFF-10's whole point - is unaffected either way.

All 29 suites green on the merged tree, including TARIFF-10's new
test_price.py section 13. All eleven mutations RE-DRIVEN on the merged
tree rather than carried over: eleven red, no survivors, every revert
fingerprinted. M11 is new - making the guard conditional on
`arbitrage.enabled(self.o)` (+86 B) is red on five named checks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-12 21:31:27 +02:00
admin 559f0eb022 Merge pull request 'TARIFF-10: a configured provider fetches prices, whether or not trading is armed' (#78) from TARIFF-10 into release/1.0
run_tariff_refresh() was created only under `if arbitrage.enabled(opts)`, and it is the sole caller of resolve_tariff(), which is the sole caller of price_history.carry(). arbitrage_enabled defaults to false, so on a shipped install with a provider configured nothing was ever fetched, no plan resolved, /data/price_history.json never written, FCST-04's Price history row read `empty` forever, and TARIFF-09's warning never fired. Every price check constructs its own resolve_tariff() call, so the suites exercised a path production never reached.

Measured, not inferred: a 40-hour soak ran two instances side by side - the shipped default made zero feed requests and never created its store across all 802 samples, while an identical instance with arbitrage armed resolved a 132-point plan from the same feed. DOCS.md:1348 has claimed the fixed behaviour all along.

The product owner's decision is Option A: a configured provider fetches. Trading stays separately gated, and that is graded directly rather than inferred - the real control loop is driven on a resolved provider plan with arbitrage_enabled the only option moving: off gives +180 W self-consumption, on gives -5000 W at the cheapest hour and +5000 W elsewhere. The armed rows are what make the unarmed row evidence.

ONE ACCEPTANCE CRITERION WAS DELIBERATELY NOT IMPLEMENTED LITERALLY, and the deviation is right. The criterion said no fetch task at all when price_provider is off, whatever arbitrage_enabled says. That breaks a configuration validate.py explicitly accepts - provider off, static_profile set, arbitrage armed - by leaving tariff_plan at None forever, silently disarming a strategy the UI reports as running. Driven both ways. The guard is therefore `provider_configured(opts) or arbitrage.enabled(opts)`; the criterion's stated harm is still fully met, because with the provider off resolve_tariff() answers out of the static profile and opens no socket, and the true shipped default starts no task at all. DECISIONS section 3 records the trade.

cheap_now() is NOT wired, deliberately. On a provider-only site static_status() returns False on every cycle, so adopting it would take the maintenance cycle's paid grid top-up from never to sometimes - a change to what spends money, and it needs its own decision rather than riding along on this one.

Evidence: 29/29 green. 13 mutations over 14 runs, each on a fresh tree copy, purged, :ro, -B. The shipped defect at -4 B is red on 7 named checks. The reachability check grades reachability from a CONFIGURATION rather than from the `if` statement - the `if` is exactly what someone meant to write. The instrument itself was mutated too, both +0 B, both red. One real survivor was found and closed in the same commit: provider_id() dropping .strip() left four price suites green. And one fixture defect was found and fixed - five minutes of P1 telegrams made the peak guard blind depending on where the wall clock sat in the billing quarter, so the armed baseline failed to trade on some runs.
2026-09-12 21:07:55 +02:00
Claude c81289527f TARIFF-12: a returned degradation is not a refresh
checks / unit (pull_request) Successful in 7m12s
`run_tariff_refresh()` promised in its own docstring that "a failure does
not clear the plan", and enforced it with `except Exception` - a branch a
supplier outage never reaches. `load_prices()` states the reason in its
own words: it "RETURNS, NEVER RAISES, for every failure a site can
actually have", because a degradation arriving as a traceback into the
planning tick would stop the controller regulating over an expired
certificate. So a dead feed RETURNS a §11.4 plan, the `try` succeeds, and
the assignment below it overwrote the good plan unconditionally. With
`static_profile` unset - the shipped default, and the normal state of a
dynamic-tariff site - `_degrade()` returns `TariffPlan(no_data, None,
"none")`: a complete 48-hour plan replaced by nothing, at the FIRST
failed fetch, ~27 minutes after the last good one. `TARIFF_MAX_AGE_S`
(5400 s) never governed anything and there was no serve-stale window at
all.

Fourth card in one family: FCST-04's M4 (`not ok` only), FCST-05's HIGH
(`(JSONDecodeError, OSError)` only), FCST-06 (`not self.ok` only), and
now `except Exception` only, for a path that never raises.

The question is now asked once, after the attempt, from the evidence
that exists - FCST-06's answer, one module over. The evidence is
`TariffPlan.from_feed` (did this resolution observe the configured feed)
and the shipped `fresh_tariff_plan()`, reused rather than transcribed. A
degradation replaces a standing observation only once that observation
has aged out, so `TARIFF_MAX_AGE_S` is now the serve-stale window it
reads like.

A §11.4 static fallback does not replace a fresh provider plan either,
although it carries real prices: every point `_degrade()` builds is
`provisional=True`, so `firm_points()` is empty, `arbitrage()` refuses
and `cheap_now()` is False - everything a consumer may act on is already
off - while a day-ahead price for a given hour does not change once
published. The rule is deliberately not symmetrical: a standing plan
that is itself a degradation IS replaced, or an outage would freeze the
operator's static profile at the window it was first generated for.

`tariff_at` does not move on a kept plan. It is the moment of the fetch
that produced the plan standing, `fresh_tariff_plan()` judges the age
from it, and an unmoved timestamp is the only thing that tells a kept
plan from a freshly fetched one.

test_arbitrage.py section 8b drives the whole chain with nothing stubbed
but the socket: the transport raises the way aiohttp does, load_prices()
returns no_data, _degrade() builds the §11.4 plan, and the shipped
run_tariff_refresh() decides. A check that patched resolve_tariff to
RAISE would have passed against the defect. Ten mutations, ten reds, no
survivors; 29/29 suites green.

Out of scope and unchanged: the persisted price store still does not
back the plan during an outage (FCST-03), and the status banner and the
Arbitrage row's lag are separately carded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-12 21:06:45 +02:00
adminandClaude Opus 5 e6c45a61e7 TARIFF-10: configuring a price provider is what fetches prices
checks / unit (pull_request) Successful in 7m19s
amain() started the background tariff refresh only when arbitrage_enabled was
on, and that option ships false. run_tariff_refresh is the only caller of
resolve_tariff(), which is the only caller of price_history.carry() - so on a
shipped install with a price provider configured nothing was ever fetched: no
plan resolved, /data/price_history.json never created, the Price history status
row reading `empty` for the life of the install, and TARIFF-09's short-horizon
warning unable to fire because there was no fetch to warn about. Measured over
a 40 h two-instance soak on one feed: 0 requests and `present: false` across all
802 samples, against a resolved 132-point plan on the instance with arbitrage
armed.

git log -S puts the refresh task, its arbitrage.enabled() guard AND the DOCS.md
sentence saying a configured provider still fetches in the same commit - OPT-04,
bdd7f18; config.yaml has carried the claim since TARIFF-02, 852fc7c. The prose
and the code disagreed from the moment both existed and nothing compared them.
config.yaml and both translations additionally carried a claim that had rotted
the other way - "nothing consumes the price series yet", untrue since OPT-04
shipped the consumer. All four surfaces now say what the add-on does, and a
check pairs DOCS against the DRIVEN behaviour so they cannot drift again.

The gate is tariff_refresh_wanted() = provider_configured(opts) or
arbitrage.enabled(opts), and the `or` is a STRICT WIDENING: every configuration
that started the task before still starts it. The second term is not there for
symmetry. price_provider: off + a static_profile + arbitrage_enabled: true is a
configuration validate.py explicitly ACCEPTS and it trades - measured, at
cycle_cost_threshold_eur 0.05 on a 0.20/0.284 day/night profile,
ARBITRAGE_CHARGE -5000 W. Gating on the provider alone would leave that site's
tariff_plan at None forever and answer "no tariff plan has been resolved yet" on
every cycle, silently disarming a strategy the UI says is running.

Nothing acquires a periodic HTTPS call. With price_provider off, resolve_tariff()
answers out of the static profile without opening a socket (driven and counted:
0 requests), and the true shipped default - no provider, no trading - starts no
task at all. What decides whether a request leaves the machine is price_provider
and only that. The literal reading of the ticket's AC2 - drop the `or` - is one
token and is recorded in DECISIONS §3 with what it costs, for the owner.

AND NOTHING TRADES THAT DID NOT TRADE BEFORE. cycle()'s
`if arbitrage.enabled(self.o):` is untouched, and that is graded directly rather
than inferred from the gate having moved: Controller.cycle() is driven on a
resolved provider plan with the peak guard asserted clear, and arbitrage_enabled
is the only option that moves between the rows - false/false give +180 W
("loop -> 180 W") on both the cheap-now and dear-now plans, true/true give
-5000 W and +5000 W ("arbitrage -> ...").

TariffPlan.cheap_now() is deliberately NOT wired, decided rather than inherited
(DECISIONS §4). A provider-backed plan now exists on every configured site, so
Cheap window could read it - and Cheap window feeds the maintenance cycle's grid
top-up, the one place a price moves energy without trading being armed. On a
provider-only site static_status() returns False on every cycle, so adopting it
would take a paid grid top-up from never to sometimes. self.cheap stays on
static_status(); what adoption would take is listed rather than left implied.

THE CHECK THAT WOULD HAVE CAUGHT THIS is test_price.py section 13, and it grades
REACHABILITY FROM A CONFIGURATION rather than the shape of an `if` - the `if` was
exactly what somebody meant to write. amain()'s guards are EVALUATED against real
option dicts, whatever they are spelled as; the call graph under app/ is walked
to see whether the tasks those guards start arrive at resolve_tariff(), wherever
it has moved to; and then the fetch is DRIVEN with a counting transport, because
a graph is an argument and a counted request is an observation. What the
instrument cannot see is printed on every run, largest hole first.

Evidence: 29/29 green. 13 mutations in 14 runs - M10 re-run as M10b once it was
closed - each applied to a fresh cp -a of the clean worktree rather than
reverted (MUTATION.md habit 18), purged, :ro, -B. The
shipped defect - the gate back at arbitrage.enabled(opts), -4 B - is red on
seven named checks. Dropping the `or` (-27 B) is red on the strict-widening row;
removing cycle()'s trading gate (+8 B) is red on both AC3 rows. The instrument
is mutated too (habit 16): _reaches() always True and _task_sites() returning []
are both +0 B and both red.

Two findings are recorded rather than rounded off. provider_id() dropping
.strip() (-8 B) was a real SURVIVOR across four price suites - nothing fed a
padded provider id - and is closed in this commit by the padded-id pair; and the
first draft of the AC3 fixture fed five minutes of P1 telegrams, so whether the
peak guard was blind depended on where the wall clock sat inside the billing
quarter and the armed baseline failed to trade on some runs. It is 900 s now,
and the guard's clear state is a check rather than an assumption.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-12 20:58:48 +02:00
admin 88c59a9918 Merge pull request 'TARIFF-09: ask the feed for the horizon the operator configured' (#77) from TARIFF-09 into release/1.0
generic_dayahead.fetch() accepted since and until and ignored both, requesting the bare URL - so the feed returned one Brussels day against a 48 h horizon. Everything past today was carry-forward, provisional, and dropped from the trade set by 8.3, which means the optimiser never saw tomorrow as firm at any hour of the day. That is not what FCST-04's fill was built for: it covers the pre-publication window until the ~13-14 h CET auction, after which a healthy feed should hand over tomorrow in full.

Measured before, 2026-09-10T09:00+02:00 on the live feed: 15 firm hours and 33 carried of 48, firm local days = [2026-09-10] only, buy and sell both on the same day. After, against a real two-day body: 39 firm and 9 carried, firm local days = two, buy 13:45 on 09-09 and sell 19:45 on 09-10. At 23:00 the difference is the whole verdict - before, 1/47 and arbitrage REFUSED at spread 0.0464; after, 25/23 and allowed. The 9 hours still carried are the tail running into a third Brussels day, which no day-ahead market publishes.

The range is derived on the BRUSSELS calendar, both ends graded and both driven red by mutation - the API takes dates while since/until are instants, so deriving either end from the UTC date loses the last one or two hours of every day. That is _bdate()'s lesson from FCST-04 one layer up, except that here the hours are never fetched at all rather than misfiled.

price_feed_url is an EXACT URL, used verbatim, nothing appended - and the deciding argument came from the codebase rather than from taste: generic_dayahead declares a second egress host, web-api.tp.entsoe.eu, whose date parameters are not start/end. An override to the other DECLARED host is exactly where merging would append parameters that host ignores and produce the short horizon anyway. The cost is made visible by a fetch-time WARNING naming the consequence and the start..end it would have asked for, with a control asserting the default path stays silent, and dated_feed_url() refuses to append to a base that already carries a query string, so blind appending is structurally unreachable.

The plugin contract is deliberately NOT enforced yet. Obliging the three unverified-endpoint plugins of TARIFF-07 to pass a range would be satisfied by an invented parameter name - TARIFF-08's defect moved from the response to the request and shipped as a guarantee. The roster of plugins that consume the range is pinned as a literal instead, so the next supplier has to decide.

Evidence: 29/29 green. 11 mutations, no survivors, each applied to a fresh copy of the clean tree rather than reverted. Includes the Brussels/UTC boundary pair, one at +1 B, and one at +0 B that changes a single price digit inside the committed capture and is caught by the re-derived sha256. Two live fetches only, 66 s apart, respecting the feed's rate limit; the second returned bytes identical to TARIFF-08's capture, so the pre-publication case is a second recorded URL for the same bytes with the identity re-derived rather than asserted.

MUTATION.md habit 20: a fixture grades the response, and nothing about the REQUEST is reachable from one. canned() had recorded the URL since TARIFF-02 and no check ever asserted on it, which is how a plugin that never asked for a date range went unnoticed for four tickets.
2026-09-10 10:44:15 +02:00
Claude Opus 5 17b7b447c4 TARIFF-09: ask the day-ahead feed for the horizon it was handed
checks / unit (pull_request) Successful in 7m10s
`generic_dayahead.fetch()` accepted `since` and `until` and requested the
bare `?bzn=BE`, which answers ONE Brussels day. Everything past today was
then filled from yesterday's profile by FCST-04, flagged provisional, and
§8.3 drops provisional hours from the trade set -- so the optimiser never
saw tomorrow as firm, at any hour of any day, and nothing reported it.

Measured on 4257e38 at 2026-09-10T09:00+02:00, 48 h horizon, live feed,
yesterday seeded through the shipped read path:

    60 firm points (15 whole hours) + 33 carried hours of 48 requested
    carried: 24 h on 2026-09-11, 9 h on 2026-09-12
    arbitrage(0.10): buy 11:45 and sell 19:45 -- BOTH on 2026-09-10
    firm local days = [2026-09-10]

The ceiling was structural: with a one-day window, at hour H a 48 h horizon
is at most 24-H firm and 24+H carried. At 23:00 that is 1 and 47, and the
remaining firm spread (0.0464 EUR/kWh) no longer clears the default 0.10
threshold, so arbitrage is refused outright. FCST-04's fill covers the
window until the ~13:00-14:00 CET auction; it was never meant to cover
every hour of every day.

The data exists and the endpoint serves it: verified 2026-09-10T07:45:49Z,
`?bzn=BE&start=2026-09-09&end=2026-09-10` returns 192 quarter-hours across
two local days. Committed unmodified beside TARIFF-08's captures, with the
URL, retrieval time and sha256 recorded and RE-DERIVED by the replay.

THE RANGE IS DERIVED ON THE BRUSSELS CALENDAR, not the UTC one. 22:30 UTC
in summer is already tomorrow in Belgium, so a UTC-derived start asks for
the day the controller has just left and drops the day it is standing in,
for the last one or two hours of every day. That is `_bdate()`'s lesson one
layer up with a worse consequence: `_bdate()` misfiles an hour, this never
fetches it. `until` is exclusive, so local midnight belongs to the day
before it; an inverted window collapses to one day rather than a backwards
range. Both ends are graded and both are driven red by mutation.

After, same code path and window, a feed serving both days: 39 firm hours
and 9 carried (was 15 and 33), firm across TWO Brussels days, and the
permitted cycle buys on one day and sells on the next.

`price_feed_url` IS THE EXACT URL AND THE PLUGIN APPENDS NOTHING. `start`/
`end` are this feed's spelling and the provider declares a second host
whose parameters are different, so merging would produce a request nobody
typed and the short horizon anyway. The cost is real, so it is not silent:
a WARNING on the shipped logger at fetch time names the consequence and the
start..end that would otherwise have been asked for. `dated_feed_url()`
refuses to append to a base that already carries a query string, so the
blind-append shape cannot come back by edit.
DECISIONS/TARIFF-09-the-range-and-the-override.md argues the alternative.

Before publication nothing regresses and it is graded, not assumed: at
09:46 CEST the two-day request came back byte-identical to the one-day
body, and that stays a `series` of what exists with FCST-04 filling the
rest. A `no_data` there would switch a working site's arbitrage off every
morning -- the mutation that adds that rule is red by name.

The plugin contract is still NOT enforced: three shipped plugins point at
endpoints nobody has verified, and a rule obliging them to pass a range
would be satisfied by an invented parameter name -- TARIFF-08's defect
moved from the response to the request. Which plugins consume the range is
pinned as a literal instead.

11 mutations, all red, no survivors; each applied to a fresh copy of the
tree rather than reverted, through tools/mutation_run.sh. One is +0 B (a
single price digit inside the committed capture) and the re-derived sha256
catches it. MUTATION.md habit 20 records why a fixture could never have
reached this: a captured body is the answer to a URL, and no property of a
body says which URL was asked for. All 29 suites green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-10 10:35:12 +02:00
admin 4257e38e5a Merge pull request 'TARIFF-08: read the feed the add-on actually ships pointed at' (#76) from TARIFF-08 into release/1.0
_points_from_json() expected doc['data'] to be a list of {start, price_eur_mwh}. api.energy-charts.info/price?bzn=BE returns a columnar document - parallel unix_seconds and price arrays - so the shipped parser raised SourceUnavailable('the feed carried no data list') against every real response. generic_dayahead had never parsed one. With the three fabricated supplier URLs of TARIFF-07, that left manual_csv as the only working provider in a five-provider table.

It survived to release because every fixture fed the invented shape: test_price.py, test_price_history.py and test_static_tariff.py all built {'data':[{'start':...,'price_eur_mwh':...}]}, so the suite validated the parser against the parser's own assumption - a closed loop that cannot notice the assumption is wrong. Section 11.4 then made the failure silent: a parse error degrades to the static profile, which is correct and says nothing about the cause.

Verified live 2026-09-10, end to end: fetch -> parse -> assemble() -> TariffPlan returns 96 quarter-hourly points at 900 s, origin provider, product DAY_AHEAD, arbitrage allowed with a firm spread of 0.4645 EUR/kWh. First time the pipeline has completed against a real feed. Belgium moved to a 15-minute MTU on 2025-10-01, so FCST-04's H1 fix - the fill was decimating sub-hourly points to one per hour - was never theoretical; it is the shape of the live feed, and it is what makes this deliverable.

The columnar shape is read by a SECOND function, not a sniff, and feeding the row shape to generic_dayahead is now a graded no_data so the sniff cannot return quietly. Equal length refused in both directions with both lengths in the reason; the unit refused if not EUR/MWh, because MAX_PLAUSIBLE_EUR_KWH only bounds prices that are too large and a EUR/kWh switch would otherwise be invisible; nulls dropped with a count, an all-null body named as no_data; a deprecated feed warns while it still works.

Two real bodies captured as fixtures - unmodified bytes, with URL, retrieval time, unit, licence-as-returned and sha256 recorded, and the checksums RE-DERIVED by the replay rather than asserted. The invented fixtures are renamed synthetic_row_feed at every call site so the closed loop cannot re-form. Only one of four network providers has a capture, and a registry-derived check says so in its own sentence: the other three have nothing to capture, which is TARIFF-07.

Evidence: 29/29 green. 12 mutations, no survivors, purged tree and -B per TEST-15. One first-draft line was decoration and the mutation said so - the parser called brussels() per instant while assemble() already renders, so removing it changed nothing. The line was deleted rather than fenced with a check, and the mutation re-aimed at assemble() where the rule lives. MUTATION.md habit 19.

Follow-up filed as TARIFF-09: generic_dayahead ignores the since/until it is handed, so the feed supplies 24 h of a 48 h horizon and tomorrow is never firm.
2026-09-10 09:14:50 +02:00
adminandClaude Opus 5 7a5281e97f Merge release/1.0 (FCST-06) into TARIFF-08
checks / unit (pull_request) Successful in 7m10s
One conflict, in CHANGELOG.md's `## Unreleased` block, exactly as expected
from two branches landing beside each other: both `_this branch_` table rows
and both sections are kept, FCST-06's first. Nothing else conflicted -
FCST-06's `test_price_history.py` section 5d and this branch's columnar
`feed_body()` merged cleanly - and all 29 test files are green on the result.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-10 09:06:39 +02:00
adminandClaude Opus 5 6eba18968c TARIFF-08: the day-ahead feed is read in the shape it actually publishes
checks / unit (pull_request) Failing after 3s
`generic_dayahead` could not read one byte of `api.energy-charts.info`, and
never had been able to. `_points_from_json()` asks for `doc["data"]`, a list of
`{start, price_eur_mwh}` objects; that document does not exist. Fetched live on
2026-09-10 the feed returns COLUMNAR parallel arrays and no `data` key at all -
`unix_seconds` (96 UTC epoch seconds, 900 s apart), `price` (96 floats),
`unit: 'EUR / MWh'`. Driving the shipped parser against that body:
`SourceUnavailable: the feed carried no 'data' list`. It is the only shipped
provider whose endpoint anybody has reached, so dynamic pricing did not work.

AND §11.4 MADE IT SILENT, which is why it survived to release: a parse error
becomes no_data, becomes the static profile, becomes arbitrage off - the
CORRECT degradation, saying nothing about the cause. A site would have sat on
the fallback indefinitely.

AND THE SUITE COULD NOT SEE IT, because every fixture was built from the
parser's own assumption. test_price.py, test_price_history.py and
test_static_tariff.py each built `json.dumps({"data": [...]})`, a shape invented
here and published by nobody, so the checks graded the parser against the
parser's guess - a closed loop no coverage inside it could ever have escaped.
MUTATION.md habit 19 records it as the fourteenth instance of habit 4 and the
first where the fixture and the code under test shared a SHAPE, not a number.

WHAT CHANGED

* `_points_from_columnar_json()` - a SECOND function, not a sniff. A parser that
  decides which document it received is one that can decide wrong, and a
  wrong-but-parseable response then becomes a price. Feeding the row shape to
  `generic_dayahead` is now a graded no_data.
* Unequal arrays are refused in both directions, with the two lengths in the
  reason. `zip()` stops at the shorter list silently, and a truncated `price`
  array would have produced a real series missing the evening peak - traded on
  as complete, where no data at all switches arbitrage off.
* A unit that is not EUR/MWh is refused. The plugin converts with a fixed
  0.001, and `MAX_PLAUSIBLE_EUR_KWH` only bounds prices that are too LARGE.
* `unix_seconds` are read as UTC instants explicitly; §9.3's Brussels rendering
  stays in `assemble()` rather than being copied into the plugin, where a
  mutation could not tell it from a no-op (M4 proved exactly that on a first
  draft, and the line was removed rather than fenced).
* A published null is dropped and logged with a count; an all-null body is
  no_data named by this parser rather than by the boundary validator.

FIXTURES

Two unmodified response bodies under `scenarios/`, captured by hand on
2026-09-10: quarter-hourly (2026-09-10, 96 points at 900 s) and hourly
(2025-09-10, 24 at 3600 s - the same endpoint before Belgium's 15-minute MTU).
`energy-charts-be-price-captures.json` records the URL, retrieval time, unit,
licence-as-returned and a sha256 of each, and the replay RE-DERIVES the
checksums, so a fixture edited to make a check pass stops being a capture. Data
CC BY 4.0, Bundesnetzagentur | SMARD.de, via Energy-Charts (Fraunhofer ISE),
published without changes - the same BE-zone licence check DATA-02 established.

The surviving invented fixtures are renamed `synthetic_row_feed` at every call
site. `engie_dynamic`, `totalenergies_dynamic` and `eneco_eddy` are untouched
and have no capture: their URLs are unverified and withdrawing them is
TARIFF-07's open decision. Nothing should read their green checks as evidence
that those feeds parse.

MUTATIONS - 12, all red, no survivors. Purged tree, mounted `:ro`, `python3 -B`
(TEST-15). M1 drop the equal-length refusal; M2 make it one-sided; M3 drop the
unit check; M4 `assemble()` stops rendering at the Brussels offset; M5 read
`unix_seconds` as a naive local clock; M6 zero-fill a null instead of dropping
it; M7 drop the all-null refusal; M8 sniff for the row shape; M9 drop the
deprecated warning; M10 drop the dropped-nulls warning; M11 edit one price in
the committed capture; M12 point `generic_dayahead` back at the row parser -
red across all three price suites.

END TO END, LIVE, 2026-09-10: `resolve_tariff()` returns a `TariffPlan`,
`origin='provider'`, 96 quarter-hourly points, spacing [900] s, no decimation,
first point 2026-09-10T00:00:00+02:00 at 0.30608 EUR/kWh all-in. FCST-04's H1
fix was never theoretical. All 29 test files green.

STILL OPEN: `generic_dayahead` asks for no date range, so the default URL
returns one published day - a full 48 h horizon is reached only through §8.3's
carry-forward fill.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-10 08:57:32 +02:00
admin 958f5cd218 Merge pull request 'FCST-06: cannot-persist is one question, asked after the attempt' (#75) from FCST-06 into release/1.0
The module claimed 'a store that cannot persist records nothing' and enforced it against `not self.ok` only. A store with ok=True whose _save() fails - a full volume, a read-only remount - kept answering yesterday() from memory until the next restart, which is the silently-forgetting cache the docstring rejects, reached by the condition the rule did not name.

Reproduced against real volumes rather than a monkeypatch: /data bind-mounted read-only (EROFS) and a full tmpfs (ENOSPC), with a healthy-/data control asserted either side. Before: yesterday() returned a 24-point series while the disk held {}.

Fixed in the code, not the docstring - weakening the doc would have documented a cache that forgets without saying so as intended.

The design answer is the part worth keeping: 'can this store persist' CANNOT be answered before the attempt, because a directory that was writable a moment ago is not a write that completed, so os.access or a probe write is a third symptom test wearing a predicate's name. It is answered after, once: _save() -> bool, true only if the file now holds what memory holds. The `not self.ok` test in _remember() is GONE rather than joined by a second one, and a check asserts its absence from the source. This is the same shape as FCST-04's M4 and FCST-05's HIGH - a rule enforced on the conditions someone thought of - and it is the first of the three to be closed by removing a test rather than adding one.

What is undone is the MERGE, not the store: a volume that fills at 14:00 leaves the yesterday written at 13:30 still answering. The snapshot is a deepcopy - measured, a shallow copy let a store holding 12 persisted points answer 24.

Evidence: 29/29 green. Six mutations, no survivors, purged tree each run with the mutated line greped off disk. One of them exposed an instrument defect now recorded in MUTATION.md: it printed 1 FAIL when the true figure was 3, because an f-string operand called len(read.points) and .points on a no_data raises Absence by design, killing the harness mid-section. One check earlier and it would have read as a survivor. A mutation's red count is a lower bound until the run is seen to reach the end of its section.
2026-09-10 08:52:09 +02:00
adminandClaude Opus 5 c45893ffad FCST-06: "cannot persist" becomes one question, asked once, about the outcome
checks / unit (pull_request) Successful in 7m14s
price_history.py's docstring states its rule unconditionally - "A store that
cannot persist now records nothing, and both of its answers say the same
thing" - and FCST-04 enforced it against `not self.ok` only, which is one
SYMPTOM: the durable volume is missing. `ok` is about the DIRECTORY, and a
read-only remount and a full disk both leave it True. Reproduced on 5059e28
against two real volumes, a /data bind-mounted read-only and a full tmpfs:

    last_error = 'could not write ([Errno 30] Read-only file system)'
    words      = '2026-01-14 (24 points over 24 h); 2026-01-15 (24 points
                  over 24 h) [could not write (...)]'
    yesterday()= series, 24 points
    ON DISK    = {}

`words` and `yesterday()` were describing two different stores, and the 24
points were gone on the next restart with nothing saying so - the
silently-forgetting cache that rule exists to refuse, reached by the condition
it did not name. No shipped path reaches it: /data is the Supervisor's durable
volume and this module is its only writer. The defect is that a guarantee the
module states about itself did not hold in one condition.

Route taken: `_remember()` refuses to retain what it could not write, and the
docstring's claim stands as written. It is now ONE derived predicate rather
than two places testing two symptoms. "Can this store persist?" cannot be
answered BEFORE the attempt - a directory that existed and was writable a
moment ago is not a write that completed - so it is answered after it, once,
by the only thing holding the evidence: `_save()` returns whether the file now
holds what memory holds. No volume is one False, an OSError is another.
`_remember()` keeps only what that one answer says was kept, and the
`not self.ok` test in `_remember()` is GONE rather than joined by a second one.
FCST-04's M4 and FCST-05's HIGH were the same shape, so a third cause now needs
no third branch.

What is undone is the MERGE, not the store. The state restored is the last one
that IS on disk, so a site whose volume fills at 14:00 goes on answering the
yesterday it wrote at 13:30 - that record survives a restart, so answering it
is not a promise the store cannot keep. It stops adding to it. The snapshot is
copy.deepcopy: a one-level copy shares the day record the merge rebinds
`points` on, and measured with one, a store holding 12 persisted points
answered 24 after a failed merge.

Not fatal, deliberately: §6 and this module's opening - a broken cache must not
be a broken controller, and an add-on that will not start because /data filled
up is a worse defect than this one. `_save()` reports and returns; the shipped
resolve_tariff() path still gives a provider plan with no §11.4 degradation.
What changed is that the failure is now acted on rather than only logged.

test_price_history.py section 5d drives it on two REAL failures with different
syscalls and different errnos - a directory where the file goes (os.replace,
EISDIR) and a volume moved out from under an open store (mkstemp, ENOENT) -
because the rule is enforced on the outcome and one cause would grade a
coincidence. DOCS.md's "where there is no /data" sentence is widened to the
condition the code now holds to.

MUTATION.md gains what M4 exposed: a mutation whose red is a TRACEBACK reports
however many checks ran before it and grades none after it. M4 first printed
1 FAIL; the true figure is 3. An f-string operand read `len(read.points)`, and
`.points` on a no_data raises `Absence` by design - why()'s rule arriving in
what a check PRINTS rather than in what it asserts. `npts()` is now beside
why(), and a red count is a lower bound until the run is seen to reach the end
of its section.

6 mutations, purged tree, every one RED on named checks, no survivors:
M1 self._days = before -> dict(self._days)          +10 B  9 red
M2 _save() OSError handler returns True             -1 B   9 red
M3 _save() returns True on `not self.ok`            -1 B   3 red
M4 self._days = before -> self._days = {}           -4 B   3 red
M5 copy.deepcopy -> dict                            -9 B   3 red
M6 `if not self.ok: return` back in _remember()     +43 B  1 red

All 29 test files green. Nothing deployed; no option added; config.yaml and
app/ behaviour on a healthy /data untouched. No version number picked - that
is the open decision the Unreleased block names.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-10 08:43:17 +02:00
admin 5059e28860 Merge pull request 'FCST-05: one byte stops stopping the add-on, and the write path gets the read path's standard' (#74) from FCST-05 into release/1.0
Both defects were found by the QA pass on merged FCST-04, and both are the same shape: a corrupt price cache taking the controller with it.

HIGH: _load() caught (JSONDecodeError, OSError) and UnicodeDecodeError is neither - it is a ValueError. One non-UTF-8 byte in /data/price_history.json escaped to Controller.__init__, which is unguarded, and the add-on did not start. The except is now (ValueError, OSError) - the content family beside the bytes family - rather than the two members known today, because a third ValueError subclass nobody thought of is exactly how this arrived. The new fixture writes real bytes and asserts the old fixture's string IS valid UTF-8, so the pair cannot collapse back into one.

MEDIUM: a malformed `start` under the day the feed was writing raised out of _remember() on every refresh forever - the add-on stayed up while the plan retired after 90 min and arbitrage stayed off, with no self-heal, and `words` reported the store healthy throughout. Read and write paths now share one validator, and the merge key is the validated instant (which also fixes a lexicographic sort putting the 25-hour day's second 02:00 first).

Also: the row counts what it says it counts, and two false claims of strengthening in _audit's docstring are corrected - a differential over the realisable domain returns 0 weaker, 0 stronger, and that differential is now a committed check with FCST-03's audit frozen beside it. PROC-01: a right decision with a fabricated reason is still a defect.

_audit completeness was NOT attempted, and the reason is measured rather than asserted: the rule as stated takes the horizon checks red on the shipped fill and a healthy feed, because `published` is the whole read while the fill emits only [since, until). The rule is wrong, not merely absent; making it right needs the window in the signature. Separate card.

Evidence: 29/29 green. 12 mutations, all red, each on a fresh tree copy fingerprinted against clean. One survivor was found and closed during the work - two rungs reported into one last_error, so a -1 byte deletion on the half of the fix that tells an operator the cache self-healed survived the whole suite. TEST-16's masking pattern, inside a check written to close a false-health defect. Recorded in MUTATION.md.
2026-09-10 01:13:41 +02:00
glenn schrooyenandClaude Opus 5 375a1e29d2 FCST-05: one byte stops stopping the add-on, and the write path gets the read path's standard
checks / unit (pull_request) Successful in 7m10s
HIGH - a non-UTF-8 byte in /data/price_history.json stopped the add-on
from starting. `_load()` caught `(json.JSONDecodeError, OSError)`.
`UnicodeDecodeError` is `ValueError -> UnicodeError -> UnicodeDecodeError`:
neither of those, and `issubclass(UnicodeDecodeError, OSError)` is False. So
it escaped `_load()` -> `_open()` -> `PriceHistory.__init__`, which
`Controller.__init__` does not guard - breaking `_load()`'s own stated
contract, "a corrupt cache must not stop the controller". The handler now
names the ValueError FAMILY (the file's content) beside OSError (its bytes),
because a third subclass nobody has thought of is how this arrived the first
time. The suite could not see it: its "unreadable file" fixture writes
"{ this is not json", which is valid UTF-8 and fails one step later, in
`json`. A fixture holding a genuinely undecodable byte is now beside it,
with the difference between the two asserted as a precondition.

MEDIUM - a malformed `start` under the day the feed is writing to killed the
tariff plan permanently. `_remember()` merged with `{q["start"]: q ...}` and
sorted on `q["start"]`, trusting a value read off /data to be a hashable,
sortable string. An int, null, a dict or a list raised TypeError out of
`carry()` -> `resolve_tariff()` -> `run_tariff_refresh`, which catches
Exception - so the add-on stayed UP while every refresh raised forever, the
plan retiring after TARIFF_MAX_AGE_S with arbitrage and the cheap window off
for good, and no self-heal because `_remember()` raised before `_save()`.
The read path was properly defensive throughout, which is why `yesterday()`
survived and the existing fixture passed - it files its malformed rows under
a day the feed is NOT writing to. Both paths now share one validator,
`_point_of()`. A row that is not a point is dropped, reported, and written
back out clean.

...and the status row said the store was healthy while the plan was dead:
`words` read '2026-01-15 (1 h)'. `_load()` now counts the rows inside a kept
record as well as the record itself, so the store stops reporting plain
health from the moment it is opened; the note names what was FOUND at open
and what was DROPPED by the refresh, and clears once the file is rewritten.

...and the merge key is the validated INSTANT rather than its spelling,
which is what rule 4 already said. The lexicographic sort beside the string
key also put the second 02:00 of the 25-hour October day (+01:00) before the
first (+02:00); stored order only, since `_parse_points()` re-sorts.

LOW - the status row counted POINTS and printed them under the HOUR's name:
'2026-01-13 (96 h)' for a 24-hour day on the quarter-hourly feed this add-on
already supports. It is now '96 points over 24 h' - two facts, each under
its own unit.

LOW - two stated reasons were false. `_audit`'s "WHY IT IS STRICTER" claimed
the rework refused the same published point emitted twice and a carried
point sharing a published hour, "which the old count never looked at". A
differential over the whole domain the audit can express finds ZERO
divergence in both directions: the old `seen` count already refused the
first (one point twice is one hour twice) and the old IDENTITY rule already
refused the second. The reworked audit is EQUAL to FCST-03's, not stronger;
the only behavioural change is the intended relaxation, and it stays. The
differential is committed as a check rather than left as prose.

...and the audit's real limit is named: it does not check COMPLETENESS. Fed
the decimation defect FCST-04 was raised for it ACCEPTS, and it accepts an
empty output. So `_audit` is not what stops that recurring - the count and
identity checks are, and the text now points at them. Not made complete:
`published` is the whole read while the fill emits only [since, until), so
the rule as stated is wrong, and building it takes the 1/12/48 h horizon
checks red on a healthy feed. Measured, not assumed.

12 mutations, purged tree, every one RED, no survivors. All 29 test files
green. Nothing deployed; no option added; config.yaml untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-10 01:04:30 +02:00
admin c2787da539 Merge pull request 'FCST-04: the carry-forward fill reaches the plan, and the day it carries survives a restart' (#73) from FCST-04 into release/1.0
Review found 2 HIGH, 4 MEDIUM, 12 LOW - all driven to a concrete failure, all fixed on the branch.

H1 was a regression this PR introduced: fill_from_yesterday keyed the published set as {_utc_hour(p): p}, so a sub-hourly feed lost three points in four. eneco_eddy is the Belgian imbalance product and settles per quarter. Measured on 96 quarter-hourly firm points: 96 -> 15 points, spread 0.143 -> 0.014 EUR/kWh. The fix passes every published point through by identity and carries only whole hours with no coverage at all; _audit's one-per-hour rule became two rules that subsume it, so it caught MORE after the change than before - a repeated published point and a carried point sharing an hour with a real print both now raise, and neither was checked before.

H2 falsified the branch's own 'a broken cache is not a broken controller': a day record that parsed but was not a dict raised out of words(), which is unguarded in Controller.__init__, so the add-on did not start and could not self-heal.

Two of the branch's own checks were decoration and are recorded as such in DECISIONS rather than quietly repaired: the prune mutation survived because the fixture seeded one day and never reached the code it graded, and the AST ordering check compared ast.walk BFS depth rather than source order - shipped and reversed scored identically. Both are now driven by behaviour.

Evidence: 29/29 test files green. 20 mutations driven across the rework, fresh tree copy and __pycache__ purge before each per MUTATION.md/TEST-15. No survivors.
2026-09-10 00:03:08 +02:00
glenn schrooyenandClaude Opus 5 b59f5ce31b FCST-04: the carry-forward stops decimating a quarter-hourly feed, and a broken cache stops stopping the add-on
checks / unit (pull_request) Successful in 7m22s
Rework of the FCST-04 branch against review. Every finding below was proven by
a runnable script before it was touched and re-run after; every check added or
repaired was driven red by a mutation, `__pycache__` purged first (MUTATION.md,
TEST-15). 29/29 test files green.

⚠️ H1 - A SUB-HOURLY FEED WAS DECIMATED TO ONE POINT AN HOUR, AND THE SPREAD
WENT WITH IT. `fill_from_yesterday()` keyed the published set as
`{_utc_hour(p.start): p for p in r.points}` - LAST POINT OF THE HOUR WINS, the
other three dropped. `eneco_eddy` is the Belgian imbalance product and settles
per QUARTER, and `_validate_raw()` accepts any strictly-increasing spacing, so
this is a shape the product supports today. Measured on a 96-point quarter-hourly
feed - ALL FIRM, FULLY PUBLISHED, nothing to carry: the plan reached the
optimiser as 15 points, the arbitrage spread fell from 0.143 to 0.014 EUR/kWh,
and the cheapest "hour" became a `:45` point whose `at()` window runs into the
next one. Not the windowing effect this card is about; a feed losing 84 % of its
own prints, with no flag anywhere reporting it. The fill now decides only WHICH
WHOLE HOURS NEED CARRYING: every published point inside the window passes
through BY IDENTITY, and a carried point is added only for a whole hour with no
published coverage at all. Same feed, same fixture: 60 of 60 firm points kept,
spread 0.134 over the hours still ahead.

⚠️ AND `_audit()` GOT STRICTER, NOT LOOSER - that was the condition on doing it
at all. The old post-condition refused any repeated `_utc_hour`, a count that
cannot express four points in one hour. Two rules replace it and SUBSUME it: a
published point must BE one of that hour's published points by identity and may
be emitted only once; a carried point may stand only in an hour absent from the
published set, must be flagged True, and may not repeat. So the duplicated
carried hour it existed to catch still raises, the same published point emitted
twice now raises, and a carried point sharing an hour with a real print now
raises too - which the count never looked at. Three mutations, three reds.

⚠️ H2 - A DAY RECORD THAT WAS NOT A RECORD STOPPED THE ADD-ON FROM STARTING.
`_load()` validated `version` and that `days` is a dict, never that each VALUE
is one. `{"version": 1, "days": {"2026-01-14": "oops"}}` loaded as ok=True,
last_error=None, and then `rec.get(...)` raised AttributeError out of `words`,
`yesterday()` and `resolve_tariff()` alike. `PriceHistory(...)` and `.words` are
both unguarded in `Controller.__init__`, so the controller did not come up - and
it never self-healed, because `_remember()` raised on the same record before it
could reach `_save()`. That falsifies the CHANGELOG's "A BROKEN CACHE IS NOT A
BROKEN CONTROLLER" in the one direction that matters. The asymmetry that
produced it is visible in the file: `_parse_points()` is carefully defensive
about every field of a POINT and `_load()` was not defensive about the record
holding them. Unusable records are now dropped and REPORTED (`words` says how
many and why), and `words` is total by construction as a second line.

M3 - `yesterday()` WAS OFF BY ONE DAY FOR 00:00-00:59 AFTER SPRING-FORWARD.
`brussels()` renders at the FIXED offset for that instant, so `brussels(stamp) -
timedelta(days=1)` subtracts 24 ABSOLUTE hours - and the Brussels day before the
March transition is 23 hours long. At 2026-03-30T00:30+02:00 it returned
2026-03-28's prices as "yesterday", or `no_data` naming a date that is not
yesterday. The arithmetic is now on the local DATE, which has no offset to lose,
and both transitions have a check - the module's headline claim had none.

M4 - WITH NO /data THE STORE STILL FILLED, IN MEMORY. `_save()` short-circuited
on `not ok` while `_remember()` populated `self._days` anyway: `words` said
"unavailable" and `yesterday()` answered A SERIES that is gone on the next
restart. The module docstring rejects exactly that ("a yesterday that silently
forgets") as its reason for not calling makedirs, so the code now matches the
docstring rather than contradicting it.

M5 - THE PRUNE MUTATION SURVIVED BECAUSE THE STIMULUS WAS DEGENERATE. §8's
clock-jump check seeded ONE day, so `if len(self._days) <= KEEP_DAYS: return`
fired and the body it is named after never ran: pruning by arithmetic against
the clock - which empties the store on exactly the morning the fill is for -
passed the whole 29-file suite. The fixture is now written to a file and
reopened (`_load()` does not prune, which is how a store legitimately holds more
than the cap) with KEEP_DAYS + 2 days, and the mutant is red.

M6 - THE AST ORDERING CHECK COULD NOT CATCH A REVERSAL, AND TWO DOCUMENTS CITED
IT AS A GUARANTEE. `_calls.index("_remember") < _calls.index("yesterday")` grades
`ast.walk` BFS DEPTH: `self._remember(...)` is a statement and
`self.yesterday(...)` is an ARGUMENT of the fill, one level deeper, so
`_remember` came out first whichever statement was written first - shipped and
reversed both scored True. It is now line numbers AND, the half that matters,
the behaviour: a feed carrying yesterday alongside today into an EMPTY store
must fill the whole horizon; reversed, the lookup finds nothing and the plan is
only the published hours. Reversing the two statements is red on both.

L14/L9 - KEEP_DAYS = 3 IS THE STEADY STATE EXACTLY, so a newest-N cap has no
slack: a feed publishing to D+2, or ONE stray point in the wrong year - which
sorts LAST and can never age out - pushes yesterday out for good. `_prune()` now
PINS the two days a lookup can actually ask for on top of the cap. Pinning is
additive, so it can only RETAIN and the WITNESS-03 property (nothing is deleted
by clock arithmetic) is preserved by construction; `stamp` stops being a dead
parameter; the store is bounded at KEEP_DAYS + 2.

L8 - the horizon end was not floored while `since` was, so a 48 h request made
at any instant off the hour returned 49 points. §1's count was green only
because its NOW was exactly :00:00, a shape production never has; the fixture
now sits at :17:43. L10 - a failed `_save()` leaked its temp file; at 48 writes
a day a persistent write failure leaked 48 inodes a day onto the volume that was
already the problem. L15 - `{"version": 1.0}` passed, since `1.0 == 1`. L16 -
the `_remember` tree walk scanned `app/*.py` only and matched a bare string; it
now parses `app/`, `tools/`, `scenarios/` and `estop/` and looks for the NAME,
so a comment does not fail it and a call in `tools/` cannot pass unseen.

CLAIM-ACCURACY DEFECTS, WHICH THIS REPO TREATS AS REAL:

* L7. DOCS said "*Tariff* on the status page reads `provisional` through the
  morning". FALSE - `tariff_words` is assigned only from the HA price-entity
  branch and from `static_status()`, both on the 30 s tick, and the plan's own
  sentence reaches `_LOG.info`. The doc now says what an operator can see (the
  Price history row, the Arbitrage row's own refusal sentence, the refresh log
  line) and a check pins the PAIRING - wiring the plan into the row is fine,
  correcting the doc is fine, doing one without the other is red.
* L11. "fills EVERY unpublished hour" is conditional: a partial yesterday leaves
  the local hours it never held out of the series (§11.4 for those hours, and
  `at()` raises rather than answering). Correct behaviour, overclaimed wording,
  and untested - all three fixed.
* L12. DECISIONS §1 leaned on `TariffPlan.cheap_now()`, which has NO PRODUCTION
  CALLER on the provider plan (`self.cheap` comes from `static_status()`, which
  builds its own). The conclusion holds on `arbitrage()`'s firm-pair refusal
  alone, which does reach control; the evidence is now stated at the reach it
  has.
* L13. §1's "passed through by identity" check compared `eur_per_kwh` between
  two SEPARATE `resolve_tariff()` calls, where `is` could never have held. The
  value comparison is kept and now says what it grades; identity is graded on a
  read the file constructs and still holds.
* L17. `PriceHistory.words` and the new status row had no check at all - and
  `words` is one of the two things H2 crashed. Both now have one.
* L18. main.py's inline note carried the "widens what the optimiser can SEE,
  never what it may act on" guarantee without the windowing caveat that the
  CHANGELOG, DOCS and `resolve_tariff()`'s docstring all carry. It is the one
  place a maintainer reads it unqualified.

EVIDENCE. All eight reviewer proofs re-run: p1/p1b return 29 March, p4 keeps
60 of 60 firm points at spread 0.134, p5 starts and reports, p6 D leaves no
temp, p6 E refuses 1.0, p6 F records nothing, p7 returns 48 for 48. p2's script
still scores True/True because it re-implements the OLD algorithm - the
reversal is now driven as a mutation instead, and is red. p3's script always
showed the shipped code correct; what was wrong was the fixture, and its mutant
is now red. Twenty mutations driven, no survivors, tree fingerprinted and
`__pycache__` purged before each: decimation (2 suites red), audit x3, unfloored
`until` (2), unvalidated day records (7 checks), instant-subtraction yesterday,
in-memory fill on a volume-less store, clock-arithmetic prune, cap without the
pin, leaked temp, loose schema type, empty `words` (5), reversed `carry()` (2),
a `_remember` leak in tools/ spelled two ways, a neighbour-filling fallback, a
copy instead of identity (2), the DOCS claim in both directions, and the caveat
removed from main.py. Baseline and the unmutated tree both green.

No option added, config.yaml untouched, and the CHANGELOG entry still picks no
version: PKG-07's rule, and only the deployer knows whether 0.9.0's image has
been pulled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-09 23:54:23 +02:00
glenn schrooyenandClaude Opus 5 0618194d9c FCST-04: the carry-forward fill reaches the plan, and the day it carries survives a restart
checks / unit (pull_request) Successful in 7m11s
FCST-03 shipped `fill_from_yesterday()` complete and checked, and shipped the
consumers written for it - `TariffPlan.firm_points()`, `cheap_now()`'s refusal
of a provisional hour, and `arbitrage()`'s refusal 3, which drops provisional
hours from the trade set rather than refusing the whole series. It shipped NO
CALLER. Nothing in `app/` named the fill, so no series ever contained a carried
point and every one of those consumers was exercising its all-firm branch.

⚠️ THE DEFECT THAT PRODUCED WAS NOT AN ERROR ANYWHERE. `load_prices()` returns
what the feed published and nothing widened it to the requested horizon, so on
a dynamic site between midnight and the 14-16 h CET publication the plan
covered the rest of today and nothing more - measured on the fixture here, 24 h
against a 48 h request. `arbitrage()` then took `max - min` over that truncated
window and reported a spread. The number was real, the window was half the one
configured, and nothing said so: tomorrow evening's peak, the hour a morning
charge would be sold into, was not in the series to be compared against
anything. A missing hour is not a wrong price - it is an hour the optimiser
never learns exists.

WHAT THIS ADDS. `app/price_history.py` keeps the PUBLISHED prices of the last
KEEP_DAYS Brussels days in /data/price_history.json; `resolve_tariff()` gains a
required keyword-only `history` and fills every unpublished hour of the horizon
from yesterday's same-wall-clock hour, flagged. Persistence rather than a
variable on the refresh task, because the window the fill exists for is
midnight -> publication and an add-on restarted at 08:00 would otherwise spend
all of it on the truncated series. Not in state.json: the refresh runs every
1800 s and that file holds the maintenance history and SAFETY-02's latch.

⚠️ ONE CHANGE HERE MOVES ENERGY, AND IT IS NOT THE ONE THE CARD IS NAMED FOR.
Carried hours are provisional, so §8.3 keeps them out of the trade set: the
fill widens what the optimiser can SEE, never what it may DO, and
test_price_history.py checks that so it cannot be overclaimed later. What does
change a trade is that the filled series is windowed to [since, until), so
hours that have ALREADY ELAPSED are no longer in the spread. Before, a cheap
night that had gone sat in the series as the day's cheapest hour - and a cycle
cannot buy in an hour that is past, so the trade was quietly never taken.
Section 2 grades this by finding the cheapest firm hour with and without the
fill and showing it MOVES, from 02:00 UTC to 08:00 UTC on the fixture.

⚠️ A DEAD FEED IS STILL §11.4, NOT YESTERDAY. The fill runs only on a `series`
read. The two are indistinguishable to the optimiser - both 100 % provisional,
so arbitrage refuses on the firm-pair rung and `cheap_now()` is False - so
preferring yesterday's feed over the `static_profile` an operator configured
FOR that case would change a documented degradation for a difference only the
status page can see. That, and two more, in
DECISIONS/FCST-04-the-day-that-is-carried.md.

⚠️ THE ANTI-DECAY RULE IS AN ORDERING, NOT A VALIDATION, and saying so is more
useful than a guarantee that would not hold. Feeding the filled series back in
is how this rots - day 2 carries day 1's guesses, day 3 carries day 2's carry
of them, behind a `provisional` flag that stays TRUE while the NUMBER decays.
No value-level check can catch it: a carried point and `eneco_eddy`'s
published-provisional quarter are the same three fields with the same flag. So
`carry()` is the only public entry, it records BEFORE it fills, the filled
series exists only as a return value the object never sees again, and
test_price_history.py walks app/ for any other module naming `_remember` and
reads `carry()`'s AST to pin the order.

EVIDENCE. 29/29 test files pass in the CI image. Eight mutations driven against
app/price_history.py and app/static_tariff.py, __pycache__ purged before each
run per MUTATION.md/TEST-15: newest-record lookup (5 red), recording the filled
series (1), replace-instead-of-merge (10), UTC-date keying (10), removing the
fill call entirely (12), a lost price field defaulting to 0.0 (2), prune by
clock arithmetic (1), a degraded reason that stops naming its cause (1). No
survivors; baseline and revert both green. Three of those first crashed the
harness instead of reporting a FAIL - "it was refused" and "it broke on the way
to being refused" must not grade the same - so `why()` and `_priced()` were
added and the run repeated.

No option added, config.yaml untouched, and the CHANGELOG entry deliberately
picks no version: PKG-07's rule, and only the deployer knows whether 0.9.0's
image has been pulled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
2026-09-09 19:37:45 +02:00
admin 587751b115 Merge pull request 'INSTR-01: grade diff()'s suppression rule at the level it is written at, one case per clause' (#72) from INSTR-01 into release/1.0 2026-09-09 01:05:36 +02:00
25 changed files with 7567 additions and 98 deletions
+569
View File
@@ -1,5 +1,574 @@
# Changelog
## Unreleased
⚠️ **THIS HAS NO VERSION AND CANNOT REACH AN INSTALL UNTIL IT GETS ONE.**
PKG-06's and PKG-07's rule: the Supervisor keys the built image by `version:`,
so `ha apps update` is a silent no-op on an unbumped change. This branch
deliberately does **not** pick the number, because the right one depends on a
fact the branch cannot observe - whether 0.9.0's image has already been built
and pulled. If it has not, this belongs inside 0.9.0's section before it ships;
if it has, it needs 0.10.0. Whoever deploys decides.
| | |
|---|---|
| _this branch_ | FCST-04 - the carry-forward fill reaches the plan, and the day it carries survives a restart |
| _this branch_ | FCST-05 - one byte stops stopping the add-on, and the write path gets the read path's standard |
| _this branch_ | FCST-06 - "cannot persist" stops meaning "has no /data" and starts meaning what it says |
| _this branch_ | TARIFF-08 - the day-ahead feed is read in the shape it actually publishes, and the fixtures stop agreeing with the parser |
| _this branch_ | TARIFF-09 - the day-ahead feed is asked for the horizon it was given, so tomorrow arrives firm instead of carried |
| _this branch_ | TARIFF-10 - configuring a price provider is what fetches prices; trading stays its own switch |
| _this branch_ | TARIFF-12 - a supplier outage stops deleting the plan on its first failed fetch, and TARIFF_MAX_AGE_S starts governing |
### TARIFF-10 - a configured provider is fetched, whether or not you trade
**`amain()` started the background tariff refresh only when
`arbitrage_enabled` was on, and that option ships `false`.**
`run_tariff_refresh` is the only caller of `resolve_tariff()`, which is the
only caller of `price_history.carry()` - so on a shipped install with a price
provider configured **nothing was ever fetched**: no plan resolved,
`/data/price_history.json` never created, the *Price history* status row
reading `empty` for the life of the install, and TARIFF-09's short-horizon
warning unable to fire because there was no fetch to warn about.
Measured over a 40 h soak of two instances side by side on the same feed:
| | shipped default + provider | same, arbitrage armed |
|---|---|---|
| feed requests | **0** | fetching |
| `/data/price_history.json` | `present: false` on **all 802 samples** | written |
| resolved plan | none | 132 points |
⚠️ **`DOCS.md` has described the fixed behaviour since the commit that wrote
the gate.** OPT-04 (`bdd7f18`) added the refresh task, the
`arbitrage.enabled()` guard on it, and the sentence *"turning a provider on
still fetches, validates and publishes prices and changes no charging
decision"* — all in one change. `config.yaml` has said the same since
TARIFF-02 (`852fc7c`). The documentation and the code have therefore disagreed
for as long as both existed, and nothing compared them. `config.yaml` and both translations carried the same
claim, plus a second one that had gone stale in the other direction (*"nothing
consumes the price series yet"*, untrue since OPT-04). All four now say what
the add-on does.
**The gate is `tariff_refresh_wanted()`, and it is a STRICT WIDENING.**
```
provider_configured(opts) or arbitrage.enabled(opts)
```
Every configuration that started the refresh before still starts it. The second
term is not redundant and is not there for symmetry: `price_provider: off` with
a `static_profile` and `arbitrage_enabled: true` is a configuration
`validate.py` explicitly **accepts**, and it trades - measured, at
`cycle_cost_threshold_eur: 0.05` on a 0.20/0.284 EUR/kWh day/night profile,
`ARBITRAGE_CHARGE -5000 W`. Gating the refresh on the provider alone would
leave that site's `tariff_plan` at `None` forever and answer *"no tariff plan
has been resolved yet"* on every cycle - silently disarming a strategy the UI
says is running.
⚠️ **Nothing acquires a periodic HTTPS call from this.** With `price_provider`
off, `resolve_tariff()` answers out of the static profile without opening a
socket; and the true shipped default - no provider, no trading - starts **no
task at all**. What decides whether a request leaves the machine is
`price_provider`, and only that.
⚠️ **AND NOTHING TRADES THAT DID NOT TRADE BEFORE.** `cycle()`'s
`if arbitrage.enabled(self.o):` is untouched. Graded directly rather than
inferred from the gate having moved: `Controller.cycle()` is driven on a
resolved provider plan whose cheapest firm hour is now, with the peak guard
asserted clear, and `arbitrage_enabled` is the single option that moves between
the rows.
| `arbitrage_enabled` | cheapest hour is now | commanded |
|---|---|---|
| `false` | yes | **+180 W** (`loop -> 180 W`) |
| `false` | no | **+180 W** (`loop -> 180 W`) |
| `true` | yes | -5000 W (`arbitrage -> -5000 W`) |
| `true` | no | +5000 W (`arbitrage -> 5000 W`) |
**`TariffPlan.cheap_now()` is deliberately NOT wired, and that is now a
decision rather than an inheritance.** A provider-backed plan exists on every
configured site from this commit, so *Cheap window* - which feeds the
maintenance cycle's grid top-up, i.e. the one place a price reaches something
that **moves energy** without trading being armed - could now read it. It does
not: `self.cheap` still comes from `static_status()`, and on a provider-only
site (`static_profile` empty) that is `False` on every cycle, so no paid grid
top-up is ever forced. Wiring it would take that from *never* to *sometimes*.
`DECISIONS/TARIFF-10-fetching-is-not-trading.md` §4 carries the argument and
the list of what adopting it would take.
**The check that would have caught this** is `test_price.py` section 13, and it
grades **reachability from a configuration**, not the shape of an `if`. The
guards in `amain()` are *evaluated* against real option dicts, whatever they are
spelled as; the call graph under `app/` is walked to see whether the tasks those
guards start can arrive at `resolve_tariff()`, wherever it has moved to; and
then the fetch is **driven** with a counting transport, because a graph is an
argument and a counted request is an observation. The shipped defect - the gate
back at `arbitrage.enabled(opts)`, **-4 B** - is red on seven named checks.
### TARIFF-12 - a returned degradation is not a refresh
**`run_tariff_refresh()` promised in its own docstring that "a failure does not
clear the plan", and enforced it with `except Exception` - a branch a supplier
outage never reaches.** `load_prices()` states the reason in its own words: it
"RETURNS, NEVER RAISES, for every failure a site can actually have", because a
degradation arriving as a traceback into the planning tick would stop the
controller regulating over an expired certificate. So a dead feed **returns** a
§11.4 plan, the `try` succeeds, and the assignment below it overwrote the good
plan unconditionally. With `static_profile` unset - the shipped default, and the
normal state of a dynamic-tariff site - `_degrade()` returns
`TariffPlan(no_data, None, "none")`: **a complete 48-hour plan replaced by
nothing.**
Measured in a 40-hour soak, `static_profile` empty, a 90-minute outage driven:
| Brussels | refresh | plan |
|---|---|---|
| 17:26:12 | 55 | `dynamic_dayahead from generic_dayahead, 141 h, provisional` |
| **17:56:12** | **56** | **`no tariff - generic_dayahead: ClientConnectorError: Cannot connect to host`** |
| 19:26:12 | 59 | restored, one refresh after the feed returned |
`origin: none`, `n_points: 0` for all 30 outage samples. The plan was gone ~27
minutes after the last good fetch, **at the first failure** - so
`TARIFF_MAX_AGE_S` (5400 s) never governed anything, and there was no
serve-stale window at all: a transient blip cost exactly what a long outage did.
**The question is now asked once, after the attempt, from the evidence that
exists** - FCST-06's answer, one module over. The evidence is
`TariffPlan.from_feed` (did this resolution observe the configured feed) and the
shipped `fresh_tariff_plan()` (is an observation still current). A degradation
replaces a standing observation only once that observation has **aged out**, so
`TARIFF_MAX_AGE_S` is now the serve-stale window it reads like.
**A §11.4 static fallback does not replace a fresh provider plan either**,
although it carries real prices. Every point `_degrade()` builds is
`provisional=True`, so `firm_points()` is empty, `arbitrage()` refuses and
`cheap_now()` is False - everything a consumer may act on is already off - while
a day-ahead price for a given hour does not change once published. The fallback
is less actionable and no more current; it takes over when the observation
retires, which is what §11.4 is for. The rule is **not** symmetrical: a standing
plan that is itself a degradation is replaced by the next one, or an outage
would freeze the operator's static profile at the window it was first generated
for.
**`tariff_at` does not move on a kept plan.** It is the moment of the fetch that
produced the plan standing, `fresh_tariff_plan()` judges the age from it, and an
unmoved timestamp is the only thing that tells a kept plan from a freshly
fetched one. Re-stamping it would be the refresh asserting the feed answered
when it did not - the "freeze at its last value and look current forever" hazard
the same docstring names.
**What did NOT change.** The persisted price history still does not back the
plan during an outage: FCST-03 exists to stop stale prices being acted on, and
"recent" is not "current". Nothing about the control loop, the arbiter, or the
30 s tick - `cycle()` still reads two attributes and awaits no fetch.
`test_arbitrage.py` section 8b drives the whole chain with **nothing stubbed but
the socket**: the transport raises the way aiohttp does, `load_prices()` returns
`no_data`, `_degrade()` builds the §11.4 plan, and the shipped
`run_tariff_refresh()` decides. A check that patched `resolve_tariff` to *raise*
would have passed against the defect. Eleven mutations, eleven reds, no
survivors - see `DECISIONS/TARIFF-12-a-degradation-is-not-a-refresh.md`.
**Merged after TARIFF-10, which widened *which installs run this task*, and the
two configurations it newly keeps alive are graded rather than reasoned about.**
A provider with trading disarmed is the case this guard was designed for and is
covered by every driven check here, because nothing in `run_tariff_refresh()`
reads a trading option at all - asserted on the AST symbols rather than on the
source text, after two drafts of that check went red on a correct tree for
saying "arbitrage" in a docstring and in a log line. `price_provider: off` + a
`static_profile` resolves to the CONFIGURED product (`origin: "static"`), so
`standing.from_feed` is false on every refresh and the guard never latches: the
plan is regenerated for the current window exactly as it always was. Both are
driven, and a mutation that makes the guard depend on `arbitrage.enabled()` is
red on five named checks.
### TARIFF-09 - the feed is asked for the range the caller wanted
**`generic_dayahead` took `since` and `until` and requested the bare feed URL,
which answers one Brussels day.** Everything past today was then filled from
yesterday's profile by FCST-04, flagged provisional, and §8.3 drops provisional
hours from the trade set - so **the optimiser never saw tomorrow as firm, at any
hour of any day.** Measured on `4257e38` at 2026-09-10 09:00 CEST over the
shipped 48 h horizon, live feed, yesterday seeded:
| | 48 h horizon, one-day feed |
|---|---|
| firm | **15 whole hours** (60 quarter-hour points) |
| carried, provisional | **33 hours** - 24 on D+1, 9 on D+2 |
| permitted cycle | buy 11:45, sell 19:45 - **both on 2026-09-10** |
| firm local days | `[2026-09-10]` |
The ceiling was structural, not incidental: with a one-day window, at hour *H*
of the local day a 48 h horizon is at most **24−H hours firm and 24+H carried**.
At 23:00 that is **1 firm hour and 47 carried**, and the remaining firm spread
(0.0464 EUR/kWh) no longer clears the default 0.10 round-trip threshold at all -
so arbitrage was refused outright. This is **not** what FCST-04's fill is for:
that covers the window until the ~13:00-14:00 CET auction, after which a healthy
feed publishes tomorrow in full.
**The data exists and the endpoint serves it.** Verified 2026-09-10T07:45:49Z:
`?bzn=BE&start=2026-09-09&end=2026-09-10` returns 192 quarter-hours across two
local days, committed unmodified beside TARIFF-08's captures. The plugin simply
never asked.
**The range is now derived from `since`/`until` on the BRUSSELS calendar**, as
two inclusive dates. Not the UTC calendar: 22:30 UTC in summer is already
tomorrow in Belgium, so a UTC-derived start asks for the day the controller has
just left and drops the day it is standing in, for the last one or two hours of
every day. That is `price_history._bdate()`'s lesson one layer up, with a worse
consequence - `_bdate()` misfiles an hour, this never fetches it. `until` is
exclusive, so a horizon ending exactly at local midnight asks for one day fewer
than the naive reading gives, and an inverted window collapses to a single day
rather than a backwards range.
Same code path, same window, a feed serving both days:
| | before | after |
|---|---|---|
| firm hours of 48 | 15 | **39** |
| carried, provisional | 33 | 9 |
| firm local days | 1 | **2** |
| permitted cycle | one calendar day | **buys on one day, sells on the next** |
The 9 remaining carried hours are the tail of the horizon that runs into a third
Brussels day, which no day-ahead market publishes. FCST-04 still fills them and
still flags them; what changed is what is firm, never what may be acted on.
**Before publication nothing regresses, and that is graded rather than
assumed.** Asking for tomorrow before the auction returns fewer days than
requested - measured, at 09:46 CEST the two-day request came back byte-identical
to the one-day body - and that stays a `series` of what exists, with FCST-04
filling the remainder. A `no_data` there would switch a working site's arbitrage
off every morning.
⚠️ **`price_feed_url` is the EXACT URL and the plugin appends nothing to it, so
an overriding site keeps the short horizon.** That is a decision, not an
oversight: `start`/`end` are this feed's spelling, the provider declares a second
host whose parameters are different, and rewriting a URL an operator typed is
how a request nobody intended gets made. **It is not silent** - the plugin logs a
`WARNING` at fetch time naming the consequence and the `start..end` it would
otherwise have asked for. See
`DECISIONS/TARIFF-09-the-range-and-the-override.md`.
The plugin contract - `fetch(self, opts, since, until, transport)` hands every
plugin the range - is still **not** enforced, because three shipped plugins point
at endpoints nobody has verified and a rule obliging them to pass a range would
be satisfied by an invented parameter name. Which plugins consume the range is
pinned as a literal in `test_price.py` instead, so the next supplier has to
decide rather than inherit the silence.
### FCST-06 - the second way to be a store that cannot persist
**A price cache that could not write went on answering a yesterday the file did
not contain.** `price_history.py`'s docstring states its rule unconditionally -
*"A store that cannot persist now records nothing, and both of its answers say
the same thing"* - and FCST-04 enforced it against `not self.ok`, which is one
**symptom**: the durable volume is missing. `ok` is about the DIRECTORY, and a
read-only remount and a full disk both leave it `True`. Reproduced on `5059e28`
against two real volumes - a `/data` bind-mounted read-only and a full tmpfs:
| | with a `/data` that cannot be written |
|---|---|
| `last_error` | `could not write ([Errno 30] Read-only file system)` |
| `words` | `2026-01-14 (24 points over 24 h); 2026-01-15 (24 points over 24 h) [could not write (...)]` |
| `yesterday()` | **series, 24 points** |
| on disk | `{}` |
`words` and `yesterday()` were describing two different stores, and the 24
points were gone on the next restart with nothing saying so - the
silently-forgetting cache that rule exists to refuse, reached by the condition
it did not name. **No shipped path reaches it**: `/data` is the Supervisor's
durable volume and this module is its only writer. The defect is that a
guarantee the module states about itself did not hold in one condition.
**"Cannot persist" is now ONE derived predicate, asked once, about the
outcome.** It cannot be answered *before* the attempt - a directory that existed
and was writable a moment ago is not a write that completed - so `_save()`
returns whether the file now holds what memory holds: `False` with no durable
volume, `False` on an `OSError` from the write, `True` otherwise. `_remember()`
keeps only what that one answer says was kept, and **the `not self.ok` test in
`_remember()` is gone** rather than joined by a second one. FCST-04's M4 and
FCST-05's HIGH were the same shape - a rule enforced on the conditions somebody
thought of - so a third cause now needs no third branch.
**What is undone is the MERGE, not the store.** The state restored is the last
one that *is on disk*, so a site whose volume fills at 14:00 goes on answering
the yesterday it wrote at 13:30 - that record survives a restart, so answering
it is not a promise the store cannot keep. It simply stops adding to it. The
snapshot is a `copy.deepcopy`: a one-level copy shares the day record the merge
rebinds `points` on, and measured with one, a store holding 12 persisted points
answered **24** after a failed merge.
⚠️ **A failed write is still not fatal, and that was a constraint rather than an
oversight.** §6 and this module's opening: a broken cache must not be a broken
controller. An add-on that will not start because `/data` filled up is a worse
defect than the one being fixed, so `_save()` reports and returns - the failure
stays in the log, in `last_error`, in `words` and in `yesterday()`'s reason -
and the shipped `resolve_tariff()` path still returns a provider plan with no
§11.4 degradation. What changed is that it is now *acted on*.
`test_price_history.py` section 5d drives it on two real failures with different
syscalls and different errnos - a directory where the file goes (`os.replace`,
EISDIR) and a volume that moves out from under an open store (`mkstemp`,
ENOENT) - because the rule is enforced on the outcome and one cause would grade
a coincidence. See `DECISIONS/FCST-06-cannot-persist-is-one-question.md`.
### TARIFF-08 - no network price provider had ever parsed a live response
**`generic_dayahead` could not read one byte of `api.energy-charts.info`, and
had never been able to.** `_points_from_json()` asks for `doc["data"]`, a list
of `{start, price_eur_mwh}` objects. That document does not exist. Fetched live
on 2026-09-10, the feed returns **columnar** parallel arrays and no `data` key
at all:
```
top-level keys: ['deprecated', 'license_info', 'price', 'unit', 'unix_seconds']
unix_seconds: 96 UTC epoch seconds, first 1788991200, spacing 900 s
price: 96 floats, first 216.08
unit: 'EUR / MWh'
```
Driving the shipped parser against that body: `SourceUnavailable: the feed
carried no 'data' list`. Since `generic_dayahead` is the only shipped provider
whose endpoint anybody has reached, **dynamic pricing did not work at all**.
⚠️ **AND §11.4 MADE THAT SILENT, WHICH IS WHY IT LASTED.** A parse error becomes
`no_data`, becomes the static profile, becomes arbitrage off - the *correct*
degradation, and it says nothing whatsoever about the cause. A site would have
sat on the fallback indefinitely while the status page reported only that the
feed was down.
⚠️ **THE SUITE COULD NOT SEE IT BECAUSE EVERY FIXTURE WAS BUILT FROM THE
PARSER'S OWN ASSUMPTION.** `test_price.py`, `test_price_history.py` and
`test_static_tariff.py` each built `json.dumps({"data": [{"start": ...,
"price_eur_mwh": ...}]})` - a shape invented here, published by nobody. The
checks graded the parser against the parser's guess: a closed loop that cannot
notice the guess is wrong, and no amount of coverage inside it would have. The
loop is broken by **bytes nobody in this repository wrote**: two unmodified
response bodies are committed under `scenarios/`, one quarter-hourly
(2026-09-10, 96 points at 900 s) and one hourly (2025-09-10, 24 points at
3600 s - the same endpoint before Belgium's 15-minute MTU), with URL, retrieval
time, licence-as-returned and a sha256 recorded beside them in
`energy-charts-be-price-captures.json` and **re-derived** by the replay, so a
fixture edited to make a check pass stops being a capture. The remaining
invented fixtures are renamed `synthetic_row_feed` at every call site.
**The columnar document is read by its own function, `_points_from_columnar_json()`.**
Not by teaching the row parser to sniff for both: a parser that decides which
document it received is a parser that can decide wrong, and a wrong-but-
parseable response then becomes a price. Two shapes, two functions, and each
plugin names the one its feed publishes. Feeding the row shape to
`generic_dayahead` is now a graded `no_data`, so the sniff cannot come back
quietly.
⚠️ **UNEQUAL ARRAYS ARE REFUSED RATHER THAN ZIPPED.** `zip()` stops at the
shorter list and says nothing, so a truncated `price` array would have produced
a **real price series with its tail missing** - and the tail of a Belgian day is
the evening peak. That is worse than no series: `no_data` switches arbitrage
off, whereas a short series is traded on as though complete, and nothing
downstream can tell them apart. Refused in both directions, with the two lengths
in the reason.
⚠️ **A UNIT CHANGE IS REFUSED FOR A REASON `_validate_raw()` CANNOT COVER.** The
plugin converts with a fixed `0.001`; if the feed ever published EUR/kWh under
the same keys, every price would come out a **thousand times too small** and
`MAX_PLAUSIBLE_EUR_KWH` would not fire, because it only bounds prices that are
too large. DATA-02's fetcher already refused on this field; the shipped path
does now too.
**`unix_seconds` are UTC instants, said so explicitly**
(`datetime.fromtimestamp(t, timezone.utc)`); §9.3's rendering at the Brussels
offset stays where it already lived, in `assemble()`, rather than being copied
into the plugin where a mutation could not tell it from a no-op. A published
`null` is dropped and logged with a count - a missing quarter is not a free
quarter - while an all-null body is `no_data` named by this parser rather than
by the boundary validator.
**End to end against the live feed, 2026-09-10:** `resolve_tariff()` returns a
`TariffPlan` with `origin='provider'`, **96 quarter-hourly points**, spacing
`[900]` s and no decimation, first point `2026-09-10T00:00:00+02:00` at 0.30608
EUR/kWh all-in. FCST-04's H1 fix was never theoretical - 900 s is the shape of
the real feed.
⚠️ **WHAT THIS DOES NOT FIX.** `engie_dynamic`, `totalenergies_dynamic` and
`eneco_eddy` are untouched: their URLs are unverified, no real body can be
captured for them, and withdrawing them is TARIFF-07's open decision. Their
fixtures are now named `synthetic_row_feed` so nothing reads them as evidence
that those feeds parse. And `generic_dayahead` still ignores `since`/`until`
and asks for no date range, so the default URL returns one published day - a
full 48 h horizon is reached only through §8.3's carry-forward fill.
### FCST-05 - what FCST-04's own contract still did not hold
**A single non-UTF-8 byte in `/data/price_history.json` stopped the add-on from
starting.** `_load()` caught `(json.JSONDecodeError, OSError)`.
`UnicodeDecodeError` is `ValueError -> UnicodeError -> UnicodeDecodeError` -
neither of those, and `issubclass(UnicodeDecodeError, OSError)` is **False** -
so it escaped `_load()` -> `_open()` -> `PriceHistory.__init__`, which
`Controller.__init__` does not guard. Reproduced: `UnicodeDecodeError: 'utf-8'
codec can't decode byte 0xff`, and **the controller did not come up**. That is the same class as the day-record defect the store was reworked
for, in the one byte class the rework left uncovered: valid-UTF-8-invalid-JSON,
a UTF-8 BOM and a non-dict day record all degraded correctly. The suite could
not see it because its "unreadable file" fixture writes `"{ this is not json"`,
which is **valid UTF-8**. The handler now names the `ValueError` family - the
file's CONTENT - beside `OSError`, the file's BYTES, and a fixture holding a
genuinely undecodable byte is in `test_price_history.py` section 5.
**A malformed `start` under the day the feed is writing to killed the tariff
plan permanently.** `_remember()` merged a day's points with
`{q["start"]: q ...}` and sorted on `q["start"]`, trusting a value read off
`/data` to be a hashable, sortable string. An `int`, `null`, a `dict` or a
`list` there raised `TypeError`; `run_tariff_refresh` catches `Exception`, so
the add-on stayed **up** while **every refresh raised, forever** - the plan
retiring after `TARIFF_MAX_AGE_S` and arbitrage and the cheap window off for
good, with no self-heal because `_remember()` raised before reaching `_save()`.
The read path (`_parse_points()`) was properly defensive throughout, which is
why `yesterday()` survived and the existing fixture passed: it files its
malformed rows under a day the feed is **not** writing to, so `_remember()`
never opened that record. **The two paths now share one validator**,
`_point_of()`, and neither has a standard of its own. A row that is not a point
is dropped and reported, and `_save()` rewrites the record clean.
⚠️ **AND THE STATUS ROW SAID THE STORE WAS HEALTHY WHILE THE PLAN WAS DEAD.**
`words` read `'2026-01-15 (1 h)'` about the record that was killing every
refresh. `_load()` now counts the rows inside a kept record as well as the
record itself, so the store stops reporting plain health from the moment it is
opened; the note names what was **found** at open and what was **dropped** by
the refresh, and clears itself once the file has been rewritten.
⚠️ **AND THE ROW COUNTED POINTS AND CALLED THEM HOURS.** `f"{d} ({_n_points(d)}
h)"` on the quarter-hourly feed this add-on already supports reads
`'2026-01-13 (96 h)'` for a 24-hour day - the same points/hours conflation the
sub-hourly defect above was about, on the row that defect's own card shipped.
The cell is now `2026-01-13 (96 points over 24 h)`: two facts, each under its
own unit, and the hour figure is the one that says whether tomorrow morning's
fill has a whole day to carry.
⚠️ **AND THE MERGE KEY IS THE INSTANT RATHER THAN ITS SPELLING**, which is what
rule 4 already said. Aware datetimes hash and compare by instant. The string
key merged by spelling, and the lexicographic sort beside it put the second
02:00 of the 25-hour October day (`+01:00`) **before** the first (`+02:00`);
only the stored order was affected, because `_parse_points()` re-sorts by
instant on the way out.
⚠️ **TWO STATED REASONS WERE FALSE, AND THIS REPOSITORY TREATS THAT AS A
DEFECT.** `price_forecast._audit()`'s "WHY IT IS STRICTER" section claimed the
rework refused two things FCST-03's audit did not. A differential over the
whole domain the audit can express - every published mask over 3 hours crossed
with every output series of length 0-3 - found **zero divergence in both
directions**: the reworked audit is exactly **EQUAL** to FCST-03's, not
stronger. The old `seen` count already refused the same published point emitted
twice (one point twice is one hour twice), and the old **identity** rule
already refused a carried point sharing a published hour. The only behavioural
change is the intended **relaxation** - several distinct published points in
one hour - and that stays. The docstring and the two check texts now say what
is true, and the differential is committed as a check in
`test_price_forecast.py` rather than left as prose.
⚠️ **AND `_audit` IS NOT WHAT STOPS THE DECIMATION DEFECT RECURRING.** It does
not check **completeness**: fed the decimation directly - four published points
in an hour, output emitting only the last - it **accepts**, and it accepts an
empty output. That is a limit, not a regression, and the module docstring said
otherwise by presenting the audit as the post-condition guarding provenance.
What actually guards it is the point **count** and the pass-through **identity**
checks in `test_price_forecast.py` and `test_price_history.py`, and the text now
points at them. It is **not** made complete: `published` is the whole read while
the fill emits only `[since, until)`, so "every published point appears" is not
a true post-condition of those two arguments. Measured - the rule as stated
takes the 1 h, 12 h and 48 h horizon checks red on legitimate series.
**No option was added, `config.yaml` is untouched, and nothing was deployed.**
**FCST-03 shipped the fill and no caller.** `price_forecast.fill_from_yesterday()`
was complete and checked, and so were the consumers written for it -
`TariffPlan.firm_points()`, `cheap_now()`'s refusal of a provisional hour, and
`arbitrage()`'s refusal 3, which drops provisional hours from the trade set
rather than refusing the whole series. Nothing in `app/` called the fill, so no
series ever contained a carried point and every one of those consumers was
exercising its all-firm branch.
⚠️ **THE DEFECT THAT PRODUCED WAS NOT AN ERROR ANYWHERE.** `load_prices()`
returns what the feed published and nothing widened it to the requested horizon,
so on a dynamic site between midnight and the 14-16 h CET day-ahead publication
the plan covered the rest of today and **nothing more** - typically 10-24 of the
48 hours `price_horizon_h` asked for. `arbitrage()` then took `max - min` over
that truncated window and reported a spread. The number was real, the window was
half the one configured, and **nothing said so**: tomorrow evening's peak, the
hour a morning charge would be sold into, was not in the series at all, so the
spread it would have created was never compared against anything.
**What changes:**
* `app/price_history.py` (new) keeps the **published** prices of the last few
Brussels days in `/data/price_history.json`, and `resolve_tariff()` now fills
an unpublished hour of the horizon from yesterday's same-wall-clock hour,
flagged provisional. ⚠️ **"An" unpublished hour, not "every" one:** the fill
can carry only an hour yesterday actually published, so a partial yesterday
leaves the local hours it never held **out of the series** - §11.4 for those
hours, and asking for one raises rather than answering. Correct behaviour,
and the wording is now the behaviour.
* ⚠️ **A sub-hourly feed keeps every print.** `eneco_eddy` is the imbalance
product and settles per **quarter**; the fill walks whole hours to decide
what needs carrying, and that is not a rule about what a published hour may
contain. Every published point inside the horizon passes through by identity,
and a carried point is added only for a whole hour with no published coverage
at all. The first draft keyed the published set by hour - last point wins -
and a 96-point quarter-hourly feed reached the optimiser as 15 points with
its spread down from 0.143 to 0.014 EUR/kWh, on an all-firm, fully-published
series.
* The horizon is **whole hours at both ends**: `since` is floored to the hour
that contains it, and so is `until`. Asked for 48 hours at 08:17, the plan is
48 hours long and not 49.
* `resolve_tariff(history=...)` is **required and keyword-only**. A caller with
no durable volume passes the explicit `NoHistory()`. A default of `None`
meaning "skip the fill" would hand a caller who forgot the argument exactly
the truncated horizon this card removes, silently.
* A new status row, **Price history**, names the days held, how many points
each holds and how many whole hours they cover (see FCST-05: it shipped
counting points and printing them under the hour's name).
⚠️ **ONE CHANGE HERE MOVES ENERGY, AND IT IS NOT THE ONE THE CARD IS NAMED FOR.**
Carried hours are provisional, so §8.3 keeps them out of the trade set: the fill
widens what the optimiser can **see**, never what it may **do**. What does
change a trade is that the filled series is windowed to `[since, until)`, so
**hours that have already elapsed are no longer in the spread**. Before, a cheap
night that had already gone sat in the series as the day's cheapest hour - and a
cycle cannot buy in an hour that is past, so the trade was quietly never taken.
`test_price_history.py` section 2 grades this by finding the cheapest firm hour
with and without the fill and showing it **moves**.
⚠️ **A DEAD FEED IS STILL §11.4, NOT YESTERDAY.** The fill runs only on a
`series` read; a feed that published nothing still degrades to the configured
`static_profile`. The two are indistinguishable to the optimiser - both are
100 % provisional, so arbitrage refuses on the firm-pair rung and `cheap_now()`
is False - so preferring yesterday would change a documented degradation for a
difference only the status page can see.
`DECISIONS/FCST-04-the-day-that-is-carried.md` carries that and two more.
⚠️ **A BROKEN CACHE IS NOT A BROKEN CONTROLLER.** No `/data`, an unreadable
file, a schema this build does not know (including a `version` of `1.0`, which
`== 1`), a **day record that is not a record**, a malformed point - each its own
state with its own reason, none of them a price and none of them an empty
series. In every one the plan is still a real provider series and the controller
still regulates; the horizon is just as short as it was before this card.
`.words` never raises, because `Controller.__init__` reads it unguarded and a
status line that throws is an add-on that does not start.
⚠️ **AND WHERE THERE IS NO `/data`, NOTHING IS RECORDED AT ALL** - not even in
memory. An in-memory yesterday answers for one run and is gone on the next
restart, which is a cache that forgets without saying so; the honest pair is
`unavailable` on the status row and no prices carried.
⚠️ **"YESTERDAY" IS A LOCAL DATE, AND THE ARITHMETIC IS ON THE DATE.**
`brussels()` renders at a fixed offset, so subtracting `timedelta(days=1)` from
it subtracts 24 **absolute** hours - and the Brussels day before a
spring-forward is 23 hours long. Between 00:00 and 00:59 on the morning after
the March transition that lands on the day *before* yesterday. **No option was
added and `config.yaml` is untouched.**
## 0.9.0
⚠️ **MINOR FOR NEW FUNCTIONALITY, NOT FOR A BREAK - AND THAT DISTINCTION IS THE
@@ -0,0 +1,181 @@
# FCST-04 — what "yesterday" is, and what the fill is allowed to reach
Four decisions, recorded because each had a defensible alternative that a later
reader would otherwise re-open from scratch.
---
## 1. The fill runs on a `series` read only. A dead feed still goes to §11.4.
`fill_from_yesterday()` accepts a `no_data` read and will happily build a
100 %-carried series from yesterday alone. `resolve_tariff()` does not let it:
a feed that published nothing still falls to `_degrade()` and §11.4's own row —
*"No price data → fall back to the static profile, mark the plan provisional,
no arbitrage."*
**Why not prefer yesterday's real prices to a coarse day/night profile?**
Because it buys nothing the optimiser can use, and it silently replaces a
fallback the operator configured.
Measured, not assumed — and stated at the reach it actually has:
- **This one reaches control.** A fully-carried series is 100 % provisional, so
`TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on
§8.3's firm-pair rung. That is the verdict the optimiser reads: `arbitrage.py`
calls `plan.arbitrage(...)` on `fresh_tariff_plan()`, and it is the only path
by which a provider plan moves energy. The §11.4 static fallback refuses on
the same rung, for the same reason.
- **This one does not, and saying so is the point.** `TariffPlan.cheap_now()`
also returns False on a fully-carried series, because `firm_points()` is
empty — but it has **no production caller on the provider plan**.
`self.cheap` comes from `static_status()`, which builds its *own* static plan
and calls `cheap_now()` on that. So it is a property of the type, checked in
`test_static_tariff.py`, and it is not evidence about the shipped control
path. The conclusion below holds on the first bullet alone.
The two paths therefore differ only in which numbers the status page shows.
So the choice is between two identical control behaviours, and the tie-breaker
is that `static_profile` is a documented option an operator fills in **for this
case**. Overriding it for a cosmetic difference is a behaviour change nobody
asked for.
⚠️ **The consequence, stated so it is not discovered later as a bug:**
`fill_from_yesterday()`'s `no_data` branch is not reached from production.
It stays checked in `test_price_forecast.py`. Turning it on is a product
decision — it would want a rule about how old a carried profile may be before it
stops being offered at all — and not a wiring one.
---
## 2. "Yesterday" is a date, not the newest record.
`newest()` is the obvious accessor and it is the one that decays. An add-on down
from Friday to Monday would answer Friday's prices to a fill whose only
vocabulary is `provisional: true` — a flag that says "this is a guess", never
"this guess is three days old". Nothing downstream can tell the difference and
nothing reports it.
So `yesterday(stamp)` computes the Brussels date before `stamp`'s Brussels date
and looks up **that** record. Anything else is `no_data` with a reason naming
both the date wanted and the dates held.
The cost is real and accepted: after an outage of more than a day, the first
morning back has no fill. That is the honest state — there is no yesterday —
and the horizon is short for one morning rather than wrong for several.
---
## 3. What is recorded is the provider's read, and the mechanism is ordering.
Feeding the filled series back into the store is how this rots: day 2 carries
day 1's guesses, day 3 carries day 2's carry of them, and after a week of a dead
feed the "yesterday" being carried is a fossil of the last real publication.
Every point still carries `provisional=True`, so the **flag never lies** — what
decays is the number, behind a flag that is telling the truth.
⚠️ **No value-level check can catch that**, and claiming one would be worse than
claiming none. A carried point and a published-provisional point (`eneco_eddy`'s
current quarter, §9.2) are the same three fields carrying the same flag. The
difference is provenance, and provenance is not in the value.
The defence is therefore an **ordering**, not a validation:
- `carry()` is the only public entry point. It records its argument, then
returns the fill.
- The filled series exists only as that return value. The object never sees it
again.
- `_remember()` is private and called from exactly one place.
- `test_price_history.py` parses every shipped module under `app/`, `tools/`,
`scenarios/` and `estop/` and fails if any of them **uses the name**
`_remember` — an attribute, a bare name, a `def`, or the string constant a
`getattr` would need. By name rather than by substring, so a comment that
merely mentions the rule does not fail it and a call in `tools/` cannot pass
unseen.
- and it pins the order two ways: the **line numbers** of the two calls in
`carry()`, and — the one that matters — the **behaviour**. A feed carrying
yesterday alongside today, into an empty store, must fill the whole horizon;
reversed, the lookup finds nothing and the plan is only the hours the feed
published. Driven, so a reversal is red.
⚠️ **The first version of that ordering check graded nothing, and this file
said it did.** It compared positions in `ast.walk`, which is breadth-first:
`self._remember(...)` is a statement and `self.yesterday(...)` is an *argument*
of `fill_from_yesterday(...)`, one level deeper, so `_remember` came out first
whichever statement was written first. Measured — shipped and reversed both
scored the same. A decorative check that a document cites as a guarantee is
worse than no check, which is why the correction is recorded here rather than
quietly made.
That ordering has a second benefit worth keeping: a feed that returns yesterday
alongside today refreshes yesterday's record from live data *before* it is
consumed as the profile.
---
## What this ticket does not claim
It does not make a trade possible that yesterday's prices justify. Carried hours
are dropped from the trade set by §8.3, and that is not a limitation to be fixed
later — it is the rule.
The one change that **does** move energy came from the windowing, not the
carrying: the filled series covers `[since, until)`, so hours that have already
elapsed are no longer in the spread. A cheap night that has gone used to be the
series' cheapest hour, and a cycle cannot buy in an hour that is past — so the
trade was silently never taken. `test_price_history.py` section 2 grades that by
finding the cheapest firm hour with and without the fill and showing it moves.
⚠️ **And section 1's identity check now grades identity.** It used to compare
`eur_per_kwh` between `_before` and `_after`, which come from two *separate*
`resolve_tariff()` calls — each parsing the feed body into its own objects, so
`is` could never have held between them however the fill were written. FCST-03's
whole anti-copy argument is `filled.points[i] is fetched.points[j]`, and floats
cannot grade it. The value comparison is kept and now says what it is (the fill
moved the window, it did not touch a price); identity is graded on a read the
file constructs and still holds a reference to.
---
## 4. A sub-hourly feed passes through whole; the fill only decides which hours are missing.
`eneco_eddy` is the Belgian imbalance product and settles **per quarter**, and
`price.py`'s `_validate_raw()` accepts any strictly-increasing spacing — so a
published hour may legitimately hold four points.
FCST-04's first caller keyed the published set as `{_utc_hour(p.start): p for p
in r.points}`. It reads perfectly ordinary and it keeps the **last** point of
each hour. Measured on a 96-point quarter-hourly feed — all firm, fully
published, nothing to carry: the plan reached the optimiser as **15 points**,
the arbitrage spread fell from **0.143 to 0.014 EUR/kWh**, and the cheapest
"hour" became a `:45` point whose `at()` window runs into the next hour. That is
not the windowing effect above; it is a feed losing 84 % of its own prints, and
no `provisional` flag would ever have reported it.
**The alternative was rejected:** refusing sub-hourly feeds at `_validate_raw()`
would have made the count correct by breaking the one supported product that
legitimately publishes per quarter.
**What was done instead** keeps FCST-03's existing contract rather than weakening
it: *every* published point inside the window passes through **by identity**,
and a carried point is added only for a whole hour with **no** published
coverage at all. A carried hour is still a single hourly point whatever the
feed's own spacing is, because yesterday's profile is only known per wall-clock
hour — and carried hours are provisional, so they are out of the trade set
anyway.
⚠️ **`_audit()` got stricter, not looser, and that was the condition on doing
this at all.** The old post-condition refused any repeated `_utc_hour` — a count
that cannot express four points in one hour. It is replaced by two rules that
**subsume** it:
- a **published** point must *be* one of that hour's published points, by
identity, and may be emitted only once;
- a **carried** point may stand only in an hour that does not appear in the
published set at all, must be flagged `True`, and may not repeat.
So a duplicated carried hour — the DST mispricing the count existed to catch —
still raises; so does the same published point emitted twice; and a carried
point sharing an hour with a real print now raises too, which the count never
looked at. Each of those three is driven by its own mutation in
`test_price_forecast.py`.
@@ -0,0 +1,125 @@
# FCST-06 — "cannot persist" is one question, and it is asked about the outcome
Two decisions. The first is the one the ticket put up for judgement; the second
is the one that decides what a store answers *after* it has failed to write.
---
## 1. One derived predicate, not two places testing two symptoms — yes
`price_history.py`'s module docstring states, unconditionally:
> ⚠️ AND THAT RULE IS CARRIED OUT IN `_remember()`, NOT ONLY IN `_save()`. …
> A store that cannot persist now records nothing, and both of its answers say
> the same thing.
It was enforced against `not self.ok`, and `ok` is **about the directory**: it
says `/data` was there when the store opened. A read-only remount and a full
volume both leave it `True`. So the second way to be a store that cannot
persist — `ok=True`, the directory present, `_save()` failing — was outside the
one condition the rule named, and in that state `words` printed two full days
beside the write error while `yesterday()` answered 24 real points, from memory,
until a restart dropped them without a word.
**The ticket's question was whether "cannot persist" should be one derived
predicate rather than two places each testing a different symptom. It should,
and the reason is not tidiness — it is that the enumeration cannot be
completed.** FCST-04's M4 enforced the rule on the symptom that was in front of
it. FCST-05's HIGH was the same shape — an `except` clause naming the one
content-failure member somebody had met (`json.JSONDecodeError`) instead of the
`ValueError` family — and its fix was to name the family. This is the third instance in
one module, and it is the last place in this file where the rule can be spelt
twice. A fourth cause — an `OSError` subclass, a filesystem that accepts the
`mkstemp` and refuses the `os.replace`, a container whose volume is unmounted
mid-refresh — needs no fourth branch now, because none of them is asked about.
### Where the one question can be asked, which is the whole design
**"Can this store persist?" cannot be answered before the attempt.** A
directory that exists and was writable a moment ago is not a write that
completed; there is no pre-check that is not itself a guess, and adding one
(`os.access`, a probe write) would be a *third* symptom test — precisely what
the ticket rules out. So the predicate is an **outcome**:
_save() -> bool # True only if the file now holds what memory holds
Both ways of not persisting are the same answer from the same call: no durable
volume returns `False` without touching the disk, an `OSError` out of the write
returns `False` after recording why. `_remember()` reads that one answer and
keeps only what it says was kept. **The `not self.ok` test in `_remember()` is
gone** — not moved, gone — because there is nothing left for a second test to
add, and a check in `test_price_history.py` 5d asserts its absence from the
source.
⚠️ **That last check is deliberately a source check, and habit 4 is why it is
allowed to be.** The claim "there are not two places testing two symptoms" is a
claim about the *shape of the code*, not about a behaviour — the behaviour is
identical whether the rule is enforced once or twice. Every other check in 5d
is driven.
### What was NOT done, and why
- **Not a `can_persist()` helper.** It would have to answer before the attempt,
which is the thing that cannot be done honestly, and it would be a second
place to keep in step with the first.
- **Not a probe write at `_open()`.** It would make `ok` mean something new, it
would write to `/data` on every start, and a volume that is writable at 08:00
and full at 14:00 would still defeat it. A predicate that can go stale is a
symptom test wearing a predicate's name.
- **Not making a failed write fatal.** §6, and this module's own opening: a
broken cache must not be a broken controller. An add-on that will not start
because `/data` filled up is a worse defect than the one being fixed. The
store degrades; it does not raise. `_save()` reports and returns.
---
## 2. The MERGE is undone, not the store
When `_save()` answers no, `_remember()` restores the state the read started
from — **not** an empty one.
The alternative considered was to clear `self._days` outright, on the argument
that a store which cannot write is a store that should stop answering. It is
wrong, and measurably so: the state being restored is the last one that **is on
disk**, so answering it is not a promise the store cannot keep. A site whose
volume fills at 14:00 goes on answering the yesterday it wrote at 13:30 —
through the restart, because that record is in the file. What it stops doing is
*adding* to it.
Clearing would throw away a good yesterday that survives a restart in order to
punish a write that failed, and it would make the morning fill worse on exactly
the day the operator is already dealing with a full disk.
That gives the file the invariant its rule was always describing, stated once
in the class docstring:
> `self._days` **never holds a point the file does not.**
⚠️ Not *"mirrors the file"* — `_usable_days()` drops records the file still
holds, and that asymmetry is deliberate. The only direction that can **invent a
yesterday** is the one named.
⚠️ **And the snapshot is `copy.deepcopy`, not `dict(self._days)`.** A one-level
copy shares the day *records*, and the merge rebinds `points` on the ones it
touches — so the "restored" store would answer the merge that never reached the
file. Measured: with a shallow copy, a store holding 12 persisted points
answered **24** after a failed merge. It would also be correct only for as long
as nobody makes the merge mutate a `points` list or a row in place, which is the
same "correct for the conditions someone thought of" this ticket is about. The
store holds at most `KEEP_DAYS + 2` records and writes 48 times a day; a copy
that is right by construction is worth more than the allocations.
---
## What this does not change
- **No shipped path reaches the defect.** `/data` is the Supervisor's durable
volume and this module is its only writer, so on a real install `_save()`
succeeds and nothing above is on the hot path. The fix is that a guarantee the
module states about itself now holds in the condition it did not.
- **The failure was already visible and still is** — `_LOG.error("could not
persist price history: %s")` and `last_error`, reported by both `words` and
`yesterday()`'s reason. What changed is that it is now *acted on*.
- **Nothing about the prices.** The series served is the feed's own, every
carried point is still flagged provisional, and a broken cache is still a
short horizon rather than a §11.4 degradation.
@@ -0,0 +1,141 @@
# TARIFF-09 — asking the feed for the horizon, and what `price_feed_url` means
Two decisions. The first had no defensible alternative once it was measured;
the second did, and the alternative is argued below rather than dismissed.
---
## 0. The measurement this card exists for
`generic_dayahead.fetch()` accepted `since` and `until` and requested the bare
`https://api.energy-charts.info/price?bzn=BE`. That URL names no range, and the
feed answers **one Brussels day**.
Measured on the merged tree (`4257e38`) at 2026-09-10 09:00 CEST, 48 h horizon,
live feed, yesterday seeded through the shipped read path:
```
60 firm points (15 whole hours) + 33 carried hours of 48 requested
carried: 24 h on 2026-09-11, 9 h on 2026-09-12
arbitrage(0.10): allowed, spread 0.4645, buy 11:45 and sell 19:45 -- BOTH on 2026-09-10
firm local days = [2026-09-10]
```
Everything past today was carry-forward, flagged provisional, and §8.3 drops
provisional hours from the trade set. **The optimiser never saw tomorrow as
firm, at any hour.** The ceiling was structural rather than incidental: with a
one-day window, at hour *H* of the local day a 48 h horizon is at most
**24−H hours firm and 24+H carried** — 15/33 at 09:00, and **1/47 at 23:00**,
where the remaining firm spread (0.0464 EUR/kWh) no longer even clears the
default 0.10 round-trip threshold and arbitrage is refused outright.
That is not what FCST-04's fill was built for. FCST-04 covers the
pre-publication window until the ~13:00–14:00 CET auction, after which a
healthy feed hands over tomorrow in full.
**The data exists and the endpoint serves it.** Verified 2026-09-10T07:45:49Z:
`?bzn=BE&start=2026-09-09&end=2026-09-10` returns **192 quarter-hours across
two local days**, committed unmodified as
`scenarios/energy-charts-be-price-2026-09-09_2026-09-10-twoday.json`. The
plugin simply never asked.
After the change, same code path, same window, a feed serving both days:
**39 firm hours and 9 carried**, firm across **two** Brussels days, and the
permitted cycle buys on one day and sells on the other. Graded in
`test_price_history.py` section 1c against both feed states.
---
## 1. The range is derived on the **Brussels** calendar, from `since`/`until`
`since` and `until` are **instants**; the API takes **dates**. Those are not
the same question, and the conversion is where the day gets lost.
`brussels_date_range()` returns two inclusive `date`s:
* the first is `brussels(since).date()`. **Not `since.date()`, and not the UTC
date.** 22:30 UTC in summer is already tomorrow in Belgium, so a UTC-derived
start asks for the day the controller has just left and drops the day it is
standing in — for the last one or two hours of *every* day. This is
`price_history._bdate()`'s lesson one layer up, with a worse consequence:
`_bdate()` misfiles an hour, this never fetches it.
* the last is `brussels(until).date()`, **stepped back a day when `until`
falls exactly on local midnight**, because `until` is exclusive and that
instant is the first tick of a day nobody asked for. Without it every
whole-hour horizon requests one extra day — harmless on this feed, but it
makes the request a claim the caller never made.
* an empty or inverted window collapses to a single day rather than producing
a backwards range, which no feed has a defined answer for.
Both ends are graded, and both are driven red by mutation — including the two
that swap `brussels()` for the UTC date (`test_price.py` section 2c).
---
## 2. `price_feed_url` is the **exact URL**, used as given. The plugin appends
nothing.
The alternative — merge our `start`/`end` into whatever query string the
operator supplied — was considered and rejected. Three reasons, in the order
they decided it:
1. **`start`/`end` are energy-charts' spelling, and this provider declares a
second host.** `generic_dayahead.capabilities().egress_hosts` is
`("api.energy-charts.info", "web-api.tp.entsoe.eu")`. That second host's
day-ahead date parameters have not been verified by this card and are not
guessed at here — but they are not `start`/`end`. So an override to the
*other declared host* — a case the declaration itself anticipates — is
exactly where merging appends two parameters that host ignores, producing
the short horizon anyway, with a URL nobody typed as the receipt.
2. **Merging means re-serialising a URL somebody else wrote.** Parsing and
rebuilding reorders parameters, collapses duplicates and re-escapes values.
"Override the feed" is what the option already says it means, and an
override that is silently rewritten is not one.
3. **Blind appending is not on the table at all.** `dated_feed_url()` refuses
to add a query string to a base that already has one, so
`?bzn=BE?bzn=BE&start=…` cannot be produced by this module even by a later
edit that folds the parameters back into the constant.
⚠️ **The cost is real and it is therefore NOT a doc footnote.** An overriding
operator gets exactly the short horizon this card exists to fix. So the plugin
emits a `WARNING` on the shipped `goodwe.price` logger at **fetch time**, naming
the consequence — one Brussels day, the rest carried and flagged, §8.3 dropping
them from the trade set — and naming the `start..end` it *would* have asked
for. §11.4 is what makes a price defect quiet; an operator who has taken the
short horizon has to be able to find out from the log that they did.
The warning is graded, with a control that the default path stays silent, and
a mutation that downgrades it to `debug()` takes that check red.
**Fetch time rather than startup**, because that is where the URL actually in
use is known. The trade-off is one line per refresh (the tariff task, not the
30 s control tick). If the status page ever grows a price-source row, this is
the sentence it should carry.
---
## 3. The plugin contract: `fetch(self, opts, since, until, transport)`
The signature promises every plugin the range, and until this card **none of
the four networked plugins used it**. The obvious follow-up is to enforce the
promise. It is **not** enforced, and the reason is TARIFF-08's:
Three of the five shipped plugins point at endpoints nobody has ever verified
(TARIFF-07 is the open decision on withdrawing them). A rule obliging them to
pass a date range would be satisfied by an **invented parameter name** — which
is the parser-graded-against-its-own-guess defect moved from the response to
the request, and it would ship looking like a guarantee.
What exists instead is a **roster pin** in `test_price.py` section 2c: which
plugins reference their `since`/`until` is asserted as a literal, derived from
the registry and the AST. A sixth supplier, or a change to any of the three,
has to edit that line and make the decision rather than inherit the silence.
It is named as a source-shape pin, not as evidence that any feed parses;
`generic_dayahead`'s behaviour is graded separately by the URL it really
requests.
**Recommendation for whoever picks up TARIFF-07:** enforce it at the moment
those three endpoints are either verified or withdrawn, not before. If they are
withdrawn, `generic_dayahead` and `manual_csv` are the whole roster and the
rule is one line — `manual_csv` reads a whole file and is windowed downstream,
so it would be the single documented exemption.
@@ -0,0 +1,277 @@
# TARIFF-10 — fetching a price and trading on one are two decisions
**Status: DECIDED 2026-09-12 — Option A. If a price provider is configured,
prices are fetched. Trading stays a separate switch (`arbitrage_enabled`), and
this card moves nothing that moves energy.**
Filed with TARIFF-10. Follows [[TARIFF-09]] and [[FCST-04]]; §4 settles the
limitation FCST-04 recorded and did not decide.
---
## 1. The defect, and why it was invisible
`app/main.py`'s `amain()` read:
```
if arbitrage.enabled(opts):
tasks.append(asyncio.create_task(controller.run_tariff_refresh()))
```
`run_tariff_refresh` is the **only** caller of `resolve_tariff()`, which is the
**only** caller of `price_history.carry()`. `arbitrage_enabled` ships `false`.
So on a shipped install with a provider configured:
* no HTTPS request was ever made to the feed;
* no plan was ever resolved, so `self.tariff_plan` stayed `None` forever;
* `/data/price_history.json` was never created;
* the *Price history* status row read `empty` for the life of the install;
* TARIFF-09's short-horizon warning could not fire, because it is logged on a
fetch and there was no fetch.
Measured, not inferred — a 40 h soak, two instances, one feed:
| | shipped default + provider | same, arbitrage armed |
|---|---|---|
| feed requests | **0** | fetching |
| `/data/price_history.json` | `present: false`, **802/802 samples** | written |
| resolved plan | none | 132 points |
⚠️ **AND FOUR SURFACES ALREADY DESCRIBED THE FIXED BEHAVIOUR.** `DOCS.md`
§*Energy price providers*: *"With `arbitrage_enabled: false`, which is the
shipped default, turning a provider on still fetches, validates and publishes
prices and changes no charging decision."* `config.yaml` and both translations
said the same, and additionally carried a claim that had gone stale in the other
direction — *"nothing consumes the price series yet"*, untrue since OPT-04
shipped the consumer. The code was the only surface that disagreed with all of
them, and no check compared the two.
The dating is worth being exact about, because it says this was never a
regression. `git log -S` puts the refresh task, its `arbitrage.enabled()` guard
**and** the DOCS sentence in the *same commit* — OPT-04, `bdd7f18`;
`config.yaml`'s version of the claim goes back further, to TARIFF-02,
`852fc7c`. The prose and the code disagreed from the moment both existed.
## 2. Why a check on the `if` would not have caught it
The `if` was exactly what somebody meant to write, and its comment argued for
it honestly:
> ⚠️ STARTED ONLY WHEN ARBITRAGE IS ARMED. Nothing else in the add-on reads the
> resolved plan … so on a default install this task would be network traffic in
> aid of an attribute nobody reads.
That was true when OPT-04 wrote it and stopped being true when FCST-04 gave the
plan a *store* and a *status row*. A pin on the expression would have gone red
on the fix and green on the defect. What was wrong is the **set of
configurations from which a fetch is reachable**, so that is what
`test_price.py` section 13 grades:
1. the guards on every `asyncio.create_task` in `amain()` are **evaluated**
against real option dicts — whatever the guard is spelled as;
2. the call graph under `app/` is walked to see whether the coroutines those
guards start can arrive at `resolve_tariff()` — wherever it has moved to;
3. and then the fetch is **driven**, with a counting transport, a real
`Controller` and a real store path, because a reachability graph is an
argument and a counted request is an observation.
⚠️ **Step 3 grades the gate only because it starts the task the way `amain()`
would.** The driver asks `_fetchers(opts)` — the evaluated guard — before it
calls the coroutine. Measured: with the shipped defect restored **and** that one
line defeated (`if False and "run_tariff_refresh" not in names:`), the four
driven rows go **green** while only the structural rows stay red. Without it
they are decoration, which is MUTATION.md's habit 4 in a harness.
What the instrument **cannot** see is printed on every run: a coroutine started
by anything but `create_task`; a guard that is not an enclosing `if` in
`amain()`; anything outside `app/`; and — because the graph is keyed by **name**
— two same-named functions as one node, which makes every *reachable* answer an
**upper** bound and every *not reachable* answer the strong one.
## 3. The gate, and why it is an `or`
```
def tariff_refresh_wanted(opts: dict) -> bool:
return provider_configured(opts) or arbitrage.enabled(opts)
```
The first term is what this card adds. **The second term is the old gate, kept
verbatim, and it is not there for symmetry.**
`price_provider: off` + a `static_profile` + `arbitrage_enabled: true` is a
configuration `validate.py` explicitly **accepts** (rule 7 refuses only the
no-prices-at-all case), and it **trades**. Measured on this branch:
```
opts: price_provider off, arbitrage_enabled true,
cycle_cost_threshold_eur 0.05,
static_profile 0.20 night / 0.284 day
plan : static_profile from static_profile, 48 h, 48 points
WITH the plan -> ARBITRAGE_CHARGE -5000.0 W
"cheapest firm hour at 0.2000 EUR/kWh, 0.0840 below the
dearest and clear of the 0.0750 this cycle had to cover"
WITHOUT the plan -> None "no tariff plan has been resolved yet"
```
Gating the refresh on the provider **alone** — which is the literal reading of
the ticket's second acceptance criterion — would leave that site's
`tariff_plan` at `None` forever and answer *"no tariff plan has been resolved
yet"* on every cycle. That is a strategy the UI says is running and that cannot
fire: the exact state `validate.py`'s OPT-04 rules exist to refuse, arrived at
from inside the code instead of from the configuration.
So the gate is a **strict widening**: every configuration that started the task
before still starts it, and `test_price.py` grades that as its own row rather
than leaving it to prose.
⚠️ **The AC's stated harm is still fully met, and it is worth being exact about
this rather than quietly declaring victory.** The criterion reads *"with
`price_provider: off`: no fetch task at all, whatever `arbitrage_enabled` says
— an operator with no provider must not acquire a periodic HTTPS call."* The
letter of it is not met in one configuration (provider off, trading armed),
because that configuration keeps the task it already had. The harm it names is:
* with `price_provider` off, `resolve_tariff()` returns `static_plan()` — pure
arithmetic, **no socket**. Driven and counted: **0 requests**.
* the true shipped default — no provider, no trading — starts **no task at
all**.
Nobody *acquires* a periodic HTTPS call from this card, and the only thing that
decides whether a request leaves the machine is `price_provider`. **If the
owner wants the literal reading instead, it is a one-token change** (drop the
`or` term) **and it costs the site in the measurement above its trading.** That
is the trade, stated rather than chosen silently.
## 4. `TariffPlan.cheap_now()` — available, and deliberately NOT consumed
**Decision: do not wire it in this card. `self.cheap` stays on
`static_status()`.**
FCST-04 recorded that `cheap_now()` has no production caller on the provider
plan: `main.read_inputs()` assigns `self.cheap, self.tariff_words =
static_status(o, now)`, which builds its **own** static plan and calls
`cheap_now()` on that. Under Option A a provider-backed plan now exists on every
configured site, so the limitation stops being structural and becomes a choice.
**It is not cosmetic, and that is the whole reason for the decision.**
`self.cheap` is read in `cycle()` by the maintenance cycle's cheap-window grid
top-up — the add-on's one shiftable load, and the one place a *price* reaches
something that **moves energy** on an install that has not armed trading.
⚠️ **And the direction it would move is not "a slightly different window".**
`static_status()` returns `(False, "no static profile")` whenever
`static_profile` is empty, so today a provider-only site — a plausible shape
for a dynamic-tariff install, and the shape this card makes work — has
`self.cheap = False` on **every** cycle and never forces a paid grid top-up at
all. Wiring `cheap_now()` would take that from *never* to *sometimes*. That is
a strictly larger behaviour change than this card's headline, and it belongs to
whoever owns the maintenance cycle rather than to the ticket that made it
reachable.
Three further reasons, in the order they would bite:
1. **Two sources, one attribute, and the loser is silent.** `validate.py`
already refuses `static_profile` alongside the Home Assistant price
entities *because* `read_inputs()` picks one and the other disappears. A
provider plan would be a third writer of the same attribute with no rule
saying which wins.
2. **The freshness question is not answered for this consumer.** The plan
retires after `TARIFF_MAX_AGE_S` and `fresh_tariff_plan()` returns `None`.
Arbitrage has an answer for that (refuse). The maintenance top-up has none:
"the plan expired" would have to mean *not cheap*, and *not cheap* on this
path means **wait for sun**, which is a real decision nobody has taken.
3. **The comparison basis changes, and so does the answer on a bad feed.**
`cheap_now()` is strictly-below-the-mean-of-the-**firm**-hours, and it is
`False` when THIS hour is provisional or is not in the series at all
(FCST-03's rule: this hour decides, not the whole plan). So a feed that
published nothing, or an hour the §8.3 fill could not cover, reads *not
cheap* where a `static_profile` reads whatever the clock says. Defensible
either way — and it is a change to the shipped maintenance cycle that needs
its own measurement rather than a side effect of this one.
**What adopting it would take**, so the next card does not start from scratch:
* a rule for which source wins when both a provider plan and a
`static_profile` are configured, enforced in `validate.py` the way the
entity/profile conflict already is;
* a decision on what an **expired** or **absent** plan means for the grid
top-up, stated as a default rather than left to `None`;
* `read_inputs()` splitting `self.cheap` from `self.tariff_words`, which are
currently one assignment from one function;
* a check that the maintenance top-up actually fires and does not fire on the
provider plan — driven through `cycle()`, not asserted on the attribute;
* and the *Tariff* status row, which FCST-04 pinned as a **pair** with
`DOCS.md` in `test_price_history.py` §9: it does not report the provider
plan, DOCS does not claim it does, and moving one without the other is
already red.
## 5. The *Arbitrage* status row, re-read and left alone
`main.status_payload()` renders `("Arbitrage", "off (arbitrage_enabled)" if
self.arb is None else self.arb.explain())`. `self.arb` is assigned only inside
`if arbitrage.enabled(self.o):` and `arbitrage.decide()` never returns `None`,
so `self.arb is None` **iff** the option is off. The string still names the
option that is actually off, and under Option A it gains a job it did not have:
on a site with a provider configured it is the row that tells an operator the
prices arriving on the *Price history* row above are moving nothing. Unchanged,
with a note at the call site saying it was checked.
## 6. What this card did not do
* It did not touch `run_tariff_refresh()`'s body (TARIFF-12 is there).
* It did not change `cycle()`'s `if arbitrage.enabled(self.o):`, the sizing,
the thresholds, the SoC bounds or the peak interlock.
* It did not wire the provider plan into the *Tariff* status row or into
`self.cheap` — §4.
* It did not pick a version number (PKG-06).
## 7. Mutation ledger
Every mutation applied to a **fresh `cp -a`** of the clean worktree rather than
reverted (habit 18: there is no revert to verify and no shared scratch path),
`__pycache__` purged, tree mounted `:ro`, run under `python3 -B`.
| id | mutation | Δ bytes | result |
|---|---|---|---|
| M1 | the shipped defect: gate back to `arbitrage.enabled(opts)` | **-4 B** | **RED**, 7 named checks |
| M2 | drop the `or arbitrage.enabled(opts)` term | -27 B | **RED**, 2 named checks (strict-widening + the static-profile row) |
| M3 | `tariff_refresh_wanted` -> always `True` | +8 B | **RED**, the AC2 reachability row |
| M4 | `provider_configured`: `"off"` counts as configured | -6 B | **RED**, the AC2 reachability row |
| M5 | `cycle()`'s trading gate -> `if True or arbitrage.enabled(self.o)` | +8 B | **RED**, both AC3 rows (`-5000 W` with trading off) |
| M6 | instrument: `_reaches()` always `True` | **+0 B** | **RED**, 4 checks incl. its own calibration row |
| M7 | instrument: `_task_sites()` returns `[]` | **+0 B** | **RED**, 10 checks incl. "it found 0 sites" |
| M8 | fixture: 5 minutes of telegrams instead of a whole quarter | **+0 B** | **RED**, the peak-guard precondition and both armed rows |
| M9 | M1 **plus** the harness's guard-driven start defeated | -4 B | the four DRIVEN rows go **GREEN** — see §2 |
| M10 | `provider_id()` drops `.strip()` | -8 B | **survivor**, then CLOSED — see below |
| M11 | `provider_configured()` -> `return True` | -32 B | **RED**, 4 checks |
| M12 | DOCS.md's sentence reverted to the old, vaguer claim | +0 B (docs) | **RED**, the AC4 pairing row (`fetches = True, DOCS says = False`) |
| M13 | the Dutch translation left behind while English moved | +0 B (docs) | **RED**, the both-languages row |
⚠️ **M10 was a real survivor and is recorded as one.** Dropping `.strip()` left
`test_price.py`, `test_price_history.py`, `test_static_tariff.py` and
`test_validate.py` all green: nothing in the repository fed a `price_provider`
with surrounding whitespace, and the `.strip()` predates this card — it was
carried into `provider_id()` from the three call sites it replaced. The
Supervisor's `list(off|…)` schema cannot produce `" off "`, but
`validate_startup_config()` is also `sim/inject.py`'s entry point and takes any
dict at all. Closed in the same commit by the padded-id pair in section 13;
re-run as **M10b**, red on two named checks.
⚠️ **M8 is the fixture defect this card walked into and then fenced.** The
first draft fed five minutes of telegrams. Whether that **covers** the current
billing quarter depends on where the wall clock sits inside it, so the peak
guard was blind and engaged on some runs and not others — and `decide()` refuses
outright while the guard is holding (§9.1 over §10.1). On those runs the armed
baseline did not trade, and "trading off never trades" would have been graded
on a fixture in which nothing could trade either way. It is now 900 s of
telegrams, and the guard's clear state is a **check** rather than an assumption.
## 8. What is still open
* The literal reading of AC2 (§3). One token, and it costs a measured trading
configuration its plan. **Owner's call.**
* `cheap_now()` (§4). Owner's call, with the adoption list above.
* The *Tariff* status row still shows the static-profile sentence on a
provider-backed site; the provider plan's own line goes to the log. Pinned as
a pair with `DOCS.md` by FCST-04 and untouched here.
@@ -0,0 +1,252 @@
# TARIFF-12 — a returned degradation is not a refresh
Three decisions. The first is what the ticket put up for judgement; the second
and third are what fall out of it and would otherwise be decided by accident.
---
## 0. What was actually wrong, stated once
`run_tariff_refresh()`'s docstring:
> ⚠️ **AND A FAILURE DOES NOT CLEAR THE PLAN.** … Zeroing the plan here would
> replace a coarse tariff with no tariff, which is the opposite of what §11.4
> asks for.
The guarantee was real. The condition it was enforced on was not:
```python
try:
plan = await resolve_tariff(...)
self.tariff_plan, self.tariff_at = plan, now # unconditional
except Exception as err:
_LOG.warning("tariff refresh failed (%s); keeping the plan ...")
```
`load_prices()` says in its own docstring that it **"RETURNS, NEVER RAISES, for
every failure a site can actually have"**, because a degradation arriving as a
traceback into the planning tick would stop the controller regulating over an
expired certificate. So a supplier outage — the case the docstring names first —
is a **return**: the `try` succeeds, `_degrade()` hands back a §11.4 plan, and
the good plan is overwritten. The `except` is reachable only by something
nobody expected, and nothing in the shipped path produces it.
This is the fourth card in one family: FCST-04's M4 (`not ok` only), FCST-05's
HIGH (`(JSONDecodeError, OSError)` only), FCST-06 (`not self.ok` only), and now
`except Exception` only, for a path that never raises. **A guarantee enforced on
the condition someone thought of, rather than the condition that occurs.**
Reproduced before the fix, driving the shipped chain with nothing stubbed but
the socket — 48 h horizon, `static_profile` empty, one good fetch then a
`ClientConnectorError` 1800 s later, `TARIFF_MAX_AGE_S` = 5400 s:
```
after a good fetch origin='provider' n_points=48 'dynamic_dayahead from generic_dayahead, 48 h'
after the failed fetch origin='none' n_points=0 'no tariff - generic_dayahead: ClientConnectorError: ...'
tariff_at moved: True
```
and after it:
```
after the failed fetch origin='provider' n_points=48 'dynamic_dayahead from generic_dayahead, 48 h'
tariff_at moved: False
```
---
## 1. The question is asked ONCE, after the attempt, from the evidence that exists
FCST-06's answer, one module over, and for the same reason: **the enumeration
of failure conditions cannot be completed.** There is no pre-check that a feed
will answer which is not itself a guess, and adding one would be a second
symptom test beside the `except`. So the predicate is an **outcome**, read off
the object the attempt produced:
```python
standing = self.fresh_tariff_plan(now)
keep = (not plan.from_feed
and standing is not None and standing.from_feed)
```
Two facts, both of which exist by the time the question is asked:
* **`plan.from_feed`** — did *this* resolution observe the configured provider?
`TariffPlan.origin` already names the four ways a plan is obtained; this
collapses them to the one question a caller holding a previous plan has to
ask. It lives on `TariffPlan` rather than as an `origin == "provider"` at the
call site for `fresh_tariff_plan()`'s own stated reason — *a check has to be
able to assert on the shipped expression rather than on a transcription that
drifts* — and because `origin`'s vocabulary belongs to `static_tariff.py`.
* **`fresh_tariff_plan(now)`** — is a standing observation still current. It
already existed, it is already the one place the age is judged, and it is
**reused, not transcribed**. `test_arbitrage.py` 8b pins that reuse on the
AST; a copy of the comparison behaves identically until the day the two
drift, which is a claim about the code's shape, so a shape check is what
grades it (FCST-06's precedent for the one source check among driven ones).
### What was NOT done
* **Not `except Exception` widened, or a second `if` beside it.** That is the
defect's own shape — a place where the rule is spelt twice, so a fifth cause
needs a fifth branch. The `except` stays, relabelled *"not the outage path"*,
keeps the plan on the same terms, and is now honestly the
something-nobody-expected branch.
* **Not the persisted price store as a cache.** Out of scope by the product
owner's call, and it is the right call: FCST-03 exists precisely to stop
stale prices being acted on, and "recent" is not "current". Keeping the last
**fetched** plan until it ages out is a different promise from serving the
store, and that one needs its own ticket.
* **Not a `no_data`-only guard.** `not plan.has_prices` is the tempting
spelling and it is wrong — see decision 2. It leaves the §11.4 fallback free
to replace a fresh observation, and it is a 1-byte edit from the shipped
line, which is why it is in the mutation table rather than in a comment.
---
## 2. A §11.4 static fallback does NOT replace a fresh provider plan
The ticket asked this directly, because the fallback is *not* nothing: it is
the operator's own configured profile, with real prices, for the requested
window. The answer is still no, and the reason is not that it is worse data —
it is that **nothing may be acted on it, and it is no more current.**
* Every point `_degrade()` builds carries `provisional=True`. So
`firm_points()` is empty, `arbitrage()` finds fewer than two firm hours and
refuses, and `cheap_now()` is False. Both of the two things `TariffPlan`
exposes for a consumer to act on are already off.
* A day-ahead price for a given hour **does not change once published**. A plan
fetched 30 minutes ago is exactly as true about 19:00 as one fetched now; at
most `TARIFF_MAX_AGE_S` of its horizon has elapsed. The observation is not
stale in any sense the fallback is fresher than.
So replacing the observation with the fallback trades a plan with firm hours
for a plan with none, and buys no currency with it. The fallback takes over
when the observation **retires** — which is exactly the row §11.4 describes,
arrived at when there is no longer an observation to prefer.
### And the rule is deliberately NOT symmetrical
`standing.from_feed` is a term in its own right. A standing plan that is itself
a degradation **is** replaced by the next degradation, and its timestamp moves.
Without that term:
* an outage on a static-profile site would freeze the operator's profile at the
window it was first generated for, and the guard would be *"never overwrite"*
wearing a bound's name;
* and on a `price_provider: off` install — the shipped default — the plan would
flip between kept and re-derived every `TARIFF_MAX_AGE_S`, for no behavioural
gain, changing which reason the status page prints on a two-hour cycle.
Each of the three terms is the only false one in some check in 8b, which is
MUTATION.md habit 17's test for whether a suite can tell an `AND` from an `OR`.
---
## 3. `tariff_at` does not move on a kept plan
The docstring's other half is that a plan must not *"freeze at its last value
and look current forever"*. `tariff_at` is what makes that work: it is the
moment of the fetch that produced the plan standing, and `fresh_tariff_plan()`
judges the age from it.
Re-stamping it on a kept plan would be this task asserting the feed answered
when it did not. The plan would be re-certified as current at every refresh, an
outage of any length would never retire it, and `TARIFF_MAX_AGE_S` would be
**exactly as dead as it was before this card** — reachable only by the refresh
task itself dying. Leaving it alone is what makes the bound govern: the plan is
served for the age bound after its own fetch, and no longer.
It is also the whole of **"distinguishable from a freshly-fetched one"**. A
kept plan and a fetched plan are the same object shape; what separates them is
that one's timestamp advanced and the other's did not, and every consumer that
asks how old the plan is already reads that number. The kept case additionally
logs at WARNING, naming the degradation's own reason, the fetch it is keeping,
its age, and the deadline it retires at.
⚠️ **What this does not give the operator, and why it is not here.** The status
page has no row for "the last refresh failed" — the banner/level not moving
when the plan is lost is separately carded, and the ticket rules it out of
scope. This card's change makes that surface *more* truthful, not less: during
the first `TARIFF_MAX_AGE_S` of an outage there is now a real plan behind the
Arbitrage row instead of a no-plan refusal, so the row says something true for
longer. The no-plan state is entered later by up to 5400 s; it is not removed,
and the 30-minute lag reported on that row is untouched by anything here.
---
## 4. Merged after TARIFF-10 — what the wider gate changed, and what it did not
TARIFF-10 landed on `release/1.0` while this card was open. It changed **which
installs run this task** — `if arbitrage.enabled(opts)` became
`tariff_refresh_wanted(opts)`, i.e. `provider_configured(opts) or
arbitrage.enabled(opts)` — while this card changed **what the task does on a
failed fetch**. Git auto-merged the two hunks, which is exactly when the
interaction needs reading rather than trusting.
**Nothing in the decisions above changed.** Two configurations are newly
reachable, and both are now driven in `test_arbitrage.py` 8b rather than
reasoned about:
* **A provider configured with trading DISARMED** — TARIFF-10's headline case.
This is precisely the configuration the guard was designed for: the plans are
`origin == "provider"`, and a dead feed must not delete one that is still
inside `TARIFF_MAX_AGE_S`. It needs no second fixture, because **nothing on
this path reads `arbitrage_enabled` at all**, and that is asserted rather
than assumed — over the AST's symbols and the non-docstring string constants,
because two earlier drafts of that check were **red against a correct tree**:
the first matched the docstring paragraph that *says* `arbitrage_enabled` is
not read, the second matched the log line's "arbitrage, where it is armed".
A substring test over a function grades its prose as much as its behaviour,
which is habit 4 in a third costume. A fixture that set the option and
asserted no difference would have been the other defect — comparing a thing
against itself.
* **`price_provider: off` + a `static_profile`** — the configuration TARIFF-10's
`or` exists to protect, and the one where a latching guard would do real harm.
`resolve_tariff()` returns `static_plan(provisional=False)`, whose origin is
`"static"`: the CONFIGURED product, not a §11.4 row. So `standing.from_feed`
is false on every refresh, `keep` is false on every refresh, and the plan is
regenerated for the current window every 30 minutes exactly as before this
card. **Driven, twice in a row**, because a guard that latched here would
freeze a static site's plan at its first window for the life of the install —
and mutation M3, which drops the `standing.from_feed` term, is red on that
check as well as on the fallback one.
Two further facts settle the rest of the interaction:
* **This decision is strictly downstream of every side effect.** The §8.3 carry
into `price_history` happens inside `resolve_tariff()`, on the `series`
branch, before `keep` is computed — so which of two plans is kept can neither
add nor lose a recorded day, and TARIFF-10's *Price history* row is
unaffected either way.
* **The kept-plan log line was reworded.** It said "arbitrage stops with it",
which after TARIFF-10 is false on a site that fetches without trading; it now
says "arbitrage, **where it is armed**, stops with it".
All eleven mutations were **re-driven on the merged tree** rather than carried
over — the surrounding code changed, so a red against the old base is not
evidence about the new one. Eleven red, no survivors, every revert
fingerprinted. The added member is M11: making the guard conditional on
`arbitrage.enabled(self.o)` (+86 B), red on five named checks.
⚠️ **And one process note that belongs in this file.** This card's mutation
harness was written to a shared scratchpad path and was **overwritten mid-run
by another agent's harness for a different card**, pointing at a different
worktree. That is MUTATION.md habit 18's second bullet almost verbatim. It was
caught because the harness fingerprints the tree after every revert and the
runs are what proved this worktree was never left mutated; the harness now
lives at a card-specific path. **A harness that edits source must not share a
name, a directory or a backup path with another one.**
---
## What this does not change
* **Nothing on the 30 s tick.** `cycle()` still reads two attributes and awaits
no fetch; the AST pin in `test_arbitrage.py` section 8 is unchanged and green.
* **Nothing about the prices themselves.** No series is invented, extended,
re-stamped or carried. The only question this card answers is *which of two
plans the controller holds*.
* **A `price_provider: off` install is bit-for-bit unaffected.** Its standing
plan is never `origin == "provider"`, so `keep` is false on every refresh it
will ever run, and the assignment happens exactly as it did before.
+157 -4
View File
@@ -1346,12 +1346,26 @@ does, and only after you have read why.
> release ships the five provider plugins, the canonical all-in format they
> emit, the config that selects one, and — new in OPT-04 — a price-arbitrage
> strategy that trades on them. With `arbitrage_enabled: false`, which is the
> shipped default, turning a provider on still fetches, validates and publishes
> prices and changes no charging decision.
> shipped default, **selecting a provider is what starts the fetch**: the feed
> is read every 30 minutes, validated, and the day recorded in
> `/data/price_history.json`, which the *Price history* status row reports and
> the add-on log names at every refresh. No charging decision changes.
>
> ⚠️ **That has been the documented behaviour since OPT-04 and was not the
> actual behaviour until TARIFF-10.** The background refresh was started only
> when `arbitrage_enabled` was on — and it ships `false` — so an install with a
> provider configured fetched nothing at all, never created the store, and read
> *Price history: empty* for as long as it ran. If you configured a provider and saw an empty
> Price history row, that is what you were looking at. Trading is unchanged and
> is still `arbitrage_enabled` alone.
>
> `static_profile` below needs no network at all, so its cheap window is
> computed on every cycle and reaches the maintenance cycle's grid top-up
> whether or not arbitrage is armed.
> whether or not arbitrage is armed. ⚠️ **And it is still the only source of
> that window.** A provider-backed plan now exists on every configured site,
> but *Cheap window* and the maintenance cycle's grid top-up read
> `static_profile` and nothing else — a deliberate limit, recorded with what it
> would take to change in `DECISIONS/TARIFF-10-fetching-is-not-trading.md`.
| option | meaning |
|---|---|
@@ -1360,7 +1374,7 @@ does, and only after you have read why.
| `price_taxes_eur_kwh` | Levies and VAT, EUR/kWh. Same rule |
| `export_price_eur_kwh` | The **injection** price. A separate scalar, not a series — see below |
| `price_csv_path` | Where `manual_csv` reads its file |
| `price_feed_url` | Override the plugin's default feed. Only a host the plugin itself declares is accepted |
| `price_feed_url` | Override the plugin's default feed. Used **exactly as written** — the add-on appends nothing to it, including the date range, so an override gets whatever horizon that URL returns. Only a host the plugin itself declares is accepted |
| `price_horizon_h` | How far ahead to ask for, hours |
| `price_timeout_s` | Per-request timeout |
| `static_profile` | A static day/night or fully-fixed tariff. Empty by default — see below |
@@ -1395,6 +1409,58 @@ flagged as high-volatility, which widens arbitrage thresholds once the
optimiser lands. That comes from the plugin's own declaration, not from its
name.
**`generic_dayahead` and the shape of the public feed.** It reads Fraunhofer
ISE's Energy-Charts API (`https://api.energy-charts.info/price?bzn=BE`, no key
and no registration), which publishes a *columnar* document — parallel
`unix_seconds` and `price` arrays, in EUR/MWh, with the unit and the licence
stated in the response. Two things follow that are worth knowing before you
point `price_feed_url` somewhere else:
* The timestamps are **UTC instants**. They are rendered at the Brussels offset
that applies to each one, so the hour that happens twice every October is two
points rather than one overwritten one.
* Belgium moved to a **15-minute market time unit** on 2025-10-01, and the feed
followed. A published day is **96 points, not 24**. Nothing in the add-on
assumes either spacing, and nothing aggregates the quarters away.
* The request **names the days it wants**, derived from the horizon on the
**Brussels** calendar — `?bzn=BE&start=2026-09-09&end=2026-09-10`, two
inclusive local dates. Without that the feed answers a single day, and
everything past today would be carried forward from yesterday's profile and
flagged provisional — which §8.3 excludes from arbitrage, so tomorrow would
never be tradeable at any hour. The calendar is Brussels rather than UTC
because 22:30 UTC in summer is already tomorrow in Belgium.
**What you get before the auction publishes.** The day-ahead auction clears at
about 13:00–14:00 CET. Ask for tomorrow before that and the feed returns fewer
days than requested — that is normal, not a failure. The add-on keeps the
series of what exists and fills the remaining hours from yesterday's profile,
flagged provisional; arbitrage runs on the firm hours and ignores the flagged
ones. So a dynamic site's plan is part firm and part carried every morning and
fully firm once tomorrow is published, and neither state is an error.
⚠️ **If you set `price_feed_url`, you keep the short horizon.** The override is
sent exactly as you wrote it and the add-on adds no `start`/`end` to it, because
those are this feed's parameter names and an override may point somewhere with
different ones — rewriting a URL you typed is how a request nobody intended gets
made. The consequence is a **`WARNING` in the log on every price refresh**,
naming the range that would otherwise have been requested. If you only wanted a
different host and not a different horizon, the fix is to clear the option.
If the arrays ever come back different lengths, or the unit stops saying
EUR/MWh, the fetch is refused outright rather than trusted in part: a price
series missing its last hours is a series missing the evening peak, and it
would be traded on as though it were complete. A refused fetch is the §11.4
row — static profile, provisional plan, no arbitrage — and the reason is in the
log.
**A refused fetch does not take the plan you already have.** The last plan that
actually came from the feed keeps standing until it is 90 minutes old (three
missed refreshes), and only then does the §11.4 row above take over. The
add-on logs a `WARNING` naming the failure, the fetch it is keeping, its age
and the moment it retires; the plan's own age is **not** reset by the failed
refresh, so a kept plan still expires on schedule and is never presented as a
current one. One failed fetch is a blip, not a lost tariff.
**`manual_csv` — the offline fallback.** It needs no internet, no supplier
account and no HTTP stack at all, and it is the supported way to run this
add-on price-aware on a site with no outbound connectivity. Write a file and
@@ -1424,6 +1490,85 @@ distinct state from "prices, all of them zero": the plan is marked provisional
and arbitrage is disabled, per the degradation matrix. It never falls back to a
price of zero, which would read as free electricity.
#### The hours the feed has not published yet (FCST-04)
A day-ahead auction publishes tomorrow's prices once a day, at **14–16 h CET**.
Before that, the feed is serving today and nothing else — so a `price_horizon_h`
of 48 asks for 48 hours and gets back the ten or fifteen that are left in the
day. Until FCST-04 those other hours were simply **absent** from the plan: not
priced badly, not flagged, just missing, and nothing said so.
Now the add-on keeps the **published prices of the last few days** in
`/data/price_history.json` and fills an unpublished hour of the horizon with
yesterday's price for the same Brussels wall-clock hour, flagged
`provisional: true`. Two things follow, and the second one matters more:
- **The plan says it is guessing** — `provisional: true` on every carried hour,
which is what the optimiser reads. Before, a half-length plan looked
completely settled.
- **Nothing may trade on a carried hour.** §8.3 forbids acting on a provisional
price, so arbitrage drops those hours from the trade set entirely — it needs a
buy hour and a sell hour that were *both* published. Carrying a price forward
widens what the optimiser can **see**; it never widens what it may **do**.
**"An unpublished hour", not "every unpublished hour"** — the fill can only
carry an hour yesterday actually published. If yesterday's record is itself
partial (a first day, an outage, a feed that only ever serves the rest of the
day), the local hours it never held are **left out of the series**: §11.4
applies to those hours, and asking the plan for one raises rather than
answering. An hour with no price is not a free hour, and the fill does not
invent a neighbour, a mean or a zero for it.
**A quarter-hourly feed keeps all four of its prints.** `eneco_eddy` settles
per quarter, and the fill decides *which whole hours need carrying* — never what
a published hour may contain. Every point the provider published inside the
horizon is passed through untouched; a carried point is added only for a whole
hour the feed did not print at all.
**What you can actually see on the status page.** *Price history* (below) says
what is held. *Arbitrage* carries the strategy's own sentence, which names the
firm/provisional split whenever that split is what refused a trade. *Tariff*
does **not** report the fill: it is the static-profile or Home Assistant
price-entity sentence, regenerated on the 30 s tick. The provider plan's own
line — `dynamic_dayahead from generic_dayahead, 48 h, provisional` — goes to the
**add-on log** at every refresh, alongside `§8.3: 33 of 48 hours carried forward
from yesterday's profile and flagged provisional; 15 published`.
The plan is also now cut to the horizon you asked for, which means **hours that
have already elapsed are no longer in it**. That is a real change to arbitrage:
a cheap night that has already gone used to sit in the series as the cheapest
hour of the day, and a cycle can never buy in an hour that is past — so the
trade was quietly never taken. The spread is now taken over hours still ahead.
A new status row, **Price history**, says which days are held, how many price
points each holds and how many whole hours those cover — `2026-01-13 (96 points
over 24 h)`. ⚠️ **Those are two different numbers and the row used to print one
of them under the other's name**: it counted points and labelled them `h`, so a
quarter-hourly feed read `96 h` for a 24-hour day. The figure that says whether
tomorrow morning's fill has a whole day to carry is the **hour** one. If it
reads `empty` or `unavailable`, nothing is carried, the horizon
is as short as the feed makes it, and every other guarantee is unchanged — a
cache that cannot answer is not allowed to be a controller that cannot regulate.
The store is rebuilt by the next successful fetch; on a brand-new install the
first fill is possible the day after the first refresh.
The same applies to a file that is *partly* broken: a day record this build
cannot read is dropped, the row says how many were dropped and why, and the rest
of the file still loads. And a store that **cannot persist records nothing** —
not even in memory. That covers a missing `/data`, and it also covers a `/data`
that is there and cannot be **written**: a full volume, a read-only remount. An
in-memory yesterday would answer for one run and vanish on the next restart,
which is a cache that forgets without saying so; the row naming the reason and
*no prices carried* are the honest pair.
⚠️ **What a failed write does not do is throw away what is already in the
file.** A volume that fills at 14:00 leaves the yesterday written at 13:30 still
answering, because that record survives a restart — the store stops *adding* to
it, nothing more. And it never stops the add-on: a write that cannot complete is
logged and reported on the status row, never raised. A cache that cannot answer
is not allowed to be a controller that cannot regulate, and that applies to the
volume being full exactly as it applies to the volume being absent.
**Injection is priced separately** (`export_price_eur_kwh`). In Belgium it is
often near zero, which by itself strongly favours self-consumption over export.
It is a single number rather than a curve because no supported supplier
@@ -1547,6 +1692,14 @@ towards the firmware's 30 s failsafe. The plan refreshes every 30 minutes and
retires itself if the refresh stops answering, at which point arbitrage stops
with it.
⚠️ **That refresh is not gated on this switch** (TARIFF-10). It runs when a
`price_provider` is configured *or* when arbitrage is armed, so turning
arbitrage off does not turn the price feed off — and a site with no provider
that arms arbitrage on a `static_profile` keeps the plan it has always had. The
only thing that decides whether a request leaves the machine is
`price_provider`: with it `off`, the refresh resolves the static profile in
memory and opens no connection.
### Site
| option | meaning |
+116
View File
@@ -127,6 +127,43 @@ Two rules come out of it, and both are cheap:
function, that pair is the first thing to try — TEST-16's two survivors were
adjacent lines in one function and neither was individually live.
⚠️ **AND FCST-05 WALKED INTO IT AGAIN, IN ITS OWN NEW CHECK, WHICH IS WHY THE
RULE IS “mutate every check you add”.** `price_history.py` reports a malformed
point row from **two** places into **one** field: `_usable_days()` records what
it FOUND when the file was opened, and `_remember()` records what it DROPPED on
the next refresh. Both land in `last_error`, and `words` prints `last_error`.
The first draft of the new check asked `"malformed point row" in words` — which
whichever rung is still standing satisfies — so deleting `_remember()`'s note
entirely was a **−1 B SURVIVOR of the whole suite**, on the half of the fix that
tells an operator the cache **self-healed**. It is graded on the TENSE now
(`… in <day>` at open, `… dropped from <day>` after the refresh), the two are
asserted to DIFFER, and the paired mutation — both notes deleted, −91 B — is
red on 15 named checks. Same shape as TEST-16 and the same fix: **grade the
reason, not the verdict**, and when two rungs answer one stimulus, mutate them
together.
⚠️ **AND FCST-06 FOUND THE COUNT ITSELF LYING — LOW, NOT SILENT.** A mutation
whose red is a **traceback** does not report one red; it reports however many
checks ran before the traceback, and grades **none** of the ones after it. The
new section drove M4 (`self._days = before` → `self._days = {}`) and the run
printed **1 FAIL**. The true figure is **3**: an f-string in the next check read
`len(read.points)`, and `.points` on a `no_data` **raises `Absence` by design**,
so the harness died mid-section with the remaining checks ungraded. That is
`why()`'s and `derived()`'s rule — *"it answered nothing" and "it broke on the
way to answering" must not grade the same* — arriving in the **printed operand**
of a check rather than in its condition, which is the half that is easy to miss
because it looks like formatting. Two things follow:
* **a mutation report's red COUNT is only a lower bound until the run is seen to
have reached the end of its section.** Read the output, not the count;
* **the total accessor has to cover what a check PRINTS, not only what it
asserts.** `test_price_history.py` 5d added `npts()` beside `why()` for
exactly this, and re-driving M4 against it moved 1 FAIL to 3.
Worse, had the traceback landed one check earlier the mutation would have
produced **no named red at all** — a result that reads like a broken harness and
gets retried rather than read.
## The third way a check lies: its DATA cannot reach the failure it names
Stale bytecode manufactures survivors from a mutation that never ran. Masking
@@ -658,6 +695,84 @@ Cheap habits fall out of it, and all of them are now used in this repository:
that two of it cannot silently overwrite each other's work. A fixed path
in a tool that edits source is a shared mutable global with a filename.
19. ⚠️ **A FIXTURE IN A SHAPE NOBODY PUBLISHES GRADES THE PARSER AGAINST THE
PARSER — FOURTEENTH INSTANCE OF HABIT 4, AND THE FIRST WHERE THE FIXTURE
AND THE CODE UNDER TEST SHARED A *SHAPE* RATHER THAN A NUMBER.** TARIFF-08.
`price.py`'s `_points_from_json()` read `doc["data"]` as a list of
`{start, price_eur_mwh}` objects. `api.energy-charts.info` publishes
parallel `unix_seconds` / `price` arrays and has no `data` key at all, so
`generic_dayahead` raised `SourceUnavailable` on **every live fetch it ever
made** — and the suite was green, because all three price suites built
their bodies with `json.dumps({"data": [...]})`. The fixture was derived
from the parser's guess, so no mutation of the parser and no amount of
coverage inside that loop could ever have reached the failure. §11.4 then
turned it into a *correct* silent degradation: parse error -> `no_data` ->
static profile -> arbitrage off, with nothing naming the cause.
Habit 2 with a network on the end of it: **the reference has to be data the
code under test did not shape.** For a parser that means BYTES, captured
from the source and committed unmodified, with the URL, the retrieval time
and a checksum beside them - `scenarios/energy-charts-be-price-*.json` and
its `-captures.json` provenance, re-derived by the replay so an edited
fixture stops being a capture. Two granularities, because the same endpoint
answers hourly before 2025-10-01 and quarter-hourly after it and a fixture
at one spacing cannot grade an assumption about the other.
⚠️ **AND THE SURVIVING SYNTHETIC FIXTURES ARE NAMED FOR WHAT THEY ARE.**
Three shipped plugins point at endpoints nobody has verified (TARIFF-07),
so no capture exists for them; their fixture builder is called
`synthetic_row_feed` at every call site, because the next reader's default
assumption will otherwise be that a green check means the feed parses.
⚠️ **ONE OF THIS CARD'S OWN NEW LINES WAS DECORATION AND THE BATTERY SAID
SO.** The columnar parser first rendered its instants with `brussels()`
before returning them. `assemble()` already renders every plugin's points,
so removing the parser's call changed nothing observable - **M4 came back a
survivor on a `:ro`, purged tree**, which is not the direction stale
bytecode fabricates. The line was removed rather than fenced with a check:
a second copy of a rule that already has an owner is indistinguishable from
a no-op, and the mutation that proves the rendering is real belongs on
`assemble()`, where it is now aimed.
20. ⚠️ **A FIXTURE GRADES THE RESPONSE. NOTHING ABOUT THE REQUEST IS REACHABLE
FROM ONE, AND THE TRANSPORT HAD BEEN RECORDING IT ALL ALONG.** TARIFF-09,
and it is habit 19's closed loop with the other half still shut.
`generic_dayahead.fetch()` took `since` and `until` and asked for the bare
`?bzn=BE`, which answers ONE Brussels day - so §8.3 dropped every hour past
today from the trade set as provisional and the optimiser never saw
tomorrow as firm, at any hour of any day. Measured at 09:00: **15 firm
hours and 33 carried of 48**, both ends of the permitted cycle inside one
calendar day; at 23:00, **1 and 47**, with the firm spread no longer
clearing the round-trip cost.
TARIFF-08 had just fixed the RESPONSE side of this same plugin by
committing real bytes. It could not have caught this: a captured body is
the answer to a URL, and **no property of any body says which URL was
asked for**. `canned()` in `test_price.py` had recorded `(url, timeout_s)`
on every call since TARIFF-02 and not one check had ever asserted on the
first element - so the harness *knew* the plugin was ignoring its arguments
and nothing asked it. The fix is one line of habit: **where a plugin takes
arguments that shape a request, assert on the REQUEST off the injected
transport, not only on what comes back.** Section 2c does, and the
shipped-defect mutation (restore the bare URL, `-173 B`) is red on six
named checks in `test_price.py` and two in `test_price_history.py`.
⚠️ **AND A MUTATION THAT TURNS A READ INTO `no_data` CRASHES A SUITE
INSTEAD OF NAMING A CHECK, UNLESS THE CHECK EXPECTS IT.** `_NoData.points`
raises `Absence` by design, so `derived(...) < derived(...)` over two
absent reads is a `TypeError` and `[p for p in plan.read.points]` is a
traceback - habit 5's shape with an exception in place of a hang. The
mutation that matters here is the "defensive" one somebody writes after
reading this card: *refuse when the feed returned fewer days than we asked
for*. It is exactly what a healthy feed does every morning before the
~13:00-14:00 CET auction, and switching arbitrage off for it would be a
worse defect than the one being fixed. It is red BY NAME in
`test_price.py` (33 checks, the AC's own line among them) and takes
`test_price_history.py` down at **FCST-04's section 1**, whose `_before`
read is unguarded - a red, but an unnamed one. That is recorded rather than
retrofitted: guarding another card's section is a change to what its
checks report, and it belongs to whoever next touches it.
And the mutation-table consequence: **a variant is not covered because a
near-miss of it is.** DEV-02's table listed the *actuator* variant of the
per-unit claim defect (`device`/`supervised` filed without `inv`), which the
@@ -715,6 +830,7 @@ commit and PR bodies.
| **SAFETY-03** reviewer survivor: `abs()` dropped from `clamp_to_rating`'s limit (6aabb83) | **re-verified — still a survivor** | re-run the same way, all 15 green. Also as recorded: unreachable under the `int(100, 5000)` schema. |
| **MAINT-05 round 3**, `migrate_maint_keys()` marker gate `if rep.ok:` → `if rep.ok or rep.kept:` (`+12 B`) | **was a survivor — now CLOSED** | measured against head `65cddf3` under the purge, tree mounted `:ro`, `19/19 PASS`. It stamps the schema marker on a migration that LOST state, defeating the older `lost` guard as well as the new `misdirected` one, because every check that reached the marker did so on a store whose `kept` list was empty. Closed by `test_maintenance.py` **9d**; the mutation is now red on three named checks. |
| **MAINT-05 round 3**, `migrate_unit_keys()` marker gate `if rep.ok:` → `if rep.ok or rep.kept:` (`+12 B`) | **found here — SAFETY-02's half, now CLOSED** | the SAME mutation one function up, on the LATCH migration rather than the history one. Measured a survivor against this branch under the purge, `:ro`, `19/19 PASS`, then closed by `test_inverter.py` part 10's new fixture — a store with `unit/inverter-a/sign_reason` already present (so `kept` is non-empty) whose `sign_tripped: true` write is dropped. Shipped: marker withheld. Mutant: marker stamped on a **disarmed latch**, and the next start skips the migration that would have re-armed it. |
| **TARIFF-09** 11 mutations over `app/price.py` and the two-day capture, all red, no survivors | **not at risk** | direction, and method: every mutation was applied to a FRESH `cp -a` of the clean worktree rather than reverted, so there is no revert to verify (habit 18) and no shared scratch path two runs could collide on. Each ran through `tools/mutation_run.sh`, which purges and fingerprints. Byte deltas recorded on the PR; one is **+0 B** - a single price digit edited inside the committed capture, the risky delta - and the re-derived sha256 catches it. |
| **TARIFF-01**'s other three survivors, **SAFETY-03**'s `describe()` survivor, **TARIFF-02**'s M3/M5/M7, **SAFETY-18**'s three | **still unverified** | recorded before the purge discipline, with no byte delta on the record, so a `+0 B` same-second revert cannot be excluded. All are survivors, which is the direction this defect fabricates. TARIFF-02's and SAFETY-18's were **closed by later commits whose claims are reds**, so what is unverified is the *diagnosis*, not the current state of the code. |
**Nothing here says a past result is wrong.** The two survivors re-run above
+203 -17
View File
@@ -67,7 +67,8 @@ from .mqtt import (MqttPublisher, announced_units, unit_slugs,
from . import peakguard
from . import packshare
from .p1 import P1Ingest, build_source, is_enabled
from .price import Uncertainty, price_option
from .price import Uncertainty, price_option, provider_configured
from .price_history import PriceHistory
from .static_tariff import resolve_tariff, static_status
from .tariff import PeakStore, PeakTracker
from .store import migrate_maint_keys, migrate_unit_keys
@@ -137,6 +138,60 @@ TARIFF_REFRESH_S = 1800.0
TARIFF_MAX_AGE_S = 3 * TARIFF_REFRESH_S
def tariff_refresh_wanted(opts: dict) -> bool:
"""TARIFF-10. Does this install need the background tariff refresh at all?
THE DEFECT THIS PREVENTS, and it shipped: the gate below used to be
`arbitrage.enabled(opts)` alone. That option defaults to `false`, and
`run_tariff_refresh` is the only caller of `resolve_tariff()`, which is the
only caller of `price_history.carry()` - so on a shipped install with a
provider configured NOTHING was ever fetched, no plan was ever resolved,
`/data/price_history.json` was never created, the *Price history* row read
`empty` forever, and TARIFF-09's short-horizon warning could not fire
because there was no fetch to warn about. Measured over a 40 h soak of two
instances side by side: zero feed requests and `present: false` across all
802 samples on the shipped default, against a resolved plan from the same
feed on the instance with arbitrage armed. ⚠️ AND `DOCS.md` HAS
DESCRIBED THE BEHAVIOUR BELOW SINCE THE COMMIT THAT WROTE THE GATE:
OPT-04 (bdd7f18) added the refresh task, the `arbitrage.enabled()` guard on
it, and the sentence saying a configured provider still fetches, all in one
change. config.yaml has said it since TARIFF-02 (852fc7c).
Two terms, and they answer two different questions:
provider_configured(opts) -> "the operator asked for a price feed", so
fetch it, validate it and record the day
(§8.3). This is the term TARIFF-10 ADDS.
arbitrage.enabled(opts) -> "the operator armed trading". This term is
what the gate USED to be, kept verbatim.
⚠️ IT IS AN `or`, SO THE GATE IS A STRICT WIDENING. Every configuration
that started this task before still starts it; none that did not start it
stops anything. That matters because the second term is not redundant:
`price_provider: off` + a `static_profile` + `arbitrage_enabled: true` is a
configuration validate.py explicitly ACCEPTS, and it trades - measured, at
`cycle_cost_threshold_eur: 0.05` on a 0.20/0.284 day/night profile,
`ARBITRAGE_CHARGE -5000 W`. Dropping that term to make the gate read
`provider_configured(opts)` alone would leave `self.tariff_plan` at None on
that site forever and `decide()` would answer "no tariff plan has been
resolved yet" on every cycle - silently disarming a strategy the UI says is
running, which is the exact failure validate.py's OPT-04 rules exist to
refuse.
⚠️ AND NO CONFIGURATION ACQUIRES A PERIODIC HTTPS CALL FROM THIS TICKET.
With `price_provider` off, `resolve_tariff()` returns `static_plan()` - pure
arithmetic, no socket - so the second term starts a task that fetches
nothing. The true shipped default (no provider, no arbitrage) starts no
task at all. What is fetched is decided by the FIRST term only.
⚠️ THIS IS NOT A TRADING SWITCH. It decides whether prices are LOOKED UP.
Whether they move energy is `arbitrage.enabled()` at the one call site in
`cycle()`, and TARIFF-10 did not touch it: `arbitrage_enabled: false` still
means no trade, ever, on any configuration.
"""
return provider_configured(opts) or arbitrage.enabled(opts)
def load_options() -> dict:
try:
with open(OPTIONS_PATH, "r", encoding="utf-8") as fh:
@@ -165,7 +220,8 @@ def pack_energy_of(inv):
class Controller:
def __init__(self, opts: dict, hass: HomeAssistant, store, mqtt_pub, *, stop,
peak_db_path: str = "/data/tariff.db"):
peak_db_path: str = "/data/tariff.db",
price_history_path: str = "/data/price_history.json"):
self.o = opts
self.hass = hass
self.store = store
@@ -582,6 +638,42 @@ class Controller:
self.peak.restore(self.p1.averager)
_LOG.info("capacity-tariff peak store: %s", self.peak.restore_state)
# FCST-04. The §8.3 carry-forward store: the published prices of the
# last few Brussels days, so the hours a day-ahead feed has not printed
# yet can be filled from yesterday's profile and FLAGGED, instead of
# being absent from the series entirely.
#
# ⚠️ IT IS AN INSTRUMENT'S BACKING FILE, NOT A STRATEGY. It commands
# nothing and claims nothing; what it changes is how many hours the
# plan covers. Before this, a dynamic site between midnight and the
# 14-16 h CET publication had a series covering only the rest of today
# - typically 10-24 of the 48 hours `price_horizon_h` asked for - and
# `TariffPlan.arbitrage()` took its spread over that truncated window
# without anything reporting that it had. Every carried hour is
# provisional, so refusal 3 drops it from the trade set: CARRYING a
# price widens what the optimiser can SEE, never what it may act on.
#
# ⚠️ AND THE CAVEAT THAT BELONGS WITH THAT SENTENCE, because this is
# the one place a maintainer reads it on its own. THE OTHER HALF OF
# THIS CHANGE DOES MOVE ENERGY, and it is the windowing rather than the
# carrying: the filled series covers `[since, until)`, so hours that
# have ALREADY ELAPSED are no longer in the spread. A cheap night that
# has gone used to sit in the series as the day's cheapest hour, and no
# cycle can buy in an hour that is past - so the trade was quietly
# never taken. `resolve_tariff()`'s docstring, DOCS.md and the CHANGELOG
# all carry this beside the guarantee; it may not appear here without it.
#
# ⚠️ A BROKEN OR ABSENT STORE IS NOT AN ERROR HERE. It answers `no_data`,
# the fill carries nothing, and the plan is the pre-FCST-04 one minus
# the elapsed hours - short, but honest and still refusing on the same
# rules. It must not be an error in the other sense either: a day
# record this build cannot read is DROPPED by `PriceHistory._load()`
# and reported in `.words`, because both the constructor above and the
# `.words` read below are unguarded, and a cache that raises here is an
# add-on that does not start.
self.price_history = PriceHistory(price_history_path)
_LOG.info("§8.3 price history: %s", self.price_history.words)
# SAFETY-07. The guard TARIFF-01 was built for. It is stateless by
# design (peakguard.py holds no history between calls), so the only
# state here is what an EDGE needs: what the last cycle concluded, so
@@ -3442,25 +3534,99 @@ class Controller:
raise out of here and take the task down, because then the plan freezes
at its last value and looks current forever. That is what
`fresh_tariff_plan()`'s age bound is the backstop for, and this is the
half that keeps the task alive so the bound rarely matters.
half that keeps the task alive. ⚠️ THE BOUND IS NO LONGER A BACKSTOP
ONLY: since TARIFF-12 it is also the serve-stale window an outage runs
on, and it is consulted below on every refresh.
⚠️ AND A FAILURE DOES NOT CLEAR THE PLAN. `resolve_tariff()` already has
a §11.4 answer for a dead feed - degrade to the static profile, mark it
provisional, arbitrage refuses - and that answer is better than the
absence of one. Zeroing the plan here would replace a coarse tariff with
no tariff, which is the opposite of what §11.4 asks for.
⚠️ AND A FAILURE DOES NOT CLEAR THE PLAN - WHICH IS A CLAIM ABOUT WHAT
`resolve_tariff()` RETURNS, NOT ABOUT WHAT IT RAISES (TARIFF-12). This
paragraph was enforced by the `except` below alone, and the `except`
below is the one path a supplier outage never takes. `load_prices()`
says so in its own words - it "RETURNS, NEVER RAISES, for every failure
a site can actually have" - because a degradation arriving as a
traceback into the planning tick would stop the controller regulating
over an expired certificate. So a dead feed is a RETURN: the `try`
succeeds, `_degrade()` hands back a §11.4 plan, and the assignment used
to overwrite the good one unconditionally. Measured in a 40 h soak with
`static_profile` unset (the shipped default, and the normal state of a
dynamic-tariff site): the FIRST failed fetch replaced a complete 48 h
plan with `TariffPlan(no_data, None, "none")` - 0 points - 27 minutes
after the last good one, and `TARIFF_MAX_AGE_S` governed nothing. A
transient blip cost the same as a 90-minute outage.
⚠️ SO THE QUESTION IS ASKED ONCE, AFTER THE ATTEMPT, FROM THE EVIDENCE
THAT EXISTS - FCST-06's answer, in this file. The evidence is
`plan.from_feed` (did this attempt observe the feed) and
`fresh_tariff_plan(now)` (is an observation still current). A
degradation replaces a standing observation only once that observation
has aged out, so `TARIFF_MAX_AGE_S` is now the serve-stale window it
reads like rather than a bound nothing could reach.
⚠️ AND A §11.4 STATIC FALLBACK DOES NOT REPLACE A FRESH PROVIDER PLAN
EITHER, although it carries real prices. Every point `_degrade()`
builds is `provisional=True`, so `firm_points()` is empty,
`arbitrage()` refuses and `cheap_now()` is False - everything a
consumer may act on is already off. A day-ahead price for a given hour
does not change once published, so a plan fetched half an hour ago is
exactly as true about that hour as one fetched now. The fallback is
less actionable and no more current; it takes over when the observation
retires, which is what §11.4 is for. The reverse is not symmetrical: a
standing plan that is ITSELF a degradation is replaced by a fresh one,
or an outage would freeze the operator's static profile at the window
it was generated for.
⚠️ AND NOTHING HERE READS `arbitrage_enabled`, WHICH TARIFF-10 MAKES
WORTH SAYING. Since that card this task also runs on a site with a
provider configured and trading DISARMED, so "keep the plan" is a
decision about what the add-on has LOOKED UP, not about what it may
trade on. It is also strictly downstream of `resolve_tariff()`: the
§8.3 carry into `price_history` happens on the `series` branch inside
that call, before this decision exists, so which of two plans is kept
cannot add or lose a recorded day.
⚠️ AND `tariff_at` DOES NOT MOVE ON A KEPT PLAN. It is the moment the
fetch that produced the standing plan happened, and
`fresh_tariff_plan()` judges the age from it - re-stamping it here
would be this task asserting the feed answered when it did not, and the
plan would "freeze at its last value and look current forever", which
is the hazard the paragraph above this one names. An unmoved
`tariff_at` is also the only thing that tells a kept plan from a
freshly fetched one, so it is what the check asserts.
"""
horizon = float(price_option(self.o, "price_horizon_h"))
while not self.stopping:
now = datetime.now(timezone.utc)
try:
plan = await resolve_tariff(
self.o, now, now + timedelta(hours=horizon), now=now)
self.tariff_plan, self.tariff_at = plan, now
_LOG.info("tariff refreshed: %s", plan.words)
self.o, now, now + timedelta(hours=horizon), now=now,
history=self.price_history)
standing = self.fresh_tariff_plan(now)
# Three terms, and each of them is the only false one in some
# check: the feed answered (replace), nothing current stands
# (take the degradation - this is what makes the age bound
# govern), or what stands is itself a degradation (replace).
keep = (not plan.from_feed
and standing is not None and standing.from_feed)
if keep:
_LOG.warning(
"tariff refresh degraded (%s); KEEPING the plan "
"resolved at %s, %.0f s ago. tariff_at is NOT moved, "
"so it retires %.0f s after that fetch; arbitrage, "
"where it is armed, stops with it. A kept plan is "
"never presented as a current one.", plan.words,
self.tariff_at,
(now - self.tariff_at).total_seconds(),
TARIFF_MAX_AGE_S)
else:
self.tariff_plan, self.tariff_at = plan, now
_LOG.info("tariff refreshed: %s", plan.words)
except Exception as err: # noqa: BLE001
# ⚠️ NOT THE OUTAGE PATH - see the docstring. `load_prices()`
# returns on every failure a site can have, so what reaches
# here is something nobody expected, and the plan is kept for
# the same reason and on the same terms.
_LOG.warning(
"tariff refresh failed (%s); keeping the plan resolved at "
"tariff refresh raised (%s); keeping the plan resolved at "
"%s. It retires on its own after %.0f s and arbitrage "
"stops with it.", err, self.tariff_at, TARIFF_MAX_AGE_S)
with contextlib.suppress(asyncio.TimeoutError):
@@ -4304,9 +4470,24 @@ class Controller:
("Maintenance due", "now" if next_due is None else next_due.strftime("%Y-%m-%d")),
("Cheap window", "yes" if self.cheap else "no"),
("Tariff", self.tariff_words),
# FCST-04. WHAT §8.3 HAS TO CARRY FORWARD, which is the difference
# between a 48-hour plan and a 12-hour one on a weekday morning.
# It is its own row rather than a clause of "Tariff" because it
# answers a different question: "Tariff" says what the plan IS, and
# this says what the plan could still be filled from if the feed is
# late again tomorrow.
("Price history", self.price_history.words),
# OPT-04. The strategy's own sentence, not a re-derivation of it -
# every refusal it can make already carries a reason a human can
# act on, and "Arbitrage: no" would throw all of them away.
#
# ⚠️ TARIFF-10 RE-READ THIS STRING AND LEFT IT ALONE, which is worth
# saying because that card is about the two switches being
# different. `self.arb` is None if and only if
# `arbitrage.enabled(self.o)` was False, so `off (arbitrage_enabled)`
# still names the option that is actually off - and on a site with a
# provider configured it is now the row that says the prices
# arriving on the *Price history* row above are moving nothing.
("Arbitrage", "off (arbitrage_enabled)" if self.arb is None
else self.arb.explain()),
]
@@ -4559,11 +4740,16 @@ async def amain() -> None:
# ESP32 heartbeat. A fetch on that path is a supplier's TLS timeout able
# to push this add-on towards layer 1's 30 s veto. See TARIFF_REFRESH_S.
#
# ⚠️ STARTED ONLY WHEN ARBITRAGE IS ARMED. Nothing else in the add-on
# reads the resolved plan - `cheap` still comes from static_status() on
# the tick, which is pure - so on a default install this task would be
# network traffic in aid of an attribute nobody reads.
if arbitrage.enabled(opts):
# ⚠️ STARTED WHEN A PROVIDER IS CONFIGURED, *OR* WHEN TRADING IS ARMED
# - TARIFF-10, and it used to be the second term alone. `arbitrage_
# enabled` ships `false`, so a site that had selected a price feed
# fetched nothing at all; the whole argument, the measurement and what
# the `or` is protecting are in `tariff_refresh_wanted()` above.
#
# ⚠️ IT DOES NOT ARM ANYTHING. Fetching a price and trading on one are
# two decisions; this is the first, and `cycle()`'s
# `if arbitrage.enabled(self.o):` is still the only gate on the second.
if tariff_refresh_wanted(opts):
tasks.append(asyncio.create_task(controller.run_tariff_refresh()))
# P1 ingestion runs as its own long-lived task. ⚠️ It must not be driven
+373 -6
View File
@@ -140,6 +140,7 @@ from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import Enum
from textwrap import dedent
from urllib.parse import urlencode
_LOG = logging.getLogger("goodwe.price")
@@ -184,6 +185,87 @@ def brussels(instant: datetime) -> datetime:
return instant.astimezone(brussels_offset(instant))
def brussels_date_range(since: datetime, until: datetime) -> tuple:
"""The Brussels calendar days a half-open window [since, until) touches.
Returns `(first, last)`, two `date`s, BOTH INCLUSIVE - which is the shape
`api.energy-charts.info` takes: `?bzn=BE&start=2026-09-09&end=2026-09-10`
answered 192 quarter-hours across exactly those two local days, measured
2026-09-10T07:45:49Z and committed as
`scenarios/energy-charts-be-price-2026-09-09_2026-09-10-twoday.json`.
THE DEFECT THIS EXISTS TO PREVENT (TARIFF-09). `generic_dayahead.fetch()`
accepted `since` and `until` and requested the bare default URL, so the
feed answered ONE Brussels day. Everything past today was then carried
forward from yesterday's profile, flagged provisional, and §8.3 dropped all
of it from the trade set - so the optimiser never saw tomorrow as firm, at
any hour. Measured on the merged tree at 2026-09-10T09:00+02:00 over a 48 h
horizon: 15 firm whole hours and 33 carried, and both ends of the only
permitted cycle inside ONE calendar day. The ceiling was structural: at
hour H of the local day a one-day window gives at most 24-H firm hours and
24+H carried - 1/47 at 23:00, where the firm spread no longer even clears
the round-trip cost. That is not what FCST-04's fill is for; it covers the
pre-publication window until the ~13:00-14:00 CET auction, after which a
healthy feed hands over tomorrow in full.
⚠️ THE CALENDAR IS BRUSSELS, NEVER UTC, AND THIS IS `_bdate()`'s LESSON ONE
LAYER UP. `since`/`until` are INSTANTS; the API takes DATES, and the two are
not the same question. 22:30 UTC in summer is already tomorrow in Belgium,
so a range derived from `since.date()` asks for the wrong FIRST day for the
last one or two hours of every single day - and the day it drops is the one
the controller is standing in. price_history.py made this mistake at the
cache layer and documents the same arithmetic; here it decides what is
fetched at all, so getting it wrong loses the hours rather than misfiling
them.
⚠️ `until` IS EXCLUSIVE, SO LOCAL MIDNIGHT BELONGS TO THE DAY BEFORE IT.
A 48 h horizon from local midnight ends at local midnight two days later,
and that instant is the first tick of a day the caller did not ask for.
Taking `brussels(until).date()` unconditionally would request a THIRD day
on every whole-hour horizon. This feed answers such a request without
complaint, so nothing would report it - which is the reason to get it right
here rather than to rely on the server's tolerance: the range is a claim
about what the caller asked for, and it should be true.
⚠️ AND AN EMPTY OR INVERTED WINDOW COLLAPSES TO ONE DAY RATHER THAN TO A
BACKWARDS RANGE. `start` after `end` is a request no feed has a defined
answer for; some return everything, some return an error, and neither is a
thing to hand a battery controller.
"""
first = brussels(since).date()
edge = brussels(until)
last = edge.date()
if (edge.hour, edge.minute, edge.second, edge.microsecond) == (0, 0, 0, 0):
last = last - timedelta(days=1)
if last < first:
last = first
return first, last
def dated_feed_url(base: str, **params) -> str:
"""`base` plus a query string. Refuses a `base` that already carries one.
⚠️ THE REFUSAL IS THE POINT, NOT THE FORMATTING. Appending `&start=...` to
a URL that may already hold its own query string is how a plugin produces a
request nobody intended - `?bzn=BE?bzn=BE&start=...`, or a duplicated
parameter whose winner is the server's choice rather than ours. So the base
is required to be parameterless and the parameters are added exactly once,
by `urlencode`, which is also what keeps a date out of the escaping
question. A future edit that folds the query back into the constant is a
refused fetch with a sentence, not a silently malformed URL.
⚠️ IT IS NOT REACHED BY AN OPERATOR'S `price_feed_url`. See
`GenericDayAhead.fetch()`: an override is used exactly as given.
"""
if "?" in base or "#" in base:
raise SourceUnavailable(
f"the feed base {base!r} already carries a query string or "
f"fragment, and this builder appends parameters. Adding a second "
f"'?' produces a URL the server parses differently from the one "
f"anybody intended; put the fixed parameters in the call instead")
return f"{base}?{urlencode(params)}" if params else base
# --------------------------------------------------------------------------- #
# Absence, borrowed wholesale from sim/riglock.py (TEST-04).
# --------------------------------------------------------------------------- #
@@ -684,6 +766,47 @@ def price_option(opts: dict, key: str):
return v
def provider_id(opts: dict) -> str:
"""The selected plugin id, normalised. `"off"` when none is selected.
⚠️ ONE SPELLING OF THIS EXPRESSION, AND TARIFF-10 IS WHY IT IS A FUNCTION.
`str(price_option(opts, "price_provider") or "off").strip()` was written out
twice - in `load_prices()` below and in `static_tariff.resolve_tariff()` -
and main.py's startup gate would have been a third. "Is a provider
configured" is now a DECISION the add-on takes (it is what starts the
background refresh), and a decision spelled out at three call sites is three
places to disagree about whether `" off "` or `None` counts.
⚠️ `validate.py` deliberately keeps its own reader: it validates the RAW
options blob, including one the Supervisor never saw, so it must not route
through `price_option()`'s defaults. Note what this does NOT do: it does not
decide
whether the id names a shipped plugin - `load_prices()` and validate.py's
rule 7 own that, and answering "unknown" here would turn a typo into
"no provider" instead of into a startup refusal.
"""
return str(price_option(opts, "price_provider") or "off").strip()
def provider_configured(opts: dict) -> bool:
"""Has the operator selected a price feed at all? (TARIFF-10)
THE DEFECT THIS PREVENTS: the background tariff refresh used to start only
when `arbitrage_enabled` was on, which ships `false` - so a site with a
provider configured fetched NOTHING, ever. `/data/price_history.json` was
never created, the *Price history* status row read `empty` forever, and
TARIFF-09's short-horizon warning could not fire because no fetch happened
to warn about. Measured over a 40 h two-instance soak: zero feed requests
and `present: false` across all 802 samples on the shipped default, against
a resolved plan on the same feed with arbitrage armed.
⚠️ THIS IS NOT A TRADING SWITCH AND MUST NEVER BECOME ONE. It answers
"fetch prices?"; `arbitrage.enabled()` answers "move energy on them?", and
§10.1's priority 3 is gated on the second alone. See
`main.tariff_refresh_wanted()` for the one place the two meet.
"""
return provider_id(opts) not in ("", "off")
def export_price_eur_kwh(opts: dict) -> float:
"""AC4 / §9.3: the injection price is a SEPARATE SCALAR, not a second series.
@@ -731,6 +854,24 @@ class PriceProvider(ABC):
`transport(url, timeout_s) -> bytes` is injected so the checks never
touch a network and `manual_csv` never needs one. `opts` is a
`PluginOptions`, so the markup and taxes are not reachable from here.
⚠️ `since`/`until` ARE THE RANGE THE CALLER WANTS, AND A PLUGIN THAT
IGNORES THEM GETS WHATEVER ITS FEED DEFAULTS TO (TARIFF-09). That is
not a small difference: `generic_dayahead` ignored both, the public
EPEX feed answered ONE Brussels day, and §8.3 then dropped every hour
past today from the trade set as provisional - so the optimiser never
saw tomorrow as firm, at any hour of any day, and nothing reported it.
This promise is NOT mechanically enforced, and the honest reason is
TARIFF-08's: three of the five shipped plugins point at endpoints
nobody has verified (TARIFF-07 is the open decision on withdrawing
them), so nobody knows how those feeds spell a date range. A rule
obliging them to pass one would be satisfied by an INVENTED parameter
name - the parser-graded-against-its-own-guess defect, moved from the
response to the request. What exists instead is a roster pin in
test_price.py: which plugins consume the range and which do not is
asserted as a literal, so a sixth supplier, or a change to one of the
three, forces the decision rather than inheriting the silence.
"""
@@ -979,9 +1120,166 @@ def _points_from_json(body, price_key, scale, provisional_default):
return out
# --------------------------------------------------------------------------- #
# TARIFF-08: the COLUMNAR document, which is what the public EPEX feed really
# publishes.
#
# ⚠️ THIS IS A SECOND SHAPE, READ BY A SECOND FUNCTION ON PURPOSE.
# `_points_from_json()` above reads a row-per-point document: `{"data": [{...},
# {...}]}`. `https://api.energy-charts.info/price?bzn=BE` does not publish that
# and never did - it publishes PARALLEL ARRAYS:
#
# {"unix_seconds": [1788991200, 1788992100, ...], <- UTC epoch seconds
# "price": [216.08, 209.90, ...], <- one per instant
# "unit": "EUR / MWh", "license_info": "...", "deprecated": false}
#
# The shipped parser was pointed at that URL and asked for `doc["data"]`, so
# every fetch ended `SourceUnavailable: the feed carried no 'data' list` ->
# §11.4 -> static profile, arbitrage off, and nothing in the status page said
# why. Verified live 2026-09-10 against both fixtures under `scenarios/`.
#
# ⚠️ AND IT IS NOT ONE FUNCTION THAT SNIFFS FOR BOTH. A parser that decides
# which document it got is a parser that can decide wrong, and a wrong-but-
# parseable response then becomes a price. Two shapes, two functions, and each
# plugin names the one its own feed publishes.
#
# The columnar shape was already known in this repository: DATA-02's
# `scenarios/fetch_epex_be.py` has read `zip(d["unix_seconds"], d["price"])`
# since 2026-08-31. Only the SHIPPED path never learned it, because every
# fixture in the suite was built from the invented row shape - the parser was
# graded against its own assumption. See test_price.py section 2b.
# --------------------------------------------------------------------------- #
COLUMNAR_TIME_KEY = "unix_seconds"
COLUMNAR_PRICE_KEY = "price"
COLUMNAR_UNIT_KEY = "unit"
def _columnar_unit(text) -> str:
"""`"EUR / MWh"` -> `"eur/mwh"`. Spacing and case are not information."""
return "".join(str(text).split()).lower()
def _points_from_columnar_json(body, expect_unit, scale,
provisional_default):
"""Parse a COLUMNAR price document: parallel `unix_seconds` / `price`.
Defects this refuses, each of which has a way of becoming a number rather
than an error:
* ⚠️ UNEQUAL ARRAYS ARE `SourceUnavailable`, NOT A SHORT SERIES. `zip()`
stops at the shorter array and says nothing, so a `price` list that
arrived truncated would produce a REAL price series with its tail
missing - and the tail of a Belgian day is the evening peak. A series
that is missing the expensive hours is worse than no series: §11.4's
no-data row switches arbitrage OFF, whereas a truncated series is
traded on. Nothing downstream can tell the two apart, so the refusal
has to happen here.
* ⚠️ A UNIT THAT IS NOT THE DECLARED ONE IS REFUSED. `scale` is the
caller's standing assumption about the unit; if the feed ever
publishes EUR/kWh under the same keys, every price silently comes out
a THOUSAND times too small. `_validate_raw()` cannot catch that -
`MAX_PLAUSIBLE_EUR_KWH` only bounds prices that are too LARGE.
DATA-02's fetcher refuses on the same field for the same reason.
* ⚠️ `unix_seconds` ARE UTC INSTANTS, SAID SO EXPLICITLY.
`datetime.fromtimestamp(t, timezone.utc)`. Without that second
argument CPython returns a NAIVE datetime in the HOST's zone, and a
naive datetime is a wall clock reading plus an assumption about whose
clock it was. `brussels()` and `_validate_raw()` both refuse one, so
the mistake surfaces as a refused fetch rather than as a mispriced
day - but they refuse ALL naive datetimes and can say nothing about
whether this one was the right instant. That part is decided here.
§9.3's rendering AT THE BRUSSELS OFFSET is `assemble()`'s, once, for
every plugin - `brussels(p.start)` there is the single line that does
it, and calling it here as well would be a second copy of a rule that
already has an owner, indistinguishable from a no-op under mutation.
A plugin's job is the right INSTANT; the framework's is the rendering.
A `null` price is DROPPED and logged, not fatal, and that is a different
judgement from the one above: an unequal pair of arrays means the pairing
of instant to price is unreliable, whereas a null means the source stated
it published no price for that one instant. A dropped instant leaves a
hole, and `_Series.at()` already answers a hole with `Absence` - §11.4 for
that hour. `scenarios/fetch_epex_be.py` dropped 0 nulls over the 12 months
to 2026-08-31, so this is a shape the source allows rather than one it uses.
"""
try:
text = body.decode("utf-8") if isinstance(body, (bytes, bytearray)) \
else str(body)
doc = json.loads(text)
except (ValueError, UnicodeDecodeError) as e:
raise SourceUnavailable(f"the feed did not parse as JSON: {e}")
if not isinstance(doc, dict):
raise SourceUnavailable(
f"the feed is a {type(doc).__name__}, not a columnar object with "
f"{COLUMNAR_TIME_KEY!r} and {COLUMNAR_PRICE_KEY!r} arrays")
times = doc.get(COLUMNAR_TIME_KEY)
prices = doc.get(COLUMNAR_PRICE_KEY)
for key, col in ((COLUMNAR_TIME_KEY, times), (COLUMNAR_PRICE_KEY, prices)):
if not isinstance(col, list):
raise SourceUnavailable(
f"the feed carried no {key!r} array; this provider reads the "
f"columnar shape (parallel {COLUMNAR_TIME_KEY}/"
f"{COLUMNAR_PRICE_KEY} arrays), and a row-per-point document "
f"is a different feed, not this one with a field missing")
if len(times) != len(prices):
raise SourceUnavailable(
f"{COLUMNAR_TIME_KEY} has {len(times)} entries and "
f"{COLUMNAR_PRICE_KEY} has {len(prices)}; refusing to zip them. "
f"The short array would truncate the series silently, and a price "
f"series missing its last hours is one missing the evening peak - "
f"traded on as if complete, where no data at all would have "
f"switched arbitrage off (§11.4)")
unit = _columnar_unit(doc.get(COLUMNAR_UNIT_KEY, ""))
if unit != _columnar_unit(expect_unit):
raise SourceUnavailable(
f"the feed reports {doc.get(COLUMNAR_UNIT_KEY)!r}, not "
f"{expect_unit!r}. This provider converts with a fixed factor of "
f"{scale}, so a changed unit is a factor of 1000 on every price "
f"with nothing downstream able to notice it")
if doc.get("deprecated"):
# Not a refusal: today's prices are still prices. But §11.4 makes a
# dead feed look exactly like a quiet one, so the warning that the
# source is going away has to be emitted while it still works.
_LOG.warning("the price feed reports deprecated=%r; it still answers, "
"but this endpoint is scheduled to stop",
doc.get("deprecated"))
out = []
dropped = []
for i, (t, v) in enumerate(zip(times, prices)):
if v is None:
dropped.append(i)
continue
if isinstance(t, bool) or not isinstance(t, (int, float)) or \
not math.isfinite(t):
raise SourceUnavailable(
f"{COLUMNAR_TIME_KEY}[{i}] is {t!r}, not epoch seconds")
if isinstance(v, bool) or not isinstance(v, (int, float)) or \
not math.isfinite(v):
raise SourceUnavailable(
f"{COLUMNAR_PRICE_KEY}[{i}] is {v!r}, which is not a price")
start = datetime.fromtimestamp(float(t), timezone.utc)
out.append(RawPoint(start=start, eur_per_kwh=float(v) * scale,
provisional=bool(provisional_default)))
if dropped:
_LOG.warning("the feed published no price for %d of %d instants "
"(index %s); those instants are left OUT of the series, "
"never zero-filled - a missing hour is not a free hour",
len(dropped), len(times),
", ".join(str(i) for i in dropped[:5])
+ (", ..." if len(dropped) > 5 else ""))
if not out:
raise SourceUnavailable(
f"the feed carried {len(times)} instants and a price for none of "
f"them")
return out
@register
class GenericDayAhead(PriceProvider):
"""Public EPEX BE day-ahead feed, in EUR/MWh.
"""Public EPEX BE day-ahead feed, COLUMNAR, in EUR/MWh.
SOURCES: energy
The framework ADDS: markup, taxes
@@ -990,6 +1288,19 @@ class GenericDayAhead(PriceProvider):
is the wholesale clearing price and nothing else, so both user-entered
components have to be added to it - and because this plugin cannot read
either number, it cannot be the one that adds them twice.
⚠️ IT READS THE COLUMNAR SHAPE, AND SAYS SO IN ITS OWN `fetch()`. Until
TARIFF-08 it called `_points_from_json()` and asked
`api.energy-charts.info` for a `"data"` list it has never published, so
every fetch this provider ever made ended in `SourceUnavailable` and §11.4
turned that into a silent fall back to the static profile. See
`_points_from_columnar_json()` for the shape and for why there are two
parsers rather than one that guesses.
⚠️ THE FEED IS QUARTER-HOURLY SINCE BELGIUM'S 15-MINUTE MTU (2025-10-01):
96 points a day at 900 s, not 24 at 3600 s. Nothing here assumes either -
the instants come from the document - and both granularities are replayed
from real captured bodies in test_price.py section 2b.
"""
id = "generic_dayahead"
@@ -1001,12 +1312,68 @@ class GenericDayAhead(PriceProvider):
sources=frozenset({Component.ENERGY}),
egress_hosts=("api.energy-charts.info", "web-api.tp.entsoe.eu"))
FEED_BASE = "https://api.energy-charts.info/price"
BZN = "BE"
async def fetch(self, opts, since, until, transport):
url = opts.get("price_feed_url") or \
"https://api.energy-charts.info/price?bzn=BE"
"""Request the range the CALLER asked for, on the Brussels calendar.
⚠️ THE ARGUMENTS ARE USED. Until TARIFF-09 this method took `since` and
`until` and asked for the bare `?bzn=BE`, which answers ONE Brussels
day - so a 48 h horizon was 15 firm hours and 33 carried at 09:00, and
1 firm and 47 carried at 23:00, with the whole trade set inside a
single calendar day at every hour of every day. See
`brussels_date_range()` for the measurement and for why the calendar
is Brussels rather than UTC.
⚠️ AN OVERRIDE IS THE EXACT URL AND THIS METHOD APPENDS NOTHING TO IT,
WHICH COSTS THE OVERRIDING SITE THE RANGE. That is a decision, not an
oversight, and the sentence below is why it is not a footnote:
* `price_feed_url` says "override the plugin's default feed", and an
override that is silently rewritten is not one. Merging parameters
into an operator's query string means parsing and re-serialising a
URL they wrote - which reorders, de-duplicates and re-escapes it,
and can hand the server a request nobody typed.
* `start`/`end` are ENERGY-CHARTS' spelling, and this provider
DECLARES A SECOND HOST: `web-api.tp.entsoe.eu`. Nothing here has
verified that host's day-ahead parameters and nothing here should
guess them - but they are not `start`/`end`, so an override to the
other DECLARED host is exactly the case where appending our
vocabulary adds two parameters that host ignores and yields the
short horizon anyway, with nothing saying so.
* and blind appending is refused outright: `dated_feed_url()` will
not add a query string to a base that has one.
So the cost is real and it is REPORTED, at fetch time, on the shipped
logger, naming the horizon consequence rather than the substitution.
§11.4 is what makes a price defect quiet; an operator who has taken the
short horizon has to be able to find out from the log that they did.
"""
override = str(opts.get("price_feed_url") or "").strip()
if override:
url = override
_LOG.warning(
"price_feed_url is set, so %s is requesting %s exactly as "
"given and is adding NO date range to it. The public feed "
"answers a single Brussels day to a request that names no "
"range, and every hour past that day is then carried forward "
"from yesterday's profile and flagged provisional - §8.3 "
"drops provisional hours from the trade set, so arbitrage "
"would never see tomorrow as firm. Clear price_feed_url to "
"let this plugin ask for the %s..%s it was given.",
self.id, url, *(d.isoformat()
for d in brussels_date_range(since, until)))
else:
first, last = brussels_date_range(since, until)
url = dated_feed_url(self.FEED_BASE, bzn=self.BZN,
start=first.isoformat(),
end=last.isoformat())
check_egress(url, self.capabilities())
body = await transport(url, float(opts.get("price_timeout_s")))
return _points_from_json(body, "price_eur_mwh", 0.001, False)
return _points_from_columnar_json(body, expect_unit="EUR / MWh",
scale=0.001,
provisional_default=False)
@register
@@ -1190,8 +1557,8 @@ async def load_prices(opts: dict, since: datetime, until: datetime,
site condition, it is a plugin trying to read the markup, and turning it
into "no price data" would hide the exact defect trap 1 exists to catch.
"""
pid = str(price_option(opts, "price_provider") or "off").strip()
if pid in ("", "off"):
pid = provider_id(opts)
if not provider_configured(opts):
return PriceRead.no_data(
"no price provider configured (price_provider is 'off'); "
"§11.4: static profile, plan provisional, no arbitrage")
+170 -27
View File
@@ -48,12 +48,24 @@ So the flag is not passed through anywhere in this file:
the object the provider produced is the object in the output series - so
there is no copy that could lose or gain a flag. `filled.points[i] is
fetched.points[j]` is a checkable property, and test_price_forecast.py
checks it.
checks it. EVERY published point in the window is passed through, not one
per hour: see the sub-hourly note below.
* `_audit()` RE-DERIVES the expected flag for every output point from the set
of hours the feed actually published, and RAISES if any point disagrees.
It is a post-condition, not a filter: it never fixes a flag, because a
module that quietly corrects its own provenance is one whose provenance
cannot be trusted.
⚠️ IT GRADES THE POINTS THAT ARE EMITTED AND NOTHING ABOUT THE ONES THAT
ARE NOT (FCST-05). `_audit()` is a provenance check, NOT a completeness
check: fed FCST-04's own decimation defect - four published points in an
hour, only the last emitted - it ACCEPTS, and it accepts an empty output
too. Measured, and the FCST-03 audit it replaced was equally blind, so
this is a limit rather than a regression. What actually guards against a
decimated series is the point COUNT and the pass-through IDENTITY checks
in `test_price_forecast.py` (section on the quarter-hourly feed) and
`test_price_history.py`; see `_audit()`'s own docstring for why the
completeness rule is not expressible from its two arguments.
* and test_price_forecast.py walks THIS FILE'S AST and fails if any
`PricePoint(...)` in it passes anything but a literal for `provisional`, so
the pass-through idiom cannot come back in a later edit and pass review by
@@ -90,6 +102,24 @@ which is the row the spec actually writes for "no price data". If NOTHING can
be produced, the result is `PriceRead.no_data(reason)` and not an empty series,
because an empty series is not constructible on purpose (price.py trap 3).
⚠️ A SUB-HOURLY FEED IS NOT DECIMATED TO ONE POINT AN HOUR, and the hourly walk
below is about WHICH HOURS NEED FILLING - never about what a published hour is
allowed to contain. `eneco_eddy` is the Belgian imbalance product and settles
per QUARTER, `_validate_raw()` accepts any strictly-increasing spacing, and a
published hour may therefore hold four points. So EVERY published point inside
the window is passed through, in order, and a carried point is added only for a
whole hour with NO published coverage at all.
The alternative was measured rather than reasoned about, because FCST-04's
first caller shipped it: keying the published set as `{_utc_hour(p.start): p
for p in r.points}` reads perfectly ordinary and keeps the LAST point of each
hour, dropping the other three. On a 96-point quarter-hourly feed - all firm,
fully published, nothing to carry - the series came out 15 points long, the
arbitrage spread collapsed from 0.143 to 0.014 EUR/kWh, and the cheapest
"hour" became a `:45` point whose `at()` window runs into the next hour. A feed
losing 84 % of its own prints is not a windowing effect and is not something a
provisional flag would ever have reported.
AC2: THE IRREVERSIBLE-ACTION GATE
=================================
@@ -156,6 +186,25 @@ def _utc_hour(instant: datetime) -> datetime:
return u.replace(minute=0, second=0, microsecond=0)
def _published_by_hour(read: PriceRead) -> dict:
"""UTC hour -> the provider's OWN points that start inside it, in order.
⚠️ A TUPLE PER HOUR, NOT A POINT, AND THAT IS THE WHOLE OF THE SUB-HOURLY
RULE. `{_utc_hour(p.start): p for p in r.points}` reads identically, is one
character shorter, and silently keeps one point per hour - the last one
written. The mapping answers two DIFFERENT questions here: "does this hour
need filling at all?" is `u in published`, and "what does the output carry
for it?" is all of `published[u]`. Collapsing them is how a quarter-hourly
feed loses three quarters of its prints.
"""
out: dict = {}
read.match(
series=lambda r: [out.setdefault(_utc_hour(p.start), []).append(p)
for p in r.points],
no_data=lambda _r: [])
return {u: tuple(ps) for u, ps in out.items()}
def _carried(start_utc: datetime, eur_per_kwh: float) -> PricePoint:
"""A point filled from yesterday's profile. §8.3: ALWAYS provisional.
@@ -172,37 +221,122 @@ def _carried(start_utc: datetime, eur_per_kwh: float) -> PricePoint:
def _audit(points, published) -> None:
"""Post-condition: every flag matches the point's TRUE provenance (AC1).
`published` maps a UTC hour to the point the provider produced for it.
`published` maps a UTC hour to the TUPLE of points the provider produced
for it (`_published_by_hour()`). An hour ABSENT from that mapping is an hour
the feed did not print, and a carried point is the only thing that may
stand in it.
The rule, in the two forms an output point can take:
* PUBLISHED - the point must BE one of `published[u]`, by identity, and
may be emitted only once. Its flag is whatever the feed said, never
forced to False: an imbalance print is published AND provisional (§9.2,
TARIFF-02), and a check that "corrected" it would be a bug wearing a
guarantee.
* CARRIED - the hour must not appear in `published` AT ALL, the flag must
be exactly True, and that hour may hold no second point.
⚠️ WHAT REPLACED "AT MOST ONE POINT PER HOUR", AND WHAT THAT DID AND DID
NOT CHANGE. This audit began as FCST-03's `seen` set: any repeated
`_utc_hour` raised, beside an identity rule `p is not src`. A bare hour
count cannot express a quarter-hourly feed, where four distinct published
points share one hour legitimately - so the count was replaced by the two
rules above, which subsume it.
⚠️ AND IT IS NOT STRICTER. FCST-04's version of this paragraph claimed two
strengthenings and BOTH WERE FALSE; FCST-05 measured it. A differential
over the whole domain this audit can express - every published mask over 3
hours crossed with every output series of length 0-3 drawn from
{published, published-provisional, carried, carried-unflagged,
copy-of-published} - found ZERO divergence in EITHER direction:
OLD raised, NEW accepted (WEAKER): 0
OLD accepted, NEW raised (STRONGER): 0
The two specific claims, one at a time:
* "so does the same published point emitted twice" - the OLD `seen`
count already raised on that, because emitting one point twice is one
HOUR twice. Preserved by the identity/`emitted` rules, not added.
* "a carried point may no longer share an hour with a published one,
which the count never checked at all" - true of the COUNT, and the old
IDENTITY rule raised on it anyway: a carried point in a published hour
took the `else` branch and failed `p is not src`. The message was about
a copy rather than about a carry, which is the only thing that moved.
The one real behavioural change is the intended RELAXATION - several
distinct published points in one hour, which the old audit could not
express at all and refused. That relaxation is why this was rewritten, and
it stays. A right decision with a fabricated reason is still a defect here.
⚠️ AND THE REAL LIMIT, WHICH IS NOT A REGRESSION AND IS NOT SMALL: THIS
AUDIT DOES NOT CHECK COMPLETENESS. It grades every point that IS emitted
and says nothing about the ones that are not. Fed the H1 decimation
directly - four published points in an hour, output emitting only the last
- it ACCEPTS; so does two of four, and so does an EMPTY output. So `_audit`
is NOT what stops H1 recurring. What stops it is the point COUNT and the
IDENTITY checks in `test_price_history.py` and `test_price_forecast.py`,
which compare the emitted series against the feed's own prints.
It is not made complete here, and the reason is structural rather than
effort: `published` is `_published_by_hour(read)` over the WHOLE read,
unwindowed, while `fill_from_yesterday()` emits only [since, until). A
published point outside the window is legitimately absent, so "every
published point appears" is not a true post-condition of these two
arguments - it would need the window passed in, which is a different
signature and a different check.
⚠️ AND THAT IS MEASURED, NOT ASSUMED. FCST-05 built the rule - raise unless
every point of every `published[u]` is emitted, +162 B - and ran it. It
takes the 1 h, 12 h and 48 h HORIZON checks in test_price_forecast.py RED,
on the shipped fill and a healthy feed: at an off-the-hour instant the
feed's already-elapsed prints are outside [since, until) and are correctly
dropped, and the rule calls that a provenance failure. (The differential
beside those checks scores the same rule as 49 STRONGER cases, so it is a
real change of behaviour and not a no-op.) The rule as stated is therefore
not merely unimplemented, it is WRONG - and stating a false post-condition
is MUTATION.md habit 16's defect: an instrument that asserts a completeness
it does not have is worse than no instrument.
⚠️ RAISES, NEVER REPAIRS. A module that silently corrects its own
provenance is one whose provenance means nothing; the only safe response to
"this point's flag disagrees with where it came from" is to refuse to emit
the series at all.
"""
seen = set()
carried_hours = set()
emitted = set()
for p in points:
u = _utc_hour(p.start)
if u in seen:
raise ValueError(f"the filled series repeats {u.isoformat()}; a "
f"duplicated hour is how a DST day gets mispriced")
seen.add(u)
src = published.get(u)
prov = Provenance.CARRIED if src is None else Provenance.PUBLISHED
if src is None:
if u in carried_hours:
raise ValueError(
f"the filled series repeats the carried hour "
f"{u.isoformat()}; a duplicated hour is how a DST day gets "
f"mispriced")
carried_hours.add(u)
if p.provisional is not True:
raise ValueError(
f"{u.isoformat()} is {prov.value} and is flagged "
f"provisional={p.provisional!r}. §8.3: an hour filled from "
f"yesterday's profile is provisional whether or not the "
f"number turns out to be right - the flag is a claim about "
f"PROVENANCE, not about accuracy.")
f"{u.isoformat()} is {Provenance.CARRIED.value} and is "
f"flagged provisional={p.provisional!r}. §8.3: an hour "
f"filled from yesterday's profile is provisional whether "
f"or not the number turns out to be right - the flag is a "
f"claim about PROVENANCE, not about accuracy.")
else:
if p is not src:
if not any(p is q for q in src):
raise ValueError(
f"{u.isoformat()} is {prov.value} but the series carries a "
f"COPY of that point rather than the point itself. A copy "
f"is where the provisional flag gets lost; published points "
f"pass through by identity.")
f"{u.isoformat()} is {Provenance.PUBLISHED.value} but the "
f"series carries a COPY of that point rather than the "
f"point itself. A copy is where the provisional flag gets "
f"lost; published points pass through by identity.")
if id(p) in emitted:
raise ValueError(
f"the filled series emits the published point at "
f"{p.start.isoformat()} TWICE. The provider's own points "
f"are strictly increasing (`_validate_raw`), so a repeat "
f"is never a second print - it is a duplicated hour, which "
f"is how a DST day gets mispriced")
emitted.add(id(p))
def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead,
@@ -217,11 +351,21 @@ def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead,
Returns a `PriceRead`:
* a series, whose points are the provider's own objects for the hours it
published and `_carried()` points for the rest, ordered by instant;
* a series, carrying the provider's own objects - ALL of them, which on
a quarter-hourly feed is four inside one hour - for the hours it
published, and one `_carried()` point for each remaining whole hour,
ordered by instant;
* `no_data` when neither source can produce a single hour - never an
empty series, which is not constructible (price.py trap 3).
⚠️ THE WINDOW IS WHOLE HOURS AT BOTH ENDS. `since` is floored to the hour
that CONTAINS it, because the current hour is still a tradeable hour; and
so is `until`, which is the half FCST-04's first draft left out. With
`since` floored and `until` compared raw, a 48 h horizon asked for at 08:17
emitted 49 hours - the hour of `until` was still `< until` - and the count
was one too many for every `now` that is not exactly on the hour, which is
every `now` a running controller has.
⚠️ IDEMPOTENT AND RE-RUNNABLE, WHICH IS THE §5.4 LATE-PUBLICATION CONTRACT.
Day-ahead publication at 14-16 h CET can be late or fail, so the fetcher
retries and this runs again against the fresher read. Published hours always
@@ -234,9 +378,7 @@ def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead,
raise ValueError(f"{name} must be tz-aware; a naive instant is "
f"ambiguous for one hour every October")
published = read.match(
series=lambda r: {_utc_hour(p.start): p for p in r.points},
no_data=lambda _r: {})
published = _published_by_hour(read)
# Yesterday's profile, keyed by BRUSSELS WALL-CLOCK HOUR. First point wins
# when yesterday itself was a 25-hour day and printed local hour 2 twice.
@@ -248,11 +390,12 @@ def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead,
out = []
u = _utc_hour(since)
end = until.astimezone(timezone.utc)
end = _utc_hour(until)
while u < end:
if u in published:
out.append(published[u]) # BY IDENTITY. No copy.
else:
got = published.get(u)
if got:
out.extend(got) # BY IDENTITY, and ALL of them. No copy,
else: # and no decimation to one an hour.
src = by_local_hour.get(brussels(u).hour)
if src is not None:
out.append(_carried(u, src.eur_per_kwh))
+864
View File
@@ -0,0 +1,864 @@
"""FCST-04 - the day the carry-forward carries, kept somewhere a restart cannot reach.
WHAT WAS MISSING, AND WHY IT WAS INVISIBLE. FCST-03 shipped
`price_forecast.fill_from_yesterday()` complete and checked, and shipped the
consumers that were written to expect it - `TariffPlan.firm_points()`,
`cheap_now()`'s refusal of a provisional hour, and `arbitrage()`'s refusal 3,
which drops provisional hours from the trade set instead of refusing the whole
series. Every one of those was live. The FILL ITSELF HAD NO CALLER: nothing in
`app/` referenced `fill_from_yesterday`, so no series ever contained a carried
point and every one of those consumers was exercising its all-firm branch.
The defect that produced is not an error anywhere. `load_prices()` returns
whatever the feed published and nothing widens it to the requested horizon, so
on a dynamic site between midnight and the 14-16 h CET day-ahead publication
the series covers the rest of TODAY and nothing more - typically 10-24 of the
48 hours `price_horizon_h` asked for. `TariffPlan.arbitrage()` then computes
`max - min` over that truncated window and reports a spread. The number is
real, the window is half the one the operator configured, and NOTHING SAYS SO:
tomorrow evening's peak - the hour a morning charge would be sold into - is
simply not in the series, so the spread it would have created cannot clear a
threshold it was never compared against. A missing hour is not a wrong price;
it is an hour the optimiser never learns exists.
So this module is the half FCST-03 left out, and it is deliberately the SMALL
half: the fill's arithmetic, its DST rule and its provenance audit all stay in
`price_forecast.py`, untouched. What lives here is the one thing that
arithmetic needs and could not have: A YESTERDAY THAT SURVIVES A RESTART.
WHY PERSISTENCE, AND NOT A VARIABLE ON THE REFRESH TASK
======================================================
An in-memory "the last full day I saw" costs nothing and is wrong exactly when
it matters. The window the fill exists for is midnight -> publication; an
add-on restarted at 08:00 - a config change, an HA update, a power cut - would
come up with no yesterday at all and spend that whole window on the truncated
series, which is the pre-FCST-03 behaviour arriving silently on the mornings
after an operator touched something. The fill's own value is a function of
surviving a restart, so the day it carries has to.
⚠️ AND NOT IN `/data/state.json`. That file holds the maintenance history and
SAFETY-02's sign latch, and `Store.set()` rewrites the whole of it on every
write. The tariff refresh runs every `TARIFF_REFRESH_S` (1800 s), so folding a
price cache into it would put 48 extra whole-file rewrites a day through the
file whose corruption reads as "no maintenance ever ran" and "never tripped".
The cache is worth having; it is not worth putting anywhere near that. Separate
file, separate blast radius.
⚠️ AND THE DIRECTORY IS NOT CREATED, which is `tariff.PeakStore`'s rule and its
reason applies unchanged: `/data` is mounted by the Supervisor and always
exists on a real install, so `makedirs` here could only ever succeed where the
durable volume is MISSING - manufacturing a cache on the container's ephemeral
layer that writes fine, reads fine, and is thrown away on the next restart.
That is a yesterday that silently forgets, which is worse than one that says it
cannot answer.
⚠️ AND THAT RULE IS CARRIED OUT IN `_remember()`, NOT ONLY IN `_save()`. FCST-04
first shipped it half-done: `_save()` short-circuited on `not self.ok` while
`_remember()` filled `self._days` anyway, so a store with no durable volume
answered `words` = "unavailable" and `yesterday()` = A SERIES - an in-memory
yesterday that works all day and is gone on the next restart, which is exactly
the silently-forgetting cache this rule exists to refuse. A store that cannot
persist now records nothing, and both of its answers say the same thing.
⚠️ AND "CANNOT PERSIST" IS ONE QUESTION, ASKED ONCE, ABOUT THE OUTCOME -
FCST-06. The paragraph above enforced that rule against `not self.ok`, which is
one SYMPTOM of not persisting: the durable volume is missing. There is a second,
and FCST-06 measured it. With `ok=True` and the directory present, `_save()` can
still fail - a read-only remount (`[Errno 30] Read-only file system`), a full
volume (`[Errno 28] No space left on device`) - and it logs, records
`last_error` and returns. `_remember()` had already populated `self._days`, so
`words` read `'2026-01-14 (24 points over 24 h); 2026-01-15 (24 points over
24 h) [could not write (...)]'` while the file on disk held `{}`, and
`yesterday()` answered 24 REAL POINTS for the rest of the process's life. Gone
on the next restart, and nothing said it would be - the same silently-forgetting
cache, reached by the condition the rule did not name.
⚠️ SO IT IS NOT TWO RULES, AND IT IS DELIBERATELY NOT TWO TESTS. "Can this
store persist?" CANNOT BE ANSWERED BEFORE THE ATTEMPT - a directory that existed
and was writable a moment ago is not a write that completed - so it is answered
AFTER it, once, by the only thing holding the evidence: `_save()` returns
whether the file now holds what memory holds. A missing volume is one of its
False cases and an `OSError` from the write is another, and `_remember()` KEEPS
ONLY WHAT THAT ONE ANSWER SAYS WAS KEPT. There is no `not self.ok` test in
`_remember()` any more, because there is nothing left for a second test to add -
and a third symptom nobody has thought of is how this defect arrived both times.
⚠️ AND WHAT IS KEPT IS THE LAST STATE THAT IS ON DISK, NOT AN EMPTY ONE. The
MERGE is undone, not the store. A site whose volume fills at 14:00 goes on
answering the yesterday it wrote at 13:30 - that record survives a restart, so
answering it is not a promise the store cannot keep. What it stops doing is
adding to it. The invariant the rule was always describing, stated once:
`self._days` NEVER HOLDS A POINT THE FILE DOES NOT. (Not "mirrors the file" -
`_usable_days()` drops records the file still holds. The only direction that can
invent a yesterday is the one named.)
THE FOUR RULES THIS FILE IS SHAPED BY
=====================================
1. IT RECORDS WHAT THE PROVIDER PUBLISHED, NEVER WHAT THE FILL PRODUCED.
Feeding a filled series back in is the defect that makes the whole mechanism
rot: day 2 carries day 1's guesses, day 3 carries day 2's carry of them, and
after a week of a dead feed the "yesterday" being carried forward is a
fossil of the last real publication with nothing anywhere reporting its age.
Every point still carries `provisional=True`, so the flag never lies - what
decays is the NUMBER, silently, behind a flag that is telling the truth.
⚠️ NO VALUE-LEVEL CHECK CAN CATCH THAT, and saying so is more useful than a
guarantee that would not hold. A carried point and a published-provisional
point (`eneco_eddy`'s current quarter, §9.2) are the same three fields with
the same flag; the difference is provenance, and provenance is not in the
value. So the mechanism is ORDERING, not validation: `carry()` is the only
public entry point, it remembers the argument and returns the fill, and the
filled series exists only as a return value that `carry()` itself never
sees again. `_remember()` is private, is called from exactly one place, and
`test_price_history.py` PARSES every shipped module under `app/`, `tools/`,
`scenarios/` and `estop/` and fails if any of them uses the NAME - an
attribute, a bare name, a def, or the string constant a `getattr` needs. By
name rather than by substring, so a comment about the rule does not fail it
and a call from `tools/` cannot pass unseen.
⚠️ AND THE ORDER IS GRADED BY DRIVING IT, NOT BY READING THE SOURCE. The
first version of that check compared positions in `ast.walk`, which is
breadth-first: `self._remember(...)` is a statement and `self.yesterday(...)`
is an ARGUMENT of the fill, one level deeper, so `_remember` came out first
whichever statement was written first - shipped and reversed scored the
same. What grades it now is the behaviour: a feed carrying yesterday
alongside today, into an EMPTY store, must fill the whole horizon, and
reversed it fills nothing.
2. "YESTERDAY" MEANS YESTERDAY, NOT "THE NEWEST DAY I HAVE".
`newest()` is the natural accessor and it is the one that decays: an add-on
down from Friday to Monday would answer Friday's prices to a fill that flags
them as one day old. The lookup is BY DATE - the Brussels day before the
Brussels day of the instant asked about - and a record for any other date is
NO_RECORD with a reason naming both dates. A three-day-old profile is not a
worse yesterday; it is not a yesterday.
3. A STORE THAT CANNOT ANSWER SAYS SO, IN THE TYPE.
`yesterday()` returns a `PriceRead`: a `series` when a full record exists,
`no_data(reason)` for every other outcome - no file, no record for that
date, an unreadable file, a record that parses to nothing. There is no
empty-series answer and no `.get(..., [])`-shaped accessor, for
`price.py`'s reason: an empty series is how "the cache is broken" gets to
wear the shape of "yesterday was free". `fill_from_yesterday()` takes a
`PriceRead` and handles `no_data` on its own, so the degraded case needs no
branch at the call site - it fills nothing and the horizon stays short,
which is exactly the behaviour that shipped before this ticket.
4. HOURS ACCUMULATE WITHIN A DAY; THEY DO NOT REPLACE IT.
A feed that serves only the REMAINING hours of the day (a legitimate shape -
nothing in the plugin contract promises a whole day) would, under
replace-the-record, overwrite a complete 24-hour profile with the two points
left at 22:00 and hand the next morning a yesterday with two hours in it. So
a record is MERGED by instant: a later read updates the hours it carries and
leaves the rest, and a revised price for an hour already held wins because
it is the newer statement about the same hour.
⚠️ KEYED BY INSTANT, NOT BY LOCAL HOUR, AND THAT IS THE DST DECISION. Storing
`hour -> price` would collapse the two 02:00s of a 25-hour October day into
one, inventing a second and different collapsing rule beside the one
`fill_from_yesterday()` already documents ("the first of its two 02:00 points
is the one used, deterministically"). Storing the instants keeps both, in
order, and lets the fill's own rule be the only one in the system.
WHAT THIS DOES NOT DO, ON PURPOSE
=================================
It does not extend the fill to a feed that published NOTHING. `resolve_tariff()`
applies `carry()` only on a `series` read; a `no_data` read still falls to
`_degrade()` and §11.4's row - "No price data -> fall back to the static
profile, mark the plan provisional, no arbitrage" - which is the fallback the
operator configured `static_profile` FOR. Preferring yesterday's feed over it
would change a documented degradation without being asked, and it would buy
nothing measurable: a fully-carried series is 100 % provisional, so
`arbitrage()` finds fewer than two firm hours and refuses, and `cheap_now()` is
False, exactly as on the §11.4 fallback. The two differ only in which numbers
the status page shows. `fill_from_yesterday()` handles a `no_data` read anyway,
and that branch is therefore not reached from production - it stays checked,
and turning it on is a product decision, not a wiring one. See
DECISIONS/FCST-04-the-day-that-is-carried.md.
"""
import copy
import json
import logging
import os
import tempfile
from datetime import datetime, timedelta, timezone
from .price import PricePoint, PriceProduct, PriceRead, brussels
from .price_forecast import fill_from_yesterday
_LOG = logging.getLogger("goodwe.price.history")
# How many Brussels days of published prices the SORT-ORDER cap keeps. The fill
# needs exactly one - yesterday - and rule 2 refuses anything older, so this is
# not a horizon: it is the number of days a record may sit unused before
# pruning removes it, and it is small because a bigger cache would only hold
# days no lookup can ask for. Three covers today, yesterday, and the day a
# fetch spanning midnight writes ahead of itself.
#
# ⚠️ IT IS A CAP, NOT THE RETENTION RULE, AND THREE HAS NO SLACK. The steady
# state is already {yesterday, today, tomorrow}; a feed that publishes to D+2,
# or ONE stray point in the wrong year - which sorts LAST and is therefore
# immortal under a newest-N cap - pushes yesterday out of a three-slot store
# and keeps it out, permanently and silently. So `_prune()` PINS the two days a
# lookup can actually ask for on top of the cap; see its docstring. The store
# is then bounded at KEEP_DAYS + 2 records rather than at KEEP_DAYS.
KEEP_DAYS = 3
SCHEMA = 1
def _bdate(instant: datetime) -> str:
"""The Brussels calendar date of an instant, as `YYYY-MM-DD`.
⚠️ BRUSSELS, NOT UTC. 23:30 UTC in July is already tomorrow in Belgium, and
a cache keyed on the UTC date would file the last 90-120 minutes of every
day under the wrong one - so a lookup on the following morning would find a
"yesterday" missing its own evening hours, which are the expensive ones.
"""
return brussels(instant).strftime("%Y-%m-%d")
def _yesterday_of(instant: datetime) -> str:
"""The Brussels date of the day BEFORE `instant`'s Brussels day.
⚠️ THE ARITHMETIC IS ON THE LOCAL DATE, NOT ON THE INSTANT, AND THAT IS A
DST DEFECT THIS FILE SHIPPED. `brussels()` renders at the FIXED offset in
force at that instant, so `brussels(stamp) - timedelta(days=1)` subtracts 24
ABSOLUTE hours from a wall clock - and the Brussels day before a
spring-forward is 23 hours long. Measured at 2026-03-30T00:30+02:00, the
first hour of the morning after the March transition: it returned
2026-03-28, so the fill carried a TWO-DAY-OLD profile flagged as yesterday's
(or, with only the last three days held, answered `no_data` naming a date
that is not yesterday).
A calendar day minus one day is `date` arithmetic, and a `date` has no
offset to lose. The 25-hour October day is right for the same reason.
"""
return (brussels(instant).date() - timedelta(days=1)).strftime("%Y-%m-%d")
def _point_of(row):
"""One stored row -> a `PricePoint`, or `None` if the row is not one.
THE ONE VALIDATOR, AND THE ASYMMETRY IT CLOSES (FCST-05's MEDIUM). This
file used to hold TWO standards for what a stored point is. The READ path
(`_parse_points()`) was properly defensive - every field checked, a
malformed row dropped and reported. The WRITE path was
`{q["start"]: q for q in raw if isinstance(q, dict) and "start" in q}`
followed by `sorted(..., key=lambda q: q["start"])`, which trusts
`q["start"]` to be a hashable, sortable string it never checked. Measured,
with the malformed row under the day the feed is currently writing to:
start is an int -> TypeError: '<' not supported between instances
of 'str' and 'int'
start is a dict -> TypeError: cannot use 'dict' as a dict key
start is null -> TypeError: '<' not supported between 'str' and
'NoneType'
start is a list -> TypeError: cannot use 'list' as a dict key
⚠️ AND THE BLAST RADIUS IS PERMANENT, WHICH IS WHY IT IS NOT MERELY UNTIDY.
`_remember()` is reached through `carry()` -> `resolve_tariff()` ->
`run_tariff_refresh`, which catches `Exception` - so the add-on stays UP
and EVERY refresh raises forever. The plan retires after
`TARIFF_MAX_AGE_S`, and arbitrage and the cheap window are off from then
on. There is no self-heal, because `_remember()` raises before it reaches
`_save()`. `_usable_days()` only required `rec["points"]` to be a LIST; the
elements were unchecked on the write path, and the existing fixture put its
malformed row under a day the feed was NOT writing to, so `_remember()`
never touched it.
So both paths call this, and neither of them has a standard of its own.
Returning `None` rather than raising is the contract: a caller decides
whether a bad row is dropped (both of them do) and how it is reported.
"""
if not isinstance(row, dict):
return None
try:
start = datetime.fromisoformat(row["start"])
value = float(row["eur_per_kwh"])
flag = row["provisional"]
except (KeyError, TypeError, ValueError):
return None
if start.tzinfo is None or start.utcoffset() is None \
or not isinstance(flag, bool):
return None
return PricePoint(start=start, eur_per_kwh=value, provisional=flag)
def _held(rec) -> tuple:
"""`(points, whole hours they cover)` for a day record. `(0, 0)` otherwise.
⚠️ POINTS AND HOURS ARE TWO NUMBERS, AND FCST-04's STATUS ROW PRINTED THE
FIRST UNDER THE SECOND'S NAME. The row was `f"{d} ({_n_points(rec)} h)"`,
which counts POINTS: on the quarter-hourly feed this add-on already
supports it read `'2026-01-13 (96 h)'` - 96 "h" for a 24-hour day. That is
the points/hours conflation H1 was raised for, on the row H1's own ticket
shipped. Both numbers are reported now, so an operator can see a day that
is 24 points over 24 h and a day that is 96 points over 24 h and tell them
apart, and neither number is wearing the other's unit.
⚠️ TOLERANT, AND IT IS THE SECOND LINE OF DEFENCE, NOT THE FIRST. `_load()`
drops a malformed day record and says so, so this should never see one. It
is written this way because `words` is read from `Controller.__init__` and
from every status render: a status LINE that raises takes the add-on down,
and "the cache is broken" must never be able to mean "the controller does
not start". That is exactly what FCST-04 shipped - `{"days": {"2026-01-14":
"oops"}}` gave `ok=True`, `last_error=None`, and an `AttributeError` out of
`words`, `yesterday()` and `resolve_tariff()` alike, with no self-heal
because `_remember()` raised before it could rewrite the file.
⚠️ AND IT COUNTS ONLY ROWS THAT ARE POINTS. A row `_point_of()` refuses is
not held prices, and a row count that included it would report a store as
healthier than it is - which is the second half of FCST-05's MEDIUM: the
status row said `'2026-01-15 (1 h)'` about a record whose single row was
killing every tariff refresh. What is dropped is named in `last_error`, and
`words` prints that beside the counts.
"""
if not isinstance(rec, dict):
return (0, 0)
raw = rec.get("points")
if not isinstance(raw, (list, tuple)):
return (0, 0)
n, hours = 0, set()
for row in raw:
pt = _point_of(row)
if pt is None:
continue
n += 1
hours.add(pt.start.astimezone(timezone.utc)
.replace(minute=0, second=0, microsecond=0))
return (n, len(hours))
def _row_of(day: str, points: int, hours: int) -> str:
"""One day's cell of the status row: what is held, in BOTH of its units.
⚠️ THE UNIT IS SPELT OUT ON EACH NUMBER. FCST-04's cell was
`f"{day} ({points} h)"` - the point count wearing the hour's name, which on
a quarter-hourly feed reads `'2026-01-13 (96 h)'` for a 24-hour day. What
an operator needs from this row is whether tomorrow morning's fill has a
whole day to carry, and that is the HOUR figure; the point count says how
finely it is priced. Neither answers for the other.
"""
return (f"{day} ({points} point{'' if points == 1 else 's'} "
f"over {hours} h)")
def _bad_rows(rec) -> int:
"""How many rows of a day record are not points. Total; never raises."""
if not isinstance(rec, dict):
return 0
raw = rec.get("points")
if not isinstance(raw, (list, tuple)):
return 0
return sum(1 for row in raw if _point_of(row) is None)
class PriceHistory:
"""The published price series of the last few Brussels days, in /data.
⚠️ `ok` IS ABOUT THE FILE, NOT ABOUT THE DATA. A healthy store with no
record for yesterday is `ok` and answers `no_data` - a normal state on a
site's first day, and a different thing from a store that could not be
opened. Both are visible: `ok`/`last_error` describe the file, and
`yesterday()`'s reason describes the answer.
⚠️ AND `ok` IS NOT "CAN PERSIST" (FCST-06). It says the directory was
there when the store opened, which a full volume and a read-only remount
both leave true. The one thing that answers "can this store persist" is
`_save()`'s return value, and `self._days` NEVER HOLDS A POINT THE FILE DOES
NOT, because `_remember()` undoes a merge that answer refused. That is what
makes `words` and `yesterday()` describe the same store in EVERY degraded
state rather than only in the ones somebody enumerated.
"""
def __init__(self, path: str = "/data/price_history.json"):
self.path = path
self.ok = False
self.last_error: str | None = None
self._days: dict = {}
self._open()
# -- the file ------------------------------------------------------------
def _open(self) -> None:
parent = os.path.dirname(self.path)
# See the module note: NOT created. A cache on the ephemeral layer is
# one that forgets without saying so.
if parent and not os.path.isdir(parent):
self.last_error = f"no durable volume at {parent}"
# INFO, not ERROR: on a dev host or in a test container there is no
# /data and nothing is wrong.
_LOG.info("no price history: %s - §8.3 carry-forward has nothing "
"to carry and the horizon stays as short as the feed",
self.last_error)
return
self.ok = True
self._load()
def _load(self) -> None:
try:
with open(self.path, "r", encoding="utf-8") as fh:
blob = json.load(fh)
except FileNotFoundError:
_LOG.info("no price history yet at %s - the first refresh writes "
"one, and the fill has nothing to carry until the day "
"after that", self.path)
return
except (ValueError, OSError) as err:
# A corrupt cache must not stop the controller, and must not be
# silently re-created either: the record is discarded, the reason
# is kept, and `yesterday()` reports it rather than answering
# "no record", which would read as a normal first day.
#
# ⚠️ `ValueError`, NOT `json.JSONDecodeError`, AND THAT IS
# FCST-05's HIGH. Reading this file has two failure families: the
# BYTES (`OSError`) and the CONTENT - and the content family has at
# least two members, because `open(..., encoding="utf-8")` decodes
# before `json` ever sees a character. `UnicodeDecodeError` is
# `ValueError -> UnicodeError -> UnicodeDecodeError`; it is NOT a
# `json.JSONDecodeError` and it is NOT an `OSError`
# (`issubclass(UnicodeDecodeError, OSError)` is False). So ONE
# non-UTF-8 byte in /data/price_history.json escaped `_load()` ->
# `_open()` -> `PriceHistory.__init__`, which is unguarded at
# main.py's `Controller.__init__`, AND THE ADD-ON DID NOT START.
# Measured: `UnicodeDecodeError: 'utf-8' codec can't decode byte
# 0xff in position 39`. Valid-UTF-8-invalid-JSON, a UTF-8 BOM and a
# non-dict day record all degraded correctly - it was the one byte
# class left uncovered, which is the same class as the defect the
# `_usable_days()` rework was raised for.
#
# The tuple names the FAMILY rather than the two members known
# today, because "the file's content is not readable as JSON" is
# exactly what this handler says, and a third `ValueError` subclass
# nobody has thought of is how this defect arrived the first time.
self.last_error = f"unreadable ({err})"
_LOG.warning("price history at %s is %s - starting from empty. "
"§8.3 carry-forward has nothing to carry until the "
"next refresh has written a full day",
self.path, self.last_error)
return
found = blob.get("version") if isinstance(blob, dict) else None
# ⚠️ THE TYPE IS PART OF THE VERSION. `1.0 == 1` in Python, so a float
# written by some other tool would pass a bare `!=` and be read as
# schema 1 - a version check that cannot tell 1 from 1.0 is one that
# would not notice a writer this build has never met.
if not isinstance(blob, dict) or isinstance(found, bool) \
or not isinstance(found, int) or found != SCHEMA:
self.last_error = f"schema {found!r}, not {SCHEMA}"
_LOG.warning("price history at %s has %s - discarding it rather "
"than guessing what its points mean",
self.path, self.last_error)
return
days = blob.get("days")
if not isinstance(days, dict):
self.last_error = "no `days` object"
_LOG.warning("price history at %s has %s", self.path, self.last_error)
return
self._days = self._usable_days(days)
def _usable_days(self, days: dict) -> dict:
"""The day records that are shaped like day records. The rest are dropped.
⚠️ THE ASYMMETRY THAT PRODUCED THE DEFECT: `_parse_points()` below is
carefully defensive about every field of a POINT, and `_load()` was not
defensive about the record holding them at all. `{"version": 1, "days":
{"2026-01-14": "oops"}}` therefore loaded as `ok=True`,
`last_error=None` - and then `rec.get(...)` raised `AttributeError` out
of `words`, `yesterday()` and `resolve_tariff()`. `PriceHistory(...)`
and `.words` are both unguarded in `Controller.__init__`, so THE ADD-ON
DID NOT START; and it never self-healed, because `_remember()` raised on
the same record before it could reach `_save()`.
A broken cache is a short horizon. It is not a controller that will not
regulate a battery, and §6's rule - absent machinery must not disable
the loop - does not stop applying because the machinery is a JSON file.
Dropped, not repaired: an unusable record has nothing in it to keep, and
the next `_remember()` writes a fresh one under that date.
⚠️ AND THE ROWS INSIDE A KEPT RECORD ARE COUNTED, NOT ONLY THE RECORD
(FCST-05). This checked that `rec["points"]` is a LIST and stopped
there, so a record holding one row with a `start` of `7` loaded as
healthy: `last_error=None`, and `words` said `'2026-01-15 (1 h)'` about
a store whose next `_remember()` would raise on that row and go on
raising forever. The rows are NOT dropped here - `_parse_points()` is
the read path's one filter and stays so - but they are COUNTED, so the
store stops reporting health it does not have from the moment it is
opened. `_remember()` drops them on the next refresh and `_save()`
rewrites the file, after which a restart clears the note.
"""
kept, dropped, malformed = {}, [], {}
for key, rec in days.items():
if not isinstance(key, str) or not isinstance(rec, dict) \
or not isinstance(rec.get("points"), list):
dropped.append(str(key))
continue
bad = _bad_rows(rec)
if bad:
malformed[key] = bad
kept[key] = rec
notes = []
if dropped:
notes.append(f"{len(dropped)} unusable day record(s) dropped "
f"({', '.join(sorted(dropped))})")
if malformed:
notes.append(
f"{sum(malformed.values())} malformed point row(s) in "
+ ", ".join(f"{d} ({n})" for d, n in sorted(malformed.items())))
if notes:
# Recorded, not silent: `yesterday()` and `words` both report
# `last_error`, and a cache that quietly discards half a file reads
# to an operator exactly like a normal first day.
self.last_error = "; ".join(notes)
_LOG.warning("price history at %s: %s - the controller starts "
"anyway and the horizon is as short as the feed makes "
"it", self.path, self.last_error)
return kept
def _save(self) -> bool:
"""Write `self._days` out. True only if the file now holds it.
⚠️ THE RETURN VALUE IS THE "CAN THIS STORE PERSIST?" PREDICATE, AND
IT IS THE ONLY ONE (FCST-06). Both ways of not persisting end here and
are the same answer: no durable volume returns False without touching
the disk, an `OSError` out of the write returns False after recording
why. `_remember()` acts on that one answer, so the second place that
used to test the first symptom on its own - and missed the second - has
nothing left to test. See the module docstring's fifth warning.
⚠️ AND IT REPORTS, IT DOES NOT RAISE. §6 and this module's opening: a
broken cache must not be a broken controller. An add-on that will not
start because /data filled up is a worse defect than the one FCST-06
fixed, so a failed write is logged, kept in `last_error` - where `words`
and `yesterday()` both read it - and RETURNED, never thrown.
⚠️ THE TEMP FILE IS REMOVED ON EVERY FAILING PATH. A write that dies
between `mkstemp` and `os.replace` used to leave its temp behind, and
the refresh task runs 48 times a day: a persistent write failure - a
full volume being the likeliest cause - leaked 48 inodes a day onto the
volume that was already the problem.
"""
if not self.ok:
return False
tmp = None
try:
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(self.path))
with os.fdopen(fd, "w", encoding="utf-8") as fh:
json.dump({"version": SCHEMA, "days": self._days}, fh, indent=1)
# Atomic: a power cut mid-write must not leave a truncated file
# that parses as an empty day.
os.replace(tmp, self.path)
tmp = None # it IS the file now; do not unlink
except OSError as err:
self.last_error = f"could not write ({err})"
_LOG.error("could not persist price history: %s", err)
return False
finally:
if tmp is not None:
try:
os.unlink(tmp)
except OSError: # nothing further to try, and a
pass # failed cleanup is not a fault
return True
# -- the one public entry point -----------------------------------------
def carry(self, read: PriceRead, *, since: datetime, until: datetime,
stamp: datetime) -> PriceRead:
"""Remember what this read published, then §8.3-fill the missing hours.
THE ORDER IS DELIBERATE AND IS THE POINT OF THE METHOD. The read is
recorded FIRST, so a feed that hands back yesterday alongside today
refreshes yesterday's record from live data before it is consumed as
the profile; and the filled series - the one thing that must never be
recorded - exists only as this method's return value, which this object
never sees again. See rule 1, including how that order is graded.
Returns whatever `fill_from_yesterday()` returns: a series over the
whole hours of [since, until), carrying EVERY point the provider
published inside it - by identity, and four of them in one hour on a
quarter-hourly feed - plus one carried point for each remaining whole
hour FOR WHICH YESTERDAY HELD THAT WALL-CLOCK HOUR. An hour neither
source can price is left out (§11.4 for that hour), and `no_data` comes
back when that is every hour.
"""
self._remember(read, stamp)
return fill_from_yesterday(read, yesterday=self.yesterday(stamp),
since=since, until=until, stamp=stamp)
# -- the halves, private because rule 1 is an ordering ------------------
def _remember(self, read: PriceRead, stamp: datetime) -> None:
"""File a provider read's points under their own Brussels days.
⚠️ PRIVATE, AND CALLED FROM EXACTLY ONE PLACE. A caller that can reach
this can record a filled series, which is the decay in rule 1. There is
no value-level defence against that - a carried point and a published
provisional point are the same three fields - so the defence is that
nothing outside this file names it, and `test_price_history.py` checks
the tree for that rather than trusting the underscore.
"""
points = read.match(series=lambda r: r.points, no_data=lambda _r: ())
if not points:
return
product = read.match(series=lambda r: r.product.value,
no_data=lambda _r: None)
provider = read.match(series=lambda r: r.provider,
no_data=lambda _r: None)
# ⚠️ THE STATE THIS READ STARTED FROM, KEPT SO IT CAN BE PUT BACK
# (FCST-06). Everything below merges into `self._days` IN PLACE, and
# whether any of it may be KEPT is not knowable until `_save()` has
# tried - so the merge runs, and it is undone if the file did not take
# it. There is no `not self.ok` branch above any more: a store with no
# durable volume is one of the answers `_save()` gives, not a separate
# symptom with a test of its own. Module docstring, fifth warning.
#
# ⚠️ DEEP, NOT `dict(self._days)`. A one-level copy shares the day
# RECORDS, and the loop below rebinds keys on the ones it touches - so
# the "previous" state would hand back records the merge had already
# edited, and a store with a good yesterday on disk would go on
# answering the merge of it that never reached the file. It would also
# be right only for as long as nobody makes this loop mutate a `points`
# list or a row in place. A copy that is right by construction outlives
# that, and the store holds at most KEEP_DAYS + 2 records, 48 times a
# day.
before = copy.deepcopy(self._days)
touched = set()
junked = {}
for p in points:
day = _bdate(p.start)
rec = self._days.setdefault(
day, {"product": product, "provider": provider, "points": []})
# Rule 4: MERGE by instant. A later read updates the hours it
# carries and leaves the rest; a revised price for an hour already
# held wins, because it is the newer statement about that hour.
#
# ⚠️ EVERY HELD ROW GOES THROUGH `_point_of()` (FCST-05's MEDIUM).
# This was `{q["start"]: q for q in raw if isinstance(q, dict) and
# "start" in q}` sorted by `q["start"]` - a key and a sort key that
# trusted a value read off disk to be a hashable, sortable string.
# An `int`, `null`, a `dict` or a `list` there raised `TypeError`
# out of `carry()` on the day the feed was writing to, and
# `run_tariff_refresh` caught it, so the add-on stayed up while
# EVERY refresh failed for good. A row that is not a point is now
# dropped and reported here, exactly as the read path drops it, and
# `_save()` below writes the record back clean - which is the
# self-heal the raising version could never reach.
#
# ⚠️ AND THE MERGE KEY IS THE VALIDATED INSTANT, NOT ITS SPELLING.
# Aware datetimes hash and compare by instant, so this is rule 4 as
# rule 4 is written. The old string key merged by SPELLING, and the
# old lexicographic sort put the second 02:00 of the 25-hour
# October day (`+01:00`) BEFORE the first (`+02:00`). Only the
# stored order was affected - `_parse_points()` re-sorts by instant
# on the way out - but the store now holds what it says it holds.
raw = rec.get("points")
held = {}
for q in (raw if isinstance(raw, list) else ()):
kept = _point_of(q)
if kept is None:
junked[day] = junked.get(day, 0) + 1
continue
held[kept.start] = q
held[p.start] = p.canonical()
order = sorted(held, key=lambda t: t.astimezone(timezone.utc))
rec["points"] = [held[t] for t in order]
rec["product"], rec["provider"] = product, provider
touched.add(day)
if junked:
# Said, not swallowed. `words` prints `last_error`, so a store that
# has just discarded rows off disk stops reporting plain health -
# and after `_save()` below the file is clean, so a restart clears
# this. `_save()` overwrites it if the write itself fails, which is
# the more urgent fact of the two - and on that path the drop is
# undone with the rest of the merge, so those rows are still in the
# file and the next write that completes drops them again.
self.last_error = (
f"{sum(junked.values())} malformed point row(s) dropped from "
+ ", ".join(f"{d} ({n})" for d, n in sorted(junked.items())))
_LOG.warning("price history at %s: %s - the record is rewritten "
"without them and the refresh continues",
self.path, self.last_error)
self._prune(stamp)
if not self._save():
# ⚠️ A STORE THAT CANNOT PERSIST RECORDS NOTHING - the rule, asked
# ONCE as a question about the outcome instead of tested twice
# against two symptoms of it. Keeping the merge here is exactly the
# cache the module docstring refuses: `yesterday()` would answer a
# series the file does not contain, all day, until a restart
# dropped it without a word - and `words` and `yesterday()` would
# be describing two different stores. The reason is already in
# `last_error` (from `_save()`, or from `_open()` when there is no
# volume at all), so both answers now describe what is on disk.
#
# It does NOT raise, and §6 is why: a broken cache must not be a
# broken controller. `carry()` still returns the fill, the horizon
# is as short as the feed makes it, and the add-on regulates.
self._days = before
return
_LOG.debug("price history: %d point(s) filed under %s; %d day(s) held",
len(points), ", ".join(sorted(touched)), len(self._days))
def _prune(self, stamp: datetime) -> None:
"""Drop records no lookup can reach. A sort-order cap, plus two pins.
⚠️ NOTHING IS DELETED BY ARITHMETIC AGAINST THE CLOCK. A clock that
jumped - the failure mode WITNESS-03 spent a session on - would
otherwise delete every record in one pass and leave the fill with
nothing on the morning after, which is the one morning it is for.
Deleting on a bad clock is unrecoverable; holding a few days too many
is not, so the cap is `sorted(...)[-KEEP_DAYS:]` and stays that way.
⚠️ AND A CAP ALONE IS DEFEATED BY ONE POINT IN THE WRONG YEAR. Sorting
is lexicographic on the date string, so a stray far-future record sorts
LAST and is immortal - it consumes a slot for good, and with the steady
state already {yesterday, today, tomorrow} filling all three, yesterday
is what falls out. A feed that publishes to D+2 does the same thing
without any stray at all. So the two days `yesterday()` can ACTUALLY be
asked for are pinned on top of the cap.
`stamp` is READ here, which it was not before: the pin is the only use
of it, and it is additive. Pinning can only RETAIN a record, never
delete one, so a jumped `stamp` pins two dates that are not held and
changes nothing - the WITNESS-03 property is preserved by construction
rather than by remembering to check it.
"""
if len(self._days) <= KEEP_DAYS:
return
keep = set(sorted(self._days)[-KEEP_DAYS:])
keep |= {_bdate(stamp), _yesterday_of(stamp)}
for day in [d for d in self._days if d not in keep]:
del self._days[day]
def yesterday(self, stamp: datetime) -> PriceRead:
"""The published series of the Brussels day BEFORE `stamp`'s, or no_data.
⚠️ THE DATE IS COMPUTED, NOT LOOKED UP. See rule 2: `newest()` would
answer Friday's prices to a Monday fill, and the fill would flag them as
one day old because that is the only thing its flag can say.
"""
if stamp.tzinfo is None or stamp.utcoffset() is None:
raise ValueError(
f"yesterday({stamp!r}): a naive datetime is not an instant, "
f"and 'which Brussels day was yesterday' is exactly the "
f"question a wall-clock reading cannot answer.")
want = _yesterday_of(stamp)
if self.last_error and not self._days:
return PriceRead.no_data(
f"the price history at {self.path} is {self.last_error}, so "
f"§8.3 has no profile to carry forward for {want}; the plan "
f"covers only the hours the feed published")
rec = self._days.get(want)
if rec is None:
have = ", ".join(sorted(self._days)) or "no days at all"
return PriceRead.no_data(
f"no published prices recorded for {want} (held: {have}), so "
f"§8.3 has no profile to carry forward. A record for another "
f"day is not a yesterday")
points, bad = _parse_points(rec.get("points"))
if not points:
why = bad or "it is empty"
return PriceRead.no_data(
f"the {want} record holds no usable point ({why}), and an "
f"empty series is not a free day")
try:
product = PriceProduct(rec.get("product"))
except ValueError:
return PriceRead.no_data(
f"the {want} record names product {rec.get('product')!r}, "
f"which is not a PriceProduct. The optimiser switches on that "
f"enum, so a record that cannot name one cannot be a series")
if bad:
_LOG.warning("price history: %s in the %s record were dropped; "
"%d point(s) carried forward", bad, want, len(points))
return PriceRead.series(points, product,
str(rec.get("provider") or "price_history"),
stamp)
# -- observability -------------------------------------------------------
@property
def words(self) -> str:
"""One line for the status page. Says what is held, or why nothing is.
⚠️ THIS MUST NOT RAISE, EVER. It is read once from `Controller.__init__`
and again on every status render, and neither call site guards it - so
an exception here is an add-on that does not start. It went out that
way: a day record that was a string instead of a dict raised
`AttributeError` from the `len(...)` below. `_load()` now drops such a
record and `_held()` is total, so this has two independent reasons not
to fail on a broken file.
⚠️ AND IT COUNTED POINTS AND CALLED THEM HOURS (FCST-05's LOW). The row
was `f"{d} ({_n_points(rec)} h)"`; on the quarter-hourly feed this
add-on already supports that reads `'2026-01-13 (96 h)'` for a 24-hour
day. Both numbers are printed now - see `_held()` - because they are
two different facts and only one of them was ever being shown.
"""
if not self.ok:
return f"unavailable - {self.last_error}"
if not self._days:
why = f" - {self.last_error}" if self.last_error else ""
return f"empty{why} - nothing to carry forward yet"
held = "; ".join(_row_of(d, *_held(self._days[d]))
for d in sorted(self._days, key=str))
# A file that PARTLY loaded says so, or the dropped days read as days
# that were simply never written.
return held + (f" [{self.last_error}]" if self.last_error else "")
def _parse_points(raw) -> tuple:
"""Canonical dicts -> PricePoints, and a phrase naming what was dropped.
⚠️ A MALFORMED POINT IS DROPPED, NOT DEFAULTED, and the drop is reported.
`float(row.get("eur_per_kwh", 0))` is the shape of the whole family of
defects this repo has been removing: a cache that lost a field would answer
"that hour was free" to a fill that flags it as an ordinary guess.
"""
if not isinstance(raw, list):
return (), "the `points` field is not a list"
out, dropped = [], 0
for row in raw:
# ⚠️ `_point_of()` IS THE ONE VALIDATOR, and this loop used to BE it.
# The write path had its own, weaker standard; see `_point_of()` for
# what that cost. One standard, two callers, no asymmetry to re-open.
pt = _point_of(row)
if pt is None:
dropped += 1
continue
out.append(pt)
out.sort(key=lambda p: p.start.astimezone(timezone.utc))
return tuple(out), (f"{dropped} malformed point(s)" if dropped else "")
class NoHistory:
"""The explicit "this caller keeps no history" - a NAME, not a `None`.
⚠️ IT EXISTS SO THE PARAMETER CAN BE REQUIRED. `history=None` meaning "skip
the fill" is the shape this repo keeps removing: a caller who forgets the
argument gets the truncated-horizon behaviour of the bug this ticket is
about, silently, and the code reads as though the fill is happening. With
no default there is nothing to forget, and a caller with no durable volume
- a check, a replay harness - has to say `NoHistory()` and be visible in
the diff.
It carries out the ordering faithfully: it records nothing and carries
nothing, so `carry()` returns the provider's read unchanged and the plan is
exactly what it was before FCST-04.
"""
ok = False
last_error = "no history is kept by this caller"
words = "not kept"
def carry(self, read: PriceRead, *, since, until, stamp) -> PriceRead:
return read
def yesterday(self, stamp: datetime) -> PriceRead:
return PriceRead.no_data(
"this caller keeps no price history (NoHistory), so §8.3 has no "
"profile to carry forward")
+78 -5
View File
@@ -207,7 +207,7 @@ from datetime import datetime, timedelta, timezone
from .price import (
MAX_PLAUSIBLE_EUR_KWH, Absence, PricePoint, PriceProduct, PriceRead,
Uncertainty, _Outcome, brussels, load_prices, price_option,
required_spread, uncertainty_of, widen_factor,
provider_configured, required_spread, uncertainty_of, widen_factor,
)
_LOG = logging.getLogger("goodwe.tariff.static")
@@ -728,6 +728,33 @@ class TariffPlan:
self.origin = origin
# -- derived facts -------------------------------------------------------
@property
def from_feed(self) -> bool:
"""Is this series an OBSERVATION of the configured provider's feed?
TARIFF-12. `origin` names four ways a plan is obtained; this collapses
them to the one question a caller holding a PREVIOUS plan has to ask
about a new one - did this resolution see the feed, or is it standing
in for one that did not? Only `"provider"` saw it. `"fallback"` and
`"none"` are the two §11.4 rows, and `"static"` is a site with no
feed to observe at all, where the question does not arise.
⚠️ IT IS NOT `has_prices`, AND THE DIFFERENCE IS THE WHOLE POINT. A
§11.4 fallback HAS prices - the operator's own static profile - but
every point `_degrade()` builds carries `provisional=True`, so
`firm_points()` is empty, `arbitrage()` refuses and `cheap_now()` is
False. It is a real tariff and it is not an observation, and a caller
deciding whether to REPLACE an observation with it needs the second
fact, not the first.
⚠️ AND IT IS A PROPERTY HERE RATHER THAN AN `origin == "provider"` AT
THE CALL SITE, for `main.fresh_tariff_plan()`'s stated reason: a check
has to be able to assert on the shipped expression rather than on a
transcription that drifts, and `origin`'s vocabulary belongs to this
module.
"""
return self.origin == "provider"
@property
def has_prices(self) -> bool:
return self.read.match(series=lambda _r: True, no_data=lambda _r: False)
@@ -1019,7 +1046,7 @@ def static_status(opts: dict, now: datetime):
async def resolve_tariff(opts: dict, since: datetime, until: datetime,
transport=None, now=None) -> TariffPlan:
transport=None, now=None, *, history) -> TariffPlan:
"""The full §9.2 + §11.4 resolution. MAY PERFORM I/O - see `static_plan`.
⚠️ NOT ON THE 30 s TICK, and that is a deliberate boundary rather than an
@@ -1033,17 +1060,63 @@ async def resolve_tariff(opts: dict, since: datetime, until: datetime,
price_provider off + profile -> the CONFIGURED product (§9.2 rows 3-4)
price_provider off + none -> no prices at all (§11.4, nothing to
fall back to)
provider series -> the feed (§9.2 rows 1-2)
provider series -> the feed, §8.3-FILLED (§9.2 rows 1-2)
provider no_data + profile -> §11.4 FALLBACK, provisional, no arbitrage
provider no_data + none -> no prices at all
⚠️ `history` IS REQUIRED, KEYWORD-ONLY, AND HAS NO DEFAULT (FCST-04). It is
the §8.3 carry-forward store - `price_history.PriceHistory`, or the explicit
`price_history.NoHistory()` for a caller that keeps none. A default of
`None` meaning "skip the fill" would give a caller who forgot the argument
the TRUNCATED HORIZON this ticket exists to remove, silently, while the code
read as though the fill were happening. There is nothing to forget now.
⚠️ AND THE FILL RUNS ONLY ON A `series` READ. A feed that published nothing
still falls to `_degrade()` and §11.4's own row - the static profile the
operator configured FOR that case - rather than to yesterday's feed. The two
are indistinguishable to the optimiser (both are 100 % provisional, so
`arbitrage()` finds fewer than two firm hours and refuses, and `cheap_now()`
is False), so preferring yesterday would change a documented degradation for
a difference only the status page can see. See price_history.py's closing
note and DECISIONS/FCST-04-the-day-that-is-carried.md.
⚠️ THE FILL IS PER WHOLE HOUR AND ITS COVERAGE IS CONDITIONAL. Every point
the provider published inside the window passes through BY IDENTITY - four
of them in one hour on a quarter-hourly product like `eneco_eddy` - and a
carried point is added only for a whole hour the feed did not print at all
AND yesterday did. An hour neither source can price is left out, which is
§11.4 for that hour: `plan.price_at()` raises rather than answering, and an
hour with no price is not a free one.
⚠️ THE FILLED SERIES IS WINDOWED TO [since, until), AND THAT IS A CHANGE.
`load_prices()` returns whatever the feed sent, past hours included, and
`TariffPlan.arbitrage()` took its spread over all of it - so an hour that
had already gone could be the cheapest hour of a cycle nothing can now buy
in. `fill_from_yesterday()` walks the requested horizon and emits exactly
that, so the spread is over hours that are still ahead. Both directions of
the difference are real: the window gains the unpublished hours it was
missing and loses the elapsed ones it should never have had.
"""
stamp = now or datetime.now(timezone.utc)
pid = str(price_option(opts, "price_provider") or "off").strip()
if pid in ("", "off"):
if not provider_configured(opts):
return static_plan(opts, since, until, provisional=False, now=stamp)
read = await load_prices(opts, since, until, transport)
return read.match(
series=lambda r: _provider_plan(opts, r, since, until, stamp, history),
no_data=lambda r: _degrade(opts, r, since, until, stamp))
def _provider_plan(opts, read, since, until, stamp, history) -> TariffPlan:
"""§8.3: record what the feed published, carry yesterday into the gaps.
The fill can itself return `no_data` - a feed whose points all fall outside
the horizon, with nothing in the history to fill it from - and that is the
§11.4 row arrived at one step later, so it degrades exactly as a failed
fetch does rather than being special-cased here.
"""
filled = history.carry(read, since=since, until=until, stamp=stamp)
return filled.match(
series=lambda r: TariffPlan(r, _quiet_profile(opts), "provider"),
no_data=lambda r: _degrade(opts, r, since, until, stamp))
+25 -3
View File
@@ -265,9 +265,17 @@ options:
# TARIFF - what energy costs, and whether to trade on it
# ==========================================================================
# `off` is the shipped default and means the §11.4 no-price-data row applies:
# static profile, plan provisional, NO ARBITRAGE. Nothing in 0.5.x consumes
# the series yet - OPT-04 is the consumer - so turning this on today fetches,
# validates and publishes prices and changes no energy decision.
# static profile, plan provisional, NO ARBITRAGE.
#
# ⚠️ SELECTING A PROVIDER IS WHAT STARTS THE FETCH, AND IT IS NOT
# `arbitrage_enabled` (TARIFF-10). With a provider set and trading off - the
# shipped default - the add-on fetches the feed every 30 minutes, validates
# it, records the day in /data/price_history.json and reports what it holds
# on the *Price history* status row. It changes NO energy decision: the cheap
# window the maintenance cycle reads still comes from `static_profile`, and
# §10.1's arbitrage is gated on `arbitrage_enabled` alone. Until TARIFF-10
# the fetch was gated on `arbitrage_enabled` too, so this option did nothing
# whatsoever on a shipped install - the store was never even created.
#
# ⚠️ THESE DEFAULTS ARE MIRRORED ONCE, IN app/price.py's
# PRICE_OPTION_DEFAULTS, and test_price.py reads THIS FILE and asserts the two
@@ -318,6 +326,14 @@ options:
# validate.py warns about exactly that.
static_profile: ""
price_csv_path: "/share/goodwe_prices.csv"
# ⚠️ AN OVERRIDE IS THE EXACT URL (TARIFF-09). The plugin sends it verbatim
# and appends nothing - including the `start`/`end` date range it derives from
# the horizon for its own feed - so an override gets whatever range that URL
# answers, which for a bare day-ahead endpoint is ONE Brussels day. Every hour
# past it is then carried from yesterday and flagged provisional, and §8.3
# keeps provisional hours out of the trade set. The plugin logs a WARNING
# naming that consequence on every fetch that uses an override; clearing this
# option is what restores the full horizon.
price_feed_url: ""
price_horizon_h: 48
price_timeout_s: 10
@@ -344,6 +360,12 @@ options:
# configuration: this is the first strategy in the add-on that moves energy
# against the meter on purpose, so it is armed deliberately or not at all.
#
# ⚠️ IT IS NOT WHAT TURNS THE PRICE FEED ON (TARIFF-10). `price_provider`
# above decides whether prices are FETCHED; this decides whether they move
# energy. Leaving this off on a site with a provider configured is a
# supported, documented state: prices are fetched, validated and recorded,
# and nothing trades.
#
# ⚠️ IT NEEDS A PRICE SOURCE AND A P1 METER, and validate.py refuses the
# combinations that cannot work rather than letting them look armed. With
# `price_provider: off` and no `static_profile` there is no series to trade
@@ -0,0 +1 @@
{"license_info":"CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de","unix_seconds":[1757455200,1757458800,1757462400,1757466000,1757469600,1757473200,1757476800,1757480400,1757484000,1757487600,1757491200,1757494800,1757498400,1757502000,1757505600,1757509200,1757512800,1757516400,1757520000,1757523600,1757527200,1757530800,1757534400,1757538000],"price":[73.0,61.31,64.32,62.49,67.0,64.48,94.52,149.26,132.43,91.61,84.8,79.61,69.23,66.75,68.91,69.31,80.29,66.91,105.2,161.05,116.73,84.2,64.1,42.41],"unit":"EUR / MWh","deprecated":false}
@@ -0,0 +1 @@
{"license_info":"CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de","unix_seconds":[1788904800,1788905700,1788906600,1788907500,1788908400,1788909300,1788910200,1788911100,1788912000,1788912900,1788913800,1788914700,1788915600,1788916500,1788917400,1788918300,1788919200,1788920100,1788921000,1788921900,1788922800,1788923700,1788924600,1788925500,1788926400,1788927300,1788928200,1788929100,1788930000,1788930900,1788931800,1788932700,1788933600,1788934500,1788935400,1788936300,1788937200,1788938100,1788939000,1788939900,1788940800,1788941700,1788942600,1788943500,1788944400,1788945300,1788946200,1788947100,1788948000,1788948900,1788949800,1788950700,1788951600,1788952500,1788953400,1788954300,1788955200,1788956100,1788957000,1788957900,1788958800,1788959700,1788960600,1788961500,1788962400,1788963300,1788964200,1788965100,1788966000,1788966900,1788967800,1788968700,1788969600,1788970500,1788971400,1788972300,1788973200,1788974100,1788975000,1788975900,1788976800,1788977700,1788978600,1788979500,1788980400,1788981300,1788982200,1788983100,1788984000,1788984900,1788985800,1788986700,1788987600,1788988500,1788989400,1788990300,1788991200,1788992100,1788993000,1788993900,1788994800,1788995700,1788996600,1788997500,1788998400,1788999300,1789000200,1789001100,1789002000,1789002900,1789003800,1789004700,1789005600,1789006500,1789007400,1789008300,1789009200,1789010100,1789011000,1789011900,1789012800,1789013700,1789014600,1789015500,1789016400,1789017300,1789018200,1789019100,1789020000,1789020900,1789021800,1789022700,1789023600,1789024500,1789025400,1789026300,1789027200,1789028100,1789029000,1789029900,1789030800,1789031700,1789032600,1789033500,1789034400,1789035300,1789036200,1789037100,1789038000,1789038900,1789039800,1789040700,1789041600,1789042500,1789043400,1789044300,1789045200,1789046100,1789047000,1789047900,1789048800,1789049700,1789050600,1789051500,1789052400,1789053300,1789054200,1789055100,1789056000,1789056900,1789057800,1789058700,1789059600,1789060500,1789061400,1789062300,1789063200,1789064100,1789065000,1789065900,1789066800,1789067700,1789068600,1789069500,1789070400,1789071300,1789072200,1789073100,1789074000,1789074900,1789075800,1789076700],"price":[150.93,151.41,150.99,147.59,153.25,147.26,144.04,141.53,147.79,146.53,143.55,139.93,145.58,139.56,139.11,138.06,138.92,138.52,138.04,138.65,137.29,139.03,144.59,150.34,154.12,174.5,178.43,186.62,199.35,200.21,200.11,197.43,220.07,202.0,196.01,174.03,198.32,167.49,152.54,142.05,161.5,145.45,128.61,100.85,139.18,115.83,94.41,80.39,111.25,89.61,84.99,74.38,80.53,70.58,62.27,57.1,62.24,64.95,68.99,71.19,66.9,87.32,109.0,123.12,68.48,125.53,155.99,161.9,138.77,160.57,167.35,173.0,142.2,172.83,196.19,211.26,205.85,218.77,217.08,222.0,210.98,208.46,208.88,210.02,209.77,202.61,200.27,191.49,194.39,195.04,197.26,190.92,188.39,198.05,203.05,191.84,216.08,216.08,214.92,200.0,223.37,201.43,196.32,190.0,199.53,194.83,180.53,172.23,179.88,175.77,167.1,173.89,163.29,165.99,171.41,178.73,170.27,177.37,187.71,201.07,193.62,207.61,218.27,226.77,227.17,234.05,236.99,229.94,248.93,227.5,216.08,195.67,215.99,195.01,177.1,155.09,174.91,152.71,139.59,123.83,140.77,123.19,114.96,94.5,102.77,96.63,94.57,116.34,114.63,126.82,129.89,123.95,105.0,119.11,121.0,136.79,123.41,135.01,135.0,156.68,146.88,152.16,162.02,181.78,154.76,184.28,194.3,215.76,197.23,234.15,253.03,296.58,345.32,389.16,482.99,559.0,551.11,446.41,358.97,295.9,271.39,268.06,241.43,238.01,239.7,226.41,223.05,211.56,218.21,210.61,197.78,171.79],"unit":"EUR / MWh","deprecated":false}
@@ -0,0 +1 @@
{"license_info":"CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de","unix_seconds":[1788991200,1788992100,1788993000,1788993900,1788994800,1788995700,1788996600,1788997500,1788998400,1788999300,1789000200,1789001100,1789002000,1789002900,1789003800,1789004700,1789005600,1789006500,1789007400,1789008300,1789009200,1789010100,1789011000,1789011900,1789012800,1789013700,1789014600,1789015500,1789016400,1789017300,1789018200,1789019100,1789020000,1789020900,1789021800,1789022700,1789023600,1789024500,1789025400,1789026300,1789027200,1789028100,1789029000,1789029900,1789030800,1789031700,1789032600,1789033500,1789034400,1789035300,1789036200,1789037100,1789038000,1789038900,1789039800,1789040700,1789041600,1789042500,1789043400,1789044300,1789045200,1789046100,1789047000,1789047900,1789048800,1789049700,1789050600,1789051500,1789052400,1789053300,1789054200,1789055100,1789056000,1789056900,1789057800,1789058700,1789059600,1789060500,1789061400,1789062300,1789063200,1789064100,1789065000,1789065900,1789066800,1789067700,1789068600,1789069500,1789070400,1789071300,1789072200,1789073100,1789074000,1789074900,1789075800,1789076700],"price":[216.08,216.08,214.92,200.0,223.37,201.43,196.32,190.0,199.53,194.83,180.53,172.23,179.88,175.77,167.1,173.89,163.29,165.99,171.41,178.73,170.27,177.37,187.71,201.07,193.62,207.61,218.27,226.77,227.17,234.05,236.99,229.94,248.93,227.5,216.08,195.67,215.99,195.01,177.1,155.09,174.91,152.71,139.59,123.83,140.77,123.19,114.96,94.5,102.77,96.63,94.57,116.34,114.63,126.82,129.89,123.95,105.0,119.11,121.0,136.79,123.41,135.01,135.0,156.68,146.88,152.16,162.02,181.78,154.76,184.28,194.3,215.76,197.23,234.15,253.03,296.58,345.32,389.16,482.99,559.0,551.11,446.41,358.97,295.9,271.39,268.06,241.43,238.01,239.7,226.41,223.05,211.56,218.21,210.61,197.78,171.79],"unit":"EUR / MWh","deprecated":false}
@@ -0,0 +1,82 @@
{
"card": "TARIFF-08, extended by TARIFF-09",
"what": "UNMODIFIED response bodies from the public EPEX BE day-ahead feed, captured so the SHIPPED parser can be replayed against what the source really publishes",
"why": "every price fixture in the suite was built from a `{\"data\": [{\"start\": ..., \"price_eur_mwh\": ...}]}` document that this feed has never published. The suite therefore graded `_points_from_json` against its own assumption - a closed loop that could not notice the assumption was wrong - and §11.4 turned the resulting failure into a silent fall back to the static profile. A captured real body is what breaks that loop, so these files are bytes, not a reconstruction: nothing here is pretty-printed, re-keyed or trimmed. TARIFF-09 adds a body from a request that names a DATE RANGE, for the same reason one layer out: the shipped plugin took `since`/`until` and asked for the bare default URL, so the feed answered ONE Brussels day and every hour past today was carried forward and dropped from the trade set as provisional. A two-day body captured from the range request is what makes \"asking for the range gets the range\" checkable against something the plugin did not shape.",
"source": "Energy-Charts API (Fraunhofer ISE), api.energy-charts.info",
"upstream": "Bundesnetzagentur | SMARD.de",
"licence": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)",
"licence_note": "the BE bidding zone is in the API's CC BY 4.0 list, so redistribution is permitted with attribution. DATA-02 established this against the API's own OpenAPI document before committing `epex-be-dayahead-2025-09-01_2026-09-01.json`; both bodies below carry the same `license_info` string, recorded per file and asserted by the replay test.",
"attribution": "Data licensed CC BY 4.0 by Bundesnetzagentur | SMARD.de, retrieved via the Energy-Charts API (Fraunhofer ISE), https://energy-charts.info. Published without changes.",
"retrieved_utc": "2026-09-10T06:35:00+00:00 (TARIFF-08's two), 2026-09-10T07:45:49+00:00 (TARIFF-09's)",
"retrieved_with": "curl, by hand; no code in `app/` fetches these",
"shape": "COLUMNAR: parallel `unix_seconds` (UTC epoch seconds) and `price` arrays, plus `unit`, `license_info` and `deprecated`. There is no `data` key and no per-point object.",
"replayed_by": "test_price.py sections 2b (TARIFF-08) and 2c (TARIFF-09)",
"files": {
"energy-charts-be-price-2026-09-10-quarterhourly.json": {
"url": "https://api.energy-charts.info/price?bzn=BE",
"note": "the SHIPPED default URL of `generic_dayahead`, verbatim and with no parameters, so the replay drives the bytes the add-on's own request produces",
"covers_local_day": "2026-09-10 (Europe/Brussels)",
"granularity_s": 900,
"intervals": 96,
"bytes": 1877,
"sha256": "176309d0020ad208320797385ef68b6a98ac248218539416cddf6579dd060487",
"unit_as_returned": "EUR / MWh",
"licence_as_returned": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de",
"deprecated_as_returned": false,
"top_level_keys": [
"deprecated",
"license_info",
"price",
"unit",
"unix_seconds"
],
"also_returned_verbatim_by": {
"url": "https://api.energy-charts.info/price?bzn=BE&start=2026-09-10&end=2026-09-11",
"retrieved_utc": "2026-09-10T07:46:55+00:00",
"note": "TARIFF-09's PRE-PUBLICATION case, and it needed no second file because the answer was BYTE-IDENTICAL to the bare-URL capture above - the same 1877 bytes and the same sha256. Asked for two days at 09:46 CEST, before the ~13:00-14:00 CET auction publishes 2026-09-11, the feed returned the ONE day that exists. That is the short answer this card requires to stay a `series` of what exists rather than becoming `no_data`, and the replay re-derives the identity from the bytes rather than restating it."
}
},
"energy-charts-be-price-2025-09-10-hourly.json": {
"url": "https://api.energy-charts.info/price?bzn=BE&start=2025-09-10&end=2025-09-10",
"note": "a day BEFORE Belgium's 15-minute MTU (2025-10-01), so the same endpoint answers HOURLY. Kept because the parser must not assume either spacing, and a fixture at only one granularity cannot grade that",
"covers_local_day": "2025-09-10 (Europe/Brussels)",
"granularity_s": 3600,
"intervals": 24,
"bytes": 572,
"sha256": "03fe41690810df224611f4016899f856457f67a3c12c7654da17b1331ff2eff4",
"unit_as_returned": "EUR / MWh",
"licence_as_returned": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de",
"deprecated_as_returned": false,
"top_level_keys": [
"deprecated",
"license_info",
"price",
"unit",
"unix_seconds"
]
},
"energy-charts-be-price-2026-09-09_2026-09-10-twoday.json": {
"url": "https://api.energy-charts.info/price?bzn=BE&start=2026-09-09&end=2026-09-10",
"retrieved_utc": "2026-09-10T07:45:49+00:00",
"note": "TARIFF-09. A RANGE request, and the counterfactual the ticket is built on: `start`/`end` are INCLUSIVE Brussels calendar dates and this one answers 192 quarter-hours across TWO local days, where the bare `?bzn=BE` of the same endpoint answers 96 across one. The shipped `generic_dayahead` now derives these two dates from the `since`/`until` it is handed.",
"covers_local_days": [
"2026-09-09",
"2026-09-10"
],
"granularity_s": 900,
"intervals": 192,
"bytes": 3579,
"sha256": "4d83f5efb4294a4a682562eac619bf35e45c95edde957bfa94c402e1777cc9b3",
"unit_as_returned": "EUR / MWh",
"licence_as_returned": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de",
"deprecated_as_returned": false,
"top_level_keys": [
"deprecated",
"license_info",
"price",
"unit",
"unix_seconds"
]
}
}
}
+320 -1
View File
@@ -92,7 +92,9 @@ from app.price import (
PRICE_OPTION_DEFAULTS, Absence, PricePoint, PriceProduct, PriceRead,
Uncertainty, WIDEN_FACTOR, brussels, required_spread, widen_factor,
)
from app.static_tariff import TariffPlan, parse_static_profile, static_plan
from app.static_tariff import (
TariffPlan, parse_static_profile, resolve_tariff, static_plan,
)
from app.validate import validate_startup_config
UTC = timezone.utc
@@ -1284,6 +1286,323 @@ check("main.py reads every arbitrage option through app/arbitrage.py or "
"arbitrage_soc_ceiling")))
# --------------------------------------------------------------------------- #
print()
print("8b. TARIFF-12 - a RETURNED degradation must not clear a fresh plan")
# ⚠️ THE STIMULUS IS A RETURN, NOT A RAISE, AND THAT IS THE WHOLE CHECK.
# `run_tariff_refresh()` guarded the plan with `except Exception` alone, and
# `load_prices()` states in its own docstring that it "RETURNS, NEVER RAISES,
# for every failure a site can actually have" - so the one case the guard was
# written for, a supplier outage, never reached it. Everything below drives the
# SHIPPED chain and stubs nothing but the socket: the transport raises the way
# aiohttp does, `load_prices()` turns that into a `no_data` read, `_degrade()`
# turns THAT into a §11.4 plan, and `run_tariff_refresh()` decides what to do
# with it. A check that patched `resolve_tariff` to raise would have passed
# against the defective code.
import asyncio # noqa: E402
from app import price as _price # noqa: E402
from app.price_history import NoHistory # noqa: E402
class ClientConnectorError(Exception):
"""aiohttp's own class name, so the §11.4 reason this drives is the
sentence the 40 h soak recorded at refresh 56, verbatim."""
_T12_FEED_UP = {"up": True}
async def _t12_transport(url, timeout_s):
"""The socket, and nothing else, is what this replaces."""
if not _T12_FEED_UP["up"]:
raise ClientConnectorError(
"Cannot connect to host api.energy-charts.info:443 ssl:default")
t0 = datetime.now(UTC).replace(minute=0, second=0, microsecond=0)
return json.dumps({
"unix_seconds": [int((t0 + timedelta(hours=i)).timestamp())
for i in range(48)],
"price": [80.0 + 40.0 * (i % 6) for i in range(48)],
"unit": "EUR / MWh"}).encode()
class _T12Controller:
"""Just enough Controller to run the SHIPPED refresh coroutine.
Both methods are bound off the shipped class rather than copied, for
`fresh_tariff_plan()`'s own stated reason: a transcription drifts.
"""
run_tariff_refresh = _main.Controller.run_tariff_refresh
fresh_tariff_plan = _main.Controller.fresh_tariff_plan
def __init__(self, opts):
self.o = opts
self.tariff_plan = None
self.tariff_at = None
self.price_history = NoHistory()
self.stop_event = asyncio.Event()
self.stop_event.set() # so the inter-refresh wait returns
self._ticks = 0
@property
def stopping(self):
self._ticks -= 1
return self._ticks < 0
def refresh_once(self):
self._ticks = 1
asyncio.run(self.run_tariff_refresh())
return self
def _t12_opts(profile=""):
# `price_timeout_s` is read with a plain `.get()` by every plugin, so it
# has to be present - an options dict that relies on price_option()'s
# defaults for it fetches with `float(None)` and this section would grade
# a TypeError instead of an outage.
return {"price_provider": "generic_dayahead", "price_timeout_s": 10,
"price_horizon_h": 48, "static_profile": profile}
_T12_HORIZON_H = 48
_T12_STALE_S = _main.TARIFF_MAX_AGE_S
_t12_saved_transport = _price.https_get
_price.https_get = _t12_transport
# -- the accepted baseline (habit 7): one good fetch, varied one thing at a
# -- time from here.
_T12_FEED_UP["up"] = True
_t12_a = derived(lambda: _T12Controller(_t12_opts()).refresh_once())
_t12_good = derived(lambda: _t12_a.tariff_plan)
check(f"BASELINE: a fetch that succeeds stores an OBSERVATION - "
f"{getattr(_t12_good, 'words', _t12_good)!r}",
getattr(_t12_good, "origin", None) == "provider"
and _t12_good.from_feed is True
and len(_t12_good.read.points) == _T12_HORIZON_H)
# -- the stimulus, graded before it is used on anything (habit 6/7) ----------
_T12_FEED_UP["up"] = False
_t12_now = datetime.now(UTC)
_t12_deg = derived(lambda: asyncio.run(resolve_tariff(
_t12_opts(), _t12_now, _t12_now + timedelta(hours=_T12_HORIZON_H),
now=_t12_now, history=NoHistory())))
check("STIMULUS: a dead feed makes resolve_tariff() RETURN a degraded plan - "
"it does not raise, so `except Exception` is not what was protecting "
"the plan and the defect was never on that branch",
isinstance(_t12_deg, TariffPlan))
check(f"...and the returned plan is the §11.4 no-profile row the soak saw - "
f"{getattr(_t12_deg, 'words', _t12_deg)!r}",
isinstance(_t12_deg, TariffPlan) and _t12_deg.origin == "none"
and _t12_deg.from_feed is False and _t12_deg.has_prices is False
and "ClientConnectorError" in _t12_deg.words)
# -- AC1: the degradation must not replace a plan still inside the bound -----
# ⚠️ 1800 s IS STATED HERE, NOT DERIVED FROM THE MODULE BEING GRADED - section
# 8's own lesson. A probe at `TARIFF_MAX_AGE_S - 1` moves with a mutation of
# the constant; half an hour is inside any reading of a 5400 s bound, and it is
# the interval the soak actually ran at.
_t12_b = derived(lambda: _T12Controller(_t12_opts()))
_t12_b.tariff_plan, _t12_b.tariff_at = _t12_good, datetime.now(UTC) - \
timedelta(seconds=1800)
_t12_at_before = _t12_b.tariff_at
check(f"PRECONDITION: the standing plan is 1800 s old and TARIFF_MAX_AGE_S is "
f"{_T12_STALE_S:.0f} s, so it IS fresh when the failed refresh runs - "
f"without this the next check grades an expired plan",
1800 < _T12_STALE_S
and _t12_b.fresh_tariff_plan(datetime.now(UTC)) is _t12_good)
_t12_b.refresh_once()
check("a RETURNED degradation does not replace a plan still inside "
"TARIFF_MAX_AGE_S - the 48 h series is still there, by identity, and "
"not a 0-point `origin: none` read",
_t12_b.tariff_plan is _t12_good
and _t12_b.tariff_plan.has_prices is True
and len(_t12_b.tariff_plan.read.points) == _T12_HORIZON_H)
check("...and `tariff_at` did NOT move, so the kept plan still ages out from "
"the fetch that produced it and is distinguishable from a freshly "
"fetched one. Re-stamping it here is how a plan freezes at its last "
"value and looks current forever",
_t12_b.tariff_at == _t12_at_before)
check("...and the two timestamps a successful refresh and a kept plan produce "
"really are different values, or the check above compares a number "
"against itself",
_t12_a.tariff_at != _t12_at_before)
# -- AC3: TARIFF_MAX_AGE_S governs - the bound is where the module says ------
_t12_c = derived(lambda: _T12Controller(_t12_opts()))
_t12_c.tariff_plan, _t12_c.tariff_at = _t12_good, datetime.now(UTC) - \
timedelta(seconds=_T12_STALE_S + 5)
_t12_c.refresh_once()
check(f"a plan that has aged PAST {_T12_STALE_S:.0f} s IS replaced by the "
f"degradation - the plan is served stale for the age bound and no "
f"longer, so TARIFF_MAX_AGE_S governs something rather than being "
f"unreachable",
_t12_c.tariff_plan is not _t12_good
and _t12_c.tariff_plan.from_feed is False
and _t12_c.tariff_plan.has_prices is False)
_t12_d = derived(lambda: _T12Controller(_t12_opts()))
_t12_d.tariff_plan, _t12_d.tariff_at = _t12_good, datetime.now(UTC) - \
timedelta(seconds=_T12_STALE_S - 60)
_t12_d.refresh_once()
check("...and one minute SHORT of the bound it is still kept, so the line "
"above is a bound rather than 'the degradation always wins'",
_t12_d.tariff_plan is _t12_good)
_t12_e = derived(lambda: _T12Controller(_t12_opts()).refresh_once())
check("a degradation with NO plan standing at all is stored - §11.4 still "
"reaches an install that has never had a good fetch, and the guard is "
"not 'never overwrite'",
_t12_e.tariff_plan is not None
and _t12_e.tariff_plan.from_feed is False
and _t12_e.tariff_at is not None)
# -- the §11.4 STATIC FALLBACK, which is the judgement this card had to make -
_t12_f = derived(lambda: _T12Controller(_t12_opts(STATIC_BE)))
_t12_fb = derived(lambda: asyncio.run(resolve_tariff(
_t12_opts(STATIC_BE), _t12_now,
_t12_now + timedelta(hours=_T12_HORIZON_H), now=_t12_now,
history=NoHistory())))
check(f"PRECONDITION: with a static_profile configured the SAME dead feed "
f"degrades to a §11.4 FALLBACK carrying real prices, not to the "
f"no-profile row - {getattr(_t12_fb, 'words', _t12_fb)!r}",
isinstance(_t12_fb, TariffPlan) and _t12_fb.origin == "fallback"
and _t12_fb.has_prices is True and _t12_fb.from_feed is False)
check("...and every one of its points is provisional, so firm_points() is "
"empty, arbitrage() refuses and cheap_now() is False - it is a real "
"tariff and nothing may be acted on it. That is why it does not replace "
"a fresh observation",
isinstance(_t12_fb, TariffPlan) and _t12_fb.provisional is True
and _t12_fb.firm_points() == ()
and _t12_fb.arbitrage(peak_guard_active=False, min_spread_eur_kwh=0.0,
uncertainty=NO_WEATHER).kind == "refused")
_t12_f.tariff_plan, _t12_f.tariff_at = _t12_good, datetime.now(UTC) - \
timedelta(seconds=1800)
_t12_f_at = _t12_f.tariff_at
_t12_f.refresh_once()
check("a §11.4 static FALLBACK does not replace a fresh provider plan either, "
"although it has prices - a day-ahead price for an hour does not change "
"once published, so the 30-minute-old observation is no less true and "
"is the only one of the two that carries a firm hour",
_t12_f.tariff_plan is _t12_good and _t12_f.tariff_at == _t12_f_at)
# -- ...and the rule is NOT symmetrical: a degradation replaces a degradation.
_t12_g = derived(lambda: _T12Controller(_t12_opts(STATIC_BE)))
_t12_g.tariff_plan, _t12_g.tariff_at = _t12_fb, datetime.now(UTC) - \
timedelta(seconds=1800)
_t12_g_at = _t12_g.tariff_at
_t12_g.refresh_once()
check("a standing plan that is ITSELF a §11.4 fallback IS replaced by the "
"next one, and its timestamp moves - otherwise an outage would freeze "
"the operator's static profile at the window it was first generated "
"for, and the guard would be 'never overwrite' wearing a bound's name",
_t12_g.tariff_plan is not _t12_fb and _t12_g.tariff_plan.origin ==
"fallback" and _t12_g.tariff_at != _t12_g_at)
# -- ...and a recovered feed replaces a kept plan, which is the third term ---
_T12_FEED_UP["up"] = True
_t12_h = derived(lambda: _T12Controller(_t12_opts()))
_t12_h.tariff_plan, _t12_h.tariff_at = _t12_good, datetime.now(UTC) - \
timedelta(seconds=1800)
_t12_h_at = _t12_h.tariff_at
_t12_h.refresh_once()
check("when the feed answers again the observation replaces the kept plan and "
"`tariff_at` moves - the guard keeps a plan against a DEGRADATION, not "
"against a fetch that worked",
_t12_h.tariff_plan is not _t12_good
and _t12_h.tariff_plan.from_feed is True
and _t12_h.tariff_at != _t12_h_at)
# -- TARIFF-10 widened WHICH INSTALLS RUN THIS TASK, so the configurations it
# -- newly keeps alive are graded here rather than reasoned about.
# ⚠️ `price_provider: off` + a `static_profile` + `arbitrage_enabled: true` is a
# configuration validate.py ACCEPTS and TARIFF-10's `or` exists to protect. On
# it `resolve_tariff()` returns `static_plan(provisional=False)` - the
# CONFIGURED product, origin "static" - so `standing.from_feed` is false on
# every refresh and this card's guard must never fire. Confirmed by driving it,
# not by reading the origin table: a guard that quietly latched here would
# freeze a static site's plan at its first window for the life of the install.
_t12_i = derived(lambda: _T12Controller(
{"price_provider": "off", "price_timeout_s": 10, "price_horizon_h": 48,
"static_profile": STATIC_BE}).refresh_once())
_t12_i_plan, _t12_i_at = _t12_i.tariff_plan, _t12_i.tariff_at
_t12_i.refresh_once()
check("PRECONDITION: a `price_provider: off` + static_profile site resolves to "
"the CONFIGURED product, not to a §11.4 row - origin 'static', prices, "
"and from_feed false because there is no feed to have observed",
getattr(_t12_i_plan, "origin", None) == "static"
and _t12_i_plan.has_prices is True and _t12_i_plan.from_feed is False)
check("...and TARIFF-10 now keeps this task alive on exactly that site, where "
"this card's guard must NEVER latch: the second refresh replaces the "
"first and the timestamp moves, so the plan is regenerated for the "
"current window every 30 minutes as it always was",
_t12_i.tariff_plan is not _t12_i_plan
and _t12_i.tariff_plan.origin == "static"
and _t12_i.tariff_at != _t12_i_at)
# ⚠️ AND THE OTHER CONFIGURATION TARIFF-10 ADDED - a provider configured with
# trading DISARMED - is graded by every driven check above without a second
# fixture, because nothing on this path reads `arbitrage_enabled` at all. That
# is a claim about the code rather than about a behaviour, so it is asserted as
# one (habit 4's exception, FCST-06's precedent): a fixture that set the option
# and asserted no difference would be comparing a thing against itself.
_T12_TRADING_OPTS = ("arbitrage_enabled", "cycle_cost_threshold_eur",
"arbitrage_soc_floor", "arbitrage_soc_ceiling")
# ⚠️ AND IT IS WHAT THE FUNCTION READS THAT IS SCANNED, NOT ITS TEXT. Two
# earlier drafts of this check were red against a CORRECT tree: the first
# because the docstring paragraph that SAYS `arbitrage_enabled` is not read is
# itself an occurrence of the string, the second because the log line says
# "arbitrage, where it is armed, stops with it" - which is a sentence, not a
# read. A substring test over a function grades its prose as much as its
# behaviour (habit 4). So the symbols are taken from the AST, and the string
# constants are taken separately and with the docstring dropped: naming
# arbitrage in a message is fine, READING a trading option is the thing claimed
# not to happen.
_t12_syms = derived(lambda: {getattr(n, "id", None) or getattr(n, "attr", None)
for n in ast.walk(_refresh)
if isinstance(n, (ast.Name, ast.Attribute))})
_t12_consts = derived(lambda: {c.value for b in _refresh.body[1:]
for c in ast.walk(b)
if isinstance(c, ast.Constant)
and isinstance(c.value, str)})
check("run_tariff_refresh() READS no trading option and calls nothing in "
"app/arbitrage.py, so keeping a plan is a decision about what the add-on "
"has LOOKED UP - TARIFF-10's own distinction, and why the checks above "
"cover a provider-configured site whether or not arbitrage is armed",
isinstance(_t12_syms, set)
and not any("arbitrage" in (n or "") for n in _t12_syms)
and not (_t12_consts & set(_T12_TRADING_OPTS))
# ...and both scanners can see what IS there, or they grade nothing: the
# one option this task does read, and a name it really does call.
and "price_horizon_h" in _t12_consts
and "fresh_tariff_plan" in _t12_syms
# ...and the docstring really was dropped, or the first draft's defect is
# back and the constant set is grading prose again.
and not any("RETURNS, NEVER RAISES" in c for c in _t12_consts))
# -- the vocabulary, graded on its own so a mutation of it has a named home --
check("TariffPlan.from_feed is true of exactly the observation - a §11.4 "
"fallback, a no-data read and a CONFIGURED static profile are all false, "
"and it is not a synonym for has_prices",
_t12_good.from_feed is True and _t12_fb.from_feed is False
and _t12_deg.from_feed is False
and static_plan({"static_profile": STATIC_BE}, D0,
D0 + timedelta(days=2), now=D0).from_feed is False
and static_plan({"static_profile": STATIC_BE}, D0,
D0 + timedelta(days=2), now=D0).has_prices is True)
# ⚠️ A SHAPE CLAIM, AND IT IS ALLOWED TO BE ONE (FCST-06's precedent). "The age
# is judged in ONE place" is a claim about the code, not about a behaviour: a
# transcribed copy of the same comparison behaves identically until the day the
# two drift. Every other check in this section is driven.
check("run_tariff_refresh() asks fresh_tariff_plan() rather than re-deriving "
"the age bound - one place judges it, which is what that method's own "
"docstring is for",
_refresh is not None
and {n.func.attr for n in ast.walk(_refresh)
if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)}
>= {"fresh_tariff_plan"})
_price.https_get = _t12_saved_transport
# --------------------------------------------------------------------------- #
print()
print("9. the startup gate - the configurations that cannot work are refused")
File diff suppressed because it is too large Load Diff
+231 -2
View File
@@ -59,6 +59,7 @@ Read MUTATION.md before you report a survivor.
import ast
import inspect
import itertools
import os
import sys
from datetime import datetime, timedelta, timezone
@@ -272,16 +273,19 @@ check("the provenance audit REFUSES a carried hour flagged non-provisional - "
"the 'stale but plausible' point, which is SAFETY-18's frozen-but-fresh "
"meter one layer up",
raises(lambda: _fc._audit([_stale_copy], {}), ValueError))
# ⚠️ `published` MAPS AN HOUR TO A TUPLE, not to a point (FCST-04 H1). An hour
# may legitimately hold four published points - see the quarter-hourly block
# below - so the audit takes the whole set and checks membership by identity.
check("...and refuses a published hour that arrived as a COPY rather than as "
"the published object, which is where a flag gets lost",
raises(lambda: _fc._audit(
[PricePoint(start=TODAY_PART.points[0].start,
eur_per_kwh=TODAY_PART.points[0].eur_per_kwh,
provisional=TODAY_PART.points[0].provisional)],
{D0: TODAY_PART.points[0]}), ValueError))
{D0: (TODAY_PART.points[0],)}), ValueError))
check("...and never repairs a flag: the audit's only outcomes are silence and "
"an exception, so a series it passed was already correct",
_fc._audit(list(FILLED.points), {u: TODAY_PART.points[i]
_fc._audit(list(FILLED.points), {u: (TODAY_PART.points[i],)
for i, u in enumerate(DAY[:PUBLISHED_N])})
is None)
@@ -298,6 +302,231 @@ check("a PUBLISHED point that the feed itself flagged provisional keeps that "
and all(p.provisional for p in _eddy_filled.points[:2])
and _eddy_filled.points[0] is _eddy.points[0])
# ⚠️ AND A SUB-HOURLY FEED IS NOT DECIMATED TO ONE POINT AN HOUR (FCST-04 H1).
# `eneco_eddy` settles per QUARTER and `_validate_raw()` accepts any
# strictly-increasing spacing, so a published hour can hold four points. The
# fill walks WHOLE HOURS to decide what needs carrying; that must not become a
# rule about what a published hour may contain. Measured on FCST-04's first
# draft, which keyed the published set `{_utc_hour(p.start): p ...}`: a
# 96-point feed came out 15 points long and its arbitrage spread fell from
# 0.143 to 0.014 EUR/kWh, on an ALL-FIRM, fully-published series.
_QN = 16 # four hours of quarters
_QUARTERS = PriceRead.series(
[PricePoint(start=brussels(D0 + timedelta(minutes=15 * n)),
eur_per_kwh=round(0.100000 + 0.011300 * n, 6),
provisional=False)
for n in range(_QN)],
PriceProduct.REALTIME, "eneco_eddy", D0)
_QHOURS = {_fc._utc_hour(p.start) for p in _QUARTERS.points}
check(f"the quarter-hourly fixture really is sub-hourly - {_QN} points across "
f"{len(_QHOURS)} hours - or every check below passes for an hourly feed "
f"too and grades nothing",
len(_QUARTERS.points) == _QN and len(_QHOURS) == _QN // 4)
_qfill = derived(lambda: fill_from_yesterday(_QUARTERS, yesterday=YESTERDAY,
since=D0, until=D1, stamp=D0))
_qfirm = derived(lambda: [p for p in _qfill.points if not p.provisional])
check(f"every one of the {_QN} published points survives the fill BY IDENTITY "
f"({len(_qfirm)} kept). One-per-hour keying would leave {_QN // 4}, and "
f"the spread over the hours the feed actually published would collapse "
f"with them",
len(_qfirm) == _QN
and all(any(p is q for q in _QUARTERS.points) for p in _qfirm))
check(f"...and the {len(DAY) - len(_QHOURS)} whole hours it published NOTHING "
f"for are carried, ONE point each: a carried hour is hourly whatever the "
f"feed's spacing is, because yesterday's profile is only known per "
f"wall-clock hour",
_qfill.kind == "series"
and len([p for p in _qfill.points if p.provisional])
== len(DAY) - len(_QHOURS))
check("...and the filled series is still ordered by instant across the two "
"kinds, so `at()` and `window()` are not reading a shuffled list",
_qfill.kind == "series"
and [p.start.astimezone(UTC) for p in _qfill.points]
== sorted(p.start.astimezone(UTC) for p in _qfill.points))
_PUB_Q = _fc._published_by_hour(_QUARTERS)
# ⚠️ `derived`, NOT A RAW CALL. Under the decimating mutation this check
# exists for, `_audit` RAISES here - and a harness that dies on the mutation it
# is grading reports a traceback instead of a FAIL, which MUTATION.md forbids.
check("the audit ACCEPTS four published points inside one hour, which is the "
"case the repeated-hour COUNT it replaced could not express",
derived(lambda: _fc._audit(list(_QUARTERS.points), _PUB_Q)) is None)
check("...and still REFUSES a repeated CARRIED hour, which is the DST "
"mispricing that count existed to catch",
raises(lambda: _fc._audit([_fc._carried(D0, 0.11),
_fc._carried(D0, 0.12)], {}), ValueError))
check("...and STILL refuses the SAME published point emitted twice - the "
"provider's points are strictly increasing, so a repeat is never a "
"second print. The old hour COUNT already refused this (one point twice "
"is one hour twice); what is graded here is that the relaxation did not "
"lose it, not that it added it",
raises(lambda: _fc._audit([_QUARTERS.points[0], _QUARTERS.points[0]],
_PUB_Q), ValueError))
check("...and STILL refuses a CARRIED point standing in an hour the feed DID "
"publish. The old COUNT never looked at it; the old IDENTITY rule did, "
"and raised - so this is preserved behaviour, not new strictness. "
"FCST-04's text called it new here and in `_audit`'s docstring, and a "
"measured differential says otherwise",
raises(lambda: _fc._audit([_fc._carried(D0, 0.11)], _PUB_Q), ValueError))
# ⚠️ AND WHAT THE AUDIT DOES NOT DO, STATED WHERE A READER OF THESE THREE
# CHECKS WOULD OTHERWISE ASSUME IT (FCST-05, MUTATION.md habit 16). `_audit` is
# a PROVENANCE post-condition, not a COMPLETENESS one: it grades every point
# that IS emitted and says nothing about points that are not. So it accepts the
# very decimation FCST-04 was raised for. That is not a regression - FCST-03's
# audit was equally blind - but it means the audit is NOT the thing that stops
# H1 coming back. The count and identity checks above this line are.
check("the audit ACCEPTS the H1 decimation itself - four published points in "
"one hour, only the last emitted - and accepts an EMPTY output. It is a "
"provenance check, not a completeness one, and the checks that guard "
"provenance completeness are the COUNT and IDENTITY ones above",
derived(lambda: _fc._audit([_QUARTERS.points[3]], _PUB_Q)) is None
and derived(lambda: _fc._audit([], _PUB_Q)) is None)
# --------------------------------------------------------------------------- #
# ⚠️ AND THE CLAIM "STRICTER" IS ITSELF A CHECK NOW (FCST-05).
#
# FCST-04 rewrote `_audit` and wrote, in the module and in the three checks
# above, that the result was STRICTER than FCST-03's. It is not. Two of the
# stated reasons were false - the old `seen` count already refused the same
# point emitted twice, and the old IDENTITY rule already refused a carried
# point in a published hour - and a right decision with a fabricated reason is
# a defect in this repository (MUTATION.md, and the reason DECISIONS files
# exist). So the claim is measured here rather than asserted in prose:
# FCST-03's audit is reproduced below from dd40c56 - its RULES verbatim, its
# message TEXT shortened, and nothing here reads a message - and the two are
# run over the whole domain `_audit` can express, in BOTH directions.
#
# THE DOMAIN, AND ITS SIZE. Every published mask over 3 hours (2**3) crossed
# with every output series of length 0-3 drawn from 3 hours x 5 point kinds.
# The nominal cross-product is 28 928; three of those kinds - published,
# published-provisional, copy-of-published - DO NOT EXIST for an hour the feed
# did not publish, and those combinations are SKIPPED rather than faked, which
# leaves 11 990 realisable comparisons. A fabricated "published point" for an
# unpublished hour is not in either audit's domain, and inventing one would
# grade the fixture rather than the code.
#
# ⚠️ THE OLD COPY IS A LITERAL, NOT A CALL. MUTATION.md habit 2: a differential
# whose two sides both come from the shipped module compares a thing against
# itself and moves with every edit. This side is frozen text.
def _audit_fcst03(points, published):
"""FCST-03's `_audit`, verbatim from dd40c56. Its `published` maps a UTC
hour to ONE point, which is exactly the expressiveness this ticket relaxed.
"""
seen = set()
for p in points:
u = _fc._utc_hour(p.start)
if u in seen:
raise ValueError("the filled series repeats " + u.isoformat())
seen.add(u)
src = published.get(u)
if src is None:
if p.provisional is not True:
raise ValueError(u.isoformat() + " is carried and is flagged "
"provisional=" + repr(p.provisional))
else:
if p is not src:
raise ValueError(u.isoformat() + " is published but the series "
"carries a COPY of that point")
def _verdict(fn, pts, pub):
try:
fn(list(pts), pub)
return "accept"
except ValueError:
return "raise"
_DH = [D0 + timedelta(hours=6 + i) for i in range(3)]
_weaker, _stronger, _compared = [], [], 0
for _mask in itertools.product((0, 1), repeat=3):
_pub = {u: PricePoint(start=u, eur_per_kwh=0.10 + i, provisional=False)
for i, (u, _on) in enumerate(zip(_DH, _mask)) if _on}
_old_pub = dict(_pub)
_new_pub = {u: (q,) for u, q in _pub.items()}
# The five shapes an output point can take at a given hour. Three of them
# do not exist for an hour the feed did not publish, and those combinations
# are skipped rather than faked - a fabricated "published point" for an
# unpublished hour is not in either audit's domain.
_kinds = []
for _u in _DH:
_src = _pub.get(_u)
_kinds.append(("published", _src))
_kinds.append(("published-provisional",
None if _src is None else
PricePoint(start=_src.start,
eur_per_kwh=_src.eur_per_kwh,
provisional=True)))
_kinds.append(("carried", _fc._carried(_u, 0.5)))
_kinds.append(("carried-unflagged",
PricePoint(start=_u, eur_per_kwh=0.5,
provisional=False)))
_kinds.append(("copy-of-published",
None if _src is None else
PricePoint(start=_src.start,
eur_per_kwh=_src.eur_per_kwh,
provisional=_src.provisional)))
for _n in range(4):
for _combo in itertools.product(_kinds, repeat=_n):
if any(k[1] is None for k in _combo):
continue
_pts = [k[1] for k in _combo]
_compared += 1
_o = _verdict(_audit_fcst03, _pts, _old_pub)
_nw = _verdict(_fc._audit, _pts, _new_pub)
if _o == "raise" and _nw == "accept":
_weaker.append((_mask, [k[0] for k in _combo]))
if _o == "accept" and _nw == "raise":
_stronger.append((_mask, [k[0] for k in _combo]))
# ⚠️ THE PRECONDITION FIRST (MUTATION.md habit 1). A differential over a
# domain that cannot reach the two disputed cases would score 0-0 for the wrong
# reason and read exactly like this one, so the two cases are driven against
# the OLD audit by name before the totals are graded.
_probe = PricePoint(start=_DH[0], eur_per_kwh=0.1, provisional=False)
check(f"the differential really covers the two claims it is here to settle, "
f"or it is {_compared} runs of nothing: FCST-03's own audit raises on "
f"BOTH 'the same published point twice' and 'a carried point in a "
f"published hour', which is what FCST-04 said it did not",
_compared > 3000
and _verdict(_audit_fcst03, [_probe, _probe], {_DH[0]: _probe}) == "raise"
and _verdict(_audit_fcst03, [_fc._carried(_DH[0], 0.5)],
{_DH[0]: _probe}) == "raise")
check(f"...and over all {_compared} of them the reworked audit is NOT WEAKER "
f"than FCST-03's: {len(_weaker)} case(s) the old one raised and this one "
f"accepts {_weaker[:2]}",
_weaker == [])
check(f"...and NOT STRONGER either: {len(_stronger)} case(s) the old one "
f"accepted and this one raises {_stronger[:2]}. FCST-04's docstring and "
f"two of these checks said it was stricter and named two reasons, and "
f"BOTH were already refused by the old count and the old identity rule. "
f"The only behavioural change is the RELAXATION below, which is what the "
f"rework was for",
_stronger == [])
_QP = [PricePoint(start=_DH[0] + timedelta(minutes=15 * i),
eur_per_kwh=0.1 + i, provisional=False) for i in range(4)]
check("...and the relaxation is real and is NOT expressible to the old audit: "
"four distinct published points inside one hour, which FCST-03's count "
"refused as a repeated hour and this one accepts",
_verdict(_fc._audit, _QP, {_DH[0]: tuple(_QP)}) == "accept"
and _verdict(_audit_fcst03, _QP, {_DH[0]: _QP[0]}) == "raise")
# ⚠️ THE HORIZON IS WHOLE HOURS AT BOTH ENDS (FCST-04 L8). `since` is floored
# and `until` was not, so a 48 h request made at any instant off the hour came
# back 49 hours long. Graded at an OFF-THE-HOUR instant, because on the hour
# the two implementations agree and the check would grade nothing.
_off = D0 + timedelta(hours=8, minutes=17, seconds=43)
for _h in (1, 12, 48):
_n = derived(lambda h=_h: len(fill_from_yesterday(
TODAY_PART, yesterday=YESTERDAY, since=_off,
until=_off + timedelta(hours=h), stamp=_off).points))
check(f"a {_h} h horizon asked for at {_off.isoformat()} - 17 min 43 s "
f"past the hour, the shape a running controller always has - is "
f"{_h} points long, not {_h + 1}",
_n == _h)
# The structural half: the constructor has nothing to pass, and the file is
# scanned so the pass-through idiom cannot come back looking ordinary.
check("_carried() has NO `provisional` parameter, so a caller has nothing to "
File diff suppressed because it is too large Load Diff
+23 -7
View File
@@ -69,6 +69,7 @@ from app.price import (
Absence, PricePoint, PriceProduct, PriceRead, arbitrage_allowed, brussels,
plan_is_provisional, required_spread, uncertainty_of,
)
from app.price_history import NoHistory
from app.static_tariff import (
MEAN_WINDOW_DAYS, ProfileInvalid, StaticProfile, TariffPlan,
parse_static_profile, resolve_tariff, static_plan, static_status,
@@ -742,11 +743,14 @@ def run(coro):
return asyncio.run(coro)
# ⚠️ NO `price_feed_url` (TARIFF-09): an override is used verbatim and the
# plugin appends no date range to it, so setting one - even to the feed's own
# address - exercises the override path rather than the shipped default.
_cfg = opts(TWO_PERIOD, price_provider="generic_dayahead",
price_feed_url="https://api.energy-charts.info/price?bzn=BE",
price_timeout_s=1, price_markup_eur_kwh=0.03,
price_taxes_eur_kwh=0.06)
_fb = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1), transport=_dead))
_fb = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1), transport=_dead,
history=NoHistory()))
check("a dead feed falls back to the static profile rather than to nothing",
_fb.origin == "fallback" and _fb.has_prices)
check("...with the SAME prices the configured-product path produces, because "
@@ -769,15 +773,27 @@ check("...and the provisional plan has no cheap window either: §8.3 forbids an
not any(_fb.cheap_now(at_local(THU, h)) for h in range(24)))
_fb_none = run(resolve_tariff(opts(None, price_provider="generic_dayahead",
price_timeout_s=1),
THU, THU + timedelta(days=1), transport=_dead))
THU, THU + timedelta(days=1), transport=_dead,
history=NoHistory()))
check("a dead feed with NO static_profile has nothing to fall back to, and "
"says that rather than inventing a tariff",
_fb_none.origin == "none" and not _fb_none.has_prices)
# A live feed still wins over the profile.
_body = json.dumps({"data": [
{"start": (THU + timedelta(hours=i)).isoformat(), "price_eur_mwh": 40 + i}
for i in range(24)]}).encode()
# ⚠️ COLUMNAR (TARIFF-08). `api.energy-charts.info` publishes parallel
# `unix_seconds`/`price` arrays and no `data` key; the `{"data": [...]}` body
# this used to build was invented by the parser it was grading, which is how a
# provider that could never parse a live response shipped green. Real captured
# bodies are replayed in test_price.py section 2b - the prices here are
# synthetic, the SHAPE is not.
_body = json.dumps({
"unix_seconds": [int((THU + timedelta(hours=i)).timestamp())
for i in range(24)],
"price": [40 + i for i in range(24)],
"unit": "EUR / MWh",
"license_info": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) "
"from Bundesnetzagentur | SMARD.de",
"deprecated": False}).encode()
async def _live(url, timeout_s):
@@ -785,7 +801,7 @@ async def _live(url, timeout_s):
_live_plan = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1),
transport=_live))
transport=_live, history=NoHistory()))
check("a live feed is used in preference to the static profile",
_live_plan.origin == "provider"
and _live_plan.product is PriceProduct.DAY_AHEAD)
+3 -3
View File
@@ -143,7 +143,7 @@ configuration:
description: "DERIVED, not picked (SAFETY-24): the reference pack holds 13.8 kWh usable and the worst modelled peak-shaving day asks for 878 Wh, which is 6.4 %, so 20 % sits ten points above the pack's own cut-out and still leaves the guard about 1.6 of its worst observed days of room. ⚠️ IT BOUNDS THE PEAK GUARD ONLY. It is not a reserve for the whole pack: ordinary self-consumption discharges straight past it and goes on down to the battery cut-out, and raising this number does not change that by a tenth of a percent. It must stay strictly above the battery cut-out or the add-on refuses to start. Do not change it without re-deriving it."
price_provider:
name: "Tariff · Dynamic price source"
description: "Where day-ahead or real-time prices come from. 'off' is what ships, and means this site runs on the static tariff below with no trading. ⚠️ Switching it on today fetches, checks and publishes prices and changes no energy decision - nothing consumes the price series yet. It may not be combined with the two legacy Home Assistant price entities further down: two price sources on one site, one of which silently wins, is refused at startup rather than left looking armed."
description: "Where day-ahead or real-time prices come from. 'off' is what ships, and means this site runs on the static tariff below with no trading. ⚠️ SWITCHING THIS ON IS WHAT STARTS THE PRICE FETCH - not the trading switch further down. With a source selected the add-on fetches the feed every half hour, checks it, records the day and shows what it holds on the Price history row, whether or not price trading is armed. On its own it still moves no energy: trading is the separate switch, and the cheap window the monthly maintenance cycle uses comes from the fixed or day/night tariff below. It may not be combined with the two legacy Home Assistant price entities further down: two price sources on one site, one of which silently wins, is refused at startup rather than left looking armed."
price_markup_eur_kwh:
name: "Tariff · Supplier markup (EUR/kWh)"
description: "Your supplier's margin. The add-on adds it ONLY when the chosen price source does not already include it - under engie_dynamic, totalenergies_dynamic, eneco_eddy and manual_csv it is ignored, and logged as ignored, never applied twice."
@@ -161,7 +161,7 @@ configuration:
description: "For price source manual_csv only: where in /share the add-on reads your own price file from."
price_feed_url:
name: "Tariff · Price feed URL override"
description: "Leave empty to use the chosen source's own feed. Only a host that source itself declares is accepted, so this cannot be pointed anywhere."
description: "Leave empty to use the chosen source's own feed. Only a host that source itself declares is accepted, so this cannot be pointed anywhere. WHAT YOU TYPE IS SENT EXACTLY AS WRITTEN: the add-on adds nothing to it, not even the date range it normally asks its own feed for, so an override gets whatever horizon that URL returns - often a single day, with the rest of the plan carried over from yesterday and excluded from price arbitrage. A warning in the log says so on every price refresh. Clear this option to get the full horizon back."
price_horizon_h:
name: "Tariff · How far ahead to fetch prices (hours)"
description: "How many hours of prices to ask the source for."
@@ -176,7 +176,7 @@ configuration:
description: "The average the current price is compared against. Only meaningful together with the current-price entity above, and like it, may not be combined with a dynamic price source."
arbitrage_enabled:
name: "Tariff · Trade on the price spread"
description: "Charge in the cheapest published hour and discharge in the dearest, but only when the spread covers what the round trip costs. OFF in the shipped configuration, deliberately: this is the first strategy in the add-on that moves energy against the meter on purpose, so it is armed on purpose or not at all. It needs both a price source and a P1 meter, and the add-on refuses the combinations that cannot work rather than letting them look armed."
description: "Charge in the cheapest published hour and discharge in the dearest, but only when the spread covers what the round trip costs. OFF in the shipped configuration, deliberately: this is the first strategy in the add-on that moves energy against the meter on purpose, so it is armed on purpose or not at all. ⚠️ LEAVING IT OFF DOES NOT TURN THE PRICE FEED OFF. A price source selected above is fetched, checked and recorded either way; this switch decides only whether the add-on ACTS on those prices. It needs both a price source and a P1 meter, and the add-on refuses the combinations that cannot work rather than letting them look armed."
cycle_cost_threshold_eur:
name: "Tariff · Cost of one full charge/discharge cycle (EUR/kWh)"
description: "What one full round trip costs per kWh: wear on the pack plus conversion losses. A price spread smaller than this loses money and battery life, so it is not traded. ⚠️ THIS ONE IS NOT DERIVED FROM A MEASUREMENT. 0.10 is a product decision and it is still an open question with the owner (OPT-08). At this value a fixed day/night site does not trade at all - the realistic Belgian day/night spread is about 0.08 EUR/kWh - and that is the correct verdict at this number, not a fault."
+3 -3
View File
@@ -123,7 +123,7 @@ configuration:
description: "AFGELEID, niet gekozen (SAFETY-24): de referentiebatterij heeft 13,8 kWh bruikbaar en de zwaarste doorgerekende afvlakdag vraagt 878 Wh, oftewel 6,4 %, dus 20 % ligt tien punten boven de uitschakelgrens van de batterij zelf en laat de bewaking nog ongeveer 1,6 van haar zwaarste waargenomen dagen speling. ⚠️ DIT BEGRENST ALLEEN DE PIEKBEWAKING. Het is geen reserve voor de hele batterij: gewoon eigenverbruik ontlaadt er dwars doorheen en gaat door tot aan de uitschakelgrens van de batterij, en dit getal verhogen verandert daar geen tiende procent aan. Het moet strikt boven de uitschakelgrens van de batterij blijven, anders start de add-on niet op. Verander het niet zonder de afleiding opnieuw te maken."
price_provider:
name: "Tarief · Bron van dynamische prijzen"
description: "Waar day-ahead- of realtimeprijzen vandaan komen. 'off' is de standaardinstelling en betekent dat deze installatie op het vaste tarief hieronder draait, zonder handel. ⚠️ Dit vandaag aanzetten haalt prijzen op, controleert en publiceert ze, en verandert geen enkele energiebeslissing - er is nog niets dat de prijsreeks gebruikt. Het mag niet gecombineerd worden met de twee oude Home Assistant-prijsentiteiten verderop: twee prijsbronnen op één installatie, waarvan er stilzwijgend één wint, wordt bij het opstarten geweigerd in plaats van schijnbaar actief gelaten."
description: "Waar day-ahead- of realtimeprijzen vandaan komen. 'off' is de standaardinstelling en betekent dat deze installatie op het vaste tarief hieronder draait, zonder handel. ⚠️ DIT AANZETTEN IS WAT HET OPHALEN VAN PRIJZEN START - niet de handelsschakelaar verderop. Met een bron ingesteld haalt de add-on de feed elk half uur op, controleert die, bewaart de dag en toont op de rij Prijsgeschiedenis wat er bewaard is, of prijshandel nu aan staat of niet. Op zichzelf verplaatst dat nog geen energie: handelen is de aparte schakelaar, en het goedkope venster dat de maandelijkse onderhoudscyclus gebruikt komt uit het vaste of dag/nacht-tarief hieronder. Het mag niet gecombineerd worden met de twee oude Home Assistant-prijsentiteiten verderop: twee prijsbronnen op één installatie, waarvan er stilzwijgend één wint, wordt bij het opstarten geweigerd in plaats van schijnbaar actief gelaten."
price_markup_eur_kwh:
name: "Tarief · Marge van de leverancier (EUR/kWh)"
description: "De marge van uw energieleverancier. De add-on telt die ALLEEN op wanneer de gekozen prijsbron ze niet al bevat - bij engie_dynamic, totalenergies_dynamic, eneco_eddy en manual_csv wordt ze genegeerd, en dat wordt ook gelogd, zodat ze nooit dubbel wordt aangerekend."
@@ -141,7 +141,7 @@ configuration:
description: "Alleen voor prijsbron manual_csv: waar in /share de add-on uw eigen prijsbestand leest."
price_feed_url:
name: "Tarief · Afwijkende URL voor de prijsfeed"
description: "Laat leeg om de eigen feed van de gekozen bron te gebruiken. Alleen een host die die bron zelf opgeeft, wordt aanvaard, zodat dit niet zomaar ergens naar kan verwijzen."
description: "Laat leeg om de eigen feed van de gekozen bron te gebruiken. Alleen een host die die bron zelf opgeeft, wordt aanvaard, zodat dit niet zomaar ergens naar kan verwijzen. WAT U HIER TYPT WORDT LETTERLIJK VERSTUURD: de add-on voegt er niets aan toe, ook niet het datumbereik dat ze normaal aan haar eigen feed vraagt. Een afwijkende URL levert dus de horizon op die die URL zelf teruggeeft - vaak één dag, waarna de rest van het plan uit de prijzen van gisteren wordt overgenomen en buiten de prijsarbitrage valt. Bij elke prijsvernieuwing verschijnt daarover een waarschuwing in het logboek. Maak dit veld leeg om de volledige horizon terug te krijgen."
price_horizon_h:
name: "Tarief · Hoe ver vooruit prijzen ophalen (uren)"
description: "Hoeveel uur aan prijzen er bij de bron opgevraagd wordt."
@@ -156,7 +156,7 @@ configuration:
description: "Het gemiddelde waarmee de huidige prijs vergeleken wordt. Alleen zinvol samen met de entiteit hierboven en, net als die, niet te combineren met een dynamische prijsbron."
arbitrage_enabled:
name: "Tarief · Handelen op het prijsverschil"
description: "Laden in het goedkoopste gepubliceerde uur en ontladen in het duurste, maar alleen wanneer het verschil de kosten van de volledige cyclus dekt. STAAT UIT in de standaardinstelling, met opzet: dit is de eerste strategie in de add-on die bewust energie tegen de meter in verplaatst, dus ze wordt bewust aangezet of helemaal niet. Ze heeft zowel een prijsbron als een P1-meter nodig, en de add-on weigert combinaties die niet kunnen werken in plaats van ze schijnbaar actief te laten."
description: "Laden in het goedkoopste gepubliceerde uur en ontladen in het duurste, maar alleen wanneer het verschil de kosten van de volledige cyclus dekt. STAAT UIT in de standaardinstelling, met opzet: dit is de eerste strategie in de add-on die bewust energie tegen de meter in verplaatst, dus ze wordt bewust aangezet of helemaal niet. ⚠️ DIT UIT LATEN STAAN ZET DE PRIJSFEED NIET UIT. Een prijsbron die hierboven gekozen is, wordt hoe dan ook opgehaald, gecontroleerd en bewaard; deze schakelaar bepaalt alleen of de add-on op die prijzen HANDELT. Ze heeft zowel een prijsbron als een P1-meter nodig, en de add-on weigert combinaties die niet kunnen werken in plaats van ze schijnbaar actief te laten."
cycle_cost_threshold_eur:
name: "Tarief · Kostprijs van één volledige laad/ontlaadcyclus (EUR/kWh)"
description: "Wat één volledige cyclus per kWh kost: slijtage van de batterij plus omzettingsverliezen. Een prijsverschil dat hier onder ligt, kost geld en levensduur en wordt dus niet verhandeld. ⚠️ DIT GETAL IS NIET AFGELEID UIT EEN METING. 0,10 is een productbeslissing en staat nog open bij de eigenaar (OPT-08). Bij deze waarde handelt een installatie met een vast dag/nachttarief helemaal niet - het realistische Belgische dag/nachtverschil is ongeveer 0,08 EUR/kWh - en dat is bij dit getal het juiste antwoord, geen fout."