FCST-04: the carry-forward stops decimating a quarter-hourly feed, and a broken cache stops stopping the add-on
checks / unit (pull_request) Successful in 7m22s
checks / unit (pull_request) Successful in 7m22s
Rework of the FCST-04 branch against review. Every finding below was proven by a runnable script before it was touched and re-run after; every check added or repaired was driven red by a mutation, `__pycache__` purged first (MUTATION.md, TEST-15). 29/29 test files green. ⚠️ H1 - A SUB-HOURLY FEED WAS DECIMATED TO ONE POINT AN HOUR, AND THE SPREAD WENT WITH IT. `fill_from_yesterday()` keyed the published set as `{_utc_hour(p.start): p for p in r.points}` - LAST POINT OF THE HOUR WINS, the other three dropped. `eneco_eddy` is the Belgian imbalance product and settles per QUARTER, and `_validate_raw()` accepts any strictly-increasing spacing, so this is a shape the product supports today. Measured on a 96-point quarter-hourly feed - ALL FIRM, FULLY PUBLISHED, nothing to carry: the plan reached the optimiser as 15 points, the arbitrage spread fell from 0.143 to 0.014 EUR/kWh, and the cheapest "hour" became a `:45` point whose `at()` window runs into the next one. Not the windowing effect this card is about; a feed losing 84 % of its own prints, with no flag anywhere reporting it. The fill now decides only WHICH WHOLE HOURS NEED CARRYING: every published point inside the window passes through BY IDENTITY, and a carried point is added only for a whole hour with no published coverage at all. Same feed, same fixture: 60 of 60 firm points kept, spread 0.134 over the hours still ahead. ⚠️ AND `_audit()` GOT STRICTER, NOT LOOSER - that was the condition on doing it at all. The old post-condition refused any repeated `_utc_hour`, a count that cannot express four points in one hour. Two rules replace it and SUBSUME it: a published point must BE one of that hour's published points by identity and may be emitted only once; a carried point may stand only in an hour absent from the published set, must be flagged True, and may not repeat. So the duplicated carried hour it existed to catch still raises, the same published point emitted twice now raises, and a carried point sharing an hour with a real print now raises too - which the count never looked at. Three mutations, three reds. ⚠️ H2 - A DAY RECORD THAT WAS NOT A RECORD STOPPED THE ADD-ON FROM STARTING. `_load()` validated `version` and that `days` is a dict, never that each VALUE is one. `{"version": 1, "days": {"2026-01-14": "oops"}}` loaded as ok=True, last_error=None, and then `rec.get(...)` raised AttributeError out of `words`, `yesterday()` and `resolve_tariff()` alike. `PriceHistory(...)` and `.words` are both unguarded in `Controller.__init__`, so the controller did not come up - and it never self-healed, because `_remember()` raised on the same record before it could reach `_save()`. That falsifies the CHANGELOG's "A BROKEN CACHE IS NOT A BROKEN CONTROLLER" in the one direction that matters. The asymmetry that produced it is visible in the file: `_parse_points()` is carefully defensive about every field of a POINT and `_load()` was not defensive about the record holding them. Unusable records are now dropped and REPORTED (`words` says how many and why), and `words` is total by construction as a second line. M3 - `yesterday()` WAS OFF BY ONE DAY FOR 00:00-00:59 AFTER SPRING-FORWARD. `brussels()` renders at the FIXED offset for that instant, so `brussels(stamp) - timedelta(days=1)` subtracts 24 ABSOLUTE hours - and the Brussels day before the March transition is 23 hours long. At 2026-03-30T00:30+02:00 it returned 2026-03-28's prices as "yesterday", or `no_data` naming a date that is not yesterday. The arithmetic is now on the local DATE, which has no offset to lose, and both transitions have a check - the module's headline claim had none. M4 - WITH NO /data THE STORE STILL FILLED, IN MEMORY. `_save()` short-circuited on `not ok` while `_remember()` populated `self._days` anyway: `words` said "unavailable" and `yesterday()` answered A SERIES that is gone on the next restart. The module docstring rejects exactly that ("a yesterday that silently forgets") as its reason for not calling makedirs, so the code now matches the docstring rather than contradicting it. M5 - THE PRUNE MUTATION SURVIVED BECAUSE THE STIMULUS WAS DEGENERATE. §8's clock-jump check seeded ONE day, so `if len(self._days) <= KEEP_DAYS: return` fired and the body it is named after never ran: pruning by arithmetic against the clock - which empties the store on exactly the morning the fill is for - passed the whole 29-file suite. The fixture is now written to a file and reopened (`_load()` does not prune, which is how a store legitimately holds more than the cap) with KEEP_DAYS + 2 days, and the mutant is red. M6 - THE AST ORDERING CHECK COULD NOT CATCH A REVERSAL, AND TWO DOCUMENTS CITED IT AS A GUARANTEE. `_calls.index("_remember") < _calls.index("yesterday")` grades `ast.walk` BFS DEPTH: `self._remember(...)` is a statement and `self.yesterday(...)` is an ARGUMENT of the fill, one level deeper, so `_remember` came out first whichever statement was written first - shipped and reversed both scored True. It is now line numbers AND, the half that matters, the behaviour: a feed carrying yesterday alongside today into an EMPTY store must fill the whole horizon; reversed, the lookup finds nothing and the plan is only the published hours. Reversing the two statements is red on both. L14/L9 - KEEP_DAYS = 3 IS THE STEADY STATE EXACTLY, so a newest-N cap has no slack: a feed publishing to D+2, or ONE stray point in the wrong year - which sorts LAST and can never age out - pushes yesterday out for good. `_prune()` now PINS the two days a lookup can actually ask for on top of the cap. Pinning is additive, so it can only RETAIN and the WITNESS-03 property (nothing is deleted by clock arithmetic) is preserved by construction; `stamp` stops being a dead parameter; the store is bounded at KEEP_DAYS + 2. L8 - the horizon end was not floored while `since` was, so a 48 h request made at any instant off the hour returned 49 points. §1's count was green only because its NOW was exactly :00:00, a shape production never has; the fixture now sits at :17:43. L10 - a failed `_save()` leaked its temp file; at 48 writes a day a persistent write failure leaked 48 inodes a day onto the volume that was already the problem. L15 - `{"version": 1.0}` passed, since `1.0 == 1`. L16 - the `_remember` tree walk scanned `app/*.py` only and matched a bare string; it now parses `app/`, `tools/`, `scenarios/` and `estop/` and looks for the NAME, so a comment does not fail it and a call in `tools/` cannot pass unseen. CLAIM-ACCURACY DEFECTS, WHICH THIS REPO TREATS AS REAL: * L7. DOCS said "*Tariff* on the status page reads `provisional` through the morning". FALSE - `tariff_words` is assigned only from the HA price-entity branch and from `static_status()`, both on the 30 s tick, and the plan's own sentence reaches `_LOG.info`. The doc now says what an operator can see (the Price history row, the Arbitrage row's own refusal sentence, the refresh log line) and a check pins the PAIRING - wiring the plan into the row is fine, correcting the doc is fine, doing one without the other is red. * L11. "fills EVERY unpublished hour" is conditional: a partial yesterday leaves the local hours it never held out of the series (§11.4 for those hours, and `at()` raises rather than answering). Correct behaviour, overclaimed wording, and untested - all three fixed. * L12. DECISIONS §1 leaned on `TariffPlan.cheap_now()`, which has NO PRODUCTION CALLER on the provider plan (`self.cheap` comes from `static_status()`, which builds its own). The conclusion holds on `arbitrage()`'s firm-pair refusal alone, which does reach control; the evidence is now stated at the reach it has. * L13. §1's "passed through by identity" check compared `eur_per_kwh` between two SEPARATE `resolve_tariff()` calls, where `is` could never have held. The value comparison is kept and now says what it grades; identity is graded on a read the file constructs and still holds. * L17. `PriceHistory.words` and the new status row had no check at all - and `words` is one of the two things H2 crashed. Both now have one. * L18. main.py's inline note carried the "widens what the optimiser can SEE, never what it may act on" guarantee without the windowing caveat that the CHANGELOG, DOCS and `resolve_tariff()`'s docstring all carry. It is the one place a maintainer reads it unqualified. EVIDENCE. All eight reviewer proofs re-run: p1/p1b return 29 March, p4 keeps 60 of 60 firm points at spread 0.134, p5 starts and reports, p6 D leaves no temp, p6 E refuses 1.0, p6 F records nothing, p7 returns 48 for 48. p2's script still scores True/True because it re-implements the OLD algorithm - the reversal is now driven as a mutation instead, and is red. p3's script always showed the shipped code correct; what was wrong was the fixture, and its mutant is now red. Twenty mutations driven, no survivors, tree fingerprinted and `__pycache__` purged before each: decimation (2 suites red), audit x3, unfloored `until` (2), unvalidated day records (7 checks), instant-subtraction yesterday, in-memory fill on a volume-less store, clock-arithmetic prune, cap without the pin, leaked temp, loose schema type, empty `words` (5), reversed `carry()` (2), a `_remember` leak in tools/ spelled two ways, a neighbour-filling fallback, a copy instead of identity (2), the DOCS claim in both directions, and the caveat removed from main.py. Baseline and the unmutated tree both green. No option added, config.yaml untouched, and the CHANGELOG entry still picks no version: PKG-07's rule, and only the deployer knows whether 0.9.0's image has been pulled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
This commit is contained in:
co-authored by
Claude Opus 5
parent
0618194d9c
commit
b59f5ce31b
@@ -1433,18 +1433,41 @@ day. Until FCST-04 those other hours were simply **absent** from the plan: not
|
||||
priced badly, not flagged, just missing, and nothing said so.
|
||||
|
||||
Now the add-on keeps the **published prices of the last few days** in
|
||||
`/data/price_history.json` and fills every unpublished hour of the horizon with
|
||||
`/data/price_history.json` and fills an unpublished hour of the horizon with
|
||||
yesterday's price for the same Brussels wall-clock hour, flagged
|
||||
`provisional: true`. Two things follow, and the second one matters more:
|
||||
|
||||
- **The plan says it is guessing.** *Tariff* on the status page reads
|
||||
`provisional` through the morning, because half its hours are. Before, a
|
||||
half-length plan looked completely settled.
|
||||
- **The plan says it is guessing** — `provisional: true` on every carried hour,
|
||||
which is what the optimiser reads. Before, a half-length plan looked
|
||||
completely settled.
|
||||
- **Nothing may trade on a carried hour.** §8.3 forbids acting on a provisional
|
||||
price, so arbitrage drops those hours from the trade set entirely — it needs a
|
||||
buy hour and a sell hour that were *both* published. Carrying a price forward
|
||||
widens what the optimiser can **see**; it never widens what it may **do**.
|
||||
|
||||
**"An unpublished hour", not "every unpublished hour"** — the fill can only
|
||||
carry an hour yesterday actually published. If yesterday's record is itself
|
||||
partial (a first day, an outage, a feed that only ever serves the rest of the
|
||||
day), the local hours it never held are **left out of the series**: §11.4
|
||||
applies to those hours, and asking the plan for one raises rather than
|
||||
answering. An hour with no price is not a free hour, and the fill does not
|
||||
invent a neighbour, a mean or a zero for it.
|
||||
|
||||
**A quarter-hourly feed keeps all four of its prints.** `eneco_eddy` settles
|
||||
per quarter, and the fill decides *which whole hours need carrying* — never what
|
||||
a published hour may contain. Every point the provider published inside the
|
||||
horizon is passed through untouched; a carried point is added only for a whole
|
||||
hour the feed did not print at all.
|
||||
|
||||
**What you can actually see on the status page.** *Price history* (below) says
|
||||
what is held. *Arbitrage* carries the strategy's own sentence, which names the
|
||||
firm/provisional split whenever that split is what refused a trade. *Tariff*
|
||||
does **not** report the fill: it is the static-profile or Home Assistant
|
||||
price-entity sentence, regenerated on the 30 s tick. The provider plan's own
|
||||
line — `dynamic_dayahead from generic_dayahead, 48 h, provisional` — goes to the
|
||||
**add-on log** at every refresh, alongside `§8.3: 33 of 48 hours carried forward
|
||||
from yesterday's profile and flagged provisional; 15 published`.
|
||||
|
||||
The plan is also now cut to the horizon you asked for, which means **hours that
|
||||
have already elapsed are no longer in it**. That is a real change to arbitrage:
|
||||
a cheap night that has already gone used to sit in the series as the cheapest
|
||||
@@ -1458,6 +1481,13 @@ cache that cannot answer is not allowed to be a controller that cannot regulate.
|
||||
The store is rebuilt by the next successful fetch; on a brand-new install the
|
||||
first fill is possible the day after the first refresh.
|
||||
|
||||
The same applies to a file that is *partly* broken: a day record this build
|
||||
cannot read is dropped, the row says how many were dropped and why, and the rest
|
||||
of the file still loads. And where there is no `/data` at all the store records
|
||||
**nothing** — not even in memory. An in-memory yesterday would answer for one
|
||||
run and vanish on the next restart, which is a cache that forgets without saying
|
||||
so; `unavailable` and *no prices carried* are the honest pair.
|
||||
|
||||
**Injection is priced separately** (`export_price_eur_kwh`). In Belgium it is
|
||||
often near zero, which by itself strongly favours self-consumption over export.
|
||||
It is a single number rather than a curve because no supported supplier
|
||||
|
||||
Reference in New Issue
Block a user