FCST-04: the carry-forward stops decimating a quarter-hourly feed, and a broken cache stops stopping the add-on
checks / unit (pull_request) Successful in 7m22s
checks / unit (pull_request) Successful in 7m22s
Rework of the FCST-04 branch against review. Every finding below was proven by a runnable script before it was touched and re-run after; every check added or repaired was driven red by a mutation, `__pycache__` purged first (MUTATION.md, TEST-15). 29/29 test files green. ⚠️ H1 - A SUB-HOURLY FEED WAS DECIMATED TO ONE POINT AN HOUR, AND THE SPREAD WENT WITH IT. `fill_from_yesterday()` keyed the published set as `{_utc_hour(p.start): p for p in r.points}` - LAST POINT OF THE HOUR WINS, the other three dropped. `eneco_eddy` is the Belgian imbalance product and settles per QUARTER, and `_validate_raw()` accepts any strictly-increasing spacing, so this is a shape the product supports today. Measured on a 96-point quarter-hourly feed - ALL FIRM, FULLY PUBLISHED, nothing to carry: the plan reached the optimiser as 15 points, the arbitrage spread fell from 0.143 to 0.014 EUR/kWh, and the cheapest "hour" became a `:45` point whose `at()` window runs into the next one. Not the windowing effect this card is about; a feed losing 84 % of its own prints, with no flag anywhere reporting it. The fill now decides only WHICH WHOLE HOURS NEED CARRYING: every published point inside the window passes through BY IDENTITY, and a carried point is added only for a whole hour with no published coverage at all. Same feed, same fixture: 60 of 60 firm points kept, spread 0.134 over the hours still ahead. ⚠️ AND `_audit()` GOT STRICTER, NOT LOOSER - that was the condition on doing it at all. The old post-condition refused any repeated `_utc_hour`, a count that cannot express four points in one hour. Two rules replace it and SUBSUME it: a published point must BE one of that hour's published points by identity and may be emitted only once; a carried point may stand only in an hour absent from the published set, must be flagged True, and may not repeat. So the duplicated carried hour it existed to catch still raises, the same published point emitted twice now raises, and a carried point sharing an hour with a real print now raises too - which the count never looked at. Three mutations, three reds. ⚠️ H2 - A DAY RECORD THAT WAS NOT A RECORD STOPPED THE ADD-ON FROM STARTING. `_load()` validated `version` and that `days` is a dict, never that each VALUE is one. `{"version": 1, "days": {"2026-01-14": "oops"}}` loaded as ok=True, last_error=None, and then `rec.get(...)` raised AttributeError out of `words`, `yesterday()` and `resolve_tariff()` alike. `PriceHistory(...)` and `.words` are both unguarded in `Controller.__init__`, so the controller did not come up - and it never self-healed, because `_remember()` raised on the same record before it could reach `_save()`. That falsifies the CHANGELOG's "A BROKEN CACHE IS NOT A BROKEN CONTROLLER" in the one direction that matters. The asymmetry that produced it is visible in the file: `_parse_points()` is carefully defensive about every field of a POINT and `_load()` was not defensive about the record holding them. Unusable records are now dropped and REPORTED (`words` says how many and why), and `words` is total by construction as a second line. M3 - `yesterday()` WAS OFF BY ONE DAY FOR 00:00-00:59 AFTER SPRING-FORWARD. `brussels()` renders at the FIXED offset for that instant, so `brussels(stamp) - timedelta(days=1)` subtracts 24 ABSOLUTE hours - and the Brussels day before the March transition is 23 hours long. At 2026-03-30T00:30+02:00 it returned 2026-03-28's prices as "yesterday", or `no_data` naming a date that is not yesterday. The arithmetic is now on the local DATE, which has no offset to lose, and both transitions have a check - the module's headline claim had none. M4 - WITH NO /data THE STORE STILL FILLED, IN MEMORY. `_save()` short-circuited on `not ok` while `_remember()` populated `self._days` anyway: `words` said "unavailable" and `yesterday()` answered A SERIES that is gone on the next restart. The module docstring rejects exactly that ("a yesterday that silently forgets") as its reason for not calling makedirs, so the code now matches the docstring rather than contradicting it. M5 - THE PRUNE MUTATION SURVIVED BECAUSE THE STIMULUS WAS DEGENERATE. §8's clock-jump check seeded ONE day, so `if len(self._days) <= KEEP_DAYS: return` fired and the body it is named after never ran: pruning by arithmetic against the clock - which empties the store on exactly the morning the fill is for - passed the whole 29-file suite. The fixture is now written to a file and reopened (`_load()` does not prune, which is how a store legitimately holds more than the cap) with KEEP_DAYS + 2 days, and the mutant is red. M6 - THE AST ORDERING CHECK COULD NOT CATCH A REVERSAL, AND TWO DOCUMENTS CITED IT AS A GUARANTEE. `_calls.index("_remember") < _calls.index("yesterday")` grades `ast.walk` BFS DEPTH: `self._remember(...)` is a statement and `self.yesterday(...)` is an ARGUMENT of the fill, one level deeper, so `_remember` came out first whichever statement was written first - shipped and reversed both scored True. It is now line numbers AND, the half that matters, the behaviour: a feed carrying yesterday alongside today into an EMPTY store must fill the whole horizon; reversed, the lookup finds nothing and the plan is only the published hours. Reversing the two statements is red on both. L14/L9 - KEEP_DAYS = 3 IS THE STEADY STATE EXACTLY, so a newest-N cap has no slack: a feed publishing to D+2, or ONE stray point in the wrong year - which sorts LAST and can never age out - pushes yesterday out for good. `_prune()` now PINS the two days a lookup can actually ask for on top of the cap. Pinning is additive, so it can only RETAIN and the WITNESS-03 property (nothing is deleted by clock arithmetic) is preserved by construction; `stamp` stops being a dead parameter; the store is bounded at KEEP_DAYS + 2. L8 - the horizon end was not floored while `since` was, so a 48 h request made at any instant off the hour returned 49 points. §1's count was green only because its NOW was exactly :00:00, a shape production never has; the fixture now sits at :17:43. L10 - a failed `_save()` leaked its temp file; at 48 writes a day a persistent write failure leaked 48 inodes a day onto the volume that was already the problem. L15 - `{"version": 1.0}` passed, since `1.0 == 1`. L16 - the `_remember` tree walk scanned `app/*.py` only and matched a bare string; it now parses `app/`, `tools/`, `scenarios/` and `estop/` and looks for the NAME, so a comment does not fail it and a call in `tools/` cannot pass unseen. CLAIM-ACCURACY DEFECTS, WHICH THIS REPO TREATS AS REAL: * L7. DOCS said "*Tariff* on the status page reads `provisional` through the morning". FALSE - `tariff_words` is assigned only from the HA price-entity branch and from `static_status()`, both on the 30 s tick, and the plan's own sentence reaches `_LOG.info`. The doc now says what an operator can see (the Price history row, the Arbitrage row's own refusal sentence, the refresh log line) and a check pins the PAIRING - wiring the plan into the row is fine, correcting the doc is fine, doing one without the other is red. * L11. "fills EVERY unpublished hour" is conditional: a partial yesterday leaves the local hours it never held out of the series (§11.4 for those hours, and `at()` raises rather than answering). Correct behaviour, overclaimed wording, and untested - all three fixed. * L12. DECISIONS §1 leaned on `TariffPlan.cheap_now()`, which has NO PRODUCTION CALLER on the provider plan (`self.cheap` comes from `static_status()`, which builds its own). The conclusion holds on `arbitrage()`'s firm-pair refusal alone, which does reach control; the evidence is now stated at the reach it has. * L13. §1's "passed through by identity" check compared `eur_per_kwh` between two SEPARATE `resolve_tariff()` calls, where `is` could never have held. The value comparison is kept and now says what it grades; identity is graded on a read the file constructs and still holds. * L17. `PriceHistory.words` and the new status row had no check at all - and `words` is one of the two things H2 crashed. Both now have one. * L18. main.py's inline note carried the "widens what the optimiser can SEE, never what it may act on" guarantee without the windowing caveat that the CHANGELOG, DOCS and `resolve_tariff()`'s docstring all carry. It is the one place a maintainer reads it unqualified. EVIDENCE. All eight reviewer proofs re-run: p1/p1b return 29 March, p4 keeps 60 of 60 firm points at spread 0.134, p5 starts and reports, p6 D leaves no temp, p6 E refuses 1.0, p6 F records nothing, p7 returns 48 for 48. p2's script still scores True/True because it re-implements the OLD algorithm - the reversal is now driven as a mutation instead, and is red. p3's script always showed the shipped code correct; what was wrong was the fixture, and its mutant is now red. Twenty mutations driven, no survivors, tree fingerprinted and `__pycache__` purged before each: decimation (2 suites red), audit x3, unfloored `until` (2), unvalidated day records (7 checks), instant-subtraction yesterday, in-memory fill on a volume-less store, clock-arithmetic prune, cap without the pin, leaked temp, loose schema type, empty `words` (5), reversed `carry()` (2), a `_remember` leak in tools/ spelled two ways, a neighbour-filling fallback, a copy instead of identity (2), the DOCS claim in both directions, and the caveat removed from main.py. Baseline and the unmutated tree both green. No option added, config.yaml untouched, and the CHANGELOG entry still picks no version: PKG-07's rule, and only the deployer knows whether 0.9.0's image has been pulled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
This commit is contained in:
co-authored by
Claude Opus 5
parent
0618194d9c
commit
b59f5ce31b
@@ -1,6 +1,6 @@
|
||||
# FCST-04 — what "yesterday" is, and what the fill is allowed to reach
|
||||
|
||||
Three decisions, recorded because each had a defensible alternative that a later
|
||||
Four decisions, recorded because each had a defensible alternative that a later
|
||||
reader would otherwise re-open from scratch.
|
||||
|
||||
---
|
||||
@@ -17,11 +17,23 @@ no arbitrage."*
|
||||
Because it buys nothing the optimiser can use, and it silently replaces a
|
||||
fallback the operator configured.
|
||||
|
||||
Measured, not assumed: a fully-carried series is 100 % provisional, so
|
||||
`TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on §8.3's
|
||||
firm-pair rung, and `cheap_now()` returns False because `firm_points()` is
|
||||
empty. The §11.4 static fallback produces *exactly* those two answers. The two
|
||||
paths differ only in which numbers the status page shows.
|
||||
Measured, not assumed — and stated at the reach it actually has:
|
||||
|
||||
- **This one reaches control.** A fully-carried series is 100 % provisional, so
|
||||
`TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on
|
||||
§8.3's firm-pair rung. That is the verdict the optimiser reads: `arbitrage.py`
|
||||
calls `plan.arbitrage(...)` on `fresh_tariff_plan()`, and it is the only path
|
||||
by which a provider plan moves energy. The §11.4 static fallback refuses on
|
||||
the same rung, for the same reason.
|
||||
- **This one does not, and saying so is the point.** `TariffPlan.cheap_now()`
|
||||
also returns False on a fully-carried series, because `firm_points()` is
|
||||
empty — but it has **no production caller on the provider plan**.
|
||||
`self.cheap` comes from `static_status()`, which builds its *own* static plan
|
||||
and calls `cheap_now()` on that. So it is a property of the type, checked in
|
||||
`test_static_tariff.py`, and it is not evidence about the shipped control
|
||||
path. The conclusion below holds on the first bullet alone.
|
||||
|
||||
The two paths therefore differ only in which numbers the status page shows.
|
||||
|
||||
So the choice is between two identical control behaviours, and the tie-breaker
|
||||
is that `static_profile` is a documented option an operator fills in **for this
|
||||
@@ -74,9 +86,26 @@ The defence is therefore an **ordering**, not a validation:
|
||||
- The filled series exists only as that return value. The object never sees it
|
||||
again.
|
||||
- `_remember()` is private and called from exactly one place.
|
||||
- `test_price_history.py` walks `app/` and fails if any other module names
|
||||
`_remember`, and reads `carry()`'s AST to confirm the record happens before
|
||||
the lookup — so a later edit that reverses them is red rather than plausible.
|
||||
- `test_price_history.py` parses every shipped module under `app/`, `tools/`,
|
||||
`scenarios/` and `estop/` and fails if any of them **uses the name**
|
||||
`_remember` — an attribute, a bare name, a `def`, or the string constant a
|
||||
`getattr` would need. By name rather than by substring, so a comment that
|
||||
merely mentions the rule does not fail it and a call in `tools/` cannot pass
|
||||
unseen.
|
||||
- and it pins the order two ways: the **line numbers** of the two calls in
|
||||
`carry()`, and — the one that matters — the **behaviour**. A feed carrying
|
||||
yesterday alongside today, into an empty store, must fill the whole horizon;
|
||||
reversed, the lookup finds nothing and the plan is only the hours the feed
|
||||
published. Driven, so a reversal is red.
|
||||
|
||||
⚠️ **The first version of that ordering check graded nothing, and this file
|
||||
said it did.** It compared positions in `ast.walk`, which is breadth-first:
|
||||
`self._remember(...)` is a statement and `self.yesterday(...)` is an *argument*
|
||||
of `fill_from_yesterday(...)`, one level deeper, so `_remember` came out first
|
||||
whichever statement was written first. Measured — shipped and reversed both
|
||||
scored the same. A decorative check that a document cites as a guarantee is
|
||||
worse than no check, which is why the correction is recorded here rather than
|
||||
quietly made.
|
||||
|
||||
That ordering has a second benefit worth keeping: a feed that returns yesterday
|
||||
alongside today refreshes yesterday's record from live data *before* it is
|
||||
@@ -96,3 +125,57 @@ elapsed are no longer in the spread. A cheap night that has gone used to be the
|
||||
series' cheapest hour, and a cycle cannot buy in an hour that is past — so the
|
||||
trade was silently never taken. `test_price_history.py` section 2 grades that by
|
||||
finding the cheapest firm hour with and without the fill and showing it moves.
|
||||
|
||||
⚠️ **And section 1's identity check now grades identity.** It used to compare
|
||||
`eur_per_kwh` between `_before` and `_after`, which come from two *separate*
|
||||
`resolve_tariff()` calls — each parsing the feed body into its own objects, so
|
||||
`is` could never have held between them however the fill were written. FCST-03's
|
||||
whole anti-copy argument is `filled.points[i] is fetched.points[j]`, and floats
|
||||
cannot grade it. The value comparison is kept and now says what it is (the fill
|
||||
moved the window, it did not touch a price); identity is graded on a read the
|
||||
file constructs and still holds a reference to.
|
||||
|
||||
---
|
||||
|
||||
## 4. A sub-hourly feed passes through whole; the fill only decides which hours are missing.
|
||||
|
||||
`eneco_eddy` is the Belgian imbalance product and settles **per quarter**, and
|
||||
`price.py`'s `_validate_raw()` accepts any strictly-increasing spacing — so a
|
||||
published hour may legitimately hold four points.
|
||||
|
||||
FCST-04's first caller keyed the published set as `{_utc_hour(p.start): p for p
|
||||
in r.points}`. It reads perfectly ordinary and it keeps the **last** point of
|
||||
each hour. Measured on a 96-point quarter-hourly feed — all firm, fully
|
||||
published, nothing to carry: the plan reached the optimiser as **15 points**,
|
||||
the arbitrage spread fell from **0.143 to 0.014 EUR/kWh**, and the cheapest
|
||||
"hour" became a `:45` point whose `at()` window runs into the next hour. That is
|
||||
not the windowing effect above; it is a feed losing 84 % of its own prints, and
|
||||
no `provisional` flag would ever have reported it.
|
||||
|
||||
**The alternative was rejected:** refusing sub-hourly feeds at `_validate_raw()`
|
||||
would have made the count correct by breaking the one supported product that
|
||||
legitimately publishes per quarter.
|
||||
|
||||
**What was done instead** keeps FCST-03's existing contract rather than weakening
|
||||
it: *every* published point inside the window passes through **by identity**,
|
||||
and a carried point is added only for a whole hour with **no** published
|
||||
coverage at all. A carried hour is still a single hourly point whatever the
|
||||
feed's own spacing is, because yesterday's profile is only known per wall-clock
|
||||
hour — and carried hours are provisional, so they are out of the trade set
|
||||
anyway.
|
||||
|
||||
⚠️ **`_audit()` got stricter, not looser, and that was the condition on doing
|
||||
this at all.** The old post-condition refused any repeated `_utc_hour` — a count
|
||||
that cannot express four points in one hour. It is replaced by two rules that
|
||||
**subsume** it:
|
||||
|
||||
- a **published** point must *be* one of that hour's published points, by
|
||||
identity, and may be emitted only once;
|
||||
- a **carried** point may stand only in an hour that does not appear in the
|
||||
published set at all, must be flagged `True`, and may not repeat.
|
||||
|
||||
So a duplicated carried hour — the DST mispricing the count existed to catch —
|
||||
still raises; so does the same published point emitted twice; and a carried
|
||||
point sharing an hour with a real print now raises too, which the count never
|
||||
looked at. Each of those three is driven by its own mutation in
|
||||
`test_price_forecast.py`.
|
||||
|
||||
Reference in New Issue
Block a user