FCST-04: the carry-forward stops decimating a quarter-hourly feed, and a broken cache stops stopping the add-on
checks / unit (pull_request) Successful in 7m22s

Rework of the FCST-04 branch against review. Every finding below was proven by
a runnable script before it was touched and re-run after; every check added or
repaired was driven red by a mutation, `__pycache__` purged first (MUTATION.md,
TEST-15). 29/29 test files green.

⚠️ H1 - A SUB-HOURLY FEED WAS DECIMATED TO ONE POINT AN HOUR, AND THE SPREAD
WENT WITH IT. `fill_from_yesterday()` keyed the published set as
`{_utc_hour(p.start): p for p in r.points}` - LAST POINT OF THE HOUR WINS, the
other three dropped. `eneco_eddy` is the Belgian imbalance product and settles
per QUARTER, and `_validate_raw()` accepts any strictly-increasing spacing, so
this is a shape the product supports today. Measured on a 96-point quarter-hourly
feed - ALL FIRM, FULLY PUBLISHED, nothing to carry: the plan reached the
optimiser as 15 points, the arbitrage spread fell from 0.143 to 0.014 EUR/kWh,
and the cheapest "hour" became a `:45` point whose `at()` window runs into the
next one. Not the windowing effect this card is about; a feed losing 84 % of its
own prints, with no flag anywhere reporting it. The fill now decides only WHICH
WHOLE HOURS NEED CARRYING: every published point inside the window passes
through BY IDENTITY, and a carried point is added only for a whole hour with no
published coverage at all. Same feed, same fixture: 60 of 60 firm points kept,
spread 0.134 over the hours still ahead.

⚠️ AND `_audit()` GOT STRICTER, NOT LOOSER - that was the condition on doing it
at all. The old post-condition refused any repeated `_utc_hour`, a count that
cannot express four points in one hour. Two rules replace it and SUBSUME it: a
published point must BE one of that hour's published points by identity and may
be emitted only once; a carried point may stand only in an hour absent from the
published set, must be flagged True, and may not repeat. So the duplicated
carried hour it existed to catch still raises, the same published point emitted
twice now raises, and a carried point sharing an hour with a real print now
raises too - which the count never looked at. Three mutations, three reds.

⚠️ H2 - A DAY RECORD THAT WAS NOT A RECORD STOPPED THE ADD-ON FROM STARTING.
`_load()` validated `version` and that `days` is a dict, never that each VALUE
is one. `{"version": 1, "days": {"2026-01-14": "oops"}}` loaded as ok=True,
last_error=None, and then `rec.get(...)` raised AttributeError out of `words`,
`yesterday()` and `resolve_tariff()` alike. `PriceHistory(...)` and `.words` are
both unguarded in `Controller.__init__`, so the controller did not come up - and
it never self-healed, because `_remember()` raised on the same record before it
could reach `_save()`. That falsifies the CHANGELOG's "A BROKEN CACHE IS NOT A
BROKEN CONTROLLER" in the one direction that matters. The asymmetry that
produced it is visible in the file: `_parse_points()` is carefully defensive
about every field of a POINT and `_load()` was not defensive about the record
holding them. Unusable records are now dropped and REPORTED (`words` says how
many and why), and `words` is total by construction as a second line.

M3 - `yesterday()` WAS OFF BY ONE DAY FOR 00:00-00:59 AFTER SPRING-FORWARD.
`brussels()` renders at the FIXED offset for that instant, so `brussels(stamp) -
timedelta(days=1)` subtracts 24 ABSOLUTE hours - and the Brussels day before the
March transition is 23 hours long. At 2026-03-30T00:30+02:00 it returned
2026-03-28's prices as "yesterday", or `no_data` naming a date that is not
yesterday. The arithmetic is now on the local DATE, which has no offset to lose,
and both transitions have a check - the module's headline claim had none.

M4 - WITH NO /data THE STORE STILL FILLED, IN MEMORY. `_save()` short-circuited
on `not ok` while `_remember()` populated `self._days` anyway: `words` said
"unavailable" and `yesterday()` answered A SERIES that is gone on the next
restart. The module docstring rejects exactly that ("a yesterday that silently
forgets") as its reason for not calling makedirs, so the code now matches the
docstring rather than contradicting it.

M5 - THE PRUNE MUTATION SURVIVED BECAUSE THE STIMULUS WAS DEGENERATE. §8's
clock-jump check seeded ONE day, so `if len(self._days) <= KEEP_DAYS: return`
fired and the body it is named after never ran: pruning by arithmetic against
the clock - which empties the store on exactly the morning the fill is for -
passed the whole 29-file suite. The fixture is now written to a file and
reopened (`_load()` does not prune, which is how a store legitimately holds more
than the cap) with KEEP_DAYS + 2 days, and the mutant is red.

M6 - THE AST ORDERING CHECK COULD NOT CATCH A REVERSAL, AND TWO DOCUMENTS CITED
IT AS A GUARANTEE. `_calls.index("_remember") < _calls.index("yesterday")` grades
`ast.walk` BFS DEPTH: `self._remember(...)` is a statement and
`self.yesterday(...)` is an ARGUMENT of the fill, one level deeper, so
`_remember` came out first whichever statement was written first - shipped and
reversed both scored True. It is now line numbers AND, the half that matters,
the behaviour: a feed carrying yesterday alongside today into an EMPTY store
must fill the whole horizon; reversed, the lookup finds nothing and the plan is
only the published hours. Reversing the two statements is red on both.

L14/L9 - KEEP_DAYS = 3 IS THE STEADY STATE EXACTLY, so a newest-N cap has no
slack: a feed publishing to D+2, or ONE stray point in the wrong year - which
sorts LAST and can never age out - pushes yesterday out for good. `_prune()` now
PINS the two days a lookup can actually ask for on top of the cap. Pinning is
additive, so it can only RETAIN and the WITNESS-03 property (nothing is deleted
by clock arithmetic) is preserved by construction; `stamp` stops being a dead
parameter; the store is bounded at KEEP_DAYS + 2.

L8 - the horizon end was not floored while `since` was, so a 48 h request made
at any instant off the hour returned 49 points. §1's count was green only
because its NOW was exactly :00:00, a shape production never has; the fixture
now sits at :17:43. L10 - a failed `_save()` leaked its temp file; at 48 writes
a day a persistent write failure leaked 48 inodes a day onto the volume that was
already the problem. L15 - `{"version": 1.0}` passed, since `1.0 == 1`. L16 -
the `_remember` tree walk scanned `app/*.py` only and matched a bare string; it
now parses `app/`, `tools/`, `scenarios/` and `estop/` and looks for the NAME,
so a comment does not fail it and a call in `tools/` cannot pass unseen.

CLAIM-ACCURACY DEFECTS, WHICH THIS REPO TREATS AS REAL:

* L7. DOCS said "*Tariff* on the status page reads `provisional` through the
  morning". FALSE - `tariff_words` is assigned only from the HA price-entity
  branch and from `static_status()`, both on the 30 s tick, and the plan's own
  sentence reaches `_LOG.info`. The doc now says what an operator can see (the
  Price history row, the Arbitrage row's own refusal sentence, the refresh log
  line) and a check pins the PAIRING - wiring the plan into the row is fine,
  correcting the doc is fine, doing one without the other is red.
* L11. "fills EVERY unpublished hour" is conditional: a partial yesterday leaves
  the local hours it never held out of the series (§11.4 for those hours, and
  `at()` raises rather than answering). Correct behaviour, overclaimed wording,
  and untested - all three fixed.
* L12. DECISIONS §1 leaned on `TariffPlan.cheap_now()`, which has NO PRODUCTION
  CALLER on the provider plan (`self.cheap` comes from `static_status()`, which
  builds its own). The conclusion holds on `arbitrage()`'s firm-pair refusal
  alone, which does reach control; the evidence is now stated at the reach it
  has.
* L13. §1's "passed through by identity" check compared `eur_per_kwh` between
  two SEPARATE `resolve_tariff()` calls, where `is` could never have held. The
  value comparison is kept and now says what it grades; identity is graded on a
  read the file constructs and still holds.
* L17. `PriceHistory.words` and the new status row had no check at all - and
  `words` is one of the two things H2 crashed. Both now have one.
* L18. main.py's inline note carried the "widens what the optimiser can SEE,
  never what it may act on" guarantee without the windowing caveat that the
  CHANGELOG, DOCS and `resolve_tariff()`'s docstring all carry. It is the one
  place a maintainer reads it unqualified.

EVIDENCE. All eight reviewer proofs re-run: p1/p1b return 29 March, p4 keeps
60 of 60 firm points at spread 0.134, p5 starts and reports, p6 D leaves no
temp, p6 E refuses 1.0, p6 F records nothing, p7 returns 48 for 48. p2's script
still scores True/True because it re-implements the OLD algorithm - the
reversal is now driven as a mutation instead, and is red. p3's script always
showed the shipped code correct; what was wrong was the fixture, and its mutant
is now red. Twenty mutations driven, no survivors, tree fingerprinted and
`__pycache__` purged before each: decimation (2 suites red), audit x3, unfloored
`until` (2), unvalidated day records (7 checks), instant-subtraction yesterday,
in-memory fill on a volume-less store, clock-arithmetic prune, cap without the
pin, leaked temp, loose schema type, empty `words` (5), reversed `carry()` (2),
a `_remember` leak in tools/ spelled two ways, a neighbour-filling fallback, a
copy instead of identity (2), the DOCS claim in both directions, and the caveat
removed from main.py. Baseline and the unmutated tree both green.

No option added, config.yaml untouched, and the CHANGELOG entry still picks no
version: PKG-07's rule, and only the deployer knows whether 0.9.0's image has
been pulled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
This commit is contained in:
glenn schrooyen
2026-09-09 23:54:23 +02:00
co-authored by Claude Opus 5
parent 0618194d9c
commit b59f5ce31b
9 changed files with 1222 additions and 112 deletions
@@ -1,6 +1,6 @@
# FCST-04 — what "yesterday" is, and what the fill is allowed to reach
Three decisions, recorded because each had a defensible alternative that a later
Four decisions, recorded because each had a defensible alternative that a later
reader would otherwise re-open from scratch.
---
@@ -17,11 +17,23 @@ no arbitrage."*
Because it buys nothing the optimiser can use, and it silently replaces a
fallback the operator configured.
Measured, not assumed: a fully-carried series is 100 % provisional, so
`TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on §8.3's
firm-pair rung, and `cheap_now()` returns False because `firm_points()` is
empty. The §11.4 static fallback produces *exactly* those two answers. The two
paths differ only in which numbers the status page shows.
Measured, not assumed — and stated at the reach it actually has:
- **This one reaches control.** A fully-carried series is 100 % provisional, so
`TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on
§8.3's firm-pair rung. That is the verdict the optimiser reads: `arbitrage.py`
calls `plan.arbitrage(...)` on `fresh_tariff_plan()`, and it is the only path
by which a provider plan moves energy. The §11.4 static fallback refuses on
the same rung, for the same reason.
- **This one does not, and saying so is the point.** `TariffPlan.cheap_now()`
also returns False on a fully-carried series, because `firm_points()` is
empty — but it has **no production caller on the provider plan**.
`self.cheap` comes from `static_status()`, which builds its *own* static plan
and calls `cheap_now()` on that. So it is a property of the type, checked in
`test_static_tariff.py`, and it is not evidence about the shipped control
path. The conclusion below holds on the first bullet alone.
The two paths therefore differ only in which numbers the status page shows.
So the choice is between two identical control behaviours, and the tie-breaker
is that `static_profile` is a documented option an operator fills in **for this
@@ -74,9 +86,26 @@ The defence is therefore an **ordering**, not a validation:
- The filled series exists only as that return value. The object never sees it
again.
- `_remember()` is private and called from exactly one place.
- `test_price_history.py` walks `app/` and fails if any other module names
`_remember`, and reads `carry()`'s AST to confirm the record happens before
the lookup — so a later edit that reverses them is red rather than plausible.
- `test_price_history.py` parses every shipped module under `app/`, `tools/`,
`scenarios/` and `estop/` and fails if any of them **uses the name**
`_remember` — an attribute, a bare name, a `def`, or the string constant a
`getattr` would need. By name rather than by substring, so a comment that
merely mentions the rule does not fail it and a call in `tools/` cannot pass
unseen.
- and it pins the order two ways: the **line numbers** of the two calls in
`carry()`, and — the one that matters — the **behaviour**. A feed carrying
yesterday alongside today, into an empty store, must fill the whole horizon;
reversed, the lookup finds nothing and the plan is only the hours the feed
published. Driven, so a reversal is red.
⚠️ **The first version of that ordering check graded nothing, and this file
said it did.** It compared positions in `ast.walk`, which is breadth-first:
`self._remember(...)` is a statement and `self.yesterday(...)` is an *argument*
of `fill_from_yesterday(...)`, one level deeper, so `_remember` came out first
whichever statement was written first. Measured — shipped and reversed both
scored the same. A decorative check that a document cites as a guarantee is
worse than no check, which is why the correction is recorded here rather than
quietly made.
That ordering has a second benefit worth keeping: a feed that returns yesterday
alongside today refreshes yesterday's record from live data *before* it is
@@ -96,3 +125,57 @@ elapsed are no longer in the spread. A cheap night that has gone used to be the
series' cheapest hour, and a cycle cannot buy in an hour that is past — so the
trade was silently never taken. `test_price_history.py` section 2 grades that by
finding the cheapest firm hour with and without the fill and showing it moves.
⚠️ **And section 1's identity check now grades identity.** It used to compare
`eur_per_kwh` between `_before` and `_after`, which come from two *separate*
`resolve_tariff()` calls — each parsing the feed body into its own objects, so
`is` could never have held between them however the fill were written. FCST-03's
whole anti-copy argument is `filled.points[i] is fetched.points[j]`, and floats
cannot grade it. The value comparison is kept and now says what it is (the fill
moved the window, it did not touch a price); identity is graded on a read the
file constructs and still holds a reference to.
---
## 4. A sub-hourly feed passes through whole; the fill only decides which hours are missing.
`eneco_eddy` is the Belgian imbalance product and settles **per quarter**, and
`price.py`'s `_validate_raw()` accepts any strictly-increasing spacing — so a
published hour may legitimately hold four points.
FCST-04's first caller keyed the published set as `{_utc_hour(p.start): p for p
in r.points}`. It reads perfectly ordinary and it keeps the **last** point of
each hour. Measured on a 96-point quarter-hourly feed — all firm, fully
published, nothing to carry: the plan reached the optimiser as **15 points**,
the arbitrage spread fell from **0.143 to 0.014 EUR/kWh**, and the cheapest
"hour" became a `:45` point whose `at()` window runs into the next hour. That is
not the windowing effect above; it is a feed losing 84 % of its own prints, and
no `provisional` flag would ever have reported it.
**The alternative was rejected:** refusing sub-hourly feeds at `_validate_raw()`
would have made the count correct by breaking the one supported product that
legitimately publishes per quarter.
**What was done instead** keeps FCST-03's existing contract rather than weakening
it: *every* published point inside the window passes through **by identity**,
and a carried point is added only for a whole hour with **no** published
coverage at all. A carried hour is still a single hourly point whatever the
feed's own spacing is, because yesterday's profile is only known per wall-clock
hour — and carried hours are provisional, so they are out of the trade set
anyway.
⚠️ **`_audit()` got stricter, not looser, and that was the condition on doing
this at all.** The old post-condition refused any repeated `_utc_hour` — a count
that cannot express four points in one hour. It is replaced by two rules that
**subsume** it:
- a **published** point must *be* one of that hour's published points, by
identity, and may be emitted only once;
- a **carried** point may stand only in an hour that does not appear in the
published set at all, must be flagged `True`, and may not repeat.
So a duplicated carried hour — the DST mispricing the count existed to catch —
still raises; so does the same published point emitted twice; and a carried
point sharing an hour with a real print now raises too, which the count never
looked at. Each of those three is driven by its own mutation in
`test_price_forecast.py`.