FCST-04: the carry-forward fill reaches the plan, and the day it carries survives a restart #73

Merged
admin merged 2 commits from FCST-04 into release/1.0 2026-09-10 00:03:09 +02:00
Owner

FCST-03 shipped fill_from_yesterday() complete and checked, together with the
consumers written for it — TariffPlan.firm_points(), cheap_now()'s refusal of
a provisional hour, and arbitrage()'s refusal 3, which drops provisional hours
from the trade set rather than refusing the whole series. It shipped no
caller.
Nothing in app/ named the fill, so no series ever contained a carried
point and every one of those consumers was exercising its all-firm branch.

⚠️ The defect that produced was not an error anywhere. load_prices()
returns what the feed published and nothing widened it to the requested horizon,
so on a dynamic site between midnight and the 14–16 h CET publication the plan
covered the rest of today and nothing more — measured on this PR's fixture, 24 h
against a 48 h price_horizon_h request
. arbitrage() then took max - min
over that truncated window and reported a spread. The number was real, the window
was half the one configured, and nothing said so: tomorrow evening's peak — the
hour a morning charge would be sold into — was not in the series to be compared
against anything. A missing hour is not a wrong price; it is an hour the
optimiser never learns exists.

What this adds

  • app/price_history.py keeps the published prices of the last KEEP_DAYS
    Brussels days in /data/price_history.json.
  • resolve_tariff() gains a required keyword-only history and fills every
    unpublished hour of the horizon from yesterday's same-wall-clock hour, flagged.
  • A new status row, Price history, names the days held and their hour counts.

Persistence rather than a variable on the refresh task, because the window the
fill exists for is midnight → publication, and an add-on restarted at 08:00 would
otherwise spend all of it on the truncated series. Not in state.json: the
refresh runs every 1800 s and that file holds the maintenance history and
SAFETY-02's latch.

⚠️ One change here moves energy, and it is not the one the card is named for

Carried hours are provisional, so §8.3 keeps them out of the trade set: the fill
widens what the optimiser can see, never what it may do — and
test_price_history.py checks that, so it cannot be overclaimed later.

What does change a trade is that the filled series is windowed to
[since, until), so hours that have already elapsed are no longer in the
spread
. Before, a cheap night that had gone sat in the series as the day's
cheapest hour — and a cycle cannot buy in an hour that is past, so the trade was
quietly never taken. Section 2 grades this by finding the cheapest firm hour with
and without the fill and showing it moves, from 02:00 UTC to 08:00 UTC on the
fixture.

⚠️ A dead feed is still §11.4, not yesterday

The fill runs only on a series read. The two are indistinguishable to the
optimiser — both 100 % provisional, so arbitrage refuses on the firm-pair rung and
cheap_now() is False — so preferring yesterday's feed over the static_profile
an operator configured for that case would change a documented degradation for
a difference only the status page can see. That, and two more, in
DECISIONS/FCST-04-the-day-that-is-carried.md.

⚠️ The anti-decay rule is an ordering, not a validation

Feeding the filled series back in is how this rots — day 2 carries day 1's
guesses, day 3 carries day 2's carry of them, behind a provisional flag that
stays true while the number decays. No value-level check can catch it: a
carried point and eneco_eddy's published-provisional quarter are the same three
fields with the same flag. So carry() is the only public entry, it records
before it fills, the filled series exists only as a return value the object
never sees again, and test_price_history.py walks app/ for any other module
naming _remember and reads carry()'s AST to pin the order.

Evidence

29/29 test files pass in the CI image (goodwe-addon-checks:1), including the
new test_price_history.py.

Eight mutations driven against app/price_history.py and
app/static_tariff.py, __pycache__ purged before each run per MUTATION.md /
TEST-15:

mutation result
yesterday() falls back to the newest record (rule 2) RED (5)
carry() records the filled series — the decay (rule 1) RED (1)
_remember() replaces a day instead of merging (rule 4) RED (10)
_bdate() keys on the UTC date RED (10)
resolve_tariff() stops filling — the pre-FCST-04 state RED (12)
a lost price field defaults to 0.0 RED (2)
prune by clock arithmetic, not sort order RED (1)
the degraded reason stops naming its cause RED (1)

No survivors; baseline and revert both green. Three of those first crashed the
harness instead of reporting a FAIL
— "it was refused" and "it broke on the way
to being refused" must not grade the same — so why() and _priced() were added
and the run repeated.

Not in this PR

  • No option added; config.yaml is untouched.
  • The CHANGELOG entry deliberately picks no version. PKG-07's rule: the
    Supervisor keys the built image by version:, and only the deployer knows
    whether 0.9.0's image has been built and pulled. If it has not, this belongs
    inside 0.9.0's section; if it has, it needs 0.10.0.
  • fill_from_yesterday()'s no_data branch is therefore not reached from
    production. It stays checked in test_price_forecast.py; turning it on is a
    product decision, recorded in the DECISIONS file.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU

FCST-03 shipped `fill_from_yesterday()` complete and checked, together with the consumers written for it — `TariffPlan.firm_points()`, `cheap_now()`'s refusal of a provisional hour, and `arbitrage()`'s refusal 3, which drops provisional hours from the trade set rather than refusing the whole series. **It shipped no caller.** Nothing in `app/` named the fill, so no series ever contained a carried point and every one of those consumers was exercising its all-firm branch. ⚠️ **The defect that produced was not an error anywhere.** `load_prices()` returns what the feed published and nothing widened it to the requested horizon, so on a dynamic site between midnight and the 14–16 h CET publication the plan covered the rest of today and nothing more — measured on this PR's fixture, **24 h against a 48 h `price_horizon_h` request**. `arbitrage()` then took `max - min` over that truncated window and reported a spread. The number was real, the window was half the one configured, and nothing said so: tomorrow evening's peak — the hour a morning charge would be sold into — was not in the series to be compared against anything. A missing hour is not a wrong price; it is an hour the optimiser never learns exists. ## What this adds - **`app/price_history.py`** keeps the *published* prices of the last `KEEP_DAYS` Brussels days in `/data/price_history.json`. - **`resolve_tariff()` gains a required keyword-only `history`** and fills every unpublished hour of the horizon from yesterday's same-wall-clock hour, flagged. - A new status row, **Price history**, names the days held and their hour counts. Persistence rather than a variable on the refresh task, because the window the fill exists for is midnight → publication, and an add-on restarted at 08:00 would otherwise spend all of it on the truncated series. **Not** in `state.json`: the refresh runs every 1800 s and that file holds the maintenance history and SAFETY-02's latch. ## ⚠️ One change here moves energy, and it is not the one the card is named for Carried hours are provisional, so §8.3 keeps them out of the trade set: the fill widens what the optimiser can **see**, never what it may **do** — and `test_price_history.py` checks that, so it cannot be overclaimed later. What *does* change a trade is that the filled series is windowed to `[since, until)`, so **hours that have already elapsed are no longer in the spread**. Before, a cheap night that had gone sat in the series as the day's cheapest hour — and a cycle cannot buy in an hour that is past, so the trade was quietly never taken. Section 2 grades this by finding the cheapest firm hour with and without the fill and showing it **moves**, from 02:00 UTC to 08:00 UTC on the fixture. ## ⚠️ A dead feed is still §11.4, not yesterday The fill runs only on a `series` read. The two are indistinguishable to the optimiser — both 100 % provisional, so arbitrage refuses on the firm-pair rung and `cheap_now()` is False — so preferring yesterday's feed over the `static_profile` an operator configured **for that case** would change a documented degradation for a difference only the status page can see. That, and two more, in `DECISIONS/FCST-04-the-day-that-is-carried.md`. ## ⚠️ The anti-decay rule is an ordering, not a validation Feeding the filled series back in is how this rots — day 2 carries day 1's guesses, day 3 carries day 2's carry of them, behind a `provisional` flag that stays **true** while the **number** decays. No value-level check can catch it: a carried point and `eneco_eddy`'s published-provisional quarter are the same three fields with the same flag. So `carry()` is the only public entry, it records **before** it fills, the filled series exists only as a return value the object never sees again, and `test_price_history.py` walks `app/` for any other module naming `_remember` and reads `carry()`'s AST to pin the order. ## Evidence **29/29 test files pass** in the CI image (`goodwe-addon-checks:1`), including the new `test_price_history.py`. **Eight mutations driven** against `app/price_history.py` and `app/static_tariff.py`, `__pycache__` purged before each run per MUTATION.md / TEST-15: | mutation | result | |---|---| | `yesterday()` falls back to the newest record (rule 2) | RED (5) | | `carry()` records the filled series — the decay (rule 1) | RED (1) | | `_remember()` replaces a day instead of merging (rule 4) | RED (10) | | `_bdate()` keys on the UTC date | RED (10) | | `resolve_tariff()` stops filling — the pre-FCST-04 state | RED (12) | | a lost price field defaults to `0.0` | RED (2) | | prune by clock arithmetic, not sort order | RED (1) | | the degraded reason stops naming its cause | RED (1) | No survivors; baseline and revert both green. **Three of those first crashed the harness instead of reporting a FAIL** — "it was refused" and "it broke on the way to being refused" must not grade the same — so `why()` and `_priced()` were added and the run repeated. ## Not in this PR - **No option added; `config.yaml` is untouched.** - The CHANGELOG entry deliberately **picks no version**. PKG-07's rule: the Supervisor keys the built image by `version:`, and only the deployer knows whether 0.9.0's image has been built and pulled. If it has not, this belongs inside 0.9.0's section; if it has, it needs 0.10.0. - `fill_from_yesterday()`'s `no_data` branch is therefore not reached from production. It stays checked in `test_price_forecast.py`; turning it on is a product decision, recorded in the DECISIONS file. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
admin added 1 commit 2026-09-09 22:39:36 +02:00
FCST-03 shipped `fill_from_yesterday()` complete and checked, and shipped the
consumers written for it - `TariffPlan.firm_points()`, `cheap_now()`'s refusal
of a provisional hour, and `arbitrage()`'s refusal 3, which drops provisional
hours from the trade set rather than refusing the whole series. It shipped NO
CALLER. Nothing in `app/` named the fill, so no series ever contained a carried
point and every one of those consumers was exercising its all-firm branch.

⚠️ THE DEFECT THAT PRODUCED WAS NOT AN ERROR ANYWHERE. `load_prices()` returns
what the feed published and nothing widened it to the requested horizon, so on
a dynamic site between midnight and the 14-16 h CET publication the plan
covered the rest of today and nothing more - measured on the fixture here, 24 h
against a 48 h request. `arbitrage()` then took `max - min` over that truncated
window and reported a spread. The number was real, the window was half the one
configured, and nothing said so: tomorrow evening's peak, the hour a morning
charge would be sold into, was not in the series to be compared against
anything. A missing hour is not a wrong price - it is an hour the optimiser
never learns exists.

WHAT THIS ADDS. `app/price_history.py` keeps the PUBLISHED prices of the last
KEEP_DAYS Brussels days in /data/price_history.json; `resolve_tariff()` gains a
required keyword-only `history` and fills every unpublished hour of the horizon
from yesterday's same-wall-clock hour, flagged. Persistence rather than a
variable on the refresh task, because the window the fill exists for is
midnight -> publication and an add-on restarted at 08:00 would otherwise spend
all of it on the truncated series. Not in state.json: the refresh runs every
1800 s and that file holds the maintenance history and SAFETY-02's latch.

⚠️ ONE CHANGE HERE MOVES ENERGY, AND IT IS NOT THE ONE THE CARD IS NAMED FOR.
Carried hours are provisional, so §8.3 keeps them out of the trade set: the
fill widens what the optimiser can SEE, never what it may DO, and
test_price_history.py checks that so it cannot be overclaimed later. What does
change a trade is that the filled series is windowed to [since, until), so
hours that have ALREADY ELAPSED are no longer in the spread. Before, a cheap
night that had gone sat in the series as the day's cheapest hour - and a cycle
cannot buy in an hour that is past, so the trade was quietly never taken.
Section 2 grades this by finding the cheapest firm hour with and without the
fill and showing it MOVES, from 02:00 UTC to 08:00 UTC on the fixture.

⚠️ A DEAD FEED IS STILL §11.4, NOT YESTERDAY. The fill runs only on a `series`
read. The two are indistinguishable to the optimiser - both 100 % provisional,
so arbitrage refuses on the firm-pair rung and `cheap_now()` is False - so
preferring yesterday's feed over the `static_profile` an operator configured
FOR that case would change a documented degradation for a difference only the
status page can see. That, and two more, in
DECISIONS/FCST-04-the-day-that-is-carried.md.

⚠️ THE ANTI-DECAY RULE IS AN ORDERING, NOT A VALIDATION, and saying so is more
useful than a guarantee that would not hold. Feeding the filled series back in
is how this rots - day 2 carries day 1's guesses, day 3 carries day 2's carry
of them, behind a `provisional` flag that stays TRUE while the NUMBER decays.
No value-level check can catch it: a carried point and `eneco_eddy`'s
published-provisional quarter are the same three fields with the same flag. So
`carry()` is the only public entry, it records BEFORE it fills, the filled
series exists only as a return value the object never sees again, and
test_price_history.py walks app/ for any other module naming `_remember` and
reads `carry()`'s AST to pin the order.

EVIDENCE. 29/29 test files pass in the CI image. Eight mutations driven against
app/price_history.py and app/static_tariff.py, __pycache__ purged before each
run per MUTATION.md/TEST-15: newest-record lookup (5 red), recording the filled
series (1), replace-instead-of-merge (10), UTC-date keying (10), removing the
fill call entirely (12), a lost price field defaulting to 0.0 (2), prune by
clock arithmetic (1), a degraded reason that stops naming its cause (1). No
survivors; baseline and revert both green. Three of those first crashed the
harness instead of reporting a FAIL - "it was refused" and "it broke on the way
to being refused" must not grade the same - so `why()` and `_priced()` were
added and the run repeated.

No option added, config.yaml untouched, and the CHANGELOG entry deliberately
picks no version: PKG-07's rule, and only the deployer knows whether 0.9.0's
image has been pulled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
admin added 1 commit 2026-09-09 23:54:31 +02:00
Rework of the FCST-04 branch against review. Every finding below was proven by
a runnable script before it was touched and re-run after; every check added or
repaired was driven red by a mutation, `__pycache__` purged first (MUTATION.md,
TEST-15). 29/29 test files green.

⚠️ H1 - A SUB-HOURLY FEED WAS DECIMATED TO ONE POINT AN HOUR, AND THE SPREAD
WENT WITH IT. `fill_from_yesterday()` keyed the published set as
`{_utc_hour(p.start): p for p in r.points}` - LAST POINT OF THE HOUR WINS, the
other three dropped. `eneco_eddy` is the Belgian imbalance product and settles
per QUARTER, and `_validate_raw()` accepts any strictly-increasing spacing, so
this is a shape the product supports today. Measured on a 96-point quarter-hourly
feed - ALL FIRM, FULLY PUBLISHED, nothing to carry: the plan reached the
optimiser as 15 points, the arbitrage spread fell from 0.143 to 0.014 EUR/kWh,
and the cheapest "hour" became a `:45` point whose `at()` window runs into the
next one. Not the windowing effect this card is about; a feed losing 84 % of its
own prints, with no flag anywhere reporting it. The fill now decides only WHICH
WHOLE HOURS NEED CARRYING: every published point inside the window passes
through BY IDENTITY, and a carried point is added only for a whole hour with no
published coverage at all. Same feed, same fixture: 60 of 60 firm points kept,
spread 0.134 over the hours still ahead.

⚠️ AND `_audit()` GOT STRICTER, NOT LOOSER - that was the condition on doing it
at all. The old post-condition refused any repeated `_utc_hour`, a count that
cannot express four points in one hour. Two rules replace it and SUBSUME it: a
published point must BE one of that hour's published points by identity and may
be emitted only once; a carried point may stand only in an hour absent from the
published set, must be flagged True, and may not repeat. So the duplicated
carried hour it existed to catch still raises, the same published point emitted
twice now raises, and a carried point sharing an hour with a real print now
raises too - which the count never looked at. Three mutations, three reds.

⚠️ H2 - A DAY RECORD THAT WAS NOT A RECORD STOPPED THE ADD-ON FROM STARTING.
`_load()` validated `version` and that `days` is a dict, never that each VALUE
is one. `{"version": 1, "days": {"2026-01-14": "oops"}}` loaded as ok=True,
last_error=None, and then `rec.get(...)` raised AttributeError out of `words`,
`yesterday()` and `resolve_tariff()` alike. `PriceHistory(...)` and `.words` are
both unguarded in `Controller.__init__`, so the controller did not come up - and
it never self-healed, because `_remember()` raised on the same record before it
could reach `_save()`. That falsifies the CHANGELOG's "A BROKEN CACHE IS NOT A
BROKEN CONTROLLER" in the one direction that matters. The asymmetry that
produced it is visible in the file: `_parse_points()` is carefully defensive
about every field of a POINT and `_load()` was not defensive about the record
holding them. Unusable records are now dropped and REPORTED (`words` says how
many and why), and `words` is total by construction as a second line.

M3 - `yesterday()` WAS OFF BY ONE DAY FOR 00:00-00:59 AFTER SPRING-FORWARD.
`brussels()` renders at the FIXED offset for that instant, so `brussels(stamp) -
timedelta(days=1)` subtracts 24 ABSOLUTE hours - and the Brussels day before the
March transition is 23 hours long. At 2026-03-30T00:30+02:00 it returned
2026-03-28's prices as "yesterday", or `no_data` naming a date that is not
yesterday. The arithmetic is now on the local DATE, which has no offset to lose,
and both transitions have a check - the module's headline claim had none.

M4 - WITH NO /data THE STORE STILL FILLED, IN MEMORY. `_save()` short-circuited
on `not ok` while `_remember()` populated `self._days` anyway: `words` said
"unavailable" and `yesterday()` answered A SERIES that is gone on the next
restart. The module docstring rejects exactly that ("a yesterday that silently
forgets") as its reason for not calling makedirs, so the code now matches the
docstring rather than contradicting it.

M5 - THE PRUNE MUTATION SURVIVED BECAUSE THE STIMULUS WAS DEGENERATE. §8's
clock-jump check seeded ONE day, so `if len(self._days) <= KEEP_DAYS: return`
fired and the body it is named after never ran: pruning by arithmetic against
the clock - which empties the store on exactly the morning the fill is for -
passed the whole 29-file suite. The fixture is now written to a file and
reopened (`_load()` does not prune, which is how a store legitimately holds more
than the cap) with KEEP_DAYS + 2 days, and the mutant is red.

M6 - THE AST ORDERING CHECK COULD NOT CATCH A REVERSAL, AND TWO DOCUMENTS CITED
IT AS A GUARANTEE. `_calls.index("_remember") < _calls.index("yesterday")` grades
`ast.walk` BFS DEPTH: `self._remember(...)` is a statement and
`self.yesterday(...)` is an ARGUMENT of the fill, one level deeper, so
`_remember` came out first whichever statement was written first - shipped and
reversed both scored True. It is now line numbers AND, the half that matters,
the behaviour: a feed carrying yesterday alongside today into an EMPTY store
must fill the whole horizon; reversed, the lookup finds nothing and the plan is
only the published hours. Reversing the two statements is red on both.

L14/L9 - KEEP_DAYS = 3 IS THE STEADY STATE EXACTLY, so a newest-N cap has no
slack: a feed publishing to D+2, or ONE stray point in the wrong year - which
sorts LAST and can never age out - pushes yesterday out for good. `_prune()` now
PINS the two days a lookup can actually ask for on top of the cap. Pinning is
additive, so it can only RETAIN and the WITNESS-03 property (nothing is deleted
by clock arithmetic) is preserved by construction; `stamp` stops being a dead
parameter; the store is bounded at KEEP_DAYS + 2.

L8 - the horizon end was not floored while `since` was, so a 48 h request made
at any instant off the hour returned 49 points. §1's count was green only
because its NOW was exactly :00:00, a shape production never has; the fixture
now sits at :17:43. L10 - a failed `_save()` leaked its temp file; at 48 writes
a day a persistent write failure leaked 48 inodes a day onto the volume that was
already the problem. L15 - `{"version": 1.0}` passed, since `1.0 == 1`. L16 -
the `_remember` tree walk scanned `app/*.py` only and matched a bare string; it
now parses `app/`, `tools/`, `scenarios/` and `estop/` and looks for the NAME,
so a comment does not fail it and a call in `tools/` cannot pass unseen.

CLAIM-ACCURACY DEFECTS, WHICH THIS REPO TREATS AS REAL:

* L7. DOCS said "*Tariff* on the status page reads `provisional` through the
  morning". FALSE - `tariff_words` is assigned only from the HA price-entity
  branch and from `static_status()`, both on the 30 s tick, and the plan's own
  sentence reaches `_LOG.info`. The doc now says what an operator can see (the
  Price history row, the Arbitrage row's own refusal sentence, the refresh log
  line) and a check pins the PAIRING - wiring the plan into the row is fine,
  correcting the doc is fine, doing one without the other is red.
* L11. "fills EVERY unpublished hour" is conditional: a partial yesterday leaves
  the local hours it never held out of the series (§11.4 for those hours, and
  `at()` raises rather than answering). Correct behaviour, overclaimed wording,
  and untested - all three fixed.
* L12. DECISIONS §1 leaned on `TariffPlan.cheap_now()`, which has NO PRODUCTION
  CALLER on the provider plan (`self.cheap` comes from `static_status()`, which
  builds its own). The conclusion holds on `arbitrage()`'s firm-pair refusal
  alone, which does reach control; the evidence is now stated at the reach it
  has.
* L13. §1's "passed through by identity" check compared `eur_per_kwh` between
  two SEPARATE `resolve_tariff()` calls, where `is` could never have held. The
  value comparison is kept and now says what it grades; identity is graded on a
  read the file constructs and still holds.
* L17. `PriceHistory.words` and the new status row had no check at all - and
  `words` is one of the two things H2 crashed. Both now have one.
* L18. main.py's inline note carried the "widens what the optimiser can SEE,
  never what it may act on" guarantee without the windowing caveat that the
  CHANGELOG, DOCS and `resolve_tariff()`'s docstring all carry. It is the one
  place a maintainer reads it unqualified.

EVIDENCE. All eight reviewer proofs re-run: p1/p1b return 29 March, p4 keeps
60 of 60 firm points at spread 0.134, p5 starts and reports, p6 D leaves no
temp, p6 E refuses 1.0, p6 F records nothing, p7 returns 48 for 48. p2's script
still scores True/True because it re-implements the OLD algorithm - the
reversal is now driven as a mutation instead, and is red. p3's script always
showed the shipped code correct; what was wrong was the fixture, and its mutant
is now red. Twenty mutations driven, no survivors, tree fingerprinted and
`__pycache__` purged before each: decimation (2 suites red), audit x3, unfloored
`until` (2), unvalidated day records (7 checks), instant-subtraction yesterday,
in-memory fill on a volume-less store, clock-arithmetic prune, cap without the
pin, leaked temp, loose schema type, empty `words` (5), reversed `carry()` (2),
a `_remember` leak in tools/ spelled two ways, a neighbour-filling fallback, a
copy instead of identity (2), the DOCS claim in both directions, and the caveat
removed from main.py. Baseline and the unmutated tree both green.

No option added, config.yaml untouched, and the CHANGELOG entry still picks no
version: PKG-07's rule, and only the deployer knows whether 0.9.0's image has
been pulled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
admin merged commit c2787da539 into release/1.0 2026-09-10 00:03:09 +02:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/goodwe-addon#73