goodwe-addon/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md

FCST-04 — what "yesterday" is, and what the fill is allowed to reach

Four decisions, recorded because each had a defensible alternative that a later reader would otherwise re-open from scratch.

---

1. The fill runs on a series read only. A dead feed still goes to §11.4.

fill_from_yesterday() accepts a no_data read and will happily build a 100 %-carried series from yesterday alone. resolve_tariff() does not let it: a feed that published nothing still falls to _degrade() and §11.4's own row — "No price data → fall back to the static profile, mark the plan provisional, no arbitrage."

Why not prefer yesterday's real prices to a coarse day/night profile? Because it buys nothing the optimiser can use, and it silently replaces a fallback the operator configured.

Measured, not assumed — and stated at the reach it actually has:

• This one reaches control. A fully-carried series is 100 % provisional, so TariffPlan.arbitrage() finds fewer than two firm hours and refuses on §8.3's firm-pair rung. That is the verdict the optimiser reads: arbitrage.py calls plan.arbitrage(...) on fresh_tariff_plan(), and it is the only path by which a provider plan moves energy. The §11.4 static fallback refuses on the same rung, for the same reason.
• This one does not, and saying so is the point. TariffPlan.cheap_now() also returns False on a fully-carried series, because firm_points() is empty — but it has no production caller on the provider plan. self.cheap comes from static_status(), which builds its own static plan and calls cheap_now() on that. So it is a property of the type, checked in test_static_tariff.py, and it is not evidence about the shipped control path. The conclusion below holds on the first bullet alone.

The two paths therefore differ only in which numbers the status page shows.

So the choice is between two identical control behaviours, and the tie-breaker is that static_profile is a documented option an operator fills in for this case. Overriding it for a cosmetic difference is a behaviour change nobody asked for.

⚠️ The consequence, stated so it is not discovered later as a bug: fill_from_yesterday()'s no_data branch is not reached from production. It stays checked in test_price_forecast.py. Turning it on is a product decision — it would want a rule about how old a carried profile may be before it stops being offered at all — and not a wiring one.

---

2. "Yesterday" is a date, not the newest record.

newest() is the obvious accessor and it is the one that decays. An add-on down from Friday to Monday would answer Friday's prices to a fill whose only vocabulary is provisional: true — a flag that says "this is a guess", never "this guess is three days old". Nothing downstream can tell the difference and nothing reports it.

So yesterday(stamp) computes the Brussels date before stamp's Brussels date and looks up that record. Anything else is no_data with a reason naming both the date wanted and the dates held.

The cost is real and accepted: after an outage of more than a day, the first morning back has no fill. That is the honest state — there is no yesterday — and the horizon is short for one morning rather than wrong for several.

---

3. What is recorded is the provider's read, and the mechanism is ordering.

Feeding the filled series back into the store is how this rots: day 2 carries day 1's guesses, day 3 carries day 2's carry of them, and after a week of a dead feed the "yesterday" being carried is a fossil of the last real publication. Every point still carries provisional=True, so the flag never lies — what decays is the number, behind a flag that is telling the truth.

⚠️ No value-level check can catch that, and claiming one would be worse than claiming none. A carried point and a published-provisional point (a real-time product's current interval, §9.2) are the same three fields carrying the same flag. The difference is provenance, and provenance is not in the value.

The defence is therefore an ordering, not a validation:

• carry() is the only public entry point. It records its argument, then returns the fill.
• The filled series exists only as that return value. The object never sees it again.
• _remember() is private and called from exactly one place.
• test_price_history.py parses every shipped module under app/, tools/, scenarios/ and estop/ and fails if any of them uses the name _remember — an attribute, a bare name, a def, or the string constant a getattr would need. By name rather than by substring, so a comment that merely mentions the rule does not fail it and a call in tools/ cannot pass unseen.
• and it pins the order two ways: the line numbers of the two calls in carry(), and — the one that matters — the behaviour. A feed carrying yesterday alongside today, into an empty store, must fill the whole horizon; reversed, the lookup finds nothing and the plan is only the hours the feed published. Driven, so a reversal is red.

⚠️ The first version of that ordering check graded nothing, and this file said it did. It compared positions in ast.walk, which is breadth-first: self._remember(...) is a statement and self.yesterday(...) is an argument of fill_from_yesterday(...), one level deeper, so _remember came out first whichever statement was written first. Measured — shipped and reversed both scored the same. A decorative check that a document cites as a guarantee is worse than no check, which is why the correction is recorded here rather than quietly made.

That ordering has a second benefit worth keeping: a feed that returns yesterday alongside today refreshes yesterday's record from live data before it is consumed as the profile.

---

What this ticket does not claim

It does not make a trade possible that yesterday's prices justify. Carried hours are dropped from the trade set by §8.3, and that is not a limitation to be fixed later — it is the rule.

The one change that does move energy came from the windowing, not the carrying: the filled series covers [since, until), so hours that have already elapsed are no longer in the spread. A cheap night that has gone used to be the series' cheapest hour, and a cycle cannot buy in an hour that is past — so the trade was silently never taken. test_price_history.py section 2 grades that by finding the cheapest firm hour with and without the fill and showing it moves.

⚠️ And section 1's identity check now grades identity. It used to compare eur_per_kwh between _before and _after, which come from two separate resolve_tariff() calls — each parsing the feed body into its own objects, so is could never have held between them however the fill were written. FCST-03's whole anti-copy argument is filled.points[i] is fetched.points[j], and floats cannot grade it. The value comparison is kept and now says what it is (the fill moved the window, it did not touch a price); identity is graded on a read the file constructs and still holds a reference to.

---

4. A sub-hourly feed passes through whole; the fill only decides which hours are missing.

⚠️ This section originally said eneco_eddy is "the Belgian imbalance product". That was false and TARIFF-07 withdrew the plugin: Eneco's dynamic product is "Zon & Wind Dynamisch" and is indexed to Belpex/EPEX day-ahead, not to imbalance, and api.eneco.be/eddy/prices never existed. The mechanism below is unaffected, because the real quarter-hourly feed is the one still shipped.

Belgian day-ahead settles per quarter — the 15-minute MTU took effect on 2025-10-01, so generic_dayahead reads 96 points a day rather than 24 — and price.py's _validate_raw() accepts any strictly-increasing spacing, so a published hour may legitimately hold four points.

FCST-04's first caller keyed the published set as {_utc_hour(p.start): p for p in r.points}. It reads perfectly ordinary and it keeps the last point of each hour. Measured on a 96-point quarter-hourly feed — all firm, fully published, nothing to carry: the plan reached the optimiser as 15 points, the arbitrage spread fell from 0.143 to 0.014 EUR/kWh, and the cheapest "hour" became a :45 point whose at() window runs into the next hour. That is not the windowing effect above; it is a feed losing 84 % of its own prints, and no provisional flag would ever have reported it.

The alternative was rejected: refusing sub-hourly feeds at _validate_raw() would have made the count correct by breaking the one supported product that legitimately publishes per quarter.

What was done instead keeps FCST-03's existing contract rather than weakening it: every published point inside the window passes through by identity, and a carried point is added only for a whole hour with no published coverage at all. A carried hour is still a single hourly point whatever the feed's own spacing is, because yesterday's profile is only known per wall-clock hour — and carried hours are provisional, so they are out of the trade set anyway.

⚠️ _audit() got stricter, not looser, and that was the condition on doing this at all. The old post-condition refused any repeated _utc_hour — a count that cannot express four points in one hour. It is replaced by two rules that subsume it:

• a published point must be one of that hour's published points, by identity, and may be emitted only once;
• a carried point may stand only in an hour that does not appear in the published set at all, must be flagged True, and may not repeat.

So a duplicated carried hour — the DST mispricing the count existed to catch — still raises; so does the same published point emitted twice; and a carried point sharing an hour with a real print now raises too, which the count never looked at. Each of those three is driven by its own mutation in test_price_forecast.py.

5. Store details (moved here from app/price_history.py by MAINT-20)

Why a file, and which one. FCST-03 shipped the fill with no caller, so no series ever contained a carried point. Without a yesterday that survives a restart, the mornings after an operator touched something spend the whole window on the truncated series. The cache is /data/price_history.json, not /data/state.json: Store.set() rewrites the whole of state.json on each write, and the refresh runs every TARIFF_REFRESH_S (1800 s), so 48 extra rewrites a day would go through the file whose corruption reads as "no maintenance ever ran". Separate file, separate blast radius. The directory is not created (tariff.PeakStore's rule): /data is always mounted, so a makedirs could only succeed where the durable volume is missing and build a cache on the ephemeral layer. "Cannot persist" is one question asked after the attempt: see FCST-06; self._days never holds a point the file does not.

Brussels date, not UTC. 23:30 UTC in July is already tomorrow in Belgium. _yesterday_of() subtracts on the local date, not the instant: the old brussels(stamp) - timedelta(days=1) took 24 absolute hours and, at 2026-03-30T00:30+02:00, returned 2026-03-28 (a two-day-old profile flagged as yesterday's). Rows are keyed by instant, not local hour, so the two 02:00s of the 25-hour October day both survive and fill_from_yesterday()'s own rule is the only collapsing rule. The merge key is the validated instant; the old string key merged by spelling and sorted the +01:00 02:00 before the +02:00. KEEP_DAYS is a sort-order cap, not a horizon: nothing is deleted by arithmetic against the clock (a jumped clock, WITNESS-03, would otherwise delete everything on the one morning the fill is for), and the two dates yesterday() can be asked for are pinned on top of the cap, which can only retain.

One validator (FCST-05, MEDIUM). The read path was defensive about every field and the write path trusted q["start"] to be hashable and sortable. An int, null, dict or list start raised TypeError out of carry() on the day being written; run_tariff_refresh caught it, so the add-on stayed up and every refresh failed for good (the plan retires after TARIFF_MAX_AGE_S, arbitrage and the cheap window go off, and _remember() raised before _save(), so no self-heal). _point_of() is now the one standard for both paths and returns None rather than raising.

A broken cache is a short horizon, never a controller that will not start (6: absent machinery must not disable the loop). PriceHistory(...) and .words are unguarded in Controller.__init__. Shipped defects, all fixed:

• _load() caught json.JSONDecodeError and OSError only; one non-UTF-8 byte raised UnicodeDecodeError (a ValueError, not an OSError) and the add-on did not start (FCST-05, HIGH). The handler names the ValueError family. The schema version is compared with its type (1.0 == 1 in Python).
• {"days": {"2026-01-14": "oops"}} loaded as healthy and raised AttributeError out of words, yesterday() and resolve_tariff(). _usable_days() drops such records, and counts (not drops) malformed rows inside a kept record so the store stops reporting health it lacks.
• words printed f"{d} ({_n_points(rec)} h)": points under the hour's name ('2026-01-13 (96 h)' for a 24-hour quarter-hourly day, FCST-05 LOW). Both numbers are printed now; only rows that are points are counted. words must never raise, _held() is total.
• A failed merge is undone from a deep copy (a one-level copy shares the day records the merge rebinds).

NO_HISTORY is a name, not None, so history= can be required: None meaning "skip the fill" is how a caller silently gets the truncated horizon. carry() records first, then fills, so the filled series exists only as a return value that the store never sees again (rule 1).