goodwe-addon/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md

# FCST-04 — what "yesterday" is, and what the fill is allowed to reach

Four decisions, recorded because each had a defensible alternative that a later
reader would otherwise re-open from scratch.

---

## 1. The fill runs on a `series` read only. A dead feed still goes to §11.4.

`fill_from_yesterday()` accepts a `no_data` read and will happily build a
100 %-carried series from yesterday alone. `resolve_tariff()` does not let it:
a feed that published nothing still falls to `_degrade()` and §11.4's own row —
*"No price data → fall back to the static profile, mark the plan provisional,
no arbitrage."*

**Why not prefer yesterday's real prices to a coarse day/night profile?**
Because it buys nothing the optimiser can use, and it silently replaces a
fallback the operator configured.

Measured, not assumed — and stated at the reach it actually has:

- **This one reaches control.** A fully-carried series is 100 % provisional, so
  `TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on
  §8.3's firm-pair rung. That is the verdict the optimiser reads: `arbitrage.py`
  calls `plan.arbitrage(...)` on `fresh_tariff_plan()`, and it is the only path
  by which a provider plan moves energy. The §11.4 static fallback refuses on
  the same rung, for the same reason.
- **This one does not, and saying so is the point.** `TariffPlan.cheap_now()`
  also returns False on a fully-carried series, because `firm_points()` is
  empty — but it has **no production caller on the provider plan**.
  `self.cheap` comes from `static_status()`, which builds its *own* static plan
  and calls `cheap_now()` on that. So it is a property of the type, checked in
  `test_static_tariff.py`, and it is not evidence about the shipped control
  path. The conclusion below holds on the first bullet alone.

The two paths therefore differ only in which numbers the status page shows.

So the choice is between two identical control behaviours, and the tie-breaker
is that `static_profile` is a documented option an operator fills in **for this
case**. Overriding it for a cosmetic difference is a behaviour change nobody
asked for.

⚠️ **The consequence, stated so it is not discovered later as a bug:**
`fill_from_yesterday()`'s `no_data` branch is not reached from production.
It stays checked in `test_price_forecast.py`. Turning it on is a product
decision — it would want a rule about how old a carried profile may be before it
stops being offered at all — and not a wiring one.

---

## 2. "Yesterday" is a date, not the newest record.

`newest()` is the obvious accessor and it is the one that decays. An add-on down
from Friday to Monday would answer Friday's prices to a fill whose only
vocabulary is `provisional: true` — a flag that says "this is a guess", never
"this guess is three days old". Nothing downstream can tell the difference and
nothing reports it.

So `yesterday(stamp)` computes the Brussels date before `stamp`'s Brussels date
and looks up **that** record. Anything else is `no_data` with a reason naming
both the date wanted and the dates held.

The cost is real and accepted: after an outage of more than a day, the first
morning back has no fill. That is the honest state — there is no yesterday —
and the horizon is short for one morning rather than wrong for several.

---

## 3. What is recorded is the provider's read, and the mechanism is ordering.

Feeding the filled series back into the store is how this rots: day 2 carries
day 1's guesses, day 3 carries day 2's carry of them, and after a week of a dead
feed the "yesterday" being carried is a fossil of the last real publication.
Every point still carries `provisional=True`, so the **flag never lies** — what
decays is the number, behind a flag that is telling the truth.

⚠️ **No value-level check can catch that**, and claiming one would be worse than
claiming none. A carried point and a published-provisional point (a real-time
product's current interval, §9.2) are the same three fields carrying the same
flag. The
difference is provenance, and provenance is not in the value.

The defence is therefore an **ordering**, not a validation:

- `carry()` is the only public entry point. It records its argument, then
  returns the fill.
- The filled series exists only as that return value. The object never sees it
  again.
- `_remember()` is private and called from exactly one place.
- `test_price_history.py` parses every shipped module under `app/`, `tools/`,
  `scenarios/` and `estop/` and fails if any of them **uses the name**
  `_remember` — an attribute, a bare name, a `def`, or the string constant a
  `getattr` would need. By name rather than by substring, so a comment that
  merely mentions the rule does not fail it and a call in `tools/` cannot pass
  unseen.
- and it pins the order two ways: the **line numbers** of the two calls in
  `carry()`, and — the one that matters — the **behaviour**. A feed carrying
  yesterday alongside today, into an empty store, must fill the whole horizon;
  reversed, the lookup finds nothing and the plan is only the hours the feed
  published. Driven, so a reversal is red.

⚠️ **The first version of that ordering check graded nothing, and this file
said it did.** It compared positions in `ast.walk`, which is breadth-first:
`self._remember(...)` is a statement and `self.yesterday(...)` is an *argument*
of `fill_from_yesterday(...)`, one level deeper, so `_remember` came out first
whichever statement was written first. Measured — shipped and reversed both
scored the same. A decorative check that a document cites as a guarantee is
worse than no check, which is why the correction is recorded here rather than
quietly made.

That ordering has a second benefit worth keeping: a feed that returns yesterday
alongside today refreshes yesterday's record from live data *before* it is
consumed as the profile.

---

## What this ticket does not claim

It does not make a trade possible that yesterday's prices justify. Carried hours
are dropped from the trade set by §8.3, and that is not a limitation to be fixed
later — it is the rule.

The one change that **does** move energy came from the windowing, not the
carrying: the filled series covers `[since, until)`, so hours that have already
elapsed are no longer in the spread. A cheap night that has gone used to be the
series' cheapest hour, and a cycle cannot buy in an hour that is past — so the
trade was silently never taken. `test_price_history.py` section 2 grades that by
finding the cheapest firm hour with and without the fill and showing it moves.

⚠️ **And section 1's identity check now grades identity.** It used to compare
`eur_per_kwh` between `_before` and `_after`, which come from two *separate*
`resolve_tariff()` calls — each parsing the feed body into its own objects, so
`is` could never have held between them however the fill were written. FCST-03's
whole anti-copy argument is `filled.points[i] is fetched.points[j]`, and floats
cannot grade it. The value comparison is kept and now says what it is (the fill
moved the window, it did not touch a price); identity is graded on a read the
file constructs and still holds a reference to.

---

## 4. A sub-hourly feed passes through whole; the fill only decides which hours are missing.

⚠️ **This section originally said `eneco_eddy` is "the Belgian imbalance
product". That was false and [[TARIFF-07]] withdrew the plugin: Eneco's dynamic
product is "Zon & Wind Dynamisch" and is indexed to Belpex/EPEX **day-ahead**,
not to imbalance, and `api.eneco.be/eddy/prices` never existed. The mechanism
below is unaffected, because the real quarter-hourly feed is the one still
shipped.**

Belgian **day-ahead** settles per **quarter** — the 15-minute MTU took effect
on 2025-10-01, so `generic_dayahead` reads 96 points a day rather than 24 — and
`price.py`'s `_validate_raw()` accepts any strictly-increasing spacing, so a
published hour may legitimately hold four points.

FCST-04's first caller keyed the published set as `{_utc_hour(p.start): p for p
in r.points}`. It reads perfectly ordinary and it keeps the **last** point of
each hour. Measured on a 96-point quarter-hourly feed — all firm, fully
published, nothing to carry: the plan reached the optimiser as **15 points**,
the arbitrage spread fell from **0.143 to 0.014 EUR/kWh**, and the cheapest
"hour" became a `:45` point whose `at()` window runs into the next hour. That is
not the windowing effect above; it is a feed losing 84 % of its own prints, and
no `provisional` flag would ever have reported it.

**The alternative was rejected:** refusing sub-hourly feeds at `_validate_raw()`
would have made the count correct by breaking the one supported product that
legitimately publishes per quarter.

**What was done instead** keeps FCST-03's existing contract rather than weakening
it: *every* published point inside the window passes through **by identity**,
and a carried point is added only for a whole hour with **no** published
coverage at all. A carried hour is still a single hourly point whatever the
feed's own spacing is, because yesterday's profile is only known per wall-clock
hour — and carried hours are provisional, so they are out of the trade set
anyway.

⚠️ **`_audit()` got stricter, not looser, and that was the condition on doing
this at all.** The old post-condition refused any repeated `_utc_hour` — a count
that cannot express four points in one hour. It is replaced by two rules that
**subsume** it:

- a **published** point must *be* one of that hour's published points, by
  identity, and may be emitted only once;
- a **carried** point may stand only in an hour that does not appear in the
  published set at all, must be flagged `True`, and may not repeat.

So a duplicated carried hour — the DST mispricing the count existed to catch —
still raises; so does the same published point emitted twice; and a carried
point sharing an hour with a real print now raises too, which the count never
looked at. Each of those three is driven by its own mutation in
`test_price_forecast.py`.