diff --git a/goodwe_controller/CHANGELOG.md b/goodwe_controller/CHANGELOG.md index 102dae9..8023f3e 100644 --- a/goodwe_controller/CHANGELOG.md +++ b/goodwe_controller/CHANGELOG.md @@ -1,5 +1,105 @@ # Changelog +## Unreleased + +⚠️ **THIS HAS NO VERSION AND CANNOT REACH AN INSTALL UNTIL IT GETS ONE.** +PKG-06's and PKG-07's rule: the Supervisor keys the built image by `version:`, +so `ha apps update` is a silent no-op on an unbumped change. This branch +deliberately does **not** pick the number, because the right one depends on a +fact the branch cannot observe - whether 0.9.0's image has already been built +and pulled. If it has not, this belongs inside 0.9.0's section before it ships; +if it has, it needs 0.10.0. Whoever deploys decides. + +| | | +|---|---| +| _this branch_ | FCST-04 - the carry-forward fill reaches the plan, and the day it carries survives a restart | + +**FCST-03 shipped the fill and no caller.** `price_forecast.fill_from_yesterday()` +was complete and checked, and so were the consumers written for it - +`TariffPlan.firm_points()`, `cheap_now()`'s refusal of a provisional hour, and +`arbitrage()`'s refusal 3, which drops provisional hours from the trade set +rather than refusing the whole series. Nothing in `app/` called the fill, so no +series ever contained a carried point and every one of those consumers was +exercising its all-firm branch. + +⚠️ **THE DEFECT THAT PRODUCED WAS NOT AN ERROR ANYWHERE.** `load_prices()` +returns what the feed published and nothing widened it to the requested horizon, +so on a dynamic site between midnight and the 14-16 h CET day-ahead publication +the plan covered the rest of today and **nothing more** - typically 10-24 of the +48 hours `price_horizon_h` asked for. `arbitrage()` then took `max - min` over +that truncated window and reported a spread. The number was real, the window was +half the one configured, and **nothing said so**: tomorrow evening's peak, the +hour a morning charge would be sold into, was not in the series at all, so the +spread it would have created was never compared against anything. + +**What changes:** + +* `app/price_history.py` (new) keeps the **published** prices of the last few + Brussels days in `/data/price_history.json`, and `resolve_tariff()` now fills + an unpublished hour of the horizon from yesterday's same-wall-clock hour, + flagged provisional. ⚠️ **"An" unpublished hour, not "every" one:** the fill + can carry only an hour yesterday actually published, so a partial yesterday + leaves the local hours it never held **out of the series** - §11.4 for those + hours, and asking for one raises rather than answering. Correct behaviour, + and the wording is now the behaviour. +* ⚠️ **A sub-hourly feed keeps every print.** `eneco_eddy` is the imbalance + product and settles per **quarter**; the fill walks whole hours to decide + what needs carrying, and that is not a rule about what a published hour may + contain. Every published point inside the horizon passes through by identity, + and a carried point is added only for a whole hour with no published coverage + at all. The first draft keyed the published set by hour - last point wins - + and a 96-point quarter-hourly feed reached the optimiser as 15 points with + its spread down from 0.143 to 0.014 EUR/kWh, on an all-firm, fully-published + series. +* The horizon is **whole hours at both ends**: `since` is floored to the hour + that contains it, and so is `until`. Asked for 48 hours at 08:17, the plan is + 48 hours long and not 49. +* `resolve_tariff(history=...)` is **required and keyword-only**. A caller with + no durable volume passes the explicit `NoHistory()`. A default of `None` + meaning "skip the fill" would hand a caller who forgot the argument exactly + the truncated horizon this card removes, silently. +* A new status row, **Price history**, names the days held and their hour + counts. + +⚠️ **ONE CHANGE HERE MOVES ENERGY, AND IT IS NOT THE ONE THE CARD IS NAMED FOR.** +Carried hours are provisional, so §8.3 keeps them out of the trade set: the fill +widens what the optimiser can **see**, never what it may **do**. What does +change a trade is that the filled series is windowed to `[since, until)`, so +**hours that have already elapsed are no longer in the spread**. Before, a cheap +night that had already gone sat in the series as the day's cheapest hour - and a +cycle cannot buy in an hour that is past, so the trade was quietly never taken. +`test_price_history.py` section 2 grades this by finding the cheapest firm hour +with and without the fill and showing it **moves**. + +⚠️ **A DEAD FEED IS STILL §11.4, NOT YESTERDAY.** The fill runs only on a +`series` read; a feed that published nothing still degrades to the configured +`static_profile`. The two are indistinguishable to the optimiser - both are +100 % provisional, so arbitrage refuses on the firm-pair rung and `cheap_now()` +is False - so preferring yesterday would change a documented degradation for a +difference only the status page can see. +`DECISIONS/FCST-04-the-day-that-is-carried.md` carries that and two more. + +⚠️ **A BROKEN CACHE IS NOT A BROKEN CONTROLLER.** No `/data`, an unreadable +file, a schema this build does not know (including a `version` of `1.0`, which +`== 1`), a **day record that is not a record**, a malformed point - each its own +state with its own reason, none of them a price and none of them an empty +series. In every one the plan is still a real provider series and the controller +still regulates; the horizon is just as short as it was before this card. +`.words` never raises, because `Controller.__init__` reads it unguarded and a +status line that throws is an add-on that does not start. + +⚠️ **AND WHERE THERE IS NO `/data`, NOTHING IS RECORDED AT ALL** - not even in +memory. An in-memory yesterday answers for one run and is gone on the next +restart, which is a cache that forgets without saying so; the honest pair is +`unavailable` on the status row and no prices carried. + +⚠️ **"YESTERDAY" IS A LOCAL DATE, AND THE ARITHMETIC IS ON THE DATE.** +`brussels()` renders at a fixed offset, so subtracting `timedelta(days=1)` from +it subtracts 24 **absolute** hours - and the Brussels day before a +spring-forward is 23 hours long. Between 00:00 and 00:59 on the morning after +the March transition that lands on the day *before* yesterday. **No option was +added and `config.yaml` is untouched.** + ## 0.9.0 ⚠️ **MINOR FOR NEW FUNCTIONALITY, NOT FOR A BREAK - AND THAT DISTINCTION IS THE diff --git a/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md b/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md new file mode 100644 index 0000000..c79ecb6 --- /dev/null +++ b/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md @@ -0,0 +1,181 @@ +# FCST-04 — what "yesterday" is, and what the fill is allowed to reach + +Four decisions, recorded because each had a defensible alternative that a later +reader would otherwise re-open from scratch. + +--- + +## 1. The fill runs on a `series` read only. A dead feed still goes to §11.4. + +`fill_from_yesterday()` accepts a `no_data` read and will happily build a +100 %-carried series from yesterday alone. `resolve_tariff()` does not let it: +a feed that published nothing still falls to `_degrade()` and §11.4's own row — +*"No price data → fall back to the static profile, mark the plan provisional, +no arbitrage."* + +**Why not prefer yesterday's real prices to a coarse day/night profile?** +Because it buys nothing the optimiser can use, and it silently replaces a +fallback the operator configured. + +Measured, not assumed — and stated at the reach it actually has: + +- **This one reaches control.** A fully-carried series is 100 % provisional, so + `TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on + §8.3's firm-pair rung. That is the verdict the optimiser reads: `arbitrage.py` + calls `plan.arbitrage(...)` on `fresh_tariff_plan()`, and it is the only path + by which a provider plan moves energy. The §11.4 static fallback refuses on + the same rung, for the same reason. +- **This one does not, and saying so is the point.** `TariffPlan.cheap_now()` + also returns False on a fully-carried series, because `firm_points()` is + empty — but it has **no production caller on the provider plan**. + `self.cheap` comes from `static_status()`, which builds its *own* static plan + and calls `cheap_now()` on that. So it is a property of the type, checked in + `test_static_tariff.py`, and it is not evidence about the shipped control + path. The conclusion below holds on the first bullet alone. + +The two paths therefore differ only in which numbers the status page shows. + +So the choice is between two identical control behaviours, and the tie-breaker +is that `static_profile` is a documented option an operator fills in **for this +case**. Overriding it for a cosmetic difference is a behaviour change nobody +asked for. + +⚠️ **The consequence, stated so it is not discovered later as a bug:** +`fill_from_yesterday()`'s `no_data` branch is not reached from production. +It stays checked in `test_price_forecast.py`. Turning it on is a product +decision — it would want a rule about how old a carried profile may be before it +stops being offered at all — and not a wiring one. + +--- + +## 2. "Yesterday" is a date, not the newest record. + +`newest()` is the obvious accessor and it is the one that decays. An add-on down +from Friday to Monday would answer Friday's prices to a fill whose only +vocabulary is `provisional: true` — a flag that says "this is a guess", never +"this guess is three days old". Nothing downstream can tell the difference and +nothing reports it. + +So `yesterday(stamp)` computes the Brussels date before `stamp`'s Brussels date +and looks up **that** record. Anything else is `no_data` with a reason naming +both the date wanted and the dates held. + +The cost is real and accepted: after an outage of more than a day, the first +morning back has no fill. That is the honest state — there is no yesterday — +and the horizon is short for one morning rather than wrong for several. + +--- + +## 3. What is recorded is the provider's read, and the mechanism is ordering. + +Feeding the filled series back into the store is how this rots: day 2 carries +day 1's guesses, day 3 carries day 2's carry of them, and after a week of a dead +feed the "yesterday" being carried is a fossil of the last real publication. +Every point still carries `provisional=True`, so the **flag never lies** — what +decays is the number, behind a flag that is telling the truth. + +⚠️ **No value-level check can catch that**, and claiming one would be worse than +claiming none. A carried point and a published-provisional point (`eneco_eddy`'s +current quarter, §9.2) are the same three fields carrying the same flag. The +difference is provenance, and provenance is not in the value. + +The defence is therefore an **ordering**, not a validation: + +- `carry()` is the only public entry point. It records its argument, then + returns the fill. +- The filled series exists only as that return value. The object never sees it + again. +- `_remember()` is private and called from exactly one place. +- `test_price_history.py` parses every shipped module under `app/`, `tools/`, + `scenarios/` and `estop/` and fails if any of them **uses the name** + `_remember` — an attribute, a bare name, a `def`, or the string constant a + `getattr` would need. By name rather than by substring, so a comment that + merely mentions the rule does not fail it and a call in `tools/` cannot pass + unseen. +- and it pins the order two ways: the **line numbers** of the two calls in + `carry()`, and — the one that matters — the **behaviour**. A feed carrying + yesterday alongside today, into an empty store, must fill the whole horizon; + reversed, the lookup finds nothing and the plan is only the hours the feed + published. Driven, so a reversal is red. + +⚠️ **The first version of that ordering check graded nothing, and this file +said it did.** It compared positions in `ast.walk`, which is breadth-first: +`self._remember(...)` is a statement and `self.yesterday(...)` is an *argument* +of `fill_from_yesterday(...)`, one level deeper, so `_remember` came out first +whichever statement was written first. Measured — shipped and reversed both +scored the same. A decorative check that a document cites as a guarantee is +worse than no check, which is why the correction is recorded here rather than +quietly made. + +That ordering has a second benefit worth keeping: a feed that returns yesterday +alongside today refreshes yesterday's record from live data *before* it is +consumed as the profile. + +--- + +## What this ticket does not claim + +It does not make a trade possible that yesterday's prices justify. Carried hours +are dropped from the trade set by §8.3, and that is not a limitation to be fixed +later — it is the rule. + +The one change that **does** move energy came from the windowing, not the +carrying: the filled series covers `[since, until)`, so hours that have already +elapsed are no longer in the spread. A cheap night that has gone used to be the +series' cheapest hour, and a cycle cannot buy in an hour that is past — so the +trade was silently never taken. `test_price_history.py` section 2 grades that by +finding the cheapest firm hour with and without the fill and showing it moves. + +⚠️ **And section 1's identity check now grades identity.** It used to compare +`eur_per_kwh` between `_before` and `_after`, which come from two *separate* +`resolve_tariff()` calls — each parsing the feed body into its own objects, so +`is` could never have held between them however the fill were written. FCST-03's +whole anti-copy argument is `filled.points[i] is fetched.points[j]`, and floats +cannot grade it. The value comparison is kept and now says what it is (the fill +moved the window, it did not touch a price); identity is graded on a read the +file constructs and still holds a reference to. + +--- + +## 4. A sub-hourly feed passes through whole; the fill only decides which hours are missing. + +`eneco_eddy` is the Belgian imbalance product and settles **per quarter**, and +`price.py`'s `_validate_raw()` accepts any strictly-increasing spacing — so a +published hour may legitimately hold four points. + +FCST-04's first caller keyed the published set as `{_utc_hour(p.start): p for p +in r.points}`. It reads perfectly ordinary and it keeps the **last** point of +each hour. Measured on a 96-point quarter-hourly feed — all firm, fully +published, nothing to carry: the plan reached the optimiser as **15 points**, +the arbitrage spread fell from **0.143 to 0.014 EUR/kWh**, and the cheapest +"hour" became a `:45` point whose `at()` window runs into the next hour. That is +not the windowing effect above; it is a feed losing 84 % of its own prints, and +no `provisional` flag would ever have reported it. + +**The alternative was rejected:** refusing sub-hourly feeds at `_validate_raw()` +would have made the count correct by breaking the one supported product that +legitimately publishes per quarter. + +**What was done instead** keeps FCST-03's existing contract rather than weakening +it: *every* published point inside the window passes through **by identity**, +and a carried point is added only for a whole hour with **no** published +coverage at all. A carried hour is still a single hourly point whatever the +feed's own spacing is, because yesterday's profile is only known per wall-clock +hour — and carried hours are provisional, so they are out of the trade set +anyway. + +⚠️ **`_audit()` got stricter, not looser, and that was the condition on doing +this at all.** The old post-condition refused any repeated `_utc_hour` — a count +that cannot express four points in one hour. It is replaced by two rules that +**subsume** it: + +- a **published** point must *be* one of that hour's published points, by + identity, and may be emitted only once; +- a **carried** point may stand only in an hour that does not appear in the + published set at all, must be flagged `True`, and may not repeat. + +So a duplicated carried hour — the DST mispricing the count existed to catch — +still raises; so does the same published point emitted twice; and a carried +point sharing an hour with a real print now raises too, which the count never +looked at. Each of those three is driven by its own mutation in +`test_price_forecast.py`. diff --git a/goodwe_controller/DOCS.md b/goodwe_controller/DOCS.md index a97875a..af8efbd 100644 --- a/goodwe_controller/DOCS.md +++ b/goodwe_controller/DOCS.md @@ -1424,6 +1424,70 @@ distinct state from "prices, all of them zero": the plan is marked provisional and arbitrage is disabled, per the degradation matrix. It never falls back to a price of zero, which would read as free electricity. +#### The hours the feed has not published yet (FCST-04) + +A day-ahead auction publishes tomorrow's prices once a day, at **14–16 h CET**. +Before that, the feed is serving today and nothing else — so a `price_horizon_h` +of 48 asks for 48 hours and gets back the ten or fifteen that are left in the +day. Until FCST-04 those other hours were simply **absent** from the plan: not +priced badly, not flagged, just missing, and nothing said so. + +Now the add-on keeps the **published prices of the last few days** in +`/data/price_history.json` and fills an unpublished hour of the horizon with +yesterday's price for the same Brussels wall-clock hour, flagged +`provisional: true`. Two things follow, and the second one matters more: + +- **The plan says it is guessing** — `provisional: true` on every carried hour, + which is what the optimiser reads. Before, a half-length plan looked + completely settled. +- **Nothing may trade on a carried hour.** §8.3 forbids acting on a provisional + price, so arbitrage drops those hours from the trade set entirely — it needs a + buy hour and a sell hour that were *both* published. Carrying a price forward + widens what the optimiser can **see**; it never widens what it may **do**. + +**"An unpublished hour", not "every unpublished hour"** — the fill can only +carry an hour yesterday actually published. If yesterday's record is itself +partial (a first day, an outage, a feed that only ever serves the rest of the +day), the local hours it never held are **left out of the series**: §11.4 +applies to those hours, and asking the plan for one raises rather than +answering. An hour with no price is not a free hour, and the fill does not +invent a neighbour, a mean or a zero for it. + +**A quarter-hourly feed keeps all four of its prints.** `eneco_eddy` settles +per quarter, and the fill decides *which whole hours need carrying* — never what +a published hour may contain. Every point the provider published inside the +horizon is passed through untouched; a carried point is added only for a whole +hour the feed did not print at all. + +**What you can actually see on the status page.** *Price history* (below) says +what is held. *Arbitrage* carries the strategy's own sentence, which names the +firm/provisional split whenever that split is what refused a trade. *Tariff* +does **not** report the fill: it is the static-profile or Home Assistant +price-entity sentence, regenerated on the 30 s tick. The provider plan's own +line — `dynamic_dayahead from generic_dayahead, 48 h, provisional` — goes to the +**add-on log** at every refresh, alongside `§8.3: 33 of 48 hours carried forward +from yesterday's profile and flagged provisional; 15 published`. + +The plan is also now cut to the horizon you asked for, which means **hours that +have already elapsed are no longer in it**. That is a real change to arbitrage: +a cheap night that has already gone used to sit in the series as the cheapest +hour of the day, and a cycle can never buy in an hour that is past — so the +trade was quietly never taken. The spread is now taken over hours still ahead. + +A new status row, **Price history**, says which days are held and how many hours +each has. If it reads `empty` or `unavailable`, nothing is carried, the horizon +is as short as the feed makes it, and every other guarantee is unchanged — a +cache that cannot answer is not allowed to be a controller that cannot regulate. +The store is rebuilt by the next successful fetch; on a brand-new install the +first fill is possible the day after the first refresh. + +The same applies to a file that is *partly* broken: a day record this build +cannot read is dropped, the row says how many were dropped and why, and the rest +of the file still loads. And where there is no `/data` at all the store records +**nothing** — not even in memory. An in-memory yesterday would answer for one +run and vanish on the next restart, which is a cache that forgets without saying +so; `unavailable` and *no prices carried* are the honest pair. + **Injection is priced separately** (`export_price_eur_kwh`). In Belgium it is often near zero, which by itself strongly favours self-consumption over export. It is a single number rather than a curve because no supported supplier diff --git a/goodwe_controller/app/main.py b/goodwe_controller/app/main.py index 2414dce..a92c3c2 100644 --- a/goodwe_controller/app/main.py +++ b/goodwe_controller/app/main.py @@ -68,6 +68,7 @@ from . import peakguard from . import packshare from .p1 import P1Ingest, build_source, is_enabled from .price import Uncertainty, price_option +from .price_history import PriceHistory from .static_tariff import resolve_tariff, static_status from .tariff import PeakStore, PeakTracker from .store import migrate_maint_keys, migrate_unit_keys @@ -165,7 +166,8 @@ def pack_energy_of(inv): class Controller: def __init__(self, opts: dict, hass: HomeAssistant, store, mqtt_pub, *, stop, - peak_db_path: str = "/data/tariff.db"): + peak_db_path: str = "/data/tariff.db", + price_history_path: str = "/data/price_history.json"): self.o = opts self.hass = hass self.store = store @@ -582,6 +584,42 @@ class Controller: self.peak.restore(self.p1.averager) _LOG.info("capacity-tariff peak store: %s", self.peak.restore_state) + # FCST-04. The §8.3 carry-forward store: the published prices of the + # last few Brussels days, so the hours a day-ahead feed has not printed + # yet can be filled from yesterday's profile and FLAGGED, instead of + # being absent from the series entirely. + # + # ⚠️ IT IS AN INSTRUMENT'S BACKING FILE, NOT A STRATEGY. It commands + # nothing and claims nothing; what it changes is how many hours the + # plan covers. Before this, a dynamic site between midnight and the + # 14-16 h CET publication had a series covering only the rest of today + # - typically 10-24 of the 48 hours `price_horizon_h` asked for - and + # `TariffPlan.arbitrage()` took its spread over that truncated window + # without anything reporting that it had. Every carried hour is + # provisional, so refusal 3 drops it from the trade set: CARRYING a + # price widens what the optimiser can SEE, never what it may act on. + # + # ⚠️ AND THE CAVEAT THAT BELONGS WITH THAT SENTENCE, because this is + # the one place a maintainer reads it on its own. THE OTHER HALF OF + # THIS CHANGE DOES MOVE ENERGY, and it is the windowing rather than the + # carrying: the filled series covers `[since, until)`, so hours that + # have ALREADY ELAPSED are no longer in the spread. A cheap night that + # has gone used to sit in the series as the day's cheapest hour, and no + # cycle can buy in an hour that is past - so the trade was quietly + # never taken. `resolve_tariff()`'s docstring, DOCS.md and the CHANGELOG + # all carry this beside the guarantee; it may not appear here without it. + # + # ⚠️ A BROKEN OR ABSENT STORE IS NOT AN ERROR HERE. It answers `no_data`, + # the fill carries nothing, and the plan is the pre-FCST-04 one minus + # the elapsed hours - short, but honest and still refusing on the same + # rules. It must not be an error in the other sense either: a day + # record this build cannot read is DROPPED by `PriceHistory._load()` + # and reported in `.words`, because both the constructor above and the + # `.words` read below are unguarded, and a cache that raises here is an + # add-on that does not start. + self.price_history = PriceHistory(price_history_path) + _LOG.info("§8.3 price history: %s", self.price_history.words) + # SAFETY-07. The guard TARIFF-01 was built for. It is stateless by # design (peakguard.py holds no history between calls), so the only # state here is what an EDGE needs: what the last cycle concluded, so @@ -3455,7 +3493,8 @@ class Controller: now = datetime.now(timezone.utc) try: plan = await resolve_tariff( - self.o, now, now + timedelta(hours=horizon), now=now) + self.o, now, now + timedelta(hours=horizon), now=now, + history=self.price_history) self.tariff_plan, self.tariff_at = plan, now _LOG.info("tariff refreshed: %s", plan.words) except Exception as err: # noqa: BLE001 @@ -4304,6 +4343,13 @@ class Controller: ("Maintenance due", "now" if next_due is None else next_due.strftime("%Y-%m-%d")), ("Cheap window", "yes" if self.cheap else "no"), ("Tariff", self.tariff_words), + # FCST-04. WHAT §8.3 HAS TO CARRY FORWARD, which is the difference + # between a 48-hour plan and a 12-hour one on a weekday morning. + # It is its own row rather than a clause of "Tariff" because it + # answers a different question: "Tariff" says what the plan IS, and + # this says what the plan could still be filled from if the feed is + # late again tomorrow. + ("Price history", self.price_history.words), # OPT-04. The strategy's own sentence, not a re-derivation of it - # every refusal it can make already carries a reason a human can # act on, and "Arbitrage: no" would throw all of them away. diff --git a/goodwe_controller/app/price_forecast.py b/goodwe_controller/app/price_forecast.py index fdb7e7a..2d8d7d2 100644 --- a/goodwe_controller/app/price_forecast.py +++ b/goodwe_controller/app/price_forecast.py @@ -48,7 +48,8 @@ So the flag is not passed through anywhere in this file: the object the provider produced is the object in the output series - so there is no copy that could lose or gain a flag. `filled.points[i] is fetched.points[j]` is a checkable property, and test_price_forecast.py - checks it. + checks it. EVERY published point in the window is passed through, not one + per hour: see the sub-hourly note below. * `_audit()` RE-DERIVES the expected flag for every output point from the set of hours the feed actually published, and RAISES if any point disagrees. It is a post-condition, not a filter: it never fixes a flag, because a @@ -90,6 +91,24 @@ which is the row the spec actually writes for "no price data". If NOTHING can be produced, the result is `PriceRead.no_data(reason)` and not an empty series, because an empty series is not constructible on purpose (price.py trap 3). +⚠️ A SUB-HOURLY FEED IS NOT DECIMATED TO ONE POINT AN HOUR, and the hourly walk +below is about WHICH HOURS NEED FILLING - never about what a published hour is +allowed to contain. `eneco_eddy` is the Belgian imbalance product and settles +per QUARTER, `_validate_raw()` accepts any strictly-increasing spacing, and a +published hour may therefore hold four points. So EVERY published point inside +the window is passed through, in order, and a carried point is added only for a +whole hour with NO published coverage at all. + +The alternative was measured rather than reasoned about, because FCST-04's +first caller shipped it: keying the published set as `{_utc_hour(p.start): p +for p in r.points}` reads perfectly ordinary and keeps the LAST point of each +hour, dropping the other three. On a 96-point quarter-hourly feed - all firm, +fully published, nothing to carry - the series came out 15 points long, the +arbitrage spread collapsed from 0.143 to 0.014 EUR/kWh, and the cheapest +"hour" became a `:45` point whose `at()` window runs into the next hour. A feed +losing 84 % of its own prints is not a windowing effect and is not something a +provisional flag would ever have reported. + AC2: THE IRREVERSIBLE-ACTION GATE ================================= @@ -156,6 +175,25 @@ def _utc_hour(instant: datetime) -> datetime: return u.replace(minute=0, second=0, microsecond=0) +def _published_by_hour(read: PriceRead) -> dict: + """UTC hour -> the provider's OWN points that start inside it, in order. + + ⚠️ A TUPLE PER HOUR, NOT A POINT, AND THAT IS THE WHOLE OF THE SUB-HOURLY + RULE. `{_utc_hour(p.start): p for p in r.points}` reads identically, is one + character shorter, and silently keeps one point per hour - the last one + written. The mapping answers two DIFFERENT questions here: "does this hour + need filling at all?" is `u in published`, and "what does the output carry + for it?" is all of `published[u]`. Collapsing them is how a quarter-hourly + feed loses three quarters of its prints. + """ + out: dict = {} + read.match( + series=lambda r: [out.setdefault(_utc_hour(p.start), []).append(p) + for p in r.points], + no_data=lambda _r: []) + return {u: tuple(ps) for u, ps in out.items()} + + def _carried(start_utc: datetime, eur_per_kwh: float) -> PricePoint: """A point filled from yesterday's profile. §8.3: ALWAYS provisional. @@ -172,37 +210,69 @@ def _carried(start_utc: datetime, eur_per_kwh: float) -> PricePoint: def _audit(points, published) -> None: """Post-condition: every flag matches the point's TRUE provenance (AC1). - `published` maps a UTC hour to the point the provider produced for it. + `published` maps a UTC hour to the TUPLE of points the provider produced + for it (`_published_by_hour()`). An hour ABSENT from that mapping is an hour + the feed did not print, and a carried point is the only thing that may + stand in it. + + The rule, in the two forms an output point can take: + + * PUBLISHED - the point must BE one of `published[u]`, by identity, and + may be emitted only once. Its flag is whatever the feed said, never + forced to False: an imbalance print is published AND provisional (§9.2, + TARIFF-02), and a check that "corrected" it would be a bug wearing a + guarantee. + * CARRIED - the hour must not appear in `published` AT ALL, the flag must + be exactly True, and that hour may hold no second point. + + ⚠️ WHAT REPLACED "AT MOST ONE POINT PER HOUR", AND WHY IT IS STRICTER. This + audit began as a `seen` set: any repeated `_utc_hour` raised, because a + duplicated hour is how a DST day gets mispriced. A bare hour count cannot + express a quarter-hourly feed, where four distinct published points share + one hour legitimately - so the count was replaced by the two rules above, + which subsume it. A repeated CARRIED hour still raises; so does the same + published point emitted twice; and a carried point may no longer share an + hour with a published one, which the count never checked at all. ⚠️ RAISES, NEVER REPAIRS. A module that silently corrects its own provenance is one whose provenance means nothing; the only safe response to "this point's flag disagrees with where it came from" is to refuse to emit the series at all. """ - seen = set() + carried_hours = set() + emitted = set() for p in points: u = _utc_hour(p.start) - if u in seen: - raise ValueError(f"the filled series repeats {u.isoformat()}; a " - f"duplicated hour is how a DST day gets mispriced") - seen.add(u) src = published.get(u) - prov = Provenance.CARRIED if src is None else Provenance.PUBLISHED if src is None: + if u in carried_hours: + raise ValueError( + f"the filled series repeats the carried hour " + f"{u.isoformat()}; a duplicated hour is how a DST day gets " + f"mispriced") + carried_hours.add(u) if p.provisional is not True: raise ValueError( - f"{u.isoformat()} is {prov.value} and is flagged " - f"provisional={p.provisional!r}. §8.3: an hour filled from " - f"yesterday's profile is provisional whether or not the " - f"number turns out to be right - the flag is a claim about " - f"PROVENANCE, not about accuracy.") + f"{u.isoformat()} is {Provenance.CARRIED.value} and is " + f"flagged provisional={p.provisional!r}. §8.3: an hour " + f"filled from yesterday's profile is provisional whether " + f"or not the number turns out to be right - the flag is a " + f"claim about PROVENANCE, not about accuracy.") else: - if p is not src: + if not any(p is q for q in src): raise ValueError( - f"{u.isoformat()} is {prov.value} but the series carries a " - f"COPY of that point rather than the point itself. A copy " - f"is where the provisional flag gets lost; published points " - f"pass through by identity.") + f"{u.isoformat()} is {Provenance.PUBLISHED.value} but the " + f"series carries a COPY of that point rather than the " + f"point itself. A copy is where the provisional flag gets " + f"lost; published points pass through by identity.") + if id(p) in emitted: + raise ValueError( + f"the filled series emits the published point at " + f"{p.start.isoformat()} TWICE. The provider's own points " + f"are strictly increasing (`_validate_raw`), so a repeat " + f"is never a second print - it is a duplicated hour, which " + f"is how a DST day gets mispriced") + emitted.add(id(p)) def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead, @@ -217,11 +287,21 @@ def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead, Returns a `PriceRead`: - * a series, whose points are the provider's own objects for the hours it - published and `_carried()` points for the rest, ordered by instant; + * a series, carrying the provider's own objects - ALL of them, which on + a quarter-hourly feed is four inside one hour - for the hours it + published, and one `_carried()` point for each remaining whole hour, + ordered by instant; * `no_data` when neither source can produce a single hour - never an empty series, which is not constructible (price.py trap 3). + ⚠️ THE WINDOW IS WHOLE HOURS AT BOTH ENDS. `since` is floored to the hour + that CONTAINS it, because the current hour is still a tradeable hour; and + so is `until`, which is the half FCST-04's first draft left out. With + `since` floored and `until` compared raw, a 48 h horizon asked for at 08:17 + emitted 49 hours - the hour of `until` was still `< until` - and the count + was one too many for every `now` that is not exactly on the hour, which is + every `now` a running controller has. + ⚠️ IDEMPOTENT AND RE-RUNNABLE, WHICH IS THE §5.4 LATE-PUBLICATION CONTRACT. Day-ahead publication at 14-16 h CET can be late or fail, so the fetcher retries and this runs again against the fresher read. Published hours always @@ -234,9 +314,7 @@ def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead, raise ValueError(f"{name} must be tz-aware; a naive instant is " f"ambiguous for one hour every October") - published = read.match( - series=lambda r: {_utc_hour(p.start): p for p in r.points}, - no_data=lambda _r: {}) + published = _published_by_hour(read) # Yesterday's profile, keyed by BRUSSELS WALL-CLOCK HOUR. First point wins # when yesterday itself was a 25-hour day and printed local hour 2 twice. @@ -248,11 +326,12 @@ def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead, out = [] u = _utc_hour(since) - end = until.astimezone(timezone.utc) + end = _utc_hour(until) while u < end: - if u in published: - out.append(published[u]) # BY IDENTITY. No copy. - else: + got = published.get(u) + if got: + out.extend(got) # BY IDENTITY, and ALL of them. No copy, + else: # and no decimation to one an hour. src = by_local_hour.get(brussels(u).hour) if src is not None: out.append(_carried(u, src.eur_per_kwh)) diff --git a/goodwe_controller/app/price_history.py b/goodwe_controller/app/price_history.py new file mode 100644 index 0000000..464c938 --- /dev/null +++ b/goodwe_controller/app/price_history.py @@ -0,0 +1,598 @@ +"""FCST-04 - the day the carry-forward carries, kept somewhere a restart cannot reach. + +WHAT WAS MISSING, AND WHY IT WAS INVISIBLE. FCST-03 shipped +`price_forecast.fill_from_yesterday()` complete and checked, and shipped the +consumers that were written to expect it - `TariffPlan.firm_points()`, +`cheap_now()`'s refusal of a provisional hour, and `arbitrage()`'s refusal 3, +which drops provisional hours from the trade set instead of refusing the whole +series. Every one of those was live. The FILL ITSELF HAD NO CALLER: nothing in +`app/` referenced `fill_from_yesterday`, so no series ever contained a carried +point and every one of those consumers was exercising its all-firm branch. + +The defect that produced is not an error anywhere. `load_prices()` returns +whatever the feed published and nothing widens it to the requested horizon, so +on a dynamic site between midnight and the 14-16 h CET day-ahead publication +the series covers the rest of TODAY and nothing more - typically 10-24 of the +48 hours `price_horizon_h` asked for. `TariffPlan.arbitrage()` then computes +`max - min` over that truncated window and reports a spread. The number is +real, the window is half the one the operator configured, and NOTHING SAYS SO: +tomorrow evening's peak - the hour a morning charge would be sold into - is +simply not in the series, so the spread it would have created cannot clear a +threshold it was never compared against. A missing hour is not a wrong price; +it is an hour the optimiser never learns exists. + +So this module is the half FCST-03 left out, and it is deliberately the SMALL +half: the fill's arithmetic, its DST rule and its provenance audit all stay in +`price_forecast.py`, untouched. What lives here is the one thing that +arithmetic needs and could not have: A YESTERDAY THAT SURVIVES A RESTART. + + +WHY PERSISTENCE, AND NOT A VARIABLE ON THE REFRESH TASK +====================================================== +An in-memory "the last full day I saw" costs nothing and is wrong exactly when +it matters. The window the fill exists for is midnight -> publication; an +add-on restarted at 08:00 - a config change, an HA update, a power cut - would +come up with no yesterday at all and spend that whole window on the truncated +series, which is the pre-FCST-03 behaviour arriving silently on the mornings +after an operator touched something. The fill's own value is a function of +surviving a restart, so the day it carries has to. + +⚠️ AND NOT IN `/data/state.json`. That file holds the maintenance history and +SAFETY-02's sign latch, and `Store.set()` rewrites the whole of it on every +write. The tariff refresh runs every `TARIFF_REFRESH_S` (1800 s), so folding a +price cache into it would put 48 extra whole-file rewrites a day through the +file whose corruption reads as "no maintenance ever ran" and "never tripped". +The cache is worth having; it is not worth putting anywhere near that. Separate +file, separate blast radius. + +⚠️ AND THE DIRECTORY IS NOT CREATED, which is `tariff.PeakStore`'s rule and its +reason applies unchanged: `/data` is mounted by the Supervisor and always +exists on a real install, so `makedirs` here could only ever succeed where the +durable volume is MISSING - manufacturing a cache on the container's ephemeral +layer that writes fine, reads fine, and is thrown away on the next restart. +That is a yesterday that silently forgets, which is worse than one that says it +cannot answer. + +⚠️ AND THAT RULE IS CARRIED OUT IN `_remember()`, NOT ONLY IN `_save()`. FCST-04 +first shipped it half-done: `_save()` short-circuited on `not self.ok` while +`_remember()` filled `self._days` anyway, so a store with no durable volume +answered `words` = "unavailable" and `yesterday()` = A SERIES - an in-memory +yesterday that works all day and is gone on the next restart, which is exactly +the silently-forgetting cache this rule exists to refuse. A store that cannot +persist now records nothing, and both of its answers say the same thing. + + +THE FOUR RULES THIS FILE IS SHAPED BY +===================================== + +1. IT RECORDS WHAT THE PROVIDER PUBLISHED, NEVER WHAT THE FILL PRODUCED. + Feeding a filled series back in is the defect that makes the whole mechanism + rot: day 2 carries day 1's guesses, day 3 carries day 2's carry of them, and + after a week of a dead feed the "yesterday" being carried forward is a + fossil of the last real publication with nothing anywhere reporting its age. + Every point still carries `provisional=True`, so the flag never lies - what + decays is the NUMBER, silently, behind a flag that is telling the truth. + + ⚠️ NO VALUE-LEVEL CHECK CAN CATCH THAT, and saying so is more useful than a + guarantee that would not hold. A carried point and a published-provisional + point (`eneco_eddy`'s current quarter, §9.2) are the same three fields with + the same flag; the difference is provenance, and provenance is not in the + value. So the mechanism is ORDERING, not validation: `carry()` is the only + public entry point, it remembers the argument and returns the fill, and the + filled series exists only as a return value that `carry()` itself never + sees again. `_remember()` is private, is called from exactly one place, and + `test_price_history.py` PARSES every shipped module under `app/`, `tools/`, + `scenarios/` and `estop/` and fails if any of them uses the NAME - an + attribute, a bare name, a def, or the string constant a `getattr` needs. By + name rather than by substring, so a comment about the rule does not fail it + and a call from `tools/` cannot pass unseen. + + ⚠️ AND THE ORDER IS GRADED BY DRIVING IT, NOT BY READING THE SOURCE. The + first version of that check compared positions in `ast.walk`, which is + breadth-first: `self._remember(...)` is a statement and `self.yesterday(...)` + is an ARGUMENT of the fill, one level deeper, so `_remember` came out first + whichever statement was written first - shipped and reversed scored the + same. What grades it now is the behaviour: a feed carrying yesterday + alongside today, into an EMPTY store, must fill the whole horizon, and + reversed it fills nothing. + +2. "YESTERDAY" MEANS YESTERDAY, NOT "THE NEWEST DAY I HAVE". + `newest()` is the natural accessor and it is the one that decays: an add-on + down from Friday to Monday would answer Friday's prices to a fill that flags + them as one day old. The lookup is BY DATE - the Brussels day before the + Brussels day of the instant asked about - and a record for any other date is + NO_RECORD with a reason naming both dates. A three-day-old profile is not a + worse yesterday; it is not a yesterday. + +3. A STORE THAT CANNOT ANSWER SAYS SO, IN THE TYPE. + `yesterday()` returns a `PriceRead`: a `series` when a full record exists, + `no_data(reason)` for every other outcome - no file, no record for that + date, an unreadable file, a record that parses to nothing. There is no + empty-series answer and no `.get(..., [])`-shaped accessor, for + `price.py`'s reason: an empty series is how "the cache is broken" gets to + wear the shape of "yesterday was free". `fill_from_yesterday()` takes a + `PriceRead` and handles `no_data` on its own, so the degraded case needs no + branch at the call site - it fills nothing and the horizon stays short, + which is exactly the behaviour that shipped before this ticket. + +4. HOURS ACCUMULATE WITHIN A DAY; THEY DO NOT REPLACE IT. + A feed that serves only the REMAINING hours of the day (a legitimate shape - + nothing in the plugin contract promises a whole day) would, under + replace-the-record, overwrite a complete 24-hour profile with the two points + left at 22:00 and hand the next morning a yesterday with two hours in it. So + a record is MERGED by instant: a later read updates the hours it carries and + leaves the rest, and a revised price for an hour already held wins because + it is the newer statement about the same hour. + +⚠️ KEYED BY INSTANT, NOT BY LOCAL HOUR, AND THAT IS THE DST DECISION. Storing +`hour -> price` would collapse the two 02:00s of a 25-hour October day into +one, inventing a second and different collapsing rule beside the one +`fill_from_yesterday()` already documents ("the first of its two 02:00 points +is the one used, deterministically"). Storing the instants keeps both, in +order, and lets the fill's own rule be the only one in the system. + +WHAT THIS DOES NOT DO, ON PURPOSE +================================= +It does not extend the fill to a feed that published NOTHING. `resolve_tariff()` +applies `carry()` only on a `series` read; a `no_data` read still falls to +`_degrade()` and §11.4's row - "No price data -> fall back to the static +profile, mark the plan provisional, no arbitrage" - which is the fallback the +operator configured `static_profile` FOR. Preferring yesterday's feed over it +would change a documented degradation without being asked, and it would buy +nothing measurable: a fully-carried series is 100 % provisional, so +`arbitrage()` finds fewer than two firm hours and refuses, and `cheap_now()` is +False, exactly as on the §11.4 fallback. The two differ only in which numbers +the status page shows. `fill_from_yesterday()` handles a `no_data` read anyway, +and that branch is therefore not reached from production - it stays checked, +and turning it on is a product decision, not a wiring one. See +DECISIONS/FCST-04-the-day-that-is-carried.md. +""" + +import json +import logging +import os +import tempfile +from datetime import datetime, timedelta, timezone + +from .price import PricePoint, PriceProduct, PriceRead, brussels +from .price_forecast import fill_from_yesterday + +_LOG = logging.getLogger("goodwe.price.history") + +# How many Brussels days of published prices the SORT-ORDER cap keeps. The fill +# needs exactly one - yesterday - and rule 2 refuses anything older, so this is +# not a horizon: it is the number of days a record may sit unused before +# pruning removes it, and it is small because a bigger cache would only hold +# days no lookup can ask for. Three covers today, yesterday, and the day a +# fetch spanning midnight writes ahead of itself. +# +# ⚠️ IT IS A CAP, NOT THE RETENTION RULE, AND THREE HAS NO SLACK. The steady +# state is already {yesterday, today, tomorrow}; a feed that publishes to D+2, +# or ONE stray point in the wrong year - which sorts LAST and is therefore +# immortal under a newest-N cap - pushes yesterday out of a three-slot store +# and keeps it out, permanently and silently. So `_prune()` PINS the two days a +# lookup can actually ask for on top of the cap; see its docstring. The store +# is then bounded at KEEP_DAYS + 2 records rather than at KEEP_DAYS. +KEEP_DAYS = 3 + +SCHEMA = 1 + + +def _bdate(instant: datetime) -> str: + """The Brussels calendar date of an instant, as `YYYY-MM-DD`. + + ⚠️ BRUSSELS, NOT UTC. 23:30 UTC in July is already tomorrow in Belgium, and + a cache keyed on the UTC date would file the last 90-120 minutes of every + day under the wrong one - so a lookup on the following morning would find a + "yesterday" missing its own evening hours, which are the expensive ones. + """ + return brussels(instant).strftime("%Y-%m-%d") + + +def _yesterday_of(instant: datetime) -> str: + """The Brussels date of the day BEFORE `instant`'s Brussels day. + + ⚠️ THE ARITHMETIC IS ON THE LOCAL DATE, NOT ON THE INSTANT, AND THAT IS A + DST DEFECT THIS FILE SHIPPED. `brussels()` renders at the FIXED offset in + force at that instant, so `brussels(stamp) - timedelta(days=1)` subtracts 24 + ABSOLUTE hours from a wall clock - and the Brussels day before a + spring-forward is 23 hours long. Measured at 2026-03-30T00:30+02:00, the + first hour of the morning after the March transition: it returned + 2026-03-28, so the fill carried a TWO-DAY-OLD profile flagged as yesterday's + (or, with only the last three days held, answered `no_data` naming a date + that is not yesterday). + + A calendar day minus one day is `date` arithmetic, and a `date` has no + offset to lose. The 25-hour October day is right for the same reason. + """ + return (brussels(instant).date() - timedelta(days=1)).strftime("%Y-%m-%d") + + +def _n_points(rec) -> int: + """How many points a day record holds. 0 for anything that is not one. + + ⚠️ TOLERANT, AND IT IS THE SECOND LINE OF DEFENCE, NOT THE FIRST. `_load()` + drops a malformed day record and says so, so this should never see one. It + is written this way because `words` is read from `Controller.__init__` and + from every status render: a status LINE that raises takes the add-on down, + and "the cache is broken" must never be able to mean "the controller does + not start". That is exactly what FCST-04 shipped - `{"days": {"2026-01-14": + "oops"}}` gave `ok=True`, `last_error=None`, and an `AttributeError` out of + `words`, `yesterday()` and `resolve_tariff()` alike, with no self-heal + because `_remember()` raised before it could rewrite the file. + """ + if not isinstance(rec, dict): + return 0 + pts = rec.get("points") + return len(pts) if isinstance(pts, (list, tuple)) else 0 + + +class PriceHistory: + """The published price series of the last few Brussels days, in /data. + + ⚠️ `ok` IS ABOUT THE FILE, NOT ABOUT THE DATA. A healthy store with no + record for yesterday is `ok` and answers `no_data` - a normal state on a + site's first day, and a different thing from a store that could not be + opened. Both are visible: `ok`/`last_error` describe the file, and + `yesterday()`'s reason describes the answer. + """ + + def __init__(self, path: str = "/data/price_history.json"): + self.path = path + self.ok = False + self.last_error: str | None = None + self._days: dict = {} + self._open() + + # -- the file ------------------------------------------------------------ + def _open(self) -> None: + parent = os.path.dirname(self.path) + # See the module note: NOT created. A cache on the ephemeral layer is + # one that forgets without saying so. + if parent and not os.path.isdir(parent): + self.last_error = f"no durable volume at {parent}" + # INFO, not ERROR: on a dev host or in a test container there is no + # /data and nothing is wrong. + _LOG.info("no price history: %s - §8.3 carry-forward has nothing " + "to carry and the horizon stays as short as the feed", + self.last_error) + return + self.ok = True + self._load() + + def _load(self) -> None: + try: + with open(self.path, "r", encoding="utf-8") as fh: + blob = json.load(fh) + except FileNotFoundError: + _LOG.info("no price history yet at %s - the first refresh writes " + "one, and the fill has nothing to carry until the day " + "after that", self.path) + return + except (json.JSONDecodeError, OSError) as err: + # A corrupt cache must not stop the controller, and must not be + # silently re-created either: the record is discarded, the reason + # is kept, and `yesterday()` reports it rather than answering + # "no record", which would read as a normal first day. + self.last_error = f"unreadable ({err})" + _LOG.warning("price history at %s is %s - starting from empty. " + "§8.3 carry-forward has nothing to carry until the " + "next refresh has written a full day", + self.path, self.last_error) + return + found = blob.get("version") if isinstance(blob, dict) else None + # ⚠️ THE TYPE IS PART OF THE VERSION. `1.0 == 1` in Python, so a float + # written by some other tool would pass a bare `!=` and be read as + # schema 1 - a version check that cannot tell 1 from 1.0 is one that + # would not notice a writer this build has never met. + if not isinstance(blob, dict) or isinstance(found, bool) \ + or not isinstance(found, int) or found != SCHEMA: + self.last_error = f"schema {found!r}, not {SCHEMA}" + _LOG.warning("price history at %s has %s - discarding it rather " + "than guessing what its points mean", + self.path, self.last_error) + return + days = blob.get("days") + if not isinstance(days, dict): + self.last_error = "no `days` object" + _LOG.warning("price history at %s has %s", self.path, self.last_error) + return + self._days = self._usable_days(days) + + def _usable_days(self, days: dict) -> dict: + """The day records that are shaped like day records. The rest are dropped. + + ⚠️ THE ASYMMETRY THAT PRODUCED THE DEFECT: `_parse_points()` below is + carefully defensive about every field of a POINT, and `_load()` was not + defensive about the record holding them at all. `{"version": 1, "days": + {"2026-01-14": "oops"}}` therefore loaded as `ok=True`, + `last_error=None` - and then `rec.get(...)` raised `AttributeError` out + of `words`, `yesterday()` and `resolve_tariff()`. `PriceHistory(...)` + and `.words` are both unguarded in `Controller.__init__`, so THE ADD-ON + DID NOT START; and it never self-healed, because `_remember()` raised on + the same record before it could reach `_save()`. + + A broken cache is a short horizon. It is not a controller that will not + regulate a battery, and §6's rule - absent machinery must not disable + the loop - does not stop applying because the machinery is a JSON file. + + Dropped, not repaired: an unusable record has nothing in it to keep, and + the next `_remember()` writes a fresh one under that date. + """ + kept, dropped = {}, [] + for key, rec in days.items(): + if not isinstance(key, str) or not isinstance(rec, dict) \ + or not isinstance(rec.get("points"), list): + dropped.append(str(key)) + continue + kept[key] = rec + if dropped: + # Recorded, not silent: `yesterday()` and `words` both report + # `last_error`, and a cache that quietly discards half a file reads + # to an operator exactly like a normal first day. + self.last_error = (f"{len(dropped)} unusable day record(s) dropped " + f"({', '.join(sorted(dropped))})") + _LOG.warning("price history at %s: %s - the controller starts " + "anyway and the horizon is as short as the feed makes " + "it", self.path, self.last_error) + return kept + + def _save(self) -> None: + if not self.ok: + return + # ⚠️ THE TEMP FILE IS REMOVED ON EVERY FAILING PATH. A write that dies + # between `mkstemp` and `os.replace` used to leave its temp behind, and + # the refresh task runs 48 times a day: a persistent write failure - + # a full volume being the likeliest cause - leaked 48 inodes a day onto + # the volume that was already the problem. + tmp = None + try: + fd, tmp = tempfile.mkstemp(dir=os.path.dirname(self.path)) + with os.fdopen(fd, "w", encoding="utf-8") as fh: + json.dump({"version": SCHEMA, "days": self._days}, fh, indent=1) + # Atomic: a power cut mid-write must not leave a truncated file + # that parses as an empty day. + os.replace(tmp, self.path) + tmp = None # it IS the file now; do not unlink + except OSError as err: + self.last_error = f"could not write ({err})" + _LOG.error("could not persist price history: %s", err) + finally: + if tmp is not None: + try: + os.unlink(tmp) + except OSError: # nothing further to try, and a + pass # failed cleanup is not a fault + + # -- the one public entry point ----------------------------------------- + def carry(self, read: PriceRead, *, since: datetime, until: datetime, + stamp: datetime) -> PriceRead: + """Remember what this read published, then §8.3-fill the missing hours. + + THE ORDER IS DELIBERATE AND IS THE POINT OF THE METHOD. The read is + recorded FIRST, so a feed that hands back yesterday alongside today + refreshes yesterday's record from live data before it is consumed as + the profile; and the filled series - the one thing that must never be + recorded - exists only as this method's return value, which this object + never sees again. See rule 1, including how that order is graded. + + Returns whatever `fill_from_yesterday()` returns: a series over the + whole hours of [since, until), carrying EVERY point the provider + published inside it - by identity, and four of them in one hour on a + quarter-hourly feed - plus one carried point for each remaining whole + hour FOR WHICH YESTERDAY HELD THAT WALL-CLOCK HOUR. An hour neither + source can price is left out (§11.4 for that hour), and `no_data` comes + back when that is every hour. + """ + self._remember(read, stamp) + return fill_from_yesterday(read, yesterday=self.yesterday(stamp), + since=since, until=until, stamp=stamp) + + # -- the halves, private because rule 1 is an ordering ------------------ + def _remember(self, read: PriceRead, stamp: datetime) -> None: + """File a provider read's points under their own Brussels days. + + ⚠️ PRIVATE, AND CALLED FROM EXACTLY ONE PLACE. A caller that can reach + this can record a filled series, which is the decay in rule 1. There is + no value-level defence against that - a carried point and a published + provisional point are the same three fields - so the defence is that + nothing outside this file names it, and `test_price_history.py` checks + the tree for that rather than trusting the underscore. + """ + if not self.ok: + # ⚠️ A STORE THAT CANNOT PERSIST RECORDS NOTHING. See the module + # docstring's third warning: filling `self._days` here would give a + # site with no /data an in-memory yesterday that answers a series + # all day and is gone on the next restart - a cache that forgets + # without saying so, which is the one thing that rule refuses. The + # honest degraded state is `words` = "unavailable" and `yesterday()` + # = `no_data`, and those two must agree. + return + points = read.match(series=lambda r: r.points, no_data=lambda _r: ()) + if not points: + return + product = read.match(series=lambda r: r.product.value, + no_data=lambda _r: None) + provider = read.match(series=lambda r: r.provider, + no_data=lambda _r: None) + + touched = set() + for p in points: + day = _bdate(p.start) + rec = self._days.setdefault( + day, {"product": product, "provider": provider, "points": []}) + # Rule 4: MERGE by instant. A later read updates the hours it + # carries and leaves the rest; a revised price for an hour already + # held wins, because it is the newer statement about that hour. + raw = rec.get("points") + held = {q["start"]: q for q in (raw if isinstance(raw, list) else ()) + if isinstance(q, dict) and "start" in q} + held[p.start.isoformat()] = p.canonical() + rec["points"] = sorted(held.values(), key=lambda q: q["start"]) + rec["product"], rec["provider"] = product, provider + touched.add(day) + + self._prune(stamp) + self._save() + _LOG.debug("price history: %d point(s) filed under %s; %d day(s) held", + len(points), ", ".join(sorted(touched)), len(self._days)) + + def _prune(self, stamp: datetime) -> None: + """Drop records no lookup can reach. A sort-order cap, plus two pins. + + ⚠️ NOTHING IS DELETED BY ARITHMETIC AGAINST THE CLOCK. A clock that + jumped - the failure mode WITNESS-03 spent a session on - would + otherwise delete every record in one pass and leave the fill with + nothing on the morning after, which is the one morning it is for. + Deleting on a bad clock is unrecoverable; holding a few days too many + is not, so the cap is `sorted(...)[-KEEP_DAYS:]` and stays that way. + + ⚠️ AND A CAP ALONE IS DEFEATED BY ONE POINT IN THE WRONG YEAR. Sorting + is lexicographic on the date string, so a stray far-future record sorts + LAST and is immortal - it consumes a slot for good, and with the steady + state already {yesterday, today, tomorrow} filling all three, yesterday + is what falls out. A feed that publishes to D+2 does the same thing + without any stray at all. So the two days `yesterday()` can ACTUALLY be + asked for are pinned on top of the cap. + + `stamp` is READ here, which it was not before: the pin is the only use + of it, and it is additive. Pinning can only RETAIN a record, never + delete one, so a jumped `stamp` pins two dates that are not held and + changes nothing - the WITNESS-03 property is preserved by construction + rather than by remembering to check it. + """ + if len(self._days) <= KEEP_DAYS: + return + keep = set(sorted(self._days)[-KEEP_DAYS:]) + keep |= {_bdate(stamp), _yesterday_of(stamp)} + for day in [d for d in self._days if d not in keep]: + del self._days[day] + + def yesterday(self, stamp: datetime) -> PriceRead: + """The published series of the Brussels day BEFORE `stamp`'s, or no_data. + + ⚠️ THE DATE IS COMPUTED, NOT LOOKED UP. See rule 2: `newest()` would + answer Friday's prices to a Monday fill, and the fill would flag them as + one day old because that is the only thing its flag can say. + """ + if stamp.tzinfo is None or stamp.utcoffset() is None: + raise ValueError( + f"yesterday({stamp!r}): a naive datetime is not an instant, " + f"and 'which Brussels day was yesterday' is exactly the " + f"question a wall-clock reading cannot answer.") + want = _yesterday_of(stamp) + if self.last_error and not self._days: + return PriceRead.no_data( + f"the price history at {self.path} is {self.last_error}, so " + f"§8.3 has no profile to carry forward for {want}; the plan " + f"covers only the hours the feed published") + rec = self._days.get(want) + if rec is None: + have = ", ".join(sorted(self._days)) or "no days at all" + return PriceRead.no_data( + f"no published prices recorded for {want} (held: {have}), so " + f"§8.3 has no profile to carry forward. A record for another " + f"day is not a yesterday") + points, bad = _parse_points(rec.get("points")) + if not points: + why = bad or "it is empty" + return PriceRead.no_data( + f"the {want} record holds no usable point ({why}), and an " + f"empty series is not a free day") + try: + product = PriceProduct(rec.get("product")) + except ValueError: + return PriceRead.no_data( + f"the {want} record names product {rec.get('product')!r}, " + f"which is not a PriceProduct. The optimiser switches on that " + f"enum, so a record that cannot name one cannot be a series") + if bad: + _LOG.warning("price history: %s in the %s record were dropped; " + "%d point(s) carried forward", bad, want, len(points)) + return PriceRead.series(points, product, + str(rec.get("provider") or "price_history"), + stamp) + + # -- observability ------------------------------------------------------- + @property + def words(self) -> str: + """One line for the status page. Says what is held, or why nothing is. + + ⚠️ THIS MUST NOT RAISE, EVER. It is read once from `Controller.__init__` + and again on every status render, and neither call site guards it - so + an exception here is an add-on that does not start. It went out that + way: a day record that was a string instead of a dict raised + `AttributeError` from the `len(...)` below. `_load()` now drops such a + record and `_n_points()` is total, so this has two independent reasons + not to fail on a broken file. + """ + if not self.ok: + return f"unavailable - {self.last_error}" + if not self._days: + why = f" - {self.last_error}" if self.last_error else "" + return f"empty{why} - nothing to carry forward yet" + held = "; ".join(f"{d} ({_n_points(self._days[d])} h)" + for d in sorted(self._days, key=str)) + # A file that PARTLY loaded says so, or the dropped days read as days + # that were simply never written. + return held + (f" [{self.last_error}]" if self.last_error else "") + + +def _parse_points(raw) -> tuple: + """Canonical dicts -> PricePoints, and a phrase naming what was dropped. + + ⚠️ A MALFORMED POINT IS DROPPED, NOT DEFAULTED, and the drop is reported. + `float(row.get("eur_per_kwh", 0))` is the shape of the whole family of + defects this repo has been removing: a cache that lost a field would answer + "that hour was free" to a fill that flags it as an ordinary guess. + """ + if not isinstance(raw, list): + return (), "the `points` field is not a list" + out, dropped = [], 0 + for row in raw: + if not isinstance(row, dict): + dropped += 1 + continue + try: + start = datetime.fromisoformat(row["start"]) + value = float(row["eur_per_kwh"]) + flag = row["provisional"] + except (KeyError, TypeError, ValueError): + dropped += 1 + continue + if start.tzinfo is None or start.utcoffset() is None \ + or not isinstance(flag, bool): + dropped += 1 + continue + out.append(PricePoint(start=start, eur_per_kwh=value, provisional=flag)) + out.sort(key=lambda p: p.start.astimezone(timezone.utc)) + return tuple(out), (f"{dropped} malformed point(s)" if dropped else "") + + +class NoHistory: + """The explicit "this caller keeps no history" - a NAME, not a `None`. + + ⚠️ IT EXISTS SO THE PARAMETER CAN BE REQUIRED. `history=None` meaning "skip + the fill" is the shape this repo keeps removing: a caller who forgets the + argument gets the truncated-horizon behaviour of the bug this ticket is + about, silently, and the code reads as though the fill is happening. With + no default there is nothing to forget, and a caller with no durable volume + - a check, a replay harness - has to say `NoHistory()` and be visible in + the diff. + + It carries out the ordering faithfully: it records nothing and carries + nothing, so `carry()` returns the provider's read unchanged and the plan is + exactly what it was before FCST-04. + """ + + ok = False + last_error = "no history is kept by this caller" + words = "not kept" + + def carry(self, read: PriceRead, *, since, until, stamp) -> PriceRead: + return read + + def yesterday(self, stamp: datetime) -> PriceRead: + return PriceRead.no_data( + "this caller keeps no price history (NoHistory), so §8.3 has no " + "profile to carry forward") diff --git a/goodwe_controller/app/static_tariff.py b/goodwe_controller/app/static_tariff.py index 102da58..dea20e3 100644 --- a/goodwe_controller/app/static_tariff.py +++ b/goodwe_controller/app/static_tariff.py @@ -1019,7 +1019,7 @@ def static_status(opts: dict, now: datetime): async def resolve_tariff(opts: dict, since: datetime, until: datetime, - transport=None, now=None) -> TariffPlan: + transport=None, now=None, *, history) -> TariffPlan: """The full §9.2 + §11.4 resolution. MAY PERFORM I/O - see `static_plan`. ⚠️ NOT ON THE 30 s TICK, and that is a deliberate boundary rather than an @@ -1033,9 +1033,42 @@ async def resolve_tariff(opts: dict, since: datetime, until: datetime, price_provider off + profile -> the CONFIGURED product (§9.2 rows 3-4) price_provider off + none -> no prices at all (§11.4, nothing to fall back to) - provider series -> the feed (§9.2 rows 1-2) + provider series -> the feed, §8.3-FILLED (§9.2 rows 1-2) provider no_data + profile -> §11.4 FALLBACK, provisional, no arbitrage provider no_data + none -> no prices at all + + ⚠️ `history` IS REQUIRED, KEYWORD-ONLY, AND HAS NO DEFAULT (FCST-04). It is + the §8.3 carry-forward store - `price_history.PriceHistory`, or the explicit + `price_history.NoHistory()` for a caller that keeps none. A default of + `None` meaning "skip the fill" would give a caller who forgot the argument + the TRUNCATED HORIZON this ticket exists to remove, silently, while the code + read as though the fill were happening. There is nothing to forget now. + + ⚠️ AND THE FILL RUNS ONLY ON A `series` READ. A feed that published nothing + still falls to `_degrade()` and §11.4's own row - the static profile the + operator configured FOR that case - rather than to yesterday's feed. The two + are indistinguishable to the optimiser (both are 100 % provisional, so + `arbitrage()` finds fewer than two firm hours and refuses, and `cheap_now()` + is False), so preferring yesterday would change a documented degradation for + a difference only the status page can see. See price_history.py's closing + note and DECISIONS/FCST-04-the-day-that-is-carried.md. + + ⚠️ THE FILL IS PER WHOLE HOUR AND ITS COVERAGE IS CONDITIONAL. Every point + the provider published inside the window passes through BY IDENTITY - four + of them in one hour on a quarter-hourly product like `eneco_eddy` - and a + carried point is added only for a whole hour the feed did not print at all + AND yesterday did. An hour neither source can price is left out, which is + §11.4 for that hour: `plan.price_at()` raises rather than answering, and an + hour with no price is not a free one. + + ⚠️ THE FILLED SERIES IS WINDOWED TO [since, until), AND THAT IS A CHANGE. + `load_prices()` returns whatever the feed sent, past hours included, and + `TariffPlan.arbitrage()` took its spread over all of it - so an hour that + had already gone could be the cheapest hour of a cycle nothing can now buy + in. `fill_from_yesterday()` walks the requested horizon and emits exactly + that, so the spread is over hours that are still ahead. Both directions of + the difference are real: the window gains the unpublished hours it was + missing and loses the elapsed ones it should never have had. """ stamp = now or datetime.now(timezone.utc) pid = str(price_option(opts, "price_provider") or "off").strip() @@ -1044,6 +1077,20 @@ async def resolve_tariff(opts: dict, since: datetime, until: datetime, read = await load_prices(opts, since, until, transport) return read.match( + series=lambda r: _provider_plan(opts, r, since, until, stamp, history), + no_data=lambda r: _degrade(opts, r, since, until, stamp)) + + +def _provider_plan(opts, read, since, until, stamp, history) -> TariffPlan: + """§8.3: record what the feed published, carry yesterday into the gaps. + + The fill can itself return `no_data` - a feed whose points all fall outside + the horizon, with nothing in the history to fill it from - and that is the + §11.4 row arrived at one step later, so it degrades exactly as a failed + fetch does rather than being special-cased here. + """ + filled = history.carry(read, since=since, until=until, stamp=stamp) + return filled.match( series=lambda r: TariffPlan(r, _quiet_profile(opts), "provider"), no_data=lambda r: _degrade(opts, r, since, until, stamp)) diff --git a/goodwe_controller/test_price_forecast.py b/goodwe_controller/test_price_forecast.py index 26e1194..cb2aeb6 100644 --- a/goodwe_controller/test_price_forecast.py +++ b/goodwe_controller/test_price_forecast.py @@ -272,16 +272,19 @@ check("the provenance audit REFUSES a carried hour flagged non-provisional - " "the 'stale but plausible' point, which is SAFETY-18's frozen-but-fresh " "meter one layer up", raises(lambda: _fc._audit([_stale_copy], {}), ValueError)) +# ⚠️ `published` MAPS AN HOUR TO A TUPLE, not to a point (FCST-04 H1). An hour +# may legitimately hold four published points - see the quarter-hourly block +# below - so the audit takes the whole set and checks membership by identity. check("...and refuses a published hour that arrived as a COPY rather than as " "the published object, which is where a flag gets lost", raises(lambda: _fc._audit( [PricePoint(start=TODAY_PART.points[0].start, eur_per_kwh=TODAY_PART.points[0].eur_per_kwh, provisional=TODAY_PART.points[0].provisional)], - {D0: TODAY_PART.points[0]}), ValueError)) + {D0: (TODAY_PART.points[0],)}), ValueError)) check("...and never repairs a flag: the audit's only outcomes are silence and " "an exception, so a series it passed was already correct", - _fc._audit(list(FILLED.points), {u: TODAY_PART.points[i] + _fc._audit(list(FILLED.points), {u: (TODAY_PART.points[i],) for i, u in enumerate(DAY[:PUBLISHED_N])}) is None) @@ -298,6 +301,81 @@ check("a PUBLISHED point that the feed itself flagged provisional keeps that " and all(p.provisional for p in _eddy_filled.points[:2]) and _eddy_filled.points[0] is _eddy.points[0]) +# ⚠️ AND A SUB-HOURLY FEED IS NOT DECIMATED TO ONE POINT AN HOUR (FCST-04 H1). +# `eneco_eddy` settles per QUARTER and `_validate_raw()` accepts any +# strictly-increasing spacing, so a published hour can hold four points. The +# fill walks WHOLE HOURS to decide what needs carrying; that must not become a +# rule about what a published hour may contain. Measured on FCST-04's first +# draft, which keyed the published set `{_utc_hour(p.start): p ...}`: a +# 96-point feed came out 15 points long and its arbitrage spread fell from +# 0.143 to 0.014 EUR/kWh, on an ALL-FIRM, fully-published series. +_QN = 16 # four hours of quarters +_QUARTERS = PriceRead.series( + [PricePoint(start=brussels(D0 + timedelta(minutes=15 * n)), + eur_per_kwh=round(0.100000 + 0.011300 * n, 6), + provisional=False) + for n in range(_QN)], + PriceProduct.REALTIME, "eneco_eddy", D0) +_QHOURS = {_fc._utc_hour(p.start) for p in _QUARTERS.points} +check(f"the quarter-hourly fixture really is sub-hourly - {_QN} points across " + f"{len(_QHOURS)} hours - or every check below passes for an hourly feed " + f"too and grades nothing", + len(_QUARTERS.points) == _QN and len(_QHOURS) == _QN // 4) +_qfill = derived(lambda: fill_from_yesterday(_QUARTERS, yesterday=YESTERDAY, + since=D0, until=D1, stamp=D0)) +_qfirm = derived(lambda: [p for p in _qfill.points if not p.provisional]) +check(f"every one of the {_QN} published points survives the fill BY IDENTITY " + f"({len(_qfirm)} kept). One-per-hour keying would leave {_QN // 4}, and " + f"the spread over the hours the feed actually published would collapse " + f"with them", + len(_qfirm) == _QN + and all(any(p is q for q in _QUARTERS.points) for p in _qfirm)) +check(f"...and the {len(DAY) - len(_QHOURS)} whole hours it published NOTHING " + f"for are carried, ONE point each: a carried hour is hourly whatever the " + f"feed's spacing is, because yesterday's profile is only known per " + f"wall-clock hour", + _qfill.kind == "series" + and len([p for p in _qfill.points if p.provisional]) + == len(DAY) - len(_QHOURS)) +check("...and the filled series is still ordered by instant across the two " + "kinds, so `at()` and `window()` are not reading a shuffled list", + _qfill.kind == "series" + and [p.start.astimezone(UTC) for p in _qfill.points] + == sorted(p.start.astimezone(UTC) for p in _qfill.points)) + +_PUB_Q = _fc._published_by_hour(_QUARTERS) +# ⚠️ `derived`, NOT A RAW CALL. Under the decimating mutation this check +# exists for, `_audit` RAISES here - and a harness that dies on the mutation it +# is grading reports a traceback instead of a FAIL, which MUTATION.md forbids. +check("the audit ACCEPTS four published points inside one hour, which is the " + "case the repeated-hour COUNT it replaced could not express", + derived(lambda: _fc._audit(list(_QUARTERS.points), _PUB_Q)) is None) +check("...and still REFUSES a repeated CARRIED hour, which is the DST " + "mispricing that count existed to catch", + raises(lambda: _fc._audit([_fc._carried(D0, 0.11), + _fc._carried(D0, 0.12)], {}), ValueError)) +check("...and refuses the SAME published point emitted twice - the provider's " + "points are strictly increasing, so a repeat is never a second print", + raises(lambda: _fc._audit([_QUARTERS.points[0], _QUARTERS.points[0]], + _PUB_Q), ValueError)) +check("...and refuses a CARRIED point standing in an hour the feed DID " + "publish, which the old count never looked at", + raises(lambda: _fc._audit([_fc._carried(D0, 0.11)], _PUB_Q), ValueError)) + +# ⚠️ THE HORIZON IS WHOLE HOURS AT BOTH ENDS (FCST-04 L8). `since` is floored +# and `until` was not, so a 48 h request made at any instant off the hour came +# back 49 hours long. Graded at an OFF-THE-HOUR instant, because on the hour +# the two implementations agree and the check would grade nothing. +_off = D0 + timedelta(hours=8, minutes=17, seconds=43) +for _h in (1, 12, 48): + _n = derived(lambda h=_h: len(fill_from_yesterday( + TODAY_PART, yesterday=YESTERDAY, since=_off, + until=_off + timedelta(hours=h), stamp=_off).points)) + check(f"a {_h} h horizon asked for at {_off.isoformat()} - 17 min 43 s " + f"past the hour, the shape a running controller always has - is " + f"{_h} points long, not {_h + 1}", + _n == _h) + # The structural half: the constructor has nothing to pass, and the file is # scanned so the pass-through idiom cannot come back looking ordinary. check("_carried() has NO `provisional` parameter, so a caller has nothing to " diff --git a/goodwe_controller/test_price_history.py b/goodwe_controller/test_price_history.py new file mode 100644 index 0000000..c547f1b --- /dev/null +++ b/goodwe_controller/test_price_history.py @@ -0,0 +1,1356 @@ +"""Runnable check for FCST-04 - the day the fill carries. `python3 test_price_history.py` + +WHAT THIS FILE IS DEFENDING. FCST-03 shipped `fill_from_yesterday()` and the +consumers written for it, and shipped NO CALLER. This file grades the caller: +that a partially-published day-ahead feed now produces a plan covering the whole +requested horizon, that the day it carries survives a restart, and that the +cache cannot decay by being fed its own output. + + AC1 THE HOLE IS CLOSED. `resolve_tariff()` on a feed that has published 10 + of 48 hours returns a series covering 48, the 38 unpublished hours are + provisional and the 10 published ones are the provider's own objects, + unchanged and unflagged. Graded against the same call with + `NoHistory()`, which is the pre-FCST-04 behaviour, so the difference is + MEASURED rather than described. + AC2 THE WINDOW STOPPED INCLUDING THE PAST, which is the half of this change + that moves energy. `load_prices()` returns whatever the feed sent, so at + 09:00 a spread was taken over hours that had already elapsed and the + "cheapest hour" could be one no cycle can buy in - the trade is then + silently never taken. The filled series is windowed to [since, until), + so the firm cheapest hour is a hour still ahead. Graded by finding the + cheapest firm hour both ways and showing it MOVES. + AC3 THE CACHE CANNOT DECAY. What is recorded is the provider's read and + never the filled series; if it were, day 2 would carry day 1's guesses + and day 3 would carry day 2's carry of them, behind a `provisional` flag + that stays true while the NUMBER rots. Graded by driving `carry()` twice + and reading the file back. + AC4 A STORE THAT CANNOT ANSWER SAYS SO. No file, no record for yesterday, an + unreadable file, a schema this build does not know, A DAY RECORD THAT IS + NOT A RECORD, a record whose points are malformed - six different + reasons, none of them a price, none of them an empty series, and the + controller keeps its pre-FCST-04 plan through all of them. ⚠️ THE + CRASHING ONE IS THE POINT: `{"days": {"2026-01-14": "oops"}}` loaded as + healthy and then raised `AttributeError` out of every accessor, and both + `PriceHistory(...)` and `.words` are unguarded in `Controller.__init__` + - so the add-on did not start, and never self-healed. + +THE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will: + + * EVERY VERDICT IS DERIVED. The carried/published split is re-derived here + from the set of hours the fixture published, by different arithmetic than + app/ uses. The horizon growth is a subtraction of two measured counts, not + a constant. The cheapest-hour move is found by looking, not asserted. + + * NO STIMULUS IS EVALUATED WHERE THE QUANTITY IT GRADES IS DEGENERATE: + - YESTERDAY AND TODAY MUST PRICE EVERY LOCAL HOUR DIFFERENTLY, or a fill + that carried the wrong day is invisible. Asserted first, as a check. + - THE PUBLISHED WINDOW MUST BE A STRICT SUBSET OF THE HORIZON, or "the + fill widened it" and "the fill did nothing" are the same observation. + - THE ELAPSED HOURS MUST CONTAIN THE CHEAPEST HOUR OF THE WHOLE FEED, or + AC2 grades nothing: the windowed and unwindowed cheapest hour would be + the same point either way. This is also the realistic shape of the + danger - a cheap night that has already gone, holding a battery out of + a trade all morning. + - THE MERGE FIXTURE MUST REVISE A PRICE TO A DIFFERENT NUMBER, or merge + and replace are indistinguishable. + - THE PRUNE FIXTURE MUST HOLD MORE DAYS THAN KEEP_DAYS, or pruning is not + exercised at all. ⚠️ THIS FILE BROKE ITS OWN RULE AND IT COST A LIVE + MUTATION: §8's clock-jump check seeded ONE day, `_prune`'s early + return fired, and the body it is named after never ran. The mutant that + pruned by arithmetic against the clock - which empties the store on + exactly the morning the fill is for - survived the whole 29-file suite. + The fixture is now WRITTEN TO A FILE and reopened, because `_load()` + does not prune and that is how a store legitimately holds more days + than the cap. + - NOW MUST NOT BE ON AN HOUR BOUNDARY. `datetime.now()` never is, and a + horizon count taken at exactly :00:00 was green for a fill that emitted + HORIZON + 1 hours at every other instant. + - A SUB-HOURLY FIXTURE MUST REALLY BE SUB-HOURLY, or "every published + print survives" passes for an hourly feed and grades nothing. + + * A FAILED READ IS NOT AN OBSERVATION. Every degraded store is graded on the + TYPE it answers with (`no_data` and its reason), never on a falsy value, + and the malformed-point case is checked for the specific defect - a price + defaulted to 0.0 - rather than for a count. + + * AND A STRUCTURAL CHECK MUST BE ABLE TO FAIL. §3's ordering check compared + `ast.walk` positions, which grade NESTING DEPTH: `self._remember(...)` is a + statement and `self.yesterday(...)` is an argument of the fill, so + `_remember` came out first whichever statement was written first - measured + by reversing them and watching the check stay green, while DECISIONS §3 and + the module docstring both cited it as making a reversal "red rather than + plausible". It is now line numbers AND the behaviour itself. + +⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run +`tools/mutation_run.sh`, which does. See MUTATION.md and TEST-15. +""" + +import ast +import asyncio +import inspect +import json +import os +import shutil +import sys +import tempfile +from datetime import datetime, timedelta, timezone + +from app import price_history as _ph +from app.price import ( + Absence, PricePoint, PriceProduct, PriceRead, brussels, +) +from app.price_history import KEEP_DAYS, NoHistory, PriceHistory, SCHEMA +from app.static_tariff import resolve_tariff + +UTC = timezone.utc +HERE = os.path.dirname(os.path.abspath(__file__)) +fails = [] +_tmpdirs = [] + + +def check(name, cond): + print(f" {'ok ' if cond else 'FAIL'} {name}") + if not cond: + fails.append(name) + + +def why(read): + """A read's reason, or "" when it is a series. + + ⚠️ NOT `read.reason`. On a `_Series` that attribute RAISES `Absence` by + design, so a mutation that makes a refusing accessor answer a series would + take this file down with a traceback instead of reporting a FAIL - and + MUTATION.md's rule is that "it was refused" and "it broke on the way to + being refused" must not grade the same. + """ + return read.match(series=lambda _r: "", no_data=lambda r: r.reason) + + +def raises(fn, exc): + try: + fn() + except exc: + return True + except Exception: + return False + return False + + +class _Raised: + """What a check saw instead of a value. Equals nothing, is falsy, and is + NOT a default.""" + + def __init__(self, e): + self.e = e + + def __eq__(self, other): + return False + + def __hash__(self): + return id(self) + + def __bool__(self): + return False + + def __repr__(self): + return f"" + + +def derived(fn): + """A value a check grades. An exception becomes `_Raised`. + + ⚠️ NOT A DEFAULT, AND NOT A BARE CALL EITHER. MUTATION.md's rule is that + "it was refused" and "it broke on the way to being refused" must not grade + the same - and the accessors below are exactly the ones the unvalidated + day record used to raise from, so a raw call here would take the harness + down on the mutation it is meant to catch and report nothing at all. + Copied from test_price_forecast.py. + """ + try: + return fn() + except Exception as e: # noqa: BLE001 + return _Raised(e) + + +def run(coro): + return asyncio.run(coro) + + +def tmpdir(): + d = tempfile.mkdtemp(prefix="fcst04-") + _tmpdirs.append(d) + return d + + +def store(): + """A history on a fresh directory that looks durable.""" + return PriceHistory(os.path.join(tmpdir(), "price_history.json")) + + +# --------------------------------------------------------------------------- # +# The fixtures. +# +# ⚠️ A DAY IS A BRUSSELS DAY, BUILT BY SEARCH. The first draft of this file made +# a day 24 UTC hours long and every count in it came out one short: in January +# Belgium is UTC+1, so 23:00 UTC already belongs to tomorrow, and a "24-hour +# day" built by adding hours to midnight UTC is 23 hours of one Brussels day +# plus one hour of the next. That is the same class of error the module is +# keyed to avoid, so the fixture must not contain it - and it is found by +# LOOKING at each instant's Brussels date rather than by adding an offset a +# helper would have to be right about. +# +# ⚠️ THE INSTANT IS 09:00 BRUSSELS ON A WEEKDAY, WHICH IS THE WHOLE POINT. The +# day-ahead auction publishes at 14-16 h CET, so at this instant a healthy feed +# is serving TODAY IN FULL and nothing of tomorrow - the window this ticket +# exists for. A fixture at 20:00 would have both days and would grade nothing. +# --------------------------------------------------------------------------- # + +TODAY_D = "2026-01-15" # a Thursday; Belgium on CET +YDAY_D = "2026-01-14" +HORIZON_H = 48 + + +def bday_hours(bdate: str) -> list: + """The UTC instants whose BRUSSELS date is `bdate`, one per local hour.""" + probe = datetime.fromisoformat(bdate + "T00:00:00+00:00") - timedelta(hours=3) + out = [] + for n in range(30): + u = probe + timedelta(hours=n) + if brussels(u).strftime("%Y-%m-%d") == bdate: + out.append(u) + return out + + +TODAY_H = bday_hours(TODAY_D) +YDAY_H = bday_hours(YDAY_D) + + +def at_local(bdate: str, hh: int) -> datetime: + """The UTC instant whose Brussels wall clock reads hh:00 on `bdate`.""" + return next(u for u in bday_hours(bdate) if brussels(u).hour == hh) + + +# ⚠️ AND IT IS NOT ON THE HOUR. A controller's `now` is +# `datetime.now(timezone.utc)`; it is :17:43 past as often as anything else and +# it is never :00:00. The first draft of this file used the exact hour, and +# §1's `_n_after == HORIZON_H` was green only because of it: `since` was +# floored to its hour while `until` was compared raw, so a 48 h request made at +# any other instant came back 49 hours long. A fixture has to have the shape +# production has, or a count grades an alignment the running system never gets. +NOW = at_local(TODAY_D, 9) + timedelta(minutes=17, seconds=43) + +# Where the WINDOW starts: the whole hour that contains NOW. The hour a +# controller is standing in is still a tradeable hour, so the fill emits from +# here and not from NOW - and every count below is derived against this rather +# than against the instant. +SINCE_HOUR = NOW.replace(minute=0, second=0, microsecond=0) + +# Today's feed, in EUR/MWh, keyed by BRUSSELS local hour and all distinct. +# ⚠️ THE CHEAPEST HOUR IS 03:00 LOCAL - ALREADY ELAPSED AT 09:00 - and that is +# AC2's stimulus, asserted below rather than assumed. +TODAY_MWH = {h: 120.0 + h for h in range(24)} +TODAY_MWH[3] = 5.0 # the cheap night that has already gone +TODAY_MWH[9] = 250.0 # the hour NOW falls in - dear, so the + # cheapest hour AHEAD is strictly + # later than NOW and "it moved to an + # hour still ahead" is exact +TODAY_MWH[19] = 260.0 # the dear evening still ahead + +# Yesterday's published prices, DIFFERENT AT EVERY LOCAL HOUR from today's, so +# a fill that carried the wrong day is visible. +YESTERDAY_MWH = {h: 400.0 + 3 * h for h in range(24)} + +MARKUP, TAXES = 0.03, 0.06 + + +def allin(mwh): + """EUR/MWh as published -> the all-in EUR/kWh `assemble()` produces.""" + return round(mwh * 0.001 + MARKUP + TAXES, 6) + + +def feed_body(instants, table): + return json.dumps({"data": [ + {"start": u.isoformat(), "price_eur_mwh": table[brussels(u).hour]} + for u in instants]}).encode() + + +def transport_for(body): + async def _t(url, timeout_s): + return body + return _t + + +CFG = {"price_provider": "generic_dayahead", + "price_feed_url": "https://api.energy-charts.info/price?bzn=BE", + "price_timeout_s": 1, + "price_markup_eur_kwh": MARKUP, + "price_taxes_eur_kwh": TAXES, + "static_profile": ""} + + +def series_of(instants, table, product=PriceProduct.DAY_AHEAD, + provider="generic_dayahead", stamp=None): + """A PriceRead.series built directly - the shape `load_prices()` returns.""" + return PriceRead.series( + [PricePoint(start=brussels(u), + eur_per_kwh=allin(table[brussels(u).hour]), + provisional=False) + for u in instants], + product, provider, stamp or NOW) + + +# --------------------------------------------------------------------------- # +print("0. the arithmetic this file grades with must not be degenerate") + +check(f"a Brussels day is 24 hours long in the fixture helper, on both fixture " + f"days ({len(TODAY_H)}, {len(YDAY_H)}) - a 23-hour 'day' built by adding " + f"to midnight UTC is what made every count below come out one short", + len(TODAY_H) == 24 and len(YDAY_H) == 24 + and {brussels(u).hour for u in TODAY_H} == set(range(24))) + +check(f"NOW is {NOW.isoformat()} - deliberately NOT on an hour boundary. A " + f"horizon count taken at exactly :00:00 is green for a fill that emits " + f"HORIZON + 1 hours at every instant a controller actually runs at", + (NOW.minute, NOW.second, NOW.microsecond) != (0, 0, 0)) + +# The feed serves TODAY IN FULL - which is what a day-ahead feed does - so the +# hours it publishes that are still AHEAD are a strict subset of the horizon. +# ⚠️ AGAINST SINCE_HOUR, NOT NOW: the 09:00 point started before NOW and its +# hour still contains NOW, so it is a published hour of the window. +_ahead = [u for u in TODAY_H if u >= SINCE_HOUR] +check(f"at 09:00 the hours the feed published that have NOT elapsed are a " + f"STRICT SUBSET of the {HORIZON_H} h horizon ({len(_ahead)} of " + f"{HORIZON_H}), so 'the fill widened the plan' and 'the fill did " + f"nothing' are different observations", + 0 < len(_ahead) < HORIZON_H) + +check("yesterday prices EVERY local hour differently from today, so a fill " + "that carried the wrong day is visible in the numbers", + all(TODAY_MWH[h] != YESTERDAY_MWH[h] for h in range(24))) + +check("...and no fixture price is zero, so 'dropped the price' is visible too", + all(TODAY_MWH.values()) and all(YESTERDAY_MWH.values())) + +_cheapest_all = min(TODAY_H, key=lambda u: TODAY_MWH[brussels(u).hour]) +_cheapest_ahead = min(_ahead, key=lambda u: TODAY_MWH[brussels(u).hour]) +check(f"AC2's stimulus is real: the cheapest hour of the whole published day is " + f"{brussels(_cheapest_all).hour:02d}:00 local, which has already elapsed " + f"at 09:00, and the cheapest hour still AHEAD is a different one " + f"({brussels(_cheapest_ahead).hour:02d}:00). Without this the windowed " + f"and unwindowed answers would be the same point and AC2 would grade " + f"nothing", + _cheapest_all != _cheapest_ahead and _cheapest_all < SINCE_HOUR + and _cheapest_ahead > NOW) + +check(f"KEEP_DAYS is {KEEP_DAYS}, and the prune fixture below files more days " + f"than that, so pruning is exercised rather than skipped", KEEP_DAYS < 6) + + +# --------------------------------------------------------------------------- # +print() +print("1. AC1 - the hole: a partially-published feed now covers the horizon") + +# ⚠️ THE FEED PUBLISHES TODAY IN FULL, WHICH IS WHAT A DAY-AHEAD FEED DOES. +# The partial publication this ticket is about is that TOMORROW is missing, not +# that this morning is - and the elapsed hours the feed is still serving are +# exactly what AC2 is about, so they have to be in the body. +_body_today = feed_body(TODAY_H, TODAY_MWH) +_until = NOW + timedelta(hours=HORIZON_H) + +# The pre-FCST-04 behaviour, by the shipped code path: NoHistory() records +# nothing and carries nothing, so this IS the old plan. +_before = run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), now=NOW, + history=NoHistory())) + +# And the same call with a history that already holds yesterday. +_h1 = store() +_h1_seed = series_of(YDAY_H, YESTERDAY_MWH) +_ph.PriceHistory._remember(_h1, _h1_seed, NOW) # seeding, not a caller +_after = run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), now=NOW, + history=_h1)) + +_n_before = len(_before.read.points) +_n_after = len(_after.read.points) +check(f"the plan's horizon GREW, measured: {_n_before} h before FCST-04, " + f"{_n_after} h after, against a {HORIZON_H} h request", + _n_after > _n_before and _n_after == HORIZON_H) + +check(f"...and before the change it was exactly what the feed sent " + f"({len(TODAY_H)} h of one day), which is the defect from both ends: the " + f"{HORIZON_H - len(_ahead)} hours past the feed's own horizon were not " + f"short-priced but ABSENT, and the {len(TODAY_H) - len(_ahead)} hours " + f"that had already elapsed were in the series anyway", + _n_before == len(TODAY_H)) + +# The split, re-derived here from the fixture rather than read off the module. +_published_instants = {u.astimezone(UTC) for u in _ahead} +_got_carried = {p.start.astimezone(UTC) for p in _after.read.points + if p.provisional} +_got_firm = {p.start.astimezone(UTC) for p in _after.read.points + if not p.provisional} +check("AC1: exactly the hours the feed published are FIRM, and exactly the " + "hours it did not are PROVISIONAL - the flag is a claim about " + "provenance, re-derived here from the fixture", + _got_firm == _published_instants + and _got_carried == {p.start.astimezone(UTC) + for p in _after.read.points} - _published_instants + and len(_got_carried) == HORIZON_H - len(_ahead)) + +_before_by_start = {p.start.astimezone(UTC): p for p in _before.read.points} +# ⚠️ THIS IS A VALUE CHECK AND SAYS SO. `_before` and `_after` come from two +# separate `resolve_tariff()` calls, each of which parsed the feed body into +# its OWN objects, so `is` could never hold between them however the fill were +# written - a check claiming identity here would be claiming a property its +# own fixture makes untestable. What it does grade is that the fill moved the +# WINDOW and did not touch a PRICE. +check("...and every published point carries the same NUMBER as the unfilled " + "plan's for that instant: the fill changed which hours are in the " + "series, not what any of them costs", + all(p.eur_per_kwh == _before_by_start[p.start.astimezone(UTC)].eur_per_kwh + for p in _after.read.points if not p.provisional)) + +# ...and identity itself, graded on a read this file constructed and still +# holds a reference to. FCST-03's whole anti-copy argument is +# `filled.points[i] is fetched.points[j]`, and floats cannot grade it. +_h1b = store() +_ph.PriceHistory._remember(_h1b, series_of(YDAY_H, YESTERDAY_MWH), NOW) +_read_id = series_of(TODAY_H, TODAY_MWH) +_filled_id = _h1b.carry(_read_id, since=NOW, until=_until, stamp=NOW) +_firm_id = [p for p in _filled_id.points if not p.provisional] +check(f"...and IDENTITY, graded as identity: all {len(_firm_id)} firm points " + f"of a filled series ARE the provider's own objects (`is`), not copies " + f"carrying the same numbers. A copy is where the provisional flag gets " + f"lost (FCST-03)", + len(_firm_id) == len(_ahead) + and all(any(p is q for q in _read_id.points) for p in _firm_id)) + +# The carried numbers are yesterday's, at the same BRUSSELS wall-clock hour. +# ⚠️ THE LOOKUP IS BUILT BY SEARCH over yesterday's own instants, not by +# subtracting 24 h from today's - a fill that subtracted a fixed day is exactly +# what this is meant to catch, and a helper that subtracts one cannot catch it. +_Y_BY_LOCAL = {} +for _u in YDAY_H: + _Y_BY_LOCAL.setdefault(brussels(_u).hour, + allin(YESTERDAY_MWH[brussels(_u).hour])) +_carried_ok = [_Y_BY_LOCAL.get(brussels(p.start).hour) is not None + and abs(p.eur_per_kwh + - _Y_BY_LOCAL[brussels(p.start).hour]) < 1e-9 + for p in _after.read.points if p.provisional] +check(f"...and every carried number is YESTERDAY's price for the same BRUSSELS " + f"wall-clock hour, not today's and not a neighbour's " + f"({sum(_carried_ok)}/{len(_carried_ok)})", + _carried_ok and all(_carried_ok)) + +# ⚠️ "FILLS EVERY UNPUBLISHED HOUR" IS CONDITIONAL, AND THIS IS THE CONDITION. +# A yesterday that is itself partial fills only the local hours it HOLDS; the +# rest are left out, which is §11.4 for those hours and emphatically not a +# zero. The behaviour is right and the wording was not - the CHANGELOG and DOCS +# said "every", unqualified, and nothing here exercised it. +_hp = store() +_PART_LOCAL = list(range(12)) # yesterday published 00:00..11:00 only +_ph.PriceHistory._remember( + _hp, series_of([at_local(YDAY_D, h) for h in _PART_LOCAL], YESTERDAY_MWH), + NOW) +_plan_part = run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), now=NOW, + history=_hp)) +_window_hours = [SINCE_HOUR + timedelta(hours=n) for n in range(HORIZON_H)] +_ahead_utc = {u.astimezone(UTC) for u in _ahead} +_expect_covered = {u for u in _window_hours + if u in _ahead_utc or brussels(u).hour in _PART_LOCAL} +_got_covered = {p.start.astimezone(UTC) for p in _plan_part.read.points} +check(f"a PARTIAL yesterday fills only the local hours it holds: " + f"{len(_got_covered)} of the {HORIZON_H} horizon hours are covered, " + f"re-derived from the fixture as {len(_expect_covered)}. The hours " + f"yesterday never published stay ABSENT", + _got_covered == _expect_covered and 0 < len(_got_covered) < HORIZON_H) +_gap = sorted(set(_window_hours) - _expect_covered)[0] +check(f"...and an uncovered hour ({_gap.isoformat()}) raises `Absence` rather " + f"than answering a price. §11.4 applies to that hour: a hole in the " + f"series is not a free hour, and this is why the wording had to stop " + f"saying 'every'", + raises(lambda: _plan_part.read.at(_gap), Absence)) + +check("the plan reports itself PROVISIONAL now, which it did not before - the " + "old plan claimed full firmness over a horizon it covered half of", + _after.provisional and not _before.provisional) + +check("...and the plan is still a provider plan, not a §11.4 fallback: the " + "fill does not degrade anything", + _after.origin == "provider" and _after.product is PriceProduct.DAY_AHEAD) + + +# --------------------------------------------------------------------------- # +print() +print("2. AC2 - the spread stopped being taken over hours that have gone") + +_A = dict(peak_guard_active=False, min_spread_eur_kwh=0.0, + uncertainty=frozenset()) +_v_before = _before.arbitrage(**_A) +_v_after = _after.arbitrage(**_A) +check("both plans still ALLOW a trade, so this section compares two verdicts " + "rather than a verdict and a refusal", + _v_before.kind == "allowed" and _v_after.kind == "allowed") + +_cheap_before = _v_before.cheapest.start.astimezone(UTC) +_cheap_after = _v_after.cheapest.start.astimezone(UTC) +check(f"AC2: the cheapest FIRM hour MOVED from {_cheap_before.isoformat()} - " + f"already elapsed at {NOW.isoformat()}, so no cycle could ever buy in it " + f"- to {_cheap_after.isoformat()}, which is still ahead", + _cheap_before < NOW <= _cheap_after and _cheap_before != _cheap_after) + +check("...and the new one is the cheapest hour the feed published that has NOT " + "gone, derived from the fixture rather than from the module", + _cheap_after == _cheapest_ahead.astimezone(UTC)) + +check("...and no point of the filled series starts before the requested " + "`since` hour: the window is [since, until) and the past is not in it", + min(p.start.astimezone(UTC) for p in _after.read.points) >= SINCE_HOUR) + +# ⚠️ THE HONEST LIMIT OF THIS TICKET, checked so it cannot be overclaimed +# later: the CARRIED hours buy no trade. §8.3 forbids acting on a provisional +# price and refusal 3 drops them from the trade set, so what widened is what +# the optimiser can SEE, and the trade above came from the WINDOWING, not from +# yesterday's numbers. +_firm_after = [p for p in _after.read.points if not p.provisional] +check("the carried hours are NOT in the trade set - the verdict's cheapest and " + "dearest are both firm, so this ticket widened what the optimiser sees " + "and not what it may act on", + not _v_after.cheapest.provisional and not _v_after.dearest.provisional + and len(_firm_after) == len(_ahead)) + + +# --------------------------------------------------------------------------- # +print() +print("2b. a QUARTER-HOURLY feed keeps every print, and its spread with them") + +# ⚠️ THE DEFECT THIS SECTION EXISTS FOR. `eneco_eddy` is the Belgian imbalance +# product and settles per QUARTER; `_validate_raw()` accepts any +# strictly-increasing spacing, so a published hour may hold four points. +# FCST-04's first draft keyed the published set `{_utc_hour(p.start): p ...}` - +# LAST POINT OF THE HOUR WINS, the other three dropped - and on an all-firm, +# fully-published 96-point feed the plan came out 15 points long with its +# arbitrage spread down from 0.143 to 0.014 EUR/kWh. That is not the windowing +# effect AC2 is about; it is prints being thrown away, and no flag reports it. +_QSTEP_MIN = 15 +_qrows = [] +for _n in range(len(TODAY_H) * (60 // _QSTEP_MIN)): + _qu = TODAY_H[0] + timedelta(minutes=_QSTEP_MIN * _n) + # A spiky within-the-hour profile, so decimation is visible in the SPREAD + # and not only in a count. + _qrows.append({"start": _qu.isoformat(), + "price_eur_mwh": 100.0 + 40.0 * (_n % 4) + brussels(_qu).hour}) +_qbody = json.dumps({"data": _qrows}).encode() +check(f"the fixture really is sub-hourly - {len(_qrows)} points across " + f"{len(TODAY_H)} Brussels hours - or every check below passes for an " + f"hourly feed too and grades nothing", + len(_qrows) == 4 * len(TODAY_H) + and len({_r["start"] for _r in _qrows}) == len(_qrows)) + +_q_before = run(resolve_tariff(CFG, NOW, _until, transport=transport_for(_qbody), + now=NOW, history=NoHistory())) +_hq = store() +_ph.PriceHistory._remember(_hq, series_of(YDAY_H, YESTERDAY_MWH), NOW) +_q_after = run(resolve_tariff(CFG, NOW, _until, transport=transport_for(_qbody), + now=NOW, history=_hq)) +# What the feed published INSIDE the window, derived from the unfilled plan. +_q_win = [p for p in _q_before.read.points + if p.start.astimezone(UTC) >= SINCE_HOUR] +_q_firm = [p for p in _q_after.read.points if not p.provisional] +check(f"every print the window contains survives the fill: {len(_q_firm)} firm " + f"points against the {len(_q_win)} the feed published in that window. " + f"One-per-hour keying leaves {len(_q_win) // 4}", + len(_q_win) > 0 and len(_q_firm) == len(_q_win)) +check("...and they are the SAME INSTANTS AND PRICES the feed published, so " + "the count is not being made up by four hourly points wearing " + "quarter-hourly starts", + {(p.start.astimezone(UTC), p.eur_per_kwh) for p in _q_firm} + == {(p.start.astimezone(UTC), p.eur_per_kwh) for p in _q_win}) +check("...and no two of them share a whole hour by accident: the fixture puts " + "four distinct prices inside every published hour, which is what the " + "decimated version could not represent", + len({p.start.astimezone(UTC).replace(minute=0) for p in _q_firm}) + == len(_q_firm) // 4 > 0) + +_q_spread_want = round(max(p.eur_per_kwh for p in _q_win) + - min(p.eur_per_kwh for p in _q_win), 6) +_qv = _q_after.arbitrage(**_A) +_q_spread_got = _qv.match(allowed=lambda v: v.spread_eur_kwh, + refused=lambda _v: None) +check(f"...and the spread is the fixture's own max - min over that window " + f"({_q_spread_want} EUR/kWh, measured {_q_spread_got}), not the " + f"flattened figure a decimated series produces", + _qv.kind == "allowed" and _q_spread_got is not None + and abs(_q_spread_got - _q_spread_want) < 1e-9) +check("...and the cheapest firm point is one the FEED published, at its own " + "sub-hourly instant - not a re-derived hourly one", + _qv.kind == "allowed" + and any(_qv.cheapest is q for q in _q_after.read.points) + and any(_qv.cheapest.start.astimezone(UTC) == q.start.astimezone(UTC) + for q in _q_win)) + + +# --------------------------------------------------------------------------- # +print() +print("3. AC3 - the cache records the FEED, never its own output") + +_h2 = store() +_read2 = series_of(TODAY_H, TODAY_MWH) +_filled2 = _h2.carry(_read2, since=NOW, until=_until, stamp=NOW) +# Round two, as the refresh task does every 1800 s: the same partial feed. +_filled2b = _h2.carry(_read2, since=NOW, until=_until, stamp=NOW) + +_disk = json.load(open(_h2.path, encoding="utf-8")) +_today_key = brussels(NOW).strftime("%Y-%m-%d") +_recorded = {p["start"] for p in _disk["days"][_today_key]["points"]} +_recorded_utc = {datetime.fromisoformat(s).astimezone(UTC) for s in _recorded} +check(f"AC3: after two `carry()` rounds the record for {_today_key} holds " + f"EXACTLY the {len(TODAY_H)} hours the feed published " + f"({len(_recorded_utc)} held) - not one of the " + f"{HORIZON_H - len(_ahead)} hours the fill invented", + _recorded_utc == {u.astimezone(UTC) for u in TODAY_H}) + +check("...and no recorded point is flagged provisional, because a carried " + "point never reached the file - the flag would still read true while the " + "NUMBER decayed, which is the failure this rule is about", + not any(p["provisional"] for p in _disk["days"][_today_key]["points"])) + +def _canon(read): + return [p.canonical() for p in read.match(series=lambda r: r.points, + no_data=lambda _r: ())] + + +check(f"...and the fill is idempotent across the two rounds, which is §5.4's " + f"late-publication contract - the refresh task runs it again every " + f"1800 s against whatever the feed says by then " + f"({len(_canon(_filled2))} h both times)", + _canon(_filled2) and _canon(_filled2) == _canon(_filled2b)) + +# The ordering is the mechanism, so the mechanism is checked in the tree. +# +# ⚠️ EVERY SHIPPED PYTHON FILE, AND BY NAME RATHER THAN BY SUBSTRING. The first +# draft scanned `app/*.py` only - not `tools/`, not `scenarios/`, not the +# repo's `estop/` - and matched the bare string, so a module that merely +# MENTIONED `_remember` in a comment failed the check while a real call from +# `tools/` passed it unseen. This parses each file and looks for the name being +# USED: an attribute, a bare name, a def, or a string constant, which is how +# `getattr` reaches a private method. +_APP = os.path.join(HERE, "app") + + +def _uses_remember(path): + """Does this module USE the name `_remember`? Prose does not count.""" + try: + tree = ast.parse(open(path, encoding="utf-8").read()) + except (OSError, SyntaxError): + # ⚠️ NOT False. A file this scan cannot read is a file it has not + # cleared, and "unreadable" must never grade the same as "clean". + return True + for n in ast.walk(tree): + if isinstance(n, ast.Attribute) and n.attr == "_remember": + return True + if isinstance(n, ast.Name) and n.id == "_remember": + return True + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) \ + and n.name == "_remember": + return True + if isinstance(n, ast.Constant) and n.value == "_remember": + return True + return False + + +_SHIPPED = [] +for _root in (_APP, os.path.join(HERE, "tools"), os.path.join(HERE, "scenarios"), + os.path.join(os.path.dirname(HERE), "estop")): + for _dirpath, _dirnames, _filenames in os.walk(_root): + for _f in sorted(_filenames): + if _f.endswith(".py") and _f != "price_history.py": + _SHIPPED.append(os.path.join(_dirpath, _f)) +check(f"the scan has a roster to walk ({len(_SHIPPED)} shipped modules across " + f"app/, tools/, scenarios/ and estop/), and it reaches beyond app/ - an " + f"empty or app-only glob passes for a leak in any other directory", + len(_SHIPPED) > 10 + and any(os.sep + "tools" + os.sep in p for p in _SHIPPED)) +_leaks = [os.path.relpath(p, HERE) for p in _SHIPPED if _uses_remember(p)] +check(f"AC3's mechanism is an ORDERING, not a validation - a carried point and " + f"a published-provisional point are the same three fields - so nothing " + f"outside app/price_history.py may name `_remember` {_leaks or 'none do'}", + not _leaks) + +_probe = os.path.join(tmpdir(), "probe.py") +open(_probe, "w", encoding="utf-8").write("def f(h):\n return h._remember(1)\n") +_probe_ok = os.path.join(tmpdir(), "probe_ok.py") +open(_probe_ok, "w", encoding="utf-8").write( + '"""A docstring about _remember."""\n# and a comment about _remember\nX = 1\n') +check("...and that scan FAILS on a real call and PASSES on a file that only " + "talks about the rule, so it is a check and not a decoration", + _uses_remember(_probe) and not _uses_remember(_probe_ok)) + +# ⚠️ `ast.walk` IS BREADTH-FIRST, SO A LIST OF CALL NAMES GRADES NESTING DEPTH +# AND NOT SOURCE ORDER. The first draft of this check was +# `_calls.index("_remember") < _calls.index("yesterday")`: `self._remember(...)` +# is a statement while `self.yesterday(...)` is an ARGUMENT of +# `fill_from_yesterday(...)`, one level deeper, so `_remember` was emitted +# first whichever statement came first. Measured by writing the reversal out +# and grading it: shipped and reversed both scored True, while DECISIONS §3 and +# price_history.py's docstring both cited the check as making a reversal "red +# rather than plausible". Line numbers here, and the behaviour itself below. +_carry_src = inspect.getsource(PriceHistory.carry) +_tree = ast.parse(_carry_src.strip()) +_lines = {} +for _n in ast.walk(_tree): + if isinstance(_n, ast.Call) and isinstance(_n.func, ast.Attribute): + _lines[_n.func.attr] = min(_lines.get(_n.func.attr, 10 ** 9), _n.lineno) +check(f"`carry()` names `_remember` on an EARLIER LINE than `yesterday` " + f"{_lines} - source order, which is what 'it records before it fills' " + f"actually means", + "_remember" in _lines and "yesterday" in _lines + and _lines["_remember"] < _lines["yesterday"]) + +# ...and the BEHAVIOUR the ordering exists for, driven rather than read. A feed +# that hands back yesterday alongside today must refresh yesterday's record +# BEFORE it is consumed as the profile. Into an EMPTY store that is the whole +# difference: record-then-fill covers the horizon, fill-then-record covers +# nothing, because at the moment of the fill there is no yesterday yet. +_h_ord = store() # empty: no record from an earlier day +_two_day = PriceRead.series( + [PricePoint(start=brussels(u), + eur_per_kwh=allin(YESTERDAY_MWH[brussels(u).hour]), + provisional=False) for u in YDAY_H] + + [PricePoint(start=brussels(u), + eur_per_kwh=allin(TODAY_MWH[brussels(u).hour]), + provisional=False) for u in TODAY_H], + PriceProduct.DAY_AHEAD, "generic_dayahead", NOW) +_ord_filled = _h_ord.carry(_two_day, since=NOW, until=_until, stamp=NOW) +_ord_n = _ord_filled.match(series=lambda r: len(r.points), no_data=lambda _r: 0) +check(f"...and the ordering is DRIVEN, not only read: a feed carrying " + f"YESTERDAY alongside today, into an empty store, fills the whole " + f"{HORIZON_H} h horizon ({_ord_n} h) because the record was written " + f"before the lookup. Reversed, the lookup finds nothing and the plan is " + f"the {len(_ahead)} published hours the window keeps", + _ord_n == HORIZON_H) +check("...and the hours it carried carry YESTERDAY's numbers - the record that " + "same call had just written - not today's", + _ord_filled.kind == "series" + and all(abs(p.eur_per_kwh + - allin(YESTERDAY_MWH[brussels(p.start).hour])) < 1e-9 + for p in _ord_filled.points if p.provisional)) + + +# --------------------------------------------------------------------------- # +print() +print("4. rule 2 - 'yesterday' is a DATE, not 'the newest day I have'") + +_h3 = store() +_ph.PriceHistory._remember( + _h3, series_of(bday_hours("2026-01-12"), YESTERDAY_MWH), NOW) +_y3 = _h3.yesterday(NOW) +check("a store holding only the day before LAST answers no_data, not those " + "prices - `newest()` is the accessor that decays, and an add-on down " + "over a weekend is when it would", + _y3.kind == "no_data") +check(f"...and the reason names BOTH the date wanted and the dates held, so an " + f"operator can see which it is: {why(_y3)!r}", + "2026-01-14" in why(_y3) and "2026-01-12" in why(_y3)) + +_h4 = store() +_ph.PriceHistory._remember( + _h4, series_of(YDAY_H, YESTERDAY_MWH), NOW) +_y4 = _h4.yesterday(NOW) +check("...while the day BEFORE this one answers a series, so the refusal above " + "is about the date and not about the store being broken", + _y4.kind == "series" and len(_y4.points) == 24) + +check("a naive instant is refused rather than assumed to be UTC - 'which " + "Brussels day was yesterday' is the question a wall clock cannot answer", + raises(lambda: _h4.yesterday(datetime(2026, 1, 15, 9)), ValueError)) + +# The Brussels-date rule, on the fixture a UTC-date rule gets wrong. A feed +# serving the 24 UTC hours of a July day spans TWO Brussels days - Belgium is +# on CEST, so 22:00 and 23:00 UTC are already 00:00 and 01:00 of tomorrow - and +# a UTC-keyed cache would file all 24 under one date. The evening hours are the +# expensive ones, so filing them under the wrong day is the costly direction. +_h5 = store() +_jul_utc = [datetime(2026, 7, 14, h, tzinfo=UTC) for h in range(24)] +_split = {brussels(u).strftime("%Y-%m-%d") for u in _jul_utc} +check(f"the July fixture really does straddle two Brussels days {sorted(_split)}" + f", or the next check passes for a UTC-keyed cache as well", + len(_split) == 2) +_ph.PriceHistory._remember( + _h5, series_of(_jul_utc, YESTERDAY_MWH), _jul_utc[-1]) +_disk5 = json.load(open(_h5.path, encoding="utf-8")) +_counts = {d: len(_disk5["days"][d]["points"]) for d in sorted(_disk5["days"])} +check(f"...and the record is keyed on the BRUSSELS date, so those 24 hours are " + f"filed under both of them {_counts} rather than all under one", + set(_counts) == _split and sum(_counts.values()) == 24 + and all(_counts.values())) + + +# --------------------------------------------------------------------------- # +print() +print("4b. the two mornings a year when 'yesterday' is a DATE and not 24 hours") + +# ⚠️ `brussels()` RETURNS A FIXED-OFFSET DATETIME, so `brussels(stamp) - +# timedelta(days=1)` subtracts 24 ABSOLUTE HOURS from a wall clock. The +# Brussels day before a spring-forward is 23 hours long, so between 00:00 and +# 00:59 on the morning after it that lands on the day BEFORE yesterday: the +# fill then carries a two-day-old profile behind a flag whose only vocabulary +# is "this is a guess", or - once pruning has dropped that day - answers +# `no_data` naming a date that is not yesterday. Measured at +# 2026-03-30T00:30+02:00 on the shipped branch: it returned 2026-03-28's +# prices. DST is this module's headline claim and it had NO CHECK. +for _label, (_dm2, _dm1, _d0) in ( + ("spring forward - yesterday was 23 h long", + ("2026-03-28", "2026-03-29", "2026-03-30")), + ("fall back - yesterday was 25 h long", + ("2026-10-24", "2026-10-25", "2026-10-26"))): + _lens = [len(bday_hours(_d)) for _d in (_dm2, _dm1, _d0)] + check(f"{_label}: the fixture days are {_lens} hours long, so one of them " + f"really is not 24 and a DATE subtraction and an INSTANT subtraction " + f"can be told apart at all", + _lens[1] != 24 and _lens[0] == 24 and _lens[2] == 24) + _hd = store() + _tag = {} + for _i, _d in enumerate((_dm2, _dm1, _d0)): + # One price band per DAY, so the answer names which day it came from. + _tag[_d] = 0.100000 * (_i + 1) + _ph.PriceHistory._remember( + _hd, + PriceRead.series( + [PricePoint(start=brussels(u), + eur_per_kwh=round(_tag[_d] + 0.000100 + * brussels(u).hour, 6), + provisional=False) + for u in bday_hours(_d)], + PriceProduct.DAY_AHEAD, "generic_dayahead", at_local(_d, 12)), + at_local(_d, 12)) + check(f"{_label}: all three days are held, so the check below is about the " + f"DATE arithmetic and not about a record having been pruned away", + sorted(_hd._days) == sorted((_dm2, _dm1, _d0))) + _stamp_dst = at_local(_d0, 0) + timedelta(minutes=30) + _yd = _hd.yesterday(_stamp_dst) + _band = _yd.match( + series=lambda r: round(min(p.eur_per_kwh for p in r.points), 4), + no_data=lambda _r: None) + check(f"{_label}: at 00:30 local on {_d0}, yesterday() answers {_dm1}'s " + f"prices (band {_band}, wanted {round(_tag[_dm1], 4)}) and not " + f"{_dm2}'s (band {round(_tag[_dm2], 4)}) {why(_yd)!r}", + _yd.kind == "series" and _band == round(_tag[_dm1], 4)) + check(f"{_label}: ...and it holds {_lens[1]} points, the length that " + f"Brussels day actually was - a 24-slot answer would be the wrong " + f"day wearing the right date", + _yd.kind == "series" and len(_yd.points) == _lens[1]) + # The control, one hour later: an implementation that is right only at + # 01:30 is the one being caught, so both are graded. + _yd2 = _hd.yesterday(at_local(_d0, 1) + timedelta(minutes=30)) + check(f"{_label}: ...and at 01:30 local it is still {_dm1} - the defect " + f"shows only in the first local hour, which is why an hourly refresh " + f"could hit it and never be noticed", + _yd2.kind == "series" + and round(min(p.eur_per_kwh for p in _yd2.points), 4) + == round(_tag[_dm1], 4)) + + +# --------------------------------------------------------------------------- # +print() +print("5. AC4 - degraded stores, one reason each, no prices") + +_missing = PriceHistory(os.path.join(tmpdir(), "gone", "price_history.json")) +check("no durable volume: the store is NOT ok, and it does not create the " + "directory - a cache on the ephemeral layer reads fine and is thrown " + "away on the next restart (tariff.PeakStore's rule)", + not _missing.ok and "no durable volume" in (_missing.last_error or "") + and not os.path.isdir(os.path.dirname(_missing.path))) + +# ⚠️ AND IT DOES NOT FILL IN MEMORY EITHER. FCST-04 shipped this half-done: +# `_save()` short-circuited on `not ok` while `_remember()` populated +# `self._days` anyway, so a site with no /data answered `words` = "unavailable" +# and `yesterday()` = A SERIES - an in-memory profile that works all day and is +# gone on the next restart. price_history.py's own third warning rejects +# exactly that ("a yesterday that silently forgets") as its reason for not +# calling makedirs, so the code and the docstring have to agree. +_carried_nowhere = _missing.carry(series_of(TODAY_H, TODAY_MWH), + since=NOW, until=_until, stamp=NOW) +check("a store with no durable volume records NOTHING, not even in memory: " + "after a full carry it still holds no day, `yesterday()` still answers " + "no_data, and `words` still says unavailable. A cache that forgets on " + "restart without saying so is worse than one that says it cannot answer", + _missing._days == {} + and _missing.yesterday(NOW + timedelta(days=1)).kind == "no_data" + and _missing.words.startswith("unavailable")) +check("...and the carry still returns a real series - the fill windows the " + "feed and carries nothing, which is the pre-FCST-04 horizon and not a " + "degradation", + _carried_nowhere.kind == "series" + and len(_carried_nowhere.points) == len(_ahead)) + +_d6 = tmpdir() +_p6 = os.path.join(_d6, "price_history.json") +open(_p6, "w", encoding="utf-8").write("{ this is not json") +_h6 = PriceHistory(_p6) +check("an unreadable file: the reason is kept and reported, and the record is " + "NOT silently re-created as an empty one that reads like a first day", + "unreadable" in (_h6.last_error or "") + and _h6.yesterday(NOW).kind == "no_data" + and "unreadable" in why(_h6.yesterday(NOW))) + +_p7 = os.path.join(tmpdir(), "price_history.json") +json.dump({"version": SCHEMA + 99, "days": {}}, open(_p7, "w", encoding="utf-8")) +_h7 = PriceHistory(_p7) +check("a file from a schema this build does not know is discarded rather than " + "guessed at - its points might mean something else", + "schema" in (_h7.last_error or "") + and _h7.yesterday(NOW).kind == "no_data") + +# A record whose points are malformed. THE DEFECT IS A DEFAULTED PRICE. +_p8 = os.path.join(tmpdir(), "price_history.json") +_good = {"start": brussels(at_local(YDAY_D, 5)).isoformat(), + "eur_per_kwh": allin(YESTERDAY_MWH[5]), "provisional": False} +json.dump({"version": SCHEMA, "days": {"2026-01-14": { + "product": "dynamic_dayahead", "provider": "generic_dayahead", + "points": [_good, + {"start": "2026-01-14T06:00:00+01:00"}, # no price + {"start": "2026-01-14 07:00:00", "eur_per_kwh": 0.2, + "provisional": False}, # naive + {"eur_per_kwh": 0.3, "provisional": False}, # no instant + "not even a dict"]}}}, + open(_p8, "w", encoding="utf-8")) +_y8 = PriceHistory(_p8).yesterday(NOW) +check("a malformed point is DROPPED, not defaulted: the four broken rows are " + "gone and the one good row survives", + _y8.kind == "series" and len(_y8.points) == 1) +check("...and specifically no point came back priced 0.0 - " + "`float(row.get('eur_per_kwh', 0))` is how a lost field becomes 'that " + "hour was free'", + all(p.eur_per_kwh != 0.0 for p in _y8.points) + and abs(_y8.points[0].eur_per_kwh - allin(YESTERDAY_MWH[5])) < 1e-9) + +_p9 = os.path.join(tmpdir(), "price_history.json") +json.dump({"version": SCHEMA, "days": {"2026-01-14": { + "product": "no_such_product", "provider": "x", + "points": [_good]}}}, open(_p9, "w", encoding="utf-8")) +check("a record naming a product that is not a PriceProduct is no_data, not a " + "series with a string in the field the optimiser switches on", + PriceHistory(_p9).yesterday(NOW).kind == "no_data") + +# ⚠️ AND THE ONE THAT STOPPED THE ADD-ON FROM STARTING. `_load()` validated +# `version` and that `days` is a dict, and never that each VALUE is one. With +# `{"version": 1, "days": {"2026-01-14": "oops"}}` the store reported ok=True +# and last_error=None, and then `rec.get(...)` raised `AttributeError` out of +# `words`, `yesterday()` and `resolve_tariff()` alike. `PriceHistory(...)` and +# `.words` are both unguarded in `Controller.__init__`, so the controller did +# not come up - and it never self-healed, because `_remember()` raised on the +# same record before it could reach `_save()`. The asymmetry that produced it +# is visible in the file: `_parse_points()` is carefully defensive about every +# field of a POINT, and `_load()` was not defensive about the record at all. +_p14 = os.path.join(tmpdir(), "price_history.json") +json.dump({"version": SCHEMA, "days": { + "2026-01-14": "oops", + "2026-01-13": {"product": "dynamic_dayahead", "provider": "generic_dayahead", + "points": [_good]}}}, + open(_p14, "w", encoding="utf-8")) +_h14 = PriceHistory(_p14) +check("a day record that is not a dict is DROPPED and REPORTED, and the rest " + "of the file still loads - one bad record is not a bad file", + _h14.ok and "unusable day record" in (_h14.last_error or "") + and sorted(_h14._days) == ["2026-01-13"]) +_w14 = derived(lambda: _h14.words) +_y14 = derived(lambda: _h14.yesterday(NOW).kind) +_plan14 = derived(lambda: run(resolve_tariff( + CFG, NOW, _until, transport=transport_for(_body_today), + now=NOW, history=_h14)).origin) +check(f"...and NOTHING raises on it: not `words` ({_w14!r}), not `yesterday()` " + f"({_y14!r}), not the shipped `resolve_tariff()` path ({_plan14!r}). A " + f"broken cache is a short horizon, not a controller that will not start", + isinstance(_w14, str) and _y14 == "no_data" and _plan14 == "provider") +for _shape in ("oops", [1, 2, 3], 7, None, {"points": "not a list"}): + _pX = os.path.join(tmpdir(), "price_history.json") + json.dump({"version": SCHEMA, "days": {"2026-01-14": _shape}}, + open(_pX, "w", encoding="utf-8")) + _hX = PriceHistory(_pX) + _wX = derived(lambda h=_hX: h.words) + _yX = derived(lambda h=_hX: h.yesterday(NOW).kind) + check(f"...and the same holds for a day record of {type(_shape).__name__} " + f"{_shape!r}: dropped, reported, and every accessor answers " + f"({_wX!r}, {_yX!r})", + _hX._days == {} and "unusable day record" in (_hX.last_error or "") + and isinstance(_wX, str) and _yX == "no_data") + +_pF = os.path.join(tmpdir(), "price_history.json") +json.dump({"version": float(SCHEMA), "days": {}}, + open(_pF, "w", encoding="utf-8")) +check(f"a `version` of {float(SCHEMA)!r} is refused although it EQUALS " + f"{SCHEMA} - a schema check that cannot tell 1 from 1.0 is one that " + f"would not notice a writer this build has never met", + "schema" in (PriceHistory(_pF).last_error or "")) + + +# --------------------------------------------------------------------------- # +print() +print("5b. `words` - the status row, and one of the two things a bad record " + "used to crash") + +def words_of(h): + return derived(lambda: h.words) + + +_w_missing = words_of(_missing) +check(f"no durable volume: the row says UNAVAILABLE and names the reason " + f"{_w_missing!r}", + isinstance(_w_missing, str) and _w_missing.startswith("unavailable") + and "no durable volume" in _w_missing) +_h_words = store() +check("a healthy but EMPTY store says something different, because it is a " + "different state and a normal one on a site's first day", + isinstance(words_of(_h_words), str) + and words_of(_h_words).startswith("empty") + and "nothing to carry forward" in words_of(_h_words)) +_ph.PriceHistory._remember(_h_words, series_of(YDAY_H, YESTERDAY_MWH), NOW) +check(f"...and a store holding a day names the DATE and the HOUR COUNT " + f"{words_of(_h_words)!r} - 'a cache exists' and 'the cache holds " + f"yesterday, all 24 hours of it' are different facts, and only the " + f"second says the horizon will be full tomorrow morning", + words_of(_h_words) == f"{YDAY_D} (24 h)") +check(f"a file that only PARTLY loaded says so in the row {_w14!r}, or the " + f"days it dropped read as days that were simply never written", + isinstance(_w14, str) and (_h14.last_error or "") in _w14 + and "2026-01-13" in _w14) +check("NoHistory() has a sentence of its own rather than an empty cell", + NoHistory().words == "not kept") +check("...and `words` never raises, whatever the file holds. It is read once " + "from Controller.__init__ and again on every status render, and neither " + "call site guards it - an exception here is an add-on that does not start", + all(isinstance(words_of(h), str) + for h in (_missing, _h6, _h7, _h14, _h_words))) + +# ⚠️ AND A FAILED WRITE LEAVES NOTHING BEHIND. `_save()` writes to a `mkstemp` +# and renames - atomic, so a power cut cannot leave a truncated file that +# parses as an empty day - and a failure BETWEEN the two used to leave the temp +# in place. The refresh task writes 48 times a day, so a persistent write +# failure leaked 48 inodes a day onto the volume whose being full is the +# likeliest reason the write failed at all. +_dleak = tmpdir() +_pleak = os.path.join(_dleak, "price_history.json") +_hleak = PriceHistory(_pleak) +check("the leak fixture starts clean, or 'nothing left behind' is not a " + "measurement", + _hleak.ok and os.listdir(_dleak) == []) +os.mkdir(_pleak) # a DIRECTORY where the file goes: the temp is + # written fine and `os.replace` cannot complete +_ph.PriceHistory._remember(_hleak, series_of(YDAY_H, YESTERDAY_MWH), NOW) +check(f"a write that cannot complete SAYS so rather than passing quietly " + f"{(_hleak.last_error or '')[:48]!r}", + "could not write" in (_hleak.last_error or "")) +_left = sorted(n for n in os.listdir(_dleak) if n != "price_history.json") +check(f"...and leaves NOTHING behind {_left} - at 48 writes a day a leaked " + f"temp per write fills the volume that was already the problem", + _left == []) +_dok = tmpdir() +_hok = PriceHistory(os.path.join(_dok, "price_history.json")) +_ph.PriceHistory._remember(_hok, series_of(YDAY_H, YESTERDAY_MWH), NOW) +check(f"...and a SUCCESSFUL write leaves exactly the one file " + f"{sorted(os.listdir(_dok))}, so the cleanup did not start deleting the " + f"cache it just wrote", + sorted(os.listdir(_dok)) == ["price_history.json"] + and _hok.last_error is None) + +# ...and every one of those leaves the CONTROLLER regulating. +# +# ⚠️ THE PLAN IS NOT BIT-FOR-BIT THE PRE-FCST-04 ONE, and the first draft of +# this check claimed it was. A broken cache still goes through the fill, which +# WINDOWS to [since, until) whether or not it has anything to carry - so the +# plan is the feed's own hours MINUS the elapsed ones. That is the AC2 half of +# this change, and it is correct with no cache at all: an hour that has gone is +# not a cheap hour. What must survive a broken cache is that the plan is a real +# provider series with no invented points and no §11.4 degradation. +for _label, _hist in (("no durable volume", _missing), + ("unreadable file", _h6), + ("wrong schema", _h7)): + _plan = run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), now=NOW, + history=_hist)) + check(f"...and with {_label} the plan is a real provider series over the " + f"{len(_ahead)} published hours still ahead, with NOTHING carried " + f"and no §11.4 degradation. A broken cache is not a broken " + f"controller - it is a short horizon, which is where this started", + _plan.origin == "provider" + and len(_plan.read.points) == len(_ahead) + and not any(p.provisional for p in _plan.read.points)) + + +# --------------------------------------------------------------------------- # +print() +print("6. rule 4 - hours accumulate within a day; they do not replace it") + +_h10 = store() +_ph.PriceHistory._remember(_h10, series_of(TODAY_H, TODAY_MWH), NOW) +# A feed that serves only the hours left in the day - a legitimate shape, and +# the one that turns replace-the-record into a two-hour yesterday. +_REVISED = dict(TODAY_MWH) +_REVISED[22], _REVISED[23] = 999.0, 998.0 +check("the revision really changes the numbers, or merge and replace are the " + "same observation", + _REVISED[22] != TODAY_MWH[22] and _REVISED[23] != TODAY_MWH[23]) +_ph.PriceHistory._remember( + _h10, series_of([at_local(TODAY_D, 22), at_local(TODAY_D, 23)], _REVISED), + NOW) + +_back = _h10.yesterday(NOW + timedelta(days=1)) +check(f"a two-hour read did not overwrite a 24-hour day: " + f"{len(_back.points) if _back.kind == 'series' else _back.kind} hours " + f"held. Under replace, the next morning's 'yesterday' would have been " + f"two hours long", + _back.kind == "series" and len(_back.points) == 24) + +# ⚠️ KEYED BY BRUSSELS HOUR, because that is what the fixture tables are keyed +# by. Keying this by UTC hour reads the wrong row of the table in winter and +# grades the module against a price nobody put there. +_by_hour = {brussels(p.start).hour: p.eur_per_kwh for p in _back.points} + + +def _priced(h, want): + """True only if hour `h` is present AND carries `want`. A MISSING hour is + False, never a KeyError: under `replace` the hours simply are not there, and + a harness that dies on the mutation it is grading reports nothing at all.""" + return h in _by_hour and abs(_by_hour[h] - want) < 1e-9 + + +check("...and the two revised hours carry the NEW price, because a later read " + "is the newer statement about the same hour", + _priced(22, allin(999.0)) and _priced(23, allin(998.0))) +check("...while every untouched hour kept its original price", + all(_priced(h, allin(TODAY_MWH[h])) for h in range(22))) + + +# --------------------------------------------------------------------------- # +print() +print("7. the restart this whole module exists for") + +_d11 = tmpdir() +_p11 = os.path.join(_d11, "price_history.json") +_first = PriceHistory(_p11) +_ph.PriceHistory._remember( + _first, series_of(YDAY_H, YESTERDAY_MWH), NOW) +del _first +_second = PriceHistory(_p11) # a brand-new add-on process +_y11 = _second.yesterday(NOW) +check("yesterday survives a restart, which is the ONLY reason this is a file " + "and not a variable on the refresh task: the window the fill exists for " + "is midnight -> publication, and an add-on restarted at 08:00 would " + "otherwise spend all of it on the truncated series", + _y11.kind == "series" and len(_y11.points) == 24) +check("...with the prices intact through the JSON round trip, to six decimals", + all(abs(p.eur_per_kwh + - allin(YESTERDAY_MWH[brussels(p.start).hour])) < 1e-9 + for p in _y11.points)) + +_plan11 = run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), now=NOW, + history=_second)) +check("...and the restarted controller's very first refresh already covers the " + "whole horizon", + len(_plan11.read.points) == HORIZON_H) + + +# --------------------------------------------------------------------------- # +print() +print("8. pruning keeps yesterday, and a jumped clock does not empty the store") + +_h12 = store() +for back in range(6, 0, -1): + _ph.PriceHistory._remember( + _h12, + series_of(bday_hours( + (datetime.fromisoformat(TODAY_D) - timedelta(days=back) + ).strftime("%Y-%m-%d")), YESTERDAY_MWH), + NOW) +_held = sorted(json.load(open(_h12.path, encoding="utf-8"))["days"]) +check(f"six days filed, {KEEP_DAYS} held {_held} - a bigger cache would only " + f"hold days rule 2 refuses to look at", + len(_held) == KEEP_DAYS) +check("...and yesterday is one of them, which is the only day that matters", + _h12.yesterday(NOW).kind == "series") + +# ⚠️ THE STIMULUS HAS TO REACH `_prune`'s BODY, AND THE FIRST DRAFT DID NOT. +# It seeded ONE day and then pruned with a jumped stamp: `if len(self._days) <= +# KEEP_DAYS: return` fired, the body never ran, and the check named after that +# body was green for an implementation that pruned by clock arithmetic. +# Measured - the body was replaced with "delete every day older than stamp +# minus KEEP_DAYS days", the guard left in place, and the whole 29-file suite +# stayed green while the mutant emptied the store on a jumped clock. +# +# A store holding more days than the cap is a REAL state, not a contrivance: +# `_load()` does not prune, so this is exactly what a restart sees after a file +# written by a build with a larger cap, or after a feed that spanned several +# days. So the fixture is written, opened, and only then pruned. +_FILED = KEEP_DAYS + 2 +_p13 = os.path.join(tmpdir(), "price_history.json") +_days13 = {} +for _back in range(_FILED - 1, -1, -1): + _d13 = (datetime.fromisoformat(TODAY_D) + - timedelta(days=_back)).strftime("%Y-%m-%d") + _days13[_d13] = { + "product": PriceProduct.DAY_AHEAD.value, + "provider": "generic_dayahead", + "points": [PricePoint(start=brussels(u), + eur_per_kwh=allin(YESTERDAY_MWH[brussels(u).hour]), + provisional=False).canonical() + for u in bday_hours(_d13)]} +json.dump({"version": SCHEMA, "days": _days13}, + open(_p13, "w", encoding="utf-8")) +_h13 = PriceHistory(_p13) +check(f"the prune fixture holds {len(_h13._days)} days, MORE than KEEP_DAYS " + f"({KEEP_DAYS}), so the early return cannot swallow the stimulus - this " + f"file's own rule, and the one its first draft broke", + len(_h13._days) == _FILED > KEEP_DAYS) +_ph.PriceHistory._prune(_h13, NOW + timedelta(days=4000)) +check(f"pruning does not delete by ARITHMETIC AGAINST THE CLOCK: a clock that " + f"jumped four thousand days keeps the newest {KEEP_DAYS} of the " + f"{_FILED} records {sorted(_h13._days)} instead of deleting all of them, " + f"which would empty the store on exactly the morning the fill is for", + len(_h13._days) == KEEP_DAYS and _h13.yesterday(NOW).kind == "series") + + +# --------------------------------------------------------------------------- # +print() +print("8b. the two days a lookup can ask for are PINNED, so a far-off record " + "cannot evict yesterday") + +# ⚠️ KEEP_DAYS = 3 IS THE STEADY STATE EXACTLY - {yesterday, today, tomorrow} - +# so a newest-N cap has NO SLACK. A feed that publishes to D+2 makes four days +# and the cap drops the oldest, which is yesterday: the only day the fill can +# use. One stray point in the wrong year does the same thing permanently, +# because a lexicographic sort puts it LAST and nothing can ever age it out. +_h15 = store() +_SPAN_D = (YDAY_D, TODAY_D, "2026-01-16", "2026-01-17") +_span = [u for _d in _SPAN_D for u in bday_hours(_d)] +check(f"the spanning fixture really covers {len(_SPAN_D)} Brussels days, more " + f"than KEEP_DAYS ({KEEP_DAYS}), or the cap is not under pressure and " + f"this grades nothing", + len(_SPAN_D) > KEEP_DAYS + and len({brussels(u).strftime("%Y-%m-%d") for u in _span}) == len(_SPAN_D)) +_ph.PriceHistory._remember(_h15, series_of(_span, YESTERDAY_MWH), NOW) +check(f"a feed spanning {len(_SPAN_D)} days does NOT evict yesterday: " + f"{sorted(_h15._days)} held, and {YDAY_D} is one of them. Under the " + f"newest-{KEEP_DAYS} cap alone it is the first record dropped, and the " + f"fill loses its only usable day for good", + YDAY_D in _h15._days and _h15.yesterday(NOW).kind == "series") + +_h16 = store() +for _d16 in (YDAY_D, TODAY_D, "2026-01-16"): + _ph.PriceHistory._remember(_h16, series_of(bday_hours(_d16), TODAY_MWH), NOW) +_STRAY_D = "2099-07-01" +_ph.PriceHistory._remember( + _h16, series_of(bday_hours(_STRAY_D)[:1], TODAY_MWH), NOW) +check(f"the stray fixture is really far future and really sorts last " + f"{sorted(_h16._days)[-1]!r}, or 'it can never age out' is not what is " + f"being graded", + sorted(_h16._days)[-1] == _STRAY_D) +check(f"...and ONE stray point in {_STRAY_D} does not push yesterday out " + f"either. It is not deleted - nothing is deleted by arithmetic here - " + f"it simply cannot take yesterday's slot", + YDAY_D in _h16._days and _h16.yesterday(NOW).kind == "series") +check(f"...while the store stays BOUNDED at KEEP_DAYS + 2 = {KEEP_DAYS + 2} " + f"records ({len(_h16._days)} held), so the pin is a pin and not a leak", + len(_h16._days) <= KEEP_DAYS + 2) + + +# --------------------------------------------------------------------------- # +print() +print("9. the parameter cannot be forgotten, and main.py does not forget it") + +_sig = inspect.signature(resolve_tariff) +_hp = _sig.parameters.get("history") +check("`resolve_tariff(history=...)` is KEYWORD-ONLY and has NO DEFAULT - a " + "default of None meaning 'skip the fill' would hand a caller who forgot " + "it the truncated horizon this ticket removes, silently", + _hp is not None and _hp.kind is inspect.Parameter.KEYWORD_ONLY + and _hp.default is inspect.Parameter.empty) + +check("...so calling it without one is a TypeError, not a quiet regression", + raises(lambda: run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), + now=NOW)), TypeError)) + +_MAIN = open(os.path.join(_APP, "main.py"), encoding="utf-8").read() +_main_tree = ast.parse(_MAIN) +_refresh = next(n for n in ast.walk(_main_tree) + if isinstance(n, ast.AsyncFunctionDef) + and n.name == "run_tariff_refresh") +_kw = [k.arg for n in ast.walk(_refresh) if isinstance(n, ast.Call) + for k in n.keywords] +check(f"the shipped refresh task passes a history {_kw} - the wiring is what " + f"this ticket IS, so it is checked in the tree rather than described", + "history" in _kw) +check("the FCST-04 note in main.py carries the WINDOWING caveat beside the " + "'widens what the optimiser can SEE, never what it may act on' " + "guarantee. That note is the one place a maintainer reads that sentence " + "unqualified, and the CHANGELOG, DOCS and resolve_tariff()'s docstring " + "all carry the pair - so it may not appear here on its own", + "never what it may act on" in _MAIN and "ALREADY ELAPSED" in _MAIN) + +check("...and it is the CONTROLLER'S store, not a NoHistory() smuggled in to " + "make the call typecheck", + "history=self.price_history" in _MAIN + and "NoHistory" not in _MAIN) + +check("...and `words` reaches the STATUS PAGE as its own row, so what the fill " + "has to carry from is a fact an operator can read rather than one that " + "only exists in a log line", + '("Price history", self.price_history.words)' in _MAIN) +_rows_fn = next(n for n in ast.walk(_main_tree) + if isinstance(n, ast.FunctionDef) + and n.name == "status_payload") +_row_labels = [e.elts[0].value for e in ast.walk(_rows_fn) + if isinstance(e, ast.Tuple) and len(e.elts) == 2 + and isinstance(e.elts[0], ast.Constant) + and isinstance(e.elts[0].value, str)] +check(f"...and it is in the ROW LIST rather than in a comment about one " + f"{[r for r in _row_labels if 'rice' in r or 'ariff' in r]}", + "Price history" in _row_labels) + +# ⚠️ AND WHAT THE STATUS PAGE DOES **NOT** SAY, PINNED AS A PAIR. DOCS.md +# claimed "*Tariff* on the status page reads `provisional` through the +# morning". It does not and never did: `self.tariff_words` is assigned only +# from the Home Assistant price-entity branch and from `static_status()`, both +# on the 30 s tick, and the provider plan's own sentence reaches `_LOG.info` +# and nothing else. This grades the PAIRING rather than the current state - so +# wiring the plan into that row is fine and correcting the doc is fine, and +# doing one without the other is red. +_MAIN_LINES = _MAIN.splitlines() +_tw_lines = sorted({n.lineno for n in ast.walk(_main_tree) + if isinstance(n, ast.Assign) + for t in n.targets + for a in ast.walk(t) + if isinstance(a, ast.Attribute) and a.attr == "tariff_words"}) +_row_from_plan = any("plan" in _MAIN_LINES[i - 1] for i in _tw_lines) +_DOCS = open(os.path.join(HERE, "DOCS.md"), encoding="utf-8").read() +_docs_claims = "**Tariff** on the status page reads `provisional`" in _DOCS +check(f"the *Tariff* row and what DOCS says about it AGREE: fed from the " + f"provider plan = {_row_from_plan} (assigned on lines {_tw_lines}), DOCS " + f"claims it reads `provisional` through the morning = {_docs_claims}. It " + f"shipped as False against True - an overclaim in a doc is a defect " + f"here, and this is the pair that has to move together", + len(_tw_lines) >= 2 and _row_from_plan == _docs_claims) + +# NoHistory is inert, and says so by identity rather than by producing a series. +_nh_read = series_of(TODAY_H, TODAY_MWH) +check("NoHistory() carries nothing and returns the read BY IDENTITY, so the " + "explicit 'I keep no history' is exactly the old behaviour and not a " + "second implementation of it", + NoHistory().carry(_nh_read, since=NOW, until=_until, stamp=NOW) + is _nh_read + and NoHistory().yesterday(NOW).kind == "no_data") + + +# --------------------------------------------------------------------------- # +for d in _tmpdirs: + shutil.rmtree(d, ignore_errors=True) + +print() +if fails: + print(f"{len(fails)} FAILED: " + "; ".join(fails)) + sys.exit(1) +print("all checks passed") diff --git a/goodwe_controller/test_static_tariff.py b/goodwe_controller/test_static_tariff.py index 84f8fa9..6e42681 100644 --- a/goodwe_controller/test_static_tariff.py +++ b/goodwe_controller/test_static_tariff.py @@ -69,6 +69,7 @@ from app.price import ( Absence, PricePoint, PriceProduct, PriceRead, arbitrage_allowed, brussels, plan_is_provisional, required_spread, uncertainty_of, ) +from app.price_history import NoHistory from app.static_tariff import ( MEAN_WINDOW_DAYS, ProfileInvalid, StaticProfile, TariffPlan, parse_static_profile, resolve_tariff, static_plan, static_status, @@ -746,7 +747,8 @@ _cfg = opts(TWO_PERIOD, price_provider="generic_dayahead", price_feed_url="https://api.energy-charts.info/price?bzn=BE", price_timeout_s=1, price_markup_eur_kwh=0.03, price_taxes_eur_kwh=0.06) -_fb = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1), transport=_dead)) +_fb = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1), transport=_dead, + history=NoHistory())) check("a dead feed falls back to the static profile rather than to nothing", _fb.origin == "fallback" and _fb.has_prices) check("...with the SAME prices the configured-product path produces, because " @@ -769,7 +771,8 @@ check("...and the provisional plan has no cheap window either: §8.3 forbids an not any(_fb.cheap_now(at_local(THU, h)) for h in range(24))) _fb_none = run(resolve_tariff(opts(None, price_provider="generic_dayahead", price_timeout_s=1), - THU, THU + timedelta(days=1), transport=_dead)) + THU, THU + timedelta(days=1), transport=_dead, + history=NoHistory())) check("a dead feed with NO static_profile has nothing to fall back to, and " "says that rather than inventing a tariff", _fb_none.origin == "none" and not _fb_none.has_prices) @@ -785,7 +788,7 @@ async def _live(url, timeout_s): _live_plan = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1), - transport=_live)) + transport=_live, history=NoHistory())) check("a live feed is used in preference to the static profile", _live_plan.origin == "provider" and _live_plan.product is PriceProduct.DAY_AHEAD)