diff --git a/goodwe_controller/CHANGELOG.md b/goodwe_controller/CHANGELOG.md index 2e29571..8023f3e 100644 --- a/goodwe_controller/CHANGELOG.md +++ b/goodwe_controller/CHANGELOG.md @@ -34,10 +34,26 @@ spread it would have created was never compared against anything. **What changes:** -* `app/price_history.py` (new) keeps the **published** prices of the last - `KEEP_DAYS` Brussels days in `/data/price_history.json`, and - `resolve_tariff()` now fills every unpublished hour of the horizon from - yesterday's same-wall-clock hour, flagged provisional. +* `app/price_history.py` (new) keeps the **published** prices of the last few + Brussels days in `/data/price_history.json`, and `resolve_tariff()` now fills + an unpublished hour of the horizon from yesterday's same-wall-clock hour, + flagged provisional. ⚠️ **"An" unpublished hour, not "every" one:** the fill + can carry only an hour yesterday actually published, so a partial yesterday + leaves the local hours it never held **out of the series** - §11.4 for those + hours, and asking for one raises rather than answering. Correct behaviour, + and the wording is now the behaviour. +* ⚠️ **A sub-hourly feed keeps every print.** `eneco_eddy` is the imbalance + product and settles per **quarter**; the fill walks whole hours to decide + what needs carrying, and that is not a rule about what a published hour may + contain. Every published point inside the horizon passes through by identity, + and a carried point is added only for a whole hour with no published coverage + at all. The first draft keyed the published set by hour - last point wins - + and a 96-point quarter-hourly feed reached the optimiser as 15 points with + its spread down from 0.143 to 0.014 EUR/kWh, on an all-firm, fully-published + series. +* The horizon is **whole hours at both ends**: `since` is floored to the hour + that contains it, and so is `until`. Asked for 48 hours at 08:17, the plan is + 48 hours long and not 49. * `resolve_tariff(history=...)` is **required and keyword-only**. A caller with no durable volume passes the explicit `NoHistory()`. A default of `None` meaning "skip the fill" would hand a caller who forgot the argument exactly @@ -64,11 +80,25 @@ difference only the status page can see. `DECISIONS/FCST-04-the-day-that-is-carried.md` carries that and two more. ⚠️ **A BROKEN CACHE IS NOT A BROKEN CONTROLLER.** No `/data`, an unreadable -file, a schema this build does not know, a malformed point - four states, four -reasons, none of them a price and none of them an empty series. In every one the -plan is still a real provider series and the controller still regulates; the -horizon is just as short as it was before this card. **No option was added and -`config.yaml` is untouched.** +file, a schema this build does not know (including a `version` of `1.0`, which +`== 1`), a **day record that is not a record**, a malformed point - each its own +state with its own reason, none of them a price and none of them an empty +series. In every one the plan is still a real provider series and the controller +still regulates; the horizon is just as short as it was before this card. +`.words` never raises, because `Controller.__init__` reads it unguarded and a +status line that throws is an add-on that does not start. + +⚠️ **AND WHERE THERE IS NO `/data`, NOTHING IS RECORDED AT ALL** - not even in +memory. An in-memory yesterday answers for one run and is gone on the next +restart, which is a cache that forgets without saying so; the honest pair is +`unavailable` on the status row and no prices carried. + +⚠️ **"YESTERDAY" IS A LOCAL DATE, AND THE ARITHMETIC IS ON THE DATE.** +`brussels()` renders at a fixed offset, so subtracting `timedelta(days=1)` from +it subtracts 24 **absolute** hours - and the Brussels day before a +spring-forward is 23 hours long. Between 00:00 and 00:59 on the morning after +the March transition that lands on the day *before* yesterday. **No option was +added and `config.yaml` is untouched.** ## 0.9.0 diff --git a/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md b/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md index e34943d..c79ecb6 100644 --- a/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md +++ b/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md @@ -1,6 +1,6 @@ # FCST-04 — what "yesterday" is, and what the fill is allowed to reach -Three decisions, recorded because each had a defensible alternative that a later +Four decisions, recorded because each had a defensible alternative that a later reader would otherwise re-open from scratch. --- @@ -17,11 +17,23 @@ no arbitrage."* Because it buys nothing the optimiser can use, and it silently replaces a fallback the operator configured. -Measured, not assumed: a fully-carried series is 100 % provisional, so -`TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on §8.3's -firm-pair rung, and `cheap_now()` returns False because `firm_points()` is -empty. The §11.4 static fallback produces *exactly* those two answers. The two -paths differ only in which numbers the status page shows. +Measured, not assumed — and stated at the reach it actually has: + +- **This one reaches control.** A fully-carried series is 100 % provisional, so + `TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on + §8.3's firm-pair rung. That is the verdict the optimiser reads: `arbitrage.py` + calls `plan.arbitrage(...)` on `fresh_tariff_plan()`, and it is the only path + by which a provider plan moves energy. The §11.4 static fallback refuses on + the same rung, for the same reason. +- **This one does not, and saying so is the point.** `TariffPlan.cheap_now()` + also returns False on a fully-carried series, because `firm_points()` is + empty — but it has **no production caller on the provider plan**. + `self.cheap` comes from `static_status()`, which builds its *own* static plan + and calls `cheap_now()` on that. So it is a property of the type, checked in + `test_static_tariff.py`, and it is not evidence about the shipped control + path. The conclusion below holds on the first bullet alone. + +The two paths therefore differ only in which numbers the status page shows. So the choice is between two identical control behaviours, and the tie-breaker is that `static_profile` is a documented option an operator fills in **for this @@ -74,9 +86,26 @@ The defence is therefore an **ordering**, not a validation: - The filled series exists only as that return value. The object never sees it again. - `_remember()` is private and called from exactly one place. -- `test_price_history.py` walks `app/` and fails if any other module names - `_remember`, and reads `carry()`'s AST to confirm the record happens before - the lookup — so a later edit that reverses them is red rather than plausible. +- `test_price_history.py` parses every shipped module under `app/`, `tools/`, + `scenarios/` and `estop/` and fails if any of them **uses the name** + `_remember` — an attribute, a bare name, a `def`, or the string constant a + `getattr` would need. By name rather than by substring, so a comment that + merely mentions the rule does not fail it and a call in `tools/` cannot pass + unseen. +- and it pins the order two ways: the **line numbers** of the two calls in + `carry()`, and — the one that matters — the **behaviour**. A feed carrying + yesterday alongside today, into an empty store, must fill the whole horizon; + reversed, the lookup finds nothing and the plan is only the hours the feed + published. Driven, so a reversal is red. + +⚠️ **The first version of that ordering check graded nothing, and this file +said it did.** It compared positions in `ast.walk`, which is breadth-first: +`self._remember(...)` is a statement and `self.yesterday(...)` is an *argument* +of `fill_from_yesterday(...)`, one level deeper, so `_remember` came out first +whichever statement was written first. Measured — shipped and reversed both +scored the same. A decorative check that a document cites as a guarantee is +worse than no check, which is why the correction is recorded here rather than +quietly made. That ordering has a second benefit worth keeping: a feed that returns yesterday alongside today refreshes yesterday's record from live data *before* it is @@ -96,3 +125,57 @@ elapsed are no longer in the spread. A cheap night that has gone used to be the series' cheapest hour, and a cycle cannot buy in an hour that is past — so the trade was silently never taken. `test_price_history.py` section 2 grades that by finding the cheapest firm hour with and without the fill and showing it moves. + +⚠️ **And section 1's identity check now grades identity.** It used to compare +`eur_per_kwh` between `_before` and `_after`, which come from two *separate* +`resolve_tariff()` calls — each parsing the feed body into its own objects, so +`is` could never have held between them however the fill were written. FCST-03's +whole anti-copy argument is `filled.points[i] is fetched.points[j]`, and floats +cannot grade it. The value comparison is kept and now says what it is (the fill +moved the window, it did not touch a price); identity is graded on a read the +file constructs and still holds a reference to. + +--- + +## 4. A sub-hourly feed passes through whole; the fill only decides which hours are missing. + +`eneco_eddy` is the Belgian imbalance product and settles **per quarter**, and +`price.py`'s `_validate_raw()` accepts any strictly-increasing spacing — so a +published hour may legitimately hold four points. + +FCST-04's first caller keyed the published set as `{_utc_hour(p.start): p for p +in r.points}`. It reads perfectly ordinary and it keeps the **last** point of +each hour. Measured on a 96-point quarter-hourly feed — all firm, fully +published, nothing to carry: the plan reached the optimiser as **15 points**, +the arbitrage spread fell from **0.143 to 0.014 EUR/kWh**, and the cheapest +"hour" became a `:45` point whose `at()` window runs into the next hour. That is +not the windowing effect above; it is a feed losing 84 % of its own prints, and +no `provisional` flag would ever have reported it. + +**The alternative was rejected:** refusing sub-hourly feeds at `_validate_raw()` +would have made the count correct by breaking the one supported product that +legitimately publishes per quarter. + +**What was done instead** keeps FCST-03's existing contract rather than weakening +it: *every* published point inside the window passes through **by identity**, +and a carried point is added only for a whole hour with **no** published +coverage at all. A carried hour is still a single hourly point whatever the +feed's own spacing is, because yesterday's profile is only known per wall-clock +hour — and carried hours are provisional, so they are out of the trade set +anyway. + +⚠️ **`_audit()` got stricter, not looser, and that was the condition on doing +this at all.** The old post-condition refused any repeated `_utc_hour` — a count +that cannot express four points in one hour. It is replaced by two rules that +**subsume** it: + +- a **published** point must *be* one of that hour's published points, by + identity, and may be emitted only once; +- a **carried** point may stand only in an hour that does not appear in the + published set at all, must be flagged `True`, and may not repeat. + +So a duplicated carried hour — the DST mispricing the count existed to catch — +still raises; so does the same published point emitted twice; and a carried +point sharing an hour with a real print now raises too, which the count never +looked at. Each of those three is driven by its own mutation in +`test_price_forecast.py`. diff --git a/goodwe_controller/DOCS.md b/goodwe_controller/DOCS.md index e657e6a..af8efbd 100644 --- a/goodwe_controller/DOCS.md +++ b/goodwe_controller/DOCS.md @@ -1433,18 +1433,41 @@ day. Until FCST-04 those other hours were simply **absent** from the plan: not priced badly, not flagged, just missing, and nothing said so. Now the add-on keeps the **published prices of the last few days** in -`/data/price_history.json` and fills every unpublished hour of the horizon with +`/data/price_history.json` and fills an unpublished hour of the horizon with yesterday's price for the same Brussels wall-clock hour, flagged `provisional: true`. Two things follow, and the second one matters more: -- **The plan says it is guessing.** *Tariff* on the status page reads - `provisional` through the morning, because half its hours are. Before, a - half-length plan looked completely settled. +- **The plan says it is guessing** — `provisional: true` on every carried hour, + which is what the optimiser reads. Before, a half-length plan looked + completely settled. - **Nothing may trade on a carried hour.** §8.3 forbids acting on a provisional price, so arbitrage drops those hours from the trade set entirely — it needs a buy hour and a sell hour that were *both* published. Carrying a price forward widens what the optimiser can **see**; it never widens what it may **do**. +**"An unpublished hour", not "every unpublished hour"** — the fill can only +carry an hour yesterday actually published. If yesterday's record is itself +partial (a first day, an outage, a feed that only ever serves the rest of the +day), the local hours it never held are **left out of the series**: §11.4 +applies to those hours, and asking the plan for one raises rather than +answering. An hour with no price is not a free hour, and the fill does not +invent a neighbour, a mean or a zero for it. + +**A quarter-hourly feed keeps all four of its prints.** `eneco_eddy` settles +per quarter, and the fill decides *which whole hours need carrying* — never what +a published hour may contain. Every point the provider published inside the +horizon is passed through untouched; a carried point is added only for a whole +hour the feed did not print at all. + +**What you can actually see on the status page.** *Price history* (below) says +what is held. *Arbitrage* carries the strategy's own sentence, which names the +firm/provisional split whenever that split is what refused a trade. *Tariff* +does **not** report the fill: it is the static-profile or Home Assistant +price-entity sentence, regenerated on the 30 s tick. The provider plan's own +line — `dynamic_dayahead from generic_dayahead, 48 h, provisional` — goes to the +**add-on log** at every refresh, alongside `§8.3: 33 of 48 hours carried forward +from yesterday's profile and flagged provisional; 15 published`. + The plan is also now cut to the horizon you asked for, which means **hours that have already elapsed are no longer in it**. That is a real change to arbitrage: a cheap night that has already gone used to sit in the series as the cheapest @@ -1458,6 +1481,13 @@ cache that cannot answer is not allowed to be a controller that cannot regulate. The store is rebuilt by the next successful fetch; on a brand-new install the first fill is possible the day after the first refresh. +The same applies to a file that is *partly* broken: a day record this build +cannot read is dropped, the row says how many were dropped and why, and the rest +of the file still loads. And where there is no `/data` at all the store records +**nothing** — not even in memory. An in-memory yesterday would answer for one +run and vanish on the next restart, which is a cache that forgets without saying +so; `unavailable` and *no prices carried* are the honest pair. + **Injection is priced separately** (`export_price_eur_kwh`). In Belgium it is often near zero, which by itself strongly favours self-consumption over export. It is a single number rather than a curve because no supported supplier diff --git a/goodwe_controller/app/main.py b/goodwe_controller/app/main.py index 5cadd9b..a92c3c2 100644 --- a/goodwe_controller/app/main.py +++ b/goodwe_controller/app/main.py @@ -596,12 +596,27 @@ class Controller: # - typically 10-24 of the 48 hours `price_horizon_h` asked for - and # `TariffPlan.arbitrage()` took its spread over that truncated window # without anything reporting that it had. Every carried hour is - # provisional, so refusal 3 drops it from the trade set: this widens - # what the optimiser can SEE, never what it may act on. + # provisional, so refusal 3 drops it from the trade set: CARRYING a + # price widens what the optimiser can SEE, never what it may act on. + # + # ⚠️ AND THE CAVEAT THAT BELONGS WITH THAT SENTENCE, because this is + # the one place a maintainer reads it on its own. THE OTHER HALF OF + # THIS CHANGE DOES MOVE ENERGY, and it is the windowing rather than the + # carrying: the filled series covers `[since, until)`, so hours that + # have ALREADY ELAPSED are no longer in the spread. A cheap night that + # has gone used to sit in the series as the day's cheapest hour, and no + # cycle can buy in an hour that is past - so the trade was quietly + # never taken. `resolve_tariff()`'s docstring, DOCS.md and the CHANGELOG + # all carry this beside the guarantee; it may not appear here without it. # # ⚠️ A BROKEN OR ABSENT STORE IS NOT AN ERROR HERE. It answers `no_data`, - # the fill carries nothing, and the plan is exactly the pre-FCST-04 - # one - short, but honest and still refusing on the same rules. + # the fill carries nothing, and the plan is the pre-FCST-04 one minus + # the elapsed hours - short, but honest and still refusing on the same + # rules. It must not be an error in the other sense either: a day + # record this build cannot read is DROPPED by `PriceHistory._load()` + # and reported in `.words`, because both the constructor above and the + # `.words` read below are unguarded, and a cache that raises here is an + # add-on that does not start. self.price_history = PriceHistory(price_history_path) _LOG.info("§8.3 price history: %s", self.price_history.words) diff --git a/goodwe_controller/app/price_forecast.py b/goodwe_controller/app/price_forecast.py index fdb7e7a..2d8d7d2 100644 --- a/goodwe_controller/app/price_forecast.py +++ b/goodwe_controller/app/price_forecast.py @@ -48,7 +48,8 @@ So the flag is not passed through anywhere in this file: the object the provider produced is the object in the output series - so there is no copy that could lose or gain a flag. `filled.points[i] is fetched.points[j]` is a checkable property, and test_price_forecast.py - checks it. + checks it. EVERY published point in the window is passed through, not one + per hour: see the sub-hourly note below. * `_audit()` RE-DERIVES the expected flag for every output point from the set of hours the feed actually published, and RAISES if any point disagrees. It is a post-condition, not a filter: it never fixes a flag, because a @@ -90,6 +91,24 @@ which is the row the spec actually writes for "no price data". If NOTHING can be produced, the result is `PriceRead.no_data(reason)` and not an empty series, because an empty series is not constructible on purpose (price.py trap 3). +⚠️ A SUB-HOURLY FEED IS NOT DECIMATED TO ONE POINT AN HOUR, and the hourly walk +below is about WHICH HOURS NEED FILLING - never about what a published hour is +allowed to contain. `eneco_eddy` is the Belgian imbalance product and settles +per QUARTER, `_validate_raw()` accepts any strictly-increasing spacing, and a +published hour may therefore hold four points. So EVERY published point inside +the window is passed through, in order, and a carried point is added only for a +whole hour with NO published coverage at all. + +The alternative was measured rather than reasoned about, because FCST-04's +first caller shipped it: keying the published set as `{_utc_hour(p.start): p +for p in r.points}` reads perfectly ordinary and keeps the LAST point of each +hour, dropping the other three. On a 96-point quarter-hourly feed - all firm, +fully published, nothing to carry - the series came out 15 points long, the +arbitrage spread collapsed from 0.143 to 0.014 EUR/kWh, and the cheapest +"hour" became a `:45` point whose `at()` window runs into the next hour. A feed +losing 84 % of its own prints is not a windowing effect and is not something a +provisional flag would ever have reported. + AC2: THE IRREVERSIBLE-ACTION GATE ================================= @@ -156,6 +175,25 @@ def _utc_hour(instant: datetime) -> datetime: return u.replace(minute=0, second=0, microsecond=0) +def _published_by_hour(read: PriceRead) -> dict: + """UTC hour -> the provider's OWN points that start inside it, in order. + + ⚠️ A TUPLE PER HOUR, NOT A POINT, AND THAT IS THE WHOLE OF THE SUB-HOURLY + RULE. `{_utc_hour(p.start): p for p in r.points}` reads identically, is one + character shorter, and silently keeps one point per hour - the last one + written. The mapping answers two DIFFERENT questions here: "does this hour + need filling at all?" is `u in published`, and "what does the output carry + for it?" is all of `published[u]`. Collapsing them is how a quarter-hourly + feed loses three quarters of its prints. + """ + out: dict = {} + read.match( + series=lambda r: [out.setdefault(_utc_hour(p.start), []).append(p) + for p in r.points], + no_data=lambda _r: []) + return {u: tuple(ps) for u, ps in out.items()} + + def _carried(start_utc: datetime, eur_per_kwh: float) -> PricePoint: """A point filled from yesterday's profile. §8.3: ALWAYS provisional. @@ -172,37 +210,69 @@ def _carried(start_utc: datetime, eur_per_kwh: float) -> PricePoint: def _audit(points, published) -> None: """Post-condition: every flag matches the point's TRUE provenance (AC1). - `published` maps a UTC hour to the point the provider produced for it. + `published` maps a UTC hour to the TUPLE of points the provider produced + for it (`_published_by_hour()`). An hour ABSENT from that mapping is an hour + the feed did not print, and a carried point is the only thing that may + stand in it. + + The rule, in the two forms an output point can take: + + * PUBLISHED - the point must BE one of `published[u]`, by identity, and + may be emitted only once. Its flag is whatever the feed said, never + forced to False: an imbalance print is published AND provisional (§9.2, + TARIFF-02), and a check that "corrected" it would be a bug wearing a + guarantee. + * CARRIED - the hour must not appear in `published` AT ALL, the flag must + be exactly True, and that hour may hold no second point. + + ⚠️ WHAT REPLACED "AT MOST ONE POINT PER HOUR", AND WHY IT IS STRICTER. This + audit began as a `seen` set: any repeated `_utc_hour` raised, because a + duplicated hour is how a DST day gets mispriced. A bare hour count cannot + express a quarter-hourly feed, where four distinct published points share + one hour legitimately - so the count was replaced by the two rules above, + which subsume it. A repeated CARRIED hour still raises; so does the same + published point emitted twice; and a carried point may no longer share an + hour with a published one, which the count never checked at all. ⚠️ RAISES, NEVER REPAIRS. A module that silently corrects its own provenance is one whose provenance means nothing; the only safe response to "this point's flag disagrees with where it came from" is to refuse to emit the series at all. """ - seen = set() + carried_hours = set() + emitted = set() for p in points: u = _utc_hour(p.start) - if u in seen: - raise ValueError(f"the filled series repeats {u.isoformat()}; a " - f"duplicated hour is how a DST day gets mispriced") - seen.add(u) src = published.get(u) - prov = Provenance.CARRIED if src is None else Provenance.PUBLISHED if src is None: + if u in carried_hours: + raise ValueError( + f"the filled series repeats the carried hour " + f"{u.isoformat()}; a duplicated hour is how a DST day gets " + f"mispriced") + carried_hours.add(u) if p.provisional is not True: raise ValueError( - f"{u.isoformat()} is {prov.value} and is flagged " - f"provisional={p.provisional!r}. §8.3: an hour filled from " - f"yesterday's profile is provisional whether or not the " - f"number turns out to be right - the flag is a claim about " - f"PROVENANCE, not about accuracy.") + f"{u.isoformat()} is {Provenance.CARRIED.value} and is " + f"flagged provisional={p.provisional!r}. §8.3: an hour " + f"filled from yesterday's profile is provisional whether " + f"or not the number turns out to be right - the flag is a " + f"claim about PROVENANCE, not about accuracy.") else: - if p is not src: + if not any(p is q for q in src): raise ValueError( - f"{u.isoformat()} is {prov.value} but the series carries a " - f"COPY of that point rather than the point itself. A copy " - f"is where the provisional flag gets lost; published points " - f"pass through by identity.") + f"{u.isoformat()} is {Provenance.PUBLISHED.value} but the " + f"series carries a COPY of that point rather than the " + f"point itself. A copy is where the provisional flag gets " + f"lost; published points pass through by identity.") + if id(p) in emitted: + raise ValueError( + f"the filled series emits the published point at " + f"{p.start.isoformat()} TWICE. The provider's own points " + f"are strictly increasing (`_validate_raw`), so a repeat " + f"is never a second print - it is a duplicated hour, which " + f"is how a DST day gets mispriced") + emitted.add(id(p)) def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead, @@ -217,11 +287,21 @@ def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead, Returns a `PriceRead`: - * a series, whose points are the provider's own objects for the hours it - published and `_carried()` points for the rest, ordered by instant; + * a series, carrying the provider's own objects - ALL of them, which on + a quarter-hourly feed is four inside one hour - for the hours it + published, and one `_carried()` point for each remaining whole hour, + ordered by instant; * `no_data` when neither source can produce a single hour - never an empty series, which is not constructible (price.py trap 3). + ⚠️ THE WINDOW IS WHOLE HOURS AT BOTH ENDS. `since` is floored to the hour + that CONTAINS it, because the current hour is still a tradeable hour; and + so is `until`, which is the half FCST-04's first draft left out. With + `since` floored and `until` compared raw, a 48 h horizon asked for at 08:17 + emitted 49 hours - the hour of `until` was still `< until` - and the count + was one too many for every `now` that is not exactly on the hour, which is + every `now` a running controller has. + ⚠️ IDEMPOTENT AND RE-RUNNABLE, WHICH IS THE §5.4 LATE-PUBLICATION CONTRACT. Day-ahead publication at 14-16 h CET can be late or fail, so the fetcher retries and this runs again against the fresher read. Published hours always @@ -234,9 +314,7 @@ def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead, raise ValueError(f"{name} must be tz-aware; a naive instant is " f"ambiguous for one hour every October") - published = read.match( - series=lambda r: {_utc_hour(p.start): p for p in r.points}, - no_data=lambda _r: {}) + published = _published_by_hour(read) # Yesterday's profile, keyed by BRUSSELS WALL-CLOCK HOUR. First point wins # when yesterday itself was a 25-hour day and printed local hour 2 twice. @@ -248,11 +326,12 @@ def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead, out = [] u = _utc_hour(since) - end = until.astimezone(timezone.utc) + end = _utc_hour(until) while u < end: - if u in published: - out.append(published[u]) # BY IDENTITY. No copy. - else: + got = published.get(u) + if got: + out.extend(got) # BY IDENTITY, and ALL of them. No copy, + else: # and no decimation to one an hour. src = by_local_hour.get(brussels(u).hour) if src is not None: out.append(_carried(u, src.eur_per_kwh)) diff --git a/goodwe_controller/app/price_history.py b/goodwe_controller/app/price_history.py index 0b2b744..464c938 100644 --- a/goodwe_controller/app/price_history.py +++ b/goodwe_controller/app/price_history.py @@ -53,6 +53,14 @@ layer that writes fine, reads fine, and is thrown away on the next restart. That is a yesterday that silently forgets, which is worse than one that says it cannot answer. +⚠️ AND THAT RULE IS CARRIED OUT IN `_remember()`, NOT ONLY IN `_save()`. FCST-04 +first shipped it half-done: `_save()` short-circuited on `not self.ok` while +`_remember()` filled `self._days` anyway, so a store with no durable volume +answered `words` = "unavailable" and `yesterday()` = A SERIES - an in-memory +yesterday that works all day and is gone on the next restart, which is exactly +the silently-forgetting cache this rule exists to refuse. A store that cannot +persist now records nothing, and both of its answers say the same thing. + THE FOUR RULES THIS FILE IS SHAPED BY ===================================== @@ -73,8 +81,20 @@ THE FOUR RULES THIS FILE IS SHAPED BY public entry point, it remembers the argument and returns the fill, and the filled series exists only as a return value that `carry()` itself never sees again. `_remember()` is private, is called from exactly one place, and - `test_price_history.py` walks the tree and fails if any module outside this - file names it. + `test_price_history.py` PARSES every shipped module under `app/`, `tools/`, + `scenarios/` and `estop/` and fails if any of them uses the NAME - an + attribute, a bare name, a def, or the string constant a `getattr` needs. By + name rather than by substring, so a comment about the rule does not fail it + and a call from `tools/` cannot pass unseen. + + ⚠️ AND THE ORDER IS GRADED BY DRIVING IT, NOT BY READING THE SOURCE. The + first version of that check compared positions in `ast.walk`, which is + breadth-first: `self._remember(...)` is a statement and `self.yesterday(...)` + is an ARGUMENT of the fill, one level deeper, so `_remember` came out first + whichever statement was written first - shipped and reversed scored the + same. What grades it now is the behaviour: a feed carrying yesterday + alongside today, into an EMPTY store, must fill the whole horizon, and + reversed it fills nothing. 2. "YESTERDAY" MEANS YESTERDAY, NOT "THE NEWEST DAY I HAVE". `newest()` is the natural accessor and it is the one that decays: an add-on @@ -139,12 +159,20 @@ from .price_forecast import fill_from_yesterday _LOG = logging.getLogger("goodwe.price.history") -# How many Brussels days of published prices are kept. The fill needs exactly -# one - yesterday - and rule 2 refuses anything older, so this is not a horizon: -# it is the number of days a record may sit unused before pruning removes it, -# and it is small because a bigger cache would only hold days no lookup can ask -# for. Three covers today, yesterday, and the day a fetch spanning midnight -# writes ahead of itself. +# How many Brussels days of published prices the SORT-ORDER cap keeps. The fill +# needs exactly one - yesterday - and rule 2 refuses anything older, so this is +# not a horizon: it is the number of days a record may sit unused before +# pruning removes it, and it is small because a bigger cache would only hold +# days no lookup can ask for. Three covers today, yesterday, and the day a +# fetch spanning midnight writes ahead of itself. +# +# ⚠️ IT IS A CAP, NOT THE RETENTION RULE, AND THREE HAS NO SLACK. The steady +# state is already {yesterday, today, tomorrow}; a feed that publishes to D+2, +# or ONE stray point in the wrong year - which sorts LAST and is therefore +# immortal under a newest-N cap - pushes yesterday out of a three-slot store +# and keeps it out, permanently and silently. So `_prune()` PINS the two days a +# lookup can actually ask for on top of the cap; see its docstring. The store +# is then bounded at KEEP_DAYS + 2 records rather than at KEEP_DAYS. KEEP_DAYS = 3 SCHEMA = 1 @@ -161,6 +189,44 @@ def _bdate(instant: datetime) -> str: return brussels(instant).strftime("%Y-%m-%d") +def _yesterday_of(instant: datetime) -> str: + """The Brussels date of the day BEFORE `instant`'s Brussels day. + + ⚠️ THE ARITHMETIC IS ON THE LOCAL DATE, NOT ON THE INSTANT, AND THAT IS A + DST DEFECT THIS FILE SHIPPED. `brussels()` renders at the FIXED offset in + force at that instant, so `brussels(stamp) - timedelta(days=1)` subtracts 24 + ABSOLUTE hours from a wall clock - and the Brussels day before a + spring-forward is 23 hours long. Measured at 2026-03-30T00:30+02:00, the + first hour of the morning after the March transition: it returned + 2026-03-28, so the fill carried a TWO-DAY-OLD profile flagged as yesterday's + (or, with only the last three days held, answered `no_data` naming a date + that is not yesterday). + + A calendar day minus one day is `date` arithmetic, and a `date` has no + offset to lose. The 25-hour October day is right for the same reason. + """ + return (brussels(instant).date() - timedelta(days=1)).strftime("%Y-%m-%d") + + +def _n_points(rec) -> int: + """How many points a day record holds. 0 for anything that is not one. + + ⚠️ TOLERANT, AND IT IS THE SECOND LINE OF DEFENCE, NOT THE FIRST. `_load()` + drops a malformed day record and says so, so this should never see one. It + is written this way because `words` is read from `Controller.__init__` and + from every status render: a status LINE that raises takes the add-on down, + and "the cache is broken" must never be able to mean "the controller does + not start". That is exactly what FCST-04 shipped - `{"days": {"2026-01-14": + "oops"}}` gave `ok=True`, `last_error=None`, and an `AttributeError` out of + `words`, `yesterday()` and `resolve_tariff()` alike, with no self-heal + because `_remember()` raised before it could rewrite the file. + """ + if not isinstance(rec, dict): + return 0 + pts = rec.get("points") + return len(pts) if isinstance(pts, (list, tuple)) else 0 + + class PriceHistory: """The published price series of the last few Brussels days, in /data. @@ -214,8 +280,13 @@ class PriceHistory: "next refresh has written a full day", self.path, self.last_error) return - if not isinstance(blob, dict) or blob.get("version") != SCHEMA: - found = blob.get("version") if isinstance(blob, dict) else None + found = blob.get("version") if isinstance(blob, dict) else None + # ⚠️ THE TYPE IS PART OF THE VERSION. `1.0 == 1` in Python, so a float + # written by some other tool would pass a bare `!=` and be read as + # schema 1 - a version check that cannot tell 1 from 1.0 is one that + # would not notice a writer this build has never met. + if not isinstance(blob, dict) or isinstance(found, bool) \ + or not isinstance(found, int) or found != SCHEMA: self.last_error = f"schema {found!r}, not {SCHEMA}" _LOG.warning("price history at %s has %s - discarding it rather " "than guessing what its points mean", @@ -226,11 +297,55 @@ class PriceHistory: self.last_error = "no `days` object" _LOG.warning("price history at %s has %s", self.path, self.last_error) return - self._days = days + self._days = self._usable_days(days) + + def _usable_days(self, days: dict) -> dict: + """The day records that are shaped like day records. The rest are dropped. + + ⚠️ THE ASYMMETRY THAT PRODUCED THE DEFECT: `_parse_points()` below is + carefully defensive about every field of a POINT, and `_load()` was not + defensive about the record holding them at all. `{"version": 1, "days": + {"2026-01-14": "oops"}}` therefore loaded as `ok=True`, + `last_error=None` - and then `rec.get(...)` raised `AttributeError` out + of `words`, `yesterday()` and `resolve_tariff()`. `PriceHistory(...)` + and `.words` are both unguarded in `Controller.__init__`, so THE ADD-ON + DID NOT START; and it never self-healed, because `_remember()` raised on + the same record before it could reach `_save()`. + + A broken cache is a short horizon. It is not a controller that will not + regulate a battery, and §6's rule - absent machinery must not disable + the loop - does not stop applying because the machinery is a JSON file. + + Dropped, not repaired: an unusable record has nothing in it to keep, and + the next `_remember()` writes a fresh one under that date. + """ + kept, dropped = {}, [] + for key, rec in days.items(): + if not isinstance(key, str) or not isinstance(rec, dict) \ + or not isinstance(rec.get("points"), list): + dropped.append(str(key)) + continue + kept[key] = rec + if dropped: + # Recorded, not silent: `yesterday()` and `words` both report + # `last_error`, and a cache that quietly discards half a file reads + # to an operator exactly like a normal first day. + self.last_error = (f"{len(dropped)} unusable day record(s) dropped " + f"({', '.join(sorted(dropped))})") + _LOG.warning("price history at %s: %s - the controller starts " + "anyway and the horizon is as short as the feed makes " + "it", self.path, self.last_error) + return kept def _save(self) -> None: if not self.ok: return + # ⚠️ THE TEMP FILE IS REMOVED ON EVERY FAILING PATH. A write that dies + # between `mkstemp` and `os.replace` used to leave its temp behind, and + # the refresh task runs 48 times a day: a persistent write failure - + # a full volume being the likeliest cause - leaked 48 inodes a day onto + # the volume that was already the problem. + tmp = None try: fd, tmp = tempfile.mkstemp(dir=os.path.dirname(self.path)) with os.fdopen(fd, "w", encoding="utf-8") as fh: @@ -238,9 +353,16 @@ class PriceHistory: # Atomic: a power cut mid-write must not leave a truncated file # that parses as an empty day. os.replace(tmp, self.path) + tmp = None # it IS the file now; do not unlink except OSError as err: self.last_error = f"could not write ({err})" _LOG.error("could not persist price history: %s", err) + finally: + if tmp is not None: + try: + os.unlink(tmp) + except OSError: # nothing further to try, and a + pass # failed cleanup is not a fault # -- the one public entry point ----------------------------------------- def carry(self, read: PriceRead, *, since: datetime, until: datetime, @@ -252,12 +374,15 @@ class PriceHistory: refreshes yesterday's record from live data before it is consumed as the profile; and the filled series - the one thing that must never be recorded - exists only as this method's return value, which this object - never sees again. See rule 1. + never sees again. See rule 1, including how that order is graded. - Returns whatever `fill_from_yesterday()` returns: a series over - [since, until) whose published hours are the provider's own point - objects and whose remaining hours are carried and flagged, or - `no_data` when neither source can produce a single hour. + Returns whatever `fill_from_yesterday()` returns: a series over the + whole hours of [since, until), carrying EVERY point the provider + published inside it - by identity, and four of them in one hour on a + quarter-hourly feed - plus one carried point for each remaining whole + hour FOR WHICH YESTERDAY HELD THAT WALL-CLOCK HOUR. An hour neither + source can price is left out (§11.4 for that hour), and `no_data` comes + back when that is every hour. """ self._remember(read, stamp) return fill_from_yesterday(read, yesterday=self.yesterday(stamp), @@ -274,6 +399,15 @@ class PriceHistory: nothing outside this file names it, and `test_price_history.py` checks the tree for that rather than trusting the underscore. """ + if not self.ok: + # ⚠️ A STORE THAT CANNOT PERSIST RECORDS NOTHING. See the module + # docstring's third warning: filling `self._days` here would give a + # site with no /data an in-memory yesterday that answers a series + # all day and is gone on the next restart - a cache that forgets + # without saying so, which is the one thing that rule refuses. The + # honest degraded state is `words` = "unavailable" and `yesterday()` + # = `no_data`, and those two must agree. + return points = read.match(series=lambda r: r.points, no_data=lambda _r: ()) if not points: return @@ -290,7 +424,8 @@ class PriceHistory: # Rule 4: MERGE by instant. A later read updates the hours it # carries and leaves the rest; a revised price for an hour already # held wins, because it is the newer statement about that hour. - held = {q["start"]: q for q in rec.get("points", []) + raw = rec.get("points") + held = {q["start"]: q for q in (raw if isinstance(raw, list) else ()) if isinstance(q, dict) and "start" in q} held[p.start.isoformat()] = p.canonical() rec["points"] = sorted(held.values(), key=lambda q: q["start"]) @@ -303,16 +438,34 @@ class PriceHistory: len(points), ", ".join(sorted(touched)), len(self._days)) def _prune(self, stamp: datetime) -> None: - """Keep the newest KEEP_DAYS records. Older ones can never be a lookup. + """Drop records no lookup can reach. A sort-order cap, plus two pins. - Pruning is by SORT ORDER, not by arithmetic against `stamp`: a clock - that jumped - the failure mode WITNESS-03 spent a session on - would + ⚠️ NOTHING IS DELETED BY ARITHMETIC AGAINST THE CLOCK. A clock that + jumped - the failure mode WITNESS-03 spent a session on - would otherwise delete every record in one pass and leave the fill with nothing on the morning after, which is the one morning it is for. + Deleting on a bad clock is unrecoverable; holding a few days too many + is not, so the cap is `sorted(...)[-KEEP_DAYS:]` and stays that way. + + ⚠️ AND A CAP ALONE IS DEFEATED BY ONE POINT IN THE WRONG YEAR. Sorting + is lexicographic on the date string, so a stray far-future record sorts + LAST and is immortal - it consumes a slot for good, and with the steady + state already {yesterday, today, tomorrow} filling all three, yesterday + is what falls out. A feed that publishes to D+2 does the same thing + without any stray at all. So the two days `yesterday()` can ACTUALLY be + asked for are pinned on top of the cap. + + `stamp` is READ here, which it was not before: the pin is the only use + of it, and it is additive. Pinning can only RETAIN a record, never + delete one, so a jumped `stamp` pins two dates that are not held and + changes nothing - the WITNESS-03 property is preserved by construction + rather than by remembering to check it. """ if len(self._days) <= KEEP_DAYS: return - for day in sorted(self._days)[:-KEEP_DAYS]: + keep = set(sorted(self._days)[-KEEP_DAYS:]) + keep |= {_bdate(stamp), _yesterday_of(stamp)} + for day in [d for d in self._days if d not in keep]: del self._days[day] def yesterday(self, stamp: datetime) -> PriceRead: @@ -327,7 +480,7 @@ class PriceHistory: f"yesterday({stamp!r}): a naive datetime is not an instant, " f"and 'which Brussels day was yesterday' is exactly the " f"question a wall-clock reading cannot answer.") - want = _bdate(brussels(stamp) - timedelta(days=1)) + want = _yesterday_of(stamp) if self.last_error and not self._days: return PriceRead.no_data( f"the price history at {self.path} is {self.last_error}, so " @@ -363,14 +516,26 @@ class PriceHistory: # -- observability ------------------------------------------------------- @property def words(self) -> str: - """One line for the status page. Says what is held, or why nothing is.""" + """One line for the status page. Says what is held, or why nothing is. + + ⚠️ THIS MUST NOT RAISE, EVER. It is read once from `Controller.__init__` + and again on every status render, and neither call site guards it - so + an exception here is an add-on that does not start. It went out that + way: a day record that was a string instead of a dict raised + `AttributeError` from the `len(...)` below. `_load()` now drops such a + record and `_n_points()` is total, so this has two independent reasons + not to fail on a broken file. + """ if not self.ok: return f"unavailable - {self.last_error}" if not self._days: why = f" - {self.last_error}" if self.last_error else "" return f"empty{why} - nothing to carry forward yet" - return "; ".join(f"{d} ({len(self._days[d].get('points') or ())} h)" - for d in sorted(self._days)) + held = "; ".join(f"{d} ({_n_points(self._days[d])} h)" + for d in sorted(self._days, key=str)) + # A file that PARTLY loaded says so, or the dropped days read as days + # that were simply never written. + return held + (f" [{self.last_error}]" if self.last_error else "") def _parse_points(raw) -> tuple: diff --git a/goodwe_controller/app/static_tariff.py b/goodwe_controller/app/static_tariff.py index dfd5ddb..dea20e3 100644 --- a/goodwe_controller/app/static_tariff.py +++ b/goodwe_controller/app/static_tariff.py @@ -1053,6 +1053,14 @@ async def resolve_tariff(opts: dict, since: datetime, until: datetime, a difference only the status page can see. See price_history.py's closing note and DECISIONS/FCST-04-the-day-that-is-carried.md. + ⚠️ THE FILL IS PER WHOLE HOUR AND ITS COVERAGE IS CONDITIONAL. Every point + the provider published inside the window passes through BY IDENTITY - four + of them in one hour on a quarter-hourly product like `eneco_eddy` - and a + carried point is added only for a whole hour the feed did not print at all + AND yesterday did. An hour neither source can price is left out, which is + §11.4 for that hour: `plan.price_at()` raises rather than answering, and an + hour with no price is not a free one. + ⚠️ THE FILLED SERIES IS WINDOWED TO [since, until), AND THAT IS A CHANGE. `load_prices()` returns whatever the feed sent, past hours included, and `TariffPlan.arbitrage()` took its spread over all of it - so an hour that diff --git a/goodwe_controller/test_price_forecast.py b/goodwe_controller/test_price_forecast.py index 26e1194..cb2aeb6 100644 --- a/goodwe_controller/test_price_forecast.py +++ b/goodwe_controller/test_price_forecast.py @@ -272,16 +272,19 @@ check("the provenance audit REFUSES a carried hour flagged non-provisional - " "the 'stale but plausible' point, which is SAFETY-18's frozen-but-fresh " "meter one layer up", raises(lambda: _fc._audit([_stale_copy], {}), ValueError)) +# ⚠️ `published` MAPS AN HOUR TO A TUPLE, not to a point (FCST-04 H1). An hour +# may legitimately hold four published points - see the quarter-hourly block +# below - so the audit takes the whole set and checks membership by identity. check("...and refuses a published hour that arrived as a COPY rather than as " "the published object, which is where a flag gets lost", raises(lambda: _fc._audit( [PricePoint(start=TODAY_PART.points[0].start, eur_per_kwh=TODAY_PART.points[0].eur_per_kwh, provisional=TODAY_PART.points[0].provisional)], - {D0: TODAY_PART.points[0]}), ValueError)) + {D0: (TODAY_PART.points[0],)}), ValueError)) check("...and never repairs a flag: the audit's only outcomes are silence and " "an exception, so a series it passed was already correct", - _fc._audit(list(FILLED.points), {u: TODAY_PART.points[i] + _fc._audit(list(FILLED.points), {u: (TODAY_PART.points[i],) for i, u in enumerate(DAY[:PUBLISHED_N])}) is None) @@ -298,6 +301,81 @@ check("a PUBLISHED point that the feed itself flagged provisional keeps that " and all(p.provisional for p in _eddy_filled.points[:2]) and _eddy_filled.points[0] is _eddy.points[0]) +# ⚠️ AND A SUB-HOURLY FEED IS NOT DECIMATED TO ONE POINT AN HOUR (FCST-04 H1). +# `eneco_eddy` settles per QUARTER and `_validate_raw()` accepts any +# strictly-increasing spacing, so a published hour can hold four points. The +# fill walks WHOLE HOURS to decide what needs carrying; that must not become a +# rule about what a published hour may contain. Measured on FCST-04's first +# draft, which keyed the published set `{_utc_hour(p.start): p ...}`: a +# 96-point feed came out 15 points long and its arbitrage spread fell from +# 0.143 to 0.014 EUR/kWh, on an ALL-FIRM, fully-published series. +_QN = 16 # four hours of quarters +_QUARTERS = PriceRead.series( + [PricePoint(start=brussels(D0 + timedelta(minutes=15 * n)), + eur_per_kwh=round(0.100000 + 0.011300 * n, 6), + provisional=False) + for n in range(_QN)], + PriceProduct.REALTIME, "eneco_eddy", D0) +_QHOURS = {_fc._utc_hour(p.start) for p in _QUARTERS.points} +check(f"the quarter-hourly fixture really is sub-hourly - {_QN} points across " + f"{len(_QHOURS)} hours - or every check below passes for an hourly feed " + f"too and grades nothing", + len(_QUARTERS.points) == _QN and len(_QHOURS) == _QN // 4) +_qfill = derived(lambda: fill_from_yesterday(_QUARTERS, yesterday=YESTERDAY, + since=D0, until=D1, stamp=D0)) +_qfirm = derived(lambda: [p for p in _qfill.points if not p.provisional]) +check(f"every one of the {_QN} published points survives the fill BY IDENTITY " + f"({len(_qfirm)} kept). One-per-hour keying would leave {_QN // 4}, and " + f"the spread over the hours the feed actually published would collapse " + f"with them", + len(_qfirm) == _QN + and all(any(p is q for q in _QUARTERS.points) for p in _qfirm)) +check(f"...and the {len(DAY) - len(_QHOURS)} whole hours it published NOTHING " + f"for are carried, ONE point each: a carried hour is hourly whatever the " + f"feed's spacing is, because yesterday's profile is only known per " + f"wall-clock hour", + _qfill.kind == "series" + and len([p for p in _qfill.points if p.provisional]) + == len(DAY) - len(_QHOURS)) +check("...and the filled series is still ordered by instant across the two " + "kinds, so `at()` and `window()` are not reading a shuffled list", + _qfill.kind == "series" + and [p.start.astimezone(UTC) for p in _qfill.points] + == sorted(p.start.astimezone(UTC) for p in _qfill.points)) + +_PUB_Q = _fc._published_by_hour(_QUARTERS) +# ⚠️ `derived`, NOT A RAW CALL. Under the decimating mutation this check +# exists for, `_audit` RAISES here - and a harness that dies on the mutation it +# is grading reports a traceback instead of a FAIL, which MUTATION.md forbids. +check("the audit ACCEPTS four published points inside one hour, which is the " + "case the repeated-hour COUNT it replaced could not express", + derived(lambda: _fc._audit(list(_QUARTERS.points), _PUB_Q)) is None) +check("...and still REFUSES a repeated CARRIED hour, which is the DST " + "mispricing that count existed to catch", + raises(lambda: _fc._audit([_fc._carried(D0, 0.11), + _fc._carried(D0, 0.12)], {}), ValueError)) +check("...and refuses the SAME published point emitted twice - the provider's " + "points are strictly increasing, so a repeat is never a second print", + raises(lambda: _fc._audit([_QUARTERS.points[0], _QUARTERS.points[0]], + _PUB_Q), ValueError)) +check("...and refuses a CARRIED point standing in an hour the feed DID " + "publish, which the old count never looked at", + raises(lambda: _fc._audit([_fc._carried(D0, 0.11)], _PUB_Q), ValueError)) + +# ⚠️ THE HORIZON IS WHOLE HOURS AT BOTH ENDS (FCST-04 L8). `since` is floored +# and `until` was not, so a 48 h request made at any instant off the hour came +# back 49 hours long. Graded at an OFF-THE-HOUR instant, because on the hour +# the two implementations agree and the check would grade nothing. +_off = D0 + timedelta(hours=8, minutes=17, seconds=43) +for _h in (1, 12, 48): + _n = derived(lambda h=_h: len(fill_from_yesterday( + TODAY_PART, yesterday=YESTERDAY, since=_off, + until=_off + timedelta(hours=h), stamp=_off).points)) + check(f"a {_h} h horizon asked for at {_off.isoformat()} - 17 min 43 s " + f"past the hour, the shape a running controller always has - is " + f"{_h} points long, not {_h + 1}", + _n == _h) + # The structural half: the constructor has nothing to pass, and the file is # scanned so the pass-through idiom cannot come back looking ordinary. check("_carried() has NO `provisional` parameter, so a caller has nothing to " diff --git a/goodwe_controller/test_price_history.py b/goodwe_controller/test_price_history.py index e5bc175..c547f1b 100644 --- a/goodwe_controller/test_price_history.py +++ b/goodwe_controller/test_price_history.py @@ -25,9 +25,14 @@ cache cannot decay by being fed its own output. that stays true while the NUMBER rots. Graded by driving `carry()` twice and reading the file back. AC4 A STORE THAT CANNOT ANSWER SAYS SO. No file, no record for yesterday, an - unreadable file, a record whose points are malformed - four different + unreadable file, a schema this build does not know, A DAY RECORD THAT IS + NOT A RECORD, a record whose points are malformed - six different reasons, none of them a price, none of them an empty series, and the - controller keeps its pre-FCST-04 plan through all four. + controller keeps its pre-FCST-04 plan through all of them. ⚠️ THE + CRASHING ONE IS THE POINT: `{"days": {"2026-01-14": "oops"}}` loaded as + healthy and then raised `AttributeError` out of every accessor, and both + `PriceHistory(...)` and `.words` are unguarded in `Controller.__init__` + - so the add-on did not start, and never self-healed. THE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will: @@ -49,13 +54,33 @@ THE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will: - THE MERGE FIXTURE MUST REVISE A PRICE TO A DIFFERENT NUMBER, or merge and replace are indistinguishable. - THE PRUNE FIXTURE MUST HOLD MORE DAYS THAN KEEP_DAYS, or pruning is not - exercised at all. + exercised at all. ⚠️ THIS FILE BROKE ITS OWN RULE AND IT COST A LIVE + MUTATION: §8's clock-jump check seeded ONE day, `_prune`'s early + return fired, and the body it is named after never ran. The mutant that + pruned by arithmetic against the clock - which empties the store on + exactly the morning the fill is for - survived the whole 29-file suite. + The fixture is now WRITTEN TO A FILE and reopened, because `_load()` + does not prune and that is how a store legitimately holds more days + than the cap. + - NOW MUST NOT BE ON AN HOUR BOUNDARY. `datetime.now()` never is, and a + horizon count taken at exactly :00:00 was green for a fill that emitted + HORIZON + 1 hours at every other instant. + - A SUB-HOURLY FIXTURE MUST REALLY BE SUB-HOURLY, or "every published + print survives" passes for an hourly feed and grades nothing. * A FAILED READ IS NOT AN OBSERVATION. Every degraded store is graded on the TYPE it answers with (`no_data` and its reason), never on a falsy value, and the malformed-point case is checked for the specific defect - a price defaulted to 0.0 - rather than for a count. + * AND A STRUCTURAL CHECK MUST BE ABLE TO FAIL. §3's ordering check compared + `ast.walk` positions, which grade NESTING DEPTH: `self._remember(...)` is a + statement and `self.yesterday(...)` is an argument of the fill, so + `_remember` came out first whichever statement was written first - measured + by reversing them and watching the check stay green, while DECISIONS §3 and + the module docstring both cited it as making a reversal "red rather than + plausible". It is now line numbers AND the behaviour itself. + ⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run `tools/mutation_run.sh`, which does. See MUTATION.md and TEST-15. """ @@ -72,7 +97,7 @@ from datetime import datetime, timedelta, timezone from app import price_history as _ph from app.price import ( - PricePoint, PriceProduct, PriceRead, brussels, + Absence, PricePoint, PriceProduct, PriceRead, brussels, ) from app.price_history import KEEP_DAYS, NoHistory, PriceHistory, SCHEMA from app.static_tariff import resolve_tariff @@ -111,6 +136,42 @@ def raises(fn, exc): return False +class _Raised: + """What a check saw instead of a value. Equals nothing, is falsy, and is + NOT a default.""" + + def __init__(self, e): + self.e = e + + def __eq__(self, other): + return False + + def __hash__(self): + return id(self) + + def __bool__(self): + return False + + def __repr__(self): + return f"" + + +def derived(fn): + """A value a check grades. An exception becomes `_Raised`. + + ⚠️ NOT A DEFAULT, AND NOT A BARE CALL EITHER. MUTATION.md's rule is that + "it was refused" and "it broke on the way to being refused" must not grade + the same - and the accessors below are exactly the ones the unvalidated + day record used to raise from, so a raw call here would take the harness + down on the mutation it is meant to catch and report nothing at all. + Copied from test_price_forecast.py. + """ + try: + return fn() + except Exception as e: # noqa: BLE001 + return _Raised(e) + + def run(coro): return asyncio.run(coro) @@ -169,13 +230,30 @@ def at_local(bdate: str, hh: int) -> datetime: return next(u for u in bday_hours(bdate) if brussels(u).hour == hh) -NOW = at_local(TODAY_D, 9) +# ⚠️ AND IT IS NOT ON THE HOUR. A controller's `now` is +# `datetime.now(timezone.utc)`; it is :17:43 past as often as anything else and +# it is never :00:00. The first draft of this file used the exact hour, and +# §1's `_n_after == HORIZON_H` was green only because of it: `since` was +# floored to its hour while `until` was compared raw, so a 48 h request made at +# any other instant came back 49 hours long. A fixture has to have the shape +# production has, or a count grades an alignment the running system never gets. +NOW = at_local(TODAY_D, 9) + timedelta(minutes=17, seconds=43) + +# Where the WINDOW starts: the whole hour that contains NOW. The hour a +# controller is standing in is still a tradeable hour, so the fill emits from +# here and not from NOW - and every count below is derived against this rather +# than against the instant. +SINCE_HOUR = NOW.replace(minute=0, second=0, microsecond=0) # Today's feed, in EUR/MWh, keyed by BRUSSELS local hour and all distinct. # ⚠️ THE CHEAPEST HOUR IS 03:00 LOCAL - ALREADY ELAPSED AT 09:00 - and that is # AC2's stimulus, asserted below rather than assumed. TODAY_MWH = {h: 120.0 + h for h in range(24)} TODAY_MWH[3] = 5.0 # the cheap night that has already gone +TODAY_MWH[9] = 250.0 # the hour NOW falls in - dear, so the + # cheapest hour AHEAD is strictly + # later than NOW and "it moved to an + # hour still ahead" is exact TODAY_MWH[19] = 260.0 # the dear evening still ahead # Yesterday's published prices, DIFFERENT AT EVERY LOCAL HOUR from today's, so @@ -230,9 +308,16 @@ check(f"a Brussels day is 24 hours long in the fixture helper, on both fixture " len(TODAY_H) == 24 and len(YDAY_H) == 24 and {brussels(u).hour for u in TODAY_H} == set(range(24))) +check(f"NOW is {NOW.isoformat()} - deliberately NOT on an hour boundary. A " + f"horizon count taken at exactly :00:00 is green for a fill that emits " + f"HORIZON + 1 hours at every instant a controller actually runs at", + (NOW.minute, NOW.second, NOW.microsecond) != (0, 0, 0)) + # The feed serves TODAY IN FULL - which is what a day-ahead feed does - so the # hours it publishes that are still AHEAD are a strict subset of the horizon. -_ahead = [u for u in TODAY_H if u >= NOW] +# ⚠️ AGAINST SINCE_HOUR, NOT NOW: the 09:00 point started before NOW and its +# hour still contains NOW, so it is a published hour of the window. +_ahead = [u for u in TODAY_H if u >= SINCE_HOUR] check(f"at 09:00 the hours the feed published that have NOT elapsed are a " f"STRICT SUBSET of the {HORIZON_H} h horizon ({len(_ahead)} of " f"{HORIZON_H}), so 'the fill widened the plan' and 'the fill did " @@ -254,8 +339,8 @@ check(f"AC2's stimulus is real: the cheapest hour of the whole published day is f"({brussels(_cheapest_ahead).hour:02d}:00). Without this the windowed " f"and unwindowed answers would be the same point and AC2 would grade " f"nothing", - _cheapest_all != _cheapest_ahead and _cheapest_all < NOW - and _cheapest_ahead >= NOW) + _cheapest_all != _cheapest_ahead and _cheapest_all < SINCE_HOUR + and _cheapest_ahead > NOW) check(f"KEEP_DAYS is {KEEP_DAYS}, and the prune fixture below files more days " f"than that, so pruning is exercised rather than skipped", KEEP_DAYS < 6) @@ -314,11 +399,33 @@ check("AC1: exactly the hours the feed published are FIRM, and exactly the " and len(_got_carried) == HORIZON_H - len(_ahead)) _before_by_start = {p.start.astimezone(UTC): p for p in _before.read.points} -check("...and every published point is the PROVIDER'S OWN OBJECT, passed " - "through by identity - a copy is where the flag gets lost (FCST-03)", +# ⚠️ THIS IS A VALUE CHECK AND SAYS SO. `_before` and `_after` come from two +# separate `resolve_tariff()` calls, each of which parsed the feed body into +# its OWN objects, so `is` could never hold between them however the fill were +# written - a check claiming identity here would be claiming a property its +# own fixture makes untestable. What it does grade is that the fill moved the +# WINDOW and did not touch a PRICE. +check("...and every published point carries the same NUMBER as the unfilled " + "plan's for that instant: the fill changed which hours are in the " + "series, not what any of them costs", all(p.eur_per_kwh == _before_by_start[p.start.astimezone(UTC)].eur_per_kwh for p in _after.read.points if not p.provisional)) +# ...and identity itself, graded on a read this file constructed and still +# holds a reference to. FCST-03's whole anti-copy argument is +# `filled.points[i] is fetched.points[j]`, and floats cannot grade it. +_h1b = store() +_ph.PriceHistory._remember(_h1b, series_of(YDAY_H, YESTERDAY_MWH), NOW) +_read_id = series_of(TODAY_H, TODAY_MWH) +_filled_id = _h1b.carry(_read_id, since=NOW, until=_until, stamp=NOW) +_firm_id = [p for p in _filled_id.points if not p.provisional] +check(f"...and IDENTITY, graded as identity: all {len(_firm_id)} firm points " + f"of a filled series ARE the provider's own objects (`is`), not copies " + f"carrying the same numbers. A copy is where the provisional flag gets " + f"lost (FCST-03)", + len(_firm_id) == len(_ahead) + and all(any(p is q for q in _read_id.points) for p in _firm_id)) + # The carried numbers are yesterday's, at the same BRUSSELS wall-clock hour. # ⚠️ THE LOOKUP IS BUILT BY SEARCH over yesterday's own instants, not by # subtracting 24 h from today's - a fill that subtracted a fixed day is exactly @@ -336,6 +443,36 @@ check(f"...and every carried number is YESTERDAY's price for the same BRUSSELS " f"({sum(_carried_ok)}/{len(_carried_ok)})", _carried_ok and all(_carried_ok)) +# ⚠️ "FILLS EVERY UNPUBLISHED HOUR" IS CONDITIONAL, AND THIS IS THE CONDITION. +# A yesterday that is itself partial fills only the local hours it HOLDS; the +# rest are left out, which is §11.4 for those hours and emphatically not a +# zero. The behaviour is right and the wording was not - the CHANGELOG and DOCS +# said "every", unqualified, and nothing here exercised it. +_hp = store() +_PART_LOCAL = list(range(12)) # yesterday published 00:00..11:00 only +_ph.PriceHistory._remember( + _hp, series_of([at_local(YDAY_D, h) for h in _PART_LOCAL], YESTERDAY_MWH), + NOW) +_plan_part = run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), now=NOW, + history=_hp)) +_window_hours = [SINCE_HOUR + timedelta(hours=n) for n in range(HORIZON_H)] +_ahead_utc = {u.astimezone(UTC) for u in _ahead} +_expect_covered = {u for u in _window_hours + if u in _ahead_utc or brussels(u).hour in _PART_LOCAL} +_got_covered = {p.start.astimezone(UTC) for p in _plan_part.read.points} +check(f"a PARTIAL yesterday fills only the local hours it holds: " + f"{len(_got_covered)} of the {HORIZON_H} horizon hours are covered, " + f"re-derived from the fixture as {len(_expect_covered)}. The hours " + f"yesterday never published stay ABSENT", + _got_covered == _expect_covered and 0 < len(_got_covered) < HORIZON_H) +_gap = sorted(set(_window_hours) - _expect_covered)[0] +check(f"...and an uncovered hour ({_gap.isoformat()}) raises `Absence` rather " + f"than answering a price. §11.4 applies to that hour: a hole in the " + f"series is not a free hour, and this is why the wording had to stop " + f"saying 'every'", + raises(lambda: _plan_part.read.at(_gap), Absence)) + check("the plan reports itself PROVISIONAL now, which it did not before - the " "old plan claimed full firmness over a horizon it covered half of", _after.provisional and not _before.provisional) @@ -370,8 +507,7 @@ check("...and the new one is the cheapest hour the feed published that has NOT " check("...and no point of the filled series starts before the requested " "`since` hour: the window is [since, until) and the past is not in it", - min(p.start.astimezone(UTC) for p in _after.read.points) - >= NOW.replace(minute=0, second=0, microsecond=0)) + min(p.start.astimezone(UTC) for p in _after.read.points) >= SINCE_HOUR) # ⚠️ THE HONEST LIMIT OF THIS TICKET, checked so it cannot be overclaimed # later: the CARRIED hours buy no trade. §8.3 forbids acting on a provisional @@ -386,6 +522,76 @@ check("the carried hours are NOT in the trade set - the verdict's cheapest and " and len(_firm_after) == len(_ahead)) +# --------------------------------------------------------------------------- # +print() +print("2b. a QUARTER-HOURLY feed keeps every print, and its spread with them") + +# ⚠️ THE DEFECT THIS SECTION EXISTS FOR. `eneco_eddy` is the Belgian imbalance +# product and settles per QUARTER; `_validate_raw()` accepts any +# strictly-increasing spacing, so a published hour may hold four points. +# FCST-04's first draft keyed the published set `{_utc_hour(p.start): p ...}` - +# LAST POINT OF THE HOUR WINS, the other three dropped - and on an all-firm, +# fully-published 96-point feed the plan came out 15 points long with its +# arbitrage spread down from 0.143 to 0.014 EUR/kWh. That is not the windowing +# effect AC2 is about; it is prints being thrown away, and no flag reports it. +_QSTEP_MIN = 15 +_qrows = [] +for _n in range(len(TODAY_H) * (60 // _QSTEP_MIN)): + _qu = TODAY_H[0] + timedelta(minutes=_QSTEP_MIN * _n) + # A spiky within-the-hour profile, so decimation is visible in the SPREAD + # and not only in a count. + _qrows.append({"start": _qu.isoformat(), + "price_eur_mwh": 100.0 + 40.0 * (_n % 4) + brussels(_qu).hour}) +_qbody = json.dumps({"data": _qrows}).encode() +check(f"the fixture really is sub-hourly - {len(_qrows)} points across " + f"{len(TODAY_H)} Brussels hours - or every check below passes for an " + f"hourly feed too and grades nothing", + len(_qrows) == 4 * len(TODAY_H) + and len({_r["start"] for _r in _qrows}) == len(_qrows)) + +_q_before = run(resolve_tariff(CFG, NOW, _until, transport=transport_for(_qbody), + now=NOW, history=NoHistory())) +_hq = store() +_ph.PriceHistory._remember(_hq, series_of(YDAY_H, YESTERDAY_MWH), NOW) +_q_after = run(resolve_tariff(CFG, NOW, _until, transport=transport_for(_qbody), + now=NOW, history=_hq)) +# What the feed published INSIDE the window, derived from the unfilled plan. +_q_win = [p for p in _q_before.read.points + if p.start.astimezone(UTC) >= SINCE_HOUR] +_q_firm = [p for p in _q_after.read.points if not p.provisional] +check(f"every print the window contains survives the fill: {len(_q_firm)} firm " + f"points against the {len(_q_win)} the feed published in that window. " + f"One-per-hour keying leaves {len(_q_win) // 4}", + len(_q_win) > 0 and len(_q_firm) == len(_q_win)) +check("...and they are the SAME INSTANTS AND PRICES the feed published, so " + "the count is not being made up by four hourly points wearing " + "quarter-hourly starts", + {(p.start.astimezone(UTC), p.eur_per_kwh) for p in _q_firm} + == {(p.start.astimezone(UTC), p.eur_per_kwh) for p in _q_win}) +check("...and no two of them share a whole hour by accident: the fixture puts " + "four distinct prices inside every published hour, which is what the " + "decimated version could not represent", + len({p.start.astimezone(UTC).replace(minute=0) for p in _q_firm}) + == len(_q_firm) // 4 > 0) + +_q_spread_want = round(max(p.eur_per_kwh for p in _q_win) + - min(p.eur_per_kwh for p in _q_win), 6) +_qv = _q_after.arbitrage(**_A) +_q_spread_got = _qv.match(allowed=lambda v: v.spread_eur_kwh, + refused=lambda _v: None) +check(f"...and the spread is the fixture's own max - min over that window " + f"({_q_spread_want} EUR/kWh, measured {_q_spread_got}), not the " + f"flattened figure a decimated series produces", + _qv.kind == "allowed" and _q_spread_got is not None + and abs(_q_spread_got - _q_spread_want) < 1e-9) +check("...and the cheapest firm point is one the FEED published, at its own " + "sub-hourly instant - not a re-derived hourly one", + _qv.kind == "allowed" + and any(_qv.cheapest is q for q in _q_after.read.points) + and any(_qv.cheapest.start.astimezone(UTC) == q.start.astimezone(UTC) + for q in _q_win)) + + # --------------------------------------------------------------------------- # print() print("3. AC3 - the cache records the FEED, never its own output") @@ -423,26 +629,114 @@ check(f"...and the fill is idempotent across the two rounds, which is §5.4's " _canon(_filled2) and _canon(_filled2) == _canon(_filled2b)) # The ordering is the mechanism, so the mechanism is checked in the tree. +# +# ⚠️ EVERY SHIPPED PYTHON FILE, AND BY NAME RATHER THAN BY SUBSTRING. The first +# draft scanned `app/*.py` only - not `tools/`, not `scenarios/`, not the +# repo's `estop/` - and matched the bare string, so a module that merely +# MENTIONED `_remember` in a comment failed the check while a real call from +# `tools/` passed it unseen. This parses each file and looks for the name being +# USED: an attribute, a bare name, a def, or a string constant, which is how +# `getattr` reaches a private method. _APP = os.path.join(HERE, "app") -_leaks = [] -for name in sorted(os.listdir(_APP)): - if not name.endswith(".py") or name == "price_history.py": - continue - src = open(os.path.join(_APP, name), encoding="utf-8").read() - if "_remember" in src: - _leaks.append(name) + + +def _uses_remember(path): + """Does this module USE the name `_remember`? Prose does not count.""" + try: + tree = ast.parse(open(path, encoding="utf-8").read()) + except (OSError, SyntaxError): + # ⚠️ NOT False. A file this scan cannot read is a file it has not + # cleared, and "unreadable" must never grade the same as "clean". + return True + for n in ast.walk(tree): + if isinstance(n, ast.Attribute) and n.attr == "_remember": + return True + if isinstance(n, ast.Name) and n.id == "_remember": + return True + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) \ + and n.name == "_remember": + return True + if isinstance(n, ast.Constant) and n.value == "_remember": + return True + return False + + +_SHIPPED = [] +for _root in (_APP, os.path.join(HERE, "tools"), os.path.join(HERE, "scenarios"), + os.path.join(os.path.dirname(HERE), "estop")): + for _dirpath, _dirnames, _filenames in os.walk(_root): + for _f in sorted(_filenames): + if _f.endswith(".py") and _f != "price_history.py": + _SHIPPED.append(os.path.join(_dirpath, _f)) +check(f"the scan has a roster to walk ({len(_SHIPPED)} shipped modules across " + f"app/, tools/, scenarios/ and estop/), and it reaches beyond app/ - an " + f"empty or app-only glob passes for a leak in any other directory", + len(_SHIPPED) > 10 + and any(os.sep + "tools" + os.sep in p for p in _SHIPPED)) +_leaks = [os.path.relpath(p, HERE) for p in _SHIPPED if _uses_remember(p)] check(f"AC3's mechanism is an ORDERING, not a validation - a carried point and " f"a published-provisional point are the same three fields - so nothing " f"outside app/price_history.py may name `_remember` {_leaks or 'none do'}", not _leaks) +_probe = os.path.join(tmpdir(), "probe.py") +open(_probe, "w", encoding="utf-8").write("def f(h):\n return h._remember(1)\n") +_probe_ok = os.path.join(tmpdir(), "probe_ok.py") +open(_probe_ok, "w", encoding="utf-8").write( + '"""A docstring about _remember."""\n# and a comment about _remember\nX = 1\n') +check("...and that scan FAILS on a real call and PASSES on a file that only " + "talks about the rule, so it is a check and not a decoration", + _uses_remember(_probe) and not _uses_remember(_probe_ok)) + +# ⚠️ `ast.walk` IS BREADTH-FIRST, SO A LIST OF CALL NAMES GRADES NESTING DEPTH +# AND NOT SOURCE ORDER. The first draft of this check was +# `_calls.index("_remember") < _calls.index("yesterday")`: `self._remember(...)` +# is a statement while `self.yesterday(...)` is an ARGUMENT of +# `fill_from_yesterday(...)`, one level deeper, so `_remember` was emitted +# first whichever statement came first. Measured by writing the reversal out +# and grading it: shipped and reversed both scored True, while DECISIONS §3 and +# price_history.py's docstring both cited the check as making a reversal "red +# rather than plausible". Line numbers here, and the behaviour itself below. _carry_src = inspect.getsource(PriceHistory.carry) _tree = ast.parse(_carry_src.strip()) -_calls = [n.func.attr for n in ast.walk(_tree) - if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)] -check(f"...and `carry()` records BEFORE it fills, so a feed carrying yesterday " - f"alongside today refreshes the profile before consuming it {_calls}", - _calls.index("_remember") < _calls.index("yesterday")) +_lines = {} +for _n in ast.walk(_tree): + if isinstance(_n, ast.Call) and isinstance(_n.func, ast.Attribute): + _lines[_n.func.attr] = min(_lines.get(_n.func.attr, 10 ** 9), _n.lineno) +check(f"`carry()` names `_remember` on an EARLIER LINE than `yesterday` " + f"{_lines} - source order, which is what 'it records before it fills' " + f"actually means", + "_remember" in _lines and "yesterday" in _lines + and _lines["_remember"] < _lines["yesterday"]) + +# ...and the BEHAVIOUR the ordering exists for, driven rather than read. A feed +# that hands back yesterday alongside today must refresh yesterday's record +# BEFORE it is consumed as the profile. Into an EMPTY store that is the whole +# difference: record-then-fill covers the horizon, fill-then-record covers +# nothing, because at the moment of the fill there is no yesterday yet. +_h_ord = store() # empty: no record from an earlier day +_two_day = PriceRead.series( + [PricePoint(start=brussels(u), + eur_per_kwh=allin(YESTERDAY_MWH[brussels(u).hour]), + provisional=False) for u in YDAY_H] + + [PricePoint(start=brussels(u), + eur_per_kwh=allin(TODAY_MWH[brussels(u).hour]), + provisional=False) for u in TODAY_H], + PriceProduct.DAY_AHEAD, "generic_dayahead", NOW) +_ord_filled = _h_ord.carry(_two_day, since=NOW, until=_until, stamp=NOW) +_ord_n = _ord_filled.match(series=lambda r: len(r.points), no_data=lambda _r: 0) +check(f"...and the ordering is DRIVEN, not only read: a feed carrying " + f"YESTERDAY alongside today, into an empty store, fills the whole " + f"{HORIZON_H} h horizon ({_ord_n} h) because the record was written " + f"before the lookup. Reversed, the lookup finds nothing and the plan is " + f"the {len(_ahead)} published hours the window keeps", + _ord_n == HORIZON_H) +check("...and the hours it carried carry YESTERDAY's numbers - the record that " + "same call had just written - not today's", + _ord_filled.kind == "series" + and all(abs(p.eur_per_kwh + - allin(YESTERDAY_MWH[brussels(p.start).hour])) < 1e-9 + for p in _ord_filled.points if p.provisional)) # --------------------------------------------------------------------------- # @@ -496,7 +790,72 @@ check(f"...and the record is keyed on the BRUSSELS date, so those 24 hours are " # --------------------------------------------------------------------------- # print() -print("5. AC4 - four degraded stores, four reasons, no prices") +print("4b. the two mornings a year when 'yesterday' is a DATE and not 24 hours") + +# ⚠️ `brussels()` RETURNS A FIXED-OFFSET DATETIME, so `brussels(stamp) - +# timedelta(days=1)` subtracts 24 ABSOLUTE HOURS from a wall clock. The +# Brussels day before a spring-forward is 23 hours long, so between 00:00 and +# 00:59 on the morning after it that lands on the day BEFORE yesterday: the +# fill then carries a two-day-old profile behind a flag whose only vocabulary +# is "this is a guess", or - once pruning has dropped that day - answers +# `no_data` naming a date that is not yesterday. Measured at +# 2026-03-30T00:30+02:00 on the shipped branch: it returned 2026-03-28's +# prices. DST is this module's headline claim and it had NO CHECK. +for _label, (_dm2, _dm1, _d0) in ( + ("spring forward - yesterday was 23 h long", + ("2026-03-28", "2026-03-29", "2026-03-30")), + ("fall back - yesterday was 25 h long", + ("2026-10-24", "2026-10-25", "2026-10-26"))): + _lens = [len(bday_hours(_d)) for _d in (_dm2, _dm1, _d0)] + check(f"{_label}: the fixture days are {_lens} hours long, so one of them " + f"really is not 24 and a DATE subtraction and an INSTANT subtraction " + f"can be told apart at all", + _lens[1] != 24 and _lens[0] == 24 and _lens[2] == 24) + _hd = store() + _tag = {} + for _i, _d in enumerate((_dm2, _dm1, _d0)): + # One price band per DAY, so the answer names which day it came from. + _tag[_d] = 0.100000 * (_i + 1) + _ph.PriceHistory._remember( + _hd, + PriceRead.series( + [PricePoint(start=brussels(u), + eur_per_kwh=round(_tag[_d] + 0.000100 + * brussels(u).hour, 6), + provisional=False) + for u in bday_hours(_d)], + PriceProduct.DAY_AHEAD, "generic_dayahead", at_local(_d, 12)), + at_local(_d, 12)) + check(f"{_label}: all three days are held, so the check below is about the " + f"DATE arithmetic and not about a record having been pruned away", + sorted(_hd._days) == sorted((_dm2, _dm1, _d0))) + _stamp_dst = at_local(_d0, 0) + timedelta(minutes=30) + _yd = _hd.yesterday(_stamp_dst) + _band = _yd.match( + series=lambda r: round(min(p.eur_per_kwh for p in r.points), 4), + no_data=lambda _r: None) + check(f"{_label}: at 00:30 local on {_d0}, yesterday() answers {_dm1}'s " + f"prices (band {_band}, wanted {round(_tag[_dm1], 4)}) and not " + f"{_dm2}'s (band {round(_tag[_dm2], 4)}) {why(_yd)!r}", + _yd.kind == "series" and _band == round(_tag[_dm1], 4)) + check(f"{_label}: ...and it holds {_lens[1]} points, the length that " + f"Brussels day actually was - a 24-slot answer would be the wrong " + f"day wearing the right date", + _yd.kind == "series" and len(_yd.points) == _lens[1]) + # The control, one hour later: an implementation that is right only at + # 01:30 is the one being caught, so both are graded. + _yd2 = _hd.yesterday(at_local(_d0, 1) + timedelta(minutes=30)) + check(f"{_label}: ...and at 01:30 local it is still {_dm1} - the defect " + f"shows only in the first local hour, which is why an hourly refresh " + f"could hit it and never be noticed", + _yd2.kind == "series" + and round(min(p.eur_per_kwh for p in _yd2.points), 4) + == round(_tag[_dm1], 4)) + + +# --------------------------------------------------------------------------- # +print() +print("5. AC4 - degraded stores, one reason each, no prices") _missing = PriceHistory(os.path.join(tmpdir(), "gone", "price_history.json")) check("no durable volume: the store is NOT ok, and it does not create the " @@ -505,6 +864,28 @@ check("no durable volume: the store is NOT ok, and it does not create the " not _missing.ok and "no durable volume" in (_missing.last_error or "") and not os.path.isdir(os.path.dirname(_missing.path))) +# ⚠️ AND IT DOES NOT FILL IN MEMORY EITHER. FCST-04 shipped this half-done: +# `_save()` short-circuited on `not ok` while `_remember()` populated +# `self._days` anyway, so a site with no /data answered `words` = "unavailable" +# and `yesterday()` = A SERIES - an in-memory profile that works all day and is +# gone on the next restart. price_history.py's own third warning rejects +# exactly that ("a yesterday that silently forgets") as its reason for not +# calling makedirs, so the code and the docstring have to agree. +_carried_nowhere = _missing.carry(series_of(TODAY_H, TODAY_MWH), + since=NOW, until=_until, stamp=NOW) +check("a store with no durable volume records NOTHING, not even in memory: " + "after a full carry it still holds no day, `yesterday()` still answers " + "no_data, and `words` still says unavailable. A cache that forgets on " + "restart without saying so is worse than one that says it cannot answer", + _missing._days == {} + and _missing.yesterday(NOW + timedelta(days=1)).kind == "no_data" + and _missing.words.startswith("unavailable")) +check("...and the carry still returns a real series - the fill windows the " + "feed and carries nothing, which is the pre-FCST-04 horizon and not a " + "degradation", + _carried_nowhere.kind == "series" + and len(_carried_nowhere.points) == len(_ahead)) + _d6 = tmpdir() _p6 = os.path.join(_d6, "price_history.json") open(_p6, "w", encoding="utf-8").write("{ this is not json") @@ -554,6 +935,127 @@ check("a record naming a product that is not a PriceProduct is no_data, not a " "series with a string in the field the optimiser switches on", PriceHistory(_p9).yesterday(NOW).kind == "no_data") +# ⚠️ AND THE ONE THAT STOPPED THE ADD-ON FROM STARTING. `_load()` validated +# `version` and that `days` is a dict, and never that each VALUE is one. With +# `{"version": 1, "days": {"2026-01-14": "oops"}}` the store reported ok=True +# and last_error=None, and then `rec.get(...)` raised `AttributeError` out of +# `words`, `yesterday()` and `resolve_tariff()` alike. `PriceHistory(...)` and +# `.words` are both unguarded in `Controller.__init__`, so the controller did +# not come up - and it never self-healed, because `_remember()` raised on the +# same record before it could reach `_save()`. The asymmetry that produced it +# is visible in the file: `_parse_points()` is carefully defensive about every +# field of a POINT, and `_load()` was not defensive about the record at all. +_p14 = os.path.join(tmpdir(), "price_history.json") +json.dump({"version": SCHEMA, "days": { + "2026-01-14": "oops", + "2026-01-13": {"product": "dynamic_dayahead", "provider": "generic_dayahead", + "points": [_good]}}}, + open(_p14, "w", encoding="utf-8")) +_h14 = PriceHistory(_p14) +check("a day record that is not a dict is DROPPED and REPORTED, and the rest " + "of the file still loads - one bad record is not a bad file", + _h14.ok and "unusable day record" in (_h14.last_error or "") + and sorted(_h14._days) == ["2026-01-13"]) +_w14 = derived(lambda: _h14.words) +_y14 = derived(lambda: _h14.yesterday(NOW).kind) +_plan14 = derived(lambda: run(resolve_tariff( + CFG, NOW, _until, transport=transport_for(_body_today), + now=NOW, history=_h14)).origin) +check(f"...and NOTHING raises on it: not `words` ({_w14!r}), not `yesterday()` " + f"({_y14!r}), not the shipped `resolve_tariff()` path ({_plan14!r}). A " + f"broken cache is a short horizon, not a controller that will not start", + isinstance(_w14, str) and _y14 == "no_data" and _plan14 == "provider") +for _shape in ("oops", [1, 2, 3], 7, None, {"points": "not a list"}): + _pX = os.path.join(tmpdir(), "price_history.json") + json.dump({"version": SCHEMA, "days": {"2026-01-14": _shape}}, + open(_pX, "w", encoding="utf-8")) + _hX = PriceHistory(_pX) + _wX = derived(lambda h=_hX: h.words) + _yX = derived(lambda h=_hX: h.yesterday(NOW).kind) + check(f"...and the same holds for a day record of {type(_shape).__name__} " + f"{_shape!r}: dropped, reported, and every accessor answers " + f"({_wX!r}, {_yX!r})", + _hX._days == {} and "unusable day record" in (_hX.last_error or "") + and isinstance(_wX, str) and _yX == "no_data") + +_pF = os.path.join(tmpdir(), "price_history.json") +json.dump({"version": float(SCHEMA), "days": {}}, + open(_pF, "w", encoding="utf-8")) +check(f"a `version` of {float(SCHEMA)!r} is refused although it EQUALS " + f"{SCHEMA} - a schema check that cannot tell 1 from 1.0 is one that " + f"would not notice a writer this build has never met", + "schema" in (PriceHistory(_pF).last_error or "")) + + +# --------------------------------------------------------------------------- # +print() +print("5b. `words` - the status row, and one of the two things a bad record " + "used to crash") + +def words_of(h): + return derived(lambda: h.words) + + +_w_missing = words_of(_missing) +check(f"no durable volume: the row says UNAVAILABLE and names the reason " + f"{_w_missing!r}", + isinstance(_w_missing, str) and _w_missing.startswith("unavailable") + and "no durable volume" in _w_missing) +_h_words = store() +check("a healthy but EMPTY store says something different, because it is a " + "different state and a normal one on a site's first day", + isinstance(words_of(_h_words), str) + and words_of(_h_words).startswith("empty") + and "nothing to carry forward" in words_of(_h_words)) +_ph.PriceHistory._remember(_h_words, series_of(YDAY_H, YESTERDAY_MWH), NOW) +check(f"...and a store holding a day names the DATE and the HOUR COUNT " + f"{words_of(_h_words)!r} - 'a cache exists' and 'the cache holds " + f"yesterday, all 24 hours of it' are different facts, and only the " + f"second says the horizon will be full tomorrow morning", + words_of(_h_words) == f"{YDAY_D} (24 h)") +check(f"a file that only PARTLY loaded says so in the row {_w14!r}, or the " + f"days it dropped read as days that were simply never written", + isinstance(_w14, str) and (_h14.last_error or "") in _w14 + and "2026-01-13" in _w14) +check("NoHistory() has a sentence of its own rather than an empty cell", + NoHistory().words == "not kept") +check("...and `words` never raises, whatever the file holds. It is read once " + "from Controller.__init__ and again on every status render, and neither " + "call site guards it - an exception here is an add-on that does not start", + all(isinstance(words_of(h), str) + for h in (_missing, _h6, _h7, _h14, _h_words))) + +# ⚠️ AND A FAILED WRITE LEAVES NOTHING BEHIND. `_save()` writes to a `mkstemp` +# and renames - atomic, so a power cut cannot leave a truncated file that +# parses as an empty day - and a failure BETWEEN the two used to leave the temp +# in place. The refresh task writes 48 times a day, so a persistent write +# failure leaked 48 inodes a day onto the volume whose being full is the +# likeliest reason the write failed at all. +_dleak = tmpdir() +_pleak = os.path.join(_dleak, "price_history.json") +_hleak = PriceHistory(_pleak) +check("the leak fixture starts clean, or 'nothing left behind' is not a " + "measurement", + _hleak.ok and os.listdir(_dleak) == []) +os.mkdir(_pleak) # a DIRECTORY where the file goes: the temp is + # written fine and `os.replace` cannot complete +_ph.PriceHistory._remember(_hleak, series_of(YDAY_H, YESTERDAY_MWH), NOW) +check(f"a write that cannot complete SAYS so rather than passing quietly " + f"{(_hleak.last_error or '')[:48]!r}", + "could not write" in (_hleak.last_error or "")) +_left = sorted(n for n in os.listdir(_dleak) if n != "price_history.json") +check(f"...and leaves NOTHING behind {_left} - at 48 writes a day a leaked " + f"temp per write fills the volume that was already the problem", + _left == []) +_dok = tmpdir() +_hok = PriceHistory(os.path.join(_dok, "price_history.json")) +_ph.PriceHistory._remember(_hok, series_of(YDAY_H, YESTERDAY_MWH), NOW) +check(f"...and a SUCCESSFUL write leaves exactly the one file " + f"{sorted(os.listdir(_dok))}, so the cleanup did not start deleting the " + f"cache it just wrote", + sorted(os.listdir(_dok)) == ["price_history.json"] + and _hok.last_error is None) + # ...and every one of those leaves the CONTROLLER regulating. # # ⚠️ THE PLAN IS NOT BIT-FOR-BIT THE PRE-FCST-04 ONE, and the first draft of @@ -671,14 +1173,88 @@ check(f"six days filed, {KEEP_DAYS} held {_held} - a bigger cache would only " check("...and yesterday is one of them, which is the only day that matters", _h12.yesterday(NOW).kind == "series") -_h13 = store() -_ph.PriceHistory._remember( - _h13, series_of(YDAY_H, YESTERDAY_MWH), NOW) +# ⚠️ THE STIMULUS HAS TO REACH `_prune`'s BODY, AND THE FIRST DRAFT DID NOT. +# It seeded ONE day and then pruned with a jumped stamp: `if len(self._days) <= +# KEEP_DAYS: return` fired, the body never ran, and the check named after that +# body was green for an implementation that pruned by clock arithmetic. +# Measured - the body was replaced with "delete every day older than stamp +# minus KEEP_DAYS days", the guard left in place, and the whole 29-file suite +# stayed green while the mutant emptied the store on a jumped clock. +# +# A store holding more days than the cap is a REAL state, not a contrivance: +# `_load()` does not prune, so this is exactly what a restart sees after a file +# written by a build with a larger cap, or after a feed that spanned several +# days. So the fixture is written, opened, and only then pruned. +_FILED = KEEP_DAYS + 2 +_p13 = os.path.join(tmpdir(), "price_history.json") +_days13 = {} +for _back in range(_FILED - 1, -1, -1): + _d13 = (datetime.fromisoformat(TODAY_D) + - timedelta(days=_back)).strftime("%Y-%m-%d") + _days13[_d13] = { + "product": PriceProduct.DAY_AHEAD.value, + "provider": "generic_dayahead", + "points": [PricePoint(start=brussels(u), + eur_per_kwh=allin(YESTERDAY_MWH[brussels(u).hour]), + provisional=False).canonical() + for u in bday_hours(_d13)]} +json.dump({"version": SCHEMA, "days": _days13}, + open(_p13, "w", encoding="utf-8")) +_h13 = PriceHistory(_p13) +check(f"the prune fixture holds {len(_h13._days)} days, MORE than KEEP_DAYS " + f"({KEEP_DAYS}), so the early return cannot swallow the stimulus - this " + f"file's own rule, and the one its first draft broke", + len(_h13._days) == _FILED > KEEP_DAYS) _ph.PriceHistory._prune(_h13, NOW + timedelta(days=4000)) -check("pruning is by SORT ORDER, not by arithmetic against the clock: a clock " - "that jumped four thousand days does not delete the record, which would " - "empty the store on exactly the morning the fill is for", - _h13.yesterday(NOW).kind == "series") +check(f"pruning does not delete by ARITHMETIC AGAINST THE CLOCK: a clock that " + f"jumped four thousand days keeps the newest {KEEP_DAYS} of the " + f"{_FILED} records {sorted(_h13._days)} instead of deleting all of them, " + f"which would empty the store on exactly the morning the fill is for", + len(_h13._days) == KEEP_DAYS and _h13.yesterday(NOW).kind == "series") + + +# --------------------------------------------------------------------------- # +print() +print("8b. the two days a lookup can ask for are PINNED, so a far-off record " + "cannot evict yesterday") + +# ⚠️ KEEP_DAYS = 3 IS THE STEADY STATE EXACTLY - {yesterday, today, tomorrow} - +# so a newest-N cap has NO SLACK. A feed that publishes to D+2 makes four days +# and the cap drops the oldest, which is yesterday: the only day the fill can +# use. One stray point in the wrong year does the same thing permanently, +# because a lexicographic sort puts it LAST and nothing can ever age it out. +_h15 = store() +_SPAN_D = (YDAY_D, TODAY_D, "2026-01-16", "2026-01-17") +_span = [u for _d in _SPAN_D for u in bday_hours(_d)] +check(f"the spanning fixture really covers {len(_SPAN_D)} Brussels days, more " + f"than KEEP_DAYS ({KEEP_DAYS}), or the cap is not under pressure and " + f"this grades nothing", + len(_SPAN_D) > KEEP_DAYS + and len({brussels(u).strftime("%Y-%m-%d") for u in _span}) == len(_SPAN_D)) +_ph.PriceHistory._remember(_h15, series_of(_span, YESTERDAY_MWH), NOW) +check(f"a feed spanning {len(_SPAN_D)} days does NOT evict yesterday: " + f"{sorted(_h15._days)} held, and {YDAY_D} is one of them. Under the " + f"newest-{KEEP_DAYS} cap alone it is the first record dropped, and the " + f"fill loses its only usable day for good", + YDAY_D in _h15._days and _h15.yesterday(NOW).kind == "series") + +_h16 = store() +for _d16 in (YDAY_D, TODAY_D, "2026-01-16"): + _ph.PriceHistory._remember(_h16, series_of(bday_hours(_d16), TODAY_MWH), NOW) +_STRAY_D = "2099-07-01" +_ph.PriceHistory._remember( + _h16, series_of(bday_hours(_STRAY_D)[:1], TODAY_MWH), NOW) +check(f"the stray fixture is really far future and really sorts last " + f"{sorted(_h16._days)[-1]!r}, or 'it can never age out' is not what is " + f"being graded", + sorted(_h16._days)[-1] == _STRAY_D) +check(f"...and ONE stray point in {_STRAY_D} does not push yesterday out " + f"either. It is not deleted - nothing is deleted by arithmetic here - " + f"it simply cannot take yesterday's slot", + YDAY_D in _h16._days and _h16.yesterday(NOW).kind == "series") +check(f"...while the store stays BOUNDED at KEEP_DAYS + 2 = {KEEP_DAYS + 2} " + f"records ({len(_h16._days)} held), so the pin is a pin and not a leak", + len(_h16._days) <= KEEP_DAYS + 2) # --------------------------------------------------------------------------- # @@ -708,11 +1284,57 @@ _kw = [k.arg for n in ast.walk(_refresh) if isinstance(n, ast.Call) check(f"the shipped refresh task passes a history {_kw} - the wiring is what " f"this ticket IS, so it is checked in the tree rather than described", "history" in _kw) +check("the FCST-04 note in main.py carries the WINDOWING caveat beside the " + "'widens what the optimiser can SEE, never what it may act on' " + "guarantee. That note is the one place a maintainer reads that sentence " + "unqualified, and the CHANGELOG, DOCS and resolve_tariff()'s docstring " + "all carry the pair - so it may not appear here on its own", + "never what it may act on" in _MAIN and "ALREADY ELAPSED" in _MAIN) + check("...and it is the CONTROLLER'S store, not a NoHistory() smuggled in to " "make the call typecheck", "history=self.price_history" in _MAIN and "NoHistory" not in _MAIN) +check("...and `words` reaches the STATUS PAGE as its own row, so what the fill " + "has to carry from is a fact an operator can read rather than one that " + "only exists in a log line", + '("Price history", self.price_history.words)' in _MAIN) +_rows_fn = next(n for n in ast.walk(_main_tree) + if isinstance(n, ast.FunctionDef) + and n.name == "status_payload") +_row_labels = [e.elts[0].value for e in ast.walk(_rows_fn) + if isinstance(e, ast.Tuple) and len(e.elts) == 2 + and isinstance(e.elts[0], ast.Constant) + and isinstance(e.elts[0].value, str)] +check(f"...and it is in the ROW LIST rather than in a comment about one " + f"{[r for r in _row_labels if 'rice' in r or 'ariff' in r]}", + "Price history" in _row_labels) + +# ⚠️ AND WHAT THE STATUS PAGE DOES **NOT** SAY, PINNED AS A PAIR. DOCS.md +# claimed "*Tariff* on the status page reads `provisional` through the +# morning". It does not and never did: `self.tariff_words` is assigned only +# from the Home Assistant price-entity branch and from `static_status()`, both +# on the 30 s tick, and the provider plan's own sentence reaches `_LOG.info` +# and nothing else. This grades the PAIRING rather than the current state - so +# wiring the plan into that row is fine and correcting the doc is fine, and +# doing one without the other is red. +_MAIN_LINES = _MAIN.splitlines() +_tw_lines = sorted({n.lineno for n in ast.walk(_main_tree) + if isinstance(n, ast.Assign) + for t in n.targets + for a in ast.walk(t) + if isinstance(a, ast.Attribute) and a.attr == "tariff_words"}) +_row_from_plan = any("plan" in _MAIN_LINES[i - 1] for i in _tw_lines) +_DOCS = open(os.path.join(HERE, "DOCS.md"), encoding="utf-8").read() +_docs_claims = "**Tariff** on the status page reads `provisional`" in _DOCS +check(f"the *Tariff* row and what DOCS says about it AGREE: fed from the " + f"provider plan = {_row_from_plan} (assigned on lines {_tw_lines}), DOCS " + f"claims it reads `provisional` through the morning = {_docs_claims}. It " + f"shipped as False against True - an overclaim in a doc is a defect " + f"here, and this is the pair that has to move together", + len(_tw_lines) >= 2 and _row_from_plan == _docs_claims) + # NoHistory is inert, and says so by identity rather than by producing a series. _nh_read = series_of(TODAY_H, TODAY_MWH) check("NoHistory() carries nothing and returns the read BY IDENTITY, so the "