diff --git a/goodwe_controller/CHANGELOG.md b/goodwe_controller/CHANGELOG.md index 102dae9..2e29571 100644 --- a/goodwe_controller/CHANGELOG.md +++ b/goodwe_controller/CHANGELOG.md @@ -1,5 +1,75 @@ # Changelog +## Unreleased + +⚠️ **THIS HAS NO VERSION AND CANNOT REACH AN INSTALL UNTIL IT GETS ONE.** +PKG-06's and PKG-07's rule: the Supervisor keys the built image by `version:`, +so `ha apps update` is a silent no-op on an unbumped change. This branch +deliberately does **not** pick the number, because the right one depends on a +fact the branch cannot observe - whether 0.9.0's image has already been built +and pulled. If it has not, this belongs inside 0.9.0's section before it ships; +if it has, it needs 0.10.0. Whoever deploys decides. + +| | | +|---|---| +| _this branch_ | FCST-04 - the carry-forward fill reaches the plan, and the day it carries survives a restart | + +**FCST-03 shipped the fill and no caller.** `price_forecast.fill_from_yesterday()` +was complete and checked, and so were the consumers written for it - +`TariffPlan.firm_points()`, `cheap_now()`'s refusal of a provisional hour, and +`arbitrage()`'s refusal 3, which drops provisional hours from the trade set +rather than refusing the whole series. Nothing in `app/` called the fill, so no +series ever contained a carried point and every one of those consumers was +exercising its all-firm branch. + +⚠️ **THE DEFECT THAT PRODUCED WAS NOT AN ERROR ANYWHERE.** `load_prices()` +returns what the feed published and nothing widened it to the requested horizon, +so on a dynamic site between midnight and the 14-16 h CET day-ahead publication +the plan covered the rest of today and **nothing more** - typically 10-24 of the +48 hours `price_horizon_h` asked for. `arbitrage()` then took `max - min` over +that truncated window and reported a spread. The number was real, the window was +half the one configured, and **nothing said so**: tomorrow evening's peak, the +hour a morning charge would be sold into, was not in the series at all, so the +spread it would have created was never compared against anything. + +**What changes:** + +* `app/price_history.py` (new) keeps the **published** prices of the last + `KEEP_DAYS` Brussels days in `/data/price_history.json`, and + `resolve_tariff()` now fills every unpublished hour of the horizon from + yesterday's same-wall-clock hour, flagged provisional. +* `resolve_tariff(history=...)` is **required and keyword-only**. A caller with + no durable volume passes the explicit `NoHistory()`. A default of `None` + meaning "skip the fill" would hand a caller who forgot the argument exactly + the truncated horizon this card removes, silently. +* A new status row, **Price history**, names the days held and their hour + counts. + +⚠️ **ONE CHANGE HERE MOVES ENERGY, AND IT IS NOT THE ONE THE CARD IS NAMED FOR.** +Carried hours are provisional, so §8.3 keeps them out of the trade set: the fill +widens what the optimiser can **see**, never what it may **do**. What does +change a trade is that the filled series is windowed to `[since, until)`, so +**hours that have already elapsed are no longer in the spread**. Before, a cheap +night that had already gone sat in the series as the day's cheapest hour - and a +cycle cannot buy in an hour that is past, so the trade was quietly never taken. +`test_price_history.py` section 2 grades this by finding the cheapest firm hour +with and without the fill and showing it **moves**. + +⚠️ **A DEAD FEED IS STILL §11.4, NOT YESTERDAY.** The fill runs only on a +`series` read; a feed that published nothing still degrades to the configured +`static_profile`. The two are indistinguishable to the optimiser - both are +100 % provisional, so arbitrage refuses on the firm-pair rung and `cheap_now()` +is False - so preferring yesterday would change a documented degradation for a +difference only the status page can see. +`DECISIONS/FCST-04-the-day-that-is-carried.md` carries that and two more. + +⚠️ **A BROKEN CACHE IS NOT A BROKEN CONTROLLER.** No `/data`, an unreadable +file, a schema this build does not know, a malformed point - four states, four +reasons, none of them a price and none of them an empty series. In every one the +plan is still a real provider series and the controller still regulates; the +horizon is just as short as it was before this card. **No option was added and +`config.yaml` is untouched.** + ## 0.9.0 ⚠️ **MINOR FOR NEW FUNCTIONALITY, NOT FOR A BREAK - AND THAT DISTINCTION IS THE diff --git a/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md b/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md new file mode 100644 index 0000000..e34943d --- /dev/null +++ b/goodwe_controller/DECISIONS/FCST-04-the-day-that-is-carried.md @@ -0,0 +1,98 @@ +# FCST-04 — what "yesterday" is, and what the fill is allowed to reach + +Three decisions, recorded because each had a defensible alternative that a later +reader would otherwise re-open from scratch. + +--- + +## 1. The fill runs on a `series` read only. A dead feed still goes to §11.4. + +`fill_from_yesterday()` accepts a `no_data` read and will happily build a +100 %-carried series from yesterday alone. `resolve_tariff()` does not let it: +a feed that published nothing still falls to `_degrade()` and §11.4's own row — +*"No price data → fall back to the static profile, mark the plan provisional, +no arbitrage."* + +**Why not prefer yesterday's real prices to a coarse day/night profile?** +Because it buys nothing the optimiser can use, and it silently replaces a +fallback the operator configured. + +Measured, not assumed: a fully-carried series is 100 % provisional, so +`TariffPlan.arbitrage()` finds fewer than two firm hours and refuses on §8.3's +firm-pair rung, and `cheap_now()` returns False because `firm_points()` is +empty. The §11.4 static fallback produces *exactly* those two answers. The two +paths differ only in which numbers the status page shows. + +So the choice is between two identical control behaviours, and the tie-breaker +is that `static_profile` is a documented option an operator fills in **for this +case**. Overriding it for a cosmetic difference is a behaviour change nobody +asked for. + +⚠️ **The consequence, stated so it is not discovered later as a bug:** +`fill_from_yesterday()`'s `no_data` branch is not reached from production. +It stays checked in `test_price_forecast.py`. Turning it on is a product +decision — it would want a rule about how old a carried profile may be before it +stops being offered at all — and not a wiring one. + +--- + +## 2. "Yesterday" is a date, not the newest record. + +`newest()` is the obvious accessor and it is the one that decays. An add-on down +from Friday to Monday would answer Friday's prices to a fill whose only +vocabulary is `provisional: true` — a flag that says "this is a guess", never +"this guess is three days old". Nothing downstream can tell the difference and +nothing reports it. + +So `yesterday(stamp)` computes the Brussels date before `stamp`'s Brussels date +and looks up **that** record. Anything else is `no_data` with a reason naming +both the date wanted and the dates held. + +The cost is real and accepted: after an outage of more than a day, the first +morning back has no fill. That is the honest state — there is no yesterday — +and the horizon is short for one morning rather than wrong for several. + +--- + +## 3. What is recorded is the provider's read, and the mechanism is ordering. + +Feeding the filled series back into the store is how this rots: day 2 carries +day 1's guesses, day 3 carries day 2's carry of them, and after a week of a dead +feed the "yesterday" being carried is a fossil of the last real publication. +Every point still carries `provisional=True`, so the **flag never lies** — what +decays is the number, behind a flag that is telling the truth. + +⚠️ **No value-level check can catch that**, and claiming one would be worse than +claiming none. A carried point and a published-provisional point (`eneco_eddy`'s +current quarter, §9.2) are the same three fields carrying the same flag. The +difference is provenance, and provenance is not in the value. + +The defence is therefore an **ordering**, not a validation: + +- `carry()` is the only public entry point. It records its argument, then + returns the fill. +- The filled series exists only as that return value. The object never sees it + again. +- `_remember()` is private and called from exactly one place. +- `test_price_history.py` walks `app/` and fails if any other module names + `_remember`, and reads `carry()`'s AST to confirm the record happens before + the lookup — so a later edit that reverses them is red rather than plausible. + +That ordering has a second benefit worth keeping: a feed that returns yesterday +alongside today refreshes yesterday's record from live data *before* it is +consumed as the profile. + +--- + +## What this ticket does not claim + +It does not make a trade possible that yesterday's prices justify. Carried hours +are dropped from the trade set by §8.3, and that is not a limitation to be fixed +later — it is the rule. + +The one change that **does** move energy came from the windowing, not the +carrying: the filled series covers `[since, until)`, so hours that have already +elapsed are no longer in the spread. A cheap night that has gone used to be the +series' cheapest hour, and a cycle cannot buy in an hour that is past — so the +trade was silently never taken. `test_price_history.py` section 2 grades that by +finding the cheapest firm hour with and without the fill and showing it moves. diff --git a/goodwe_controller/DOCS.md b/goodwe_controller/DOCS.md index a97875a..e657e6a 100644 --- a/goodwe_controller/DOCS.md +++ b/goodwe_controller/DOCS.md @@ -1424,6 +1424,40 @@ distinct state from "prices, all of them zero": the plan is marked provisional and arbitrage is disabled, per the degradation matrix. It never falls back to a price of zero, which would read as free electricity. +#### The hours the feed has not published yet (FCST-04) + +A day-ahead auction publishes tomorrow's prices once a day, at **14–16 h CET**. +Before that, the feed is serving today and nothing else — so a `price_horizon_h` +of 48 asks for 48 hours and gets back the ten or fifteen that are left in the +day. Until FCST-04 those other hours were simply **absent** from the plan: not +priced badly, not flagged, just missing, and nothing said so. + +Now the add-on keeps the **published prices of the last few days** in +`/data/price_history.json` and fills every unpublished hour of the horizon with +yesterday's price for the same Brussels wall-clock hour, flagged +`provisional: true`. Two things follow, and the second one matters more: + +- **The plan says it is guessing.** *Tariff* on the status page reads + `provisional` through the morning, because half its hours are. Before, a + half-length plan looked completely settled. +- **Nothing may trade on a carried hour.** §8.3 forbids acting on a provisional + price, so arbitrage drops those hours from the trade set entirely — it needs a + buy hour and a sell hour that were *both* published. Carrying a price forward + widens what the optimiser can **see**; it never widens what it may **do**. + +The plan is also now cut to the horizon you asked for, which means **hours that +have already elapsed are no longer in it**. That is a real change to arbitrage: +a cheap night that has already gone used to sit in the series as the cheapest +hour of the day, and a cycle can never buy in an hour that is past — so the +trade was quietly never taken. The spread is now taken over hours still ahead. + +A new status row, **Price history**, says which days are held and how many hours +each has. If it reads `empty` or `unavailable`, nothing is carried, the horizon +is as short as the feed makes it, and every other guarantee is unchanged — a +cache that cannot answer is not allowed to be a controller that cannot regulate. +The store is rebuilt by the next successful fetch; on a brand-new install the +first fill is possible the day after the first refresh. + **Injection is priced separately** (`export_price_eur_kwh`). In Belgium it is often near zero, which by itself strongly favours self-consumption over export. It is a single number rather than a curve because no supported supplier diff --git a/goodwe_controller/app/main.py b/goodwe_controller/app/main.py index 2414dce..5cadd9b 100644 --- a/goodwe_controller/app/main.py +++ b/goodwe_controller/app/main.py @@ -68,6 +68,7 @@ from . import peakguard from . import packshare from .p1 import P1Ingest, build_source, is_enabled from .price import Uncertainty, price_option +from .price_history import PriceHistory from .static_tariff import resolve_tariff, static_status from .tariff import PeakStore, PeakTracker from .store import migrate_maint_keys, migrate_unit_keys @@ -165,7 +166,8 @@ def pack_energy_of(inv): class Controller: def __init__(self, opts: dict, hass: HomeAssistant, store, mqtt_pub, *, stop, - peak_db_path: str = "/data/tariff.db"): + peak_db_path: str = "/data/tariff.db", + price_history_path: str = "/data/price_history.json"): self.o = opts self.hass = hass self.store = store @@ -582,6 +584,27 @@ class Controller: self.peak.restore(self.p1.averager) _LOG.info("capacity-tariff peak store: %s", self.peak.restore_state) + # FCST-04. The §8.3 carry-forward store: the published prices of the + # last few Brussels days, so the hours a day-ahead feed has not printed + # yet can be filled from yesterday's profile and FLAGGED, instead of + # being absent from the series entirely. + # + # ⚠️ IT IS AN INSTRUMENT'S BACKING FILE, NOT A STRATEGY. It commands + # nothing and claims nothing; what it changes is how many hours the + # plan covers. Before this, a dynamic site between midnight and the + # 14-16 h CET publication had a series covering only the rest of today + # - typically 10-24 of the 48 hours `price_horizon_h` asked for - and + # `TariffPlan.arbitrage()` took its spread over that truncated window + # without anything reporting that it had. Every carried hour is + # provisional, so refusal 3 drops it from the trade set: this widens + # what the optimiser can SEE, never what it may act on. + # + # ⚠️ A BROKEN OR ABSENT STORE IS NOT AN ERROR HERE. It answers `no_data`, + # the fill carries nothing, and the plan is exactly the pre-FCST-04 + # one - short, but honest and still refusing on the same rules. + self.price_history = PriceHistory(price_history_path) + _LOG.info("§8.3 price history: %s", self.price_history.words) + # SAFETY-07. The guard TARIFF-01 was built for. It is stateless by # design (peakguard.py holds no history between calls), so the only # state here is what an EDGE needs: what the last cycle concluded, so @@ -3455,7 +3478,8 @@ class Controller: now = datetime.now(timezone.utc) try: plan = await resolve_tariff( - self.o, now, now + timedelta(hours=horizon), now=now) + self.o, now, now + timedelta(hours=horizon), now=now, + history=self.price_history) self.tariff_plan, self.tariff_at = plan, now _LOG.info("tariff refreshed: %s", plan.words) except Exception as err: # noqa: BLE001 @@ -4304,6 +4328,13 @@ class Controller: ("Maintenance due", "now" if next_due is None else next_due.strftime("%Y-%m-%d")), ("Cheap window", "yes" if self.cheap else "no"), ("Tariff", self.tariff_words), + # FCST-04. WHAT §8.3 HAS TO CARRY FORWARD, which is the difference + # between a 48-hour plan and a 12-hour one on a weekday morning. + # It is its own row rather than a clause of "Tariff" because it + # answers a different question: "Tariff" says what the plan IS, and + # this says what the plan could still be filled from if the feed is + # late again tomorrow. + ("Price history", self.price_history.words), # OPT-04. The strategy's own sentence, not a re-derivation of it - # every refusal it can make already carries a reason a human can # act on, and "Arbitrage: no" would throw all of them away. diff --git a/goodwe_controller/app/price_history.py b/goodwe_controller/app/price_history.py new file mode 100644 index 0000000..0b2b744 --- /dev/null +++ b/goodwe_controller/app/price_history.py @@ -0,0 +1,433 @@ +"""FCST-04 - the day the carry-forward carries, kept somewhere a restart cannot reach. + +WHAT WAS MISSING, AND WHY IT WAS INVISIBLE. FCST-03 shipped +`price_forecast.fill_from_yesterday()` complete and checked, and shipped the +consumers that were written to expect it - `TariffPlan.firm_points()`, +`cheap_now()`'s refusal of a provisional hour, and `arbitrage()`'s refusal 3, +which drops provisional hours from the trade set instead of refusing the whole +series. Every one of those was live. The FILL ITSELF HAD NO CALLER: nothing in +`app/` referenced `fill_from_yesterday`, so no series ever contained a carried +point and every one of those consumers was exercising its all-firm branch. + +The defect that produced is not an error anywhere. `load_prices()` returns +whatever the feed published and nothing widens it to the requested horizon, so +on a dynamic site between midnight and the 14-16 h CET day-ahead publication +the series covers the rest of TODAY and nothing more - typically 10-24 of the +48 hours `price_horizon_h` asked for. `TariffPlan.arbitrage()` then computes +`max - min` over that truncated window and reports a spread. The number is +real, the window is half the one the operator configured, and NOTHING SAYS SO: +tomorrow evening's peak - the hour a morning charge would be sold into - is +simply not in the series, so the spread it would have created cannot clear a +threshold it was never compared against. A missing hour is not a wrong price; +it is an hour the optimiser never learns exists. + +So this module is the half FCST-03 left out, and it is deliberately the SMALL +half: the fill's arithmetic, its DST rule and its provenance audit all stay in +`price_forecast.py`, untouched. What lives here is the one thing that +arithmetic needs and could not have: A YESTERDAY THAT SURVIVES A RESTART. + + +WHY PERSISTENCE, AND NOT A VARIABLE ON THE REFRESH TASK +====================================================== +An in-memory "the last full day I saw" costs nothing and is wrong exactly when +it matters. The window the fill exists for is midnight -> publication; an +add-on restarted at 08:00 - a config change, an HA update, a power cut - would +come up with no yesterday at all and spend that whole window on the truncated +series, which is the pre-FCST-03 behaviour arriving silently on the mornings +after an operator touched something. The fill's own value is a function of +surviving a restart, so the day it carries has to. + +⚠️ AND NOT IN `/data/state.json`. That file holds the maintenance history and +SAFETY-02's sign latch, and `Store.set()` rewrites the whole of it on every +write. The tariff refresh runs every `TARIFF_REFRESH_S` (1800 s), so folding a +price cache into it would put 48 extra whole-file rewrites a day through the +file whose corruption reads as "no maintenance ever ran" and "never tripped". +The cache is worth having; it is not worth putting anywhere near that. Separate +file, separate blast radius. + +⚠️ AND THE DIRECTORY IS NOT CREATED, which is `tariff.PeakStore`'s rule and its +reason applies unchanged: `/data` is mounted by the Supervisor and always +exists on a real install, so `makedirs` here could only ever succeed where the +durable volume is MISSING - manufacturing a cache on the container's ephemeral +layer that writes fine, reads fine, and is thrown away on the next restart. +That is a yesterday that silently forgets, which is worse than one that says it +cannot answer. + + +THE FOUR RULES THIS FILE IS SHAPED BY +===================================== + +1. IT RECORDS WHAT THE PROVIDER PUBLISHED, NEVER WHAT THE FILL PRODUCED. + Feeding a filled series back in is the defect that makes the whole mechanism + rot: day 2 carries day 1's guesses, day 3 carries day 2's carry of them, and + after a week of a dead feed the "yesterday" being carried forward is a + fossil of the last real publication with nothing anywhere reporting its age. + Every point still carries `provisional=True`, so the flag never lies - what + decays is the NUMBER, silently, behind a flag that is telling the truth. + + ⚠️ NO VALUE-LEVEL CHECK CAN CATCH THAT, and saying so is more useful than a + guarantee that would not hold. A carried point and a published-provisional + point (`eneco_eddy`'s current quarter, §9.2) are the same three fields with + the same flag; the difference is provenance, and provenance is not in the + value. So the mechanism is ORDERING, not validation: `carry()` is the only + public entry point, it remembers the argument and returns the fill, and the + filled series exists only as a return value that `carry()` itself never + sees again. `_remember()` is private, is called from exactly one place, and + `test_price_history.py` walks the tree and fails if any module outside this + file names it. + +2. "YESTERDAY" MEANS YESTERDAY, NOT "THE NEWEST DAY I HAVE". + `newest()` is the natural accessor and it is the one that decays: an add-on + down from Friday to Monday would answer Friday's prices to a fill that flags + them as one day old. The lookup is BY DATE - the Brussels day before the + Brussels day of the instant asked about - and a record for any other date is + NO_RECORD with a reason naming both dates. A three-day-old profile is not a + worse yesterday; it is not a yesterday. + +3. A STORE THAT CANNOT ANSWER SAYS SO, IN THE TYPE. + `yesterday()` returns a `PriceRead`: a `series` when a full record exists, + `no_data(reason)` for every other outcome - no file, no record for that + date, an unreadable file, a record that parses to nothing. There is no + empty-series answer and no `.get(..., [])`-shaped accessor, for + `price.py`'s reason: an empty series is how "the cache is broken" gets to + wear the shape of "yesterday was free". `fill_from_yesterday()` takes a + `PriceRead` and handles `no_data` on its own, so the degraded case needs no + branch at the call site - it fills nothing and the horizon stays short, + which is exactly the behaviour that shipped before this ticket. + +4. HOURS ACCUMULATE WITHIN A DAY; THEY DO NOT REPLACE IT. + A feed that serves only the REMAINING hours of the day (a legitimate shape - + nothing in the plugin contract promises a whole day) would, under + replace-the-record, overwrite a complete 24-hour profile with the two points + left at 22:00 and hand the next morning a yesterday with two hours in it. So + a record is MERGED by instant: a later read updates the hours it carries and + leaves the rest, and a revised price for an hour already held wins because + it is the newer statement about the same hour. + +⚠️ KEYED BY INSTANT, NOT BY LOCAL HOUR, AND THAT IS THE DST DECISION. Storing +`hour -> price` would collapse the two 02:00s of a 25-hour October day into +one, inventing a second and different collapsing rule beside the one +`fill_from_yesterday()` already documents ("the first of its two 02:00 points +is the one used, deterministically"). Storing the instants keeps both, in +order, and lets the fill's own rule be the only one in the system. + +WHAT THIS DOES NOT DO, ON PURPOSE +================================= +It does not extend the fill to a feed that published NOTHING. `resolve_tariff()` +applies `carry()` only on a `series` read; a `no_data` read still falls to +`_degrade()` and §11.4's row - "No price data -> fall back to the static +profile, mark the plan provisional, no arbitrage" - which is the fallback the +operator configured `static_profile` FOR. Preferring yesterday's feed over it +would change a documented degradation without being asked, and it would buy +nothing measurable: a fully-carried series is 100 % provisional, so +`arbitrage()` finds fewer than two firm hours and refuses, and `cheap_now()` is +False, exactly as on the §11.4 fallback. The two differ only in which numbers +the status page shows. `fill_from_yesterday()` handles a `no_data` read anyway, +and that branch is therefore not reached from production - it stays checked, +and turning it on is a product decision, not a wiring one. See +DECISIONS/FCST-04-the-day-that-is-carried.md. +""" + +import json +import logging +import os +import tempfile +from datetime import datetime, timedelta, timezone + +from .price import PricePoint, PriceProduct, PriceRead, brussels +from .price_forecast import fill_from_yesterday + +_LOG = logging.getLogger("goodwe.price.history") + +# How many Brussels days of published prices are kept. The fill needs exactly +# one - yesterday - and rule 2 refuses anything older, so this is not a horizon: +# it is the number of days a record may sit unused before pruning removes it, +# and it is small because a bigger cache would only hold days no lookup can ask +# for. Three covers today, yesterday, and the day a fetch spanning midnight +# writes ahead of itself. +KEEP_DAYS = 3 + +SCHEMA = 1 + + +def _bdate(instant: datetime) -> str: + """The Brussels calendar date of an instant, as `YYYY-MM-DD`. + + ⚠️ BRUSSELS, NOT UTC. 23:30 UTC in July is already tomorrow in Belgium, and + a cache keyed on the UTC date would file the last 90-120 minutes of every + day under the wrong one - so a lookup on the following morning would find a + "yesterday" missing its own evening hours, which are the expensive ones. + """ + return brussels(instant).strftime("%Y-%m-%d") + + +class PriceHistory: + """The published price series of the last few Brussels days, in /data. + + ⚠️ `ok` IS ABOUT THE FILE, NOT ABOUT THE DATA. A healthy store with no + record for yesterday is `ok` and answers `no_data` - a normal state on a + site's first day, and a different thing from a store that could not be + opened. Both are visible: `ok`/`last_error` describe the file, and + `yesterday()`'s reason describes the answer. + """ + + def __init__(self, path: str = "/data/price_history.json"): + self.path = path + self.ok = False + self.last_error: str | None = None + self._days: dict = {} + self._open() + + # -- the file ------------------------------------------------------------ + def _open(self) -> None: + parent = os.path.dirname(self.path) + # See the module note: NOT created. A cache on the ephemeral layer is + # one that forgets without saying so. + if parent and not os.path.isdir(parent): + self.last_error = f"no durable volume at {parent}" + # INFO, not ERROR: on a dev host or in a test container there is no + # /data and nothing is wrong. + _LOG.info("no price history: %s - §8.3 carry-forward has nothing " + "to carry and the horizon stays as short as the feed", + self.last_error) + return + self.ok = True + self._load() + + def _load(self) -> None: + try: + with open(self.path, "r", encoding="utf-8") as fh: + blob = json.load(fh) + except FileNotFoundError: + _LOG.info("no price history yet at %s - the first refresh writes " + "one, and the fill has nothing to carry until the day " + "after that", self.path) + return + except (json.JSONDecodeError, OSError) as err: + # A corrupt cache must not stop the controller, and must not be + # silently re-created either: the record is discarded, the reason + # is kept, and `yesterday()` reports it rather than answering + # "no record", which would read as a normal first day. + self.last_error = f"unreadable ({err})" + _LOG.warning("price history at %s is %s - starting from empty. " + "§8.3 carry-forward has nothing to carry until the " + "next refresh has written a full day", + self.path, self.last_error) + return + if not isinstance(blob, dict) or blob.get("version") != SCHEMA: + found = blob.get("version") if isinstance(blob, dict) else None + self.last_error = f"schema {found!r}, not {SCHEMA}" + _LOG.warning("price history at %s has %s - discarding it rather " + "than guessing what its points mean", + self.path, self.last_error) + return + days = blob.get("days") + if not isinstance(days, dict): + self.last_error = "no `days` object" + _LOG.warning("price history at %s has %s", self.path, self.last_error) + return + self._days = days + + def _save(self) -> None: + if not self.ok: + return + try: + fd, tmp = tempfile.mkstemp(dir=os.path.dirname(self.path)) + with os.fdopen(fd, "w", encoding="utf-8") as fh: + json.dump({"version": SCHEMA, "days": self._days}, fh, indent=1) + # Atomic: a power cut mid-write must not leave a truncated file + # that parses as an empty day. + os.replace(tmp, self.path) + except OSError as err: + self.last_error = f"could not write ({err})" + _LOG.error("could not persist price history: %s", err) + + # -- the one public entry point ----------------------------------------- + def carry(self, read: PriceRead, *, since: datetime, until: datetime, + stamp: datetime) -> PriceRead: + """Remember what this read published, then §8.3-fill the missing hours. + + THE ORDER IS DELIBERATE AND IS THE POINT OF THE METHOD. The read is + recorded FIRST, so a feed that hands back yesterday alongside today + refreshes yesterday's record from live data before it is consumed as + the profile; and the filled series - the one thing that must never be + recorded - exists only as this method's return value, which this object + never sees again. See rule 1. + + Returns whatever `fill_from_yesterday()` returns: a series over + [since, until) whose published hours are the provider's own point + objects and whose remaining hours are carried and flagged, or + `no_data` when neither source can produce a single hour. + """ + self._remember(read, stamp) + return fill_from_yesterday(read, yesterday=self.yesterday(stamp), + since=since, until=until, stamp=stamp) + + # -- the halves, private because rule 1 is an ordering ------------------ + def _remember(self, read: PriceRead, stamp: datetime) -> None: + """File a provider read's points under their own Brussels days. + + ⚠️ PRIVATE, AND CALLED FROM EXACTLY ONE PLACE. A caller that can reach + this can record a filled series, which is the decay in rule 1. There is + no value-level defence against that - a carried point and a published + provisional point are the same three fields - so the defence is that + nothing outside this file names it, and `test_price_history.py` checks + the tree for that rather than trusting the underscore. + """ + points = read.match(series=lambda r: r.points, no_data=lambda _r: ()) + if not points: + return + product = read.match(series=lambda r: r.product.value, + no_data=lambda _r: None) + provider = read.match(series=lambda r: r.provider, + no_data=lambda _r: None) + + touched = set() + for p in points: + day = _bdate(p.start) + rec = self._days.setdefault( + day, {"product": product, "provider": provider, "points": []}) + # Rule 4: MERGE by instant. A later read updates the hours it + # carries and leaves the rest; a revised price for an hour already + # held wins, because it is the newer statement about that hour. + held = {q["start"]: q for q in rec.get("points", []) + if isinstance(q, dict) and "start" in q} + held[p.start.isoformat()] = p.canonical() + rec["points"] = sorted(held.values(), key=lambda q: q["start"]) + rec["product"], rec["provider"] = product, provider + touched.add(day) + + self._prune(stamp) + self._save() + _LOG.debug("price history: %d point(s) filed under %s; %d day(s) held", + len(points), ", ".join(sorted(touched)), len(self._days)) + + def _prune(self, stamp: datetime) -> None: + """Keep the newest KEEP_DAYS records. Older ones can never be a lookup. + + Pruning is by SORT ORDER, not by arithmetic against `stamp`: a clock + that jumped - the failure mode WITNESS-03 spent a session on - would + otherwise delete every record in one pass and leave the fill with + nothing on the morning after, which is the one morning it is for. + """ + if len(self._days) <= KEEP_DAYS: + return + for day in sorted(self._days)[:-KEEP_DAYS]: + del self._days[day] + + def yesterday(self, stamp: datetime) -> PriceRead: + """The published series of the Brussels day BEFORE `stamp`'s, or no_data. + + ⚠️ THE DATE IS COMPUTED, NOT LOOKED UP. See rule 2: `newest()` would + answer Friday's prices to a Monday fill, and the fill would flag them as + one day old because that is the only thing its flag can say. + """ + if stamp.tzinfo is None or stamp.utcoffset() is None: + raise ValueError( + f"yesterday({stamp!r}): a naive datetime is not an instant, " + f"and 'which Brussels day was yesterday' is exactly the " + f"question a wall-clock reading cannot answer.") + want = _bdate(brussels(stamp) - timedelta(days=1)) + if self.last_error and not self._days: + return PriceRead.no_data( + f"the price history at {self.path} is {self.last_error}, so " + f"§8.3 has no profile to carry forward for {want}; the plan " + f"covers only the hours the feed published") + rec = self._days.get(want) + if rec is None: + have = ", ".join(sorted(self._days)) or "no days at all" + return PriceRead.no_data( + f"no published prices recorded for {want} (held: {have}), so " + f"§8.3 has no profile to carry forward. A record for another " + f"day is not a yesterday") + points, bad = _parse_points(rec.get("points")) + if not points: + why = bad or "it is empty" + return PriceRead.no_data( + f"the {want} record holds no usable point ({why}), and an " + f"empty series is not a free day") + try: + product = PriceProduct(rec.get("product")) + except ValueError: + return PriceRead.no_data( + f"the {want} record names product {rec.get('product')!r}, " + f"which is not a PriceProduct. The optimiser switches on that " + f"enum, so a record that cannot name one cannot be a series") + if bad: + _LOG.warning("price history: %s in the %s record were dropped; " + "%d point(s) carried forward", bad, want, len(points)) + return PriceRead.series(points, product, + str(rec.get("provider") or "price_history"), + stamp) + + # -- observability ------------------------------------------------------- + @property + def words(self) -> str: + """One line for the status page. Says what is held, or why nothing is.""" + if not self.ok: + return f"unavailable - {self.last_error}" + if not self._days: + why = f" - {self.last_error}" if self.last_error else "" + return f"empty{why} - nothing to carry forward yet" + return "; ".join(f"{d} ({len(self._days[d].get('points') or ())} h)" + for d in sorted(self._days)) + + +def _parse_points(raw) -> tuple: + """Canonical dicts -> PricePoints, and a phrase naming what was dropped. + + ⚠️ A MALFORMED POINT IS DROPPED, NOT DEFAULTED, and the drop is reported. + `float(row.get("eur_per_kwh", 0))` is the shape of the whole family of + defects this repo has been removing: a cache that lost a field would answer + "that hour was free" to a fill that flags it as an ordinary guess. + """ + if not isinstance(raw, list): + return (), "the `points` field is not a list" + out, dropped = [], 0 + for row in raw: + if not isinstance(row, dict): + dropped += 1 + continue + try: + start = datetime.fromisoformat(row["start"]) + value = float(row["eur_per_kwh"]) + flag = row["provisional"] + except (KeyError, TypeError, ValueError): + dropped += 1 + continue + if start.tzinfo is None or start.utcoffset() is None \ + or not isinstance(flag, bool): + dropped += 1 + continue + out.append(PricePoint(start=start, eur_per_kwh=value, provisional=flag)) + out.sort(key=lambda p: p.start.astimezone(timezone.utc)) + return tuple(out), (f"{dropped} malformed point(s)" if dropped else "") + + +class NoHistory: + """The explicit "this caller keeps no history" - a NAME, not a `None`. + + ⚠️ IT EXISTS SO THE PARAMETER CAN BE REQUIRED. `history=None` meaning "skip + the fill" is the shape this repo keeps removing: a caller who forgets the + argument gets the truncated-horizon behaviour of the bug this ticket is + about, silently, and the code reads as though the fill is happening. With + no default there is nothing to forget, and a caller with no durable volume + - a check, a replay harness - has to say `NoHistory()` and be visible in + the diff. + + It carries out the ordering faithfully: it records nothing and carries + nothing, so `carry()` returns the provider's read unchanged and the plan is + exactly what it was before FCST-04. + """ + + ok = False + last_error = "no history is kept by this caller" + words = "not kept" + + def carry(self, read: PriceRead, *, since, until, stamp) -> PriceRead: + return read + + def yesterday(self, stamp: datetime) -> PriceRead: + return PriceRead.no_data( + "this caller keeps no price history (NoHistory), so §8.3 has no " + "profile to carry forward") diff --git a/goodwe_controller/app/static_tariff.py b/goodwe_controller/app/static_tariff.py index 102da58..dfd5ddb 100644 --- a/goodwe_controller/app/static_tariff.py +++ b/goodwe_controller/app/static_tariff.py @@ -1019,7 +1019,7 @@ def static_status(opts: dict, now: datetime): async def resolve_tariff(opts: dict, since: datetime, until: datetime, - transport=None, now=None) -> TariffPlan: + transport=None, now=None, *, history) -> TariffPlan: """The full §9.2 + §11.4 resolution. MAY PERFORM I/O - see `static_plan`. ⚠️ NOT ON THE 30 s TICK, and that is a deliberate boundary rather than an @@ -1033,9 +1033,34 @@ async def resolve_tariff(opts: dict, since: datetime, until: datetime, price_provider off + profile -> the CONFIGURED product (§9.2 rows 3-4) price_provider off + none -> no prices at all (§11.4, nothing to fall back to) - provider series -> the feed (§9.2 rows 1-2) + provider series -> the feed, §8.3-FILLED (§9.2 rows 1-2) provider no_data + profile -> §11.4 FALLBACK, provisional, no arbitrage provider no_data + none -> no prices at all + + ⚠️ `history` IS REQUIRED, KEYWORD-ONLY, AND HAS NO DEFAULT (FCST-04). It is + the §8.3 carry-forward store - `price_history.PriceHistory`, or the explicit + `price_history.NoHistory()` for a caller that keeps none. A default of + `None` meaning "skip the fill" would give a caller who forgot the argument + the TRUNCATED HORIZON this ticket exists to remove, silently, while the code + read as though the fill were happening. There is nothing to forget now. + + ⚠️ AND THE FILL RUNS ONLY ON A `series` READ. A feed that published nothing + still falls to `_degrade()` and §11.4's own row - the static profile the + operator configured FOR that case - rather than to yesterday's feed. The two + are indistinguishable to the optimiser (both are 100 % provisional, so + `arbitrage()` finds fewer than two firm hours and refuses, and `cheap_now()` + is False), so preferring yesterday would change a documented degradation for + a difference only the status page can see. See price_history.py's closing + note and DECISIONS/FCST-04-the-day-that-is-carried.md. + + ⚠️ THE FILLED SERIES IS WINDOWED TO [since, until), AND THAT IS A CHANGE. + `load_prices()` returns whatever the feed sent, past hours included, and + `TariffPlan.arbitrage()` took its spread over all of it - so an hour that + had already gone could be the cheapest hour of a cycle nothing can now buy + in. `fill_from_yesterday()` walks the requested horizon and emits exactly + that, so the spread is over hours that are still ahead. Both directions of + the difference are real: the window gains the unpublished hours it was + missing and loses the elapsed ones it should never have had. """ stamp = now or datetime.now(timezone.utc) pid = str(price_option(opts, "price_provider") or "off").strip() @@ -1044,6 +1069,20 @@ async def resolve_tariff(opts: dict, since: datetime, until: datetime, read = await load_prices(opts, since, until, transport) return read.match( + series=lambda r: _provider_plan(opts, r, since, until, stamp, history), + no_data=lambda r: _degrade(opts, r, since, until, stamp)) + + +def _provider_plan(opts, read, since, until, stamp, history) -> TariffPlan: + """§8.3: record what the feed published, carry yesterday into the gaps. + + The fill can itself return `no_data` - a feed whose points all fall outside + the horizon, with nothing in the history to fill it from - and that is the + §11.4 row arrived at one step later, so it degrades exactly as a failed + fetch does rather than being special-cased here. + """ + filled = history.carry(read, since=since, until=until, stamp=stamp) + return filled.match( series=lambda r: TariffPlan(r, _quiet_profile(opts), "provider"), no_data=lambda r: _degrade(opts, r, since, until, stamp)) diff --git a/goodwe_controller/test_price_history.py b/goodwe_controller/test_price_history.py new file mode 100644 index 0000000..e5bc175 --- /dev/null +++ b/goodwe_controller/test_price_history.py @@ -0,0 +1,734 @@ +"""Runnable check for FCST-04 - the day the fill carries. `python3 test_price_history.py` + +WHAT THIS FILE IS DEFENDING. FCST-03 shipped `fill_from_yesterday()` and the +consumers written for it, and shipped NO CALLER. This file grades the caller: +that a partially-published day-ahead feed now produces a plan covering the whole +requested horizon, that the day it carries survives a restart, and that the +cache cannot decay by being fed its own output. + + AC1 THE HOLE IS CLOSED. `resolve_tariff()` on a feed that has published 10 + of 48 hours returns a series covering 48, the 38 unpublished hours are + provisional and the 10 published ones are the provider's own objects, + unchanged and unflagged. Graded against the same call with + `NoHistory()`, which is the pre-FCST-04 behaviour, so the difference is + MEASURED rather than described. + AC2 THE WINDOW STOPPED INCLUDING THE PAST, which is the half of this change + that moves energy. `load_prices()` returns whatever the feed sent, so at + 09:00 a spread was taken over hours that had already elapsed and the + "cheapest hour" could be one no cycle can buy in - the trade is then + silently never taken. The filled series is windowed to [since, until), + so the firm cheapest hour is a hour still ahead. Graded by finding the + cheapest firm hour both ways and showing it MOVES. + AC3 THE CACHE CANNOT DECAY. What is recorded is the provider's read and + never the filled series; if it were, day 2 would carry day 1's guesses + and day 3 would carry day 2's carry of them, behind a `provisional` flag + that stays true while the NUMBER rots. Graded by driving `carry()` twice + and reading the file back. + AC4 A STORE THAT CANNOT ANSWER SAYS SO. No file, no record for yesterday, an + unreadable file, a record whose points are malformed - four different + reasons, none of them a price, none of them an empty series, and the + controller keeps its pre-FCST-04 plan through all four. + +THE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will: + + * EVERY VERDICT IS DERIVED. The carried/published split is re-derived here + from the set of hours the fixture published, by different arithmetic than + app/ uses. The horizon growth is a subtraction of two measured counts, not + a constant. The cheapest-hour move is found by looking, not asserted. + + * NO STIMULUS IS EVALUATED WHERE THE QUANTITY IT GRADES IS DEGENERATE: + - YESTERDAY AND TODAY MUST PRICE EVERY LOCAL HOUR DIFFERENTLY, or a fill + that carried the wrong day is invisible. Asserted first, as a check. + - THE PUBLISHED WINDOW MUST BE A STRICT SUBSET OF THE HORIZON, or "the + fill widened it" and "the fill did nothing" are the same observation. + - THE ELAPSED HOURS MUST CONTAIN THE CHEAPEST HOUR OF THE WHOLE FEED, or + AC2 grades nothing: the windowed and unwindowed cheapest hour would be + the same point either way. This is also the realistic shape of the + danger - a cheap night that has already gone, holding a battery out of + a trade all morning. + - THE MERGE FIXTURE MUST REVISE A PRICE TO A DIFFERENT NUMBER, or merge + and replace are indistinguishable. + - THE PRUNE FIXTURE MUST HOLD MORE DAYS THAN KEEP_DAYS, or pruning is not + exercised at all. + + * A FAILED READ IS NOT AN OBSERVATION. Every degraded store is graded on the + TYPE it answers with (`no_data` and its reason), never on a falsy value, + and the malformed-point case is checked for the specific defect - a price + defaulted to 0.0 - rather than for a count. + +⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run +`tools/mutation_run.sh`, which does. See MUTATION.md and TEST-15. +""" + +import ast +import asyncio +import inspect +import json +import os +import shutil +import sys +import tempfile +from datetime import datetime, timedelta, timezone + +from app import price_history as _ph +from app.price import ( + PricePoint, PriceProduct, PriceRead, brussels, +) +from app.price_history import KEEP_DAYS, NoHistory, PriceHistory, SCHEMA +from app.static_tariff import resolve_tariff + +UTC = timezone.utc +HERE = os.path.dirname(os.path.abspath(__file__)) +fails = [] +_tmpdirs = [] + + +def check(name, cond): + print(f" {'ok ' if cond else 'FAIL'} {name}") + if not cond: + fails.append(name) + + +def why(read): + """A read's reason, or "" when it is a series. + + ⚠️ NOT `read.reason`. On a `_Series` that attribute RAISES `Absence` by + design, so a mutation that makes a refusing accessor answer a series would + take this file down with a traceback instead of reporting a FAIL - and + MUTATION.md's rule is that "it was refused" and "it broke on the way to + being refused" must not grade the same. + """ + return read.match(series=lambda _r: "", no_data=lambda r: r.reason) + + +def raises(fn, exc): + try: + fn() + except exc: + return True + except Exception: + return False + return False + + +def run(coro): + return asyncio.run(coro) + + +def tmpdir(): + d = tempfile.mkdtemp(prefix="fcst04-") + _tmpdirs.append(d) + return d + + +def store(): + """A history on a fresh directory that looks durable.""" + return PriceHistory(os.path.join(tmpdir(), "price_history.json")) + + +# --------------------------------------------------------------------------- # +# The fixtures. +# +# ⚠️ A DAY IS A BRUSSELS DAY, BUILT BY SEARCH. The first draft of this file made +# a day 24 UTC hours long and every count in it came out one short: in January +# Belgium is UTC+1, so 23:00 UTC already belongs to tomorrow, and a "24-hour +# day" built by adding hours to midnight UTC is 23 hours of one Brussels day +# plus one hour of the next. That is the same class of error the module is +# keyed to avoid, so the fixture must not contain it - and it is found by +# LOOKING at each instant's Brussels date rather than by adding an offset a +# helper would have to be right about. +# +# ⚠️ THE INSTANT IS 09:00 BRUSSELS ON A WEEKDAY, WHICH IS THE WHOLE POINT. The +# day-ahead auction publishes at 14-16 h CET, so at this instant a healthy feed +# is serving TODAY IN FULL and nothing of tomorrow - the window this ticket +# exists for. A fixture at 20:00 would have both days and would grade nothing. +# --------------------------------------------------------------------------- # + +TODAY_D = "2026-01-15" # a Thursday; Belgium on CET +YDAY_D = "2026-01-14" +HORIZON_H = 48 + + +def bday_hours(bdate: str) -> list: + """The UTC instants whose BRUSSELS date is `bdate`, one per local hour.""" + probe = datetime.fromisoformat(bdate + "T00:00:00+00:00") - timedelta(hours=3) + out = [] + for n in range(30): + u = probe + timedelta(hours=n) + if brussels(u).strftime("%Y-%m-%d") == bdate: + out.append(u) + return out + + +TODAY_H = bday_hours(TODAY_D) +YDAY_H = bday_hours(YDAY_D) + + +def at_local(bdate: str, hh: int) -> datetime: + """The UTC instant whose Brussels wall clock reads hh:00 on `bdate`.""" + return next(u for u in bday_hours(bdate) if brussels(u).hour == hh) + + +NOW = at_local(TODAY_D, 9) + +# Today's feed, in EUR/MWh, keyed by BRUSSELS local hour and all distinct. +# ⚠️ THE CHEAPEST HOUR IS 03:00 LOCAL - ALREADY ELAPSED AT 09:00 - and that is +# AC2's stimulus, asserted below rather than assumed. +TODAY_MWH = {h: 120.0 + h for h in range(24)} +TODAY_MWH[3] = 5.0 # the cheap night that has already gone +TODAY_MWH[19] = 260.0 # the dear evening still ahead + +# Yesterday's published prices, DIFFERENT AT EVERY LOCAL HOUR from today's, so +# a fill that carried the wrong day is visible. +YESTERDAY_MWH = {h: 400.0 + 3 * h for h in range(24)} + +MARKUP, TAXES = 0.03, 0.06 + + +def allin(mwh): + """EUR/MWh as published -> the all-in EUR/kWh `assemble()` produces.""" + return round(mwh * 0.001 + MARKUP + TAXES, 6) + + +def feed_body(instants, table): + return json.dumps({"data": [ + {"start": u.isoformat(), "price_eur_mwh": table[brussels(u).hour]} + for u in instants]}).encode() + + +def transport_for(body): + async def _t(url, timeout_s): + return body + return _t + + +CFG = {"price_provider": "generic_dayahead", + "price_feed_url": "https://api.energy-charts.info/price?bzn=BE", + "price_timeout_s": 1, + "price_markup_eur_kwh": MARKUP, + "price_taxes_eur_kwh": TAXES, + "static_profile": ""} + + +def series_of(instants, table, product=PriceProduct.DAY_AHEAD, + provider="generic_dayahead", stamp=None): + """A PriceRead.series built directly - the shape `load_prices()` returns.""" + return PriceRead.series( + [PricePoint(start=brussels(u), + eur_per_kwh=allin(table[brussels(u).hour]), + provisional=False) + for u in instants], + product, provider, stamp or NOW) + + +# --------------------------------------------------------------------------- # +print("0. the arithmetic this file grades with must not be degenerate") + +check(f"a Brussels day is 24 hours long in the fixture helper, on both fixture " + f"days ({len(TODAY_H)}, {len(YDAY_H)}) - a 23-hour 'day' built by adding " + f"to midnight UTC is what made every count below come out one short", + len(TODAY_H) == 24 and len(YDAY_H) == 24 + and {brussels(u).hour for u in TODAY_H} == set(range(24))) + +# The feed serves TODAY IN FULL - which is what a day-ahead feed does - so the +# hours it publishes that are still AHEAD are a strict subset of the horizon. +_ahead = [u for u in TODAY_H if u >= NOW] +check(f"at 09:00 the hours the feed published that have NOT elapsed are a " + f"STRICT SUBSET of the {HORIZON_H} h horizon ({len(_ahead)} of " + f"{HORIZON_H}), so 'the fill widened the plan' and 'the fill did " + f"nothing' are different observations", + 0 < len(_ahead) < HORIZON_H) + +check("yesterday prices EVERY local hour differently from today, so a fill " + "that carried the wrong day is visible in the numbers", + all(TODAY_MWH[h] != YESTERDAY_MWH[h] for h in range(24))) + +check("...and no fixture price is zero, so 'dropped the price' is visible too", + all(TODAY_MWH.values()) and all(YESTERDAY_MWH.values())) + +_cheapest_all = min(TODAY_H, key=lambda u: TODAY_MWH[brussels(u).hour]) +_cheapest_ahead = min(_ahead, key=lambda u: TODAY_MWH[brussels(u).hour]) +check(f"AC2's stimulus is real: the cheapest hour of the whole published day is " + f"{brussels(_cheapest_all).hour:02d}:00 local, which has already elapsed " + f"at 09:00, and the cheapest hour still AHEAD is a different one " + f"({brussels(_cheapest_ahead).hour:02d}:00). Without this the windowed " + f"and unwindowed answers would be the same point and AC2 would grade " + f"nothing", + _cheapest_all != _cheapest_ahead and _cheapest_all < NOW + and _cheapest_ahead >= NOW) + +check(f"KEEP_DAYS is {KEEP_DAYS}, and the prune fixture below files more days " + f"than that, so pruning is exercised rather than skipped", KEEP_DAYS < 6) + + +# --------------------------------------------------------------------------- # +print() +print("1. AC1 - the hole: a partially-published feed now covers the horizon") + +# ⚠️ THE FEED PUBLISHES TODAY IN FULL, WHICH IS WHAT A DAY-AHEAD FEED DOES. +# The partial publication this ticket is about is that TOMORROW is missing, not +# that this morning is - and the elapsed hours the feed is still serving are +# exactly what AC2 is about, so they have to be in the body. +_body_today = feed_body(TODAY_H, TODAY_MWH) +_until = NOW + timedelta(hours=HORIZON_H) + +# The pre-FCST-04 behaviour, by the shipped code path: NoHistory() records +# nothing and carries nothing, so this IS the old plan. +_before = run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), now=NOW, + history=NoHistory())) + +# And the same call with a history that already holds yesterday. +_h1 = store() +_h1_seed = series_of(YDAY_H, YESTERDAY_MWH) +_ph.PriceHistory._remember(_h1, _h1_seed, NOW) # seeding, not a caller +_after = run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), now=NOW, + history=_h1)) + +_n_before = len(_before.read.points) +_n_after = len(_after.read.points) +check(f"the plan's horizon GREW, measured: {_n_before} h before FCST-04, " + f"{_n_after} h after, against a {HORIZON_H} h request", + _n_after > _n_before and _n_after == HORIZON_H) + +check(f"...and before the change it was exactly what the feed sent " + f"({len(TODAY_H)} h of one day), which is the defect from both ends: the " + f"{HORIZON_H - len(_ahead)} hours past the feed's own horizon were not " + f"short-priced but ABSENT, and the {len(TODAY_H) - len(_ahead)} hours " + f"that had already elapsed were in the series anyway", + _n_before == len(TODAY_H)) + +# The split, re-derived here from the fixture rather than read off the module. +_published_instants = {u.astimezone(UTC) for u in _ahead} +_got_carried = {p.start.astimezone(UTC) for p in _after.read.points + if p.provisional} +_got_firm = {p.start.astimezone(UTC) for p in _after.read.points + if not p.provisional} +check("AC1: exactly the hours the feed published are FIRM, and exactly the " + "hours it did not are PROVISIONAL - the flag is a claim about " + "provenance, re-derived here from the fixture", + _got_firm == _published_instants + and _got_carried == {p.start.astimezone(UTC) + for p in _after.read.points} - _published_instants + and len(_got_carried) == HORIZON_H - len(_ahead)) + +_before_by_start = {p.start.astimezone(UTC): p for p in _before.read.points} +check("...and every published point is the PROVIDER'S OWN OBJECT, passed " + "through by identity - a copy is where the flag gets lost (FCST-03)", + all(p.eur_per_kwh == _before_by_start[p.start.astimezone(UTC)].eur_per_kwh + for p in _after.read.points if not p.provisional)) + +# The carried numbers are yesterday's, at the same BRUSSELS wall-clock hour. +# ⚠️ THE LOOKUP IS BUILT BY SEARCH over yesterday's own instants, not by +# subtracting 24 h from today's - a fill that subtracted a fixed day is exactly +# what this is meant to catch, and a helper that subtracts one cannot catch it. +_Y_BY_LOCAL = {} +for _u in YDAY_H: + _Y_BY_LOCAL.setdefault(brussels(_u).hour, + allin(YESTERDAY_MWH[brussels(_u).hour])) +_carried_ok = [_Y_BY_LOCAL.get(brussels(p.start).hour) is not None + and abs(p.eur_per_kwh + - _Y_BY_LOCAL[brussels(p.start).hour]) < 1e-9 + for p in _after.read.points if p.provisional] +check(f"...and every carried number is YESTERDAY's price for the same BRUSSELS " + f"wall-clock hour, not today's and not a neighbour's " + f"({sum(_carried_ok)}/{len(_carried_ok)})", + _carried_ok and all(_carried_ok)) + +check("the plan reports itself PROVISIONAL now, which it did not before - the " + "old plan claimed full firmness over a horizon it covered half of", + _after.provisional and not _before.provisional) + +check("...and the plan is still a provider plan, not a §11.4 fallback: the " + "fill does not degrade anything", + _after.origin == "provider" and _after.product is PriceProduct.DAY_AHEAD) + + +# --------------------------------------------------------------------------- # +print() +print("2. AC2 - the spread stopped being taken over hours that have gone") + +_A = dict(peak_guard_active=False, min_spread_eur_kwh=0.0, + uncertainty=frozenset()) +_v_before = _before.arbitrage(**_A) +_v_after = _after.arbitrage(**_A) +check("both plans still ALLOW a trade, so this section compares two verdicts " + "rather than a verdict and a refusal", + _v_before.kind == "allowed" and _v_after.kind == "allowed") + +_cheap_before = _v_before.cheapest.start.astimezone(UTC) +_cheap_after = _v_after.cheapest.start.astimezone(UTC) +check(f"AC2: the cheapest FIRM hour MOVED from {_cheap_before.isoformat()} - " + f"already elapsed at {NOW.isoformat()}, so no cycle could ever buy in it " + f"- to {_cheap_after.isoformat()}, which is still ahead", + _cheap_before < NOW <= _cheap_after and _cheap_before != _cheap_after) + +check("...and the new one is the cheapest hour the feed published that has NOT " + "gone, derived from the fixture rather than from the module", + _cheap_after == _cheapest_ahead.astimezone(UTC)) + +check("...and no point of the filled series starts before the requested " + "`since` hour: the window is [since, until) and the past is not in it", + min(p.start.astimezone(UTC) for p in _after.read.points) + >= NOW.replace(minute=0, second=0, microsecond=0)) + +# ⚠️ THE HONEST LIMIT OF THIS TICKET, checked so it cannot be overclaimed +# later: the CARRIED hours buy no trade. §8.3 forbids acting on a provisional +# price and refusal 3 drops them from the trade set, so what widened is what +# the optimiser can SEE, and the trade above came from the WINDOWING, not from +# yesterday's numbers. +_firm_after = [p for p in _after.read.points if not p.provisional] +check("the carried hours are NOT in the trade set - the verdict's cheapest and " + "dearest are both firm, so this ticket widened what the optimiser sees " + "and not what it may act on", + not _v_after.cheapest.provisional and not _v_after.dearest.provisional + and len(_firm_after) == len(_ahead)) + + +# --------------------------------------------------------------------------- # +print() +print("3. AC3 - the cache records the FEED, never its own output") + +_h2 = store() +_read2 = series_of(TODAY_H, TODAY_MWH) +_filled2 = _h2.carry(_read2, since=NOW, until=_until, stamp=NOW) +# Round two, as the refresh task does every 1800 s: the same partial feed. +_filled2b = _h2.carry(_read2, since=NOW, until=_until, stamp=NOW) + +_disk = json.load(open(_h2.path, encoding="utf-8")) +_today_key = brussels(NOW).strftime("%Y-%m-%d") +_recorded = {p["start"] for p in _disk["days"][_today_key]["points"]} +_recorded_utc = {datetime.fromisoformat(s).astimezone(UTC) for s in _recorded} +check(f"AC3: after two `carry()` rounds the record for {_today_key} holds " + f"EXACTLY the {len(TODAY_H)} hours the feed published " + f"({len(_recorded_utc)} held) - not one of the " + f"{HORIZON_H - len(_ahead)} hours the fill invented", + _recorded_utc == {u.astimezone(UTC) for u in TODAY_H}) + +check("...and no recorded point is flagged provisional, because a carried " + "point never reached the file - the flag would still read true while the " + "NUMBER decayed, which is the failure this rule is about", + not any(p["provisional"] for p in _disk["days"][_today_key]["points"])) + +def _canon(read): + return [p.canonical() for p in read.match(series=lambda r: r.points, + no_data=lambda _r: ())] + + +check(f"...and the fill is idempotent across the two rounds, which is §5.4's " + f"late-publication contract - the refresh task runs it again every " + f"1800 s against whatever the feed says by then " + f"({len(_canon(_filled2))} h both times)", + _canon(_filled2) and _canon(_filled2) == _canon(_filled2b)) + +# The ordering is the mechanism, so the mechanism is checked in the tree. +_APP = os.path.join(HERE, "app") +_leaks = [] +for name in sorted(os.listdir(_APP)): + if not name.endswith(".py") or name == "price_history.py": + continue + src = open(os.path.join(_APP, name), encoding="utf-8").read() + if "_remember" in src: + _leaks.append(name) +check(f"AC3's mechanism is an ORDERING, not a validation - a carried point and " + f"a published-provisional point are the same three fields - so nothing " + f"outside app/price_history.py may name `_remember` {_leaks or 'none do'}", + not _leaks) + +_carry_src = inspect.getsource(PriceHistory.carry) +_tree = ast.parse(_carry_src.strip()) +_calls = [n.func.attr for n in ast.walk(_tree) + if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)] +check(f"...and `carry()` records BEFORE it fills, so a feed carrying yesterday " + f"alongside today refreshes the profile before consuming it {_calls}", + _calls.index("_remember") < _calls.index("yesterday")) + + +# --------------------------------------------------------------------------- # +print() +print("4. rule 2 - 'yesterday' is a DATE, not 'the newest day I have'") + +_h3 = store() +_ph.PriceHistory._remember( + _h3, series_of(bday_hours("2026-01-12"), YESTERDAY_MWH), NOW) +_y3 = _h3.yesterday(NOW) +check("a store holding only the day before LAST answers no_data, not those " + "prices - `newest()` is the accessor that decays, and an add-on down " + "over a weekend is when it would", + _y3.kind == "no_data") +check(f"...and the reason names BOTH the date wanted and the dates held, so an " + f"operator can see which it is: {why(_y3)!r}", + "2026-01-14" in why(_y3) and "2026-01-12" in why(_y3)) + +_h4 = store() +_ph.PriceHistory._remember( + _h4, series_of(YDAY_H, YESTERDAY_MWH), NOW) +_y4 = _h4.yesterday(NOW) +check("...while the day BEFORE this one answers a series, so the refusal above " + "is about the date and not about the store being broken", + _y4.kind == "series" and len(_y4.points) == 24) + +check("a naive instant is refused rather than assumed to be UTC - 'which " + "Brussels day was yesterday' is the question a wall clock cannot answer", + raises(lambda: _h4.yesterday(datetime(2026, 1, 15, 9)), ValueError)) + +# The Brussels-date rule, on the fixture a UTC-date rule gets wrong. A feed +# serving the 24 UTC hours of a July day spans TWO Brussels days - Belgium is +# on CEST, so 22:00 and 23:00 UTC are already 00:00 and 01:00 of tomorrow - and +# a UTC-keyed cache would file all 24 under one date. The evening hours are the +# expensive ones, so filing them under the wrong day is the costly direction. +_h5 = store() +_jul_utc = [datetime(2026, 7, 14, h, tzinfo=UTC) for h in range(24)] +_split = {brussels(u).strftime("%Y-%m-%d") for u in _jul_utc} +check(f"the July fixture really does straddle two Brussels days {sorted(_split)}" + f", or the next check passes for a UTC-keyed cache as well", + len(_split) == 2) +_ph.PriceHistory._remember( + _h5, series_of(_jul_utc, YESTERDAY_MWH), _jul_utc[-1]) +_disk5 = json.load(open(_h5.path, encoding="utf-8")) +_counts = {d: len(_disk5["days"][d]["points"]) for d in sorted(_disk5["days"])} +check(f"...and the record is keyed on the BRUSSELS date, so those 24 hours are " + f"filed under both of them {_counts} rather than all under one", + set(_counts) == _split and sum(_counts.values()) == 24 + and all(_counts.values())) + + +# --------------------------------------------------------------------------- # +print() +print("5. AC4 - four degraded stores, four reasons, no prices") + +_missing = PriceHistory(os.path.join(tmpdir(), "gone", "price_history.json")) +check("no durable volume: the store is NOT ok, and it does not create the " + "directory - a cache on the ephemeral layer reads fine and is thrown " + "away on the next restart (tariff.PeakStore's rule)", + not _missing.ok and "no durable volume" in (_missing.last_error or "") + and not os.path.isdir(os.path.dirname(_missing.path))) + +_d6 = tmpdir() +_p6 = os.path.join(_d6, "price_history.json") +open(_p6, "w", encoding="utf-8").write("{ this is not json") +_h6 = PriceHistory(_p6) +check("an unreadable file: the reason is kept and reported, and the record is " + "NOT silently re-created as an empty one that reads like a first day", + "unreadable" in (_h6.last_error or "") + and _h6.yesterday(NOW).kind == "no_data" + and "unreadable" in why(_h6.yesterday(NOW))) + +_p7 = os.path.join(tmpdir(), "price_history.json") +json.dump({"version": SCHEMA + 99, "days": {}}, open(_p7, "w", encoding="utf-8")) +_h7 = PriceHistory(_p7) +check("a file from a schema this build does not know is discarded rather than " + "guessed at - its points might mean something else", + "schema" in (_h7.last_error or "") + and _h7.yesterday(NOW).kind == "no_data") + +# A record whose points are malformed. THE DEFECT IS A DEFAULTED PRICE. +_p8 = os.path.join(tmpdir(), "price_history.json") +_good = {"start": brussels(at_local(YDAY_D, 5)).isoformat(), + "eur_per_kwh": allin(YESTERDAY_MWH[5]), "provisional": False} +json.dump({"version": SCHEMA, "days": {"2026-01-14": { + "product": "dynamic_dayahead", "provider": "generic_dayahead", + "points": [_good, + {"start": "2026-01-14T06:00:00+01:00"}, # no price + {"start": "2026-01-14 07:00:00", "eur_per_kwh": 0.2, + "provisional": False}, # naive + {"eur_per_kwh": 0.3, "provisional": False}, # no instant + "not even a dict"]}}}, + open(_p8, "w", encoding="utf-8")) +_y8 = PriceHistory(_p8).yesterday(NOW) +check("a malformed point is DROPPED, not defaulted: the four broken rows are " + "gone and the one good row survives", + _y8.kind == "series" and len(_y8.points) == 1) +check("...and specifically no point came back priced 0.0 - " + "`float(row.get('eur_per_kwh', 0))` is how a lost field becomes 'that " + "hour was free'", + all(p.eur_per_kwh != 0.0 for p in _y8.points) + and abs(_y8.points[0].eur_per_kwh - allin(YESTERDAY_MWH[5])) < 1e-9) + +_p9 = os.path.join(tmpdir(), "price_history.json") +json.dump({"version": SCHEMA, "days": {"2026-01-14": { + "product": "no_such_product", "provider": "x", + "points": [_good]}}}, open(_p9, "w", encoding="utf-8")) +check("a record naming a product that is not a PriceProduct is no_data, not a " + "series with a string in the field the optimiser switches on", + PriceHistory(_p9).yesterday(NOW).kind == "no_data") + +# ...and every one of those leaves the CONTROLLER regulating. +# +# ⚠️ THE PLAN IS NOT BIT-FOR-BIT THE PRE-FCST-04 ONE, and the first draft of +# this check claimed it was. A broken cache still goes through the fill, which +# WINDOWS to [since, until) whether or not it has anything to carry - so the +# plan is the feed's own hours MINUS the elapsed ones. That is the AC2 half of +# this change, and it is correct with no cache at all: an hour that has gone is +# not a cheap hour. What must survive a broken cache is that the plan is a real +# provider series with no invented points and no §11.4 degradation. +for _label, _hist in (("no durable volume", _missing), + ("unreadable file", _h6), + ("wrong schema", _h7)): + _plan = run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), now=NOW, + history=_hist)) + check(f"...and with {_label} the plan is a real provider series over the " + f"{len(_ahead)} published hours still ahead, with NOTHING carried " + f"and no §11.4 degradation. A broken cache is not a broken " + f"controller - it is a short horizon, which is where this started", + _plan.origin == "provider" + and len(_plan.read.points) == len(_ahead) + and not any(p.provisional for p in _plan.read.points)) + + +# --------------------------------------------------------------------------- # +print() +print("6. rule 4 - hours accumulate within a day; they do not replace it") + +_h10 = store() +_ph.PriceHistory._remember(_h10, series_of(TODAY_H, TODAY_MWH), NOW) +# A feed that serves only the hours left in the day - a legitimate shape, and +# the one that turns replace-the-record into a two-hour yesterday. +_REVISED = dict(TODAY_MWH) +_REVISED[22], _REVISED[23] = 999.0, 998.0 +check("the revision really changes the numbers, or merge and replace are the " + "same observation", + _REVISED[22] != TODAY_MWH[22] and _REVISED[23] != TODAY_MWH[23]) +_ph.PriceHistory._remember( + _h10, series_of([at_local(TODAY_D, 22), at_local(TODAY_D, 23)], _REVISED), + NOW) + +_back = _h10.yesterday(NOW + timedelta(days=1)) +check(f"a two-hour read did not overwrite a 24-hour day: " + f"{len(_back.points) if _back.kind == 'series' else _back.kind} hours " + f"held. Under replace, the next morning's 'yesterday' would have been " + f"two hours long", + _back.kind == "series" and len(_back.points) == 24) + +# ⚠️ KEYED BY BRUSSELS HOUR, because that is what the fixture tables are keyed +# by. Keying this by UTC hour reads the wrong row of the table in winter and +# grades the module against a price nobody put there. +_by_hour = {brussels(p.start).hour: p.eur_per_kwh for p in _back.points} + + +def _priced(h, want): + """True only if hour `h` is present AND carries `want`. A MISSING hour is + False, never a KeyError: under `replace` the hours simply are not there, and + a harness that dies on the mutation it is grading reports nothing at all.""" + return h in _by_hour and abs(_by_hour[h] - want) < 1e-9 + + +check("...and the two revised hours carry the NEW price, because a later read " + "is the newer statement about the same hour", + _priced(22, allin(999.0)) and _priced(23, allin(998.0))) +check("...while every untouched hour kept its original price", + all(_priced(h, allin(TODAY_MWH[h])) for h in range(22))) + + +# --------------------------------------------------------------------------- # +print() +print("7. the restart this whole module exists for") + +_d11 = tmpdir() +_p11 = os.path.join(_d11, "price_history.json") +_first = PriceHistory(_p11) +_ph.PriceHistory._remember( + _first, series_of(YDAY_H, YESTERDAY_MWH), NOW) +del _first +_second = PriceHistory(_p11) # a brand-new add-on process +_y11 = _second.yesterday(NOW) +check("yesterday survives a restart, which is the ONLY reason this is a file " + "and not a variable on the refresh task: the window the fill exists for " + "is midnight -> publication, and an add-on restarted at 08:00 would " + "otherwise spend all of it on the truncated series", + _y11.kind == "series" and len(_y11.points) == 24) +check("...with the prices intact through the JSON round trip, to six decimals", + all(abs(p.eur_per_kwh + - allin(YESTERDAY_MWH[brussels(p.start).hour])) < 1e-9 + for p in _y11.points)) + +_plan11 = run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), now=NOW, + history=_second)) +check("...and the restarted controller's very first refresh already covers the " + "whole horizon", + len(_plan11.read.points) == HORIZON_H) + + +# --------------------------------------------------------------------------- # +print() +print("8. pruning keeps yesterday, and a jumped clock does not empty the store") + +_h12 = store() +for back in range(6, 0, -1): + _ph.PriceHistory._remember( + _h12, + series_of(bday_hours( + (datetime.fromisoformat(TODAY_D) - timedelta(days=back) + ).strftime("%Y-%m-%d")), YESTERDAY_MWH), + NOW) +_held = sorted(json.load(open(_h12.path, encoding="utf-8"))["days"]) +check(f"six days filed, {KEEP_DAYS} held {_held} - a bigger cache would only " + f"hold days rule 2 refuses to look at", + len(_held) == KEEP_DAYS) +check("...and yesterday is one of them, which is the only day that matters", + _h12.yesterday(NOW).kind == "series") + +_h13 = store() +_ph.PriceHistory._remember( + _h13, series_of(YDAY_H, YESTERDAY_MWH), NOW) +_ph.PriceHistory._prune(_h13, NOW + timedelta(days=4000)) +check("pruning is by SORT ORDER, not by arithmetic against the clock: a clock " + "that jumped four thousand days does not delete the record, which would " + "empty the store on exactly the morning the fill is for", + _h13.yesterday(NOW).kind == "series") + + +# --------------------------------------------------------------------------- # +print() +print("9. the parameter cannot be forgotten, and main.py does not forget it") + +_sig = inspect.signature(resolve_tariff) +_hp = _sig.parameters.get("history") +check("`resolve_tariff(history=...)` is KEYWORD-ONLY and has NO DEFAULT - a " + "default of None meaning 'skip the fill' would hand a caller who forgot " + "it the truncated horizon this ticket removes, silently", + _hp is not None and _hp.kind is inspect.Parameter.KEYWORD_ONLY + and _hp.default is inspect.Parameter.empty) + +check("...so calling it without one is a TypeError, not a quiet regression", + raises(lambda: run(resolve_tariff(CFG, NOW, _until, + transport=transport_for(_body_today), + now=NOW)), TypeError)) + +_MAIN = open(os.path.join(_APP, "main.py"), encoding="utf-8").read() +_main_tree = ast.parse(_MAIN) +_refresh = next(n for n in ast.walk(_main_tree) + if isinstance(n, ast.AsyncFunctionDef) + and n.name == "run_tariff_refresh") +_kw = [k.arg for n in ast.walk(_refresh) if isinstance(n, ast.Call) + for k in n.keywords] +check(f"the shipped refresh task passes a history {_kw} - the wiring is what " + f"this ticket IS, so it is checked in the tree rather than described", + "history" in _kw) +check("...and it is the CONTROLLER'S store, not a NoHistory() smuggled in to " + "make the call typecheck", + "history=self.price_history" in _MAIN + and "NoHistory" not in _MAIN) + +# NoHistory is inert, and says so by identity rather than by producing a series. +_nh_read = series_of(TODAY_H, TODAY_MWH) +check("NoHistory() carries nothing and returns the read BY IDENTITY, so the " + "explicit 'I keep no history' is exactly the old behaviour and not a " + "second implementation of it", + NoHistory().carry(_nh_read, since=NOW, until=_until, stamp=NOW) + is _nh_read + and NoHistory().yesterday(NOW).kind == "no_data") + + +# --------------------------------------------------------------------------- # +for d in _tmpdirs: + shutil.rmtree(d, ignore_errors=True) + +print() +if fails: + print(f"{len(fails)} FAILED: " + "; ".join(fails)) + sys.exit(1) +print("all checks passed") diff --git a/goodwe_controller/test_static_tariff.py b/goodwe_controller/test_static_tariff.py index 84f8fa9..6e42681 100644 --- a/goodwe_controller/test_static_tariff.py +++ b/goodwe_controller/test_static_tariff.py @@ -69,6 +69,7 @@ from app.price import ( Absence, PricePoint, PriceProduct, PriceRead, arbitrage_allowed, brussels, plan_is_provisional, required_spread, uncertainty_of, ) +from app.price_history import NoHistory from app.static_tariff import ( MEAN_WINDOW_DAYS, ProfileInvalid, StaticProfile, TariffPlan, parse_static_profile, resolve_tariff, static_plan, static_status, @@ -746,7 +747,8 @@ _cfg = opts(TWO_PERIOD, price_provider="generic_dayahead", price_feed_url="https://api.energy-charts.info/price?bzn=BE", price_timeout_s=1, price_markup_eur_kwh=0.03, price_taxes_eur_kwh=0.06) -_fb = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1), transport=_dead)) +_fb = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1), transport=_dead, + history=NoHistory())) check("a dead feed falls back to the static profile rather than to nothing", _fb.origin == "fallback" and _fb.has_prices) check("...with the SAME prices the configured-product path produces, because " @@ -769,7 +771,8 @@ check("...and the provisional plan has no cheap window either: §8.3 forbids an not any(_fb.cheap_now(at_local(THU, h)) for h in range(24))) _fb_none = run(resolve_tariff(opts(None, price_provider="generic_dayahead", price_timeout_s=1), - THU, THU + timedelta(days=1), transport=_dead)) + THU, THU + timedelta(days=1), transport=_dead, + history=NoHistory())) check("a dead feed with NO static_profile has nothing to fall back to, and " "says that rather than inventing a tariff", _fb_none.origin == "none" and not _fb_none.has_prices) @@ -785,7 +788,7 @@ async def _live(url, timeout_s): _live_plan = run(resolve_tariff(_cfg, THU, THU + timedelta(days=1), - transport=_live)) + transport=_live, history=NoHistory())) check("a live feed is used in preference to the static profile", _live_plan.origin == "provider" and _live_plan.product is PriceProduct.DAY_AHEAD)