diff --git a/goodwe_controller/CHANGELOG.md b/goodwe_controller/CHANGELOG.md index 8023f3e..6ee3d27 100644 --- a/goodwe_controller/CHANGELOG.md +++ b/goodwe_controller/CHANGELOG.md @@ -13,6 +13,89 @@ if it has, it needs 0.10.0. Whoever deploys decides. | | | |---|---| | _this branch_ | FCST-04 - the carry-forward fill reaches the plan, and the day it carries survives a restart | +| _this branch_ | FCST-05 - one byte stops stopping the add-on, and the write path gets the read path's standard | + +### FCST-05 - what FCST-04's own contract still did not hold + +**A single non-UTF-8 byte in `/data/price_history.json` stopped the add-on from +starting.** `_load()` caught `(json.JSONDecodeError, OSError)`. +`UnicodeDecodeError` is `ValueError -> UnicodeError -> UnicodeDecodeError` - +neither of those, and `issubclass(UnicodeDecodeError, OSError)` is **False** - +so it escaped `_load()` -> `_open()` -> `PriceHistory.__init__`, which +`Controller.__init__` does not guard. Reproduced: `UnicodeDecodeError: 'utf-8' +codec can't decode byte 0xff`, and **the controller did not come up**. That is the same class as the day-record defect the store was reworked +for, in the one byte class the rework left uncovered: valid-UTF-8-invalid-JSON, +a UTF-8 BOM and a non-dict day record all degraded correctly. The suite could +not see it because its "unreadable file" fixture writes `"{ this is not json"`, +which is **valid UTF-8**. The handler now names the `ValueError` family - the +file's CONTENT - beside `OSError`, the file's BYTES, and a fixture holding a +genuinely undecodable byte is in `test_price_history.py` section 5. + +**A malformed `start` under the day the feed is writing to killed the tariff +plan permanently.** `_remember()` merged a day's points with +`{q["start"]: q ...}` and sorted on `q["start"]`, trusting a value read off +`/data` to be a hashable, sortable string. An `int`, `null`, a `dict` or a +`list` there raised `TypeError`; `run_tariff_refresh` catches `Exception`, so +the add-on stayed **up** while **every refresh raised, forever** - the plan +retiring after `TARIFF_MAX_AGE_S` and arbitrage and the cheap window off for +good, with no self-heal because `_remember()` raised before reaching `_save()`. +The read path (`_parse_points()`) was properly defensive throughout, which is +why `yesterday()` survived and the existing fixture passed: it files its +malformed rows under a day the feed is **not** writing to, so `_remember()` +never opened that record. **The two paths now share one validator**, +`_point_of()`, and neither has a standard of its own. A row that is not a point +is dropped and reported, and `_save()` rewrites the record clean. + +⚠️ **AND THE STATUS ROW SAID THE STORE WAS HEALTHY WHILE THE PLAN WAS DEAD.** +`words` read `'2026-01-15 (1 h)'` about the record that was killing every +refresh. `_load()` now counts the rows inside a kept record as well as the +record itself, so the store stops reporting plain health from the moment it is +opened; the note names what was **found** at open and what was **dropped** by +the refresh, and clears itself once the file has been rewritten. + +⚠️ **AND THE ROW COUNTED POINTS AND CALLED THEM HOURS.** `f"{d} ({_n_points(d)} +h)"` on the quarter-hourly feed this add-on already supports reads +`'2026-01-13 (96 h)'` for a 24-hour day - the same points/hours conflation the +sub-hourly defect above was about, on the row that defect's own card shipped. +The cell is now `2026-01-13 (96 points over 24 h)`: two facts, each under its +own unit, and the hour figure is the one that says whether tomorrow morning's +fill has a whole day to carry. + +⚠️ **AND THE MERGE KEY IS THE INSTANT RATHER THAN ITS SPELLING**, which is what +rule 4 already said. Aware datetimes hash and compare by instant. The string +key merged by spelling, and the lexicographic sort beside it put the second +02:00 of the 25-hour October day (`+01:00`) **before** the first (`+02:00`); +only the stored order was affected, because `_parse_points()` re-sorts by +instant on the way out. + +⚠️ **TWO STATED REASONS WERE FALSE, AND THIS REPOSITORY TREATS THAT AS A +DEFECT.** `price_forecast._audit()`'s "WHY IT IS STRICTER" section claimed the +rework refused two things FCST-03's audit did not. A differential over the +whole domain the audit can express - every published mask over 3 hours crossed +with every output series of length 0-3 - found **zero divergence in both +directions**: the reworked audit is exactly **EQUAL** to FCST-03's, not +stronger. The old `seen` count already refused the same published point emitted +twice (one point twice is one hour twice), and the old **identity** rule +already refused a carried point sharing a published hour. The only behavioural +change is the intended **relaxation** - several distinct published points in +one hour - and that stays. The docstring and the two check texts now say what +is true, and the differential is committed as a check in +`test_price_forecast.py` rather than left as prose. + +⚠️ **AND `_audit` IS NOT WHAT STOPS THE DECIMATION DEFECT RECURRING.** It does +not check **completeness**: fed the decimation directly - four published points +in an hour, output emitting only the last - it **accepts**, and it accepts an +empty output. That is a limit, not a regression, and the module docstring said +otherwise by presenting the audit as the post-condition guarding provenance. +What actually guards it is the point **count** and the pass-through **identity** +checks in `test_price_forecast.py` and `test_price_history.py`, and the text now +points at them. It is **not** made complete: `published` is the whole read while +the fill emits only `[since, until)`, so "every published point appears" is not +a true post-condition of those two arguments. Measured - the rule as stated +takes the 1 h, 12 h and 48 h horizon checks red on legitimate series. + +**No option was added, `config.yaml` is untouched, and nothing was deployed.** + **FCST-03 shipped the fill and no caller.** `price_forecast.fill_from_yesterday()` was complete and checked, and so were the consumers written for it - @@ -58,8 +141,9 @@ spread it would have created was never compared against anything. no durable volume passes the explicit `NoHistory()`. A default of `None` meaning "skip the fill" would hand a caller who forgot the argument exactly the truncated horizon this card removes, silently. -* A new status row, **Price history**, names the days held and their hour - counts. +* A new status row, **Price history**, names the days held, how many points + each holds and how many whole hours they cover (see FCST-05: it shipped + counting points and printing them under the hour's name). ⚠️ **ONE CHANGE HERE MOVES ENERGY, AND IT IS NOT THE ONE THE CARD IS NAMED FOR.** Carried hours are provisional, so §8.3 keeps them out of the trade set: the fill diff --git a/goodwe_controller/DOCS.md b/goodwe_controller/DOCS.md index af8efbd..fe54bce 100644 --- a/goodwe_controller/DOCS.md +++ b/goodwe_controller/DOCS.md @@ -1474,8 +1474,13 @@ a cheap night that has already gone used to sit in the series as the cheapest hour of the day, and a cycle can never buy in an hour that is past — so the trade was quietly never taken. The spread is now taken over hours still ahead. -A new status row, **Price history**, says which days are held and how many hours -each has. If it reads `empty` or `unavailable`, nothing is carried, the horizon +A new status row, **Price history**, says which days are held, how many price +points each holds and how many whole hours those cover — `2026-01-13 (96 points +over 24 h)`. ⚠️ **Those are two different numbers and the row used to print one +of them under the other's name**: it counted points and labelled them `h`, so a +quarter-hourly feed read `96 h` for a 24-hour day. The figure that says whether +tomorrow morning's fill has a whole day to carry is the **hour** one. If it +reads `empty` or `unavailable`, nothing is carried, the horizon is as short as the feed makes it, and every other guarantee is unchanged — a cache that cannot answer is not allowed to be a controller that cannot regulate. The store is rebuilt by the next successful fetch; on a brand-new install the diff --git a/goodwe_controller/MUTATION.md b/goodwe_controller/MUTATION.md index 3e2af21..03dbf2d 100644 --- a/goodwe_controller/MUTATION.md +++ b/goodwe_controller/MUTATION.md @@ -127,6 +127,21 @@ Two rules come out of it, and both are cheap: function, that pair is the first thing to try — TEST-16's two survivors were adjacent lines in one function and neither was individually live. +⚠️ **AND FCST-05 WALKED INTO IT AGAIN, IN ITS OWN NEW CHECK, WHICH IS WHY THE +RULE IS “mutate every check you add”.** `price_history.py` reports a malformed +point row from **two** places into **one** field: `_usable_days()` records what +it FOUND when the file was opened, and `_remember()` records what it DROPPED on +the next refresh. Both land in `last_error`, and `words` prints `last_error`. +The first draft of the new check asked `"malformed point row" in words` — which +whichever rung is still standing satisfies — so deleting `_remember()`'s note +entirely was a **−1 B SURVIVOR of the whole suite**, on the half of the fix that +tells an operator the cache **self-healed**. It is graded on the TENSE now +(`… in ` at open, `… dropped from ` after the refresh), the two are +asserted to DIFFER, and the paired mutation — both notes deleted, −91 B — is +red on 15 named checks. Same shape as TEST-16 and the same fix: **grade the +reason, not the verdict**, and when two rungs answer one stimulus, mutate them +together. + ## The third way a check lies: its DATA cannot reach the failure it names Stale bytecode manufactures survivors from a mutation that never ran. Masking diff --git a/goodwe_controller/app/price_forecast.py b/goodwe_controller/app/price_forecast.py index 2d8d7d2..8e1dd44 100644 --- a/goodwe_controller/app/price_forecast.py +++ b/goodwe_controller/app/price_forecast.py @@ -55,6 +55,17 @@ So the flag is not passed through anywhere in this file: It is a post-condition, not a filter: it never fixes a flag, because a module that quietly corrects its own provenance is one whose provenance cannot be trusted. + + ⚠️ IT GRADES THE POINTS THAT ARE EMITTED AND NOTHING ABOUT THE ONES THAT + ARE NOT (FCST-05). `_audit()` is a provenance check, NOT a completeness + check: fed FCST-04's own decimation defect - four published points in an + hour, only the last emitted - it ACCEPTS, and it accepts an empty output + too. Measured, and the FCST-03 audit it replaced was equally blind, so + this is a limit rather than a regression. What actually guards against a + decimated series is the point COUNT and the pass-through IDENTITY checks + in `test_price_forecast.py` (section on the quarter-hourly feed) and + `test_price_history.py`; see `_audit()`'s own docstring for why the + completeness rule is not expressible from its two arguments. * and test_price_forecast.py walks THIS FILE'S AST and fails if any `PricePoint(...)` in it passes anything but a literal for `provisional`, so the pass-through idiom cannot come back in a later edit and pass review by @@ -225,14 +236,67 @@ def _audit(points, published) -> None: * CARRIED - the hour must not appear in `published` AT ALL, the flag must be exactly True, and that hour may hold no second point. - ⚠️ WHAT REPLACED "AT MOST ONE POINT PER HOUR", AND WHY IT IS STRICTER. This - audit began as a `seen` set: any repeated `_utc_hour` raised, because a - duplicated hour is how a DST day gets mispriced. A bare hour count cannot - express a quarter-hourly feed, where four distinct published points share - one hour legitimately - so the count was replaced by the two rules above, - which subsume it. A repeated CARRIED hour still raises; so does the same - published point emitted twice; and a carried point may no longer share an - hour with a published one, which the count never checked at all. + ⚠️ WHAT REPLACED "AT MOST ONE POINT PER HOUR", AND WHAT THAT DID AND DID + NOT CHANGE. This audit began as FCST-03's `seen` set: any repeated + `_utc_hour` raised, beside an identity rule `p is not src`. A bare hour + count cannot express a quarter-hourly feed, where four distinct published + points share one hour legitimately - so the count was replaced by the two + rules above, which subsume it. + + ⚠️ AND IT IS NOT STRICTER. FCST-04's version of this paragraph claimed two + strengthenings and BOTH WERE FALSE; FCST-05 measured it. A differential + over the whole domain this audit can express - every published mask over 3 + hours crossed with every output series of length 0-3 drawn from + {published, published-provisional, carried, carried-unflagged, + copy-of-published} - found ZERO divergence in EITHER direction: + + OLD raised, NEW accepted (WEAKER): 0 + OLD accepted, NEW raised (STRONGER): 0 + + The two specific claims, one at a time: + + * "so does the same published point emitted twice" - the OLD `seen` + count already raised on that, because emitting one point twice is one + HOUR twice. Preserved by the identity/`emitted` rules, not added. + * "a carried point may no longer share an hour with a published one, + which the count never checked at all" - true of the COUNT, and the old + IDENTITY rule raised on it anyway: a carried point in a published hour + took the `else` branch and failed `p is not src`. The message was about + a copy rather than about a carry, which is the only thing that moved. + + The one real behavioural change is the intended RELAXATION - several + distinct published points in one hour, which the old audit could not + express at all and refused. That relaxation is why this was rewritten, and + it stays. A right decision with a fabricated reason is still a defect here. + + ⚠️ AND THE REAL LIMIT, WHICH IS NOT A REGRESSION AND IS NOT SMALL: THIS + AUDIT DOES NOT CHECK COMPLETENESS. It grades every point that IS emitted + and says nothing about the ones that are not. Fed the H1 decimation + directly - four published points in an hour, output emitting only the last + - it ACCEPTS; so does two of four, and so does an EMPTY output. So `_audit` + is NOT what stops H1 recurring. What stops it is the point COUNT and the + IDENTITY checks in `test_price_history.py` and `test_price_forecast.py`, + which compare the emitted series against the feed's own prints. + + It is not made complete here, and the reason is structural rather than + effort: `published` is `_published_by_hour(read)` over the WHOLE read, + unwindowed, while `fill_from_yesterday()` emits only [since, until). A + published point outside the window is legitimately absent, so "every + published point appears" is not a true post-condition of these two + arguments - it would need the window passed in, which is a different + signature and a different check. + + ⚠️ AND THAT IS MEASURED, NOT ASSUMED. FCST-05 built the rule - raise unless + every point of every `published[u]` is emitted, +162 B - and ran it. It + takes the 1 h, 12 h and 48 h HORIZON checks in test_price_forecast.py RED, + on the shipped fill and a healthy feed: at an off-the-hour instant the + feed's already-elapsed prints are outside [since, until) and are correctly + dropped, and the rule calls that a provenance failure. (The differential + beside those checks scores the same rule as 49 STRONGER cases, so it is a + real change of behaviour and not a no-op.) The rule as stated is therefore + not merely unimplemented, it is WRONG - and stating a false post-condition + is MUTATION.md habit 16's defect: an instrument that asserts a completeness + it does not have is worse than no instrument. ⚠️ RAISES, NEVER REPAIRS. A module that silently corrects its own provenance is one whose provenance means nothing; the only safe response to diff --git a/goodwe_controller/app/price_history.py b/goodwe_controller/app/price_history.py index 464c938..40efc91 100644 --- a/goodwe_controller/app/price_history.py +++ b/goodwe_controller/app/price_history.py @@ -208,8 +208,65 @@ def _yesterday_of(instant: datetime) -> str: return (brussels(instant).date() - timedelta(days=1)).strftime("%Y-%m-%d") -def _n_points(rec) -> int: - """How many points a day record holds. 0 for anything that is not one. +def _point_of(row): + """One stored row -> a `PricePoint`, or `None` if the row is not one. + + THE ONE VALIDATOR, AND THE ASYMMETRY IT CLOSES (FCST-05's MEDIUM). This + file used to hold TWO standards for what a stored point is. The READ path + (`_parse_points()`) was properly defensive - every field checked, a + malformed row dropped and reported. The WRITE path was + `{q["start"]: q for q in raw if isinstance(q, dict) and "start" in q}` + followed by `sorted(..., key=lambda q: q["start"])`, which trusts + `q["start"]` to be a hashable, sortable string it never checked. Measured, + with the malformed row under the day the feed is currently writing to: + + start is an int -> TypeError: '<' not supported between instances + of 'str' and 'int' + start is a dict -> TypeError: cannot use 'dict' as a dict key + start is null -> TypeError: '<' not supported between 'str' and + 'NoneType' + start is a list -> TypeError: cannot use 'list' as a dict key + + ⚠️ AND THE BLAST RADIUS IS PERMANENT, WHICH IS WHY IT IS NOT MERELY UNTIDY. + `_remember()` is reached through `carry()` -> `resolve_tariff()` -> + `run_tariff_refresh`, which catches `Exception` - so the add-on stays UP + and EVERY refresh raises forever. The plan retires after + `TARIFF_MAX_AGE_S`, and arbitrage and the cheap window are off from then + on. There is no self-heal, because `_remember()` raises before it reaches + `_save()`. `_usable_days()` only required `rec["points"]` to be a LIST; the + elements were unchecked on the write path, and the existing fixture put its + malformed row under a day the feed was NOT writing to, so `_remember()` + never touched it. + + So both paths call this, and neither of them has a standard of its own. + Returning `None` rather than raising is the contract: a caller decides + whether a bad row is dropped (both of them do) and how it is reported. + """ + if not isinstance(row, dict): + return None + try: + start = datetime.fromisoformat(row["start"]) + value = float(row["eur_per_kwh"]) + flag = row["provisional"] + except (KeyError, TypeError, ValueError): + return None + if start.tzinfo is None or start.utcoffset() is None \ + or not isinstance(flag, bool): + return None + return PricePoint(start=start, eur_per_kwh=value, provisional=flag) + + +def _held(rec) -> tuple: + """`(points, whole hours they cover)` for a day record. `(0, 0)` otherwise. + + ⚠️ POINTS AND HOURS ARE TWO NUMBERS, AND FCST-04's STATUS ROW PRINTED THE + FIRST UNDER THE SECOND'S NAME. The row was `f"{d} ({_n_points(rec)} h)"`, + which counts POINTS: on the quarter-hourly feed this add-on already + supports it read `'2026-01-13 (96 h)'` - 96 "h" for a 24-hour day. That is + the points/hours conflation H1 was raised for, on the row H1's own ticket + shipped. Both numbers are reported now, so an operator can see a day that + is 24 points over 24 h and a day that is 96 points over 24 h and tell them + apart, and neither number is wearing the other's unit. ⚠️ TOLERANT, AND IT IS THE SECOND LINE OF DEFENCE, NOT THE FIRST. `_load()` drops a malformed day record and says so, so this should never see one. It @@ -220,11 +277,52 @@ def _n_points(rec) -> int: "oops"}}` gave `ok=True`, `last_error=None`, and an `AttributeError` out of `words`, `yesterday()` and `resolve_tariff()` alike, with no self-heal because `_remember()` raised before it could rewrite the file. + + ⚠️ AND IT COUNTS ONLY ROWS THAT ARE POINTS. A row `_point_of()` refuses is + not held prices, and a row count that included it would report a store as + healthier than it is - which is the second half of FCST-05's MEDIUM: the + status row said `'2026-01-15 (1 h)'` about a record whose single row was + killing every tariff refresh. What is dropped is named in `last_error`, and + `words` prints that beside the counts. """ + if not isinstance(rec, dict): + return (0, 0) + raw = rec.get("points") + if not isinstance(raw, (list, tuple)): + return (0, 0) + n, hours = 0, set() + for row in raw: + pt = _point_of(row) + if pt is None: + continue + n += 1 + hours.add(pt.start.astimezone(timezone.utc) + .replace(minute=0, second=0, microsecond=0)) + return (n, len(hours)) + + +def _row_of(day: str, points: int, hours: int) -> str: + """One day's cell of the status row: what is held, in BOTH of its units. + + ⚠️ THE UNIT IS SPELT OUT ON EACH NUMBER. FCST-04's cell was + `f"{day} ({points} h)"` - the point count wearing the hour's name, which on + a quarter-hourly feed reads `'2026-01-13 (96 h)'` for a 24-hour day. What + an operator needs from this row is whether tomorrow morning's fill has a + whole day to carry, and that is the HOUR figure; the point count says how + finely it is priced. Neither answers for the other. + """ + return (f"{day} ({points} point{'' if points == 1 else 's'} " + f"over {hours} h)") + + +def _bad_rows(rec) -> int: + """How many rows of a day record are not points. Total; never raises.""" if not isinstance(rec, dict): return 0 - pts = rec.get("points") - return len(pts) if isinstance(pts, (list, tuple)) else 0 + raw = rec.get("points") + if not isinstance(raw, (list, tuple)): + return 0 + return sum(1 for row in raw if _point_of(row) is None) class PriceHistory: @@ -269,11 +367,33 @@ class PriceHistory: "one, and the fill has nothing to carry until the day " "after that", self.path) return - except (json.JSONDecodeError, OSError) as err: + except (ValueError, OSError) as err: # A corrupt cache must not stop the controller, and must not be # silently re-created either: the record is discarded, the reason # is kept, and `yesterday()` reports it rather than answering # "no record", which would read as a normal first day. + # + # ⚠️ `ValueError`, NOT `json.JSONDecodeError`, AND THAT IS + # FCST-05's HIGH. Reading this file has two failure families: the + # BYTES (`OSError`) and the CONTENT - and the content family has at + # least two members, because `open(..., encoding="utf-8")` decodes + # before `json` ever sees a character. `UnicodeDecodeError` is + # `ValueError -> UnicodeError -> UnicodeDecodeError`; it is NOT a + # `json.JSONDecodeError` and it is NOT an `OSError` + # (`issubclass(UnicodeDecodeError, OSError)` is False). So ONE + # non-UTF-8 byte in /data/price_history.json escaped `_load()` -> + # `_open()` -> `PriceHistory.__init__`, which is unguarded at + # main.py's `Controller.__init__`, AND THE ADD-ON DID NOT START. + # Measured: `UnicodeDecodeError: 'utf-8' codec can't decode byte + # 0xff in position 39`. Valid-UTF-8-invalid-JSON, a UTF-8 BOM and a + # non-dict day record all degraded correctly - it was the one byte + # class left uncovered, which is the same class as the defect the + # `_usable_days()` rework was raised for. + # + # The tuple names the FAMILY rather than the two members known + # today, because "the file's content is not readable as JSON" is + # exactly what this handler says, and a third `ValueError` subclass + # nobody has thought of is how this defect arrived the first time. self.last_error = f"unreadable ({err})" _LOG.warning("price history at %s is %s - starting from empty. " "§8.3 carry-forward has nothing to carry until the " @@ -318,20 +438,41 @@ class PriceHistory: Dropped, not repaired: an unusable record has nothing in it to keep, and the next `_remember()` writes a fresh one under that date. + + ⚠️ AND THE ROWS INSIDE A KEPT RECORD ARE COUNTED, NOT ONLY THE RECORD + (FCST-05). This checked that `rec["points"]` is a LIST and stopped + there, so a record holding one row with a `start` of `7` loaded as + healthy: `last_error=None`, and `words` said `'2026-01-15 (1 h)'` about + a store whose next `_remember()` would raise on that row and go on + raising forever. The rows are NOT dropped here - `_parse_points()` is + the read path's one filter and stays so - but they are COUNTED, so the + store stops reporting health it does not have from the moment it is + opened. `_remember()` drops them on the next refresh and `_save()` + rewrites the file, after which a restart clears the note. """ - kept, dropped = {}, [] + kept, dropped, malformed = {}, [], {} for key, rec in days.items(): if not isinstance(key, str) or not isinstance(rec, dict) \ or not isinstance(rec.get("points"), list): dropped.append(str(key)) continue + bad = _bad_rows(rec) + if bad: + malformed[key] = bad kept[key] = rec + notes = [] if dropped: + notes.append(f"{len(dropped)} unusable day record(s) dropped " + f"({', '.join(sorted(dropped))})") + if malformed: + notes.append( + f"{sum(malformed.values())} malformed point row(s) in " + + ", ".join(f"{d} ({n})" for d, n in sorted(malformed.items()))) + if notes: # Recorded, not silent: `yesterday()` and `words` both report # `last_error`, and a cache that quietly discards half a file reads # to an operator exactly like a normal first day. - self.last_error = (f"{len(dropped)} unusable day record(s) dropped " - f"({', '.join(sorted(dropped))})") + self.last_error = "; ".join(notes) _LOG.warning("price history at %s: %s - the controller starts " "anyway and the horizon is as short as the feed makes " "it", self.path, self.last_error) @@ -417,6 +558,7 @@ class PriceHistory: no_data=lambda _r: None) touched = set() + junked = {} for p in points: day = _bdate(p.start) rec = self._days.setdefault( @@ -424,14 +566,52 @@ class PriceHistory: # Rule 4: MERGE by instant. A later read updates the hours it # carries and leaves the rest; a revised price for an hour already # held wins, because it is the newer statement about that hour. + # + # ⚠️ EVERY HELD ROW GOES THROUGH `_point_of()` (FCST-05's MEDIUM). + # This was `{q["start"]: q for q in raw if isinstance(q, dict) and + # "start" in q}` sorted by `q["start"]` - a key and a sort key that + # trusted a value read off disk to be a hashable, sortable string. + # An `int`, `null`, a `dict` or a `list` there raised `TypeError` + # out of `carry()` on the day the feed was writing to, and + # `run_tariff_refresh` caught it, so the add-on stayed up while + # EVERY refresh failed for good. A row that is not a point is now + # dropped and reported here, exactly as the read path drops it, and + # `_save()` below writes the record back clean - which is the + # self-heal the raising version could never reach. + # + # ⚠️ AND THE MERGE KEY IS THE VALIDATED INSTANT, NOT ITS SPELLING. + # Aware datetimes hash and compare by instant, so this is rule 4 as + # rule 4 is written. The old string key merged by SPELLING, and the + # old lexicographic sort put the second 02:00 of the 25-hour + # October day (`+01:00`) BEFORE the first (`+02:00`). Only the + # stored order was affected - `_parse_points()` re-sorts by instant + # on the way out - but the store now holds what it says it holds. raw = rec.get("points") - held = {q["start"]: q for q in (raw if isinstance(raw, list) else ()) - if isinstance(q, dict) and "start" in q} - held[p.start.isoformat()] = p.canonical() - rec["points"] = sorted(held.values(), key=lambda q: q["start"]) + held = {} + for q in (raw if isinstance(raw, list) else ()): + kept = _point_of(q) + if kept is None: + junked[day] = junked.get(day, 0) + 1 + continue + held[kept.start] = q + held[p.start] = p.canonical() + order = sorted(held, key=lambda t: t.astimezone(timezone.utc)) + rec["points"] = [held[t] for t in order] rec["product"], rec["provider"] = product, provider touched.add(day) + if junked: + # Said, not swallowed. `words` prints `last_error`, so a store that + # has just discarded rows off disk stops reporting plain health - + # and after `_save()` below the file is clean, so a restart clears + # this. `_save()` overwrites it if the write itself fails, which is + # the more urgent fact of the two. + self.last_error = ( + f"{sum(junked.values())} malformed point row(s) dropped from " + + ", ".join(f"{d} ({n})" for d, n in sorted(junked.items()))) + _LOG.warning("price history at %s: %s - the record is rewritten " + "without them and the refresh continues", + self.path, self.last_error) self._prune(stamp) self._save() _LOG.debug("price history: %d point(s) filed under %s; %d day(s) held", @@ -523,15 +703,21 @@ class PriceHistory: an exception here is an add-on that does not start. It went out that way: a day record that was a string instead of a dict raised `AttributeError` from the `len(...)` below. `_load()` now drops such a - record and `_n_points()` is total, so this has two independent reasons - not to fail on a broken file. + record and `_held()` is total, so this has two independent reasons not + to fail on a broken file. + + ⚠️ AND IT COUNTED POINTS AND CALLED THEM HOURS (FCST-05's LOW). The row + was `f"{d} ({_n_points(rec)} h)"`; on the quarter-hourly feed this + add-on already supports that reads `'2026-01-13 (96 h)'` for a 24-hour + day. Both numbers are printed now - see `_held()` - because they are + two different facts and only one of them was ever being shown. """ if not self.ok: return f"unavailable - {self.last_error}" if not self._days: why = f" - {self.last_error}" if self.last_error else "" return f"empty{why} - nothing to carry forward yet" - held = "; ".join(f"{d} ({_n_points(self._days[d])} h)" + held = "; ".join(_row_of(d, *_held(self._days[d])) for d in sorted(self._days, key=str)) # A file that PARTLY loaded says so, or the dropped days read as days # that were simply never written. @@ -550,21 +736,14 @@ def _parse_points(raw) -> tuple: return (), "the `points` field is not a list" out, dropped = [], 0 for row in raw: - if not isinstance(row, dict): + # ⚠️ `_point_of()` IS THE ONE VALIDATOR, and this loop used to BE it. + # The write path had its own, weaker standard; see `_point_of()` for + # what that cost. One standard, two callers, no asymmetry to re-open. + pt = _point_of(row) + if pt is None: dropped += 1 continue - try: - start = datetime.fromisoformat(row["start"]) - value = float(row["eur_per_kwh"]) - flag = row["provisional"] - except (KeyError, TypeError, ValueError): - dropped += 1 - continue - if start.tzinfo is None or start.utcoffset() is None \ - or not isinstance(flag, bool): - dropped += 1 - continue - out.append(PricePoint(start=start, eur_per_kwh=value, provisional=flag)) + out.append(pt) out.sort(key=lambda p: p.start.astimezone(timezone.utc)) return tuple(out), (f"{dropped} malformed point(s)" if dropped else "") diff --git a/goodwe_controller/test_price_forecast.py b/goodwe_controller/test_price_forecast.py index cb2aeb6..eff1c4f 100644 --- a/goodwe_controller/test_price_forecast.py +++ b/goodwe_controller/test_price_forecast.py @@ -59,6 +59,7 @@ Read MUTATION.md before you report a survivor. import ast import inspect +import itertools import os import sys from datetime import datetime, timedelta, timezone @@ -354,14 +355,164 @@ check("...and still REFUSES a repeated CARRIED hour, which is the DST " "mispricing that count existed to catch", raises(lambda: _fc._audit([_fc._carried(D0, 0.11), _fc._carried(D0, 0.12)], {}), ValueError)) -check("...and refuses the SAME published point emitted twice - the provider's " - "points are strictly increasing, so a repeat is never a second print", +check("...and STILL refuses the SAME published point emitted twice - the " + "provider's points are strictly increasing, so a repeat is never a " + "second print. The old hour COUNT already refused this (one point twice " + "is one hour twice); what is graded here is that the relaxation did not " + "lose it, not that it added it", raises(lambda: _fc._audit([_QUARTERS.points[0], _QUARTERS.points[0]], _PUB_Q), ValueError)) -check("...and refuses a CARRIED point standing in an hour the feed DID " - "publish, which the old count never looked at", +check("...and STILL refuses a CARRIED point standing in an hour the feed DID " + "publish. The old COUNT never looked at it; the old IDENTITY rule did, " + "and raised - so this is preserved behaviour, not new strictness. " + "FCST-04's text called it new here and in `_audit`'s docstring, and a " + "measured differential says otherwise", raises(lambda: _fc._audit([_fc._carried(D0, 0.11)], _PUB_Q), ValueError)) +# ⚠️ AND WHAT THE AUDIT DOES NOT DO, STATED WHERE A READER OF THESE THREE +# CHECKS WOULD OTHERWISE ASSUME IT (FCST-05, MUTATION.md habit 16). `_audit` is +# a PROVENANCE post-condition, not a COMPLETENESS one: it grades every point +# that IS emitted and says nothing about points that are not. So it accepts the +# very decimation FCST-04 was raised for. That is not a regression - FCST-03's +# audit was equally blind - but it means the audit is NOT the thing that stops +# H1 coming back. The count and identity checks above this line are. +check("the audit ACCEPTS the H1 decimation itself - four published points in " + "one hour, only the last emitted - and accepts an EMPTY output. It is a " + "provenance check, not a completeness one, and the checks that guard " + "provenance completeness are the COUNT and IDENTITY ones above", + derived(lambda: _fc._audit([_QUARTERS.points[3]], _PUB_Q)) is None + and derived(lambda: _fc._audit([], _PUB_Q)) is None) + + +# --------------------------------------------------------------------------- # +# ⚠️ AND THE CLAIM "STRICTER" IS ITSELF A CHECK NOW (FCST-05). +# +# FCST-04 rewrote `_audit` and wrote, in the module and in the three checks +# above, that the result was STRICTER than FCST-03's. It is not. Two of the +# stated reasons were false - the old `seen` count already refused the same +# point emitted twice, and the old IDENTITY rule already refused a carried +# point in a published hour - and a right decision with a fabricated reason is +# a defect in this repository (MUTATION.md, and the reason DECISIONS files +# exist). So the claim is measured here rather than asserted in prose: +# FCST-03's audit is reproduced below from dd40c56 - its RULES verbatim, its +# message TEXT shortened, and nothing here reads a message - and the two are +# run over the whole domain `_audit` can express, in BOTH directions. +# +# THE DOMAIN, AND ITS SIZE. Every published mask over 3 hours (2**3) crossed +# with every output series of length 0-3 drawn from 3 hours x 5 point kinds. +# The nominal cross-product is 28 928; three of those kinds - published, +# published-provisional, copy-of-published - DO NOT EXIST for an hour the feed +# did not publish, and those combinations are SKIPPED rather than faked, which +# leaves 11 990 realisable comparisons. A fabricated "published point" for an +# unpublished hour is not in either audit's domain, and inventing one would +# grade the fixture rather than the code. +# +# ⚠️ THE OLD COPY IS A LITERAL, NOT A CALL. MUTATION.md habit 2: a differential +# whose two sides both come from the shipped module compares a thing against +# itself and moves with every edit. This side is frozen text. +def _audit_fcst03(points, published): + """FCST-03's `_audit`, verbatim from dd40c56. Its `published` maps a UTC + hour to ONE point, which is exactly the expressiveness this ticket relaxed. + """ + seen = set() + for p in points: + u = _fc._utc_hour(p.start) + if u in seen: + raise ValueError("the filled series repeats " + u.isoformat()) + seen.add(u) + src = published.get(u) + if src is None: + if p.provisional is not True: + raise ValueError(u.isoformat() + " is carried and is flagged " + "provisional=" + repr(p.provisional)) + else: + if p is not src: + raise ValueError(u.isoformat() + " is published but the series " + "carries a COPY of that point") + + +def _verdict(fn, pts, pub): + try: + fn(list(pts), pub) + return "accept" + except ValueError: + return "raise" + + +_DH = [D0 + timedelta(hours=6 + i) for i in range(3)] +_weaker, _stronger, _compared = [], [], 0 +for _mask in itertools.product((0, 1), repeat=3): + _pub = {u: PricePoint(start=u, eur_per_kwh=0.10 + i, provisional=False) + for i, (u, _on) in enumerate(zip(_DH, _mask)) if _on} + _old_pub = dict(_pub) + _new_pub = {u: (q,) for u, q in _pub.items()} + # The five shapes an output point can take at a given hour. Three of them + # do not exist for an hour the feed did not publish, and those combinations + # are skipped rather than faked - a fabricated "published point" for an + # unpublished hour is not in either audit's domain. + _kinds = [] + for _u in _DH: + _src = _pub.get(_u) + _kinds.append(("published", _src)) + _kinds.append(("published-provisional", + None if _src is None else + PricePoint(start=_src.start, + eur_per_kwh=_src.eur_per_kwh, + provisional=True))) + _kinds.append(("carried", _fc._carried(_u, 0.5))) + _kinds.append(("carried-unflagged", + PricePoint(start=_u, eur_per_kwh=0.5, + provisional=False))) + _kinds.append(("copy-of-published", + None if _src is None else + PricePoint(start=_src.start, + eur_per_kwh=_src.eur_per_kwh, + provisional=_src.provisional))) + for _n in range(4): + for _combo in itertools.product(_kinds, repeat=_n): + if any(k[1] is None for k in _combo): + continue + _pts = [k[1] for k in _combo] + _compared += 1 + _o = _verdict(_audit_fcst03, _pts, _old_pub) + _nw = _verdict(_fc._audit, _pts, _new_pub) + if _o == "raise" and _nw == "accept": + _weaker.append((_mask, [k[0] for k in _combo])) + if _o == "accept" and _nw == "raise": + _stronger.append((_mask, [k[0] for k in _combo])) + +# ⚠️ THE PRECONDITION FIRST (MUTATION.md habit 1). A differential over a +# domain that cannot reach the two disputed cases would score 0-0 for the wrong +# reason and read exactly like this one, so the two cases are driven against +# the OLD audit by name before the totals are graded. +_probe = PricePoint(start=_DH[0], eur_per_kwh=0.1, provisional=False) +check(f"the differential really covers the two claims it is here to settle, " + f"or it is {_compared} runs of nothing: FCST-03's own audit raises on " + f"BOTH 'the same published point twice' and 'a carried point in a " + f"published hour', which is what FCST-04 said it did not", + _compared > 3000 + and _verdict(_audit_fcst03, [_probe, _probe], {_DH[0]: _probe}) == "raise" + and _verdict(_audit_fcst03, [_fc._carried(_DH[0], 0.5)], + {_DH[0]: _probe}) == "raise") +check(f"...and over all {_compared} of them the reworked audit is NOT WEAKER " + f"than FCST-03's: {len(_weaker)} case(s) the old one raised and this one " + f"accepts {_weaker[:2]}", + _weaker == []) +check(f"...and NOT STRONGER either: {len(_stronger)} case(s) the old one " + f"accepted and this one raises {_stronger[:2]}. FCST-04's docstring and " + f"two of these checks said it was stricter and named two reasons, and " + f"BOTH were already refused by the old count and the old identity rule. " + f"The only behavioural change is the RELAXATION below, which is what the " + f"rework was for", + _stronger == []) +_QP = [PricePoint(start=_DH[0] + timedelta(minutes=15 * i), + eur_per_kwh=0.1 + i, provisional=False) for i in range(4)] +check("...and the relaxation is real and is NOT expressible to the old audit: " + "four distinct published points inside one hour, which FCST-03's count " + "refused as a repeated hour and this one accepts", + _verdict(_fc._audit, _QP, {_DH[0]: tuple(_QP)}) == "accept" + and _verdict(_audit_fcst03, _QP, {_DH[0]: _QP[0]}) == "raise") + # ⚠️ THE HORIZON IS WHOLE HOURS AT BOTH ENDS (FCST-04 L8). `since` is floored # and `until` was not, so a 48 h request made at any instant off the hour came # back 49 hours long. Graded at an OFF-THE-HOUR instant, because on the hour diff --git a/goodwe_controller/test_price_history.py b/goodwe_controller/test_price_history.py index c547f1b..c90fb87 100644 --- a/goodwe_controller/test_price_history.py +++ b/goodwe_controller/test_price_history.py @@ -34,6 +34,18 @@ cache cannot decay by being fed its own output. `PriceHistory(...)` and `.words` are unguarded in `Controller.__init__` - so the add-on did not start, and never self-healed. + AC5 (FCST-05) THE CACHE'S OWN CONTRACT HOLDS FOR EVERY BYTE AND EVERY ROW. + A file holding one non-UTF-8 byte no longer stops the add-on from + starting - `_load()` caught `(json.JSONDecodeError, OSError)` and + `UnicodeDecodeError` is neither, so it escaped `PriceHistory.__init__` + and `Controller.__init__` does not guard it. And a malformed `start` + under the day the feed is WRITING to no longer kills the tariff plan + for good: the write path shares `_point_of()` with the read path, + drops the row, says so, and `_save()` rewrites the record clean. + ⚠️ THE STATUS ROW IS PART OF THIS. It said `'2026-01-15 (1 h)'` about a + store whose every refresh was raising - health it did not have, in a + unit that was not its own (it counts POINTS). + THE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will: * EVERY VERDICT IS DERIVED. The carried/published split is re-derived here @@ -262,6 +274,13 @@ YESTERDAY_MWH = {h: 400.0 + 3 * h for h in range(24)} MARKUP, TAXES = 0.03, 0.06 +# ⚠️ FCST-05's HIGH, AS BYTES. A cache that is structurally valid JSON up to +# the point where one byte is not UTF-8 - which is what a truncated write, a +# half-flushed page or a foreign writer produces. Written with `open(..., "wb")` +# and never through a text handle, or the fixture cannot hold the byte it is +# named for. `0xff` is not a valid UTF-8 start byte in any position. +_UTF8_BOMB = b'{"version": 1, "days": {"2026-01-14": "\xff\xfe"}}' + def allin(mwh): """EUR/MWh as published -> the all-in EUR/kWh `assemble()` produces.""" @@ -896,6 +915,55 @@ check("an unreadable file: the reason is kept and reported, and the record is " and _h6.yesterday(NOW).kind == "no_data" and "unreadable" in why(_h6.yesterday(NOW))) +# ⚠️ AND THE BYTE CLASS THAT FIXTURE CANNOT REACH - FCST-05's HIGH. The row +# above is INVALID JSON but it is PERFECTLY VALID UTF-8, so it is decoded and +# then refused by `json`, which raises `json.JSONDecodeError`. A file holding +# one byte that is not UTF-8 at all fails EARLIER, in the decode `open(..., +# encoding="utf-8")` performs before `json` sees a character, and +# `UnicodeDecodeError` is `ValueError -> UnicodeError -> UnicodeDecodeError`: +# NOT a `JSONDecodeError`, and NOT an `OSError` either. `_load()` caught +# `(json.JSONDecodeError, OSError)`, so it escaped `_load()` -> `_open()` -> +# `PriceHistory.__init__` - unguarded in `Controller.__init__` - AND THE ADD-ON +# DID NOT START, on a cache whose own contract says "a corrupt cache must not +# stop the controller". +# +# The precondition is asserted first, because it is the whole reason the check +# above could not see this: if `"{ this is not json"` were undecodable, the two +# fixtures would be one fixture and this would grade nothing (MUTATION.md +# habit 1). +check("the two unreadable-file fixtures are DIFFERENT FAILURES: the one above " + "is valid UTF-8 that is not JSON, and the one below is not UTF-8 at all. " + "A file that is merely bad JSON cannot reach the decode step, which is " + "why the suite had this hole", + open(_p6, "rb").read().decode("utf-8") == "{ this is not json" + and raises(lambda: _UTF8_BOMB.decode("utf-8"), UnicodeDecodeError)) + +_d6b = tmpdir() +_p6b = os.path.join(_d6b, "price_history.json") +open(_p6b, "wb").write(_UTF8_BOMB) +_h6b = derived(lambda: PriceHistory(_p6b)) +check(f"a single non-UTF-8 byte in the cache does NOT stop the store from " + f"being constructed {_h6b!r}. `PriceHistory(...)` is unguarded in " + f"`Controller.__init__`, so an exception here is an add-on that does not " + f"start - which is what shipped", + isinstance(_h6b, PriceHistory)) +check(f"...and it degrades the SAME WAY the bad-JSON file does: the reason is " + f"kept, named and reported " + f"{(_h6b.last_error if isinstance(_h6b, PriceHistory) else _h6b)!r}", + isinstance(_h6b, PriceHistory) + and "unreadable" in (_h6b.last_error or "") + and "codec can't decode byte" in (_h6b.last_error or "") + and _h6b.yesterday(NOW).kind == "no_data" + and "unreadable" in why(_h6b.yesterday(NOW))) +_w6b = derived(lambda: _h6b.words) +_plan6b = derived(lambda: run(resolve_tariff( + CFG, NOW, _until, transport=transport_for(_body_today), + now=NOW, history=_h6b)).origin) +check(f"...and NOTHING downstream raises on it either: not `words` ({_w6b!r}), " + f"not the shipped `resolve_tariff()` path ({_plan6b!r}). §6: absent " + f"machinery must not disable the loop, and a JSON file is machinery", + isinstance(_w6b, str) and _plan6b == "provider") + _p7 = os.path.join(tmpdir(), "price_history.json") json.dump({"version": SCHEMA + 99, "days": {}}, open(_p7, "w", encoding="utf-8")) _h7 = PriceHistory(_p7) @@ -1008,11 +1076,49 @@ check("a healthy but EMPTY store says something different, because it is a " and words_of(_h_words).startswith("empty") and "nothing to carry forward" in words_of(_h_words)) _ph.PriceHistory._remember(_h_words, series_of(YDAY_H, YESTERDAY_MWH), NOW) -check(f"...and a store holding a day names the DATE and the HOUR COUNT " - f"{words_of(_h_words)!r} - 'a cache exists' and 'the cache holds " - f"yesterday, all 24 hours of it' are different facts, and only the " - f"second says the horizon will be full tomorrow morning", - words_of(_h_words) == f"{YDAY_D} (24 h)") +check(f"...and a store holding a day names the DATE, the POINT COUNT and the " + f"HOUR COUNT {words_of(_h_words)!r} - 'a cache exists' and 'the cache " + f"holds yesterday, all 24 hours of it' are different facts, and only " + f"the second says the horizon will be full tomorrow morning", + words_of(_h_words) == f"{YDAY_D} (24 points over 24 h)") + +# ⚠️ AND THE TWO NUMBERS ARE NOT THE SAME NUMBER - FCST-05's LOW. The row +# shipped as `f"{d} ({_n_points(rec)} h)"`, which counts POINTS and prints them +# under the HOUR's name. On an hourly feed the two are equal and nothing shows; +# on the quarter-hourly feed this add-on already supports (`_published_by_hour` +# keeps a TUPLE per hour precisely for it, and §7 below drives one) the row +# read '2026-01-13 (96 h)' - 96 "h" for a 24-hour day. That is the points/hours +# conflation H1 was raised for, on the row H1's own ticket shipped. +# +# MUTATION.md's rule: this check asserts a DIFFERENCE, so the fixture is run +# with the two numbers actually made different. 96 != 24, asserted first. +_h_q = store() +_QUARTERS = [u + timedelta(minutes=15 * q) for u in YDAY_H for q in range(4)] +_ph.PriceHistory._remember( + _h_q, PriceRead.series( + [PricePoint(start=brussels(u), + eur_per_kwh=allin(YESTERDAY_MWH[brussels(u).hour] + + brussels(u).minute), + provisional=False) for u in _QUARTERS], + PriceProduct.DAY_AHEAD, "generic_dayahead", NOW), + NOW) +_qw = words_of(_h_q) +check(f"the quarter-hourly fixture really is sub-hourly and its two counts " + f"really differ ({len(_QUARTERS)} points, " + f"{len({brussels(u).hour for u in _QUARTERS})} local hours), or 'points' " + f"and 'hours' are the same observation and the row grades nothing", + len(_QUARTERS) == 96 + and len({brussels(u).hour for u in _QUARTERS}) == 24 + and len({u.minute for u in _QUARTERS}) == 4) +check(f"...and the row reports BOTH, each under its own unit: {_qw!r}. " + f"A day of 96 quarter-hourly prints is 96 points over 24 h, and the " + f"figure that says whether tomorrow's fill has a whole day to carry is " + f"the HOUR one", + _qw == f"{YDAY_D} (96 points over 24 h)") +check(f"...and specifically the row does NOT say '96 h' {_qw!r} - a point " + f"count wearing the hour's name is H1's defect, and this row is where " + f"an operator would have read it", + isinstance(_qw, str) and "(96 h)" not in _qw and " (96 points " in _qw) check(f"a file that only PARTLY loaded says so in the row {_w14!r}, or the " f"days it dropped read as days that were simply never written", isinstance(_w14, str) and (_h14.last_error or "") in _w14 @@ -1023,7 +1129,7 @@ check("...and `words` never raises, whatever the file holds. It is read once " "from Controller.__init__ and again on every status render, and neither " "call site guards it - an exception here is an add-on that does not start", all(isinstance(words_of(h), str) - for h in (_missing, _h6, _h7, _h14, _h_words))) + for h in (_missing, _h6, _h6b, _h7, _h14, _h_words))) # ⚠️ AND A FAILED WRITE LEAVES NOTHING BEHIND. `_save()` writes to a `mkstemp` # and renames - atomic, so a power cut cannot leave a truncated file that @@ -1067,6 +1173,7 @@ check(f"...and a SUCCESSFUL write leaves exactly the one file " # provider series with no invented points and no §11.4 degradation. for _label, _hist in (("no durable volume", _missing), ("unreadable file", _h6), + ("a non-UTF-8 byte in the file", _h6b), ("wrong schema", _h7)): _plan = run(resolve_tariff(CFG, NOW, _until, transport=transport_for(_body_today), now=NOW, @@ -1080,6 +1187,198 @@ for _label, _hist in (("no durable volume", _missing), and not any(p.provisional for p in _plan.read.points)) +# --------------------------------------------------------------------------- # +print() +print("5c. FCST-05 - the WRITE path is as defensive as the read path") + +# THE DEFECT. `_remember()` merged a day's held points with +# +# held = {q["start"]: q for q in raw +# if isinstance(q, dict) and "start" in q} +# rec["points"] = sorted(held.values(), key=lambda q: q["start"]) +# +# which trusts a value read off /data to be a HASHABLE, SORTABLE STRING. It is +# whatever is in the file. `_usable_days()` required `rec["points"]` to be a +# LIST and never looked at the elements, and `_parse_points()` - the READ path - +# was properly defensive, so `yesterday()` survived and the §5 `_p8` fixture +# passed. Two standards for what a stored point is; the weaker one on the path +# that WRITES. +# +# ⚠️ AND THE PLACEMENT IS THE WHOLE FIXTURE. `_p8` files its malformed rows +# under a day the feed is NOT writing to, so `_remember()` never touches that +# record and the defect is invisible. What reaches it is a malformed row under +# TODAY - the record the running feed merges into on every refresh - so that is +# asserted before anything is graded (MUTATION.md habit 1), and the harmless +# placement is run as the accepted baseline it varies from (habit 7). +# +# ⚠️ AND THE BLAST RADIUS IS WHY IT IS NOT MERELY UNTIDY. `carry()` is reached +# from `resolve_tariff()` from `run_tariff_refresh`, which catches `Exception`: +# the add-on stays UP and EVERY refresh raises, forever. The plan retires after +# TARIFF_MAX_AGE_S and arbitrage and the cheap window are off permanently, with +# no self-heal, because `_remember()` raises before it reaches `_save()`. + +_BAD_STARTS = ( + ("an int", 7), # TypeError: '<' between 'str' and 'int' + ("a dict", {"a": 1}), # TypeError: unhashable as a dict key + ("null", None), # TypeError: '<' between 'str', 'NoneType' + ("a list", [1, 2]), # TypeError: unhashable as a dict key + ("a string that is not an instant", "nope"), # sorts fine, never a point +) + + +def _seeded(day, bad_start): + """A store on disk holding ONE malformed row under `day`, plus a good one.""" + d = tmpdir() + path = os.path.join(d, "price_history.json") + good = {"start": brussels(at_local(day, 5)).isoformat(), + "eur_per_kwh": allin(YESTERDAY_MWH[5]), "provisional": False} + json.dump({"version": SCHEMA, "days": {day: { + "product": "dynamic_dayahead", "provider": "generic_dayahead", + "points": [good, {"start": bad_start, "eur_per_kwh": 0.2, + "provisional": False}]}}}, + open(path, "w", encoding="utf-8")) + return path, PriceHistory(path) + + +_TODAY_FEED = series_of(TODAY_H, TODAY_MWH) +check(f"the fixture files its malformed row under {TODAY_D}, which is exactly " + f"the Brussels day the feed being carried writes to " + f"{sorted({_ph._bdate(p.start) for p in _TODAY_FEED.points})} - under any " + f"other day `_remember()` never opens the record and this whole section " + f"grades nothing, which is why the suite already had this hole", + {_ph._bdate(p.start) for p in _TODAY_FEED.points} == {TODAY_D}) + +# The accepted baseline: the SAME row, one day over. Shipped and broken alike +# pass this, and that is the point of running it (habit 7). +_pB, _hB = _seeded(YDAY_D, 7) +_cB = derived(lambda: _hB.carry(_TODAY_FEED, since=NOW, until=_until, stamp=NOW)) +check("baseline: the same malformed row under a day the feed is NOT writing " + "to is carried without incident - one variable moves between this and " + "every check below, and it is the DAY", + not isinstance(_cB, _Raised) and _cB.kind == "series") + +for _shape, _bad in _BAD_STARTS: + _pM, _hM = _seeded(TODAY_D, _bad) + _wload = words_of(_hM) + _c1 = derived(lambda h=_hM: h.carry(_TODAY_FEED, since=NOW, until=_until, + stamp=NOW)) + _saw = repr(_c1) if isinstance(_c1, _Raised) else _c1.kind + check(f"a `start` that is {_shape} ({_bad!r}) under TODAY's own record " + f"does not take the refresh down: `carry()` returns a series " + f"({_saw}). It raised " + f"`TypeError` out of the shipped path, and `run_tariff_refresh` " + f"catches `Exception`, so the add-on stayed up while every refresh " + f"failed for good", + not isinstance(_c1, _Raised) and _c1.kind == "series") + # ⚠️ THE VERDICT IS NOT THE CLAIM. "It did not raise" is satisfied by a + # store that silently kept the poison row; what this section promises is + # that the row is GONE FROM THE FILE, so the next start is clean. + _disk = json.load(open(_pM, encoding="utf-8")) + _rows = _disk["days"][TODAY_D]["points"] + # DERIVED, not a constant: the surviving rows are exactly the DISTINCT + # instants the feed published for that day plus the fixture's one good + # seeded row - which is 05:00, an hour the feed also prints, so it merges. + _want = len({p.start.astimezone(UTC) for p in _TODAY_FEED.points} + | {at_local(TODAY_D, 5).astimezone(UTC)}) + check(f"...and the row is DROPPED FROM /data, not carried: {len(_rows)} " + f"row(s) on disk against {_want} distinct instants, every one of " + f"them a point. `_remember()` used to raise BEFORE `_save()`, so " + f"there was no self-heal at all - the store came back poisoned on " + f"every restart", + all(_ph._point_of(r) is not None for r in _rows) + and len(_rows) == _want) + # ...and the store stops reporting health, at BOTH moments it can be read: + # when it is opened, and after the refresh that cleaned it. + # ⚠️ THE COUNTS ARE ASSERTED TOO, NOT ONLY THE NOTE. The seeded record holds + # ONE point and one row that is not one, so a row printing "2 points" would + # be reporting the poison as held prices - which is the half of this defect + # that reached the operator: `words` said "2026-01-15 (1 h)" while every + # tariff refresh was dying on that record. + check(f"...and the status row does not call that store healthy - at open " + f"{_wload!r}: ONE point held, and the row that is not a point named " + f"rather than counted", + isinstance(_wload, str) + and _wload.startswith(f"{TODAY_D} (1 point over 1 h) [") + and "1 malformed point row(s) in" in _wload) + # ⚠️ AND THE TWO NOTES ARE GRADED APART, BECAUSE THEY MASK EACH OTHER + # OTHERWISE (MUTATION.md TEST-16). `_usable_days()` records what it FOUND + # at open and `_remember()` records what it DROPPED on the refresh; both + # land in `last_error`, so a check asking only "does the row mention a + # malformed point" is answered by whichever rung is still standing - the + # first draft of this check was exactly that, and deleting `_remember()`'s + # note left the whole 29-file suite green at -1 B. The tense is the + # difference an operator reads: "in" is still there, "dropped from" is + # gone, and after the refresh only the second is true. + _wafter = words_of(_hM) + check(f"...nor after the refresh that dropped it {_wafter!r}. It said " + f"'{TODAY_D} (1 h)' - the status row reporting a full cache while " + f"every tariff refresh was dying on it", + isinstance(_wafter, str) + and "malformed point row(s) dropped from" in _wafter + and TODAY_D in _wafter) + check(f"...and the row MOVED from what was found to what was dropped " + f"{_wload.split('[')[-1]!r} -> {_wafter.split('[')[-1]!r}, so the " + f"two notes are not one note read twice", + isinstance(_wload, str) and isinstance(_wafter, str) + and "malformed point row(s) in" in _wload + and "malformed point row(s) in" not in _wafter + and "malformed point row(s) dropped from" not in _wload) + # ...and a SECOND refresh over the same store is clean, which is the state + # a restarted add-on comes up in. + _hM2 = PriceHistory(_pM) + check(f"...and the store reopened from that file is CLEAN: the whole day " + f"held and nothing to report {words_of(_hM2)!r}, so the note clears " + f"itself rather than becoming permanent furniture", + words_of(_hM2) == f"{TODAY_D} ({_want} points over {_want} h)" + and _hM2.last_error is None) + +# ...and the shipped path, end to end, through the caller that catches +# `Exception` and would otherwise have hidden this for good. +_pP, _hP = _seeded(TODAY_D, 7) +_o1 = derived(lambda: run(resolve_tariff( + CFG, NOW, _until, transport=transport_for(_body_today), now=NOW, + history=_hP)).origin) +_o2 = derived(lambda: run(resolve_tariff( + CFG, NOW, _until, transport=transport_for(_body_today), now=NOW, + history=_hP)).origin) +check(f"the SHIPPED `resolve_tariff()` path survives it TWICE ({_o1!r}, " + f"{_o2!r}) - once for the refresh that finds the poison and once for " + f"the one after it. A guard that refuses once and dies on the next call " + f"is a delay, not a guard (MUTATION.md habit 9)", + _o1 == "provider" and _o2 == "provider") + +# ⚠️ AND THE MERGE KEY IS THE INSTANT, WHICH IS WHAT RULE 4 SAYS. The old key +# was `q["start"]` - the SPELLING - so the same instant written at two offsets +# was held twice, and the lexicographic sort beside it put the second 02:00 of +# the 25-hour October day (`+01:00`) BEFORE the first (`+02:00`). Aware +# datetimes hash and compare by instant, so the validated `start` is the key. +_d15 = tmpdir() +_p15 = os.path.join(_d15, "price_history.json") +_alt = at_local(YDAY_D, 6) +json.dump({"version": SCHEMA, "days": {YDAY_D: { + "product": "dynamic_dayahead", "provider": "generic_dayahead", + "points": [{"start": _alt.astimezone(UTC).isoformat(), + "eur_per_kwh": 0.999, "provisional": False}]}}}, + open(_p15, "w", encoding="utf-8")) +_h15 = PriceHistory(_p15) +check(f"the two spellings really are one instant and really are two strings " + f"({_alt.astimezone(UTC).isoformat()!r} vs " + f"{brussels(_alt).isoformat()!r}), or 'merged by instant' and 'merged by " + f"spelling' are the same observation", + _alt.astimezone(UTC).isoformat() != brussels(_alt).isoformat() + and _alt.astimezone(UTC) == brussels(_alt)) +_ph.PriceHistory._remember( + _h15, series_of([_alt], YESTERDAY_MWH), NOW) +_y15 = _h15.yesterday(NOW) +_n15 = len(_y15.points) if _y15.kind == "series" else _y15.kind +_v15 = _y15.points[0].eur_per_kwh if _y15.kind == "series" else None +check(f"...and a revision of that hour REPLACES it rather than sitting beside " + f"it: {_n15} point held, priced {_v15} and not 0.999. Rule 4 says " + f"'MERGE BY INSTANT'; the key was the string", + _y15.kind == "series" and len(_y15.points) == 1 + and abs(_y15.points[0].eur_per_kwh - allin(YESTERDAY_MWH[6])) < 1e-9) + + # --------------------------------------------------------------------------- # print() print("6. rule 4 - hours accumulate within a day; they do not replace it")