admin/goodwe-addon
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

FCST-05: one byte stops stopping the add-on, and the write path gets the read path's standard #74

Merged
admin merged 1 commits from FCST-05 into release/1.0 2026-09-10 01:13:41 +02:00
Conversation 0 Commits 1 Files Changed 7
+847 -50
7 changed files with 847 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
goodwe_controller/CHANGELOG.md
+86 -2
View File
@@ -13,6 +13,89 @@ if it has, it needs 0.10.0. Whoever deploys decides.
| | |
|---|---|
| _this branch_ | FCST-04 - the carry-forward fill reaches the plan, and the day it carries survives a restart |
| _this branch_ | FCST-05 - one byte stops stopping the add-on, and the write path gets the read path's standard |
### FCST-05 - what FCST-04's own contract still did not hold
**A single non-UTF-8 byte in `/data/price_history.json` stopped the add-on from
starting.** `_load()` caught `(json.JSONDecodeError, OSError)`.
`UnicodeDecodeError` is `ValueError -> UnicodeError -> UnicodeDecodeError` -
neither of those, and `issubclass(UnicodeDecodeError, OSError)` is **False** -
so it escaped `_load()` -> `_open()` -> `PriceHistory.__init__`, which
`Controller.__init__` does not guard. Reproduced: `UnicodeDecodeError: 'utf-8'
codec can't decode byte 0xff`, and **the controller did not come up**. That is the same class as the day-record defect the store was reworked
for, in the one byte class the rework left uncovered: valid-UTF-8-invalid-JSON,
a UTF-8 BOM and a non-dict day record all degraded correctly. The suite could
not see it because its "unreadable file" fixture writes `"{ this is not json"`,
which is **valid UTF-8**. The handler now names the `ValueError` family - the
file's CONTENT - beside `OSError`, the file's BYTES, and a fixture holding a
genuinely undecodable byte is in `test_price_history.py` section 5.
**A malformed `start` under the day the feed is writing to killed the tariff
plan permanently.** `_remember()` merged a day's points with
`{q["start"]: q ...}` and sorted on `q["start"]`, trusting a value read off
`/data` to be a hashable, sortable string. An `int`, `null`, a `dict` or a
`list` there raised `TypeError`; `run_tariff_refresh` catches `Exception`, so
the add-on stayed **up** while **every refresh raised, forever** - the plan
retiring after `TARIFF_MAX_AGE_S` and arbitrage and the cheap window off for
good, with no self-heal because `_remember()` raised before reaching `_save()`.
The read path (`_parse_points()`) was properly defensive throughout, which is
why `yesterday()` survived and the existing fixture passed: it files its
malformed rows under a day the feed is **not** writing to, so `_remember()`
never opened that record. **The two paths now share one validator**,
`_point_of()`, and neither has a standard of its own. A row that is not a point
is dropped and reported, and `_save()` rewrites the record clean.
⚠️ **AND THE STATUS ROW SAID THE STORE WAS HEALTHY WHILE THE PLAN WAS DEAD.**
`words` read `'2026-01-15 (1 h)'` about the record that was killing every
refresh. `_load()` now counts the rows inside a kept record as well as the
record itself, so the store stops reporting plain health from the moment it is
opened; the note names what was **found** at open and what was **dropped** by
the refresh, and clears itself once the file has been rewritten.
⚠️ **AND THE ROW COUNTED POINTS AND CALLED THEM HOURS.** `f"{d} ({_n_points(d)}
h)"` on the quarter-hourly feed this add-on already supports reads
`'2026-01-13 (96 h)'` for a 24-hour day - the same points/hours conflation the
sub-hourly defect above was about, on the row that defect's own card shipped.
The cell is now `2026-01-13 (96 points over 24 h)`: two facts, each under its
own unit, and the hour figure is the one that says whether tomorrow morning's
fill has a whole day to carry.
⚠️ **AND THE MERGE KEY IS THE INSTANT RATHER THAN ITS SPELLING**, which is what
rule 4 already said. Aware datetimes hash and compare by instant. The string
key merged by spelling, and the lexicographic sort beside it put the second
02:00 of the 25-hour October day (`+01:00`) **before** the first (`+02:00`);
only the stored order was affected, because `_parse_points()` re-sorts by
instant on the way out.
⚠️ **TWO STATED REASONS WERE FALSE, AND THIS REPOSITORY TREATS THAT AS A
DEFECT.** `price_forecast._audit()`'s "WHY IT IS STRICTER" section claimed the
rework refused two things FCST-03's audit did not. A differential over the
whole domain the audit can express - every published mask over 3 hours crossed
with every output series of length 0-3 - found **zero divergence in both
directions**: the reworked audit is exactly **EQUAL** to FCST-03's, not
stronger. The old `seen` count already refused the same published point emitted
twice (one point twice is one hour twice), and the old **identity** rule
already refused a carried point sharing a published hour. The only behavioural
change is the intended **relaxation** - several distinct published points in
one hour - and that stays. The docstring and the two check texts now say what
is true, and the differential is committed as a check in
`test_price_forecast.py` rather than left as prose.
⚠️ **AND `_audit` IS NOT WHAT STOPS THE DECIMATION DEFECT RECURRING.** It does
not check **completeness**: fed the decimation directly - four published points
in an hour, output emitting only the last - it **accepts**, and it accepts an
empty output. That is a limit, not a regression, and the module docstring said
otherwise by presenting the audit as the post-condition guarding provenance.
What actually guards it is the point **count** and the pass-through **identity**
checks in `test_price_forecast.py` and `test_price_history.py`, and the text now
points at them. It is **not** made complete: `published` is the whole read while
the fill emits only `[since, until)`, so "every published point appears" is not
a true post-condition of those two arguments. Measured - the rule as stated
takes the 1 h, 12 h and 48 h horizon checks red on legitimate series.
**No option was added, `config.yaml` is untouched, and nothing was deployed.**
**FCST-03 shipped the fill and no caller.** `price_forecast.fill_from_yesterday()`
was complete and checked, and so were the consumers written for it -
@@ -58,8 +141,9 @@ spread it would have created was never compared against anything.
no durable volume passes the explicit `NoHistory()`. A default of `None`
meaning "skip the fill" would hand a caller who forgot the argument exactly
the truncated horizon this card removes, silently.
* A new status row, **Price history**, names the days held and their hour
counts.
* A new status row, **Price history**, names the days held, how many points
each holds and how many whole hours they cover (see FCST-05: it shipped
counting points and printing them under the hour's name).
⚠️ **ONE CHANGE HERE MOVES ENERGY, AND IT IS NOT THE ONE THE CARD IS NAMED FOR.**
Carried hours are provisional, so §8.3 keeps them out of the trade set: the fill
goodwe_controller/DOCS.md
+7 -2
View File
@@ -1474,8 +1474,13 @@ a cheap night that has already gone used to sit in the series as the cheapest
hour of the day, and a cycle can never buy in an hour that is past — so the
trade was quietly never taken. The spread is now taken over hours still ahead.
A new status row, **Price history**, says which days are held and how many hours
each has. If it reads `empty` or `unavailable`, nothing is carried, the horizon
A new status row, **Price history**, says which days are held, how many price
points each holds and how many whole hours those cover — `2026-01-13 (96 points
over 24 h)`. ⚠️ **Those are two different numbers and the row used to print one
of them under the other's name**: it counted points and labelled them `h`, so a
quarter-hourly feed read `96 h` for a 24-hour day. The figure that says whether
tomorrow morning's fill has a whole day to carry is the **hour** one. If it
reads `empty` or `unavailable`, nothing is carried, the horizon
is as short as the feed makes it, and every other guarantee is unchanged — a
cache that cannot answer is not allowed to be a controller that cannot regulate.
The store is rebuilt by the next successful fetch; on a brand-new install the
goodwe_controller/MUTATION.md
+15
View File
@@ -127,6 +127,21 @@ Two rules come out of it, and both are cheap:
function, that pair is the first thing to try — TEST-16's two survivors were
adjacent lines in one function and neither was individually live.
⚠️ **AND FCST-05 WALKED INTO IT AGAIN, IN ITS OWN NEW CHECK, WHICH IS WHY THE
RULE IS “mutate every check you add”.** `price_history.py` reports a malformed
point row from **two** places into **one** field: `_usable_days()` records what
it FOUND when the file was opened, and `_remember()` records what it DROPPED on
the next refresh. Both land in `last_error`, and `words` prints `last_error`.
The first draft of the new check asked `"malformed point row" in words` — which
whichever rung is still standing satisfies — so deleting `_remember()`'s note
entirely was a **−1 B SURVIVOR of the whole suite**, on the half of the fix that
tells an operator the cache **self-healed**. It is graded on the TENSE now
(`… in <day>` at open, `… dropped from <day>` after the refresh), the two are
asserted to DIFFER, and the paired mutation — both notes deleted, −91 B — is
red on 15 named checks. Same shape as TEST-16 and the same fix: **grade the
reason, not the verdict**, and when two rungs answer one stimulus, mutate them
together.
## The third way a check lies: its DATA cannot reach the failure it names
Stale bytecode manufactures survivors from a mutation that never ran. Masking
goodwe_controller/app/price_forecast.py
+72 -8
View File
@@ -55,6 +55,17 @@ So the flag is not passed through anywhere in this file:
It is a post-condition, not a filter: it never fixes a flag, because a
module that quietly corrects its own provenance is one whose provenance
cannot be trusted.
⚠️ IT GRADES THE POINTS THAT ARE EMITTED AND NOTHING ABOUT THE ONES THAT
ARE NOT (FCST-05). `_audit()` is a provenance check, NOT a completeness
check: fed FCST-04's own decimation defect - four published points in an
hour, only the last emitted - it ACCEPTS, and it accepts an empty output
too. Measured, and the FCST-03 audit it replaced was equally blind, so
this is a limit rather than a regression. What actually guards against a
decimated series is the point COUNT and the pass-through IDENTITY checks
in `test_price_forecast.py` (section on the quarter-hourly feed) and
`test_price_history.py`; see `_audit()`'s own docstring for why the
completeness rule is not expressible from its two arguments.
* and test_price_forecast.py walks THIS FILE'S AST and fails if any
`PricePoint(...)` in it passes anything but a literal for `provisional`, so
the pass-through idiom cannot come back in a later edit and pass review by
@@ -225,14 +236,67 @@ def _audit(points, published) -> None:
* CARRIED - the hour must not appear in `published` AT ALL, the flag must
be exactly True, and that hour may hold no second point.
⚠️ WHAT REPLACED "AT MOST ONE POINT PER HOUR", AND WHY IT IS STRICTER. This
audit began as a `seen` set: any repeated `_utc_hour` raised, because a
duplicated hour is how a DST day gets mispriced. A bare hour count cannot
express a quarter-hourly feed, where four distinct published points share
one hour legitimately - so the count was replaced by the two rules above,
which subsume it. A repeated CARRIED hour still raises; so does the same
published point emitted twice; and a carried point may no longer share an
hour with a published one, which the count never checked at all.
⚠️ WHAT REPLACED "AT MOST ONE POINT PER HOUR", AND WHAT THAT DID AND DID
NOT CHANGE. This audit began as FCST-03's `seen` set: any repeated
`_utc_hour` raised, beside an identity rule `p is not src`. A bare hour
count cannot express a quarter-hourly feed, where four distinct published
points share one hour legitimately - so the count was replaced by the two
rules above, which subsume it.
⚠️ AND IT IS NOT STRICTER. FCST-04's version of this paragraph claimed two
strengthenings and BOTH WERE FALSE; FCST-05 measured it. A differential
over the whole domain this audit can express - every published mask over 3
hours crossed with every output series of length 0-3 drawn from
{published, published-provisional, carried, carried-unflagged,
copy-of-published} - found ZERO divergence in EITHER direction:
OLD raised, NEW accepted (WEAKER): 0
OLD accepted, NEW raised (STRONGER): 0
The two specific claims, one at a time:
* "so does the same published point emitted twice" - the OLD `seen`
count already raised on that, because emitting one point twice is one
HOUR twice. Preserved by the identity/`emitted` rules, not added.
* "a carried point may no longer share an hour with a published one,
which the count never checked at all" - true of the COUNT, and the old
IDENTITY rule raised on it anyway: a carried point in a published hour
took the `else` branch and failed `p is not src`. The message was about
a copy rather than about a carry, which is the only thing that moved.
The one real behavioural change is the intended RELAXATION - several
distinct published points in one hour, which the old audit could not
express at all and refused. That relaxation is why this was rewritten, and
it stays. A right decision with a fabricated reason is still a defect here.
⚠️ AND THE REAL LIMIT, WHICH IS NOT A REGRESSION AND IS NOT SMALL: THIS
AUDIT DOES NOT CHECK COMPLETENESS. It grades every point that IS emitted
and says nothing about the ones that are not. Fed the H1 decimation
directly - four published points in an hour, output emitting only the last
- it ACCEPTS; so does two of four, and so does an EMPTY output. So `_audit`
is NOT what stops H1 recurring. What stops it is the point COUNT and the
IDENTITY checks in `test_price_history.py` and `test_price_forecast.py`,
which compare the emitted series against the feed's own prints.
It is not made complete here, and the reason is structural rather than
effort: `published` is `_published_by_hour(read)` over the WHOLE read,
unwindowed, while `fill_from_yesterday()` emits only [since, until). A
published point outside the window is legitimately absent, so "every
published point appears" is not a true post-condition of these two
arguments - it would need the window passed in, which is a different
signature and a different check.
⚠️ AND THAT IS MEASURED, NOT ASSUMED. FCST-05 built the rule - raise unless
every point of every `published[u]` is emitted, +162 B - and ran it. It
takes the 1 h, 12 h and 48 h HORIZON checks in test_price_forecast.py RED,
on the shipped fill and a healthy feed: at an off-the-hour instant the
feed's already-elapsed prints are outside [since, until) and are correctly
dropped, and the rule calls that a provenance failure. (The differential
beside those checks scores the same rule as 49 STRONGER cases, so it is a
real change of behaviour and not a no-op.) The rule as stated is therefore
not merely unimplemented, it is WRONG - and stating a false post-condition
is MUTATION.md habit 16's defect: an instrument that asserts a completeness
it does not have is worse than no instrument.
⚠️ RAISES, NEVER REPAIRS. A module that silently corrects its own
provenance is one whose provenance means nothing; the only safe response to
goodwe_controller/app/price_history.py
+207 -28
View File
@@ -208,8 +208,65 @@ def _yesterday_of(instant: datetime) -> str:
return (brussels(instant).date() - timedelta(days=1)).strftime("%Y-%m-%d")
def _n_points(rec) -> int:
"""How many points a day record holds. 0 for anything that is not one.
def _point_of(row):
"""One stored row -> a `PricePoint`, or `None` if the row is not one.
THE ONE VALIDATOR, AND THE ASYMMETRY IT CLOSES (FCST-05's MEDIUM). This
file used to hold TWO standards for what a stored point is. The READ path
(`_parse_points()`) was properly defensive - every field checked, a
malformed row dropped and reported. The WRITE path was
`{q["start"]: q for q in raw if isinstance(q, dict) and "start" in q}`
followed by `sorted(..., key=lambda q: q["start"])`, which trusts
`q["start"]` to be a hashable, sortable string it never checked. Measured,
with the malformed row under the day the feed is currently writing to:
start is an int -> TypeError: '<' not supported between instances
of 'str' and 'int'
start is a dict -> TypeError: cannot use 'dict' as a dict key
start is null -> TypeError: '<' not supported between 'str' and
'NoneType'
start is a list -> TypeError: cannot use 'list' as a dict key
⚠️ AND THE BLAST RADIUS IS PERMANENT, WHICH IS WHY IT IS NOT MERELY UNTIDY.
`_remember()` is reached through `carry()` -> `resolve_tariff()` ->
`run_tariff_refresh`, which catches `Exception` - so the add-on stays UP
and EVERY refresh raises forever. The plan retires after
`TARIFF_MAX_AGE_S`, and arbitrage and the cheap window are off from then
on. There is no self-heal, because `_remember()` raises before it reaches
`_save()`. `_usable_days()` only required `rec["points"]` to be a LIST; the
elements were unchecked on the write path, and the existing fixture put its
malformed row under a day the feed was NOT writing to, so `_remember()`
never touched it.
So both paths call this, and neither of them has a standard of its own.
Returning `None` rather than raising is the contract: a caller decides
whether a bad row is dropped (both of them do) and how it is reported.
"""
if not isinstance(row, dict):
return None
try:
start = datetime.fromisoformat(row["start"])
value = float(row["eur_per_kwh"])
flag = row["provisional"]
except (KeyError, TypeError, ValueError):
return None
if start.tzinfo is None or start.utcoffset() is None \
or not isinstance(flag, bool):
return None
return PricePoint(start=start, eur_per_kwh=value, provisional=flag)
def _held(rec) -> tuple:
"""`(points, whole hours they cover)` for a day record. `(0, 0)` otherwise.
⚠️ POINTS AND HOURS ARE TWO NUMBERS, AND FCST-04's STATUS ROW PRINTED THE
FIRST UNDER THE SECOND'S NAME. The row was `f"{d} ({_n_points(rec)} h)"`,
which counts POINTS: on the quarter-hourly feed this add-on already
supports it read `'2026-01-13 (96 h)'` - 96 "h" for a 24-hour day. That is
the points/hours conflation H1 was raised for, on the row H1's own ticket
shipped. Both numbers are reported now, so an operator can see a day that
is 24 points over 24 h and a day that is 96 points over 24 h and tell them
apart, and neither number is wearing the other's unit.
⚠️ TOLERANT, AND IT IS THE SECOND LINE OF DEFENCE, NOT THE FIRST. `_load()`
drops a malformed day record and says so, so this should never see one. It
@@ -220,11 +277,52 @@ def _n_points(rec) -> int:
"oops"}}` gave `ok=True`, `last_error=None`, and an `AttributeError` out of
`words`, `yesterday()` and `resolve_tariff()` alike, with no self-heal
because `_remember()` raised before it could rewrite the file.
⚠️ AND IT COUNTS ONLY ROWS THAT ARE POINTS. A row `_point_of()` refuses is
not held prices, and a row count that included it would report a store as
healthier than it is - which is the second half of FCST-05's MEDIUM: the
status row said `'2026-01-15 (1 h)'` about a record whose single row was
killing every tariff refresh. What is dropped is named in `last_error`, and
`words` prints that beside the counts.
"""
if not isinstance(rec, dict):
return (0, 0)
raw = rec.get("points")
if not isinstance(raw, (list, tuple)):
return (0, 0)
n, hours = 0, set()
for row in raw:
pt = _point_of(row)
if pt is None:
continue
n += 1
hours.add(pt.start.astimezone(timezone.utc)
.replace(minute=0, second=0, microsecond=0))
return (n, len(hours))
def _row_of(day: str, points: int, hours: int) -> str:
"""One day's cell of the status row: what is held, in BOTH of its units.
⚠️ THE UNIT IS SPELT OUT ON EACH NUMBER. FCST-04's cell was
`f"{day} ({points} h)"` - the point count wearing the hour's name, which on
a quarter-hourly feed reads `'2026-01-13 (96 h)'` for a 24-hour day. What
an operator needs from this row is whether tomorrow morning's fill has a
whole day to carry, and that is the HOUR figure; the point count says how
finely it is priced. Neither answers for the other.
"""
return (f"{day} ({points} point{'' if points == 1 else 's'} "
f"over {hours} h)")
def _bad_rows(rec) -> int:
"""How many rows of a day record are not points. Total; never raises."""
if not isinstance(rec, dict):
return 0
pts = rec.get("points")
return len(pts) if isinstance(pts, (list, tuple)) else 0
raw = rec.get("points")
if not isinstance(raw, (list, tuple)):
return 0
return sum(1 for row in raw if _point_of(row) is None)
class PriceHistory:
@@ -269,11 +367,33 @@ class PriceHistory:
"one, and the fill has nothing to carry until the day "
"after that", self.path)
return
except (json.JSONDecodeError, OSError) as err:
except (ValueError, OSError) as err:
# A corrupt cache must not stop the controller, and must not be
# silently re-created either: the record is discarded, the reason
# is kept, and `yesterday()` reports it rather than answering
# "no record", which would read as a normal first day.
#
# ⚠️ `ValueError`, NOT `json.JSONDecodeError`, AND THAT IS
# FCST-05's HIGH. Reading this file has two failure families: the
# BYTES (`OSError`) and the CONTENT - and the content family has at
# least two members, because `open(..., encoding="utf-8")` decodes
# before `json` ever sees a character. `UnicodeDecodeError` is
# `ValueError -> UnicodeError -> UnicodeDecodeError`; it is NOT a
# `json.JSONDecodeError` and it is NOT an `OSError`
# (`issubclass(UnicodeDecodeError, OSError)` is False). So ONE
# non-UTF-8 byte in /data/price_history.json escaped `_load()` ->
# `_open()` -> `PriceHistory.__init__`, which is unguarded at
# main.py's `Controller.__init__`, AND THE ADD-ON DID NOT START.
# Measured: `UnicodeDecodeError: 'utf-8' codec can't decode byte
# 0xff in position 39`. Valid-UTF-8-invalid-JSON, a UTF-8 BOM and a
# non-dict day record all degraded correctly - it was the one byte
# class left uncovered, which is the same class as the defect the
# `_usable_days()` rework was raised for.
#
# The tuple names the FAMILY rather than the two members known
# today, because "the file's content is not readable as JSON" is
# exactly what this handler says, and a third `ValueError` subclass
# nobody has thought of is how this defect arrived the first time.
self.last_error = f"unreadable ({err})"
_LOG.warning("price history at %s is %s - starting from empty. "
"§8.3 carry-forward has nothing to carry until the "
@@ -318,20 +438,41 @@ class PriceHistory:
Dropped, not repaired: an unusable record has nothing in it to keep, and
the next `_remember()` writes a fresh one under that date.
⚠️ AND THE ROWS INSIDE A KEPT RECORD ARE COUNTED, NOT ONLY THE RECORD
(FCST-05). This checked that `rec["points"]` is a LIST and stopped
there, so a record holding one row with a `start` of `7` loaded as
healthy: `last_error=None`, and `words` said `'2026-01-15 (1 h)'` about
a store whose next `_remember()` would raise on that row and go on
raising forever. The rows are NOT dropped here - `_parse_points()` is
the read path's one filter and stays so - but they are COUNTED, so the
store stops reporting health it does not have from the moment it is
opened. `_remember()` drops them on the next refresh and `_save()`
rewrites the file, after which a restart clears the note.
"""
kept, dropped = {}, []
kept, dropped, malformed = {}, [], {}
for key, rec in days.items():
if not isinstance(key, str) or not isinstance(rec, dict) \
or not isinstance(rec.get("points"), list):
dropped.append(str(key))
continue
bad = _bad_rows(rec)
if bad:
malformed[key] = bad
kept[key] = rec
notes = []
if dropped:
notes.append(f"{len(dropped)} unusable day record(s) dropped "
f"({', '.join(sorted(dropped))})")
if malformed:
notes.append(
f"{sum(malformed.values())} malformed point row(s) in "
+ ", ".join(f"{d} ({n})" for d, n in sorted(malformed.items())))
if notes:
# Recorded, not silent: `yesterday()` and `words` both report
# `last_error`, and a cache that quietly discards half a file reads
# to an operator exactly like a normal first day.
self.last_error = (f"{len(dropped)} unusable day record(s) dropped "
f"({', '.join(sorted(dropped))})")
self.last_error = "; ".join(notes)
_LOG.warning("price history at %s: %s - the controller starts "
"anyway and the horizon is as short as the feed makes "
"it", self.path, self.last_error)
@@ -417,6 +558,7 @@ class PriceHistory:
no_data=lambda _r: None)
touched = set()
junked = {}
for p in points:
day = _bdate(p.start)
rec = self._days.setdefault(
@@ -424,14 +566,52 @@ class PriceHistory:
# Rule 4: MERGE by instant. A later read updates the hours it
# carries and leaves the rest; a revised price for an hour already
# held wins, because it is the newer statement about that hour.
#
# ⚠️ EVERY HELD ROW GOES THROUGH `_point_of()` (FCST-05's MEDIUM).
# This was `{q["start"]: q for q in raw if isinstance(q, dict) and
# "start" in q}` sorted by `q["start"]` - a key and a sort key that
# trusted a value read off disk to be a hashable, sortable string.
# An `int`, `null`, a `dict` or a `list` there raised `TypeError`
# out of `carry()` on the day the feed was writing to, and
# `run_tariff_refresh` caught it, so the add-on stayed up while
# EVERY refresh failed for good. A row that is not a point is now
# dropped and reported here, exactly as the read path drops it, and
# `_save()` below writes the record back clean - which is the
# self-heal the raising version could never reach.
#
# ⚠️ AND THE MERGE KEY IS THE VALIDATED INSTANT, NOT ITS SPELLING.
# Aware datetimes hash and compare by instant, so this is rule 4 as
# rule 4 is written. The old string key merged by SPELLING, and the
# old lexicographic sort put the second 02:00 of the 25-hour
# October day (`+01:00`) BEFORE the first (`+02:00`). Only the
# stored order was affected - `_parse_points()` re-sorts by instant
# on the way out - but the store now holds what it says it holds.
raw = rec.get("points")
held = {q["start"]: q for q in (raw if isinstance(raw, list) else ())
if isinstance(q, dict) and "start" in q}
held[p.start.isoformat()] = p.canonical()
rec["points"] = sorted(held.values(), key=lambda q: q["start"])
held = {}
for q in (raw if isinstance(raw, list) else ()):
kept = _point_of(q)
if kept is None:
junked[day] = junked.get(day, 0) + 1
continue
held[kept.start] = q
held[p.start] = p.canonical()
order = sorted(held, key=lambda t: t.astimezone(timezone.utc))
rec["points"] = [held[t] for t in order]
rec["product"], rec["provider"] = product, provider
touched.add(day)
if junked:
# Said, not swallowed. `words` prints `last_error`, so a store that
# has just discarded rows off disk stops reporting plain health -
# and after `_save()` below the file is clean, so a restart clears
# this. `_save()` overwrites it if the write itself fails, which is
# the more urgent fact of the two.
self.last_error = (
f"{sum(junked.values())} malformed point row(s) dropped from "
+ ", ".join(f"{d} ({n})" for d, n in sorted(junked.items())))
_LOG.warning("price history at %s: %s - the record is rewritten "
"without them and the refresh continues",
self.path, self.last_error)
self._prune(stamp)
self._save()
_LOG.debug("price history: %d point(s) filed under %s; %d day(s) held",
@@ -523,15 +703,21 @@ class PriceHistory:
an exception here is an add-on that does not start. It went out that
way: a day record that was a string instead of a dict raised
`AttributeError` from the `len(...)` below. `_load()` now drops such a
record and `_n_points()` is total, so this has two independent reasons
not to fail on a broken file.
record and `_held()` is total, so this has two independent reasons not
to fail on a broken file.
⚠️ AND IT COUNTED POINTS AND CALLED THEM HOURS (FCST-05's LOW). The row
was `f"{d} ({_n_points(rec)} h)"`; on the quarter-hourly feed this
add-on already supports that reads `'2026-01-13 (96 h)'` for a 24-hour
day. Both numbers are printed now - see `_held()` - because they are
two different facts and only one of them was ever being shown.
"""
if not self.ok:
return f"unavailable - {self.last_error}"
if not self._days:
why = f" - {self.last_error}" if self.last_error else ""
return f"empty{why} - nothing to carry forward yet"
held = "; ".join(f"{d} ({_n_points(self._days[d])} h)"
held = "; ".join(_row_of(d, *_held(self._days[d]))
for d in sorted(self._days, key=str))
# A file that PARTLY loaded says so, or the dropped days read as days
# that were simply never written.
@@ -550,21 +736,14 @@ def _parse_points(raw) -> tuple:
return (), "the `points` field is not a list"
out, dropped = [], 0
for row in raw:
if not isinstance(row, dict):
# ⚠️ `_point_of()` IS THE ONE VALIDATOR, and this loop used to BE it.
# The write path had its own, weaker standard; see `_point_of()` for
# what that cost. One standard, two callers, no asymmetry to re-open.
pt = _point_of(row)
if pt is None:
dropped += 1
continue
try:
start = datetime.fromisoformat(row["start"])
value = float(row["eur_per_kwh"])
flag = row["provisional"]
except (KeyError, TypeError, ValueError):
dropped += 1
continue
if start.tzinfo is None or start.utcoffset() is None \
or not isinstance(flag, bool):
dropped += 1
continue
out.append(PricePoint(start=start, eur_per_kwh=value, provisional=flag))
out.append(pt)
out.sort(key=lambda p: p.start.astimezone(timezone.utc))
return tuple(out), (f"{dropped} malformed point(s)" if dropped else "")
goodwe_controller/test_price_forecast.py
+155 -4
View File
@@ -59,6 +59,7 @@ Read MUTATION.md before you report a survivor.
import ast
import inspect
import itertools
import os
import sys
from datetime import datetime, timedelta, timezone
@@ -354,14 +355,164 @@ check("...and still REFUSES a repeated CARRIED hour, which is the DST "
"mispricing that count existed to catch",
raises(lambda: _fc._audit([_fc._carried(D0, 0.11),
_fc._carried(D0, 0.12)], {}), ValueError))
check("...and refuses the SAME published point emitted twice - the provider's "
"points are strictly increasing, so a repeat is never a second print",
check("...and STILL refuses the SAME published point emitted twice - the "
"provider's points are strictly increasing, so a repeat is never a "
"second print. The old hour COUNT already refused this (one point twice "
"is one hour twice); what is graded here is that the relaxation did not "
"lose it, not that it added it",
raises(lambda: _fc._audit([_QUARTERS.points[0], _QUARTERS.points[0]],
_PUB_Q), ValueError))
check("...and refuses a CARRIED point standing in an hour the feed DID "
"publish, which the old count never looked at",
check("...and STILL refuses a CARRIED point standing in an hour the feed DID "
"publish. The old COUNT never looked at it; the old IDENTITY rule did, "
"and raised - so this is preserved behaviour, not new strictness. "
"FCST-04's text called it new here and in `_audit`'s docstring, and a "
"measured differential says otherwise",
raises(lambda: _fc._audit([_fc._carried(D0, 0.11)], _PUB_Q), ValueError))
# ⚠️ AND WHAT THE AUDIT DOES NOT DO, STATED WHERE A READER OF THESE THREE
# CHECKS WOULD OTHERWISE ASSUME IT (FCST-05, MUTATION.md habit 16). `_audit` is
# a PROVENANCE post-condition, not a COMPLETENESS one: it grades every point
# that IS emitted and says nothing about points that are not. So it accepts the
# very decimation FCST-04 was raised for. That is not a regression - FCST-03's
# audit was equally blind - but it means the audit is NOT the thing that stops
# H1 coming back. The count and identity checks above this line are.
check("the audit ACCEPTS the H1 decimation itself - four published points in "
"one hour, only the last emitted - and accepts an EMPTY output. It is a "
"provenance check, not a completeness one, and the checks that guard "
"provenance completeness are the COUNT and IDENTITY ones above",
derived(lambda: _fc._audit([_QUARTERS.points[3]], _PUB_Q)) is None
and derived(lambda: _fc._audit([], _PUB_Q)) is None)
# --------------------------------------------------------------------------- #
# ⚠️ AND THE CLAIM "STRICTER" IS ITSELF A CHECK NOW (FCST-05).
#
# FCST-04 rewrote `_audit` and wrote, in the module and in the three checks
# above, that the result was STRICTER than FCST-03's. It is not. Two of the
# stated reasons were false - the old `seen` count already refused the same
# point emitted twice, and the old IDENTITY rule already refused a carried
# point in a published hour - and a right decision with a fabricated reason is
# a defect in this repository (MUTATION.md, and the reason DECISIONS files
# exist). So the claim is measured here rather than asserted in prose:
# FCST-03's audit is reproduced below from dd40c56 - its RULES verbatim, its
# message TEXT shortened, and nothing here reads a message - and the two are
# run over the whole domain `_audit` can express, in BOTH directions.
#
# THE DOMAIN, AND ITS SIZE. Every published mask over 3 hours (2**3) crossed
# with every output series of length 0-3 drawn from 3 hours x 5 point kinds.
# The nominal cross-product is 28 928; three of those kinds - published,
# published-provisional, copy-of-published - DO NOT EXIST for an hour the feed
# did not publish, and those combinations are SKIPPED rather than faked, which
# leaves 11 990 realisable comparisons. A fabricated "published point" for an
# unpublished hour is not in either audit's domain, and inventing one would
# grade the fixture rather than the code.
#
# ⚠️ THE OLD COPY IS A LITERAL, NOT A CALL. MUTATION.md habit 2: a differential
# whose two sides both come from the shipped module compares a thing against
# itself and moves with every edit. This side is frozen text.
def _audit_fcst03(points, published):
"""FCST-03's `_audit`, verbatim from dd40c56. Its `published` maps a UTC
hour to ONE point, which is exactly the expressiveness this ticket relaxed.
"""
seen = set()
for p in points:
u = _fc._utc_hour(p.start)
if u in seen:
raise ValueError("the filled series repeats " + u.isoformat())
seen.add(u)
src = published.get(u)
if src is None:
if p.provisional is not True:
raise ValueError(u.isoformat() + " is carried and is flagged "
"provisional=" + repr(p.provisional))
else:
if p is not src:
raise ValueError(u.isoformat() + " is published but the series "
"carries a COPY of that point")
def _verdict(fn, pts, pub):
try:
fn(list(pts), pub)
return "accept"
except ValueError:
return "raise"
_DH = [D0 + timedelta(hours=6 + i) for i in range(3)]
_weaker, _stronger, _compared = [], [], 0
for _mask in itertools.product((0, 1), repeat=3):
_pub = {u: PricePoint(start=u, eur_per_kwh=0.10 + i, provisional=False)
for i, (u, _on) in enumerate(zip(_DH, _mask)) if _on}
_old_pub = dict(_pub)
_new_pub = {u: (q,) for u, q in _pub.items()}
# The five shapes an output point can take at a given hour. Three of them
# do not exist for an hour the feed did not publish, and those combinations
# are skipped rather than faked - a fabricated "published point" for an
# unpublished hour is not in either audit's domain.
_kinds = []
for _u in _DH:
_src = _pub.get(_u)
_kinds.append(("published", _src))
_kinds.append(("published-provisional",
None if _src is None else
PricePoint(start=_src.start,
eur_per_kwh=_src.eur_per_kwh,
provisional=True)))
_kinds.append(("carried", _fc._carried(_u, 0.5)))
_kinds.append(("carried-unflagged",
PricePoint(start=_u, eur_per_kwh=0.5,
provisional=False)))
_kinds.append(("copy-of-published",
None if _src is None else
PricePoint(start=_src.start,
eur_per_kwh=_src.eur_per_kwh,
provisional=_src.provisional)))
for _n in range(4):
for _combo in itertools.product(_kinds, repeat=_n):
if any(k[1] is None for k in _combo):
continue
_pts = [k[1] for k in _combo]
_compared += 1
_o = _verdict(_audit_fcst03, _pts, _old_pub)
_nw = _verdict(_fc._audit, _pts, _new_pub)
if _o == "raise" and _nw == "accept":
_weaker.append((_mask, [k[0] for k in _combo]))
if _o == "accept" and _nw == "raise":
_stronger.append((_mask, [k[0] for k in _combo]))
# ⚠️ THE PRECONDITION FIRST (MUTATION.md habit 1). A differential over a
# domain that cannot reach the two disputed cases would score 0-0 for the wrong
# reason and read exactly like this one, so the two cases are driven against
# the OLD audit by name before the totals are graded.
_probe = PricePoint(start=_DH[0], eur_per_kwh=0.1, provisional=False)
check(f"the differential really covers the two claims it is here to settle, "
f"or it is {_compared} runs of nothing: FCST-03's own audit raises on "
f"BOTH 'the same published point twice' and 'a carried point in a "
f"published hour', which is what FCST-04 said it did not",
_compared > 3000
and _verdict(_audit_fcst03, [_probe, _probe], {_DH[0]: _probe}) == "raise"
and _verdict(_audit_fcst03, [_fc._carried(_DH[0], 0.5)],
{_DH[0]: _probe}) == "raise")
check(f"...and over all {_compared} of them the reworked audit is NOT WEAKER "
f"than FCST-03's: {len(_weaker)} case(s) the old one raised and this one "
f"accepts {_weaker[:2]}",
_weaker == [])
check(f"...and NOT STRONGER either: {len(_stronger)} case(s) the old one "
f"accepted and this one raises {_stronger[:2]}. FCST-04's docstring and "
f"two of these checks said it was stricter and named two reasons, and "
f"BOTH were already refused by the old count and the old identity rule. "
f"The only behavioural change is the RELAXATION below, which is what the "
f"rework was for",
_stronger == [])
_QP = [PricePoint(start=_DH[0] + timedelta(minutes=15 * i),
eur_per_kwh=0.1 + i, provisional=False) for i in range(4)]
check("...and the relaxation is real and is NOT expressible to the old audit: "
"four distinct published points inside one hour, which FCST-03's count "
"refused as a repeated hour and this one accepts",
_verdict(_fc._audit, _QP, {_DH[0]: tuple(_QP)}) == "accept"
and _verdict(_audit_fcst03, _QP, {_DH[0]: _QP[0]}) == "raise")
# ⚠️ THE HORIZON IS WHOLE HOURS AT BOTH ENDS (FCST-04 L8). `since` is floored
# and `until` was not, so a 48 h request made at any instant off the hour came
# back 49 hours long. Graded at an OFF-THE-HOUR instant, because on the hour
goodwe_controller/test_price_history.py
+305 -6
View File
@@ -34,6 +34,18 @@ cache cannot decay by being fed its own output.
`PriceHistory(...)` and `.words` are unguarded in `Controller.__init__`
- so the add-on did not start, and never self-healed.
AC5 (FCST-05) THE CACHE'S OWN CONTRACT HOLDS FOR EVERY BYTE AND EVERY ROW.
A file holding one non-UTF-8 byte no longer stops the add-on from
starting - `_load()` caught `(json.JSONDecodeError, OSError)` and
`UnicodeDecodeError` is neither, so it escaped `PriceHistory.__init__`
and `Controller.__init__` does not guard it. And a malformed `start`
under the day the feed is WRITING to no longer kills the tariff plan
for good: the write path shares `_point_of()` with the read path,
drops the row, says so, and `_save()` rewrites the record clean.
⚠️ THE STATUS ROW IS PART OF THIS. It said `'2026-01-15 (1 h)'` about a
store whose every refresh was raising - health it did not have, in a
unit that was not its own (it counts POINTS).
THE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
* EVERY VERDICT IS DERIVED. The carried/published split is re-derived here
@@ -262,6 +274,13 @@ YESTERDAY_MWH = {h: 400.0 + 3 * h for h in range(24)}
MARKUP, TAXES = 0.03, 0.06
# ⚠️ FCST-05's HIGH, AS BYTES. A cache that is structurally valid JSON up to
# the point where one byte is not UTF-8 - which is what a truncated write, a
# half-flushed page or a foreign writer produces. Written with `open(..., "wb")`
# and never through a text handle, or the fixture cannot hold the byte it is
# named for. `0xff` is not a valid UTF-8 start byte in any position.
_UTF8_BOMB = b'{"version": 1, "days": {"2026-01-14": "\xff\xfe"}}'
def allin(mwh):
"""EUR/MWh as published -> the all-in EUR/kWh `assemble()` produces."""
@@ -896,6 +915,55 @@ check("an unreadable file: the reason is kept and reported, and the record is "
and _h6.yesterday(NOW).kind == "no_data"
and "unreadable" in why(_h6.yesterday(NOW)))
# ⚠️ AND THE BYTE CLASS THAT FIXTURE CANNOT REACH - FCST-05's HIGH. The row
# above is INVALID JSON but it is PERFECTLY VALID UTF-8, so it is decoded and
# then refused by `json`, which raises `json.JSONDecodeError`. A file holding
# one byte that is not UTF-8 at all fails EARLIER, in the decode `open(...,
# encoding="utf-8")` performs before `json` sees a character, and
# `UnicodeDecodeError` is `ValueError -> UnicodeError -> UnicodeDecodeError`:
# NOT a `JSONDecodeError`, and NOT an `OSError` either. `_load()` caught
# `(json.JSONDecodeError, OSError)`, so it escaped `_load()` -> `_open()` ->
# `PriceHistory.__init__` - unguarded in `Controller.__init__` - AND THE ADD-ON
# DID NOT START, on a cache whose own contract says "a corrupt cache must not
# stop the controller".
#
# The precondition is asserted first, because it is the whole reason the check
# above could not see this: if `"{ this is not json"` were undecodable, the two
# fixtures would be one fixture and this would grade nothing (MUTATION.md
# habit 1).
check("the two unreadable-file fixtures are DIFFERENT FAILURES: the one above "
"is valid UTF-8 that is not JSON, and the one below is not UTF-8 at all. "
"A file that is merely bad JSON cannot reach the decode step, which is "
"why the suite had this hole",
open(_p6, "rb").read().decode("utf-8") == "{ this is not json"
and raises(lambda: _UTF8_BOMB.decode("utf-8"), UnicodeDecodeError))
_d6b = tmpdir()
_p6b = os.path.join(_d6b, "price_history.json")
open(_p6b, "wb").write(_UTF8_BOMB)
_h6b = derived(lambda: PriceHistory(_p6b))
check(f"a single non-UTF-8 byte in the cache does NOT stop the store from "
f"being constructed {_h6b!r}. `PriceHistory(...)` is unguarded in "
f"`Controller.__init__`, so an exception here is an add-on that does not "
f"start - which is what shipped",
isinstance(_h6b, PriceHistory))
check(f"...and it degrades the SAME WAY the bad-JSON file does: the reason is "
f"kept, named and reported "
f"{(_h6b.last_error if isinstance(_h6b, PriceHistory) else _h6b)!r}",
isinstance(_h6b, PriceHistory)
and "unreadable" in (_h6b.last_error or "")
and "codec can't decode byte" in (_h6b.last_error or "")
and _h6b.yesterday(NOW).kind == "no_data"
and "unreadable" in why(_h6b.yesterday(NOW)))
_w6b = derived(lambda: _h6b.words)
_plan6b = derived(lambda: run(resolve_tariff(
CFG, NOW, _until, transport=transport_for(_body_today),
now=NOW, history=_h6b)).origin)
check(f"...and NOTHING downstream raises on it either: not `words` ({_w6b!r}), "
f"not the shipped `resolve_tariff()` path ({_plan6b!r}). §6: absent "
f"machinery must not disable the loop, and a JSON file is machinery",
isinstance(_w6b, str) and _plan6b == "provider")
_p7 = os.path.join(tmpdir(), "price_history.json")
json.dump({"version": SCHEMA + 99, "days": {}}, open(_p7, "w", encoding="utf-8"))
_h7 = PriceHistory(_p7)
@@ -1008,11 +1076,49 @@ check("a healthy but EMPTY store says something different, because it is a "
and words_of(_h_words).startswith("empty")
and "nothing to carry forward" in words_of(_h_words))
_ph.PriceHistory._remember(_h_words, series_of(YDAY_H, YESTERDAY_MWH), NOW)
check(f"...and a store holding a day names the DATE and the HOUR COUNT "
f"{words_of(_h_words)!r} - 'a cache exists' and 'the cache holds "
f"yesterday, all 24 hours of it' are different facts, and only the "
f"second says the horizon will be full tomorrow morning",
words_of(_h_words) == f"{YDAY_D} (24 h)")
check(f"...and a store holding a day names the DATE, the POINT COUNT and the "
f"HOUR COUNT {words_of(_h_words)!r} - 'a cache exists' and 'the cache "
f"holds yesterday, all 24 hours of it' are different facts, and only "
f"the second says the horizon will be full tomorrow morning",
words_of(_h_words) == f"{YDAY_D} (24 points over 24 h)")
# ⚠️ AND THE TWO NUMBERS ARE NOT THE SAME NUMBER - FCST-05's LOW. The row
# shipped as `f"{d} ({_n_points(rec)} h)"`, which counts POINTS and prints them
# under the HOUR's name. On an hourly feed the two are equal and nothing shows;
# on the quarter-hourly feed this add-on already supports (`_published_by_hour`
# keeps a TUPLE per hour precisely for it, and §7 below drives one) the row
# read '2026-01-13 (96 h)' - 96 "h" for a 24-hour day. That is the points/hours
# conflation H1 was raised for, on the row H1's own ticket shipped.
#
# MUTATION.md's rule: this check asserts a DIFFERENCE, so the fixture is run
# with the two numbers actually made different. 96 != 24, asserted first.
_h_q = store()
_QUARTERS = [u + timedelta(minutes=15 * q) for u in YDAY_H for q in range(4)]
_ph.PriceHistory._remember(
_h_q, PriceRead.series(
[PricePoint(start=brussels(u),
eur_per_kwh=allin(YESTERDAY_MWH[brussels(u).hour]
+ brussels(u).minute),
provisional=False) for u in _QUARTERS],
PriceProduct.DAY_AHEAD, "generic_dayahead", NOW),
NOW)
_qw = words_of(_h_q)
check(f"the quarter-hourly fixture really is sub-hourly and its two counts "
f"really differ ({len(_QUARTERS)} points, "
f"{len({brussels(u).hour for u in _QUARTERS})} local hours), or 'points' "
f"and 'hours' are the same observation and the row grades nothing",
len(_QUARTERS) == 96
and len({brussels(u).hour for u in _QUARTERS}) == 24
and len({u.minute for u in _QUARTERS}) == 4)
check(f"...and the row reports BOTH, each under its own unit: {_qw!r}. "
f"A day of 96 quarter-hourly prints is 96 points over 24 h, and the "
f"figure that says whether tomorrow's fill has a whole day to carry is "
f"the HOUR one",
_qw == f"{YDAY_D} (96 points over 24 h)")
check(f"...and specifically the row does NOT say '96 h' {_qw!r} - a point "
f"count wearing the hour's name is H1's defect, and this row is where "
f"an operator would have read it",
isinstance(_qw, str) and "(96 h)" not in _qw and " (96 points " in _qw)
check(f"a file that only PARTLY loaded says so in the row {_w14!r}, or the "
f"days it dropped read as days that were simply never written",
isinstance(_w14, str) and (_h14.last_error or "") in _w14
@@ -1023,7 +1129,7 @@ check("...and `words` never raises, whatever the file holds. It is read once "
"from Controller.__init__ and again on every status render, and neither "
"call site guards it - an exception here is an add-on that does not start",
all(isinstance(words_of(h), str)
for h in (_missing, _h6, _h7, _h14, _h_words)))
for h in (_missing, _h6, _h6b, _h7, _h14, _h_words)))
# ⚠️ AND A FAILED WRITE LEAVES NOTHING BEHIND. `_save()` writes to a `mkstemp`
# and renames - atomic, so a power cut cannot leave a truncated file that
@@ -1067,6 +1173,7 @@ check(f"...and a SUCCESSFUL write leaves exactly the one file "
# provider series with no invented points and no §11.4 degradation.
for _label, _hist in (("no durable volume", _missing),
("unreadable file", _h6),
("a non-UTF-8 byte in the file", _h6b),
("wrong schema", _h7)):
_plan = run(resolve_tariff(CFG, NOW, _until,
transport=transport_for(_body_today), now=NOW,
@@ -1080,6 +1187,198 @@ for _label, _hist in (("no durable volume", _missing),
and not any(p.provisional for p in _plan.read.points))
# --------------------------------------------------------------------------- #
print()
print("5c. FCST-05 - the WRITE path is as defensive as the read path")
# THE DEFECT. `_remember()` merged a day's held points with
#
# held = {q["start"]: q for q in raw
# if isinstance(q, dict) and "start" in q}
# rec["points"] = sorted(held.values(), key=lambda q: q["start"])
#
# which trusts a value read off /data to be a HASHABLE, SORTABLE STRING. It is
# whatever is in the file. `_usable_days()` required `rec["points"]` to be a
# LIST and never looked at the elements, and `_parse_points()` - the READ path -
# was properly defensive, so `yesterday()` survived and the §5 `_p8` fixture
# passed. Two standards for what a stored point is; the weaker one on the path
# that WRITES.
#
# ⚠️ AND THE PLACEMENT IS THE WHOLE FIXTURE. `_p8` files its malformed rows
# under a day the feed is NOT writing to, so `_remember()` never touches that
# record and the defect is invisible. What reaches it is a malformed row under
# TODAY - the record the running feed merges into on every refresh - so that is
# asserted before anything is graded (MUTATION.md habit 1), and the harmless
# placement is run as the accepted baseline it varies from (habit 7).
#
# ⚠️ AND THE BLAST RADIUS IS WHY IT IS NOT MERELY UNTIDY. `carry()` is reached
# from `resolve_tariff()` from `run_tariff_refresh`, which catches `Exception`:
# the add-on stays UP and EVERY refresh raises, forever. The plan retires after
# TARIFF_MAX_AGE_S and arbitrage and the cheap window are off permanently, with
# no self-heal, because `_remember()` raises before it reaches `_save()`.
_BAD_STARTS = (
("an int", 7), # TypeError: '<' between 'str' and 'int'
("a dict", {"a": 1}), # TypeError: unhashable as a dict key
("null", None), # TypeError: '<' between 'str', 'NoneType'
("a list", [1, 2]), # TypeError: unhashable as a dict key
("a string that is not an instant", "nope"), # sorts fine, never a point
)
def _seeded(day, bad_start):
"""A store on disk holding ONE malformed row under `day`, plus a good one."""
d = tmpdir()
path = os.path.join(d, "price_history.json")
good = {"start": brussels(at_local(day, 5)).isoformat(),
"eur_per_kwh": allin(YESTERDAY_MWH[5]), "provisional": False}
json.dump({"version": SCHEMA, "days": {day: {
"product": "dynamic_dayahead", "provider": "generic_dayahead",
"points": [good, {"start": bad_start, "eur_per_kwh": 0.2,
"provisional": False}]}}},
open(path, "w", encoding="utf-8"))
return path, PriceHistory(path)
_TODAY_FEED = series_of(TODAY_H, TODAY_MWH)
check(f"the fixture files its malformed row under {TODAY_D}, which is exactly "
f"the Brussels day the feed being carried writes to "
f"{sorted({_ph._bdate(p.start) for p in _TODAY_FEED.points})} - under any "
f"other day `_remember()` never opens the record and this whole section "
f"grades nothing, which is why the suite already had this hole",
{_ph._bdate(p.start) for p in _TODAY_FEED.points} == {TODAY_D})
# The accepted baseline: the SAME row, one day over. Shipped and broken alike
# pass this, and that is the point of running it (habit 7).
_pB, _hB = _seeded(YDAY_D, 7)
_cB = derived(lambda: _hB.carry(_TODAY_FEED, since=NOW, until=_until, stamp=NOW))
check("baseline: the same malformed row under a day the feed is NOT writing "
"to is carried without incident - one variable moves between this and "
"every check below, and it is the DAY",
not isinstance(_cB, _Raised) and _cB.kind == "series")
for _shape, _bad in _BAD_STARTS:
_pM, _hM = _seeded(TODAY_D, _bad)
_wload = words_of(_hM)
_c1 = derived(lambda h=_hM: h.carry(_TODAY_FEED, since=NOW, until=_until,
stamp=NOW))
_saw = repr(_c1) if isinstance(_c1, _Raised) else _c1.kind
check(f"a `start` that is {_shape} ({_bad!r}) under TODAY's own record "
f"does not take the refresh down: `carry()` returns a series "
f"({_saw}). It raised "
f"`TypeError` out of the shipped path, and `run_tariff_refresh` "
f"catches `Exception`, so the add-on stayed up while every refresh "
f"failed for good",
not isinstance(_c1, _Raised) and _c1.kind == "series")
# ⚠️ THE VERDICT IS NOT THE CLAIM. "It did not raise" is satisfied by a
# store that silently kept the poison row; what this section promises is
# that the row is GONE FROM THE FILE, so the next start is clean.
_disk = json.load(open(_pM, encoding="utf-8"))
_rows = _disk["days"][TODAY_D]["points"]
# DERIVED, not a constant: the surviving rows are exactly the DISTINCT
# instants the feed published for that day plus the fixture's one good
# seeded row - which is 05:00, an hour the feed also prints, so it merges.
_want = len({p.start.astimezone(UTC) for p in _TODAY_FEED.points}
| {at_local(TODAY_D, 5).astimezone(UTC)})
check(f"...and the row is DROPPED FROM /data, not carried: {len(_rows)} "
f"row(s) on disk against {_want} distinct instants, every one of "
f"them a point. `_remember()` used to raise BEFORE `_save()`, so "
f"there was no self-heal at all - the store came back poisoned on "
f"every restart",
all(_ph._point_of(r) is not None for r in _rows)
and len(_rows) == _want)
# ...and the store stops reporting health, at BOTH moments it can be read:
# when it is opened, and after the refresh that cleaned it.
# ⚠️ THE COUNTS ARE ASSERTED TOO, NOT ONLY THE NOTE. The seeded record holds
# ONE point and one row that is not one, so a row printing "2 points" would
# be reporting the poison as held prices - which is the half of this defect
# that reached the operator: `words` said "2026-01-15 (1 h)" while every
# tariff refresh was dying on that record.
check(f"...and the status row does not call that store healthy - at open "
f"{_wload!r}: ONE point held, and the row that is not a point named "
f"rather than counted",
isinstance(_wload, str)
and _wload.startswith(f"{TODAY_D} (1 point over 1 h) [")
and "1 malformed point row(s) in" in _wload)
# ⚠️ AND THE TWO NOTES ARE GRADED APART, BECAUSE THEY MASK EACH OTHER
# OTHERWISE (MUTATION.md TEST-16). `_usable_days()` records what it FOUND
# at open and `_remember()` records what it DROPPED on the refresh; both
# land in `last_error`, so a check asking only "does the row mention a
# malformed point" is answered by whichever rung is still standing - the
# first draft of this check was exactly that, and deleting `_remember()`'s
# note left the whole 29-file suite green at -1 B. The tense is the
# difference an operator reads: "in" is still there, "dropped from" is
# gone, and after the refresh only the second is true.
_wafter = words_of(_hM)
check(f"...nor after the refresh that dropped it {_wafter!r}. It said "
f"'{TODAY_D} (1 h)' - the status row reporting a full cache while "
f"every tariff refresh was dying on it",
isinstance(_wafter, str)
and "malformed point row(s) dropped from" in _wafter
and TODAY_D in _wafter)
check(f"...and the row MOVED from what was found to what was dropped "
f"{_wload.split('[')[-1]!r} -> {_wafter.split('[')[-1]!r}, so the "
f"two notes are not one note read twice",
isinstance(_wload, str) and isinstance(_wafter, str)
and "malformed point row(s) in" in _wload
and "malformed point row(s) in" not in _wafter
and "malformed point row(s) dropped from" not in _wload)
# ...and a SECOND refresh over the same store is clean, which is the state
# a restarted add-on comes up in.
_hM2 = PriceHistory(_pM)
check(f"...and the store reopened from that file is CLEAN: the whole day "
f"held and nothing to report {words_of(_hM2)!r}, so the note clears "
f"itself rather than becoming permanent furniture",
words_of(_hM2) == f"{TODAY_D} ({_want} points over {_want} h)"
and _hM2.last_error is None)
# ...and the shipped path, end to end, through the caller that catches
# `Exception` and would otherwise have hidden this for good.
_pP, _hP = _seeded(TODAY_D, 7)
_o1 = derived(lambda: run(resolve_tariff(
CFG, NOW, _until, transport=transport_for(_body_today), now=NOW,
history=_hP)).origin)
_o2 = derived(lambda: run(resolve_tariff(
CFG, NOW, _until, transport=transport_for(_body_today), now=NOW,
history=_hP)).origin)
check(f"the SHIPPED `resolve_tariff()` path survives it TWICE ({_o1!r}, "
f"{_o2!r}) - once for the refresh that finds the poison and once for "
f"the one after it. A guard that refuses once and dies on the next call "
f"is a delay, not a guard (MUTATION.md habit 9)",
_o1 == "provider" and _o2 == "provider")
# ⚠️ AND THE MERGE KEY IS THE INSTANT, WHICH IS WHAT RULE 4 SAYS. The old key
# was `q["start"]` - the SPELLING - so the same instant written at two offsets
# was held twice, and the lexicographic sort beside it put the second 02:00 of
# the 25-hour October day (`+01:00`) BEFORE the first (`+02:00`). Aware
# datetimes hash and compare by instant, so the validated `start` is the key.
_d15 = tmpdir()
_p15 = os.path.join(_d15, "price_history.json")
_alt = at_local(YDAY_D, 6)
json.dump({"version": SCHEMA, "days": {YDAY_D: {
"product": "dynamic_dayahead", "provider": "generic_dayahead",
"points": [{"start": _alt.astimezone(UTC).isoformat(),
"eur_per_kwh": 0.999, "provisional": False}]}}},
open(_p15, "w", encoding="utf-8"))
_h15 = PriceHistory(_p15)
check(f"the two spellings really are one instant and really are two strings "
f"({_alt.astimezone(UTC).isoformat()!r} vs "
f"{brussels(_alt).isoformat()!r}), or 'merged by instant' and 'merged by "
f"spelling' are the same observation",
_alt.astimezone(UTC).isoformat() != brussels(_alt).isoformat()
and _alt.astimezone(UTC) == brussels(_alt))
_ph.PriceHistory._remember(
_h15, series_of([_alt], YESTERDAY_MWH), NOW)
_y15 = _h15.yesterday(NOW)
_n15 = len(_y15.points) if _y15.kind == "series" else _y15.kind
_v15 = _y15.points[0].eur_per_kwh if _y15.kind == "series" else None
check(f"...and a revision of that hour REPLACES it rather than sitting beside "
f"it: {_n15} point held, priced {_v15} and not 0.999. Rule 4 says "
f"'MERGE BY INSTANT'; the key was the string",
_y15.kind == "series" and len(_y15.points) == 1
and abs(_y15.points[0].eur_per_kwh - allin(YESTERDAY_MWH[6])) < 1e-9)
# --------------------------------------------------------------------------- #
print()
print("6. rule 4 - hours accumulate within a day; they do not replace it")