FCST-06: "cannot persist" becomes one question, asked once, about the outcome

price_history.py's docstring states its rule unconditionally - "A store that
cannot persist now records nothing, and both of its answers say the same
thing" - and FCST-04 enforced it against `not self.ok` only, which is one
SYMPTOM: the durable volume is missing. `ok` is about the DIRECTORY, and a
read-only remount and a full disk both leave it True. Reproduced on 5059e28
against two real volumes, a /data bind-mounted read-only and a full tmpfs:

    last_error = 'could not write ([Errno 30] Read-only file system)'
    words      = '2026-01-14 (24 points over 24 h); 2026-01-15 (24 points
                  over 24 h) [could not write (...)]'
    yesterday()= series, 24 points
    ON DISK    = {}

`words` and `yesterday()` were describing two different stores, and the 24
points were gone on the next restart with nothing saying so - the
silently-forgetting cache that rule exists to refuse, reached by the condition
it did not name. No shipped path reaches it: /data is the Supervisor's durable
volume and this module is its only writer. The defect is that a guarantee the
module states about itself did not hold in one condition.

Route taken: `_remember()` refuses to retain what it could not write, and the
docstring's claim stands as written. It is now ONE derived predicate rather
than two places testing two symptoms. "Can this store persist?" cannot be
answered BEFORE the attempt - a directory that existed and was writable a
moment ago is not a write that completed - so it is answered after it, once,
by the only thing holding the evidence: `_save()` returns whether the file now
holds what memory holds. No volume is one False, an OSError is another.
`_remember()` keeps only what that one answer says was kept, and the
`not self.ok` test in `_remember()` is GONE rather than joined by a second one.
FCST-04's M4 and FCST-05's HIGH were the same shape, so a third cause now needs
no third branch.

What is undone is the MERGE, not the store. The state restored is the last one
that IS on disk, so a site whose volume fills at 14:00 goes on answering the
yesterday it wrote at 13:30 - that record survives a restart, so answering it
is not a promise the store cannot keep. It stops adding to it. The snapshot is
copy.deepcopy: a one-level copy shares the day record the merge rebinds
`points` on, and measured with one, a store holding 12 persisted points
answered 24 after a failed merge.

Not fatal, deliberately: §6 and this module's opening - a broken cache must not
be a broken controller, and an add-on that will not start because /data filled
up is a worse defect than this one. `_save()` reports and returns; the shipped
resolve_tariff() path still gives a provider plan with no §11.4 degradation.
What changed is that the failure is now acted on rather than only logged.

test_price_history.py section 5d drives it on two REAL failures with different
syscalls and different errnos - a directory where the file goes (os.replace,
EISDIR) and a volume moved out from under an open store (mkstemp, ENOENT) -
because the rule is enforced on the outcome and one cause would grade a
coincidence. DOCS.md's "where there is no /data" sentence is widened to the
condition the code now holds to.

MUTATION.md gains what M4 exposed: a mutation whose red is a TRACEBACK reports
however many checks ran before it and grades none after it. M4 first printed
1 FAIL; the true figure is 3. An f-string operand read `len(read.points)`, and
`.points` on a no_data raises `Absence` by design - why()'s rule arriving in
what a check PRINTS rather than in what it asserts. `npts()` is now beside
why(), and a red count is a lower bound until the run is seen to reach the end
of its section.

6 mutations, purged tree, every one RED on named checks, no survivors:
M1 self._days = before -> dict(self._days)          +10 B  9 red
M2 _save() OSError handler returns True             -1 B   9 red
M3 _save() returns True on `not self.ok`            -1 B   3 red
M4 self._days = before -> self._days = {}           -4 B   3 red
M5 copy.deepcopy -> dict                            -9 B   3 red
M6 `if not self.ok: return` back in _remember()     +43 B  1 red

All 29 test files green. Nothing deployed; no option added; config.yaml and
app/ behaviour on a healthy /data untouched. No version number picked - that
is the open decision the Unreleased block names.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU

goodwe_controller/CHANGELOG.md

@@ -13,6 +13,146 @@ if it has, it needs 0.10.0. Whoever deploys decides.
 | | |
 |---|---|
 | _this branch_ | FCST-04 - the carry-forward fill reaches the plan, and the day it carries survives a restart |
+| _this branch_ | FCST-05 - one byte stops stopping the add-on, and the write path gets the read path's standard |
+| _this branch_ | FCST-06 - "cannot persist" stops meaning "has no /data" and starts meaning what it says |
+
+### FCST-06 - the second way to be a store that cannot persist
+
+**A price cache that could not write went on answering a yesterday the file did
+not contain.** `price_history.py`'s docstring states its rule unconditionally -
+*"A store that cannot persist now records nothing, and both of its answers say
+the same thing"* - and FCST-04 enforced it against `not self.ok`, which is one
+**symptom**: the durable volume is missing. `ok` is about the DIRECTORY, and a
+read-only remount and a full disk both leave it `True`. Reproduced on `5059e28`
+against two real volumes - a `/data` bind-mounted read-only and a full tmpfs:
+
+| | with a `/data` that cannot be written |
+|---|---|
+| `last_error` | `could not write ([Errno 30] Read-only file system)` |
+| `words` | `2026-01-14 (24 points over 24 h); 2026-01-15 (24 points over 24 h) [could not write (...)]` |
+| `yesterday()` | **series, 24 points** |
+| on disk | `{}` |
+
+`words` and `yesterday()` were describing two different stores, and the 24
+points were gone on the next restart with nothing saying so - the
+silently-forgetting cache that rule exists to refuse, reached by the condition
+it did not name. **No shipped path reaches it**: `/data` is the Supervisor's
+durable volume and this module is its only writer. The defect is that a
+guarantee the module states about itself did not hold in one condition.
+
+**"Cannot persist" is now ONE derived predicate, asked once, about the
+outcome.** It cannot be answered *before* the attempt - a directory that existed
+and was writable a moment ago is not a write that completed - so `_save()`
+returns whether the file now holds what memory holds: `False` with no durable
+volume, `False` on an `OSError` from the write, `True` otherwise. `_remember()`
+keeps only what that one answer says was kept, and **the `not self.ok` test in
+`_remember()` is gone** rather than joined by a second one. FCST-04's M4 and
+FCST-05's HIGH were the same shape - a rule enforced on the conditions somebody
+thought of - so a third cause now needs no third branch.
+
+**What is undone is the MERGE, not the store.** The state restored is the last
+one that *is on disk*, so a site whose volume fills at 14:00 goes on answering
+the yesterday it wrote at 13:30 - that record survives a restart, so answering
+it is not a promise the store cannot keep. It simply stops adding to it. The
+snapshot is a `copy.deepcopy`: a one-level copy shares the day record the merge
+rebinds `points` on, and measured with one, a store holding 12 persisted points
+answered **24** after a failed merge.
+
+⚠️ **A failed write is still not fatal, and that was a constraint rather than an
+oversight.** §6 and this module's opening: a broken cache must not be a broken
+controller. An add-on that will not start because `/data` filled up is a worse
+defect than the one being fixed, so `_save()` reports and returns - the failure
+stays in the log, in `last_error`, in `words` and in `yesterday()`'s reason -
+and the shipped `resolve_tariff()` path still returns a provider plan with no
+§11.4 degradation. What changed is that it is now *acted on*.
+
+`test_price_history.py` section 5d drives it on two real failures with different
+syscalls and different errnos - a directory where the file goes (`os.replace`,
+EISDIR) and a volume that moves out from under an open store (`mkstemp`,
+ENOENT) - because the rule is enforced on the outcome and one cause would grade
+a coincidence. See `DECISIONS/FCST-06-cannot-persist-is-one-question.md`.
+
+### FCST-05 - what FCST-04's own contract still did not hold
+
+**A single non-UTF-8 byte in `/data/price_history.json` stopped the add-on from
+starting.** `_load()` caught `(json.JSONDecodeError, OSError)`.
+`UnicodeDecodeError` is `ValueError -> UnicodeError -> UnicodeDecodeError` -
+neither of those, and `issubclass(UnicodeDecodeError, OSError)` is **False** -
+so it escaped `_load()` -> `_open()` -> `PriceHistory.__init__`, which
+`Controller.__init__` does not guard. Reproduced: `UnicodeDecodeError: 'utf-8'
+codec can't decode byte 0xff`, and **the controller did not come up**. That is the same class as the day-record defect the store was reworked
+for, in the one byte class the rework left uncovered: valid-UTF-8-invalid-JSON,
+a UTF-8 BOM and a non-dict day record all degraded correctly. The suite could
+not see it because its "unreadable file" fixture writes `"{ this is not json"`,
+which is **valid UTF-8**. The handler now names the `ValueError` family - the
+file's CONTENT - beside `OSError`, the file's BYTES, and a fixture holding a
+genuinely undecodable byte is in `test_price_history.py` section 5.
+
+**A malformed `start` under the day the feed is writing to killed the tariff
+plan permanently.** `_remember()` merged a day's points with
+`{q["start"]: q ...}` and sorted on `q["start"]`, trusting a value read off
+`/data` to be a hashable, sortable string. An `int`, `null`, a `dict` or a
+`list` there raised `TypeError`; `run_tariff_refresh` catches `Exception`, so
+the add-on stayed **up** while **every refresh raised, forever** - the plan
+retiring after `TARIFF_MAX_AGE_S` and arbitrage and the cheap window off for
+good, with no self-heal because `_remember()` raised before reaching `_save()`.
+The read path (`_parse_points()`) was properly defensive throughout, which is
+why `yesterday()` survived and the existing fixture passed: it files its
+malformed rows under a day the feed is **not** writing to, so `_remember()`
+never opened that record. **The two paths now share one validator**,
+`_point_of()`, and neither has a standard of its own. A row that is not a point
+is dropped and reported, and `_save()` rewrites the record clean.
+
+⚠️ **AND THE STATUS ROW SAID THE STORE WAS HEALTHY WHILE THE PLAN WAS DEAD.**
+`words` read `'2026-01-15 (1 h)'` about the record that was killing every
+refresh. `_load()` now counts the rows inside a kept record as well as the
+record itself, so the store stops reporting plain health from the moment it is
+opened; the note names what was **found** at open and what was **dropped** by
+the refresh, and clears itself once the file has been rewritten.
+
+⚠️ **AND THE ROW COUNTED POINTS AND CALLED THEM HOURS.** `f"{d} ({_n_points(d)}
+h)"` on the quarter-hourly feed this add-on already supports reads
+`'2026-01-13 (96 h)'` for a 24-hour day - the same points/hours conflation the
+sub-hourly defect above was about, on the row that defect's own card shipped.
+The cell is now `2026-01-13 (96 points over 24 h)`: two facts, each under its
+own unit, and the hour figure is the one that says whether tomorrow morning's
+fill has a whole day to carry.
+
+⚠️ **AND THE MERGE KEY IS THE INSTANT RATHER THAN ITS SPELLING**, which is what
+rule 4 already said. Aware datetimes hash and compare by instant. The string
+key merged by spelling, and the lexicographic sort beside it put the second
+02:00 of the 25-hour October day (`+01:00`) **before** the first (`+02:00`);
+only the stored order was affected, because `_parse_points()` re-sorts by
+instant on the way out.
+
+⚠️ **TWO STATED REASONS WERE FALSE, AND THIS REPOSITORY TREATS THAT AS A
+DEFECT.** `price_forecast._audit()`'s "WHY IT IS STRICTER" section claimed the
+rework refused two things FCST-03's audit did not. A differential over the
+whole domain the audit can express - every published mask over 3 hours crossed
+with every output series of length 0-3 - found **zero divergence in both
+directions**: the reworked audit is exactly **EQUAL** to FCST-03's, not
+stronger. The old `seen` count already refused the same published point emitted
+twice (one point twice is one hour twice), and the old **identity** rule
+already refused a carried point sharing a published hour. The only behavioural
+change is the intended **relaxation** - several distinct published points in
+one hour - and that stays. The docstring and the two check texts now say what
+is true, and the differential is committed as a check in
+`test_price_forecast.py` rather than left as prose.
+
+⚠️ **AND `_audit` IS NOT WHAT STOPS THE DECIMATION DEFECT RECURRING.** It does
+not check **completeness**: fed the decimation directly - four published points
+in an hour, output emitting only the last - it **accepts**, and it accepts an
+empty output. That is a limit, not a regression, and the module docstring said
+otherwise by presenting the audit as the post-condition guarding provenance.
+What actually guards it is the point **count** and the pass-through **identity**
+checks in `test_price_forecast.py` and `test_price_history.py`, and the text now
+points at them. It is **not** made complete: `published` is the whole read while
+the fill emits only `[since, until)`, so "every published point appears" is not
+a true post-condition of those two arguments. Measured - the rule as stated
+takes the 1 h, 12 h and 48 h horizon checks red on legitimate series.
+
+**No option was added, `config.yaml` is untouched, and nothing was deployed.**
+
 
 **FCST-03 shipped the fill and no caller.** `price_forecast.fill_from_yesterday()`
 was complete and checked, and so were the consumers written for it -
@@ -58,8 +198,9 @@ spread it would have created was never compared against anything.
   no durable volume passes the explicit `NoHistory()`. A default of `None`
   meaning "skip the fill" would hand a caller who forgot the argument exactly
   the truncated horizon this card removes, silently.
-* A new status row, **Price history**, names the days held and their hour
-  counts.
+* A new status row, **Price history**, names the days held, how many points
+  each holds and how many whole hours they cover (see FCST-05: it shipped
+  counting points and printing them under the hour's name).
 
 ⚠️ **ONE CHANGE HERE MOVES ENERGY, AND IT IS NOT THE ONE THE CARD IS NAMED FOR.**
 Carried hours are provisional, so §8.3 keeps them out of the trade set: the fill

goodwe_controller/DECISIONS/FCST-06-cannot-persist-is-one-question.md

@@ -0,0 +1,125 @@
+# FCST-06 — "cannot persist" is one question, and it is asked about the outcome
+
+Two decisions. The first is the one the ticket put up for judgement; the second
+is the one that decides what a store answers *after* it has failed to write.
+
+---
+
+## 1. One derived predicate, not two places testing two symptoms — yes
+
+`price_history.py`'s module docstring states, unconditionally:
+
+> ⚠️ AND THAT RULE IS CARRIED OUT IN `_remember()`, NOT ONLY IN `_save()`. …
+> A store that cannot persist now records nothing, and both of its answers say
+> the same thing.
+
+It was enforced against `not self.ok`, and `ok` is **about the directory**: it
+says `/data` was there when the store opened. A read-only remount and a full
+volume both leave it `True`. So the second way to be a store that cannot
+persist — `ok=True`, the directory present, `_save()` failing — was outside the
+one condition the rule named, and in that state `words` printed two full days
+beside the write error while `yesterday()` answered 24 real points, from memory,
+until a restart dropped them without a word.
+
+**The ticket's question was whether "cannot persist" should be one derived
+predicate rather than two places each testing a different symptom. It should,
+and the reason is not tidiness — it is that the enumeration cannot be
+completed.** FCST-04's M4 enforced the rule on the symptom that was in front of
+it. FCST-05's HIGH was the same shape — an `except` clause naming the one
+content-failure member somebody had met (`json.JSONDecodeError`) instead of the
+`ValueError` family — and its fix was to name the family. This is the third instance in
+one module, and it is the last place in this file where the rule can be spelt
+twice. A fourth cause — an `OSError` subclass, a filesystem that accepts the
+`mkstemp` and refuses the `os.replace`, a container whose volume is unmounted
+mid-refresh — needs no fourth branch now, because none of them is asked about.
+
+### Where the one question can be asked, which is the whole design
+
+**"Can this store persist?" cannot be answered before the attempt.** A
+directory that exists and was writable a moment ago is not a write that
+completed; there is no pre-check that is not itself a guess, and adding one
+(`os.access`, a probe write) would be a *third* symptom test — precisely what
+the ticket rules out. So the predicate is an **outcome**:
+
+    _save() -> bool      # True only if the file now holds what memory holds
+
+Both ways of not persisting are the same answer from the same call: no durable
+volume returns `False` without touching the disk, an `OSError` out of the write
+returns `False` after recording why. `_remember()` reads that one answer and
+keeps only what it says was kept. **The `not self.ok` test in `_remember()` is
+gone** — not moved, gone — because there is nothing left for a second test to
+add, and a check in `test_price_history.py` 5d asserts its absence from the
+source.
+
+⚠️ **That last check is deliberately a source check, and habit 4 is why it is
+allowed to be.** The claim "there are not two places testing two symptoms" is a
+claim about the *shape of the code*, not about a behaviour — the behaviour is
+identical whether the rule is enforced once or twice. Every other check in 5d
+is driven.
+
+### What was NOT done, and why
+
+- **Not a `can_persist()` helper.** It would have to answer before the attempt,
+  which is the thing that cannot be done honestly, and it would be a second
+  place to keep in step with the first.
+- **Not a probe write at `_open()`.** It would make `ok` mean something new, it
+  would write to `/data` on every start, and a volume that is writable at 08:00
+  and full at 14:00 would still defeat it. A predicate that can go stale is a
+  symptom test wearing a predicate's name.
+- **Not making a failed write fatal.** §6, and this module's own opening: a
+  broken cache must not be a broken controller. An add-on that will not start
+  because `/data` filled up is a worse defect than the one being fixed. The
+  store degrades; it does not raise. `_save()` reports and returns.
+
+---
+
+## 2. The MERGE is undone, not the store
+
+When `_save()` answers no, `_remember()` restores the state the read started
+from — **not** an empty one.
+
+The alternative considered was to clear `self._days` outright, on the argument
+that a store which cannot write is a store that should stop answering. It is
+wrong, and measurably so: the state being restored is the last one that **is on
+disk**, so answering it is not a promise the store cannot keep. A site whose
+volume fills at 14:00 goes on answering the yesterday it wrote at 13:30 —
+through the restart, because that record is in the file. What it stops doing is
+*adding* to it.
+
+Clearing would throw away a good yesterday that survives a restart in order to
+punish a write that failed, and it would make the morning fill worse on exactly
+the day the operator is already dealing with a full disk.
+
+That gives the file the invariant its rule was always describing, stated once
+in the class docstring:
+
+> `self._days` **never holds a point the file does not.**
+
+⚠️ Not *"mirrors the file"* — `_usable_days()` drops records the file still
+holds, and that asymmetry is deliberate. The only direction that can **invent a
+yesterday** is the one named.
+
+⚠️ **And the snapshot is `copy.deepcopy`, not `dict(self._days)`.** A one-level
+copy shares the day *records*, and the merge rebinds `points` on the ones it
+touches — so the "restored" store would answer the merge that never reached the
+file. Measured: with a shallow copy, a store holding 12 persisted points
+answered **24** after a failed merge. It would also be correct only for as long
+as nobody makes the merge mutate a `points` list or a row in place, which is the
+same "correct for the conditions someone thought of" this ticket is about. The
+store holds at most `KEEP_DAYS + 2` records and writes 48 times a day; a copy
+that is right by construction is worth more than the allocations.
+
+---
+
+## What this does not change
+
+- **No shipped path reaches the defect.** `/data` is the Supervisor's durable
+  volume and this module is its only writer, so on a real install `_save()`
+  succeeds and nothing above is on the hot path. The fix is that a guarantee the
+  module states about itself now holds in the condition it did not.
+- **The failure was already visible and still is** — `_LOG.error("could not
+  persist price history: %s")` and `last_error`, reported by both `words` and
+  `yesterday()`'s reason. What changed is that it is now *acted on*.
+- **Nothing about the prices.** The series served is the feed's own, every
+  carried point is still flagged provisional, and a broken cache is still a
+  short horizon rather than a §11.4 degradation.

goodwe_controller/DOCS.md

@@ -1474,8 +1474,13 @@ a cheap night that has already gone used to sit in the series as the cheapest
 hour of the day, and a cycle can never buy in an hour that is past — so the
 trade was quietly never taken. The spread is now taken over hours still ahead.
 
-A new status row, **Price history**, says which days are held and how many hours
-each has. If it reads `empty` or `unavailable`, nothing is carried, the horizon
+A new status row, **Price history**, says which days are held, how many price
+points each holds and how many whole hours those cover — `2026-01-13 (96 points
+over 24 h)`. ⚠️ **Those are two different numbers and the row used to print one
+of them under the other's name**: it counted points and labelled them `h`, so a
+quarter-hourly feed read `96 h` for a 24-hour day. The figure that says whether
+tomorrow morning's fill has a whole day to carry is the **hour** one. If it
+reads `empty` or `unavailable`, nothing is carried, the horizon
 is as short as the feed makes it, and every other guarantee is unchanged — a
 cache that cannot answer is not allowed to be a controller that cannot regulate.
 The store is rebuilt by the next successful fetch; on a brand-new install the
@@ -1483,10 +1488,20 @@ first fill is possible the day after the first refresh.
 
 The same applies to a file that is *partly* broken: a day record this build
 cannot read is dropped, the row says how many were dropped and why, and the rest
-of the file still loads. And where there is no `/data` at all the store records
-**nothing** — not even in memory. An in-memory yesterday would answer for one
-run and vanish on the next restart, which is a cache that forgets without saying
-so; `unavailable` and *no prices carried* are the honest pair.
+of the file still loads. And a store that **cannot persist records nothing** —
+not even in memory. That covers a missing `/data`, and it also covers a `/data`
+that is there and cannot be **written**: a full volume, a read-only remount. An
+in-memory yesterday would answer for one run and vanish on the next restart,
+which is a cache that forgets without saying so; the row naming the reason and
+*no prices carried* are the honest pair.
+
+⚠️ **What a failed write does not do is throw away what is already in the
+file.** A volume that fills at 14:00 leaves the yesterday written at 13:30 still
+answering, because that record survives a restart — the store stops *adding* to
+it, nothing more. And it never stops the add-on: a write that cannot complete is
+logged and reported on the status row, never raised. A cache that cannot answer
+is not allowed to be a controller that cannot regulate, and that applies to the
+volume being full exactly as it applies to the volume being absent.
 
 **Injection is priced separately** (`export_price_eur_kwh`). In Belgium it is
 often near zero, which by itself strongly favours self-consumption over export.

goodwe_controller/MUTATION.md

@@ -127,6 +127,43 @@ Two rules come out of it, and both are cheap:
    function, that pair is the first thing to try — TEST-16's two survivors were
    adjacent lines in one function and neither was individually live.
 
+⚠️ **AND FCST-05 WALKED INTO IT AGAIN, IN ITS OWN NEW CHECK, WHICH IS WHY THE
+RULE IS “mutate every check you add”.** `price_history.py` reports a malformed
+point row from **two** places into **one** field: `_usable_days()` records what
+it FOUND when the file was opened, and `_remember()` records what it DROPPED on
+the next refresh. Both land in `last_error`, and `words` prints `last_error`.
+The first draft of the new check asked `"malformed point row" in words` — which
+whichever rung is still standing satisfies — so deleting `_remember()`'s note
+entirely was a **−1 B SURVIVOR of the whole suite**, on the half of the fix that
+tells an operator the cache **self-healed**. It is graded on the TENSE now
+(`… in <day>` at open, `… dropped from <day>` after the refresh), the two are
+asserted to DIFFER, and the paired mutation — both notes deleted, −91 B — is
+red on 15 named checks. Same shape as TEST-16 and the same fix: **grade the
+reason, not the verdict**, and when two rungs answer one stimulus, mutate them
+together.
+
+⚠️ **AND FCST-06 FOUND THE COUNT ITSELF LYING — LOW, NOT SILENT.** A mutation
+whose red is a **traceback** does not report one red; it reports however many
+checks ran before the traceback, and grades **none** of the ones after it. The
+new section drove M4 (`self._days = before` → `self._days = {}`) and the run
+printed **1 FAIL**. The true figure is **3**: an f-string in the next check read
+`len(read.points)`, and `.points` on a `no_data` **raises `Absence` by design**,
+so the harness died mid-section with the remaining checks ungraded. That is
+`why()`'s and `derived()`'s rule — *"it answered nothing" and "it broke on the
+way to answering" must not grade the same* — arriving in the **printed operand**
+of a check rather than in its condition, which is the half that is easy to miss
+because it looks like formatting. Two things follow:
+
+* **a mutation report's red COUNT is only a lower bound until the run is seen to
+  have reached the end of its section.** Read the output, not the count;
+* **the total accessor has to cover what a check PRINTS, not only what it
+  asserts.** `test_price_history.py` 5d added `npts()` beside `why()` for
+  exactly this, and re-driving M4 against it moved 1 FAIL to 3.
+
+Worse, had the traceback landed one check earlier the mutation would have
+produced **no named red at all** — a result that reads like a broken harness and
+gets retried rather than read.
+
 ## The third way a check lies: its DATA cannot reach the failure it names
 
 Stale bytecode manufactures survivors from a mutation that never ran. Masking

goodwe_controller/app/price_forecast.py

@@ -55,6 +55,17 @@ So the flag is not passed through anywhere in this file:
     It is a post-condition, not a filter: it never fixes a flag, because a
     module that quietly corrects its own provenance is one whose provenance
     cannot be trusted.
+
+    ⚠️ IT GRADES THE POINTS THAT ARE EMITTED AND NOTHING ABOUT THE ONES THAT
+    ARE NOT (FCST-05). `_audit()` is a provenance check, NOT a completeness
+    check: fed FCST-04's own decimation defect - four published points in an
+    hour, only the last emitted - it ACCEPTS, and it accepts an empty output
+    too. Measured, and the FCST-03 audit it replaced was equally blind, so
+    this is a limit rather than a regression. What actually guards against a
+    decimated series is the point COUNT and the pass-through IDENTITY checks
+    in `test_price_forecast.py` (section on the quarter-hourly feed) and
+    `test_price_history.py`; see `_audit()`'s own docstring for why the
+    completeness rule is not expressible from its two arguments.
   * and test_price_forecast.py walks THIS FILE'S AST and fails if any
     `PricePoint(...)` in it passes anything but a literal for `provisional`, so
     the pass-through idiom cannot come back in a later edit and pass review by
@@ -225,14 +236,67 @@ def _audit(points, published) -> None:
       * CARRIED - the hour must not appear in `published` AT ALL, the flag must
         be exactly True, and that hour may hold no second point.
 
-    ⚠️ WHAT REPLACED "AT MOST ONE POINT PER HOUR", AND WHY IT IS STRICTER. This
-    audit began as a `seen` set: any repeated `_utc_hour` raised, because a
-    duplicated hour is how a DST day gets mispriced. A bare hour count cannot
-    express a quarter-hourly feed, where four distinct published points share
-    one hour legitimately - so the count was replaced by the two rules above,
-    which subsume it. A repeated CARRIED hour still raises; so does the same
-    published point emitted twice; and a carried point may no longer share an
-    hour with a published one, which the count never checked at all.
+    ⚠️ WHAT REPLACED "AT MOST ONE POINT PER HOUR", AND WHAT THAT DID AND DID
+    NOT CHANGE. This audit began as FCST-03's `seen` set: any repeated
+    `_utc_hour` raised, beside an identity rule `p is not src`. A bare hour
+    count cannot express a quarter-hourly feed, where four distinct published
+    points share one hour legitimately - so the count was replaced by the two
+    rules above, which subsume it.
+
+    ⚠️ AND IT IS NOT STRICTER. FCST-04's version of this paragraph claimed two
+    strengthenings and BOTH WERE FALSE; FCST-05 measured it. A differential
+    over the whole domain this audit can express - every published mask over 3
+    hours crossed with every output series of length 0-3 drawn from
+    {published, published-provisional, carried, carried-unflagged,
+    copy-of-published} - found ZERO divergence in EITHER direction:
+
+        OLD raised, NEW accepted (WEAKER):   0
+        OLD accepted, NEW raised (STRONGER): 0
+
+    The two specific claims, one at a time:
+
+      * "so does the same published point emitted twice" - the OLD `seen`
+        count already raised on that, because emitting one point twice is one
+        HOUR twice. Preserved by the identity/`emitted` rules, not added.
+      * "a carried point may no longer share an hour with a published one,
+        which the count never checked at all" - true of the COUNT, and the old
+        IDENTITY rule raised on it anyway: a carried point in a published hour
+        took the `else` branch and failed `p is not src`. The message was about
+        a copy rather than about a carry, which is the only thing that moved.
+
+    The one real behavioural change is the intended RELAXATION - several
+    distinct published points in one hour, which the old audit could not
+    express at all and refused. That relaxation is why this was rewritten, and
+    it stays. A right decision with a fabricated reason is still a defect here.
+
+    ⚠️ AND THE REAL LIMIT, WHICH IS NOT A REGRESSION AND IS NOT SMALL: THIS
+    AUDIT DOES NOT CHECK COMPLETENESS. It grades every point that IS emitted
+    and says nothing about the ones that are not. Fed the H1 decimation
+    directly - four published points in an hour, output emitting only the last
+    - it ACCEPTS; so does two of four, and so does an EMPTY output. So `_audit`
+    is NOT what stops H1 recurring. What stops it is the point COUNT and the
+    IDENTITY checks in `test_price_history.py` and `test_price_forecast.py`,
+    which compare the emitted series against the feed's own prints.
+
+    It is not made complete here, and the reason is structural rather than
+    effort: `published` is `_published_by_hour(read)` over the WHOLE read,
+    unwindowed, while `fill_from_yesterday()` emits only [since, until). A
+    published point outside the window is legitimately absent, so "every
+    published point appears" is not a true post-condition of these two
+    arguments - it would need the window passed in, which is a different
+    signature and a different check.
+
+    ⚠️ AND THAT IS MEASURED, NOT ASSUMED. FCST-05 built the rule - raise unless
+    every point of every `published[u]` is emitted, +162 B - and ran it. It
+    takes the 1 h, 12 h and 48 h HORIZON checks in test_price_forecast.py RED,
+    on the shipped fill and a healthy feed: at an off-the-hour instant the
+    feed's already-elapsed prints are outside [since, until) and are correctly
+    dropped, and the rule calls that a provenance failure. (The differential
+    beside those checks scores the same rule as 49 STRONGER cases, so it is a
+    real change of behaviour and not a no-op.) The rule as stated is therefore
+    not merely unimplemented, it is WRONG - and stating a false post-condition
+    is MUTATION.md habit 16's defect: an instrument that asserts a completeness
+    it does not have is worse than no instrument.
 
     ⚠️ RAISES, NEVER REPAIRS. A module that silently corrects its own
     provenance is one whose provenance means nothing; the only safe response to

goodwe_controller/app/price_history.py

@@ -61,6 +61,38 @@ yesterday that works all day and is gone on the next restart, which is exactly
 the silently-forgetting cache this rule exists to refuse. A store that cannot
 persist now records nothing, and both of its answers say the same thing.
 
+⚠️ AND "CANNOT PERSIST" IS ONE QUESTION, ASKED ONCE, ABOUT THE OUTCOME -
+FCST-06. The paragraph above enforced that rule against `not self.ok`, which is
+one SYMPTOM of not persisting: the durable volume is missing. There is a second,
+and FCST-06 measured it. With `ok=True` and the directory present, `_save()` can
+still fail - a read-only remount (`[Errno 30] Read-only file system`), a full
+volume (`[Errno 28] No space left on device`) - and it logs, records
+`last_error` and returns. `_remember()` had already populated `self._days`, so
+`words` read `'2026-01-14 (24 points over 24 h); 2026-01-15 (24 points over
+24 h) [could not write (...)]'` while the file on disk held `{}`, and
+`yesterday()` answered 24 REAL POINTS for the rest of the process's life. Gone
+on the next restart, and nothing said it would be - the same silently-forgetting
+cache, reached by the condition the rule did not name.
+
+⚠️ SO IT IS NOT TWO RULES, AND IT IS DELIBERATELY NOT TWO TESTS. "Can this
+store persist?" CANNOT BE ANSWERED BEFORE THE ATTEMPT - a directory that existed
+and was writable a moment ago is not a write that completed - so it is answered
+AFTER it, once, by the only thing holding the evidence: `_save()` returns
+whether the file now holds what memory holds. A missing volume is one of its
+False cases and an `OSError` from the write is another, and `_remember()` KEEPS
+ONLY WHAT THAT ONE ANSWER SAYS WAS KEPT. There is no `not self.ok` test in
+`_remember()` any more, because there is nothing left for a second test to add -
+and a third symptom nobody has thought of is how this defect arrived both times.
+
+⚠️ AND WHAT IS KEPT IS THE LAST STATE THAT IS ON DISK, NOT AN EMPTY ONE. The
+MERGE is undone, not the store. A site whose volume fills at 14:00 goes on
+answering the yesterday it wrote at 13:30 - that record survives a restart, so
+answering it is not a promise the store cannot keep. What it stops doing is
+adding to it. The invariant the rule was always describing, stated once:
+`self._days` NEVER HOLDS A POINT THE FILE DOES NOT. (Not "mirrors the file" -
+`_usable_days()` drops records the file still holds. The only direction that can
+invent a yesterday is the one named.)
+
 
 THE FOUR RULES THIS FILE IS SHAPED BY
 =====================================
@@ -148,6 +180,7 @@ and turning it on is a product decision, not a wiring one. See
 DECISIONS/FCST-04-the-day-that-is-carried.md.
 """
 
+import copy
 import json
 import logging
 import os
@@ -208,8 +241,65 @@ def _yesterday_of(instant: datetime) -> str:
     return (brussels(instant).date() - timedelta(days=1)).strftime("%Y-%m-%d")
 
 
-def _n_points(rec) -> int:
-    """How many points a day record holds. 0 for anything that is not one.
+def _point_of(row):
+    """One stored row -> a `PricePoint`, or `None` if the row is not one.
+
+    THE ONE VALIDATOR, AND THE ASYMMETRY IT CLOSES (FCST-05's MEDIUM). This
+    file used to hold TWO standards for what a stored point is. The READ path
+    (`_parse_points()`) was properly defensive - every field checked, a
+    malformed row dropped and reported. The WRITE path was
+    `{q["start"]: q for q in raw if isinstance(q, dict) and "start" in q}`
+    followed by `sorted(..., key=lambda q: q["start"])`, which trusts
+    `q["start"]` to be a hashable, sortable string it never checked. Measured,
+    with the malformed row under the day the feed is currently writing to:
+
+        start is an int   -> TypeError: '<' not supported between instances
+                             of 'str' and 'int'
+        start is a dict   -> TypeError: cannot use 'dict' as a dict key
+        start is null     -> TypeError: '<' not supported between 'str' and
+                             'NoneType'
+        start is a list   -> TypeError: cannot use 'list' as a dict key
+
+    ⚠️ AND THE BLAST RADIUS IS PERMANENT, WHICH IS WHY IT IS NOT MERELY UNTIDY.
+    `_remember()` is reached through `carry()` -> `resolve_tariff()` ->
+    `run_tariff_refresh`, which catches `Exception` - so the add-on stays UP
+    and EVERY refresh raises forever. The plan retires after
+    `TARIFF_MAX_AGE_S`, and arbitrage and the cheap window are off from then
+    on. There is no self-heal, because `_remember()` raises before it reaches
+    `_save()`. `_usable_days()` only required `rec["points"]` to be a LIST; the
+    elements were unchecked on the write path, and the existing fixture put its
+    malformed row under a day the feed was NOT writing to, so `_remember()`
+    never touched it.
+
+    So both paths call this, and neither of them has a standard of its own.
+    Returning `None` rather than raising is the contract: a caller decides
+    whether a bad row is dropped (both of them do) and how it is reported.
+    """
+    if not isinstance(row, dict):
+        return None
+    try:
+        start = datetime.fromisoformat(row["start"])
+        value = float(row["eur_per_kwh"])
+        flag = row["provisional"]
+    except (KeyError, TypeError, ValueError):
+        return None
+    if start.tzinfo is None or start.utcoffset() is None \
+            or not isinstance(flag, bool):
+        return None
+    return PricePoint(start=start, eur_per_kwh=value, provisional=flag)
+
+
+def _held(rec) -> tuple:
+    """`(points, whole hours they cover)` for a day record. `(0, 0)` otherwise.
+
+    ⚠️ POINTS AND HOURS ARE TWO NUMBERS, AND FCST-04's STATUS ROW PRINTED THE
+    FIRST UNDER THE SECOND'S NAME. The row was `f"{d} ({_n_points(rec)} h)"`,
+    which counts POINTS: on the quarter-hourly feed this add-on already
+    supports it read `'2026-01-13 (96 h)'` - 96 "h" for a 24-hour day. That is
+    the points/hours conflation H1 was raised for, on the row H1's own ticket
+    shipped. Both numbers are reported now, so an operator can see a day that
+    is 24 points over 24 h and a day that is 96 points over 24 h and tell them
+    apart, and neither number is wearing the other's unit.
 
     ⚠️ TOLERANT, AND IT IS THE SECOND LINE OF DEFENCE, NOT THE FIRST. `_load()`
     drops a malformed day record and says so, so this should never see one. It
@@ -220,11 +310,52 @@ def _n_points(rec) -> int:
     "oops"}}` gave `ok=True`, `last_error=None`, and an `AttributeError` out of
     `words`, `yesterday()` and `resolve_tariff()` alike, with no self-heal
     because `_remember()` raised before it could rewrite the file.
+
+    ⚠️ AND IT COUNTS ONLY ROWS THAT ARE POINTS. A row `_point_of()` refuses is
+    not held prices, and a row count that included it would report a store as
+    healthier than it is - which is the second half of FCST-05's MEDIUM: the
+    status row said `'2026-01-15 (1 h)'` about a record whose single row was
+    killing every tariff refresh. What is dropped is named in `last_error`, and
+    `words` prints that beside the counts.
     """
+    if not isinstance(rec, dict):
+        return (0, 0)
+    raw = rec.get("points")
+    if not isinstance(raw, (list, tuple)):
+        return (0, 0)
+    n, hours = 0, set()
+    for row in raw:
+        pt = _point_of(row)
+        if pt is None:
+            continue
+        n += 1
+        hours.add(pt.start.astimezone(timezone.utc)
+                  .replace(minute=0, second=0, microsecond=0))
+    return (n, len(hours))
+
+
+def _row_of(day: str, points: int, hours: int) -> str:
+    """One day's cell of the status row: what is held, in BOTH of its units.
+
+    ⚠️ THE UNIT IS SPELT OUT ON EACH NUMBER. FCST-04's cell was
+    `f"{day} ({points} h)"` - the point count wearing the hour's name, which on
+    a quarter-hourly feed reads `'2026-01-13 (96 h)'` for a 24-hour day. What
+    an operator needs from this row is whether tomorrow morning's fill has a
+    whole day to carry, and that is the HOUR figure; the point count says how
+    finely it is priced. Neither answers for the other.
+    """
+    return (f"{day} ({points} point{'' if points == 1 else 's'} "
+            f"over {hours} h)")
+
+
+def _bad_rows(rec) -> int:
+    """How many rows of a day record are not points. Total; never raises."""
     if not isinstance(rec, dict):
         return 0
-    pts = rec.get("points")
-    return len(pts) if isinstance(pts, (list, tuple)) else 0
+    raw = rec.get("points")
+    if not isinstance(raw, (list, tuple)):
+        return 0
+    return sum(1 for row in raw if _point_of(row) is None)
 
 
 class PriceHistory:
@@ -235,6 +366,14 @@ class PriceHistory:
     site's first day, and a different thing from a store that could not be
     opened. Both are visible: `ok`/`last_error` describe the file, and
     `yesterday()`'s reason describes the answer.
+
+    ⚠️ AND `ok` IS NOT "CAN PERSIST" (FCST-06). It says the directory was
+    there when the store opened, which a full volume and a read-only remount
+    both leave true. The one thing that answers "can this store persist" is
+    `_save()`'s return value, and `self._days` NEVER HOLDS A POINT THE FILE DOES
+    NOT, because `_remember()` undoes a merge that answer refused. That is what
+    makes `words` and `yesterday()` describe the same store in EVERY degraded
+    state rather than only in the ones somebody enumerated.
     """
 
     def __init__(self, path: str = "/data/price_history.json"):
@@ -269,11 +408,33 @@ class PriceHistory:
                       "one, and the fill has nothing to carry until the day "
                       "after that", self.path)
             return
-        except (json.JSONDecodeError, OSError) as err:
+        except (ValueError, OSError) as err:
             # A corrupt cache must not stop the controller, and must not be
             # silently re-created either: the record is discarded, the reason
             # is kept, and `yesterday()` reports it rather than answering
             # "no record", which would read as a normal first day.
+            #
+            # ⚠️ `ValueError`, NOT `json.JSONDecodeError`, AND THAT IS
+            # FCST-05's HIGH. Reading this file has two failure families: the
+            # BYTES (`OSError`) and the CONTENT - and the content family has at
+            # least two members, because `open(..., encoding="utf-8")` decodes
+            # before `json` ever sees a character. `UnicodeDecodeError` is
+            # `ValueError -> UnicodeError -> UnicodeDecodeError`; it is NOT a
+            # `json.JSONDecodeError` and it is NOT an `OSError`
+            # (`issubclass(UnicodeDecodeError, OSError)` is False). So ONE
+            # non-UTF-8 byte in /data/price_history.json escaped `_load()` ->
+            # `_open()` -> `PriceHistory.__init__`, which is unguarded at
+            # main.py's `Controller.__init__`, AND THE ADD-ON DID NOT START.
+            # Measured: `UnicodeDecodeError: 'utf-8' codec can't decode byte
+            # 0xff in position 39`. Valid-UTF-8-invalid-JSON, a UTF-8 BOM and a
+            # non-dict day record all degraded correctly - it was the one byte
+            # class left uncovered, which is the same class as the defect the
+            # `_usable_days()` rework was raised for.
+            #
+            # The tuple names the FAMILY rather than the two members known
+            # today, because "the file's content is not readable as JSON" is
+            # exactly what this handler says, and a third `ValueError` subclass
+            # nobody has thought of is how this defect arrived the first time.
             self.last_error = f"unreadable ({err})"
             _LOG.warning("price history at %s is %s - starting from empty. "
                          "§8.3 carry-forward has nothing to carry until the "
@@ -318,33 +479,71 @@ class PriceHistory:
 
         Dropped, not repaired: an unusable record has nothing in it to keep, and
         the next `_remember()` writes a fresh one under that date.
+
+        ⚠️ AND THE ROWS INSIDE A KEPT RECORD ARE COUNTED, NOT ONLY THE RECORD
+        (FCST-05). This checked that `rec["points"]` is a LIST and stopped
+        there, so a record holding one row with a `start` of `7` loaded as
+        healthy: `last_error=None`, and `words` said `'2026-01-15 (1 h)'` about
+        a store whose next `_remember()` would raise on that row and go on
+        raising forever. The rows are NOT dropped here - `_parse_points()` is
+        the read path's one filter and stays so - but they are COUNTED, so the
+        store stops reporting health it does not have from the moment it is
+        opened. `_remember()` drops them on the next refresh and `_save()`
+        rewrites the file, after which a restart clears the note.
         """
-        kept, dropped = {}, []
+        kept, dropped, malformed = {}, [], {}
         for key, rec in days.items():
             if not isinstance(key, str) or not isinstance(rec, dict) \
                     or not isinstance(rec.get("points"), list):
                 dropped.append(str(key))
                 continue
+            bad = _bad_rows(rec)
+            if bad:
+                malformed[key] = bad
             kept[key] = rec
+        notes = []
         if dropped:
+            notes.append(f"{len(dropped)} unusable day record(s) dropped "
+                         f"({', '.join(sorted(dropped))})")
+        if malformed:
+            notes.append(
+                f"{sum(malformed.values())} malformed point row(s) in "
+                + ", ".join(f"{d} ({n})" for d, n in sorted(malformed.items())))
+        if notes:
             # Recorded, not silent: `yesterday()` and `words` both report
             # `last_error`, and a cache that quietly discards half a file reads
             # to an operator exactly like a normal first day.
-            self.last_error = (f"{len(dropped)} unusable day record(s) dropped "
-                               f"({', '.join(sorted(dropped))})")
+            self.last_error = "; ".join(notes)
             _LOG.warning("price history at %s: %s - the controller starts "
                          "anyway and the horizon is as short as the feed makes "
                          "it", self.path, self.last_error)
         return kept
 
-    def _save(self) -> None:
+    def _save(self) -> bool:
+        """Write `self._days` out. True only if the file now holds it.
+
+        ⚠️ THE RETURN VALUE IS THE "CAN THIS STORE PERSIST?" PREDICATE, AND
+        IT IS THE ONLY ONE (FCST-06). Both ways of not persisting end here and
+        are the same answer: no durable volume returns False without touching
+        the disk, an `OSError` out of the write returns False after recording
+        why. `_remember()` acts on that one answer, so the second place that
+        used to test the first symptom on its own - and missed the second - has
+        nothing left to test. See the module docstring's fifth warning.
+
+        ⚠️ AND IT REPORTS, IT DOES NOT RAISE. §6 and this module's opening: a
+        broken cache must not be a broken controller. An add-on that will not
+        start because /data filled up is a worse defect than the one FCST-06
+        fixed, so a failed write is logged, kept in `last_error` - where `words`
+        and `yesterday()` both read it - and RETURNED, never thrown.
+
+        ⚠️ THE TEMP FILE IS REMOVED ON EVERY FAILING PATH. A write that dies
+        between `mkstemp` and `os.replace` used to leave its temp behind, and
+        the refresh task runs 48 times a day: a persistent write failure - a
+        full volume being the likeliest cause - leaked 48 inodes a day onto the
+        volume that was already the problem.
+        """
         if not self.ok:
-            return
-        # ⚠️ THE TEMP FILE IS REMOVED ON EVERY FAILING PATH. A write that dies
-        # between `mkstemp` and `os.replace` used to leave its temp behind, and
-        # the refresh task runs 48 times a day: a persistent write failure -
-        # a full volume being the likeliest cause - leaked 48 inodes a day onto
-        # the volume that was already the problem.
+            return False
         tmp = None
         try:
             fd, tmp = tempfile.mkstemp(dir=os.path.dirname(self.path))
@@ -357,12 +556,14 @@ class PriceHistory:
         except OSError as err:
             self.last_error = f"could not write ({err})"
             _LOG.error("could not persist price history: %s", err)
+            return False
         finally:
             if tmp is not None:
                 try:
                     os.unlink(tmp)
                 except OSError:             # nothing further to try, and a
                     pass                    # failed cleanup is not a fault
+        return True
 
     # -- the one public entry point -----------------------------------------
     def carry(self, read: PriceRead, *, since: datetime, until: datetime,
@@ -399,15 +600,6 @@ class PriceHistory:
         nothing outside this file names it, and `test_price_history.py` checks
         the tree for that rather than trusting the underscore.
         """
-        if not self.ok:
-            # ⚠️ A STORE THAT CANNOT PERSIST RECORDS NOTHING. See the module
-            # docstring's third warning: filling `self._days` here would give a
-            # site with no /data an in-memory yesterday that answers a series
-            # all day and is gone on the next restart - a cache that forgets
-            # without saying so, which is the one thing that rule refuses. The
-            # honest degraded state is `words` = "unavailable" and `yesterday()`
-            # = `no_data`, and those two must agree.
-            return
         points = read.match(series=lambda r: r.points, no_data=lambda _r: ())
         if not points:
             return
@@ -416,7 +608,27 @@ class PriceHistory:
         provider = read.match(series=lambda r: r.provider,
                               no_data=lambda _r: None)
 
+        # ⚠️ THE STATE THIS READ STARTED FROM, KEPT SO IT CAN BE PUT BACK
+        # (FCST-06). Everything below merges into `self._days` IN PLACE, and
+        # whether any of it may be KEPT is not knowable until `_save()` has
+        # tried - so the merge runs, and it is undone if the file did not take
+        # it. There is no `not self.ok` branch above any more: a store with no
+        # durable volume is one of the answers `_save()` gives, not a separate
+        # symptom with a test of its own. Module docstring, fifth warning.
+        #
+        # ⚠️ DEEP, NOT `dict(self._days)`. A one-level copy shares the day
+        # RECORDS, and the loop below rebinds keys on the ones it touches - so
+        # the "previous" state would hand back records the merge had already
+        # edited, and a store with a good yesterday on disk would go on
+        # answering the merge of it that never reached the file. It would also
+        # be right only for as long as nobody makes this loop mutate a `points`
+        # list or a row in place. A copy that is right by construction outlives
+        # that, and the store holds at most KEEP_DAYS + 2 records, 48 times a
+        # day.
+        before = copy.deepcopy(self._days)
+
         touched = set()
+        junked = {}
         for p in points:
             day = _bdate(p.start)
             rec = self._days.setdefault(
@@ -424,16 +636,71 @@ class PriceHistory:
             # Rule 4: MERGE by instant. A later read updates the hours it
             # carries and leaves the rest; a revised price for an hour already
             # held wins, because it is the newer statement about that hour.
+            #
+            # ⚠️ EVERY HELD ROW GOES THROUGH `_point_of()` (FCST-05's MEDIUM).
+            # This was `{q["start"]: q for q in raw if isinstance(q, dict) and
+            # "start" in q}` sorted by `q["start"]` - a key and a sort key that
+            # trusted a value read off disk to be a hashable, sortable string.
+            # An `int`, `null`, a `dict` or a `list` there raised `TypeError`
+            # out of `carry()` on the day the feed was writing to, and
+            # `run_tariff_refresh` caught it, so the add-on stayed up while
+            # EVERY refresh failed for good. A row that is not a point is now
+            # dropped and reported here, exactly as the read path drops it, and
+            # `_save()` below writes the record back clean - which is the
+            # self-heal the raising version could never reach.
+            #
+            # ⚠️ AND THE MERGE KEY IS THE VALIDATED INSTANT, NOT ITS SPELLING.
+            # Aware datetimes hash and compare by instant, so this is rule 4 as
+            # rule 4 is written. The old string key merged by SPELLING, and the
+            # old lexicographic sort put the second 02:00 of the 25-hour
+            # October day (`+01:00`) BEFORE the first (`+02:00`). Only the
+            # stored order was affected - `_parse_points()` re-sorts by instant
+            # on the way out - but the store now holds what it says it holds.
             raw = rec.get("points")
-            held = {q["start"]: q for q in (raw if isinstance(raw, list) else ())
-                    if isinstance(q, dict) and "start" in q}
-            held[p.start.isoformat()] = p.canonical()
-            rec["points"] = sorted(held.values(), key=lambda q: q["start"])
+            held = {}
+            for q in (raw if isinstance(raw, list) else ()):
+                kept = _point_of(q)
+                if kept is None:
+                    junked[day] = junked.get(day, 0) + 1
+                    continue
+                held[kept.start] = q
+            held[p.start] = p.canonical()
+            order = sorted(held, key=lambda t: t.astimezone(timezone.utc))
+            rec["points"] = [held[t] for t in order]
             rec["product"], rec["provider"] = product, provider
             touched.add(day)
 
+        if junked:
+            # Said, not swallowed. `words` prints `last_error`, so a store that
+            # has just discarded rows off disk stops reporting plain health -
+            # and after `_save()` below the file is clean, so a restart clears
+            # this. `_save()` overwrites it if the write itself fails, which is
+            # the more urgent fact of the two - and on that path the drop is
+            # undone with the rest of the merge, so those rows are still in the
+            # file and the next write that completes drops them again.
+            self.last_error = (
+                f"{sum(junked.values())} malformed point row(s) dropped from "
+                + ", ".join(f"{d} ({n})" for d, n in sorted(junked.items())))
+            _LOG.warning("price history at %s: %s - the record is rewritten "
+                         "without them and the refresh continues",
+                         self.path, self.last_error)
         self._prune(stamp)
-        self._save()
+        if not self._save():
+            # ⚠️ A STORE THAT CANNOT PERSIST RECORDS NOTHING - the rule, asked
+            # ONCE as a question about the outcome instead of tested twice
+            # against two symptoms of it. Keeping the merge here is exactly the
+            # cache the module docstring refuses: `yesterday()` would answer a
+            # series the file does not contain, all day, until a restart
+            # dropped it without a word - and `words` and `yesterday()` would
+            # be describing two different stores. The reason is already in
+            # `last_error` (from `_save()`, or from `_open()` when there is no
+            # volume at all), so both answers now describe what is on disk.
+            #
+            # It does NOT raise, and §6 is why: a broken cache must not be a
+            # broken controller. `carry()` still returns the fill, the horizon
+            # is as short as the feed makes it, and the add-on regulates.
+            self._days = before
+            return
         _LOG.debug("price history: %d point(s) filed under %s; %d day(s) held",
                    len(points), ", ".join(sorted(touched)), len(self._days))
 
@@ -523,15 +790,21 @@ class PriceHistory:
         an exception here is an add-on that does not start. It went out that
         way: a day record that was a string instead of a dict raised
         `AttributeError` from the `len(...)` below. `_load()` now drops such a
-        record and `_n_points()` is total, so this has two independent reasons
-        not to fail on a broken file.
+        record and `_held()` is total, so this has two independent reasons not
+        to fail on a broken file.
+
+        ⚠️ AND IT COUNTED POINTS AND CALLED THEM HOURS (FCST-05's LOW). The row
+        was `f"{d} ({_n_points(rec)} h)"`; on the quarter-hourly feed this
+        add-on already supports that reads `'2026-01-13 (96 h)'` for a 24-hour
+        day. Both numbers are printed now - see `_held()` - because they are
+        two different facts and only one of them was ever being shown.
         """
         if not self.ok:
             return f"unavailable - {self.last_error}"
         if not self._days:
             why = f" - {self.last_error}" if self.last_error else ""
             return f"empty{why} - nothing to carry forward yet"
-        held = "; ".join(f"{d} ({_n_points(self._days[d])} h)"
+        held = "; ".join(_row_of(d, *_held(self._days[d]))
                          for d in sorted(self._days, key=str))
         # A file that PARTLY loaded says so, or the dropped days read as days
         # that were simply never written.
@@ -550,21 +823,14 @@ def _parse_points(raw) -> tuple:
         return (), "the `points` field is not a list"
     out, dropped = [], 0
     for row in raw:
-        if not isinstance(row, dict):
+        # ⚠️ `_point_of()` IS THE ONE VALIDATOR, and this loop used to BE it.
+        # The write path had its own, weaker standard; see `_point_of()` for
+        # what that cost. One standard, two callers, no asymmetry to re-open.
+        pt = _point_of(row)
+        if pt is None:
             dropped += 1
             continue
-        try:
-            start = datetime.fromisoformat(row["start"])
-            value = float(row["eur_per_kwh"])
-            flag = row["provisional"]
-        except (KeyError, TypeError, ValueError):
-            dropped += 1
-            continue
-        if start.tzinfo is None or start.utcoffset() is None \
-                or not isinstance(flag, bool):
-            dropped += 1
-            continue
-        out.append(PricePoint(start=start, eur_per_kwh=value, provisional=flag))
+        out.append(pt)
     out.sort(key=lambda p: p.start.astimezone(timezone.utc))
     return tuple(out), (f"{dropped} malformed point(s)" if dropped else "")
 

goodwe_controller/test_price_forecast.py

@@ -59,6 +59,7 @@ Read MUTATION.md before you report a survivor.
 
 import ast
 import inspect
+import itertools
 import os
 import sys
 from datetime import datetime, timedelta, timezone
@@ -354,14 +355,164 @@ check("...and still REFUSES a repeated CARRIED hour, which is the DST "
       "mispricing that count existed to catch",
       raises(lambda: _fc._audit([_fc._carried(D0, 0.11),
                                  _fc._carried(D0, 0.12)], {}), ValueError))
-check("...and refuses the SAME published point emitted twice - the provider's "
-      "points are strictly increasing, so a repeat is never a second print",
+check("...and STILL refuses the SAME published point emitted twice - the "
+      "provider's points are strictly increasing, so a repeat is never a "
+      "second print. The old hour COUNT already refused this (one point twice "
+      "is one hour twice); what is graded here is that the relaxation did not "
+      "lose it, not that it added it",
       raises(lambda: _fc._audit([_QUARTERS.points[0], _QUARTERS.points[0]],
                                 _PUB_Q), ValueError))
-check("...and refuses a CARRIED point standing in an hour the feed DID "
-      "publish, which the old count never looked at",
+check("...and STILL refuses a CARRIED point standing in an hour the feed DID "
+      "publish. The old COUNT never looked at it; the old IDENTITY rule did, "
+      "and raised - so this is preserved behaviour, not new strictness. "
+      "FCST-04's text called it new here and in `_audit`'s docstring, and a "
+      "measured differential says otherwise",
       raises(lambda: _fc._audit([_fc._carried(D0, 0.11)], _PUB_Q), ValueError))
 
+# ⚠️ AND WHAT THE AUDIT DOES NOT DO, STATED WHERE A READER OF THESE THREE
+# CHECKS WOULD OTHERWISE ASSUME IT (FCST-05, MUTATION.md habit 16). `_audit` is
+# a PROVENANCE post-condition, not a COMPLETENESS one: it grades every point
+# that IS emitted and says nothing about points that are not. So it accepts the
+# very decimation FCST-04 was raised for. That is not a regression - FCST-03's
+# audit was equally blind - but it means the audit is NOT the thing that stops
+# H1 coming back. The count and identity checks above this line are.
+check("the audit ACCEPTS the H1 decimation itself - four published points in "
+      "one hour, only the last emitted - and accepts an EMPTY output. It is a "
+      "provenance check, not a completeness one, and the checks that guard "
+      "provenance completeness are the COUNT and IDENTITY ones above",
+      derived(lambda: _fc._audit([_QUARTERS.points[3]], _PUB_Q)) is None
+      and derived(lambda: _fc._audit([], _PUB_Q)) is None)
+
+
+# --------------------------------------------------------------------------- #
+# ⚠️ AND THE CLAIM "STRICTER" IS ITSELF A CHECK NOW (FCST-05).
+#
+# FCST-04 rewrote `_audit` and wrote, in the module and in the three checks
+# above, that the result was STRICTER than FCST-03's. It is not. Two of the
+# stated reasons were false - the old `seen` count already refused the same
+# point emitted twice, and the old IDENTITY rule already refused a carried
+# point in a published hour - and a right decision with a fabricated reason is
+# a defect in this repository (MUTATION.md, and the reason DECISIONS files
+# exist). So the claim is measured here rather than asserted in prose:
+# FCST-03's audit is reproduced below from dd40c56 - its RULES verbatim, its
+# message TEXT shortened, and nothing here reads a message - and the two are
+# run over the whole domain `_audit` can express, in BOTH directions.
+#
+# THE DOMAIN, AND ITS SIZE. Every published mask over 3 hours (2**3) crossed
+# with every output series of length 0-3 drawn from 3 hours x 5 point kinds.
+# The nominal cross-product is 28 928; three of those kinds - published,
+# published-provisional, copy-of-published - DO NOT EXIST for an hour the feed
+# did not publish, and those combinations are SKIPPED rather than faked, which
+# leaves 11 990 realisable comparisons. A fabricated "published point" for an
+# unpublished hour is not in either audit's domain, and inventing one would
+# grade the fixture rather than the code.
+#
+# ⚠️ THE OLD COPY IS A LITERAL, NOT A CALL. MUTATION.md habit 2: a differential
+# whose two sides both come from the shipped module compares a thing against
+# itself and moves with every edit. This side is frozen text.
+def _audit_fcst03(points, published):
+    """FCST-03's `_audit`, verbatim from dd40c56. Its `published` maps a UTC
+    hour to ONE point, which is exactly the expressiveness this ticket relaxed.
+    """
+    seen = set()
+    for p in points:
+        u = _fc._utc_hour(p.start)
+        if u in seen:
+            raise ValueError("the filled series repeats " + u.isoformat())
+        seen.add(u)
+        src = published.get(u)
+        if src is None:
+            if p.provisional is not True:
+                raise ValueError(u.isoformat() + " is carried and is flagged "
+                                 "provisional=" + repr(p.provisional))
+        else:
+            if p is not src:
+                raise ValueError(u.isoformat() + " is published but the series "
+                                 "carries a COPY of that point")
+
+
+def _verdict(fn, pts, pub):
+    try:
+        fn(list(pts), pub)
+        return "accept"
+    except ValueError:
+        return "raise"
+
+
+_DH = [D0 + timedelta(hours=6 + i) for i in range(3)]
+_weaker, _stronger, _compared = [], [], 0
+for _mask in itertools.product((0, 1), repeat=3):
+    _pub = {u: PricePoint(start=u, eur_per_kwh=0.10 + i, provisional=False)
+            for i, (u, _on) in enumerate(zip(_DH, _mask)) if _on}
+    _old_pub = dict(_pub)
+    _new_pub = {u: (q,) for u, q in _pub.items()}
+    # The five shapes an output point can take at a given hour. Three of them
+    # do not exist for an hour the feed did not publish, and those combinations
+    # are skipped rather than faked - a fabricated "published point" for an
+    # unpublished hour is not in either audit's domain.
+    _kinds = []
+    for _u in _DH:
+        _src = _pub.get(_u)
+        _kinds.append(("published", _src))
+        _kinds.append(("published-provisional",
+                       None if _src is None else
+                       PricePoint(start=_src.start,
+                                  eur_per_kwh=_src.eur_per_kwh,
+                                  provisional=True)))
+        _kinds.append(("carried", _fc._carried(_u, 0.5)))
+        _kinds.append(("carried-unflagged",
+                       PricePoint(start=_u, eur_per_kwh=0.5,
+                                  provisional=False)))
+        _kinds.append(("copy-of-published",
+                       None if _src is None else
+                       PricePoint(start=_src.start,
+                                  eur_per_kwh=_src.eur_per_kwh,
+                                  provisional=_src.provisional)))
+    for _n in range(4):
+        for _combo in itertools.product(_kinds, repeat=_n):
+            if any(k[1] is None for k in _combo):
+                continue
+            _pts = [k[1] for k in _combo]
+            _compared += 1
+            _o = _verdict(_audit_fcst03, _pts, _old_pub)
+            _nw = _verdict(_fc._audit, _pts, _new_pub)
+            if _o == "raise" and _nw == "accept":
+                _weaker.append((_mask, [k[0] for k in _combo]))
+            if _o == "accept" and _nw == "raise":
+                _stronger.append((_mask, [k[0] for k in _combo]))
+
+# ⚠️ THE PRECONDITION FIRST (MUTATION.md habit 1). A differential over a
+# domain that cannot reach the two disputed cases would score 0-0 for the wrong
+# reason and read exactly like this one, so the two cases are driven against
+# the OLD audit by name before the totals are graded.
+_probe = PricePoint(start=_DH[0], eur_per_kwh=0.1, provisional=False)
+check(f"the differential really covers the two claims it is here to settle, "
+      f"or it is {_compared} runs of nothing: FCST-03's own audit raises on "
+      f"BOTH 'the same published point twice' and 'a carried point in a "
+      f"published hour', which is what FCST-04 said it did not",
+      _compared > 3000
+      and _verdict(_audit_fcst03, [_probe, _probe], {_DH[0]: _probe}) == "raise"
+      and _verdict(_audit_fcst03, [_fc._carried(_DH[0], 0.5)],
+                   {_DH[0]: _probe}) == "raise")
+check(f"...and over all {_compared} of them the reworked audit is NOT WEAKER "
+      f"than FCST-03's: {len(_weaker)} case(s) the old one raised and this one "
+      f"accepts {_weaker[:2]}",
+      _weaker == [])
+check(f"...and NOT STRONGER either: {len(_stronger)} case(s) the old one "
+      f"accepted and this one raises {_stronger[:2]}. FCST-04's docstring and "
+      f"two of these checks said it was stricter and named two reasons, and "
+      f"BOTH were already refused by the old count and the old identity rule. "
+      f"The only behavioural change is the RELAXATION below, which is what the "
+      f"rework was for",
+      _stronger == [])
+_QP = [PricePoint(start=_DH[0] + timedelta(minutes=15 * i),
+                  eur_per_kwh=0.1 + i, provisional=False) for i in range(4)]
+check("...and the relaxation is real and is NOT expressible to the old audit: "
+      "four distinct published points inside one hour, which FCST-03's count "
+      "refused as a repeated hour and this one accepts",
+      _verdict(_fc._audit, _QP, {_DH[0]: tuple(_QP)}) == "accept"
+      and _verdict(_audit_fcst03, _QP, {_DH[0]: _QP[0]}) == "raise")
+
 # ⚠️ THE HORIZON IS WHOLE HOURS AT BOTH ENDS (FCST-04 L8). `since` is floored
 # and `until` was not, so a 48 h request made at any instant off the hour came
 # back 49 hours long. Graded at an OFF-THE-HOUR instant, because on the hour

goodwe_controller/test_price_history.py

@@ -34,6 +34,18 @@ cache cannot decay by being fed its own output.
        `PriceHistory(...)` and `.words` are unguarded in `Controller.__init__`
        - so the add-on did not start, and never self-healed.
 
+  AC5  (FCST-05) THE CACHE'S OWN CONTRACT HOLDS FOR EVERY BYTE AND EVERY ROW.
+       A file holding one non-UTF-8 byte no longer stops the add-on from
+       starting - `_load()` caught `(json.JSONDecodeError, OSError)` and
+       `UnicodeDecodeError` is neither, so it escaped `PriceHistory.__init__`
+       and `Controller.__init__` does not guard it. And a malformed `start`
+       under the day the feed is WRITING to no longer kills the tariff plan
+       for good: the write path shares `_point_of()` with the read path,
+       drops the row, says so, and `_save()` rewrites the record clean.
+       ⚠️ THE STATUS ROW IS PART OF THIS. It said `'2026-01-15 (1 h)'` about a
+       store whose every refresh was raising - health it did not have, in a
+       unit that was not its own (it counts POINTS).
+
 THE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
 
   * EVERY VERDICT IS DERIVED. The carried/published split is re-derived here
@@ -262,6 +274,13 @@ YESTERDAY_MWH = {h: 400.0 + 3 * h for h in range(24)}
 
 MARKUP, TAXES = 0.03, 0.06
 
+# ⚠️ FCST-05's HIGH, AS BYTES. A cache that is structurally valid JSON up to
+# the point where one byte is not UTF-8 - which is what a truncated write, a
+# half-flushed page or a foreign writer produces. Written with `open(..., "wb")`
+# and never through a text handle, or the fixture cannot hold the byte it is
+# named for. `0xff` is not a valid UTF-8 start byte in any position.
+_UTF8_BOMB = b'{"version": 1, "days": {"2026-01-14": "\xff\xfe"}}'
+
 
 def allin(mwh):
     """EUR/MWh as published -> the all-in EUR/kWh `assemble()` produces."""
@@ -896,6 +915,55 @@ check("an unreadable file: the reason is kept and reported, and the record is "
       and _h6.yesterday(NOW).kind == "no_data"
       and "unreadable" in why(_h6.yesterday(NOW)))
 
+# ⚠️ AND THE BYTE CLASS THAT FIXTURE CANNOT REACH - FCST-05's HIGH. The row
+# above is INVALID JSON but it is PERFECTLY VALID UTF-8, so it is decoded and
+# then refused by `json`, which raises `json.JSONDecodeError`. A file holding
+# one byte that is not UTF-8 at all fails EARLIER, in the decode `open(...,
+# encoding="utf-8")` performs before `json` sees a character, and
+# `UnicodeDecodeError` is `ValueError -> UnicodeError -> UnicodeDecodeError`:
+# NOT a `JSONDecodeError`, and NOT an `OSError` either. `_load()` caught
+# `(json.JSONDecodeError, OSError)`, so it escaped `_load()` -> `_open()` ->
+# `PriceHistory.__init__` - unguarded in `Controller.__init__` - AND THE ADD-ON
+# DID NOT START, on a cache whose own contract says "a corrupt cache must not
+# stop the controller".
+#
+# The precondition is asserted first, because it is the whole reason the check
+# above could not see this: if `"{ this is not json"` were undecodable, the two
+# fixtures would be one fixture and this would grade nothing (MUTATION.md
+# habit 1).
+check("the two unreadable-file fixtures are DIFFERENT FAILURES: the one above "
+      "is valid UTF-8 that is not JSON, and the one below is not UTF-8 at all. "
+      "A file that is merely bad JSON cannot reach the decode step, which is "
+      "why the suite had this hole",
+      open(_p6, "rb").read().decode("utf-8") == "{ this is not json"
+      and raises(lambda: _UTF8_BOMB.decode("utf-8"), UnicodeDecodeError))
+
+_d6b = tmpdir()
+_p6b = os.path.join(_d6b, "price_history.json")
+open(_p6b, "wb").write(_UTF8_BOMB)
+_h6b = derived(lambda: PriceHistory(_p6b))
+check(f"a single non-UTF-8 byte in the cache does NOT stop the store from "
+      f"being constructed {_h6b!r}. `PriceHistory(...)` is unguarded in "
+      f"`Controller.__init__`, so an exception here is an add-on that does not "
+      f"start - which is what shipped",
+      isinstance(_h6b, PriceHistory))
+check(f"...and it degrades the SAME WAY the bad-JSON file does: the reason is "
+      f"kept, named and reported "
+      f"{(_h6b.last_error if isinstance(_h6b, PriceHistory) else _h6b)!r}",
+      isinstance(_h6b, PriceHistory)
+      and "unreadable" in (_h6b.last_error or "")
+      and "codec can't decode byte" in (_h6b.last_error or "")
+      and _h6b.yesterday(NOW).kind == "no_data"
+      and "unreadable" in why(_h6b.yesterday(NOW)))
+_w6b = derived(lambda: _h6b.words)
+_plan6b = derived(lambda: run(resolve_tariff(
+    CFG, NOW, _until, transport=transport_for(_body_today),
+    now=NOW, history=_h6b)).origin)
+check(f"...and NOTHING downstream raises on it either: not `words` ({_w6b!r}), "
+      f"not the shipped `resolve_tariff()` path ({_plan6b!r}). §6: absent "
+      f"machinery must not disable the loop, and a JSON file is machinery",
+      isinstance(_w6b, str) and _plan6b == "provider")
+
 _p7 = os.path.join(tmpdir(), "price_history.json")
 json.dump({"version": SCHEMA + 99, "days": {}}, open(_p7, "w", encoding="utf-8"))
 _h7 = PriceHistory(_p7)
@@ -1008,11 +1076,49 @@ check("a healthy but EMPTY store says something different, because it is a "
       and words_of(_h_words).startswith("empty")
       and "nothing to carry forward" in words_of(_h_words))
 _ph.PriceHistory._remember(_h_words, series_of(YDAY_H, YESTERDAY_MWH), NOW)
-check(f"...and a store holding a day names the DATE and the HOUR COUNT "
-      f"{words_of(_h_words)!r} - 'a cache exists' and 'the cache holds "
-      f"yesterday, all 24 hours of it' are different facts, and only the "
-      f"second says the horizon will be full tomorrow morning",
-      words_of(_h_words) == f"{YDAY_D} (24 h)")
+check(f"...and a store holding a day names the DATE, the POINT COUNT and the "
+      f"HOUR COUNT {words_of(_h_words)!r} - 'a cache exists' and 'the cache "
+      f"holds yesterday, all 24 hours of it' are different facts, and only "
+      f"the second says the horizon will be full tomorrow morning",
+      words_of(_h_words) == f"{YDAY_D} (24 points over 24 h)")
+
+# ⚠️ AND THE TWO NUMBERS ARE NOT THE SAME NUMBER - FCST-05's LOW. The row
+# shipped as `f"{d} ({_n_points(rec)} h)"`, which counts POINTS and prints them
+# under the HOUR's name. On an hourly feed the two are equal and nothing shows;
+# on the quarter-hourly feed this add-on already supports (`_published_by_hour`
+# keeps a TUPLE per hour precisely for it, and §7 below drives one) the row
+# read '2026-01-13 (96 h)' - 96 "h" for a 24-hour day. That is the points/hours
+# conflation H1 was raised for, on the row H1's own ticket shipped.
+#
+# MUTATION.md's rule: this check asserts a DIFFERENCE, so the fixture is run
+# with the two numbers actually made different. 96 != 24, asserted first.
+_h_q = store()
+_QUARTERS = [u + timedelta(minutes=15 * q) for u in YDAY_H for q in range(4)]
+_ph.PriceHistory._remember(
+    _h_q, PriceRead.series(
+        [PricePoint(start=brussels(u),
+                    eur_per_kwh=allin(YESTERDAY_MWH[brussels(u).hour]
+                                      + brussels(u).minute),
+                    provisional=False) for u in _QUARTERS],
+        PriceProduct.DAY_AHEAD, "generic_dayahead", NOW),
+    NOW)
+_qw = words_of(_h_q)
+check(f"the quarter-hourly fixture really is sub-hourly and its two counts "
+      f"really differ ({len(_QUARTERS)} points, "
+      f"{len({brussels(u).hour for u in _QUARTERS})} local hours), or 'points' "
+      f"and 'hours' are the same observation and the row grades nothing",
+      len(_QUARTERS) == 96
+      and len({brussels(u).hour for u in _QUARTERS}) == 24
+      and len({u.minute for u in _QUARTERS}) == 4)
+check(f"...and the row reports BOTH, each under its own unit: {_qw!r}. "
+      f"A day of 96 quarter-hourly prints is 96 points over 24 h, and the "
+      f"figure that says whether tomorrow's fill has a whole day to carry is "
+      f"the HOUR one",
+      _qw == f"{YDAY_D} (96 points over 24 h)")
+check(f"...and specifically the row does NOT say '96 h' {_qw!r} - a point "
+      f"count wearing the hour's name is H1's defect, and this row is where "
+      f"an operator would have read it",
+      isinstance(_qw, str) and "(96 h)" not in _qw and " (96 points " in _qw)
 check(f"a file that only PARTLY loaded says so in the row {_w14!r}, or the "
       f"days it dropped read as days that were simply never written",
       isinstance(_w14, str) and (_h14.last_error or "") in _w14
@@ -1023,7 +1129,7 @@ check("...and `words` never raises, whatever the file holds. It is read once "
       "from Controller.__init__ and again on every status render, and neither "
       "call site guards it - an exception here is an add-on that does not start",
       all(isinstance(words_of(h), str)
-          for h in (_missing, _h6, _h7, _h14, _h_words)))
+          for h in (_missing, _h6, _h6b, _h7, _h14, _h_words)))
 
 # ⚠️ AND A FAILED WRITE LEAVES NOTHING BEHIND. `_save()` writes to a `mkstemp`
 # and renames - atomic, so a power cut cannot leave a truncated file that
@@ -1067,6 +1173,7 @@ check(f"...and a SUCCESSFUL write leaves exactly the one file "
 # provider series with no invented points and no §11.4 degradation.
 for _label, _hist in (("no durable volume", _missing),
                       ("unreadable file", _h6),
+                      ("a non-UTF-8 byte in the file", _h6b),
                       ("wrong schema", _h7)):
     _plan = run(resolve_tariff(CFG, NOW, _until,
                                transport=transport_for(_body_today), now=NOW,
@@ -1080,6 +1187,387 @@ for _label, _hist in (("no durable volume", _missing),
           and not any(p.provisional for p in _plan.read.points))
 
 
+# --------------------------------------------------------------------------- #
+print()
+print("5c. FCST-05 - the WRITE path is as defensive as the read path")
+
+# THE DEFECT. `_remember()` merged a day's held points with
+#
+#     held = {q["start"]: q for q in raw
+#             if isinstance(q, dict) and "start" in q}
+#     rec["points"] = sorted(held.values(), key=lambda q: q["start"])
+#
+# which trusts a value read off /data to be a HASHABLE, SORTABLE STRING. It is
+# whatever is in the file. `_usable_days()` required `rec["points"]` to be a
+# LIST and never looked at the elements, and `_parse_points()` - the READ path -
+# was properly defensive, so `yesterday()` survived and the §5 `_p8` fixture
+# passed. Two standards for what a stored point is; the weaker one on the path
+# that WRITES.
+#
+# ⚠️ AND THE PLACEMENT IS THE WHOLE FIXTURE. `_p8` files its malformed rows
+# under a day the feed is NOT writing to, so `_remember()` never touches that
+# record and the defect is invisible. What reaches it is a malformed row under
+# TODAY - the record the running feed merges into on every refresh - so that is
+# asserted before anything is graded (MUTATION.md habit 1), and the harmless
+# placement is run as the accepted baseline it varies from (habit 7).
+#
+# ⚠️ AND THE BLAST RADIUS IS WHY IT IS NOT MERELY UNTIDY. `carry()` is reached
+# from `resolve_tariff()` from `run_tariff_refresh`, which catches `Exception`:
+# the add-on stays UP and EVERY refresh raises, forever. The plan retires after
+# TARIFF_MAX_AGE_S and arbitrage and the cheap window are off permanently, with
+# no self-heal, because `_remember()` raises before it reaches `_save()`.
+
+_BAD_STARTS = (
+    ("an int", 7),                     # TypeError: '<' between 'str' and 'int'
+    ("a dict", {"a": 1}),              # TypeError: unhashable as a dict key
+    ("null", None),                    # TypeError: '<' between 'str', 'NoneType'
+    ("a list", [1, 2]),                # TypeError: unhashable as a dict key
+    ("a string that is not an instant", "nope"),   # sorts fine, never a point
+)
+
+
+def _seeded(day, bad_start):
+    """A store on disk holding ONE malformed row under `day`, plus a good one."""
+    d = tmpdir()
+    path = os.path.join(d, "price_history.json")
+    good = {"start": brussels(at_local(day, 5)).isoformat(),
+            "eur_per_kwh": allin(YESTERDAY_MWH[5]), "provisional": False}
+    json.dump({"version": SCHEMA, "days": {day: {
+        "product": "dynamic_dayahead", "provider": "generic_dayahead",
+        "points": [good, {"start": bad_start, "eur_per_kwh": 0.2,
+                          "provisional": False}]}}},
+              open(path, "w", encoding="utf-8"))
+    return path, PriceHistory(path)
+
+
+_TODAY_FEED = series_of(TODAY_H, TODAY_MWH)
+check(f"the fixture files its malformed row under {TODAY_D}, which is exactly "
+      f"the Brussels day the feed being carried writes to "
+      f"{sorted({_ph._bdate(p.start) for p in _TODAY_FEED.points})} - under any "
+      f"other day `_remember()` never opens the record and this whole section "
+      f"grades nothing, which is why the suite already had this hole",
+      {_ph._bdate(p.start) for p in _TODAY_FEED.points} == {TODAY_D})
+
+# The accepted baseline: the SAME row, one day over. Shipped and broken alike
+# pass this, and that is the point of running it (habit 7).
+_pB, _hB = _seeded(YDAY_D, 7)
+_cB = derived(lambda: _hB.carry(_TODAY_FEED, since=NOW, until=_until, stamp=NOW))
+check("baseline: the same malformed row under a day the feed is NOT writing "
+      "to is carried without incident - one variable moves between this and "
+      "every check below, and it is the DAY",
+      not isinstance(_cB, _Raised) and _cB.kind == "series")
+
+for _shape, _bad in _BAD_STARTS:
+    _pM, _hM = _seeded(TODAY_D, _bad)
+    _wload = words_of(_hM)
+    _c1 = derived(lambda h=_hM: h.carry(_TODAY_FEED, since=NOW, until=_until,
+                                        stamp=NOW))
+    _saw = repr(_c1) if isinstance(_c1, _Raised) else _c1.kind
+    check(f"a `start` that is {_shape} ({_bad!r}) under TODAY's own record "
+          f"does not take the refresh down: `carry()` returns a series "
+          f"({_saw}). It raised "
+          f"`TypeError` out of the shipped path, and `run_tariff_refresh` "
+          f"catches `Exception`, so the add-on stayed up while every refresh "
+          f"failed for good",
+          not isinstance(_c1, _Raised) and _c1.kind == "series")
+    # ⚠️ THE VERDICT IS NOT THE CLAIM. "It did not raise" is satisfied by a
+    # store that silently kept the poison row; what this section promises is
+    # that the row is GONE FROM THE FILE, so the next start is clean.
+    _disk = json.load(open(_pM, encoding="utf-8"))
+    _rows = _disk["days"][TODAY_D]["points"]
+    # DERIVED, not a constant: the surviving rows are exactly the DISTINCT
+    # instants the feed published for that day plus the fixture's one good
+    # seeded row - which is 05:00, an hour the feed also prints, so it merges.
+    _want = len({p.start.astimezone(UTC) for p in _TODAY_FEED.points}
+                | {at_local(TODAY_D, 5).astimezone(UTC)})
+    check(f"...and the row is DROPPED FROM /data, not carried: {len(_rows)} "
+          f"row(s) on disk against {_want} distinct instants, every one of "
+          f"them a point. `_remember()` used to raise BEFORE `_save()`, so "
+          f"there was no self-heal at all - the store came back poisoned on "
+          f"every restart",
+          all(_ph._point_of(r) is not None for r in _rows)
+          and len(_rows) == _want)
+    # ...and the store stops reporting health, at BOTH moments it can be read:
+    # when it is opened, and after the refresh that cleaned it.
+    # ⚠️ THE COUNTS ARE ASSERTED TOO, NOT ONLY THE NOTE. The seeded record holds
+    # ONE point and one row that is not one, so a row printing "2 points" would
+    # be reporting the poison as held prices - which is the half of this defect
+    # that reached the operator: `words` said "2026-01-15 (1 h)" while every
+    # tariff refresh was dying on that record.
+    check(f"...and the status row does not call that store healthy - at open "
+          f"{_wload!r}: ONE point held, and the row that is not a point named "
+          f"rather than counted",
+          isinstance(_wload, str)
+          and _wload.startswith(f"{TODAY_D} (1 point over 1 h) [")
+          and "1 malformed point row(s) in" in _wload)
+    # ⚠️ AND THE TWO NOTES ARE GRADED APART, BECAUSE THEY MASK EACH OTHER
+    # OTHERWISE (MUTATION.md TEST-16). `_usable_days()` records what it FOUND
+    # at open and `_remember()` records what it DROPPED on the refresh; both
+    # land in `last_error`, so a check asking only "does the row mention a
+    # malformed point" is answered by whichever rung is still standing - the
+    # first draft of this check was exactly that, and deleting `_remember()`'s
+    # note left the whole 29-file suite green at -1 B. The tense is the
+    # difference an operator reads: "in" is still there, "dropped from" is
+    # gone, and after the refresh only the second is true.
+    _wafter = words_of(_hM)
+    check(f"...nor after the refresh that dropped it {_wafter!r}. It said "
+          f"'{TODAY_D} (1 h)' - the status row reporting a full cache while "
+          f"every tariff refresh was dying on it",
+          isinstance(_wafter, str)
+          and "malformed point row(s) dropped from" in _wafter
+          and TODAY_D in _wafter)
+    check(f"...and the row MOVED from what was found to what was dropped "
+          f"{_wload.split('[')[-1]!r} -> {_wafter.split('[')[-1]!r}, so the "
+          f"two notes are not one note read twice",
+          isinstance(_wload, str) and isinstance(_wafter, str)
+          and "malformed point row(s) in" in _wload
+          and "malformed point row(s) in" not in _wafter
+          and "malformed point row(s) dropped from" not in _wload)
+    # ...and a SECOND refresh over the same store is clean, which is the state
+    # a restarted add-on comes up in.
+    _hM2 = PriceHistory(_pM)
+    check(f"...and the store reopened from that file is CLEAN: the whole day "
+          f"held and nothing to report {words_of(_hM2)!r}, so the note clears "
+          f"itself rather than becoming permanent furniture",
+          words_of(_hM2) == f"{TODAY_D} ({_want} points over {_want} h)"
+          and _hM2.last_error is None)
+
+# ...and the shipped path, end to end, through the caller that catches
+# `Exception` and would otherwise have hidden this for good.
+_pP, _hP = _seeded(TODAY_D, 7)
+_o1 = derived(lambda: run(resolve_tariff(
+    CFG, NOW, _until, transport=transport_for(_body_today), now=NOW,
+    history=_hP)).origin)
+_o2 = derived(lambda: run(resolve_tariff(
+    CFG, NOW, _until, transport=transport_for(_body_today), now=NOW,
+    history=_hP)).origin)
+check(f"the SHIPPED `resolve_tariff()` path survives it TWICE ({_o1!r}, "
+      f"{_o2!r}) - once for the refresh that finds the poison and once for "
+      f"the one after it. A guard that refuses once and dies on the next call "
+      f"is a delay, not a guard (MUTATION.md habit 9)",
+      _o1 == "provider" and _o2 == "provider")
+
+# ⚠️ AND THE MERGE KEY IS THE INSTANT, WHICH IS WHAT RULE 4 SAYS. The old key
+# was `q["start"]` - the SPELLING - so the same instant written at two offsets
+# was held twice, and the lexicographic sort beside it put the second 02:00 of
+# the 25-hour October day (`+01:00`) BEFORE the first (`+02:00`). Aware
+# datetimes hash and compare by instant, so the validated `start` is the key.
+_d15 = tmpdir()
+_p15 = os.path.join(_d15, "price_history.json")
+_alt = at_local(YDAY_D, 6)
+json.dump({"version": SCHEMA, "days": {YDAY_D: {
+    "product": "dynamic_dayahead", "provider": "generic_dayahead",
+    "points": [{"start": _alt.astimezone(UTC).isoformat(),
+                "eur_per_kwh": 0.999, "provisional": False}]}}},
+          open(_p15, "w", encoding="utf-8"))
+_h15 = PriceHistory(_p15)
+check(f"the two spellings really are one instant and really are two strings "
+      f"({_alt.astimezone(UTC).isoformat()!r} vs "
+      f"{brussels(_alt).isoformat()!r}), or 'merged by instant' and 'merged by "
+      f"spelling' are the same observation",
+      _alt.astimezone(UTC).isoformat() != brussels(_alt).isoformat()
+      and _alt.astimezone(UTC) == brussels(_alt))
+_ph.PriceHistory._remember(
+    _h15, series_of([_alt], YESTERDAY_MWH), NOW)
+_y15 = _h15.yesterday(NOW)
+_n15 = len(_y15.points) if _y15.kind == "series" else _y15.kind
+_v15 = _y15.points[0].eur_per_kwh if _y15.kind == "series" else None
+check(f"...and a revision of that hour REPLACES it rather than sitting beside "
+      f"it: {_n15} point held, priced {_v15} and not 0.999. Rule 4 says "
+      f"'MERGE BY INSTANT'; the key was the string",
+      _y15.kind == "series" and len(_y15.points) == 1
+      and abs(_y15.points[0].eur_per_kwh - allin(YESTERDAY_MWH[6])) < 1e-9)
+
+
+# --------------------------------------------------------------------------- #
+print()
+print("5d. FCST-06 - the OTHER way to be a store that cannot persist")
+
+# THE DEFECT. Section 5 above grades "a store that cannot persist records
+# nothing" against `not ok` - the missing-durable-volume SYMPTOM - and that is
+# the only condition the rule was ever enforced on. `ok` stays TRUE on a volume
+# that is present and cannot be written: a read-only remount, a full disk. In
+# that state `_remember()` had already populated `self._days`, `_save()` logged
+# and returned, and the store went on answering a series for the rest of the
+# process's life. Measured on 5059e28, twice, with a `/data` bind-mounted
+# read-only and again on a full tmpfs:
+#
+#     last_error = 'could not write ([Errno 30] Read-only file system)'
+#     words      = '2026-01-14 (24 points over 24 h); 2026-01-15 (24 points
+#                   over 24 h) [could not write (...)]'
+#     yesterday()= series, 24 points
+#     ON DISK    = {}                         <- and gone on the next restart
+#
+# `words` and `yesterday()` were describing two different stores, which is
+# exactly what section 5's own comment refuses - reached by the condition the
+# rule did not name.
+#
+# ⚠️ THE LEVERS ARE REAL FAILURES, NOT PATCHED ONES, AND THERE ARE TWO OF THEM
+# ON PURPOSE. The fix is written against the OUTCOME - `_save()` answering "no"
+# - and not against an errno, so one cause would grade a coincidence. A
+# DIRECTORY where the file goes fails `os.replace`; a volume that moves out
+# from under an already-open store fails `mkstemp`. Different syscall,
+# different errno, one answer (habit 10: grade the family).
+
+def npts(read):
+    """A read's point count, or "no_data". Never raises.
+
+    ⚠️ NOT `len(read.points)`. On a `no_data` that attribute RAISES `Absence`
+    by design, so a mutation that empties a store where this section expects
+    one would take the harness down with a traceback instead of reporting a
+    FAIL - and MUTATION.md's rule is that "it answered nothing" and "it broke
+    on the way to answering" must not grade the same. Found by driving M4.
+    """
+    return read.match(series=lambda r: len(r.points),
+                      no_data=lambda _r: "no_data")
+
+
+_d16 = tmpdir()
+_p16 = os.path.join(_d16, "price_history.json")
+_h16 = PriceHistory(_p16)
+os.mkdir(_p16)                   # `os.replace` cannot complete onto a directory
+check(f"the fixture is an `ok` store on a directory that IS there "
+      f"(ok={_h16.ok}, last_error={_h16.last_error!r}) - not the "
+      f"no-durable-volume case section 5 already grades, or this section is "
+      f"that one again (habit 1)",
+      _h16.ok and _h16.last_error is None and os.path.isdir(_d16))
+_carried16 = _h16.carry(series_of(YDAY_H + TODAY_H, TODAY_MWH),
+                        since=NOW, until=_until, stamp=NOW)
+_y16 = _h16.yesterday(NOW)
+check(f"a write that could not complete is REPORTED and leaves the store "
+      f"`ok` (ok={_h16.ok}, {(_h16.last_error or '')[:44]!r}). `ok` is about "
+      f"the DIRECTORY, and a full volume and a read-only remount both leave "
+      f"it True - which is why testing `ok` alone could not see this",
+      _h16.ok and "could not write" in (_h16.last_error or ""))
+check(f"...and the file really does not hold the day "
+      f"(isfile={os.path.isfile(_p16)}), or 'it answers what is not there' is "
+      f"not a measurement (habit 1)",
+      not os.path.isfile(_p16))
+check(f"A STORE THAT COULD NOT WRITE RECORDS NOTHING, exactly as one with no "
+      f"durable volume does: it holds {sorted(_h16._days)} and `yesterday()` "
+      f"is {_y16.kind!r}. Retaining the merge is an in-memory yesterday that "
+      f"answers all day and is gone on the next restart - the "
+      f"silently-forgetting cache this module's docstring exists to refuse",
+      _h16._days == {} and _y16.kind == "no_data")
+check(f"...and `words` and `yesterday()` AGREE about that store "
+      f"{_h16.words[:44]!r}: neither names a day the file does not hold, and "
+      f"both name the write failure. They disagreed - `words` printed two "
+      f"full days BESIDE the error while `yesterday()` answered 24 points",
+      YDAY_D not in _h16.words and TODAY_D not in _h16.words
+      and "could not write" in _h16.words
+      and _y16.kind == "no_data" and "could not write" in why(_y16))
+check(f"...and it DEGRADES rather than raising: the carry still returned a "
+      f"real series ({_carried16.kind!r}, {npts(_carried16)} points) with "
+      f"nothing carried. §6 and this module's opening - an add-on that will "
+      f"not start because /data filled up is a WORSE defect than the one this "
+      f"section is about",
+      _carried16.kind == "series"
+      and not _carried16.match(
+          series=lambda r: any(p.provisional for p in r.points),
+          no_data=lambda _r: True))
+_plan16 = derived(lambda: run(resolve_tariff(
+    CFG, NOW, _until, transport=transport_for(_body_today), now=NOW,
+    history=_h16)).origin)
+check(f"...and the SHIPPED `resolve_tariff()` path is unmoved by it "
+      f"({_plan16!r}): a provider plan, no §11.4 degradation, on a store that "
+      f"cannot write a byte",
+      _plan16 == "provider")
+# Habit 9: the same wrong input a SECOND time. A refresh runs 48 times a day,
+# and a volume that is full at 14:00 is still full at 14:30.
+_h16.carry(series_of(YDAY_H, YESTERDAY_MWH), since=NOW, until=_until,
+           stamp=NOW)
+check(f"...and the SECOND failing refresh behaves the same rather than "
+      f"accumulating: still {sorted(_h16._days)} held and "
+      f"{_h16.yesterday(NOW).kind!r}. The write fails 48 times a day, so the "
+      f"call that matters is the one after the first (habit 9)",
+      _h16._days == {} and _h16.yesterday(NOW).kind == "no_data")
+
+# ⚠️ AND WHAT IS KEPT IS THE LAST STATE THAT IS ON DISK, NOT AN EMPTY ONE. The
+# MERGE is undone, not the store. A site whose volume fills at 14:00 goes on
+# answering the yesterday it wrote at 13:30 - that record survives a restart,
+# so answering it is not a promise the store cannot keep. What it stops doing
+# is adding to it. This is also the fixture that separates a deep snapshot from
+# `dict(self._days)`: a one-level copy shares the day RECORD the merge rebinds
+# `points` on, so the "restored" store would answer the 24 points the merge
+# made and the file never took.
+_d17 = os.path.join(tmpdir(), "data")
+os.mkdir(_d17)
+_p17 = os.path.join(_d17, "price_history.json")
+_h17 = PriceHistory(_p17)
+_ph.PriceHistory._remember(_h17, series_of(YDAY_H[:12], YESTERDAY_MWH), NOW)
+_first17 = _h17.yesterday(NOW)
+check(f"a HALF day is persisted first ({npts(_first17)} points, "
+      f"last_error={_h17.last_error!r}), or 'the merge is undone and the "
+      f"store is not' grades nothing (habit 1)",
+      _h17.last_error is None and os.path.isfile(_p17)
+      and npts(_first17) == 12)
+_persisted17 = open(_p17, encoding="utf-8").read()
+_moved17 = _d17 + ".moved"
+os.rename(_d17, _moved17)        # the volume goes out from under an OPEN store:
+                                 # `mkstemp` fails, a different call and a
+                                 # different errno from the fixture above, and
+                                 # the bytes already written stay intact
+_ph.PriceHistory._remember(_h17, series_of(YDAY_H[12:], YESTERDAY_MWH), NOW)
+_y17 = _h17.yesterday(NOW)
+_n17 = npts(_y17)
+check(f"the second lever failed the write by a DIFFERENT route "
+      f"{(_h17.last_error or '')[:44]!r} - `mkstemp`, not `os.replace`, and "
+      f"ENOENT, not EISDIR. The rule is enforced on the outcome, so it must "
+      f"be driven by more than one cause (habit 10)",
+      "could not write" in (_h17.last_error or "")
+      and "No such file" in (_h17.last_error or ""))
+check(f"...and the store keeps the last state that IS ON DISK, not an empty "
+      f"one: {_n17} points held - the 12 it wrote before the volume went, not "
+      f"the 24 the unwritten merge made, and not nothing. The MERGE is undone; "
+      f"the store is not",
+      npts(_y17) == 12)
+_ph.PriceHistory._remember(_h17, series_of(YDAY_H[12:], YESTERDAY_MWH), NOW)
+_y17b = _h17.yesterday(NOW)
+check(f"...and again on the second failing refresh ({npts(_y17b)} points), "
+      f"because a volume that has gone is still gone 30 minutes later "
+      f"(habit 9)",
+      npts(_y17b) == 12)
+os.rename(_moved17, _d17)        # the volume comes back: what a RESTART finds
+_r17 = PriceHistory(_p17).yesterday(NOW)
+check(f"...and that is EXACTLY what a restart finds ({npts(_r17)} points, the "
+      f"file byte-for-byte as it was left), which is the whole claim: the "
+      f"store answers what survives a restart and nothing else",
+      open(_p17, encoding="utf-8").read() == _persisted17
+      and npts(_r17) == 12
+      and _r17.match(series=lambda a: _y17.match(
+          series=lambda b: [p.start for p in a.points]
+          == [p.start for p in b.points],
+          no_data=lambda _r: False), no_data=lambda _r: False))
+
+# ⚠️ AND "CANNOT PERSIST" IS ONE PREDICATE, ASKED ONCE. FCST-04's M4 and
+# FCST-05's HIGH were both a rule enforced on the conditions somebody thought
+# of rather than on the family; this is the third of the shape and the last
+# place it can be spelt twice. `_save()` returns whether the file now holds
+# what memory holds - no volume is one False, an `OSError` is another - and
+# `_remember()` keeps only what that answer says was kept.
+check(f"`_save()` answers for BOTH symptoms and is the only thing that does: "
+      f"no durable volume -> {_ph.PriceHistory._save(_missing)!r}, a write "
+      f"that fails -> {_ph.PriceHistory._save(_h16)!r}, a store that "
+      f"persisted -> {_ph.PriceHistory._save(_hok)!r}",
+      _ph.PriceHistory._save(_missing) is False
+      and _ph.PriceHistory._save(_h16) is False
+      and _ph.PriceHistory._save(_hok) is True)
+check(f"...and asking it once leaves the two symptoms with ONE behaviour: "
+      f"no volume holds {sorted(_missing._days)}/"
+      f"{_missing.yesterday(NOW).kind!r} and a failed write holds "
+      f"{sorted(_h16._days)}/{_h16.yesterday(NOW).kind!r}. A third cause "
+      f"nobody has thought of needs no third branch - which is how this defect "
+      f"arrived, twice",
+      (_missing._days, _missing.yesterday(NOW).kind)
+      == (_h16._days, _h16.yesterday(NOW).kind))
+check("...and `_remember()` no longer tests a symptom of its own: the source "
+      "holds no `self.ok` test between the docstring and the merge. This one "
+      "IS about the code's shape - two places testing two symptoms is the "
+      "defect, not a behaviour (habit 4's stated exception)",
+      "if not self.ok:" not in inspect.getsource(_ph.PriceHistory._remember)
+      and "if not self.ok:" in inspect.getsource(_ph.PriceHistory._save))
+
+
 # --------------------------------------------------------------------------- #
 print()
 print("6. rule 4 - hours accumulate within a day; they do not replace it")