FCST-06: "cannot persist" becomes one question, asked once, about the outcome #75

Merged
admin merged 1 commits from FCST-06 into release/1.0 2026-09-10 08:52:10 +02:00
Owner

The defect

app/price_history.py's module docstring states its rule unconditionally:

⚠️ AND THAT RULE IS CARRIED OUT IN _remember(), NOT ONLY IN _save(). … A store that cannot persist now records nothing, and both of its answers say the same thing.

That was FCST-04's M4 fix, and it was enforced against not self.ok only — one symptom of not persisting: the durable volume is missing. ok is about the directory, and a read-only remount and a full disk both leave it True.

There is a second way to be a store that cannot persist: ok=True, /data present, and _save() failing. In that state _remember() had already populated self._days, _save() logged and returned, and yesterday() went on answering a series for the rest of the process's life — gone on the next restart, with nothing saying so. That is the silently-forgetting cache the docstring rejects, reached by the condition the rule did not name.

Not urgent, and that shaped the fix. No shipped path reaches it: /data is the Supervisor's durable volume and this module is its only writer. The failure was already logged and in last_error. Nothing unsafe happened meanwhile — real prices from a real feed, every carried point still flagged provisional. The defect is that a guarantee the module states about itself did not hold in one condition.

The reproduction

Driven on 5059e28 against two real volumes — not a monkeypatch — inside the checks image, with the store's carry() given a feed covering yesterday and today:

case 1: /data bind-mounted read-only case 2: full tmpfs case 3: control, healthy /data
ok True True True
last_error could not write ([Errno 30] Read-only file system) could not write ([Errno 28] No space left on device) None
words 2026-01-14 (24 points over 24 h); 2026-01-15 (24 points over 24 h) [could not write (…)] same 2026-01-14 (…); 2026-01-15 (…)
yesterday() series, 24 points series, 24 points series, 24 points
on disk {} no file at all {'2026-01-14': 24, '2026-01-15': 24}
after a restart — — series, 24 points

words and yesterday() were describing two different stores. The control is asserted first, so cases 1–2 are not grading a fixture that never persisted anything.

After the fix, same script, same volumes:

case 1 case 2 case 3 (control)
words empty - could not write ([Errno 30] Read-only file system) - nothing to carry forward yet empty - could not write ([Errno 28] No space left on device) - … unchanged
yesterday() no_data, reason names the write failure no_data, same unchanged: series, 24 points, surviving a restart
verdict DEFECT ABSENT DEFECT ABSENT control still persists

Which acceptance route, and why

Route 1: _remember() refuses to retain what it could not write. The docstring's claim stands exactly as written, because it was the right claim — the code was the half that was wrong. Weakening the doc to describe the not ok-only behaviour would have documented a cache that forgets without saying so as intended, and that is the one thing this module's third warning exists to refuse.

The thing actually worth judgement: one derived predicate — yes

"Cannot persist" is now one question, asked once, about the outcome. The reason is not tidiness: the enumeration cannot be completed. FCST-04's M4 enforced the rule on the symptom in front of it; FCST-05's HIGH was the same shape (an except clause naming the one content-failure member somebody had met instead of the ValueError family). This is the third instance in one module, and it is the last place in this file where the rule can be spelt twice.

The design point is where the one question can be asked. It cannot be answered before the attempt — a directory that existed and was writable a moment ago is not a write that completed, and any pre-check (os.access, a probe write) would be a third symptom test wearing a predicate's name, plus a value that can go stale between 08:00 and 14:00. So it is answered after, once, by the only thing holding the evidence:

_save() -> bool      # True only if the file now holds what memory holds

No durable volume returns False without touching the disk; an OSError from the write returns False after recording why. _remember() reads that one answer and keeps only what it says was kept. The not self.ok test in _remember() is gone — not moved, gone — and a check asserts its absence from the source. A fourth cause needs no fourth branch, because none of them is asked about.

What is undone is the MERGE, not the store. The state restored is the last one that is on disk, so a site whose volume fills at 14:00 goes on answering the yesterday it wrote at 13:30 — that record survives a restart, so answering it is not a promise the store cannot keep. It simply stops adding to it. Clearing outright would throw away a good yesterday in order to punish a failed write, on exactly the morning the operator is already dealing with a full disk.

That gives the file the invariant its rule was always describing, stated once in the class docstring: self._days never holds a point the file does not. (Not "mirrors the file" — _usable_days() drops records the file still holds; the only direction that can invent a yesterday is the one named.)

⚠️ The snapshot is copy.deepcopy, not dict(self._days). A one-level copy shares the day records, and the merge rebinds points on the ones it touches. Measured with a shallow copy: a store holding 12 persisted points answered 24 after a failed merge.

⚠️ A failed write is still not fatal, per the ticket's constraint and §6: the store degrades, it does not raise. resolve_tariff() still returns a provider plan with no §11.4 degradation on a store that cannot write a byte, and that is checked.

The mutation table

Purged tree before every run (TEST-15), PYTHONDONTWRITEBYTECODE=1, mutated line greped off disk before each run. Both suites that touch this module were run each time.

mutation Δ bytes test_price_history.py test_static_tariff.py
M1 self._days = before → self._days = dict(self._days) (the merge is retained — the shipped behaviour) +10 B 9 red, incl. A STORE THAT COULD NOT WRITE RECORDS NOTHING… and FCST-04's own a store with no durable volume records NOTHING… green
M2 _save()'s except OSError returns True −1 B 9 red, incl. `_save()` answers for BOTH symptoms… green
M3 _save() returns True on not self.ok −1 B 3 red, incl. FCST-04's M4 check green
M4 self._days = before → self._days = {} (wipe instead of undo) −4 B 3 red, incl. …keeps the last state that IS ON DISK, not an empty one green
M5 copy.deepcopy(self._days) → dict(self._days) −9 B 3 red — 24 points held, not 12 green
M6 if not self.ok: return put back into _remember() +43 B 1 red: …_remember() no longer tests a symptom of its own green

No survivors. M6 exists to prove the one source-shape check in the section is not decoration — behaviour is unchanged under it, and it is red on exactly that check.

What M4 exposed about the instrument, recorded in MUTATION.md

M4 first printed 1 FAIL. The true figure is 3. An f-string operand in the next check read len(read.points), and .points on a no_data raises Absence by design, so the harness died mid-section with the remaining checks ungraded. That is why()'s and derived()'s existing rule — "it answered nothing" and "it broke on the way to answering" must not grade the same — arriving in what a check prints rather than in what it asserts, which is the half that looks like formatting. npts() is now beside why(); re-driving M4 moved 1 FAIL to 3. Had the traceback landed one check earlier, M4 would have produced no named red at all — a result that reads like a broken harness and gets retried rather than read.

Also in this branch

  • DOCS.md's "where there is no /data at all the store records nothing" is widened to the condition the code now holds to, with the "a failed write does not throw away what is already in the file" half stated explicitly.
  • DECISIONS/FCST-06-cannot-persist-is-one-question.md — the two decisions, including what was not done (no can_persist() helper, no probe write at _open(), not fatal) and why.
  • CHANGELOG.md under the existing ## Unreleased convention. No version number picked — that is the open decision the Unreleased block names, and it depends on a fact this branch cannot observe.

Not fixed, deliberately

  • The not ok store still does the merge work before throwing it away. With the single predicate, a store with no durable volume merges into self._days, is refused by _save(), and is rolled back. Adding a fast-path not self.ok return would restore the second symptom test this ticket removes, to save a few dozen dict operations on a host that has no /data — dev hosts and the checks container only, since production always takes the successful path. Judged not worth re-opening the defect's shape for.
  • Nothing about /data/state.json, PeakStore, or any other writer. This ticket is one module.

29/29 test files green on the final tree. Nothing deployed — no ha addons update, no writes to /addons/ or the inverter.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU

## The defect `app/price_history.py`'s module docstring states its rule unconditionally: > ⚠️ AND THAT RULE IS CARRIED OUT IN `_remember()`, NOT ONLY IN `_save()`. … A store that cannot persist now records nothing, and both of its answers say the same thing. That was FCST-04's M4 fix, and it was enforced against **`not self.ok`** only — one **symptom** of not persisting: the durable volume is missing. `ok` is about the **directory**, and a read-only remount and a full disk both leave it `True`. There is a second way to be a store that cannot persist: `ok=True`, `/data` present, and `_save()` failing. In that state `_remember()` had already populated `self._days`, `_save()` logged and returned, and `yesterday()` went on answering **a series** for the rest of the process's life — gone on the next restart, with nothing saying so. That is the silently-forgetting cache the docstring rejects, reached by the condition the rule did not name. **Not urgent, and that shaped the fix.** No shipped path reaches it: `/data` is the Supervisor's durable volume and this module is its only writer. The failure was already logged and in `last_error`. Nothing unsafe happened meanwhile — real prices from a real feed, every carried point still flagged provisional. The defect is that a guarantee the module states about itself did not hold in one condition. ## The reproduction Driven on `5059e28` against **two real volumes** — not a monkeypatch — inside the checks image, with the store's `carry()` given a feed covering yesterday and today: | | case 1: `/data` bind-mounted **read-only** | case 2: **full** tmpfs | case 3: control, healthy `/data` | |---|---|---|---| | `ok` | `True` | `True` | `True` | | `last_error` | `could not write ([Errno 30] Read-only file system)` | `could not write ([Errno 28] No space left on device)` | `None` | | `words` | `2026-01-14 (24 points over 24 h); 2026-01-15 (24 points over 24 h) [could not write (…)]` | same | `2026-01-14 (…); 2026-01-15 (…)` | | `yesterday()` | **series, 24 points** | **series, 24 points** | series, 24 points | | **on disk** | **`{}`** | **no file at all** | `{'2026-01-14': 24, '2026-01-15': 24}` | | after a restart | — | — | series, 24 points | `words` and `yesterday()` were describing two different stores. The control is asserted first, so cases 1–2 are not grading a fixture that never persisted anything. **After the fix, same script, same volumes:** | | case 1 | case 2 | case 3 (control) | |---|---|---|---| | `words` | `empty - could not write ([Errno 30] Read-only file system) - nothing to carry forward yet` | `empty - could not write ([Errno 28] No space left on device) - …` | unchanged | | `yesterday()` | **`no_data`**, reason names the write failure | **`no_data`**, same | unchanged: series, 24 points, surviving a restart | | verdict | `DEFECT ABSENT` | `DEFECT ABSENT` | control still persists | ## Which acceptance route, and why **Route 1: `_remember()` refuses to retain what it could not write.** The docstring's claim stands exactly as written, because it was the right claim — the code was the half that was wrong. Weakening the doc to describe the `not ok`-only behaviour would have documented a cache that forgets without saying so as intended, and that is the one thing this module's third warning exists to refuse. ## The thing actually worth judgement: one derived predicate — yes **"Cannot persist" is now one question, asked once, about the outcome.** The reason is not tidiness: **the enumeration cannot be completed.** FCST-04's M4 enforced the rule on the symptom in front of it; FCST-05's HIGH was the same shape (an `except` clause naming the one content-failure member somebody had met instead of the `ValueError` family). This is the third instance in one module, and it is the last place in this file where the rule can be spelt twice. The design point is **where** the one question can be asked. It **cannot be answered before the attempt** — a directory that existed and was writable a moment ago is not a write that completed, and any pre-check (`os.access`, a probe write) would be a *third* symptom test wearing a predicate's name, plus a value that can go stale between 08:00 and 14:00. So it is answered **after**, once, by the only thing holding the evidence: ``` _save() -> bool # True only if the file now holds what memory holds ``` No durable volume returns `False` without touching the disk; an `OSError` from the write returns `False` after recording why. `_remember()` reads that one answer and keeps only what it says was kept. **The `not self.ok` test in `_remember()` is gone** — not moved, gone — and a check asserts its absence from the source. A fourth cause needs no fourth branch, because none of them is asked about. **What is undone is the MERGE, not the store.** The state restored is the last one that *is on disk*, so a site whose volume fills at 14:00 goes on answering the yesterday it wrote at 13:30 — that record survives a restart, so answering it is not a promise the store cannot keep. It simply stops adding to it. Clearing outright would throw away a good yesterday in order to punish a failed write, on exactly the morning the operator is already dealing with a full disk. That gives the file the invariant its rule was always describing, stated once in the class docstring: **`self._days` never holds a point the file does not.** (Not "mirrors the file" — `_usable_days()` drops records the file still holds; the only direction that can *invent a yesterday* is the one named.) ⚠️ **The snapshot is `copy.deepcopy`, not `dict(self._days)`.** A one-level copy shares the day *records*, and the merge rebinds `points` on the ones it touches. Measured with a shallow copy: a store holding **12** persisted points answered **24** after a failed merge. ⚠️ **A failed write is still not fatal**, per the ticket's constraint and §6: the store degrades, it does not raise. `resolve_tariff()` still returns a `provider` plan with no §11.4 degradation on a store that cannot write a byte, and that is checked. ## The mutation table Purged tree before every run (TEST-15), `PYTHONDONTWRITEBYTECODE=1`, mutated line greped off disk before each run. Both suites that touch this module were run each time. | | mutation | Δ bytes | `test_price_history.py` | `test_static_tariff.py` | |---|---|---|---|---| | **M1** | `self._days = before` → `self._days = dict(self._days)` (the merge is retained — the shipped behaviour) | +10 B | **9 red**, incl. `A STORE THAT COULD NOT WRITE RECORDS NOTHING…` and FCST-04's own `a store with no durable volume records NOTHING…` | green | | **M2** | `_save()`'s `except OSError` returns `True` | −1 B | **9 red**, incl. `` `_save()` answers for BOTH symptoms… `` | green | | **M3** | `_save()` returns `True` on `not self.ok` | −1 B | **3 red**, incl. FCST-04's M4 check | green | | **M4** | `self._days = before` → `self._days = {}` (wipe instead of undo) | −4 B | **3 red**, incl. `…keeps the last state that IS ON DISK, not an empty one` | green | | **M5** | `copy.deepcopy(self._days)` → `dict(self._days)` | −9 B | **3 red** — `24 points held`, not 12 | green | | **M6** | `if not self.ok: return` put back into `_remember()` | +43 B | **1 red**: `…_remember() no longer tests a symptom of its own` | green | **No survivors.** M6 exists to prove the one source-shape check in the section is not decoration — behaviour is unchanged under it, and it is red on exactly that check. ## What M4 exposed about the instrument, recorded in MUTATION.md M4 first printed **1 FAIL**. The true figure is **3**. An f-string operand in the next check read `len(read.points)`, and `.points` on a `no_data` **raises `Absence` by design**, so the harness died mid-section with the remaining checks ungraded. That is `why()`'s and `derived()`'s existing rule — *"it answered nothing" and "it broke on the way to answering" must not grade the same* — arriving in what a check **prints** rather than in what it **asserts**, which is the half that looks like formatting. `npts()` is now beside `why()`; re-driving M4 moved 1 FAIL to 3. Had the traceback landed one check earlier, M4 would have produced **no named red at all** — a result that reads like a broken harness and gets retried rather than read. ## Also in this branch - `DOCS.md`'s *"where there is no `/data` at all the store records nothing"* is widened to the condition the code now holds to, with the "a failed write does not throw away what is already in the file" half stated explicitly. - `DECISIONS/FCST-06-cannot-persist-is-one-question.md` — the two decisions, including what was **not** done (no `can_persist()` helper, no probe write at `_open()`, not fatal) and why. - `CHANGELOG.md` under the existing `## Unreleased` convention. **No version number picked** — that is the open decision the Unreleased block names, and it depends on a fact this branch cannot observe. ## Not fixed, deliberately - **The `not ok` store still does the merge work before throwing it away.** With the single predicate, a store with no durable volume merges into `self._days`, is refused by `_save()`, and is rolled back. Adding a fast-path `not self.ok` return would restore the second symptom test this ticket removes, to save a few dozen dict operations on a host that has no `/data` — dev hosts and the checks container only, since production always takes the successful path. Judged not worth re-opening the defect's shape for. - **Nothing about `/data/state.json`, `PeakStore`, or any other writer.** This ticket is one module. **29/29 test files green** on the final tree. Nothing deployed — no `ha addons update`, no writes to `/addons/` or the inverter. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
admin added 1 commit 2026-09-10 08:44:21 +02:00
price_history.py's docstring states its rule unconditionally - "A store that
cannot persist now records nothing, and both of its answers say the same
thing" - and FCST-04 enforced it against `not self.ok` only, which is one
SYMPTOM: the durable volume is missing. `ok` is about the DIRECTORY, and a
read-only remount and a full disk both leave it True. Reproduced on 5059e28
against two real volumes, a /data bind-mounted read-only and a full tmpfs:

    last_error = 'could not write ([Errno 30] Read-only file system)'
    words      = '2026-01-14 (24 points over 24 h); 2026-01-15 (24 points
                  over 24 h) [could not write (...)]'
    yesterday()= series, 24 points
    ON DISK    = {}

`words` and `yesterday()` were describing two different stores, and the 24
points were gone on the next restart with nothing saying so - the
silently-forgetting cache that rule exists to refuse, reached by the condition
it did not name. No shipped path reaches it: /data is the Supervisor's durable
volume and this module is its only writer. The defect is that a guarantee the
module states about itself did not hold in one condition.

Route taken: `_remember()` refuses to retain what it could not write, and the
docstring's claim stands as written. It is now ONE derived predicate rather
than two places testing two symptoms. "Can this store persist?" cannot be
answered BEFORE the attempt - a directory that existed and was writable a
moment ago is not a write that completed - so it is answered after it, once,
by the only thing holding the evidence: `_save()` returns whether the file now
holds what memory holds. No volume is one False, an OSError is another.
`_remember()` keeps only what that one answer says was kept, and the
`not self.ok` test in `_remember()` is GONE rather than joined by a second one.
FCST-04's M4 and FCST-05's HIGH were the same shape, so a third cause now needs
no third branch.

What is undone is the MERGE, not the store. The state restored is the last one
that IS on disk, so a site whose volume fills at 14:00 goes on answering the
yesterday it wrote at 13:30 - that record survives a restart, so answering it
is not a promise the store cannot keep. It stops adding to it. The snapshot is
copy.deepcopy: a one-level copy shares the day record the merge rebinds
`points` on, and measured with one, a store holding 12 persisted points
answered 24 after a failed merge.

Not fatal, deliberately: §6 and this module's opening - a broken cache must not
be a broken controller, and an add-on that will not start because /data filled
up is a worse defect than this one. `_save()` reports and returns; the shipped
resolve_tariff() path still gives a provider plan with no §11.4 degradation.
What changed is that the failure is now acted on rather than only logged.

test_price_history.py section 5d drives it on two REAL failures with different
syscalls and different errnos - a directory where the file goes (os.replace,
EISDIR) and a volume moved out from under an open store (mkstemp, ENOENT) -
because the rule is enforced on the outcome and one cause would grade a
coincidence. DOCS.md's "where there is no /data" sentence is widened to the
condition the code now holds to.

MUTATION.md gains what M4 exposed: a mutation whose red is a TRACEBACK reports
however many checks ran before it and grades none after it. M4 first printed
1 FAIL; the true figure is 3. An f-string operand read `len(read.points)`, and
`.points` on a no_data raises `Absence` by design - why()'s rule arriving in
what a check PRINTS rather than in what it asserts. `npts()` is now beside
why(), and a red count is a lower bound until the run is seen to reach the end
of its section.

6 mutations, purged tree, every one RED on named checks, no survivors:
M1 self._days = before -> dict(self._days)          +10 B  9 red
M2 _save() OSError handler returns True             -1 B   9 red
M3 _save() returns True on `not self.ok`            -1 B   3 red
M4 self._days = before -> self._days = {}           -4 B   3 red
M5 copy.deepcopy -> dict                            -9 B   3 red
M6 `if not self.ok: return` back in _remember()     +43 B  1 red

All 29 test files green. Nothing deployed; no option added; config.yaml and
app/ behaviour on a healthy /data untouched. No version number picked - that
is the open decision the Unreleased block names.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
admin merged commit 958f5cd218 into release/1.0 2026-09-10 08:52:10 +02:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/goodwe-addon#75