goodwe-addon/goodwe_controller/app/price_history.py

"""FCST-04 - the day the carry-forward carries, kept somewhere a restart cannot reach.

WHAT WAS MISSING, AND WHY IT WAS INVISIBLE. FCST-03 shipped
`price_forecast.fill_from_yesterday()` complete and checked, and shipped the
consumers that were written to expect it - `TariffPlan.firm_points()`,
`cheap_now()`'s refusal of a provisional hour, and `arbitrage()`'s refusal 3,
which drops provisional hours from the trade set instead of refusing the whole
series. Every one of those was live. The FILL ITSELF HAD NO CALLER: nothing in
`app/` referenced `fill_from_yesterday`, so no series ever contained a carried
point and every one of those consumers was exercising its all-firm branch.

The defect that produced is not an error anywhere. `load_prices()` returns
whatever the feed published and nothing widens it to the requested horizon, so
on a dynamic site between midnight and the 14-16 h CET day-ahead publication
the series covers the rest of TODAY and nothing more - typically 10-24 of the
48 hours `price_horizon_h` asked for. `TariffPlan.arbitrage()` then computes
`max - min` over that truncated window and reports a spread. The number is
real, the window is half the one the operator configured, and NOTHING SAYS SO:
tomorrow evening's peak - the hour a morning charge would be sold into - is
simply not in the series, so the spread it would have created cannot clear a
threshold it was never compared against. A missing hour is not a wrong price;
it is an hour the optimiser never learns exists.

So this module is the half FCST-03 left out, and it is deliberately the SMALL
half: the fill's arithmetic, its DST rule and its provenance audit all stay in
`price_forecast.py`, untouched. What lives here is the one thing that
arithmetic needs and could not have: A YESTERDAY THAT SURVIVES A RESTART.


WHY PERSISTENCE, AND NOT A VARIABLE ON THE REFRESH TASK
======================================================
An in-memory "the last full day I saw" costs nothing and is wrong exactly when
it matters. The window the fill exists for is midnight -> publication; an
add-on restarted at 08:00 - a config change, an HA update, a power cut - would
come up with no yesterday at all and spend that whole window on the truncated
series, which is the pre-FCST-03 behaviour arriving silently on the mornings
after an operator touched something. The fill's own value is a function of
surviving a restart, so the day it carries has to.

⚠️ AND NOT IN `/data/state.json`. That file holds the maintenance history and
SAFETY-02's sign latch, and `Store.set()` rewrites the whole of it on every
write. The tariff refresh runs every `TARIFF_REFRESH_S` (1800 s), so folding a
price cache into it would put 48 extra whole-file rewrites a day through the
file whose corruption reads as "no maintenance ever ran" and "never tripped".
The cache is worth having; it is not worth putting anywhere near that. Separate
file, separate blast radius.

⚠️ AND THE DIRECTORY IS NOT CREATED, which is `tariff.PeakStore`'s rule and its
reason applies unchanged: `/data` is mounted by the Supervisor and always
exists on a real install, so `makedirs` here could only ever succeed where the
durable volume is MISSING - manufacturing a cache on the container's ephemeral
layer that writes fine, reads fine, and is thrown away on the next restart.
That is a yesterday that silently forgets, which is worse than one that says it
cannot answer.

⚠️ AND THAT RULE IS CARRIED OUT IN `_remember()`, NOT ONLY IN `_save()`. FCST-04
first shipped it half-done: `_save()` short-circuited on `not self.ok` while
`_remember()` filled `self._days` anyway, so a store with no durable volume
answered `words` = "unavailable" and `yesterday()` = A SERIES - an in-memory
yesterday that works all day and is gone on the next restart, which is exactly
the silently-forgetting cache this rule exists to refuse. A store that cannot
persist now records nothing, and both of its answers say the same thing.

⚠️ AND "CANNOT PERSIST" IS ONE QUESTION, ASKED ONCE, ABOUT THE OUTCOME -
FCST-06. The paragraph above enforced that rule against `not self.ok`, which is
one SYMPTOM of not persisting: the durable volume is missing. There is a second,
and FCST-06 measured it. With `ok=True` and the directory present, `_save()` can
still fail - a read-only remount (`[Errno 30] Read-only file system`), a full
volume (`[Errno 28] No space left on device`) - and it logs, records
`last_error` and returns. `_remember()` had already populated `self._days`, so
`words` read `'2026-01-14 (24 points over 24 h); 2026-01-15 (24 points over
24 h) [could not write (...)]'` while the file on disk held `{}`, and
`yesterday()` answered 24 REAL POINTS for the rest of the process's life. Gone
on the next restart, and nothing said it would be - the same silently-forgetting
cache, reached by the condition the rule did not name.

⚠️ SO IT IS NOT TWO RULES, AND IT IS DELIBERATELY NOT TWO TESTS. "Can this
store persist?" CANNOT BE ANSWERED BEFORE THE ATTEMPT - a directory that existed
and was writable a moment ago is not a write that completed - so it is answered
AFTER it, once, by the only thing holding the evidence: `_save()` returns
whether the file now holds what memory holds. A missing volume is one of its
False cases and an `OSError` from the write is another, and `_remember()` KEEPS
ONLY WHAT THAT ONE ANSWER SAYS WAS KEPT. There is no `not self.ok` test in
`_remember()` any more, because there is nothing left for a second test to add -
and a third symptom nobody has thought of is how this defect arrived both times.

⚠️ AND WHAT IS KEPT IS THE LAST STATE THAT IS ON DISK, NOT AN EMPTY ONE. The
MERGE is undone, not the store. A site whose volume fills at 14:00 goes on
answering the yesterday it wrote at 13:30 - that record survives a restart, so
answering it is not a promise the store cannot keep. What it stops doing is
adding to it. The invariant the rule was always describing, stated once:
`self._days` NEVER HOLDS A POINT THE FILE DOES NOT. (Not "mirrors the file" -
`_usable_days()` drops records the file still holds. The only direction that can
invent a yesterday is the one named.)


THE FOUR RULES THIS FILE IS SHAPED BY
=====================================

1. IT RECORDS WHAT THE PROVIDER PUBLISHED, NEVER WHAT THE FILL PRODUCED.
   Feeding a filled series back in is the defect that makes the whole mechanism
   rot: day 2 carries day 1's guesses, day 3 carries day 2's carry of them, and
   after a week of a dead feed the "yesterday" being carried forward is a
   fossil of the last real publication with nothing anywhere reporting its age.
   Every point still carries `provisional=True`, so the flag never lies - what
   decays is the NUMBER, silently, behind a flag that is telling the truth.

   ⚠️ NO VALUE-LEVEL CHECK CAN CATCH THAT, and saying so is more useful than a
   guarantee that would not hold. A carried point and a published-provisional
   point (a real-time product's current interval, §9.2) are the same three
   fields with
   the same flag; the difference is provenance, and provenance is not in the
   value. So the mechanism is ORDERING, not validation: `carry()` is the only
   public entry point, it remembers the argument and returns the fill, and the
   filled series exists only as a return value that `carry()` itself never
   sees again. `_remember()` is private, is called from exactly one place, and
   `test_price_history.py` PARSES every shipped module under `app/`, `tools/`,
   `scenarios/` and `estop/` and fails if any of them uses the NAME - an
   attribute, a bare name, a def, or the string constant a `getattr` needs. By
   name rather than by substring, so a comment about the rule does not fail it
   and a call from `tools/` cannot pass unseen.

   ⚠️ AND THE ORDER IS GRADED BY DRIVING IT, NOT BY READING THE SOURCE. The
   first version of that check compared positions in `ast.walk`, which is
   breadth-first: `self._remember(...)` is a statement and `self.yesterday(...)`
   is an ARGUMENT of the fill, one level deeper, so `_remember` came out first
   whichever statement was written first - shipped and reversed scored the
   same. What grades it now is the behaviour: a feed carrying yesterday
   alongside today, into an EMPTY store, must fill the whole horizon, and
   reversed it fills nothing.

2. "YESTERDAY" MEANS YESTERDAY, NOT "THE NEWEST DAY I HAVE".
   `newest()` is the natural accessor and it is the one that decays: an add-on
   down from Friday to Monday would answer Friday's prices to a fill that flags
   them as one day old. The lookup is BY DATE - the Brussels day before the
   Brussels day of the instant asked about - and a record for any other date is
   NO_RECORD with a reason naming both dates. A three-day-old profile is not a
   worse yesterday; it is not a yesterday.

3. A STORE THAT CANNOT ANSWER SAYS SO, IN THE TYPE.
   `yesterday()` returns a `PriceRead`: a `series` when a full record exists,
   `no_data(reason)` for every other outcome - no file, no record for that
   date, an unreadable file, a record that parses to nothing. There is no
   empty-series answer and no `.get(..., [])`-shaped accessor, for
   `price.py`'s reason: an empty series is how "the cache is broken" gets to
   wear the shape of "yesterday was free". `fill_from_yesterday()` takes a
   `PriceRead` and handles `no_data` on its own, so the degraded case needs no
   branch at the call site - it fills nothing and the horizon stays short,
   which is exactly the behaviour that shipped before this ticket.

4. HOURS ACCUMULATE WITHIN A DAY; THEY DO NOT REPLACE IT.
   A feed that serves only the REMAINING hours of the day (a legitimate shape -
   nothing in the plugin contract promises a whole day) would, under
   replace-the-record, overwrite a complete 24-hour profile with the two points
   left at 22:00 and hand the next morning a yesterday with two hours in it. So
   a record is MERGED by instant: a later read updates the hours it carries and
   leaves the rest, and a revised price for an hour already held wins because
   it is the newer statement about the same hour.

⚠️ KEYED BY INSTANT, NOT BY LOCAL HOUR, AND THAT IS THE DST DECISION. Storing
`hour -> price` would collapse the two 02:00s of a 25-hour October day into
one, inventing a second and different collapsing rule beside the one
`fill_from_yesterday()` already documents ("the first of its two 02:00 points
is the one used, deterministically"). Storing the instants keeps both, in
order, and lets the fill's own rule be the only one in the system.

WHAT THIS DOES NOT DO, ON PURPOSE
=================================
It does not extend the fill to a feed that published NOTHING. `resolve_tariff()`
applies `carry()` only on a `series` read; a `no_data` read still falls to
`_degrade()` and §11.4's row - "No price data -> fall back to the static
profile, mark the plan provisional, no arbitrage" - which is the fallback the
operator configured `static_profile` FOR. Preferring yesterday's feed over it
would change a documented degradation without being asked, and it would buy
nothing measurable: a fully-carried series is 100 % provisional, so
`arbitrage()` finds fewer than two firm hours and refuses, and `cheap_now()` is
False, exactly as on the §11.4 fallback. The two differ only in which numbers
the status page shows. `fill_from_yesterday()` handles a `no_data` read anyway,
and that branch is therefore not reached from production - it stays checked,
and turning it on is a product decision, not a wiring one. See
DECISIONS/FCST-04-the-day-that-is-carried.md.
"""

import copy
import json
import logging
import os
import tempfile
from datetime import date, datetime, timedelta, timezone

from . import messages
from .price import PricePoint, PriceProduct, PriceRead, brussels
from .price_forecast import fill_from_yesterday

_LOG = logging.getLogger("goodwe.price.history")

# How many Brussels days of published prices the SORT-ORDER cap keeps. The fill
# needs exactly one - yesterday - and rule 2 refuses anything older, so this is
# not a horizon: it is the number of days a record may sit unused before
# pruning removes it, and it is small because a bigger cache would only hold
# days no lookup can ask for. Three covers today, yesterday, and the day a
# fetch spanning midnight writes ahead of itself.
#
# ⚠️ IT IS A CAP, NOT THE RETENTION RULE, AND THREE HAS NO SLACK. The steady
# state is already {yesterday, today, tomorrow}; a feed that publishes to D+2,
# or ONE stray point in the wrong year - which sorts LAST and is therefore
# immortal under a newest-N cap - pushes yesterday out of a three-slot store
# and keeps it out, permanently and silently. So `_prune()` PINS the two days a
# lookup can actually ask for on top of the cap; see its docstring. The store
# is then bounded at KEEP_DAYS + 2 records rather than at KEEP_DAYS.
KEEP_DAYS = 3

SCHEMA = 1


def _bdate(instant: datetime) -> str:
    """The Brussels calendar date of an instant, as `YYYY-MM-DD`.

    ⚠️ BRUSSELS, NOT UTC. 23:30 UTC in July is already tomorrow in Belgium, and
    a cache keyed on the UTC date would file the last 90-120 minutes of every
    day under the wrong one - so a lookup on the following morning would find a
    "yesterday" missing its own evening hours, which are the expensive ones.
    """
    return brussels(instant).strftime("%Y-%m-%d")


def _yesterday_of(instant: datetime) -> str:
    """The Brussels date of the day BEFORE `instant`'s Brussels day.

    ⚠️ THE ARITHMETIC IS ON THE LOCAL DATE, NOT ON THE INSTANT, AND THAT IS A
    DST DEFECT THIS FILE SHIPPED. `brussels()` renders at the FIXED offset in
    force at that instant, so `brussels(stamp) - timedelta(days=1)` subtracts 24
    ABSOLUTE hours from a wall clock - and the Brussels day before a
    spring-forward is 23 hours long. Measured at 2026-03-30T00:30+02:00, the
    first hour of the morning after the March transition: it returned
    2026-03-28, so the fill carried a TWO-DAY-OLD profile flagged as yesterday's
    (or, with only the last three days held, answered `no_data` naming a date
    that is not yesterday).

    A calendar day minus one day is `date` arithmetic, and a `date` has no
    offset to lose. The 25-hour October day is right for the same reason.
    """
    return (brussels(instant).date() - timedelta(days=1)).strftime("%Y-%m-%d")


def _point_of(row):
    """One stored row -> a `PricePoint`, or `None` if the row is not one.

    THE ONE VALIDATOR, AND THE ASYMMETRY IT CLOSES (FCST-05's MEDIUM). This
    file used to hold TWO standards for what a stored point is. The READ path
    (`_parse_points()`) was properly defensive - every field checked, a
    malformed row dropped and reported. The WRITE path was
    `{q["start"]: q for q in raw if isinstance(q, dict) and "start" in q}`
    followed by `sorted(..., key=lambda q: q["start"])`, which trusts
    `q["start"]` to be a hashable, sortable string it never checked. Measured,
    with the malformed row under the day the feed is currently writing to:

        start is an int   -> TypeError: '<' not supported between instances
                             of 'str' and 'int'
        start is a dict   -> TypeError: cannot use 'dict' as a dict key
        start is null     -> TypeError: '<' not supported between 'str' and
                             'NoneType'
        start is a list   -> TypeError: cannot use 'list' as a dict key

    ⚠️ AND THE BLAST RADIUS IS PERMANENT, WHICH IS WHY IT IS NOT MERELY UNTIDY.
    `_remember()` is reached through `carry()` -> `resolve_tariff()` ->
    `run_tariff_refresh`, which catches `Exception` - so the add-on stays UP
    and EVERY refresh raises forever. The plan retires after
    `TARIFF_MAX_AGE_S`, and arbitrage and the cheap window are off from then
    on. There is no self-heal, because `_remember()` raises before it reaches
    `_save()`. `_usable_days()` only required `rec["points"]` to be a LIST; the
    elements were unchecked on the write path, and the existing fixture put its
    malformed row under a day the feed was NOT writing to, so `_remember()`
    never touched it.

    So both paths call this, and neither of them has a standard of its own.
    Returning `None` rather than raising is the contract: a caller decides
    whether a bad row is dropped (both of them do) and how it is reported.
    """
    if not isinstance(row, dict):
        return None
    try:
        start = datetime.fromisoformat(row["start"])
        value = float(row["eur_per_kwh"])
        flag = row["provisional"]
    except (KeyError, TypeError, ValueError):
        return None
    if start.tzinfo is None or start.utcoffset() is None \
            or not isinstance(flag, bool):
        return None
    return PricePoint(start=start, eur_per_kwh=value, provisional=flag)


def _held(rec) -> tuple:
    """`(points, whole hours they cover)` for a day record. `(0, 0)` otherwise.

    ⚠️ POINTS AND HOURS ARE TWO NUMBERS, AND FCST-04's STATUS ROW PRINTED THE
    FIRST UNDER THE SECOND'S NAME. The row was `f"{d} ({_n_points(rec)} h)"`,
    which counts POINTS: on the quarter-hourly feed this add-on already
    supports it read `'2026-01-13 (96 h)'` - 96 "h" for a 24-hour day. That is
    the points/hours conflation H1 was raised for, on the row H1's own ticket
    shipped. Both numbers are reported now, so an operator can see a day that
    is 24 points over 24 h and a day that is 96 points over 24 h and tell them
    apart, and neither number is wearing the other's unit.

    ⚠️ TOLERANT, AND IT IS THE SECOND LINE OF DEFENCE, NOT THE FIRST. `_load()`
    drops a malformed day record and says so, so this should never see one. It
    is written this way because `words` is read from `Controller.__init__` and
    from every status render: a status LINE that raises takes the add-on down,
    and "the cache is broken" must never be able to mean "the controller does
    not start". That is exactly what FCST-04 shipped - `{"days": {"2026-01-14":
    "oops"}}` gave `ok=True`, `last_error=None`, and an `AttributeError` out of
    `words`, `yesterday()` and `resolve_tariff()` alike, with no self-heal
    because `_remember()` raised before it could rewrite the file.

    ⚠️ AND IT COUNTS ONLY ROWS THAT ARE POINTS. A row `_point_of()` refuses is
    not held prices, and a row count that included it would report a store as
    healthier than it is - which is the second half of FCST-05's MEDIUM: the
    status row said `'2026-01-15 (1 h)'` about a record whose single row was
    killing every tariff refresh. What is dropped is named in `last_error`, and
    `words` prints that beside the counts.
    """
    if not isinstance(rec, dict):
        return (0, 0)
    raw = rec.get("points")
    if not isinstance(raw, (list, tuple)):
        return (0, 0)
    n, hours = 0, set()
    for row in raw:
        pt = _point_of(row)
        if pt is None:
            continue
        n += 1
        hours.add(pt.start.astimezone(timezone.utc)
                  .replace(minute=0, second=0, microsecond=0))
    return (n, len(hours))


def _row_of(day: str, points: int, hours: int) -> str:
    """One day's cell of the status row: what is held, in BOTH of its units.

    ⚠️ THE UNIT IS SPELT OUT ON EACH NUMBER. FCST-04's cell was
    `f"{day} ({points} h)"` - the point count wearing the hour's name, which on
    a quarter-hourly feed reads `'2026-01-13 (96 h)'` for a 24-hour day. What
    an operator needs from this row is whether tomorrow morning's fill has a
    whole day to carry, and that is the HOUR figure; the point count says how
    finely it is priced. Neither answers for the other.
    """
    # UX-10: the day as the page writes dates ("26 Sep"), not ISO.
    return (f"{points} price{'' if points == 1 else 's'} over {hours} "
            f"hour{'' if hours == 1 else 's'} for {day_words(day)}")


def day_words(day) -> str:
    """`'2026-09-26'` -> `'26 Sep'`, as the page writes a date. A key that is
    not a date is shown as it is, because this feeds a status row."""
    try:
        d = date.fromisoformat(str(day))
    except ValueError:
        return str(day)
    return f"{d.day} {d:%b}"


def _bad_rows(rec) -> int:
    """How many rows of a day record are not points. Total; never raises."""
    if not isinstance(rec, dict):
        return 0
    raw = rec.get("points")
    if not isinstance(raw, (list, tuple)):
        return 0
    return sum(1 for row in raw if _point_of(row) is None)


class PriceHistory:
    """The published price series of the last few Brussels days, in /data.

    ⚠️ `ok` IS ABOUT THE FILE, NOT ABOUT THE DATA. A healthy store with no
    record for yesterday is `ok` and answers `no_data` - a normal state on a
    site's first day, and a different thing from a store that could not be
    opened. Both are visible: `ok`/`last_error` describe the file, and
    `yesterday()`'s reason describes the answer.

    ⚠️ AND `ok` IS NOT "CAN PERSIST" (FCST-06). It says the directory was
    there when the store opened, which a full volume and a read-only remount
    both leave true. The one thing that answers "can this store persist" is
    `_save()`'s return value, and `self._days` NEVER HOLDS A POINT THE FILE DOES
    NOT, because `_remember()` undoes a merge that answer refused. That is what
    makes `words` and `yesterday()` describe the same store in EVERY degraded
    state rather than only in the ones somebody enumerated.
    """

    def __init__(self, path: str = "/data/price_history.json"):
        self.path = path
        self.ok = False
        self.last_error: str | None = None
        self._days: dict = {}
        self._open()

    # -- the file ------------------------------------------------------------
    def _open(self) -> None:
        parent = os.path.dirname(self.path)
        # See the module note: NOT created. A cache on the ephemeral layer is
        # one that forgets without saying so.
        if parent and not os.path.isdir(parent):
            self.last_error = f"no durable volume at {parent}"
            # INFO, not ERROR: on a dev host or in a test container there is no
            # /data and nothing is wrong.
            _LOG.info("No price history is kept (%s), so missing hours are "
                      "not filled from yesterday's prices.", self.last_error)
            return
        self.ok = True
        self._load()

    def _load(self) -> None:
        try:
            with open(self.path, "r", encoding="utf-8") as fh:
                blob = json.load(fh)
        except FileNotFoundError:
            _LOG.info("no price history yet at %s - the first refresh writes "
                      "one, and the fill has nothing to carry until the day "
                      "after that", self.path)
            return
        except (ValueError, OSError) as err:
            # A corrupt cache must not stop the controller, and must not be
            # silently re-created either: the record is discarded, the reason
            # is kept, and `yesterday()` reports it rather than answering
            # "no record", which would read as a normal first day.
            #
            # ⚠️ `ValueError`, NOT `json.JSONDecodeError`, AND THAT IS
            # FCST-05's HIGH. Reading this file has two failure families: the
            # BYTES (`OSError`) and the CONTENT - and the content family has at
            # least two members, because `open(..., encoding="utf-8")` decodes
            # before `json` ever sees a character. `UnicodeDecodeError` is
            # `ValueError -> UnicodeError -> UnicodeDecodeError`; it is NOT a
            # `json.JSONDecodeError` and it is NOT an `OSError`
            # (`issubclass(UnicodeDecodeError, OSError)` is False). So ONE
            # non-UTF-8 byte in /data/price_history.json escaped `_load()` ->
            # `_open()` -> `PriceHistory.__init__`, which is unguarded at
            # main.py's `Controller.__init__`, AND THE ADD-ON DID NOT START.
            # Measured: `UnicodeDecodeError: 'utf-8' codec can't decode byte
            # 0xff in position 39`. Valid-UTF-8-invalid-JSON, a UTF-8 BOM and a
            # non-dict day record all degraded correctly - it was the one byte
            # class left uncovered, which is the same class as the defect the
            # `_usable_days()` rework was raised for.
            #
            # The tuple names the FAMILY rather than the two members known
            # today, because "the file's content is not readable as JSON" is
            # exactly what this handler says, and a third `ValueError` subclass
            # nobody has thought of is how this defect arrived the first time.
            self.last_error = f"unreadable ({err})"
            _LOG.warning("Price history at %s is %s, so it starts empty. "
                         "Missing hours cannot be filled in until a full day "
                         "is saved.",
                         self.path, self.last_error)
            return
        found = blob.get("version") if isinstance(blob, dict) else None
        # ⚠️ THE TYPE IS PART OF THE VERSION. `1.0 == 1` in Python, so a float
        # written by some other tool would pass a bare `!=` and be read as
        # schema 1 - a version check that cannot tell 1 from 1.0 is one that
        # would not notice a writer this build has never met.
        if not isinstance(blob, dict) or isinstance(found, bool) \
                or not isinstance(found, int) or found != SCHEMA:
            self.last_error = f"version {found!r}, not {SCHEMA}"
            _LOG.warning("Price history at %s has %s, so it is not used.",
                         self.path, self.last_error)
            return
        days = blob.get("days")
        if not isinstance(days, dict):
            self.last_error = "no days in it"
            _LOG.warning("Price history at %s has %s, so it is not used.",
                         self.path, self.last_error)
            return
        self._days = self._usable_days(days)

    def _usable_days(self, days: dict) -> dict:
        """The day records that are shaped like day records. The rest are dropped.

        ⚠️ THE ASYMMETRY THAT PRODUCED THE DEFECT: `_parse_points()` below is
        carefully defensive about every field of a POINT, and `_load()` was not
        defensive about the record holding them at all. `{"version": 1, "days":
        {"2026-01-14": "oops"}}` therefore loaded as `ok=True`,
        `last_error=None` - and then `rec.get(...)` raised `AttributeError` out
        of `words`, `yesterday()` and `resolve_tariff()`. `PriceHistory(...)`
        and `.words` are both unguarded in `Controller.__init__`, so THE ADD-ON
        DID NOT START; and it never self-healed, because `_remember()` raised on
        the same record before it could reach `_save()`.

        A broken cache is a short horizon. It is not a controller that will not
        regulate a battery, and §6's rule - absent machinery must not disable
        the loop - does not stop applying because the machinery is a JSON file.

        Dropped, not repaired: an unusable record has nothing in it to keep, and
        the next `_remember()` writes a fresh one under that date.

        ⚠️ AND THE ROWS INSIDE A KEPT RECORD ARE COUNTED, NOT ONLY THE RECORD
        (FCST-05). This checked that `rec["points"]` is a LIST and stopped
        there, so a record holding one row with a `start` of `7` loaded as
        healthy: `last_error=None`, and `words` said `'2026-01-15 (1 h)'` about
        a store whose next `_remember()` would raise on that row and go on
        raising forever. The rows are NOT dropped here - `_parse_points()` is
        the read path's one filter and stays so - but they are COUNTED, so the
        store stops reporting health it does not have from the moment it is
        opened. `_remember()` drops them on the next refresh and `_save()`
        rewrites the file, after which a restart clears the note.
        """
        kept, dropped, malformed = {}, [], {}
        for key, rec in days.items():
            if not isinstance(key, str) or not isinstance(rec, dict) \
                    or not isinstance(rec.get("points"), list):
                dropped.append(str(key))
                continue
            bad = _bad_rows(rec)
            if bad:
                malformed[key] = bad
            kept[key] = rec
        notes = []
        if dropped:
            notes.append(f"{messages.count(len(dropped), 'unusable day')} "
                         f"dropped, " + ", ".join(day_words(d)
                                                  for d in sorted(dropped)))
        if malformed:
            notes.append(
                f"{messages.count(sum(malformed.values()), 'bad price')} in "
                + ", ".join(f"{day_words(d)} ({n})"
                            for d, n in sorted(malformed.items())))
        if notes:
            # Recorded, not silent: `yesterday()` and `words` both report
            # `last_error`, and a cache that quietly discards half a file reads
            # to an operator exactly like a normal first day.
            self.last_error = " and ".join(notes)
            _LOG.warning("Price history at %s: %s. The add-on starts anyway, "
                         "and fills in only what the rest holds.",
                         self.path, self.last_error)
        return kept

    def _save(self) -> bool:
        """Write `self._days` out. True only if the file now holds it.

        ⚠️ THE RETURN VALUE IS THE "CAN THIS STORE PERSIST?" PREDICATE, AND
        IT IS THE ONLY ONE (FCST-06). Both ways of not persisting end here and
        are the same answer: no durable volume returns False without touching
        the disk, an `OSError` out of the write returns False after recording
        why. `_remember()` acts on that one answer, so the second place that
        used to test the first symptom on its own - and missed the second - has
        nothing left to test. See the module docstring's fifth warning.

        ⚠️ AND IT REPORTS, IT DOES NOT RAISE. §6 and this module's opening: a
        broken cache must not be a broken controller. An add-on that will not
        start because /data filled up is a worse defect than the one FCST-06
        fixed, so a failed write is logged, kept in `last_error` - where `words`
        and `yesterday()` both read it - and RETURNED, never thrown.

        ⚠️ THE TEMP FILE IS REMOVED ON EVERY FAILING PATH. A write that dies
        between `mkstemp` and `os.replace` used to leave its temp behind, and
        the refresh task runs 48 times a day: a persistent write failure - a
        full volume being the likeliest cause - leaked 48 inodes a day onto the
        volume that was already the problem.
        """
        if not self.ok:
            return False
        tmp = None
        try:
            fd, tmp = tempfile.mkstemp(dir=os.path.dirname(self.path))
            with os.fdopen(fd, "w", encoding="utf-8") as fh:
                json.dump({"version": SCHEMA, "days": self._days}, fh, indent=1)
            # Atomic: a power cut mid-write must not leave a truncated file
            # that parses as an empty day.
            os.replace(tmp, self.path)
            tmp = None                      # it IS the file now; do not unlink
        except OSError as err:
            self.last_error = f"could not write ({err})"
            _LOG.error("could not persist price history: %s", err)
            return False
        finally:
            if tmp is not None:
                try:
                    os.unlink(tmp)
                except OSError:             # nothing further to try, and a
                    pass                    # failed cleanup is not a fault
        return True

    # -- the one public entry point -----------------------------------------
    def carry(self, read: PriceRead, *, since: datetime, until: datetime,
              stamp: datetime, remember: bool = True) -> PriceRead:
        """Remember what this read published, then §8.3-fill the missing hours.

        THE ORDER IS DELIBERATE AND IS THE POINT OF THE METHOD. The read is
        recorded FIRST, so a feed that hands back yesterday alongside today
        refreshes yesterday's record from live data before it is consumed as
        the profile; and the filled series - the one thing that must never be
        recorded - exists only as this method's return value, which this object
        never sees again. See rule 1, including how that order is graded.

        Returns whatever `fill_from_yesterday()` returns: a series over the
        whole hours of [since, until), carrying EVERY point the provider
        published inside it - by identity, and four of them in one hour on a
        quarter-hourly feed - plus one carried point for each remaining whole
        hour FOR WHICH YESTERDAY HELD THAT WALL-CLOCK HOUR. An hour neither
        source can price is left out (§11.4 for that hour), and `no_data` comes
        back when that is every hour.
        """
        if remember:
            # PRICE-01: False only for a read that CAME FROM this store
            # (`stored()`), which is already filed; filing it again would
            # rewrite the same file for nothing, and nothing else may pass it.
            self._remember(read, stamp)
        return fill_from_yesterday(read, yesterday=self.yesterday(stamp),
                                   since=since, until=until, stamp=stamp)

    # -- the halves, private because rule 1 is an ordering ------------------
    def _remember(self, read: PriceRead, stamp: datetime) -> None:
        """File a provider read's points under their own Brussels days.

        ⚠️ PRIVATE, AND CALLED FROM EXACTLY ONE PLACE. A caller that can reach
        this can record a filled series, which is the decay in rule 1. There is
        no value-level defence against that - a carried point and a published
        provisional point are the same three fields - so the defence is that
        nothing outside this file names it, and `test_price_history.py` checks
        the tree for that rather than trusting the underscore.
        """
        points = read.match(series=lambda r: r.points, no_data=lambda _r: ())
        if not points:
            return
        product = read.match(series=lambda r: r.product.value,
                             no_data=lambda _r: None)
        provider = read.match(series=lambda r: r.provider,
                              no_data=lambda _r: None)

        # ⚠️ THE STATE THIS READ STARTED FROM, KEPT SO IT CAN BE PUT BACK
        # (FCST-06). Everything below merges into `self._days` IN PLACE, and
        # whether any of it may be KEPT is not knowable until `_save()` has
        # tried - so the merge runs, and it is undone if the file did not take
        # it. There is no `not self.ok` branch above any more: a store with no
        # durable volume is one of the answers `_save()` gives, not a separate
        # symptom with a test of its own. Module docstring, fifth warning.
        #
        # ⚠️ DEEP, NOT `dict(self._days)`. A one-level copy shares the day
        # RECORDS, and the loop below rebinds keys on the ones it touches - so
        # the "previous" state would hand back records the merge had already
        # edited, and a store with a good yesterday on disk would go on
        # answering the merge of it that never reached the file. It would also
        # be right only for as long as nobody makes this loop mutate a `points`
        # list or a row in place. A copy that is right by construction outlives
        # that, and the store holds at most KEEP_DAYS + 2 records, 48 times a
        # day.
        before = copy.deepcopy(self._days)

        touched = set()
        junked = {}
        for p in points:
            day = _bdate(p.start)
            rec = self._days.setdefault(
                day, {"product": product, "provider": provider, "points": []})
            # Rule 4: MERGE by instant. A later read updates the hours it
            # carries and leaves the rest; a revised price for an hour already
            # held wins, because it is the newer statement about that hour.
            #
            # ⚠️ EVERY HELD ROW GOES THROUGH `_point_of()` (FCST-05's MEDIUM).
            # This was `{q["start"]: q for q in raw if isinstance(q, dict) and
            # "start" in q}` sorted by `q["start"]` - a key and a sort key that
            # trusted a value read off disk to be a hashable, sortable string.
            # An `int`, `null`, a `dict` or a `list` there raised `TypeError`
            # out of `carry()` on the day the feed was writing to, and
            # `run_tariff_refresh` caught it, so the add-on stayed up while
            # EVERY refresh failed for good. A row that is not a point is now
            # dropped and reported here, exactly as the read path drops it, and
            # `_save()` below writes the record back clean - which is the
            # self-heal the raising version could never reach.
            #
            # ⚠️ AND THE MERGE KEY IS THE VALIDATED INSTANT, NOT ITS SPELLING.
            # Aware datetimes hash and compare by instant, so this is rule 4 as
            # rule 4 is written. The old string key merged by SPELLING, and the
            # old lexicographic sort put the second 02:00 of the 25-hour
            # October day (`+01:00`) BEFORE the first (`+02:00`). Only the
            # stored order was affected - `_parse_points()` re-sorts by instant
            # on the way out - but the store now holds what it says it holds.
            raw = rec.get("points")
            held = {}
            for q in (raw if isinstance(raw, list) else ()):
                kept = _point_of(q)
                if kept is None:
                    junked[day] = junked.get(day, 0) + 1
                    continue
                held[kept.start] = q
            held[p.start] = p.canonical()
            order = sorted(held, key=lambda t: t.astimezone(timezone.utc))
            rec["points"] = [held[t] for t in order]
            rec["product"], rec["provider"] = product, provider
            touched.add(day)

        if junked:
            # Said, not swallowed. `words` prints `last_error`, so a store that
            # has just discarded rows off disk stops reporting plain health -
            # and after `_save()` below the file is clean, so a restart clears
            # this. `_save()` overwrites it if the write itself fails, which is
            # the more urgent fact of the two - and on that path the drop is
            # undone with the rest of the merge, so those rows are still in the
            # file and the next write that completes drops them again.
            self.last_error = (
                f"{messages.count(sum(junked.values()), 'bad price')} dropped "
                f"from " + ", ".join(f"{day_words(d)} ({n})"
                                     for d, n in sorted(junked.items())))
            _LOG.warning("Price history at %s: %s. The file is saved without "
                         "them and the price update goes on.",
                         self.path, self.last_error)
        self._prune(stamp)
        if not self._save():
            # ⚠️ A STORE THAT CANNOT PERSIST RECORDS NOTHING - the rule, asked
            # ONCE as a question about the outcome instead of tested twice
            # against two symptoms of it. Keeping the merge here is exactly the
            # cache the module docstring refuses: `yesterday()` would answer a
            # series the file does not contain, all day, until a restart
            # dropped it without a word - and `words` and `yesterday()` would
            # be describing two different stores. The reason is already in
            # `last_error` (from `_save()`, or from `_open()` when there is no
            # volume at all), so both answers now describe what is on disk.
            #
            # It does NOT raise, and §6 is why: a broken cache must not be a
            # broken controller. `carry()` still returns the fill, the horizon
            # is as short as the feed makes it, and the add-on regulates.
            self._days = before
            return
        _LOG.debug("price history: %d point(s) filed under %s; %d day(s) held",
                   len(points), ", ".join(sorted(touched)), len(self._days))

    def _prune(self, stamp: datetime) -> None:
        """Drop records no lookup can reach. A sort-order cap, plus two pins.

        ⚠️ NOTHING IS DELETED BY ARITHMETIC AGAINST THE CLOCK. A clock that
        jumped - the failure mode WITNESS-03 spent a session on - would
        otherwise delete every record in one pass and leave the fill with
        nothing on the morning after, which is the one morning it is for.
        Deleting on a bad clock is unrecoverable; holding a few days too many
        is not, so the cap is `sorted(...)[-KEEP_DAYS:]` and stays that way.

        ⚠️ AND A CAP ALONE IS DEFEATED BY ONE POINT IN THE WRONG YEAR. Sorting
        is lexicographic on the date string, so a stray far-future record sorts
        LAST and is immortal - it consumes a slot for good, and with the steady
        state already {yesterday, today, tomorrow} filling all three, yesterday
        is what falls out. A feed that publishes to D+2 does the same thing
        without any stray at all. So the two days `yesterday()` can ACTUALLY be
        asked for are pinned on top of the cap.

        `stamp` is READ here, which it was not before: the pin is the only use
        of it, and it is additive. Pinning can only RETAIN a record, never
        delete one, so a jumped `stamp` pins two dates that are not held and
        changes nothing - the WITNESS-03 property is preserved by construction
        rather than by remembering to check it.
        """
        if len(self._days) <= KEEP_DAYS:
            return
        keep = set(sorted(self._days)[-KEEP_DAYS:])
        keep |= {_bdate(stamp), _yesterday_of(stamp)}
        for day in [d for d in self._days if d not in keep]:
            del self._days[day]

    def yesterday(self, stamp: datetime) -> PriceRead:
        """The published series of the Brussels day BEFORE `stamp`'s, or no_data.

        ⚠️ THE DATE IS COMPUTED, NOT LOOKED UP. See rule 2: `newest()` would
        answer Friday's prices to a Monday fill, and the fill would flag them as
        one day old because that is the only thing its flag can say.
        """
        if stamp.tzinfo is None or stamp.utcoffset() is None:
            raise ValueError(
                f"yesterday({stamp!r}): a naive datetime is not an instant, "
                f"and 'which Brussels day was yesterday' is exactly the "
                f"question a wall-clock reading cannot answer.")
        want = _yesterday_of(stamp)
        if self.last_error and not self._days:
            return PriceRead.no_data(
                f"the price history at {self.path} is {self.last_error}, so "
                f"there are no prices from yesterday to fill in for "
                f"{day_words(want)}. The plan covers only the hours the feed "
                f"published")
        rec = self._days.get(want)
        if rec is None:
            have = (", ".join(day_words(d) for d in sorted(self._days))
                    or "no days at all")
            return PriceRead.no_data(
                f"no published prices are kept for {day_words(want)}, only for "
                f"{have}, so there are no prices from yesterday to fill in. A "
                f"record for another day is not a yesterday")
        points, bad = _parse_points(rec.get("points"))
        if not points:
            why = bad or "it is empty"
            return PriceRead.no_data(
                f"the {day_words(want)} record holds no usable price, {why}, "
                f"and an empty series is not a free day")
        try:
            product = PriceProduct(rec.get("product"))
        except ValueError:
            return PriceRead.no_data(
                f"the {want} record names product {rec.get('product')!r}, "
                f"which is not a PriceProduct. The optimiser switches on that "
                f"enum, so a record that cannot name one cannot be a series")
        if bad:
            _LOG.warning("Price history: %s dropped from the %s record, and "
                         "%s kept.", bad, day_words(want),
                         messages.count(len(points), "price"))
        return PriceRead.series(points, product,
                                str(rec.get("provider") or "price_history"),
                                stamp)

    def stored(self, stamp: datetime) -> PriceRead:
        """PRICE-01: every price this store holds, as one series, or no_data.

        This is the LAST step of the price server's fallback (price_chain.py):
        the answer of the last fetch that worked, kept on disk, so a site can
        plan for as long as the days it holds last. It is a read of what was
        PUBLISHED and filed, in the order they happened, and it is never the
        carried fill (rule 1 holds: `carry(..., remember=False)` is how this
        read is filled, and it is not filed again).

        The prices are the all-in ones stored when they were fetched. They are
        not worked out again, so a change of markup or taxes shows from the
        next fetch that works, not in these.
        """
        if stamp.tzinfo is None or stamp.utcoffset() is None:
            raise ValueError(f"stored({stamp!r}): a naive datetime is not an "
                             f"instant")
        points = []
        for day in sorted(self._days):
            rec = self._days[day]
            got, _bad = _parse_points(rec.get("points") if isinstance(
                rec, dict) else None)
            points.extend(got)
        if not points:
            why = self.last_error or "it holds no prices"
            return PriceRead.no_data(
                f"the price history at {self.path} gives nothing: {why}")
        points.sort(key=lambda p: p.start.astimezone(timezone.utc))
        return PriceRead.series(points, PriceProduct.DAY_AHEAD,
                                "price_history", stamp)

    # -- observability -------------------------------------------------------
    @property
    def words(self) -> str:
        """One line for the status page. Says what is held, or why nothing is.

        ⚠️ THIS MUST NOT RAISE, EVER. It is read once from `Controller.__init__`
        and again on every status render, and neither call site guards it - so
        an exception here is an add-on that does not start. It went out that
        way: a day record that was a string instead of a dict raised
        `AttributeError` from the `len(...)` below. `_load()` now drops such a
        record and `_held()` is total, so this has two independent reasons not
        to fail on a broken file.

        ⚠️ AND IT COUNTED POINTS AND CALLED THEM HOURS (FCST-05's LOW). The row
        was `f"{d} ({_n_points(rec)} h)"`; on the quarter-hourly feed this
        add-on already supports that reads `'2026-01-13 (96 h)'` for a 24-hour
        day. Both numbers are printed now - see `_held()` - because they are
        two different facts and only one of them was ever being shown.
        """
        if not self.ok:
            return f"unavailable - {self.last_error}"
        if not self._days:
            why = f" - {self.last_error}" if self.last_error else ""
            return f"empty{why} - nothing to carry forward yet"
        held = ", ".join(_row_of(d, *_held(self._days[d]))
                         for d in sorted(self._days, key=str))
        # A file that PARTLY loaded says so, or the dropped days read as days
        # that were simply never written.
        return held + (f" [{self.last_error}]" if self.last_error else "")


def _parse_points(raw) -> tuple:
    """Canonical dicts -> PricePoints, and a phrase naming what was dropped.

    ⚠️ A MALFORMED POINT IS DROPPED, NOT DEFAULTED, and the drop is reported.
    `float(row.get("eur_per_kwh", 0))` is the shape of the whole family of
    defects this repo has been removing: a cache that lost a field would answer
    "that hour was free" to a fill that flags it as an ordinary guess.
    """
    if not isinstance(raw, list):
        return (), "the `points` field is not a list"
    out, dropped = [], 0
    for row in raw:
        # ⚠️ `_point_of()` IS THE ONE VALIDATOR, and this loop used to BE it.
        # The write path had its own, weaker standard; see `_point_of()` for
        # what that cost. One standard, two callers, no asymmetry to re-open.
        pt = _point_of(row)
        if pt is None:
            dropped += 1
            continue
        out.append(pt)
    out.sort(key=lambda p: p.start.astimezone(timezone.utc))
    return tuple(out), (messages.count(dropped, "bad price") if dropped else "")


class NoHistory:
    """The explicit "this caller keeps no history" - a NAME, not a `None`.

    ⚠️ IT EXISTS SO THE PARAMETER CAN BE REQUIRED. `history=None` meaning "skip
    the fill" is the shape this repo keeps removing: a caller who forgets the
    argument gets the truncated-horizon behaviour of the bug this ticket is
    about, silently, and the code reads as though the fill is happening. With
    no default there is nothing to forget, and a caller with no durable volume
    - a check, a replay harness - has to say `NoHistory()` and be visible in
    the diff.

    It carries out the ordering faithfully: it records nothing and carries
    nothing, so `carry()` returns the provider's read unchanged and the plan is
    exactly what it was before FCST-04.
    """

    ok = False
    last_error = "no history is kept by this caller"
    words = "not kept"

    def carry(self, read: PriceRead, *, since, until, stamp,
              remember: bool = True) -> PriceRead:
        return read

    def stored(self, stamp: datetime) -> PriceRead:
        return PriceRead.no_data("this caller keeps no price history, so "
                                 "there are no stored prices")

    def yesterday(self, stamp: datetime) -> PriceRead:
        return PriceRead.no_data(
            "this caller keeps no price history, so there are no prices from "
            "yesterday to fill in")