goodwe-addon/goodwe_controller/app/price_history.py

"""FCST-04 - the day the carry-forward carries, kept somewhere a restart cannot reach.

WHAT WAS MISSING, AND WHY IT WAS INVISIBLE. FCST-03 shipped
`price_forecast.fill_from_yesterday()` complete and checked, and shipped the
consumers that were written to expect it - `TariffPlan.firm_points()`,
`cheap_now()`'s refusal of a provisional hour, and `arbitrage()`'s refusal 3,
which drops provisional hours from the trade set instead of refusing the whole
series. Every one of those was live. The FILL ITSELF HAD NO CALLER: nothing in
`app/` referenced `fill_from_yesterday`, so no series ever contained a carried
point and every one of those consumers was exercising its all-firm branch.

The defect that produced is not an error anywhere. `load_prices()` returns
whatever the feed published and nothing widens it to the requested horizon, so
on a dynamic site between midnight and the 14-16 h CET day-ahead publication
the series covers the rest of TODAY and nothing more - typically 10-24 of the
48 hours `price_horizon_h` asked for. `TariffPlan.arbitrage()` then computes
`max - min` over that truncated window and reports a spread. The number is
real, the window is half the one the operator configured, and NOTHING SAYS SO:
tomorrow evening's peak - the hour a morning charge would be sold into - is
simply not in the series, so the spread it would have created cannot clear a
threshold it was never compared against. A missing hour is not a wrong price;
it is an hour the optimiser never learns exists.

So this module is the half FCST-03 left out, and it is deliberately the SMALL
half: the fill's arithmetic, its DST rule and its provenance audit all stay in
`price_forecast.py`, untouched. What lives here is the one thing that
arithmetic needs and could not have: A YESTERDAY THAT SURVIVES A RESTART.


WHY PERSISTENCE, AND NOT A VARIABLE ON THE REFRESH TASK
======================================================
An in-memory "the last full day I saw" costs nothing and is wrong exactly when
it matters. The window the fill exists for is midnight -> publication; an
add-on restarted at 08:00 - a config change, an HA update, a power cut - would
come up with no yesterday at all and spend that whole window on the truncated
series, which is the pre-FCST-03 behaviour arriving silently on the mornings
after an operator touched something. The fill's own value is a function of
surviving a restart, so the day it carries has to.

⚠️ AND NOT IN `/data/state.json`. That file holds the maintenance history and
SAFETY-02's sign latch, and `Store.set()` rewrites the whole of it on every
write. The tariff refresh runs every `TARIFF_REFRESH_S` (1800 s), so folding a
price cache into it would put 48 extra whole-file rewrites a day through the
file whose corruption reads as "no maintenance ever ran" and "never tripped".
The cache is worth having; it is not worth putting anywhere near that. Separate
file, separate blast radius.

⚠️ AND THE DIRECTORY IS NOT CREATED, which is `tariff.PeakStore`'s rule and its
reason applies unchanged: `/data` is mounted by the Supervisor and always
exists on a real install, so `makedirs` here could only ever succeed where the
durable volume is MISSING - manufacturing a cache on the container's ephemeral
layer that writes fine, reads fine, and is thrown away on the next restart.
That is a yesterday that silently forgets, which is worse than one that says it
cannot answer.


THE FOUR RULES THIS FILE IS SHAPED BY
=====================================

1. IT RECORDS WHAT THE PROVIDER PUBLISHED, NEVER WHAT THE FILL PRODUCED.
   Feeding a filled series back in is the defect that makes the whole mechanism
   rot: day 2 carries day 1's guesses, day 3 carries day 2's carry of them, and
   after a week of a dead feed the "yesterday" being carried forward is a
   fossil of the last real publication with nothing anywhere reporting its age.
   Every point still carries `provisional=True`, so the flag never lies - what
   decays is the NUMBER, silently, behind a flag that is telling the truth.

   ⚠️ NO VALUE-LEVEL CHECK CAN CATCH THAT, and saying so is more useful than a
   guarantee that would not hold. A carried point and a published-provisional
   point (`eneco_eddy`'s current quarter, §9.2) are the same three fields with
   the same flag; the difference is provenance, and provenance is not in the
   value. So the mechanism is ORDERING, not validation: `carry()` is the only
   public entry point, it remembers the argument and returns the fill, and the
   filled series exists only as a return value that `carry()` itself never
   sees again. `_remember()` is private, is called from exactly one place, and
   `test_price_history.py` walks the tree and fails if any module outside this
   file names it.

2. "YESTERDAY" MEANS YESTERDAY, NOT "THE NEWEST DAY I HAVE".
   `newest()` is the natural accessor and it is the one that decays: an add-on
   down from Friday to Monday would answer Friday's prices to a fill that flags
   them as one day old. The lookup is BY DATE - the Brussels day before the
   Brussels day of the instant asked about - and a record for any other date is
   NO_RECORD with a reason naming both dates. A three-day-old profile is not a
   worse yesterday; it is not a yesterday.

3. A STORE THAT CANNOT ANSWER SAYS SO, IN THE TYPE.
   `yesterday()` returns a `PriceRead`: a `series` when a full record exists,
   `no_data(reason)` for every other outcome - no file, no record for that
   date, an unreadable file, a record that parses to nothing. There is no
   empty-series answer and no `.get(..., [])`-shaped accessor, for
   `price.py`'s reason: an empty series is how "the cache is broken" gets to
   wear the shape of "yesterday was free". `fill_from_yesterday()` takes a
   `PriceRead` and handles `no_data` on its own, so the degraded case needs no
   branch at the call site - it fills nothing and the horizon stays short,
   which is exactly the behaviour that shipped before this ticket.

4. HOURS ACCUMULATE WITHIN A DAY; THEY DO NOT REPLACE IT.
   A feed that serves only the REMAINING hours of the day (a legitimate shape -
   nothing in the plugin contract promises a whole day) would, under
   replace-the-record, overwrite a complete 24-hour profile with the two points
   left at 22:00 and hand the next morning a yesterday with two hours in it. So
   a record is MERGED by instant: a later read updates the hours it carries and
   leaves the rest, and a revised price for an hour already held wins because
   it is the newer statement about the same hour.

⚠️ KEYED BY INSTANT, NOT BY LOCAL HOUR, AND THAT IS THE DST DECISION. Storing
`hour -> price` would collapse the two 02:00s of a 25-hour October day into
one, inventing a second and different collapsing rule beside the one
`fill_from_yesterday()` already documents ("the first of its two 02:00 points
is the one used, deterministically"). Storing the instants keeps both, in
order, and lets the fill's own rule be the only one in the system.

WHAT THIS DOES NOT DO, ON PURPOSE
=================================
It does not extend the fill to a feed that published NOTHING. `resolve_tariff()`
applies `carry()` only on a `series` read; a `no_data` read still falls to
`_degrade()` and §11.4's row - "No price data -> fall back to the static
profile, mark the plan provisional, no arbitrage" - which is the fallback the
operator configured `static_profile` FOR. Preferring yesterday's feed over it
would change a documented degradation without being asked, and it would buy
nothing measurable: a fully-carried series is 100 % provisional, so
`arbitrage()` finds fewer than two firm hours and refuses, and `cheap_now()` is
False, exactly as on the §11.4 fallback. The two differ only in which numbers
the status page shows. `fill_from_yesterday()` handles a `no_data` read anyway,
and that branch is therefore not reached from production - it stays checked,
and turning it on is a product decision, not a wiring one. See
DECISIONS/FCST-04-the-day-that-is-carried.md.
"""

import json
import logging
import os
import tempfile
from datetime import datetime, timedelta, timezone

from .price import PricePoint, PriceProduct, PriceRead, brussels
from .price_forecast import fill_from_yesterday

_LOG = logging.getLogger("goodwe.price.history")

# How many Brussels days of published prices are kept. The fill needs exactly
# one - yesterday - and rule 2 refuses anything older, so this is not a horizon:
# it is the number of days a record may sit unused before pruning removes it,
# and it is small because a bigger cache would only hold days no lookup can ask
# for. Three covers today, yesterday, and the day a fetch spanning midnight
# writes ahead of itself.
KEEP_DAYS = 3

SCHEMA = 1


def _bdate(instant: datetime) -> str:
    """The Brussels calendar date of an instant, as `YYYY-MM-DD`.

    ⚠️ BRUSSELS, NOT UTC. 23:30 UTC in July is already tomorrow in Belgium, and
    a cache keyed on the UTC date would file the last 90-120 minutes of every
    day under the wrong one - so a lookup on the following morning would find a
    "yesterday" missing its own evening hours, which are the expensive ones.
    """
    return brussels(instant).strftime("%Y-%m-%d")


class PriceHistory:
    """The published price series of the last few Brussels days, in /data.

    ⚠️ `ok` IS ABOUT THE FILE, NOT ABOUT THE DATA. A healthy store with no
    record for yesterday is `ok` and answers `no_data` - a normal state on a
    site's first day, and a different thing from a store that could not be
    opened. Both are visible: `ok`/`last_error` describe the file, and
    `yesterday()`'s reason describes the answer.
    """

    def __init__(self, path: str = "/data/price_history.json"):
        self.path = path
        self.ok = False
        self.last_error: str | None = None
        self._days: dict = {}
        self._open()

    # -- the file ------------------------------------------------------------
    def _open(self) -> None:
        parent = os.path.dirname(self.path)
        # See the module note: NOT created. A cache on the ephemeral layer is
        # one that forgets without saying so.
        if parent and not os.path.isdir(parent):
            self.last_error = f"no durable volume at {parent}"
            # INFO, not ERROR: on a dev host or in a test container there is no
            # /data and nothing is wrong.
            _LOG.info("no price history: %s - §8.3 carry-forward has nothing "
                      "to carry and the horizon stays as short as the feed",
                      self.last_error)
            return
        self.ok = True
        self._load()

    def _load(self) -> None:
        try:
            with open(self.path, "r", encoding="utf-8") as fh:
                blob = json.load(fh)
        except FileNotFoundError:
            _LOG.info("no price history yet at %s - the first refresh writes "
                      "one, and the fill has nothing to carry until the day "
                      "after that", self.path)
            return
        except (json.JSONDecodeError, OSError) as err:
            # A corrupt cache must not stop the controller, and must not be
            # silently re-created either: the record is discarded, the reason
            # is kept, and `yesterday()` reports it rather than answering
            # "no record", which would read as a normal first day.
            self.last_error = f"unreadable ({err})"
            _LOG.warning("price history at %s is %s - starting from empty. "
                         "§8.3 carry-forward has nothing to carry until the "
                         "next refresh has written a full day",
                         self.path, self.last_error)
            return
        if not isinstance(blob, dict) or blob.get("version") != SCHEMA:
            found = blob.get("version") if isinstance(blob, dict) else None
            self.last_error = f"schema {found!r}, not {SCHEMA}"
            _LOG.warning("price history at %s has %s - discarding it rather "
                         "than guessing what its points mean",
                         self.path, self.last_error)
            return
        days = blob.get("days")
        if not isinstance(days, dict):
            self.last_error = "no `days` object"
            _LOG.warning("price history at %s has %s", self.path, self.last_error)
            return
        self._days = days

    def _save(self) -> None:
        if not self.ok:
            return
        try:
            fd, tmp = tempfile.mkstemp(dir=os.path.dirname(self.path))
            with os.fdopen(fd, "w", encoding="utf-8") as fh:
                json.dump({"version": SCHEMA, "days": self._days}, fh, indent=1)
            # Atomic: a power cut mid-write must not leave a truncated file
            # that parses as an empty day.
            os.replace(tmp, self.path)
        except OSError as err:
            self.last_error = f"could not write ({err})"
            _LOG.error("could not persist price history: %s", err)

    # -- the one public entry point -----------------------------------------
    def carry(self, read: PriceRead, *, since: datetime, until: datetime,
              stamp: datetime) -> PriceRead:
        """Remember what this read published, then §8.3-fill the missing hours.

        THE ORDER IS DELIBERATE AND IS THE POINT OF THE METHOD. The read is
        recorded FIRST, so a feed that hands back yesterday alongside today
        refreshes yesterday's record from live data before it is consumed as
        the profile; and the filled series - the one thing that must never be
        recorded - exists only as this method's return value, which this object
        never sees again. See rule 1.

        Returns whatever `fill_from_yesterday()` returns: a series over
        [since, until) whose published hours are the provider's own point
        objects and whose remaining hours are carried and flagged, or
        `no_data` when neither source can produce a single hour.
        """
        self._remember(read, stamp)
        return fill_from_yesterday(read, yesterday=self.yesterday(stamp),
                                   since=since, until=until, stamp=stamp)

    # -- the halves, private because rule 1 is an ordering ------------------
    def _remember(self, read: PriceRead, stamp: datetime) -> None:
        """File a provider read's points under their own Brussels days.

        ⚠️ PRIVATE, AND CALLED FROM EXACTLY ONE PLACE. A caller that can reach
        this can record a filled series, which is the decay in rule 1. There is
        no value-level defence against that - a carried point and a published
        provisional point are the same three fields - so the defence is that
        nothing outside this file names it, and `test_price_history.py` checks
        the tree for that rather than trusting the underscore.
        """
        points = read.match(series=lambda r: r.points, no_data=lambda _r: ())
        if not points:
            return
        product = read.match(series=lambda r: r.product.value,
                             no_data=lambda _r: None)
        provider = read.match(series=lambda r: r.provider,
                              no_data=lambda _r: None)

        touched = set()
        for p in points:
            day = _bdate(p.start)
            rec = self._days.setdefault(
                day, {"product": product, "provider": provider, "points": []})
            # Rule 4: MERGE by instant. A later read updates the hours it
            # carries and leaves the rest; a revised price for an hour already
            # held wins, because it is the newer statement about that hour.
            held = {q["start"]: q for q in rec.get("points", [])
                    if isinstance(q, dict) and "start" in q}
            held[p.start.isoformat()] = p.canonical()
            rec["points"] = sorted(held.values(), key=lambda q: q["start"])
            rec["product"], rec["provider"] = product, provider
            touched.add(day)

        self._prune(stamp)
        self._save()
        _LOG.debug("price history: %d point(s) filed under %s; %d day(s) held",
                   len(points), ", ".join(sorted(touched)), len(self._days))

    def _prune(self, stamp: datetime) -> None:
        """Keep the newest KEEP_DAYS records. Older ones can never be a lookup.

        Pruning is by SORT ORDER, not by arithmetic against `stamp`: a clock
        that jumped - the failure mode WITNESS-03 spent a session on - would
        otherwise delete every record in one pass and leave the fill with
        nothing on the morning after, which is the one morning it is for.
        """
        if len(self._days) <= KEEP_DAYS:
            return
        for day in sorted(self._days)[:-KEEP_DAYS]:
            del self._days[day]

    def yesterday(self, stamp: datetime) -> PriceRead:
        """The published series of the Brussels day BEFORE `stamp`'s, or no_data.

        ⚠️ THE DATE IS COMPUTED, NOT LOOKED UP. See rule 2: `newest()` would
        answer Friday's prices to a Monday fill, and the fill would flag them as
        one day old because that is the only thing its flag can say.
        """
        if stamp.tzinfo is None or stamp.utcoffset() is None:
            raise ValueError(
                f"yesterday({stamp!r}): a naive datetime is not an instant, "
                f"and 'which Brussels day was yesterday' is exactly the "
                f"question a wall-clock reading cannot answer.")
        want = _bdate(brussels(stamp) - timedelta(days=1))
        if self.last_error and not self._days:
            return PriceRead.no_data(
                f"the price history at {self.path} is {self.last_error}, so "
                f"§8.3 has no profile to carry forward for {want}; the plan "
                f"covers only the hours the feed published")
        rec = self._days.get(want)
        if rec is None:
            have = ", ".join(sorted(self._days)) or "no days at all"
            return PriceRead.no_data(
                f"no published prices recorded for {want} (held: {have}), so "
                f"§8.3 has no profile to carry forward. A record for another "
                f"day is not a yesterday")
        points, bad = _parse_points(rec.get("points"))
        if not points:
            why = bad or "it is empty"
            return PriceRead.no_data(
                f"the {want} record holds no usable point ({why}), and an "
                f"empty series is not a free day")
        try:
            product = PriceProduct(rec.get("product"))
        except ValueError:
            return PriceRead.no_data(
                f"the {want} record names product {rec.get('product')!r}, "
                f"which is not a PriceProduct. The optimiser switches on that "
                f"enum, so a record that cannot name one cannot be a series")
        if bad:
            _LOG.warning("price history: %s in the %s record were dropped; "
                         "%d point(s) carried forward", bad, want, len(points))
        return PriceRead.series(points, product,
                                str(rec.get("provider") or "price_history"),
                                stamp)

    # -- observability -------------------------------------------------------
    @property
    def words(self) -> str:
        """One line for the status page. Says what is held, or why nothing is."""
        if not self.ok:
            return f"unavailable - {self.last_error}"
        if not self._days:
            why = f" - {self.last_error}" if self.last_error else ""
            return f"empty{why} - nothing to carry forward yet"
        return "; ".join(f"{d} ({len(self._days[d].get('points') or ())} h)"
                         for d in sorted(self._days))


def _parse_points(raw) -> tuple:
    """Canonical dicts -> PricePoints, and a phrase naming what was dropped.

    ⚠️ A MALFORMED POINT IS DROPPED, NOT DEFAULTED, and the drop is reported.
    `float(row.get("eur_per_kwh", 0))` is the shape of the whole family of
    defects this repo has been removing: a cache that lost a field would answer
    "that hour was free" to a fill that flags it as an ordinary guess.
    """
    if not isinstance(raw, list):
        return (), "the `points` field is not a list"
    out, dropped = [], 0
    for row in raw:
        if not isinstance(row, dict):
            dropped += 1
            continue
        try:
            start = datetime.fromisoformat(row["start"])
            value = float(row["eur_per_kwh"])
            flag = row["provisional"]
        except (KeyError, TypeError, ValueError):
            dropped += 1
            continue
        if start.tzinfo is None or start.utcoffset() is None \
                or not isinstance(flag, bool):
            dropped += 1
            continue
        out.append(PricePoint(start=start, eur_per_kwh=value, provisional=flag))
    out.sort(key=lambda p: p.start.astimezone(timezone.utc))
    return tuple(out), (f"{dropped} malformed point(s)" if dropped else "")


class NoHistory:
    """The explicit "this caller keeps no history" - a NAME, not a `None`.

    ⚠️ IT EXISTS SO THE PARAMETER CAN BE REQUIRED. `history=None` meaning "skip
    the fill" is the shape this repo keeps removing: a caller who forgets the
    argument gets the truncated-horizon behaviour of the bug this ticket is
    about, silently, and the code reads as though the fill is happening. With
    no default there is nothing to forget, and a caller with no durable volume
    - a check, a replay harness - has to say `NoHistory()` and be visible in
    the diff.

    It carries out the ordering faithfully: it records nothing and carries
    nothing, so `carry()` returns the provider's read unchanged and the plan is
    exactly what it was before FCST-04.
    """

    ok = False
    last_error = "no history is kept by this caller"
    words = "not kept"

    def carry(self, read: PriceRead, *, since, until, stamp) -> PriceRead:
        return read

    def yesterday(self, stamp: datetime) -> PriceRead:
        return PriceRead.no_data(
            "this caller keeps no price history (NoHistory), so §8.3 has no "
            "profile to carry forward")