goodwe-addon/goodwe_controller/app/price_forecast.py

"""FCST-03 - the hours the feed has not published yet, and what may not rest on them.

WHERE THIS LIVES. specs.md §2.2 puts price ingest under `ems/ingest/prices/`.
That layout does not exist and is not being built; the product owner's decision
of 2026-08-30 is that everything ships in `goodwe_controller/app/`. The
BEHAVIOUR §8.3 describes is unchanged and still authoritative - only the path
moved. Read `ems/...` as `app/...`.

§8.3, in full, because every line below is one clause of it:

    "Day-ahead prices arrive once daily. Until they do, unknown hours use
     yesterday's profile and are flagged `provisional: true` - and NO
     IRREVERSIBLE ACTION MAY BE TAKEN ON A PROVISIONAL PRICE. Real-time /
     imbalance products are treated as high-volatility: arbitrage thresholds
     widen."

Three separate things, and this module owns the first two. The third - the
widening arithmetic - lives in price.py beside the product enum it is derived
from (`Uncertainty`, `widen_factor`, `required_spread`), and is spent by
`static_tariff.TariffPlan.arbitrage()`, because that is where the verdict this
repo already refuses to hand out casually is produced. FCST-03 added a refusal
to that existing ladder rather than a second mechanism beside it.


THE DEFECT THIS MODULE IS SHAPED BY: A COPY THAT KEEPS THE WRONG FLAG
=====================================================================
A point filled from yesterday and a point published this morning are the SAME
SHAPE. Three floats and a bool; only the bool tells them apart. And the natural
way to write a carry-forward fill is:

    PricePoint(start=t, eur_per_kwh=y.eur_per_kwh, provisional=y.provisional)
                                                                ^^^^^^^^^^^^^

which copies yesterday's `False` onto today's guess and produces exactly the
thing the ticket exists to prevent: STALE DATA PRESENTED AS CURRENT. It reads
fine, it type-checks, it round-trips through §9.3's JSON, and every price on
the status page looks like a price. This is the meter-side SAFETY-18 defect one
layer up: the reading arrives on time, looks fresh, and is wrong - and a
freshness check that only asks "did something arrive" cannot see it.

So the flag is not passed through anywhere in this file:

  * A CARRIED POINT IS BUILT BY `_carried()`, WHICH HAS NO PARAMETER FOR THE
    FLAG. `provisional=True` is a literal in one place. There is no argument a
    caller can pass, no keyword to forget, and no variable that could hold
    yesterday's value at the moment the point is constructed.
  * A PUBLISHED POINT IS NEVER COPIED AT ALL. It is passed through BY IDENTITY -
    the object the provider produced is the object in the output series - so
    there is no copy that could lose or gain a flag. `filled.points[i] is
    fetched.points[j]` is a checkable property, and test_price_forecast.py
    checks it.
  * `_audit()` RE-DERIVES the expected flag for every output point from the set
    of hours the feed actually published, and RAISES if any point disagrees.
    It is a post-condition, not a filter: it never fixes a flag, because a
    module that quietly corrects its own provenance is one whose provenance
    cannot be trusted.
  * and test_price_forecast.py walks THIS FILE'S AST and fails if any
    `PricePoint(...)` in it passes anything but a literal for `provisional`, so
    the pass-through idiom cannot come back in a later edit and pass review by
    looking ordinary.

⚠️ "PUBLISHED" DOES NOT MEAN "FIRM". `eneco_eddy` is an imbalance product whose
current-quarter print is genuinely provisional and says so per point (§9.2,
TARIFF-02). Such a point is published AND provisional, and the audit's rule for
a published point is "the flag is whatever the feed said", never "False". A
provenance check that forced published points to False would be a bug that
looks like a guarantee.


THE FILL, AND WHY IT KEYS ON THE WALL CLOCK
===========================================
For every whole hour in the requested horizon that the feed has not published,
take yesterday's value for THE SAME BRUSSELS WALL-CLOCK HOUR - not `t - 86400
s`. On the two days a year that matter those are different instants, and the
difference is not academic: a 23-hour day has no 02:00 at all and a 25-hour day
has two, so a fill that subtracts a fixed 24 h shifts every carried hour by one
across a transition and misprices the whole day in a way nothing reports.

On the 25-hour day the two 02:00 hours - `02:00+02:00` and `02:00+01:00` - both
key to local hour 2 and are BOTH filled from yesterday's single 02:00 value.
That is deliberate: they are both guesses, they are both flagged, and nothing
irreversible may rest on either. If yesterday itself was a 25-hour day, the
first of its two 02:00 points is the one used, deterministically.

An hour with no published price AND no yesterday value is LEFT OUT of the
series. It is not filled with a neighbour, a mean, or a zero. §11.4 applies to
that hour - `_Series.at()` raises `Absence` for it - and the caller that wanted
a tariff for the whole horizon falls back to the static profile (TARIFF-03),
which is the row the spec actually writes for "no price data". If NOTHING can
be produced, the result is `PriceRead.no_data(reason)` and not an empty series,
because an empty series is not constructible on purpose (price.py trap 3).


AC2: THE IRREVERSIBLE-ACTION GATE
=================================
"The distinction should be an explicit property of the action, not a judgement
call at each call site" - so an irreversible action is a value, `Irreversible`,
carrying the number of hours it COMMITS. `price_permits()` looks at exactly
those hours and answers with a `Gate` variant that carries the evidence on the
permitting arm and a reason on the blocking one, the `_Outcome` shape price.py
borrowed from `sim/riglock.py`.

The shipped example is the maintenance drain (§12): starting one commits hours
of forced discharge that cannot be undone if the price turns out to have been
yesterday's. Its commitment window is DERIVED from `maintenance.DRAIN_TIMEOUT_MIN`
rather than typed here, so the gate and the phase machine cannot drift apart.

⚠️ WHAT DOES NOT GO THROUGH THIS GATE, AND WHY THAT MATTERS MORE THAN WHAT
DOES. The maintenance cycle is a CALENDAR obligation (§12.2: a cycle that
silently never fires is indistinguishable from one that is not due yet), and a
site with `price_provider: off` - the shipped default - has no price series at
all. Routing every maintenance start through a gate that blocks on `no_data`
would stop the monthly cycle on every default install, permanently and
silently: the exact failure §12 opens by naming. So `begin_phase()` is the
entry point for a start that is being JUSTIFIED BY A PRICE, main.py's existing
calendar path is untouched by this ticket, and OPT-04/MAINT-05 own the decision
of which one a site runs. Nothing here disables anything that runs today.

Reversible actions - ordinary self-consumption modulation, the loop that
actually holds the grid at zero - are not gated at all and cannot be: §6 says
absent machinery must not disable the loop.
"""

import logging
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import Enum

from .maintenance import CHARGE_TIMEOUT_MIN, DRAIN_TIMEOUT_MIN
from .price import PricePoint, PriceRead, _Outcome, brussels

_LOG = logging.getLogger("goodwe.price.forecast")


class Provenance(Enum):
    """Where a point in a filled series came from. INTERNAL - §9.3's wire format
    carries the boolean and nothing else, so this never leaves the module.

    It exists so `_audit()` has a vocabulary to state its post-condition in, and
    so the two cases have names in the log rather than being "the ones with the
    flag" and "the others".
    """
    PUBLISHED = "published"    # the provider printed this hour. Flag: the
                               # feed's own - an imbalance print may be
                               # published AND provisional (§9.2).
    CARRIED = "carried"        # yesterday's same-wall-clock hour (§8.3).
                               # Flag: ALWAYS True, by construction.


# --------------------------------------------------------------------------- #
# The fill
# --------------------------------------------------------------------------- #

def _utc_hour(instant: datetime) -> datetime:
    u = instant.astimezone(timezone.utc)
    return u.replace(minute=0, second=0, microsecond=0)


def _carried(start_utc: datetime, eur_per_kwh: float) -> PricePoint:
    """A point filled from yesterday's profile. §8.3: ALWAYS provisional.

    ⚠️ THERE IS NO `provisional` PARAMETER, AND THAT IS THE MECHANISM. This is
    the only constructor of a carried point, the flag is a literal, and no
    caller - including a future one editing in a hurry - has anything to pass,
    default, or copy from the point being carried.
    """
    return PricePoint(start=brussels(start_utc),
                      eur_per_kwh=round(float(eur_per_kwh), 6),
                      provisional=True)


def _audit(points, published) -> None:
    """Post-condition: every flag matches the point's TRUE provenance (AC1).

    `published` maps a UTC hour to the point the provider produced for it.

    ⚠️ RAISES, NEVER REPAIRS. A module that silently corrects its own
    provenance is one whose provenance means nothing; the only safe response to
    "this point's flag disagrees with where it came from" is to refuse to emit
    the series at all.
    """
    seen = set()
    for p in points:
        u = _utc_hour(p.start)
        if u in seen:
            raise ValueError(f"the filled series repeats {u.isoformat()}; a "
                             f"duplicated hour is how a DST day gets mispriced")
        seen.add(u)
        src = published.get(u)
        prov = Provenance.CARRIED if src is None else Provenance.PUBLISHED
        if src is None:
            if p.provisional is not True:
                raise ValueError(
                    f"{u.isoformat()} is {prov.value} and is flagged "
                    f"provisional={p.provisional!r}. §8.3: an hour filled from "
                    f"yesterday's profile is provisional whether or not the "
                    f"number turns out to be right - the flag is a claim about "
                    f"PROVENANCE, not about accuracy.")
        else:
            if p is not src:
                raise ValueError(
                    f"{u.isoformat()} is {prov.value} but the series carries a "
                    f"COPY of that point rather than the point itself. A copy "
                    f"is where the provisional flag gets lost; published points "
                    f"pass through by identity.")


def fill_from_yesterday(read: PriceRead, *, yesterday: PriceRead,
                        since: datetime, until: datetime,
                        stamp: datetime) -> PriceRead:
    """§8.3: fill the hours the feed has not published with yesterday's profile.

    Every argument is required and keyword-only past the first. In particular
    `yesterday` has NO DEFAULT: a fill with no yesterday to fill from is a
    different situation from a fill that found nothing useful in one, and a
    default `None` would make the two arrive looking identical.

    Returns a `PriceRead`:

      * a series, whose points are the provider's own objects for the hours it
        published and `_carried()` points for the rest, ordered by instant;
      * `no_data` when neither source can produce a single hour - never an
        empty series, which is not constructible (price.py trap 3).

    ⚠️ IDEMPOTENT AND RE-RUNNABLE, WHICH IS THE §5.4 LATE-PUBLICATION CONTRACT.
    Day-ahead publication at 14-16 h CET can be late or fail, so the fetcher
    retries and this runs again against the fresher read. Published hours always
    win over carried ones, so a re-run REPLACES a carried point with the real
    print and the hour becomes firm - the flag and the number move together
    because the whole series is rebuilt rather than patched in place.
    """
    for name, value in (("since", since), ("until", until), ("stamp", stamp)):
        if value.tzinfo is None or value.utcoffset() is None:
            raise ValueError(f"{name} must be tz-aware; a naive instant is "
                             f"ambiguous for one hour every October")

    published = read.match(
        series=lambda r: {_utc_hour(p.start): p for p in r.points},
        no_data=lambda _r: {})

    # Yesterday's profile, keyed by BRUSSELS WALL-CLOCK HOUR. First point wins
    # when yesterday itself was a 25-hour day and printed local hour 2 twice.
    by_local_hour = {}
    yesterday.match(
        series=lambda r: [by_local_hour.setdefault(brussels(p.start).hour, p)
                          for p in r.points],
        no_data=lambda _r: [])

    out = []
    u = _utc_hour(since)
    end = until.astimezone(timezone.utc)
    while u < end:
        if u in published:
            out.append(published[u])                    # BY IDENTITY. No copy.
        else:
            src = by_local_hour.get(brussels(u).hour)
            if src is not None:
                out.append(_carried(u, src.eur_per_kwh))
        u += timedelta(hours=1)

    if not out:
        reason = read.match(
            series=lambda r: (f"the {len(r.points)} published hours all fall "
                              f"outside {since.isoformat()}..{until.isoformat()}"),
            no_data=lambda r: r.reason)
        return PriceRead.no_data(
            f"§8.3 carry-forward produced no hours: {reason}, and yesterday's "
            f"profile has nothing to fill them with either. §11.4 applies: "
            f"static profile, plan provisional, no arbitrage")

    _audit(out, published)
    carried = sum(1 for p in out if _utc_hour(p.start) not in published)
    if carried:
        _LOG.info("§8.3: %d of %d hours carried forward from yesterday's "
                  "profile and flagged provisional; %d published",
                  carried, len(out), len(out) - carried)

    product = read.match(series=lambda r: r.product,
                         no_data=lambda _r: yesterday.product)
    provider = read.match(series=lambda r: r.provider,
                          no_data=lambda r: yesterday.provider)
    return PriceRead.series(out, product, provider, stamp)


# --------------------------------------------------------------------------- #
# AC2: irreversibility as a property of the ACTION, and the gate that reads it
# --------------------------------------------------------------------------- #

@dataclass(frozen=True)
class Irreversible:
    """An action that cannot be undone once started, and how far it commits.

    `commits_h` is the horizon the action ties up - the hours whose prices must
    therefore be firm before it may be justified by a price at all.
    """

    name: str            # the §10.2 reason code
    commits_h: float

    def __post_init__(self):
        if not self.name or not isinstance(self.name, str):
            raise ValueError("an irreversible action needs a reason code (§10.2)")
        if isinstance(self.commits_h, bool) or \
                not isinstance(self.commits_h, (int, float)) or \
                not self.commits_h > 0:
            raise ValueError(
                f"commits_h must be a positive number of hours, not "
                f"{self.commits_h!r}. An action that commits nothing is a "
                f"reversible one and does not belong here.")


# ⚠️ DERIVED FROM THE PHASE MACHINE'S OWN BOUNDS, NOT TYPED HERE. maintenance.py
# already states how long each phase may run before it is a fault; a second copy
# of those numbers here would be a place for them to drift apart, and the drift
# would show up as a gate that checked the wrong hours.
MAINTENANCE_DRAIN = Irreversible("MAINTENANCE_DRAIN", DRAIN_TIMEOUT_MIN / 60.0)
MAINTENANCE_CHARGE = Irreversible("MAINTENANCE_CHARGE", CHARGE_TIMEOUT_MIN / 60.0)


class Gate(_Outcome):
    """May an irreversible action proceed on the price evidence available?

      * `permitted` - carries `.action` and `.hours`, the firm points that were
                      actually checked. The evidence exists only on this arm.
      * `blocked`   - carries `.reason`. Nothing else; `.hours` raises `Absence`.

    No `__bool__`, both `match()` arms required and keyword-only, for the reason
    price.py's `_Outcome` gives: `if gate:` must not look like a guard while
    being none.
    """

    def match(self, *, permitted, blocked):
        return {"permitted": permitted, "blocked": blocked}[self.kind](self)

    @staticmethod
    def permit(action, hours):
        return _Permitted(action, hours)

    @staticmethod
    def block(reason):
        if not reason or not isinstance(reason, str):
            raise ValueError(
                "Gate.block() takes a reason a human can act on. §6 requires "
                "every decision to explain itself, and 'the maintenance cycle "
                "did not start' with no reason is indistinguishable from a bug.")
        return _Blocked(reason)


class _Permitted(Gate):
    kind = "permitted"

    def __init__(self, action, hours):
        if not isinstance(action, Irreversible):
            raise TypeError(f"action must be an Irreversible, not {action!r}")
        pts = tuple(hours)
        if not pts:
            raise ValueError(
                "a permission granted over ZERO checked hours is a permission "
                "granted on no evidence - the shape this whole file exists to "
                "refuse. Block instead.")
        if any(p.provisional for p in pts):
            raise ValueError(
                "a permitted gate cannot carry a provisional hour: §8.3 is the "
                "one rule this type has.")
        self.action = action
        self.hours = pts
        self.words = (f"{action.name} permitted on {len(pts)} firm hours "
                      f"({pts[0].start.isoformat()} ..)")


class _Blocked(Gate):
    kind = "blocked"

    def __init__(self, reason):
        self.reason = reason
        self.words = f"blocked - {reason}"


def committed_hours(now: datetime, action: Irreversible) -> tuple:
    """The whole hours `action` started at `now` would run through.

    Split out because the gate and its caller must agree on what "the hours it
    commits" means, and because a window that quietly excluded the last hour
    would be a gate that permits an action on an hour it never looked at.
    """
    if not isinstance(action, Irreversible):
        raise TypeError(f"action must be an Irreversible, not {action!r}")
    if now.tzinfo is None or now.utcoffset() is None:
        raise ValueError("now must be tz-aware")
    out = []
    u = _utc_hour(now)
    end = now.astimezone(timezone.utc) + timedelta(hours=action.commits_h)
    while u < end:
        out.append(u)
        u += timedelta(hours=1)
    return tuple(out)


def price_permits(read: PriceRead, now: datetime,
                  action: Irreversible) -> Gate:
    """§8.3's rule, as a value: may this irreversible action rest on this price?

    Blocks when the read observed nothing (§11.4 - an action justified by a
    price nobody has is an action justified by nothing), when any hour it
    commits has no price at all, and when any of those hours is provisional.
    Names the offending hour, so an operator reading the log knows which one.
    """
    hours = committed_hours(now, action)
    return read.match(
        series=lambda r: _walk(r, hours, action),
        no_data=lambda r: Gate.block(
            f"{action.name} is gated on a price and there is no price data: "
            f"{r.reason}. §11.4 - and a missing price is not a cheap one"))


def _walk(series, hours, action) -> Gate:
    pts = []
    for u in hours:
        if not series.covers(u):
            return Gate.block(
                f"{action.name} commits {action.commits_h:g} h from "
                f"{hours[0].isoformat()} and no price covers "
                f"{u.isoformat()}. An hour outside the series is an hour with "
                f"no price (§11.4 for that hour), not a free one")
        p = series.at(u)
        if p.provisional:
            return Gate.block(
                f"{action.name} commits {action.commits_h:g} h from "
                f"{hours[0].isoformat()} and {u.isoformat()} is PROVISIONAL - "
                f"carried from yesterday's profile or standing in for a dead "
                f"feed. §8.3: no irreversible action may be taken on a "
                f"provisional price. It becomes available when the day-ahead "
                f"print lands (~14-16 h CET, §5.4)")
        pts.append(p)
    return Gate.permit(action, pts)


def begin_phase(maint, now: datetime, read: PriceRead,
                action: Irreversible) -> Gate:
    """Start a maintenance phase BECAUSE OF A PRICE, or refuse and say why.

    The AC2 example made real: `maint.force_start(now)` - the call that moves
    the phase machine out of `idle` and commits the pack to hours of forced
    discharge - happens on the permitting arm and cannot happen on the other
    one, because the blocking arm has no code path to it.

    ⚠️ READ THE MODULE DOCSTRING'S LAST WARNING BEFORE WIRING THIS IN. This is
    the price-justified entry point, not the calendar one; a site with no price
    provider must keep starting its monthly cycle the way it does today, or
    §12's opening sentence comes true - a cycle that silently never fires.
    """
    gate = price_permits(read, now, action)
    return gate.match(permitted=lambda g: _go(maint, now, g),
                      blocked=lambda g: _no(g))


def _go(maint, now, gate) -> Gate:
    maint.force_start(now)
    _LOG.info("%s: %s", gate.action.name, gate.words)
    return gate


def _no(gate) -> Gate:
    _LOG.warning("irreversible action refused - %s", gate.reason)
    return gate