SAFETY-128: control does not act on late Home Assistant readings

Readings that Home Assistant delivers late but still changing passed
every staleness gate, so the loop corrected again each cycle an error
its last command had already fixed (10 s late: the give run swung and
exported to -1.9 kW), and the peak guard pushed to the inverter limit
on the same late figures (OPT-34 F-2).

app/lateness.py fits how long after a power request the meter shows
it. Over 8 s the readings are late: a request the battery reading has
not answered holds the loop (it may not lead the reading by more than
100 W) and caps the peak guard at what stands; unanswered for 23 s it
holds, then 0 W after stale_input_s. An amber row and one log line per
episode say so. With readings on time nothing changes (digests equal).

DECISIONS/SAFETY-128-late-readings.md; test_safety128.py.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bdqr7PiaHcBnzPituu2yTC

goodwe_controller/app/lateness.py

@@ -0,0 +1,218 @@
+"""SAFETY-128: a reading that does not answer the last command is not current.
+
+When Home Assistant delivers readings late but still changing, every gate that
+judges arrival (SAFETY-01) or an unchanged value (SAFETY-18, SAFETY-37) passes
+them, and the loop corrects again, each cycle, an error its last command has
+already fixed. The readings themselves show it: after a command the meter moves
+the other way and the battery reading follows (r = 0.998, freshness.py), and
+late readings do both late.
+
+`Answer` watches the site command on the wire, the meter and the site battery
+reading (less the batteries' idle reading, SAFETY-97):
+- `Delay` fits how long after a command the meter shows it. Over DELAY_S the
+  readings run late (`late_seen`); within it they do not. A battery that does
+  not follow gives no fit, so no evidence either way;
+- a new command more than ANSWER_W from the battery reading is PENDING until
+  that reading comes within ANSWER_W of it, or half way. The first move of
+  either reading towards it (ANSWER_W) is its START;
+- while `late_seen` and pending, the loop may not lead the battery reading by
+  more than ANSWER_W that way (`hold`, `hold_w`; not while `exempt`: a
+  battery that gives nothing under its ask or refuses is SAFETY-52/94's case)
+  and the peak guard may not push past what stands (`peak_cap`). Pending for LATE_S with no start is LATE: control
+  holds, then 0 W after `stale_input_s`, as for a missing input.
+
+With readings on time nothing here acts. DECISIONS/SAFETY-128-late-readings.md.
+"""
+
+import bisect
+import math
+from collections import deque
+
+from . import freshness
+from .control import LOOP_TICK_S, PROBE_ANSWER_W
+
+# Within this of the command the reading has answered it (a probe's answer).
+ANSWER_W = PROBE_ANSWER_W                                         # 100
+# A fresh meter shows a command within the plant's settle (SAFETY-75's 6 s dead
+# time) and one meter tick; later than that the readings run late.
+DELAY_S = freshness.PLANT_SETTLE_S + LOOP_TICK_S                   # 8.0
+# The slowest healthy battery answer: SAFETY-75's 6 s dead time, the 6 s plant
+# settle, one 10 s publish and one 1 s read.
+LATE_S = (2 * freshness.PLANT_SETTLE_S + freshness.PUBLISH_PERIOD_S
+          + freshness.READ_PERIOD_S)                              # 23.0
+
+
+# The meter's delay is fitted over this much history, up to this delay.
+FIT_S = 40.0
+MAX_DELAY_S = 90
+FIT_EVERY_S = 4.0
+# A slow battery answers at once but takes this long (SAFETY-85/94 rigs).
+SLOW_TAUS_S = (10.0, 20.0)
+# With no fit for this long the evidence is dropped (a steady site shows none).
+FIT_GONE_S = 120.0
+# A fit counts only if the commands explain this share of the meter's swing.
+FIT_SHARE = 0.5
+# ...and a late delay only if it fits this much better than any fresh one.
+FIT_BETTER = 0.7
+
+
+class Delay:
+    """How long after a command the meter shows it, fitted from history.
+
+    The meter is the house less the batteries, and the batteries follow the
+    commands, so `grid(t) + command(t - d)` is the house alone at the right
+    `d`: the `d` that leaves it flattest is the delay. Only new meter values
+    count, so a silent meter (SAFETY-99) is not a late one. None while the
+    commands have not moved enough to tell, or explain too little of the meter
+    (a battery that does not follow); not late if a slow battery that answers
+    at once (SLOW_TAUS_S) fits as well."""
+
+    def __init__(self):
+        self.cmds = deque()      # (t, command) when it changed
+        self.grid = deque()      # (t, meter)
+
+    def add(self, now, cmd, grid):
+        if not self.cmds or self.cmds[-1][1] != cmd:
+            self.cmds.append((now, cmd))
+        if grid is not None and (not self.grid or self.grid[-1][1] != grid):
+            self.grid.append((now, grid))         # new meter values only
+        while self.grid and self.grid[0][0] < now - FIT_S:
+            self.grid.popleft()
+        while (len(self.cmds) > 1
+               and self.cmds[1][0] <= now - FIT_S - MAX_DELAY_S):
+            self.cmds.popleft()
+
+    def estimate(self):
+        if len(self.grid) < 10 or len(self.cmds) < 2:
+            return None
+        ts = [t for t, _c in self.cmds]
+        t0 = self.cmds[0][0]
+        rows = [(t, g) for t, g in self.grid if t >= t0 + MAX_DELAY_S]
+        if len(rows) < 10:
+            return None
+        cs = [c for _t, c in self.cmds]
+        if max(cs) - min(cs) < ANSWER_W:
+            return None
+        g_mean = sum(g for _t, g in rows) / len(rows)
+        g_var = sum((g - g_mean) ** 2 for _t, g in rows) / len(rows)
+
+        def var(f, d):
+            e = [g + f(t - d) for t, g in rows]
+            m = sum(e) / len(e)
+            return sum((x - m) ** 2 for x in e) / len(e)
+        step = lambda t: cs[bisect.bisect_right(ts, t) - 1]     # noqa: E731
+        pure = [var(step, d) for d in range(0, MAX_DELAY_S + 1)]
+        best = min(range(len(pure)), key=lambda d: (pure[d], d))
+        if g_var <= 0 or pure[best] > (1 - FIT_SHARE) * g_var:
+            return None
+        if best <= DELAY_S:
+            return float(best)
+        # Late only if it fits clearly better than any fresh delay, and better
+        # than a slow battery answering at once (SAFETY-85/94's 10-20 s lag).
+        if pure[best] > FIT_BETTER * min(pure[:int(DELAY_S) + 1]):
+            return None
+        first, last = int(t0), int(rows[-1][0])
+        for tau in SLOW_TAUS_S:
+            k, f, slow = 1 - math.exp(-1.0 / tau), None, []
+            for sec in range(first, last + 1):        # the battery, each second
+                f = step(sec) if f is None else f + (step(sec) - f) * k
+                slow.append(f)
+            if var(lambda t: slow[min(int(t), last) - first], 0) <= pure[best]:
+                return None
+        return float(best)
+
+
+class Answer:
+    """The answer gate. `observe()` once per cycle, before the loop."""
+
+    def __init__(self):
+        self.ask = None          # the pending command, W (site, + = give)
+        self.from_w = None       # the battery reading when it was asked
+        self.from_grid = None    # the meter when it was asked
+        self.since = None        # when it was asked
+        self.started = False     # a reading has moved towards it
+        self.late_seen = False   # the last fitted delay was over DELAY_S
+        self.late_at = None      # when the late verdict started
+        self.exempt = False      # a battery that cannot follow explains it
+        self._was = None         # (command, reading, meter) at the last call
+        self.delay = Delay()
+        self.delay_s = None      # the meter's last fitted delay, s
+        self._fit_at = self._fit_ok = self._exempt_at = None
+
+    @property
+    def active(self) -> bool:
+        return self.ask is not None and self.late_seen
+
+    @property
+    def hold(self) -> int:
+        """+1 / -1: the loop may not push further that way. 0: no hold."""
+        if not self.active or self.exempt:
+            return 0
+        return 1 if self.ask > self.from_w else -1
+
+    @property
+    def hold_w(self):
+        """How far the loop may go while held: ANSWER_W past the reading."""
+        if not self.hold:
+            return None
+        return self._was[1] + self.hold * ANSWER_W
+
+    def peak_cap(self, standing_w):
+        """The most the peak guard may ask, or None."""
+        return max(float(standing_w), 0.0) if self.active else None
+
+    def late_s(self, now):
+        """Seconds the late verdict has stood, or None."""
+        return None if self.late_at is None else max(0.0, now - self.late_at)
+
+    def observe(self, now, standing_w, batt_w, *, grid_w=None, exempt=False,
+                idle_w=0.0):
+        """`standing_w` the site command on the wire, `batt_w` the site
+        battery reading (None: unknown, nothing changes), `grid_w` the meter
+        (+ = import), `idle_w` what the batteries read while giving nothing.
+        `exempt`: a battery gives nothing under its ask, or refuses."""
+        if batt_w is None or standing_w is None:
+            return
+        cmd, r = float(standing_w), float(batt_w) - float(idle_w)
+        g = None if grid_w is None else float(grid_w)
+        self.exempt = bool(exempt)
+        if self.exempt:
+            self._exempt_at = now
+        was, self._was = self._was, (cmd, r, g)
+        self.delay.add(now, cmd, g)
+        if self._fit_at is None or now - self._fit_at >= FIT_EVERY_S:
+            self._fit_at = now
+            d = self.delay.estimate()
+            if self._exempt_at is not None and now - self._exempt_at < FIT_S:
+                d = None              # a battery that does not follow spoils it
+            if d is not None:
+                self.delay_s, self.late_seen, self._fit_ok = d, d > DELAY_S, now
+            elif self._fit_ok is None or now - self._fit_ok > FIT_GONE_S:
+                self.late_seen = False            # nothing to tell it by
+        if self.ask is None:
+            if was is None or cmd == was[0] or abs(cmd - was[1]) < ANSWER_W:
+                return
+            self.ask, self.from_w, self.from_grid = cmd, was[1], was[2]
+            self.since, self.started = now, False
+        self.ask = cmd
+        age = now - self.since
+        gap = self.ask - self.from_w
+        way = 1.0 if gap > 0 else -1.0
+        batt_to = (r - self.from_w) * way
+        grid_to = (-1e9 if g is None or self.from_grid is None
+                   else (self.from_grid - g) * way)
+        if not self.started and max(batt_to, grid_to) >= ANSWER_W:
+            self.started, self.late_at = True, None
+        if (self.ask - r) * way < ANSWER_W or batt_to >= abs(gap) / 2:
+            self._clear()                         # answered
+        elif abs(gap) < ANSWER_W:
+            self._clear()                         # withdrawn
+        elif exempt:
+            self.late_at = None
+        elif (self.late_at is None and self.late_seen and not self.started
+              and age >= LATE_S):
+            self.late_at = now
+
+    def _clear(self):
+        self.ask = self.from_w = self.from_grid = self.since = None
+        self.started, self.late_at = False, None