SAFETY-128: control does not act on late Home Assistant readings

Readings that Home Assistant delivers late but still changing passed
every staleness gate, so the loop corrected again each cycle an error
its last command had already fixed (10 s late: the give run swung and
exported to -1.9 kW), and the peak guard pushed to the inverter limit
on the same late figures (OPT-34 F-2).

app/lateness.py fits how long after a power request the meter shows
it. Over 8 s the readings are late: a request the battery reading has
not answered holds the loop (it may not lead the reading by more than
100 W) and caps the peak guard at what stands; unanswered for 23 s it
holds, then 0 W after stale_input_s. An amber row and one log line per
episode say so. With readings on time nothing changes (digests equal).

DECISIONS/SAFETY-128-late-readings.md; test_safety128.py.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bdqr7PiaHcBnzPituu2yTC

goodwe_controller/app/control.py

@@ -298,6 +298,9 @@ PROBE_S = 34.0
 # command: every battery that may move this way is at its mode's limit.
 MODE_BOUND = "mode-bound"
 
+# SAFETY-128. The loop's reason while it waits for the readings to answer.
+WAITING = "waiting for readings"
+
 
 @dataclass(frozen=True)
 class Tuning:
@@ -486,6 +489,8 @@ def compute(
     bound: tuple | None = None,
     gave_nothing: bool | None = None,
     idle_w: float = 0.0,
+    hold: int = 0,
+    hold_w: float | None = None,
 ) -> Decision:
     """One control cycle. A cycle is one meter update (~5 s on a HomeWizard P1).
 
@@ -506,6 +511,9 @@ def compute(
     `idle_w`   SAFETY-97: what `actual_w` reads while the batteries give
                nothing (app/idle.py). The refusal's band and its end are
                centred on it. 0 is the law as it was.
+    `hold`     SAFETY-128: +1 / -1, the batteries have not answered the last
+               command that way (app/lateness.py), so the loop may not go
+               past `hold_w` (None: `prev_w`) that way. 0 is the law as it was.
     """
     reason = "tracking"
     # OPT-11: a bound that leaves out 0 W would FORCE a direction, which no
@@ -622,6 +630,11 @@ def compute(
             i_w = min(moved, i_w)
         else:
             i_w = max(moved, i_w)
+    # SAFETY-128: no further than `hold_w` until the readings answer the ask.
+    held_at = float(prev_w if hold_w is None else hold_w)
+    if hold and hold * (i_w - held_at) > 0:
+        i_w = held_at
+        reason = WAITING
 
     # OPT-11: the integrator does not wind past what the modes let the
     # batteries place. Only a side tighter than the rail is left in `bound`

goodwe_controller/app/lateness.py

@@ -0,0 +1,218 @@
+"""SAFETY-128: a reading that does not answer the last command is not current.
+
+When Home Assistant delivers readings late but still changing, every gate that
+judges arrival (SAFETY-01) or an unchanged value (SAFETY-18, SAFETY-37) passes
+them, and the loop corrects again, each cycle, an error its last command has
+already fixed. The readings themselves show it: after a command the meter moves
+the other way and the battery reading follows (r = 0.998, freshness.py), and
+late readings do both late.
+
+`Answer` watches the site command on the wire, the meter and the site battery
+reading (less the batteries' idle reading, SAFETY-97):
+- `Delay` fits how long after a command the meter shows it. Over DELAY_S the
+  readings run late (`late_seen`); within it they do not. A battery that does
+  not follow gives no fit, so no evidence either way;
+- a new command more than ANSWER_W from the battery reading is PENDING until
+  that reading comes within ANSWER_W of it, or half way. The first move of
+  either reading towards it (ANSWER_W) is its START;
+- while `late_seen` and pending, the loop may not lead the battery reading by
+  more than ANSWER_W that way (`hold`, `hold_w`; not while `exempt`: a
+  battery that gives nothing under its ask or refuses is SAFETY-52/94's case)
+  and the peak guard may not push past what stands (`peak_cap`). Pending for LATE_S with no start is LATE: control
+  holds, then 0 W after `stale_input_s`, as for a missing input.
+
+With readings on time nothing here acts. DECISIONS/SAFETY-128-late-readings.md.
+"""
+
+import bisect
+import math
+from collections import deque
+
+from . import freshness
+from .control import LOOP_TICK_S, PROBE_ANSWER_W
+
+# Within this of the command the reading has answered it (a probe's answer).
+ANSWER_W = PROBE_ANSWER_W                                         # 100
+# A fresh meter shows a command within the plant's settle (SAFETY-75's 6 s dead
+# time) and one meter tick; later than that the readings run late.
+DELAY_S = freshness.PLANT_SETTLE_S + LOOP_TICK_S                   # 8.0
+# The slowest healthy battery answer: SAFETY-75's 6 s dead time, the 6 s plant
+# settle, one 10 s publish and one 1 s read.
+LATE_S = (2 * freshness.PLANT_SETTLE_S + freshness.PUBLISH_PERIOD_S
+          + freshness.READ_PERIOD_S)                              # 23.0
+
+
+# The meter's delay is fitted over this much history, up to this delay.
+FIT_S = 40.0
+MAX_DELAY_S = 90
+FIT_EVERY_S = 4.0
+# A slow battery answers at once but takes this long (SAFETY-85/94 rigs).
+SLOW_TAUS_S = (10.0, 20.0)
+# With no fit for this long the evidence is dropped (a steady site shows none).
+FIT_GONE_S = 120.0
+# A fit counts only if the commands explain this share of the meter's swing.
+FIT_SHARE = 0.5
+# ...and a late delay only if it fits this much better than any fresh one.
+FIT_BETTER = 0.7
+
+
+class Delay:
+    """How long after a command the meter shows it, fitted from history.
+
+    The meter is the house less the batteries, and the batteries follow the
+    commands, so `grid(t) + command(t - d)` is the house alone at the right
+    `d`: the `d` that leaves it flattest is the delay. Only new meter values
+    count, so a silent meter (SAFETY-99) is not a late one. None while the
+    commands have not moved enough to tell, or explain too little of the meter
+    (a battery that does not follow); not late if a slow battery that answers
+    at once (SLOW_TAUS_S) fits as well."""
+
+    def __init__(self):
+        self.cmds = deque()      # (t, command) when it changed
+        self.grid = deque()      # (t, meter)
+
+    def add(self, now, cmd, grid):
+        if not self.cmds or self.cmds[-1][1] != cmd:
+            self.cmds.append((now, cmd))
+        if grid is not None and (not self.grid or self.grid[-1][1] != grid):
+            self.grid.append((now, grid))         # new meter values only
+        while self.grid and self.grid[0][0] < now - FIT_S:
+            self.grid.popleft()
+        while (len(self.cmds) > 1
+               and self.cmds[1][0] <= now - FIT_S - MAX_DELAY_S):
+            self.cmds.popleft()
+
+    def estimate(self):
+        if len(self.grid) < 10 or len(self.cmds) < 2:
+            return None
+        ts = [t for t, _c in self.cmds]
+        t0 = self.cmds[0][0]
+        rows = [(t, g) for t, g in self.grid if t >= t0 + MAX_DELAY_S]
+        if len(rows) < 10:
+            return None
+        cs = [c for _t, c in self.cmds]
+        if max(cs) - min(cs) < ANSWER_W:
+            return None
+        g_mean = sum(g for _t, g in rows) / len(rows)
+        g_var = sum((g - g_mean) ** 2 for _t, g in rows) / len(rows)
+
+        def var(f, d):
+            e = [g + f(t - d) for t, g in rows]
+            m = sum(e) / len(e)
+            return sum((x - m) ** 2 for x in e) / len(e)
+        step = lambda t: cs[bisect.bisect_right(ts, t) - 1]     # noqa: E731
+        pure = [var(step, d) for d in range(0, MAX_DELAY_S + 1)]
+        best = min(range(len(pure)), key=lambda d: (pure[d], d))
+        if g_var <= 0 or pure[best] > (1 - FIT_SHARE) * g_var:
+            return None
+        if best <= DELAY_S:
+            return float(best)
+        # Late only if it fits clearly better than any fresh delay, and better
+        # than a slow battery answering at once (SAFETY-85/94's 10-20 s lag).
+        if pure[best] > FIT_BETTER * min(pure[:int(DELAY_S) + 1]):
+            return None
+        first, last = int(t0), int(rows[-1][0])
+        for tau in SLOW_TAUS_S:
+            k, f, slow = 1 - math.exp(-1.0 / tau), None, []
+            for sec in range(first, last + 1):        # the battery, each second
+                f = step(sec) if f is None else f + (step(sec) - f) * k
+                slow.append(f)
+            if var(lambda t: slow[min(int(t), last) - first], 0) <= pure[best]:
+                return None
+        return float(best)
+
+
+class Answer:
+    """The answer gate. `observe()` once per cycle, before the loop."""
+
+    def __init__(self):
+        self.ask = None          # the pending command, W (site, + = give)
+        self.from_w = None       # the battery reading when it was asked
+        self.from_grid = None    # the meter when it was asked
+        self.since = None        # when it was asked
+        self.started = False     # a reading has moved towards it
+        self.late_seen = False   # the last fitted delay was over DELAY_S
+        self.late_at = None      # when the late verdict started
+        self.exempt = False      # a battery that cannot follow explains it
+        self._was = None         # (command, reading, meter) at the last call
+        self.delay = Delay()
+        self.delay_s = None      # the meter's last fitted delay, s
+        self._fit_at = self._fit_ok = self._exempt_at = None
+
+    @property
+    def active(self) -> bool:
+        return self.ask is not None and self.late_seen
+
+    @property
+    def hold(self) -> int:
+        """+1 / -1: the loop may not push further that way. 0: no hold."""
+        if not self.active or self.exempt:
+            return 0
+        return 1 if self.ask > self.from_w else -1
+
+    @property
+    def hold_w(self):
+        """How far the loop may go while held: ANSWER_W past the reading."""
+        if not self.hold:
+            return None
+        return self._was[1] + self.hold * ANSWER_W
+
+    def peak_cap(self, standing_w):
+        """The most the peak guard may ask, or None."""
+        return max(float(standing_w), 0.0) if self.active else None
+
+    def late_s(self, now):
+        """Seconds the late verdict has stood, or None."""
+        return None if self.late_at is None else max(0.0, now - self.late_at)
+
+    def observe(self, now, standing_w, batt_w, *, grid_w=None, exempt=False,
+                idle_w=0.0):
+        """`standing_w` the site command on the wire, `batt_w` the site
+        battery reading (None: unknown, nothing changes), `grid_w` the meter
+        (+ = import), `idle_w` what the batteries read while giving nothing.
+        `exempt`: a battery gives nothing under its ask, or refuses."""
+        if batt_w is None or standing_w is None:
+            return
+        cmd, r = float(standing_w), float(batt_w) - float(idle_w)
+        g = None if grid_w is None else float(grid_w)
+        self.exempt = bool(exempt)
+        if self.exempt:
+            self._exempt_at = now
+        was, self._was = self._was, (cmd, r, g)
+        self.delay.add(now, cmd, g)
+        if self._fit_at is None or now - self._fit_at >= FIT_EVERY_S:
+            self._fit_at = now
+            d = self.delay.estimate()
+            if self._exempt_at is not None and now - self._exempt_at < FIT_S:
+                d = None              # a battery that does not follow spoils it
+            if d is not None:
+                self.delay_s, self.late_seen, self._fit_ok = d, d > DELAY_S, now
+            elif self._fit_ok is None or now - self._fit_ok > FIT_GONE_S:
+                self.late_seen = False            # nothing to tell it by
+        if self.ask is None:
+            if was is None or cmd == was[0] or abs(cmd - was[1]) < ANSWER_W:
+                return
+            self.ask, self.from_w, self.from_grid = cmd, was[1], was[2]
+            self.since, self.started = now, False
+        self.ask = cmd
+        age = now - self.since
+        gap = self.ask - self.from_w
+        way = 1.0 if gap > 0 else -1.0
+        batt_to = (r - self.from_w) * way
+        grid_to = (-1e9 if g is None or self.from_grid is None
+                   else (self.from_grid - g) * way)
+        if not self.started and max(batt_to, grid_to) >= ANSWER_W:
+            self.started, self.late_at = True, None
+        if (self.ask - r) * way < ANSWER_W or batt_to >= abs(gap) / 2:
+            self._clear()                         # answered
+        elif abs(gap) < ANSWER_W:
+            self._clear()                         # withdrawn
+        elif exempt:
+            self.late_at = None
+        elif (self.late_at is None and self.late_seen and not self.started
+              and age >= LATE_S):
+            self.late_at = now
+
+    def _clear(self):
+        self.ask = self.from_w = self.from_grid = self.since = None
+        self.started, self.late_at = False, None

goodwe_controller/app/main.py

@@ -64,6 +64,7 @@ from .clamp import (
     shortfall_note, write_clamp_note,
 )
 from . import freshness
+from . import lateness
 from . import dayview, homeview
 from .daycurve import SolarToday
 from . import messages
@@ -1281,6 +1282,8 @@ class Controller:
         self.cycle_at = None
         self._arb_action = None
         self.inputs_bad_since = None
+        self.answer = lateness.Answer()           # SAFETY-128
+        self._late_logged = self._late_zero = False
         # SAFETY-03. What the loop's OWN bounds did last cycle - a
         # (engaged, note) pair so the alarm can say WHICH bound bit without
         # re-deriving it from a string. ⚠️ THIS ONE STAYS SITE-LEVEL, and it is
@@ -3116,6 +3119,36 @@ class Controller:
         return (decision if self._peak_needed
                 else dataclasses.replace(decision, within_aim=True))
 
+    def _readings_late(self, standing_w):
+        """SAFETY-128: update the answer gate; seconds the late verdict has
+        stood, or None. Readings found late are logged once per episode.
+        DECISIONS/SAFETY-128-late-readings.md."""
+        a = self.answer
+        if self._now_mono is None:
+            return None
+        a.observe(self._now_mono, standing_w, self.site_batt_w,
+                  grid_w=self.grid, exempt=bool(self.batt_refused) or any(
+                      i.refused or (
+                          i.batt is not None
+                          and abs(i.standing_w()) >= lateness.ANSWER_W
+                          and abs(i.batt - i.idle_w) < lateness.ANSWER_W)
+                      for i in self.inverters),
+                  idle_w=self._site_idle_w())
+        if a.late_seen != self._late_logged:
+            self._late_logged = a.late_seen
+            if a.late_seen:
+                self.log_event(f"readings late ({a.delay_s:.0f} s) - holding",
+                               hint=messages.hint("readings_late"))
+                _LOG.warning("Home Assistant readings arrive late, so control "
+                             "holds the battery: READINGS_LATE.")
+                _LOG.debug("READINGS_LATE: the meter shows a power request "
+                           "%.0f s after it is sent.", a.delay_s)
+            else:
+                self.log_event("readings on time", hint=messages.hint(
+                    "readings_back"))
+                _LOG.info("Home Assistant readings are on time again.")
+        return a.late_s(self._now_mono)
+
     def _peak_reserve(self) -> frozenset:
         """SAFETY-87: the batteries that give nothing for the peak this
         cycle: under their reserve while the others carry a peak floor."""
@@ -4647,6 +4680,8 @@ class Controller:
                     "away a partly delivering unit's output.")
 
         stale_after = float(self.o.get("stale_input_s", 15))
+        late_s = self._readings_late(site_standing)          # SAFETY-128
+        self._late_zero = late_s is not None and late_s >= stale_after
         # ⚠️ `any`, NOT the sole unit's. The hold exists so a single missed poll
         # does not become a fault; a LATCHED inverter must still reach the claim
         # list below, because its `failsafe` SET is what parks it. One tripped
@@ -4672,6 +4707,13 @@ class Controller:
                 self.why_words = messages.why_waiting(bad_for)
                 self._why_waiting_since = now - timedelta(seconds=bad_for)
                 return
+        if late_s is not None and late_s < stale_after and self.inputs_ok \
+                and (self.auto or self.shadow) \
+                and not any(i.sign_wd.tripped for i in self.inverters) \
+                and not (n_units > 1 and self.site_sign_wd.tripped):
+            self.reason = f"readings late {late_s:.0f}s"     # SAFETY-128: hold
+            self.why_words = messages.why_late()
+            return
 
         claims = []
 
@@ -4850,6 +4892,9 @@ class Controller:
                                hint=messages.hint("inputs_missing"))
             claims.append(Claim.set("safety", P_SAFETY, 0.0, "inputs missing",
                                     actuator=self.actuator_of("safety")))
+        elif late_s is not None:
+            claims.append(Claim.set("safety", P_SAFETY, 0.0, "readings late",
+                                    actuator=self.actuator_of("safety")))
 
         # --- SAFETY-07: the quarter-peak projection guard --------------------
         # ONE `lo` limit at P_PEAK, and peakguard.py carries the whole argument.
@@ -4973,6 +5018,8 @@ class Controller:
             self.peak_guard = self._peak_on_able(self.peak_guard,
                                                  maint_results)
             self.peak_guard = self._peak_landing(self.peak_guard)
+            self.peak_guard = peakguard.cap_floor(          # SAFETY-128
+                self.peak_guard, self.answer.peak_cap(site_standing))
         # SITE, and peakguard.claim() says so on the object it builds. The
         # quantity is the 15-minute average OFFTAKE AT THE GRID CONNECTION,
         # which is what the capacity tariff bills; there is one quarter-hour
@@ -5437,6 +5484,7 @@ class Controller:
                               if len(self.inverters) == 1 else None),
                 # SAFETY-97: "gives nothing" is centred on the idle reading.
                 idle_w=self._site_idle_w(),
+                hold=self.answer.hold, hold_w=self.answer.hold_w,  # SAFETY-128
                 # OPT-11: None unless battery modes are on. SAFETY-85: or while
                 # maintenance holds a battery at N>=2, so the loop does not
                 # wind past what the others can give and jump when it ends.
@@ -7333,6 +7381,10 @@ class Controller:
                               (inv.refusal_note(), False)):
                 if note:
                     out.append({"ok": False, "warn": not red, "text": note})
+        if self.answer.late_seen:                             # SAFETY-128
+            out.append({"ok": False, "warn": True, "id": "late",
+                        "text": messages.row_readings_late(
+                            self.answer.delay_s, self._late_zero)})
 
         # SAFETY-03. Red while an over-rating clamp is engaged (amber for the
         # integrator bound, SAFETY-40), and the engagement COUNT is shown even

goodwe_controller/app/messages.py

@@ -644,6 +644,19 @@ def row_controller_stalled(age_s, veto_s) -> str:
               f"stuurmodule zet de omvormer na {seconden(veto_s)} op 0 W.")
 
 
+def row_readings_late(delay_s, zero=False) -> str:
+    """SAFETY-128: Home Assistant readings arrive late; past the hold, 0 W."""
+    if zero:
+        return Bi(f"Home Assistant readings arrive about {seconds(delay_s)} "
+                  f"late, so every inverter is at 0 W.",
+                  f"Metingen uit Home Assistant komen zo'n "
+                  f"{seconden(delay_s)} te laat, omvormers op 0 W.")
+    return Bi(f"Home Assistant readings arrive about {seconds(delay_s)} late, "
+              f"so control holds the batteries.",
+              f"Metingen uit Home Assistant komen zo'n {seconden(delay_s)} te "
+              f"laat, dus de batterijen wachten.")
+
+
 def row_meter_waiting(limit_s) -> str:
     """UX-10: no reading yet since the start, and still inside the limit."""
     return Bi(f"Meter: waiting for the first reading, limit "
@@ -1270,6 +1283,7 @@ WHY_WINNER = {
     "circulation": "Safety stop, inverters worked against each other",
     "stopped": "Control is stopped",
     "inputs missing": "A reading is missing",
+    "readings late": "Home Assistant readings arrive late",
     "no claim was made": "Nothing asked for power",
     # UX-10: the claim reasons main.why_key() can return for a safety or
     # failsafe winner. test_messages.py part 7 finds every one in the code.
@@ -1320,6 +1334,11 @@ def why_missing(what, limit_s) -> str:
     return f"No usable {what} reading for over {seconds(limit_s)}: 0 W"
 
 
+def why_late() -> str:
+    """SAFETY-128: readings arrive late, so control holds the batteries."""
+    return "Home Assistant readings arrive late, control holds"
+
+
 def why_waiting(missing_s) -> str:
     """A reading went missing a moment ago: control holds and waits."""
     return f"A reading is missing for {seconds(missing_s)}, control waits"
@@ -1337,6 +1356,8 @@ HINTS = {
     "roster": "Inverter list read: {n} inverters",
     "maint_history": "Maintenance history did not move over cleanly",
     "stop_restored": "{unit}: an earlier safety stop is back, it stays at 0 W",
+    "readings_late": "Home Assistant readings arrive late, control holds",
+    "readings_back": "Home Assistant readings are on time again",
     "battery_stuck": "{unit}: battery power reading stopped changing",
     "battery_back": "{unit}: battery power reading is used again",
     "battery_refuses": "The battery refuses to give power, the add-on asks 0 W",
@@ -1488,6 +1509,8 @@ def status_samples() -> list:
         ("row_controller_stalled", RED, row_controller_stalled(45, 30)),
         ("row_meter_stale", RED, row_meter_stale(40, 10)),
         ("row_meter_waiting", AMBER, row_meter_waiting(30)),
+        ("row_readings_late", AMBER, row_readings_late(30)),
+        ("row_readings_late", AMBER, row_readings_late(30, True)),
         ("row_meter", GREEN, row_meter(-350, 2, 3, 340)),
         ("row_meter_frozen", RED, row_meter_frozen()),
         *(("row_freeze_off", AMBER, row_freeze_off(k)) for k in FREEZE_OFF),

goodwe_controller/app/peakguard.py

@@ -446,6 +446,19 @@ def shaping_suspended(decision, peak_forecast_w, cap_w) -> bool:
     return decision is not None and decision.engaged
 
 
+def cap_floor(decision, cap_w):
+    """SAFETY-128: the floor no higher than `cap_w` (the command standing while
+    the readings have not answered it), or `decision` unchanged when None.
+    DECISIONS/SAFETY-128-late-readings.md."""
+    if (cap_w is None or decision is None or decision.floor_w is None
+            or decision.floor_w <= cap_w):
+        return decision
+    return replace(
+        decision, floor_w=float(cap_w), saturated=False,
+        reason=decision.reason + f", held at {float(cap_w):+.0f} W until the "
+        "readings answer the last power request")
+
+
 def claim(decision: PeakDecision) -> Claim | None:
     """The decision as one arbiter claim (`lo` only, `hi` stays `+inf`), or None."""
     if not decision.engaged or decision.floor_w is None:

goodwe_controller/app/ui_text/en.json

@@ -476,6 +476,7 @@
       "stopped": "Control is stopped. Every inverter is at 0 W."
     },
     "attn": {
+      "late": "Readings arrive late",
       "home_assistant": "Home Assistant is not answering",
       "controller": "Control is not running",
       "meter": "Meter reading",

goodwe_controller/app/ui_text/nl.json

@@ -397,6 +397,7 @@
     "peak_not_covered": "Geen piekgrens: deze meter meet geen kwartieren.",
     "control": {"shadow": "Schaduwmodus: elke omvormer blijft op 0 W.", "running": "De regeling is actief.", "stopped": "De regeling is gestopt. Elke omvormer staat op 0 W."},
     "attn": {
+      "late": "Metingen komen te laat",
       "home_assistant": "Home Assistant antwoordt niet",
       "controller": "Regeling is niet actief",
       "meter": "Meting van de meter",