SAFETY-10: the whole decision path runs and nothing reaches the inverter
checks / unit (pull_request) Successful in 7m58s
checks / unit (pull_request) Successful in 7m58s
Every optimiser defect this project has found was invisible from outside while it was happening, and each was paid for at the meter first: SAFETY-45's 28 charge onsets an hour against a binding peak_cap_w, TARIFF-14's wrong quarter-hour for 45 minutes in every 60, SAFETY-46's 598 Wh spent to gain 310 Wh. There was no way to watch what the optimiser WOULD do without letting it do it. `shadow_mode` runs strategies, arbiter, clamps and guards, records the decision with its reason, and commands nothing. ⚠️ IT WRITES. It writes 0 W, on the ordinary heartbeat, through the ordinary single writer. The inverter holds its last command forever - it has no meter timeout - so "commands nothing" implemented as "stops writing" would leave whatever was standing when shadow mode came on standing for as long as it was left on, under a page saying nothing is being commanded. That is the failure this repository keeps finding. Refreshing 0 W is also what firmware/goodwe-master.yaml's 30 s stale-input watchdog requires of us; going quiet would keep that board tripped for the whole run, which is an alarm being used as a mode. ⚠️ IT IS NOT A SECOND WRITE PATH. No claim, no transport, no arithmetic of its own. It is a substitution inside Controller._write_unit() - the single choke point every write already reaches - in the same form as the three above it, and it is the LAST of them, so SAFETY-28's gate and SAFETY-03's clamp still grade the number the decision path produced. ⚠️ THE DEFAULT IS `auto`, NOT `true`, AND THAT IS THE CARD'S REAL CONTENT. The Supervisor composes options as {**config.yaml defaults, **stored user options}, so a new key with a new default reaches EVERY EXISTING INSTALL on the next update - including the one regulating a real battery. A boolean default-on would have stopped it silently. There is no mechanism by which one key can default differently for a new install and an existing one, so "default-on for new" is a value the add-on RESOLVES: `auto` is shadow while control is STOPPED and live once it is STARTED. A fresh install ships `auto_start: false` and comes up observing; an install that is controlling is started and does not change by one watt. THE FAILSAFES. Sign watchdogs (SAFETY-02/30): INERT - they grade Inverter.standing_w(), which shadow mode holds at 0 W. Fed the shadow decision they would disagree with a meter that never saw it and LATCH A TRIP THAT SURVIVES INTO LIVE OPERATION AND A RESTART. Clamp (SAFETY-03), write gate (SAFETY-28), meter freeze (SAFETY-18), peak guard: OBSERVE. Circulation (SAFETY-31) and delivery (SAFETY-42): observe and may legitimately latch, because their evidence is measured power. The operator stop is the one deliberate change: enforced at the wire rather than as a P_SAFETY claim, which would otherwise make every recorded decision read "stopped". ⚠️ RECORDED LIMITATION: _shadow_settle() re-seeds the carried command to what stands on the wire, which makes the exit safe by construction - and therefore the loop leg's shadow decision is ONE SLEW STEP, not a trajectory. A shadow run answers "what would it command now", not "what would it have done next". The strategies are unaffected; all three defects above are visible from one. Graded on the writes and on the rendered surface, never on the flag: 22 mutations, no survivors, including one that lets a write escape and one that goes silent instead of commanding 0 W. Closed loop on TEST-23 stage 3's rig, same hour twice: live drives the pack to -2000 W and SoC 50.000 -> 62.331 %; shadow commands 0 W throughout, SoC stays 50.0 exactly, and the arbitrage decision is identical in action, instant, signed power, reason and arbiter winner. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
This commit is contained in:
Claude
@@ -66,6 +66,7 @@ from .mqtt import (MqttPublisher, announced_units, unit_slugs,
|
||||
unit_state_key)
|
||||
from . import peakguard
|
||||
from . import packshare
|
||||
from . import shadow
|
||||
from .p1 import P1Ingest, build_source, is_enabled
|
||||
from .price import Uncertainty, price_option, provider_configured
|
||||
from .price_history import PriceHistory
|
||||
@@ -830,6 +831,24 @@ class Controller:
|
||||
|
||||
# live state
|
||||
self.auto = bool(store.data.get("auto", opts.get("auto_start", False)))
|
||||
# --- SAFETY-10 -------------------------------------------------------
|
||||
# What the decision path reached on the last shadow cycle, in W, or
|
||||
# None before the first one. ⚠️ IT IS KEPT SEPARATELY FROM
|
||||
# `self.target` BECAUSE THEY ARE DIFFERENT FACTS: `self.target` is what
|
||||
# is STANDING ON THE WIRE - held at `shadow.WIRE_W` for the whole of a
|
||||
# shadow run, and re-seeded to it at the end of every shadow cycle so
|
||||
# the loop and the sign watchdogs track reality - and this is what a
|
||||
# live controller WOULD have put there. Folding them into one field is
|
||||
# how a shadow run starts telling an operator a number nothing is
|
||||
# doing.
|
||||
self.shadow_would_w = None
|
||||
# Edge state for the action log, both of them. `_shadow_was` is None
|
||||
# until the first cycle, so an install that comes up IN shadow mode
|
||||
# says so in the log it will be read from rather than leaving the
|
||||
# operator to infer it from a page nobody is looking at yet. Coming up
|
||||
# LIVE is silent - see `_shadow_settle()`, which says why.
|
||||
self._shadow_was = None
|
||||
self._shadow_winner = None
|
||||
self.target = 0.0
|
||||
self.sat_count = 0
|
||||
self.i_w = 0.0 # the loop's integrator, carried between cycles
|
||||
@@ -1739,6 +1758,42 @@ class Controller:
|
||||
"reports a different value.",
|
||||
inv.name, inv.batt_fresh.reason)
|
||||
|
||||
@property
|
||||
def shadow(self) -> bool:
|
||||
"""SAFETY-10. Is the battery being left alone right now?
|
||||
|
||||
⚠️ A PROPERTY, NOT A FIELD, AND FOR THE REASON `self.units` IS ONE.
|
||||
Three surfaces ask this question at three different moments - the
|
||||
cycle, the write, and the status page rendered between two cycles - and
|
||||
a cached flag is a fourth answer that can disagree with the option and
|
||||
the button it is derived from. Derived, they cannot.
|
||||
|
||||
⚠️ AND THE WRITE GATE ASKS THIS, NOT A PARAMETER PASSED DOWN TO IT. A
|
||||
suppression that had to be threaded through `write_setpoint()` would be
|
||||
absent from every call site that forgot it - the heartbeat, the stop
|
||||
button, `shutdown()` - which is exactly the shape of hole this add-on's
|
||||
single choke point exists to close.
|
||||
"""
|
||||
return shadow.engaged(self.o, control_started=self.auto)
|
||||
|
||||
def shadow_latches(self) -> tuple:
|
||||
"""The failsafe latches that are UP, named, for the shadow banner.
|
||||
|
||||
A latch that is standing while the controller is inert is standing when
|
||||
it stops being inert. `shadow.banner()` carries them so an operator who
|
||||
turns shadow mode off is not handed a battery that will not move and a
|
||||
page that never mentioned why.
|
||||
"""
|
||||
out = []
|
||||
if self.circ_wd.tripped:
|
||||
out.append("circulating power")
|
||||
if self.site_sign_wd.tripped and len(self.inverters) > 1:
|
||||
out.append("site sign watchdog")
|
||||
for inv in self.inverters:
|
||||
if inv.sign_wd.tripped:
|
||||
out.append(f"{inv.name} sign watchdog")
|
||||
return tuple(out)
|
||||
|
||||
def write_refusal(self, value: float, inv=None) -> str | None:
|
||||
"""Why `value` may NOT be written to `inv` right now, or None if it may.
|
||||
|
||||
@@ -1930,6 +1985,40 @@ class Controller:
|
||||
self.log_event(f"{inv.name}: {result.describe()}")
|
||||
value = result.applied_w
|
||||
|
||||
# --- SAFETY-10: AND THIS IS THE LAST THING BEFORE THE WIRE -----------
|
||||
# Shadow mode runs the whole decision path and commands nothing. The
|
||||
# suppression is HERE, in the one method every write in this add-on
|
||||
# reaches - the cycle, the heartbeat, the stop button, shutdown() and
|
||||
# maint_abort() alike - so there is no call site that can route round
|
||||
# it. app/shadow.py carries the whole argument; three things about the
|
||||
# placement belong here:
|
||||
#
|
||||
# ⚠️ IT IS A SUBSTITUTION, NOT A RETURN, exactly like the two refusals
|
||||
# above it. Returning would leave the inverter LATCHED ON ITS LAST
|
||||
# COMMAND with the ESP32's heartbeat still being fed - a battery held
|
||||
# at whatever it was doing when shadow mode was switched on, under a
|
||||
# status page saying nothing is being commanded. That is the failure
|
||||
# this add-on keeps finding, and shadow mode must not be a new way to
|
||||
# reach it. `shadow.WIRE_W` is 0.0 W and it is REFRESHED on the
|
||||
# ordinary heartbeat, which is also what firmware/goodwe-master.yaml's
|
||||
# 30 s stale-input watchdog requires of us.
|
||||
#
|
||||
# ⚠️ IT IS BELOW SAFETY-28's GATE AND BELOW SAFETY-03's CLAMP, so both
|
||||
# still grade the number the decision path actually produced. A shadow
|
||||
# run that could not say "it would have hit the rating here" would be
|
||||
# answering a different question from the one it is for.
|
||||
#
|
||||
# ⚠️ IT IS NOT A SECOND WRITER. No claim is filed, no transport is
|
||||
# opened and no arithmetic of its own reaches the wire: one actuator,
|
||||
# one writer, and this is that writer emitting the value it emits for
|
||||
# every other refusal.
|
||||
if self.shadow and value != shadow.WIRE_W:
|
||||
_LOG.info("SHADOW MODE: not commanding %+.0f W to %s - holding "
|
||||
"%+.0f W. The decision was reached and recorded; "
|
||||
"nothing was sent to the inverter.",
|
||||
value, inv.name, shadow.WIRE_W)
|
||||
value = shadow.WIRE_W
|
||||
|
||||
changed = (inv.last_write is None or value != inv.last_write)
|
||||
if not (changed or force):
|
||||
return
|
||||
@@ -2549,7 +2638,16 @@ class Controller:
|
||||
# operator switching control off for the whole site; "inputs missing" is
|
||||
# the SITE's own telemetry gone - the meter this controller regulates
|
||||
# against. Neither is about a pack, and both must stop every inverter.
|
||||
if not self.auto:
|
||||
# ⚠️ SAFETY-10: `and not self.shadow`, AND IT IS THE ONE FAILSAFE THIS
|
||||
# CARD CHANGES. `P_SAFETY` out-ranks every other claim, so leaving the
|
||||
# stop claim in while shadow mode is engaged would make the recorded
|
||||
# decision "stopped" on every cycle - a shadow run that answers the
|
||||
# wrong question, which is the same as no shadow run. The actuator ends
|
||||
# at the same number either way, and the substitution in `_write_unit()`
|
||||
# that replaces it is STRICTLY STRONGER than this claim: a claim can be
|
||||
# out-ranked by a higher priority, and the wire gate cannot be
|
||||
# out-ranked by anything.
|
||||
if not self.auto and not self.shadow:
|
||||
claims.append(Claim.set("safety", P_SAFETY, 0.0, "stopped",
|
||||
actuator=self.actuator_of("safety")))
|
||||
elif not self.inputs_ok:
|
||||
@@ -2893,7 +2991,14 @@ class Controller:
|
||||
inv)))
|
||||
|
||||
# --- the grid-following loop -----------------------------------------
|
||||
if self.auto and self.inputs_ok:
|
||||
# ⚠️ SAFETY-10: `or self.shadow`. The loop is the leg a shadow run most
|
||||
# needs and the one that is switched off by `auto_start: false`, so
|
||||
# without this an install in `auto` shadow mode would record every
|
||||
# strategy's decision and a permanent silence where the grid-following
|
||||
# loop should be. It is safe because the loop's `prev_w` is
|
||||
# `self.target`, which the end of this method re-seeds to what is
|
||||
# actually standing on the wire.
|
||||
if (self.auto or self.shadow) and self.inputs_ok:
|
||||
# ⚠️ prev is the ARBITER's last output, not the loop's own last
|
||||
# wish. If something outranked the loop, that is what the hardware
|
||||
# actually did, and the controller must track reality or it jumps
|
||||
@@ -3227,6 +3332,116 @@ class Controller:
|
||||
# reads only `self.p1`, which no line in this method writes.
|
||||
self.peak.sync(self.p1)
|
||||
|
||||
# --- SAFETY-10, AND IT IS LAST FOR A REASON THAT IS NOT TIDINESS -----
|
||||
# Every guard above has now read `self.target` - the resolved intent -
|
||||
# and SAFETY-18 in particular grades exactly that. Re-seeding before
|
||||
# them would hand each one a 0 W that no decision produced. Re-seeding
|
||||
# AFTER them hands the NEXT cycle, and the sign watchdogs, what is
|
||||
# actually standing on the wire. See `_shadow_settle()`.
|
||||
self._shadow_settle(resolution)
|
||||
|
||||
def _shadow_settle(self, resolution) -> None:
|
||||
"""SAFETY-10. Record the decision, then re-seed the carried command.
|
||||
|
||||
TWO JOBS, AND THEY ARE IN THIS ORDER BECAUSE THE SECOND DESTROYS THE
|
||||
FIRST'S EVIDENCE.
|
||||
|
||||
1. THE RECORD. A shadow run is only worth running if it answers "what
|
||||
would it have done, and why". `self.shadow_would_w` and
|
||||
`Resolution.explain()` are that answer, and the action log gets one
|
||||
line per CHANGE OF DECISION rather than one per cycle - the log
|
||||
holds 40 rows, and a level-logged shadow run would flush the context
|
||||
of its own findings inside four minutes at a 15 s cycle floor. The
|
||||
same edge rule the peak guard and the arbitrage strategy already
|
||||
use, for the same reason.
|
||||
|
||||
2. THE RE-SEED. `self.target` and each `inv.target` are set back to
|
||||
`shadow.WIRE_W` - what the write gate actually put on the wire.
|
||||
|
||||
⚠️ WHY THE RE-SEED IS A SAFETY PROPERTY AND NOT BOOKKEEPING. Three
|
||||
machines read the carried command as a statement about the HARDWARE:
|
||||
|
||||
* `compute(prev_w=self.target, ...)` - the loop's slew limiter and
|
||||
its integrator. Left holding a shadow decision, the first cycle
|
||||
after shadow mode is turned off would slew from a base that never
|
||||
existed: up to a full `slew_w` step onto a battery sitting at 0 W.
|
||||
Re-seeded, the exit is safe BY CONSTRUCTION and needs no separate
|
||||
transition code - the carried command already equals the standing
|
||||
one.
|
||||
* `Inverter.standing_w()`, which is what SAFETY-02/30's sign
|
||||
watchdogs grade. Fed a shadow decision they would compare a number
|
||||
against a meter that never saw it, disagree for as long as the
|
||||
decision asked for power, and LATCH A TRIP THAT SURVIVES INTO LIVE
|
||||
OPERATION AND A RESTART. Shadow mode would manufacture the failsafe
|
||||
it exists to help debug. ⚠️ THIS ONE IS ALREADY HELD BY THE WIRE
|
||||
GATE - `standing_w()` prefers `last_write`, and that is the 0.0 W
|
||||
the gate accepted - so the re-seed is the SECOND rung, covering
|
||||
the fallback before any write has been accepted. Two rungs
|
||||
answering one question is MUTATION.md's masking shape, and the
|
||||
mutation that reproduces the hazard has to defeat BOTH: driven
|
||||
singly, dropping this line leaves the detector inert and names
|
||||
nothing about the latch. Measured.
|
||||
* SAFETY-42's delivery watchdog compares `inv.target` against what
|
||||
the pack is producing. A share nothing was told to produce is a
|
||||
delinquency report on every cycle of every shadow run.
|
||||
|
||||
⚠️ AND EVERY CYCLE'S DECISION IS THEREFORE COMPUTED FROM WHERE THE
|
||||
PLANT REALLY IS, not from a counterfactual plant that drifts further
|
||||
from reality every tick. "What would this controller command NOW,
|
||||
given this meter and a battery at 0 W" is a true statement about the
|
||||
present; "what would the eleventh cycle of a run that never happened
|
||||
have commanded" is not a statement about anything.
|
||||
"""
|
||||
engaged = self.shadow
|
||||
if engaged != self._shadow_was:
|
||||
# ⚠️ `first` EXISTS SO A LIVE INSTALL'S FIRST CYCLE IS SILENT. The
|
||||
# edge is against `None` on cycle one, and an install that comes up
|
||||
# CONTROLLING has nothing to report - a "shadow mode off" line at
|
||||
# every start is one more row of boilerplate in a 40-row log, and
|
||||
# it was measured pushing a real clamp episode's context out of
|
||||
# test_clamp.py part 2's ring. Coming up IN shadow mode is the
|
||||
# other direction and it is always worth a line: it is the log an
|
||||
# operator reads to find out why nothing is being commanded.
|
||||
first = self._shadow_was is None
|
||||
self._shadow_was = engaged
|
||||
if engaged:
|
||||
self.log_event(
|
||||
"SHADOW MODE ON - " + shadow.why(
|
||||
self.o, control_started=self.auto)
|
||||
+ f"; inverter held at {shadow.WIRE_W:.0f} W")
|
||||
_LOG.warning(
|
||||
"SAFETY-10 SHADOW MODE IS ON. The full decision path runs "
|
||||
"and NOTHING is commanded: every inverter is held at "
|
||||
"%+.0f W on the ordinary heartbeat. %s", shadow.WIRE_W,
|
||||
shadow.why(self.o, control_started=self.auto))
|
||||
elif not first:
|
||||
self.log_event(
|
||||
"SHADOW MODE OFF - the battery is being controlled again, "
|
||||
f"resuming from {shadow.WIRE_W:.0f} W")
|
||||
_LOG.warning(
|
||||
"SAFETY-10 shadow mode is OFF. This controller is "
|
||||
"commanding the battery again, starting from the "
|
||||
"%+.0f W that was standing on the wire.", shadow.WIRE_W)
|
||||
if not engaged:
|
||||
self.shadow_would_w = None
|
||||
self._shadow_winner = None
|
||||
return
|
||||
self.shadow_would_w = float(resolution.target_w)
|
||||
# ⚠️ THE WINNER AND THE ROUNDED WATTS, NOT THE WATTS ALONE. A decision
|
||||
# that moves from the loop to the peak guard at the same number is a
|
||||
# different decision and the operator needs the line; a loop target
|
||||
# that wanders by a watt is not, and 15 s of those would be the whole
|
||||
# log.
|
||||
edge = (resolution.winner, round(float(resolution.target_w)))
|
||||
if edge != self._shadow_winner:
|
||||
self._shadow_winner = edge
|
||||
self.log_event(
|
||||
f"SHADOW: would have commanded {resolution.target_w:+.0f} W - "
|
||||
+ resolution.explain())
|
||||
self.target = shadow.WIRE_W
|
||||
for inv in self.inverters:
|
||||
inv.target = shadow.WIRE_W
|
||||
|
||||
async def check_output_clamp(self) -> None:
|
||||
"""SAFETY-03. Alarm whenever a clamp actually engaged this cycle.
|
||||
|
||||
@@ -4016,6 +4231,14 @@ class Controller:
|
||||
or (self.site_sign_wd.tripped
|
||||
and len(self.inverters) > 1)
|
||||
or self.circ_wd.tripped
|
||||
# SAFETY-10: "shadow" is its own value and not a
|
||||
# flavour of "stopped". A fleet dashboard reporting a
|
||||
# shadow install as `running` would be calling a
|
||||
# battery nobody is controlling healthy; one reporting
|
||||
# it as `stopped` would lose the fact that the decision
|
||||
# path is live and being recorded. ⚠️ BELOW `failsafe`:
|
||||
# a latch that is up is up, and shadow does not clear it.
|
||||
else "shadow" if self.shadow
|
||||
else "running" if self.auto else "stopped"),
|
||||
# SAFETY-31. ⚠️ MEASURED, not the allocation record - see
|
||||
# `site_circulating_w`. `publish()` drops None values, so a cycle
|
||||
@@ -4516,7 +4739,24 @@ class Controller:
|
||||
def status_payload(self) -> dict:
|
||||
checks = self.checks()
|
||||
bad = [c for c in checks if not c["ok"] and not c["warn"]]
|
||||
if self.circ_wd.tripped:
|
||||
# --- SAFETY-10, AND IT IS FIRST IN THE CASCADE -----------------------
|
||||
# "The battery is NOT being controlled" is the one sentence on this
|
||||
# page that may never be displaced by another condition. A banner that
|
||||
# can be out-ranked does not STATE a thing plainly; it states it
|
||||
# whenever nothing else is happening, which is the half of the time
|
||||
# nobody needs telling.
|
||||
#
|
||||
# ⚠️ AND IT CARRIES A STANDING LATCH RATHER THAN HIDING ONE. The three
|
||||
# `bad` branches below all say some version of "held at 0 W", which is
|
||||
# still true in shadow mode - but an operator who turns shadow mode off
|
||||
# with a latch up gets a battery that will not move and a page that
|
||||
# never mentioned it. `shadow.banner()` names the latch and goes red
|
||||
# for it, so nothing this cascade would have said is lost.
|
||||
_shadow = shadow.banner(self.o, control_started=self.auto,
|
||||
latched=self.shadow_latches())
|
||||
if _shadow is not None:
|
||||
level, banner = _shadow
|
||||
elif self.circ_wd.tripped:
|
||||
# ⚠️ ABOVE THE SIGN FAILSAFE, and the order is a decision. Both are
|
||||
# latched P_SAFETY stops, so the site is at 0 W either way and the
|
||||
# banner can only carry one sentence. This one goes first because it
|
||||
@@ -4616,6 +4856,16 @@ class Controller:
|
||||
("State of charge", "—" if self.pack_floor_soc is None
|
||||
else f"{self.pack_floor_soc:g} %"),
|
||||
("Commanded", w(self.target)),
|
||||
# SAFETY-10, and it sits BETWEEN the two rows it reconciles.
|
||||
# "Commanded 0 W" above and a "Why" below that explains a trade is
|
||||
# the exact pair an operator would otherwise have to guess at; this
|
||||
# row is what makes the two of them consistent, and it is rendered
|
||||
# in EVERY state - including `off`, where it says the battery IS
|
||||
# being controlled - because a surface whose ABSENCE is how you
|
||||
# learn something is a surface you have to already know about.
|
||||
("Shadow mode", shadow.words(
|
||||
self.o, control_started=self.auto,
|
||||
would_w=self.shadow_would_w, reason=self.reason)),
|
||||
("Why", self.reason),
|
||||
("Maintenance phase", self.maint.phase if len(self.inverters) <= 1
|
||||
else f"{self.maint.phase} ({self.maint_unit.name})"),
|
||||
|
||||
Reference in New Issue
Block a user