"""SAFETY-10 - shadow mode: the whole decision path, and no command. WHAT DEFECT THIS PREVENTS ========================= Not one defect - the CLASS of them. Every optimiser fault this repository has found was invisible from outside while it was happening, and each one was paid for at the meter before anybody could see it: * SAFETY-45 arbitrage against a binding `peak_cap_w` ran 28 charge onsets an hour, the grid swinging -455 .. +3200 W, and put a billed quarter 59 W over the cap; * TARIFF-14 the price lookup answered for the wrong quarter-hour 45 minutes in every 60, and traded on it; * SAFETY-46 the SoC ceiling churns 598 Wh to gain 310 Wh, no hysteresis. There was no way to WATCH what the optimiser would do without letting it do it. A commissioning engineer could arm arbitrage and find out, or not arm it and learn nothing. Shadow mode is the third option: strategies, arbiter, clamps and guards all run, the decision is recorded with its reason, and NOTHING THE CONTROLLER DECIDES REACHES THE INVERTER. ⚠️ AND OF THOSE THREE, A SHADOW RUN CAN SHOW YOU EXACTLY ONE - SEE THE NEXT SECTION. This header used to name SAFETY-45's limit cycle as the example of what shadow mode surfaces. That was false, and SAFETY-47 measured it. ⚠️ WHAT A SHADOW RUN CAN AND CANNOT SHOW (SAFETY-47) ===================================================== SHADOW MODE CANNOT SURFACE A CLOSED-LOOP DEFECT. Not badly, not partially - NOT AT ALL, and by the same construction that makes it safe. The actuator is held at WIRE_W and the carried command is re-seeded to WIRE_W, so every MEASURED input - the meter, the quarter accumulator behind it, the battery power, the state of charge - describes a plant that DID NOT RESPOND. A closed-loop defect is by definition one whose stimulus is the plant's response to the controller. No response, no stimulus, no defect to see. That is not a gap to be closed later. Feeding the guards a simulated plant instead is what SAFETY-10 §4 refuses for the sign watchdogs, for a reason that applies to all of this: a controller reacting to a plant that does not exist drifts further from reality every tick, and three cycles in its numbers describe nothing. THE CLOSED-LOOP ANSWER COMES FROM A CLOSED-LOOP RIG (TEST-23 stage 3) OR FROM A LIVE RUN. There is no third place. So, of the three defects above, measured on that rig at `peak_cap_w` 1500 / house 1200 / `max_w` 2000, one hour at a 5 s cycle: * SAFETY-45's limit cycle NOT VISIBLE. Live: 28 charge onsets, the grid swinging -455.5 .. +3200.0 W, the guard engaged on 410 of 720 cycles. Shadow, same hour, same code: 1 onset, the grid flat at the 1200 W house, the guard engaged on 0 of 720. Every term of `wanted = batt_w + excess` is measured. * SAFETY-46's SoC churn NOT VISIBLE. The SoC moves 0.000000000 pp in a shadow run, because it is the pack's integral of what the pack was commanded and the pack was commanded 0 W. No SoC limit can be REACHED in a shadow run; the ones already crossed when it started are reported, and that is all. * TARIFF-14's wrong quarter VISIBLE, and it is the one. The price lookup's inputs are the CLOCK and the PRICE SERIES, and neither is a function of what the battery did. Measured: which hour the strategy picked, its price and the spread it cleared are identical live and shadow on 720 of 720 charging cycles, AT A BINDING CAP. ⚠️ THE EXEMPTION THAT USED TO BE HERE WAS TRUE UNTIL SAFETY-45 MERGED. This module, DOCS.md and DECISIONS/SAFETY-10-shadow-mode.md all said "the strategies are unaffected - they ask for a fixed power from inputs that do not depend on what the battery did". SAFETY-45 made the buy leg `min(asked, headroom)` with headroom = (900 / remaining) x (ceiling - projected) and `projected` reads the OPEN QUARTER'S ACCUMULATOR - the meter - which is a function of what the battery did earlier in that same quarter. The strategy became closed-loop the day that landed. What survives of the exemption, and only this: WHICH hour a strategy picks, and WHY, are unaffected. HOW HARD it trades is not. ⚠️ AND THE SIZE A SHADOW RUN RECORDS IS THE RIGHT ANSWER TO A QUESTION NOBODY ASKED. `900/remaining` is correct arithmetic for a meter that is genuinely under the ceiling - and in shadow the meter genuinely IS under it, because the battery is not charging. So the number is not a bug; it is an answer computed for a plant that did not exist. Measured on the same rig, shipped code: cap / house live commanded shadow recorded over 1500 / 1200 -302.4 W -2000.0 W 6.61x 3500 / 3000 -502.6 W -2000.0 W 3.98x 3500 / 2000 -1502.6 W -2000.0 W 1.33x 3500 / 1800 -1702.6 W -2000.0 W 1.17x 15000 / 350 -2000.0 W -2000.0 W 1.00x The last row is the fixture SAFETY-10's own evidence was taken on, which is why nothing reported this. `words()` carries the caveat rather than a re-computed number, and `DECISIONS/SAFETY-47-what-a-shadow-run-can-show.md` §3 records the three alternatives that were rejected and why. ⚠️ "REACHES NOTHING" IS NOT "WRITES NOTHING", AND THE DIFFERENCE IS THE WHOLE SAFETY ARGUMENT ======================================================================= The inverter holds its last command FOREVER - it has no meter timeout (HANDOVER.md §6), and firmware/goodwe-master.yaml's header records what that cost when a controller stopped refreshing: Illusmart went silent mid-command and the inverter held +5000 W for 113 s. main.py's own header carries the same measurement. So a shadow mode that simply stopped writing would leave whatever was standing when it was switched on standing for as long as it was left on. That is the exact failure this repository keeps finding - LOOKS HEALTHY BUT IS INERT - with the status page calmly reporting that nothing is being commanded. So shadow mode does write, and it writes ONE number: the actuator is held at WIRE_W (0.0 W), refreshed on the ordinary heartbeat, by the ordinary single writer. That is a DEFINED, SAFE actuator state, it is the same state the operator's stop button reaches, and it honours the contract firmware/goodwe-master.yaml §"CONTRACT CHANGE" imposes on us - HA must rewrite the setpoint every ~10 s or the ESP32's 30 s stale-input watchdog cannot tell "steady" from "dead". Going silent would put that board into a tripped-watchdog state for the whole shadow run, which is an ALARM being used as a MODE: the operator who learns that `wd_tripped` is normal during shadow runs is the operator who ignores it when the add-on has actually died. ⚠️ AND IT IS NOT A SECOND WRITE PATH. SAFETY-28's rule is one actuator, one writer. Shadow mode files no claim, owns no transport and calls nothing in hass.py. It is a SUBSTITUTION inside `Controller._write_unit()` - the single choke point every write in this add-on already goes through - in exactly the form the three substitutions above it take (non-finite -> 0 W, unauthorised -> 0 W, clamped -> the rating). It is the LAST of them, so SAFETY-28's authorisation gate and SAFETY-03's clamp still grade the number the decision path actually produced, and a shadow run therefore records that the clamp WOULD have engaged. THE DEFAULT, AND WHY IT IS NOT A PLAIN `true` (§ the ticket's own warning) ========================================================================== The card is titled "default-on". Taken literally - `shadow_mode: true` in `config.yaml`'s `options:` block - it is the worst change in this repository's history. Home Assistant's Supervisor composes an add-on's effective options as `{**config.yaml defaults, **the stored user options}`, so a NEW key with a new default reaches EVERY install on the next update, including the one regulating a real battery in the author's house today. Literal default-on silently stops an existing install from controlling, and the only symptom is a battery that has stopped doing anything. There is no mechanism in the add-on options system that lets one key default differently for a new install and an existing one: `options:` IS the default, and there is exactly one of it. So "default-on for new" has to be expressed as a value the add-on RESOLVES, not as a constant the Supervisor merges. Hence three values rather than two, defaulting to `auto`: off never shadow. The controller commands the battery. on always shadow, whatever the start/stop button says. auto shadow WHILE CONTROL IS STOPPED; live once it is started. `auto` is default-on for a new install and a no-op for an existing one, and it is that without any persisted marker, any migration, and any guess: * a FRESH install ships `auto_start: false` and has no stored `auto` key, so it comes up stopped -> shadow. It observes a day of decisions before it is ever allowed to command anything. That is the card's intent, delivered. * an install that is CONTROLLING - `auto_start: true`, or the operator has pressed *Start / stop control* and the store carries `auto: true` - is started, so `auto` resolves to LIVE and its behaviour does not change by one watt. * an install that is STOPPED keeps its actuator at 0 W exactly as before - the same number, from the same writer, on the same heartbeat - and gains the recording. The only thing that changes for it is what the status page and the action log SAY. ⚠️ AND `auto` MEANS THE START BUTTON LEAVES SHADOW MODE. That is deliberate: under `auto` the button reads "Start / stop control" and pressing Start is the operator saying "control the battery". An operator who wants the decisions recorded WHILE the loop runs sets `shadow_mode: on` explicitly - that is the configuration this card exists to make possible, and it is the one that cannot be reached by accident. ⚠️ AN UNRECOGNISED VALUE RESOLVES TO SHADOW, NOT TO LIVE. `validate.py` refuses it at startup so this is not meant to be reachable, but the fail-safe direction for "the operator's intent is unreadable" is the one that does not move energy. THE FAILSAFES, ONE BY ONE (the card asks for each to be decided and recorded) ============================================================================ | failsafe | in shadow mode | |--------------------------------|-------------------------------------------| | SAFETY-02/30 sign watchdogs | INERT, and by construction rather than by | | (per-unit and site) | a special case. They grade | | | `Inverter.standing_w()` - what this | | | controller has standing on the wire - and | | | shadow mode holds that at 0 W. A 0 W | | | command cannot oppose a meter, so the | | | detector never arms. ⚠️ THIS IS THE ONE | | | THAT HAD TO BE GOT RIGHT: fed the SHADOW | | | decision instead, every shadow run would | | | latch a trip within `sign_disagree_s` - | | | the decision opposes a meter that never | | | saw it - and that latch SURVIVES INTO | | | LIVE OPERATION and a restart. Shadow mode | | | would manufacture the failsafe it is | | | meant to help debug. | | SAFETY-03 clamp | OBSERVES. It runs on the number the | | | decision path produced, `clamp_last` is | | | set and every engagement goes to the | | | action log, because "it would have hit | | | the rating" is part of what a shadow run | | | is for. It changes no command: the value | | | it narrows is replaced by 0 W below it. | | SAFETY-28 write authorisation | OBSERVES, for the same reason and in the | | | same place - above the substitution, so | | | it still asks who chose the number. | | SAFETY-18 meter freeze | OBSERVES, unchanged. It takes no action | | | in live operation either; it grades the | | | resolved intent against a meter that has | | | stopped moving, and both halves are as | | | true in shadow as out of it. | | SAFETY-07/24/25 peak guard | OBSERVES, AND SEES ONLY THE HOUSE. It is | | | level-triggered and unlatched and it files| | | its claim as usual, so the recorded | | | decision is the real one FOR THIS METER - | | | but every term of `wanted = batt_w + | | | excess` is measured, and in shadow mode | | | the battery contributes nothing to either.| | | ⚠️ SAFETY-47: this row used to claim | | | SAFETY-45's limit cycle was visible from a| | | shadow run. It is not - 410/720 engaged | | | cycles live against 0/720 in shadow, on | | | the same hour and the same code. | | SAFETY-31 circulation | OBSERVES AND MAY LATCH, legitimately. Its | | SAFETY-42 delivery | evidence is MEASURED power, not a | | | command: packs pushing against each other | | | while every one of them is told 0 W is a | | | real fault and a latch is the right | | | answer to it. | | operator stop (`auto` False) | SUPPRESSED AS A CLAIM, ENFORCED AT THE | | | WIRE. It is the one deliberate change. | | | Its `P_SAFETY` 0 W claim out-ranks | | | everything, so leaving it in would make | | | the recorded decision "stopped" on every | | | cycle - the shadow run would answer the | | | wrong question. The actuator state is | | | identical either way, and the wire | | | substitution is STRICTLY STRONGER than | | | the claim: a claim can be out-ranked, and | | | this cannot. | | inputs missing, non-finite, | UNCHANGED. All of them resolve to 0 W, | | allocator refusal, arbiter | which is what the wire gets in shadow | | contradiction | mode anyway, so they act redundantly and | | | are recorded as they always were. | ⚠️ MAINTENANCE IS NOT IN THAT TABLE BECAUSE SHADOW MODE DOES NOT TOUCH IT. The state machines are ticked on every cycle today whether control is started or not, so a stopped install already runs them against a battery that does not move. Shadow mode inherits that, unchanged and unimproved; it is a pre-existing property of `auto_start: false` and it is recorded here rather than quietly fixed under another card's name. LEAVING SHADOW MODE =================== `Controller.cycle()` re-seeds its carried command (`self.target` and each `inv.target`) to WIRE_W at the END of every shadow cycle, AFTER every in-cycle guard has read the decision. Two things follow, and both are safety properties: * the loop's `prev_w` and its integrator track WHAT IS ACTUALLY STANDING ON THE WIRE, so every shadow cycle answers "from where the plant really is, what would this controller do NOW" - a true statement about the present, rather than a simulation of a counterfactual plant that drifts further from reality every tick; * the exit is safe BY CONSTRUCTION. The carried command already equals the standing command, so the first live cycle slews from 0 W like any other cycle. Carrying the shadow decision instead would let the slew limiter depart from a base that never existed - up to a full `slew_w` step onto a battery sitting at zero. The entry is safe for the ordinary reason: the substitution happens on the very first write of the very first shadow cycle, and `last_write` was not 0 W, so that write is issued rather than skipped as unchanged. """ from __future__ import annotations # ⚠️ CFG-03's mirror. `test_defaults_pin.py` part 2 discovers every # module-level name ending in OPTION_DEFAULTS and asserts each value # reproduces config.yaml EXACTLY, so this table cannot drift from the shipped # default. Nothing in the add-on may carry a literal fallback for a key a # mirror answers for - part 3 scans for that - which is why `main.py` asks the # functions below and never reads `shadow_mode` itself. SHADOW_OPTION_DEFAULTS = { "shadow_mode": "auto", } #: The three values `shadow_mode` may take, in the order config.yaml's schema #: lists them. `validate.py` refuses anything else by name. MODES = ("auto", "on", "off") #: What stands on the wire while shadow mode is engaged. ⚠️ A NUMBER, NOT A #: SILENCE - see this module's header. It is 0.0 W for the same reason every #: other refusal in this add-on is 0.0 W: fail toward inaction, and keep #: commanding it so the inverter cannot be left latched on anything else. WIRE_W = 0.0 #: SAFETY-47. What the recorded watts mean when a MEASURED billed quarter sized #: them. #: #: ⚠️ "UPPER BOUND" AND NOT "SMALLER", AND THE DIFFERENCE WAS MEASURED RATHER #: THAN ARGUED. Charging only ADDS offtake, so a live meter's `projected` is #: never below a shadow meter's and the headroom is never above it: the shadow #: size is >= the live size, with EQUALITY at the start of every quarter, where #: `900/remaining` is 1.0 and neither plant has moved. Measured pairwise over #: 3600 charging cycles in five configurations - 0 cycles below, 730 exactly #: equal. A sentence reading "smaller" would have been false on every one of #: those 730. #: #: ⚠️ AND IT NAMES NO FACTOR. How far under the bound a live run lands is a #: property of the site - 6.61x at a 1500 W cap under a 1200 W house, 1.00x at #: a cap that does not bind - and a number invented for the row would be a #: second helping of the defect this constant exists to disclose. SIZED_FROM_METER = ( "⚠️ AND ITS SIZE CAME FROM THE BILLED-QUARTER METER, WHICH HAS ONLY THE " "HOUSE IN IT WHILE THE BATTERY IS HELD AT 0 W. A live run sizes the same " "trade against a meter the battery is moving, so this is an UPPER BOUND " "and not a prediction; how far under it a live run lands is a closed-loop " "question and a shadow run cannot answer it") def mode(opts: dict) -> str: """The configured `shadow_mode`, normalised to one of `MODES`. ⚠️ AN UNRECOGNISED VALUE ANSWERS `"on"`, NOT `"auto"` AND NOT `"off"`. `validate.py` refuses such a value at startup, so this branch is not meant to be reachable at all; it exists because the fail-safe direction for "the operator's intent cannot be read" is the one that does not move energy. A typo that silently resolved to `off` would put a battery under a controller whose configuration nobody could parse. """ raw = opts.get("shadow_mode", SHADOW_OPTION_DEFAULTS["shadow_mode"]) text = str("" if raw is None else raw).strip().lower() if text in MODES: return text return "on" def engaged(opts: dict, *, control_started: bool) -> bool: """Is the battery being left alone right now? `control_started` is `Controller.auto` - the persisted start/stop state, which a fresh install gets from `auto_start` and an operator moves with the *Start / stop control* button. See the header on why `auto` reads it. """ m = mode(opts) if m == "off": return False if m == "on": return True return not control_started def why(opts: dict, *, control_started: bool) -> str: """Which rung decided, in the words an operator can act on. ⚠️ THE REASON, NOT ONLY THE VERDICT - MUTATION.md's masking rule. `on` and `auto`-while-stopped produce the same boolean and want different operator actions: one is turned off by editing an option, the other by pressing a button. A surface that showed only "shadow" would make a mutation that swapped the two rungs invisible. """ # ⚠️ THE LIVE SENTENCES SAY "NOT IN SHADOW MODE", NOT "THE BATTERY IS # BEING CONTROLLED". The second is FALSE on a stopped install with # `shadow_mode: off` - nothing is regulating there either - and a row that # is false in one of the four live states is a row an operator learns to # discount. What is true in every live state is that what this add-on # decides reaches the inverter; whether control is RUNNING is the # `Stopped` banner's question and it is answered there. m = mode(opts) if m == "off": return ("off (shadow_mode: off) - NOT in shadow mode: what this " "add-on decides DOES reach the inverter") if m == "on": return ("on (shadow_mode: on) - decisions are recorded and nothing " "is commanded") if control_started: return ("off (shadow_mode: auto, control is started) - NOT in shadow " "mode: what this add-on decides DOES reach the inverter") return ("on (shadow_mode: auto, control is stopped) - decisions are " "recorded and nothing is commanded") def words(opts: dict, *, control_started: bool, would_w: float | None, sized_from_meter: bool, reason: str = "") -> str: """The status page's `Shadow mode` row. ⚠️ THE ROW IS RENDERED IN EVERY STATE, INCLUDING `off`. A surface whose ABSENCE is how you learn which mode you are in is a surface an operator has to already know about; TARIFF-11's rule is that the rendered text is what gets graded, so there has to be text in both states. `would_w` is the decision the path reached this cycle - what a live controller would have put on the wire GIVEN THIS METER AND A BATTERY AT 0 W - and `reason` is `Resolution.explain()`, the same sentence the `Why` row carries. Together they are the whole of "what would it have done, and why"; without them the row would say a shadow run happened and nothing about what it found. ⚠️ `sized_from_meter` HAS NO DEFAULT, AND THAT IS SAFETY-47's WHOLE POINT. It is True when a MEASURED billed-quarter headroom sized the number being recorded - `arbitrage.sized_from_meter()` answers it, and it is the one place the question is asked. A default of False here would be a caller that forgot the caveat rendering a row that reads exactly like a caller that decided it did not apply, which is the shape of defect this card is about. TEST-14's rule with a new name on it. ⚠️ AND THE PHRASE IS "ON THIS CYCLE", UNCONDITIONALLY, NOT ONLY WHEN THE CAVEAT FIRES. Every number a shadow run records was reached from a plant that did not respond; "WOULD HAVE COMMANDED -2 000 W" reads as a forecast of a live run and no number here is one. The ⚠️ below is the STRONGER, conditional statement about the cycles where the answer can be multiples out - see this module's header for the measured table. ⚠️ AND IT IS NOT KEYED ON WHETHER THE TRADE WAS CAPPED. `arbitrage`'s own sentence says "SIZED BY THE CAPACITY-TARIFF GUARD" only when the headroom came out BELOW the ask - which in a shadow run is the START of the quarter, where the two runs still agree. By the end of the quarter `900/remaining` has multiplied the headroom past the ask, the trade is NOT capped, the sentence is silent and the recorded number is at its most wrong. A caveat keyed on that sentence would be quietest exactly where it is needed loudest. `test_arbitrage.py` part 3d-8 drives that. """ head = why(opts, control_started=control_started) if not engaged(opts, control_started=control_started): return head if would_w is None: return (f"{head}; no decision has been reached yet - the inverter is " f"held at {WIRE_W:.0f} W") # ⚠️ THE THOUSANDS SEPARATOR IS SWAPPED ON THE NUMBER ALONE. Applied to # the whole sentence it ate the comma out of `head` - and `head` is the # half that says WHICH RUNG decided, which is the claim MUTATION.md's # masking rule says must survive. num = f"{float(would_w):+,.0f}".replace(",", " ") out = (f"{head}; ON THIS CYCLE WOULD HAVE COMMANDED {num} W, inverter " f"held at {WIRE_W:.0f} W") if sized_from_meter: out += f" {SIZED_FROM_METER}" if reason: out += f" - {reason}" return out def log_line(*, would_w: float, reason: str, sized_from_meter: bool) -> str: """The action-log line for one change of shadow decision. ⚠️ IT IS A FUNCTION FOR THE REASON `words()` IS ONE, AND SAFETY-47 LEARNED IT THE EXPENSIVE WAY. This sentence used to be an f-string inside `Controller._shadow_settle()`. Deleting the caveat from it there left all 29 files GREEN - the only check that could see the line drove a fixture with `meter_source: off`, where no arbitrage decision and therefore no caveat can ever be reached. A sentence assembled at a call site is a sentence no check can grade, and the page would have said one thing while the record a later analysis reads said another. ⚠️ AND IT IS THE SAME `SIZED_FROM_METER` CONSTANT THE ROW CARRIES, not a second wording of it. Two sentences meaning one thing is how the two surfaces start disagreeing about which cycles are caveated. """ out = (f"SHADOW: would have commanded {float(would_w):+.0f} W ON THIS " f"CYCLE - {reason}") if sized_from_meter: out += f" {SIZED_FROM_METER}" return out def banner(opts: dict, *, control_started: bool, latched: tuple = ()) -> tuple: """`(level, text)` for the top of the status page, or `None` when live. ⚠️ IT NAMES A STANDING LATCH AND GOES RED FOR ONE. The shadow banner sits ABOVE every other branch of the cascade, because "the battery is not being controlled" is the one sentence that may never be displaced - but a failsafe latch that is up while the controller is inert is up when it stops being inert, and an operator who turns shadow mode off without clearing it gets a battery that will not move and a green page that never mentioned why. So the latch is carried INTO the shadow banner rather than losing its own. """ if not engaged(opts, control_started=control_started): return None text = ("SHADOW MODE - the battery is NOT being controlled. Every " f"inverter is held at {WIRE_W:.0f} W and nothing this add-on " "decides reaches the inverter.") if latched: return ("bad", text + " ⚠️ A FAILSAFE LATCH IS STILL UP (" + ", ".join(latched) + ") and it will still be up when shadow " "mode is turned off.") return ("warn", text) def problems(opts: dict) -> list: """`validate.py`'s rule: `shadow_mode` must be one of the three values. ⚠️ THE SENTENCE SPELLS `shadow_mode`. `configui.problems_by_field()` files a refusal under the field whose key its text contains; a rule that never names a key lands in the page's top-level block instead, where the operator cannot see which control is wrong. UX-02's whole point is that the shipped validator is the only rule surface, so the rule has to be readable there. ⚠️ AND IT IS A REFUSAL RATHER THAN A COERCION. `mode()` resolves an unreadable value to shadow, which is the safe direction but is also a SILENT reinterpretation of what the operator wrote - and an install left in shadow mode by a typo commands nothing at all, indefinitely. """ raw = opts.get("shadow_mode", SHADOW_OPTION_DEFAULTS["shadow_mode"]) text = str("" if raw is None else raw).strip().lower() if text in MODES: return [] return [f"shadow_mode is {raw!r}, which is not one of " + ", ".join(f"'{m}'" for m in MODES) + ". 'auto' leaves the battery alone while control is stopped and " "commands it once control is started; 'on' records every " "decision and commands nothing at all; 'off' is ordinary " "control."]