# SAFETY-10 — shadow mode, and why "default-on" is not `true` **Status:** decided and implemented on branch `SAFETY-10`. ⚠️ **AMENDED BY SAFETY-47, and three of the claims below were false when this was merged.** §5's "the strategies are unaffected" exemption, §4's peak-guard row and §6's evidence table are corrected in place and each correction is marked. The amendments are marked rather than rewritten silently, because what this card believed is part of why the defect shipped. See `DECISIONS/SAFETY-47-what-a-shadow-run-can-show.md`. **Card:** SAFETY-10 — *"Shadow mode (default-on, log-only decisions)"*. **Nothing here reached the inverter.** No deployment, no `ha addons update`, no contact with 192.168.2.158 or 192.168.2.221. Everything below was measured in the CI image and on TEST-23 stage 3's closed-loop rig. --- ## 0. The four things a reader should take away 1. **"Default-on" is delivered as `auto`, not as `true`, and the difference is an install that is regulating a real battery today.** The Supervisor composes an add-on's options as *its defaults, overlaid with the stored user options*, so a new key with a new default reaches **every existing install** on the next update. A boolean `shadow_mode: true` would silently stop the reference site. §1. 2. **Shadow mode WRITES. It writes 0 W, on the ordinary heartbeat, through the ordinary single writer.** "Commands nothing" implemented as "stops writing" would leave the inverter latched on whatever was standing — it has no meter timeout — and would put the ESP32 into a tripped stale-input watchdog for the whole run. §2. 3. **The sign watchdogs are inert, and that was the decision that had to be got right.** Fed the shadow decision they would trip within `sign_disagree_s` and **latch a trip that survives into live operation and a restart**. §4. 4. **The loop leg's shadow decision is one slew step, not a trajectory, and that is a real limitation of this feature.** §5. 5. ⚠️ **SAFETY-47: and neither is anything else that depends on the plant answering back. A shadow run cannot surface a closed-loop defect at all**, by the same construction that makes it safe — the battery is held at 0 W, so every measured input describes a plant that did not respond. Of the three defects §"WHAT DEFECT THIS PREVENTS" was written for, exactly one — TARIFF-14's wrong quarter-hour — is visible from a shadow run. §5, §6. --- ## 1. The default ### The literal reading, and what it costs `config.yaml`'s `options:` block **is** the default. There is exactly one of it, and HA's Supervisor resolves an add-on's effective options as ``` {**config.yaml options, **the stored user options} ``` A key that has never been stored therefore takes the file's value — on a fresh install and on an install that has been running for months, identically. There is **no mechanism in the add-on options system** by which one key can default differently for the two. So the card's title, taken at face value, is: > the next `ha addons update` stops the battery at the reference site, and the > only symptom is a battery that has stopped doing anything. That is strictly worse than every defect shadow mode exists to catch. It is refused. ### What was shipped instead `shadow_mode` is a three-valued `list(auto|on|off)` defaulting to `"auto"`: | value | resolves to | |---|---| | `off` | live | | `on` | shadow, always | | `auto` | shadow **while control is stopped**; live once it is started | `auto` reads `Controller.auto` — the persisted start/stop state, which a fresh install takes from `auto_start` and which the *Start / stop control* button moves. No new persisted state, no migration, no guess: * a **fresh** install ships `auto_start: false`, has no stored `auto` key, and therefore comes up **in shadow mode**. It observes a day of decisions before it is ever allowed to command anything. That is the card's intent, delivered where it is safe; * an install that is **controlling** is started, so `auto` resolves to live and **not one watt changes**; * an install that is **stopped** keeps its actuator at 0 W exactly as before — the same number, from the same writer, on the same heartbeat. What changes is what the status page and the action log **say**, and that the decision path now runs and is recorded. ⚠️ **Under `auto`, pressing Start leaves shadow mode.** That is the point of `auto`: the button means "control the battery". An operator who wants the decisions recorded *while the loop runs* — the configuration this card exists to make possible — sets `on` explicitly, and cannot reach it by accident. ⚠️ **An unrecognised value resolves to SHADOW, and is REFUSED at startup.** `shadow.mode()` fails toward inaction; `validate.py` rule 11 refuses the value by name so the reinterpretation is never silent. A typo is not a decision, and an install silently left in shadow mode commands nothing indefinitely while its status page calmly explains that this is intentional. --- ## 2. "Commands nothing" is a number, not a silence The inverter holds its last command **forever** — `HANDOVER.md` §6 says so, and `firmware/goodwe-master.yaml`'s header records what it cost when a controller stopped refreshing: *"Illusmart went silent mid-command and the inverter held +5000 W for 113 s"*. `firmware/goodwe-master.yaml` states the contract that follows — *"HA must now REWRITE the setpoint every ~10 s even when the value has not changed"* — because without a refresh, **"stale" and "steady" are the same thing on the wire**. So shadow mode holds the actuator at `shadow.WIRE_W` = **0.0 W**, refreshed on the ordinary heartbeat. Two consequences were weighed and are recorded: * **it is not silence.** A shadow mode that stopped writing would leave the standing command up until the ESP32's 30 s watchdog noticed — and at a site with no such board, forever; * **it does not use an alarm as a mode.** Letting that watchdog trip for the duration of a shadow run teaches the operator that `wd_tripped` is normal, which is exactly the training that makes it useless when the add-on has actually died. ### It is not a second writer SAFETY-28's rule is one actuator, one writer. Shadow mode files **no claim**, owns **no transport**, and calls nothing in `hass.py`. It is a **substitution** inside `Controller._write_unit()` — the single choke point every write in this add-on already reaches, including the heartbeat, the stop button, `shutdown()` and `maint_abort()` — in exactly the form of the three substitutions above it (non-finite → 0 W, unauthorised → 0 W, clamped → the rating). It is the **last** of them. Below SAFETY-28's gate and below SAFETY-03's clamp, so both still grade the number the decision path actually produced, and a shadow run can report *"it would have hit the rating here"*. --- ## 3. What the run records | surface | what it carries | |---|---| | status banner | `SHADOW MODE - the battery is NOT being controlled...`, **first in the cascade**, amber — red, with the latch named, if one is standing | | status row `Shadow mode` | which rung decided, the watts it **would have commanded on this cycle**, where the inverter actually is, and `Resolution.explain()` — plus, when a measured billed quarter sized those watts, SAFETY-47's caveat saying so (`shadow.SIZED_FROM_METER`) | | status row `Commanded` | `0 W` — the truth, which the row above reconciles with the `Why` row below it | | action log | one line when shadow mode comes on; one line per **change of decision** (winner, rounded watts, or SAFETY-47's caveat state), carrying the arbiter's own sentence and the same caveat the row carries — `shadow.log_line()` is the one sentence both read | | MQTT | `status: shadow`, its own value, below `failsafe` | The action log is **edge**-logged, not level-logged: the log holds 40 rows, and a line per cycle would flush the context of the run's own findings inside four minutes at the 15 s cycle floor. --- ## 4. The failsafes, one by one | failsafe | in shadow mode | why | |---|---|---| | SAFETY-02/30 sign watchdogs | **INERT** | They grade `Inverter.standing_w()` — what this controller has standing on the wire — and shadow mode holds that at 0 W, so they cannot arm. ⚠️ Fed the shadow decision instead they would disagree with a meter that never saw it and **latch a trip that survives into live operation and a restart**: shadow mode would manufacture the failsafe it exists to help debug. ⚠️ **Two rungs hold it there** — `last_write` (the wire gate) and `inv.target` (the re-seed) — which is MUTATION.md's masking shape, so the mutation that reproduces the hazard defeats both as a pair (M6). Driven singly, dropping the re-seed does not reproduce it at all — measured | | SAFETY-03 clamp | **OBSERVES** | Runs on the decision, sets `clamp_last`, logs every engagement. Changes no command | | SAFETY-28 write gate | **OBSERVES** | Same placement, same reason: the question is who chose the number | | SAFETY-18 meter freeze | **OBSERVES**, unchanged | It takes no action live either; both halves of its conjunction are as true in shadow | | SAFETY-07/24/25 peak guard | **OBSERVES, AND SEES ONLY THE HOUSE** | Level-triggered, unlatched, files its claim as usual, so the recorded decision is the real one *for this meter*. ⚠️ **AMENDED BY SAFETY-47.** This row used to end "SAFETY-45's limit cycle is visible from a shadow run, which is the point". It is not: every term of `wanted = batt_w + excess` is measured, and in shadow mode the battery contributes nothing to either. Measured on the rig, same hour, same unsized code — **410 of 720 cycles engaged live, 0 of 720 in shadow** | | SAFETY-31 circulation | **OBSERVES AND MAY LATCH** | Its evidence is *measured* power. Packs pushing against each other while all are told 0 W is a real fault | | SAFETY-42 delivery | **OBSERVES AND MAY LATCH** | Same, against a share that is 0 W | | operator stop | **SUPPRESSED AS A CLAIM, ENFORCED AT THE WIRE** | The one deliberate change. Its `P_SAFETY` 0 W claim out-ranks everything, so leaving it in makes the recorded decision `"stopped"` on every cycle and the run answers the wrong question. The actuator ends at the same number either way, and the wire substitution is **strictly stronger**: a claim can be out-ranked, and it cannot | | inputs missing, non-finite, allocator refusal, arbiter contradiction | **UNCHANGED** | All resolve to 0 W, which is what the wire gets anyway | ⚠️ **A latch that is already up stays up.** Nothing here clears one. The shadow banner **names** any standing latch and goes red for it, because a latch that is up while the controller is inert is up the moment it stops being inert. ⚠️ **Maintenance is not in that table because shadow mode does not touch it.** The state machines are ticked on every cycle whether control is started or not, so a stopped install already runs them against a battery that does not move. Shadow mode inherits that, unchanged and unimproved. Recorded here rather than quietly fixed under another card's name. --- ## 5. The limitation, stated rather than discovered later `Controller._shadow_settle()` re-seeds the carried command (`self.target` and each `inv.target`) to `WIRE_W` at the **end** of every shadow cycle — after every in-cycle guard has read the decision. Three machines read that value as a statement about the **hardware**: the loop's `prev_w` and integrator, `standing_w()`, and SAFETY-42's share comparison. Re-seeded, they track reality, and **the exit is safe by construction** — the carried command already equals the standing one, so the first live cycle slews from 0 W like any other. ⚠️ **The cost is real.** The loop's `prev_w` is therefore 0 W on every shadow cycle, so the loop leg's recorded decision is **one slew step from the standing command** and stays there. A shadow run answers *"what would this controller command NOW, from where the plant really is"* — a true statement about the present — and **not** *"what trajectory would it have followed"*. The trajectory is a closed-loop property, it did not happen, and reporting it would be a simulation of a plant that does not exist; three cycles in, such a number describes nothing. ⚠️ **AMENDED BY SAFETY-47 — THE PARAGRAPH THAT STOOD HERE WAS THE DEFECT.** It read: > The strategies are unaffected: arbitrage, the peak guard and maintenance all > file `SET` claims at a fixed power from level-triggered inputs. That is why > the three defects this card was written for — SAFETY-45's 28 onsets an hour, > TARIFF-14's wrong quarter-hour, SAFETY-46's ceiling churn — are all visible > from a shadow run, and the loop's ramp is not. It was true until **SAFETY-45 merged, one commit before this card**. That change makes the buy leg `min(asked, headroom)` with ``` headroom = (900 / remaining) x (ceiling - projected) ``` and `projected` reads the **open quarter's accumulator** — the meter — which is a function of what the battery did earlier in that same quarter. The strategy is closed-loop, and the loop's one-slew-step limitation above is not a special case of anything: **it is the general case**, and the limitation is the whole feature's, not the loop leg's. **The correct statement, and it is now graded rather than asserted:** > A shadow run cannot surface a **closed-loop** defect — one whose stimulus is > the plant's response to the controller — because the battery is held at 0 W > and there is no response to be the stimulus. **Which** hour a strategy picks > and **why** are unaffected, because the clock and the price series are not > functions of the battery. **How hard** it trades is not. Of the three defects this card was written for: | | visible from a shadow run? | measured | |---|---|---| | SAFETY-45's limit cycle | **no** | 28 charge onsets live, **1** in shadow; grid −455.5 … +3200.0 W live, **flat at the 1200 W house** in shadow; guard engaged 410/720 live, **0/720** in shadow | | SAFETY-46's SoC churn | **no** | the SoC moves **0.000000000 pp** in a shadow run. No SoC limit can be *reached* in one | | TARIFF-14's wrong quarter-hour | **yes**, and it is the one | which hour, at what price, clearing what spread: identical live and shadow on **720 of 720** charging cycles, at a binding cap | Graded in `test_arbitrage.py` part **3d** — the same closed loop part 3c uses, run once live and once in shadow — and the loop leg's own limitation stays named in `test_watchdog.py` part 10 as `RECORDED LIMITATION`. --- ## 6. How "nothing reaches the inverter" is graded **Not on a flag.** A check that asserted `Controller.shadow` would pass on a build in which the flag is set and the substitution is deleted — which is exactly mutation **M1**. `test_clamp.py` part 8 drives the **shipped** `Controller.cycle()` and the shipped heartbeat against a recording Home Assistant stub, and asserts on what reached `set_number()`: * **the precondition first** — the same stimulus with `shadow_mode: off` puts 30 non-zero writes on the wire, the largest 5000 W. MUTATION.md habit 4: a check whose data cannot reach the failure it names is decoration; * then `set(writes) == {0.0}`, **and** that there were 26 of them — a check that only said "no non-zero write" would pass on the silent build too; * entry and exit are measured on `inv.last_write` — what the inverter is latched on, updated only on an **accepted** write — either side of the boundary, against the same run that never entered shadow mode. The closed-loop half is `/home/homelab/s10-mut/shadowloop.py` on TEST-23 stage 3's rig: the same hour, twice, against `plantsim.Plant` through the firmware's own frame encoder. ⚠️ **SUPERSEDED BY SAFETY-47 — THE TABLE THAT STOOD HERE COULD NEVER HAVE GONE RED, FOR TWO INDEPENDENT REASONS.** It read `arb_power_w` −2000 W *"identical"*, `arb_reason` *"identical, verbatim"*, arbiter winner *"identical"*, and concluded *"the decision happened, at the same instant, for the same reason, at the same signed power — and the battery did not move."* The battery half is true and is unchanged. The **decision** half was decoration: 1. **the fixture is `peak_cap_w: 15000` against a 350 W house**, so the capacity guard never engages and SAFETY-45's sizing is in the run and inert. The harness's own comment says so — *"at this scenario's `peak_cap_w: 15000` it is not binding anyway"* — which is MUTATION.md habit 4 written down beside the check it disables; 2. **every identity row reads the FIRST charging sample.** At `elapsed = 0` the projection *is* the held level, so the two runs agree to the float there **even at a cap that does bind**. Measured: closing reason (1) alone leaves that comparison passing. Both had to be closed. Re-taken below at **`peak_cap_w` 1500 / house 1200 / `max_w` 2000**, one hour at a 5 s cycle, **across the whole run** — harness `/home/homelab/s47-mut/shadowloop47.py`, same rig, same three transcribed lines. | | live | shadow | |---|---|---| | grid | 1224.1 … 1506.0 W — **held up at the ceiling** | **1200.0 W flat = the house, exactly** | | SoC | 50.000019 → **51.856789 %** | 50.000000 → **50.000000 %** | | `ARBITRAGE_CHARGE` cycles | 720 / 720 | **720 / 720** | | first charging cycle, `arb_power_w` | −300.0000 W | **identical — and this is why the old table passed** | | `arb_power_w`, **whole run** | −307.799 … −299.930 W | **−2000.000 … −300.000 W** | | …cycles where the two differ | — | **716 of 720** | | `arb_reason`, whole run | the strategy's own sentence | **differs on 719 of 720** | | measured charging headroom | 299.93 … 307.80 W | **300.0 … 54 000.0 W** | | final recorded command | **−302.4 W** | **−2000.0 W — 6.61× the live figure** | | price / spread / which hour | | **identical on 720 of 720** | | arbiter winner | `arbitrage` | **identical** | **What the re-taken table says.** The decision still *happens*, at the same instant, on the same hour, at the same price, for the same *kind* of reason, and the battery still does not move — those were the card's real claims and they hold. What does **not** hold is that the decision is the *same decision*: its **size** is a function of a meter the battery never moved, and at this cap the recorded figure is 6.61× what a live controller commands. 54 000 W of "headroom" under a 1500 W ceiling is `900/remaining` multiplying a gap the battery never closed. ⚠️ **And the SoC row is the second half of it.** 0.000000000 pp of movement is not a curiosity: it is the proof that **no SoC-limit behaviour is reachable from a shadow run at all**, which §5 used to promise. SAFETY-47 puts the caveat on both surfaces that carry the number and narrows the three documents' claims; `test_arbitrage.py` part 3d is the CI-side grader so this class cannot recur. ⚠️ **Three lines of each harness are a TRANSCRIPTION and are marked as one.** `priceloop.Rig` has no `_write_unit()`, no clamp and no failsafe machine (divergence 2 of its README), so the substitution's **placement** is transcribed there. The shipped placement is graded by `test_clamp.py` part 8, which drives the real `Controller`. ⚠️ **And the rig's transcribed cycle predates SAFETY-45**: `arbitrage.decide()` gained a required `charge_headroom_w` after `478ba93`, and the harness supplies it from the **shipped** `peakguard.charge_headroom_w()` reading the **shipped** peak decision rather than from a constant, so SAFETY-45 stays in the run. --- ## 7. The mutation battery — 22 mutations, no survivors Harness: `/home/homelab/s10-mut/run.py`, backups beside it. Each run restores from backup, purges `__pycache__` **on the host**, and grades through the CI image with the tree bind-mounted `:ro`, so the interpreter cannot write a cache into the tree under test (TEST-15). The full table is in the branch report; one per new behaviour, including **M1** (the substitution deleted — a write escapes), **M2** (it returns instead of substituting — the inverter is left latched), **M3** (it moves above SAFETY-28 and SAFETY-03 — the observers go blind), **M8** (`auto` resolves to shadow unconditionally — the literal default-on that stops the live install), **M12** (the stop claim is filed in shadow — every recorded decision reads `stopped`), **M21** (the substitution and the re-seed together) and **M22** (the re-seed moved one line earlier — SAFETY-18 grades a 0 W no decision produced and goes permanently blind). ⚠️ **Two findings from driving it, both of the kind MUTATION.md says to expect.** * **M2 first came back a SURVIVOR and was not one.** The "stop writing altogether" build left `inv.last_write` as `None`, the check did `float()` on it, and the section died with a `TypeError` — grading none of the checks after it and reporting **zero reds**, which reads exactly like a survivor. That is FCST-06's shape. The fixture now hands `None` back and names it as its own check; M2 is red on three. * **M6 had to become a PAIR, and only the battery said so.** `standing_w()` prefers `last_write` — the 0.0 W the wire gate accepted — and falls back to `inv.target`, which the re-seed also holds at 0.0. Either rung alone keeps the detector inert, so dropping the re-seed **singly** came back red on the limitation check and on **nothing about the latch**: the hazard it names was not reproduced. TEST-16's rule with a new name on it. * **M7/M8/M9/M19 had to MOVE SUITES.** They defeat `shadow.engaged()`, and a defeated default stops `test_clamp.py` part 3's wired run reaching the wire at all — aborting the file with an `IndexError` before part 8 is graded. The red count there was a lower bound naming the saturation freeze. The mode table now lives in `test_validate.py`, which drives no Controller. * **`shadow_would_w = float(self.target)` is an EQUIVALENT mutation and was measured as one.** `_shadow_settle()` runs before the re-seed, so at that line `self.target` **is** `resolution.target_w`. It is recorded as equivalent rather than reported as a survivor, and the non-equivalent form of the same defect — the record becoming the **wire** — is driven instead. ### The table Each row: the mutation, the file, the byte delta, the suite it was graded in, and how many **named** checks went red. `⚠️` marks a row whose run did not reach the end of its file, so its count is a **lower bound** (MUTATION.md). | # | mutation | file | Δ | suite | reds | |---|---|---|---|---|---| | M1 | **a write escapes** — the substitution deleted | `main.py` | −6 B | clamp | 5 | | M2 | **silence instead of 0 W** — it returns rather than substituting | `main.py` | −15 B | clamp | 3 | | M3 | **the observers go blind** — it moves above SAFETY-28 and SAFETY-03 | `main.py` | +58 B | clamp | 1 | | M4 | the site re-seed dropped — the exit resumes from a fiction | `main.py` | −36 B | clamp | 3 | | M5 | the per-unit re-seed dropped | `main.py` | −22 B | clamp | 1 | | M6 | **PAIRED**: `standing_w()` reads the target **and** the re-seed dropped — the sign latch that survives into live operation | `inverter.py` + `main.py` | −170 B | watchdog | 3 | | M7 | `auto` inverted — shadow while started, live while stopped | `shadow.py` | +2 B | validate | 4 | | M8 | **default-on for everyone** — `auto` always shadow; the literal reading that stops the live install | `shadow.py` | −15 B | validate | 2 | | M9 | an unreadable value resolves to LIVE instead of shadow | `shadow.py` | +1 B | validate | 1 | | M10 | `validate.py` stops refusing an unknown value | `shadow.py` | −9 B | validate | 9 | | M11 | the loop leg gated on `auto` alone again — the decision path stops running | `main.py` | −17 B | clamp | 1 | | M12 | the stop claim filed in shadow — every decision reads `stopped` | `main.py` | −20 B | clamp | 1 | | M13 | the banner goes green while nothing is commanded | `shadow.py` | −2 B | clamp | 1 | | M14 | the banner loses the standing latch | `shadow.py` | −2 B | clamp | 1 | | M15 | the row renders the standing command instead of the decision | `shadow.py` | −8 B | clamp | 1 | | M16 ⚠️ | the row disappears when live | `main.py` | +43 B | clamp | 1 | | M17 | the record becomes the wire | `main.py` | −6 B | clamp | 7 | | M18 | level-logged — the 40-row action log floods | `main.py` | −23 B | clamp | 1 | | M19 | the two rungs render identically (masking) | `shadow.py` | +22 B | validate | 1 | | M20 | the shipped default moves to `off` in `config.yaml` alone | `config.yaml` | −1 B | defaults_pin | 1 | | M21 | **PAIRED**: the substitution **and** the re-seed deleted together | `main.py` | −103 B | clamp | 10 | | M22 | the re-seed moved one line earlier — SAFETY-18 grades a 0 W no decision produced and goes permanently blind | `main.py` | +40 B | clamp | 1 | **Survivors: none.** ⚠️ **M17's `+0 B`-adjacent sibling** — `shadow_would_w = float(self.target)` — is recorded in §7 as **equivalent** rather than counted here.