# SAFETY-10 — shadow mode, and why "default-on" is not `true` **Status:** decided and implemented on branch `SAFETY-10`. **Card:** SAFETY-10 — *"Shadow mode (default-on, log-only decisions)"*. **Nothing here reached the inverter.** No deployment, no `ha addons update`, no contact with 192.168.2.158 or 192.168.2.221. Everything below was measured in the CI image and on TEST-23 stage 3's closed-loop rig. --- ## 0. The four things a reader should take away 1. **"Default-on" is delivered as `auto`, not as `true`, and the difference is an install that is regulating a real battery today.** The Supervisor composes an add-on's options as *its defaults, overlaid with the stored user options*, so a new key with a new default reaches **every existing install** on the next update. A boolean `shadow_mode: true` would silently stop the reference site. §1. 2. **Shadow mode WRITES. It writes 0 W, on the ordinary heartbeat, through the ordinary single writer.** "Commands nothing" implemented as "stops writing" would leave the inverter latched on whatever was standing — it has no meter timeout — and would put the ESP32 into a tripped stale-input watchdog for the whole run. §2. 3. **The sign watchdogs are inert, and that was the decision that had to be got right.** Fed the shadow decision they would trip within `sign_disagree_s` and **latch a trip that survives into live operation and a restart**. §4. 4. **The loop leg's shadow decision is one slew step, not a trajectory, and that is a real limitation of this feature.** §5. --- ## 1. The default ### The literal reading, and what it costs `config.yaml`'s `options:` block **is** the default. There is exactly one of it, and HA's Supervisor resolves an add-on's effective options as ``` {**config.yaml options, **the stored user options} ``` A key that has never been stored therefore takes the file's value — on a fresh install and on an install that has been running for months, identically. There is **no mechanism in the add-on options system** by which one key can default differently for the two. So the card's title, taken at face value, is: > the next `ha addons update` stops the battery at the reference site, and the > only symptom is a battery that has stopped doing anything. That is strictly worse than every defect shadow mode exists to catch. It is refused. ### What was shipped instead `shadow_mode` is a three-valued `list(auto|on|off)` defaulting to `"auto"`: | value | resolves to | |---|---| | `off` | live | | `on` | shadow, always | | `auto` | shadow **while control is stopped**; live once it is started | `auto` reads `Controller.auto` — the persisted start/stop state, which a fresh install takes from `auto_start` and which the *Start / stop control* button moves. No new persisted state, no migration, no guess: * a **fresh** install ships `auto_start: false`, has no stored `auto` key, and therefore comes up **in shadow mode**. It observes a day of decisions before it is ever allowed to command anything. That is the card's intent, delivered where it is safe; * an install that is **controlling** is started, so `auto` resolves to live and **not one watt changes**; * an install that is **stopped** keeps its actuator at 0 W exactly as before — the same number, from the same writer, on the same heartbeat. What changes is what the status page and the action log **say**, and that the decision path now runs and is recorded. ⚠️ **Under `auto`, pressing Start leaves shadow mode.** That is the point of `auto`: the button means "control the battery". An operator who wants the decisions recorded *while the loop runs* — the configuration this card exists to make possible — sets `on` explicitly, and cannot reach it by accident. ⚠️ **An unrecognised value resolves to SHADOW, and is REFUSED at startup.** `shadow.mode()` fails toward inaction; `validate.py` rule 11 refuses the value by name so the reinterpretation is never silent. A typo is not a decision, and an install silently left in shadow mode commands nothing indefinitely while its status page calmly explains that this is intentional. --- ## 2. "Commands nothing" is a number, not a silence The inverter holds its last command **forever** — `HANDOVER.md` §6 says so, and `firmware/goodwe-master.yaml`'s header records what it cost when a controller stopped refreshing: *"Illusmart went silent mid-command and the inverter held +5000 W for 113 s"*. `firmware/goodwe-master.yaml` states the contract that follows — *"HA must now REWRITE the setpoint every ~10 s even when the value has not changed"* — because without a refresh, **"stale" and "steady" are the same thing on the wire**. So shadow mode holds the actuator at `shadow.WIRE_W` = **0.0 W**, refreshed on the ordinary heartbeat. Two consequences were weighed and are recorded: * **it is not silence.** A shadow mode that stopped writing would leave the standing command up until the ESP32's 30 s watchdog noticed — and at a site with no such board, forever; * **it does not use an alarm as a mode.** Letting that watchdog trip for the duration of a shadow run teaches the operator that `wd_tripped` is normal, which is exactly the training that makes it useless when the add-on has actually died. ### It is not a second writer SAFETY-28's rule is one actuator, one writer. Shadow mode files **no claim**, owns **no transport**, and calls nothing in `hass.py`. It is a **substitution** inside `Controller._write_unit()` — the single choke point every write in this add-on already reaches, including the heartbeat, the stop button, `shutdown()` and `maint_abort()` — in exactly the form of the three substitutions above it (non-finite → 0 W, unauthorised → 0 W, clamped → the rating). It is the **last** of them. Below SAFETY-28's gate and below SAFETY-03's clamp, so both still grade the number the decision path actually produced, and a shadow run can report *"it would have hit the rating here"*. --- ## 3. What the run records | surface | what it carries | |---|---| | status banner | `SHADOW MODE - the battery is NOT being controlled...`, **first in the cascade**, amber — red, with the latch named, if one is standing | | status row `Shadow mode` | which rung decided, the watts it **would have commanded**, where the inverter actually is, and `Resolution.explain()` | | status row `Commanded` | `0 W` — the truth, which the row above reconciles with the `Why` row below it | | action log | one line when shadow mode comes on; one line per **change of decision** (winner or rounded watts), carrying the arbiter's own sentence | | MQTT | `status: shadow`, its own value, below `failsafe` | The action log is **edge**-logged, not level-logged: the log holds 40 rows, and a line per cycle would flush the context of the run's own findings inside four minutes at the 15 s cycle floor. --- ## 4. The failsafes, one by one | failsafe | in shadow mode | why | |---|---|---| | SAFETY-02/30 sign watchdogs | **INERT** | They grade `Inverter.standing_w()` — what this controller has standing on the wire — and shadow mode holds that at 0 W, so they cannot arm. ⚠️ Fed the shadow decision instead they would disagree with a meter that never saw it and **latch a trip that survives into live operation and a restart**: shadow mode would manufacture the failsafe it exists to help debug. ⚠️ **Two rungs hold it there** — `last_write` (the wire gate) and `inv.target` (the re-seed) — which is MUTATION.md's masking shape, so the mutation that reproduces the hazard defeats both as a pair (M6). Driven singly, dropping the re-seed does not reproduce it at all — measured | | SAFETY-03 clamp | **OBSERVES** | Runs on the decision, sets `clamp_last`, logs every engagement. Changes no command | | SAFETY-28 write gate | **OBSERVES** | Same placement, same reason: the question is who chose the number | | SAFETY-18 meter freeze | **OBSERVES**, unchanged | It takes no action live either; both halves of its conjunction are as true in shadow | | SAFETY-07/24/25 peak guard | **OBSERVES** | Level-triggered, unlatched, files its claim as usual — SAFETY-45's limit cycle is visible from a shadow run, which is the point | | SAFETY-31 circulation | **OBSERVES AND MAY LATCH** | Its evidence is *measured* power. Packs pushing against each other while all are told 0 W is a real fault | | SAFETY-42 delivery | **OBSERVES AND MAY LATCH** | Same, against a share that is 0 W | | operator stop | **SUPPRESSED AS A CLAIM, ENFORCED AT THE WIRE** | The one deliberate change. Its `P_SAFETY` 0 W claim out-ranks everything, so leaving it in makes the recorded decision `"stopped"` on every cycle and the run answers the wrong question. The actuator ends at the same number either way, and the wire substitution is **strictly stronger**: a claim can be out-ranked, and it cannot | | inputs missing, non-finite, allocator refusal, arbiter contradiction | **UNCHANGED** | All resolve to 0 W, which is what the wire gets anyway | ⚠️ **A latch that is already up stays up.** Nothing here clears one. The shadow banner **names** any standing latch and goes red for it, because a latch that is up while the controller is inert is up the moment it stops being inert. ⚠️ **Maintenance is not in that table because shadow mode does not touch it.** The state machines are ticked on every cycle whether control is started or not, so a stopped install already runs them against a battery that does not move. Shadow mode inherits that, unchanged and unimproved. Recorded here rather than quietly fixed under another card's name. --- ## 5. The limitation, stated rather than discovered later `Controller._shadow_settle()` re-seeds the carried command (`self.target` and each `inv.target`) to `WIRE_W` at the **end** of every shadow cycle — after every in-cycle guard has read the decision. Three machines read that value as a statement about the **hardware**: the loop's `prev_w` and integrator, `standing_w()`, and SAFETY-42's share comparison. Re-seeded, they track reality, and **the exit is safe by construction** — the carried command already equals the standing one, so the first live cycle slews from 0 W like any other. ⚠️ **The cost is real.** The loop's `prev_w` is therefore 0 W on every shadow cycle, so the loop leg's recorded decision is **one slew step from the standing command** and stays there. A shadow run answers *"what would this controller command NOW, from where the plant really is"* — a true statement about the present — and **not** *"what trajectory would it have followed"*. The trajectory is a closed-loop property, it did not happen, and reporting it would be a simulation of a plant that does not exist; three cycles in, such a number describes nothing. The strategies are unaffected: arbitrage, the peak guard and maintenance all file `SET` claims at a fixed power from level-triggered inputs. That is why the three defects this card was written for — SAFETY-45's 28 onsets an hour, TARIFF-14's wrong quarter-hour, SAFETY-46's ceiling churn — are all visible from a shadow run, and the loop's ramp is not. Graded, named, in `test_watchdog.py` part 10 as `RECORDED LIMITATION`. --- ## 6. How "nothing reaches the inverter" is graded **Not on a flag.** A check that asserted `Controller.shadow` would pass on a build in which the flag is set and the substitution is deleted — which is exactly mutation **M1**. `test_clamp.py` part 8 drives the **shipped** `Controller.cycle()` and the shipped heartbeat against a recording Home Assistant stub, and asserts on what reached `set_number()`: * **the precondition first** — the same stimulus with `shadow_mode: off` puts 30 non-zero writes on the wire, the largest 5000 W. MUTATION.md habit 4: a check whose data cannot reach the failure it names is decoration; * then `set(writes) == {0.0}`, **and** that there were 26 of them — a check that only said "no non-zero write" would pass on the silent build too; * entry and exit are measured on `inv.last_write` — what the inverter is latched on, updated only on an **accepted** write — either side of the boundary, against the same run that never entered shadow mode. The closed-loop half is `/home/homelab/s10-mut/shadowloop.py` on TEST-23 stage 3's rig: the same hour, twice, against `plantsim.Plant` through the firmware's own frame encoder. | | live | shadow | |---|---|---| | `plant_cmd_w` | −2000 W throughout | **0 W throughout** | | `batt_w` | −2000 .. −40.6 W | **0.0 W, exactly** | | SoC | 50.000 → **62.331 %** | 50.0 → **50.0 %** | | grid | 390.6 .. 2350 W | **350 W = the house, exactly** | | `ARBITRAGE_CHARGE` samples | 3600 | **3600** | | first charge instant | 2026-01-15T00:00:00+00:00 | **identical** | | `arb_power_w` | −2000 W | **identical** | | `arb_reason` | the strategy's own sentence | **identical, verbatim** | | arbiter winner | `arbitrage` | **identical** | The decision happened, at the same instant, for the same reason, at the same signed power — and the battery did not move. ⚠️ **Three lines of that harness are a TRANSCRIPTION and are marked as one.** `priceloop.Rig` has no `_write_unit()`, no clamp and no failsafe machine (divergence 2 of its README), so the substitution's **placement** is transcribed there. The shipped placement is graded by `test_clamp.py` part 8, which drives the real `Controller`. ⚠️ **And the rig's transcribed cycle predates SAFETY-45**: `arbitrage.decide()` gained a required `charge_headroom_w` after `478ba93`, and the harness supplies it from the **shipped** `peakguard.charge_headroom_w()` reading the **shipped** peak decision rather than from a constant, so SAFETY-45 stays in the run. --- ## 7. The mutation battery — 22 mutations, no survivors Harness: `/home/homelab/s10-mut/run.py`, backups beside it. Each run restores from backup, purges `__pycache__` **on the host**, and grades through the CI image with the tree bind-mounted `:ro`, so the interpreter cannot write a cache into the tree under test (TEST-15). The full table is in the branch report; one per new behaviour, including **M1** (the substitution deleted — a write escapes), **M2** (it returns instead of substituting — the inverter is left latched), **M3** (it moves above SAFETY-28 and SAFETY-03 — the observers go blind), **M8** (`auto` resolves to shadow unconditionally — the literal default-on that stops the live install), **M12** (the stop claim is filed in shadow — every recorded decision reads `stopped`), **M21** (the substitution and the re-seed together) and **M22** (the re-seed moved one line earlier — SAFETY-18 grades a 0 W no decision produced and goes permanently blind). ⚠️ **Two findings from driving it, both of the kind MUTATION.md says to expect.** * **M2 first came back a SURVIVOR and was not one.** The "stop writing altogether" build left `inv.last_write` as `None`, the check did `float()` on it, and the section died with a `TypeError` — grading none of the checks after it and reporting **zero reds**, which reads exactly like a survivor. That is FCST-06's shape. The fixture now hands `None` back and names it as its own check; M2 is red on three. * **M6 had to become a PAIR, and only the battery said so.** `standing_w()` prefers `last_write` — the 0.0 W the wire gate accepted — and falls back to `inv.target`, which the re-seed also holds at 0.0. Either rung alone keeps the detector inert, so dropping the re-seed **singly** came back red on the limitation check and on **nothing about the latch**: the hazard it names was not reproduced. TEST-16's rule with a new name on it. * **M7/M8/M9/M19 had to MOVE SUITES.** They defeat `shadow.engaged()`, and a defeated default stops `test_clamp.py` part 3's wired run reaching the wire at all — aborting the file with an `IndexError` before part 8 is graded. The red count there was a lower bound naming the saturation freeze. The mode table now lives in `test_validate.py`, which drives no Controller. * **`shadow_would_w = float(self.target)` is an EQUIVALENT mutation and was measured as one.** `_shadow_settle()` runs before the re-seed, so at that line `self.target` **is** `resolution.target_w`. It is recorded as equivalent rather than reported as a survivor, and the non-equivalent form of the same defect — the record becoming the **wire** — is driven instead. ### The table Each row: the mutation, the file, the byte delta, the suite it was graded in, and how many **named** checks went red. `⚠️` marks a row whose run did not reach the end of its file, so its count is a **lower bound** (MUTATION.md). | # | mutation | file | Δ | suite | reds | |---|---|---|---|---|---| | M1 | **a write escapes** — the substitution deleted | `main.py` | −6 B | clamp | 5 | | M2 | **silence instead of 0 W** — it returns rather than substituting | `main.py` | −15 B | clamp | 3 | | M3 | **the observers go blind** — it moves above SAFETY-28 and SAFETY-03 | `main.py` | +58 B | clamp | 1 | | M4 | the site re-seed dropped — the exit resumes from a fiction | `main.py` | −36 B | clamp | 3 | | M5 | the per-unit re-seed dropped | `main.py` | −22 B | clamp | 1 | | M6 | **PAIRED**: `standing_w()` reads the target **and** the re-seed dropped — the sign latch that survives into live operation | `inverter.py` + `main.py` | −170 B | watchdog | 3 | | M7 | `auto` inverted — shadow while started, live while stopped | `shadow.py` | +2 B | validate | 4 | | M8 | **default-on for everyone** — `auto` always shadow; the literal reading that stops the live install | `shadow.py` | −15 B | validate | 2 | | M9 | an unreadable value resolves to LIVE instead of shadow | `shadow.py` | +1 B | validate | 1 | | M10 | `validate.py` stops refusing an unknown value | `shadow.py` | −9 B | validate | 9 | | M11 | the loop leg gated on `auto` alone again — the decision path stops running | `main.py` | −17 B | clamp | 1 | | M12 | the stop claim filed in shadow — every decision reads `stopped` | `main.py` | −20 B | clamp | 1 | | M13 | the banner goes green while nothing is commanded | `shadow.py` | −2 B | clamp | 1 | | M14 | the banner loses the standing latch | `shadow.py` | −2 B | clamp | 1 | | M15 | the row renders the standing command instead of the decision | `shadow.py` | −8 B | clamp | 1 | | M16 ⚠️ | the row disappears when live | `main.py` | +43 B | clamp | 1 | | M17 | the record becomes the wire | `main.py` | −6 B | clamp | 7 | | M18 | level-logged — the 40-row action log floods | `main.py` | −23 B | clamp | 1 | | M19 | the two rungs render identically (masking) | `shadow.py` | +22 B | validate | 1 | | M20 | the shipped default moves to `off` in `config.yaml` alone | `config.yaml` | −1 B | defaults_pin | 1 | | M21 | **PAIRED**: the substitution **and** the re-seed deleted together | `main.py` | −103 B | clamp | 10 | | M22 | the re-seed moved one line earlier — SAFETY-18 grades a 0 W no decision produced and goes permanently blind | `main.py` | +40 B | clamp | 1 | **Survivors: none.** ⚠️ **M17's `+0 B`-adjacent sibling** — `shadow_would_w = float(self.target)` — is recorded in §7 as **equivalent** rather than counted here.