SAFETY-10: the whole decision path runs and nothing reaches the inverter

Every optimiser defect this project has found was invisible from outside
while it was happening, and each was paid for at the meter first: SAFETY-45's
28 charge onsets an hour against a binding peak_cap_w, TARIFF-14's wrong
quarter-hour for 45 minutes in every 60, SAFETY-46's 598 Wh spent to gain
310 Wh. There was no way to watch what the optimiser WOULD do without letting
it do it.

`shadow_mode` runs strategies, arbiter, clamps and guards, records the
decision with its reason, and commands nothing.

⚠️ IT WRITES. It writes 0 W, on the ordinary heartbeat, through the ordinary
single writer. The inverter holds its last command forever - it has no meter
timeout - so "commands nothing" implemented as "stops writing" would leave
whatever was standing when shadow mode came on standing for as long as it was
left on, under a page saying nothing is being commanded. That is the failure
this repository keeps finding. Refreshing 0 W is also what
firmware/goodwe-master.yaml's 30 s stale-input watchdog requires of us; going
quiet would keep that board tripped for the whole run, which is an alarm being
used as a mode.

⚠️ IT IS NOT A SECOND WRITE PATH. No claim, no transport, no arithmetic of its
own. It is a substitution inside Controller._write_unit() - the single choke
point every write already reaches - in the same form as the three above it,
and it is the LAST of them, so SAFETY-28's gate and SAFETY-03's clamp still
grade the number the decision path produced.

⚠️ THE DEFAULT IS `auto`, NOT `true`, AND THAT IS THE CARD'S REAL CONTENT. The
Supervisor composes options as {**config.yaml defaults, **stored user
options}, so a new key with a new default reaches EVERY EXISTING INSTALL on
the next update - including the one regulating a real battery. A boolean
default-on would have stopped it silently. There is no mechanism by which one
key can default differently for a new install and an existing one, so
"default-on for new" is a value the add-on RESOLVES: `auto` is shadow while
control is STOPPED and live once it is STARTED. A fresh install ships
`auto_start: false` and comes up observing; an install that is controlling is
started and does not change by one watt.

THE FAILSAFES. Sign watchdogs (SAFETY-02/30): INERT - they grade
Inverter.standing_w(), which shadow mode holds at 0 W. Fed the shadow decision
they would disagree with a meter that never saw it and LATCH A TRIP THAT
SURVIVES INTO LIVE OPERATION AND A RESTART. Clamp (SAFETY-03), write gate
(SAFETY-28), meter freeze (SAFETY-18), peak guard: OBSERVE. Circulation
(SAFETY-31) and delivery (SAFETY-42): observe and may legitimately latch,
because their evidence is measured power. The operator stop is the one
deliberate change: enforced at the wire rather than as a P_SAFETY claim, which
would otherwise make every recorded decision read "stopped".

⚠️ RECORDED LIMITATION: _shadow_settle() re-seeds the carried command to what
stands on the wire, which makes the exit safe by construction - and therefore
the loop leg's shadow decision is ONE SLEW STEP, not a trajectory. A shadow
run answers "what would it command now", not "what would it have done next".
The strategies are unaffected; all three defects above are visible from one.

Graded on the writes and on the rendered surface, never on the flag: 22
mutations, no survivors, including one that lets a write escape and one that
goes silent instead of commanding 0 W. Closed loop on TEST-23 stage 3's rig,
same hour twice: live drives the pack to -2000 W and SoC 50.000 -> 62.331 %;
shadow commands 0 W throughout, SoC stays 50.0 exactly, and the arbitrage
decision is identical in action, instant, signed power, reason and arbiter
winner.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU

goodwe_controller/DECISIONS/SAFETY-10-shadow-mode.md

@@ -0,0 +1,321 @@
+# SAFETY-10 — shadow mode, and why "default-on" is not `true`
+
+**Status:** decided and implemented on branch `SAFETY-10`.
+**Card:** SAFETY-10 — *"Shadow mode (default-on, log-only decisions)"*.
+**Nothing here reached the inverter.** No deployment, no `ha addons update`, no
+contact with 192.168.2.158 or 192.168.2.221. Everything below was measured in
+the CI image and on TEST-23 stage 3's closed-loop rig.
+
+---
+
+## 0. The four things a reader should take away
+
+1. **"Default-on" is delivered as `auto`, not as `true`, and the difference is
+   an install that is regulating a real battery today.** The Supervisor
+   composes an add-on's options as *its defaults, overlaid with the stored user
+   options*, so a new key with a new default reaches **every existing install**
+   on the next update. A boolean `shadow_mode: true` would silently stop the
+   reference site. §1.
+2. **Shadow mode WRITES. It writes 0 W, on the ordinary heartbeat, through the
+   ordinary single writer.** "Commands nothing" implemented as "stops writing"
+   would leave the inverter latched on whatever was standing — it has no meter
+   timeout — and would put the ESP32 into a tripped stale-input watchdog for
+   the whole run. §2.
+3. **The sign watchdogs are inert, and that was the decision that had to be got
+   right.** Fed the shadow decision they would trip within `sign_disagree_s`
+   and **latch a trip that survives into live operation and a restart**. §4.
+4. **The loop leg's shadow decision is one slew step, not a trajectory, and
+   that is a real limitation of this feature.** §5.
+
+---
+
+## 1. The default
+
+### The literal reading, and what it costs
+
+`config.yaml`'s `options:` block **is** the default. There is exactly one of it,
+and HA's Supervisor resolves an add-on's effective options as
+
+```
+{**config.yaml options, **the stored user options}
+```
+
+A key that has never been stored therefore takes the file's value — on a fresh
+install and on an install that has been running for months, identically. There
+is **no mechanism in the add-on options system** by which one key can default
+differently for the two. So the card's title, taken at face value, is:
+
+> the next `ha addons update` stops the battery at the reference site, and the
+> only symptom is a battery that has stopped doing anything.
+
+That is strictly worse than every defect shadow mode exists to catch. It is
+refused.
+
+### What was shipped instead
+
+`shadow_mode` is a three-valued `list(auto|on|off)` defaulting to `"auto"`:
+
+| value | resolves to |
+|---|---|
+| `off` | live |
+| `on` | shadow, always |
+| `auto` | shadow **while control is stopped**; live once it is started |
+
+`auto` reads `Controller.auto` — the persisted start/stop state, which a fresh
+install takes from `auto_start` and which the *Start / stop control* button
+moves. No new persisted state, no migration, no guess:
+
+* a **fresh** install ships `auto_start: false`, has no stored `auto` key, and
+  therefore comes up **in shadow mode**. It observes a day of decisions before
+  it is ever allowed to command anything. That is the card's intent, delivered
+  where it is safe;
+* an install that is **controlling** is started, so `auto` resolves to live and
+  **not one watt changes**;
+* an install that is **stopped** keeps its actuator at 0 W exactly as before —
+  the same number, from the same writer, on the same heartbeat. What changes is
+  what the status page and the action log **say**, and that the decision path
+  now runs and is recorded.
+
+⚠️ **Under `auto`, pressing Start leaves shadow mode.** That is the point of
+`auto`: the button means "control the battery". An operator who wants the
+decisions recorded *while the loop runs* — the configuration this card exists
+to make possible — sets `on` explicitly, and cannot reach it by accident.
+
+⚠️ **An unrecognised value resolves to SHADOW, and is REFUSED at startup.**
+`shadow.mode()` fails toward inaction; `validate.py` rule 11 refuses the value
+by name so the reinterpretation is never silent. A typo is not a decision, and
+an install silently left in shadow mode commands nothing indefinitely while its
+status page calmly explains that this is intentional.
+
+---
+
+## 2. "Commands nothing" is a number, not a silence
+
+The inverter holds its last command **forever** — `HANDOVER.md` §6 says so, and
+`firmware/goodwe-master.yaml`'s header records what it cost when a controller
+stopped refreshing: *"Illusmart went silent mid-command and the inverter held
++5000 W for 113 s"*. `firmware/goodwe-master.yaml` states the contract that follows — *"HA
+must now REWRITE the setpoint every ~10 s even when the value has not
+changed"* — because without a refresh, **"stale" and "steady" are the same
+thing on the wire**.
+
+So shadow mode holds the actuator at `shadow.WIRE_W` = **0.0 W**, refreshed on
+the ordinary heartbeat. Two consequences were weighed and are recorded:
+
+* **it is not silence.** A shadow mode that stopped writing would leave the
+  standing command up until the ESP32's 30 s watchdog noticed — and at a site
+  with no such board, forever;
+* **it does not use an alarm as a mode.** Letting that watchdog trip for the
+  duration of a shadow run teaches the operator that `wd_tripped` is normal,
+  which is exactly the training that makes it useless when the add-on has
+  actually died.
+
+### It is not a second writer
+
+SAFETY-28's rule is one actuator, one writer. Shadow mode files **no claim**,
+owns **no transport**, and calls nothing in `hass.py`. It is a **substitution**
+inside `Controller._write_unit()` — the single choke point every write in this
+add-on already reaches, including the heartbeat, the stop button, `shutdown()`
+and `maint_abort()` — in exactly the form of the three substitutions above it
+(non-finite → 0 W, unauthorised → 0 W, clamped → the rating).
+
+It is the **last** of them. Below SAFETY-28's gate and below SAFETY-03's clamp,
+so both still grade the number the decision path actually produced, and a
+shadow run can report *"it would have hit the rating here"*.
+
+---
+
+## 3. What the run records
+
+| surface | what it carries |
+|---|---|
+| status banner | `SHADOW MODE - the battery is NOT being controlled...`, **first in the cascade**, amber — red, with the latch named, if one is standing |
+| status row `Shadow mode` | which rung decided, the watts it **would have commanded**, where the inverter actually is, and `Resolution.explain()` |
+| status row `Commanded` | `0 W` — the truth, which the row above reconciles with the `Why` row below it |
+| action log | one line when shadow mode comes on; one line per **change of decision** (winner or rounded watts), carrying the arbiter's own sentence |
+| MQTT | `status: shadow`, its own value, below `failsafe` |
+
+The action log is **edge**-logged, not level-logged: the log holds 40 rows, and
+a line per cycle would flush the context of the run's own findings inside four
+minutes at the 15 s cycle floor.
+
+---
+
+## 4. The failsafes, one by one
+
+| failsafe | in shadow mode | why |
+|---|---|---|
+| SAFETY-02/30 sign watchdogs | **INERT** | They grade `Inverter.standing_w()` — what this controller has standing on the wire — and shadow mode holds that at 0 W, so they cannot arm. ⚠️ Fed the shadow decision instead they would disagree with a meter that never saw it and **latch a trip that survives into live operation and a restart**: shadow mode would manufacture the failsafe it exists to help debug. ⚠️ **Two rungs hold it there** — `last_write` (the wire gate) and `inv.target` (the re-seed) — which is MUTATION.md's masking shape, so the mutation that reproduces the hazard defeats both as a pair (M6). Driven singly, dropping the re-seed does not reproduce it at all — measured |
+| SAFETY-03 clamp | **OBSERVES** | Runs on the decision, sets `clamp_last`, logs every engagement. Changes no command |
+| SAFETY-28 write gate | **OBSERVES** | Same placement, same reason: the question is who chose the number |
+| SAFETY-18 meter freeze | **OBSERVES**, unchanged | It takes no action live either; both halves of its conjunction are as true in shadow |
+| SAFETY-07/24/25 peak guard | **OBSERVES** | Level-triggered, unlatched, files its claim as usual — SAFETY-45's limit cycle is visible from a shadow run, which is the point |
+| SAFETY-31 circulation | **OBSERVES AND MAY LATCH** | Its evidence is *measured* power. Packs pushing against each other while all are told 0 W is a real fault |
+| SAFETY-42 delivery | **OBSERVES AND MAY LATCH** | Same, against a share that is 0 W |
+| operator stop | **SUPPRESSED AS A CLAIM, ENFORCED AT THE WIRE** | The one deliberate change. Its `P_SAFETY` 0 W claim out-ranks everything, so leaving it in makes the recorded decision `"stopped"` on every cycle and the run answers the wrong question. The actuator ends at the same number either way, and the wire substitution is **strictly stronger**: a claim can be out-ranked, and it cannot |
+| inputs missing, non-finite, allocator refusal, arbiter contradiction | **UNCHANGED** | All resolve to 0 W, which is what the wire gets anyway |
+
+⚠️ **A latch that is already up stays up.** Nothing here clears one. The shadow
+banner **names** any standing latch and goes red for it, because a latch that is
+up while the controller is inert is up the moment it stops being inert.
+
+⚠️ **Maintenance is not in that table because shadow mode does not touch it.**
+The state machines are ticked on every cycle whether control is started or not,
+so a stopped install already runs them against a battery that does not move.
+Shadow mode inherits that, unchanged and unimproved. Recorded here rather than
+quietly fixed under another card's name.
+
+---
+
+## 5. The limitation, stated rather than discovered later
+
+`Controller._shadow_settle()` re-seeds the carried command (`self.target` and
+each `inv.target`) to `WIRE_W` at the **end** of every shadow cycle — after
+every in-cycle guard has read the decision. Three machines read that value as a
+statement about the **hardware**: the loop's `prev_w` and integrator,
+`standing_w()`, and SAFETY-42's share comparison. Re-seeded, they track reality,
+and **the exit is safe by construction** — the carried command already equals
+the standing one, so the first live cycle slews from 0 W like any other.
+
+⚠️ **The cost is real.** The loop's `prev_w` is therefore 0 W on every shadow
+cycle, so the loop leg's recorded decision is **one slew step from the standing
+command** and stays there. A shadow run answers *"what would this controller
+command NOW, from where the plant really is"* — a true statement about the
+present — and **not** *"what trajectory would it have followed"*. The trajectory
+is a closed-loop property, it did not happen, and reporting it would be a
+simulation of a plant that does not exist; three cycles in, such a number
+describes nothing.
+
+The strategies are unaffected: arbitrage, the peak guard and maintenance all
+file `SET` claims at a fixed power from level-triggered inputs. That is why the
+three defects this card was written for — SAFETY-45's 28 onsets an hour,
+TARIFF-14's wrong quarter-hour, SAFETY-46's ceiling churn — are all visible from
+a shadow run, and the loop's ramp is not. Graded, named, in
+`test_watchdog.py` part 10 as `RECORDED LIMITATION`.
+
+---
+
+## 6. How "nothing reaches the inverter" is graded
+
+**Not on a flag.** A check that asserted `Controller.shadow` would pass on a
+build in which the flag is set and the substitution is deleted — which is
+exactly mutation **M1**.
+
+`test_clamp.py` part 8 drives the **shipped** `Controller.cycle()` and the
+shipped heartbeat against a recording Home Assistant stub, and asserts on what
+reached `set_number()`:
+
+* **the precondition first** — the same stimulus with `shadow_mode: off` puts
+  30 non-zero writes on the wire, the largest 5000 W. MUTATION.md habit 4: a
+  check whose data cannot reach the failure it names is decoration;
+* then `set(writes) == {0.0}`, **and** that there were 26 of them — a check
+  that only said "no non-zero write" would pass on the silent build too;
+* entry and exit are measured on `inv.last_write` — what the inverter is
+  latched on, updated only on an **accepted** write — either side of the
+  boundary, against the same run that never entered shadow mode.
+
+The closed-loop half is `/home/homelab/s10-mut/shadowloop.py` on TEST-23 stage
+3's rig: the same hour, twice, against `plantsim.Plant` through the firmware's
+own frame encoder.
+
+| | live | shadow |
+|---|---|---|
+| `plant_cmd_w` | −2000 W throughout | **0 W throughout** |
+| `batt_w` | −2000 .. −40.6 W | **0.0 W, exactly** |
+| SoC | 50.000 → **62.331 %** | 50.0 → **50.0 %** |
+| grid | 390.6 .. 2350 W | **350 W = the house, exactly** |
+| `ARBITRAGE_CHARGE` samples | 3600 | **3600** |
+| first charge instant | 2026-01-15T00:00:00+00:00 | **identical** |
+| `arb_power_w` | −2000 W | **identical** |
+| `arb_reason` | the strategy's own sentence | **identical, verbatim** |
+| arbiter winner | `arbitrage` | **identical** |
+
+The decision happened, at the same instant, for the same reason, at the same
+signed power — and the battery did not move.
+
+⚠️ **Three lines of that harness are a TRANSCRIPTION and are marked as one.**
+`priceloop.Rig` has no `_write_unit()`, no clamp and no failsafe machine
+(divergence 2 of its README), so the substitution's **placement** is
+transcribed there. The shipped placement is graded by `test_clamp.py` part 8,
+which drives the real `Controller`. ⚠️ **And the rig's transcribed cycle
+predates SAFETY-45**: `arbitrage.decide()` gained a required
+`charge_headroom_w` after `478ba93`, and the harness supplies it from the
+**shipped** `peakguard.charge_headroom_w()` reading the **shipped** peak
+decision rather than from a constant, so SAFETY-45 stays in the run.
+
+---
+
+## 7. The mutation battery — 22 mutations, no survivors
+
+Harness: `/home/homelab/s10-mut/run.py`, backups beside it. Each run restores
+from backup, purges `__pycache__` **on the host**, and grades through the CI
+image with the tree bind-mounted `:ro`, so the interpreter cannot write a cache
+into the tree under test (TEST-15). The full table is in the branch report;
+one per new behaviour, including **M1** (the substitution deleted — a write
+escapes), **M2** (it returns instead of substituting — the inverter is left
+latched), **M3** (it moves above SAFETY-28 and SAFETY-03 — the observers go
+blind), **M8** (`auto` resolves to shadow unconditionally — the literal
+default-on that stops the live install), **M12** (the stop claim is filed in
+shadow — every recorded decision reads `stopped`), **M21** (the substitution
+and the re-seed together) and **M22** (the re-seed moved one line earlier —
+SAFETY-18 grades a 0 W no decision produced and goes permanently blind).
+
+⚠️ **Two findings from driving it, both of the kind MUTATION.md says to expect.**
+
+* **M2 first came back a SURVIVOR and was not one.** The "stop writing
+  altogether" build left `inv.last_write` as `None`, the check did `float()` on
+  it, and the section died with a `TypeError` — grading none of the checks
+  after it and reporting **zero reds**, which reads exactly like a survivor.
+  That is FCST-06's shape. The fixture now hands `None` back and names it as
+  its own check; M2 is red on three.
+* **M6 had to become a PAIR, and only the battery said so.** `standing_w()`
+  prefers `last_write` — the 0.0 W the wire gate accepted — and falls back to
+  `inv.target`, which the re-seed also holds at 0.0. Either rung alone keeps
+  the detector inert, so dropping the re-seed **singly** came back red on the
+  limitation check and on **nothing about the latch**: the hazard it names was
+  not reproduced. TEST-16's rule with a new name on it.
+* **M7/M8/M9/M19 had to MOVE SUITES.** They defeat `shadow.engaged()`, and a
+  defeated default stops `test_clamp.py` part 3's wired run reaching the wire
+  at all — aborting the file with an `IndexError` before part 8 is graded. The
+  red count there was a lower bound naming the saturation freeze. The mode
+  table now lives in `test_validate.py`, which drives no Controller.
+* **`shadow_would_w = float(self.target)` is an EQUIVALENT mutation and was
+  measured as one.** `_shadow_settle()` runs before the re-seed, so at that
+  line `self.target` **is** `resolution.target_w`. It is recorded as equivalent
+  rather than reported as a survivor, and the non-equivalent form of the same
+  defect — the record becoming the **wire** — is driven instead.
+
+### The table
+
+Each row: the mutation, the file, the byte delta, the suite it was graded in,
+and how many **named** checks went red. `⚠️` marks a row whose run did not
+reach the end of its file, so its count is a **lower bound** (MUTATION.md).
+
+| # | mutation | file | Δ | suite | reds |
+|---|---|---|---|---|---|
+| M1 | **a write escapes** — the substitution deleted | `main.py` | −6 B | clamp | 5 |
+| M2 | **silence instead of 0 W** — it returns rather than substituting | `main.py` | −15 B | clamp | 3 |
+| M3 | **the observers go blind** — it moves above SAFETY-28 and SAFETY-03 | `main.py` | +58 B | clamp | 1 |
+| M4 | the site re-seed dropped — the exit resumes from a fiction | `main.py` | −36 B | clamp | 3 |
+| M5 | the per-unit re-seed dropped | `main.py` | −22 B | clamp | 1 |
+| M6 | **PAIRED**: `standing_w()` reads the target **and** the re-seed dropped — the sign latch that survives into live operation | `inverter.py` + `main.py` | −170 B | watchdog | 3 |
+| M7 | `auto` inverted — shadow while started, live while stopped | `shadow.py` | +2 B | validate | 4 |
+| M8 | **default-on for everyone** — `auto` always shadow; the literal reading that stops the live install | `shadow.py` | −15 B | validate | 2 |
+| M9 | an unreadable value resolves to LIVE instead of shadow | `shadow.py` | +1 B | validate | 1 |
+| M10 | `validate.py` stops refusing an unknown value | `shadow.py` | −9 B | validate | 9 |
+| M11 | the loop leg gated on `auto` alone again — the decision path stops running | `main.py` | −17 B | clamp | 1 |
+| M12 | the stop claim filed in shadow — every decision reads `stopped` | `main.py` | −20 B | clamp | 1 |
+| M13 | the banner goes green while nothing is commanded | `shadow.py` | −2 B | clamp | 1 |
+| M14 | the banner loses the standing latch | `shadow.py` | −2 B | clamp | 1 |
+| M15 | the row renders the standing command instead of the decision | `shadow.py` | −8 B | clamp | 1 |
+| M16 ⚠️ | the row disappears when live | `main.py` | +43 B | clamp | 1 |
+| M17 | the record becomes the wire | `main.py` | −6 B | clamp | 7 |
+| M18 | level-logged — the 40-row action log floods | `main.py` | −23 B | clamp | 1 |
+| M19 | the two rungs render identically (masking) | `shadow.py` | +22 B | validate | 1 |
+| M20 | the shipped default moves to `off` in `config.yaml` alone | `config.yaml` | −1 B | defaults_pin | 1 |
+| M21 | **PAIRED**: the substitution **and** the re-seed deleted together | `main.py` | −103 B | clamp | 10 |
+| M22 | the re-seed moved one line earlier — SAFETY-18 grades a 0 W no decision produced and goes permanently blind | `main.py` | +40 B | clamp | 1 |
+
+**Survivors: none.** ⚠️ **M17's `+0 B`-adjacent sibling** —
+`shadow_would_w = float(self.target)` — is recorded in §7 as **equivalent**
+rather than counted here.