diff --git a/goodwe_controller/DOCS.md b/goodwe_controller/DOCS.md index e6e3aa7..6eb01ac 100644 --- a/goodwe_controller/DOCS.md +++ b/goodwe_controller/DOCS.md @@ -1574,6 +1574,112 @@ battery power, circulating power, state of charge, maintenance phase and controller status by MQTT discovery. This is observability only — the controller works fine without a broker, and MQTT problems can never affect control. +## Before you commission inverter B: `tools/rehearse_roster.py` (ARCH-02) + +The day the `inverters:` roster is declared, two things change that are +**one-way on a live box and invisible until after the restart that applies +them**: the store migration renames the keys that hold the SAFETY-02 latch and +the maintenance history, and the entity ids Home Assistant composes change +shape. This tool runs the **real `Controller`** through that transition against +a **copy** of the state file and reports the difference. + + docker run --rm -e PYTHONDONTWRITEBYTECODE=1 -v :/w:ro \ + goodwe-addon-checks:1 sh -c 'cd /w/goodwe_controller && \ + python3 tools/rehearse_roster.py \ + --state /w/state-copy.json \ + --before /w/live-options.json \ + --after /w/roster-options.json \ + --live-entities /w/live-entities.txt \ + --reverse --round-trip' + +Exit `0` clean · `1` at least one finding · `2` could not run. ⚠️ **`2` is not a +pass.** + +### The owner steps, because this session could not take them + +⚠️ **The rehearsal is only as good as the state it is pointed at, and nothing in +this repository can fetch that.** SSH to the live box is blocked by policy and +that is correct — `/data/state.json` is the file the running add-on is writing. +So three things have to be exported by hand, onto the machine that runs the +check, **before** the roster is declared: + +1. **The state file.** From the Supervisor host: + + ha addons info local_goodwe_controller # confirm the slug first + docker cp addon_local_goodwe_controller:/data/state.json ./state-copy.json + + or take it out of a full add-on backup, which is the same bytes. ⚠️ **Copy + it, never move it, and never write to the original.** The tool opens the path + it is given exactly once and does all of its writing on a scratch copy — but + the safe habit is to hand it a file the add-on has never heard of. + +2. **The options blob**, as `--before`. It is not reachable through the Home + Assistant REST API and no entity carries it; export it from the Supervisor, + the same way `tools/predeploy_check.py` needs it. Write the roster you intend + to deploy into a second file and pass it as `--after`. + +3. **The entity list**, as `--live-entities` — one entity id per line: + + Developer tools -> Template: + {{ states | map(attribute='entity_id') | select('search','goodwe') | join('\n') }} + + ⚠️ **Without it the tool cannot see a change to the id composition at all**, + because both sides of its own before/after diff are composed by the same + build: rename the MQTT device and every id moves equally on both sides, and + the diff reports "nothing orphaned" about a change that orphans everything. + `tools/rehearsal/live-entities-recorded.txt` is the recorded stand-in that + keeps the check live in CI; the real export is what makes it true of *this* + install. The install carries roughly 49 `goodwe*` entities and this add-on + announces eight of them — the tool separates the ids it owns from the ids it + does not, rather than claiming orphans it cannot cause. + +⚠️ **No result in this repository was produced against real state.** Everything +recorded here and in `test_rehearse.py` was measured on the synthetic fixtures +in `tools/rehearsal/`. + +### What it found on the synthetic fixtures + +⚠️ **A pre-DEV-02 flat store that declares a multi-inverter roster comes up with +its SAFETY-02 latch DISARMED, unless the first roster entry happens to be named +`inverter-a`.** `Controller.__init__` calls `migrate_unit_keys(store, UNIT_A)` +with a hardcoded constant, so the latch lands in `unit/inverter-a/sign_tripped`; +a roster of `["garage", "shed"]` reads `unit/garage/sign_tripped`, which is +absent. And `_grade_store_migration()` — the guard written for exactly this — +sees the migration's own misdirected key satisfy its `namespaced_latched` test +and returns without re-arming. The file says `sign_tripped: true`, the banner is +green, and nothing is held at 0 W. **This is SAFETY-44, and the answer to it is +that the asymmetry is not safe on that path.** It is reported here and pinned by +`test_rehearse.py` part 5; the fix belongs to its own card, shown red first. + +The same end state is reached whether the roster is declared in the same step as +the upgrade or a restart later — in the staged case because the flat start +stamps `schema_version` and the roster start's migration then returns early. + +### The reverse direction: the retreat is supported, the round trip is not + +If commissioning B goes wrong, **backing out to the flat single-inverter config +is supported and does not corrupt state.** Measured on the fixtures: + +* the SAFETY-02 latch and the first pack's maintenance history both survive, + because `store.py` rule 2 keeps the flat spellings **live** rather than merely + present — `maintenance.UnitHistory.mirror_flat` writes them on every change, + so a single-FSM add-on finds a current history rather than one dated the day + the roster was declared; +* **it orphans eight Home Assistant entities** — the four per-unit rows for each + of the two units. Their history stops and any automation naming them stops + firing, silently. That is the price of the retreat and it is not recoverable + by advancing again: HA keeps the id it composed at first discovery. + +⚠️ **What is NOT supported is the round trip.** A maintenance cycle that +completes *while the install is rolled back* is **discarded** when it +re-advances: `maint_schema_version` is already stamped, so `migrate_maint_keys()` +takes its early return and the pack reads the namespaced history from before the +retreat. The pack is then recalibrated on a schedule that does not know it +already was. **So: retreat freely, but treat a retreat as ending the current +maintenance bookkeeping — and if a cycle ran during it, correct +`unit//last_completed` by hand before advancing again.** This is a stated +decision, not a discovery made at the inverter. + ## Before you deploy: `tools/predeploy_check.py` (PKG-09) Two ways a deploy has gone wrong on this system, both of which the Supervisor diff --git a/goodwe_controller/MUTATION.md b/goodwe_controller/MUTATION.md index 9f3c565..fd741ab 100644 --- a/goodwe_controller/MUTATION.md +++ b/goodwe_controller/MUTATION.md @@ -225,6 +225,32 @@ same place twice, the check is decoration. forever, *including after the call site is fixed*. The check was true and passed for the wrong reason. Habit **9** below is what falls out of it. +* ⚠️ **ARCH-02's OWN REHEARSAL HARNESS, CAUGHT BY ITS OWN MUTATION BATTERY — + the THIRTEENTH instance, and the third in an INSTRUMENT rather than in a + check.** `tools/rehearse_roster.py` reports orphaned Home Assistant entities + by composing the ids the add-on announces **before** the roster switch and + **after** it, and diffing. Both sides are composed by the same build. So the + two mutations that change the id composition itself — renaming the MQTT device + (`"GoodWe"` -> `"GoodWe Controller"`), renaming one site sensor (`"Grid power"` + -> `"Grid power W"`) — moved **every id on both sides equally**, and the + harness reported *"no orphans"* about a change that orphans the entire live + entity list. It read exactly like a clean transition. The first draft's + M5/M6 came back `exit 0` and `exit 2`, and neither was the failure they name. + + The fix is habit 2 with a filename: **the reference has to be data that does + not move with the code.** `--live-entities` takes a list of the ids the + install actually has, `tools/rehearsal/live-entities-recorded.txt` is the + recorded stand-in that keeps it live in CI, and a build that announces an id + the list does not carry is now a `COMPOSITION DRIFT` finding rather than a + silence. ⚠️ **The anchor moved for the same reason**: it had been + `compose(mqtt.DEVICE["name"], p1["name"]) == "sensor.goodwe_p1_sample_age"`, + which any renaming of the device satisfies by moving both operands, and it is + now composed from two recorded **literals**. + + Ask of an instrument what habit 4 asks of a check: *where did each side of + this comparison come from* — and if a mutation of the subject moves both + sides, the instrument is decoration. + * **CFG-04's `tools/supervisor_shape_probe.py` §B** — and this one is the rule reaching outside the suite. The probe is the committed evidence that `config.yaml`'s `inverters: []` default is load-bearing, and four documents diff --git a/goodwe_controller/test_rehearse.py b/goodwe_controller/test_rehearse.py new file mode 100644 index 0000000..7114a3c --- /dev/null +++ b/goodwe_controller/test_rehearse.py @@ -0,0 +1,410 @@ +"""Runnable check for the roster rehearsal. `python3 test_rehearse.py` + +ARCH-02. `tools/rehearse_roster.py` is the instrument that decides whether the +flat -> roster switch on 2026-09-12 loses a safety latch, a month of maintenance +history, or a Home Assistant entity that an automation is watching. This file +grades the instrument. + +⚠️ AN INSTRUMENT THAT HAS NEVER BEEN SEEN TO GO RED IS NOT EVIDENCE. This +repository's most-repeated defect, `MUTATION.md` habit 4 and now THIRTEEN +recorded instances, is "a check whose DATA cannot reach the failure it names" - +and a rehearsal that reports "migration OK" because it compared a structure to +itself is the purest example of it there is. So this file does two things in +order, and the first one is the one that makes the second mean anything: + + Part 2 TWO CLEAN CONTROLS. Two (state file, roster) pairs on which the + shipped code is CORRECT and the harness must report NOTHING. A + harness that is red on every input has measured nothing, and every + red below it would be free. + + Part 3 ⭐ SIX DELIBERATE MUTATIONS OF THE SHIPPED MIGRATION AND OF THE + ENTITY-ID COMPOSITION, each applied to a private copy of the tree, + each required to flip its control from green to red, AND each + required to produce the SPECIFIC finding it should - not merely a + non-zero exit. A mutation that goes red for the wrong reason is a + harness that would go red on the real defect for the wrong reason + too. + + Part 4 ONE MUTATION THAT DELIBERATELY SURVIVES, with the reason written + down. A battery in which everything dies is a battery whose author + stopped as soon as the number looked good. + + Part 5 SAFETY-44's exact path, pinned as a finding, by name, in both + directions. + +⚠️ THE MUTATIONS ARE SOURCE EDITS IN A COPIED TREE, NOT MONKEYPATCHES. A +monkeypatch grades the name a module was asked for; an edit grades the code that +runs. Each copy is made with `mkdtemp()`, so two runs of this file cannot +overwrite each other's work - habit 18 - and `__pycache__` is never copied, so +no mutant is ever graded through a stale `.pyc` - habit 1 / TEST-15. +""" + +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +HERE = Path(__file__).resolve().parent +FIX = HERE / "tools" / "rehearsal" + +fails = [] + + +def check(name, cond): + print(f" {'ok ' if cond else 'FAIL'} {name}") + if not cond: + fails.append(name) + + +# --------------------------------------------------------------------------- # +print("1. the harness's own preconditions") +# --------------------------------------------------------------------------- # +sys.path.insert(0, str(HERE / "tools")) +import rehearse_roster as RR # noqa: E402 + +check("the tool refuses rather than passing when it cannot run: its three exit " + "codes are distinct and 2 is not 0", + (RR.EXIT_OK, RR.EXIT_FINDING, RR.EXIT_COULD_NOT_RUN) == (0, 1, 2)) + +# ⚠️ THE ANCHOR. `app/mqtt.py` records `sensor.goodwe_p1_sample_age` as VERIFIED +# against the live system on 2026-08-26, and the ESP32's ingest guard subscribes +# to it. The tool's entity-id model is written independently of `mqtt._slug` on +# purpose (habit 4), so this is the one external fact that says the independent +# model is right. If it fails, every orphan claim the tool makes is worthless +# and the tool exits 2 rather than reporting. +try: + RR._check_anchor() + anchored = True +except RR.Refused: + anchored = False +check("the entity-id model reproduces the ONE id verified against the live " + f"system ({RR.ANCHOR_ID})", anchored) + +check("...and it is NOT built out of `mqtt._slug`, so a mutation in the " + "shipped slugifier moves one side of the comparison and not both", + "_slug" not in RR.compose.__code__.co_names) + +_opts = RR.load_options(str(FIX / "roster.yaml")) +check("the built-in YAML reader parses the roster fixture the check image " + "cannot parse with PyYAML (there is no `yaml` module in it)", + [e["name"] for e in _opts["inverters"]] == ["garage", "shed"] + and _opts["max_w"] == 5000 and _opts["batt_invert"] is False + and _opts["meter_source"] == "off") +try: + RR._mini_yaml("inverters: [a, b]\n") + _refused = False +except RR.Refused: + _refused = True +check("...and it REFUSES a construct it does not understand rather than " + "guessing, because a config silently mis-parsed here rehearses a " + "configuration nobody is going to deploy", _refused) + +_flat = RR.load_options(str(FIX / "flat.yaml")) +check("⚠️ the two configs differ in the ONE way the transition is about: the " + "BEFORE roster is empty and the AFTER roster is declared - if they did " + "not, every diff below would be a structure compared to itself", + _flat.get("inverters") == [] and len(_opts["inverters"]) == 2) +check("...and the AFTER roster's first entry is NOT `inverter-a`, which is " + "habit 4's seventh instance in one line: with it, `UNIT_A` and " + "`inverters[0].name` are the same string and a misdirected namespace " + "cannot be seen", + _opts["inverters"][0]["name"] != RR.UNIT_A) + +_legacy = json.loads((FIX / "state-legacy.json").read_text()) +check("⚠️ the synthetic state carries MAINTENANCE HISTORY - state built by the " + "happy path cannot detect a migration that drops it", + _legacy.get("last_completed") and _legacy.get("phase")) +check("⚠️ ...and a LEGACY-SHAPED KEY WHOSE LOSS WOULD BE SILENT: a LATCHED " + "`sign_tripped` with no `schema_version` beside it, i.e. written by a " + "controller that had never heard of unit namespaces", + _legacy.get("sign_tripped") is True + and "schema_version" not in _legacy + and "maint_schema_version" not in _legacy) + + +# --------------------------------------------------------------------------- # +print() +print("2. THE CLEAN CONTROLS: two transitions on which the shipped code is " + "right and the harness must say nothing") +# --------------------------------------------------------------------------- # +# ⚠️ THIS PART IS WHAT MAKES PART 3 EVIDENCE. Six reds prove nothing about an +# instrument that is red on its own fixtures. Each control below is a genuine +# flat -> roster switch that the shipped migration handles correctly, and the +# required answer is EXIT 0. +# +# CONTROL-A a PRE-DEV-02 flat store, latch tripped, onto a roster whose +# first entry IS `inverter-a`. This is the path SAFETY-44's +# documented reasoning describes, and on it that reasoning holds. +# CONTROL-B a store that has already run flat under 0.7.0 (schema stamped, +# no latch), onto a roster named `garage`/`shed`. This is the +# MAINT-05 half, and the names are deliberately not `inverter-a`. +CONTROLS = { + "CONTROL-A": ("state-legacy.json", "roster-inverter-a.yaml"), + "CONTROL-B": ("state-flat-070.json", "roster.yaml"), +} + + +def run(tree: Path, state: str, after: str, *extra) -> "tuple[int, str]": + """The harness, as the owner runs it, over `tree`. + + ⚠️ ALWAYS WITH `--live-entities`. That file is the only part of the + entity-id comparison that does not move with the code, so a run without it + is a run that cannot see a renamed device. + """ + fixtures = tree / "tools" / "rehearsal" + proc = subprocess.run( + [sys.executable, str(tree / "tools" / "rehearse_roster.py"), + "--state", str(fixtures / state), + "--before", str(fixtures / "flat.yaml"), + "--after", str(fixtures / after), + "--live-entities", str(fixtures / "live-entities-recorded.txt"), + *extra], + capture_output=True, text=True, + env={**os.environ, "PYTHONDONTWRITEBYTECODE": "1"}) + return proc.returncode, proc.stdout + proc.stderr + + +for label, (state, after) in CONTROLS.items(): + code, out = run(HERE, state, after) + check(f"{label}: the shipped code, {state} -> {after}, exits 0 with no " + f"findings", code == RR.EXIT_OK and "NO FINDINGS" in out) + +# ⚠️ AND THE INPUT FILE IS NEVER WRITTEN. The owner runs this against a copy of +# a live state file; a tool that edited it would be a tool that had to be +# trusted rather than one that cannot do harm. +_before_bytes = (FIX / "state-legacy.json").read_bytes() +run(HERE, "state-legacy.json", "roster.yaml", "--reverse", "--round-trip") +check("the state file the harness was pointed at is BYTE-IDENTICAL afterwards, " + "including after the reverse and round-trip legs, which write", + (FIX / "state-legacy.json").read_bytes() == _before_bytes) + + +# --------------------------------------------------------------------------- # +print() +print("3. ⭐ THE MUTATIONS: each one breaks the shipped migration or the " + "entity-id composition, and each must turn its control RED") +# --------------------------------------------------------------------------- # +def mutant_tree(edits) -> str: + """A private copy of the add-on with `edits` applied. + + ⚠️ EVERY EDIT IS ASSERTED TO HAVE MATCHED. Four defects in this session came + from `str.replace` calls whose pattern did not match: the call silently + returns the original string, the mutation never happens, and the harness + reports a SURVIVOR on code it never changed. That is a fabricated result. + """ + tmp = tempfile.mkdtemp(prefix="rehearse-mut-") + dest = Path(tmp) / "tree" + ignore = shutil.ignore_patterns("__pycache__", "*.pyc") + (dest / "tools").mkdir(parents=True) + shutil.copytree(HERE / "app", dest / "app", ignore=ignore) + shutil.copytree(FIX, dest / "tools" / "rehearsal", ignore=ignore) + shutil.copyfile(HERE / "tools" / "rehearse_roster.py", + dest / "tools" / "rehearse_roster.py") + for rel, old, new in edits: + path = dest / rel + text = path.read_text(encoding="utf-8") + hits = text.count(old) + if hits != 1: + raise AssertionError( + f"mutation target appears {hits} times in {rel}, expected 1: " + f"{old!r} - the mutation did NOT happen and any result from it " + f"would be fabricated") + path.write_text(text.replace(old, new), encoding="utf-8") + return str(dest) + + +MUTATIONS = [ + # (id, control, what it breaks, edits, the finding it must produce) + ("M1", "CONTROL-B", + "the maintenance migration is handed the hardcoded `UNIT_A` instead of " + "the roster's own first name - MAINT-05's exact shipped regression, which " + "left all 19 suites green", + [("app/main.py", + "store, self.inverters[0].name, [inv.name for inv in self.inverters])", + "store, UNIT_A, [inv.name for inv in self.inverters])")], + "MAINTENANCE HISTORY LOST"), + + ("M2", "CONTROL-B", + "the maintenance namespace is DROPPED: the migration moves no keys at all, " + "so a month of history stays under a flat spelling the new packs do not " + "read", + [("app/store.py", + "_move_flat_keys(store, unit, MAINT_UNIT_KEYS, rep)", + "_move_flat_keys(store, unit, (), rep)")], + "MAINTENANCE HISTORY LOST"), + + ("M3", "CONTROL-A", + "the SAFETY-02 pair loses HALF of itself: `sign_reason` drops out of " + "`LEGACY_UNIT_KEYS`, so the latch migrates without the sentence that says " + "why it tripped", + [("app/store.py", + 'LEGACY_UNIT_KEYS = ("sign_tripped", "sign_reason")', + 'LEGACY_UNIT_KEYS = ("sign_tripped",)')], + "LATCH KEY NOT UNDER A ROSTER NAME"), + + ("M4", "CONTROL-A", + "the READER's key composition changes while the migration's does not - " + "`Inverter.key()` upper-cases the unit - so the latch is written " + "perfectly, read back perfectly by the migration, and looked for under a " + "name that is not there", + [("app/inverter.py", + "return unit_key(self.name, name)", + "return unit_key(self.name.upper(), name)")], + "LATCH CAME UP DISARMED"), + + ("M5", "CONTROL-A", + "the entity-id composition changes on the DEVICE half: the MQTT device is " + "renamed, which is the 'tidy it up' change `app/mqtt.py` spends thirty " + "lines in capitals refusing. ⚠️ THE BEFORE/AFTER DIFF IS BLIND TO THIS - " + "both sides are composed by the mutant, so every id moves equally and " + "nothing looks orphaned. Only the recorded list catches it", + [("app/mqtt.py", '"name": "GoodWe",', '"name": "GoodWe Controller",')], + "COMPOSITION DRIFT"), + + ("M6", "CONTROL-A", + "the entity-id composition changes on the ENTITY half: one site sensor is " + "renamed, which orphans exactly one id and leaves the other seven alone", + [("app/mqtt.py", + '("grid", "goodwe_grid_power", "Grid power", "W", "power", "measurement", None),', + '("grid", "goodwe_grid_power", "Grid power W", "W", "power", "measurement", None),')], + "COMPOSITION DRIFT: this build announces sensor.goodwe_grid_power_w"), +] + +for mid, control, what, edits, must_say in MUTATIONS: + state, after = CONTROLS[control] + tree = mutant_tree(edits) + code, out = run(Path(tree), state, after) + red = code == RR.EXIT_FINDING + named = must_say in out + print() + print(f" --- {mid} ({control}) {what}") + print(f" exit {code} (0=clean 1=finding 2=could-not-run)") + for line in out.splitlines(): + if line.strip().startswith("⚠️ "): + print(f" {line.strip()[:150]}") + check(f"{mid} turns {control} RED", red) + check(f"{mid} says WHY, and it is the right why: {must_say!r}", named) + shutil.rmtree(Path(tree).parent, ignore_errors=True) + + +# --------------------------------------------------------------------------- # +print() +print("4. THE MUTATION THAT SURVIVES, ON PURPOSE, WITH THE REASON WRITTEN DOWN") +# --------------------------------------------------------------------------- # +# ⚠️ A BATTERY IN WHICH EVERYTHING DIES IS A BATTERY SOMEBODY STOPPED WRITING +# AS SOON AS THE NUMBER LOOKED GOOD. `store.unit_key()` is the ONE spelling of +# the namespaced key, read by `Inverter.key()`, by `UnitHistory.key()` and by +# both migrations - that is store.py's stated design, "one spelling, in one +# place". Changing it therefore moves the writer AND every reader together, and +# the store comes out internally consistent under a different name. +# +# The harness is BLIND to it and cannot be otherwise: nothing inside one version +# of this add-on can see that the key shape stopped matching what an OLDER +# version reads. That is a ROLLBACK contract, and grading it needs two versions +# of the code, which is a different instrument (`tools/predeploy_check.py`'s +# territory). Stated here rather than left as an unexplained green. +_tree = mutant_tree([("app/store.py", + 'return f"unit/{unit}/{key}"', + 'return f"ns.{unit}.{key}"')]) +_code, _out = run(Path(_tree), *CONTROLS["CONTROL-A"]) +print(f" renaming the ONE key spelling -> exit {_code}") +check("renaming `unit_key()` survives, because writer and readers move " + "together - a fact about the design, recorded, not a gap being hidden", + _code == RR.EXIT_OK) +check("...and the store really did change shape underneath it, so this is a " + "survivor of a mutation that HAPPENED rather than of one that did not", + "ns.inverter-a.sign_tripped" in _out and "unit/inverter-a/" not in _out) +shutil.rmtree(Path(_tree).parent, ignore_errors=True) + + +# --------------------------------------------------------------------------- # +print() +print("5. SAFETY-44: a pre-DEV-02 flat store meeting a DECLARED ROSTER, which " + "is the path nobody had established either way") +# --------------------------------------------------------------------------- # +# The question on the card: `migrate_unit_keys(store, UNIT_A)` takes a hardcoded +# constant and has no `misdirected` guard, while `migrate_maint_keys(store, +# unit, roster)` takes the roster and does. Is that asymmetry safe? +# +# ⚠️ MEASURED, NOT ARGUED. The two checks below are the answer, and they are +# written as REQUIREMENTS so that a fix to `migrate_unit_keys` turns them red +# and this file has to be updated deliberately rather than drifting. +code44, out44 = run(HERE, "state-legacy.json", "roster.yaml") +print(f" flat store + roster ['garage','shed'] -> exit {code44}") +check("⚠️ SAFETY-44 IS A DEFECT AGAIN: a store that ran pre-DEV-02 flat and " + "declares a multi-inverter roster comes up with its SAFETY-02 latch " + "DISARMED - the file says `sign_tripped: true` and not one inverter is " + "latched", + code44 == RR.EXIT_FINDING and "LATCH CAME UP DISARMED" in out44) +check("...and the harness names the two latch keys SAFETY-44 asks about, by " + "the namespace they were stranded in", + "unit/inverter-a/sign_tripped" in out44 + and "unit/inverter-a/sign_reason" in out44) +check("...and the mechanism is visible in the report: the keys are stranded " + "under `inverter-a`, which is not on the roster, so no machine reads them", + "STRANDED STORE KEY" in out44 and "['garage', 'shed']" in out44) +check("⚠️ AND IT IS NOT THE SAME-STEP PATH ALONE. Running the flat config " + "first and declaring the roster afterwards - the staged upgrade - " + "reaches the same disarmed latch, because the first start stamps " + "`schema_version` and the second start's migration then returns early", + out44.count("LATCH CAME UP DISARMED") >= 2) + +# ⚠️ AND THE CONTRAST, WHICH IS WHAT STOPS THE CHECK ABOVE BEING VACUOUS. The +# SAME state file onto a roster whose first entry IS `inverter-a` comes up +# LATCHED. The two runs differ in one thing - the first roster name - so the +# finding is attributable to it and to nothing else. +code_ok, out_ok = run(HERE, "state-legacy.json", "roster-inverter-a.yaml") +check("the SAME state file onto a roster whose first entry IS `inverter-a` " + "comes up latched and clean - so the finding above is caused by the " + "roster NAME and not by the fixture being tripped", + code_ok == RR.EXIT_OK and "LATCH CAME UP DISARMED" not in out_ok) + + +# --------------------------------------------------------------------------- # +print() +print("6. THE REVERSE DIRECTION: the retreat, and the return from it") +# --------------------------------------------------------------------------- # +code_r, out_r = run(HERE, "state-flat-070.json", "roster.yaml", + "--reverse", "--round-trip") +_orphaned = set(re.findall(r"ORPHANED ENTITY: (sensor\.\S+)", out_r)) +print(f" the retreat orphans: {sorted(_orphaned)}") +check("the RETREAT is rehearsed at all, and it reports the entities it " + "orphans - the eight per-unit rows, which is the cost of backing out", + "THE RETREAT" in out_r and _orphaned == { + f"sensor.goodwe_{u}_{k}" + for u in ("garage", "shed") + for k in ("setpoint", "battery_power", "battery_soc", "sign_latch")}) +check("...and NOT one site entity: the eight ids the install has today survive " + "the retreat, which is what makes the eight above attributable to the " + "per-unit rows and not to a blanket rename", + not any(o in _orphaned for o in + ("sensor.goodwe_grid_power", "sensor.goodwe_setpoint", + "sensor.goodwe_battery_power", "sensor.goodwe_p1_sample_age"))) +check("...and the first pack's maintenance history SURVIVES the retreat, " + "because `UnitHistory.mirror_flat` keeps the flat keys live - so the " + "retreat itself does not lose state", + "MAINTENANCE HISTORY LOST" not in out_r.split("THE ROUND TRIP")[0] + .split("THE RETREAT")[-1]) +check("⚠️ ...but the ROUND TRIP does not: a maintenance cycle completed while " + "the install is rolled back is DISCARDED when it re-advances, because " + "`maint_schema_version` is already stamped and the migration skips", + "RETREAT-WINDOW WORK DISCARDED" in out_r) +check("...and that is a leg whose data can reach the failure it names: the " + "retreat window actually contains a completion, written through the " + "shipped `UnitHistory`, at a timestamp the starting fixture does not have", + RR.RETREAT_WINDOW_COMPLETION in out_r + and RR.RETREAT_WINDOW_COMPLETION + not in (FIX / "state-flat-070.json").read_text()) + + +print() +if fails: + print(f"FAILED {len(fails)}:") + for f in fails: + print(f" - {f}") + sys.exit(1) +print("all checks passed") diff --git a/goodwe_controller/tools/rehearsal/flat.yaml b/goodwe_controller/tools/rehearsal/flat.yaml new file mode 100644 index 0000000..b81b063 --- /dev/null +++ b/goodwe_controller/tools/rehearsal/flat.yaml @@ -0,0 +1,22 @@ +# ARCH-02 fixture: THE SHAPE THE LIVE BOX IS IN TODAY (a flat, single-inverter +# config). ⚠️ ILLUSTRATIVE ENTITY NAMES, NOT THE LIVE ONES - the owner runs the +# rehearsal against a copy of the real options blob, and this file exists so +# that the harness has a synthetic transition it can be graded on. +setpoint_entity: number.goodwe_setpoint +soc_entity: sensor.goodwe_battery_soc_pct +batt_entity: sensor.goodwe_battery_power_w +batt_invert: false +max_w: 5000 +bms_soc_floor: 10 +meter_source: 'off' +meter_entity: sensor.p1_active_power +inverters: [] +gain: 0.6 +slew_w: 0 +deadband_w: 15 +target_grid_w: 0 +step_w: 10 +heartbeat_s: 10 +stale_input_s: 15 +maintenance_discharge_w: 2000 +maintenance_charge_w: 2000 diff --git a/goodwe_controller/tools/rehearsal/live-entities-recorded.txt b/goodwe_controller/tools/rehearsal/live-entities-recorded.txt new file mode 100644 index 0000000..d991ef4 --- /dev/null +++ b/goodwe_controller/tools/rehearsal/live-entities-recorded.txt @@ -0,0 +1,34 @@ +# The entity ids this add-on ANNOUNCES at the shipped roster of one. +# +# ⚠️ THIS FILE IS EXTERNAL DATA AND THAT IS ITS ENTIRE PURPOSE. The before/after +# entity diff inside `rehearse_roster.py` cannot see a change to the id +# COMPOSITION, because both sides of that diff are composed by the same build: +# rename the MQTT device and every id moves on both sides equally, and the diff +# reports "nothing orphaned" about a change that orphans everything. That is +# `MUTATION.md` habit 4 - a check whose data cannot reach the failure it names - +# and a list that does NOT move with the code is the only thing that closes it. +# +# PROVENANCE, stated exactly: +# - recorded 2026-09-08 by running this tool against `release/1.0` at +# `meter_source: firmware`, roster of one. It is what the code announces. +# - `sensor.goodwe_p1_sample_age` is additionally recorded in `app/mqtt.py` as +# VERIFIED AGAINST THE LIVE SYSTEM on 2026-08-26, and the ESP32's ingest +# guard subscribes to that exact string. +# - ⚠️ IT IS NOT A DUMP OF THE LIVE INSTALL'S ENTITY REGISTRY. Live HA carries +# roughly 49 `goodwe*` entities (card ARCH-02, read 2026-09-06); the other +# ~41 belong to the ESPHome firmware, to template sensors and to helpers, +# and this add-on announces none of them. THE OWNER STEP in DOCS.md replaces +# this file with the real export, and the tool then separates the ids it +# owns from the ids it does not rather than claiming orphans it cannot +# cause. +# +# Editing this file is how you say "the ids changed on purpose". Do not edit it +# to make a check pass. +sensor.goodwe_battery_power +sensor.goodwe_battery_soc +sensor.goodwe_circulating_power +sensor.goodwe_controller_status +sensor.goodwe_grid_power +sensor.goodwe_maintenance_phase +sensor.goodwe_p1_sample_age +sensor.goodwe_setpoint diff --git a/goodwe_controller/tools/rehearsal/roster-inverter-a.yaml b/goodwe_controller/tools/rehearsal/roster-inverter-a.yaml new file mode 100644 index 0000000..13dc3e6 --- /dev/null +++ b/goodwe_controller/tools/rehearsal/roster-inverter-a.yaml @@ -0,0 +1,42 @@ +# ARCH-02 fixture: THE SHAPE AFTER INVERTER B IS COMMISSIONED. +# +# ⚠️ THE FIRST ROSTER ENTRY IS **NOT** CALLED `inverter-a`, AND THAT IS THE +# WHOLE FIXTURE. `MUTATION.md` habit 4's seventh instance is MAINT-05 shipping +# `migrate_maint_keys(store, UNIT_A, ...)` under a comment saying it must not: +# every two-pack fixture in the suite named its first entry `inverter-a`, so +# `UNIT_A` and `inverters[0].name` were the same string in every check that +# existed and the defect was invisible. Names that are not `inverter-a` are what +# make a misdirected namespace something a rehearsal can SEE. +# +# `roster-inverter-a.yaml` beside this file is the same roster with the first +# entry named `inverter-a`, so the two can be compared - see DOCS.md. +setpoint_entity: number.goodwe_setpoint +soc_entity: sensor.goodwe_battery_soc_pct +batt_entity: sensor.goodwe_battery_power_w +batt_invert: false +max_w: 5000 +bms_soc_floor: 10 +meter_source: 'off' +meter_entity: sensor.p1_active_power +gain: 0.6 +slew_w: 0 +deadband_w: 15 +target_grid_w: 0 +step_w: 10 +heartbeat_s: 10 +stale_input_s: 15 +maintenance_discharge_w: 2000 +maintenance_charge_w: 2000 +inverters: + - name: inverter-a + setpoint_entity: number.goodwe_setpoint + soc_entity: sensor.goodwe_battery_soc_pct + batt_entity: sensor.goodwe_battery_power_w + max_w: 5000 + bms_soc_floor: 10 + - name: inverter-b + setpoint_entity: number.goodwe_b_setpoint + soc_entity: sensor.goodwe_b_battery_soc_pct + batt_entity: sensor.goodwe_b_battery_power_w + max_w: 4600 + bms_soc_floor: 10 diff --git a/goodwe_controller/tools/rehearsal/roster.yaml b/goodwe_controller/tools/rehearsal/roster.yaml new file mode 100644 index 0000000..0060f78 --- /dev/null +++ b/goodwe_controller/tools/rehearsal/roster.yaml @@ -0,0 +1,42 @@ +# ARCH-02 fixture: THE SHAPE AFTER INVERTER B IS COMMISSIONED. +# +# ⚠️ THE FIRST ROSTER ENTRY IS **NOT** CALLED `inverter-a`, AND THAT IS THE +# WHOLE FIXTURE. `MUTATION.md` habit 4's seventh instance is MAINT-05 shipping +# `migrate_maint_keys(store, UNIT_A, ...)` under a comment saying it must not: +# every two-pack fixture in the suite named its first entry `inverter-a`, so +# `UNIT_A` and `inverters[0].name` were the same string in every check that +# existed and the defect was invisible. Names that are not `inverter-a` are what +# make a misdirected namespace something a rehearsal can SEE. +# +# `roster-inverter-a.yaml` beside this file is the same roster with the first +# entry named `inverter-a`, so the two can be compared - see DOCS.md. +setpoint_entity: number.goodwe_setpoint +soc_entity: sensor.goodwe_battery_soc_pct +batt_entity: sensor.goodwe_battery_power_w +batt_invert: false +max_w: 5000 +bms_soc_floor: 10 +meter_source: 'off' +meter_entity: sensor.p1_active_power +gain: 0.6 +slew_w: 0 +deadband_w: 15 +target_grid_w: 0 +step_w: 10 +heartbeat_s: 10 +stale_input_s: 15 +maintenance_discharge_w: 2000 +maintenance_charge_w: 2000 +inverters: + - name: garage + setpoint_entity: number.goodwe_setpoint + soc_entity: sensor.goodwe_battery_soc_pct + batt_entity: sensor.goodwe_battery_power_w + max_w: 5000 + bms_soc_floor: 10 + - name: shed + setpoint_entity: number.goodwe_b_setpoint + soc_entity: sensor.goodwe_b_battery_soc_pct + batt_entity: sensor.goodwe_b_battery_power_w + max_w: 4600 + bms_soc_floor: 10 diff --git a/goodwe_controller/tools/rehearsal/state-flat-070.json b/goodwe_controller/tools/rehearsal/state-flat-070.json new file mode 100644 index 0000000..8b9069c --- /dev/null +++ b/goodwe_controller/tools/rehearsal/state-flat-070.json @@ -0,0 +1,11 @@ +{ + "_comment": "ARCH-02 SYNTHETIC FIXTURE - a store that has ALREADY run the flat config under 0.7.0+, so DEV-02's `schema_version` is stamped and the SAFETY-02 latch is not tripped. NOT a copy of the live box's /data/state.json. This is the CLEAN CONTROL for the MAINT-05 half of the rehearsal: the flat->roster transition over it must come out with NO findings, which is what makes each deliberate mutation's RED mean something. A harness that is red on every input has not measured anything.", + "schema_version": 1, + "phase": "idle", + "phase_started": "2026-08-07T10:00:00+00:00", + "last_completed": "2026-08-07T18:42:00+00:00", + "last_start_attempt": "2026-08-07T10:00:00+00:00", + "sign_tripped": false, + "sign_reason": "", + "auto": true +} diff --git a/goodwe_controller/tools/rehearsal/state-legacy.json b/goodwe_controller/tools/rehearsal/state-legacy.json new file mode 100644 index 0000000..ba75d1a --- /dev/null +++ b/goodwe_controller/tools/rehearsal/state-legacy.json @@ -0,0 +1,10 @@ +{ + "_comment": "ARCH-02 SYNTHETIC FIXTURE - a PRE-DEV-02 flat state file. NOT a copy of the live box's /data/state.json; this session cannot read that one and did not try. It is built to carry, deliberately, the two shapes that state built by the happy path does not have: a LATCHED SAFETY-02 failsafe under the flat spelling, and a real maintenance history under the flat spelling. Both are LEGACY-SHAPED: there is no schema_version and no maint_schema_version, so this file was last written by a controller that had never heard of unit namespaces. Losing either one is silent.", + "phase": "idle", + "phase_started": "2026-08-07T10:00:00+00:00", + "last_completed": "2026-08-07T18:42:00+00:00", + "last_start_attempt": "2026-08-07T10:00:00+00:00", + "sign_tripped": true, + "sign_reason": "SIGN_DISAGREEMENT: commanded -1800 W against a meter reading -1450 W for 190 s", + "auto": true +} diff --git a/goodwe_controller/tools/rehearse_roster.py b/goodwe_controller/tools/rehearse_roster.py new file mode 100644 index 0000000..5ff9f20 --- /dev/null +++ b/goodwe_controller/tools/rehearse_roster.py @@ -0,0 +1,840 @@ +"""ARCH-02: rehearse the flat -> roster switch against a COPY of real state. + + python3 tools/rehearse_roster.py --state \ + --before --after + +WHAT THIS IS FOR, in one sentence: on the day inverter B is commissioned the +add-on stops reading the flat single-inverter options and starts reading a +declared `inverters:` roster, and two things happen at that restart that are +ONE-WAY and INVISIBLE UNTIL AFTERWARDS - the store migration renames the keys +that hold a safety latch and a month of maintenance history, and the entity ids +Home Assistant composes change shape. + + THE STORE. /data/state.json on the live box was written by a flat-config + controller. A latch that migrates to a name nothing reads is a + SAFETY GUARD THAT COMES UP DISARMED on an install whose state + file says it is protected (store.py's own header, and SAFETY-02: + "a latch that a restart clears is not a latch"). MAINT-05 had to + fix exactly this shape once already. + + THE ENTITIES. Home Assistant keeps the entity_id it composed at FIRST + discovery. An id that stops being announced is not deleted - it + is ORPHANED: history stays behind on the dead id, and every + automation whose trigger names it SILENTLY STOPS FIRING. An + automation with a vanished trigger entity does not error. It + never runs again. + +⚠️ THIS TOOL RUNS THE REAL `Controller`, NOT A MODEL OF IT. `snapshot()` builds +`app.main.Controller` over the real `app.store.Store`, so the migration under +test is the shipped one and the readers that are asked "can you still find your +latch" are the shipped readers. A rehearsal that re-implemented the migration +would be comparing a structure to itself, which is `MUTATION.md` habit 4 and +this repository's most-repeated defect. + +⚠️ IT NEVER WRITES TO THE FILE IT IS GIVEN. `--state` is opened once, copied to +a scratch file made with `mkstemp()`, and every `Store.save()` in the rehearsal +lands on the copy. The scratch file is unique per invocation, which is habit 18. + +⚠️ IT DOES NOT TOUCH THE LIVE BOX AND CANNOT. There is no SSH, no Supervisor +call and no HTTP in this file. Fetching the state file is an OWNER STEP and it +is written down in DOCS.md under "Rehearsing the roster switch (ARCH-02)". + +EXIT CODES + 0 the transition is clean: nothing vanished, nothing was stranded under a + name no machine reads, no entity was orphaned, no history was lost. + 1 ⭐ AT LEAST ONE OF THOSE HAPPENED. This is the code that must be + reachable: `test_rehearse.py` mutates the shipped migration five + different ways and requires this exit code for each. + 2 the rehearsal could not be performed (bad file, unparseable config, the + entity-id model failed its own anchor). ⚠️ NOT A PASS. +""" + +import argparse +import json +import os +import shutil +import sys +import tempfile +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from app.inverter import UNIT_A, UNIT_STORE_KEYS, Inverter, roster_names # noqa: E402 +from app.main import Controller # noqa: E402 +from app import mqtt # noqa: E402 +from app.p1 import is_enabled # noqa: E402 +from app.store import (DEFAULTS, MAINT_UNIT_KEYS, Store, # noqa: E402 + unit_key) + +EXIT_OK, EXIT_FINDING, EXIT_COULD_NOT_RUN = 0, 1, 2 + + +class Refused(Exception): + """The rehearsal could not be performed. Exit 2, never exit 0.""" + + +# --------------------------------------------------------------------------- # +# 1. THE CONFIGS +# --------------------------------------------------------------------------- # +# The live box's options blob is JSON (`/data/options.json`); the card's command +# line spells the two configs as `.yaml` because that is what an operator edits. +# Both are accepted. ⚠️ THE YAML READER REFUSES WHAT IT DOES NOT UNDERSTAND +# RATHER THAN GUESSING: a config silently mis-parsed here would produce a +# rehearsal of a configuration nobody is going to deploy, which is worse than no +# rehearsal at all. PyYAML is used when it is importable; the add-on's check +# image has no `yaml` module, so the subset below is what actually runs in CI. + +def _scalar(text: str): + raw = text.strip() + if raw.startswith("#"): + raise Refused("comment where a value was expected") + if (raw.startswith('"') and raw.endswith('"') and len(raw) >= 2) or \ + (raw.startswith("'") and raw.endswith("'") and len(raw) >= 2): + return raw[1:-1] + low = raw.lower() + if low in ("true", "yes", "on"): + return True + if low in ("false", "no", "off"): + return False + if low in ("null", "~", ""): + return None + if raw == "[]": + return [] + if raw == "{}": + return {} + try: + return int(raw) + except ValueError: + pass + try: + return float(raw) + except ValueError: + pass + if raw.startswith("[") or raw.startswith("{"): + raise Refused(f"inline collection {raw!r} is not supported by the " + "built-in reader - install PyYAML or use JSON") + return raw + + +def _mini_yaml(text: str) -> dict: + """The subset an add-on options file is written in, and nothing else. + + Block mappings, and lists of block mappings (`inverters:`). Two indent + levels of nesting, which is all `config.yaml`'s option surface has. Every + other construct raises `Refused`. + """ + root: dict = {} + cur_list = None + cur_entry = None + list_indent = None + for lineno, line in enumerate(text.splitlines(), 1): + if not line.strip() or line.lstrip().startswith("#"): + continue + indent = len(line) - len(line.lstrip(" ")) + if "\t" in line[:indent]: + raise Refused(f"line {lineno}: tab in indentation") + body = line.strip() + try: + if body.startswith("- "): + if cur_list is None: + raise Refused(f"line {lineno}: list item outside a key") + if list_indent is None: + list_indent = indent + if indent != list_indent: + raise Refused(f"line {lineno}: ragged list indentation") + cur_entry = {} + cur_list.append(cur_entry) + item = body[2:].strip() + if item: + k, _, v = item.partition(":") + if not _: + raise Refused(f"line {lineno}: bare list scalar") + cur_entry[k.strip()] = _scalar(v) + continue + if ":" not in body: + raise Refused(f"line {lineno}: {body!r} is not `key: value`") + key, _, value = body.partition(":") + key, value = key.strip(), value.strip() + if cur_entry is not None and list_indent is not None \ + and indent > list_indent: + cur_entry[key] = _scalar(value) + continue + cur_list = cur_entry = None + list_indent = None + if value == "": + cur_list = root[key] = [] + continue + root[key] = _scalar(value) + except Refused: + raise + # A key written `foo:` with nothing under it is an EMPTY list here, which is + # exactly what `inverters: []` means to `roster_entries()`. + return root + + +def load_options(path: str) -> dict: + text = Path(path).read_text(encoding="utf-8") + try: + obj = json.loads(text) + except ValueError: + try: + import yaml # noqa: PLC0415 + except ImportError: + obj = _mini_yaml(text) + else: + obj = yaml.safe_load(text) + if not isinstance(obj, dict): + raise Refused(f"{path}: options must be a mapping, got {type(obj).__name__}") + return obj + + +# --------------------------------------------------------------------------- # +# 2. THE STORE, AND THE READ LOG THAT MAKES "NOTHING READS THIS" MEASURABLE +# --------------------------------------------------------------------------- # +class _ReadLog(dict): + """`Store.data`, with every value-read remembered. + + ⚠️ THIS IS THE INSTRUMENT THE WHOLE SAFETY-44 QUESTION TURNS ON. "A key + nothing reads" cannot be established by reasoning about which name the + migration was handed; it is established by asking the SHIPPED READERS to + restore themselves and recording which keys they actually touched. A key in + the file that no reader touched is a key whose value is not information. + + `__contains__` is deliberately NOT recorded: `_move_flat_keys()` tests + `target in data` to decide whether a namespaced value already exists, and + that is a write-side decision, not a machine reading its latch. + """ + + def __init__(self, *a, **kw): + super().__init__(*a, **kw) + self.reads = set() + + def get(self, key, default=None): + self.reads.add(key) + return super().get(key, default) + + def __getitem__(self, key): + self.reads.add(key) + return super().__getitem__(key) + + +class RehearsalStore(Store): + """The SHIPPED `Store`, pointed at a scratch copy of the operator's file. + + ⚠️ THE REAL CLASS, NOT A STAND-IN. `load()`, `save()`, `get_time()` and + `set_time()` are the ones the add-on runs, so the ISO-8601 round trip that + the module docstring says once took the controller down for hours is the + round trip under rehearsal. A `FakeStore` here would rehearse the fake. + """ + + def __init__(self, path: str): + super().__init__(path) + self.data = _ReadLog(self.data) + self.data.reads.clear() + + +def _scratch_copy(state_path: str, tmpdir: str, tag: str) -> str: + """A per-invocation copy. ⚠️ The operator's file is never opened for write.""" + dest = os.path.join(tmpdir, f"state-{tag}.json") + if state_path is None: + Path(dest).write_text("{}", encoding="utf-8") + else: + shutil.copyfile(state_path, dest) + return dest + + +# --------------------------------------------------------------------------- # +# 3. THE ENTITY IDS +# --------------------------------------------------------------------------- # +# ⚠️ THE SLUGIFIER IS DELIBERATELY NOT `mqtt._slug`. Importing the module's own +# spelling would make the model and the subject move together, so a mutation in +# the composition would change both sides equally and this tool would report +# "no orphans" about a rename that orphaned everything - habit 4, written in one +# import. `test_mqtt.py` part 6 makes the same move for the same reason. +import re # noqa: E402 + + +def compose(device_name, entity_name) -> str: + """The entity_id Home Assistant composes, per `app/mqtt.py`'s own note.""" + s = re.sub(r"[^a-z0-9]+", "_", f"{device_name} {entity_name}".lower()) + return "sensor." + s.strip("_") + + +# The ONE id verified against the live system (2026-08-26, recorded in +# `app/mqtt.py`) and subscribed to by the ESP32 ingest guard. If the model above +# cannot reproduce it, the model is wrong and every orphan claim below it is +# worthless - so the tool REFUSES rather than reporting. +ANCHOR_ID = "sensor.goodwe_p1_sample_age" + + +def entity_ids(opts: dict) -> "dict[str, str]": + """`{entity_id: unique_id}` for every row this add-on announces on `opts`. + + ⚠️ `discovery_payloads()` IS THE SHIPPED ANNOUNCER and `roster_names()` is + the shipped naming rule, so the roster shape decision is not restated here. + """ + payloads = mqtt.discovery_payloads( + units=roster_names(opts), + omit=() if is_enabled(opts) else ("p1_age",)) + out = {} + for _topic, cfg in payloads: + out[compose(cfg["device"]["name"], cfg["name"])] = cfg["unique_id"] + return out + + +# The device name and entity name that composed it, recorded as LITERALS. +# ⚠️ LITERALS, NOT `mqtt.DEVICE["name"]`. Anchoring on the shipped dict would +# make this check pass for any renaming of the device - the model and the +# subject would move together and the anchor would be decoration. These two +# strings are what `release/1.0` had on the day the id was verified. +ANCHOR_DEVICE_NAME = "GoodWe" +ANCHOR_ENTITY_NAME = "P1 sample age" + + +def _check_anchor() -> None: + """Is the MODEL right? Nothing about the shipped code is asked here.""" + got = compose(ANCHOR_DEVICE_NAME, ANCHOR_ENTITY_NAME) + if got != ANCHOR_ID: + raise Refused( + f"the entity-id model composes {got!r} where app/mqtt.py records " + f"{ANCHOR_ID!r} as verified against the live system on 2026-08-26. " + "The model is wrong; no orphan claim from it can be believed.") + + +def load_entity_list(path: str) -> set: + """The entity ids that exist on the install, one per line, `#` comments. + + ⚠️ EXTERNAL DATA, AND THE TOOL IS BLIND WITHOUT IT. The before/after diff is + composed twice by the SAME build, so a change to the id composition itself - + a renamed MQTT device, a renamed sensor - moves both sides equally and the + diff reports nothing. `tools/rehearsal/live-entities-recorded.txt` carries + the recorded list and DOCS.md's owner step replaces it with a real export. + """ + out = set() + for line in Path(path).read_text(encoding="utf-8").splitlines(): + line = line.split("#", 1)[0].strip() + if line: + out.add(line) + if not out: + raise Refused(f"{path}: no entity ids in it") + return out + + +# --------------------------------------------------------------------------- # +# 4. ONE CONTROLLER START, OBSERVED +# --------------------------------------------------------------------------- # +class Hass: + """No I/O. `Controller.__init__` performs none; this only has to exist.""" + + async def number(self, entity, invert=False): + return None + + async def limits(self, entity): + return (None, None, None) + + async def set_number(self, entity, value): + return False + + async def call(self, domain, service, data): + return False + + +class Pub: + enabled = False + + def publish(self, *a, **k): + pass + + def close(self): + pass + + +class Snapshot: + """What one `Controller` start over one state file looks like.""" + + def __init__(self, label, opts, store, controller): + self.label = label + self.opts = opts + self.store = store + self.units = [inv.name for inv in controller.inverters] + self.data = dict(store.data) + self.read = set(store.data.reads) + self.entities = entity_ids(opts) + self.latched = {inv.name: bool(inv.sign_wd.tripped) + for inv in controller.inverters} + self.latch_reason = {inv.name: inv.sign_wd.reason + for inv in controller.inverters} + self.flat_tripped = bool(store.data.get("sign_tripped", False)) + self.history = {} + for m in controller.maints: + self.history[m.name] = {k: m.store.get(k) for k in MAINT_UNIT_KEYS} + self.store_migration = controller.store_migration + self.maint_migration = controller.maint_migration + self.machine_read = machine_reads(controller, store) + + @property + def unit_keys(self): + return sorted(k for k in self.data if k.startswith("unit/")) + + +def machine_reads(controller, store) -> set: + """Every store key the CONSTRUCTED machines read, measured not transcribed. + + ⚠️ THE PROBE IS THE SHIPPED CONSTRUCTOR, RUN AGAIN. `Inverter.__init__` is + the code that restores a unit's SAFETY-02 and SAFETY-42 latches, so building + one more `Inverter` per roster entry against the post-migration store + replays EXACTLY the reads a restart performs - including any key this file + has never heard of. Writing out the key names here instead would be a second + transcription of `UNIT_STORE_KEYS`, and the copy that drifts is always the + one that reads (store.py, `unit_key()`). + + The maintenance half iterates `MAINT_UNIT_KEYS`, which is the same tuple + `migrate_maint_keys()` moves and `maintenance.UnitHistory` indexes - so a + key renamed there cannot silently stop being probed here either. + """ + store.data.reads.clear() + for inv in controller.inverters: + Inverter(inv.name, store=store) + for m in controller.maints: + for key in MAINT_UNIT_KEYS: + m.store.get(key) + m.store.get_time(key) + seen = set(store.data.reads) + store.data.reads.clear() + return seen + + +def snapshot(label: str, opts: dict, state_path: str, tmpdir: str) -> Snapshot: + store = RehearsalStore(_scratch_copy(state_path, tmpdir, label)) + controller = Controller(dict(opts), Hass(), store, Pub(), stop=None) + return Snapshot(label, opts, store, controller) + + +def snapshot_over(label: str, opts: dict, prior: Snapshot) -> Snapshot: + """A second start over the store the previous start LEFT BEHIND. + + ⚠️ THE SAME FILE, NOT A SECOND COPY OF THE ORIGINAL. This is the sequence a + real box performs: the add-on restarts against `/data/state.json` as its own + previous run wrote it. + """ + store = RehearsalStore(prior.store.path) + controller = Controller(dict(opts), Hass(), store, Pub(), stop=None) + return Snapshot(label, opts, store, controller) + + +# --------------------------------------------------------------------------- # +# 5. THE DIFF +# --------------------------------------------------------------------------- # +class Finding: + def __init__(self, kind, text): + self.kind = kind + self.text = text + + def __str__(self): + return f"{self.kind}: {self.text}" + + +def diff(before: Snapshot, after: Snapshot) -> list: + """Everything that changed, and which of those changes is a defect.""" + findings = [] + + # -- store keys --------------------------------------------------------- + gone = sorted(set(before.data) - set(after.data)) + for key in gone: + findings.append(Finding( + "VANISHED STORE KEY", + f"{key} = {before.data[key]!r} was in the store before the " + f"transition and is not there afterwards")) + + # ⭐ THE ONE SAFETY-44 IS ABOUT. A key that is present, correct, verified by + # the migration's own read-back - and that no machine on the new roster will + # ever look under. `MigrationReport.misdirected` says the same thing about + # the keys the migration itself wrote; this says it about the WHOLE FILE, + # including keys written by a version that is no longer installed. + for key in sorted(after.data): + if not key.startswith("unit/"): + continue + if key in after.machine_read: + continue + findings.append(Finding( + "STRANDED STORE KEY", + f"{key} = {after.data[key]!r} is in the store and NO machine on " + f"the roster {after.units!r} read it during a full start. Its " + f"value is not information.")) + + # -- the SAFETY-02 latch, by name, because SAFETY-44 asks for it by name -- + for key in ("sign_tripped", "sign_reason"): + findings.extend(_latch_finding(key, before, after)) + + if before.flat_tripped and not any(after.latched.values()): + findings.append(Finding( + "LATCH CAME UP DISARMED", + "the state file carries a LATCHED SAFETY-02 sign-disagreement " + "failsafe under the pre-DEV-02 flat key `sign_tripped`, and NOT " + f"ONE inverter on the roster {after.units!r} came up tripped. " + "SAFETY-02: a latch that a restart clears is not a latch.")) + + # -- maintenance history ------------------------------------------------- + findings.extend(_history_findings(before, after)) + + # -- entity ids ---------------------------------------------------------- + orphans = sorted(set(before.entities) - set(after.entities)) + for eid in orphans: + findings.append(Finding( + "ORPHANED ENTITY", + f"{eid} (unique_id {before.entities[eid]}) is announced before the " + "transition and NOT afterwards. Home Assistant does not delete it: " + "its history stops, and every automation whose trigger names it " + "silently stops firing - without an error.")) + return findings + + +def _latch_finding(key, before, after): + """Did the two SAFETY-44 keys land somewhere a machine reads? + + ⚠️ THE FIRST ROSTER ENTRY, AND ONLY THE FIRST. The flat pair belongs to + whichever inverter the single-inverter install was driving, which is + `inverters[0]` - that is what `migrate_maint_keys()` targets for the history + and what `Controller._grade_store_migration()` re-arms. Asking whether the + flat latch reached inverter B's namespace would report a finding on a + CORRECT migration, and an instrument that is red on the right answer is + worth nothing when it is red on the wrong one. + """ + out = [] + if not after.units or key not in UNIT_STORE_KEYS: + return out + unit = after.units[0] + target = unit_key(unit, key) + was = before.data.get(key) + if target not in after.data and was not in (None, DEFAULTS.get(key)): + out.append(Finding( + "LATCH KEY NOT UNDER A ROSTER NAME", + f"the flat `{key}` = {was!r} did not reach {target}, which is the " + f"key `Inverter({unit!r})` restores from")) + elif target in after.data and target not in after.machine_read: + out.append(Finding( + "LATCH KEY NOT READ BY ANY MACHINE", + f"{target} = {after.data[target]!r} exists but no machine on the " + f"roster {after.units!r} read it during a full start")) + return out + + +def entity_list_findings(recorded: set, before: Snapshot, + after: Snapshot) -> list: + """Grade both snapshots against a list that does NOT move with the code.""" + out = [] + announced_before = set(before.entities) + announced_after = set(after.entities) + for eid in sorted(announced_before - recorded): + out.append(Finding( + "COMPOSITION DRIFT", + f"this build announces {eid} in the BEFORE shape, and that id is " + "not on the recorded list of ids the install has. The id " + "composition has changed under the install: Home Assistant keeps " + "the id it composed at first discovery, so what this actually " + "creates is a NEW entity beside a dead one.")) + owned = recorded & announced_before + for eid in sorted(owned - announced_after): + out.append(Finding( + "ORPHANED ENTITY (against the recorded list)", + f"{eid} exists on the install and is announced in the BEFORE " + "shape, and the AFTER shape does not announce it.")) + return out + + +def report_entity_list(recorded: set, before: Snapshot, after: Snapshot) -> None: + announced_before = set(before.entities) + foreign = sorted(recorded - announced_before) + print() + print(" -- AGAINST THE RECORDED ENTITY LIST ------------------------------") + print(f" {len(recorded)} ids on the list, " + f"{len(recorded & announced_before)} of them announced by this " + f"add-on in the BEFORE shape") + print(f" {len(foreign)} on the list that this add-on announces in " + "NEITHER shape - the roster switch cannot orphan these, they belong " + "to the") + print(" ESPHome firmware, to template sensors or to helpers:") + for eid in foreign[:60]: + print(f" {eid}") + if len(foreign) > 60: + print(f" ... and {len(foreign) - 60} more") + + +def _history_findings(before, after): + """MAINT-05 moved this once. A second move must not drop it.""" + out = [] + if not before.history or not after.history: + return out + was = list(before.history.values())[0] + now = list(after.history.values())[0] + inheritor = list(after.history)[0] + for key in ("phase", "last_completed"): + old, new = was.get(key), now.get(key) + if old in (None, DEFAULTS.get(key)): + continue + if old != new: + out.append(Finding( + "MAINTENANCE HISTORY LOST", + f"`{key}` read {old!r} through the first pack's own history " + f"before the transition and reads {new!r} through " + f"{inheritor!r}'s afterwards")) + return out + + +# --------------------------------------------------------------------------- # +# 6. THE REPORT +# --------------------------------------------------------------------------- # +RETREAT_WINDOW_COMPLETION = "2026-09-20T18:00:00+00:00" + + +def _recalibrate_while_retreated(retreated: Snapshot) -> str: + """A maintenance cycle finishes while the install is rolled back to flat. + + ⚠️ WRITTEN THROUGH THE SHIPPED WRITER. `Maintenance.store` is a + `maintenance.UnitHistory`, which is the one object that decides which key a + pack's history lands under and whether the flat rollback view is mirrored. + A raw `store.set("last_completed", ...)` here would be this file's own + opinion about where a completion goes, and the round trip would then be + grading that opinion instead of the add-on. + """ + store = RehearsalStore(retreated.store.path) + controller = Controller(dict(retreated.opts), Hass(), store, Pub(), stop=None) + controller.maints[0].store.set("last_completed", RETREAT_WINDOW_COMPLETION) + return RETREAT_WINDOW_COMPLETION + + +def _round_trip_findings(completed: str, readvanced: Snapshot) -> list: + """Can the re-advanced roster still see work done during the retreat?""" + out = [] + if not readvanced.history: + return out + inheritor = list(readvanced.history)[0] + seen = readvanced.history[inheritor].get("last_completed") + if seen != completed: + out.append(Finding( + "RETREAT-WINDOW WORK DISCARDED", + f"a maintenance cycle completed at {completed} while the install " + f"was rolled back to the flat config, and after re-advancing to " + f"the roster {inheritor!r} reads `last_completed` = {seen!r}. The " + "pack will be recalibrated again on a schedule that does not know " + "it already was.")) + return out + + +def report_transition(name, before, after, findings) -> None: + print() + print("=" * 78) + print(f" {name}") + print("=" * 78) + print(f" roster before : {before.units}") + print(f" roster after : {after.units}") + print(f" DEV-02 migration : {after.store_migration.explain()}") + print(f" MAINT-05 migration: {after.maint_migration.explain()}") + + print() + print(" -- STORE KEYS ----------------------------------------------------") + keys = sorted(set(before.data) | set(after.data)) + width = max([len(k) for k in keys] + [10]) + print(f" {'key'.ljust(width)} {'before':<26} {'after':<26} status") + for key in keys: + inb, ina = key in before.data, key in after.data + b = repr(before.data[key]) if inb else "-" + a = repr(after.data[key]) if ina else "-" + if inb and not ina: + status = "VANISHED" + elif ina and not inb: + status = "appeared" + elif before.data[key] != after.data[key]: + status = "changed" + else: + status = "" + if key.startswith("unit/") and ina and key not in after.machine_read: + status = (status + " " if status else "") + "STRANDED-nothing-reads-it" + print(f" {key.ljust(width)} {b[:26]:<26} {a[:26]:<26} {status}") + + print() + print(" keys the AFTER roster's machines actually read, measured by " + "replaying") + print(" the shipped constructors against the post-transition store:") + for key in sorted(after.machine_read): + mark = " " if key in after.data else " (absent, defaulted) " + print(f" {key}{mark}") + + print() + print(" -- MAINTENANCE HISTORY (MAINT-05: a second move must not drop it) -") + for label, hist in (("before", before.history), ("after", after.history)): + for unit, vals in hist.items(): + print(f" {label:<7} {unit:<12} " + + " ".join(f"{k}={vals.get(k)!r}" for k in MAINT_UNIT_KEYS)) + + print() + print(" -- SAFETY-02 LATCH (SAFETY-44 asks about these two keys by name) -") + print(f" flat `sign_tripped` in the file : {before.flat_tripped}") + print(f" flat `sign_reason` in the file : " + f"{before.data.get('sign_reason')!r}") + print(f" before: which units came up latched : " + f"{[u for u, v in before.latched.items() if v] or 'none'}") + print(f" after : which units came up latched : " + f"{[u for u, v in after.latched.items() if v] or 'none'}") + for unit in after.units: + target = unit_key(unit, "sign_tripped") + print(f" {target} = {after.data.get(target, '')!r} " + f"(read by a machine: {target in after.machine_read})") + + print() + print(" -- ENTITY IDS ----------------------------------------------------") + allids = sorted(set(before.entities) | set(after.entities)) + for eid in allids: + inb, ina = eid in before.entities, eid in after.entities + mark = "both" if (inb and ina) else ("ORPHANED" if inb else "new") + print(f" {eid:<48} {mark}") + print(f" {len(before.entities)} announced before, " + f"{len(after.entities)} after") + + print() + print(" -- FINDINGS ------------------------------------------------------") + if not findings: + print(" none: nothing vanished, nothing stranded, nothing orphaned, " + "no history lost") + for f in findings: + print(f" ⚠️ {f}") + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--state", required=True, + help="path to a COPY of /data/state.json. Never written to.") + ap.add_argument("--before", required=True, + help="the options in force now (flat single-inverter)") + ap.add_argument("--after", required=True, + help="the options being switched to (declared roster)") + ap.add_argument("--reverse", action="store_true", + help="also rehearse the RETREAT: roster -> flat") + ap.add_argument("--live-entities", default=None, + help="a file of entity ids the install actually has, one " + "per line. ⚠️ WITHOUT IT THE TOOL CANNOT SEE A CHANGE " + "TO THE ID COMPOSITION, because both sides of its own " + "diff are composed by the same build. " + "tools/rehearsal/live-entities-recorded.txt is the " + "recorded default; DOCS.md's owner step replaces it " + "with a real export.") + ap.add_argument("--round-trip", action="store_true", + help="also rehearse RETREAT then RE-ADVANCE, with a " + "maintenance cycle completing while retreated. ⚠️ A " + "retreat is only a retreat if you can come back from " + "it, so the round trip is the leg that decides " + "whether the reverse direction is supported.") + ap.add_argument("--paths", default="both", choices=("staged", "same-step", + "both"), + help="staged: the box runs the BEFORE config once, then " + "restarts on AFTER. same-step: the roster is declared " + "in the SAME step as the upgrade, so the BEFORE " + "controller never runs against this store. " + "Default: both.") + args = ap.parse_args(argv) + + try: + _check_anchor() + before_opts = load_options(args.before) + after_opts = load_options(args.after) + if not Path(args.state).is_file(): + raise Refused(f"{args.state}: not a file") + except Refused as err: + print(f"REFUSED: {err}", file=sys.stderr) + return EXIT_COULD_NOT_RUN + + print(__doc__.split("EXIT CODES")[0].rstrip()) + print() + print(f" state file (copied, never written): {args.state}") + print(f" before config: {args.before}") + print(f" after config: {args.after}") + + findings = [] + recorded = None + try: + if args.live_entities: + recorded = load_entity_list(args.live_entities) + except (OSError, Refused) as err: + print(f"REFUSED: {err}", file=sys.stderr) + return EXIT_COULD_NOT_RUN + tmpdir = tempfile.mkdtemp(prefix="rehearse-roster-") + try: + if args.paths in ("staged", "both"): + b = snapshot("staged-before", before_opts, args.state, tmpdir) + a = snapshot_over("staged-after", after_opts, b) + f = diff(b, a) + if recorded: + f += entity_list_findings(recorded, b, a) + report_transition( + "STAGED: the box runs the flat config, then restarts on the " + "roster", b, a, f) + if recorded: + report_entity_list(recorded, b, a) + print(" (findings from the recorded list are in the list " + "above)") + findings += f + + if args.paths in ("same-step", "both"): + # ⚠️ SAFETY-44's EXACT PATH. A store that ran pre-DEV-02 flat, + # upgraded straight to 0.7.0+, AND declared a multi-inverter roster + # in the same step - so nothing ever ran the flat config under a + # version that knew about namespaces. + b2 = snapshot("samestep-before", before_opts, args.state, tmpdir) + a2 = snapshot("samestep-after", after_opts, args.state, tmpdir) + f2 = diff(b2, a2) + if recorded: + f2 += entity_list_findings(recorded, b2, a2) + report_transition( + "SAME-STEP (SAFETY-44): a pre-DEV-02 flat store meets a " + "declared roster on its FIRST start under this version", + b2, a2, f2) + findings += f2 + + if args.reverse or args.round_trip: + r1 = snapshot("retreat-before", after_opts, args.state, tmpdir) + r2 = snapshot_over("retreat-after", before_opts, r1) + fr = diff(r1, r2) + report_transition( + "THE RETREAT: roster -> flat, which is the back-out if " + "commissioning B goes wrong", r1, r2, fr) + findings += fr + + if args.round_trip: + # ⚠️ THE RETREAT WINDOW HAS TO CONTAIN WORK, OR THIS LEG IS HABIT 4. + # A round trip over a store that nothing changed in between compares + # a structure to itself and reports "clean" whatever the code does. + # So a maintenance cycle COMPLETES while the install is retreated - + # written through `maintenance.UnitHistory`, which is the shipped + # writer, not through a raw `store.set()` - and the question this + # leg asks is whether the pack can still see it after re-advancing. + q1 = snapshot("roundtrip-roster", after_opts, args.state, tmpdir) + q2 = snapshot_over("roundtrip-retreated", before_opts, q1) + worked = _recalibrate_while_retreated(q2) + q3 = snapshot_over("roundtrip-readvanced", after_opts, q2) + fq = diff(q2, q3) + _round_trip_findings(worked, q3) + report_transition( + "THE ROUND TRIP: roster -> flat -> roster, with a maintenance " + f"cycle completing at {worked!r} while retreated", + q2, q3, fq) + findings += fq + except Refused as err: + print(f"REFUSED: {err}", file=sys.stderr) + return EXIT_COULD_NOT_RUN + finally: + shutil.rmtree(tmpdir, ignore_errors=True) + + print() + print("=" * 78) + if findings: + print(f" {len(findings)} FINDING(S) - this transition is not clean") + for f in findings: + print(f" ⚠️ {f}") + print("=" * 78) + return EXIT_FINDING + print(" NO FINDINGS in the rehearsed transitions") + print("=" * 78) + return EXIT_OK + + +if __name__ == "__main__": + sys.exit(main())