CFG-04 fix round: the probe now measures the key it is cited for
checks / unit (pull_request) Successful in 4m33s

Review blocked this on ONE finding, and it was in the EVIDENCE, not the code.
The design and the shipped behaviour are unchanged.

THE INSTRUMENT DID NOT MEASURE WHAT FOUR DOCUMENTS SAID IT MEASURED

tools/supervisor_shape_probe.py section B validated the RECONSTRUCTED LIVE
options blob - which is missing many required keys that have nothing to do with
this card. Its refusal therefore named whichever of THOSE the validator reached
first: the committed transcript recorded `meter_freeze_s`, the reviewer's run
printed `p1_phase_export_entities`, and a run today printed `sign_disagree_w`.
The string those documents quoted - `Missing option 'inverters' in root` - was
never produced by the instrument.

And section B asserted a bare `not ok`. Any refusal at all satisfied it, so
deleting `inverters: []` from config.yaml left it printing PASS: the probe could
not detect the one regression it is the committed evidence for.

THE UNDERLYING CLAIM IS TRUE AND IS NOW MEASURED PROPERLY

Section B is an ISOLATED COUNTERFACTUAL now: the full shipped `options:` block
(expect accept), and the SAME block with `inverters` removed and nothing else
varying (expect refusal). One key differs, so the refusal has nowhere else to
come from - and the assertion is on the KEY NAME IN THE REFUSAL STRING, so the
line goes red if the refusal is ever about a different key. The probe also exits
non-zero when any case fails; without that a transcript is only evidence to
someone who reads it.

The old live-blob case survives as section B2, relabelled for what it is: an
incomplete reconstruction whose RAW half is refused for unrelated missing keys,
kept because its MERGED half genuinely demonstrates what App.options is.

VERIFIED BY BREAKING IT, WHICH IS THE WHOLE POINT

`inverters: []` deleted from config.yaml, probe re-run against
ghcr.io/home-assistant/amd64-hassio-supervisor:2026.08.0:

  == B. THE LOAD-BEARING ONE: an options blob with no `inverters` key ==
    FAIL  the shipped `options:` block validates AS IT STANDS - ...
          Missing option 'inverters' in root in GoodWe RS485 Controller
  ...
  == 2 CASE(S) FAILED ==

exit 1. Key restored, same image, same command: `== all cases passed ==`, exit
0. The transcript in DECISIONS is that restored run, re-run today, not the old
one.

THE FOUR MIS-CITATIONS

DECISIONS/CFG-04-inverter-roster.md section 4's "section B shows both halves"
sentence and its line-103 transcript; CHANGELOG.md; test_inverter.py's part-9
comment and its part-12 mutation row; and the PR description. All now attribute
the string to section B and say plainly what section B2 does not show. The
part-12 row additionally records that the probe was blind to that mutation until
review, because a mutation table that lists a defeat nothing catches is the
failure this repository keeps finding.

MUTATION.md gains it as the sixth instance of "the check's data cannot reach the
failure it names" - the first found OUTSIDE the suite, in a committed evidence
artifact, because nothing in that document had ever been pointed at tools/. Three
habits with it: assert on the CONTENT of a refusal rather than that one happened;
vary ONE thing, from an ACCEPTED baseline; and treat a committed transcript as a
check subject to the mutation battery.

TWO NON-BLOCKING FINDINGS, ALSO FIXED

  validate.py rules 5 and 6 NAMED THE WRONG INVERTER. `unit_caps()` SKIPS an
  entry with no usable `max_w`, so caps position i stops being roster position i
  the moment one entry is malformed - and rule 5 fed `caps.index(weakest)` to
  `_unit_label()`, which indexes the roster, while rule 6 paired `caps[0]` with a
  hard-wired `_unit_label(opts, 0)`. Right number, another inverter's name: an
  operator sent to the wrong device by the message that exists to stop that. New
  `unit_caps_indexed()` carries the roster index with the cap. It is latent -
  rule 9(d) refuses such a config in the same pass - but that was a property of
  today's rule ordering, not of the code, and validate_startup_config() is also
  sim/inject.py's entry point. Part 10 gains a check that asserts the fixture
  really does desynchronise the two lists, then that both messages name the right
  unit, then the counterfactual: the OLD expressions, on that same config, still
  evaluate to the wrong names.

  DOCS.md called a downgrade "safe". True of the SCHEMA - the older schema drops
  the key with a warning - and incomplete about the ADD-ON: rule 9(e) requires the
  flat `setpoint_entity` be cleared when a roster is declared, so a config that
  downgrades cleanly is by construction one whose flat actuator is blank, and the
  older version has nothing to write to. Confirmed by running release/1.0's own
  validate_startup_config() on that blob: one refusal, naming setpoint_entity.
  Recoverable and loud, but an outage until a field is refilled. The note says so
  now, and tells the operator to write the entity down before downgrading.

19/19 green in the goodwe-addon-ci image.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
This commit is contained in:
glenn schrooyen
2026-09-04 10:33:51 +02:00
co-authored by Claude Opus 5
parent 6b85acb0f1
commit 22c016fadd
7 changed files with 276 additions and 43 deletions
+43 -4
View File
@@ -135,9 +135,11 @@ manufactures them from a **fixture**: the check runs, the assertion is
evaluated, the sentence above it is true — and the numbers it compares are
arranged so that the failure it describes could not have changed the answer.
It is not hypothetical and it is not rare. Five instances in two weeks; the
first three were found by review rather than by a red line, and the last two -
CFG-04's - by an author's own mutation battery, which is the cheaper place:
It is not hypothetical and it is not rare. Six instances in two weeks; three
were found by review, two - CFG-04's fixture pair - by an author's own mutation
battery, which is the cheaper place, and the sixth by review again, in the one
place nobody had thought to point the rule at: not a check in the suite, but a
COMMITTED EVIDENCE ARTIFACT four documents cite.
* **ARB-03** shipped a call-site scan asserting that every `claims.append` in
`cycle()` routes through `actuator_of()`. Its own parse **saw 3 of the 8
@@ -165,6 +167,26 @@ CFG-04's - by an author's own mutation battery, which is the cheaper place:
than against its behaviour. It now drives the shipped `run_control()` on a
two-inverter config, **under a deadline**, because the mutation's real symptom
is that the controller enters its loop and never returns.
* **CFG-04's `tools/supervisor_shape_probe.py` §B** — and this one is the rule
reaching outside the suite. The probe is the committed evidence that
`config.yaml`'s `inverters: []` default is load-bearing, and four documents
quoted it as answering *"Missing option 'inverters' in root"*. §B validated
the reconstructed LIVE options blob, which is missing many required keys
unrelated to this card, so its refusal named whichever of THOSE the validator
reached first — `meter_freeze_s`, `p1_phase_export_entities` and
`sign_disagree_w` on three different runs. The quoted string was never
produced by the instrument. And the assertion was a bare **`not ok`**: delete
`inverters: []` from `config.yaml` and §B went on printing **PASS**, blind to
the exact regression it was cited for. §B is now an isolated counterfactual —
the shipped defaults, and the same defaults minus that one key — asserting on
the **key name in the refusal**; the mis-cited blob survives, relabelled, as
§B2. Re-run with the default deleted it prints `2 CASE(S) FAILED` and exits 1.
⚠️ **A committed transcript is a check, and it decays like one.** Nothing in
this document had been pointed at `tools/`, because the artifacts there are
filed as *measurements* rather than as *checks* — and a measurement whose
assertion cannot fail is exactly the thing this section is about. Two habits
follow, and both are habit 5 and 6 below.
Note what these have in common with a masked rung: the mutation *runs*, and the
suite is *honestly* green. Nothing in a purge, a fingerprint or a disassembly
@@ -177,7 +199,7 @@ check was handed.
> **Any check asserting "X differs from Y" must be run once with X and Y
> actually made different.**
Three cheap habits fall out of it, and all three are now used in this repository:
Cheap habits fall out of it, and all of them are now used in this repository:
1. **Write the PRECONDITION as its own check.** `test_peakguard.py` 10g asserts
that A and B have different ratings *before* it asserts that B's bound is
@@ -202,6 +224,23 @@ Three cheap habits fall out of it, and all three are now used in this repository
5. **A check whose failure mode is a HANG needs a deadline.** Otherwise the
mutation that defeats it produces a CI timeout with no name attached, which
reads like flaky infrastructure and gets retried rather than read.
6. **Assert on the CONTENT of a refusal, not on the fact that one happened.**
`not ok` is satisfied by any error at all, including one about a different
key on a different blob. `"Missing option 'inverters' in root" in out` goes
red the moment the refusal stops being about the thing it is cited for. This
is habit 2 in the negative direction: a refusal, like a difference, has to be
asserted against a literal.
7. **Vary ONE thing in a counterfactual, and vary it from an ACCEPTED
baseline.** `supervisor_shape_probe.py` §B validates the shipped defaults
first, precisely so the refusal on the next line has nowhere else to come
from. Without the accepted baseline the second line is a refusal of unknown
provenance, which is what §B used to print.
8. **A committed transcript is a check.** Anything under `tools/` that a
document quotes is subject to every rule above, and to the mutation battery:
break the property, re-run the instrument, and confirm the transcript goes
red. `inverters: []` was deleted from `config.yaml` and the probe re-run
against `ghcr.io/home-assistant/amd64-hassio-supervisor:2026.08.0` for
exactly that reason.
And the mutation-table consequence: **a variant is not covered because a
near-miss of it is.** DEV-02's table listed the *actuator* variant of the