glenn schrooyenandClaude Opus 5
7b4877d56d
checks / unit (pull_request) Successful in 3m35s
FCST-03's developer reverted `self.provisional` to `here.provisional`, re-ran, and reported the mutation as a survivor. It was not one. CPython validates a cached `.pyc` on (source size, source mtime truncated to whole seconds), and that edit moved neither number - same length, same second - so the interpreter loaded the bytecode it already had. Disassembly settled it at the time: the source read `here`, the running code loaded `self`. A mutation run had graded code that was not there, with no error and exit status 0. This does not make the results already on the record wrong. It makes them UNVERIFIED, and that distinction is the whole card - so nothing here re-runs a campaign and nothing here quietly assumes the old numbers hold. REPRODUCED, THEN CLOSED. test_bytecode.py builds a real package whose two return values are the same length, imports it in a real subprocess, rewrites it inside the same second (retrying until both writes land in one whole second - milliseconds, no sleep) and reads back which value ran. Part 1 shows the interpreter running SELF off a file that says HERE. AND THE TWO OBVIOUS MITIGATIONS DO NOT WORK. Measured, not read off documentation: `python3 -B` and PYTHONDONTWRITEBYTECODE=1 are write-side only. Neither stops an EXISTING stale .pyc being read, and a mutation run always has one, because the previous run wrote it. Part 2 grades both still running the old code. The advice in test_control.py's coverage audit led with `-B`; it is corrected here - the purge is the half that works, and `-B` earns its place by keeping THIS run from leaving a cache for the NEXT edit to inherit. NO SLEEP ANYWHERE. One-second mtime resolution is the mechanism, not the enemy. Waiting it out would slow every run down and would still only cover the edit somebody remembered to slow down. THE HARNESS. tools/mutation_run.sh purges, prints a sha256/byte fingerprint of every app/*.py it is about to grade - so a mutation report can say WHICH source produced it, which is the reviewer's addition to this card - runs each suite under `python3 -B`, refuses a tree with no test_*.py (CI-01's rule, for CI-01's reason), and fails if a cache appears during the run. It takes the tree as an argument so the infra repo's replay harnesses can call it without being moved. WHERE IT IS SAID (AC 2). MUTATION.md carries the mechanism, the measurement, the reporting rules and the survey, and every one of the 16 suites now points at it from its docstring. That list is DERIVED FROM THE TREE by test_bytecode.py part 5, so a new suite that joins this directory without the pointer is red - the same failure shape CI-01's file glob exists to prevent. CI WAS ALREADY IMMUNE, AND IS DESCRIBED THAT WAY (AC 4). Verified rather than reasoned: with the checkout bind mounted `-v "$WS:/w:ro"` the interpreter writes no __pycache__ into the tree at all, and it fails SILENTLY - no error, exit status 0 - while the same probe on a writable mount produces one. Each job also rm -rf's its workspace and clones afresh, so nothing crosses between jobs on the shared runner. PYTHONDONTWRITEBYTECODE=1 is added as belt and braces only: the `:ro` is one character wide and the next person debugging a write will be tempted by it. Part 6 grades both ON THE DOCKER COMMAND LINE rather than on the file - the first draft searched the whole workflow and stayed green after the flag was deleted, because the comment explaining it still named it. THE SHIPPED ADD-ON IS NOT AFFECTED and the question is closed: goodwe_controller/Dockerfile has set PYTHONDONTWRITEBYTECODE=1 since it was written. There is no deploy-side finding. THE SURVEY (AC 3), in MUTATION.md, scoped BY THE PROPERTY. The defect is one-directional - a mutation that never took leaves the suite green, so stale bytecode manufactures SURVIVORS and cannot manufacture a red - which puts every "all red, no survivors" claim in the window out of scope, along with mutations executed in-process (SAFETY-07's B1/B2/B3, test_peakguard parts 7 and 10, where no source file is edited) and everything run after the purge discipline landed on 2026-08-31 (SAFETY-24, SAFETY-26, OPT-04, FCST-03's re-run). What is left is the recorded SURVIVORS from source-edit runs before that date. Two were re-run here under the harness and both still stand: TARIFF-01's record_block() ignoring its policy argument, and SAFETY-03's abs() on the clamp limit - 15 suites green each, on the tree as it stood. The rest are labelled still-unverified rather than assumed or re-run; each is one harness run to close, and the moment to spend it is when somebody next touches that module. OUT OF SCOPE AND RECORDED: the infra repo's sim/ replay harnesses import the add-on and write bytecode into both trees (goodwe/sim/__pycache__ exists today), so SAFETY-02's headline replay carries the same exposure. mutation_run.sh takes a tree argument for exactly that, but the change belongs in that repo and some of those harnesses want the rig. RED AND GREEN. Every new check was broken and watched fail: the purge commented out of the harness turns part 4 red (and its non-vacuity probe already proves the fixture would mis-grade without it); one docstring pointer removed turns part 5 red naming the file; `:ro` and the `-e` flag removed from the workflow turn part 6 red. Full suite green, 16 files, in the CI image on a read-only mount - the shape the runner uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
154 lines
9.9 KiB
Markdown
154 lines
9.9 KiB
Markdown
# Mutation runs in this repository — and the one way they lie
|
|
|
|
Nearly every safety claim in this add-on was established the same way: change
|
|
the shipped module, run the suites, watch one go **red**, put the module back,
|
|
watch it go **green**. There is no coverage tool here and there is not meant to
|
|
be — the check that matters is whether a *check* notices when the behaviour it
|
|
names is removed.
|
|
|
|
That method has exactly one silent failure mode, and TEST-15 exists because we
|
|
walked into it.
|
|
|
|
## The defect
|
|
|
|
FCST-03's developer reverted `self.provisional` to `here.provisional`, re-ran,
|
|
and reported the mutation as a **survivor**. It was not. CPython decides a
|
|
cached `.pyc` is still valid from two numbers:
|
|
|
|
* the source file's **size in bytes**, and
|
|
* the source file's **mtime truncated to whole seconds**.
|
|
|
|
`self` and `here` are the same length, and the revert happened in the same
|
|
second as the edit. Neither number moved, so the interpreter loaded the
|
|
bytecode it already had. Confirmed at the time by disassembly: **the source read
|
|
`here`, the running code loaded `self`.** The suite had graded code that was not
|
|
there — no error, no warning, exit status 0.
|
|
|
|
The same collision, in the other direction, can leave a **red** standing on
|
|
correct source after a revert that never took.
|
|
|
|
### Which way it lies, and why that scopes everything
|
|
|
|
A mutation that never took effect leaves the suite **green**, and a green run of
|
|
a mutated tree is reported as a survivor. So stale bytecode **manufactures
|
|
survivors**; it cannot manufacture a red. The false-red case exists but
|
|
announces itself, because a developer chases a failure they cannot explain.
|
|
|
|
Two consequences, and both are used below:
|
|
|
|
* a recorded **"all red, no survivors"** result is not at risk from this defect;
|
|
* a recorded **survivor** — and a recorded *zero*, such as a coverage-audit
|
|
count of 0 — is exactly what this defect produces.
|
|
|
|
`test_control.py`'s coverage-audit table says the same thing in its own terms:
|
|
the error is one-directional, so every figure there is a lower bound at worst.
|
|
|
|
## The rule
|
|
|
|
**Purge `__pycache__` before every mutation run.** Not afterwards, not usually,
|
|
not when the edit looks big enough.
|
|
|
|
```sh
|
|
cd goodwe_controller
|
|
tools/mutation_run.sh # every test_*.py in this directory
|
|
tools/mutation_run.sh . test_tariff.py # or just the ones you are watching
|
|
tools/mutation_run.sh "$GOODWE_ADDON_REPO/goodwe_controller" # from elsewhere
|
|
```
|
|
|
|
The harness purges, prints a `sha256`/byte-count fingerprint of every `app/*.py`
|
|
it is about to grade, runs each suite under `python3 -B`, and fails if a cache
|
|
appeared during the run. Diff two fingerprints and the file you mutated is the
|
|
line that changed — which is also how a mutation report proves *which source*
|
|
produced it.
|
|
|
|
### What does **not** fix this
|
|
|
|
| | writes bytecode | reads a stale `.pyc` |
|
|
|---|---|---|
|
|
| `python3 -B` | no | **yes** |
|
|
| `PYTHONDONTWRITEBYTECODE=1` | no | **yes** |
|
|
| purging `__pycache__` | (unchanged) | **no — nothing to read** |
|
|
|
|
Both flags are *write-side only*. A mutation run always has a cache already,
|
|
because the previous run wrote it, so a harness that only adds `-B` closes
|
|
nothing. This is measured, not read off documentation: `test_bytecode.py`
|
|
part 2 arranges the collision and shows both flags still running the old code.
|
|
|
|
`-B` is still worth passing — it stops **this** run leaving a cache for the
|
|
**next** edit to inherit, which is the one nobody is thinking about three
|
|
mutations into a batch. Purge for this run, `-B` for the next one.
|
|
|
|
### And not a `sleep`
|
|
|
|
One-second mtime resolution is the *mechanism*, not the enemy. Waiting out the
|
|
clock would slow every run down, and would still only cover the edit somebody
|
|
remembered to slow down. Stop trusting mtime instead.
|
|
|
|
## Reporting a mutation
|
|
|
|
State the things a reviewer cannot reconstruct afterwards:
|
|
|
|
1. **the byte delta** of the mutated file (`+5 B`, `+0 B`). `+0 B` is the risky
|
|
one — say so;
|
|
2. **that the mutated line was on disk** when the run started (the harness
|
|
fingerprint, or a `grep` of the line);
|
|
3. **which suite went red, and on which named check** — "the suite is red" and
|
|
"the check that watches this is red" are different claims;
|
|
4. for a **survivor**, that the run was purged. An unpurged survivor is not a
|
|
result.
|
|
|
|
## Where each surface stands
|
|
|
|
* **Local and dev-harness runs — the exposed one.** This is where FCST-03 hit
|
|
it. `goodwe_controller/__pycache__` and `app/__pycache__` are gitignored and
|
|
are sitting in most working trees right now. Use the harness.
|
|
* **CI — structurally immune, and it was before this card.** `.gitea/workflows/
|
|
checks.yml` bind mounts the checkout `-v "$WS:/w:ro"`, so the interpreter
|
|
cannot write a `__pycache__` into the tree under test at all; measured for
|
|
TEST-15, and it fails *silently* — no error, exit status 0. Each job also
|
|
`rm -rf`s its workspace and clones afresh, so nothing crosses between jobs on
|
|
the shared runner. `PYTHONDONTWRITEBYTECODE=1` is now set there too, as belt
|
|
and braces: the `:ro` is one character wide and the next person debugging a
|
|
write will be tempted by it.
|
|
* **The shipped add-on — not affected, and this is closed.**
|
|
`goodwe_controller/Dockerfile` has carried `PYTHONDONTWRITEBYTECODE=1` since
|
|
it was written. There is no deploy-side finding here; do not re-open it.
|
|
* **The infra repo's `sim/` harnesses — exposed, and NOT fixed by this card.**
|
|
`sim/replay_sign.py` and friends import the add-on out of
|
|
`$GOODWE_ADDON_REPO` and write `__pycache__` into *both* trees
|
|
(`goodwe/sim/__pycache__` exists today). SAFETY-02's headline validation is a
|
|
replay, not a suite, so a mutation graded through it carries the same
|
|
exposure. `tools/mutation_run.sh` takes the tree to purge as its first
|
|
argument precisely so that repo can call it, but the change belongs in that
|
|
repo and some of those harnesses want the rig. **Anyone mutating a module
|
|
graded by a replay must purge both trees by hand until then.**
|
|
|
|
## Survey — mutation claims of 2026-08-25 … 2026-09-01
|
|
|
|
Scoped **by the property, not by re-running everything**. A claim is at risk
|
|
only if a mutation could have preserved the file's byte count *and* been
|
|
reverted inside the same second, *and* the claim is of the kind stale bytecode
|
|
can fabricate (see "which way it lies"). Sources: `git log release/1.0` and the
|
|
commit and PR bodies.
|
|
|
|
| claim | verdict | why |
|
|
|---|---|---|
|
|
| **FCST-03** M9 `self.provisional`→`here.provisional` (dd40c56, 71cd784) | **re-verified** | this is the original false survivor. Re-run with the cache cleared, M9 is caught. The shipped source was correct throughout. |
|
|
| **FCST-03** X1, X6 degenerate stimuli (71cd784) | **not at risk** | run after the purge discipline landed in that branch. |
|
|
| **SAFETY-24** 11 author + 9 reviewer mutations (e709554, 4601682) | **not at risk** | the commit records them run under `python -B` with a purged `__pycache__` and real byte deltas. Its one survivor is an equivalent mutant, argued structurally rather than by a green run. |
|
|
| **SAFETY-26** tolerance sweep over `arbitrage.py` and `static_tariff.py`, 2 survivors (2b8b8cb) | **not at risk** | purge, printed `+5 B` delta, mutated line asserted present, tree re-read before each run. One survivor is explicitly recorded at `+0 B` — the risky delta — and was produced *under* the purge. |
|
|
| **OPT-04** mutations reported on PR #22 (bdd7f18) | **not at risk** | `test_arbitrage.py`'s header records the exact command: purge, then `python3 -B`. |
|
|
| **TEST-17** `checks()` dropping `when` (504e8da) | **not at risk** | a *red* claim, and dropping an argument changes the file length. Neither half of the collision is available. |
|
|
| **SAFETY-07** wiring mutations B1/B2/B3 and `test_peakguard.py` parts 7 and 10 (2d570d8, e7c95bf) | **not at risk** | those mutations are **executed in-process**, monkeypatched onto the imported module. No source file is edited, so no `.pyc` is consulted. |
|
|
| every **"all red, no survivors"** claim in the window — TARIFF-02 (24/0), TARIFF-03 (11), MAINT-04 (10), SAFETY-18 (6+4), SAFETY-03 (6), TEL-04 (10), TEL-05 (6) | **not at risk** | direction. Stale bytecode turns a red into a green survivor, never the reverse. A red is evidence that *something* changed under the check. |
|
|
| **TEST-06**'s deleted assertion, "no new notification while parked at 0 W" (c34b5a2) | **not at risk / superseded** | it was recorded as undecidable and then *falsified* — `test_freeze.py` records the one-line mutation that takes the file red 9 times. The stronger later claim is a red one. |
|
|
| **TARIFF-01** reviewer survivor: `record_block()` ignoring its `policy` argument (0dbd485) | **re-verified — still a survivor** | re-run here through `tools/mutation_run.sh` against the 15 suites that were on the record, all green with `billed_w(block, policy)` mutated to `billed_w(block, POLICY_NET)`. The recorded diagnosis holds: every block reaching the store is single-phase, where NET, SUM and MAX collapse. |
|
|
| **SAFETY-03** reviewer survivor: `abs()` dropped from `clamp_to_rating`'s limit (6aabb83) | **re-verified — still a survivor** | re-run the same way, all 15 green. Also as recorded: unreachable under the `int(100, 5000)` schema. |
|
|
| **TARIFF-01**'s other three survivors, **SAFETY-03**'s `describe()` survivor, **TARIFF-02**'s M3/M5/M7, **SAFETY-18**'s three | **still unverified** | recorded before the purge discipline, with no byte delta on the record, so a `+0 B` same-second revert cannot be excluded. All are survivors, which is the direction this defect fabricates. TARIFF-02's and SAFETY-18's were **closed by later commits whose claims are reds**, so what is unverified is the *diagnosis*, not the current state of the code. |
|
|
|
|
**Nothing here says a past result is wrong.** The two survivors re-run above
|
|
both stood. What changed is that they are now on the record as measured rather
|
|
than assumed, and the rest are labelled honestly. The still-unverified rows are
|
|
cheap to close — one harness run each — and the right moment is when somebody
|
|
next touches that module, not in a re-run campaign nobody will read.
|