154 lines
9.9 KiB
Markdown
154 lines
9.9 KiB
Markdown
# Mutation runs in this repository — and the one way they lie
|
|
|
|
Nearly every safety claim in this add-on was established the same way: change
|
|
the shipped module, run the suites, watch one go **red**, put the module back,
|
|
watch it go **green**. There is no coverage tool here and there is not meant to
|
|
be — the check that matters is whether a *check* notices when the behaviour it
|
|
names is removed.
|
|
|
|
That method has exactly one silent failure mode, and TEST-15 exists because we
|
|
walked into it.
|
|
|
|
## The defect
|
|
|
|
FCST-03's developer reverted `self.provisional` to `here.provisional`, re-ran,
|
|
and reported the mutation as a **survivor**. It was not. CPython decides a
|
|
cached `.pyc` is still valid from two numbers:
|
|
|
|
* the source file's **size in bytes**, and
|
|
* the source file's **mtime truncated to whole seconds**.
|
|
|
|
`self` and `here` are the same length, and the revert happened in the same
|
|
second as the edit. Neither number moved, so the interpreter loaded the
|
|
bytecode it already had. Confirmed at the time by disassembly: **the source read
|
|
`here`, the running code loaded `self`.** The suite had graded code that was not
|
|
there — no error, no warning, exit status 0.
|
|
|
|
The same collision, in the other direction, can leave a **red** standing on
|
|
correct source after a revert that never took.
|
|
|
|
### Which way it lies, and why that scopes everything
|
|
|
|
A mutation that never took effect leaves the suite **green**, and a green run of
|
|
a mutated tree is reported as a survivor. So stale bytecode **manufactures
|
|
survivors**; it cannot manufacture a red. The false-red case exists but
|
|
announces itself, because a developer chases a failure they cannot explain.
|
|
|
|
Two consequences, and both are used below:
|
|
|
|
* a recorded **"all red, no survivors"** result is not at risk from this defect;
|
|
* a recorded **survivor** — and a recorded *zero*, such as a coverage-audit
|
|
count of 0 — is exactly what this defect produces.
|
|
|
|
`test_control.py`'s coverage-audit table says the same thing in its own terms:
|
|
the error is one-directional, so every figure there is a lower bound at worst.
|
|
|
|
## The rule
|
|
|
|
**Purge `__pycache__` before every mutation run.** Not afterwards, not usually,
|
|
not when the edit looks big enough.
|
|
|
|
```sh
|
|
cd goodwe_controller
|
|
tools/mutation_run.sh # every test_*.py in this directory
|
|
tools/mutation_run.sh . test_tariff.py # or just the ones you are watching
|
|
tools/mutation_run.sh "$GOODWE_ADDON_REPO/goodwe_controller" # from elsewhere
|
|
```
|
|
|
|
The harness purges, prints a `sha256`/byte-count fingerprint of every `app/*.py`
|
|
it is about to grade, runs each suite under `python3 -B`, and fails if a cache
|
|
appeared during the run. Diff two fingerprints and the file you mutated is the
|
|
line that changed — which is also how a mutation report proves *which source*
|
|
produced it.
|
|
|
|
### What does **not** fix this
|
|
|
|
| | writes bytecode | reads a stale `.pyc` |
|
|
|---|---|---|
|
|
| `python3 -B` | no | **yes** |
|
|
| `PYTHONDONTWRITEBYTECODE=1` | no | **yes** |
|
|
| purging `__pycache__` | (unchanged) | **no — nothing to read** |
|
|
|
|
Both flags are *write-side only*. A mutation run always has a cache already,
|
|
because the previous run wrote it, so a harness that only adds `-B` closes
|
|
nothing. This is measured, not read off documentation: `test_bytecode.py`
|
|
part 2 arranges the collision and shows both flags still running the old code.
|
|
|
|
`-B` is still worth passing — it stops **this** run leaving a cache for the
|
|
**next** edit to inherit, which is the one nobody is thinking about three
|
|
mutations into a batch. Purge for this run, `-B` for the next one.
|
|
|
|
### And not a `sleep`
|
|
|
|
One-second mtime resolution is the *mechanism*, not the enemy. Waiting out the
|
|
clock would slow every run down, and would still only cover the edit somebody
|
|
remembered to slow down. Stop trusting mtime instead.
|
|
|
|
## Reporting a mutation
|
|
|
|
State the things a reviewer cannot reconstruct afterwards:
|
|
|
|
1. **the byte delta** of the mutated file (`+5 B`, `+0 B`). `+0 B` is the risky
|
|
one — say so;
|
|
2. **that the mutated line was on disk** when the run started (the harness
|
|
fingerprint, or a `grep` of the line);
|
|
3. **which suite went red, and on which named check** — "the suite is red" and
|
|
"the check that watches this is red" are different claims;
|
|
4. for a **survivor**, that the run was purged. An unpurged survivor is not a
|
|
result.
|
|
|
|
## Where each surface stands
|
|
|
|
* **Local and dev-harness runs — the exposed one.** This is where FCST-03 hit
|
|
it. `goodwe_controller/__pycache__` and `app/__pycache__` are gitignored and
|
|
are sitting in most working trees right now. Use the harness.
|
|
* **CI — structurally immune, and it was before this card.** `.gitea/workflows/
|
|
checks.yml` bind mounts the checkout `-v "$WS:/w:ro"`, so the interpreter
|
|
cannot write a `__pycache__` into the tree under test at all; measured for
|
|
TEST-15, and it fails *silently* — no error, exit status 0. Each job also
|
|
`rm -rf`s its workspace and clones afresh, so nothing crosses between jobs on
|
|
the shared runner. `PYTHONDONTWRITEBYTECODE=1` is now set there too, as belt
|
|
and braces: the `:ro` is one character wide and the next person debugging a
|
|
write will be tempted by it.
|
|
* **The shipped add-on — not affected, and this is closed.**
|
|
`goodwe_controller/Dockerfile` has carried `PYTHONDONTWRITEBYTECODE=1` since
|
|
it was written. There is no deploy-side finding here; do not re-open it.
|
|
* **The infra repo's `sim/` harnesses — exposed, and NOT fixed by this card.**
|
|
`sim/replay_sign.py` and friends import the add-on out of
|
|
`$GOODWE_ADDON_REPO` and write `__pycache__` into *both* trees
|
|
(`goodwe/sim/__pycache__` exists today). SAFETY-02's headline validation is a
|
|
replay, not a suite, so a mutation graded through it carries the same
|
|
exposure. `tools/mutation_run.sh` takes the tree to purge as its first
|
|
argument precisely so that repo can call it, but the change belongs in that
|
|
repo and some of those harnesses want the rig. **Anyone mutating a module
|
|
graded by a replay must purge both trees by hand until then.**
|
|
|
|
## Survey — mutation claims of 2026-08-25 … 2026-09-01
|
|
|
|
Scoped **by the property, not by re-running everything**. A claim is at risk
|
|
only if a mutation could have preserved the file's byte count *and* been
|
|
reverted inside the same second, *and* the claim is of the kind stale bytecode
|
|
can fabricate (see "which way it lies"). Sources: `git log release/1.0` and the
|
|
commit and PR bodies.
|
|
|
|
| claim | verdict | why |
|
|
|---|---|---|
|
|
| **FCST-03** M9 `self.provisional`→`here.provisional` (dd40c56, 71cd784) | **re-verified** | this is the original false survivor. Re-run with the cache cleared, M9 is caught. The shipped source was correct throughout. |
|
|
| **FCST-03** X1, X6 degenerate stimuli (71cd784) | **not at risk** | run after the purge discipline landed in that branch. |
|
|
| **SAFETY-24** 11 author + 9 reviewer mutations (e709554, 4601682) | **not at risk** | the commit records them run under `python -B` with a purged `__pycache__` and real byte deltas. Its one survivor is an equivalent mutant, argued structurally rather than by a green run. |
|
|
| **SAFETY-26** tolerance sweep over `arbitrage.py` and `static_tariff.py`, 2 survivors (2b8b8cb) | **not at risk** | purge, printed `+5 B` delta, mutated line asserted present, tree re-read before each run. One survivor is explicitly recorded at `+0 B` — the risky delta — and was produced *under* the purge. |
|
|
| **OPT-04** mutations reported on PR #22 (bdd7f18) | **not at risk** | `test_arbitrage.py`'s header records the exact command: purge, then `python3 -B`. |
|
|
| **TEST-17** `checks()` dropping `when` (504e8da) | **not at risk** | a *red* claim, and dropping an argument changes the file length. Neither half of the collision is available. |
|
|
| **SAFETY-07** wiring mutations B1/B2/B3 and `test_peakguard.py` parts 7 and 10 (2d570d8, e7c95bf) | **not at risk** | those mutations are **executed in-process**, monkeypatched onto the imported module. No source file is edited, so no `.pyc` is consulted. |
|
|
| every **"all red, no survivors"** claim in the window — TARIFF-02 (24/0), TARIFF-03 (11), MAINT-04 (10), SAFETY-18 (6+4), SAFETY-03 (6), TEL-04 (10), TEL-05 (6) | **not at risk** | direction. Stale bytecode turns a red into a green survivor, never the reverse. A red is evidence that *something* changed under the check. |
|
|
| **TEST-06**'s deleted assertion, "no new notification while parked at 0 W" (c34b5a2) | **not at risk / superseded** | it was recorded as undecidable and then *falsified* — `test_freeze.py` records the one-line mutation that takes the file red 9 times. The stronger later claim is a red one. |
|
|
| **TARIFF-01** reviewer survivor: `record_block()` ignoring its `policy` argument (0dbd485) | **re-verified — still a survivor** | re-run here through `tools/mutation_run.sh` against the 15 suites that were on the record, all green with `billed_w(block, policy)` mutated to `billed_w(block, POLICY_NET)`. The recorded diagnosis holds: every block reaching the store is single-phase, where NET, SUM and MAX collapse. |
|
|
| **SAFETY-03** reviewer survivor: `abs()` dropped from `clamp_to_rating`'s limit (6aabb83) | **re-verified — still a survivor** | re-run the same way, all 15 green. Also as recorded: unreachable under the `int(100, 5000)` schema. |
|
|
| **TARIFF-01**'s other three survivors, **SAFETY-03**'s `describe()` survivor, **TARIFF-02**'s M3/M5/M7, **SAFETY-18**'s three | **still unverified** | recorded before the purge discipline, with no byte delta on the record, so a `+0 B` same-second revert cannot be excluded. All are survivors, which is the direction this defect fabricates. TARIFF-02's and SAFETY-18's were **closed by later commits whose claims are reds**, so what is unverified is the *diagnosis*, not the current state of the code. |
|
|
|
|
**Nothing here says a past result is wrong.** The two survivors re-run above
|
|
both stood. What changed is that they are now on the record as measured rather
|
|
than assumed, and the rest are labelled honestly. The still-unverified rows are
|
|
cheap to close — one harness run each — and the right moment is when somebody
|
|
next touches that module, not in a re-run campaign nobody will read.
|