TEST-15: purge the bytecode cache before a mutation run, not after
checks / unit (pull_request) Successful in 3m35s

FCST-03's developer reverted `self.provisional` to `here.provisional`, re-ran,
and reported the mutation as a survivor. It was not one. CPython validates a
cached `.pyc` on (source size, source mtime truncated to whole seconds), and
that edit moved neither number - same length, same second - so the interpreter
loaded the bytecode it already had. Disassembly settled it at the time: the
source read `here`, the running code loaded `self`. A mutation run had graded
code that was not there, with no error and exit status 0.

This does not make the results already on the record wrong. It makes them
UNVERIFIED, and that distinction is the whole card - so nothing here re-runs a
campaign and nothing here quietly assumes the old numbers hold.

REPRODUCED, THEN CLOSED. test_bytecode.py builds a real package whose two
return values are the same length, imports it in a real subprocess, rewrites it
inside the same second (retrying until both writes land in one whole second -
milliseconds, no sleep) and reads back which value ran. Part 1 shows the
interpreter running SELF off a file that says HERE.

AND THE TWO OBVIOUS MITIGATIONS DO NOT WORK. Measured, not read off
documentation: `python3 -B` and PYTHONDONTWRITEBYTECODE=1 are write-side only.
Neither stops an EXISTING stale .pyc being read, and a mutation run always has
one, because the previous run wrote it. Part 2 grades both still running the old
code. The advice in test_control.py's coverage audit led with `-B`; it is
corrected here - the purge is the half that works, and `-B` earns its place by
keeping THIS run from leaving a cache for the NEXT edit to inherit.

NO SLEEP ANYWHERE. One-second mtime resolution is the mechanism, not the enemy.
Waiting it out would slow every run down and would still only cover the edit
somebody remembered to slow down.

THE HARNESS. tools/mutation_run.sh purges, prints a sha256/byte fingerprint of
every app/*.py it is about to grade - so a mutation report can say WHICH source
produced it, which is the reviewer's addition to this card - runs each suite
under `python3 -B`, refuses a tree with no test_*.py (CI-01's rule, for CI-01's
reason), and fails if a cache appears during the run. It takes the tree as an
argument so the infra repo's replay harnesses can call it without being moved.

WHERE IT IS SAID (AC 2). MUTATION.md carries the mechanism, the measurement,
the reporting rules and the survey, and every one of the 16 suites now points at
it from its docstring. That list is DERIVED FROM THE TREE by test_bytecode.py
part 5, so a new suite that joins this directory without the pointer is red -
the same failure shape CI-01's file glob exists to prevent.

CI WAS ALREADY IMMUNE, AND IS DESCRIBED THAT WAY (AC 4). Verified rather than
reasoned: with the checkout bind mounted `-v "$WS:/w:ro"` the interpreter writes
no __pycache__ into the tree at all, and it fails SILENTLY - no error, exit
status 0 - while the same probe on a writable mount produces one. Each job also
rm -rf's its workspace and clones afresh, so nothing crosses between jobs on the
shared runner. PYTHONDONTWRITEBYTECODE=1 is added as belt and braces only: the
`:ro` is one character wide and the next person debugging a write will be
tempted by it. Part 6 grades both ON THE DOCKER COMMAND LINE rather than on the
file - the first draft searched the whole workflow and stayed green after the
flag was deleted, because the comment explaining it still named it.

THE SHIPPED ADD-ON IS NOT AFFECTED and the question is closed:
goodwe_controller/Dockerfile has set PYTHONDONTWRITEBYTECODE=1 since it was
written. There is no deploy-side finding.

THE SURVEY (AC 3), in MUTATION.md, scoped BY THE PROPERTY. The defect is
one-directional - a mutation that never took leaves the suite green, so stale
bytecode manufactures SURVIVORS and cannot manufacture a red - which puts every
"all red, no survivors" claim in the window out of scope, along with mutations
executed in-process (SAFETY-07's B1/B2/B3, test_peakguard parts 7 and 10, where
no source file is edited) and everything run after the purge discipline landed
on 2026-08-31 (SAFETY-24, SAFETY-26, OPT-04, FCST-03's re-run). What is left is
the recorded SURVIVORS from source-edit runs before that date. Two were re-run
here under the harness and both still stand: TARIFF-01's record_block() ignoring
its policy argument, and SAFETY-03's abs() on the clamp limit - 15 suites green
each, on the tree as it stood. The rest are labelled still-unverified rather
than assumed or re-run; each is one harness run to close, and the moment to
spend it is when somebody next touches that module.

OUT OF SCOPE AND RECORDED: the infra repo's sim/ replay harnesses import the
add-on and write bytecode into both trees (goodwe/sim/__pycache__ exists today),
so SAFETY-02's headline replay carries the same exposure. mutation_run.sh takes
a tree argument for exactly that, but the change belongs in that repo and some
of those harnesses want the rig.

RED AND GREEN. Every new check was broken and watched fail: the purge commented
out of the harness turns part 4 red (and its non-vacuity probe already proves
the fixture would mis-grade without it); one docstring pointer removed turns
part 5 red naming the file; `:ro` and the `-e` flag removed from the workflow
turn part 6 red. Full suite green, 16 files, in the CI image on a read-only
mount - the shape the runner uses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
glenn schrooyen
2026-09-01 02:07:09 +02:00
co-authored by Claude Opus 5
parent 358fbb0172
commit 7b4877d56d
22 changed files with 755 additions and 10 deletions
+13 -1
View File
@@ -101,10 +101,22 @@ jobs:
# hwsim section was skipped - so "test_freeze.py: PASS" below is a weaker
# claim than it looks. That belongs to whoever owns test_freeze.py next;
# naming it here is the least this gate can do about it.
# ⚠️ PYTHONDONTWRITEBYTECODE=1 IS BELT AND BRACES, NOT A FIX FOR A LIVE
# HOLE. TEST-15 measured this runner's actual exposure and found none: the
# checkout below is bind mounted `:ro`, so the interpreter physically
# cannot write a __pycache__ into the tree it is grading - it fails
# SILENTLY, no error and exit status 0 - and every job `rm -rf`s $WS and
# clones afresh, so nothing survives from a previous job on this shared
# runner either. The variable is here because the `:ro` is the load-bearing
# part and it is one character wide: delete it while debugging a write and
# the gate quietly starts caching bytecode across a rebuild again. Note
# what it does NOT do, per TEST-15 part 2 - it stops bytecode being
# WRITTEN, never a stale one being READ. It is not a substitute for the
# purge in goodwe_controller/tools/mutation_run.sh.
- name: the add-on's own tests
run: |
set -eu
docker run --rm -v "$WS:/w:ro" goodwe-addon-checks:1 sh -eu -c '
docker run --rm -e PYTHONDONTWRITEBYTECODE=1 -v "$WS:/w:ro" goodwe-addon-checks:1 sh -eu -c '
cd /w/goodwe-addon/goodwe_controller
set -- test_*.py
if [ "$1" = "test_*.py" ]; then
+13
View File
@@ -57,6 +57,14 @@ python3 test_control.py && python3 test_maintenance.py && python3 test_arbiter.p
&& python3 test_loop.py
```
⚠️ **Mutating a module to prove a check bites? Read `goodwe_controller/
MUTATION.md` first, and run `goodwe_controller/tools/mutation_run.sh`.** A
same-size edit reverted inside one second changes neither the file size nor the
whole-second mtime, so CPython reuses the bytecode it already compiled and the
suite grades code that is not there - once already, reported as a survivor and
disproved by disassembly (TEST-15). `python3 -B` does **not** save you: it stops
bytecode being written, not a stale `.pyc` being read. The purge is the fix.
`test_watchdog.py` covers SAFETY-02's logic, its arming gate (derived from the
real arbiter, not passed in), its trip path, and the fact that the latch survives
an add-on restart. Its headline validation is not here at all: `sim/replay_sign.py`
@@ -165,6 +173,11 @@ Each of these was found the expensive way. They are in the code comments too.
`timestamp` attribute.
- **Alpine ships paho-mqtt 1.x**, which has no `CallbackAPIVersion`, and musl has
no aiohttp wheel — install from apk, not pip.
- **A mutation run can grade code that is not there.** CPython validates a
`.pyc` on (source size, source mtime in whole seconds), so a same-size edit
reverted within one second is invisible to it and the old bytecode runs. It
manufactures *survivors*, never reds. Purge `__pycache__` -
`tools/mutation_run.sh` does - and see `goodwe_controller/MUTATION.md`.
- **The inverter holds its last command forever.** It has no meter-timeout. Every
failsafe here exists because of that one fact, and anything in doubt commands
**0 W** rather than holding.
+8 -1
View File
@@ -19,6 +19,8 @@ goodwe_controller/ the add-on
test_control.py runnable checks — no framework needed
test_maintenance.py runnable checks — walks a full cycle in fake time
DOCS.md the add-on's documentation tab
MUTATION.md how to run a mutation so the result means something
tools/mutation_run.sh the mutation harness - purges bytecode, then grades
firmware/goodwe-master.yaml ESPHome config for the T-CAN485
estop/rs485_log.py optional RS485 e-stop / bus witness
docs/GoodWe-RS485-Field-Guide.pdf the guide techs carry (10pp, A4)
@@ -82,7 +84,12 @@ python3 test_control.py
python3 test_maintenance.py
```
Both must pass before shipping any change. They are not unit-test theatre —
Both must pass before shipping any change. To prove a check *bites* - the way
almost every claim in this repository was established - mutate the module and
run `tools/mutation_run.sh`, which clears `__pycache__` first. It is not
optional: a same-size edit reverted inside one second leaves CPython reusing the
old bytecode, and the run grades code that is not there. `goodwe_controller/MUTATION.md`
has the mechanism, the measurement, and what the flags do not do. They are not unit-test theatre —
each assertion corresponds to a rule whose absence produced an observed failure
on real hardware, and the comments in `control.py` say which.
+56
View File
@@ -2,6 +2,62 @@
## Unreleased
**A mutation run could grade code that was not there, and the fix is a purge
rather than a flag (TEST-15).** FCST-03's developer reverted `self.provisional`
to `here.provisional`, re-ran, and reported a survivor; disassembly showed the
source read `here` while the running code loaded `self`. CPython validates a
`.pyc` on (source size, source mtime truncated to whole seconds), and a
size-preserving edit reverted inside one second moves neither number, so the
stale bytecode was reused. That does not make past results wrong - it made them
unverified, and the distinction is the whole card.
Reproduced rather than reasoned about. `test_bytecode.py` builds a real package
whose two return values are the same length, imports it in a real subprocess,
rewrites it inside the same second and reads back which value ran. Part 1 shows
the interpreter running `SELF` off a file that says `HERE`.
**And it shows the two obvious mitigations do not work.** `python3 -B` and
`PYTHONDONTWRITEBYTECODE=1` are write-side only: neither stops an EXISTING stale
`.pyc` being read, and a mutation run always has one because the previous run
wrote it. The advice already in `test_control.py`'s coverage audit led with `-B`
and is corrected here - the purge is the half that works, and `-B` earns its
place by protecting the NEXT edit rather than this one. No `sleep` anywhere:
one-second mtime resolution is the mechanism, and waiting it out would slow every
run down while still only covering the edit somebody remembered to slow.
`goodwe_controller/tools/mutation_run.sh` is the harness the house adopts: it
purges, prints a sha256/byte fingerprint of every `app/*.py` it is about to
grade - so a mutation report can say WHICH source produced it - runs each suite
under `python3 -B`, refuses a tree with no `test_*.py` (CI-01's rule), and fails
if a cache appears during the run. It takes the tree as an argument so the infra
repo's replay harnesses can call it.
`MUTATION.md` states the guarantee where a mutation gets written, and every one
of the 16 suites now points at it from its docstring - a list `test_bytecode.py`
derives from the tree, so a new suite joining without the pointer is red.
CI was already immune and is described that way rather than as a hole being
closed: `checks.yml` mounts the checkout `-v "$WS:/w:ro"`, and a read-only tree
cannot take a `__pycache__` at all - measured, and it fails silently, exit
status 0 - while each job `rm -rf`s its workspace and clones afresh.
`PYTHONDONTWRITEBYTECODE=1` is added as belt and braces, because the `:ro` is one
character wide. The shipped add-on was never affected: `Dockerfile` has set
`PYTHONDONTWRITEBYTECODE=1` since it was written.
**The survey of this week's mutation claims is in `MUTATION.md`, scoped by the
property rather than by re-running everything.** The defect is one-directional -
stale bytecode turns a red into a green survivor and never the reverse - so every
"all red, no survivors" result in the window is not at risk, and the exposure is
the recorded SURVIVORS from source-edit runs made before the purge discipline
landed on 2026-08-31. Two of them were re-run here under the harness and both
still stand (TARIFF-01's `record_block()` ignoring its `policy` argument;
SAFETY-03's `abs()` on the clamp limit). The rest are labelled still-unverified
rather than quietly assumed or noisily re-run.
Out of scope and recorded: the infra repo's `sim/` replay harnesses import the
add-on and write bytecode into both trees. Anyone mutating a module graded by a
replay must purge both by hand until that repo adopts the harness.
**`release/1.0` went red at midnight on a commit nobody had touched (TEST-17).**
`Controller.checks()` read the capacity-tariff peak with `self.peak.peak()` - no
argument - so `tariff.py` resolved the month from `datetime.now()`, while
+153
View File
@@ -0,0 +1,153 @@
# Mutation runs in this repository — and the one way they lie
Nearly every safety claim in this add-on was established the same way: change
the shipped module, run the suites, watch one go **red**, put the module back,
watch it go **green**. There is no coverage tool here and there is not meant to
be — the check that matters is whether a *check* notices when the behaviour it
names is removed.
That method has exactly one silent failure mode, and TEST-15 exists because we
walked into it.
## The defect
FCST-03's developer reverted `self.provisional` to `here.provisional`, re-ran,
and reported the mutation as a **survivor**. It was not. CPython decides a
cached `.pyc` is still valid from two numbers:
* the source file's **size in bytes**, and
* the source file's **mtime truncated to whole seconds**.
`self` and `here` are the same length, and the revert happened in the same
second as the edit. Neither number moved, so the interpreter loaded the
bytecode it already had. Confirmed at the time by disassembly: **the source read
`here`, the running code loaded `self`.** The suite had graded code that was not
there — no error, no warning, exit status 0.
The same collision, in the other direction, can leave a **red** standing on
correct source after a revert that never took.
### Which way it lies, and why that scopes everything
A mutation that never took effect leaves the suite **green**, and a green run of
a mutated tree is reported as a survivor. So stale bytecode **manufactures
survivors**; it cannot manufacture a red. The false-red case exists but
announces itself, because a developer chases a failure they cannot explain.
Two consequences, and both are used below:
* a recorded **"all red, no survivors"** result is not at risk from this defect;
* a recorded **survivor** — and a recorded *zero*, such as a coverage-audit
count of 0 — is exactly what this defect produces.
`test_control.py`'s coverage-audit table says the same thing in its own terms:
the error is one-directional, so every figure there is a lower bound at worst.
## The rule
**Purge `__pycache__` before every mutation run.** Not afterwards, not usually,
not when the edit looks big enough.
```sh
cd goodwe_controller
tools/mutation_run.sh # every test_*.py in this directory
tools/mutation_run.sh . test_tariff.py # or just the ones you are watching
tools/mutation_run.sh "$GOODWE_ADDON_REPO/goodwe_controller" # from elsewhere
```
The harness purges, prints a `sha256`/byte-count fingerprint of every `app/*.py`
it is about to grade, runs each suite under `python3 -B`, and fails if a cache
appeared during the run. Diff two fingerprints and the file you mutated is the
line that changed — which is also how a mutation report proves *which source*
produced it.
### What does **not** fix this
| | writes bytecode | reads a stale `.pyc` |
|---|---|---|
| `python3 -B` | no | **yes** |
| `PYTHONDONTWRITEBYTECODE=1` | no | **yes** |
| purging `__pycache__` | (unchanged) | **no — nothing to read** |
Both flags are *write-side only*. A mutation run always has a cache already,
because the previous run wrote it, so a harness that only adds `-B` closes
nothing. This is measured, not read off documentation: `test_bytecode.py`
part 2 arranges the collision and shows both flags still running the old code.
`-B` is still worth passing — it stops **this** run leaving a cache for the
**next** edit to inherit, which is the one nobody is thinking about three
mutations into a batch. Purge for this run, `-B` for the next one.
### And not a `sleep`
One-second mtime resolution is the *mechanism*, not the enemy. Waiting out the
clock would slow every run down, and would still only cover the edit somebody
remembered to slow down. Stop trusting mtime instead.
## Reporting a mutation
State the things a reviewer cannot reconstruct afterwards:
1. **the byte delta** of the mutated file (`+5 B`, `+0 B`). `+0 B` is the risky
one — say so;
2. **that the mutated line was on disk** when the run started (the harness
fingerprint, or a `grep` of the line);
3. **which suite went red, and on which named check** — "the suite is red" and
"the check that watches this is red" are different claims;
4. for a **survivor**, that the run was purged. An unpurged survivor is not a
result.
## Where each surface stands
* **Local and dev-harness runs — the exposed one.** This is where FCST-03 hit
it. `goodwe_controller/__pycache__` and `app/__pycache__` are gitignored and
are sitting in most working trees right now. Use the harness.
* **CI — structurally immune, and it was before this card.** `.gitea/workflows/
checks.yml` bind mounts the checkout `-v "$WS:/w:ro"`, so the interpreter
cannot write a `__pycache__` into the tree under test at all; measured for
TEST-15, and it fails *silently* — no error, exit status 0. Each job also
`rm -rf`s its workspace and clones afresh, so nothing crosses between jobs on
the shared runner. `PYTHONDONTWRITEBYTECODE=1` is now set there too, as belt
and braces: the `:ro` is one character wide and the next person debugging a
write will be tempted by it.
* **The shipped add-on — not affected, and this is closed.**
`goodwe_controller/Dockerfile` has carried `PYTHONDONTWRITEBYTECODE=1` since
it was written. There is no deploy-side finding here; do not re-open it.
* **The infra repo's `sim/` harnesses — exposed, and NOT fixed by this card.**
`sim/replay_sign.py` and friends import the add-on out of
`$GOODWE_ADDON_REPO` and write `__pycache__` into *both* trees
(`goodwe/sim/__pycache__` exists today). SAFETY-02's headline validation is a
replay, not a suite, so a mutation graded through it carries the same
exposure. `tools/mutation_run.sh` takes the tree to purge as its first
argument precisely so that repo can call it, but the change belongs in that
repo and some of those harnesses want the rig. **Anyone mutating a module
graded by a replay must purge both trees by hand until then.**
## Survey — mutation claims of 2026-08-25 … 2026-09-01
Scoped **by the property, not by re-running everything**. A claim is at risk
only if a mutation could have preserved the file's byte count *and* been
reverted inside the same second, *and* the claim is of the kind stale bytecode
can fabricate (see "which way it lies"). Sources: `git log release/1.0` and the
commit and PR bodies.
| claim | verdict | why |
|---|---|---|
| **FCST-03** M9 `self.provisional`→`here.provisional` (dd40c56, 71cd784) | **re-verified** | this is the original false survivor. Re-run with the cache cleared, M9 is caught. The shipped source was correct throughout. |
| **FCST-03** X1, X6 degenerate stimuli (71cd784) | **not at risk** | run after the purge discipline landed in that branch. |
| **SAFETY-24** 11 author + 9 reviewer mutations (e709554, 4601682) | **not at risk** | the commit records them run under `python -B` with a purged `__pycache__` and real byte deltas. Its one survivor is an equivalent mutant, argued structurally rather than by a green run. |
| **SAFETY-26** tolerance sweep over `arbitrage.py` and `static_tariff.py`, 2 survivors (2b8b8cb) | **not at risk** | purge, printed `+5 B` delta, mutated line asserted present, tree re-read before each run. One survivor is explicitly recorded at `+0 B` — the risky delta — and was produced *under* the purge. |
| **OPT-04** mutations reported on PR #22 (bdd7f18) | **not at risk** | `test_arbitrage.py`'s header records the exact command: purge, then `python3 -B`. |
| **TEST-17** `checks()` dropping `when` (504e8da) | **not at risk** | a *red* claim, and dropping an argument changes the file length. Neither half of the collision is available. |
| **SAFETY-07** wiring mutations B1/B2/B3 and `test_peakguard.py` parts 7 and 10 (2d570d8, e7c95bf) | **not at risk** | those mutations are **executed in-process**, monkeypatched onto the imported module. No source file is edited, so no `.pyc` is consulted. |
| every **"all red, no survivors"** claim in the window — TARIFF-02 (24/0), TARIFF-03 (11), MAINT-04 (10), SAFETY-18 (6+4), SAFETY-03 (6), TEL-04 (10), TEL-05 (6) | **not at risk** | direction. Stale bytecode turns a red into a green survivor, never the reverse. A red is evidence that *something* changed under the check. |
| **TEST-06**'s deleted assertion, "no new notification while parked at 0 W" (c34b5a2) | **not at risk / superseded** | it was recorded as undecidable and then *falsified* — `test_freeze.py` records the one-line mutation that takes the file red 9 times. The stronger later claim is a red one. |
| **TARIFF-01** reviewer survivor: `record_block()` ignoring its `policy` argument (0dbd485) | **re-verified — still a survivor** | re-run here through `tools/mutation_run.sh` against the 15 suites that were on the record, all green with `billed_w(block, policy)` mutated to `billed_w(block, POLICY_NET)`. The recorded diagnosis holds: every block reaching the store is single-phase, where NET, SUM and MAX collapse. |
| **SAFETY-03** reviewer survivor: `abs()` dropped from `clamp_to_rating`'s limit (6aabb83) | **re-verified — still a survivor** | re-run the same way, all 15 green. Also as recorded: unreachable under the `int(100, 5000)` schema. |
| **TARIFF-01**'s other three survivors, **SAFETY-03**'s `describe()` survivor, **TARIFF-02**'s M3/M5/M7, **SAFETY-18**'s three | **still unverified** | recorded before the purge discipline, with no byte delta on the record, so a `+0 B` same-second revert cannot be excluded. All are survivors, which is the direction this defect fabricates. TARIFF-02's and SAFETY-18's were **closed by later commits whose claims are reds**, so what is unverified is the *diagnosis*, not the current state of the code. |
**Nothing here says a past result is wrong.** The two survivors re-run above
both stood. What changed is that they are now on the record as measured rather
than assumed, and the rest are labelled honestly. The still-unverified rows are
cheap to close — one harness run each — and the right moment is when somebody
next touches that module, not in a re-run campaign nobody will read.
+7
View File
@@ -3,6 +3,13 @@
Each assertion is a precedence question someone will eventually ask in the
field: "why did it charge during a peak?", "why did it discharge while the
maintenance cycle was charging?", "what happens if two things disagree?".
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import sys
+7
View File
@@ -62,6 +62,13 @@ THREE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
aged out, an hour the series does not cover, an unread state of charge and a
peak guard that never ran are five DIFFERENT reasons - and none of them is
an answer. Section 5 drives all five and asserts they are distinguishable.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import ast
+296
View File
@@ -0,0 +1,296 @@
"""Runnable check for TEST-15 - the mutation harness's own honesty.
`python3 test_bytecode.py`
Every other file in this directory grades the add-on. This one grades the thing
that grades the add-on, because in this repository a mutation run IS the
evidence: "I broke it, the suite went red, I put it back, the suite went green"
is how nearly every safety claim here was established.
THE DEFECT, found by FCST-03's developer and reproduced below. CPython decides a
cached `.pyc` is still good from two numbers: the source's SIZE, and its
MTIME TRUNCATED TO WHOLE SECONDS. Reverting `self.provisional` to
`here.provisional` moves neither - same length, same second - so the interpreter
loaded the bytecode it already had. Confirmed then by disassembly: the source
read `here`, the running code loaded `self`. The mutation was reported as a
survivor when in truth it had never run.
WHICH DIRECTION IT LIES IN. A mutation that never took effect makes the suite
stay GREEN, so it is reported as a survivor: stale bytecode MANUFACTURES
SURVIVORS. The opposite - a revert that never took, leaving a red on correct
source - is possible too but announces itself, because a developer chases an
unexplained failure. That asymmetry is why the coverage-audit table in
test_control.py is a lower bound rather than nonsense, and it is what the survey
in MUTATION.md is scoped by.
WHAT THIS FILE PINS, in the order the card's acceptance criteria name it:
AC1 The defect reproduces (part 1), the two mitigations that DO NOT fix it
are shown not to (part 2), the one that does is shown to (part 3), and
the shipped harness is shown to close it end to end on a tree that would
otherwise mis-grade (part 4). Executed, not reasoned: every part builds a
real package, imports it in a real subprocess, and reads back which
value the running code returned.
AC2 The guarantee is where a developer writing a mutation reads it: part 5
requires EVERY test_*.py in this directory to point at MUTATION.md. The
list is derived from the tree, so a new suite that joins without the
pointer is red - CI-01's rule, for CI-01's reason.
AC4 Part 6 pins the two properties that make CI immune, on the workflow file
itself.
⚠️ NOTHING HERE SLEEPS. One-second mtime resolution is the mechanism; a harness
that waits out the clock would be slower on every run and would still only cover
the edit somebody remembered to slow down. Part 1 REPRODUCES the collision
deliberately (it retries until two writes land in the same second, which takes
milliseconds) rather than avoiding it.
⚠️ This file writes only under a temporary directory. It must stay that way: CI
mounts the checkout READ-ONLY, and a check that needs to write into the tree
would fail there for a reason that has nothing to do with what it grades.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or use
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ. See MUTATION.md.
"""
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
HERE = Path(__file__).resolve().parent
fails = []
def check(name, cond):
if cond:
print(f" ok {name}")
else:
print(f" FAIL {name}")
fails.append(name)
# ---------------------------------------------------------------------------
# The fixture. `who()` returns one of two names that are THE SAME LENGTH, so
# swapping them leaves the file byte count identical - the first half of
# CPython's staleness test. The second half is the whole-second mtime, which
# part 1 arranges by writing both versions inside one second.
#
# It is deliberately not a copy of a shipped module: the property under test is
# CPython's, and using real code here would make this file go red whenever that
# module changed for unrelated reasons.
# ---------------------------------------------------------------------------
V_OLD = 'def who():\n self = "SELF"\n here = "HERE"\n return self\n'
V_NEW = 'def who():\n self = "SELF"\n here = "HERE"\n return here\n'
assert len(V_OLD) == len(V_NEW), "the fixture must be size-preserving or it grades nothing"
PROBE = "from app.m import who\nprint(who())\n"
# The harness runs `test_*.py` files and grades them by exit status, so the
# probe wears that shape in part 4: it FAILS when it sees the pre-mutation value.
TEST_PROBE = (
"from app.m import who\n"
"v = who()\n"
"print('RUNS:', v)\n"
"raise SystemExit(0 if v == 'HERE' else 1)\n"
)
def run(tree, args, env_extra=None, script="probe.py"):
env = dict(os.environ)
env.pop("PYTHONDONTWRITEBYTECODE", None)
if env_extra:
env.update(env_extra)
out = subprocess.run(
[sys.executable] + args + [script],
cwd=tree, env=env, capture_output=True, text=True,
)
return out.stdout.strip()
def caches(tree):
return sorted(str(p.relative_to(tree)) for p in Path(tree).rglob("__pycache__"))
def seed(tree):
"""Leave `tree` holding V_NEW on disk and V_OLD in its bytecode cache.
Returns True when the collision was actually arranged. Both writes must land
in the same whole second; at a few milliseconds per cycle that is the normal
case, and the caller retries rather than asserting on one attempt, because a
machine that happens to cross a second boundary mid-cycle would otherwise
report a defect in CPython instead of a slow disk.
"""
app = Path(tree) / "app"
shutil.rmtree(app, ignore_errors=True)
app.mkdir(parents=True)
(app / "__init__.py").write_text("")
(Path(tree) / "probe.py").write_text(PROBE)
(Path(tree) / "test_probe.py").write_text(TEST_PROBE)
m = app / "m.py"
m.write_text(V_OLD)
t0 = m.stat().st_mtime
run(tree, []) # compiles V_OLD, writes app/__pycache__
m.write_text(V_NEW) # the "revert": same size, same second
t1 = m.stat().st_mtime
return int(t0) == int(t1) and caches(tree) != []
def seeded_tree(stack):
tree = tempfile.mkdtemp(prefix="test_bytecode.")
stack.append(tree)
for _ in range(40):
if seed(tree):
return tree
return None
tmpdirs = []
print("stale bytecode (TEST-15)")
# ---------------------------------------------------------------------------
print("\n1. the defect: a same-size edit inside one second is not seen at all")
# ---------------------------------------------------------------------------
tree = seeded_tree(tmpdirs)
check("the collision can be arranged at all (same size, same second)", tree is not None)
if tree is None:
print(" cannot continue - the fixture never landed two writes in one second")
raise SystemExit(1)
src = (Path(tree) / "app" / "m.py").read_text()
check("the source on disk says HERE", "return here" in src and "return self" not in src)
check("...and a plain interpreter runs SELF anyway", run(tree, []) == "SELF")
# Stated as its own check because this is the whole ticket in one line: what the
# file says and what the process executes have come apart, with no error, no
# warning, and a zero exit status.
# ---------------------------------------------------------------------------
print("\n2. the two mitigations that do NOT close it")
# ---------------------------------------------------------------------------
# ⚠️ READ THIS BEFORE 'FIXING' A HARNESS WITH -B ALONE. Both of these stop
# bytecode being WRITTEN. Neither stops a stale one being READ, and a mutation
# run always has one already: the previous run wrote it. The advice in
# test_control.py's coverage audit said `-B` first and CLEAR the cache second;
# the order is not a style choice, the clear is the half that works.
check("`python3 -B` over an existing cache still runs SELF", run(tree, ["-B"]) == "SELF")
check(
"PYTHONDONTWRITEBYTECODE=1 over an existing cache still runs SELF",
run(tree, [], {"PYTHONDONTWRITEBYTECODE": "1"}) == "SELF",
)
# ---------------------------------------------------------------------------
print("\n3. the purge closes it, and -B keeps the next run clean")
# ---------------------------------------------------------------------------
for p in Path(tree).rglob("__pycache__"):
shutil.rmtree(p, ignore_errors=True)
check("after the purge, the plain interpreter runs HERE", run(tree, []) == "HERE")
check("...and it has written a fresh cache back", caches(tree) != [])
tree2 = seeded_tree(tmpdirs)
check("second fixture seeded", tree2 is not None)
if tree2:
for p in Path(tree2).rglob("__pycache__"):
shutil.rmtree(p, ignore_errors=True)
check("purge + -B runs HERE", run(tree2, ["-B"]) == "HERE")
check("...and leaves NO cache for the next mutation to inherit", caches(tree2) == [])
# The second half is why the harness passes -B at all. Purging before a run
# protects THIS run; not writing one protects the NEXT edit, which is the
# one nobody is thinking about when they are three mutations into a batch.
# ---------------------------------------------------------------------------
print("\n4. the shipped harness, executed against a tree that would mis-grade")
# ---------------------------------------------------------------------------
harness = HERE / "tools" / "mutation_run.sh"
check("tools/mutation_run.sh exists", harness.is_file())
check("...and is executable", os.access(harness, os.X_OK))
tree3 = seeded_tree(tmpdirs)
check("third fixture seeded", tree3 is not None)
if tree3 and harness.is_file():
# NON-VACUITY FIRST. If the naked run below were already green, the harness
# run after it would prove nothing - it would be passing on a tree that
# needed no help. This is the same trap test_peakguard.py part 7 documents:
# a mutation harness that patches nothing passes beautifully.
naked = subprocess.run(
[sys.executable, "test_probe.py"], cwd=tree3, capture_output=True, text=True
)
check("without the harness the tree grades the STALE source (rc!=0)", naked.returncode != 0)
check("...and says so: it saw SELF", "SELF" in naked.stdout)
got = subprocess.run(
["sh", str(harness), tree3], cwd=tree3, capture_output=True, text=True
)
check("the harness grades the source on disk (rc==0)", got.returncode == 0)
check("...having reported the purge", "no bytecode cache" in got.stdout)
check("...and it leaves no cache behind", caches(tree3) == [])
if got.returncode != 0:
print(got.stdout[-2000:])
print(got.stderr[-2000:])
# The harness must also refuse to report on nothing - CI-01's zero-file rule.
empty = tempfile.mkdtemp(prefix="test_bytecode.empty.")
tmpdirs.append(empty)
none = subprocess.run(
["sh", str(harness), empty], cwd=empty, capture_output=True, text=True
)
check("the harness refuses a tree with no test_*.py", none.returncode != 0)
# ---------------------------------------------------------------------------
print("\n5. the guarantee is stated where a mutation gets written")
# ---------------------------------------------------------------------------
# ⚠️ THE LIST IS DERIVED FROM THE TREE. A hand-written one is how a new suite
# joins this directory carrying no warning at all, which is the exact shape of
# failure CI-01's file glob exists to prevent. If you add a test_*.py, put the
# pointer in its docstring; that is the whole cost.
doc = HERE / "MUTATION.md"
check("MUTATION.md exists", doc.is_file())
if doc.is_file():
text = doc.read_text()
check("...and it names the purge as the load-bearing half", "__pycache__" in text and "purge" in text.lower())
check("...and warns that -B alone does not read fresh", "-B" in text)
check("...and points at the harness", "mutation_run.sh" in text)
suites = sorted(p.name for p in HERE.glob("test_*.py"))
check("there are test suites to survey at all", len(suites) >= 15)
missing = [n for n in suites if "MUTATION.md" not in (HERE / n).read_text()]
check(f"every one of the {len(suites)} suites points at MUTATION.md", missing == [])
if missing:
print(" missing the pointer:", ", ".join(missing))
# ---------------------------------------------------------------------------
print("\n6. CI cannot carry a stale artefact between jobs")
# ---------------------------------------------------------------------------
# ⚠️ THE FIRST TWO PROPERTIES ARE THE ONES THAT ACTUALLY DO THE WORK, AND THEY
# PREDATE THIS CARD. Measured for TEST-15, not assumed: with the checkout bind
# mounted READ-ONLY the interpreter writes no __pycache__ into the tree at all,
# and it does so SILENTLY - no error, no warning, exit status 0. The workflow
# also `rm -rf`s its workspace and clones afresh, so nothing survives from a
# previous job even on the same runner. PYTHONDONTWRITEBYTECODE=1 is belt and
# braces on top: it is what keeps this true if somebody ever needs to drop the
# `:ro`, and dropping the `:ro` is a one-character edit that nothing else here
# would notice.
wf = HERE.parent / ".gitea" / "workflows" / "checks.yml"
check("the workflow file is where this expects it", wf.is_file())
if wf.is_file():
y = wf.read_text()
# ⚠️ GRADED ON THE COMMAND LINE, NOT ON THE FILE. A `in y` search over the
# whole workflow passes on the PROSE around the step - this file's first
# draft did exactly that, and stayed green after the `-e` flag and the `:ro`
# were deleted, because the comment explaining them still named both.
runs = [l for l in y.splitlines() if "docker run" in l and ":/w" in l]
check("exactly one docker run mounts the checkout", len(runs) == 1)
line = runs[0] if runs else ""
check("the checkout is mounted read-only", ':/w:ro"' in line)
check("PYTHONDONTWRITEBYTECODE=1 is set on that run", "-e PYTHONDONTWRITEBYTECODE=1" in line)
check("the workspace is removed before each checkout", 'rm -rf "$WS"' in y)
check("no sleep was introduced to dodge the mtime resolution", "sleep " not in y)
for d in tmpdirs:
shutil.rmtree(d, ignore_errors=True)
print()
if fails:
print(f"{len(fails)} FAILED: " + "; ".join(fails))
sys.exit(1)
print("all checks passed")
+7
View File
@@ -47,6 +47,13 @@ WHAT THIS CHECK DOES NOT COVER
asserted here over every value that reached `set_number()`, which is the last
thing this process controls; proving it at the inverter needs the witness log
and belongs with DRIVER-07.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import asyncio
+23 -8
View File
@@ -6,6 +6,13 @@ its absence caused an observed failure on real hardware.
If you change control.py, run this. If it fails, the inverter would have done
something you did not intend.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import sys
@@ -71,14 +78,22 @@ def check(name, cond):
# Hence: A FIXTURE MUST SIT CLEAR OF EVERY RAIL IT IS NOT TESTING. Where a test
# names one mechanism, make that mechanism the binding one (see TCLAMP and TF).
#
# ⚠️ RUN MUTATIONS WITH `python -B` AND CLEAR app/__pycache__. CPython
# invalidates a .pyc on (source mtime in whole seconds, source size), so a
# same-second rewrite that also preserves the file size reuses stale bytecode
# and the suite reports on code you are no longer running. It under-reported one
# mutation here as 2 where the true figure is 6. The error is one-directional -
# stale bytecode can only under-report - so every figure above is a lower bound
# at worst, and the two zeros ever recorded were both confirmed by fixing them
# and watching the count rise, which a caching artefact cannot do.
# ⚠️ RUN MUTATIONS THROUGH `tools/mutation_run.sh`, WHICH CLEARS __pycache__
# BEFORE IT RUNS ANYTHING. CPython invalidates a .pyc on (source mtime in whole
# seconds, source size), so a same-second rewrite that also preserves the file
# size reuses stale bytecode and the suite reports on code you are no longer
# running. It under-reported one mutation here as 2 where the true figure is 6.
#
# ⚠️ AND THE CLEAR IS THE HALF THAT WORKS. This note used to lead with
# `python -B`; TEST-15 measured what that flag actually does, and it is
# write-only - it does not stop an EXISTING stale .pyc being read, and a
# mutation run always has one, because the previous run wrote it. -B is still
# worth passing, for the NEXT edit rather than this one.
#
# The error is one-directional - stale bytecode can only under-report - so
# every figure above is a lower bound at worst, and the two zeros ever recorded
# were both confirmed by fixing them and watching the count rise, which a
# caching artefact cannot do.
# ---------------------------------------------------------------------------
print("control law")
+7
View File
@@ -62,6 +62,13 @@ load the meter's POWER FIGURE never moved - 10130.0 W for 26 consecutive polls -
and only the kWh registers kept `unchanged_s` at zero. See the note in
freeze.py; it is why the fingerprint's register terms are the negative case
rather than a nicety.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import asyncio
+7
View File
@@ -19,6 +19,13 @@ HANDOVER.md. An understated runtime reads as a hang.
`cycle_floor_s` is read from options with no lower bound in code, so a huge
value reproduces the pre-fix behaviour exactly. Every claim below is therefore
an A/B on one binary, not a comparison against a remembered old one.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import asyncio
+7
View File
@@ -3,6 +3,13 @@
Walks a whole cycle in simulated time, which is the test the YAML version never
had - and it is exactly the phase transitions that a real cycle takes ten hours
to exercise once a month.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import sys
+7
View File
@@ -8,6 +8,13 @@ Every assert here is a rule whose absence poisons something downstream: an
inverted sign inverts the control loop, a fabricated zero hides a dead meter, a
naive per-phase sum misbills the capacity tariff, and a plain mean of samples
computes the wrong quarter-hour figure whenever the telegram cadence changes.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import asyncio
+7
View File
@@ -72,6 +72,13 @@ Two days is also not thirty. AC 3's 95 %-over-30-days criterion is TARIFF-01's
and is still not met; `test_tariff.py`'s part 9 says so at length and this file
does not re-litigate it. What part 6 reports is the comparison that two days CAN
support: the same quarters, with and without the guard.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import asyncio
+7
View File
@@ -34,6 +34,13 @@ TWO RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
- the DST checks run on the 23-hour and 25-hour days, not on a normal
24-hour day where a fixed-24-slot implementation and a correct one
agree exactly.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import ast
+7
View File
@@ -48,6 +48,13 @@ THREE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
and a stale one is not a fresh one: the gate blocks on `no_data`, blocks on
an hour the series does not cover, and blocks on a provisional hour - three
different reasons, none of them an answer.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import ast
+7
View File
@@ -45,6 +45,13 @@ THREE RULES THIS FILE HOLDS ITSELF TO, because the reviewer will:
price, and zero is exactly the value that makes everything look free. So
`static_plan` with no profile is checked to produce a read with NO
`.points`, and `price_at` on it must RAISE rather than answer 0.0.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import gzip
+7
View File
@@ -85,6 +85,13 @@ regression fixture that CI cannot see is not a regression fixture.
The recorder's window rolls: 2026-08-01 leaves it around 2026-09-15, which is
why the committed copy is the artefact and not the API call.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import asyncio
+7
View File
@@ -5,6 +5,13 @@ three refusable configurations are refused. The second asserts that everything
on validate.py's "deliberately not refused" list still STARTS - because the way
a validator dies is not by missing a fault, it is by refusing something harmless
until somebody deletes the call.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import asyncio
+7
View File
@@ -8,6 +8,13 @@ person switches it off.
The headline validation is not here - it is `sim/replay_sign.py` in the infra
repo, which drives this module over the two captured bus days. This file is the
table-driven half: boundaries, timers, the latch, and the arming gate.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or run
`tools/mutation_run.sh`, which does. `python3 -B` alone does NOT save you: it
stops bytecode being WRITTEN, not a stale one being READ, and a size-preserving
edit reverted inside one second is invisible to CPython's staleness test. That
is TEST-15, and it has already turned one live mutation into a false survivor.
Read MUTATION.md before you report a survivor.
"""
import sys
+95
View File
@@ -0,0 +1,95 @@
#!/bin/sh
# The house mutation harness (TEST-15). `tools/mutation_run.sh [TREE] [test_*.py ...]`
#
# WHY THIS EXISTS. Nearly every safety claim in this repository was verified by
# mutation: change the shipped module, run the suite, watch it go red, revert,
# watch it go green. FCST-03's developer found that the middle two steps can be
# a lie. CPython decides a `.pyc` is still valid from (source size, source mtime
# TRUNCATED TO WHOLE SECONDS). Revert `self.provisional` to `here.provisional`
# and neither term moves - same length, same second - so the interpreter loads
# the bytecode it already has. Confirmed by disassembly: the source read `here`,
# the running code loaded `self`. The mutation "survived" because it never ran.
#
# ⚠️ `python3 -B` IS NOT THE FIX, AND NEITHER IS PYTHONDONTWRITEBYTECODE=1.
# Both stop bytecode being WRITTEN. Neither stops a stale `.pyc` that is already
# on disk from being READ - measured, and graded by test_bytecode.py part 2.
# The load-bearing half is the PURGE. `-B` is here so the run leaves no cache
# behind for the NEXT mutation to inherit.
#
# ⚠️ NOT A SLEEP. One-second mtime resolution is the mechanism, not the enemy;
# the answer is to stop trusting mtime, not to make every edit wait a second.
# A sleep also only covers the edit you remembered to slow down.
#
# It prints a fingerprint of the tree it is about to grade, because the other
# half of a mutation report is WHICH SOURCE produced it: a report that cannot
# say what was on disk is the same unverifiable claim in a different dress.
# Diff two fingerprints and the mutated file is the line that changed.
set -eu
TREE=""
case "${1-}" in
"" ) : ;;
-h|--help ) sed -n '2,30p' "$0"; exit 0 ;;
* ) if [ -d "$1" ]; then TREE=$1; shift; fi ;;
esac
if [ -z "$TREE" ]; then
# Default: the add-on directory this script ships in, so `tools/mutation_run.sh`
# works from anywhere. An explicit TREE is what lets the infra repo's replay
# harnesses purge $GOODWE_ADDON_REPO before they grade it - see MUTATION.md.
TREE=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
fi
PY=${PY:-python3}
cd "$TREE"
# CI-01's rule, for the same reason: a loop over an empty glob exits 0, and a
# green run that examined nothing is worse than a red one.
if [ $# -eq 0 ]; then set -- test_*.py; fi
if [ ! -f "$1" ]; then
echo "mutation_run: no test_*.py under $TREE - NOTHING has been graded." >&2
exit 1
fi
echo "== purge"
find . -name '__pycache__' -type d -prune -exec rm -rf {} + 2>/dev/null || true
find . -name '*.pyc' -type f -exec rm -f {} + 2>/dev/null || true
left=$(find . -name '__pycache__' -type d -o -name '*.pyc' -type f | head -5)
if [ -n "$left" ]; then
# A read-only tree cannot hold a cache written during this run either, so this
# is only reachable when the purge itself failed on a tree that HAS one - the
# one state where continuing would grade unknown bytecode.
echo "mutation_run: could not purge, and a cache remains:" >&2
echo "$left" >&2
exit 1
fi
echo " no bytecode cache under $TREE"
echo "== tree graded (sha256, bytes)"
find app -name '*.py' 2>/dev/null | sort | while read -r f; do
printf ' %s %8d %s\n' "$(sha256sum "$f" | cut -c1-16)" "$(wc -c < "$f")" "$f"
done
echo "== run ($PY -B)"
rc=0
for f in "$@"; do
start=$(date +%s)
if "$PY" -B "$f" > /tmp/mutation_run.out 2>&1; then
echo " PASS $f ($(( $(date +%s) - start ))s)"
else
echo " FAIL $f ($(( $(date +%s) - start ))s)"
tail -40 /tmp/mutation_run.out
rc=1
fi
done
# -B was asked for; check it was honoured rather than assuming. A cache here
# would be inherited by the next mutation in the batch, which is exactly the
# state this script exists to prevent.
after=$(find . -name '__pycache__' -type d -o -name '*.pyc' -type f | head -5)
if [ -n "$after" ]; then
echo "mutation_run: a bytecode cache appeared DURING the run - the next" >&2
echo "mutation in this batch would inherit it:" >&2
echo "$after" >&2
rc=1
fi
exit $rc