checks / unit (pull_request) Successful in 6m16s
Two ways a deploy of this add-on has gone wrong, and the Supervisor reports both as success. THE SILENT NO-OP. The Supervisor keys the built image by `version:`, so `ha apps update` against an unchanged version installs nothing and says it worked. PKG-06 recorded this on 2026-09-05 with six merges behind an unbumped version; PKG-07 recorded it again on 2026-09-06 with six. Both times a human noticed. Nothing else was going to. THE REFUSED CONFIG. A config the shipped validator rejects is SystemExit(78) - the add-on does not start. ⚠️ AND THE INGRESS PANEL GOES WITH IT, because the add-on serving it is not running, so recovery means the Supervisor UI. ⚠️ IT CAUGHT A LIVE ONE ON ITS OWN PULL REQUEST. Run against release/1.0 while this branch was in review: version 0.8.0 on both sides with UX-02's and OPT-09's runtime commits already merged behind it. The class this exists to catch had recurred, the same day, unnoticed - which is the argument for PKG-10, a non-required post-merge check, since this script only helps someone who remembers to run it. ⚠️ THE FIRST VERSION WAS BROKEN ON EVERY REAL REPOSITORY AND ITS TESTS COULD NOT SEE IT. The "when did the version last change" lookup was `git log -S`, the pickaxe, which finds commits that change the COUNT of a literal string. A bump edits the line IN PLACE, so the count never moves and it returned only the commit that first INTRODUCED the line. Measured here: -S ->0a1e61d2026-08-23 Packaged as a Home Assistant add-on -G ->06661c52026-09-06 PKG-07: bump 0.7.0 -> 0.8.0 so a correctly bumped tree was reported as 87 COMMITS BEHIND AN UNBUMPED VERSION, PKG-06's and PKG-07's own bump commits among the offenders. A tool whose whole purpose is to be trusted cried wolf on every run. ⚠️ AND `-G` IS NOT THE FIX EITHER, ONLY A BETTER FILTER - it matches a reformat, a quote change, a comment moved onto the line. `-G` narrows the candidates and the VALUE is then compared against the parent's. ⚠️ THREE DISTINCT SHAPES OF HABIT 4 IN ONE FILE, each found by a different means, which is the honest record of how this landed: 1. Fixtures built fresh repositories whose version line is introduced once and never edited - the one shape in which `-S` looks correct. Found by review. Answered with a three-state history; the old lookup goes red on 4 checks. 2. That three-state history still could not tell "value-compared" from "filtered-only": review swapped in `-G`'s newest raw hit and THE WHOLE SUITE STAYED GREEN, so the docstring's central claim was pinned by nothing. Answered with a cosmetic touch of the version line - `-G` matches it, the value comparison does not. That mutant now goes red on 3 checks. 3. The counts below were claimed fixed and two of them had not landed at all. Two git shapes it still gets wrong are recorded in the docstring rather than fixed - a bump resolved inside a merge commit (false refusal) and a bump reverted while the work is kept (false clearance, the dangerous direction). Neither has ever occurred here; both were constructed in review. Card PKG-11. ⚠️ AND THE PROSE COUNTS, WHICH I REPORTED AS FIXED WHEN THEY WERE NOT. Round 1 found "PKG-06 ... four merges" (its own commit message says six) and "recorded eleven times" (MUTATION.md:550 says TWELFTH). I replaced them without asserting the replacement matched, said "all applied", and round 2 found both still in the tree - one fixed in DOCS.md but not in the script, one not fixed anywhere. A claim that a correction landed is itself a claim that needs checking, and this is PROC-01's tenth instance. WHAT IT DOES NOT DO. It does not read the live box: the options blob needs the Supervisor API. Without --options the validator half SKIPS - loudly, exit 2 - and never pretends. On a bare host `app/validate.py` needs aiohttp, so that half skips too and names the container to re-run in. A missing or malformed --options file is a REFUSAL with a sentence, not a traceback. EXIT CODES: 0 cleared, 1 REFUSED, 2 NOT CLEARED. ⚠️ 2 IS NOT A PASS. Suite: 26 files, all pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BwRLw7eKYj2byVdwB6kbjU
245 lines
11 KiB
Python
245 lines
11 KiB
Python
"""PKG-09 - grade the check that refuses a no-op deploy. `python3 test_version_gate.py`
|
|
|
|
⚠️ THE POINT OF THIS FILE IS THE REFUSALS, NOT THE CLEARANCES. A pre-deploy
|
|
check that only ever says "CLEARED" is the defect `MUTATION.md` habit 4 names -
|
|
"a check whose DATA cannot reach the failure it names" - which this repository
|
|
has recorded twelve times. So every refusal below is DRIVEN: a repository is
|
|
built in which the failure is really present, and the check is required to say
|
|
so. A clearance is only meaningful next to a refusal from the same harness.
|
|
|
|
The third property is the one that is easy to lose: a SKIPPED check must not
|
|
look like a PASSED one. Omitting `--options` cannot exit 0.
|
|
"""
|
|
import json
|
|
import os
|
|
import shutil
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
|
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
GATE = os.path.join(HERE, "tools", "predeploy_check.py")
|
|
sys.path.insert(0, os.path.join(HERE, "tools"))
|
|
|
|
fails = []
|
|
tmpdirs = []
|
|
|
|
|
|
def check(name, ok):
|
|
print(f" {'ok ' if ok else 'FAIL'} {name}")
|
|
if not ok:
|
|
fails.append(name)
|
|
|
|
|
|
def git(repo, *args):
|
|
return subprocess.run(
|
|
["git", "-C", repo, "-c", "user.email=t@t", "-c", "user.name=t", *args],
|
|
capture_output=True, text=True)
|
|
|
|
|
|
def make_repo(version="0.7.0"):
|
|
"""A repo shaped like this one: <root>/goodwe_controller/{config.yaml,app,tools}."""
|
|
root = tempfile.mkdtemp(prefix="test_version_gate.")
|
|
tmpdirs.append(root)
|
|
addon = os.path.join(root, "goodwe_controller")
|
|
os.makedirs(os.path.join(addon, "app"))
|
|
write(addon, "config.yaml", f'name: t\nversion: "{version}"\n')
|
|
write(addon, "app/control.py", "X = 1\n")
|
|
write(addon, "CHANGELOG.md", "# Changelog\n")
|
|
git(root, "init", "-q", ".")
|
|
git(root, "add", "-A")
|
|
git(root, "commit", "-q", "-m", f"initial at {version}")
|
|
return root, addon
|
|
|
|
|
|
def write(addon, rel, text):
|
|
p = os.path.join(addon, rel)
|
|
with open(p, "w", encoding="utf-8") as fh:
|
|
fh.write(text)
|
|
|
|
|
|
def run_gate(addon, root, deployed, options=None):
|
|
"""Run the REAL gate against a throwaway tree.
|
|
|
|
⚠️ The script is invoked where it SHIPS, not copied into the fixture. It
|
|
imports the validator from its own sibling `app/`, which is the behaviour
|
|
that matters: the gate must check a config against the validator it ships
|
|
beside, not against whatever happens to sit next to a copy. Copying it into
|
|
the fixture made every validator assertion silently SKIP - the first draft
|
|
of this file did exactly that and reported four green skips as passes.
|
|
"""
|
|
cmd = [sys.executable, GATE,
|
|
"--deployed-version", deployed, "--tree", addon, "--repo", root]
|
|
if options:
|
|
cmd += ["--options", options]
|
|
return subprocess.run(cmd, capture_output=True, text=True)
|
|
|
|
|
|
VALID = {"max_w": 2000, "slew_w": 0, "target_grid_w": -10, "gain": 0.6,
|
|
"inverters": [], "setpoint_entity": "number.goodwe_setpoint_w"}
|
|
|
|
|
|
def opts_file(d):
|
|
fd, p = tempfile.mkstemp(suffix=".json")
|
|
with os.fdopen(fd, "w") as fh:
|
|
json.dump(d, fh)
|
|
return p
|
|
|
|
|
|
print("classification - what forces a rebuild and what does not")
|
|
from predeploy_check import is_runtime, shipped_version # noqa: E402
|
|
|
|
for p in ("goodwe_controller/app/control.py", "goodwe_controller/config.yaml",
|
|
"goodwe_controller/translations/en.yaml", "goodwe_controller/run.sh"):
|
|
check(f"runtime: {p}", is_runtime(p))
|
|
for p in ("goodwe_controller/CHANGELOG.md", "goodwe_controller/DOCS.md",
|
|
"goodwe_controller/test_control.py", "goodwe_controller/tools/x.py",
|
|
"goodwe_controller/DECISIONS/a.md", "README.md"):
|
|
check(f"NOT runtime: {p}", not is_runtime(p))
|
|
|
|
print("\nTHE REFUSAL - a runtime change behind an unchanged version")
|
|
root, addon = make_repo("0.7.0")
|
|
write(addon, "app/control.py", "X = 2\n")
|
|
git(root, "commit", "-qam", "SAFETY-99: change behaviour, bump nothing")
|
|
r = run_gate(addon, root, "0.7.0", opts_file(VALID))
|
|
check("refuses (exit 1)", r.returncode == 1)
|
|
check("...and says the deploy would be a NO-OP", "NO-OP" in r.stdout)
|
|
check("...and names the offending commit", "SAFETY-99" in r.stdout)
|
|
check("...and does NOT say cleared", "CLEARED for deploy" not in r.stdout)
|
|
|
|
print("\nPOSITIVE CONTROL - the same check must NOT fire on a docs-only change")
|
|
root2, addon2 = make_repo("0.7.0")
|
|
write(addon2, "CHANGELOG.md", "# Changelog\n\n## notes\n")
|
|
git(root2, "commit", "-qam", "PKG-99: documentation only")
|
|
r2 = run_gate(addon2, root2, "0.7.0", opts_file(VALID))
|
|
check("does not refuse a docs-only tree", r2.returncode == 0)
|
|
check("...and says there is nothing to deploy",
|
|
"nothing to deploy" in r2.stdout)
|
|
|
|
print("\nTHE CLEARANCE - a bumped version with a valid config")
|
|
root3, addon3 = make_repo("0.7.0")
|
|
write(addon3, "app/control.py", "X = 3\n")
|
|
write(addon3, "config.yaml", 'name: t\nversion: "0.8.0"\n')
|
|
git(root3, "commit", "-qam", "PKG-99: change and bump together")
|
|
r3 = run_gate(addon3, root3, "0.7.0", opts_file(VALID))
|
|
check("clears (exit 0)", r3.returncode == 0)
|
|
check("...and says so", "CLEARED for deploy" in r3.stdout)
|
|
|
|
print("\n⭐ A SKIPPED CHECK MUST NOT LOOK LIKE A PASSED ONE")
|
|
r4 = run_gate(addon3, root3, "0.7.0") # no --options
|
|
check("omitting --options cannot exit 0", r4.returncode != 0)
|
|
check("...says SKIPPED", "SKIPPED" in r4.stdout)
|
|
check("...says the config was NEVER CHECKED", "NEVER CHECKED" in r4.stdout)
|
|
check("...and does NOT claim clearance", "CLEARED for deploy" not in r4.stdout)
|
|
|
|
print("\nTHE OTHER REFUSAL - a config this version's validator rejects")
|
|
bad = dict(VALID, slew_w=1000) # rule 10: > 2/18 x 2000 = 222 W
|
|
r5 = run_gate(addon3, root3, "0.7.0", opts_file(bad))
|
|
check("refuses a config the validator rejects", r5.returncode == 1)
|
|
check("...and warns it is exit 78", "78" in r5.stdout)
|
|
check("...and warns the ingress panel goes too", "ingress panel" in r5.stdout)
|
|
check("...and prints the validator's own words", "slew_w is 1000 W" in r5.stdout)
|
|
|
|
print()
|
|
print("⭐ HISTORY WITH MORE THAN ONE BUMP - THE FIXTURE THE FIRST DRAFT LACKED")
|
|
# ⚠️ EVERY FIXTURE ABOVE BUILDS A FRESH REPOSITORY WHOSE `version:` LINE IS
|
|
# INTRODUCED ONCE AND NEVER EDITED. That is precisely the shape in which the
|
|
# original `git log -S 'version: "'` looked correct - the pickaxe finds commits
|
|
# that change a string's COUNT, and introducing the line changes it from 0 to 1.
|
|
# A real repository edits the line IN PLACE, the count never moves again, and
|
|
# the lookup silently returns the commit that first added it.
|
|
#
|
|
# MEASURED on admin/goodwe-addon before the fix: `-S` named `0a1e61d`
|
|
# (2026-08-23, the original packaging commit) as "the last bump", so a correctly
|
|
# bumped tree was reported as 87 commits behind an unbumped version - with
|
|
# PKG-06's and PKG-07's own bump commits listed among the offenders. The tool
|
|
# cried wolf on every run, and none of the fixtures above could see it, because
|
|
# a test that builds its own one-bump history is a test asserting against a
|
|
# structure it built itself. MUTATION.md habit 4, in the file written to
|
|
# prevent a recurrence.
|
|
from predeploy_check import last_version_change # noqa: E402
|
|
|
|
root4 = tempfile.mkdtemp(prefix="test_version_gate.hist.")
|
|
tmpdirs.append(root4)
|
|
addon4 = os.path.join(root4, "goodwe_controller")
|
|
os.makedirs(os.path.join(addon4, "app"))
|
|
write(addon4, "config.yaml", 'name: t\nversion: "0.5.0"\n')
|
|
write(addon4, "app/control.py", "X = 1\n")
|
|
git(root4, "init", "-q", ".")
|
|
git(root4, "add", "-A")
|
|
git(root4, "commit", "-q", "-m", "initial at 0.5.0")
|
|
_first = git(root4, "rev-parse", "HEAD").stdout.strip()
|
|
|
|
write(addon4, "app/control.py", "X = 2\n")
|
|
git(root4, "commit", "-qam", "runtime change between bumps")
|
|
write(addon4, "config.yaml", 'name: t\nversion: "0.6.0"\n')
|
|
git(root4, "commit", "-qam", "bump to 0.6.0")
|
|
write(addon4, "app/control.py", "X = 3\n")
|
|
git(root4, "commit", "-qam", "another runtime change")
|
|
write(addon4, "config.yaml", 'name: t\nversion: "0.7.0"\n')
|
|
git(root4, "commit", "-qam", "bump to 0.7.0")
|
|
_last_bump = git(root4, "rev-parse", "HEAD").stdout.strip()
|
|
|
|
check("⭐ THE LAST BUMP IS FOUND, NOT THE FIRST. An in-place edit of the "
|
|
"version line does not change how many times the string occurs, so a "
|
|
"pickaxe search returns the commit that introduced it and every bump "
|
|
"after that is invisible",
|
|
last_version_change(root4) == _last_bump)
|
|
check("...and it is emphatically not the commit that introduced the line",
|
|
last_version_change(root4) != _first)
|
|
|
|
r6 = run_gate(addon4, root4, "0.7.0", opts_file(VALID))
|
|
check("a tree bumped with nothing after it clears, even with bumps in history",
|
|
r6.returncode == 0 and "CLEARED for deploy" in r6.stdout)
|
|
|
|
write(addon4, "app/control.py", "X = 4\n")
|
|
git(root4, "commit", "-qam", "SAFETY-98: runtime change after the last bump")
|
|
r7 = run_gate(addon4, root4, "0.7.0", opts_file(VALID))
|
|
check("and one runtime commit after that last bump is refused", r7.returncode == 1)
|
|
check("...naming only the commit since the LAST bump, not since the first",
|
|
"SAFETY-98" in r7.stdout and "runtime change between bumps" not in r7.stdout)
|
|
|
|
|
|
# ⚠️ AND THE FIXTURE ABOVE STILL COULD NOT TELL TWO IMPLEMENTATIONS APART.
|
|
# Review swapped `last_version_change` for "-G's newest raw hit, skip the value
|
|
# comparison entirely" and THE WHOLE SUITE STAYED GREEN, including the section
|
|
# above - because a linear history of real bumps never produces a commit that
|
|
# touches the version LINE without changing the version VALUE. So the docstring's
|
|
# central claim ("the VALUE is compared, and that is the only thing that answers
|
|
# it") was not pinned by anything, and `-G` alone would have shipped as if it
|
|
# were the fix. That is the third distinct shape of habit 4 in this one file.
|
|
#
|
|
# A cosmetic touch of the version line is the discriminator: `-G '^version: "'`
|
|
# matches it, and the value comparison does not.
|
|
write(addon4, "config.yaml", 'name: t\nversion: "0.7.0" # pinned deliberately\n')
|
|
git(root4, "commit", "-qam", "COSMETIC: annotate the version line, same value")
|
|
_cosmetic = git(root4, "rev-parse", "HEAD").stdout.strip()
|
|
|
|
check("⭐ A COMMIT THAT TOUCHES THE VERSION LINE WITHOUT CHANGING THE VERSION "
|
|
"IS NOT A BUMP. `-G` matches it and a filter-only lookup would name it; "
|
|
"only comparing the VALUE against the parent's answers the question the "
|
|
"tool is actually asking",
|
|
last_version_change(root4) == _last_bump)
|
|
check("...and it is emphatically not the cosmetic commit",
|
|
last_version_change(root4) != _cosmetic)
|
|
|
|
r8 = run_gate(addon4, root4, "0.7.0", opts_file(VALID))
|
|
check("...and because config.yaml IS a runtime path, that cosmetic commit is "
|
|
"still counted as something unshipped - conservative, and correct: the "
|
|
"Supervisor rebuilds on the version string alone, so a changed file "
|
|
"behind an unchanged version is a no-op whatever changed in it",
|
|
r8.returncode == 1 and "COSMETIC" in r8.stdout)
|
|
|
|
|
|
print("\nthe shipped tree's own version is readable")
|
|
check("shipped_version() finds a version", bool(shipped_version(HERE)))
|
|
|
|
for d in tmpdirs:
|
|
shutil.rmtree(d, ignore_errors=True)
|
|
|
|
print()
|
|
if fails:
|
|
print(f"{len(fails)} FAILED: " + "; ".join(fails))
|
|
sys.exit(1)
|
|
print("all checks passed")
|