glenn schrooyenandClaude Opus 5.5
550bd6fa52
A missing or comment-only .gitea/infra-pin now ends on the workflow's own FAIL line instead of git's `fatal: path ... does not exist` (still red). Every x-access-token remote URL becomes an env-set Authorization header, so no token lands in .git/config or argv; the pin workflows' temp dir is removed by an EXIT trap, also on failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
407 lines
20 KiB
Python
407 lines
20 KiB
Python
"""Runnable check for CI-02 - what a green from `checks / unit` is a claim about.
|
|
`python3 test_ci_gate.py`
|
|
|
|
test_bytecode.py grades the thing that grades the add-on. This file grades the
|
|
thing that decides whether anything was graded at all.
|
|
|
|
THE DEFECT, and the correction the measurement forced. CI-02 was filed saying
|
|
that a CONFLICTED pull request loses its merge ref, so that
|
|
|
|
git fetch -q --depth 1 origin "$GITHUB_REF" \
|
|
|| git fetch -q --depth 1 origin "$GITHUB_SHA"
|
|
|
|
falls through to the second half and quietly grades the bare branch head. PR #14
|
|
did report `checks / unit -> success` while `mergeable: false`, so the symptom
|
|
was real. The mechanism was not. Measured on a deliberately conflicted throwaway
|
|
pull request (#29, runs 558 and 559 - one question per STEP, because this
|
|
Gitea's Actions log endpoints 404 and a step's conclusion is the only channel
|
|
out of a job):
|
|
|
|
GITHUB_REF == refs/pull/<n>/head P01 passed
|
|
GITHUB_REF != refs/pull/<n>/merge P02 failed
|
|
GITHUB_SHA == the pull request head sha P03 passed
|
|
fetching GITHUB_REF succeeds P04 passed <- even conflicted
|
|
...and it lands on the bare head P05 passed
|
|
refs/pull/<n>/merge does not exist P06 failed
|
|
|
|
`git ls-remote origin 'refs/pull/*'` on this server advertises only `/head`, for
|
|
open and closed pull requests alike. THERE ARE NO MERGE REFS HERE. So the first
|
|
half of the `||` always succeeded, the fallback never fired once, and both halves
|
|
named the same commit anyway. The gate has never graded merged content on any
|
|
pull request in this repository's history. Green has meant "this branch passed
|
|
on its own" - which, on a base branch that has moved underneath it, is not the
|
|
content that lands. `block_on_outdated_branch` is off on release/1.0, so the
|
|
base usually has moved.
|
|
|
|
WHAT THIS FILE PINS.
|
|
|
|
Part 1 The context string branch protection requires is assembled from two
|
|
tokens in checks.yml. Renaming either makes every open and future
|
|
pull request unmergeable forever, silently, because Gitea waits on a
|
|
context it will now never receive. This part is the tripwire on that.
|
|
Part 2 The three absences in checks.yml that are load-bearing, each graded
|
|
on the command lines rather than on the prose around them.
|
|
Part 3 THE DEFECT, REPRODUCED. The old one-line checkout is run verbatim
|
|
against a real conflicted pull request built here, and is shown to
|
|
exit 0 having checked out content that cannot land.
|
|
Part 4 THE FIX, EXECUTED. The checkout and merge steps are EXTRACTED FROM
|
|
checks.yml AND RUN - the shipped commands, not a paraphrase - against
|
|
a pull request that merges. The tree they leave behind is the merge,
|
|
and is shown to differ from the head the old line would have graded.
|
|
Part 5 The same two steps against a pull request that does NOT merge: the
|
|
merge step exits non-zero, says nothing was tested, and never prints
|
|
a GRADED line.
|
|
Part 6 The gate announces which content it graded, on both paths, so that a
|
|
reader can tell merged content from a bare ref (CI-02 AC2).
|
|
|
|
⚠️ THIS FILE RUNS THE WORKFLOW'S OWN SHELL. That is the point: a check that
|
|
read those commands as a string and matched them against an expectation would
|
|
be exactly the failure this repository keeps finding - a reading that never
|
|
observed its subject. The extractor below is deliberately brittle about finding
|
|
the step it wants: if checks.yml is restructured so that the step cannot be
|
|
found, this file goes RED rather than quietly grading nothing.
|
|
|
|
⚠️ NEEDS `git`, which is why .gitea/ci/Dockerfile installs it. No network: the
|
|
fixtures are local repositories built in a temp directory; the workflow's
|
|
token header (TOOL-9) is ignored for a local path, which is what lets the
|
|
shipped lines be used verbatim.
|
|
|
|
⚠️ This file writes only under a temporary directory. It must stay that way:
|
|
CI mounts the checkout READ-ONLY.
|
|
|
|
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or use
|
|
`tools/mutation_run.sh`, which does. See MUTATION.md. (There is no module here
|
|
to mutate; the subject is checks.yml. Break a line of it and re-run this file
|
|
expecting red - if it stays green, it was never watching.)
|
|
"""
|
|
|
|
import os
|
|
import shutil
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
from pathlib import Path
|
|
|
|
HERE = Path(__file__).resolve().parent
|
|
WF = HERE.parent / ".gitea" / "workflows" / "checks.yml"
|
|
fails = []
|
|
|
|
|
|
def check(name, cond):
|
|
if cond:
|
|
print(f" ok {name}")
|
|
else:
|
|
print(f" FAIL {name}")
|
|
fails.append(name)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Reading checks.yml. There is no yaml module in the image the checks run in
|
|
# (alpine py3-aiohttp py3-paho-mqtt, and adding a parser to grade five lines
|
|
# would be a worse trade than a small parser that fails loudly), so the two
|
|
# things this file needs are pulled out by hand:
|
|
#
|
|
# * the workflow name and the job keys, for part 1
|
|
# * the `run:` block of a named step, verbatim, for parts 3 to 6
|
|
#
|
|
# COMMAND LINES ONLY WHERE IT MATTERS. test_bytecode.py's part 6 learned this
|
|
# the hard way: an `in y` search over a whole workflow passes on the PROSE that
|
|
# explains a flag, and stayed green after the flag itself was deleted. Every
|
|
# absence checked in part 2 is checked against the file with its comment lines
|
|
# stripped.
|
|
# ---------------------------------------------------------------------------
|
|
if not WF.is_file():
|
|
print(f"cannot find the workflow at {WF} - nothing to grade")
|
|
raise SystemExit(1)
|
|
YAML = WF.read_text()
|
|
CODE = "\n".join(l for l in YAML.splitlines() if not l.lstrip().startswith("#"))
|
|
|
|
|
|
def workflow_name():
|
|
for line in YAML.splitlines():
|
|
if line.startswith("name:"):
|
|
return line.split(":", 1)[1].strip()
|
|
return None
|
|
|
|
|
|
def job_keys():
|
|
out, seen = [], False
|
|
for line in YAML.splitlines():
|
|
if line.startswith("jobs:"):
|
|
seen = True
|
|
continue
|
|
if seen:
|
|
if line and not line[0].isspace():
|
|
break
|
|
if line.startswith(" ") and not line.startswith(" ") and line.rstrip().endswith(":"):
|
|
out.append(line.strip()[:-1])
|
|
return out
|
|
|
|
|
|
def step_script(name_prefix):
|
|
"""Return the `run: |` body of the step whose name starts with name_prefix."""
|
|
lines = YAML.splitlines()
|
|
i = 0
|
|
while i < len(lines):
|
|
if lines[i].strip().startswith("- name:") and name_prefix in lines[i]:
|
|
break
|
|
i += 1
|
|
else:
|
|
return None
|
|
if i >= len(lines):
|
|
return None
|
|
while i < len(lines) and lines[i].strip() != "run: |":
|
|
# stop at the next step so a missing block cannot borrow the next one's
|
|
if i > 0 and lines[i].strip().startswith("- name:") and name_prefix not in lines[i]:
|
|
return None
|
|
i += 1
|
|
if i >= len(lines):
|
|
return None
|
|
body, i = [], i + 1
|
|
while i < len(lines):
|
|
l = lines[i]
|
|
if l.strip() == "":
|
|
body.append("")
|
|
elif l.startswith(" " * 10):
|
|
body.append(l[10:])
|
|
else:
|
|
break
|
|
i += 1
|
|
return "\n".join(body).strip("\n")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# The fixture: a real bare repository with a base branch and a refs/pull/N/head,
|
|
# built to order. `conflict=True` makes the pull request touch the same lines
|
|
# the base moved, which is how PR #29 was made unmergeable on the server.
|
|
#
|
|
# The shapes matter. In BOTH cases the pull request adds `from_pr.txt` and the
|
|
# base adds `from_base.txt` AFTER the branch point, so the branch head is never
|
|
# the content that lands even when it merges perfectly - that difference is what
|
|
# part 4 measures, and it is the ordinary case, not the pathological one.
|
|
# ---------------------------------------------------------------------------
|
|
GIT = ["git", "-c", "user.email=fixture@localhost", "-c", "user.name=fixture",
|
|
"-c", "init.defaultBranch=release/1.0", "-c", "commit.gpgsign=false"]
|
|
PRNUM = "7"
|
|
BASEREF = "release/1.0"
|
|
|
|
|
|
def git(cwd, *args, check_rc=True):
|
|
r = subprocess.run(GIT + list(args), cwd=cwd, capture_output=True, text=True)
|
|
if check_rc and r.returncode != 0:
|
|
raise RuntimeError(f"git {' '.join(args)} failed in {cwd}:\n{r.stderr}")
|
|
return r.stdout.strip()
|
|
|
|
|
|
def build_origin(stack, conflict):
|
|
"""A bare repo carrying refs/heads/release/1.0 and refs/pull/7/head."""
|
|
root = Path(tempfile.mkdtemp(prefix="test_ci_gate."))
|
|
stack.append(str(root))
|
|
work, bare = root / "work", root / "origin.git"
|
|
work.mkdir()
|
|
git(work, "init", "-q", ".")
|
|
(work / "shared.txt").write_text("one\ntwo\nthree\n")
|
|
git(work, "add", "-A")
|
|
git(work, "commit", "-qm", "branch point")
|
|
fork = git(work, "rev-parse", "HEAD")
|
|
|
|
# the base moves on
|
|
(work / "from_base.txt").write_text("only the base has this\n")
|
|
if conflict:
|
|
(work / "shared.txt").write_text("one\nTWO FROM BASE\nthree\n")
|
|
git(work, "add", "-A")
|
|
git(work, "commit", "-qm", "the base branch moves on")
|
|
base_sha = git(work, "rev-parse", "HEAD")
|
|
|
|
# the pull request branches from the fork point
|
|
git(work, "checkout", "-q", "-b", "pr", fork)
|
|
(work / "from_pr.txt").write_text("only the pull request has this\n")
|
|
if conflict:
|
|
(work / "shared.txt").write_text("one\nTWO FROM THE PULL REQUEST\nthree\n")
|
|
git(work, "add", "-A")
|
|
git(work, "commit", "-qm", "the pull request")
|
|
pr_sha = git(work, "rev-parse", "HEAD")
|
|
|
|
git(root, "init", "-q", "--bare", str(bare))
|
|
git(work, "remote", "add", "origin", str(bare))
|
|
git(work, "push", "-q", "origin", f"{base_sha}:refs/heads/{BASEREF}")
|
|
git(work, "push", "-q", "origin", f"{pr_sha}:refs/pull/{PRNUM}/head")
|
|
return str(bare), base_sha, pr_sha, str(root)
|
|
|
|
|
|
def run_script(script, ws, origin, pr_sha, extra=None):
|
|
env = dict(os.environ)
|
|
env.pop("PYTHONDONTWRITEBYTECODE", None)
|
|
env.update(
|
|
WS=ws, PRNUM=PRNUM, BASEREF=BASEREF, URL=origin, TOKEN="",
|
|
GITHUB_REF=f"refs/pull/{PRNUM}/head", GITHUB_SHA=pr_sha,
|
|
GIT_TERMINAL_PROMPT="0",
|
|
)
|
|
if extra:
|
|
env.update(extra)
|
|
return subprocess.run(["sh", "-c", script], env=env, capture_output=True, text=True)
|
|
|
|
|
|
tmpdirs = []
|
|
print("the gate's own honesty (CI-02)")
|
|
|
|
# ---------------------------------------------------------------------------
|
|
print("\n1. the required context string, from the two tokens that build it")
|
|
# ---------------------------------------------------------------------------
|
|
# ⚠️ READ BEFORE RENAMING ANYTHING IN checks.yml. Branch protection on
|
|
# release/1.0 has `enable_status_check: true` and
|
|
# `status_check_contexts: ['checks / unit (pull_request)']`. Gitea assembles
|
|
# that string from `name:` at the top of the workflow and the job KEY under
|
|
# `jobs:`. Change either and Gitea starts posting a context nobody is waiting
|
|
# for, while the one branch protection requires never arrives - so it stays
|
|
# `pending` forever, on every open pull request and every future one, and
|
|
# admin override is blocked. There is no error message anywhere in that
|
|
# sequence. This check is the only thing between a tidy-up and that.
|
|
REQUIRED = "checks / unit (pull_request)"
|
|
name, jobs = workflow_name(), job_keys()
|
|
check("the workflow still has a name", name is not None)
|
|
check("the workflow declares exactly one job", len(jobs) == 1)
|
|
built = f"{name} / {jobs[0]} (pull_request)" if name and len(jobs) == 1 else "<unparseable>"
|
|
check(f"the context it builds is exactly {REQUIRED!r}", built == REQUIRED)
|
|
if built != REQUIRED:
|
|
print(f" it now builds {built!r} - branch protection will wait forever for {REQUIRED!r}")
|
|
check("the workflow still triggers on pull_request", "pull_request:" in CODE)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
print("\n2. the three absences that are load-bearing")
|
|
# ---------------------------------------------------------------------------
|
|
# `paths:` is CI-01's: a workflow that learns to sit a pull request out leaves
|
|
# the required context pending forever, which is unmergeable forever.
|
|
check("no paths: filter", not any(l.strip().startswith("paths:") or l.strip().startswith("paths-ignore:")
|
|
for l in CODE.splitlines()))
|
|
# `concurrency:` is CI-02's, and it was MEASURED rather than reasoned about, on
|
|
# throwaway PR #29 with one run holding the single runner slot and two more
|
|
# pushes behind it:
|
|
# no block at all -> runs 561 and 562 both queued, both executed
|
|
# group + cancel-in-progress: false
|
|
# -> run 564 sat `Blocked by required conditions`
|
|
# and was cancelled at 0s the moment 565 existed
|
|
# A shared group cancels a run that has not started regardless of that flag, and
|
|
# a cancelled run posts `failure` on the required context ("Canceled after 0s"),
|
|
# which branch protection cannot tell from a real test failure. Runs 531 and 533
|
|
# on PR #14 died exactly this way, each at the same second the next run was
|
|
# created. Having no group is the only setting measured to leave a queued run
|
|
# alive.
|
|
check("no concurrency: group", not any(l.startswith("concurrency:") for l in CODE.splitlines()))
|
|
check("no cancel-in-progress anywhere in the commands", "cancel-in-progress" not in CODE)
|
|
# The fallback itself. Graded on the command lines: the prose above the step
|
|
# quotes the old line on purpose, and an `in YAML` search would match that.
|
|
check("the || $GITHUB_SHA fallback is gone from the commands",
|
|
not any("||" in l and "GITHUB_SHA" in l for l in CODE.splitlines()))
|
|
check("the checkout is no longer shallow (a merge needs the merge base)",
|
|
"--depth 1" not in CODE)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
print("\n3. the defect, reproduced: the old line grades a conflicted head, happily")
|
|
# ---------------------------------------------------------------------------
|
|
# NON-VACUITY FOR EVERYTHING BELOW. If this part were already red - if the old
|
|
# line refused a conflicted pull request on its own - then parts 4 and 5 would
|
|
# be measuring nothing, and the ticket would have no subject.
|
|
origin, base_sha, pr_sha, _root = build_origin(tmpdirs, conflict=True)
|
|
ws = tempfile.mkdtemp(prefix="test_ci_gate.ws.")
|
|
tmpdirs.append(ws)
|
|
OLD = '''
|
|
set -eu
|
|
rm -rf "$WS" && mkdir -p "$WS/goodwe-addon" && cd "$WS/goodwe-addon"
|
|
git init -q .
|
|
git remote add origin "$(echo "$URL" | sed "s#://#://x-access-token:${TOKEN}@#")"
|
|
git fetch -q --depth 1 origin "$GITHUB_REF" || git fetch -q --depth 1 origin "$GITHUB_SHA"
|
|
git checkout -qf FETCH_HEAD
|
|
echo "checked out $(git log --oneline -1)"
|
|
'''
|
|
old = run_script(OLD, ws, origin, pr_sha)
|
|
check("the pull request genuinely does not merge (the fixture is not lying)",
|
|
subprocess.run(GIT + ["merge-tree", "--write-tree", base_sha, pr_sha],
|
|
cwd=origin, capture_output=True, text=True).returncode != 0)
|
|
check("...and the old checkout exits 0 on it anyway", old.returncode == 0)
|
|
tree = Path(ws) / "goodwe-addon"
|
|
check("...having checked out the bare head", (tree / "from_pr.txt").is_file())
|
|
check("...which is missing what the base branch added", not (tree / "from_base.txt").is_file())
|
|
check("...and it says nothing about which of the two refs it used",
|
|
"GRADED" not in old.stdout and "REFUSED" not in old.stdout)
|
|
if old.returncode != 0:
|
|
print(old.stdout[-800:], old.stderr[-800:])
|
|
|
|
# ---------------------------------------------------------------------------
|
|
print("\n4. the shipped steps, executed, on a pull request that MERGES")
|
|
# ---------------------------------------------------------------------------
|
|
CO = step_script("check out the base branch and the pull request head")
|
|
MERGE = step_script("grade the merge, not the branch")
|
|
check("the checkout step can be found in checks.yml", CO is not None)
|
|
check("the merge/refuse step can be found in checks.yml", MERGE is not None)
|
|
if CO is None or MERGE is None:
|
|
print(" checks.yml has been restructured; this file grades nothing until it is updated")
|
|
print()
|
|
print(f"{len(fails)} FAILED")
|
|
raise SystemExit(1)
|
|
|
|
origin2, base2, pr2, _r2 = build_origin(tmpdirs, conflict=False)
|
|
ws2 = tempfile.mkdtemp(prefix="test_ci_gate.ws2.")
|
|
tmpdirs.append(ws2)
|
|
co = run_script(CO, ws2, origin2, pr2)
|
|
mg = run_script(MERGE, ws2, origin2, pr2)
|
|
check("the checkout step succeeds", co.returncode == 0)
|
|
check("the merge step succeeds on a mergeable pull request", mg.returncode == 0)
|
|
if co.returncode or mg.returncode:
|
|
print(co.stdout[-600:], co.stderr[-600:], mg.stdout[-600:], mg.stderr[-600:])
|
|
t2 = Path(ws2) / "goodwe-addon"
|
|
check("what is left on disk carries the pull request's file", (t2 / "from_pr.txt").is_file())
|
|
check("...AND the file the base branch added after the branch point", (t2 / "from_base.txt").is_file())
|
|
# That second one is the whole difference. The old line graded a tree WITHOUT
|
|
# from_base.txt - green about content that is not what lands - and nothing in
|
|
# the run said so. This is the ordinary case: no conflict anywhere in it.
|
|
check("the graded tree is exactly the tree of a real merge",
|
|
git(t2, "rev-parse", "HEAD^{tree}") ==
|
|
subprocess.run(GIT + ["merge-tree", "--write-tree", base2, pr2],
|
|
cwd=origin2, capture_output=True, text=True).stdout.strip())
|
|
|
|
# ---------------------------------------------------------------------------
|
|
print("\n5. the same two steps on a pull request that does NOT merge")
|
|
# ---------------------------------------------------------------------------
|
|
ws3 = tempfile.mkdtemp(prefix="test_ci_gate.ws3.")
|
|
tmpdirs.append(ws3)
|
|
co3 = run_script(CO, ws3, origin, pr_sha)
|
|
mg3 = run_script(MERGE, ws3, origin, pr_sha)
|
|
check("the checkout step still succeeds - a conflict does not stop a fetch",
|
|
co3.returncode == 0)
|
|
# Stated as its own check because it is why the defect was invisible: nothing
|
|
# about a conflicted pull request makes any git fetch fail on this server.
|
|
check("the merge step REFUSES (non-zero)", mg3.returncode != 0)
|
|
check("...and says nothing was tested, not that a test failed",
|
|
"REFUSED" in mg3.stdout and "NOTHING HAS BEEN TESTED" in mg3.stdout)
|
|
check("...and never claims to have graded anything", "GRADED:" not in mg3.stdout)
|
|
check("...and leaves no half-merged tree behind for a later step to grade",
|
|
not (Path(ws3) / "goodwe-addon" / "from_pr.txt").is_file()
|
|
or git(Path(ws3) / "goodwe-addon", "rev-parse", "HEAD") == base_sha)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
print("\n6. the run says which content it graded (CI-02 AC2)")
|
|
# ---------------------------------------------------------------------------
|
|
check("the mergeable run announces the merge it graded", "GRADED:" in mg.stdout)
|
|
check("...naming the base branch it merged into", BASEREF in mg.stdout)
|
|
check("...and the tree, which is the only unambiguous name for content",
|
|
"tree" in mg.stdout and git(t2, "rev-parse", "HEAD^{tree}")[:12] in mg.stdout)
|
|
# The no-pull-request path (workflow_dispatch) keeps a fallback, because a
|
|
# branch really can vanish between the dispatch and the fetch. It has to say so.
|
|
disp = run_script(CO, tempfile.mkdtemp(prefix="test_ci_gate.ws4."), origin2, pr2,
|
|
extra={"PRNUM": "", "GITHUB_REF": f"refs/heads/{BASEREF}"})
|
|
check("the no-pull-request path still checks something out", disp.returncode == 0)
|
|
check("...and announces that it is NOT merged content",
|
|
"GRADED:" in disp.stdout and "NOT merged content" in disp.stdout)
|
|
if disp.returncode != 0:
|
|
print(disp.stdout[-600:], disp.stderr[-600:])
|
|
|
|
for d in tmpdirs:
|
|
shutil.rmtree(d, ignore_errors=True)
|
|
|
|
print()
|
|
if fails:
|
|
print(f"{len(fails)} FAILED: " + "; ".join(fails))
|
|
sys.exit(1)
|
|
print("all checks passed")
|