CI-02: grade the merge, and stop a cancelled run reading as a test failure
checks / unit (pull_request) Successful in 3m44s

Two things a green from `checks / unit (pull_request)` was claiming without
having observed. Both were measured on a deliberately conflicted throwaway pull
request (#29, now closed and its branch deleted) rather than read off the
workflow, because the Actions log endpoints 404 on this Gitea and a step's own
conclusion is the only channel out of a job.

CASE 1, AND THE TICKET'S MECHANISM IS WRONG IN A WAY THAT MAKES IT WORSE.
The card says a conflicted pull request loses its merge ref, so that

    git fetch -q --depth 1 origin "$GITHUB_REF" \
      || git fetch -q --depth 1 origin "$GITHUB_SHA"

falls through to the second half and quietly grades the bare branch head. The
symptom is real - PR #14 reported success while `mergeable: false` - but that
is not how. Asking one question per step on the conflicted probe (runs 558,
559):

    GITHUB_REF == refs/pull/<n>/head          P01 passed
    GITHUB_REF != refs/pull/<n>/merge         P02 failed
    GITHUB_SHA == the pull request head sha   P03 passed
    fetching GITHUB_REF succeeds              P04 passed  <- even conflicted
    ...and it lands on the bare head          P05 passed
    refs/pull/<n>/merge does not exist        P06 failed

`git ls-remote origin 'refs/pull/*'` on this server advertises only `/head`, for
open and closed pull requests alike. There are no merge refs here at all. The
first fetch has therefore always succeeded and the fallback has never fired
once; both halves name the same commit anyway. So the gate did not lose merged
content on conflicted pull requests - it has never had merged content, on any
pull request in this repository's history. Green has meant "this branch passed
on its own", and with `block_on_outdated_branch` off on release/1.0 the base has
usually moved underneath it.

Reproduced end to end on one pull request, same conflict, gate swapped:

    run 567  the ORIGINAL gate   -> success, "Successful in 3m38s"
    run 566  the gate below      -> failure in 2s, merge step red,
                                    both test steps `skipped`

The checkout now fetches the base tip and the pull request head, merges them,
and refuses if the merge does not apply - as its own NAMED step, because
/actions/runs/<id>/jobs carries a per-step conclusion and that is what lets a
reader tell "nothing was tested" from "the tests spoke". `--depth 1` is gone: a
merge needs a merge base, and a full fetch of this repository measured 6.8 MB
and 1.1 s. The fallback survives only on the workflow_dispatch path, which has
no pull request to merge and now announces which bare ref it took.

CASE 2, AND `cancel-in-progress` IS NOT THE KNOB.
Runs 531 and 533 on PR #14 were cancelled at 0s, each at the exact second the
next run on that pull request was created:

    531  created 17:08:19 `Waiting to run`, cancelled 17:10:44 = 532's creation
    533  created 17:45:11 `Waiting to run`, cancelled 18:11:22 = 534's creation

Both have `started_at` at the Unix epoch. A cancelled run posts `failure` on the
required context, described `Canceled after 0s`; branch protection compares
state, not description, so "nothing ran" and "your tests are broken" arrive as
the same red wanting opposite responses. Three settings were measured, each with
one run holding the single runner slot and two further pushes behind it:

    no concurrency block at all   runs 561 and 562 both queued, both executed
    group, cancel-in-progress:false
                                  run 564 sat `Blocked by required conditions`
                                  and was cancelled at 0s the second 565 existed

A shared group cancels a run that has not started regardless of that flag -
GitHub documents it and Gitea 1.27 implements it. Having no group is the only
configuration measured to leave a queued run alive, so the block is gone and the
measurement is written above it. The cost is stated in the file: a developer who
pushes four times occupies the shared runner four times. That is CI-01's own
trade written the other way round - a delayed verdict, never a destroyed one -
and a destroyed verdict is worse here because it is indistinguishable from a
real failure. A sha-keyed group was considered and rejected: the only
cancellations it still permits are same-sha ones, whose red lands on a commit
that IS still the head.

WHAT IS NOT ESTABLISHED, SAID PLAINLY. Nothing here makes a cancelled run
distinguishable at the gate BY STATE; that is Gitea posting the status, not this
workflow, and no edit to this file can change it. What ships removes the cause
instead of relabelling the symptom. And a commit that never had a run carries an
empty `/statuses/<sha>` list, while a queued one carries `pending / Waiting to
run` within seconds and a concurrency-held one `pending / Blocked by required
conditions` - those were already distinguishable at the API. `/actions/tasks`,
which is what was polled when this was first reported, lists only tasks that
started and cannot answer the question at all.

test_ci_gate.py grades the gate by RUNNING it. It extracts the checkout and
merge steps out of checks.yml and executes the shipped commands against two
throwaway local repositories it builds - one whose pull request merges, one
whose does not - and pins that the old one-liner exits 0 on the unmergeable one
while the new steps refuse and say nothing was tested. Part 1 pins the two
tokens that assemble `checks / unit (pull_request)`: renaming `name:` or the
`unit:` job key would leave every open and future pull request waiting forever
on a context that never arrives, with no error anywhere in that sequence, and
admin override blocked. Seven mutations of checks.yml were each confirmed red,
including both renames and a restored `--depth 1`. .gitea/ci/Dockerfile gains
`git` for it; the fixtures are local paths and no network is touched.

17/17 green in the checks image.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-01 03:42:54 +02:00
co-authored by Claude Opus 5
parent 6ea79f1043
commit 1e4bad8aa7
4 changed files with 651 additions and 14 deletions
+13 -1
View File
@@ -18,5 +18,17 @@
# python3, no aiohttp, no pip (`No module named pip`; checked on the box, not # python3, no aiohttp, no pip (`No module named pip`; checked on the box, not
# assumed). An image is not gold-plating here, it is the only option that does # assumed). An image is not gold-plating here, it is the only option that does
# not mutate a shared long-lived runner. # not mutate a shared long-lived runner.
# ⚠️ `git` IS A TEST DEPENDENCY, NOT A CONVENIENCE (CI-02). test_ci_gate.py
# grades the gate's own checkout by EXTRACTING the shell out of
# .gitea/workflows/checks.yml and RUNNING it - against two throwaway local
# repositories it builds itself, one whose pull request merges and one whose
# does not. Reading those commands as a string would be the thing this
# repository keeps finding: a check that never observed its subject. Running
# them needs git in the image the checks run in.
#
# It costs one apk package and no network at test time: the fixtures are local
# paths, and the token substitution in the workflow's `git remote add` is a
# no-op on a path with no `://` in it, which is exactly why the test can use
# the shipped line verbatim instead of a paraphrase of it.
FROM alpine:3 FROM alpine:3
RUN apk add --no-cache python3 py3-aiohttp py3-paho-mqtt RUN apk add --no-cache python3 py3-aiohttp py3-paho-mqtt git
+166 -13
View File
@@ -18,7 +18,9 @@
# so it QUEUES behind a rig campaign - a delayed verdict, never a destroyed one. # so it QUEUES behind a rig campaign - a delayed verdict, never a destroyed one.
# #
# NO JS ACTIONS: the runner image has no node, so `actions/checkout` cannot run. # NO JS ACTIONS: the runner image has no node, so `actions/checkout` cannot run.
# Checkout is seven lines of git below. # Checkout is plain git below - and since CI-02 it is a checkout of the MERGE of
# the pull request into its base, not of the branch tip. See the long note on
# that step; a green from this workflow is a claim about content that can land.
name: checks name: checks
on: on:
@@ -26,15 +28,53 @@ on:
branches: [release/1.0, master] branches: [release/1.0, master]
workflow_dispatch: workflow_dispatch:
concurrency: # ⚠️ NO `concurrency:` BLOCK, AND ITS ABSENCE IS AS LOAD-BEARING AS THE MISSING
# ⚠️ PER PULL REQUEST, NOT A BARE NAME. A shared group holds one running and # `paths:` FILTER ABOVE. CI-01 put one here - `group: addon-checks-<pr number>`,
# one pending run, so a bare name lets a second PR supersede a first PR's run # `cancel-in-progress: true` - to stop a second pull request superseding a
# before any runner has picked it up - which is exactly what happened to the # first one's run. It did stop that. It also did this, twice, on PR #14:
# other repo's run 521 (four jobs, `started_at` at the Unix epoch, #
# `runner_id: 0`, concluded `cancelled`, and the PR merged anyway). Do not # run 531 created 17:08:19 `pending / Waiting to run`
# replace this with a constant. # cancelled 17:10:44 <- the exact second run 532 was created
group: addon-checks-${{ github.event.pull_request.number || github.ref }} # run 533 created 17:45:11 `pending / Waiting to run`
cancel-in-progress: true # cancelled 18:11:22 <- the exact second run 534 was created
#
# Both cancelled runs have `started_at` at the Unix epoch: they never executed.
# A cancelled run posts `failure` on the required context, described
# `Canceled after 0s`. Branch protection compares STATE, not description, so
# "nothing ran" and "your tests are broken" arrive at the gate as the same red,
# and the correct operator response to each is the opposite of the other.
#
# CI-02 MEASURED THE THREE CANDIDATE SETTINGS ON A THROWAWAY PR (#29) RATHER
# THAN READING THE DOCS, because the docs and this Gitea disagree about which
# knob does it. In each trial one run was made to hold the single slot and two
# further pushes followed:
#
# trial 1, NO concurrency block - runs 561 and 562 both sat queued together
# and both executed. Nothing was cancelled.
# trial 2, group kept, cancel-in-progress: FALSE
# - run 564 sat `pending / Blocked by required
# conditions` and was cancelled at 0s the
# second run 565 was created.
#
# So `cancel-in-progress` IS NOT THE KNOB. A shared group cancels a run that has
# not started no matter what that flag says - GitHub documents this ("any
# previously pending run in the group will be cancelled") and Gitea 1.27
# implements it. The only configuration that leaves a queued run alive is having
# no group at all, which is what is written here.
#
# ⚠️ DO NOT REPLACE THIS WITH A SHA-KEYED GROUP as a compromise. It looks
# tempting - two runs on different commits could never share a group, so nothing
# would be cancelled - but the only cancellations it still permits are the
# same-sha ones, and those are precisely the dangerous half: a red for a run
# that never started, landing on a commit that IS still the head. It
# concentrates the failure instead of removing it.
#
# WHAT THIS COSTS, stated honestly: the runner is `capacity: 1` and shared with
# the other repo's rig, so a developer who pushes four times in a minute now
# occupies it four times instead of once. That is CI-01's own trade written the
# other way round - it accepted "a delayed verdict, never a destroyed one" for
# the rig, and this accepts the same for pushes. A destroyed verdict here is
# worse than a slow one, because it is indistinguishable from a real failure.
jobs: jobs:
unit: unit:
@@ -58,19 +98,132 @@ jobs:
# exist, so the test dies on `IndexError: 1` before its first assertion. # exist, so the test dies on `IndexError: 1` before its first assertion.
# Measured during CI-01; the flat layout was tried first and did exactly # Measured during CI-01; the flat layout was tried first and did exactly
# that. $WS/goodwe-addon/goodwe_controller gives parents[1] == $WS. # that. $WS/goodwe-addon/goodwe_controller gives parents[1] == $WS.
- name: checkout into a path the host shares (git; no node for actions/checkout) #
# ⚠️ WHAT THIS CHECKS OUT IS THE MERGE, NOT THE BRANCH, AND THAT IS THE
# WHOLE OF CI-02. What was here before was one line:
#
# git fetch -q --depth 1 origin "$GITHUB_REF" \
# || git fetch -q --depth 1 origin "$GITHUB_SHA"
#
# CI-02 measured what those two names actually are on this Gitea, on a
# deliberately conflicted throwaway PR (#29, runs 558 and 559), by asking
# one question per STEP - the Actions log endpoints 404 here, so a step's
# own conclusion is the only channel out of a job. The answers:
#
# GITHUB_REF == refs/pull/<n>/head (step P01 passed)
# GITHUB_REF != refs/pull/<n>/merge (step P02 failed)
# GITHUB_SHA == the pull request head sha (step P03 passed)
# fetching GITHUB_REF succeeds (step P04 passed)
# ...and lands on the bare head (step P05 passed)
# refs/pull/<n>/merge does not exist (step P06 failed)
#
# ⚠️ SO THE FALLBACK NEVER FIRED, AND THE DEFECT IS WORSE THAN THE ONE
# REPORTED. The ticket says a conflicted PR loses its merge ref and the
# `||` quietly grades the bare head instead. It does not: this Gitea has
# no merge refs at all - `git ls-remote origin 'refs/pull/*'` advertises
# only `/head`, for open and closed pull requests alike - so the FIRST
# half always succeeds, and both halves name the same commit anyway
# (GITHUB_REF resolves to GITHUB_SHA). The gate has never once graded
# merged content, on any pull request in this repository's history. Green
# has meant "this branch passed on its own", which on a base that has
# moved - and `block_on_outdated_branch` is off here, so it usually has -
# is not the content that lands. PR #14 reporting success while
# `mergeable: false` was not the fallback misfiring; it was the gate
# working exactly as written.
#
# A fallback that cannot fire cannot be "kept for the case it was for", so
# it is gone from this path and kept, ANNOUNCED, on the one path that has
# no pull request to merge (workflow_dispatch, below).
#
# ⚠️ NOT `--depth 1` ANY MORE. A merge needs the merge base, and a shallow
# pair does not have one. Measured before changing it: a full fetch of
# this repository is 6.8 MB and 1.1 s against this server, over 95
# commits. The depth flag was buying nothing worth a wrong verdict.
#
# ⚠️ THE WORKFLOW AND THE CONTENT NOW COME FROM DIFFERENT TREES. Gitea
# picks the workflow file from refs/pull/<n>/head, so THIS file is always
# the branch's; the tree graded below - including .gitea/ci/Dockerfile -
# is the merge. That is the right way round (a pull request must not be
# able to grade itself with a gate the base has not seen) but it means a
# change to this file is tested by the branch it is on, which is how you
# are reading this.
- name: check out the base branch and the pull request head (git; no node for actions/checkout)
env: env:
URL: ${{ github.server_url }}/${{ github.repository }}.git URL: ${{ github.server_url }}/${{ github.repository }}.git
TOKEN: ${{ secrets.GITHUB_TOKEN }} TOKEN: ${{ secrets.GITHUB_TOKEN }}
PRNUM: ${{ github.event.pull_request.number }}
BASEREF: ${{ github.base_ref }}
run: | run: |
set -eu set -eu
rm -rf "$WS" && mkdir -p "$WS/goodwe-addon" && cd "$WS/goodwe-addon" rm -rf "$WS" && mkdir -p "$WS/goodwe-addon" && cd "$WS/goodwe-addon"
git init -q . git init -q .
git remote add origin "$(echo "$URL" | sed "s#://#://x-access-token:${TOKEN}@#")" git remote add origin "$(echo "$URL" | sed "s#://#://x-access-token:${TOKEN}@#")"
git fetch -q --depth 1 origin "$GITHUB_REF" || git fetch -q --depth 1 origin "$GITHUB_SHA" if [ -n "$PRNUM" ]; then
git checkout -qf FETCH_HEAD # Two fetches, not one, so that a failure to reach either ref fails
# HERE with the ref in the message, rather than inside the merge
# step where it would read as a conflict. `github.base_ref` being
# populated at all was checked on the probe PR too (step Q01).
git fetch -q origin "+refs/heads/$BASEREF:refs/remotes/origin/base"
git fetch -q origin "+refs/pull/$PRNUM/head:refs/remotes/origin/pr"
git checkout -qf refs/remotes/origin/base
echo "base $BASEREF $(git rev-parse refs/remotes/origin/base)"
echo "head refs/pull/$PRNUM/head $(git rev-parse refs/remotes/origin/pr)"
else
# NO PULL REQUEST (workflow_dispatch). There is nothing to merge
# into, so this path grades a bare ref and SAYS SO - which is all
# the old `||` ever needed to do. It is kept here because a branch
# can be deleted or renamed between the dispatch and the fetch, and
# unlike the pull_request path that is a case nobody has ruled out.
if git fetch -q origin "$GITHUB_REF"; then
echo "GRADED: the bare ref $GITHUB_REF - NOT merged content (no pull request in this event)."
else
git fetch -q origin "$GITHUB_SHA"
echo "GRADED: the bare commit $GITHUB_SHA - NOT merged content, and NOT $GITHUB_REF either, which would not fetch."
fi
git checkout -qf FETCH_HEAD
fi
echo "checked out $(git log --oneline -1)" echo "checked out $(git log --oneline -1)"
# ⚠️ THIS STEP IS THE ONE THAT MAKES A GREEN MEAN SOMETHING. It exists as
# its OWN step rather than as three lines of the checkout because the
# Actions log endpoints 404 on this Gitea and
# /api/v1/repos/.../actions/runs/<id>/jobs - which carries a per-step
# conclusion - is the only machine-readable account of what a run did. A
# reader who sees this step red knows the pull request does not apply and
# that NO TEST RAN; a reader who sees the next two red knows the tests ran
# and spoke. Collapse them back together and that distinction is gone
# again, which is the shape of defect this whole ticket is about.
#
# The refusal is a FAILURE rather than a skip on purpose. "Reports
# nothing" would leave the required context pending forever, which this
# file's own header explains is unmergeable-forever; and a conflicted pull
# request cannot be merged by Gitea anyway, so the red blocks nothing that
# was not already blocked. What it does is stop a green being posted about
# content that cannot land.
- name: grade the merge, not the branch - refuse a pull request that does not apply
env:
PRNUM: ${{ github.event.pull_request.number }}
BASEREF: ${{ github.base_ref }}
run: |
set -eu
cd "$WS/goodwe-addon"
if [ -z "$PRNUM" ]; then
echo "no pull request in this event - nothing to merge, and the previous step said what it graded."
exit 0
fi
if git -c user.email=checks@localhost -c user.name=checks \
merge --no-edit refs/remotes/origin/pr; then
echo "GRADED: refs/pull/$PRNUM/head merged into $BASEREF"
echo " result $(git rev-parse HEAD)"
echo " tree $(git rev-parse 'HEAD^{tree}')"
else
git merge --abort 2>/dev/null || true
echo "REFUSED: refs/pull/$PRNUM/head does not merge into $BASEREF."
echo " NOTHING HAS BEEN TESTED. This is not a test failure - no test ran."
echo " Rebase or merge $BASEREF into the branch and push again."
exit 1
fi
- name: build the test image (shipped deps, from apk - see .gitea/ci/Dockerfile) - name: build the test image (shipped deps, from apk - see .gitea/ci/Dockerfile)
run: | run: |
set -eu set -eu
+65
View File
@@ -2,6 +2,71 @@
## Unreleased ## Unreleased
**The gate has never once graded the content a pull request would land, and a
run that never started reported as a test failure (CI-02).** Both halves were
measured on a deliberately conflicted throwaway pull request rather than read
off the workflow, because the Actions log endpoints 404 on this Gitea and a
step's own conclusion is the only channel out of a job.
**Case 1, and the ticket's mechanism was wrong in a way that made it worse.**
CI-02 was filed saying a conflicted pull request loses its merge ref, so that
`git fetch $GITHUB_REF || git fetch $GITHUB_SHA` falls through and grades the
bare branch head. The symptom was real — PR #14 reported `checks / unit ->
success` while `mergeable: false` — but the mechanism is not. On this server
`GITHUB_REF` **is** `refs/pull/<n>/head`, `GITHUB_SHA` is the same commit, and
`git ls-remote origin 'refs/pull/*'` advertises no `/merge` refs at all, for
open and closed pull requests alike. The first fetch has always succeeded; **the
fallback has never fired once.** So the gate did not lose merged content on
conflicted pull requests — it has never had merged content, on any pull request
in this repository's history. Green has meant "this branch passed on its own",
which on a base that has moved is not what lands, and `block_on_outdated_branch`
is off on release/1.0 so the base usually has moved.
**The checkout now fetches the base tip and the pull request head, merges them,
and refuses if the merge does not apply** — as its own named step, so
`/actions/runs/<id>/jobs` distinguishes "nothing was tested" from "the tests
spoke". Verified by producing it: with the fix on a conflicted pull request the
required context went `failure` in 2 s with the two test steps `skipped`. The
`--depth 1` is gone (a merge needs a merge base; a full fetch of this repository
is 6.8 MB and 1.1 s), and the fallback survives only on the `workflow_dispatch`
path, which has no pull request to merge and now says so in its output.
**Case 2: `concurrency:` is gone, and `cancel-in-progress` was never the knob.**
Runs 531 and 533 on PR #14 were cancelled at 0 s each at the exact second the
next run was created, having never started; a cancelled run posts `failure` on
the required context, described `Canceled after 0s`, and branch protection
compares state and not description — so "nothing ran" and "your tests are
broken" arrive at the gate as the same red, wanting opposite responses. Three
settings were measured with one run holding the single runner slot and two
pushes behind it: with **no block at all** both queued runs survived and
executed; with the group kept and **`cancel-in-progress: false`** the queued run
was still cancelled at 0 s. A shared group cancels a run that has not started
regardless of that flag. Having no group is the only configuration measured to
leave a queued run alive, and it is what ships. The cost — a developer who
pushes four times occupies the shared runner four times — is CI-01's own trade
written the other way round: a delayed verdict, never a destroyed one.
**`test_ci_gate.py` grades the gate by running it, not by reading it.** It
extracts the checkout and merge steps out of `checks.yml` and executes the
shipped commands against two throwaway local repositories it builds — one whose
pull request merges, one whose does not — and pins that the old one-liner exits
0 on the unmergeable one while the new steps refuse. It also pins the two tokens
that assemble `checks / unit (pull_request)`: renaming `name:` or the `unit:`
job key would leave every open and future pull request waiting forever on a
context that never arrives, with no error anywhere in that sequence. Seven
mutations of `checks.yml` were each confirmed red. `.gitea/ci/Dockerfile` gains
`git` for it.
**What was NOT established, stated so nobody reads more into this.** The
original report that "no run was ever created for `0942228d`" was already
corrected on the card — run 533 was created and cancelled — and this work
confirms that from the runs list and the per-commit status history. A commit
that never had a run carries an empty `/statuses/<sha>` list; a queued one
carries `pending / Waiting to run` within seconds, and a concurrency-held one
`pending / Blocked by required conditions`. Those are distinguishable at the
API. `/actions/tasks`, which is what was polled originally, lists only tasks
that started, and cannot answer the question at all.
**A setpoint device smaller than `max_w` silently disarmed the frozen-meter **A setpoint device smaller than `max_w` silently disarmed the frozen-meter
detector, and the status page reported it as armed (SAFETY-20).** SAFETY-18's detector, and the status page reported it as armed (SAFETY-20).** SAFETY-18's
conjunction needs `abs(commanded_w) >= 0.95 * max_w`, but the emitted command is conjunction needs `abs(commanded_w) >= 0.95 * max_w`, but the emitted command is
+407
View File
@@ -0,0 +1,407 @@
"""Runnable check for CI-02 - what a green from `checks / unit` is a claim about.
`python3 test_ci_gate.py`
test_bytecode.py grades the thing that grades the add-on. This file grades the
thing that decides whether anything was graded at all.
THE DEFECT, and the correction the measurement forced. CI-02 was filed saying
that a CONFLICTED pull request loses its merge ref, so that
git fetch -q --depth 1 origin "$GITHUB_REF" \
|| git fetch -q --depth 1 origin "$GITHUB_SHA"
falls through to the second half and quietly grades the bare branch head. PR #14
did report `checks / unit -> success` while `mergeable: false`, so the symptom
was real. The mechanism was not. Measured on a deliberately conflicted throwaway
pull request (#29, runs 558 and 559 - one question per STEP, because this
Gitea's Actions log endpoints 404 and a step's conclusion is the only channel
out of a job):
GITHUB_REF == refs/pull/<n>/head P01 passed
GITHUB_REF != refs/pull/<n>/merge P02 failed
GITHUB_SHA == the pull request head sha P03 passed
fetching GITHUB_REF succeeds P04 passed <- even conflicted
...and it lands on the bare head P05 passed
refs/pull/<n>/merge does not exist P06 failed
`git ls-remote origin 'refs/pull/*'` on this server advertises only `/head`, for
open and closed pull requests alike. THERE ARE NO MERGE REFS HERE. So the first
half of the `||` always succeeded, the fallback never fired once, and both halves
named the same commit anyway. The gate has never graded merged content on any
pull request in this repository's history. Green has meant "this branch passed
on its own" - which, on a base branch that has moved underneath it, is not the
content that lands. `block_on_outdated_branch` is off on release/1.0, so the
base usually has moved.
WHAT THIS FILE PINS.
Part 1 The context string branch protection requires is assembled from two
tokens in checks.yml. Renaming either makes every open and future
pull request unmergeable forever, silently, because Gitea waits on a
context it will now never receive. This part is the tripwire on that.
Part 2 The three absences in checks.yml that are load-bearing, each graded
on the command lines rather than on the prose around them.
Part 3 THE DEFECT, REPRODUCED. The old one-line checkout is run verbatim
against a real conflicted pull request built here, and is shown to
exit 0 having checked out content that cannot land.
Part 4 THE FIX, EXECUTED. The checkout and merge steps are EXTRACTED FROM
checks.yml AND RUN - the shipped commands, not a paraphrase - against
a pull request that merges. The tree they leave behind is the merge,
and is shown to differ from the head the old line would have graded.
Part 5 The same two steps against a pull request that does NOT merge: the
merge step exits non-zero, says nothing was tested, and never prints
a GRADED line.
Part 6 The gate announces which content it graded, on both paths, so that a
reader can tell merged content from a bare ref (CI-02 AC2).
⚠️ THIS FILE RUNS THE WORKFLOW'S OWN SHELL. That is the point: a check that
read those commands as a string and matched them against an expectation would
be exactly the failure this repository keeps finding - a reading that never
observed its subject. The extractor below is deliberately brittle about finding
the step it wants: if checks.yml is restructured so that the step cannot be
found, this file goes RED rather than quietly grading nothing.
⚠️ NEEDS `git`, which is why .gitea/ci/Dockerfile installs it. No network: the
fixtures are local repositories built in a temp directory, and the workflow's
`git remote add origin "$(echo "$URL" | sed "s#://#://x-access-token:...@#")"`
is a no-op substitution on a path with no `://` in it, which is what lets the
shipped line be used verbatim.
⚠️ This file writes only under a temporary directory. It must stay that way:
CI mounts the checkout READ-ONLY.
⚠️ MUTATING THE MODULE THIS FILE GRADES? PURGE `__pycache__` FIRST - or use
`tools/mutation_run.sh`, which does. See MUTATION.md. (There is no module here
to mutate; the subject is checks.yml. Break a line of it and re-run this file
expecting red - if it stays green, it was never watching.)
"""
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
HERE = Path(__file__).resolve().parent
WF = HERE.parent / ".gitea" / "workflows" / "checks.yml"
fails = []
def check(name, cond):
if cond:
print(f" ok {name}")
else:
print(f" FAIL {name}")
fails.append(name)
# ---------------------------------------------------------------------------
# Reading checks.yml. There is no yaml module in the image the checks run in
# (alpine py3-aiohttp py3-paho-mqtt, and adding a parser to grade five lines
# would be a worse trade than a small parser that fails loudly), so the two
# things this file needs are pulled out by hand:
#
# * the workflow name and the job keys, for part 1
# * the `run:` block of a named step, verbatim, for parts 3 to 6
#
# COMMAND LINES ONLY WHERE IT MATTERS. test_bytecode.py's part 6 learned this
# the hard way: an `in y` search over a whole workflow passes on the PROSE that
# explains a flag, and stayed green after the flag itself was deleted. Every
# absence checked in part 2 is checked against the file with its comment lines
# stripped.
# ---------------------------------------------------------------------------
if not WF.is_file():
print(f"cannot find the workflow at {WF} - nothing to grade")
raise SystemExit(1)
YAML = WF.read_text()
CODE = "\n".join(l for l in YAML.splitlines() if not l.lstrip().startswith("#"))
def workflow_name():
for line in YAML.splitlines():
if line.startswith("name:"):
return line.split(":", 1)[1].strip()
return None
def job_keys():
out, seen = [], False
for line in YAML.splitlines():
if line.startswith("jobs:"):
seen = True
continue
if seen:
if line and not line[0].isspace():
break
if line.startswith(" ") and not line.startswith(" ") and line.rstrip().endswith(":"):
out.append(line.strip()[:-1])
return out
def step_script(name_prefix):
"""Return the `run: |` body of the step whose name starts with name_prefix."""
lines = YAML.splitlines()
i = 0
while i < len(lines):
if lines[i].strip().startswith("- name:") and name_prefix in lines[i]:
break
i += 1
else:
return None
if i >= len(lines):
return None
while i < len(lines) and lines[i].strip() != "run: |":
# stop at the next step so a missing block cannot borrow the next one's
if i > 0 and lines[i].strip().startswith("- name:") and name_prefix not in lines[i]:
return None
i += 1
if i >= len(lines):
return None
body, i = [], i + 1
while i < len(lines):
l = lines[i]
if l.strip() == "":
body.append("")
elif l.startswith(" " * 10):
body.append(l[10:])
else:
break
i += 1
return "\n".join(body).strip("\n")
# ---------------------------------------------------------------------------
# The fixture: a real bare repository with a base branch and a refs/pull/N/head,
# built to order. `conflict=True` makes the pull request touch the same lines
# the base moved, which is how PR #29 was made unmergeable on the server.
#
# The shapes matter. In BOTH cases the pull request adds `from_pr.txt` and the
# base adds `from_base.txt` AFTER the branch point, so the branch head is never
# the content that lands even when it merges perfectly - that difference is what
# part 4 measures, and it is the ordinary case, not the pathological one.
# ---------------------------------------------------------------------------
GIT = ["git", "-c", "user.email=fixture@localhost", "-c", "user.name=fixture",
"-c", "init.defaultBranch=release/1.0", "-c", "commit.gpgsign=false"]
PRNUM = "7"
BASEREF = "release/1.0"
def git(cwd, *args, check_rc=True):
r = subprocess.run(GIT + list(args), cwd=cwd, capture_output=True, text=True)
if check_rc and r.returncode != 0:
raise RuntimeError(f"git {' '.join(args)} failed in {cwd}:\n{r.stderr}")
return r.stdout.strip()
def build_origin(stack, conflict):
"""A bare repo carrying refs/heads/release/1.0 and refs/pull/7/head."""
root = Path(tempfile.mkdtemp(prefix="test_ci_gate."))
stack.append(str(root))
work, bare = root / "work", root / "origin.git"
work.mkdir()
git(work, "init", "-q", ".")
(work / "shared.txt").write_text("one\ntwo\nthree\n")
git(work, "add", "-A")
git(work, "commit", "-qm", "branch point")
fork = git(work, "rev-parse", "HEAD")
# the base moves on
(work / "from_base.txt").write_text("only the base has this\n")
if conflict:
(work / "shared.txt").write_text("one\nTWO FROM BASE\nthree\n")
git(work, "add", "-A")
git(work, "commit", "-qm", "the base branch moves on")
base_sha = git(work, "rev-parse", "HEAD")
# the pull request branches from the fork point
git(work, "checkout", "-q", "-b", "pr", fork)
(work / "from_pr.txt").write_text("only the pull request has this\n")
if conflict:
(work / "shared.txt").write_text("one\nTWO FROM THE PULL REQUEST\nthree\n")
git(work, "add", "-A")
git(work, "commit", "-qm", "the pull request")
pr_sha = git(work, "rev-parse", "HEAD")
git(root, "init", "-q", "--bare", str(bare))
git(work, "remote", "add", "origin", str(bare))
git(work, "push", "-q", "origin", f"{base_sha}:refs/heads/{BASEREF}")
git(work, "push", "-q", "origin", f"{pr_sha}:refs/pull/{PRNUM}/head")
return str(bare), base_sha, pr_sha, str(root)
def run_script(script, ws, origin, pr_sha, extra=None):
env = dict(os.environ)
env.pop("PYTHONDONTWRITEBYTECODE", None)
env.update(
WS=ws, PRNUM=PRNUM, BASEREF=BASEREF, URL=origin, TOKEN="",
GITHUB_REF=f"refs/pull/{PRNUM}/head", GITHUB_SHA=pr_sha,
GIT_TERMINAL_PROMPT="0",
)
if extra:
env.update(extra)
return subprocess.run(["sh", "-c", script], env=env, capture_output=True, text=True)
tmpdirs = []
print("the gate's own honesty (CI-02)")
# ---------------------------------------------------------------------------
print("\n1. the required context string, from the two tokens that build it")
# ---------------------------------------------------------------------------
# ⚠️ READ BEFORE RENAMING ANYTHING IN checks.yml. Branch protection on
# release/1.0 has `enable_status_check: true` and
# `status_check_contexts: ['checks / unit (pull_request)']`. Gitea assembles
# that string from `name:` at the top of the workflow and the job KEY under
# `jobs:`. Change either and Gitea starts posting a context nobody is waiting
# for, while the one branch protection requires never arrives - so it stays
# `pending` forever, on every open pull request and every future one, and
# admin override is blocked. There is no error message anywhere in that
# sequence. This check is the only thing between a tidy-up and that.
REQUIRED = "checks / unit (pull_request)"
name, jobs = workflow_name(), job_keys()
check("the workflow still has a name", name is not None)
check("the workflow declares exactly one job", len(jobs) == 1)
built = f"{name} / {jobs[0]} (pull_request)" if name and len(jobs) == 1 else "<unparseable>"
check(f"the context it builds is exactly {REQUIRED!r}", built == REQUIRED)
if built != REQUIRED:
print(f" it now builds {built!r} - branch protection will wait forever for {REQUIRED!r}")
check("the workflow still triggers on pull_request", "pull_request:" in CODE)
# ---------------------------------------------------------------------------
print("\n2. the three absences that are load-bearing")
# ---------------------------------------------------------------------------
# `paths:` is CI-01's: a workflow that learns to sit a pull request out leaves
# the required context pending forever, which is unmergeable forever.
check("no paths: filter", not any(l.strip().startswith("paths:") or l.strip().startswith("paths-ignore:")
for l in CODE.splitlines()))
# `concurrency:` is CI-02's, and it was MEASURED rather than reasoned about, on
# throwaway PR #29 with one run holding the single runner slot and two more
# pushes behind it:
# no block at all -> runs 561 and 562 both queued, both executed
# group + cancel-in-progress: false
# -> run 564 sat `Blocked by required conditions`
# and was cancelled at 0s the moment 565 existed
# A shared group cancels a run that has not started regardless of that flag, and
# a cancelled run posts `failure` on the required context ("Canceled after 0s"),
# which branch protection cannot tell from a real test failure. Runs 531 and 533
# on PR #14 died exactly this way, each at the same second the next run was
# created. Having no group is the only setting measured to leave a queued run
# alive.
check("no concurrency: group", not any(l.startswith("concurrency:") for l in CODE.splitlines()))
check("no cancel-in-progress anywhere in the commands", "cancel-in-progress" not in CODE)
# The fallback itself. Graded on the command lines: the prose above the step
# quotes the old line on purpose, and an `in YAML` search would match that.
check("the || $GITHUB_SHA fallback is gone from the commands",
not any("||" in l and "GITHUB_SHA" in l for l in CODE.splitlines()))
check("the checkout is no longer shallow (a merge needs the merge base)",
"--depth 1" not in CODE)
# ---------------------------------------------------------------------------
print("\n3. the defect, reproduced: the old line grades a conflicted head, happily")
# ---------------------------------------------------------------------------
# NON-VACUITY FOR EVERYTHING BELOW. If this part were already red - if the old
# line refused a conflicted pull request on its own - then parts 4 and 5 would
# be measuring nothing, and the ticket would have no subject.
origin, base_sha, pr_sha, _root = build_origin(tmpdirs, conflict=True)
ws = tempfile.mkdtemp(prefix="test_ci_gate.ws.")
tmpdirs.append(ws)
OLD = '''
set -eu
rm -rf "$WS" && mkdir -p "$WS/goodwe-addon" && cd "$WS/goodwe-addon"
git init -q .
git remote add origin "$(echo "$URL" | sed "s#://#://x-access-token:${TOKEN}@#")"
git fetch -q --depth 1 origin "$GITHUB_REF" || git fetch -q --depth 1 origin "$GITHUB_SHA"
git checkout -qf FETCH_HEAD
echo "checked out $(git log --oneline -1)"
'''
old = run_script(OLD, ws, origin, pr_sha)
check("the pull request genuinely does not merge (the fixture is not lying)",
subprocess.run(GIT + ["merge-tree", "--write-tree", base_sha, pr_sha],
cwd=origin, capture_output=True, text=True).returncode != 0)
check("...and the old checkout exits 0 on it anyway", old.returncode == 0)
tree = Path(ws) / "goodwe-addon"
check("...having checked out the bare head", (tree / "from_pr.txt").is_file())
check("...which is missing what the base branch added", not (tree / "from_base.txt").is_file())
check("...and it says nothing about which of the two refs it used",
"GRADED" not in old.stdout and "REFUSED" not in old.stdout)
if old.returncode != 0:
print(old.stdout[-800:], old.stderr[-800:])
# ---------------------------------------------------------------------------
print("\n4. the shipped steps, executed, on a pull request that MERGES")
# ---------------------------------------------------------------------------
CO = step_script("check out the base branch and the pull request head")
MERGE = step_script("grade the merge, not the branch")
check("the checkout step can be found in checks.yml", CO is not None)
check("the merge/refuse step can be found in checks.yml", MERGE is not None)
if CO is None or MERGE is None:
print(" checks.yml has been restructured; this file grades nothing until it is updated")
print()
print(f"{len(fails)} FAILED")
raise SystemExit(1)
origin2, base2, pr2, _r2 = build_origin(tmpdirs, conflict=False)
ws2 = tempfile.mkdtemp(prefix="test_ci_gate.ws2.")
tmpdirs.append(ws2)
co = run_script(CO, ws2, origin2, pr2)
mg = run_script(MERGE, ws2, origin2, pr2)
check("the checkout step succeeds", co.returncode == 0)
check("the merge step succeeds on a mergeable pull request", mg.returncode == 0)
if co.returncode or mg.returncode:
print(co.stdout[-600:], co.stderr[-600:], mg.stdout[-600:], mg.stderr[-600:])
t2 = Path(ws2) / "goodwe-addon"
check("what is left on disk carries the pull request's file", (t2 / "from_pr.txt").is_file())
check("...AND the file the base branch added after the branch point", (t2 / "from_base.txt").is_file())
# That second one is the whole difference. The old line graded a tree WITHOUT
# from_base.txt - green about content that is not what lands - and nothing in
# the run said so. This is the ordinary case: no conflict anywhere in it.
check("the graded tree is exactly the tree of a real merge",
git(t2, "rev-parse", "HEAD^{tree}") ==
subprocess.run(GIT + ["merge-tree", "--write-tree", base2, pr2],
cwd=origin2, capture_output=True, text=True).stdout.strip())
# ---------------------------------------------------------------------------
print("\n5. the same two steps on a pull request that does NOT merge")
# ---------------------------------------------------------------------------
ws3 = tempfile.mkdtemp(prefix="test_ci_gate.ws3.")
tmpdirs.append(ws3)
co3 = run_script(CO, ws3, origin, pr_sha)
mg3 = run_script(MERGE, ws3, origin, pr_sha)
check("the checkout step still succeeds - a conflict does not stop a fetch",
co3.returncode == 0)
# Stated as its own check because it is why the defect was invisible: nothing
# about a conflicted pull request makes any git fetch fail on this server.
check("the merge step REFUSES (non-zero)", mg3.returncode != 0)
check("...and says nothing was tested, not that a test failed",
"REFUSED" in mg3.stdout and "NOTHING HAS BEEN TESTED" in mg3.stdout)
check("...and never claims to have graded anything", "GRADED:" not in mg3.stdout)
check("...and leaves no half-merged tree behind for a later step to grade",
not (Path(ws3) / "goodwe-addon" / "from_pr.txt").is_file()
or git(Path(ws3) / "goodwe-addon", "rev-parse", "HEAD") == base_sha)
# ---------------------------------------------------------------------------
print("\n6. the run says which content it graded (CI-02 AC2)")
# ---------------------------------------------------------------------------
check("the mergeable run announces the merge it graded", "GRADED:" in mg.stdout)
check("...naming the base branch it merged into", BASEREF in mg.stdout)
check("...and the tree, which is the only unambiguous name for content",
"tree" in mg.stdout and git(t2, "rev-parse", "HEAD^{tree}")[:12] in mg.stdout)
# The no-pull-request path (workflow_dispatch) keeps a fallback, because a
# branch really can vanish between the dispatch and the fetch. It has to say so.
disp = run_script(CO, tempfile.mkdtemp(prefix="test_ci_gate.ws4."), origin2, pr2,
extra={"PRNUM": "", "GITHUB_REF": f"refs/heads/{BASEREF}"})
check("the no-pull-request path still checks something out", disp.returncode == 0)
check("...and announces that it is NOT merged content",
"GRADED:" in disp.stdout and "NOT merged content" in disp.stdout)
if disp.returncode != 0:
print(disp.stdout[-600:], disp.stderr[-600:])
for d in tmpdirs:
shutil.rmtree(d, ignore_errors=True)
print()
if fails:
print(f"{len(fails)} FAILED: " + "; ".join(fails))
sys.exit(1)
print("all checks passed")