ci(signal): name the job that ejected a PR, and run the ci-red reader nothing invoked

[wshallwshall]

Two measured defects in CI signalling. Neither is speculative; the command that proved each is in its commit.

1. A step promised a job name and emitted a workflow name

failure-signal.yml's step "Say which job ejected it, when the red came from the merge queue" interpolated github.event.workflow_run.name — the WORKFLOW name. Every merge-queue ejection comment read "CI failed ...", which the reader already had from the label and the checks tab. The title and the behaviour disagreed.

git show origin/main:.github/workflows/failure-signal.yml
# step "Say which job ejected it ..." -> RUN_NAME: github.event.workflow_run.name

BACKLOG #1403's banner reads closed over that gap. This is its unmet acceptance.

It now names the leg and its first failing step:

CI failed in web console tests (windows-2025, py3.14) / Step margin -- web console suite while this pull request was in the merge queue, so the queue ejected it.

It refuses to name CI gate — the roll-up fails in every red run and its own failing step points at a leg it does not name. That is the same refusal scripts/ci/report_ci_red.py makes, and a test now compares the two instead of trusting them to agree.

The token question, resolved rather than assumed

A previous investigation left this unresolved: the jobs endpoint answers unauthenticated, but nobody had shown it answers for a scope-less GITHUB_TOKEN. I measured it on throwaway run 35171248441, a workflow carrying this job's exact permissions block minus the new line:

probe	result
runner's own GITHUB_TOKEN Permissions	Issues: write, Metadata: read, PullRequests: write — no Actions entry, so the scope really was none
GET /actions/runs/<id>/jobs with that token	200, job names returned
GET /actions/runs?status=failure with that token	200, runs returned
same URL, no token at all	200
control: GET /actions/permissions	refused — so the successes are not a probe blind to 403s

It answers. actions: read is declared anyway, alongside the existing pull-requests: write and issues: write. The reason is in the same response: X-Accepted-Github-Permissions: actions=read. GitHub states the requirement and serves it without the scope only because this repository is public. Resting the attribution on that would make the job name vanish the day the repo turns private or GitHub enforces its own declaration — silently, with the comment still posting. That is this defect wearing the same clothes, found months later. It is a read scope and cannot modify code.

Why a Python program in env: and not inline jq

The rule has to be tested. This workflow may not check the repository out (the zizmor dangerous-triggers suppression rests on it running no third-party code), so it cannot import the identical rule from report_ci_red.py. Held in env: it is a plain string the tests read out of the file and run — so what is exercised is the bytes that ship, not a copy beside them.

It fails soft, and says which way

An ejection comment is the only record there is. A refused fetch degrades to the old workflow-only text rather than posting nothing, and a rule that crashed gets its own annotation — distinct from "no job failed", which a bare || true would have made indistinguishable. All four paths were run end to end against the shipped shell with a stubbed gh.

2. A written, tested reader that nothing ran

scripts/ci/report_ci_red.py landed 2026-09-04 with tests/test_ci_red_reader.py beside it. No workflow invoked it.

git grep -l "report_ci_red" origin/main -- .github   ->   no output, exit 1

Meanwhile failure-signal.yml writes the ci-red label and nothing triggers on it. Written and never read — BACKLOG #1385's residual 3, verbatim. Every test in that file passed the whole time, because they test the rule and nothing tested the delivery.

ci-red-report.yml runs it on a 07:35 UTC cron plus dispatch, copying required-workflow-state.yml's shape. It holds no required status context, and a test asserts it never gains one.

--warn-only is a measurement, not a preference

Measured 2026-09-16: 22 of the 41 open pull requests carry ci-red, and nothing removes the label — the writer only ever adds it. A cron exiting 1 on a finding would be red from the day it landed and every day after, which is decoration. So the job rules on whether it could measure, not on what it found — the same split quality-advisory.yml's liveness job makes. It does not touch the fail-closed path: an unreadable query still exits 2 and reds the job, and a new test pins that.

Run live before wiring: 22 pull requests, 3 reddened by a merge_group run their own PR page shows as green, 4 unattributed with the single-page truncation caveat firing correctly.

Mutation proofs

Eleven mutations, each reverting one claim and confirming the test that asserts it goes red, with the baseline green after restore.

mutation	test that reddened
delete ci-red-report.yml (the pre-fix state of main)	test_a_workflow_actually_runs_this_script
point the invocation at a missing script	same
promote the reader to a required context	test_the_workflow_that_runs_it_holds_no_required_context
--warn-only swallows exit 2	test_warn_only_still_fails_closed_on_a_query_it_could_not_run
stop fetching the run's jobs (the pre-fix behaviour)	test_the_attribution_reads_the_jobs_of_the_run_it_is_commenting_on
drop the roll-up refusal	..._never_names_the_roll_up_when_a_real_leg_failed
name the job but not its step	..._names_the_failing_job_and_step
count a cancelled sibling as a cause	test_a_cancelled_sibling_is_never_named_as_the_cause
drift the roll-up name from the reader's	test_the_roll_up_the_workflow_refuses_is_the_one_the_reader_refuses
unguard the jobs fetch	..._still_posts_a_comment_and_says_what_is_missing
revoke actions: read	exact-block test only — write-set test stays green
add contents: write	write-set test, on its own

The last two are why test_it_is_least_privilege_and_cannot_modify_code is split in two. It used to pin the write set and the exact block in one assertion, so granting any scope re-blessed the security claim in the same keystroke.

One test was mutation-proven useless and fixed. The first wiring test grepped the whole workflow file and survived deleting the invocation — the new file's own header names the script five times. It now reads run: bodies only. Prose about running a script satisfying a check for running it is this PR in miniature.

Known residual — recorded, not fixed

failure-signal.yml watches CI, Security and backlog-hygiene. It does not watch the three advisory crons that rule on CI's own health: required-workflow-state.yml, the new ci-red-report.yml, and stalled-prs.yml. A red one reaches nobody — .github/required-contexts.txt already records that happening for about a day.

Written as prose in that file's header, not as a not-watched: line: that register is scoped to REQUIRED workflows, and an exclusion naming a non-required one reds test_a_recorded_exclusion_names_a_real_workflow_that_really_is_required. Same precedent as CodeQL's exclusion. Adding them to the watch list is not the fix either — the label says a REQUIRED check went red, which is false of an advisory cron. Closing this needs a second signal whose text is true.

Verification

gate	result
ruff format --check .	1310 files already formatted
ruff check .	All checks passed
mypy messagefoundry messagefoundry_webconsole --exclude 'messagefoundry/tray/'	no issues, 293 files
mypy --platform win32 messagefoundry	no issues, 275 files
the four named test files	87 passed
all 57 test files that read .github/workflows	1064 passed, 25 skipped
packaging/messagefoundry-webconsole/tests	505 passed, 3 skipped

14 failures in test_ci_retry_native_crash.py and test_nightly_notice.py are pre-existing on this Windows box — the identical 14 fail on a clean origin/main checkout of the same worktree, confirmed by stashing.

No branch protection changed. No required context added. No BACKLOG number allocated — #1385 and #1403 are cited only.

🤖 Generated with Claude Code

[wshallwshall]

Two stale line references found while building #1242, in the two files this PR edits. Recording them here rather than fixing them, because #1242 deliberately touches neither file and correcting them from that branch would manufacture the conflict it was shaped to avoid. Both measured on origin/main today; neither is urgent and neither blocks anything.

1. .github/workflows/failure-signal.yml:15 — "the required contexts span five of them". They span four.

backlog-hygiene.yml: 1
ci.yml: 4
cla.yml: 1
security.yml: 9
distinct workflows = 4, total contexts = 15

Resolved through tests/_workflow_contexts.resolve() over .github/required-contexts.txt. The sentence is load-bearing in context — it is the argument for workflow_run over a per-workflow if: failure() — and that argument holds at four just as well as at five.

2. tests/test_ci_red_reader.py:146 cites failure-signal.yml:51 for the conclusion gate. It is at line 99.

if: github.event.workflow_run.conclusion == 'failure' is line 99 on origin/main. Line 51 is inside the header prose.

---

One interaction worth knowing about, since it lands on this PR's header rather than its code. ci-red-report.yml's header states as fact that nothing removes the label, and justifies --warn-only on that basis. #1242 adds scripts/ci/clear_stale_ci_red.py, so whichever of the two merges second should refresh that paragraph.

The --warn-only decision itself survives, and the measurement supports it more strongly than the header claims. Measured 2026-09-17 against the live repository: of the 20 labelled open pull requests, 17 have a required context failing on their current head right now and one is a live merge-queue ejection whose own head is green (#1229, mergeStateStatus=CLEAN, 42 green check runs). Only three are stale. So a cron exiting 1 on a finding would still be red most days — not because the label is unclearable, but because it is mostly earned. That is a better reason for --warn-only than the one currently written, and it does not depend on whether a remover exists.