Skip to content

ci: run the test surface nothing else runs — hard-failing, no baseline - #3255

Closed
noahgift wants to merge 1 commit into
mainfrom
PMAT-1098-full-surface-nightly
Closed

noahgift wants to merge 1 commit into
mainfrom
PMAT-1098-full-surface-nightly

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

DRAFT until its dependencies land. This job hard-fails, so it must not merge before the
five PRs that take the failing set to zero: #3250 (34 rows), #3248, #3238, #3253
(the last one), and #3241 (which adds scripts/cargo_step.sh, used in step 1). Merged
ahead of them it turns a nightly red on day one — the exact outcome the design argues against.

Implements the cadence in #3249.

What it runs, and why nothing else does

--tests appears exactly once in every other workflow — ci.yml's BSE-17 quick tier, scoped
to the crates a PR touched. make coverage, the only other whole-workspace run, is --lib.
nightly.yml runs no tests at all.

merge-queue full tier   82,217 tests /  73 binaries   (job 103868350604)
this job                98,621 tests / 727 binaries   (1141 s run, measured)

654 test binaries have never been linked in CI, let alone run. A binary that never links
cannot fail, cannot flake, and cannot appear in a report.

No baseline. Hard fail on day one.

The first draft proposed a shrink-only ratchet. A 4-lane agy quorum refused it, and the objection
is mechanical rather than stylistic: --update writes a file and needs a commit, and a scheduled
workflow has no PR and no author.
Every other ratchet here is maintained by a human inside a PR
because it runs in one. A cron's baseline would only ever be edited by someone who already
noticed the job was red — the exact population a ratchet exists to stop relying on.

So the 37 failures are fixed first and this ships with nothing to maintain.

Not on gx10 — and the mechanism is measured

The build produces a 39 GB target directory (808 binaries). gx10 is the smallest of the three
Linux hosts — 916 GB against intel's 3.6 TB and yoga's 935 GB — and it hit zero bytes free on
2026-09-14, dequeuing a PR from merge-queue position 1 (#3247).

X64 is the exclusion: over 25 runs, no job requesting X64 has ever landed on gx10 (it is
ARM64). No new label.

Four refusals, each earned by a defect already paid for here

step why
cargo_step.sh --self-test first a sweep of 727 binaries is exactly where an unclassified cargo death reads as 650 new defects
preflight ≥ 120 GB free fail ENV: before the build, not half-way through — that is how #3247 presented
--no-fail-fast plain cargo test stops at the first failing binary; seven rounds went into four tests
vacuity floor, 600 binaries a selection that built nothing reports zero failures, which reads exactly like a pass

It reclaims the 39 GB in-job under always(), because the host reaper's critical mode already
cannot reclaim under keep=3 (#3247) — leaking there compounds a known-unfixed defect.

The timeout is honest about being a guess

timeout-minutes: 90, deliberately generous and not derived. BSE-05 wants
T = max(15, ceil(1.5·p99), p99+20) over the job's own history and this job has none. The only
reading in hand is 21 min on a 48-core box with a warm sccache; a clean-room runner measured
2.34× slower per test on the same workload, extrapolating to ~51 min — an extrapolation, not a
measurement. Re-derive from three real runs and tighten.

Also carries the selector-touch trigger

ci_test_tier.sh, gate_touched_crates.sh, tree_reader_tests.txt, ci.yml. A change to how the
tier chooses what to run is exactly when the dark set moves.

Verified: yaml parses; check_no_hosted_runners, check_runner_labels,
check_perf_concurrency_groups, check_ci_reusable_workflow_pinned, check_no_timing_in_required,
check_cargo_install_private_root, check_hardcoded_paths, check_guards_are_wired and
check_tool_versions --audit-workflow all pass on the new file.

Refs #3239

no-close: #3239 stays OPEN until this is undrafted and has run green three times — a workflow that
has never fired is a claim, not a gate.

🤖 Generated with Claude Code

`--tests` appears exactly ONCE in every other workflow: ci.yml's BSE-17 quick
tier, scoped to the crates a PR touched. `make coverage` — the only other
whole-workspace run — is `--lib`. nightly.yml runs no tests at all. So 654 test
binaries have never been LINKED in CI, let alone run, and a binary that never
links cannot fail, flake, or appear in a report.

    merge-queue full tier   82,217 tests /  73 binaries   job 103868350604
    this job                98,621 tests / 727 binaries   1141 s run, measured

NO BASELINE, HARD FAIL ON DAY ONE. The first draft of APR-DARK-TARGETS-001
proposed a shrink-only ratchet. A 4-lane agy quorum refused it and the objection
is mechanical: `--update` writes a file and needs a COMMIT, and a scheduled
workflow has no PR and no author. Every other ratchet here is maintained by a
human inside a PR BECAUSE it runs in one. A cron's baseline would only ever be
edited by someone who already noticed the job was red — the exact population a
ratchet exists to stop relying on. So the 37 failures are FIXED first, in five
PRs (#3250 34 rows, #3248, #3238, #3253), and this ships with nothing to
maintain.

NOT ON gx10, and the mechanism is measured rather than declared: the build
produces a 39 GB target dir (808 binaries), gx10 is the smallest of the three
Linux hosts at 916 GB against intel 3.6 TB and yoga 935 GB, and it hit ZERO BYTES
FREE on 2026-09-14 — dequeuing a PR from merge-queue position 1 (#3247). `X64` is
the exclusion: over 25 runs, no job requesting X64 has ever landed on gx10. No
new label.

Four things this job refuses to do quietly, each earned by a defect already paid
for here:

  cargo_step.sh --self-test FIRST   a sweep of 727 binaries is exactly where an
                                    unclassified cargo death reads as 650 defects
  preflight >= 120 GB free          fail ENV: BEFORE the build, not half-way
                                    through it — that is how #3247 presented
  --no-fail-fast                    plain cargo test stops at the first failing
                                    BINARY; seven rounds went into four tests
  vacuity floor, 600 binaries       a selection that built nothing reports zero
                                    failures, which reads exactly like a pass

and it reclaims the 39 GB in-job under `always()`, because the host reaper's
critical mode already cannot reclaim under keep=3 (#3247) — leaking there
compounds a known-unfixed defect.

`timeout-minutes: 90` is deliberately generous and is NOT derived. BSE-05 wants
T = max(15, ceil(1.5*p99), p99+20) over the job's own history and this job has
none; the only reading in hand is 21 min on a 48-core box with a warm sccache,
and a clean-room runner measured 2.34x slower per test on the same workload. That
extrapolates to ~51 min — an extrapolation, not a measurement. Re-derive from
three real runs.

Also carries the selector-touch PR trigger from the spec: ci_test_tier.sh,
gate_touched_crates.sh, tree_reader_tests.txt and ci.yml. A change to how the
tier chooses what to run is exactly when the dark set moves.

Verified: yaml parses; check_no_hosted_runners, check_runner_labels,
check_perf_concurrency_groups, check_ci_reusable_workflow_pinned,
check_no_timing_in_required, check_cargo_install_private_root,
check_hardcoded_paths, check_guards_are_wired and
check_tool_versions --audit-workflow all pass on the new file.

Refs #3239

Pmat-Ticket: PMAT-1098

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3255 head=685e348efcef9556f48e8eb3222b9614ffe0eb68 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

noahgift added a commit that referenced this pull request Sep 14, 2026
…ack out

The operator's challenge was right and I had argued from assumption: slowness is
what a nightly is FOR, and `.config/nextest.toml` already answers HANGING with
`slow-timeout = { period = "60s", terminate-after = 20 }` so "a genuine hang dies
at 20 minutes WITH A NAME". `check_examples_run` in aprender-orchestrate even
sidesteps stdin the obvious way: `cargo run --example X -- --help`.

So I wrote the clause. Then ran it on a random sample of five:

    HANG     aprender-core::grid_search_tuning
    HANG     aprender-serve::imp_700_realworld_verification
    exit101  aprender-serve::debug_q4k_gemv
    ok       aprender-simulate::reproducibility
    HANG     aprender-serve::bench_barrier_overhead

THREE OF FIVE ran past 60 s on `--help`. These are not CLIs — they are compute
demos that ignore argv and do the work. `--help` does not make them exit, and
"hang" is the wrong word: they are RUNNING. A per-example timeout would report a
working example as a FAILURE, at 980 rows.

Five is a small sample and the ratio is not the claim; the KIND is, and the three
names are self-evidently long-running demos.

So "every `cargo run --example`" cannot be implemented as the spec words it
against this workspace. Two shapes that could work are written into the header
for whoever decides: redefine as "starts and does not crash within N seconds"
with timeout treated as PASS (~2 h at N=10, so it belongs in #3255's
full-surface lane, not this 5-minute one), or have examples DECLARE runnability.

Clause 3 — every example BUILDS, 980 targets, 83 s — already catches the common
rot, which is compile breakage. That is the part that is cheap and unambiguous,
and it stays.

Also recorded: `check_examples_run` is worth a ticket of its own. Its only
callers are its own tests, and it fails OPEN — a non-zero runtime exit counts as
a pass because "--help exiting non-zero is OK", so a crash passes.

The feature matrix stays out and that one is genuinely undefined: cargo-hack is
not installed and nothing here references --each-feature, --feature-powerset or
any matrix tooling. No definition and no tool.

Pmat-Ticket: PMAT-1098

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift noahgift modified the milestones: 0.68.0, 0.70.0, 0.71.0 Sep 14, 2026
@noahgift

Copy link
Copy Markdown
Contributor Author

stale — APR-RELEASE-001 §6, applied mechanically 2026-09-20.

No activity since 2026-09-17 13:47Z; 2 release trains have been cut since (v0.68.2, v0.68.1). Age is the only input to this sweep — it is not a judgement on the work, and a train label such as pp-066/carry-* is not an exemption.

What happens next: if one more train passes while this is still labelled stale, §6 closes it. The branch is kept — closing a PR here discards a review thread, never commits.

To clear it: push, rebase, or say on the thread what it is waiting for. Any of the three removes the label at the next sweep. If it is blocked on something external, name that here — a blocker with an owner is not sprawl, and it stops the clock.

@noahgift noahgift added the stale APR-RELEASE-001 §6: >=2 release trains with no activity label Sep 20, 2026
@noahgift

Copy link
Copy Markdown
Contributor Author

closed by 0.70 batch triage: no quorum receipt and no live owner; branch kept, reopen with an owner

Authority: per operator order (quoted) "yes, clearn up. no sovereign repo can have triage, ticket, branche issues and arbiter shoudl be doing this! tell that agent to get to work" + cop (aprender-cf) ruling on the close criteria (a draft, or >7d old with no live owner, and no quorum receipt). The operator did not select this PR individually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale APR-RELEASE-001 §6: >=2 release trains with no activity

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1,424 of 1,469 integration targets run only when a PR touches their own crate — four of tonight's defects trace to this

1 participant