Conversation
…ubstrings (#3839) libtest's --skip is a substring match on the FULL test path. `make coverage` passed 19 bare substrings, which removed 2,713 lib tests from the instrumented run (all=86,807, kept=84,094, measured with --list at fc942f6). 1,352 of them matched only on a MODULE path, never the test's name. By pattern: gpu_ 1,147 (includes every wgpu_*), falsification 1,084 (whole modules), chaos 135, load_test 112, cuda 106, heavy 67, slow 48, disconnect 29. Their code then read as uncovered, so the gate measured a subset over the whole denominator, and #2307 / pmat --coverage-gaps ranked well-tested modules as the top gaps. Measured which of them need skipping: all 2,700 unique names run under nextest on intel (no working NVIDIA driver), 30 s slow period, 120 s terminate. 2,702 passed, the slowest in 41 s, 121 s wall. The ONE timeout is apr-cli test_llm_band::cuda_without_the_server_feature_is_refused. It is not a GPU test: build_provenance hashes current_exe twice per call, and the test calls it twice, i.e. four passes over a 494 MB debug binary. The skip list is now scripts/coverage-skips.txt, one exact path per line, each with its measured reason, passed as `--exact --skip <path>`. Checked on the real apr-cli binary: all 7,416 → exact entry 7,415; `--exact --skip gpu_` 7,416 (cannot over-match); old substring gpu_ 7,343. COV_FLOOR is unchanged. Coverage before/after numbers follow in the next commit, measured with the same instrumented build. Refs #3839 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 3cc4c3c)
…ing test that blanked the nightly (#3839) 1. coverage, coverage-html and coverage-full renamed ~/.cargo/config.toml for the whole run (the bashrs "mold breaks LLVM coverage" workaround). That file is read by every cargo process of the user, so on a shared host it changed other agents' builds mid-run. No fleet host has one (lambda, intel, yoga, gx10 checked), so the move was a no-op. COV_REFUSE_GLOBAL_MOLD refuses to measure when a global config enables mold, and never touches the file. Case table through make with a throwaway CARGO_HOME: mold config -> refused and left in place; plain config -> runs; none -> runs. The table also caught a `#` in the message that truncated the make variable into a shell syntax error. 2. aprender-zram-core benchmark::tests::test_f060_no_performance_regression asserts two LZ4 runs agree within 15x. Under instrumentation on yoga it measured 26.21x and failed Coverage Nightly 35800448700, so no lcov was written and no number was reported. Added to scripts/coverage-skips.txt with that reason. The path is unique across all 69 lib binaries; it still runs in the uninstrumented test gates. Refs #3839 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit a2833e4)
…measure" is never a verdict (#3839) The 2026-09-23 Coverage Nightly wrote no lcov because one timing test failed, and reported nothing. The 0.69.1 rehearsal died the same way on a receipt-coupled test and printed "[FAIL] coverage" with no percentage. - cargo llvm-cov test --ignore-run-fail: a failing test no longer blanks the measurement. Output is teed to target/coverage/test.log, and rc=${PIPESTATUS[0]} is llvm-cov's own status, not tee's. - Failing tests are extracted to target/coverage/failed-tests.txt, printed with a count, and appended to summary.txt. Nothing is swallowed: every test here also runs in CI's workspace-test, which fails on it (checked for the one failure seen today, release_section_doc, which fails uninstrumented too). - "DID NOT MEASURE", exit 1, with no percentage: llvm-cov non-zero (build or report failure), no lcov.info, or LF=0. A stale lcov.info from an earlier run is deleted first, so it can't be re-read. - Below COV_FLOOR still exits non-zero. COV_FLOOR is unchanged at 88. Case table through `make coverage` with a stub cargo on PATH: measures 90% with 1 failing test -> rc 0, failure listed measures 80% -> rc 2, REGRESSION llvm-cov exits 101 -> rc 2, DID NOT MEASURE no lcov written -> rc 2, DID NOT MEASURE stale 99% lcov from an earlier run -> rc 2, DID NOT MEASURE Refs #3839 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 2fe5858)
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
Contributor
Author
|
Proof run Caveat: 76% is not a baseline yet. 🤖 Generated with Claude Code |
…n-fail swallowed it (#3839) Proof run 35868368976 (this branch, yoga): earlyoom SIGTERMed `realizar` (aprender-serve --lib, VmRSS 25752 MiB of 28 GB, yoga journal 14:04:02Z) after 15,950 of 16,022 tests. --ignore-run-fail treated the killed binary like a failing test, llvm-cov wrote no profile for it, and the recipe printed "76% ... REGRESSION" with the largest crate missing: a partial number presented as a verdict. scripts/check_coverage_log_complete.sh refuses a verdict when any test binary died by signal (`process didn't exit successfully: ... (signal: N`). An ordinary failing test exits with a status and is still listed and measured, as #3839 intends. Case table (--self-test): SIGTERM and SIGKILL lines flagged, `exit status: 101` and a clean log pass, 4/4. On the real run log it names `realizar-5fd3723e2da2d1b3 (signal: 15, SIGTERM)`. bashrs: 0 errors; Makefile lint count unchanged (14). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… so none is OOM-killed (#4023) aprender-serve's lib tests build up memory across tests in one process (#4028). Measured on gx10, uninstrumented, peak RSS by --test-threads: 22 → 45.3 GB, 8 → 43.7, 4 → 40.4, 2 → 36.8, 1 → 30.1. MALLOC_ARENA_MAX=2 changes nothing. On yoga (28 GB) earlyoom SIGTERMed the instrumented binary at 25.7 GB (run 35868368976). No thread cap can fit it. `make coverage` now runs: 1. the workspace lib tests except aprender-serve, as before (--no-report); 2. aprender-serve's lib tests split by top-level module into 9 processes (scripts/coverage_serve_shards.py: a module is never split; modules with >= 1000 tests run alone, the rest are packed <= 2000; the partition is CHECKED so every non-skipped test is in exactly one shard, else it exits 1). Each shard runs `--exact <its names>`, so the skip list stays exact (#3839). Measured on gx10 at 22 threads: every shard peaks <= 7.8 GB, and the 9 shards pass 15,913 tests, the same as one process; 3. ONE `cargo llvm-cov report` merging every run. Every run MUST be --no-report. Measured with a two-module scratch crate: a second run WITH a report cleans the first run's profiles (only_a hits 1 → 0), which would silently cover only the last shard. The killed-binary guard (previous commit) keeps any remaining OOM loud. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Proof run
Why it OOMed: on yoga, wgpu uses the real RTX 4060, and the long-context, streaming-KV and large-vocab GPU-path tests allocate large host-side buffers at 22 threads. My pre-dispatch measurement on gx10 (≤7.8 GB per shard) was uninstrumented, on unified-memory hardware, so it did not predict yoga. The next step, capping threads for the gpu shard only or skipping named tests with measured reasons, is waiting on the release cops. 🤖 Generated with Claude Code |
…solo with measured peak RSS (#4023) Proof run 35881004821 (02fe2e0): shards quantize/gguf/api exited normally, but the `gpu` shard (1,280 tests) was SIGTERMed by earlyoom on yoga at 25,855 MiB, where wgpu uses the real RTX 4060. The killed-binary guard held (tracking issue: "Line coverage: unknown"). My gx10 pre-measurement (<= 7.8 GB) was uninstrumented, on unified memory, so it did not transfer. - COV_GPU_SHARD_THREADS ?= 4: only shard-NN-gpu.txt runs with --test-threads=4 (cop decision (a)). - scripts/coverage-solo.txt: 11 tests that completed just before the kill, plus two gx10 suspects, each run in its OWN process. The recipe prints `coverage-solo-maxrss <KB> <test> rc=N` (RUSAGE_CHILDREN.ru_maxrss of the test binary; checked locally: 316032 KB for a 300 MB allocation two process levels down, rc passed through). Any later skip then has a measured reason. - coverage_serve_shards.py: shard files are named by module (shard-03-gpu.txt); solo tests become solo-NN.txt; a solo entry that is not a listed test is an error; the exactly-once check covers shards + solos. Real list: 9 shards (gpu=1269) + 11 solo, 15,971 tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…first; run everything before refusing (#4023) Run 35885731831 (4a266eb): the gpu shard at --test-threads=4, with the 11 heavy tests moved to solo, was STILL SIGTERMed by earlyoom on yoga at 26,514 MiB (journal 16:35:34Z, cmdline `--exact --test-threads=4 gpu::adapters::apr::...`). So thread count is not the lever; the gpu module builds up memory within one process on yoga (#4028). And because the shard loop stopped at the first failure, the solo tests never ran and no per-test RSS was measured. - coverage_serve_shards.py: DEEP = {"gpu"} is chunked into processes of <= 200 tests (gpu.00 .. gpu.06), which still run at COV_GPU_SHARD_THREADS. - Solo tests now run FIRST, so their `coverage-solo-maxrss` lines are always captured. - Every solo and every shard runs even when one fails. Failures go to target/coverage/failed-runs.txt, and ONE check afterwards refuses the verdict, naming each failed run. A dispatch now yields the whole picture instead of stopping at the first OOM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…report covers only the root facade (#4023) Proof run 35892421393 (7c5ffda) is the first where EVERY aprender-serve run exited normally: all 11 solo tests and all 15 shards returned rc 0, with no earlyoom kill on a shared yoga. Then `cargo llvm-cov report` printed "Finished report saved to target/coverage/lcov.info", and the parse found no non-empty lcov. Cause, reproduced with cargo-llvm-cov 0.9.0 (the version CI runs; 0.8.5 behaves the same) on a workspace whose root is also a package, like this repo's `apr` facade: an unqualified `report` covers only the root package (lcov SF = src/lib.rs); `report --workspace` covers every member. The single-invocation recipe passed --workspace to `test`; splitting out `report` dropped it. Now `report --workspace --exclude aprender-gpu`, the same package set the test runs cover. The report path is absolute, and the lcov size, lcov files and profraw count are logged after it. Also measured in that run: every solo test's peak RSS is ~3.3 GB (`coverage-solo-maxrss 3.31-3.33 GB` for all 11), i.e. no single test is heavy. The OOMs come from build-up across tests in one process (#4028), and skipping any of them would be unjustified. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main's
coverage-nightlyhas produced no coverage number for 25 days (run 35800448700).benchmark::tests::test_f060_no_performance_regressionpanics under llvm-cov, and one failing test blanks the entire run. The release tree already fixed this under #3839. This PR ports that fix exactly: threegit cherry-pick -xcommits, no other changes.8e3bbea3b(from 3cc4c3c): skip tests by exact test path, read from a reasoned list inscripts/coverage-skips.txt, instead of 19 substring matches.edff015ba(from a2833e4): nevermv ~/.cargo/config.toml; exact-skip the timing test (test_f060) that blanked the nightly.cbcec709f(from 2fe5858):--ignore-run-fail, so failing tests are listed and no longer blank the number. "Did not measure" is never reported as a verdict.What changes: only
Makefileand a newscripts/coverage-skips.txt. After the port, thecoverage:recipe and the skip list are byte-identical torelease/0.69.1-batch-2..github/workflows/coverage-nightly.ymlis already identical on both branches and callsmake coverage, so no workflow edit is needed.Checks:
scripts/check_coverage_has_producers.sh: PASS.make -n coverageexpands to--ignore-run-failplus--exact --skip <list>.bashrs make lint: 14 findings, the same count as on main.workflow_dispatchof coverage-nightly on this branch, run 35868368976. Its result will be added here.This is the baseline for ratchet pillar A (#3997/#4003). Do not arm auto-merge until 0.69.1 ships.
🤖 Generated with Claude Code