fix(mail): sweep every box, bounded per pass, and say when a pass stopped short - #1245
Merged
Merged
Conversation
added 4 commits
September 17, 2026 10:56
The retention sweep read only the current worktree's box. A box is keyed by
worktree, a worktree is removed once its work lands, and a removed worktree's
box is never drained again -- so the rule only ever reached the boxes whose
contents were newest, and every other box kept everything forever.
Measured read-only on the live spool 2026-09-17, over 188 boxes: 23,904 files
in seen/ and expired/ are already past a rule written to delete them, plus 36
markers. receipts/ is one flat directory for the whole queue that no box-scoped
loop could reach at all, and holds 33,121 aged files of 33,474.
receipts/ is now swept under three guards, because a receipt is the only record
of what was observed about a message:
- its message is still in inbox/, claiming/ or stranded/ in any box (13),
- something outside mail/ and _retired-2026-08-22/ quotes its stem (16),
- the keep set is read before any delete, so a receipt and the message it
describes are never removed by one pass (23,910 held this pass).
A guard that cannot be evaluated keeps every receipt, and the counter block
says so rather than rendering as a sweep with nothing to do.
inbox/, claiming/ and stranded/ stay out of the sweep. At the same reading 285
undelivered messages and 6 stranded records sit inside the window's reach.
Tests, section 8g: one planted survivor per guard, a file that must be deleted
beside them, a build with the guard removed in which every survivor dies, and
fault injection for the fail-closed path.
… owner
Follow-ups from the /simplify review of the previous commit. Two are behaviour,
the rest remove copies.
THE CITATION GUARD WAS PINNED TO ONE DATE. It excluded the literal
`_retired-2026-08-22`, but scripts/coord/handoff.ps1 mints
`_retired-<yyyy-MM-dd>` on every -Retire, so the next retirement would have
produced a frozen tree the guard reads. Every stem quoted inside it would pin
its receipt forever while the counter line went on reporting a sweep that ran.
It now matches the `_retired-` family as a whole path segment, so a live note
called `_retired-notes.md` is still read. Both directions are planted: the test
tree's frozen directory is dated 2099, which only a family match can exclude.
AN ABSENT RECEIPT STOPPED MEANING ONE THING the day the sweep shipped, and
mail.ps1 still told every sender it meant "no drain has run yet". That is a
false statement rather than a missing one -- the failure the receipt guards
exist to prevent, one file over. Both it and docs/SESSION-MAIL.md now name the
second cause.
The copies removed:
- Test-MailStem is composed from Get-MailStemPattern rather than carrying a
second literal of the shape, so the two cannot drift and the test that
policed them is replaced by one that pins what is left: that the pattern
finds a stem quoted mid-sentence, and that the planted names really are
names this channel mints.
- Split-ReceiptFileName joins the other two name validators, so the <stem>.json
shape is decoded in one place instead of inline in the sweep.
- run_drain takes the script to run, so the control build no longer needs a
third copy of the drain-invocation block and its fail-open assertions.
- age_out replaces three inline copies of the ageing arithmetic.
- The candidate list is a List rather than `+=`, which is quadratic on the
one pass that is largest; the citation walk reads text once rather than
bytes-then-string.
…ped short The widened sweep gave its first pass a five-week backlog, against a hook that is killed at 20 seconds. Measured read-only on the live spool at 0bb2605, over 188 boxes: 23,861 aged and minted files in seen/, 52 in expired/, 36 markers in shown/, and 9,210 receipts still deletable after the three guards -- 33,159 files in one pass. The registration is "timeout": 20 on SessionStart and on Stop, in the settings.json of all six config roots, twelve rows, every one of them 20. THAT IS NOT DATA LOSS. The sweep is idempotent and every guard fails closed, so a killed pass resumes. What breaks is quieter. The kill lands at a turn boundary, so the counter line never renders at all -- no partial output, no sentence saying the pass was cut -- and a pass stopped inside the MESSAGE loop never reaches the receipt sweep. receipts/ is the largest class, so the record that grows fastest is the one that silently never drains. Three per-pass delete budgets, one per deleting phase: seen/ and expired/ across every box, shown/ markers across both marker sweeps, and receipts/. EACH PHASE GETS ITS OWN RATHER THAN A SHARE OF ONE. A pooled budget spent entirely on seen/ leaves receipts/ unreached, which is the defect rather than a smaller version of it. Independent budgets also make the worst case the SUM, which is the number there is any point sizing against the timeout. THE ARITHMETIC SITS AT THE CONSTANTS, measured on this machine 2026-09-17: 0.77 ms for the Remove-Item plus Test-Path pair over 3,000 files, 709-998 ms to build the keep set over 188 boxes, 729-929 ms for the receipt scan over 33,480 files, 706 ms warm for the citation walk. At 1000 + 250 + 1000 deletes a pass costs 7.0s of the 20s timeout; at five times the benchmark, 14.0s. The benchmark ran in a throwaway temp directory with no on-access scanner in the path, which is why it is sized against a multiple of itself. A BOUNDED PASS SAYS SO, AND NAMES ITS PHASE. "Swept 1,000" and "swept 1,000, and did not reach the end" are different facts about the queue, and a hook killed at its timeout prints neither -- so the only thing that can report a bounded sweep is a sweep that chose to stop. The flag means "did not reach the end of its input" exactly: it is set where a file or a box was left UNEXAMINED, never by comparing a counter to its cap, so a pass whose last delete was its last candidate is finished and stays silent about it. The receipt phase budgets its DELETES and not its scan. A guard-protected receipt stays a candidate until its guard releases it, so capping the candidate list would let a block of protected receipts at the front of the enumeration fill it on every pass and hide everything behind them forever. Nothing protects an aged, well-named file in seen/, so that loop may stop where it stands. Tests, section 8h: a pile larger than one pass may remove, asserting the remainder survives and the sentence prints; a second pass that clears the remainder and stops printing it; the receipt sweep still reaching an unguarded receipt when the message phase stops short; and a build with the budgets lifted in which the whole pile dies and no truncation line appears. Both behavioural arms were run against the drain at 0bb2605 first and were red there.
…d reading 0bb2605 replaced a hard-coded `_retired-2026-08-22` in the citation guard with a `_retired-` prefix matched as a whole path segment, because scripts/coord/handoff.ps1:310 builds "_retired-$stamp" from the current date on every -Retire. docs/SESSION-MAIL.md went on describing the exclusion as "the frozen `_retired-2026-08-22/` tree", so the page named a date no code reads. That is a documented claim that no longer matches its code -- the same defect class the guard was rewritten to close. The page now states the RULE: the `_retired-` family, matched as a path SEGMENT, so a live file called `_retired-notes.md` is still scanned and the stems in it stay live citations. The old wording is quoted in the retraction rather than swapped out silently. A pointer and the thing it points at are two edits, and nothing fails when only the first one is made. C8'S MARGINAL PROTECTION IS 1 RECEIPT, NOT 16, and the page now says so. Measured read-only on the live spool at 0bb2605, over 188 boxes: the citation walk finds 16 cited stems among the aged receipts and 15 are already held by C5, whose message is still in a box. The previous commit's message published 16 as C8's contribution, which overstates it. An overstated guard invites the next reader to delete it the day they measure the real figure; one file is still the right answer, because a dangling citation is the failure nothing else reports.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The retention sweep was scoped to the current worktree's own box, so a box whose worktree is gone
never drained. Measured before this branch: 23,812 of 24,205
seen/files were already past a rulemeant to delete them.
Widening it is correct and it means the first pass has a five-week backlog to clear. That is the rest
of this branch.
The first widened pass would have been killed, and taken the mail with it
mail-drain.ps1is registered onSessionStartandStopwith"timeout": 20-- twelveregistrations across six config roots, read from each root's
settings.json.The first pass reaches 33,159 files: 23,861 in
seen/, 52 inexpired/, 36 markers, and 9,210receipts after the guards. At a measured 0.77 ms per
Remove-ItemplusTest-Path, that is about25.5s of deletes plus roughly 5s of fixed cost, against a 20s kill.
The consequence is worse than a slow sweep. Delivery is the LAST thing the drain does -- box loop,
receipt sweep, dead-owner sweep, marker-orphan sweep, then the delivery passes and the injection. A
pass killed inside the message loop delivers no mail at all that turn.
Three budgets, not one
seen/andexpired/, every box$MESSAGE_DELETE_BUDGETshown/markers$MARKER_DELETE_BUDGETreceipts/$RECEIPT_DELETE_BUDGETSeparate budgets are the structural fix for the failure being repaired: a
seen/backlog can neverspend the receipt phase's allowance, so the receipt sweep cannot be starved by the phase ahead of it.
2,250 deletes cost 1.73s, so a full pass is about 7.0s -- 35 percent of the 20s cap. Sized against
multiples because the benchmark had no on-access scanner in its path: 10.5s at 3x, 14.0s at 5x. The
backlog clears in 24 message passes and 10 receipt passes.
The receipt phase budgets its deletes, not its scan. Capping the candidate list would wedge: a
block of guard-protected receipts at the front of the enumeration would refill it every pass and hide
everything behind them forever.
A truncated pass says so, and a finished one stays silent
Next pass, same tree:
25 message(s) ... removed.The flag means "did not reach the end of its input", set only where a file or box was left
unexamined. It is never derived by comparing a counter to its cap, so a pass whose last delete
happened to be its last candidate is finished and says nothing.
Readings
Red first, against the drain at
0bb2605b4extracted to a throwaway file so both arms run the same way:Green after:
The third is a permanent planted control: budgets rewritten to 10,000,000, the whole pile dies,
and no truncation line prints. A bound sweep and a broken one cannot pass identically.
Full suite across both session-mail files:
122 passed in 182.59s, exit 0, against a 119 baseline.HEAD and porcelain identical before and after.
The guards, verified independently before this branch opened
Each of C5, C8 and never-both-halves has a planted survivor AND a must-delete plant beside it. The
branch ships its own negative control that rewrites
Test-ReceiptProtectedtoreturn $falseandasserts all six survivors die. A verifier separately mutated two guards and watched them go red, then
restored the file to a matching SHA256.
The state filter is byte-for-byte unchanged.
@('seen', 'expired')--inbox,claimingandstrandedare absent from both refs. That filter alone stands between this sweep and 285 undeliveredmessages plus 6 rescue records: with
inboxadded to the delete list, the C5 receipt test stillpassed while the messages it protects were destroyed. C5 does not backstop it.
Corrections carried in this branch
C8 protects 1 file today, not 16. It finds 16 cited stems; 15 are already held by C5. The "(16)"
appears only in an earlier commit message, which cannot be amended, so the accurate figure is now in
docs/SESSION-MAIL.mdbeside the guard, with the reason an overstated guard invites its own deletion.The docs named a date the code stopped reading. The exclusion became the
_retired-family,matched as a whole path segment, because
scripts/coord/handoff.ps1builds"_retired-$stamp"fromthe current date on every retire.
docs/SESSION-MAIL.mdstill said_retired-2026-08-22/.The condition not varied
Every figure is from this machine on 2026-09-17. The delete benchmark ran in a throwaway temp
directory with no on-access scanner, which is why the budget is sized against multiples of it rather
than against the number itself.