Skip to content

fix(mail): sweep every box, bounded per pass, and say when a pass stopped short - #1245

Merged
wshallwshall merged 4 commits into
mainfrom
guard-maildrain
Sep 17, 2026
Merged

wshallwshall merged 4 commits into
mainfrom
guard-maildrain

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

The retention sweep was scoped to the current worktree's own box, so a box whose worktree is gone
never drained. Measured before this branch: 23,812 of 24,205 seen/ files were already past a rule
meant to delete them.

Widening it is correct and it means the first pass has a five-week backlog to clear. That is the rest
of this branch.

The first widened pass would have been killed, and taken the mail with it

mail-drain.ps1 is registered on SessionStart and Stop with "timeout": 20 -- twelve
registrations across six config roots
, read from each root's settings.json.

The first pass reaches 33,159 files: 23,861 in seen/, 52 in expired/, 36 markers, and 9,210
receipts after the guards. At a measured 0.77 ms per Remove-Item plus Test-Path, that is about
25.5s of deletes plus roughly 5s of fixed cost, against a 20s kill.

The consequence is worse than a slow sweep. Delivery is the LAST thing the drain does -- box loop,
receipt sweep, dead-owner sweep, marker-orphan sweep, then the delivery passes and the injection. A
pass killed inside the message loop delivers no mail at all that turn.

Three budgets, not one

Phase Constant Value
seen/ and expired/, every box $MESSAGE_DELETE_BUDGET 1000
shown/ markers $MARKER_DELETE_BUDGET 250
receipts/ $RECEIPT_DELETE_BUDGET 1000

Separate budgets are the structural fix for the failure being repaired: a seen/ backlog can never
spend the receipt phase's allowance, so the receipt sweep cannot be starved by the phase ahead of it.

2,250 deletes cost 1.73s, so a full pass is about 7.0s -- 35 percent of the 20s cap. Sized against
multiples because the benchmark had no on-access scanner in its path: 10.5s at 3x, 14.0s at 5x. The
backlog clears in 24 message passes and 10 receipt passes.

The receipt phase budgets its deletes, not its scan. Capping the candidate list would wedge: a
block of guard-protected receipts at the front of the enumeration would refill it every pass and hide
everything behind them forever.

A truncated pass says so, and a finished one stays silent

1000 message(s) older than 7 days were removed from seen/ and expired/ across 2 box(es) examined.
The seen/ and expired/ sweep stopped at its per-pass budget of 1000 delete(s)
without reaching the end of its input. The rest is still on disk and the next drain resumes.

Next pass, same tree: 25 message(s) ... removed.

The flag means "did not reach the end of its input", set only where a file or box was left
unexamined. It is never derived by comparing a counter to its cap, so a pass whose last delete
happened to be its last candidate is finished and says nothing.

Readings

Red first, against the drain at 0bb2605b4 extracted to a throwaway file so both arms run the same way:

AssertionError: the pass did not stop at the budget, so the hook's 20-second kill is still reachable
AssertionError: the message phase never hit its budget, so this arm proves nothing
2 failed          exit: 1

Green after:

3 passed, 44 deselected in 13.33s          exit: 0

The third is a permanent planted control: budgets rewritten to 10,000,000, the whole pile dies,
and no truncation line prints. A bound sweep and a broken one cannot pass identically.

Full suite across both session-mail files: 122 passed in 182.59s, exit 0, against a 119 baseline.
HEAD and porcelain identical before and after.

The guards, verified independently before this branch opened

Each of C5, C8 and never-both-halves has a planted survivor AND a must-delete plant beside it. The
branch ships its own negative control that rewrites Test-ReceiptProtected to return $false and
asserts all six survivors die. A verifier separately mutated two guards and watched them go red, then
restored the file to a matching SHA256.

The state filter is byte-for-byte unchanged. @('seen', 'expired') -- inbox, claiming and
stranded are absent from both refs. That filter alone stands between this sweep and 285 undelivered
messages plus 6 rescue records: with inbox added to the delete list, the C5 receipt test still
passed while the messages it protects were destroyed. C5 does not backstop it.

Corrections carried in this branch

C8 protects 1 file today, not 16. It finds 16 cited stems; 15 are already held by C5. The "(16)"
appears only in an earlier commit message, which cannot be amended, so the accurate figure is now in
docs/SESSION-MAIL.md beside the guard, with the reason an overstated guard invites its own deletion.

The docs named a date the code stopped reading. The exclusion became the _retired- family,
matched as a whole path segment, because scripts/coord/handoff.ps1 builds "_retired-$stamp" from
the current date on every retire. docs/SESSION-MAIL.md still said _retired-2026-08-22/.

The condition not varied

Every figure is from this machine on 2026-09-17. The delete benchmark ran in a throwaway temp
directory with no on-access scanner, which is why the budget is sized against multiples of it rather
than against the number itself.

wshallwshall added 4 commits September 17, 2026 10:56
The retention sweep read only the current worktree's box. A box is keyed by
worktree, a worktree is removed once its work lands, and a removed worktree's
box is never drained again -- so the rule only ever reached the boxes whose
contents were newest, and every other box kept everything forever.

Measured read-only on the live spool 2026-09-17, over 188 boxes: 23,904 files
in seen/ and expired/ are already past a rule written to delete them, plus 36
markers. receipts/ is one flat directory for the whole queue that no box-scoped
loop could reach at all, and holds 33,121 aged files of 33,474.

receipts/ is now swept under three guards, because a receipt is the only record
of what was observed about a message:

  - its message is still in inbox/, claiming/ or stranded/ in any box (13),
  - something outside mail/ and _retired-2026-08-22/ quotes its stem (16),
  - the keep set is read before any delete, so a receipt and the message it
    describes are never removed by one pass (23,910 held this pass).

A guard that cannot be evaluated keeps every receipt, and the counter block
says so rather than rendering as a sweep with nothing to do.

inbox/, claiming/ and stranded/ stay out of the sweep. At the same reading 285
undelivered messages and 6 stranded records sit inside the window's reach.

Tests, section 8g: one planted survivor per guard, a file that must be deleted
beside them, a build with the guard removed in which every survivor dies, and
fault injection for the fail-closed path.
… owner

Follow-ups from the /simplify review of the previous commit. Two are behaviour,
the rest remove copies.

THE CITATION GUARD WAS PINNED TO ONE DATE. It excluded the literal
`_retired-2026-08-22`, but scripts/coord/handoff.ps1 mints
`_retired-<yyyy-MM-dd>` on every -Retire, so the next retirement would have
produced a frozen tree the guard reads. Every stem quoted inside it would pin
its receipt forever while the counter line went on reporting a sweep that ran.
It now matches the `_retired-` family as a whole path segment, so a live note
called `_retired-notes.md` is still read. Both directions are planted: the test
tree's frozen directory is dated 2099, which only a family match can exclude.

AN ABSENT RECEIPT STOPPED MEANING ONE THING the day the sweep shipped, and
mail.ps1 still told every sender it meant "no drain has run yet". That is a
false statement rather than a missing one -- the failure the receipt guards
exist to prevent, one file over. Both it and docs/SESSION-MAIL.md now name the
second cause.

The copies removed:

  - Test-MailStem is composed from Get-MailStemPattern rather than carrying a
    second literal of the shape, so the two cannot drift and the test that
    policed them is replaced by one that pins what is left: that the pattern
    finds a stem quoted mid-sentence, and that the planted names really are
    names this channel mints.
  - Split-ReceiptFileName joins the other two name validators, so the <stem>.json
    shape is decoded in one place instead of inline in the sweep.
  - run_drain takes the script to run, so the control build no longer needs a
    third copy of the drain-invocation block and its fail-open assertions.
  - age_out replaces three inline copies of the ageing arithmetic.
  - The candidate list is a List rather than `+=`, which is quadratic on the
    one pass that is largest; the citation walk reads text once rather than
    bytes-then-string.
…ped short

The widened sweep gave its first pass a five-week backlog, against a hook that
is killed at 20 seconds. Measured read-only on the live spool at 0bb2605,
over 188 boxes: 23,861 aged and minted files in seen/, 52 in expired/, 36
markers in shown/, and 9,210 receipts still deletable after the three guards --
33,159 files in one pass. The registration is "timeout": 20 on SessionStart and
on Stop, in the settings.json of all six config roots, twelve rows, every one
of them 20.

THAT IS NOT DATA LOSS. The sweep is idempotent and every guard fails closed, so
a killed pass resumes. What breaks is quieter. The kill lands at a turn
boundary, so the counter line never renders at all -- no partial output, no
sentence saying the pass was cut -- and a pass stopped inside the MESSAGE loop
never reaches the receipt sweep. receipts/ is the largest class, so the record
that grows fastest is the one that silently never drains.

Three per-pass delete budgets, one per deleting phase: seen/ and expired/
across every box, shown/ markers across both marker sweeps, and receipts/.

EACH PHASE GETS ITS OWN RATHER THAN A SHARE OF ONE. A pooled budget spent
entirely on seen/ leaves receipts/ unreached, which is the defect rather than a
smaller version of it. Independent budgets also make the worst case the SUM,
which is the number there is any point sizing against the timeout.

THE ARITHMETIC SITS AT THE CONSTANTS, measured on this machine 2026-09-17: 0.77
ms for the Remove-Item plus Test-Path pair over 3,000 files, 709-998 ms to
build the keep set over 188 boxes, 729-929 ms for the receipt scan over 33,480
files, 706 ms warm for the citation walk. At 1000 + 250 + 1000 deletes a pass
costs 7.0s of the 20s timeout; at five times the benchmark, 14.0s. The
benchmark ran in a throwaway temp directory with no on-access scanner in the
path, which is why it is sized against a multiple of itself.

A BOUNDED PASS SAYS SO, AND NAMES ITS PHASE. "Swept 1,000" and "swept 1,000,
and did not reach the end" are different facts about the queue, and a hook
killed at its timeout prints neither -- so the only thing that can report a
bounded sweep is a sweep that chose to stop. The flag means "did not reach the
end of its input" exactly: it is set where a file or a box was left UNEXAMINED,
never by comparing a counter to its cap, so a pass whose last delete was its
last candidate is finished and stays silent about it.

The receipt phase budgets its DELETES and not its scan. A guard-protected
receipt stays a candidate until its guard releases it, so capping the candidate
list would let a block of protected receipts at the front of the enumeration
fill it on every pass and hide everything behind them forever. Nothing protects
an aged, well-named file in seen/, so that loop may stop where it stands.

Tests, section 8h: a pile larger than one pass may remove, asserting the
remainder survives and the sentence prints; a second pass that clears the
remainder and stops printing it; the receipt sweep still reaching an unguarded
receipt when the message phase stops short; and a build with the budgets lifted
in which the whole pile dies and no truncation line appears. Both behavioural
arms were run against the drain at 0bb2605 first and were red there.
…d reading

0bb2605 replaced a hard-coded `_retired-2026-08-22` in the citation guard with
a `_retired-` prefix matched as a whole path segment, because
scripts/coord/handoff.ps1:310 builds "_retired-$stamp" from the current date on
every -Retire. docs/SESSION-MAIL.md went on describing the exclusion as "the
frozen `_retired-2026-08-22/` tree", so the page named a date no code reads.

That is a documented claim that no longer matches its code -- the same defect
class the guard was rewritten to close. The page now states the RULE: the
`_retired-` family, matched as a path SEGMENT, so a live file called
`_retired-notes.md` is still scanned and the stems in it stay live citations.

The old wording is quoted in the retraction rather than swapped out silently.
A pointer and the thing it points at are two edits, and nothing fails when only
the first one is made.

C8'S MARGINAL PROTECTION IS 1 RECEIPT, NOT 16, and the page now says so.
Measured read-only on the live spool at 0bb2605, over 188 boxes: the citation
walk finds 16 cited stems among the aged receipts and 15 are already held by
C5, whose message is still in a box. The previous commit's message published 16
as C8's contribution, which overstates it. An overstated guard invites the next
reader to delete it the day they measure the real figure; one file is still the
right answer, because a dangling citation is the failure nothing else reports.
@wshallwshall
wshallwshall added this pull request to the merge queue Sep 17, 2026
Merged via the queue into main with commit 852a3fa Sep 17, 2026
41 checks passed
@wshallwshall
wshallwshall deleted the guard-maildrain branch September 17, 2026 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant