Orphaned nextRenderTime index entries make the claim scan non-monotonic; one break at row 0 silently stalls a node's entire backlog (2 of 4 nodes affected)

[harper-joseph]

Summary

A claim pass that sees no due rows advances the floor to nowMinute - guard. That is correct on a
caught-up queue, but it is an unconditional forward jump with no way back: any row that later appears
below the floor is invisible to every subsequent pass, because the scan is a single
nextRenderTime >= floor condition. Nothing in the claim path can lower the floor again.

What makes the pass see no due rows is a single orphaned secondary-index entry pointing at a row
whose stored value has since moved forward. The walk visits the orphan, loads the record, and yields a
future value at the head of an otherwise ascending scan. The pass breaks on it at iteration 0 —
every pass, forever — and then ratchets the floor past everything below.

Observed in production as one node of four silently ceasing to render a growing slice of its shard:
92,873 rows stranded below the floor across 2,101 distinct due-minutes, oldest due 2026-08-01 — 18
days. Three orphaned entries were withholding ~93k rows of work.

Two defects:

1. Index maintenance leaves an entry that no longer describes the record. Upstream — see status below.
2. The claim path trusts the index ordering absolutely. One break on the first future row turns a
   single bad entry into a permanent, silent, whole-node queue stall.

Defect 2 is the one worth fixing. It is cheap, local to this plugin, and it makes the claim path
robust to any ordering violation regardless of who caused it. It is also still entirely present — see
"Where defect 2 lives today".

Upstream status — updated 2026-09-16

HarperFast/harper#2211, filed alongside this issue, is closed: fixed 2026-08-20 in efd5e535,
"keep same-key writes in staging order so a delete can't strip a live record's indexes". A replicated
delete K; put K transaction could leave the follower holding a live, correct record with none of
its secondary index entries.

That is not the signature measured here, and it is not established that the fix covers this one.
The state on this cluster was an entry at the record's old value and no entry at the new one — a
walk starting at the record's own nextRenderTime did not return the key, while a point read returned
the later value. That matches the updateIndices early-continue:

if (value === existingValue && !isIndexing) continue;   // no index work at all, in either direction

— the index untouched while the record moved forward — rather than a delete stripping entries on the
way through. So: re-verify against a Harper build carrying efd5e535 before treating defect 1 as gone.
Defect 2 is unaffected either way, and is what this issue now tracks.

Root cause, as measured

Scanning each node's nextRenderTime index from 0, limit: 200, ascending:

node	monotonic violations	head of walk	future rows in window	overdue in window
cd5	1	2026-08-20T11:24	2	198 / 200
yc0	1	2026-08-29T23:03	1	199 / 200
e9v	0	2026-08-19T16:01	158	42
v3t	0	2026-08-19T16:01	126	74

Two of four nodes affected, independently. The orphans, with the position bisected out of the index:

cd5  <product-A>|mobile   record 2026-08-20T11:24  indexed 2026-08-14T02:06  fromSitemap=true
cd5  <product-B>|desktop  record 2026-08-20T11:24  indexed 2026-08-14T02:06  fromSitemap=true
yc0  <product-C>|desktop  record 2026-08-29T23:03  indexed 2026-08-15T23:02  fromSitemap=false

(Those keys are pre-v0.66.0 shapes — RenderSchedule is keyed by URL now, one row per URL rather
than one per URL+device. The mechanism is unchanged: residency, the index, and the floor all work the
same way on the new key.)

Frequency was roughly 3 rows in ~1.3M over several days. The precise trigger was never established —
the affected rows have since been repaired and re-rendered, and dating the divergent writes from
record.nextRenderTime - interval is unreliable because the demand ladder rewrites intervals.

The aborted transactions were not the cause

Recorded so nobody re-walks it. Record and index writes share one transaction
(updateIndices(id, existingRecord, recordToStore, transaction && { transaction }), with that same
transaction passed into index.remove(...)/add), so an abort rolls back both and cannot half-apply.
And the aborts named only PrerenderedPage/ and VisitFilter/ while the orphans were in
render_schedule — different databases, independent commits.

Where defect 2 lives today

Still unguarded, and now in two places rather than one — v0.50.0's ready-set sweep took over
observing the index on nodes serving claims from the ready set, and it repeats the same shape:

• util/renderSchedule.js — the claim pass breaks on the first future row, then
  floorTo = Math.max(0, observed ?? nowMinute - guard).
• util/renderSchedule.js — the ready sweep breaks the same way, then
  advanceFloor(floorFrom, Math.max(0, firstDueMinute ?? nowMinute - guardMinutes())).

So the ready set did not remove this: a node whose index carries one orphan at the head strands its
backlog through whichever path is serving claims.

Why every recovery fails

• reset-claim-floor is futile. Confirmed live: reset, and the floor was back at now - guard
  immediately. From a zeroed floor the walk returns the same orphan first and the pass breaks at
  iteration 0 again.
• claimFloor.resetInterval is the same reset, so equally futile.
• unpinAfter is irrelevant — the pass never gets far enough to name a pinned row.
• claimFloor.enabled: false is worse than futile. Over 75s it recovered zero stranded rows and
  the overdue count accelerated from ~7/min drift to ~200/min, because maybeUnpinFloor early-returns
  when the floor is disabled (if (!config.queue.claimFloor.enabled) return null; — still there today)
  and runClaimPass then calls resetFloor() every pass, restarting from the absolute index minimum
  while still capped at ~100 rows.

Current-minute work still renders (a pass whose window happens to open on a due row reports
sawDue: true), which is why the node looks healthy and QueueStatus reads queued.

Ruled out

• Tie-pileup at one minute — the stranded set spans 2,101 distinct minutes at ~44 rows/minute.
• A single poison row wedging the head — the head of the index is ordinary product URLs.
• unpinAfter not firing — pin age lives in the SharedBuffer and resets on restart; node uptime was
  under the 1h default, so zero unpin warnings is expected.

Asks — restated against the tree as of 2026-09-16

1. Bound the forward advance. OPEN. A pass should only advance the floor as far as it actually
   scanned — to earliestNotYetDueMinute when it reached a not-yet-due row, and not at all when the
   scan was cap-truncated. observed ?? nowMinute - guard is the defect, and it is unchanged in both
   the claim pass and the ready sweep. Note the signal now exists where it did not before: the pass
   computes scanTruncated (rows.length >= scanLimit && earliestNotYetDueMinute === 0) and reports
   it. It just does not gate the advance on it.
2. advanceFloor's contract. LANDED. It is now a CAS against the value the pass started from,
   abandoning on conflict — "a conflict means a funnel write lowered the floor for a row this pass
   never saw". That is the assertion this ask wanted; it does not by itself stop the empty-scan
   ratchet, which is ask 1.
3. Document or repair the kill switch. OPEN. claimFloor.enabled: false still disables
   maybeUnpinFloor (renderSchedule.js, the second guard in that function). Either stop it doing
   that, or say plainly in the option text that it does — it is presented as a safe "changes nothing
   else" switch and is a downgrade for a node already stranded.
4. Make below_floor honest. OPEN. It is still counted inside a management.scanCap-bounded walk,
   so it saturates at the cap: a node with 92,873 stranded rows and a node with 1,998 publish the same
   number. backlogSnapshot already computes truncated for the walk — emit it alongside, or count
   belowFloor uncapped, so an alarm can tell "1,998" from "at least 2,000 and we stopped counting".
5. Per-node ConfigOverride scoping. OPEN. ConfigOverride.path is still the sole primary key and
   the table is replicated and not residency-pinned, so every lever for a node-local condition is
   cluster-wide. QueueControl already implements the pattern — scope as primary key, 'all' plus
   per-hostname rows, delete to inherit.

One unexplained detail still worth a look: oldestBelowFloorMs read 2026-08-14T02:07 while the table's
true minimum nextRenderTime was 2026-08-01T00:57. If the walk really is ascending from the absolute
minimum, those should agree.

Production evidence

Plugin 0.49.0, harper-pro 5.2.3, 4-node cluster, ~1.6M keys, all queue.* at defaults.

rows below floor        92,873      <- invisible to claim
rows floor..now              2      <- all claim can see
distinct due-minutes     2,101
oldest stranded row      2026-08-01T00:57Z

Other three nodes: 507 / 3 / 1 overdue. This bites whichever node's floor gets ahead, not the cluster
uniformly. Claim was granting nothing — granted 0 of 5 in 256 of 270 warned passes.

#111 is the companion: writeSchedule lowers only the writing node's floor while the row is
residency-routed to its owner, so ~75% of writes leave the owner's floor untouched — which is what
keeps depositing rows underneath a floor that has already moved past them.