Reproducible 202609 chain, step 7: make the search-index builder deterministic

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: keep going on Part B under the standing rule. What I did: found that tools/build_search_index.py produced different indexes from identical inputs, traced it to four order-dependent choices in the hot-token path, fixed them, and proved two builds byte-identical. No published file changes; the live 202608 index is untouched.

Finding

Two builds of the same 202609 inputs: 915 vs 901 shard files, 138 vs 137 hot tokens, 217 shared files with different bytes — only one base shard differed; everything else was under hot/. So the hot set itself was unstable.

Causes → fixes

cause	fix
est (avg posting width) sampled with LIMIT 500000 and no ORDER BY → different rows each run → the hot threshold moved	average over all rows
hot set query had no ORDER BY → hot/ collision-suffix keys assigned in scan order	ORDER BY token
per-shard promotion ORDER BY n DESC LIMIT 1 → ties arbitrary	ORDER BY n DESC, token ASC
hot_topk per-pid sum(c) — parallel float summation order	sum(c ORDER BY field) (field now carried through contrib; ≤ 1 row per field per (token, pid), so the order is total)
(Codex) stale files from a previous build could sit beside a new one	refuse a non-empty index dir unless --force, which removes it; --tag must be one filename component and the target must resolve under --outdir before any rmtree
(Codex nit) top_df_tokens tie	ORDER BY df DESC, token ASC

Evidence

• After the fix: all 1,029 files byte-identical across two builds (build_stats.json excluded — it carries timestamps); 1,025 shards, max 4.93 MiB under the 5 MiB cap; ~15 min per build. A third build with the final code is compared against run 1 below.
• Reader contract unchanged (FNV token routing, hash(pid) % M sub-sharding, hot_tokens.json as locator). Layout changes because the estimate is now over the full table: vs the live 202608 index, 0 hot tokens disappeared, 0 fetchable flips among tokens present in both, 20 tokens newly hot (all fetchable). (202609 has more OC records than 202608, so the sets aren't strictly comparable; the no-flip check is the point.)

Review

Codex ×3: R1 changes-required (P2 stale output dir; nit df tie) → R2 changes-required (P2 --force could delete outside --outdir via a traversal tag) → R3 LGTM. Verdict posted below.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LtTxB4jfTZgaTR7CK4zKqy

[rdhyee]

🤖 rbotyee — Codex round 3 (final) verdict, verbatim:

No P1/P2 findings remain in the scope-locked changes.

The tag validation, canonical parent check, and symlink rejection all occur before the sole rmtree call in build_search_index.py. Direct checks confirmed /tmp/victim, ../victim, a/b, ., and .. exit 2. Compilation and git diff --check pass.

The full pytest suite could not start because the read-only sandbox provides no writable temporary directory.

Verdict: LGTM.

[rdhyee]

🤖 rbotyee — confirmation build with the final code: CONFIRMED: post-patch build byte-identical to run 1 (1029 files)

[rdhyee]

🤖 rbotyee — staging check before merge: fork Pages run 33211721352 built green; explorer-smoke 4 passed (6.1s) against https://rdhyee.github.io/isamplesorg.github.io. Merged under RY's standing rule.