Add exact native mutation batch partitioning

[kylebernhardy]

Summary

Add a handle-bound logical mutation API for Harper's derived-index adapter. A caller can submit one logical batch, have the wrapper partition it into native-size frames, and publish only after every frame is accepted. Partial native application latches the handle until rollback, preventing a later publication from exposing half a logical batch.

The change also adds read-only option validation, index-scoped retired-tree reclamation, explicit active-versus-incomplete errors, and benchmark coverage for the logical and low-level mutation drivers. The package continues to expose only Tantivy's native storage.

This advances Implement mutation batching, commit publication, reader reload, and status.

Design

• ts/native.ts owns the logical-batch state machine. It snapshots caller-owned IDs before awaiting, partitions and applies frames sequentially, validates native mutation counts, batches replacement deletes, and distinguishes retryable E_BATCH_ACTIVE from rollback-required E_BATCH_INCOMPLETE. apply() rechecks the latch after typed-array coercion, while reload() participates in the same writer fence.
• ts/codec.ts implements bounded frame encoding, consumed-prefix accounting, duplicate-ID validation, and record-local rejection reporting. Oversized strings and IDs are rejected before allocation, and chunk coalescing bounds writer metadata for dense multi-value records.
• ts/native.ts retired-tree reclamation validates hints even when the retirement root is absent, refuses a symlinked root, confines deletion to the requested basename, and does not broaden scope through a symlinked live path.
• ts/native.ts open validation and close handling normalize malformed options, preserve closed-handle diagnostics after cleanup failures, and expose quiesced cleanup failure through { cleanupError }.
• ts/errors.ts and src/error.rs keep the new active/incomplete error codes in cross-language parity.
• src/lib.rs, ts/addon.d.ts, and ts/load-addon.ts bump native ABI compatibility to 5 and expose the commit-payload runtime limit with a capability gate.
• src/native.rs adds stateless native option validation and keeps retirement naming aligned with the caller's source basename. src/protocol.rs rejects frame limits that cannot hold one minimal mutation.

Developer surface

• README.md documents logical mutation batches, rejection policy, latch recovery, validation, retirement cleanup, and benchmark usage.
• docs/native-backend-implementation.md records the wrapper/native ownership boundary and mutation flow.
• docs/native-reset-quiescence.md defines reset results, same-path reclamation, collision-safe names, and cleanup ownership.
• benchmarks/native.mjs adds logical versus low-level mutation drivers and logical-batch latency percentiles. package.json runs both drivers in benchmark smoke CI.

Decisions for the human reviewer

• Record-local E_INVALID_ARGUMENT and E_BATCH_TOO_LARGE failures may become replacement deletes only when the caller opts into rejectedUpsert: "delete". Schema mismatch always fails the whole call to avoid mass deletion under configuration drift.
• A failure after native admission invalidates the handle's complete uncommitted window, not only the current logical batch. Recovery is rollback-close and replay from the durable checkpoint.
• assumeDistinctIds: true is a trusted hot-path escape hatch. Duplicate IDs under that option are outside the API contract.
• Retirement cleanup sweeps generated names for one exact path basename. The same path value must be used for reset and reclaim.
• Review depth checkpoint: five full outside-review passes plus targeted delta passes were run. The third full pass accepted five findings and led to the latch, snapshot, allocation, batching, and cleanup-scope hardening above. Further whole-branch passes were stopped after the fifth pass; the final adjudicated findings were implemented and verified locally.
• Where to look hardest: the admission boundary and rollback latch in ts/native.ts, and recursive deletion scope in reclaimRetiredNativeFullTextIndexes().

Verification

• npm run lint — TypeScript and Clippy passed.
• npm test — 31 Rust tests, 99 primary Node tests, 3 worker tests, and 1 packed-package test passed.
• npm run benchmark:smoke — both logical and low-level mutation drivers completed. On this local smoke workload they produced about 4.4k and 4.5k durable documents/second respectively; these are sanity results, not release benchmarks.
• Prettier and cargo fmt --check passed.
• test/native-index.test.mjs covers multi-frame application, visibility and reopen, reentrancy, overload, record snapshots, replacement-delete batching, size bounds, and reclamation scope against the real addon.
• test/native.test.mjs covers runtime capabilities and closed-handle behavior after injected cleanup failure.
• test/error-codes.test.mjs pins Rust/TypeScript limits and error-code parity.
• test/native-admission.test.mjs and test/native-publication.test.mjs consume the runtime-provided payload limit rather than duplicating it.
• Final read-only review coverage was Claude plus Harper-domain adjudication. Gemini was authorized but the installed review harness could not authenticate it, so Gemini is not represented as having reviewed the final artifact.

Complexity: high — public mutation semantics, durable-state recovery, native ABI capability checks, and recursive cleanup boundaries.

Comment generated by kAIle (GPT-5)

_{Review-Coverage: authored=codex; ran=cursor-composer,claude,cursor-grok; adjudicated=domain; blocked=gemini(auth); rounds=18; full=5 @ 522d38b}

_{Human-Review-Need: 4 @ 522d38b}