perf(index): reuse primary-key index groups with retired source files

[wangyong9999]

Purpose

Linked issue: close #252

A source-backed primary-key index payload uses one immutable ordered source list as its ordinal namespace. After an update retires only part of that list, the current reader requires the payload to equal the complete active source set of its data level. It rejects the still-valid payload and degrades every surviving indexed file to an ordinary scan. Query results remain correct; this PR only avoids that performance fallback.

The same optimization is tracked for Java and Python in apache/paimon#9458, with the aligned change in apache/paimon#9460.

Key code and data-flow changes

• Keep the payload's complete source list for group-ordinal localization, but compute coverage from its intersection with eligible active files at source_meta.DataLevel().
• Preserve the existing one-group-per-level invariant. Multiple valid payload candidates for one level are rejected and fall back to an ordinary scan.
• Map an accepted group only to files allowed by PrimaryKeyIndexSourcePolicy::ShouldRead, using data level + file name + row count as the scan-side source identity.
• Preserve retired prefix or middle source offsets and deletion files when converting surviving files to IndexedSplit.
• Leave uncovered, newly written, or cross-level files on the existing ordinary-scan path.

This restores index reuse for surviving compact files without introducing a new writer lifecycle or allowing an uncovered file to depend on stale index metadata.

Error and compatibility boundaries

• Source names remain canonically ordered; an active source must match the payload's recorded row count.
• A payload with no active source at its metadata-declared level is rejected. A file with the same name and row count at another level cannot inherit the group.
• Invalid field/type metadata, row range, source metadata, or multiple candidates at one level fail closed to the existing ordinary scan.
• No public include API, storage format, index protocol, write path, or concurrency behavior changes.

Tests

• Current x86_64 GCC 8 internal branch: focused PkSortedBucketIndexStateTest.*:PrimaryKeySortedIndexScanTest.* — 37/37 passed.
• Current x86_64 GCC 8 internal branch: full core_test — 1,747 passed, 100 skipped, 0 failed (1,847 total).
• Follow-up DataLevel binding regression set — 3/3 passed: cross-level source fallback, retired-source ordinal preservation, and wrong-level payload rejection.
• Added/retained coverage for partial active intersection, one group per level, metadata level mismatch, retired prefix/middle ordinal offsets, row-count mismatch, a mixed update with deletion-vector preservation, a non-eligible APPEND source, and cross-level source movement.
• clang-format dry-run and git diff --check pass.

API and Format

No public include API, storage format, or index protocol changes. The change only broadens safe read-side reuse of one existing source-backed payload per level.

Documentation

Updated the primary-key global-index user guide to describe immutable source groups, partial active coverage, the one-group-per-level boundary, declared-level validation, and fallback behavior.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

[lxy-9602]

The current changes seem reasonable to me. That said, to avoid PkSortedBucketIndexState in C++ gradually diverging too much from the Java implementation and becoming harder to align later — for example, if additional features or refactors are introduced on the Java side in the future — I’d suggest merging this logic into paimon-java first, and then bringing it into C++.

[wangyong9999]

Thanks. I narrowed the revision to preserve the existing one-group-per-level lifecycle and added strict source_meta.DataLevel() validation; it does not add a writer/protocol state or a C++-only multi-group model. The remaining change fixes C++ read-side validation of an already-written immutable source group after partial retirement. Because no Java behavior or persisted format changes, could this narrow reader correction proceed independently while keeping the Java/C++ lifecycle aligned?

[wangyong9999]

I checked the current Paimon Java and Python readers and confirmed the same exact-active-set rejection there. I filed apache/paimon#9458 and opened the aligned Java/Python fix in apache/paimon#9460. This C++ branch now also binds scan-side source matching to the group's data level, with a regression test for a source moving to another level.

[lxy-9602]

@wangyong9999 I’m not sure this is best described as a bug, since it doesn’t affect correctness and seems more like a performance fallback when the index is not applied. With that in mind, the current title may be slightly misleading.

Also, I think it may be better to wait for the Java side to land first before moving this forward, since the Java implementation covers more complex scenarios and the C++ path is currently a subset of that functionality. From what I can tell, the optimization on the C++ side looks reasonable, but I’d feel more comfortable waiting for full validation on the Java side before treating the overall behavior as fully confirmed. Thanks again for the patience here.

[wangyong9999]

Agreed. This is a performance fallback, not a correctness issue. I changed the PR title to perf, changed both tracking issues to Improvement, and clarified that query results remain correct. The aligned Java/Python change is apache/paimon#9460; I will leave this PR pending until that change completes validation and lands, then update here. No further C++ behavior change is planned.