[core][python] Reuse PK sorted indexes with retired source files

[wangyong9999]

Purpose

Close #9458.

A source-backed primary-key sorted-index payload keeps one immutable, ordered source-file list as its row-ordinal namespace. After a later snapshot retires only part of that list, the current Java and Python readers compare the payload with the complete active file set at that level. The mismatch rejects the still-valid payload, so surviving indexed files fall back to raw scans. Query results remain correct; this change only restores index reuse.

Key code and data-flow changes

• Keep the complete payload source list for group-ordinal localization, while deriving snapshot coverage from its active intersection at the metadata-declared data level.
• Preserve the existing one-group-per-level invariant; ambiguous payload candidates at one level still fall back instead of introducing a new lifecycle.
• Bind scan-side source lookup to both the group data level and source identity so a file that moves to another level cannot inherit a stale group.
• Leave newly active or otherwise uncovered files on the existing raw-scan path.
• Apply the same lifecycle and scan-planning semantics to Java core and Python.
• Route Python sorted-state, sorted-scan, vector-scan, and full-text-scan source eligibility through one PrimaryKeyIndexSourcePolicy counterpart, matching Java and preventing the four readers from drifting.

The equivalent C++ reader correction is tracked by apache/paimon-cpp#253.

Error and compatibility boundaries

• Reject malformed or misordered source metadata, active row-count mismatches, zero-active intersections, and multiple valid candidates for one level.
• Preserve the payload's retired prefix or middle sources so surviving files retain their original group ordinal offsets.
• No public API, storage format, index protocol, writer lifecycle, or concurrency behavior changes.

Tests

• Java: mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=PkSortedBucketIndexStateTest,PrimaryKeySortedIndexScanTest test — 20 tests passed.
• Python: targeted sorted-index/source-policy tests — 23 tests passed, including five invalid-payload subcases and source-policy parity checks.
• Java Spotless, Python project-config flake8 and py_compile, and git diff --check passed.
• Added coverage for retired-source ordinal offsets, newly active raw fallback, row-count mismatch, misordered metadata, zero-active payloads, ambiguous candidates, cross-level source movement, and shared source eligibility.

[wangyong9999]

CI update for the latest commit:

• 20 of 21 checks pass, including all Java, Flink, Spark, and the other Python matrix jobs.
• The only failure is lint-python (3.12). It stopped during Maven dependency resolution because Maven Central returned HTTP 502 for kerby-provider, joda-time, and crt-core; source lint and tests did not start.
• The previous Python 3.13 setup failure and JDK 11 ThreadPoolUtilsTest failure both pass unchanged on this run.

Could a maintainer rerun only lint-python (3.12)?