vector-search: support scalar pre-filter before Top-K

[shyjsarah]

Summary

Support scalar pre-filtering before Top-K for data-evolution/global-index vector search, including DataFusion literal and lateral vector_search queries.

The scalar predicate is evaluated against the same pinned snapshot as vector search, producing global row IDs that are localized per vector-index shard and passed to the vector backend as an allow-list. A normal Paimon read is used so scalar global indexes such as BTree can prune the filter read while residual evaluation, partial index coverage, and deletion visibility remain exact.

Changes

• Enable VectorSearchBuilder::with_filter and BatchVectorSearchBuilder::with_filter on the data-evolution path.
• Pin scalar filtering, deletion-vector handling, raw fallback, refine, and materialization to one resolved snapshot.
• Apply the same global row-ID allow-list to indexed search and unindexed raw fallback.
• Localize a shared bitmap once per vector-index shard, skip empty shards, and share bitmaps across batch queries without deep copies.
• Add a reusable prepared-filter API so lateral execution evaluates a target-side scalar predicate once per physical plan.
• Push fully translated DataFusion predicates into literal vector_search while retaining an Inexact residual correctness check.
• Push fully translated target-only conjuncts into lateral vector search while preserving left-side, cross-side, unsupported, and inexact residual predicates.
• Reject NaN predicates from vector pre-filter pushdown because Paimon and Arrow comparison semantics differ.
• Show pushed scalar filters in logical and physical EXPLAIN output.
• Document scalar pre-filter semantics and lateral-query behavior.

Testing

• cargo fmt --all --check
• git diff --check
• cargo test -p paimon --lib table::vector_search_builder::tests
• cargo test -p paimon-datafusion --lib filter_pushdown::tests::test_vector_prefilter_rejects_nan_literals
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vindex_build_then_vector_search_query -- --nocapture
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_lateral_join_uses_query_vectors -- --nocapture
• cargo clippy -p paimon --lib --tests -- -D warnings
• cargo clippy -p paimon-datafusion --lib --tests -- -D warnings

Covered cases include filter-before-Top-K ordering, empty matches, batch reuse, non-zero shard offsets, raw fallback for unindexed rows, time-travel snapshot pinning, literal SQL, lateral SQL, mixed/cross-side conjuncts, and NaN pushdown safety.

Notes

• No on-disk format change.
• No SQL function signature change.
• Existing VectorSearch::include_row_ids public API remains unchanged; this PR adds PreparedVectorSearchFilter and shared-filter builder support.
• This is scalar pre-filtering for vector search, not the existing vector/full-text route-fusion hybrid_search feature.

[jerry-024]

[MAJOR] The execution-scoped prefilter cache only retains a weak reference to the prepared filter, so one logical execution can resolve different target snapshots across partitions. Once an earlier partition stream is exhausted and drops the last strong reference, a later partition using the same live TaskContext creates a new OnceCell and resolves the table snapshot again.

Keep the prepared filter strongly owned for the complete execution across all partitions, releasing it only when that execution finishes. Please add a sequential-partition test that updates the target between partitions.

[jerry-024]

[MAJOR] Filtered batch searches discard Lumina batch execution and rebuild the same shared allow-list once per query. Each fallback call materializes the shared RoaringTreemap into a new Vec<u64>, producing O(Q * B) conversion/allocation work plus Q sequential searches per shard.

When every query shares the same bitmap, materialize it once and call search_with_filter with the full query batch. Keep the per-query fallback only for differing filters.