perf(gfql): cut per-hop and rows-stage overhead for seeded polars chains

[lmeyerov]

Problem

LDBC SNB IC8 recent-replies — a 3-hop seeded chain (2.6k → 2.1k → 464 frontier) that projects six columns, orders and takes 20 — ran 29.3 ms on the measured Polars lane against Neo4j 5.4 and Memgraph 6.3. Profiling on merged master showed the cost is not the traversal: each hop runs about thirteen Polars operations over ~2k-row frames, and each one pays fixed plan and thread fan-out cost. The same query at 4 Polars threads ran 12.9 ms against 22.2 at 20 threads on identical data.

Changes

All four are result-preserving: same rows, same order, same dtypes, same index-trace values.

1. Array-side join estimate. estimate_inner_join_rows computes its exact sum-of-products over null-free integer keys with unique/searchsorted instead of a group-by-and-join plan. Nulls and non-integer keys keep the frame path, because engines disagree on null-key matching.
2. Predicate-first node gather. The endpoint filter runs on just the predicate columns plus a row-position column, and only the survivors are gathered at full width. An absent predicate column keeps the wide path so its three-valued verdict is unchanged.
3. Identity semi-join skip. When the endpoint filter dropped no ids, narrowing the edges to those ids is the identity, so the semi-join is skipped.
4. Array-side path expand join. For eager Polars over null-free integer keys and tiebreaks, the per-hop join-and-sort becomes searchsorted ranges over step rows pre-sorted by (key, tiebreaks) plus two row gathers.
5. Projection pushdown. rows() accepts attach_prop_columns, and a bare rows() immediately followed by select is rewritten to attach only the alias.column names the projection reads — the same rewrite in the pandas/cuDF and Polars twins, and in both bindings finishers. Any item that is not a literal, a bare alias id, an edge-alias column, or an existing node column keeps attach-all.
6. Small: the temporal-constructor text scan checks for a literal ( before the anchored regex, and resolve_engine recognizes Polars frames before probing for pyspark.

Measured

LDBC SNB SF0.1, the H684 index lane recipe (native Polars frames, gfql_index_all, auto node-property indexes, 8 warmups, 31 samples, full record materialization). Local box, 24 threads.

query	polars master	polars this PR	pandas master	pandas this PR
recent-replies	23.58	11.14	61.7	40.6
message-replies	8.96	5.25	25.3	22.4
seed-lookup	1.16	1.02	3.2	2.1

cuDF on a local RTX: recent-replies 372 → 174, message-replies 224 → 109. Point queries are unchanged: interleaved A/B of message-creator over 201 samples gives master 0.590 / 0.683 / 0.564 against 0.611 / 0.578 / 0.579 ms.

Thread sensitivity drops with the array-side work: recent-replies is 9.85 ms at 24 threads and 8.14 at 4, where master was 22.2 and 12.9.

DGX A/B on the benchmark recipe

Three runs per arm on the same H684 recipe against merged master 65c359b, canonical rows identical in every cell.

scale	engine	query	master	this PR	delta
SF0.1	polars	recent-replies	29.291	16.080	-45.1%
SF0.1	polars	message-replies	8.102	5.106	-37.0%
SF0.1	polars	seed-lookup	1.068	1.012	-5.2%
SF0.1	pandas	recent-replies	44.743	27.057	-39.5%
SF0.1	pandas	message-replies	11.325	11.745	+3.7%

Every other cell lands within about two percent, except two that deserve naming.

SF1 polars seed-lookup reads +13.5%, but the per-run values overlap completely (master 0.971 / 1.325 / 1.181 against 1.362 / 1.167 / 1.341), so that cell is noise at three runs.

SF1 pandas new-topics read +14.3%, and an interleaved re-run on an idle box reproduced +19.4%, so it was investigated rather than waved off. It is not a code effect. That query's GFQL surface is a one-hop undirected traversal with rows(source=...), which reaches none of the changed code: the projection rewrite requires source to be absent, and instrumenting the bindings module's own references shows zero calls to the three changed helpers. Bisecting to the first commit of this branch still showed +13.5% even though that commit provably makes no calls on the path. The measurement record explains it: the same cell on this harness reads 574.0 / 570.6 / 528.8 on master f7a7253 and 567.1 / 522.8 / 568.8 on master 92ad8c0, both of which predate this work entirely, and an A/A control settles it: running the same interleaved driver with BOTH arms at the current master, one from each of two byte-identical checkouts, gives 567.49 and 476.62 on one checkout against 472.81 and 517.53 on the other — four runs of identical code spanning 472.8 to 567.5, with the first interleaved pair 16.7% apart. The cell is bimodal across roughly 474 to 598 whatever the code is, and this branch drew from the high end.

The wider lesson is written up in the benchmark evidence: a three-run median of a bulk cell whose own A/A spread approaches the acceptance gate cannot support a product-vs-product claim, so such cells need an A/A control or many more repetitions before a delta is quoted.

Tests

• tests/compute/gfql/index/test_indexed_bindings_hop_overhead.py — the array estimate against the frame estimate across pandas, Polars and cuDF including null, float, nullable-integer and lazy deferrals; predicate-first gather against gather-then-filter over drops, unsorted positions, an absent column that must raise identically, and an empty gather; the covers-ids check; end-to-end indexed against canonical with and without endpoint drops, asserting the semi-join is skipped only when it is the identity; and a twelve-seed fuzz of the array expand join against the frame plan.
• tests/compute/gfql/test_rows_select_projection_pushdown.py — the pushdown plan itself, then every case run with and without the pushdown on pandas, Polars and cuDF including the decline cases, a spy asserting the narrowed request reaches the builder, and parameter serialization and validation.
• Broad local suite over tests/compute/gfql, chain specializations, chain and hops: 16067 passed, 1162 skipped, 102 xfailed.
• tck-gfql at main 84b098c against this branch: 4426 passed, 689 xfailed, 14 failed — the same 14 fail on merged master and are local cuDF-version errors, not regressions.
• Type-hygiene and comment-encoding guards pass with no growth; ruff and mypy clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_012Me1E7ZdDuGqJGu3mMEzhp