perf(gfql): execute a seeded fixed-hop bindings projection on index arrays

[lmeyerov]

Problem

After #2084 the seeded fixed-hop bindings path still builds a frame at every hop and a full bindings table before projecting. On LDBC SNB IC8 recent-replies that is about eight Polars operations per hop over ~2k-row frames, each paying fixed plan cost, plus an attach-and-sort over 2114 rows whose six projected columns are all the query ever reads. Two of the four aliases in that query need no properties at all, yet their full 32-column node rows are gathered on every hop.

A probe against the same indexes measured the ceiling: the identical result set, computed entirely in index arrays, takes 1.8 ms where the current route takes 9.9.

Change

A new specialization recognizes a seeded fixed-hop pattern whose rows() is immediately followed by a bounded select, and runs it without materializing anything but the projected columns.

• graphistry/compute/gfql/index/array_bindings.py carries the traversal. A path bag is one row position per alias per matched path, held in index arrays. Hops walk the same CSR adjacency indexes, orientation and path ordering are lexsort plus searchsorted ranges, and the endpoint join is the range expansion that perf(gfql): cut per-hop and rows-stage overhead for seeded polars chains #2084 already introduced for the frame path.
• chain_specializations/bindings_select.py gates and projects. It admits only Polars, directed fixed hops, an indexed seed, and a projection whose every item is a literal, a bare node alias, or alias.column over a column that exists. Everything else declines to the canonical route.

Three behaviors are preserved deliberately, because each is observable:

1. Filters still run through the unchanged filter_by_dict on a narrow gathered frame, so three-valued and dtype semantics stay the engine's own.
2. The seed still goes through _seed_rows_via_property_index as a module attribute, because the benchmark harness counts that seam by patching it.
3. The same served trace step is recorded, so index-engagement receipts are unchanged, and the result publishes the same emptied, alias-marked edge frame and clears the row execution context the way the canonical route does.

Measured

LDBC SNB SF0.1, the H684 index lane recipe, local box.

query (polars)	master	with #2084	this branch
recent-replies	23.58	9.85	6.54
message-replies	8.96	5.25	2.70

Point queries are unchanged: seed-lookup 1.18, message-content 0.42, message-creator 0.61.

DGX A/B on the benchmark recipe

Measured at 4c1912a. The head since then differs only by route registration, the polars
coverage floor, the CI matrix and test wiring; the sole product-code change is hoisting one
import to module scope so the route switch can patch what the chain calls, which does not
touch the served path.

Three runs per arm against merged master 65c359b, same H684 recipe, canonical rows identical in every cell.

scale	engine	query	master	this branch	delta
SF0.1	polars	recent-replies	29.291	10.231	-65.1%
SF0.1	polars	message-replies	8.102	2.991	-63.1%
SF0.1	pandas	recent-replies	44.743	27.166	-39.3%

The pandas gain comes from #2084, which this branch includes; the specialization itself is Polars-only.

Every other cell is flat or inside that cell's own noise. Three deserve naming rather than burying:

• SF0.1 polars seed-lookup reads +8.8%, but master's own three runs span 0.983 to 1.334 and this branch's span 0.939 to 1.285, so the ranges overlap completely.
• SF1 polars seed-lookup reads -19.6% for the same reason, in the other direction. Neither number is a claim.
• SF1 pandas new-topics reads +2.9% here. That is worth stating because the same cell read +19.4% on perf(gfql): cut per-hop and rows-stage overhead for seeded polars chains #2084's branch, which is the measurement-variance finding written up in that PR: an A/A control with both arms at the same commit spans 472.8 to 567.5 ms. This run is independent corroboration that the cell, not the code, was moving.

Against the competitor arms at SF0.1, message-replies now beats Kuzu's 23.712 by about eight times. recent-replies is still a loss against Neo4j's 5.425 and Memgraph's 6.333, but by roughly two times rather than the five it was. The remaining budget is measured and tracked: about 2.3 ms in filters that still route through polars, 0.6 in projection, and 2.1 in dispatch.

Tests

graphistry/tests/compute/gfql/index/test_array_bindings_select.py is a differential suite against the canonical route as oracle. Every case runs twice, once served and once with the specialization disabled, and the two frames must match in columns, dtypes, values and order. It covers seven pattern shapes crossed with four projection shapes, edge payload projection with ordering, all three index policies, declines for duplicate node ids, string ids, null endpoints and missing indexes, index-trace equality between the two routes, and a 24-seed fuzz over random graphs, hop counts, directions, filters and projections. A guard asserts the corpus actually serves, since a differential suite that silently stopped serving would pass while proving nothing.

That suite caught a real defect during development: literal projection items were built from a Python list, giving Int64 where the canonical route lowers through pl.lit and gives Int32.

Broad local suite over tests/compute/gfql, chain specializations, chain and hops: 16126 passed, 3 failed, matching the #2084 baseline exactly (two lane-registration tests for the new file, since fixed, and one cuDF zero-hop test that fails identically on master with the local cuDF version). Type-hygiene and comment-encoding guards pass with both new files at baseline zero; mypy and ruff clean.

Stacked on #2084; review that first. Base moves to master once #2084 lands.

🤖 Generated with Claude Code

https://claude.ai/code/session_012Me1E7ZdDuGqJGu3mMEzhp

[lmeyerov]

You were right to push on this, and it found real defects. I audited every admission/decline boundary in the diff against the tests, and the honest answer was that the suite looked broader than it was.

Two contract divergences, both fixed (01f113e76)

Reproduced with the served leg instrumented to prove it actually took the route, not assumed:

1. rows(attach_prop_columns=…) and rows(attach_prop_aliases=…) were silently ignored. The suffix check rejected source, alias_endpoints, alias_prefilters and a non-node table — but not those two. So this route answered a query the canonical polars route refuses outright with NotImplementedError. The answer looked right, which is what makes it worse: parity-or-error was broken quietly.

Fixed by inverting the check to an allow-list (set(params) - {"table", "binding_ops"} must be empty). Rejecting known-bad parameters fails open; admitting known-good fails safe, so a rows() parameter added later declines until someone teaches this route what it means.

2. A repeated select output name leaked polars.exceptions.DuplicateError instead of GFQLTypeError [invalid-node-reference]. The projection now declines on a repeated name and the canonical route raises its own error.

Why the existing suite could not see either

The bigger finding. point-rows admits several of the corpus shapes first, so in those cases neither leg ran the code under test — the differential compared point-rows against point-rows. Measured: test_matches_canonical_under_every_index_policy never engaged this route for any of its three policies, and only one of six middles in the corpus-engagement test actually served.

Fixed structurally rather than case by case:

• _both now turns point-rows off and asserts the array route served. That assertion immediately failed six existing cases — exactly the ones the audit flagged.
• The four test_declines_and_matches_* cases are now explicit negative tests (expect_served=False), so a decline is asserted rather than silently producing a canonical-vs-canonical comparison.
• The index-policy test now states which policy serves instead of assuming all three do. On this graph use declines on the cost gate; pinning that is what would catch the gate silently opening or closing.

Both new tests were verified by reinstating the bugs: with the old code they fail and the rest pass; with the fix, 61 pass.

Performance regressions: measured, and it is nothing

The decline tax is the thing to worry about, so I measured it directly rather than at query level where box noise dominates. Time spent inside try_bindings_select_polars on calls that return None, LDBC SNB SF0.1:

query	gate calls / query	decline ms / query	share of query
seed-lookup	0.0	0	0%
message-content	0.0	0	0%
message-creator	0.0	0	0%
new-topics	1.0	0.00513	0.008%

Three of the four non-qualifying queries never reach the gate at all; the fourth pays 5 microseconds.

A query-level route-on/route-off comparison over 6 alternating rounds reads seed-lookup +1.55%, message-content −1.42%, message-creator +7.88%, new-topics +2.36% against −63.9% and −68.3% for the two that serve. The +7.88% is noise, and the gate timing proves it rather than arguing it: that query calls the gate zero times, so this route cannot be responsible for any part of its number.

Local suite at this head: 13250 passed, 24 failed — the 24 are a polars-conformance set that reproduces identically on the untouched parent commit.

[lmeyerov]

Three more commits on this branch since the comment above, head is now d9aaf0ee0.

91aef602b — the id-dtype promotion boundary, both sides. numpy promotes (int64, uint64) to float64, which cannot tell 2^53+1 from 2^53+2, so an id→row probe that promoted would land on the wrong node. I checked whether that is reachable: uniform UInt64 ids above 2^53 serve and match the canonical route exactly, and mixed UInt64/Int64 ids decline on the schema-equality guard. So the seam is unreachable rather than merely unlikely — but nothing said so, and now two tests do.

d9aaf0ee0 — the property-index seed seam. _seed_rows has two branches, and every case in the file seeded on the graph's own id column, so the second was executed by no test. It is also the only branch that reports public_seed_scan in the trace. Verified correct against the canonical route on a multi-seed shape, with the expected rows asserted, plus a direct pin that the seam is the one that ran.

419df0390 — a CI lesson worth recording. Asserting "the route actually served" turns a test into an engagement pin, and the gfql-routes-off matrix switches each route off session-wide. That produced 52 divergences in the indexed-kernel lane, all the same assertion firing where the route is disabled on purpose. The repo already has the contract — @pytest.mark.route_engaged(...) — so it is applied at module level here. Verified across all ten modes: 63 pass in eight, skipped in indexed-kernel and polars-bindings-select, and it still runs in polars-point-rows, the lane where it gains cases rather than losing them.

Still unpinned and worth naming rather than hiding: undirected and variable-length hops all decline today, but nothing distinguishes why, so a regression that flipped one sub-condition would be caught by no test.