fix: keep paginate going past a page whose matches were all dropped

[vishal-bala]

Summary

Two independent bugs in redisvl/index/index.py, both of which silently return fewer documents than the server matched.

1. paginate stopped early on an unmaterializable page. SearchIndex.paginate / AsyncSearchIndex.paginate terminated on if not results: break, treating an empty page as an exhausted result set. But process_results deliberately drops matched documents whose field payload came back missing — a key that expires or is updated mid-query is returned by the server as a matched id with a nil field array, and is still counted in the server's total. So a page can be empty while matches remain, and the loop would exit and discard every remaining page.

2. NOCONTENT queries returned nothing at all. This one is live on main today and needs no race. Under NOCONTENT the server returns ids and no field data for every healthy match, which is byte-for-byte the shape the drop heuristic uses to detect a race victim — so it dropped every document. redis-py's Query.no_content() sets only _no_content and leaves _return_fields untouched, so neither detection predicate noticed.

Measured against a real Redis 8.4, four documents indexed, main vs this branch:

query	main	this branch
VectorQuery(...).no_content()	0 of 4	4 of 4
VectorQuery(..., normalize_vector_distance=True).no_content()	0 of 4	4 of 4
FilterQuery(...).no_content() (hash)	4 of 4	4 of 4

Hash FilterQuery escaped because it trips neither predicate. Vector and range queries (which always project vector_distance when return_score=True, the default) and JSON full-object unpack both returned an empty list for every NOCONTENT call.

What changed

Termination now asks whether the server reported any match, via SearchResults.dropped_count, rather than whether the page yielded documents. The offset advances unconditionally, so a page that cannot be materialized can never wedge the loop.

• _page_had_matches(results) — a page had matches when it either yielded documents or dropped some. Uses truthiness rather than len(), because process_results returns a bare int for a CountQuery.
• _fold_carried_drops(results, carried) — paginate never yields an empty batch, so a fully-dropped page's dropped_count would otherwise vanish and leave later batches reporting complete is True. Its count is folded into the next yielded batch instead.
• An explicit TypeError for CountQuery in both paginate methods. It returns a match count, not documents; previously it yielded that integer forever.
• _has_missing_field_payload short-circuits on _no_content, and the vector-normalize branch is guarded on the distance actually being present (otherwise VectorQuery(normalize_vector_distance=True).no_content() raised KeyError).
• SearchResults is now in the API reference, and docs/concepts/queries.md documents the pagination behaviour.

SearchResults remains a drop-in list, so no caller needs to change.

Known limitations

These are real and deliberate — this PR does not claim to make offset pagination correct in general, only to stop it truncating on a droppable page.

Trailing dropped pages are only logged. A result set whose final pages are entirely dropped has no subsequent batch to fold those drops into, so they are reported only by the process_results warning and never reach SearchResults.dropped_count.

paginate is offset-based, and that is inherently unstable. Redis documents LIMIT without sorting as non-deterministic — subsequent queries may return duplicated or missing values. Stable paging requires SORTBY on a unique field; sorting on a non-unique field (age, say) is not sufficient. This is independent of the expiry race and is not fixed here. The durable fix is a cursor-backed paginate (FT.AGGREGATE ... WITHCURSOR, which _iter_keys_by_filter in this same file already uses); it is explicitly not attempted in this PR.

Deep pagination is still bounded server-side. search-max-search-results caps the reachable offset — 1,000,000 by default (verified on Redis 8.4), but 10,000 on Redis Cloud Free & Fixed tiers. Past that the search errors rather than ending cleanly.

Deliberately deferred. Short-page termination (stopping when len(batch) + dropped_count < page_size, which would save one round trip per iteration) and a public SearchResults.total. Both are additive and neither is needed for correctness.

Verification

Run against Redis 8.4 via testcontainers, on Python 3.14, with uv sync --all-extras.

• Full suite (make test): 1900 passed, 257 skipped, 2 xfailed, 0 failed.
• make check-types (mypy over ./redisvl): clean, 119 source files.
• make docs-build: succeeds; 11 warnings, all pre-existing and none from files touched here.
• pre-commit run --all-files: passes.
• make test-notebooks on 01_getting_started.ipynb (which demonstrates paginate()): 24 passed, 1 skipped, no external API calls.

The regression tests were confirmed to actually fail against the pre-fix code, not merely to pass:

• Reverting _page_had_matches to return bool(results) fails 2 of the 5 paginate tests.
• Deleting the carried_drops = 0 reset fails the same 2, with dropped_count going [2, 5] instead of [2, 3].
• Removing the _no_content short-circuit fails both NOCONTENT unit tests and the new integration test.

tests/unit/test_paginate_dropped_page.py serves canned FT.SEARCH replies keyed by paging offset through the real _query → process_results → SearchResults chain, so the dropped_count the fix depends on is produced by production code. Every fake bounds its request count, so a termination regression fails loudly instead of hanging the suite. An audit cut this file from 27 tests to 5 that kill strictly more mutants.

A note on the "Redis 8.8+" attribution

Seven docstrings in redisvl/index/index.py (inherited from merged commit 0cfabe2, not written on this branch) attribute the race to Redis 8.8+ changing the RediSearch worker pool default to a nonzero background executor. A reviewer disputed this, citing a search-workers default of 0. Checked empirically with CONFIG GET search-workers:

image	redis_version	search-workers
redis:8.4	8.4.6	0
redis:8.8.0	8.8.0	12
redis:latest	8.10.0	12

The docstrings are correct and the reviewer's number came from the Redis Software REST API object reference, which does not govern Redis Open Source. The --search-workers 0 pin and comment in tests/docker-compose.yml are also accurate. No change needed.

---

Note

Medium Risk
Changes search result processing and pagination termination, which can alter which documents callers receive and how completeness is reported. Behavior is covered by unit and integration tests, but it sits on a core query path.

Overview
Stops paginate from treating an empty page as end-of-results when matches were dropped (expiry/update race), and stops NOCONTENT queries from being dropped as if they were race victims.

Sync and async paginate now advance while the server still reported matches (dropped_count or documents), skip empty batches, and fold skipped-page drops into the next yielded SearchResults. CountQuery is rejected instead of looping forever.

_has_missing_field_payload short-circuits on _no_content, and distance normalization is skipped when no score is present, so id-only vector/JSON queries return ids instead of an empty list or KeyError. Docs note the remaining caveats: trailing fully-dropped pages only log, and offset paging still needs a unique sort_by.

^{Reviewed by Cursor Bugbot for commit 564f96b. Bugbot is set up for automated code reviews on this repo. Configure here.}