fix: permit clear() when create_index=False, and drop its FT.INFO dependency

[vishal-bala]

Motivation

An application whose Redis ACL is assembled from +@read +@write is denied FT.INFO and FT.CREATE together, because neither command belongs to either category. Such an application cannot create an index and cannot ask whether one exists, so the index is provisioned out of band by whoever holds a privileged credential. create_index=False exists for that deployment: it skips the existence check, the schema comparison and creation, and issues no index command at all.

The change that added the flag also made delete() and clear() refuse to run under it, documented as protecting an externally managed index from being destroyed through an attach-only instance. That reasoning holds for delete(), which calls _index.delete(drop=True). It does not hold for clear(), which removes entries and leaves the index in place. Grouping the two conflated "delete every entry" with "delete the index", and left a caller attached to a platform-provisioned index with no public way to invalidate a stale cache, which is the one runtime operation such a credential still needs.

The guard was not an enforcement boundary in any case. drop(keys=...) is public and unguarded under the same flag, so the entire cache was always reachable through public API by enumerating the keys first. That is what downstream callers did, taking RedisVL's key-layout knowledge with them and reimplementing the keyspace walk against private attributes.

Changes

Entry removal is permitted; dropping the index is not

clear() and aclear() no longer raise under create_index=False on SemanticCache, MessageHistory, SemanticMessageHistory or SemanticRouter. delete() and adelete() still do.

EXTERNAL_INDEX_LIFECYCLE_CONFLICT is renamed to EXTERNAL_INDEX_DROP_CONFLICT, since it now guards exactly one operation, and its message points the caller at clear() rather than at the provisioning path alone. No alias is kept for the old name: the message text changed as well, so anyone matching on it is affected either way.

SearchIndex.clear() no longer reads FT.INFO

Three of the four extension clear() methods delegate to SearchIndex.clear(), which read FT.INFO's num_docs to size a runaway backstop for its delete loop. FT.INFO carries only the @search category, so that single call, made for a loop bound, denied the whole method to the credential this flag exists to serve. The commands the sweep is actually made of are both granted: FT.SEARCH is @read @search, and DEL is @keyspace @write @slow. Measured on Redis 8.4.5 against a user created as on ~* &* +@read +@write -@dangerous, FT.INFO raises and FT.SEARCH does not. Without this half, a uniform-looking fix would have meant one extension working and three failing on a permission error.

The backstop now comes from a CountQuery, which drop_by_filter has used for the same purpose all along and which is also FT.SEARCH. Under that restricted credential, clear() removed 1500 of 1500 documents and the index survived.

A page whose keys cannot all be deleted now advances the paging offset rather than re-reading the same head, so a blockage no longer hides the documents behind it. Measured with 600 of 2000 keys made undeletable, the sweep removes exactly the other 1400 and returns. Termination was also checked against a writer inserting 1.04 million keys mid-sweep, against every delete failing, and against a second clear() running concurrently on the same index.

The failure modes are documented rather than removed

Neither shape of clear() verifies its key selection against the live index, because verifying it needs the FT.INFO this flag exists to avoid. They fail silently in opposite directions, and both are now stated on the methods and in the ACL guide.

Prefix-based clearing, which is SemanticCache, deletes every key under {name}:. That reaches another writer's entries and any unrelated application data sharing the namespace root. It also misses: if the live index covers a different prefix, or is an alias onto one, clear() deletes only what this instance itself wrote, returns successfully, and leaves the cache serving the stale hits it was called to invalidate.

Index-based clearing, which is the other three, deletes whatever the live index covers. Against a differently-prefixed, multi-PREFIX or aliased index that means another application's documents, while this instance's own unindexed entries stay behind.

Separately, and unchanged by this PR, clear() races an in-progress background index scan, since it can only delete what the index currently returns. It drains the documents indexed so far, sees an empty page while unindexed keys remain, and stops, reporting what it deleted as if it were done. Measured on Redis 8.4.6 over 20,000 hashes indexed immediately after loading, two runs cleared 125 and 57; against a fully indexed copy of the same data both cleared all 20,000. The num_docs bound this replaces cleared 57 on the same setup, so the early stop is long-standing rather than a consequence of the new bound. It is now stated on the method.

Smaller changes

• remove_route() gains the stored-config caveat add_route() already carried, because clear()'s new docstring points readers at it.
• Two stubs in tests/unit/test_error_handling.py patched SearchIndex.info for a call clear() no longer makes, so they passed while asserting nothing. They now raise, which makes re-introducing FT.INFO into either twin fail loudly.
• The new backstop test drives the cluster branch of _delete_batch, the only path that reaches the backstop, and carries its own query cap: pytest-timeout is not installed, so a regression would otherwise hang the suite rather than fail it.
• A MessageHistory integration test proves the FT.INFO removal end to end against a real restricted credential in about two seconds and with no vectorizer. The pre-existing ACL coverage exercises SemanticCache, whose prefix-based clear() never needed this change.
• Assertions compare against the constant rather than a regex on its message.

Notes

This is a behaviour change, not only a permissions fix. A call that raised ValueError now performs a destructive operation, and for the SearchIndex.clear() half the loop bound changes on the default create_index=True path too. Callers who wrote except ValueError around clear() as a "clearing is unavailable here" idiom will now see real deletions. The exposure is small, since the flag itself is recent, but the change wants a minor release rather than a patch, and a release note that says so plainly.

SearchIndex.clear() is on every clearing path in the library, so this touches more than the four extensions. The cluster branch of _delete_batch is exercised only by the new hermetic test, because requires_cluster tests are skipped in every environment including CI. That test pins control flow given _delete_batch's documented return contract; it says nothing about cross-slot behaviour, node targeting, or whether FT.SEARCH enumerates every shard.

This overlaps the branch that reworks BaseCache.clear() for cluster cursor handling, which rewrites the same SCAN loop to use scan_iter. The two do not conflict as written, and the assertions here deliberately avoid pinning that loop's call shape, but landing the cluster work first keeps the sequencing clean.

Four follow-ups are left out deliberately. drop_by_filter keeps the same hang-forever hole when every delete in a batch fails, since its backstop only advances on successful deletes. SearchIndex.delete(drop=True) on cluster calls clear() and then issues FT.DROPINDEX without DD, so an incomplete sweep orphans keys with no index left to enumerate them. The prefix scans in BaseCache interpolate an unescaped name straight into SCAN MATCH, so a name containing a glob metacharacter silently matches the wrong set: a cache named tenant[a] clears nothing. And clear() returning a bare count means 0 cannot distinguish an already-empty index from a sweep that deleted nothing, which BulkResult already solves for drop_by_filter.

Next Steps

1. Decide the release type. A minor version is the recommendation, for the reason in Notes.
2. File the four deferred items above as issues; each has a reproduction.
3. Confirm the sequencing against the cluster cursor branch before merging either.

Release Notes

SemanticCache, MessageHistory, SemanticMessageHistory and SemanticRouter constructed with create_index=False can now call clear() to remove their entries. Previously both clear() and delete() raised ValueError under that flag; delete() still does, because it drops the index. This is a behavioural change: code that caught ValueError from clear() and treated it as "unavailable" will now delete data.

SearchIndex.clear() and AsyncSearchIndex.clear() no longer issue FT.INFO. They size their internal delete loop with a CountQuery instead, so clearing an index is available to a credential built from +@read +@write, which is denied FT.INFO. Clearing also no longer stops early on a page whose keys could not be deleted.

EXTERNAL_INDEX_LIFECYCLE_CONFLICT in redisvl.extensions.constants is renamed to EXTERNAL_INDEX_DROP_CONFLICT, and its message has changed. Code importing the old name, or matching on the old text, needs updating.