Skip to content

fix: escape glob metacharacters in SCAN match patterns - #702

Open
vishal-bala wants to merge 2 commits into
mainfrom
fix/escape-glob-in-scan-patterns
Open

fix: escape glob metacharacters in SCAN match patterns#702
vishal-bala wants to merge 2 commits into
mainfrom
fix/escape-glob-in-scan-patterns

Conversation

@vishal-bala

@vishal-bala vishal-bala commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

What's wrong

BaseCache.clear() built its SCAN MATCH pattern by interpolating the cache name directly. Per the SCAN docs that pattern is glob-style, and the name is caller-supplied and unvalidated — so a name containing *, ?, [ or \ produced a pattern matching keys the cache does not own, and clear() then deleted them.

Reproduced on Redis 8.4.5:

redis-cli -n 9 MSET 'cache[ab]:x' 1 'cachea:y' 2 'cacheb:z' 3
redis-cli -n 9 --scan --pattern 'cache[ab]:*'     # -> cachea:y, cacheb:z
redis-cli -n 9 --scan --pattern 'cache\[ab]:*'    # -> cache[ab]:x

So SemanticCache(name="cache[ab]").clear() wipes two unrelated caches and leaves its own entries intact. It is silent — no error, and the caller believes their cache was cleared.

This is data loss on a destructive path, not a matching quirk: low likelihood (unusual names) but high consequence, with a blast radius covering other keys in the same keyspace. In a multi-tenant naming scheme such as EmbeddingsCache(name=f"cache:{tenant_id}") a tenant registering x[ab] destroys other tenants' entries.

Grepping match= across the repo found the same shape in three more places, two of them also destructive:

Site Consequence when unescaped
BaseCache.clear / aclear deletes other caches' keys
SemanticRouter._route_pattern deletes other routes' keys via delete_route_references
build_scan_match_patterns rewrites other indices' keys on the executor's RENAME / DUMP-RESTORE-DEL path
MigrationPlanner._sample_keys and the async twin wrong key sample, so a wrong migration plan

The fix

A match_pattern(*segments) builder in redisvl/utils/utils.py escapes each literal segment and appends the trailing glob. All four sites now build patterns through it; build_scan_match_patterns is itself a chokepoint, so fixing it covers executor, async_executor, validation and async_validation at once.

Exposing a builder rather than a bare escape() is deliberate. Escaping at the call site is a step the next contributor can forget, and no lint rule can express the invariant here: ruff and flake8 are unconfigured, mypy is not strict, and patterns are consumed several call sites away from where they are built. A builder makes the omission unrepresentable — there is no other way to make a pattern.

The escape set is \, *, ?, [. ], ^ and - are deliberately absent: they are only meaningful inside a [...] class, which can never open once [ is escaped. Confirmed empirically — cache\[ab]:* and cache\[ab\]:* return identical results.

Escaping rather than rejecting such names. Rejecting at construction would be a breaking change that orphans keys: a deployment whose cache is already named cache[ab] could no longer construct the object to clean up after itself. If maintainers prefer validation, it wants a deprecation cycle rather than a straight raise.

Testing

Two questions, separated by what can actually answer each:

  • Does RedisVL emit the right pattern? An exact-string assertion, no matcher needed — tests/unit/test_scan_pattern_escaping.py covers all four sites hermetically, including aclear and both planners.
  • Does Redis then interpret it as intended? Only a real server can say. tests/integration/test_embedcache.py adds the cache regression (three caches colliding under an unescaped glob; clear one, assert the others survive), and tests/integration/test_migration_v1.py adds the migration and router prefix case.

No glob matcher is reimplemented anywhere, so nothing can drift from Redis. Every new test was checked to fail without the fix.

Verified: mypy clean (119 files), 1421 unit tests pass, integration green across embedcache and migration.

Deliberately out of scope

Four pre-existing issues surfaced while working in this code. None is caused or worsened by this change, and each wants its own PR:

  • BaseCache.clear() has a containment bug escaping cannot fix. _get_prefix() is f"{name}:", so a cache named foo clearing foo:* deletes the entries of a cache named foo:bar — ordinary names, no metacharacters, same blast radius. SearchIndex.clear() already does this correctly by deleting the doc ids FT.SEARCH returns; SemanticCache should route through it, and EmbeddingsCache (which has no index) should verify each scanned key's prefix segment before deleting.
  • clear() / aclear() never advance a Mapping cursor on Redis Cluster, so they re-scan page 1 forever. Note cursor = cursor_int is not the fix — a dict cannot be passed back as a SCAN cursor; per-node iteration via scan_iter is. Already addressed on a separate branch.
  • async_planner appends the key separator where the other three sites do not, so it samples a narrower key set than the index covers — FT.CREATE PREFIX is a literal string-prefix match (verified: PREFIX 1 zzglobgl[ab] indexes only its own keys), which makes the sync planner the correct one. Flagged in a comment here rather than changed, since fixing it is a behaviour change. The natural follow-up consolidates all three pattern builders and drops build_scan_match_patterns' unused key_separator parameter.
  • delete_route_references splits with a hardcoded ":" rather than key_separator, and raises after drop_keys has already deleted.

Note

Medium Risk
Changes destructive SCAN/delete and migration enumeration behavior for names with glob metacharacters (fixes incorrect cross-key deletion); otherwise localized to pattern building with broad test coverage.

Overview
Fixes a data-loss bug where caller-supplied cache names, index prefixes, or route names containing Redis glob characters (*, ?, [, \) were interpolated into SCAN MATCH patterns unescaped, so destructive paths could delete or touch keys the caller did not own (e.g. cache[ab] clearing cachea/cacheb).

Adds match_pattern(*segments) in redisvl/utils/utils.py to build escaped literal-prefix patterns and routes all pattern construction through it: BaseCache.clear / aclear, SemanticRouter._route_pattern, build_scan_match_patterns (migration executor/validation), and sync/async migration key sampling.

docs/user_guide/installation.md gains ACL notes (TTL refresh needs EXPIRE + write key access) and documents that ACL key patterns use the same glob semantics as SCAN MATCH.

Unit tests assert emitted patterns; integration tests verify Redis behavior for colliding cache names and migration/router prefixes.

Reviewed by Cursor Bugbot for commit edc99b3. Bugbot is set up for automated code reviews on this repo. Configure here.

SCAN/KEYS MATCH patterns are glob-style, but several call sites
interpolated a caller-supplied literal straight into one. A cache name,
index prefix or route name containing *, ?, [ or \ therefore produced a
pattern matching keys the caller did not own -- and on the destructive
paths those keys were then deleted or rewritten.

Verified on Redis 8.4.5: SemanticCache(name="cache[ab]").clear() deletes
every entry under cachea: and cacheb: while leaving its own keys intact,
silently and with no error. The same shape reaches delete_route_references
and the migration executor's RENAME/DUMP-RESTORE-DEL path.

Patterns are now built with match_pattern(), which escapes each literal
segment and appends the trailing glob. Exposing a builder rather than a
bare escaper is deliberate: escaping stops being a step a future call site
can forget, which matters because no lint rule can express the invariant
(ruff and flake8 are unconfigured, mypy is not strict, and patterns are
consumed several call sites away from where they are built).

Covered sites: BaseCache.clear/aclear, SemanticRouter._route_pattern,
build_scan_match_patterns (and through it executor, async_executor,
validation, async_validation), and both planners' key sampling.

Escaping rather than rejecting such names, because rejecting is a breaking
change that orphans keys: a deployment whose cache is already named
"cache[ab]" could no longer construct the object to clean up after itself.

Tests split by what can actually answer each question. That RedisVL emits
the right pattern is an exact-string assertion, needing no matcher. That
Redis then interprets it as intended is asked of a real server: the
existing cache regression in test_embedcache.py, and a new migration and
router prefix case in test_migration_v1.py. No glob matcher is reimplemented
anywhere, so nothing can drift from Redis.

Two pre-existing issues in the same neighbourhood are deliberately left
alone: BaseCache.clear/aclear never advance a Mapping cursor on Redis
Cluster (fixed separately by replacing the hand-rolled loop with
scan_iter), and async_planner appends the key separator where the other
three sites do not, so it samples a narrower key set than the index covers.
Two things that surprise anyone assembling a least-privilege credential,
both measured on Redis 8.4.5 rather than inferred.

A cache read is not read-only: with a TTL configured, every hit refreshes
the matched entries' TTL, so the read path issues EXPIRE, which is in
@Write and not @READ. A lookup-only credential therefore fails on a cache
hit rather than on the write that populated the entry. Granting the command
is only half of it -- ACL DRYRUN shows +expire under a read-only key
pattern such as %R~llmcache:* is denied on the key, and that is the shape
the Key permissions table presents as sufficient for querying. Placed in
"Roles built from @READ and @Write", where a reader assembling exactly that
role will be.

ACL key patterns are also glob-style, matched by the same engine as SCAN
MATCH, so they carry the same metacharacter trap: ~cache[ab]:* grants
cachea:1 and cacheb:1 and denies the literal cache[ab]:1.
@vishal-bala vishal-bala added the auto:patch Increment the patch version when merged label Aug 24, 2026
@vishal-bala
vishal-bala marked this pull request as ready for review August 25, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:patch Increment the patch version when merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant