Skip to content

improvement(redis): pair lock acquire failures with connection state - #7669

Merged
waleedlatif1 merged 2 commits into
stagingfrom
fix/redis-lock-timeout-diagnostics
Sep 9, 2026
Merged

improvement(redis): pair lock acquire failures with connection state#7669
waleedlatif1 merged 2 commits into
stagingfrom
fix/redis-lock-timeout-diagnostics

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • A lock acquire is often a process's first Redis call, so an unusable connection surfaces there as Error: Command timed out — a rejection carrying only ioredis timer frames, no app frame, and nothing to say which of several very different causes applied
  • describeRedisConnection() already exists for exactly this and is wired into the usage-reservation path; this extends it to acquireLock, which covers every lock caller
  • status is what separates the cases: connecting/reconnecting means the command was parked waiting on connection setup, ready means it went to a live socket that never answered
  • Read before the reclaim, which awaits and would otherwise report the state it left behind rather than the one that failed

Logging only — no behavior change. The failure path is unchanged: same error, same rethrow, same reclaim. Nothing is logged on the taken or contended paths, which the poll routes run constantly.

Type of Change

  • Improvement (observability)

Testing

Three tests added: the failure is paired with connection state, the state is read before the reclaim, and the success/contended paths stay quiet. Each verified to fail when its behavior is reverted — the ordering test goes red when the diagnostic read is moved after the reclaim.

bun run type-check clean. 3,893 tests pass across lib/core/config, lib/table, lib/billing, background, lib/concurrency, lib/mcp, lib/workspace-files. bun run lint, block-registry check, docs-manifest:check, and all 46 audits in check:audits pass.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

A lock acquire is often a process's first Redis call, so an unusable
connection surfaces there as `Error: Command timed out` — a rejection
carrying only ioredis timer frames, no app frame, and nothing to separate a
handshake still in flight from a socket that died silently. `status` is what
separates them, so log it alongside the failure.

Read before the reclaim, which awaits and would otherwise report the state it
left behind rather than the one that failed.
@vercel

vercel Bot commented Sep 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Sep 9, 2026 7:20pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR improves Redis lock-acquisition failure diagnostics by logging the state of the exact client that executed the failed command.

  • Captures connection status before the optional reclaim operation can alter it.
  • Extends describeRedisConnection to accept a specific Redis client while withholding timestamps that belong to another client.
  • Adds coverage for failures, client replacement, reclaim ordering, and quiet success/contention paths.
  • The previously reported replacement-client diagnostic issue is fixed in the current code.

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness, security, or repository-rule issues identified.

The lock failure path preserves its existing rethrow and reclaim behavior while recording diagnostics from the failing client before asynchronous cleanup can change connection state. The previous replacement-client finding is fully addressed by passing the captured client and withholding globally tracked ages after client identity diverges.

Important Files Changed

Filename Overview
apps/sim/lib/core/config/redis.ts Adds failure-path lock diagnostics and correctly associates status with the Redis client that executed the command.
apps/sim/lib/core/config/redis.test.ts Adds focused regression tests for diagnostic state, ordering, client replacement, and non-failure logging behavior.

Reviews (2): Last reviewed commit: "fix(redis): describe the client that ran..." | Re-trigger Greptile

Comment thread apps/sim/lib/core/config/redis.ts Outdated
A command can outlive the client that issued it: the PING health check drops
`state.client` after consecutive failures, which is the same unhealthy stretch
in which that command is timing out. Reading the global in the failure path
then described the replacement — reporting `no-client` or a fresh `connecting`
for a failure belonging to the connection before it, misclassifying the very
timeout the diagnostic exists to explain.

Take the client as an argument, and withhold the ages when `state` no longer
holds it rather than dating a connection its timestamps never measured.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@waleedlatif1
waleedlatif1 merged commit 9fd3177 into staging Sep 9, 2026
25 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/redis-lock-timeout-diagnostics branch September 9, 2026 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant