Skip to content

Bound storage usage cache task runtime - #11875

Open
kvnloo wants to merge 1 commit into
CenterForOpenScience:developfrom
kvnloo:fix/11816-storage-usage-time-limits
Open

Bound storage usage cache task runtime#11875
kvnloo wants to merge 1 commit into
CenterForOpenScience:developfrom
kvnloo:fix/11816-storage-usage-time-limits

Conversation

@kvnloo

@kvnloo kvnloo commented Aug 24, 2026

Copy link
Copy Markdown

Closes #11816

Problem

update_storage_usage_cache can hold a Celery worker indefinitely when its paginated storage query stalls.

Approach

  • set a 270-second soft limit and 300-second hard limit on the task
  • log the affected target_guid with traceback when the soft limit fires
  • re-raise SoftTimeLimitExceeded so Celery preserves timeout semantics

Verification

  • focused regression tests: 2 passed
  • flake8 api/caching/tasks.py api/caching/test_tasks.py
  • python -m compileall -q api/caching/tasks.py api/caching/test_tasks.py
  • git diff --check origin/develop...HEAD
  • strict RED → GREEN verification, including sabotage runs for both the task limits and timeout logging behavior
  • independently reviewed with no blocking safety, security, or test-adequacy findings

Risk

The change only affects the storage-usage cache task after 270 seconds of execution. The existing disabled-cache early return remains outside the timeout handler, and timed-out work is logged then propagated rather than silently swallowed.

AI assistance was used for implementation and test review; the contributor inspected the diff and ran the verification listed above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Celery task update_storage_usage_cache has no time_limit — worker may hang indefinitely

1 participant