fix(celery): stop soft-limit signals escaping as worker-killing errors

[vpetersson-bot]

What's wrong today

Two separate causes were feeding the same family of Sentry issues on the current release — the SoftTimeLimitExceeded events (ANTHIAS-5K, ANTHIAS-5Y, ANTHIAS-45) and, downstream of them, the hard-timeout kill trio (ANTHIAS-A / ANTHIAS-9 / ANTHIAS-B) plus the pool-replacement timeout (ANTHIAS-1Q).

1. revalidate_asset_url left its preamble unguarded

The task wrapped its probe and write-back in except SoftTimeLimitExceeded, but everything before that ran bare:

try:
    asset = Asset.objects.get(asset_id=asset_id)
except Asset.DoesNotExist:
    return
...
if not r.set(asset_recheck_lock_key(asset_id), '1', nx=True, ex=RECHECK_COOLDOWN_S):
    return

That preamble is all blocking I/O — a SQLite SELECT and a Redis round trip. On a memory-pressured board it is slow enough to matter: a reported event has the soft-limit signal landing inside Asset.objects.get, on a fetch that had been running ~85s, with the traceback bottoming out in django/utils/dateparse.py under billiard's soft_timeout_sighandler. except Asset.DoesNotExist cannot catch that, so it escaped as a task failure — and left the task running into the 90s hard limit, which SIGKILLs the pool child.

That last part is the expensive half. Sampling the hard-limit issue's events on 2026.8.2, revalidate_asset_url is the single largest contributor (23 of 53 sampled kills, ahead of apply_display_power_schedule at 12 and get_display_power at 10) — and every kill also files a separate signal 9 (SIGKILL) event and can strand the pool waiting on a replacement child.

2. Soft-limit signals arriving after the job already finished

billiard sends SIGUSR1 from the parent when a job passes its soft_time_limit, and the parent only learns the job completed once the result reaches it over the pipe. A job that finishes just inside its budget therefore leaves a signal in flight, and it lands in whatever the child does next:

• celery.fixups.django.on_task_postrun → close_cache, closing Django's caches after the task returned (ANTHIAS-5Y — the breadcrumbs show the task's own Redis GET/DEL completing first)
• the pool child's idle workloop → receive → _recv, blocked waiting for its next job (ANTHIAS-45)

Neither has a task body to catch it, so celery reports an unhandled error for work that succeeded. No try/except inside a task can fix this — the task isn't on the stack.

What this changes

revalidate_asset_url — the soft-limit guard now covers the whole body rather than just the probe and write-back. The inner handler is unchanged and still comes first, so a probe that genuinely runs out of budget is recorded as unreachable rather than abandoned.

_sentry_before_send — drops a SoftTimeLimitExceeded whose traceback contains no task body, i.e. no Anthias frame and no celery.app.trace frame. A task wedged deep inside a third-party library still traces back through celery's tracer, so real overruns are kept; only the post-completion strays are dropped. Same expected-transient rationale as the redis / gaierror / client-disconnect arms already in the filter.

How I know it's right

Five new tests, four of which fail on master and pass here:

• soft limit raised from the opening Asset.objects.get → task succeeds, probe never runs (fails on master)
• soft limit raised from the SETNX cooldown gate → same (fails on master)
• soft limit raised from the probe → still recorded as unreachable (guards the inner handler against being shadowed by the widened outer one)
• before_send drops a signal whose frame belongs to celery.fixups.django and one belonging to billiard.pool (both fail on master)
• before_send keeps one whose frame belongs to anthias_server.celery_tasks, one belonging to celery.app.trace, and one with no traceback at all

The before_send tests synthesise a frame with the module name under test rather than importing celery's pool internals, which keeps them independent of billiard's private layout.

Full suite: 2213 passed, 3 skipped. ruff check, ruff format --check and mypy . are clean (the two pre-existing tools/image_builder import-not-found errors are unrelated and unchanged).

Follow-ups not in this PR

apply_display_power_schedule and get_display_power still reach the 60s hard limit occasionally despite catching the soft limit at 30s — that is the backstop firing for a call stuck where the signal can't be delivered, and needs its own investigation. The webview launch failures (ANTHIAS-D, ANTHIAS-1W, ANTHIAS-3) are unrelated to this change.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

Fix the post-task traceback filtering case and remove the unnecessary type suppression.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Avoid type-ignore by narrowing the dictionary value at runtime · New

What changed in this PR

This PR hardens Celery soft-timeout handling and filters late timeout signals from Sentry.

Changes:

• Guards the complete asset revalidation flow.
• Adds traceback-based Sentry filtering.
• Adds regression tests for timeout and filtering behavior.

File	Summary
tests/​test_sentry.py	Adds filtering tests; avoid the unnecessary type: ignore suppression.
tests/​test_celery_tasks.py	Adds coverage for soft-limit handling paths.
src/​anthias_server/​django_project/​settings.py	Adds traceback filtering; post-task signals may still be retained through celery.app.trace.
src/​anthias_server/​celery_tasks.py	Expands soft-limit protection across the task body.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Copilot]

Copilot review overview

🔵 Needs a closer look

The Sentry filtering logic needs to distinguish post-task teardown from active task execution, with corresponding coverage.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Avoid type-ignore by narrowing the dictionary value at runtime

Previously missed (1)

In code that hasn't changed since last review

Postrun signal check misidentifies active Celery task frames

src/​anthias_server/​django_project/​settings.py:159

task_postrun receivers are invoked from Celery's celery.app.trace.trace_task while that frame is still on the stack. Therefore a soft signal delivered in celery.fixups.django.on_task_postrun can still have a celery.app.trace traceback frame, making this check return False and retaining the ANTHIAS-5Y event that the new test claims to drop. The synthetic test only creates the receiver frame, so it does not exercise the real caller stack; distinguish post-task teardown from an executing task (or use the actual task-frame boundary) and add coverage with the tracer frame present.

[vpetersson-bot]

Closing this — it has been superseded by the celery soft-limit fix that landed on master while this was open. That change carries both of this PR's fixes plus the root cause this one missed, so rebasing would leave nothing behind.

I checked rather than assumed: running this PR's 140 tests against master's source, 139 pass. The single failure is the one behavioural difference between the two, and master is right and this PR is wrong on it — see below.

What master already has:

• The same whole-body soft-limit guard on revalidate_asset_url, with its own regression tests for a signal landing in the prologue SELECT and in the cooldown SETNX.
• The same before_send rule for soft-limit signals delivered after the task body returned, with tests for both landing sites.

What master has that this PR missed: the actual root cause of the hard-timeout kills. SoftTimeLimitExceeded is a plain Exception subclass, and two library helpers on the periodic-poke paths (cec_client.available() and telemetry._get_asset_counts()) caught it in a blanket except Exception, logged it as an ordinary fault and carried on — so the task's own handler never ran and the task continued into the hard limit. I had flagged the display-power share of those kills as needing its own investigation; that is the answer, and it explains why three previous rounds of soft limits never closed the group.

Two ways master's version is better than this one:

1. This PR imported celery.exceptions inside _sentry_before_send. The viewer imports that settings module and its image ships no celery, so the import would have raised ModuleNotFoundError in the viewer process the first time it tried to report an event — breaking viewer crash reporting on every board. Master matches by name and module instead, the way the existing yt-dlp rule does, and adds an import-closure guard test that would have caught it.
2. Master's rule is "no Anthias frame anywhere in the traceback", where this PR additionally kept anything with a celery.app.trace frame. That extra clause is exactly what Copilot's second review flagged, and the objection holds: a signal landing in celery's own machinery after the body returned has nothing to catch it either, so keeping it just preserves noise. Master's simpler rule drops it.

For the record on Copilot's finding as written — that a celery.app.trace frame would cause ANTHIAS-5Y to be retained — the retention would not actually have happened on that signature. Celery's Signal.send catches receiver exceptions itself, so the traceback is truncated at send and never reaches the tracer frame, which matches the captured event exactly. But the remedy Copilot pointed at was the right one regardless, and master took it.