Skip to content

fix(python): Shut workers down without raising from signal handlers - #784

Merged
untitaker merged 4 commits into
mainfrom
markusunterwaditzer/stream-1649-fix-taskworker-crashes-during-shutdown
Aug 25, 2026
Merged

fix(python): Shut workers down without raising from signal handlers#784
untitaker merged 4 commits into
mainfrom
markusunterwaditzer/stream-1649-fix-taskworker-crashes-during-shutdown

Conversation

@untitaker

@untitaker untitaker commented Aug 25, 2026

Copy link
Copy Markdown
Member

Reland of #764 (reverted in e9d1494). See that PR for the original
investigation into the KeyboardInterrupt-from-a-signal-handler crashes; the
approach here is unchanged. What follows is only what is new.

Why it was reverted

One inverted boolean:

if server.wait_for_termination(timeout=SHUTDOWN_POLL_INTERVAL_SEC):
    break

grpc returns True when the timeout elapsed (server healthy) and False once
it terminated — the inverse of Event.wait(), despite the docstring
(grpc/_common.py:161-164). So healthy workers left the loop after one 0.5s
poll and exited 0: clean, no traceback, half a second after warmup. Push mode
is the production default. Now read the right way round, plus a log line when
the server dies unasked.

Every other exit path from start() was checked: the early returns all require
the shutdown bool, and exceptions exit non-zero. This was the only
spontaneous-exit vector.

Why CI missed it

test_push_start_exits_cleanly_on_sigterm fired SIGTERM from the first poll,
so the loop exited after one iteration whichever way the condition was read.
Verified by reintroducing the bug: one test failed, one hung, this one passed.
Fixed by signalling on the third poll, so surviving to it proves the condition
runs; all three now fail fast.

Caveat: these tests mock wait_for_termination, so they prove the loop matches
our belief about grpc's contract, not the contract itself. I checked that
against a real server by hand.

Other changes from review

  • fetch_task() re-checks the flag after the RPC and drops the activation.
    This does not speed up shutdown; it stops us claiming a task we won't run,
    which would otherwise sit until it expires on the broker. Metric:
    taskworker.worker.fetch_task.dropped_during_shutdown.
  • An earlier revision here dropped wait_for_termination entirely and slept on
    ShutdownSignal.wait(). Immune to the inversion, but it lost detection of a
    server that failed on its own, leaving the parent alive with a green health
    check.

Known regression: pull-worker shutdown latency

stub.GetTask() (client.py:415) has no deadline. cygrpc turns a
KeyboardInterrupt raised from a handler into RPC cancellation, so the old code
aborted an in-flight GetTask immediately. Flipping a bool does not, so the
worker now stays in that call until it returns. Normally that is 20-50ms and
irrelevant, but on a hung connection it means ignoring SIGTERM until k8s
SIGKILLs the pod. The fetch_task re-check above limits the damage (we drop the
task rather than run it) but not the delay. Fixing it properly means putting a
deadline on the RPC — follow-up below.

Follow-ups (not in this PR)
  • Deadline on GetTask and on the client's all-hosts-unavailable
    time.sleep(), per the regression above. This is the one that keeps
    pull-worker shutdown from being fully solved.
  • The child's SIGTERM handler calls threading.Event.set()
    (workerchild.py:207), which is exactly what this PR's ShutdownSignal
    docstring says is unsafe from a handler. Latent rather than live: children are
    spawned, not forked (--process-type defaults to spawn and nothing
    overrides it), so the child installs that handler as part of its own startup
    and there is no inherited-handler window. Still worth making the two sides
    consistent.
  • Unexpected server termination returns 0, so it looks identical to a clean
    SIGTERM to anything watching exit codes. Arguably should be non-zero.
  • A second SIGTERM is now inert — the bool is already set, so a hung drain
    can only be escaped with SIGKILL. Fine under k8s, annoying for local Ctrl-C.
  • No test drives a real grpc.server(). The serve-loop tests still encode
    grpc's semantics in a mock, now the correct way round.

ref STREAM-1649

Reland of #764 (reverted in e9d1494) with the bug that forced the revert
fixed.

Signal handlers now only flip a bool instead of raising KeyboardInterrupt.
Raising unwinds at an arbitrary bytecode and can leave locks held by the
interrupted code in a broken state, which is where the
`ValueError: semaphore or lock released too many times` crashes came from.
Anything that takes a lock, including `Event.set()` and `server.stop()`, is
also unsafe to call from a handler, so `ShutdownSignal.request()` does
nothing but assign.

The previous attempt exited on its own shortly after startup because the
serve loop read the return value of
`grpc.Server.wait_for_termination(timeout=...)` as "the server terminated".
It actually returns True when the timeout elapsed, i.e. while the server is
healthy, and False once it has terminated - the inverse of `Event.wait()`.
A healthy worker therefore broke out of the loop after one poll interval and
shut down cleanly with exit code 0.

The serve loop no longer looks at that return value at all. It sleeps on
`ShutdownSignal.wait()`, so our own flag is the only thing that can end it.

ref STREAM-1649
@linear-code

linear-code Bot commented Aug 25, 2026

Copy link
Copy Markdown

STREAM-1649

- Notice a gRPC server that terminated on its own again. Polling only the
  shutdown flag meant an internally failed server left the parent running
  with live children and a green health check. Read
  wait_for_termination(timeout=...) with its real semantics: True means the
  timeout elapsed and the server is still up, False means it terminated.

- Drop a pull-mode task claimed while shutting down. get_task() blocks with
  no deadline, so SIGTERM can land mid-RPC; handing the activation to a
  child claims work we won't run, which then has to expire on the broker
  before anyone else picks it up.
…tion

Verified by reintroducing the inverted boolean and re-running: previously
one test failed, one hung until timeout, and the SIGTERM test passed
regardless. Now all three fail fast.

- test_push_start_exits_cleanly_on_sigterm fired SIGTERM from the first
  poll, so the loop exited after one iteration whichever way the exit
  condition was read. That is how the inverted boolean got through review.
  Deliver the signal on the third poll instead, so surviving to it proves
  the condition is being exercised.

- test_push_start_exits_when_server_terminates_unexpectedly spun forever
  when the loop ignored a terminated server. Bail out after a few polls so
  it fails with a message instead of hanging CI.
Mutation check: making request() also call _event.set() -- which destroys
the whole point of the class, since Event.set() takes the lock a signal
handler must not touch -- passed all 18 shutdown tests.

The two wakeup tests look like they cover this but cannot: one allows <5s,
loose enough to pass whether request() polls or wakes instantly, and the
other goes through set(), which the mutation makes identical to request().

Assert the contract directly instead: request() flips the bool and leaves
the event alone, set() sets it. The mutation now fails.
@untitaker
untitaker marked this pull request as ready for review August 25, 2026 09:52
@untitaker
untitaker requested a review from a team as a code owner August 25, 2026 09:52
@untitaker
untitaker merged commit a1f106f into main Aug 25, 2026
29 checks passed
@untitaker
untitaker deleted the markusunterwaditzer/stream-1649-fix-taskworker-crashes-during-shutdown branch August 25, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants