fix(python): Shut workers down without raising from signal handlers - #784
Merged
untitaker merged 4 commits intoAug 25, 2026
Conversation
Reland of #764 (reverted in e9d1494) with the bug that forced the revert fixed. Signal handlers now only flip a bool instead of raising KeyboardInterrupt. Raising unwinds at an arbitrary bytecode and can leave locks held by the interrupted code in a broken state, which is where the `ValueError: semaphore or lock released too many times` crashes came from. Anything that takes a lock, including `Event.set()` and `server.stop()`, is also unsafe to call from a handler, so `ShutdownSignal.request()` does nothing but assign. The previous attempt exited on its own shortly after startup because the serve loop read the return value of `grpc.Server.wait_for_termination(timeout=...)` as "the server terminated". It actually returns True when the timeout elapsed, i.e. while the server is healthy, and False once it has terminated - the inverse of `Event.wait()`. A healthy worker therefore broke out of the loop after one poll interval and shut down cleanly with exit code 0. The serve loop no longer looks at that return value at all. It sleeps on `ShutdownSignal.wait()`, so our own flag is the only thing that can end it. ref STREAM-1649
- Notice a gRPC server that terminated on its own again. Polling only the shutdown flag meant an internally failed server left the parent running with live children and a green health check. Read wait_for_termination(timeout=...) with its real semantics: True means the timeout elapsed and the server is still up, False means it terminated. - Drop a pull-mode task claimed while shutting down. get_task() blocks with no deadline, so SIGTERM can land mid-RPC; handing the activation to a child claims work we won't run, which then has to expire on the broker before anyone else picks it up.
…tion Verified by reintroducing the inverted boolean and re-running: previously one test failed, one hung until timeout, and the SIGTERM test passed regardless. Now all three fail fast. - test_push_start_exits_cleanly_on_sigterm fired SIGTERM from the first poll, so the loop exited after one iteration whichever way the exit condition was read. That is how the inverted boolean got through review. Deliver the signal on the third poll instead, so surviving to it proves the condition is being exercised. - test_push_start_exits_when_server_terminates_unexpectedly spun forever when the loop ignored a terminated server. Bail out after a few polls so it fails with a message instead of hanging CI.
Mutation check: making request() also call _event.set() -- which destroys the whole point of the class, since Event.set() takes the lock a signal handler must not touch -- passed all 18 shutdown tests. The two wakeup tests look like they cover this but cannot: one allows <5s, loose enough to pass whether request() polls or wakes instantly, and the other goes through set(), which the mutation makes identical to request(). Assert the contract directly instead: request() flips the bool and leaves the event alone, set() sets it. The mutation now fails.
untitaker
marked this pull request as ready for review
August 25, 2026 09:52
lvthanh03
approved these changes
Aug 25, 2026
untitaker
deleted the
markusunterwaditzer/stream-1649-fix-taskworker-crashes-during-shutdown
branch
August 25, 2026 14:48
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reland of #764 (reverted in
e9d1494). See that PR for the originalinvestigation into the
KeyboardInterrupt-from-a-signal-handler crashes; theapproach here is unchanged. What follows is only what is new.
Why it was reverted
One inverted boolean:
grpc returns True when the timeout elapsed (server healthy) and False once
it terminated — the inverse of
Event.wait(), despite the docstring(
grpc/_common.py:161-164). So healthy workers left the loop after one 0.5spoll and exited 0: clean, no traceback, half a second after warmup. Push mode
is the production default. Now read the right way round, plus a log line when
the server dies unasked.
Every other exit path from
start()was checked: the early returns all requirethe shutdown bool, and exceptions exit non-zero. This was the only
spontaneous-exit vector.
Why CI missed it
test_push_start_exits_cleanly_on_sigtermfired SIGTERM from the first poll,so the loop exited after one iteration whichever way the condition was read.
Verified by reintroducing the bug: one test failed, one hung, this one passed.
Fixed by signalling on the third poll, so surviving to it proves the condition
runs; all three now fail fast.
Caveat: these tests mock
wait_for_termination, so they prove the loop matchesour belief about grpc's contract, not the contract itself. I checked that
against a real server by hand.
Other changes from review
fetch_task()re-checks the flag after the RPC and drops the activation.This does not speed up shutdown; it stops us claiming a task we won't run,
which would otherwise sit until it expires on the broker. Metric:
taskworker.worker.fetch_task.dropped_during_shutdown.wait_for_terminationentirely and slept onShutdownSignal.wait(). Immune to the inversion, but it lost detection of aserver that failed on its own, leaving the parent alive with a green health
check.
Known regression: pull-worker shutdown latency
stub.GetTask()(client.py:415) has no deadline. cygrpc turns aKeyboardInterruptraised from a handler into RPC cancellation, so the old codeaborted an in-flight
GetTaskimmediately. Flipping a bool does not, so theworker now stays in that call until it returns. Normally that is 20-50ms and
irrelevant, but on a hung connection it means ignoring SIGTERM until k8s
SIGKILLs the pod. The
fetch_taskre-check above limits the damage (we drop thetask rather than run it) but not the delay. Fixing it properly means putting a
deadline on the RPC — follow-up below.
Follow-ups (not in this PR)
GetTaskand on the client's all-hosts-unavailabletime.sleep(), per the regression above. This is the one that keepspull-worker shutdown from being fully solved.
threading.Event.set()(
workerchild.py:207), which is exactly what this PR'sShutdownSignaldocstring says is unsafe from a handler. Latent rather than live: children are
spawned, not forked (
--process-typedefaults tospawnand nothingoverrides it), so the child installs that handler as part of its own startup
and there is no inherited-handler window. Still worth making the two sides
consistent.
SIGTERM to anything watching exit codes. Arguably should be non-zero.
can only be escaped with SIGKILL. Fine under k8s, annoying for local Ctrl-C.
grpc.server(). The serve-loop tests still encodegrpc's semantics in a mock, now the correct way round.
ref STREAM-1649