Skip to content

Fix async thread-pool deadlock from semaphore-blocked B-team workers - #9372

Merged
mcourteaux merged 1 commit into
mainfrom
alexreinking/async-deadlock
Aug 22, 2026
Merged

Fix async thread-pool deadlock from semaphore-blocked B-team workers#9372
mcourteaux merged 1 commit into
mainfrom
alexreinking/async-deadlock

Conversation

@alexreinking

Copy link
Copy Markdown
Member

A worker that found a job it could run but for an unavailable semaphore went to sleep on one of the two idle worker condition variables (the A/B teams), chosen by pool-size bookkeeping unrelated to why it slept. A semaphore release only ever broadcast wake_a_team, so a semaphore-blocked worker demoted to the B team was never woken by the release that made its job runnable, and the pipeline could deadlock with every thread parked.

The A/B teams model idle capacity (no runnable work), which is a different state from being blocked on an external event. Give blocked-on-semaphore workers their own wait channel and wake it on every release, alongside stalled owners. Genuinely-idle A/B-team workers are never waiting on a semaphore, so they are left undisturbed.

This also removes the old 0 -> 1-transition gate on the release wakeup, which was independently unsound for acquires of count > 1 satisfied by several count-1 releases in a row.

Adds test/correctness/async_deadlock.cpp, pinned to a low thread count (where the pool reshuffles teams and reliably exposes the bug).

Checklist

  • Tests added or updated (not required for docs, CI config, or typo fixes)
  • Documentation updated (if public API changed)
  • Python bindings updated (if public API changed)
  • Benchmarks are included here if the change is intended to affect performance.
  • Commits include AI attribution where applicable (see Code of Conduct)

A worker that found a job it could run but for an unavailable semaphore
went to sleep on one of the two idle worker condition variables (the A/B
teams), chosen by pool-size bookkeeping unrelated to why it slept. A
semaphore release only ever broadcast wake_a_team, so a semaphore-blocked
worker demoted to the B team was never woken by the release that made its
job runnable, and the pipeline could deadlock with every thread parked.

The A/B teams model idle capacity (no runnable work), which is a different
state from being blocked on an external event. Give blocked-on-semaphore
workers their own wait channel and wake it on every release, alongside
stalled owners. Genuinely-idle A/B-team workers are never waiting on a
semaphore, so they are left undisturbed.

This also removes the old 0 -> 1-transition gate on the release wakeup,
which was independently unsound for acquires of count > 1 satisfied by
several count-1 releases in a row.

Adds test/correctness/async_deadlock.cpp, pinned to a low thread count
(where the pool reshuffles teams and reliably exposes the bug).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alexreinking
alexreinking requested a review from abadams August 20, 2026 18:58

@abadams abadams left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in future we could use a parallel deadlock fuzzer that tries random nestings of .parallel and .async and runs under tsan.

@mcourteaux mcourteaux left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea of the separate channel looks great, but I think there is still a race condition.

Comment thread src/runtime/thread_pool_common.h
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.06%. Comparing base (bb5426d) to head (168926f).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9372      +/-   ##
==========================================
- Coverage   70.08%   70.06%   -0.02%     
==========================================
  Files         259      260       +1     
  Lines       79158    79287     +129     
  Branches    19293    19327      +34     
==========================================
+ Hits        55477    55553      +76     
- Misses      17886    17916      +30     
- Partials     5795     5818      +23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcourteaux
mcourteaux merged commit abbf7ff into main Aug 22, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants