Skip to content

feat(taskworker): Add child fetch wait duration metric - #787

Merged
enochtangg merged 1 commit into
mainfrom
track-children-wait
Aug 27, 2026
Merged

feat(taskworker): Add child fetch wait duration metric#787
enochtangg merged 1 commit into
mainfrom
track-children-wait

Conversation

@enochtangg

Copy link
Copy Markdown
Contributor

Description

Adds taskworker.worker.child.fetch_wait_duration distribution metric of how long each child blocks in child_tasks.get() waiting for work.

Question we'd like to answer:
When a child is idle, is it because the parent has nothing to give it, or because it can't get at work that's already sitting in the queue?

In process-segments-push (s4s2), children are idle ~55% of the time while child_tasks.size averages 8.97. A get() on a queue that deep should return in quickly. We can't currently tell which of those is wrong.

Existing metrics don't answer it. child_task_queue_empty only fires after a full 1s timeout, so it counts famines and misses the normal case. Occupancy gives the aggregate but not the shape, and the shape is what identifies the cause:

Signature Cause
Long waits while child_tasks.size > 0 _rlock contention, children serializing on one queue lock
Long waits only while size == 0 Supply-limited upstream of the parent

That distinction decides how to update the autoscaling metric

@enochtangg
enochtangg requested a review from a team as a code owner August 26, 2026 21:28
@enochtangg
enochtangg merged commit ca76843 into main Aug 27, 2026
29 checks passed
@enochtangg
enochtangg deleted the track-children-wait branch August 27, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants