Skip to content

refactor(sc): unify capture retries and extract orchestration components - #4073

Draft
pthombre wants to merge 5 commits into
pranav/reassembler-group-fetchfrom
pranav/capture-orchestration
Draft

refactor(sc): unify capture retries and extract orchestration components#4073
pthombre wants to merge 5 commits into
pranav/reassembler-group-fetchfrom
pranav/capture-orchestration

Conversation

@pthombre

Copy link
Copy Markdown
Contributor

What does this PR do?

Capture dispatch did not honor the legacy path's skipped-prompt budget, and retry accounting was duplicated. Share retry handling, make capture honor the budget and replacement policy, and extract the actor-pool and worker capture responsibilities into explicit components.

  • A capture prompt that exhausts data retries can now be skipped within max_skipped_prompts; zero retains fail-fast behavior. Replacement selection and lineage/shortfall accounting remain checkpoint-atomic.
  • Keep cleanup failures fatal and preserve cancellation, separate infrastructure/data counters, and downstream enrichment failure behavior.
  • Extract RolloutReassemblerPool while the controller retains its mutation cut across the RPC, replay-buffer commit, and recovery-ledger update. Unknown publication outcomes remain fatal and actors are not retried or reused.
  • Extract TokenCaptureHost, finish actor-field renaming, and preserve optional Gym imports, the pinned Gym contract, and existing dashboard metric keys.
  • Remove the obsolete configuration warning and update the single-controller guide.

Issues

Follow-up to #3837. Cross-repository Gym route-fallback diagnostics are deferred.

PR stack

This is the second PR in the stack: mainpranav/reassembler-group-fetchpranav/capture-orchestration.

Depends on #4072 (reassembler performance).

Base this PR on pranav/reassembler-group-fetch so the diff contains only orchestration changes. After the performance PR merges, rebase/retarget this PR to main.

Validation

  • Slurm job 3656944, final commit 59eeafcf9b6be51352dc4fa39be5b98674f60ce9: completed with exit 0:0.
  • 1,069 default/controller tests + 76 Gym tests passed, zero failures or skips; includes 16 new orchestration regression cases.
  • Scoped pyrefly comparison in the same container: 39 current diagnostics versus 40 on the original base, with no new diagnostics. Existing worker/setup/config diagnostics remain; this is not a clean repository-wide type check.
  • Ruff, import sorting, formatting, compilation, and diff checks passed.
  • Used the existing rl-gym container and its Gym environment. Full multi-node training was not run.

Before ready for review

  • Followed contributor guidelines; commits include sign-off trailers.
  • Added and ran relevant unit/regression tests through Slurm.
  • Updated the single-controller guide and type-check allowlist.
  • Run a representative full training/functional workload.

This PR is intentionally a draft.

Pranav Thombre added 5 commits September 9, 2026 23:23
Signed-off-by: Pranav Thombre <pthombre@aws-cmh-slurm-1-vscode-01.cm.cluster>
Signed-off-by: Pranav Thombre <pthombre@aws-cmh-slurm-1-vscode-01.cm.cluster>
Signed-off-by: Pranav Thombre <pthombre@aws-cmh-slurm-1-vscode-01.cm.cluster>
Signed-off-by: Pranav Thombre <pthombre@aws-cmh-slurm-1-vscode-01.cm.cluster>
Signed-off-by: Pranav Thombre <pthombre@aws-cmh-slurm-1-vscode-01.cm.cluster>
@pthombre pthombre added the CI:L1 Run doctests, unit tests, and functional tests label Sep 10, 2026
@copy-pr-bot

copy-pr-bot Bot commented Sep 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Sep 10, 2026
@pthombre
pthombre added this pull request to stack #4074 September 10, 2026 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant