Skip to content

[#344] 🐛 - Throttle workflow queue polling and increase workflow timeouts - #345

Merged
efraespada merged 6 commits into
developfrom
bugfix/344-throttle-workflow-queue-polling-and-increase-workflow-timeouts
Sep 1, 2026
Merged

[#344] 🐛 - Throttle workflow queue polling and increase workflow timeouts#345
efraespada merged 6 commits into
developfrom
bugfix/344-throttle-workflow-queue-polling-and-increase-workflow-timeouts

Conversation

@efraespada

@efraespada efraespada commented Sep 1, 2026

Copy link
Copy Markdown
Member

📌 Summary

Improves the repository-wide Copilot workflow queue to withstand bursts without exhausting GitHub API limits, while preserving strict sequential execution and allowing long queues to drain reliably.


🎯 Related Issues / Tickets


🧩 Scope of Changes

  • Added:

    • Centralized 90-minute queue policy with exponential backoff, capped delays, and jitter.
    • Rate-limit-aware retry handling for GitHub API responses.
    • Deterministic queue, retry, and workflow-contract regression tests.
  • Updated:

    • Workflow polling to use one paginated repository traversal per poll with local filtering.
    • Handling of HTTP 429, rate-limited 403, transient HTTP, and network failures.
    • Queue-bearing workflow timeouts from 30 to 120 minutes in active and setup workflows.
    • Documentation, package version, and generated action/CLI bundles.
  • Removed:

    • Fixed two-second polling and repeated queries for each active workflow status.
  • Refactored:

    • Queue ports and composition to inject clock, randomness, delays, and polling observers.
    • Workflow contract validation around a manifest of the seven queue workflows.

🛠️ Technical Details

  • Each poll requests up to 100 workflow runs through a single paginated traversal, then filters locally by the seven Copilot/Task workflow names, active statuses, and lower run IDs.
  • Pagination continues through all pages so older matching runs are not missed.
  • Polling uses 5s, 10s, 20s, 40s, and 60s maximum delays with bounded ±20% jitter.
  • Rate-limit responses honor Retry-After and x-ratelimit-reset, with bounded fallback backoff when headers are unavailable.
  • Provider failures and queue deadlines fail closed; they cannot be interpreted as an empty queue.
  • Provider diagnostics are replaced with a canonical queue failure message to avoid exposing response bodies, URLs, headers, or credentials.
  • Queue-bearing jobs use 120-minute timeouts, providing headroom beyond the 90-minute queue deadline.
  • Mutation workflows intentionally retain no GitHub Actions concurrency block, preventing cancellation or replacement of queued runs.

🔍 How to Test

  1. Run the deterministic queue and retry tests:

    corepack pnpm@​10.12.4 exec jest \
      src/application/usecases/workflow/__tests__/wait_for_previous_workflow_runs_use_case.test.ts \
      src/data/repository/workflow/__tests__/active_previous_workflow_runs_repository.test.ts \
      src/data/repository/workflow/__tests__/workflow_runs_retry_policy.test.ts \
      src/tooling/__tests__/validate_workflow_contract.test.ts
  2. Run the project quality gates:

    corepack pnpm@​10.12.4 run typecheck
    corepack pnpm@​10.12.4 run lint
    corepack pnpm@​10.12.4 run test:coverage
    corepack pnpm@​10.12.4 run build
    corepack pnpm@​10.12.4 run validate:workflows
    corepack pnpm@​10.12.4 run validate:agent-docs
    corepack pnpm@​10.12.4 run validate:docs-page
    git diff --check
  3. Confirm the workflow contract validator accepts both .github/workflows/ and setup/workflows/, including the 120-minute queue-job timeouts and absence of mutation-workflow concurrency settings.


🧪 Test Coverage

  • Unit tests
  • Integration tests
  • End-to-end (E2E) tests
  • Manual testing only

📸 Screenshots / Recordings (UI changes only)

Not applicable — no UI changes.


⚠️ Breaking Changes

  • None to action inputs or workflow configuration.
  • Internal queue polling behavior now uses a bounded 90-minute deadline and fail-closed provider errors.

🚀 Deployment Notes

  • Requires database migration
  • Requires environment variable changes
  • Requires feature flag toggle
  • No special deployment steps

Details: Generated action and CLI bundles are included and should be released through the normal action release process. Setup workflow templates are synchronized with the active workflows.


🔒 Security Considerations

  • No security impact
  • Input validation changes
  • Authentication / authorization changes
  • Sensitive data handling changes

Provider error details are no longer interpolated into logs or externally visible failures. Diagnostic logging is limited to retry reason, attempt, delay, and reset metadata.


📈 Performance Impact

  • No performance impact
  • Improves performance
  • Potential performance regression (explain)

Reduces API fan-out from multiple status-specific queries to one paginated traversal per poll and increases polling intervals adaptively. Complete pagination and retry restarts remain necessary to preserve queue correctness.


📝 Notes for Reviewers

Please pay particular attention to the fail-closed behavior, later-page pagination coverage, rate-limit header handling, and synchronization between active and setup workflow contracts.


✅ Checklist

  • I have self-reviewed my code
  • Code follows project standards and conventions
  • Tests have been added or updated
  • Documentation has been updated (if applicable)
  • No new warnings or lint errors
  • Changes are backward compatible or breaking changes are documented

📚 Additional Context

The queue remains repository-wide across the seven Copilot/Task mutation workflows so intermediate issue, branch, and release mutations are never cancelled, skipped, or overwritten.

@docs-page

docs-page Bot commented Sep 1, 2026

Copy link
Copy Markdown

To preview the documentation for this pull request, visit the following URL:

docs.page/vypdev/copilot~345

Documentation is deployed and generated using docs.page

@vypbot vypbot changed the title Bugfix/344 throttle workflow queue polling and increase workflow timeouts [#344] 🐛 - Throttle workflow queue polling and increase workflow timeouts Sep 1, 2026
@vypbot
vypbot self-requested a review September 1, 2026 13:45
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.60870% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.58%. Comparing base (274db40) to head (75dad46).
⚠️ Report is 554 commits behind head on develop.

Files with missing lines Patch % Lines
...rc/data/repository/workflow/workflow_runs_retry.ts 74.66% 7 Missing and 12 partials ⚠️
...rkflow/active_previous_workflow_runs_repository.ts 89.65% 1 Missing and 2 partials ⚠️
...rkflow/wait_for_previous_workflow_runs_use_case.ts 90.00% 1 Missing and 1 partial ⚠️
...ogging/logger_workflow_polling_observer_adapter.ts 0.00% 2 Missing ⚠️
src/application/policies/workflow_queue_policy.ts 88.88% 1 Missing ⚠️
...ure/time/system_workflow_polling_random_adapter.ts 50.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #345      +/-   ##
===========================================
+ Coverage    90.94%   92.58%   +1.64%     
===========================================
  Files          135      467     +332     
  Lines         6251     8229    +1978     
  Branches      1338     1643     +305     
===========================================
+ Hits          5685     7619    +1934     
- Misses         172      229      +57     
+ Partials       394      381      -13     
Files with missing lines Coverage Δ
src/actions/common_action.ts 92.85% <100.00%> (-5.70%) ⬇️
src/actions/github_action.ts 85.10% <100.00%> (+3.17%) ⬆️
src/actions/main_run_lifecycle.ts 95.00% <100.00%> (ø)
...ure/composition/workflow_queue_composition_root.ts 100.00% <100.00%> (ø)
...ucture/time/system_workflow_queue_clock_adapter.ts 100.00% <100.00%> (ø)
src/utils/constants.ts 100.00% <ø> (ø)
src/application/policies/workflow_queue_policy.ts 90.00% <88.88%> (ø)
...ure/time/system_workflow_polling_random_adapter.ts 50.00% <50.00%> (ø)
...rkflow/wait_for_previous_workflow_runs_use_case.ts 92.85% <90.00%> (ø)
...ogging/logger_workflow_polling_observer_adapter.ts 66.66% <0.00%> (ø)
... and 2 more

... and 506 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vypbot vypbot added this to vypdev Sep 1, 2026
@vypbot vypbot moved this to In Progress in vypdev Sep 1, 2026
@vypbot
vypbot changed the base branch from develop to master September 1, 2026 13:46
@efraespada efraespada linked an issue Sep 1, 2026 that may be closed by this pull request
1 task
@vypbot
vypbot changed the base branch from master to develop September 1, 2026 13:46
@vypbot vypbot added size: XXL Indicates an extremely large task or issue that requires significant time and effort to complete. 95% Progress: 95% copilot:state:changes-requested and removed size: XXL Indicates an extremely large task or issue that requires significant time and effort to complete. 95% Progress: 95% labels Sep 1, 2026
@vypbot

vypbot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

🐛 Bugfix Actions

  1. The pull request's title was updated from Bugfix/344 throttle workflow queue polling and increase workflow timeouts to [#344] 🐛 - Throttle workflow queue polling and increase workflow timeouts.

  2. The pull request was assigned to @​efraespada (creator).

  3. @​vypbot was requested to review the pull request.

  4. The pull request was linked to vypdev and moved to the column In Progress.

  5. The base branch was temporarily updated to master.

  6. The description was temporarily modified to include a reference to issue 🐛🧑‍💻 - Throttle workflow queue polling and increase workflow timeouts #344.

  7. The base branch was reverted to its original value: develop.

  8. The temporary issue reference 🐛🧑‍💻 - Throttle workflow queue polling and increase workflow timeouts #344 was removed from the description.

  9. Priority set to P0 in vypdev.

  10. Potential problems detection completed (no new findings, no resolved). States: open=2, fixed=3.

  11. Lifecycle state synchronized to changes-requested.

image

Debug log
[DEBUG] Setup done. Issue number: 344, isSingleAction: false, isIssue: false, isPullRequest: true, isPush: false
[INFO] Running PullRequestUseCase for PR #345.
[INFO] 🔀 Executing PullRequestUseCase.
[DEBUG] PR action opened
[DEBUG] PR isOpened true
[DEBUG] PR isMerged false
[DEBUG] PR isClosed false
[INFO] ✏️ Executing UpdateTitleUseCase.
[DEBUG] Issue title updated to: [#344] 🐛 - Throttle workflow queue polling and increase workflow timeouts
[INFO] 👤 Executing AssignMemberToIssueUseCase.
[DEBUG] #345 needs 1 assignees.
[DEBUG] Assigned pull request creator @​efraespada to #345.
[INFO] 👀 Executing AssignReviewersToIssueUseCase.
[DEBUG] #345 needs 1 reviewers.
[DEBUG] #345 needs 1 more reviewers.
[INFO] 🔗 Executing LinkPullRequestProjectUseCase.
[DEBUG] Linked PR_kwDONSeBW88AAAABBwUmmA with id PVTI_lADOA8wRO84Ajh7Bzg46fA8 to project PVT_kwDOA8wRO84Ajh7B
[DEBUG] Target field ID: PVTSSF_lADOA8wRO84Ajh7Bzgb4L2E
[DEBUG] Target option ID: 47fc9ee4
[INFO] 🔗 Executing LinkPullRequestIssueUseCase.
[DEBUG] Changed base branch to master
[DEBUG] Updated PR #345 description with: &lt;!--
Thank you for opening a Pull Request 🚀
Please fill out this template as completely as possible.
Clear PRs = faster reviews = better software.
--&gt;

# 📌 Summary
&lt;!--
Briefly explain what this PR does and why it is needed.
Focus on the intent, not the implementation details.
--&gt;

---

## 🎯 Related Issues / Tickets
&lt;!--
Link any related issues, tickets, or discussions.
Use GitHub keywords to auto-close issues when merged.
--&gt;
- Closes #
- Related to #

---

## 🧩 Scope of Changes
&lt;!--
Describe the scope and nature of the changes at a high level.
--&gt;
- Added:
- Updated:
- Removed:
- Refactored:

---

## 🛠️ Technical Details
&lt;!--
Explain important technical decisions, trade-offs, or architecture changes.
Include anything that may not be obvious from reading the code.
--&gt;

---

## 🔍 How to Test
&lt;!--
Provide clear, step-by-step instructions to test this change.
Assume the reviewer is unfamiliar with the context.
--&gt;
1.
2.
3.

---

## 🧪 Test Coverage
&lt;!--
Indicate what types of tests were added or updated.
--&gt;
- [ ] Unit tests
- [ ] Integration tests
- [ ] End-to-end (E2E) tests
- [ ] Manual testing only (explain why)

---

## 📸 Screenshots / Recordings (UI changes only)
&lt;!--
Include before/after screenshots or short videos if this PR affects the UI.
--&gt;

---

## ⚠️ Breaking Changes
&lt;!--
List any breaking changes and required migration steps.
If none, state "None".
--&gt;
- None

---

## 🚀 Deployment Notes
&lt;!--
Anything required or important during deployment:
- environment variables
- database migrations
- feature flags
- deployment order
--&gt;
- [ ] Requires database migration
- [ ] Requires environment variable changes
- [ ] Requires feature flag toggle
- [ ] No special deployment steps

Details:

---

## 🔒 Security Considerations
&lt;!--
Mention any security-related implications or validations.
--&gt;
- [ ] No security impact
- [ ] Input validation changes
- [ ] Authentication / authorization changes
- [ ] Sensitive data handling changes

---

## 📈 Performance Impact
&lt;!--
Describe any performance considerations, improvements, or regressions.
--&gt;
- [ ] No performance impact
- [ ] Improves performance
- [ ] Potential performance regression (explain)

---

## 📝 Notes for Reviewers
&lt;!--
Highlight areas that need special attention or context for reviewers.
--&gt;

---

## ✅ Checklist
&lt;!--
Confirm all items before requesting review.
--&gt;
- [ ] I have self-reviewed my code
- [ ] Code follows project standards and conventions
- [ ] Tests have been added or updated
- [ ] Documentation has been updated (if applicable)
- [ ] No new warnings or lint errors
- [ ] Changes are backward compatible or breaking changes are documented

---

## 📚 Additional Context
&lt;!--
Any extra information that may help reviewers understand this PR.
Links, references, or follow-up work.
--&gt;


Resolves #344
[DEBUG] Changed base branch to develop
[DEBUG] Updated PR #345 description with: &lt;!--
Thank you for opening a Pull Request 🚀
Please fill out this template as completely as possible.
Clear PRs = faster reviews = better software.
--&gt;

# 📌 Summary
&lt;!--
Briefly explain what this PR does and why it is needed.
Focus on the intent, not the implementation details.
--&gt;

---

## 🎯 Related Issues / Tickets
&lt;!--
Link any related issues, tickets, or discussions.
Use GitHub keywords to auto-close issues when merged.
--&gt;
- Closes #
- Related to #

---

## 🧩 Scope of Changes
&lt;!--
Describe the scope and nature of the changes at a high level.
--&gt;
- Added:
- Updated:
- Removed:
- Refactored:

---

## 🛠️ Technical Details
&lt;!--
Explain important technical decisions, trade-offs, or architecture changes.
Include anything that may not be obvious from reading the code.
--&gt;

---

## 🔍 How to Test
&lt;!--
Provide clear, step-by-step instructions to test this change.
Assume the reviewer is unfamiliar with the context.
--&gt;
1.
2.
3.

---

## 🧪 Test Coverage
&lt;!--
Indicate what types of tests were added or updated.
--&gt;
- [ ] Unit tests
- [ ] Integration tests
- [ ] End-to-end (E2E) tests
- [ ] Manual testing only (explain why)

---

## 📸 Screenshots / Recordings (UI changes only)
&lt;!--
Include before/after screenshots or short videos if this PR affects the UI.
--&gt;

---

## ⚠️ Breaking Changes
&lt;!--
List any breaking changes and required migration steps.
If none, state "None".
--&gt;
- None

---

## 🚀 Deployment Notes
&lt;!--
Anything required or important during deployment:
- environment variables
- database migrations
- feature flags
- deployment order
--&gt;
- [ ] Requires database migration
- [ ] Requires environment variable changes
- [ ] Requires feature flag toggle
- [ ] No special deployment steps

Details:

---

## 🔒 Security Considerations
&lt;!--
Mention any security-related implications or validations.
--&gt;
- [ ] No security impact
- [ ] Input validation changes
- [ ] Authentication / authorization changes
- [ ] Sensitive data handling changes

---

## 📈 Performance Impact
&lt;!--
Describe any performance considerations, improvements, or regressions.
--&gt;
- [ ] No performance impact
- [ ] Improves performance
- [ ] Potential performance regression (explain)

---

## 📝 Notes for Reviewers
&lt;!--
Highlight areas that need special attention or context for reviewers.
--&gt;

---

## ✅ Checklist
&lt;!--
Confirm all items before requesting review.
--&gt;
- [ ] I have self-reviewed my code
- [ ] Code follows project standards and conventions
- [ ] Tests have been added or updated
- [ ] Documentation has been updated (if applicable)
- [ ] No new warnings or lint errors
- [ ] Changes are backward compatible or breaking changes are documented

---

## 📚 Additional Context
&lt;!--
Any extra information that may help reviewers understand this PR.
Links, references, or follow-up work.
--&gt;

[INFO] 🔄 Executing SyncSizeAndProgressLabelsFromIssueToPrUseCase.
[DEBUG] Synced size/progress labels from issue #344 to PR #345: size: XXL, 95%
[INFO] 📏 Executing CheckPriorityPullRequestSizeUseCase.
[DEBUG] Priority: priority: high
[DEBUG] Github Priority Label: P0
[DEBUG] Target field ID: PVTSSF_lADOA8wRO84Ajh7Bzgb4MPQ
[DEBUG] Target option ID: 4af6496f
[INFO] ✏️ Executing UpdatePullRequestDescriptionUseCase (AI PR description).
[DEBUG] PR description will be generated from workspace diff: base "develop", head "bugfix/344-throttle-workflow-queue-polling-and-increase-workflow-timeouts" (configured agent will run git diff).
[DEBUG] UpdatePullRequestDescription: prompt length=11475, issue description length=7250. Calling configured agent.
[DEBUG] UpdatePullRequestDescription: agent response received. Description length=5413.
[DEBUG] Updated PR #345 description with: # 📌 Summary

Improves the repository-wide Copilot workflow queue to withstand bursts without exhausting GitHub API limits, while preserving strict sequential execution and allowing long queues to drain reliably.

---

## 🎯 Related Issues / Tickets

- Closes #344

---

## 🧩 Scope of Changes

- Added:
  - Centralized 90-minute queue policy with exponential backoff, capped delays, and jitter.
  - Rate-limit-aware retry handling for GitHub API responses.
  - Deterministic queue, retry, and workflow-contract regression tests.

- Updated:
  - Workflow polling to use one paginated repository traversal per poll with local filtering.
  - Handling of HTTP 429, rate-limited 403, transient HTTP, and network failures.
  - Queue-bearing workflow timeouts from 30 to 120 minutes in active and setup workflows.
  - Documentation, package version, and generated action/CLI bundles.

- Removed:
  - Fixed two-second polling and repeated queries for each active workflow status.

- Refactored:
  - Queue ports and composition to inject clock, randomness, delays, and polling observers.
  - Workflow contract validation around a manifest of the seven queue workflows.

---

## 🛠️ Technical Details

- Each poll requests up to 100 workflow runs through a single paginated traversal, then filters locally by the seven Copilot/Task workflow names, active statuses, and lower run IDs.
- Pagination continues through all pages so older matching runs are not missed.
- Polling uses 5s, 10s, 20s, 40s, and 60s maximum delays with bounded ±20% jitter.
- Rate-limit responses honor `Retry-After` and `x-ratelimit-reset`, with bounded fallback backoff when headers are unavailable.
- Provider failures and queue deadlines fail closed; they cannot be interpreted as an empty queue.
- Provider diagnostics are replaced with a canonical queue failure message to avoid exposing response bodies, URLs, headers, or credentials.
- Queue-bearing jobs use 120-minute timeouts, providing headroom beyond the 90-minute queue deadline.
- Mutation workflows intentionally retain no GitHub Actions `concurrency` block, preventing cancellation or replacement of queued runs.

---

## 🔍 How to Test

1. Run the deterministic queue and retry tests:

   bash
   corepack pnpm@​10.12.4 exec jest \
     src/application/usecases/workflow/__tests__/wait_for_previous_workflow_runs_use_case.test.ts \
     src/data/repository/workflow/__tests__/active_previous_workflow_runs_repository.test.ts \
     src/data/repository/workflow/__tests__/workflow_runs_retry_policy.test.ts \
     src/tooling/__tests__/validate_workflow_contract.test.ts
   

2. Run the project quality gates:

   bash
   corepack pnpm@​10.12.4 run typecheck
   corepack pnpm@​10.12.4 run lint
   corepack pnpm@​10.12.4 run test:coverage
   corepack pnpm@​10.12.4 run build
   corepack pnpm@​10.12.4 run validate:workflows
   corepack pnpm@​10.12.4 run validate:agent-docs
   corepack pnpm@​10.12.4 run validate:docs-page
   git diff --check
   

3. Confirm the workflow contract validator accepts both `.github/workflows/` and `setup/workflows/`, including the 120-minute queue-job timeouts and absence of mutation-workflow concurrency settings.

---

## 🧪 Test Coverage

- [x] Unit tests
- [ ] Integration tests
- [ ] End-to-end (E2E) tests
- [ ] Manual testing only

---

## 📸 Screenshots / Recordings (UI changes only)

Not applicable — no UI changes.

---

## ⚠️ Breaking Changes

- None to action inputs or workflow configuration.
- Internal queue polling behavior now uses a bounded 90-minute deadline and fail-closed provider errors.

---

## 🚀 Deployment Notes

- [ ] Requires database migration
- [ ] Requires environment variable changes
- [ ] Requires feature flag toggle
- [x] No special deployment steps

Details: Generated action and CLI bundles are included and should be released through the normal action release process. Setup workflow templates are synchronized with the active workflows.

---

## 🔒 Security Considerations

- [ ] No security impact
- [ ] Input validation changes
- [ ] Authentication / authorization changes
- [x] Sensitive data handling changes

Provider error details are no longer interpolated into logs or externally visible failures. Diagnostic logging is limited to retry reason, attempt, delay, and reset metadata.

---

#
[untrusted content truncated]

🚀 Happy coding!

Made with ❤️ by vypdev/copilot

@efraespada
efraespada merged commit 6289499 into develop Sep 1, 2026
14 of 21 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in vypdev Sep 1, 2026
@vypbot

vypbot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

🐛 Bugfix Actions

  1. 🐛🧑‍💻 - Throttle workflow queue polling and increase workflow timeouts #344 was automatically closed after merging this pull request.

  2. Lifecycle state synchronized to verified.

image

Debug log
[DEBUG] Setup done. Issue number: 344, isSingleAction: false, isIssue: false, isPullRequest: true, isPush: false
[INFO] Running PullRequestUseCase for PR #345.
[INFO] 🔀 Executing PullRequestUseCase.
[DEBUG] PR action closed
[DEBUG] PR isOpened false
[DEBUG] PR isMerged true
[DEBUG] PR isClosed true
[INFO] ✅ Executing CloseIssueAfterMergingUseCase.
[DEBUG] Issue #344 state: open
[DEBUG] Issue #344 has been closed.
[INFO] Issue #344 closed after merging PR #345.
[DEBUG] Comment added to Issue 344.
[INFO] Main run finished. Results: 1, total steps: 1.
[INFO] Publishing result: 2 result(s), 2 step(s), 0 error(s).
[INFO] 📄 Executing PublishResultUseCase.

🚀 Happy coding!

Made with ❤️ by vypdev/copilot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

🐛🧑‍💻 - Throttle workflow queue polling and increase workflow timeouts

3 participants