Skip to content

fix(sandbox): wait for refused connections within the job deadline - #32

Merged
rschlaefli merged 18 commits into
mainfrom
rs/storage-reliability-investigation
Sep 20, 2026
Merged

rschlaefli merged 18 commits into
mainfrom
rs/storage-reliability-investigation

Conversation

@rschlaefli

@rschlaefli rschlaefli commented Sep 20, 2026

Copy link
Copy Markdown
Member

Cold execution can fail with ECONNREFUSED while a sandbox scales from zero. The HTTP backend now waits only after a refused connection, within the existing absolute job deadline, and sends the same signed request when the endpoint becomes available.

Retries stop on cancellation or deadline expiry. HTTP failures, disconnects, timeouts and other ambiguous failures propagate immediately. Redirects are disabled so an accepted POST cannot be replayed through a failing redirect target. No timeout increase, new setting or dependency is introduced.

Real-transport tests cover repeated refusals followed by exactly one accepted request, unchanged signed bytes, cancellation, expired/invalid/fallback deadlines, hanging accepted requests, HTTP errors, disconnects and redirects. This branch also persists the storage recovery and upstream v1.4.0 integration plan and its rollout receipts. Upstream history and the policy of no fork release tags are preserved.

Validation on current source: service build passed with no new type warnings; scoped ESLint has zero errors and one existing warning; the full service suite passed 1,115 tests with 12 skips and zero failures in Bun 1.3.14 (3,374 assertions across 97 files). All 48 focused HTTP/worker-error/cancellation tests also pass. Independent correctness and integrated final review found no required source fixes at 92d6940. A retry-telemetry suggestion is deferred as nonblocking. Current CI remains pending.

STG storage recovery passed its fifteen-minute observation. The v1.4.0 chart/image reconciliation completed and Argo is Synced/Healthy, but that deployed pre-fix image failed the cold execution probe. This change has not passed live STG execution acceptance. PRD promotion remains blocked on corrected STG cold/warm execution, artifact reuse/download, timeout/cancellation and consumer verification, plus PRD storage recovery and alert-delivery prerequisites.

@rschlaefli
rschlaefli marked this pull request as ready for review September 20, 2026 14:18
@rschlaefli
rschlaefli merged commit f6ec42c into main Sep 20, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant