Skip to content

fix(runtime): stop the reaper sweeping LIVE jobs' dind bind staging dirs - #201

Merged
luthermonson merged 1 commit into
mainfrom
fix/reaper-sweeps-live-job-binds
Sep 18, 2026
Merged

luthermonson merged 1 commit into
mainfrom
fix/reaper-sweeps-live-job-binds

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Regression introduced by #187 (shipped in v0.2.8, present in every release since including v0.2.12). Actively breaking jobs on the Linux amd64 nodes.

Reported symptom

docker: Error response from daemon: bind mount /home/runner/_work/ephpm/ephpm -> /w rejected:
  creating bind staging mountpoint /var/lib/ephemerd/dind-binds/ephemerd-github-ephpm-sure_hopper/2:
  mkdir /var/lib/ephemerd/dind-binds/... : no such file or directory
##[error]Process completed with exit code 127

Cause

#187's periodic dead-container reaper calls the whole-tree dind.SweepStagedBinds once per pass (pkg/runtime/runtime.go). That function is documented STARTUP ONLY, and its own comment states the failure mode precisely:

"it does not know which jobs are live, and unmounting a running job's staged bind would not break that job's already-running containers but would break any container it starts next."

Sequence:

  1. A job is running with staged binds under <data>/dind-binds/<jobID>/
  2. The reaper tick finds some unrelated dead container to reap
  3. It sweeps the entire dind-binds tree first — deleting live jobs' directories too
  4. The live job's already-running containers are unaffected (their mounts are in their namespaces), so nothing fails loudly
  5. Its next docker run -v cannot create its mountpoint: ensureDirLocked has latched m.ready and will not recreate the parent, and os.Mkdir is non-recursive → ENOENT

The index /2 in the error is the tell: binds 0 and 1 staged fine, then a tick landed.

Why ephpm surfaced it: its release workflow runs several docker run -v "$PWD":/w steps per job (since the node24 de-containerization), so it has the most exposure.

Fix

Sweep only the staging dir of the container actually being reaped, via a new SweepStagedBindsForJob. The reaping motive is unchanged — a leaked staged mount pins that container's rootfs so WithSnapshotCleanup fails with "device or resource busy" — and scoping it preserves that without touching anyone else's directory.

The startup-only whole-tree sweep in CleanOrphans is left exactly as-is; that one is correct, because nothing is live at startup.

Tests

TestSweepStagedBindsForJob_SparesOtherJobs creates a dead and a live job staging dir, sweeps the dead one, and asserts the live one survives with its mountpoint subdirectory intact — the precise property that was violated. Plus a tolerates-missing case (a job with no -v anywhere, and empty-arg guards). Deliberately mount-free so it runs in CI without privileges.

Verification

GOOS=linux go vet ./pkg/dind/... ./pkg/runtime/... clean; go test ./pkg/dind/... ./pkg/runtime/... green on the dev box.

Caveat: the dev box is Windows, so the new Linux tests were compile-verified (GOOS=linux go vet) but not executed locally — CI's "Lint, Unit, E2E & Build (Linux)" is the gate. The existing bind-staging tests need real mount privileges and can only run there.

Regression introduced by #187 (v0.2.8). Its periodic dead-container
reaper called dind.SweepStagedBinds once per pass. That function is
documented STARTUP ONLY, and its own comment says why:

  "it does not know which jobs are live, and unmounting a running job's
   staged bind would not break that job's already-running containers but
   would break any container it starts next."

Which is exactly what happened on the fleet's Linux amd64 nodes:

  docker: Error response from daemon: bind mount /home/runner/_work/... -> /w
    rejected: creating bind staging mountpoint
    /var/lib/ephemerd/dind-binds/ephemerd-github-ephpm-sure_hopper/2:
    mkdir ...: no such file or directory

A reaper tick reaps some unrelated dead container, sweeps the whole
<data>/dind-binds tree, and deletes the staging directory of every LIVE
job with it. The running job's existing containers keep working (their
mounts are already in their namespaces), so nothing fails loudly -- but
its NEXT `docker run -v` cannot create its mountpoint, because
ensureDirLocked has latched m.ready and will not recreate the parent.
Hence the index /2: binds 0 and 1 staged fine before the tick landed.

ephpm's release workflow is the heaviest user of `docker run -v "$PWD":/w"
(since the node24 de-containerization), which is why it surfaced there.

Fix: sweep only the staging dir of the container actually being reaped,
via a new SweepStagedBindsForJob. The reaping motive is unchanged -- a
leaked staged mount pins that container's rootfs so its snapshot cannot
be deleted -- and scoping it keeps that without touching anyone else's.
The startup-only whole-tree sweep in CleanOrphans is left alone; that one
is correct, because nothing is live yet.
@ephpm

ephpm Bot commented Sep 14, 2026

Copy link
Copy Markdown

ePHPm Preview — removed

Preview deployment has been torn down.

@ephpm
ephpm Bot temporarily deployed to preview-pr-201 September 14, 2026 01:18 Inactive
@ephpm
ephpm Bot temporarily deployed to preview-pr-201 September 14, 2026 01:18 Inactive
@luthermonson
luthermonson merged commit 22f0308 into main Sep 18, 2026
4 checks passed

This branch was successfully deployed

1 active deployment
preview-pr-201 7c49348a Deployed Sep 14, 2026 by ephpm[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant