fix(tunnel): stop leaking a cloudflared process per restart on Windows - #206
Open
luthermonson wants to merge 1 commit into
Open
luthermonson wants to merge 1 commit into
luthermonson wants to merge 1 commit into
Conversation
Linux binds cloudflared to ephemerd at the kernel level with Pdeathsig, which — as its own comment says — "survives SIGKILL of ephemerd, panics, or any exit path that skips deferred cleanup". Windows had nothing. cloudflared_other.go claimed "the graceful Close() path is the sole shutdown mechanism there", but that path calls cmd.Process.Signal(syscall.SIGTERM), and Go's os.Process.Signal rejects every signal except os.Kill on Windows. The error was logged at Debug and dropped. Stopping the service — which is what every `mayfly apply` and `node upgrade` does — killed ephemerd without running close(), and the tunnel client simply kept running. Measured on mfl-win-amd64-102 on 2026-09-22: live ephemerd pid : 6908 children of live : 1 orphaned : 25 orphan parents still alive: 0 orphan RAM MB : 868 oldest orphan dated 2026-09-09. All 25 share one tunnel config and held 96 established connections to Cloudflare's edge against 4 for the live daemon. Cloudflare load-balances across every connection registered for a tunnel, so the large majority of inbound webhook deliveries were being routed to clients whose ephemerd was gone. That is a strong candidate for the "webhook listener wedged for hours" behaviour seen after upgrades. The fix is the Windows counterpart to Pdeathsig: a Job Object marked KILL_ON_JOB_CLOSE. The kernel terminates the child when the last handle to the job closes, which happens when ephemerd exits by any means — service stop, crash, or kill. Closing the job is also a guaranteed kill, so close() releases it AFTER the graceful attempts rather than instead of them: a clean cloudflared shutdown closes its edge connections properly, an abrupt one leaves Cloudflare to time them out. A bind failure warns instead of failing Listen — a tunnel that works but might leak beats no tunnel — but it is a warning, not a debug line, because it is the difference between one stray process and one per restart forever. Tests run for real on Windows: the child dies when the job is released, and an unstarted process is rejected rather than panicking on a nil cmd.Process. Revert-verified — reducing bindChildLifetime to the old no-op fails with "the child is unbound". Builds and vets clean for linux, windows and darwin.
|
ePHPm Preview — deployed (health check pending)
Preview updates automatically on each push to this PR. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linux binds cloudflared to ephemerd at the kernel level with
Pdeathsig, which — as its own comment says — "survives SIGKILL of ephemerd, panics, or any exit path that skips deferred cleanup."Windows had nothing.
cloudflared_other.goclaimed "the graceful Close() path is the sole shutdown mechanism there", but that path callscmd.Process.Signal(syscall.SIGTERM)— and Go’sos.Process.Signalrejects every signal exceptos.Killon Windows. The error was logged atDebugand dropped.So stopping the service, which is what every
mayfly applyandnode upgradedoes, killed ephemerd without runningclose(), and the tunnel client just kept running.Measured on the live node (2026-09-22)
Oldest orphan dated 2026-09-09. All 25 share one tunnel config and held 96 established connections to Cloudflare’s edge, against 4 for the live daemon.
Cloudflare load-balances across every connection registered for a tunnel — so the large majority of inbound webhook deliveries were being routed to clients whose ephemerd was gone. That is a strong candidate for the "webhook listener wedged for hours after an upgrade" behaviour.
Fix
The Windows counterpart to
Pdeathsig: a Job Object markedKILL_ON_JOB_CLOSE. The kernel terminates the child when the last handle to the job closes, which happens when ephemerd exits by any means — service stop, crash, or kill.Closing the job is itself a guaranteed kill, so
close()releases it after the graceful attempts rather than instead of them: a clean cloudflared shutdown closes its edge connections properly, an abrupt one leaves Cloudflare to time them out.A bind failure warns rather than failing
Listen— a tunnel that works but might leak beats no tunnel at all — but it is a warning, not a debug line, because it is the difference between one stray process and one per restart forever.Testing
The tests run for real on Windows: the child dies when the job is released, and an unstarted process is rejected rather than panicking on a nil
cmd.Process.Revert-verified — reducing
bindChildLifetimeto the old no-op fails withbindChildLifetime returned a nil closer on windows; the child is unbound.Builds and
go vetclean for linux, windows and darwin. The 25 existing orphans still need reaping by hand; this stops new ones.