Skip to content

fix(tunnel): stop leaking a cloudflared process per restart on Windows - #206

Open
luthermonson wants to merge 1 commit into
mainfrom
fix/windows-cloudflared-orphans
Open

luthermonson wants to merge 1 commit into
mainfrom
fix/windows-cloudflared-orphans

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Linux binds cloudflared to ephemerd at the kernel level with Pdeathsig, which — as its own comment says — "survives SIGKILL of ephemerd, panics, or any exit path that skips deferred cleanup."

Windows had nothing. cloudflared_other.go claimed "the graceful Close() path is the sole shutdown mechanism there", but that path calls cmd.Process.Signal(syscall.SIGTERM) — and Go’s os.Process.Signal rejects every signal except os.Kill on Windows. The error was logged at Debug and dropped.

So stopping the service, which is what every mayfly apply and node upgrade does, killed ephemerd without running close(), and the tunnel client just kept running.

Measured on the live node (2026-09-22)

live ephemerd pid : 6908
children of live  : 1
orphaned          : 25
orphan parents still alive: 0
orphan RAM MB     : 868

Oldest orphan dated 2026-09-09. All 25 share one tunnel config and held 96 established connections to Cloudflare’s edge, against 4 for the live daemon.

Cloudflare load-balances across every connection registered for a tunnel — so the large majority of inbound webhook deliveries were being routed to clients whose ephemerd was gone. That is a strong candidate for the "webhook listener wedged for hours after an upgrade" behaviour.

Fix

The Windows counterpart to Pdeathsig: a Job Object marked KILL_ON_JOB_CLOSE. The kernel terminates the child when the last handle to the job closes, which happens when ephemerd exits by any means — service stop, crash, or kill.

Closing the job is itself a guaranteed kill, so close() releases it after the graceful attempts rather than instead of them: a clean cloudflared shutdown closes its edge connections properly, an abrupt one leaves Cloudflare to time them out.

A bind failure warns rather than failing Listen — a tunnel that works but might leak beats no tunnel at all — but it is a warning, not a debug line, because it is the difference between one stray process and one per restart forever.

Testing

The tests run for real on Windows: the child dies when the job is released, and an unstarted process is rejected rather than panicking on a nil cmd.Process.

Revert-verified — reducing bindChildLifetime to the old no-op fails with bindChildLifetime returned a nil closer on windows; the child is unbound.

Builds and go vet clean for linux, windows and darwin. The 25 existing orphans still need reaping by hand; this stops new ones.

Linux binds cloudflared to ephemerd at the kernel level with Pdeathsig,
which — as its own comment says — "survives SIGKILL of ephemerd, panics,
or any exit path that skips deferred cleanup".

Windows had nothing. cloudflared_other.go claimed "the graceful Close()
path is the sole shutdown mechanism there", but that path calls
cmd.Process.Signal(syscall.SIGTERM), and Go's os.Process.Signal rejects
every signal except os.Kill on Windows. The error was logged at Debug
and dropped. Stopping the service — which is what every `mayfly apply`
and `node upgrade` does — killed ephemerd without running close(), and
the tunnel client simply kept running.

Measured on mfl-win-amd64-102 on 2026-09-22:

  live ephemerd pid : 6908
  children of live  : 1
  orphaned          : 25
  orphan parents still alive: 0
  orphan RAM MB     : 868

oldest orphan dated 2026-09-09. All 25 share one tunnel config and held
96 established connections to Cloudflare's edge against 4 for the live
daemon. Cloudflare load-balances across every connection registered for
a tunnel, so the large majority of inbound webhook deliveries were being
routed to clients whose ephemerd was gone. That is a strong candidate
for the "webhook listener wedged for hours" behaviour seen after
upgrades.

The fix is the Windows counterpart to Pdeathsig: a Job Object marked
KILL_ON_JOB_CLOSE. The kernel terminates the child when the last handle
to the job closes, which happens when ephemerd exits by any means —
service stop, crash, or kill.

Closing the job is also a guaranteed kill, so close() releases it AFTER
the graceful attempts rather than instead of them: a clean cloudflared
shutdown closes its edge connections properly, an abrupt one leaves
Cloudflare to time them out.

A bind failure warns instead of failing Listen — a tunnel that works but
might leak beats no tunnel — but it is a warning, not a debug line,
because it is the difference between one stray process and one per
restart forever.

Tests run for real on Windows: the child dies when the job is released,
and an unstarted process is rejected rather than panicking on a nil
cmd.Process. Revert-verified — reducing bindChildLifetime to the old
no-op fails with "the child is unbound". Builds and vets clean for
linux, windows and darwin.
@ephpm

ephpm Bot commented Sep 22, 2026

Copy link
Copy Markdown

ePHPm Preview — deployed (health check pending)

URL https://ephpm-ephemerd-pr-206.preview.ephpm.dev
Framework PHP
PHP 8.5
Deployed in 61.7s

Preview updates automatically on each push to this PR.

@ephpm
ephpm Bot temporarily deployed to preview-pr-206 September 22, 2026 19:56 Inactive
@ephpm
ephpm Bot temporarily deployed to preview-pr-206 September 22, 2026 19:56 Inactive

This branch was successfully deployed

1 active deployment
preview-pr-206 8f70c5fd Deployed Sep 22, 2026 by ephpm[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant