headscale-verify-proxy

headscale-verify-proxy is a small Go service that sits between a Tailscale DERP server and one or more Headscale /verify endpoints. It caches only successful node verification, tries independent Headscale authorities in a deterministic order, retains the authority that allowed a node, and emits a stable final-denial record suitable for Fail2Ban.

The proxy does not modify firewall state and has no database, Redis, Prometheus, or web-framework dependency.

Architecture

Tailscale client
      |
      v
DERP server
      |
      | POST /verify
      v
headscale-verify-proxy
      |
      +--> Headscale A /verify
      +--> Headscale B /verify
      +--> Headscale C /verify

The request path is implemented by the HTTP handler, cache-aware service, and sequential upstream verifier. Configuration is loaded once at startup by the YAML parser.

Multi-Upstream Semantics

Upstreams are attempted sequentially in their YAML order. The decision rule is exact:

Observed results	Final result	Cached	VERIFY_DENY
Any upstream returns HTTP 200 with Allow=true	HTTP 200, Allow=true	Yes, with that upstream name	No
Every upstream returns HTTP 200 with Allow=false	HTTP 200, Allow=false	No	Exactly once
No upstream allows and at least one attempt has an infrastructure failure	HTTP 502	No	No

An allow immediately stops the sequence. A denial from one authority never overrides a later allow. Timeouts, DNS failures, connection failures, TLS failures, redirects, non-200 responses, oversized responses, malformed JSON, and responses without a Boolean Allow field are infrastructure failures.

The configured timeout applies to each upstream attempt. With $N$ upstreams and timeout $T$, worst-case upstream time is approximately $N \times T$, plus small transport overhead.

Caching

Only Allow=true results are cached. Denials and infrastructure failures are never cached.

The cache key is NodePublic, not Source and not the upstream name. A cache entry contains:

• the configured name of the upstream that allowed the node;
• its individual expiration time.

This lets a node roam between source IP addresses without repeating verification while its positive entry is valid. Cache hits skip every upstream and retain the original upstream in logs and counters.

Expiration is checked on every lookup, so correctness does not depend on periodic cleanup. cache.cleanup_interval only removes stale map entries proactively; 0 disables that maintenance task.

Revocation delay

Positive caching creates a deliberate revocation window. With positive_ttl: 1h, a node that was allowed and then removed or revoked in Headscale may continue receiving cached approval for approximately one hour. Choose the TTL according to the acceptable revocation delay. Send SIGHUP when immediate local cache invalidation is required.

Build and Run

Go 1.24.5 or a compatible Go 1.24 toolchain is required.

mkdir -p bin
go build -o bin/headscale-verify-proxy ./cmd/headscale-verify-proxy
cp config.example.yaml config.yaml
./bin/headscale-verify-proxy --config=./config.yaml

--config is required. Runtime settings do not come from environment variables or additional command-line flags.

Configuration

The configuration is one YAML document. Unknown YAML fields, duplicate keys, malformed values, multiple documents, and files larger than 1 MiB are rejected at startup. Duration values use Go duration syntax such as 3s, 10m, and 1h.

server:
  listen: ":8080"
  read_header_timeout: 5s
  idle_timeout: 60s
  shutdown_timeout: 10s

cache:
  positive_ttl: 1h
  cleanup_interval: 10m

upstream:
  timeout: 3s

upstreams:
  - name: headscale-primary
    url: https://headscale-primary.example.com/verify

  - name: headscale-secondary
    url: https://headscale-secondary.example.com/verify

logging:
  level: info

The complete commented example is config.example.yaml.

Field	Default	Validation and behavior
server.listen	:8080	Numeric TCP port from 1 through 65535; bind it only where DERP can reach it
server.read_header_timeout	5s	Must be positive
server.idle_timeout	60s	Must be positive
server.shutdown_timeout	10s	Must be positive; bounds graceful shutdown
cache.positive_ttl	1h	Must be positive; controls revocation delay
cache.cleanup_interval	10m	Must not be negative; 0 disables periodic cleanup
upstream.timeout	3s	Must be positive and applies independently to each attempt
upstreams	None	At least one ordered entry is required
upstreams[].name	None	Non-empty, unique, and without surrounding whitespace
upstreams[].url	None	Absolute http or https URL with a host and no fragment
logging.level	info	debug, info, warn, or error, case-insensitive

Upstream order is preserved exactly. The configured name, rather than the URL, is used in cache metadata, logs, counters, and diagnostics. Normal request logs do not expose full upstream URLs.

HTTP API

POST /verify

The incoming body is limited to 16 KiB. It does not need a Content-Type header. Unknown JSON fields are accepted for DERP compatibility, while malformed JSON and trailing JSON values are rejected.

{
  "NodePublic": "nodekey:example",
  "Source": "192.0.2.10"
}

NodePublic is required, must start with nodekey:, and is limited to 256 bytes. If Source is present, it must parse as an IPv4 or IPv6 address. Source is request metadata supplied by DERP; the proxy does not replace it with the HTTP peer address.

Successful decisions return:

{"Allow":true}

or:

{"Allow":false}

Input errors return HTTP 400 or 413, unsupported methods return HTTP 405, and an indeterminate upstream result returns HTTP 502 rather than a false denial.

GET /healthz

Returns HTTP 200 with ok. This is process health only and never contacts Headscale.

No cache-clear HTTP endpoint is exposed.

Singleflight and Request Cancellation

Concurrent cache misses are coalesced by NodePublic with golang.org/x/sync/singleflight. The cache is checked again inside the shared function before any upstream request.

Shared verification uses an application-lifetime context rather than the initiating HTTP request context. A caller that disconnects stops waiting, but does not cancel work needed by other callers. The bounded per-upstream timeout prevents abandoned shared work from running indefinitely. SIGINT or SIGTERM lets active requests finish up to server.shutdown_timeout, then cancels application work if forced shutdown is necessary.

Different node keys use different singleflight keys and can verify independently.

Logging and Fail2Ban

Logs use log/slog text output. Request and cache detail is emitted at DEBUG, final decisions and individual upstream decisions at INFO, recoverable upstream failures at WARN, and indeterminate or internal verification failures at ERROR.

Every upstream event uses the configured upstream name. Cache-store, cache-hit, and cache-expiration records also carry the successful upstream name.

Only the final case in which every configured upstream explicitly returned Allow=false emits the mandatory audit record. Its stable field sequence is:

msg=VERIFY_DENY source=192.0.2.10 node=nodekey:abc reason=all_upstreams_denied upstreams=headscale-a,headscale-b event=verify_denied

The event is emitted exactly once per final denied request. Infrastructure failures, invalid input, mixed deny/error results, and deny-then-allow results never produce it. Mandatory VERIFY_DENY and SIGHUP cache_clear audit records remain enabled even when logging.level is error.

A minimal Fail2Ban filter expression for the default text handler is:

failregex = ^.*msg=VERIFY_DENY source=<HOST> node=\S+ reason=all_upstreams_denied upstreams=\S+ event=verify_denied$
ignoreregex =

The proxy only emits logs. It never calls Fail2Ban, manipulates iptables or nftables, requests CAP_NET_ADMIN, or accesses the Docker socket. Monitoring DERP's own rejection logs can be safer because DERP knows the real client connection address, whereas this proxy must trust the Source metadata in the request body.

Metrics

The process maintains concurrency-safe internal counters for requests, cache hits and misses, current cache entries, final allows, denials, errors, and per-upstream requests, allows, denials, and errors. Upstream maps are keyed by configured name.

There is intentionally no /metrics endpoint. This keeps the service and attack surface small; structured logs remain available to external collectors.

Signals

Signal	Behavior
SIGHUP	Clears all positive cache entries, logs the number removed, and keeps running
SIGINT	Starts bounded graceful shutdown
SIGTERM	Starts bounded graceful shutdown

With the example Compose deployment, clear the cache with:

docker compose -f compose.example.yaml kill --signal=SIGHUP headscale-verify-proxy

Restarting the process also clears the in-memory cache.

Security

The proxy must be placed behind DERP and should not be publicly accessible. A direct caller can forge Source, because it is metadata and not independently tied to the TCP client by this service.

Recommended controls:

• bind or route the proxy only on a private host or container network;
• do not publish port 8080 to the public Internet;
• restrict which workloads can join the private network;
• mount configuration read-only;
• use HTTPS upstream URLs across untrusted networks;
• keep normal upstream URL contents out of per-request logs;
• run the container as non-root with all Linux capabilities dropped.

The upstream client uses the system trust store, does not disable TLS verification, does not follow redirects, reuses connections, limits response bodies to 16 KiB, and applies a timeout to every attempt.

Docker

Build the non-root, multi-stage image:

docker build -t headscale-verify-proxy:local .

The runtime image contains the statically linked binary and CA certificates but no shell. It runs as the distroless nonroot user. Mount a validated config at /etc/headscale-verify-proxy/config.yaml; the file must be readable by container UID 65532.

The provided compose.example.yaml uses three networks:

• derp-public for DERP's client-facing ports;
• internal verify-internal for DERP-to-proxy traffic;
• upstream-access for proxy-to-Headscale traffic.

The proxy has no published port. The Compose example requires DERPER_IMAGE because DERP container packaging varies; choose and pin an audited image. It assumes that image accepts DERP_VERIFY_CLIENTS and DERP_VERIFY_CLIENT_URL. If it uses command-line flags instead, configure the equivalent verification URL as http://headscale-verify-proxy:8080/verify.

cp config.example.yaml config.yaml
export DERPER_IMAGE='your-registry.example/derper:audited-version'
export DERP_DOMAIN='derp.example.com'
docker compose -f compose.example.yaml up --build

Do not add a public ports mapping to headscale-verify-proxy.

Testing

Run the normal tests, race detector, and static checks:

go test ./...
go test -race ./...
go vet ./...

Tests cover YAML validation, positive-only expiration, upstream provenance, exact multi-authority semantics, malformed and oversized input, HTTP upstream failures, stable denial logs, same-node singleflight, different-node independence, caller cancellation, SIGHUP clearing, and graceful shutdown.