fix: level-triggered reconcile so previews converge across the cluster [DO NOT MERGE — review]

[luthermonson]

DO NOT MERGE — for review. Permanent fix for preview config/state drift across the 3-node preview cluster.

The bug (confirmed in current source)

Preview fan-out is edge-triggered on the switchboard:gen counter. The proximate short-circuit is in switchboard-api:

• switchboard-api/src/Cluster/DrainHandler.php:120-124 — if ($gen === $lastGen) { return … 'checked' => 0, 'queued' => 0 } returns before materialize(). Once a node advances last_gen to the current generation, every later drain no-ops and never re-walks switchboard:index.

gen is a separate gossip key from the content it guards (switchboard:preview:<label>, written by ClusterState::publish in ClusterState.php). A gen increment that is lost or not-yet-replicated (the incr is best-effort, ClusterState.php:117-119) leaves a node "current" at a generation whose teardown it never materialized — the torn-down preview stays served, its DB and <key>.toml override on disk, every health check green (switchboard#24, and the gen-vs-key-content propagation race). On top of that, switchboard:index is a read-modify-write with an acknowledged lost-update window (ClusterState.php:35-50).

Both are the same fault: an edge trigger that misses a state it never observed an event for. Live symptom: override .toml counts drifting between nodes (11/10/7) and orphaned overrides for torn-down previews lingering as 404s.

The fix

A level-triggered reconcile pass in the daemon (src/reconcile.rs), on its own interval, independent of gen, converging this node's on-disk previews to the KV desired state read directly:

• Prune — classify each preview directory under sites_dir from its own switchboard:preview:<label> key. That key is a plain set (only teardown gives it a TTL), so it is immune to both the gen propagation race and the index lost-update race. A genuine nil (retired) or intent:teardown → orphan → removed via the existing KEEP-guarded, exact-path teardown::teardown_preview (site dir, <key>.db*, <key>.toml, /tmp/ephpm-vhosts/<key>-<hash>, preserving *.single-db-bak). Pruning never trusts the index, so a live preview that transiently fell out of it is not false-positived into an orphan.
• Add — a label the index lists whose preview key says deploy but whose directory is absent → enqueue the exact job document (Queue::enqueue) so the normal claim → coalesce → validate → deploy path runs it.

Correctness no longer depends on gen; it stays as the PHP fast-path's optimization hint (kept as-is). Idempotent under concurrent drains — teardown tolerates already-absent artifacts; a materialized job gets a fresh per-node filename and is coalesced — so two nodes converge rather than fight.

Fail-safe posture

• A KV read failure aborts the pass (prunes nothing) — never act against an unreadable authority.
• Infra/non-preview vhosts protected by --reconcile-keep-sites plus the API's own site key.
• Pruning is opt-in (--reconcile-prune); until set, dry-run logs each orphan it would remove at WARN — which alone would have surfaced the original incident (found by hand-diffing three nodes).
• Per-cycle prune cap bounds blast radius.
• Whole feature off by default (--reconcile-interval-secs 0), so upgrade is a no-op.

switchboard-api companion change?

Not required. The daemon is the process that owns on-disk state and already reads KV, and reading the per-site content keys directly sidesteps the index lost-update race the PHP materialize() is subject to. The DrainHandler gen short-circuit is left as the fast-path optimization; a PHP-side level-triggered materialize would be redundant with this and still exposed to the index race. Flagging for a maintainer decision if you'd prefer symmetry.

Tests added

• reconcile.rs: live-vs-orphan classification, teardown-intent-while-still-in-index prunes, orphan-not-in-index prunes, live-preview-missing-from-index is NOT pruned (the false-positive guard), add-when-missing, add opt-in, dir listing filters (infra/.tmp/files), dry-run removes nothing, real prune removes the dir, per-cycle cap defers.
• kv.rs: ClusterReader::snapshot end-to-end against a mock RESP server (AUTH → index → per-label previews, incl. an on-disk-only extra label and a genuine nil), dead-addr error, parse_index.
• queue.rs: enqueue round-trips through the consume path and leaves no temp file; has_job_for_label sees pending and claimed.

Gates green locally: cargo check --all-targets, cargo clippy --all-targets -- -D warnings, cargo test (330 passed), cargo fmt --all -- --check, cargo +1.85 check --all-targets (MSRV).

[ephpm]

ePHPm Preview — removed

Preview deployment has been torn down.