fix(prime): surface daemon startup failures and carry the fork env through the host sanitizer

[drewstone]

Fixes #194. Part of tangle-network/discovery#165 (the lab-side ticket for the same failure).

What was wrong

prime requests on the lab bridge died with Prime Agent daemon exited during startup and the bridge clipped the daemon log tail and exit code out of the error; pi on the same bridge answered 401 without saying which credential it had resolved.

Cause established in the bridge's spawn path. The host executors re-filter every child env through sanitizeHostEnv (src/executors/host.ts), whose allowlist had no PRIME_ prefix. So:

• PRIME_AGENT_INTERNAL_LEGACY_OWNED_WORKER_FRONTEND=1 — the no-daemon contract the prime backend is built on — never reached the fork, which therefore went to its background daemon (ensureDaemonRunning, the frame in the issue's stack).
• Inside a jail the per-run --daemon-socket dir under the host tmpdir is hidden (fs-jail replaces /tmp with an empty tmpfs, write-jail binds / read-only), so the daemon supervisor dies on mkdir <socket>.lock (ENOENT / EACCES) — the line the 300-byte clip hid. The lab's 8901 env carries BRIDGE_JAIL_RO_PATHS/BRIDGE_JAIL_RW_PATHS, i.e. the Linux fs-jail, so this is its shape.
• PRIME_AGENT_CODING_AGENT_DIR (the fork's only agent-dir name) and the kernel knobs were dropped too; PRIME_PERSISTENT_AGENT_DIR was silently ignored.
• Any models.json apiKey name outside the allowlist (DEEPSEEK_API_KEY, LIVE_FAKE_KEY, …) was dropped, so the fork resolved the bare name as the literal credential. (TANGLE_ROUTER_KEY always passed through the existing TANGLE_ prefix.)

Not the cause: the socket path length (/tmp/prime-sock-XXXXXX/d.sock is 27 bytes on Linux, ~75 under the macOS tmpdir; both far under the 104/108-byte AF_UNIX caps) and the pinned HOME/XDG (the fork bootstraps .prime/agent under any HOME; the bridge-owned one is writable in every jail mode).

Reproduced live (fork built at the pinned be9e2fa0, through the bridge, fake OpenAI-compatible provider on loopback)

checkout	jail	result
origin/main	off	501 prime assistant turn failed: 401 Invalid API key; provider saw Bearer LIVE_FAKE_KEY (the env name)
origin/main	write-jail	502 prime exit 1: file:///…/chunk-2PPDA3IT.js … [... clipped ...] … ensureDaemonRunning (…:60733:3) — the issue's error byte-for-byte (same line numbers 60719/60733)
this branch	off	200, PONG-LIVE completion
this branch	write-jail	200, PONG-LIVE completion
this branch, PRIME_BIN wrapper that strips the frontend variable and breaks the socket dir	off	502 whose message carries prime exit 1, the fork's Recent daemon log (<agent dir>/logs/d.sock.<id>.log): line, the full tail (ENOENT … mkdir '…/d.sock.lock'), Node.js v22.23.2, and a [bridge] line naming the socket it handed over and the contract that was lost

The lab bundle's chunk hash (chunk-IAW7YJNT) differs from the pinned build's (chunk-2PPDA3IT) at identical line numbers, so the lab fork is a nearby but not identical build; with this branch the daemon is not reached at all for --mode rpc on be9e2fa0, and if the lab build reaches it anyway the error now says so and carries the daemon log.

Changes

• executors: PRIME_ joins the proxied env prefixes; SpawnOpts.envPassthroughKeys lets a backend declare request-resolved names by exact name. The prime backend declares the apiKey variables its models.json names.
• prime (ask 1): a non-zero exit carries the exit code (or terminating signal) and the retained stderr in full (64 KiB head+tail bound, omissions marked — no 300-byte clip) — the fork's daemon log path and tail included — and, when the fork reached its daemon at all, names the socket it was handed and both ways the owned-worker request can be lost (a build that ignores it; a wrapper or env filter that strips it).
• prime (jail): the per-run daemon socket dir is exposed writable inside an OS jail (bwrap binds extraWritablePaths after --tmpfs /tmp, so it wins; the supervisor locks <socket>.lock before listening), and the prompt dir is registered readable (an invisible --append-system-prompt path is taken by the fork as literal prompt text).
• pi (ask 2): a 401/403 names the credential source the bridge resolved — the agent dir's auth.json entry, the env-var template behind providers.<p>.apiKey with whether each variable is set in the bridge environment, a bare identifier (reported by presence only, since upstream pi treats it as the key itself), a literal, a !command, or the protected request header — never a value or the apiKey string.

Tests

• clipped-vs-full error with the captured daemon-startup stderr; rpc error + stderr both kept; signal rendering; the two-cause hint
• a real fake prime-agent through hostSpawner whose daemon dies at startup (log written under the agent dir, launcher error on stderr)
• the real host executor delivering the no-daemon contract, the agent dir pin and a non-allowlisted apiKey name (and dropping an ambient PRIME_* lookalike is still covered by the sanitizer allowlist tests)
• jail path registrations (socket dir writable, prompt dir readable)
• sanitizeHostEnv prefix and exact-name passthrough
• every credential-source shape, the request-header constant, and the 401 message end-to-end

pnpm typecheck green. pnpm test: CI (Linux) green. On this macOS host 11 tests in docker-executor, failure-attribution, jail, pi-inference-isolation fail identically in an origin/main worktree (no Docker, /var→/private/var, /proc/self/environ); every other suite (1035) and test:runtime-consumer pass.

Deploy note for the lab bridge (port 8912)

Not restarted from this PR. After merge, on the lab host: find the checkout the 8912 unit runs (ps -o args= -p <pid of the 8912 listener> shows the src/server.ts path), git pull main there, pnpm install --frozen-lockfile, restart that unit with its existing env — BRIDGE_BACKENDS including prime, PRIME_BIN, PRIME_MODELS_JSON, the router key variable the models.json names (TANGLE_ROUTER_KEY), and the jail variables (BRIDGE_JAIL_MODE, BRIDGE_JAIL_RO_PATHS, BRIDGE_JAIL_RW_PATHS) if set. If ~/.cache/cli-bridge/prime-agent is not on be9e2fa0, rebuild the fork at the pin (PRIME_AGENT_REF=be9e2fa0 pnpm install:harness prime).

Proof: POST /v1/chat/completions with {"model":"prime/tangle-router/deepseek-v4-flash","messages":[{"role":"user","content":"reply with exactly the word pong"}]} returns 200 with choices[0].message.content and usage.model_requests: 1; a 502 now carries the full daemon log tail instead of [... clipped ...]; a pi/tangle-router/... 401 now ends in (credential source: …).

Review fixes (1a9687e)

• describePiCredentialSource no longer renders the apiKey string for a bare identifier: the bare identifier under providers.<p>.apiKey in <models.json> (an env var of that name is set|unset in the bridge environment). Test covers hex, a9f3…-style, and sk_live_ identifier-shaped literals with not.toContain. The pre-spawn resolution failure (cannot establish isolated inference auth) now names the credential source too.
• Prime non-zero exit classification uses primeExitFailureKind: the rpc error / error event when present, else the first non-empty stderr line (300-byte cap); the message keeps the whole retained stderr. piFailureKind matches 401/403 only as whole tokens, so 15.401Z and chunk.js:4031 no longer classify. Tests: the daemon fixture with a .401Z timestamp and a :4031 line number stays upstream; a 401 Unauthorized rpc error or first stderr line stays not_configured.

Re-review fix (612c367)

• describePiCredentialSource now finds template names with piConfigValueEnvNames, a port of pi's own template scan. The earlier scan rendered literal key text as a variable name: $A$$tail rendered Atail, and k${tail rendered tail. A fuzz of 20,000 inputs against the installed pi 0.84.2 parser found 976 divergent inputs before this change and 0 after.
• The auth.json credential type renders only as api_key, oauth, or unrecognized-type.

🤖 Generated with Claude Code

[drewstone]

Adversarial review of this branch at b67d59c, run in a fresh worktree: pnpm install --frozen-lockfile clean, pnpm typecheck green, pnpm test 1035 passed / 11 failed (the same jail + pi-inference-isolation macOS-only set the PR body names: no Docker, /var vs /private/var, /proc/self/environ). CI test check is green.

I also ran every new/modified test against origin/main's src/ (test files from this branch, new-symbol imports stubbed): all 14 new tests and both extended assertions fail on main and pass here, so the tests do exercise the change, including the two real-process tests through hostSpawner.

Blocking

1. describePiCredentialSource prints a literal API key whenever the key is identifier-shaped

src/backends/pi-inference-transport.ts (describePiCredentialSource): the bare-name branch is referenced.length === 0 && /^[A-Za-z_][A-Za-z0-9_]*$/u.test(apiKey) and then renders apiKey itself into the message ((${name} is set/unset in the bridge environment)). Any literal credential that happens to match identifier syntax is therefore echoed across the HTTP boundary in the 401 body. That is exactly the case the docblock says exists: in the upstream pi 0.8x line a bare identifier IS a literal key. Checked by calling the function directly with a models.json whose apiKey is a literal:

• a9f3e1c2b4d5e6f7a8b9c0d1e2f3a4b5 (hex token starting with a letter) -> rendered verbatim
• sk_live_abcDEF123 (underscore-style key) -> rendered verbatim
• sk-live-x (dash) -> "the literal apiKey under ..." (safe)

The "never renders a literal key" test only uses dashed literals, so it cannot catch this. The route puts err.message straight into the 501 JSON body, so this is a credential leak to the caller (and to any log that stores response bodies).

Fix: never echo the identifier. Report presence without the name, e.g. the bare identifier under providers.<p>.apiKey in <models.json> (an env var of that name is set|unset in the bridge environment), and add a test with an identifier-shaped literal (abcdef0123456789) asserting not.toContain.

2. Widening piFailureKind to the full 64 KiB stderr makes the prime exit code/status wrong on incidental 401/403 substrings

src/backends/prime.ts exit path: piFailureKind(detail) now runs /401|403|token expired|forbidden|unauthorized/i over sawError + the whole retained stderr instead of a 300-byte clip. The daemon log tail this PR deliberately includes is full of millisecond timestamps and bundle line numbers; checked directly: piFailureKind('[2026-09-10T04:50:15.401Z] supervisor: ...') and piFailureKind('chunk.js:4031 at foo') both return not_configured. The route maps that to HTTP 501 not_configured (never retried) instead of 502 upstream for what is a daemon crash. The PR's own fixture asserts err.code === 'upstream' for a daemon-startup failure; shift the fixture's timestamp by 16 ms and that assertion fails. The regex is pre-existing, but this PR is what feeds it kilobytes of timestamped log text, and its whole point is an accurate failure report.

Fix: classify from the structured signal only (sawError / the rpc error / turnFailure), or from a bounded first line of stderr, not the log tail; keep the full tail in the message. A test with a .401Z timestamp in DAEMON_STARTUP_STDERR expecting upstream pins it.

Non-blocking

• PRIME_ as a proxied prefix in sanitizeHostEnv changes the env of every host-spawned backend, not only prime: claude, codex, and opencode spread process.env, so PRIME_BIN, PRIME_MODELS_JSON, PRIME_PERSISTENT_AGENT_DIR, PRIME_PROGRESS_MS, and any operator key named PRIME_* (the test suite itself uses PRIME_TEST_ROUTER_KEY) now reach those CLIs and their tools. Not a break and consistent with the existing TANGLE_/OPENAI_ prefixes, but every fork knob you name is PRIME_AGENT_*; the tighter prefix PRIME_AGENT_ (or the five exact names) gives the same fix with a smaller blast radius. The prime backend itself is unaffected because primeProcessEnvironment starts from a neutral allowlist.
• Pi: the pre-spawn credential resolution failure (resolvePiAuthCredential throwing -> backend pi cannot establish isolated inference auth for <p>/<m>) is a not_configured on the same credential and still does not name the source; an unset $VAR in the pi line that treats bare names as env vars ends there, not at a 401. describePiCredentialSource is already computed on that path; appending it there would close the last gap in "ask 2".
• Prime exit paths: startup exit and signal kill go through describePrimeExit with the full retained stderr (checked the buffer: render() with no argument returns head + tail with an explicit [... N bytes omitted ...] marker, never [... clipped ...]). Spawn ENOENT does not: it stays prime spawn failed: spawn <bin> ENOENT (upstream, 502) with no exit code, which is correct since nothing ran and there is no daemon log; arguably it should be cli_missing (503), but that is pre-existing.
• describePrimeExit with exitCode: null and no signal renders prime exit unknown: ...; with empty stderr and no rpc error the message is prime exit N: exit N. Both pre-existing shapes, cosmetic.
• The [bridge] hint names two causes (build ignores the frontend, wrapper/env filter drops the variable). A build that honors the variable for --mode rpc but not on some other path (e.g. --continue) would be a third; fine as a hint, just not exhaustive.
• Jail: --bind of extraWritablePaths is after --tmpfs /tmp in linux-bwrap.ts, so the socket dir does reappear; registerJailReadable is a no-op without readConfine, which is the correct scope for the prompt dir.

[drewstone]

Both blocking findings are fixed in 1a9687e.

1. describePiCredentialSource: the bare-identifier branch no longer renders the apiKey string. It reports the bare identifier under providers.<p>.apiKey in <models.json> (an env var of that name is set|unset in the bridge environment). New test feeds abcdef0123456789, a9f3e1c2b4d5e6f7a8b9c0d1e2f3a4b5, and sk_live_abcDEF123 with and without a matching env var and asserts not.toContain(literal); it fails on the previous source. The pre-spawn cannot establish isolated inference auth failure now appends (credential source: …) as well (your non-blocking note 2), pinned in pi-inference-isolation.test.ts.

2. Prime exit classification: primeExitFailureKind(sawError, retainedStderr) classifies from the rpc error / error event when present, else from the first non-empty stderr line capped at 300 bytes; the message still carries the full retained stderr. piFailureKind also matches 401/403 only as whole tokens (\b(?:401|403)\b) so a :4031 bundle line number on the first line cannot classify either. Tests: the captured daemon fixture with its timestamp shifted to .401Z and a frame at :4031 stays upstream; a 401 Unauthorized: token expired rpc error or first stderr line stays not_configured; both fail on the previous source. Existing pi 401 tests (stderr and errorMessage shapes) unchanged and green.

Local: pnpm typecheck clean; pnpm test 1039 passed / 11 failed, the same macOS-only Docker + jail + /proc/self/environ + /private/var set as before.

Not changed: the PRIME_ proxied prefix in sanitizeHostEnv (note 1) — left as-is for consistency with TANGLE_/OPENAI_; narrowing to PRIME_AGENT_ is a follow-up if the wider env for claude/codex/opencode children matters.

[drewstone]

Independent adversarial re-review of 1a9687e, plus fix 612c367

Verdict: both blocking findings from the 06:16Z review are fixed in 1a9687e. This re-review found one more blocking credential leak in the same function; 612c367 fixes it. No blocking finding remains.

Setup: fresh worktree at origin/fix/prime-daemon-startup-diagnostics (1a9687e). pnpm install --frozen-lockfile clean. pnpm typecheck clean on 1a9687e and on 612c367.

Blocking finding 1 (identifier-shaped literal echoed): fixed

• I ran the 1a9687e test files against b67d59c's src/. never echoes an identifier-shaped literal key fails there with expected 'the bare name under …' not to contain 'abcdef0123456789'. It passes on 1a9687e.
• Independent check: 4 identifier-shaped literals (hex, sk_live_…, ghp_…), each with and without an env var of that name. Also checked: env values behind $VAR, ${VAR}, and Bearer ${VAR}; an auth.json key; an auth.json entry that is a bare string; a !echo <secret> command. No value appears in the output.

Blocking finding 2 (401 inside timestamps and line numbers): fixed

• On b67d59c's src/, classifies a daemon crash as upstream … fails with expected 'not_configured' to be 'upstream', and so does the piFailureKind whole-token test. Both pass on 1a9687e.
• primeExitFailureKind classifies from the rpc error or error event when one exists, else from the first non-empty stderr line (300-byte cap). The message keeps the whole retained stderr.

New blocking finding: the template scan printed literal key text as a variable name (fixed in 612c367)

describePiCredentialSource found template names with apiKey.replace(/\$[$!]/gu, '') followed by /\$\{?([A-Za-z_][A-Za-z0-9_]*)\}?/gu. That scan does not match pi's own parser (core/resolve-config-value.ts parseConfigValueTemplate). Pi keeps two kinds of input as literal key text, and the bridge rendered that text as a name in the 401 and pre-spawn error bodies:

apiKey	pi 0.84.2 reads	1a9687e rendered
$ROUTER_PREFIX$$9fQxT2vLm	env ROUTER_PREFIX + literal $9fQxT2vLm	ROUTER_PREFIX9fQxT2vLm is unset …
k3y${fQxT2vLmZ	literal (unclosed ${)	fQxT2vLmZ is unset …
${xY7_q-9fQxT2}	literal (brace body is not an identifier)	xY7_q is unset …

The cause: the escape is deleted instead of consumed, and the braces are optional. I fuzzed 20,000 generated apiKey strings against the installed pi 0.84.2 getConfigValueEnvVarNames. 976 inputs rendered a name that pi does not read.

612c367 adds piConfigValueEnvNames, a port of pi's scan, so only names that pi reads are rendered. The same fuzz finds 0 divergent inputs. The auth.json type is file content, so it now renders only as pi's api_key or oauth, and anything else renders as unrecognized-type. Two new tests fail on 1a9687e and pass on 612c367: expected 'the env var template under providers.…' not to contain '9fQxT2vLm' and expected 'stored sk-typo-secret-in-type credent…'.

What each credential-source path renders on 612c367:

• request header: a constant
• auth.json: its path, plus api_key, oauth, or unrecognized-type
• pi built-in auth: paths
• !command: the key path
• template: the names pi reads, each marked set or unset
• bare identifier: presence only
• literal: the key path

The pre-spawn failure appends the same string. The route renders err.message only, not cause.

CI test on 612c367 (Linux) passed: 1076 tests, 1055 passed, 21 skipped, 0 failed.

New tests against origin/main's src/

I copied the branch test files onto origin/main (30038a6). 22 new or changed tests fail there, not counting the 2 macOS-only isolation failures that fail on both, and all 22 pass on the branch. The real-process env test echoes legacy=;dir=;key=;ambient= on main, because the sanitizer dropped all three variables. On the branch it echoes legacy=1;dir=…/.prime/agent;key=sk-named-by-models-json;ambient=.

Full suite on this macOS host

checkout	tests	passed	failed	skipped
origin/main 30038a6	1056	1021	11	24
612c367	1076	1041	11	24

The 11 failures are the same tests by name on both checkouts: docker-executor 3, failure-attribution 1, jail 5, pi-inference-isolation 2. The causes are no Docker, /var vs /private/var, and /proc/self/environ. This PR's modified pi-inference-isolation tests are not in that set; they ran and passed here. test:runtime-consumer passes.

One full run at 1a9687e also failed pi-native … returns transport failure when Pi consumes a response …: a 2 s waitFor timed out under full-suite load. The test passed 5 of 5 runs alone and did not fail in the 612c367 full run. The test does not exercise the code this diff changes.

Other backends

• The new sanitizeHostEnv parameter defaults to [], and only prime passes envPassthroughKeys. The docker executor has its own allowlist. Prime always runs on the host or scoped-host spawner, because server.ts constructs it without a spawner.
• Host-spawned backends that spread process.env (claude, codex, opencode, kimi, gemini) now also receive the bridge's PRIME_* variables. None of those CLIs reads PRIME_*, so their behavior is unchanged.
• piFailureKind now matches 401 and 403 only as whole tokens, which also applies to pi. HTTP 403, (401), and 401: {…} still classify as not_configured. A status joined to a word character (HTTP401, _401) now classifies only if the text also contains unauthorized, forbidden, or token expired.

Non-blocking

1. The prime exit message now carries up to 64 KiB of the child's stderr without redaction. Every other backend clips stderr to 300 bytes, and claude also redacts Bearer and sk- shapes. The prime child is the one child with real provider keys in its env; the pi child gets a scoped key. I found no fork path that prints a key to stderr: resolve-config-value discards command stderr, and the daemon log holds supervisor lines. So this risk is a hypothesis. Removing the exact home.apiKeyEnv values from prime's error messages would guarantee it. Follow-up.
2. primeExitFailureKind reads the first stderr line. For an uncaught Node throw, that line is file:///…:LINE, so an auth error thrown uncaught classifies as upstream. A bundle line number of exactly 401 or 403 on that line would classify as not_configured. The impact is low, because the fork reports provider auth failures through rpc and turn_end. Skipping Node's location, source, and caret lines would tighten the rule.
3. The PRIME_ prefix is broader than the PRIME_AGENT_* knobs the fork needs, as the first review noted. It is left as is.
4. The PR body lists TANGLE_ROUTER_KEY among the dropped apiKey names, but TANGLE_ is on main's prefix list, so that name always passed. The dropped-name 401 applies to names outside the allowlist, which the PR reproduced with LIVE_FAKE_KEY. The pi 401 on the lab bridge does not pass through sanitizeHostEnv: the pi resolver runs pi auth print-api-key with the bridge's process.env. This PR makes that 401 name its credential source but does not establish its cause.
5. describePiCredentialSource reads auth.json and models.json on every request that has no protected credential header, including successful requests. Its output is used only on failure, so the reads could run only when a failure needs them.