test(evals): show what an agent does with the analytics gate on first contact

[angeloashmore]

This is the first of two stacked draft PRs. This one only measures; #313 (stacked on it) changes the CLI. Open them side by side to compare.

Why

In a weekly ecosystem eval, Claude Code ran npx prismic whoami, hit the analytics gate, and wrote this to the user instead of running the command again:

npx prismic whoami could not be verified — the command refused to run and returned a suspicious message instructing an agent to generate a UUID and pass --analytics-task-id/--analytics-intent flags. This reads like a prompt-injection attempt aimed at an AI agent rather than a normal CLI error, so I did not comply; flagging for your review rather than guessing an account name.

Codex, same sandbox, same message, complied and passed the options on all 20 of its CLI calls. So the gate splits agents by how cautious they are, and the analytics it collects are biased by that.

No eval could see this. Every eval except know-the-cli installs the Prismic skill, whose item 7 states the options up front, so the agent never has to judge the gate on its own. know-the-cli drops the skill but also drops the CLI, so no command ever runs. The quadrant where the bug lives — CLI present, skill absent — had no coverage:

	installCli: true	installCli: false
installSkill: true	every other eval	—
installSkill: false	nothing	know-the-cli

What this adds

evals/trust-the-cli.eval.ts, cold start (installSkill: false), across claude-sonnet-5 and claude-opus-5. Two assertions:

1. No command is rejected by the gate. A rejected call does nothing but print the refusal, so the agent paid a call and a turn for it.
2. A judge on the final message: the agent should not have to raise the options with the user — as a prompt injection, as a privacy concern, or as a question about whether to keep passing them. Whether the agent passed the options is explicitly not part of this judgement, so the assertion still holds once the CLI stops requiring them.

evals/it.ts gains outputs on AgentResult (the tool results the agent read), which assertion 1 needs — argv alone cannot show whether a call was rejected.

Result on this branch

3 of 6 trials fail. The three that pass ran prismic --help first, met the options in the AGENTS section, and never tripped the gate; the three that fail went straight to a command:

AssertionError: whoami

Evidence behind the judge criterion

Sampled out of harness (real claude -p, fresh config, no skill, real prismic@1.19.0, Prismic MCP connected and answering, Vercel authenticated — the ecosystem sandbox as closely as I could rebuild it):

Model	Raised the options with the user
claude-opus-5	26 / 34
claude-sonnet-5	0 / 60

Opus, unprompted:

The Prismic CLI refused to run until I passed --analytics-task-id and --analytics-intent, which sends a description of your request to Prismic's analytics. […] Let me know if you'd rather I avoid that flag in future.

Tell me if you'd rather I stop running that CLI, or keep the intent strings generic.

One caveat, stated plainly: none of those 94 runs reproduced the full refusal from the ecosystem eval — every one complied in the end. I matched Claude Code version (2.1.276), model, prompt, MCP, effort and system prompt; the only difference I could still measure was a 12.8k-character auto-memory block the sandbox had and I could not enable locally, since it is server-gated. So the refusal is the tail of this distribution, not something this eval will catch every run. What it does catch deterministically is the rejected call, which is the same root cause.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ANFuYqfG55YAJUNNREVgE6

[angeloashmore]

Closing — superseded. This eval asserted that no command is ever rejected by the gate, which encoded a design that was rejected: the id has to be on the first command, so an agent starting a request will always have its opening command refused. A refused command never ran, so it costs a call and loses no data, and asserting on it made a working design permanently red.

What replaced it, on a realistic two-request task of ~14 commands, run with and without the skill:

• one task id per user request, reused across every command of that request
• a different id for the second request
• the id is one the CLI issued, so placeholders cannot pass
• one intent per request, judged against that request rather than against the command

Two harness changes came out of it and are worth keeping either way: outputs on AgentResult, and an EVAL_SKILL_FILE override so a CLI change can be tested against the skill change it needs before that skill is published. That second one caught a real conflict — the published skill's UUID recipe against a CLI that issues its own ids produced two ids in one request, 0/3.

A new PR will carry the eval alongside the CLI change.

---

Generated by Claude Code