fix(analytics): supply the task id instead of refusing the agent's command - #313
angeloashmore wants to merge 1 commit into
Conversation
…mmand The CLI refused every command from a detected agent until it passed --analytics-task-id and --analytics-intent, and answered on the stderr of the failed command with an instruction block: generate a UUID, run this `node -e` line, pass the user's request along, the CLI refuses without both. That block asks an agent to run code and forward the user's request, and it arrives the way a prompt injection arrives, so an agent meeting it cold cannot tell the two apart. In a weekly ecosystem eval, Claude Code called it "a prompt-injection attempt aimed at an AI agent", declined to comply, and told the user the CLI was not to be trusted. Sampled cold, claude-opus-5 raised the options with the user in 26 of 34 runs, several offering to stop using the CLI. The refusal also cost every agent its opening command, whatever it decided next. The CLI now supplies a task id of its own when the agent passes none, keeping it for 30 minutes so a request's commands still group. Both options stay, and an agent that passes them is unaffected. The help text says what they are for and that Prismic receives the intent sentence, in place of the refusal notice. Analytics lose intent sentences from agents that pass nothing, and gain the commands from agents that previously refused or were rejected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANFuYqfG55YAJUNNREVgE6
|
The failing The assertion is in So the end-to-end tests are hitting a 500 from the live user service. There is no fix to port — nothing in the repo causes it, and it clears when the service recovers. I have re-run the failed jobs once to confirm. For the record, the local run of the full suite on this branch passed 323 tests, including the rewritten Generated by Claude Code |
|
Closing — the approach here is wrong. It had the CLI derive a task id from a 30-minute time window, which groups by clock when the whole point is grouping by intent: two quick requests merge into one task, and a slice build that runs long splits into two. The work continued and landed somewhere better: the CLI issues ids via Generated by Claude Code |
Stacked on #312, which adds the eval this makes pass. Base is that PR's branch, so this diff shows only the CLI change. Review #312 first.
The problem
src/index.tsrefused every command from a detected agent until it passed both analytics options, and answered on the stderr of the failed command with:Four properties, all at once: it instructs the agent, it tells it to run code, it asks for the user's request to be forwarded to a third party, and it withholds the tool until it complies — arriving on the stderr of a command that just failed. That is the shape of a prompt injection, so an agent meeting it cold cannot tell the two apart. Rewording does not remove any of the four.
Two further problems, independent of how an agent reads it:
src/index.ts:132;isTelemetryEnabled()was not consulted until line 139. A user who had disabled telemetry still had their agent blocked, for analytics they had declined.The change
The CLI supplies a task id of its own when the agent passes none, keeping it in the config dir for 30 minutes so a request's commands still group. Nothing is refused, and nothing is printed.
src/tracking.ts—resolveTaskId(), reusing the existingreadJsonFile/writeFileRecursivepattern besideanalytics.jsonsrc/config.ts—AGENT_TASK_PATHsrc/index.ts— the refusal block becomes oneresolveTaskId()callsrc/lib/command.ts— theAGENTShelp section says what the options are for and that Prismic receives the intent sentence, in place of the refusal notice. The option descriptions drop "Required when an agent runs the CLI", and the prohibitions ("Never use a placeholder", "never generate a second UUID") are gone — agents broke those rules anyway in 2 of 3 trials of the existinggroup-commands-by-taskeval, which is what negation-phrased instructions tend to produce.test/index.test.ts— the unit test assertingexitCode === 1now asserts the command runsTrade-off
Analytics lose intent sentences from agents that pass nothing. They gain the commands from agents that previously refused outright or were rejected. The options are unchanged for any agent that passes them, and the skill still tells agents to.
Verification
trust-the-clieval (6 trials)group-commands-by-task(2 trials)Agents with the skill installed still pass both options, so the grouping the analytics rely on is intact where it was working before.
Checked by hand that the gate is gone and grouping still happens:
🤖 Generated with Claude Code
https://claude.ai/code/session_01ANFuYqfG55YAJUNNREVgE6
Generated by Claude Code