Plugin telemetry: routing funnel, discovery link, MCP headers (GRO-305) by teallarson · Pull Request #11 · ArcadeAI/arcade-plugin

teallarson · 2026-09-15T15:03:13Z

Linear: https://linear.app/arcadedev/issue/GRO-305/evaluate-optional-plugin-side-telemetry-via-hooks

Summary

Adds a small plugin telemetry stream and a correlation bridge to authenticated gateway usage:

routing events --host_session_hash--> discovery link --query_id--> gateway SelectTools Usage --principalId--> account usage

The bridge is intentionally partial. It applies when Arcade_SelectTools successfully returns a query_id.

What this PR adds

Hook events for session start, routing context, bare-continuation skips, subagent start, and sanitized hook errors.
Plugin discovery linked after a successful SelectTools call when its result contains a valid query_id.
Host adapters for Cursor's result_json and the MCP structuredContent and JSON text-content envelopes used by Claude Code and Codex.
Runtime guards that narrow untrusted hook input and result values before reading fields; malformed and non-object payloads are ignored.
Static Arcade-Plugin: arcade and Arcade-Plugin-Version headers in generated MCP manifests.
Default-on hook telemetry with ARCADE_PLUGIN_TELEMETRY=0 opt-out.

Failure hooks are intentionally omitted: a failed tool call does not return the query_id needed for the correlation bridge, and Codex does not expose a PostToolUseFailure event.

Hook events use a hashed host session and do not send prompt text, tool arguments, tool results, paths, tokens, emails, or error messages. Events without a host session ID are skipped.

Correlation contract

Stream	Identity	Join properties
Plugin hooks	`host_session_hash`	Discovery link also carries `query_id`
Gateway Usage	authenticated `principalId`	Plugin requests carry `plugin_source`; SelectTools will carry `query_id`

plugin_version is useful for debugging but is not required for correlation or dashboards. Plugin headers are client-declared analytics metadata, never authorization.

Follow-up

The monorepo change described in docs/research/gateway-telemetry-followup.md will:

Record Arcade-Plugin as plugin_source on Usage events and preserve plugin_version when present.
Add the server-generated SelectTools query_id to the principal-attributed MCP tool recommendation queried Usage event on both stateful and stateless paths.

Then run one real SelectTools call in Cursor, Claude Code, and Codex to confirm the headers and result envelopes survive installation and authentication.

Limits

query_id can be absent, so some conversations will not be cross-referenceable.
Multi-task SelectTools exposes the first nonempty task query ID.
Direct app-tool calls have no SelectTools bridge, though plugin_source still attributes their gateway usage.
Hook delivery is best effort; a missing link event is unknown coverage, not a failed discovery.

Canonical design: docs/research/telemetry-design.md

Validation

npm run verify — 68 tests plus manifest generation, schema validation, plugin discovery, and Claude validation
Real-host SelectTools conformance in Cursor, Claude Code, and Codex

teallarson · 2026-09-15T15:46:03Z

@BugBot review

teallarson · 2026-09-15T15:46:20Z

@BugBot review

cursor

Stale Bugbot comment from a previous run.

teallarson · 2026-09-15T16:02:22Z

@BugBot review

cursor

Stale Bugbot comment from a previous run.

teallarson · 2026-09-15T17:47:12Z

@BugBot review

cursor

Stale Bugbot comment from a previous run.

teallarson · 2026-09-15T18:30:39Z

@BugBot review

cursor

Stale Bugbot comment from a previous run.

Add gateway-auth guardrails across try-arcade, arcade-operator, routing guidance, and the Cursor rule, plus a beforeSubmitPrompt hook so Cursor gets the same per-turn reminder Claude already has. Co-authored-by: Cursor <cursoragent@cursor.com>

SESSION_CONTEXT is injected into both Cursor and Claude SessionStart hooks; referring to Cursor only sent Claude users to the wrong place. Co-authored-by: Cursor <cursoragent@cursor.com>

Cursor's beforeSubmitPrompt only allows or blocks prompts; additional_context is ignored. Per-turn needsAuth guidance stays on the always-apply arcade rule. Co-authored-by: Cursor <cursoragent@cursor.com>

Rebase onto GRO-353 so SUBAGENT_CONTEXT includes the same no-substitute auth block as session and prompt guidance, keeping CI marker checks green. Co-authored-by: Cursor <cursoragent@cursor.com>

Supplemental funnel events post to PostHog via p.arcade.dev with per-machine install_id correlation, opt-out via ARCADE_PLUGIN_TELEMETRY=0, and no prompt text in payloads. Co-authored-by: Cursor <cursoragent@cursor.com>

Stacks on the Codex adapter work so SubagentStart emits the same supplemental funnel events as other lifecycle hooks. Co-authored-by: Cursor <cursoragent@cursor.com>

Wire resolvePosthogIngestHost through the detached capture script so the documented ingest override applies to actual event delivery. Co-authored-by: Cursor <cursoragent@cursor.com>

…nts. Rebased onto GRO-353 contract stack. Adds routing_context_emitted, routing_skipped_bare_continuation, hook_error, and post-arcade-tool MCP telemetry wired for Claude (PostToolUse/PostToolUseFailure) and Cursor (afterMCPExecution/postToolUseFailure). Codex post-tool hooks skipped: shared hooks/hooks.json is loaded by Codex and has no confirmed PostToolUse surface in the com.openai extension manifest. Co-authored-by: Cursor <cursoragent@cursor.com>

Record sanitized hook failures under ~/.arcade-plugin/self-reports/ with zero egress, independent of ARCADE_PLUGIN_TELEMETRY opt-out. Rebase onto GRO-353 hook manifest inventory hashing. Co-authored-by: Cursor <cursoragent@cursor.com>

Move PostToolUse telemetry to clients/claude/hooks/hooks.json and wire both hook paths from .claude-plugin/plugin.json so Codex no longer registers undocumented post-tool handlers from hooks/hooks.json. Co-authored-by: Cursor <cursoragent@cursor.com>

Same cross-host maintainer doc as GRO-353; includes Claude post-tool manifest split from the telemetry stack. Co-authored-by: Cursor <cursoragent@cursor.com>

Document plugin_version on every capture, fix Codex hook manifest split wording, and update the support matrix for Claude post-tool adapters. Co-authored-by: Cursor <cursoragent@cursor.com>

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 57525c0. Configure here.}

cursor · 2026-09-15T20:55:53Z

+  "hooks": [
+    "./hooks/hooks.json",
+    "./clients/claude/hooks/hooks.json"
+  ]


Claude hooks may fire twice

Medium Severity

.claude-plugin/plugin.json now lists ./hooks/hooks.json plus the Claude-only post-tool file. Claude Code already auto-discovers hooks/hooks.json, and custom hook paths supplement that default rather than replacing it. SessionStart and UserPromptSubmit can therefore run twice, injecting duplicate routing context and double-counting telemetry.

Additional Locations (1)

scripts/generate-manifests.mjs#L74-L75

^{Reviewed by Cursor Bugbot for commit 57525c0. Configure here.}

Document which hosts consume root extensions.* (Codex only today), why sidecars remain for Cursor and Claude, and how we verified that with host docs plus local claude validate and cursor agent probes. Co-authored-by: Cursor <cursoragent@cursor.com>

teallarson · 2026-09-16T17:04:06Z

+      error_class: errorClassFrom(error),
+    },
+  });
+};


Can we make this opt-out instead? Telemetry should be enabled by default and disabled only with an explicit ARCADE_PLUGIN_TELEMETRY=0 (or equivalent) setting. Please update the docs and tests to match.

Done in 97b8c1c on the #12 stack. isTelemetryEnabled now defaults on and only opts out for ARCADE_PLUGIN_TELEMETRY=0 / off / false / no. README, ARCHITECTURE, and tests updated.

teallarson · 2026-09-16T17:08:26Z

-      }
-    ],
    "SubagentStart": [
      {


Can you please audit the hook/event coverage across all supported clients? Several telemetry signals are currently wired for only one client even though comparable events appear to exist elsewhere (for example, Codex supports PostToolUse for MCP calls). Please add the appropriate cross-client coverage where supported, and update the implementation, documentation, and tests to match. Where parity is not possible, document the host-specific limitation explicitly.

Done in 97b8c1c. Wired Codex PostToolUse / PostToolUseFailure in com.openai/hooks/hooks.json, added the cross-host coverage table in ARCHITECTURE, and documented Cursor's MCP-specific hook names in support-matrix.

Default hook telemetry on with ARCADE_PLUGIN_TELEMETRY=0 opt-out, wire PostToolUse telemetry for Codex, and document cross-host hook coverage. Co-authored-by: Cursor <cursoragent@cursor.com>

Inspect hook payloads for error shapes instead of trusting success-hook argv alone on afterMCPExecution and PostToolUse paths. Co-authored-by: Cursor <cursoragent@cursor.com>

Shrink hooks to routing funnel and SelectTools query_id links, add static MCP headers for gateway attribution, and document the monorepo follow-up spec. Co-authored-by: Cursor <cursoragent@cursor.com>

teallarson changed the base branch from main to teal/gro-353-add-codexchatgpt-plugin-hooks-for-telemetry-and-lifecycle September 15, 2026 15:10

teallarson mentioned this pull request Sep 15, 2026

Distinguish Arcade auth blockers from gateway failures (GRO-354) #10

Merged

cursor Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread hooks/telemetry-send.mjs Outdated

cursor Bot reviewed Sep 15, 2026

View reviewed changes

teallarson force-pushed the teal/gro-305-evaluate-optional-plugin-side-telemetry-via-hooks branch from cc2b089 to 349a978 Compare September 15, 2026 17:22

teallarson mentioned this pull request Sep 15, 2026

portable manifest adapters (GRO-353) #9

Open

teallarson force-pushed the teal/gro-305-evaluate-optional-plugin-side-telemetry-via-hooks branch from 349a978 to b93b696 Compare September 15, 2026 17:32

teallarson mentioned this pull request Sep 15, 2026

Improve Arcade client listing name and icon (GRO-356) #12

Open

cursor Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread hooks/hooks.json Outdated

teallarson force-pushed the teal/gro-305-evaluate-optional-plugin-side-telemetry-via-hooks branch from b93b696 to a265175 Compare September 15, 2026 17:58

cursor Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread hooks/telemetry.mjs

Comment thread clients/cursor/hooks/hooks.json

teallarson force-pushed the teal/gro-305-evaluate-optional-plugin-side-telemetry-via-hooks branch from df8480d to 5640e06 Compare September 15, 2026 20:18

teallarson changed the base branch from teal/gro-353-add-codexchatgpt-plugin-hooks-for-telemetry-and-lifecycle to teallarson/stop-fallback-on-arcade-needsauth September 15, 2026 20:18

teallarson changed the title ~~Add optional plugin-side telemetry via hooks (GRO-305)~~ Add opt-in plugin-side telemetry via hooks (GRO-305) Sep 15, 2026

teallarson marked this pull request as ready for review September 15, 2026 20:41

teallarson and others added 5 commits September 15, 2026 16:45

Use host-agnostic auth wording in shared session context.

020404e

SESSION_CONTEXT is injected into both Cursor and Claude SessionStart hooks; referring to Cursor only sent Claude users to the wrong place. Co-authored-by: Cursor <cursoragent@cursor.com>

Remove Cursor beforeSubmitPrompt hook that cannot inject context.

5bd1b43

Cursor's beforeSubmitPrompt only allows or blocks prompts; additional_context is ignored. Per-turn needsAuth guidance stays on the always-apply arcade rule. Co-authored-by: Cursor <cursoragent@cursor.com>

Extend needsAuth guardrails to subagent routing context.

b16a7c7

Rebase onto GRO-353 so SUBAGENT_CONTEXT includes the same no-substitute auth block as session and prompt guidance, keeping CI marker checks green. Co-authored-by: Cursor <cursoragent@cursor.com>

fix: distinguish auth from gateway failures

ad43a97

teallarson force-pushed the teallarson/stop-fallback-on-arcade-needsauth branch from 75f74a6 to ad43a97 Compare September 15, 2026 20:46

teallarson and others added 4 commits September 15, 2026 16:46

Add optional hook-side telemetry for Cursor and Claude Code.

b2b9aac

Supplemental funnel events post to PostHog via p.arcade.dev with per-machine install_id correlation, opt-out via ARCADE_PLUGIN_TELEMETRY=0, and no prompt text in payloads. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge Codex hook routing from GRO-353 and add SubagentStart telemetry.

c2a6ecd

Stacks on the Codex adapter work so SubagentStart emits the same supplemental funnel events as other lifecycle hooks. Co-authored-by: Cursor <cursoragent@cursor.com>

Honor ARCADE_PLUGIN_POSTHOG_HOST in telemetry-send worker.

c76a0be

Wire resolvePosthogIngestHost through the detached capture script so the documented ingest override applies to actual event delivery. Co-authored-by: Cursor <cursoragent@cursor.com>

teallarson and others added 5 commits September 15, 2026 16:46

Move hook adapter guidance from Cursor rules to AGENTS.md.

febedf4

Same cross-host maintainer doc as GRO-353; includes Claude post-tool manifest split from the telemetry stack. Co-authored-by: Cursor <cursoragent@cursor.com>

fix: harden opt-in hook telemetry

5bdd85a

docs: align telemetry and hook manifest docs with stack head.

57525c0

Document plugin_version on every capture, fix Codex hook manifest split wording, and update the support matrix for Claude post-tool adapters. Co-authored-by: Cursor <cursoragent@cursor.com>

teallarson force-pushed the teal/gro-305-evaluate-optional-plugin-side-telemetry-via-hooks branch from 0232f29 to 57525c0 Compare September 15, 2026 20:47

cursor Bot reviewed Sep 15, 2026

View reviewed changes

teallarson marked this pull request as draft September 16, 2026 13:16

Base automatically changed from teallarson/stop-fallback-on-arcade-needsauth to teal/gro-353-add-codexchatgpt-plugin-hooks-for-telemetry-and-lifecycle September 16, 2026 13:22

teallarson marked this pull request as ready for review September 16, 2026 16:55

teallarson marked this pull request as draft September 16, 2026 16:55

teallarson marked this pull request as ready for review September 16, 2026 16:55

teallarson commented Sep 16, 2026

View reviewed changes

teallarson marked this pull request as draft September 16, 2026 17:09

teallarson and others added 3 commits September 16, 2026 13:19

Address PR review: opt-out telemetry and Codex post-tool hooks.

3b8e7d5

Default hook telemetry on with ARCADE_PLUGIN_TELEMETRY=0 opt-out, wire PostToolUse telemetry for Codex, and document cross-host hook coverage. Co-authored-by: Cursor <cursoragent@cursor.com>

Detect in-band MCP failures in post-arcade-tool telemetry.

d4758d5

Inspect hook payloads for error shapes instead of trusting success-hook argv alone on afterMCPExecution and PostToolUse paths. Co-authored-by: Cursor <cursoragent@cursor.com>

Reshape hook telemetry per gateway-first design (GRO-305).

b8a9fb4

Shrink hooks to routing funnel and SelectTools query_id links, add static MCP headers for gateway attribution, and document the monorepo follow-up spec. Co-authored-by: Cursor <cursoragent@cursor.com>

teallarson changed the title ~~Add opt-in plugin-side telemetry via hooks (GRO-305)~~ Plugin telemetry: routing funnel, discovery link, MCP headers (GRO-305) Sep 16, 2026

teallarson added 2 commits September 16, 2026 18:52

Clarify telemetry correlation contract

c1c5b81

Align telemetry hooks with host adapters

63f148f

Conversation

teallarson commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What this PR adds

Correlation contract

Follow-up

Limits

Validation

Uh oh!

teallarson commented Sep 15, 2026

Uh oh!

teallarson commented Sep 15, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

teallarson commented Sep 15, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teallarson commented Sep 15, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

teallarson commented Sep 15, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 15, 2026

Choose a reason for hiding this comment

Claude hooks may fire twice

Uh oh!

teallarson Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

teallarson Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

teallarson Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

teallarson Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

teallarson commented Sep 15, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading