Skip to content

fix(mcp): neutralize agent-visible tool metadata - #340

Merged
hmishra2250 merged 4 commits into
mainfrom
fix/openai-neutral-mcp-metadata-v3
Jul 29, 2026
Merged

fix(mcp): neutralize agent-visible tool metadata#340
hmishra2250 merged 4 commits into
mainfrom
fix/openai-neutral-mcp-metadata-v3

Conversation

@hmishra2250

@hmishra2250 hmishra2250 commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

OpenAI remediation scope

This is a focused, compatibility-preserving remediation for the OpenAI tool-metadata review. It keeps all existing MCP tool names, schemas, routes, and the full /v2/mcp tool surface unchanged. It removes promotional/comparative, native-tool-displacing, and coercive routing language from the rendered initialize instructions and tool descriptions, replacing it with neutral operation boundaries: what the tool does, when it applies, side effects, and what it returns.

It intentionally does not include the selector/alwaysLoad work from #337 and does not modify that PR.

Final v3 lineage and behavior

  • Final agent-visible source wording comes from the compact-neutral v3 lineage: c090912 -> ca2b689 -> a4221e8 (not the historical branch wholesale).
  • The focused branch squashes only that final net metadata behavior plus final policy/rendered-metadata tests.
  • The policy hardening direction is from 2c6b9ca through d0f57f6: it blocks promotion/comparison, Firecrawl-vs-native displacement, always/default coercion, critical mandatory routing, and feedback-for-credit/refund inducement (including split-sentence forms); it permits neutral operational terms such as best for, not recommended, accounting fields, and critical status.
  • No tool is renamed. The rendered full profile remains 26 tools; integration tests assert tool names/order and non-language behavior remain stable.

AX R02: immutable hosted-MCP confirmation

The candidate was evaluated against the production-baseline hosted-MCP state in EXP-032 R02:

  • 360 paired hosted traces = 30 fresh prompts × 2 conditions × 2 lanes (Claude Sonnet 5 and Codex GPT-5.6) × 3 replicates; 144 positive pairs.
  • Each trace used the immutable E2B template z5rhuwx4dcfg3zajn5o8; dispatch reached 100 concurrent physical E2B sandboxes. Per-trace initialized/tools-list state matched the frozen condition fingerprint.
  • Formal deterministic verdict: scoped_equivalent. Candidate minus baseline task success: +2.78pp; preregistered 90% bootstrap CI [0.00pp, +5.56pp], wholly within the ±10pp equivalence margin.
  • Lane deltas: Claude +1.39pp, Codex +4.17pp. There were no positive prompt/lane collapses and 0 unpinned-negative false hijacks in either condition.
  • Exactly 8 discordant pairs received independent hash-bound Claude Sonnet 5 adjudication. The deterministic scorer/verdict remains authoritative; judges only explain discordance.

Evidence bindings

  • Frozen inputs SHA-256: 607ee86944d6b1fdb7ea621d27cf1a1f33f83310697e1cb57d103bfa4df44ee2
  • Frozen input manifest SHA-256: 12578212de0cec00064ac426d844f477689e77bccb0e09d93a75552ed40813ed
  • Raw trace manifest SHA-256: 2ee805f18ecf85a57569dd8ebe62528bf70da47e7ee5a2700346de3ce9595a33
  • Raw trace tree SHA-256: ca05d2067748b6329d50667223062736bd2678e3fc45840a3947e863d904a9ed
  • Score manifest SHA-256: 51f2fe37f6bdf2fbf128079470cb8716f1f57a706027249719770c7d8497d41d
  • Score tree SHA-256: c1894a7dcb10fe35696527053e2cbbf24c2710056e1e036c4357f15e0fc2a530
  • Exact raw spend: $62.468649 / $120.00 hard cap (append-only cost ledger).

Disclosure

R02 had four Codex traces whose actual model cost exceeded the original $1 per-trace acceptance cap: C04 baseline r0, C04 baseline r1, C04 v3 r1, and B16 baseline r2. The runner records these as scoreable policy failures, not missing/provenance-invalid traces; all 360 raw trace identities, pairs, and fingerprints remained complete. They are included in the formal outcome. A preflight-invalid R01 is explicitly excluded from this evidence.

Validation

  • pnpm run build
  • node --test tests/agent-metadata-policy.test.mjs tests/mcp-smoke.test.mjs tests/mcp-search-profile.test.mjs (50 passing)
  • pnpm run lint
  • git diff --check

Review

Code review: APPROVE. Nonblocking P2 follow-up: persist the baked-template verifier JSON directly in raw trace documents; current persisted state fingerprints already provide equivalent R02 integrity evidence, so this does not require a rerun.

Additional nonblocking code-review follow-up: firecrawl_interact accurately discloses that form submission can persist, while its pre-existing destructiveHint: false annotation remains inconsistent. Correct that annotation in a separate safety-focused change before general marketplace resubmission; it is outside this neutral-copy remediation.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 7 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/agent-metadata-policy.mjs">

<violation number="1" location="scripts/agent-metadata-policy.mjs:10">
P2: Neutral operational descriptions using `over` as a transport preposition are rejected as native-tool displacement when both names occur. Restrict `over` to a comparison involving the native/built-in tool instead of treating every occurrence as displacement.</violation>
</file>

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread scripts/agent-metadata-policy.mjs
'\\b(?:us(?:e|es|ed|ing)|choos(?:e|es|ing|en)|select(?:s|ed|ing)?|call(?:s|ed|ing)?|invok(?:e|es|ed|ing)|rout(?:e|es|ed|ing)|prioriti[sz](?:e|es|ed|ing)|prefer(?:s|red|ring)?|default\\s+to)\\b';
const MANDATORY =
'\\b(?:must|mandatory|required|always|only|need(?:s)?\\s+to|have\\s+to)\\b';
const DISPLACEMENT =

@cubic-dev-ai cubic-dev-ai Bot Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Neutral operational descriptions using over as a transport preposition are rejected as native-tool displacement when both names occur. Restrict over to a comparison involving the native/built-in tool instead of treating every occurrence as displacement.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At scripts/agent-metadata-policy.mjs, line 10:

<comment>Neutral operational descriptions using `over` as a transport preposition are rejected as native-tool displacement when both names occur. Restrict `over` to a comparison involving the native/built-in tool instead of treating every occurrence as displacement.</comment>

<file context>
@@ -0,0 +1,306 @@
+  '\\b(?:us(?:e|es|ed|ing)|choos(?:e|es|ing|en)|select(?:s|ed|ing)?|call(?:s|ed|ing)?|invok(?:e|es|ed|ing)|rout(?:e|es|ed|ing)|prioriti[sz](?:e|es|ed|ing)|prefer(?:s|red|ring)?|default\\s+to)\\b';
+const MANDATORY =
+  '\\b(?:must|mandatory|required|always|only|need(?:s)?\\s+to|have\\s+to)\\b';
+const DISPLACEMENT =
+  '\\b(?:instead\\s+of|rather\\s+than|over|in\\s+preference\\s+to|replace(?:s|d|ment|ing)?|substitut(?:e|es|ed|ing|ion)?|supersede(?:s|d|ing)?|take(?:s)?\\s+precedence(?:\\s+over)?|prioriti[sz]e|choose|select|prefer)\\b';
+const NEGATED_NATIVE_SELECTION =
</file context>
Fix with cubic

Comment thread scripts/agent-metadata-policy.mjs
Comment thread scripts/agent-metadata-policy.mjs Outdated
Comment thread src/monitor.ts Outdated
hmishra2250 and others added 2 commits July 29, 2026 00:50
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Marks the neutral-metadata surface with a distinct version and syncs
the stale server.json top-level version (3.7.5) with the package.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hmishra2250
hmishra2250 merged commit 7c764ef into main Jul 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants