Add opt-in documentation discovery (llms.txt) to the MCP server

[rohitkumarbhagat]

Summary

Add optional, client-controlled guidance for discovering Data Commons documentation through llms.txt.

The guidance directs clients to read the index for API, library, schema, dataset coverage, concept, and integration questions, then follow only relevant documentation links. Statistical queries continue to use the existing MCP tools and skills.

Configuration

Documentation guidance is disabled by default, including autopush.

• DC_ENABLE_DOCUMENTATION sets the server default and configures stdio deployments.
• HTTP clients can override it using X-DC-Enable-Documentation: true or false.
• An absent header inherits the server default.
• Values are case-insensitive and trimmed; invalid values are rejected.
• Clients should reconnect after changing their preference.

Disabling the feature omits the guidance; it does not restrict access to public documentation.

Implementation

• Keep server.md unchanged.
• Store the guidance in doc_instructions_extension.md, overridable through DC_INSTRUCTIONS_DIR.
• Prepare base-only and base-plus-documentation instructions in DCApp.
• Select instructions per client without modifying shared server state.
• Remove the documentation resource and server-side fetching. Clients fetch the index and linked pages directly.

The direct-link approach avoids resource registration and filtering because clients already need URL-reading capability to access the linked documentation.

FastMCP remains pinned to 3.4.2. The isolated private session-options dependency is documented and covered by initialization tests.

Validation

• All 50 MCP tests pass.
• HTTP tests cover both server defaults, client overrides, and concurrent-client isolation.
• Tests cover custom instructions, header validation, and headerless fallback.
• Lint, formatting, and diff checks pass.
• A temporary CLI experiment validated direct-index discovery and documentation-based example generation without general web search. The final refactored code has not yet been rerun through that CLI.

[gemini-code-assist]

Code Review

This pull request introduces the option to expose the official Data Commons documentation resource, controlled by the new DC_ENABLE_DOCUMENTATION_RESOURCE setting. When enabled, the server registers a documentation index resource and appends a routing hint to the server instructions. The feedback recommends moving the requests import from the module level to inside the registration function to defer loading and keep server startup fast when the feature is disabled.

[keyurva]

Thanks Rohit! Consider whether we need the server flag. LGTM otherwise.

[rohitkumarbhagat]

Thanks Rohit! Consider whether we need the server flag. LGTM otherwise.

Thanks Keyur. Removed server side flag.