feat(cli): add sdmx query commands under a new client group

[dwnoble]

Adds SDMX 3.0 query commands to the Data Commons CLI under a new client command group, alongside a reusable SdmxClient.

Why a client group

Per review feedback, querying observations is an end-user operation rather than an administrative one, so these commands no longer live under datacommons admin:

datacommons client sdmx-data ...
datacommons client sdmx-availability ...

The implementation also moves from the datacommons-admin package to datacommons-cli, keeping end-user functionality out of the admin package. datacommons admin sdmx is removed.

Connecting to an endpoint

An endpoint is selected in one of two ways, depending on whether its URL is publicly reachable:

Mode	Flags	Authentication
Public API (default)	none, or --url	API key
Private DCP instance	--project-id + --instance-name	Google Cloud IAM

Public API (default). Commands target https://api.datacommons.org and authenticate with an API key from --api-key or the DATACOMMONS_API_KEY environment variable. --url retargets to any reachable endpoint and accepts a bare host or a full URL (api.datacommons.org, https://api.datacommons.org, http://localhost:8080).

Private DCP instance. A DCP instance on Cloud Run is private by default and sits behind IAM, so it cannot be reached by URL alone. --project-id and --instance-name resolve its service URL from remote Terraform state in GCS and sign requests with your Google Cloud credentials.

The two modes are mutually exclusive, and the flags are validated at parse time so conflicting input fails before any network or state access.

Handling the two API path layouts

The two deployment flavors serve the SDMX API under different prefixes:

Deployment	SDMX path
Public Data Commons API	/sdmx/v3/…
DCP instance	/core/api/sdmx/v3/…

Rather than exposing this as a flag, the client picks the likely prefix for the endpoint and retries once on a 404, then reuses the discovered prefix for the rest of the session. A valid SDMX query never returns 404 (an unknown variable returns 200 with an empty result), so the status unambiguously signals the other layout. In the common case this costs no extra request.

Key Changes

• client command group (packages/datacommons-cli/datacommons_cli/client/):
  • client_cli.py: the client group and its two commands. Shared query options are factored into a single decorator, endpoint resolution is lazy so --help never touches the network, and informational output goes to stderr so stdout stays a clean CSV/JSON stream.
  • connection.py: endpoint resolution, URL normalization, API key and Google ID token auth, and the guidance shown when an endpoint rejects credentials.
  • sdmx_client.py: the SDMX HTTP client, query construction, API-root discovery, and error extraction.
• Commands:
  • sdmx-data: observations matching a variable and dimension filters, as SDMX-CSV.
  • sdmx-availability <component_id>: available values and constraints for a dimension or attribute, as SDMX-JSON Structure.
  • Shared: -v/--variable, -f/--filter (repeatable, key=value), -o/--output, --log/--no-log, --multi-entity/--no-multi-entity, --accept.
• Error reporting: 401/403 responses append guidance specific to the connection mode, pointing at --api-key/DATACOMMONS_API_KEY for the public API or gcloud auth application-default login for an instance.
• tf_utils: get_datacommons_service_url accepts an explicit TerraformStateConfig so callers outside a Click context can resolve a named instance.
• Dependencies: datacommons-cli now declares requests and google-auth directly instead of relying on them transitively.
• Documentation: rewrote the SDMX section of packages/datacommons-cli/README.md around the new group, with a command-group overview, endpoint selection guide, and refreshed cheatsheet.

Sample CLI Usage

# Public API, key from the environment:
export DATACOMMONS_API_KEY=your-api-key
datacommons client sdmx-data -v Count_Person -f observationAbout=country/USA

# Discover which provenances carry data for a variable:
datacommons client sdmx-availability provenance -v Count_Person

# Multiple filters, saved to a file:
datacommons client sdmx-data -v FinancialTrade \
    -f sourceCountry=country/FRA -f provenance=FooBarTrade -o output.csv

# A private DCP instance:
datacommons client --project-id datcom-dcp --instance-name testbed-1 \
    sdmx-data -v FinancialTrade -f sourceCountry=country/FRA

# A local development server:
datacommons client --url http://localhost:8080 sdmx-data -v Count_Person -f observationAbout=country/USA

Sample Responses

sdmx-data (SDMX-CSV)

STRUCTURE,STRUCTURE_ID,ACTION,variableMeasured,observationAbout,unit,measurementMethod,observationPeriod,provenance,TIME_PERIOD,OBS_VALUE,scalingFactor,facetId
dataflow,DC:DF_OBS(1.0.0),I,Count_Person,country/USA,NotApplicable,CensusACS5yrSurvey,NotApplicable,dc/base/CensusACS5YearSurvey,2024,3.34922499E8,,10169881228856630405
dataflow,DC:DF_OBS(1.0.0),I,Count_Person,country/USA,NotApplicable,CensusACS5yrSurvey,NotApplicable,dc/base/CensusACS5YearSurvey,2023,3.3238754E8,,10169881228856630405
dataflow,DC:DF_OBS(1.0.0),I,Count_Person,country/USA,NotApplicable,WikidataPopulation,NotApplicable,dc/base/WikidataPopulation,2021,3.322782E8,,12850099660527362240

sdmx-availability (SDMX-JSON Structure)

{
  "meta": {
    "schema": "https://json.sdmx.org/2.0.0/sdmx-json-structure-schema.json",
    "id": "DF_OBS_AVAILABILITY",
    "prepared": "2026-09-18T23:43:05Z",
    "sender": {
      "id": "DC"
    }
  },
  "data": {
    "dataConstraints": [
      {
        "id": "DF_OBS_AVAILABILITY",
        "agencyID": "DC",
        "version": "1.0.0",
        "name": "Available DF_OBS data",
        "role": "Actual",
        "cubeRegions": [
          {
            "include": true,
            "keyValues": [
              {
                "id": "provenance",
                "include": true,
                "values": [
                  { "value": "dc/base/CensusACS5YearSurvey" },
                  { "value": "dc/base/WikidataPopulation" }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}

Programmatic Usage

from datacommons_cli.client import ConnectionOptions, SdmxClient, resolve_connection

connection = resolve_connection(ConnectionOptions(api_key="your-api-key"))
client = SdmxClient(connection)

csv_text = client.get_data("Count_Person", {"observationAbout": "country/USA"})
availability = client.get_availability("provenance", "Count_Person")

Verification

• Live, against api.datacommons.org: both commands, the default endpoint and --url in bare-host and full-URL forms, --api-key and DATACOMMONS_API_KEY, multi-value filters, -o file output, and the 401 guidance when no key is supplied.
• Live, API-root discovery: a connection deliberately configured with the wrong preferred prefix recovered on the 404, then reused the discovered prefix for subsequent queries.
• Unit: 57 tests in packages/datacommons-cli/tests/client/ covering URL normalization, flag validation, both connection modes, query construction, header handling, error extraction, root discovery and caching, and CLI behavior. Full suite: 231 passing.
• Integration: tests/integration/suites/03_serving_api/test_sdmx.py updated to the new commands and Connection-based client.
• Ruff check and format pass on all touched files.

[gemini-code-assist]

Code Review

This pull request renames the namespace variable to instance_name across Terraform modules and the admin CLI, while maintaining backward compatibility. It also introduces a new sdmx command group to query custom SDMX v3 observation APIs and enhances tf_utils.py to support fetching Terraform outputs directly from GCS remote state. Feedback on these changes focuses on improving the manual ADC loading logic for Windows compatibility and standard precedence, initializing the GCS client with the target project ID for robustness, and refining error handling when parsing non-OK HTTP responses.

[clincoln8]

Do we want this to be under "admin"? Or should it be like datcom-cli api sdmx? or just datcom-cli sdmx?

These docs are a bit outdated, but capture the concept of things that are end-usery vs purely admin:
https://docs.google.com/document/d/1SELfRfNKnUZEvjZdRDN6V8feDhM848e6nnS7HqEPj8k/edit?resourcekey=0-sgcFjpIJQBftf8ssZxu4wg&tab=t.t7xn652lw7rd

https://docs.google.com/document/d/1SELfRfNKnUZEvjZdRDN6V8feDhM848e6nnS7HqEPj8k/edit?resourcekey=0-sgcFjpIJQBftf8ssZxu4wg&tab=t.ditqyzpaettb#heading=h.jlr05xdw2h1t

Can we create a cli where we "connect" to an instance and then use cli to issue api calls? Like

datacommons-cli api set-default --url=dcp.datacommons.org
datacommons-cli api sdmx [params] [--url (optional, required if default is not set)]

In that case, can/should we move the datacommons-client into this repo?

With all that in mind, do we still want to addsdmx to admin as a short-term thing?