Make failures actionable and harden validation, APIs, and CI - #387
Open
thodson-usgs wants to merge 10 commits into
Open
Make failures actionable and harden validation, APIs, and CI#387thodson-usgs wants to merge 10 commits into
thodson-usgs wants to merge 10 commits into
Conversation
thodson-usgs
force-pushed
the
docs/refresh-agents-context
branch
from
August 20, 2026 18:36
595a8ef to
b0fd59d
Compare
thodson-usgs
added a commit
to thodson-usgs/dataretrieval-python
that referenced
this pull request
Aug 20, 2026
Four message defects found reviewing DOI-USGS#387, all of the shape the new AGENTS.md rule names: following the remedy literally must produce a working call. nldi._validate_feature_source_comid checked the feature pair before the origin conflict, so `get_features(comid=13294314, feature_id='X')` was told "Pass both, e.g. feature_source='WQP'" -- and the corrected call then raised "Provide exactly one of comid or feature_source". Two round trips for one mistake, in both directions of the pair. A comid now rejects either half up front, and both checks offer the same two ways forward from one shared hint. nwdc._resolve_locations closed its empty-value complaint with "(exactly one of state, county, or huc must be given)" -- but `state=[]` did give exactly one, so the parenthetical sent the caller to change a different selector than the one at fault. _nwdc_error_detail is annotated `str | None` and returned `body["detail"]` unguarded; a validation envelope spells that as a list of error objects, which would have been interpolated verbatim into the message. It also appended a period to a detail that already ended in one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Point agents at the shared vocabulary and authoritative dependency map, replace stale file inventories with placement rules, document the current quality gates, and correct the definition of a configuration setting.
Centralize the recurring missing, grouped, alternative, and conflicting argument checks. Adopt them across adapters, make every rejection name an executable remedy, prevent NLDI and nearest-observation corrections from producing silent wrong answers, and keep the associated NLDI tests offline.
Expose the OGC engine's existing max_rows limit through get_cql so callers can cap total results rather than accidentally turning a small page size into hundreds of requests.
Enable branch coverage, close behaviorally meaningful gaps, exclude only generated or environment-unreachable paths, and set the threshold at the measured value so future regressions fail without encouraging hollow tests.
Correct stale service and collection documentation, make the date-formatting hint parameter describe rather than imply enforcement, and document how to interpret the coverage threshold when local platform skips differ from CI.
Move the coverage gate into the existing test matrix, keep Windows informational where POSIX-only tests skip, and exclude version-conditional branches so every supported Linux interpreter measures the same ratchet.
Add the permissive any-of shape and the closed-vocabulary remedy extension needed by legacy NWIS and CQL callers, then replace the remaining hand-written recurring checks without changing their caller-facing intent.
Record the public behavior changes in NEWS and remove comments that merely restated measured values or nearby code while retaining rationale and constraints.
Add the five inhabited US territories to state normalization, allow Water Data and NGWMN queries the upstream services already support, and make endpoint-specific remedies use arguments their public getter accepts.
Consolidate shared message rendering and context handling, finish adoption in NLDI, NWIS, states, and ratings, return validated selections instead of re-deriving them, and standardize malformed argument values on ValueError while improving the NWIS ratings replacement guidance.
thodson-usgs
force-pushed
the
docs/refresh-agents-context
branch
from
August 21, 2026 14:23
a939ca4 to
5d7ff50
Compare
thodson-usgs
marked this pull request as ready for review
August 21, 2026 14:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR started as an agent-documentation refresh and an error-message cleanup. Exercising each proposed remedy exposed several cases where following the message caused a second failure or, worse, returned a plausible but wrong result. The scope now covers the resulting validation, correctness, API, geographic-code, test, and CI changes as one caller-correctness pass.
Actionable argument validation
dataretrieval._validationnow owns the recurring argument-check shapes and their message vocabulary:require_one_ofrequire_argumentrequire_togetherrequire_any_ofrequire_exactly_one(name, value)reject_togetherThe helpers are adopted across NLDI, NWDC, NWIS, WQP, OGC dates, Water Data CQL/reference/nearest, and state normalization. Messages identify the caller's parameter spelling, state the problem, and name a correction the caller can execute.
The ratings
file_typevalidator remains local deliberately: it validates a collection and reports every invalid member, whilerequire_one_ofvalidates one scalar. Their shared formatting is centralized throughrender_options; adding a second collection-helper interface for one caller would be speculative.Compatibility note
Malformed argument values now consistently raise
ValueError. The deprecatednwis.query_waterdata,query_waterservices, andget_recordpaths therefore no longer use their historicTypeErrorfor missing filters, incomplete bounding boxes, or unknown services.TypeErrorremains for genuinely mistyped arguments such as a non-stringsitesvalue.Correctness fixes found by executing the remedies
data_sourcebefore constructing a navigation URL. A missing source can no longer become a literal/Nonepath whose 200 response looks like a genuine empty result.waterdata.get_nearest_continuousaddstimeandmonitoring_location_idto an explicitpropertieslist because both are required to select one row per target and monitoring location. This is a documented returned-column-shape change.nwdcempty selectors and error envelopes now produce actionable text rather than advice that repeats the same failure or prints a raw validation-object list.query_waterdata("ratings", ...)route now points towaterdata.get_ratings, including the dict return shape and feature-ID keys callers need to use it.nldi._query_nldistill turns a 200 non-JSON body into an empty GeoDataFrame. That is an existing documented exception, not changed here, and is called out inAGENTS.mdso it is not mistaken for the general error policy.Additional API and data behavior
waterdata.get_cql(..., max_rows=N)exposes the OGC engine's existing total-row cap.limitremains page size; a smalllimitalone can increase request count rather than cap results.US:<FIPS>forms normalize alike for adapters whose upstream service supports them.countries, and its hand-maintained reference docstring is contract-tested against that vocabulary.NEWS.mdrecords the public behavior and exception changes.Coverage and CI
Coverage is now a branch-coverage ratchet rather than an informational number. The branch adds behavior-focused tests for dispatch, malformed response envelopes, NLDI navigation, nearest-row grouping, configuration fallbacks, date handling, pagination, and previously uncovered public getters.
The ratchet runs in the existing OS/Python test matrix, blocks on every non-Windows leg, and reports informationally on Windows where POSIX-only skips measure a smaller suite. Version-conditional and environment-unreachable import arms are excluded explicitly so Python 3.10, 3.13, and 3.14 enforce the same measurement. Complexity and dependency-direction jobs remain fast and independent.
Agent and contributor documentation
AGENTS.mdnow starts fromCONTEXT.md, explains placement rules rather than carrying a stale file inventory, names.importlinteras the authoritative dependency map, documents the current test/tooling gates, and records the executable-remedy policy.CONTEXT.mdcorrects the definition of a setting: a public keyword such asssl_checkis not automatically part of configuration resolution.CONTRIBUTING.mddocuments branch coverage, platform-dependent local measurements, complexity ratchets, import-contract ownership, and the periodic structural-health sweep.Validation
ruff check .andruff format --check .mypyin strict modelint-imports: 8 contracts keptxenonandcomplexipy