perf: fast feature-frame builds and response-body release in the chunked OGC path

[thodson-usgs]

Summary

A design review of the async parallel chunking stack (planning → fan-out → pagination → shaping/combining → transport), with each candidate improvement evaluated experimentally against real Water Data queries. Two changes survived the bar; everything else is reported below with the numbers that killed it. A post-review cleanup pass (4 parallel review agents: reuse / simplification / efficiency / altitude) converged the fast-path code and made it faster still, and a follow-up pass trimmed comments to current-code constraints and deprecated the one dead function a package-wide sweep found.

Accepted (this PR):

1. perf(shaping): vectorized feature-frame fast paths. Flat feature properties (every Water Data / NGWMN collection) build via pd.DataFrame instead of pd.json_normalize (~2.5x — nested values are detected by scanning only object-dtype columns after the cheap build); all-2D-point pages build geometry via one vectorized geopandas.points_from_xy call over two flat coordinate lists instead of per-feature GeoDataFrame.from_features (~4.8x). Nested properties, non-point/malformed geometry, or a geometry property column all fall back to the previous path. This CPU runs on the fan-out's event loop, so it sits on every chunked call's critical path.
2. perf(transport): free response bodies once parsed. Every per-chunk aggregate stored for resume shared its first page's decompressed body, so a ~1-page-per-chunk fan-out held the whole download in RAM until the call finished. paginate likewise pinned its first page's body for the whole walk. Aggregates now carry an empty body via a single named helper (combining._drop_body); status/headers/URL/elapsed unchanged. Behavior change (in NEWS): the aggregate httpx.Response behind a call's metadata and FanOutInterrupted.partial_response no longer carries body bytes — it was previously one arbitrary page's fragment, not the query's data.
3. chore(utils): deprecate format_datetime. A dead-function sweep over the whole package found exactly one orphan: it shaped qw-service responses, lost its last caller in 491eb5c3, and appears in no docs or demos. Public name, so it warns through _deprecation.warn_deprecated with a 2027-08-22 horizon instead of vanishing.

Two cleanup passes (each: 4 parallel review agents across reuse / simplification / efficiency / altitude) also landed on the branch. The second pass replaced the direct shapely.points call with geopandas.points_from_xy — the idiom nwis already uses, retiring the package's only direct shapely import — and gated _properties_frame's Python dict scan behind infer_dtype, worth 2.3x on that helper under pandas 2 (supported by this package, and where every string column is object dtype).

Efficiency benchmarks (current branch head)

Measurement	Baseline	This PR
Frame build, 100k rows, spatial (offline MRE)	554 ms	116 ms (4.8x)
Frame build, 100k rows, plain (offline MRE)	225 ms	86 ms (2.6x)
Full-call CPU path, 93k rows / 12 chunks (recorded-response replay, zero latency)	0.90 s	0.46 s (−49%; unfanned −54%)
End-to-end wall, live cache-hot 12-chunk get_daily, median of 7 trials/arm	1.33 s	0.95 s (−29%)
End-to-end wall, replay with recorded latencies	—	−2 to −6%
Peak heap, fanned-12 replay, sequential (zero-latency)	168.1 MB	89.8 MB (−47%)
Peak heap, fanned-12 replay, concurrent arrival	164 MB	133 MB (−19%)

Correctness: outputs verified identical (assert_frame_equal) against recorded API responses for full/missing/mixed geometry, polygons, nested properties, 3-D coordinates, single-feature pages, and a properties-level id column; full suite (1004 tests) and mypy --strict pass.

Evaluated and rejected (with numbers)

• HTTP/2 (server negotiates h2 via ALPN): consistently slower — per-request median 0.33 s (h1) → 0.42 s (h2), in both pool-of-32 and single-multiplexed-connection modes, over 15 interleaved cold-window trials.
• orjson decode (the practical "Rust" option): −15 pp more CPU-path on top of this PR, but ≤1% end-to-end with realistic latencies — doesn't justify a new dependency. (A custom Rust/C extension is strictly worse: pure-Python package, conda-forge feedstock, and the remaining CPU tail is already sub-second per 100k rows.)
• Columnar frame construction (dict-of-lists instead of DataFrame(records)): 1.08x best case — pandas 3's record path is already good.
• Pipelined page prefetch (fetch page N+1 while building frame N): the inter-page CPU gap is ~0.4 s per boundary on 40 MB pages (~3% of a 2-page pull, less after this PR) — not worth the cancellation complexity.
• Speculative parallel pagination within a chunk: impossible — cursor-based pagination (cursor=<opaque>; numberMatched absent for daily), so page N+1's URL cannot be constructed ahead of page N.
• Persistent client/event loop across calls: pre-first-request overhead measured at 5–6 ms; nothing to save.
• Streaming JSON parse for the transient decode spike (a 40 MB page peaks ~150 MB during its own parse — the residual unfanned peak): incremental parsers are slower; the spike is bounded by one page.
• Fan-out dispatch layer: no change needed — 12 parallel chunks complete within ~50 ms of the slowest single request.

MRE

Offline, deterministic, no API key or quota (compares against the pre-change implementation inlined):

import time
import pandas as pd
from pandas.testing import assert_frame_equal
import dataretrieval.ogc.shaping as shaping

features = [
    {
        "id": f"id-{i}",
        "properties": {
            "monitoring_location_id": f"USGS-{i % 500:08d}",
            "parameter_code": "00060", "statistic_id": "00003",
            "time": f"2020-{1 + i % 12:02d}-{1 + i % 28:02d}",
            "value": str(i), "approval_status": "Approved", "qualifier": None,
        },
        "geometry": {"type": "Point", "coordinates": [-77.0 - i * 1e-6, 38.9]},
    }
    for i in range(100_000)
]

def old_spatial(feats):  # pre-change implementation
    df = shaping._geo_feature_frame(feats)
    df["id"] = [f.get("id") for f in feats]
    return df[["id"] + [c for c in df.columns if c != "id"]]

t0 = time.perf_counter(); old = old_spatial(features); t_old = time.perf_counter() - t0
t0 = time.perf_counter(); new = shaping._spatial_feature_frame(features); t_new = time.perf_counter() - t0
assert_frame_equal(old, new)
print(f"from_features={t_old*1000:.0f}ms vectorized={t_new*1000:.0f}ms ({t_old/t_new:.1f}x, identical output)")

Typical output on an M-series laptop: from_features=554ms vectorized=116ms (4.8x, identical output).

Methodology notes

• Live A/B trials interleaved arm order and rotated distinct time windows (the API caches by data window).
• Replay benchmarks drive the real fan-out/pagination machinery over 14 recorded API responses (152 MB) via httpx.MockTransport, sleeping each request's recorded wall time — exact pairing, no quota, no cache noise.
• Memory measured with tracemalloc peak, one arm per process, response bodies freshly allocated per request.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UKweWmHg8cu1WuJ17kiHdh