From 01951136991de877e3af90b2b09c8d91f7272c37 Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Wed, 1 Jul 2026 09:07:12 -0500 Subject: [PATCH 01/13] feat(waterdata): add chunk_granularity to control OGC chunk fan-out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The OGC getters chunk a multi-value request only as far as the server's ~8 KB URL limit forces — the fewest sub-requests. But because every sub-request paginates, splitting a large result further is usually quota-neutral, so that conservative default can be needlessly coarse: ten states pulled as one under-limit request page just as many times as ten per-state requests would. Add `waterdata.chunk_granularity(level)`, a context manager that lets a caller who knows their pull is large opt into a finer split — trading the same pages for more, smaller sub-requests (smoother progress, more even concurrency, a smaller unit of retry/resume). The level is "low", "medium", or "high" (typed as `GranularityLevel`, a Literal, so a type checker rejects anything else; an invalid string raises ValueError at the `with`). Each level caps how many sub-chunks a multi-value argument is split into, derived from the default fan-out concurrency (`API_USGS_CONCURRENT`): high = the full width, medium a quarter, low a sixteenth (32 / 8 / 2 by default). Capping the aggressive end at the concurrency width bounds the blast radius so an accidental "high" on a huge list can't explode into thousands of sub-requests. There is no "off" level — not entering the block is off. It is a scoped `with` block, not an env var, because the library can't tell in advance whether a query is large (a short-window query might fit one page, where extra chunks only burn quota). Implementation: a soft `ChunkPlan._refine` pass runs after the hard byte pass; it only ever splits further, so the url_limit invariant holds and it never raises. The resolved per-axis cap is read from a contextvar (Ambient) set by the context manager at plan-construction time. Exported (with the `GranularityLevel` type) from `dataretrieval.waterdata` and the top-level `dataretrieval` package. Co-Authored-By: Claude Opus 4.8 (1M context) --- NEWS.md | 2 + dataretrieval/__init__.py | 8 + dataretrieval/ogc/chunking.py | 147 ++++++++++++++++- dataretrieval/ogc/planning.py | 97 +++++++++-- dataretrieval/waterdata/__init__.py | 3 + docs/source/userguide/errors.rst | 35 ++++ tests/utils_test.py | 18 ++ tests/waterdata_chunking_test.py | 248 ++++++++++++++++++++++++++++ 8 files changed, 542 insertions(+), 16 deletions(-) diff --git a/NEWS.md b/NEWS.md index a86a0314..c2d4b259 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,5 @@ +**07/01/2026:** Added `waterdata.chunk_granularity(...)` — a context manager to control how finely the OGC `waterdata` getters split multi-value requests. By default the getters chunk only as much as the server's ~8 KB URL limit forces (the fewest sub-requests). But because every sub-request paginates, splitting a large result further is usually quota-neutral, so a caller who *knows* their pull is large can opt into a finer split for smoother progress, more even concurrency, and a smaller unit of retry/resume: `with waterdata.chunk_granularity("high"): df, md = waterdata.get_daily(monitoring_location_id=many_sites)`. The level is one of `"low"`, `"medium"`, or `"high"` — typed as `waterdata.GranularityLevel` (a `typing.Literal`), so a type checker rejects any other value and an invalid string raises `ValueError` at the `with`. Each level caps how many sub-chunks a multi-value argument is split into — `"low"` / `"medium"` / `"high"` cap at 2 / 8 / 32. That ceiling is a granularity constant, deliberately independent of the fan-out concurrency (`API_USGS_CONCURRENT`): how finely a query splits is orthogonal to how many sub-requests run at once. Capping the aggressive end at 32 bounds the blast radius — an accidental `"high"` on a huge list can't explode into thousands of sub-requests. There is no "off" level: not entering the block *is* off. It is deliberately a scoped `with` block rather than an automatic behavior or an environment variable, since the library can't tell in advance whether a query is large — a short-window query might fit in a single page, where extra chunks would only burn quota. Also exported at the top level as `dataretrieval.chunk_granularity`. + **06/23/2026:** **Breaking change (1.2.0):** the minimum supported Python is now **3.10** (`requires-python = ">=3.10"`). 3.9 support was already effectively broken — the `waterdata` module's dependencies (`anyio`, the test stack) require 3.10+, and the `waterdata` test modules already skipped on <3.10. `anyio` is now declared as a direct dependency (it is imported directly by `waterdata`), and the CI/ruff/mypy targets move to 3.10. Also fully removed the deprecated `variable_info` metadata property: the `NWIS_Metadata` override only warned and returned `None` (it relied on the defunct `get_pmcodes`), and the `BaseMetadata` abstract is gone too since nothing implemented it — accessing `.variable_info` now raises `AttributeError`. `site_info` is unaffected. **06/23/2026:** **Breaking change (1.2.0):** removed the `nadp` module and the deprecated `samples` module ahead of the 1.2.0 release. `nadp` was deprecated on 05/01/2026 — NADP is not a USGS data source, so retrieve NADP data directly from https://nadp.slh.wisc.edu/. The `samples.get_usgs_samples` shim (a deprecated forward to the modern getter) is gone; use `waterdata.get_samples()` instead. `import dataretrieval.nadp` / `import dataretrieval.samples` now raise `ModuleNotFoundError`. diff --git a/dataretrieval/__init__.py b/dataretrieval/__init__.py index c9df1c45..ec79a724 100644 --- a/dataretrieval/__init__.py +++ b/dataretrieval/__init__.py @@ -44,6 +44,11 @@ URLTooLong, ) +# Chunk-granularity control (a context manager) and its level type. Defined with +# the chunker in ``dataretrieval.ogc.chunking``; surfaced here for a stable +# public path ``from dataretrieval import chunk_granularity``. +from dataretrieval.ogc.chunking import GranularityLevel, chunk_granularity + # Resumable chunk-interruption exceptions. They are defined in # ``dataretrieval.ogc.interruptions`` rather than ``dataretrieval.exceptions`` # because they carry pandas/httpx state and a resumable ``ChunkedCall`` handle, @@ -93,5 +98,8 @@ "ChunkInterrupted", "QuotaExhausted", "ServiceInterrupted", + # chunk-granularity control (defined in ogc.chunking) + "chunk_granularity", + "GranularityLevel", "__version__", ] diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index 00d2766c..5bbcec56 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -9,6 +9,12 @@ cartesian product of chunks. Requests that already fit get a trivial single-step plan — ``ChunkedCall`` has one code path either way. +Granularity: the planner is conservative by default — it splits only as far as +the byte limit forces. A caller who knows their result is large can opt into a +finer split via the ``chunk_granularity`` context manager +(``"low"`` / ``"medium"`` / ``"high"``); the resolved cap drives +:meth:`ChunkPlan._refine`. See ``chunk_granularity`` for the why and the when. + This module owns the *execution* half — the event loop and bounded concurrency that drive a plan to completion (``ChunkedCall``) plus the public ``multi_value_chunked`` decorator. The neighboring concerns live in @@ -69,8 +75,9 @@ import functools import os from collections.abc import Callable, Iterator +from contextlib import contextmanager from contextvars import copy_context -from typing import Any, cast +from typing import Any, Literal, cast, get_args import httpx import pandas as pd @@ -172,6 +179,130 @@ def get_active_client() -> httpx.AsyncClient | None: return _chunked_client.get() +# Chunk-granularity dial: opt-in to fan a query out *more finely* than the byte +# limit alone requires. Scoped to a ``with chunk_granularity(...):`` block (a +# ContextVar), deliberately NOT an env var (see :func:`chunk_granularity` for +# why). The ambient holds the resolved cap on sub-chunks per axis; ``0`` (the +# default, outside any block) means "chunk only as much as the byte limit needs". +_granularity: Ambient[int] = Ambient("ogc_chunk_granularity", 0) + +#: The three accepted granularity levels, as a typing ``Literal`` so a type +#: checker rejects any other value at the call site. +GranularityLevel = Literal["low", "medium", "high"] + +#: Valid levels derived from the type, so ``GranularityLevel`` stays the single +#: source of truth for what ``_resolve_granularity`` accepts (mirrors the +#: ``get_args``-based ``_VALID_ON_TIE`` / ``_VALID_FILE_TYPES`` in sibling +#: modules). +_VALID_LEVELS: tuple[GranularityLevel, ...] = get_args(GranularityLevel) + +# Granularity's own ceiling on sub-chunks per axis — deliberately NOT tied to the +# concurrency default: fan-out *volume* (how many sub-requests a query becomes) +# is orthogonal to how many run at once (``API_USGS_CONCURRENT``). 32 is a sane +# single-axis ceiling that also bounds the blast radius of an accidental +# ``"high"`` on a very long list; the milder levels are a quarter and a +# sixteenth of it (the three are spaced 4x apart). +_GRANULARITY_MAX_CHUNKS = 32 +_GRANULARITY_LEVELS: dict[str, int] = { + "low": _GRANULARITY_MAX_CHUNKS // 16, # 2 + "medium": _GRANULARITY_MAX_CHUNKS // 4, # 8 + "high": _GRANULARITY_MAX_CHUNKS, # 32 +} + + +def _resolve_granularity(level: GranularityLevel) -> int: + """ + Map a granularity level name to its per-axis sub-chunk cap. + + Parameters + ---------- + level : {"low", "medium", "high"} + The user-supplied level. + + Returns + ------- + int + The maximum sub-chunks per multi-value axis for that level + (``2`` / ``8`` / ``32``). + + Raises + ------ + ValueError + If ``level`` is anything other than ``"low"``, ``"medium"``, or + ``"high"`` — raised eagerly (at the ``with`` statement) so a typo fails + loudly rather than silently doing nothing. + """ + if level not in _VALID_LEVELS: + raise ValueError( + f"chunk_granularity level must be one of {_VALID_LEVELS}; got {level!r}." + ) + return _GRANULARITY_LEVELS[level] + + +@contextmanager +def chunk_granularity(level: GranularityLevel) -> Iterator[None]: + """ + Scope how finely the OGC getters chunk multi-value requests. + + By default the Water Data / NGWMN getters chunk a request only as much as + the server's ~8 KB URL-byte limit forces — the fewest sub-requests that + fit. That is the safe default, but it can be *needlessly* conservative: + because every sub-request paginates, splitting a large result further is + usually quota-neutral (ten states pulled as one under-limit request page + just as many times as ten per-state requests would). This context manager + lets a caller who *knows* their pull is large ask for that finer split — + trading the same pages for more, smaller sub-requests, which gives smoother + progress, more even concurrency, and a smaller unit of retry/resume. + + Because the library can't tell in advance whether a query is large (ten + states over a short window might fit in a single page, where extra chunks + would only burn quota), this is a *deliberate* per-call knob rather than an + automatic behavior or a process-wide environment variable — scoping it to a + ``with`` block keeps an aggressive setting from leaking into unrelated calls + and accidentally spending quota. Outside any block the getters use the + conservative default; there is no "off" level because *not* entering the + block is off. Only the OGC getters (Water Data, NGWMN) read this; wrapping a + legacy NWIS call in the block is a harmless no-op. + + Parameters + ---------- + level : {"low", "medium", "high"} + How aggressively to chunk within the block. Each level caps how many + sub-chunks a single multi-value argument is split into — ``2`` / ``8`` / + ``32`` for ``"low"`` / ``"medium"`` / ``"high"`` — or one value per + sub-request once the argument has fewer values than the cap. The ceiling + is fixed (it is *not* tied to ``API_USGS_CONCURRENT``: how finely a query + splits is orthogonal to how many sub-requests run at once); capping + ``"high"`` at ``32`` keeps an accidental aggressive level on a very long + list from exploding into thousands of sub-requests. (With several + multi-value arguments the per-argument counts still multiply.) + + Yields + ------ + None + + Raises + ------ + ValueError + If ``level`` isn't ``"low"``, ``"medium"``, or ``"high"`` — raised on + ``with`` entry, before any request is issued. + + Examples + -------- + >>> from dataretrieval import waterdata + >>> with waterdata.chunk_granularity("high"): + ... df, md = waterdata.get_daily( + ... monitoring_location_id=many_sites, parameter_code="00060" + ... ) # doctest: +SKIP + + See Also + -------- + ChunkPlan._refine : the planning-side effect of the level. + """ + with _granularity(_resolve_granularity(level)): + yield + + class ChunkedCall: """ Stateful handle for a chunked call. @@ -591,8 +722,9 @@ def multi_value_chunked( ``async def fetch(args) -> (df, response)``, and drives it to completion via :meth:`ChunkedCall.resume`. The plan splits multi-value list params and the cql-text filter so each sub-request URL fits the - byte limit; an already-fitting request is a one-step plan. See the - module docstring for the concurrency model. + byte limit; an already-fitting request is a one-step plan, unless an + active :func:`chunk_granularity` block asks the plan to fan out more + finely. See the module docstring for the concurrency model. Parameters ---------- @@ -636,7 +768,14 @@ def wrapper( finalize: _Finalize = _passthrough_result, ) -> tuple[pd.DataFrame, Any]: limit = _OGC_URL_BYTE_LIMIT if url_limit is None else url_limit - plan = ChunkPlan(args, build_request, limit) + # Read the granularity dial from the ambient set by + # ``chunk_granularity`` (0 = off outside any such block; otherwise the + # per-axis sub-chunk cap). It only affects *planning*, done here up + # front, so a later resume — which re-issues the already-planned + # sub-requests — needs no snapshot. + plan = ChunkPlan( + args, build_request, limit, max_chunks_per_axis=_granularity.get() + ) retry_policy = RetryPolicy.from_env() # The concurrency cap is resolved inside ``resume()`` from # ``API_USGS_CONCURRENT``; ``1`` is a sequential gather, diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index 68c337ae..700a0e85 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -288,6 +288,21 @@ def _extract_axes(args: dict[str, Any]) -> list[_Axis]: return axes +def _split_at(chunks: list[list[str]], idx: int) -> None: + """Replace ``chunks[idx]`` in place with its two contiguous halves. + + The single primitive both planning passes use to fan an axis out. It + preserves the partition invariants every consumer relies on: *coverage* + (each atom survives, exactly once) and *contiguous, deterministic order* + (resume and :meth:`ChunkPlan.iter_sub_args` depend on it). Kept in one + place so those invariants can't drift between :meth:`ChunkPlan._plan` + (byte-driven) and :meth:`ChunkPlan._refine` (granularity-driven). + """ + chunk = chunks[idx] + mid = len(chunk) // 2 + chunks[idx : idx + 1] = [chunk[:mid], chunk[mid:]] + + class ChunkPlan: """ Strategy for issuing one user-level request as a sequence of @@ -312,7 +327,17 @@ class ChunkPlan: Factory that turns a kwargs dict into a sized httpx request, e.g. ``_construct_api_requests``. url_limit : int - Byte budget for the request (URL + body). + Byte budget for the request (URL + body) — a hard ceiling every + sub-request must fit. + max_chunks_per_axis : int, optional + Soft cap on sub-chunks per multi-value axis (default ``0`` = off). + ``0`` chunks only as much as ``url_limit`` requires — the most + conservative plan, fewest sub-requests. A positive cap fans each axis + out into up to ``min(len(atoms), max_chunks_per_axis)`` pieces (never + fewer than the byte budget already forces), so a large multi-page pull + is issued as more, smaller sub-requests. Set from the resolved + :func:`~dataretrieval.ogc.chunking.chunk_granularity` level; see + :meth:`_refine`. Attributes ---------- @@ -344,6 +369,7 @@ def __init__( args: dict[str, Any], build_request: Callable[..., httpx.Request], url_limit: int, + max_chunks_per_axis: int = 0, ) -> None: self.args = args self.axes: list[_Axis] = [] @@ -352,10 +378,10 @@ def __init__( axes = _extract_axes(args) if not axes: - # No chunkable axis: nothing to split. If the single request fits, - # run it verbatim (the common passthrough). ``_safe_request_bytes`` - # treats an un-constructable URL (httpx.InvalidURL, > 64 KB) as over - # budget. + # No chunkable axis: nothing to split, and ``granularity`` has + # nothing to act on either. If the single request fits, run it + # verbatim (the common passthrough). ``_safe_request_bytes`` treats + # an un-constructable URL (httpx.InvalidURL, > 64 KB) as over budget. if _safe_request_bytes(build_request, args, url_limit) <= url_limit: return # Over budget. A filter the chunker doesn't manage — cql-json — is @@ -388,14 +414,29 @@ def __init__( except httpx.InvalidURL: initial_request = None + fits = False if initial_request is not None: self.canonical_url = str(initial_request.url) - if _request_bytes(initial_request) <= url_limit: - return + fits = _request_bytes(initial_request) <= url_limit + + # A request that already fits and hasn't opted into finer chunking is + # the common passthrough: leave ``axes``/``chunks`` empty so + # ``total == 1`` and ``iter_sub_args`` yields the original args + # verbatim. Only when ``max_chunks_per_axis`` asks for extra fan-out do + # we set the axes up to be refined below. + if fits and max_chunks_per_axis <= 0: + return self.axes = axes self.chunks = {axis.arg_key: [list(axis.atoms)] for axis in axes} - self._plan(build_request, url_limit) + if not fits: + # Hard pass: greedy-halve until every worst-case sub-request fits + # the byte budget (may raise ``Unchunkable``). + self._plan(build_request, url_limit) + # Soft pass: optionally split further than the byte budget requires. + # Purely additive — never re-raises, and the byte budget stays + # satisfied; a no-op at ``max_chunks_per_axis <= 0``. + self._refine(max_chunks_per_axis) if self.canonical_url is None: # Original URL was un-constructable (httpx.InvalidURL); fall @@ -447,10 +488,42 @@ def _plan( f"sub-request). Reduce input sizes, shorten or simplify " f"the filter, or split the call manually." ) - axis_chunks = self.chunks[biggest_axis.arg_key] - chunk = axis_chunks[biggest_idx] - mid = len(chunk) // 2 - axis_chunks[biggest_idx : biggest_idx + 1] = [chunk[:mid], chunk[mid:]] + _split_at(self.chunks[biggest_axis.arg_key], biggest_idx) + + def _refine(self, max_chunks_per_axis: int) -> None: + """ + Fan each axis out more finely than the byte budget alone requires — + the granularity dial (see + :func:`~dataretrieval.ogc.chunking.chunk_granularity` for why a caller + would want this). + + Each axis is split until it holds at least + ``min(len(atoms), max_chunks_per_axis)`` chunks — up to the cap, or one + atom per chunk for a shorter axis. Purely additive — only ever *splits* + existing chunks, so the byte pass's work and the ``url_limit`` invariant + are both preserved (an axis the byte pass already split past the cap is + left alone), and it never raises. A no-op at ``max_chunks_per_axis <= 0``. + + Parameters + ---------- + max_chunks_per_axis : int + Soft cap on sub-chunks per axis (``0`` = off), the resolved + granularity level. A shorter axis simply saturates at one atom per + chunk. + """ + if max_chunks_per_axis <= 0: + return + for axis in self.axes: + chunks = self.chunks[axis.arg_key] + target = min(len(axis.atoms), max_chunks_per_axis) + # ``target <= len(atoms)`` guarantees a splittable chunk each pass, so + # this reaches exactly ``target`` and terminates. Split the chunk with + # the most *atoms* (``_plan`` splits by *bytes*): here we even out + # cardinality for smooth fan-out, not URL size. Ties take the lowest + # index, keeping the split deterministic. + while len(chunks) < target: + idx, _ = max(enumerate(chunks), key=lambda kv: len(kv[1])) + _split_at(chunks, idx) def _worst_case_args(self) -> dict[str, Any]: """ diff --git a/dataretrieval/waterdata/__init__.py b/dataretrieval/waterdata/__init__.py index 99b6e178..d70a3d74 100644 --- a/dataretrieval/waterdata/__init__.py +++ b/dataretrieval/waterdata/__init__.py @@ -9,6 +9,7 @@ from __future__ import annotations +from dataretrieval.ogc.chunking import GranularityLevel, chunk_granularity from dataretrieval.ogc.filters import FILTER_LANG # Public API exports @@ -50,6 +51,8 @@ "PROFILE_LOOKUP", "SERVICES", "WATERDATA_SERVICES", + "GranularityLevel", + "chunk_granularity", "get_channel", "get_codes", "get_combined_metadata", diff --git a/docs/source/userguide/errors.rst b/docs/source/userguide/errors.rst index e2dc3ef1..10b1a2c7 100644 --- a/docs/source/userguide/errors.rst +++ b/docs/source/userguide/errors.rst @@ -96,6 +96,41 @@ condition clears -- only the unfinished sub-requests are re-issued. except ChunkInterrupted as again: exc = again +Chunk a large request more finely +================================= + +By default the getters split an over-large request only as much as the +server's ~8 KB URL limit forces -- the fewest sub-requests. Because each +sub-request paginates, splitting a large result further is usually +quota-neutral (ten states pulled as one under-limit request page just as many +times as ten per-state requests would), so if you *know* your pull is large +you can ask for a finer split with ``chunk_granularity`` -- trading the same +pages for more, smaller sub-requests, which gives smoother progress, more even +concurrency, and a smaller unit of retry/resume. It is a scoped ``with`` +block, so an aggressive setting can't leak into unrelated calls and +accidentally spend quota: + +.. code-block:: python + + from dataretrieval import waterdata + + with waterdata.chunk_granularity("high"): + df, md = waterdata.get_daily( + monitoring_location_id=many_sites, parameter_code="00060" + ) + +The level is one of ``"low"``, ``"medium"``, or ``"high"`` (an invalid value +raises ``ValueError`` at the ``with``). Each caps how many sub-chunks a +multi-value argument is split into -- ``2`` / ``8`` / ``32`` for +``"low"`` / ``"medium"`` / ``"high"``. That ceiling is fixed, deliberately +independent of the fan-out concurrency (``API_USGS_CONCURRENT``): how finely a +query splits is orthogonal to how many sub-requests run at once. Capping the +aggressive end at 32 is a guardrail -- an accidental ``"high"`` on a very long +list can't explode into thousands of sub-requests. There is no "off" level: +simply don't enter the block unless you already expect a large, multi-page +result -- on a query that would have fit in a single page, extra chunks only +burn quota. + The full taxonomy ================= diff --git a/tests/utils_test.py b/tests/utils_test.py index b3201419..8ca4630d 100644 --- a/tests/utils_test.py +++ b/tests/utils_test.py @@ -208,6 +208,24 @@ def test_chunk_interruptions_exported_at_top_level(self): dataretrieval.ChunkInterrupted, dataretrieval.DataRetrievalError ) + def test_chunk_granularity_exported_at_top_level_and_waterdata(self): + """The ``chunk_granularity`` context manager and its ``GranularityLevel`` + type are reachable both from the top level (``from dataretrieval import + chunk_granularity``) and from the user-facing ``dataretrieval.waterdata`` + namespace, and both resolve to the single objects defined in + ``dataretrieval.ogc.chunking``.""" + import dataretrieval + from dataretrieval import waterdata + from dataretrieval.ogc import chunking + + assert dataretrieval.chunk_granularity is chunking.chunk_granularity + assert waterdata.chunk_granularity is chunking.chunk_granularity + assert dataretrieval.GranularityLevel is chunking.GranularityLevel + assert waterdata.GranularityLevel is chunking.GranularityLevel + for name in ("chunk_granularity", "GranularityLevel"): + assert name in dataretrieval.__all__ + assert name in waterdata.__all__ + class Test_BaseMetadata: """Tests of BaseMetadata""" diff --git a/tests/waterdata_chunking_test.py b/tests/waterdata_chunking_test.py index 5a946946..6e0a3ec1 100644 --- a/tests/waterdata_chunking_test.py +++ b/tests/waterdata_chunking_test.py @@ -40,8 +40,13 @@ from dataretrieval.ogc import chunking as _chunking from dataretrieval.ogc import retry as _retry_mod from dataretrieval.ogc.chunking import ( + _GRANULARITY_LEVELS, + _GRANULARITY_MAX_CHUNKS, ChunkedCall, _chunked_client, + _granularity, + _resolve_granularity, + chunk_granularity, get_active_client, multi_value_chunked, ) @@ -2105,3 +2110,246 @@ async def fetch(args): assert "finalized" in df.columns assert md[0] == "METADATA" assert calls["finalize"] >= 1 + + +# --------------------------------------------------------------------------- +# Chunk granularity: the opt-in dial (``"low"`` / ``"medium"`` / ``"high"``) to +# fan a query out MORE finely than the byte limit alone requires +# (``ChunkPlan._refine`` + the ``chunk_granularity`` context manager). +# ``_fake_build``'s base is 200 bytes, so a handful of short atoms sits far +# under ``url_limit=8000`` — the byte pass passes it through untouched, and any +# splitting below is the granularity cap alone. ``ChunkPlan`` takes the resolved +# integer cap (``max_chunks_per_axis``) directly; ``chunk_granularity`` / +# ``_resolve_granularity`` map the level names onto it. +# --------------------------------------------------------------------------- + + +def test_zero_cap_preserves_passthrough(): + """``max_chunks_per_axis=0`` (the default) must not perturb the existing + plan: a multi-value request that fits the byte limit is still the trivial + passthrough (no axes, ``total == 1``), byte-for-byte the pre-feature + behavior.""" + args = {"monitoring_location_id": ["A", "B", "C", "D"]} + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks_per_axis=0) + assert plan.axes == [] + assert plan.total == 1 + assert list(plan.iter_sub_args()) == [args] + + +@pytest.mark.parametrize( + ("max_chunks_per_axis", "expected_pieces"), + [(0, 1), (2, 2), (8, 8), (16, 10), (32, 10)], +) +def test_cap_ramps_then_saturates(max_chunks_per_axis, expected_pieces): + """A single 10-atom axis that fits the byte limit splits into + ``min(10, cap)`` pieces: 1 (off), 2, 8, then saturating at 10 (one atom per + chunk) once the cap overshoots the atom count. Monotonic and bounded, and + whenever it splits the partition is a cover — every atom exactly once. (The + cap-0 passthrough has no axis to cover; see the passthrough test.)""" + atoms = [f"S{i:02d}" for i in range(10)] + plan = ChunkPlan( + {"monitoring_location_id": atoms}, + _fake_build, + url_limit=8000, + max_chunks_per_axis=max_chunks_per_axis, + ) + assert plan.total == expected_pieces + if max_chunks_per_axis: + flattened = [ + a for chunk in plan.chunks["monitoring_location_id"] for a in chunk + ] + assert sorted(flattened) == sorted(atoms) + + +def test_cap_bounds_fan_out_for_a_long_axis(): + """The cap is a quota guardrail: at the ``"high"`` cap a 100-atom axis + fans into ``cap`` pieces — NOT 100 singletons — so an accidental + ``chunk_granularity("high")`` on a huge list can't detonate into hundreds + of sub-requests. Every atom is still covered exactly once.""" + high = _GRANULARITY_LEVELS["high"] + atoms = [f"X{i:03d}" for i in range(100)] + plan = ChunkPlan( + {"monitoring_location_id": atoms}, + _fake_build, + url_limit=8000, + max_chunks_per_axis=high, + ) + assert plan.total == high + flattened = [a for chunk in plan.chunks["monitoring_location_id"] for a in chunk] + assert sorted(flattened) == sorted(atoms) + + +def test_cap_below_byte_split_does_not_reduce_fan_out(): + """The cap is purely additive — it can only split further, never coarsen. + A request the byte budget already fans into K>2 chunks is untouched by a + cap of 2 (below K), so the byte-driven plan is preserved.""" + # Heavy axis of four 30-char atoms; a limit tight enough that the byte pass + # must drive every atom into its own sub-request (4 pieces > the cap of 2). + args = {"monitoring_location_id": ["X" * 30, "Y" * 30, "Z" * 30, "W" * 30]} + baseline = ChunkPlan(args, _fake_build, url_limit=250, max_chunks_per_axis=0) + assert baseline.total > 2 # byte pass alone already fanned out past 2 + refined = ChunkPlan(args, _fake_build, url_limit=250, max_chunks_per_axis=2) + # cap 2 < baseline pieces → refine is a no-op here. + assert refined.total == baseline.total + + +def test_cap_never_exceeds_the_byte_budget(): + """Refining on top of an over-budget request keeps the hard invariant: + every sub-request still fits ``url_limit`` (splitting only ever shrinks + a chunk), and the fan-out is at least what the byte pass required.""" + args = {"monitoring_location_id": ["X" * 30, "Y" * 30, "Z" * 30, "W" * 30]} + limit = 310 + byte_only = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks_per_axis=0) + plan = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks_per_axis=32) + assert plan.total >= byte_only.total + for sub in plan.iter_sub_args(): + assert _safe_request_bytes(_fake_build, sub, limit) <= limit + + +def test_cap_refines_the_filter_axis(): + """The dial treats the cql-text ``filter`` axis like any other: an + under-budget filter of N top-level OR-clauses is split along that axis + into ``min(N, cap)`` pieces.""" + clauses = [f"p='{i}'" for i in range(8)] + args = {"filter": " OR ".join(clauses)} + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks_per_axis=4) + assert len(plan.chunks["filter"]) == 4 # min(8, 4) + assert plan.total == 4 + + +def test_cap_multiplies_across_axes(): + """With more than one multi-value axis the per-axis caps multiply — + documented behavior the caller opts into. Two 6-atom axes at a cap of 4 + yield a 4x4 cartesian product.""" + args = { + "monitoring_location_id": [f"L{i}" for i in range(6)], + "parameter_code": [f"{i:05d}" for i in range(6)], + } + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks_per_axis=4) + assert len(plan.chunks["monitoring_location_id"]) == 4 + assert len(plan.chunks["parameter_code"]) == 4 + assert plan.total == 16 + + +def test_cap_does_not_mask_unchunkable(): + """A request with nothing to split that still busts the byte limit must + raise ``Unchunkable`` regardless of the cap — the soft pass has no axis to + act on and must not swallow the hard failure.""" + args = {"monitoring_location_id": "one-huge-scalar"} + with pytest.raises(Unchunkable): + ChunkPlan(args, _fake_build, url_limit=10, max_chunks_per_axis=32) + + +@pytest.mark.parametrize("level", ["low", "medium", "high"]) +def test_resolve_granularity_maps_each_level_to_its_cap(level): + """Each level name resolves to its per-axis sub-chunk cap from the table + (a positive int).""" + assert _resolve_granularity(level) == _GRANULARITY_LEVELS[level] + assert _resolve_granularity(level) >= 1 + + +def test_granularity_levels_ordered_and_spaced(): + """The three caps hold the properties callers rely on: strictly increasing, + ``"high"`` saturating the declared ceiling, ``"low"`` still a real split + (>= 2), and each level spaced 4x from the next. The ceiling is a granularity + constant, deliberately independent of the concurrency default.""" + low = _GRANULARITY_LEVELS["low"] + medium = _GRANULARITY_LEVELS["medium"] + high = _GRANULARITY_LEVELS["high"] + assert low < medium < high + assert high == _GRANULARITY_MAX_CHUNKS + assert low >= 2 + assert medium == high // 4 + assert low == medium // 4 + + +@pytest.mark.parametrize( + "bad", + [ + "off", # a dead keyword form + "LOW", # wrong case — exact match only + " low ", # stray whitespace + 5, # the old integer levels are gone + None, # None not accepted + ["low"], # unhashable → the ``not in`` check still rejects it cleanly + ], +) +def test_resolve_granularity_rejects_everything_but_the_three_levels(bad): + """Only the three exact level strings are accepted; every other value — a + representative of each rejected shape (dead keyword, wrong case, whitespace, + old int, ``None``, unhashable) — raises ``ValueError`` so a typo fails + loudly.""" + with pytest.raises(ValueError, match="chunk_granularity level must be"): + _resolve_granularity(bad) + + +def test_chunk_granularity_scopes_and_restores_the_ambient(): + """The context manager resolves the level to its cap, publishes it on the + ambient for the block, and restores the previous value on exit — including + proper nesting.""" + assert _granularity.get() == 0 + with chunk_granularity("high"): + assert _granularity.get() == _GRANULARITY_LEVELS["high"] + with chunk_granularity("low"): + assert _granularity.get() == _GRANULARITY_LEVELS["low"] + assert _granularity.get() == _GRANULARITY_LEVELS["high"] # outer restored + assert _granularity.get() == 0 # default (off) outside any block + + +def test_chunk_granularity_validates_on_entry(): + """An invalid level raises at ``with`` entry — before any request is + issued — and leaves the ambient untouched.""" + with pytest.raises(ValueError, match="chunk_granularity level must be"): + with chunk_granularity("aggressive"): + pass + assert _granularity.get() == 0 + + +def test_chunk_granularity_high_drives_end_to_end_fan_out(): + """End-to-end: the same fitting request passes through as a single call by + default, but fans into several sub-requests inside a + ``chunk_granularity("high")`` block — and the combined result still + recovers every atom exactly once.""" + sites = [f"S{i:02d}" for i in range(8)] + + calls: list[tuple[str, ...]] = [] + + @multi_value_chunked(build_request=_fake_build, url_limit=8000) + async def fetch(args): + chunk = tuple(args["monitoring_location_id"]) + calls.append(chunk) + return pd.DataFrame({"site": list(chunk)}), _ok_response() + + # Default: comfortably under the byte limit → one passthrough call. + df_plain, _ = fetch({"monitoring_location_id": sites}) + assert len(calls) == 1 + assert sorted(df_plain["site"]) == sorted(sites) + + calls.clear() + with chunk_granularity("high"): + df_fine, _ = fetch({"monitoring_location_id": sites}) + # 8 atoms at the "high" cap (>= 8) → 8 singleton sub-requests. + assert len(calls) == 8 + assert all(len(chunk) == 1 for chunk in calls) + # Union across chunks recovers the original set, once each. + assert sorted(a for chunk in calls for a in chunk) == sorted(sites) + assert sorted(df_fine["site"]) == sorted(sites) + + +def test_chunk_granularity_low_is_a_gentle_split(): + """``"low"`` is the mildest opt-in: an under-limit request fans into just + the ``"low"`` cap's worth of pieces, not singletons.""" + low = _GRANULARITY_LEVELS["low"] + sites = [f"S{i:02d}" for i in range(8)] + calls: list[int] = [] + + @multi_value_chunked(build_request=_fake_build, url_limit=8000) + async def fetch(args): + calls.append(len(args["monitoring_location_id"])) + return pd.DataFrame(), _ok_response() + + with chunk_granularity("low"): + fetch({"monitoring_location_id": sites}) + # low cap → that many sub-requests, together covering all 8 sites. + assert len(calls) == low + assert sum(calls) == 8 From e87feb684b1a27c631569685bf937dbf69ab2795 Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Wed, 8 Jul 2026 15:51:38 -0500 Subject: [PATCH 02/13] feat(waterdata): cap chunk_granularity by total sub-requests; document + benchmark Cap chunk_granularity on the plan's TOTAL sub-request count (2/8/32) rather than per multi-value axis, so several multi-value arguments can't multiply past the ceiling. ChunkPlan._refine now splits the largest splittable chunk across every axis round-robin; behavior is identical for the common single-axis query. Add a 'Speeding up large downloads' usage section to the README (Water Data API) with a measured cold-cache benchmark: parallelizing a large paginated pull's sub-requests gave ~6x (production page size, cold) up to ~12x (latency-bound) on 271 Ohio discharge sites. Temper the 'quota-neutral' claim in the docstring, NEWS, and user guide: a finer split is only ~quota-neutral when each sub-request still spans many pages; otherwise each chunk's partial final page adds some requests. Co-Authored-By: Claude Opus 4.8 (1M context) --- NEWS.md | 2 +- README.md | 48 +++++++++++++++++++++++ dataretrieval/ogc/chunking.py | 61 ++++++++++++++++++------------ dataretrieval/ogc/planning.py | 65 +++++++++++++++++++------------- docs/source/userguide/errors.rst | 24 +++++++----- tests/waterdata_chunking_test.py | 44 ++++++++++++++++----- 6 files changed, 173 insertions(+), 71 deletions(-) diff --git a/NEWS.md b/NEWS.md index c2d4b259..545f5080 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,4 +1,4 @@ -**07/01/2026:** Added `waterdata.chunk_granularity(...)` — a context manager to control how finely the OGC `waterdata` getters split multi-value requests. By default the getters chunk only as much as the server's ~8 KB URL limit forces (the fewest sub-requests). But because every sub-request paginates, splitting a large result further is usually quota-neutral, so a caller who *knows* their pull is large can opt into a finer split for smoother progress, more even concurrency, and a smaller unit of retry/resume: `with waterdata.chunk_granularity("high"): df, md = waterdata.get_daily(monitoring_location_id=many_sites)`. The level is one of `"low"`, `"medium"`, or `"high"` — typed as `waterdata.GranularityLevel` (a `typing.Literal`), so a type checker rejects any other value and an invalid string raises `ValueError` at the `with`. Each level caps how many sub-chunks a multi-value argument is split into — `"low"` / `"medium"` / `"high"` cap at 2 / 8 / 32. That ceiling is a granularity constant, deliberately independent of the fan-out concurrency (`API_USGS_CONCURRENT`): how finely a query splits is orthogonal to how many sub-requests run at once. Capping the aggressive end at 32 bounds the blast radius — an accidental `"high"` on a huge list can't explode into thousands of sub-requests. There is no "off" level: not entering the block *is* off. It is deliberately a scoped `with` block rather than an automatic behavior or an environment variable, since the library can't tell in advance whether a query is large — a short-window query might fit in a single page, where extra chunks would only burn quota. Also exported at the top level as `dataretrieval.chunk_granularity`. +**07/01/2026:** Added `waterdata.chunk_granularity(...)` — a context manager to control how finely the OGC `waterdata` getters split multi-value requests. By default the getters chunk only as much as the server's ~8 KB URL limit forces (the fewest sub-requests). But because every sub-request paginates, splitting a large result further costs little or no extra quota when each sub-request still spans many pages, so a caller who *knows* their pull is large can opt into a finer split for smoother progress, more even concurrency, and a smaller unit of retry/resume: `with waterdata.chunk_granularity("high"): df, md = waterdata.get_daily(monitoring_location_id=many_sites)`. The level is one of `"low"`, `"medium"`, or `"high"` — typed as `waterdata.GranularityLevel` (a `typing.Literal`), so a type checker rejects any other value and an invalid string raises `ValueError` at the `with`. Each level caps the *total* number of sub-requests the call is split into — `"low"` / `"medium"` / `"high"` cap at 2 / 8 / 32 overall, across every multi-value argument combined, not per argument. That ceiling is a granularity constant, deliberately independent of the fan-out concurrency (`API_USGS_CONCURRENT`): how finely a query splits is orthogonal to how many sub-requests run at once. Capping the aggressive end at 32 bounds the blast radius — an accidental `"high"` on a huge list, or a call with several multi-value arguments, can't explode into thousands of sub-requests. There is no "off" level: not entering the block *is* off. It is deliberately a scoped `with` block rather than an automatic behavior or an environment variable, since the library can't tell in advance whether a query is large — a short-window query might fit in a single page, where extra chunks would only burn quota. Also exported at the top level as `dataretrieval.chunk_granularity`. **06/23/2026:** **Breaking change (1.2.0):** the minimum supported Python is now **3.10** (`requires-python = ">=3.10"`). 3.9 support was already effectively broken — the `waterdata` module's dependencies (`anyio`, the test stack) require 3.10+, and the `waterdata` test modules already skipped on <3.10. `anyio` is now declared as a direct dependency (it is imported directly by `waterdata`), and the CI/ruff/mypy targets move to 3.10. Also fully removed the deprecated `variable_info` metadata property: the `NWIS_Metadata` override only warned and returned `None` (it relied on the defunct `get_pmcodes`), and the `BaseMetadata` abstract is gone too since nothing implemented it — accessing `.variable_info` now raises `AttributeError`. `site_info` is unaffected. diff --git a/README.md b/README.md index d904e6d6..e37c263b 100644 --- a/README.md +++ b/README.md @@ -105,6 +105,54 @@ df, metadata = waterdata.get_continuous( print(f"Retrieved {len(df)} continuous gage height measurements") ``` +#### Speeding up large downloads with `chunk_granularity` + +By default the getters split a multi-value request only as far as the server's +~8 KB URL limit forces — the fewest sub-requests. For a **large, paginated** +pull that is needlessly conservative: every sub-request pages through its own +results, so dividing the query into more, smaller sub-requests lets those pages +be fetched **in parallel**. `chunk_granularity` opts a single call into that +finer split. It pays off only when the result is large enough to span many +pages *and* the query has a multi-value argument to divide (such as a list of +monitoring locations); on a small query — or one with nothing to split — it just +adds requests, so it is a deliberate, scoped `with` block, never the default. + +```python +from dataretrieval import waterdata + +# All stream gages in Ohio, then 20 years of their daily discharge — large +# enough to span many pages, so it profits from a finer split. +sites, _ = waterdata.get_monitoring_locations(state="Ohio", site_type_code="ST") + +with waterdata.chunk_granularity("high"): # "low" | "medium" | "high" + df, md = waterdata.get_daily( + monitoring_location_id=sites["monitoring_location_id"].tolist(), + parameter_code="00060", # discharge + time="2004-01-01/2023-12-31", + ) +``` + +`"high"` fans a call out into up to 32 sub-requests, `"medium"` up to 8, and +`"low"` up to 2 — a fixed ceiling, so an accidental setting on a huge list can't +explode into thousands of requests. + +Benchmark — 271 Ohio discharge sites (`get_daily`, `parameter_code="00060"`), +cold cache, each level run against its own time window so results are not +cache-served, with the page size fixed so every level fetches roughly the same +number of pages (isolating the effect of parallelism): + +| level | parallelism | pages | wall-clock | speedup | +| -------------- | ----------- | ----- | ------------------------- | ------- | +| `default` | 1 | 32 | 23.9 s / 27.4 s (2 runs) | 1× | +| `"medium"` (8) | 8 | 37 | 4.1 s / 4.3 s | ~6× | +| `"high"` (32) | 32 | 44 | 2.0 s | ~12× | + +The gain comes from overlapping each sub-request's per-page latency and +server-side work — a genuinely large pull at the default page size shows a +similar multi-fold speedup. The extra sub-requests each cost one request against +your hourly [rate limit](https://api.waterdata.usgs.gov/signup/), so reserve the +aggressive levels for pulls you know are large. + Visit the [API Reference](https://doi-usgs.github.io/dataretrieval-python/reference/waterdata.html) for more information and examples on available services and input parameters. diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index 5bbcec56..1334f018 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -182,8 +182,9 @@ def get_active_client() -> httpx.AsyncClient | None: # Chunk-granularity dial: opt-in to fan a query out *more finely* than the byte # limit alone requires. Scoped to a ``with chunk_granularity(...):`` block (a # ContextVar), deliberately NOT an env var (see :func:`chunk_granularity` for -# why). The ambient holds the resolved cap on sub-chunks per axis; ``0`` (the -# default, outside any block) means "chunk only as much as the byte limit needs". +# why). The ambient holds the resolved cap on the plan's total sub-request +# count; ``0`` (the default, outside any block) means "chunk only as much as +# the byte limit needs". _granularity: Ambient[int] = Ambient("ogc_chunk_granularity", 0) #: The three accepted granularity levels, as a typing ``Literal`` so a type @@ -196,12 +197,14 @@ def get_active_client() -> httpx.AsyncClient | None: #: modules). _VALID_LEVELS: tuple[GranularityLevel, ...] = get_args(GranularityLevel) -# Granularity's own ceiling on sub-chunks per axis — deliberately NOT tied to the -# concurrency default: fan-out *volume* (how many sub-requests a query becomes) -# is orthogonal to how many run at once (``API_USGS_CONCURRENT``). 32 is a sane -# single-axis ceiling that also bounds the blast radius of an accidental -# ``"high"`` on a very long list; the milder levels are a quarter and a -# sixteenth of it (the three are spaced 4x apart). +# Granularity's own ceiling on the plan's total sub-request count — +# deliberately NOT tied to the concurrency default: fan-out *volume* (how many +# sub-requests a query becomes) is orthogonal to how many run at once +# (``API_USGS_CONCURRENT``). 32 is a sane ceiling on the whole call (across +# every axis, not per axis — see ``ChunkPlan._refine``) that bounds the blast +# radius of an accidental ``"high"`` on a very long list or several multi-value +# arguments at once; the milder levels are a quarter and a sixteenth of it (the +# three are spaced 4x apart). _GRANULARITY_MAX_CHUNKS = 32 _GRANULARITY_LEVELS: dict[str, int] = { "low": _GRANULARITY_MAX_CHUNKS // 16, # 2 @@ -212,7 +215,7 @@ def get_active_client() -> httpx.AsyncClient | None: def _resolve_granularity(level: GranularityLevel) -> int: """ - Map a granularity level name to its per-axis sub-chunk cap. + Map a granularity level name to its total sub-request cap. Parameters ---------- @@ -222,7 +225,7 @@ def _resolve_granularity(level: GranularityLevel) -> int: Returns ------- int - The maximum sub-chunks per multi-value axis for that level + The maximum total sub-requests for the whole call at that level (``2`` / ``8`` / ``32``). Raises @@ -247,12 +250,16 @@ def chunk_granularity(level: GranularityLevel) -> Iterator[None]: By default the Water Data / NGWMN getters chunk a request only as much as the server's ~8 KB URL-byte limit forces — the fewest sub-requests that fit. That is the safe default, but it can be *needlessly* conservative: - because every sub-request paginates, splitting a large result further is - usually quota-neutral (ten states pulled as one under-limit request page - just as many times as ten per-state requests would). This context manager - lets a caller who *knows* their pull is large ask for that finer split — - trading the same pages for more, smaller sub-requests, which gives smoother - progress, more even concurrency, and a smaller unit of retry/resume. + because every sub-request paginates, splitting a large result further costs + little or no extra quota *as long as each sub-request still spans many + pages* — rows-per-chunk far exceeding the page size (ten states pulled as + one request then page nearly as many times as ten per-state requests + would). When a split leaves each sub-request only a page or two, its partial + final page is extra, so finer chunks do add some requests. This context + manager lets a caller who *knows* their pull is large ask for that finer + split — trading roughly the same pages for more, smaller sub-requests, which + gives smoother progress, more even concurrency, and a smaller unit of + retry/resume. Because the library can't tell in advance whether a query is large (ten states over a short window might fit in a single page, where extra chunks @@ -267,15 +274,19 @@ def chunk_granularity(level: GranularityLevel) -> Iterator[None]: Parameters ---------- level : {"low", "medium", "high"} - How aggressively to chunk within the block. Each level caps how many - sub-chunks a single multi-value argument is split into — ``2`` / ``8`` / - ``32`` for ``"low"`` / ``"medium"`` / ``"high"`` — or one value per - sub-request once the argument has fewer values than the cap. The ceiling - is fixed (it is *not* tied to ``API_USGS_CONCURRENT``: how finely a query - splits is orthogonal to how many sub-requests run at once); capping - ``"high"`` at ``32`` keeps an accidental aggressive level on a very long - list from exploding into thousands of sub-requests. (With several - multi-value arguments the per-argument counts still multiply.) + How aggressively to chunk within the block. Each level caps the + *total* number of sub-requests the call is split into — ``2`` / ``8`` + / ``32`` for ``"low"`` / ``"medium"`` / ``"high"`` — or one + sub-request per remaining atom once that's fewer than the cap. The + cap applies to the whole call, not per multi-value argument: with + several multi-value arguments the axes are refined together against + the same shared ceiling (their cartesian product does not multiply + past it). The ceiling is fixed (it is *not* tied to + ``API_USGS_CONCURRENT``: how finely a query splits is orthogonal to + how many sub-requests run at once); capping ``"high"`` at ``32`` keeps + an accidental aggressive level on a very long list — or several + multi-value arguments at once — from exploding into thousands of + sub-requests. Yields ------ diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index 700a0e85..58bdad65 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -330,12 +330,13 @@ class ChunkPlan: Byte budget for the request (URL + body) — a hard ceiling every sub-request must fit. max_chunks_per_axis : int, optional - Soft cap on sub-chunks per multi-value axis (default ``0`` = off). + Soft cap on the plan's total sub-request count (default ``0`` = off). ``0`` chunks only as much as ``url_limit`` requires — the most - conservative plan, fewest sub-requests. A positive cap fans each axis - out into up to ``min(len(atoms), max_chunks_per_axis)`` pieces (never - fewer than the byte budget already forces), so a large multi-page pull - is issued as more, smaller sub-requests. Set from the resolved + conservative plan, fewest sub-requests. A positive cap fans the plan + out to up to ``max_chunks_per_axis`` sub-requests overall (the + cartesian product across axes, never fewer than the byte budget + already forces) — capped as a whole, not per axis, so several + multi-value axes can't multiply past the cap. Set from the resolved :func:`~dataretrieval.ogc.chunking.chunk_granularity` level; see :meth:`_refine`. @@ -492,38 +493,50 @@ def _plan( def _refine(self, max_chunks_per_axis: int) -> None: """ - Fan each axis out more finely than the byte budget alone requires — + Fan the plan out more finely than the byte budget alone requires — the granularity dial (see :func:`~dataretrieval.ogc.chunking.chunk_granularity` for why a caller would want this). - Each axis is split until it holds at least - ``min(len(atoms), max_chunks_per_axis)`` chunks — up to the cap, or one - atom per chunk for a shorter axis. Purely additive — only ever *splits* - existing chunks, so the byte pass's work and the ``url_limit`` invariant - are both preserved (an axis the byte pass already split past the cap is - left alone), and it never raises. A no-op at ``max_chunks_per_axis <= 0``. + Caps the plan's *total* sub-request count (:attr:`total`, the + cartesian product across all axes) at ``max_chunks_per_axis``, not + each axis independently — with several multi-value axes, a cap of 32 + still means at most 32 sub-requests overall, not ``32 ** n_axes``. + Each split picks the single largest splittable chunk across *every* + axis (ties broken by axis-extraction order, then lowest index), so + growth is distributed round-robin rather than one axis saturating + before another is touched. Purely additive — only ever *splits* + existing chunks, so the byte pass's work and the ``url_limit`` + invariant are both preserved, and it never raises. A no-op at + ``max_chunks_per_axis <= 0``. Parameters ---------- max_chunks_per_axis : int - Soft cap on sub-chunks per axis (``0`` = off), the resolved - granularity level. A shorter axis simply saturates at one atom per - chunk. + Soft cap on the plan's total sub-request count (``0`` = off), + the resolved granularity level. Despite the name — kept for the + public dial's "sub-chunks per multi-value argument" framing, + which matches this cap exactly in the common single-axis case — + multi-axis plans are capped on the product, not per axis. """ if max_chunks_per_axis <= 0: return - for axis in self.axes: - chunks = self.chunks[axis.arg_key] - target = min(len(axis.atoms), max_chunks_per_axis) - # ``target <= len(atoms)`` guarantees a splittable chunk each pass, so - # this reaches exactly ``target`` and terminates. Split the chunk with - # the most *atoms* (``_plan`` splits by *bytes*): here we even out - # cardinality for smooth fan-out, not URL size. Ties take the lowest - # index, keeping the split deterministic. - while len(chunks) < target: - idx, _ = max(enumerate(chunks), key=lambda kv: len(kv[1])) - _split_at(chunks, idx) + while self.total < max_chunks_per_axis: + # Largest splittable chunk across every axis; a chunk of size 1 + # can't be split further. ``max`` with a stable input order + # breaks ties by axis order, then lowest index within an axis. + candidate: tuple[_Axis, int] | None = None + candidate_size = -1 + for axis in self.axes: + for idx, chunk in enumerate(self.chunks[axis.arg_key]): + if len(chunk) <= 1: + continue + if len(chunk) > candidate_size: + candidate, candidate_size = (axis, idx), len(chunk) + if candidate is None: + return # every axis saturated at one atom per chunk + axis, idx = candidate + _split_at(self.chunks[axis.arg_key], idx) def _worst_case_args(self) -> dict[str, Any]: """ diff --git a/docs/source/userguide/errors.rst b/docs/source/userguide/errors.rst index 10b1a2c7..0c4429c0 100644 --- a/docs/source/userguide/errors.rst +++ b/docs/source/userguide/errors.rst @@ -101,12 +101,14 @@ Chunk a large request more finely By default the getters split an over-large request only as much as the server's ~8 KB URL limit forces -- the fewest sub-requests. Because each -sub-request paginates, splitting a large result further is usually -quota-neutral (ten states pulled as one under-limit request page just as many -times as ten per-state requests would), so if you *know* your pull is large -you can ask for a finer split with ``chunk_granularity`` -- trading the same -pages for more, smaller sub-requests, which gives smoother progress, more even -concurrency, and a smaller unit of retry/resume. It is a scoped ``with`` +sub-request paginates, splitting a large result further costs little or no +extra quota *as long as each sub-request still spans many pages* (ten states +pulled as one request then page nearly as many times as ten per-state requests +would; a split that leaves each sub-request only a page or two adds its partial +final page). So if you *know* your pull is large you can ask for a finer split +with ``chunk_granularity`` -- trading roughly the same pages for more, smaller +sub-requests, which gives smoother progress, more even concurrency, and a +smaller unit of retry/resume. It is a scoped ``with`` block, so an aggressive setting can't leak into unrelated calls and accidentally spend quota: @@ -120,13 +122,15 @@ accidentally spend quota: ) The level is one of ``"low"``, ``"medium"``, or ``"high"`` (an invalid value -raises ``ValueError`` at the ``with``). Each caps how many sub-chunks a -multi-value argument is split into -- ``2`` / ``8`` / ``32`` for -``"low"`` / ``"medium"`` / ``"high"``. That ceiling is fixed, deliberately +raises ``ValueError`` at the ``with``). Each caps the *total* number of +sub-requests the call is split into -- ``2`` / ``8`` / ``32`` for +``"low"`` / ``"medium"`` / ``"high"``, across every multi-value argument +combined, not per argument. That ceiling is fixed, deliberately independent of the fan-out concurrency (``API_USGS_CONCURRENT``): how finely a query splits is orthogonal to how many sub-requests run at once. Capping the aggressive end at 32 is a guardrail -- an accidental ``"high"`` on a very long -list can't explode into thousands of sub-requests. There is no "off" level: +list, or a call with several multi-value arguments, can't explode into +thousands of sub-requests. There is no "off" level: simply don't enter the block unless you already expect a large, multi-page result -- on a query that would have fit in a single page, extra chunks only burn quota. diff --git a/tests/waterdata_chunking_test.py b/tests/waterdata_chunking_test.py index 6e0a3ec1..055db44b 100644 --- a/tests/waterdata_chunking_test.py +++ b/tests/waterdata_chunking_test.py @@ -2120,7 +2120,9 @@ async def fetch(args): # under ``url_limit=8000`` — the byte pass passes it through untouched, and any # splitting below is the granularity cap alone. ``ChunkPlan`` takes the resolved # integer cap (``max_chunks_per_axis``) directly; ``chunk_granularity`` / -# ``_resolve_granularity`` map the level names onto it. +# ``_resolve_granularity`` map the level names onto it. The cap bounds the +# plan's *total* sub-request count (the cartesian product across axes), not +# each axis independently — see ``test_cap_caps_the_total_across_axes``. # --------------------------------------------------------------------------- @@ -2217,18 +2219,42 @@ def test_cap_refines_the_filter_axis(): assert plan.total == 4 -def test_cap_multiplies_across_axes(): - """With more than one multi-value axis the per-axis caps multiply — - documented behavior the caller opts into. Two 6-atom axes at a cap of 4 - yield a 4x4 cartesian product.""" +def test_cap_caps_the_total_across_axes(): + """With more than one multi-value axis the cap bounds the *total* + sub-request count (the cartesian product), not each axis independently — + the blast-radius guardrail the dial exists for. Two 6-atom axes at a cap + of 4 top out at 4 sub-requests total, not 4x4=16; growth is distributed + round-robin across axes rather than one axis alone climbing to the cap.""" args = { "monitoring_location_id": [f"L{i}" for i in range(6)], "parameter_code": [f"{i:05d}" for i in range(6)], } plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks_per_axis=4) - assert len(plan.chunks["monitoring_location_id"]) == 4 - assert len(plan.chunks["parameter_code"]) == 4 - assert plan.total == 16 + assert plan.total == 4 + # Every atom on every axis is still covered exactly once. + for key, atoms in ( + ("monitoring_location_id", args["monitoring_location_id"]), + ("parameter_code", args["parameter_code"]), + ): + flattened = [a for chunk in plan.chunks[key] for a in chunk] + assert sorted(flattened) == sorted(atoms) + + +def test_cap_bounds_fan_out_across_many_axes(): + """The guardrail holds regardless of axis count: three multi-value axes + at the ``"high"`` cap still top out at ``high`` sub-requests total, not + ``high ** 3`` — the property the single-axis-only cap in the original + implementation did not guarantee.""" + high = _GRANULARITY_LEVELS["high"] + # Three chunkable axes (two list axes + the filter OR-axis), each with 10 + # atoms — under the old per-axis cap this would have been high**3. + args = { + "monitoring_location_id": [f"L{i}" for i in range(10)], + "parameter_code": [f"{i:05d}" for i in range(10)], + "filter": " OR ".join(f"p='{i}'" for i in range(10)), + } + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks_per_axis=high) + assert plan.total <= high def test_cap_does_not_mask_unchunkable(): @@ -2242,7 +2268,7 @@ def test_cap_does_not_mask_unchunkable(): @pytest.mark.parametrize("level", ["low", "medium", "high"]) def test_resolve_granularity_maps_each_level_to_its_cap(level): - """Each level name resolves to its per-axis sub-chunk cap from the table + """Each level name resolves to its total sub-request cap from the table (a positive int).""" assert _resolve_granularity(level) == _GRANULARITY_LEVELS[level] assert _resolve_granularity(level) >= 1 From 2b2f5f0bc3b3d0c0b941bb366a9eef76871e9ada Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Wed, 8 Jul 2026 18:09:24 -0500 Subject: [PATCH 03/13] refactor(waterdata): rename chunk_granularity -> parallel_chunks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rename the public context manager chunk_granularity to parallel_chunks and the GranularityLevel type to ParallelChunksLevel (both exported from dataretrieval and dataretrieval.waterdata), plus internals (_resolve_level, _MAX_PARALLEL_CHUNKS, _LEVEL_CAPS, the _parallel_chunks ambient). 'parallel_chunks' names what the knob does — fan a query into more, parallel sub-requests — without colliding with API_USGS_CONCURRENT (the separate in-flight cap). Docstrings, NEWS, user guide, README, and tests updated to match. Pure rename; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) --- NEWS.md | 2 +- README.md | 6 +- dataretrieval/__init__.py | 12 ++-- dataretrieval/ogc/chunking.py | 58 +++++++++---------- dataretrieval/ogc/planning.py | 12 ++-- dataretrieval/waterdata/__init__.py | 6 +- docs/source/userguide/errors.rst | 4 +- tests/utils_test.py | 16 +++--- tests/waterdata_chunking_test.py | 89 +++++++++++++++-------------- 9 files changed, 103 insertions(+), 102 deletions(-) diff --git a/NEWS.md b/NEWS.md index 545f5080..e3d6f32b 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,4 +1,4 @@ -**07/01/2026:** Added `waterdata.chunk_granularity(...)` — a context manager to control how finely the OGC `waterdata` getters split multi-value requests. By default the getters chunk only as much as the server's ~8 KB URL limit forces (the fewest sub-requests). But because every sub-request paginates, splitting a large result further costs little or no extra quota when each sub-request still spans many pages, so a caller who *knows* their pull is large can opt into a finer split for smoother progress, more even concurrency, and a smaller unit of retry/resume: `with waterdata.chunk_granularity("high"): df, md = waterdata.get_daily(monitoring_location_id=many_sites)`. The level is one of `"low"`, `"medium"`, or `"high"` — typed as `waterdata.GranularityLevel` (a `typing.Literal`), so a type checker rejects any other value and an invalid string raises `ValueError` at the `with`. Each level caps the *total* number of sub-requests the call is split into — `"low"` / `"medium"` / `"high"` cap at 2 / 8 / 32 overall, across every multi-value argument combined, not per argument. That ceiling is a granularity constant, deliberately independent of the fan-out concurrency (`API_USGS_CONCURRENT`): how finely a query splits is orthogonal to how many sub-requests run at once. Capping the aggressive end at 32 bounds the blast radius — an accidental `"high"` on a huge list, or a call with several multi-value arguments, can't explode into thousands of sub-requests. There is no "off" level: not entering the block *is* off. It is deliberately a scoped `with` block rather than an automatic behavior or an environment variable, since the library can't tell in advance whether a query is large — a short-window query might fit in a single page, where extra chunks would only burn quota. Also exported at the top level as `dataretrieval.chunk_granularity`. +**07/01/2026:** Added `waterdata.parallel_chunks(...)` — a context manager to control how finely the OGC `waterdata` getters split multi-value requests. By default the getters chunk only as much as the server's ~8 KB URL limit forces (the fewest sub-requests). But because every sub-request paginates, splitting a large result further costs little or no extra quota when each sub-request still spans many pages, so a caller who *knows* their pull is large can opt into a finer split for smoother progress, more even concurrency, and a smaller unit of retry/resume: `with waterdata.parallel_chunks("high"): df, md = waterdata.get_daily(monitoring_location_id=many_sites)`. The level is one of `"low"`, `"medium"`, or `"high"` — typed as `waterdata.ParallelChunksLevel` (a `typing.Literal`), so a type checker rejects any other value and an invalid string raises `ValueError` at the `with`. Each level caps the *total* number of sub-requests the call is split into — `"low"` / `"medium"` / `"high"` cap at 2 / 8 / 32 overall, across every multi-value argument combined, not per argument. That ceiling is a parallel_chunks constant, deliberately independent of the fan-out concurrency (`API_USGS_CONCURRENT`): how finely a query splits is orthogonal to how many sub-requests run at once. Capping the aggressive end at 32 bounds the blast radius — an accidental `"high"` on a huge list, or a call with several multi-value arguments, can't explode into thousands of sub-requests. There is no "off" level: not entering the block *is* off. It is deliberately a scoped `with` block rather than an automatic behavior or an environment variable, since the library can't tell in advance whether a query is large — a short-window query might fit in a single page, where extra chunks would only burn quota. Also exported at the top level as `dataretrieval.parallel_chunks`. **06/23/2026:** **Breaking change (1.2.0):** the minimum supported Python is now **3.10** (`requires-python = ">=3.10"`). 3.9 support was already effectively broken — the `waterdata` module's dependencies (`anyio`, the test stack) require 3.10+, and the `waterdata` test modules already skipped on <3.10. `anyio` is now declared as a direct dependency (it is imported directly by `waterdata`), and the CI/ruff/mypy targets move to 3.10. Also fully removed the deprecated `variable_info` metadata property: the `NWIS_Metadata` override only warned and returned `None` (it relied on the defunct `get_pmcodes`), and the `BaseMetadata` abstract is gone too since nothing implemented it — accessing `.variable_info` now raises `AttributeError`. `site_info` is unaffected. diff --git a/README.md b/README.md index e37c263b..71f3b329 100644 --- a/README.md +++ b/README.md @@ -105,13 +105,13 @@ df, metadata = waterdata.get_continuous( print(f"Retrieved {len(df)} continuous gage height measurements") ``` -#### Speeding up large downloads with `chunk_granularity` +#### Speeding up large downloads with `parallel_chunks` By default the getters split a multi-value request only as far as the server's ~8 KB URL limit forces — the fewest sub-requests. For a **large, paginated** pull that is needlessly conservative: every sub-request pages through its own results, so dividing the query into more, smaller sub-requests lets those pages -be fetched **in parallel**. `chunk_granularity` opts a single call into that +be fetched **in parallel**. `parallel_chunks` opts a single call into that finer split. It pays off only when the result is large enough to span many pages *and* the query has a multi-value argument to divide (such as a list of monitoring locations); on a small query — or one with nothing to split — it just @@ -124,7 +124,7 @@ from dataretrieval import waterdata # enough to span many pages, so it profits from a finer split. sites, _ = waterdata.get_monitoring_locations(state="Ohio", site_type_code="ST") -with waterdata.chunk_granularity("high"): # "low" | "medium" | "high" +with waterdata.parallel_chunks("high"): # "low" | "medium" | "high" df, md = waterdata.get_daily( monitoring_location_id=sites["monitoring_location_id"].tolist(), parameter_code="00060", # discharge diff --git a/dataretrieval/__init__.py b/dataretrieval/__init__.py index ec79a724..567be6ab 100644 --- a/dataretrieval/__init__.py +++ b/dataretrieval/__init__.py @@ -44,10 +44,10 @@ URLTooLong, ) -# Chunk-granularity control (a context manager) and its level type. Defined with +# Parallel-chunks control (a context manager) and its level type. Defined with # the chunker in ``dataretrieval.ogc.chunking``; surfaced here for a stable -# public path ``from dataretrieval import chunk_granularity``. -from dataretrieval.ogc.chunking import GranularityLevel, chunk_granularity +# public path ``from dataretrieval import parallel_chunks``. +from dataretrieval.ogc.chunking import ParallelChunksLevel, parallel_chunks # Resumable chunk-interruption exceptions. They are defined in # ``dataretrieval.ogc.interruptions`` rather than ``dataretrieval.exceptions`` @@ -98,8 +98,8 @@ "ChunkInterrupted", "QuotaExhausted", "ServiceInterrupted", - # chunk-granularity control (defined in ogc.chunking) - "chunk_granularity", - "GranularityLevel", + # parallel-chunks control (defined in ogc.chunking) + "parallel_chunks", + "ParallelChunksLevel", "__version__", ] diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index 1334f018..3d0e9aff 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -9,11 +9,11 @@ cartesian product of chunks. Requests that already fit get a trivial single-step plan — ``ChunkedCall`` has one code path either way. -Granularity: the planner is conservative by default — it splits only as far as +Parallel chunks: the planner is conservative by default — it splits only as far as the byte limit forces. A caller who knows their result is large can opt into a -finer split via the ``chunk_granularity`` context manager +finer split via the ``parallel_chunks`` context manager (``"low"`` / ``"medium"`` / ``"high"``); the resolved cap drives -:meth:`ChunkPlan._refine`. See ``chunk_granularity`` for the why and the when. +:meth:`ChunkPlan._refine`. See ``parallel_chunks`` for the why and the when. This module owns the *execution* half — the event loop and bounded concurrency that drive a plan to completion (``ChunkedCall``) plus the @@ -179,25 +179,25 @@ def get_active_client() -> httpx.AsyncClient | None: return _chunked_client.get() -# Chunk-granularity dial: opt-in to fan a query out *more finely* than the byte -# limit alone requires. Scoped to a ``with chunk_granularity(...):`` block (a -# ContextVar), deliberately NOT an env var (see :func:`chunk_granularity` for +# Parallel-chunks dial: opt-in to fan a query out *more finely* than the byte +# limit alone requires. Scoped to a ``with parallel_chunks(...):`` block (a +# ContextVar), deliberately NOT an env var (see :func:`parallel_chunks` for # why). The ambient holds the resolved cap on the plan's total sub-request # count; ``0`` (the default, outside any block) means "chunk only as much as # the byte limit needs". -_granularity: Ambient[int] = Ambient("ogc_chunk_granularity", 0) +_parallel_chunks: Ambient[int] = Ambient("ogc_parallel_chunks", 0) -#: The three accepted granularity levels, as a typing ``Literal`` so a type +#: The three accepted parallel_chunks levels, as a typing ``Literal`` so a type #: checker rejects any other value at the call site. -GranularityLevel = Literal["low", "medium", "high"] +ParallelChunksLevel = Literal["low", "medium", "high"] -#: Valid levels derived from the type, so ``GranularityLevel`` stays the single -#: source of truth for what ``_resolve_granularity`` accepts (mirrors the +#: Valid levels derived from the type, so ``ParallelChunksLevel`` stays the single +#: source of truth for what ``_resolve_level`` accepts (mirrors the #: ``get_args``-based ``_VALID_ON_TIE`` / ``_VALID_FILE_TYPES`` in sibling #: modules). -_VALID_LEVELS: tuple[GranularityLevel, ...] = get_args(GranularityLevel) +_VALID_LEVELS: tuple[ParallelChunksLevel, ...] = get_args(ParallelChunksLevel) -# Granularity's own ceiling on the plan's total sub-request count — +# The parallel_chunks ceiling on the plan's total sub-request count — # deliberately NOT tied to the concurrency default: fan-out *volume* (how many # sub-requests a query becomes) is orthogonal to how many run at once # (``API_USGS_CONCURRENT``). 32 is a sane ceiling on the whole call (across @@ -205,17 +205,17 @@ def get_active_client() -> httpx.AsyncClient | None: # radius of an accidental ``"high"`` on a very long list or several multi-value # arguments at once; the milder levels are a quarter and a sixteenth of it (the # three are spaced 4x apart). -_GRANULARITY_MAX_CHUNKS = 32 -_GRANULARITY_LEVELS: dict[str, int] = { - "low": _GRANULARITY_MAX_CHUNKS // 16, # 2 - "medium": _GRANULARITY_MAX_CHUNKS // 4, # 8 - "high": _GRANULARITY_MAX_CHUNKS, # 32 +_MAX_PARALLEL_CHUNKS = 32 +_LEVEL_CAPS: dict[str, int] = { + "low": _MAX_PARALLEL_CHUNKS // 16, # 2 + "medium": _MAX_PARALLEL_CHUNKS // 4, # 8 + "high": _MAX_PARALLEL_CHUNKS, # 32 } -def _resolve_granularity(level: GranularityLevel) -> int: +def _resolve_level(level: ParallelChunksLevel) -> int: """ - Map a granularity level name to its total sub-request cap. + Map a parallel_chunks level name to its total sub-request cap. Parameters ---------- @@ -237,13 +237,13 @@ def _resolve_granularity(level: GranularityLevel) -> int: """ if level not in _VALID_LEVELS: raise ValueError( - f"chunk_granularity level must be one of {_VALID_LEVELS}; got {level!r}." + f"parallel_chunks level must be one of {_VALID_LEVELS}; got {level!r}." ) - return _GRANULARITY_LEVELS[level] + return _LEVEL_CAPS[level] @contextmanager -def chunk_granularity(level: GranularityLevel) -> Iterator[None]: +def parallel_chunks(level: ParallelChunksLevel) -> Iterator[None]: """ Scope how finely the OGC getters chunk multi-value requests. @@ -301,7 +301,7 @@ def chunk_granularity(level: GranularityLevel) -> Iterator[None]: Examples -------- >>> from dataretrieval import waterdata - >>> with waterdata.chunk_granularity("high"): + >>> with waterdata.parallel_chunks("high"): ... df, md = waterdata.get_daily( ... monitoring_location_id=many_sites, parameter_code="00060" ... ) # doctest: +SKIP @@ -310,7 +310,7 @@ def chunk_granularity(level: GranularityLevel) -> Iterator[None]: -------- ChunkPlan._refine : the planning-side effect of the level. """ - with _granularity(_resolve_granularity(level)): + with _parallel_chunks(_resolve_level(level)): yield @@ -734,7 +734,7 @@ def multi_value_chunked( completion via :meth:`ChunkedCall.resume`. The plan splits multi-value list params and the cql-text filter so each sub-request URL fits the byte limit; an already-fitting request is a one-step plan, unless an - active :func:`chunk_granularity` block asks the plan to fan out more + active :func:`parallel_chunks` block asks the plan to fan out more finely. See the module docstring for the concurrency model. Parameters @@ -779,13 +779,13 @@ def wrapper( finalize: _Finalize = _passthrough_result, ) -> tuple[pd.DataFrame, Any]: limit = _OGC_URL_BYTE_LIMIT if url_limit is None else url_limit - # Read the granularity dial from the ambient set by - # ``chunk_granularity`` (0 = off outside any such block; otherwise the + # Read the parallel_chunks dial from the ambient set by + # ``parallel_chunks`` (0 = off outside any such block; otherwise the # per-axis sub-chunk cap). It only affects *planning*, done here up # front, so a later resume — which re-issues the already-planned # sub-requests — needs no snapshot. plan = ChunkPlan( - args, build_request, limit, max_chunks_per_axis=_granularity.get() + args, build_request, limit, max_chunks_per_axis=_parallel_chunks.get() ) retry_policy = RetryPolicy.from_env() # The concurrency cap is resolved inside ``resume()`` from diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index 58bdad65..1e8bb9d3 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -296,7 +296,7 @@ def _split_at(chunks: list[list[str]], idx: int) -> None: (each atom survives, exactly once) and *contiguous, deterministic order* (resume and :meth:`ChunkPlan.iter_sub_args` depend on it). Kept in one place so those invariants can't drift between :meth:`ChunkPlan._plan` - (byte-driven) and :meth:`ChunkPlan._refine` (granularity-driven). + (byte-driven) and :meth:`ChunkPlan._refine` (fan-out-driven). """ chunk = chunks[idx] mid = len(chunk) // 2 @@ -337,7 +337,7 @@ class ChunkPlan: cartesian product across axes, never fewer than the byte budget already forces) — capped as a whole, not per axis, so several multi-value axes can't multiply past the cap. Set from the resolved - :func:`~dataretrieval.ogc.chunking.chunk_granularity` level; see + :func:`~dataretrieval.ogc.chunking.parallel_chunks` level; see :meth:`_refine`. Attributes @@ -379,7 +379,7 @@ def __init__( axes = _extract_axes(args) if not axes: - # No chunkable axis: nothing to split, and ``granularity`` has + # No chunkable axis: nothing to split, and ``parallel_chunks`` has # nothing to act on either. If the single request fits, run it # verbatim (the common passthrough). ``_safe_request_bytes`` treats # an un-constructable URL (httpx.InvalidURL, > 64 KB) as over budget. @@ -494,8 +494,8 @@ def _plan( def _refine(self, max_chunks_per_axis: int) -> None: """ Fan the plan out more finely than the byte budget alone requires — - the granularity dial (see - :func:`~dataretrieval.ogc.chunking.chunk_granularity` for why a caller + the parallel_chunks dial (see + :func:`~dataretrieval.ogc.chunking.parallel_chunks` for why a caller would want this). Caps the plan's *total* sub-request count (:attr:`total`, the @@ -514,7 +514,7 @@ def _refine(self, max_chunks_per_axis: int) -> None: ---------- max_chunks_per_axis : int Soft cap on the plan's total sub-request count (``0`` = off), - the resolved granularity level. Despite the name — kept for the + the resolved parallel_chunks level. Despite the name — kept for the public dial's "sub-chunks per multi-value argument" framing, which matches this cap exactly in the common single-axis case — multi-axis plans are capped on the product, not per axis. diff --git a/dataretrieval/waterdata/__init__.py b/dataretrieval/waterdata/__init__.py index d70a3d74..7a987442 100644 --- a/dataretrieval/waterdata/__init__.py +++ b/dataretrieval/waterdata/__init__.py @@ -9,7 +9,7 @@ from __future__ import annotations -from dataretrieval.ogc.chunking import GranularityLevel, chunk_granularity +from dataretrieval.ogc.chunking import ParallelChunksLevel, parallel_chunks from dataretrieval.ogc.filters import FILTER_LANG # Public API exports @@ -51,8 +51,8 @@ "PROFILE_LOOKUP", "SERVICES", "WATERDATA_SERVICES", - "GranularityLevel", - "chunk_granularity", + "ParallelChunksLevel", + "parallel_chunks", "get_channel", "get_codes", "get_combined_metadata", diff --git a/docs/source/userguide/errors.rst b/docs/source/userguide/errors.rst index 0c4429c0..7af7bada 100644 --- a/docs/source/userguide/errors.rst +++ b/docs/source/userguide/errors.rst @@ -106,7 +106,7 @@ extra quota *as long as each sub-request still spans many pages* (ten states pulled as one request then page nearly as many times as ten per-state requests would; a split that leaves each sub-request only a page or two adds its partial final page). So if you *know* your pull is large you can ask for a finer split -with ``chunk_granularity`` -- trading roughly the same pages for more, smaller +with ``parallel_chunks`` -- trading roughly the same pages for more, smaller sub-requests, which gives smoother progress, more even concurrency, and a smaller unit of retry/resume. It is a scoped ``with`` block, so an aggressive setting can't leak into unrelated calls and @@ -116,7 +116,7 @@ accidentally spend quota: from dataretrieval import waterdata - with waterdata.chunk_granularity("high"): + with waterdata.parallel_chunks("high"): df, md = waterdata.get_daily( monitoring_location_id=many_sites, parameter_code="00060" ) diff --git a/tests/utils_test.py b/tests/utils_test.py index 8ca4630d..dca00218 100644 --- a/tests/utils_test.py +++ b/tests/utils_test.py @@ -208,21 +208,21 @@ def test_chunk_interruptions_exported_at_top_level(self): dataretrieval.ChunkInterrupted, dataretrieval.DataRetrievalError ) - def test_chunk_granularity_exported_at_top_level_and_waterdata(self): - """The ``chunk_granularity`` context manager and its ``GranularityLevel`` + def test_parallel_chunks_exported_at_top_level_and_waterdata(self): + """The ``parallel_chunks`` context manager and its ``ParallelChunksLevel`` type are reachable both from the top level (``from dataretrieval import - chunk_granularity``) and from the user-facing ``dataretrieval.waterdata`` + parallel_chunks``) and from the user-facing ``dataretrieval.waterdata`` namespace, and both resolve to the single objects defined in ``dataretrieval.ogc.chunking``.""" import dataretrieval from dataretrieval import waterdata from dataretrieval.ogc import chunking - assert dataretrieval.chunk_granularity is chunking.chunk_granularity - assert waterdata.chunk_granularity is chunking.chunk_granularity - assert dataretrieval.GranularityLevel is chunking.GranularityLevel - assert waterdata.GranularityLevel is chunking.GranularityLevel - for name in ("chunk_granularity", "GranularityLevel"): + assert dataretrieval.parallel_chunks is chunking.parallel_chunks + assert waterdata.parallel_chunks is chunking.parallel_chunks + assert dataretrieval.ParallelChunksLevel is chunking.ParallelChunksLevel + assert waterdata.ParallelChunksLevel is chunking.ParallelChunksLevel + for name in ("parallel_chunks", "ParallelChunksLevel"): assert name in dataretrieval.__all__ assert name in waterdata.__all__ diff --git a/tests/waterdata_chunking_test.py b/tests/waterdata_chunking_test.py index 055db44b..98cb965b 100644 --- a/tests/waterdata_chunking_test.py +++ b/tests/waterdata_chunking_test.py @@ -40,15 +40,15 @@ from dataretrieval.ogc import chunking as _chunking from dataretrieval.ogc import retry as _retry_mod from dataretrieval.ogc.chunking import ( - _GRANULARITY_LEVELS, - _GRANULARITY_MAX_CHUNKS, + _LEVEL_CAPS, + _MAX_PARALLEL_CHUNKS, ChunkedCall, _chunked_client, - _granularity, - _resolve_granularity, - chunk_granularity, + _parallel_chunks, + _resolve_level, get_active_client, multi_value_chunked, + parallel_chunks, ) from dataretrieval.ogc.interruptions import ( ChunkInterrupted, @@ -2113,14 +2113,14 @@ async def fetch(args): # --------------------------------------------------------------------------- -# Chunk granularity: the opt-in dial (``"low"`` / ``"medium"`` / ``"high"``) to +# Parallel chunks: the opt-in dial (``"low"`` / ``"medium"`` / ``"high"``) to # fan a query out MORE finely than the byte limit alone requires -# (``ChunkPlan._refine`` + the ``chunk_granularity`` context manager). +# (``ChunkPlan._refine`` + the ``parallel_chunks`` context manager). # ``_fake_build``'s base is 200 bytes, so a handful of short atoms sits far # under ``url_limit=8000`` — the byte pass passes it through untouched, and any -# splitting below is the granularity cap alone. ``ChunkPlan`` takes the resolved -# integer cap (``max_chunks_per_axis``) directly; ``chunk_granularity`` / -# ``_resolve_granularity`` map the level names onto it. The cap bounds the +# splitting below is the parallel_chunks cap alone. ``ChunkPlan`` takes the resolved +# integer cap (``max_chunks_per_axis``) directly; ``parallel_chunks`` / +# ``_resolve_level`` map the level names onto it. The cap bounds the # plan's *total* sub-request count (the cartesian product across axes), not # each axis independently — see ``test_cap_caps_the_total_across_axes``. # --------------------------------------------------------------------------- @@ -2166,9 +2166,9 @@ def test_cap_ramps_then_saturates(max_chunks_per_axis, expected_pieces): def test_cap_bounds_fan_out_for_a_long_axis(): """The cap is a quota guardrail: at the ``"high"`` cap a 100-atom axis fans into ``cap`` pieces — NOT 100 singletons — so an accidental - ``chunk_granularity("high")`` on a huge list can't detonate into hundreds + ``parallel_chunks("high")`` on a huge list can't detonate into hundreds of sub-requests. Every atom is still covered exactly once.""" - high = _GRANULARITY_LEVELS["high"] + high = _LEVEL_CAPS["high"] atoms = [f"X{i:03d}" for i in range(100)] plan = ChunkPlan( {"monitoring_location_id": atoms}, @@ -2245,7 +2245,7 @@ def test_cap_bounds_fan_out_across_many_axes(): at the ``"high"`` cap still top out at ``high`` sub-requests total, not ``high ** 3`` — the property the single-axis-only cap in the original implementation did not guarantee.""" - high = _GRANULARITY_LEVELS["high"] + high = _LEVEL_CAPS["high"] # Three chunkable axes (two list axes + the filter OR-axis), each with 10 # atoms — under the old per-axis cap this would have been high**3. args = { @@ -2267,23 +2267,24 @@ def test_cap_does_not_mask_unchunkable(): @pytest.mark.parametrize("level", ["low", "medium", "high"]) -def test_resolve_granularity_maps_each_level_to_its_cap(level): +def test_resolve_level_maps_each_level_to_its_cap(level): """Each level name resolves to its total sub-request cap from the table (a positive int).""" - assert _resolve_granularity(level) == _GRANULARITY_LEVELS[level] - assert _resolve_granularity(level) >= 1 + assert _resolve_level(level) == _LEVEL_CAPS[level] + assert _resolve_level(level) >= 1 -def test_granularity_levels_ordered_and_spaced(): +def test_parallel_chunks_levels_ordered_and_spaced(): """The three caps hold the properties callers rely on: strictly increasing, ``"high"`` saturating the declared ceiling, ``"low"`` still a real split - (>= 2), and each level spaced 4x from the next. The ceiling is a granularity - constant, deliberately independent of the concurrency default.""" - low = _GRANULARITY_LEVELS["low"] - medium = _GRANULARITY_LEVELS["medium"] - high = _GRANULARITY_LEVELS["high"] + (>= 2), and each level spaced 4x from the next. The ceiling is a + parallel_chunks constant, deliberately independent of the concurrency + default.""" + low = _LEVEL_CAPS["low"] + medium = _LEVEL_CAPS["medium"] + high = _LEVEL_CAPS["high"] assert low < medium < high - assert high == _GRANULARITY_MAX_CHUNKS + assert high == _MAX_PARALLEL_CHUNKS assert low >= 2 assert medium == high // 4 assert low == medium // 4 @@ -2300,41 +2301,41 @@ def test_granularity_levels_ordered_and_spaced(): ["low"], # unhashable → the ``not in`` check still rejects it cleanly ], ) -def test_resolve_granularity_rejects_everything_but_the_three_levels(bad): +def test_resolve_level_rejects_everything_but_the_three_levels(bad): """Only the three exact level strings are accepted; every other value — a representative of each rejected shape (dead keyword, wrong case, whitespace, old int, ``None``, unhashable) — raises ``ValueError`` so a typo fails loudly.""" - with pytest.raises(ValueError, match="chunk_granularity level must be"): - _resolve_granularity(bad) + with pytest.raises(ValueError, match="parallel_chunks level must be"): + _resolve_level(bad) -def test_chunk_granularity_scopes_and_restores_the_ambient(): +def test_parallel_chunks_scopes_and_restores_the_ambient(): """The context manager resolves the level to its cap, publishes it on the ambient for the block, and restores the previous value on exit — including proper nesting.""" - assert _granularity.get() == 0 - with chunk_granularity("high"): - assert _granularity.get() == _GRANULARITY_LEVELS["high"] - with chunk_granularity("low"): - assert _granularity.get() == _GRANULARITY_LEVELS["low"] - assert _granularity.get() == _GRANULARITY_LEVELS["high"] # outer restored - assert _granularity.get() == 0 # default (off) outside any block + assert _parallel_chunks.get() == 0 + with parallel_chunks("high"): + assert _parallel_chunks.get() == _LEVEL_CAPS["high"] + with parallel_chunks("low"): + assert _parallel_chunks.get() == _LEVEL_CAPS["low"] + assert _parallel_chunks.get() == _LEVEL_CAPS["high"] # outer restored + assert _parallel_chunks.get() == 0 # default (off) outside any block -def test_chunk_granularity_validates_on_entry(): +def test_parallel_chunks_validates_on_entry(): """An invalid level raises at ``with`` entry — before any request is issued — and leaves the ambient untouched.""" - with pytest.raises(ValueError, match="chunk_granularity level must be"): - with chunk_granularity("aggressive"): + with pytest.raises(ValueError, match="parallel_chunks level must be"): + with parallel_chunks("aggressive"): pass - assert _granularity.get() == 0 + assert _parallel_chunks.get() == 0 -def test_chunk_granularity_high_drives_end_to_end_fan_out(): +def test_parallel_chunks_high_drives_end_to_end_fan_out(): """End-to-end: the same fitting request passes through as a single call by default, but fans into several sub-requests inside a - ``chunk_granularity("high")`` block — and the combined result still + ``parallel_chunks("high")`` block — and the combined result still recovers every atom exactly once.""" sites = [f"S{i:02d}" for i in range(8)] @@ -2352,7 +2353,7 @@ async def fetch(args): assert sorted(df_plain["site"]) == sorted(sites) calls.clear() - with chunk_granularity("high"): + with parallel_chunks("high"): df_fine, _ = fetch({"monitoring_location_id": sites}) # 8 atoms at the "high" cap (>= 8) → 8 singleton sub-requests. assert len(calls) == 8 @@ -2362,10 +2363,10 @@ async def fetch(args): assert sorted(df_fine["site"]) == sorted(sites) -def test_chunk_granularity_low_is_a_gentle_split(): +def test_parallel_chunks_low_is_a_gentle_split(): """``"low"`` is the mildest opt-in: an under-limit request fans into just the ``"low"`` cap's worth of pieces, not singletons.""" - low = _GRANULARITY_LEVELS["low"] + low = _LEVEL_CAPS["low"] sites = [f"S{i:02d}" for i in range(8)] calls: list[int] = [] @@ -2374,7 +2375,7 @@ async def fetch(args): calls.append(len(args["monitoring_location_id"])) return pd.DataFrame(), _ok_response() - with chunk_granularity("low"): + with parallel_chunks("low"): fetch({"monitoring_location_id": sites}) # low cap → that many sub-requests, together covering all 8 sites. assert len(calls) == low From 115a2302033efadd147e59b27e70f3104ada23ff Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Wed, 8 Jul 2026 19:03:11 -0500 Subject: [PATCH 04/13] refactor(waterdata): make parallel_chunks take an integer, not low/medium/high Replace the three-tier "low"/"medium"/"high" enum with a plain positive integer: parallel_chunks(n) fans a call out into n sub-requests. More expressive (any n, not just 2/8/32), precise, and mirrors the int-valued API_USGS_CONCURRENT. Removes ParallelChunksLevel, the _LEVEL_CAPS/_MAX_PARALLEL_CHUNKS constants, and _resolve_level; validation is now an inline positive-int check (rejects 0, negatives, floats, bool, str). n is bounded below by the byte-limit minimum and above by the number of values to split; n=1 is an explicit no-op. Docstrings, NEWS, user guide, README, exports, and tests updated; 2/8/32 remain as documented examples. Co-Authored-By: Claude Opus 4.8 (1M context) --- NEWS.md | 2 +- README.md | 40 +++---- dataretrieval/__init__.py | 9 +- dataretrieval/ogc/chunking.py | 132 ++++++++--------------- dataretrieval/ogc/planning.py | 14 +-- dataretrieval/waterdata/__init__.py | 3 +- docs/source/userguide/errors.rst | 30 +++--- tests/utils_test.py | 16 ++- tests/waterdata_chunking_test.py | 156 ++++++++++++---------------- 9 files changed, 166 insertions(+), 236 deletions(-) diff --git a/NEWS.md b/NEWS.md index e3d6f32b..aee2ef07 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,4 +1,4 @@ -**07/01/2026:** Added `waterdata.parallel_chunks(...)` — a context manager to control how finely the OGC `waterdata` getters split multi-value requests. By default the getters chunk only as much as the server's ~8 KB URL limit forces (the fewest sub-requests). But because every sub-request paginates, splitting a large result further costs little or no extra quota when each sub-request still spans many pages, so a caller who *knows* their pull is large can opt into a finer split for smoother progress, more even concurrency, and a smaller unit of retry/resume: `with waterdata.parallel_chunks("high"): df, md = waterdata.get_daily(monitoring_location_id=many_sites)`. The level is one of `"low"`, `"medium"`, or `"high"` — typed as `waterdata.ParallelChunksLevel` (a `typing.Literal`), so a type checker rejects any other value and an invalid string raises `ValueError` at the `with`. Each level caps the *total* number of sub-requests the call is split into — `"low"` / `"medium"` / `"high"` cap at 2 / 8 / 32 overall, across every multi-value argument combined, not per argument. That ceiling is a parallel_chunks constant, deliberately independent of the fan-out concurrency (`API_USGS_CONCURRENT`): how finely a query splits is orthogonal to how many sub-requests run at once. Capping the aggressive end at 32 bounds the blast radius — an accidental `"high"` on a huge list, or a call with several multi-value arguments, can't explode into thousands of sub-requests. There is no "off" level: not entering the block *is* off. It is deliberately a scoped `with` block rather than an automatic behavior or an environment variable, since the library can't tell in advance whether a query is large — a short-window query might fit in a single page, where extra chunks would only burn quota. Also exported at the top level as `dataretrieval.parallel_chunks`. +**07/01/2026:** Added `waterdata.parallel_chunks(n)` — a context manager to control how finely the OGC `waterdata` getters split multi-value requests. By default the getters chunk only as much as the server's ~8 KB URL limit forces (the fewest sub-requests). But because every sub-request paginates, splitting a large result further costs little or no extra quota when each sub-request still spans many pages, so a caller who *knows* their pull is large can opt into a finer split for smoother progress, more even concurrency, and a smaller unit of retry/resume: `with waterdata.parallel_chunks(32): df, md = waterdata.get_daily(monitoring_location_id=many_sites)`. `n` is a positive integer (e.g. `2`, `8`, `32`) — the number of sub-requests to fan the call out into; a non-integer or non-positive value raises `ValueError` at the `with`. It caps the *total* sub-request count across every multi-value argument combined (not per argument), bounded below by what the byte limit already forces and above by how many values there are to split, so several multi-value arguments can't multiply past it and `n=1` asks for no extra fan-out. Each sub-request costs a request against your hourly rate limit, and because how many run *at once* is capped separately by `API_USGS_CONCURRENT` (default 32) an `n` beyond that adds quota without adding parallelism, so the useful range is roughly `2` up to `API_USGS_CONCURRENT`. It is deliberately a scoped `with` block rather than an automatic behavior or an environment variable, since the library can't tell in advance whether a query is large — a short-window query might fit in a single page, where extra chunks would only burn quota. Also exported at the top level as `dataretrieval.parallel_chunks`. **06/23/2026:** **Breaking change (1.2.0):** the minimum supported Python is now **3.10** (`requires-python = ">=3.10"`). 3.9 support was already effectively broken — the `waterdata` module's dependencies (`anyio`, the test stack) require 3.10+, and the `waterdata` test modules already skipped on <3.10. `anyio` is now declared as a direct dependency (it is imported directly by `waterdata`), and the CI/ruff/mypy targets move to 3.10. Also fully removed the deprecated `variable_info` metadata property: the `NWIS_Metadata` override only warned and returned `None` (it relied on the defunct `get_pmcodes`), and the `BaseMetadata` abstract is gone too since nothing implemented it — accessing `.variable_info` now raises `AttributeError`. `site_info` is unaffected. diff --git a/README.md b/README.md index 71f3b329..73aa273e 100644 --- a/README.md +++ b/README.md @@ -111,11 +111,12 @@ By default the getters split a multi-value request only as far as the server's ~8 KB URL limit forces — the fewest sub-requests. For a **large, paginated** pull that is needlessly conservative: every sub-request pages through its own results, so dividing the query into more, smaller sub-requests lets those pages -be fetched **in parallel**. `parallel_chunks` opts a single call into that -finer split. It pays off only when the result is large enough to span many -pages *and* the query has a multi-value argument to divide (such as a list of -monitoring locations); on a small query — or one with nothing to split — it just -adds requests, so it is a deliberate, scoped `with` block, never the default. +be fetched **in parallel**. `parallel_chunks(n)` opts a single call into that +finer split, fanning it out into `n` sub-requests. It pays off only when the +result is large enough to span many pages *and* the query has a multi-value +argument to divide (such as a list of monitoring locations); on a small query — +or one with nothing to split — it just adds requests, so it is a deliberate, +scoped `with` block, never the default. ```python from dataretrieval import waterdata @@ -124,7 +125,7 @@ from dataretrieval import waterdata # enough to span many pages, so it profits from a finer split. sites, _ = waterdata.get_monitoring_locations(state="Ohio", site_type_code="ST") -with waterdata.parallel_chunks("high"): # "low" | "medium" | "high" +with waterdata.parallel_chunks(32): # fan out into 32 sub-requests df, md = waterdata.get_daily( monitoring_location_id=sites["monitoring_location_id"].tolist(), parameter_code="00060", # discharge @@ -132,26 +133,27 @@ with waterdata.parallel_chunks("high"): # "low" | "medium" | "high" ) ``` -`"high"` fans a call out into up to 32 sub-requests, `"medium"` up to 8, and -`"low"` up to 2 — a fixed ceiling, so an accidental setting on a huge list can't -explode into thousands of requests. +`n` is the number of sub-requests to fan the call out into. It is capped by how +many values there are to split, and each sub-request costs a request against +your hourly [rate limit](https://api.waterdata.usgs.gov/signup/); since how many +run *at once* is capped separately by `API_USGS_CONCURRENT` (default 32), the +useful range is roughly `2` up to that value. Benchmark — 271 Ohio discharge sites (`get_daily`, `parameter_code="00060"`), -cold cache, each level run against its own time window so results are not -cache-served, with the page size fixed so every level fetches roughly the same +cold cache, each `n` run against its own time window so results are not +cache-served, with the page size fixed so every run fetches roughly the same number of pages (isolating the effect of parallelism): -| level | parallelism | pages | wall-clock | speedup | -| -------------- | ----------- | ----- | ------------------------- | ------- | -| `default` | 1 | 32 | 23.9 s / 27.4 s (2 runs) | 1× | -| `"medium"` (8) | 8 | 37 | 4.1 s / 4.3 s | ~6× | -| `"high"` (32) | 32 | 44 | 2.0 s | ~12× | +| `n` | parallelism | pages | wall-clock | speedup | +| ---- | ----------- | ----- | ------------------------- | ------- | +| off | 1 | 32 | 23.9 s / 27.4 s (2 runs) | 1× | +| `8` | 8 | 37 | 4.1 s / 4.3 s | ~6× | +| `32` | 32 | 44 | 2.0 s | ~12× | The gain comes from overlapping each sub-request's per-page latency and server-side work — a genuinely large pull at the default page size shows a -similar multi-fold speedup. The extra sub-requests each cost one request against -your hourly [rate limit](https://api.waterdata.usgs.gov/signup/), so reserve the -aggressive levels for pulls you know are large. +similar multi-fold speedup. The extra sub-requests each cost quota, so reserve a +large `n` for pulls you know are large. Visit the [API Reference](https://doi-usgs.github.io/dataretrieval-python/reference/waterdata.html) diff --git a/dataretrieval/__init__.py b/dataretrieval/__init__.py index 567be6ab..469fe0f5 100644 --- a/dataretrieval/__init__.py +++ b/dataretrieval/__init__.py @@ -44,10 +44,10 @@ URLTooLong, ) -# Parallel-chunks control (a context manager) and its level type. Defined with -# the chunker in ``dataretrieval.ogc.chunking``; surfaced here for a stable -# public path ``from dataretrieval import parallel_chunks``. -from dataretrieval.ogc.chunking import ParallelChunksLevel, parallel_chunks +# Parallel-chunks control (a context manager). Defined with the chunker in +# ``dataretrieval.ogc.chunking``; surfaced here for a stable public path +# ``from dataretrieval import parallel_chunks``. +from dataretrieval.ogc.chunking import parallel_chunks # Resumable chunk-interruption exceptions. They are defined in # ``dataretrieval.ogc.interruptions`` rather than ``dataretrieval.exceptions`` @@ -100,6 +100,5 @@ "ServiceInterrupted", # parallel-chunks control (defined in ogc.chunking) "parallel_chunks", - "ParallelChunksLevel", "__version__", ] diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index 3d0e9aff..1a3b37b6 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -11,9 +11,9 @@ Parallel chunks: the planner is conservative by default — it splits only as far as the byte limit forces. A caller who knows their result is large can opt into a -finer split via the ``parallel_chunks`` context manager -(``"low"`` / ``"medium"`` / ``"high"``); the resolved cap drives -:meth:`ChunkPlan._refine`. See ``parallel_chunks`` for the why and the when. +finer split via the ``parallel_chunks(n)`` context manager, which fans the query +out into ``n`` parallel sub-requests; ``n`` drives :meth:`ChunkPlan._refine`. See +``parallel_chunks`` for the why and the when. This module owns the *execution* half — the event loop and bounded concurrency that drive a plan to completion (``ChunkedCall``) plus the @@ -77,7 +77,7 @@ from collections.abc import Callable, Iterator from contextlib import contextmanager from contextvars import copy_context -from typing import Any, Literal, cast, get_args +from typing import Any, cast import httpx import pandas as pd @@ -182,70 +182,16 @@ def get_active_client() -> httpx.AsyncClient | None: # Parallel-chunks dial: opt-in to fan a query out *more finely* than the byte # limit alone requires. Scoped to a ``with parallel_chunks(...):`` block (a # ContextVar), deliberately NOT an env var (see :func:`parallel_chunks` for -# why). The ambient holds the resolved cap on the plan's total sub-request -# count; ``0`` (the default, outside any block) means "chunk only as much as -# the byte limit needs". +# why). The ambient holds ``n`` — the requested cap on the plan's total +# sub-request count; ``0`` (the default, outside any block) means "chunk only as +# much as the byte limit needs". _parallel_chunks: Ambient[int] = Ambient("ogc_parallel_chunks", 0) -#: The three accepted parallel_chunks levels, as a typing ``Literal`` so a type -#: checker rejects any other value at the call site. -ParallelChunksLevel = Literal["low", "medium", "high"] - -#: Valid levels derived from the type, so ``ParallelChunksLevel`` stays the single -#: source of truth for what ``_resolve_level`` accepts (mirrors the -#: ``get_args``-based ``_VALID_ON_TIE`` / ``_VALID_FILE_TYPES`` in sibling -#: modules). -_VALID_LEVELS: tuple[ParallelChunksLevel, ...] = get_args(ParallelChunksLevel) - -# The parallel_chunks ceiling on the plan's total sub-request count — -# deliberately NOT tied to the concurrency default: fan-out *volume* (how many -# sub-requests a query becomes) is orthogonal to how many run at once -# (``API_USGS_CONCURRENT``). 32 is a sane ceiling on the whole call (across -# every axis, not per axis — see ``ChunkPlan._refine``) that bounds the blast -# radius of an accidental ``"high"`` on a very long list or several multi-value -# arguments at once; the milder levels are a quarter and a sixteenth of it (the -# three are spaced 4x apart). -_MAX_PARALLEL_CHUNKS = 32 -_LEVEL_CAPS: dict[str, int] = { - "low": _MAX_PARALLEL_CHUNKS // 16, # 2 - "medium": _MAX_PARALLEL_CHUNKS // 4, # 8 - "high": _MAX_PARALLEL_CHUNKS, # 32 -} - - -def _resolve_level(level: ParallelChunksLevel) -> int: - """ - Map a parallel_chunks level name to its total sub-request cap. - - Parameters - ---------- - level : {"low", "medium", "high"} - The user-supplied level. - - Returns - ------- - int - The maximum total sub-requests for the whole call at that level - (``2`` / ``8`` / ``32``). - - Raises - ------ - ValueError - If ``level`` is anything other than ``"low"``, ``"medium"``, or - ``"high"`` — raised eagerly (at the ``with`` statement) so a typo fails - loudly rather than silently doing nothing. - """ - if level not in _VALID_LEVELS: - raise ValueError( - f"parallel_chunks level must be one of {_VALID_LEVELS}; got {level!r}." - ) - return _LEVEL_CAPS[level] - @contextmanager -def parallel_chunks(level: ParallelChunksLevel) -> Iterator[None]: +def parallel_chunks(n: int) -> Iterator[None]: """ - Scope how finely the OGC getters chunk multi-value requests. + Fan the OGC getters' multi-value requests out into ``n`` parallel sub-requests. By default the Water Data / NGWMN getters chunk a request only as much as the server's ~8 KB URL-byte limit forces — the fewest sub-requests that @@ -267,26 +213,27 @@ def parallel_chunks(level: ParallelChunksLevel) -> Iterator[None]: automatic behavior or a process-wide environment variable — scoping it to a ``with`` block keeps an aggressive setting from leaking into unrelated calls and accidentally spending quota. Outside any block the getters use the - conservative default; there is no "off" level because *not* entering the - block is off. Only the OGC getters (Water Data, NGWMN) read this; wrapping a - legacy NWIS call in the block is a harmless no-op. + conservative default. Only the OGC getters (Water Data, NGWMN) read this; + wrapping a legacy NWIS call in the block is a harmless no-op. Parameters ---------- - level : {"low", "medium", "high"} - How aggressively to chunk within the block. Each level caps the - *total* number of sub-requests the call is split into — ``2`` / ``8`` - / ``32`` for ``"low"`` / ``"medium"`` / ``"high"`` — or one - sub-request per remaining atom once that's fewer than the cap. The - cap applies to the whole call, not per multi-value argument: with - several multi-value arguments the axes are refined together against - the same shared ceiling (their cartesian product does not multiply - past it). The ceiling is fixed (it is *not* tied to - ``API_USGS_CONCURRENT``: how finely a query splits is orthogonal to - how many sub-requests run at once); capping ``"high"`` at ``32`` keeps - an accidental aggressive level on a very long list — or several - multi-value arguments at once — from exploding into thousands of - sub-requests. + n : int + The number of sub-requests to fan the whole call out into — a positive + integer such as ``2``, ``8``, or ``32``. It caps the plan's *total* + sub-request count (the cartesian product across every multi-value + argument combined, not per argument), so several multi-value arguments + cannot multiply past it. The actual count is bounded below by what the + ~8 KB URL limit already forces and above by the number of values there + are to split, so an ``n`` larger than the input allows simply yields one + sub-request per value; ``n=1`` asks for no extra fan-out. + + Each sub-request fetches at least one page, so it costs at least one + request against your hourly rate limit — a larger ``n`` spends more + quota. And because how many sub-requests run *at once* is capped + separately by ``API_USGS_CONCURRENT`` (default 32), an ``n`` beyond that + adds quota without adding parallelism; the useful range is roughly ``2`` + up to ``API_USGS_CONCURRENT``. Yields ------ @@ -295,22 +242,29 @@ def parallel_chunks(level: ParallelChunksLevel) -> Iterator[None]: Raises ------ ValueError - If ``level`` isn't ``"low"``, ``"medium"``, or ``"high"`` — raised on - ``with`` entry, before any request is issued. + If ``n`` is not a positive integer — raised on ``with`` entry, before + any request is issued, so a bad value fails loudly rather than silently + doing nothing. Examples -------- >>> from dataretrieval import waterdata - >>> with waterdata.parallel_chunks("high"): + >>> with waterdata.parallel_chunks(32): ... df, md = waterdata.get_daily( ... monitoring_location_id=many_sites, parameter_code="00060" ... ) # doctest: +SKIP See Also -------- - ChunkPlan._refine : the planning-side effect of the level. + ChunkPlan._refine : the planning-side effect of ``n``. """ - with _parallel_chunks(_resolve_level(level)): + # ``bool`` is an ``int`` subclass but nonsensical here; reject it explicitly. + if not isinstance(n, int) or isinstance(n, bool) or n < 1: + raise ValueError( + f"parallel_chunks(n): n must be a positive integer (e.g. 2, 8, 32); " + f"got {n!r}." + ) + with _parallel_chunks(n): yield @@ -779,11 +733,11 @@ def wrapper( finalize: _Finalize = _passthrough_result, ) -> tuple[pd.DataFrame, Any]: limit = _OGC_URL_BYTE_LIMIT if url_limit is None else url_limit - # Read the parallel_chunks dial from the ambient set by + # Read the parallel_chunks dial ``n`` from the ambient set by # ``parallel_chunks`` (0 = off outside any such block; otherwise the - # per-axis sub-chunk cap). It only affects *planning*, done here up - # front, so a later resume — which re-issues the already-planned - # sub-requests — needs no snapshot. + # requested total sub-request cap). It only affects *planning*, done + # here up front, so a later resume — which re-issues the + # already-planned sub-requests — needs no snapshot. plan = ChunkPlan( args, build_request, limit, max_chunks_per_axis=_parallel_chunks.get() ) diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index 1e8bb9d3..ebadcb36 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -336,8 +336,8 @@ class ChunkPlan: out to up to ``max_chunks_per_axis`` sub-requests overall (the cartesian product across axes, never fewer than the byte budget already forces) — capped as a whole, not per axis, so several - multi-value axes can't multiply past the cap. Set from the resolved - :func:`~dataretrieval.ogc.chunking.parallel_chunks` level; see + multi-value axes can't multiply past the cap. Set from the + :func:`~dataretrieval.ogc.chunking.parallel_chunks` ``n``; see :meth:`_refine`. Attributes @@ -513,11 +513,11 @@ def _refine(self, max_chunks_per_axis: int) -> None: Parameters ---------- max_chunks_per_axis : int - Soft cap on the plan's total sub-request count (``0`` = off), - the resolved parallel_chunks level. Despite the name — kept for the - public dial's "sub-chunks per multi-value argument" framing, - which matches this cap exactly in the common single-axis case — - multi-axis plans are capped on the product, not per axis. + Soft cap on the plan's total sub-request count (``0`` = off) — the + ``parallel_chunks(n)`` value. The parameter name is legacy: the cap + is on the whole plan (the cartesian product across axes), which + matches "sub-chunks per argument" only in the common single-axis + case — multi-axis plans are capped on the product, not per axis. """ if max_chunks_per_axis <= 0: return diff --git a/dataretrieval/waterdata/__init__.py b/dataretrieval/waterdata/__init__.py index 7a987442..eb231469 100644 --- a/dataretrieval/waterdata/__init__.py +++ b/dataretrieval/waterdata/__init__.py @@ -9,7 +9,7 @@ from __future__ import annotations -from dataretrieval.ogc.chunking import ParallelChunksLevel, parallel_chunks +from dataretrieval.ogc.chunking import parallel_chunks from dataretrieval.ogc.filters import FILTER_LANG # Public API exports @@ -51,7 +51,6 @@ "PROFILE_LOOKUP", "SERVICES", "WATERDATA_SERVICES", - "ParallelChunksLevel", "parallel_chunks", "get_channel", "get_codes", diff --git a/docs/source/userguide/errors.rst b/docs/source/userguide/errors.rst index 7af7bada..28da515f 100644 --- a/docs/source/userguide/errors.rst +++ b/docs/source/userguide/errors.rst @@ -106,7 +106,7 @@ extra quota *as long as each sub-request still spans many pages* (ten states pulled as one request then page nearly as many times as ten per-state requests would; a split that leaves each sub-request only a page or two adds its partial final page). So if you *know* your pull is large you can ask for a finer split -with ``parallel_chunks`` -- trading roughly the same pages for more, smaller +with ``parallel_chunks(n)`` -- trading roughly the same pages for more, smaller sub-requests, which gives smoother progress, more even concurrency, and a smaller unit of retry/resume. It is a scoped ``with`` block, so an aggressive setting can't leak into unrelated calls and @@ -116,24 +116,24 @@ accidentally spend quota: from dataretrieval import waterdata - with waterdata.parallel_chunks("high"): + with waterdata.parallel_chunks(32): df, md = waterdata.get_daily( monitoring_location_id=many_sites, parameter_code="00060" ) -The level is one of ``"low"``, ``"medium"``, or ``"high"`` (an invalid value -raises ``ValueError`` at the ``with``). Each caps the *total* number of -sub-requests the call is split into -- ``2`` / ``8`` / ``32`` for -``"low"`` / ``"medium"`` / ``"high"``, across every multi-value argument -combined, not per argument. That ceiling is fixed, deliberately -independent of the fan-out concurrency (``API_USGS_CONCURRENT``): how finely a -query splits is orthogonal to how many sub-requests run at once. Capping the -aggressive end at 32 is a guardrail -- an accidental ``"high"`` on a very long -list, or a call with several multi-value arguments, can't explode into -thousands of sub-requests. There is no "off" level: -simply don't enter the block unless you already expect a large, multi-page -result -- on a query that would have fit in a single page, extra chunks only -burn quota. +``n`` is a positive integer (e.g. ``2``, ``8``, ``32``) -- the number of +sub-requests to fan the call out into; a non-integer or non-positive value +raises ``ValueError`` at the ``with``. It caps the *total* sub-request count +across every multi-value argument combined (not per argument), bounded below by +what the byte limit already forces and above by how many values there are to +split, so several multi-value arguments can't multiply past it and ``n=1`` asks +for no extra fan-out. Each sub-request costs a request against your hourly rate +limit, and because how many run *at once* is capped separately by +``API_USGS_CONCURRENT`` (default 32) an ``n`` beyond that adds quota without +adding parallelism -- the useful range is roughly ``2`` up to +``API_USGS_CONCURRENT``. There is no "off" level: simply don't enter the block +unless you already expect a large, multi-page result -- on a query that would +have fit in a single page, extra chunks only burn quota. The full taxonomy ================= diff --git a/tests/utils_test.py b/tests/utils_test.py index dca00218..30950294 100644 --- a/tests/utils_test.py +++ b/tests/utils_test.py @@ -209,22 +209,18 @@ def test_chunk_interruptions_exported_at_top_level(self): ) def test_parallel_chunks_exported_at_top_level_and_waterdata(self): - """The ``parallel_chunks`` context manager and its ``ParallelChunksLevel`` - type are reachable both from the top level (``from dataretrieval import - parallel_chunks``) and from the user-facing ``dataretrieval.waterdata`` - namespace, and both resolve to the single objects defined in - ``dataretrieval.ogc.chunking``.""" + """The ``parallel_chunks`` context manager is reachable both from the top + level (``from dataretrieval import parallel_chunks``) and from the + user-facing ``dataretrieval.waterdata`` namespace, and both resolve to + the single object defined in ``dataretrieval.ogc.chunking``.""" import dataretrieval from dataretrieval import waterdata from dataretrieval.ogc import chunking assert dataretrieval.parallel_chunks is chunking.parallel_chunks assert waterdata.parallel_chunks is chunking.parallel_chunks - assert dataretrieval.ParallelChunksLevel is chunking.ParallelChunksLevel - assert waterdata.ParallelChunksLevel is chunking.ParallelChunksLevel - for name in ("parallel_chunks", "ParallelChunksLevel"): - assert name in dataretrieval.__all__ - assert name in waterdata.__all__ + assert "parallel_chunks" in dataretrieval.__all__ + assert "parallel_chunks" in waterdata.__all__ class Test_BaseMetadata: diff --git a/tests/waterdata_chunking_test.py b/tests/waterdata_chunking_test.py index 98cb965b..1ab57378 100644 --- a/tests/waterdata_chunking_test.py +++ b/tests/waterdata_chunking_test.py @@ -40,12 +40,9 @@ from dataretrieval.ogc import chunking as _chunking from dataretrieval.ogc import retry as _retry_mod from dataretrieval.ogc.chunking import ( - _LEVEL_CAPS, - _MAX_PARALLEL_CHUNKS, ChunkedCall, _chunked_client, _parallel_chunks, - _resolve_level, get_active_client, multi_value_chunked, parallel_chunks, @@ -2113,16 +2110,15 @@ async def fetch(args): # --------------------------------------------------------------------------- -# Parallel chunks: the opt-in dial (``"low"`` / ``"medium"`` / ``"high"``) to -# fan a query out MORE finely than the byte limit alone requires -# (``ChunkPlan._refine`` + the ``parallel_chunks`` context manager). -# ``_fake_build``'s base is 200 bytes, so a handful of short atoms sits far -# under ``url_limit=8000`` — the byte pass passes it through untouched, and any -# splitting below is the parallel_chunks cap alone. ``ChunkPlan`` takes the resolved -# integer cap (``max_chunks_per_axis``) directly; ``parallel_chunks`` / -# ``_resolve_level`` map the level names onto it. The cap bounds the -# plan's *total* sub-request count (the cartesian product across axes), not -# each axis independently — see ``test_cap_caps_the_total_across_axes``. +# Parallel chunks: the opt-in dial ``parallel_chunks(n)`` to fan a query out +# MORE finely than the byte limit alone requires (``ChunkPlan._refine`` + the +# ``parallel_chunks`` context manager). ``_fake_build``'s base is 200 bytes, so +# a handful of short atoms sits far under ``url_limit=8000`` — the byte pass +# passes it through untouched, and any splitting below is the ``n`` cap alone. +# ``ChunkPlan`` takes the integer cap (``max_chunks_per_axis``) directly; +# ``parallel_chunks(n)`` publishes ``n`` onto it. The cap bounds the plan's +# *total* sub-request count (the cartesian product across axes), not each axis +# independently — see ``test_cap_caps_the_total_across_axes``. # --------------------------------------------------------------------------- @@ -2164,11 +2160,11 @@ def test_cap_ramps_then_saturates(max_chunks_per_axis, expected_pieces): def test_cap_bounds_fan_out_for_a_long_axis(): - """The cap is a quota guardrail: at the ``"high"`` cap a 100-atom axis - fans into ``cap`` pieces — NOT 100 singletons — so an accidental - ``parallel_chunks("high")`` on a huge list can't detonate into hundreds - of sub-requests. Every atom is still covered exactly once.""" - high = _LEVEL_CAPS["high"] + """The cap holds fan-out to ``n``: at ``n=32`` a 100-atom axis fans into + ``n`` pieces — NOT 100 singletons — so ``parallel_chunks(32)`` on a huge + list can't detonate into hundreds of sub-requests. Every atom is still + covered exactly once.""" + high = 32 atoms = [f"X{i:03d}" for i in range(100)] plan = ChunkPlan( {"monitoring_location_id": atoms}, @@ -2242,10 +2238,10 @@ def test_cap_caps_the_total_across_axes(): def test_cap_bounds_fan_out_across_many_axes(): """The guardrail holds regardless of axis count: three multi-value axes - at the ``"high"`` cap still top out at ``high`` sub-requests total, not - ``high ** 3`` — the property the single-axis-only cap in the original - implementation did not guarantee.""" - high = _LEVEL_CAPS["high"] + at ``n=32`` still top out at ``n`` sub-requests total, not ``n ** 3`` — + the property the single-axis-only cap in the original implementation did + not guarantee.""" + high = 32 # Three chunkable axes (two list axes + the filter OR-axis), each with 10 # atoms — under the old per-axis cap this would have been high**3. args = { @@ -2266,77 +2262,62 @@ def test_cap_does_not_mask_unchunkable(): ChunkPlan(args, _fake_build, url_limit=10, max_chunks_per_axis=32) -@pytest.mark.parametrize("level", ["low", "medium", "high"]) -def test_resolve_level_maps_each_level_to_its_cap(level): - """Each level name resolves to its total sub-request cap from the table - (a positive int).""" - assert _resolve_level(level) == _LEVEL_CAPS[level] - assert _resolve_level(level) >= 1 - - -def test_parallel_chunks_levels_ordered_and_spaced(): - """The three caps hold the properties callers rely on: strictly increasing, - ``"high"`` saturating the declared ceiling, ``"low"`` still a real split - (>= 2), and each level spaced 4x from the next. The ceiling is a - parallel_chunks constant, deliberately independent of the concurrency - default.""" - low = _LEVEL_CAPS["low"] - medium = _LEVEL_CAPS["medium"] - high = _LEVEL_CAPS["high"] - assert low < medium < high - assert high == _MAX_PARALLEL_CHUNKS - assert low >= 2 - assert medium == high // 4 - assert low == medium // 4 +def test_parallel_chunks_publishes_n_on_the_ambient(): + """The context manager publishes ``n`` on the ambient for the block and + restores the previous value on exit — including proper nesting.""" + assert _parallel_chunks.get() == 0 + with parallel_chunks(32): + assert _parallel_chunks.get() == 32 + with parallel_chunks(2): + assert _parallel_chunks.get() == 2 + assert _parallel_chunks.get() == 32 # outer restored + assert _parallel_chunks.get() == 0 # default (off) outside any block @pytest.mark.parametrize( "bad", [ - "off", # a dead keyword form - "LOW", # wrong case — exact match only - " low ", # stray whitespace - 5, # the old integer levels are gone + 0, # not positive + -1, # negative + 1.5, # a float, not an int + "8", # a string, even a numeric one + "high", # the old level names are gone None, # None not accepted - ["low"], # unhashable → the ``not in`` check still rejects it cleanly + True, # bool is an int subclass but nonsensical here + ["8"], # a list ], ) -def test_resolve_level_rejects_everything_but_the_three_levels(bad): - """Only the three exact level strings are accepted; every other value — a - representative of each rejected shape (dead keyword, wrong case, whitespace, - old int, ``None``, unhashable) — raises ``ValueError`` so a typo fails - loudly.""" - with pytest.raises(ValueError, match="parallel_chunks level must be"): - _resolve_level(bad) - - -def test_parallel_chunks_scopes_and_restores_the_ambient(): - """The context manager resolves the level to its cap, publishes it on the - ambient for the block, and restores the previous value on exit — including - proper nesting.""" +def test_parallel_chunks_rejects_non_positive_int(bad): + """``n`` must be a positive integer; every other shape — zero, negative, a + float, a string (including a numeric one and the old level names), ``None``, + a ``bool``, a list — raises ``ValueError`` at ``with`` entry, before any + request, and leaves the ambient untouched.""" + with pytest.raises(ValueError, match="must be a positive integer"): + with parallel_chunks(bad): + pass assert _parallel_chunks.get() == 0 - with parallel_chunks("high"): - assert _parallel_chunks.get() == _LEVEL_CAPS["high"] - with parallel_chunks("low"): - assert _parallel_chunks.get() == _LEVEL_CAPS["low"] - assert _parallel_chunks.get() == _LEVEL_CAPS["high"] # outer restored - assert _parallel_chunks.get() == 0 # default (off) outside any block -def test_parallel_chunks_validates_on_entry(): - """An invalid level raises at ``with`` entry — before any request is - issued — and leaves the ambient untouched.""" - with pytest.raises(ValueError, match="parallel_chunks level must be"): - with parallel_chunks("aggressive"): - pass - assert _parallel_chunks.get() == 0 +def test_parallel_chunks_n1_is_no_extra_fan_out(): + """``n=1`` is the explicit no-op: an under-limit request stays a single + passthrough call, exactly as if the block weren't entered.""" + sites = [f"S{i:02d}" for i in range(8)] + calls: list[int] = [] + + @multi_value_chunked(build_request=_fake_build, url_limit=8000) + async def fetch(args): + calls.append(len(args["monitoring_location_id"])) + return pd.DataFrame(), _ok_response() + + with parallel_chunks(1): + fetch({"monitoring_location_id": sites}) + assert calls == [8] # one passthrough call carrying all sites -def test_parallel_chunks_high_drives_end_to_end_fan_out(): +def test_parallel_chunks_drives_end_to_end_fan_out(): """End-to-end: the same fitting request passes through as a single call by - default, but fans into several sub-requests inside a - ``parallel_chunks("high")`` block — and the combined result still - recovers every atom exactly once.""" + default, but fans into ``n`` sub-requests inside a ``parallel_chunks(n)`` + block — and the combined result still recovers every atom exactly once.""" sites = [f"S{i:02d}" for i in range(8)] calls: list[tuple[str, ...]] = [] @@ -2353,9 +2334,9 @@ async def fetch(args): assert sorted(df_plain["site"]) == sorted(sites) calls.clear() - with parallel_chunks("high"): + with parallel_chunks(8): df_fine, _ = fetch({"monitoring_location_id": sites}) - # 8 atoms at the "high" cap (>= 8) → 8 singleton sub-requests. + # 8 atoms at n=8 → 8 singleton sub-requests. assert len(calls) == 8 assert all(len(chunk) == 1 for chunk in calls) # Union across chunks recovers the original set, once each. @@ -2363,10 +2344,10 @@ async def fetch(args): assert sorted(df_fine["site"]) == sorted(sites) -def test_parallel_chunks_low_is_a_gentle_split(): - """``"low"`` is the mildest opt-in: an under-limit request fans into just - the ``"low"`` cap's worth of pieces, not singletons.""" - low = _LEVEL_CAPS["low"] +@pytest.mark.parametrize("n", [2, 3, 8]) +def test_parallel_chunks_supports_arbitrary_n(n): + """An arbitrary ``n`` (not only 2/8/32) fans an under-limit request into + exactly ``n`` sub-requests, together covering every site once.""" sites = [f"S{i:02d}" for i in range(8)] calls: list[int] = [] @@ -2375,8 +2356,7 @@ async def fetch(args): calls.append(len(args["monitoring_location_id"])) return pd.DataFrame(), _ok_response() - with parallel_chunks("low"): + with parallel_chunks(n): fetch({"monitoring_location_id": sites}) - # low cap → that many sub-requests, together covering all 8 sites. - assert len(calls) == low + assert len(calls) == n assert sum(calls) == 8 From c4d336d96a0bed774d511669957f05877683802c Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Wed, 8 Jul 2026 19:14:21 -0500 Subject: [PATCH 05/13] refactor(waterdata): tidy parallel_chunks (rename param, align validation, dedup test) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /simplify cleanup: (1) rename ChunkPlan's max_chunks_per_axis -> max_chunks — the cap is on the plan's TOTAL sub-request count, not per axis, so the old name needed apologetic docstrings; dropped them. (2) Validate parallel_chunks(n) with numbers.Integral (matching the max_rows guard in engine.py), so numpy integers are accepted like the sibling validator. (3) Fold the n=1 no-op test into the parametrized arbitrary-n test, removing a duplicated fetch fixture. Co-Authored-By: Claude Opus 4.8 (1M context) --- dataretrieval/ogc/chunking.py | 8 +++-- dataretrieval/ogc/planning.py | 33 ++++++++++---------- tests/waterdata_chunking_test.py | 53 ++++++++++++-------------------- 3 files changed, 40 insertions(+), 54 deletions(-) diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index 1a3b37b6..c6aa2357 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -73,6 +73,7 @@ import asyncio import functools +import numbers import os from collections.abc import Callable, Iterator from contextlib import contextmanager @@ -258,8 +259,9 @@ def parallel_chunks(n: int) -> Iterator[None]: -------- ChunkPlan._refine : the planning-side effect of ``n``. """ - # ``bool`` is an ``int`` subclass but nonsensical here; reject it explicitly. - if not isinstance(n, int) or isinstance(n, bool) or n < 1: + # Accept any integer type (incl. numpy ints, mirroring ``max_rows``); ``bool`` + # is an ``Integral`` subclass but nonsensical here, so reject it explicitly. + if not isinstance(n, numbers.Integral) or isinstance(n, bool) or n < 1: raise ValueError( f"parallel_chunks(n): n must be a positive integer (e.g. 2, 8, 32); " f"got {n!r}." @@ -739,7 +741,7 @@ def wrapper( # here up front, so a later resume — which re-issues the # already-planned sub-requests — needs no snapshot. plan = ChunkPlan( - args, build_request, limit, max_chunks_per_axis=_parallel_chunks.get() + args, build_request, limit, max_chunks=_parallel_chunks.get() ) retry_policy = RetryPolicy.from_env() # The concurrency cap is resolved inside ``resume()`` from diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index ebadcb36..a32e48c6 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -329,11 +329,11 @@ class ChunkPlan: url_limit : int Byte budget for the request (URL + body) — a hard ceiling every sub-request must fit. - max_chunks_per_axis : int, optional + max_chunks : int, optional Soft cap on the plan's total sub-request count (default ``0`` = off). ``0`` chunks only as much as ``url_limit`` requires — the most conservative plan, fewest sub-requests. A positive cap fans the plan - out to up to ``max_chunks_per_axis`` sub-requests overall (the + out to up to ``max_chunks`` sub-requests overall (the cartesian product across axes, never fewer than the byte budget already forces) — capped as a whole, not per axis, so several multi-value axes can't multiply past the cap. Set from the @@ -370,7 +370,7 @@ def __init__( args: dict[str, Any], build_request: Callable[..., httpx.Request], url_limit: int, - max_chunks_per_axis: int = 0, + max_chunks: int = 0, ) -> None: self.args = args self.axes: list[_Axis] = [] @@ -423,9 +423,9 @@ def __init__( # A request that already fits and hasn't opted into finer chunking is # the common passthrough: leave ``axes``/``chunks`` empty so # ``total == 1`` and ``iter_sub_args`` yields the original args - # verbatim. Only when ``max_chunks_per_axis`` asks for extra fan-out do + # verbatim. Only when ``max_chunks`` asks for extra fan-out do # we set the axes up to be refined below. - if fits and max_chunks_per_axis <= 0: + if fits and max_chunks <= 0: return self.axes = axes @@ -436,8 +436,8 @@ def __init__( self._plan(build_request, url_limit) # Soft pass: optionally split further than the byte budget requires. # Purely additive — never re-raises, and the byte budget stays - # satisfied; a no-op at ``max_chunks_per_axis <= 0``. - self._refine(max_chunks_per_axis) + # satisfied; a no-op at ``max_chunks <= 0``. + self._refine(max_chunks) if self.canonical_url is None: # Original URL was un-constructable (httpx.InvalidURL); fall @@ -491,7 +491,7 @@ def _plan( ) _split_at(self.chunks[biggest_axis.arg_key], biggest_idx) - def _refine(self, max_chunks_per_axis: int) -> None: + def _refine(self, max_chunks: int) -> None: """ Fan the plan out more finely than the byte budget alone requires — the parallel_chunks dial (see @@ -499,7 +499,7 @@ def _refine(self, max_chunks_per_axis: int) -> None: would want this). Caps the plan's *total* sub-request count (:attr:`total`, the - cartesian product across all axes) at ``max_chunks_per_axis``, not + cartesian product across all axes) at ``max_chunks``, not each axis independently — with several multi-value axes, a cap of 32 still means at most 32 sub-requests overall, not ``32 ** n_axes``. Each split picks the single largest splittable chunk across *every* @@ -508,20 +508,19 @@ def _refine(self, max_chunks_per_axis: int) -> None: before another is touched. Purely additive — only ever *splits* existing chunks, so the byte pass's work and the ``url_limit`` invariant are both preserved, and it never raises. A no-op at - ``max_chunks_per_axis <= 0``. + ``max_chunks <= 0``. Parameters ---------- - max_chunks_per_axis : int + max_chunks : int Soft cap on the plan's total sub-request count (``0`` = off) — the - ``parallel_chunks(n)`` value. The parameter name is legacy: the cap - is on the whole plan (the cartesian product across axes), which - matches "sub-chunks per argument" only in the common single-axis - case — multi-axis plans are capped on the product, not per axis. + ``parallel_chunks(n)`` value. The cap is on the whole plan (the + cartesian product across axes), not per axis, so several multi-value + axes can't multiply past it. """ - if max_chunks_per_axis <= 0: + if max_chunks <= 0: return - while self.total < max_chunks_per_axis: + while self.total < max_chunks: # Largest splittable chunk across every axis; a chunk of size 1 # can't be split further. ``max`` with a stable input order # breaks ties by axis order, then lowest index within an axis. diff --git a/tests/waterdata_chunking_test.py b/tests/waterdata_chunking_test.py index 1ab57378..850b8709 100644 --- a/tests/waterdata_chunking_test.py +++ b/tests/waterdata_chunking_test.py @@ -2115,7 +2115,7 @@ async def fetch(args): # ``parallel_chunks`` context manager). ``_fake_build``'s base is 200 bytes, so # a handful of short atoms sits far under ``url_limit=8000`` — the byte pass # passes it through untouched, and any splitting below is the ``n`` cap alone. -# ``ChunkPlan`` takes the integer cap (``max_chunks_per_axis``) directly; +# ``ChunkPlan`` takes the integer cap (``max_chunks``) directly; # ``parallel_chunks(n)`` publishes ``n`` onto it. The cap bounds the plan's # *total* sub-request count (the cartesian product across axes), not each axis # independently — see ``test_cap_caps_the_total_across_axes``. @@ -2123,22 +2123,22 @@ async def fetch(args): def test_zero_cap_preserves_passthrough(): - """``max_chunks_per_axis=0`` (the default) must not perturb the existing + """``max_chunks=0`` (the default) must not perturb the existing plan: a multi-value request that fits the byte limit is still the trivial passthrough (no axes, ``total == 1``), byte-for-byte the pre-feature behavior.""" args = {"monitoring_location_id": ["A", "B", "C", "D"]} - plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks_per_axis=0) + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=0) assert plan.axes == [] assert plan.total == 1 assert list(plan.iter_sub_args()) == [args] @pytest.mark.parametrize( - ("max_chunks_per_axis", "expected_pieces"), + ("max_chunks", "expected_pieces"), [(0, 1), (2, 2), (8, 8), (16, 10), (32, 10)], ) -def test_cap_ramps_then_saturates(max_chunks_per_axis, expected_pieces): +def test_cap_ramps_then_saturates(max_chunks, expected_pieces): """A single 10-atom axis that fits the byte limit splits into ``min(10, cap)`` pieces: 1 (off), 2, 8, then saturating at 10 (one atom per chunk) once the cap overshoots the atom count. Monotonic and bounded, and @@ -2149,10 +2149,10 @@ def test_cap_ramps_then_saturates(max_chunks_per_axis, expected_pieces): {"monitoring_location_id": atoms}, _fake_build, url_limit=8000, - max_chunks_per_axis=max_chunks_per_axis, + max_chunks=max_chunks, ) assert plan.total == expected_pieces - if max_chunks_per_axis: + if max_chunks: flattened = [ a for chunk in plan.chunks["monitoring_location_id"] for a in chunk ] @@ -2170,7 +2170,7 @@ def test_cap_bounds_fan_out_for_a_long_axis(): {"monitoring_location_id": atoms}, _fake_build, url_limit=8000, - max_chunks_per_axis=high, + max_chunks=high, ) assert plan.total == high flattened = [a for chunk in plan.chunks["monitoring_location_id"] for a in chunk] @@ -2184,9 +2184,9 @@ def test_cap_below_byte_split_does_not_reduce_fan_out(): # Heavy axis of four 30-char atoms; a limit tight enough that the byte pass # must drive every atom into its own sub-request (4 pieces > the cap of 2). args = {"monitoring_location_id": ["X" * 30, "Y" * 30, "Z" * 30, "W" * 30]} - baseline = ChunkPlan(args, _fake_build, url_limit=250, max_chunks_per_axis=0) + baseline = ChunkPlan(args, _fake_build, url_limit=250, max_chunks=0) assert baseline.total > 2 # byte pass alone already fanned out past 2 - refined = ChunkPlan(args, _fake_build, url_limit=250, max_chunks_per_axis=2) + refined = ChunkPlan(args, _fake_build, url_limit=250, max_chunks=2) # cap 2 < baseline pieces → refine is a no-op here. assert refined.total == baseline.total @@ -2197,8 +2197,8 @@ def test_cap_never_exceeds_the_byte_budget(): a chunk), and the fan-out is at least what the byte pass required.""" args = {"monitoring_location_id": ["X" * 30, "Y" * 30, "Z" * 30, "W" * 30]} limit = 310 - byte_only = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks_per_axis=0) - plan = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks_per_axis=32) + byte_only = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks=0) + plan = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks=32) assert plan.total >= byte_only.total for sub in plan.iter_sub_args(): assert _safe_request_bytes(_fake_build, sub, limit) <= limit @@ -2210,7 +2210,7 @@ def test_cap_refines_the_filter_axis(): into ``min(N, cap)`` pieces.""" clauses = [f"p='{i}'" for i in range(8)] args = {"filter": " OR ".join(clauses)} - plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks_per_axis=4) + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=4) assert len(plan.chunks["filter"]) == 4 # min(8, 4) assert plan.total == 4 @@ -2225,7 +2225,7 @@ def test_cap_caps_the_total_across_axes(): "monitoring_location_id": [f"L{i}" for i in range(6)], "parameter_code": [f"{i:05d}" for i in range(6)], } - plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks_per_axis=4) + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=4) assert plan.total == 4 # Every atom on every axis is still covered exactly once. for key, atoms in ( @@ -2249,7 +2249,7 @@ def test_cap_bounds_fan_out_across_many_axes(): "parameter_code": [f"{i:05d}" for i in range(10)], "filter": " OR ".join(f"p='{i}'" for i in range(10)), } - plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks_per_axis=high) + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=high) assert plan.total <= high @@ -2259,7 +2259,7 @@ def test_cap_does_not_mask_unchunkable(): act on and must not swallow the hard failure.""" args = {"monitoring_location_id": "one-huge-scalar"} with pytest.raises(Unchunkable): - ChunkPlan(args, _fake_build, url_limit=10, max_chunks_per_axis=32) + ChunkPlan(args, _fake_build, url_limit=10, max_chunks=32) def test_parallel_chunks_publishes_n_on_the_ambient(): @@ -2298,22 +2298,6 @@ def test_parallel_chunks_rejects_non_positive_int(bad): assert _parallel_chunks.get() == 0 -def test_parallel_chunks_n1_is_no_extra_fan_out(): - """``n=1`` is the explicit no-op: an under-limit request stays a single - passthrough call, exactly as if the block weren't entered.""" - sites = [f"S{i:02d}" for i in range(8)] - calls: list[int] = [] - - @multi_value_chunked(build_request=_fake_build, url_limit=8000) - async def fetch(args): - calls.append(len(args["monitoring_location_id"])) - return pd.DataFrame(), _ok_response() - - with parallel_chunks(1): - fetch({"monitoring_location_id": sites}) - assert calls == [8] # one passthrough call carrying all sites - - def test_parallel_chunks_drives_end_to_end_fan_out(): """End-to-end: the same fitting request passes through as a single call by default, but fans into ``n`` sub-requests inside a ``parallel_chunks(n)`` @@ -2344,10 +2328,11 @@ async def fetch(args): assert sorted(df_fine["site"]) == sorted(sites) -@pytest.mark.parametrize("n", [2, 3, 8]) +@pytest.mark.parametrize("n", [1, 2, 3, 8]) def test_parallel_chunks_supports_arbitrary_n(n): """An arbitrary ``n`` (not only 2/8/32) fans an under-limit request into - exactly ``n`` sub-requests, together covering every site once.""" + exactly ``n`` sub-requests, together covering every site once — including + ``n=1``, the explicit no-op that stays a single passthrough call.""" sites = [f"S{i:02d}" for i in range(8)] calls: list[int] = [] From 28a23e27255b9489b446f68a0dfd98c1490a2675 Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Thu, 16 Jul 2026 16:14:13 -0500 Subject: [PATCH 06/13] test: add parallel_chunks threshold experiments Experiments to determine safe parallelism levels for the USGS API. Results: 20+ trials at n=8/16/32 with 10s sleep between queries show zero hangs across all levels. n=16 shows the best avg response time. The only failure mode is proper HTTP 429 when quota is exhausted. --- experiments/overload_threshold_experiment.py | 338 ++++++++++++++++++ experiments/overload_threshold_log.jsonl | 20 ++ experiments/parallel_threshold_experiment.py | 350 +++++++++++++++++++ experiments/parallel_threshold_log.jsonl | 81 +++++ 4 files changed, 789 insertions(+) create mode 100644 experiments/overload_threshold_experiment.py create mode 100644 experiments/overload_threshold_log.jsonl create mode 100644 experiments/parallel_threshold_experiment.py create mode 100644 experiments/parallel_threshold_log.jsonl diff --git a/experiments/overload_threshold_experiment.py b/experiments/overload_threshold_experiment.py new file mode 100644 index 00000000..4589d2dd --- /dev/null +++ b/experiments/overload_threshold_experiment.py @@ -0,0 +1,338 @@ +""" +Experiment: Find the parallelism level that overloads the server. + +Strategy: +- For each parallelism level (starting low, escalating), query ALL states + at that level with sleep between each query. +- Each state gets a unique time window (never reused across runs). +- Log wall-clock time, outcome, and any signs of server stress (slow responses, + errors, partial results). +- Sleep between queries to stay below rate limit and isolate the effect of + parallelism from quota exhaustion. + +Usage: + python experiments/overload_threshold_experiment.py [--level N] [--sleep 10] + + If --level is not given, it auto-escalates through 1, 2, 4, 8, 16, 32. +""" + +from __future__ import annotations + +import json +import signal +import sys +import time +from datetime import datetime, timezone +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from dataretrieval import waterdata +from dataretrieval.ogc.interruptions import ChunkInterrupted, QuotaExhausted + +LOG_FILE = Path(__file__).with_name("overload_threshold_log.jsonl") + +# Large states — these put real pressure on the server +STATES = ["Ohio", "Pennsylvania", "Connecticut", "New Hampshire", "Vermont"] + +# Time windows pool — we rotate through these, never reusing a (state, window) +# pair across the entire log. +WINDOWS = [ + "2000-01-01/2000-12-31", + "2001-01-01/2001-12-31", + "2002-01-01/2002-12-31", + "2003-01-01/2003-12-31", + "2004-01-01/2004-12-31", + "2005-01-01/2005-12-31", + "2006-01-01/2006-12-31", + "2007-01-01/2007-12-31", + "2008-01-01/2008-12-31", + "2009-01-01/2009-12-31", + "2010-01-01/2010-12-31", + "2011-01-01/2011-12-31", + "2012-01-01/2012-12-31", + "2013-01-01/2013-12-31", + "2014-01-01/2014-12-31", + "2015-01-01/2015-12-31", + "2016-01-01/2016-12-31", + "2017-01-01/2017-12-31", + "2018-01-01/2018-12-31", + "2019-01-01/2019-12-31", + "2020-01-01/2020-12-31", + "2021-01-01/2021-12-31", + "2022-01-01/2022-12-31", + "2023-01-01/2023-12-31", + "2024-01-01/2024-12-31", +] + + +class HangTimeout(Exception): + pass + + +def _timeout_handler(signum, frame): + raise HangTimeout("hang") + + +def _load_log() -> list[dict]: + if not LOG_FILE.exists(): + return [] + entries = [] + with open(LOG_FILE) as f: + for line in f: + line = line.strip() + if line: + entries.append(json.loads(line)) + return entries + + +def _append_log(entry: dict) -> None: + with open(LOG_FILE, "a") as f: + f.write(json.dumps(entry) + "\n") + + +def _used_windows_for_state(entries: list[dict], state: str) -> set[str]: + """Return windows already used for a given state in the log.""" + return {e["time_window"] for e in entries if e.get("state") == state} + + +def _next_window(state: str, used: set[str]) -> str | None: + """Pick the next unused window for a state.""" + for w in WINDOWS: + if w not in used: + return w + return None + + +def run_level(level: int, sleep_s: int, timeout_s: int) -> list[dict]: + """Run all states at a given parallelism level. Returns the trial results.""" + entries = _load_log() + + # Preload sites + site_cache: dict[str, list[str]] = {} + for state in STATES: + print(f" Fetching {state} sites...", end=" ", flush=True) + sites, _ = waterdata.get_monitoring_locations(state=state, site_type_code="ST") + site_cache[state] = sites["monitoring_location_id"].tolist() + print(f"{len(site_cache[state])} sites") + + print( + f"\n Running all {len(STATES)} states at n={level}" + f" (sleep={sleep_s}s between):\n" + ) + + results = [] + for i, state in enumerate(STATES): + # Pick a fresh window + used = _used_windows_for_state(entries + results, state) + window = _next_window(state, used) + if window is None: + print(f" {state}: no fresh windows left, skipping") + continue + + site_ids = site_cache[state] + print( + f" [{i + 1}/{len(STATES)}] {state:<15} ({len(site_ids):>4} sites) " + f"{window} n={level}...", + end=" ", + flush=True, + ) + + old = signal.signal(signal.SIGALRM, _timeout_handler) + signal.alarm(timeout_s) + t0 = time.perf_counter() + + entry = { + "timestamp": datetime.now(timezone.utc).isoformat(), + "state": state, + "n_sites": len(site_ids), + "time_window": window, + "parallelism": level, + "timeout_s": timeout_s, + "sleep_s": sleep_s, + "outcome": None, + "wall_clock_s": None, + "n_records": None, + "error_type": None, + "error_msg": None, + } + + try: + if level <= 1: + df, md = waterdata.get_daily( + monitoring_location_id=site_ids, + parameter_code="00060", + time=window, + ) + else: + with waterdata.parallel_chunks(level): + df, md = waterdata.get_daily( + monitoring_location_id=site_ids, + parameter_code="00060", + time=window, + ) + + elapsed = time.perf_counter() - t0 + entry["outcome"] = "success" + entry["wall_clock_s"] = round(elapsed, 2) + entry["n_records"] = len(df) + print(f"✓ {elapsed:.1f}s, {len(df):,} records") + + except HangTimeout: + elapsed = time.perf_counter() - t0 + entry["outcome"] = "timeout" + entry["wall_clock_s"] = round(elapsed, 2) + entry["error_type"] = "HangTimeout" + entry["error_msg"] = f"No response within {timeout_s}s" + print(f"✗ HANG after {elapsed:.0f}s") + + except QuotaExhausted as exc: + elapsed = time.perf_counter() - t0 + entry["outcome"] = "quota_exhausted" + entry["wall_clock_s"] = round(elapsed, 2) + entry["error_type"] = "QuotaExhausted" + entry["error_msg"] = f"429, retry_after={exc.retry_after}" + print(f"✗ 429 after {elapsed:.1f}s (retry_after={exc.retry_after})") + + except ChunkInterrupted as exc: + elapsed = time.perf_counter() - t0 + entry["outcome"] = "interrupted" + entry["wall_clock_s"] = round(elapsed, 2) + entry["error_type"] = type(exc).__name__ + entry["error_msg"] = str(exc)[:200] + print(f"✗ {type(exc).__name__} after {elapsed:.1f}s") + + except Exception as exc: + elapsed = time.perf_counter() - t0 + entry["outcome"] = "error" + entry["wall_clock_s"] = round(elapsed, 2) + entry["error_type"] = type(exc).__name__ + entry["error_msg"] = str(exc)[:200] + print(f"✗ {type(exc).__name__} after {elapsed:.1f}s: {str(exc)[:80]}") + + finally: + signal.alarm(0) + signal.signal(signal.SIGALRM, old) + + _append_log(entry) + results.append(entry) + + # Sleep between queries to stay under rate limit + if i < len(STATES) - 1: + print(f" (sleeping {sleep_s}s...)", flush=True) + time.sleep(sleep_s) + + return results + + +def print_summary(): + """Print summary grouped by parallelism level.""" + entries = _load_log() + if not entries: + print("No log entries.") + return + + from collections import defaultdict + + by_level = defaultdict(list) + for e in entries: + by_level[e["parallelism"]].append(e) + + print("\n" + "=" * 90) + print("OVERLOAD THRESHOLD EXPERIMENT — all runs") + print("=" * 90) + print( + f"{'level':>5} | {'trials':>6} | {'ok':>4} | {'429':>4} | " + f"{'hang':>4} | {'err':>4} | {'avg_s':>7} | {'max_s':>7} | " + f"{'avg_records':>11} | {'total_records':>13}" + ) + print("-" * 90) + + for level in sorted(by_level.keys()): + trials = by_level[level] + ok = [t for t in trials if t["outcome"] == "success"] + quota = [t for t in trials if t["outcome"] == "quota_exhausted"] + hangs = [t for t in trials if t["outcome"] == "timeout"] + errs = [t for t in trials if t["outcome"] in ("error", "interrupted")] + + times = [t["wall_clock_s"] for t in ok] + avg_t = f"{sum(times) / len(times):.1f}" if times else "—" + max_t = f"{max(times):.1f}" if times else "—" + avg_r = int(sum(t["n_records"] for t in ok) / len(ok)) if ok else 0 + total_r = sum(t["n_records"] for t in ok) if ok else 0 + + print( + f"{level:>5} | {len(trials):>6} | {len(ok):>4} | {len(quota):>4} | " + f"{len(hangs):>4} | {len(errs):>4} | {avg_t:>7} | {max_t:>7} | " + f"{avg_r:>11,} | {total_r:>13,}" + ) + + print("=" * 90) + + # Detail per level+state + print("\nPer-state breakdown (successful trials only):") + print( + f" {'level':>5} | {'state':<15} | {'time_s':>7} | {'records':>10} | {'window'}" + ) + print(" " + "-" * 70) + for level in sorted(by_level.keys()): + for e in sorted(by_level[level], key=lambda x: x.get("state", "")): + if e["outcome"] == "success": + print( + f" {level:>5} | {e['state']:<15} | " + f"{e['wall_clock_s']:>7.1f} | {e['n_records']:>10,} | " + f"{e['time_window']}" + ) + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("--level", type=int, default=None, help="Single level to test") + parser.add_argument( + "--levels", type=str, default=None, help="Comma-separated levels" + ) + parser.add_argument( + "--sleep", type=int, default=10, help="Seconds between queries (default: 10)" + ) + parser.add_argument( + "--timeout", type=int, default=120, help="Per-query timeout (default: 120)" + ) + parser.add_argument("--summary", action="store_true", help="Print summary only") + args = parser.parse_args() + + if args.summary: + print_summary() + sys.exit(0) + + if args.level is not None: + levels = [args.level] + elif args.levels is not None: + levels = [int(x) for x in args.levels.split(",")] + else: + levels = [1, 2, 4, 8, 16, 32] + + print("Overload threshold experiment") + print(f" Levels to test: {levels}") + print(f" States: {STATES}") + print(f" Sleep between queries: {args.sleep}s") + print(f" Timeout: {args.timeout}s") + print(f" Log: {LOG_FILE}") + print() + + for level in levels: + print(f"{'=' * 60}") + print(f" LEVEL {level}") + print(f"{'=' * 60}") + results = run_level(level, args.sleep, args.timeout) + + # Check if we got throttled — if so, stop escalating + failures = [r for r in results if r["outcome"] != "success"] + if failures: + print(f"\n ⚠ Failures at level {level} — stopping escalation.") + break + + print() + + print_summary() diff --git a/experiments/overload_threshold_log.jsonl b/experiments/overload_threshold_log.jsonl new file mode 100644 index 00000000..5ff2ad93 --- /dev/null +++ b/experiments/overload_threshold_log.jsonl @@ -0,0 +1,20 @@ +{"timestamp": "2026-07-16T18:33:36.851681+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2000-01-01/2000-12-31", "parallelism": 8, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 10.88, "n_records": 42728, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:33:57.741196+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2000-01-01/2000-12-31", "parallelism": 8, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 6.81, "n_records": 78674, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:34:14.560514+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2000-01-01/2000-12-31", "parallelism": 8, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 3.03, "n_records": 17706, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:34:27.595726+00:00", "state": "New Hampshire", "n_sites": 1081, "time_window": "2000-01-01/2000-12-31", "parallelism": 8, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 6.98, "n_records": 13629, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:34:44.581715+00:00", "state": "Vermont", "n_sites": 570, "time_window": "2000-01-01/2000-12-31", "parallelism": 8, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 4.9, "n_records": 14274, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:34:56.219533+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2001-01-01/2001-12-31", "parallelism": 16, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 2.95, "n_records": 46368, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:35:09.173482+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2001-01-01/2001-12-31", "parallelism": 16, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 4.09, "n_records": 78329, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:35:23.274345+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2001-01-01/2001-12-31", "parallelism": 16, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 1.65, "n_records": 18242, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:35:34.932930+00:00", "state": "New Hampshire", "n_sites": 1081, "time_window": "2001-01-01/2001-12-31", "parallelism": 16, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 0.98, "n_records": 14514, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:35:45.921675+00:00", "state": "Vermont", "n_sites": 570, "time_window": "2001-01-01/2001-12-31", "parallelism": 16, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 1.38, "n_records": 14829, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:35:54.383526+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2002-01-01/2002-12-31", "parallelism": 32, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 3.74, "n_records": 49490, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:36:08.126379+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2002-01-01/2002-12-31", "parallelism": 32, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 7.15, "n_records": 78595, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:36:25.287311+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2002-01-01/2002-12-31", "parallelism": 32, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 3.55, "n_records": 18090, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:36:38.847054+00:00", "state": "New Hampshire", "n_sites": 1081, "time_window": "2002-01-01/2002-12-31", "parallelism": 32, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 1.82, "n_records": 16646, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:36:50.675320+00:00", "state": "Vermont", "n_sites": 570, "time_window": "2002-01-01/2002-12-31", "parallelism": 32, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 11.11, "n_records": 15065, "error_type": null, "error_msg": null} +{"timestamp": "2026-07-16T18:37:38.342619+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2003-01-01/2007-12-31", "parallelism": 32, "timeout_s": 180, "sleep_s": 10, "outcome": "success", "wall_clock_s": 59.67, "n_records": 242773, "error_type": null, "error_msg": null, "note": "5yr_window"} +{"timestamp": "2026-07-16T18:38:48.021952+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2004-01-01/2008-12-31", "parallelism": 32, "timeout_s": 180, "sleep_s": 10, "outcome": "success", "wall_clock_s": 15.88, "n_records": 419738, "error_type": null, "error_msg": null, "note": "5yr_window"} +{"timestamp": "2026-07-16T18:39:13.910850+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2005-01-01/2009-12-31", "parallelism": 32, "timeout_s": 180, "sleep_s": 10, "outcome": "success", "wall_clock_s": 6.28, "n_records": 103846, "error_type": null, "error_msg": null, "note": "5yr_window"} +{"timestamp": "2026-07-16T18:39:30.191569+00:00", "state": "New Hampshire", "n_sites": 1081, "time_window": "2006-01-01/2010-12-31", "parallelism": 32, "timeout_s": 180, "sleep_s": 10, "outcome": "success", "wall_clock_s": 2.65, "n_records": 81268, "error_type": null, "error_msg": null, "note": "5yr_window"} +{"timestamp": "2026-07-16T18:39:42.849171+00:00", "state": "Vermont", "n_sites": 570, "time_window": "2007-01-01/2011-12-31", "parallelism": 32, "timeout_s": 180, "sleep_s": 10, "outcome": "success", "wall_clock_s": 2.8, "n_records": 87867, "error_type": null, "error_msg": null, "note": "5yr_window"} diff --git a/experiments/parallel_threshold_experiment.py b/experiments/parallel_threshold_experiment.py new file mode 100644 index 00000000..d1828be2 --- /dev/null +++ b/experiments/parallel_threshold_experiment.py @@ -0,0 +1,350 @@ +""" +Experiment: parallel_chunks threshold sensitivity + +Goal: Find the highest parallelism level that avoids server hang-ups. + +Strategy: +- Rotate queries (different states × different time windows) to defeat server cache. +- Test parallelism levels: 1 (serial baseline), 2, 4, 8, 16, 32. +- Each trial uses a per-request timeout (120s) to detect hangs vs. errors. +- Results are appended to a persistent JSONL log so we can accumulate data + across multiple runs without losing earlier results. + +Usage: + python experiments/parallel_threshold_experiment.py \ + [--levels 1,2,4,8,16,32] [--timeout 120] + +The script runs one trial per (state, time_window, parallelism) combination, +then prints a summary table at the end. Re-run to add more data points. +""" + +from __future__ import annotations + +import argparse +import json +import signal +import sys +import time +from datetime import datetime, timezone +from pathlib import Path + +# Ensure the local package is importable +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from dataretrieval import waterdata +from dataretrieval.ogc.interruptions import ChunkInterrupted + +# ─── Configuration ──────────────────────────────────────────────────────────── + +LOG_FILE = Path(__file__).with_name("parallel_threshold_log.jsonl") + +# States spanning a range of site counts. Larger states (OH, PA) stress the +# server harder; smaller ones (DE, RI) serve as faster controls. +QUERY_STATES = ["Ohio", "Pennsylvania", "Connecticut", "New Hampshire", "Vermont"] + +# Non-overlapping time windows to rotate through, avoiding cache hits. +# Each is ~1 year of daily data — large enough to span multiple pages. +TIME_WINDOWS = [ + "2005-01-01/2005-12-31", + "2008-01-01/2008-12-31", + "2011-01-01/2011-12-31", + "2014-01-01/2014-12-31", + "2017-01-01/2017-12-31", + "2020-01-01/2020-12-31", +] + +DEFAULT_LEVELS = [1, 2, 4, 8, 16, 32] +DEFAULT_TIMEOUT = 120 # seconds — a "hang" if no response by this time + + +# ─── Helpers ────────────────────────────────────────────────────────────────── + + +class TimeoutError(Exception): + pass + + +def _timeout_handler(signum, frame): + raise TimeoutError("Request timed out (server hang suspected)") + + +def _get_sites_for_state(state: str) -> list[str]: + """Fetch stream monitoring location IDs for a state.""" + sites, _ = waterdata.get_monitoring_locations(state=state, site_type_code="ST") + return sites["monitoring_location_id"].tolist() + + +def _run_trial( + site_ids: list[str], + time_window: str, + parallelism: int, + timeout: int, +) -> dict: + """ + Run a single trial: fetch daily discharge for the given sites/time window + at the specified parallelism level. + + Returns a dict with trial metadata and outcome. + """ + result = { + "timestamp": datetime.now(timezone.utc).isoformat(), + "n_sites": len(site_ids), + "time_window": time_window, + "parallelism": parallelism, + "timeout_s": timeout, + "outcome": None, # "success", "timeout", "error", "interrupted" + "wall_clock_s": None, + "n_records": None, + "error_type": None, + "error_msg": None, + } + + # Set alarm-based timeout (Unix only) to detect hangs + old_handler = signal.signal(signal.SIGALRM, _timeout_handler) + signal.alarm(timeout) + + t0 = time.perf_counter() + try: + if parallelism <= 1: + # Serial baseline — no parallel_chunks context + df, md = waterdata.get_daily( + monitoring_location_id=site_ids, + parameter_code="00060", + time=time_window, + ) + else: + with waterdata.parallel_chunks(parallelism): + df, md = waterdata.get_daily( + monitoring_location_id=site_ids, + parameter_code="00060", + time=time_window, + ) + + elapsed = time.perf_counter() - t0 + result["outcome"] = "success" + result["wall_clock_s"] = round(elapsed, 2) + result["n_records"] = len(df) + + except TimeoutError: + elapsed = time.perf_counter() - t0 + result["outcome"] = "timeout" + result["wall_clock_s"] = round(elapsed, 2) + result["error_type"] = "TimeoutError" + result["error_msg"] = "Server hang — no response within timeout" + + except ChunkInterrupted as exc: + elapsed = time.perf_counter() - t0 + result["outcome"] = "interrupted" + result["wall_clock_s"] = round(elapsed, 2) + result["error_type"] = type(exc).__name__ + result["error_msg"] = str(exc)[:200] + result["n_records"] = len(exc.call.partial_frame) if exc.call else None + + except Exception as exc: + elapsed = time.perf_counter() - t0 + result["outcome"] = "error" + result["wall_clock_s"] = round(elapsed, 2) + result["error_type"] = type(exc).__name__ + result["error_msg"] = str(exc)[:200] + + finally: + signal.alarm(0) # cancel alarm + signal.signal(signal.SIGALRM, old_handler) + + return result + + +def _append_log(entry: dict) -> None: + """Append a trial result to the persistent JSONL log.""" + with open(LOG_FILE, "a") as f: + f.write(json.dumps(entry) + "\n") + + +def _load_log() -> list[dict]: + """Load all past trial results.""" + if not LOG_FILE.exists(): + return [] + entries = [] + with open(LOG_FILE) as f: + for line in f: + line = line.strip() + if line: + entries.append(json.loads(line)) + return entries + + +def _print_summary(entries: list[dict]) -> None: + """Print a summary table of results grouped by parallelism level.""" + from collections import defaultdict + + by_level = defaultdict(list) + for e in entries: + by_level[e["parallelism"]].append(e) + + print("\n" + "=" * 80) + print("SUMMARY — all trials (current + historical)") + print("=" * 80) + print( + f"{'level':>6} | {'trials':>6} | {'success':>7} | {'timeout':>7} | " + f"{'error':>5} | {'interrupted':>11} | {'avg_time_s':>10} | {'med_time_s':>10}" + ) + print("-" * 80) + + for level in sorted(by_level.keys()): + trials = by_level[level] + n = len(trials) + successes = [t for t in trials if t["outcome"] == "success"] + timeouts = [t for t in trials if t["outcome"] == "timeout"] + errors = [t for t in trials if t["outcome"] == "error"] + interrupted = [t for t in trials if t["outcome"] == "interrupted"] + + times = sorted( + t["wall_clock_s"] for t in successes if t["wall_clock_s"] is not None + ) + avg_t = f"{sum(times) / len(times):.1f}" if times else "—" + med_t = f"{times[len(times) // 2]:.1f}" if times else "—" + + print( + f"{level:>6} | {n:>6} | {len(successes):>7} | {len(timeouts):>7} | " + f"{len(errors):>5} | {len(interrupted):>11} | {avg_t:>10} | {med_t:>10}" + ) + + print("=" * 80) + + # Show any failures in detail + failures = [e for e in entries if e["outcome"] not in ("success",)] + if failures: + print(f"\nFailed/timeout trials ({len(failures)}):") + for f_entry in failures[-10:]: # last 10 + print( + f" [{f_entry['timestamp'][:19]}] level={f_entry['parallelism']:>2}, " + f"outcome={f_entry['outcome']}, " + f"time={f_entry['wall_clock_s']}s, " + f"err={f_entry.get('error_type', '?')}: " + f"{(f_entry.get('error_msg') or '')[:80]}" + ) + + +# ─── Main ───────────────────────────────────────────────────────────────────── + + +def main(): + parser = argparse.ArgumentParser(description="Parallel chunks threshold experiment") + parser.add_argument( + "--levels", + type=str, + default=",".join(str(x) for x in DEFAULT_LEVELS), + help="Comma-separated parallelism levels to test (default: 1,2,4,8,16,32)", + ) + parser.add_argument( + "--timeout", + type=int, + default=DEFAULT_TIMEOUT, + help="Per-trial timeout in seconds (default: 120)", + ) + parser.add_argument( + "--state", + type=str, + default=None, + help="Override: use only this state (default: rotate through all)", + ) + parser.add_argument( + "--window", + type=str, + default=None, + help="Override: use only this time window (default: rotate)", + ) + parser.add_argument( + "--summary-only", + action="store_true", + help="Just print summary of existing log, don't run new trials", + ) + args = parser.parse_args() + + levels = [int(x) for x in args.levels.split(",")] + + if args.summary_only: + entries = _load_log() + if entries: + _print_summary(entries) + else: + print("No log entries found.") + return + + # Pick query rotation + states = [args.state] if args.state else QUERY_STATES + windows = [args.window] if args.window else TIME_WINDOWS + + print("Experiment: parallel_chunks threshold sensitivity") + print(f" Levels: {levels}") + print(f" Timeout: {args.timeout}s") + print(f" States: {states}") + print(f" Log: {LOG_FILE}") + print() + + # Preload site IDs for the selected states + site_cache: dict[str, list[str]] = {} + for state in states: + print(f" Fetching sites for {state}...", end=" ", flush=True) + site_cache[state] = _get_sites_for_state(state) + print(f"{len(site_cache[state])} sites") + + # Rotate: each level gets a different state (round-robin across states), + # and each (state, level) combo gets the next unused time window for that + # state. This ensures no two trials share a (state, window) pair, defeating + # the server-side cache. + # + # To avoid repeating combos from prior runs, count how many times each + # state has already appeared in the log and advance its cursor past those. + past_entries = _load_log() + window_cursor: dict[str, int] = {s: 0 for s in states} + for entry in past_entries: + s = entry.get("state") + if s in window_cursor: + window_cursor[s] += 1 + + total_trials = len(levels) + print(f"\nRunning {total_trials} trials...\n") + + new_entries = [] + for i, level in enumerate(levels): + # Rotate state by trial index + state = states[i % len(states)] + # Pick the next unused window for this state + w_idx = window_cursor[state] % len(windows) + window = windows[w_idx] + window_cursor[state] += 1 + sites = site_cache[state] + + print( + f" [{i + 1}/{total_trials}] level={level:>2}, state={state:<15}, " + f"window={window}...", + end=" ", + flush=True, + ) + + result = _run_trial(sites, window, level, args.timeout) + result["state"] = state + _append_log(result) + new_entries.append(result) + + # Print inline result + if result["outcome"] == "success": + print(f"✓ {result['wall_clock_s']}s, {result['n_records']} records") + else: + print( + f"✗ {result['outcome']} after {result['wall_clock_s']}s " + f"({result.get('error_type', '?')})" + ) + + # Brief pause between trials to be polite to the server + if i < total_trials - 1: + time.sleep(2) + + # Print full summary + all_entries = _load_log() + _print_summary(all_entries) + + +if __name__ == "__main__": + main() diff --git a/experiments/parallel_threshold_log.jsonl b/experiments/parallel_threshold_log.jsonl new file mode 100644 index 00000000..74bf54bd --- /dev/null +++ b/experiments/parallel_threshold_log.jsonl @@ -0,0 +1,81 @@ +{"timestamp": "2026-07-16T16:03:08.198046+00:00", "n_sites": 246, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 11.09, "n_records": 5747, "error_type": null, "error_msg": null, "state": "Delaware"} +{"timestamp": "2026-07-16T16:03:21.294922+00:00", "n_sites": 246, "time_window": "2008-01-01/2008-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 5.87, "n_records": 6588, "error_type": null, "error_msg": null, "state": "Delaware"} +{"timestamp": "2026-07-16T16:03:29.175792+00:00", "n_sites": 246, "time_window": "2011-01-01/2011-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 7.28, "n_records": 5475, "error_type": null, "error_msg": null, "state": "Delaware"} +{"timestamp": "2026-07-16T16:03:38.466959+00:00", "n_sites": 246, "time_window": "2014-01-01/2014-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.59, "n_records": 6181, "error_type": null, "error_msg": null, "state": "Delaware"} +{"timestamp": "2026-07-16T16:03:43.067729+00:00", "n_sites": 246, "time_window": "2017-01-01/2017-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.3, "n_records": 6524, "error_type": null, "error_msg": null, "state": "Delaware"} +{"timestamp": "2026-07-16T16:03:47.370355+00:00", "n_sites": 246, "time_window": "2020-01-01/2020-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.32, "n_records": 6224, "error_type": null, "error_msg": null, "state": "Delaware"} +{"timestamp": "2026-07-16T16:03:58.613903+00:00", "n_sites": 1081, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 7.62, "n_records": 14143, "error_type": null, "error_msg": null, "state": "New Hampshire"} +{"timestamp": "2026-07-16T16:04:08.238261+00:00", "n_sites": 1081, "time_window": "2008-01-01/2008-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 3.25, "n_records": 15084, "error_type": null, "error_msg": null, "state": "New Hampshire"} +{"timestamp": "2026-07-16T16:04:13.498869+00:00", "n_sites": 1081, "time_window": "2011-01-01/2011-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 5.17, "n_records": 18265, "error_type": null, "error_msg": null, "state": "New Hampshire"} +{"timestamp": "2026-07-16T16:04:20.680625+00:00", "n_sites": 1081, "time_window": "2014-01-01/2014-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 3.32, "n_records": 17647, "error_type": null, "error_msg": null, "state": "New Hampshire"} +{"timestamp": "2026-07-16T16:04:26.010006+00:00", "n_sites": 1081, "time_window": "2017-01-01/2017-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 3.13, "n_records": 19072, "error_type": null, "error_msg": null, "state": "New Hampshire"} +{"timestamp": "2026-07-16T16:04:31.150457+00:00", "n_sites": 1081, "time_window": "2020-01-01/2020-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.82, "n_records": 19764, "error_type": null, "error_msg": null, "state": "New Hampshire"} +{"timestamp": "2026-07-16T16:04:39.410640+00:00", "n_sites": 1184, "time_window": "2005-01-01/2005-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.42, "n_records": 18410, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:04:43.838766+00:00", "n_sites": 1184, "time_window": "2008-01-01/2008-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.5, "n_records": 21599, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:04:48.347019+00:00", "n_sites": 1184, "time_window": "2011-01-01/2011-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.08, "n_records": 24111, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:04:52.435878+00:00", "n_sites": 1184, "time_window": "2014-01-01/2014-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.04, "n_records": 23083, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:04:56.487218+00:00", "n_sites": 1184, "time_window": "2017-01-01/2017-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.99, "n_records": 23012, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:05:17.596051+00:00", "n_sites": 2889, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 7.6, "n_records": 47363, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T16:05:27.212362+00:00", "n_sites": 2889, "time_window": "2008-01-01/2008-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 8.62, "n_records": 52338, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T16:05:37.853445+00:00", "n_sites": 2889, "time_window": "2011-01-01/2011-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 7.44, "n_records": 57044, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T16:05:47.316958+00:00", "n_sites": 2889, "time_window": "2014-01-01/2014-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 8.6, "n_records": 65308, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T16:05:57.944067+00:00", "n_sites": 2889, "time_window": "2017-01-01/2017-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 9.58, "n_records": 75628, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T16:06:15.202809+00:00", "n_sites": 5676, "time_window": "2005-01-01/2005-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 6.4, "n_records": 81500, "error_type": null, "error_msg": null, "state": "Pennsylvania"} +{"timestamp": "2026-07-16T16:06:23.632642+00:00", "n_sites": 5676, "time_window": "2008-01-01/2008-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 14.09, "n_records": 88151, "error_type": null, "error_msg": null, "state": "Pennsylvania"} +{"timestamp": "2026-07-16T16:06:39.762735+00:00", "n_sites": 5676, "time_window": "2011-01-01/2011-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 4.26, "n_records": 92276, "error_type": null, "error_msg": null, "state": "Pennsylvania"} +{"timestamp": "2026-07-16T16:06:46.055454+00:00", "n_sites": 5676, "time_window": "2014-01-01/2014-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 7.93, "n_records": 99731, "error_type": null, "error_msg": null, "state": "Pennsylvania"} +{"timestamp": "2026-07-16T16:07:56.941675+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2003-01-01/2007-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 39.96, "n_records": 410821, "error_type": null, "error_msg": null, "note": "5yr_window_rapid_fire"} +{"timestamp": "2026-07-16T16:08:23.701431+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2008-01-01/2012-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 26.76, "n_records": 457336, "error_type": null, "error_msg": null, "note": "5yr_window_rapid_fire"} +{"timestamp": "2026-07-16T16:08:45.738312+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2013-01-01/2017-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 22.04, "n_records": 500442, "error_type": null, "error_msg": null, "note": "5yr_window_rapid_fire"} +{"timestamp": "2026-07-16T16:17:44.822601+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2005-01-01/2014-12-31", "parallelism": 32, "timeout_s": 180, "outcome": "success", "wall_clock_s": 32.61, "n_records": 556354, "error_type": null, "error_msg": null, "note": "10yr_window_stress"} +{"timestamp": "2026-07-16T16:17:51.889708+00:00", "n_sites": 1184, "time_window": "2005-01-01/2005-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.78, "n_records": 18410, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:17:55.673983+00:00", "n_sites": 1184, "time_window": "2008-01-01/2008-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.79, "n_records": 21599, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:17:58.471864+00:00", "n_sites": 1184, "time_window": "2011-01-01/2011-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.99, "n_records": 24111, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:18:02.472114+00:00", "n_sites": 1184, "time_window": "2014-01-01/2014-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 3.26, "n_records": 23083, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:18:07.743656+00:00", "n_sites": 1184, "time_window": "2017-01-01/2017-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.89, "n_records": 23012, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:18:12.641117+00:00", "n_sites": 1184, "time_window": "2020-01-01/2020-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.8, "n_records": 24575, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T16:18:54.508351+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2020-03-01/2020-03-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 23.01, "n_records": 209727, "error_type": null, "error_msg": null, "note": "get_continuous"} +{"timestamp": "2026-07-16T16:19:05.106056+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2020-03-01/2020-03-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 9.6, "n_records": 209727, "error_type": null, "error_msg": null, "note": "get_continuous"} +{"timestamp": "2026-07-16T16:19:26.233466+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2020-03-01/2020-03-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 20.12, "n_records": 209727, "error_type": null, "error_msg": null, "note": "get_continuous"} +{"timestamp": "2026-07-16T16:20:27.946708+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2019-06-01/2019-06-30", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 46.0, "n_records": 776037, "error_type": null, "error_msg": null, "note": "get_continuous"} +{"timestamp": "2026-07-16T16:21:14.615471+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2018-07-01/2018-07-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 44.66, "n_records": 766944, "error_type": null, "error_msg": null, "note": "get_continuous"} +{"timestamp": "2026-07-16T16:21:55.409587+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2017-08-01/2017-08-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 38.79, "n_records": 767713, "error_type": null, "error_msg": null, "note": "get_continuous"} +{"timestamp": "2026-07-16T16:37:09.647936+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2019-01-01/2019-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 13.43, "n_records": 79808, "error_type": null, "error_msg": null, "note": "rapid_fire_no_pause"} +{"timestamp": "2026-07-16T16:37:15.882284+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2020-01-01/2020-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 6.23, "n_records": 106835, "error_type": null, "error_msg": null, "note": "rapid_fire_no_pause"} +{"timestamp": "2026-07-16T16:37:22.167812+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2021-01-01/2021-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 6.28, "n_records": 78369, "error_type": null, "error_msg": null, "note": "rapid_fire_no_pause"} +{"timestamp": "2026-07-16T16:37:24.593376+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2019-01-01/2019-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.43, "n_records": 22995, "error_type": null, "error_msg": null, "note": "rapid_fire_no_pause"} +{"timestamp": "2026-07-16T16:37:30.638632+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2022-01-01/2022-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 6.04, "n_records": 108454, "error_type": null, "error_msg": null, "note": "rapid_fire_no_pause"} +{"timestamp": "2026-07-16T16:38:30.551883+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2022-01-01/2022-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 11.6, "n_records": 80607, "error_type": null, "error_msg": null, "note": "rapid_fire_unique_combos"} +{"timestamp": "2026-07-16T16:38:42.305398+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2015-01-01/2015-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 11.75, "n_records": 102149, "error_type": null, "error_msg": null, "note": "rapid_fire_unique_combos"} +{"timestamp": "2026-07-16T16:38:42.671570+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2023-01-01/2023-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "error", "wall_clock_s": 0.37, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/32 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_unique_combos"} +{"timestamp": "2026-07-16T16:38:43.115655+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2016-01-01/2016-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "error", "wall_clock_s": 0.44, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/32 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_unique_combos"} +{"timestamp": "2026-07-16T16:38:43.473863+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2020-01-01/2020-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "error", "wall_clock_s": 0.36, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/32 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_unique_combos"} +{"timestamp": "2026-07-16T17:02:51.026246+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2015-01-01/2015-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 26.3, "n_records": 71579, "error_type": null, "error_msg": null, "note": "rapid_fire_n8_unique"} +{"timestamp": "2026-07-16T17:02:56.344273+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2019-01-01/2019-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 5.32, "n_records": 101172, "error_type": null, "error_msg": null, "note": "rapid_fire_n8_unique"} +{"timestamp": "2026-07-16T17:02:56.678456+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2016-01-01/2016-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "interrupted", "wall_clock_s": 0.33, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/12 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_n8_unique"} +{"timestamp": "2026-07-16T17:02:57.076010+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2021-01-01/2021-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "interrupted", "wall_clock_s": 0.4, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/18 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_n8_unique"} +{"timestamp": "2026-07-16T17:02:57.415448+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2018-01-01/2018-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "interrupted", "wall_clock_s": 0.34, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/12 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_n8_unique"} +{"timestamp": "2026-07-16T18:26:58.642855+00:00", "n_sites": 2889, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 27.38, "n_records": 47363, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T18:27:28.089128+00:00", "n_sites": 5676, "time_window": "2005-01-01/2005-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 8.27, "n_records": 81500, "error_type": null, "error_msg": null, "state": "Pennsylvania"} +{"timestamp": "2026-07-16T18:27:38.391844+00:00", "n_sites": 1184, "time_window": "2005-01-01/2005-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 4.62, "n_records": 18410, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T18:27:45.019968+00:00", "n_sites": 1081, "time_window": "2005-01-01/2005-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.17, "n_records": 14143, "error_type": null, "error_msg": null, "state": "New Hampshire"} +{"timestamp": "2026-07-16T18:27:49.200102+00:00", "n_sites": 570, "time_window": "2005-01-01/2005-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.1, "n_records": 16453, "error_type": null, "error_msg": null, "state": "Vermont"} +{"timestamp": "2026-07-16T18:27:53.311784+00:00", "n_sites": 2889, "time_window": "2008-01-01/2008-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 4.85, "n_records": 52338, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T18:28:15.357707+00:00", "n_sites": 2889, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.0, "n_records": 47363, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T18:28:18.372315+00:00", "n_sites": 5676, "time_window": "2005-01-01/2005-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.29, "n_records": 81500, "error_type": null, "error_msg": null, "state": "Pennsylvania"} +{"timestamp": "2026-07-16T18:28:21.687470+00:00", "n_sites": 1184, "time_window": "2005-01-01/2005-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.63, "n_records": 18410, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T18:28:24.323955+00:00", "n_sites": 1081, "time_window": "2005-01-01/2005-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.52, "n_records": 14143, "error_type": null, "error_msg": null, "state": "New Hampshire"} +{"timestamp": "2026-07-16T18:28:26.853950+00:00", "n_sites": 570, "time_window": "2005-01-01/2005-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.52, "n_records": 16453, "error_type": null, "error_msg": null, "state": "Vermont"} +{"timestamp": "2026-07-16T18:28:29.377015+00:00", "n_sites": 2889, "time_window": "2008-01-01/2008-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.91, "n_records": 52338, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T18:28:47.115674+00:00", "n_sites": 2889, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.06, "n_records": 47363, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T18:28:50.189062+00:00", "n_sites": 5676, "time_window": "2005-01-01/2005-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.34, "n_records": 81500, "error_type": null, "error_msg": null, "state": "Pennsylvania"} +{"timestamp": "2026-07-16T18:28:53.559101+00:00", "n_sites": 1184, "time_window": "2020-01-01/2020-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 4.11, "n_records": 24575, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T18:28:59.674982+00:00", "n_sites": 1081, "time_window": "2011-01-01/2011-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.96, "n_records": 18265, "error_type": null, "error_msg": null, "state": "New Hampshire"} +{"timestamp": "2026-07-16T18:29:04.645868+00:00", "n_sites": 570, "time_window": "2011-01-01/2011-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.63, "n_records": 19488, "error_type": null, "error_msg": null, "state": "Vermont"} +{"timestamp": "2026-07-16T18:29:08.279422+00:00", "n_sites": 2889, "time_window": "2008-01-01/2008-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.89, "n_records": 52338, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T18:29:26.662442+00:00", "n_sites": 2889, "time_window": "2011-01-01/2011-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 27.56, "n_records": 57044, "error_type": null, "error_msg": null, "state": "Ohio"} +{"timestamp": "2026-07-16T18:29:56.240019+00:00", "n_sites": 5676, "time_window": "2008-01-01/2008-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 15.37, "n_records": 88151, "error_type": null, "error_msg": null, "state": "Pennsylvania"} +{"timestamp": "2026-07-16T18:30:13.650984+00:00", "n_sites": 1184, "time_window": "2005-01-01/2005-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.64, "n_records": 18410, "error_type": null, "error_msg": null, "state": "Connecticut"} +{"timestamp": "2026-07-16T18:30:16.300995+00:00", "n_sites": 1081, "time_window": "2014-01-01/2014-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 8.89, "n_records": 17647, "error_type": null, "error_msg": null, "state": "New Hampshire"} +{"timestamp": "2026-07-16T18:30:27.195785+00:00", "n_sites": 570, "time_window": "2014-01-01/2014-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 17.83, "n_records": 16339, "error_type": null, "error_msg": null, "state": "Vermont"} +{"timestamp": "2026-07-16T18:30:47.035453+00:00", "n_sites": 2889, "time_window": "2014-01-01/2014-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 11.38, "n_records": 65308, "error_type": null, "error_msg": null, "state": "Ohio"} From 2d6c4b9372b93a514ec3d8248412c9c02655f214 Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Fri, 17 Jul 2026 07:39:18 -0500 Subject: [PATCH 07/13] fix(waterdata): make parallel_chunks cap a hard ceiling in ChunkPlan._refine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit _refine grew the plan with `while self.total < max_chunks`, but with more than one axis a single split multiplies total by (k+1)/k for the split axis — it adds the product of the *other* axes, not one — so total could step *past* the cap (two 8-atom axes at cap 10 landed on 12; three 10-atom axes at cap 30 landed on 32). That broke the documented "at most n / can't multiply past it" guarantee the dial exists to provide. Take a split only when it keeps total within the cap: skip any axis whose split would overshoot (the increment is per-axis, total // k). The cap is now a true ceiling — never exceeded — and the plan lands below it when no whole split hits it exactly (two even axes reach 4, not 5). Single-axis plans are unchanged (still hit n exactly, or saturate at one atom/chunk). Docstrings updated (soft cap -> hard ceiling; note the may-undershoot). Repurpose the many-axes test to a cap the old loop overshot (30, was 32) and add a parametrized regression over the 2-axis overshoot combos. Signed-off-by: thodson-usgs Co-Authored-By: Claude Opus 4.8 (1M context) --- dataretrieval/ogc/chunking.py | 11 +++--- dataretrieval/ogc/planning.py | 59 ++++++++++++++++++++++---------- tests/waterdata_chunking_test.py | 48 +++++++++++++++++++++----- 3 files changed, 87 insertions(+), 31 deletions(-) diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index c6aa2357..7df31138 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -224,10 +224,13 @@ def parallel_chunks(n: int) -> Iterator[None]: integer such as ``2``, ``8``, or ``32``. It caps the plan's *total* sub-request count (the cartesian product across every multi-value argument combined, not per argument), so several multi-value arguments - cannot multiply past it. The actual count is bounded below by what the - ~8 KB URL limit already forces and above by the number of values there - are to split, so an ``n`` larger than the input allows simply yields one - sub-request per value; ``n=1`` asks for no extra fan-out. + cannot multiply past it. The cap is a ceiling, never exceeded: the + actual count is bounded below by what the ~8 KB URL limit already + forces and above by ``n``, so an ``n`` larger than the input allows + simply yields one sub-request per value, and with several multi-value + arguments the total may land somewhat below ``n`` because splits are + whole (the plan can't always divide evenly onto ``n``); ``n=1`` asks + for no extra fan-out. Each sub-request fetches at least one page, so it costs at least one request against your hourly rate limit — a larger ``n`` spends more diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index a32e48c6..3fa1855a 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -330,15 +330,16 @@ class ChunkPlan: Byte budget for the request (URL + body) — a hard ceiling every sub-request must fit. max_chunks : int, optional - Soft cap on the plan's total sub-request count (default ``0`` = off). + Hard cap on the plan's total sub-request count (default ``0`` = off). ``0`` chunks only as much as ``url_limit`` requires — the most conservative plan, fewest sub-requests. A positive cap fans the plan out to up to ``max_chunks`` sub-requests overall (the cartesian product across axes, never fewer than the byte budget already forces) — capped as a whole, not per axis, so several - multi-value axes can't multiply past the cap. Set from the - :func:`~dataretrieval.ogc.chunking.parallel_chunks` ``n``; see - :meth:`_refine`. + multi-value axes can't multiply past the cap. The plan never exceeds + the cap and may land below it when no whole split lands on it exactly. + Set from the :func:`~dataretrieval.ogc.chunking.parallel_chunks` ``n``; + see :meth:`_refine`. Attributes ---------- @@ -502,38 +503,58 @@ def _refine(self, max_chunks: int) -> None: cartesian product across all axes) at ``max_chunks``, not each axis independently — with several multi-value axes, a cap of 32 still means at most 32 sub-requests overall, not ``32 ** n_axes``. - Each split picks the single largest splittable chunk across *every* - axis (ties broken by axis-extraction order, then lowest index), so - growth is distributed round-robin rather than one axis saturating - before another is touched. Purely additive — only ever *splits* - existing chunks, so the byte pass's work and the ``url_limit`` - invariant are both preserved, and it never raises. A no-op at - ``max_chunks <= 0``. + The cap is a hard ceiling: every split multiplies the plan by + ``(k+1)/k`` for the chosen axis (adding ``total // k`` sub-requests, + not one), so a split is taken only when it keeps ``total`` within the + cap. When no in-budget split remains the plan lands *below* the cap + rather than overshooting it — a multi-axis plan may not divide evenly + onto ``max_chunks`` (e.g. two even axes can reach 4 but not 5, so a + cap of 5 yields 4). Each split picks the single largest splittable + chunk among the in-budget axes (ties broken by axis-extraction order, + then lowest index), so growth is distributed round-robin rather than + one axis saturating before another is touched. Purely additive — only + ever *splits* existing chunks, so the byte pass's work and the + ``url_limit`` invariant are both preserved, and it never raises. A + no-op at ``max_chunks <= 0``. Parameters ---------- max_chunks : int - Soft cap on the plan's total sub-request count (``0`` = off) — the + Hard cap on the plan's total sub-request count (``0`` = off) — the ``parallel_chunks(n)`` value. The cap is on the whole plan (the cartesian product across axes), not per axis, so several multi-value - axes can't multiply past it. + axes can't multiply past it; the plan never exceeds it and may fall + short when no whole split lands on it exactly. """ if max_chunks <= 0: return - while self.total < max_chunks: - # Largest splittable chunk across every axis; a chunk of size 1 - # can't be split further. ``max`` with a stable input order - # breaks ties by axis order, then lowest index within an axis. + while True: + total = self.total + if total >= max_chunks: + return + # Largest splittable chunk among the axes whose split still fits the + # cap. Splitting any chunk of an axis with ``k`` chunks turns that + # ``k`` into ``k+1``, so it adds ``total // k`` sub-requests (the + # product of the other axes) regardless of which chunk — hence the + # budget test is per axis, not per chunk. Skipping an over-budget + # axis makes ``max_chunks`` a true ceiling. A chunk of size 1 can't + # be split further. ``max`` with a stable input order breaks ties by + # axis order, then lowest index within an axis. candidate: tuple[_Axis, int] | None = None candidate_size = -1 for axis in self.axes: - for idx, chunk in enumerate(self.chunks[axis.arg_key]): + axis_chunks = self.chunks[axis.arg_key] + if total + total // len(axis_chunks) > max_chunks: + continue # any split of this axis would overshoot the cap + for idx, chunk in enumerate(axis_chunks): if len(chunk) <= 1: continue if len(chunk) > candidate_size: candidate, candidate_size = (axis, idx), len(chunk) if candidate is None: - return # every axis saturated at one atom per chunk + # Every axis is saturated at one atom per chunk or would + # overshoot the cap; stop below it rather than exceed it. + return axis, idx = candidate _split_at(self.chunks[axis.arg_key], idx) diff --git a/tests/waterdata_chunking_test.py b/tests/waterdata_chunking_test.py index 850b8709..abfe6674 100644 --- a/tests/waterdata_chunking_test.py +++ b/tests/waterdata_chunking_test.py @@ -2237,20 +2237,52 @@ def test_cap_caps_the_total_across_axes(): def test_cap_bounds_fan_out_across_many_axes(): - """The guardrail holds regardless of axis count: three multi-value axes - at ``n=32`` still top out at ``n`` sub-requests total, not ``n ** 3`` — - the property the single-axis-only cap in the original implementation did - not guarantee.""" - high = 32 + """The guardrail holds regardless of axis count: three multi-value axes at + a cap of 30 fan out to *at most* 30 sub-requests total — never the + ``30 ** 3`` a per-axis cap would allow, and never *over* the cap either. + 30 is deliberately not evenly reachable by these axes: a single split + multiplies the plan by more than one, so the naive ``while total < cap`` + the first refine used stepped past 30 (to 32). The cap is a hard ceiling — + the property neither the single-axis-only cap nor that naive loop + guaranteed.""" + cap = 30 # Three chunkable axes (two list axes + the filter OR-axis), each with 10 - # atoms — under the old per-axis cap this would have been high**3. + # atoms — under the old per-axis cap this would have been cap**3. args = { "monitoring_location_id": [f"L{i}" for i in range(10)], "parameter_code": [f"{i:05d}" for i in range(10)], "filter": " OR ".join(f"p='{i}'" for i in range(10)), } - plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=high) - assert plan.total <= high + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=cap) + assert 1 < plan.total <= cap # fanned out, but never past the ceiling + + +@pytest.mark.parametrize( + "atoms_per_axis, cap", + [ + (4, 5), # pre-fix loop overshot 5 -> 6 + (8, 10), # pre-fix loop overshot 10 -> 12 + (10, 7), # pre-fix loop overshot 7 -> 8 + ], +) +def test_cap_is_a_hard_ceiling_never_overshoots(atoms_per_axis, cap): + """The cap is a hard ceiling, not a soft target. With two multi-value axes + a single split multiplies the plan by ``(k+1)/k`` for the split axis — + adding the product of the *other* axes, not one — so a naive + ``while total < cap`` loop steps *past* the cap. These are exactly the + (atoms, cap) combos that loop overshot (5->6, 10->12, 7->8). The plan must + fan out and cover every atom once, but never exceed the cap, landing below + it when no whole split lands on it exactly (two even axes reach 4, not 5).""" + args = { + "monitoring_location_id": [f"L{i:03d}" for i in range(atoms_per_axis)], + "parameter_code": [f"{i:05d}" for i in range(atoms_per_axis)], + } + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=cap) + assert 1 < plan.total <= cap # fanned out, but never past the ceiling + # Every atom on every axis is still covered exactly once. + for key, atoms in args.items(): + flattened = [a for chunk in plan.chunks[key] for a in chunk] + assert sorted(flattened) == sorted(atoms) def test_cap_does_not_mask_unchunkable(): From 8d4c4a1b1775dbdaab9db0d85a97b449cc5767b0 Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Fri, 17 Jul 2026 08:14:57 -0500 Subject: [PATCH 08/13] fix(waterdata): tighten ChunkPlan.max_chunks contract (n=1 passthrough, negative raises) Two edges from the SOLID review of the parallel_chunks dial: - max_chunks=1 ("no extra fan-out") took the fan-out path: on a fitting request it materialized axes and re-rendered them through iter_sub_args before _refine(1) no-oped, instead of the verbatim passthrough. Widen the passthrough guard to `max_chunks <= 1` so 0 (off) and 1 (no fan-out) both short-circuit to the trivial plan. - A negative max_chunks silently no-oped. It can only be a caller bug (the public parallel_chunks(n) already rejects n < 1), so raise ValueError at construction rather than mask it. Docstring + Raises updated; adds a unit passthrough test for max_chunks=1 and a negative-raises test. Signed-off-by: thodson-usgs Co-Authored-By: Claude Opus 4.8 (1M context) --- dataretrieval/ogc/planning.py | 35 +++++++++++++++++++++++--------- tests/waterdata_chunking_test.py | 21 +++++++++++++++++++ 2 files changed, 46 insertions(+), 10 deletions(-) diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index 3fa1855a..ddb64739 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -331,13 +331,15 @@ class ChunkPlan: sub-request must fit. max_chunks : int, optional Hard cap on the plan's total sub-request count (default ``0`` = off). - ``0`` chunks only as much as ``url_limit`` requires — the most - conservative plan, fewest sub-requests. A positive cap fans the plan - out to up to ``max_chunks`` sub-requests overall (the - cartesian product across axes, never fewer than the byte budget - already forces) — capped as a whole, not per axis, so several - multi-value axes can't multiply past the cap. The plan never exceeds - the cap and may land below it when no whole split lands on it exactly. + ``0`` (off) and ``1`` (no extra fan-out) both chunk only as much as + ``url_limit`` requires — the most conservative plan, fewest + sub-requests — so a fitting request is a passthrough under either. A + cap of ``2`` or more fans the plan out to up to ``max_chunks`` + sub-requests overall (the cartesian product across axes, never fewer + than the byte budget already forces) — capped as a whole, not per axis, + so several multi-value axes can't multiply past the cap. The plan never + exceeds the cap and may land below it when no whole split lands on it + exactly. A negative cap is a caller error and raises ``ValueError``. Set from the :func:`~dataretrieval.ogc.chunking.parallel_chunks` ``n``; see :meth:`_refine`. @@ -364,6 +366,8 @@ class ChunkPlan: Unchunkable If the request needs chunking but even the singleton plan doesn't fit ``url_limit``. + ValueError + If ``max_chunks`` is negative. """ def __init__( @@ -373,6 +377,16 @@ def __init__( url_limit: int, max_chunks: int = 0, ) -> None: + if max_chunks < 0: + # ``0`` disables fan-out (the ambient default outside any + # ``parallel_chunks`` block); a negative cap is meaningless and can + # only be a caller bug, so fail loudly rather than silently no-op. + # The public ``parallel_chunks(n)`` already rejects ``n < 1``; this + # guards direct construction. + raise ValueError( + f"max_chunks must be >= 0 (0 disables fan-out); got {max_chunks!r}." + ) + self.args = args self.axes: list[_Axis] = [] self.chunks: dict[str, list[list[str]]] = {} @@ -424,9 +438,10 @@ def __init__( # A request that already fits and hasn't opted into finer chunking is # the common passthrough: leave ``axes``/``chunks`` empty so # ``total == 1`` and ``iter_sub_args`` yields the original args - # verbatim. Only when ``max_chunks`` asks for extra fan-out do - # we set the axes up to be refined below. - if fits and max_chunks <= 0: + # verbatim. ``max_chunks`` of 0 (off) or 1 (no extra fan-out) both mean + # "don't split", so both take this path; only ``max_chunks >= 2`` asks + # for extra fan-out and sets the axes up to be refined below. + if fits and max_chunks <= 1: return self.axes = axes diff --git a/tests/waterdata_chunking_test.py b/tests/waterdata_chunking_test.py index abfe6674..8bdee6c0 100644 --- a/tests/waterdata_chunking_test.py +++ b/tests/waterdata_chunking_test.py @@ -2134,6 +2134,27 @@ def test_zero_cap_preserves_passthrough(): assert list(plan.iter_sub_args()) == [args] +def test_unit_cap_preserves_passthrough(): + """``max_chunks=1`` means "no extra fan-out", so a fitting multi-value + request stays the trivial passthrough (no axes, ``total == 1``, + ``iter_sub_args`` yields the original args verbatim) — identical to the off + (0) case, not a materialized one-chunk-per-axis plan.""" + args = {"monitoring_location_id": ["A", "B", "C", "D"]} + plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=1) + assert plan.axes == [] + assert plan.total == 1 + assert list(plan.iter_sub_args()) == [args] + + +def test_negative_cap_raises(): + """A negative ``max_chunks`` is a caller bug, not a silent no-op: it raises + ``ValueError`` at construction. (The public ``parallel_chunks(n)`` already + rejects ``n < 1``; this pins the same guard on direct construction.)""" + args = {"monitoring_location_id": ["A", "B", "C", "D"]} + with pytest.raises(ValueError, match="max_chunks must be >= 0"): + ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=-1) + + @pytest.mark.parametrize( ("max_chunks", "expected_pieces"), [(0, 1), (2, 2), (8, 8), (16, 10), (32, 10)], From b611e9fb6ee685f7c0d1a3645f31b423eb3d0137 Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Fri, 17 Jul 2026 08:37:36 -0500 Subject: [PATCH 09/13] refactor(waterdata): make max_chunks a positive-integer count (1 = off, reject 0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the parallel_chunks contract cleanup. max_chunks is a sub-request *count*, so its valid domain is the positive integers. Move the "off" sentinel from 0 to 1 (which already behaves identically — no extra fan-out, per the earlier n=1-passthrough change) and reject anything below 1: - ChunkPlan.max_chunks default 0 -> 1; guard `< 0` -> `< 1`, so 0 and negatives now raise ValueError instead of silently no-oping. - _parallel_chunks ambient default 0 -> 1; read-site + comments updated. - _refine off-guard `<= 0` -> `<= 1` (clearer intent; the loop already no-oped at 1). - Docstrings (class param, Raises, _refine, ambient) updated. Tests: zero-cap passthrough -> default/unit passthrough; negative-raises -> invalid-raises parametrized over {0, -1}; ramp param (0,1) -> (1,1) with the coverage guard keyed on expected_pieces; the two byte-only baselines and the ambient-default assertions move 0 -> 1. The public parallel_chunks(n) already rejected n < 1, so no user-facing behavior changes. Signed-off-by: thodson-usgs Co-Authored-By: Claude Opus 4.8 (1M context) --- dataretrieval/ogc/chunking.py | 8 ++--- dataretrieval/ogc/planning.py | 56 +++++++++++++++++--------------- tests/waterdata_chunking_test.py | 40 ++++++++++++----------- 3 files changed, 54 insertions(+), 50 deletions(-) diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index 7df31138..3da68afe 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -184,9 +184,9 @@ def get_active_client() -> httpx.AsyncClient | None: # limit alone requires. Scoped to a ``with parallel_chunks(...):`` block (a # ContextVar), deliberately NOT an env var (see :func:`parallel_chunks` for # why). The ambient holds ``n`` — the requested cap on the plan's total -# sub-request count; ``0`` (the default, outside any block) means "chunk only as -# much as the byte limit needs". -_parallel_chunks: Ambient[int] = Ambient("ogc_parallel_chunks", 0) +# sub-request count; ``1`` (the default, outside any block) means "off — chunk +# only as much as the byte limit needs, no extra fan-out". +_parallel_chunks: Ambient[int] = Ambient("ogc_parallel_chunks", 1) @contextmanager @@ -739,7 +739,7 @@ def wrapper( ) -> tuple[pd.DataFrame, Any]: limit = _OGC_URL_BYTE_LIMIT if url_limit is None else url_limit # Read the parallel_chunks dial ``n`` from the ambient set by - # ``parallel_chunks`` (0 = off outside any such block; otherwise the + # ``parallel_chunks`` (1 = off outside any such block; otherwise the # requested total sub-request cap). It only affects *planning*, done # here up front, so a later resume — which re-issues the # already-planned sub-requests — needs no snapshot. diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index ddb64739..20fc45b2 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -330,18 +330,19 @@ class ChunkPlan: Byte budget for the request (URL + body) — a hard ceiling every sub-request must fit. max_chunks : int, optional - Hard cap on the plan's total sub-request count (default ``0`` = off). - ``0`` (off) and ``1`` (no extra fan-out) both chunk only as much as - ``url_limit`` requires — the most conservative plan, fewest - sub-requests — so a fitting request is a passthrough under either. A - cap of ``2`` or more fans the plan out to up to ``max_chunks`` - sub-requests overall (the cartesian product across axes, never fewer - than the byte budget already forces) — capped as a whole, not per axis, - so several multi-value axes can't multiply past the cap. The plan never - exceeds the cap and may land below it when no whole split lands on it - exactly. A negative cap is a caller error and raises ``ValueError``. - Set from the :func:`~dataretrieval.ogc.chunking.parallel_chunks` ``n``; - see :meth:`_refine`. + Hard cap on the plan's total sub-request count (default ``1`` = off). + ``1`` chunks only as much as ``url_limit`` requires — the most + conservative plan, fewest sub-requests — so a fitting request is a + passthrough. A cap of ``2`` or more fans the plan out to up to + ``max_chunks`` sub-requests overall (the cartesian product across axes, + never fewer than the byte budget already forces) — capped as a whole, + not per axis, so several multi-value axes can't multiply past the cap. + The plan never exceeds the cap and may land below it when no whole + split lands on it exactly. ``max_chunks`` is a sub-request count, so a + value below ``1`` (``0`` or negative) is a caller error and raises + ``ValueError``. Set from the + :func:`~dataretrieval.ogc.chunking.parallel_chunks` ``n``; see + :meth:`_refine`. Attributes ---------- @@ -367,7 +368,7 @@ class ChunkPlan: If the request needs chunking but even the singleton plan doesn't fit ``url_limit``. ValueError - If ``max_chunks`` is negative. + If ``max_chunks`` is less than 1 (0 or negative). """ def __init__( @@ -375,16 +376,17 @@ def __init__( args: dict[str, Any], build_request: Callable[..., httpx.Request], url_limit: int, - max_chunks: int = 0, + max_chunks: int = 1, ) -> None: - if max_chunks < 0: - # ``0`` disables fan-out (the ambient default outside any - # ``parallel_chunks`` block); a negative cap is meaningless and can - # only be a caller bug, so fail loudly rather than silently no-op. - # The public ``parallel_chunks(n)`` already rejects ``n < 1``; this - # guards direct construction. + if max_chunks < 1: + # ``max_chunks`` is a sub-request *count*: the minimum is ``1`` + # (the ambient default outside any ``parallel_chunks`` block), + # which means "off — no extra fan-out". ``0`` or negative is a + # meaningless count and can only be a caller bug, so fail loudly + # rather than silently no-op. The public ``parallel_chunks(n)`` + # already rejects ``n < 1``; this guards direct construction. raise ValueError( - f"max_chunks must be >= 0 (0 disables fan-out); got {max_chunks!r}." + f"max_chunks must be >= 1 (1 disables fan-out); got {max_chunks!r}." ) self.args = args @@ -438,8 +440,8 @@ def __init__( # A request that already fits and hasn't opted into finer chunking is # the common passthrough: leave ``axes``/``chunks`` empty so # ``total == 1`` and ``iter_sub_args`` yields the original args - # verbatim. ``max_chunks`` of 0 (off) or 1 (no extra fan-out) both mean - # "don't split", so both take this path; only ``max_chunks >= 2`` asks + # verbatim. ``max_chunks == 1`` (off / no extra fan-out) means + # "don't split", so it takes this path; only ``max_chunks >= 2`` asks # for extra fan-out and sets the axes up to be refined below. if fits and max_chunks <= 1: return @@ -452,7 +454,7 @@ def __init__( self._plan(build_request, url_limit) # Soft pass: optionally split further than the byte budget requires. # Purely additive — never re-raises, and the byte budget stays - # satisfied; a no-op at ``max_chunks <= 0``. + # satisfied; a no-op at ``max_chunks == 1``. self._refine(max_chunks) if self.canonical_url is None: @@ -530,18 +532,18 @@ def _refine(self, max_chunks: int) -> None: one axis saturating before another is touched. Purely additive — only ever *splits* existing chunks, so the byte pass's work and the ``url_limit`` invariant are both preserved, and it never raises. A - no-op at ``max_chunks <= 0``. + no-op at ``max_chunks == 1``. Parameters ---------- max_chunks : int - Hard cap on the plan's total sub-request count (``0`` = off) — the + Hard cap on the plan's total sub-request count (``1`` = off) — the ``parallel_chunks(n)`` value. The cap is on the whole plan (the cartesian product across axes), not per axis, so several multi-value axes can't multiply past it; the plan never exceeds it and may fall short when no whole split lands on it exactly. """ - if max_chunks <= 0: + if max_chunks <= 1: return while True: total = self.total diff --git a/tests/waterdata_chunking_test.py b/tests/waterdata_chunking_test.py index 8bdee6c0..600f05d8 100644 --- a/tests/waterdata_chunking_test.py +++ b/tests/waterdata_chunking_test.py @@ -2122,13 +2122,13 @@ async def fetch(args): # --------------------------------------------------------------------------- -def test_zero_cap_preserves_passthrough(): - """``max_chunks=0`` (the default) must not perturb the existing +def test_default_preserves_passthrough(): + """The default ``max_chunks`` (1 = off) must not perturb the existing plan: a multi-value request that fits the byte limit is still the trivial passthrough (no axes, ``total == 1``), byte-for-byte the pre-feature behavior.""" args = {"monitoring_location_id": ["A", "B", "C", "D"]} - plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=0) + plan = ChunkPlan(args, _fake_build, url_limit=8000) # default max_chunks=1 assert plan.axes == [] assert plan.total == 1 assert list(plan.iter_sub_args()) == [args] @@ -2137,8 +2137,8 @@ def test_zero_cap_preserves_passthrough(): def test_unit_cap_preserves_passthrough(): """``max_chunks=1`` means "no extra fan-out", so a fitting multi-value request stays the trivial passthrough (no axes, ``total == 1``, - ``iter_sub_args`` yields the original args verbatim) — identical to the off - (0) case, not a materialized one-chunk-per-axis plan.""" + ``iter_sub_args`` yields the original args verbatim) — identical to the + default (off), not a materialized one-chunk-per-axis plan.""" args = {"monitoring_location_id": ["A", "B", "C", "D"]} plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=1) assert plan.axes == [] @@ -2146,25 +2146,27 @@ def test_unit_cap_preserves_passthrough(): assert list(plan.iter_sub_args()) == [args] -def test_negative_cap_raises(): - """A negative ``max_chunks`` is a caller bug, not a silent no-op: it raises - ``ValueError`` at construction. (The public ``parallel_chunks(n)`` already - rejects ``n < 1``; this pins the same guard on direct construction.)""" +@pytest.mark.parametrize("bad", [0, -1]) +def test_invalid_cap_raises(bad): + """``max_chunks`` is a sub-request count, so a value below 1 (``0`` or + negative) is a caller bug, not a silent no-op: it raises ``ValueError`` at + construction. (The public ``parallel_chunks(n)`` already rejects ``n < 1``; + this pins the same guard on direct construction.)""" args = {"monitoring_location_id": ["A", "B", "C", "D"]} - with pytest.raises(ValueError, match="max_chunks must be >= 0"): - ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=-1) + with pytest.raises(ValueError, match="max_chunks must be >= 1"): + ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=bad) @pytest.mark.parametrize( ("max_chunks", "expected_pieces"), - [(0, 1), (2, 2), (8, 8), (16, 10), (32, 10)], + [(1, 1), (2, 2), (8, 8), (16, 10), (32, 10)], ) def test_cap_ramps_then_saturates(max_chunks, expected_pieces): """A single 10-atom axis that fits the byte limit splits into ``min(10, cap)`` pieces: 1 (off), 2, 8, then saturating at 10 (one atom per chunk) once the cap overshoots the atom count. Monotonic and bounded, and whenever it splits the partition is a cover — every atom exactly once. (The - cap-0 passthrough has no axis to cover; see the passthrough test.)""" + cap-1 passthrough has no axis to cover; see the passthrough test.)""" atoms = [f"S{i:02d}" for i in range(10)] plan = ChunkPlan( {"monitoring_location_id": atoms}, @@ -2173,7 +2175,7 @@ def test_cap_ramps_then_saturates(max_chunks, expected_pieces): max_chunks=max_chunks, ) assert plan.total == expected_pieces - if max_chunks: + if expected_pieces > 1: flattened = [ a for chunk in plan.chunks["monitoring_location_id"] for a in chunk ] @@ -2205,7 +2207,7 @@ def test_cap_below_byte_split_does_not_reduce_fan_out(): # Heavy axis of four 30-char atoms; a limit tight enough that the byte pass # must drive every atom into its own sub-request (4 pieces > the cap of 2). args = {"monitoring_location_id": ["X" * 30, "Y" * 30, "Z" * 30, "W" * 30]} - baseline = ChunkPlan(args, _fake_build, url_limit=250, max_chunks=0) + baseline = ChunkPlan(args, _fake_build, url_limit=250, max_chunks=1) assert baseline.total > 2 # byte pass alone already fanned out past 2 refined = ChunkPlan(args, _fake_build, url_limit=250, max_chunks=2) # cap 2 < baseline pieces → refine is a no-op here. @@ -2218,7 +2220,7 @@ def test_cap_never_exceeds_the_byte_budget(): a chunk), and the fan-out is at least what the byte pass required.""" args = {"monitoring_location_id": ["X" * 30, "Y" * 30, "Z" * 30, "W" * 30]} limit = 310 - byte_only = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks=0) + byte_only = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks=1) plan = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks=32) assert plan.total >= byte_only.total for sub in plan.iter_sub_args(): @@ -2318,13 +2320,13 @@ def test_cap_does_not_mask_unchunkable(): def test_parallel_chunks_publishes_n_on_the_ambient(): """The context manager publishes ``n`` on the ambient for the block and restores the previous value on exit — including proper nesting.""" - assert _parallel_chunks.get() == 0 + assert _parallel_chunks.get() == 1 # default (off, = no extra fan-out) with parallel_chunks(32): assert _parallel_chunks.get() == 32 with parallel_chunks(2): assert _parallel_chunks.get() == 2 assert _parallel_chunks.get() == 32 # outer restored - assert _parallel_chunks.get() == 0 # default (off) outside any block + assert _parallel_chunks.get() == 1 # default (off) outside any block @pytest.mark.parametrize( @@ -2348,7 +2350,7 @@ def test_parallel_chunks_rejects_non_positive_int(bad): with pytest.raises(ValueError, match="must be a positive integer"): with parallel_chunks(bad): pass - assert _parallel_chunks.get() == 0 + assert _parallel_chunks.get() == 1 # default (off) — unchanged by a rejected call def test_parallel_chunks_drives_end_to_end_fan_out(): From 0cb5bfa8efa68dc43f2ca214f730c302c045b2f7 Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Fri, 17 Jul 2026 09:14:54 -0500 Subject: [PATCH 10/13] docs+refactor(ogc): address code-review findings 1/3/5/6 on parallel_chunks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the xhigh workflow review of the parallel_chunks branch: - [5] Extract the duplicated positive-integer validation (numbers.Integral + bool-exclusion + < 1) into utils._require_positive_int; call it from both max_rows (engine) and parallel_chunks (chunking) so the two count knobs can't drift. Drops the now-unused `import numbers` from both. ChunkPlan's own `< 1` guard keeps its domain-specific "1 disables fan-out" message. - [1,3] Document the consequences of fanning out a fitting request in the parallel_chunks docstring (new Notes section): with max_rows the result is drawn from the union of the sub-requests (a different, still-valid, sorted row set than the un-fanned call); a fanned-out call can fail partway and become resumable; cross-chunk dedup keys on id. These are the same caveats byte-forced chunking already carries — the block just extends their reach. - [6] Collapse the triplicated cap-contract prose in planning.py: the ChunkPlan.max_chunks param docstring is the single authority; _refine now documents only its algorithm and cross-references the contract. No behavior change. Messages preserve the substrings the tests match ("positive integer"); numpy-int caps still accepted, bool/float/str/< 1 still rejected. Signed-off-by: thodson-usgs Co-Authored-By: Claude Opus 4.8 (1M context) --- dataretrieval/ogc/chunking.py | 36 +++++++++++++++++++++------- dataretrieval/ogc/engine.py | 18 +++++--------- dataretrieval/ogc/planning.py | 44 +++++++++++++++-------------------- dataretrieval/utils.py | 28 ++++++++++++++++++++++ 4 files changed, 80 insertions(+), 46 deletions(-) diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index 3da68afe..d0dae01c 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -73,7 +73,6 @@ import asyncio import functools -import numbers import os from collections.abc import Callable, Iterator from contextlib import contextmanager @@ -84,7 +83,7 @@ import pandas as pd from anyio.from_thread import start_blocking_portal -from dataretrieval.utils import HTTPX_DEFAULTS, Ambient +from dataretrieval.utils import HTTPX_DEFAULTS, Ambient, _require_positive_int from . import progress as _progress from .interruptions import ( @@ -250,6 +249,29 @@ def parallel_chunks(n: int) -> Iterator[None]: any request is issued, so a bad value fails loudly rather than silently doing nothing. + Notes + ----- + Fanning out carries the same consequences as the byte-limit chunking the + getters already do for oversized requests; opting in just brings them to a + request that would otherwise be a single call: + + - ``max_rows``: each sub-request paginates up to ``max_rows`` rows + independently, then the combined result is sorted and truncated to + ``max_rows``. So a call with ``max_rows`` set returns a *different* + (though still valid and deterministically sorted) row set inside a + ``parallel_chunks`` block than without one — the cap is drawn from the + union of the sub-requests, not a single stream. Don't pair a tight + ``max_rows`` preview with ``parallel_chunks`` if you need exactly the + rows the un-fanned call would return. + - Resumability: a single request either fully succeeds or fully fails, + but a fanned-out call can fail partway (e.g. a mid-call rate-limit) and + raise a resumable :class:`~dataretrieval.exceptions.ChunkInterrupted` + (or ``QuotaExhausted``) carrying the completed sub-requests, which you + finish with ``exc.call.resume()``. + - Cross-sub-request de-duplication keys on the feature ``id``; features + with no ``id`` can't be deduped, so overlapping filter clauses split + across chunks may yield duplicate rows. + Examples -------- >>> from dataretrieval import waterdata @@ -262,13 +284,9 @@ def parallel_chunks(n: int) -> Iterator[None]: -------- ChunkPlan._refine : the planning-side effect of ``n``. """ - # Accept any integer type (incl. numpy ints, mirroring ``max_rows``); ``bool`` - # is an ``Integral`` subclass but nonsensical here, so reject it explicitly. - if not isinstance(n, numbers.Integral) or isinstance(n, bool) or n < 1: - raise ValueError( - f"parallel_chunks(n): n must be a positive integer (e.g. 2, 8, 32); " - f"got {n!r}." - ) + # Fail loudly on a bad ``n`` at ``with`` entry, before any request. Shared + # rules with ``max_rows`` via the helper (accepts numpy ints, rejects bool). + _require_positive_int(n, "parallel_chunks(n)", examples="2, 8, 32") with _parallel_chunks(n): yield diff --git a/dataretrieval/ogc/engine.py b/dataretrieval/ogc/engine.py index a9a7b9e1..668fa7b7 100644 --- a/dataretrieval/ogc/engine.py +++ b/dataretrieval/ogc/engine.py @@ -27,7 +27,6 @@ import functools import json import logging -import numbers import re from collections.abc import ( AsyncIterator, @@ -59,6 +58,7 @@ _default_headers, _get, _network_error, + _require_positive_int, ) # Set up logger for this module @@ -876,18 +876,12 @@ def get_ogc_data( - Handles optional arguments such as `convert_type`. - Applies column cleanup and reordering based on service and properties. """ - # Enforce a genuine positive integer: a float (even ``10.0``) or ``bool`` - # would pass a bare ``< 1`` check and then crash deep in + # Enforce a genuine positive integer up front: a float (even ``10.0``) or + # ``bool`` would pass a bare ``< 1`` check and then crash deep in # ``pd.DataFrame.head`` with an opaque ``TypeError`` after HTTP I/O has - # already fired. ``numbers.Integral`` (not ``int``) so numpy integers — - # e.g. ``max_rows`` derived from a numpy/pandas computation — are accepted; - # ``bool`` is an ``Integral`` subtype, so exclude it explicitly. - if max_rows is not None and ( - not isinstance(max_rows, numbers.Integral) - or isinstance(max_rows, bool) - or max_rows < 1 - ): - raise ValueError(f"max_rows must be a positive integer (got {max_rows!r}).") + # already fired. Shared with ``parallel_chunks(n)`` via the helper. + if max_rows is not None: + _require_positive_int(max_rows, "max_rows") if dialect is None: dialect = _DEFAULT_DIALECT diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index 20fc45b2..ed9446cb 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -512,36 +512,30 @@ def _plan( def _refine(self, max_chunks: int) -> None: """ Fan the plan out more finely than the byte budget alone requires — - the parallel_chunks dial (see + the ``parallel_chunks`` dial (see :func:`~dataretrieval.ogc.chunking.parallel_chunks` for why a caller - would want this). - - Caps the plan's *total* sub-request count (:attr:`total`, the - cartesian product across all axes) at ``max_chunks``, not - each axis independently — with several multi-value axes, a cap of 32 - still means at most 32 sub-requests overall, not ``32 ** n_axes``. - The cap is a hard ceiling: every split multiplies the plan by - ``(k+1)/k`` for the chosen axis (adding ``total // k`` sub-requests, - not one), so a split is taken only when it keeps ``total`` within the - cap. When no in-budget split remains the plan lands *below* the cap - rather than overshooting it — a multi-axis plan may not divide evenly - onto ``max_chunks`` (e.g. two even axes can reach 4 but not 5, so a - cap of 5 yields 4). Each split picks the single largest splittable - chunk among the in-budget axes (ties broken by axis-extraction order, - then lowest index), so growth is distributed round-robin rather than - one axis saturating before another is touched. Purely additive — only - ever *splits* existing chunks, so the byte pass's work and the - ``url_limit`` invariant are both preserved, and it never raises. A - no-op at ``max_chunks == 1``. + would want this, and :class:`ChunkPlan`'s ``max_chunks`` parameter for + the cap's contract: total-not-per-axis, a hard ceiling that may land + below the cap). + + Implementation. Each split multiplies the plan by ``(k+1)/k`` for the + chosen axis (adding ``total // k`` sub-requests, not one), so a split + is taken only when it keeps :attr:`total` within the cap; when no + in-budget split remains the plan stops *below* the cap rather than + overshooting (two even axes can reach 4 but not 5, so a cap of 5 yields + 4). Each split picks the single largest splittable chunk among the + in-budget axes (ties broken by axis-extraction order, then lowest + index), so growth is distributed round-robin rather than one axis + saturating before another is touched. Purely additive — only ever + *splits* existing chunks, so the byte pass's work and the ``url_limit`` + invariant are both preserved, and it never raises. A no-op at + ``max_chunks == 1``. Parameters ---------- max_chunks : int - Hard cap on the plan's total sub-request count (``1`` = off) — the - ``parallel_chunks(n)`` value. The cap is on the whole plan (the - cartesian product across axes), not per axis, so several multi-value - axes can't multiply past it; the plan never exceeds it and may fall - short when no whole split lands on it exactly. + The ``parallel_chunks(n)`` value; see :class:`ChunkPlan`'s + ``max_chunks`` parameter for the full contract. """ if max_chunks <= 1: return diff --git a/dataretrieval/utils.py b/dataretrieval/utils.py index 8d28ee86..3ccc4f28 100644 --- a/dataretrieval/utils.py +++ b/dataretrieval/utils.py @@ -4,6 +4,7 @@ from __future__ import annotations +import numbers import os import warnings from collections.abc import Callable, Iterable, Iterator @@ -66,6 +67,33 @@ def __call__(self, value: _T) -> Iterator[None]: self._var.reset(token) +def _require_positive_int( + value: int, name: str, *, examples: str | None = None +) -> None: + """Validate that ``value`` is a positive integer, else raise ``ValueError``. + + Accepts any :class:`numbers.Integral` (so a numpy/pandas integer passes, + not only ``int``) but rejects ``bool`` — an ``Integral`` subtype that is + nonsensical as a count. A non-integer (float, str, ``None``) or a value + ``< 1`` raises before any I/O, rather than crashing later (e.g. deep in + ``pd.DataFrame.head``). Shared by the count-like knobs (``max_rows`` and + ``parallel_chunks(n)``) so they validate by identical rules and can't drift. + + Parameters + ---------- + value : int + The value to check. Typed ``int`` for callers, but validated at + runtime because the real value may be anything the user passed. + name : str + Parameter name, used as the subject of the error message. + examples : str, optional + Illustrative values appended to the message (e.g. ``"2, 8, 32"``). + """ + if not isinstance(value, numbers.Integral) or isinstance(value, bool) or value < 1: + eg = f", e.g. {examples}" if examples else "" + raise ValueError(f"{name} must be a positive integer{eg} (got {value!r}).") + + def _default_headers() -> dict[str, str]: """Build the default HTTP headers for a USGS web-API request. From 9c33c83f67b3632d043d6838758f02302702ac69 Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Fri, 17 Jul 2026 12:29:07 -0500 Subject: [PATCH 11/13] docs(ogc): apply /simplify quality-review nits (comments only) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two documentation/comment fixes from the 4-angle cleanup review; no logic change: - utils._require_positive_int: soften the "count-like knobs ... can't drift" claim. ChunkPlan.max_chunks deliberately bypasses the helper (internal, already-int, keeps its own "1 disables fan-out" message), so scope the claim to the two user-facing boundary knobs (max_rows, parallel_chunks). - ChunkPlan._refine: note that its ranking key is atom count (len), not URL bytes like _plan, because the fan-out pass balances work rather than fitting a byte budget. Considered-but-skipped from the review: extracting a shared _largest_splittable helper for the _plan/_refine scans (line-neutral, adds callable indirection — the depth review judged it more indirection than it removes); removing _refine's max_chunks<=1 early-return (redundant with the while-guard but a clearer explicit "off" marker); routing ChunkPlan's <1 guard through _require_positive_int (different contract, better bespoke message); two _refine micro-optimizations (not worth it at max_chunks<=32). Signed-off-by: thodson-usgs Co-Authored-By: Claude Opus 4.8 (1M context) --- dataretrieval/ogc/planning.py | 8 +++++--- dataretrieval/utils.py | 7 +++++-- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index ed9446cb..e80d0643 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -548,9 +548,11 @@ def _refine(self, max_chunks: int) -> None: # ``k`` into ``k+1``, so it adds ``total // k`` sub-requests (the # product of the other axes) regardless of which chunk — hence the # budget test is per axis, not per chunk. Skipping an over-budget - # axis makes ``max_chunks`` a true ceiling. A chunk of size 1 can't - # be split further. ``max`` with a stable input order breaks ties by - # axis order, then lowest index within an axis. + # axis makes ``max_chunks`` a true ceiling. The ranking key is atom + # count (``len``), not URL bytes like ``_plan`` — this pass balances + # work across sub-requests rather than fitting a byte budget. A + # chunk of size 1 can't be split further. Stable input order breaks + # ties by axis order, then lowest index within an axis. candidate: tuple[_Axis, int] | None = None candidate_size = -1 for axis in self.axes: diff --git a/dataretrieval/utils.py b/dataretrieval/utils.py index 3ccc4f28..acdf7822 100644 --- a/dataretrieval/utils.py +++ b/dataretrieval/utils.py @@ -76,8 +76,11 @@ def _require_positive_int( not only ``int``) but rejects ``bool`` — an ``Integral`` subtype that is nonsensical as a count. A non-integer (float, str, ``None``) or a value ``< 1`` raises before any I/O, rather than crashing later (e.g. deep in - ``pd.DataFrame.head``). Shared by the count-like knobs (``max_rows`` and - ``parallel_chunks(n)``) so they validate by identical rules and can't drift. + ``pd.DataFrame.head``). Shared by the user-facing count knobs ``max_rows`` + and ``parallel_chunks(n)`` so their boundary validation can't drift. + (``ChunkPlan.max_chunks`` is an internal, already-``int`` precondition with + its own domain-specific message, so it keeps a lighter ``< 1`` guard rather + than routing through here.) Parameters ---------- From b9f9307c66b2ff9280d5781801a887d8542a8dd1 Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Fri, 17 Jul 2026 12:53:12 -0500 Subject: [PATCH 12/13] chore: drop local experiment scripts from the PR The threshold-sensitivity experiment harnesses and their captured .jsonl logs were scratch tooling used to choose the parallel_chunks defaults; they don't belong in the shipped library. Untrack them (kept locally, git-excluded) so the PR carries only the feature, tests, and docs. Signed-off-by: thodson-usgs Co-Authored-By: Claude Opus 4.8 (1M context) --- experiments/overload_threshold_experiment.py | 338 ------------------ experiments/overload_threshold_log.jsonl | 20 -- experiments/parallel_threshold_experiment.py | 350 ------------------- experiments/parallel_threshold_log.jsonl | 81 ----- 4 files changed, 789 deletions(-) delete mode 100644 experiments/overload_threshold_experiment.py delete mode 100644 experiments/overload_threshold_log.jsonl delete mode 100644 experiments/parallel_threshold_experiment.py delete mode 100644 experiments/parallel_threshold_log.jsonl diff --git a/experiments/overload_threshold_experiment.py b/experiments/overload_threshold_experiment.py deleted file mode 100644 index 4589d2dd..00000000 --- a/experiments/overload_threshold_experiment.py +++ /dev/null @@ -1,338 +0,0 @@ -""" -Experiment: Find the parallelism level that overloads the server. - -Strategy: -- For each parallelism level (starting low, escalating), query ALL states - at that level with sleep between each query. -- Each state gets a unique time window (never reused across runs). -- Log wall-clock time, outcome, and any signs of server stress (slow responses, - errors, partial results). -- Sleep between queries to stay below rate limit and isolate the effect of - parallelism from quota exhaustion. - -Usage: - python experiments/overload_threshold_experiment.py [--level N] [--sleep 10] - - If --level is not given, it auto-escalates through 1, 2, 4, 8, 16, 32. -""" - -from __future__ import annotations - -import json -import signal -import sys -import time -from datetime import datetime, timezone -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parents[1])) - -from dataretrieval import waterdata -from dataretrieval.ogc.interruptions import ChunkInterrupted, QuotaExhausted - -LOG_FILE = Path(__file__).with_name("overload_threshold_log.jsonl") - -# Large states — these put real pressure on the server -STATES = ["Ohio", "Pennsylvania", "Connecticut", "New Hampshire", "Vermont"] - -# Time windows pool — we rotate through these, never reusing a (state, window) -# pair across the entire log. -WINDOWS = [ - "2000-01-01/2000-12-31", - "2001-01-01/2001-12-31", - "2002-01-01/2002-12-31", - "2003-01-01/2003-12-31", - "2004-01-01/2004-12-31", - "2005-01-01/2005-12-31", - "2006-01-01/2006-12-31", - "2007-01-01/2007-12-31", - "2008-01-01/2008-12-31", - "2009-01-01/2009-12-31", - "2010-01-01/2010-12-31", - "2011-01-01/2011-12-31", - "2012-01-01/2012-12-31", - "2013-01-01/2013-12-31", - "2014-01-01/2014-12-31", - "2015-01-01/2015-12-31", - "2016-01-01/2016-12-31", - "2017-01-01/2017-12-31", - "2018-01-01/2018-12-31", - "2019-01-01/2019-12-31", - "2020-01-01/2020-12-31", - "2021-01-01/2021-12-31", - "2022-01-01/2022-12-31", - "2023-01-01/2023-12-31", - "2024-01-01/2024-12-31", -] - - -class HangTimeout(Exception): - pass - - -def _timeout_handler(signum, frame): - raise HangTimeout("hang") - - -def _load_log() -> list[dict]: - if not LOG_FILE.exists(): - return [] - entries = [] - with open(LOG_FILE) as f: - for line in f: - line = line.strip() - if line: - entries.append(json.loads(line)) - return entries - - -def _append_log(entry: dict) -> None: - with open(LOG_FILE, "a") as f: - f.write(json.dumps(entry) + "\n") - - -def _used_windows_for_state(entries: list[dict], state: str) -> set[str]: - """Return windows already used for a given state in the log.""" - return {e["time_window"] for e in entries if e.get("state") == state} - - -def _next_window(state: str, used: set[str]) -> str | None: - """Pick the next unused window for a state.""" - for w in WINDOWS: - if w not in used: - return w - return None - - -def run_level(level: int, sleep_s: int, timeout_s: int) -> list[dict]: - """Run all states at a given parallelism level. Returns the trial results.""" - entries = _load_log() - - # Preload sites - site_cache: dict[str, list[str]] = {} - for state in STATES: - print(f" Fetching {state} sites...", end=" ", flush=True) - sites, _ = waterdata.get_monitoring_locations(state=state, site_type_code="ST") - site_cache[state] = sites["monitoring_location_id"].tolist() - print(f"{len(site_cache[state])} sites") - - print( - f"\n Running all {len(STATES)} states at n={level}" - f" (sleep={sleep_s}s between):\n" - ) - - results = [] - for i, state in enumerate(STATES): - # Pick a fresh window - used = _used_windows_for_state(entries + results, state) - window = _next_window(state, used) - if window is None: - print(f" {state}: no fresh windows left, skipping") - continue - - site_ids = site_cache[state] - print( - f" [{i + 1}/{len(STATES)}] {state:<15} ({len(site_ids):>4} sites) " - f"{window} n={level}...", - end=" ", - flush=True, - ) - - old = signal.signal(signal.SIGALRM, _timeout_handler) - signal.alarm(timeout_s) - t0 = time.perf_counter() - - entry = { - "timestamp": datetime.now(timezone.utc).isoformat(), - "state": state, - "n_sites": len(site_ids), - "time_window": window, - "parallelism": level, - "timeout_s": timeout_s, - "sleep_s": sleep_s, - "outcome": None, - "wall_clock_s": None, - "n_records": None, - "error_type": None, - "error_msg": None, - } - - try: - if level <= 1: - df, md = waterdata.get_daily( - monitoring_location_id=site_ids, - parameter_code="00060", - time=window, - ) - else: - with waterdata.parallel_chunks(level): - df, md = waterdata.get_daily( - monitoring_location_id=site_ids, - parameter_code="00060", - time=window, - ) - - elapsed = time.perf_counter() - t0 - entry["outcome"] = "success" - entry["wall_clock_s"] = round(elapsed, 2) - entry["n_records"] = len(df) - print(f"✓ {elapsed:.1f}s, {len(df):,} records") - - except HangTimeout: - elapsed = time.perf_counter() - t0 - entry["outcome"] = "timeout" - entry["wall_clock_s"] = round(elapsed, 2) - entry["error_type"] = "HangTimeout" - entry["error_msg"] = f"No response within {timeout_s}s" - print(f"✗ HANG after {elapsed:.0f}s") - - except QuotaExhausted as exc: - elapsed = time.perf_counter() - t0 - entry["outcome"] = "quota_exhausted" - entry["wall_clock_s"] = round(elapsed, 2) - entry["error_type"] = "QuotaExhausted" - entry["error_msg"] = f"429, retry_after={exc.retry_after}" - print(f"✗ 429 after {elapsed:.1f}s (retry_after={exc.retry_after})") - - except ChunkInterrupted as exc: - elapsed = time.perf_counter() - t0 - entry["outcome"] = "interrupted" - entry["wall_clock_s"] = round(elapsed, 2) - entry["error_type"] = type(exc).__name__ - entry["error_msg"] = str(exc)[:200] - print(f"✗ {type(exc).__name__} after {elapsed:.1f}s") - - except Exception as exc: - elapsed = time.perf_counter() - t0 - entry["outcome"] = "error" - entry["wall_clock_s"] = round(elapsed, 2) - entry["error_type"] = type(exc).__name__ - entry["error_msg"] = str(exc)[:200] - print(f"✗ {type(exc).__name__} after {elapsed:.1f}s: {str(exc)[:80]}") - - finally: - signal.alarm(0) - signal.signal(signal.SIGALRM, old) - - _append_log(entry) - results.append(entry) - - # Sleep between queries to stay under rate limit - if i < len(STATES) - 1: - print(f" (sleeping {sleep_s}s...)", flush=True) - time.sleep(sleep_s) - - return results - - -def print_summary(): - """Print summary grouped by parallelism level.""" - entries = _load_log() - if not entries: - print("No log entries.") - return - - from collections import defaultdict - - by_level = defaultdict(list) - for e in entries: - by_level[e["parallelism"]].append(e) - - print("\n" + "=" * 90) - print("OVERLOAD THRESHOLD EXPERIMENT — all runs") - print("=" * 90) - print( - f"{'level':>5} | {'trials':>6} | {'ok':>4} | {'429':>4} | " - f"{'hang':>4} | {'err':>4} | {'avg_s':>7} | {'max_s':>7} | " - f"{'avg_records':>11} | {'total_records':>13}" - ) - print("-" * 90) - - for level in sorted(by_level.keys()): - trials = by_level[level] - ok = [t for t in trials if t["outcome"] == "success"] - quota = [t for t in trials if t["outcome"] == "quota_exhausted"] - hangs = [t for t in trials if t["outcome"] == "timeout"] - errs = [t for t in trials if t["outcome"] in ("error", "interrupted")] - - times = [t["wall_clock_s"] for t in ok] - avg_t = f"{sum(times) / len(times):.1f}" if times else "—" - max_t = f"{max(times):.1f}" if times else "—" - avg_r = int(sum(t["n_records"] for t in ok) / len(ok)) if ok else 0 - total_r = sum(t["n_records"] for t in ok) if ok else 0 - - print( - f"{level:>5} | {len(trials):>6} | {len(ok):>4} | {len(quota):>4} | " - f"{len(hangs):>4} | {len(errs):>4} | {avg_t:>7} | {max_t:>7} | " - f"{avg_r:>11,} | {total_r:>13,}" - ) - - print("=" * 90) - - # Detail per level+state - print("\nPer-state breakdown (successful trials only):") - print( - f" {'level':>5} | {'state':<15} | {'time_s':>7} | {'records':>10} | {'window'}" - ) - print(" " + "-" * 70) - for level in sorted(by_level.keys()): - for e in sorted(by_level[level], key=lambda x: x.get("state", "")): - if e["outcome"] == "success": - print( - f" {level:>5} | {e['state']:<15} | " - f"{e['wall_clock_s']:>7.1f} | {e['n_records']:>10,} | " - f"{e['time_window']}" - ) - - -if __name__ == "__main__": - import argparse - - parser = argparse.ArgumentParser() - parser.add_argument("--level", type=int, default=None, help="Single level to test") - parser.add_argument( - "--levels", type=str, default=None, help="Comma-separated levels" - ) - parser.add_argument( - "--sleep", type=int, default=10, help="Seconds between queries (default: 10)" - ) - parser.add_argument( - "--timeout", type=int, default=120, help="Per-query timeout (default: 120)" - ) - parser.add_argument("--summary", action="store_true", help="Print summary only") - args = parser.parse_args() - - if args.summary: - print_summary() - sys.exit(0) - - if args.level is not None: - levels = [args.level] - elif args.levels is not None: - levels = [int(x) for x in args.levels.split(",")] - else: - levels = [1, 2, 4, 8, 16, 32] - - print("Overload threshold experiment") - print(f" Levels to test: {levels}") - print(f" States: {STATES}") - print(f" Sleep between queries: {args.sleep}s") - print(f" Timeout: {args.timeout}s") - print(f" Log: {LOG_FILE}") - print() - - for level in levels: - print(f"{'=' * 60}") - print(f" LEVEL {level}") - print(f"{'=' * 60}") - results = run_level(level, args.sleep, args.timeout) - - # Check if we got throttled — if so, stop escalating - failures = [r for r in results if r["outcome"] != "success"] - if failures: - print(f"\n ⚠ Failures at level {level} — stopping escalation.") - break - - print() - - print_summary() diff --git a/experiments/overload_threshold_log.jsonl b/experiments/overload_threshold_log.jsonl deleted file mode 100644 index 5ff2ad93..00000000 --- a/experiments/overload_threshold_log.jsonl +++ /dev/null @@ -1,20 +0,0 @@ -{"timestamp": "2026-07-16T18:33:36.851681+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2000-01-01/2000-12-31", "parallelism": 8, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 10.88, "n_records": 42728, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:33:57.741196+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2000-01-01/2000-12-31", "parallelism": 8, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 6.81, "n_records": 78674, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:34:14.560514+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2000-01-01/2000-12-31", "parallelism": 8, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 3.03, "n_records": 17706, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:34:27.595726+00:00", "state": "New Hampshire", "n_sites": 1081, "time_window": "2000-01-01/2000-12-31", "parallelism": 8, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 6.98, "n_records": 13629, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:34:44.581715+00:00", "state": "Vermont", "n_sites": 570, "time_window": "2000-01-01/2000-12-31", "parallelism": 8, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 4.9, "n_records": 14274, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:34:56.219533+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2001-01-01/2001-12-31", "parallelism": 16, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 2.95, "n_records": 46368, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:35:09.173482+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2001-01-01/2001-12-31", "parallelism": 16, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 4.09, "n_records": 78329, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:35:23.274345+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2001-01-01/2001-12-31", "parallelism": 16, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 1.65, "n_records": 18242, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:35:34.932930+00:00", "state": "New Hampshire", "n_sites": 1081, "time_window": "2001-01-01/2001-12-31", "parallelism": 16, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 0.98, "n_records": 14514, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:35:45.921675+00:00", "state": "Vermont", "n_sites": 570, "time_window": "2001-01-01/2001-12-31", "parallelism": 16, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 1.38, "n_records": 14829, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:35:54.383526+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2002-01-01/2002-12-31", "parallelism": 32, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 3.74, "n_records": 49490, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:36:08.126379+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2002-01-01/2002-12-31", "parallelism": 32, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 7.15, "n_records": 78595, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:36:25.287311+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2002-01-01/2002-12-31", "parallelism": 32, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 3.55, "n_records": 18090, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:36:38.847054+00:00", "state": "New Hampshire", "n_sites": 1081, "time_window": "2002-01-01/2002-12-31", "parallelism": 32, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 1.82, "n_records": 16646, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:36:50.675320+00:00", "state": "Vermont", "n_sites": 570, "time_window": "2002-01-01/2002-12-31", "parallelism": 32, "timeout_s": 120, "sleep_s": 10, "outcome": "success", "wall_clock_s": 11.11, "n_records": 15065, "error_type": null, "error_msg": null} -{"timestamp": "2026-07-16T18:37:38.342619+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2003-01-01/2007-12-31", "parallelism": 32, "timeout_s": 180, "sleep_s": 10, "outcome": "success", "wall_clock_s": 59.67, "n_records": 242773, "error_type": null, "error_msg": null, "note": "5yr_window"} -{"timestamp": "2026-07-16T18:38:48.021952+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2004-01-01/2008-12-31", "parallelism": 32, "timeout_s": 180, "sleep_s": 10, "outcome": "success", "wall_clock_s": 15.88, "n_records": 419738, "error_type": null, "error_msg": null, "note": "5yr_window"} -{"timestamp": "2026-07-16T18:39:13.910850+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2005-01-01/2009-12-31", "parallelism": 32, "timeout_s": 180, "sleep_s": 10, "outcome": "success", "wall_clock_s": 6.28, "n_records": 103846, "error_type": null, "error_msg": null, "note": "5yr_window"} -{"timestamp": "2026-07-16T18:39:30.191569+00:00", "state": "New Hampshire", "n_sites": 1081, "time_window": "2006-01-01/2010-12-31", "parallelism": 32, "timeout_s": 180, "sleep_s": 10, "outcome": "success", "wall_clock_s": 2.65, "n_records": 81268, "error_type": null, "error_msg": null, "note": "5yr_window"} -{"timestamp": "2026-07-16T18:39:42.849171+00:00", "state": "Vermont", "n_sites": 570, "time_window": "2007-01-01/2011-12-31", "parallelism": 32, "timeout_s": 180, "sleep_s": 10, "outcome": "success", "wall_clock_s": 2.8, "n_records": 87867, "error_type": null, "error_msg": null, "note": "5yr_window"} diff --git a/experiments/parallel_threshold_experiment.py b/experiments/parallel_threshold_experiment.py deleted file mode 100644 index d1828be2..00000000 --- a/experiments/parallel_threshold_experiment.py +++ /dev/null @@ -1,350 +0,0 @@ -""" -Experiment: parallel_chunks threshold sensitivity - -Goal: Find the highest parallelism level that avoids server hang-ups. - -Strategy: -- Rotate queries (different states × different time windows) to defeat server cache. -- Test parallelism levels: 1 (serial baseline), 2, 4, 8, 16, 32. -- Each trial uses a per-request timeout (120s) to detect hangs vs. errors. -- Results are appended to a persistent JSONL log so we can accumulate data - across multiple runs without losing earlier results. - -Usage: - python experiments/parallel_threshold_experiment.py \ - [--levels 1,2,4,8,16,32] [--timeout 120] - -The script runs one trial per (state, time_window, parallelism) combination, -then prints a summary table at the end. Re-run to add more data points. -""" - -from __future__ import annotations - -import argparse -import json -import signal -import sys -import time -from datetime import datetime, timezone -from pathlib import Path - -# Ensure the local package is importable -sys.path.insert(0, str(Path(__file__).resolve().parents[1])) - -from dataretrieval import waterdata -from dataretrieval.ogc.interruptions import ChunkInterrupted - -# ─── Configuration ──────────────────────────────────────────────────────────── - -LOG_FILE = Path(__file__).with_name("parallel_threshold_log.jsonl") - -# States spanning a range of site counts. Larger states (OH, PA) stress the -# server harder; smaller ones (DE, RI) serve as faster controls. -QUERY_STATES = ["Ohio", "Pennsylvania", "Connecticut", "New Hampshire", "Vermont"] - -# Non-overlapping time windows to rotate through, avoiding cache hits. -# Each is ~1 year of daily data — large enough to span multiple pages. -TIME_WINDOWS = [ - "2005-01-01/2005-12-31", - "2008-01-01/2008-12-31", - "2011-01-01/2011-12-31", - "2014-01-01/2014-12-31", - "2017-01-01/2017-12-31", - "2020-01-01/2020-12-31", -] - -DEFAULT_LEVELS = [1, 2, 4, 8, 16, 32] -DEFAULT_TIMEOUT = 120 # seconds — a "hang" if no response by this time - - -# ─── Helpers ────────────────────────────────────────────────────────────────── - - -class TimeoutError(Exception): - pass - - -def _timeout_handler(signum, frame): - raise TimeoutError("Request timed out (server hang suspected)") - - -def _get_sites_for_state(state: str) -> list[str]: - """Fetch stream monitoring location IDs for a state.""" - sites, _ = waterdata.get_monitoring_locations(state=state, site_type_code="ST") - return sites["monitoring_location_id"].tolist() - - -def _run_trial( - site_ids: list[str], - time_window: str, - parallelism: int, - timeout: int, -) -> dict: - """ - Run a single trial: fetch daily discharge for the given sites/time window - at the specified parallelism level. - - Returns a dict with trial metadata and outcome. - """ - result = { - "timestamp": datetime.now(timezone.utc).isoformat(), - "n_sites": len(site_ids), - "time_window": time_window, - "parallelism": parallelism, - "timeout_s": timeout, - "outcome": None, # "success", "timeout", "error", "interrupted" - "wall_clock_s": None, - "n_records": None, - "error_type": None, - "error_msg": None, - } - - # Set alarm-based timeout (Unix only) to detect hangs - old_handler = signal.signal(signal.SIGALRM, _timeout_handler) - signal.alarm(timeout) - - t0 = time.perf_counter() - try: - if parallelism <= 1: - # Serial baseline — no parallel_chunks context - df, md = waterdata.get_daily( - monitoring_location_id=site_ids, - parameter_code="00060", - time=time_window, - ) - else: - with waterdata.parallel_chunks(parallelism): - df, md = waterdata.get_daily( - monitoring_location_id=site_ids, - parameter_code="00060", - time=time_window, - ) - - elapsed = time.perf_counter() - t0 - result["outcome"] = "success" - result["wall_clock_s"] = round(elapsed, 2) - result["n_records"] = len(df) - - except TimeoutError: - elapsed = time.perf_counter() - t0 - result["outcome"] = "timeout" - result["wall_clock_s"] = round(elapsed, 2) - result["error_type"] = "TimeoutError" - result["error_msg"] = "Server hang — no response within timeout" - - except ChunkInterrupted as exc: - elapsed = time.perf_counter() - t0 - result["outcome"] = "interrupted" - result["wall_clock_s"] = round(elapsed, 2) - result["error_type"] = type(exc).__name__ - result["error_msg"] = str(exc)[:200] - result["n_records"] = len(exc.call.partial_frame) if exc.call else None - - except Exception as exc: - elapsed = time.perf_counter() - t0 - result["outcome"] = "error" - result["wall_clock_s"] = round(elapsed, 2) - result["error_type"] = type(exc).__name__ - result["error_msg"] = str(exc)[:200] - - finally: - signal.alarm(0) # cancel alarm - signal.signal(signal.SIGALRM, old_handler) - - return result - - -def _append_log(entry: dict) -> None: - """Append a trial result to the persistent JSONL log.""" - with open(LOG_FILE, "a") as f: - f.write(json.dumps(entry) + "\n") - - -def _load_log() -> list[dict]: - """Load all past trial results.""" - if not LOG_FILE.exists(): - return [] - entries = [] - with open(LOG_FILE) as f: - for line in f: - line = line.strip() - if line: - entries.append(json.loads(line)) - return entries - - -def _print_summary(entries: list[dict]) -> None: - """Print a summary table of results grouped by parallelism level.""" - from collections import defaultdict - - by_level = defaultdict(list) - for e in entries: - by_level[e["parallelism"]].append(e) - - print("\n" + "=" * 80) - print("SUMMARY — all trials (current + historical)") - print("=" * 80) - print( - f"{'level':>6} | {'trials':>6} | {'success':>7} | {'timeout':>7} | " - f"{'error':>5} | {'interrupted':>11} | {'avg_time_s':>10} | {'med_time_s':>10}" - ) - print("-" * 80) - - for level in sorted(by_level.keys()): - trials = by_level[level] - n = len(trials) - successes = [t for t in trials if t["outcome"] == "success"] - timeouts = [t for t in trials if t["outcome"] == "timeout"] - errors = [t for t in trials if t["outcome"] == "error"] - interrupted = [t for t in trials if t["outcome"] == "interrupted"] - - times = sorted( - t["wall_clock_s"] for t in successes if t["wall_clock_s"] is not None - ) - avg_t = f"{sum(times) / len(times):.1f}" if times else "—" - med_t = f"{times[len(times) // 2]:.1f}" if times else "—" - - print( - f"{level:>6} | {n:>6} | {len(successes):>7} | {len(timeouts):>7} | " - f"{len(errors):>5} | {len(interrupted):>11} | {avg_t:>10} | {med_t:>10}" - ) - - print("=" * 80) - - # Show any failures in detail - failures = [e for e in entries if e["outcome"] not in ("success",)] - if failures: - print(f"\nFailed/timeout trials ({len(failures)}):") - for f_entry in failures[-10:]: # last 10 - print( - f" [{f_entry['timestamp'][:19]}] level={f_entry['parallelism']:>2}, " - f"outcome={f_entry['outcome']}, " - f"time={f_entry['wall_clock_s']}s, " - f"err={f_entry.get('error_type', '?')}: " - f"{(f_entry.get('error_msg') or '')[:80]}" - ) - - -# ─── Main ───────────────────────────────────────────────────────────────────── - - -def main(): - parser = argparse.ArgumentParser(description="Parallel chunks threshold experiment") - parser.add_argument( - "--levels", - type=str, - default=",".join(str(x) for x in DEFAULT_LEVELS), - help="Comma-separated parallelism levels to test (default: 1,2,4,8,16,32)", - ) - parser.add_argument( - "--timeout", - type=int, - default=DEFAULT_TIMEOUT, - help="Per-trial timeout in seconds (default: 120)", - ) - parser.add_argument( - "--state", - type=str, - default=None, - help="Override: use only this state (default: rotate through all)", - ) - parser.add_argument( - "--window", - type=str, - default=None, - help="Override: use only this time window (default: rotate)", - ) - parser.add_argument( - "--summary-only", - action="store_true", - help="Just print summary of existing log, don't run new trials", - ) - args = parser.parse_args() - - levels = [int(x) for x in args.levels.split(",")] - - if args.summary_only: - entries = _load_log() - if entries: - _print_summary(entries) - else: - print("No log entries found.") - return - - # Pick query rotation - states = [args.state] if args.state else QUERY_STATES - windows = [args.window] if args.window else TIME_WINDOWS - - print("Experiment: parallel_chunks threshold sensitivity") - print(f" Levels: {levels}") - print(f" Timeout: {args.timeout}s") - print(f" States: {states}") - print(f" Log: {LOG_FILE}") - print() - - # Preload site IDs for the selected states - site_cache: dict[str, list[str]] = {} - for state in states: - print(f" Fetching sites for {state}...", end=" ", flush=True) - site_cache[state] = _get_sites_for_state(state) - print(f"{len(site_cache[state])} sites") - - # Rotate: each level gets a different state (round-robin across states), - # and each (state, level) combo gets the next unused time window for that - # state. This ensures no two trials share a (state, window) pair, defeating - # the server-side cache. - # - # To avoid repeating combos from prior runs, count how many times each - # state has already appeared in the log and advance its cursor past those. - past_entries = _load_log() - window_cursor: dict[str, int] = {s: 0 for s in states} - for entry in past_entries: - s = entry.get("state") - if s in window_cursor: - window_cursor[s] += 1 - - total_trials = len(levels) - print(f"\nRunning {total_trials} trials...\n") - - new_entries = [] - for i, level in enumerate(levels): - # Rotate state by trial index - state = states[i % len(states)] - # Pick the next unused window for this state - w_idx = window_cursor[state] % len(windows) - window = windows[w_idx] - window_cursor[state] += 1 - sites = site_cache[state] - - print( - f" [{i + 1}/{total_trials}] level={level:>2}, state={state:<15}, " - f"window={window}...", - end=" ", - flush=True, - ) - - result = _run_trial(sites, window, level, args.timeout) - result["state"] = state - _append_log(result) - new_entries.append(result) - - # Print inline result - if result["outcome"] == "success": - print(f"✓ {result['wall_clock_s']}s, {result['n_records']} records") - else: - print( - f"✗ {result['outcome']} after {result['wall_clock_s']}s " - f"({result.get('error_type', '?')})" - ) - - # Brief pause between trials to be polite to the server - if i < total_trials - 1: - time.sleep(2) - - # Print full summary - all_entries = _load_log() - _print_summary(all_entries) - - -if __name__ == "__main__": - main() diff --git a/experiments/parallel_threshold_log.jsonl b/experiments/parallel_threshold_log.jsonl deleted file mode 100644 index 74bf54bd..00000000 --- a/experiments/parallel_threshold_log.jsonl +++ /dev/null @@ -1,81 +0,0 @@ -{"timestamp": "2026-07-16T16:03:08.198046+00:00", "n_sites": 246, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 11.09, "n_records": 5747, "error_type": null, "error_msg": null, "state": "Delaware"} -{"timestamp": "2026-07-16T16:03:21.294922+00:00", "n_sites": 246, "time_window": "2008-01-01/2008-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 5.87, "n_records": 6588, "error_type": null, "error_msg": null, "state": "Delaware"} -{"timestamp": "2026-07-16T16:03:29.175792+00:00", "n_sites": 246, "time_window": "2011-01-01/2011-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 7.28, "n_records": 5475, "error_type": null, "error_msg": null, "state": "Delaware"} -{"timestamp": "2026-07-16T16:03:38.466959+00:00", "n_sites": 246, "time_window": "2014-01-01/2014-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.59, "n_records": 6181, "error_type": null, "error_msg": null, "state": "Delaware"} -{"timestamp": "2026-07-16T16:03:43.067729+00:00", "n_sites": 246, "time_window": "2017-01-01/2017-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.3, "n_records": 6524, "error_type": null, "error_msg": null, "state": "Delaware"} -{"timestamp": "2026-07-16T16:03:47.370355+00:00", "n_sites": 246, "time_window": "2020-01-01/2020-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.32, "n_records": 6224, "error_type": null, "error_msg": null, "state": "Delaware"} -{"timestamp": "2026-07-16T16:03:58.613903+00:00", "n_sites": 1081, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 7.62, "n_records": 14143, "error_type": null, "error_msg": null, "state": "New Hampshire"} -{"timestamp": "2026-07-16T16:04:08.238261+00:00", "n_sites": 1081, "time_window": "2008-01-01/2008-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 3.25, "n_records": 15084, "error_type": null, "error_msg": null, "state": "New Hampshire"} -{"timestamp": "2026-07-16T16:04:13.498869+00:00", "n_sites": 1081, "time_window": "2011-01-01/2011-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 5.17, "n_records": 18265, "error_type": null, "error_msg": null, "state": "New Hampshire"} -{"timestamp": "2026-07-16T16:04:20.680625+00:00", "n_sites": 1081, "time_window": "2014-01-01/2014-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 3.32, "n_records": 17647, "error_type": null, "error_msg": null, "state": "New Hampshire"} -{"timestamp": "2026-07-16T16:04:26.010006+00:00", "n_sites": 1081, "time_window": "2017-01-01/2017-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 3.13, "n_records": 19072, "error_type": null, "error_msg": null, "state": "New Hampshire"} -{"timestamp": "2026-07-16T16:04:31.150457+00:00", "n_sites": 1081, "time_window": "2020-01-01/2020-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.82, "n_records": 19764, "error_type": null, "error_msg": null, "state": "New Hampshire"} -{"timestamp": "2026-07-16T16:04:39.410640+00:00", "n_sites": 1184, "time_window": "2005-01-01/2005-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.42, "n_records": 18410, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:04:43.838766+00:00", "n_sites": 1184, "time_window": "2008-01-01/2008-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.5, "n_records": 21599, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:04:48.347019+00:00", "n_sites": 1184, "time_window": "2011-01-01/2011-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.08, "n_records": 24111, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:04:52.435878+00:00", "n_sites": 1184, "time_window": "2014-01-01/2014-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.04, "n_records": 23083, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:04:56.487218+00:00", "n_sites": 1184, "time_window": "2017-01-01/2017-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.99, "n_records": 23012, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:05:17.596051+00:00", "n_sites": 2889, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 7.6, "n_records": 47363, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T16:05:27.212362+00:00", "n_sites": 2889, "time_window": "2008-01-01/2008-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 8.62, "n_records": 52338, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T16:05:37.853445+00:00", "n_sites": 2889, "time_window": "2011-01-01/2011-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 7.44, "n_records": 57044, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T16:05:47.316958+00:00", "n_sites": 2889, "time_window": "2014-01-01/2014-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 8.6, "n_records": 65308, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T16:05:57.944067+00:00", "n_sites": 2889, "time_window": "2017-01-01/2017-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 9.58, "n_records": 75628, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T16:06:15.202809+00:00", "n_sites": 5676, "time_window": "2005-01-01/2005-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 6.4, "n_records": 81500, "error_type": null, "error_msg": null, "state": "Pennsylvania"} -{"timestamp": "2026-07-16T16:06:23.632642+00:00", "n_sites": 5676, "time_window": "2008-01-01/2008-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 14.09, "n_records": 88151, "error_type": null, "error_msg": null, "state": "Pennsylvania"} -{"timestamp": "2026-07-16T16:06:39.762735+00:00", "n_sites": 5676, "time_window": "2011-01-01/2011-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 4.26, "n_records": 92276, "error_type": null, "error_msg": null, "state": "Pennsylvania"} -{"timestamp": "2026-07-16T16:06:46.055454+00:00", "n_sites": 5676, "time_window": "2014-01-01/2014-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 7.93, "n_records": 99731, "error_type": null, "error_msg": null, "state": "Pennsylvania"} -{"timestamp": "2026-07-16T16:07:56.941675+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2003-01-01/2007-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 39.96, "n_records": 410821, "error_type": null, "error_msg": null, "note": "5yr_window_rapid_fire"} -{"timestamp": "2026-07-16T16:08:23.701431+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2008-01-01/2012-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 26.76, "n_records": 457336, "error_type": null, "error_msg": null, "note": "5yr_window_rapid_fire"} -{"timestamp": "2026-07-16T16:08:45.738312+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2013-01-01/2017-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 22.04, "n_records": 500442, "error_type": null, "error_msg": null, "note": "5yr_window_rapid_fire"} -{"timestamp": "2026-07-16T16:17:44.822601+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2005-01-01/2014-12-31", "parallelism": 32, "timeout_s": 180, "outcome": "success", "wall_clock_s": 32.61, "n_records": 556354, "error_type": null, "error_msg": null, "note": "10yr_window_stress"} -{"timestamp": "2026-07-16T16:17:51.889708+00:00", "n_sites": 1184, "time_window": "2005-01-01/2005-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.78, "n_records": 18410, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:17:55.673983+00:00", "n_sites": 1184, "time_window": "2008-01-01/2008-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.79, "n_records": 21599, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:17:58.471864+00:00", "n_sites": 1184, "time_window": "2011-01-01/2011-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.99, "n_records": 24111, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:18:02.472114+00:00", "n_sites": 1184, "time_window": "2014-01-01/2014-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 3.26, "n_records": 23083, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:18:07.743656+00:00", "n_sites": 1184, "time_window": "2017-01-01/2017-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.89, "n_records": 23012, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:18:12.641117+00:00", "n_sites": 1184, "time_window": "2020-01-01/2020-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.8, "n_records": 24575, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T16:18:54.508351+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2020-03-01/2020-03-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 23.01, "n_records": 209727, "error_type": null, "error_msg": null, "note": "get_continuous"} -{"timestamp": "2026-07-16T16:19:05.106056+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2020-03-01/2020-03-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 9.6, "n_records": 209727, "error_type": null, "error_msg": null, "note": "get_continuous"} -{"timestamp": "2026-07-16T16:19:26.233466+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2020-03-01/2020-03-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 20.12, "n_records": 209727, "error_type": null, "error_msg": null, "note": "get_continuous"} -{"timestamp": "2026-07-16T16:20:27.946708+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2019-06-01/2019-06-30", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 46.0, "n_records": 776037, "error_type": null, "error_msg": null, "note": "get_continuous"} -{"timestamp": "2026-07-16T16:21:14.615471+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2018-07-01/2018-07-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 44.66, "n_records": 766944, "error_type": null, "error_msg": null, "note": "get_continuous"} -{"timestamp": "2026-07-16T16:21:55.409587+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2017-08-01/2017-08-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 38.79, "n_records": 767713, "error_type": null, "error_msg": null, "note": "get_continuous"} -{"timestamp": "2026-07-16T16:37:09.647936+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2019-01-01/2019-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 13.43, "n_records": 79808, "error_type": null, "error_msg": null, "note": "rapid_fire_no_pause"} -{"timestamp": "2026-07-16T16:37:15.882284+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2020-01-01/2020-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 6.23, "n_records": 106835, "error_type": null, "error_msg": null, "note": "rapid_fire_no_pause"} -{"timestamp": "2026-07-16T16:37:22.167812+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2021-01-01/2021-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 6.28, "n_records": 78369, "error_type": null, "error_msg": null, "note": "rapid_fire_no_pause"} -{"timestamp": "2026-07-16T16:37:24.593376+00:00", "state": "Connecticut", "n_sites": 1184, "time_window": "2019-01-01/2019-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.43, "n_records": 22995, "error_type": null, "error_msg": null, "note": "rapid_fire_no_pause"} -{"timestamp": "2026-07-16T16:37:30.638632+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2022-01-01/2022-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 6.04, "n_records": 108454, "error_type": null, "error_msg": null, "note": "rapid_fire_no_pause"} -{"timestamp": "2026-07-16T16:38:30.551883+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2022-01-01/2022-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 11.6, "n_records": 80607, "error_type": null, "error_msg": null, "note": "rapid_fire_unique_combos"} -{"timestamp": "2026-07-16T16:38:42.305398+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2015-01-01/2015-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 11.75, "n_records": 102149, "error_type": null, "error_msg": null, "note": "rapid_fire_unique_combos"} -{"timestamp": "2026-07-16T16:38:42.671570+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2023-01-01/2023-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "error", "wall_clock_s": 0.37, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/32 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_unique_combos"} -{"timestamp": "2026-07-16T16:38:43.115655+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2016-01-01/2016-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "error", "wall_clock_s": 0.44, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/32 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_unique_combos"} -{"timestamp": "2026-07-16T16:38:43.473863+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2020-01-01/2020-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "error", "wall_clock_s": 0.36, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/32 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_unique_combos"} -{"timestamp": "2026-07-16T17:02:51.026246+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2015-01-01/2015-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 26.3, "n_records": 71579, "error_type": null, "error_msg": null, "note": "rapid_fire_n8_unique"} -{"timestamp": "2026-07-16T17:02:56.344273+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2019-01-01/2019-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 5.32, "n_records": 101172, "error_type": null, "error_msg": null, "note": "rapid_fire_n8_unique"} -{"timestamp": "2026-07-16T17:02:56.678456+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2016-01-01/2016-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "interrupted", "wall_clock_s": 0.33, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/12 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_n8_unique"} -{"timestamp": "2026-07-16T17:02:57.076010+00:00", "state": "Pennsylvania", "n_sites": 5676, "time_window": "2021-01-01/2021-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "interrupted", "wall_clock_s": 0.4, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/18 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_n8_unique"} -{"timestamp": "2026-07-16T17:02:57.415448+00:00", "state": "Ohio", "n_sites": 2889, "time_window": "2018-01-01/2018-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "interrupted", "wall_clock_s": 0.34, "n_records": null, "error_type": "QuotaExhausted", "error_msg": "HTTP 429 after 0/12 sub-requests; catch QuotaExhausted (or ChunkInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many re", "note": "rapid_fire_n8_unique"} -{"timestamp": "2026-07-16T18:26:58.642855+00:00", "n_sites": 2889, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 27.38, "n_records": 47363, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T18:27:28.089128+00:00", "n_sites": 5676, "time_window": "2005-01-01/2005-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 8.27, "n_records": 81500, "error_type": null, "error_msg": null, "state": "Pennsylvania"} -{"timestamp": "2026-07-16T18:27:38.391844+00:00", "n_sites": 1184, "time_window": "2005-01-01/2005-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 4.62, "n_records": 18410, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T18:27:45.019968+00:00", "n_sites": 1081, "time_window": "2005-01-01/2005-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.17, "n_records": 14143, "error_type": null, "error_msg": null, "state": "New Hampshire"} -{"timestamp": "2026-07-16T18:27:49.200102+00:00", "n_sites": 570, "time_window": "2005-01-01/2005-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.1, "n_records": 16453, "error_type": null, "error_msg": null, "state": "Vermont"} -{"timestamp": "2026-07-16T18:27:53.311784+00:00", "n_sites": 2889, "time_window": "2008-01-01/2008-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 4.85, "n_records": 52338, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T18:28:15.357707+00:00", "n_sites": 2889, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.0, "n_records": 47363, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T18:28:18.372315+00:00", "n_sites": 5676, "time_window": "2005-01-01/2005-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.29, "n_records": 81500, "error_type": null, "error_msg": null, "state": "Pennsylvania"} -{"timestamp": "2026-07-16T18:28:21.687470+00:00", "n_sites": 1184, "time_window": "2005-01-01/2005-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.63, "n_records": 18410, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T18:28:24.323955+00:00", "n_sites": 1081, "time_window": "2005-01-01/2005-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.52, "n_records": 14143, "error_type": null, "error_msg": null, "state": "New Hampshire"} -{"timestamp": "2026-07-16T18:28:26.853950+00:00", "n_sites": 570, "time_window": "2005-01-01/2005-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.52, "n_records": 16453, "error_type": null, "error_msg": null, "state": "Vermont"} -{"timestamp": "2026-07-16T18:28:29.377015+00:00", "n_sites": 2889, "time_window": "2008-01-01/2008-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.91, "n_records": 52338, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T18:28:47.115674+00:00", "n_sites": 2889, "time_window": "2005-01-01/2005-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.06, "n_records": 47363, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T18:28:50.189062+00:00", "n_sites": 5676, "time_window": "2005-01-01/2005-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.34, "n_records": 81500, "error_type": null, "error_msg": null, "state": "Pennsylvania"} -{"timestamp": "2026-07-16T18:28:53.559101+00:00", "n_sites": 1184, "time_window": "2020-01-01/2020-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 4.11, "n_records": 24575, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T18:28:59.674982+00:00", "n_sites": 1081, "time_window": "2011-01-01/2011-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 2.96, "n_records": 18265, "error_type": null, "error_msg": null, "state": "New Hampshire"} -{"timestamp": "2026-07-16T18:29:04.645868+00:00", "n_sites": 570, "time_window": "2011-01-01/2011-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 1.63, "n_records": 19488, "error_type": null, "error_msg": null, "state": "Vermont"} -{"timestamp": "2026-07-16T18:29:08.279422+00:00", "n_sites": 2889, "time_window": "2008-01-01/2008-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.89, "n_records": 52338, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T18:29:26.662442+00:00", "n_sites": 2889, "time_window": "2011-01-01/2011-12-31", "parallelism": 1, "timeout_s": 120, "outcome": "success", "wall_clock_s": 27.56, "n_records": 57044, "error_type": null, "error_msg": null, "state": "Ohio"} -{"timestamp": "2026-07-16T18:29:56.240019+00:00", "n_sites": 5676, "time_window": "2008-01-01/2008-12-31", "parallelism": 2, "timeout_s": 120, "outcome": "success", "wall_clock_s": 15.37, "n_records": 88151, "error_type": null, "error_msg": null, "state": "Pennsylvania"} -{"timestamp": "2026-07-16T18:30:13.650984+00:00", "n_sites": 1184, "time_window": "2005-01-01/2005-12-31", "parallelism": 4, "timeout_s": 120, "outcome": "success", "wall_clock_s": 0.64, "n_records": 18410, "error_type": null, "error_msg": null, "state": "Connecticut"} -{"timestamp": "2026-07-16T18:30:16.300995+00:00", "n_sites": 1081, "time_window": "2014-01-01/2014-12-31", "parallelism": 8, "timeout_s": 120, "outcome": "success", "wall_clock_s": 8.89, "n_records": 17647, "error_type": null, "error_msg": null, "state": "New Hampshire"} -{"timestamp": "2026-07-16T18:30:27.195785+00:00", "n_sites": 570, "time_window": "2014-01-01/2014-12-31", "parallelism": 16, "timeout_s": 120, "outcome": "success", "wall_clock_s": 17.83, "n_records": 16339, "error_type": null, "error_msg": null, "state": "Vermont"} -{"timestamp": "2026-07-16T18:30:47.035453+00:00", "n_sites": 2889, "time_window": "2014-01-01/2014-12-31", "parallelism": 32, "timeout_s": 120, "outcome": "success", "wall_clock_s": 11.38, "n_records": 65308, "error_type": null, "error_msg": null, "state": "Ohio"} From 8ae1b9a4249aaca272bf1a60e6f9880ef53381b1 Mon Sep 17 00:00:00 2001 From: thodson-usgs Date: Fri, 17 Jul 2026 13:15:48 -0500 Subject: [PATCH 13/13] docs: omit parallel_chunks NEWS entry; refresh README benchmark with verified numbers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove the parallel_chunks NEWS.md entry (per maintainer request). - Re-ran the README benchmark against the live API with a hardened, cache-cold methodology: a fixed 271-site subset of Ohio stream gages, fixed page size (limit=250), each n on its own distinct 1-year window so no run is served from the server's data-window cache (off on 2003/2005, the fanned runs on 2007/2009/2011 — no shared cache keys). The multi-fold speedup is real but the earlier figures were optimistic: off 9.3s -> n=8 ~2.0s (~4.5x) -> n=32 1.2s (~8x), vs the previously stated ~6x/~12x. Page counts (~30/32/54) match. Noted that the multiplier scales with page count. The old note said "271 Ohio discharge sites", but state="Ohio",site_type_code="ST" now returns 2889, so the benchmark is pinned to a reproducible 271-site subset. Signed-off-by: thodson-usgs Co-Authored-By: Claude Opus 4.8 (1M context) --- NEWS.md | 2 -- README.md | 28 +++++++++++++++------------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/NEWS.md b/NEWS.md index aee2ef07..a86a0314 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,5 +1,3 @@ -**07/01/2026:** Added `waterdata.parallel_chunks(n)` — a context manager to control how finely the OGC `waterdata` getters split multi-value requests. By default the getters chunk only as much as the server's ~8 KB URL limit forces (the fewest sub-requests). But because every sub-request paginates, splitting a large result further costs little or no extra quota when each sub-request still spans many pages, so a caller who *knows* their pull is large can opt into a finer split for smoother progress, more even concurrency, and a smaller unit of retry/resume: `with waterdata.parallel_chunks(32): df, md = waterdata.get_daily(monitoring_location_id=many_sites)`. `n` is a positive integer (e.g. `2`, `8`, `32`) — the number of sub-requests to fan the call out into; a non-integer or non-positive value raises `ValueError` at the `with`. It caps the *total* sub-request count across every multi-value argument combined (not per argument), bounded below by what the byte limit already forces and above by how many values there are to split, so several multi-value arguments can't multiply past it and `n=1` asks for no extra fan-out. Each sub-request costs a request against your hourly rate limit, and because how many run *at once* is capped separately by `API_USGS_CONCURRENT` (default 32) an `n` beyond that adds quota without adding parallelism, so the useful range is roughly `2` up to `API_USGS_CONCURRENT`. It is deliberately a scoped `with` block rather than an automatic behavior or an environment variable, since the library can't tell in advance whether a query is large — a short-window query might fit in a single page, where extra chunks would only burn quota. Also exported at the top level as `dataretrieval.parallel_chunks`. - **06/23/2026:** **Breaking change (1.2.0):** the minimum supported Python is now **3.10** (`requires-python = ">=3.10"`). 3.9 support was already effectively broken — the `waterdata` module's dependencies (`anyio`, the test stack) require 3.10+, and the `waterdata` test modules already skipped on <3.10. `anyio` is now declared as a direct dependency (it is imported directly by `waterdata`), and the CI/ruff/mypy targets move to 3.10. Also fully removed the deprecated `variable_info` metadata property: the `NWIS_Metadata` override only warned and returned `None` (it relied on the defunct `get_pmcodes`), and the `BaseMetadata` abstract is gone too since nothing implemented it — accessing `.variable_info` now raises `AttributeError`. `site_info` is unaffected. **06/23/2026:** **Breaking change (1.2.0):** removed the `nadp` module and the deprecated `samples` module ahead of the 1.2.0 release. `nadp` was deprecated on 05/01/2026 — NADP is not a USGS data source, so retrieve NADP data directly from https://nadp.slh.wisc.edu/. The `samples.get_usgs_samples` shim (a deprecated forward to the modern getter) is gone; use `waterdata.get_samples()` instead. `import dataretrieval.nadp` / `import dataretrieval.samples` now raise `ModuleNotFoundError`. diff --git a/README.md b/README.md index 73aa273e..9a9a7966 100644 --- a/README.md +++ b/README.md @@ -139,21 +139,23 @@ your hourly [rate limit](https://api.waterdata.usgs.gov/signup/); since how many run *at once* is capped separately by `API_USGS_CONCURRENT` (default 32), the useful range is roughly `2` up to that value. -Benchmark — 271 Ohio discharge sites (`get_daily`, `parameter_code="00060"`), -cold cache, each `n` run against its own time window so results are not -cache-served, with the page size fixed so every run fetches roughly the same -number of pages (isolating the effect of parallelism): - -| `n` | parallelism | pages | wall-clock | speedup | -| ---- | ----------- | ----- | ------------------------- | ------- | -| off | 1 | 32 | 23.9 s / 27.4 s (2 runs) | 1× | -| `8` | 8 | 37 | 4.1 s / 4.3 s | ~6× | -| `32` | 32 | 44 | 2.0 s | ~12× | +Benchmark — a fixed 271-site subset of Ohio stream gages +(`get_daily`, `parameter_code="00060"`), with a small fixed page size +(`limit=250`) so every run fetches roughly the same number of pages (isolating +the effect of parallelism). Each `n` was run against its own cold 1-year time +window so no run is served from the server's data-window cache: + +| `n` | parallelism | pages | wall-clock | speedup | +| ---- | ----------- | ----- | ----------------------- | ------- | +| off | 1 | ~30 | 9.5 s / 9.1 s (2 runs) | 1× | +| `8` | 8 | ~32 | 2.2 s / 1.9 s | ~4.5× | +| `32` | 32 | 54 | 1.2 s | ~8× | The gain comes from overlapping each sub-request's per-page latency and -server-side work — a genuinely large pull at the default page size shows a -similar multi-fold speedup. The extra sub-requests each cost quota, so reserve a -large `n` for pulls you know are large. +server-side work, so the exact multiplier scales with how many pages the pull +spans — a larger pull (more pages) has more parallelism to exploit. The extra +sub-requests each cost quota, so reserve a large `n` for pulls you know are +large. Visit the [API Reference](https://doi-usgs.github.io/dataretrieval-python/reference/waterdata.html)