Companion package: TxGIO address-point index for lat/long resolution (consumed via [geo] extra)

[jreakin]

Summary

Proposal for a new standalone package that acquires the TxGIO StratMap Address
Points dataset (every addressable structure in Texas, with lat/long) and resolves an
address — full string or components — to a coordinate.

It should ship as its own package, with RyanData-Address-Utils taking it as an
optional dependency (e.g. ryandata-address-utils[geo]). The split matters because
the two halves have genuinely different shapes:

	RyanData-Address-Utils	proposed package
Job	parse/normalize an address string	resolve a normalized address → lat/long
State	stateless, pure	~450 MB managed on-disk cache
Deps	pydantic, usaddress	+ pyarrow/duckdb, geo stack
Cadence	library releases	annual data vintages

Forcing a 450 MB data cache and a DB engine into the parser would penalize every
consumer who only wants parse(). Keeping it separate lets the parser stay light and
lets the geo package version on the data cadence rather than the code cadence.

I've already built and run the acquisition end to end (in voterfile-audit-pipeline,
for an address-quality audit) — all 254 counties, 12,142,647 address points. Everything
below is verified against the real endpoints, not inferred.

---

1. Where the data actually lives

https://data.geographic.texas.gov is the TxGIO DataHub. Do not try to scrape it — it's
a JS SPA and the HTML has no listings. It's backed by a clean public REST API:

https://api.tnris.org/api/v1/collections_catalog   # dataset catalog (2,219 records)
https://api.tnris.org/api/v1/resources?collection_id=<uuid>   # download URLs

I found these by pulling the SPA bundle (/static/index.<hash>.js) and extracting the
API base + route templates from the minified source.

Resolve the newest vintage dynamically — don't pin the UUID

Address Points has shipped seven vintages, and the collection UUID changes each time:

acquisition_date	collection_id
2019-08-31	117cf9e1-3b1e-48f2-97a3-47020d871035
2021-07-07	94502179-9389-4bfa-b753-5e43f6d477bf
2022-02-01	842bafc7-cabf-483b-b5fb-ab21638b5c72
2023-02-01	578ac3ee-0d9f-4bd3-bdeb-509af9b5b646
2024-02-01	6d9c4a2e-b5bb-49b3-9ceb-0727f4711c5b
2025-04-01	f9bfd25d-cf2c-4e2f-9b44-43d4bb915390
2026-03-18	be6d5f05-2f5f-4bb6-9717-1165de96e6cb ← current

Filter the catalog on name == "Address Points", sort by acquisition_date desc, take
the newest. New vintages then land with no code change — and pin the resolved UUID in a
manifest so an audit stays reproducible.

/resources returns exactly 255 entries

254 county ZIPs (area_type: "county") + 1 statewide rollup (area_type: "state",
331 MB). Skip the statewide file — it duplicates the counties. County-only is 823 MB.

URL pattern is stable and FIPS-keyed:

https://data.geographic.texas.gov/{collection_id}/resources/stratmap-2026-address-points_{fips5}_ap.zip

⚠️ The CloudFront WAF gotcha (this will cost you an hour)

The download host 403s any User-Agent that doesn't start with Mozilla/5.0:

User-Agent	Result
curl default	403
python-httpx/0.27	403
Mozilla/5.0 (bare)	403
Mozilla/5.0 (compatible; my-tool/0.3; +https://github.com/...)	200 ✅

So you do not need to impersonate a browser — just prefix an honest UA with
Mozilla/5.0 (compatible; ...). Note api.tnris.org has no such restriction; only
the CDN download host does. Range requests work (206), so downloads are resumable.

License

CC0-1.0 (https://spdx.org/licenses/CC0-1.0.html) — public domain dedication,
source "Various 911 Districts". No redistribution restriction, so caching and even
republishing a derived artifact is fine.

---

2. What's inside each ZIP

Both a FileGDB and a Shapefile. EPSG:4326 already — no reprojection needed. Read the
shapefile in place via GDAL's /vsizip/ handler; never extract:

inner = next(n for n in zipfile.ZipFile(zp).namelist() if n.lower().endswith(".shp"))
gdf = geopandas.read_file(f"/vsizip/{zp}/{inner}")   # ~0.3s for a small county

Schema is NENA/FGDC — pre-parsed components, which lines up almost 1:1 with the
fields RyanData-Address-Utils already produces:

Add_Number  AddNum_Suf  St_PreMod  St_PreDir  St_PreTyp  St_PreSep
St_Name     St_PosTyp   St_PosDir  St_PosMod  Full_Addr
Building    Floor       Unit       Post_Comm  Post_Code
State       County      FIPS       Source     geometry(Point)

That's the real win: no need to parse the reference side. St_PreDir/St_Name/
St_PosTyp map onto usaddress's StreetNamePreDirectional/StreetName/
StreetNamePostType, so matching happens component-to-component instead of
string-to-string.

---

3. Verified results from the full pull

Metric	Value
Counties	254 / 254, 0 failures
Address points	12,142,647
Raw ZIPs	823 MB
Parquet (zstd)	434 MB
Coordinates outside TX bounds	0
Largest county	Harris (48201) — 1,453,510 points

Field completeness — read this before designing the matcher:

Field	Missing	Note
house number	0.00% (57 rows)	essentially perfect
street name	0.02%	essentially perfect
ZIP	22.11%	⚠️ see below
city (Post_Comm)	9.59%
unit	93.7% absent	only 766k rows carry one
unique num+street+zip key	91.54%	~8% legitimate collisions (units/duplexes)

The 22% missing-ZIP rate is the single most important design constraint. A naive
housenum + street + zip5 join silently drops a fifth of the reference set — and it
won't fail loudly, it'll just look like a poor match rate. Rural counties are worst hit.
County FIPS is always present (it's in the filename), so county is the reliable
partition key, not ZIP.

---

4. Proposed package

Name — something that admits other states later, since ~20 states publish the same
NENA schema: abstract-address-points, or openaddress-index. I'd avoid txgio-*;
the TX source belongs in a provider module, not the package name.

src/<pkg>/
  providers/txgio.py      # catalog resolve, resource list, download, WAF-safe UA
  store/                  # parquet writer, duckdb builder, cache dir mgmt
  match/                  # tiered resolver
  cli.py                  # `<pkg> fetch --state TX`, `<pkg> resolve "<address>"`

Core API sketch:

idx = AddressIndex.load()                      # uses managed cache dir
idx.resolve("15671 FM 652, Van Horn TX 79855") # -> Match(lat, lon, tier, confidence)
idx.resolve_components(number="15671", street="FM 652", zip5="79855")
idx.resolve_batch(df, on=["addr","city","zip"])  # vectorized, the important one

Every result should carry tier and confidence, never a bare lat/long — an audit
needs to distinguish "exact rooftop" from "guessed from street". That's the whole point
for the voter-file use case.

Tiered matching (accounts for the missing-ZIP problem)

Tier	Key	Confidence
1	county + number + street + zip5	exact
2	county + number + street + city	high
3	county + number + street	high (ZIP absent on ref side)
4	county + number + fuzzy(street) via rapidfuzz	medium
5	county + street → interpolate/centroid	low
—	no match	report, don't fabricate

Tier 3 is what recovers most of the 22%. Unit-level matching should be opt-in — only
6.3% of points carry a unit, so requiring one mostly produces false negatives.

---

5. Storage — recommendation

Options considered, against the real 12.1M rows:

Option	Size	Batch join	Point lookup	Verdict
Raw shapefiles	823 MB	terrible	terrible	staging only
Parquet, partitioned by county	434 MB	excellent	good	✅ storage format
DuckDB	~500 MB	excellent	excellent (indexed)	✅ query engine
SQLite + R*Tree/FTS5	~2–3 GB	poor	good	too fat, slow bulk load
PostGIS	n/a	excellent	excellent	too heavy for a library

Recommendation: Parquet as the canonical artifact, DuckDB as the query engine.

Concretely:

1. Parquet is what you cache and ship — one file per county, zstd, hive-style
   county_fips=48201/. 434 MB total. Portable, versionable, readable by polars/pandas/
   pyarrow with no engine, and partition pruning means a single-county query touches
   one file. This is also the redistributable artifact (CC0 permits it) — consumers
   could pull a prebuilt release asset instead of hammering TxGIO with 254 requests.

2. DuckDB is what you query — it reads the Parquet directly (read_parquet('.../*.parquet')),
   so it needs no duplicate copy. For hot lookup paths, materialize a .duckdb with an
   index on the normalized key; for one-shot batch jobs, query the Parquet in place.
   DuckDB gives real SQL joins, levenshtein/jaro_winkler for tier 4, and the
   spatial extension for nearest-neighbor in tier 5.

Why not SQLite: 12M rows × 20 text columns balloons to multiple GB, and bulk load is
far slower. Its FTS5/R*Tree are nice but DuckDB covers the same ground at a third the
size with much better scan throughput. Why not PostGIS: correct for a server, wrong as
a library dependency — nobody pip installs a package that needs a database server.

Cache location: platformdirs user cache dir, overridable by env var, with the
manifest (collection_id, acquisition_date, per-county sha256) alongside so staleness
and provenance are both checkable. Ship a fetch CLI rather than downloading implicitly
on import — 823 MB should never be a surprise side effect.

---

6. Reference implementation

The full acquisition + conversion has been run end to end and verified against the
live endpoints (that's where every number above comes from): dynamic vintage resolution,
the WAF-safe UA, size-verified resumable downloads across all 254 counties, /vsizip/
reading, key normalization, and manifest generation with per-county sha256.

It currently exists as a shell downloader plus a Python conversion script in
voterfile-audit-pipeline, not yet as a packaged module — the recipe is proven, the
packaging is what this issue is asking for. Happy to hand over both as the starting
point for providers/txgio.py.

Suggested acceptance criteria

• fetch pulls all 254 counties, verifies size, resumes cleanly, writes a manifest
• Vintage resolved dynamically; resolved UUID pinned in the manifest
• Parquet output partitioned by county FIPS
• resolve() / resolve_batch() return tier + confidence, never a bare coordinate
• Tier 3 fallback verified to recover the missing-ZIP cohort
• Benchmark: batch-resolve 100k addresses, report match rate by tier
• RyanData-Address-Utils consumes it under a [geo] extra, no hard dependency