Skip to content

Add compression-ratio check to zip bomb guard, raise entry size cap - #114

Merged
iandees merged 2 commits into
masterfrom
nad-zip-bomb-ratio-guard
Sep 3, 2026
Merged

iandees merged 2 commits into
masterfrom
nad-zip-bomb-ratio-guard

Conversation

@iandees

@iandees iandees commented Sep 3, 2026 •

Copy link
Copy Markdown
Member

Summary

  • The zip-bomb guard's flat 2GB per-entry declared-size cap was rejecting the national address database (NAD) countrywide source: its gdb table legitimately declares ~34GB uncompressed, well within a normal compression ratio (~3.5:1 against the ~9.7GB downloaded zip), nothing like an actual zip bomb.
  • Raises MAX_ZIP_ENTRY_BYTES to 50GB as a generous backstop on worst-case disk usage.
  • Adds a new MAX_ZIP_RATIO check (declared:compressed size, default 100x) that catches real zip bombs (tiny compressed payload inflating enormously) regardless of the absolute cap, since ratio is what actually distinguishes a bomb from a big well-compressed dataset.
  • The first version of this ratio check broke CI: real Esri file geodatabases (e.g. the us/nj/statewide test fixture) contain tiny (a few KB) but highly-compressible index files (.gdbtablx, .atx) that legitimately exceed 100x ratio on their own. Added MIN_RATIO_CHECK_BYTES (500MB) so the ratio check only applies to entries large enough that an extreme ratio could actually threaten disk space - a few KB inflating to a few hundred KB isn't a real risk regardless of ratio.

Test plan

  • Added test_extreme_compression_ratio_raises_decompression_error, test_low_compression_ratio_entry_does_not_raise, and test_small_highly_compressible_entry_does_not_raise to TestZipDecompressTask - the last one reproduces the exact CI failure (a tiny, highly-compressible entry) as a regression test.
  • python3 test.py (full suite, matching CI) - 123/123 passing in the project's Docker test image, including test_single_us_nj_statewide which failed on the first push.

@iandees
iandees merged commit b929d42 into master Sep 3, 2026
1 check passed
@iandees
iandees deleted the nad-zip-bomb-ratio-guard branch September 3, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant