Skip to content

feat: Add big endian fixtures with predictor 2 and predictor 3 - #73

Open
james-willis wants to merge 1 commit into
developmentseed:mainfrom
james-willis:jw/big-endian-fixtures
Open

james-willis wants to merge 1 commit into
developmentseed:mainfrom
james-willis:jw/big-endian-fixtures

Conversation

@james-willis

@james-willis james-willis commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Adds two big endian (MM) fixtures, as requested in developmentseed/deck.gl-raster#687.

Fixture Exercises
uint16_1band_lzw_block128_predictor2_big_endian Samples must be swapped to native order before undoing horizontal differencing, which adds neighbouring samples as integers.
float32_1band_deflate_block64_predictor3_big_endian Predictor 3 byte planes are most significant byte first whatever the file's byte order, so these samples must not be swapped.

Both are 128×128 with 64×64 tiles and one overview, and pass rio cogeo validate.

How they're made

GDAL's COG driver has no ENDIANNESS option, so a new rewrite_big_endian helper in write_utils.py copies a little endian COG through the GTiff driver with COPY_SRC_OVERVIEWS=YES and ENDIANNESS=BIG, which keeps the COG layout. The uint16 fixture is rewritten from uint16_1band_lzw_block128_predictor2; the float32 one from a np.linspace COG built the same way as float32_1band_lerc_block32.

Tiling, compression and predictor are copied from the source, which relies on #74 (write_cog now writes the predictor into the COG).

Tests

tests/test_big_endian.py checks, for each fixture, that the file starts with MM, that every IFD carries the expected predictor (via tifffile), and that every level matches its little endian source as read by rasterio. pyproject.toml gains pythonpath = ["."] so the test can import the generators. pixi run check and pixi run test pass locally, and regenerating is byte-for-byte stable.

Downstream check

With deck.gl-raster's fixtures submodule pointed at this branch and both fixtures added to integration-rasterio.test.ts:

GDAL's COG driver can't write big endian files, so these are rewritten
from a little endian COG through the GTiff driver with COPY_SRC_OVERVIEWS,
which keeps the COG layout.

- uint16_1band_lzw_block128_predictor2_big_endian: readers must swap
  samples to native order before undoing horizontal differencing.
- float32_1band_deflate_block64_predictor3_big_endian: predictor 3 byte
  planes are most significant byte first in every file, so readers must
  not swap these samples.

A test checks that both files are MM, carry their predictor on every IFD,
and match their little endian source at every level.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The fixtures, generation path, metadata, and regression coverage consistently satisfy the stated endian and predictor requirements.

Review effort: Balanced
Findings: None

What changed in this PR

Adds big-endian predictor fixtures to validate downstream GeoTIFF decoding behavior.

Changes:

  • Adds Predictor 2 and Predictor 3 big-endian COG fixtures.
  • Adds reusable endian-rewriting logic and fixture validation tests.
  • Configures pytest imports for generator modules.
File Description
pyproject.toml Adds the repository root to pytest’s Python path.
rasterio_generated/​write_utils.py Adds the big-endian COG rewrite helper.
tests/​test_big_endian.py Validates byte order, predictors, and pixels.
rasterio_generated/​fixtures/​uint16_1band_lzw_block128_predictor2_big_endian.py Generates the Predictor 2 fixture.
rasterio_generated/​fixtures/​uint16_1band_lzw_block128_predictor2_big_endian.tif Provides the generated uint16 fixture.
rasterio_generated/​fixtures/​uint16_1band_lzw_block128_predictor2_big_endian_info.md Records uint16 fixture metadata.
rasterio_generated/​fixtures/​float32_1band_deflate_block64_predictor3_big_endian.py Generates the Predictor 3 fixture.
rasterio_generated/​fixtures/​float32_1band_deflate_block64_predictor3_big_endian.tif Provides the generated float32 fixture.
rasterio_generated/​fixtures/​float32_1band_deflate_block64_predictor3_big_endian_info.md Records float32 fixture metadata.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants