Skip to content

Validate the grid, the history dimension and the latitudes and longitudes - #204

Open
priyDe (priyald) wants to merge 1 commit into
microsoft:mainfrom
priyald:validate-grid-history-latlon
Open

Validate the grid, the history dimension and the latitudes and longitudes#204
priyDe (priyald) wants to merge 1 commit into
microsoft:mainfrom
priyald:validate-grid-history-latlon

Conversation

@priyald

Copy link
Copy Markdown

Builds on #196, and should be merged after it.

#196 validates that metadata.time and metadata.atmos_levels match the data. This adds the
remaining shape invariants that Batch documents but does not enforce. There is no functional
overlap: nothing here touches metadata.time or atmos_levels.

What it checks

  1. Every surface-level, static and atmospheric variable describes the same grid.
  2. The surface-level and atmospheric variables agree on the history dimension.
  3. The latitudes and longitudes match that grid, in both the vector and the matrix form.

Why these three

The latitudes and longitudes are already checked in Aurora3DEncoder.forward, but with

assert lat.shape[0] == H and lon.shape[-1] == W

which carries no message, fires deep inside the forward pass rather than where the batch was
built, and is removed entirely under python -O. The other two currently surface as shape errors
from inside torch, for example Sizes of tensors must match except in dimension 2, which does not
tell the caller which variable is wrong.

Raising at construction, next to the existing latitude and longitude validation in
Metadata.__post_init__, points at the actual problem.

Two implementation notes

The spatial shape is read from the last two dimensions rather than the full shape. The static
variables are (h, w) at the public API but Aurora.forward expands them to (b, t, h, w), and
Perceiver3DDecoder constructs a Batch before the history dimension is inserted. Comparing full
shapes would reject both.

For the same reason the history dimension is only checked on variables that still have one, that
is surface-level variables with four dimensions and atmospheric variables with five.

Empty variable dictionaries

An empty surf_vars or atmos_vars is left alone rather than treated as an error, since running
without atmospheric variables is something you suggested trying in #176. There is a test for it.

Merge conflict, and how I would like to handle it

This creates Batch.__post_init__, and so does #196. They will conflict textually even though the
checks are disjoint. #196 is approved and should land first. Once it does, say the word and I will
rebase this onto it so the two sets of checks sit in one method. Happy to do that at any point,
including before you review, if that is easier.

Tests

Added to tests/test_batch.py:

  • latitudes or longitudes that do not match the grid raise, covering four combinations;
  • matrix latitudes and longitudes are accepted, and mismatched matrices raise;
  • variables that disagree with each other on the grid raise;
  • variables that disagree on the history dimension raise;
  • an empty atmos_vars is accepted;
  • to, crop and regrid still produce valid batches.

Full suite passes apart from test_aurora_small, which fails for me at 0.04224444 / 276.22 on
unmodified main as well. That looks like the machine-dependent numerics in #169, so I am flagging
it rather than claiming it is related to this change. ruff check and ruff format --check are
clean.

…udes

`Batch` documents the shapes of its variables but does not enforce them. Add a
`__post_init__` that checks the three invariants not covered by microsoft#196:

* every surface-level, static and atmospheric variable describes the same grid;
* the surface-level and atmospheric variables agree on the history dimension;
* the latitudes and longitudes match that grid, in both the vector and the
  matrix form.

The latitudes and longitudes are checked in the encoder today, but with a bare
`assert` that carries no message and is removed under `python -O`. The other two
surface as shape errors from inside torch, which do not say which variable is
wrong. Raising at construction is consistent with the existing latitude and
longitude validation in `Metadata.__post_init__`.

The spatial shape is read from the last two dimensions because the static
variables are expanded to `(b, t, h, w)` inside the model and the decoder builds
a batch before the history dimension is inserted. The history dimension is
checked only on variables that still have one, for the same reason.

An empty `surf_vars` or `atmos_vars` is left alone rather than treated as an
error, since running without atmospheric variables is suggested in microsoft#176.
@priyald
priyDe (priyald) requested a review from a team September 9, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant