Skip to content

Four fixes from reviewing the spec builder - #950

Merged
FBumann merged 4 commits into
spec-builderfrom
spec-review-fixes
Sep 8, 2026
Merged

Four fixes from reviewing the spec builder#950
FBumann merged 4 commits into
spec-builderfrom
spec-review-fixes

Conversation

@FBumann

@FBumann FBumann commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Note

The following content was generated by AI.

Stacked on #949. Four bugs found reviewing #922, one commit each. Every one is
reproduced by a test that fails on the parent commit and passes on the fix.

1. read_netcdf relabels a hand-added variable (fix(io))

restamp_coords puts a spec's master coordinates back on every container
carrying a dimension of that name, so the whole model agrees on one dtype per
dimension however the engine returned it. It asked only whether the dtypes
differed, never whether the labels were the same — and a spec-built model may
hold hand-added variables of its own.

  • a variable on snapshot = [10, 11, 12] beside a spec built on [0, 1, 2]
    came back silently relabelled to the spec's;
  • one on a snapshot of another length failed the read with
    ValueError: conflicting sizes for dimension 'snapshot'.

The CSRConstraint branch rebuilt its Grid unconditionally, without even the
dtype guard. Both go unseen on netcdf4, which preserves the dtype; it is scipy
narrowing int64 to int32 that makes an index look stale.

An index is now restamped only where it holds the master's own labels, which is
what Index.equals asks — it compares labels and not dtypes, so a narrowed int
and a widened bool still match.

2. A lookup that maps nothing (fix(spec))

grouped_sum promises that "a group no member reaches holds the empty sum,
which is 0 and not an absence", and delivers it by filtering the operand down to
its mapped members and grouping what is left. Where a lookup maps no member at
all, what is left is empty — and an empty dimension is one xarray refuses to
group over, so the promise came out as ValueError: s must not be empty from
inside the groupby, on the fold and at build time alike.

The empty case is now answered directly, the way sum_over already answers a
term beside an empty dimension: zeros over the operand's remaining dimensions
and every declared label of the group, as data or as a constant term.

3. repr(model.spec) on an unreached dimension (fix(spec))

attach lets a dimension the spec declares but no declaration reaches go
without a source, deliberately (test_unreached_dimension_needs_no_source), so
it never lands in coords. The repr indexed coords[d] for every declared
dimension anyway — so a model that built perfectly well could not be looked at
in a REPL or a notebook. Shown as unreached now.

4. A half-built model on a retain failure (fix(spec))

retain="all" reaches every declared parameter, including ones no declaration
reads, and did so after build() had already put the variables and
constraints on the model. A parameter with no source therefore raised with the
model half-built and _spec still unset — and the corrected retry was then
refused by the guard that add_spec builds into an empty model, so the only way
on was a fresh model. The retained set needs nothing the build produces, so it
is resolved first.

Verification (3dadc43)
uv run --extra dev --extra solvers --group spec pytest test/ -q --ignore=test/remote
9055 passed, 1061 skipped

uv run --extra dev --group spec mypy linopy
Found 13 errors in 3 files (checked 49 source files)
# identical to the base commit: missing types-PyYAML and pre-existing
# linopy/expressions.py union-attr errors in this local environment

test/remote is excluded: its OETC cloud-provider tests fail identically on the
base commit in this environment and touch nothing here.

Each fix was also checked in isolation by stashing only the source change and
re-running its test:

commit test without the fix
62860f72 test_a_hand_added_variable_keeps_its_own_labels 2 failed (scipy, both label sets)
fd0f2669 test_a_lookup_that_maps_nothing_leaves_every_group_at_the_empty_sum ValueError: s must not be empty
673eb6b1 test_a_declared_dimension_with_no_source_still_reprs KeyError
3dadc434 test_a_build_that_cannot_retain_leaves_the_model_buildable model left holding p and power_balance

Checklist

  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented.
  • Unit tests for new features were added.
  • A note for the release notes doc/release_notes.rst is included.

🤖 Generated with Claude Code

https://claude.ai/code/session_015ZfHfTFENUy6WxnFFri5Td

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy time — v1 / legacy
peak v1/legacy time v1/legacy
Full table (time + peak, mean)
benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08996 (1.09)   │   15.03 (1.00) 
 (v1)        0.08253 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1031 (1.17)   │   12.04 (1.00) 
 (v1)       0.08838 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03943 (1.09)   │   15.20 (1.0) 
 (v1)        0.03605 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05698 (1.06)   │   44.93 (1.0) 
 (v1)        0.05379 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0431 (1.08)   │   11.51 (1.0) 
 (v1)       0.03987 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1018 (1.07)   │   24.34 (1.06) 
 (v1)       0.09491 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                time (s)         peak (MiB) 
 name               mean   │           mean 
────────────────────────────────────────────
 (legacy)   0.112 (1.07)   │   16.12 (1.00) 
 (v1)       0.1048 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02416 (1.07)   │   752.18 (1.10) 
 (v1)        0.02264 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02394 (1.07)   │   3.12 (1.33) 
 (v1)        0.02247 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06599 (1.24)   │   126.16 (1.44) 
 (v1)        0.05339 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06501 (1.24)   │   126.16 (1.44) 
 (v1)        0.05261 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06555 (1.24)   │   126.16 (1.44) 
 (v1)        0.05278 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05602 (1.02)   │    715.12 (1.0) 
 (v1)        0.05475 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05455 (1.10)   │   4.54 (1.27) 
 (v1)         0.0495 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                time (s)          peak (KiB) 
 name               mean   │            mean 
─────────────────────────────────────────────
 (legacy)   0.379 (1.05)   │   704.12 (1.09) 
 (v1)       0.3613 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.4002 (1.06)   │   18.34 (1.0) 
 (v1)        0.3782 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.3946 (1.05)   │   9.54 (1.0) 
 (v1)        0.3765 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.07728 (1.10)   │   3.77 (1.12) 
 (v1)        0.07052 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07735 (1.09)   │   216.59 (1.10) 
 (v1)        0.07082 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03894 (1.09)   │   938.49 (1.0) 
 (v1)        0.03569 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.04006 (1.10)   │   9.66 (1.0) 
 (v1)        0.03631 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.04034 (1.12)   │   5.32 (1.0) 
 (v1)        0.03617 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02212 (1.05)   │   1.47 (1.0) 
 (v1)          0.021 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0219 (1.03)   │   1.47 (1.0) 
 (v1)       0.02117 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02153 (1.02)   │   1.47 (1.0) 
 (v1)        0.02109 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1948 (1.06)   │   946.85 (1.06) 
 (v1)        0.1834 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1899 (1.05)   │   12.01 (1.00) 
 (v1)        0.1812 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04893 (1.08)   │   147.70 (1.06) 
 (v1)        0.04532 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04859 (1.08)   │   2.60 (1.09) 
 (v1)        0.04517 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03965 (1.09)   │   696.75 (1.03) 
 (v1)        0.03622 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08516 (1.00)   │   137.97 (1.0) 
 (v1)        0.08499 (1.0)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06179 (1.09)   │   69.22 (1.0) 
 (v1)        0.05662 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04664 (1.12)   │   402.33 (1.00) 
 (v1)        0.04168 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04634 (1.12)   │   3.19 (1.19) 
 (v1)        0.04129 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05138 (1.10)   │   29.00 (1.54) 
 (v1)        0.04684 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06197 (1.14)   │   37.95 (1.43) 
 (v1)        0.05413 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0982 (1.00)   │    410.93 (1.0) 
 (v1)       0.09804 (1.0)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)    0.1047 (1.0)   │     9.94 (1.0) 
 (v1)       0.1058 (1.01)   │   10.22 (1.03) 

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.

@FBumann
FBumann force-pushed the spec-parameter-ownership branch from 041ab2d to 4d4a508 Compare September 8, 2026 17:24
Base automatically changed from spec-parameter-ownership to spec-builder September 8, 2026 17:24
FBumann and others added 4 commits September 8, 2026 19:24
`restamp_coords` puts a spec's master coordinates back on every container
carrying a dimension of that name, so the whole model agrees on one dtype
per dimension however the netcdf engine returned it. It asked only whether
the dtypes differed, not whether the labels were the same -- and a spec-built
model may hold hand-added variables of its own.

So a variable added on `snapshot = [10, 11, 12]` beside a spec built on
`[0, 1, 2]` came back relabelled to the spec's, silently; and one on a
`snapshot` of another length failed the read outright with `conflicting sizes
for dimension 'snapshot'`. The `CSRConstraint` branch replaced its `Grid`
indexes unconditionally, without even the dtype guard. Both went unseen on
netcdf4, which preserves the dtype; scipy narrowing int64 to int32 is what
makes an index look stale.

An index is now restamped only where it holds the master's own labels, which
is what `Index.equals` asks -- it compares labels and not dtypes, so a
narrowed int and a widened bool still match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ZfHfTFENUy6WxnFFri5Td
`grouped_sum` promises that "a group no member reaches holds the empty sum,
which is 0 and not an absence", and delivers it by filtering the operand down
to its mapped members and grouping what is left. Where a lookup maps no
member at all, what is left is empty -- and an empty dimension is one xarray
refuses to group over, so the promise came out as `ValueError: s must not be
empty` from inside the groupby, on the fold and at build time alike.

The empty case is now answered directly, the way `sum_over` already answers a
term beside an empty dimension: zeros over the operand's remaining dimensions
and every declared label of the group, as data or as a constant term.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ZfHfTFENUy6WxnFFri5Td
`attach` lets a dimension the spec declares but no declaration reaches go
without a source, deliberately, so it never lands in `coords`. The
`ModelSpec` repr indexed `coords[d]` for every declared dimension anyway, so
a model that built perfectly well could not be looked at: `repr(model.spec)`
raised `KeyError` in a REPL or a notebook. Such a dimension is now shown as
`unreached`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ZfHfTFENUy6WxnFFri5Td
`retain="all"` reaches every declared parameter, including ones no
declaration reads, and it did so after `build()` had already put the
variables and constraints on the model. A parameter with no source therefore
raised with the model half-built and `_spec` still unset -- and the corrected
retry was then refused by the guard that `add_spec` builds into an empty
model, so the only way on was a fresh model.

The retained set is resolved first. It needs nothing the build produces, so
the failure now lands before anything is added and the same model takes the
corrected sources.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ZfHfTFENUy6WxnFFri5Td
@FBumann
FBumann merged commit 87b2ff1 into spec-builder Sep 8, 2026
7 of 8 checks passed
@FBumann
FBumann deleted the spec-review-fixes branch September 8, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant