From d916f759dca7ee41bfa56f51a2295b8b0e90a81a Mon Sep 17 00:00:00 2001 From: Ilan Gold Date: Tue, 14 Jul 2026 17:13:03 +0200 Subject: [PATCH 1/8] chore: add Ilan Gold to the team list (#4145) --- TEAM.md | 1 + 1 file changed, 1 insertion(+) diff --git a/TEAM.md b/TEAM.md index dc22a1ee87..ce9de1d486 100644 --- a/TEAM.md +++ b/TEAM.md @@ -11,6 +11,7 @@ - @dcherian (Deepak Cherian) - @TomAugspurger (Tom Augspurger) - @maxrjones (Max Jones) +- @ilan-gold (Ilan Gold) ## Emeritus core-developers - @alimanfoo (Alistair Miles) From 91a8ec06dc4d475fe40691e0ab668dd655d3751a Mon Sep 17 00:00:00 2001 From: Davis Bennett Date: Thu, 16 Jul 2026 16:34:50 +0200 Subject: [PATCH 2/8] docs: 3.3.0 release notes (#4148) * chore(deps): bump the actions group across 1 directory with 8 updates (#176) Bumps the actions group with 8 updates in the / directory: | Package | From | To | | --- | --- | --- | | [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` | | [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` | | [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` | | [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` | Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6 - [Release notes](https://github.com/prefix-dev/setup-pixi/releases) - [Commits](https://github.com/prefix-dev/setup-pixi/compare/1b2de7f3351f171c8b4dfeb558c639cb58ed4ec0...5185adfbffb4bd703da3010310260805d89ebb11) Updates `codecov/codecov-action` from 6.0.0 to 6.0.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/codecov/codecov-action/compare/57e3a136b779b570ffcdbf80b3bdc90e7fab3de2...e79a6962e0d4c0c17b229090214935d2e33f8354) Updates `github/issue-metrics` from 4.2.2 to 4.2.7 - [Release notes](https://github.com/github/issue-metrics/releases) - [Commits](https://github.com/github/issue-metrics/compare/c9e9838147fd355dace335ba787f01b6641a400a...1e38d5e62363e14db8019ed7d106b9855bdba6cc) Updates `j178/prek-action` from 2.0.3 to 2.0.4 - [Release notes](https://github.com/j178/prek-action/releases) - [Commits](https://github.com/j178/prek-action/compare/6ad80277337ad479fe43bd70701c3f7f8aa74db3...bdca6f102f98e2b4c7029491a53dfd366469e33d) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v7...043fb46d1a93c77aae656e7c1c64a875d1fc6a0a) Updates `actions/download-artifact` from 7.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/v7...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](https://github.com/pypa/gh-action-pypi-publish/compare/v1.13.0...cef221092ed1bacb1cc03d23a2d87d1d172e277b) Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6 - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](https://github.com/zizmorcore/zizmor-action/compare/b1d7e1fb5de872772f31590499237e7cce841e8e...5f14fd08f7cf1cb1609c1e344975f152c7ee938d) --- updated-dependencies: - dependency-name: prefix-dev/setup-pixi dependency-version: 0.9.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: github/issue-metrics dependency-version: 4.2.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: j178/prek-action dependency-version: 2.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: zizmorcore/zizmor-action dependency-version: 0.5.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: byte-order handling for structured dtypes in the bytes codec (#220) * fix: byte-order handling for structured dtypes in the bytes codec The bytes codec neither byte-swapped structured-dtype fields to its configured endian on encode (numpy reports byteorder '|' for void dtypes, so the top-level byteorder comparison never detected a mismatch) nor honored its endian when decoding, silently corrupting any structured data whose field byte order differed from the stored one (e.g. virtual references to external big-endian data). Encode now detects byte-order mismatches by comparing full dtypes via newbyteorder, and decode reinterprets raw bytes in the stored byte order before converting to the data type's declared byte order, so the stored layout (codec state) and the in-memory layout (array data type) are independent. Closes #4141 Assisted-by: ClaudeCode:claude-fable-5 * test: fold structured byte-order cases into existing bytes codec tests Extend test_endian's parametrization with structured dtypes and test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus stored-layout and decoded-dtype assertions, instead of adding parallel test functions for the same properties. Assisted-by: ClaudeCode:claude-fable-5 * refactor: rename stored_dtype to view_dtype in BytesCodec decode The variable is the dtype used to view the raw chunk bytes (byte order from the codec's endian configuration), not a property of the stored data or of the returned buffer, which always carries the array's declared dtype. Assisted-by: ClaudeCode:claude-fable-5 * docs: note that the decode-side byte-order conversion copies the chunk Assisted-by: ClaudeCode:claude-fable-5 * docs: 3.3.0 release notes Add missing changelog fragments for #3955 (datetime64/timedelta64 V3 metadata types), #3966 (writes to 0-dimensional sharded arrays), and the public alias renames that accompanied the #3963/#3968 enum deprecations, then build the 3.3.0 release notes with towncrier, consuming all fragments accumulated since v3.2.1. Co-Authored-By: Claude Fable 5 --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Claude Fable 5 --- changes/202.bugfix.md | 1 - changes/214.misc.md | 1 - changes/215.misc.md | 1 - changes/2929.bugfix.md | 3 - changes/3004.feature.md | 4 - changes/3009.doc.md | 1 - changes/3417.bugfix.md | 4 - changes/3469.bugfix.md | 1 - changes/3826.feature.md | 1 - changes/3885.bugfix.md | 1 - changes/3885.feature.1.md | 1 - changes/3885.feature.md | 1 - changes/3899.bugfix.md | 7 -- changes/3908.misc.md | 1 - changes/3925.feature.md | 1 - changes/3954.bugfix.md | 1 - changes/3963.removal.md | 6 -- changes/3968.removal.md | 17 ----- changes/3972.misc.md | 1 - changes/3973.removal.md | 1 - changes/3975.misc.md | 1 - changes/3977.bugfix.md | 1 - changes/3979.misc.md | 1 - changes/3984.doc.md | 4 - changes/3987.feature.md | 1 - changes/3990.misc.md | 5 -- changes/3998.misc.md | 1 - changes/4000.misc.md | 1 - changes/4001.misc.md | 6 -- changes/4003.bugfix.md | 19 ----- changes/4016.bugfix.md | 1 - changes/4032.bugfix.md | 1 - changes/4046.misc.md | 1 - changes/4052.doc.md | 21 ----- changes/4053.doc.md | 1 - changes/4054.misc.md | 1 - changes/4059.bugfix.md | 1 - changes/4073.misc.md | 1 - changes/4074.bugfix.md | 7 -- changes/4086.misc.md | 1 - changes/4116.bugfix.md | 1 - changes/4132.doc.md | 6 -- changes/4133.doc.md | 1 - changes/4138.misc.md | 1 - changes/4141.bugfix.md | 1 - docs/release-notes.md | 156 ++++++++++++++++++++++++++++++++++++++ 46 files changed, 156 insertions(+), 141 deletions(-) delete mode 100644 changes/202.bugfix.md delete mode 100644 changes/214.misc.md delete mode 100644 changes/215.misc.md delete mode 100644 changes/2929.bugfix.md delete mode 100644 changes/3004.feature.md delete mode 100644 changes/3009.doc.md delete mode 100644 changes/3417.bugfix.md delete mode 100644 changes/3469.bugfix.md delete mode 100644 changes/3826.feature.md delete mode 100644 changes/3885.bugfix.md delete mode 100644 changes/3885.feature.1.md delete mode 100644 changes/3885.feature.md delete mode 100644 changes/3899.bugfix.md delete mode 100644 changes/3908.misc.md delete mode 100644 changes/3925.feature.md delete mode 100644 changes/3954.bugfix.md delete mode 100644 changes/3963.removal.md delete mode 100644 changes/3968.removal.md delete mode 100644 changes/3972.misc.md delete mode 100644 changes/3973.removal.md delete mode 100644 changes/3975.misc.md delete mode 100644 changes/3977.bugfix.md delete mode 100644 changes/3979.misc.md delete mode 100644 changes/3984.doc.md delete mode 100644 changes/3987.feature.md delete mode 100644 changes/3990.misc.md delete mode 100644 changes/3998.misc.md delete mode 100644 changes/4000.misc.md delete mode 100644 changes/4001.misc.md delete mode 100644 changes/4003.bugfix.md delete mode 100644 changes/4016.bugfix.md delete mode 100644 changes/4032.bugfix.md delete mode 100644 changes/4046.misc.md delete mode 100644 changes/4052.doc.md delete mode 100644 changes/4053.doc.md delete mode 100644 changes/4054.misc.md delete mode 100644 changes/4059.bugfix.md delete mode 100644 changes/4073.misc.md delete mode 100644 changes/4074.bugfix.md delete mode 100644 changes/4086.misc.md delete mode 100644 changes/4116.bugfix.md delete mode 100644 changes/4132.doc.md delete mode 100644 changes/4133.doc.md delete mode 100644 changes/4138.misc.md delete mode 100644 changes/4141.bugfix.md diff --git a/changes/202.bugfix.md b/changes/202.bugfix.md deleted file mode 100644 index 9c9bd40f21..0000000000 --- a/changes/202.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Stop emitting an `UnstableSpecificationWarning` when serializing the `struct` data type to Zarr V3 metadata. The `struct` data type now has a stable Zarr V3 specification. The legacy `structured` alias and the unspecified `null_terminated_bytes`, `raw_bytes`, and `variable_length_bytes` data types continue to warn. diff --git a/changes/214.misc.md b/changes/214.misc.md deleted file mode 100644 index 8611362e84..0000000000 --- a/changes/214.misc.md +++ /dev/null @@ -1 +0,0 @@ -Bind the shared moto S3 test server to an ephemeral port instead of a fixed one. The fixed port deadlocked `pytest -n auto`: session-scoped fixtures run once per xdist worker, so concurrent workers raced to bind the same port and the losers blocked forever in `ThreadedMotoServer.start()`. diff --git a/changes/215.misc.md b/changes/215.misc.md deleted file mode 100644 index f8df7cc632..0000000000 --- a/changes/215.misc.md +++ /dev/null @@ -1 +0,0 @@ -Enable pytest's `strict = true` config option (strict config, markers, xfail, and parametrization ids), replacing the `--strict-config`/`--strict-markers` addopts flags that pytest silently ignored before 9.1, and fix the seven duplicate parametrization ids it surfaced — including restoring float-JSON roundtrip cases that were meant to cover numpy scalars but ran plain-float cases twice instead. Also arm pytest's faulthandler watchdog (`faulthandler_timeout = 600` with `faulthandler_exit_on_timeout`) so a deadlocked test dumps every thread's traceback and fails the run instead of hanging indefinitely. diff --git a/changes/2929.bugfix.md b/changes/2929.bugfix.md deleted file mode 100644 index 533a5f86c1..0000000000 --- a/changes/2929.bugfix.md +++ /dev/null @@ -1,3 +0,0 @@ -Fix equality comparison of `ArrayV2Metadata` and `ArrayV3Metadata` objects with a -`NaN` fill value. Such objects are now compared by their JSON-serialized form, so two -otherwise-identical metadata objects with a `NaN` (or infinite) fill value compare equal. diff --git a/changes/3004.feature.md b/changes/3004.feature.md deleted file mode 100644 index 9d3816a50c..0000000000 --- a/changes/3004.feature.md +++ /dev/null @@ -1,4 +0,0 @@ -Optimizes reading multiple chunks from a shard. Serial calls to `Store.get()` -in the sharding codec have been replaced with a single call to -`Store.get_ranges()`, which coalesces nearby byte ranges and fetches them -concurrently. diff --git a/changes/3009.doc.md b/changes/3009.doc.md deleted file mode 100644 index 777672b77b..0000000000 --- a/changes/3009.doc.md +++ /dev/null @@ -1 +0,0 @@ -Document the changes to `zarr.errors` in the 3.0 migration guide, including the removal of v2 exception classes and the introduction of `NodeNotFoundError`. diff --git a/changes/3417.bugfix.md b/changes/3417.bugfix.md deleted file mode 100644 index be5b44f3e9..0000000000 --- a/changes/3417.bugfix.md +++ /dev/null @@ -1,4 +0,0 @@ -Fixed `BytesCodec.from_dict` so that `BytesCodec` instances roundtrip to / from -their dict representation. `BytesCodec.from_dict` now interprets a missing -`endian` configuration as `endian=None` (matching what `BytesCodec.to_dict` -emits), instead of falling back to the system's native byte order. diff --git a/changes/3469.bugfix.md b/changes/3469.bugfix.md deleted file mode 100644 index eb56e87476..0000000000 --- a/changes/3469.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed `save_array`, `Group.__setitem__`, and `load` for 0-dimensional arrays. diff --git a/changes/3826.feature.md b/changes/3826.feature.md deleted file mode 100644 index 8909e777c5..0000000000 --- a/changes/3826.feature.md +++ /dev/null @@ -1 +0,0 @@ -Added a `subchunk_write_order` option to `ShardingCodec` to control the physical order of subchunks within a shard. Supported values are `morton`, `unordered`, `lexicographic`, and `colexicographic`. `unordered` makes no guarantee about subchunk layout. This setting affects only on-disk layout, not the data read back, and is not persisted in array metadata: it applies per codec instance and is not recovered when reopening a sharded array. diff --git a/changes/3885.bugfix.md b/changes/3885.bugfix.md deleted file mode 100644 index d4c1dd5c03..0000000000 --- a/changes/3885.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed inner-codec spec evolution for sharded arrays. The sharding codec now threads the array spec through its inner codec chain when evolving codecs, so a codec that changes the dtype upstream of `BytesCodec` no longer leaves the inner chain evolved against the wrong spec (which previously failed at decode time). This runs on the default `BatchedCodecPipeline` as well. Standard inner chains (`[BytesCodec]`, `[BytesCodec, ZstdCodec]`, transpose + bytes) are byte-identical to before. Restores the behavior of #2179. diff --git a/changes/3885.feature.1.md b/changes/3885.feature.1.md deleted file mode 100644 index 0a51ba6844..0000000000 --- a/changes/3885.feature.1.md +++ /dev/null @@ -1 +0,0 @@ -Added `SyncByteGetter` and `SyncByteSetter` runtime-checkable protocols and a `get_ranges_sync` method on the `Store` ABC. These let custom byte getters/setters opt into the synchronous codec pipeline's fast path for in-memory IO, which the sharding codec uses for its inner chunks. diff --git a/changes/3885.feature.md b/changes/3885.feature.md deleted file mode 100644 index 13010f521e..0000000000 --- a/changes/3885.feature.md +++ /dev/null @@ -1 +0,0 @@ -Added `FusedCodecPipeline`, an opt-in codec pipeline that runs codec compute synchronously and in bulk (avoiding the per-chunk async scheduling overhead of the default `BatchedCodecPipeline`), giving large speedups for sharded arrays (up to ~24x writes / ~14x reads on many-chunks-per-shard layouts, more with compression) and no regressions on compute-bound workloads. The default `BatchedCodecPipeline` is unchanged for standard configurations, so existing code keeps working unless you opt in; enable the new pipeline with `zarr.config.set({"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"})`. diff --git a/changes/3899.bugfix.md b/changes/3899.bugfix.md deleted file mode 100644 index e2b2177a39..0000000000 --- a/changes/3899.bugfix.md +++ /dev/null @@ -1,7 +0,0 @@ -Make chunk normalization properly handle `-1` as a compact representation of the -length of an entire axis. Reject several previously-accepted but ill-defined -chunk specifications: `chunks=True` (previously silently produced size-1 chunks), -chunk tuples shorter than the array's number of dimensions (previously padded to -the array's shape), and `None` as a per-dimension chunk size. These all now -raise informative errors. Also fix chunk handling for 0-length array dimensions, -and add explicit rejection of 0-length chunks. diff --git a/changes/3908.misc.md b/changes/3908.misc.md deleted file mode 100644 index 66717e8444..0000000000 --- a/changes/3908.misc.md +++ /dev/null @@ -1 +0,0 @@ -Reuse a constant `ArraySpec` during indexing when possible. \ No newline at end of file diff --git a/changes/3925.feature.md b/changes/3925.feature.md deleted file mode 100644 index ed07be309c..0000000000 --- a/changes/3925.feature.md +++ /dev/null @@ -1 +0,0 @@ -Add `zarr.abc.store.Store.get_ranges` for concurrent, coalesced multi-range reads from a single key. The method is defined on the `Store` ABC with a default implementation built on `Store.get`, so every store inherits a working version; stores with native multi-range backends (e.g. `FsspecStore`) can override for efficiency. Coalescing knobs (`max_concurrency`, `max_gap_bytes`, `max_coalesced_bytes`) are passed as keyword arguments to `get_ranges`. Failures from underlying fetches surface as a `BaseExceptionGroup` (PEP 654); callers should use `except*` to filter for specific exception types such as `FileNotFoundError`. diff --git a/changes/3954.bugfix.md b/changes/3954.bugfix.md deleted file mode 100644 index 1a6c93a8ac..0000000000 --- a/changes/3954.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Handle missing consolidated metadata in leaf Group nodes. diff --git a/changes/3963.removal.md b/changes/3963.removal.md deleted file mode 100644 index a83fd53853..0000000000 --- a/changes/3963.removal.md +++ /dev/null @@ -1,6 +0,0 @@ -The ``BloscShuffle`` and ``BloscCname`` enums (``zarr.codecs.BloscShuffle``, -``zarr.codecs.BloscCname``) are now deprecated. Pass the equivalent literal -string (e.g. ``"zstd"``, ``"bitshuffle"``) when constructing a ``BloscCodec``. -The enum classes remain importable but emit ``DeprecationWarning`` on member -access, and will be removed in a future release. ``BloscCodec.cname`` and -``BloscCodec.shuffle`` are now plain strings rather than enum members. diff --git a/changes/3968.removal.md b/changes/3968.removal.md deleted file mode 100644 index 075e4a15a9..0000000000 --- a/changes/3968.removal.md +++ /dev/null @@ -1,17 +0,0 @@ -The ``Endian`` (``zarr.codecs.bytes.Endian``) and ``ShardingCodecIndexLocation`` -(``zarr.codecs.ShardingCodecIndexLocation``) enums are now deprecated. Pass the -equivalent literal string instead (e.g. ``"little"`` / ``"big"``, ``"start"`` / -``"end"``). The enum classes remain importable but emit ``DeprecationWarning`` -on member access, and will be removed in a future release. ``BytesCodec.endian`` -and ``ShardingCodec.index_location`` are now plain strings rather than enum -members. - -Two follow-on changes from this deprecation: - -- ``NDBuffer.byteorder`` now returns a literal string (``"little"`` or - ``"big"``) rather than an ``Endian`` member. Subclasses overriding this - property should update their return type. -- The module-level binding ``zarr.codecs.bytes.default_system_endian`` was - removed. ``BytesCodec()`` continues to default to ``sys.byteorder``; - external callers that imported ``default_system_endian`` should use - ``sys.byteorder`` directly. diff --git a/changes/3972.misc.md b/changes/3972.misc.md deleted file mode 100644 index 60e5f75cca..0000000000 --- a/changes/3972.misc.md +++ /dev/null @@ -1 +0,0 @@ -Run `mypy` via `uv run mypy` instead of `pre-commit`'s isolated venv. The `dev` dependency group in `pyproject.toml`, locked by `uv.lock`, is now the single source of truth for `mypy`'s dependency set, eliminating the duplicate dependency list previously maintained in `.pre-commit-config.yaml` and giving every contributor and CI an identical, reproducible type-checking environment. diff --git a/changes/3973.removal.md b/changes/3973.removal.md deleted file mode 100644 index c25ac967a2..0000000000 --- a/changes/3973.removal.md +++ /dev/null @@ -1 +0,0 @@ -Removed the NumPy 1.x implementation of the `VariableLengthUTF8` data type because NumPy 1.x is no longer supported under [SPEC0](https://scientific-python.org/specs/spec-0000/). \ No newline at end of file diff --git a/changes/3975.misc.md b/changes/3975.misc.md deleted file mode 100644 index b90147988a..0000000000 --- a/changes/3975.misc.md +++ /dev/null @@ -1 +0,0 @@ -Store `chunks_per_shard` explicitly as a field on `_ShardIndex` instead of inferring it from `offsets_and_lengths.shape[:-1]`. The previous derivation collapsed to rank-1 for 0-D arrays, requiring a numpy-compat cast workaround that is now removed. Also removes the unused `_ShardIndex.is_dense` method, which was ported from an earlier prototype and never had any call sites or tests. diff --git a/changes/3977.bugfix.md b/changes/3977.bugfix.md deleted file mode 100644 index 6a8d9b4244..0000000000 --- a/changes/3977.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fix flaky stateful test bookkeeping when `delete_dir` matches string prefixes instead of true directory descendants. Previously a path such as `6/faNT…` could be incorrectly removed when deleting `6/f`. (See [issue #3977](https://github.com/zarr-developers/zarr-python/issues/3977).) diff --git a/changes/3979.misc.md b/changes/3979.misc.md deleted file mode 100644 index 3d49547148..0000000000 --- a/changes/3979.misc.md +++ /dev/null @@ -1 +0,0 @@ -Remove the `UnstableSpecificationWarning` emitted when serializing a `FixedLengthUTF32` data type instance, as a spec for this data type has been published to zarr-extensions. \ No newline at end of file diff --git a/changes/3984.doc.md b/changes/3984.doc.md deleted file mode 100644 index 1b92fb8e74..0000000000 --- a/changes/3984.doc.md +++ /dev/null @@ -1,4 +0,0 @@ -Clarify the difference between `zarr.load` and `zarr.open` in their docstrings. -`load` eagerly reads data into an in-memory array, while `open` returns a -lazy `Array` or `Group` backed by the store, with `See Also` cross-references -linking the two. diff --git a/changes/3987.feature.md b/changes/3987.feature.md deleted file mode 100644 index 2492b4d7dd..0000000000 --- a/changes/3987.feature.md +++ /dev/null @@ -1 +0,0 @@ -Two new fields on `ArrayConfig` control how the sharding codec coalesces partial-shard reads: `sharding_coalesce_max_gap_bytes` (default 1 MiB) and `sharding_coalesce_max_bytes` (default 16 MiB). When reading multiple chunks from the same shard, nearby byte ranges are merged into a single request to the store if separated by no more than `sharding_coalesce_max_gap_bytes` and the merged read stays within `sharding_coalesce_max_bytes`. Defaults are seeded from the matching `array.sharding_coalesce_max_gap_bytes` / `array.sharding_coalesce_max_bytes` keys in [`zarr.config`][] at array-creation time, and can be overridden per array by passing `config={...}` to [`zarr.create_array`][]. diff --git a/changes/3990.misc.md b/changes/3990.misc.md deleted file mode 100644 index ff3fcf4cf2..0000000000 --- a/changes/3990.misc.md +++ /dev/null @@ -1,5 +0,0 @@ -Widen `ChunksLike` type alias to use `Iterable` instead of `Sequence`, and also -remove `None` from the type union. This supports a broader range of types, -removing the necessity to "materialize" iterable values simply to satisfy type -annotations. It also allows use of `ChunksLike` in cases where `None` should -not be permitted. diff --git a/changes/3998.misc.md b/changes/3998.misc.md deleted file mode 100644 index bacfa93a8b..0000000000 --- a/changes/3998.misc.md +++ /dev/null @@ -1 +0,0 @@ -Centralized JSON document I/O behind free functions in `zarr.core._json` and removed the unused private `Store._get_bytes`/`_get_json` methods and their per-store overrides. diff --git a/changes/4000.misc.md b/changes/4000.misc.md deleted file mode 100644 index 17d48d3016..0000000000 --- a/changes/4000.misc.md +++ /dev/null @@ -1 +0,0 @@ -Run all doctests via pytest and fix all broken doctests. diff --git a/changes/4001.misc.md b/changes/4001.misc.md deleted file mode 100644 index adbba988d9..0000000000 --- a/changes/4001.misc.md +++ /dev/null @@ -1,6 +0,0 @@ -Restore sharding write performance for shards with many inner chunks. The -`subchunk_write_order` feature inadvertently rebuilt the per-shard chunk -coordinate grid (up to tens of thousands of coordinate tuples) on every shard -write. These coordinates are now computed once per shard shape and cached, so -repeated writes to same-shaped shards reuse them, restoring write throughput to -its previous level. diff --git a/changes/4003.bugfix.md b/changes/4003.bugfix.md deleted file mode 100644 index 36327b55df..0000000000 --- a/changes/4003.bugfix.md +++ /dev/null @@ -1,19 +0,0 @@ -`FsspecStore.from_url()` and `from_mapper()` now close the async filesystem -they create when `store.close()` is called. Previously the underlying aiohttp -`ClientSession` was left open until garbage collection, producing -`"Unclosed client session"` `ResourceWarning`s from aiohttp. - -The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when -`FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` -when a sync→async conversion was performed). When `_owns_fs` is ``True``, -`store.close()` calls the new `_close_fs()` helper, which invokes -`fs.set_session()` and closes the returned client. Callers who supply their own -filesystem instance to `FsspecStore()` directly remain responsible for its -lifecycle; `_owns_fs` is ``False`` for those stores. - -**Scope note**: This fix closes the S3 client session that is active at the time -`store.close()` is called. Some S3-backed filesystem implementations (e.g. -s3fs with ``cache_regions=True``) may internally refresh and replace their -client during I/O operations, abandoning prior sessions before ``store.close()`` -is invoked. Those intermediate sessions are outside the scope of this fix and -are an issue in the upstream filesystem library. diff --git a/changes/4016.bugfix.md b/changes/4016.bugfix.md deleted file mode 100644 index 01984110f7..0000000000 --- a/changes/4016.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed an invalid `zarr.create_array` example in the quick-start documentation (it passed an unsupported `mode` argument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested. diff --git a/changes/4032.bugfix.md b/changes/4032.bugfix.md deleted file mode 100644 index 4c9476f583..0000000000 --- a/changes/4032.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed `ObjectStore.list_dir` for object-store listings that include a directory-marker object matching the requested non-root prefix. diff --git a/changes/4046.misc.md b/changes/4046.misc.md deleted file mode 100644 index 96c0c7d78a..0000000000 --- a/changes/4046.misc.md +++ /dev/null @@ -1 +0,0 @@ -Replace the internal `DefaultFillValue` sentinel class with a `typing_extensions.Sentinel`, and raise the minimum `typing_extensions` version to 4.14. diff --git a/changes/4052.doc.md b/changes/4052.doc.md deleted file mode 100644 index 90b4679db5..0000000000 --- a/changes/4052.doc.md +++ /dev/null @@ -1,21 +0,0 @@ -Updated the custom dtype example in `examples/custom_dtype/custom_dtype.py` to -use only the public API, eliminating all non-public imports, illustrating what -users should do. - -To better support this, the following types and functions were made available -from public modules: - -| Type/Function | Non-public module | Public module | -| ------------------------- | ------------------------ | ------------- | -| `DataTypeValidationError` | `zarr.core.dtype.common` | `zarr.errors` | -| `JSON` | `zarr.core.common` | `zarr.types` | -| `ZarrFormat` | `zarr.core.common` | `zarr.types` | -| `DTypeConfig_V2` | `zarr.core.dtype.common` | `zarr.types` | -| `DTypeJSON` | `zarr.core.dtype.common` | `zarr.types` | -| `DTypeSpec_V2` | `zarr.core.dtype.common` | `zarr.dtype` | -| `check_dtype_spec_v2` | `zarr.core.dtype.common` | `zarr.dtype` | - -`DataTypeValidationError` was *moved* to `zarr.errors`. Importing it from -`zarr.core.dtype.common` (its original location), `zarr.core.dtype`, or -`zarr.dtype` still works but now raises a `ZarrDeprecationWarning`. The remaining -types and functions are simply re-exported from the listed public module. diff --git a/changes/4053.doc.md b/changes/4053.doc.md deleted file mode 100644 index 3220b2b4ec..0000000000 --- a/changes/4053.doc.md +++ /dev/null @@ -1 +0,0 @@ -Document a self-merge policy in the contributor guide, describing when a core developer may merge their own pull request without a second reviewer and which changes warrant more caution. diff --git a/changes/4054.misc.md b/changes/4054.misc.md deleted file mode 100644 index 4aa9435b84..0000000000 --- a/changes/4054.misc.md +++ /dev/null @@ -1 +0,0 @@ -Add Hypothesis property tests for block and mask indexing (`test_block_indexing`, `test_mask_indexing`), along with a `block_indices` strategy in `zarr.testing.strategies`. These extend the existing randomized indexing coverage (basic, orthogonal, and vectorized) to the block and mask selection methods. diff --git a/changes/4059.bugfix.md b/changes/4059.bugfix.md deleted file mode 100644 index 16fb582f65..0000000000 --- a/changes/4059.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Prevents mutation of the attributes dict provided by the user by copying them instead of keeping the reference diff --git a/changes/4073.misc.md b/changes/4073.misc.md deleted file mode 100644 index bcc6b8281e..0000000000 --- a/changes/4073.misc.md +++ /dev/null @@ -1 +0,0 @@ -Extend the `test_block_indexing` Hypothesis property test to cover rectilinear chunk grids and sharded regular grids, and generalize the `block_indices` strategy in `zarr.testing.strategies` to build its array-space oracle from cumulative chunk offsets (`chunk_sizes` parameter) instead of a uniform chunk size. diff --git a/changes/4074.bugfix.md b/changes/4074.bugfix.md deleted file mode 100644 index d55a52b887..0000000000 --- a/changes/4074.bugfix.md +++ /dev/null @@ -1,7 +0,0 @@ -Fixed several storage and codec bugs: - -- Reading a value with a `SuffixByteRequest` larger than the value now correctly returns the whole value (matching HTTP `bytes=-N` suffix-range semantics), instead of silently returning incorrect data for `MemoryStore`. -- `LoggingStore.get_partial_values` and `FsspecStore.get_partial_values` no longer return empty results when `key_ranges` is passed as a one-shot iterable (e.g. a generator). -- `Store.getsize_prefix` no longer over-counts sibling keys that merely share a string prefix (e.g. `getsize_prefix("foo")` no longer includes keys under `foobar/`). -- `ZipStore.close()` no longer raises `AttributeError` when the store was created but never opened (including when used as a context manager without any I/O). -- `codecs_from_list` now raises a descriptive `TypeError` when a `BytesBytesCodec` immediately follows an `ArrayArrayCodec`, instead of a misleading "Required ArrayBytesCodec was not found" `ValueError`. diff --git a/changes/4086.misc.md b/changes/4086.misc.md deleted file mode 100644 index 2564cf9ea1..0000000000 --- a/changes/4086.misc.md +++ /dev/null @@ -1 +0,0 @@ -Fix `test_multiprocessing[fork]` failing on Python 3.15, where `os.fork()` in a multi-threaded process emits a `DeprecationWarning` that the test suite promotes to an error. diff --git a/changes/4116.bugfix.md b/changes/4116.bugfix.md deleted file mode 100644 index e99eac8c95..0000000000 --- a/changes/4116.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed writing Fortran-ordered (F-contiguous) arrays through the variable-length string and bytes codecs and through numcodecs array-array filters such as `Delta`, `FixedScaleOffset` and `PackBits`. Chunks are now passed to numcodecs as C-contiguous arrays, so elements are no longer stored in transposed order. diff --git a/changes/4132.doc.md b/changes/4132.doc.md deleted file mode 100644 index 3325c82f43..0000000000 --- a/changes/4132.doc.md +++ /dev/null @@ -1,6 +0,0 @@ -Fixed many documentation errors found in a full review of the user guide, including -prose contradicted by rendered example output on the performance page, invisible -code blocks, an incorrect S3 example, stale "not yet implemented" claims in the -v3 migration guide, and undocumented optional dependency groups. Also improved -navigation order, cross-linking between pages, and coverage of group member -enumeration, bulk attribute updates, and the `use_consolidated` keyword. diff --git a/changes/4133.doc.md b/changes/4133.doc.md deleted file mode 100644 index 9690318b46..0000000000 --- a/changes/4133.doc.md +++ /dev/null @@ -1 +0,0 @@ -Fixed the documented default of ``max_age_seconds`` in the ``CacheStore`` docstring: the default is ``"infinity"`` (no expiration), not ``None``, which is rejected. Also noted that ``cache_store`` must support deletes. diff --git a/changes/4138.misc.md b/changes/4138.misc.md deleted file mode 100644 index 2cf0365bb9..0000000000 --- a/changes/4138.misc.md +++ /dev/null @@ -1 +0,0 @@ -Pass a list rather than a generator to `pytest.mark.parametrize` in `tests/test_docs.py`, so the test suite collects cleanly under pytest 9.1 (which deprecates non-Collection iterables and, under `filterwarnings = error`, turns that into a collection error). diff --git a/changes/4141.bugfix.md b/changes/4141.bugfix.md deleted file mode 100644 index 6a132da3f5..0000000000 --- a/changes/4141.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fix silent byte-order corruption for structured dtypes with the `bytes` codec: multi-byte fields are now byte-swapped to the codec's configured `endian` on write and decoded honoring it on read, so non-native-endian structured data (e.g. big-endian fields, as produced by virtual references to external data) round-trips correctly. diff --git a/docs/release-notes.md b/docs/release-notes.md index 3ba870ac92..e85137f2b6 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -2,6 +2,162 @@ +## 3.3.0 (2026-07-15) + +### Features + +- Optimizes reading multiple chunks from a shard. Serial calls to `Store.get()` + in the sharding codec have been replaced with a single call to + `Store.get_ranges()`, which coalesces nearby byte ranges and fetches them + concurrently. ([#3004](https://github.com/zarr-developers/zarr-python/issues/3004)) +- Added a `subchunk_write_order` option to `ShardingCodec` to control the physical order of subchunks within a shard. Supported values are `morton`, `unordered`, `lexicographic`, and `colexicographic`. `unordered` makes no guarantee about subchunk layout. This setting affects only on-disk layout, not the data read back, and is not persisted in array metadata: it applies per codec instance and is not recovered when reopening a sharded array. ([#3826](https://github.com/zarr-developers/zarr-python/issues/3826)) +- Added `SyncByteGetter` and `SyncByteSetter` runtime-checkable protocols and a `get_ranges_sync` method on the `Store` ABC. These let custom byte getters/setters opt into the synchronous codec pipeline's fast path for in-memory IO, which the sharding codec uses for its inner chunks. ([#3885](https://github.com/zarr-developers/zarr-python/issues/3885)) +- Added `FusedCodecPipeline`, an opt-in codec pipeline that runs codec compute synchronously and in bulk (avoiding the per-chunk async scheduling overhead of the default `BatchedCodecPipeline`), giving large speedups for sharded arrays (up to ~24x writes / ~14x reads on many-chunks-per-shard layouts, more with compression) and no regressions on compute-bound workloads. The default `BatchedCodecPipeline` is unchanged for standard configurations, so existing code keeps working unless you opt in; enable the new pipeline with `zarr.config.set({"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"})`. ([#3885](https://github.com/zarr-developers/zarr-python/issues/3885)) +- Add `zarr.abc.store.Store.get_ranges` for concurrent, coalesced multi-range reads from a single key. The method is defined on the `Store` ABC with a default implementation built on `Store.get`, so every store inherits a working version; stores with native multi-range backends (e.g. `FsspecStore`) can override for efficiency. Coalescing knobs (`max_concurrency`, `max_gap_bytes`, `max_coalesced_bytes`) are passed as keyword arguments to `get_ranges`. Failures from underlying fetches surface as a `BaseExceptionGroup` (PEP 654); callers should use `except*` to filter for specific exception types such as `FileNotFoundError`. ([#3925](https://github.com/zarr-developers/zarr-python/issues/3925)) +- Two new fields on `ArrayConfig` control how the sharding codec coalesces partial-shard reads: `sharding_coalesce_max_gap_bytes` (default 1 MiB) and `sharding_coalesce_max_bytes` (default 16 MiB). When reading multiple chunks from the same shard, nearby byte ranges are merged into a single request to the store if separated by no more than `sharding_coalesce_max_gap_bytes` and the merged read stays within `sharding_coalesce_max_bytes`. Defaults are seeded from the matching `array.sharding_coalesce_max_gap_bytes` / `array.sharding_coalesce_max_bytes` keys in [`zarr.config`][] at array-creation time, and can be overridden per array by passing `config={...}` to [`zarr.create_array`][]. ([#3987](https://github.com/zarr-developers/zarr-python/issues/3987)) + +### Bugfixes + +- Stop emitting an `UnstableSpecificationWarning` when serializing the `struct` data type to Zarr V3 metadata. The `struct` data type now has a stable Zarr V3 specification. The legacy `structured` alias and the unspecified `null_terminated_bytes`, `raw_bytes`, and `variable_length_bytes` data types continue to warn. ([#202](https://github.com/zarr-developers/zarr-python/issues/202)) +- Fix equality comparison of `ArrayV2Metadata` and `ArrayV3Metadata` objects with a + `NaN` fill value. Such objects are now compared by their JSON-serialized form, so two + otherwise-identical metadata objects with a `NaN` (or infinite) fill value compare equal. ([#2929](https://github.com/zarr-developers/zarr-python/issues/2929)) +- Fixed `BytesCodec.from_dict` so that `BytesCodec` instances roundtrip to / from + their dict representation. `BytesCodec.from_dict` now interprets a missing + `endian` configuration as `endian=None` (matching what `BytesCodec.to_dict` + emits), instead of falling back to the system's native byte order. ([#3417](https://github.com/zarr-developers/zarr-python/issues/3417)) +- Fixed `save_array`, `Group.__setitem__`, and `load` for 0-dimensional arrays. ([#3469](https://github.com/zarr-developers/zarr-python/issues/3469)) +- Fixed inner-codec spec evolution for sharded arrays. The sharding codec now threads the array spec through its inner codec chain when evolving codecs, so a codec that changes the dtype upstream of `BytesCodec` no longer leaves the inner chain evolved against the wrong spec (which previously failed at decode time). This runs on the default `BatchedCodecPipeline` as well. Standard inner chains (`[BytesCodec]`, `[BytesCodec, ZstdCodec]`, transpose + bytes) are byte-identical to before. Restores the behavior of #2179. ([#3885](https://github.com/zarr-developers/zarr-python/issues/3885)) +- Make chunk normalization properly handle `-1` as a compact representation of the + length of an entire axis. Reject several previously-accepted but ill-defined + chunk specifications: `chunks=True` (previously silently produced size-1 chunks), + chunk tuples shorter than the array's number of dimensions (previously padded to + the array's shape), and `None` as a per-dimension chunk size. These all now + raise informative errors. Also fix chunk handling for 0-length array dimensions, + and add explicit rejection of 0-length chunks. ([#3899](https://github.com/zarr-developers/zarr-python/issues/3899)) +- Handle missing consolidated metadata in leaf Group nodes. ([#3954](https://github.com/zarr-developers/zarr-python/issues/3954)) +- Corrected the JSON type definitions for the `numpy.datetime64` and + `numpy.timedelta64` data types in Zarr V3 metadata: the `configuration` object + (holding `unit` and `scale_factor`) is now required, matching the published + specifications for these data types. Also updated the specification links in + the docstrings to point to the zarr-extensions repository. ([#3955](https://github.com/zarr-developers/zarr-python/issues/3955)) +- Fixed writing to 0-dimensional arrays that use the sharding codec. Previously + assigning to a 0-dimensional sharded array raised an error. ([#3966](https://github.com/zarr-developers/zarr-python/issues/3966)) +- Fix flaky stateful test bookkeeping when `delete_dir` matches string prefixes instead of true directory descendants. Previously a path such as `6/faNT…` could be incorrectly removed when deleting `6/f`. (See [issue #3977](https://github.com/zarr-developers/zarr-python/issues/3977).) ([#3977](https://github.com/zarr-developers/zarr-python/issues/3977)) +- `FsspecStore.from_url()` and `from_mapper()` now close the async filesystem + they create when `store.close()` is called. Previously the underlying aiohttp + `ClientSession` was left open until garbage collection, producing + `"Unclosed client session"` `ResourceWarning`s from aiohttp. + + The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when + `FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` + when a sync→async conversion was performed). When `_owns_fs` is ``True``, + `store.close()` calls the new `_close_fs()` helper, which invokes + `fs.set_session()` and closes the returned client. Callers who supply their own + filesystem instance to `FsspecStore()` directly remain responsible for its + lifecycle; `_owns_fs` is ``False`` for those stores. + + **Scope note**: This fix closes the S3 client session that is active at the time + `store.close()` is called. Some S3-backed filesystem implementations (e.g. + s3fs with ``cache_regions=True``) may internally refresh and replace their + client during I/O operations, abandoning prior sessions before ``store.close()`` + is invoked. Those intermediate sessions are outside the scope of this fix and + are an issue in the upstream filesystem library. ([#4003](https://github.com/zarr-developers/zarr-python/issues/4003)) +- Fixed an invalid `zarr.create_array` example in the quick-start documentation (it passed an unsupported `mode` argument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested. ([#4016](https://github.com/zarr-developers/zarr-python/issues/4016)) +- Fixed `ObjectStore.list_dir` for object-store listings that include a directory-marker object matching the requested non-root prefix. ([#4032](https://github.com/zarr-developers/zarr-python/issues/4032)) +- Prevents mutation of the attributes dict provided by the user by copying them instead of keeping the reference ([#4059](https://github.com/zarr-developers/zarr-python/issues/4059)) +- Fixed several storage and codec bugs: + + - Reading a value with a `SuffixByteRequest` larger than the value now correctly returns the whole value (matching HTTP `bytes=-N` suffix-range semantics), instead of silently returning incorrect data for `MemoryStore`. + - `LoggingStore.get_partial_values` and `FsspecStore.get_partial_values` no longer return empty results when `key_ranges` is passed as a one-shot iterable (e.g. a generator). + - `Store.getsize_prefix` no longer over-counts sibling keys that merely share a string prefix (e.g. `getsize_prefix("foo")` no longer includes keys under `foobar/`). + - `ZipStore.close()` no longer raises `AttributeError` when the store was created but never opened (including when used as a context manager without any I/O). + - `codecs_from_list` now raises a descriptive `TypeError` when a `BytesBytesCodec` immediately follows an `ArrayArrayCodec`, instead of a misleading "Required ArrayBytesCodec was not found" `ValueError`. + + ([#4074](https://github.com/zarr-developers/zarr-python/issues/4074)) +- Fixed writing Fortran-ordered (F-contiguous) arrays through the variable-length string and bytes codecs and through numcodecs array-array filters such as `Delta`, `FixedScaleOffset` and `PackBits`. Chunks are now passed to numcodecs as C-contiguous arrays, so elements are no longer stored in transposed order. ([#4116](https://github.com/zarr-developers/zarr-python/issues/4116)) +- Fix silent byte-order corruption for structured dtypes with the `bytes` codec: multi-byte fields are now byte-swapped to the codec's configured `endian` on write and decoded honoring it on read, so non-native-endian structured data (e.g. big-endian fields, as produced by virtual references to external data) round-trips correctly. ([#4141](https://github.com/zarr-developers/zarr-python/issues/4141)) + +### Improved Documentation + +- Document the changes to `zarr.errors` in the 3.0 migration guide, including the removal of v2 exception classes and the introduction of `NodeNotFoundError`. ([#3009](https://github.com/zarr-developers/zarr-python/issues/3009)) +- Clarify the difference between `zarr.load` and `zarr.open` in their docstrings. + `load` eagerly reads data into an in-memory array, while `open` returns a + lazy `Array` or `Group` backed by the store, with `See Also` cross-references + linking the two. ([#3984](https://github.com/zarr-developers/zarr-python/issues/3984)) +- Updated the custom dtype example in `examples/custom_dtype/custom_dtype.py` to + use only the public API, eliminating all non-public imports, illustrating what + users should do. + + To better support this, the following types and functions were made available + from public modules: + + | Type/Function | Non-public module | Public module | + | ------------------------- | ------------------------ | ------------- | + | `DataTypeValidationError` | `zarr.core.dtype.common` | `zarr.errors` | + | `JSON` | `zarr.core.common` | `zarr.types` | + | `ZarrFormat` | `zarr.core.common` | `zarr.types` | + | `DTypeConfig_V2` | `zarr.core.dtype.common` | `zarr.types` | + | `DTypeJSON` | `zarr.core.dtype.common` | `zarr.types` | + | `DTypeSpec_V2` | `zarr.core.dtype.common` | `zarr.dtype` | + | `check_dtype_spec_v2` | `zarr.core.dtype.common` | `zarr.dtype` | + + `DataTypeValidationError` was *moved* to `zarr.errors`. Importing it from + `zarr.core.dtype.common` (its original location), `zarr.core.dtype`, or + `zarr.dtype` still works but now raises a `ZarrDeprecationWarning`. The remaining + types and functions are simply re-exported from the listed public module. ([#4052](https://github.com/zarr-developers/zarr-python/issues/4052)) +- Document a self-merge policy in the contributor guide, describing when a core developer may merge their own pull request without a second reviewer and which changes warrant more caution. ([#4053](https://github.com/zarr-developers/zarr-python/issues/4053)) +- Fixed many documentation errors found in a full review of the user guide, including + prose contradicted by rendered example output on the performance page, invisible + code blocks, an incorrect S3 example, stale "not yet implemented" claims in the + v3 migration guide, and undocumented optional dependency groups. Also improved + navigation order, cross-linking between pages, and coverage of group member + enumeration, bulk attribute updates, and the `use_consolidated` keyword. ([#4132](https://github.com/zarr-developers/zarr-python/issues/4132)) +- Fixed the documented default of ``max_age_seconds`` in the ``CacheStore`` docstring: the default is ``"infinity"`` (no expiration), not ``None``, which is rejected. Also noted that ``cache_store`` must support deletes. ([#4133](https://github.com/zarr-developers/zarr-python/issues/4133)) + +### Deprecations and Removals + +- The ``BloscShuffle`` and ``BloscCname`` enums (``zarr.codecs.BloscShuffle``, + ``zarr.codecs.BloscCname``) are now deprecated. Pass the equivalent literal + string (e.g. ``"zstd"``, ``"bitshuffle"``) when constructing a ``BloscCodec``. + The enum classes remain importable but emit ``DeprecationWarning`` on member + access, and will be removed in a future release. ``BloscCodec.cname`` and + ``BloscCodec.shuffle`` are now plain strings rather than enum members. + + Additional renames in ``zarr.codecs.blosc`` from the same change: the type + aliases ``Shuffle`` and ``CName`` are now ``BloscShuffleLiteral`` and + ``BloscCnameLiteral``, the constant ``SHUFFLE`` is now ``BLOSC_SHUFFLE`` + (with a new ``BLOSC_CNAME`` alongside it), and ``BloscShuffle.from_int`` + now returns a literal string rather than an enum member. ([#3963](https://github.com/zarr-developers/zarr-python/issues/3963)) +- The ``Endian`` (``zarr.codecs.bytes.Endian``) and ``ShardingCodecIndexLocation`` + (``zarr.codecs.ShardingCodecIndexLocation``) enums are now deprecated. Pass the + equivalent literal string instead (e.g. ``"little"`` / ``"big"``, ``"start"`` / + ``"end"``). The enum classes remain importable but emit ``DeprecationWarning`` + on member access, and will be removed in a future release. ``BytesCodec.endian`` + and ``ShardingCodec.index_location`` are now plain strings rather than enum + members. + + Two follow-on changes from this deprecation: + + - ``NDBuffer.byteorder`` now returns a literal string (``"little"`` or + ``"big"``) rather than an ``Endian`` member. Subclasses overriding this + property should update their return type. + - The module-level binding ``zarr.codecs.bytes.default_system_endian`` was + removed. ``BytesCodec()`` continues to default to ``sys.byteorder``; + external callers that imported ``default_system_endian`` should use + ``sys.byteorder`` directly. + + Additionally, the module-level function ``zarr.codecs.sharding.parse_index_location`` + was made private as part of this change. + + ([#3968](https://github.com/zarr-developers/zarr-python/issues/3968)) +- Removed the NumPy 1.x implementation of the `VariableLengthUTF8` data type because NumPy 1.x is no longer supported under [SPEC0](https://scientific-python.org/specs/spec-0000/). ([#3973](https://github.com/zarr-developers/zarr-python/issues/3973)) + +### Misc + +- [#214](https://github.com/zarr-developers/zarr-python/issues/214), [#215](https://github.com/zarr-developers/zarr-python/issues/215), [#3908](https://github.com/zarr-developers/zarr-python/issues/3908), [#3972](https://github.com/zarr-developers/zarr-python/issues/3972), [#3975](https://github.com/zarr-developers/zarr-python/issues/3975), [#3979](https://github.com/zarr-developers/zarr-python/issues/3979), [#3990](https://github.com/zarr-developers/zarr-python/issues/3990), [#3998](https://github.com/zarr-developers/zarr-python/issues/3998), [#4000](https://github.com/zarr-developers/zarr-python/issues/4000), [#4001](https://github.com/zarr-developers/zarr-python/issues/4001), [#4046](https://github.com/zarr-developers/zarr-python/issues/4046), [#4054](https://github.com/zarr-developers/zarr-python/issues/4054), [#4073](https://github.com/zarr-developers/zarr-python/issues/4073), [#4086](https://github.com/zarr-developers/zarr-python/issues/4086), [#4138](https://github.com/zarr-developers/zarr-python/issues/4138) + + ## 3.2.1 (2026-05-05) ### Bugfixes From 23d96d38ae1cb1b722681f16eec7af171890b367 Mon Sep 17 00:00:00 2001 From: Max Jones <14077947+maxrjones@users.noreply.github.com> Date: Thu, 16 Jul 2026 11:30:12 -0400 Subject: [PATCH 3/8] chore: CI worfklows for linting docs (#4076) * chore: CI worfklows for missing docs and RST docstrings * Add linter for docs * Add link checking workflow * fix links * fix: apply suggestions from code review * refactor: readability improvements * chore: switch lychee action to opening issue on failure --------- Co-authored-by: Davis Bennett --- .github/workflows/docs.yml | 5 + .github/workflows/links.yml | 32 +++ .markdownlint-cli2.jsonc | 54 +++++ .pre-commit-config.yaml | 10 + ci/check_documented_exports.py | 161 +++++++++++++ ci/lint_docs.py | 284 +++++++++++++++++++++++ docs/contributing.md | 23 +- docs/index.md | 14 +- docs/quick-start.md | 2 - docs/release-notes.md | 2 + docs/user-guide/attributes.md | 1 + docs/user-guide/consolidated_metadata.md | 1 - docs/user-guide/data_types.md | 7 + docs/user-guide/experimental.md | 1 - docs/user-guide/extending.md | 1 + docs/user-guide/groups.md | 1 - docs/user-guide/performance.md | 6 +- docs/user-guide/storage.md | 8 +- lychee.toml | 20 ++ src/zarr/core/metadata/v2.py | 2 +- src/zarr/testing/store.py | 6 +- 21 files changed, 608 insertions(+), 33 deletions(-) create mode 100644 .github/workflows/links.yml create mode 100644 .markdownlint-cli2.jsonc create mode 100644 ci/check_documented_exports.py create mode 100644 ci/lint_docs.py create mode 100644 lychee.toml diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 59eca8d9b9..c72b493b12 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -24,6 +24,11 @@ jobs: persist-credentials: false - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 - run: uv sync --group docs + # Fast source-level guards that need no built site, so they run before the (slower) + # build for a quick failure: every public export is in the API reference, and no + # docstring/Markdown carries reStructuredText markup that MkDocs won't render. + - run: uv run python ci/check_documented_exports.py docs/api + - run: uv run python ci/lint_docs.py # --strict turns warnings into errors, so a docs code block that fails to execute # at build time (e.g. a non-exec python fence disrupting a later exec="true" block) # fails CI instead of merging as a silent warning. diff --git a/.github/workflows/links.yml b/.github/workflows/links.yml new file mode 100644 index 0000000000..a53de8da83 --- /dev/null +++ b/.github/workflows/links.yml @@ -0,0 +1,32 @@ +name: Check links + +on: + repository_dispatch: + workflow_dispatch: + # pull_request: + schedule: + - cron: "00 18 * * *" + +jobs: + linkChecker: + runs-on: ubuntu-latest + permissions: + issues: write # required for peter-evans/create-issue-from-file + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + with: + persist-credentials: false + + - name: Link Checker + id: lychee + uses: lycheeverse/lychee-action@8646ba30535128ac92d33dfc9133794bfdd9b411 # v2.8.0 + with: + fail: false + + - name: Create Issue From File + if: steps.lychee.outputs.exit_code != 0 + uses: peter-evans/create-issue-from-file@fca9117c27cdc29c6c4db3b86c48e4115a786710 # v6.0.0 + with: + title: Link Checker Report + content-filepath: ./lychee/out.md + labels: report, automated issue diff --git a/.markdownlint-cli2.jsonc b/.markdownlint-cli2.jsonc new file mode 100644 index 0000000000..3dfdf96856 --- /dev/null +++ b/.markdownlint-cli2.jsonc @@ -0,0 +1,54 @@ +// markdownlint-cli2 configuration for zarr-python docs. +// +// We keep the rules that catch real rendering/structure problems and disable those that +// are pure style, conflict with house conventions, or fire false positives against our +// MkDocs/mkdocstrings + pymdownx toolchain. Complementary, not overlapping, with +// ci/lint_docs.py (RST residue + list-breaking fences) and `mkdocs build --strict`. +{ + "config": { + "default": true, + + // House style: Markdown paragraphs are single unwrapped lines, so line length is not + // a meaningful constraint. + "MD013": false, + + // Purely stylistic marker/emphasis choices -- not worth the churn across existing docs. + "MD004": false, // ul bullet style (-, *, +) + "MD007": false, // ul indentation width + "MD050": false, // strong (bold) style + "MD035": false, // hr style + + // False positives from our toolchain: + // mkdocstrings cross-refs `[`X`][zarr.X]` read as undefined reference links (MD052); + // pymdownx.magiclink auto-links bare URLs (MD034); + // md_in_html lets us embed intentional raw HTML (MD033); + // generated/included files (api stubs, snippets) need not open with an H1 (MD041). + "MD052": false, + "MD034": false, + "MD033": false, + "MD041": false, + + // Duplicate headings are legitimate under different sections (e.g. repeated + // "Documentation"); only flag true sibling duplicates. + "MD024": { "siblings_only": true }, + + // Opinionated table/link/command rules with low value for these docs. + "MD055": false, // table pipe style + "MD060": false, // table column style + "MD059": false, // "descriptive" link text (no "click here") + "MD014": false, // $ before commands without shown output + + // markdownlint does not understand MkDocs `!!!` admonitions, so it reads their + // 4-space-indented bodies as indented code blocks and flags them (and, via inferred + // file style, flags real fenced blocks too). Cannot coexist with our admonitions. + "MD046": false // code block style (fenced vs indented) + // Kept on (structural / real rendering bugs): MD012 (multiple blanks), MD022/MD031/MD032 + // (blanks around headings/fences/lists), MD025 (single H1), MD029 (ordered-list prefix), + // MD040 (fenced code language), MD042 (empty links), + // MD047 (trailing newline), MD056 (table column count), among others. + }, + "globs": ["docs/**/*.md"], + "ignores": [ + "docs/api/**" // mkdocstrings stubs (`::: zarr.X`) + ] +} diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index e08c98abca..57a1d0d4f7 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -31,6 +31,16 @@ repos: - id: check-yaml exclude: mkdocs.yml - id: trailing-whitespace + - repo: https://github.com/DavidAnson/markdownlint-cli2 + rev: v0.22.1 + hooks: + # Markdown structure/hygiene. Rule selection and ignores are in + # .markdownlint-cli2.jsonc; complements ci/lint_docs.py (RST residue, + # list-breaking fences) and `mkdocs build --strict`. Scoped to docs/ to + # match the config's globs (pre-commit passes filenames, which would + # otherwise override that scoping and lint all repo Markdown). + - id: markdownlint-cli2 + files: ^docs/ - repo: local hooks: - id: mypy diff --git a/ci/check_documented_exports.py b/ci/check_documented_exports.py new file mode 100644 index 0000000000..0772954399 --- /dev/null +++ b/ci/check_documented_exports.py @@ -0,0 +1,161 @@ +"""Check that every public top-level export is in the API reference. + +The API reference is authored as explicit mkdocstrings directives (``::: target``) +under ``docs/api/`` -- one per documented symbol -- rather than autodoc, so a newly +added ``zarr.__all__`` entry will not appear in the docs until someone writes a page +for it (or it becomes a rendered member of an already-documented module). This script +catches that gap: it resolves every ``:::`` target, expands module directives into the +members they render (honoring ``members: false``), and asserts each name in +``zarr.__all__`` resolves to a documented object. + +Usage: + python ci/check_documented_exports.py [API_DOCS_DIR] + +API_DOCS_DIR defaults to the repo-root ``docs/api``. Exits non-zero (and prints the +undocumented exports to stderr) if any public export is missing from the reference. +""" + +from __future__ import annotations + +import importlib +import re +import sys +from pathlib import Path +from types import ModuleType +from typing import TYPE_CHECKING, Any + +import zarr + +if TYPE_CHECKING: + from collections.abc import Iterator + +REPO_ROOT = Path(__file__).parent.parent.resolve() +DEFAULT_API_DOCS_ROOT = REPO_ROOT / "docs" / "api" + +# Names in zarr.__all__ that are intentionally absent from the API reference. +# Keep this list short and justified -- it is the only escape hatch from the guard. +EXEMPT_EXPORTS = { + "__version__", # version string, not an API symbol + "print_debug_info", # debugging helper, deliberately not in the reference +} + +# A mkdocstrings autodoc directive: `::: some.dotted.target` at the start of a line. +DIRECTIVE_RE = re.compile(r"^:::[ \t]+(?P\S+)") +# `members: false` (or `members: []`) within a directive's option block disables +# rendering of a module's members. +MEMBERS_DISABLED_RE = re.compile(r"^\s+members:\s*(false|\[\s*\])\s*$") + + +def resolve(target: str) -> Any: + """Resolve a `:::` target (a dotted path) to the Python object it documents.""" + try: + return importlib.import_module(target) + except ImportError: + pass + module_path, _, attr = target.rpartition(".") + try: + return getattr(importlib.import_module(module_path), attr) + except (ImportError, AttributeError): + return None + + +def iter_directives(text: str) -> Iterator[tuple[str, bool]]: + """Yield ``(target, members_enabled)`` for each ``:::`` directive in ``text``. + + The file is split into lines once; for each directive we scan its indented option + block -- stopping at the first non-indented line, which ends the block -- so options + belonging to a later directive are never consulted. ``members_enabled`` is False when + that block sets ``members: false`` (or ``members: []``).""" + lines = text.splitlines() + i = 0 + while i < len(lines): + match = DIRECTIVE_RE.match(lines[i]) + if match is None: + i += 1 + continue + members_enabled = True + i += 1 + while i < len(lines): + line = lines[i] + if line.strip() == "": + i += 1 + continue + if not line.startswith((" ", "\t")): + break # non-indented line: end of this directive's option block + if MEMBERS_DISABLED_RE.match(line): + members_enabled = False + i += 1 + yield match.group("target"), members_enabled + + +def module_member_ids(module: ModuleType) -> Iterator[int]: + """Yield the id() of each public member a module directive renders. + + The rendered members are the module's ``__all__`` if defined, else its public + (non-underscore) attributes.""" + member_names = getattr(module, "__all__", None) or [ + name for name in dir(module) if not name.startswith("_") + ] + for name in member_names: + member = getattr(module, name, None) + if member is not None: + yield id(member) + + +def documented_object_ids(api_docs_root: Path) -> set[int]: + """Collect the id()s of every object rendered by a `:::` directive under api_docs_root. + + A directive pointing at an object documents that object. A directive pointing at a + module documents the module's public members unless the directive sets + ``members: false``.""" + documented: set[int] = set() + for md_file in sorted(api_docs_root.rglob("*.md")): + for target, members_enabled in iter_directives(md_file.read_text(encoding="utf-8")): + obj = resolve(target) + if obj is None: + continue + documented.add(id(obj)) + if isinstance(obj, ModuleType) and members_enabled: + documented.update(module_member_ids(obj)) + return documented + + +def find_undocumented_exports(api_docs_root: Path) -> list[str]: + documented = documented_object_ids(api_docs_root) + return sorted( + name + for name in zarr.__all__ + if name not in EXEMPT_EXPORTS and id(getattr(zarr, name)) not in documented + ) + + +def main() -> int: + args = sys.argv[1:] + api_docs_root = Path(args[0]).resolve() if args else DEFAULT_API_DOCS_ROOT + if not api_docs_root.exists(): + print(f"{api_docs_root} does not exist.", file=sys.stderr) + return 1 + + missing = find_undocumented_exports(api_docs_root) + if not missing: + print(f"All {len(zarr.__all__)} public exports are documented.") + return 0 + + print( + f"Found {len(missing)} public export(s) in zarr.__all__ missing from the API " + "reference (docs/api/):\n", + file=sys.stderr, + ) + for name in missing: + print(f" - zarr.{name}", file=sys.stderr) + print( + "\nAdd a `::: zarr.` page under docs/api/zarr/ (and register it in " + "mkdocs.yml and docs/api/zarr/index.md), or -- if the export is intentionally " + "undocumented -- add it to EXEMPT_EXPORTS in this script with a reason.", + file=sys.stderr, + ) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/ci/lint_docs.py b/ci/lint_docs.py new file mode 100644 index 0000000000..ff7b56649b --- /dev/null +++ b/ci/lint_docs.py @@ -0,0 +1,284 @@ +"""Lint docstrings and Markdown for reStructuredText markup that won't render. + +This project renders API docs with mkdocstrings (``docstring_style: numpy``) and prose +with MkDocs + Markdown -- not Sphinx/reStructuredText. RST constructs that survive from +older docstrings (or muscle memory) are not interpreted: a Sphinx role passes through as +literal text instead of becoming a link, an ``.. note::`` directive renders as a stray +line, and a ``:param:`` field list never becomes a documented parameter. + +Crucially, none of this is caught by the rest of the docs CI. ``mkdocs build --strict`` +sees the residue as ordinary prose (no warning), and ``ci/check_unlinked_types.py`` only +finds cross-references mkdocstrings *attempted* to resolve -- a raw ``:class:`` role is +never attempted, so it leaves no unlinked-type span. This linter fills that gap with a +fast, source-level check that needs no docs build. + +Checks fall into two groups -- RST markup that silently fails under MkDocs/mkdocstrings, +and a Markdown structural problem that renders as valid-but-wrong HTML (so `mkdocs build` +emits no warning): + + sphinx-role :class:`X`, :func:`X`, :py:meth:`X` -> [`X`][zarr.X] + rst-directive .. note:: / .. code-block:: python -> MkDocs admonition / fenced code + rst-field :param x:, :returns:, :rtype: -> numpydoc Parameters/Returns/Raises + rst-link `text `_ -> [text](https://example) + list-break unindented code fence between list items -> indent the fence under its item + +The ``list-break`` check catches a fenced code block at column 0 placed *between* two list +items: because the fence is not indented into the preceding item, Markdown ends the list at +the fence and the following item starts a fresh list -- renumbering an ordered list (1, 1, 2 +instead of 1, 2, 3) or breaking the grouping/spacing of any list. markdownlint's MD029 only +notices this for sequentially-numbered ordered lists; lazily-numbered (1., 1.) and unordered +lists slip past it, so this structural check covers the gap. + +Usage: + python ci/lint_docs.py [PATH ...] + +PATH defaults to the repo-root ``src/zarr`` and ``docs``. Each PATH may be a file or a +directory (directories are searched for ``*.py`` and ``*.md``). Exits non-zero if any +issues are found. +""" + +from __future__ import annotations + +import ast +import re +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import NamedTuple + +REPO_ROOT = Path(__file__).parent.parent.resolve() +DEFAULT_PATHS = (REPO_ROOT / "src" / "zarr", REPO_ROOT / "docs") + +# A Sphinx interpreted-text role: an optional domain, a role name, then a backtick +# target -- e.g. :class:`Foo` or :py:meth:`Foo.bar`. Requires the trailing backtick so +# plain "::" (RST literal markers, time strings, mkdocs-material :icon: shortcodes) and +# URLs ("https://") never match. +SPHINX_ROLE = re.compile(r":[a-zA-Z_]\w*(?::[a-zA-Z_]\w*)?:`[^`\n]+`") + +# An RST directive line: ".. name::" (with or without an argument after it). RST hyperlink +# targets (".. _label:") and comments (".. text") lack the "::" and are not flagged. +RST_DIRECTIVE = re.compile(r"^\s*\.\.[ \t]+[\w-]+::") + +# An RST field-list entry used for docstring fields. The role names above (class, func, +# ...) are deliberately excluded so a role is reported as a role, not a field. +RST_FIELD = re.compile( + r"^\s*:(param|parameter|arg|argument|key|keyword|kwarg|type|returns?|rtype" + r"|raises?|except|exception|yields?|ytype|var|cvar|ivar)\b[^:]*:" +) + +# An RST external hyperlink: `text `_ +RST_LINK = re.compile(r"`[^`\n]+\n]+>`_") + +# A list item at column 0: an ordered marker (1. / 1)) or a bullet (-, *, +) followed by +# whitespace and content. Leading-whitespace (nested/continuation) lines are intentionally +# not matched -- the list-break check only fires on top-level items. +LIST_ITEM = re.compile(r"^(?:\d+[.)]|[-*+])\s+\S") + + +class Check(NamedTuple): + """One docs-residue check: its category, the line pattern that flags it (None for a + structural check matched outside ``_scan_line``), and the user-facing remediation + shown by ``main()``. Keeping ``example``/``fix`` here makes this the single source for + the help text, so adding a check can't leave the help out of date.""" + + category: str + pattern: re.Pattern[str] | None + example: str + fix: str + + +# ``list-break`` carries no pattern -- it is detected structurally in find_list_breaking_fences, +# not by scanning a single line -- but it appears here so it shares the remediation help. +CHECKS = ( + Check("sphinx-role", SPHINX_ROLE, ":class:`X`", "[`X`][zarr.X]"), + Check("rst-directive", RST_DIRECTIVE, ".. note::", "MkDocs admonition (!!! note)"), + Check("rst-field", RST_FIELD, ":param x:", "numpydoc Parameters/Returns/Raises section"), + Check("rst-link", RST_LINK, "`text `_", "[text](url)"), + Check("list-break", None, "fence between items", "indent the fence 4 spaces to nest it"), +) + + +@dataclass(frozen=True) +class Finding: + path: Path + line: int + category: str + snippet: str + + def format(self) -> str: + try: + location: Path | str = self.path.relative_to(REPO_ROOT) + except ValueError: + location = self.path + return f" {location}:{self.line}: [{self.category}] {self.snippet.strip()}" + + +def _scan_line(text: str) -> list[str]: + """Return every RST-residue category found in a single line (a line can carry more + than one, e.g. a role and an external link).""" + return [c.category for c in CHECKS if c.pattern is not None and c.pattern.search(text)] + + +def lint_python(path: Path) -> list[Finding]: + """Scan the docstrings (module, classes, functions) of a Python file. + + Only docstrings are checked -- they are what mkdocstrings renders -- so RST-looking + text inside ordinary code or string literals is never misreported.""" + source = path.read_text(encoding="utf-8") + try: + tree = ast.parse(source) + except SyntaxError as exc: # pragma: no cover - surfaced, not silently skipped + return [Finding(path, exc.lineno or 0, "syntax-error", str(exc.msg))] + + doc_nodes = (ast.Module, ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef) + # node.body[0].value is the docstring literal; its lineno is the line the string opens + # on, so content line i maps to source line (start + i). + docstrings = [ + (docstring, node.body[0].value.lineno) # type: ignore[attr-defined] + for node in ast.walk(tree) + if isinstance(node, doc_nodes) + if (docstring := ast.get_docstring(node, clean=False)) + ] + return [ + Finding(path, start + offset, category, line) + for docstring, start in docstrings + for offset, line in enumerate(docstring.splitlines()) + for category in _scan_line(line) + ] + + +class Fence(NamedTuple): + """A fenced code block, by 0-based line index. ``terminated`` is False when the fence + has no closing delimiter before EOF, in which case ``close`` is the last line.""" + + open: int + close: int + terminated: bool + + +def fenced_blocks(lines: list[str]) -> list[Fence]: + """Index every fenced code block in ``lines``. + + An unterminated fence is malformed Markdown that `mkdocs build` surfaces anyway; it is + still returned (with ``terminated=False``, ``close`` at the last line) so callers that + skip code can skip to EOF.""" + blocks: list[Fence] = [] + fence: str | None = None + open_idx = -1 + for i, line in enumerate(lines): + stripped = line.lstrip() + if fence is None: + if stripped.startswith(("```", "~~~")): + fence, open_idx = stripped[:3], i + elif stripped.startswith(fence): + blocks.append(Fence(open_idx, i, terminated=True)) + fence = None + if fence is not None: + blocks.append(Fence(open_idx, len(lines) - 1, terminated=False)) + return blocks + + +def find_list_breaking_fences(lines: list[str], blocks: list[Fence]) -> list[tuple[int, str]]: + """Return ``(lineno, snippet)`` for each fenced code block at column 0 that splits a + list -- i.e. one whose nearest non-blank neighbours on both sides are top-level list + items. Such a fence is not indented into the preceding item, so Markdown closes the + list at the fence and the following item starts a new one. The fix is to indent the + fence (4 spaces) so it nests inside its list item. See the module docstring. + + Conservative on purpose: it requires a list item *directly* before and after (a + continuation line or paragraph in between is not matched), keeping false positives low + for a check that fails CI. Unterminated fences are ignored.""" + + def neighbour(start: int, step: int) -> str | None: + j = start + step + while 0 <= j < len(lines): + if lines[j].strip(): + return lines[j] + j += step + return None + + def splits_a_list(open_i: int, close_i: int) -> bool: + if lines[open_i][:1].isspace(): + return False # indented fence: already nested in the list item, not a break + before = neighbour(open_i, -1) + after = neighbour(close_i, +1) + return bool(before and after and LIST_ITEM.match(before) and LIST_ITEM.match(after)) + + return [ + (fence.open + 1, lines[fence.open]) + for fence in blocks + if fence.terminated and splits_a_list(fence.open, fence.close) + ] + + +def lint_markdown(path: Path) -> list[Finding]: + """Scan a Markdown file: RST residue in prose (skipping fenced code blocks), plus + fenced code blocks that break a list (see find_list_breaking_fences).""" + lines = path.read_text(encoding="utf-8").splitlines() + blocks = fenced_blocks(lines) + in_code = {i for fence in blocks for i in range(fence.open, fence.close + 1)} + + prose = [ + Finding(path, lineno, category, line) + for lineno, line in enumerate(lines, start=1) + if lineno - 1 not in in_code + for category in _scan_line(line) + ] + breaks = [ + Finding(path, lineno, "list-break", snippet) + for lineno, snippet in find_list_breaking_fences(lines, blocks) + ] + return prose + breaks + + +def iter_files(paths: tuple[Path, ...]) -> list[Path]: + files: list[Path] = [] + for path in paths: + if path.is_file(): + files.append(path) + elif path.is_dir(): + files.extend(sorted(path.rglob("*.py"))) + files.extend(sorted(path.rglob("*.md"))) + else: + raise FileNotFoundError(f"{path} does not exist") + return files + + +LINTERS = {".py": lint_python, ".md": lint_markdown} + + +def lint(paths: tuple[Path, ...]) -> list[Finding]: + return [ + finding + for file in iter_files(paths) + if file.suffix in LINTERS + for finding in LINTERS[file.suffix](file) + ] + + +def main() -> int: + args = sys.argv[1:] + paths = tuple(Path(a).resolve() for a in args) if args else DEFAULT_PATHS + findings = lint(paths) + + if not findings: + print("No reStructuredText residue or list-breaking fences found in docs.") + return 0 + + print( + f"Found {len(findings)} docs issue(s) -- RST markup that will not render under " + "MkDocs/mkdocstrings, or Markdown that renders as valid-but-wrong HTML:\n", + file=sys.stderr, + ) + for finding in findings: + print(finding.format(), file=sys.stderr) + remediation = "\n".join(f" {c.category:<13} {c.example:<19} -> {c.fix}" for c in CHECKS) + print( + f"\nFix each issue (see ci/lint_docs.py header):\n{remediation}", + file=sys.stderr, + ) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/contributing.md b/docs/contributing.md index eaab26fbc7..df46f381de 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -12,23 +12,23 @@ If you find a bug, please raise a [GitHub issue](https://github.com/zarr-develop 1. A minimal, self-contained snippet of Python code reproducing the problem. You can format the code nicely using markdown, e.g.: -```python exec="false" reason="illustrative pseudocode with a '# etc.' placeholder, not runnable" -import zarr -g = zarr.group() -# etc. -``` + ```python exec="false" reason="illustrative pseudocode with a '# etc.' placeholder, not runnable" + import zarr + g = zarr.group() + # etc. + ``` 2. An explanation of why the current behavior is wrong/not desired, and what you expect instead. 3. Information about the version of Zarr, along with versions of dependencies and the Python interpreter, and installation information. The version of Zarr can be obtained from the `zarr.__version__` attribute. Please also state how Zarr was installed, e.g., "installed via pip into a virtual environment", or "installed using conda". Information about other packages installed can be obtained by executing `pip freeze` (if using pip to install packages) or `conda env export` (if using conda to install packages) from the operating system command prompt. The version of the Python interpreter can be obtained by running a Python interactive session, e.g.: -```console -python -``` + ```console + python + ``` -```ansi -Python 3.12.7 | packaged by conda-forge | (main, Oct 4 2024, 15:57:01) [Clang 17.0.6 ] on darwin -``` + ```ansi + Python 3.12.7 | packaged by conda-forge | (main, Oct 4 2024, 15:57:01) [Clang 17.0.6 ] on darwin + ``` ## Enhancement proposals @@ -401,7 +401,6 @@ The Zarr library is an implementation of a file format standard defined external If an existing Zarr format version changes, or a new version of the Zarr format is released, then the Zarr library will generally require changes. It is very likely that a new Zarr format will require extensive breaking changes to the Zarr library, and so support for a new Zarr format in the Zarr library will almost certainly come in a new `major` release. When the Zarr library adds support for a new Zarr format, there may be a period of accelerated changes as developers refine newly added APIs and deprecate old APIs. In such a transitional phase breaking changes may be more frequent than usual. - ## Experimental API policy The `zarr.experimental` namespace contains features that are under active development and may change without notice. When contributing to or depending on experimental features, please keep the following in mind: diff --git a/docs/index.md b/docs/index.md index 1ec5e93c75..ee4098a8ea 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,7 +6,6 @@ [Developer Chat](https://ossci.zulipchat.com/) | [Zarr specifications](https://zarr-specs.readthedocs.io) - Zarr is a powerful library for storage of n-dimensional arrays, supporting chunking, compression, and various backends, making it a versatile choice for scientific and large-scale data. @@ -37,22 +36,20 @@ conda install -c conda-forge zarr
-- [:material-clock-fast:{ .lg .middle } __Quick start__](quick-start.md) +- [:material-clock-fast:{ .lg .middle } __Quick start__](quick-start.md) --- New to Zarr? Check out the quick start guide. It contains a brief introduction to Zarr's main concepts and links to additional tutorials. - -- [:material-book-open:{ .lg .middle } __User guide__](user-guide/index.md) +- [:material-book-open:{ .lg .middle } __User guide__](user-guide/index.md) --- A detailed guide for how to use Zarr-Python. - -- [:material-api:{ .lg .middle } __API Reference__](api/zarr/index.md) +- [:material-api:{ .lg .middle } __API Reference__](api/zarr/index.md) --- @@ -61,8 +58,7 @@ conda install -c conda-forge zarr which parameters can be used. It assumes that you have an understanding of the key concepts. - -- [:material-account-group:{ .lg .middle } __Contributor's Guide__](contributing.md) +- [:material-account-group:{ .lg .middle } __Contributor's Guide__](contributing.md) --- @@ -72,7 +68,6 @@ conda install -c conda-forge zarr
- ## Project Status More information about the Zarr format can be found on the [main website](https://zarr.dev). @@ -80,6 +75,7 @@ More information about the Zarr format can be found on the [main website](https: If you are using Zarr-Python, we would [love to hear about it](https://github.com/zarr-developers/community/issues/19). ### Funding and Support + The project is fiscally sponsored by [NumFOCUS](https://numfocus.org/), a US 501(c)(3) public charity, and development has been supported by the [MRC Centre for Genomics and Global Health](https://github.com/cggh/) diff --git a/docs/quick-start.md b/docs/quick-start.md index 29232d2a04..17cb1c599a 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -50,7 +50,6 @@ written to a `LocalStore` in the `data/example-1.zarr` directory. Zarr supports data compression and filters. For example, to use Blosc compression: - ```python exec="true" session="quickstart" source="above" result="ansi" # Create a 2D Zarr array with Blosc compression @@ -73,7 +72,6 @@ print(z.info) This compresses the data using the Blosc codec with shuffle enabled for better compression. - ## Hierarchical Groups Zarr allows you to create hierarchical groups, similar to directories: diff --git a/docs/release-notes.md b/docs/release-notes.md index e85137f2b6..7a5b12f59f 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -1,5 +1,7 @@ # Release notes + + ## 3.3.0 (2026-07-15) diff --git a/docs/user-guide/attributes.md b/docs/user-guide/attributes.md index 96a5d67584..8c11c853f1 100644 --- a/docs/user-guide/attributes.md +++ b/docs/user-guide/attributes.md @@ -20,6 +20,7 @@ print('foo' in root.attrs) ```python exec="true" session="attributes" source="above" result="ansi" print(root.attrs['foo']) ``` + ```python exec="true" session="attributes" source="above" result="ansi" print(sorted(z.attrs)) ``` diff --git a/docs/user-guide/consolidated_metadata.md b/docs/user-guide/consolidated_metadata.md index 7030ed6cbc..9cb4d87c89 100644 --- a/docs/user-guide/consolidated_metadata.md +++ b/docs/user-guide/consolidated_metadata.md @@ -119,7 +119,6 @@ removed, or modified, consolidated metadata may not be desirable. `use_consolidated=False` to [`zarr.open_group`][] to always read child metadata directly from the store. - ## Stores Without Support for Consolidated Metadata Some stores may want to opt out of the consolidated metadata mechanism. This diff --git a/docs/user-guide/data_types.md b/docs/user-guide/data_types.md index 642a09d714..ac966f885f 100644 --- a/docs/user-guide/data_types.md +++ b/docs/user-guide/data_types.md @@ -194,9 +194,11 @@ Python supports nearly all of the data types in NumPy. If you need a data type t here, it's possible to create it yourself: see [Adding new data types](#adding-new-data-types). #### Boolean + - [Boolean][zarr.dtype.Bool] #### Integral + - [Signed 8-bit integer][zarr.dtype.Int8] - [Signed 16-bit integer][zarr.dtype.Int16] - [Signed 32-bit integer][zarr.dtype.Int32] @@ -207,6 +209,7 @@ here, it's possible to create it yourself: see [Adding new data types](#adding-n - [Unsigned 64-bit integer][zarr.dtype.UInt64] #### Floating-point + - [16-bit floating-point][zarr.dtype.Float16] - [32-bit floating-point][zarr.dtype.Float32] - [64-bit floating-point][zarr.dtype.Float64] @@ -214,19 +217,23 @@ here, it's possible to create it yourself: see [Adding new data types](#adding-n - [128-bit complex floating-point][zarr.dtype.Complex128] #### String + - [Fixed-length UTF-32 string][zarr.dtype.FixedLengthUTF32] - [Variable-length UTF-8 string][zarr.dtype.VariableLengthUTF8] #### Bytes + - [Fixed-length null-terminated bytes][zarr.dtype.NullTerminatedBytes] - [Fixed-length raw bytes][zarr.dtype.RawBytes] - [Variable-length bytes][zarr.dtype.VariableLengthBytes] #### Temporal + - [DateTime64][zarr.dtype.DateTime64] - [TimeDelta64][zarr.dtype.TimeDelta64] #### Struct-like + - [Structured][zarr.dtype.Structured] !!! note "Zarr V3 Structured Data Types" diff --git a/docs/user-guide/experimental.md b/docs/user-guide/experimental.md index a70ee923fe..e14146610c 100644 --- a/docs/user-guide/experimental.md +++ b/docs/user-guide/experimental.md @@ -168,7 +168,6 @@ print(f"Speedup is {speedup}") Cache effectiveness is particularly pronounced with repeated access to the same data chunks. - ### Cache Configuration The CacheStore can be configured with several parameters: diff --git a/docs/user-guide/extending.md b/docs/user-guide/extending.md index 8602ec8c3b..f852f9105e 100644 --- a/docs/user-guide/extending.md +++ b/docs/user-guide/extending.md @@ -14,6 +14,7 @@ in the following ways: [numcodecs.registry.register_codec](https://numcodecs.readthedocs.io/en/stable/registry.html#numcodecs.registry.register_codec). There are three types of codecs in Zarr: + - array-to-array - array-to-bytes - bytes-to-bytes diff --git a/docs/user-guide/groups.md b/docs/user-guide/groups.md index a2b40dfe8a..337ad39554 100644 --- a/docs/user-guide/groups.md +++ b/docs/user-guide/groups.md @@ -174,4 +174,3 @@ Groups also have the [`zarr.Group.tree`][] method, e.g.: ```python exec="true" session="groups" source="above" result="ansi" print(root.tree()) ``` - diff --git a/docs/user-guide/performance.md b/docs/user-guide/performance.md index 818386578c..52c1cf0d71 100644 --- a/docs/user-guide/performance.md +++ b/docs/user-guide/performance.md @@ -120,7 +120,6 @@ The order of chunks **within each shard** can be changed via the `subchunk_write By default [`morton`](https://en.wikipedia.org/wiki/Z-order_curve) order provides good spatial locality. [`lexicographic` (i.e., row-major)](https://en.wikipedia.org/wiki/Row-_and_column-major_order), for example, may be better suited to "batched" workflows where some form of sequential reading through a fixed number of outer dimensions is desired, and `colexicographic` is its reverse. `unordered` makes no guarantee about the order in which subchunks are laid out within a shard. - ### Empty chunks It is possible to configure how Zarr handles the storage of chunks that are "empty" @@ -221,11 +220,13 @@ zarr.config.set({'async.concurrency': 128}) ``` Higher concurrency values can improve throughput when: + - Working with remote storage (e.g., S3, GCS) where network latency is high - Reading/writing many small chunks in parallel - The storage backend can handle many concurrent requests Lower concurrency values may be beneficial when: + - Working with local storage with limited I/O bandwidth - Memory is constrained (each concurrent operation requires buffer space) - Using Zarr within a parallel computing framework (see below) @@ -258,7 +259,7 @@ concurrently. **Important**: When using many Dask threads, you may need to reduce both Zarr's `async.concurrency` and `threading.max_workers` settings to avoid creating too many concurrent operations. The total number of concurrent I/O operations can be roughly estimated as: -``` +```text total_concurrency ≈ dask_threads × zarr_async_concurrency ``` @@ -298,6 +299,7 @@ You may need to experiment with different values to find the optimal balance for Zarr arrays are designed to be thread-safe for concurrent reads and writes from multiple threads within the same process. However, proper synchronization is required when writing to overlapping regions from multiple threads. For multi-process parallelism, Zarr provides safe concurrent writes as long as: + - Different processes write to different chunks - The storage backend supports atomic writes (most do) diff --git a/docs/user-guide/storage.md b/docs/user-guide/storage.md index d32fe217c9..7e0154b2a0 100644 --- a/docs/user-guide/storage.md +++ b/docs/user-guide/storage.md @@ -41,17 +41,21 @@ group = zarr.create_group(store=data) print(group) ``` + [](){#user-guide-store-like} + ### StoreLike `StoreLike` values can be: - a `Path` or string indicating a location on the local file system. This will create a [local store](#local-store): + ```python exec="true" session="storage" source="above" result="ansi" group = zarr.open_group(store='data/foo/bar') print(group) ``` + ```python exec="true" session="storage" source="above" result="ansi" from pathlib import Path group = zarr.open_group(store=Path('data/foo/bar')) @@ -59,6 +63,7 @@ print(group) ``` - an FSSpec URI string, indicating a [remote store](#remote-store) location: + ```python exec="true" session="storage" source="above" result="ansi" # Note: requires s3fs to be installed group = zarr.open_group( @@ -70,10 +75,12 @@ print(group) ``` - an empty dictionary or None, which will create a new [memory store](#memory-store): + ```python exec="true" session="storage" source="above" result="ansi" group = zarr.create_group(store={}) print(group) ``` + ```python exec="true" session="storage" source="above" result="ansi" group = zarr.create_group(store=None) print(group) @@ -152,7 +159,6 @@ print(store) When using an S3-compatible service other than AWS, pass the service endpoint to the filesystem via `client_kwargs={'endpoint_url': 'https://...'}`. - ### Memory Store The [`zarr.storage.MemoryStore`][] stores Zarr data (metadata and chunks) in an diff --git a/lychee.toml b/lychee.toml new file mode 100644 index 0000000000..38b2b8ab7a --- /dev/null +++ b/lychee.toml @@ -0,0 +1,20 @@ +# Configuration for the lychee link checker (https://lychee.cli.rs/). +# Auto-discovered as ./lychee.toml by the lychee GitHub Action. + +# Files lychee should not scan for links. +exclude_path = [ + # mkdocs-material theme overrides: hrefs are Jinja expressions like + # `{{ '../' ~ base_url }}`, not real URLs, so lychee cannot resolve them. + "docs/overrides", + # Design notes: working records that point at transient artifacts (commits, + # fork branches, compare URLs) which are expected to disappear over time. + "design", +] + +# URL patterns to ignore (regex, matched against the full URL). +exclude = [ + # Local docs preview server shown in the contributing guide ("hatch run serve"), + # documentation of a command rather than a reachable link. + '^https?://0\.0\.0\.0', + '^https?://(localhost|127\.0\.0\.1)(:\d+)?', +] diff --git a/src/zarr/core/metadata/v2.py b/src/zarr/core/metadata/v2.py index ac32521239..91515d87b9 100644 --- a/src/zarr/core/metadata/v2.py +++ b/src/zarr/core/metadata/v2.py @@ -120,7 +120,7 @@ def ndim(self) -> int: def chunk_grid(self) -> ChunkGrid: """Backwards-compatible chunk grid property. - .. deprecated:: + !!! warning "Deprecated" Access the chunk grid via the array layer instead. This property will be removed in a future release. """ diff --git a/src/zarr/testing/store.py b/src/zarr/testing/store.py index 80a0996ca6..46287ccffb 100644 --- a/src/zarr/testing/store.py +++ b/src/zarr/testing/store.py @@ -40,21 +40,21 @@ class StoreTests[S: Store, B: Buffer]: @staticmethod def _require_get_sync(store: S) -> SupportsGetSync: - """Skip unless *store* implements :class:`SupportsGetSync`.""" + """Skip unless *store* implements [`SupportsGetSync`][zarr.abc.store.SupportsGetSync].""" if not isinstance(store, SupportsGetSync): pytest.skip("store does not implement SupportsGetSync") return store # type: ignore[unreachable] @staticmethod def _require_set_sync(store: S) -> SupportsSetSync: - """Skip unless *store* implements :class:`SupportsSetSync`.""" + """Skip unless *store* implements [`SupportsSetSync`][zarr.abc.store.SupportsSetSync].""" if not isinstance(store, SupportsSetSync): pytest.skip("store does not implement SupportsSetSync") return store # type: ignore[unreachable] @staticmethod def _require_delete_sync(store: S) -> SupportsDeleteSync: - """Skip unless *store* implements :class:`SupportsDeleteSync`.""" + """Skip unless *store* implements [`SupportsDeleteSync`][zarr.abc.store.SupportsDeleteSync].""" if not isinstance(store, SupportsDeleteSync): pytest.skip("store does not implement SupportsDeleteSync") return store # type: ignore[unreachable] From 7f8834ab14d607d1ec03209d9509a153fe81b173 Mon Sep 17 00:00:00 2001 From: Max Jones <14077947+maxrjones@users.noreply.github.com> Date: Thu, 16 Jul 2026 13:42:21 -0400 Subject: [PATCH 4/8] chore: cleanup indentation and guidance for release (#4153) * chore: cleanup indentation and guidance for release * Make explanation a comment --- .github/ISSUE_TEMPLATE/release-checklist.md | 7 +- .github/PULL_REQUEST_TEMPLATE.md | 2 +- .github/workflows/prepare_release.yml | 74 ------ ci/lint_docs.py | 67 ++++- docs/contributing.md | 21 +- docs/release-notes.md | 258 ++++++++++---------- docs/user-guide/data_types.md | 20 +- docs/user-guide/v3_migration.md | 73 +++--- 8 files changed, 251 insertions(+), 271 deletions(-) delete mode 100644 .github/workflows/prepare_release.yml diff --git a/.github/ISSUE_TEMPLATE/release-checklist.md b/.github/ISSUE_TEMPLATE/release-checklist.md index 5e073cb59f..309c76b4dc 100644 --- a/.github/ISSUE_TEMPLATE/release-checklist.md +++ b/.github/ISSUE_TEMPLATE/release-checklist.md @@ -18,8 +18,8 @@ assignees: '' - [ ] Check [SPEC 0](https://scientific-python.org/specs/spec-0000/#support-window) to see if the minimum supported version of Python or NumPy needs bumping. - [ ] Verify that the latest CI workflows on `main` are passing: [Tests](https://github.com/zarr-developers/zarr-python/actions/workflows/test.yml), [GPU Tests](https://github.com/zarr-developers/zarr-python/actions/workflows/gpu_test.yml), [Hypothesis](https://github.com/zarr-developers/zarr-python/actions/workflows/hypothesis.yaml), [Docs](https://github.com/zarr-developers/zarr-python/actions/workflows/docs.yml), [Lint](https://github.com/zarr-developers/zarr-python/actions/workflows/lint.yml), [Wheels](https://github.com/zarr-developers/zarr-python/actions/workflows/releases.yml). -- [ ] Run the ["Prepare release" workflow](https://github.com/zarr-developers/zarr-python/actions/workflows/prepare_release.yml) with the target version. This will build the changelog and open a release PR with the `run-downstream` label. -- [ ] Verify that the [downstream tests](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml) (triggered automatically by the `run-downstream` label) pass on the release PR. +- [ ] Run the [downstream tests](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml) against `main`: go to the workflow page, click "Run workflow", and select the `main` branch. Verify that the Xarray and numcodecs integration tests pass. +- [ ] Open a release PR with the changelog entries for the upcoming release, generated with `uv run --only-group release towncrier build --version x.y.z`. - [ ] Review the release PR and verify the changelog in `docs/release-notes.md` looks correct. - [ ] Merge the release PR. @@ -45,7 +45,8 @@ In rare cases (e.g. patch releases for an older minor version), you may need to - Create the release branch from the appropriate tag if it doesn't already exist. - Cherry-pick or backport the necessary commits onto the branch. -- Run `towncrier build --version x.y.z` and commit the result to the release branch instead of `main`. +- Run `towncrier build --version x.y.z` and open the release PR against the release branch instead of `main`. +- Run the downstream tests against the release branch instead of `main`. - When drafting the GitHub Release, set the target to the release branch instead of `main`. - After the release, ensure any relevant changelog updates are also reflected on `main`. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 1d12aa02eb..47adb4b19e 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -15,7 +15,7 @@ - [ ] I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct. -AI coding assistance is welcome, but a human must be the author and is responsible for the contents of the PR. The description and any review responses must be in your own words. Please read [AI-assisted contributions](https://zarr.readthedocs.io/en/stable/contributing/#ai-assisted-contributions) before opening. + ## TODO diff --git a/.github/workflows/prepare_release.yml b/.github/workflows/prepare_release.yml deleted file mode 100644 index d558779bf2..0000000000 --- a/.github/workflows/prepare_release.yml +++ /dev/null @@ -1,74 +0,0 @@ -name: Prepare release notes - -on: - workflow_dispatch: - inputs: - version: - description: 'Release version notes (e.g. 3.2.0)' - required: true - type: string - target_branch: - description: 'Branch to target' - required: false - default: 'main' - type: string - -permissions: - contents: write - pull-requests: write - -jobs: - prepare: - name: Build changelog and open PR - runs-on: ubuntu-latest - steps: - - name: Validate inputs - run: | - if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+([-\.][a-zA-Z0-9]+)*$ ]]; then - echo "::error::Invalid version format: '$VERSION'" - exit 1 - fi - if [[ ! "$TARGET_BRANCH" =~ ^[a-zA-Z0-9._/-]+$ ]]; then - echo "::error::Invalid branch name: '$TARGET_BRANCH'" - exit 1 - fi - env: - VERSION: ${{ inputs.version }} - TARGET_BRANCH: ${{ inputs.target_branch }} - - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - with: - ref: ${{ inputs.target_branch }} - fetch-depth: 0 - persist-credentials: false - - - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 - - - name: Build changelog - # Use the pinned towncrier from the `release` dependency group (single - # source of truth) rather than an unpinned standalone install. - run: uv run --only-group release towncrier build --version "$VERSION" --yes - env: - VERSION: ${{ inputs.version }} - - - name: Create pull request - uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1 - with: - branch: release/v${{ inputs.version }} - base: ${{ inputs.target_branch }} - title: "Release v${{ inputs.version }}" - body: | - Automated release preparation for v${{ inputs.version }}. - - This PR was generated by the "Prepare release" workflow. It includes: - - Rendered changelog via `towncrier build --version ${{ inputs.version }}` - - Removal of consumed changelog fragments from `changes/` - - ## Checklist - - - [ ] Review the rendered changelog in `docs/release-notes.md` - - [ ] Downstream tests pass (see [downstream workflow](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml)) - - [ ] Merge this PR, then [draft a GitHub Release](https://github.com/zarr-developers/zarr-python/releases/new) targeting `${{ inputs.target_branch }}` with tag `v${{ inputs.version }}` - commit-message: "chore: build changelog for v${{ inputs.version }}" - labels: run-downstream - delete-branch: true diff --git a/ci/lint_docs.py b/ci/lint_docs.py index ff7b56649b..a847e8aa9d 100644 --- a/ci/lint_docs.py +++ b/ci/lint_docs.py @@ -13,7 +13,7 @@ fast, source-level check that needs no docs build. Checks fall into two groups -- RST markup that silently fails under MkDocs/mkdocstrings, -and a Markdown structural problem that renders as valid-but-wrong HTML (so `mkdocs build` +and Markdown structural problems that render as valid-but-wrong HTML (so `mkdocs build` emits no warning): sphinx-role :class:`X`, :func:`X`, :py:meth:`X` -> [`X`][zarr.X] @@ -21,6 +21,8 @@ rst-field :param x:, :returns:, :rtype: -> numpydoc Parameters/Returns/Raises rst-link `text `_ -> [text](https://example) list-break unindented code fence between list items -> indent the fence under its item + list-indent continuation block indented < 4 spaces -> indent it 4 spaces + list-blank list item directly after indented block -> blank line before the item The ``list-break`` check catches a fenced code block at column 0 placed *between* two list items: because the fence is not indented into the preceding item, Markdown ends the list at @@ -29,6 +31,15 @@ notices this for sequentially-numbered ordered lists; lazily-numbered (1., 1.) and unordered lists slip past it, so this structural check covers the gap. +The ``list-indent`` and ``list-blank`` checks catch the two halves of Python-Markdown's +strict list-continuation rules, which differ from CommonMark. A blank-line-separated +block (paragraph, nested list, table) belongs to a list item only when indented at least +4 spaces; at the 2-space indent other renderers accept, Python-Markdown ends the list and +the block escapes to the top level (``list-indent``). And a new list item can not start +directly after an indented continuation block: without a blank line first, the ``- `` line +is lazily absorbed into the preceding paragraph as literal text (``list-blank``). Both +produced silently-broken changelog rendering in ``docs/release-notes.md``. + Usage: python ci/lint_docs.py [PATH ...] @@ -87,14 +98,16 @@ class Check(NamedTuple): fix: str -# ``list-break`` carries no pattern -- it is detected structurally in find_list_breaking_fences, -# not by scanning a single line -- but it appears here so it shares the remediation help. +# The ``list-*`` checks carry no pattern -- they are detected structurally, not by scanning +# a single line -- but they appear here so they share the remediation help. CHECKS = ( Check("sphinx-role", SPHINX_ROLE, ":class:`X`", "[`X`][zarr.X]"), Check("rst-directive", RST_DIRECTIVE, ".. note::", "MkDocs admonition (!!! note)"), Check("rst-field", RST_FIELD, ":param x:", "numpydoc Parameters/Returns/Raises section"), Check("rst-link", RST_LINK, "`text `_", "[text](url)"), Check("list-break", None, "fence between items", "indent the fence 4 spaces to nest it"), + Check("list-indent", None, "2-space continuation", "indent the block 4 spaces under its item"), + Check("list-blank", None, "item after indented block", "add a blank line before the item"), ) @@ -211,9 +224,49 @@ def splits_a_list(open_i: int, close_i: int) -> bool: ] +def find_list_continuation_issues( + lines: list[str], in_code: set[int] +) -> list[tuple[int, str, str]]: + """Return ``(lineno, category, snippet)`` for list continuations Python-Markdown will + mis-render (see the module docstring): + + - ``list-indent``: a blank-line-separated block inside a list item indented 1-3 + spaces. Python-Markdown requires 4; at less, the block escapes the list. + - ``list-blank``: a top-level list item directly after a line indented 4+ spaces. + Without a blank line in between, the item is absorbed into the preceding paragraph + as literal ``- `` text. + + Lazy continuations (an indented line with no blank line before it) are valid at any + indent and are not flagged. Fenced-code lines are opaque: never flagged themselves, + but they keep the item scope open and their indent feeds the ``list-blank`` check so + an item directly after an indented fence is still caught.""" + findings: list[tuple[int, str, str]] = [] + in_item = False # inside a top-level list item's scope + prev_blank = True + prev_indent = 0 + for i, line in enumerate(lines): + stripped = line.strip() + if not stripped: + prev_blank = True + continue + indent = len(line) - len(line.lstrip(" ")) + if i not in in_code: + if indent == 0: + is_item = bool(LIST_ITEM.match(line)) + if is_item and in_item and not prev_blank and prev_indent >= 4: + findings.append((i + 1, "list-blank", line)) + in_item = is_item + elif in_item and prev_blank and indent < 4: + findings.append((i + 1, "list-indent", line)) + prev_blank = False + prev_indent = indent + return findings + + def lint_markdown(path: Path) -> list[Finding]: """Scan a Markdown file: RST residue in prose (skipping fenced code blocks), plus - fenced code blocks that break a list (see find_list_breaking_fences).""" + list-structure problems (see find_list_breaking_fences and + find_list_continuation_issues).""" lines = path.read_text(encoding="utf-8").splitlines() blocks = fenced_blocks(lines) in_code = {i for fence in blocks for i in range(fence.open, fence.close + 1)} @@ -228,7 +281,11 @@ def lint_markdown(path: Path) -> list[Finding]: Finding(path, lineno, "list-break", snippet) for lineno, snippet in find_list_breaking_fences(lines, blocks) ] - return prose + breaks + continuations = [ + Finding(path, lineno, category, snippet) + for lineno, category, snippet in find_list_continuation_issues(lines, in_code) + ] + return prose + breaks + continuations def iter_files(paths: tuple[Path, ...]) -> list[Path]: diff --git a/docs/contributing.md b/docs/contributing.md index df46f381de..aeb88e6ce1 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -354,18 +354,7 @@ This policy exists to lower the cost of routine work and to help newer core deve ### Release procedure -Open an issue on GitHub announcing the release using the release checklist template: -[https://github.com/zarr-developers/zarr-python/issues/new?template=release-checklist.md](https://github.com/zarr-developers/zarr-python/issues/new?template=release-checklist.md). The release checklist includes all steps necessary for the release. - -#### Preparing a release - -Releases are prepared using the ["Prepare release notes"](https://github.com/zarr-developers/zarr-python/actions/workflows/prepare_release.yml) workflow. To run it: - -1. Go to the [workflow page](https://github.com/zarr-developers/zarr-python/actions/workflows/prepare_release.yml) and click "Run workflow". -2. Enter the release version (e.g. `3.2.0`) and the target branch (defaults to `main`). -3. The workflow will run `towncrier build` to render the changelog, remove consumed fragments from `changes/`, and open a pull request on the `release/v` branch. -4. The release PR is automatically labeled `run-downstream`, which triggers the [downstream test workflow](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml) to run Xarray and numcodecs integration tests against the release branch. -5. Review the rendered changelog in `docs/release-notes.md` and verify downstream tests pass before merging. +To give the release visibility and a single place to track progress, open an issue on GitHub announcing the release using the [release checklist template](https://github.com/zarr-developers/zarr-python/issues/new?template=release-checklist.md). The release checklist includes all steps necessary for the release. ## Compatibility and versioning policies @@ -377,17 +366,17 @@ Releases are classified by the library changes contained in that release. This c * **major** releases (for example, `2.18.0` -> `3.0.0`) are for changes that will require extensive adaptation efforts from many users and downstream projects. For example, breaking changes to widely-used user-facing APIs should only be applied in a major release. - Users and downstream projects should carefully consider the impact of a major release before adopting it. In advance of a major release, developers should communicate the scope of the upcoming changes, and help users prepare for them. + Users and downstream projects should carefully consider the impact of a major release before adopting it. In advance of a major release, developers should communicate the scope of the upcoming changes, and help users prepare for them. * **minor** releases (for example, `3.0.0` -> `3.1.0`) are for changes that do not require significant effort from most users or downstream projects to respond to. API changes are possible in minor releases if the burden on users imposed by those changes is sufficiently small. - For example, a recently released API may need fixes or refinements that are breaking, but low impact due to the recency of the feature. Such API changes are permitted in a minor release. + For example, a recently released API may need fixes or refinements that are breaking, but low impact due to the recency of the feature. Such API changes are permitted in a minor release. - Minor releases are safe for most users and downstream projects to adopt. + Minor releases are safe for most users and downstream projects to adopt. * **patch** releases (for example, `3.1.0` -> `3.1.1`) are for changes that contain no breaking or behavior changes for downstream projects or users. Examples of changes suitable for a patch release are bugfixes and documentation improvements. - Users should always feel safe upgrading to the latest patch release. + Users should always feel safe upgrading to the latest patch release. Note that this versioning scheme is not consistent with [Semantic Versioning](https://semver.org/). Contrary to SemVer, the Zarr library may release breaking changes in `minor` releases, or even `patch` releases under exceptional circumstances. But we should strive to avoid doing so. diff --git a/docs/release-notes.md b/docs/release-notes.md index 7a5b12f59f..3fd8a5f360 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -51,32 +51,34 @@ `ClientSession` was left open until garbage collection, producing `"Unclosed client session"` `ResourceWarning`s from aiohttp. - The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when - `FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` - when a sync→async conversion was performed). When `_owns_fs` is ``True``, - `store.close()` calls the new `_close_fs()` helper, which invokes - `fs.set_session()` and closes the returned client. Callers who supply their own - filesystem instance to `FsspecStore()` directly remain responsible for its - lifecycle; `_owns_fs` is ``False`` for those stores. - - **Scope note**: This fix closes the S3 client session that is active at the time - `store.close()` is called. Some S3-backed filesystem implementations (e.g. - s3fs with ``cache_regions=True``) may internally refresh and replace their - client during I/O operations, abandoning prior sessions before ``store.close()`` - is invoked. Those intermediate sessions are outside the scope of this fix and - are an issue in the upstream filesystem library. ([#4003](https://github.com/zarr-developers/zarr-python/issues/4003)) + The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when + `FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` + when a sync→async conversion was performed). When `_owns_fs` is ``True``, + `store.close()` calls the new `_close_fs()` helper, which invokes + `fs.set_session()` and closes the returned client. Callers who supply their own + filesystem instance to `FsspecStore()` directly remain responsible for its + lifecycle; `_owns_fs` is ``False`` for those stores. + + **Scope note**: This fix closes the S3 client session that is active at the time + `store.close()` is called. Some S3-backed filesystem implementations (e.g. + s3fs with ``cache_regions=True``) may internally refresh and replace their + client during I/O operations, abandoning prior sessions before ``store.close()`` + is invoked. Those intermediate sessions are outside the scope of this fix and + are an issue in the upstream filesystem library. ([#4003](https://github.com/zarr-developers/zarr-python/issues/4003)) + - Fixed an invalid `zarr.create_array` example in the quick-start documentation (it passed an unsupported `mode` argument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested. ([#4016](https://github.com/zarr-developers/zarr-python/issues/4016)) - Fixed `ObjectStore.list_dir` for object-store listings that include a directory-marker object matching the requested non-root prefix. ([#4032](https://github.com/zarr-developers/zarr-python/issues/4032)) - Prevents mutation of the attributes dict provided by the user by copying them instead of keeping the reference ([#4059](https://github.com/zarr-developers/zarr-python/issues/4059)) - Fixed several storage and codec bugs: - - Reading a value with a `SuffixByteRequest` larger than the value now correctly returns the whole value (matching HTTP `bytes=-N` suffix-range semantics), instead of silently returning incorrect data for `MemoryStore`. - - `LoggingStore.get_partial_values` and `FsspecStore.get_partial_values` no longer return empty results when `key_ranges` is passed as a one-shot iterable (e.g. a generator). - - `Store.getsize_prefix` no longer over-counts sibling keys that merely share a string prefix (e.g. `getsize_prefix("foo")` no longer includes keys under `foobar/`). - - `ZipStore.close()` no longer raises `AttributeError` when the store was created but never opened (including when used as a context manager without any I/O). - - `codecs_from_list` now raises a descriptive `TypeError` when a `BytesBytesCodec` immediately follows an `ArrayArrayCodec`, instead of a misleading "Required ArrayBytesCodec was not found" `ValueError`. + - Reading a value with a `SuffixByteRequest` larger than the value now correctly returns the whole value (matching HTTP `bytes=-N` suffix-range semantics), instead of silently returning incorrect data for `MemoryStore`. + - `LoggingStore.get_partial_values` and `FsspecStore.get_partial_values` no longer return empty results when `key_ranges` is passed as a one-shot iterable (e.g. a generator). + - `Store.getsize_prefix` no longer over-counts sibling keys that merely share a string prefix (e.g. `getsize_prefix("foo")` no longer includes keys under `foobar/`). + - `ZipStore.close()` no longer raises `AttributeError` when the store was created but never opened (including when used as a context manager without any I/O). + - `codecs_from_list` now raises a descriptive `TypeError` when a `BytesBytesCodec` immediately follows an `ArrayArrayCodec`, instead of a misleading "Required ArrayBytesCodec was not found" `ValueError`. + + ([#4074](https://github.com/zarr-developers/zarr-python/issues/4074)) - ([#4074](https://github.com/zarr-developers/zarr-python/issues/4074)) - Fixed writing Fortran-ordered (F-contiguous) arrays through the variable-length string and bytes codecs and through numcodecs array-array filters such as `Delta`, `FixedScaleOffset` and `PackBits`. Chunks are now passed to numcodecs as C-contiguous arrays, so elements are no longer stored in transposed order. ([#4116](https://github.com/zarr-developers/zarr-python/issues/4116)) - Fix silent byte-order corruption for structured dtypes with the `bytes` codec: multi-byte fields are now byte-swapped to the codec's configured `endian` on write and decoded honoring it on read, so non-native-endian structured data (e.g. big-endian fields, as produced by virtual references to external data) round-trips correctly. ([#4141](https://github.com/zarr-developers/zarr-python/issues/4141)) @@ -91,23 +93,24 @@ use only the public API, eliminating all non-public imports, illustrating what users should do. - To better support this, the following types and functions were made available - from public modules: - - | Type/Function | Non-public module | Public module | - | ------------------------- | ------------------------ | ------------- | - | `DataTypeValidationError` | `zarr.core.dtype.common` | `zarr.errors` | - | `JSON` | `zarr.core.common` | `zarr.types` | - | `ZarrFormat` | `zarr.core.common` | `zarr.types` | - | `DTypeConfig_V2` | `zarr.core.dtype.common` | `zarr.types` | - | `DTypeJSON` | `zarr.core.dtype.common` | `zarr.types` | - | `DTypeSpec_V2` | `zarr.core.dtype.common` | `zarr.dtype` | - | `check_dtype_spec_v2` | `zarr.core.dtype.common` | `zarr.dtype` | - - `DataTypeValidationError` was *moved* to `zarr.errors`. Importing it from - `zarr.core.dtype.common` (its original location), `zarr.core.dtype`, or - `zarr.dtype` still works but now raises a `ZarrDeprecationWarning`. The remaining - types and functions are simply re-exported from the listed public module. ([#4052](https://github.com/zarr-developers/zarr-python/issues/4052)) + To better support this, the following types and functions were made available + from public modules: + + | Type/Function | Non-public module | Public module | + | ------------------------- | ------------------------ | ------------- | + | `DataTypeValidationError` | `zarr.core.dtype.common` | `zarr.errors` | + | `JSON` | `zarr.core.common` | `zarr.types` | + | `ZarrFormat` | `zarr.core.common` | `zarr.types` | + | `DTypeConfig_V2` | `zarr.core.dtype.common` | `zarr.types` | + | `DTypeJSON` | `zarr.core.dtype.common` | `zarr.types` | + | `DTypeSpec_V2` | `zarr.core.dtype.common` | `zarr.dtype` | + | `check_dtype_spec_v2` | `zarr.core.dtype.common` | `zarr.dtype` | + + `DataTypeValidationError` was *moved* to `zarr.errors`. Importing it from + `zarr.core.dtype.common` (its original location), `zarr.core.dtype`, or + `zarr.dtype` still works but now raises a `ZarrDeprecationWarning`. The remaining + types and functions are simply re-exported from the listed public module. ([#4052](https://github.com/zarr-developers/zarr-python/issues/4052)) + - Document a self-merge policy in the contributor guide, describing when a core developer may merge their own pull request without a second reviewer and which changes warrant more caution. ([#4053](https://github.com/zarr-developers/zarr-python/issues/4053)) - Fixed many documentation errors found in a full review of the user guide, including prose contradicted by rendered example output on the performance page, invisible @@ -126,11 +129,12 @@ access, and will be removed in a future release. ``BloscCodec.cname`` and ``BloscCodec.shuffle`` are now plain strings rather than enum members. - Additional renames in ``zarr.codecs.blosc`` from the same change: the type - aliases ``Shuffle`` and ``CName`` are now ``BloscShuffleLiteral`` and - ``BloscCnameLiteral``, the constant ``SHUFFLE`` is now ``BLOSC_SHUFFLE`` - (with a new ``BLOSC_CNAME`` alongside it), and ``BloscShuffle.from_int`` - now returns a literal string rather than an enum member. ([#3963](https://github.com/zarr-developers/zarr-python/issues/3963)) + Additional renames in ``zarr.codecs.blosc`` from the same change: the type + aliases ``Shuffle`` and ``CName`` are now ``BloscShuffleLiteral`` and + ``BloscCnameLiteral``, the constant ``SHUFFLE`` is now ``BLOSC_SHUFFLE`` + (with a new ``BLOSC_CNAME`` alongside it), and ``BloscShuffle.from_int`` + now returns a literal string rather than an enum member. ([#3963](https://github.com/zarr-developers/zarr-python/issues/3963)) + - The ``Endian`` (``zarr.codecs.bytes.Endian``) and ``ShardingCodecIndexLocation`` (``zarr.codecs.ShardingCodecIndexLocation``) enums are now deprecated. Pass the equivalent literal string instead (e.g. ``"little"`` / ``"big"``, ``"start"`` / @@ -139,20 +143,21 @@ and ``ShardingCodec.index_location`` are now plain strings rather than enum members. - Two follow-on changes from this deprecation: + Two follow-on changes from this deprecation: - - ``NDBuffer.byteorder`` now returns a literal string (``"little"`` or - ``"big"``) rather than an ``Endian`` member. Subclasses overriding this - property should update their return type. - - The module-level binding ``zarr.codecs.bytes.default_system_endian`` was - removed. ``BytesCodec()`` continues to default to ``sys.byteorder``; - external callers that imported ``default_system_endian`` should use - ``sys.byteorder`` directly. + - ``NDBuffer.byteorder`` now returns a literal string (``"little"`` or + ``"big"``) rather than an ``Endian`` member. Subclasses overriding this + property should update their return type. + - The module-level binding ``zarr.codecs.bytes.default_system_endian`` was + removed. ``BytesCodec()`` continues to default to ``sys.byteorder``; + external callers that imported ``default_system_endian`` should use + ``sys.byteorder`` directly. - Additionally, the module-level function ``zarr.codecs.sharding.parse_index_location`` - was made private as part of this change. + Additionally, the module-level function ``zarr.codecs.sharding.parse_index_location`` + was made private as part of this change. + + ([#3968](https://github.com/zarr-developers/zarr-python/issues/3968)) - ([#3968](https://github.com/zarr-developers/zarr-python/issues/3968)) - Removed the NumPy 1.x implementation of the `VariableLengthUTF8` data type because NumPy 1.x is no longer supported under [SPEC0](https://scientific-python.org/specs/spec-0000/). ([#3973](https://github.com/zarr-developers/zarr-python/issues/3973)) ### Misc @@ -191,19 +196,19 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Add support for rectilinear (variable-sized) chunk grids. This feature is experimental and must be explicitly enabled via `zarr.config.set({'array.rectilinear_chunks': True})`. - Rectilinear chunks can be used through: + Rectilinear chunks can be used through: - - **Creating arrays**: Pass nested sequences (e.g., `[[10, 20, 30], [50, 50]]`) to `chunks` - in `zarr.create_array`, `zarr.from_array`, `zarr.zeros`, `zarr.ones`, `zarr.full`, - `zarr.open`, and related functions, or to `chunk_shape` in `zarr.create`. - - **Opening existing arrays**: Arrays stored with the `rectilinear` chunk grid are read - transparently via `zarr.open` and `zarr.open_array`. - - **Rectilinear sharding**: Shard boundaries can be rectilinear while inner chunks remain regular. + - **Creating arrays**: Pass nested sequences (e.g., `[[10, 20, 30], [50, 50]]`) to `chunks` + in `zarr.create_array`, `zarr.from_array`, `zarr.zeros`, `zarr.ones`, `zarr.full`, + `zarr.open`, and related functions, or to `chunk_shape` in `zarr.create`. + - **Opening existing arrays**: Arrays stored with the `rectilinear` chunk grid are read + transparently via `zarr.open` and `zarr.open_array`. + - **Rectilinear sharding**: Shard boundaries can be rectilinear while inner chunks remain regular. - **Breaking change**: The `validate` method on `BaseCodec` and `CodecPipeline` now receives - a `ChunkGridMetadata` instance instead of a `ChunkGrid` instance for the `chunk_grid` - parameter. Third-party codecs that override `validate` and inspect the chunk grid will need to - update their type annotations. No known downstream packages were using this parameter. ([#3802](https://github.com/zarr-developers/zarr-python/issues/3802)) + **Breaking change**: The `validate` method on `BaseCodec` and `CodecPipeline` now receives + a `ChunkGridMetadata` instance instead of a `ChunkGrid` instance for the `chunk_grid` + parameter. Third-party codecs that override `validate` and inspect the chunk grid will need to + update their type annotations. No known downstream packages were using this parameter. ([#3802](https://github.com/zarr-developers/zarr-python/issues/3802)) - Add `cast_value` and `scale_offset` codecs. ([#3874](https://github.com/zarr-developers/zarr-python/issues/3874)) @@ -394,52 +399,52 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Ensure that invocations of `create_array` use consistent keyword arguments, with consistent defaults. - [`zarr.api.synchronous.create_array`][] now takes a `write_data` keyword argument - The `Group.create_array` method takes `data` and `write_data` keyword arguments. - The functions [`zarr.api.asynchronous.create`][], [`zarr.api.asynchronous.create_array`] - and the methods `Group.create_array`, `Group.array`, had the default - `fill_value` changed from `0` to the `DEFAULT_FILL_VALUE` value, which instructs Zarr to - use the default scalar value associated with the array's data type as the fill value. These are - all functions or methods for array creation that mirror, wrap or are wrapped by, another function - that already has a default `fill_value` set to `DEFAULT_FILL_VALUE`. This change is necessary - to make these functions consistent across the entire codebase, but as this changes default values, - new data might have a different fill value than expected after this change. - - For data types where 0 is meaningful, like integers or floats, the default scalar is 0, so this - change should not be noticeable. For data types where 0 is ambiguous, like fixed-length unicode - strings, the default fill value might be different after this change. Users who were relying on how - Zarr interpreted `0` as a non-numeric scalar value should set their desired fill value explicitly - after this change. + [`zarr.api.synchronous.create_array`][] now takes a `write_data` keyword argument + The `Group.create_array` method takes `data` and `write_data` keyword arguments. + The functions [`zarr.api.asynchronous.create`][], [`zarr.api.asynchronous.create_array`] + and the methods `Group.create_array`, `Group.array`, had the default + `fill_value` changed from `0` to the `DEFAULT_FILL_VALUE` value, which instructs Zarr to + use the default scalar value associated with the array's data type as the fill value. These are + all functions or methods for array creation that mirror, wrap or are wrapped by, another function + that already has a default `fill_value` set to `DEFAULT_FILL_VALUE`. This change is necessary + to make these functions consistent across the entire codebase, but as this changes default values, + new data might have a different fill value than expected after this change. + + For data types where 0 is meaningful, like integers or floats, the default scalar is 0, so this + change should not be noticeable. For data types where 0 is ambiguous, like fixed-length unicode + strings, the default fill value might be different after this change. Users who were relying on how + Zarr interpreted `0` as a non-numeric scalar value should set their desired fill value explicitly + after this change. - Added public API for Buffer ABCs and implementations. - Use `zarr.buffer` to access buffer implementations, and - `zarr.abc.buffer` for the interface to implement new buffer types. + Use `zarr.buffer` to access buffer implementations, and + `zarr.abc.buffer` for the interface to implement new buffer types. - Users previously importing buffer from `zarr.core.buffer` should update their - imports to use `zarr.buffer`. As a reminder, all of `zarr.core` is - considered a private API that's not covered by zarr-python's versioning policy. ([#2871](https://github.com/zarr-developers/zarr-python/issues/2871)) + Users previously importing buffer from `zarr.core.buffer` should update their + imports to use `zarr.buffer`. As a reminder, all of `zarr.core` is + considered a private API that's not covered by zarr-python's versioning policy. ([#2871](https://github.com/zarr-developers/zarr-python/issues/2871)) - Adds zarr-specific data type classes. - This change adds a `ZDType` base class for Zarr V2 and Zarr V3 data types. Child classes are - defined for each NumPy data type. Each child class defines routines for `JSON` serialization. - New data types can be created and registered dynamically. + This change adds a `ZDType` base class for Zarr V2 and Zarr V3 data types. Child classes are + defined for each NumPy data type. Each child class defines routines for `JSON` serialization. + New data types can be created and registered dynamically. - Prior to this change, Zarr Python had two streams for handling data types. For Zarr V2 arrays, - we used NumPy data type identifiers. For Zarr V3 arrays, we used a fixed set of string enums. Both - of these systems proved hard to extend. + Prior to this change, Zarr Python had two streams for handling data types. For Zarr V2 arrays, + we used NumPy data type identifiers. For Zarr V3 arrays, we used a fixed set of string enums. Both + of these systems proved hard to extend. - This change is largely internal, but it does change the type of the `dtype` and `data_type` - fields on the `ArrayV2Metadata` and `ArrayV3Metadata` classes. Previously, `ArrayV2Metadata.dtype` - was a NumPy `dtype` object, and `ArrayV3Metadata.data_type` was an internally-defined `enum`. - After this change, both `ArrayV2Metadata.dtype` and `ArrayV3Metadata.data_type` are instances of - `ZDType`. A NumPy data type can be generated from a `ZDType` via the `ZDType.to_native_dtype()` - method. The internally-defined Zarr V3 `enum` class is gone entirely, but the `ZDType.to_json(zarr_format=3)` - method can be used to generate either a string, or dictionary that has a string `name` field, that - represents the string value previously associated with that `enum`. + This change is largely internal, but it does change the type of the `dtype` and `data_type` + fields on the `ArrayV2Metadata` and `ArrayV3Metadata` classes. Previously, `ArrayV2Metadata.dtype` + was a NumPy `dtype` object, and `ArrayV3Metadata.data_type` was an internally-defined `enum`. + After this change, both `ArrayV2Metadata.dtype` and `ArrayV3Metadata.data_type` are instances of + `ZDType`. A NumPy data type can be generated from a `ZDType` via the `ZDType.to_native_dtype()` + method. The internally-defined Zarr V3 `enum` class is gone entirely, but the `ZDType.to_json(zarr_format=3)` + method can be used to generate either a string, or dictionary that has a string `name` field, that + represents the string value previously associated with that `enum`. - For more on this new feature, see the [documentation](user-guide/data_types.md) ([#2874](https://github.com/zarr-developers/zarr-python/issues/2874)) + For more on this new feature, see the [documentation](user-guide/data_types.md) ([#2874](https://github.com/zarr-developers/zarr-python/issues/2874)) - Added `NDBuffer.empty` method for faster ndbuffer initialization. ([#3191](https://github.com/zarr-developers/zarr-python/issues/3191)) @@ -451,10 +456,10 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Fixes a variety of issues related to string data types. - - Brings the `VariableLengthUTF8` data type Zarr V3 identifier in alignment with Zarr Python 3.0.8 - - Disallows creation of 0-length fixed-length data types - - Adds a regression test for the `VariableLengthUTF8` data type that checks against version 3.0.8 - - Allows users to request the `VariableLengthUTF8` data type with `str`, `"str"`, or `"string"`. ([#3170](https://github.com/zarr-developers/zarr-python/issues/3170)) + - Brings the `VariableLengthUTF8` data type Zarr V3 identifier in alignment with Zarr Python 3.0.8 + - Disallows creation of 0-length fixed-length data types + - Adds a regression test for the `VariableLengthUTF8` data type that checks against version 3.0.8 + - Allows users to request the `VariableLengthUTF8` data type with `str`, `"str"`, or `"string"`. ([#3170](https://github.com/zarr-developers/zarr-python/issues/3170)) - Add human readable size for No. bytes stored to `info_complete` ([#3190](https://github.com/zarr-developers/zarr-python/issues/3190)) @@ -475,26 +480,26 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Add a description on how to create a RemoteStore of a specific filesystem to the `Remote Store` section in `docs/user-guide/storage.md`. State in the docstring of `FsspecStore.from_url` that the filesystem type is inferred from the URL scheme. - It should help a user handling the case when the type of FsspecStore doesn't match the URL scheme. ([#3212](https://github.com/zarr-developers/zarr-python/issues/3212)) + It should help a user handling the case when the type of FsspecStore doesn't match the URL scheme. ([#3212](https://github.com/zarr-developers/zarr-python/issues/3212)) ### Deprecations and Removals - Removes default chunk encoding settings (filters, serializer, compressors) from the global configuration object. - This removal is justified on the basis that storing chunk encoding settings in the config required - a brittle, confusing, and inaccurate categorization of array data types, which was particularly - unsuitable after the recent addition of new data types that didn't fit naturally into the - pre-existing categories. + This removal is justified on the basis that storing chunk encoding settings in the config required + a brittle, confusing, and inaccurate categorization of array data types, which was particularly + unsuitable after the recent addition of new data types that didn't fit naturally into the + pre-existing categories. - The default chunk encoding is the same (Zstandard compression, and the required object codecs for - variable length data types), but the chunk encoding is now generated by functions that cannot be - reconfigured at runtime. Users who relied on setting the default chunk encoding via the global configuration object should - instead specify the desired chunk encoding explicitly when creating an array. + The default chunk encoding is the same (Zstandard compression, and the required object codecs for + variable length data types), but the chunk encoding is now generated by functions that cannot be + reconfigured at runtime. Users who relied on setting the default chunk encoding via the global configuration object should + instead specify the desired chunk encoding explicitly when creating an array. - This change also adds an extra validation step to the creation of Zarr V2 arrays, which ensures that - arrays with a `VariableLengthUTF8` or `VariableLengthBytes` data type cannot be created without the - correct "object codec". ([#3228](https://github.com/zarr-developers/zarr-python/issues/3228)) + This change also adds an extra validation step to the creation of Zarr V2 arrays, which ensures that + arrays with a `VariableLengthUTF8` or `VariableLengthBytes` data type cannot be created without the + correct "object codec". ([#3228](https://github.com/zarr-developers/zarr-python/issues/3228)) - Removes support for passing keyword-only arguments positionally to the following functions and methods: `save_array`, `open`, `group`, `open_group`, `create`, `get_basic_selection`, `set_basic_selection`, @@ -546,8 +551,8 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr For large arrays this will result in smaller default chunk sizes. To retain previous behaviour, explicitly set the chunk shape to the data shape. - This fix matches the existing chunking behaviour of - `zarr.save_array` and `zarr.api.asynchronous.AsyncArray.create`. ([#3103](https://github.com/zarr-developers/zarr-python/issues/3103)) + This fix matches the existing chunking behaviour of + `zarr.save_array` and `zarr.api.asynchronous.AsyncArray.create`. ([#3103](https://github.com/zarr-developers/zarr-python/issues/3103)) - When `zarr.save` has an argument `path=some/path/` and multiple arrays in `args`, the path resulted in `some/path/some/path` due to using the `path` argument twice while building the array path. This is now fixed. ([#3127](https://github.com/zarr-developers/zarr-python/issues/3127)) @@ -556,12 +561,12 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Suppress `FileNotFoundError` when deleting non-existent keys in the `obstore` adapter. - When writing empty chunks (i.e. chunks where all values are equal to the array's fill value) to a zarr array, zarr - will delete those chunks from the underlying store. For zarr arrays backed by the `obstore` adapter, this will potentially - raise a `FileNotFoundError` if the chunk doesn't already exist. - Since whether or not a delete of a non-existing object raises an error depends on the behavior of the underlying store, - suppressing the error in all cases results in consistent behavior across stores, and is also what `zarr` seems to expect - from the store. ([#3140](https://github.com/zarr-developers/zarr-python/issues/3140)) + When writing empty chunks (i.e. chunks where all values are equal to the array's fill value) to a zarr array, zarr + will delete those chunks from the underlying store. For zarr arrays backed by the `obstore` adapter, this will potentially + raise a `FileNotFoundError` if the chunk doesn't already exist. + Since whether or not a delete of a non-existing object raises an error depends on the behavior of the underlying store, + suppressing the error in all cases results in consistent behavior across stores, and is also what `zarr` seems to expect + from the store. ([#3140](https://github.com/zarr-developers/zarr-python/issues/3140)) - Trying to open a StorePath/Array with `mode='r'` when the store is not read-only creates a read-only copy of the store. ([#3156](https://github.com/zarr-developers/zarr-python/issues/3156)) @@ -583,8 +588,9 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - It is now possible to specify no compressor when creating a zarr format 2 array. This can be done by passing `compressor=None` to the various array creation routines. - The default behaviour of automatically choosing a suitable default compressor remains if the compressor argument is not given. - To reproduce the behaviour in previous zarr-python versions when `compressor=None` was passed, pass `compressor='auto'` instead. ([#3039](https://github.com/zarr-developers/zarr-python/issues/3039)) + The default behaviour of automatically choosing a suitable default compressor remains if the compressor argument is not given. + To reproduce the behaviour in previous zarr-python versions when `compressor=None` was passed, pass `compressor='auto'` instead. ([#3039](https://github.com/zarr-developers/zarr-python/issues/3039)) + - Fixed the typing of `dimension_names` arguments throughout so that it now accepts iterables that contain `None` alongside `str`. ([#3045](https://github.com/zarr-developers/zarr-python/issues/3045)) - Using various functions to open data with `mode='a'` no longer deletes existing data in the store. ([#3062](https://github.com/zarr-developers/zarr-python/issues/3062)) - Internally use `typesize` constructor parameter for `numcodecs.blosc.Blosc` to improve compression ratios back to the v2-package levels. ([#2962](https://github.com/zarr-developers/zarr-python/issues/2962)) diff --git a/docs/user-guide/data_types.md b/docs/user-guide/data_types.md index ac966f885f..91f828a738 100644 --- a/docs/user-guide/data_types.md +++ b/docs/user-guide/data_types.md @@ -139,17 +139,17 @@ Zarr V3 introduced several key changes to how data types are represented: The basic data types are identified by strings like `"int8"`, `"int16"`, etc., and data types that require a configuration can be identified by a JSON object. - For example, this JSON object declares a datetime data type: - - ```json - { - "name": "numpy.datetime64", - "configuration": { - "unit": "s", - "scale_factor": 10 + For example, this JSON object declares a datetime data type: + + ```json + { + "name": "numpy.datetime64", + "configuration": { + "unit": "s", + "scale_factor": 10 + } } - } - ``` + ``` - Zarr V3 data types do not have endianness. This is a departure from Zarr V2, where multi-byte data types are defined with endianness information. Instead, Zarr V3 requires that the endianness diff --git a/docs/user-guide/v3_migration.md b/docs/user-guide/v3_migration.md index 6d9d516ebe..4d97963be2 100644 --- a/docs/user-guide/v3_migration.md +++ b/docs/user-guide/v3_migration.md @@ -42,40 +42,40 @@ the following actions in order: will be compatible in Zarr-Python 3. However, the following breaking API changes are planned: - - `numcodecs.*` will no longer be available in `zarr.*`. To migrate, import codecs - directly from `numcodecs`: - - ```python exec="false" reason="intentionally shows the old/incorrect import for contrast" - from numcodecs import Blosc - # instead of: - # from zarr import Blosc - ``` - - - The `zarr.v3_api_available` feature flag is being removed. In Zarr-Python 3 - the v3 API is always available, so you shouldn't need to use this flag. - - `zarr.errors` has been consolidated. Several exception classes from - Zarr-Python 2 (such as `zarr.errors.PathNotFoundError`) have been removed - or replaced. For example, missing nodes now raise `zarr.errors.NodeNotFoundError` - (which subclasses both `BaseZarrError` and `FileNotFoundError`) instead of - `zarr.errors.PathNotFoundError`. Review any code that catches exceptions - from `zarr.errors` after migrating. - - The following internal modules are being removed or significantly changed. If - your application relies on imports from any of the below modules, you will need - to either a) modify your application to no longer rely on these imports or b) - vendor the parts of the specific modules that you need. - - * `zarr.attrs` has gone, with no replacement - * `zarr.codecs` has changed, see "Codecs" section below for more information - * `zarr.context` has gone, with no replacement - * `zarr.core` remains but should be considered private API - * `zarr.hierarchy` has gone, with no replacement (use `zarr.Group` in place of `zarr.hierarchy.Group`) - * `zarr.indexing` has gone, with no replacement - * `zarr.meta` has gone, with no replacement - * `zarr.meta_v1` has gone, with no replacement - * `zarr.sync` has gone, with no replacement - * `zarr.types` has gone, with no replacement - * `zarr.util` has gone, with no replacement - * `zarr.n5` has gone, see below for an alternative N5 option + - `numcodecs.*` will no longer be available in `zarr.*`. To migrate, import codecs + directly from `numcodecs`: + + ```python exec="false" reason="intentionally shows the old/incorrect import for contrast" + from numcodecs import Blosc + # instead of: + # from zarr import Blosc + ``` + + - The `zarr.v3_api_available` feature flag is being removed. In Zarr-Python 3 + the v3 API is always available, so you shouldn't need to use this flag. + - `zarr.errors` has been consolidated. Several exception classes from + Zarr-Python 2 (such as `zarr.errors.PathNotFoundError`) have been removed + or replaced. For example, missing nodes now raise `zarr.errors.NodeNotFoundError` + (which subclasses both `BaseZarrError` and `FileNotFoundError`) instead of + `zarr.errors.PathNotFoundError`. Review any code that catches exceptions + from `zarr.errors` after migrating. + - The following internal modules are being removed or significantly changed. If + your application relies on imports from any of the below modules, you will need + to either a) modify your application to no longer rely on these imports or b) + vendor the parts of the specific modules that you need. + + * `zarr.attrs` has gone, with no replacement + * `zarr.codecs` has changed, see "Codecs" section below for more information + * `zarr.context` has gone, with no replacement + * `zarr.core` remains but should be considered private API + * `zarr.hierarchy` has gone, with no replacement (use `zarr.Group` in place of `zarr.hierarchy.Group`) + * `zarr.indexing` has gone, with no replacement + * `zarr.meta` has gone, with no replacement + * `zarr.meta_v1` has gone, with no replacement + * `zarr.sync` has gone, with no replacement + * `zarr.types` has gone, with no replacement + * `zarr.util` has gone, with no replacement + * `zarr.n5` has gone, see below for an alternative N5 option 3. Test that your package works with version 3. 4. Update the pin to include `zarr>=3,<4`. @@ -123,8 +123,9 @@ The following sections provide details on breaking changes in Zarr-Python 3. 2. The h5py compatibility methods `create_dataset` and `require_dataset` have been removed. Use the following replacements: - - [`zarr.Group.create_array`][] in place of `Group.create_dataset` - - [`zarr.Group.require_array`][] in place of `Group.require_dataset` + - [`zarr.Group.create_array`][] in place of `Group.create_dataset` + - [`zarr.Group.require_array`][] in place of `Group.require_dataset` + 3. Disallow "." syntax for getting group members. To get a member of a group named `foo`, use `group["foo"]` in place of `group.foo`. 4. The `zarr.storage.init_group` low-level helper function has been removed. Use From 845a850a4c71f1365d5c89b22efaca4b112b36b6 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 16 Jul 2026 22:48:44 +0200 Subject: [PATCH 5/8] fix(store): copy buffers on write in MemoryStore Encoding an uncompressed chunk hands the store a zero-copy view of the caller's array, and MemoryStore keeps whatever it is given alive in a dict rather than serializing it. Mutating the source array after a write therefore rewrote chunks already committed to the store, silently. Stores that serialize on write (LocalStore, ZipStore, remote stores) are unaffected, so they keep the full benefit of #3885. Only MemoryStore pays the copy, and only where it was aliasing to begin with: an uncompressed 34 MB write goes from ~23 ms to ~39 ms, while compressed writes are unchanged. Copying at the store boundary rather than narrowing the fast path in _merge_chunk_array also fixes the single-chunk case, which aliased in v3.2.1 too. Assisted-by: ClaudeCode:claude-opus-4.8 --- src/zarr/core/buffer/core.py | 2 ++ src/zarr/storage/_memory.py | 24 ++++++++++++++-- tests/test_store/test_memory.py | 51 ++++++++++++++++++++++++++++++++- 3 files changed, 73 insertions(+), 4 deletions(-) diff --git a/src/zarr/core/buffer/core.py b/src/zarr/core/buffer/core.py index 497543f88f..b8f7f11cd4 100644 --- a/src/zarr/core/buffer/core.py +++ b/src/zarr/core/buffer/core.py @@ -45,6 +45,8 @@ def __getitem__(self, key: slice) -> Self: ... def __setitem__(self, key: slice, value: Any) -> None: ... + def copy(self) -> Self: ... + @runtime_checkable class NDArrayLike(Protocol): diff --git a/src/zarr/storage/_memory.py b/src/zarr/storage/_memory.py index 5f5b632d4a..97dd355515 100644 --- a/src/zarr/storage/_memory.py +++ b/src/zarr/storage/_memory.py @@ -26,6 +26,18 @@ logger = getLogger(__name__) +def _copy_buffer(value: Buffer) -> Buffer: + """Copy `value` so the store does not retain the caller's memory. + + Encoding a chunk can hand the store a zero-copy view of the user's array + (an uncompressed write is the common case), and unlike stores that + serialize on write, this one keeps whatever it is given alive in a dict. + Without this copy a later mutation of the user's array would rewrite + chunks already committed to the store. + """ + return type(value).from_array_like(value.as_array_like().copy()) + + class MemoryStore(Store): """ Store for local memory. @@ -42,6 +54,12 @@ class MemoryStore(Store): supports_writes supports_deletes supports_listing + + Notes + ----- + Writes copy the buffer they are given, so the store never aliases the + caller's memory. Buffers passed via `store_dict` are the caller's + responsibility and are stored as-is. """ supports_writes: bool = True @@ -117,7 +135,7 @@ def set_sync(self, key: str, value: Buffer) -> None: raise TypeError( f"MemoryStore.set(): `value` must be a Buffer instance. Got an instance of {type(value)} instead." ) - self._store_dict[key] = value + self._store_dict[key] = _copy_buffer(value) def delete_sync(self, key: str) -> None: self._check_writable() @@ -178,13 +196,13 @@ async def set(self, key: str, value: Buffer, byte_range: tuple[int, int] | None buf[byte_range[0] : byte_range[1]] = value self._store_dict[key] = buf else: - self._store_dict[key] = value + self._store_dict[key] = _copy_buffer(value) async def set_if_not_exists(self, key: str, value: Buffer) -> None: # docstring inherited self._check_writable() await self._ensure_open() - self._store_dict.setdefault(key, value) + self._store_dict.setdefault(key, _copy_buffer(value)) async def delete(self, key: str) -> None: # docstring inherited diff --git a/tests/test_store/test_memory.py b/tests/test_store/test_memory.py index 35504718a7..36265423e6 100644 --- a/tests/test_store/test_memory.py +++ b/tests/test_store/test_memory.py @@ -8,7 +8,7 @@ import pytest import zarr -from zarr.core.buffer import Buffer, cpu, gpu +from zarr.core.buffer import Buffer, cpu, default_buffer_prototype, gpu from zarr.errors import ZarrUserWarning from zarr.storage import GpuMemoryStore, ManagedMemoryStore, MemoryStore from zarr.testing.store import StoreTests @@ -76,6 +76,55 @@ async def test_deterministic_size( np.testing.assert_array_equal(a[:3], 1) np.testing.assert_array_equal(a[3:], 0) + @pytest.mark.parametrize("method", ["set", "set_sync", "set_if_not_exists"]) + async def test_set_does_not_retain_caller_buffer(self, store: MemoryStore, method: str) -> None: + """Writing a buffer must not alias the caller's memory. + + MemoryStore keeps whatever it is handed alive in a dict, so retaining + the caller's buffer lets a later mutation of that buffer rewrite data + already committed to the store. + """ + source = np.frombuffer(bytearray(b"\x01\x02\x03\x04"), dtype="B") + value = cpu.Buffer.from_array_like(source) + + if method == "set_sync": + store.set_sync("k", value) + else: + await getattr(store, method)("k", value) + + source[:] = 0xF # mutate the caller's memory after the write + stored = await store.get("k", prototype=default_buffer_prototype()) + assert stored is not None + assert stored.to_bytes() == b"\x01\x02\x03\x04" + + @pytest.mark.parametrize( + "pipeline", + [ + "zarr.core.codec_pipeline.BatchedCodecPipeline", + "zarr.core.codec_pipeline.FusedCodecPipeline", + ], + ) + @pytest.mark.parametrize(("shape", "chunks"), [((30,), (10,)), ((8,), (4,)), ((4,), (4,))]) + def test_write_does_not_alias_source_array( + self, pipeline: str, shape: tuple[int], chunks: tuple[int] + ) -> None: + """Mutating the source array after a write must not corrupt stored chunks. + + Without compression the encoded buffer is a zero-copy view of the + caller's array all the way down to the store, so this covers both the + single-chunk and multi-chunk write paths. + """ + with zarr.config.set({"codec_pipeline.path": pipeline}): + array = zarr.create_array( + store=MemoryStore(), shape=shape, chunks=chunks, dtype="i4", compressors=None + ) + source = np.arange(shape[0], dtype="i4") + expected = source.copy() + array[:] = source + source[:] = -1 + + np.testing.assert_array_equal(array[:], expected) + # --- byte-range-write tests: disabled --- # Byte-range-write support (set_range / set_range_sync / SupportsSetRange) # was removed from this PR pending a decision on the store interface. These From af10bfe051cf73d5e15bce82118f9952c77cfbb9 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 16 Jul 2026 22:49:19 +0200 Subject: [PATCH 6/8] docs: add changelog entry for MemoryStore buffer copy Assisted-by: ClaudeCode:claude-opus-4.8 --- changes/224.bugfix.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 changes/224.bugfix.md diff --git a/changes/224.bugfix.md b/changes/224.bugfix.md new file mode 100644 index 0000000000..6b0d0fcc67 --- /dev/null +++ b/changes/224.bugfix.md @@ -0,0 +1,11 @@ +`MemoryStore` now copies buffers as they are written, so it never retains the +caller's memory. Previously an uncompressed write handed the store a zero-copy +view of the user's array, and mutating that array afterwards would silently +rewrite chunks already committed to the store. + +Only `MemoryStore` is affected: stores that serialize on write, such as +`LocalStore` and `ZipStore`, never aliased the caller's memory. Uncompressed +writes to a `MemoryStore` are correspondingly slower, since the copy that makes +the stored data independent is now actually performed; compressed writes are +unchanged. Buffers supplied through the `store_dict` argument remain the +caller's responsibility and are stored as-is. From 326545217ffe60112b2da1175a7423ff4acb7219 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 17 Jul 2026 13:18:56 +0200 Subject: [PATCH 7/8] chore(deps): bump the actions group across 1 directory with 6 updates (#4152) Bumps the actions group with 6 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `6.0.3` | `7.0.0` | | [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `8.2.0` | `8.3.2` | | [CodSpeedHQ/action](https://github.com/codspeedhq/action) | `4.18.1` | `4.18.5` | | [github-community-projects/issue-metrics](https://github.com/github-community-projects/issue-metrics) | `4.2.8` | `5.0.0` | | [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) | `2.8.0` | `2.9.0` | | [actions/labeler](https://github.com/actions/labeler) | `6.1.0` | `6.2.0` | Updates `actions/checkout` from 6.0.3 to 7.0.0 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v6.0.3...9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0) Updates `astral-sh/setup-uv` from 8.2.0 to 8.3.2 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](https://github.com/astral-sh/setup-uv/compare/fac544c07dec837d0ccb6301d7b5580bf5edae39...11f9893b081a58869d3b5fccaea48c9e9e46f990) Updates `CodSpeedHQ/action` from 4.18.1 to 4.18.5 - [Release notes](https://github.com/codspeedhq/action/releases) - [Changelog](https://github.com/CodSpeedHQ/action/blob/main/CHANGELOG.md) - [Commits](https://github.com/codspeedhq/action/compare/a4a36bb07c0638b0b4ca52bf1f3dad1b4289e52f...f99becdce5e5d51fd556489ebef684f4ecfd6286) Updates `github-community-projects/issue-metrics` from 4.2.8 to 5.0.0 - [Release notes](https://github.com/github-community-projects/issue-metrics/releases) - [Commits](https://github.com/github-community-projects/issue-metrics/compare/44173f9e0a3b2144a777a10a340e4c09a25ac9f8...df8c49d20958f9345281fa2124858bd0ad227e1f) Updates `lycheeverse/lychee-action` from 2.8.0 to 2.9.0 - [Release notes](https://github.com/lycheeverse/lychee-action/releases) - [Commits](https://github.com/lycheeverse/lychee-action/compare/8646ba30535128ac92d33dfc9133794bfdd9b411...e7477775783ea5526144ba13e8db5eec57747ce8) Updates `actions/labeler` from 6.1.0 to 6.2.0 - [Release notes](https://github.com/actions/labeler/releases) - [Commits](https://github.com/actions/labeler/compare/f27b608878404679385c85cfa523b85ccb86e213...b8dd2d9be0f68b860e7dae5dae7d772984eacd6d) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/labeler dependency-version: 6.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: astral-sh/setup-uv dependency-version: 8.3.2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: CodSpeedHQ/action dependency-version: 4.18.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: github-community-projects/issue-metrics dependency-version: 5.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: lycheeverse/lychee-action dependency-version: 2.9.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/check_changelogs.yml | 2 +- .github/workflows/codspeed.yml | 2 +- .github/workflows/docs.yml | 2 +- .github/workflows/downstream.yml | 4 ++-- .github/workflows/gpu_test.yml | 2 +- .github/workflows/hypothesis.yaml | 2 +- .github/workflows/issue-metrics.yml | 2 +- .github/workflows/links.yml | 4 ++-- .github/workflows/lint.yml | 2 +- .github/workflows/needs_release_notes.yml | 2 +- .github/workflows/test.yml | 8 ++++---- .github/workflows/zarr-metadata-release.yml | 2 +- .github/workflows/zarr-metadata.yml | 6 +++--- 13 files changed, 20 insertions(+), 20 deletions(-) diff --git a/.github/workflows/check_changelogs.yml b/.github/workflows/check_changelogs.yml index 25034b868d..0033b43db2 100644 --- a/.github/workflows/check_changelogs.yml +++ b/.github/workflows/check_changelogs.yml @@ -22,7 +22,7 @@ jobs: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Check zarr-python changelog entries run: uv run --no-sync python ci/check_changelog_entries.py diff --git a/.github/workflows/codspeed.yml b/.github/workflows/codspeed.yml index 1d8e847ec5..39cd8eb261 100644 --- a/.github/workflows/codspeed.yml +++ b/.github/workflows/codspeed.yml @@ -32,7 +32,7 @@ jobs: with: version: '1.16.5' - name: Run the benchmarks - uses: CodSpeedHQ/action@a4a36bb07c0638b0b4ca52bf1f3dad1b4289e52f # v4.18.1 + uses: CodSpeedHQ/action@f99becdce5e5d51fd556489ebef684f4ecfd6286 # v4.18.5 with: mode: walltime run: hatch run test.py3.12-minimal:pytest tests/benchmarks --codspeed diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index c72b493b12..baf9233fc7 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -22,7 +22,7 @@ jobs: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 with: persist-credentials: false - - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + - uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - run: uv sync --group docs # Fast source-level guards that need no built site, so they run before the (slower) # build for a quick failure: every public export is in the API reference, and no diff --git a/.github/workflows/downstream.yml b/.github/workflows/downstream.yml index d8aa2907be..f65f8d47e3 100644 --- a/.github/workflows/downstream.yml +++ b/.github/workflows/downstream.yml @@ -45,7 +45,7 @@ jobs: python-version: '3.13' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install xarray and test dependencies working-directory: xarray @@ -102,7 +102,7 @@ jobs: python-version: '3.13' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install numcodecs with test-zarr-main group working-directory: numcodecs diff --git a/.github/workflows/gpu_test.yml b/.github/workflows/gpu_test.yml index f2bf4907fa..bbbb3e5133 100644 --- a/.github/workflows/gpu_test.yml +++ b/.github/workflows/gpu_test.yml @@ -62,7 +62,7 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch uses: pypa/hatch@257e27e51a6a5616ed08a39a408a21c35c9931bc with: diff --git a/.github/workflows/hypothesis.yaml b/.github/workflows/hypothesis.yaml index 01212dfb56..e836f30a5b 100644 --- a/.github/workflows/hypothesis.yaml +++ b/.github/workflows/hypothesis.yaml @@ -57,7 +57,7 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch uses: pypa/hatch@257e27e51a6a5616ed08a39a408a21c35c9931bc with: diff --git a/.github/workflows/issue-metrics.yml b/.github/workflows/issue-metrics.yml index 53860d21c8..adbd2748a0 100644 --- a/.github/workflows/issue-metrics.yml +++ b/.github/workflows/issue-metrics.yml @@ -33,7 +33,7 @@ jobs: echo "last_month=$first_day..$last_day" >> "$GITHUB_ENV" - name: Run issue-metrics tool - uses: github-community-projects/issue-metrics@44173f9e0a3b2144a777a10a340e4c09a25ac9f8 # v4.2.8 + uses: github-community-projects/issue-metrics@df8c49d20958f9345281fa2124858bd0ad227e1f # v5.0.0 env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} SEARCH_QUERY: 'repo:zarr-developers/zarr-python is:issue created:${{ env.last_month }} -reason:"not planned"' diff --git a/.github/workflows/links.yml b/.github/workflows/links.yml index a53de8da83..f606f12a3e 100644 --- a/.github/workflows/links.yml +++ b/.github/workflows/links.yml @@ -13,13 +13,13 @@ jobs: permissions: issues: write # required for peter-evans/create-issue-from-file steps: - - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 with: persist-credentials: false - name: Link Checker id: lychee - uses: lycheeverse/lychee-action@8646ba30535128ac92d33dfc9133794bfdd9b411 # v2.8.0 + uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2.9.0 with: fail: false diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 8190b1e061..dacba6648f 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -27,7 +27,7 @@ jobs: with: python-version: "3.12" - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 with: enable-cache: true - uses: j178/prek-action@e98a699c41eb69ab013a45817a0406469a748f8d # v2.0.5 diff --git a/.github/workflows/needs_release_notes.yml b/.github/workflows/needs_release_notes.yml index 1f79725b86..fa555d1478 100644 --- a/.github/workflows/needs_release_notes.yml +++ b/.github/workflows/needs_release_notes.yml @@ -21,7 +21,7 @@ jobs: pull-requests: write # Required to add labels to PRs runs-on: ubuntu-latest steps: - - uses: actions/labeler@f27b608878404679385c85cfa523b85ccb86e213 # v6.1.0 + - uses: actions/labeler@b8dd2d9be0f68b860e7dae5dae7d772984eacd6d # v6.2.0 with: repo-token: ${{ secrets.GITHUB_TOKEN }} sync-labels: true diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 7bc43512b5..ab78cfbe2b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -66,7 +66,7 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch run: python -m pip install hatch==1.16.5 - name: Set Up Hatch Env @@ -115,7 +115,7 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch run: python -m pip install hatch==1.16.5 - name: Set Up Hatch Env @@ -150,7 +150,7 @@ jobs: python-version: '3.13' cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch run: python -m pip install hatch==1.16.5 - name: Set Up Hatch Env @@ -174,7 +174,7 @@ jobs: python-version: '3.13' cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch run: python -m pip install hatch==1.16.5 - name: Run Benchmarks diff --git a/.github/workflows/zarr-metadata-release.yml b/.github/workflows/zarr-metadata-release.yml index db05489798..5021f79d2e 100644 --- a/.github/workflows/zarr-metadata-release.yml +++ b/.github/workflows/zarr-metadata-release.yml @@ -51,7 +51,7 @@ jobs: path: dist - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 with: enable-cache: false diff --git a/.github/workflows/zarr-metadata.yml b/.github/workflows/zarr-metadata.yml index 95e8251227..4e5bb0fb1a 100644 --- a/.github/workflows/zarr-metadata.yml +++ b/.github/workflows/zarr-metadata.yml @@ -36,7 +36,7 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 with: enable-cache: true - name: Set up Python ${{ matrix.python-version }} @@ -58,7 +58,7 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Run ruff run: uvx ruff check . @@ -74,7 +74,7 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 with: enable-cache: true - name: Set up Python From cd68f10c832a678848d4939e77d79bc87f4215a2 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 17 Jul 2026 13:45:34 +0200 Subject: [PATCH 8/8] docs: correct changelog --- changes/{224.bugfix.md => 4157.bugfix.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename changes/{224.bugfix.md => 4157.bugfix.md} (100%) diff --git a/changes/224.bugfix.md b/changes/4157.bugfix.md similarity index 100% rename from changes/224.bugfix.md rename to changes/4157.bugfix.md