Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions changes/4174.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Array creation is now O(1) in the number of chunks per dimension. Chunk
normalization returns a `ChunkGrid` whose uniform dimensions are stored as a
size + extent pair (`FixedDimension`) instead of being expanded to one entry
per chunk, so creating arrays like
`zarr.create_array(store, shape=(2**62,), chunks=(1,), dtype='int32')` succeeds
instantly instead of raising `ValueError` or allocating gigabytes of memory.
The intermediate `ChunksTuple` representation was removed in the process, and
`ChunksLike` now admits per-dimension specs that mix a bare int (uniform chunk
size) with explicit edge-length sequences, matching what the normalizer and
the rectilinear grid spec already accepted.
This is the creation-time counterpart of the indexing fix in #4172.
22 changes: 22 additions & 0 deletions changes/4272.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
Explicit per-chunk size lists now always produce a rectilinear chunk grid,
even when the sizes happen to describe a regular grid (all equal, or all equal
with a smaller trailing chunk). Previously such input was silently collapsed to
a regular grid, which changed resize semantics: a regular grid grows by
extending the uniform pattern, while a rectilinear grid appends a new edge
chunk — the behavior an append-oriented layout like `(168,) * 13 + (24,)`
relies on. The grid kind now follows the input syntax, matching 3.2.x:
scalar chunk sizes (including numpy integers and the `-1` sentinel) produce a
regular grid, nested sequences produce a rectilinear grid. Rectilinear grids
remain gated behind `zarr.config.set({"array.rectilinear_chunks": True})`.
See #4174 for the accompanying O(1) chunk normalization change.

`zarr.from_array` with the default `chunks="keep"` / `shards="keep"` now
reproduces the source's stored grid exactly: a rectilinear grid is passed
through in O(number of dimensions), with uniform dimensions keeping their
bare-int shorthand; sharding under a rectilinear shard grid is preserved
instead of being silently dropped; and the default `write_data=True` copy
works for every grid kind. `Array.chunks` is now defined for any sharded array
(the inner chunks of a shard are always regular), and for sharded arrays with
a rectilinear shard grid `Array.info` no longer raises — it reports the shard
shape as `<variable>` — while `Array.nchunks_initialized` counts the chunks of
each initialized shard individually instead of raising.
18 changes: 16 additions & 2 deletions docs/user-guide/arrays.md
Original file line number Diff line number Diff line change
Expand Up @@ -674,8 +674,11 @@ z_regular = zarr.create_array(
print(z_regular.write_chunk_sizes)
```

Note that the `.chunks` property is only available for regular chunk grids. For
rectilinear arrays, use `.write_chunk_sizes` (or `.read_chunk_sizes`) instead.
Note that the `.chunks` property is not available for non-sharded rectilinear
arrays, since there is no single uniform chunk shape — use `.write_chunk_sizes`
(or `.read_chunk_sizes`) instead. Sharded arrays always have `.chunks`: it
returns the inner chunk shape, which is regular even when the shard grid is
rectilinear (see [Rectilinear shard boundaries](#rectilinear-shard-boundaries)).

### Resizing and appending

Expand Down Expand Up @@ -746,6 +749,17 @@ print(z[50:70, 40:60])
Note that rectilinear inner chunks with sharding are not supported — only the
shard boundaries can be rectilinear.

For such arrays, `.chunks` returns the (regular) inner chunk shape, while
`.shards` raises `NotImplementedError` since there is no single uniform shard
shape — use `.write_chunk_sizes` for the per-dimension shard sizes. `.info`
reports the shard shape as `<variable>`:

```python exec="true" session="arrays" source="above" result="ansi"
print(f"chunks={z.chunks}")
print(f"shard sizes={z.write_chunk_sizes}")
print(z.info)
```

### Metadata format

Rectilinear chunk grid metadata uses run-length encoding (RLE) for compact
Expand Down
4 changes: 3 additions & 1 deletion src/zarr/core/_info.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,9 @@ class ArrayInfo:
_data_type: ZDType[TBaseDType, TBaseScalar]
_fill_value: object
_shape: tuple[int, ...]
_shard_shape: tuple[int, ...] | None = None
# "<variable>" marks a sharded array whose shard grid is rectilinear, so
# there is no uniform shard shape; None means the array is not sharded.
_shard_shape: tuple[int, ...] | Literal["<variable>"] | None = None
_chunk_shape: tuple[int, ...] | None = None
_order: Literal["C", "F"]
_read_only: bool
Expand Down
Loading
Loading