fix(chunk-grids): make array creation O(1) in per-dimension chunk count

[d-v-b]

Summary

this is a claude-authored fix of ##4174. in ChunkLayout, instead of materializing all chunk specifications, even regular ones, as numpy array, we specifically model regular chunk grid dimensions as a single shape + extent. This means ChunksTuple can just disappear, and we can re-use our chunk grid abstractions (FixedDimension and VaryingDimension), which is nice and simple. This avoids o(num chunks) memory problems and avoids some unnecessary abstractions.

this is basically what @maxrjones suggested here: #3899 (comment). My instinct to make the API "uniform" came at the cost of an inefficient representation, so I'm pulling back on the commitment to "uniformity" here.

Here's claude's summary:

Chunk normalization now returns a ChunkGrid whose uniform dimensions are stored as FixedDimension (size + extent) instead of one array entry per chunk, so create_array(shape=(262,), chunks=(1,)) succeeds instantly instead of raising "array is too big", and chunks=(1, 1) on a (231, 2**31) array no longer allocates ~17 GB per dimension.

Explicit per-chunk lists collapse to FixedDimension when they describe a regular grid; genuinely irregular lists become VaryingDimension. Both variants bind chunk sizes to their extent, so the two forms carry the same invariants. The intermediate ChunksTuple type and as_regular_shape helper are removed; create_chunk_grid_metadata consumes the grid directly and serializes uniform dimensions of mixed rectilinear grids as the spec's bare-int step-size shorthand.

Creation-time counterpart of the gh-4174 indexing fix.

Assisted-by: ClaudeCode:claude-fable-5

[Describe what this PR changes and why, in your own words.]

For reviewers

Check if this purported simplification is in fact simpler, and confirm that we don't see any unpleasant performance side effects.

Author attestation

• I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.md
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.20%. Comparing base (7729d92) to head (ed26f62).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4218      +/-   ##
==========================================
+ Coverage   94.19%   94.20%   +0.01%     
==========================================
  Files          92       92              
  Lines       12825    12830       +5     
==========================================
+ Hits        12080    12087       +7     
+ Misses        745      743       -2     

Files with missing lines	Coverage Δ
src/zarr/core/array.py	98.05% <100.00%> (+0.10%)	⬆️
src/zarr/core/chunk_grids.py	96.91% <100.00%> (-0.06%)	⬇️
src/zarr/core/common.py	90.47% <100.00%> (+0.59%)	⬆️
src/zarr/core/metadata/v3.py	94.21% <100.00%> (+0.10%)	⬆️
src/zarr/testing/strategies.py	95.45% <100.00%> (+0.02%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codspeed-hq]

Merging this PR will improve performance by 11.82%

⚡ 1 improved benchmark
✅ 108 untouched benchmarks
🆕 4 new benchmarks
⏩ 37 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	WallTime	test_read_array[latency=0.03-batched-memory-Layout(shape=(100000000,), chunks=(100000,), shards=(100000000,))-None-repeated]	572.2 ms	511.7 ms	+11.82%
🆕	WallTime	test_read_array_concurrent[latency=0-local-None-batched]	N/A	43.7 ms	N/A
🆕	WallTime	test_read_array_concurrent[latency=0-local-None-fused_full_threaded]	N/A	30.3 ms	N/A
🆕	WallTime	test_read_array_concurrent[latency=0-local-zstd-batched]	N/A	70.2 ms	N/A
🆕	WallTime	test_read_array_concurrent[latency=0-local-zstd-fused_full_threaded]	N/A	61.9 ms	N/A

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing d-v-b:claude/optimistic-diffie-2ef839 (8c192b2) with main (80e00ae)²}

Footnotes

1. 37 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on main (6f9724c) during the generation of this report, so 80e00ae was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[d-v-b]

🤖 AI text below 🤖

Rebased onto current main (7729d92) and added a fix for #4272 on top. Summary of what changed:

Rebase

The conflict was with #4257 ("accept numpy integers as chunk sizes"), which landed after this branch was cut and rewrote the same region of normalize_chunks_1d. Its semantics are preserved inside the new FixedDimension/VaryingDimension code:

• scalars are narrowed with numbers.Integral (not int) and coerced via int(), so numpy integer scalars take the uniform-chunk path;
• the -1 sentinel check lives inside the integral branch, so numpy-array chunk specs never hit an ambiguous-truth-value error on chunks == -1;
• the improved TypeError for non-integer, non-iterable specs is kept, along with fix: accept numpy integers as chunk sizes #4257's tests (translated to the FixedDimension/VaryingDimension expectation format).

normalize_chunks_1d no longer needs main's span == 0 special case — FixedDimension handles zero extents natively. In zarr/testing/stateful.py the branch's chunks=-reconstruction hunk was dropped in favor of main's newer zarr.from_array(..., write_data=False) approach; chunks_param_from_rectilinear stays, since the arrays() strategy still uses it.

New commit: explicit per-chunk lists stay rectilinear (fixes #4272)

The explicit-list branch of normalize_chunks_1d previously collapsed uniform (or uniform-plus-short-tail) lists to FixedDimension, so an explicitly rectilinear spec like [[10, 10, 4]] was silently stored as a regular grid — which changes resize semantics (a regular grid grows by extending the uniform pattern; a rectilinear grid appends an edge chunk), breaking append-oriented layouts like (168,) * 13 + (24,). That collapse is now removed: the grid kind follows the input syntax, matching 3.2.x — scalar specs (ints, numpy integers, -1) produce FixedDimension → regular grid (the O(1) path this PR is about), explicit lists always produce VaryingDimension → rectilinear grid. Uniform dimensions of mixed scalar/list specs still serialize as the spec's bare-int step-size shorthand. New tests cover the syntax→kind mapping and the resize divergence (the rectilinear append touches exactly one new chunk; the regular pattern-extension straddles two).

Relation to #4290

#4290 fixes the same issue on current main by threading a requested_rectilinear flag from the raw input into create_chunk_grid_metadata, because main's ChunksTuple normalization erases the form of the request. With this PR's representation the flag isn't needed — FixedDimension vs VaryingDimension records the request's form, so create_chunk_grid_metadata dispatches on the grid with no call-site plumbing, and rectilinear shards= producing a rectilinear outer grid falls out automatically. End behavior matches #4290 (including uniform nested specs now hitting the array.rectilinear_chunks gate). Credit to #4290 for the touched-chunk-keys assertion style, which the resize regression test here adopts.

[d-v-b]

@maxrjones could you review this when you get a chance? It fixes two issues and so I consider it high priority for our next release