feat(zarr-metadata): Zarr metadata model layer

.github/workflows/zarr-metadata.yml

@@ -82,7 +82,10 @@ jobs:
       - name: Sync test dependency group
         run: uv sync --group test --python 3.11
       - name: Run pyright
-        run: uv run --group test --with pyright pyright src
+        # Pinned to the last version that types PEP 661 sentinels in class
+        # attributes correctly; 1.1.405+ regressed (microsoft/pyright#11115).
+        # Unpin when the fix lands.
+        run: uv run --group test --with 'pyright==1.1.404' pyright src
 
   zarr-metadata-complete:
     name: zarr-metadata complete

packages/zarr-metadata/README.md

@@ -20,12 +20,12 @@ Zarr metadata. Pair them with a runtime validator like
 ```python
 import json
 from pydantic import TypeAdapter
-from zarr_metadata.v3.array import ArrayMetadataV3
+from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON
 
 with open("zarr.json", "rb") as f:
     raw = json.load(f)
 
-metadata = TypeAdapter(ArrayMetadataV3).validate_python(raw)
+metadata = TypeAdapter(ZarrV3ArrayMetadataJSON).validate_python(raw)
 ```
 
 ## What this is *not*

packages/zarr-metadata/changes/210.feature.md

@@ -0,0 +1,69 @@
+Added `zarr_metadata.model`: frozen-dataclass models (`ZarrV2ArrayMetadata`,
+`ZarrV3ArrayMetadata`, `ZarrV2GroupMetadata`, `ZarrV3GroupMetadata`,
+`ZarrV2ConsolidatedMetadata`, `ZarrV3ConsolidatedMetadata`, `ZarrV3NamedConfig`)
+that are canonical, lossless representations of Zarr metadata documents, plus
+structural validators (`validate_*` / `is_*` / `parse_*`). Every v3 extension
+point (data type, chunk grid, chunk key encoding, codecs, storage transformers)
+is held as a name + configuration pair; nothing is interpreted. Model fields
+are annotated with the role alias `ZarrV3MetadataField` (today exactly
+`ZarrV3NamedConfig`), so the annotations convey the logical meaning of the
+field and stay put if the spec ever adds a new field form.
+
+Validation is strict about what the types declare: v2 `dtype` / `order` /
+`compressor` / `filters` / `dimension_separator` shapes and the fixed
+`zarr_format` / `node_type` literals are all enforced. Every
+`ValidationProblem` carries a machine-readable `kind`
+(`missing_key` / `invalid_type` / `invalid_value` / `invalid_json`) so
+consumers can dispatch on the failure mode without matching message strings,
+and every ingestion failure — including missing store keys and undecodable
+bytes in `from_key_value` — surfaces as `MetadataValidationError`. An
+adversarial review added further structural checks: JSON booleans are not
+accepted as dimension lengths, dimensions are non-negative,
+`dimension_names` must have one entry per dimension of `shape`, `attributes`
+and `configuration` values are JSON-checked recursively (like `fill_value`),
+and the inline consolidated-metadata envelope and entries are deep-validated
+so the group validator's verdict always agrees with the model constructor.
+
+The v3 models expose `must_understand_fields`: the subset of `extra_fields`
+not explicitly waived with `must_understand: false` (fields are implicitly
+must-understand per the spec). Readers discharge the spec's fail-to-open
+duty by subtracting the extension names they recognize; the model only
+partitions by obligation, since recognition is reader-specific.
+
+Optional pydantic integration ships as `zarr_metadata.pydantic` (importing it
+requires pydantic v2; the core package does not depend on it): one `Annotated`
+field type per model, validating raw documents through `from_json`, passing
+core-model instances through unchanged, and serializing via `to_json`. The
+instances are the core model classes, so values interoperate freely with
+non-pydantic code.
+
+`create_default` keeps its output self-consistent: overriding `shape` without
+a chunk grid derives one regular chunk covering the array (v3
+`chunk_shape == shape`; v2 `chunks == shape`) instead of silently keeping the
+scalar default's 0-d grid.
+
+A v2 `.zarray` that omits `dimension_separator` is interpreted with the v2
+convention's default `"."` (the model previously normalized absence to `"/"`,
+which would misaddress the chunks of real-world default-separator arrays).
+The value is never null: absent, `"."`, or `"/"` are the only spellings.
+
+Optional document keys use `UNSET` — a PEP 661 sentinel
+(`typing_extensions.Sentinel`), usable directly in type expressions — never
+`None`: in a model, `None` always corresponds to a JSON `null` in the
+document (a v2 `compressor`, an unnamed dimension inside `dimension_names`),
+and `UNSET` always means the key is absent. Checker note: ty types the
+sentinel exactly; pyright needs `<= 1.1.404` until microsoft/pyright#11115
+is fixed (this package's CI pins it); mypy users need a `cast` or
+`type: ignore` at narrowing sites until python/mypy#21647 merges. This keeps semantically distinct spellings
+distinct — an absent `dimension_names` ("there are no dimension names") and
+an explicit `[null, null]` ("every dimension has a name, which is null") are
+different documents and round-trip as such. The `consolidated_metadata: null`
+written by a historical zarr-python bug is the one deliberate exception to
+faithful round-tripping: those stores remain readable, but the bug spelling
+is repaired to absence on read and never written back.
+
+The v2 models treat the `.zattrs` file's presence as part of the store:
+`attributes` is `UNSET` when no `.zattrs` file exists (and `to_key_value`
+emits none), while an explicit empty `.zattrs` is `{}` and round-trips as a
+file. Previously `to_key_value` always emitted `.zattrs`, silently adding a
+file to stores that never had one.

packages/zarr-metadata/changes/4119.removal.md

@@ -0,0 +1,23 @@
+The document (TypedDict) types are renamed to put the format version at the
+front of the name and to mark the JSON-document form with a `JSON` suffix,
+so a format version can never be misread as a class revision and the bare
+entity names are reserved for the `zarr_metadata.model` dataclasses:
+
+- `ArrayMetadataV2` → `ZarrV2ArrayMetadataJSON` (and `...Partial` accordingly)
+- `ArrayMetadataV3` → `ZarrV3ArrayMetadataJSON` (and `...Partial` accordingly)
+- `GroupMetadataV2` → `ZarrV2GroupMetadataJSON` (and `...Partial` accordingly)
+- `GroupMetadataV3` → `ZarrV3GroupMetadataJSON` (and `...Partial` accordingly)
+- `ConsolidatedMetadataV2` → `ZarrV2ConsolidatedMetadataJSON`
+- `ConsolidatedMetadataV3` → `ZarrV3ConsolidatedMetadataJSON`
+- `NamedConfigV3` → `ZarrV3NamedConfigJSON`
+- `MetadataV3` → `ZarrV3MetadataFieldJSON` (the union of the bare-name and
+  named-configuration spellings of one metadata field)
+- `ExtensionFieldV3` → `ZarrV3ExtensionField`
+- `CodecMetadataV2` → `ZarrV2CodecMetadata`
+- `DataTypeMetadataV2` → `ZarrV2DataTypeMetadata`
+- `ArrayOrderV2` → `ZarrV2ArrayOrder`
+- `ArrayDimensionSeparatorV2` → `ZarrV2ArrayDimensionSeparator`
+
+The old names are removed, not aliased. The `zarr_metadata.pydantic` field
+types take the bare entity names (`ZarrV3ArrayMetadata`, ...), matching the
+model classes they validate into.

packages/zarr-metadata/pyproject.toml

@@ -32,7 +32,11 @@ classifiers = [
 ]
 keywords = ["zarr"]
 dependencies = [
-    "typing_extensions>=4.13",
+    # >=4.16: first release where `Sentinel` pickles by reference
+    # (`__reduce__` returns the sentinel's name), so `UNSET` — and any model
+    # holding it — can cross process boundaries and be deep-copied with its
+    # singleton identity intact. 4.15 and earlier refuse to pickle sentinels.
+    "typing_extensions>=4.16",
 ]
 
 [project.urls]
@@ -82,6 +86,9 @@ checks = [
     "PR06",
 ]
 
+# CI pins pyright==1.1.404: later versions regress PEP 661 sentinel typing in
+# class attributes (microsoft/pyright#11115), which zarr_metadata.model._sentinel
+# relies on. Use the same pin locally; unpin when the fix lands.
 [tool.pyright]
 include = ["src"]
 enableExperimentalFeatures = true

packages/zarr-metadata/src/zarr_metadata/__init__.py

@@ -1,22 +1,48 @@
 from importlib.metadata import version
 
-from zarr_metadata._common import JSONValue, NamedConfigV3
+from zarr_metadata._common import JSONValue, ZarrV3NamedConfigJSON
+from zarr_metadata.model import (
+    UNSET,
+    MetadataValidationError,
+    ProblemKind,
+    ValidationProblem,
+    ZarrV2ArrayMetadata,
+    ZarrV2ArrayMetadataPartial,
+    ZarrV2ConsolidatedMetadata,
+    ZarrV2GroupMetadata,
+    ZarrV2GroupMetadataPartial,
+    ZarrV3ArrayMetadata,
+    ZarrV3ArrayMetadataPartial,
+    ZarrV3ConsolidatedMetadata,
+    ZarrV3GroupMetadata,
+    ZarrV3GroupMetadataPartial,
+    ZarrV3MetadataField,
+    ZarrV3NamedConfig,
+)
 from zarr_metadata.v2.array import (
     ARRAY_DIMENSION_SEPARATOR_V2,
     ARRAY_ORDER_V2,
-    ArrayDimensionSeparatorV2,
-    ArrayMetadataV2,
-    ArrayMetadataV2Partial,
-    ArrayOrderV2,
-    DataTypeMetadataV2,
     ZArrayMetadata,
+    ZarrV2ArrayDimensionSeparator,
+    ZarrV2ArrayMetadataJSON,
+    ZarrV2ArrayMetadataJSONPartial,
+    ZarrV2ArrayOrder,
+    ZarrV2DataTypeMetadata,
 )
 from zarr_metadata.v2.attributes import ZAttrsMetadata
-from zarr_metadata.v2.codec import CodecMetadataV2
-from zarr_metadata.v2.consolidated import ConsolidatedMetadataV2
-from zarr_metadata.v2.group import GroupMetadataV2, GroupMetadataV2Partial, ZGroupMetadata
-from zarr_metadata.v3._common import MetadataV3
-from zarr_metadata.v3.array import ArrayMetadataV3, ArrayMetadataV3Partial, ExtensionFieldV3
+from zarr_metadata.v2.codec import ZarrV2CodecMetadata
+from zarr_metadata.v2.consolidated import ZarrV2ConsolidatedMetadataJSON
+from zarr_metadata.v2.group import (
+    ZarrV2GroupMetadataJSON,
+    ZarrV2GroupMetadataJSONPartial,
+    ZGroupMetadata,
+)
+from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON
+from zarr_metadata.v3.array import (
+    ZarrV3ArrayMetadataJSON,
+    ZarrV3ArrayMetadataJSONPartial,
+    ZarrV3ExtensionField,
+)
 from zarr_metadata.v3.chunk_grid.rectilinear import (
     RECTILINEAR_CHUNK_GRID_NAME,
     RectilinearChunkGridMetadata,
@@ -86,7 +112,7 @@
     TransposeCodecName,
 )
 from zarr_metadata.v3.codec.zstd import ZSTD_CODEC_NAME, ZstdCodecMetadata, ZstdCodecName
-from zarr_metadata.v3.consolidated import ConsolidatedMetadataV3
+from zarr_metadata.v3.consolidated import ZarrV3ConsolidatedMetadataJSON
 from zarr_metadata.v3.data_type.bool import (
     BOOL_DATA_TYPE_NAME,
     BoolDataTypeName,
@@ -185,7 +211,7 @@
     Uint64DataTypeName,
     Uint64FillValue,
 )
-from zarr_metadata.v3.group import GroupMetadataV3, GroupMetadataV3Partial
+from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON, ZarrV3GroupMetadataJSONPartial
 
 __version__ = version("zarr-metadata")
 
@@ -231,15 +257,10 @@
     "UINT16_DATA_TYPE_NAME",
     "UINT32_DATA_TYPE_NAME",
     "UINT64_DATA_TYPE_NAME",
+    "UNSET",
     "V2_CHUNK_KEY_ENCODING_NAME",
     "V2_CHUNK_KEY_ENCODING_SEPARATOR",
     "ZSTD_CODEC_NAME",
-    "ArrayDimensionSeparatorV2",
-    "ArrayMetadataV2",
-    "ArrayMetadataV2Partial",
-    "ArrayMetadataV3",
-    "ArrayMetadataV3Partial",
-    "ArrayOrderV2",
     "BloscCName",
     "BloscCodecMetadata",
     "BloscCodecName",
@@ -254,31 +275,22 @@
     "CastRoundingMode",
     "CastValueCodecMetadata",
     "CastValueCodecName",
-    "CodecMetadataV2",
     "Complex64DataTypeName",
     "Complex64FillValue",
     "Complex128DataTypeName",
     "Complex128FillValue",
-    "ConsolidatedMetadataV2",
-    "ConsolidatedMetadataV3",
     "Crc32cCodecMetadata",
     "Crc32cCodecName",
-    "DataTypeMetadataV2",
     "DefaultChunkKeyEncodingMetadata",
     "DefaultChunkKeyEncodingName",
     "DefaultChunkKeyEncodingSeparator",
     "Endianness",
-    "ExtensionFieldV3",
     "Float16DataTypeName",
     "Float16FillValue",
     "Float32DataTypeName",
     "Float32FillValue",
     "Float64DataTypeName",
     "Float64FillValue",
-    "GroupMetadataV2",
-    "GroupMetadataV2Partial",
-    "GroupMetadataV3",
-    "GroupMetadataV3Partial",
     "GzipCodecMetadata",
     "GzipCodecName",
     "Int8DataTypeName",
@@ -290,13 +302,13 @@
     "Int64DataTypeName",
     "Int64FillValue",
     "JSONValue",
-    "MetadataV3",
-    "NamedConfigV3",
+    "MetadataValidationError",
     "NumpyDatetime64DataTypeName",
     "NumpyDatetime64FillValue",
     "NumpyTimeUnit",
     "NumpyTimedelta64DataTypeName",
     "NumpyTimedelta64FillValue",
+    "ProblemKind",
     "RawBytesDataTypeName",
     "RawBytesFillValue",
     "RectilinearChunkGridMetadata",
@@ -325,9 +337,39 @@
     "V2ChunkKeyEncodingMetadata",
     "V2ChunkKeyEncodingName",
     "V2ChunkKeyEncodingSeparator",
+    "ValidationProblem",
     "ZArrayMetadata",
     "ZAttrsMetadata",
     "ZGroupMetadata",
+    "ZarrV2ArrayDimensionSeparator",
+    "ZarrV2ArrayMetadata",
+    "ZarrV2ArrayMetadataJSON",
+    "ZarrV2ArrayMetadataJSONPartial",
+    "ZarrV2ArrayMetadataPartial",
+    "ZarrV2ArrayOrder",
+    "ZarrV2CodecMetadata",
+    "ZarrV2ConsolidatedMetadata",
+    "ZarrV2ConsolidatedMetadataJSON",
+    "ZarrV2DataTypeMetadata",
+    "ZarrV2GroupMetadata",
+    "ZarrV2GroupMetadataJSON",
+    "ZarrV2GroupMetadataJSONPartial",
+    "ZarrV2GroupMetadataPartial",
+    "ZarrV3ArrayMetadata",
+    "ZarrV3ArrayMetadataJSON",
+    "ZarrV3ArrayMetadataJSONPartial",
+    "ZarrV3ArrayMetadataPartial",
+    "ZarrV3ConsolidatedMetadata",
+    "ZarrV3ConsolidatedMetadataJSON",
+    "ZarrV3ExtensionField",
+    "ZarrV3GroupMetadata",
+    "ZarrV3GroupMetadataJSON",
+    "ZarrV3GroupMetadataJSONPartial",
+    "ZarrV3GroupMetadataPartial",
+    "ZarrV3MetadataField",
+    "ZarrV3MetadataFieldJSON",
+    "ZarrV3NamedConfig",
+    "ZarrV3NamedConfigJSON",
     "ZstdCodecMetadata",
     "ZstdCodecName",
     "__version__",

packages/zarr-metadata/src/zarr_metadata/_common.py

@@ -13,7 +13,7 @@
 
 JSONValue = TypeAliasType(
     "JSONValue",
-    "int | float | bool | None | str | list[JSONValue] | tuple[JSONValue, ...] | Mapping[str, JSONValue]",  # type: ignore[reportInvalidTypeForm]
+    "int | float | bool | None | str | list[JSONValue] | tuple[JSONValue, ...] | Mapping[str, JSONValue]",
 )
 """A recursive type alias for JSON-encodable values.
 
@@ -24,7 +24,7 @@
 """
 
 
-class NamedConfigV3(TypedDict):
+class ZarrV3NamedConfigJSON(TypedDict):
     """
     Externally-tagged union member for a metadata field.