Feature Request: Expose public synchronous encoding/decoding API in Codec / CodecPipeline

[ElemTran]

Describe the Feature

Currently, in Zarr v3 (zarr-python 3.x), the core encoding and decoding APIs in BatchedCodecPipeline (and individual codecs) are predominantly asynchronous (async def encode(), async def decode()).

However, many high-throughput data processing pipelines (e.g., converting large histopathology WSI images using pyvips/OpenCV in thread pools) run in pure synchronous, CPU-bound multi-threaded environments.

In these architectures, bridging the sync thread pool to async APIs via asyncio.run() introduces severe overhead. Based on our micro-benchmarks, running asyncio.run(pipeline.encode(...)) to compress 512x512 chunks:

• Async API via asyncio.run(): ~1.65 ms/iter (nearly 0.8 ms is wasted on event loop creation/destruction per iteration)
• Direct internal sync encoding (_encode_sync chain): ~0.65 ms/iter (2.5x faster than async bridging, 4.2x faster than creating ephemeral MemoryStore + zarr.Array templates)

Since underlying codecs already implement synchronous processing logic via _encode_sync(chunk_array, chunk_spec), it would be extremely beneficial to expose them as part of the public API (e.g., CodecPipeline.encode_sync or public Codec.encode_sync).

Proposed Solution

Expose public synchronous methods on Codec and CodecPipeline (or BatchedCodecPipeline):

class CodecPipeline:
    def encode_sync(
        self, 
        chunk_arrays_and_specs: Iterable[tuple[NDBuffer | None, ArraySpec]]
    ) -> Iterable[Buffer | None]:
        ...

This would allow CPU-bound, sync-heavy pipelines to directly leverage fast compression pipelines without any async event loop overhead.

Additional Context

Using undocumented internal _encode_sync methods works perfectly as a workaround but is prone to breakage across minor version upgrades of zarr-python.

[d-v-b]

hi @ElemTran thanks for this issue. we are aware that the async path is inefficient for many storage backends. That being said, I don't think we want to make the sync methods public. Here's why:

we added those methods as a stop-gap measure to support the work in this PR, which defines a codec pipeline that opportunistically avoids to_thread and asyncio, relying instead on threading. This codec pipeline, which you can read about in this blog post, uses those currently-private sync methods.

Why is it a stop-gap measure? Because ultimately we want to completely separate classes that compute / fetch using async from the classes that compute / fetch using plain synchronous functions. So instead of store classes that support sync and async methods, which is confusing for callers who don't know which method is performant, we want two completely separate object hierarchies for sync and async-native classes. That means we will need big changes in our codebase, because our core Store and Codec abstractions use async for all compute / io. Those changes will take some work -- see #4217 for more detail.

I hope that makes sense. Until we evolve our core APIs to fix the underlying issue, I recommend you try out the new FusedCodecPipeline; it should avoid all the async / to_thread overhead you've been fighting, and if not, let us know.

[ElemTran]

Hi @d-v-b,

Thank you for the detailed response and explanation regarding the architectural roadmap (#4217)!

We upgraded to Zarr 3.3.0 and ran micro-benchmarks comparing FusedCodecPipeline against direct _encode_sync calls on 512x512 uint8 WSI (histopathology) image chunks (with BytesCodec + ZstdCodec):

• Default BatchedCodecPipeline (via zarr.Array write): ~1.47 ms/chunk (~680 chunks/sec)
• FusedCodecPipeline (via zarr.Array write): ~0.91 ms/chunk (~1,100 chunks/sec) — 1.6x speedup over default.
• Direct _encode_sync (pure stateless chunk encoding without Store/Array overhead): ~0.27 ms/chunk (~10,000 chunks/sec in an 8-thread pool).

FusedCodecPipeline is definitely a great improvement for standard zarr.Array / Store indexing workloads!

However, for high-throughput image generation pipelines, our worker threads generate chunks in parallel and stream encoded raw chunk bytes directly to downstream queues or custom object storage writers. In this pattern, passing data through zarr.Array slice indexing and MemoryStore introduces unnecessary coordinate/store locking overhead.

As you refactor the core abstractions for sync-native object hierarchies in #4217, it would be extremely valuable to have a public, stateless synchronous chunk encoding/decoding entry point (e.g., CodecPipeline.encode_chunk_sync(data, spec) -> bytes or equivalent) that bypasses Store and Array instantiation completely.

Thanks again for the great work on Zarr v3!