Add array_chunks by ronnodas · Pull Request #1023 · rust-itertools/itertools

ronnodas · 2025-03-02T18:42:28Z

Added an implementation of arrays based on the work in next_array(). Currently the N = 0 produces a post-monomorphization error, see the discussion in #1012.

Some other choices that I'm not sure about:

If the number of items yielded by the input iterator is not a multiple of N then there is exactly one allocation the first time next() returns None. I don't think this is a performance issue but this means the method has to be gated behind use_alloc. The alternative would be something like implementing IntoIterator for ArrayBuilder, which seems possible but the safety invariants get more complicated. A middle ground would be to only gate the remainder() method, but this seems worse.
Should ArrayChunks::remainder() return a custom type, so that it can be ExactSizeIterator?
Is it okay to not specify the behavior if the input iterator is not fused? I think the current implementation behaves slightly differently depending on whether the None is on a multiple of N.

codecov · 2025-03-02T18:43:55Z

Codecov Report

Attention: Patch coverage is 91.25000% with 7 lines in your changes missing coverage. Please review.

Project coverage is 94.32%. Comparing base (6814180) to head (db21b2f).
Report is 137 commits behind head on master.

Files with missing lines	Patch %	Lines
src/arrays.rs	82.92%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1023      +/-   ##
==========================================
- Coverage   94.38%   94.32%   -0.07%     
==========================================
  Files          48       51       +3     
  Lines        6665     6963     +298     
==========================================
+ Hits         6291     6568     +277     
- Misses        374      395      +21

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Iterator::array_chunks exists on nightly and it's better to avoid the name collision especially if and when that's stabilized

ronnodas · 2025-03-02T23:53:01Z

Re the MSRV failure, I'm not sure what the idiomatic way is to trigger a PME on 1.63 based on N > 0.

jswrenn · 2025-03-03T00:07:13Z

Re the MSRV failure, I'm not sure what the idiomatic way is to trigger a PME on 1.63 based on N > 0.

You can do it like this: https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=233fb2f6ca3df52d15e3df96b8c4cb61

jswrenn · 2025-03-05T15:04:56Z

src/arrays.rs

+/// Effectively assert!(N > 0) post-monomorphization
+fn assert_positive<const N: usize>() {
+    trait StaticAssert<const N: usize> {
+        const ASSERT: bool;
+    }
+
+    impl<const N: usize> StaticAssert<N> for () {
+        const ASSERT: bool = {
+            assert!(N > 0);
+            true
+        };
+    }
+
+    assert!(<() as StaticAssert<N>>::ASSERT);
+}


Rust doesn't provide stacktraces for monomorphization errors, so folks will only see that an error occured in assert_positive — and not the code that actually required the assertion.

Define this as a macro and call that instead.

Is the change in db21b2f what you meant?

jswrenn · 2025-03-05T15:05:48Z

src/lib.rs

+    /// assert_eq!(Some([]), it.next());
+    /// ```
+    #[cfg(feature = "use_alloc")]
+    fn arrays<const N: usize>(self) -> Arrays<Self, N>


IMO array_chunks is the better name, since it differentiates it from array_windows.

To add context, I was trying to avoid triggering the unstable-name-collisions lint, because of rust-lang/rust#100450. Should I change it back to array_chunks?

jswrenn · 2025-03-05T15:28:02Z

src/arrays.rs

+pub struct Arrays<I: Iterator, const N: usize> {
+    iter: I,
+    partial: Vec<I::Item>,
+}


It strikes me as a code smell that N doesn't appear in the definition here. Shouldn't partial be an ArrayBuilder<T, N>?

Making it ArrayBuilder<T, N> a priori makes sense, but I had trouble with the bounds for the impl Clone in that case. Note that MaybeUninit<T>: Clone requires T: Copy.

Maybe it could also be array::IntoIter<T, N> or a hypothetical ArrayBuilder::IntoIter<T, N> (this is close to what the unstable Iterator::ArrayChunks does) but I think the unsafe code for that would need to be more careful.

The state you need to keep track of doesn't really depend on N (except partial.len() <= N but this is sort of incidental). However Arrays needs to have N as a parameter for Arrays::Item to be well-defined.

Suggestions?

king-11 · 2025-10-12T10:40:37Z

what are the thoughts about having an array_chunks which can fill the partially filled last item by fetching from a filler that the caller provides? I think it would be a very useful API in case people always want a certain length chunks

bend-n · 2025-11-11T07:18:52Z

im curious why you didnt want to just copy the previous stdlib implementation? you can simply implement this on top of a manual as_chunks?
returning an owned array and using a vec for remainder is kind of funky.

ronnodas · 2025-11-11T07:30:14Z

@bend-n Do you mean slice::as_chunks? That works for slices, but this is for a general iterator.

There's also a nightly version of this in std: Iterator::array_chunks. If you're asking about the implementation there, I don't think the code in this PR is that different, just trying to reuse things already in the crate. But TBF I haven't looked at either that implementation or this PR closely in a while.

bend-n · 2025-11-11T07:42:23Z

oh, my bad. i forgot that iterator::array_chunks was still unstable and assumed this was with regards to slice array chunks for some reason.

you could still copy the implementation of iterator::array_chunks, though?

ronnodas · 2025-12-23T03:57:18Z

you could still copy the implementation of iterator::array_chunks, though?

If you go through the source, Iterator::array_chunks is implemented in terms of (also unstable) Iterator::next_chunk, which is equivalent to Itertools::next_array. It made more sense to use the thing that's already in the crate rather than copy over something with duplicate functionality.

array_chunks

a7489e4

ronnodas added 2 commits March 3, 2025 00:12

copy-paste artefact

7fa1908

PME on array_chunks::<0>

d866b7d

ronnodas force-pushed the array-chunks branch from 12e26ae to d866b7d Compare March 2, 2025 23:19

ronnodas added 3 commits March 3, 2025 00:28

impl ExactSizeIterator

14f004d

exact size tests

1bb3739

rename to arrays

532758a

Iterator::array_chunks exists on nightly and it's better to avoid the name collision especially if and when that's stabilized

avoid const assert to keep msrv

6bede55

ronnodas force-pushed the array-chunks branch from edaabbf to 6bede55 Compare March 3, 2025 00:16

jswrenn requested changes Mar 5, 2025

View reviewed changes

make assert_positive a macro

db21b2f

ronnodas requested a review from jswrenn June 20, 2025 18:47

Conversation

ronnodas commented Mar 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ronnodas commented Mar 2, 2025

Uh oh!

jswrenn commented Mar 3, 2025

Uh oh!

jswrenn Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

ronnodas Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

jswrenn Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

ronnodas Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

jswrenn Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

ronnodas Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

king-11 commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bend-n commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ronnodas commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bend-n commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ronnodas commented Dec 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ronnodas commented Mar 2, 2025 •

edited

Loading

codecov bot commented Mar 2, 2025 •

edited

Loading

ronnodas Mar 5, 2025 •

edited

Loading

king-11 commented Oct 12, 2025 •

edited

Loading

bend-n commented Nov 11, 2025 •

edited

Loading

ronnodas commented Nov 11, 2025 •

edited

Loading

bend-n commented Nov 11, 2025 •

edited

Loading