No way to express fixed-size heterogeneous iterator/sequence w/o extending tuple

[ms-pablo-ruiz]

Describe the Bug

Iter in python is used both for implementing iterations as in loops or as for defining the unpacking of the class, but doesn't seems to produce correct types, and fails in a somewhat strange manner.

class DummyIter:
    def __iter__(self) -> tuple[int, str]:
        return 1, "s"

a,b = DummyIter() # Type `DummyIter` is not iterable
# but this is valid in python and works, 
print(a)  # 1 
print(b) # "s"

The real code that made me realize this was a bit more confusing as makes you believe that you are iterating the element in the loop instead of the list:

my_list = [DummyIter(), DummyIter()]
for i, j in my_list:  # Type `DummyIter` is not iterable
    # but this is valid in python and works:
    print(i) # 1
    print(j) # "s"

In this case DummyIter is a iterable as far as i know even if is only iterable for unpacking porpoises

For tuples / NamedTuples this does work and inheriting from tuple can make it work, but that solution has its own drawbacks.

Sandbox Link

https://pyrefly.org/sandbox/?project=N4IgZglgNgpgziAXKOBDAdgEwEYHsAeAdAA4CeS4ATrgLYAEmqALqgMZSpxzx0Q3G5KTBszYcuAHXRT2nOHQAiAVxo1SASSYxKiKXX0MYYOgH0TELZTMAKblDABKOgFoAfHSZLisANoR0TAA0dHBMlAC6uugGMXSUMJ6U0QCMwRIgcOlSUqiB2HQAvIoqapra1g5SxJT%2BTNaolejVtdbYjdnoaiZQEKGFdD7KqhqWFcFDpaMO4VJggrzBAFa80V09oVExzQHWEI1bNTuL7dLoIIEgSkzQcCTkiCAAxHQAqtc9TKR0YErorNe4dCZU6YIzfQQ0ZgmdAqbDlfCIFZMJxuEJhTYGeKJaJgdIAOVh2kRwHwAF8smcLmR4mAoKRCExaFAKM8AAqkGl0kIYHAEOisQGQADmSkozAggMIUmeAGUYDA6AALJhMYhwRAAeg11KMdMIgiFGpg6A1mFwrDgGoF6GFovFgI14ModFQADdUNBUNhYPzBRARWKAdFcMQg7cqqQmIrAc5Xdo4BLokV0gBmQjJABMFJApIubGucYAYtAYBQ0Fg8EQyDmgA

(Only applicable for extension issues) IDE Information

No response

[asukaminato0721]

class DummyIter:
    def __iter__(self) -> tuple[int, str]:
        return 1, "s"

a,b = DummyIter() # Type `DummyIter` is not iterable
# but this is valid in python and works, 
print(a)  # 1 
print(b) # "s"

it has Runtime error

Traceback (most recent call last):
  File "/tmp/exec_project_2289570104/code/main.py", line 9, in <module>
    a,b = DummyIter() # Type `DummyIter` is not iterable
    ^^^
TypeError: iter() returned non-iterator of type 'tuple'

[yangdanny97]

Interestingly, Pyright also raises identical errors in both cases (that's not to say we can't improve on that, though!)

https://pyright-play.net/?code=MYGwhgzhAEAiCuBbRBPAkgFwKYCcBcAUNMdACZYBm0A%2BtQJbY60AUEWIFAlNALQB80DPAAOILAG06AOwwAaaBAw4AuoRLroOLEJxToARnkAiCEYIEwsgEbQAvHCSpMuZtwDE0ACophWaAAMEZHRGf2g6GCkAewxwxjArMQIPK3hYjAALCPCYADcwEDpScL1hFEyovTApYoB3KJwAawh5AmEcaQxmMG5oD31oNo6ZZit3aBMzAlRqQsU7aHEgp0ZXeWWQl05lAgoG8PkAKxLoGbmMPGIPb18AjeccMOzo2IZcBKT1FLTBLJhs-KFYrSaBlCpVGrQepNCBqEjtTrMOjjfREeHDLqHcaTIA

I think what's happening in your second example is that

for i, j in my_list roughly desugars to to for _x in my_list & i, j = _x (so you’re iterating thru my_list, but also iterating thru each element of my_list to do the unpacking)

the error is being raised for the i, j = _x part, which is the same pattern as your first example, hence the error message is the same.

[ms-pablo-ruiz]

@asukaminato0721 yes you are right, i think i striped the original code too much and didn't realize, maybe a more interesting case (that it actually runs ) is:

from typing import Iterator

class DummyIter:
    def __iter__(self) -> Iterator[tuple[int, str]]:
        return iter((1, "s"))

a,b = DummyIter()  # a , b are both infered to be tuple[int,str] instead of int  and str
print("a", a)
print("b", a)

a, b = (1,"s") # for a tuple or a class that inherits from tuple or namedtuple is identified correctly ofc
print("a", a)
print("b", a)

my_list = [DummyIter(), DummyIter()]
for i, j in my_list:
    print("i", i)
    print("j", j)

I think this is just a problem related how python / type checkers special cases tuples.

Probably in pyrefly we could display an error if iter doesn't give back and iterable (or self),

And i guess I either we consider that non-tuples should not be unpacked in the same way as tuples even if is technically allowed, or we consider the case of iter returning a typed tuple as a unpacking case that should be supported, not sure.

@yangdanny97 yes that is the intentoin, is like when you iterate and unpack a list of tuples or a zip.

[yangdanny97]

I think the type annotations are being interpreted correctly per the spec, annotating it as Iterator[tuple[int, str]] means that each element produced is tuple[int, str]. There's no way to annotate an __iter__ method that produces a fixed number of heterogeneous elements, since as you said that behavior is special-cased for tuples.

If you want a custom class to behave like a tuple during iteration, you can make your class extend tuple or NamedTuple. Pyrefly will apply the special tuple handling to your class, so it behaves as expected for the following two snippets.

class DummyIter(tuple[int, str]):
    pass

a,b = DummyIter()  # a is inferred as int, b is inferred as str
print("a", a)
print("b", a)

or

from typing import NamedTuple
class DummyIter(NamedTuple):
    a: int
    b: str

a,b = DummyIter(a=1, b="")  # a is inferred as int, b is inferred as str
print("a", a)
print("b", a)

[ms-pablo-ruiz]

Yes the type annotation is correct, but is still not great is that the type-checker doesn't realize that for unpacking an iterable of tuples means element by element instead of every element. I don't see why tuple / namedtuples should be special for this behaviour.

The reason i try to avoid replacing the class by a namedTuple is that there is a very big performance difference (like 30%) between instantiating named tuples and a dataclass with slots. In code that is performance critical that is a big change.

So suddenly i need to chose precise typehints or real performance, not great.

[yangdanny97]

Right now it's not possible to express that in Python's type system, within the constraints you mentioned.

We can't change how we interpret the Iterator[tuple[...]] because that annotation already means something else, and it would break anything that want each element of the iterator to actually be a tuple.

This discussion thread on the Python forum presents some workarounds (only making it extend NamedTuple/tuple for type checking but not runtime), but it's also unsatisfying since it's verbose and type checking diverges from runtime behavior.

https://discuss.python.org/t/custom-fixed-size-heterogeneous-sequence/93855/6

Ultimately we probably need a PEP to add a new way to express the type you want & standardize the behavior here

[yangdanny97]

python/mypy#8441 similar issue in Mypy