Using Generic and __new__ in class def, mypy misses incorrect parameter if parameter created inline

[Dr-Irv]

Bug Report

This came up in pandas-stubs, issue pandas-dev/pandas-stubs#412

If you have a stub that defines a Generic class with __new__(), with overloads, and the final overload doesn't specify the type of the generic, then a function call that creates an object of the generic class in the function call is not picked up as an error, but it is picked up if stored in another variable.

To Reproduce

Two files are needed. First is gptst2.pyi :

from typing import Generic, TypeVar, Type, overload

TV = TypeVar("TV", int, str)

class Foo(Generic[TV]):
    @overload
    def __new__(cls, ty: Type[TV]) -> Foo[TV]: ...
    @overload
    def __new__(cls, ty=...) -> Foo: ...

Then the tester code genparam.py:

from gptst2 import Foo


def fun(param: Foo[int]) -> Foo[str]:
    ...


fun(param=Foo(str))
p = Foo(str)
fun(p)

Expected Behavior

mypy should report errors on both calls to fun()

Actual Behavior
mypy does not pick up that the first call to fun is incorrect. It does pick up that the second call is incorrect.
pyright picks up both as being incorrect.

Output from mypy:

genparam.py:10: error: Argument 1 to "fun" has incompatible type "Foo[str]"; expected "Foo[int]"

Your Environment

• Mypy version used: 0.971
• Mypy command-line flags: --no-incremental
• Mypy configuration options from mypy.ini (and other config files): None
• Python version used: 3.10

[erictraut]

I think mypy is correct here. The behavior you cited for pyright was due to a bug which was fixed a while ago. Pyright and mypy now agree.

[Dr-Irv]

I think mypy is correct here. The behavior you cited for pyright was due to a bug which was fixed a while ago. Pyright and mypy now agree.

Why do you think that both pyright and mypy are correct?

Why should the 2 calls

fun(param=Foo(str))
p = Foo(str)
fun(p)

to the function fun() be interpreted differently?

[JelleZijlstra]

I believe it's due to type context (Eric calls the equivalent technique in pyright 'bidirectional inference'). When a particular type is expected, type checkers try to infer that type to minimize errors. However, when the value is assigned to a variable, that assignment is type checked independently and sets the type.

Consider code like this:

def f(x: list[object]) -> None: pass

y = [1]
f(y)

f([1])

Here mypy shows an error for the first call, but not the second.

[Dr-Irv]

When a particular type is expected, type checkers try to infer that type to minimize errors. However, when the value is assigned to a variable, that assignment is type checked independently and sets the type.

I don't think your example fits in this case, because object is more of a supertype.

In my example, you have

def fun(param: Foo[int]) -> Foo[str]:
    ...

fun(param=Foo(str))
p = Foo(str)
fun(p)

So I don't see how Foo(str), which has type Foo[str] could ever match Foo[int] in the first call to fun()

[erictraut]

When you call fun(param=Foo(str)) (or the equivalent fun(Foo(str))), a type checker needs to determine whether there a way for the expression Foo(str) to be evaluated such that it can satisfy the call to fun. In this case, there is a way because there is an overload that returns a Foo[Any], and Foo[Any] is compatible with Foo[int].

By contrast, if you construct a Foo[str] independent of the call to fun, there is no way to transform that Fun[str] into a Fun[int] such that it's compatible with parameter param.

It's similar to this:

a1 = [1, 2, 3]
x1: list[float] = a # Type error

x2: list[float] = [1, 2, 3] # No type error

[Dr-Irv]

@erictraut thanks for the example. It's very subtle, and difficult for people using type checkers to understand.