[red-knot] Several failing tests for generics (#16509)

To kick off the work of supporting generics, this adds many new
(currently failing) tests, showing the behavior we plan to support.

This is still missing a lot!  Not included:

- typevar tuples
- param specs
- variance
- `Self`

But it's a good start! We can add more failing tests for those once we
tackle these.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
This commit is contained in:
Douglas Creager 2025-03-05 17:21:19 -05:00 committed by GitHub
parent 114abc7cfb
commit ebd172e732
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 808 additions and 81 deletions

View file

@ -1,81 +0,0 @@
# PEP 695 Generics
## Class Declarations
Basic PEP 695 generics
```py
class MyBox[T]:
data: T
box_model_number = 695
def __init__(self, data: T):
self.data = data
box: MyBox[int] = MyBox(5)
# TODO should emit a diagnostic here (str is not assignable to int)
wrong_innards: MyBox[int] = MyBox("five")
# TODO reveal int, do not leak the typevar
reveal_type(box.data) # revealed: T
reveal_type(MyBox.box_model_number) # revealed: Unknown | Literal[695]
```
## Subclassing
```py
class MyBox[T]:
data: T
def __init__(self, data: T):
self.data = data
# TODO not error on the subscripting
# error: [non-subscriptable]
class MySecureBox[T](MyBox[T]): ...
secure_box: MySecureBox[int] = MySecureBox(5)
reveal_type(secure_box) # revealed: MySecureBox
# TODO reveal int
# The @Todo(…) is misleading here. We currently treat `MyBox[T]` as a dynamic base class because we
# don't understand generics and therefore infer `Unknown` for the `MyBox[T]` base of `MySecureBox[T]`.
reveal_type(secure_box.data) # revealed: @Todo(instance attribute on class with dynamic base)
```
## Cyclical class definition
In type stubs, classes can reference themselves in their base class definitions. For example, in
`typeshed`, we have `class str(Sequence[str]): ...`.
This should hold true even with generics at play.
```pyi
class Seq[T]: ...
# TODO not error on the subscripting
class S[T](Seq[S]): ... # error: [non-subscriptable]
reveal_type(S) # revealed: Literal[S]
```
## Type params
A PEP695 type variable defines a value of type `typing.TypeVar`.
```py
def f[T]():
reveal_type(T) # revealed: T
reveal_type(T.__name__) # revealed: Literal["T"]
```
## Minimum two constraints
A typevar with less than two constraints emits a diagnostic:
```py
# error: [invalid-type-variable-constraints] "TypeVar must have at least two constrained types"
def f[T: (int,)]():
pass
```

View file

@ -0,0 +1,186 @@
# Generic classes
## PEP 695 syntax
TODO: Add a `red_knot_extension` function that asserts whether a function or class is generic.
This is a generic class defined using PEP 695 syntax:
```py
class C[T]: ...
```
A class that inherits from a generic class, and fills its type parameters with typevars, is generic:
```py
# TODO: no error
# error: [non-subscriptable]
class D[U](C[U]): ...
```
A class that inherits from a generic class, but fills its type parameters with concrete types, is
_not_ generic:
```py
# TODO: no error
# error: [non-subscriptable]
class E(C[int]): ...
```
A class that inherits from a generic class, and doesn't fill its type parameters at all, implicitly
uses the default value for the typevar. In this case, that default type is `Unknown`, so `F`
inherits from `C[Unknown]` and is not itself generic.
```py
class F(C): ...
```
## Legacy syntax
This is a generic class defined using the legacy syntax:
```py
from typing import Generic, TypeVar
T = TypeVar("T")
# TODO: no error
# error: [invalid-base]
class C(Generic[T]): ...
```
A class that inherits from a generic class, and fills its type parameters with typevars, is generic.
```py
class D(C[T]): ...
```
(Examples `E` and `F` from above do not have analogues in the legacy syntax.)
## Inferring generic class parameters
The type parameter can be specified explicitly:
```py
class C[T]:
x: T
# TODO: no error
# TODO: revealed: C[int]
# error: [non-subscriptable]
reveal_type(C[int]()) # revealed: Unknown
```
We can infer the type parameter from a type context:
```py
c: C[int] = C()
# TODO: revealed: C[int]
reveal_type(c) # revealed: C
```
The typevars of a fully specialized generic class should no longer be visible:
```py
# TODO: revealed: int
reveal_type(c.x) # revealed: T
```
If the type parameter is not specified explicitly, and there are no constraints that let us infer a
specific type, we infer the typevar's default type:
```py
class D[T = int]: ...
# TODO: revealed: D[int]
reveal_type(D()) # revealed: D
```
If a typevar does not provide a default, we use `Unknown`:
```py
# TODO: revealed: C[Unknown]
reveal_type(C()) # revealed: C
```
If the type of a constructor parameter is a class typevar, we can use that to infer the type
parameter:
```py
class E[T]:
def __init__(self, x: T) -> None: ...
# TODO: revealed: E[int] or E[Literal[1]]
reveal_type(E(1)) # revealed: E
```
The types inferred from a type context and from a constructor parameter must be consistent with each
other:
```py
# TODO: error
wrong_innards: E[int] = E("five")
```
## Generic subclass
When a generic subclass fills its superclass's type parameter with one of its own, the actual types
propagate through:
```py
class Base[T]:
x: T
# TODO: no error
# error: [non-subscriptable]
class Sub[U](Base[U]): ...
# TODO: no error
# TODO: revealed: int
# error: [non-subscriptable]
reveal_type(Base[int].x) # revealed: Unknown
# TODO: revealed: int
reveal_type(Sub[int].x) # revealed: Unknown
```
## Cyclic class definition
A class can use itself as the type parameter of one of its superclasses. (This is also known as the
[curiously recurring template pattern][crtp] or [F-bounded quantification][f-bound].)
Here, `Sub` is not a generic class, since it fills its superclass's type parameter (with itself).
`stub.pyi`:
```pyi
class Base[T]: ...
# TODO: no error
# error: [non-subscriptable]
class Sub(Base[Sub]): ...
reveal_type(Sub) # revealed: Literal[Sub]
```
`string_annotation.py`:
```py
class Base[T]: ...
# TODO: no error
# error: [non-subscriptable]
class Sub(Base["Sub"]): ...
reveal_type(Sub) # revealed: Literal[Sub]
```
`bare_annotation.py`:
```py
class Base[T]: ...
# TODO: error: [unresolved-reference]
class Sub(Base[Sub]): ...
```
[crtp]: https://en.wikipedia.org/wiki/Curiously_recurring_template_pattern
[f-bound]: https://en.wikipedia.org/wiki/Bounded_quantification#F-bounded_quantification

View file

@ -0,0 +1,244 @@
# Generic functions
## Typevar must be used at least twice
If you're only using a typevar for a single parameter, you don't need the typevar — just use
`object` (or the typevar's upper bound):
```py
# TODO: error, should be (x: object)
def typevar_not_needed[T](x: T) -> None:
pass
# TODO: error, should be (x: int)
def bounded_typevar_not_needed[T: int](x: T) -> None:
pass
```
Typevars are only needed if you use them more than once. For instance, to specify that two
parameters must both have the same type:
```py
def two_params[T](x: T, y: T) -> T:
return x
```
or to specify that a return value is the same as a parameter:
```py
def return_value[T](x: T) -> T:
return x
```
Each typevar must also appear _somewhere_ in the parameter list:
```py
def absurd[T]() -> T:
# There's no way to construct a T!
...
```
## Inferring generic function parameter types
If the type of a generic function parameter is a typevar, then we can infer what type that typevar
is bound to at each call site.
TODO: Note that some of the TODO revealed types have two options, since we haven't decided yet
whether we want to infer a more specific `Literal` type where possible, or use heuristics to weaken
the inferred type to e.g. `int`.
```py
def f[T](x: T) -> T: ...
# TODO: no error
# TODO: revealed: int or Literal[1]
# error: [invalid-argument-type]
reveal_type(f(1)) # revealed: T
# TODO: no error
# TODO: revealed: float
# error: [invalid-argument-type]
reveal_type(f(1.0)) # revealed: T
# TODO: no error
# TODO: revealed: bool or Literal[true]
# error: [invalid-argument-type]
reveal_type(f(True)) # revealed: T
# TODO: no error
# TODO: revealed: str or Literal["string"]
# error: [invalid-argument-type]
reveal_type(f("string")) # revealed: T
```
## Inferring “deep” generic parameter types
The matching up of call arguments and discovery of constraints on typevars can be a recursive
process for arbitrarily-nested generic types in parameters.
```py
def f[T](x: list[T]) -> T: ...
# TODO: revealed: float
reveal_type(f([1.0, 2.0])) # revealed: T
```
## Typevar constraints
If a type parameter has an upper bound, that upper bound constrains which types can be used for that
typevar. This effectively adds the upper bound as an intersection to every appearance of the typevar
in the function.
```py
def good_param[T: int](x: T) -> None:
# TODO: revealed: T & int
reveal_type(x) # revealed: T
```
If the function is annotated as returning the typevar, this means that the upper bound is _not_
assignable to that typevar, since return types are contravariant. In `bad`, we can infer that
`x + 1` has type `int`. But `T` might be instantiated with a narrower type than `int`, and so the
return value is not guaranteed to be compatible for all `T: int`.
```py
def good_return[T: int](x: T) -> T:
return x
def bad_return[T: int](x: T) -> T:
# TODO: error: int is not assignable to T
# error: [unsupported-operator] "Operator `+` is unsupported between objects of type `T` and `Literal[1]`"
return x + 1
```
## All occurrences of the same typevar have the same type
If a typevar appears multiple times in a function signature, all occurrences have the same type.
```py
def different_types[T, S](cond: bool, t: T, s: S) -> T:
if cond:
return t
else:
# TODO: error: S is not assignable to T
return s
def same_types[T](cond: bool, t1: T, t2: T) -> T:
if cond:
return t1
else:
return t2
```
## All occurrences of the same constrained typevar have the same type
The above is true even when the typevars are constrained. Here, both `int` and `str` have `__add__`
methods that are compatible with the return type, so the `return` expression is always well-typed:
```py
def same_constrained_types[T: (int, str)](t1: T, t2: T) -> T:
# TODO: no error
# error: [unsupported-operator] "Operator `+` is unsupported between objects of type `T` and `T`"
return t1 + t2
```
This is _not_ the same as a union type, because of this additional constraint that the two
occurrences have the same type. In `unions_are_different`, `t1` and `t2` might have different types,
and an `int` and a `str` cannot be added together:
```py
def unions_are_different(t1: int | str, t2: int | str) -> int | str:
# error: [unsupported-operator] "Operator `+` is unsupported between objects of type `int | str` and `int | str`"
return t1 + t2
```
## Typevar inference is a unification problem
When inferring typevar assignments in a generic function call, we cannot simply solve constraints
eagerly for each parameter in turn. We must solve a unification problem involving all of the
parameters simultaneously.
```py
def two_params[T](x: T, y: T) -> T:
return x
# TODO: no error
# TODO: revealed: str
# error: [invalid-argument-type]
# error: [invalid-argument-type]
reveal_type(two_params("a", "b")) # revealed: T
# TODO: no error
# TODO: revealed: str | int
# error: [invalid-argument-type]
# error: [invalid-argument-type]
reveal_type(two_params("a", 1)) # revealed: T
```
```py
def param_with_union[T](x: T | int, y: T) -> T:
return y
# TODO: no error
# TODO: revealed: str
# error: [invalid-argument-type]
reveal_type(param_with_union(1, "a")) # revealed: T
# TODO: no error
# TODO: revealed: str
# error: [invalid-argument-type]
# error: [invalid-argument-type]
reveal_type(param_with_union("a", "a")) # revealed: T
# TODO: no error
# TODO: revealed: int
# error: [invalid-argument-type]
reveal_type(param_with_union(1, 1)) # revealed: T
# TODO: no error
# TODO: revealed: str | int
# error: [invalid-argument-type]
# error: [invalid-argument-type]
reveal_type(param_with_union("a", 1)) # revealed: T
```
```py
def tuple_param[T, S](x: T | S, y: tuple[T, S]) -> tuple[T, S]:
return y
# TODO: no error
# TODO: revealed: tuple[str, int]
# error: [invalid-argument-type]
# error: [invalid-argument-type]
reveal_type(tuple_param("a", ("a", 1))) # revealed: tuple[T, S]
# TODO: no error
# TODO: revealed: tuple[str, int]
# error: [invalid-argument-type]
# error: [invalid-argument-type]
reveal_type(tuple_param(1, ("a", 1))) # revealed: tuple[T, S]
```
## Inferring nested generic function calls
We can infer type assignments in nested calls to multiple generic functions. If they use the same
type variable, we do not confuse the two; `T@f` and `T@g` have separate types in each example below.
```py
def f[T](x: T) -> tuple[T, int]:
return (x, 1)
def g[T](x: T) -> T | None:
return x
# TODO: no error
# TODO: revealed: tuple[str | None, int]
# error: [invalid-argument-type]
# error: [invalid-argument-type]
reveal_type(f(g("a"))) # revealed: tuple[T, int]
# TODO: no error
# TODO: revealed: tuple[str, int] | None
# error: [invalid-argument-type]
# error: [invalid-argument-type]
reveal_type(g(f("a"))) # revealed: T | None
```

View file

@ -0,0 +1,72 @@
# Legacy type variables
The tests in this file focus on how type variables are defined using the legacy notation. Most
_uses_ of type variables are tested in other files in this directory; we do not duplicate every test
for both type variable syntaxes.
Unless otherwise specified, all quotations come from the [Generics] section of the typing spec.
## Type variables
### Defining legacy type variables
> Generics can be parameterized by using a factory available in `typing` called `TypeVar`.
This was the only way to create type variables prior to PEP 695/Python 3.12. It is still available
in newer Python releases.
```py
from typing import TypeVar
T = TypeVar("T")
```
### Directly assigned to a variable
> A `TypeVar()` expression must always directly be assigned to a variable (it should not be used as
> part of a larger expression).
```py
from typing import TypeVar
# TODO: error
TestList = list[TypeVar("W")]
```
### `TypeVar` parameter must match variable name
> The argument to `TypeVar()` must be a string equal to the variable name to which it is assigned.
```py
from typing import TypeVar
# TODO: error
T = TypeVar("Q")
```
### No redefinition
> Type variables must not be redefined.
```py
from typing import TypeVar
T = TypeVar("T")
# TODO: error
T = TypeVar("T")
```
### Cannot have only one constraint
> `TypeVar` supports constraining parametric types to a fixed set of possible types...There should
> be at least two constraints, if any; specifying a single constraint is disallowed.
```py
from typing import TypeVar
# TODO: error: [invalid-type-variable-constraints]
T = TypeVar("T", int)
```
[generics]: https://typing.readthedocs.io/en/latest/spec/generics.html

View file

@ -0,0 +1,51 @@
# PEP 695 Generics
[PEP 695] and Python 3.12 introduced new, more ergonomic syntax for type variables.
## Type variables
### Defining PEP 695 type variables
PEP 695 introduces a new syntax for defining type variables. The resulting type variables are
instances of `typing.TypeVar`, just like legacy type variables.
```py
def f[T]():
reveal_type(type(T)) # revealed: Literal[TypeVar]
reveal_type(T) # revealed: T
reveal_type(T.__name__) # revealed: Literal["T"]
```
### Cannot have only one constraint
> `TypeVar` supports constraining parametric types to a fixed set of possible types...There should
> be at least two constraints, if any; specifying a single constraint is disallowed.
```py
# error: [invalid-type-variable-constraints] "TypeVar must have at least two constrained types"
def f[T: (int,)]():
pass
```
## Invalid uses
Note that many of the invalid uses of legacy typevars do not apply to PEP 695 typevars, since the
PEP 695 syntax is only allowed places where typevars are allowed.
## Displaying typevars
We use a suffix when displaying the typevars of a generic function or class. This helps distinguish
different uses of the same typevar.
```py
def f[T](x: T, y: T) -> None:
# TODO: revealed: T@f
reveal_type(x) # revealed: T
class C[T]:
def m(self, x: T) -> None:
# TODO: revealed: T@c
reveal_type(x) # revealed: T
```
[pep 695]: https://peps.python.org/pep-0695/

View file

@ -0,0 +1,255 @@
# Scoping rules for type variables
Most of these tests come from the [Scoping rules for type variables][scoping] section of the typing
spec.
## Typevar used outside of generic function or class
Typevars may only be used in generic function or class definitions.
```py
from typing import TypeVar
T = TypeVar("T")
# TODO: error
x: T
class C:
# TODO: error
x: T
def f() -> None:
# TODO: error
x: T
```
## Legacy typevar used multiple times
> A type variable used in a generic function could be inferred to represent different types in the
> same code block.
This only applies to typevars defined using the legacy syntax, since the PEP 695 syntax creates a
new distinct typevar for each occurrence.
```py
from typing import TypeVar
T = TypeVar("T")
def f1(x: T) -> T: ...
def f2(x: T) -> T: ...
f1(1)
f2("a")
```
## Typevar inferred multiple times
> A type variable used in a generic function could be inferred to represent different types in the
> same code block.
This also applies to a single generic function being used multiple times, instantiating the typevar
to a different type each time.
```py
def f[T](x: T) -> T: ...
# TODO: no error
# TODO: revealed: int or Literal[1]
# error: [invalid-argument-type]
reveal_type(f(1)) # revealed: T
# TODO: no error
# TODO: revealed: str or Literal["a"]
# error: [invalid-argument-type]
reveal_type(f("a")) # revealed: T
```
## Methods can mention class typevars
> A type variable used in a method of a generic class that coincides with one of the variables that
> parameterize this class is always bound to that variable.
```py
class C[T]:
def m1(self, x: T) -> T: ...
def m2(self, x: T) -> T: ...
c: C[int] = C()
# TODO: no error
# error: [invalid-argument-type]
c.m1(1)
# TODO: no error
# error: [invalid-argument-type]
c.m2(1)
# TODO: expected type `int`
# error: [invalid-argument-type] "Object of type `Literal["string"]` cannot be assigned to parameter 2 (`x`) of bound method `m2`; expected type `T`"
c.m2("string")
```
## Methods can mention other typevars
> A type variable used in a method that does not match any of the variables that parameterize the
> class makes this method a generic function in that variable.
```py
from typing import TypeVar, Generic
T = TypeVar("T")
S = TypeVar("S")
# TODO: no error
# error: [invalid-base]
class Legacy(Generic[T]):
def m(self, x: T, y: S) -> S: ...
legacy: Legacy[int] = Legacy()
# TODO: revealed: str
reveal_type(legacy.m(1, "string")) # revealed: @Todo(Invalid or unsupported `Instance` in `Type::to_type_expression`)
```
With PEP 695 syntax, it is clearer that the method uses a separate typevar:
```py
class C[T]:
def m[S](self, x: T, y: S) -> S: ...
c: C[int] = C()
# TODO: no errors
# TODO: revealed: str
# error: [invalid-argument-type]
# error: [invalid-argument-type]
reveal_type(c.m(1, "string")) # revealed: S
```
## Unbound typevars
> Unbound type variables should not appear in the bodies of generic functions, or in the class
> bodies apart from method definitions.
This is true with the legacy syntax:
```py
from typing import TypeVar, Generic
T = TypeVar("T")
S = TypeVar("S")
def f(x: T) -> None:
x: list[T] = []
# TODO: error
y: list[S] = []
# TODO: no error
# error: [invalid-base]
class C(Generic[T]):
# TODO: error
x: list[S] = []
# This is not an error, as shown in the previous test
def m(self, x: S) -> S: ...
```
This is true with PEP 695 syntax, as well, though we must use the legacy syntax to define the
unbound typevars:
`pep695.py`:
```py
from typing import TypeVar
S = TypeVar("S")
def f[T](x: T) -> None:
x: list[T] = []
# TODO: error
y: list[S] = []
class C[T]:
# TODO: error
x: list[S] = []
def m1(self, x: S) -> S: ...
def m2[S](self, x: S) -> S: ...
```
## Nested formal typevars must be distinct
Generic functions and classes can be nested in each other, but it is an error for the same typevar
to be used in nested generic definitions.
Note that the typing spec only mentions two specific versions of this rule:
> A generic class definition that appears inside a generic function should not use type variables
> that parameterize the generic function.
and
> A generic class nested in another generic class cannot use the same type variables.
We assume that the more general form holds.
### Generic function within generic function
```py
def f[T](x: T, y: T) -> None:
def ok[S](a: S, b: S) -> None: ...
# TODO: error
def bad[T](a: T, b: T) -> None: ...
```
### Generic method within generic class
```py
class C[T]:
def ok[S](self, a: S, b: S) -> None: ...
# TODO: error
def bad[T](self, a: T, b: T) -> None: ...
```
### Generic class within generic function
```py
from typing import Iterable
def f[T](x: T, y: T) -> None:
class Ok[S]: ...
# TODO: error
class Bad1[T]: ...
# TODO: error
class Bad2(Iterable[T]): ...
```
### Generic class within generic class
```py
from typing import Iterable
class C[T]:
class Ok1[S]: ...
# TODO: error
class Bad1[T]: ...
# TODO: error
class Bad2(Iterable[T]): ...
```
## Class scopes do not cover inner scopes
Just like regular symbols, the typevars of a generic class are only available in that class's scope,
and are not available in nested scopes.
```py
class C[T]:
ok1: list[T] = []
class Bad:
# TODO: error
bad: list[T] = []
class Inner[S]: ...
ok2: Inner[T]
```
[scoping]: https://typing.readthedocs.io/en/latest/spec/generics.html#scoping-rules-for-type-variables