
## Summary Modify the (external) signature of instance methods such that the first parameter uses `Self` unless it is explicitly annotated. This allows us to correctly type-check more code, and allows us to infer correct return types for many functions that return `Self`. For example: ```py from pathlib import Path from datetime import datetime, timedelta reveal_type(Path(".config") / ".ty") # now Path, previously Unknown def _(dt: datetime, delta: timedelta): reveal_type(dt - delta) # now datetime, previously Unknown ``` part of https://github.com/astral-sh/ty/issues/159 ## Performance I ran benchmarks locally on `attrs`, `freqtrade` and `colour`, the projects with the largest regressions on CodSpeed. I see much smaller effects locally, but can definitely reproduce the regression on `attrs`. From looking at the profiling results (on Codspeed), it seems that we simply do more type inference work, which seems plausible, given that we now understand much more return types (of many stdlib functions). In particular, whenever a function uses an implicit `self` and returns `Self` (without mentioning `Self` anywhere else in its signature), we will now infer the correct type, whereas we would previously return `Unknown`. This also means that we need to invoke the generics solver in more cases. Comparing half a million lines of log output on attrs, I can see that we do 5% more "work" (number of lines in the log), and have a lot more `apply_specialization` events (7108 vs 4304). On freqtrade, I see similar numbers for `apply_specialization` (11360 vs 5138 calls). Given these results, I'm not sure if it's generally worth doing more performance work, especially since none of the code modifications themselves seem to be likely candidates for regressions. | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `./ty_main check /home/shark/ecosystem/attrs` | 92.6 ± 3.6 | 85.9 | 102.6 | 1.00 | | `./ty_self check /home/shark/ecosystem/attrs` | 101.7 ± 3.5 | 96.9 | 113.8 | 1.10 ± 0.06 | | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `./ty_main check /home/shark/ecosystem/freqtrade` | 599.0 ± 20.2 | 568.2 | 627.5 | 1.00 | | `./ty_self check /home/shark/ecosystem/freqtrade` | 607.9 ± 11.5 | 594.9 | 626.4 | 1.01 ± 0.04 | | Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | `./ty_main check /home/shark/ecosystem/colour` | 423.9 ± 17.9 | 394.6 | 447.4 | 1.00 | | `./ty_self check /home/shark/ecosystem/colour` | 426.9 ± 24.9 | 373.8 | 456.6 | 1.01 ± 0.07 | ## Test Plan New Markdown tests ## Ecosystem report * apprise: ~300 new diagnostics related to problematic stubs in apprise 😩 * attrs: a new true positive, since [this function](4e2c89c823/tests/test_make.py (L2135)
) is missing a `@staticmethod`? * Some legitimate true positives * sympy: lots of new `invalid-operator` false positives in [matrix multiplication](cf9f4b6805/sympy/matrices/matrixbase.py (L3267-L3269)
) due to our limited understanding of [generic `Callable[[Callable[[T1, T2], T3]], Callable[[T1, T2], T3]]` "identity" types](cf9f4b6805/sympy/core/decorators.py (L83-L84)
) of decorators. This is not related to type-of-self. ## Typing conformance results The changes are all correct, except for ```diff +generics_self_usage.py:50:5: error[invalid-assignment] Object of type `def foo(self) -> int` is not assignable to `(typing.Self, /) -> int` ``` which is related to an assignability problem involving type variables on both sides: ```py class CallableAttribute: def foo(self) -> int: return 0 bar: Callable[[Self], int] = foo # <- we currently error on this assignment ``` --------- Co-authored-by: Shaygan Hooshyari <sh.hooshyari@gmail.com>
11 KiB
Self
[environment]
python-version = "3.13"
Self
is treated as if it were a TypeVar
bound to the class it's being used on.
typing.Self
is only available in Python 3.11 and later.
Methods
from typing import Self
class Shape:
def set_scale(self: Self, scale: float) -> Self:
reveal_type(self) # revealed: Self@set_scale
return self
def nested_type(self: Self) -> list[Self]:
return [self]
def nested_func(self: Self) -> Self:
def inner() -> Self:
reveal_type(self) # revealed: Self@nested_func
return self
return inner()
def nested_func_without_enclosing_binding(self):
def inner(x: Self):
reveal_type(x) # revealed: Self@nested_func_without_enclosing_binding
inner(self)
reveal_type(Shape().nested_type()) # revealed: list[Shape]
reveal_type(Shape().nested_func()) # revealed: Shape
class Circle(Shape):
def set_scale(self: Self, scale: float) -> Self:
reveal_type(self) # revealed: Self@set_scale
return self
class Outer:
class Inner:
def foo(self: Self) -> Self:
reveal_type(self) # revealed: Self@foo
return self
Type of (unannotated) self
parameters
In instance methods, the first parameter (regardless of its name) is assumed to have the type
typing.Self
, unless it has an explicit annotation. This does not apply to @classmethod
and
@staticmethod
s.
[environment]
python-version = "3.11"
from typing import Self
class A:
def implicit_self(self) -> Self:
# TODO: This should be Self@implicit_self
reveal_type(self) # revealed: Unknown
return self
def a_method(self) -> int:
def first_arg_is_not_self(a: int) -> int:
reveal_type(a) # revealed: int
return a
return first_arg_is_not_self(1)
@classmethod
def a_classmethod(cls) -> Self:
# TODO: This should be type[Self@bar]
reveal_type(cls) # revealed: Unknown
return cls()
@staticmethod
def a_staticmethod(x: int): ...
a = A()
reveal_type(a.implicit_self()) # revealed: A
reveal_type(a.implicit_self) # revealed: bound method A.implicit_self() -> A
Calling an instance method explicitly verifies the first argument:
A.implicit_self(a)
# error: [invalid-argument-type] "Argument to function `implicit_self` is incorrect: Argument type `Literal[1]` does not satisfy upper bound `A` of type variable `Self`"
A.implicit_self(1)
Passing self
implicitly also verifies the type:
from typing import Never
class Strange:
def can_not_be_called(self: Never) -> None: ...
# error: [invalid-argument-type] "Argument to bound method `can_not_be_called` is incorrect: Expected `Never`, found `Strange`"
Strange().can_not_be_called()
If the method is a class or static method then first argument is not inferred as Self
:
A.a_classmethod()
A.a_classmethod(a) # error: [too-many-positional-arguments]
A.a_staticmethod(1)
a.a_staticmethod(1)
A.a_staticmethod(a) # error: [invalid-argument-type]
The first parameter of instance methods always has type Self
, if it is not explicitly annotated.
The name self
is not special in any way.
class B:
def name_does_not_matter(this) -> Self:
# TODO: Should reveal Self@name_does_not_matter
reveal_type(this) # revealed: Unknown
return this
def positional_only(self, /, x: int) -> Self:
# TODO: Should reveal Self@positional_only
reveal_type(self) # revealed: Unknown
return self
def keyword_only(self, *, x: int) -> Self:
# TODO: Should reveal Self@keyword_only
reveal_type(self) # revealed: Unknown
return self
@property
def a_property(self) -> Self:
# TODO: Should reveal Self@a_property
reveal_type(self) # revealed: Unknown
return self
reveal_type(B().name_does_not_matter()) # revealed: B
reveal_type(B().positional_only(1)) # revealed: B
reveal_type(B().keyword_only(x=1)) # revealed: B
# TODO: this should be B
reveal_type(B().a_property) # revealed: Unknown
This also works for generic classes:
from typing import Self, Generic, TypeVar
T = TypeVar("T")
class G(Generic[T]):
def id(self) -> Self:
# TODO: Should reveal Self@id
reveal_type(self) # revealed: Unknown
return self
reveal_type(G[int]().id()) # revealed: G[int]
reveal_type(G[str]().id()) # revealed: G[str]
Free functions and nested functions do not use implicit Self
:
def not_a_method(self):
reveal_type(self) # revealed: Unknown
# error: [invalid-type-form]
def does_not_return_self(self) -> Self:
return self
class C:
def outer(self) -> None:
def inner(self):
reveal_type(self) # revealed: Unknown
reveal_type(not_a_method) # revealed: def not_a_method(self) -> Unknown
typing_extensions
[environment]
python-version = "3.10"
from typing_extensions import Self
class C:
def method(self: Self) -> Self:
return self
reveal_type(C().method()) # revealed: C
Class Methods
from typing import Self, TypeVar
class Shape:
def foo(self: Self) -> Self:
return self
@classmethod
def bar(cls: type[Self]) -> Self:
# TODO: type[Shape]
reveal_type(cls) # revealed: @Todo(unsupported type[X] special form)
return cls()
class Circle(Shape): ...
reveal_type(Shape().foo()) # revealed: Shape
# TODO: Shape
reveal_type(Shape.bar()) # revealed: Unknown
Attributes
TODO: The use of Self
to annotate the next_node
attribute should be
modeled as a property, using Self
in its parameter and return type.
from typing import Self
class LinkedList:
value: int
next_node: Self
def next(self: Self) -> Self:
reveal_type(self.value) # revealed: int
# TODO: no error
# error: [invalid-return-type]
return self.next_node
reveal_type(LinkedList().next()) # revealed: LinkedList
Generic Classes
from typing import Self, Generic, TypeVar
T = TypeVar("T")
class Container(Generic[T]):
value: T
def set_value(self: Self, value: T) -> Self:
return self
int_container: Container[int] = Container[int]()
reveal_type(int_container) # revealed: Container[int]
reveal_type(int_container.set_value(1)) # revealed: Container[int]
Protocols
TODO: https://typing.python.org/en/latest/spec/generics.html#use-in-protocols
Annotations
from typing import Self
class Shape:
def union(self: Self, other: Self | None):
reveal_type(other) # revealed: Self@union | None
return self
Self
for classes with a default value for their generic parameter
This is a regression test for https://github.com/astral-sh/ty/issues/1156.
from typing import Self
class Container[T = bytes]:
def __init__(self: Self, data: T | None = None) -> None:
self.data = data
reveal_type(Container()) # revealed: Container[bytes]
reveal_type(Container(1)) # revealed: Container[int]
reveal_type(Container("a")) # revealed: Container[str]
reveal_type(Container(b"a")) # revealed: Container[bytes]
Invalid Usage
Self
cannot be used in the signature of a function or variable.
from typing import Self, Generic, TypeVar
T = TypeVar("T")
# error: [invalid-type-form]
def x(s: Self): ...
# error: [invalid-type-form]
b: Self
# TODO: "Self" cannot be used in a function with a `self` or `cls` parameter that has a type annotation other than "Self"
class Foo:
# TODO: rejected Self because self has a different type
def has_existing_self_annotation(self: T) -> Self:
return self # error: [invalid-return-type]
def return_concrete_type(self) -> Self:
# TODO: tell user to use "Foo" instead of "Self"
# error: [invalid-return-type]
return Foo()
@staticmethod
# TODO: reject because of staticmethod
def make() -> Self:
# error: [invalid-return-type]
return Foo()
class Bar(Generic[T]):
foo: T
def bar(self) -> T:
return self.foo
# error: [invalid-type-form]
class Baz(Bar[Self]): ...
class MyMetaclass(type):
# TODO: rejected
def __new__(cls) -> Self:
return super().__new__(cls)
Explicit annotations override implicit Self
If the first parameter is explicitly annotated, that annotation takes precedence over the implicit
Self
type.
[environment]
python-version = "3.12"
from __future__ import annotations
from typing import final
@final
class Disjoint: ...
class Explicit:
# TODO: We could emit a warning if the annotated type of `self` is disjoint from `Explicit`
def bad(self: Disjoint) -> None:
reveal_type(self) # revealed: Disjoint
def forward(self: Explicit) -> None:
reveal_type(self) # revealed: Explicit
# error: [invalid-argument-type] "Argument to bound method `bad` is incorrect: Expected `Disjoint`, found `Explicit`"
Explicit().bad()
Explicit().forward()
class ExplicitGeneric[T]:
def special(self: ExplicitGeneric[int]) -> None:
reveal_type(self) # revealed: ExplicitGeneric[int]
ExplicitGeneric[int]().special()
# TODO: this should be an `invalid-argument-type` error
ExplicitGeneric[str]().special()
Binding a method fixes Self
When a method is bound, any instances of Self
in its signature are "fixed", since we now know the
specific type of the bound parameter.
from typing import Self
class C:
def instance_method(self, other: Self) -> Self:
return self
@classmethod
def class_method(cls) -> Self:
return cls()
# revealed: bound method C.instance_method(other: C) -> C
reveal_type(C().instance_method)
# revealed: bound method <class 'C'>.class_method() -> C
reveal_type(C.class_method)
class D(C): ...
# revealed: bound method D.instance_method(other: D) -> D
reveal_type(D().instance_method)
# revealed: bound method <class 'D'>.class_method() -> D
reveal_type(D.class_method)
In nested functions self
binds to the method. So in the following example the self
in C.b
is
bound at C.f
.
from typing import Self
from ty_extensions import generic_context
class C[T]():
def f(self: Self):
def b(x: Self):
reveal_type(x) # revealed: Self@f
reveal_type(generic_context(b)) # revealed: None
reveal_type(generic_context(C.f)) # revealed: tuple[Self@f]
Even if the Self
annotation appears first in the nested function, it is the method that binds
Self
.
from typing import Self
from ty_extensions import generic_context
class C:
def f(self: "C"):
def b(x: Self):
reveal_type(x) # revealed: Self@f
reveal_type(generic_context(b)) # revealed: None
reveal_type(generic_context(C.f)) # revealed: None