mirror of
https://github.com/astral-sh/ruff.git
synced 2025-10-05 16:10:36 +00:00
0092794302
## Summary
Modify the (external) signature of instance methods such that the first
parameter uses `Self` unless it is explicitly annotated. This allows us
to correctly type-check more code, and allows us to infer correct return
types for many functions that return `Self`. For example:
```py
from pathlib import Path
from datetime import datetime, timedelta
reveal_type(Path(".config") / ".ty") # now Path, previously Unknown
def _(dt: datetime, delta: timedelta):
reveal_type(dt - delta) # now datetime, previously Unknown
```
part of https://github.com/astral-sh/ty/issues/159
## Performance
I ran benchmarks locally on `attrs`, `freqtrade` and `colour`, the
projects with the largest regressions on CodSpeed. I see much smaller
effects locally, but can definitely reproduce the regression on `attrs`.
From looking at the profiling results (on Codspeed), it seems that we
simply do more type inference work, which seems plausible, given that we
now understand much more return types (of many stdlib functions). In
particular, whenever a function uses an implicit `self` and returns
`Self` (without mentioning `Self` anywhere else in its signature), we
will now infer the correct type, whereas we would previously return
`Unknown`. This also means that we need to invoke the generics solver in
more cases. Comparing half a million lines of log output on attrs, I can
see that we do 5% more "work" (number of lines in the log), and have a
lot more `apply_specialization` events (7108 vs 4304). On freqtrade, I
see similar numbers for `apply_specialization` (11360 vs 5138 calls).
Given these results, I'm not sure if it's generally worth doing more
performance work, especially since none of the code modifications
themselves seem to be likely candidates for regressions.
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `./ty_main check /home/shark/ecosystem/attrs` | 92.6 ± 3.6 | 85.9 |
102.6 | 1.00 |
| `./ty_self check /home/shark/ecosystem/attrs` | 101.7 ± 3.5 | 96.9 |
113.8 | 1.10 ± 0.06 |
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `./ty_main check /home/shark/ecosystem/freqtrade` | 599.0 ± 20.2 |
568.2 | 627.5 | 1.00 |
| `./ty_self check /home/shark/ecosystem/freqtrade` | 607.9 ± 11.5 |
594.9 | 626.4 | 1.01 ± 0.04 |
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `./ty_main check /home/shark/ecosystem/colour` | 423.9 ± 17.9 | 394.6
| 447.4 | 1.00 |
| `./ty_self check /home/shark/ecosystem/colour` | 426.9 ± 24.9 | 373.8
| 456.6 | 1.01 ± 0.07 |
## Test Plan
New Markdown tests
## Ecosystem report
* apprise: ~300 new diagnostics related to problematic stubs in apprise
😩
* attrs: a new true positive, since [this
function](4e2c89c823/tests/test_make.py (L2135))
is missing a `@staticmethod`?
* Some legitimate true positives
* sympy: lots of new `invalid-operator` false positives in [matrix
multiplication](cf9f4b6805/sympy/matrices/matrixbase.py (L3267-L3269))
due to our limited understanding of [generic `Callable[[Callable[[T1,
T2], T3]], Callable[[T1, T2], T3]]` "identity"
types](cf9f4b6805/sympy/core/decorators.py (L83-L84))
of decorators. This is not related to type-of-self.
## Typing conformance results
The changes are all correct, except for
```diff
+generics_self_usage.py:50:5: error[invalid-assignment] Object of type `def foo(self) -> int` is not assignable to `(typing.Self, /) -> int`
```
which is related to an assignability problem involving type variables on
both sides:
```py
class CallableAttribute:
def foo(self) -> int:
return 0
bar: Callable[[Self], int] = foo # <- we currently error on this assignment
```
---------
Co-authored-by: Shaygan Hooshyari <sh.hooshyari@gmail.com>
6.7 KiB
6.7 KiB
Async with statements
Basic async with statement
The type of the target variable in a with statement should be the return type from the context
manager's __aenter__ method. However, async with statements aren't supported yet. This test
asserts that it doesn't emit any context manager-related errors.
class Target: ...
class Manager:
async def __aenter__(self) -> Target:
return Target()
async def __aexit__(self, exc_type, exc_value, traceback): ...
async def test():
async with Manager() as f:
reveal_type(f) # revealed: Target
Multiple targets
class Manager:
async def __aenter__(self) -> tuple[int, str]:
return 42, "hello"
async def __aexit__(self, exc_type, exc_value, traceback): ...
async def test():
async with Manager() as (x, y):
reveal_type(x) # revealed: int
reveal_type(y) # revealed: str
Context manager without an __aenter__ or __aexit__ method
class Manager: ...
async def main():
# error: [invalid-context-manager] "Object of type `Manager` cannot be used with `async with` because it does not implement `__aenter__` and `__aexit__`"
async with Manager():
...
Context manager without an __aenter__ method
class Manager:
async def __aexit__(self, exc_tpe, exc_value, traceback): ...
async def main():
# error: [invalid-context-manager] "Object of type `Manager` cannot be used with `async with` because it does not implement `__aenter__`"
async with Manager():
...
Context manager without an __aexit__ method
class Manager:
async def __aenter__(self): ...
async def main():
# error: [invalid-context-manager] "Object of type `Manager` cannot be used with `async with` because it does not implement `__aexit__`"
async with Manager():
...
Context manager with non-callable __aenter__ attribute
class Manager:
__aenter__: int = 42
async def __aexit__(self, exc_tpe, exc_value, traceback): ...
async def main():
# error: [invalid-context-manager] "Object of type `Manager` cannot be used with `async with` because it does not correctly implement `__aenter__`"
async with Manager():
...
Context manager with non-callable __aexit__ attribute
from typing_extensions import Self
class Manager:
def __aenter__(self) -> Self:
return self
__aexit__: int = 32
async def main():
# error: [invalid-context-manager] "Object of type `Manager` cannot be used with `async with` because it does not correctly implement `__aexit__`"
async with Manager():
...
Context expression with possibly-unbound union variants
async def _(flag: bool):
class Manager1:
def __aenter__(self) -> str:
return "foo"
def __aexit__(self, exc_type, exc_value, traceback): ...
class NotAContextManager: ...
context_expr = Manager1() if flag else NotAContextManager()
# error: [invalid-context-manager] "Object of type `Manager1 | NotAContextManager` cannot be used with `async with` because the methods `__aenter__` and `__aexit__` are possibly missing"
async with context_expr as f:
reveal_type(f) # revealed: str
Context expression with "sometimes" callable __aenter__ method
async def _(flag: bool):
class Manager:
if flag:
async def __aenter__(self) -> str:
return "abcd"
async def __exit__(self, *args): ...
# error: [invalid-context-manager] "Object of type `Manager` cannot be used with `async with` because the method `__aenter__` may be missing"
async with Manager() as f:
reveal_type(f) # revealed: CoroutineType[Any, Any, str]
Invalid __aenter__ signature
class Manager:
async def __aenter__() -> str:
return "foo"
async def __aexit__(self, exc_type, exc_value, traceback): ...
async def main():
context_expr = Manager()
# error: [invalid-context-manager] "Object of type `Manager` cannot be used with `async with` because it does not correctly implement `__aenter__`"
async with context_expr as f:
reveal_type(f) # revealed: CoroutineType[Any, Any, str]
Accidental use of async async with
If a asynchronous async with statement is used on a type with __enter__ and __exit__, we show
a diagnostic hint that the user might have intended to use with instead.
class Manager:
def __enter__(self): ...
def __exit__(self, *args): ...
async def main():
# error: [invalid-context-manager] "Object of type `Manager` cannot be used with `async with` because it does not implement `__aenter__` and `__aexit__`"
async with Manager():
...
Incorrect signatures
The sub-diagnostic is also provided if the signatures of __enter__ and __exit__ do not match the
expected signatures for a context manager:
class Manager:
def __enter__(self): ...
def __exit__(self, typ: str, exc, traceback): ...
async def main():
# error: [invalid-context-manager] "Object of type `Manager` cannot be used with `async with` because it does not implement `__aenter__` and `__aexit__`"
async with Manager():
...
Incorrect number of arguments
Similarly, we also show the hint if the functions have the wrong number of arguments:
class Manager:
def __enter__(self, wrong_extra_arg): ...
def __exit__(self, typ, exc, traceback, wrong_extra_arg): ...
async def main():
# error: [invalid-context-manager] "Object of type `Manager` cannot be used with `async with` because it does not implement `__aenter__` and `__aexit__`"
async with Manager():
...
@asynccontextmanager
from contextlib import asynccontextmanager
from typing import AsyncGenerator
class Session: ...
@asynccontextmanager
async def connect() -> AsyncGenerator[Session]:
yield Session()
# TODO: this should be `() -> _AsyncGeneratorContextManager[Session, None]`
reveal_type(connect) # revealed: (...) -> _AsyncGeneratorContextManager[Unknown, None]
async def main():
async with connect() as session:
# TODO: should be `Session`
reveal_type(session) # revealed: Unknown
asyncio.timeout
[environment]
python-version = "3.11"
import asyncio
async def long_running_task():
await asyncio.sleep(5)
async def main():
async with asyncio.timeout(1):
await long_running_task()
asyncio.TaskGroup
[environment]
python-version = "3.11"
import asyncio
async def long_running_task():
await asyncio.sleep(5)
async def main():
async with asyncio.TaskGroup() as tg:
reveal_type(tg) # revealed: TaskGroup
tg.create_task(long_running_task())