## Summary
Part of #15383, this PR adds support for overloaded callables.
Typing spec: https://typing.python.org/en/latest/spec/overload.html
Specifically, it does the following:
1. Update the `FunctionType::signature` method to return signatures from
a possibly overloaded callable using a new `FunctionSignature` enum
2. Update `CallableType` to accommodate overloaded callable by updating
the inner type to `Box<[Signature]>`
3. Update the relation methods on `CallableType` with logic specific to
overloads
4. Update the display of callable type to display a list of signatures
enclosed by parenthesis
5. Update `CallableTypeOf` special form to recognize overloaded callable
6. Update subtyping, assignability and fully static check to account for
callables (equivalence is planned to be done as a follow-up)
For (2), it is required to be done in this PR because otherwise I'd need
to add some workaround for `into_callable_type` and I though it would be
best to include it in here.
For (2), another possible design would be convert `CallableType` in an
enum with two variants `CallableType::Single` and
`CallableType::Overload` but I decided to go with `Box<[Signature]>` for
now to (a) mirror it to be equivalent to `overload` field on
`CallableSignature` and (b) to avoid any refactor in this PR. This could
be done in a follow-up to better split the two kind of callables.
### Design
There were two main candidates on how to represent the overloaded
definition:
1. To include it in the existing infrastructure which is what this PR is
doing by recognizing all the signatures within the
`FunctionType::signature` method
2. To create a new `Overload` type variant
<details><summary>For context, this is what I had in mind with the new
type variant:</summary>
<p>
```rs
pub enum Type {
FunctionLiteral(FunctionType),
Overload(OverloadType),
BoundMethod(BoundMethodType),
...
}
pub struct OverloadType {
// FunctionLiteral or BoundMethod
overloads: Box<[Type]>,
// FunctionLiteral or BoundMethod
implementation: Option<Type>
}
pub struct BoundMethodType {
kind: BoundMethodKind,
self_instance: Type,
}
pub enum BoundMethodKind {
Function(FunctionType),
Overload(OverloadType),
}
```
</p>
</details>
The main reasons to choose (1) are the simplicity in the implementation,
reusing the existing infrastructure, avoiding any complications that the
new type variant has specifically around the different variants between
function and methods which would require the overload type to use `Type`
instead.
### Implementation
The core logic is how to collect all the overloaded functions. The way
this is done in this PR is by recording a **use** on the `Identifier`
node that represents the function name in the use-def map. This is then
used to fetch the previous symbol using the same name. This way the
signatures are going to be propagated from top to bottom (from first
overload to the final overload or the implementation) with each function
/ method. For example:
```py
from typing import overload
@overload
def foo(x: int) -> int: ...
@overload
def foo(x: str) -> str: ...
def foo(x: int | str) -> int | str:
return x
```
Here, each definition of `foo` knows about all the signatures that comes
before itself. So, the first overload would only see itself, the second
would see the first and itself and so on until the implementation or the
final overload.
This approach required some updates specifically recognizing
`Identifier` node to record the function use because it doesn't use
`ExprName`.
## Test Plan
Update existing test cases which were limited by the overload support
and add test cases for the following cases:
* Valid overloads as functions, methods, generics, version specific
* Invalid overloads as stated in
https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions
(implementation will be done in a follow-up)
* Various relation: fully static, subtyping, and assignability (others
in a follow-up)
## Ecosystem changes
_WIP_
After going through the ecosystem changes (there are a lot!), here's
what I've found:
We need assignability check between a callable type and a class literal
because a lot of builtins are defined as classes in typeshed whose
constructor method is overloaded e.g., `map`, `sorted`, `list.sort`,
`max`, `min` with the `key` parameter, `collections.abc.defaultdict`,
etc. (https://github.com/astral-sh/ruff/issues/17343). This makes up
most of the ecosystem diff **roughly 70 diagnostics**. For example:
```py
from collections import defaultdict
# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(int)
# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(list)
class Foo:
def __init__(self, x: int):
self.x = x
# red-knot: No overload of function `__new__` matches arguments [lint:no-matching-overload]
map(Foo, ["a", "b", "c"])
```
Duplicate diagnostics in unpacking
(https://github.com/astral-sh/ruff/issues/16514) has **~16
diagnostics**.
Support for the `callable` builtin which requires `TypeIs` support. This
is **5 diagnostics**. For example:
```py
from typing import Any
def _(x: Any | None) -> None:
if callable(x):
# red-knot: `Any | None`
# Pyright: `(...) -> object`
# mypy: `Any`
# pyrefly: `(...) -> object`
reveal_type(x)
```
Narrowing on `assert` which has **11 diagnostics**. This is being worked
on in https://github.com/astral-sh/ruff/pull/17345. For example:
```py
import re
match = re.search("", "")
assert match
match.group() # error: [possibly-unbound-attribute]
```
Others:
* `Self`: 2
* Type aliases: 6
* Generics: 3
* Protocols: 13
* Unpacking in comprehension: 1
(https://github.com/astral-sh/ruff/pull/17396)
## Performance
Refer to
https://github.com/astral-sh/ruff/pull/17366#issuecomment-2814053046.
## Summary
Add support for decorators on function as well as support
for properties by adding special handling for `@property` and `@<name of
property>.setter`/`.getter` decorators.
closes https://github.com/astral-sh/ruff/issues/16987
## Ecosystem results
- ✔️ A lot of false positives are fixed by our new
understanding of properties
- 🔴 A bunch of new false positives (typically
`possibly-unbound-attribute` or `invalid-argument-type`) occur because
we currently do not perform type narrowing on attributes. And with the
new understanding of properties, this becomes even more relevant. In
many cases, the narrowing occurs through an assertion, so this is also
something that we need to implement to get rid of these false positives.
- 🔴 A few new false positives occur because we do not
understand generics, and therefore some calls to custom setters fail.
- 🔴 Similarly, some false positives occur because we do not
understand protocols yet.
- ✔️ Seems like a true positive to me. [The
setter](e624d8edfa/src/packaging/specifiers.py (L752-L754))
only accepts `bools`, but `None` is assigned in [this
line](e624d8edfa/tests/test_specifiers.py (L688)).
```
+ error[lint:invalid-assignment]
/tmp/mypy_primer/projects/packaging/tests/test_specifiers.py:688:9:
Invalid assignment to data descriptor attribute `prereleases` on type
`SpecifierSet` with custom `__set__` method
```
- ✔️ This is arguable also a true positive. The setter
[here](0c6c75644f/rich/table.py (L359-L363))
returns `Table`, but typeshed wants [setters to return
`None`](bf8d2a9912/stdlib/builtins.pyi (L1298)).
```
+ error[lint:invalid-argument-type]
/tmp/mypy_primer/projects/rich/rich/table.py:359:5: Object of type
`Literal[padding]` cannot be assigned to parameter 2 (`fset`) of bound
method `setter`; expected type `(Any, Any, /) -> None`
```
## Follow ups
- Fix the `@no_type_check` regression
- Implement class decorators
## Test Plan
New Markdown test suites for decorators and properties.
## Summary
From #16861, and the continuation of #16915.
This PR fixes the incorrect behavior of
`TypeInferenceBuilder::infer_name_load` in eager nested scopes.
And this PR closes#16341.
## Test Plan
New test cases are added in `annotations/deferred.md`.
This cleans up how we handle calling unions of types. #16568 adding a
three-level structure for callable signatures (`Signatures`,
`CallableSignature`, and `Signature`) to handle unions and overloads.
This PR updates the bindings side to mimic that structure. What used to
be called `CallOutcome` is now `Bindings`, and represents the result of
binding actual arguments against a possible union of callables.
`CallableBinding` is the result of binding a single, possibly
overloaded, callable type. `Binding` is the result of binding a single
overload.
While we're here, this also cleans up `CallError` greatly. It was
previously extracting error information from the bindings and storing it
in the error result. It is now a simple enum, carrying no data, that's
used as a status code to talk about whether the overall binding was
successful or not. We are now more consistent about walking the binding
itself to get detailed information about _how_ the binding was
unsucessful.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR closes#16248.
If the return type of the function isn't assignable to the one
specified, an `invalid-return-type` error occurs.
I thought it would be better to report this as a different kind of error
than the `invalid-assignment` error, so I defined this as a new error.
## Test Plan
All type inconsistencies in the test cases have been replaced with
appropriate ones.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Model dunder-calls correctly (and in one single place), by implementing
this behavior (using `__getitem__` as an example).
```py
def getitem_desugared(obj: object, key: object) -> object:
getitem_callable = find_in_mro(type(obj), "__getitem__")
if hasattr(getitem_callable, "__get__"):
getitem_callable = getitem_callable.__get__(obj, type(obj))
return getitem_callable(key)
```
See the new `calls/dunder.md` test suite for more information. The new
behavior also needs much fewer lines of code (the diff is positive due
to new tests).
## Test Plan
New tests; fix TODOs in existing tests.
## Summary
Add support for `@classmethod`s.
```py
class C:
@classmethod
def f(cls, x: int) -> str:
return "a"
reveal_type(C.f(1)) # revealed: str
```
## Test Plan
New Markdown tests
When adjusting the existing tests, I aimed to avoid dealing with the
special case in other tests if it's not necessary to do so (that is,
avoid using `float` and `complex` as examples where we just need "some
type"), and keep the tests for the special case mostly collected in the
mdtest dedicated to that purpose.
Fixes https://github.com/astral-sh/ruff/issues/14932
## Summary
Use `Unknown | T_inferred` as the type for *undeclared* public symbols.
## Test Plan
- Updated existing tests
- New test for external `__slots__` modifications.
- New tests for external modifications of public symbols.
## Summary
Adds meta information to `Type::Todo`, allowing developers to easily
trace back the origin of a particular `@Todo` type they encounter.
Instead of `Type::Todo`, we now write either `type_todo!()` which
creates a `@Todo[path/to/source.rs:123]` type with file and line
information, or using `type_todo!("PEP 604 unions not supported")`,
which creates a variant with a custom message.
`Type::Todo` now contains a `TodoType` field. In release mode, this is
just a zero-sized struct, in order not to create any overhead. In debug
mode, this is an `enum` that contains the meta information.
`Type` implements `Copy`, which means that `TodoType` also needs to be
copyable. This limits the design space. We could intern `TodoType`, but
I discarded this option, as it would require us to have access to the
salsa DB everywhere we want to use `Type::Todo`. And it would have made
the macro invocations less ergonomic (requiring us to pass `db`).
So for now, the meta information is simply a `&'static str` / `u32` for
the file/line variant, or a `&'static str` for the custom message.
Anything involving a chain/backtrace of several `@Todo`s or similar is
therefore currently not implemented. Also because we currently don't see
any direct use cases for this, and because all of this will eventually
go away.
Note that the size of `Type` increases from 16 to 24 bytes, but only in
debug mode.
## Test Plan
- Observed the changes in Markdown tests.
- Added custom messages for all `Type::Todo`s that were revealed in the
tests
- Ran red knot in release and debug mode on the following Python file:
```py
def f(x: int) -> int:
reveal_type(x)
```
Prints `@Todo` in release mode and `@Todo(function parameter type)` in
debug mode.
## Summary
This fixes an edge case that @carljm and I missed when implementing
https://github.com/astral-sh/ruff/pull/13800. Namely, if the left-hand
operand is the _exact same type_ as the right-hand operand, the
reflected dunder on the right-hand operand is never tried:
```pycon
>>> class Foo:
... def __radd__(self, other):
... return 42
...
>>> Foo() + Foo()
Traceback (most recent call last):
File "<python-input-1>", line 1, in <module>
Foo() + Foo()
~~~~~~^~~~~~~
TypeError: unsupported operand type(s) for +: 'Foo' and 'Foo'
```
This edge case _is_ covered in Brett's blog at
https://snarky.ca/unravelling-binary-arithmetic-operations-in-python/,
but I missed it amongst all the other subtleties of this algorithm. The
motivations and history behind it were discussed in
https://mail.python.org/archives/list/python-dev@python.org/thread/7NZUCODEAPQFMRFXYRMGJXDSIS3WJYIV/
## Test Plan
I added an mdtest for this cornercase.
