## Summary
This PR closes#16248.
If the return type of the function isn't assignable to the one
specified, an `invalid-return-type` error occurs.
I thought it would be better to report this as a different kind of error
than the `invalid-assignment` error, so I defined this as a new error.
## Test Plan
All type inconsistencies in the test cases have been replaced with
appropriate ones.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
This updates the `Signature` and `CallBinding` machinery to support
multiple overloads for a callable. This is currently only used for
`KnownFunction`s that we special-case in our type inference code. It
does **_not_** yet update the semantic index builder to handle
`@overload` decorators and construct a multi-signature `Overloads`
instance for real Python functions.
While I was here, I updated many of the `try_call` special cases to use
signatures (possibly overloaded ones now) and `bind_call` to check
parameter lists. We still need some of the mutator methods on
`OverloadBinding` for the special cases where we need to update return
types based on some Rust code.
## Summary
This mostly fixes#14899
My motivation was similar to the last comment by @sharkdp there. I ran
red_knot on a codebase and the most common error was patterns like this
failing:
```
def foo(x: str): ...
x: Any = ...
if isinstance(x, str):
foo(x) # Object of type `Any & str` cannot be assigned to parameter 1 (`x`) of function `foo`; expected type `str`
```
The desired behavior is pretty much to ignore Any/Unknown when resolving
intersection assignability - `Any & str` should be assignable to `str`,
and `str` should be assignable to `str & Any`
The fix is actually very similar to the existing code in
`is_subtype_of`, we need to correctly handle intersections on either
side, while being careful to handle dynamic types as desired.
This does not fix the second test case from that issue:
```
static_assert(is_assignable_to(Intersection[Unrelated, Any], Not[tuple[Unrelated, Any]]))
```
but that's misleading because the root cause there has nothing to do
with gradual types. I added a simpler test case that also fails:
```
static_assert(is_assignable_to(Unrelated, Not[tuple[Unrelated]]))
```
This is because we don't determine that Unrelated does not subclass from
tuple so we can't rule out this relation. If that logic is improved then
this fix should also handle the case of the intersection
## Test Plan
Added a bunch of is_assignable_to tests, most of which failed before
this fix.
## Summary
Part of https://github.com/astral-sh/ruff/issues/15382
This PR adds support for inferring the `lambda` expression and return
the `CallableType`.
Currently, this is only limited to inferring the parameters and a todo
type for the return type.
For posterity, I tried using the `file_expression_type` to infer the
return type of lambda but it would always lead to cycle. The main reason
is that in `infer_parameter_definition`, the default expression is being
inferred using `file_expression_type`, which is correct, but it then
Take the following source code as an example:
```py
lambda x=1: x
```
Here's how the code will flow:
* `infer_scope_types` for the global scope
* `infer_lambda_expression`
* `infer_expression` for the default value `1`
* `file_expression_type` for the return type using the body expression.
This is because the body creates it's own scope
* `infer_scope_types` (lambda body scope)
* `infer_name_load` for the symbol `x` whose visible binding is the
lambda parameter `x`
* `infer_parameter_definition` for parameter `x`
* `file_expression_type` for the default value `1`
* `infer_scope_types` for the global scope because of the default
expression
This will then reach to `infer_definition` for the parameter `x` again
which then creates the cycle.
## Test Plan
Add tests around `lambda` expression inference.
## Summary
Fixes a small nit of mine -- we are currently inconsistent in our
spelling between "metaclass" and "meta class", and between "meta type"
and "meta-type". This PR means that we consistently use "metaclass" and
"meta-type".
## Test Plan
`uvx pre-commit run -a`
## Summary
Part of https://github.com/astral-sh/ruff/issues/15382
This PR implements a general callable type that wraps around a
`Signature` and it uses that new type to represent `typing.Callable`.
It also implements `Display` support for `Callable`. The format is as:
```
([<arg name>][: <arg type>][ = <default type>], ...) -> <return type>
```
The `/` and `*` separators are added at the correct boundary for
positional-only and keyword-only parameters. Now, as `typing.Callable`
only has positional-only parameters, the rendered signature would be:
```py
Callable[[int, str], None]
# (int, str, /) -> None
```
The `/` separator represents that all the arguments are positional-only.
The relationship methods that check assignability, subtype relationship,
etc. are not yet implemented and will be done so as a follow-up.
## Test Plan
Add test cases for display support for `Signature` and various mdtest
for `typing.Callable`.
## Summary
Resolves#16365
Add support for unpacking `with` statement targets.
## Test Plan
Added some test cases, alike the ones added by #15058.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
* Attributes/method are now properly looked up on metaclasses, when
called on class objects
* We properly distinguish between data descriptors and non-data
descriptors (but we do not yet support them in store-context, i.e.
`obj.data_descr = …`)
* The descriptor protocol is now implemented in a single unified place
for instances, classes and dunder-calls. Unions and possibly-unbound
symbols are supported in all possible stages of the process by creating
union types as results.
* In general, the handling of "possibly-unbound" symbols has been
improved in a lot of places: meta-class attributes, attributes,
descriptors with possibly-unbound `__get__` methods, instance
attributes, …
* We keep track of type qualifiers in a lot more places. I anticipate
that this will be useful if we import e.g. `Final` symbols from other
modules (see relevant change to typing spec:
https://github.com/python/typing/pull/1937).
* Detection and special-casing of the `typing.Protocol` special form in
order to avoid lots of changes in the test suite due to new `@Todo`
types when looking up attributes on builtin types which have `Protocol`
in their MRO. We previously
looked up attributes in a wrong way, which is why this didn't come up
before.
closes#16367closes#15966
## Context
The way attribute lookup in `Type::member` worked before was simply
wrong (mostly my own fault). The whole instance-attribute lookup should
probably never have been integrated into `Type::member`. And the
`Type::static_member` function that I introduced in my last descriptor
PR was the wrong abstraction. It's kind of fascinating how far this
approach took us, but I am pretty confident that the new approach
proposed here is what we need to model this correctly.
There are three key pieces that are required to implement attribute
lookups:
- **`Type::class_member`**/**`Type::find_in_mro`**: The
`Type::find_in_mro` method that can look up attributes on class bodies
(and corresponding bases). This is a partial function on types, as it
can not be called on instance types like`Type::Instance(…)` or
`Type::IntLiteral(…)`. For this reason, we usually call it through
`Type::class_member`, which is essentially just
`type.to_meta_type().find_in_mro(…)` plus union/intersection handling.
- **`Type::instance_member`**: This new function is basically the
type-level equivalent to `obj.__dict__[name]` when called on
`Type::Instance(…)`. We use this to discover instance attributes such as
those that we see as declarations on class bodies or as (annotated)
assignments to `self.attr` in methods of a class.
- The implementation of the descriptor protocol. It works slightly
different for instances and for class objects, but it can be described
by the general framework:
- Call `type.class_member("attribute")` to look up "attribute" in the
MRO of the meta type of `type`. Call the resulting `Symbol` `meta_attr`
(even if it's unbound).
- Use `meta_attr.class_member("__get__")` to look up `__get__` on the
*meta type* of `meta_attr`. Call it with `__get__(meta_attr, self,
self.to_meta_type())`. If this fails (either the lookup or the call),
just proceed with `meta_attr`. Otherwise, replace `meta_attr` in the
following with the return type of `__get__`. In this step, we also probe
if a `__set__` or `__delete__` method exists and store it in
`meta_attr_kind` (can be either "data descriptor" or "normal attribute
or non-data descriptor").
- Compute a `fallback` type.
- For instances, we use `self.instance_member("attribute")`
- For class objects, we use `class_attr =
self.find_in_mro("attribute")`, and then try to invoke the descriptor
protocol on `class_attr`, i.e. we look up `__get__` on the meta type of
`class_attr` and call it with `__get__(class_attr, None, self)`. This
additional invocation of the descriptor protocol on the fallback type is
one major asymmetry in the otherwise universal descriptor protocol
implementation.
- Finally, we look at `meta_attr`, `meta_attr_kind` and `fallback`, and
handle various cases of (possible) unboundness of these symbols.
- If `meta_attr` is bound and a data descriptor, just return `meta_attr`
- If `meta_attr` is not a data descriptor, and `fallback` is bound, just
return `fallback`
- If `meta_attr` is not a data descriptor, and `fallback` is unbound,
return `meta_attr`
- Return unions of these three possibilities for partially-bound
symbols.
This allows us to handle class objects and instances within the same
framework. There is a minor additional detail where for instances, we do
not allow the fallback type (the instance attribute) to completely
shadow the non-data descriptor. We do this because we (currently) don't
want to pretend that we can statically infer that an instance attribute
is always set.
Dunder method calls can also be embedded into this framework. The only
thing that changes is that *there is no fallback type*. If a dunder
method is called on an instance, we do not fall back to instance
variables. If a dunder method is called on a class object, we only look
it up on the meta class, never on the class itself.
## Test Plan
New Markdown tests.
## Summary
This PR closes#15199.
The change I just made is to set all variables to type `Unknown` if
unpacking fails, but in some cases this may be excessive.
For example:
```py
a, b, c = "ab"
reveal_type(a) # Unknown, but it would be reasonable to think of it as LiteralString
reveal_type(c) # Unknown
```
```py
# Failed to unpack before the starred expression
(a, b, *c, d, e) = (1,)
reveal_type(a) # Unknown
reveal_type(b) # Unknown
...
# Failed to unpack after the starred expression
(a, b, *c, d, e) = (1, 2, 3)
reveal_type(a) # Unknown, but should it be Literal[1]?
reveal_type(b) # Unknown, but should it be Literal[2]?
reveal_type(c) # Todo
reveal_type(d) # Unknown
reveal_type(e) # Unknown
```
I will modify it if you think it would be better to make it a different
type than just `Unknown`.
## Test Plan
I have made appropriate modifications to the test cases affected by this
change, and also added some more test cases.
## Summary
- `Never` is callable
- `Never` is iterable
- Arbitrary attributes can be accessed on `Never`
Split out from #16416 that is going to be required.
## Test Plan
Tests for all properties above.
## Summary
This PR introduces a new mdtest option `system` that can either be
`in-memory` or `os`
where `in-memory` is the default.
The motivation for supporting `os` is so that we can write OS/system
specific tests
with mdtests. Specifically, I want to write mdtests for the module
resolver,
testing that module resolution is case sensitive.
## Test Plan
I tested that the case-sensitive module resolver test start failing when
setting `system = "os"`
## Summary
Python's module resolver is case sensitive.
This PR adds mdtests that assert that our module resolution is case
sensitive.
The tests currently all pass because our in memory file system is case
sensitive.
I'll add support for using the real file system to the mdtest framework
in a separate PR.
This PR also adds support for specifying extra search paths to the
mdtest framework.
## Test Plan
The tests fail when running them using the real file system.
To kick off the work of supporting generics, this adds many new
(currently failing) tests, showing the behavior we plan to support.
This is still missing a lot! Not included:
- typevar tuples
- param specs
- variance
- `Self`
But it's a good start! We can add more failing tests for those once we
tackle these.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR adds more features to #16468.
* Adds a new error rule `invalid-type-checking-constant`, which occurs
when we try to assign a value other than `False` to a user-defined
`TYPE_CHECKING` variable (it is possible to assign `...` in a stub
file).
* Allows annotated assignment to `TYPE_CHECKING`. Only types that
`False` can be assigned to are allowed. However, the type of
`TYPE_CHECKING` will be inferred to be `Literal[True]` regardless of
what the type is specified.
## Test plan
I ran the tests with `cargo test -p red_knot_python_semantic` and
confirmed that all tests passed.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR closes#15722.
The change is that if the variable `TYPE_CHECKING` is defined/imported,
the type of the variable is interpreted as `Literal[True]` regardless of
what the value is.
This is compatible with the behavior of other type checkers (e.g. mypy,
pyright).
## Test Plan
I ran the tests with `cargo test -p red_knot_python_semantic` and
confirmed that all tests passed.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Minor follow-up to https://github.com/astral-sh/ruff/pull/16161
This `not_callable` flag wasn't functional, because it could never be
`false`. It was initialized to `true` and then only ever updated with
`|=`, which can never make it `false`.
Add a test that exercises the case where it _should_ be `false` (all of
the union elements are callable) but `bindings` is also empty (all union
elements have binding errors). Before this PR, the added test wrongly
emits a diagnostic that the union `Literal[f1] | Literal[f2]` is not
callable.
And add a test where a union call results in one binding error and one
not-callable error, where we currently give the wrong result (we show
only the binding error), with a TODO.
Also add TODO comments in a couple other tests where ideally we'd report
more than just one error out of a union call.
Also update the flag name to `all_errors_not_callable` to more clearly
indicate the semantics of the flag.
## Summary
Model dunder-calls correctly (and in one single place), by implementing
this behavior (using `__getitem__` as an example).
```py
def getitem_desugared(obj: object, key: object) -> object:
getitem_callable = find_in_mro(type(obj), "__getitem__")
if hasattr(getitem_callable, "__get__"):
getitem_callable = getitem_callable.__get__(obj, type(obj))
return getitem_callable(key)
```
See the new `calls/dunder.md` test suite for more information. The new
behavior also needs much fewer lines of code (the diff is positive due
to new tests).
## Test Plan
New tests; fix TODOs in existing tests.
## Summary
Add a diagnostic if a pure instance variable is accessed on a class object. For example
```py
class C:
instance_only: str
def __init__(self):
self.instance_only = "a"
# error: Attribute `instance_only` can only be accessed on instances, not on the class object `Literal[C]` itself.
C.instance_only
```
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Add support for `@classmethod`s.
```py
class C:
@classmethod
def f(cls, x: int) -> str:
return "a"
reveal_type(C.f(1)) # revealed: str
```
## Test Plan
New Markdown tests
## Summary
I spotted a minor mistake in my descriptor protocol implementation where
`C.descriptor` would pass the meta type (`type`) of the type of `C`
(`Literal[C]`) as the owner argument to `__get__`, instead of passing
`Literal[C]` directly.
## Test Plan
New test.
## Summary
This PR achieves the following:
* Add support for checking method calls, and inferring return types from
method calls. For example:
```py
reveal_type("abcde".find("abc")) # revealed: int
reveal_type("foo".encode(encoding="utf-8")) # revealed: bytes
"abcde".find(123) # error: [invalid-argument-type]
class C:
def f(self) -> int:
pass
reveal_type(C.f) # revealed: <function `f`>
reveal_type(C().f) # revealed: <bound method: `f` of `C`>
C.f() # error: [missing-argument]
reveal_type(C().f()) # revealed: int
```
* Implement the descriptor protocol, i.e. properly call the `__get__`
method when a descriptor object is accessed through a class object or an
instance of a class. For example:
```py
from typing import Literal
class Ten:
def __get__(self, instance: object, owner: type | None = None) ->
Literal[10]:
return 10
class C:
ten: Ten = Ten()
reveal_type(C.ten) # revealed: Literal[10]
reveal_type(C().ten) # revealed: Literal[10]
```
* Add support for member lookup on intersection types.
* Support type inference for `inspect.getattr_static(obj, attr)` calls.
This was mostly used as a debugging tool during development, but seems
more generally useful. It can be used to bypass the descriptor protocol.
For the example above:
```py
from inspect import getattr_static
reveal_type(getattr_static(C, "ten")) # revealed: Ten
```
* Add a new `Type::Callable(…)` variant with the following sub-variants:
* `Type::Callable(CallableType::BoundMethod(…))` — represents bound
method objects, e.g. `C().f` above
* `Type::Callable(CallableType::MethodWrapperDunderGet(…))` — represents
`f.__get__` where `f` is a function
* `Type::Callable(WrapperDescriptorDunderGet)` — represents
`FunctionType.__get__`
* Add new known classes:
* `types.MethodType`
* `types.MethodWrapperType`
* `types.WrapperDescriptorType`
* `builtins.range`
## Performance analysis
On this branch, we do more work. We need to do more call checking, since
we now check all method calls. We also need to do ~twice as many member
lookups, because we need to check if a `__get__` attribute exists on
accessed members.
A brief analysis on `tomllib` shows that we now call `Type::call` 1780
times, compared to 612 calls before.
## Limitations
* Data descriptors are not yet supported, i.e. we do not infer correct
types for descriptor attribute accesses in `Store` context and do not
check writes to descriptor attributes. I felt like this was something
that could be split out as a follow-up without risking a major
architectural change.
* We currently distinguish between `Type::member` (with descriptor
protocol) and `Type::static_member` (without descriptor protocol). The
former corresponds to `obj.attr`, the latter corresponds to
`getattr_static(obj, "attr")`. However, to model some details correctly,
we would also need to distinguish between a static member lookup *with*
and *without* instance variables. The lookup without instance variables
corresponds to `find_name_in_mro`
[here](https://docs.python.org/3/howto/descriptor.html#invocation-from-an-instance).
We currently approximate both using `member_static`, which leads to two
open TODOs. Changing this would be a larger refactoring of
`Type::own_instance_member`, so I chose to leave it out of this PR.
## Test Plan
* New `call/methods.md` test suite for method calls
* New tests in `descriptor_protocol.md`
* New `call/getattr_static.md` test suite for `inspect.getattr_static`
* Various updated tests
## Summary
Follow up on the discussion
[here](https://github.com/astral-sh/ruff/pull/16121#discussion_r1962973298).
Replace builtin classes with custom placeholder names, which should
hopefully make the tests a bit easier to understand.
I carefully renamed things one after the other, to make sure that there
is no functional change in the tests.
We now resolve references in "eager" scopes correctly — using the
bindings and declarations that are visible at the point where the eager
scope is created, not the "public" type of the symbol (typically the
bindings visible at the end of the scope).
---------
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
This uses the refactoring and support for secondary diagnostic messages
to improve the diagnostic for "invalid argument type." The main
improvement here is that we show where the function being called is
defined, and annotate the span corresponding to the invalid parameter.
When adjusting the existing tests, I aimed to avoid dealing with the
special case in other tests if it's not necessary to do so (that is,
avoid using `float` and `complex` as examples where we just need "some
type"), and keep the tests for the special case mostly collected in the
mdtest dedicated to that purpose.
Fixes https://github.com/astral-sh/ruff/issues/14932
This is an alternative implementation to #15848.
## Summary
This PR adds support for re-export conventions for imports for stub
files.
**How does this work?**
* Add a new flag on the `Import` and `ImportFrom` definitions to
indicate whether they're being exported or not
* Add a new enum to indicate whether the symbol lookup is happening
within the same file or is being queried from another file (e.g., an
import statement)
* When a `Symbol` is being queried, we'll skip the definitions that are
(a) coming from a stub file (b) external lookup and (c) check the
re-export flag on the definition
This implementation does not yet support `__all__` and `*` imports as
both are features that needs to be implemented independently.
closes: #14099closes: #15476
## Test Plan
Add test cases, update existing ones if required.
## Summary
After I was asked twice within the same day, I thought it would be a
good idea to write some *user facing* documentation that explains our
reasoning behind inferring `Unknown | T_inferred` for public uses of
undeclared symbols. This is a major deviation from the behavior of other
type checkers and it seems like a good practice to defend our choice
like this.
## Summary
- Simplify unions with `object` to `object`.
- Add a new `Type::object(db)` constructor to abbreviate
`KnownClass::Object.to_instance(db)` in some places.
- Add a `Type::is_object` and `Class::is_object` function to make some
tests for a bit easier to read.
closes#16084
## Test Plan
New Markdown tests.
## Summary
* Support assignments to attributes in more cases:
- assignments in `for` loops
- in unpacking assignments
* Add test for multi-target assignments
* Add tests for all other possible assignments to attributes that could
possibly occur (in decreasing order of likeliness):
- augmented attribute assignments
- attribute assignments in `with` statements
- attribute assignments in comprehensions
- Note: assignments to attributes in named expressions are not
syntactically allowed
closes#15962
## Test Plan
New Markdown tests
## Summary
This PR reverts the behavior changes from
https://github.com/astral-sh/ruff/pull/15990
But it isn't just a revert, it also:
* Adds a test covering this specific behavior
* Preserves the improvement to use `saturating_sub` in the package case
to avoid overflows in the case of invalid syntax
* Use `ancestors` instead of a `for` loop
## Test Plan
Added test
## Summary
I noticed that the diagnostic range in specific unpacking assignments is
wrong. For this example
```py
a, b = 1
```
we previously got (see first commit):
```
error: lint:not-iterable
--> /src/mdtest_snippet.py:1:1
|
1 | a, b = 1
| ^^^^ Object of type `Literal[1]` is not iterable
|
```
and with this change, we get:
```
error: lint:not-iterable
--> /src/mdtest_snippet.py:1:8
|
1 | a, b = 1
| ^ Object of type `Literal[1]` is not iterable
|
```
## Test Plan
New snapshot tests.
## Summary
Fixes https://github.com/astral-sh/ruff/issues/15989
Red Knot failed to resolve relative imports if the importing module is
located at a search path root.
The issue was that the module resolver returned an `Err(TooManyDots)` as
soon as the parent of the current module is `None` (which is the case
for a module at the search path root).
However, this is incorrect if a `tail` (a module name) exists.
## Summary
- Minor wording update
- Code improvement (thanks Alex)
- Removed all unnecessary filenames throughout our Markdown tests (two
new ones were added in the meantime)
- Minor rewording of the statically-known-branches introduction
This example from @sharkdp shows how terminal statements can appear in
statically known branches:
https://github.com/astral-sh/ruff/pull/15676#issuecomment-2618809716
```py
def _(cond: bool):
x = "a"
if cond:
x = "b"
if True:
return
reveal_type(x) # revealed: "a", "b"; should be "a"
```
We now use visibility constraints to track reachability, which allows us
to model this correctly. There are two related changes as a result:
- New bindings are not assumed to be visible; they inherit the current
"scope start" visibility, which effectively means that new bindings are
visible if/when the current flow is reachable
- When simplifying visibility constraints after branching control flow,
we only simplify if none of the intervening branches included a terminal
statement. That is, earlier unaffected bindings are only _actually_
unaffected if all branches make it to the merge point.
