## Summary
Currently if I run `uv run crates/red_knot_python_semantic/mdtest.py`
from the Ruff repo root, I get this output:
```
~/dev/ruff (main)⚡ % uv run crates/red_knot_python_semantic/mdtest.py
Ready to watch for changes...
```
...And I then have to make some spurious whitespace changes or something
to a test file in order to get the script to actually run mdtest. This
PR changes mdtest.py so that it does an initial run of all mdtests when
you invoke the script, and _then_ starts watching for changes in test
files/Rust code.
## Summary
This PR fixes a bug in callable subtyping to consider both the
positional and keyword form of the standard parameter in the supertype
when matching against variadic, keyword-only and keyword-variadic
parameter in the subtype.
This is done by collecting the unmatched standard parameters and then
checking them against the keyword-only / keyword-variadic parameters
after the positional loop.
## Test Plan
Add test cases.
## Summary
There are quite a few places we infer `Todo` types currently, and some
of them are nested somewhat deeply in type expressions. These can cause
spurious issues for the new `redundant-cast` diagnostics. We fixed all
the false positives we saw in the mypy_primer report before merging
https://github.com/astral-sh/ruff/pull/17100, but I think there are
still lots of places where we'd emit false positives due to this check
-- we currently don't run on that many projects at all in our
mypy_primer check:
d0c8eaa092/.github/workflows/mypy_primer.yaml (L71)
This PR fixes some more false positives from this diagnostic by making
the `Type::contains_todo()` method more expansive.
## Test Plan
I added a regression test which causes us to emit a spurious diagnostic
on `main`, but does not with this PR.
## Summary
In https://github.com/python/typeshed/pull/13520 the typeshed definition
of `typing.Any` was changed from `Any = object()` to `class Any: ...`.
Our automated typeshed updater pulled down this change in
https://github.com/astral-sh/ruff/pull/17106, with the consequence that
we no longer understand `Any`, which is... not good.
This PR gives us the ability to understand `Any` defined as a class
instead of `object()`. It doesn't remove our ability to understand the
old form. Perhaps at some point we'll want to remove it, but for now we
may as well support both old and new typeshed?
This also directly patches typeshed to use the new form of `Any`; this
is purely to work around our tests that no known class is inferred as
`Unknown`, which otherwise fail with the old typeshed and the changes in
this PR. (All other tests pass.) This patch to typeshed will shortly be
subsumed by https://github.com/astral-sh/ruff/pull/17106 anyway.
## Test Plan
Without the typeshed change in this PR, all tests pass except for the
two `known_class_doesnt_fallback_to_unknown_unexpectedly_*` tests (so we
still support the old form of defining `Any`). With the typeshed change
in this PR, all tests pass, so we now support the new form in a way that
is indistinguishable to our test suite from the old form. And
indistinguishable to the ecosystem check: after rebasing
https://github.com/astral-sh/ruff/pull/17106 on this PR, there's zero
ecosystem impact.
## Summary
I noticed we were inferring `Todo` as the declared type for annotations
such as `x: tuple[list[int], list[int]]`. This PR reworks our annotation
parsing so that we instead infer `tuple[Todo, Todo]` for this
annotation, which is quite a bit more precise.
## Test Plan
Existing mdtest updated.
## Summary
Following up from earlier discussion on Discord, this PR adds logic to
flag casts as redundant when the inferred type of the expression is the
same as the target type. It should follow the semantics from
[mypy](https://github.com/python/mypy/pull/1705).
Example:
```python
def f() -> int:
return 10
# error: [redundant-cast] "Value is already of type `int`"
cast(int, f())
```
## Summary
Part of #13694
Seems there a bit more to cover regarding `in` and other types, but i
can cover them in different PRs
## Test Plan
Add `in.md` file in narrowing conditionals folder
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Rewrites the virtual env discovery to:
* Only use of `System` APIs, this ensures that the discovery will also
work when using a memory file system (testing or WASM)
* Don't traverse ancestor directories. We're not convinced that this is
necessary. Let's wait until someone shows us a use case where it is
needed
* Start from the project root and not from the current working
directory. This ensures that Red Knot picks up the right venv even when
using `knot --project ../other-dir`
## Test Plan
Existing tests, @ntBre tested that the `file_watching` tests no longer
pick up his virtual env in a parent directory
## Summary
In preparation for #17017, where we will need them to suppress new false
positives (once we understand the `ParamSpec.args`/`ParamSpec.kwargs`
properties).
## Test Plan
Tested on branch #17017
## Summary
Disallow empty `todo_type!()`s without a custom message. They can lead
to spurious diffs in `mypy_primer` where the only thing that's changed
is the file/line information.
## Summary
A quick fix for how union/intersection member search ins performed in
Knot.
## Test Plan
* Added a dunder method call test for Union, which exhibits the error
* Also added an intersection error, but it is not triggering currently
due to `call` logic not being fully implemented for intersections.
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Closes#16661
This PR includes two changes:
- `NotImplementedType` is now a member of `KnownClass`
- We skip `is_assignable_to` checks for `NotImplemented` when checking
return types
### Limitation
```py
def f(cond: bool) -> int:
return 1 if cond else NotImplemented
```
The implementation covers cases where `NotImplemented` appears inside a
`Union`.
However, for more complex types (ex. `Intersection`) it will not worked.
In my opinion, supporting such complexity is unnecessary at this point.
## Test Plan
Two `mdtest` files were updated:
- `mdtest/function/return_type.md`
- `mdtest/type_properties/is_singleton.md`
To test `KnownClass`, run:
```bash
cargo test -p red_knot_python_semantic -- types::class::
```
## Summary
This PR refactors the common logic for unpacking in assignment, for loops, and with items.
## Test Plan
Make sure existing tests pass.
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Fixes#16744
Code from
bbf4f830b5/crates/uv-python/src/virtualenv.rs (L124-L144)
## Test Plan
Manual testing
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
From #16861, and the continuation of #16915.
This PR fixes the incorrect behavior of
`TypeInferenceBuilder::infer_name_load` in eager nested scopes.
And this PR closes#16341.
## Test Plan
New test cases are added in `annotations/deferred.md`.
## Summary
Part of #13694
Narrow in or-patterns by taking the type union of the type constraints
in each disjunct pattern.
## Test Plan
Add new tests to narrow/match.md
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Part of #13694
The implementation here was suspiciously straightforward so please lmk
if I missed something
Also some drive-by changes to DRY things up a bit
## Test Plan
Add new tests to narrow/match.md
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR adds some branches so that we infer `Todo` types for attribute
access on instances of `super()` and subtypes of `type[Enum]`. It reduces
false positives in the short term until we implement full support for
these features.
## Test Plan
New mdtests added + mypy_primer report
## Summary
Mainly for partially fixing #16953
## Test Plan
Update is_subtype tests. And should maybe do these checks for many other
types (is subtype of object but object is not subtype)
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
## Summary
Further work towards https://github.com/astral-sh/ruff/issues/14169.
We currently panic on encountering cyclic `*` imports. This is easily
fixed using fixpoint iteration.
## Test Plan
Added a test that panics on `main`, but passes with this PR
## Summary
As mentioned in
https://github.com/astral-sh/ruff/pull/16698#discussion_r2004920075,
part of #15382, this PR updates the `is_gradual_equivalent_to`
implementation between callable types to be similar to
`is_equivalent_to` and checks other attributes of parameters like name,
optionality, and parameter kind.
## Test Plan
Expand the existing test cases to consider other properties but not all
similar to how the tests are structured for subtyping and assignability.
## Summary
This PR adds initial support for `*` imports to red-knot. The approach
is to implement a standalone query, called from semantic indexing, that
visits the module referenced by the `*` import and collects all
global-scope public names that will be imported by the `*` import. The
`SemanticIndexBuilder` then adds separate definitions for each of these
names, all keyed to the same `ast::Alias` node that represents the `*`
import.
There are many pieces of `*`-import semantics that are still yet to be
done, even with this PR:
- This PR does not attempt to implement any of the semantics to do with
`__all__`. (If a module defines `__all__`, then only the symbols
included in `__all__` are imported, _not_ all public global-scope
symbols.
- With the logic implemented in this PR as it currently stands, we
sometimes incorrectly consider a symbol bound even though it is defined
in a branch that is statically known to be dead code, e.g. (assuming the
target Python version is set to 3.11):
```py
# a.py
import sys
if sys.version_info < (3, 10):
class Foo: ...
```
```py
# b.py
from a import *
print(Foo) # this is unbound at runtime on 3.11,
# but we currently consider it bound with the logic in this PR
```
Implementing these features is important, but is for now deferred to
followup PRs.
Many thanks to @ntBre, who contributed to this PR in a pairing session
on Friday!
## Test Plan
Assertions in existing mdtests are adjusted, and several new ones are
added.
## Summary
Here I fix the last English spelling errors I could find in the repo.
Again, I am trying not to touch variable/function names, or anything
that might be misspelled in the API. The goal is to make this PR safe
and easy to merge.
## Test Plan
I have run all the unit tests. Though, again, all of the changes I make
here are to docs and docstrings. I make no code changes, which I believe
should greatly mitigate the testing concerns.
## Summary
Resolves#16895.
`abstractmethod` is now a `KnownFunction`. When a function is decorated
by `abstractmethod` or when the parent class inherits directly from
`Protocol`, `invalid-return-type` won't be emitted for that function.
## Test Plan
Markdown tests.
---------
Co-authored-by: Carl Meyer <carl@oddbird.net>
## Summary
Fixes#16912
Create a new type `DisplayMaybeParenthesizedType` that is now used in
Union and Intersection display
## Test Plan
Update callable annotations
## Summary
From #16861
This PR fixes the incorrect `ClassDef` handling of
`SemanticIndexBuilder::visit_stmt`, which fixes some of the incorrect
behavior of referencing the class itself in the class scope (a complete
fix requires a different fix, which will be done in the another PR).
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
This is a cleanup PR. I am fixing various English language spelling
errors. This is mostly in docs and docstrings.
## Test Plan
The usual CI tests were run. I tried to build the docs (though I had
some troubles there). The testing needs here are, I trust, very low
impact. (Though I would happily test more.)
## Summary
Log the origin of the sys path prefix. This should help with debugging
if someone doesn't understand
why Red Knot picks up a certain venv.
## Test Plan
Ran the CLI and tested that it logs the origin
## Summary
Part of #15382, this PR adds support for calling a variable that's
annotated with `typing.Callable`.
## Test Plan
Add test cases in a new `call/annotation.md` file.
## Summary
Part of #15382
This PR adds support for checking the assignability of two general
callable types.
This is built on top of #16804 by including the gradual parameters check
and accepting a function that performs the check between the two types.
## Test Plan
Update `is_assignable_to.md` with callable types section.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
When callables are displayed in unions, like:
```py
from typing import Callable
def foo(x: Callable[[], int] | None):
# red-knot: Revealed type is `() -> int | None` [revealed-type]
reveal_type(x)
```
This leaves the type rather ambiguous, to fix this we can add
parenthesis to callable type in union
Fixes#16893
## Test Plan
Update callable annotations tests
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
Summary
--
Detects duplicate type parameter names in function definitions, class
definitions, and type alias statements.
I also boxed the `type_params` field on `StmtTypeAlias` to make it
easier to
`match` with functions and classes. (That's the reason for the red-knot
code
owner review requests, sorry!)
Test Plan
--
New `ruff_python_syntax_errors` unit tests.
Fixes#11119.
## Summary
This PR removes false-positive diagnostics for `*` imports. Currently we
always emit a diagnostic for these statements unless the module we're
importing from has a symbol named `"*"` in its symbol table for the
global scope. (And if we were doing everything correctly, no module ever
would have a symbol named `"*"` in its global scope!)
The fix here is sort-of hacky and won't be what we'll want to do
long-term. However, I think it's useful to do this as a first step
since:
- It significantly reduces false positives when running on code that
uses `*` imports
- It "resets" the tests to a cleaner state with many fewer TODOs, making
it easier to see what the hard work is that's still to be done.
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
This PR adds a suite of tests for wildcard (`*`) imports. The tests
nearly all fail for now, and those that don't, ahem, pass for the wrong
reasons...
I've tried to add TODO comments in all instances for places where we are
currently inferring the incorrect thing, incorrectly emitting a
diagnostic, or emitting a diagnostic with a bad error message.
## Test Plan
`cargo test -p red_knot_python_semantic`
This breaks up call binding into two phases:
- **_Matching parameters_** just looks at the names and kinds
(positional/keyword) of each formal and actual parameters, and matches
them up. Most of the current call binding errors happen during this
phase.
- Once we have matched up formal and actual parameters, we can **_infer
types_** of each actual parameter, and **_check_** that each one is
assignable to the corresponding formal parameter type.
As part of this, we add information to each formal parameter about
whether it is a type form or not. Once [PEP
747](https://peps.python.org/pep-0747/) is finalized, we can hook that
up to this internal type form representation. This replaces the
`ParameterExpectations` type, which did the same thing in a more ad hoc
way.
While we're here, we add a new fluent API for building `Parameter`s,
which makes our signature constructors a bit nicer to read. We also
eliminate a TODO where we were consuming types from the argument list
instead of the bound parameter list when evaluating our special-case
known functions.
Closes#15460
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
Part of #15382
This PR adds support for checking the subtype relationship between the
two callable types.
The main source of reference used for implementation is
https://typing.python.org/en/latest/spec/callables.html#assignability-rules-for-callables.
The implementation is split into two phases:
1. Check all the positional parameters which includes positional-only,
standard (positional or keyword) and variadic kind
2. Collect all the keywords in a `HashMap` to do the keyword parameters
check via name lookup
For (1), there's a helper struct which is similar to `.zip_longest`
(from `itertools`) except that it allows control over one of the
iterator as that's required when processing a variadic parameter. This
is required because positional parameters needs to be checked as per
their position between the two callable types. The struct also keeps
track of the current iteration element because when the loop is exited
(to move on to the phase 2) the current iteration element would be
carried over to the phase 2 check.
This struct is internal to the `is_subtype_of` method as I don't think
it makes sense to expose it outside. It also allows me to use "self" and
"other" suffixed field names as that's only relevant in that context.
## Test Plan
Add extensive tests in markdown.
Converted all of the code snippets from
https://typing.python.org/en/latest/spec/callables.html#assignability-rules-for-callables
to use `knot_extensions.is_subtype_of` and verified the result.
## Summary
This PR checks whether two callable types are equivalent or not.
This is required because for an equivalence relationship, the default
value does not necessarily need to be the same but if the parameter in
one of the callable has a default value then the corresponding parameter
in the other callable should also have a default value. This is the main
reason a manual implementation is required.
And, as per https://typing.python.org/en/latest/spec/callables.html#id4,
the default _type_ doesn't participate in a subtype relationship, only
the optionality (required or not) participates. This means that the
following two callable types are equivalent:
```py
def f1(a: int = 1) -> None: ...
def f2(a: int = 2) -> None: ...
```
Additionally, the name of positional-only, variadic and keyword-variadic
are not required to be the same for an equivalence relation.
A potential solution to avoid the manual implementation would be to only
store whether a parameter has a default value or not but the type is
currently required to check for assignability.
## Test plan
Add tests for callable types in `is_equivalent_to.md`
## Summary
Catch some Instances, but raise type error for the rest of them
Fixes#16851
## Test Plan
Extend invalid.md in annotations
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Currently for something like `X = typing.Tuple[str, str]`, we infer the
value of `X` as `object`. That's because `Tuple` (like many of the
symbols in the typing module) is annotated as a `_SpecialForm` instance
in typeshed's stubs:
23382f5f8c/crates/red_knot_vendored/vendor/typeshed/stdlib/typing.pyi (L215)
and we don't understand implicit type aliases yet, and the stub for
`_SpecialForm.__getitem__` says it always returns `object`:
23382f5f8c/crates/red_knot_vendored/vendor/typeshed/stdlib/typing.pyi (L198-L200)
We have existing false positives in our test suite due to this:
23382f5f8c/crates/red_knot_python_semantic/resources/mdtest/annotations/annotated.md (L76-L78)
and it's causing _many_ new false positives in #16872, which tries to
make our annotation-expression parsing stricter in some ways.
This PR therefore adds some small special casing for `KnownInstanceType`
variants that fallback to `_SpecialForm`, so that these false positives
can be avoided.
## Test Plan
Existing mdtest altered.
Cc. @MatthewMckee4
## Summary
From #16641
The previous PR attempted to fix the errors presented in this PR, but as
discussed in the conversation, it was concluded that the approach was
undesirable and that further work would be needed to fix the errors with
a correct general solution.
In this PR, I instead add the test cases from the previous PR as TODOs,
as a starting point for future work.
## Test Plan
---------
Co-authored-by: Carl Meyer <carl@oddbird.net>
## Summary
Fixes#16744
Allows the cli to find a virtual environment from the VIRTUAL_ENV
environment variable if no `--python` is set
## Test Plan
Manual testing, of:
- Virtual environments explicitly activated using `source .venv/bin/activate`
- Virtual environments implicilty activated via `uv run`
- Broken virtual environments with no `pyvenv.cfg` file
