We now handle generic constructor methods on generic classes correctly:
```py
class C[T]:
def __init__[S](self, t: T, s: S): ...
x = C(1, "str")
```
Here, constructing `C` requires us to infer a specialization for the
generic contexts of `C` and `__init__` at the same time.
At first I thought I would need to track the full stack of nested
generic contexts here (since the `[S]` context is nested within the
`[T]` context). But I think this is the only way that we might need to
specialize more than one generic context at once — in all other cases, a
containing generic context must be specialized before we get to a nested
one, and so we can just special-case this.
While we're here, we also construct the generic context for a generic
function lazily, when its signature is accessed, instead of eagerly when
inferring the function body.
## Summary
Model assignability of class instances with a `__call__` method to
`Callable` types. This should solve some false positives related to
`functools.partial` (yes, 1098 fewer diagnostics!).
Reference:
https://github.com/astral-sh/ruff/issues/17343#issuecomment-2824618483
## Test Plan
New Markdown tests.
## Summary
Many symbols in typeshed are defined without being declared. For
example:
```pyi
# builtins:
IOError = OSError
# types
LambdaType = FunctionType
NotImplementedType = _NotImplementedType
# typing
Text = str
# random
uniform = _inst.uniform
# optparse
make_option = Option
# all over the place:
_T = TypeVar("_T")
```
Here, we introduce a change that skips widening the public type of these
symbols (by unioning with `Unknown`).
fixes#17032
## Ecosystem analysis
This is difficult to analyze in detail, but I went over most changes and
it looks very favorable to me overall. The diff on the overall numbers
is:
```
errors: 1287 -> 859 (reduction by 428)
warnings: 45 -> 59 (increase by 14)
```
### Removed false positives
`invalid-base` examples:
```diff
- error[lint:invalid-base] /tmp/mypy_primer/projects/pip/src/pip/_vendor/rich/console.py:548:27: Invalid class base with type `Unknown | Literal[_local]` (all bases must be a class, `Any`, `Unknown` or `Todo`)
- error[lint:invalid-base] /tmp/mypy_primer/projects/tornado/tornado/iostream.py:84:25: Invalid class base with type `Unknown | Literal[OSError]` (all bases must be a class, `Any`, `Unknown` or `Todo`)
- error[lint:invalid-base] /tmp/mypy_primer/projects/mitmproxy/test/conftest.py:35:40: Invalid class base with type `Unknown | Literal[_UnixDefaultEventLoopPolicy]` (all bases must be a class, `Any`, `Unknown` or `Todo`)
```
`invalid-exception-caught` examples:
```diff
- error[lint:invalid-exception-caught] /tmp/mypy_primer/projects/cloud-init/cloudinit/cmd/status.py:334:16: Cannot catch object of type `Literal[ProcessExecutionError]` in an exception handler (must be a `BaseException` subclass or a tuple of `BaseException` subclasses)
- error[lint:invalid-exception-caught] /tmp/mypy_primer/projects/jinja/src/jinja2/loaders.py:537:16: Cannot catch object of type `Literal[TemplateNotFound]` in an exception handler (must be a `BaseException` subclass or a tuple of `BaseException` subclasses)
```
`unresolved-reference` examples
7a0265d36e/cloudinit/handlers/jinja_template.py (L120-L123)
(we now understand the `isinstance` narrowing)
```diff
- error[lint:unresolved-attribute] /tmp/mypy_primer/projects/cloud-init/cloudinit/handlers/jinja_template.py:123:16: Type `Exception` has no attribute `errno`
```
`unknown-argument` examples
https://github.com/hauntsaninja/boostedblob/blob/master/boostedblob/request.py#L53
```diff
- error[lint:unknown-argument] /tmp/mypy_primer/projects/boostedblob/boostedblob/request.py:53:17: Argument `connect` does not match any known parameter of bound method `__init__`
```
`unknown-argument`
There are a lot of `__init__`-related changes because we now understand
[`@attr.s`](3d42a6978a/src/attr/__init__.pyi (L387))
as a `@dataclass_transform` annotated symbol. For example:
```diff
- error[lint:unknown-argument] /tmp/mypy_primer/projects/attrs/tests/test_hooks.py:72:18: Argument `x` does not match any known parameter of bound method `__init__`
```
### New false positives
This can happen if a symbol that previously was inferred as `X |
Unknown` was assigned-to, but we don't yet understand the assignability
to `X`:
https://github.com/strawberry-graphql/strawberry/blob/main/strawberry/exceptions/handler.py#L90
```diff
+ error[lint:invalid-assignment] /tmp/mypy_primer/projects/strawberry/strawberry/exceptions/handler.py:90:9: Object of type `def strawberry_threading_exception_handler(args: tuple[type[BaseException], BaseException | None, TracebackType | None, Thread | None]) -> None` is not assignable to attribute `excepthook` of type `(_ExceptHookArgs, /) -> Any`
```
### New true positives
6bbb5519fe/tests/tracer/test_span.py (L714)
```diff
+ error[lint:invalid-argument-type] /tmp/mypy_primer/projects/dd-trace-py/tests/tracer/test_span.py:714:33: Argument to this function is incorrect: Expected `str`, found `Literal[b"\xf0\x9f\xa4\x94"]`
```
### Changed diagnostics
A lot of changed diagnostics because we now show `@Todo(Support for
`typing.TypeVar` instances in type expressions)` instead of `Unknown`
for all kinds of symbols that used a `_T = TypeVar("_T")` as a type. One
prominent example is the `list.__getitem__` method:
`builtins.pyi`:
```pyi
_T = TypeVar("_T") # previously `TypeVar | Unknown`, now just `TypeVar`
# …
class list(MutableSequence[_T]):
# …
@overload
def __getitem__(self, i: SupportsIndex, /) -> _T: ...
# …
```
which causes this change in diagnostics:
```py
xs = [1, 2]
reveal_type(xs[0]) # previously `Unknown`, now `@Todo(Support for `typing.TypeVar` instances in type expressions)`
```
## Test Plan
Updated Markdown tests
## Summary
Apply auto fixes to cases where the names have changed in Airflow 3
## Test Plan
Add `AIR301_names_fix.py` and `AIR301_provider_names_fix.py` test fixtures
This pull request fixes https://github.com/astral-sh/ruff/issues/17014
changes this
```python
from __future__ import annotations
flag1 = True
flag2 = True
if flag1 == True or flag2 == True:
pass
if flag1 == False and flag2 == False:
pass
flag3 = True
if flag1 == flag3 and (flag2 == False or flag3 == True): # Should become: if flag1==flag3 and (not flag2 or flag3)
pass
if flag1 == True and (flag2 == False or not flag3 == True): # Should become: if flag1 and (not flag2 or not flag3)
pass
if flag1 != True and (flag2 != False or not flag3 == True): # Should become: if not flag1 and (flag2 or not flag3)
pass
flag = True
while flag == True: # Should become: while flag
flag = False
flag = True
x = 5
if flag == True and x > 0: # Should become: if flag and x > 0
print("ok")
flag = True
result = "yes" if flag == True else "no" # Should become: result = "yes" if flag else "no"
x = flag == True < 5
x = (flag == True) == False < 5
```
to this
```python
from __future__ import annotations
flag1 = True
flag2 = True
if flag1 or flag2:
pass
if not flag1 and not flag2:
pass
flag3 = True
if flag1 == flag3 and (not flag2 or flag3): # Should become: if flag1 == flag3 and (not flag2 or flag3)
pass
if flag1 and (not flag2 or not flag3): # Should become: if flag1 and (not flag2 or not flag3)
pass
if not flag1 and (flag2 or not flag3): # Should become: if not flag1 and (flag2 or not flag3)
pass
flag = True
while flag: # Should become: while flag
flag = False
flag = True
x = 5
if flag and x > 0: # Should become: if flag and x > 0
print("ok")
flag = True
result = "yes" if flag else "no" # Should become: result = "yes" if flag else "no"
x = flag is True < 5
x = (flag) is False < 5
```
---------
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
Summary
--
This PR extends semantic syntax error detection to red-knot. The main
changes here are:
1. Adding `SemanticSyntaxChecker` and `Vec<SemanticSyntaxError>` fields
to the `SemanticIndexBuilder`
2. Calling `SemanticSyntaxChecker::visit_stmt` and `visit_expr` in the
`SemanticIndexBuilder`'s `visit_stmt` and `visit_expr` methods
3. Implementing `SemanticSyntaxContext` for `SemanticIndexBuilder`
4. Adding new mdtests to test the context implementation and show
diagnostics
(3) is definitely the trickiest and required (I think) a minor addition
to the `SemanticIndexBuilder`. I tried to look around for existing code
performing the necessary checks, but I definitely could have missed
something or misused the existing code even when I found it.
There's still one TODO around `global` statement handling. I don't think
there's an existing way to look this up, but I'm happy to work on that
here or in a separate PR. This currently only affects detection of one
error (`LoadBeforeGlobalDeclaration` or
[PLE0118](https://docs.astral.sh/ruff/rules/load-before-global-declaration/)
in ruff), so it's not too big of a problem even if we leave the TODO.
Test Plan
--
New mdtests, as well as new errors for existing mdtests
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Allow (instances of) subclasses of `Any` and `Unknown` to be assignable
to (instances of) other classes, unless they are final. This allows us
to get rid of ~1000 false positives, mostly when mock-objects like
`unittest.mock.MagicMock` are assigned to various targets.
## Test Plan
Adapted and new Markdown tests.
## Summary
mypy_primer changes included here:
ebaa9fd27b..4c22d192a4
- Add strawberry as a `good.txt` project (was previously included in our
fork)
- Print Red Knot compilation errors to stderr (thanks @MichaReiser)
## Summary
We currently emit a diagnostic for code like the following:
```py
from typing import Any
# error: Invalid class base with type `GenericAlias` (all bases must be a class, `Any`, `Unknown` or `Todo`)
class C(tuple[Any, ...]): ...
```
The changeset here silences this diagnostic by recognizing instances of
`GenericAlias` in `ClassBase::try_from_type`, and inferring a `@Todo`
type for them. This is a change in preparation for #17557, because `C`
previously had `Unknown` in its MRO …
```py
reveal_type(C.__mro__) # tuple[Literal[C], Unknown, Literal[object]]
```
… which would cause us to think that `C` is assignable to everything.
The changeset also removes some false positive `invalid-base`
diagnostics across the ecosystem.
## Test Plan
Updated Markdown tests.
## Summary
Add parentheses to multi-element intersections, when displayed in a
context that's otherwise potentially ambiguous.
## Test Plan
Update mdtest files
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
#17451 was incomplete. `AlwaysFalsy` and `AlwaysTruthy` are not the only
two types that are super-types of some literals (of a given kind) and
not others. That set also includes intersections containing
`AlwaysTruthy` or `AlwaysFalsy`, and intersections containing literal
types of the same kind. Cover these cases as well.
Fixes#17478.
## Test Plan
Added mdtests.
`QUICKCHECK_TESTS=1000000 cargo test -p red_knot_python_semantic --
--ignored types::property_tests::stable` failed on both
`all_fully_static_type_pairs_are_subtypes_of_their_union` and
`all_type_pairs_are_assignable_to_their_union` prior to this PR, passes
after it.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
I gave up trying to do this one lint at a time and just (mostly)
mechanically translated this entire file in one go.
Generally the messages stay the same (with most moving from an
annotation message to the diagnostic's main message). I added a couple
of `info` sub-diagnostics where it seemed to be the obvious intent.
This finishes the migration for the `INVALID_ASSIGNMENT` lint.
Notice how I'm steadily losing steam in terms of actually improving the
diagnostics. This change is more mechanical, because taking the time to
revamp every diagnostic is a ton of effort. Probably future migrations
will be similar unless there are easy pickings.
We mostly keep things the same here, but the message has been moved from
the annotation to the diagnostic's top-line message. I think this is
perhaps a little worse, but some bigger improvements could be made here.
Indeed, we could perhaps even add a "fix" here.
This moves all INVALID_ASSIGNMENT lints related to unpacking over to the new
diagnostic model.
While we're here, we improve the diagnostic a bit by adding a secondary
annotation covering where the value is. We also split apart the original
singular message into one message for the diagnostic and the "expected
versus got" into annotation messages.
This tests the diagnostic rendering of a case that wasn't previously
covered by snapshots: when unpacking fails because there are too few
values, but where the left hand side can tolerate "N or more." In the
code, this is a distinct diagnostic, so we capture it here.
(Sorry about the diff here, but it made sense to rename the other
sections and that changes the name of the snapshot file.)
I believe this was an artifact of an older iteration of the diagnostic
reporting API. But this is strictly not necessary now, and indeed, might
even be annoying. It is okay, but perhaps looks a little odd, to do
`builder.into_diagnostic("...")` if you don't want to add anything else
to the diagnostic.
I suspect this will be used pretty frequently (I wanted it
immediately). And more practically, this avoids needing to
import `Annotation` to create it.
## Summary
I ran red-knot on every project in mypy-primer. I moved every project
where red-knot ran to completion (fast enough, and mypy-primer could
handle its output) into `good.txt`, so it will run in our CI.
The remaining projects I left listed in `bad.txt`, with a comment
summarizing the failure mode (a few don't fail, they are just slow -- on
a debug build, at least -- or output too many diagnostics for
mypy-primer to handle.)
We will now run CI on 109 projects; 34 are left in `bad.txt`.
## Test Plan
CI on this PR!
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Takes the `good.txt` changes from #17474, and removes the following
projects:
- arrow (not part of mypy_primer upstream)
- freqtrade, hydpy, ibis, pandera, xarray (saw panics locally, all
related to try_metaclass cycles)
Increases the mypy_primer CI run time to ~4 min.
## Test Plan
Three successful CI runs.
## Summary
* Add initial support for `typing.dataclass_transform`
* Support decorating a function decorator with `@dataclass_transform(…)`
(used by `attrs`, `strawberry`)
* Support decorating a metaclass with `@dataclass_transform(…)` (used by
`pydantic`, but doesn't work yet, because we don't seem to model
`__new__` calls correctly?)
* *No* support yet for decorating base classes with
`@dataclass_transform(…)`. I haven't figured out how this even supposed
to work. And haven't seen it being used.
* Add `strawberry` as an ecosystem project, as it makes heavy use of
`@dataclass_transform`
## Test Plan
New Markdown tests
This is an implementation of the discussion from #16719.
This change will allow list function calls to be replaced with
comprehensions:
```python
result = list()
for i in range(3):
result.append(i + 1)
# becomes
result = [i + 1 for i in range(3)]
```
I added a new test to `PERF401.py` to verify that this fix will now work
for `list()`.
## Summary
This PR is a follow-up to #16852.
Instance variables bound in comprehensions are recorded, allowing type
inference to work correctly.
This required adding support for unpacking in comprehension which
resolves https://github.com/astral-sh/ruff/issues/15369.
## Test Plan
One TODO in `mdtest/attributes.md` is now resolved, and some new test
cases are added.
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
## Summary
If two types are gradually-equivalent, that means they share the same
set of possible materializations. There's no need to keep two such types
in the same union or intersection; we should simplify them.
Fixes https://github.com/astral-sh/ruff/issues/17465
The one downside here is that now we will simplify e.g. `Unknown |
Todo(...)` to just `Unknown`, if `Unknown` was added to the union first.
This is correct from a type perspective (they are equivalent types), but
it can mean we lose visibility into part of the cause for the type
inferring as unknown. I think this is OK, but if we think it's important
to avoid this, I can add a special case to try to preserve `Todo` over
`Unknown`, if we see them both in the same union or intersection.
## Test Plan
Added and updated mdtests.
## Summary
The long line of projects in `mypy_primer.yaml` is hard to work with
when adding projects or checking whether they are currently run. Use a
one-per-line text file instead.
## Test Plan
Ecosystem check on this PR.
