Commit graph

862 commits

Author SHA1 Message Date
Douglas Creager
7c976dc570 Merge branch 'main' into dcreager/function-enum
* main:
  Update pre-commit dependencies (#17506)
  [red-knot] Simplify visibility constraint handling for `*`-import definitions (#17486)
  [red-knot] Detect (some) invalid protocols (#17488)
  [red-knot] Correctly identify protocol classes (#17487)
  Update dependency ruff to v0.11.6 (#17516)
  Update Rust crate shellexpand to v3.1.1 (#17512)
  Update Rust crate proc-macro2 to v1.0.95 (#17510)
  Update Rust crate rand to v0.9.1 (#17511)
  Update Rust crate libc to v0.2.172 (#17509)
  Update Rust crate jiff to v0.2.9 (#17508)
  Update Rust crate clap to v4.5.37 (#17507)
  Update astral-sh/setup-uv action to v5.4.2 (#17504)
  Update taiki-e/install-action digest to 09dc018 (#17503)
  [red-knot] infer attribute assignments bound in comprehensions (#17396)
  [red-knot] simplify gradually-equivalent types out of unions and intersections (#17467)
  [red-knot] pull primer projects to run from file (#17473)
2025-04-21 13:18:36 -04:00
Douglas Creager
b44fb47f25 create generic context lazily 2025-04-21 13:15:09 -04:00
Alex Waygood
be54b840e9
[red-knot] Simplify visibility constraint handling for *-import definitions (#17486) 2025-04-21 15:33:35 +00:00
Alex Waygood
45b5dedee2
[red-knot] Detect (some) invalid protocols (#17488) 2025-04-21 16:24:19 +01:00
Alex Waygood
9ff4772a2c
[red-knot] Correctly identify protocol classes (#17487) 2025-04-21 16:17:06 +01:00
Shunsuke Shibayama
da6b68cb58
[red-knot] infer attribute assignments bound in comprehensions (#17396)
Some checks failed
CI / Determine changes (push) Has been cancelled
CI / cargo fmt (push) Has been cancelled
CI / cargo build (release) (push) Has been cancelled
CI / python package (push) Has been cancelled
CI / pre-commit (push) Has been cancelled
CI / mkdocs (push) Has been cancelled
[Knot Playground] Release / publish (push) Has been cancelled
CI / cargo clippy (push) Has been cancelled
CI / cargo test (linux) (push) Has been cancelled
CI / cargo test (linux, release) (push) Has been cancelled
CI / cargo test (windows) (push) Has been cancelled
CI / cargo test (wasm) (push) Has been cancelled
CI / cargo build (msrv) (push) Has been cancelled
CI / cargo fuzz build (push) Has been cancelled
CI / fuzz parser (push) Has been cancelled
CI / test scripts (push) Has been cancelled
CI / ecosystem (push) Has been cancelled
CI / cargo shear (push) Has been cancelled
CI / formatter instabilities and black similarity (push) Has been cancelled
CI / test ruff-lsp (push) Has been cancelled
CI / check playground (push) Has been cancelled
CI / benchmarks (push) Has been cancelled
## Summary

This PR is a follow-up to #16852.

Instance variables bound in comprehensions are recorded, allowing type
inference to work correctly.

This required adding support for unpacking in comprehension which
resolves https://github.com/astral-sh/ruff/issues/15369.

## Test Plan

One TODO in `mdtest/attributes.md` is now resolved, and some new test
cases are added.

---------

Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
2025-04-19 06:42:48 +05:30
Carl Meyer
2a478ce1b2
[red-knot] simplify gradually-equivalent types out of unions and intersections (#17467)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

If two types are gradually-equivalent, that means they share the same
set of possible materializations. There's no need to keep two such types
in the same union or intersection; we should simplify them.

Fixes https://github.com/astral-sh/ruff/issues/17465

The one downside here is that now we will simplify e.g. `Unknown |
Todo(...)` to just `Unknown`, if `Unknown` was added to the union first.
This is correct from a type perspective (they are equivalent types), but
it can mean we lose visibility into part of the cause for the type
inferring as unknown. I think this is OK, but if we think it's important
to avoid this, I can add a special case to try to preserve `Todo` over
`Unknown`, if we see them both in the same union or intersection.

## Test Plan

Added and updated mdtests.
2025-04-18 15:08:57 -07:00
Carl Meyer
8fe2dd5e03
[red-knot] pull primer projects to run from file (#17473)
## Summary

The long line of projects in `mypy_primer.yaml` is hard to work with
when adding projects or checking whether they are currently run. Use a
one-per-line text file instead.

## Test Plan

Ecosystem check on this PR.
2025-04-18 21:20:18 +00:00
Douglas Creager
0a4dec0323 start pulling out enum 2025-04-18 17:04:26 -04:00
Alex Waygood
454ad15aee
[red-knot] Fix MRO inference for protocol classes; allow inheritance from subscripted Generic[]; forbid subclassing unsubscripted Generic (#17452) 2025-04-18 19:55:53 +00:00
Carl Meyer
27a315b740
[red-knot] add fixpoint iteration for Type::member_lookup_with_policy (#17464)
## Summary

Member lookup can be cyclic, with type inference of implicit members. A
sample case is shown in the added mdtest.

There's no clear way to handle such cases other than to fixpoint-iterate
the cycle.

Fixes #17457.

## Test Plan

Added test.
2025-04-18 10:20:03 -07:00
Douglas Creager
787bcd1c6a
[red-knot] Handle explicit class specialization in type expressions (#17434)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
You can now use subscript expressions in a type expression to explicitly
specialize generic classes, just like you could already do in value
expressions.

This still does not implement bidirectional checking, so a type
annotation on an assignment does not influence how we infer a
specialization for a (not explicitly specialized) constructor call. You
might get an `invalid-assignment` error if (a) we cannot infer a class
specialization from the constructor call (in which case you end up e.g.
trying to assign `C[Unknown]` to `C[int]`) or if (b) we can infer a
specialization, but it doesn't match the annotation.

Closes https://github.com/astral-sh/ruff/issues/17432
2025-04-18 11:49:22 -04:00
Matthew Mckee
5853eb28dd
[red-knot] allow assignment expression in call compare narrowing (#17461)
## Summary

There was some narrowing constraints not covered from the previous PR

```py
def _(x: object):
    if (type(y := x)) is bool:
        reveal_type(y)  # revealed: bool
```

Also, refactored a bit

## Test Plan

Update type_api.md
2025-04-18 08:46:15 -07:00
Carl Meyer
84d064a14c
[red-knot] fix building unions with literals and AlwaysTruthy/AlwaysFalsy (#17451)
In #17403 I added a comment asserting that all same-kind literal types
share all the same super-types. This is true, with two notable
exceptions: the types `AlwaysTruthy` and `AlwaysFalsy`. These two types
are super-types of some literal types within a given kind and not
others: `Literal[0]`, `Literal[""]`, and `Literal[b""]` inhabit
`AlwaysFalsy`, while other literals inhabit `AlwaysTruthy`.

This PR updates the literal-unions optimization to handle these types
correctly.

Fixes https://github.com/astral-sh/ruff/issues/17447

Verified locally that `QUICKCHECK_TESTS=100000 cargo test -p
red_knot_python_semantic -- --ignored types::property_tests::stable` now
passes again.
2025-04-18 08:20:03 -07:00
Carl Meyer
e4e405d2a1
[red-knot] Type narrowing for assertions (take 2) (#17345)
## Summary

Fixes #17147.

This was landed in #17149 and then reverted in #17335 because it caused
cycle panics in checking pybind11. #17456 fixed the cause of that panic.

## Test Plan

Add new narrow/assert.md test file

Co-authored-by: Matthew Mckee <matthewmckee04@yahoo.co.uk>
2025-04-18 08:11:07 -07:00
Carl Meyer
1918c61623
[red-knot] class bases are not affected by __future__.annotations (#17456)
## Summary

We were over-conflating the conditions for deferred name resolution.
`from __future__ import annotations` defers annotations, but not class
bases. In stub files, class bases are also deferred. Modeling this
correctly also reduces likelihood of cycles in Python files using `from
__future__ import annotations` (since deferred resolution is inherently
cycle-prone). The same cycles are still possible in `.pyi` files, but
much less likely, since typically there isn't anything in a `pyi` file
that would cause an early return from a scope, or otherwise cause
visibility constraints to persist to end of scope. Usually there is only
code at module global scope and class scope, which can't have `return`
statements, and `raise` or `assert` statements in a stub file would be
very strange. (Technically according to the spec we'd be within our
rights to just forbid a whole bunch of syntax outright in a stub file,
but I kinda like minimizing unnecessary differences between the handling
of Python files and stub files.)

## Test Plan

Added mdtests.
2025-04-18 06:46:21 -07:00
Dhruv Manilawala
44ad201262
[red-knot] Add support for overloaded functions (#17366)
Some checks are pending
CI / cargo build (msrv) (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

Part of #15383, this PR adds support for overloaded callables.

Typing spec: https://typing.python.org/en/latest/spec/overload.html

Specifically, it does the following:
1. Update the `FunctionType::signature` method to return signatures from
a possibly overloaded callable using a new `FunctionSignature` enum
2. Update `CallableType` to accommodate overloaded callable by updating
the inner type to `Box<[Signature]>`
3. Update the relation methods on `CallableType` with logic specific to
overloads
4. Update the display of callable type to display a list of signatures
enclosed by parenthesis
5. Update `CallableTypeOf` special form to recognize overloaded callable
6. Update subtyping, assignability and fully static check to account for
callables (equivalence is planned to be done as a follow-up)

For (2), it is required to be done in this PR because otherwise I'd need
to add some workaround for `into_callable_type` and I though it would be
best to include it in here.

For (2), another possible design would be convert `CallableType` in an
enum with two variants `CallableType::Single` and
`CallableType::Overload` but I decided to go with `Box<[Signature]>` for
now to (a) mirror it to be equivalent to `overload` field on
`CallableSignature` and (b) to avoid any refactor in this PR. This could
be done in a follow-up to better split the two kind of callables.

### Design

There were two main candidates on how to represent the overloaded
definition:
1. To include it in the existing infrastructure which is what this PR is
doing by recognizing all the signatures within the
`FunctionType::signature` method
2. To create a new `Overload` type variant

<details><summary>For context, this is what I had in mind with the new
type variant:</summary>
<p>

```rs
pub enum Type {
	FunctionLiteral(FunctionType),
    Overload(OverloadType),
    BoundMethod(BoundMethodType),
    ...
}

pub struct OverloadType {
	// FunctionLiteral or BoundMethod
    overloads: Box<[Type]>,
	// FunctionLiteral or BoundMethod
    implementation: Option<Type>
}

pub struct BoundMethodType {
    kind: BoundMethodKind,
    self_instance: Type,
}

pub enum BoundMethodKind {
    Function(FunctionType),
    Overload(OverloadType),
}
```

</p>
</details> 

The main reasons to choose (1) are the simplicity in the implementation,
reusing the existing infrastructure, avoiding any complications that the
new type variant has specifically around the different variants between
function and methods which would require the overload type to use `Type`
instead.

### Implementation

The core logic is how to collect all the overloaded functions. The way
this is done in this PR is by recording a **use** on the `Identifier`
node that represents the function name in the use-def map. This is then
used to fetch the previous symbol using the same name. This way the
signatures are going to be propagated from top to bottom (from first
overload to the final overload or the implementation) with each function
/ method. For example:

```py
from typing import overload

@overload
def foo(x: int) -> int: ...
@overload
def foo(x: str) -> str: ...
def foo(x: int | str) -> int | str:
	return x
```

Here, each definition of `foo` knows about all the signatures that comes
before itself. So, the first overload would only see itself, the second
would see the first and itself and so on until the implementation or the
final overload.

This approach required some updates specifically recognizing
`Identifier` node to record the function use because it doesn't use
`ExprName`.

## Test Plan

Update existing test cases which were limited by the overload support
and add test cases for the following cases:
* Valid overloads as functions, methods, generics, version specific
* Invalid overloads as stated in
https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions
(implementation will be done in a follow-up)
* Various relation: fully static, subtyping, and assignability (others
in a follow-up)

## Ecosystem changes

_WIP_

After going through the ecosystem changes (there are a lot!), here's
what I've found:

We need assignability check between a callable type and a class literal
because a lot of builtins are defined as classes in typeshed whose
constructor method is overloaded e.g., `map`, `sorted`, `list.sort`,
`max`, `min` with the `key` parameter, `collections.abc.defaultdict`,
etc. (https://github.com/astral-sh/ruff/issues/17343). This makes up
most of the ecosystem diff **roughly 70 diagnostics**. For example:

```py
from collections import defaultdict

# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(int)
# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(list)

class Foo:
    def __init__(self, x: int):
        self.x = x

# red-knot: No overload of function `__new__` matches arguments [lint:no-matching-overload]
map(Foo, ["a", "b", "c"])
```

Duplicate diagnostics in unpacking
(https://github.com/astral-sh/ruff/issues/16514) has **~16
diagnostics**.

Support for the `callable` builtin which requires `TypeIs` support. This
is **5 diagnostics**. For example:
```py
from typing import Any

def _(x: Any | None) -> None:
    if callable(x):
        # red-knot: `Any | None`
        # Pyright: `(...) -> object`
        # mypy: `Any`
        # pyrefly: `(...) -> object`
        reveal_type(x)
```

Narrowing on `assert` which has **11 diagnostics**. This is being worked
on in https://github.com/astral-sh/ruff/pull/17345. For example:
```py
import re

match = re.search("", "")
assert match
match.group()  # error: [possibly-unbound-attribute]
```

Others:
* `Self`: 2
* Type aliases: 6
* Generics: 3
* Protocols: 13
* Unpacking in comprehension: 1
(https://github.com/astral-sh/ruff/pull/17396)

## Performance

Refer to
https://github.com/astral-sh/ruff/pull/17366#issuecomment-2814053046.
2025-04-18 09:57:40 +05:30
Eric Mark Martin
de8f4e62e2
[red-knot] more type-narrowing in match statements (#17302)
## Summary

Add more narrowing analysis for match statements:
* add narrowing constraints from guard expressions
* add negated constraints from previous predicates and guards to
subsequent cases

This PR doesn't address that guards can mutate your subject, and so
theoretically invalidate some of these narrowing constraints that you've
previously accumulated. Some prior art on this issue [here][mutable
guards].

[mutable guards]:
https://www.irif.fr/~scherer/research/mutable-patterns/mutable-patterns-mlworkshop2024-abstract.pdf

## Test Plan

Add some new tests, and update some existing ones


---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2025-04-17 18:18:34 -07:00
Matthew Mckee
edfa03a692
[red-knot] Add some narrowing for assignment expressions (#17448)
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

Fixes #14866
Fixes #17437

## Test Plan

Update mdtests in `narrow/`
2025-04-17 17:28:06 -07:00
Alex Waygood
9965cee998
[red-knot] Understand typing.Protocol and typing_extensions.Protocol as equivalent (#17446)
Some checks are pending
CI / cargo fuzz build (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
2025-04-17 21:54:22 +01:00
Brent Westbrook
9c47b6dbb0
[red-knot] Detect version-related syntax errors (#16379)
## Summary
This PR extends version-related syntax error detection to red-knot. The
main changes here are:

1. Passing `ParseOptions` specifying a `PythonVersion` to parser calls
2. Adding a `python_version` method to the `Db` trait to make this
possible
3. Converting `UnsupportedSyntaxError`s to `Diagnostic`s
4. Updating existing mdtests  to avoid unrelated syntax errors

My initial draft of (1) and (2) in #16090 instead tried passing a
`PythonVersion` down to every parser call, but @MichaReiser suggested
the `Db` approach instead
[here](https://github.com/astral-sh/ruff/pull/16090#discussion_r1969198407),
and I think it turned out much nicer.

All of the new `python_version` methods look like this:

```rust
fn python_version(&self) -> ruff_python_ast::PythonVersion {
    Program::get(self).python_version(self)
}
```

with the exception of the `TestDb` in `ruff_db`, which hard-codes
`PythonVersion::latest()`.

## Test Plan

Existing mdtests, plus a new mdtest to see at least one of the new
diagnostics.
2025-04-17 14:00:30 -04:00
Alex Waygood
c36f3f5304
[red-knot] Add KnownFunction variants for is_protocol, get_protocol_members and runtime_checkable (#17450) 2025-04-17 14:49:52 +01:00
Alex Waygood
bd89838212
[red-knot] Initial tests for protocols (#17436) 2025-04-17 11:36:41 +00:00
David Peter
b32407b6f3
[red-knot] Dataclasses: synthesize __init__ with proper signature (#17428)
Some checks are pending
CI / test ruff-lsp (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

This changeset allows us to generate the signature of synthesized
`__init__` functions in dataclasses by analyzing the fields on the class
(and its superclasses). There are certain things that I have not yet
attempted to model in this PR, like `kw_only`,
[`dataclasses.KW_ONLY`](https://docs.python.org/3/library/dataclasses.html#dataclasses.KW_ONLY)
or functionality around
[`dataclasses.field`](https://docs.python.org/3/library/dataclasses.html#dataclasses.field).

ticket: https://github.com/astral-sh/ruff/issues/16651

## Ecosystem analysis

These two seem to depend on missing features in generics (see [relevant
code
here](9898ccbb78/tests/core/test_generics.py (L54))):

> ```diff
> + error[lint:unknown-argument]
/tmp/mypy_primer/projects/dacite/tests/core/test_generics.py:54:24:
Argument `x` does not match any known parameter
> + error[lint:unknown-argument]
/tmp/mypy_primer/projects/dacite/tests/core/test_generics.py:54:38:
Argument `y` does not match any known parameter
> ```



These two are true positives. See [relevant code
here](9898ccbb78/tests/core/test_config.py (L154-L161)).

> ```diff
> + error[lint:invalid-argument-type]
/tmp/mypy_primer/projects/dacite/tests/core/test_config.py:161:24:
Argument to this function is incorrect: Expected `int`, found
`Literal["test"]`
> + error[lint:invalid-argument-type]
/tmp/mypy_primer/projects/dacite/tests/core/test_config.py:172:24:
Argument to this function is incorrect: Expected `int | float`, found
`Literal["test"]`
> ```


This one depends on `**` unpacking of dictionaries, which we don't
support yet:

> ```diff
> + error[lint:missing-argument]
/tmp/mypy_primer/projects/mypy_primer/mypy_primer/globals.py:218:11: No
arguments provided for required parameters `new`, `old`, `repo`,
`type_checker`, `mypyc_compile_level`, `custom_typeshed_repo`,
`new_typeshed`, `old_typeshed`, `new_prepend_path`, `old_prepend_path`,
`additional_flags`, `project_selector`, `known_dependency_selector`,
`local_project`, `expected_success`, `project_date`, `shard_index`,
`num_shards`, `output`, `old_success`, `coverage`, `bisect`,
`bisect_output`, `validate_expected_success`,
`measure_project_runtimes`, `concurrency`, `base_dir`, `debug`, `clear`
> ```



## Test Plan

New Markdown tests.
2025-04-17 09:30:59 +02:00
David Peter
b4de245a5a
[red-knot] Dataclasses: support order=True (#17406)
## Summary

Support dataclasses with `order=True`:

```py
@dataclass(order=True)
class WithOrder:
    x: int

WithOrder(1) < WithOrder(2)  # no error
```

Also adds some additional tests to `dataclasses.md`.

ticket: #16651

## Test Plan

New Markdown tests
2025-04-17 08:58:46 +02:00
Douglas Creager
914095d08f
[red-knot] Super-basic generic inference at call sites (#17301)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / mkdocs (push) Waiting to run
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
This PR adds **_very_** basic inference of generic typevars at call
sites. It does not bring in a full unification algorithm, and there are
a few TODOs in the test suite that are not discharged by this. But it
handles a good number of useful cases! And the PR does not add anything
that would go away with a more sophisticated constraint solver.

In short, we just look for typevars in the formal parameters, and assume
that the inferred type of the corresponding argument is what that
typevar should map to. If a typevar appears more than once, we union
together the corresponding argument types.

Cases we are not yet handling:

- We are not widening literals.
- We are not recursing into parameters that are themselves generic
aliases.
- We are not being very clever with parameters that are union types.

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
2025-04-16 15:07:36 -04:00
Dhruv Manilawala
5350288d07
[red-knot] Check assignability of bound methods to callables (#17430)
## Summary

This is similar to https://github.com/astral-sh/ruff/pull/17095, it adds
assignability check for bound methods to callables.

## Test Plan

Add test cases to for assignability; specifically it uses gradual types
because otherwise it would just delegate to `is_subtype_of`.
2025-04-17 00:21:59 +05:30
cake-monotone
649610cc98
[red-knot] Support super (#17174)
## Summary

closes #16615 

This PR includes:

- Introduces a new type: `Type::BoundSuper`
- Implements member lookup for `Type::BoundSuper`, resolving attributes
by traversing the MRO starting from the specified class
- Adds support for inferring appropriate arguments (`pivot_class` and
`owner`) for `super()` when it is used without arguments

When `super(..)` appears in code, it can be inferred into one of the
following:

- `Type::Unknown`: when a runtime error would occur (e.g. calling
`super()` out of method scope, or when parameter validation inside
`super` fails)
- `KnownClass::Super::to_instance()`: when the result is an *unbound
super object* or when a dynamic type is used as parameters (MRO
traversing is meaningless)
- `Type::BoundSuper`: the common case, representing a properly
constructed `super` instance that is ready for MRO traversal and
attribute resolution

### Terminology

Python defines the terms *bound super object* and *unbound super
object*.

An **unbound super object** is created when `super` is called with only
one argument (e.g.
`super(A)`). This object may later be bound via the `super.__get__`
method. However, this form is rarely used in practice.

A **bound super object** is created either by calling
`super(pivot_class, owner)` or by using the implicit form `super()`,
where both arguments are inferred from the context. This is the most
common usage.

### Follow-ups

- Add diagnostics for `super()` calls that would result in runtime
errors (marked as TODO)
- Add property tests for `Type::BoundSuper`

## Test Plan

- Added `mdtest/class/super.md`

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2025-04-16 18:41:55 +00:00
Carl Meyer
b67590bfde
[red-knot] simplify union size limit handling (#17429) 2025-04-16 09:22:16 -07:00
Carl Meyer
c7b5067ef8
[red-knot] set a size limit on unions of literals (#17419)
## Summary

Until we optimize our full union/intersection representation to
efficiently handle large numbers of same-kind literal types "as a
block", set a fairly low limit on the size of unions of literals.

We will want to increase this limit once we've made the broader
efficiency improvement (tracked in
https://github.com/astral-sh/ruff/issues/17420).

## Test Plan

`cargo bench --bench red_knot`
2025-04-16 14:23:11 +00:00
Carl Meyer
a1f361949e
[red-knot] optimize building large unions of literals (#17403)
## Summary

Special-case literal types in `UnionBuilder` to speed up building large
unions of literals.

This optimization is extremely effective at speeding up building even a
very large union (it improves the large-unions benchmark by 41x!). The
problem we can run into is that it is easy to then run into another
operation on the very large union (for instance, narrowing may add it to
an intersection, which then distributes it over the intersection) which
is still slow.

I think it is possible to avoid this by extending this optimized
"grouped" representation throughout not just `UnionBuilder`, but all of
our union and intersection representations. I have some work in this
direction, but rather than spending more time on it right now, I'd
rather just land this much, along with a limit on the size of these
unions (to avoid building really big unions quickly and then hitting
issues where they are used.)

## Test Plan

Existing tests and benchmarks.

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2025-04-16 13:55:37 +00:00
Matthew Mckee
13ea4e5d0e
[red-knot] Fix comments in type_api.md (#17425) 2025-04-16 11:19:48 +00:00
Matthew Mckee
a2a7b1e268
[red-knot] Do not assume that x != 0 if x inhabits ~Literal[0] (#17370)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

Fixes incorrect negated type eq and ne assertions in
infer_binary_intersection_type_comparison

fixes #17360

## Test Plan

Remove and update some now incorrect tests
2025-04-15 22:27:27 -07:00
Douglas Creager
807a8a7a29
[red-knot] Acknowledge that T & anything is assignable to T (#17413)
Some checks are pending
CI / cargo clippy (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
This reworks the assignability/subtyping relations a bit to handle
typevars better:

1. For the most part, types are not assignable to typevars, since
there's no guarantee what type the typevar will be specialized to.

2. An intersection is an exception, if it contains the typevar itself as
one of the positive elements. This should fall out from the other
clauses automatically, since a typevar is assignable to itself, and an
intersection is assignable to something if any positive element is
assignable to that something.

3. Constrained typevars are an exception, since they must be specialized
to _exactly_ one of the constraints, not to a _subtype_ of a constraint.
If a type is assignable to every constraint, then the type is also
assignable to the constrained typevar.

We already had a special case for (3), but the ordering of it relative
to the intersection clauses meant we weren't catching (2) correctly. To
fix this, we keep the special case for (3), but fall through to the
other match arms for non-constrained typevars and if the special case
isn't true for a constrained typevar.

Closes https://github.com/astral-sh/ruff/issues/17364
2025-04-15 16:34:07 -04:00
Alex Waygood
312a487ea7
[red-knot] Add some knowledge of __all__ to *-import machinery (#17373) 2025-04-15 12:56:40 +01:00
Alex Waygood
79b921179c
[red-knot] Further optimize *-import visibility constraints (#17375) 2025-04-15 12:32:22 +01:00
David Peter
1f85e0d0a0
[red-knot] Minor 'member_lookup_with_policy' fix (#17407)
## Summary

Couldn't really think of a regression test, but it's probably better to
fix this if we ever add new member-lookup-policies.
2025-04-15 13:28:51 +02:00
David Peter
03adae80dc
[red-knot] Initial support for dataclasses (#17353)
## Summary

Add very early support for dataclasses. This is mostly to make sure that
we do not emit false positives on dataclass construction, but it also
lies some foundations for future extensions.

This seems like a good initial step to merge to me, as it basically
removes all false positives on dataclass constructor calls. This allows
us to use the ecosystem checks for making sure we don't introduce new
false positives as we continue to work on dataclasses.

## Ecosystem analysis

I re-ran the mypy_primer evaluation of [the `__init__`
PR](https://github.com/astral-sh/ruff/pull/16512) locally with our
current mypy_primer version and project selection. It introduced 1597
new diagnostics. Filtering those by searching for `__init__` and
rejecting those that contain `invalid-argument-type` (those could not
possibly be solved by this PR) leaves 1281 diagnostics. The current
version of this PR removes 1171 diagnostics, which leaves 110
unaccounted for. I extracted the lint + file path for all of these
diagnostics and generated a diff (of diffs), to see which
`__init__`-diagnostics remain. I looked at a subset of these: There are
a lot of `SomeClass(*args)` calls where we don't understand the
unpacking yet (this is not even related to `__init__`). Some others are
related to `NamedTuple`, which we also don't support yet. And then there
are some errors related to `@attrs.define`-decorated classes, which
would probably require support for `dataclass_transform`, which I made
no attempt to include in this PR.

## Test Plan

New Markdown tests.
2025-04-15 10:39:21 +02:00
Mike Perlov
3b24fe5c07
[red-knot] improve function/bound method type display (#17294)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / mkdocs (push) Waiting to run
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

* Partial #17238
* Flyby from discord discussion - `todo_type!` now statically checks for
no parens in the message to avoid issues between debug & release build
tests

## Test Plan

many mdtests are changing
2025-04-14 15:56:18 -07:00
Dhruv Manilawala
b5d529e976
[red-knot] Move relation methods from CallableType to Signature (#17365)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / mkdocs (push) Waiting to run
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

This PR moves all the relation methods from `CallableType` to
`Signature`.

The main reason for this is that `Signature` is going to be the common
denominator between normal and overloaded callables and the core logic
to check a certain relationship is going to just require the information
that would exists on `Signature`. For example, to check whether an
overloaded callable is a subtype of a normal callable, we need to check
whether _every_ overloaded signature is a subtype of the normal
callable's signature. This "every" logic would become part of the
`CallableType` and the core logic of checking the subtyping would exists
on `Signature`.
2025-04-15 03:32:25 +05:30
Carl Meyer
e2a38e4c00
[red-knot] optimize is_subtype_of for literals (#17394)
## Summary

Allows us to establish that two literals do not have a subtype
relationship with each other, without having to fallback to a typeshed
Instance type, which is comparatively slow.

Improves the performance of the many-string-literals union benchmark by
5x.

## Test Plan

`cargo test -p red_knot_python_semantic` and `cargo bench --bench
red_knot`.
2025-04-14 09:42:44 -07:00
David Peter
850360a0b4
[red-knot] Document limitations of diagnostics-silencing in unreachable code (#17387)
## Summary

Document the limitations of our current approach to silencing only a
subset of diagnostics in unreachable sections.
2025-04-14 12:55:14 +02:00
Shunsuke Shibayama
dfd8eaeb32
[red-knot] detect unreachable attribute assignments (#16852)
## Summary

This PR closes #15967.

Attribute assignments that are statically known to be unreachable are
excluded from consideration for implicit instance attribute type
inference. If none of the assignments are found to be reachable, an
`unresolved-attribute` error is reported.

## Test Plan

[A test
case](https://github.com/astral-sh/ruff/blob/main/crates/red_knot_python_semantic/resources/mdtest/attributes.md#attributes-defined-in-statically-known-to-be-false-branches)
marked as TODO now work as intended, and new test cases have been added.

---------

Co-authored-by: David Peter <mail@david-peter.de>
2025-04-14 09:23:20 +02:00
David Peter
f0c3abd198
[red-knot] Minor followup on str.startswith (#17362)
Some checks are pending
CI / mkdocs (push) Waiting to run
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

Minor follow-up to https://github.com/astral-sh/ruff/pull/17351, thank
you @AlexWaygood.
2025-04-11 18:40:08 +00:00
Andrew Gallant
7d11ef1564 red_knot_python_semantic: make TextRange required for reporting a lint diagnostic
This commit shuffles the reporting API around a little bit such that a
range is required, up front, when reporting a lint diagnostic. This in
turn enables us to make suppression checking eager.

In order to avoid callers needing to provide the range twice, we create
a primary annotation *without* a message inside the `Diagnostic`
encapsulated by the guard. We do this instead of requiring the message
up front because we're concerned about API complexity and the effort
involved in creating the message.

In order to provide a means of attaching a message to the primary
annotation, we expose a convenience API on `LintDiagnosticGuard` for
setting the message. This isn't generally possible for a `Diagnostic`,
but since a `LintDiagnosticGuard` knows how the `Diagnostic` was
constructed, we can offer this API correctly.

It strikes me that it might be easy to forget to attach a primary
annotation message, btu I think this the "least" bad failure mode. And
in particular, it should be somewhat obvious that it's missing once one
adds a snapshot test for how the diagnostic renders.

Otherwise, this API gives us the ability to eagerly check whether a
diagnostic should be reported with nearly minimal information. It also
shouldn't have any footguns since it guarantees that the primary
annotation is tied to the file in the typing context. And it keeps
things pretty simple: callers only need to provide what is actually
strictly necessary to make a diagnostic.
2025-04-11 12:36:36 -04:00
Andrew Gallant
a4d3c4bf8b red_knot_python_semantic: make the guards implement DerefMut
This is the payoff from removing a bit of indirection. The types still
exist, but now callers don't need to do builder -> reporter ->
diagnostic. They can just conceptually think of it as builder ->
diagnostic.
2025-04-11 12:36:36 -04:00
Andrew Gallant
f3cc287a40 red_knot_python_semantic: tweak naming
We're going to make the guards deref to `Diagnostic` in order to remove
a layer of indirection in the reporter API. (Well, technically the layer
is not removed since the types still exist, but in actual _usage_ the
layer will be removed. We'll see how it shakes out in the next commit.)
2025-04-11 12:36:36 -04:00
David Peter
47956db567
[red-knot] Specialize str.startswith for string literals (#17351)
## Summary

Infer precise Boolean literal types for `str.startswith` calls where the
instance and the prefix are both string literals. This allows us to
understand `sys.platform.startswith(…)` branches.

## Test Plan

New Markdown tests
2025-04-11 16:26:45 +02:00
David Peter
1a3b73720c
[red-knot] Silence errors in unreachable type annotations / class bases (#17342)
Some checks are pending
CI / test ruff-lsp (push) Blocked by required conditions
CI / cargo clippy (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

For silencing `invalid-type-form` diagnostics in unreachable code, we
use the same approach that we use before and check the reachability that
we already record.

For silencing `invalid-bases`, we simply check if the type of the base
is `Never`. If so, we silence the diagnostic with the argument that the
class construction would never happen.

## Test Plan

Updated Markdown tests.
2025-04-10 22:47:47 +02:00
David Peter
8b2727cf67
[red-knot] Silence unresolved-import in unreachable code (#17336)
Some checks are pending
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

Similar to what we did for `unresolved-reference` and
`unresolved-attribute`, we now also silence `unresolved-import`
diagnostics if the corresponding `import` statement is unreachable.

This addresses the (already closed) issue #17049.

## Test Plan

Adapted Markdown tests.
2025-04-10 21:13:28 +02:00