Commit graph

6236 commits

Author SHA1 Message Date
Dhruv Manilawala
08fa9b4a90
[red-knot] Add CallableTypeFromFunction special form (#16683)
## Summary

This PR adds a new `CallableTypeFromFunction` special form to allow
extracting the abstract signature of a function literal i.e., convert a
`Type::Function` into a `Type::Callable` (`CallableType::General`).

This is done to support testing the `is_gradual_equivalent_to` type
relation specifically the case we want to make sure that a function that
has parameters with no annotations and does not have a return type
annotation is gradual equivalent to `Callable[[Any, Any, ...], Any]`
where the number of parameters should match between the function literal
and callable type.

Refer
https://github.com/astral-sh/ruff/pull/16634#discussion_r1989976692

### Bikeshedding

The name `CallableTypeFromFunction` is a bit too verbose. A possibly
alternative from Carl is `CallableTypeOf` but that would be similar to
`TypeOf` albeit with a limitation that the former only accepts function
literal types and errors on other types.

Some other alternatives:
* `FunctionSignature`
* `SignatureOf` (similar issues as `TypeOf`?)
* ...

## Test Plan

Update `type_api.md` with a new section that tests this special form,
both invalid and valid forms.
2025-03-13 07:49:34 +05:30
David Peter
dd2313ab0f
[red-knot] Add mypy_primer usage documentation (#16679)
## Summary

Add documentation on how to run mypy_primer locally.
2025-03-12 16:47:10 +01:00
David Peter
083df0cf84
[red-knot] Support custom __getattr__ methods (#16668)
## Summary

Add support for calling custom `__getattr__` methods in case an
attribute is not otherwise found. This allows us to get rid of many
ecosystem false positives where we previously emitted errors when
accessing attributes on `argparse.Namespace`.

closes #16614

## Test Plan

* New Markdown tests
* Observed expected ecosystem changes (the changes for `arrow` also look
fine, since the `Arrow` class has a custom [`__getattr__`
here](1d70d00919/arrow/arrow.py (L802-L815)))
2025-03-12 13:44:11 +01:00
Carl Meyer
a176c1ac80
[red-knot] use fixpoint iteration for all cycles (#14029)
Pulls in the latest Salsa main branch, which supports fixpoint
iteration, and uses it to handle all query cycles.

With this, we no longer need to skip any corpus files to avoid panics.

Latest perf results show a 6% incremental and 1% cold-check regression.
This is not a "no cycles" regression, as tomllib and typeshed do trigger
some definition cycles (previously handled by our old
`infer_definition_types` fallback to `Unknown`). We don't currently have
a benchmark we can use to measure the pure no-cycles regression, though
I expect there would still be some regression; the fixpoint iteration
feature in Salsa does add some overhead even for non-cyclic queries.

I think this regression is within the reasonable range for this feature.
We can do further optimization work later, but I don't think it's the
top priority right now. So going ahead and acknowledging the regression
on CodSpeed.

Mypy primer is happy, so this doesn't regress anything on our
currently-checked projects. I expect it probably unlocks adding a number
of new projects to our ecosystem check that previously would have
panicked.

Fixes #13792
Fixes #14672
2025-03-12 12:41:40 +00:00
David Peter
a6572a57c4
[red-knot] Attribute access on intersection types (#16665)
## Summary

Implements attribute access on intersection types, which didn't
previously work. For example:

```py
from typing import Any

class P: ...
class Q: ...

class A:
    x: P = P()

class B:
    x: Any = Q()

def _(obj: A):
    if isinstance(obj, B):
        reveal_type(obj.x)  # revealed: P & Any
```

Refers to [this comment].

[this comment]:
https://github.com/astral-sh/ruff/pull/16416#discussion_r1985040363

## Test Plan

New Markdown tests
2025-03-12 13:20:17 +01:00
Joey Bar
b250304ad3
[red-knot] Improve is_disjoint for two intersections (#16636)
## Summary

Background - as a follow up to #16611 I noticed that there's a lot of
code duplicated between the `is_assignable_to` and `is_subtype_of`
functions and considered trying to merge them.

[A subtype and an assignable type are pretty much the
same](https://typing.python.org/en/latest/spec/concepts.html#the-assignable-to-or-consistent-subtyping-relation),
except that subtypes are by definition fully static, so I think we can
replace the whole of `is_subtype_of` with:

```
if !self.is_fully_static(db) || !target.is_fully_static(db) {
    return false;
}
return self.is_assignable_to(target)
```

if we move all of the logic to is_assignable_to and delete duplicate
code. Then we can discuss if it even makes sense to have a separate
is_subtype_of function (I think the answer is yes since it's used by a
bunch of other places, but we may be able to basically rip out the
concept).

Anyways while playing with combining the functions I noticed is that the
handling of Intersections in `is_subtype_of` has a special case for two
intersections, which I didn't include in the last PR - rather I first
handled right hand intersections before left hand, which should properly
handle double intersections (hand-wavy explanation I can justify if
needed - (A & B & C) is assignable to (A & B) because the left is
assignable to both A and B, but none of A, B, or C is assignable to (A &
B)).

I took a look at what breaks if I remove the handling for double
intersections, and the reason it is needed is because is_disjoint does
not properly handle intersections with negative conditions (so instead
`is_subtype_of` basically implements the check correctly).

This PR adds support to is_disjoint for properly checking negative
branches, which also lets us simplify `is_subtype_of`, bringing it in
line with `is_assignable_to`

## Test Plan

Added a bunch of tests, most of which failed before this fix

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2025-03-12 12:13:04 +00:00
David Peter
11b5cbcd2f
[red-knot] Restructure attribute-access and descriptor-protocol test suites. (#16664)
## Summary

This is a pure restructuring of the `attributes.md` and
`descriptor_protocol.md` test suites. They have grown organically and I
didn't want to make major structural changes in my recent PR to keep the
diff clean.
2025-03-12 09:52:21 +01:00
David Peter
3228545598
[red-knot] Minor optimization/cleanup in member lookup (#16663)
## Summary

A follow up to address [this comment]:

> Similarly here, it might be a little more performant to have a single
`Type::instance()` branch with an inner match over `class.known()`
rather than having multiple branches with `if class.is_known()` guards

[this comment]:
https://github.com/astral-sh/ruff/pull/16416#discussion_r1985159037
2025-03-12 09:11:05 +01:00
David Peter
860b95a318
[red-knot] Binary operator inference for union types (#16601)
## Summary

Properly handle binary operator inference for union types.

This fixes a bug I noticed while looking at ecosystem results. The MRE
version of it is this:

```py
def sub(x: float, y: float):
    # Red Knot: Operator `-` is unsupported between objects of type `int | float` and `int | float`
    return x - y
```

## Test Plan

- New Markdown tests.
- Expected diff in the ecosystem checks
2025-03-12 08:21:54 +01:00
Dhruv Manilawala
6de2b2873b
[red-knot] Check if callable type is fully static (#16633)
## Summary

Part of #15382 

This PR adds the check for whether a callable type is fully static or
not.

A callable type is fully static if all of the parameter types are fully
static _and_ the return type is fully static _and_ if it does not use
the gradual form (`...`) for its parameters.

## Test Plan

Update `is_fully_static.md` with callable types.

It seems that currently this test is grouped into either fully static or
not, I think it would be useful to split them up in groups like
callable, etc. I intentionally avoided that in this PR but I'll put up a
PR for an appropriate split.

Note: I've an explicit goal of updating the property tests with the new
callable types once all relations are implemented.
2025-03-12 12:13:22 +05:30
Dhruv Manilawala
6b84253679
[red-knot] Callable member lookup, meta type impl (#16618)
## Summary

This PR is a follow-up to https://github.com/astral-sh/ruff/pull/16493
that implements member lookup for the general callable type.

Based on the discussion around [member lookup
here](https://github.com/astral-sh/ruff/pull/16493#discussion_r1982041180)
and [`.to_meta_type()`
here](https://github.com/astral-sh/ruff/pull/16493#discussion_r1985104664).

## Test Plan

Add a new test cases.
2025-03-12 12:01:38 +05:30
Carl Meyer
0340e23395
[red-knot] remove redundant sentence in test (#16660)
Removes a redundant sentence I accidentally left in the test suite from
in #16540 (my mistake).
2025-03-12 04:20:31 +00:00
Shunsuke Shibayama
78b5f0b165
[red-knot] detect invalid return type (#16540)
## Summary

This PR closes #16248.

If the return type of the function isn't assignable to the one
specified, an `invalid-return-type` error occurs.
I thought it would be better to report this as a different kind of error
than the `invalid-assignment` error, so I defined this as a new error.

## Test Plan

All type inconsistencies in the test cases have been replaced with
appropriate ones.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2025-03-12 01:58:59 +00:00
Douglas Creager
e17cd350b6
[red-knot] Support multiple overloads when binding parameters at call sites (#16568)
This updates the `Signature` and `CallBinding` machinery to support
multiple overloads for a callable. This is currently only used for
`KnownFunction`s that we special-case in our type inference code. It
does **_not_** yet update the semantic index builder to handle
`@overload` decorators and construct a multi-signature `Overloads`
instance for real Python functions.

While I was here, I updated many of the `try_call` special cases to use
signatures (possibly overloaded ones now) and `bind_call` to check
parameter lists. We still need some of the mutator methods on
`OverloadBinding` for the special cases where we need to update return
types based on some Rust code.
2025-03-11 15:08:17 -04:00
Alex Waygood
c16237ddc0
[red-knot] Rework Type::to_instance() to return Option<Type> (#16428)
## Summary

This PR fixes https://github.com/astral-sh/ruff/issues/16302.

The PR reworks `Type::to_instance()` to return `Option<Type>` rather
than `Type`. This reflects more accurately the fact that some variants
cannot be "turned into an instance", since they _already_ represent
instances of some kind. On `main`, we silently fallback to `Unknown` for
these variants, but this implicit behaviour can be somewhat surprising
and lead to unexpected bugs.

Returning `Option<Type>` rather than `Type` means that each callsite has
to account for the possibility that the type might already represent an
instance, and decide what to do about it.
In general, I think this increases the robustness of the code. Working
on this PR revealed two latent bugs in the code:
- One which has already been fixed by
https://github.com/astral-sh/ruff/pull/16427
- One which is fixed as part of https://github.com/astral-sh/ruff/pull/16608

I added special handling to `KnownClass::to_instance()`: If we fail to find one of these classes and the `test` feature is
_not_ enabled, we log a warning to the terminal saying that we failed to
find the class in typeshed and that we will be falling back to
`Type::Unknown`. A cache is maintained so that we record all classes
that we have already logged a warning for; we only log a warning for
failing to lookup a `KnownClass` if we know that it's the first time
we're looking it up.

## Test Plan

- All existing tests pass
- I ran the property tests via `QUICKCHECK_TESTS=1000000 cargo test
--release -p red_knot_python_semantic -- --ignored
types::property_tests::stable`

I also manually checked that warnings are appropriately printed to the
terminal when `KnownClass::to_instance()` falls back to `Unknown` and
the `test` feature is not enabled. To do this, I applied this diff to
the PR branch:

<details>
<summary>Patch deleting `int` and `str` from buitins</summary>

```diff
diff --git a/crates/red_knot_vendored/vendor/typeshed/stdlib/builtins.pyi b/crates/red_knot_vendored/vendor/typeshed/stdlib/builtins.pyi
index 0a6dc57b0..86636a05b 100644
--- a/crates/red_knot_vendored/vendor/typeshed/stdlib/builtins.pyi
+++ b/crates/red_knot_vendored/vendor/typeshed/stdlib/builtins.pyi
@@ -228,111 +228,6 @@ _PositiveInteger: TypeAlias = Literal[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
 _NegativeInteger: TypeAlias = Literal[-1, -2, -3, -4, -5, -6, -7, -8, -9, -10, -11, -12, -13, -14, -15, -16, -17, -18, -19, -20]
 _LiteralInteger = _PositiveInteger | _NegativeInteger | Literal[0]  # noqa: Y026  # TODO: Use TypeAlias once mypy bugs are fixed
 
-class int:
-    @overload
-    def __new__(cls, x: ConvertibleToInt = ..., /) -> Self: ...
-    @overload
-    def __new__(cls, x: str | bytes | bytearray, /, base: SupportsIndex) -> Self: ...
-    def as_integer_ratio(self) -> tuple[int, Literal[1]]: ...
-    @property
-    def real(self) -> int: ...
-    @property
-    def imag(self) -> Literal[0]: ...
-    @property
-    def numerator(self) -> int: ...
-    @property
-    def denominator(self) -> Literal[1]: ...
-    def conjugate(self) -> int: ...
-    def bit_length(self) -> int: ...
-    if sys.version_info >= (3, 10):
-        def bit_count(self) -> int: ...
-
-    if sys.version_info >= (3, 11):
-        def to_bytes(
-            self, length: SupportsIndex = 1, byteorder: Literal["little", "big"] = "big", *, signed: bool = False
-        ) -> bytes: ...
-        @classmethod
-        def from_bytes(
-            cls,
-            bytes: Iterable[SupportsIndex] | SupportsBytes | ReadableBuffer,
-            byteorder: Literal["little", "big"] = "big",
-            *,
-            signed: bool = False,
-        ) -> Self: ...
-    else:
-        def to_bytes(self, length: SupportsIndex, byteorder: Literal["little", "big"], *, signed: bool = False) -> bytes: ...
-        @classmethod
-        def from_bytes(
-            cls,
-            bytes: Iterable[SupportsIndex] | SupportsBytes | ReadableBuffer,
-            byteorder: Literal["little", "big"],
-            *,
-            signed: bool = False,
-        ) -> Self: ...
-
-    if sys.version_info >= (3, 12):
-        def is_integer(self) -> Literal[True]: ...
-
-    def __add__(self, value: int, /) -> int: ...
-    def __sub__(self, value: int, /) -> int: ...
-    def __mul__(self, value: int, /) -> int: ...
-    def __floordiv__(self, value: int, /) -> int: ...
-    def __truediv__(self, value: int, /) -> float: ...
-    def __mod__(self, value: int, /) -> int: ...
-    def __divmod__(self, value: int, /) -> tuple[int, int]: ...
-    def __radd__(self, value: int, /) -> int: ...
-    def __rsub__(self, value: int, /) -> int: ...
-    def __rmul__(self, value: int, /) -> int: ...
-    def __rfloordiv__(self, value: int, /) -> int: ...
-    def __rtruediv__(self, value: int, /) -> float: ...
-    def __rmod__(self, value: int, /) -> int: ...
-    def __rdivmod__(self, value: int, /) -> tuple[int, int]: ...
-    @overload
-    def __pow__(self, x: Literal[0], /) -> Literal[1]: ...
-    @overload
-    def __pow__(self, value: Literal[0], mod: None, /) -> Literal[1]: ...
-    @overload
-    def __pow__(self, value: _PositiveInteger, mod: None = None, /) -> int: ...
-    @overload
-    def __pow__(self, value: _NegativeInteger, mod: None = None, /) -> float: ...
-    # positive __value -> int; negative __value -> float
-    # return type must be Any as `int | float` causes too many false-positive errors
-    @overload
-    def __pow__(self, value: int, mod: None = None, /) -> Any: ...
-    @overload
-    def __pow__(self, value: int, mod: int, /) -> int: ...
-    def __rpow__(self, value: int, mod: int | None = None, /) -> Any: ...
-    def __and__(self, value: int, /) -> int: ...
-    def __or__(self, value: int, /) -> int: ...
-    def __xor__(self, value: int, /) -> int: ...
-    def __lshift__(self, value: int, /) -> int: ...
-    def __rshift__(self, value: int, /) -> int: ...
-    def __rand__(self, value: int, /) -> int: ...
-    def __ror__(self, value: int, /) -> int: ...
-    def __rxor__(self, value: int, /) -> int: ...
-    def __rlshift__(self, value: int, /) -> int: ...
-    def __rrshift__(self, value: int, /) -> int: ...
-    def __neg__(self) -> int: ...
-    def __pos__(self) -> int: ...
-    def __invert__(self) -> int: ...
-    def __trunc__(self) -> int: ...
-    def __ceil__(self) -> int: ...
-    def __floor__(self) -> int: ...
-    def __round__(self, ndigits: SupportsIndex = ..., /) -> int: ...
-    def __getnewargs__(self) -> tuple[int]: ...
-    def __eq__(self, value: object, /) -> bool: ...
-    def __ne__(self, value: object, /) -> bool: ...
-    def __lt__(self, value: int, /) -> bool: ...
-    def __le__(self, value: int, /) -> bool: ...
-    def __gt__(self, value: int, /) -> bool: ...
-    def __ge__(self, value: int, /) -> bool: ...
-    def __float__(self) -> float: ...
-    def __int__(self) -> int: ...
-    def __abs__(self) -> int: ...
-    def __hash__(self) -> int: ...
-    def __bool__(self) -> bool: ...
-    def __index__(self) -> int: ...
-
 class float:
     def __new__(cls, x: ConvertibleToFloat = ..., /) -> Self: ...
     def as_integer_ratio(self) -> tuple[int, int]: ...
@@ -437,190 +332,6 @@ class _FormatMapMapping(Protocol):
 class _TranslateTable(Protocol):
     def __getitem__(self, key: int, /) -> str | int | None: ...
 
-class str(Sequence[str]):
-    @overload
-    def __new__(cls, object: object = ...) -> Self: ...
-    @overload
-    def __new__(cls, object: ReadableBuffer, encoding: str = ..., errors: str = ...) -> Self: ...
-    @overload
-    def capitalize(self: LiteralString) -> LiteralString: ...
-    @overload
-    def capitalize(self) -> str: ...  # type: ignore[misc]
-    @overload
-    def casefold(self: LiteralString) -> LiteralString: ...
-    @overload
-    def casefold(self) -> str: ...  # type: ignore[misc]
-    @overload
-    def center(self: LiteralString, width: SupportsIndex, fillchar: LiteralString = " ", /) -> LiteralString: ...
-    @overload
-    def center(self, width: SupportsIndex, fillchar: str = " ", /) -> str: ...  # type: ignore[misc]
-    def count(self, sub: str, start: SupportsIndex | None = ..., end: SupportsIndex | None = ..., /) -> int: ...
-    def encode(self, encoding: str = "utf-8", errors: str = "strict") -> bytes: ...
-    def endswith(
-        self, suffix: str | tuple[str, ...], start: SupportsIndex | None = ..., end: SupportsIndex | None = ..., /
-    ) -> bool: ...
-    @overload
-    def expandtabs(self: LiteralString, tabsize: SupportsIndex = 8) -> LiteralString: ...
-    @overload
-    def expandtabs(self, tabsize: SupportsIndex = 8) -> str: ...  # type: ignore[misc]
-    def find(self, sub: str, start: SupportsIndex | None = ..., end: SupportsIndex | None = ..., /) -> int: ...
-    @overload
-    def format(self: LiteralString, *args: LiteralString, **kwargs: LiteralString) -> LiteralString: ...
-    @overload
-    def format(self, *args: object, **kwargs: object) -> str: ...
-    def format_map(self, mapping: _FormatMapMapping, /) -> str: ...
-    def index(self, sub: str, start: SupportsIndex | None = ..., end: SupportsIndex | None = ..., /) -> int: ...
-    def isalnum(self) -> bool: ...
-    def isalpha(self) -> bool: ...
-    def isascii(self) -> bool: ...
-    def isdecimal(self) -> bool: ...
-    def isdigit(self) -> bool: ...
-    def isidentifier(self) -> bool: ...
-    def islower(self) -> bool: ...
-    def isnumeric(self) -> bool: ...
-    def isprintable(self) -> bool: ...
-    def isspace(self) -> bool: ...
-    def istitle(self) -> bool: ...
-    def isupper(self) -> bool: ...
-    @overload
-    def join(self: LiteralString, iterable: Iterable[LiteralString], /) -> LiteralString: ...
-    @overload
-    def join(self, iterable: Iterable[str], /) -> str: ...  # type: ignore[misc]
-    @overload
-    def ljust(self: LiteralString, width: SupportsIndex, fillchar: LiteralString = " ", /) -> LiteralString: ...
-    @overload
-    def ljust(self, width: SupportsIndex, fillchar: str = " ", /) -> str: ...  # type: ignore[misc]
-    @overload
-    def lower(self: LiteralString) -> LiteralString: ...
-    @overload
-    def lower(self) -> str: ...  # type: ignore[misc]
-    @overload
-    def lstrip(self: LiteralString, chars: LiteralString | None = None, /) -> LiteralString: ...
-    @overload
-    def lstrip(self, chars: str | None = None, /) -> str: ...  # type: ignore[misc]
-    @overload
-    def partition(self: LiteralString, sep: LiteralString, /) -> tuple[LiteralString, LiteralString, LiteralString]: ...
-    @overload
-    def partition(self, sep: str, /) -> tuple[str, str, str]: ...  # type: ignore[misc]
-    if sys.version_info >= (3, 13):
-        @overload
-        def replace(
-            self: LiteralString, old: LiteralString, new: LiteralString, /, count: SupportsIndex = -1
-        ) -> LiteralString: ...
-        @overload
-        def replace(self, old: str, new: str, /, count: SupportsIndex = -1) -> str: ...  # type: ignore[misc]
-    else:
-        @overload
-        def replace(
-            self: LiteralString, old: LiteralString, new: LiteralString, count: SupportsIndex = -1, /
-        ) -> LiteralString: ...
-        @overload
-        def replace(self, old: str, new: str, count: SupportsIndex = -1, /) -> str: ...  # type: ignore[misc]
-    if sys.version_info >= (3, 9):
-        @overload
-        def removeprefix(self: LiteralString, prefix: LiteralString, /) -> LiteralString: ...
-        @overload
-        def removeprefix(self, prefix: str, /) -> str: ...  # type: ignore[misc]
-        @overload
-        def removesuffix(self: LiteralString, suffix: LiteralString, /) -> LiteralString: ...
-        @overload
-        def removesuffix(self, suffix: str, /) -> str: ...  # type: ignore[misc]
-
-    def rfind(self, sub: str, start: SupportsIndex | None = ..., end: SupportsIndex | None = ..., /) -> int: ...
-    def rindex(self, sub: str, start: SupportsIndex | None = ..., end: SupportsIndex | None = ..., /) -> int: ...
-    @overload
-    def rjust(self: LiteralString, width: SupportsIndex, fillchar: LiteralString = " ", /) -> LiteralString: ...
-    @overload
-    def rjust(self, width: SupportsIndex, fillchar: str = " ", /) -> str: ...  # type: ignore[misc]
-    @overload
-    def rpartition(self: LiteralString, sep: LiteralString, /) -> tuple[LiteralString, LiteralString, LiteralString]: ...
-    @overload
-    def rpartition(self, sep: str, /) -> tuple[str, str, str]: ...  # type: ignore[misc]
-    @overload
-    def rsplit(self: LiteralString, sep: LiteralString | None = None, maxsplit: SupportsIndex = -1) -> list[LiteralString]: ...
-    @overload
-    def rsplit(self, sep: str | None = None, maxsplit: SupportsIndex = -1) -> list[str]: ...  # type: ignore[misc]
-    @overload
-    def rstrip(self: LiteralString, chars: LiteralString | None = None, /) -> LiteralString: ...
-    @overload
-    def rstrip(self, chars: str | None = None, /) -> str: ...  # type: ignore[misc]
-    @overload
-    def split(self: LiteralString, sep: LiteralString | None = None, maxsplit: SupportsIndex = -1) -> list[LiteralString]: ...
-    @overload
-    def split(self, sep: str | None = None, maxsplit: SupportsIndex = -1) -> list[str]: ...  # type: ignore[misc]
-    @overload
-    def splitlines(self: LiteralString, keepends: bool = False) -> list[LiteralString]: ...
-    @overload
-    def splitlines(self, keepends: bool = False) -> list[str]: ...  # type: ignore[misc]
-    def startswith(
-        self, prefix: str | tuple[str, ...], start: SupportsIndex | None = ..., end: SupportsIndex | None = ..., /
-    ) -> bool: ...
-    @overload
-    def strip(self: LiteralString, chars: LiteralString | None = None, /) -> LiteralString: ...
-    @overload
-    def strip(self, chars: str | None = None, /) -> str: ...  # type: ignore[misc]
-    @overload
-    def swapcase(self: LiteralString) -> LiteralString: ...
-    @overload
-    def swapcase(self) -> str: ...  # type: ignore[misc]
-    @overload
-    def title(self: LiteralString) -> LiteralString: ...
-    @overload
-    def title(self) -> str: ...  # type: ignore[misc]
-    def translate(self, table: _TranslateTable, /) -> str: ...
-    @overload
-    def upper(self: LiteralString) -> LiteralString: ...
-    @overload
-    def upper(self) -> str: ...  # type: ignore[misc]
-    @overload
-    def zfill(self: LiteralString, width: SupportsIndex, /) -> LiteralString: ...
-    @overload
-    def zfill(self, width: SupportsIndex, /) -> str: ...  # type: ignore[misc]
-    @staticmethod
-    @overload
-    def maketrans(x: dict[int, _T] | dict[str, _T] | dict[str | int, _T], /) -> dict[int, _T]: ...
-    @staticmethod
-    @overload
-    def maketrans(x: str, y: str, /) -> dict[int, int]: ...
-    @staticmethod
-    @overload
-    def maketrans(x: str, y: str, z: str, /) -> dict[int, int | None]: ...
-    @overload
-    def __add__(self: LiteralString, value: LiteralString, /) -> LiteralString: ...
-    @overload
-    def __add__(self, value: str, /) -> str: ...  # type: ignore[misc]
-    # Incompatible with Sequence.__contains__
-    def __contains__(self, key: str, /) -> bool: ...  # type: ignore[override]
-    def __eq__(self, value: object, /) -> bool: ...
-    def __ge__(self, value: str, /) -> bool: ...
-    @overload
-    def __getitem__(self: LiteralString, key: SupportsIndex | slice, /) -> LiteralString: ...
-    @overload
-    def __getitem__(self, key: SupportsIndex | slice, /) -> str: ...  # type: ignore[misc]
-    def __gt__(self, value: str, /) -> bool: ...
-    def __hash__(self) -> int: ...
-    @overload
-    def __iter__(self: LiteralString) -> Iterator[LiteralString]: ...
-    @overload
-    def __iter__(self) -> Iterator[str]: ...  # type: ignore[misc]
-    def __le__(self, value: str, /) -> bool: ...
-    def __len__(self) -> int: ...
-    def __lt__(self, value: str, /) -> bool: ...
-    @overload
-    def __mod__(self: LiteralString, value: LiteralString | tuple[LiteralString, ...], /) -> LiteralString: ...
-    @overload
-    def __mod__(self, value: Any, /) -> str: ...
-    @overload
-    def __mul__(self: LiteralString, value: SupportsIndex, /) -> LiteralString: ...
-    @overload
-    def __mul__(self, value: SupportsIndex, /) -> str: ...  # type: ignore[misc]
-    def __ne__(self, value: object, /) -> bool: ...
-    @overload
-    def __rmul__(self: LiteralString, value: SupportsIndex, /) -> LiteralString: ...
-    @overload
-    def __rmul__(self, value: SupportsIndex, /) -> str: ...  # type: ignore[misc]
-    def __getnewargs__(self) -> tuple[str]: ...
-
 class bytes(Sequence[int]):
```

</details>

And then ran red-knot on my
[typeshed-stats](https://github.com/AlexWaygood/typeshed-stats) project
using the command

```
cargo run -p red_knot -- check --project ../typeshed-stats --python-version="3.12" --verbose
```

I observed that the following logs were printed to the terminal, but
that each warning was only printed once (the desired behaviour):

```
INFO Python version: Python 3.12, platform: all
INFO Indexed 15 file(s)
INFO Could not find class `builtins.int` in typeshed on Python 3.12. Falling back to `Unknown` for the symbol instead.
INFO Could not find class `builtins.str` in typeshed on Python 3.12. Falling back to `Unknown` for the symbol instead.
```
2025-03-11 16:42:44 +00:00
Alex Waygood
989075dc16
[red-knot] Add tests asserting that KnownClass::to_instance() doesn't unexpectedly fallback to Type::Unknown with full typeshed stubs (#16608)
## Summary

One of the motivations in https://github.com/astral-sh/ruff/pull/16428
for panicking when the `test` or `debug_assertions` features are enabled
and a lookup of a `KnownClass` fails is that we've had some latent bugs
in our code where certain variants have been silently falling back to
`Unknown` in every typeshed lookup without us realising. But that in
itself isn't a great motivation for panicking in
`KnownClass::to_instance()`, since we can fairly easily add some tests
that assert that we don't unexpectedly fallback to `Unknown` for any
`KnownClass` variant. This PR adds those tests.

## Test Plan

`cargo test -p red_knot_python_semantic`
2025-03-11 16:12:44 +00:00
Joey Bar
e8e24310fb
[red-knot] Handle gradual intersection types in assignability (#16611)
## Summary

This mostly fixes #14899

My motivation was similar to the last comment by @sharkdp there. I ran
red_knot on a codebase and the most common error was patterns like this
failing:

```
def foo(x: str): ...

x: Any = ...
if isinstance(x, str):
    foo(x) # Object of type `Any & str` cannot be assigned to parameter 1 (`x`) of function `foo`; expected type `str`
```

The desired behavior is pretty much to ignore Any/Unknown when resolving
intersection assignability - `Any & str` should be assignable to `str`,
and `str` should be assignable to `str & Any`
 
The fix is actually very similar to the existing code in
`is_subtype_of`, we need to correctly handle intersections on either
side, while being careful to handle dynamic types as desired.

This does not fix the second test case from that issue:

```
static_assert(is_assignable_to(Intersection[Unrelated, Any], Not[tuple[Unrelated, Any]]))
```

but that's misleading because the root cause there has nothing to do
with gradual types. I added a simpler test case that also fails:

```
static_assert(is_assignable_to(Unrelated, Not[tuple[Unrelated]]))
```
This is because we don't determine that Unrelated does not subclass from
tuple so we can't rule out this relation. If that logic is improved then
this fix should also handle the case of the intersection

## Test Plan

Added a bunch of is_assignable_to tests, most of which failed before
this fix.
2025-03-11 07:58:56 -07:00
Dhruv Manilawala
da069aa00c
[red-knot] Infer lambda expression (#16547)
## Summary

Part of https://github.com/astral-sh/ruff/issues/15382

This PR adds support for inferring the `lambda` expression and return
the `CallableType`.

Currently, this is only limited to inferring the parameters and a todo
type for the return type.

For posterity, I tried using the `file_expression_type` to infer the
return type of lambda but it would always lead to cycle. The main reason
is that in `infer_parameter_definition`, the default expression is being
inferred using `file_expression_type`, which is correct, but it then

Take the following source code as an example:
```py
lambda x=1: x
```

Here's how the code will flow:
* `infer_scope_types` for the global scope
* `infer_lambda_expression`
* `infer_expression` for the default value `1`
* `file_expression_type` for the return type using the body expression.
This is because the body creates it's own scope
* `infer_scope_types` (lambda body scope)
* `infer_name_load` for the symbol `x` whose visible binding is the
lambda parameter `x`
* `infer_parameter_definition` for parameter `x`
* `file_expression_type` for the default value `1`
* `infer_scope_types` for the global scope because of the default
expression

This will then reach to `infer_definition` for the parameter `x` again
which then creates the cycle.

## Test Plan

Add tests around `lambda` expression inference.
2025-03-11 11:25:20 +05:30
David Peter
c60e8a037a
[red-knot] Add support for calling type[…] (#16597)
## Summary

This fixes the non-diagnostics part of #15948.

## Test Plan

New Markdown tests.

Negative diff on the ecosystem checks:

```diff
zipp (https://github.com/jaraco/zipp)
- error: lint:call-non-callable
-    --> /tmp/mypy_primer/projects/zipp/zipp/__init__.py:393:16
-     |
- 392 |     def _next(self, at):
- 393 |         return self.__class__(self.root, at)
-     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Object of type `type[Unknown]` is not callable
- 394 |
- 395 |     def is_dir(self):
-     |
- 
- Found 9 diagnostics
+ Found 8 diagnostics

arrow (https://github.com/arrow-py/arrow)
+     |
+     |
+ warning: lint:unused-ignore-comment
+    --> /tmp/mypy_primer/projects/arrow/arrow/arrow.py:576:66
+ 574 |                 values.append(1)
+ 575 |
+ 576 |             floor = self.__class__(*values, tzinfo=self.tzinfo)  # type: ignore[misc]
+     |                                                                  -------------------- Unused blanket `type: ignore` directive
+ 577 |
+ 578 |             if frame_absolute == "week":
- error: lint:call-non-callable
-     --> /tmp/mypy_primer/projects/arrow/arrow/arrow.py:1080:16
-      |
- 1078 |           dt = self._datetime.astimezone(tz)
- 1079 |
- 1080 |           return self.__class__(
-      |  ________________^
- 1081 | |             dt.year,
- 1082 | |             dt.month,
- 1083 | |             dt.day,
- 1084 | |             dt.hour,
- 1085 | |             dt.minute,
- 1086 | |             dt.second,
- 1087 | |             dt.microsecond,
- 1088 | |             dt.tzinfo,
- 1089 | |             fold=getattr(dt, "fold", 0),
- 1090 | |         )
-      | |_________^ Object of type `type[Unknown]` is not callable
- 1091 |
- 1092 |       # string output and formatting
-      |

black (https://github.com/psf/black)
- 
-     |
-     |
- error: lint:call-non-callable
-    --> /tmp/mypy_primer/projects/black/src/blib2to3/pgen2/grammar.py:135:15
- 133 |         Copy the grammar.
- 134 |         """
- 135 |         new = self.__class__()
-     |               ^^^^^^^^^^^^^^^^ Object of type `type[@Todo]` is not callable
- 136 |         for dict_attr in (
- 137 |             "symbol2number",
- Found 328 diagnostics
+ Found 327 diagnostics
```
2025-03-10 13:24:13 +01:00
David Peter
ca974706dd
[red-knot] Do not ignore typeshed stubs for 'venv' module (#16596)
## Summary

We currently fail to add the stubs for the `venv` stdlib module because
there is a `venv/` ignore pattern in the top-level `.gitignore` file.

## Test Plan

Ran the typeshed sync workflow manually once to see if the `venv/`
folder is now correctly added.
2025-03-10 09:07:48 +01:00
Alex Waygood
b6c7ba4f8e
[red-knot] Reduce Salsa lookups in Type::find_name_in_mro (#16582)
## Summary

Theoretically this should be slightly more performant, since the
`class.is_known()` calls each do a separate Salsa lookup, which we can
avoid if we do a single `match` on the value of `class.known()`. It also
ends up being two lines less code overall!

## Test Plan

`cargo test -p red_knot_python_semantic`
2025-03-10 07:55:22 +01:00
Alex Waygood
c970b794d0
Fix broken red-knot property tests (#16574)
## Summary

Fixes #16566, fixes #16575

The semantics of `Type::class_member` changed in
https://github.com/astral-sh/ruff/pull/16416, but the property-test
infrastructure was not updated. That means that the property tests were
panicking on the second `expect_type` call here:


0361021863/crates/red_knot_python_semantic/src/types/property_tests.rs (L151-L158)

With the somewhat unhelpful message:

```
Expected a (possibly unbound) type, not an unbound symbol
```

Applying this patch, and then running `QUICKCHECK_TESTS=1000000 cargo
test --release -p red_knot_python_semantic -- --ignored
types::property_tests::stable::equivalent_to_is_reflexive` showed
clearly that it was no longer able to find _any_ methods on _any_
classes due to the change in semantics of `Type::class_member`:

```diff
--- a/crates/red_knot_python_semantic/src/types/property_tests.rs
+++ b/crates/red_knot_python_semantic/src/types/property_tests.rs
@@ -27,7 +27,7 @@
 use std::sync::{Arc, Mutex, MutexGuard, OnceLock};
 
 use crate::db::tests::{setup_db, TestDb};
-use crate::symbol::{builtins_symbol, known_module_symbol};
+use crate::symbol::{builtins_symbol, known_module_symbol, Symbol};
 use crate::types::{
     BoundMethodType, CallableType, IntersectionBuilder, KnownClass, KnownInstanceType,
     SubclassOfType, TupleType, Type, UnionType,
@@ -150,10 +150,11 @@ impl Ty {
             Ty::BuiltinsFunction(name) => builtins_symbol(db, name).symbol.expect_type(),
             Ty::BuiltinsBoundMethod { class, method } => {
                 let builtins_class = builtins_symbol(db, class).symbol.expect_type();
-                let function = builtins_class
-                    .class_member(db, method.into())
-                    .symbol
-                    .expect_type();
+                let Symbol::Type(function, ..) =
+                    builtins_class.class_member(db, method.into()).symbol
+                else {
+                    panic!("no method `{method}` on class `{class}`");
+                };
 
                 create_bound_method(db, function, builtins_class)
             }
```

This PR updates the property-test infrastructure to use `Type::member`
rather than `Type::class_member`.

## Test Plan

- Ran `QUICKCHECK_TESTS=1000000 cargo test --release -p
red_knot_python_semantic -- --ignored types::property_tests::stable`
successfully
- Checked that there were no remaining uses of `Type::class_member` in
`property_tests.rs`
2025-03-09 17:40:08 +00:00
Alex Waygood
335b264fe2
[red-knot] Consistent spelling of "metaclass" and "meta-type" (#16576)
## Summary

Fixes a small nit of mine -- we are currently inconsistent in our
spelling between "metaclass" and "meta class", and between "meta type"
and "meta-type". This PR means that we consistently use "metaclass" and
"meta-type".

## Test Plan

`uvx pre-commit run -a`
2025-03-09 12:30:32 +00:00
Dhruv Manilawala
0361021863
[red-knot] Understand typing.Callable (#16493)
## Summary

Part of https://github.com/astral-sh/ruff/issues/15382

This PR implements a general callable type that wraps around a
`Signature` and it uses that new type to represent `typing.Callable`.

It also implements `Display` support for `Callable`. The format is as:
```
([<arg name>][: <arg type>][ = <default type>], ...) -> <return type>
```

The `/` and `*` separators are added at the correct boundary for
positional-only and keyword-only parameters. Now, as `typing.Callable`
only has positional-only parameters, the rendered signature would be:

```py
Callable[[int, str], None]
# (int, str, /) -> None
```

The `/` separator represents that all the arguments are positional-only.

The relationship methods that check assignability, subtype relationship,
etc. are not yet implemented and will be done so as a follow-up.

## Test Plan

Add test cases for display support for `Signature` and various mdtest
for `typing.Callable`.
2025-03-08 03:58:52 +00:00
Eric Mark Martin
24c8b1242e
[red-knot] Support unpacking with target (#16469)
## Summary

Resolves #16365

Add support for unpacking `with` statement targets.

## Test Plan

Added some test cases, alike the ones added by #15058.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2025-03-08 02:36:35 +00:00
David Peter
820a31af5d
[red-knot] Attribute access and the descriptor protocol (#16416)
## Summary

* Attributes/method are now properly looked up on metaclasses, when
called on class objects
* We properly distinguish between data descriptors and non-data
descriptors (but we do not yet support them in store-context, i.e.
`obj.data_descr = …`)
* The descriptor protocol is now implemented in a single unified place
for instances, classes and dunder-calls. Unions and possibly-unbound
symbols are supported in all possible stages of the process by creating
union types as results.
* In general, the handling of "possibly-unbound" symbols has been
improved in a lot of places: meta-class attributes, attributes,
descriptors with possibly-unbound `__get__` methods, instance
attributes, …
* We keep track of type qualifiers in a lot more places. I anticipate
that this will be useful if we import e.g. `Final` symbols from other
modules (see relevant change to typing spec:
https://github.com/python/typing/pull/1937).
* Detection and special-casing of the `typing.Protocol` special form in
order to avoid lots of changes in the test suite due to new `@Todo`
types when looking up attributes on builtin types which have `Protocol`
in their MRO. We previously
looked up attributes in a wrong way, which is why this didn't come up
before.

closes #16367
closes #15966

## Context

The way attribute lookup in `Type::member` worked before was simply
wrong (mostly my own fault). The whole instance-attribute lookup should
probably never have been integrated into `Type::member`. And the
`Type::static_member` function that I introduced in my last descriptor
PR was the wrong abstraction. It's kind of fascinating how far this
approach took us, but I am pretty confident that the new approach
proposed here is what we need to model this correctly.

There are three key pieces that are required to implement attribute
lookups:

- **`Type::class_member`**/**`Type::find_in_mro`**: The
`Type::find_in_mro` method that can look up attributes on class bodies
(and corresponding bases). This is a partial function on types, as it
can not be called on instance types like`Type::Instance(…)` or
`Type::IntLiteral(…)`. For this reason, we usually call it through
`Type::class_member`, which is essentially just
`type.to_meta_type().find_in_mro(…)` plus union/intersection handling.
- **`Type::instance_member`**: This new function is basically the
type-level equivalent to `obj.__dict__[name]` when called on
`Type::Instance(…)`. We use this to discover instance attributes such as
those that we see as declarations on class bodies or as (annotated)
assignments to `self.attr` in methods of a class.
- The implementation of the descriptor protocol. It works slightly
different for instances and for class objects, but it can be described
by the general framework:
- Call `type.class_member("attribute")` to look up "attribute" in the
MRO of the meta type of `type`. Call the resulting `Symbol` `meta_attr`
(even if it's unbound).
- Use `meta_attr.class_member("__get__")` to look up `__get__` on the
*meta type* of `meta_attr`. Call it with `__get__(meta_attr, self,
self.to_meta_type())`. If this fails (either the lookup or the call),
just proceed with `meta_attr`. Otherwise, replace `meta_attr` in the
following with the return type of `__get__`. In this step, we also probe
if a `__set__` or `__delete__` method exists and store it in
`meta_attr_kind` (can be either "data descriptor" or "normal attribute
or non-data descriptor").
  - Compute a `fallback` type.
    - For instances, we use `self.instance_member("attribute")`
- For class objects, we use `class_attr =
self.find_in_mro("attribute")`, and then try to invoke the descriptor
protocol on `class_attr`, i.e. we look up `__get__` on the meta type of
`class_attr` and call it with `__get__(class_attr, None, self)`. This
additional invocation of the descriptor protocol on the fallback type is
one major asymmetry in the otherwise universal descriptor protocol
implementation.
- Finally, we look at `meta_attr`, `meta_attr_kind` and `fallback`, and
handle various cases of (possible) unboundness of these symbols.
- If `meta_attr` is bound and a data descriptor, just return `meta_attr`
- If `meta_attr` is not a data descriptor, and `fallback` is bound, just
return `fallback`
- If `meta_attr` is not a data descriptor, and `fallback` is unbound,
return `meta_attr`
- Return unions of these three possibilities for partially-bound
symbols.

This allows us to handle class objects and instances within the same
framework. There is a minor additional detail where for instances, we do
not allow the fallback type (the instance attribute) to completely
shadow the non-data descriptor. We do this because we (currently) don't
want to pretend that we can statically infer that an instance attribute
is always set.

Dunder method calls can also be embedded into this framework. The only
thing that changes is that *there is no fallback type*. If a dunder
method is called on an instance, we do not fall back to instance
variables. If a dunder method is called on a class object, we only look
it up on the meta class, never on the class itself.

## Test Plan

New Markdown tests.
2025-03-07 22:03:28 +01:00
InSync
a18d8bfa7d
[pep8-naming] Add links to ignore-names options in various rules' documentation (#16557)
## Summary

Resolves #16551.

All rules using
[`lint.pep8-naming.ignore-names`](https://docs.astral.sh/ruff/settings/#lint_pep8-naming_ignore-names)
and
[`lint.pep8-naming.extend-ignore-names`](https://docs.astral.sh/ruff/settings/#lint_pep8-naming_extend-ignore-names)
now have their documentation linked to these two options.

## Test Plan

None.
2025-03-07 14:49:08 -05:00
Shunsuke Shibayama
348c196cb3
[red-knot] avoid inferring types if unpacking fails (#16530)
## Summary

This PR closes #15199.

The change I just made is to set all variables to type `Unknown` if
unpacking fails, but in some cases this may be excessive.
For example:

```py
a, b, c = "ab"
reveal_type(a)  # Unknown, but it would be reasonable to think of it as LiteralString
reveal_type(c)  # Unknown
```

```py
# Failed to unpack before the starred expression
(a, b, *c, d, e) = (1,)
reveal_type(a)  # Unknown
reveal_type(b)  # Unknown
...
# Failed to unpack after the starred expression
(a, b, *c, d, e) = (1, 2, 3)
reveal_type(a)  # Unknown, but should it be Literal[1]?
reveal_type(b)  # Unknown, but should it be Literal[2]?
reveal_type(c)  # Todo
reveal_type(d)  # Unknown
reveal_type(e)  # Unknown
```

I will modify it if you think it would be better to make it a different
type than just `Unknown`.

## Test Plan

I have made appropriate modifications to the test cases affected by this
change, and also added some more test cases.
2025-03-07 11:04:44 -08:00
Vasco Schiavo
6d6e524b90
[flake8-bandit] Fix mixed-case hash algorithm names (S324) (#16552)
The PR solves issue #16525
2025-03-07 15:21:07 +00:00
Dylan
0dfa810e9a
Bump 0.9.10 (#16556) 2025-03-07 09:00:08 -06:00
Micha Reiser
9cd0cdefd3
Assert that formatted code doesn't introduce any new unsupported syntax errors (#16549)
## Summary

This should give us better coverage for the unsupported syntax error
features and
increases our confidence that the formatter doesn't accidentially
introduce new unsupported
syntax errors. 

A feature like this would have been very useful when working on f-string
formatting
where it took a lot of iteration to find all Python 3.11 or older
incompatibilities.

## Test Plan

I applied my changes on top of
https://github.com/astral-sh/ruff/pull/16523 and
removed the target version check in the with-statement formatting code.
As expected,
the integration tests now failed
2025-03-07 09:12:00 +01:00
Eric Mark Martin
05a4c29344
print MDTEST_TEST_FILTER value in single-quotes (and escaped) (#16548)
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

<!-- What's the purpose of the change? What does it do, and why? -->

If an mdtest fails, the error output will include an example command
that can be run to re-run just the failing test, e.g

```
To rerun this specific test, set the environment variable: MDTEST_TEST_FILTER="sync.md - With statements - Context manager with non-callable `__exit__` attribute"
MDTEST_TEST_FILTER="sync.md - With statements - Context manager with non-callable `__exit__` attribute" cargo test -p red_knot_python_semantic --test mdtest -- mdtest__with_sync
```
This is very helpful, but because we're printing the envvar value
surrounded in double-quotes, the bits between backticks in this example
get interpreted as a shell interpolation. When running this in zsh, for
example, I see

```console
❯ MDTEST_TEST_FILTER="sync.md - With statements - Context manager with non-callable `__exit__` attribute" cargo test -p red_knot_python_semantic --test mdtest -- mdtest__with_sync  
zsh: command not found: __exit__
   Compiling red_knot_python_semantic v0.0.0 (/home/ericmarkmartin/Development/ruff/crates/red_knot_python_semantic)
   Compiling red_knot_test v0.0.0 (/home/ericmarkmartin/Development/ruff/crates/red_knot_test)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 6.09s
     Running tests/mdtest.rs (target/debug/deps/mdtest-149b8f9d937e36bc)

running 1 test
test mdtest__with_sync ... ok
```
[^1]

This is a minor annoyance which we can solve by using single-quotes
instead of double-quotes for this string. To do so safely, we also
escape single-quotes possibly contained within the string.

There is a [shell-quote](https://github.com/allenap/shell-quote) crate,
which seems to handle all this escaping stuff for you but fixing this
issue perfectly isn't a big deal (if there are more things to escape we
can deal with it then), so adding a new dependency (even a dev one)
seemed overkill.

[^1]: The filter does still work---it turns out that the filter
`MDTEST_TEST_FILTER="sync.md - With statements - Context manager with
non-callable attribute"` (what you get after the failed interpolation)
is still good enough

## Test Plan
<!-- How was it tested? -->

I broke the ``## Context manager with non-callable `__exit__`
attribute`` test by deleting the error assertion, then successfully ran
the new command it printed out.
2025-03-07 09:04:52 +01:00
Brent Westbrook
b3c884f4f3
[syntax-errors] Parenthesized keyword argument names after Python 3.8 (#16482)
Summary
--

Unlike the other syntax errors detected so far, parenthesized keyword
arguments are only allowed *before* 3.8. It sounds like they were only
accidentally allowed before that [^1].

As an aside, you get a pretty confusing error from Python for this, so
it's nice that we can catch it:

```pycon
>>> def f(**kwargs): ...
... f((a)=1)
...
  File "<python-input-0>", line 2
    f((a)=1)
       ^^^
SyntaxError: expression cannot contain assignment, perhaps you meant "=="?
>>>
```
Test Plan
--
Inline tests.

[^1]: https://github.com/python/cpython/issues/78822
2025-03-06 12:18:13 -05:00
Brent Westbrook
6c14225c66
[syntax-errors] Tuple unpacking in return and yield before Python 3.8 (#16485)
Summary
--

Checks for tuple unpacking in `return` and `yield` statements before
Python 3.8, as described [here].

Test Plan
--
Inline tests.

[here]: https://github.com/python/cpython/issues/76298
2025-03-06 11:57:20 -05:00
David Peter
0a627ef216
[red-knot] Never is callable and iterable. Arbitrary attributes can be accessed. (#16533)
## Summary

- `Never` is callable
- `Never` is iterable
- Arbitrary attributes can be accessed on `Never`

Split out from #16416 that is going to be required.

## Test Plan

Tests for all properties above.
2025-03-06 15:59:19 +00:00
Micha Reiser
a25be4610a
Clarify that D417 only checks docstrings with an arguments section (#16494)
## Summary

This came up in https://github.com/astral-sh/ruff/issues/16477

It's not obvious from the D417 rule's documentation that it only checks
docstrings
with an arguments section. Functions without such a section aren't
checked.

This PR tries to make this clearer in the documentation.
2025-03-06 09:49:35 +00:00
Micha Reiser
ce0018c3cb
Add OsSystem support to mdtests (#16518)
## Summary

This PR introduces a new mdtest option `system` that can either be
`in-memory` or `os`
where `in-memory` is the default.

The motivation for supporting `os` is so that we can write OS/system
specific tests
with mdtests. Specifically, I want to write mdtests for the module
resolver,
testing that module resolution is case sensitive. 

## Test Plan

I tested that the case-sensitive module resolver test start failing when
setting `system = "os"`
2025-03-06 10:41:40 +01:00
Micha Reiser
48f906e06c
Add tests for case-sensitive module resolution (#16517)
## Summary

Python's module resolver is case sensitive. 

This PR adds mdtests that assert that our module resolution is case
sensitive.

The tests currently all pass because our in memory file system is case
sensitive.
I'll add support for using the real file system to the mdtest framework
in a separate PR.

This PR also adds support for specifying extra search paths to the
mdtest framework.

## Test Plan
The tests fail when running them using the real file system.
2025-03-06 10:19:23 +01:00
Douglas Creager
ebd172e732
[red-knot] Several failing tests for generics (#16509)
To kick off the work of supporting generics, this adds many new
(currently failing) tests, showing the behavior we plan to support.

This is still missing a lot!  Not included:

- typevar tuples
- param specs
- variance
- `Self`

But it's a good start! We can add more failing tests for those once we
tackle these.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2025-03-05 17:21:19 -05:00
Carl Meyer
114abc7cfb
[red-knot] support empty TypeInference with fallback type (#16510)
This is split out of https://github.com/astral-sh/ruff/pull/14029, to
reduce the size of that PR, and to validate that this "fallback type"
support in `TypeInference` doesn't come with a performance cost. It also
improves the reliability and debuggability of our current (temporary)
cycle handling.

In order to recover from a cycle, we have to be able to construct a
"default" `TypeInference` where all expressions and definitions have
some "default" type. In our current cycle handling, this "default" type
is just unknown or a todo type. With fixpoint iteration, the "default"
type will be `Type::Never`, which is the "bottom" type that fixpoint
iteration starts from.

Since it would be costly (both in space and time) to actually enumerate
all expressions and definitions in a scope, just to insert the same
default type for all of them, instead we add an optional "missing type"
fallback to `TypeInference`, which (if set) is the fallback type for any
expression or definition which doesn't have an explicit type set.

With this change, cycles can no longer result in the dreaded "Missing
key" errors looking up the type of some expression.
2025-03-05 09:12:27 -08:00
Brent Westbrook
318f503714
[syntax-errors] Named expressions in decorators before Python 3.9 (#16386)
Summary
--

This PR detects the relaxed grammar for decorators proposed in [PEP
614](https://peps.python.org/pep-0614/) on Python 3.8 and lower.

The 3.8 grammar for decorators is
[here](https://docs.python.org/3.8/reference/compound_stmts.html#grammar-token-decorators):

```
decorators                ::=  decorator+
decorator                 ::=  "@" dotted_name ["(" [argument_list [","]] ")"] NEWLINE
dotted_name               ::=  identifier ("." identifier)*
```

in contrast to the current grammar
[here](https://docs.python.org/3/reference/compound_stmts.html#grammar-token-python-grammar-decorators)

```
decorators                ::= decorator+
decorator                 ::= "@" assignment_expression NEWLINE
assignment_expression ::= [identifier ":="] expression
```

Test Plan
--

New inline parser tests.
2025-03-05 17:08:18 +00:00
Brent Westbrook
d0623888b3
[syntax-errors] Positional-only parameters before Python 3.8 (#16481)
Summary
--

Detect positional-only parameters before Python 3.8, as marked by the
`/` separator in a parameter list.

Test Plan
--
Inline tests.
2025-03-05 13:46:43 +00:00
Shaygan Hooshyari
23fd4927ae
Auto generate ast expression nodes (#16285)
## Summary

Part of https://github.com/astral-sh/ruff/issues/15655

- Auto generate AST nodes using definitions in `ast.toml`. I added
attributes similar to
[`Field`](https://github.com/python/cpython/blob/main/Parser/asdl.py#L67)
in ASDL to hold field information

## Test Plan

Nothing outside the `ruff_python_ast` package should change.

---------

Co-authored-by: Douglas Creager <dcreager@dcreager.net>
2025-03-05 08:25:55 -05:00
Andrew Gallant
cc324abcc2 ruff_db: add new Diagnostic type
... with supporting types. This is meant to give us a base to work with
in terms of our new diagnostic data model. I expect the representations
to be tweaked over time, but I think this is a decent start.

I would also like to add doctest examples, but I think it's better if we
wait until an initial version of the renderer is done for that.
2025-03-05 08:23:02 -05:00
Andrew Gallant
80be0a0115 ruff_db: move ParseDiagnostic to old submodule too
This should have been with the previous two commits, but I missed it.
2025-03-05 08:23:02 -05:00
Andrew Gallant
b2e90c3f5c ruff_db: rename ParseDiagnostic to OldParseDiagnostic
I missed this in the previous commits.
2025-03-05 08:23:02 -05:00
Andrew Gallant
d7cbe6b7df ruff_db: move old types into their own sub-module
This puts them out of the way so that they can hopefully be removed more
easily in the (near) future, and so that they don't get in the way of
the new types. This also makes the intent of the migration a bit clearer
in the code and hopefully results in less confusion.
2025-03-05 08:23:02 -05:00
Andrew Gallant
021640a7a6 ruff_db: rename Diagnostic to OldDiagnosticTrait
This trait should eventually go away, so we rename it (and supporting
types) to make room for a new concrete `Diagnostic` type.

This commit is just the rename. In the next commit, we'll move it to a
different module.
2025-03-05 08:23:02 -05:00
Brent Westbrook
81bcdcebd3
[syntax-errors] Type parameter lists before Python 3.12 (#16479)
Summary
--

Another simple one, just detect type parameter lists in functions
and classes. Like pyright, we don't emit a second diagnostic for
`type` alias statements, which were also introduced in 3.12.

Test Plan
--
Inline tests.
2025-03-05 13:19:09 +00:00
Dhruv Manilawala
d94a78a134
[red-knot] De-duplicate symbol table query (#16515)
## Summary

This PR does a small refactor to avoid double
`symbol_table(...).symbol(...)` call to check for `__slots__` and
`TYPE_CHECKING`. It merges them into a single call.

I noticed this while looking at
https://github.com/astral-sh/ruff/pull/16468.
2025-03-05 07:36:21 +00:00