Commit graph

859 commits

Author SHA1 Message Date
Dhruv Manilawala
0360c6b219
[red-knot] Support calling a typing.Callable (#16888)
Some checks are pending
CI / cargo fmt (push) Waiting to run
CI / Determine changes (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

Part of #15382, this PR adds support for calling a variable that's
annotated with `typing.Callable`.

## Test Plan

Add test cases in a new `call/annotation.md` file.
2025-03-23 02:39:33 +05:30
Dhruv Manilawala
1cffb323bc
[red-knot] Check assignability for two callable types (#16845)
## Summary

Part of #15382

This PR adds support for checking the assignability of two general
callable types.

This is built on top of #16804 by including the gradual parameters check
and accepting a function that performs the check between the two types.

## Test Plan

Update `is_assignable_to.md` with callable types section.
2025-03-23 02:28:44 +05:30
Matthew Mckee
92028efe3d
[red-knot] Fix disambiguate display for union types (#16907)
Some checks are pending
CI / cargo clippy (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

When callables are displayed in unions, like:
```py
from typing import Callable


def foo(x: Callable[[], int] | None):
    # red-knot: Revealed type is `() -> int | None` [revealed-type]
    reveal_type(x)
```

This leaves the type rather ambiguous, to fix this we can add
parenthesis to callable type in union

Fixes #16893

## Test Plan

Update callable annotations tests

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2025-03-22 13:08:51 +01:00
Brent Westbrook
e4f5fe8cf7
[syntax-errors] Duplicate type parameter names (#16858)
Some checks are pending
CI / cargo fmt (push) Waiting to run
CI / Determine changes (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
Summary
--

Detects duplicate type parameter names in function definitions, class
definitions, and type alias statements.

I also boxed the `type_params` field on `StmtTypeAlias` to make it
easier to
`match` with functions and classes. (That's the reason for the red-knot
code
owner review requests, sorry!)

Test Plan
--

New `ruff_python_syntax_errors` unit tests.

Fixes #11119.
2025-03-21 15:06:22 -04:00
Alex Waygood
d21d639ee0
[red-knot] Avoid false-positive diagnostics on * import statements (#16899)
## Summary

This PR removes false-positive diagnostics for `*` imports. Currently we
always emit a diagnostic for these statements unless the module we're
importing from has a symbol named `"*"` in its symbol table for the
global scope. (And if we were doing everything correctly, no module ever
would have a symbol named `"*"` in its global scope!)

The fix here is sort-of hacky and won't be what we'll want to do
long-term. However, I think it's useful to do this as a first step
since:
- It significantly reduces false positives when running on code that
uses `*` imports
- It "resets" the tests to a cleaner state with many fewer TODOs, making
it easier to see what the hard work is that's still to be done.

## Test Plan

`cargo test -p red_knot_python_semantic`
2025-03-21 14:41:49 +00:00
Alex Waygood
14eb4cac88
[red-knot] Add failing tests for * imports (#16873)
## Summary

This PR adds a suite of tests for wildcard (`*`) imports. The tests
nearly all fail for now, and those that don't, ahem, pass for the wrong
reasons...

I've tried to add TODO comments in all instances for places where we are
currently inferring the incorrect thing, incorrectly emitting a
diagnostic, or emitting a diagnostic with a bad error message.

## Test Plan

`cargo test -p red_knot_python_semantic`
2025-03-21 14:17:15 +00:00
Douglas Creager
c03c28d199
[red-knot] Break up call binding into two phases (#16546)
This breaks up call binding into two phases:

- **_Matching parameters_** just looks at the names and kinds
(positional/keyword) of each formal and actual parameters, and matches
them up. Most of the current call binding errors happen during this
phase.

- Once we have matched up formal and actual parameters, we can **_infer
types_** of each actual parameter, and **_check_** that each one is
assignable to the corresponding formal parameter type.

As part of this, we add information to each formal parameter about
whether it is a type form or not. Once [PEP
747](https://peps.python.org/pep-0747/) is finalized, we can hook that
up to this internal type form representation. This replaces the
`ParameterExpectations` type, which did the same thing in a more ad hoc
way.

While we're here, we add a new fluent API for building `Parameter`s,
which makes our signature constructors a bit nicer to read. We also
eliminate a TODO where we were consuming types from the argument list
instead of the bound parameter list when evaluating our special-case
known functions.

Closes #15460

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2025-03-21 09:38:11 -04:00
Dhruv Manilawala
04a8756379
[red-knot] Check subtype relation between callable types (#16804)
## Summary

Part of #15382

This PR adds support for checking the subtype relationship between the
two callable types.

The main source of reference used for implementation is
https://typing.python.org/en/latest/spec/callables.html#assignability-rules-for-callables.

The implementation is split into two phases:
1. Check all the positional parameters which includes positional-only,
standard (positional or keyword) and variadic kind
2. Collect all the keywords in a `HashMap` to do the keyword parameters
check via name lookup

For (1), there's a helper struct which is similar to `.zip_longest`
(from `itertools`) except that it allows control over one of the
iterator as that's required when processing a variadic parameter. This
is required because positional parameters needs to be checked as per
their position between the two callable types. The struct also keeps
track of the current iteration element because when the loop is exited
(to move on to the phase 2) the current iteration element would be
carried over to the phase 2 check.

This struct is internal to the `is_subtype_of` method as I don't think
it makes sense to expose it outside. It also allows me to use "self" and
"other" suffixed field names as that's only relevant in that context.

## Test Plan

Add extensive tests in markdown.

Converted all of the code snippets from
https://typing.python.org/en/latest/spec/callables.html#assignability-rules-for-callables
to use `knot_extensions.is_subtype_of` and verified the result.
2025-03-21 03:27:22 +00:00
Dhruv Manilawala
193c38199e
[red-knot] Check whether two callable types are equivalent (#16698)
## Summary

This PR checks whether two callable types are equivalent or not.

This is required because for an equivalence relationship, the default
value does not necessarily need to be the same but if the parameter in
one of the callable has a default value then the corresponding parameter
in the other callable should also have a default value. This is the main
reason a manual implementation is required.

And, as per https://typing.python.org/en/latest/spec/callables.html#id4,
the default _type_ doesn't participate in a subtype relationship, only
the optionality (required or not) participates. This means that the
following two callable types are equivalent:

```py
def f1(a: int = 1) -> None: ...
def f2(a: int = 2) -> None: ...
```

Additionally, the name of positional-only, variadic and keyword-variadic
are not required to be the same for an equivalence relation.

A potential solution to avoid the manual implementation would be to only
store whether a parameter has a default value or not but the type is
currently required to check for assignability.

## Test plan

Add tests for callable types in `is_equivalent_to.md`
2025-03-21 03:19:07 +00:00
Matthew Mckee
63e78b41cd
[red-knot] Ban most Type::Instance types in type expressions (#16872)
Some checks are pending
CI / cargo clippy (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

Catch some Instances, but raise type error for the rest of them
Fixes #16851 

## Test Plan

Extend invalid.md in annotations

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2025-03-20 15:19:56 -07:00
Alex Waygood
296d67a496
Special-case value-expression inference of special form subscriptions (#16877)
## Summary

Currently for something like `X = typing.Tuple[str, str]`, we infer the
value of `X` as `object`. That's because `Tuple` (like many of the
symbols in the typing module) is annotated as a `_SpecialForm` instance
in typeshed's stubs:


23382f5f8c/crates/red_knot_vendored/vendor/typeshed/stdlib/typing.pyi (L215)

and we don't understand implicit type aliases yet, and the stub for
`_SpecialForm.__getitem__` says it always returns `object`:


23382f5f8c/crates/red_knot_vendored/vendor/typeshed/stdlib/typing.pyi (L198-L200)

We have existing false positives in our test suite due to this:


23382f5f8c/crates/red_knot_python_semantic/resources/mdtest/annotations/annotated.md (L76-L78)

and it's causing _many_ new false positives in #16872, which tries to
make our annotation-expression parsing stricter in some ways.

This PR therefore adds some small special casing for `KnownInstanceType`
variants that fallback to `_SpecialForm`, so that these false positives
can be avoided.

## Test Plan

Existing mdtest altered.

Cc. @MatthewMckee4
2025-03-20 21:46:02 +00:00
Shunsuke Shibayama
23382f5f8c
[red-knot] add test cases result in false positive errors (#16856)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

From #16641

The previous PR attempted to fix the errors presented in this PR, but as
discussed in the conversation, it was concluded that the approach was
undesirable and that further work would be needed to fix the errors with
a correct general solution.

In this PR, I instead add the test cases from the previous PR as TODOs,
as a starting point for future work.

## Test Plan

---------

Co-authored-by: Carl Meyer <carl@oddbird.net>
2025-03-20 17:17:54 +00:00
Matthew Mckee
cdafd8e32b
Allow discovery of venv in VIRTUAL_ENV env variable (#16853)
## Summary

Fixes #16744 

Allows the cli to find a virtual environment from the VIRTUAL_ENV
environment variable if no `--python` is set

## Test Plan

Manual testing, of:
- Virtual environments explicitly activated using `source .venv/bin/activate`
- Virtual environments implicilty activated via `uv run`
- Broken virtual environments with no `pyvenv.cfg` file
2025-03-20 13:55:35 +00:00
Matthew Mckee
4ed93b4311
Show more precise messages in invalid type expressions (#16850)
Some checks are pending
CI / cargo fmt (push) Waiting to run
CI / Determine changes (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[Knot Playground] Release / publish (push) Waiting to run
## Summary

Some error messages were not very specific; this PR improves them

## Test Plan

New mdtests added; existing mdtests tweaked
2025-03-19 17:00:30 +00:00
Alex Waygood
a3f3d734a1
[red-knot] Ban list literals in most contexts in type expressions (#16847)
## Summary

This PR reworks `TypeInferenceBuilder::infer_type_expression()` so that
we emit diagnostics when encountering a list literal in a type
expression. The only place where a list literal is allowed in a type
expression is if it appears as the first argument to `Callable[]`, and
`Callable` is already heavily special-cased in our type-expression
parsing.

In order to ensure that list literals are _always_ allowed as the
_first_ argument to `Callabler` (but never allowed as the second, third,
etc. argument), I had to do some refactoring of our type-expression
parsing for `Callable` annotations.

## Test Plan

New mdtests added, and existing ones updated
2025-03-19 14:42:42 +00:00
Matthew Mckee
3a5f1d46c0
[red-knot] Make' Type::in_type_expression()' exhaustive for Type::KnownInstance (#16836)
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

fixes #15048 
We want to handle more types from Type::KnownInstance 

## Test Plan

Add tests for each type added explicitly in the match

---------

Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
2025-03-19 07:36:28 -07:00
Alex Waygood
f3f3e55d97
[red-knot] Minor cleanup to infer_parameterized_known_instance_type_expression (#16846)
## Summary

These are just cosmetic changes, but I'm separating them out into a
standalone PR to make a branch I have stacked on top of this easier to
review

## Test Plan

Existing tests all pass
2025-03-19 14:19:13 +00:00
Dhruv Manilawala
a69f6240cc
[red-knot] Infer lambda return type as Unknown (#16695)
## Summary

Part of #15382

This PR infers the return type `lambda` expression as `Unknown`. In the
future, it would be more useful to infer the expression type considering
the surrounding context (#16696).

## Test Plan

Update existing test cases from `@todo` to the (verified) return type.
2025-03-18 22:48:10 +05:30
Dhruv Manilawala
c3d429ddd8
[red-knot] Move name field on parameter kind (#16830)
## Summary

Previously, the `name` field was on `Parameter` which required it to be
always optional regardless of the parameter kind because a
`typing.Callable` signature does not have name for the parameters. This
is the case for positional-only parameters. This wasn't enforced at the
type level which meant that downstream usages would have to unwrap on
`name` even though it's guaranteed to be present.

This commit moves the `name` field from `Parameter` to the
`ParameterKind` variants and makes it optional only for
`ParameterKind::PositionalOnly` variant while required for all other
variants.

One change that's now required is that a `Callable` form using a gradual
form for parameter types (`...`) would have a default `args` and
`kwargs` name used for variadic and keyword-variadic parameter kind
respectively. This is also the case for invalid `Callable` type forms. I
think this is fine as names are not relevant in this context but happy
to make it optional even in variadic variants.

## Test Plan

No new tests; make sure existing tests are passing.
2025-03-18 22:47:44 +05:30
Matthew Mckee
ab3ec4de6a
[red-knot] Emit errors for more AST nodes that are invalid (or only valid in specific contexts) in type expressions (#16822)
## Summary

Add error messages for invalid nodes in type expressions

Fixes #16816 

## Test Plan

Extend annotations/invalid.md to handle these invalid AST nodes error
messages
2025-03-18 17:16:50 +00:00
cake-monotone
4ab529803f
[red-knot] Refactor property_tests.rs into property_tests module structure (#16827)
## Summary

For now, `property_tests.rs` has grown larger and larger, making the
file difficult to read and maintain.

Although the code has been split, the test paths and full names remain
unchanged. There are no changes affecting test execution.
2025-03-18 12:59:14 +00:00
Alex Waygood
23b7df9b29
[red-knot] Simplify IterationError and ContextManagerError (#16820)
## Summary

This PR simplifies `IterationError` and `ContextManagerError` so that
they no longer "remember" what type it was that was (respectively) not
iterable or not valid as a context manager. Instead, the type that was
iterated over (or was used as a context manager) is passed back in when
calling the error struct's `report_diagnostic` method.

The motivations for this are:
- It significantly simplifies the code
- It reduces the size of these types on the stack

## Test Plan

`cargo test -p red_knot_python_semantic`
2025-03-18 11:30:41 +00:00
cake-monotone
3e2cf5d7c4
[red-knot] Improve property test performance by cloning db instead of holding MutexGuard (#16823)
Some checks are pending
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
## Summary

This PR brings an optimization.

- `get_cached_db` no longer returns a `MutexGuard`; instead, it returns
a cloned database.

### `get_cached_db`

Previously, the `MutexGuard` was held inside the property test function
(defined in the macro), which prevented multiple property tests from
running in parallel. More specifically, the program could only test one
random test case at a time, which likely caused a significant
bottleneck.

On my local machine, running:

```
QUICKCHECK_TESTS=100000 cargo test --release -p red_knot_python_semantic -- --ignored stable
```

showed about **a 75% speedup** (from \~60s to \~15s).
2025-03-18 09:09:57 +01:00
Andrew Gallant
bd9eab059f red_knot: update diagnostic output snapshots
These should all be minor cosmetic changes. To summarize:

* In many cases, `-` was replaced with `^` for primary annotations.
This is because, previously, whether `-` or `^` was used depended
on the severity. But in the new data model, it's based on whether
the annotation is "primary" or not. We could of course change this
in whatever way we want, but I think we should roll with this for now.

* The "secondary messages" in the old API are rendered as
sub-diagnostics. This in turn results in a small change in the output
format, since previously, the secondary messages were represented as
just another snippet. We use sub-diagnostics because that's the intended
way to enforce relative ordering between messages within a diagnostic.

* The "info:" prefix used in some annotation messages has been dropped.
We could re-add this, but I think I like it better without this prefix.

I believe those 3 cover all of the snapshot changes here.
2025-03-17 12:46:49 -04:00
Douglas Creager
23ccb52fa6
[red-knot] Handle unions of callables better (#16716)
This cleans up how we handle calling unions of types. #16568 adding a
three-level structure for callable signatures (`Signatures`,
`CallableSignature`, and `Signature`) to handle unions and overloads.

This PR updates the bindings side to mimic that structure. What used to
be called `CallOutcome` is now `Bindings`, and represents the result of
binding actual arguments against a possible union of callables.
`CallableBinding` is the result of binding a single, possibly
overloaded, callable type. `Binding` is the result of binding a single
overload.

While we're here, this also cleans up `CallError` greatly. It was
previously extracting error information from the bindings and storing it
in the error result. It is now a simple enum, carrying no data, that's
used as a status code to talk about whether the overall binding was
successful or not. We are now more consistent about walking the binding
itself to get detailed information about _how_ the binding was
unsucessful.

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
2025-03-17 10:35:52 -04:00
Dhruv Manilawala
3ccc8dbbf9
[red-knot] Fix fully static check for callable type (#16803)
## Summary

This PR fixes a bug in the check for fully static callable type where we
would skip unannotated parameter type.

## Test Plan

Add tests using the new `CallableTypeFromFunction` special form.
2025-03-17 20:01:30 +05:30
Alex Waygood
50b66dc025
[red-knot] Stabilize negation_reverses_subtype_order property test (#16801)
## Summary

This is a re-creation of https://github.com/astral-sh/ruff/pull/16764 by
@mtshiba, which I closed meaning to immediately reopen (GitHub wasn't
updating the PR with the latest pushed changes), and which GitHub will
not allow me to reopen for some reason. Pasting the summary from that PR
below:

> From https://github.com/astral-sh/ruff/pull/16641
> 
> As stated in this comment
(https://github.com/astral-sh/ruff/pull/16641#discussion_r1996153702),
the current ordering implementation for intersection types is incorrect.
So, I will introduce lexicographic ordering for intersection types.

## Test Plan

One property test stabilised (tested locally with
`QUICKCHECK_TESTS=2000000 cargo test --release -p
red_knot_python_semantic -- --ignored
types::property_tests::stable::negation_reverses_subtype_order`), and
existing mdtests that previously failed now pass.

Primarily-authored-by:
[mtshiba](https://github.com/astral-sh/ruff/commits?author=mtshiba)

---------

Co-authored-by: Shunsuke Shibayama <sbym1346@gmail.com>
2025-03-17 12:33:38 +00:00
Matthew Mckee
24707777af
[red-knot] Emit error if int/float/complex/bytes/boolean literals appear in type expressions outside typing.Literal[] (#16765)
## Summary
Fixes https://github.com/astral-sh/ruff/issues/16532

## Test Plan

New mdtest assertions added

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2025-03-17 11:56:16 +00:00
Micha Reiser
c100d519e9
[internal]: Upgrade salsa (#16794)
## Summary

Another salsa upgrade. 

The main motivation is to stay on a recent salsa version because there
are still a lot of breaking changes happening.
The most significant changes in this update:

* Salsa no longer derives `Debug` by default. It now requires
`interned(debug)` (or similar)
* This version ships the foundation for garbage collecting interned
values. However, this comes at the cost that queries now track which
interned values they created (or read). The micro benchmarks in the
salsa repo showed a significant perf regression. Will see if this also
visible in our benchmarks.

## Test Plan

`cargo test`
2025-03-17 11:05:54 +01:00
David Peter
ebcad6e641
[red-knot] Use try_call_dunder for augmented assignment (#16717)
## Summary

Uses the `try_call_dunder` infrastructure for augmented assignment and
fixes the logic to work for types other than `Type::Instance(…)`. This
allows us to infer the correct type here:
```py
x = (1, 2)
x += (3, 4)
reveal_type(x)  # revealed: tuple[Literal[1], Literal[2], Literal[3], Literal[4]]
```
Or in this (extremely weird) scenario:
```py
class Meta(type):
    def __iadd__(cls, other: int) -> str:
        return ""

class C(metaclass=Meta): ...

cls = C
cls += 1

reveal_type(cls)  # revealed: str
```

Union and intersection handling could also be improved here, but I made
no attempt to do so in this PR.

## Test Plan

New MD tests
2025-03-14 20:36:09 +01:00
David Peter
fe275725e0
[red-knot] Document current state of attribute assignment diagnostics (#16746)
## Summary

A follow-up to https://github.com/astral-sh/ruff/pull/16705 which
documents various kinds of diagnostics that can appear when assigning to
an attribute.

## Test Plan

New snapshot tests.
2025-03-14 20:34:43 +01:00
Micha Reiser
a467e7c8d3
[red-knot] Case sensitive module resolver (#16521)
## Summary

This PR implements the first part of
https://github.com/astral-sh/ruff/discussions/16440. It ensures that Red
Knot's module resolver is case sensitive on all systems.

This PR combines a few approaches:

1. It uses `canonicalize` on non-case-sensitive systems to get the real
casing of a path. This works for as long as no symlinks or mapped
network drives (the windows `E:\` is mapped to `\\server\share` thingy).
This is the same as what Pyright does
2. If 1. fails, fall back to recursively list the parent directory and
test if the path's file name matches the casing exactly as listed in by
list dir. This is the same approach as CPython takes in its module
resolver. The main downside is that it requires more syscalls because,
unlike CPython, we Red Knot needs to invalidate its caches if a file
name gets renamed (CPython assumes that the folders are immutable).

It's worth noting that the file watching test that I added that renames
`lib.py` to `Lib.py` currently doesn't pass on case-insensitive systems.
Making it pass requires some more involved changes to `Files`. I plan to
work on this next. There's the argument that landing this PR on its own
isn't worth it without this issue being addressed. I think it's still a
good step in the right direction even when some of the details on how
and where the path case sensitive comparison is implemented.

## Test plan

I added multiple integration tests (including a failing one). I tested
that the `case-sensitivity` detection works as expected on Windows,
MacOS and Linux and that the fast-paths are taken accordingly.
2025-03-14 19:16:44 +00:00
David Peter
a22d206db2
[red-knot] Preliminary tests for typing.Final (#15917)
## Summary

WIP.

Adds some preliminary tests for `typing.Final`.

## Test Plan

New MD tests
2025-03-14 12:30:13 +01:00
cake-monotone
270318c2e0
[red-knot] fix: improve type inference for binary ops on tuples (#16725)
## Summary

This PR includes minor improvements to binary operation inference,
specifically for tuple concatenation.

### Before

```py
reveal_type((1, 2) + (3, 4))  # revealed: @Todo(return type of decorated function)
# If TODO is ignored, the revealed type would be `tuple[1|2|3|4, ...]`
```

The `builtins.tuple` type stub defines `__add__`, but it appears to only
work for homogeneous tuples. However, I think this limitation is not
ideal for many use cases.

### After

```py
reveal_type((1, 2) + (3, 4))  # revealed: tuple[Literal[1], Literal[2], Literal[3], Literal[4]]
```

## Test Plan

### Added
- `mdtest/binary/tuples.md`

### Affected
- `mdtest/slots.md` (a test have been moved out of the `False-Negative`
block.)
2025-03-14 12:29:57 +01:00
David Peter
d03b12e711
[red-knot] Assignments to attributes (#16705)
## Summary

This changeset adds proper support for assignments to attributes:
```py
obj.attr = value
```

In particular, the following new features are now available:

* We previously didn't raise any errors if you tried to assign to a
non-existing attribute `attr`. This is now fixed.
* If `type(obj).attr` is a data descriptor, we now call its `__set__`
method instead of trying to assign to the load-context type of
`obj.attr`, which can be different for data descriptors.
* An initial attempt was made to support unions and intersections, as
well as possibly-unbound situations. There are some remaining TODOs in
tests, but they only affect edge cases. Having nested diagnostics would
be one way that could help solve the remaining cases, I believe.

## Follow ups

The following things are planned as follow-ups:

- Write a test suite with snapshot diagnostics for various attribute
assignment errors
- Improve the diagnostics. An easy improvement would be to highlight the
right hand side of the assignment as a secondary span (with the rhs type
as additional information). Some other ideas are mentioned in TODO
comments in this PR.
- Improve the union/intersection/possible-unboundness handling
- Add support for calling custom `__setattr__` methods (see new false
positive in the ecosystem results)

## Ecosystem changes

Some changes are related to assignments on attributes with a custom
`__setattr__` method (see above). Since we didn't notice missing
attributes at all in store context previously, these are new.

The other changes are related to properties. We previously used their
read-context type to test the assignment. That results in weird error
messages, as we often see assignments to `self.property` and then we
think that those are instance attributes *and* descriptors, leading to
union types. Now we properly look them up on the meta type, see the
decorated function, and try to overwrite it with the new value (as we
don't understand decorators yet). Long story short: the errors are still
weird, we need to understand decorators to make them go away.

## Test Plan

New Markdown tests
2025-03-14 12:15:41 +01:00
Carl Meyer
abaa18993b
[red-knot] handle cycles in MRO/bases resolution (#16693)
There can be semi-cyclic inheritance patterns (e.g. recursive generics)
that are not technically inheritance cycles, but that can cause us to
hit Salsa query cycles in evaluating a type's MRO. Add fixed-point
handling to these MRO-related queries so we don't panic on these cycles.

The details of what queries we hit in what order in this case will
change as we implement support for generics, but ultimately we will
probably need cycle handling for all queries that can re-enter type
inference, otherwise we are susceptible to small changes in query
execution order causing panics.

Fixes #14333
Further reduces the panicking set of seeds in #14737
2025-03-13 08:16:03 -07:00
Dhruv Manilawala
58d5fe982e
[red-knot] Check gradual equivalence between callable types (#16634) 2025-03-13 08:16:51 +05:30
Dhruv Manilawala
08fa9b4a90
[red-knot] Add CallableTypeFromFunction special form (#16683)
## Summary

This PR adds a new `CallableTypeFromFunction` special form to allow
extracting the abstract signature of a function literal i.e., convert a
`Type::Function` into a `Type::Callable` (`CallableType::General`).

This is done to support testing the `is_gradual_equivalent_to` type
relation specifically the case we want to make sure that a function that
has parameters with no annotations and does not have a return type
annotation is gradual equivalent to `Callable[[Any, Any, ...], Any]`
where the number of parameters should match between the function literal
and callable type.

Refer
https://github.com/astral-sh/ruff/pull/16634#discussion_r1989976692

### Bikeshedding

The name `CallableTypeFromFunction` is a bit too verbose. A possibly
alternative from Carl is `CallableTypeOf` but that would be similar to
`TypeOf` albeit with a limitation that the former only accepts function
literal types and errors on other types.

Some other alternatives:
* `FunctionSignature`
* `SignatureOf` (similar issues as `TypeOf`?)
* ...

## Test Plan

Update `type_api.md` with a new section that tests this special form,
both invalid and valid forms.
2025-03-13 07:49:34 +05:30
David Peter
083df0cf84
[red-knot] Support custom __getattr__ methods (#16668)
## Summary

Add support for calling custom `__getattr__` methods in case an
attribute is not otherwise found. This allows us to get rid of many
ecosystem false positives where we previously emitted errors when
accessing attributes on `argparse.Namespace`.

closes #16614

## Test Plan

* New Markdown tests
* Observed expected ecosystem changes (the changes for `arrow` also look
fine, since the `Arrow` class has a custom [`__getattr__`
here](1d70d00919/arrow/arrow.py (L802-L815)))
2025-03-12 13:44:11 +01:00
Carl Meyer
a176c1ac80
[red-knot] use fixpoint iteration for all cycles (#14029)
Pulls in the latest Salsa main branch, which supports fixpoint
iteration, and uses it to handle all query cycles.

With this, we no longer need to skip any corpus files to avoid panics.

Latest perf results show a 6% incremental and 1% cold-check regression.
This is not a "no cycles" regression, as tomllib and typeshed do trigger
some definition cycles (previously handled by our old
`infer_definition_types` fallback to `Unknown`). We don't currently have
a benchmark we can use to measure the pure no-cycles regression, though
I expect there would still be some regression; the fixpoint iteration
feature in Salsa does add some overhead even for non-cyclic queries.

I think this regression is within the reasonable range for this feature.
We can do further optimization work later, but I don't think it's the
top priority right now. So going ahead and acknowledging the regression
on CodSpeed.

Mypy primer is happy, so this doesn't regress anything on our
currently-checked projects. I expect it probably unlocks adding a number
of new projects to our ecosystem check that previously would have
panicked.

Fixes #13792
Fixes #14672
2025-03-12 12:41:40 +00:00
David Peter
a6572a57c4
[red-knot] Attribute access on intersection types (#16665)
## Summary

Implements attribute access on intersection types, which didn't
previously work. For example:

```py
from typing import Any

class P: ...
class Q: ...

class A:
    x: P = P()

class B:
    x: Any = Q()

def _(obj: A):
    if isinstance(obj, B):
        reveal_type(obj.x)  # revealed: P & Any
```

Refers to [this comment].

[this comment]:
https://github.com/astral-sh/ruff/pull/16416#discussion_r1985040363

## Test Plan

New Markdown tests
2025-03-12 13:20:17 +01:00
Joey Bar
b250304ad3
[red-knot] Improve is_disjoint for two intersections (#16636)
## Summary

Background - as a follow up to #16611 I noticed that there's a lot of
code duplicated between the `is_assignable_to` and `is_subtype_of`
functions and considered trying to merge them.

[A subtype and an assignable type are pretty much the
same](https://typing.python.org/en/latest/spec/concepts.html#the-assignable-to-or-consistent-subtyping-relation),
except that subtypes are by definition fully static, so I think we can
replace the whole of `is_subtype_of` with:

```
if !self.is_fully_static(db) || !target.is_fully_static(db) {
    return false;
}
return self.is_assignable_to(target)
```

if we move all of the logic to is_assignable_to and delete duplicate
code. Then we can discuss if it even makes sense to have a separate
is_subtype_of function (I think the answer is yes since it's used by a
bunch of other places, but we may be able to basically rip out the
concept).

Anyways while playing with combining the functions I noticed is that the
handling of Intersections in `is_subtype_of` has a special case for two
intersections, which I didn't include in the last PR - rather I first
handled right hand intersections before left hand, which should properly
handle double intersections (hand-wavy explanation I can justify if
needed - (A & B & C) is assignable to (A & B) because the left is
assignable to both A and B, but none of A, B, or C is assignable to (A &
B)).

I took a look at what breaks if I remove the handling for double
intersections, and the reason it is needed is because is_disjoint does
not properly handle intersections with negative conditions (so instead
`is_subtype_of` basically implements the check correctly).

This PR adds support to is_disjoint for properly checking negative
branches, which also lets us simplify `is_subtype_of`, bringing it in
line with `is_assignable_to`

## Test Plan

Added a bunch of tests, most of which failed before this fix

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2025-03-12 12:13:04 +00:00
David Peter
11b5cbcd2f
[red-knot] Restructure attribute-access and descriptor-protocol test suites. (#16664)
## Summary

This is a pure restructuring of the `attributes.md` and
`descriptor_protocol.md` test suites. They have grown organically and I
didn't want to make major structural changes in my recent PR to keep the
diff clean.
2025-03-12 09:52:21 +01:00
David Peter
3228545598
[red-knot] Minor optimization/cleanup in member lookup (#16663)
## Summary

A follow up to address [this comment]:

> Similarly here, it might be a little more performant to have a single
`Type::instance()` branch with an inner match over `class.known()`
rather than having multiple branches with `if class.is_known()` guards

[this comment]:
https://github.com/astral-sh/ruff/pull/16416#discussion_r1985159037
2025-03-12 09:11:05 +01:00
David Peter
860b95a318
[red-knot] Binary operator inference for union types (#16601)
## Summary

Properly handle binary operator inference for union types.

This fixes a bug I noticed while looking at ecosystem results. The MRE
version of it is this:

```py
def sub(x: float, y: float):
    # Red Knot: Operator `-` is unsupported between objects of type `int | float` and `int | float`
    return x - y
```

## Test Plan

- New Markdown tests.
- Expected diff in the ecosystem checks
2025-03-12 08:21:54 +01:00
Dhruv Manilawala
6de2b2873b
[red-knot] Check if callable type is fully static (#16633)
## Summary

Part of #15382 

This PR adds the check for whether a callable type is fully static or
not.

A callable type is fully static if all of the parameter types are fully
static _and_ the return type is fully static _and_ if it does not use
the gradual form (`...`) for its parameters.

## Test Plan

Update `is_fully_static.md` with callable types.

It seems that currently this test is grouped into either fully static or
not, I think it would be useful to split them up in groups like
callable, etc. I intentionally avoided that in this PR but I'll put up a
PR for an appropriate split.

Note: I've an explicit goal of updating the property tests with the new
callable types once all relations are implemented.
2025-03-12 12:13:22 +05:30
Dhruv Manilawala
6b84253679
[red-knot] Callable member lookup, meta type impl (#16618)
## Summary

This PR is a follow-up to https://github.com/astral-sh/ruff/pull/16493
that implements member lookup for the general callable type.

Based on the discussion around [member lookup
here](https://github.com/astral-sh/ruff/pull/16493#discussion_r1982041180)
and [`.to_meta_type()`
here](https://github.com/astral-sh/ruff/pull/16493#discussion_r1985104664).

## Test Plan

Add a new test cases.
2025-03-12 12:01:38 +05:30
Carl Meyer
0340e23395
[red-knot] remove redundant sentence in test (#16660)
Removes a redundant sentence I accidentally left in the test suite from
in #16540 (my mistake).
2025-03-12 04:20:31 +00:00
Shunsuke Shibayama
78b5f0b165
[red-knot] detect invalid return type (#16540)
## Summary

This PR closes #16248.

If the return type of the function isn't assignable to the one
specified, an `invalid-return-type` error occurs.
I thought it would be better to report this as a different kind of error
than the `invalid-assignment` error, so I defined this as a new error.

## Test Plan

All type inconsistencies in the test cases have been replaced with
appropriate ones.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2025-03-12 01:58:59 +00:00
Douglas Creager
e17cd350b6
[red-knot] Support multiple overloads when binding parameters at call sites (#16568)
This updates the `Signature` and `CallBinding` machinery to support
multiple overloads for a callable. This is currently only used for
`KnownFunction`s that we special-case in our type inference code. It
does **_not_** yet update the semantic index builder to handle
`@overload` decorators and construct a multi-signature `Overloads`
instance for real Python functions.

While I was here, I updated many of the `try_call` special cases to use
signatures (possibly overloaded ones now) and `bind_call` to check
parameter lists. We still need some of the mutator methods on
`OverloadBinding` for the special cases where we need to update return
types based on some Rust code.
2025-03-11 15:08:17 -04:00