Commit graph

365 commits

Author SHA1 Message Date
Micha Reiser
5fc8e5d80e
[red-knot] Add infrastructure to declare lints (#14873)
## Summary

This is the second PR out of three that adds support for
enabling/disabling lint rules in Red Knot. You may want to take a look
at the [first PR](https://github.com/astral-sh/ruff/pull/14869) in this
stack to familiarize yourself with the used terminology.

This PR adds a new syntax to define a lint: 

```rust
declare_lint! {
    /// ## What it does
    /// Checks for references to names that are not defined.
    ///
    /// ## Why is this bad?
    /// Using an undefined variable will raise a `NameError` at runtime.
    ///
    /// ## Example
    ///
    /// ```python
    /// print(x)  # NameError: name 'x' is not defined
    /// ```
    pub(crate) static UNRESOLVED_REFERENCE = {
        summary: "detects references to names that are not defined",
        status: LintStatus::preview("1.0.0"),
        default_level: Level::Warn,
    }
}
```

A lint has a name and metadata about its status (preview, stable,
removed, deprecated), the default diagnostic level (unless the
configuration changes), and documentation. I use a macro here to derive
the kebab-case name and extract the documentation automatically.

This PR doesn't yet add any mechanism to discover all known lints. This
will be added in the next and last PR in this stack.


## Documentation
I documented some rules but then decided that it's probably not my best
use of time if I document all of them now (it also means that I play
catch-up with all of you forever). That's why I left some rules
undocumented (marked with TODO)

## Where is the best place to define all lints?

I'm not sure. I think what I have in this PR is fine but I also don't
love it because most lints are in a single place but not all of them. If
you have ideas, let me know.


## Why is the message not part of the lint, unlike Ruff's `Violation`

I understand that the main motivation for defining `message` on
`Violation` in Ruff is to remove the need to repeat the same message
over and over again. I'm not sure if this is an actual problem. Most
rules only emit a diagnostic in a single place and they commonly use
different messages if they emit diagnostics in different code paths,
requiring extra fields on the `Violation` struct.

That's why I'm not convinced that there's an actual need for it and
there are alternatives that can reduce the repetition when creating a
diagnostic:

* Create a helper function. We already do this in red knot with the
`add_xy` methods
* Create a custom `Diagnostic` implementation that tailors the entire
diagnostic and pre-codes e.g. the message

Avoiding an extra field on the `Violation` also removes the need to
allocate intermediate strings as it is commonly the place in Ruff.
Instead, Red Knot can use a borrowed string with `format_args`

## Test Plan

`cargo test`
2024-12-10 16:14:44 +00:00
Micha Reiser
5f548072d9
[red-knot] Typed diagnostic id (#14869)
## Summary

This PR introduces a structured `DiagnosticId` instead of using a plain
`&'static str`. It is the first of three in a stack that implements a
basic rules infrastructure for Red Knot.

`DiagnosticId` is an enum over all known diagnostic codes. A closed enum
reduces the risk of accidentally introducing two identical diagnostic
codes. It also opens the possibility of generating reference
documentation from the enum in the future (not part of this PR).

The enum isn't *fully closed* because it uses a `&'static str` for lint
names. This is because we want the flexibility to define lints in
different crates, and all names are only known in `red_knot_linter` or
above. Still, lower-level crates must already reference the lint names
to emit diagnostics. We could define all lint-names in `DiagnosticId`
but I decided against it because:

* We probably want to share the `DiagnosticId` type between Ruff and Red
Knot to avoid extra complexity in the diagnostic crate, and both tools
use different lint names.
* Lints require a lot of extra metadata beyond just the name. That's why
I think defining them close to their implementation is important.

In the long term, we may also want to support plugins, which would make
it impossible to know all lint names at compile time. The next PR in the
stack introduces extra syntax for defining lints.

A closed enum does have a few disadvantages:

* rustc can't help us detect unused diagnostic codes because the enum is
public
* Adding a new diagnostic in the workspace crate now requires changes to
at least two crates: It requires changing the workspace crate to add the
diagnostic and the `ruff_db` crate to define the diagnostic ID. I
consider this an acceptable trade. We may want to move `DiagnosticId` to
its own crate or into a shared `red_knot_diagnostic` crate.


## Preventing duplicate diagnostic identifiers

One goal of this PR is to make it harder to introduce ambiguous
diagnostic IDs, which is achieved by defining a closed enum. However,
the enum isn't fully "closed" because it doesn't explicitly list the IDs
for all lint rules. That leaves the possibility that a lint rule and a
diagnostic ID share the same name.

I made the names unambiguous in this PR by separating them into
different namespaces by using `lint/<rule>` for lint rule codes. I don't
mind the `lint` prefix in a *Ruff next* context, but it is a bit weird
for a standalone type checker. I'd like to not overfocus on this for now
because I see a few different options:

* We remove the `lint` prefix and add a unit test in a top-level crate
that iterates over all known lint rules and diagnostic IDs to ensure the
names are non-overlapping.
* We only render `[lint]` as the error code and add a note to the
diagnostic mentioning the lint rule. This is similar to clippy and has
the advantage that the header line remains short
(`lint/some-long-rule-name` is very long ;))
* Any other form of adjusting the diagnostic rendering to make the
distinction clear

I think we can defer this decision for now because the `DiagnosticId`
contains all the relevant information to change the rendering
accordingly.


## Why `Lint` and not `LintRule`

I see three kinds of diagnostics in Red Knot:

* Non-suppressable: Reveal type, IO errors, configuration errors, etc.
(any `DiagnosticId`)
* Lints: code-related diagnostics that are suppressable. 
* Lint rules: The same as lints, but they can be enabled or disabled in
the configuration. The majority of lints in Red Knot and the Ruff
linter.

Our current implementation doesn't distinguish between lints and Lint
rules because we aren't aware of a suppressible code-related lint that
can't be configured in the configuration. The only lint that comes to my
mind is maybe `division-by-zero` if we're 99.99% sure that it is always
right. However, I want to keep the door open to making this distinction
in the future if it proves useful.

Another reason why I chose lint over lint rule (or just rule) is that I
want to leave room for a future lint rule and lint phase concept:

* lint is the *what*: a specific code smell, pattern, or violation 
* the lint rule is the *how*: I could see a future `LintRule` trait in
`red_knot_python_linter` that provides the necessary hooks to run as
part of the linter. A lint rule produces diagnostics for exactly one
lint. A lint rule differs from all lints in `red_knot_python_semantic`
because they don't run as "rules" in the Ruff sense. Instead, they're a
side-product of type inference.
* the lint phase is a different form of *how*: A lint phase can produce
many different lints in a single pass. This is a somewhat common pattern
in Ruff where running one analysis collects the necessary information
for finding many different lints
* diagnostic is the *presentation*: Unlike a lint, the diagnostic isn't
the what, but how a specific lint gets presented. I expect that many
lints can use one generic `LintDiagnostic`, but a few lints might need
more flexibility and implement their custom diagnostic rendering (at
least custom `Diagnostic` implementation).


## Test Plan

`cargo test`
2024-12-10 15:58:07 +00:00
InSync
15fe540251
Improve mdtests style (#14884)
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
2024-12-10 13:05:51 +00:00
Alex Waygood
ab26d9cf9a
[red-knot] Improve type inference for except handlers (#14838) 2024-12-09 22:49:58 +00:00
Dimitri Papadopoulos Orfanos
64944f2cf5
More typos found by codespell (#14880) 2024-12-09 22:47:34 +00:00
Carl Meyer
533e8a6ee6
[red-knot] move standalone expression_ty to TypeInferenceBuilder::file_expression_ty (#14879)
## Summary

Per suggestion in
https://github.com/astral-sh/ruff/pull/14802#discussion_r1875455417

This is a bit less error-prone and allows us to handle both expressions
in the current scope or a different scope. Also, there's currently no
need for this method outside of `TypeInferenceBuilder`, so no reason to
expose it in `types.rs`.

## Test Plan

Pure refactor, no functional change; existing tests pass.

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-12-09 17:02:14 +00:00
InSync
3865fb6641
[red-knot] Understanding type[Union[A, B]] (#14858)
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
2024-12-09 12:47:14 +00:00
Dimitri Papadopoulos Orfanos
59145098d6
Fix typos found by codespell (#14863)
## Summary

Just fix typos.

## Test Plan

CI tests.

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2024-12-09 09:32:12 +00:00
Shaygan Hooshyari
269e47be96
Understand type[A | B] special form in annotations (#14830)
resolves https://github.com/astral-sh/ruff/issues/14703

I decided to use recursion to get the type, so if anything is added to
the single element inference it will be applied for the union.
Also added this
[change](https://github.com/astral-sh/ruff/issues/14703#issuecomment-2510286217)
in this PR since it was easy.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2024-12-07 17:34:50 +00:00
Douglas Creager
8fdd88013d
Support type[a.X] with qualified class names (#14825)
This adds support for `type[a.X]`, where the `type` special form is
applied to a qualified name that resolves to a class literal. This works
for both nested classes and classes imported from another module.

Closes #14545
2024-12-06 17:14:51 -05:00
Carl Meyer
3017b3b687
[red-knot] function parameter types (#14802)
## Summary

Inferred and declared types for function parameters, in the function
body scope.

Fixes #13693.

## Test Plan

Added mdtests.

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-12-06 12:55:56 -08:00
David Peter
6b9f3d7d7c
[red-knot] Import LiteralString/Never from typing_extensions (#14817)
## Summary

`typing.Never` and `typing.LiteralString` are only conditionally
exported from `typing` for Python versions 3.11 and later. We run the
Markdown tests with the default Python version of 3.9, so here we change
the import to `typing_extensions` instead, and add a new test to make
sure we'll continue to understand the `typing`-version of these symbols
for newer versions.

This didn't cause problems so far, as we don't understand
`sys.version_info` branches yet.

## Test Plan

New Markdown tests to make sure this will continue to work in the
future.
2024-12-06 13:57:51 +01:00
Douglas Creager
918358aaa6
Migrate some inference tests to mdtests (#14795)
As part of #13696, this PR ports a smallish number of inference tests
over to the mdtest framework.
2024-12-06 11:19:22 +01:00
David Peter
b01a651e69
[red-knot] Support for TOML configs in Markdown tests (#14785)
## Summary

This adds support for specifying the target Python version from a
Markdown test. It is a somewhat limited ad-hoc solution, but designed to
be future-compatible. TOML blocks can be added to arbitrary sections in
the Markdown block. They have the following format:

````markdown
```toml
[tool.knot.environment]
target-version = "3.13"
```
````

So far, there is nothing else that can be configured, but it should be
straightforward to extend this to things like a custom typeshed path.

This is in preparation for the statically-known branches feature where
we are going to have to specify the target version for lots of tests.

## Test Plan

- New Markdown test that fails without the explicitly specified
`target-version`.
- Manually tested various error paths when specifying a wrong
`target-version` field.
- Made sure that running tests is as fast as before.
2024-12-06 10:22:08 +01:00
Dhruv Manilawala
40b0b67dd9
[red-knot] Separate invalid syntax code snippets (#14803)
Ref: https://github.com/astral-sh/ruff/pull/14788#discussion_r1872242283

This PR:
* Separates code snippets as individual tests for the invalid syntax
cases
* Adds a general comment explaining why the parser could emit more
syntax errors than expected
2024-12-06 02:41:33 +00:00
Dhruv Manilawala
e9941cd714
[red-knot] Move standalone expr inference to for non-name target (#14788)
## Summary

Ref: https://github.com/astral-sh/ruff/pull/14754#discussion_r1871040646

## Test Plan

Remove the TODO comment and update the mdtest.
2024-12-05 18:06:20 +05:30
Dhruv Manilawala
43bf1a8907
Add tests for "keyword as identifier" syntax errors (#14754)
## Summary

This is related to #13778, more specifically
https://github.com/astral-sh/ruff/issues/13778#issuecomment-2513556004.

This PR adds various test cases where a keyword is being where an
identifier is expected. The tests are to make sure that red knot doesn't
panic, raises the syntax error and the identifier is added to the symbol
table. The final part allows editor related features like renaming the
symbol.
2024-12-05 17:32:48 +05:30
David Peter
2d3f557875
[red-knot] Fallback for typing._NoDefaultType (#14783)
## Summary

`typing_extensions` has a `>=3.13` re-export for the `typing.NoDefault`
singleton, but not for `typing._NoDefaultType`. This causes problems as
soon as we understand `sys.version_info` branches, so we explicity
switch to `typing._NoDefaultType` for Python 3.13 and later.

This is a part of #14759 that I thought might make sense to break out
and merge in isolation.

## Test Plan

New test that will become more meaningful with #12700

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2024-12-05 09:17:55 +01:00
David Peter
bd27bfab5d
[red-knot] Unify setup_db() functions, add TestDb builder (#14777)
## Summary

- Instead of seven (more or less similar) `setup_db` functions, use just
one in a single central place.
- For every test that needs customization beyond that, offer a
`TestDbBuilder` that can control the Python target version, custom
typeshed, and pre-existing files.

The main motivation for this is that we're soon going to need
customization of the Python version, and I didn't feel like adding this
to each of the existing `setup_db` functions.
2024-12-04 21:36:54 +01:00
InSync
155d34bbb9
[red-knot] Infer precise types for len() calls (#14599)
## Summary

Resolves #14598.

## Test Plan

Markdown tests.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2024-12-04 11:16:53 -08:00
David Peter
af43bd4b0f
[red-knot] Gradual forms do not participate in equivalence/subtyping (#14758)
## Summary

This changeset contains various improvements concerning non-fully-static
types and their relationships:

- Make sure that non-fully-static types do not participate in
equivalence or subtyping.
- Clarify what `Type::is_equivalent_to` actually implements.
- Introduce `Type::is_fully_static`
- New tests making sure that multiple `Any`/`Unknown`s inside unions and
intersections are collapsed.

closes #14524

## Test Plan

- Added new unit tests for union and intersection builder
- Added new unit tests for `Type::is_equivalent_to`
- Added new unit tests for `Type::is_subtype_of`
- Added new property test making sure that non-fully-static types do not
participate in subtyping
2024-12-04 17:11:25 +01:00
Douglas Creager
8b23086eac
[red-knot] Add typing.Any as a spelling for the Any type (#14742)
We already had a representation for the Any type, which we would use
e.g. for expressions without type annotations. We now recognize
`typing.Any` as a way to refer to this type explicitly. Like other
special forms, this is tracked correctly through aliasing, and isn't
confused with local definitions that happen to have the same name.

Closes #14544
2024-12-04 09:56:36 -05:00
David Peter
948549fcdc
[red-knot] Test: Hashable/Sized => A/B (#14769)
## Summary

Minor change that uses two plain classes `A` and `B` instead of
`typing.Sized` and `typing.Hashable`.

The motivation is twofold: I remember that I was confused when I first
saw this test. Was there anything specific to `Sized` and `Hashable`
that was relevant here? (there is, these classes are not overlapping;
and you can build a proper intersection from them; but that's true for
almost all non-builtin classes).

I now ran into another problem while working on #14758: `Sized` and
`Hashable` are protocols that we don't fully understand yet. This
causing some trouble when trying to infer whether these are fully-static
types or not.
2024-12-04 15:00:27 +01:00
David Peter
74309008fd
[red-knot] Property tests (#14178)
## Summary

This PR adds a new `property_tests` module with quickcheck-based tests
that verify certain properties of types. The following properties are
currently checked:

* `is_equivalent_to`:
  * is reflexive: `T` is equivalent to itself
* `is_subtype_of`:
  * is reflexive: `T` is a subtype of `T`
* is antisymmetric: if `S <: T` and `T <: S`, then `S` is equivalent to
`T`
  * is transitive: `S <: T` & `T <: U` => `S <: U`
* `is_disjoint_from`:
  * is irreflexive: `T` is not disjoint from `T`
  * is symmetric: `S` disjoint from `T` => `T` disjoint from `S`
* `is_assignable_to`:
  * is reflexive
* `negate`:
  * is an involution: `T.negate().negate()` is equivalent to `T`

There are also some tests that validate higher-level properties like:

* `S <: T` implies that `S` is not disjoint from `T`
* `S <: T` implies that `S` is assignable to `T`
* A singleton type must also be single-valued

These tests found a few bugs so far:

- #14177 
- #14195 
- #14196 
- #14210
- #14731

Some additional notes:

- Quickcheck-based property tests are non-deterministic and finding
counter-examples might take an arbitrary long time. This makes them bad
candidates for running in CI (for every PR). We can think of running
them in a cron-job way from time to time, similar to fuzzing. But for
now, it's only possible to run them locally (see instructions in source
code).
- Some tests currently find false positive "counterexamples" because our
understanding of equivalence of types is not yet complete. We do not
understand that `int | str` is the same as `str | int`, for example.
These tests are in a separate `property_tests::flaky` module.
- Properties can not be formulated in every way possible, due to the
fact that `is_disjoint_from` and `is_subtype_of` can produce false
negative answers.
- The current shrinking implementation is very naive, which leads to
counterexamples that are very long (`str & Any & ~tuple[Any] &
~tuple[Unknown] & ~Literal[""] & ~Literal["a"] | str & int & ~tuple[Any]
& ~tuple[Unknown]`), requiring the developer to simplify manually. It
has not been a major issue so far, but there is a comment in the code
how this can be improved.
- The tests are currently implemented using a macro. This is a single
commit on top which can easily be reverted, if we prefer the plain code
instead. With the macro:
  ```rs
  // `S <: T` implies that `S` can be assigned to `T`.
  type_property_test!(
      subtype_of_implies_assignable_to, db,
forall types s, t. s.is_subtype_of(db, t) => s.is_assignable_to(db, t)
  );
  ```
  without the macro:
  ```rs
  /// `S <: T` implies that `S` can be assigned to `T`.
  #[quickcheck]
  fn subtype_of_implies_assignable_to(s: Ty, t: Ty) -> bool {
      let db = get_cached_db();
  
      let s = s.into_type(&db);
      let t = t.into_type(&db);
  
      !s.is_subtype_of(&*db, t) || s.is_assignable_to(&*db, t)
  }
  ```

## Test Plan

```bash
while cargo test --release -p red_knot_python_semantic --features property_tests types::property_tests; do :; done
```
2024-12-03 13:54:54 +01:00
David Peter
a255d79087
[red-knot] is_subtype_of fix for KnownInstance types (#14750)
## Summary

`KnownInstance::instance_fallback` may return instances of supertypes.
For example, it returns an instance of `_SpecialForm` for `Literal`.
This means it can't be used on the right-hand side of `is_subtype_of`
relationships, because it might lead to false positives.

I can lead to false negatives on the left hand side of `is_subtype_of`,
but this is at least a known limitation. False negatives are fine for
most applications, but false positives can lead to wrong results in
intersection-simplification, for example.

closes #14731

## Test Plan

Added regression test
2024-12-03 12:03:26 +01:00
David Peter
a69dfd4a74
[red-knot] Simplify tuples containing Never (#14744)
## Summary

Simplify tuples containing `Never` to `Never`:

```py
from typing import Never

def never() -> Never: ...

reveal_type((1, never(), "foo"))  # revealed: Never
```

I should note that mypy and pyright do *not* perform this
simplification. I don't know why.


There is [only one
place](5137fcc9c8/crates/red_knot_python_semantic/src/types/infer.rs (L1477-L1484))
where we use `TupleType::new` directly (instead of `Type::tuple`, which
changes behavior here). This appears when creating `TypeVar`
constraints, and it looks to me like it should stay this way, because
we're using `TupleType` to store a list of constraints there, instead of
an actual type. We also store `tuple[constraint1, constraint2, …]` as
the type for the `constraint1, constraint2, …` tuple expression. This
would mean that we infer a type of `tuple[str, Never]` for the following
type variable constraints, without simplifying it to `Never`. This seems
like a weird edge case that's maybe not worth looking further into?!
```py
from typing import Never

#         vvvvvvvvvv
def f[T: (str, Never)](x: T):
    pass
```

## Test Plan

- Added a new unit test. Did not add additional Markdown tests as that
seems superfluous.
- Tested the example above using red knot, mypy, pyright.
- Verified that this allows us to remove `contains_never` from the
property tests
(https://github.com/astral-sh/ruff/pull/14178#discussion_r1866473192)
2024-12-03 08:28:36 +01:00
InSync
246a6df87d
[red-knot] Deeper understanding of LiteralString (#14649)
## Summary

Resolves #14648.

## Test Plan

Markdown tests.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2024-12-03 03:31:58 +00:00
Connor Skees
3e702e12f7
red-knot: support narrowing for bool(E) (#14668)
Resolves https://github.com/astral-sh/ruff/issues/14547 by delegating
narrowing to `E` for `bool(E)` where `E` is some expression.

This change does not include other builtin class constructors which
should also work in this position, like `int(..)` or `float(..)`, as the
original issue does not mention these. It should be easy enough to add
checks for these as well if we want to.

I don't see a lot of markdown tests for malformed input, maybe there's a
better place for the no args and too many args cases to go?

I did see after the fact that it looks like this task was intended for a
new hire.. my apologies. I got here from
https://github.com/astral-sh/ruff/issues/13694, which is marked
help-wanted.

---------

Co-authored-by: David Peter <mail@david-peter.de>
2024-12-03 03:04:59 +00:00
Connor Skees
579ef01294
mdtest: include test name in printed rerun command (#14684)
Co-authored-by: Micha Reiser <micha@reiser.io>
2024-11-30 11:01:06 +00:00
Micha Reiser
b63c2e126b
Upgrade Rust toolchain to 1.83 (#14677) 2024-11-29 12:05:05 +00:00
Samodya Abeysiriwardane
3f6c65e78c
[red-knot] Fix merged type after if-else without explicit else branch (#14621)
## Summary

Closes: https://github.com/astral-sh/ruff/issues/14593

The final type of a variable after if-statement without explicit else
branch should be similar to having an explicit else branch.

## Test Plan

Originally failed test cases from the bug are added.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-11-28 06:23:55 -08:00
David Peter
a378ff38dc
[red-knot] Fix Boolean flags in mdtests (#14654)
## Summary

Similar to #14652, but now with conditions that are `Literal[True]`
(instead of `Literal[False]`), where we want them to be `bool`.
2024-11-28 14:29:35 +01:00
David Peter
6f1cf5b686
[red-knot] Minor fix in MRO tests (#14652)
## Summary

`bool()` is equal to `False`, and we infer `Literal[False]` for it. Which
means that the test here will fail as soon as we treat the body of
this `if` as unreachable.
2024-11-28 10:17:15 +01:00
David Peter
b94d6cf567
[red-knot] Fix panic related to f-strings in annotations (#14613)
## Summary

Fix panics related to expressions without inferred types in invalid
syntax examples like:
```py
x: f"Literal[{1 + 2}]" = 3
```
where the `1 + 2` expression (and its sub-expressions) inside the
annotation did not have an inferred type.

## Test Plan

Added new corpus test.
2024-11-26 16:35:44 +01:00
David Peter
0e71c9e3bb
[red-knot] Fix unit tests in release mode (#14604)
## Summary

This is about the easiest patch that I can think of. It has a drawback
in that there is no real guarantee this won't happen again. I think this
might be acceptable, given that all of this is a temporary thing.

And we also add a new CI job to prevent regressions like this in the
future.

For the record though, I'm listing alternative approaches I thought of:

- We could get rid of the debug/release distinction and just add `@Todo`
type metadata everywhere. This has possible affects on runtime. The main
reason I didn't follow through with this is that the size of `Type`
increases. We would either have to adapt the `assert_eq_size!` test or
get rid of it. Even if we add messages everywhere and get rid of the
file-and-line-variant in the enum, it's not enough to get back to the
current release-mode size of `Type`.
- We could generally discard `@Todo` meta information when using it in
tests. I think this would be a huge drawback. I like that we can have
the actual messages in the mdtest. And make sure we get the expected
`@Todo` type, not just any `@Todo`. It's also helpful when debugging
tests.

closes #14594

## Test Plan

```rs
cargo nextest run --release
```
2024-11-26 15:40:02 +01:00
Shaygan Hooshyari
557d583e32
Support typing.NoReturn and typing.Never (#14559)
Fix #14558 
## Summary

- Add `typing.NoReturn` and `typing.Never` to known instances and infer
them as `Type::Never`
- Add `is_assignable_to` cases for `Type::Never`

I skipped emitting diagnostic for when a function is annotated as
`NoReturn` but it actually returns.

## Test Plan

Added tests from

https://github.com/python/typing/blob/main/conformance/tests/specialtypes_never.py
except from generics and checking if the return value of the function
and the annotations match.

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
2024-11-25 21:37:55 +00:00
cake-monotone
f98eebdbab
[red-knot] Fix Leaking Narrowing Constraint in ast::ExprIf (#14590)
## Summary

Closes #14588


```py
x: Literal[42, "hello"] = 42 if bool_instance() else "hello"
reveal_type(x)  # revealed: Literal[42] | Literal["hello"]

_ = ... if isinstance(x, str) else ...

# The `isinstance` test incorrectly narrows the type of `x`.
# As a result, `x` is revealed as Literal["hello"], but it should remain Literal[42, "hello"].
reveal_type(x)  # revealed: Literal["hello"]
```

## Test Plan
mdtest included!

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-11-25 10:36:37 -08:00
Dhruv Manilawala
5a30ec0df6
Avoid inferring invalid expr types for string annotation (#14447)
## Summary

fixes: #14440

## Test Plan

Add a test case with all the invalid expressions in a string annotation
context.
2024-11-25 21:27:03 +05:30
Carl Meyer
4ba847f250
[red-knot] remove wrong typevar attribute implementations (#14540) 2024-11-22 13:17:16 -08:00
David Peter
f6b2cd5588
[red-knot] Semantic index: handle invalid breaks (#14522)
## Summary

This fix addresses panics related to invalid syntax like the following
where a `break` statement is used in a nested definition inside a
loop:

```py
while True:

    def b():
        x: int

        break
```

closes #14342

## Test Plan

* New corpus regression tests.
* New unit test to make sure we handle nested while loops correctly.
This test is passing on `main`, but can easily fail if the
`is_inside_loop` state isn't properly saved/restored.
2024-11-22 13:13:55 +01:00
David Peter
a90e404c3f
[red-knot] PEP 695 type aliases (#14357)
## Summary

Add support for (non-generic) type aliases. The main motivation behind
this was to get rid of panics involving expressions in (generic) type
aliases. But it turned out the best way to fix it was to implement
(partial) support for type aliases.

```py
type IntOrStr = int | str

reveal_type(IntOrStr)  # revealed: typing.TypeAliasType
reveal_type(IntOrStr.__name__)  # revealed: Literal["IntOrStr"]

x: IntOrStr = 1

reveal_type(x)  # revealed: Literal[1]

def f() -> None:
    reveal_type(x)  # revealed: int | str
```

## Test Plan

- Updated corpus test allow list to reflect that we don't panic anymore.
- Added Markdown-based test for type aliases (`type_alias.md`)
2024-11-22 08:47:14 +01:00
Micha Reiser
87043a2415
Limit type size assertion to 64bit (#14514) 2024-11-21 12:49:55 +00:00
David Peter
f684b6fff4
[red-knot] Fix: Infer type for typing.Union[..] tuple expression (#14510)
## Summary

Fixes a panic related to sub-expressions of `typing.Union` where we fail
to store a type for the `int, str` tuple-expression in code like this:
```
x: Union[int, str] = 1
```

relates to [my
comment](https://github.com/astral-sh/ruff/pull/14499#discussion_r1851794467)
on #14499.

## Test Plan

New corpus test
2024-11-21 11:49:20 +01:00
David Peter
47f39ed1a0
[red-knot] Meta data for Type::Todo (#14500)
## Summary

Adds meta information to `Type::Todo`, allowing developers to easily
trace back the origin of a particular `@Todo` type they encounter.

Instead of `Type::Todo`, we now write either `type_todo!()` which
creates a `@Todo[path/to/source.rs:123]` type with file and line
information, or using `type_todo!("PEP 604 unions not supported")`,
which creates a variant with a custom message.

`Type::Todo` now contains a `TodoType` field. In release mode, this is
just a zero-sized struct, in order not to create any overhead. In debug
mode, this is an `enum` that contains the meta information.

`Type` implements `Copy`, which means that `TodoType` also needs to be
copyable. This limits the design space. We could intern `TodoType`, but
I discarded this option, as it would require us to have access to the
salsa DB everywhere we want to use `Type::Todo`. And it would have made
the macro invocations less ergonomic (requiring us to pass `db`).

So for now, the meta information is simply a `&'static str` / `u32` for
the file/line variant, or a `&'static str` for the custom message.
Anything involving a chain/backtrace of several `@Todo`s or similar is
therefore currently not implemented. Also because we currently don't see
any direct use cases for this, and because all of this will eventually
go away.

Note that the size of `Type` increases from 16 to 24 bytes, but only in
debug mode.

## Test Plan

- Observed the changes in Markdown tests.
- Added custom messages for all `Type::Todo`s that were revealed in the
tests
- Ran red knot in release and debug mode on the following Python file:
  ```py
  def f(x: int) -> int:
      reveal_type(x)
  ```
Prints `@Todo` in release mode and `@Todo(function parameter type)` in
debug mode.
2024-11-21 09:59:47 +01:00
Shaygan Hooshyari
aecdb8c144
[red-knot] support typing.Union in type annotations (#14499)
Fix #14498

## Summary

This PR adds `typing.Union` support

## Test Plan

I created new tests in mdtest.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
2024-11-20 21:55:33 +00:00
cake-monotone
6a4d207db7
[red-knot] Refactoring the inference logic of lexicographic comparisons (#14422)
## Summary

closes #14279

### Limitations of the Current Implementation
#### Incorrect Error Propagation

In the current implementation of lexicographic comparisons, if the
result of an Eq operation is Ambiguous, the comparison stops
immediately, returning a bool instance. While this may yield correct
inferences, it fails to capture unsupported-operation errors that might
occur in subsequent comparisons.
```py
class A: ...

(int_instance(), A()) < (int_instance(), A())  # should error
```

#### Weak Inference in Specific Cases

> Example: `(int_instance(), "foo") == (int_instance(), "bar")`
> Current result: `bool`
> Expected result: `Literal[False]`

`Eq` and `NotEq` have unique behavior in lexicographic comparisons
compared to other operators. Specifically:
- For `Eq`, if any non-equal pair exists within the tuples being
compared, we can immediately conclude that the tuples are not equal.
- For `NotEq`, if any equal pair exists, we can conclude that the tuples
are unequal.

```py
a = (str_instance(), int_instance(), "foo")

reveal_type(a == a)  # revealed: bool
reveal_type(a != a)  # revealed: bool

b = (str_instance(), int_instance(), "bar")

reveal_type(a == b)  # revealed: bool  # should be Literal[False]
reveal_type(a != b)  # revealed: bool  # should be Literal[True]
```
#### Incorrect Support for Non-Boolean Rich Comparisons

In CPython, aside from `==` and `!=`, tuple comparisons return a
non-boolean result as-is. Tuples do not convert the value into `bool`.

Note: If all pairwise `==` comparisons between elements in the tuples
return Truthy, the comparison then considers the tuples' lengths.
Regardless of the return type of the dunder methods, the final result
can still be a boolean.

```py
from __future__ import annotations

class A:
    def __eq__(self, o: object) -> str:
        return "hello"

    def __ne__(self, o: object) -> bytes:
        return b"world"

    def __lt__(self, o: A) -> float:
        return 3.14

a = (A(), A())

reveal_type(a == a)  # revealed: bool
reveal_type(a != a)  # revealed: bool
reveal_type(a < a)  # revealed: bool # should be: `float | Literal[False]`

```

### Key Changes
One of the major changes is that comparisons no longer end with a `bool`
result when a pairwise `Eq` result is `Ambiguous`. Instead, the function
attempts to infer all possible cases and unions the results. This
improvement allows for more robust type inference and better error
detection.

Additionally, as the function is now optimized for tuple comparisons,
the name has been changed from the more general
`infer_lexicographic_comparison` to `infer_tuple_rich_comparison`.

## Test Plan

mdtest included
2024-11-19 17:32:43 -08:00
David Peter
f8c20258ae
[red-knot] Do not panic on f-string format spec expressions (#14436)
## Summary

Previously, we panicked on expressions like `f"{v:{f'0.2f'}}"` because
we did not infer types for expressions nested inside format spec
elements.

## Test Plan

```
cargo nextest run -p red_knot_workspace -- --ignored linter_af linter_gz
```
2024-11-19 10:04:51 +01:00
David Peter
d8538d8c98
[red-knot] Narrowing for type(x) is C checks (#14432)
## Summary

Add type narrowing for `type(x) is C` conditions (and `else` clauses of
`type(x) is not C` conditionals):

```py
if type(x) is A:
    reveal_type(x)  # revealed: A
else:
    reveal_type(x)  # revealed: A | B
```

closes: #14431, part of: #13694

## Test Plan

New Markdown-based tests.
2024-11-18 16:21:46 +01:00
David Peter
d81b6cd334
[red-knot] Types for subexpressions of annotations (#14426)
## Summary

This patches up various missing paths where sub-expressions of type
annotations previously had no type attached. Examples include:
```py
tuple[int, str]
#     ~~~~~~~~

type[MyClass]
#    ~~~~~~~

Literal["foo"]
#       ~~~~~

Literal["foo", Literal[1, 2]]
#              ~~~~~~~~~~~~~

Literal[1, "a", random.illegal(sub[expr + ession])]
#               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

## Test Plan

```
cargo nextest run -p red_knot_workspace -- --ignored linter_af linter_gz
```
2024-11-18 13:03:27 +01:00
Micha Reiser
d99210c049
[red-knot] Default to python 3.9 (#14429) 2024-11-18 11:27:40 +00:00