## Summary
Currently for something like `X = typing.Tuple[str, str]`, we infer the
value of `X` as `object`. That's because `Tuple` (like many of the
symbols in the typing module) is annotated as a `_SpecialForm` instance
in typeshed's stubs:
23382f5f8c/crates/red_knot_vendored/vendor/typeshed/stdlib/typing.pyi (L215)
and we don't understand implicit type aliases yet, and the stub for
`_SpecialForm.__getitem__` says it always returns `object`:
23382f5f8c/crates/red_knot_vendored/vendor/typeshed/stdlib/typing.pyi (L198-L200)
We have existing false positives in our test suite due to this:
23382f5f8c/crates/red_knot_python_semantic/resources/mdtest/annotations/annotated.md (L76-L78)
and it's causing _many_ new false positives in #16872, which tries to
make our annotation-expression parsing stricter in some ways.
This PR therefore adds some small special casing for `KnownInstanceType`
variants that fallback to `_SpecialForm`, so that these false positives
can be avoided.
## Test Plan
Existing mdtest altered.
Cc. @MatthewMckee4
Summary
--
Fixes#16874. I previously emitted a syntax error when starred
annotations were _allowed_ rather than when they were actually used.
This caused false positives for any starred parameter name because these
are allowed to have starred annotations but not required to. The fix is
to check if the annotation is actually starred after parsing it.
Test Plan
--
New inline parser tests derived from the initial report and more
examples from the comments, although I think the first case should cover
them all.
Summary
--
This updates the regex in `ruff-ecosystem` to catch syntax errors in an
effort to prevent bugs like #16874. This should catch `ParseError`s,
`UnsupportedSyntaxError`s, and the upcoming `SemanticSyntaxError`s.
Test Plan
--
I ran the ecosystem check locally comparing v0.11.0 and v0.11.1 and saw
a large number (2757!) of new syntax errors. I also manually tested the
regex on a few lines before that.
If we merge this before #16878, I'd expect to see that number decrease
substantially in that PR too, as another test.
## Summary
From #16641
The previous PR attempted to fix the errors presented in this PR, but as
discussed in the conversation, it was concluded that the approach was
undesirable and that further work would be needed to fix the errors with
a correct general solution.
In this PR, I instead add the test cases from the previous PR as TODOs,
as a starting point for future work.
## Test Plan
---------
Co-authored-by: Carl Meyer <carl@oddbird.net>
## Summary
Fixes#16744
Allows the cli to find a virtual environment from the VIRTUAL_ENV
environment variable if no `--python` is set
## Test Plan
Manual testing, of:
- Virtual environments explicitly activated using `source .venv/bin/activate`
- Virtual environments implicilty activated via `uv run`
- Broken virtual environments with no `pyvenv.cfg` file
`base.sha` appears to be the commit of the base branch when the pull
request was opened, not the base commit that's used to construct the
test merge commit — which can lead to incorrect "determine changes"
results where commits made to the base ref since the pull request are
opened are included in the results.
We use `git merge-base` to find the correct sha, as I don't think that
GitHub provides this. They provide `merge_commit_sha` but my
understanding is that is equivalent to the actual merge commit we're
testing in CI.
I tested this locally on an example pull request. I don't think it's
worth trying to reproduce a specific situation here.
---------
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
## Summary
This change follows up on the bug-fix requested in #16747 --
`ruff_python_ast::OperatorPrecedence` had an enum variant, `BitXorOr`,
which which gave the same precedence to the `|` and `^` operators. This
goes against [Python's documentation for operator
precedence](https://docs.python.org/3/reference/expressions.html#operator-precedence),
so this PR changes the code so that it's correct.
This is part of the overall effort to unify redundant definitions of
`OperatorPrecedence` throughout the codebase (#16071)
## Test Plan
Because this is an internal change, I only ran existing tests to ensure
nothing was broken.
The single flag `has_syntax_error` on `LinterResult` is replaced with
two (private) flags: `has_valid_syntax` and
`has_no_unsupported_syntax_errors`, which record whether there are
`ParseError`s or `UnsupportedSyntaxError`s, respectively. Only the
former is used to prevent a `FixAll` action.
An attempt has been made to make consistent the usage of the phrases
"valid syntax" (which seems to be used to refer only to _parser_ errors)
and "syntax error" (which refers to both _parser_ errors and
version-specific syntax errors).
Closes#16841
## Summary
Fixes#8191 by introducing `--exit-non-zero-on-format` to `ruff format`
which pretty much does what it says on the tin.
## Test Plan
Added a new test!
---------
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
## Summary
This PR reworks `TypeInferenceBuilder::infer_type_expression()` so that
we emit diagnostics when encountering a list literal in a type
expression. The only place where a list literal is allowed in a type
expression is if it appears as the first argument to `Callable[]`, and
`Callable` is already heavily special-cased in our type-expression
parsing.
In order to ensure that list literals are _always_ allowed as the
_first_ argument to `Callabler` (but never allowed as the second, third,
etc. argument), I had to do some refactoring of our type-expression
parsing for `Callable` annotations.
## Test Plan
New mdtests added, and existing ones updated
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
fixes#15048
We want to handle more types from Type::KnownInstance
## Test Plan
Add tests for each type added explicitly in the match
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
## Summary
These are just cosmetic changes, but I'm separating them out into a
standalone PR to make a branch I have stacked on top of this easier to
review
## Test Plan
Existing tests all pass
Summary
--
This PR updates `check_path` in the `ruff_linter` crate to return a
`Vec<Message>` instead of a `Vec<Diagnostic>`. The main motivation for
this is to make it easier to convert semantic syntax errors directly
into `Message`s rather than `Diagnostic`s in #16106. However, this also
has the benefit of keeping the preview check on unsupported syntax
errors in `check_path`, as suggested in
https://github.com/astral-sh/ruff/pull/16429#discussion_r1974748024.
All of the interesting changes are in the first commit. The second
commit just renames variables like `diagnostics` to `messages`, and the
third commit is a tiny import fix.
I also updated the `ExpandedMessage::location` field name, which caused
a few extra commits tidying up the playground code. I thought it was
nicely symmetric with `end_location`, but I'm happy to revert that too.
Test Plan
--
Existing tests. I also tested the playground and server manually.
## Summary
Part of #15382
This PR infers the return type `lambda` expression as `Unknown`. In the
future, it would be more useful to infer the expression type considering
the surrounding context (#16696).
## Test Plan
Update existing test cases from `@todo` to the (verified) return type.
## Summary
Previously, the `name` field was on `Parameter` which required it to be
always optional regardless of the parameter kind because a
`typing.Callable` signature does not have name for the parameters. This
is the case for positional-only parameters. This wasn't enforced at the
type level which meant that downstream usages would have to unwrap on
`name` even though it's guaranteed to be present.
This commit moves the `name` field from `Parameter` to the
`ParameterKind` variants and makes it optional only for
`ParameterKind::PositionalOnly` variant while required for all other
variants.
One change that's now required is that a `Callable` form using a gradual
form for parameter types (`...`) would have a default `args` and
`kwargs` name used for variadic and keyword-variadic parameter kind
respectively. This is also the case for invalid `Callable` type forms. I
think this is fine as names are not relevant in this context but happy
to make it optional even in variadic variants.
## Test Plan
No new tests; make sure existing tests are passing.
## Summary
Add error messages for invalid nodes in type expressions
Fixes#16816
## Test Plan
Extend annotations/invalid.md to handle these invalid AST nodes error
messages
## Summary
This PR adds a playground for Red Knot
[Screencast from 2024-08-14
10-33-54.webm](https://github.com/user-attachments/assets/ae81d85f-74a3-4ba6-bb61-4a871b622f05)
Sharing does work 😆 I just forgot to start wrangler.
It supports:
* Multiple files
* Showing the AST
* Showing the tokens
* Sharing
* Persistence to local storage
Future extensions:
* Configuration support: The `pyproject.toml` would *just* be another
file.
* Showing type information on hover
## Blockers
~~Salsa uses `catch_unwind` to break cycles, which Red Knot uses
extensively when inferring types in the standard library.
However, WASM (at least `wasm32-unknown-unknown`) doesn't support
`catch_unwind` today, so the playground always crashes when the type
inference encounters a cycle.~~
~~I created a discussion in the [salsa
zulip](https://salsa.zulipchat.com/#narrow/stream/333573-salsa-3.2E0/topic/WASM.20support)
to see if it would be possible to **not** use catch unwind to break
cycles.~~
~~[Rust tracking issue for WASM catch unwind
support](https://github.com/rust-lang/rust/issues/118168)~~
~~I tried to build the WASM with the nightly compiler option but ran
into problems because wasm-bindgen doesn't support WASM-exceptions. We
could try to write the binding code by hand.~~
~~Another alternative is to use `wasm32-unknown-emscripten` but it's
rather painful to build~~
## Summary
This PR detects the use of PEP 701 f-strings before 3.12. This one
sounded difficult and ended up being pretty easy, so I think there's a
good chance I've over-simplified things. However, from experimenting in
the Python REPL and checking with [pyright], I think this is correct.
pyright actually doesn't even flag the comment case, but Python does.
I also checked pyright's implementation for
[quotes](98dc4469cc/packages/pyright-internal/src/analyzer/checker.ts (L1379-L1398))
and
[escapes](98dc4469cc/packages/pyright-internal/src/analyzer/checker.ts (L1365-L1377))
and think I've approximated how they do it.
Python's error messages also point to the simple approach of these
characters simply not being allowed:
```pycon
Python 3.11.11 (main, Feb 12 2025, 14:51:05) [Clang 19.1.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f'''multiline {
... expression # comment
... }'''
File "<stdin>", line 3
}'''
^
SyntaxError: f-string expression part cannot include '#'
>>> f'''{not a line \
... continuation}'''
File "<stdin>", line 2
continuation}'''
^
SyntaxError: f-string expression part cannot include a backslash
>>> f'hello {'world'}'
File "<stdin>", line 1
f'hello {'world'}'
^^^^^
SyntaxError: f-string: expecting '}'
```
And since escapes aren't allowed, I don't think there are any tricky
cases where nested quotes or comments can sneak in.
It's also slightly annoying that the error is repeated for every nested
quote character, but that also mirrors pyright, although they highlight
the whole nested string, which is a little nicer. However, their check
is in the analysis phase, so I don't think we have such easy access to
the quoted range, at least without adding another mini visitor.
## Test Plan
New inline tests
[pyright]:
https://pyright-play.net/?pythonVersion=3.11&strict=true&code=EYQw5gBAvBAmCWBjALgCgO4gHaygRgEoAoEaCAIgBpyiiBiCLAUwGdknYIBHAVwHt2LIgDMA5AFlwSCJhwAuCAG8IoMAG1Rs2KIC6EAL6iIxosbPmLlq5foRWiEAAcmERAAsQAJxAomnltY2wuSKogA6WKIAdABWfPBYqCAE%2BuSBVqbpWVm2iHwAtvlMWMgB2ekiolUAgq4FjgA2TAAeEMieSADWCsoV5qoaqrrGDJ5MiDz%2B8ABuLqosAIREhlXlaybrmyYMXsDw7V4AnoysyAmQ5SIhwYo3d9cheADUeKlv5O%2BpQA
## Summary
For now, `property_tests.rs` has grown larger and larger, making the
file difficult to read and maintain.
Although the code has been split, the test paths and full names remain
unchanged. There are no changes affecting test execution.
## Summary
This PR simplifies `IterationError` and `ContextManagerError` so that
they no longer "remember" what type it was that was (respectively) not
iterable or not valid as a context manager. Instead, the type that was
iterated over (or was used as a context manager) is passed back in when
calling the error struct's `report_diagnostic` method.
The motivations for this are:
- It significantly simplifies the code
- It reduces the size of these types on the stack
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
This PR brings an optimization.
- `get_cached_db` no longer returns a `MutexGuard`; instead, it returns
a cloned database.
### `get_cached_db`
Previously, the `MutexGuard` was held inside the property test function
(defined in the macro), which prevented multiple property tests from
running in parallel. More specifically, the program could only test one
random test case at a time, which likely caused a significant
bottleneck.
On my local machine, running:
```
QUICKCHECK_TESTS=100000 cargo test --release -p red_knot_python_semantic -- --ignored stable
```
showed about **a 75% speedup** (from \~60s to \~15s).
Fixes#16751
## Summary
Previously, unsafe fixes were counted as "fixable" in
`Printer::write_statistics`, in contrast to the behaviour in
`Printer::write_once`. This changes the behaviour to align with
`write_once`, including them only if `--unsafe-fixes` is set.
We now also reuse `Printer::write_summary` to avoid duplicating the
logic for whether or not to report if there are hidden fixes.
## Test Plan
Existing tests modified to use an unsafe-fixable rule, and new ones
added to cover the case with `--unsafe-fixes`
## Summary
This is prep-work for the Red Knot playground. We'll have two
playgrounds, one for Red Knot and Ruff.
I want to share some components between the two, a "shared" NPM package
in a local workspace is a great fit for that.
I also want to share the dev dependencies and dev scripts. Again, NPM
workspaces are great for that.
This PR also sets up a CI workflow for the playground to prevent
surprises during the release.
## Test Plan
CI, local `npm install`, `npm start`, ...
I verified that the new CI step fails if there's a typescript or
formatting error.
* [Deployment test
run](3890552435)
These should all be minor cosmetic changes. To summarize:
* In many cases, `-` was replaced with `^` for primary annotations.
This is because, previously, whether `-` or `^` was used depended
on the severity. But in the new data model, it's based on whether
the annotation is "primary" or not. We could of course change this
in whatever way we want, but I think we should roll with this for now.
* The "secondary messages" in the old API are rendered as
sub-diagnostics. This in turn results in a small change in the output
format, since previously, the secondary messages were represented as
just another snippet. We use sub-diagnostics because that's the intended
way to enforce relative ordering between messages within a diagnostic.
* The "info:" prefix used in some annotation messages has been dropped.
We could re-add this, but I think I like it better without this prefix.
I believe those 3 cover all of the snapshot changes here.
... and switch to the new one.
We do this switch by converting the old diagnostics to a
`Diagnostic`, and then rendering that.
This does not quite emit identical output. There are some
changes. They *could* be fixed to remain the same, but the
changes aren't obviously worse to me and I think the right
way to *improve* them is to move Red Knot to the new `Diagnostic`
API.
The next commit will have the snapshot changes.
In our existing diagnostics, our message is just the diagnostic
ID, and the message goes to the annotation. In reality, the
diagnostic can have its own message distinct from the optional
messages associated with an annotation.
In order to make the outputs match, we do a small tweak here:
when the main diagnostic message is empty, we drop the colon
after the diagnostic ID.
I expect that we'll want to rejigger this output format more
in the future, but for now this was a very simple change to
preserve the status quo.
When moving over to the new renderer, I noticed that it
was emitting an extra line terminator compared to the status
quo. This removes it by turning the line terminator into a
line delimiter between diagnostics.
Previously, unless you had some other configuration that impacts
ripgrep, `rg -tyaml uses:` would return zero results. After this
changes, it returns more of what you might expect.
This is because ripgrep ignores hidden files and directories by default.
But arguably, searching `.github` by default is probably what we want.
I do the same thing in ripgrep's repository:
de4baa1002/.ignore (L1)
This cleans up how we handle calling unions of types. #16568 adding a
three-level structure for callable signatures (`Signatures`,
`CallableSignature`, and `Signature`) to handle unions and overloads.
This PR updates the bindings side to mimic that structure. What used to
be called `CallOutcome` is now `Bindings`, and represents the result of
binding actual arguments against a possible union of callables.
`CallableBinding` is the result of binding a single, possibly
overloaded, callable type. `Binding` is the result of binding a single
overload.
While we're here, this also cleans up `CallError` greatly. It was
previously extracting error information from the bindings and storing it
in the error result. It is now a simple enum, carrying no data, that's
used as a status code to talk about whether the overall binding was
successful or not. We are now more consistent about walking the binding
itself to get detailed information about _how_ the binding was
unsucessful.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
