Commit graph

217 commits

Author SHA1 Message Date
Micha Reiser
ce938fe205
[ty] Speedup project file discovery (#19913) 2025-08-14 19:38:39 +01:00
Brent Westbrook
7f8f1ab2c1
[pyflakes] Add secondary annotation showing previous definition (F811) (#19900)
## Summary

This is a second attempt at a first use of a new diagnostic feature
after #19886. I'll blame rustc for this one because it also has a
similar diagnostic:

<img width="735" height="335" alt="image"
src="https://github.com/user-attachments/assets/572fe1c3-1742-4ce4-b575-1d9196ff0932"
/>

We end up with a very similar diagnostic:

<img width="764" height="401" alt="image"
src="https://github.com/user-attachments/assets/01eaf0c7-2567-467b-a5d8-a27206b2c74c"
/>

## Test Plan

New snapshots and manual tests above
2025-08-14 13:23:43 -04:00
Ibraheem Ahmed
f34b65b7a0
[ty] Track heap usage of salsa structs (#19790)
Co-authored-by: Micha Reiser <micha@reiser.io>
2025-08-12 13:28:44 +02:00
Brent Westbrook
c433865801
Avoid underflow in default ranges before a BOM (#19839)
Summary
--

This fixes a regression caused by the BOM handling in #19806. Most
diagnostics already account for the BOM in their ranges, but those that
use `TextRange::default` to mean the beginning of the file do not,
causing an underflow in `RenderableAnnotation::new` when subtracting the
BOM-shifted `snippet_start` from the annotation range.

I ran into this when trying to run benchmarks on CPython in preparation
for caching work. The file `cpython/Lib/test/bad_coding2.py` was causing
a crash because it had a default-range `I002` diagnostic, with a BOM.


7cc3f1ebe9/crates/ruff_linter/src/rules/isort/rules/add_required_imports.rs (L122-L126)

The fix here is just to saturate to zero instead of panicking. I
considered adding a `TextRange::saturating_sub` method, but I wasn't
sure it was worth it for this one use. I'm happy to do that if
preferred, though.

Saturating seemed easier than shifting the affected annotations over,
but that could be another solution.

Test Plan
--

A new `ruff_db` test that reproduced the issue and manual testing
against the CPython file mentioned above
2025-08-11 08:52:27 -04:00
Brent Westbrook
44755e6e86
Move full diagnostic rendering to ruff_db (#19415)
## Summary

This PR switches the `full` output format in Ruff over to use the
rendering code
in `ruff_db`. As proposed in the design doc, this involves a lot of
changes to the snapshot output.

I also had to comment out this assertion with a TODO to replace it after
https://github.com/astral-sh/ruff/issues/19688 because many of Ruff's
"file-level" annotations aren't actually file-level. They just happen to
occur at the start of the file, especially in tests with very short
snippets.


529d81daca/crates/ruff_annotate_snippets/src/renderer/display_list.rs (L1204-L1208)

I broke up the snapshot commits at the end into several blocks, but I
don't think it's enough to help with review. The first few (notebooks,
syntax errors, and test rules) are small enough to look at, but I
couldn't really think of other categories beyond that. I'm happy to
break those up or pick out specific examples beyond what I have below,
if that would help.

The minimal code changes are in this
[range](abd28f1e77),
with the snapshot commits following. Moving the `FullRenderer` and
updating the `EmitterFlags` aren't strictly necessary either. I even
dropped the renderer commit this morning but figured it made sense to
keep it since we have the `full` module for tests. I don't feel strongly
either way.

## Test Plan

I did actually click through all 1700 snapshots individually instead of
accepting them all at once, although I moved through them quickly. There
are a
few main categories:

### Lint diagnostics

```diff
-unused.py:8:19: F401 [*] `pathlib` imported but unused
+F401 [*] `pathlib` imported but unused
+  --> unused.py:8:19
    |
  7 | # Unused, _not_ marked as required (due to the alias).
  8 | import pathlib as non_alias
-   |                   ^^^^^^^^^ F401
+   |                   ^^^^^^^^^
  9 |
 10 | # Unused, marked as required.
    |
-   = help: Remove unused import: `pathlib`
+help: Remove unused import: `pathlib`
```

- The filename and line numbers are moved to the second line
- The second noqa code next to the underline is removed

### Syntax errors

These are much like the above.

```diff
-    -:1:16: invalid-syntax: Expected one or more symbol names after import
+    invalid-syntax: Expected one or more symbol names after import
+     --> -:1:16
       |
     1 | from foo import
       |                ^
```

One thing I noticed while reviewing some of these, but I don't think is
strictly syntax-error-related, is that some of the new diagnostics have
a little less context after the error. I don't think this is a problem,
but it's one small discrepancy I hadn't noticed before. Here's a minor
example:

```diff
-syntax_errors.py:1:15: invalid-syntax: Expected one or more symbol names after import
+invalid-syntax: Expected one or more symbol names after import
+ --> syntax_errors.py:1:15
   |
 1 | from os import
   |               ^
 2 |
 3 | if call(foo
-4 |     def bar():
   |
```

And one of the biggest examples:

```diff
-E30_syntax_error.py:18:11: invalid-syntax: Expected ')', found newline
+invalid-syntax: Expected ')', found newline
+  --> E30_syntax_error.py:18:11
    |
 16 |         pass
 17 |
 18 | foo = Foo(
    |           ^
-19 |
-20 |
-21 | def top(
    |
```

Similarly, a few of the lint diagnostics showed that the cut indicator
calculation for overly long lines is also slightly different, but I
think that's okay too.

### Full-file diagnostics

```diff
-comment.py:1:1: I002 [*] Missing required import: `from __future__ import annotations`
+I002 [*] Missing required import: `from __future__ import annotations`
+--> comment.py:1:1
+help: Insert required import: `from __future__ import annotations`
+
```

As noted above, these will be much more rare after #19688 too. This case
isn't a true full-file diagnostic and will render a snippet in the
future, but you can see that we're now rendering the help message that
would have been discarded before. In contrast, this is a true full-file
diagnostic and should still look like this after #19688:

```diff
-__init__.py:1:1: A005 Module `logging` shadows a Python standard-library module
+A005 Module `logging` shadows a Python standard-library module
+--> __init__.py:1:1
```

### Jupyter notebooks

There's nothing particularly different about these, just showing off the
cell index again.

```diff
-    Jupyter.ipynb:cell 3:1:7: F821 Undefined name `x`
+    F821 Undefined name `x`
+     --> Jupyter.ipynb:cell 3:1:7
       |
     1 | print(x)
-      |       ^ F821
+      |       ^
       |
```
2025-08-08 12:56:23 -04:00
Brent Westbrook
8199154d54
[ty] Fix a few more diagnostic differences from Ruff (#19806)
## Summary

Fixes the remaining range reporting differences between the `ruff_db`
diagnostic rendering and Ruff's existing rendering, as noted in
https://github.com/astral-sh/ruff/pull/19415#issuecomment-3160525595.

This PR is structured as a series of three pairs. The first commit in
each pair adds a test showing the previous behavior, followed by a fix
and the updated snapshot. It's quite a small PR, but that might be
helpful just for the contrast.

You can also look at [this
range](052e656c6c..c3ea51030d)
of commits from #19415 to see the impact on real Ruff diagnostics. I
spun these commits out of that PR.

## Test Plan

New `ruff_db` tests
2025-08-08 11:31:19 -04:00
ember91
50e1ecc086
[pylint] Use lowercase hex characters to match the formatter (PLE2513) (#19808)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks-instrumented (push) Blocked by required conditions
CI / benchmarks-walltime (push) Blocked by required conditions
[ty Playground] Release / publish (push) Waiting to run
PLE2513 --fix changes ESC and SUB to uppercase hexadecimal values such
as \x1B while the formatter changes them to lowercase \x1b

<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
  requests.)
- Does this pull request include references to any relevant issues?
-->

## Summary

<!-- What's the purpose of the change? What does it do, and why? -->

## Test Plan

<!-- How was it tested? -->

---------

Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2025-08-08 12:25:11 +00:00
Micha Reiser
7dfde3b929
Update Rust toolchain to 1.89 (#19807) 2025-08-07 18:21:50 +02:00
Ibraheem Ahmed
21ac16db85
[ty] Avoid overcounting shared memory usage (#19773)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks-instrumented (push) Blocked by required conditions
CI / benchmarks-walltime (push) Blocked by required conditions
[ty Playground] Release / publish (push) Waiting to run
## Summary

Use a global tracker to avoid double counting `Arc` instances.
2025-08-06 15:32:02 -04:00
Brent Westbrook
5bfffe1aa7
[ty] Remap Jupyter notebook cell indices in ruff_db (#19698)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks-instrumented (push) Blocked by required conditions
CI / benchmarks-walltime (push) Blocked by required conditions
[ty Playground] Release / publish (push) Waiting to run
## Summary

This PR remaps ranges in Jupyter notebooks from simple `row:column`
indices in the concatenated source code to `cell:row:col` to match
Ruff's output. This is probably not a likely change to land upstream in
`annotate-snippets`, but I didn't see a good way around it.

The remapping logic is taken nearly verbatim from here:


cd6bf1457d/crates/ruff_linter/src/message/text.rs (L212-L222)


## Test Plan

New `full` rendering test for a notebook

I was mainly focused on Ruff, but in local tests this also works for ty:

```
error[invalid-assignment]: Object of type `Literal[1]` is not assignable to `str`
 --> Untitled.ipynb:cell 1:3:1
  |
1 | import math
2 |
3 | x: str = 1
  | ^
  |
info: rule `invalid-assignment` is enabled by default

error[invalid-assignment]: Object of type `Literal[1]` is not assignable to `str`
 --> Untitled.ipynb:cell 2:3:1
  |
1 | import math
2 |
3 | x: str = 1
  | ^
  |
info: rule `invalid-assignment` is enabled by default
```

This isn't a duplicate diagnostic, just an unimaginative example:

```py
# cell 1
import math

x: str = 1
# cell 2
import math

x: str = 1
```
2025-08-05 14:10:35 -04:00
Brent Westbrook
b324ae1be3
Hide empty snippets for full-file diagnostics (#19653)
Summary
--

This is the other commit I wanted to spin off from #19415, currently
stacked on #19644.

This PR suppresses blank snippets for empty ranges at the very beginning
of a file, and for empty ranges in non-existent files. Ruff includes
empty ranges for IO errors, for example.


f4e93b6335/crates/ruff_linter/src/message/text.rs (L100-L110)

The diagnostics now look like this (new snapshot test):

```
error[test-diagnostic]: main diagnostic message
--> example.py:1:1                             
```

Instead of [^*]

```
error[test-diagnostic]: main diagnostic message
--> example.py:1:1
 |
 |
```

Test Plan
--

A new `ruff_db` test showing the expected output format

[^*]: This doesn't correspond precisely to the example in the PR because
of some details of the diagnostic builder helper methods in `ruff_db`,
but you can see another example in the current version of the summary in
#19415.
2025-08-05 11:20:31 -04:00
Brent Westbrook
78e5fe0a51
Allow hiding the diagnostic severity in ruff_db (#19644)
## Summary

This PR is a spin-off from https://github.com/astral-sh/ruff/pull/19415.
It enables replacing the severity and lint name in a ty-style
diagnostic:

```
error[unused-import]: `os` imported but unused
```

with the noqa code and optional fix availability icon for a Ruff
diagnostic:

```
F401 [*] `os` imported but unused
F821 Undefined name `a`
```

or nothing at all for a Ruff syntax error:

```
SyntaxError: Expected one or more symbol names after import
```

Ruff adds the `SyntaxError` prefix to these messages manually.

Initially (d912458), I just passed a `hide_severity` flag through a
bunch of calls to get it into `annotate-snippets`, but after looking at
it again today, I think reusing the `None` severity/level gave a nicer
result. As I note in a lengthy code comment, I think all of this code
should be temporary and reverted when Ruff gets real severities, so
hopefully it's okay if it feels a little hacky.

I think the main visible downside of this approach is that we can't
style the asterisk in the fix availabilty icon in cyan, as in Ruff's
current output. It's part of the message in this PR and any styling gets
overwritten in `annotate-snippets`.

<img width="400" height="342" alt="image"
src="https://github.com/user-attachments/assets/57542ec9-a81c-4a01-91c7-bd6d7ec99f99"
/>

Hmm, I guess reusing `Level::None` also means the `F401` isn't red
anymore. Maybe my initial approach was better after all. In any case,
the rest of the PR should be basically the same, it just depends how we
want to toggle the severity.

## Test Plan

New `ruff_db` tests. These snapshots should be compared to the two tests
just above them (`hide_severity_output` vs `output` and
`hide_severity_syntax_errors` against `syntax_errors`).
2025-08-05 09:56:18 -04:00
Ibraheem Ahmed
8f8c39c435
Simplify get_size2 usage (#19643)
## Summary

These were added in the 0.5.0 release.
2025-07-30 15:31:37 -04:00
Brent Westbrook
8979271ea8
Always expand tabs to four spaces in diagnostics (#19618)
## Summary

I was a bit stuck on some snapshot differences I was seeing in #19415,
but @BurntSushi pointed out that `annotate-snippets` already normalizes
tabs on its own, which was very helpful! Instead of applying this change
directly to the other branch, I wanted to try applying it in
`ruff_linter` first. This should very slightly reduce the number of
changes in #19415 proper.

It looks like `annotate-snippets` always expands a tab to four spaces,
whereas I think we were aligning to tab stops:

```diff
  6 | spam(ham[1], { eggs: 2})
  7 | #: E201:1:6
- 8 | spam(   ham[1], {eggs: 2})
-   |      ^^^ E201
+ 8 | spam(    ham[1], {eggs: 2})
+   |      ^^^^ E201
```

```diff
61 | #: E203:2:15 E702:2:16
 62 | if x == 4:
-63 |     print(x, y) ; x, y = y, x
-   |                ^ E203
+63 |     print(x, y)    ; x, y = y, x
+   |                ^^^^ E203
```

```diff
 E27.py:15:6: E271 [*] Multiple spaces after keyword
    |
-13 | True        and False
+13 | True        and    False
 14 | #: E271
 15 | a and  b
    |      ^^ E271
```

I don't think this is too bad and has the major benefit of allowing us
to pass the non-tab-expanded range to `annotate-snippets` in #19415,
where it's also displayed in the header. Ruff doesn't have this problem
currently because it uses its own concise diagnostic output as the
header for full diagnostics, where the pre-expansion range is used
directly.

## Test Plan

Existing tests with a few snapshot updates
2025-07-30 11:00:36 -04:00
Micha Reiser
2a5ace6e55
[ty] Implement diagnostic caching (#19605) 2025-07-30 11:04:34 +01:00
Brent Westbrook
a54061e757
[ty] Fix empty spans following a line terminator and unprintable character spans in diagnostics (#19535)
## Summary

This was previously the last commit in #19415, split out to make it
easier to review. This applies the fixes from c9b99e4, 5021f32, and
2922490cb8 to the new rendering code in `ruff_db`. I initially intended
only to fix the empty span after a line terminator (as you can see in
the branch name), but the two fixes were tied pretty closely together,
and my initial fix for the empty spans needed a big change after trying
to handle unprintable characters too. I can still split this up if it
would help with review. I would just start with the unprintable
characters first.

The implementation here is essentially copy-pasted from
`ruff_linter::message::text.rs`, with the `SourceCode` struct renamed to
`EscapedSourceCode` since there's already a `SourceCode` in scope in
`render.rs`. It's also updated slightly to account for the multiple
annotations for a single snippet. The original implementation used some
types from the `line_width` module from `ruff_linter`. I copied over
heavily stripped-down versions of these instead of trying to import
them. We could inline the remaining code entirely, if we want, but I
thought it was nice enough to keep.

I also moved over `ceil_char_boundary`, which is unchanged except to
make it a free function taking a `&str` instead of a `Locator` method.
All of this code could be deleted from `ruff_linter` if we also move
over the `grouped` output format, which will be the last user after
#19415.

## Test Plan

I added new tests in `ruff_linter` that call into the new rendering code
to snapshot the diagnostics for the affected cases. These are copies of
existing snapshots in Ruff, so it's helpful to compare them. These are a
bit noisy because of the other rendering differences in the header, but
all of the `^^^` indicators should be the same.

<details><summary>`empty_span_after_line_terminator` diff</summary>

```diff
diff --git a/crates/ruff_linter/src/rules/pycodestyle/snapshots/ruff_linter__rules__pycodestyle__tests__E112_E11.py.snap b/crates/ruff_linter/src/message/snapshots/ruff_linter__message__text__tests__empty_span_after_line_terminator.snap
index 5ade4346e0..6df75c16f0 100644
--- a/crates/ruff_linter/src/rules/pycodestyle/snapshots/ruff_linter__rules__pycodestyle__tests__E112_E11.py.snap
+++ b/crates/ruff_linter/src/message/snapshots/ruff_linter__message__text__tests__empty_span_after_line_terminator.snap
@@ -1,17 +1,20 @@
 ---
-source: crates/ruff_linter/src/rules/pycodestyle/mod.rs
+source: crates/ruff_linter/src/message/text.rs
+expression: value.to_string()
 ---
-E11.py:9:1: E112 Expected an indented block
+error[no-indented-block]: Expected an indented block
+  --> E11.py:9:1
    |
  7 | #: E112
  8 | if False:
  9 | print()
-   | ^ E112
+   | ^
 10 | #: E113
 11 | print()
    |
 
-E11.py:9:1: SyntaxError: Expected an indented block after `if` statement
+error[invalid-syntax]: SyntaxError: Expected an indented block after `if` statement
+  --> E11.py:9:1
    |
  7 | #: E112
  8 | if False:
@@ -21,7 +24,8 @@ E11.py:9:1: SyntaxError: Expected an indented block after `if` statement
 11 | print()
    |
 
-E11.py:12:1: SyntaxError: Unexpected indentation
+error[invalid-syntax]: SyntaxError: Unexpected indentation
+  --> E11.py:12:1
    |
 10 | #: E113
 11 | print()
@@ -31,7 +35,8 @@ E11.py:12:1: SyntaxError: Unexpected indentation
 14 | mimetype = 'application/x-directory'
    |
 
-E11.py:14:1: SyntaxError: Expected a statement
+error[invalid-syntax]: SyntaxError: Expected a statement
+  --> E11.py:14:1
    |
 12 |     print()
 13 | #: E114 E116
@@ -41,17 +46,19 @@ E11.py:14:1: SyntaxError: Expected a statement
 16 | create_date = False
    |
 
-E11.py:45:1: E112 Expected an indented block
+error[no-indented-block]: Expected an indented block
+  --> E11.py:45:1
    |
 43 | #: E112
 44 | if False:  #
 45 | print()
-   | ^ E112
+   | ^
 46 | #:
 47 | if False:
    |
 
-E11.py:45:1: SyntaxError: Expected an indented block after `if` statement
+error[invalid-syntax]: SyntaxError: Expected an indented block after `if` statement
+  --> E11.py:45:1
    |
 43 | #: E112
 44 | if False:  #
```

</details>

<details><summary>`unprintable_characters` diff</summary>

```diff
diff --git a/crates/ruff_linter/src/rules/pylint/snapshots/ruff_linter__rules__pylint__tests__PLE2512_invalid_characters.py.snap b/crates/ruff_linter/src/message/snapshots/ruff_linter__message__text__tests__unprintable_characters.snap
index 52cfdf9cce..fcfa1ac9f1 100644
--- a/crates/ruff_linter/src/rules/pylint/snapshots/ruff_linter__rules__pylint__tests__PLE2512_invalid_characters.py.snap
+++ b/crates/ruff_linter/src/message/snapshots/ruff_linter__message__text__tests__unprintable_characters.snap
@@ -1,161 +1,115 @@
 ---
-source: crates/ruff_linter/src/rules/pylint/mod.rs
+source: crates/ruff_linter/src/message/text.rs
+expression: value.to_string()
 ---
-invalid_characters.py:24:12: PLE2512 [*] Invalid unescaped character SUB, use "\x1A" instead
+error[invalid-character-sub]: Invalid unescaped character SUB, use "\x1A" instead
+  --> invalid_characters.py:24:12
    |
 22 | cr_ok = f'\\r'
 23 |
 24 | sub = 'sub '
-   |            ^ PLE2512
+   |            ^
 25 | sub = f'sub '
    |
-   = help: Replace with escape sequence
+help: Replace with escape sequence
 
-ℹ Safe fix
-21 21 | cr_ok = '\\r'
-22 22 | cr_ok = f'\\r'
-23 23 | 
-24    |-sub = 'sub '
-   24 |+sub = 'sub \x1A'
-25 25 | sub = f'sub '
-26 26 | 
-27 27 | sub_ok = '\x1a'
-
-invalid_characters.py:25:13: PLE2512 [*] Invalid unescaped character SUB, use "\x1A" instead
+error[invalid-character-sub]: Invalid unescaped character SUB, use "\x1A" instead
+  --> invalid_characters.py:25:13
    |
 24 | sub = 'sub '
 25 | sub = f'sub '
-   |             ^ PLE2512
+   |             ^
 26 |
 27 | sub_ok = '\x1a'
    |
-   = help: Replace with escape sequence
-
-ℹ Safe fix
-22 22 | cr_ok = f'\\r'
-23 23 | 
-24 24 | sub = 'sub '
-25    |-sub = f'sub '
-   25 |+sub = f'sub \x1A'
-26 26 | 
-27 27 | sub_ok = '\x1a'
-28 28 | sub_ok = f'\x1a'
+help: Replace with escape sequence
 
-invalid_characters.py:55:25: PLE2512 [*] Invalid unescaped character SUB, use "\x1A" instead
+error[invalid-character-sub]: Invalid unescaped character SUB, use "\x1A" instead
+  --> invalid_characters.py:55:25
    |
 53 | zwsp_after_multicharacter_grapheme_cluster = f"ಫ್ರಾನ್ಸಿಸ್ಕೊ ​​"
 54 |
 55 | nested_fstrings = f'␈{f'{f'␛'}'}'
-   |                         ^ PLE2512
+   |                         ^
 56 |
 57 | # https://github.com/astral-sh/ruff/issues/7455#issuecomment-1741998106
    |
-   = help: Replace with escape sequence
-
-ℹ Safe fix
-52 52 | zwsp_after_multicharacter_grapheme_cluster = "ಫ್ರಾನ್ಸಿಸ್ಕೊ ​​"
-53 53 | zwsp_after_multicharacter_grapheme_cluster = f"ಫ್ರಾನ್ಸಿಸ್ಕೊ ​​"
-54 54 | 
-55    |-nested_fstrings = f'␈{f'{f'␛'}'}'
-   55 |+nested_fstrings = f'␈{f'\x1A{f'␛'}'}'
-56 56 | 
-57 57 | # https://github.com/astral-sh/ruff/issues/7455#issuecomment-1741998106
-58 58 | x = f"""}}ab"""
+help: Replace with escape sequence
 
-invalid_characters.py:58:12: PLE2512 [*] Invalid unescaped character SUB, use "\x1A" instead
+error[invalid-character-sub]: Invalid unescaped character SUB, use "\x1A" instead
+  --> invalid_characters.py:58:12
    |
 57 | # https://github.com/astral-sh/ruff/issues/7455#issuecomment-1741998106
 58 | x = f"""}}ab"""
-   |            ^ PLE2512
+   |            ^
 59 | # https://github.com/astral-sh/ruff/issues/7455#issuecomment-1741998256
 60 | x = f"""}}a␛b"""
    |
-   = help: Replace with escape sequence
+help: Replace with escape sequence
 
-ℹ Safe fix
-55 55 | nested_fstrings = f'␈{f'{f'␛'}'}'
-56 56 | 
-57 57 | # https://github.com/astral-sh/ruff/issues/7455#issuecomment-1741998106
-58    |-x = f"""}}ab"""
-   58 |+x = f"""}}a\x1Ab"""
-59 59 | # https://github.com/astral-sh/ruff/issues/7455#issuecomment-1741998256
-60 60 | x = f"""}}a␛b"""
-61 61 | 
-
-invalid_characters.py:64:12: PLE2512 Invalid unescaped character SUB, use "\x1A" instead
+error[invalid-character-sub]: Invalid unescaped character SUB, use "\x1A" instead
+  --> invalid_characters.py:64:12
    |
 63 | # https://github.com/astral-sh/ruff/issues/13294
 64 | print(r"""␈␛�​
-   |            ^ PLE2512
+   |            ^
 65 | """)
 66 | print(fr"""␈␛�​
    |
-   = help: Replace with escape sequence
+help: Replace with escape sequence
 
-invalid_characters.py:66:13: PLE2512 Invalid unescaped character SUB, use "\x1A" instead
+error[invalid-character-sub]: Invalid unescaped character SUB, use "\x1A" instead
+  --> invalid_characters.py:66:13
    |
 64 | print(r"""␈␛�​
 65 | """)
 66 | print(fr"""␈␛�​
-   |             ^ PLE2512
+   |             ^
 67 | """)
 68 | print(Rf"""␈␛�​
    |
-   = help: Replace with escape sequence
+help: Replace with escape sequence
 
-invalid_characters.py:68:13: PLE2512 Invalid unescaped character SUB, use "\x1A" instead
+error[invalid-character-sub]: Invalid unescaped character SUB, use "\x1A" instead
+  --> invalid_characters.py:68:13
    |
 66 | print(fr"""␈␛�​
 67 | """)
 68 | print(Rf"""␈␛�​
-   |             ^ PLE2512
+   |             ^
 69 | """)
    |
-   = help: Replace with escape sequence
+help: Replace with escape sequence
 
-invalid_characters.py:73:9: PLE2512 Invalid unescaped character SUB, use "\x1A" instead
+error[invalid-character-sub]: Invalid unescaped character SUB, use "\x1A" instead
+  --> invalid_characters.py:73:9
    |
 71 | # https://github.com/astral-sh/ruff/issues/18815
 72 | b = "\␈"
 73 | sub = "\"
-   |         ^ PLE2512
+   |         ^
 74 | esc = "\␛"
 75 | zwsp = "\​"
    |
-   = help: Replace with escape sequence
+help: Replace with escape sequence
 
-invalid_characters.py:80:25: PLE2512 [*] Invalid unescaped character SUB, use "\x1A" instead
+error[invalid-character-sub]: Invalid unescaped character SUB, use "\x1A" instead
+  --> invalid_characters.py:80:25
    |
 78 | # tstrings
 79 | esc = t'esc esc ␛'
 80 | nested_tstrings = t'␈{t'{t'␛'}'}'
-   |                         ^ PLE2512
+   |                         ^
 81 | nested_ftstrings = t'␈{f'{t'␛'}'}'
    |
-   = help: Replace with escape sequence
-
-ℹ Safe fix
-77 77 | 
-78 78 | # tstrings
-79 79 | esc = t'esc esc ␛'
-80    |-nested_tstrings = t'␈{t'{t'␛'}'}'
-   80 |+nested_tstrings = t'␈{t'\x1A{t'␛'}'}'
-81 81 | nested_ftstrings = t'␈{f'{t'␛'}'}'
-82 82 | 
+help: Replace with escape sequence
 
-invalid_characters.py:81:26: PLE2512 [*] Invalid unescaped character SUB, use "\x1A" instead
+error[invalid-character-sub]: Invalid unescaped character SUB, use "\x1A" instead
+  --> invalid_characters.py:81:26
    |
 79 | esc = t'esc esc ␛'
 80 | nested_tstrings = t'␈{t'{t'␛'}'}'
 81 | nested_ftstrings = t'␈{f'{t'␛'}'}'
-   |                          ^ PLE2512
+   |                          ^
    |
-   = help: Replace with escape sequence
-
-ℹ Safe fix
-78 78 | # tstrings
-79 79 | esc = t'esc esc ␛'
-80 80 | nested_tstrings = t'␈{t'{t'␛'}'}'
-81    |-nested_ftstrings = t'␈{f'{t'␛'}'}'
-   81 |+nested_ftstrings = t'␈{f'\x1A{t'␛'}'}'
-82 82 |
+help: Replace with escape sequence
```

</details>
2025-07-29 08:25:58 -04:00
Brent Westbrook
4daf59e5e7
Move concise diagnostic rendering to ruff_db (#19398)
## Summary

This PR moves most of the work of rendering concise diagnostics in Ruff
into `ruff_db`, where the code is shared with ty. To accomplish this
without breaking backwards compatibility in Ruff, there are two main
changes on the `ruff_db`/ty side:
- Added the logic from Ruff for remapping notebook line numbers to cells
- Reordered the fields in the diagnostic to match Ruff and rustc
  ```text
  # old
error[invalid-assignment] try.py:3:1: Object of type `Literal[1]` is not
assignable to `str`
  # new
try.py:3:1: error[invalid-assignment]: Object of type `Literal[1]` is
not assignable to `str`
  ```

I don't think the notebook change failed any tests on its own, and only
a handful of snaphots changed in ty after reordering the fields, but
this will obviously affect any other uses of the concise format, outside
of tests, too.

The other big change should only affect Ruff:

- Added three new `DisplayDiagnosticConfig` options
Micha and I hoped that we could get by with one option
(`hide_severity`), but Ruff also toggles `show_fix_status` itself,
independently (there are cases where we want neither severity nor the
fix status), and during the implementation I realized we also needed
access to an `Applicability`. The main goal here is to suppress the
severity (`error` above) because ruff only uses the `error` severity and
to use the secondary/noqa code instead of the line name
(`invalid-assignment` above).
  ```text
  # ty - same as "new" above
try.py:3:1: error[invalid-assignment]: Object of type `Literal[1]` is
not assignable to `str`
  # ruff
try.py:3:1: RUF123 [*] Object of type `Literal[1]` is not assignable to
`str`
  ```

This part of the concise diagnostic is actually shared with the `full`
output format in Ruff, but with the settings above, there are no
snapshot changes to either format.

## Test Plan

Existing tests with the handful of updates mentioned above, as well as
some new tests in the `concise` module.

Also this PR. Swapping the fields might have broken mypy_primer, unless
it occasionally times out on its own.

I also ran this script in the root of my Ruff checkout, which also has
CPython in it:

```shell
flags=(--isolated --no-cache --no-respect-gitignore --output-format concise .)
diff <(target/release/ruff check ${flags[@]} 2> /dev/null) \
     <(ruff check ${flags[@]} 2> /dev/null)
```

This yielded an expected diff due to some t-string error changes on main
since 0.12.4:
```diff
33622c33622
< crates/ruff_python_parser/resources/inline/err/f_string_lambda_without_parentheses.py:1:15: SyntaxError: Expected an element of or the end of the f-string
---
> crates/ruff_python_parser/resources/inline/err/f_string_lambda_without_parentheses.py:1:15: SyntaxError: Expected an f-string or t-string element or the end of the f-string or t-string
33742c33742
< crates/ruff_python_parser/resources/inline/err/implicitly_concatenated_unterminated_string_multiline.py:4:1: SyntaxError: Expected an element of or the end of the f-string
---
> crates/ruff_python_parser/resources/inline/err/implicitly_concatenated_unterminated_string_multiline.py:4:1: SyntaxError: Expected an f-string or t-string element or the end of the f-string or t-string
34131c34131
< crates/ruff_python_parser/resources/inline/err/t_string_lambda_without_parentheses.py:2:15: SyntaxError: Expected an element of or the end of the t-string
---
> crates/ruff_python_parser/resources/inline/err/t_string_lambda_without_parentheses.py:2:15: SyntaxError: Expected an f-string or t-string element or the end of the f-string or t-string
```

So modulo color, the results are identical on 38,186 errors in our test
suite and CPython 3.10.

---------

Co-authored-by: David Peter <mail@david-peter.de>
2025-07-23 11:43:32 -04:00
Brent Westbrook
fd335eb8b7
Move fix suggestion to subdiagnostic (#19464)
Summary
--

This PR tweaks Ruff's internal usage of the new diagnostic model to more
closely
match the intended use, as I understand it. Specifically, it moves the
fix/help
suggestion from the primary annotation's message to a subdiagnostic. In
turn, it
adds the secondary/noqa code as the new primary annotation message. As
shown in
the new `ruff_db` tests, this more closely mirrors Ruff's current
diagnostic
output.

I also added `Severity::Help` to render the fix suggestion with a
`help:` prefix
instead of `info:`.

These changes don't have any external impact now but should help a bit
with #19415.

Test Plan
--

New full output format tests in `ruff_db`

Rendered Diagnostics
--

Full diagnostic output from `annotate-snippets` in this PR:

``` 
error[unused-import]: `os` imported but unused
  --> fib.py:1:8
   |
 1 | import os
   |        ^^
   |
 help: Remove unused import: `os`
```

Current Ruff output for the same code:

```
fib.py:1:8: F401 [*] `os` imported but unused
  |
1 | import os
  |        ^^ F401
  |
  = help: Remove unused import: `os`
```

Proposed final output after #19415:

``` 
F401 [*] `os` imported but unused
  --> fib.py:1:8
   |
 1 | import os
   |        ^^
   |
 help: Remove unused import: `os`
```

These are slightly updated from
https://github.com/astral-sh/ruff/pull/19464#issuecomment-3097377634
below to remove the extra noqa codes in the primary annotation messages
for the first and third cases.
2025-07-22 10:03:58 -04:00
Micha Reiser
5e29278aa2
[ty] Reduce size of TypeInference (#19435) 2025-07-22 11:36:36 +02:00
Andrew Gallant
64f9481fd0
[ty] Add caching for submodule completion suggestions (#19408)
This change makes it so we aren't doing a directory traversal every time
we ask for completions from a module. Specifically, submodules that
aren't attributes of their parent module can only be discovered by
looking at the directory tree. But we want to avoid doing a directory
scan unless we think there are changes.

To make this work, this change does a little bit of surgery to
`FileRoot`. Previously, a `FileRoot` was only used for library search
paths. Its revision was bumped whenever a file in that tree was added,
deleted or even modified (to support the discovery of `pth` files and
changes to its contents). This generally seems fine since these are
presumably dependency paths that shouldn't change frequently.

In this change, we add a `FileRoot` for the project. But having the
`FileRoot`'s revision bumped for every change in the project makes
caching based on that `FileRoot` rather ineffective. That is, cache
invalidation will occur too aggressively. To the point that there is
little point in adding caching in the first place. To mitigate this, a
`FileRoot`'s revision is only bumped on a change to a child file's
contents when the `FileRoot` is a `LibrarySearchPath`. Otherwise, we
only bump the revision when a file is created or added.

The effect is that, at least in VS Code, when a new module is added or
removed, this change is picked up and the cache is properly invalidated.
Other LSP clients with worse support for file watching (which seems to
be the case for the CoC vim plugin that I use) don't work as well. Here,
the cache is less likely to be invalidated which might cause completions
to have stale results. Unless there's an obvious way to fix or improve
this, I propose punting on improvements here for now.
2025-07-18 11:54:27 -04:00
Dhruv Manilawala
99d0ac60b4
[ty] Track open files in the server (#19264)
## Summary

This PR updates the server to keep track of open files both system and
virtual files.

This is done by updating the project by adding the file in the open file
set in `didOpen` notification and removing it in `didClose`
notification.

This does mean that for workspace diagnostics, ty will only check open
files because the behavior of different diagnostic builder is to first
check `is_file_open` and only add diagnostics for open files. So, this
required updating the `is_file_open` model to be `should_check_file`
model which validates whether the file needs to be checked based on the
`CheckMode`. If the check mode is open files only then it will check
whether the file is open. If it's all files then it'll return `true` by
default.

Closes: astral-sh/ty#619

## Test Plan

### Before

There are two files in the project: `__init__.py` and `diagnostics.py`.

In the video, I'm demonstrating the old behavior where making changes to
the (open) `diagnostics.py` file results in re-parsing the file:


https://github.com/user-attachments/assets/c2ac0ecd-9c77-42af-a924-c3744b146045

### After

Same setup as above.

In the video, I'm demonstrating the new behavior where making changes to
the (open) `diagnostics.py` file doesn't result in re-parting the file:


https://github.com/user-attachments/assets/7b82fe92-f330-44c7-b527-c841c4545f8f
2025-07-18 19:33:35 +05:30
Andrew Gallant
ba7ed3a6f9
[ty] Use as the "cut" indicator in diagnostic rendering (#19420)
This makes ty match ruff's behavior. Specifically, we want to use `…`
instead of the default `...` because `...` has special significance in
Python.
2025-07-18 07:46:48 -04:00
Brent Westbrook
997dc2e7cc
Move JUnit rendering to ruff_db (#19370)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks-instrumented (push) Blocked by required conditions
CI / benchmarks-walltime (push) Blocked by required conditions
[ty Playground] Release / publish (push) Waiting to run
Summary
--

This PR moves the JUnit output format to the new rendering
infrastructure. As I
mention in a TODO in the code, there's some code that will be shared
with the
`grouped` output format. Hopefully I'll have that PR up too by the time
this one
is reviewed.

Test Plan
--

Existing tests moved to `ruff_db`

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2025-07-17 18:24:13 -04:00
Brent Westbrook
e9cac3684a
Move Pylint rendering to ruff_db (#19340)
Summary
--

This is a very simple output format, the only decision is what to do if
the file
is missing from the diagnostic. For now, I opted to `unwrap_or_default`
both the
path and the `OneIndexed` row number, giving `:1: main diagnostic
message` in
the test without a file.

Another quirk here is that the path is relativized. I just pasted in the
`relativize_path` and `get_cwd` implementations from `ruff_linter::fs`
for now,
but maybe there's a better place for them.

I didn't see any details about why this needs to be relativized in the
original
[issue](https://github.com/astral-sh/ruff/issues/1953),
[PR](https://github.com/astral-sh/ruff/pull/1995), or in the pylint

[docs](https://flake8.pycqa.org/en/latest/internal/formatters.html#pylint-formatter),
but it did change the results of the CLI integration test when I tried
deleting
it. I haven't been able to reproduce that in the CLI, though, so it may
only
happen with `Command::current_dir`.

Test Plan
--

Tests ported from `ruff_linter` and a new test for the case with no file

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2025-07-15 10:14:49 -04:00
Brent Westbrook
e9b0c33703
Move RDJSON rendering to ruff_db (#19293)
## Summary

Another output format like #19133. This is the
[reviewdog](https://github.com/reviewdog/reviewdog) output format, which
is somewhat similar to regular JSON. Like #19270, in the first commit I
converted from using `json!` to `Serialize` structs, then in the second
commit I moved the module to `ruff_db`.

The reviewdog
[schema](320a8e73a9/proto/rdf/jsonschema/DiagnosticResult.json)
seems a bit more flexible than our JSON schema, so I'm not sure if we
need any preview checks here. I'll flag the places I wasn't sure about
as review comments.

## Test Plan

New tests in `rdjson.rs`, ported from the old `rjdson.rs` module, as
well as the new CLI output tests.

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2025-07-15 12:39:21 +00:00
Micha Reiser
3da8b51dc1
[ty] Fix server version (#19284) 2025-07-14 09:06:34 +02:00
Brent Westbrook
b5c5f710fc
Render Azure, JSON, and JSON lines output with the new diagnostics (#19133)
## Summary

This was originally stacked on #19129, but some of the changes I made
for JSON also impacted the Azure format, so I went ahead and combined
them. The main changes here are:

- Implementing `FileResolver` for Ruff's `EmitterContext`
- Adding `FileResolver::notebook_index` and `FileResolver::is_notebook`
methods
- Adding a `DisplayDiagnostics` (with an "s") type for rendering a group
of diagnostics at once
- Adding `Azure`, `Json`, and `JsonLines` as new `DiagnosticFormat`s

I tried a couple of alternatives to the `FileResolver::notebook` methods
like passing down the `NotebookIndex` separately and trying to reparse a
`Notebook` from Ruff's `SourceFile`. The latter seemed promising, but
the `SourceFile` only stores the concatenated plain text of the
notebook, not the re-parsable JSON. I guess the current version is just
a variation on passing the `NotebookIndex`, but at least we can reuse
the existing `resolver` argument. I think a lot of this can be cleaned
up once Ruff has its own actual file resolver.

As suggested, I also tried deleting the corresponding `Emitter` files in
`ruff_linter`, but it doesn't look like git was able to follow this as a
rename. It did, however, track that the tests were moved, so the
snapshots should be easy to review.

## Test Plan

Existing Ruff tests ported to tests in `ruff_db`. I think some other
existing ruff tests also cover parts of this refactor.

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2025-07-11 15:04:46 -04:00
Andrew Gallant
100d765ddf [ty] Document path separator usage in VendoredFileSystem
Ref https://github.com/astral-sh/ruff/pull/19266#discussion_r2198530383
2025-07-11 10:06:35 -04:00
Andrew Gallant
948463aafa [ty] Move SystemOrVendoredPathRef
This moves the type and adds a few methods so that it can
be used elsewhere.
2025-07-11 10:06:35 -04:00
Andrew Gallant
729fa12575 [ty] Add "readdir" for vendored file systems
This is mostly just holding a zip file in the right way
to simulate reading a directory. We want this to be able
to discover sub-modules for completions.
2025-07-11 10:06:35 -04:00
Brent Westbrook
f14ee9edd5
Use structs for JSON serialization (#19270)
## Summary

See https://github.com/astral-sh/ruff/pull/19133#discussion_r2198413586
for recent discussion. This PR moves to using structs for the types in
our JSON output format instead of the `json!` macro.

I didn't rename any of the `message` references because that should be
handled when rebasing #19133 onto this.

My plan for handling the `preview` behavior with the new diagnostics is
to use a wrapper enum. Something like:

```rust
#[derive(Serialize)]
#[serde(untagged)]
pub(crate) enum JsonDiagnostic<'a> {
    Old(OldJsonDiagnostic<'a>),
}

#[derive(Serialize)]
pub(crate) struct OldJsonDiagnostic<'a> {
    // ...
}
```

Initially I thought I could use a `&dyn Serialize` for the affected
fields, but I see that `Serialize` isn't dyn-compatible in testing this
now.

## Test Plan

Existing tests. One quirk of the new types is that their fields are in
alphabetical order. I guess `json!` sorts the fields alphabetically? The
tests were failing before I sorted the struct fields.

## Other formats

It looks like the `rdjson`, `sarif`, and `gitlab` formats also use
`json!`, so if we decide to merge this, I can do something similar for
those before moving them to the new diagnostic format.
2025-07-11 09:37:44 -04:00
Charlie Marsh
3ee3434187
Auto-generate environment variable references for ty (#19205)
## Summary

This PR mirrors the environment variable implementation we have in uv:
efc361223c/crates/uv-static/src/env_vars.rs (L6-L7).

See: https://github.com/astral-sh/ty/issues/773.
2025-07-08 10:48:31 -04:00
Brent Westbrook
2643dc5b7a
Rename Diagnostic::syntax_error methods, separate Ord implementation (#19179)
## Summary

This PR addresses some additional feedback on #19053:

- Renaming the `syntax_error` methods to `invalid_syntax` to match the
lint id
- Moving the standalone `diagnostic_from_violation` function to
`Violation::into_diagnostic`
- Removing the `Ord` and `PartialOrd` implementations from `Diagnostic`
in favor of `Diagnostic::start_ordering`

## Test Plan

Existing tests

## Additional Follow-ups

Besides these, I also put the following comments on my todo list, but
they seemed like they might be big enough to have their own PRs:

- [Use `LintId::IOError` for IO
errors](https://github.com/astral-sh/ruff/pull/19053#discussion_r2189425922)
- [Move `Fix` and
`Edit`](https://github.com/astral-sh/ruff/pull/19053#discussion_r2189448647)
- [Avoid so many
unwraps](https://github.com/astral-sh/ruff/pull/19053#discussion_r2189465980)
2025-07-08 09:54:19 -04:00
Brent Westbrook
77a5c5ac80
Combine OldDiagnostic and Diagnostic (#19053)
## Summary

This PR is a collaboration with @AlexWaygood from our pairing session
last Friday.

The main goal here is removing `ruff_linter::message::OldDiagnostic` in
favor of
using `ruff_db::diagnostic::Diagnostic` directly. This involved a few
major steps:

- Transferring the fields
- Transferring the methods and trait implementations, where possible
- Converting some constructor methods to free functions
- Moving the `SecondaryCode` struct
- Updating the method names

I'm hoping that some of the methods, especially those in the
`expect_ruff_*`
family, won't be necessary long-term, but I avoided trying to replace
them
entirely for now to keep the already-large diff a bit smaller.

### Related refactors

Alex and I noticed a few refactoring opportunities while looking at the
code,
specifically the very similar implementations for
`create_parse_diagnostic`,
`create_unsupported_syntax_diagnostic`, and
`create_semantic_syntax_diagnostic`.
We combined these into a single generic function, which I then copied
into
`ruff_linter::message` with some small changes and a TODO to combine
them in the
future.

I also deleted the `DisplayParseErrorType` and `TruncateAtNewline` types
for
reporting parse errors. These were added in #4124, I believe to work
around the
error messages from LALRPOP. Removing these didn't affect any tests, so
I think
they were unnecessary now that we fully control the error messages from
the
parser.

On a more minor note, I factored out some calls to the
`OldDiagnostic::filename`
(now `Diagnostic::expect_ruff_filename`) function to avoid repeatedly
allocating
`String`s in some places.

### Snapshot changes

The `show_statistics_syntax_errors` integration test changed because the
`OldDiagnostic::name` method used `syntax-error` instead of
`invalid-syntax`
like in ty. I think this (`--statistics`) is one of the only places we
actually
use this name for syntax errors, so I hope this is okay. An alternative
is to
use `syntax-error` in ty too.

The other snapshot changes are from removing this code, as discussed on

[Discord](1388252408):


34052a1185/crates/ruff_linter/src/message/mod.rs (L128-L135)

I think both of these are technically breaking changes, but they only
affect
syntax errors and are very narrow in scope, while also pretty
substantially
simplifying the refactor, so I hope they're okay to include in a patch
release.

## Test plan

Existing tests, with the adjustments mentioned above

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2025-07-03 13:01:09 -04:00
Ibraheem Ahmed
ebc70a4002
[ty] Support LSP go-to with vendored typeshed stubs (#19057)
## Summary

Extracts the vendored typeshed stubs lazily and caches them on the local
filesystem to support go-to in the LSP.

Resolves https://github.com/astral-sh/ty/issues/77.
2025-07-02 07:58:58 -04:00
Micha Reiser
f7fc8fb084
[ty] Request configuration from client (#18984)
Some checks are pending
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks-instrumented (push) Blocked by required conditions
CI / benchmarks-walltime (push) Blocked by required conditions
[ty Playground] Release / publish (push) Waiting to run
## Summary

This PR makes the necessary changes to the server that it can request
configurations from the client using the `configuration` request.
This PR doesn't make use of the request yet. It only sets up the
foundation (mainly the coordination between client and server)
so that future PRs could pull specific settings. 

I plan to use this for pulling the Python environment from the Python
extension.

Deno does something very similar to this.

## Test Plan

Tested that diagnostics are still shown.
2025-07-02 14:31:41 +05:30
Micha Reiser
29927f2b59
Update Rust toolchain to 1.88 and MSRV to 1.86 (#19011) 2025-06-28 20:24:00 +02:00
Ibraheem Ahmed
6f7b1c9bb3
[ty] Add environment variable to dump Salsa memory usage stats (#18928)
## Summary

Setting `TY_MEMORY_REPORT=full` will generate and print a memory usage
report to the CLI after a `ty check` run:

```
=======SALSA STRUCTS=======
`Definition`                                       metadata=7.24MB   fields=17.38MB  count=181062
`Expression`                                       metadata=4.45MB   fields=5.94MB   count=92804
`member_lookup_with_policy_::interned_arguments`   metadata=1.97MB   fields=2.25MB   count=35176
...
=======SALSA QUERIES=======
`File -> ty_python_semantic::semantic_index::SemanticIndex`
    metadata=11.46MB  fields=88.86MB  count=1638
`Definition -> ty_python_semantic::types::infer::TypeInference`
    metadata=24.52MB  fields=86.68MB  count=146018
`File -> ruff_db::parsed::ParsedModule`
    metadata=0.12MB   fields=69.06MB  count=1642
...
=======SALSA SUMMARY=======
TOTAL MEMORY USAGE: 577.61MB
    struct metadata = 29.00MB
    struct fields = 35.68MB
    memo metadata = 103.87MB
    memo fields = 409.06MB
```

Eventually, we should integrate these numbers into CI in some form. The
one limitation currently is that heap allocations in salsa structs (e.g.
interned values) are not tracked, but memoized values should have full
coverage. We may also want a peak memory usage counter (that accounts
for non-salsa memory), but that is relatively simple to profile manually
(e.g. `time -v ty check`) and would require a compile-time option to
avoid runtime overhead.
2025-06-26 21:27:51 +00:00
Micha Reiser
76387295a5
[ty] Move venv and conda env discovery to SearchPath::from_settings (#18938)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2025-06-26 16:39:27 +02:00
Micha Reiser
0194452928
[ty] Rename src.root setting to environment.root (#18760)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2025-06-24 14:40:44 +02:00
Micha Reiser
23261a38a0
[ty] Add more benchmarks (#18714)
Some checks are pending
CI / benchmarks-instrumented (push) Blocked by required conditions
CI / benchmarks-walltime (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
[ty Playground] Release / publish (push) Waiting to run
2025-06-18 13:41:38 +02:00
Micha Reiser
3a430fa6da
[ty] Allow overriding rules for specific files (#18648)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[ty Playground] Release / publish (push) Waiting to run
2025-06-15 14:27:39 +01:00
Ibraheem Ahmed
c9dff5c7d5
[ty] AST garbage collection (#18482)
## Summary

Garbage collect ASTs once we are done checking a given file. Queries
with a cross-file dependency on the AST will reparse the file on demand.
This reduces ty's peak memory usage by ~20-30%.

The primary change of this PR is adding a `node_index` field to every
AST node, that is assigned by the parser. `ParsedModule` can use this to
create a flat index of AST nodes any time the file is parsed (or
reparsed). This allows `AstNodeRef` to simply index into the current
instance of the `ParsedModule`, instead of storing a pointer directly.

The indices are somewhat hackily (using an atomic integer) assigned by
the `parsed_module` query instead of by the parser directly. Assigning
the indices in source-order in the (recursive) parser turns out to be
difficult, and collecting the nodes during semantic indexing is
impossible as `SemanticIndex` does not hold onto a specific
`ParsedModuleRef`, which the pointers in the flat AST are tied to. This
means that we have to do an extra AST traversal to assign and collect
the nodes into a flat index, but the small performance impact (~3% on
cold runs) seems worth it for the memory savings.

Part of https://github.com/astral-sh/ty/issues/214.
2025-06-13 08:40:11 -04:00
Micha Reiser
1f27d53fd5
[ty] File inclusion and exclusion (#18498) 2025-06-12 19:07:31 +02:00
renovate[bot]
475a02b725
Update pre-commit dependencies (#18581)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions
[ty Playground] Release / publish (push) Waiting to run
2025-06-09 08:08:17 +02:00
Micha Reiser
86e5a311f0
[ty] Introduce and use System::env_var for better test isolation (#18538)
Some checks failed
CI / cargo test (linux) (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / benchmarks (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
[ty Playground] Release / publish (push) Has been cancelled
2025-06-07 19:56:58 +02:00
Ibraheem Ahmed
8531f4b3ca
[ty] Add infrastructure for AST garbage collection (#18445)
## Summary

https://github.com/astral-sh/ty/issues/214 will require a couple
invasive changes that I would like to get merged even before garbage
collection is fully implemented (to avoid rebasing):
- `ParsedModule` can no longer be dereferenced directly. Instead you
need to load a `ParsedModuleRef` to access the AST, which requires a
reference to the salsa database (as it may require re-parsing the AST if
it was collected).
- `AstNodeRef` can only be dereferenced with the `node` method, which
takes a reference to the `ParsedModuleRef`. This allows us to encode the
fact that ASTs do not live as long as the database and may be collected
as soon a given instance of a `ParsedModuleRef` is dropped. There are a
number of places where we currently merge the `'db` and `'ast`
lifetimes, so this requires giving some types/functions two separate
lifetime parameters.
2025-06-05 11:43:18 -04:00
Micha Reiser
8005ebb405
Update salsa past generational id change (#18362) 2025-05-30 15:31:33 +02:00
Alex Waygood
a5ebb3f3a2
[ty] Support ephemeral uv virtual environments (#18335) 2025-05-28 14:54:59 +00:00
Micha Reiser
d8216fa328
[ty] Gracefully handle salsa cancellations and panics in background request handlers (#18254) 2025-05-26 13:37:49 +01:00