language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-07-09 22:25:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Enable token-based rules on source with syntax errors (#11950)

Browse source 
## Summary

This PR updates the linter, specifically the token-based rules, to work
on the tokens that come after a syntax error.

For context, the token-based rules only diagnose the tokens up to the
first lexical error. This PR builds up an error resilience by
introducing a `TokenIterWithContext` which updates the `nesting` level
and tries to reflect it with what the lexer is seeing. This isn't 100%
accurate because if the parser recovered from an unclosed parenthesis in
the middle of the line, the context won't reduce the nesting level until
it sees the newline token at the end of the line.

resolves: #11915

## Test Plan

* Add test cases for a bunch of rules that are affected by this change.
* Run the fuzzer for a long time, making sure to fix any other bugs.
This commit is contained in:
Dhruv Manilawala 2024-07-02 14:27:46 +05:30 • committed by GitHub
parent 88a4cc41f7
commit 8f40928534
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
29 changed files with 916 additions and 153 deletions
Download patch file Download diff file Expand all files Collapse all files

12
crates/ruff_python_index/src/indexer.rs
View file

@ -39,7 +39,7 @@ impl Indexer {
let mut prev_end = TextSize::default();
let mut line_start = TextSize::default();
for token in tokens.up_to_first_unknown() {
for token in tokens {
let trivia = locator.slice(TextRange::new(prev_end, token.start()));
// Get the trivia between the previous and the current token and detect any newlines.
@ -80,16 +80,6 @@ impl Indexer {
prev_end = token.end();
}
// TODO(dhruvmanila): This is temporary until Ruff becomes error resilient. To understand
// why this is required, refer to https://github.com/astral-sh/ruff/pull/11457#issuecomment-2144990269
// which was released at the time of this writing. Now we can't just revert that behavior,
// so we need to visit the remaining tokens if there are any for the comment ranges.
for token in tokens.after(prev_end) {
if token.kind() == TokenKind::Comment {
comment_ranges.push(token.range());
}
}
Self {
continuation_lines,
fstring_ranges: fstring_ranges_builder.finish(),