ruff/crates/ruff_python_parser/src
Dhruv Manilawala 8f40928534
Enable token-based rules on source with syntax errors (#11950)
## Summary

This PR updates the linter, specifically the token-based rules, to work
on the tokens that come after a syntax error.

For context, the token-based rules only diagnose the tokens up to the
first lexical error. This PR builds up an error resilience by
introducing a `TokenIterWithContext` which updates the `nesting` level
and tries to reflect it with what the lexer is seeing. This isn't 100%
accurate because if the parser recovered from an unclosed parenthesis in
the middle of the line, the context won't reduce the nesting level until
it sees the newline token at the end of the line.

resolves: #11915

## Test Plan

* Add test cases for a bunch of rules that are affected by this change.
* Run the fuzzer for a long time, making sure to fix any other bugs.
2024-07-02 08:57:46 +00:00
..
lexer Mention that Cursor is based on rustc's implementation. (#12109) 2024-06-30 16:53:25 +01:00
parser Use CompactString for Identifier (#12101) 2024-07-01 10:06:02 +02:00
snapshots Use CompactString for Identifier (#12101) 2024-07-01 10:06:02 +02:00
error.rs Revert "Use correct range to highlight line continuation error" (#12089) 2024-06-28 18:10:00 +05:30
lexer.rs Use CompactString for Identifier (#12101) 2024-07-01 10:06:02 +02:00
lib.rs Enable token-based rules on source with syntax errors (#11950) 2024-07-02 08:57:46 +00:00
string.rs Move token and error structs into related modules (#11957) 2024-06-21 10:07:19 +00:00
token.rs Use CompactString for Identifier (#12101) 2024-07-01 10:06:02 +02:00
token_set.rs Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
token_source.rs Use TokenSource to find new location for re-lexing (#12060) 2024-06-27 17:12:39 +05:30
typing.rs Update type annotation parsing API to return Parsed (#11739) 2024-06-05 12:59:43 +05:30