language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-08-24 04:14:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Use re-lexing for normal list parsing (#11871)

Browse source 
## Summary

This PR is a follow-up on #11845 to add the re-lexing logic for normal
list parsing.

A normal list parsing is basically parsing elements without any
separator in between i.e., there can only be trivia tokens in between
the two elements. Currently, this is only being used for parsing
**assignment statement** and **f-string elements**. Assignment
statements cannot be in a parenthesized context, but f-string can have
curly braces so this PR is specifically for them.

I don't think this is an ideal recovery but the problem is that both
lexer and parser could add an error for f-strings. If the lexer adds an
error it'll emit an `Unknown` token instead while the parser adds the
error directly. I think we'd need to move all f-string errors to be
emitted by the parser instead. This way the parser can correctly inform
the lexer that it's out of an f-string and then the lexer can pop the
current f-string context out of the stack.

## Test Plan

Add test cases, update the snapshots, and run the fuzzer.
This commit is contained in:
Dhruv Manilawala 2024-06-18 12:14:41 +05:30 • committed by GitHub
parent 5e5a81b05f
commit 1e0642fac8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 614 additions and 322 deletions
Download patch file Download diff file Expand all files Collapse all files

19
crates/ruff_python_parser/src/parser/mod.rs
View file

@ -475,21 +475,22 @@ impl<'src> Parser<'src> {
if recovery_context_kind.is_list_element(self) {
parse_element(self);
} else if recovery_context_kind.is_list_terminator(self) {
} else if recovery_context_kind.is_regular_list_terminator(self) {
break;
} else {
// Not a recognised element. Add an error and either skip the token or break
// parsing the list if the token is recognised as an element or terminator of an
// enclosing list.
let error = recovery_context_kind.create_error(self);
self.add_error(error, self.current_token_range());
// Run the error recovery: This also handles the case when an element is missing
// between two commas: `a,,b`
// Run the error recovery: If the token is recognised as an element or terminator
// of an enclosing list, then we try to re-lex in the context of a logical line and
// break out of list parsing.
if self.is_enclosing_list_element_or_terminator() {
self.tokens.re_lex_logical_token();
break;
}
self.add_error(
recovery_context_kind.create_error(self),
self.current_token_range(),
);
self.bump_any();
}
}