Use re-lexing for normal list parsing (#11871)

## Summary

This PR is a follow-up on #11845 to add the re-lexing logic for normal
list parsing.

A normal list parsing is basically parsing elements without any
separator in between i.e., there can only be trivia tokens in between
the two elements. Currently, this is only being used for parsing
**assignment statement** and **f-string elements**. Assignment
statements cannot be in a parenthesized context, but f-string can have
curly braces so this PR is specifically for them.

I don't think this is an ideal recovery but the problem is that both
lexer and parser could add an error for f-strings. If the lexer adds an
error it'll emit an `Unknown` token instead while the parser adds the
error directly. I think we'd need to move all f-string errors to be
emitted by the parser instead. This way the parser can correctly inform
the lexer that it's out of an f-string and then the lexer can pop the
current f-string context out of the stack.

## Test Plan

Add test cases, update the snapshots, and run the fuzzer.
This commit is contained in:
Dhruv Manilawala 2024-06-18 12:14:41 +05:30 committed by GitHub
parent 5e5a81b05f
commit 1e0642fac8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 614 additions and 322 deletions

View file

@ -42,5 +42,16 @@ if call(foo, [a, b)
if call(foo, [a,
b
)
def bar():
pass
# F-strings uses normal list parsing, so test those as well
if call(f"hello {x
def bar():
pass
if call(f"hello
def bar():
pass