Avoid moving back the lexer for triple-quoted fstring (#11939)

## Summary

This PR avoids moving back the lexer for a triple-quoted f-string during
the re-lexing phase.

The reason this is a problem is that for a triple-quoted f-string the
newlines are part of the f-string itself, specifically they'll be part
of the `FStringMiddle` token. So, if we moved the lexer back, there
would be a `Newline` token whose range would be in between an
`FStringMiddle` token. This creates a panic in downstream usage.

fixes: #11937 

## Test Plan

Add test cases and validate the snapshots.
This commit is contained in:
Dhruv Manilawala 2024-06-20 16:27:36 +05:30 committed by GitHub
parent 22733cb7c7
commit ed948eaefb
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 380 additions and 159 deletions

View file

@ -0,0 +1,6 @@
# There are trailing whitespace before the newline character but those whitespaces are
# part of the comment token.
# https://github.com/astral-sh/ruff/issues/11929
f"""hello {x # comment
y = 1

View file

@ -0,0 +1,6 @@
# The lexer can't be moved back for a triple-quoted f-string because the newlines are
# part of the f-string itself.
# https://github.com/astral-sh/ruff/issues/11937
f'''{foo:.3f
'''

View file

@ -0,0 +1,7 @@
# Here, the nesting level is 2 when the parser is trying to recover from an unclosed `{`
# This test demonstrates that we need to reduce the nesting level when recovering from
# within an f-string but the lexer shouldn't go back.
if call(f'''{x:.3f
'''
pass