Avoid moving back the lexer for triple-quoted fstring (#11939)

## Summary

This PR avoids moving back the lexer for a triple-quoted f-string during
the re-lexing phase.

The reason this is a problem is that for a triple-quoted f-string the
newlines are part of the f-string itself, specifically they'll be part
of the `FStringMiddle` token. So, if we moved the lexer back, there
would be a `Newline` token whose range would be in between an
`FStringMiddle` token. This creates a panic in downstream usage.

fixes: #11937 

## Test Plan

Add test cases and validate the snapshots.
This commit is contained in:
Dhruv Manilawala 2024-06-20 16:27:36 +05:30 committed by GitHub
parent 22733cb7c7
commit ed948eaefb
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 380 additions and 159 deletions

View file

@ -1370,6 +1370,12 @@ impl<'src> Lexer<'src> {
// i.e., it recovered from an unclosed parenthesis (`(`, `[`, or `{`).
self.nesting -= 1;
// The lexer can't be moved back for a triple-quoted f-string because the newlines are
// part of the f-string itself, so there is no newline token to be emitted.
if self.current_flags.is_triple_quoted_fstring() {
return false;
}
let mut current_position = self.current_range().start();
let reverse_chars = self.source[..current_position.to_usize()].chars().rev();
let mut newline_position = None;
@ -1578,6 +1584,11 @@ impl TokenFlags {
self.intersects(TokenFlags::F_STRING)
}
/// Returns `true` if the token is a triple-quoted f-string.
fn is_triple_quoted_fstring(self) -> bool {
self.contains(TokenFlags::F_STRING | TokenFlags::TRIPLE_QUOTED_STRING)
}
/// Returns `true` if the token is a raw string.
const fn is_raw_string(self) -> bool {
self.intersects(TokenFlags::RAW_STRING)