Do not include newline for unterminated string range (#12017)

## Summary

This PR updates the unterminated string error range to not include the
final newline character.

This is a follow-up to #12016 and required for #12019

This is not done for when the unterminated string goes till the end of
file (not a newline character). The unterminated f-string range is
correct.

### Why is this required for #12019 ?

Because otherwise the token ranges will overlap. For example:
```py
f"{"
f"{foo!r"
```

Here, the re-lexing logic recovers from an unterminated f-string and
thus emitting a `Newline` token for the one at the end of the first
line. But, currently the `Unknown` and the `Newline` token would overlap
because the `Unknown` token (unterminated string literal) range would
include the newline character.

## Test Plan

Update and validate the snapshot.
This commit is contained in:
Dhruv Manilawala 2024-06-25 13:40:07 +05:30 committed by GitHub
parent 9c1b6ec411
commit d930e97212
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 53 additions and 65 deletions

View file

@ -973,10 +973,10 @@ impl<'src> Lexer<'src> {
}
match ch {
Some('\r' | '\n') => {
Some(newline @ ('\r' | '\n')) => {
return self.push_error(LexicalError::new(
LexicalErrorType::UnclosedStringError,
self.token_range(),
self.token_range().sub_end(newline.text_len()),
));
}
Some(ch) if ch == quote => {