Avoid consuming newline for unterminated string (#12067)

## Summary

This PR fixes the lexer logic to **not** consume the newline character
for an unterminated string literal.

Currently, the lexer would consume it to be part of the string itself
but that would be bad for recovery because then the lexer wouldn't emit
the newline token ever. This PR fixes that to avoid consuming the
newline character in that case.

This was discovered during https://github.com/astral-sh/ruff/pull/12060.

## Test Plan

Update the snapshots and validate them.
This commit is contained in:
Dhruv Manilawala 2024-06-27 17:02:48 +05:30 committed by GitHub
parent 55f4812051
commit e137c824c3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 22 additions and 8 deletions

View file

@ -160,6 +160,15 @@ Module(
|
|
1 | 'hello' 'world
| ^ Syntax Error: Expected a statement
2 | 1 + 1
3 | 'hello' f'world {x}
4 | 2 + 2
|
|
1 | 'hello' 'world
2 | 1 + 1

View file

@ -335,8 +335,8 @@ Module(
|
5 | f'middle {'string':\
6 | 'format spec'}
| ^ Syntax Error: Expected a statement
7 |
| ^ Syntax Error: Expected a statement
8 | f'middle {'string':\\
9 | 'format spec'}
|