ruff/crates/ruff_python_parser/resources
Dhruv Manilawala 68a8978454
Consider line continuation character for re-lexing (#12008)
## Summary

This PR fixes a bug where the re-lexing logic didn't consider the line
continuation character being present before the newline character. This
meant that the lexer was being moved back to the newline character which
is actually ignored via `\`.

Considering the following code:
```py
f'middle {'string':\
        'format spec'}

```

The old token stream is:
```
...
Colon 18..19
FStringMiddle 19..29 (flags = F_STRING)
Newline 20..21
Indent 21..29
String 29..42
Rbrace 42..43
...
```

Notice how the ranges are overlapping between the `FStringMiddle` token
and the tokens emitted after moving the lexer backwards.

After this fix, the new token stream which is without moving the lexer
backwards in this scenario:
```
FStringStart 0..2 (flags = F_STRING)
FStringMiddle 2..9 (flags = F_STRING)
Lbrace 9..10
String 10..18
Colon 18..19
FStringMiddle 19..29 (flags = F_STRING)
FStringEnd 29..30 (flags = F_STRING)
Name 30..36
Name 37..41
Unknown 41..44
Newline 44..45
```

fixes: #12004 

## Test Plan

Add test cases and update the snapshots.
2024-06-25 02:13:54 +00:00
..
inline Implement re-lexing logic for better error recovery (#11845) 2024-06-17 06:47:00 +00:00
invalid Consider line continuation character for re-lexing (#12008) 2024-06-25 02:13:54 +00:00
valid Add support for PEP 696 syntax (#11120) 2024-04-26 09:47:29 +02:00
.editorconfig Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30