Parse triple quoted string annotations as if parenthesized (#15387)

## Summary

Resolves #9467 

Parse quoted annotations as if the string content is inside parenthesis.
With this logic `x` and `y` in this example are equal:

```python
y: """
   int |
   str
"""

z: """(
    int |
    str
)
"""
```

Also this rule only applies to triple
quotes([link](https://github.com/python/typing-council/issues/9#issuecomment-1890808610)).

This PR is based on the
[comments](https://github.com/astral-sh/ruff/issues/9467#issuecomment-2579180991)
on the issue.

I did one extra change, since we don't want any indentation tokens I am
setting the `State::Other` as the initial state of the Lexer.

Remaining work:

- [x] Add a test case for red-knot.
- [x] Add more tests.

## Test Plan

Added a test which previously failed because quoted annotation contained
indentation.
Added an mdtest for red-knot.
Updated previous test.

Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
Co-authored-by: Micha Reiser <micha@reiser.io>
This commit is contained in:
Shaygan Hooshyari 2025-01-16 07:08:15 +01:00 committed by GitHub
parent d2656e88a3
commit cf4ab7cba1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 295 additions and 31 deletions

View file

@ -84,15 +84,21 @@ impl<'src> Lexer<'src> {
"Lexer only supports files with a size up to 4GB"
);
let (state, nesting) = if mode == Mode::ParenthesizedExpression {
(State::Other, 1)
} else {
(State::AfterNewline, 0)
};
let mut lexer = Lexer {
source,
cursor: Cursor::new(source),
state: State::AfterNewline,
state,
current_kind: TokenKind::EndOfFile,
current_range: TextRange::empty(start_offset),
current_value: TokenValue::None,
current_flags: TokenFlags::empty(),
nesting: 0,
nesting,
indentations: Indentations::default(),
pending_indentation: None,
mode,
@ -1309,7 +1315,11 @@ impl<'src> Lexer<'src> {
fn consume_end(&mut self) -> TokenKind {
// We reached end of file.
// First of all, we need all nestings to be finished.
if self.nesting > 0 {
// For Mode::ParenthesizedExpression we start with nesting level 1.
// So we check if we end with that level.
let init_nesting = u32::from(self.mode == Mode::ParenthesizedExpression);
if self.nesting > init_nesting {
// Reset the nesting to avoid going into infinite loop.
self.nesting = 0;
return self.push_error(LexicalError::new(LexicalErrorType::Eof, self.token_range()));