mirrors/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-08-03 18:28:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Maintain synchronicity between the lexer and the parser (#11457)

Browse source 
## Summary

This PR updates the entire parser stack in multiple ways:

### Make the lexer lazy

* https://github.com/astral-sh/ruff/pull/11244
* https://github.com/astral-sh/ruff/pull/11473

Previously, Ruff's lexer would act as an iterator. The parser would
collect all the tokens in a vector first and then process the tokens to
create the syntax tree.

The first task in this project is to update the entire parsing flow to
make the lexer lazy. This includes the `Lexer`, `TokenSource`, and
`Parser`. For context, the `TokenSource` is a wrapper around the `Lexer`
to filter out the trivia tokens[^1]. Now, the parser will ask the token
source to get the next token and only then the lexer will continue and
emit the token. This means that the lexer needs to be aware of the
"current" token. When the `next_token` is called, the current token will
be updated with the newly lexed token.

The main motivation to make the lexer lazy is to allow re-lexing a token
in a different context. This is going to be really useful to make the
parser error resilience. For example, currently the emitted tokens
remains the same even if the parser can recover from an unclosed
parenthesis. This is important because the lexer emits a
`NonLogicalNewline` in parenthesized context while a normal `Newline` in
non-parenthesized context. This different kinds of newline is also used
to emit the indentation tokens which is important for the parser as it's
used to determine the start and end of a block.

Additionally, this allows us to implement the following functionalities:
1. Checkpoint - rewind infrastructure: The idea here is to create a
checkpoint and continue lexing. At a later point, this checkpoint can be
used to rewind the lexer back to the provided checkpoint.
2. Remove the `SoftKeywordTransformer` and instead use lookahead or
speculative parsing to determine whether a soft keyword is a keyword or
an identifier
3. Remove the `Tok` enum. The `Tok` enum represents the tokens emitted
by the lexer but it contains owned data which makes it expensive to
clone. The new `TokenKind` enum just represents the type of token which
is very cheap.

This brings up a question as to how will the parser get the owned value
which was stored on `Tok`. This will be solved by introducing a new
`TokenValue` enum which only contains a subset of token kinds which has
the owned value. This is stored on the lexer and is requested by the
parser when it wants to process the data. For example:
8196720f80/crates/ruff_python_parser/src/parser/expression.rs (L1260-L1262)

[^1]: Trivia tokens are `NonLogicalNewline` and `Comment`

### Remove `SoftKeywordTransformer`

* https://github.com/astral-sh/ruff/pull/11441
* https://github.com/astral-sh/ruff/pull/11459
* https://github.com/astral-sh/ruff/pull/11442
* https://github.com/astral-sh/ruff/pull/11443
* https://github.com/astral-sh/ruff/pull/11474

For context,
https://github.com/RustPython/RustPython/pull/4519/files#diff-5de40045e78e794aa5ab0b8aacf531aa477daf826d31ca129467703855408220
added support for soft keywords in the parser which uses infinite
lookahead to classify a soft keyword as a keyword or an identifier. This
is a brilliant idea as it basically wraps the existing Lexer and works
on top of it which means that the logic for lexing and re-lexing a soft
keyword remains separate. The change here is to remove
`SoftKeywordTransformer` and let the parser determine this based on
context, lookahead and speculative parsing.

* **Context:** The transformer needs to know the position of the lexer
between it being at a statement position or a simple statement position.
This is because a `match` token starts a compound statement while a
`type` token starts a simple statement. **The parser already knows
this.**
* **Lookahead:** Now that the parser knows the context it can perform
lookahead of up to two tokens to classify the soft keyword. The logic
for this is mentioned in the PR implementing it for `type` and `match
soft keyword.
* **Speculative parsing:** This is where the checkpoint - rewind
infrastructure helps. For `match` soft keyword, there are certain cases
for which we can't classify based on lookahead. The idea here is to
create a checkpoint and keep parsing. Based on whether the parsing was
successful and what tokens are ahead we can classify the remaining
cases. Refer to #11443 for more details.

If the soft keyword is being parsed in an identifier context, it'll be
converted to an identifier and the emitted token will be updated as
well. Refer
8196720f80/crates/ruff_python_parser/src/parser/expression.rs (L487-L491).

The `case` soft keyword doesn't require any special handling because
it'll be a keyword only in the context of a match statement.

### Update the parser API

* https://github.com/astral-sh/ruff/pull/11494
* https://github.com/astral-sh/ruff/pull/11505

Now that the lexer is in sync with the parser, and the parser helps to
determine whether a soft keyword is a keyword or an identifier, the
lexer cannot be used on its own. The reason being that it's not
sensitive to the context (which is correct). This means that the parser
API needs to be updated to not allow any access to the lexer.

Previously, there were multiple ways to parse the source code:
1. Passing the source code itself
2. Or, passing the tokens

Now that the lexer and parser are working together, the API
corresponding to (2) cannot exists. The final API is mentioned in this
PR description: https://github.com/astral-sh/ruff/pull/11494.

### Refactor the downstream tools (linter and formatter)

* https://github.com/astral-sh/ruff/pull/11511
* https://github.com/astral-sh/ruff/pull/11515
* https://github.com/astral-sh/ruff/pull/11529
* https://github.com/astral-sh/ruff/pull/11562
* https://github.com/astral-sh/ruff/pull/11592

And, the final set of changes involves updating all references of the
lexer and `Tok` enum. This was done in two-parts:
1. Update all the references in a way that doesn't require any changes
from this PR i.e., it can be done independently
	* https://github.com/astral-sh/ruff/pull/11402
	* https://github.com/astral-sh/ruff/pull/11406
	* https://github.com/astral-sh/ruff/pull/11418
	* https://github.com/astral-sh/ruff/pull/11419
	* https://github.com/astral-sh/ruff/pull/11420
	* https://github.com/astral-sh/ruff/pull/11424
2. Update all the remaining references to use the changes made in this
PR

For (2), there were various strategies used:
1. Introduce a new `Tokens` struct which wraps the token vector and add
methods to query a certain subset of tokens. These includes:
	1. `up_to_first_unknown` which replaces the `tokenize` function
2. `in_range` and `after` which replaces the `lex_starts_at` function
where the former returns the tokens within the given range while the
latter returns all the tokens after the given offset
2. Introduce a new `TokenFlags` which is a set of flags to query certain
information from a token. Currently, this information is only limited to
any string type token but can be expanded to include other information
in the future as needed. https://github.com/astral-sh/ruff/pull/11578
3. Move the `CommentRanges` to the parsed output because this
information is common to both the linter and the formatter. This removes
the need for `tokens_and_ranges` function.

## Test Plan

- [x] Update and verify the test snapshots
- [x] Make sure the entire test suite is passing
- [x] Make sure there are no changes in the ecosystem checks
- [x] Run the fuzzer on the parser
- [x] Run this change on dozens of open-source projects

### Running this change on dozens of open-source projects

Refer to the PR description to get the list of open source projects used
for testing.

Now, the following tests were done between `main` and this branch:
1. Compare the output of `--select=E999` (syntax errors)
2. Compare the output of default rule selection
3. Compare the output of `--select=ALL`

**Conclusion: all output were same**

## What's next?

The next step is to introduce re-lexing logic and update the parser to
feed the recovery information to the lexer so that it can emit the
correct token. This moves us one step closer to having error resilience
in the parser and provides Ruff the possibility to lint even if the
source code contains syntax errors.
This commit is contained in:
Dhruv Manilawala 2024-06-03 18:23:50 +05:30 • committed by GitHub
parent c69a789aa5
commit bf5b62edac
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
262 changed files with 8174 additions and 6132 deletions
Download patch file Download diff file Expand all files Collapse all files

4
crates/ruff_python_parser/src/error.rs
View file

@ -7,7 +7,7 @@ use crate::TokenKind;
/// Represents represent errors that occur during parsing and are
/// returned by the `parse_*` functions.
#[derive(Debug, PartialEq)]
#[derive(Debug, PartialEq, Clone)]
pub struct ParseError {
pub error: ParseErrorType,
pub location: TextRange,
@ -85,7 +85,7 @@ impl std::fmt::Display for FStringErrorType {
}
/// Represents the different types of errors that can occur during parsing.
#[derive(Debug, PartialEq)]
#[derive(Debug, PartialEq, Clone)]
pub enum ParseErrorType {
/// An unexpected error occurred.
OtherError(String),

1424
crates/ruff_python_parser/src/lexer.rs
View file

File diff suppressed because it is too large Load diff

41
crates/ruff_python_parser/src/lexer/cursor.rs
View file

@ -1,18 +1,26 @@
use ruff_text_size::{TextLen, TextSize};
use std::str::Chars;
use ruff_text_size::{TextLen, TextSize};
pub(crate) const EOF_CHAR: char = '\0';
/// A cursor represents a pointer in the source code.
#[derive(Clone, Debug)]
pub(super) struct Cursor<'a> {
chars: Chars<'a>,
pub(super) struct Cursor<'src> {
/// An iterator over the [`char`]'s of the source code.
chars: Chars<'src>,
/// Length of the source code. This is used as a marker to indicate the start of the current
/// token which is being lexed.
source_length: TextSize,
/// Stores the previous character for debug assertions.
#[cfg(debug_assertions)]
prev_char: char,
}
impl<'a> Cursor<'a> {
pub(crate) fn new(source: &'a str) -> Self {
impl<'src> Cursor<'src> {
pub(crate) fn new(source: &'src str) -> Self {
Self {
source_length: source.text_len(),
chars: source.chars(),
@ -21,14 +29,14 @@ impl<'a> Cursor<'a> {
}
}
/// Returns the previous token. Useful for debug assertions.
/// Returns the previous character. Useful for debug assertions.
#[cfg(debug_assertions)]
pub(super) const fn previous(&self) -> char {
self.prev_char
}
/// Peeks the next character from the input stream without consuming it.
/// Returns [`EOF_CHAR`] if the file is at the end of the file.
/// Returns [`EOF_CHAR`] if the position is past the end of the file.
pub(super) fn first(&self) -> char {
self.chars.clone().next().unwrap_or(EOF_CHAR)
}
@ -42,29 +50,44 @@ impl<'a> Cursor<'a> {
}
/// Returns the remaining text to lex.
pub(super) fn rest(&self) -> &'a str {
///
/// Use [`Cursor::text_len`] to get the length of the remaining text.
pub(super) fn rest(&self) -> &'src str {
self.chars.as_str()
}
/// Returns the length of the remaining text.
///
/// Use [`Cursor::rest`] to get the remaining text.
// SAFETY: The `source.text_len` call in `new` would panic if the string length is larger than a `u32`.
#[allow(clippy::cast_possible_truncation)]
pub(super) fn text_len(&self) -> TextSize {
TextSize::new(self.chars.as_str().len() as u32)
}
/// Returns the length of the current token length.
///
/// This is to be used after setting the start position of the token using
/// [`Cursor::start_token`].
pub(super) fn token_len(&self) -> TextSize {
self.source_length - self.text_len()
}
/// Mark the current position of the cursor as the start of the token which is going to be
/// lexed.
///
/// Use [`Cursor::token_len`] to get the length of the lexed token.
pub(super) fn start_token(&mut self) {
self.source_length = self.text_len();
}
/// Returns `true` if the cursor is at the end of file.
pub(super) fn is_eof(&self) -> bool {
self.chars.as_str().is_empty()
}
/// Consumes the next character
/// Moves the cursor to the next character, returning the previous character.
/// Returns [`None`] if there is no next character.
pub(super) fn bump(&mut self) -> Option<char> {
let prev = self.chars.next()?;

27
crates/ruff_python_parser/src/lexer/fstring.rs
View file

@ -1,9 +1,11 @@
use ruff_python_ast::{AnyStringFlags, StringFlags};
use ruff_python_ast::StringFlags;
use super::TokenFlags;
/// The context representing the current f-string that the lexer is in.
#[derive(Debug)]
#[derive(Clone, Debug)]
pub(crate) struct FStringContext {
flags: AnyStringFlags,
flags: TokenFlags,
/// The level of nesting for the lexer when it entered the current f-string.
/// The nesting level includes all kinds of parentheses i.e., round, square,
@ -17,8 +19,9 @@ pub(crate) struct FStringContext {
}
impl FStringContext {
pub(crate) const fn new(flags: AnyStringFlags, nesting: u32) -> Self {
debug_assert!(flags.is_f_string());
pub(crate) const fn new(flags: TokenFlags, nesting: u32) -> Self {
assert!(flags.is_f_string());
Self {
flags,
nesting,
@ -26,8 +29,7 @@ impl FStringContext {
}
}
pub(crate) const fn flags(&self) -> AnyStringFlags {
debug_assert!(self.flags.is_f_string());
pub(crate) const fn flags(&self) -> TokenFlags {
self.flags
}
@ -127,4 +129,15 @@ impl FStrings {
pub(crate) fn current_mut(&mut self) -> Option<&mut FStringContext> {
self.stack.last_mut()
}
pub(crate) fn checkpoint(&self) -> FStringsCheckpoint {
FStringsCheckpoint(self.stack.clone())
}
pub(crate) fn rewind(&mut self, checkpoint: FStringsCheckpoint) {
self.stack = checkpoint.0;
}
}
#[derive(Debug, Clone)]
pub(crate) struct FStringsCheckpoint(Vec<FStringContext>);

15
crates/ruff_python_parser/src/lexer/indentation.rs
View file

@ -82,8 +82,8 @@ impl Indentation {
#[derive(Debug, Copy, Clone, PartialEq)]
pub(super) struct UnexpectedIndentation;
// The indentations stack is used to keep track of the current indentation level
// [See Indentation](docs.python.org/3/reference/lexical_analysis.html#indentation).
/// The indentations stack is used to keep track of the current indentation level
/// [See Indentation](docs.python.org/3/reference/lexical_analysis.html#indentation).
#[derive(Debug, Clone, Default)]
pub(super) struct Indentations {
stack: Vec<Indentation>,
@ -124,8 +124,19 @@ impl Indentations {
static ROOT: Indentation = Indentation::root();
self.stack.last().unwrap_or(&ROOT)
}
pub(crate) fn checkpoint(&self) -> IndentationsCheckpoint {
IndentationsCheckpoint(self.stack.clone())
}
pub(crate) fn rewind(&mut self, checkpoint: IndentationsCheckpoint) {
self.stack = checkpoint.0;
}
}
#[derive(Debug, Clone)]
pub(crate) struct IndentationsCheckpoint(Vec<Indentation>);
assert_eq_size!(Indentation, u64);
#[cfg(test)]

779
crates/ruff_python_parser/src/lib.rs
View file

@ -57,81 +57,37 @@
//!
//! - token: This module contains the definition of the tokens that are generated by the lexer.
//! - [lexer]: This module contains the lexer and is responsible for generating the tokens.
//! - parser: This module contains an interface to the [Program] and is responsible for generating the AST.
//! - parser: This module contains an interface to the [Parsed] and is responsible for generating the AST.
//! - mode: This module contains the definition of the different modes that the `ruff_python_parser` can be in.
//!
//! # Examples
//!
//! For example, to get a stream of tokens from a given string, one could do this:
//!
//! ```
//! use ruff_python_parser::{lexer::lex, Mode};
//!
//! let python_source = r#"
//! def is_odd(i):
//! return bool(i & 1)
//! "#;
//! let mut tokens = lex(python_source, Mode::Module);
//! assert!(tokens.all(|t| t.is_ok()));
//! ```
//!
//! These tokens can be directly fed into the `ruff_python_parser` to generate an AST:
//!
//! ```
//! use ruff_python_parser::lexer::lex;
//! use ruff_python_parser::{Mode, parse_tokens};
//!
//! let python_source = r#"
//! def is_odd(i):
//! return bool(i & 1)
//! "#;
//! let tokens = lex(python_source, Mode::Module);
//! let ast = parse_tokens(tokens.collect(), python_source, Mode::Module);
//!
//! assert!(ast.is_ok());
//! ```
//!
//! Alternatively, you can use one of the other `parse_*` functions to parse a string directly without using a specific
//! mode or tokenizing the source beforehand:
//!
//! ```
//! use ruff_python_parser::parse_suite;
//!
//! let python_source = r#"
//! def is_odd(i):
//! return bool(i & 1)
//! "#;
//! let ast = parse_suite(python_source);
//!
//! assert!(ast.is_ok());
//! ```
//!
//! [lexical analysis]: https://en.wikipedia.org/wiki/Lexical_analysis
//! [parsing]: https://en.wikipedia.org/wiki/Parsing
//! [lexer]: crate::lexer
use std::iter::FusedIterator;
use std::cell::OnceCell;
use std::ops::Deref;
use ruff_python_ast::{Expr, Mod, ModModule, PySourceType, Suite};
use ruff_text_size::{TextRange, TextSize};
pub use crate::error::{FStringErrorType, ParseError, ParseErrorType};
use crate::lexer::{lex, lex_starts_at, LexResult};
pub use crate::parser::Program;
pub use crate::token::{Tok, TokenKind};
pub use crate::lexer::Token;
pub use crate::token::TokenKind;
use crate::parser::Parser;
use itertools::Itertools;
use ruff_python_ast::{Expr, Mod, ModExpression, ModModule, PySourceType, Suite};
use ruff_python_trivia::CommentRanges;
use ruff_text_size::{Ranged, TextRange, TextSize};
mod error;
pub mod lexer;
mod parser;
mod soft_keywords;
mod string;
mod token;
mod token_set;
mod token_source;
pub mod typing;
/// Parse a full Python program usually consisting of multiple lines.
/// Parse a full Python module usually consisting of multiple lines.
///
/// This is a convenience function that can be used to parse a full Python program without having to
/// specify the [`Mode`] or the location. It is probably what you want to use most of the time.
@ -141,7 +97,7 @@ pub mod typing;
/// For example, parsing a simple function definition and a call to that function:
///
/// ```
/// use ruff_python_parser::parse_program;
/// use ruff_python_parser::parse_module;
///
/// let source = r#"
/// def foo():
@ -150,41 +106,15 @@ pub mod typing;
/// print(foo())
/// "#;
///
/// let program = parse_program(source);
/// assert!(program.is_ok());
/// let module = parse_module(source);
/// assert!(module.is_ok());
/// ```
pub fn parse_program(source: &str) -> Result<ModModule, ParseError> {
let lexer = lex(source, Mode::Module);
match parse_tokens(lexer.collect(), source, Mode::Module)? {
Mod::Module(m) => Ok(m),
Mod::Expression(_) => unreachable!("Mode::Module doesn't return other variant"),
}
}
/// Parse a full Python program into a [`Suite`].
///
/// This function is similar to [`parse_program`] except that it returns the module body
/// instead of the module itself.
///
/// # Example
///
/// For example, parsing a simple function definition and a call to that function:
///
/// ```
/// use ruff_python_parser::parse_suite;
///
/// let source = r#"
/// def foo():
/// return 42
///
/// print(foo())
/// "#;
///
/// let body = parse_suite(source);
/// assert!(body.is_ok());
/// ```
pub fn parse_suite(source: &str) -> Result<Suite, ParseError> {
parse_program(source).map(|m| m.body)
pub fn parse_module(source: &str) -> Result<Parsed<ModModule>, ParseError> {
Parser::new(source, Mode::Module)
.parse()
.try_into_module()
.unwrap()
.into_result()
}
/// Parses a single Python expression.
@ -202,37 +132,40 @@ pub fn parse_suite(source: &str) -> Result<Suite, ParseError> {
/// let expr = parse_expression("1 + 2");
/// assert!(expr.is_ok());
/// ```
pub fn parse_expression(source: &str) -> Result<Expr, ParseError> {
let lexer = lex(source, Mode::Expression).collect();
match parse_tokens(lexer, source, Mode::Expression)? {
Mod::Expression(expression) => Ok(*expression.body),
Mod::Module(_m) => unreachable!("Mode::Expression doesn't return other variant"),
}
pub fn parse_expression(source: &str) -> Result<Parsed<ModExpression>, ParseError> {
Parser::new(source, Mode::Expression)
.parse()
.try_into_expression()
.unwrap()
.into_result()
}
/// Parses a Python expression from a given location.
/// Parses a Python expression for the given range in the source.
///
/// This function allows to specify the location of the expression in the source code, other than
/// This function allows to specify the range of the expression in the source code, other than
/// that, it behaves exactly like [`parse_expression`].
///
/// # Example
///
/// Parsing a single expression denoting the addition of two numbers, but this time specifying a different,
/// somewhat silly, location:
/// Parsing one of the numeric literal which is part of an addition expression:
///
/// ```
/// use ruff_python_parser::parse_expression_starts_at;
/// # use ruff_text_size::TextSize;
/// use ruff_python_parser::parse_expression_range;
/// # use ruff_text_size::{TextRange, TextSize};
///
/// let expr = parse_expression_starts_at("1 + 2", TextSize::from(400));
/// assert!(expr.is_ok());
/// let parsed = parse_expression_range("11 + 22 + 33", TextRange::new(TextSize::new(5), TextSize::new(7)));
/// assert!(parsed.is_ok());
/// ```
pub fn parse_expression_starts_at(source: &str, offset: TextSize) -> Result<Expr, ParseError> {
let lexer = lex_starts_at(source, Mode::Module, offset).collect();
match parse_tokens(lexer, source, Mode::Expression)? {
Mod::Expression(expression) => Ok(*expression.body),
Mod::Module(_m) => unreachable!("Mode::Expression doesn't return other variant"),
}
pub fn parse_expression_range(
source: &str,
range: TextRange,
) -> Result<Parsed<ModExpression>, ParseError> {
let source = &source[..range.end().to_usize()];
Parser::new_starts_at(source, Mode::Expression, range.start())
.parse()
.try_into_expression()
.unwrap()
.into_result()
}
/// Parse the given Python source code using the specified [`Mode`].
@ -249,8 +182,8 @@ pub fn parse_expression_starts_at(source: &str, offset: TextSize) -> Result<Expr
/// ```
/// use ruff_python_parser::{Mode, parse};
///
/// let expr = parse("1 + 2", Mode::Expression);
/// assert!(expr.is_ok());
/// let parsed = parse("1 + 2", Mode::Expression);
/// assert!(parsed.is_ok());
/// ```
///
/// Alternatively, we can parse a full Python program consisting of multiple lines:
@ -264,8 +197,8 @@ pub fn parse_expression_starts_at(source: &str, offset: TextSize) -> Result<Expr
/// def greet(self):
/// print("Hello, world!")
/// "#;
/// let program = parse(source, Mode::Module);
/// assert!(program.is_ok());
/// let parsed = parse(source, Mode::Module);
/// assert!(parsed.is_ok());
/// ```
///
/// Additionally, we can parse a Python program containing IPython escapes:
@ -278,189 +211,308 @@ pub fn parse_expression_starts_at(source: &str, offset: TextSize) -> Result<Expr
/// ?str.replace
/// !ls
/// "#;
/// let program = parse(source, Mode::Ipython);
/// assert!(program.is_ok());
/// let parsed = parse(source, Mode::Ipython);
/// assert!(parsed.is_ok());
/// ```
pub fn parse(source: &str, mode: Mode) -> Result<Mod, ParseError> {
let lxr = lexer::lex(source, mode);
parse_tokens(lxr.collect(), source, mode)
pub fn parse(source: &str, mode: Mode) -> Result<Parsed<Mod>, ParseError> {
parse_unchecked(source, mode).into_result()
}
/// Parse the given Python source code using the specified [`Mode`] and [`TextSize`].
/// Parse the given Python source code using the specified [`Mode`].
///
/// This function allows to specify the location of the source code, other than
/// that, it behaves exactly like [`parse`].
///
/// # Example
///
/// ```
/// # use ruff_text_size::TextSize;
/// use ruff_python_parser::{Mode, parse_starts_at};
///
/// let source = r#"
/// def fib(i):
/// a, b = 0, 1
/// for _ in range(i):
/// a, b = b, a + b
/// return a
///
/// print(fib(42))
/// "#;
/// let program = parse_starts_at(source, Mode::Module, TextSize::from(0));
/// assert!(program.is_ok());
/// ```
pub fn parse_starts_at(source: &str, mode: Mode, offset: TextSize) -> Result<Mod, ParseError> {
let lxr = lexer::lex_starts_at(source, mode, offset);
parse_tokens(lxr.collect(), source, mode)
/// This is same as the [`parse`] function except that it doesn't check for any [`ParseError`]
/// and returns the [`Parsed`] as is.
pub fn parse_unchecked(source: &str, mode: Mode) -> Parsed<Mod> {
Parser::new(source, mode).parse()
}
/// Parse an iterator of [`LexResult`]s using the specified [`Mode`].
///
/// This could allow you to perform some preprocessing on the tokens before parsing them.
///
/// # Example
///
/// As an example, instead of parsing a string, we can parse a list of tokens after we generate
/// them using the [`lexer::lex`] function:
///
/// ```
/// use ruff_python_parser::lexer::lex;
/// use ruff_python_parser::{Mode, parse_tokens};
///
/// let source = "1 + 2";
/// let tokens = lex(source, Mode::Expression);
/// let expr = parse_tokens(tokens.collect(), source, Mode::Expression);
/// assert!(expr.is_ok());
/// ```
pub fn parse_tokens(tokens: Vec<LexResult>, source: &str, mode: Mode) -> Result<Mod, ParseError> {
let program = Program::parse_tokens(source, tokens, mode);
if program.is_valid() {
Ok(program.into_ast())
} else {
Err(program.into_errors().into_iter().next().unwrap())
/// Parse the given Python source code using the specified [`PySourceType`].
pub fn parse_unchecked_source(source: &str, source_type: PySourceType) -> Parsed<ModModule> {
// SAFETY: Safe because `PySourceType` always parses to a `ModModule`
Parser::new(source, source_type.as_mode())
.parse()
.try_into_module()
.unwrap()
}
/// Represents the parsed source code.
#[derive(Debug, Clone)]
pub struct Parsed<T> {
syntax: T,
tokens: Tokens,
errors: Vec<ParseError>,
comment_ranges: CommentRanges,
}
impl<T> Parsed<T> {
/// Returns the syntax node represented by this parsed output.
pub fn syntax(&self) -> &T {
&self.syntax
}
/// Returns all the tokens for the parsed output.
pub fn tokens(&self) -> &Tokens {
&self.tokens
}
/// Returns a list of syntax errors found during parsing.
pub fn errors(&self) -> &[ParseError] {
&self.errors
}
/// Returns the comment ranges for the parsed output.
pub fn comment_ranges(&self) -> &CommentRanges {
&self.comment_ranges
}
/// Consumes the [`Parsed`] output and returns the contained syntax node.
pub fn into_syntax(self) -> T {
self.syntax
}
/// Consumes the [`Parsed`] output and returns a list of syntax errors found during parsing.
pub fn into_errors(self) -> Vec<ParseError> {
self.errors
}
/// Returns `true` if the parsed source code is valid i.e., it has no syntax errors.
pub fn is_valid(&self) -> bool {
self.errors.is_empty()
}
/// Returns the [`Parsed`] output as a [`Result`], returning [`Ok`] if it has no syntax errors,
/// or [`Err`] containing the first [`ParseError`] encountered.
pub fn as_result(&self) -> Result<&Parsed<T>, &ParseError> {
if let [error, ..] = self.errors() {
Err(error)
} else {
Ok(self)
}
}
/// Consumes the [`Parsed`] output and returns a [`Result`] which is [`Ok`] if it has no syntax
/// errors, or [`Err`] containing the first [`ParseError`] encountered.
pub(crate) fn into_result(self) -> Result<Parsed<T>, ParseError> {
if self.is_valid() {
Ok(self)
} else {
Err(self.into_errors().into_iter().next().unwrap())
}
}
}
/// Tokens represents a vector of [`LexResult`].
///
/// This should only include tokens up to and including the first error. This struct is created
/// by the [`tokenize`] function.
impl Parsed<Mod> {
/// Attempts to convert the [`Parsed<Mod>`] into a [`Parsed<ModModule>`].
///
/// This method checks if the `syntax` field of the output is a [`Mod::Module`]. If it is, the
/// method returns [`Some(Parsed<ModModule>)`] with the contained module. Otherwise, it
/// returns [`None`].
///
/// [`Some(Parsed<ModModule>)`]: Some
fn try_into_module(self) -> Option<Parsed<ModModule>> {
match self.syntax {
Mod::Module(module) => Some(Parsed {
syntax: module,
tokens: self.tokens,
errors: self.errors,
comment_ranges: self.comment_ranges,
}),
Mod::Expression(_) => None,
}
}
/// Attempts to convert the [`Parsed<Mod>`] into a [`Parsed<ModExpression>`].
///
/// This method checks if the `syntax` field of the output is a [`Mod::Expression`]. If it is,
/// the method returns [`Some(Parsed<ModExpression>)`] with the contained expression.
/// Otherwise, it returns [`None`].
///
/// [`Some(Parsed<ModExpression>)`]: Some
fn try_into_expression(self) -> Option<Parsed<ModExpression>> {
match self.syntax {
Mod::Module(_) => None,
Mod::Expression(expression) => Some(Parsed {
syntax: expression,
tokens: self.tokens,
errors: self.errors,
comment_ranges: self.comment_ranges,
}),
}
}
}
impl Parsed<ModModule> {
/// Returns the module body contained in this parsed output as a [`Suite`].
pub fn suite(&self) -> &Suite {
&self.syntax.body
}
/// Consumes the [`Parsed`] output and returns the module body as a [`Suite`].
pub fn into_suite(self) -> Suite {
self.syntax.body
}
}
impl Parsed<ModExpression> {
/// Returns the expression contained in this parsed output.
pub fn expr(&self) -> &Expr {
&self.syntax.body
}
/// Consumes the [`Parsed`] output and returns the contained [`Expr`].
pub fn into_expr(self) -> Expr {
*self.syntax.body
}
}
/// Tokens represents a vector of lexed [`Token`].
#[derive(Debug, Clone)]
pub struct Tokens(Vec<LexResult>);
pub struct Tokens {
raw: Vec<Token>,
/// Index of the first [`TokenKind::Unknown`] token or the length of the token vector.
first_unknown_or_len: OnceCell<usize>,
}
impl Tokens {
/// Returns an iterator over the [`TokenKind`] and the range corresponding to the tokens.
pub fn kinds(&self) -> TokenKindIter {
TokenKindIter::new(&self.0)
pub(crate) fn new(tokens: Vec<Token>) -> Tokens {
Tokens {
raw: tokens,
first_unknown_or_len: OnceCell::new(),
}
}
/// Consumes the [`Tokens`], returning the underlying vector of [`LexResult`].
pub fn into_inner(self) -> Vec<LexResult> {
self.0
/// Returns a slice of tokens up to (and excluding) the first [`TokenKind::Unknown`] token or
/// all the tokens if there is none.
pub fn up_to_first_unknown(&self) -> &[Token] {
let end = *self.first_unknown_or_len.get_or_init(|| {
self.raw
.iter()
.find_position(|token| token.kind() == TokenKind::Unknown)
.map(|(idx, _)| idx)
.unwrap_or_else(|| self.raw.len())
});
&self.raw[..end]
}
/// Returns a slice of [`Token`] that are within the given `range`.
///
/// The start and end offset of the given range should be either:
/// 1. Token boundary
/// 2. Gap between the tokens
///
/// For example, considering the following tokens and their corresponding range:
///
/// | Token | Range |
/// |---------------------|-----------|
/// | `Def` | `0..3` |
/// | `Name` | `4..7` |
/// | `Lpar` | `7..8` |
/// | `Rpar` | `8..9` |
/// | `Colon` | `9..10` |
/// | `Newline` | `10..11` |
/// | `Comment` | `15..24` |
/// | `NonLogicalNewline` | `24..25` |
/// | `Indent` | `25..29` |
/// | `Pass` | `29..33` |
///
/// Here, for (1) a token boundary is considered either the start or end offset of any of the
/// above tokens. For (2), the gap would be any offset between the `Newline` and `Comment`
/// token which are 12, 13, and 14.
///
/// Examples:
/// 1) `4..10` would give `Name`, `Lpar`, `Rpar`, `Colon`
/// 2) `11..25` would give `Comment`, `NonLogicalNewline`
/// 3) `12..25` would give same as (2) and offset 12 is in the "gap"
/// 4) `9..12` would give `Colon`, `Newline` and offset 12 is in the "gap"
/// 5) `18..27` would panic because both the start and end offset is within a token
///
/// ## Note
///
/// The returned slice can contain the [`TokenKind::Unknown`] token if there was a lexical
/// error encountered within the given range.
///
/// # Panics
///
/// If either the start or end offset of the given range is within a token range.
pub fn in_range(&self, range: TextRange) -> &[Token] {
let tokens_after_start = self.after(range.start());
match tokens_after_start.binary_search_by_key(&range.end(), Ranged::end) {
Ok(idx) => {
// If we found the token with the end offset, that token should be included in the
// return slice.
&tokens_after_start[..=idx]
}
Err(idx) => {
if let Some(token) = tokens_after_start.get(idx) {
// If it's equal to the start offset, then it's at a token boundary which is
// valid. If it's less than the start offset, then it's in the gap between the
// tokens which is valid as well.
assert!(
range.end() <= token.start(),
"End offset {:?} is inside a token range {:?}",
range.end(),
token.range()
);
}
// This index is where the token with the offset _could_ be, so that token should
// be excluded from the return slice.
&tokens_after_start[..idx]
}
}
}
/// Returns a slice of tokens after the given [`TextSize`] offset.
///
/// If the given offset is between two tokens, the returned slice will start from the following
/// token. In other words, if the offset is between the end of previous token and start of next
/// token, the returned slice will start from the next token.
///
/// # Panics
///
/// If the given offset is inside a token range.
pub fn after(&self, offset: TextSize) -> &[Token] {
match self.binary_search_by(|token| token.start().cmp(&offset)) {
Ok(idx) => &self[idx..],
Err(idx) => {
// We can't use `saturating_sub` here because a file could contain a BOM header, in
// which case the token starts at offset 3 for UTF-8 encoded file content.
if idx > 0 {
if let Some(prev) = self.get(idx - 1) {
// If it's equal to the end offset, then it's at a token boundary which is
// valid. If it's greater than the end offset, then it's in the gap between
// the tokens which is valid as well.
assert!(
offset >= prev.end(),
"Offset {:?} is inside a token range {:?}",
offset,
prev.range()
);
}
}
&self[idx..]
}
}
}
}
impl<'a> IntoIterator for &'a Tokens {
type Item = &'a Token;
type IntoIter = std::slice::Iter<'a, Token>;
fn into_iter(self) -> Self::IntoIter {
self.iter()
}
}
impl Deref for Tokens {
type Target = [LexResult];
type Target = [Token];
fn deref(&self) -> &Self::Target {
&self.0
}
}
/// An iterator over the [`TokenKind`] and the corresponding range.
///
/// This struct is created by the [`Tokens::kinds`] method.
#[derive(Clone, Default)]
pub struct TokenKindIter<'a> {
inner: std::iter::Flatten<std::slice::Iter<'a, LexResult>>,
}
impl<'a> TokenKindIter<'a> {
/// Create a new iterator from a slice of [`LexResult`].
pub fn new(tokens: &'a [LexResult]) -> Self {
Self {
inner: tokens.iter().flatten(),
}
}
/// Return the next value without advancing the iterator.
pub fn peek(&mut self) -> Option<(TokenKind, TextRange)> {
self.clone().next()
}
}
impl Iterator for TokenKindIter<'_> {
type Item = (TokenKind, TextRange);
fn next(&mut self) -> Option<Self::Item> {
let &(ref tok, range) = self.inner.next()?;
Some((TokenKind::from_token(tok), range))
}
}
impl FusedIterator for TokenKindIter<'_> {}
impl DoubleEndedIterator for TokenKindIter<'_> {
fn next_back(&mut self) -> Option<Self::Item> {
let &(ref tok, range) = self.inner.next_back()?;
Some((TokenKind::from_token(tok), range))
}
}
/// Collect tokens up to and including the first error.
pub fn tokenize(contents: &str, mode: Mode) -> Tokens {
let mut tokens: Vec<LexResult> = allocate_tokens_vec(contents);
for tok in lexer::lex(contents, mode) {
let is_err = tok.is_err();
tokens.push(tok);
if is_err {
break;
}
}
Tokens(tokens)
}
/// Tokenizes all tokens.
///
/// It differs from [`tokenize`] in that it tokenizes all tokens and doesn't stop
/// after the first `Err`.
pub fn tokenize_all(contents: &str, mode: Mode) -> Vec<LexResult> {
let mut tokens = allocate_tokens_vec(contents);
for token in lexer::lex(contents, mode) {
tokens.push(token);
}
tokens
}
/// Allocates a [`Vec`] with an approximated capacity to fit all tokens
/// of `contents`.
///
/// See [#9546](https://github.com/astral-sh/ruff/pull/9546) for a more detailed explanation.
pub fn allocate_tokens_vec(contents: &str) -> Vec<LexResult> {
Vec::with_capacity(approximate_tokens_lower_bound(contents))
}
/// Approximates the number of tokens when lexing `contents`.
fn approximate_tokens_lower_bound(contents: &str) -> usize {
contents.len().saturating_mul(15) / 100
}
/// Parse a full Python program from its tokens.
pub fn parse_program_tokens(
tokens: Tokens,
source: &str,
is_jupyter_notebook: bool,
) -> anyhow::Result<Suite, ParseError> {
let mode = if is_jupyter_notebook {
Mode::Ipython
} else {
Mode::Module
};
match parse_tokens(tokens.into_inner(), source, mode)? {
Mod::Module(m) => Ok(m.body),
Mod::Expression(_) => unreachable!("Mode::Module doesn't return other variant"),
&self.raw
}
}
@ -529,3 +581,174 @@ impl std::fmt::Display for ModeParseError {
write!(f, r#"mode must be "exec", "eval", "ipython", or "single""#)
}
}
#[cfg(test)]
mod tests {
use std::ops::Range;
use crate::lexer::TokenFlags;
use super::*;
/// Test case containing a "gap" between two tokens.
///
/// Code: <https://play.ruff.rs/a3658340-6df8-42c5-be80-178744bf1193>
const TEST_CASE_WITH_GAP: [(TokenKind, Range<u32>); 10] = [
(TokenKind::Def, 0..3),
(TokenKind::Name, 4..7),
(TokenKind::Lpar, 7..8),
(TokenKind::Rpar, 8..9),
(TokenKind::Colon, 9..10),
(TokenKind::Newline, 10..11),
// Gap ||..||
(TokenKind::Comment, 15..24),
(TokenKind::NonLogicalNewline, 24..25),
(TokenKind::Indent, 25..29),
(TokenKind::Pass, 29..33),
// No newline at the end to keep the token set full of unique tokens
];
/// Test case containing [`TokenKind::Unknown`] token.
///
/// Code: <https://play.ruff.rs/ea722760-9bf5-4d00-be9f-dc441793f88e>
const TEST_CASE_WITH_UNKNOWN: [(TokenKind, Range<u32>); 5] = [
(TokenKind::Name, 0..1),
(TokenKind::Equal, 2..3),
(TokenKind::Unknown, 4..11),
(TokenKind::Plus, 11..12),
(TokenKind::Int, 13..14),
// No newline at the end to keep the token set full of unique tokens
];
/// Helper function to create [`Tokens`] from an iterator of (kind, range).
fn new_tokens(tokens: impl Iterator<Item = (TokenKind, Range<u32>)>) -> Tokens {
Tokens::new(
tokens
.map(|(kind, range)| {
Token::new(
kind,
TextRange::new(TextSize::new(range.start), TextSize::new(range.end)),
TokenFlags::empty(),
)
})
.collect(),
)
}
#[test]
fn tokens_up_to_first_unknown_empty() {
let tokens = Tokens::new(vec![]);
assert_eq!(tokens.up_to_first_unknown(), &[]);
}
#[test]
fn tokens_up_to_first_unknown_noop() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
let up_to_first_unknown = tokens.up_to_first_unknown();
assert_eq!(up_to_first_unknown.len(), tokens.len());
}
#[test]
fn tokens_up_to_first_unknown() {
let tokens = new_tokens(TEST_CASE_WITH_UNKNOWN.into_iter());
let up_to_first_unknown = tokens.up_to_first_unknown();
assert_eq!(up_to_first_unknown.len(), 2);
}
#[test]
fn tokens_after_offset_at_token_start() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
let after = tokens.after(TextSize::new(8));
assert_eq!(after.len(), 7);
assert_eq!(after.first().unwrap().kind(), TokenKind::Rpar);
}
#[test]
fn tokens_after_offset_at_token_end() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
let after = tokens.after(TextSize::new(11));
assert_eq!(after.len(), 4);
assert_eq!(after.first().unwrap().kind(), TokenKind::Comment);
}
#[test]
fn tokens_after_offset_between_tokens() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
let after = tokens.after(TextSize::new(13));
assert_eq!(after.len(), 4);
assert_eq!(after.first().unwrap().kind(), TokenKind::Comment);
}
#[test]
fn tokens_after_offset_at_last_token_end() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
let after = tokens.after(TextSize::new(33));
assert_eq!(after.len(), 0);
}
#[test]
#[should_panic(expected = "Offset 5 is inside a token range 4..7")]
fn tokens_after_offset_inside_token() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
tokens.after(TextSize::new(5));
}
#[test]
fn tokens_in_range_at_token_offset() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
let in_range = tokens.in_range(TextRange::new(4.into(), 10.into()));
assert_eq!(in_range.len(), 4);
assert_eq!(in_range.first().unwrap().kind(), TokenKind::Name);
assert_eq!(in_range.last().unwrap().kind(), TokenKind::Colon);
}
#[test]
fn tokens_in_range_start_offset_at_token_end() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
let in_range = tokens.in_range(TextRange::new(11.into(), 29.into()));
assert_eq!(in_range.len(), 3);
assert_eq!(in_range.first().unwrap().kind(), TokenKind::Comment);
assert_eq!(in_range.last().unwrap().kind(), TokenKind::Indent);
}
#[test]
fn tokens_in_range_end_offset_at_token_start() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
let in_range = tokens.in_range(TextRange::new(8.into(), 15.into()));
assert_eq!(in_range.len(), 3);
assert_eq!(in_range.first().unwrap().kind(), TokenKind::Rpar);
assert_eq!(in_range.last().unwrap().kind(), TokenKind::Newline);
}
#[test]
fn tokens_in_range_start_offset_between_tokens() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
let in_range = tokens.in_range(TextRange::new(13.into(), 29.into()));
assert_eq!(in_range.len(), 3);
assert_eq!(in_range.first().unwrap().kind(), TokenKind::Comment);
assert_eq!(in_range.last().unwrap().kind(), TokenKind::Indent);
}
#[test]
fn tokens_in_range_end_offset_between_tokens() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
let in_range = tokens.in_range(TextRange::new(9.into(), 13.into()));
assert_eq!(in_range.len(), 2);
assert_eq!(in_range.first().unwrap().kind(), TokenKind::Colon);
assert_eq!(in_range.last().unwrap().kind(), TokenKind::Newline);
}
#[test]
#[should_panic(expected = "Offset 5 is inside a token range 4..7")]
fn tokens_in_range_start_offset_inside_token() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
tokens.in_range(TextRange::new(5.into(), 10.into()));
}
#[test]
#[should_panic(expected = "End offset 6 is inside a token range 4..7")]
fn tokens_in_range_end_offset_inside_token() {
let tokens = new_tokens(TEST_CASE_WITH_GAP.into_iter());
tokens.in_range(TextRange::new(0.into(), 6.into()));
}
}

124
crates/ruff_python_parser/src/parser/expression.rs
View file

@ -11,11 +11,12 @@ use ruff_python_ast::{
};
use ruff_text_size::{Ranged, TextLen, TextRange, TextSize};
use crate::lexer::TokenValue;
use crate::parser::progress::ParserProgress;
use crate::parser::{helpers, FunctionKind, Parser};
use crate::string::{parse_fstring_literal_element, parse_string_literal, StringType};
use crate::token_set::TokenSet;
use crate::{FStringErrorType, Mode, ParseErrorType, Tok, TokenKind};
use crate::{FStringErrorType, Mode, ParseErrorType, TokenKind};
use super::{Parenthesized, RecoveryContextKind};
@ -106,9 +107,24 @@ pub(super) const END_EXPR_SET: TokenSet = TokenSet::new([
const END_SEQUENCE_SET: TokenSet = END_EXPR_SET.remove(TokenKind::Comma);
impl<'src> Parser<'src> {
/// Returns `true` if the parser is at a name or keyword (including soft keyword) token.
pub(super) fn at_name_or_keyword(&self) -> bool {
self.at(TokenKind::Name) || self.current_token_kind().is_keyword()
}
/// Returns `true` if the parser is at a name or soft keyword token.
pub(super) fn at_name_or_soft_keyword(&self) -> bool {
self.at(TokenKind::Name) || self.at_soft_keyword()
}
/// Returns `true` if the parser is at a soft keyword token.
pub(super) fn at_soft_keyword(&self) -> bool {
self.current_token_kind().is_soft_keyword()
}
/// Returns `true` if the current token is the start of an expression.
pub(super) fn at_expr(&self) -> bool {
self.at_ts(EXPR_SET)
self.at_ts(EXPR_SET) || self.at_soft_keyword()
}
/// Returns `true` if the current token ends a sequence.
@ -459,36 +475,43 @@ impl<'src> Parser<'src> {
let range = self.current_token_range();
if self.at(TokenKind::Name) {
let (Tok::Name { name }, _) = self.bump(TokenKind::Name) else {
let TokenValue::Name(name) = self.bump_value(TokenKind::Name) else {
unreachable!();
};
ast::Identifier {
return ast::Identifier {
id: name.to_string(),
range,
}
} else {
if self.current_token_kind().is_keyword() {
let (tok, range) = self.next_token();
self.add_error(
ParseErrorType::OtherError(format!(
"Expected an identifier, but found a keyword '{tok}' that cannot be used here"
)),
range,
);
};
}
ast::Identifier {
id: tok.to_string(),
range,
}
} else {
self.add_error(
ParseErrorType::OtherError("Expected an identifier".into()),
range,
);
ast::Identifier {
id: String::new(),
range: self.missing_node_range(),
}
if self.current_token_kind().is_soft_keyword() {
let id = self.src_text(range).to_string();
self.bump_soft_keyword_as_name();
return ast::Identifier { id, range };
}
if self.current_token_kind().is_keyword() {
// Non-soft keyword
self.add_error(
ParseErrorType::OtherError(format!(
"Expected an identifier, but found a keyword {} that cannot be used here",
self.current_token_kind()
)),
range,
);
let id = self.src_text(range).to_string();
self.bump_any();
ast::Identifier { id, range }
} else {
self.add_error(
ParseErrorType::OtherError("Expected an identifier".into()),
range,
);
ast::Identifier {
id: String::new(),
range: self.missing_node_range(),
}
}
}
@ -501,7 +524,7 @@ impl<'src> Parser<'src> {
let lhs = match self.current_token_kind() {
TokenKind::Float => {
let (Tok::Float { value }, _) = self.bump(TokenKind::Float) else {
let TokenValue::Float(value) = self.bump_value(TokenKind::Float) else {
unreachable!()
};
@ -511,7 +534,7 @@ impl<'src> Parser<'src> {
})
}
TokenKind::Complex => {
let (Tok::Complex { real, imag }, _) = self.bump(TokenKind::Complex) else {
let TokenValue::Complex { real, imag } = self.bump_value(TokenKind::Complex) else {
unreachable!()
};
Expr::NumberLiteral(ast::ExprNumberLiteral {
@ -520,7 +543,7 @@ impl<'src> Parser<'src> {
})
}
TokenKind::Int => {
let (Tok::Int { value }, _) = self.bump(TokenKind::Int) else {
let TokenValue::Int(value) = self.bump_value(TokenKind::Int) else {
unreachable!()
};
Expr::NumberLiteral(ast::ExprNumberLiteral {
@ -1231,7 +1254,10 @@ impl<'src> Parser<'src> {
///
/// See: <https://docs.python.org/3.13/reference/lexical_analysis.html#string-and-bytes-literals>
fn parse_string_or_byte_literal(&mut self) -> StringType {
let (Tok::String { value, flags }, range) = self.bump(TokenKind::String) else {
let range = self.current_token_range();
let flags = self.tokens.current_flags().as_any_string_flags();
let TokenValue::String(value) = self.bump_value(TokenKind::String) else {
unreachable!()
};
@ -1277,18 +1303,17 @@ impl<'src> Parser<'src> {
/// See: <https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals>
fn parse_fstring(&mut self) -> ast::FString {
let start = self.node_start();
let flags = self.tokens.current_flags().as_any_string_flags();
let (Tok::FStringStart(kind), _) = self.bump(TokenKind::FStringStart) else {
unreachable!()
};
let elements = self.parse_fstring_elements();
self.bump(TokenKind::FStringStart);
let elements = self.parse_fstring_elements(flags);
self.expect(TokenKind::FStringEnd);
ast::FString {
elements,
range: self.node_range(start),
flags: kind.into(),
flags: ast::FStringFlags::from(flags),
}
}
@ -1297,16 +1322,18 @@ impl<'src> Parser<'src> {
/// # Panics
///
/// If the parser isn't positioned at a `{` or `FStringMiddle` token.
fn parse_fstring_elements(&mut self) -> FStringElements {
fn parse_fstring_elements(&mut self, flags: ast::AnyStringFlags) -> FStringElements {
let mut elements = vec![];
self.parse_list(RecoveryContextKind::FStringElements, |parser| {
let element = match parser.current_token_kind() {
TokenKind::Lbrace => {
FStringElement::Expression(parser.parse_fstring_expression_element())
FStringElement::Expression(parser.parse_fstring_expression_element(flags))
}
TokenKind::FStringMiddle => {
let (Tok::FStringMiddle { value, flags, .. }, range) = parser.next_token()
let range = parser.current_token_range();
let TokenValue::FStringMiddle(value) =
parser.bump_value(TokenKind::FStringMiddle)
else {
unreachable!()
};
@ -1332,7 +1359,7 @@ impl<'src> Parser<'src> {
// `Invalid` tokens are created when there's a lexical error, so
// we ignore it here to avoid creating unexpected token errors
TokenKind::Unknown => {
parser.next_token();
parser.bump_any();
return;
}
tok => {
@ -1356,7 +1383,10 @@ impl<'src> Parser<'src> {
/// # Panics
///
/// If the parser isn't positioned at a `{` token.
fn parse_fstring_expression_element(&mut self) -> ast::FStringExpressionElement {
fn parse_fstring_expression_element(
&mut self,
flags: ast::AnyStringFlags,
) -> ast::FStringExpressionElement {
let start = self.node_start();
self.bump(TokenKind::Lbrace);
@ -1396,7 +1426,10 @@ impl<'src> Parser<'src> {
let conversion = if self.eat(TokenKind::Exclamation) {
let conversion_flag_range = self.current_token_range();
if let Tok::Name { name } = self.next_token().0 {
if self.at(TokenKind::Name) {
let TokenValue::Name(name) = self.bump_value(TokenKind::Name) else {
unreachable!();
};
match &*name {
"s" => ConversionFlag::Str,
"r" => ConversionFlag::Repr,
@ -1419,6 +1452,8 @@ impl<'src> Parser<'src> {
ParseErrorType::FStringError(FStringErrorType::InvalidConversionFlag),
conversion_flag_range,
);
// TODO(dhruvmanila): Avoid dropping this token
self.bump_any();
ConversionFlag::None
}
} else {
@ -1427,7 +1462,7 @@ impl<'src> Parser<'src> {
let format_spec = if self.eat(TokenKind::Colon) {
let spec_start = self.node_start();
let elements = self.parse_fstring_elements();
let elements = self.parse_fstring_elements(flags);
Some(Box::new(ast::FStringFormatSpec {
range: self.node_range(spec_start),
elements,
@ -2229,7 +2264,8 @@ impl<'src> Parser<'src> {
fn parse_ipython_escape_command_expression(&mut self) -> ast::ExprIpyEscapeCommand {
let start = self.node_start();
let (Tok::IpyEscapeCommand { value, kind }, _) = self.bump(TokenKind::IpyEscapeCommand)
let TokenValue::IpyEscapeCommand { value, kind } =
self.bump_value(TokenKind::IpyEscapeCommand)
else {
unreachable!()
};

368
crates/ruff_python_parser/src/parser/mod.rs
View file

@ -2,20 +2,16 @@ use std::cmp::Ordering;
use bitflags::bitflags;
use ast::Mod;
use ruff_python_ast as ast;
use ruff_python_ast::{Mod, ModExpression, ModModule};
use ruff_text_size::{Ranged, TextRange, TextSize};
use crate::lexer::lex;
use crate::lexer::TokenValue;
use crate::parser::expression::ExpressionContext;
use crate::parser::progress::{ParserProgress, TokenId};
use crate::{
lexer::{LexResult, Spanned},
token_set::TokenSet,
token_source::TokenSource,
Mode, ParseError, ParseErrorType, Tok, TokenKind,
};
use self::expression::ExpressionContext;
use crate::token_set::TokenSet;
use crate::token_source::{TokenSource, TokenSourceCheckpoint};
use crate::{Mode, ParseError, ParseErrorType, TokenKind};
use crate::{Parsed, Tokens};
mod expression;
mod helpers;
@ -26,57 +22,12 @@ mod statement;
#[cfg(test)]
mod tests;
/// Represents the parsed source code.
///
/// This includes the AST and all of the errors encountered during parsing.
#[derive(Debug)]
pub struct Program {
ast: ast::Mod,
parse_errors: Vec<ParseError>,
}
impl Program {
/// Returns the parsed AST.
pub fn ast(&self) -> &ast::Mod {
&self.ast
}
/// Returns a list of syntax errors found during parsing.
pub fn errors(&self) -> &[ParseError] {
&self.parse_errors
}
/// Consumes the [`Program`] and returns the parsed AST.
pub fn into_ast(self) -> ast::Mod {
self.ast
}
/// Consumes the [`Program`] and returns a list of syntax errors found during parsing.
pub fn into_errors(self) -> Vec<ParseError> {
self.parse_errors
}
/// Returns `true` if the program is valid i.e., it has no syntax errors.
pub fn is_valid(&self) -> bool {
self.parse_errors.is_empty()
}
/// Parse the given Python source code using the specified [`Mode`].
pub fn parse_str(source: &str, mode: Mode) -> Program {
let tokens = lex(source, mode);
Self::parse_tokens(source, tokens.collect(), mode)
}
/// Parse a vector of [`LexResult`]s using the specified [`Mode`].
pub fn parse_tokens(source: &str, tokens: Vec<LexResult>, mode: Mode) -> Program {
Parser::new(source, mode, TokenSource::new(tokens)).parse_program()
}
}
#[derive(Debug)]
pub(crate) struct Parser<'src> {
source: &'src str,
tokens: TokenSource,
/// Token source for the parser that skips over any non-trivia token.
tokens: TokenSource<'src>,
/// Stores all the syntax errors found during the parsing.
errors: Vec<ParseError>,
@ -84,37 +35,29 @@ pub(crate) struct Parser<'src> {
/// Specify the mode in which the code will be parsed.
mode: Mode,
/// Current token along with its range.
current: Spanned,
/// The ID of the current token. This is used to track the progress of the parser
/// to avoid infinite loops when the parser is stuck.
current_token_id: TokenId,
/// The end of the last processed. Used to determine a node's end.
last_token_end: TextSize,
/// The range of the tokens to parse.
///
/// The range is equal to `[0; source.len())` when parsing an entire file. The range can be
/// different when parsing only a part of a file using the [`crate::lex_starts_at`] and
/// [`crate::parse_expression_starts_at`] APIs in which case the the range is equal to
/// `[offset; subrange.len())`.
tokens_range: TextRange,
/// The end of the previous token processed. This is used to determine a node's end.
prev_token_end: TextSize,
/// The recovery context in which the parser is currently in.
recovery_context: RecoveryContext,
/// The start offset in the source code from which to start parsing at.
start_offset: TextSize,
}
impl<'src> Parser<'src> {
pub(crate) fn new(source: &'src str, mode: Mode, mut tokens: TokenSource) -> Parser<'src> {
let tokens_range = TextRange::new(
tokens.position().unwrap_or_default(),
tokens.end().unwrap_or_default(),
);
/// Create a new parser for the given source code.
pub(crate) fn new(source: &'src str, mode: Mode) -> Self {
Parser::new_starts_at(source, mode, TextSize::new(0))
}
let current = tokens
.next()
.unwrap_or_else(|| (Tok::EndOfFile, TextRange::empty(tokens_range.end())));
/// Create a new parser for the given source code which starts parsing at the given offset.
pub(crate) fn new_starts_at(source: &'src str, mode: Mode, start_offset: TextSize) -> Self {
let tokens = TokenSource::from_source(source, mode, start_offset);
Parser {
mode,
@ -122,24 +65,20 @@ impl<'src> Parser<'src> {
errors: Vec::new(),
tokens,
recovery_context: RecoveryContext::empty(),
last_token_end: tokens_range.start(),
current,
prev_token_end: TextSize::new(0),
start_offset,
current_token_id: TokenId::default(),
tokens_range,
}
}
/// Consumes the [`Parser`] and returns the parsed [`Program`].
pub(crate) fn parse_program(mut self) -> Program {
let ast = match self.mode {
/// Consumes the [`Parser`] and returns the parsed [`Parsed`].
pub(crate) fn parse(mut self) -> Parsed<Mod> {
let syntax = match self.mode {
Mode::Expression => Mod::Expression(self.parse_single_expression()),
Mode::Module | Mode::Ipython => Mod::Module(self.parse_module()),
};
Program {
ast,
parse_errors: self.finish(),
}
self.finish(syntax)
}
/// Parses a single expression.
@ -150,7 +89,7 @@ impl<'src> Parser<'src> {
///
/// After parsing a single expression, an error is reported and all remaining tokens are
/// dropped by the parser.
fn parse_single_expression(&mut self) -> ast::ModExpression {
fn parse_single_expression(&mut self) -> ModExpression {
let start = self.node_start();
let parsed_expr = self.parse_expression_list(ExpressionContext::default());
@ -170,13 +109,13 @@ impl<'src> Parser<'src> {
if self.at(TokenKind::EndOfFile) {
break;
}
self.next_token();
self.bump_any();
}
}
self.bump(TokenKind::EndOfFile);
ast::ModExpression {
ModExpression {
body: Box::new(parsed_expr.expr),
range: self.node_range(start),
}
@ -185,7 +124,7 @@ impl<'src> Parser<'src> {
/// Parses a Python module.
///
/// This is to be used for [`Mode::Module`] and [`Mode::Ipython`].
fn parse_module(&mut self) -> ast::ModModule {
fn parse_module(&mut self) -> ModModule {
let body = self.parse_list_into_vec(
RecoveryContextKind::ModuleStatements,
Parser::parse_statement,
@ -193,13 +132,13 @@ impl<'src> Parser<'src> {
self.bump(TokenKind::EndOfFile);
ast::ModModule {
ModModule {
body,
range: self.tokens_range,
range: TextRange::new(self.start_offset, self.current_token_range().end()),
}
}
fn finish(self) -> Vec<ParseError> {
fn finish(self, syntax: Mod) -> Parsed<Mod> {
assert_eq!(
self.current_token_kind(),
TokenKind::EndOfFile,
@ -208,13 +147,18 @@ impl<'src> Parser<'src> {
// TODO consider re-integrating lexical error handling into the parser?
let parse_errors = self.errors;
let lex_errors = self.tokens.finish();
let (tokens, comment_ranges, lex_errors) = self.tokens.finish();
// Fast path for when there are no lex errors.
// There's no fast path for when there are no parse errors because a lex error
// always results in a parse error.
if lex_errors.is_empty() {
return parse_errors;
return Parsed {
syntax,
tokens: Tokens::new(tokens),
comment_ranges,
errors: parse_errors,
};
}
let mut merged = Vec::with_capacity(parse_errors.len().saturating_add(lex_errors.len()));
@ -241,7 +185,12 @@ impl<'src> Parser<'src> {
merged.extend(parse_errors);
merged.extend(lex_errors.map(ParseError::from));
merged
Parsed {
syntax,
tokens: Tokens::new(tokens),
comment_ranges,
errors: merged,
}
}
/// Returns the start position for a node that starts at the current token.
@ -280,7 +229,7 @@ impl<'src> Parser<'src> {
//
// In either of the above cases, there's a "gap" between the end of the last token and start
// of the current token.
if self.last_token_end <= start {
if self.prev_token_end <= start {
// We need to create an empty range at the last token end instead of the start because
// otherwise this node range will fall outside the range of it's parent node. Taking
// the above example:
@ -302,9 +251,9 @@ impl<'src> Parser<'src> {
// def foo # comment
// def bar(): ...
// def baz
TextRange::empty(self.last_token_end)
TextRange::empty(self.prev_token_end)
} else {
TextRange::new(start, self.last_token_end)
TextRange::new(start, self.prev_token_end)
}
}
@ -319,65 +268,48 @@ impl<'src> Parser<'src> {
// # ^^^^ expression range
// # ^ last token end
// ```
TextRange::empty(self.last_token_end)
TextRange::empty(self.prev_token_end)
}
/// Moves the parser to the next token.
///
/// Returns the old current token as an owned value.
fn next_token(&mut self) -> Spanned {
let next = self
.tokens
.next()
.unwrap_or_else(|| (Tok::EndOfFile, TextRange::empty(self.tokens_range.end())));
self.current_token_id.increment();
let current = std::mem::replace(&mut self.current, next);
fn do_bump(&mut self, kind: TokenKind) {
if !matches!(
current.0,
self.current_token_kind(),
// TODO explore including everything up to the dedent as part of the body.
Tok::Dedent
TokenKind::Dedent
// Don't include newlines in the body
| Tok::Newline
| TokenKind::Newline
// TODO(micha): Including the semi feels more correct but it isn't compatible with lalrpop and breaks the
// formatters semicolon detection. Exclude it for now
| Tok::Semi
| TokenKind::Semi
) {
self.last_token_end = current.1.end();
self.prev_token_end = self.current_token_range().end();
}
current
self.tokens.bump(kind);
self.current_token_id.increment();
}
/// Returns the next token kind without consuming it.
fn peek(&self) -> TokenKind {
self.tokens
.peek()
.map_or(TokenKind::EndOfFile, |spanned| spanned.0)
fn peek(&mut self) -> TokenKind {
self.tokens.peek()
}
/// Returns the current token kind along with its range.
///
/// Use [`Parser::current_token_kind`] or [`Parser::current_token_range`] to only get the kind
/// or range respectively.
#[inline]
fn current_token(&self) -> (TokenKind, TextRange) {
(self.current_token_kind(), self.current_token_range())
/// Returns the next two token kinds without consuming it.
fn peek2(&mut self) -> (TokenKind, TokenKind) {
self.tokens.peek2()
}
/// Returns the current token kind.
#[inline]
fn current_token_kind(&self) -> TokenKind {
// TODO: Converting the token kind over and over again can be expensive.
TokenKind::from_token(&self.current.0)
self.tokens.current_kind()
}
/// Returns the range of the current token.
#[inline]
fn current_token_range(&self) -> TextRange {
self.current.1
self.tokens.current_range()
}
/// Returns the current token ID.
@ -386,50 +318,88 @@ impl<'src> Parser<'src> {
self.current_token_id
}
/// Eat the current token if it is of the given kind, returning `true` in
/// that case. Otherwise, return `false`.
/// Bumps the current token assuming it is of the given kind.
///
/// # Panics
///
/// If the current token is not of the given kind.
fn bump(&mut self, kind: TokenKind) {
assert_eq!(self.current_token_kind(), kind);
self.do_bump(kind);
}
/// Take the token value from the underlying token source and bump the current token.
///
/// # Panics
///
/// If the current token is not of the given kind.
fn bump_value(&mut self, kind: TokenKind) -> TokenValue {
let value = self.tokens.take_value();
self.bump(kind);
value
}
/// Bumps the current token assuming it is found in the given token set.
///
/// # Panics
///
/// If the current token is not found in the given token set.
fn bump_ts(&mut self, ts: TokenSet) {
let kind = self.current_token_kind();
assert!(ts.contains(kind));
self.do_bump(kind);
}
/// Bumps the current token regardless of its kind and advances to the next token.
///
/// # Panics
///
/// If the parser is at end of file.
fn bump_any(&mut self) {
let kind = self.current_token_kind();
assert_ne!(kind, TokenKind::EndOfFile);
self.do_bump(kind);
}
/// Bumps the soft keyword token as a `Name` token.
///
/// # Panics
///
/// If the current token is not a soft keyword.
pub(crate) fn bump_soft_keyword_as_name(&mut self) {
assert!(self.at_soft_keyword());
self.do_bump(TokenKind::Name);
}
/// Consume the current token if it is of the given kind. Returns `true` if it matches, `false`
/// otherwise.
fn eat(&mut self, kind: TokenKind) -> bool {
if self.at(kind) {
self.next_token();
self.do_bump(kind);
true
} else {
false
}
}
/// Bumps the current token assuming it is of the given kind.
///
/// Returns the current token as an owned value.
///
/// # Panics
///
/// If the current token is not of the given kind.
fn bump(&mut self, kind: TokenKind) -> (Tok, TextRange) {
assert_eq!(self.current_token_kind(), kind);
self.next_token()
}
/// Bumps the current token assuming it is found in the given token set.
///
/// Returns the current token as an owned value.
///
/// # Panics
///
/// If the current token is not found in the given token set.
fn bump_ts(&mut self, ts: TokenSet) -> (Tok, TextRange) {
assert!(ts.contains(self.current_token_kind()));
self.next_token()
}
/// Eat the current token if its of the expected kind, otherwise adds an appropriate error.
fn expect(&mut self, expected: TokenKind) -> bool {
if self.eat(expected) {
return true;
}
let (found, range) = self.current_token();
self.add_error(ParseErrorType::ExpectedToken { found, expected }, range);
self.add_error(
ParseErrorType::ExpectedToken {
found: self.current_token_kind(),
expected,
},
self.current_token_range(),
);
false
}
@ -468,11 +438,7 @@ impl<'src> Parser<'src> {
where
T: Ranged,
{
let range = ranged.range();
// `ranged` uses absolute ranges to the source text of an entire file. Fix the source by
// subtracting the start offset when parsing only a part of a file (when parsing the tokens
// from `lex_starts_at`).
&self.source[range - self.tokens_range.start()]
&self.source[ranged.range()]
}
/// Parses a list of elements into a vector where each element is parsed using
@ -531,7 +497,7 @@ impl<'src> Parser<'src> {
break;
}
self.next_token();
self.bump_any();
}
}
@ -615,7 +581,7 @@ impl<'src> Parser<'src> {
trailing_comma_range = None;
}
self.next_token();
self.bump_any();
}
}
@ -641,6 +607,42 @@ impl<'src> Parser<'src> {
false
}
/// Creates a checkpoint to which the parser can later return to using [`Self::rewind`].
fn checkpoint(&self) -> ParserCheckpoint<'src> {
ParserCheckpoint {
tokens: self.tokens.checkpoint(),
errors_position: self.errors.len(),
current_token_id: self.current_token_id,
prev_token_end: self.prev_token_end,
recovery_context: self.recovery_context,
}
}
/// Restore the parser to the given checkpoint.
fn rewind(&mut self, checkpoint: ParserCheckpoint<'src>) {
let ParserCheckpoint {
tokens,
errors_position,
current_token_id,
prev_token_end,
recovery_context,
} = checkpoint;
self.tokens.rewind(tokens);
self.errors.truncate(errors_position);
self.current_token_id = current_token_id;
self.prev_token_end = prev_token_end;
self.recovery_context = recovery_context;
}
}
struct ParserCheckpoint<'src> {
tokens: TokenSourceCheckpoint<'src>,
errors_position: usize,
current_token_id: TokenId,
prev_token_end: TextSize,
recovery_context: RecoveryContext,
}
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
@ -872,7 +874,7 @@ impl RecoveryContextKind {
fn is_list_terminator(self, p: &Parser) -> bool {
match self {
// The program must consume all tokens until the end
// The parser must consume all tokens until the end
RecoveryContextKind::ModuleStatements => false,
RecoveryContextKind::BlockStatements => p.at(TokenKind::Dedent),
@ -1008,9 +1010,9 @@ impl RecoveryContextKind {
RecoveryContextKind::Except => p.at(TokenKind::Except),
RecoveryContextKind::AssignmentTargets => p.at(TokenKind::Equal),
RecoveryContextKind::TypeParams => p.at_type_param(),
RecoveryContextKind::ImportNames => p.at(TokenKind::Name),
RecoveryContextKind::ImportNames => p.at_name_or_soft_keyword(),
RecoveryContextKind::ImportFromAsNames(_) => {
matches!(p.current_token_kind(), TokenKind::Star | TokenKind::Name)
p.at(TokenKind::Star) || p.at_name_or_soft_keyword()
}
RecoveryContextKind::Slices => p.at(TokenKind::Colon) || p.at_expr(),
RecoveryContextKind::ListElements
@ -1029,11 +1031,13 @@ impl RecoveryContextKind {
RecoveryContextKind::MatchPatternClassArguments => p.at_pattern_start(),
RecoveryContextKind::Arguments => p.at_expr(),
RecoveryContextKind::DeleteTargets => p.at_expr(),
RecoveryContextKind::Identifiers => p.at(TokenKind::Name),
RecoveryContextKind::Parameters(_) => matches!(
p.current_token_kind(),
TokenKind::Name | TokenKind::Star | TokenKind::DoubleStar | TokenKind::Slash
),
RecoveryContextKind::Identifiers => p.at_name_or_soft_keyword(),
RecoveryContextKind::Parameters(_) => {
matches!(
p.current_token_kind(),
TokenKind::Star | TokenKind::DoubleStar | TokenKind::Slash
) || p.at_name_or_soft_keyword()
}
RecoveryContextKind::WithItems(_) => p.at_expr(),
RecoveryContextKind::FStringElements => matches!(
p.current_token_kind(),

106
crates/ruff_python_parser/src/parser/pattern.rs
View file

@ -1,10 +1,11 @@
use ruff_python_ast::{self as ast, Expr, ExprContext, Number, Operator, Pattern, Singleton};
use ruff_text_size::{Ranged, TextSize};
use crate::lexer::TokenValue;
use crate::parser::progress::ParserProgress;
use crate::parser::{recovery, Parser, RecoveryContextKind, SequenceMatchPatternParentheses};
use crate::token_set::TokenSet;
use crate::{ParseErrorType, Tok, TokenKind};
use crate::{ParseErrorType, TokenKind};
use super::expression::ExpressionContext;
@ -50,12 +51,12 @@ const MAPPING_PATTERN_START_SET: TokenSet = TokenSet::new([
impl<'src> Parser<'src> {
/// Returns `true` if the current token is a valid start of a pattern.
pub(super) fn at_pattern_start(&self) -> bool {
self.at_ts(PATTERN_START_SET)
self.at_ts(PATTERN_START_SET) || self.at_soft_keyword()
}
/// Returns `true` if the current token is a valid start of a mapping pattern.
pub(super) fn at_mapping_pattern_start(&self) -> bool {
self.at_ts(MAPPING_PATTERN_START_SET)
self.at_ts(MAPPING_PATTERN_START_SET) || self.at_soft_keyword()
}
/// Entry point to start parsing a pattern.
@ -397,7 +398,7 @@ impl<'src> Parser<'src> {
})
}
TokenKind::Complex => {
let (Tok::Complex { real, imag }, _) = self.bump(TokenKind::Complex) else {
let TokenValue::Complex { real, imag } = self.bump_value(TokenKind::Complex) else {
unreachable!()
};
let range = self.node_range(start);
@ -411,7 +412,7 @@ impl<'src> Parser<'src> {
})
}
TokenKind::Int => {
let (Tok::Int { value }, _) = self.bump(TokenKind::Int) else {
let TokenValue::Int(value) = self.bump_value(TokenKind::Int) else {
unreachable!()
};
let range = self.node_range(start);
@ -425,7 +426,7 @@ impl<'src> Parser<'src> {
})
}
TokenKind::Float => {
let (Tok::Float { value }, _) = self.bump(TokenKind::Float) else {
let TokenValue::Float(value) = self.bump_value(TokenKind::Float) else {
unreachable!()
};
let range = self.node_range(start);
@ -438,46 +439,6 @@ impl<'src> Parser<'src> {
range,
})
}
TokenKind::Name if self.peek() == TokenKind::Dot => {
let (Tok::Name { name }, _) = self.bump(TokenKind::Name) else {
unreachable!()
};
let id = Expr::Name(ast::ExprName {
id: name.to_string(),
ctx: ExprContext::Load,
range: self.node_range(start),
});
let attribute = self.parse_attr_expr_for_match_pattern(id, start);
Pattern::MatchValue(ast::PatternMatchValue {
value: Box::new(attribute),
range: self.node_range(start),
})
}
TokenKind::Name => {
let (Tok::Name { name }, _) = self.bump(TokenKind::Name) else {
unreachable!()
};
let range = self.node_range(start);
// test_ok match_as_pattern
// match foo:
// case foo_bar: ...
// case _: ...
Pattern::MatchAs(ast::PatternMatchAs {
range,
pattern: None,
name: if &*name == "_" {
None
} else {
Some(ast::Identifier {
id: name.to_string(),
range,
})
},
})
}
kind => {
// The `+` is only for better error recovery.
if let Some(unary_arithmetic_op) = kind.as_unary_arithmetic_operator() {
@ -506,26 +467,57 @@ impl<'src> Parser<'src> {
}
}
// Upon encountering an unexpected token, return a `Pattern::MatchValue` containing
// an empty `Expr::Name`.
let invalid_node = if kind.is_keyword() {
Expr::Name(self.parse_name())
if self.at_name_or_keyword() {
if self.peek() == TokenKind::Dot {
// test_ok match_attr_pattern_soft_keyword
// match foo:
// case match.bar: ...
// case case.bar: ...
// case type.bar: ...
// case match.case.type.bar.type.case.match: ...
let id = Expr::Name(self.parse_name());
let attribute = self.parse_attr_expr_for_match_pattern(id, start);
Pattern::MatchValue(ast::PatternMatchValue {
value: Box::new(attribute),
range: self.node_range(start),
})
} else {
// test_ok match_as_pattern_soft_keyword
// match foo:
// case case: ...
// case match: ...
// case type: ...
let ident = self.parse_identifier();
// test_ok match_as_pattern
// match foo:
// case foo_bar: ...
// case _: ...
Pattern::MatchAs(ast::PatternMatchAs {
range: ident.range,
pattern: None,
name: if &ident == "_" { None } else { Some(ident) },
})
}
} else {
// Upon encountering an unexpected token, return a `Pattern::MatchValue` containing
// an empty `Expr::Name`.
self.add_error(
ParseErrorType::OtherError("Expected a pattern".to_string()),
self.current_token_range(),
);
Expr::Name(ast::ExprName {
let invalid_node = Expr::Name(ast::ExprName {
range: self.missing_node_range(),
id: String::new(),
ctx: ExprContext::Invalid,
});
Pattern::MatchValue(ast::PatternMatchValue {
range: invalid_node.range(),
value: Box::new(invalid_node),
})
};
Pattern::MatchValue(ast::PatternMatchValue {
range: invalid_node.range(),
value: Box::new(invalid_node),
})
}
}
}
}

353
crates/ruff_python_parser/src/parser/statement.rs
View file

@ -8,13 +8,14 @@ use ruff_python_ast::{
};
use ruff_text_size::{Ranged, TextSize};
use crate::lexer::TokenValue;
use crate::parser::expression::{GeneratorExpressionInParentheses, ParsedExpr, EXPR_SET};
use crate::parser::progress::ParserProgress;
use crate::parser::{
helpers, FunctionKind, Parser, RecoveryContext, RecoveryContextKind, WithItemKind,
};
use crate::token_set::TokenSet;
use crate::{Mode, ParseErrorType, Tok, TokenKind};
use crate::{Mode, ParseErrorType, TokenKind};
use super::expression::{ExpressionContext, OperatorPrecedence};
use super::Parenthesized;
@ -84,13 +85,13 @@ impl<'src> Parser<'src> {
/// Returns `true` if the current token is the start of a simple statement,
/// including expressions.
fn at_simple_stmt(&self) -> bool {
self.at_ts(SIMPLE_STMT_WITH_EXPR_SET)
self.at_ts(SIMPLE_STMT_WITH_EXPR_SET) || self.at_soft_keyword()
}
/// Returns `true` if the current token is the start of a simple, compound or expression
/// statement.
pub(super) fn at_stmt(&self) -> bool {
self.at_ts(STMTS_SET)
self.at_ts(STMTS_SET) || self.at_soft_keyword()
}
/// Checks if the parser is currently positioned at the start of a type parameter.
@ -120,8 +121,26 @@ impl<'src> Parser<'src> {
TokenKind::With => Stmt::With(self.parse_with_statement(start)),
TokenKind::At => self.parse_decorators(),
TokenKind::Async => self.parse_async_statement(),
TokenKind::Match => Stmt::Match(self.parse_match_statement()),
_ => self.parse_single_simple_statement(),
token => {
if token == TokenKind::Match {
// Match is considered a soft keyword, so we will treat it as an identifier if
// it's followed by an unexpected token.
match self.classify_match_token() {
MatchTokenKind::Keyword => {
return Stmt::Match(self.parse_match_statement());
}
MatchTokenKind::KeywordOrIdentifier => {
if let Some(match_stmt) = self.try_parse_match_statement() {
return Stmt::Match(match_stmt);
}
}
MatchTokenKind::Identifier => {}
}
}
self.parse_single_simple_statement()
}
}
}
@ -252,11 +271,22 @@ impl<'src> Parser<'src> {
TokenKind::Assert => Stmt::Assert(self.parse_assert_statement()),
TokenKind::Global => Stmt::Global(self.parse_global_statement()),
TokenKind::Nonlocal => Stmt::Nonlocal(self.parse_nonlocal_statement()),
TokenKind::Type => Stmt::TypeAlias(self.parse_type_alias_statement()),
TokenKind::IpyEscapeCommand => {
Stmt::IpyEscapeCommand(self.parse_ipython_escape_command_statement())
}
_ => {
token => {
if token == TokenKind::Type {
// Type is considered a soft keyword, so we will treat it as an identifier if
// it's followed by an unexpected token.
let (first, second) = self.peek2();
if (first == TokenKind::Name || first.is_soft_keyword())
&& matches!(second, TokenKind::Lsqb | TokenKind::Equal)
{
return Stmt::TypeAlias(self.parse_type_alias_statement());
}
}
let start = self.node_start();
// simple_stmt: `... | yield_stmt | star_expressions | ...`
@ -498,7 +528,12 @@ impl<'src> Parser<'src> {
}
}
let module = if self.at(TokenKind::Name) {
let module = if self.at_name_or_soft_keyword() {
// test_ok from_import_soft_keyword_module_name
// from match import pattern
// from type import bar
// from case import pattern
// from match.type.case import foo
Some(self.parse_dotted_name())
} else {
if leading_dots == 0 {
@ -603,7 +638,11 @@ impl<'src> Parser<'src> {
};
let asname = if self.eat(TokenKind::As) {
if self.at(TokenKind::Name) {
if self.at_name_or_soft_keyword() {
// test_ok import_as_name_soft_keyword
// import foo as match
// import bar as case
// import baz as type
Some(self.parse_identifier())
} else {
// test_err import_alias_missing_asname
@ -872,7 +911,8 @@ impl<'src> Parser<'src> {
fn parse_ipython_escape_command_statement(&mut self) -> ast::StmtIpyEscapeCommand {
let start = self.node_start();
let (Tok::IpyEscapeCommand { value, kind }, _) = self.bump(TokenKind::IpyEscapeCommand)
let TokenValue::IpyEscapeCommand { value, kind } =
self.bump_value(TokenKind::IpyEscapeCommand)
else {
unreachable!()
};
@ -1469,7 +1509,12 @@ impl<'src> Parser<'src> {
};
let name = if self.eat(TokenKind::As) {
if self.at(TokenKind::Name) {
if self.at_name_or_soft_keyword() {
// test_ok except_stmt_as_name_soft_keyword
// try: ...
// except Exception as match: ...
// except Exception as case: ...
// except Exception as type: ...
Some(self.parse_identifier())
} else {
// test_err except_stmt_missing_as_name
@ -2327,6 +2372,84 @@ impl<'src> Parser<'src> {
target
}
/// Try parsing a `match` statement.
///
/// This uses speculative parsing to remove the ambiguity of whether the `match` token is used
/// as a keyword or an identifier. This ambiguity arises only in if the `match` token is
/// followed by certain tokens. For example, if `match` is followed by `[`, we can't know if
/// it's used in the context of a subscript expression or as a list expression:
///
/// ```python
/// # Subcript expression; `match` is an identifier
/// match[x]
///
/// # List expression; `match` is a keyword
/// match [x, y]:
/// case [1, 2]:
/// pass
/// ```
///
/// This is done by parsing the subject expression considering `match` as a keyword token.
/// Then, based on certain heuristics we'll determine if our assumption is true. If so, we'll
/// continue parsing the entire match statement. Otherwise, return `None`.
///
/// # Panics
///
/// If the parser isn't positioned at a `match` token.
///
/// See: <https://docs.python.org/3/reference/compound_stmts.html#the-match-statement>
fn try_parse_match_statement(&mut self) -> Option<ast::StmtMatch> {
let checkpoint = self.checkpoint();
let start = self.node_start();
self.bump(TokenKind::Match);
let subject = self.parse_match_subject_expression();
match self.current_token_kind() {
TokenKind::Colon => {
// `match` is a keyword
self.bump(TokenKind::Colon);
let cases = self.parse_match_body();
Some(ast::StmtMatch {
subject: Box::new(subject),
cases,
range: self.node_range(start),
})
}
TokenKind::Newline if matches!(self.peek2(), (TokenKind::Indent, TokenKind::Case)) => {
// `match` is a keyword
// test_err match_expected_colon
// match [1, 2]
// case _: ...
self.add_error(
ParseErrorType::ExpectedToken {
found: self.current_token_kind(),
expected: TokenKind::Colon,
},
self.current_token_range(),
);
let cases = self.parse_match_body();
Some(ast::StmtMatch {
subject: Box::new(subject),
cases,
range: self.node_range(start),
})
}
_ => {
// `match` is an identifier
self.rewind(checkpoint);
None
}
}
}
/// Parses a match statement.
///
/// # Panics
@ -2338,7 +2461,21 @@ impl<'src> Parser<'src> {
let start = self.node_start();
self.bump(TokenKind::Match);
let subject_start = self.node_start();
let subject = self.parse_match_subject_expression();
self.expect(TokenKind::Colon);
let cases = self.parse_match_body();
ast::StmtMatch {
subject: Box::new(subject),
cases,
range: self.node_range(start),
}
}
/// Parses the subject expression for a `match` statement.
fn parse_match_subject_expression(&mut self) -> Expr {
let start = self.node_start();
// Subject expression grammar is:
//
@ -2370,13 +2507,12 @@ impl<'src> Parser<'src> {
// case _: ...
// match yield x:
// case _: ...
let subject = if self.at(TokenKind::Comma) {
let tuple =
self.parse_tuple_expression(subject.expr, subject_start, Parenthesized::No, |p| {
p.parse_named_expression_or_higher(ExpressionContext::starred_bitwise_or())
});
if self.at(TokenKind::Comma) {
let tuple = self.parse_tuple_expression(subject.expr, start, Parenthesized::No, |p| {
p.parse_named_expression_or_higher(ExpressionContext::starred_bitwise_or())
});
Expr::Tuple(tuple).into()
Expr::Tuple(tuple)
} else {
if subject.is_unparenthesized_starred_expr() {
// test_err match_stmt_single_starred_subject
@ -2384,11 +2520,15 @@ impl<'src> Parser<'src> {
// case _: ...
self.add_error(ParseErrorType::InvalidStarredExpressionUsage, &subject);
}
subject
};
self.expect(TokenKind::Colon);
subject.expr
}
}
/// Parses the body of a `match` statement.
///
/// This method expects that the parser is positioned at a `Newline` token. If not, it adds a
/// syntax error and continues parsing.
fn parse_match_body(&mut self) -> Vec<ast::MatchCase> {
// test_err match_stmt_no_newline_before_case
// match foo: case _: ...
self.expect(TokenKind::Newline);
@ -2411,11 +2551,7 @@ impl<'src> Parser<'src> {
// TODO(dhruvmanila): Should we expect `Dedent` only if there was an `Indent` present?
self.expect(TokenKind::Dedent);
ast::StmtMatch {
subject: Box::new(subject.expr),
cases,
range: self.node_range(start),
}
cases
}
/// Parses a list of match case blocks.
@ -2458,7 +2594,6 @@ impl<'src> Parser<'src> {
self.bump(TokenKind::Case);
// test_err match_stmt_missing_pattern
// # TODO(dhruvmanila): Here, `case` is a name token because of soft keyword transformer
// match x:
// case : ...
let pattern = self.parse_match_patterns();
@ -2557,8 +2692,6 @@ impl<'src> Parser<'src> {
// async while test: ...
// async x = 1
// async async def foo(): ...
// # TODO(dhruvmanila): Here, `match` is actually a Name token because
// # of the soft keyword # transformer
// async match test:
// case _: ...
self.add_error(
@ -2890,7 +3023,7 @@ impl<'src> Parser<'src> {
let star_range = parser.current_token_range();
parser.bump(TokenKind::Star);
if parser.at(TokenKind::Name) {
if parser.at_name_or_soft_keyword() {
let param = parser.parse_parameter(param_start, function_kind, AllowStarAnnotation::Yes);
let param_star_range = parser.node_range(star_range.start());
@ -3049,7 +3182,7 @@ impl<'src> Parser<'src> {
last_keyword_only_separator_range = None;
}
TokenKind::Name => {
_ if parser.at_name_or_soft_keyword() => {
let param = parser.parse_parameter_with_default(param_start, function_kind);
// TODO(dhruvmanila): Pyright seems to only highlight the first non-default argument
@ -3386,6 +3519,122 @@ impl<'src> Parser<'src> {
}
}
/// Classify the `match` soft keyword token.
///
/// # Panics
///
/// If the parser isn't positioned at a `match` token.
fn classify_match_token(&mut self) -> MatchTokenKind {
assert_eq!(self.current_token_kind(), TokenKind::Match);
let (first, second) = self.peek2();
match first {
// test_ok match_classify_as_identifier_1
// match not in case
TokenKind::Not if second == TokenKind::In => MatchTokenKind::Identifier,
// test_ok match_classify_as_keyword_1
// match foo:
// case _: ...
// match 1:
// case _: ...
// match 1.0:
// case _: ...
// match 1j:
// case _: ...
// match "foo":
// case _: ...
// match f"foo {x}":
// case _: ...
// match {1, 2}:
// case _: ...
// match ~foo:
// case _: ...
// match ...:
// case _: ...
// match not foo:
// case _: ...
// match await foo():
// case _: ...
// match lambda foo: foo:
// case _: ...
// test_err match_classify_as_keyword
// match yield foo:
// case _: ...
TokenKind::Name
| TokenKind::Int
| TokenKind::Float
| TokenKind::Complex
| TokenKind::String
| TokenKind::FStringStart
| TokenKind::Lbrace
| TokenKind::Tilde
| TokenKind::Ellipsis
| TokenKind::Not
| TokenKind::Await
| TokenKind::Yield
| TokenKind::Lambda => MatchTokenKind::Keyword,
// test_ok match_classify_as_keyword_or_identifier
// match (1, 2) # Identifier
// match (1, 2): # Keyword
// case _: ...
// match [1:] # Identifier
// match [1, 2]: # Keyword
// case _: ...
// match * foo # Identifier
// match - foo # Identifier
// match -foo: # Keyword
// case _: ...
// test_err match_classify_as_keyword_or_identifier
// match *foo: # Keyword
// case _: ...
TokenKind::Lpar
| TokenKind::Lsqb
| TokenKind::Star
| TokenKind::Plus
| TokenKind::Minus => MatchTokenKind::KeywordOrIdentifier,
_ => {
if first.is_soft_keyword() || first.is_singleton() {
// test_ok match_classify_as_keyword_2
// match match:
// case _: ...
// match case:
// case _: ...
// match type:
// case _: ...
// match None:
// case _: ...
// match True:
// case _: ...
// match False:
// case _: ...
MatchTokenKind::Keyword
} else {
// test_ok match_classify_as_identifier_2
// match
// match != foo
// (foo, match)
// [foo, match]
// {foo, match}
// match;
// match: int
// match,
// match.foo
// match / foo
// match << foo
// match and foo
// match is not foo
MatchTokenKind::Identifier
}
}
}
}
/// Specialized [`Parser::parse_list_into_vec`] for parsing a sequence of clauses.
///
/// The difference is that the parser only continues parsing for as long as it sees the token
@ -3477,6 +3726,46 @@ impl Display for Clause {
}
}
/// The classification of the `match` token.
///
/// The `match` token is a soft keyword which means, depending on the context, it can be used as a
/// keyword or an identifier.
#[derive(Debug, Clone, Copy)]
enum MatchTokenKind {
/// The `match` token is used as a keyword.
///
/// For example:
/// ```python
/// match foo:
/// case _:
/// pass
/// ```
Keyword,
/// The `match` token is used as an identifier.
///
/// For example:
/// ```python
/// match.values()
/// match is None
/// ````
Identifier,
/// The `match` token is used as either a keyword or an identifier.
///
/// For example:
/// ```python
/// # Used as a keyword
/// match [x, y]:
/// case [1, 2]:
/// pass
///
/// # Used as an identifier
/// match[x]
/// ```
KeywordOrIdentifier,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum WithItemParsingState {
/// The parser is currently parsing a with item without any ambiguity.

30
crates/ruff_python_parser/src/parser/tests.rs
View file

@ -1,4 +1,4 @@
use crate::{lex, parse, parse_expression, parse_suite, parse_tokens, Mode};
use crate::{parse, parse_expression, parse_module, Mode};
#[test]
fn test_modes() {
@ -45,23 +45,23 @@ fn test_expr_mode_valid_syntax() {
let source = "first
";
let expr = parse_expression(source).unwrap();
let parsed = parse_expression(source).unwrap();
insta::assert_debug_snapshot!(expr);
insta::assert_debug_snapshot!(parsed.expr());
}
#[test]
fn test_unicode_aliases() {
// https://github.com/RustPython/RustPython/issues/4566
let source = r#"x = "\N{BACKSPACE}another cool trick""#;
let parse_ast = parse_suite(source).unwrap();
let suite = parse_module(source).unwrap().into_suite();
insta::assert_debug_snapshot!(parse_ast);
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_ipython_escape_commands() {
let parse_ast = parse(
let parsed = parse(
r"
# Normal Python code
(
@ -132,21 +132,5 @@ foo.bar[0].baz[2].egg??
Mode::Ipython,
)
.unwrap();
insta::assert_debug_snapshot!(parse_ast);
}
#[test]
fn test_ipython_escape_command_parse_error() {
let source = r"
a = 1
%timeit a == 1
"
.trim();
let lxr = lex(source, Mode::Ipython);
let parse_err = parse_tokens(lxr.collect(), source, Mode::Module).unwrap_err();
assert_eq!(
parse_err.to_string(),
"IPython escape commands are only allowed in `Mode::Ipython` at byte range 6..20"
.to_string()
);
insta::assert_debug_snapshot!(parsed.syntax());
}

27
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__assignment.snap
View file

@ -2,11 +2,13 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
Name {
name: "a_variable",
},
Name(
"a_variable",
),
0..10,
),
(
@ -14,9 +16,9 @@ expression: lex_source(source)
11..12,
),
(
Int {
value: 99,
},
Int(
99,
),
13..15,
),
(
@ -24,9 +26,9 @@ expression: lex_source(source)
16..17,
),
(
Int {
value: 2,
},
Int(
2,
),
18..19,
),
(
@ -34,9 +36,9 @@ expression: lex_source(source)
19..20,
),
(
Int {
value: 0,
},
Int(
0,
),
20..21,
),
(
@ -44,3 +46,4 @@ expression: lex_source(source)
21..21,
),
]
```

19
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__comment_until_mac_eol.snap
View file

@ -2,17 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: comment_until_eol(MAC_EOL)
---
## Tokens
```
[
(
Int {
value: 123,
},
Int(
123,
),
0..3,
),
(
Comment(
"# Foo",
),
Comment,
5..10,
),
(
@ -20,9 +20,9 @@ expression: comment_until_eol(MAC_EOL)
10..11,
),
(
Int {
value: 456,
},
Int(
456,
),
11..14,
),
(
@ -30,3 +30,4 @@ expression: comment_until_eol(MAC_EOL)
14..14,
),
]
```

19
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__comment_until_unix_eol.snap
View file

@ -2,17 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: comment_until_eol(UNIX_EOL)
---
## Tokens
```
[
(
Int {
value: 123,
},
Int(
123,
),
0..3,
),
(
Comment(
"# Foo",
),
Comment,
5..10,
),
(
@ -20,9 +20,9 @@ expression: comment_until_eol(UNIX_EOL)
10..11,
),
(
Int {
value: 456,
},
Int(
456,
),
11..14,
),
(
@ -30,3 +30,4 @@ expression: comment_until_eol(UNIX_EOL)
14..14,
),
]
```

19
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__comment_until_windows_eol.snap
View file

@ -2,17 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: comment_until_eol(WINDOWS_EOL)
---
## Tokens
```
[
(
Int {
value: 123,
},
Int(
123,
),
0..3,
),
(
Comment(
"# Foo",
),
Comment,
5..10,
),
(
@ -20,9 +20,9 @@ expression: comment_until_eol(WINDOWS_EOL)
10..12,
),
(
Int {
value: 456,
},
Int(
456,
),
12..15,
),
(
@ -30,3 +30,4 @@ expression: comment_until_eol(WINDOWS_EOL)
15..15,
),
]
```

79
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__dedent_after_whitespace.snap Normal file
View file

@ -0,0 +1,79 @@
---
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
If,
0..2,
),
(
Name(
"first",
),
3..8,
),
(
Colon,
8..9,
),
(
Newline,
9..10,
),
(
Indent,
10..14,
),
(
If,
14..16,
),
(
Name(
"second",
),
17..23,
),
(
Colon,
23..24,
),
(
Newline,
24..25,
),
(
Indent,
25..33,
),
(
Pass,
33..37,
),
(
Newline,
37..38,
),
(
Dedent,
42..42,
),
(
Name(
"foo",
),
42..45,
),
(
Newline,
45..46,
),
(
Dedent,
46..46,
),
]
```

21
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__double_dedent_with_mac_eol.snap
View file

@ -2,15 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: double_dedent_with_eol(MAC_EOL)
---
## Tokens
```
[
(
Def,
0..3,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
4..7,
),
(
@ -38,9 +40,9 @@ expression: double_dedent_with_eol(MAC_EOL)
12..14,
),
(
Name {
name: "x",
},
Name(
"x",
),
15..16,
),
(
@ -64,9 +66,9 @@ expression: double_dedent_with_eol(MAC_EOL)
21..27,
),
(
Int {
value: 99,
},
Int(
99,
),
28..30,
),
(
@ -86,3 +88,4 @@ expression: double_dedent_with_eol(MAC_EOL)
32..32,
),
]
```

21
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__double_dedent_with_tabs_mac_eol.snap
View file

@ -2,15 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: double_dedent_with_tabs_eol(MAC_EOL)
---
## Tokens
```
[
(
Def,
0..3,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
4..7,
),
(
@ -38,9 +40,9 @@ expression: double_dedent_with_tabs_eol(MAC_EOL)
12..14,
),
(
Name {
name: "x",
},
Name(
"x",
),
15..16,
),
(
@ -64,9 +66,9 @@ expression: double_dedent_with_tabs_eol(MAC_EOL)
22..28,
),
(
Int {
value: 99,
},
Int(
99,
),
29..31,
),
(
@ -86,3 +88,4 @@ expression: double_dedent_with_tabs_eol(MAC_EOL)
33..33,
),
]
```

21
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__double_dedent_with_tabs_unix_eol.snap
View file

@ -2,15 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: double_dedent_with_tabs_eol(UNIX_EOL)
---
## Tokens
```
[
(
Def,
0..3,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
4..7,
),
(
@ -38,9 +40,9 @@ expression: double_dedent_with_tabs_eol(UNIX_EOL)
12..14,
),
(
Name {
name: "x",
},
Name(
"x",
),
15..16,
),
(
@ -64,9 +66,9 @@ expression: double_dedent_with_tabs_eol(UNIX_EOL)
22..28,
),
(
Int {
value: 99,
},
Int(
99,
),
29..31,
),
(
@ -86,3 +88,4 @@ expression: double_dedent_with_tabs_eol(UNIX_EOL)
33..33,
),
]
```

21
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__double_dedent_with_tabs_windows_eol.snap
View file

@ -2,15 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: double_dedent_with_tabs_eol(WINDOWS_EOL)
---
## Tokens
```
[
(
Def,
0..3,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
4..7,
),
(
@ -38,9 +40,9 @@ expression: double_dedent_with_tabs_eol(WINDOWS_EOL)
13..15,
),
(
Name {
name: "x",
},
Name(
"x",
),
16..17,
),
(
@ -64,9 +66,9 @@ expression: double_dedent_with_tabs_eol(WINDOWS_EOL)
25..31,
),
(
Int {
value: 99,
},
Int(
99,
),
32..34,
),
(
@ -86,3 +88,4 @@ expression: double_dedent_with_tabs_eol(WINDOWS_EOL)
38..38,
),
]
```

21
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__double_dedent_with_unix_eol.snap
View file

@ -2,15 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: double_dedent_with_eol(UNIX_EOL)
---
## Tokens
```
[
(
Def,
0..3,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
4..7,
),
(
@ -38,9 +40,9 @@ expression: double_dedent_with_eol(UNIX_EOL)
12..14,
),
(
Name {
name: "x",
},
Name(
"x",
),
15..16,
),
(
@ -64,9 +66,9 @@ expression: double_dedent_with_eol(UNIX_EOL)
21..27,
),
(
Int {
value: 99,
},
Int(
99,
),
28..30,
),
(
@ -86,3 +88,4 @@ expression: double_dedent_with_eol(UNIX_EOL)
32..32,
),
]
```

21
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__double_dedent_with_windows_eol.snap
View file

@ -2,15 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: double_dedent_with_eol(WINDOWS_EOL)
---
## Tokens
```
[
(
Def,
0..3,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
4..7,
),
(
@ -38,9 +40,9 @@ expression: double_dedent_with_eol(WINDOWS_EOL)
13..15,
),
(
Name {
name: "x",
},
Name(
"x",
),
16..17,
),
(
@ -64,9 +66,9 @@ expression: double_dedent_with_eol(WINDOWS_EOL)
24..30,
),
(
Int {
value: 99,
},
Int(
99,
),
31..33,
),
(
@ -86,3 +88,4 @@ expression: double_dedent_with_eol(WINDOWS_EOL)
37..37,
),
]
```

24
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__emoji_identifier.snap Normal file
View file

@ -0,0 +1,24 @@
---
source: crates/ruff_python_parser/src/lexer.rs
expression: "lex_invalid(source, Mode::Module)"
---
## Tokens
```
[
(
Unknown,
0..4,
),
]
```
## Errors
```
[
LexicalError {
error: UnrecognizedToken {
tok: '🐦',
},
location: 0..4,
},
]
```

120
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__empty_fstrings.snap
View file

@ -2,115 +2,97 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
2..3,
),
(
String {
value: "",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Double,
},
},
4..6,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
String(
"",
),
4..6,
TokenFlags(
DOUBLE_QUOTES,
),
),
(
FStringStart,
7..9,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
9..10,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
),
FStringStart,
11..13,
TokenFlags(
F_STRING,
),
),
(
FStringEnd,
13..14,
TokenFlags(
F_STRING,
),
),
(
String {
value: "",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"",
),
15..17,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Double,
},
),
FStringStart,
18..22,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringEnd,
22..25,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
),
FStringStart,
26..30,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringEnd,
30..33,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Newline,
33..33,
),
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__empty_ipython_escape_command.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_jupyter_source(source)
---
## Tokens
```
[
(
IpyEscapeCommand {
@ -103,3 +105,4 @@ expression: lex_jupyter_source(source)
20..20,
),
]
```

19
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__escape_unicode_name.snap
View file

@ -2,22 +2,21 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
String {
value: "\\N{EN SPACE}",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Double,
},
},
String(
"\\N{EN SPACE}",
),
0..14,
TokenFlags(
DOUBLE_QUOTES,
),
),
(
Newline,
14..14,
),
]
```

101
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring.snap
View file

@ -2,40 +2,33 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle {
value: "normal ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"normal ",
),
2..9,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
9..10,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
10..13,
),
(
@ -43,26 +36,22 @@ expression: lex_source(source)
13..14,
),
(
FStringMiddle {
value: " {another} ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" {another} ",
),
14..27,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
27..28,
),
(
Name {
name: "bar",
},
Name(
"bar",
),
28..31,
),
(
@ -70,26 +59,22 @@ expression: lex_source(source)
31..32,
),
(
FStringMiddle {
value: " {",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" {",
),
32..35,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
35..36,
),
(
Name {
name: "three",
},
Name(
"three",
),
36..41,
),
(
@ -97,24 +82,24 @@ expression: lex_source(source)
41..42,
),
(
FStringMiddle {
value: "}",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"}",
),
42..44,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
44..45,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
45..45,
),
]
```

61
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_comments.snap
View file

@ -2,40 +2,31 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Double,
},
),
FStringStart,
0..4,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringMiddle {
value: "\n# not a comment ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Double,
},
},
FStringMiddle(
"\n# not a comment ",
),
4..21,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Lbrace,
21..22,
),
(
Comment(
"# comment {",
),
Comment,
23..34,
),
(
@ -43,9 +34,9 @@ expression: lex_source(source)
34..35,
),
(
Name {
name: "x",
},
Name(
"x",
),
39..40,
),
(
@ -57,24 +48,24 @@ expression: lex_source(source)
41..42,
),
(
FStringMiddle {
value: " # not a comment\n",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Double,
},
},
FStringMiddle(
" # not a comment\n",
),
42..59,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringEnd,
59..62,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Newline,
62..62,
),
]
```

113
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_conversion.snap
View file

@ -2,27 +2,24 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
2..3,
),
(
Name {
name: "x",
},
Name(
"x",
),
3..4,
),
(
@ -30,9 +27,9 @@ expression: lex_source(source)
4..5,
),
(
Name {
name: "s",
},
Name(
"s",
),
5..6,
),
(
@ -40,26 +37,22 @@ expression: lex_source(source)
6..7,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" ",
),
7..8,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
8..9,
),
(
Name {
name: "x",
},
Name(
"x",
),
9..10,
),
(
@ -71,9 +64,9 @@ expression: lex_source(source)
11..12,
),
(
Name {
name: "r",
},
Name(
"r",
),
12..13,
),
(
@ -81,26 +74,22 @@ expression: lex_source(source)
13..14,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" ",
),
14..15,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
15..16,
),
(
Name {
name: "x",
},
Name(
"x",
),
16..17,
),
(
@ -108,41 +97,37 @@ expression: lex_source(source)
17..18,
),
(
FStringMiddle {
value: ".3f!r",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
".3f!r",
),
18..23,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Rbrace,
23..24,
),
(
FStringMiddle {
value: " {x!r}",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" {x!r}",
),
24..32,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
32..33,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
33..33,
),
]
```

79
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_escape.snap
View file

@ -2,40 +2,33 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle {
value: "\\",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"\\",
),
2..3,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
3..4,
),
(
Name {
name: "x",
},
Name(
"x",
),
4..5,
),
(
@ -43,26 +36,22 @@ expression: lex_source(source)
5..6,
),
(
FStringMiddle {
value: "\\\"\\",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"\\\"\\",
),
6..9,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
9..10,
),
(
Name {
name: "x",
},
Name(
"x",
),
10..11,
),
(
@ -74,24 +63,24 @@ expression: lex_source(source)
12..13,
),
(
FStringMiddle {
value: " \\\"\\\"\\\n end",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" \\\"\\\"\\\n end",
),
13..24,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
24..25,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
25..25,
),
]
```

155
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_escape_braces.snap
View file

@ -2,40 +2,33 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
),
FStringStart,
0..2,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle {
value: "\\",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"\\",
),
2..3,
TokenFlags(
F_STRING,
),
),
(
Lbrace,
3..4,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
4..7,
),
(
@ -45,40 +38,34 @@ expression: lex_source(source)
(
FStringEnd,
8..9,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
TokenFlags(
F_STRING,
),
10..12,
),
(
FStringMiddle {
value: "\\\\",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringStart,
10..12,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle(
"\\\\",
),
12..14,
TokenFlags(
F_STRING,
),
),
(
Lbrace,
14..15,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
15..18,
),
(
@ -88,67 +75,59 @@ expression: lex_source(source)
(
FStringEnd,
19..20,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
TokenFlags(
F_STRING,
),
21..23,
),
(
FStringMiddle {
value: "\\{foo}",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringStart,
21..23,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle(
"\\{foo}",
),
23..31,
TokenFlags(
F_STRING,
),
),
(
FStringEnd,
31..32,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
TokenFlags(
F_STRING,
),
33..35,
),
(
FStringMiddle {
value: "\\\\{foo}",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringStart,
33..35,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle(
"\\\\{foo}",
),
35..44,
TokenFlags(
F_STRING,
),
),
(
FStringEnd,
44..45,
TokenFlags(
F_STRING,
),
),
(
Newline,
45..45,
),
]
```

87
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_escape_raw.snap
View file

@ -2,44 +2,33 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..3,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringMiddle {
value: "\\",
flags: AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"\\",
),
3..4,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
Lbrace,
4..5,
),
(
Name {
name: "x",
},
Name(
"x",
),
5..6,
),
(
@ -47,28 +36,22 @@ expression: lex_source(source)
6..7,
),
(
FStringMiddle {
value: "\\\"\\",
flags: AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"\\\"\\",
),
7..10,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
Lbrace,
10..11,
),
(
Name {
name: "x",
},
Name(
"x",
),
11..12,
),
(
@ -80,26 +63,24 @@ expression: lex_source(source)
13..14,
),
(
FStringMiddle {
value: " \\\"\\\"\\\n end",
flags: AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" \\\"\\\"\\\n end",
),
14..25,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringEnd,
25..26,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
Newline,
26..26,
),
]
```

63
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_expression_multiline.snap
View file

@ -2,31 +2,24 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle {
value: "first ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"first ",
),
2..8,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -37,9 +30,9 @@ expression: lex_source(source)
9..10,
),
(
Name {
name: "x",
},
Name(
"x",
),
14..15,
),
(
@ -55,9 +48,9 @@ expression: lex_source(source)
25..26,
),
(
Name {
name: "y",
},
Name(
"y",
),
38..39,
),
(
@ -69,24 +62,24 @@ expression: lex_source(source)
40..41,
),
(
FStringMiddle {
value: " second",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" second",
),
41..48,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
48..49,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
49..49,
),
]
```

161
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_multiline.snap
View file

@ -2,127 +2,99 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Double,
},
),
FStringStart,
0..4,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringMiddle {
value: "\nhello\n world\n",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Double,
},
},
FStringMiddle(
"\nhello\n world\n",
),
4..21,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringEnd,
21..24,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
25..29,
),
(
FStringMiddle {
value: "\n world\nhello\n",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
},
FStringStart,
25..29,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringMiddle(
"\n world\nhello\n",
),
29..46,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringEnd,
46..49,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
50..52,
),
(
FStringMiddle {
value: "some ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringStart,
50..52,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle(
"some ",
),
52..57,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
57..58,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Double,
},
),
FStringStart,
58..62,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringMiddle {
value: "multiline\nallowed ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Double,
},
},
FStringMiddle(
"multiline\nallowed ",
),
62..80,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Lbrace,
80..81,
),
(
Name {
name: "x",
},
Name(
"x",
),
81..82,
),
(
@ -132,30 +104,33 @@ expression: lex_source(source)
(
FStringEnd,
83..86,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Rbrace,
86..87,
),
(
FStringMiddle {
value: " string",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" string",
),
87..94,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
94..95,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
95..95,
),
]
```

35
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_named_unicode.snap
View file

@ -2,38 +2,35 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle {
value: "\\N{BULLET} normal \\Nope \\N",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"\\N{BULLET} normal \\Nope \\N",
),
2..28,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
28..29,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
29..29,
),
]
```

63
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_named_unicode_raw.snap
View file

@ -2,44 +2,33 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..3,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringMiddle {
value: "\\N",
flags: AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"\\N",
),
3..5,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
Lbrace,
5..6,
),
(
Name {
name: "BULLET",
},
Name(
"BULLET",
),
6..12,
),
(
@ -47,26 +36,24 @@ expression: lex_source(source)
12..13,
),
(
FStringMiddle {
value: " normal",
flags: AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" normal",
),
13..20,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringEnd,
20..21,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
Newline,
21..21,
),
]
```

227
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_nested.snap
View file

@ -2,69 +2,53 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle {
value: "foo ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"foo ",
),
2..6,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
6..7,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
7..9,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle {
value: "bar ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"bar ",
),
9..13,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
13..14,
),
(
Name {
name: "x",
},
Name(
"x",
),
14..15,
),
(
@ -72,25 +56,20 @@ expression: lex_source(source)
16..17,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
18..20,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
20..21,
),
(
Name {
name: "wow",
},
Name(
"wow",
),
21..24,
),
(
@ -100,6 +79,9 @@ expression: lex_source(source)
(
FStringEnd,
25..26,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Rbrace,
@ -108,135 +90,112 @@ expression: lex_source(source)
(
FStringEnd,
27..28,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Rbrace,
28..29,
),
(
FStringMiddle {
value: " baz",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" baz",
),
29..33,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
33..34,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
35..37,
),
(
FStringMiddle {
value: "foo ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringStart,
35..37,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle(
"foo ",
),
37..41,
TokenFlags(
F_STRING,
),
),
(
Lbrace,
41..42,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
),
FStringStart,
42..44,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle {
value: "bar",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"bar",
),
44..47,
TokenFlags(
F_STRING,
),
),
(
FStringEnd,
47..48,
TokenFlags(
F_STRING,
),
),
(
Rbrace,
48..49,
),
(
FStringMiddle {
value: " some ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
" some ",
),
49..55,
TokenFlags(
F_STRING,
),
),
(
Lbrace,
55..56,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
56..58,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle {
value: "another",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"another",
),
58..65,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
65..66,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Rbrace,
@ -245,9 +204,13 @@ expression: lex_source(source)
(
FStringEnd,
67..68,
TokenFlags(
F_STRING,
),
),
(
Newline,
68..68,
),
]
```

247
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_parentheses.snap
View file

@ -2,18 +2,15 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -26,60 +23,48 @@ expression: lex_source(source)
(
FStringEnd,
4..5,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
6..8,
),
(
FStringMiddle {
value: "{}",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringStart,
6..8,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle(
"{}",
),
8..12,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
12..13,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
14..16,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringStart,
14..16,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle(
" ",
),
16..17,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -92,31 +77,25 @@ expression: lex_source(source)
(
FStringEnd,
19..20,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
21..23,
),
(
FStringMiddle {
value: "{",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringStart,
21..23,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle(
"{",
),
23..25,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -127,75 +106,59 @@ expression: lex_source(source)
26..27,
),
(
FStringMiddle {
value: "}",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"}",
),
27..29,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
29..30,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
31..33,
),
(
FStringMiddle {
value: "{{}}",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringStart,
31..33,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle(
"{{}}",
),
33..41,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
41..42,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
43..45,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringStart,
43..45,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle(
" ",
),
45..46,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -206,17 +169,13 @@ expression: lex_source(source)
47..48,
),
(
FStringMiddle {
value: " {} {",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" {} {",
),
48..56,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -227,24 +186,24 @@ expression: lex_source(source)
57..58,
),
(
FStringMiddle {
value: "} {{}} ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"} {{}} ",
),
58..71,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
71..72,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
72..72,
),
]
```

179
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_prefix.snap
View file

@ -2,185 +2,152 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
2..3,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
4..6,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
6..7,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
8..11,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringEnd,
11..12,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
13..16,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringEnd,
16..17,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: true,
},
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
18..21,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_UPPERCASE,
),
),
(
FStringEnd,
21..22,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_UPPERCASE,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: true,
},
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
23..26,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_UPPERCASE,
),
),
(
FStringEnd,
26..27,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_UPPERCASE,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
28..31,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringEnd,
31..32,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: false,
},
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
33..36,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringEnd,
36..37,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_LOWERCASE,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: true,
},
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
38..41,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_UPPERCASE,
),
),
(
FStringEnd,
41..42,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_UPPERCASE,
),
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Raw {
uppercase_r: true,
},
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
43..46,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_UPPERCASE,
),
),
(
FStringEnd,
46..47,
TokenFlags(
DOUBLE_QUOTES | F_STRING | RAW_STRING_UPPERCASE,
),
),
(
Newline,
47..47,
),
]
```

35
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_single_quote_escape_mac_eol.snap
View file

@ -2,38 +2,35 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: fstring_single_quote_escape_eol(MAC_EOL)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
),
FStringStart,
0..2,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle {
value: "text \\\r more text",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"text \\\r more text",
),
2..19,
TokenFlags(
F_STRING,
),
),
(
FStringEnd,
19..20,
TokenFlags(
F_STRING,
),
),
(
Newline,
20..20,
),
]
```

35
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_single_quote_escape_unix_eol.snap
View file

@ -2,38 +2,35 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: fstring_single_quote_escape_eol(UNIX_EOL)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
),
FStringStart,
0..2,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle {
value: "text \\\n more text",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"text \\\n more text",
),
2..19,
TokenFlags(
F_STRING,
),
),
(
FStringEnd,
19..20,
TokenFlags(
F_STRING,
),
),
(
Newline,
20..20,
),
]
```

35
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_single_quote_escape_windows_eol.snap
View file

@ -2,38 +2,35 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: fstring_single_quote_escape_eol(WINDOWS_EOL)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
),
FStringStart,
0..2,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle {
value: "text \\\r\n more text",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"text \\\r\n more text",
),
2..20,
TokenFlags(
F_STRING,
),
),
(
FStringEnd,
20..21,
TokenFlags(
F_STRING,
),
),
(
Newline,
21..21,
),
]
```

220
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_with_format_spec.snap
View file

@ -2,27 +2,24 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
2..3,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
3..6,
),
(
@ -34,26 +31,22 @@ expression: lex_source(source)
7..8,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" ",
),
8..9,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
9..10,
),
(
Name {
name: "x",
},
Name(
"x",
),
10..11,
),
(
@ -65,9 +58,9 @@ expression: lex_source(source)
12..13,
),
(
Name {
name: "s",
},
Name(
"s",
),
13..14,
),
(
@ -75,43 +68,35 @@ expression: lex_source(source)
14..15,
),
(
FStringMiddle {
value: ".3f",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
".3f",
),
15..18,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Rbrace,
18..19,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" ",
),
19..20,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
20..21,
),
(
Name {
name: "x",
},
Name(
"x",
),
21..22,
),
(
@ -119,26 +104,22 @@ expression: lex_source(source)
22..23,
),
(
FStringMiddle {
value: ".",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
".",
),
23..24,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
24..25,
),
(
Name {
name: "y",
},
Name(
"y",
),
25..26,
),
(
@ -146,50 +127,35 @@ expression: lex_source(source)
26..27,
),
(
FStringMiddle {
value: "f",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"f",
),
27..28,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Rbrace,
28..29,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" ",
),
29..30,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
30..31,
),
(
String {
value: "",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"",
),
31..33,
),
(
@ -197,26 +163,22 @@ expression: lex_source(source)
33..34,
),
(
FStringMiddle {
value: "*^",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"*^",
),
34..36,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
36..37,
),
(
Int {
value: 1,
},
Int(
1,
),
37..38,
),
(
@ -228,9 +190,9 @@ expression: lex_source(source)
39..40,
),
(
Int {
value: 1,
},
Int(
1,
),
40..41,
),
(
@ -246,26 +208,22 @@ expression: lex_source(source)
43..44,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" ",
),
44..45,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
45..46,
),
(
Name {
name: "x",
},
Name(
"x",
),
46..47,
),
(
@ -281,9 +239,9 @@ expression: lex_source(source)
49..50,
),
(
Int {
value: 1,
},
Int(
1,
),
50..51,
),
(
@ -295,9 +253,9 @@ expression: lex_source(source)
52..53,
),
(
Name {
name: "pop",
},
Name(
"pop",
),
53..56,
),
(
@ -319,9 +277,13 @@ expression: lex_source(source)
(
FStringEnd,
60..61,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
61..61,
),
]
```

57
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_with_ipy_escape_command.snap
View file

@ -2,31 +2,24 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringMiddle {
value: "foo ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"foo ",
),
2..6,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -37,9 +30,9 @@ expression: lex_source(source)
7..8,
),
(
Name {
name: "pwd",
},
Name(
"pwd",
),
8..11,
),
(
@ -47,24 +40,24 @@ expression: lex_source(source)
11..12,
),
(
FStringMiddle {
value: " bar",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" bar",
),
12..16,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
FStringEnd,
16..17,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
17..17,
),
]
```

59
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_with_lambda_expression.snap
View file

@ -2,18 +2,15 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -24,9 +21,9 @@ expression: lex_source(source)
3..9,
),
(
Name {
name: "x",
},
Name(
"x",
),
10..11,
),
(
@ -38,9 +35,9 @@ expression: lex_source(source)
12..13,
),
(
Name {
name: "x",
},
Name(
"x",
),
13..14,
),
(
@ -54,22 +51,20 @@ expression: lex_source(source)
(
FStringEnd,
16..17,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
17..18,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
18..20,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -84,9 +79,9 @@ expression: lex_source(source)
22..28,
),
(
Name {
name: "x",
},
Name(
"x",
),
29..30,
),
(
@ -98,9 +93,9 @@ expression: lex_source(source)
31..32,
),
(
Name {
name: "x",
},
Name(
"x",
),
32..33,
),
(
@ -118,9 +113,13 @@ expression: lex_source(source)
(
FStringEnd,
36..37,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
37..37,
),
]
```

289
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_with_multiline_format_spec.snap
View file

@ -2,31 +2,24 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
),
FStringStart,
0..4,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringMiddle {
value: "__",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
},
FStringMiddle(
"__",
),
4..6,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Lbrace,
@ -37,9 +30,9 @@ expression: lex_source(source)
7..8,
),
(
Name {
name: "x",
},
Name(
"x",
),
12..13,
),
(
@ -47,67 +40,53 @@ expression: lex_source(source)
13..14,
),
(
FStringMiddle {
value: "d\n",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
},
FStringMiddle(
"d\n",
),
14..16,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Rbrace,
16..17,
),
(
FStringMiddle {
value: "__",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
},
FStringMiddle(
"__",
),
17..19,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringEnd,
19..22,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Newline,
22..23,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
),
FStringStart,
23..27,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringMiddle {
value: "__",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
},
FStringMiddle(
"__",
),
27..29,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Lbrace,
@ -118,9 +97,9 @@ expression: lex_source(source)
30..31,
),
(
Name {
name: "x",
},
Name(
"x",
),
35..36,
),
(
@ -128,67 +107,53 @@ expression: lex_source(source)
36..37,
),
(
FStringMiddle {
value: "a\n b\n c\n",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
},
FStringMiddle(
"a\n b\n c\n",
),
37..61,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Rbrace,
61..62,
),
(
FStringMiddle {
value: "__",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: true,
quote_style: Single,
},
},
FStringMiddle(
"__",
),
62..64,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
FStringEnd,
64..67,
TokenFlags(
TRIPLE_QUOTED_STRING | F_STRING,
),
),
(
Newline,
67..68,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
),
FStringStart,
68..70,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle {
value: "__",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"__",
),
70..72,
TokenFlags(
F_STRING,
),
),
(
Lbrace,
@ -199,9 +164,9 @@ expression: lex_source(source)
73..74,
),
(
Name {
name: "x",
},
Name(
"x",
),
78..79,
),
(
@ -209,17 +174,13 @@ expression: lex_source(source)
79..80,
),
(
FStringMiddle {
value: "d",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"d",
),
80..81,
TokenFlags(
F_STRING,
),
),
(
NonLogicalNewline,
@ -230,50 +191,40 @@ expression: lex_source(source)
82..83,
),
(
FStringMiddle {
value: "__",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"__",
),
83..85,
TokenFlags(
F_STRING,
),
),
(
FStringEnd,
85..86,
TokenFlags(
F_STRING,
),
),
(
Newline,
86..87,
),
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
),
FStringStart,
87..89,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle {
value: "__",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"__",
),
89..91,
TokenFlags(
F_STRING,
),
),
(
Lbrace,
@ -284,9 +235,9 @@ expression: lex_source(source)
92..93,
),
(
Name {
name: "x",
},
Name(
"x",
),
97..98,
),
(
@ -294,26 +245,22 @@ expression: lex_source(source)
98..99,
),
(
FStringMiddle {
value: "a",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"a",
),
99..100,
TokenFlags(
F_STRING,
),
),
(
NonLogicalNewline,
100..101,
),
(
Name {
name: "b",
},
Name(
"b",
),
109..110,
),
(
@ -325,24 +272,24 @@ expression: lex_source(source)
111..112,
),
(
FStringMiddle {
value: "__",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"__",
),
112..114,
TokenFlags(
F_STRING,
),
),
(
FStringEnd,
114..115,
TokenFlags(
F_STRING,
),
),
(
Newline,
115..116,
),
]
```

131
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_with_named_expression.snap
View file

@ -2,27 +2,24 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
),
FStringStart,
0..2,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
2..3,
),
(
Name {
name: "x",
},
Name(
"x",
),
3..4,
),
(
@ -30,34 +27,26 @@ expression: lex_source(source)
4..5,
),
(
FStringMiddle {
value: "=10",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
"=10",
),
5..8,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Rbrace,
8..9,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" ",
),
9..10,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -68,9 +57,9 @@ expression: lex_source(source)
11..12,
),
(
Name {
name: "x",
},
Name(
"x",
),
12..13,
),
(
@ -78,9 +67,9 @@ expression: lex_source(source)
13..15,
),
(
Int {
value: 10,
},
Int(
10,
),
15..17,
),
(
@ -92,26 +81,22 @@ expression: lex_source(source)
18..19,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" ",
),
19..20,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
20..21,
),
(
Name {
name: "x",
},
Name(
"x",
),
21..22,
),
(
@ -123,9 +108,9 @@ expression: lex_source(source)
23..24,
),
(
Name {
name: "y",
},
Name(
"y",
),
24..25,
),
(
@ -133,9 +118,9 @@ expression: lex_source(source)
25..27,
),
(
Int {
value: 10,
},
Int(
10,
),
27..29,
),
(
@ -147,17 +132,13 @@ expression: lex_source(source)
30..31,
),
(
FStringMiddle {
value: " ",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Double,
},
},
FStringMiddle(
" ",
),
31..32,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Lbrace,
@ -168,9 +149,9 @@ expression: lex_source(source)
33..34,
),
(
Name {
name: "x",
},
Name(
"x",
),
34..35,
),
(
@ -178,9 +159,9 @@ expression: lex_source(source)
35..37,
),
(
Int {
value: 10,
},
Int(
10,
),
37..39,
),
(
@ -194,9 +175,13 @@ expression: lex_source(source)
(
FStringEnd,
41..42,
TokenFlags(
DOUBLE_QUOTES | F_STRING,
),
),
(
Newline,
42..42,
),
]
```

35
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__fstring_with_nul_char.snap
View file

@ -2,38 +2,35 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
FStringStart(
AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
),
FStringStart,
0..2,
TokenFlags(
F_STRING,
),
),
(
FStringMiddle {
value: "\\0",
flags: AnyStringFlags {
prefix: Format(
Regular,
),
triple_quoted: false,
quote_style: Single,
},
},
FStringMiddle(
"\\0",
),
2..4,
TokenFlags(
F_STRING,
),
),
(
FStringEnd,
4..5,
TokenFlags(
F_STRING,
),
),
(
Newline,
5..5,
),
]
```

15
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__indentation_with_mac_eol.snap
View file

@ -2,15 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: indentation_with_eol(MAC_EOL)
---
## Tokens
```
[
(
Def,
0..3,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
4..7,
),
(
@ -38,9 +40,9 @@ expression: indentation_with_eol(MAC_EOL)
15..21,
),
(
Int {
value: 99,
},
Int(
99,
),
22..24,
),
(
@ -56,3 +58,4 @@ expression: indentation_with_eol(MAC_EOL)
26..26,
),
]
```

15
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__indentation_with_unix_eol.snap
View file

@ -2,15 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: indentation_with_eol(UNIX_EOL)
---
## Tokens
```
[
(
Def,
0..3,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
4..7,
),
(
@ -38,9 +40,9 @@ expression: indentation_with_eol(UNIX_EOL)
15..21,
),
(
Int {
value: 99,
},
Int(
99,
),
22..24,
),
(
@ -56,3 +58,4 @@ expression: indentation_with_eol(UNIX_EOL)
26..26,
),
]
```

15
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__indentation_with_windows_eol.snap
View file

@ -2,15 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: indentation_with_eol(WINDOWS_EOL)
---
## Tokens
```
[
(
Def,
0..3,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
4..7,
),
(
@ -38,9 +40,9 @@ expression: indentation_with_eol(WINDOWS_EOL)
16..22,
),
(
Int {
value: 99,
},
Int(
99,
),
23..25,
),
(
@ -56,3 +58,4 @@ expression: indentation_with_eol(WINDOWS_EOL)
29..29,
),
]
```

22
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__invalid_leading_zero_big.snap
View file

@ -1,12 +1,28 @@
---
source: crates/ruff_python_parser/src/lexer.rs
expression: tokens
expression: "lex_invalid(source, Mode::Module)"
---
Err(
## Tokens
```
[
(
Unknown,
0..85,
),
(
Newline,
85..85,
),
]
```
## Errors
```
[
LexicalError {
error: OtherError(
"Invalid decimal integer literal",
),
location: 0..85,
},
)
]
```

22
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__invalid_leading_zero_small.snap
View file

@ -1,12 +1,28 @@
---
source: crates/ruff_python_parser/src/lexer.rs
expression: tokens
expression: "lex_invalid(source, Mode::Module)"
---
Err(
## Tokens
```
[
(
Unknown,
0..3,
),
(
Newline,
3..3,
),
]
```
## Errors
```
[
LexicalError {
error: OtherError(
"Invalid decimal integer literal",
),
location: 0..3,
},
)
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__ipython_escape_command.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_jupyter_source(source)
---
## Tokens
```
[
(
IpyEscapeCommand {
@ -125,3 +127,4 @@ expression: lex_jupyter_source(source)
180..180,
),
]
```

27
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__ipython_escape_command_assignment.snap
View file

@ -2,11 +2,13 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_jupyter_source(source)
---
## Tokens
```
[
(
Name {
name: "pwd",
},
Name(
"pwd",
),
0..3,
),
(
@ -25,9 +27,9 @@ expression: lex_jupyter_source(source)
10..11,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
11..14,
),
(
@ -46,9 +48,9 @@ expression: lex_jupyter_source(source)
30..31,
),
(
Name {
name: "bar",
},
Name(
"bar",
),
31..34,
),
(
@ -67,9 +69,9 @@ expression: lex_jupyter_source(source)
50..51,
),
(
Name {
name: "baz",
},
Name(
"baz",
),
51..54,
),
(
@ -88,3 +90,4 @@ expression: lex_jupyter_source(source)
85..85,
),
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__ipython_escape_command_indentation.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_jupyter_source(source)
---
## Tokens
```
[
(
If,
@ -39,3 +41,4 @@ expression: lex_jupyter_source(source)
43..43,
),
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_mac_eol.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: ipython_escape_command_line_continuation_eol(MAC_EOL)
---
## Tokens
```
[
(
IpyEscapeCommand {
@ -15,3 +17,4 @@ expression: ipython_escape_command_line_continuation_eol(MAC_EOL)
24..24,
),
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_unix_eol.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: ipython_escape_command_line_continuation_eol(UNIX_EOL)
---
## Tokens
```
[
(
IpyEscapeCommand {
@ -15,3 +17,4 @@ expression: ipython_escape_command_line_continuation_eol(UNIX_EOL)
24..24,
),
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_windows_eol.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: ipython_escape_command_line_continuation_eol(WINDOWS_EOL)
---
## Tokens
```
[
(
IpyEscapeCommand {
@ -15,3 +17,4 @@ expression: ipython_escape_command_line_continuation_eol(WINDOWS_EOL)
25..25,
),
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_with_mac_eol_and_eof.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: ipython_escape_command_line_continuation_with_eol_and_eof(MAC_EOL)
---
## Tokens
```
[
(
IpyEscapeCommand {
@ -15,3 +17,4 @@ expression: ipython_escape_command_line_continuation_with_eol_and_eof(MAC_EOL)
14..14,
),
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_with_unix_eol_and_eof.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: ipython_escape_command_line_continuation_with_eol_and_eof(UNIX_EOL)
---
## Tokens
```
[
(
IpyEscapeCommand {
@ -15,3 +17,4 @@ expression: ipython_escape_command_line_continuation_with_eol_and_eof(UNIX_EOL)
14..14,
),
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_with_windows_eol_and_eof.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: ipython_escape_command_line_continuation_with_eol_and_eof(WINDOWS_EOL)
---
## Tokens
```
[
(
IpyEscapeCommand {
@ -15,3 +17,4 @@ expression: ipython_escape_command_line_continuation_with_eol_and_eof(WINDOWS_EO
15..15,
),
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__ipython_help_end_escape_command.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_jupyter_source(source)
---
## Tokens
```
[
(
IpyEscapeCommand {
@ -180,3 +182,4 @@ expression: lex_jupyter_source(source)
132..132,
),
]
```

13
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__line_comment_empty.snap
View file

@ -2,17 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(&source)
---
## Tokens
```
[
(
Int {
value: 99232,
},
Int(
99232,
),
0..5,
),
(
Comment(
"#",
),
Comment,
7..8,
),
(
@ -20,3 +20,4 @@ expression: lex_source(&source)
8..8,
),
]
```

13
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__line_comment_long.snap
View file

@ -2,17 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(&source)
---
## Tokens
```
[
(
Int {
value: 99232,
},
Int(
99232,
),
0..5,
),
(
Comment(
"# foo",
),
Comment,
7..12,
),
(
@ -20,3 +20,4 @@ expression: lex_source(&source)
12..12,
),
]
```

13
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__line_comment_single_whitespace.snap
View file

@ -2,17 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(&source)
---
## Tokens
```
[
(
Int {
value: 99232,
},
Int(
99232,
),
0..5,
),
(
Comment(
"# ",
),
Comment,
7..9,
),
(
@ -20,3 +20,4 @@ expression: lex_source(&source)
9..9,
),
]
```

13
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__line_comment_whitespace.snap
View file

@ -2,17 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(&source)
---
## Tokens
```
[
(
Int {
value: 99232,
},
Int(
99232,
),
0..5,
),
(
Comment(
"# ",
),
Comment,
7..10,
),
(
@ -20,3 +20,4 @@ expression: lex_source(&source)
10..10,
),
]
```

11
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__logical_newline_line_comment.snap
View file

@ -2,11 +2,11 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
Comment(
"#Hello",
),
Comment,
0..6,
),
(
@ -14,9 +14,7 @@ expression: lex_source(source)
6..7,
),
(
Comment(
"#World",
),
Comment,
7..13,
),
(
@ -24,3 +22,4 @@ expression: lex_source(source)
13..14,
),
]
```

15
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__match_softkeyword_in_notebook.snap
View file

@ -2,15 +2,17 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_jupyter_source(source)
---
## Tokens
```
[
(
Match,
0..5,
),
(
Name {
name: "foo",
},
Name(
"foo",
),
6..9,
),
(
@ -30,9 +32,9 @@ expression: lex_jupyter_source(source)
15..19,
),
(
Name {
name: "bar",
},
Name(
"bar",
),
20..23,
),
(
@ -64,3 +66,4 @@ expression: lex_jupyter_source(source)
37..37,
),
]
```

51
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__newline_in_brackets_mac_eol.snap
View file

@ -2,11 +2,13 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: newline_in_brackets_eol(MAC_EOL)
---
## Tokens
```
[
(
Name {
name: "x",
},
Name(
"x",
),
0..1,
),
(
@ -26,9 +28,9 @@ expression: newline_in_brackets_eol(MAC_EOL)
6..7,
),
(
Int {
value: 1,
},
Int(
1,
),
11..12,
),
(
@ -36,9 +38,9 @@ expression: newline_in_brackets_eol(MAC_EOL)
12..13,
),
(
Int {
value: 2,
},
Int(
2,
),
13..14,
),
(
@ -54,9 +56,9 @@ expression: newline_in_brackets_eol(MAC_EOL)
16..17,
),
(
Int {
value: 3,
},
Int(
3,
),
17..18,
),
(
@ -68,9 +70,9 @@ expression: newline_in_brackets_eol(MAC_EOL)
19..20,
),
(
Int {
value: 4,
},
Int(
4,
),
20..21,
),
(
@ -98,9 +100,9 @@ expression: newline_in_brackets_eol(MAC_EOL)
27..28,
),
(
Int {
value: 5,
},
Int(
5,
),
28..29,
),
(
@ -112,9 +114,9 @@ expression: newline_in_brackets_eol(MAC_EOL)
30..31,
),
(
Int {
value: 6,
},
Int(
6,
),
31..32,
),
(
@ -122,9 +124,9 @@ expression: newline_in_brackets_eol(MAC_EOL)
32..33,
),
(
Int {
value: 7,
},
Int(
7,
),
35..36,
),
(
@ -140,3 +142,4 @@ expression: newline_in_brackets_eol(MAC_EOL)
38..39,
),
]
```

51
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__newline_in_brackets_unix_eol.snap
View file

@ -2,11 +2,13 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: newline_in_brackets_eol(UNIX_EOL)
---
## Tokens
```
[
(
Name {
name: "x",
},
Name(
"x",
),
0..1,
),
(
@ -26,9 +28,9 @@ expression: newline_in_brackets_eol(UNIX_EOL)
6..7,
),
(
Int {
value: 1,
},
Int(
1,
),
11..12,
),
(
@ -36,9 +38,9 @@ expression: newline_in_brackets_eol(UNIX_EOL)
12..13,
),
(
Int {
value: 2,
},
Int(
2,
),
13..14,
),
(
@ -54,9 +56,9 @@ expression: newline_in_brackets_eol(UNIX_EOL)
16..17,
),
(
Int {
value: 3,
},
Int(
3,
),
17..18,
),
(
@ -68,9 +70,9 @@ expression: newline_in_brackets_eol(UNIX_EOL)
19..20,
),
(
Int {
value: 4,
},
Int(
4,
),
20..21,
),
(
@ -98,9 +100,9 @@ expression: newline_in_brackets_eol(UNIX_EOL)
27..28,
),
(
Int {
value: 5,
},
Int(
5,
),
28..29,
),
(
@ -112,9 +114,9 @@ expression: newline_in_brackets_eol(UNIX_EOL)
30..31,
),
(
Int {
value: 6,
},
Int(
6,
),
31..32,
),
(
@ -122,9 +124,9 @@ expression: newline_in_brackets_eol(UNIX_EOL)
32..33,
),
(
Int {
value: 7,
},
Int(
7,
),
35..36,
),
(
@ -140,3 +142,4 @@ expression: newline_in_brackets_eol(UNIX_EOL)
38..39,
),
]
```

51
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__newline_in_brackets_windows_eol.snap
View file

@ -2,11 +2,13 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: newline_in_brackets_eol(WINDOWS_EOL)
---
## Tokens
```
[
(
Name {
name: "x",
},
Name(
"x",
),
0..1,
),
(
@ -26,9 +28,9 @@ expression: newline_in_brackets_eol(WINDOWS_EOL)
7..9,
),
(
Int {
value: 1,
},
Int(
1,
),
13..14,
),
(
@ -36,9 +38,9 @@ expression: newline_in_brackets_eol(WINDOWS_EOL)
14..15,
),
(
Int {
value: 2,
},
Int(
2,
),
15..16,
),
(
@ -54,9 +56,9 @@ expression: newline_in_brackets_eol(WINDOWS_EOL)
19..20,
),
(
Int {
value: 3,
},
Int(
3,
),
20..21,
),
(
@ -68,9 +70,9 @@ expression: newline_in_brackets_eol(WINDOWS_EOL)
22..24,
),
(
Int {
value: 4,
},
Int(
4,
),
24..25,
),
(
@ -98,9 +100,9 @@ expression: newline_in_brackets_eol(WINDOWS_EOL)
32..34,
),
(
Int {
value: 5,
},
Int(
5,
),
34..35,
),
(
@ -112,9 +114,9 @@ expression: newline_in_brackets_eol(WINDOWS_EOL)
36..38,
),
(
Int {
value: 6,
},
Int(
6,
),
38..39,
),
(
@ -122,9 +124,9 @@ expression: newline_in_brackets_eol(WINDOWS_EOL)
39..40,
),
(
Int {
value: 7,
},
Int(
7,
),
43..44,
),
(
@ -140,3 +142,4 @@ expression: newline_in_brackets_eol(WINDOWS_EOL)
46..48,
),
]
```

55
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__non_logical_newline_in_string_continuation.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
Lpar,
@ -12,16 +14,9 @@ expression: lex_source(source)
1..2,
),
(
String {
value: "a",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"a",
),
6..9,
),
(
@ -29,16 +24,9 @@ expression: lex_source(source)
9..10,
),
(
String {
value: "b",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"b",
),
14..17,
),
(
@ -50,29 +38,15 @@ expression: lex_source(source)
18..19,
),
(
String {
value: "c",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"c",
),
23..26,
),
(
String {
value: "d",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"d",
),
33..36,
),
(
@ -88,3 +62,4 @@ expression: lex_source(source)
38..38,
),
]
```

75
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__numbers.snap
View file

@ -2,59 +2,61 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
Int {
value: 47,
},
Int(
47,
),
0..4,
),
(
Int {
value: 10,
},
Int(
10,
),
5..9,
),
(
Int {
value: 13,
},
Int(
13,
),
10..16,
),
(
Int {
value: 0,
},
Int(
0,
),
17..18,
),
(
Int {
value: 123,
},
Int(
123,
),
19..22,
),
(
Int {
value: 1234567890,
},
Int(
1234567890,
),
23..36,
),
(
Float {
value: 0.2,
},
Float(
0.2,
),
37..40,
),
(
Float {
value: 100.0,
},
Float(
100.0,
),
41..45,
),
(
Float {
value: 2100.0,
},
Float(
2100.0,
),
46..51,
),
(
@ -72,21 +74,21 @@ expression: lex_source(source)
55..59,
),
(
Int {
value: 0,
},
Int(
0,
),
60..63,
),
(
Int {
value: 11051210869376104954,
},
Int(
11051210869376104954,
),
64..82,
),
(
Int {
value: 0x995DC9BBDF1939FA995DC9BBDF1939FA,
},
Int(
0x995DC9BBDF1939FA995DC9BBDF1939FA,
),
83..117,
),
(
@ -94,3 +96,4 @@ expression: lex_source(source)
117..117,
),
]
```

3
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__operators.snap
View file

@ -2,6 +2,8 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
DoubleSlash,
@ -28,3 +30,4 @@ expression: lex_source(source)
10..10,
),
]
```

131
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__string.snap
View file

@ -2,124 +2,70 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: lex_source(source)
---
## Tokens
```
[
(
String {
value: "double",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Double,
},
},
String(
"double",
),
0..8,
TokenFlags(
DOUBLE_QUOTES,
),
),
(
String {
value: "single",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"single",
),
9..17,
),
(
String {
value: "can\\'t",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"can\\'t",
),
18..26,
),
(
String {
value: "\\\\\\\"",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Double,
},
},
String(
"\\\\\\\"",
),
27..33,
TokenFlags(
DOUBLE_QUOTES,
),
),
(
String {
value: "\\t\\r\\n",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"\\t\\r\\n",
),
34..42,
),
(
String {
value: "\\g",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"\\g",
),
43..47,
),
(
String {
value: "raw\\'",
flags: AnyStringFlags {
prefix: Regular(
Raw {
uppercase: false,
},
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"raw\\'",
),
48..56,
TokenFlags(
RAW_STRING_LOWERCASE,
),
),
(
String {
value: "\\420",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"\\420",
),
57..63,
),
(
String {
value: "\\200\\0a",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Single,
},
},
String(
"\\200\\0a",
),
64..73,
),
(
@ -127,3 +73,4 @@ expression: lex_source(source)
73..73,
),
]
```

19
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__string_continuation_with_mac_eol.snap
View file

@ -2,22 +2,21 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: string_continuation_with_eol(MAC_EOL)
---
## Tokens
```
[
(
String {
value: "abc\\\rdef",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Double,
},
},
String(
"abc\\\rdef",
),
0..10,
TokenFlags(
DOUBLE_QUOTES,
),
),
(
Newline,
10..10,
),
]
```

19
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__string_continuation_with_unix_eol.snap
View file

@ -2,22 +2,21 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: string_continuation_with_eol(UNIX_EOL)
---
## Tokens
```
[
(
String {
value: "abc\\\ndef",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Double,
},
},
String(
"abc\\\ndef",
),
0..10,
TokenFlags(
DOUBLE_QUOTES,
),
),
(
Newline,
10..10,
),
]
```

19
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__string_continuation_with_windows_eol.snap
View file

@ -2,22 +2,21 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: string_continuation_with_eol(WINDOWS_EOL)
---
## Tokens
```
[
(
String {
value: "abc\\\r\ndef",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: false,
quote_style: Double,
},
},
String(
"abc\\\r\ndef",
),
0..11,
TokenFlags(
DOUBLE_QUOTES,
),
),
(
Newline,
11..11,
),
]
```

94
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__tet_too_low_dedent.snap
View file

@ -1,66 +1,58 @@
---
source: crates/ruff_python_parser/src/lexer.rs
expression: tokens
expression: "lex_invalid(source, Mode::Module)"
---
## Tokens
```
[
Ok(
(
If,
0..2,
),
(
If,
0..2,
),
Ok(
(
True,
3..7,
),
(
True,
3..7,
),
Ok(
(
Colon,
7..8,
),
(
Colon,
7..8,
),
Ok(
(
Newline,
8..9,
),
(
Newline,
8..9,
),
Ok(
(
Indent,
9..13,
),
(
Indent,
9..13,
),
Ok(
(
Pass,
13..17,
),
(
Pass,
13..17,
),
Ok(
(
Newline,
17..18,
),
(
Newline,
17..18,
),
Err(
LexicalError {
error: IndentationError,
location: 18..20,
},
(
Unknown,
18..20,
),
Ok(
(
Pass,
20..24,
),
(
Pass,
20..24,
),
Ok(
(
Newline,
24..24,
),
(
Newline,
24..24,
),
]
```
## Errors
```
[
LexicalError {
error: IndentationError,
location: 18..20,
},
]
```

19
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__triple_quoted_mac_eol.snap
View file

@ -2,22 +2,21 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: triple_quoted_eol(MAC_EOL)
---
## Tokens
```
[
(
String {
value: "\r test string\r ",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: true,
quote_style: Double,
},
},
String(
"\r test string\r ",
),
0..21,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING,
),
),
(
Newline,
21..21,
),
]
```

19
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__triple_quoted_unix_eol.snap
View file

@ -2,22 +2,21 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: triple_quoted_eol(UNIX_EOL)
---
## Tokens
```
[
(
String {
value: "\n test string\n ",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: true,
quote_style: Double,
},
},
String(
"\n test string\n ",
),
0..21,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING,
),
),
(
Newline,
21..21,
),
]
```

19
crates/ruff_python_parser/src/snapshots/ruff_python_parser__lexer__tests__triple_quoted_windows_eol.snap
View file

@ -2,22 +2,21 @@
source: crates/ruff_python_parser/src/lexer.rs
expression: triple_quoted_eol(WINDOWS_EOL)
---
## Tokens
```
[
(
String {
value: "\r\n test string\r\n ",
flags: AnyStringFlags {
prefix: Regular(
Empty,
),
triple_quoted: true,
quote_style: Double,
},
},
String(
"\r\n test string\r\n ",
),
0..23,
TokenFlags(
DOUBLE_QUOTES | TRIPLE_QUOTED_STRING,
),
),
(
Newline,
23..23,
),
]
```

224
crates/ruff_python_parser/src/soft_keywords.rs
View file

@ -1,224 +0,0 @@
use itertools::{Itertools, MultiPeek};
use crate::{lexer::LexResult, token::Tok, Mode};
/// An [`Iterator`] that transforms a token stream to accommodate soft keywords (namely, `match`
/// `case`, and `type`).
///
/// [PEP 634](https://www.python.org/dev/peps/pep-0634/) introduced the `match` and `case` keywords
/// as soft keywords, meaning that they can be used as identifiers (e.g., variable names) in certain
/// contexts.
///
/// Later, [PEP 695](https://peps.python.org/pep-0695/#generic-type-alias) introduced the `type`
/// soft keyword.
///
/// This function modifies a token stream to accommodate this change. In particular, it replaces
/// soft keyword tokens with `identifier` tokens if they are used as identifiers.
///
/// Handling soft keywords in this intermediary pass allows us to simplify both the lexer and
/// `ruff_python_parser`, as neither of them need to be aware of soft keywords.
pub struct SoftKeywordTransformer<I>
where
I: Iterator<Item = LexResult>,
{
underlying: MultiPeek<I>,
position: Position,
}
impl<I> SoftKeywordTransformer<I>
where
I: Iterator<Item = LexResult>,
{
pub fn new(lexer: I, mode: Mode) -> Self {
Self {
underlying: lexer.multipeek(), // spell-checker:ignore multipeek
position: if mode == Mode::Expression {
Position::Other
} else {
Position::Statement
},
}
}
}
impl<I> Iterator for SoftKeywordTransformer<I>
where
I: Iterator<Item = LexResult>,
{
type Item = LexResult;
#[inline]
fn next(&mut self) -> Option<LexResult> {
let mut next = self.underlying.next();
if let Some(Ok((tok, range))) = next.as_ref() {
// If the token is a soft keyword e.g. `type`, `match`, or `case`, check if it's
// used as an identifier. We assume every soft keyword use is an identifier unless
// a heuristic is met.
match tok {
// For `match` and `case`, all of the following conditions must be met:
// 1. The token is at the start of a logical line.
// 2. The logical line contains a top-level colon (that is, a colon that is not nested
// inside a parenthesized expression, list, or dictionary).
// 3. The top-level colon is not the immediate sibling of a `match` or `case` token.
// (This is to avoid treating `match` or `case` as identifiers when annotated with
// type hints.)
Tok::Match | Tok::Case => {
if matches!(self.position, Position::Statement) {
let mut nesting = 0;
let mut first = true;
let mut seen_colon = false;
let mut seen_lambda = false;
while let Some(Ok((tok, _))) = self.underlying.peek() {
match tok {
Tok::Newline => break,
Tok::Lambda if nesting == 0 => seen_lambda = true,
Tok::Colon if nesting == 0 => {
if seen_lambda {
seen_lambda = false;
} else if !first {
seen_colon = true;
}
}
Tok::Lpar | Tok::Lsqb | Tok::Lbrace => nesting += 1,
Tok::Rpar | Tok::Rsqb | Tok::Rbrace => nesting -= 1,
_ => {}
}
first = false;
}
if !seen_colon {
next = Some(Ok((soft_to_name(tok), *range)));
}
} else {
next = Some(Ok((soft_to_name(tok), *range)));
}
}
// For `type` all of the following conditions must be met:
// 1. The token is at the start of a logical line.
// 2. The type token is immediately followed by a name token.
// 3. The name token is eventually followed by an equality token.
Tok::Type => {
if matches!(
self.position,
Position::Statement | Position::SimpleStatement
) {
let mut is_type_alias = false;
if let Some(Ok((tok, _))) = self.underlying.peek() {
if matches!(
tok,
Tok::Name { .. } |
// We treat a soft keyword token following a type token as a
// name to support cases like `type type = int` or `type match = int`
Tok::Type | Tok::Match | Tok::Case
) {
let mut nesting = 0;
while let Some(Ok((tok, _))) = self.underlying.peek() {
match tok {
Tok::Newline => break,
Tok::Equal if nesting == 0 => {
is_type_alias = true;
break;
}
Tok::Lsqb => nesting += 1,
Tok::Rsqb => nesting -= 1,
// Allow arbitrary content within brackets for now
_ if nesting > 0 => {}
// Exit if unexpected tokens are seen
_ => break,
}
}
}
}
if !is_type_alias {
next = Some(Ok((soft_to_name(tok), *range)));
}
} else {
next = Some(Ok((soft_to_name(tok), *range)));
}
}
_ => (), // Not a soft keyword token
}
}
// Update the position, to track whether we're at the start of a logical line.
if let Some(lex_result) = next.as_ref() {
if let Ok((tok, _)) = lex_result.as_ref() {
match tok {
Tok::NonLogicalNewline | Tok::Comment { .. } => {
// Nothing to do.
}
Tok::Newline | Tok::Indent | Tok::Dedent => {
self.position = Position::Statement;
}
// If we see a semicolon, assume we're at the start of a simple statement, as in:
// ```python
// type X = int; type Y = float
// ```
Tok::Semi => {
self.position = Position::SimpleStatement;
}
// If we see a colon, and we're not in a nested context, assume we're at the
// start of a simple statement, as in:
// ```python
// class Class: type X = int
// ```
Tok::Colon if self.position == Position::Other => {
self.position = Position::SimpleStatement;
}
Tok::Lpar | Tok::Lsqb | Tok::Lbrace => {
self.position = if let Position::Nested(depth) = self.position {
Position::Nested(depth.saturating_add(1))
} else {
Position::Nested(1)
};
}
Tok::Rpar | Tok::Rsqb | Tok::Rbrace => {
self.position = if let Position::Nested(depth) = self.position {
let depth = depth.saturating_sub(1);
if depth > 0 {
Position::Nested(depth)
} else {
Position::Other
}
} else {
Position::Other
};
}
_ => {
self.position = Position::Other;
}
}
}
}
next
}
}
#[inline]
fn soft_to_name(tok: &Tok) -> Tok {
let name = match tok {
Tok::Match => "match",
Tok::Case => "case",
Tok::Type => "type",
_ => unreachable!("other tokens never reach here"),
};
Tok::Name {
name: name.to_string().into_boxed_str(),
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Position {
/// The lexer is at the start of a logical line, i.e., the start of a simple or compound statement.
Statement,
/// The lexer is at the start of a simple statement, e.g., a statement following a semicolon
/// or colon, as in:
/// ```python
/// class Class: type X = int
/// ```
SimpleStatement,
/// The lexer is within brackets, with the given bracket nesting depth.
Nested(u32),
/// The lexer is some other location.
Other,
}

192
crates/ruff_python_parser/src/string.rs
View file

@ -469,13 +469,19 @@ pub(crate) fn parse_fstring_literal_element(
#[cfg(test)]
mod tests {
use ruff_python_ast::Suite;
use crate::lexer::LexicalErrorType;
use crate::{parse_suite, FStringErrorType, ParseErrorType, Suite};
use crate::{parse_module, FStringErrorType, ParseError, ParseErrorType, Parsed};
const WINDOWS_EOL: &str = "\r\n";
const MAC_EOL: &str = "\r";
const UNIX_EOL: &str = "\n";
fn parse_suite(source: &str) -> Result<Suite, ParseError> {
parse_module(source).map(Parsed::into_suite)
}
fn string_parser_escaped_eol(eol: &str) -> Suite {
let source = format!(r"'text \{eol}more text'");
parse_suite(&source).unwrap()
@ -483,73 +489,69 @@ mod tests {
#[test]
fn test_string_parser_escaped_unix_eol() {
let parse_ast = string_parser_escaped_eol(UNIX_EOL);
insta::assert_debug_snapshot!(parse_ast);
let suite = string_parser_escaped_eol(UNIX_EOL);
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_string_parser_escaped_mac_eol() {
let parse_ast = string_parser_escaped_eol(MAC_EOL);
insta::assert_debug_snapshot!(parse_ast);
let suite = string_parser_escaped_eol(MAC_EOL);
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_string_parser_escaped_windows_eol() {
let parse_ast = string_parser_escaped_eol(WINDOWS_EOL);
insta::assert_debug_snapshot!(parse_ast);
let suite = string_parser_escaped_eol(WINDOWS_EOL);
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_fstring() {
let source = r#"f"{a}{ b }{{foo}}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_fstring_nested_spec() {
let source = r#"f"{foo:{spec}}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_fstring_not_nested_spec() {
let source = r#"f"{foo:spec}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_empty_fstring() {
insta::assert_debug_snapshot!(parse_suite(r#"f"""#,).unwrap());
let source = r#"f"""#;
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_fstring_parse_self_documenting_base() {
let source = r#"f"{user=}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_fstring_parse_self_documenting_base_more() {
let source = r#"f"mix {user=} with text and {second=}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_fstring_parse_self_documenting_format() {
let source = r#"f"{user=:>10}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
fn parse_fstring_error(source: &str) -> FStringErrorType {
@ -577,240 +579,236 @@ mod tests {
// error appears after the unexpected `FStringMiddle` token, which is between the
// `:` and the `{`.
// assert_eq!(parse_fstring_error("f'{lambda x: {x}}'"), LambdaWithoutParentheses);
assert!(parse_suite(r#"f"{class}""#,).is_err());
assert!(parse_suite(r#"f"{class}""#).is_err());
}
#[test]
fn test_parse_fstring_not_equals() {
let source = r#"f"{1 != 2}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_fstring_equals() {
let source = r#"f"{42 == 42}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_fstring_self_doc_prec_space() {
let source = r#"f"{x =}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_fstring_self_doc_trailing_space() {
let source = r#"f"{x= }""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_fstring_yield_expr() {
let source = r#"f"{yield}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_string_concat() {
let source = "'Hello ' 'world'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_u_string_concat_1() {
let source = "'Hello ' u'world'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_u_string_concat_2() {
let source = "u'Hello ' 'world'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_f_string_concat_1() {
let source = "'Hello ' f'world'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_f_string_concat_2() {
let source = "'Hello ' f'world'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_f_string_concat_3() {
let source = "'Hello ' f'world{\"!\"}'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_f_string_concat_4() {
let source = "'Hello ' f'world{\"!\"}' 'again!'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_u_f_string_concat_1() {
let source = "u'Hello ' f'world'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_u_f_string_concat_2() {
let source = "u'Hello ' f'world' '!'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_string_triple_quotes_with_kind() {
let source = "u'''Hello, world!'''";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_single_quoted_byte() {
// single quote
let source = r##"b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff'"##;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_double_quoted_byte() {
// double quote
let source = r##"b"\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff""##;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_escape_char_in_byte_literal() {
// backslash does not escape
let source = r#"b"omkmok\Xaa""#; // spell-checker:ignore omkmok
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_raw_byte_literal_1() {
let source = r"rb'\x1z'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_raw_byte_literal_2() {
let source = r"rb'\\'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_escape_octet() {
let source = r"b'\43a\4\1234'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_fstring_escaped_newline() {
let source = r#"f"\n{x}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_fstring_constant_range() {
let source = r#"f"aaa{bbb}ccc{ddd}eee""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_fstring_unescaped_newline() {
let source = r#"f"""
{x}""""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_fstring_escaped_character() {
let source = r#"f"\\{x}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_raw_fstring() {
let source = r#"rf"{x}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_triple_quoted_raw_fstring() {
let source = r#"rf"""{x}""""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_fstring_line_continuation() {
let source = r#"rf"\
{x}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_fstring_nested_string_spec() {
let source = r#"f"{foo:{''}}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_parse_fstring_nested_concatenation_string_spec() {
let source = r#"f"{foo:{'' ''}}""#;
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
/// <https://github.com/astral-sh/ruff/issues/8355>
#[test]
fn test_dont_panic_on_8_in_octal_escape() {
let source = r"bold = '\038[1m'";
let parse_ast = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(source).unwrap();
insta::assert_debug_snapshot!(suite);
}
#[test]
fn test_invalid_unicode_literal() {
let source = r"'\x1ó34'";
let error = parse_suite(source).unwrap_err();
insta::assert_debug_snapshot!(error);
}
@ -818,7 +816,6 @@ mod tests {
fn test_missing_unicode_lbrace_error() {
let source = r"'\N '";
let error = parse_suite(source).unwrap_err();
insta::assert_debug_snapshot!(error);
}
@ -826,7 +823,6 @@ mod tests {
fn test_missing_unicode_rbrace_error() {
let source = r"'\N{SPACE'";
let error = parse_suite(source).unwrap_err();
insta::assert_debug_snapshot!(error);
}
@ -834,7 +830,6 @@ mod tests {
fn test_invalid_unicode_name_error() {
let source = r"'\N{INVALID}'";
let error = parse_suite(source).unwrap_err();
insta::assert_debug_snapshot!(error);
}
@ -842,7 +837,6 @@ mod tests {
fn test_invalid_byte_literal_error() {
let source = r"b'123a𝐁c'";
let error = parse_suite(source).unwrap_err();
insta::assert_debug_snapshot!(error);
}
@ -852,8 +846,8 @@ mod tests {
#[test]
fn $name() {
let source = format!(r#""\N{{{0}}}""#, $alias);
let parse_ast = parse_suite(&source).unwrap();
insta::assert_debug_snapshot!(parse_ast);
let suite = parse_suite(&source).unwrap();
insta::assert_debug_snapshot!(suite);
}
)*
}

634
crates/ruff_python_parser/src/token.rs
View file

@ -1,4 +1,4 @@
//! Token type for Python source code created by the lexer and consumed by the `ruff_python_parser`.
//! Token kinds for Python source code created by the lexer and consumed by the `ruff_python_parser`.
//!
//! This module defines the tokens that the lexer recognizes. The tokens are
//! loosely based on the token definitions found in the [CPython source].
@ -7,482 +7,140 @@
use std::fmt;
use ruff_python_ast::{AnyStringFlags, BoolOp, Int, IpyEscapeKind, Operator, StringFlags, UnaryOp};
use ruff_python_ast::{BoolOp, Operator, UnaryOp};
/// The set of tokens the Python source code can be tokenized in.
#[derive(Clone, Debug, PartialEq, is_macro::Is)]
pub enum Tok {
/// Token value for a name, commonly known as an identifier.
Name {
/// The name value.
///
/// Unicode names are NFKC-normalized by the lexer,
/// matching [the behaviour of Python's lexer](https://docs.python.org/3/reference/lexical_analysis.html#identifiers)
name: Box<str>,
},
/// Token value for an integer.
Int {
/// The integer value.
value: Int,
},
/// Token value for a floating point number.
Float {
/// The float value.
value: f64,
},
/// Token value for a complex number.
Complex {
/// The real part of the complex number.
real: f64,
/// The imaginary part of the complex number.
imag: f64,
},
/// Token value for a string.
String {
/// The string value.
value: Box<str>,
/// Flags that can be queried to determine the quote style
/// and prefixes of the string
flags: AnyStringFlags,
},
/// Token value for the start of an f-string. This includes the `f`/`F`/`fr` prefix
/// and the opening quote(s).
FStringStart(AnyStringFlags),
/// Token value that includes the portion of text inside the f-string that's not
/// part of the expression part and isn't an opening or closing brace.
FStringMiddle {
/// The string value.
value: Box<str>,
/// Flags that can be queried to determine the quote style
/// and prefixes of the string
flags: AnyStringFlags,
},
/// Token value for the end of an f-string. This includes the closing quote.
FStringEnd,
/// Token value for IPython escape commands. These are recognized by the lexer
/// only when the mode is [`Ipython`].
///
/// [`Ipython`]: crate::Mode::Ipython
IpyEscapeCommand {
/// The magic command value.
value: Box<str>,
/// The kind of magic command.
kind: IpyEscapeKind,
},
/// Token value for a comment. These are filtered out of the token stream prior to parsing.
Comment(Box<str>),
/// Token value for a newline.
Newline,
/// Token value for a newline that is not a logical line break. These are filtered out of
/// the token stream prior to parsing.
NonLogicalNewline,
/// Token value for an indent.
Indent,
/// Token value for a dedent.
Dedent,
EndOfFile,
/// Token value for a question mark `?`. This is only used in [`Ipython`].
///
/// [`Ipython`]: crate::Mode::Ipython
Question,
/// Token value for a exclamation mark `!`.
Exclamation,
/// Token value for a left parenthesis `(`.
Lpar,
/// Token value for a right parenthesis `)`.
Rpar,
/// Token value for a left square bracket `[`.
Lsqb,
/// Token value for a right square bracket `]`.
Rsqb,
/// Token value for a colon `:`.
Colon,
/// Token value for a comma `,`.
Comma,
/// Token value for a semicolon `;`.
Semi,
/// Token value for plus `+`.
Plus,
/// Token value for minus `-`.
Minus,
/// Token value for star `*`.
Star,
/// Token value for slash `/`.
Slash,
/// Token value for vertical bar `|`.
Vbar,
/// Token value for ampersand `&`.
Amper,
/// Token value for less than `<`.
Less,
/// Token value for greater than `>`.
Greater,
/// Token value for equal `=`.
Equal,
/// Token value for dot `.`.
Dot,
/// Token value for percent `%`.
Percent,
/// Token value for left bracket `{`.
Lbrace,
/// Token value for right bracket `}`.
Rbrace,
/// Token value for double equal `==`.
EqEqual,
/// Token value for not equal `!=`.
NotEqual,
/// Token value for less than or equal `<=`.
LessEqual,
/// Token value for greater than or equal `>=`.
GreaterEqual,
/// Token value for tilde `~`.
Tilde,
/// Token value for caret `^`.
CircumFlex,
/// Token value for left shift `<<`.
LeftShift,
/// Token value for right shift `>>`.
RightShift,
/// Token value for double star `**`.
DoubleStar,
/// Token value for double star equal `**=`.
DoubleStarEqual,
/// Token value for plus equal `+=`.
PlusEqual,
/// Token value for minus equal `-=`.
MinusEqual,
/// Token value for star equal `*=`.
StarEqual,
/// Token value for slash equal `/=`.
SlashEqual,
/// Token value for percent equal `%=`.
PercentEqual,
/// Token value for ampersand equal `&=`.
AmperEqual,
/// Token value for vertical bar equal `|=`.
VbarEqual,
/// Token value for caret equal `^=`.
CircumflexEqual,
/// Token value for left shift equal `<<=`.
LeftShiftEqual,
/// Token value for right shift equal `>>=`.
RightShiftEqual,
/// Token value for double slash `//`.
DoubleSlash,
/// Token value for double slash equal `//=`.
DoubleSlashEqual,
/// Token value for colon equal `:=`.
ColonEqual,
/// Token value for at `@`.
At,
/// Token value for at equal `@=`.
AtEqual,
/// Token value for arrow `->`.
Rarrow,
/// Token value for ellipsis `...`.
Ellipsis,
// Self documenting.
// Keywords (alphabetically):
False,
None,
True,
And,
As,
Assert,
Async,
Await,
Break,
Class,
Continue,
Def,
Del,
Elif,
Else,
Except,
Finally,
For,
From,
Global,
If,
Import,
In,
Is,
Lambda,
Nonlocal,
Not,
Or,
Pass,
Raise,
Return,
Try,
While,
Match,
Type,
Case,
With,
Yield,
Unknown,
}
impl Tok {
#[inline]
pub fn kind(&self) -> TokenKind {
TokenKind::from_token(self)
}
}
impl fmt::Display for Tok {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
#[allow(clippy::enum_glob_use)]
use Tok::*;
match self {
Name { name } => write!(f, "{name}"),
Int { value } => write!(f, "{value}"),
Float { value } => write!(f, "{value}"),
Complex { real, imag } => write!(f, "{real}j{imag}"),
String { value, flags } => {
write!(f, "{}", flags.format_string_contents(value))
}
FStringStart(_) => f.write_str("FStringStart"),
FStringMiddle { value, .. } => f.write_str(value),
FStringEnd => f.write_str("FStringEnd"),
IpyEscapeCommand { kind, value } => write!(f, "{kind}{value}"),
Newline => f.write_str("Newline"),
NonLogicalNewline => f.write_str("NonLogicalNewline"),
Indent => f.write_str("Indent"),
Dedent => f.write_str("Dedent"),
EndOfFile => f.write_str("EOF"),
Question => f.write_str("?"),
Exclamation => f.write_str("!"),
Lpar => f.write_str("("),
Rpar => f.write_str(")"),
Lsqb => f.write_str("["),
Rsqb => f.write_str("]"),
Colon => f.write_str(":"),
Comma => f.write_str(","),
Comment(value) => f.write_str(value),
Semi => f.write_str(";"),
Plus => f.write_str("+"),
Minus => f.write_str("-"),
Star => f.write_str("*"),
Slash => f.write_str("/"),
Vbar => f.write_str("|"),
Amper => f.write_str("&"),
Less => f.write_str("<"),
Greater => f.write_str(">"),
Equal => f.write_str("="),
Dot => f.write_str("."),
Percent => f.write_str("%"),
Lbrace => f.write_str("{"),
Rbrace => f.write_str("}"),
EqEqual => f.write_str("=="),
NotEqual => f.write_str("!="),
LessEqual => f.write_str("<="),
GreaterEqual => f.write_str(">="),
Tilde => f.write_str("~"),
CircumFlex => f.write_str("^"),
LeftShift => f.write_str("<<"),
RightShift => f.write_str(">>"),
DoubleStar => f.write_str("**"),
DoubleStarEqual => f.write_str("**="),
PlusEqual => f.write_str("+="),
MinusEqual => f.write_str("-="),
StarEqual => f.write_str("*="),
SlashEqual => f.write_str("/="),
PercentEqual => f.write_str("%="),
AmperEqual => f.write_str("&="),
VbarEqual => f.write_str("|="),
CircumflexEqual => f.write_str("^="),
LeftShiftEqual => f.write_str("<<="),
RightShiftEqual => f.write_str(">>="),
DoubleSlash => f.write_str("//"),
DoubleSlashEqual => f.write_str("//="),
At => f.write_str("@"),
AtEqual => f.write_str("@="),
Rarrow => f.write_str("->"),
Ellipsis => f.write_str("..."),
False => f.write_str("False"),
None => f.write_str("None"),
True => f.write_str("True"),
And => f.write_str("and"),
As => f.write_str("as"),
Assert => f.write_str("assert"),
Async => f.write_str("async"),
Await => f.write_str("await"),
Break => f.write_str("break"),
Class => f.write_str("class"),
Continue => f.write_str("continue"),
Def => f.write_str("def"),
Del => f.write_str("del"),
Elif => f.write_str("elif"),
Else => f.write_str("else"),
Except => f.write_str("except"),
Finally => f.write_str("finally"),
For => f.write_str("for"),
From => f.write_str("from"),
Global => f.write_str("global"),
If => f.write_str("if"),
Import => f.write_str("import"),
In => f.write_str("in"),
Is => f.write_str("is"),
Lambda => f.write_str("lambda"),
Nonlocal => f.write_str("nonlocal"),
Not => f.write_str("not"),
Or => f.write_str("or"),
Pass => f.write_str("pass"),
Raise => f.write_str("raise"),
Return => f.write_str("return"),
Try => f.write_str("try"),
While => f.write_str("while"),
Match => f.write_str("match"),
Type => f.write_str("type"),
Case => f.write_str("case"),
With => f.write_str("with"),
Yield => f.write_str("yield"),
ColonEqual => f.write_str(":="),
Unknown => f.write_str("<Unknown>>"),
}
}
}
/// A kind of token.
///
/// This is a lightweight representation of [`Tok`] which doesn't contain any information
/// about the token itself.
/// A kind of a token.
#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)]
pub enum TokenKind {
/// Token value for a name, commonly known as an identifier.
/// Token kind for a name, commonly known as an identifier.
Name,
/// Token value for an integer.
/// Token kind for an integer.
Int,
/// Token value for a floating point number.
/// Token kind for a floating point number.
Float,
/// Token value for a complex number.
/// Token kind for a complex number.
Complex,
/// Token value for a string.
/// Token kind for a string.
String,
/// Token value for the start of an f-string. This includes the `f`/`F`/`fr` prefix
/// Token kind for the start of an f-string. This includes the `f`/`F`/`fr` prefix
/// and the opening quote(s).
FStringStart,
/// Token value that includes the portion of text inside the f-string that's not
/// Token kind that includes the portion of text inside the f-string that's not
/// part of the expression part and isn't an opening or closing brace.
FStringMiddle,
/// Token value for the end of an f-string. This includes the closing quote.
/// Token kind for the end of an f-string. This includes the closing quote.
FStringEnd,
/// Token value for a IPython escape command.
/// Token kind for a IPython escape command.
IpyEscapeCommand,
/// Token value for a comment. These are filtered out of the token stream prior to parsing.
/// Token kind for a comment. These are filtered out of the token stream prior to parsing.
Comment,
/// Token value for a newline.
/// Token kind for a newline.
Newline,
/// Token value for a newline that is not a logical line break. These are filtered out of
/// Token kind for a newline that is not a logical line break. These are filtered out of
/// the token stream prior to parsing.
NonLogicalNewline,
/// Token value for an indent.
/// Token kind for an indent.
Indent,
/// Token value for a dedent.
/// Token kind for a dedent.
Dedent,
EndOfFile,
/// Token value for a question mark `?`.
/// Token kind for a question mark `?`.
Question,
/// Token value for an exclamation mark `!`.
/// Token kind for an exclamation mark `!`.
Exclamation,
/// Token value for a left parenthesis `(`.
/// Token kind for a left parenthesis `(`.
Lpar,
/// Token value for a right parenthesis `)`.
/// Token kind for a right parenthesis `)`.
Rpar,
/// Token value for a left square bracket `[`.
/// Token kind for a left square bracket `[`.
Lsqb,
/// Token value for a right square bracket `]`.
/// Token kind for a right square bracket `]`.
Rsqb,
/// Token value for a colon `:`.
/// Token kind for a colon `:`.
Colon,
/// Token value for a comma `,`.
/// Token kind for a comma `,`.
Comma,
/// Token value for a semicolon `;`.
/// Token kind for a semicolon `;`.
Semi,
/// Token value for plus `+`.
/// Token kind for plus `+`.
Plus,
/// Token value for minus `-`.
/// Token kind for minus `-`.
Minus,
/// Token value for star `*`.
/// Token kind for star `*`.
Star,
/// Token value for slash `/`.
/// Token kind for slash `/`.
Slash,
/// Token value for vertical bar `|`.
/// Token kind for vertical bar `|`.
Vbar,
/// Token value for ampersand `&`.
/// Token kind for ampersand `&`.
Amper,
/// Token value for less than `<`.
/// Token kind for less than `<`.
Less,
/// Token value for greater than `>`.
/// Token kind for greater than `>`.
Greater,
/// Token value for equal `=`.
/// Token kind for equal `=`.
Equal,
/// Token value for dot `.`.
/// Token kind for dot `.`.
Dot,
/// Token value for percent `%`.
/// Token kind for percent `%`.
Percent,
/// Token value for left bracket `{`.
/// Token kind for left bracket `{`.
Lbrace,
/// Token value for right bracket `}`.
/// Token kind for right bracket `}`.
Rbrace,
/// Token value for double equal `==`.
/// Token kind for double equal `==`.
EqEqual,
/// Token value for not equal `!=`.
/// Token kind for not equal `!=`.
NotEqual,
/// Token value for less than or equal `<=`.
/// Token kind for less than or equal `<=`.
LessEqual,
/// Token value for greater than or equal `>=`.
/// Token kind for greater than or equal `>=`.
GreaterEqual,
/// Token value for tilde `~`.
/// Token kind for tilde `~`.
Tilde,
/// Token value for caret `^`.
/// Token kind for caret `^`.
CircumFlex,
/// Token value for left shift `<<`.
/// Token kind for left shift `<<`.
LeftShift,
/// Token value for right shift `>>`.
/// Token kind for right shift `>>`.
RightShift,
/// Token value for double star `**`.
/// Token kind for double star `**`.
DoubleStar,
/// Token value for double star equal `**=`.
/// Token kind for double star equal `**=`.
DoubleStarEqual,
/// Token value for plus equal `+=`.
/// Token kind for plus equal `+=`.
PlusEqual,
/// Token value for minus equal `-=`.
/// Token kind for minus equal `-=`.
MinusEqual,
/// Token value for star equal `*=`.
/// Token kind for star equal `*=`.
StarEqual,
/// Token value for slash equal `/=`.
/// Token kind for slash equal `/=`.
SlashEqual,
/// Token value for percent equal `%=`.
/// Token kind for percent equal `%=`.
PercentEqual,
/// Token value for ampersand equal `&=`.
/// Token kind for ampersand equal `&=`.
AmperEqual,
/// Token value for vertical bar equal `|=`.
/// Token kind for vertical bar equal `|=`.
VbarEqual,
/// Token value for caret equal `^=`.
/// Token kind for caret equal `^=`.
CircumflexEqual,
/// Token value for left shift equal `<<=`.
/// Token kind for left shift equal `<<=`.
LeftShiftEqual,
/// Token value for right shift equal `>>=`.
/// Token kind for right shift equal `>>=`.
RightShiftEqual,
/// Token value for double slash `//`.
/// Token kind for double slash `//`.
DoubleSlash,
/// Token value for double slash equal `//=`.
/// Token kind for double slash equal `//=`.
DoubleSlashEqual,
/// Token value for colon equal `:=`.
/// Token kind for colon equal `:=`.
ColonEqual,
/// Token value for at `@`.
/// Token kind for at `@`.
At,
/// Token value for at equal `@=`.
/// Token kind for at equal `@=`.
AtEqual,
/// Token value for arrow `->`.
/// Token kind for arrow `->`.
Rarrow,
/// Token value for ellipsis `...`.
/// Token kind for ellipsis `...`.
Ellipsis,
// The keywords should be sorted in alphabetical order. If the boundary tokens for the
@ -534,6 +192,11 @@ pub enum TokenKind {
}
impl TokenKind {
#[inline]
pub const fn is_eof(self) -> bool {
matches!(self, TokenKind::EndOfFile)
}
#[inline]
pub const fn is_newline(self) -> bool {
matches!(self, TokenKind::Newline | TokenKind::NonLogicalNewline)
@ -541,7 +204,10 @@ impl TokenKind {
/// Returns `true` if the token is a keyword (including soft keywords).
///
/// See also [`TokenKind::is_soft_keyword`], [`TokenKind::is_non_soft_keyword`].
/// See also [`is_soft_keyword`], [`is_non_soft_keyword`].
///
/// [`is_soft_keyword`]: TokenKind::is_soft_keyword
/// [`is_non_soft_keyword`]: TokenKind::is_non_soft_keyword
#[inline]
pub fn is_keyword(self) -> bool {
TokenKind::And <= self && self <= TokenKind::Type
@ -549,7 +215,10 @@ impl TokenKind {
/// Returns `true` if the token is strictly a soft keyword.
///
/// See also [`TokenKind::is_keyword`], [`TokenKind::is_non_soft_keyword`].
/// See also [`is_keyword`], [`is_non_soft_keyword`].
///
/// [`is_keyword`]: TokenKind::is_keyword
/// [`is_non_soft_keyword`]: TokenKind::is_non_soft_keyword
#[inline]
pub fn is_soft_keyword(self) -> bool {
TokenKind::Case <= self && self <= TokenKind::Type
@ -557,7 +226,10 @@ impl TokenKind {
/// Returns `true` if the token is strictly a non-soft keyword.
///
/// See also [`TokenKind::is_keyword`], [`TokenKind::is_soft_keyword`].
/// See also [`is_keyword`], [`is_soft_keyword`].
///
/// [`is_keyword`]: TokenKind::is_keyword
/// [`is_soft_keyword`]: TokenKind::is_soft_keyword
#[inline]
pub fn is_non_soft_keyword(self) -> bool {
TokenKind::And <= self && self <= TokenKind::Yield
@ -677,10 +349,12 @@ impl TokenKind {
matches!(self, TokenKind::Plus | TokenKind::Minus)
}
/// Returns the [`UnaryOp`] that corresponds to this token kind, if it is an arithmetic unary
/// Returns the [`UnaryOp`] that corresponds to this token kind, if it is a unary arithmetic
/// operator, otherwise return [None].
///
/// Use [`TokenKind::as_unary_operator`] to match against any unary operator.
/// Use [`as_unary_operator`] to match against any unary operator.
///
/// [`as_unary_operator`]: TokenKind::as_unary_operator
#[inline]
pub(crate) const fn as_unary_arithmetic_operator(self) -> Option<UnaryOp> {
Some(match self {
@ -693,8 +367,9 @@ impl TokenKind {
/// Returns the [`UnaryOp`] that corresponds to this token kind, if it is a unary operator,
/// otherwise return [None].
///
/// Use [`TokenKind::as_unary_arithmetic_operator`] to match against only an arithmetic unary
/// operator.
/// Use [`as_unary_arithmetic_operator`] to match against only an arithmetic unary operator.
///
/// [`as_unary_arithmetic_operator`]: TokenKind::as_unary_arithmetic_operator
#[inline]
pub(crate) const fn as_unary_operator(self) -> Option<UnaryOp> {
Some(match self {
@ -720,8 +395,9 @@ impl TokenKind {
/// Returns the binary [`Operator`] that corresponds to the current token, if it's a binary
/// operator, otherwise return [None].
///
/// Use [`TokenKind::as_augmented_assign_operator`] to match against an augmented assignment
/// token.
/// Use [`as_augmented_assign_operator`] to match against an augmented assignment token.
///
/// [`as_augmented_assign_operator`]: TokenKind::as_augmented_assign_operator
pub(crate) const fn as_binary_operator(self) -> Option<Operator> {
Some(match self {
TokenKind::Plus => Operator::Add,
@ -762,126 +438,6 @@ impl TokenKind {
_ => return None,
})
}
pub const fn from_token(token: &Tok) -> Self {
match token {
Tok::Name { .. } => TokenKind::Name,
Tok::Int { .. } => TokenKind::Int,
Tok::Float { .. } => TokenKind::Float,
Tok::Complex { .. } => TokenKind::Complex,
Tok::String { .. } => TokenKind::String,
Tok::FStringStart(_) => TokenKind::FStringStart,
Tok::FStringMiddle { .. } => TokenKind::FStringMiddle,
Tok::FStringEnd => TokenKind::FStringEnd,
Tok::IpyEscapeCommand { .. } => TokenKind::IpyEscapeCommand,
Tok::Comment(_) => TokenKind::Comment,
Tok::Newline => TokenKind::Newline,
Tok::NonLogicalNewline => TokenKind::NonLogicalNewline,
Tok::Indent => TokenKind::Indent,
Tok::Dedent => TokenKind::Dedent,
Tok::EndOfFile => TokenKind::EndOfFile,
Tok::Question => TokenKind::Question,
Tok::Exclamation => TokenKind::Exclamation,
Tok::Lpar => TokenKind::Lpar,
Tok::Rpar => TokenKind::Rpar,
Tok::Lsqb => TokenKind::Lsqb,
Tok::Rsqb => TokenKind::Rsqb,
Tok::Colon => TokenKind::Colon,
Tok::Comma => TokenKind::Comma,
Tok::Semi => TokenKind::Semi,
Tok::Plus => TokenKind::Plus,
Tok::Minus => TokenKind::Minus,
Tok::Star => TokenKind::Star,
Tok::Slash => TokenKind::Slash,
Tok::Vbar => TokenKind::Vbar,
Tok::Amper => TokenKind::Amper,
Tok::Less => TokenKind::Less,
Tok::Greater => TokenKind::Greater,
Tok::Equal => TokenKind::Equal,
Tok::Dot => TokenKind::Dot,
Tok::Percent => TokenKind::Percent,
Tok::Lbrace => TokenKind::Lbrace,
Tok::Rbrace => TokenKind::Rbrace,
Tok::EqEqual => TokenKind::EqEqual,
Tok::NotEqual => TokenKind::NotEqual,
Tok::LessEqual => TokenKind::LessEqual,
Tok::GreaterEqual => TokenKind::GreaterEqual,
Tok::Tilde => TokenKind::Tilde,
Tok::CircumFlex => TokenKind::CircumFlex,
Tok::LeftShift => TokenKind::LeftShift,
Tok::RightShift => TokenKind::RightShift,
Tok::DoubleStar => TokenKind::DoubleStar,
Tok::DoubleStarEqual => TokenKind::DoubleStarEqual,
Tok::PlusEqual => TokenKind::PlusEqual,
Tok::MinusEqual => TokenKind::MinusEqual,
Tok::StarEqual => TokenKind::StarEqual,
Tok::SlashEqual => TokenKind::SlashEqual,
Tok::PercentEqual => TokenKind::PercentEqual,
Tok::AmperEqual => TokenKind::AmperEqual,
Tok::VbarEqual => TokenKind::VbarEqual,
Tok::CircumflexEqual => TokenKind::CircumflexEqual,
Tok::LeftShiftEqual => TokenKind::LeftShiftEqual,
Tok::RightShiftEqual => TokenKind::RightShiftEqual,
Tok::DoubleSlash => TokenKind::DoubleSlash,
Tok::DoubleSlashEqual => TokenKind::DoubleSlashEqual,
Tok::ColonEqual => TokenKind::ColonEqual,
Tok::At => TokenKind::At,
Tok::AtEqual => TokenKind::AtEqual,
Tok::Rarrow => TokenKind::Rarrow,
Tok::Ellipsis => TokenKind::Ellipsis,
Tok::False => TokenKind::False,
Tok::None => TokenKind::None,
Tok::True => TokenKind::True,
Tok::And => TokenKind::And,
Tok::As => TokenKind::As,
Tok::Assert => TokenKind::Assert,
Tok::Async => TokenKind::Async,
Tok::Await => TokenKind::Await,
Tok::Break => TokenKind::Break,
Tok::Class => TokenKind::Class,
Tok::Continue => TokenKind::Continue,
Tok::Def => TokenKind::Def,
Tok::Del => TokenKind::Del,
Tok::Elif => TokenKind::Elif,
Tok::Else => TokenKind::Else,
Tok::Except => TokenKind::Except,
Tok::Finally => TokenKind::Finally,
Tok::For => TokenKind::For,
Tok::From => TokenKind::From,
Tok::Global => TokenKind::Global,
Tok::If => TokenKind::If,
Tok::Import => TokenKind::Import,
Tok::In => TokenKind::In,
Tok::Is => TokenKind::Is,
Tok::Lambda => TokenKind::Lambda,
Tok::Nonlocal => TokenKind::Nonlocal,
Tok::Not => TokenKind::Not,
Tok::Or => TokenKind::Or,
Tok::Pass => TokenKind::Pass,
Tok::Raise => TokenKind::Raise,
Tok::Return => TokenKind::Return,
Tok::Try => TokenKind::Try,
Tok::While => TokenKind::While,
Tok::Match => TokenKind::Match,
Tok::Case => TokenKind::Case,
Tok::Type => TokenKind::Type,
Tok::With => TokenKind::With,
Tok::Yield => TokenKind::Yield,
Tok::Unknown => TokenKind::Unknown,
}
}
}
impl From<&Tok> for TokenKind {
fn from(value: &Tok) -> Self {
Self::from_token(value)
}
}
impl From<Tok> for TokenKind {
fn from(value: Tok) -> Self {
Self::from_token(&value)
}
}
impl From<BoolOp> for TokenKind {
@ -1041,10 +597,8 @@ impl fmt::Display for TokenKind {
#[cfg(target_pointer_width = "64")]
mod sizes {
use crate::lexer::{LexicalError, LexicalErrorType};
use crate::Tok;
use static_assertions::assert_eq_size;
assert_eq_size!(Tok, [u8; 24]);
assert_eq_size!(LexicalErrorType, [u8; 24]);
assert_eq_size!(Result<Tok, LexicalError>, [u8; 32]);
assert_eq_size!(LexicalError, [u8; 32]);
}

248
crates/ruff_python_parser/src/token_source.rs
View file

@ -1,115 +1,189 @@
use std::iter::FusedIterator;
use ruff_python_trivia::CommentRanges;
use ruff_text_size::{TextRange, TextSize};
use crate::lexer::{LexResult, LexicalError, Spanned};
use crate::{Tok, TokenKind};
use crate::lexer::{Lexer, LexerCheckpoint, LexicalError, Token, TokenFlags, TokenValue};
use crate::{Mode, TokenKind};
#[derive(Clone, Debug)]
pub(crate) struct TokenSource {
tokens: std::vec::IntoIter<LexResult>,
errors: Vec<LexicalError>,
/// Token source for the parser that skips over any trivia tokens.
#[derive(Debug)]
pub(crate) struct TokenSource<'src> {
/// The underlying source for the tokens.
lexer: Lexer<'src>,
/// A vector containing all the tokens emitted by the lexer. This is returned when the parser
/// is finished consuming all the tokens. Note that unlike the emitted tokens, this vector
/// holds both the trivia and non-trivia tokens.
tokens: Vec<Token>,
/// A vector containing the range of all the comment tokens emitted by the lexer.
comments: Vec<TextRange>,
}
impl TokenSource {
pub(crate) fn new(tokens: Vec<LexResult>) -> Self {
Self {
tokens: tokens.into_iter(),
errors: Vec::new(),
impl<'src> TokenSource<'src> {
/// Create a new token source for the given lexer.
pub(crate) fn new(lexer: Lexer<'src>) -> Self {
// TODO(dhruvmanila): Use `allocate_tokens_vec`
TokenSource {
lexer,
tokens: vec![],
comments: vec![],
}
}
/// Returns the position of the current token.
/// Create a new token source from the given source code which starts at the given offset.
pub(crate) fn from_source(source: &'src str, mode: Mode, start_offset: TextSize) -> Self {
let lexer = Lexer::new(source, mode, start_offset);
let mut source = TokenSource::new(lexer);
// Initialize the token source so that the current token is set correctly.
source.do_bump();
source
}
/// Returns the kind of the current token.
pub(crate) fn current_kind(&self) -> TokenKind {
self.lexer.current_kind()
}
/// Returns the range of the current token.
pub(crate) fn current_range(&self) -> TextRange {
self.lexer.current_range()
}
/// Returns the flags for the current token.
pub(crate) fn current_flags(&self) -> TokenFlags {
self.lexer.current_flags()
}
/// Calls the underlying [`take_value`] method on the lexer. Refer to its documentation
/// for more info.
///
/// This is the position before any whitespace or comments.
pub(crate) fn position(&self) -> Option<TextSize> {
let first = self.tokens.as_slice().first()?;
let range = match first {
Ok((_, range)) => *range,
Err(error) => error.location(),
};
Some(range.start())
/// [`take_value`]: Lexer::take_value
pub(crate) fn take_value(&mut self) -> TokenValue {
self.lexer.take_value()
}
/// Returns the end of the last token
pub(crate) fn end(&self) -> Option<TextSize> {
let last = self.tokens.as_slice().last()?;
let range = match last {
Ok((_, range)) => *range,
Err(error) => error.location(),
};
Some(range.end())
/// Returns the next non-trivia token without consuming it.
///
/// Use [`peek2`] to get the next two tokens.
///
/// [`peek2`]: TokenSource::peek2
pub(crate) fn peek(&mut self) -> TokenKind {
let checkpoint = self.lexer.checkpoint();
let next = self.next_non_trivia_token();
self.lexer.rewind(checkpoint);
next
}
/// Returns the next token kind and its range without consuming it.
pub(crate) fn peek(&self) -> Option<(TokenKind, TextRange)> {
let mut iter = self.tokens.as_slice().iter();
/// Returns the next two non-trivia tokens without consuming it.
///
/// Use [`peek`] to only get the next token.
///
/// [`peek`]: TokenSource::peek
pub(crate) fn peek2(&mut self) -> (TokenKind, TokenKind) {
let checkpoint = self.lexer.checkpoint();
let first = self.next_non_trivia_token();
let second = self.next_non_trivia_token();
self.lexer.rewind(checkpoint);
(first, second)
}
/// Bumps the token source to the next non-trivia token.
///
/// It pushes the given kind to the token vector with the current token range.
pub(crate) fn bump(&mut self, kind: TokenKind) {
self.tokens
.push(Token::new(kind, self.current_range(), self.current_flags()));
self.do_bump();
}
/// Bumps the token source to the next non-trivia token without adding the current token to the
/// token vector. It does add the trivia tokens to the token vector.
fn do_bump(&mut self) {
loop {
let next = iter.next()?;
if next.as_ref().is_ok_and(is_trivia) {
let kind = self.lexer.next_token();
if is_trivia(kind) {
if kind == TokenKind::Comment {
self.comments.push(self.current_range());
}
self.tokens
.push(Token::new(kind, self.current_range(), self.current_flags()));
continue;
}
break Some(match next {
Ok((token, range)) => (TokenKind::from_token(token), *range),
Err(error) => (TokenKind::Unknown, error.location()),
});
break;
}
}
pub(crate) fn finish(self) -> Vec<LexicalError> {
/// Returns the next non-trivia token without adding it to the token vector.
fn next_non_trivia_token(&mut self) -> TokenKind {
loop {
let kind = self.lexer.next_token();
if is_trivia(kind) {
continue;
}
break kind;
}
}
/// Creates a checkpoint to which the token source can later return to using [`Self::rewind`].
pub(crate) fn checkpoint(&self) -> TokenSourceCheckpoint<'src> {
TokenSourceCheckpoint {
lexer_checkpoint: self.lexer.checkpoint(),
tokens_position: self.tokens.len(),
comments_position: self.comments.len(),
}
}
/// Restore the token source to the given checkpoint.
pub(crate) fn rewind(&mut self, checkpoint: TokenSourceCheckpoint<'src>) {
let TokenSourceCheckpoint {
lexer_checkpoint,
tokens_position,
comments_position,
} = checkpoint;
self.lexer.rewind(lexer_checkpoint);
self.tokens.truncate(tokens_position);
self.comments.truncate(comments_position);
}
/// Consumes the token source, returning the collected tokens, comment ranges, and any errors
/// encountered during lexing. The token collection includes both the trivia and non-trivia
/// tokens.
pub(crate) fn finish(mut self) -> (Vec<Token>, CommentRanges, Vec<LexicalError>) {
assert_eq!(
self.tokens.as_slice(),
&[],
"TokenSource was not fully consumed."
self.current_kind(),
TokenKind::EndOfFile,
"TokenSource was not fully consumed"
);
self.errors
}
}
impl FromIterator<LexResult> for TokenSource {
#[inline]
fn from_iter<T: IntoIterator<Item = LexResult>>(iter: T) -> Self {
Self::new(Vec::from_iter(iter))
}
}
impl Iterator for TokenSource {
type Item = Spanned;
#[inline]
fn next(&mut self) -> Option<Self::Item> {
loop {
let next = self.tokens.next()?;
match next {
Ok(token) => {
if is_trivia(&token) {
continue;
}
break Some(token);
}
Err(error) => {
let location = error.location();
self.errors.push(error);
break Some((Tok::Unknown, location));
}
}
// The `EndOfFile` token shouldn't be included in the token stream, it's mainly to signal
// the parser to stop. This isn't in `do_bump` because it only needs to be done once.
if let Some(last) = self.tokens.pop() {
assert_eq!(last.kind(), TokenKind::EndOfFile);
}
let comment_ranges = CommentRanges::new(self.comments);
(self.tokens, comment_ranges, self.lexer.finish())
}
}
impl FusedIterator for TokenSource {}
const fn is_trivia(result: &Spanned) -> bool {
matches!(result, (Tok::Comment(_) | Tok::NonLogicalNewline, _))
pub(crate) struct TokenSourceCheckpoint<'src> {
lexer_checkpoint: LexerCheckpoint<'src>,
tokens_position: usize,
comments_position: usize,
}
/// Allocates a [`Vec`] with an approximated capacity to fit all tokens
/// of `contents`.
///
/// See [#9546](https://github.com/astral-sh/ruff/pull/9546) for a more detailed explanation.
#[allow(dead_code)]
fn allocate_tokens_vec(contents: &str) -> Vec<Token> {
let lower_bound = contents.len().saturating_mul(15) / 100;
Vec::with_capacity(lower_bound)
}
fn is_trivia(token: TokenKind) -> bool {
matches!(token, TokenKind::Comment | TokenKind::NonLogicalNewline)
}

19
crates/ruff_python_parser/src/typing.rs
View file

@ -6,7 +6,7 @@ use ruff_python_ast::relocate::relocate_expr;
use ruff_python_ast::{str, Expr};
use ruff_text_size::{TextLen, TextRange};
use crate::{parse_expression, parse_expression_starts_at};
use crate::{parse_expression, parse_expression_range};
#[derive(is_macro::Is, Copy, Clone, Debug)]
pub enum AnnotationKind {
@ -22,25 +22,30 @@ pub enum AnnotationKind {
Complex,
}
/// Parse a type annotation from a string.
/// Parses the value of a string literal node (`parsed_contents`) with `range` as a type
/// annotation. The given `source` is the entire source code.
pub fn parse_type_annotation(
value: &str,
parsed_contents: &str,
range: TextRange,
source: &str,
) -> Result<(Expr, AnnotationKind)> {
let expression = &source[range];
if str::raw_contents(expression).is_some_and(|body| body == value) {
if str::raw_contents(expression).is_some_and(|raw_contents| raw_contents == parsed_contents) {
// The annotation is considered "simple" if and only if the raw representation (e.g.,
// `List[int]` within "List[int]") exactly matches the parsed representation. This
// isn't the case, e.g., for implicit concatenations, or for annotations that contain
// escaped quotes.
let leading_quote = str::leading_quote(expression).unwrap();
let expr = parse_expression_starts_at(value, range.start() + leading_quote.text_len())?;
let leading_quote_len = str::leading_quote(expression).unwrap().text_len();
let trailing_quote_len = str::trailing_quote(expression).unwrap().text_len();
let range = range
.add_start(leading_quote_len)
.sub_end(trailing_quote_len);
let expr = parse_expression_range(source, range)?.into_expr();
Ok((expr, AnnotationKind::Simple))
} else {
// Otherwise, consider this a "complex" annotation.
let mut expr = parse_expression(value)?;
let mut expr = parse_expression(parsed_contents)?.into_expr();
relocate_expr(&mut expr, range);
Ok((expr, AnnotationKind::Complex))
}