language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-08-17 09:00:48 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Maintain synchronicity between the lexer and the parser (#11457)

Browse source 
## Summary

This PR updates the entire parser stack in multiple ways:

### Make the lexer lazy

* https://github.com/astral-sh/ruff/pull/11244
* https://github.com/astral-sh/ruff/pull/11473

Previously, Ruff's lexer would act as an iterator. The parser would
collect all the tokens in a vector first and then process the tokens to
create the syntax tree.

The first task in this project is to update the entire parsing flow to
make the lexer lazy. This includes the `Lexer`, `TokenSource`, and
`Parser`. For context, the `TokenSource` is a wrapper around the `Lexer`
to filter out the trivia tokens[^1]. Now, the parser will ask the token
source to get the next token and only then the lexer will continue and
emit the token. This means that the lexer needs to be aware of the
"current" token. When the `next_token` is called, the current token will
be updated with the newly lexed token.

The main motivation to make the lexer lazy is to allow re-lexing a token
in a different context. This is going to be really useful to make the
parser error resilience. For example, currently the emitted tokens
remains the same even if the parser can recover from an unclosed
parenthesis. This is important because the lexer emits a
`NonLogicalNewline` in parenthesized context while a normal `Newline` in
non-parenthesized context. This different kinds of newline is also used
to emit the indentation tokens which is important for the parser as it's
used to determine the start and end of a block.

Additionally, this allows us to implement the following functionalities:
1. Checkpoint - rewind infrastructure: The idea here is to create a
checkpoint and continue lexing. At a later point, this checkpoint can be
used to rewind the lexer back to the provided checkpoint.
2. Remove the `SoftKeywordTransformer` and instead use lookahead or
speculative parsing to determine whether a soft keyword is a keyword or
an identifier
3. Remove the `Tok` enum. The `Tok` enum represents the tokens emitted
by the lexer but it contains owned data which makes it expensive to
clone. The new `TokenKind` enum just represents the type of token which
is very cheap.

This brings up a question as to how will the parser get the owned value
which was stored on `Tok`. This will be solved by introducing a new
`TokenValue` enum which only contains a subset of token kinds which has
the owned value. This is stored on the lexer and is requested by the
parser when it wants to process the data. For example:
8196720f80/crates/ruff_python_parser/src/parser/expression.rs (L1260-L1262)

[^1]: Trivia tokens are `NonLogicalNewline` and `Comment`

### Remove `SoftKeywordTransformer`

* https://github.com/astral-sh/ruff/pull/11441
* https://github.com/astral-sh/ruff/pull/11459
* https://github.com/astral-sh/ruff/pull/11442
* https://github.com/astral-sh/ruff/pull/11443
* https://github.com/astral-sh/ruff/pull/11474

For context,
https://github.com/RustPython/RustPython/pull/4519/files#diff-5de40045e78e794aa5ab0b8aacf531aa477daf826d31ca129467703855408220
added support for soft keywords in the parser which uses infinite
lookahead to classify a soft keyword as a keyword or an identifier. This
is a brilliant idea as it basically wraps the existing Lexer and works
on top of it which means that the logic for lexing and re-lexing a soft
keyword remains separate. The change here is to remove
`SoftKeywordTransformer` and let the parser determine this based on
context, lookahead and speculative parsing.

* **Context:** The transformer needs to know the position of the lexer
between it being at a statement position or a simple statement position.
This is because a `match` token starts a compound statement while a
`type` token starts a simple statement. **The parser already knows
this.**
* **Lookahead:** Now that the parser knows the context it can perform
lookahead of up to two tokens to classify the soft keyword. The logic
for this is mentioned in the PR implementing it for `type` and `match
soft keyword.
* **Speculative parsing:** This is where the checkpoint - rewind
infrastructure helps. For `match` soft keyword, there are certain cases
for which we can't classify based on lookahead. The idea here is to
create a checkpoint and keep parsing. Based on whether the parsing was
successful and what tokens are ahead we can classify the remaining
cases. Refer to #11443 for more details.

If the soft keyword is being parsed in an identifier context, it'll be
converted to an identifier and the emitted token will be updated as
well. Refer
8196720f80/crates/ruff_python_parser/src/parser/expression.rs (L487-L491).

The `case` soft keyword doesn't require any special handling because
it'll be a keyword only in the context of a match statement.

### Update the parser API

* https://github.com/astral-sh/ruff/pull/11494
* https://github.com/astral-sh/ruff/pull/11505

Now that the lexer is in sync with the parser, and the parser helps to
determine whether a soft keyword is a keyword or an identifier, the
lexer cannot be used on its own. The reason being that it's not
sensitive to the context (which is correct). This means that the parser
API needs to be updated to not allow any access to the lexer.

Previously, there were multiple ways to parse the source code:
1. Passing the source code itself
2. Or, passing the tokens

Now that the lexer and parser are working together, the API
corresponding to (2) cannot exists. The final API is mentioned in this
PR description: https://github.com/astral-sh/ruff/pull/11494.

### Refactor the downstream tools (linter and formatter)

* https://github.com/astral-sh/ruff/pull/11511
* https://github.com/astral-sh/ruff/pull/11515
* https://github.com/astral-sh/ruff/pull/11529
* https://github.com/astral-sh/ruff/pull/11562
* https://github.com/astral-sh/ruff/pull/11592

And, the final set of changes involves updating all references of the
lexer and `Tok` enum. This was done in two-parts:
1. Update all the references in a way that doesn't require any changes
from this PR i.e., it can be done independently
	* https://github.com/astral-sh/ruff/pull/11402
	* https://github.com/astral-sh/ruff/pull/11406
	* https://github.com/astral-sh/ruff/pull/11418
	* https://github.com/astral-sh/ruff/pull/11419
	* https://github.com/astral-sh/ruff/pull/11420
	* https://github.com/astral-sh/ruff/pull/11424
2. Update all the remaining references to use the changes made in this
PR

For (2), there were various strategies used:
1. Introduce a new `Tokens` struct which wraps the token vector and add
methods to query a certain subset of tokens. These includes:
	1. `up_to_first_unknown` which replaces the `tokenize` function
2. `in_range` and `after` which replaces the `lex_starts_at` function
where the former returns the tokens within the given range while the
latter returns all the tokens after the given offset
2. Introduce a new `TokenFlags` which is a set of flags to query certain
information from a token. Currently, this information is only limited to
any string type token but can be expanded to include other information
in the future as needed. https://github.com/astral-sh/ruff/pull/11578
3. Move the `CommentRanges` to the parsed output because this
information is common to both the linter and the formatter. This removes
the need for `tokens_and_ranges` function.

## Test Plan

- [x] Update and verify the test snapshots
- [x] Make sure the entire test suite is passing
- [x] Make sure there are no changes in the ecosystem checks
- [x] Run the fuzzer on the parser
- [x] Run this change on dozens of open-source projects

### Running this change on dozens of open-source projects

Refer to the PR description to get the list of open source projects used
for testing.

Now, the following tests were done between `main` and this branch:
1. Compare the output of `--select=E999` (syntax errors)
2. Compare the output of default rule selection
3. Compare the output of `--select=ALL`

**Conclusion: all output were same**

## What's next?

The next step is to introduce re-lexing logic and update the parser to
feed the recovery information to the lexer so that it can emit the
correct token. This moves us one step closer to having error resilience
in the parser and provides Ruff the possibility to lint even if the
source code contains syntax errors.
This commit is contained in:
Dhruv Manilawala 2024-06-03 18:23:50 +05:30 • committed by GitHub
parent c69a789aa5
commit bf5b62edac
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
262 changed files with 8174 additions and 6132 deletions
Download patch file Download diff file Expand all files Collapse all files

634
crates/ruff_python_parser/src/token.rs
View file

@ -1,4 +1,4 @@
//! Token type for Python source code created by the lexer and consumed by the `ruff_python_parser`.
//! Token kinds for Python source code created by the lexer and consumed by the `ruff_python_parser`.
//!
//! This module defines the tokens that the lexer recognizes. The tokens are
//! loosely based on the token definitions found in the [CPython source].
@ -7,482 +7,140 @@
use std::fmt;
use ruff_python_ast::{AnyStringFlags, BoolOp, Int, IpyEscapeKind, Operator, StringFlags, UnaryOp};
use ruff_python_ast::{BoolOp, Operator, UnaryOp};
/// The set of tokens the Python source code can be tokenized in.
#[derive(Clone, Debug, PartialEq, is_macro::Is)]
pub enum Tok {
/// Token value for a name, commonly known as an identifier.
Name {
/// The name value.
///
/// Unicode names are NFKC-normalized by the lexer,
/// matching [the behaviour of Python's lexer](https://docs.python.org/3/reference/lexical_analysis.html#identifiers)
name: Box<str>,
},
/// Token value for an integer.
Int {
/// The integer value.
value: Int,
},
/// Token value for a floating point number.
Float {
/// The float value.
value: f64,
},
/// Token value for a complex number.
Complex {
/// The real part of the complex number.
real: f64,
/// The imaginary part of the complex number.
imag: f64,
},
/// Token value for a string.
String {
/// The string value.
value: Box<str>,
/// Flags that can be queried to determine the quote style
/// and prefixes of the string
flags: AnyStringFlags,
},
/// Token value for the start of an f-string. This includes the `f`/`F`/`fr` prefix
/// and the opening quote(s).
FStringStart(AnyStringFlags),
/// Token value that includes the portion of text inside the f-string that's not
/// part of the expression part and isn't an opening or closing brace.
FStringMiddle {
/// The string value.
value: Box<str>,
/// Flags that can be queried to determine the quote style
/// and prefixes of the string
flags: AnyStringFlags,
},
/// Token value for the end of an f-string. This includes the closing quote.
FStringEnd,
/// Token value for IPython escape commands. These are recognized by the lexer
/// only when the mode is [`Ipython`].
///
/// [`Ipython`]: crate::Mode::Ipython
IpyEscapeCommand {
/// The magic command value.
value: Box<str>,
/// The kind of magic command.
kind: IpyEscapeKind,
},
/// Token value for a comment. These are filtered out of the token stream prior to parsing.
Comment(Box<str>),
/// Token value for a newline.
Newline,
/// Token value for a newline that is not a logical line break. These are filtered out of
/// the token stream prior to parsing.
NonLogicalNewline,
/// Token value for an indent.
Indent,
/// Token value for a dedent.
Dedent,
EndOfFile,
/// Token value for a question mark `?`. This is only used in [`Ipython`].
///
/// [`Ipython`]: crate::Mode::Ipython
Question,
/// Token value for a exclamation mark `!`.
Exclamation,
/// Token value for a left parenthesis `(`.
Lpar,
/// Token value for a right parenthesis `)`.
Rpar,
/// Token value for a left square bracket `[`.
Lsqb,
/// Token value for a right square bracket `]`.
Rsqb,
/// Token value for a colon `:`.
Colon,
/// Token value for a comma `,`.
Comma,
/// Token value for a semicolon `;`.
Semi,
/// Token value for plus `+`.
Plus,
/// Token value for minus `-`.
Minus,
/// Token value for star `*`.
Star,
/// Token value for slash `/`.
Slash,
/// Token value for vertical bar `|`.
Vbar,
/// Token value for ampersand `&`.
Amper,
/// Token value for less than `<`.
Less,
/// Token value for greater than `>`.
Greater,
/// Token value for equal `=`.
Equal,
/// Token value for dot `.`.
Dot,
/// Token value for percent `%`.
Percent,
/// Token value for left bracket `{`.
Lbrace,
/// Token value for right bracket `}`.
Rbrace,
/// Token value for double equal `==`.
EqEqual,
/// Token value for not equal `!=`.
NotEqual,
/// Token value for less than or equal `<=`.
LessEqual,
/// Token value for greater than or equal `>=`.
GreaterEqual,
/// Token value for tilde `~`.
Tilde,
/// Token value for caret `^`.
CircumFlex,
/// Token value for left shift `<<`.
LeftShift,
/// Token value for right shift `>>`.
RightShift,
/// Token value for double star `**`.
DoubleStar,
/// Token value for double star equal `**=`.
DoubleStarEqual,
/// Token value for plus equal `+=`.
PlusEqual,
/// Token value for minus equal `-=`.
MinusEqual,
/// Token value for star equal `*=`.
StarEqual,
/// Token value for slash equal `/=`.
SlashEqual,
/// Token value for percent equal `%=`.
PercentEqual,
/// Token value for ampersand equal `&=`.
AmperEqual,
/// Token value for vertical bar equal `|=`.
VbarEqual,
/// Token value for caret equal `^=`.
CircumflexEqual,
/// Token value for left shift equal `<<=`.
LeftShiftEqual,
/// Token value for right shift equal `>>=`.
RightShiftEqual,
/// Token value for double slash `//`.
DoubleSlash,
/// Token value for double slash equal `//=`.
DoubleSlashEqual,
/// Token value for colon equal `:=`.
ColonEqual,
/// Token value for at `@`.
At,
/// Token value for at equal `@=`.
AtEqual,
/// Token value for arrow `->`.
Rarrow,
/// Token value for ellipsis `...`.
Ellipsis,
// Self documenting.
// Keywords (alphabetically):
False,
None,
True,
And,
As,
Assert,
Async,
Await,
Break,
Class,
Continue,
Def,
Del,
Elif,
Else,
Except,
Finally,
For,
From,
Global,
If,
Import,
In,
Is,
Lambda,
Nonlocal,
Not,
Or,
Pass,
Raise,
Return,
Try,
While,
Match,
Type,
Case,
With,
Yield,
Unknown,
}
impl Tok {
#[inline]
pub fn kind(&self) -> TokenKind {
TokenKind::from_token(self)
}
}
impl fmt::Display for Tok {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
#[allow(clippy::enum_glob_use)]
use Tok::*;
match self {
Name { name } => write!(f, "{name}"),
Int { value } => write!(f, "{value}"),
Float { value } => write!(f, "{value}"),
Complex { real, imag } => write!(f, "{real}j{imag}"),
String { value, flags } => {
write!(f, "{}", flags.format_string_contents(value))
}
FStringStart(_) => f.write_str("FStringStart"),
FStringMiddle { value, .. } => f.write_str(value),
FStringEnd => f.write_str("FStringEnd"),
IpyEscapeCommand { kind, value } => write!(f, "{kind}{value}"),
Newline => f.write_str("Newline"),
NonLogicalNewline => f.write_str("NonLogicalNewline"),
Indent => f.write_str("Indent"),
Dedent => f.write_str("Dedent"),
EndOfFile => f.write_str("EOF"),
Question => f.write_str("?"),
Exclamation => f.write_str("!"),
Lpar => f.write_str("("),
Rpar => f.write_str(")"),
Lsqb => f.write_str("["),
Rsqb => f.write_str("]"),
Colon => f.write_str(":"),
Comma => f.write_str(","),
Comment(value) => f.write_str(value),
Semi => f.write_str(";"),
Plus => f.write_str("+"),
Minus => f.write_str("-"),
Star => f.write_str("*"),
Slash => f.write_str("/"),
Vbar => f.write_str("|"),
Amper => f.write_str("&"),
Less => f.write_str("<"),
Greater => f.write_str(">"),
Equal => f.write_str("="),
Dot => f.write_str("."),
Percent => f.write_str("%"),
Lbrace => f.write_str("{"),
Rbrace => f.write_str("}"),
EqEqual => f.write_str("=="),
NotEqual => f.write_str("!="),
LessEqual => f.write_str("<="),
GreaterEqual => f.write_str(">="),
Tilde => f.write_str("~"),
CircumFlex => f.write_str("^"),
LeftShift => f.write_str("<<"),
RightShift => f.write_str(">>"),
DoubleStar => f.write_str("**"),
DoubleStarEqual => f.write_str("**="),
PlusEqual => f.write_str("+="),
MinusEqual => f.write_str("-="),
StarEqual => f.write_str("*="),
SlashEqual => f.write_str("/="),
PercentEqual => f.write_str("%="),
AmperEqual => f.write_str("&="),
VbarEqual => f.write_str("|="),
CircumflexEqual => f.write_str("^="),
LeftShiftEqual => f.write_str("<<="),
RightShiftEqual => f.write_str(">>="),
DoubleSlash => f.write_str("//"),
DoubleSlashEqual => f.write_str("//="),
At => f.write_str("@"),
AtEqual => f.write_str("@="),
Rarrow => f.write_str("->"),
Ellipsis => f.write_str("..."),
False => f.write_str("False"),
None => f.write_str("None"),
True => f.write_str("True"),
And => f.write_str("and"),
As => f.write_str("as"),
Assert => f.write_str("assert"),
Async => f.write_str("async"),
Await => f.write_str("await"),
Break => f.write_str("break"),
Class => f.write_str("class"),
Continue => f.write_str("continue"),
Def => f.write_str("def"),
Del => f.write_str("del"),
Elif => f.write_str("elif"),
Else => f.write_str("else"),
Except => f.write_str("except"),
Finally => f.write_str("finally"),
For => f.write_str("for"),
From => f.write_str("from"),
Global => f.write_str("global"),
If => f.write_str("if"),
Import => f.write_str("import"),
In => f.write_str("in"),
Is => f.write_str("is"),
Lambda => f.write_str("lambda"),
Nonlocal => f.write_str("nonlocal"),
Not => f.write_str("not"),
Or => f.write_str("or"),
Pass => f.write_str("pass"),
Raise => f.write_str("raise"),
Return => f.write_str("return"),
Try => f.write_str("try"),
While => f.write_str("while"),
Match => f.write_str("match"),
Type => f.write_str("type"),
Case => f.write_str("case"),
With => f.write_str("with"),
Yield => f.write_str("yield"),
ColonEqual => f.write_str(":="),
Unknown => f.write_str("<Unknown>>"),
}
}
}
/// A kind of token.
///
/// This is a lightweight representation of [`Tok`] which doesn't contain any information
/// about the token itself.
/// A kind of a token.
#[derive(Copy, Clone, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)]
pub enum TokenKind {
/// Token value for a name, commonly known as an identifier.
/// Token kind for a name, commonly known as an identifier.
Name,
/// Token value for an integer.
/// Token kind for an integer.
Int,
/// Token value for a floating point number.
/// Token kind for a floating point number.
Float,
/// Token value for a complex number.
/// Token kind for a complex number.
Complex,
/// Token value for a string.
/// Token kind for a string.
String,
/// Token value for the start of an f-string. This includes the `f`/`F`/`fr` prefix
/// Token kind for the start of an f-string. This includes the `f`/`F`/`fr` prefix
/// and the opening quote(s).
FStringStart,
/// Token value that includes the portion of text inside the f-string that's not
/// Token kind that includes the portion of text inside the f-string that's not
/// part of the expression part and isn't an opening or closing brace.
FStringMiddle,
/// Token value for the end of an f-string. This includes the closing quote.
/// Token kind for the end of an f-string. This includes the closing quote.
FStringEnd,
/// Token value for a IPython escape command.
/// Token kind for a IPython escape command.
IpyEscapeCommand,
/// Token value for a comment. These are filtered out of the token stream prior to parsing.
/// Token kind for a comment. These are filtered out of the token stream prior to parsing.
Comment,
/// Token value for a newline.
/// Token kind for a newline.
Newline,
/// Token value for a newline that is not a logical line break. These are filtered out of
/// Token kind for a newline that is not a logical line break. These are filtered out of
/// the token stream prior to parsing.
NonLogicalNewline,
/// Token value for an indent.
/// Token kind for an indent.
Indent,
/// Token value for a dedent.
/// Token kind for a dedent.
Dedent,
EndOfFile,
/// Token value for a question mark `?`.
/// Token kind for a question mark `?`.
Question,
/// Token value for an exclamation mark `!`.
/// Token kind for an exclamation mark `!`.
Exclamation,
/// Token value for a left parenthesis `(`.
/// Token kind for a left parenthesis `(`.
Lpar,
/// Token value for a right parenthesis `)`.
/// Token kind for a right parenthesis `)`.
Rpar,
/// Token value for a left square bracket `[`.
/// Token kind for a left square bracket `[`.
Lsqb,
/// Token value for a right square bracket `]`.
/// Token kind for a right square bracket `]`.
Rsqb,
/// Token value for a colon `:`.
/// Token kind for a colon `:`.
Colon,
/// Token value for a comma `,`.
/// Token kind for a comma `,`.
Comma,
/// Token value for a semicolon `;`.
/// Token kind for a semicolon `;`.
Semi,
/// Token value for plus `+`.
/// Token kind for plus `+`.
Plus,
/// Token value for minus `-`.
/// Token kind for minus `-`.
Minus,
/// Token value for star `*`.
/// Token kind for star `*`.
Star,
/// Token value for slash `/`.
/// Token kind for slash `/`.
Slash,
/// Token value for vertical bar `|`.
/// Token kind for vertical bar `|`.
Vbar,
/// Token value for ampersand `&`.
/// Token kind for ampersand `&`.
Amper,
/// Token value for less than `<`.
/// Token kind for less than `<`.
Less,
/// Token value for greater than `>`.
/// Token kind for greater than `>`.
Greater,
/// Token value for equal `=`.
/// Token kind for equal `=`.
Equal,
/// Token value for dot `.`.
/// Token kind for dot `.`.
Dot,
/// Token value for percent `%`.
/// Token kind for percent `%`.
Percent,
/// Token value for left bracket `{`.
/// Token kind for left bracket `{`.
Lbrace,
/// Token value for right bracket `}`.
/// Token kind for right bracket `}`.
Rbrace,
/// Token value for double equal `==`.
/// Token kind for double equal `==`.
EqEqual,
/// Token value for not equal `!=`.
/// Token kind for not equal `!=`.
NotEqual,
/// Token value for less than or equal `<=`.
/// Token kind for less than or equal `<=`.
LessEqual,
/// Token value for greater than or equal `>=`.
/// Token kind for greater than or equal `>=`.
GreaterEqual,
/// Token value for tilde `~`.
/// Token kind for tilde `~`.
Tilde,
/// Token value for caret `^`.
/// Token kind for caret `^`.
CircumFlex,
/// Token value for left shift `<<`.
/// Token kind for left shift `<<`.
LeftShift,
/// Token value for right shift `>>`.
/// Token kind for right shift `>>`.
RightShift,
/// Token value for double star `**`.
/// Token kind for double star `**`.
DoubleStar,
/// Token value for double star equal `**=`.
/// Token kind for double star equal `**=`.
DoubleStarEqual,
/// Token value for plus equal `+=`.
/// Token kind for plus equal `+=`.
PlusEqual,
/// Token value for minus equal `-=`.
/// Token kind for minus equal `-=`.
MinusEqual,
/// Token value for star equal `*=`.
/// Token kind for star equal `*=`.
StarEqual,
/// Token value for slash equal `/=`.
/// Token kind for slash equal `/=`.
SlashEqual,
/// Token value for percent equal `%=`.
/// Token kind for percent equal `%=`.
PercentEqual,
/// Token value for ampersand equal `&=`.
/// Token kind for ampersand equal `&=`.
AmperEqual,
/// Token value for vertical bar equal `|=`.
/// Token kind for vertical bar equal `|=`.
VbarEqual,
/// Token value for caret equal `^=`.
/// Token kind for caret equal `^=`.
CircumflexEqual,
/// Token value for left shift equal `<<=`.
/// Token kind for left shift equal `<<=`.
LeftShiftEqual,
/// Token value for right shift equal `>>=`.
/// Token kind for right shift equal `>>=`.
RightShiftEqual,
/// Token value for double slash `//`.
/// Token kind for double slash `//`.
DoubleSlash,
/// Token value for double slash equal `//=`.
/// Token kind for double slash equal `//=`.
DoubleSlashEqual,
/// Token value for colon equal `:=`.
/// Token kind for colon equal `:=`.
ColonEqual,
/// Token value for at `@`.
/// Token kind for at `@`.
At,
/// Token value for at equal `@=`.
/// Token kind for at equal `@=`.
AtEqual,
/// Token value for arrow `->`.
/// Token kind for arrow `->`.
Rarrow,
/// Token value for ellipsis `...`.
/// Token kind for ellipsis `...`.
Ellipsis,
// The keywords should be sorted in alphabetical order. If the boundary tokens for the
@ -534,6 +192,11 @@ pub enum TokenKind {
}
impl TokenKind {
#[inline]
pub const fn is_eof(self) -> bool {
matches!(self, TokenKind::EndOfFile)
}
#[inline]
pub const fn is_newline(self) -> bool {
matches!(self, TokenKind::Newline | TokenKind::NonLogicalNewline)
@ -541,7 +204,10 @@ impl TokenKind {
/// Returns `true` if the token is a keyword (including soft keywords).
///
/// See also [`TokenKind::is_soft_keyword`], [`TokenKind::is_non_soft_keyword`].
/// See also [`is_soft_keyword`], [`is_non_soft_keyword`].
///
/// [`is_soft_keyword`]: TokenKind::is_soft_keyword
/// [`is_non_soft_keyword`]: TokenKind::is_non_soft_keyword
#[inline]
pub fn is_keyword(self) -> bool {
TokenKind::And <= self && self <= TokenKind::Type
@ -549,7 +215,10 @@ impl TokenKind {
/// Returns `true` if the token is strictly a soft keyword.
///
/// See also [`TokenKind::is_keyword`], [`TokenKind::is_non_soft_keyword`].
/// See also [`is_keyword`], [`is_non_soft_keyword`].
///
/// [`is_keyword`]: TokenKind::is_keyword
/// [`is_non_soft_keyword`]: TokenKind::is_non_soft_keyword
#[inline]
pub fn is_soft_keyword(self) -> bool {
TokenKind::Case <= self && self <= TokenKind::Type
@ -557,7 +226,10 @@ impl TokenKind {
/// Returns `true` if the token is strictly a non-soft keyword.
///
/// See also [`TokenKind::is_keyword`], [`TokenKind::is_soft_keyword`].
/// See also [`is_keyword`], [`is_soft_keyword`].
///
/// [`is_keyword`]: TokenKind::is_keyword
/// [`is_soft_keyword`]: TokenKind::is_soft_keyword
#[inline]
pub fn is_non_soft_keyword(self) -> bool {
TokenKind::And <= self && self <= TokenKind::Yield
@ -677,10 +349,12 @@ impl TokenKind {
matches!(self, TokenKind::Plus | TokenKind::Minus)
}
/// Returns the [`UnaryOp`] that corresponds to this token kind, if it is an arithmetic unary
/// Returns the [`UnaryOp`] that corresponds to this token kind, if it is a unary arithmetic
/// operator, otherwise return [None].
///
/// Use [`TokenKind::as_unary_operator`] to match against any unary operator.
/// Use [`as_unary_operator`] to match against any unary operator.
///
/// [`as_unary_operator`]: TokenKind::as_unary_operator
#[inline]
pub(crate) const fn as_unary_arithmetic_operator(self) -> Option<UnaryOp> {
Some(match self {
@ -693,8 +367,9 @@ impl TokenKind {
/// Returns the [`UnaryOp`] that corresponds to this token kind, if it is a unary operator,
/// otherwise return [None].
///
/// Use [`TokenKind::as_unary_arithmetic_operator`] to match against only an arithmetic unary
/// operator.
/// Use [`as_unary_arithmetic_operator`] to match against only an arithmetic unary operator.
///
/// [`as_unary_arithmetic_operator`]: TokenKind::as_unary_arithmetic_operator
#[inline]
pub(crate) const fn as_unary_operator(self) -> Option<UnaryOp> {
Some(match self {
@ -720,8 +395,9 @@ impl TokenKind {
/// Returns the binary [`Operator`] that corresponds to the current token, if it's a binary
/// operator, otherwise return [None].
///
/// Use [`TokenKind::as_augmented_assign_operator`] to match against an augmented assignment
/// token.
/// Use [`as_augmented_assign_operator`] to match against an augmented assignment token.
///
/// [`as_augmented_assign_operator`]: TokenKind::as_augmented_assign_operator
pub(crate) const fn as_binary_operator(self) -> Option<Operator> {
Some(match self {
TokenKind::Plus => Operator::Add,
@ -762,126 +438,6 @@ impl TokenKind {
_ => return None,
})
}
pub const fn from_token(token: &Tok) -> Self {
match token {
Tok::Name { .. } => TokenKind::Name,
Tok::Int { .. } => TokenKind::Int,
Tok::Float { .. } => TokenKind::Float,
Tok::Complex { .. } => TokenKind::Complex,
Tok::String { .. } => TokenKind::String,
Tok::FStringStart(_) => TokenKind::FStringStart,
Tok::FStringMiddle { .. } => TokenKind::FStringMiddle,
Tok::FStringEnd => TokenKind::FStringEnd,
Tok::IpyEscapeCommand { .. } => TokenKind::IpyEscapeCommand,
Tok::Comment(_) => TokenKind::Comment,
Tok::Newline => TokenKind::Newline,
Tok::NonLogicalNewline => TokenKind::NonLogicalNewline,
Tok::Indent => TokenKind::Indent,
Tok::Dedent => TokenKind::Dedent,
Tok::EndOfFile => TokenKind::EndOfFile,
Tok::Question => TokenKind::Question,
Tok::Exclamation => TokenKind::Exclamation,
Tok::Lpar => TokenKind::Lpar,
Tok::Rpar => TokenKind::Rpar,
Tok::Lsqb => TokenKind::Lsqb,
Tok::Rsqb => TokenKind::Rsqb,
Tok::Colon => TokenKind::Colon,
Tok::Comma => TokenKind::Comma,
Tok::Semi => TokenKind::Semi,
Tok::Plus => TokenKind::Plus,
Tok::Minus => TokenKind::Minus,
Tok::Star => TokenKind::Star,
Tok::Slash => TokenKind::Slash,
Tok::Vbar => TokenKind::Vbar,
Tok::Amper => TokenKind::Amper,
Tok::Less => TokenKind::Less,
Tok::Greater => TokenKind::Greater,
Tok::Equal => TokenKind::Equal,
Tok::Dot => TokenKind::Dot,
Tok::Percent => TokenKind::Percent,
Tok::Lbrace => TokenKind::Lbrace,
Tok::Rbrace => TokenKind::Rbrace,
Tok::EqEqual => TokenKind::EqEqual,
Tok::NotEqual => TokenKind::NotEqual,
Tok::LessEqual => TokenKind::LessEqual,
Tok::GreaterEqual => TokenKind::GreaterEqual,
Tok::Tilde => TokenKind::Tilde,
Tok::CircumFlex => TokenKind::CircumFlex,
Tok::LeftShift => TokenKind::LeftShift,
Tok::RightShift => TokenKind::RightShift,
Tok::DoubleStar => TokenKind::DoubleStar,
Tok::DoubleStarEqual => TokenKind::DoubleStarEqual,
Tok::PlusEqual => TokenKind::PlusEqual,
Tok::MinusEqual => TokenKind::MinusEqual,
Tok::StarEqual => TokenKind::StarEqual,
Tok::SlashEqual => TokenKind::SlashEqual,
Tok::PercentEqual => TokenKind::PercentEqual,
Tok::AmperEqual => TokenKind::AmperEqual,
Tok::VbarEqual => TokenKind::VbarEqual,
Tok::CircumflexEqual => TokenKind::CircumflexEqual,
Tok::LeftShiftEqual => TokenKind::LeftShiftEqual,
Tok::RightShiftEqual => TokenKind::RightShiftEqual,
Tok::DoubleSlash => TokenKind::DoubleSlash,
Tok::DoubleSlashEqual => TokenKind::DoubleSlashEqual,
Tok::ColonEqual => TokenKind::ColonEqual,
Tok::At => TokenKind::At,
Tok::AtEqual => TokenKind::AtEqual,
Tok::Rarrow => TokenKind::Rarrow,
Tok::Ellipsis => TokenKind::Ellipsis,
Tok::False => TokenKind::False,
Tok::None => TokenKind::None,
Tok::True => TokenKind::True,
Tok::And => TokenKind::And,
Tok::As => TokenKind::As,
Tok::Assert => TokenKind::Assert,
Tok::Async => TokenKind::Async,
Tok::Await => TokenKind::Await,
Tok::Break => TokenKind::Break,
Tok::Class => TokenKind::Class,
Tok::Continue => TokenKind::Continue,
Tok::Def => TokenKind::Def,
Tok::Del => TokenKind::Del,
Tok::Elif => TokenKind::Elif,
Tok::Else => TokenKind::Else,
Tok::Except => TokenKind::Except,
Tok::Finally => TokenKind::Finally,
Tok::For => TokenKind::For,
Tok::From => TokenKind::From,
Tok::Global => TokenKind::Global,
Tok::If => TokenKind::If,
Tok::Import => TokenKind::Import,
Tok::In => TokenKind::In,
Tok::Is => TokenKind::Is,
Tok::Lambda => TokenKind::Lambda,
Tok::Nonlocal => TokenKind::Nonlocal,
Tok::Not => TokenKind::Not,
Tok::Or => TokenKind::Or,
Tok::Pass => TokenKind::Pass,
Tok::Raise => TokenKind::Raise,
Tok::Return => TokenKind::Return,
Tok::Try => TokenKind::Try,
Tok::While => TokenKind::While,
Tok::Match => TokenKind::Match,
Tok::Case => TokenKind::Case,
Tok::Type => TokenKind::Type,
Tok::With => TokenKind::With,
Tok::Yield => TokenKind::Yield,
Tok::Unknown => TokenKind::Unknown,
}
}
}
impl From<&Tok> for TokenKind {
fn from(value: &Tok) -> Self {
Self::from_token(value)
}
}
impl From<Tok> for TokenKind {
fn from(value: Tok) -> Self {
Self::from_token(&value)
}
}
impl From<BoolOp> for TokenKind {
@ -1041,10 +597,8 @@ impl fmt::Display for TokenKind {
#[cfg(target_pointer_width = "64")]
mod sizes {
use crate::lexer::{LexicalError, LexicalErrorType};
use crate::Tok;
use static_assertions::assert_eq_size;
assert_eq_size!(Tok, [u8; 24]);
assert_eq_size!(LexicalErrorType, [u8; 24]);
assert_eq_size!(Result<Tok, LexicalError>, [u8; 32]);
assert_eq_size!(LexicalError, [u8; 32]);
}