mirror of
https://github.com/astral-sh/ruff.git
synced 2025-08-20 10:31:36 +00:00

## Summary This PR updates the entire parser stack in multiple ways: ### Make the lexer lazy * https://github.com/astral-sh/ruff/pull/11244 * https://github.com/astral-sh/ruff/pull/11473 Previously, Ruff's lexer would act as an iterator. The parser would collect all the tokens in a vector first and then process the tokens to create the syntax tree. The first task in this project is to update the entire parsing flow to make the lexer lazy. This includes the `Lexer`, `TokenSource`, and `Parser`. For context, the `TokenSource` is a wrapper around the `Lexer` to filter out the trivia tokens[^1]. Now, the parser will ask the token source to get the next token and only then the lexer will continue and emit the token. This means that the lexer needs to be aware of the "current" token. When the `next_token` is called, the current token will be updated with the newly lexed token. The main motivation to make the lexer lazy is to allow re-lexing a token in a different context. This is going to be really useful to make the parser error resilience. For example, currently the emitted tokens remains the same even if the parser can recover from an unclosed parenthesis. This is important because the lexer emits a `NonLogicalNewline` in parenthesized context while a normal `Newline` in non-parenthesized context. This different kinds of newline is also used to emit the indentation tokens which is important for the parser as it's used to determine the start and end of a block. Additionally, this allows us to implement the following functionalities: 1. Checkpoint - rewind infrastructure: The idea here is to create a checkpoint and continue lexing. At a later point, this checkpoint can be used to rewind the lexer back to the provided checkpoint. 2. Remove the `SoftKeywordTransformer` and instead use lookahead or speculative parsing to determine whether a soft keyword is a keyword or an identifier 3. Remove the `Tok` enum. The `Tok` enum represents the tokens emitted by the lexer but it contains owned data which makes it expensive to clone. The new `TokenKind` enum just represents the type of token which is very cheap. This brings up a question as to how will the parser get the owned value which was stored on `Tok`. This will be solved by introducing a new `TokenValue` enum which only contains a subset of token kinds which has the owned value. This is stored on the lexer and is requested by the parser when it wants to process the data. For example:8196720f80/crates/ruff_python_parser/src/parser/expression.rs (L1260-L1262)
[^1]: Trivia tokens are `NonLogicalNewline` and `Comment` ### Remove `SoftKeywordTransformer` * https://github.com/astral-sh/ruff/pull/11441 * https://github.com/astral-sh/ruff/pull/11459 * https://github.com/astral-sh/ruff/pull/11442 * https://github.com/astral-sh/ruff/pull/11443 * https://github.com/astral-sh/ruff/pull/11474 For context, https://github.com/RustPython/RustPython/pull/4519/files#diff-5de40045e78e794aa5ab0b8aacf531aa477daf826d31ca129467703855408220 added support for soft keywords in the parser which uses infinite lookahead to classify a soft keyword as a keyword or an identifier. This is a brilliant idea as it basically wraps the existing Lexer and works on top of it which means that the logic for lexing and re-lexing a soft keyword remains separate. The change here is to remove `SoftKeywordTransformer` and let the parser determine this based on context, lookahead and speculative parsing. * **Context:** The transformer needs to know the position of the lexer between it being at a statement position or a simple statement position. This is because a `match` token starts a compound statement while a `type` token starts a simple statement. **The parser already knows this.** * **Lookahead:** Now that the parser knows the context it can perform lookahead of up to two tokens to classify the soft keyword. The logic for this is mentioned in the PR implementing it for `type` and `match soft keyword. * **Speculative parsing:** This is where the checkpoint - rewind infrastructure helps. For `match` soft keyword, there are certain cases for which we can't classify based on lookahead. The idea here is to create a checkpoint and keep parsing. Based on whether the parsing was successful and what tokens are ahead we can classify the remaining cases. Refer to #11443 for more details. If the soft keyword is being parsed in an identifier context, it'll be converted to an identifier and the emitted token will be updated as well. Refer8196720f80/crates/ruff_python_parser/src/parser/expression.rs (L487-L491)
. The `case` soft keyword doesn't require any special handling because it'll be a keyword only in the context of a match statement. ### Update the parser API * https://github.com/astral-sh/ruff/pull/11494 * https://github.com/astral-sh/ruff/pull/11505 Now that the lexer is in sync with the parser, and the parser helps to determine whether a soft keyword is a keyword or an identifier, the lexer cannot be used on its own. The reason being that it's not sensitive to the context (which is correct). This means that the parser API needs to be updated to not allow any access to the lexer. Previously, there were multiple ways to parse the source code: 1. Passing the source code itself 2. Or, passing the tokens Now that the lexer and parser are working together, the API corresponding to (2) cannot exists. The final API is mentioned in this PR description: https://github.com/astral-sh/ruff/pull/11494. ### Refactor the downstream tools (linter and formatter) * https://github.com/astral-sh/ruff/pull/11511 * https://github.com/astral-sh/ruff/pull/11515 * https://github.com/astral-sh/ruff/pull/11529 * https://github.com/astral-sh/ruff/pull/11562 * https://github.com/astral-sh/ruff/pull/11592 And, the final set of changes involves updating all references of the lexer and `Tok` enum. This was done in two-parts: 1. Update all the references in a way that doesn't require any changes from this PR i.e., it can be done independently * https://github.com/astral-sh/ruff/pull/11402 * https://github.com/astral-sh/ruff/pull/11406 * https://github.com/astral-sh/ruff/pull/11418 * https://github.com/astral-sh/ruff/pull/11419 * https://github.com/astral-sh/ruff/pull/11420 * https://github.com/astral-sh/ruff/pull/11424 2. Update all the remaining references to use the changes made in this PR For (2), there were various strategies used: 1. Introduce a new `Tokens` struct which wraps the token vector and add methods to query a certain subset of tokens. These includes: 1. `up_to_first_unknown` which replaces the `tokenize` function 2. `in_range` and `after` which replaces the `lex_starts_at` function where the former returns the tokens within the given range while the latter returns all the tokens after the given offset 2. Introduce a new `TokenFlags` which is a set of flags to query certain information from a token. Currently, this information is only limited to any string type token but can be expanded to include other information in the future as needed. https://github.com/astral-sh/ruff/pull/11578 3. Move the `CommentRanges` to the parsed output because this information is common to both the linter and the formatter. This removes the need for `tokens_and_ranges` function. ## Test Plan - [x] Update and verify the test snapshots - [x] Make sure the entire test suite is passing - [x] Make sure there are no changes in the ecosystem checks - [x] Run the fuzzer on the parser - [x] Run this change on dozens of open-source projects ### Running this change on dozens of open-source projects Refer to the PR description to get the list of open source projects used for testing. Now, the following tests were done between `main` and this branch: 1. Compare the output of `--select=E999` (syntax errors) 2. Compare the output of default rule selection 3. Compare the output of `--select=ALL` **Conclusion: all output were same** ## What's next? The next step is to introduce re-lexing logic and update the parser to feed the recovery information to the lexer so that it can emit the correct token. This moves us one step closer to having error resilience in the parser and provides Ruff the possibility to lint even if the source code contains syntax errors.
433 lines
12 KiB
Rust
433 lines
12 KiB
Rust
use crate::comments::Comments;
|
|
use crate::other::f_string_element::FStringExpressionElementContext;
|
|
use crate::PyFormatOptions;
|
|
use ruff_formatter::{Buffer, FormatContext, GroupId, IndentWidth, SourceCode};
|
|
use ruff_python_ast::str::Quote;
|
|
use ruff_python_parser::Tokens;
|
|
use ruff_source_file::Locator;
|
|
use std::fmt::{Debug, Formatter};
|
|
use std::ops::{Deref, DerefMut};
|
|
|
|
#[derive(Clone)]
|
|
pub struct PyFormatContext<'a> {
|
|
options: PyFormatOptions,
|
|
contents: &'a str,
|
|
comments: Comments<'a>,
|
|
tokens: &'a Tokens,
|
|
node_level: NodeLevel,
|
|
indent_level: IndentLevel,
|
|
/// Set to a non-None value when the formatter is running on a code
|
|
/// snippet within a docstring. The value should be the quote character of the
|
|
/// docstring containing the code snippet.
|
|
///
|
|
/// Various parts of the formatter may inspect this state to change how it
|
|
/// works. For example, multi-line strings will always be written with a
|
|
/// quote style that is inverted from the one here in order to ensure that
|
|
/// the formatted Python code will be valid.
|
|
docstring: Option<Quote>,
|
|
/// The state of the formatter with respect to f-strings.
|
|
f_string_state: FStringState,
|
|
}
|
|
|
|
impl<'a> PyFormatContext<'a> {
|
|
pub(crate) fn new(
|
|
options: PyFormatOptions,
|
|
contents: &'a str,
|
|
comments: Comments<'a>,
|
|
tokens: &'a Tokens,
|
|
) -> Self {
|
|
Self {
|
|
options,
|
|
contents,
|
|
comments,
|
|
tokens,
|
|
node_level: NodeLevel::TopLevel(TopLevelStatementPosition::Other),
|
|
indent_level: IndentLevel::new(0),
|
|
docstring: None,
|
|
f_string_state: FStringState::Outside,
|
|
}
|
|
}
|
|
|
|
pub(crate) fn source(&self) -> &'a str {
|
|
self.contents
|
|
}
|
|
|
|
#[allow(unused)]
|
|
pub(crate) fn locator(&self) -> Locator<'a> {
|
|
Locator::new(self.contents)
|
|
}
|
|
|
|
pub(crate) fn set_node_level(&mut self, level: NodeLevel) {
|
|
self.node_level = level;
|
|
}
|
|
|
|
pub(crate) fn node_level(&self) -> NodeLevel {
|
|
self.node_level
|
|
}
|
|
|
|
pub(crate) fn set_indent_level(&mut self, level: IndentLevel) {
|
|
self.indent_level = level;
|
|
}
|
|
|
|
pub(crate) fn indent_level(&self) -> IndentLevel {
|
|
self.indent_level
|
|
}
|
|
|
|
pub(crate) fn comments(&self) -> &Comments<'a> {
|
|
&self.comments
|
|
}
|
|
|
|
pub(crate) fn tokens(&self) -> &'a Tokens {
|
|
self.tokens
|
|
}
|
|
|
|
/// Returns a non-None value only if the formatter is running on a code
|
|
/// snippet within a docstring.
|
|
///
|
|
/// The quote character returned corresponds to the quoting used for the
|
|
/// docstring containing the code snippet currently being formatted.
|
|
pub(crate) fn docstring(&self) -> Option<Quote> {
|
|
self.docstring
|
|
}
|
|
|
|
/// Return a new context suitable for formatting code snippets within a
|
|
/// docstring.
|
|
///
|
|
/// The quote character given should correspond to the quote character used
|
|
/// for the docstring containing the code snippets.
|
|
pub(crate) fn in_docstring(self, quote: Quote) -> PyFormatContext<'a> {
|
|
PyFormatContext {
|
|
docstring: Some(quote),
|
|
..self
|
|
}
|
|
}
|
|
|
|
pub(crate) fn f_string_state(&self) -> FStringState {
|
|
self.f_string_state
|
|
}
|
|
|
|
pub(crate) fn set_f_string_state(&mut self, f_string_state: FStringState) {
|
|
self.f_string_state = f_string_state;
|
|
}
|
|
|
|
/// Returns `true` if preview mode is enabled.
|
|
#[allow(unused)]
|
|
pub(crate) const fn is_preview(&self) -> bool {
|
|
self.options.preview().is_enabled()
|
|
}
|
|
}
|
|
|
|
impl FormatContext for PyFormatContext<'_> {
|
|
type Options = PyFormatOptions;
|
|
|
|
fn options(&self) -> &Self::Options {
|
|
&self.options
|
|
}
|
|
|
|
fn source_code(&self) -> SourceCode {
|
|
SourceCode::new(self.contents)
|
|
}
|
|
}
|
|
|
|
impl Debug for PyFormatContext<'_> {
|
|
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
|
|
f.debug_struct("PyFormatContext")
|
|
.field("options", &self.options)
|
|
.field("comments", &self.comments.debug(self.source_code()))
|
|
.field("node_level", &self.node_level)
|
|
.field("source", &self.contents)
|
|
.finish()
|
|
}
|
|
}
|
|
|
|
#[derive(Clone, Copy, Debug, Default)]
|
|
pub(crate) enum FStringState {
|
|
/// The formatter is inside an f-string expression element i.e., between the
|
|
/// curly brace in `f"foo {x}"`.
|
|
///
|
|
/// The containing `FStringContext` is the surrounding f-string context.
|
|
InsideExpressionElement(FStringExpressionElementContext),
|
|
/// The formatter is outside an f-string.
|
|
#[default]
|
|
Outside,
|
|
}
|
|
|
|
/// The position of a top-level statement in the module.
|
|
#[derive(Copy, Clone, Debug, Eq, PartialEq, Default)]
|
|
pub(crate) enum TopLevelStatementPosition {
|
|
/// This is the last top-level statement in the module.
|
|
Last,
|
|
/// Any other top-level statement.
|
|
#[default]
|
|
Other,
|
|
}
|
|
|
|
/// What's the enclosing level of the outer node.
|
|
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
|
|
pub(crate) enum NodeLevel {
|
|
/// Formatting statements on the module level.
|
|
TopLevel(TopLevelStatementPosition),
|
|
|
|
/// Formatting the body statements of a [compound statement](https://docs.python.org/3/reference/compound_stmts.html#compound-statements)
|
|
/// (`if`, `while`, `match`, etc.).
|
|
CompoundStatement,
|
|
|
|
/// The root or any sub-expression.
|
|
Expression(Option<GroupId>),
|
|
|
|
/// Formatting nodes that are enclosed by a parenthesized (any `[]`, `{}` or `()`) expression.
|
|
ParenthesizedExpression,
|
|
}
|
|
|
|
impl Default for NodeLevel {
|
|
fn default() -> Self {
|
|
Self::TopLevel(TopLevelStatementPosition::Other)
|
|
}
|
|
}
|
|
|
|
impl NodeLevel {
|
|
/// Returns `true` if the expression is in a parenthesized context.
|
|
pub(crate) const fn is_parenthesized(self) -> bool {
|
|
matches!(
|
|
self,
|
|
NodeLevel::Expression(Some(_)) | NodeLevel::ParenthesizedExpression
|
|
)
|
|
}
|
|
|
|
/// Returns `true` if this is the last top-level statement in the module.
|
|
pub(crate) const fn is_last_top_level_statement(self) -> bool {
|
|
matches!(self, NodeLevel::TopLevel(TopLevelStatementPosition::Last))
|
|
}
|
|
}
|
|
|
|
/// Change the [`NodeLevel`] of the formatter for the lifetime of this struct
|
|
pub(crate) struct WithNodeLevel<'ast, 'buf, B>
|
|
where
|
|
B: Buffer<Context = PyFormatContext<'ast>>,
|
|
{
|
|
buffer: &'buf mut B,
|
|
saved_level: NodeLevel,
|
|
}
|
|
|
|
impl<'ast, 'buf, B> WithNodeLevel<'ast, 'buf, B>
|
|
where
|
|
B: Buffer<Context = PyFormatContext<'ast>>,
|
|
{
|
|
pub(crate) fn new(level: NodeLevel, buffer: &'buf mut B) -> Self {
|
|
let context = buffer.state_mut().context_mut();
|
|
let saved_level = context.node_level();
|
|
|
|
context.set_node_level(level);
|
|
|
|
Self {
|
|
buffer,
|
|
saved_level,
|
|
}
|
|
}
|
|
}
|
|
|
|
impl<'ast, 'buf, B> Deref for WithNodeLevel<'ast, 'buf, B>
|
|
where
|
|
B: Buffer<Context = PyFormatContext<'ast>>,
|
|
{
|
|
type Target = B;
|
|
|
|
fn deref(&self) -> &Self::Target {
|
|
self.buffer
|
|
}
|
|
}
|
|
|
|
impl<'ast, 'buf, B> DerefMut for WithNodeLevel<'ast, 'buf, B>
|
|
where
|
|
B: Buffer<Context = PyFormatContext<'ast>>,
|
|
{
|
|
fn deref_mut(&mut self) -> &mut Self::Target {
|
|
self.buffer
|
|
}
|
|
}
|
|
|
|
impl<'ast, B> Drop for WithNodeLevel<'ast, '_, B>
|
|
where
|
|
B: Buffer<Context = PyFormatContext<'ast>>,
|
|
{
|
|
fn drop(&mut self) {
|
|
self.buffer
|
|
.state_mut()
|
|
.context_mut()
|
|
.set_node_level(self.saved_level);
|
|
}
|
|
}
|
|
|
|
/// The current indent level of the formatter.
|
|
///
|
|
/// One can determine the width of the indent itself (in number of ASCII
|
|
/// space characters) by multiplying the indent level by the configured indent
|
|
/// width.
|
|
///
|
|
/// This is specifically used inside the docstring code formatter for
|
|
/// implementing its "dynamic" line width mode. Namely, in the nested call to
|
|
/// the formatter, when "dynamic" mode is enabled, the line width is set to
|
|
/// `min(1, line_width - indent_level * indent_width)`, where `line_width` in
|
|
/// this context is the global line width setting.
|
|
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
|
|
pub(crate) struct IndentLevel {
|
|
/// The numeric level. It is incremented for every whole indent in Python
|
|
/// source code.
|
|
///
|
|
/// Note that the first indentation level is actually 1, since this starts
|
|
/// at 0 and is incremented when the first top-level statement is seen. So
|
|
/// even though the first top-level statement in Python source will have no
|
|
/// indentation, its indentation level is 1.
|
|
level: u16,
|
|
}
|
|
|
|
impl IndentLevel {
|
|
/// Returns a new indent level for the given value.
|
|
pub(crate) fn new(level: u16) -> IndentLevel {
|
|
IndentLevel { level }
|
|
}
|
|
|
|
/// Returns the next indent level.
|
|
pub(crate) fn increment(self) -> IndentLevel {
|
|
IndentLevel {
|
|
level: self.level.saturating_add(1),
|
|
}
|
|
}
|
|
|
|
/// Convert this indent level into a specific number of ASCII whitespace
|
|
/// characters based on the given indent width.
|
|
pub(crate) fn to_ascii_spaces(self, width: IndentWidth) -> u16 {
|
|
let width = u16::try_from(width.value()).unwrap_or(u16::MAX);
|
|
// Why the subtraction? IndentLevel starts at 0 and asks for the "next"
|
|
// indent level before seeing the first top-level statement. So it's
|
|
// always 1 more than what we expect it to be.
|
|
let level = self.level.saturating_sub(1);
|
|
width.saturating_mul(level)
|
|
}
|
|
}
|
|
|
|
/// Change the [`IndentLevel`] of the formatter for the lifetime of this
|
|
/// struct.
|
|
pub(crate) struct WithIndentLevel<'a, B, D>
|
|
where
|
|
D: DerefMut<Target = B>,
|
|
B: Buffer<Context = PyFormatContext<'a>>,
|
|
{
|
|
buffer: D,
|
|
saved_level: IndentLevel,
|
|
}
|
|
|
|
impl<'a, B, D> WithIndentLevel<'a, B, D>
|
|
where
|
|
D: DerefMut<Target = B>,
|
|
B: Buffer<Context = PyFormatContext<'a>>,
|
|
{
|
|
pub(crate) fn new(level: IndentLevel, mut buffer: D) -> Self {
|
|
let context = buffer.state_mut().context_mut();
|
|
let saved_level = context.indent_level();
|
|
|
|
context.set_indent_level(level);
|
|
|
|
Self {
|
|
buffer,
|
|
saved_level,
|
|
}
|
|
}
|
|
}
|
|
|
|
impl<'a, B, D> Deref for WithIndentLevel<'a, B, D>
|
|
where
|
|
D: DerefMut<Target = B>,
|
|
B: Buffer<Context = PyFormatContext<'a>>,
|
|
{
|
|
type Target = B;
|
|
|
|
fn deref(&self) -> &Self::Target {
|
|
&self.buffer
|
|
}
|
|
}
|
|
|
|
impl<'a, B, D> DerefMut for WithIndentLevel<'a, B, D>
|
|
where
|
|
D: DerefMut<Target = B>,
|
|
B: Buffer<Context = PyFormatContext<'a>>,
|
|
{
|
|
fn deref_mut(&mut self) -> &mut Self::Target {
|
|
&mut self.buffer
|
|
}
|
|
}
|
|
|
|
impl<'a, B, D> Drop for WithIndentLevel<'a, B, D>
|
|
where
|
|
D: DerefMut<Target = B>,
|
|
B: Buffer<Context = PyFormatContext<'a>>,
|
|
{
|
|
fn drop(&mut self) {
|
|
self.buffer
|
|
.state_mut()
|
|
.context_mut()
|
|
.set_indent_level(self.saved_level);
|
|
}
|
|
}
|
|
|
|
pub(crate) struct WithFStringState<'a, B, D>
|
|
where
|
|
D: DerefMut<Target = B>,
|
|
B: Buffer<Context = PyFormatContext<'a>>,
|
|
{
|
|
buffer: D,
|
|
saved_location: FStringState,
|
|
}
|
|
|
|
impl<'a, B, D> WithFStringState<'a, B, D>
|
|
where
|
|
D: DerefMut<Target = B>,
|
|
B: Buffer<Context = PyFormatContext<'a>>,
|
|
{
|
|
pub(crate) fn new(expr_location: FStringState, mut buffer: D) -> Self {
|
|
let context = buffer.state_mut().context_mut();
|
|
let saved_location = context.f_string_state();
|
|
|
|
context.set_f_string_state(expr_location);
|
|
|
|
Self {
|
|
buffer,
|
|
saved_location,
|
|
}
|
|
}
|
|
}
|
|
|
|
impl<'a, B, D> Deref for WithFStringState<'a, B, D>
|
|
where
|
|
D: DerefMut<Target = B>,
|
|
B: Buffer<Context = PyFormatContext<'a>>,
|
|
{
|
|
type Target = B;
|
|
|
|
fn deref(&self) -> &Self::Target {
|
|
&self.buffer
|
|
}
|
|
}
|
|
|
|
impl<'a, B, D> DerefMut for WithFStringState<'a, B, D>
|
|
where
|
|
D: DerefMut<Target = B>,
|
|
B: Buffer<Context = PyFormatContext<'a>>,
|
|
{
|
|
fn deref_mut(&mut self) -> &mut Self::Target {
|
|
&mut self.buffer
|
|
}
|
|
}
|
|
|
|
impl<'a, B, D> Drop for WithFStringState<'a, B, D>
|
|
where
|
|
D: DerefMut<Target = B>,
|
|
B: Buffer<Context = PyFormatContext<'a>>,
|
|
{
|
|
fn drop(&mut self) {
|
|
self.buffer
|
|
.state_mut()
|
|
.context_mut()
|
|
.set_f_string_state(self.saved_location);
|
|
}
|
|
}
|