mirror of
https://github.com/astral-sh/ruff.git
synced 2025-09-27 12:29:28 +00:00
bf5b62edac
## Summary This PR updates the entire parser stack in multiple ways: ### Make the lexer lazy * https://github.com/astral-sh/ruff/pull/11244 * https://github.com/astral-sh/ruff/pull/11473 Previously, Ruff's lexer would act as an iterator. The parser would collect all the tokens in a vector first and then process the tokens to create the syntax tree. The first task in this project is to update the entire parsing flow to make the lexer lazy. This includes the `Lexer`, `TokenSource`, and `Parser`. For context, the `TokenSource` is a wrapper around the `Lexer` to filter out the trivia tokens[^1]. Now, the parser will ask the token source to get the next token and only then the lexer will continue and emit the token. This means that the lexer needs to be aware of the "current" token. When the `next_token` is called, the current token will be updated with the newly lexed token. The main motivation to make the lexer lazy is to allow re-lexing a token in a different context. This is going to be really useful to make the parser error resilience. For example, currently the emitted tokens remains the same even if the parser can recover from an unclosed parenthesis. This is important because the lexer emits a `NonLogicalNewline` in parenthesized context while a normal `Newline` in non-parenthesized context. This different kinds of newline is also used to emit the indentation tokens which is important for the parser as it's used to determine the start and end of a block. Additionally, this allows us to implement the following functionalities: 1. Checkpoint - rewind infrastructure: The idea here is to create a checkpoint and continue lexing. At a later point, this checkpoint can be used to rewind the lexer back to the provided checkpoint. 2. Remove the `SoftKeywordTransformer` and instead use lookahead or speculative parsing to determine whether a soft keyword is a keyword or an identifier 3. Remove the `Tok` enum. The `Tok` enum represents the tokens emitted by the lexer but it contains owned data which makes it expensive to clone. The new `TokenKind` enum just represents the type of token which is very cheap. This brings up a question as to how will the parser get the owned value which was stored on `Tok`. This will be solved by introducing a new `TokenValue` enum which only contains a subset of token kinds which has the owned value. This is stored on the lexer and is requested by the parser when it wants to process the data. For example: |
||
|---|---|---|
| .. | ||
| assets | ||
| docs | ||
| resources/test/fixtures | ||
| src | ||
| tests | ||
| Cargo.toml | ||
| CONTRIBUTING.md | ||
| README.md | ||
The Ruff Language Server
Welcome!
ruff server is a language server that powers Ruff's editor integrations.
The job of the language server is to listen for requests from the client (in this case, the code editor of your choice) and call into Ruff's linter and formatter crates to construct real-time diagnostics or formatted code, which is then sent back to the client. It also tracks configuration files in your editor's workspace, and will refresh its in-memory configuration whenever those files are modified.
Setup
We have specific setup instructions depending on your editor. If you don't see your editor on this list and would like a setup guide, please open an issue.
If you're transferring your configuration from ruff-lsp, regardless of
editor, there are several settings which have changed or are no longer available. See the migration guide for
more.
VS Code
Install the Ruff extension from the VS Code Marketplace.
As this server is still in Beta, you will need to enable the "Native Server" extension setting, either in the settings UI:
Or in your settings.json:
{
"ruff.nativeServer": true
}
From there, you can configure Ruff to format Python code on-save with:
{
"[python]": {
"editor.formatOnSave": true,
"editor.defaultFormatter": "charliermarsh.ruff"
}
}
For more, see Configuring VS Code in the Ruff extension documentation.
By default, the extension will run against the ruff binary that it discovers in your environment. If you don't have
ruff installed, the extension will fall back to a bundled version of the binary.
Neovim
See the Neovim setup guide.
Helix
See the Helix setup guide.
Vim
See the Vim setup guide.
Kate
See the Kate setup guide.
Contributing
If you're interested in contributing to ruff server - well, first of all, thank you! Second of all, you might find the
contribution guide to be a useful resource.
Finally, don't hesitate to reach out on Discord if you have questions.