mirrors/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-09-29 13:24:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Add Tokens newtype wrapper, TokenKind iterator (#11361)

Browse source 
## Summary

Alternative to #11237 

This PR adds a new `Tokens` struct which is a newtype wrapper around a
vector of lexer output. This allows us to add a `kinds` method which
returns an iterator over the corresponding `TokenKind`. This iterator is
implemented as a separate `TokenKindIter` struct to allow using the type
and provide additional methods like `peek` directly on the iterator.

This exposes the linter to access the stream of `TokenKind` instead of
`Tok`.

Edit: I've made the necessary downstream changes and plan to merge the
entire stack at once.
This commit is contained in:
Dhruv Manilawala 2024-05-14 22:15:04 +05:30 • committed by GitHub
parent 50f14d017e
commit 025768d303
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 142 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

6
crates/ruff_linter/src/test.rs
View file

@ -16,7 +16,6 @@ use ruff_notebook::NotebookError;
use ruff_python_ast::PySourceType;
use ruff_python_codegen::Stylist;
use ruff_python_index::Indexer;
use ruff_python_parser::lexer::LexResult;
use ruff_python_parser::AsMode;
use ruff_python_trivia::textwrap::dedent;
use ruff_source_file::{Locator, SourceFileBuilder};
@ -111,8 +110,7 @@ pub(crate) fn test_contents<'a>(
settings: &LinterSettings,
) -> (Vec<Message>, Cow<'a, SourceKind>) {
let source_type = PySourceType::from(path);
let tokens: Vec<LexResult> =
ruff_python_parser::tokenize(source_kind.source_code(), source_type.as_mode());
let tokens = ruff_python_parser::tokenize(source_kind.source_code(), source_type.as_mode());
let locator = Locator::new(source_kind.source_code());
let stylist = Stylist::from_tokens(&tokens, &locator);
let indexer = Indexer::from_tokens(&tokens, &locator);
@ -177,7 +175,7 @@ pub(crate) fn test_contents<'a>(
transformed = Cow::Owned(transformed.updated(fixed_contents, &source_map));
let tokens: Vec<LexResult> =
let tokens =
ruff_python_parser::tokenize(transformed.source_code(), source_type.as_mode());
let locator = Locator::new(transformed.source_code());
let stylist = Stylist::from_tokens(&tokens, &locator);