language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-09-28 04:45:01 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Use CommentRanges in backwards lexing (#7360)

Browse source 
## Summary

The tokenizer was split into a forward and a backwards tokenizer. The
backwards tokenizer uses the same names as the forwards ones (e.g.
`next_token`). The backwards tokenizer gets the comment ranges that we
already built to skip comments.

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
This commit is contained in:
konsti 2023-09-16 05:21:45 +02:00 • committed by GitHub
parent 1f6e1485f9
commit 2cbe1733c8
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
41 changed files with 744 additions and 628 deletions
Download patch file Download diff file Expand all files Collapse all files

2
crates/ruff_python_trivia/src/cursor.rs
View file

@ -44,7 +44,7 @@ impl<'a> Cursor<'a> {
self.chars.clone().next_back().unwrap_or(EOF_CHAR)
}
// SAFETY: THe `source.text_len` call in `new` would panic if the string length is larger than a `u32`.
// SAFETY: The `source.text_len` call in `new` would panic if the string length is larger than a `u32`.
#[allow(clippy::cast_possible_truncation)]
pub fn text_len(&self) -> TextSize {
TextSize::new(self.chars.as_str().len() as u32)