language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-08-28 14:25:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Treat form feed as whitespace in SimpleTokenizer (#7626)

Browse source 
## Summary

This is whitespace as per `is_python_whitespace`, and right now it tends
to lead to panics in the formatter. Seems reasonable to treat it as
whitespace in the `SimpleTokenizer` too.

Closes .https://github.com/astral-sh/ruff/issues/7624.
This commit is contained in:
Charlie Marsh 2023-09-25 10:34:59 -04:00 • committed by GitHub
parent 17ceb5dcb3
commit 65aebf127a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3 changed files with 43 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

17
crates/ruff_python_trivia/src/tokenizer.rs
View file

@ -566,8 +566,10 @@ impl<'a> SimpleTokenizer<'a> {
kind
}
' ' | '\t' => {
self.cursor.eat_while(|c| matches!(c, ' ' | '\t'));
// Space, tab, or form feed. We ignore the true semantics of form feed, and treat it as
// whitespace.
' ' | '\t' | '\x0C' => {
self.cursor.eat_while(|c| matches!(c, ' ' | '\t' | '\x0C'));
SimpleTokenKind::Whitespace
}
@ -837,10 +839,13 @@ impl<'a> BackwardsTokenizer<'a> {
}
let kind = match last {
// This may not be 100% correct because it will lex-out trailing whitespace from a comment
// as whitespace rather than being part of the token. This shouldn't matter for what we use the lexer for.
' ' | '\t' => {
self.cursor.eat_back_while(|c| matches!(c, ' ' | '\t'));
// Space, tab, or form feed. We ignore the true semantics of form feed, and treat it as
// whitespace. Note that this will lex-out trailing whitespace from a comment as
// whitespace rather than as part of the comment token, but this shouldn't matter for
// our use case.
' ' | '\t' | '\x0C' => {
self.cursor
.eat_back_while(|c| matches!(c, ' ' | '\t' | '\x0C'));
SimpleTokenKind::Whitespace
}