language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-07-24 21:44:37 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Add support for multi-character operator tokens to SimpleTokenizer (#6563)

Browse source 
## Summary

Allows for proper lexing of tokens like `->`.

The main challenge is to ensure that our forward and backwards
representations are the same for cases like `===`. Specifically, we want
that to lex as `==` followed by `=` regardless of whether it's a
forwards or backwards lex. To do so, we identify the range of the
sequential characters (the full span of `===`), lex it forwards, then
return the last token.

## Test Plan

`cargo test`
This commit is contained in:
Charlie Marsh 2023-08-16 09:09:19 -04:00 • committed by GitHub
parent e28858bb29
commit 86ccdcc9d9
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
8 changed files with 538 additions and 149 deletions
Download patch file Download diff file Expand all files Collapse all files

36
crates/ruff_python_formatter/src/comments/placement.rs
View file

@ -951,39 +951,11 @@ fn handle_dict_unpacking_comment<'a>(
// if the remaining tokens from the previous node are exactly `**`,
// re-assign the comment to the one that follows the stars
let mut count = 0u32;
// we start from the preceding node but we skip its token
if let Some(token) = tokens.next() {
// The Keyword case
if token.kind == SimpleTokenKind::Star {
count += 1;
} else {
// The dict case
debug_assert!(
matches!(
token,
SimpleToken {
kind: SimpleTokenKind::LBrace
| SimpleTokenKind::Comma
| SimpleTokenKind::Colon,
..
}
),
"{token:?}",
);
}
if tokens.any(|token| token.kind == SimpleTokenKind::DoubleStar) {
CommentPlacement::trailing(following, comment)
} else {
CommentPlacement::Default(comment)
}
for token in tokens {
debug_assert!(token.kind == SimpleTokenKind::Star, "Expected star token");
count += 1;
}
if count == 2 {
return CommentPlacement::trailing(following, comment);
}
CommentPlacement::Default(comment)
}
/// Own line comments coming after the node are always dangling comments