language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-08-16 08:30:30 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Apply NFKC normalization to unicode identifiers in the lexer (#10412)

Browse source
This commit is contained in:
Alex Waygood 2024-03-18 11:56:56 +00:00 • committed by GitHub
parent bb540718c2
commit 92e6026446
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 68 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

3
crates/ruff_python_parser/src/token.rs
View file

@ -16,6 +16,9 @@ pub enum Tok {
/// Token value for a name, commonly known as an identifier.
Name {
/// The name value.
///
/// Unicode names are NFKC-normalized by the lexer,
/// matching [the behaviour of Python's lexer](https://docs.python.org/3/reference/lexical_analysis.html#identifiers)
name: Box<str>,
},
/// Token value for an integer.