language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-09-27 12:29:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Always use identifier ranges to store bindings (#5110)

Browse source 
## Summary

At present, when we store a binding, we include a `TextRange` alongside
it. The `TextRange` _sometimes_ matches the exact range of the
identifier to which the `Binding` is linked, but... not always.

For example, given:

```python
x = 1
```

The binding we create _will_ use the range of `x`, because the left-hand
side is an `Expr::Name`, which has a valid range on it.

However, given:

```python
try:
  pass
except ValueError as e:
  pass
```

When we create a binding for `e`, we don't have a `TextRange`... The AST
doesn't give us one. So we end up extracting it via lexing.

This PR extends that pattern to the rest of the binding kinds, to ensure
that whenever we create a binding, we always use the range of the bound
name. This leads to better diagnostics in cases like pattern matching,
whereby the diagnostic for "unused variable `x`" here used to include
`*x`, instead of just `x`:

```python
def f(provided: int) -> int:
    match provided:
        case [_, *x]:
            pass
```

This is _also_ required for symbol renames, since we track writes as
bindings -- so we need to know the ranges of the bound symbols.

By storing these bindings precisely, we can also remove the
`binding.trimmed_range` abstraction -- since bindings already use the
"trimmed range".

To implement this behavior, I took some of our existing utilities (like
the code we had for `except ValueError as e` above), migrated them from
a full lexer to a zero-allocation lexer that _only_ identifies
"identifiers", and moved the behavior into a trait, so we can now do
`stmt.identifier(locator)` to get the range for the identifier.

Honestly, we might end up discarding much of this if we decide to put
ranges on all identifiers
(https://github.com/astral-sh/RustPython-Parser/pull/8). But even if we
do, this will _still_ be a good change, because the lexer introduced
here is useful beyond names (e.g., we use it find the `except` keyword
in an exception handler, to find the `else` after a `for` loop, and so
on). So, I'm fine committing this even if we end up changing our minds
about the right approach.

Closes #5090.

## Benchmarks

No significant change, with one statistically significant improvement
(-2.1654% on `linter/all-rules/large/dataset.py`):

```
linter/default-rules/numpy/globals.py
                        time:   [73.922 µs 73.955 µs 73.986 µs]
                        thrpt:  [39.882 MiB/s 39.898 MiB/s 39.916 MiB/s]
                 change:
                        time:   [-0.5579% -0.4732% -0.3980%] (p = 0.00 < 0.05)
                        thrpt:  [+0.3996% +0.4755% +0.5611%]
                        Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
  4 (4.00%) low severe
  1 (1.00%) low mild
  1 (1.00%) high mild
linter/default-rules/pydantic/types.py
                        time:   [1.4909 ms 1.4917 ms 1.4926 ms]
                        thrpt:  [17.087 MiB/s 17.096 MiB/s 17.106 MiB/s]
                 change:
                        time:   [+0.2140% +0.2741% +0.3392%] (p = 0.00 < 0.05)
                        thrpt:  [-0.3380% -0.2734% -0.2136%]
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
linter/default-rules/numpy/ctypeslib.py
                        time:   [688.97 µs 691.34 µs 694.15 µs]
                        thrpt:  [23.988 MiB/s 24.085 MiB/s 24.168 MiB/s]
                 change:
                        time:   [-1.3282% -0.7298% -0.1466%] (p = 0.02 < 0.05)
                        thrpt:  [+0.1468% +0.7351% +1.3461%]
                        Change within noise threshold.
Found 15 outliers among 100 measurements (15.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  12 (12.00%) high severe
linter/default-rules/large/dataset.py
                        time:   [3.3872 ms 3.4032 ms 3.4191 ms]
                        thrpt:  [11.899 MiB/s 11.954 MiB/s 12.011 MiB/s]
                 change:
                        time:   [-0.6427% -0.2635% +0.0906%] (p = 0.17 > 0.05)
                        thrpt:  [-0.0905% +0.2642% +0.6469%]
                        No change in performance detected.
Found 20 outliers among 100 measurements (20.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
  4 (4.00%) high mild
  13 (13.00%) high severe

linter/all-rules/numpy/globals.py
                        time:   [148.99 µs 149.21 µs 149.42 µs]
                        thrpt:  [19.748 MiB/s 19.776 MiB/s 19.805 MiB/s]
                 change:
                        time:   [-0.7340% -0.5068% -0.2778%] (p = 0.00 < 0.05)
                        thrpt:  [+0.2785% +0.5094% +0.7395%]
                        Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high severe
linter/all-rules/pydantic/types.py
                        time:   [3.0362 ms 3.0396 ms 3.0441 ms]
                        thrpt:  [8.3779 MiB/s 8.3903 MiB/s 8.3997 MiB/s]
                 change:
                        time:   [-0.0957% +0.0618% +0.2125%] (p = 0.45 > 0.05)
                        thrpt:  [-0.2121% -0.0618% +0.0958%]
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low severe
  3 (3.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe
linter/all-rules/numpy/ctypeslib.py
                        time:   [1.6879 ms 1.6894 ms 1.6909 ms]
                        thrpt:  [9.8478 MiB/s 9.8562 MiB/s 9.8652 MiB/s]
                 change:
                        time:   [-0.2279% -0.0888% +0.0436%] (p = 0.18 > 0.05)
                        thrpt:  [-0.0435% +0.0889% +0.2284%]
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) low mild
  1 (1.00%) high severe
linter/all-rules/large/dataset.py
                        time:   [7.1520 ms 7.1586 ms 7.1654 ms]
                        thrpt:  [5.6777 MiB/s 5.6831 MiB/s 5.6883 MiB/s]
                 change:
                        time:   [-2.5626% -2.1654% -1.7780%] (p = 0.00 < 0.05)
                        thrpt:  [+1.8102% +2.2133% +2.6300%]
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
```
This commit is contained in:
Charlie Marsh 2023-06-15 14:43:19 -04:00 • committed by GitHub
parent 66089e1a2e
commit 5ea3e42513
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
58 changed files with 1001 additions and 576 deletions
Download patch file Download diff file Expand all files Collapse all files

621
crates/ruff_python_ast/src/identifier.rs Normal file
View file

@ -0,0 +1,621 @@
//! Extract [`TextRange`] information from AST nodes.
//!
//! In the `RustPython` AST, each node has a `range` field that contains the
//! start and end byte offsets of the node. However, attributes on those
//! nodes may not have their own ranges. In particular, identifiers are
//! not given their own ranges, unless they're part of a name expression.
//!
//! For example, given:
//! ```python
//! def f():
//! ...
//! ```
//!
//! The statement defining `f` has a range, but the identifier `f` does not.
//!
//! This module assists with extracting [`TextRange`] ranges from AST nodes
//! via manual lexical analysis.
use std::ops::{Add, Sub};
use std::str::Chars;
use ruff_text_size::{TextLen, TextRange, TextSize};
use rustpython_ast::{Alias, Arg, Pattern};
use rustpython_parser::ast::{self, Excepthandler, Ranged, Stmt};
use ruff_python_whitespace::is_python_whitespace;
use crate::source_code::Locator;
pub trait Identifier {
/// Return the [`TextRange`] of the identifier in the given AST node.
fn identifier(&self, locator: &Locator) -> TextRange;
}
pub trait TryIdentifier {
/// Return the [`TextRange`] of the identifier in the given AST node, or `None` if
/// the node does not have an identifier.
fn try_identifier(&self, locator: &Locator) -> Option<TextRange>;
}
impl Identifier for Stmt {
/// Return the [`TextRange`] of the identifier in the given statement.
///
/// For example, return the range of `f` in:
/// ```python
/// def f():
/// ...
/// ```
fn identifier(&self, locator: &Locator) -> TextRange {
match self {
Stmt::ClassDef(ast::StmtClassDef {
decorator_list,
range,
..
})
| Stmt::FunctionDef(ast::StmtFunctionDef {
decorator_list,
range,
..
}) => {
let range = decorator_list.last().map_or(*range, |last_decorator| {
TextRange::new(last_decorator.end(), range.end())
});
// The first "identifier" is the `def` or `class` keyword.
// The second "identifier" is the function or class name.
IdentifierTokenizer::starts_at(range.start(), locator.contents())
.nth(1)
.expect("Unable to identify identifier in function or class definition")
}
Stmt::AsyncFunctionDef(ast::StmtAsyncFunctionDef {
decorator_list,
range,
..
}) => {
let range = decorator_list.last().map_or(*range, |last_decorator| {
TextRange::new(last_decorator.end(), range.end())
});
// The first "identifier" is the `async` keyword.
// The second "identifier" is the `def` or `class` keyword.
// The third "identifier" is the function or class name.
IdentifierTokenizer::starts_at(range.start(), locator.contents())
.nth(2)
.expect("Unable to identify identifier in function or class definition")
}
_ => self.range(),
}
}
}
impl Identifier for Arg {
/// Return the [`TextRange`] for the identifier defining an [`Arg`].
///
/// For example, return the range of `x` in:
/// ```python
/// def f(x: int = 0):
/// ...
/// ```
fn identifier(&self, locator: &Locator) -> TextRange {
IdentifierTokenizer::new(locator.contents(), self.range())
.next()
.expect("Failed to find argument identifier")
}
}
impl Identifier for Alias {
/// Return the [`TextRange`] for the identifier defining an [`Alias`].
///
/// For example, return the range of `x` in:
/// ```python
/// from foo import bar as x
/// ```
fn identifier(&self, locator: &Locator) -> TextRange {
if matches!(self.name.as_str(), "*") {
self.range()
} else if self.asname.is_none() {
// The first identifier is the module name.
IdentifierTokenizer::new(locator.contents(), self.range())
.next()
.expect("Failed to find alias identifier")
} else {
// The first identifier is the module name.
// The second identifier is the "as" keyword.
// The third identifier is the alias name.
IdentifierTokenizer::new(locator.contents(), self.range())
.last()
.expect("Failed to find alias identifier")
}
}
}
impl TryIdentifier for Pattern {
/// Return the [`TextRange`] of the identifier in the given pattern.
///
/// For example, return the range of `z` in:
/// ```python
/// match x:
/// # Pattern::MatchAs
/// case z:
/// ...
/// ```
///
/// Or:
/// ```python
/// match x:
/// # Pattern::MatchAs
/// case y as z:
/// ...
/// ```
///
/// Or :
/// ```python
/// match x:
/// # Pattern::MatchMapping
/// case {"a": 1, **z}
/// ...
/// ```
///
/// Or :
/// ```python
/// match x:
/// # Pattern::MatchStar
/// case *z:
/// ...
/// ```
fn try_identifier(&self, locator: &Locator) -> Option<TextRange> {
match self {
Pattern::MatchAs(ast::PatternMatchAs {
name: Some(_),
pattern,
range,
}) => {
Some(if let Some(pattern) = pattern {
// Identify `z` in:
// ```python
// match x:
// case Foo(bar) as z:
// ...
// ```
IdentifierTokenizer::starts_at(pattern.end(), locator.contents())
.nth(1)
.expect("Unable to identify identifier in pattern")
} else {
// Identify `z` in:
// ```python
// match x:
// case z:
// ...
// ```
*range
})
}
Pattern::MatchMapping(ast::PatternMatchMapping {
patterns,
rest: Some(_),
..
}) => {
Some(if let Some(pattern) = patterns.last() {
// Identify `z` in:
// ```python
// match x:
// case {"a": 1, **z}
// ...
// ```
//
// A mapping pattern can contain at most one double-star pattern,
// and it must be the last pattern in the mapping.
IdentifierTokenizer::starts_at(pattern.end(), locator.contents())
.next()
.expect("Unable to identify identifier in pattern")
} else {
// Identify `z` in:
// ```python
// match x:
// case {**z}
// ...
// ```
IdentifierTokenizer::starts_at(self.start(), locator.contents())
.next()
.expect("Unable to identify identifier in pattern")
})
}
Pattern::MatchStar(ast::PatternMatchStar { name: Some(_), .. }) => {
// Identify `z` in:
// ```python
// match x:
// case *z:
// ...
// ```
Some(
IdentifierTokenizer::starts_at(self.start(), locator.contents())
.next()
.expect("Unable to identify identifier in pattern"),
)
}
_ => None,
}
}
}
impl TryIdentifier for Excepthandler {
/// Return the [`TextRange`] of a named exception in an [`Excepthandler`].
///
/// For example, return the range of `e` in:
/// ```python
/// try:
/// ...
/// except ValueError as e:
/// ...
/// ```
fn try_identifier(&self, locator: &Locator) -> Option<TextRange> {
let Excepthandler::ExceptHandler(ast::ExcepthandlerExceptHandler { type_, name, .. }) =
self;
if name.is_none() {
return None;
}
let Some(type_) = type_ else {
return None;
};
// The exception name is the first identifier token after the `as` keyword.
Some(
IdentifierTokenizer::starts_at(type_.end(), locator.contents())
.nth(1)
.expect("Failed to find exception identifier in exception handler"),
)
}
}
/// Return the [`TextRange`] for every name in a [`Stmt`].
///
/// Intended to be used for `global` and `nonlocal` statements.
///
/// For example, return the ranges of `x` and `y` in:
/// ```python
/// global x, y
/// ```
pub fn names<'a>(stmt: &Stmt, locator: &'a Locator<'a>) -> impl Iterator<Item = TextRange> + 'a {
// Given `global x, y`, the first identifier is `global`, and the remaining identifiers are
// the names.
IdentifierTokenizer::new(locator.contents(), stmt.range()).skip(1)
}
/// Return the [`TextRange`] of the `except` token in an [`Excepthandler`].
pub fn except(handler: &Excepthandler, locator: &Locator) -> TextRange {
IdentifierTokenizer::new(locator.contents(), handler.range())
.next()
.expect("Failed to find `except` token in `Excepthandler`")
}
/// Return the [`TextRange`] of the `else` token in a `For`, `AsyncFor`, or `While` statement.
pub fn else_(stmt: &Stmt, locator: &Locator) -> Option<TextRange> {
let (Stmt::For(ast::StmtFor { body, orelse, .. })
| Stmt::AsyncFor(ast::StmtAsyncFor { body, orelse, .. })
| Stmt::While(ast::StmtWhile { body, orelse, .. })) = stmt else {
return None;
};
if orelse.is_empty() {
return None;
}
IdentifierTokenizer::starts_at(
body.last().expect("Expected body to be non-empty").end(),
locator.contents(),
)
.next()
}
/// Return `true` if the given character starts a valid Python identifier.
///
/// Python identifiers must start with an alphabetic character or an underscore.
fn is_python_identifier_start(c: char) -> bool {
c.is_alphabetic() || c == '_'
}
/// Return `true` if the given character is a valid Python identifier continuation character.
///
/// Python identifiers can contain alphanumeric characters and underscores, but cannot start with a
/// number.
fn is_python_identifier_continue(c: char) -> bool {
c.is_alphanumeric() || c == '_'
}
/// Simple zero allocation tokenizer for Python identifiers.
///
/// The tokenizer must operate over a range that can only contain identifiers, keywords, and
/// comments (along with whitespace and continuation characters). It does not support other tokens,
/// like operators, literals, or delimiters. It also does not differentiate between keywords and
/// identifiers, treating every valid token as an "identifier".
///
/// This is useful for cases like, e.g., identifying the alias name in an aliased import (`bar` in
/// `import foo as bar`), where we're guaranteed to only have identifiers and keywords in the
/// relevant range.
pub(crate) struct IdentifierTokenizer<'a> {
cursor: Cursor<'a>,
offset: TextSize,
}
impl<'a> IdentifierTokenizer<'a> {
pub(crate) fn new(source: &'a str, range: TextRange) -> Self {
Self {
cursor: Cursor::new(&source[range]),
offset: range.start(),
}
}
pub(crate) fn starts_at(offset: TextSize, source: &'a str) -> Self {
let range = TextRange::new(offset, source.text_len());
Self::new(source, range)
}
fn next_token(&mut self) -> Option<TextRange> {
while let Some(c) = self.cursor.bump() {
match c {
c if is_python_identifier_start(c) => {
let start = self.offset.add(self.cursor.offset()).sub(c.text_len());
self.cursor.eat_while(is_python_identifier_continue);
let end = self.offset.add(self.cursor.offset());
return Some(TextRange::new(start, end));
}
c if is_python_whitespace(c) => {
self.cursor.eat_while(is_python_whitespace);
}
'#' => {
self.cursor.eat_while(|c| !matches!(c, '\n' | '\r'));
}
'\r' => {
self.cursor.eat_char('\n');
}
'\n' => {
// Nothing to do.
}
'\\' => {
// Nothing to do.
}
_ => {
// Nothing to do.
}
};
}
None
}
}
impl Iterator for IdentifierTokenizer<'_> {
type Item = TextRange;
fn next(&mut self) -> Option<Self::Item> {
self.next_token()
}
}
const EOF_CHAR: char = '\0';
#[derive(Debug, Clone)]
struct Cursor<'a> {
chars: Chars<'a>,
offset: TextSize,
}
impl<'a> Cursor<'a> {
fn new(source: &'a str) -> Self {
Self {
chars: source.chars(),
offset: TextSize::from(0),
}
}
const fn offset(&self) -> TextSize {
self.offset
}
/// Peeks the next character from the input stream without consuming it.
/// Returns [`EOF_CHAR`] if the file is at the end of the file.
fn first(&self) -> char {
self.chars.clone().next().unwrap_or(EOF_CHAR)
}
/// Returns `true` if the file is at the end of the file.
fn is_eof(&self) -> bool {
self.chars.as_str().is_empty()
}
/// Consumes the next character.
fn bump(&mut self) -> Option<char> {
if let Some(char) = self.chars.next() {
self.offset += char.text_len();
Some(char)
} else {
None
}
}
/// Eats the next character if it matches the given character.
fn eat_char(&mut self, c: char) -> bool {
if self.first() == c {
self.bump();
true
} else {
false
}
}
/// Eats symbols while predicate returns true or until the end of file is reached.
fn eat_while(&mut self, mut predicate: impl FnMut(char) -> bool) {
while predicate(self.first()) && !self.is_eof() {
self.bump();
}
}
}
#[cfg(test)]
mod tests {
use anyhow::Result;
use ruff_text_size::{TextRange, TextSize};
use rustpython_ast::Stmt;
use rustpython_parser::Parse;
use crate::identifier;
use crate::identifier::Identifier;
use crate::source_code::Locator;
#[test]
fn extract_arg_range() -> Result<()> {
let contents = "def f(x): pass".trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let function_def = stmt.as_function_def_stmt().unwrap();
let args = &function_def.args.args;
let arg = &args[0];
let locator = Locator::new(contents);
assert_eq!(
arg.identifier(&locator),
TextRange::new(TextSize::from(6), TextSize::from(7))
);
let contents = "def f(x: int): pass".trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let function_def = stmt.as_function_def_stmt().unwrap();
let args = &function_def.args.args;
let arg = &args[0];
let locator = Locator::new(contents);
assert_eq!(
arg.identifier(&locator),
TextRange::new(TextSize::from(6), TextSize::from(7))
);
let contents = r#"
def f(
x: int, # Comment
):
pass
"#
.trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let function_def = stmt.as_function_def_stmt().unwrap();
let args = &function_def.args.args;
let arg = &args[0];
let locator = Locator::new(contents);
assert_eq!(
arg.identifier(&locator),
TextRange::new(TextSize::from(11), TextSize::from(12))
);
Ok(())
}
#[test]
fn extract_identifier_range() -> Result<()> {
let contents = "def f(): pass".trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let locator = Locator::new(contents);
assert_eq!(
stmt.identifier(&locator),
TextRange::new(TextSize::from(4), TextSize::from(5))
);
let contents = "async def f(): pass".trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let locator = Locator::new(contents);
assert_eq!(
stmt.identifier(&locator),
TextRange::new(TextSize::from(10), TextSize::from(11))
);
let contents = r#"
def \
f():
pass
"#
.trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let locator = Locator::new(contents);
assert_eq!(
stmt.identifier(&locator),
TextRange::new(TextSize::from(8), TextSize::from(9))
);
let contents = "class Class(): pass".trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let locator = Locator::new(contents);
assert_eq!(
stmt.identifier(&locator),
TextRange::new(TextSize::from(6), TextSize::from(11))
);
let contents = "class Class: pass".trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let locator = Locator::new(contents);
assert_eq!(
stmt.identifier(&locator),
TextRange::new(TextSize::from(6), TextSize::from(11))
);
let contents = r#"
@decorator()
class Class():
pass
"#
.trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let locator = Locator::new(contents);
assert_eq!(
stmt.identifier(&locator),
TextRange::new(TextSize::from(19), TextSize::from(24))
);
let contents = r#"
@decorator() # Comment
class Class():
pass
"#
.trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let locator = Locator::new(contents);
assert_eq!(
stmt.identifier(&locator),
TextRange::new(TextSize::from(30), TextSize::from(35))
);
let contents = r#"x = y + 1"#.trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let locator = Locator::new(contents);
assert_eq!(
stmt.identifier(&locator),
TextRange::new(TextSize::from(0), TextSize::from(9))
);
Ok(())
}
#[test]
fn extract_else_range() -> Result<()> {
let contents = r#"
for x in y:
pass
else:
pass
"#
.trim();
let stmt = Stmt::parse(contents, "<filename>")?;
let locator = Locator::new(contents);
let range = identifier::else_(&stmt, &locator).unwrap();
assert_eq!(&contents[range], "else");
assert_eq!(
range,
TextRange::new(TextSize::from(21), TextSize::from(25))
);
Ok(())
}
}