language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-11-01 20:31:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

[ty] AST garbage collection (#18482)

Browse source 
## Summary

Garbage collect ASTs once we are done checking a given file. Queries
with a cross-file dependency on the AST will reparse the file on demand.
This reduces ty's peak memory usage by ~20-30%.

The primary change of this PR is adding a `node_index` field to every
AST node, that is assigned by the parser. `ParsedModule` can use this to
create a flat index of AST nodes any time the file is parsed (or
reparsed). This allows `AstNodeRef` to simply index into the current
instance of the `ParsedModule`, instead of storing a pointer directly.

The indices are somewhat hackily (using an atomic integer) assigned by
the `parsed_module` query instead of by the parser directly. Assigning
the indices in source-order in the (recursive) parser turns out to be
difficult, and collecting the nodes during semantic indexing is
impossible as `SemanticIndex` does not hold onto a specific
`ParsedModuleRef`, which the pointers in the flat AST are tied to. This
means that we have to do an extra AST traversal to assign and collect
the nodes into a flat index, but the small performance impact (~3% on
cold runs) seems worth it for the memory savings.

Part of https://github.com/astral-sh/ty/issues/214.
This commit is contained in:
Ibraheem Ahmed 2025-06-13 08:40:11 -04:00 • committed by GitHub
parent 76d9009a6e
commit c9dff5c7d5
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
824 changed files with 25243 additions and 804 deletions
Download patch file Download diff file Expand all files Collapse all files

42
crates/ruff_python_parser/tests/snapshots/valid_syntax@expressions__attribute.py.snap
View file

@ -1,23 +1,26 @@
---
source: crates/ruff_python_parser/tests/fixtures.rs
input_file: crates/ruff_python_parser/resources/valid/expressions/attribute.py
snapshot_kind: text
---
## AST
```
Module(
ModModule {
node_index: AtomicNodeIndex(..),
range: 0..90,
body: [
Expr(
StmtExpr {
node_index: AtomicNodeIndex(..),
range: 0..10,
value: Attribute(
ExprAttribute {
node_index: AtomicNodeIndex(..),
range: 0..10,
value: Name(
ExprName {
node_index: AtomicNodeIndex(..),
range: 0..5,
id: Name("value"),
ctx: Load,
@ -26,6 +29,7 @@ Module(
attr: Identifier {
id: Name("attr"),
range: 6..10,
node_index: AtomicNodeIndex(..),
},
ctx: Load,
},
@ -34,15 +38,19 @@ Module(
),
Expr(
StmtExpr {
node_index: AtomicNodeIndex(..),
range: 11..23,
value: Call(
ExprCall {
node_index: AtomicNodeIndex(..),
range: 11..23,
func: Attribute(
ExprAttribute {
node_index: AtomicNodeIndex(..),
range: 11..21,
value: Name(
ExprName {
node_index: AtomicNodeIndex(..),
range: 11..16,
id: Name("value"),
ctx: Load,
@ -51,12 +59,14 @@ Module(
attr: Identifier {
id: Name("attr"),
range: 17..21,
node_index: AtomicNodeIndex(..),
},
ctx: Load,
},
),
arguments: Arguments {
range: 21..23,
node_index: AtomicNodeIndex(..),
args: [],
keywords: [],
},
@ -66,15 +76,19 @@ Module(
),
Expr(
StmtExpr {
node_index: AtomicNodeIndex(..),
range: 24..36,
value: Attribute(
ExprAttribute {
node_index: AtomicNodeIndex(..),
range: 24..36,
value: Call(
ExprCall {
node_index: AtomicNodeIndex(..),
range: 24..31,
func: Name(
ExprName {
node_index: AtomicNodeIndex(..),
range: 24..29,
id: Name("value"),
ctx: Load,
@ -82,6 +96,7 @@ Module(
),
arguments: Arguments {
range: 29..31,
node_index: AtomicNodeIndex(..),
args: [],
keywords: [],
},
@ -90,6 +105,7 @@ Module(
attr: Identifier {
id: Name("attr"),
range: 32..36,
node_index: AtomicNodeIndex(..),
},
ctx: Load,
},
@ -98,21 +114,27 @@ Module(
),
Expr(
StmtExpr {
node_index: AtomicNodeIndex(..),
range: 37..55,
value: Attribute(
ExprAttribute {
node_index: AtomicNodeIndex(..),
range: 37..55,
value: Call(
ExprCall {
node_index: AtomicNodeIndex(..),
range: 37..51,
func: Attribute(
ExprAttribute {
node_index: AtomicNodeIndex(..),
range: 37..49,
value: Call(
ExprCall {
node_index: AtomicNodeIndex(..),
range: 37..44,
func: Name(
ExprName {
node_index: AtomicNodeIndex(..),
range: 37..42,
id: Name("value"),
ctx: Load,
@ -120,6 +142,7 @@ Module(
),
arguments: Arguments {
range: 42..44,
node_index: AtomicNodeIndex(..),
args: [],
keywords: [],
},
@ -128,12 +151,14 @@ Module(
attr: Identifier {
id: Name("attr"),
range: 45..49,
node_index: AtomicNodeIndex(..),
},
ctx: Load,
},
),
arguments: Arguments {
range: 49..51,
node_index: AtomicNodeIndex(..),
args: [],
keywords: [],
},
@ -142,6 +167,7 @@ Module(
attr: Identifier {
id: Name("foo"),
range: 52..55,
node_index: AtomicNodeIndex(..),
},
ctx: Load,
},
@ -150,15 +176,19 @@ Module(
),
Expr(
StmtExpr {
node_index: AtomicNodeIndex(..),
range: 56..70,
value: Attribute(
ExprAttribute {
node_index: AtomicNodeIndex(..),
range: 56..70,
value: Attribute(
ExprAttribute {
node_index: AtomicNodeIndex(..),
range: 56..66,
value: Name(
ExprName {
node_index: AtomicNodeIndex(..),
range: 56..61,
id: Name("value"),
ctx: Load,
@ -167,6 +197,7 @@ Module(
attr: Identifier {
id: Name("attr"),
range: 62..66,
node_index: AtomicNodeIndex(..),
},
ctx: Load,
},
@ -174,6 +205,7 @@ Module(
attr: Identifier {
id: Name("foo"),
range: 67..70,
node_index: AtomicNodeIndex(..),
},
ctx: Load,
},
@ -182,18 +214,23 @@ Module(
),
Expr(
StmtExpr {
node_index: AtomicNodeIndex(..),
range: 71..89,
value: Attribute(
ExprAttribute {
node_index: AtomicNodeIndex(..),
range: 71..89,
value: Call(
ExprCall {
node_index: AtomicNodeIndex(..),
range: 71..85,
func: Attribute(
ExprAttribute {
node_index: AtomicNodeIndex(..),
range: 71..83,
value: Name(
ExprName {
node_index: AtomicNodeIndex(..),
range: 72..77,
id: Name("value"),
ctx: Load,
@ -202,12 +239,14 @@ Module(
attr: Identifier {
id: Name("attr"),
range: 79..83,
node_index: AtomicNodeIndex(..),
},
ctx: Load,
},
),
arguments: Arguments {
range: 83..85,
node_index: AtomicNodeIndex(..),
args: [],
keywords: [],
},
@ -216,6 +255,7 @@ Module(
attr: Identifier {
id: Name("foo"),
range: 86..89,
node_index: AtomicNodeIndex(..),
},
ctx: Load,
},