[ty] AST garbage collection (#18482)

## Summary

Garbage collect ASTs once we are done checking a given file. Queries
with a cross-file dependency on the AST will reparse the file on demand.
This reduces ty's peak memory usage by ~20-30%.

The primary change of this PR is adding a `node_index` field to every
AST node, that is assigned by the parser. `ParsedModule` can use this to
create a flat index of AST nodes any time the file is parsed (or
reparsed). This allows `AstNodeRef` to simply index into the current
instance of the `ParsedModule`, instead of storing a pointer directly.

The indices are somewhat hackily (using an atomic integer) assigned by
the `parsed_module` query instead of by the parser directly. Assigning
the indices in source-order in the (recursive) parser turns out to be
difficult, and collecting the nodes during semantic indexing is
impossible as `SemanticIndex` does not hold onto a specific
`ParsedModuleRef`, which the pointers in the flat AST are tied to. This
means that we have to do an extra AST traversal to assign and collect
the nodes into a flat index, but the small performance impact (~3% on
cold runs) seems worth it for the memory savings.

Part of https://github.com/astral-sh/ty/issues/214.
This commit is contained in:
Ibraheem Ahmed 2025-06-13 08:40:11 -04:00 committed by GitHub
parent 76d9009a6e
commit c9dff5c7d5
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
824 changed files with 25243 additions and 804 deletions

View file

@ -4,11 +4,11 @@ use {
regex::Regex,
};
use ruff_python_ast::visitor::transformer::Transformer;
use ruff_python_ast::{
self as ast, BytesLiteralFlags, Expr, FStringFlags, FStringPart, InterpolatedStringElement,
InterpolatedStringLiteralElement, Stmt, StringFlags,
};
use ruff_python_ast::{AtomicNodeIndex, visitor::transformer::Transformer};
use ruff_python_ast::{StringLiteralFlags, visitor::transformer};
use ruff_text_size::{Ranged, TextRange};
@ -82,6 +82,7 @@ impl Transformer for Normalizer {
value: Box::from(string.value.to_str()),
range: string.range,
flags: StringLiteralFlags::empty(),
node_index: AtomicNodeIndex::dummy(),
});
}
}
@ -98,6 +99,7 @@ impl Transformer for Normalizer {
value: bytes.value.bytes().collect(),
range: bytes.range,
flags: BytesLiteralFlags::empty(),
node_index: AtomicNodeIndex::dummy(),
});
}
}
@ -141,6 +143,7 @@ impl Transformer for Normalizer {
InterpolatedStringLiteralElement {
range,
value: literal.into(),
node_index: AtomicNodeIndex::dummy(),
},
));
}
@ -182,6 +185,7 @@ impl Transformer for Normalizer {
elements: collector.elements.into(),
range: fstring.range,
flags: FStringFlags::empty(),
node_index: AtomicNodeIndex::dummy(),
});
}
}