language-servers/rust-analyzer
1
0
Fork 0
mirror of https://github.com/rust-lang/rust-analyzer.git synced 2025-11-14 01:41:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Store token trees in contiguous Vec instead of as a tree

Browse source 
I expected this to be faster (due to less allocations and better cache locality), but benchmarked it is not (neither it is slower). Memory usage, however, drops by ~50mb (of `analysis-stats .`). I guess tt construction is just not hot.

This also simplifies using even less memory for token trees by compressing equal span, which I plan to do right after.

Some workflows are more easily expressed with a flat tt, while some are better expressed with a tree. With the right helpers, though (which was mostly a matter of trial and error), even the worst workflows become very easy indeed.
This commit is contained in:
Chayim Refael Friedman 2024-10-18 10:16:08 +03:00
parent 1c6b83852b
commit ceba289f80
50 changed files with 2356 additions and 2286 deletions
Download patch file Download diff file Expand all files Collapse all files

21
crates/syntax-bridge/src/tests.rs
View file

@ -2,10 +2,7 @@ use rustc_hash::FxHashMap;
use span::Span;
use syntax::{ast, AstNode};
use test_utils::extract_annotations;
use tt::{
buffer::{TokenBuffer, TokenTreeRef},
Leaf, Punct, Spacing,
};
use tt::{buffer::Cursor, Leaf, Punct, Spacing};
use crate::{
dummy_test_span_utils::{DummyTestSpanMap, DUMMY},
@ -32,22 +29,22 @@ fn check_punct_spacing(fixture: &str) {
})
.collect();
let buf = TokenBuffer::from_subtree(&subtree);
let mut cursor = buf.begin();
let mut cursor = Cursor::new(&subtree.0);
while !cursor.eof() {
while let Some(token_tree) = cursor.token_tree() {
if let TokenTreeRef::Leaf(
Leaf::Punct(Punct { spacing, span: Span { range, .. }, .. }),
_,
) = token_tree
if let tt::TokenTree::Leaf(Leaf::Punct(Punct {
spacing,
span: Span { range, .. },
..
})) = token_tree
{
if let Some(expected) = annotations.remove(range) {
assert_eq!(expected, *spacing);
}
}
cursor = cursor.bump_subtree();
cursor.bump();
}
cursor = cursor.bump();
cursor.bump_or_end();
}
assert!(annotations.is_empty(), "unchecked annotations: {annotations:?}");