language-servers/rust-analyzer
1
0
Fork 0
mirror of https://github.com/rust-lang/rust-analyzer.git synced 2025-09-27 04:19:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

internal: replace L_DOLLAR/R_DOLLAR with parenthesis hack

Browse source 
The general problem we are dealing with here is this:

```
macro_rules! thrice {
    ($e:expr) => { $e * 3}
}

fn main() {
    let x = thrice!(1 + 2);
}
```

we really want this to print 9 rather than 7.

The way rustc solves this is rather ad-hoc. In rustc, token trees are
allowed to include whole AST fragments, so 1+2 is passed through macro
expansion as a single unit. This is a significant violation of token
tree model.

In rust-analyzer, we intended to handle this in a more elegant way,
using token trees with "invisible" delimiters. The idea was is that we
introduce a new kind of parenthesis, "left $"/"right $", and let the
parser intelligently handle this.

The idea was inspired by the relevant comment in the proc_macro crate:

https://doc.rust-lang.org/stable/proc_macro/enum.Delimiter.html#variant.None

> An implicit delimiter, that may, for example, appear around tokens
> coming from a “macro variable” $var. It is important to preserve
> operator priorities in cases like $var * 3 where $var is 1 + 2.
> Implicit delimiters might not survive roundtrip of a token stream
> through a string.

Now that we are older and wiser, we conclude that the idea doesn't work.

_First_, the comment in the proc-macro crate is wishful thinking. Rustc
currently completely ignores none delimiters. It solves the (1 + 2) * 3
problem by having magical token trees which can't be duplicated:

* https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Frust-analyzer/topic/TIL.20that.20token.20streams.20are.20magic
* https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Handling.20of.20Delimiter.3A.3ANone.20by.20the.20parser

_Second_, it's not like our implementation in rust-analyzer works. We
special-case expressions (as opposed to treating all kinds of $var
captures the same) and we don't know how parser error recovery should
work with these dollar-parenthesis.

So, in this PR we simplify the whole thing away by not pretending that
we are doing something proper and instead just explicitly special-casing
expressions by wrapping them into real `()`.

In the future, to maintain bug-parity with `rustc` what we are going to
do is probably adding an explicit `CAPTURED_EXPR` *token* which we can
explicitly account for in the parser.

If/when rustc starts handling delimiter=none properly, we'll port that
logic as well, in addition to special handling.
This commit is contained in:
Aleksey Kladov 2021-10-23 20:08:42 +03:00
parent 9d33d05d85
commit 5a83d1be66
7 changed files with 169 additions and 134 deletions
Download patch file Download diff file Expand all files Collapse all files

99
crates/mbe/src/syntax_bridge.rs
View file

@ -632,12 +632,11 @@ impl<'a> TtTreeSink<'a> {
}
}
fn delim_to_str(d: Option<tt::DelimiterKind>, closing: bool) -> &'static str {
fn delim_to_str(d: tt::DelimiterKind, closing: bool) -> &'static str {
let texts = match d {
Some(tt::DelimiterKind::Parenthesis) => "()",
Some(tt::DelimiterKind::Brace) => "{}",
Some(tt::DelimiterKind::Bracket) => "[]",
None => return "",
tt::DelimiterKind::Parenthesis => "()",
tt::DelimiterKind::Brace => "{}",
tt::DelimiterKind::Bracket => "[]",
};
let idx = closing as usize;
@ -646,10 +645,6 @@ fn delim_to_str(d: Option<tt::DelimiterKind>, closing: bool) -> &'static str {
impl<'a> TreeSink for TtTreeSink<'a> {
fn token(&mut self, kind: SyntaxKind, mut n_tokens: u8) {
if kind == L_DOLLAR || kind == R_DOLLAR {
self.cursor = self.cursor.bump_subtree();
return;
}
if kind == LIFETIME_IDENT {
n_tokens = 2;
}
@ -661,48 +656,54 @@ impl<'a> TreeSink for TtTreeSink<'a> {
break;
}
last = self.cursor;
let text: &str = match self.cursor.token_tree() {
Some(tt::buffer::TokenTreeRef::Leaf(leaf, _)) => {
// Mark the range if needed
let (text, id) = match leaf {
tt::Leaf::Ident(ident) => (&ident.text, ident.id),
tt::Leaf::Punct(punct) => {
assert!(punct.char.is_ascii());
let char = &(punct.char as u8);
tmp_str = SmolStr::new_inline(
std::str::from_utf8(std::slice::from_ref(char)).unwrap(),
);
(&tmp_str, punct.id)
}
tt::Leaf::Literal(lit) => (&lit.text, lit.id),
};
let range = TextRange::at(self.text_pos, TextSize::of(text.as_str()));
self.token_map.insert(id, range);
self.cursor = self.cursor.bump();
text
}
Some(tt::buffer::TokenTreeRef::Subtree(subtree, _)) => {
self.cursor = self.cursor.subtree().unwrap();
if let Some(id) = subtree.delimiter.map(|it| it.id) {
self.open_delims.insert(id, self.text_pos);
}
delim_to_str(subtree.delimiter_kind(), false)
}
None => {
if let Some(parent) = self.cursor.end() {
self.cursor = self.cursor.bump();
if let Some(id) = parent.delimiter.map(|it| it.id) {
if let Some(open_delim) = self.open_delims.get(&id) {
let open_range = TextRange::at(*open_delim, TextSize::of('('));
let close_range = TextRange::at(self.text_pos, TextSize::of('('));
self.token_map.insert_delim(id, open_range, close_range);
let text: &str = loop {
break match self.cursor.token_tree() {
Some(tt::buffer::TokenTreeRef::Leaf(leaf, _)) => {
// Mark the range if needed
let (text, id) = match leaf {
tt::Leaf::Ident(ident) => (&ident.text, ident.id),
tt::Leaf::Punct(punct) => {
assert!(punct.char.is_ascii());
let char = &(punct.char as u8);
tmp_str = SmolStr::new_inline(
std::str::from_utf8(std::slice::from_ref(char)).unwrap(),
);
(&tmp_str, punct.id)
}
}
delim_to_str(parent.delimiter_kind(), true)
} else {
continue;
tt::Leaf::Literal(lit) => (&lit.text, lit.id),
};
let range = TextRange::at(self.text_pos, TextSize::of(text.as_str()));
self.token_map.insert(id, range);
self.cursor = self.cursor.bump();
text
}
}
Some(tt::buffer::TokenTreeRef::Subtree(subtree, _)) => {
self.cursor = self.cursor.subtree().unwrap();
match subtree.delimiter {
Some(d) => {
self.open_delims.insert(d.id, self.text_pos);
delim_to_str(d.kind, false)
}
None => continue,
}
}
None => {
let parent = self.cursor.end().unwrap();
self.cursor = self.cursor.bump();
match parent.delimiter {
Some(d) => {
if let Some(open_delim) = self.open_delims.get(&d.id) {
let open_range = TextRange::at(*open_delim, TextSize::of('('));
let close_range =
TextRange::at(self.text_pos, TextSize::of('('));
self.token_map.insert_delim(d.id, open_range, close_range);
}
delim_to_str(d.kind, true)
}
None => continue,
}
}
};
};
self.buf += text;
self.text_pos += TextSize::of(text);