Auto merge of #13854 - lowr:fix/mbe-glue-punct, r=Veykril

Support multi-character punct tokens in MBE

Fixes #11497

In the context of MBE, consecutive puncts are parsed as multi-character punct tokens whenever possible. For example, `:::` is parsed as ``[Punct(`::`), Punct(`:`)]`` and shouldn't get matched to patterns like `: : :` or `: ::`.

We have implemented this behavior only for when we match puncts against `tt` fragments, but not when we match puncts literally. This PR extracts the multi-character punct handling procedure into a separate method and extends its support for literal matching.

For good measure, this PR adds support for `<-` token, which is still [considered as one token in rustc](e396186407/compiler/rustc_ast/src/token.rs (L249)) despite the placement syntax having been removed.
This commit is contained in:
bors 2023-01-02 13:07:42 +00:00
commit 790759fb11
6 changed files with 258 additions and 126 deletions

View file

@ -1630,3 +1630,48 @@ const _: i32 = -0--1--2;
"#]],
);
}
#[test]
fn test_punct_without_space() {
// Puncts are "glued" greedily.
check(
r#"
macro_rules! foo {
(: : :) => { "1 1 1" };
(: ::) => { "1 2" };
(:: :) => { "2 1" };
(: : : :) => { "1 1 1 1" };
(:: : :) => { "2 1 1" };
(: :: :) => { "1 2 1" };
(: : ::) => { "1 1 2" };
(:: ::) => { "2 2" };
}
fn test() {
foo!(:::);
foo!(: :::);
foo!(::::);
}
"#,
expect![[r#"
macro_rules! foo {
(: : :) => { "1 1 1" };
(: ::) => { "1 2" };
(:: :) => { "2 1" };
(: : : :) => { "1 1 1 1" };
(:: : :) => { "2 1 1" };
(: :: :) => { "1 2 1" };
(: : ::) => { "1 1 2" };
(:: ::) => { "2 2" };
}
fn test() {
"2 1";
"1 2 1";
"2 2";
}
"#]],
);
}