Charlie Marsh 86ccdcc9d9 Add support for multi-character operator tokens to SimpleTokenizer (#6563) ... ## Summary Allows for proper lexing of tokens like `->`. The main challenge is to ensure that our forward and backwards representations are the same for cases like `===`. Specifically, we want that to lex as `==` followed by `=` regardless of whether it's a forwards or backwards lex. To do so, we identify the range of the sequential characters (the full span of `===`), lex it forwards, then return the last token. ## Test Plan `cargo test`		2023-08-16 09:09:19 -04:00
..
src	Add support for multi-character operator tokens to SimpleTokenizer (#6563)	2023-08-16 09:09:19 -04:00
Cargo.toml	Remove some extraneous newlines in Cargo.toml (#6577)	2023-08-14 23:39:41 +00:00