Dhruv Manilawala 025768d303 Add Tokens newtype wrapper, TokenKind iterator (#11361) ... ## Summary Alternative to #11237 This PR adds a new `Tokens` struct which is a newtype wrapper around a vector of lexer output. This allows us to add a `kinds` method which returns an iterator over the corresponding `TokenKind`. This iterator is implemented as a separate `TokenKindIter` struct to allow using the type and provide additional methods like `peek` directly on the iterator. This exposes the linter to access the stream of `TokenKind` instead of `Tok`. Edit: I've made the necessary downstream changes and plan to merge the entire stack at once.		2024-05-14 16:45:04 +00:00
..
snapshots
block_comments.rs	Add Tokens newtype wrapper, TokenKind iterator (#11361)	2024-05-14 16:45:04 +00:00
simple_tokenizer.rs
whitespace.rs