Move token and error structs into related modules (#11957)

## Summary

This PR does some housekeeping into moving certain structs into related
modules. Specifically,
1. Move `LexicalError` from `lexer.rs` to `error.rs` which also contains
the `ParseError`
2. Move `Token`, `TokenFlags` and `TokenValue` from `lexer.rs` to
`token.rs`
This commit is contained in:
Dhruv Manilawala 2024-06-21 15:37:19 +05:30 committed by GitHub
parent 4667d8697c
commit 96da136e6a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 352 additions and 342 deletions

View file

@ -5,9 +5,9 @@ use bitflags::bitflags;
use ruff_python_ast::{Mod, ModExpression, ModModule};
use ruff_text_size::{Ranged, TextRange, TextSize};
use crate::lexer::TokenValue;
use crate::parser::expression::ExpressionContext;
use crate::parser::progress::{ParserProgress, TokenId};
use crate::token::TokenValue;
use crate::token_set::TokenSet;
use crate::token_source::{TokenSource, TokenSourceCheckpoint};
use crate::{Mode, ParseError, ParseErrorType, TokenKind};