mirror of
https://github.com/python/cpython.git
synced 2025-07-10 04:45:36 +00:00
gh-104169: Refactor tokenizer into lexer and wrappers (#110684)
* The lexer, which include the actual lexeme producing logic, goes into the `lexer` directory. * The wrappers, one wrapper per input mode (file, string, utf-8, and readline), go into the `tokenizer` directory and include logic for creating a lexer instance and managing the buffer for different modes. --------- Co-authored-by: Pablo Galindo <pablogsal@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
This commit is contained in:
parent
eb50cd37ea
commit
01481f2dc1
29 changed files with 3185 additions and 2988 deletions
|
@ -1,6 +1,8 @@
|
|||
#include "Python.h"
|
||||
#include "errcode.h"
|
||||
#include "../Parser/tokenizer.h"
|
||||
#include "../Parser/lexer/state.h"
|
||||
#include "../Parser/lexer/lexer.h"
|
||||
#include "../Parser/tokenizer/tokenizer.h"
|
||||
#include "../Parser/pegen.h" // _PyPegen_byte_offset_to_character_offset()
|
||||
#include "../Parser/pegen.h" // _PyPegen_byte_offset_to_character_offset()
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue