mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-07-12 22:05:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 14

gh-104169: Refactor tokenizer into lexer and wrappers (#110684)

Browse source 
* The lexer, which include the actual lexeme producing logic, goes into
  the `lexer` directory.
* The wrappers, one wrapper per input mode (file, string, utf-8, and
  readline), go into the `tokenizer` directory and include logic for
  creating a lexer instance and managing the buffer for different modes.
---------

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
This commit is contained in:
Lysandros Nikolaou 2023-10-11 17:14:44 +02:00 • committed by GitHub
parent eb50cd37ea
commit 01481f2dc1
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
29 changed files with 3185 additions and 2988 deletions
Download patch file Download diff file Expand all files Collapse all files

4
Python/Python-tokenize.c
View file

@ -1,6 +1,8 @@
#include "Python.h"
#include "errcode.h"
#include "../Parser/tokenizer.h"
#include "../Parser/lexer/state.h"
#include "../Parser/lexer/lexer.h"
#include "../Parser/tokenizer/tokenizer.h"
#include "../Parser/pegen.h" // _PyPegen_byte_offset_to_character_offset()
#include "../Parser/pegen.h" // _PyPegen_byte_offset_to_character_offset()