mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-10-17 20:28:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

gh-104169: Refactor tokenizer into lexer and wrappers (#110684)

Browse source 
* The lexer, which include the actual lexeme producing logic, goes into
  the `lexer` directory.
* The wrappers, one wrapper per input mode (file, string, utf-8, and
  readline), go into the `tokenizer` directory and include logic for
  creating a lexer instance and managing the buffer for different modes.
---------

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
This commit is contained in:
Lysandros Nikolaou 2023-10-11 17:14:44 +02:00 • committed by GitHub
parent eb50cd37ea
commit 01481f2dc1
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
29 changed files with 3185 additions and 2988 deletions
Download patch file Download diff file Expand all files Collapse all files

2
Tools/c-analyzer/cpython/ignored.tsv
View file

@ -335,7 +335,7 @@ Objects/unicodeobject.c unicode_encode_call_errorhandler argparse -
Objects/unicodeobject.c unicode_translate_call_errorhandler argparse -
Parser/parser.c - reserved_keywords -
Parser/parser.c - soft_keywords -
Parser/tokenizer.c - type_comment_prefix -
Parser/lexer/lexer.c - type_comment_prefix -
Python/ast_opt.c fold_unaryop ops -
Python/ceval.c - _PyEval_BinaryOps -
Python/ceval.c - _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS -

Can't render this file because it has a wrong number of fields in line 4.