mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-07-07 19:35:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

gh-104169: Fix test_peg_generator after tokenizer refactoring (#110727)

Browse source 
* Fix test_peg_generator after tokenizer refactoring
* Remove references to tokenizer.c in comments etc.
This commit is contained in:
Lysandros Nikolaou 2023-10-12 09:34:35 +02:00 • committed by GitHub
parent 23645420dc
commit 17d65547df
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
10 changed files with 24 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

2
Lib/tokenize.py
View file

@ -298,7 +298,7 @@ def untokenize(iterable):
def _get_normal_name(orig_enc):
"""Imitates get_normal_name in tokenizer.c."""
"""Imitates get_normal_name in Parser/tokenizer/helpers.c."""
# Only care about the first 12 characters.
enc = orig_enc[:12].lower().replace("_", "-")
if enc == "utf-8" or enc.startswith("utf-8-"):