mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-12-04 00:30:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

handle names starting with non-ascii characters correctly #9712

Browse source
This commit is contained in:
Benjamin Peterson 2010-08-30 14:41:20 +00:00
parent e01de8f2f3
commit 33856de84d
3 changed files with 25 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

13
Lib/test/test_tokenize.py
View file

@ -531,6 +531,7 @@ pass the '-ucompiler' option to process the full directory.
True
Evil tabs
>>> dump_tokens("def f():\\n\\tif x\\n \\tpass")
ENCODING 'utf-8' (0, 0) (0, 0)
NAME 'def' (1, 0) (1, 3)
@ -547,6 +548,18 @@ Evil tabs
NAME 'pass' (3, 9) (3, 13)
DEDENT '' (4, 0) (4, 0)
DEDENT '' (4, 0) (4, 0)
Non-ascii identifiers
>>> dump_tokens("Örter = 'places'\\ngrün = 'green'")
ENCODING 'utf-8' (0, 0) (0, 0)
NAME 'Örter' (1, 0) (1, 5)
OP '=' (1, 6) (1, 7)
STRING "'places'" (1, 8) (1, 16)
NEWLINE '\\n' (1, 16) (1, 17)
NAME 'grün' (2, 0) (2, 4)
OP '=' (2, 5) (2, 6)
STRING "'green'" (2, 7) (2, 14)
"""
from test import support