mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-07-30 06:34:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

Added PEP 3127 support to tokenize (with tests); added PEP 3127 to NEWS.

Browse source
This commit is contained in:
Eric Smith 2008-03-17 19:43:40 +00:00
parent 6f778cfb5b
commit 0aed07ad80
3 changed files with 17 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

12
Lib/test/test_tokenize.py
View file

@ -4,7 +4,7 @@ Tests for the tokenize module.
>>> import glob, random, sys
The tests can be really simple. Given a small fragment of source
code, print out a table with thokens. The ENDMARK is omitted for
code, print out a table with tokens. The ENDMARK is omitted for
brevity.
>>> dump_tokens("1 + 1")
@ -106,7 +106,7 @@ Some error-handling code
... "else: print 'Loaded'\\n")
True
Balancing contunuation
Balancing continuation
>>> roundtrip("a = (3,4, \\n"
... "5,6)\\n"
@ -126,6 +126,14 @@ Ordinary integers and binary operators
NUMBER '0xff' (1, 0) (1, 4)
OP '<=' (1, 5) (1, 7)
NUMBER '255' (1, 8) (1, 11)
>>> dump_tokens("0b10 <= 255")
NUMBER '0b10' (1, 0) (1, 4)
OP '<=' (1, 5) (1, 7)
NUMBER '255' (1, 8) (1, 11)
>>> dump_tokens("0o123 <= 0123")
NUMBER '0o123' (1, 0) (1, 5)
OP '<=' (1, 6) (1, 8)
NUMBER '0123' (1, 9) (1, 13)
>>> dump_tokens("01234567 > ~0x15")
NUMBER '01234567' (1, 0) (1, 8)
OP '>' (1, 9) (1, 10)