mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-09-27 02:39:58 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 13

bpo-36654: Add examples for using tokenize module programmically (#12947)

Browse source
This commit is contained in:
Windson yang 2020-01-26 03:23:00 +08:00 • committed by Berker Peksag
parent 7de617455e
commit 4b09dc79f4
1 changed files with 19 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

19
Doc/library/tokenize.rst
View file

@ -278,3 +278,22 @@ The exact token type names can be displayed using the :option:`-e` option:
4,10-4,11: RPAR ')' 4,10-4,11: RPAR ')'
4,11-4,12: NEWLINE '\n' 4,11-4,12: NEWLINE '\n'
5,0-5,0: ENDMARKER '' 5,0-5,0: ENDMARKER ''
Example of tokenizing a file programmatically, reading unicode
strings instead of bytes with :func:`generate_tokens`::
import tokenize
with tokenize.open('hello.py') as f:
tokens = tokenize.generate_tokens(f.readline)
for token in tokens:
print(token)
Or reading bytes directly with :func:`.tokenize`::
import tokenize
with open('hello.py', 'rb') as f:
tokens = tokenize.tokenize(f.readline)
for token in tokens:
print(token)