mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-07-07 19:35:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

gh-127303: Add docs for token.EXACT_TOKEN_TYPES (#127304)

Browse source 
---------

Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
This commit is contained in:
Илья Любавский 2024-11-29 12:00:50 +03:00 • committed by GitHub
parent b83be9c971
commit dd3a87d2a8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 13 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

7
Doc/library/token.rst
View file

@ -79,6 +79,13 @@ the :mod:`tokenize` module.
``type_comments=True``.
.. data:: EXACT_TOKEN_TYPES
A dictionary mapping the string representation of a token to its numeric code.
.. versionadded:: 3.8
.. versionchanged:: 3.5
Added :data:`!AWAIT` and :data:`!ASYNC` tokens.

3
Lib/token.py generated
View file

@ -1,7 +1,8 @@
"""Token constants."""
# Auto-generated by Tools/build/generate_token.py
__all__ = ['tok_name', 'ISTERMINAL', 'ISNONTERMINAL', 'ISEOF']
__all__ = ['tok_name', 'ISTERMINAL', 'ISNONTERMINAL', 'ISEOF',
'EXACT_TOKEN_TYPES']
ENDMARKER = 0
NAME = 1

1
Misc/ACKS
View file

@ -1154,6 +1154,7 @@ Mark Lutz
Taras Lyapun
Jim Lynch
Mikael Lyngvig
Ilya Lyubavski
Jeff MacDonald
John Machin
Andrew I MacIntyre

1
Misc/NEWS.d/next/Library/2024-11-27-16-06-10.gh-issue-127303.asqkgh.rst Normal file
View file

@ -0,0 +1 @@
Publicly expose :data:`~token.EXACT_TOKEN_TYPES` in :attr:`!token.__all__`.

3
Tools/build/generate_token.py
View file

@ -226,7 +226,8 @@ token_py_template = f'''\
# {AUTO_GENERATED_BY_SCRIPT}
'''
token_py_template += '''
__all__ = ['tok_name', 'ISTERMINAL', 'ISNONTERMINAL', 'ISEOF']
__all__ = ['tok_name', 'ISTERMINAL', 'ISNONTERMINAL', 'ISEOF',
'EXACT_TOKEN_TYPES']
%s
N_TOKENS = %d