mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-08-04 17:08:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 13

bpo-25324: Move the description of tokenize tokens to token.rst. (#1911)

Browse source
This commit is contained in:
Serhiy Storchaka 2017-06-06 18:43:35 +03:00 • committed by GitHub
parent 6260d9f203
commit 5cefb6cfdd
2 changed files with 39 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

41
Doc/library/token.rst
View file

@ -101,18 +101,37 @@ The token constants are:
AWAIT
ASYNC
ERRORTOKEN
COMMENT
NL
ENCODING
N_TOKENS
NT_OFFSET
.. versionchanged:: 3.5
Added :data:`AWAIT` and :data:`ASYNC` tokens. Starting with
Python 3.7, "async" and "await" will be tokenized as :data:`NAME`
tokens, and :data:`AWAIT` and :data:`ASYNC` will be removed.
.. versionchanged:: 3.7
Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` to bring
the tokens in the C code in line with the tokens needed in
:mod:`tokenize` module. These tokens aren't used by the C tokenizer.
The following token type values aren't used by the C tokenizer but are needed for
the :mod:`tokenize` module.
.. data:: COMMENT
Token value used to indicate a comment.
.. data:: NL
Token value used to indicate a non-terminating newline. The
:data:`NEWLINE` token indicates the end of a logical line of Python code;
``NL`` tokens are generated when a logical line of code is continued over
multiple physical lines.
.. data:: ENCODING
Token value that indicates the encoding used to decode the source bytes
into text. The first token returned by :func:`tokenize.tokenize` will
always be an ``ENCODING`` token.
.. versionchanged:: 3.5
Added :data:`AWAIT` and :data:`ASYNC` tokens. Starting with
Python 3.7, "async" and "await" will be tokenized as :data:`NAME`
tokens, and :data:`AWAIT` and :data:`ASYNC` will be removed.
.. versionchanged:: 3.7
Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` tokens.