mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-10-02 13:22:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

Merged revisions 76914 via svnmerge from

Browse source 
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r76914 | ezio.melotti | 2009-12-20 14:24:57 +0200 (Sun, 20 Dec 2009) | 1 line

  Fixed capitalization and markup; added imports in the example
........
This commit is contained in:
Ezio Melotti 2009-12-20 12:26:45 +00:00
parent 4b9fbebaee
commit c5c8ff962f
1 changed files with 6 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

9
Doc/library/tokenize.rst
View file

@ -94,16 +94,19 @@ function it uses to do this is available:
(as a string) and a list of any lines (not decoded from bytes) it has read (as a string) and a list of any lines (not decoded from bytes) it has read
in. in.
It detects the encoding from the presence of a utf-8 bom or an encoding It detects the encoding from the presence of a UTF-8 BOM or an encoding
cookie as specified in pep-0263. If both a bom and a cookie are present, cookie as specified in :pep:`263`. If both a BOM and a cookie are present,
but disagree, a SyntaxError will be raised. but disagree, a SyntaxError will be raised.
If no encoding is specified, then the default of 'utf-8' will be returned. If no encoding is specified, then the default of ``'utf-8'`` will be returned.
Example of a script re-writer that transforms float literals into Decimal Example of a script re-writer that transforms float literals into Decimal
objects:: objects::
from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP
from io import BytesIO
def decistmt(s): def decistmt(s):
"""Substitute Decimals for floats in a string of statements. """Substitute Decimals for floats in a string of statements.