mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-09-27 10:50:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 13

Merged revisions 76063,76068 via svnmerge from

Browse source 
svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
  r76063 | benjamin.peterson | 2009-11-02 12:16:28 -0600 (Mon, 02 Nov 2009) | 77 lines

  Merged revisions 76062 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ................
    r76062 | benjamin.peterson | 2009-11-02 12:12:12 -0600 (Mon, 02 Nov 2009) | 70 lines

    Merged revisions 74359,75081,75088,75213,75278,75303,75427-75428,75734-75736,75865,76059-76061 via svnmerge from
    svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

    ........
      r74359 | benjamin.peterson | 2009-08-12 17:23:13 -0500 (Wed, 12 Aug 2009) | 1 line

      don't pass the deprecated print_function option
    ........
      r75081 | benjamin.peterson | 2009-09-26 22:02:57 -0500 (Sat, 26 Sep 2009) | 1 line

      let 2to3 work with extended iterable unpacking
    ........
      r75088 | benjamin.peterson | 2009-09-27 11:25:21 -0500 (Sun, 27 Sep 2009) | 1 line

      look on the type only for __call__
    ........
      r75213 | benjamin.peterson | 2009-10-03 10:09:46 -0500 (Sat, 03 Oct 2009) | 5 lines

      revert 75212; it's not correct

      People can use isinstance(x, collections.Callable) if they expect objects with
      __call__ in their instance dictionaries.
    ........
      r75278 | benjamin.peterson | 2009-10-07 16:25:56 -0500 (Wed, 07 Oct 2009) | 4 lines

      fix whitespace problems with fix_idioms #3563

      Patch by Joe Amenta.
    ........
      r75303 | benjamin.peterson | 2009-10-09 16:59:11 -0500 (Fri, 09 Oct 2009) | 1 line

      port latin-1 and utf-8 cookie improvements
    ........
      r75427 | benjamin.peterson | 2009-10-14 20:35:57 -0500 (Wed, 14 Oct 2009) | 1 line

      force floor division
    ........
      r75428 | benjamin.peterson | 2009-10-14 20:39:21 -0500 (Wed, 14 Oct 2009) | 1 line

      silence -3 warnings about __hash__
    ........
      r75734 | benjamin.peterson | 2009-10-26 16:25:53 -0500 (Mon, 26 Oct 2009) | 2 lines

      warn on map(None, ...) with more than 2 arguments #7203
    ........
      r75735 | benjamin.peterson | 2009-10-26 16:28:25 -0500 (Mon, 26 Oct 2009) | 1 line

      remove unused result
    ........
      r75736 | benjamin.peterson | 2009-10-26 16:29:02 -0500 (Mon, 26 Oct 2009) | 1 line

      using get() here is a bit pointless
    ........
      r75865 | benjamin.peterson | 2009-10-27 15:49:00 -0500 (Tue, 27 Oct 2009) | 1 line

      explain reason for warning
    ........
      r76059 | benjamin.peterson | 2009-11-02 11:43:47 -0600 (Mon, 02 Nov 2009) | 1 line

      tuples are no longer used for children
    ........
      r76060 | benjamin.peterson | 2009-11-02 11:55:40 -0600 (Mon, 02 Nov 2009) | 1 line

      revert r76059; apparently some fixers rely on Leaf no () for children
    ........
      r76061 | benjamin.peterson | 2009-11-02 12:06:17 -0600 (Mon, 02 Nov 2009) | 1 line

      make fix_tuple_params keep the tree valid #7253
    ........
  ................
................
  r76068 | benjamin.peterson | 2009-11-02 12:30:48 -0600 (Mon, 02 Nov 2009) | 24 lines

  Merged revisions 76064,76066-76067 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ................
    r76064 | benjamin.peterson | 2009-11-02 12:16:36 -0600 (Mon, 02 Nov 2009) | 1 line

    add space
  ................
    r76066 | benjamin.peterson | 2009-11-02 12:22:53 -0600 (Mon, 02 Nov 2009) | 9 lines

    Merged revisions 76065 via svnmerge from
    svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

    ........
      r76065 | benjamin.peterson | 2009-11-02 12:21:25 -0600 (Mon, 02 Nov 2009) | 1 line

      don't print stuff in tests
    ........
  ................
    r76067 | benjamin.peterson | 2009-11-02 12:24:57 -0600 (Mon, 02 Nov 2009) | 1 line

    enable test_parser in lib2to3
  ................
................
This commit is contained in:
Benjamin Peterson 2009-11-02 18:33:36 +00:00
parent 868b578929
commit e80b51fab7
10 changed files with 118 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

15
Lib/lib2to3/pgen2/tokenize.py
View file

@ -231,6 +231,17 @@ class Untokenizer:
cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
def _get_normal_name(orig_enc):
"""Imitates get_normal_name in tokenizer.c."""
# Only care about the first 12 characters.
enc = orig_enc[:12].lower().replace("_", "-")
if enc == "utf-8" or enc.startswith("utf-8-"):
return "utf-8"
if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
return "iso-8859-1"
return orig_enc
def detect_encoding(readline):
"""
The detect_encoding() function is used to detect the encoding that should
@ -265,7 +276,7 @@ def detect_encoding(readline):
matches = cookie_re.findall(line_string)
if not matches:
return None
encoding = matches[0]
encoding = _get_normal_name(matches[0])
try:
codec = lookup(encoding)
except LookupError:
@ -375,7 +386,7 @@ def generate_tokens(readline):
column = 0
while pos < max: # measure leading whitespace
if line[pos] == ' ': column = column + 1
elif line[pos] == '\t': column = (column/tabsize + 1)*tabsize
elif line[pos] == '\t': column = (column//tabsize + 1)*tabsize
elif line[pos] == '\f': column = 0
else: break
pos = pos + 1