String annotations [PEP 563] (#4390)

* Document `from __future__ import annotations`
* Provide plumbing and tests for `from __future__ import annotations`
* Implement unparsing the AST back to string form

This is required for PEP 563 and as such only implements a part of the
unparsing process that covers expressions.

Doc/glossary.rst

@@ -372,9 +372,11 @@ Glossary
       may be accessed via the :attr:`__annotations__` special attribute of a
       function object.
 
-      Python itself does not assign any particular meaning to function
+      See also the :term:`variable annotation` glossary entry.
-      annotations. They are intended to be interpreted by third-party libraries
+
-      or tools.  See :pep:`3107`, which describes some of their potential uses.
+      Annotations are meant to provide a standard way for programmers to
+      document types of functions they design.  See :pep:`484`, which
+      describes this functionality.
 
    __future__
       A pseudo-module which programmers can use to enable new language features
@@ -1021,10 +1023,11 @@ Glossary
       attribute of a class or module object and can be accessed using
       :func:`typing.get_type_hints`.
 
-      Python itself does not assign any particular meaning to variable
+      See also the :term:`function annotation` glossary entry.
-      annotations. They are intended to be interpreted by third-party libraries
+
-      or type checking tools. See :pep:`526`, :pep:`484` which describe
+      Annotations are meant to provide a standard way for programmers to
-      some of their potential uses.
+      document types of functions they design.  See :pep:`484` and :pep:`526`
+      which describe this functionality.
 
    virtual environment
       A cooperatively isolated runtime environment that allows Python users

Doc/library/__future__.rst

@@ -90,6 +90,11 @@ language using this mechanism:
 | generator_stop   | 3.5.0b1     | 3.7          | :pep:`479`:                                 |
 |                  |             |              | *StopIteration handling inside generators*  |
 +------------------+-------------+--------------+---------------------------------------------+
+| annotations      | 3.7.0b1     | 4.0          | :pep:`563`:                                 |
+|                  |             |              | *Postponed evaluation of annotations*       |
++------------------+-------------+--------------+---------------------------------------------+
+
+.. XXX Adding a new entry?  Remember to update simple_stmts.rst, too.
 
 
 .. seealso::

Doc/reference/compound_stmts.rst

@@ -559,12 +559,14 @@ Parameters may have annotations of the form "``: expression``" following the
 parameter name.  Any parameter may have an annotation even those of the form
 ``*identifier`` or ``**identifier``.  Functions may have "return" annotation of
 the form "``-> expression``" after the parameter list.  These annotations can be
-any valid Python expression and are evaluated when the function definition is
+any valid Python expression.  The presence of annotations does not change the
-executed.  Annotations may be evaluated in a different order than they appear in
+semantics of a function.  The annotation values are available as values of
-the source code.  The presence of annotations does not change the semantics of a
+a dictionary keyed by the parameters' names in the :attr:`__annotations__`
-function.  The annotation values are available as values of a dictionary keyed
+attribute of the function object.  If the ``annotations`` import from
-by the parameters' names in the :attr:`__annotations__` attribute of the
+:mod:`__future__` is used, annotations are preserved as strings at runtime which
-function object.
+enables postponed evaluation.  Otherwise, they are evaluated when the function
+definition is executed.  In this case annotations may be evaluated in
+a different order than they appear in the source code.
 
 .. index:: pair: lambda; expression
 
@@ -587,6 +589,17 @@ access the local variables of the function containing the def.  See section
    :pep:`3107` - Function Annotations
       The original specification for function annotations.
 
+   :pep:`484` - Type Hints
+      Definition of a standard meaning for annotations: type hints.
+
+   :pep:`526` - Syntax for Variable Annotations
+      Ability to type hint variable declarations, including class
+      variables and instance variables
+
+   :pep:`563` - Postponed Evaluation of Annotations
+      Support for forward references within annotations by preserving
+      annotations in a string form at runtime instead of eager evaluation.
+
 
 .. _class:
 

Doc/reference/simple_stmts.rst

@@ -853,12 +853,15 @@ can appear before a future statement are:
 * blank lines, and
 * other future statements.
 
-.. XXX change this if future is cleaned out
+The only feature in Python 3.7 that requires using the future statement is
+``annotations``.
 
-The features recognized by Python 3.0 are ``absolute_import``, ``division``,
+All historical features enabled by the future statement are still recognized
-``generators``, ``unicode_literals``, ``print_function``, ``nested_scopes`` and
+by Python 3.  The list includes ``absolute_import``, ``division``,
-``with_statement``.  They are all redundant because they are always enabled, and
+``generators``, ``generator_stop``, ``unicode_literals``,
-only kept for backwards compatibility.
+``print_function``, ``nested_scopes`` and ``with_statement``.  They are
+all redundant because they are always enabled, and only kept for
+backwards compatibility.
 
 A future statement is recognized and treated specially at compile time: Changes
 to the semantics of core constructs are often implemented by generating

Doc/whatsnew/3.7.rst

@@ -179,6 +179,57 @@ a normal ``__getattr__`` method, except that it will be defined on module
        PEP written and implemented by Ivan Levkivskyi
 
 
+PEP 563: Postponed evaluation of annotations
+--------------------------------------------
+
+The advent of type hints in Python uncovered two glaring usability issues
+with the functionality of annotations added in :pep:`3107` and refined
+further in :pep:`526`:
+
+* annotations could only use names which were already available in the
+  current scope, in other words they didn't support forward references
+  of any kind; and
+
+* annotating source code had adverse effects on startup time of Python
+  programs.
+
+Both of these issues are fixed by postponing the evaluation of
+annotations.  Instead of compiling code which executes expressions in
+annotations at their definition time, the compiler stores the annotation
+in a string form equivalent to the AST of the expression in question.
+If needed, annotations can be resolved at runtime using
+``typing.get_type_hints()``.  In the common case where this is not
+required, the annotations are cheaper to store (since short strings
+are interned by the interpreter) and make startup time faster.
+
+Usability-wise, annotations now support forward references, making the
+following syntax valid::
+
+    class C:
+        @classmethod
+        def from_string(cls, source: str) -> C:
+            ...
+
+        def validate_b(self, obj: B) -> bool:
+            ...
+
+    class B:
+        ...
+
+Since this change breaks compatibility, the new behavior can be enabled
+on a per-module basis in Python 3.7 using a ``__future__`` import, like
+this::
+
+    from __future__ import annotations
+
+It will become the default in Python 4.0.
+
+.. seealso::
+
+    :pep:`563` -- Postponed evaluation of annotations
+       PEP written and implemented by Łukasz Langa.
+
+
 PEP 564: Add new time functions with nanosecond resolution
 ----------------------------------------------------------
 

Include/ast.h

@@ -16,6 +16,15 @@ PyAPI_FUNC(mod_ty) PyAST_FromNodeObject(
     PyObject *filename,
     PyArena *arena);
 
+#ifndef Py_LIMITED_API
+
+/* _PyAST_ExprAsUnicode is defined in ast_unparse.c */
+PyAPI_FUNC(PyObject *) _PyAST_ExprAsUnicode(
+    expr_ty e,
+    int omit_parens);
+
+#endif /* !Py_LIMITED_API */
+
 #ifdef __cplusplus
 }
 #endif

Include/code.h

@@ -82,6 +82,7 @@ typedef struct {
 
 #define CO_FUTURE_BARRY_AS_BDFL  0x40000
 #define CO_FUTURE_GENERATOR_STOP  0x80000
+#define CO_FUTURE_ANNOTATIONS    0x100000
 
 /* This value is found in the co_cell2arg array when the associated cell
    variable does not correspond to an argument. */

Include/compile.h

@@ -16,7 +16,7 @@ PyAPI_FUNC(PyCodeObject *) PyNode_Compile(struct _node *, const char *);
 #define PyCF_MASK (CO_FUTURE_DIVISION | CO_FUTURE_ABSOLUTE_IMPORT | \
                    CO_FUTURE_WITH_STATEMENT | CO_FUTURE_PRINT_FUNCTION | \
                    CO_FUTURE_UNICODE_LITERALS | CO_FUTURE_BARRY_AS_BDFL | \
-                   CO_FUTURE_GENERATOR_STOP)
+                   CO_FUTURE_GENERATOR_STOP | CO_FUTURE_ANNOTATIONS)
 #define PyCF_MASK_OBSOLETE (CO_NESTED)
 #define PyCF_SOURCE_IS_UTF8  0x0100
 #define PyCF_DONT_IMPLY_DEDENT 0x0200
@@ -45,6 +45,7 @@ typedef struct {
 #define FUTURE_UNICODE_LITERALS "unicode_literals"
 #define FUTURE_BARRY_AS_BDFL "barry_as_FLUFL"
 #define FUTURE_GENERATOR_STOP "generator_stop"
+#define FUTURE_ANNOTATIONS "annotations"
 
 struct _mod; /* Declare the existence of this type */
 #define PyAST_Compile(mod, s, f, ar) PyAST_CompileEx(mod, s, f, -1, ar)

Lib/__future__.py

@@ -57,13 +57,14 @@ all_feature_names = [
     "unicode_literals",
     "barry_as_FLUFL",
     "generator_stop",
+    "annotations",
 ]
 
 __all__ = ["all_feature_names"] + all_feature_names
 
-# The CO_xxx symbols are defined here under the same names used by
+# The CO_xxx symbols are defined here under the same names defined in
-# compile.h, so that an editor search will find them here.  However,
+# code.h and used by compile.h, so that an editor search will find them here.
-# they're not exported in __all__, because they don't really belong to
+# However, they're not exported in __all__, because they don't really belong to
 # this module.
 CO_NESTED            = 0x0010   # nested_scopes
 CO_GENERATOR_ALLOWED = 0        # generators (obsolete, was 0x1000)
@@ -74,6 +75,7 @@ CO_FUTURE_PRINT_FUNCTION  = 0x10000   # print function
 CO_FUTURE_UNICODE_LITERALS = 0x20000 # unicode string literals
 CO_FUTURE_BARRY_AS_BDFL = 0x40000
 CO_FUTURE_GENERATOR_STOP  = 0x80000 # StopIteration becomes RuntimeError in generators
+CO_FUTURE_ANNOTATIONS     = 0x100000  # annotations become strings at runtime
 
 class _Feature:
     def __init__(self, optionalRelease, mandatoryRelease, compiler_flag):
@@ -138,3 +140,7 @@ barry_as_FLUFL = _Feature((3, 1, 0, "alpha", 2),
 generator_stop = _Feature((3, 5, 0, "beta", 1),
                           (3, 7, 0, "alpha", 0),
                           CO_FUTURE_GENERATOR_STOP)
+
+annotations = _Feature((3, 7, 0, "beta", 1),
+                       (4, 0, 0, "alpha", 0),
+                       CO_FUTURE_ANNOTATIONS)

Lib/test/test_future.py

@@ -1,7 +1,9 @@
 # Test various flavors of legal and illegal future statements
 
+from functools import partial
 import unittest
 from test import support
+from textwrap import dedent
 import os
 import re
 
@@ -102,6 +104,167 @@ class FutureTest(unittest.TestCase):
         exec("from __future__ import unicode_literals; x = ''", {}, scope)
         self.assertIsInstance(scope["x"], str)
 
+class AnnotationsFutureTestCase(unittest.TestCase):
+    template = dedent(
+        """
+        from __future__ import annotations
+        def f() -> {ann}:
+            ...
+        def g(arg: {ann}) -> None:
+            ...
+        var: {ann}
+        var2: {ann} = None
+        """
+    )
+
+    def getActual(self, annotation):
+        scope = {}
+        exec(self.template.format(ann=annotation), {}, scope)
+        func_ret_ann = scope['f'].__annotations__['return']
+        func_arg_ann = scope['g'].__annotations__['arg']
+        var_ann1 = scope['__annotations__']['var']
+        var_ann2 = scope['__annotations__']['var2']
+        self.assertEqual(func_ret_ann, func_arg_ann)
+        self.assertEqual(func_ret_ann, var_ann1)
+        self.assertEqual(func_ret_ann, var_ann2)
+        return func_ret_ann
+
+    def assertAnnotationEqual(
+        self, annotation, expected=None, drop_parens=False, is_tuple=False,
+    ):
+        actual = self.getActual(annotation)
+        if expected is None:
+            expected = annotation if not is_tuple else annotation[1:-1]
+        if drop_parens:
+            self.assertNotEqual(actual, expected)
+            actual = actual.replace("(", "").replace(")", "")
+
+        self.assertEqual(actual, expected)
+
+    def test_annotations(self):
+        eq = self.assertAnnotationEqual
+        eq('...')
+        eq("'some_string'")
+        eq("b'\\xa3'")
+        eq('Name')
+        eq('None')
+        eq('True')
+        eq('False')
+        eq('1')
+        eq('1.0')
+        eq('1j')
+        eq('True or False')
+        eq('True or False or None')
+        eq('True and False')
+        eq('True and False and None')
+        eq('(Name1 and Name2) or Name3')
+        eq('Name1 or (Name2 and Name3)')
+        eq('(Name1 and Name2) or (Name3 and Name4)')
+        eq('Name1 or (Name2 and Name3) or Name4')
+        eq('v1 << 2')
+        eq('1 >> v2')
+        eq(r'1 % finished')
+        eq('((1 + v2) - (v3 * 4)) ^ (((5 ** v6) / 7) // 8)')
+        eq('not great')
+        eq('~great')
+        eq('+value')
+        eq('-1')
+        eq('(~int) and (not ((v1 ^ (123 + v2)) | True))')
+        eq('lambda arg: None')
+        eq('lambda a=True: a')
+        eq('lambda a, b, c=True: a')
+        eq("lambda a, b, c=True, *, d=(1 << v2), e='str': a")
+        eq("lambda a, b, c=True, *vararg, d=(v1 << 2), e='str', **kwargs: a + b")
+        eq('1 if True else 2')
+        eq('(str or None) if True else (str or bytes or None)')
+        eq('(str or None) if (1 if True else 2) else (str or bytes or None)')
+        eq("{'2.7': dead, '3.7': (long_live or die_hard)}")
+        eq("{'2.7': dead, '3.7': (long_live or die_hard), **{'3.6': verygood}}")
+        eq("{**a, **b, **c}")
+        eq("{'2.7', '3.6', '3.7', '3.8', '3.9', ('4.0' if gilectomy else '3.10')}")
+        eq("({'a': 'b'}, (True or False), (+value), 'string', b'bytes') or None")
+        eq("()")
+        eq("(1,)")
+        eq("(1, 2)")
+        eq("(1, 2, 3)")
+        eq("[]")
+        eq("[1, 2, 3, 4, 5, 6, 7, 8, 9, (10 or A), (11 or B), (12 or C)]")
+        eq("{i for i in (1, 2, 3)}")
+        eq("{(i ** 2) for i in (1, 2, 3)}")
+        eq("{(i ** 2) for i, _ in ((1, 'a'), (2, 'b'), (3, 'c'))}")
+        eq("{((i ** 2) + j) for i in (1, 2, 3) for j in (1, 2, 3)}")
+        eq("[i for i in (1, 2, 3)]")
+        eq("[(i ** 2) for i in (1, 2, 3)]")
+        eq("[(i ** 2) for i, _ in ((1, 'a'), (2, 'b'), (3, 'c'))]")
+        eq("[((i ** 2) + j) for i in (1, 2, 3) for j in (1, 2, 3)]")
+        eq(r"{i: 0 for i in (1, 2, 3)}")
+        eq("{i: j for i, j in ((1, 'a'), (2, 'b'), (3, 'c'))}")
+        eq("Python3 > Python2 > COBOL")
+        eq("Life is Life")
+        eq("call()")
+        eq("call(arg)")
+        eq("call(kwarg='hey')")
+        eq("call(arg, kwarg='hey')")
+        eq("call(arg, another, kwarg='hey', **kwargs)")
+        eq("lukasz.langa.pl")
+        eq("call.me(maybe)")
+        eq("1 .real")
+        eq("1.0 .real")
+        eq("....__class__")
+        eq("list[str]")
+        eq("dict[str, int]")
+        eq("tuple[str, ...]")
+        eq("tuple[str, int, float, dict[str, int]]")
+        eq("slice[0]")
+        eq("slice[0:1]")
+        eq("slice[0:1:2]")
+        eq("slice[:]")
+        eq("slice[:-1]")
+        eq("slice[1:]")
+        eq("slice[::-1]")
+        eq('(str or None) if (sys.version_info[0] > (3,)) else (str or bytes or None)')
+        eq("f'f-string without formatted values is just a string'")
+        eq("f'{{NOT a formatted value}}'")
+        eq("f'some f-string with {a} {few():.2f} {formatted.values!r}'")
+        eq('''f"{f'{nested} inner'} outer"''')
+        eq("f'space between opening braces: { {a for a in (1, 2, 3)}}'")
+
+    def test_annotations_inexact(self):
+        """Source formatting is not always preserved
+
+        This is due to reconstruction from AST.  We *need to* put the parens
+        in nested expressions because we don't know if the source code
+        had them in the first place or not.
+        """
+        eq = partial(self.assertAnnotationEqual, drop_parens=True)
+        eq('Name1 and Name2 or Name3')
+        eq('Name1 or Name2 and Name3')
+        eq('Name1 and Name2 or Name3 and Name4')
+        eq('Name1 or Name2 and Name3 or Name4')
+        eq('1 + v2 - v3 * 4 ^ v5 ** 6 / 7 // 8')
+        eq('~int and not v1 ^ 123 + v2 | True')
+        eq('str or None if True else str or bytes or None')
+        eq("{'2.7': dead, '3.7': long_live or die_hard}")
+        eq("{'2.7', '3.6', '3.7', '3.8', '3.9', '4.0' if gilectomy else '3.10'}")
+        eq("[1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or A, 11 or B, 12 or C]")
+        # Consequently, we always drop unnecessary parens if they were given in
+        # the outer scope:
+        some_name = self.getActual("(SomeName)")
+        self.assertEqual(some_name, 'SomeName')
+        # Interestingly, in the case of tuples (and generator expressions) the
+        # parens are *required* by the Python syntax in the annotation context.
+        # But there's no point storing that detail in __annotations__ so we're
+        # fine with the parens-less form.
+        eq = partial(self.assertAnnotationEqual, is_tuple=True)
+        eq("(Good, Bad, Ugly)")
+        eq("(i for i in (1, 2, 3))")
+        eq("((i ** 2) for i in (1, 2, 3))")
+        eq("((i ** 2) for i, _ in ((1, 'a'), (2, 'b'), (3, 'c')))")
+        eq("(((i ** 2) + j) for i in (1, 2, 3) for j in (1, 2, 3))")
+        eq("(*starred)")
+        eq('(yield from outside_of_generator)')
+        eq('(yield)')
+        eq('(await some.complicated[0].call(with_args=(True or (1 is not 1))))')
 
 
 if __name__ == "__main__":

Makefile.pre.in

@@ -325,6 +325,7 @@ PYTHON_OBJS=	\
 		Python/asdl.o \
 		Python/ast.o \
 		Python/ast_opt.o \
+		Python/ast_unparse.o \
 		Python/bltinmodule.o \
 		Python/ceval.o \
 		Python/compile.o \
@@ -840,7 +841,7 @@ regen-opcode:
 		$(srcdir)/Include/opcode.h.new
 	$(UPDATE_FILE) $(srcdir)/Include/opcode.h $(srcdir)/Include/opcode.h.new
 
-Python/compile.o Python/symtable.o Python/ast.o: $(srcdir)/Include/graminit.h $(srcdir)/Include/Python-ast.h
+Python/compile.o Python/symtable.o Python/ast_unparse.o Python/ast.o: $(srcdir)/Include/graminit.h $(srcdir)/Include/Python-ast.h
 
 Python/getplatform.o: $(srcdir)/Python/getplatform.c
 		$(CC) -c $(PY_CORE_CFLAGS) -DPLATFORM='"$(MACHDEP)"' -o $@ $(srcdir)/Python/getplatform.c

PCbuild/pythoncore.vcxproj

@@ -358,6 +358,7 @@
     <ClCompile Include="..\Python\asdl.c" />
     <ClCompile Include="..\Python\ast.c" />
     <ClCompile Include="..\Python\ast_opt.c" />
+    <ClCompile Include="..\Python\ast_unparse.c" />
     <ClCompile Include="..\Python\bltinmodule.c" />
     <ClCompile Include="..\Python\bootstrap_hash.c" />
     <ClCompile Include="..\Python\ceval.c" />

PCbuild/pythoncore.vcxproj.filters

@@ -839,6 +839,9 @@
     <ClCompile Include="..\Python\ast_opt.c">
       <Filter>Python</Filter>
     </ClCompile>
+    <ClCompile Include="..\Python\ast_unparse.c">
+      <Filter>Python</Filter>
+    </ClCompile>
     <ClCompile Include="..\Python\bltinmodule.c">
       <Filter>Python</Filter>
     </ClCompile>

Python/compile.c

@@ -1699,13 +1699,30 @@ error:
     return 0;
 }
 
+static int
+compiler_visit_annexpr(struct compiler *c, expr_ty annotation)
+{
+    PyObject *ann_as_str;
+    ann_as_str = _PyAST_ExprAsUnicode(annotation, 1);
+    if (!ann_as_str) {
+        return 0;
+    }
+    ADDOP_N(c, LOAD_CONST, ann_as_str, consts);
+    return 1;
+}
+
 static int
 compiler_visit_argannotation(struct compiler *c, identifier id,
     expr_ty annotation, PyObject *names)
 {
     if (annotation) {
         PyObject *mangled;
+        if (c->c_future->ff_features & CO_FUTURE_ANNOTATIONS) {
+            VISIT(c, annexpr, annotation)
+        }
+        else {
             VISIT(c, expr, annotation);
+        }
         mangled = _Py_Mangle(c->u->u_private, id);
         if (!mangled)
             return 0;
@@ -4688,7 +4705,12 @@ compiler_annassign(struct compiler *c, stmt_ty s)
             if (!mangled) {
                 return 0;
             }
+            if (c->c_future->ff_features & CO_FUTURE_ANNOTATIONS) {
+                VISIT(c, annexpr, s->v.AnnAssign.annotation)
+            }
+            else {
                 VISIT(c, expr, s->v.AnnAssign.annotation);
+            }
             /* ADDOP_N decrefs its argument */
             ADDOP_N(c, STORE_ANNOTATION, mangled, names);
         }

Python/future.c

@@ -42,6 +42,8 @@ future_check_features(PyFutureFeatures *ff, stmt_ty s, PyObject *filename)
             ff->ff_features |= CO_FUTURE_BARRY_AS_BDFL;
         } else if (strcmp(feature, FUTURE_GENERATOR_STOP) == 0) {
             ff->ff_features |= CO_FUTURE_GENERATOR_STOP;
+        } else if (strcmp(feature, FUTURE_ANNOTATIONS) == 0) {
+            ff->ff_features |= CO_FUTURE_ANNOTATIONS;
         } else if (strcmp(feature, "braces") == 0) {
             PyErr_SetString(PyExc_SyntaxError,
                             "not a chance");