Issue #24619: Simplify async/await tokenization.

This commit simplifies async/await tokenization in tokenizer.c,
tokenize.py & lib2to3/tokenize.py.  Previous solution was to keep
a stack of async-def & def blocks, whereas the new approach is just
to remember position of the outermost async-def block.

This change won't bring any parsing performance improvements, but
it makes the code much easier to read and validate.
This commit is contained in:
Yury Selivanov 2015-07-23 15:01:58 +03:00
parent f315c1c016
commit 96ec934e75
7 changed files with 183 additions and 132 deletions

View file

@ -67,10 +67,32 @@ class TestAsyncAwait(GrammarTest):
await x
""")
self.validate("""async def foo():
def foo(): pass
def foo(): pass
await x
""")
self.validate("""async def foo(): return await a""")
self.validate("""def foo():
def foo(): pass
async def foo(): await x
""")
self.invalid_syntax("await x")
self.invalid_syntax("""def foo():
await x""")
self.invalid_syntax("""def foo():
def foo(): pass
async def foo(): pass
await x
""")
def test_async_var(self):
self.validate("""async = 1""")
self.validate("""await = 1""")