There were only grammar changes between 3.5 and 3.6, which are pretty trivial to put in. So, do that trivial bit and get us completely covered for production Python 3 versions. This could use better testing, but it is good enough as-is and we can address issues with GitHub's issue tracker.
This adds the ability to subdivide the grammar into different python versions,
as well as the ability to change the tokenizer per python version. I use this
to add support for Python 3.6 since this is a supported version we run on. This
can be used for others to add support for older or newer Python versions, regardless
of tokenizer/grammar changes.
I think this is a test ordering issue, since its failing to find something in cache, but it doesn't fail on my environment. So, this should be the fix.
We've forked parso internally, so we no longer need a dependency on it. This
exposed an unused version info function that still referenced parso, so I
deleted it.
Fork both of these and convert internal LibCST stuff to point at them.
Removes the TODO from previous commit now that the tokenizer is in
complete agreement.
Brings a copy of utils.py internal to LibCST, rewriting our uses of it
to use our internal fork. A compatibility shim (__iter__) is currently
included to make PythonVersionInfo compatible with Parso directly.
This is useful when you are using `parse_expression` or `parse_statement`
to generate a tree from a source string that is meant to be later added
to an existing module. It allows you to more easily configure both of these
parser functions to output a tree which has the same defaults as the `Module`
that you previously parsed out.
`parse_statement` and `parse_expresssion` aren't guaranteed to round-trip
exactly due to the fact that they don't have a wrapping Module to encapsulate
miscelaneous spacing. So, update the tests to verify that the rendered code
is identical to the original code based on the AST. Also, while I'm at it,
bump up Hypothesis's maximums in order to stress LibCST more.
Hypothesis found that when we have a statement like `pass\r`, we detect that
`\r` is the default and parse the trailing newline as `Newline(None)`. However, when
we render the statement back out again, since we don't have a module, we construct
a default module which treats `Newline(None)` as a `\n` not a '\r'. So, when we are
parsing statements or expressions, disable auto-inferring the default newline and always
infer the default rendered newline (`\n`) so that rendering a statement/expression back
out behaves as expected.
There are a lot of nodes that cannot be removed or converted to maybes, such as
most of the Op tokens. It would be a bit of a lie to codegen leave_* methods
that allow these nodes to be converted, only to throw a runtime error later. So,
upgrade the codegen to allow us to see whether certain nodes are used in conjunction
with a MaybeSentinel/None, or inside a Sequence, to inform ourselves as to when to
allow MaybeSentinel or RemovalSentinel.
We want to make sure that the generated function stubs stay in sync with
the node definitions. So, make a unit test that fails if codegen generates
a different file than the existing file, so that somebody modifying code
knows they need to re-run codegen.
