* Add Python 3.9 to tox envlist
* Require newer typing_extensions for 3.9
For simplicity, use the new version in all cases.
* Improve default-version selection to work on 3.9
While were at it, improve the code to work with a likely 3.10 by
allowing multiple digits for minor version.
Several of the python 2 features are gated on these in addition to
version (like `with_statement`), and a refactoring tool like Bowler
commonly needs this information anyway.
If you have such a program like "pass\\\n", this is technically a program without a trailing newline, since line continuations are defined as being a `\` followed by a newline. We were misdetecting this as having a trailing newline, thus making it impossible to parse the continuation. Add some tests to verify this behavior and then fix the problem.
Note that this was found via hypothesis.
This makes sure we always wrap elements in a SubscriptElement, even when there
is only one element. This makes things more regular while still being backwards
compatible with existing creation. The meat of this is in two halves, which can't
be split due to not wanting to break the build between commits. The first half
is just the changes to the parser and updates to tests. This includes a test to
be sure we can still render code that uses old construction types. The second half
is changes to codegen which made assumptions about `Subscript` and demonstrates
the need to make this change in the first place. This includes a fix to
`CSTNode.with_deep_changes` type to make it more correct and also more usable in
transforms without additional type assertions.
This is somewhat complicated by the fact that we need to not just allow
construction of nodes/matchers using `ExtSlice` still for backwards compatibility,
but we also need to be able to call `visit_ExtSlice` and `leave_ExtSlice` on
old visitors even though the new node is named `SubscriptElement`. The
construction/instance check/matching side of things will work since internally we
refer to everything as `SubscriptElement` and alias `ExtSlice` to this everywhere,
but for string-based function lookup, we need to get a little more clever and make
the default `visit_SubscriptElement` delegate onward to `visit_ExtSlice` so that
either form works.
This can all be removed again once we're past the deprecation period for ExtSlice.
This introduces a few new gotchas (namely attribute access and a bug with
Union[Callable]), but it also removes a whole host of pyre-fixmes and gets us
updated to a release of pyre that came out after May.
Standardize on the convention that private modules (those we don't expect people to directly import) are prefixed with an underscore. Everything under a directory/module that has an underscore is considered private, unless it is re-exported from a non-underscored module. Most things are exported from libcst directly, but there are a few things in libcst.tool, libcst.codegen and libcst.metadata that are namedspaced as such.
There were only grammar changes between 3.5 and 3.6, which are pretty trivial to put in. So, do that trivial bit and get us completely covered for production Python 3 versions. This could use better testing, but it is good enough as-is and we can address issues with GitHub's issue tracker.
This adds the ability to subdivide the grammar into different python versions,
as well as the ability to change the tokenizer per python version. I use this
to add support for Python 3.6 since this is a supported version we run on. This
can be used for others to add support for older or newer Python versions, regardless
of tokenizer/grammar changes.
I think this is a test ordering issue, since its failing to find something in cache, but it doesn't fail on my environment. So, this should be the fix.
We've forked parso internally, so we no longer need a dependency on it. This
exposed an unused version info function that still referenced parso, so I
deleted it.
Fork both of these and convert internal LibCST stuff to point at them.
Removes the TODO from previous commit now that the tokenizer is in
complete agreement.
Brings a copy of utils.py internal to LibCST, rewriting our uses of it
to use our internal fork. A compatibility shim (__iter__) is currently
included to make PythonVersionInfo compatible with Parso directly.
Hypothesis found that when we have a statement like `pass\r`, we detect that
`\r` is the default and parse the trailing newline as `Newline(None)`. However, when
we render the statement back out again, since we don't have a module, we construct
a default module which treats `Newline(None)` as a `\n` not a '\r'. So, when we are
parsing statements or expressions, disable auto-inferring the default newline and always
infer the default rendered newline (`\n`) so that rendering a statement/expression back
out behaves as expected.
Because the parser conversion functions may not understand their current
position in the code, we instead need to construct a `ParserSyntaxError`
without the current line and column information, filling it in later.
Originally I did this by allowing ParserSyntaxError to be partially
initialized, but this wasn't very type-safe, and required some ugly
assertions.
Instead, I now construct and raise a PartialParserSyntaxError. The
BaseParser class is responsible for catching that partial error object
and constructing a full ParserSyntaxError.
For anything that's not an internal logic error, conversion functions
should raise a ParserSyntaxError.
Internal logic errors should probably use an AssertionError or an assert
statement, but that's not as important and is out of scope for this PR.
This removes the hard-coded logic about encountered/expected, and moves
it into a separate helper method.
Line and column can now be initialized lazily. We'll use this later to
raise `ParserSyntaxError`s inside of conversion functions, backfilling
the positions inside `_base_parser`, since conversion functions don't
always have access to position information.
Adds details about how various configuration options can be used, and
updates some details to reflect reality (e.g. we don't currently infer
the python version from your execution environment).
We only support parsing code as 3.7 right now.
This raises a descriptive error message if we receive an unsupported
version number, instead of silently trying to use it with the tokenizer
and failing in potentially strange ways.
We pass down the correct default annotation indicator to use in 100% of code rendering places, so this becomes a useless bit of initialization. We already set this to a sentinel by default, and the only thing that having an explicit str gives us is the inability to copy an annotation from a param to a return or vice versa. So, out it goes. This means we can't render Annotation by itself, so the test that was using this behavior is out too.
