* Add support for methods with func type comment excluding self/cls
PEP 484 doesn't really specify carefully how function type
comments should work on methods, but since usually the type of
`self` / `cls` is automatic, most use cases choose to only annotate
the other arguments.
As a result, this commit modifies our codemod so that non-static
methods can specify either all the arguments, or all but one of
them. We'll correctly zip together the inline and func-type-comment
types either way, typically getting no type for `cls` or `self`.
We accomplish this by using matchers to trigger the visit
method for FunctionDef rather than using visit_FunctionDef, which gives
us enough context to determine when a function def is a regular
function versus a method (plus also matching the decorators against
`@staticmethod`, so that we trigger the normal function logic in
that case).
Co-authored-by: Zsolt Dollenstein <zsol.zsol@gmail.com>
* Handle syntax errors in the ast parse function.
If we encounter a syntax error in either the type comment extraction
or the type comment parsing stages, ignore type information on that
cst node.
* Quote the FunctionType type, which does not exist in Python x3.6
I've tested all of the edge cases I know of: type comments in various
locations, non-type-comments, arity mismatches where we should skip,
etc.
Assuming that all type comments parse, this should work as far as I
know. I'll make a separate PR to deal with SyntaxErrors when parsing
types, because that is cross-cutting and not specific to FunctionDef.
* Add full support type comment -> PEP 526 conversion
Summary:
In the previous PR, I added basic support for converting an
Assign with a type comment to an AnnAssign, as long as there was
only one target.
This PR handles all fully PEP 484 compliant cases:
- multiple assignments
- multiple elements in the LHS l-value
We cannot handle arity errors because there's no way to do it. And
we don't try to handle the ambiguous case of multiple assignments with
mismatched arities (PEP 484 isn't super clear on which LHS is supposed
to pick up the type, we are conservative here). The ambiguous case is
probably very uncommon in real code anyway, multiple assignment is not
a widely used feature.
Test Plan:
There are new test cases covering:
- multiple elements in the LHS
- multiple assignment
- both of the above together
- semicolon expansion, which is handled differently in the cases
where we have to add type declarations
- new error cases:
- mismatched arity in both directions on one assignment
- mismatched arity in multiple assignment
```
> python -m unittest libcst.codemod.commands.tests.test_convert_type_comments
.....
----------------------------------------------------------------------
Ran 5 tests in 0.150s
OK
```
* Tracks TypeVars that are used in type annotations in the pyi file, and
adds their Assign statements to the merged file.
* Adds Generic[T] as a base class if needed.
* Codemod for PEP 484 Assign w / type comments -> PEP 526 AnnAssign
Summary:
This codemod is intended to eventually handle all type comments from
PEP 484. This is a partial implementation specifically handling
assignment type comments, which as of PEP 526 are better dealt
with using AnnAssign nodes.
There is more work to do because there are two other kinds of
comments to support: function heading comments and function parameter
inline comments. But the PEP 526 functionality is complete so I feel
like it's worth havign a PR / CI signals / code review at this stage.
Test Plan:
```
python -m unittest libcst.codemod.commands.tests.test_convert_type_comments
```
* Disable on python 3.6, 3.7
The ast module didn't get the `type_comment` information we need
until python 3.8.
It is possible but not a priority right now to enable 3.6 and 3.7
via the typed_ast library, for now I just throw a NotImplementedError
with a nice description. There's a note in the code about where to look
for a typed_ast example in case anyone wants to add support in the
future.
* Fix type errors on the 3.8+ testing fix
* Do a better job of complaining on Python < 3.8
* Updates based on code review
Summary:
Do not strip type comments in the visitor pattern; instead,
reach down from the parent to do it because this makes it
much more reliable that we won't accidentally remove
other comments in a codemod (using visitor state to do this
isn't really feasible once we handle complex statements like
FunctionDef, With, For).
Handle multi-statement statement lines; this works since the
trailing whitespace can only apply to the final statement on
the line. It's not really a critical edge case to handle, but
the code is no more complicated so we might as well.
* Prevent comment stripping for multi-assign
* Note in the docstring that this is a limited WIP
* Reorder checks so the next step will be cleaner
* Use precise signature matching when inserting function type annotations
* add type annotations
* Add an argument for strict annotation matching.
* don't use Any
* Support relative imports in AddImportsVisitor.
* Adds an Import dataclass to represent a single imported object
* Refactors AddImportsVisitor to pass around Import objects
* Separates out the main logic in get_absolute_module_for_import so that
it can be used to resolve relative module names outside of a cst.Import
node
* Resolves relative module names in AddImportsVisitor if we have a
current module name set.
Fixes#578
On the python side, we can add parentheses from MaybeSentinel.DEFAULT if the whitespace requires it.
On the rust side, we support the new grammar but codegen will only add explicitly included parentheses for now - it should be possible to match python behavior but it's not urgent so I've left a TODO
It took me some time to track down the root cause of `children`
not matching codegen, having the error message directly hint that
visit and codegen are probably mimatched (my visit was running
out-of-order) will likely help newbies get going faster.
* ParenthesizedNode implementation for Box
* match statement rust CST and grammar
* match statement python CST and docs
* run rust unit tests in release mode for now
This massive PR implements an alternative Python parser that will allow LibCST to parse Python 3.10's new grammar features. The parser is implemented in Rust, but it's turned off by default through the `LIBCST_PARSER_TYPE` environment variable. Set it to `native` to enable. The PR also enables new CI steps that test just the Rust parser, as well as steps that produce binary wheels for a variety of CPython versions and platforms.
Note: this PR aims to be roughly feature-equivalent to the main branch, so it doesn't include new 3.10 syntax features. That will be addressed as a follow-up PR.
The new parser is implemented in the `native/` directory, and is organized into two rust crates: `libcst_derive` contains some macros to facilitate various features of CST nodes, and `libcst` contains the `parser` itself (including the Python grammar), a `tokenizer` implementation by @bgw, and a very basic representation of CST `nodes`. Parsing is done by
1. **tokenizing** the input utf-8 string (bytes are not supported at the Rust layer, they are converted to utf-8 strings by the python wrapper)
2. running the **PEG parser** on the tokenized input, which also captures certain anchor tokens in the resulting syntax tree
3. using the anchor tokens to **inflate** the syntax tree into a proper CST
Co-authored-by: Benjamin Woodruff <github@benjam.info>
