* Fix type of evaluated_value on string
This can return bytes if the string is a bytestring, e.g.:
In [1]: import libcst as cst
In [2]: cst.parse_expression('b"foo"').evaluated_value
Out[2]: b'foo'
* Fix type errors from changed signature
* Fixes an issue where ApplyTypeAnnotationsVisitor would crash on code
like `SomeClass.some_attribute = 42` with a "Name is not a valid
identifier" error message.
* Changes the above-mentioned error message to include the bad name in
the message, for easier debugging.
* Adds tests for all valid assignment targets, as described here:
https://libcst.readthedocs.io/en/latest/nodes.html#libcst.BaseAssignTargetExpression.
* Simplify command specifier parsing
* Allow running codemods without configuring in YAML
This enables codemodding things by just plonking a CodemodCommand class
into any old importable module and running
`python -m libcst.tool codemod -x some_module.SomeClass ...`
Moves PEP 621 metadata from `setup.py` and `requirements*.txt` into the
`[project]` table of `pyproject.toml`. This enables using hatch as a
task runner for the project, where previously one would need to remember
a bunch of different commands, or repeatedly consult the readme's
developer guide to find all of the relevant commands.
This creates the following hatch commands:
- docs
- fixtures
- format
- lint
- test
- typecheck
It also updates all of the github actions workflows to use the
appropriate hatch commands, and the readme's developer guide, so that
there is only one source of truth for what constitutes running tests.
The "test" workflows now drop the matrix distinction between "pure" or
"native", and run tests in both modes from a single build.
ghstack-source-id: 8834da7825
Pull Request resolved: https://github.com/Instagram/LibCST/pull/893
* Enumeration members are singletons. Copying on them would be no-op
* Avoid generating unnecessary `pass` statement
* Several trivial refactor
* Avoid building unnecessary intermediate lists, which are mere slight waste of time and space
* Remove unused import, an overlook from commit 8e6bf9e9
* `collections.abc.Mapping.get()` defaults to return `None` when key doesn't exist
* Just use unittest's `assertRaises` to specify expected exception types, instead of catching every possible `Exception`s, which could suppress legitimate errors and hide bugs
* We know for sure that the body of `CSTTypedTransformerFunctions` won't be empty, so don't bother with complex formal completeness
* When codemod, specify the black formatter to use the same target Python version we use
* Fix the `test_codemod_formatter_error_input` unit test
* Remove an unused import in `test_codemod_cli` module
Correct the renamed import from structure when renaming last imported name from a module.
Given
from a.b import qux
print(qux)
And providing old_name="a.b.qux" and new_name="a:b.qux" I expect the
following output (as described int the command description):
from a import b
print(b.qux)
But what I get is:
from a import b.qux
print(b.qux)
It pulls the old name up into the new one.
The provided test is the important part but I've attempted a fix too. I
suspect there is a better one and that the special casing of the "this
is that last name" situation shouldn't be needed. For instance there is
import removing code in leave_Module and renaming the first of many
names (as opposed to the last) happily adds a correct import line.
I didn't manage to grok the code and all the concepts it requires to
provide a better fix though.
This leaves the alias adjustments to the existing code and just does the
module renaming the int he special casing block.
I don't know why scheduling removal of the updated node is required, it makes
the tests pass though.
* Qualify imported symbols when the dequalified form would cause a conflict.
Adds a preliminary pass that scans the stub file for all imported
symbols, and collects the ones that cannot be safely dequalified.
Fixes#673
* review fixes
* handle symbol conflicts between the stub and the main file
* fix type errors
* Always compute a module and package name
* Update name_provider to correctly support __main__ (also updated the tests to use data_provider)
* Update name_provider to correctly handle relative imports and package name
* Update relative module resolution to work on package names
* Use full_package_name in libcst.codemod.visitors.GatherImportsVisitor
* Use full_package_name in libcst.codemod.visitors.RemovedNodeVisitor
* Use full_package_name in libcst.codemod.visitors.AddImportsVisitor
* Fix failing test
* Fix typo in variable name
* PR feedback
* Force rebuild
* Support module and package names in the codemod context
* PR feedback
* Reorganize module name and relative name logic to libcst.helpers.module
* Force rebuild
* Port c3b44cb9d3
* Port 138c97cb70
* Test harness for the next commit
* Port 2cdc4ba237
* Test harness for next commit
* Port 71c5da8169
* Remove no-longer-used import
* Remove unneeded block
* Improve function name, add docstring
* Rename _is_set -> _is_non_sentinel
* Add docstring for FunctionKey
* Add class attributes with doc blocks to TypeCollector
* Extract Annotations into a single abstraction, not two
* Nits + fix flake8
* Add ApplyTrailingCommas codemod
This codemod adds trailing commas to parameter and arguments
lists when there are sufficient arguments or parameters.
The idea is this:
- both black and yapf will generally split lines when there
are trailing commas at the end of a parameter / arguments list
- It's easier on my eye to have names and types in more predictable
locations within a function header, i.e. left-aligned. And in
function calls, I also find it easier to compare arguments to
function parameters whenever the arguments are one-per line, at
least when there are more than two arguments.
By default, we ensure trailing commas for functions with one or more
parameters (but do not include `self` or `cls` method arguments) which
is suitable for `black`, and calls with 3 or more arguments.
Both the parameter count and the argument count can be overridden.
Moreover, by passing `--formatter yapf` someone can use the
yapf-suitable default of 2 parameters which is handy since then the
user doesn't have to memorize black vs yapf settings; this is necesary
because yapf does not split lines after a trailing comma in one-argument
defines.
```
> python -m unittest libcst.codemod.commands.tests.test_add_trailing_commas
......
----------------------------------------------------------------------
Ran 6 tests in 0.134s
OK
```
* Run ufmt, fix type error
* Bump argument counts down to 2
I find it difficult sometimes to read method and function signatures
when there are multiple arguments with type annotations - left-aligning
the arguments makes it much easier for me to skim and see, using mostly
my automatic visual resoning,
- the argument names
- the argument types
- the return type
Without this, I feel like I'm trying to run a parser in my head, which
is not as fast and distracts me from code-skimming.
This change was generated using thte new AddTrailingCommas codemod
(which I'll put in a separate PR) via the command
```
python -m libcst.tool codemod add_trailing_commas.AddTrailingCommas ./libcst/codemod/visitors/_apply_type_annotations.py
```
Wait for CI - this is pure formatting, it should be very safe
* Add support for methods with func type comment excluding self/cls
PEP 484 doesn't really specify carefully how function type
comments should work on methods, but since usually the type of
`self` / `cls` is automatic, most use cases choose to only annotate
the other arguments.
As a result, this commit modifies our codemod so that non-static
methods can specify either all the arguments, or all but one of
them. We'll correctly zip together the inline and func-type-comment
types either way, typically getting no type for `cls` or `self`.
We accomplish this by using matchers to trigger the visit
method for FunctionDef rather than using visit_FunctionDef, which gives
us enough context to determine when a function def is a regular
function versus a method (plus also matching the decorators against
`@staticmethod`, so that we trigger the normal function logic in
that case).
Co-authored-by: Zsolt Dollenstein <zsol.zsol@gmail.com>
* Handle syntax errors in the ast parse function.
If we encounter a syntax error in either the type comment extraction
or the type comment parsing stages, ignore type information on that
cst node.
* Quote the FunctionType type, which does not exist in Python x3.6
I've tested all of the edge cases I know of: type comments in various
locations, non-type-comments, arity mismatches where we should skip,
etc.
Assuming that all type comments parse, this should work as far as I
know. I'll make a separate PR to deal with SyntaxErrors when parsing
types, because that is cross-cutting and not specific to FunctionDef.
* Add full support type comment -> PEP 526 conversion
Summary:
In the previous PR, I added basic support for converting an
Assign with a type comment to an AnnAssign, as long as there was
only one target.
This PR handles all fully PEP 484 compliant cases:
- multiple assignments
- multiple elements in the LHS l-value
We cannot handle arity errors because there's no way to do it. And
we don't try to handle the ambiguous case of multiple assignments with
mismatched arities (PEP 484 isn't super clear on which LHS is supposed
to pick up the type, we are conservative here). The ambiguous case is
probably very uncommon in real code anyway, multiple assignment is not
a widely used feature.
Test Plan:
There are new test cases covering:
- multiple elements in the LHS
- multiple assignment
- both of the above together
- semicolon expansion, which is handled differently in the cases
where we have to add type declarations
- new error cases:
- mismatched arity in both directions on one assignment
- mismatched arity in multiple assignment
```
> python -m unittest libcst.codemod.commands.tests.test_convert_type_comments
.....
----------------------------------------------------------------------
Ran 5 tests in 0.150s
OK
```
* Tracks TypeVars that are used in type annotations in the pyi file, and
adds their Assign statements to the merged file.
* Adds Generic[T] as a base class if needed.
* Codemod for PEP 484 Assign w / type comments -> PEP 526 AnnAssign
Summary:
This codemod is intended to eventually handle all type comments from
PEP 484. This is a partial implementation specifically handling
assignment type comments, which as of PEP 526 are better dealt
with using AnnAssign nodes.
There is more work to do because there are two other kinds of
comments to support: function heading comments and function parameter
inline comments. But the PEP 526 functionality is complete so I feel
like it's worth havign a PR / CI signals / code review at this stage.
Test Plan:
```
python -m unittest libcst.codemod.commands.tests.test_convert_type_comments
```
* Disable on python 3.6, 3.7
The ast module didn't get the `type_comment` information we need
until python 3.8.
It is possible but not a priority right now to enable 3.6 and 3.7
via the typed_ast library, for now I just throw a NotImplementedError
with a nice description. There's a note in the code about where to look
for a typed_ast example in case anyone wants to add support in the
future.
* Fix type errors on the 3.8+ testing fix
* Do a better job of complaining on Python < 3.8
* Updates based on code review
Summary:
Do not strip type comments in the visitor pattern; instead,
reach down from the parent to do it because this makes it
much more reliable that we won't accidentally remove
other comments in a codemod (using visitor state to do this
isn't really feasible once we handle complex statements like
FunctionDef, With, For).
Handle multi-statement statement lines; this works since the
trailing whitespace can only apply to the final statement on
the line. It's not really a critical edge case to handle, but
the code is no more complicated so we might as well.
* Prevent comment stripping for multi-assign
* Note in the docstring that this is a limited WIP
* Reorder checks so the next step will be cleaner
* Use precise signature matching when inserting function type annotations
* add type annotations
* Add an argument for strict annotation matching.
* don't use Any
* Support relative imports in AddImportsVisitor.
* Adds an Import dataclass to represent a single imported object
* Refactors AddImportsVisitor to pass around Import objects
* Separates out the main logic in get_absolute_module_for_import so that
it can be used to resolve relative module names outside of a cst.Import
node
* Resolves relative module names in AddImportsVisitor if we have a
current module name set.
Fixes#578
This massive PR implements an alternative Python parser that will allow LibCST to parse Python 3.10's new grammar features. The parser is implemented in Rust, but it's turned off by default through the `LIBCST_PARSER_TYPE` environment variable. Set it to `native` to enable. The PR also enables new CI steps that test just the Rust parser, as well as steps that produce binary wheels for a variety of CPython versions and platforms.
Note: this PR aims to be roughly feature-equivalent to the main branch, so it doesn't include new 3.10 syntax features. That will be addressed as a follow-up PR.
The new parser is implemented in the `native/` directory, and is organized into two rust crates: `libcst_derive` contains some macros to facilitate various features of CST nodes, and `libcst` contains the `parser` itself (including the Python grammar), a `tokenizer` implementation by @bgw, and a very basic representation of CST `nodes`. Parsing is done by
1. **tokenizing** the input utf-8 string (bytes are not supported at the Rust layer, they are converted to utf-8 strings by the python wrapper)
2. running the **PEG parser** on the tokenized input, which also captures certain anchor tokens in the resulting syntax tree
3. using the anchor tokens to **inflate** the syntax tree into a proper CST
Co-authored-by: Benjamin Woodruff <github@benjam.info>
* Add ImportAssignment class and record it from Scope
* Add overrides for LocalScope and ClassScope
* Clean scope_provider code and use ImportAssignment class in `unusued_imports` codemod
* Add missing types
* Fix fixit errors
Based on diff review of https://github.com/Instagram/LibCST/pull/536,
I investigated relatvie import handling and realized that with minor
changes we can now handle them correctly.
Relative imports aren't likely in code coming from an automated
tool, but they could happen in hand-written stubs if anyone tries
to use this codemod tool to merge stubs with code.
Added a new test:
```
> python -m unittest libcst.codemod.visitors.tests.test_apply_type_annotations
.............................................
----------------------------------------------------------------------
Ran 45 tests in 2.195s
OK
```
