Instead of sharing instances of a Codemod across many files, this PR allows passing in a Codemod class to `parallel_exec_transform_with_prettyprint` which will then instantiate the Codemod for each file. `tool._codemod_impl` now starts using this API.
The old behavior is deprecated, because sharing codemod instances across files is a surprising behavior, and causes hard-to-diagnose bugs when a Codemod keeps track of its state via instance variables.
Fixes#1160.
This PR also
- fixes `whitespace_before_colon` being swallowed during visitation on `MatchCase`s
- adds a new type of roundtrip test that catches issues of this class: the test applies a noop transformer to exercise the visitation API and compares the result with the original source.
- adds a few more cases to the match fixture
#453 fixed scratch leaking between files by setting it to empty, but that drops all the scratch space that was set up before the codemod runs (e.g. in the transformer's constructor)
This PR improves the fix by preserving the initial scratch.
* Make the nodes fields filtering process - from libcst.tool - public, so that other libraries may provide their own custom representation of LibCST graphs.
* Create functions to access & filter CST-node fields (with appropriate docstrings & tests), in libcst.helpers
* Add new CST-node fields functions to helpers documentation.
Summary:
This PR removes the `typing_extensions` and `typing_inspect` dependencies as we can now rely on the built-in `typing` module since Python 3.9.
Test Plan:
existing tests
This PR adds support for parsing and representing Type Parameters and Type Aliases as specified by PEP 695. What's missing are the scope rules, to be implemented in a future PR.
Notable (user visible) changes:
- new `TypeAlias` CST node, which is a `SmallStatement`
- new CST nodes to represent TypeVarLikes: `TypeVar`, `TypeVarTuple`, `ParamSpec`
- new helper CST nodes: `TypeParameters` to serve as a container for multiple TypeVarLikes, and `TypeParam` which is a single item in a `TypeParameters` (owning the separating comma)
- extended `FunctionDef` and `ClassDef` with an optional `type_parameters` field, as well as `whitespace_after_type_parameters` to own the extra whitespace between type parameters and the following token
- these new fields are added after all others to avoid breaking callers passing in fields as positional arguments
- in `FunctionDef` and `ClassDef`, `whitespace_after_name` now owns the whitespace before the type parameters if they exist
Moves PEP 621 metadata from `setup.py` and `requirements*.txt` into the
`[project]` table of `pyproject.toml`. This enables using hatch as a
task runner for the project, where previously one would need to remember
a bunch of different commands, or repeatedly consult the readme's
developer guide to find all of the relevant commands.
This creates the following hatch commands:
- docs
- fixtures
- format
- lint
- test
- typecheck
It also updates all of the github actions workflows to use the
appropriate hatch commands, and the readme's developer guide, so that
there is only one source of truth for what constitutes running tests.
The "test" workflows now drop the matrix distinction between "pure" or
"native", and run tests in both modes from a single build.
ghstack-source-id: 8834da7825
Pull Request resolved: https://github.com/Instagram/LibCST/pull/893
Upgrading Pyre requires updating test fixtures with any upstream changes
to Pyre's query results for the `simple_class.py` fixture.
This adds a new `scripts/` directory to the repo, with a script to
regenerate test fixtures. The script regenerates the cache data fixture,
and updates the `TypeInferenceProvider` tests to use `assertDictEqual`
and helpful error messages for better behavior in future mismatches.
This also includes a slight bump to Pyre 0.9.10 to fix install issues on
Apple Silicon M1 Macs, and regenerated fixtures using the script above.
This massive PR implements an alternative Python parser that will allow LibCST to parse Python 3.10's new grammar features. The parser is implemented in Rust, but it's turned off by default through the `LIBCST_PARSER_TYPE` environment variable. Set it to `native` to enable. The PR also enables new CI steps that test just the Rust parser, as well as steps that produce binary wheels for a variety of CPython versions and platforms.
Note: this PR aims to be roughly feature-equivalent to the main branch, so it doesn't include new 3.10 syntax features. That will be addressed as a follow-up PR.
The new parser is implemented in the `native/` directory, and is organized into two rust crates: `libcst_derive` contains some macros to facilitate various features of CST nodes, and `libcst` contains the `parser` itself (including the Python grammar), a `tokenizer` implementation by @bgw, and a very basic representation of CST `nodes`. Parsing is done by
1. **tokenizing** the input utf-8 string (bytes are not supported at the Rust layer, they are converted to utf-8 strings by the python wrapper)
2. running the **PEG parser** on the tokenized input, which also captures certain anchor tokens in the resulting syntax tree
3. using the anchor tokens to **inflate** the syntax tree into a proper CST
Co-authored-by: Benjamin Woodruff <github@benjam.info>
which ensures we won't have inconsistent black-vs-isort errors
going forward. We can always format by running `ufmt format .`
at the root, and check with `ufmt check .` in our CI actions.
* Read install requirements from requirements.txt
* read extras_require from requirements-dev.txt
* add requirements-dev.txt to MANIFEST.in
* apply fixes for new version of Black and Flake8
* don't upgrade Pyre
* re-format
It seems that @mark_no_op was not correctly preserving type signatures
in pyre previous to this change. Preserving type signatures in the
decorator caused some pyre errors in tests where visitors are defined
that don't share the same arg names as their parent.
Let's split from just fuzzing on the default python to having a separate fuzzer for each version of python we support. This should help us catch any issues should they arise in differences with grammars across releases. Right now we're fuzz-clean as well, so I also bumped up the thresholds to stress test LibCST more. I anticipate that this will be useful when we begin work on 3.8 support.
This makes sure we always wrap elements in a SubscriptElement, even when there
is only one element. This makes things more regular while still being backwards
compatible with existing creation. The meat of this is in two halves, which can't
be split due to not wanting to break the build between commits. The first half
is just the changes to the parser and updates to tests. This includes a test to
be sure we can still render code that uses old construction types. The second half
is changes to codegen which made assumptions about `Subscript` and demonstrates
the need to make this change in the first place. This includes a fix to
`CSTNode.with_deep_changes` type to make it more correct and also more usable in
transforms without additional type assertions.
