Currently, a lot of libcst node tests use data providers that use tuples
instead of dictionaries. This just makes the tests unreadable, so this
is one of several diffs aiming to solve this issue.
Refactors various metadata related classes to better reflect intended
usage of the API.
The metadata runner now deep clones the module and returns a copy
containing the metadata information. Metadata providers also return the
tree to enforce the idea that the tree is immutable (even though no
copying is actually done and providers write directly to the original
tree).
Currently, a lot of libcst node tests use data providers that use tuples
instead of dictionaries. This just makes the tests unreadable, so this
is one of several diffs aiming to solve this issue.
Previously, while `f"abc" f"def"` and `"abc" "def"` would parse as a
ConcatenatedString, `f"" ""` wouldn't because it mixes f-strings and
simple strings.
This joins `atom_string` and `atom_fstring` into a single `atom_string`
rule, which has an added benefit of simplifying the code.
Update codegen and tests to cover list, list comprehension and generator
expressions. Also cleans up some extraneous position tracking code in
codegen.
Instead of generating Pass nodes on removal, allow empty IndentedBlock,
SimpleStatementLine and SimpleStatementSuite nodes. Have them insert a
'pass' token manually when they encounter having nothing inside of
them. This allows us to more easily figure out when we should keep a
node around, and means that we no longer construct new nodes with
surprising subnodes based on somewhat-related user input. It also
matches how we do MaybeSentinel support for the rest of Python's
syntactic sugar. As a bonus (well, the point of this diff actually), we
no longer add a pass statement every time we remove another statement,
only when we actually need to.
Changes `CSTNode.visit()` to `CSTNode._visit_impl()` and uses `visit()`
as a public entry point to prevent users from running visitors with
metadata dependencies on nodes other than Modules.
Converts metadata providers to extend from either CSTVisitor or from
BatchableCSTVisitor and updates the runner to batch providers when
possible. There is a known issue in this diff where metadata may be
computed multiple times on the same pass which is addressed in
a later diff. There should be no correctness issues in this diff as
metadata computation should have no side effects.
Defines two new visitor base classes for libcst:
- `CSTTransformer` provides an visitor that can be used to create a new
tree. This is a drop in replacement for the old `CSTVisitor` in
`_base_visitor.py`.
- `CSTVisitor` provides a visitor that does not allow the user to mutate
the tree by restricting access to the updated_node in `on_leave`.
I noticed that this had some bugs when I was reading through it. Let's
fix it!
- It's safe to use a Comparison with a word operator as long as it's
first or last child is safe. Previously, this would incorrectly fail
validation on an expression like `a if(b) < (c)else d`
- We have to validate every comparator against it's next operator's
leading whitespace, not just the first comparator (`self.left`).
This adds support for
```
(a for b in c if d)
```
As well as various nested versions of the expression.
This does not include nodes for other comprehension types, they'll come
in later diffs.
Add metadata access and dependency logic in `CSTVisitor` and
`Module.visit` to generate all metadata dependencies before performing a
visit pass over a tree and validate access to metadata.
There's a lot of different places tuples can get used, and
`Element`/`StarredElement` require some tricky handling of commas, so
this took a while.
Hopefully, this work should make `List` easy to implement.
`dictsetmaker` and comprehensions are still going to be a pain though...
Along the way, I found out that `Element`/`StarredElement`'s
`whitespace_after` wasn't needed, because the tuple's parenthesis (if
they exist) or parent node would be a better owner anyways, so that's
removed in this commit.
Refactors codegen to write position information to the `__metadata__`
fields in nodes keyed by `BasicPositionProvider` and
`SyntacticPositionProvider` as defined in position_metadata.py.
This commit also updates `deep_equals` to ignore dataclass fields that
are marked `compare=False` to avoid comparing metadata when doing
equality checks.
This ports `CSTNode.validate_types_shallow` and
`CSTNode.validate_types_deep`, as well as `libcst._type_enforce` to the
open-source release.
These are useful if someone wants to use LibCST without a static type
checker.
These weren't originally included because `libcst._type_enforce` wasn't
3.7 compatible.
Converts `_codegen` methods into `_codegen_impl` to wrap implementations
to calls to update the position of each node in the `CodegenState`. The
stored position is the syntactic position of a node (that includes any
whitespace attached to that particular node).
Also updates implementation of tool and `CSTNode.__repr__` to not print
fields of `CSTNode` objects prefixed with "_".
Converts all nodes to use new CodegenState methods as defined in the
previous commit.
Ran codemods:
codemod -d libcst 'state.tokens.append\((.*)\)' 'state.add_token(\1)'
codemod -d libcst 'state.tokens.extend\(state.indent\)' 'state.add_indent_tokens()'
codemod -d libcst 'state.indent.pop()' 'state.decrease_indent()'
codemod -d libcst 'state.indent.append\((.*)\)' 'state.increase_indent(\1)'
Where `codemod` refers to https://github.com/facebook/codemod
- `pyproject.toml` is supported by isort and black, and lets us call
those tools without supplying a ton of arguments.
- `.editorconfig` is supported by a wide range of editors, and can
automatically set per-project configuration.
- `.pyre_configuration` is used by pyre.
- I added test discovery to the `setup.py` file, which required some
extra `__init__.py` files.
