Currently, a lot of libcst node tests use data providers that use tuples
instead of dictionaries. This just makes the tests unreadable, so this
is one of several diffs aiming to solve this issue.
Refactors various metadata related classes to better reflect intended
usage of the API.
The metadata runner now deep clones the module and returns a copy
containing the metadata information. Metadata providers also return the
tree to enforce the idea that the tree is immutable (even though no
copying is actually done and providers write directly to the original
tree).
Currently, a lot of libcst node tests use data providers that use tuples
instead of dictionaries. This just makes the tests unreadable, so this
is one of several diffs aiming to solve this issue.
Previously, while `f"abc" f"def"` and `"abc" "def"` would parse as a
ConcatenatedString, `f"" ""` wouldn't because it mixes f-strings and
simple strings.
This joins `atom_string` and `atom_fstring` into a single `atom_string`
rule, which has an added benefit of simplifying the code.
Update codegen and tests to cover list, list comprehension and generator
expressions. Also cleans up some extraneous position tracking code in
codegen.
Instead of generating Pass nodes on removal, allow empty IndentedBlock,
SimpleStatementLine and SimpleStatementSuite nodes. Have them insert a
'pass' token manually when they encounter having nothing inside of
them. This allows us to more easily figure out when we should keep a
node around, and means that we no longer construct new nodes with
surprising subnodes based on somewhat-related user input. It also
matches how we do MaybeSentinel support for the rest of Python's
syntactic sugar. As a bonus (well, the point of this diff actually), we
no longer add a pass statement every time we remove another statement,
only when we actually need to.
Changes `CSTNode.visit()` to `CSTNode._visit_impl()` and uses `visit()`
as a public entry point to prevent users from running visitors with
metadata dependencies on nodes other than Modules.
Converts metadata providers to extend from either CSTVisitor or from
BatchableCSTVisitor and updates the runner to batch providers when
possible. There is a known issue in this diff where metadata may be
computed multiple times on the same pass which is addressed in
a later diff. There should be no correctness issues in this diff as
metadata computation should have no side effects.
Defines two new visitor base classes for libcst:
- `CSTTransformer` provides an visitor that can be used to create a new
tree. This is a drop in replacement for the old `CSTVisitor` in
`_base_visitor.py`.
- `CSTVisitor` provides a visitor that does not allow the user to mutate
the tree by restricting access to the updated_node in `on_leave`.
I noticed that this had some bugs when I was reading through it. Let's
fix it!
- It's safe to use a Comparison with a word operator as long as it's
first or last child is safe. Previously, this would incorrectly fail
validation on an expression like `a if(b) < (c)else d`
- We have to validate every comparator against it's next operator's
leading whitespace, not just the first comparator (`self.left`).
This adds support for
```
(a for b in c if d)
```
As well as various nested versions of the expression.
This does not include nodes for other comprehension types, they'll come
in later diffs.
Add metadata access and dependency logic in `CSTVisitor` and
`Module.visit` to generate all metadata dependencies before performing a
visit pass over a tree and validate access to metadata.
There's a lot of different places tuples can get used, and
`Element`/`StarredElement` require some tricky handling of commas, so
this took a while.
Hopefully, this work should make `List` easy to implement.
`dictsetmaker` and comprehensions are still going to be a pain though...
Along the way, I found out that `Element`/`StarredElement`'s
`whitespace_after` wasn't needed, because the tuple's parenthesis (if
they exist) or parent node would be a better owner anyways, so that's
removed in this commit.
Refactors codegen to write position information to the `__metadata__`
fields in nodes keyed by `BasicPositionProvider` and
`SyntacticPositionProvider` as defined in position_metadata.py.
This commit also updates `deep_equals` to ignore dataclass fields that
are marked `compare=False` to avoid comparing metadata when doing
equality checks.
This ports `CSTNode.validate_types_shallow` and
`CSTNode.validate_types_deep`, as well as `libcst._type_enforce` to the
open-source release.
These are useful if someone wants to use LibCST without a static type
checker.
These weren't originally included because `libcst._type_enforce` wasn't
3.7 compatible.
Converts `_codegen` methods into `_codegen_impl` to wrap implementations
to calls to update the position of each node in the `CodegenState`. The
stored position is the syntactic position of a node (that includes any
whitespace attached to that particular node).
Also updates implementation of tool and `CSTNode.__repr__` to not print
fields of `CSTNode` objects prefixed with "_".
Converts all nodes to use new CodegenState methods as defined in the
previous commit.
Ran codemods:
codemod -d libcst 'state.tokens.append\((.*)\)' 'state.add_token(\1)'
codemod -d libcst 'state.tokens.extend\(state.indent\)' 'state.add_indent_tokens()'
codemod -d libcst 'state.indent.pop()' 'state.decrease_indent()'
codemod -d libcst 'state.indent.append\((.*)\)' 'state.increase_indent(\1)'
Where `codemod` refers to https://github.com/facebook/codemod
- `pyproject.toml` is supported by isort and black, and lets us call
those tools without supplying a ton of arguments.
- `.editorconfig` is supported by a wide range of editors, and can
automatically set per-project configuration.
- `.pyre_configuration` is used by pyre.
- I added test discovery to the `setup.py` file, which required some
extra `__init__.py` files.
- Add `libcst/__init__.py` back which I accidentally deleted in another
commit.
- Add `*.egg-info/` to the gitignore, because `libcst.egg-info` is it's
created by pip/setuptools when locally installing libcst, and it's
annoying.
- Changed the version number from `0.1.dev` to `0.1.dev0`, since pip was
warning that it was normalizing the version number from the former to
the later.
- Add a `python_requires` field, since we know that libcst only works on
3.6+.
- Add an `install_requires`. Pip uses this to find dependencies, and
ignores `requirements.txt` (since `requirements.txt` is really only
intended to be a freeze file).
- Add the dataclasses backport as a dependency for Python 3.6. I
validated that installing and using libcst works in both 3.6 and 3.7.
**Test Plan:**
```
$ python3 -m venv libcst-install-test # my system python is 3.7
$ libcst-install-test/bin/pip install --upgrade pip ipython
Cache entry deserialization failed, entry ignored
Collecting pip
Using cached
be401c0032/pip-19.1.1-py2.py3-none-any.whl
Collecting ipython
... # lots of output
$ ~/libcst-install-test/bin/pip install ~/libcst/
Processing ./libcst
Requirement already satisfied: parso in
./libcst-install-test/lib/python3.7/site-packages (from
libcst==0.1.dev0) (0.4.0)
Collecting typing_extensions (from libcst==0.1.dev0)
Using cached
c66e553258/typing_extensions-3.7.2-py3-none-any.whl
Installing collected packages: typing-extensions, libcst
Running setup.py install for libcst ... done
Successfully installed libcst-0.1.dev0 typing-extensions-3.7.2
$ ~/libcst-install-test/bin/ipython
Python 3.7.3 (default, Apr 3 2019, 05:39:12)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from libcst import parser
In [2]: parser.parse_expression("None")
Out[2]:
Name(
value='None',
lpar=[],
rpar=[],
)
In [3]:
```
I then repeated the same with a copy of CPython 3.6 that I built from
source.
This ended up being pretty complicated, so the parser stuff will come in
another diff.
Hopefully this should set up up nicely for dicts, sets, and lists too.
