This adds support for
```
(a for b in c if d)
```
As well as various nested versions of the expression.
This does not include nodes for other comprehension types, they'll come
in later diffs.
Add metadata access and dependency logic in `CSTVisitor` and
`Module.visit` to generate all metadata dependencies before performing a
visit pass over a tree and validate access to metadata.
There's a lot of different places tuples can get used, and
`Element`/`StarredElement` require some tricky handling of commas, so
this took a while.
Hopefully, this work should make `List` easy to implement.
`dictsetmaker` and comprehensions are still going to be a pain though...
Along the way, I found out that `Element`/`StarredElement`'s
`whitespace_after` wasn't needed, because the tuple's parenthesis (if
they exist) or parent node would be a better owner anyways, so that's
removed in this commit.
Refactors codegen to write position information to the `__metadata__`
fields in nodes keyed by `BasicPositionProvider` and
`SyntacticPositionProvider` as defined in position_metadata.py.
This commit also updates `deep_equals` to ignore dataclass fields that
are marked `compare=False` to avoid comparing metadata when doing
equality checks.
This ports `CSTNode.validate_types_shallow` and
`CSTNode.validate_types_deep`, as well as `libcst._type_enforce` to the
open-source release.
These are useful if someone wants to use LibCST without a static type
checker.
These weren't originally included because `libcst._type_enforce` wasn't
3.7 compatible.
Converts `_codegen` methods into `_codegen_impl` to wrap implementations
to calls to update the position of each node in the `CodegenState`. The
stored position is the syntactic position of a node (that includes any
whitespace attached to that particular node).
Also updates implementation of tool and `CSTNode.__repr__` to not print
fields of `CSTNode` objects prefixed with "_".
Converts all nodes to use new CodegenState methods as defined in the
previous commit.
Ran codemods:
codemod -d libcst 'state.tokens.append\((.*)\)' 'state.add_token(\1)'
codemod -d libcst 'state.tokens.extend\(state.indent\)' 'state.add_indent_tokens()'
codemod -d libcst 'state.indent.pop()' 'state.decrease_indent()'
codemod -d libcst 'state.indent.append\((.*)\)' 'state.increase_indent(\1)'
Where `codemod` refers to https://github.com/facebook/codemod
- `pyproject.toml` is supported by isort and black, and lets us call
those tools without supplying a ton of arguments.
- `.editorconfig` is supported by a wide range of editors, and can
automatically set per-project configuration.
- `.pyre_configuration` is used by pyre.
- I added test discovery to the `setup.py` file, which required some
extra `__init__.py` files.
@jreese had some suggestions in my previous pull request after it got
merged, so this addresses some of those suggestions:
- Uses the PEP 508 `python_version` environment marker instead of
conditional logic inside `setup.py`. I've updated `requirements.txt`
to use this too.
- Split dev requirements into a separate `requirements-dev.txt`, and
updated the README to include instructions for it.
This PR does not use pyup, because it looks like it's free for
non-commercial use only (I don't know that that means in this context),
and because this project isn't public yet.
It also does not include a makefile yet, because Jennifer and I need to
talk through where we'd stick the virtualenv in that case.
I tested these changes on 3.6 and 3.7.
- Reorganized the docs a bit to separate "getting started" from
"development".
- Added a section about setting up a virtualenv (and tested that
workflow on my machine).
- Added isort, pyre-check, and black to the `requirements.txt`, and
updated the `requirements.txt` to include exact versions so that we
can be sure that our tools will behave the same (e.g. there are
frequently breaking changes between releases of pyre).
- Added a `dev` section to `setup.py`'s `extras_require`. This allows
you to `pip install -e '.[dev]'` to get a set of development packages
installed. I used this to help build the updated `requirements.txt`.
This appears to be a pattern that other projects use.
- Add `libcst/__init__.py` back which I accidentally deleted in another
commit.
- Add `*.egg-info/` to the gitignore, because `libcst.egg-info` is it's
created by pip/setuptools when locally installing libcst, and it's
annoying.
- Changed the version number from `0.1.dev` to `0.1.dev0`, since pip was
warning that it was normalizing the version number from the former to
the later.
- Add a `python_requires` field, since we know that libcst only works on
3.6+.
- Add an `install_requires`. Pip uses this to find dependencies, and
ignores `requirements.txt` (since `requirements.txt` is really only
intended to be a freeze file).
- Add the dataclasses backport as a dependency for Python 3.6. I
validated that installing and using libcst works in both 3.6 and 3.7.
**Test Plan:**
```
$ python3 -m venv libcst-install-test # my system python is 3.7
$ libcst-install-test/bin/pip install --upgrade pip ipython
Cache entry deserialization failed, entry ignored
Collecting pip
Using cached
be401c0032/pip-19.1.1-py2.py3-none-any.whl
Collecting ipython
... # lots of output
$ ~/libcst-install-test/bin/pip install ~/libcst/
Processing ./libcst
Requirement already satisfied: parso in
./libcst-install-test/lib/python3.7/site-packages (from
libcst==0.1.dev0) (0.4.0)
Collecting typing_extensions (from libcst==0.1.dev0)
Using cached
c66e553258/typing_extensions-3.7.2-py3-none-any.whl
Installing collected packages: typing-extensions, libcst
Running setup.py install for libcst ... done
Successfully installed libcst-0.1.dev0 typing-extensions-3.7.2
$ ~/libcst-install-test/bin/ipython
Python 3.7.3 (default, Apr 3 2019, 05:39:12)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from libcst import parser
In [2]: parser.parse_expression("None")
Out[2]:
Name(
value='None',
lpar=[],
rpar=[],
)
In [3]:
```
I then repeated the same with a copy of CPython 3.6 that I built from
source.
This ended up being pretty complicated, so the parser stuff will come in
another diff.
Hopefully this should set up up nicely for dicts, sets, and lists too.
