This PR adds support for parsing and representing Type Parameters and Type Aliases as specified by PEP 695. What's missing are the scope rules, to be implemented in a future PR.
Notable (user visible) changes:
- new `TypeAlias` CST node, which is a `SmallStatement`
- new CST nodes to represent TypeVarLikes: `TypeVar`, `TypeVarTuple`, `ParamSpec`
- new helper CST nodes: `TypeParameters` to serve as a container for multiple TypeVarLikes, and `TypeParam` which is a single item in a `TypeParameters` (owning the separating comma)
- extended `FunctionDef` and `ClassDef` with an optional `type_parameters` field, as well as `whitespace_after_type_parameters` to own the extra whitespace between type parameters and the following token
- these new fields are added after all others to avoid breaking callers passing in fields as positional arguments
- in `FunctionDef` and `ClassDef`, `whitespace_after_name` now owns the whitespace before the type parameters if they exist
* ParenthesizedNode implementation for Box
* match statement rust CST and grammar
* match statement python CST and docs
* run rust unit tests in release mode for now
which ensures we won't have inconsistent black-vs-isort errors
going forward. We can always format by running `ufmt format .`
at the root, and check with `ufmt check .` in our CI actions.
* Use setuptools-scm to derive the current version from git metadata
* Add Github Action equivalent to the current circleci tasks
* Run pyre integration test in GH action / tox
* Add flatten_sentinal
* Add FlattenSentinal to __all__
* Fix lint errors
* Fix type errors
* Update test to use leave_Return
* Update and run codegen
* Add empty test
* Update docs
* autofix
While these classes are used by the codegen implementation, conceptually
they're part of `libcst.metadata`, so we should export them from
`libcst.metadata` instead of the top-level `libcst` package.
This is somewhat complicated by the fact that we need to not just allow
construction of nodes/matchers using `ExtSlice` still for backwards compatibility,
but we also need to be able to call `visit_ExtSlice` and `leave_ExtSlice` on
old visitors even though the new node is named `SubscriptElement`. The
construction/instance check/matching side of things will work since internally we
refer to everything as `SubscriptElement` and alias `ExtSlice` to this everywhere,
but for string-based function lookup, we need to get a little more clever and make
the default `visit_SubscriptElement` delegate onward to `visit_ExtSlice` so that
either form works.
This can all be removed again once we're past the deprecation period for ExtSlice.
Add a RemoveFromParent() function as a convenience to returning RemovalSentinel.REMOVE.
Introduce a `deep_remove()` on CSTNode analogous to `deep_replace()` but for removing.
Standardize on the convention that private modules (those we don't expect people to directly import) are prefixed with an underscore. Everything under a directory/module that has an underscore is considered private, unless it is re-exported from a non-underscored module. Most things are exported from libcst directly, but there are a few things in libcst.tool, libcst.codegen and libcst.metadata that are namedspaced as such.
This commit mostly focuses on f-strings, collections, comprehensions,
and subscripts/slices.
I added intersphinx support to the sphinx config so I could link to
`ast.literal_eval`.
I exported BaseSimpleComp, since I felt it had some documentation value.
Groups the expression nodes into much smaller subcategories for improved
readablility.
Some nodes defined in the expression module were grouped with statements
because they made more sense there (from a documentation perspective).
This also adds a local table of contents to the top of the nodes page
which better reveals all of the subcategories.
It also exports and categorizes
`BaseAssignTargetExpression`/`BaseDelTargetExpression`. These nodes were
referenced in type annotations of other nodes, so they need to be
exported if only for documentation reasons.
I added this originally because it's part of Python's grammar, but since
we merged parenthesis and friends into expressions instead of forming
separate nodes, BaseAtom doesn't have much use.
This fixes also a few bugs:
- A decorator requires a `dotted_name`, so it can only take a `Name` or
an `Attribute` node, not any atom.
- A `Call` node's function can be any expression, not just a fixed list
of `BaseExpression` subclasses.
- Now that we're exporting everything from the top-level libcst package,
automodule makes a mess of things.
- While it does create more boilerplate, using autoclass/autofunction
instead of automodule gives us a lot more control over the order in
which nodes are presented, so we can group them by various categories.
This diff also exports BaseParenthesizableWhitespace, because nodes
refer to it by type, and so it's needed for documentation purposes.
This was a pain, because apparently we can't refer to
"static_analysis.libcst" as a module before its finished executing, due
to https://bugs.python.org/issue25294. To combat this, I had to change
to direct importing inside parser, which was a bit of a pain due to the
number of nodes used. But, it works.
Go through and clean up where we export things from in the top level
directories. This is the first pass at cleaning up the public API. In
this diff I clean up all top-level exports that were coming out of .py
files and instead export them from libcst's base module itself.
I also didn't export a few of the things defined in newly underscored
files because they were only used internally.
- Add `libcst/__init__.py` back which I accidentally deleted in another
commit.
- Add `*.egg-info/` to the gitignore, because `libcst.egg-info` is it's
created by pip/setuptools when locally installing libcst, and it's
annoying.
- Changed the version number from `0.1.dev` to `0.1.dev0`, since pip was
warning that it was normalizing the version number from the former to
the later.
- Add a `python_requires` field, since we know that libcst only works on
3.6+.
- Add an `install_requires`. Pip uses this to find dependencies, and
ignores `requirements.txt` (since `requirements.txt` is really only
intended to be a freeze file).
- Add the dataclasses backport as a dependency for Python 3.6. I
validated that installing and using libcst works in both 3.6 and 3.7.
**Test Plan:**
```
$ python3 -m venv libcst-install-test # my system python is 3.7
$ libcst-install-test/bin/pip install --upgrade pip ipython
Cache entry deserialization failed, entry ignored
Collecting pip
Using cached
be401c0032/pip-19.1.1-py2.py3-none-any.whl
Collecting ipython
... # lots of output
$ ~/libcst-install-test/bin/pip install ~/libcst/
Processing ./libcst
Requirement already satisfied: parso in
./libcst-install-test/lib/python3.7/site-packages (from
libcst==0.1.dev0) (0.4.0)
Collecting typing_extensions (from libcst==0.1.dev0)
Using cached
c66e553258/typing_extensions-3.7.2-py3-none-any.whl
Installing collected packages: typing-extensions, libcst
Running setup.py install for libcst ... done
Successfully installed libcst-0.1.dev0 typing-extensions-3.7.2
$ ~/libcst-install-test/bin/ipython
Python 3.7.3 (default, Apr 3 2019, 05:39:12)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from libcst import parser
In [2]: parser.parse_expression("None")
Out[2]:
Name(
value='None',
lpar=[],
rpar=[],
)
In [3]:
```
I then repeated the same with a copy of CPython 3.6 that I built from
source.
This ended up being pretty complicated, so the parser stuff will come in
another diff.
Hopefully this should set up up nicely for dicts, sets, and lists too.
