This is an obscure one.
`_ if 0else _` failed to parse with some very weird errors. It turns out that the tokenizer tries to parse `0else` as a single number, but when it encounters `l` it realizes it can't be a single number and it backtracks.
Unfortunately the backtracking logic was broken, and it failed to correctly backtrack one of the offsets used for whitespace parsing (the byte offset since the start of the line). This caused whitespace nodes to refer to incorrect parts of the input text, eventually resulting in the above behavior.
This PR fixes the bookkeeping when the tokenizer backtracks.
Reported in #930.
For an expression like `f"{one:{two:}{three}}"`, `three` is not in an f-string spec, and should be tokenized accordingly.
This PR fixes the `format_spec_count` bookkeeping in the tokenizer, so it properly decrements it when a closing `}` is encountered but only if the `}` closes a format_spec.
Reported in #930.
* Fix type of evaluated_value on string
This can return bytes if the string is a bytestring, e.g.:
In [1]: import libcst as cst
In [2]: cst.parse_expression('b"foo"').evaluated_value
Out[2]: b'foo'
* Fix type errors from changed signature
* Fixes an issue where ApplyTypeAnnotationsVisitor would crash on code
like `SomeClass.some_attribute = 42` with a "Name is not a valid
identifier" error message.
* Changes the above-mentioned error message to include the bad name in
the message, for easier debugging.
* Adds tests for all valid assignment targets, as described here:
https://libcst.readthedocs.io/en/latest/nodes.html#libcst.BaseAssignTargetExpression.
* Simplify command specifier parsing
* Allow running codemods without configuring in YAML
This enables codemodding things by just plonking a CodemodCommand class
into any old importable module and running
`python -m libcst.tool codemod -x some_module.SomeClass ...`
Moves PEP 621 metadata from `setup.py` and `requirements*.txt` into the
`[project]` table of `pyproject.toml`. This enables using hatch as a
task runner for the project, where previously one would need to remember
a bunch of different commands, or repeatedly consult the readme's
developer guide to find all of the relevant commands.
This creates the following hatch commands:
- docs
- fixtures
- format
- lint
- test
- typecheck
It also updates all of the github actions workflows to use the
appropriate hatch commands, and the readme's developer guide, so that
there is only one source of truth for what constitutes running tests.
The "test" workflows now drop the matrix distinction between "pure" or
"native", and run tests in both modes from a single build.
ghstack-source-id: 8834da7825
Pull Request resolved: https://github.com/Instagram/LibCST/pull/893
Caches file path information on the root `Module` node.
Resolves paths when caching, so they are always absolute paths.
Adds a new `chdir` helper to change working directory and automatically
revert to previous directory, which makes testing file paths with the
`"."` repo root easier.
ghstack-source-id: 3413905fc1
Pull Request resolved: https://github.com/Instagram/LibCST/pull/892
pypi_upload has been broken since #810, because `actions/checkout` defaults to a shallow checkout that only checks out the revision triggering the workflow. This causes setuptools-scm to miss the most recent tag, causing the version to be detected as `0.1`.
With the latest setup-python actions, there is a better caching
mechanism available that also requires less setup, and provides better
fallback behavior that should help avoid the random CI failures that
have been happening on 3.11 for setuptools-rust. This ensures that we
install the necessary dependencies before attempting to build the
package or run tests, while still enabling speedups in best case
scenario when requirements files haven't changed.
See the upstream readme for details:
https://github.com/actions/setup-python#caching-packages-dependencies
This allows FullyQualifiedNameProvider to work with absolute paths,
rather than assuming all paths given will be relative to the current
directory. This enables tools like Fixit to provide a root path, and
have the FullyQualifiedNameProvider correctly scope the final results
relative to that root path.
This does require that both the root path and the given file paths
match the other as relative or absolute, due to the
`calculate_module_and_package` helper comparing file paths relative
to the root path, but this seems like a reasonable tradeoff, and
unlikely to cause a problem in normal use cases.
Upgrading Pyre requires updating test fixtures with any upstream changes
to Pyre's query results for the `simple_class.py` fixture.
This adds a new `scripts/` directory to the repo, with a script to
regenerate test fixtures. The script regenerates the cache data fixture,
and updates the `TypeInferenceProvider` tests to use `assertDictEqual`
and helpful error messages for better behavior in future mismatches.
This also includes a slight bump to Pyre 0.9.10 to fix install issues on
Apple Silicon M1 Macs, and regenerated fixtures using the script above.
* Fix Github issue 855 - fail to parse with statement
When we added support for parenthesized with statements, the
grammar on the with itself was correct (it's a right and left
parenthesis around a comma-separated list of with-items, with
a possible trailing comma).
But inside of the "as" variation of the with_item rule we have a peek at
the next character, which was allowing for a comma or a colon. That peek
needs to also accept right parentheses - otherwise, if the last item
contains an `as` and has no trailing comma we fail to parse.
The bug is exercisecd by, for example, this code snippet:
```
with (foo, bar as bar,):
pass
```
The with_wickedness test fixture has been revised to include both
the plain and async variations of this example snippet with and without
trailing comma, and tests pass after the peek rule fix.
* Add more tests covering the plain expression form of `with_item`
