When the input doesn't have a trailing newline, but the last line had
exactly the amount of bytes as the current indentation level, the
tokenizer didn't emit a fake newline, causing parse errors (the grammar
expects newlines to conform with the Python spec).
I don't see any reason for fake newlines to be omitted in these cases,
so this PR removes that condition from the tokenizer.
Reported in #930.
* Allow walrus in slices
See https://github.com/python/cpython/pull/23317
Raised in #930.
* Fix parsing of nested f-string specifiers
For an expression like `f"{one:{two:}{three}}"`, `three` is not in an f-string spec, and should be tokenized accordingly.
This PR fixes the `format_spec_count` bookkeeping in the tokenizer, so it properly decrements it when a closing `}` is encountered but only if the `}` closes a format_spec.
Reported in #930.
* Fix tokenizing `0else`
This is an obscure one.
`_ if 0else _` failed to parse with some very weird errors. It turns out that the tokenizer tries to parse `0else` as a single number, but when it encounters `l` it realizes it can't be a single number and it backtracks.
Unfortunately the backtracking logic was broken, and it failed to correctly backtrack one of the offsets used for whitespace parsing (the byte offset since the start of the line). This caused whitespace nodes to refer to incorrect parts of the input text, eventually resulting in the above behavior.
This PR fixes the bookkeeping when the tokenizer backtracks.
Reported in #930.
* Allow no whitespace between lambda keyword and params in certain cases
Python accepts code where `lambda` follows a `*`, so this PR relaxes validation rules for Lambdas.
Raised in #930.
* Allow any expression in comprehensions' evaluated expression
This PR relaxes the accepted types for the `elt` field in `ListComp`, `SetComp`, and `GenExp`, as well as the `key` and `value` fields in `DictComp`.
Fixes#500.
* Allow no space around an ifexp in certain cases
For example in `_ if _ else""if _ else _`.
Raised in #930. Also fixes#854.
* Allow no spaces after `as` in a contextmanager in certain cases
Like in `with foo()as():pass`
Raised in #930.
* Allow no spaces around walrus in certain cases
Like in `[_:=''for _ in _]`
Raised in #930.
* Allow no whitespace after lambda body in certain cases
Like in `[lambda:()for _ in _]`
Reported in #930.
* Fix type of evaluated_value on string
This can return bytes if the string is a bytestring, e.g.:
In [1]: import libcst as cst
In [2]: cst.parse_expression('b"foo"').evaluated_value
Out[2]: b'foo'
* Fix type errors from changed signature
* Fixes an issue where ApplyTypeAnnotationsVisitor would crash on code
like `SomeClass.some_attribute = 42` with a "Name is not a valid
identifier" error message.
* Changes the above-mentioned error message to include the bad name in
the message, for easier debugging.
* Adds tests for all valid assignment targets, as described here:
https://libcst.readthedocs.io/en/latest/nodes.html#libcst.BaseAssignTargetExpression.
* Simplify command specifier parsing
* Allow running codemods without configuring in YAML
This enables codemodding things by just plonking a CodemodCommand class
into any old importable module and running
`python -m libcst.tool codemod -x some_module.SomeClass ...`
Moves PEP 621 metadata from `setup.py` and `requirements*.txt` into the
`[project]` table of `pyproject.toml`. This enables using hatch as a
task runner for the project, where previously one would need to remember
a bunch of different commands, or repeatedly consult the readme's
developer guide to find all of the relevant commands.
This creates the following hatch commands:
- docs
- fixtures
- format
- lint
- test
- typecheck
It also updates all of the github actions workflows to use the
appropriate hatch commands, and the readme's developer guide, so that
there is only one source of truth for what constitutes running tests.
The "test" workflows now drop the matrix distinction between "pure" or
"native", and run tests in both modes from a single build.
ghstack-source-id: 8834da7825
Pull Request resolved: https://github.com/Instagram/LibCST/pull/893
Caches file path information on the root `Module` node.
Resolves paths when caching, so they are always absolute paths.
Adds a new `chdir` helper to change working directory and automatically
revert to previous directory, which makes testing file paths with the
`"."` repo root easier.
ghstack-source-id: 3413905fc1
Pull Request resolved: https://github.com/Instagram/LibCST/pull/892
