Per the Cargo Book, `license-file` is only to be used if a package uses
a non-standard license; see https://doc.rust-lang.org/cargo/reference/manifest.html#the-license-and-license-file-fields
Declare the licenses directly, and verify that the LICENSE file
containing the license breakdown is still included
```
…n LibCST/native/libcst_derive on cargo-fixes [!] is 📦 v1.4.0 via 🦀 v1.77.1
⬢ [fedora:40] ❯ cargo package --list --allow-dirty | grep LICENSE
LICENSE
…n LibCST/native/libcst_derive on cargo-fixes [!] is 📦 v1.4.0 via 🦀 v1.77.1
⬢ [fedora:40] ❯ cd ../libcst
michel in LibCST/native/libcst on cargo-fixes [!] is 📦 v1.4.0 via 🦀 v1.77.1
⬢ [fedora:40] ❯ cargo package --list --allow-dirty | grep LICENSE
LICENSE
src/tokenizer/core/LICENSE
```
Signed-off-by: Michel Lind <salimma@fedoraproject.org>
Fixes#1160.
This PR also
- fixes `whitespace_before_colon` being swallowed during visitation on `MatchCase`s
- adds a new type of roundtrip test that catches issues of this class: the test applies a noop transformer to exercise the visitation API and compares the result with the original source.
- adds a few more cases to the match fixture
```
match a:
case 1, 2: pass
```
This is parsed correctly by the grammar, but the default values of `MatchList.lbracket` and `MatchList.rbracket` are inconsistent between Python and Rust, causing the above snippet to round-trip (from Python) to:
```
match a:
case [1, 2]: pass
```
Fixes#1096.
removing Regexes from whitespace parser allows ditching of thread local storage + lazy initialization cost
This shows a modest 2% improvement in overall parse time (inflate is improved by 10%)
This PR adds support for parsing and representing Type Parameters and Type Aliases as specified by PEP 695. What's missing are the scope rules, to be implemented in a future PR.
Notable (user visible) changes:
- new `TypeAlias` CST node, which is a `SmallStatement`
- new CST nodes to represent TypeVarLikes: `TypeVar`, `TypeVarTuple`, `ParamSpec`
- new helper CST nodes: `TypeParameters` to serve as a container for multiple TypeVarLikes, and `TypeParam` which is a single item in a `TypeParameters` (owning the separating comma)
- extended `FunctionDef` and `ClassDef` with an optional `type_parameters` field, as well as `whitespace_after_type_parameters` to own the extra whitespace between type parameters and the following token
- these new fields are added after all others to avoid breaking callers passing in fields as positional arguments
- in `FunctionDef` and `ClassDef`, `whitespace_after_name` now owns the whitespace before the type parameters if they exist
When the input doesn't have a trailing newline, but the last line had
exactly the amount of bytes as the current indentation level, the
tokenizer didn't emit a fake newline, causing parse errors (the grammar
expects newlines to conform with the Python spec).
I don't see any reason for fake newlines to be omitted in these cases,
so this PR removes that condition from the tokenizer.
Reported in #930.
* Allow walrus in slices
See https://github.com/python/cpython/pull/23317
Raised in #930.
* Fix parsing of nested f-string specifiers
For an expression like `f"{one:{two:}{three}}"`, `three` is not in an f-string spec, and should be tokenized accordingly.
This PR fixes the `format_spec_count` bookkeeping in the tokenizer, so it properly decrements it when a closing `}` is encountered but only if the `}` closes a format_spec.
Reported in #930.
* Fix tokenizing `0else`
This is an obscure one.
`_ if 0else _` failed to parse with some very weird errors. It turns out that the tokenizer tries to parse `0else` as a single number, but when it encounters `l` it realizes it can't be a single number and it backtracks.
Unfortunately the backtracking logic was broken, and it failed to correctly backtrack one of the offsets used for whitespace parsing (the byte offset since the start of the line). This caused whitespace nodes to refer to incorrect parts of the input text, eventually resulting in the above behavior.
This PR fixes the bookkeeping when the tokenizer backtracks.
Reported in #930.
* Allow no whitespace between lambda keyword and params in certain cases
Python accepts code where `lambda` follows a `*`, so this PR relaxes validation rules for Lambdas.
Raised in #930.
* Allow any expression in comprehensions' evaluated expression
This PR relaxes the accepted types for the `elt` field in `ListComp`, `SetComp`, and `GenExp`, as well as the `key` and `value` fields in `DictComp`.
Fixes#500.
* Allow no space around an ifexp in certain cases
For example in `_ if _ else""if _ else _`.
Raised in #930. Also fixes#854.
* Allow no spaces after `as` in a contextmanager in certain cases
Like in `with foo()as():pass`
Raised in #930.
* Allow no spaces around walrus in certain cases
Like in `[_:=''for _ in _]`
Raised in #930.
* Allow no whitespace after lambda body in certain cases
Like in `[lambda:()for _ in _]`
Reported in #930.
* Fix Github issue 855 - fail to parse with statement
When we added support for parenthesized with statements, the
grammar on the with itself was correct (it's a right and left
parenthesis around a comma-separated list of with-items, with
a possible trailing comma).
But inside of the "as" variation of the with_item rule we have a peek at
the next character, which was allowing for a comma or a colon. That peek
needs to also accept right parentheses - otherwise, if the last item
contains an `as` and has no trailing comma we fail to parse.
The bug is exercisecd by, for example, this code snippet:
```
with (foo, bar as bar,):
pass
```
The with_wickedness test fixture has been revised to include both
the plain and async variations of this example snippet with and without
trailing comma, and tests pass after the peek rule fix.
* Add more tests covering the plain expression form of `with_item`
