Per the Cargo Book, `license-file` is only to be used if a package uses
a non-standard license; see https://doc.rust-lang.org/cargo/reference/manifest.html#the-license-and-license-file-fields
Declare the licenses directly, and verify that the LICENSE file
containing the license breakdown is still included
```
…n LibCST/native/libcst_derive on cargo-fixes [!] is 📦 v1.4.0 via 🦀 v1.77.1
⬢ [fedora:40] ❯ cargo package --list --allow-dirty | grep LICENSE
LICENSE
…n LibCST/native/libcst_derive on cargo-fixes [!] is 📦 v1.4.0 via 🦀 v1.77.1
⬢ [fedora:40] ❯ cd ../libcst
michel in LibCST/native/libcst on cargo-fixes [!] is 📦 v1.4.0 via 🦀 v1.77.1
⬢ [fedora:40] ❯ cargo package --list --allow-dirty | grep LICENSE
LICENSE
src/tokenizer/core/LICENSE
```
Signed-off-by: Michel Lind <salimma@fedoraproject.org>
This massive PR implements an alternative Python parser that will allow LibCST to parse Python 3.10's new grammar features. The parser is implemented in Rust, but it's turned off by default through the `LIBCST_PARSER_TYPE` environment variable. Set it to `native` to enable. The PR also enables new CI steps that test just the Rust parser, as well as steps that produce binary wheels for a variety of CPython versions and platforms.
Note: this PR aims to be roughly feature-equivalent to the main branch, so it doesn't include new 3.10 syntax features. That will be addressed as a follow-up PR.
The new parser is implemented in the `native/` directory, and is organized into two rust crates: `libcst_derive` contains some macros to facilitate various features of CST nodes, and `libcst` contains the `parser` itself (including the Python grammar), a `tokenizer` implementation by @bgw, and a very basic representation of CST `nodes`. Parsing is done by
1. **tokenizing** the input utf-8 string (bytes are not supported at the Rust layer, they are converted to utf-8 strings by the python wrapper)
2. running the **PEG parser** on the tokenized input, which also captures certain anchor tokens in the resulting syntax tree
3. using the anchor tokens to **inflate** the syntax tree into a proper CST
Co-authored-by: Benjamin Woodruff <github@benjam.info>
