13ffb5bc19
(Supersedes #9152, authored by @LaBatata101) ## Summary This PR replaces the current parser generated from LALRPOP to a hand-written recursive descent parser. It also updates the grammar for [PEP 646](https://peps.python.org/pep-0646/) so that the parser outputs the correct AST. For example, in `data[*x]`, the index expression is now a tuple with a single starred expression instead of just a starred expression. Beyond the performance improvements, the parser is also error resilient and can provide better error messages. The behavior as seen by any downstream tools isn't changed. That is, the linter and formatter can still assume that the parser will _stop_ at the first syntax error. This will be updated in the following months. For more details about the change here, refer to the PR corresponding to the individual commits and the release blog post. ## Test Plan Write _lots_ and _lots_ of tests for both valid and invalid syntax and verify the output. ## Acknowledgements - @MichaReiser for reviewing 100+ parser PRs and continuously providing guidance throughout the project - @LaBatata101 for initiating the transition to a hand-written parser in #9152 - @addisoncrump for implementing the fuzzer which helped [catch](https://github.com/astral-sh/ruff/pull/10903) [a](https://github.com/astral-sh/ruff/pull/10910) [lot](https://github.com/astral-sh/ruff/pull/10966) [of](https://github.com/astral-sh/ruff/pull/10896) [bugs](https://github.com/astral-sh/ruff/pull/10877) --------- Co-authored-by: Victor Hugo Gomes <labatata101@linuxmail.org> Co-authored-by: Micha Reiser <micha@reiser.io>
1.3 KiB
Contributing to the Python Parser
Development
Inline tests
The parser crate supports writing inline tests. These are tests that are written in the source code itself, and are extracted to a separate file and run with the test suite. They are written in the form of comments with a specific format. There are two forms of inline tests:
Test that the parser successfully parses the input with no syntax errors. They're written in the following format:
// test_ok this_is_the_test_name
// def foo():
// pass
println!("some rust code");
Test that the parser fails to parse the input with a syntax error. They're written in the following format:
// test_err this_is_the_test_name
// [1, 2
println!("some rust code");
Note that the difference between the two is the test_ok and test_err keywords.
The comment block must be independent of any other comment blocks. For example, the
following is not extracted:
// Some random comment
//
// test_ok this_is_the_test_name
// def foo():
// pass
println!("some rust code");
To generate the corresponding Python files for the inline tests, run the following command:
cargo test --package ruff_python_parser --test generate_inline_tests
Then, run the Parser test suite with the following command:
cargo test --package ruff_python_parser