ruff/crates
Dhruv Manilawala 13ffb5bc19
Replace LALRPOP parser with hand-written parser (#10036)
(Supersedes #9152, authored by @LaBatata101)

## Summary

This PR replaces the current parser generated from LALRPOP to a
hand-written recursive descent parser.

It also updates the grammar for [PEP
646](https://peps.python.org/pep-0646/) so that the parser outputs the
correct AST. For example, in `data[*x]`, the index expression is now a
tuple with a single starred expression instead of just a starred
expression.

Beyond the performance improvements, the parser is also error resilient
and can provide better error messages. The behavior as seen by any
downstream tools isn't changed. That is, the linter and formatter can
still assume that the parser will _stop_ at the first syntax error. This
will be updated in the following months.

For more details about the change here, refer to the PR corresponding to
the individual commits and the release blog post.

## Test Plan

Write _lots_ and _lots_ of tests for both valid and invalid syntax and
verify the output.

## Acknowledgements

- @MichaReiser for reviewing 100+ parser PRs and continuously providing
guidance throughout the project
- @LaBatata101 for initiating the transition to a hand-written parser in
#9152
- @addisoncrump for implementing the fuzzer which helped
[catch](https://github.com/astral-sh/ruff/pull/10903)
[a](https://github.com/astral-sh/ruff/pull/10910)
[lot](https://github.com/astral-sh/ruff/pull/10966)
[of](https://github.com/astral-sh/ruff/pull/10896)
[bugs](https://github.com/astral-sh/ruff/pull/10877)

---------

Co-authored-by: Victor Hugo Gomes <labatata101@linuxmail.org>
Co-authored-by: Micha Reiser <micha@reiser.io>
2024-04-18 17:57:39 +05:30
..
ruff Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
ruff_benchmark Run doctests as part of CI pipeline (#9939) 2024-02-12 10:18:58 +01:00
ruff_cache chore: remove repetitive words (#10952) 2024-04-15 12:57:48 +00:00
ruff_dev Docs: Link inline settings when not part of options section (#10499) 2024-03-21 16:33:58 +00:00
ruff_diagnostics Respect # noqa directives on __all__ openers (#10798) 2024-04-06 14:51:17 +00:00
ruff_formatter chore: remove repetitive words (#10952) 2024-04-15 12:57:48 +00:00
ruff_index Run doctests as part of CI pipeline (#9939) 2024-02-12 10:18:58 +01:00
ruff_linter Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
ruff_macros Fix comment copy/paste typo in newtype_index (#10892) 2024-04-11 18:43:52 -06:00
ruff_notebook Remove unused dependencies (#10475) 2024-03-19 17:33:47 +01:00
ruff_python_ast Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
ruff_python_codegen Track casing of r-string prefixes in the tokenizer and AST (#10314) 2024-03-18 17:18:04 +00:00
ruff_python_formatter Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
ruff_python_index Fix Indexer fails to identify continuation preceded by newline #10351 (#10354) 2024-03-12 00:35:41 -04:00
ruff_python_literal Remove unused dependencies (#10475) 2024-03-19 17:33:47 +01:00
ruff_python_parser Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
ruff_python_resolver chore(deps): update rust crate insta to v1.38.0 (#10701) 2024-04-01 15:44:30 +00:00
ruff_python_semantic Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
ruff_python_stdlib Improve handling of builtin symbols in linter rules (#10919) 2024-04-16 11:37:31 +01:00
ruff_python_trivia Remove unused dependencies (#10475) 2024-03-19 17:33:47 +01:00
ruff_server ruff server: Introduce settings for directly configuring the linter and formatter (#10984) 2024-04-18 07:53:48 +00:00
ruff_shrinking Bump version to v0.3.7 (#10895) 2024-04-12 03:39:45 +00:00
ruff_source_file Remove unused dependencies (#10475) 2024-03-19 17:33:47 +01:00
ruff_text_size Range formatting: Fix invalid syntax after parenthesizing expression (#9751) 2024-02-02 17:56:25 +01:00
ruff_wasm Run doctests as part of CI pipeline (#9939) 2024-02-12 10:18:58 +01:00
ruff_workspace Fix Typo for extend-aliases Option (#10978) 2024-04-16 12:23:09 -04:00