mirror of
https://github.com/astral-sh/ruff.git
synced 2025-08-03 10:23:11 +00:00
04a922866a
## Summary This PR adds a basic README for the `ruff_python_parser` crate and updates the CONTRIBUTING docs with the fuzzer and benchmark section. Additionally, it also updates some inline documentation within the parser crate and splits the `parse_program` function into `parse_single_expression` and `parse_module` which will be called by matching against the `Mode`. This PR doesn't go into too much internal detail around the parser logic due to the following reasons: 1. Where should the docs go? Should it be as a module docs in `lib.rs` or in README? 2. The parser is still evolving and could include a lot of refactors with the future work (feedback loop and improved error recovery and resilience) --------- Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
100 lines
2.6 KiB
Markdown
100 lines
2.6 KiB
Markdown
# Contributing to the Python Parser
|
|
|
|
## Development
|
|
|
|
### Inline tests
|
|
|
|
The parser crate supports writing inline tests. These are tests that are written
|
|
in the source code itself, and are extracted to a separate file and run with the
|
|
test suite. They are written in the form of comments with a specific format. There
|
|
are two forms of inline tests:
|
|
|
|
Test that the parser successfully parses the input with no syntax errors. They're
|
|
written in the following format:
|
|
|
|
```rs
|
|
// test_ok this_is_the_test_name
|
|
// def foo():
|
|
// pass
|
|
println!("some rust code");
|
|
```
|
|
|
|
Test that the parser fails to parse the input with a syntax error. They're written
|
|
in the following format:
|
|
|
|
```rs
|
|
// test_err this_is_the_test_name
|
|
// [1, 2
|
|
println!("some rust code");
|
|
```
|
|
|
|
Note that the difference between the two is the `test_ok` and `test_err` keywords.
|
|
The comment block must be independent of any other comment blocks. For example, the
|
|
following is not extracted:
|
|
|
|
```rs
|
|
// Some random comment
|
|
//
|
|
// test_ok this_is_the_test_name
|
|
// def foo():
|
|
// pass
|
|
println!("some rust code");
|
|
```
|
|
|
|
To generate the corresponding Python files for the inline tests, run the following command:
|
|
|
|
```sh
|
|
cargo test --package ruff_python_parser --test generate_inline_tests
|
|
```
|
|
|
|
Then, run the Parser test suite with the following command:
|
|
|
|
```sh
|
|
cargo test --package ruff_python_parser
|
|
```
|
|
|
|
### Python-based fuzzer
|
|
|
|
The Ruff project includes a Python-based fuzzer that can be used to run the parser on
|
|
randomly generated (but syntactically valid) Python source code files.
|
|
|
|
To run the fuzzer, first install the required dependencies:
|
|
|
|
```sh
|
|
uv pip install -r scripts/fuzz-parser/requirements.txt
|
|
```
|
|
|
|
Then, run the fuzzer with the following command:
|
|
|
|
```sh
|
|
python scripts/fuzz-parser/fuzz.py
|
|
```
|
|
|
|
Refer to the [fuzz.py](https://github.com/astral-sh/ruff/blob/main/scripts/fuzz-parser/fuzz.py)
|
|
script for more information or use the `--help` flag to see the available options.
|
|
|
|
#### CI
|
|
|
|
The fuzzer is run as part of the CI pipeline. The purpose of running the fuzzer in the CI is to
|
|
catch any regresssions introduced by any new changes to the parser. This is why the fuzzer is run on
|
|
the same set of seeds on every run.
|
|
|
|
## Benchmarks
|
|
|
|
The `ruff_benchmark` crate can benchmark both the lexer and the parser.
|
|
|
|
To run the lexer benchmarks, use the following command:
|
|
|
|
```sh
|
|
cargo bench --package ruff_benchmark --bench lexer
|
|
```
|
|
|
|
And to run the parser benchmarks, use the following command:
|
|
|
|
```sh
|
|
cargo bench --package ruff_benchmark --bench parser
|
|
```
|
|
|
|
Refer to the [Benchmarking and
|
|
Profiling](https://docs.astral.sh/ruff/contributing/#benchmark-driven-development) section in the
|
|
contributing guide for more information.
|