ruff/scripts/knot_benchmark
Micha Reiser c4578162d5
[red-knot] Add support for knot check <paths> (#16375)
## Summary

This PR adds support for an optional list of paths that should be
checked to `knot check`.

E.g. to only check the `src` directory

```sh
knot check src
```

The default is to check all files in the project but users can reduce
the included files by specifying one or multiple optional paths.

The main two challenges with adding this feature were:

* We now need to show an error when one of the provided paths doesn't
exist. That's why this PR now collects errors from the project file
indexing phase and adds them to the output diagnostics. The diagnostic
looks similar to ruffs (see CLI test)
* The CLI should pick up new files added to included folders. For
example, `knot check src --watch` should pick up new files that are
added to the `src` folder. This requires that we now filter the files
before adding them to the project. This is a good first step to
supporting `include` and `exclude`.


The PR makes two simplifications:

1. I didn't test the changes with case-insensitive file systems. We may
need to do some extra path normalization to support those well. See
https://github.com/astral-sh/ruff/issues/16400
2. Ideally, we'd accumulate the IO errors from the initial indexing
phase and subsequent incremental indexing operations. For example, we
should preserve the IO diagnostic for a non existing `test.py` if it was
specified as an explicit CLI argument until the file gets created and we
should show it again when the file gets deleted. However, this is
somewhat complicated because we'd need to track which files we revisited
(or were removed because the entire directory is gone). I considered
this too low a priority as it's worth dealing with right now.

The implementation doesn't support symlinks within the project but that
is the same as Ruff and is unchanged from before this PR.



Closes https://github.com/astral-sh/ruff/issues/14193

## Test Plan

Added CLI and file watching integration tests. Manually testing.
2025-03-03 12:59:56 +00:00
..
src/benchmark [red-knot] Add support for knot check <paths> (#16375) 2025-03-03 12:59:56 +00:00
pyproject.toml Add basic red knot benchmark (#13026) 2024-08-23 08:22:42 +02:00
README.md Add basic red knot benchmark (#13026) 2024-08-23 08:22:42 +02:00
uv.lock Add basic red knot benchmark (#13026) 2024-08-23 08:22:42 +02:00

Getting started

  1. Install uv
  • Unix: curl -LsSf https://astral.sh/uv/install.sh | sh
  • Windows: powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
  1. Build red_knot: cargo build --bin red_knot --release
  2. cd into the benchmark directory: cd scripts/knot_benchmark
  3. Run benchmarks: uv run benchmark

Known limitations

Red Knot only implements a tiny fraction of Mypy's and Pyright's functionality, so the benchmarks aren't in any way a fair comparison today. However, they'll become more meaningful as we build out more type checking features in Red Knot.

Windows support

The script should work on Windows, but we haven't tested it yet. We do make use of shlex which has known limitations when using non-POSIX shells.