Commit graph

161 commits

Author SHA1 Message Date
Charlie Marsh
ddfdc3bb01
Add rule documentation URL to JSON output (#5187)
## Summary

I want to include URLs to the rule documentation in the LSP (the LSP has
a native `code_description` field for this, which, if specified, causes
the source to be rendered as a link to the docs). This PR exposes the
URL to the documentation in the Ruff JSON output.
2023-06-19 21:09:15 +00:00
Dhruv Manilawala
48f4f2d63d
Maintain consistency when deserializing to JSON (#5114)
## Summary

Maintain consistency while deserializing Jupyter notebook to JSON. The
following changes were made:

1. Use string array to store the source value as that's the default
(5781720423/nbformat/v4/nbjson.py (L56-L57))
2. Remove unused structs and enums
3. Reorder the keys in alphabetical order as that's the default.
(5781720423/nbformat/v4/nbjson.py (L51))

### Side effect

Removing the `preserve_order` feature means that the order of keys in
JSON output (`--format json`) will be in alphabetical order. This is
because the value is represented using `serde_json::Value` which
internally is a `BTreeMap`, thus sorting it as per the string key. For
posterity if this turns out to be not ideal, then we could define a
struct representing the JSON object and the order of struct fields will
determine the order in the JSON string.

## Test Plan

Add a test case to assert the raw JSON string.
2023-06-19 23:47:56 +05:30
Thomas de Zeeuw
e3c12764f8
Only use a single cache file per Python package (#5117)
## Summary

This changes the caching design from one cache file per source file, to
one cache file per package. This greatly reduces the amount of cache
files that are opened and written, while maintaining roughly the same
(combined) size as bincode is very compact.

Below are some very much not scientific performance tests. It uses
projects/sources to check:

* small.py: single, 31 bytes Python file with 2 errors.
* test.py: single, 43k Python file with 8 errors.
* fastapi: FastAPI repo, 1134 files checked, 0 errors.

Source   | Before # files | After # files | Before size | After size
-------|-------|-------|-------|-------
small.py | 1              | 1             | 20 K        | 20 K
test.py  | 1              | 1             | 60 K        | 60 K
fastapi  | 1134           | 518           | 4.5 M       | 2.3 M

One question that might come up is why fastapi still has 518 cache files
and not 1? That is because this is using the existing package
resolution, which sees examples, docs, etc. as separate from the "main"
source code (in the fastapi directory in the repo). In this future it
might be worth consider switching to a one cache file per repo strategy.

This new design is not perfect and does have a number of known issues.
First, like the old design it doesn't remove the cache for a source file
that has been (re)moved until `ruff clean` is called.

Second, this currently uses a large mutex around the mutation of the
package cache (e.g. inserting result). This could be (or become) a
bottleneck. It's future work to test and improve this (if needed).

Third, currently the packages and opened and stored in a sequential
loop, this could be done parallel. This is also future work.


## Test Plan

Run `ruff check` (with caching enabled) twice on any Python source code
and it should produce the same results.
2023-06-19 17:46:13 +02:00
konstin
b8d378b0a3
Add a script that tests formatter stability on repositories (#5055)
## Summary

We want to ensure that once formatted content stays the same when
formatted again, which is known as formatter stability or formatter
idempotency, and that the formatter prints syntactically valid code. As
our test cases cover only a limited amount of code, this allows checking
entire repositories.

This adds a new subcommand to `ruff_dev` which can be invoked as `cargo
run --bin ruff_dev -- check-formatter-stability <repo>`. While initially
only intended to check stability, it has also found cases where the
formatter printed invalid syntax or panicked.

 ## Test Plan

Running this on cpython is already identifying bugs
(https://github.com/astral-sh/ruff/pull/5089)
2023-06-19 14:13:38 +00:00
konstin
361d45f2b2
Add cargo dev repeat for profiling (#5144)
## Summary

This adds a new subcommand that can be used as

```shell
cargo build --bin ruff_dev --profile=release-debug
perf record -g -F 999 target/release-debug/ruff_dev repeat --repeat 30 --exit-zero --no-cache path/to/cpython > /dev/null
flamegraph --perfdata perf.data
```

## Test Plan

This is a ruff internal script. I successfully used it to profile
cpython with the instructions above
2023-06-19 11:40:09 +02:00
Charlie Marsh
732b0405d7
Remove FixMode::None (#5087)
## Summary

We now _always_ generate fixes, so `FixMode::None` and
`FixMode::Generate` are redundant. We can also remove the TODO around
`--fix-dry-run`, since that's our default behavior.

Closes #5081.
2023-06-14 11:17:09 -04:00
Aarni Koskela
7b4dde0c6c
Add JSON Lines (NDJSON) message serialization (#5048)
## Summary

This adds `json-lines` (https://jsonlines.org/ or http://ndjson.org/) as
an output format.

I'm sure you already know, but

* JSONL is more greppable (each record is a single line) than the pretty
JSON
* JSONL is faster to ingest piecewise (and/or in parallel) than JSON

## Test Plan

Snapshot test in the new module :)
2023-06-13 14:15:55 +00:00
Dhruv Manilawala
d8f5d2d767
Add support for auto-fix in Jupyter notebooks (#4665)
## Summary

Add support for applying auto-fixes in Jupyter Notebook.

### Solution

Cell offsets are the boundaries for each cell in the concatenated source
code. They are represented using `TextSize`. It includes the start and
end offset as well, thus creating a range for each cell. These offsets
are updated using the `SourceMap` markers.

### SourceMap

`SourceMap` contains markers constructed from each edits which tracks
the original source code position to the transformed positions. The
following drawing might make it clear:

![SourceMap visualization](3c94e591-70a7-4b57-bd32-0baa91cc7858)

The center column where the dotted lines are present are the markers
included in the `SourceMap`. The `Notebook` looks at these markers and
updates the cell offsets after each linter loop. If you notice closely,
the destination takes into account all of the markers before it.

The index is constructed only when required as it's only used to render
the diagnostics. So, a `OnceCell` is used for this purpose. The cell
offsets, cell content and the index will be updated after each iteration
of linting in the mentioned order. The order is important here as the
content is updated as per the new offsets and index is updated as per
the new content.

## Limitations

### 1

Styling rules such as the ones in `pycodestyle` will not be applicable
everywhere in Jupyter notebook, especially at the cell boundaries. Let's
take an example where a rule suggests to have 2 blank lines before a
function and the cells contains the following code:

```python
import something
# ---
def first():
	pass

def second():
	pass
```

(Again, the comment is only to visualize cell boundaries.)

In the concatenated source code, the 2 blank lines will be added but it
shouldn't actually be added when we look in terms of Jupyter notebook.
It's as if the function `first` is at the start of a file.

`nbqa` solves this by recording newlines before and after running
`autopep8`, then running the tool and restoring the newlines at the end
(refer https://github.com/nbQA-dev/nbQA/pull/807).

## Test Plan

Three commands were run in order with common flags (`--select=ALL
--no-cache --isolated`) to isolate which stage the problem is occurring:
1. Only diagnostics
2. Fix with diff (`--fix --diff`)
3. Fix (`--fix`)

### https://github.com/facebookresearch/segment-anything

```
-------------------------------------------------------------------------------
 Jupyter Notebooks       3            0            0            0            0
 |- Markdown             3           98            0           94            4
 |- Python               3          513          468            4           41
 (Total)                            611          468           98           45
-------------------------------------------------------------------------------
```

```console
$ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/segment-anything/**/*.ipynb --fix
...
Found 180 errors (89 fixed, 91 remaining).
```

### https://github.com/openai/openai-cookbook

```
-------------------------------------------------------------------------------
 Jupyter Notebooks      65            0            0            0            0
 |- Markdown            64         3475           12         2507          956
 |- Python              65         9700         7362         1101         1237
 (Total)                          13175         7374         3608         2193
===============================================================================
```

```console
$ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/openai-cookbook/**/*.ipynb --fix
error: Failed to parse /path/to/openai-cookbook/examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb:cell 4:29:18: unexpected token '-'
...
Found 4227 errors (2165 fixed, 2062 remaining).
```

### https://github.com/tensorflow/docs

```
-------------------------------------------------------------------------------
 Jupyter Notebooks     150            0            0            0            0
 |- Markdown             1           55            0           46            9
 |- Python               1          402          289           60           53
 (Total)                            457          289          106           62
-------------------------------------------------------------------------------
```

```console
$ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/tensorflow-docs/**/*.ipynb --fix
error: Failed to parse /path/to/tensorflow-docs/site/en/guide/extension_type.ipynb:cell 80:1:1: unexpected token Indent
error: Failed to parse /path/to/tensorflow-docs/site/en/r1/tutorials/eager/custom_layers.ipynb:cell 20:1:1: unexpected token Indent
error: Failed to parse /path/to/tensorflow-docs/site/en/guide/data.ipynb:cell 175:5:14: unindent does not match any outer indentation level
error: Failed to parse /path/to/tensorflow-docs/site/en/r1/tutorials/representation/unicode.ipynb:cell 30:1:1: unexpected token Indent
...
Found 12726 errors (5140 fixed, 7586 remaining).
```

### https://github.com/tensorflow/models

```
-------------------------------------------------------------------------------
 Jupyter Notebooks      46            0            0            0            0
 |- Markdown             1           11            0            6            5
 |- Python               1          328          249           19           60
 (Total)                            339          249           25           65
-------------------------------------------------------------------------------
```

```console
$ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/tensorflow-models/**/*.ipynb --fix
...
Found 4856 errors (2690 fixed, 2166 remaining).
```

resolves: #1218
fixes: #4556
2023-06-12 14:14:15 +00:00
Dhruv Manilawala
c3d1fa851e
Ignore pyproject.toml for adding noqa directives (#5013)
## Summary

Ignore pyproject.toml file for adding noqa directives using `--add-noqa`

## Test Plan

`cargo run --bin ruff -- check --add-noqa .`

fixes: #5012
2023-06-11 20:21:24 -04:00
Charlie Marsh
68b6d30c46
Use consistent Cargo.toml metadata in all crates (#5015) 2023-06-12 00:02:40 +00:00
Dhruv Manilawala
07cc4bcb0f
Update links to point to Astral org (#4949) 2023-06-08 11:43:40 -04:00
Charlie Marsh
5235977abc
Bump version to 0.0.272 (#4948) 2023-06-08 02:17:29 +00:00
Charlie Marsh
2b5fb70482
Bump version to 0.0.271 (#4890) 2023-06-06 15:11:48 -04:00
Charlie Marsh
621718784a
Replace deletion-tracking with enforced isolation levels (#4766) 2023-06-02 02:45:56 +00:00
konstin
d4027d8b65
Use new formatter infrastructure in CLI and test (#4767)
* Use dummy verbatim formatter for all nodes

* Use new formatter infrastructure in CLI and test

* Expose the new formatter in the CLI

* Merge import blocks
2023-06-01 11:55:04 +02:00
Charlie Marsh
399eb84d5e
Add a ruff_textwrap crate (#4731) 2023-05-31 16:35:23 +00:00
konstin
b6a382eeaf
Lint pyproject.toml (#4496)
This adds a new rule `InvalidPyprojectToml` that lints pyproject.toml by checking if https://github.com/PyO3/pyproject-toml-rs can parse it. This means the linting is currently very basic, e.g. we don't check whether the name is actually a valid python project name or appropriately normalized. It does catch errors e.g. with invalid dependency requirements or problems withs the license specifications. It is open to be extended in the future (validate name, SPDX expressions, classifiers, ...), either in ruff or in pyproject-toml-rs.

Test plan:

```
scripts/ecosystem_all_check.sh check --select RUF200
```
This lead to a bunch of 
```
RUF200 Failed to parse pyproject.toml: missing field `name`
```
(e.g. https://github.com/amitsk/fastapi-todos/blob/main/pyproject.toml) which is indeed invalid (https://packaging.python.org/en/latest/specifications/declaring-project-metadata/#specification).

Filtering those out, the following other problems were found by `cd target/ecosystem_all_results/ && rg RUF200`:
```
UCL-ARC:rred-reports.stdout.txt
1:pyproject.toml:27:16: RUF200 Failed to parse pyproject.toml: Version specifier `>='3.9'` doesn't match PEP 440 rules
EndlessTrax:python-start-project.stdout.txt
1:pyproject.toml:14:16: RUF200 Failed to parse pyproject.toml: Expected package name starting with an alphanumeric character, found '#'
redjax:gardening-api.stdout.txt
1:pyproject.toml:7:11: RUF200 Failed to parse pyproject.toml: Version `` doesn't match PEP 440 rules
ajslater:codex.stdout.txt
2:  3:17 RUF200 Failed to parse pyproject.toml: invalid type: sequence, expected a string
LDmitriy7:404_AvatarsBot.stdout.txt
1:pyproject.toml:3:11: RUF200 Failed to parse pyproject.toml: Version `` doesn't match PEP 440 rules
ajslater:comicbox.stdout.txt
1:pyproject.toml:3:17: RUF200 Failed to parse pyproject.toml: invalid type: sequence, expected a string
manueldevillena:forecast-earnings.stdout.txt
1:pyproject.toml:24:12: RUF200 Failed to parse pyproject.toml: Expected one of `@`, `(`, `<`, `=`, `>`, `~`, `!`, `;`, found `^`
redjax:ohio_utility_scraper.stdout.txt
1:pyproject.toml:11:11: RUF200 Failed to parse pyproject.toml: Version `` doesn't match PEP 440 rules
agronholm:typeguard.stdout.txt
1:pyproject.toml:40:8: RUF200 Failed to parse pyproject.toml: Expected a valid marker name, found 'python_implementation'
cyuss:decathlon-turnover.stdout.txt
1:pyproject.toml:7:12: RUF200 Failed to parse pyproject.toml: invalid type: string "Youcef", expected a table with 'name' and 'email' keys
ajslater:boilerplate.stdout.txt
1:pyproject.toml:3:17: RUF200 Failed to parse pyproject.toml: invalid type: sequence, expected a string
kaparoo:lightning-project-template.stdout.txt
1:pyproject.toml:56:16: RUF200 Failed to parse pyproject.toml: You can't mix a >= operator with a local version (`+cu117`)
dijital20:pytexas2023-decorators.stdout.txt
1:pyproject.toml:5:11: RUF200 Failed to parse pyproject.toml: Version `` doesn't match PEP 440 rules
pfouque:django-anymail-history.stdout.txt
1:pyproject.toml:137:12: RUF200 Failed to parse pyproject.toml: Version specifier `> = 1.2.0` doesn't match PEP 440 rules
pfouque:django-fakemessages.stdout.txt
1:pyproject.toml:130:12: RUF200 Failed to parse pyproject.toml: Version specifier `> = 1.2.0` doesn't match PEP 440 rules
pypa:build.stdout.txt
1:tests/packages/test-invalid-requirements/pyproject.toml:2:12: RUF200 Failed to parse pyproject.toml: Expected one of `@`, `(`, `<`, `=`, `>`, `~`, `!`, `;`, found `i`
4:tests/packages/test-no-requires/pyproject.toml:1:1: RUF200 Failed to parse pyproject.toml: missing field `requires`
UnoYakshi:FRAAND.stdout.txt
2:  3:11 RUF200 Failed to parse pyproject.toml: Version `` doesn't match PEP 440 rules
DHolmanCoding:python-template.stdout.txt
1:pyproject.toml:22:1: RUF200 Failed to parse pyproject.toml: missing field `requires`
```
Overall, this emitted errors in 43 out of 3408 projects (`rg -c RUF200 target/ecosystem_all_results/ | wc -l`)


Co-authored-by: Micha Reiser <micha@reiser.io>
2023-05-25 12:05:28 +00:00
Charlie Marsh
f4572fe40b
Bump version to 0.0.270 (#4637) 2023-05-24 16:34:29 +00:00
Micha Reiser
85f094f592
Improve Message sorting performance (#4624) 2023-05-24 16:34:48 +02:00
Charlie Marsh
040fb9cef4
Use a separate PrinterFlag for including fix diffs (#4615) 2023-05-24 10:22:37 -04:00
Jonathan Plasse
c6a760e298
Introduce tab-size to correcly calculate the line length with tabulations (#4167) 2023-05-24 08:37:24 +02:00
konstin
3644695bf2
Include hidden ecosystem_ci option to show fixes without feature (#4528) 2023-05-23 22:22:23 -04:00
Charlie Marsh
19c4b7bee6
Rename ruff_python_semantic's Context struct to SemanticModel (#4565) 2023-05-22 02:35:03 +00:00
Aaron Cunningham
41a681531d
Support new extend-per-file-ignores setting (#4265) 2023-05-19 12:24:04 -04:00
konstin
32f1edc555
Create dummy format CLI (#4453)
* Create dummy format CLI

* Hide format from clap, too

Missed that this is a separate option from `#[doc(hidden)]`

* Remove cargo feature and replace with warning

* No-alloc files parameter matching

* beta warning: warn -> warn_user_once

* Rephrase warning
2023-05-19 11:45:52 +02:00
Charlie Marsh
15cb21a6f4
Implement --extend-fixable option (#4297) 2023-05-18 22:20:19 -04:00
Charlie Marsh
d4c0a41b00
Bump version to 0.0.269 (#4506) 2023-05-18 19:45:20 +00:00
Charlie Marsh
8702b5a40a
Bump version to 0.0.268 (#4501) 2023-05-18 15:35:46 -04:00
Charlie Marsh
6b1062ccc3
Enable pycodestyle rules under new "nursery" category (#4407) 2023-05-16 21:21:58 +00:00
Charlie Marsh
dcedd5cd9d
Bump version to 0.0.267 (#4400) 2023-05-12 19:04:56 +00:00
Charlie Marsh
67076b2dcb
Bump version to 0.0.266 (#4391) 2023-05-12 13:11:03 -04:00
konstin
09dbd2029c
Update maturin to maturin 0.15 (#3999)
* Update maturin to maturin>=0.14.17

This allows removing the deprecated `[package.metadata.maturin]`

* Update to maturin 0.15
2023-05-12 15:43:06 +02:00
Jonathan Plasse
1380bd94da
Expose more fields in rule explanation (#4367) 2023-05-11 19:22:23 -04:00
Jonathan Plasse
c10a4535b9
Disallow unreachable_pub (#4314) 2023-05-11 18:00:00 -04:00
Jonathan Plasse
4fd4a65718
Isolate show statistic integration test (#4383) 2023-05-11 21:42:34 +00:00
konstin
0096938789
Optionally show fixes when using --features ecosystem_ci with cargo and --show-fixes at runtime (#4191)
* Generate fixes when using --show-fixes

Example command: `cargo run --bin ruff -- --no-cache --select F401
--show-source --show-fixes
crates/ruff/resources/test/fixtures/pyflakes/F401_9.py`

Before, `--show-fixes` was ignored:

```
crates/ruff/resources/test/fixtures/pyflakes/F401_9.py:4:22: F401 [*] `foo.baz` imported but unused
  |
4 | __all__ = ("bar",)
5 | from foo import bar, baz
  |                      ^^^ F401
  |
  = help: Remove unused import: `foo.baz`

Found 1 error.
[*] 1 potentially fixable with the --fix option.
```

After:

```
crates/ruff/resources/test/fixtures/pyflakes/F401_9.py:4:22: F401 [*] `foo.baz` imported but unused
  |
4 | __all__ = ("bar",)
5 | from foo import bar, baz
  |                      ^^^ F401
  |
  = help: Remove unused import: `foo.baz`

ℹ Suggested fix
1 1 | """Test: late-binding of `__all__`."""
2 2 |
3 3 | __all__ = ("bar",)
4   |-from foo import bar, baz
  4 |+from foo import bar

Found 1 error.
[*] 1 potentially fixable with the --fix option.
```

* Add `--format ecosystem-ci`

* cargo dev generate-all

* Put behind cargo feature

* Regenerate docs

* Don't test ecosystem_ci feature on CI

* Use top level flag instead

* Fix

* Simplify code based on #4191

* Remove old TODO comment
2023-05-10 17:45:57 +02:00
Micha Reiser
853d8354cb
JSON Emitter: Use one indexed column numbers for edits (#4007)
I noticed in the byte-offsets refactor that the `JsonEmitter` uses one indexed column numbers for the diagnostic start and end locations but not for `edits`.

This PR changes the `JsonEmitter` to emit one-indexed column numbers for edits, as we already do for `Message::location` and `Message::end_location`.

## Open questions

~We'll need to change the LSP to subtract 1 from the columns in `_parse_fix`~

6e44fadf8a/ruff_lsp/server.py (L129-L150)

~@charliermarsh is there a way to get the ruff version in that method? If not, then I recommend adding a `version` that we increment whenever we make incompatible changes to the serialized message. We can then use it in the LSP to correctly compute the column offset.~

I'll use the presence of the `Fix::applicability` field to detect if the Ruff version uses one or zero-based column indices.

See https://github.com/charliermarsh/ruff-lsp/pull/103
2023-05-10 17:21:02 +02:00
Micha Reiser
ddbe5a1243
Add Fix::applicability to JSON output (#4341) 2023-05-10 14:34:53 +00:00
Micha Reiser
8969ad5879
Always generate fixes (#4239) 2023-05-10 07:06:14 +00:00
Charlie Marsh
8c2cfade90
Move show_source onto CLI settings group (#4317) 2023-05-09 17:26:25 +00:00
Charlie Marsh
a435c0df4b
Remove deprecated update-check setting (#4313) 2023-05-09 13:10:02 -04:00
Charlie Marsh
d3b71f1e04
Run autofix on initial watcher pass (#4311) 2023-05-09 12:35:32 -04:00
Mikko Leppänen
04e8e74499
Feat: detect changes also in configuration files (#4169) 2023-05-09 16:22:52 +00:00
Zanie Adkins
0801f14046
Refactor Fix and Edit API (#4198) 2023-05-08 11:57:03 +02:00
Charlie Marsh
11e1380df4
Bump version to 0.0.265 (#4248) 2023-05-05 13:16:05 -04:00
Charlie Marsh
b71cc3789f
Change --fix-only exit semantics to mirror --fix (#4146) 2023-05-04 19:03:15 +00:00
Dhruv Manilawala
59d40f9f81
Show settings path in --show-settings output (#4199) 2023-05-04 08:22:31 +02:00
Charlie Marsh
8cb76f85eb
Bump version to 0.0.264 (#4179) 2023-05-01 23:33:38 -07:00
Micha Reiser
cab65b25da
Replace row/column based Location with byte-offsets. (#3931) 2023-04-26 18:11:02 +00:00
Dhruv Manilawala
b9c06b48e1
Document that --diff implies --fix-only (#4098) 2023-04-25 21:19:44 -06:00