## Summary
This is a non-behavior-changing refactor to follow-up
https://github.com/astral-sh/ruff/pull/9321 by modifying
`DisplayParseError` to use owned data and make it useable as a
standalone error type (rather than using references and implementing
`Display`). I don't feel very strongly either way. I thought it was
awkward that the `FormatCommandError` had two branches in the display
path, and wanted to represent the `Parse` vs. other cases as a separate
enum, so here we are.
## Summary
I always found it odd that we had to pass this in, since it's really
higher-level context for the error. The awkwardness is further evidenced
by the fact that we pass in fake values everywhere (even outside of
tests). The source path isn't actually used to display the error; it's
only accessed elsewhere to _re-display_ the error in certain cases. This
PR modifies to instead pass the path directly in those cases.
## Summary
This helps a bit with (but does not close) the issues described in
https://github.com/astral-sh/ruff/issues/9311. E.g., now, we at least
see: `error: Failed to format main.py: source contains syntax errors:
invalid syntax. Got unexpected token '=' at byte offset 20`.
## Summary
This PR adds some helper structs to the linter paths to enable passing
in the pre-computed tokens and parsed source code during benchmarking,
to remove lexing and parsing from the overall linter benchmark
measurement. We already remove parsing for the formatter, and we have
separate benchmarks for the lexer and the parser, so this should make it
much easier to measure linter performance changes.
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
New messages for "format" mode.
Fixes#9132
## Test Plan
<!-- How was it tested? -->
I ran the tests specified in `CONTRIBUTING.md`
```bash
cargo run -p ruff_cli -- check /path/to/some_files.py --no-cache
cargo run -p ruff_cli -- format --check /path/to/some_files.py --no-cache
cargo clippy --workspace --all-targets --all-features -- -D warnings
RUFF_UPDATE_SCHEMA=1 cargo test
pre-commit run --all-files --show-diff-on-failure
```
**Note:** In case no files are detected, either correctly formatted,
changed, or unchanged, it does not display a message. Wouldn't it be
better to show some message in this case?
This PR does the plumbing to make a new formatting option,
`docstring-code-format`, available in the configuration for end users.
It is disabled by default (opt-in). It is opt-in at least initially to
reflect a conservative posture. The intent is to make it opt-out at some
point in the future.
This was split out from #8811 in order to make #8811 easier to merge.
Namely, once this is merged, docstring code snippet formatting will
become available to end users. (See comments below for how we arrived at
the name.)
Closes#7146
## Test Plan
Other than the standard test suite, I ran the formatter over the CPython
and polars projects to ensure both that the result looked sensible and
that tests still passed. At time of writing, one issue that currently
appears is that reformatting code snippets trips the long line lint:
1905886802
## Summary
Adds support for sarif v2.1.0 output to cli, usable via the
output-format paramter.
`ruff . --output-format=sarif`
Includes a few changes I wasn't sure of, namely:
* Adds a few derives for Clone & Copy, which I think could be removed
with a little extra work as well.
## Test Plan
I built and ran this against several large open source projects and
verified that the output sarif was valid, using [Microsoft's SARIF
validator tool](https://sarifweb.azurewebsites.net/Validation)
I've also attached an output of the sarif generated by this version of
ruff on the main branch of django at commit: b287af5dc9
[django_main_b287af5dc9_sarif.json](13626222/django_main_b287af5dc9_sarif.json)
Note: this needs to be regenerated with the latest changes and
confirmed.
## Open Points
[ ] Convert to just using all Rules all the time
[ ] Fix the issue with getting the file URI when compiling for web
assembly
Hides hints about unsafe fixes when they are disabled e.g. with
`--no-unsafe-fixes` or `unsafe-fixes = false`. By default, unsafe fix
hints are still displayed. This seems like a nice way to remove the nag
for users who have chosen not to apply unsafe fixes.
Inspired by comment at
https://github.com/astral-sh/ruff/issues/9063#issuecomment-1850289675
## Summary
This refactors the `ruff_cli` integration tests to create a new
`RuffCheck` struct -- this holds options to configure the "common case"
flags that we want to pass to Ruff (e.g. `--no-cache`, `--isolated`,
etc). This helps reduce the boilerplate and (IMO) makes it more obvious
what the core logic of each test is by keeping only the "interesting"
parameters.
## Test Plan
`cargo test`
Closes https://github.com/astral-sh/ruff/issues/7347Closes#3970 via use of `include`
We could update examples in our documentation, but I worry since we do
not have versioned documentation users on older versions would be
confused. Instead, I'll open an issue to track updating use of `ruff
check .` in the documentation sometime in the future.
## Summary
When running ruff in verbose mode with `-v`, the first debug logs show
where the config settings are taken from. For example:
```
❯ ruff check ./some_file.py -v
[2023-11-17][00:16:25][ruff_cli::resolve][DEBUG] Using pyproject.toml (parent) at /Users/vince/demo/ruff.toml
```
This threw me off for a second because I knew I had no python project
there, and therefore no `pyproject.toml` file. Then I realised it was
actually reading a `ruff.toml` file (obvious when you read the whole
print I suppose) and that the pyproject.toml is a hardcoded string in
the debug log.
I think it would be nice to tweak the wording slightly so it is clear
that the settings don't neccessarily have to come from a
`pyproject.toml` file.
Update to [Rust
1.74](https://blog.rust-lang.org/2023/11/16/Rust-1.74.0.html) and use
the new clippy lints table.
The update itself introduced a new clippy lint about superfluous hashes
in raw strings, which got removed.
I moved our lint config from `rustflags` to the newly stabilized
[workspace.lints](https://doc.rust-lang.org/stable/cargo/reference/workspaces.html#the-lints-table).
One consequence is that we have to `unsafe_code = "warn"` instead of
"forbid" because the latter now actually bans unsafe code:
```
error[E0453]: allow(unsafe_code) incompatible with previous forbid
--> crates/ruff_source_file/src/newlines.rs:62:17
|
62 | #[allow(unsafe_code)]
| ^^^^^^^^^^^ overruled by previous forbid
|
= note: `forbid` lint level was set on command line
```
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
This fixes#2606 by moving where we apply the convention ignores --
instead of applying that at the very end, e track, we now track which
rules have been specifically enabled (via `Specificity::Rule`). If they
have, then we do *not* apply the docstring overrides at the end.
## Test Plan
Added unit tests to `ruff_workspace` and an integration test to
`ruff_cli`
## Summary
This PR updates the formatter to preserve trailing semicolon for Jupyter
Notebooks.
The motivation behind the change is that semicolons in notebooks are
typically used to hide the output, for example when plotting. This is
highlighted in the linked issue.
The conditions required as to when the trailing semicolon should be
preserved are:
1. It should be a top-level statement which is last in the module.
2. For statement, it can be either assignment, annotated assignment, or
augmented assignment. Here, the target should only be a single
identifier i.e., multiple assignments or tuple unpacking isn't
considered.
3. For expression, it can be any.
## Test Plan
Add a new integration test in `ruff_cli`. The test notebook basically
acts as a document as to which trailing semicolons are to be preserved.
fixes: #8254
## Summary
When you run Ruff via stdin, and pass `format` or `check --fix`, we
typically write the changed or unchanged contents to stdout. It turns
out we forgot to do this when the file is _excluded_, so if you run
`ruff format /path/to/excluded/file.py`, we don't write _anything_ to
`stdout`. This led to a bug in the LSP whereby we deleted file contents
for third-party files.
The right thing to do here is write back the unchanged contents, as it
should always be safe to write the output of stdout back to a file.
## Summary
This PR addresses the incompatibility with `jupyterlab-lsp` +
`python-lsp-ruff` arising from the inference of source type from file
extension, raised in #6847.
In particular it follows the suggestion in
https://github.com/astral-sh/ruff/issues/6847#issuecomment-1765724679 to
specify a mapping from file extension to source type.
The source types are
- python
- pyi
- ipynb
Usage:
```sh
ruff check --no-cache --stdin-filename Untitled.ipynb --extension ipynb:python
```
Unlike the original suggestion, `:` instead of `=` is used to associate
file extensions to language since that is what is used with
`--per-file-ignores` which is an existing option that accepts a mapping.
## Test Plan
2 tests added to `integration_test.rs` to ensure the override works as
expected
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
Prior to this change `extend_unsafe_fixes` took precedence over
`extend_safe_fixes` selectors, so any conflicts were resolved in favour
of `extend_unsafe_fixes`. Thanks to that ruff were conservatively
assuming that if configs conlict the fix corresponding to selected rule
will be treated as unsafe.
After this change we take into account Specificity of the selectors. For
conflicts between selectors of the same Specificity we will treat the
corresponding fixes as unsafe. But if the conflicting selectors are of
different specificity the more specific one will win.
## Test Plan
Tests were added for the `FixSafetyTable` struct. The
`check_extend_unsafe_fixes_conflict_with_extend_safe_fixes_by_specificity`
integration test was added to test conflicting rules of different
specificity.
Fixes#8404
---------
Co-authored-by: Zanie <contact@zanie.dev>
## Summary
Being able to set `--no-cache` without touching the command line makes
comparing formatter speed with e.g. Hyperfine a lot easier; Black allows
one to set `BLACK_CACHE_DIR=/dev/null`, but setting
`RUFF_CACHE_DIR=/dev/null` has Ruff choke:
```
error: Failed to initialize cache at /dev/null: Not a directory (os error 20)
error: Failed to initialize cache at /dev/null: Not a directory (os error 20)
warning: Failed to open cache file '/dev/null/0.1.4/18160934645386409287': Not a directory (os error 20)
```
Alternately, we could make a `/dev/null` (or `nul` on Windows) cache
directory imply `--no-cache`?
## Test Plan
None yet.
## Summary
We typically avoid enforcing exclusions if a file was passed to Ruff
directly on the CLI. However, we also allow `--force-exclude`, which
ignores excluded files _even_ if they're passed to Ruff directly. This
is really important for pre-commit, which always passes changed files --
we need to exclude files passed by pre-commit if they're in the
`exclude` lists.
Turns out the new `lint.exclude` and `format.exclude` settings weren't
respecting `--force-exclude`.
Closes https://github.com/astral-sh/ruff/issues/8391.
## Summary
Uses `warn_user_once!` instead of `warn!` to ensure that every warning
is shown exactly once, regardless of whether there are duplicates in the
list, or warnings that are raised by multiple configuration files.
Closes#8271.
## Summary
If a file has no diagnostics, then we can read and write that
information from and to the cache, even if the fix mode is `--fix` or
`--diff`. (Typically, we can't read or write such results from or to the
cache, because `--fix` and `--diff` have side effects that take place
during diagnostic analysis (writing to disk or outputting the diff).)
This greatly improves performance when running `--fix` on a codebase in
the common case (few diagnostics).
Closes#8311.
Closes https://github.com/astral-sh/ruff/issues/8315.
## Summary
Since `--format` was changed to `--output-format` for `check`, it feels
like it makes sense for the same to work for the auxiliary commands.
This
* adds the same deprecation warning that used to be a thing in #7514
(and un-became a thing in #7984)
Fixes#7990.
## Test Plan
* `cargo run --bin=ruff -- rule --all --output-format=json` works
* `cargo run --bin=ruff -- rule --format=json` works with warnings
## Summary
Based on [this
feedback](https://github.com/astral-sh/ruff/issues/8185#issuecomment-1780092525).
Avoid warning about `force-wrap-aliases` and `split-on-trailing-comma`
if `force-single-line` is true (which creates a dedicated import for
each imported member).
## Test Plan
Ran `ruff format . --no-cache` and verified that the warning show up
when `force-single-line=false` and aren't shown when
`force-single-line=true`
## Summary
Avoid warning about incompatible rules except if their configuration
directly conflicts with the formatter. This should reduce the noise and
potentially the need for https://github.com/astral-sh/ruff/issues/8175
and https://github.com/astral-sh/ruff/issues/8185
I also extended the rule and option documentation to mention any
potential formatter incompatibilities or whether they're redundant when
using the formatter.
* `LineTooLong`: This is a use case we explicitly want to support. Don't
warn about it
* `TabIndentation`, `IndentWithSpaces`: Only warn if
`indent-style="tab"`
* `IndentationWithInvalidMultiple`,
`IndentationWithInvalidMultipleComment`: Only warn if `indent-width !=
4`
* `OverIndented`: Don't warn, but mention that the rule is redundant
* `BadQuotesInlineString`: Warn if quote setting is different from
`format.quote-style`
* `BadQuotesMultilineString`, `BadQuotesDocstring`: Warn if `quote !=
"double"`
## Test Plan
I added a new integration test for the default configuration with `ALL`.
`ruff format` now only shows two incompatible rules, which feels more
reasonable.
## Summary
This PR refactors the formatter diff code to reuse the
`SourceKind::diff` logic. This has the benefit that the Notebook diff
now includes the cell numbers which was not present before.
## Test Plan
Update the snapshots and verified the cell numbers.
## Summary
This PR removes the `todo!()` around `IpyEscapeCommand` in the
formatter.
The `NeedsParentheses` trait needs to be implemented which always return
`Never`. The reason being that if an escape command is parenthesized,
then that's not parsed as an escape command. IOW, the parentheses
shouldn't be present around an escape command.
In the similar way, the `CanSkipOptionalParenthesesVisitor` will skip
this node.
## Test Plan
Updated the `unformatted.ipynb` fixture with new cells containing
IPython escape commands and the corresponding snapshot was verified.
Also, tested it out in a few open source repositories containing
notebooks (`openai/openai-cookbook`, `huggingface/notebooks`).
#### New cells in `unformatted.ipynb`
**Cell 2**
```markdown
A markdown cell
```
**Cell 3**
```python
def some_function(foo, bar):
pass
%matplotlib inline
```
**Cell 4**
```python
foo = %pwd
def some_function(foo,bar,):
foo = %pwd
print(foo
)
```
fixes: #8204
## Summary
Related to https://github.com/astral-sh/ruff/issues/8135.
If we're not printing a `--diff`, or a summary of `--check` changes, we
can avoid sorting the list of results. Further, when sorting, we only
need to sort a small subset of the entries, in the common case (i.e., in
general, it's much more likely that a file is formatted than not).
## Test Plan
Local benchmarks suggest a 5-10% speedup on the cached behavior:
```
❯ hyperfine --warmup 3 "./target/release/ruff format ../airflow" "./target/release/sort format ../airflow"
Benchmark 1: ./target/release/ruff format ../airflow
Time (mean ± σ): 70.3 ms ± 5.2 ms [User: 52.1 ms, System: 59.0 ms]
Range (min … max): 68.3 ms … 101.7 ms 42 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Benchmark 2: ./target/release/sort format ../airflow
Time (mean ± σ): 66.0 ms ± 1.4 ms [User: 48.3 ms, System: 58.4 ms]
Range (min … max): 64.7 ms … 71.8 ms 44 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Summary
'./target/release/sort format ../airflow' ran
1.07 ± 0.08 times faster than './target/release/ruff format ../airflow'
```
## Summary
This PR renames the `tab-size` configuration option to `indent-width` to
express that the formatter uses the option to determine the indentation
width AND as tab width.
I first preferred naming the option `tab-width` but then decided to go
with `indent-width` because:
* It aligns with the `indent-style` option
* It would allow us to write a lint rule that asserts that each
indentation uses `indent-width` spaces.
Closes#7643
## Test Plan
Added integration test
## Summary
This PR introduces a new `pycodestyl.max-line-length` option that allows overriding the global `line-length` option for `E501` only.
This is useful when using the formatter and `E501` together, where the formatter uses a lower limit and `E501` is only used to catch extra-long lines.
Closes#7644
## Considerations
~~Our fix infrastructure asserts in some places that the fix doesn't exceed the configured `line-width`. With this change, the question is whether it should use the `pycodestyle.max-line-width` or `line-width` option to make that decision.
I opted for the global `line-width` for now, considering that it should be the lower limit. However, this constraint isn't enforced and users not using the formatter may only specify `pycodestyle.max-line-width` because they're unaware of the global option (and it solves their need).~~
~~I'm interested to hear your thoughts on whether we should use `pycodestyle.max-line-width` or `line-width` to decide on whether to emit a fix or not.~~
Edit: The linter users `pycodestyle.max-line-width`. The `line-width` option has been removed from the `LinterSettings`
## Test Plan
Added integration test. Built the documentation and verified that the links are correct.
