Commit graph

700 commits

Author SHA1 Message Date
konsti
af95cbaeef
Add newline after module docstrings in preview style (#8283)
Change
```python
"""Test docstring"""
a = 1
```
to
```python
"""Test docstring"""

a = 1
```
in preview style, but don't touch the docstring otherwise.

Do we want to ask black to also format the content of module level
docstrings? Seems inconsistent to me that we change function and class
docstring indentation/contents but not module docstrings.

Fixes https://github.com/astral-sh/ruff/issues/7995
2023-10-28 01:16:50 +00:00
konsti
317d3dd612
Add test and basic implementation for formatter preview mode (#8044)
**Summary** Prepare for the black preview style becoming the black
stable style at the end of the year.

This adds a new test file to compare stable and preview on some relevant
preview options in black, and makes `format_dev` understand the black
preview flag. I've added poetry as a project that uses preview.

I've implemented one specific deviation (collapsing of stub
implementation in non-stub files) which showed up in poetry for testing.
This also improves poetry compatibility from 0.99891 to 0.99919.

Fixes #7440

New compatibility stats:
| project | similarity index | total files | changed files |

|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75803 | 1799 | 1647 |
| django | 0.99983 | 2772 | 35 |
| home-assistant | 0.99953 | 10596 | 189 |
| poetry | 0.99919 | 317 | 12 |
| transformers | 0.99963 | 2657 | 332 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99978 | 3669 | 20 |
| warehouse | 0.99969 | 654 | 15 |
| zulip | 0.99970 | 1459 | 22 |
2023-10-26 15:33:26 +00:00
Micha Reiser
f5e850745c
Only omit optional parentheses for starting or ending with parentheses (#8238) 2023-10-26 07:28:58 +01:00
Charlie Marsh
88c8b47326
Avoid introducing new parentheses in annotated assignments (#8233)
## Summary

We decided to avoid changing this in
https://github.com/astral-sh/ruff/issues/7315, but it's been reported
multiple times (e.g., in https://github.com/astral-sh/ruff/issues/8226,
also on Discord). I suggest we change it to improve compatibility. In
general, it also seems to lend itself to better code style.

Closes #8188 
Closes #8226

## Test Plan

Shows improvements for CPython, home-assistant, Poetry, and typeshed.

Before:

| project | similarity index | total files | changed files |

|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75803 | 1799 | 1647 |
| django | 0.99983 | 2772 | 34 |
| home-assistant | 0.99953 | 10596 | 186 |
| poetry | 0.99891 | 317 | 17 |
| transformers | 0.99966 | 2657 | 330 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99978 | 3669 | 20 |
| warehouse | 0.99977 | 654 | 13 |
| zulip | 0.99970 | 1459 | 22 |

After:

| project | similarity index | total files | changed files |

|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75804 | 1799 | 1647 |
| django | 0.99983 | 2772 | 34 |
| home-assistant | 0.99960 | 10596 | 156 |
| poetry | 0.99897 | 317 | 17 |
| transformers | 0.99966 | 2657 | 330 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99980 | 3669 | 18 |
| warehouse | 0.99977 | 654 | 13 |
| zulip | 0.99970 | 1459 | 22 |
2023-10-25 22:51:50 -04:00
Micha Reiser
6983d96d27
Fix fmt:off with trailing child comment (#8234) 2023-10-26 01:03:34 +00:00
Charlie Marsh
3c3d9ab173
Insert necessary blank line between class and leading comments (#8224)
## Summary

Given:

```python
# comment

class A:
    def foo(self):
        pass
```

We need to insert an additional newline between `# comment` and `class
A`. We were missing this handling for the case in which `# comment` is a
leading comment on `class A`, as opposed to a trailing comment of some
preceding statement.

In practice, I think this only applies to the specific case in which a
class or function is the first statement in a module, and there's a
single empty line between a leading comment and that class or function.
If there are no empty lines, then the comment "sticks" to the
definition; if there are two or more, then `leading_comments` will
truncate appropriately. If the class or function is nested, then we only
need one empty line anyway.

Closes https://github.com/astral-sh/ruff/issues/8215.

## Test Plan

No change in similarity.

Before:

| project | similarity index | total files | changed files |

|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75803 | 1799 | 1647 |
| django | 0.99983 | 2772 | 34 |
| home-assistant | 0.99953 | 10596 | 186 |
| poetry | 0.99891 | 317 | 17 |
| transformers | 0.99966 | 2657 | 330 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99978 | 3669 | 20 |
| warehouse | 0.99977 | 654 | 13 |
| zulip | 0.99970 | 1459 | 22 |

After:

| project | similarity index | total files | changed files |

|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75803 | 1799 | 1648 |
| django | 0.99983 | 2772 | 34 |
| home-assistant | 0.99953 | 10596 | 186 |
| poetry | 0.99891 | 317 | 17 |
| transformers | 0.99966 | 2657 | 330 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99978 | 3669 | 20 |
| warehouse | 0.99977 | 654 | 13 |
| zulip | 0.99970 | 1459 | 22 |
2023-10-25 20:31:59 -04:00
Dhruv Manilawala
dbd84c947b
Formatter parentheses support for IpyEscapeCommand (#8207)
## Summary

This PR removes the `todo!()` around `IpyEscapeCommand` in the
formatter.

The `NeedsParentheses` trait needs to be implemented which always return
`Never`. The reason being that if an escape command is parenthesized,
then that's not parsed as an escape command. IOW, the parentheses
shouldn't be present around an escape command.

In the similar way, the `CanSkipOptionalParenthesesVisitor` will skip
this node.

## Test Plan

Updated the `unformatted.ipynb` fixture with new cells containing
IPython escape commands and the corresponding snapshot was verified.
Also, tested it out in a few open source repositories containing
notebooks (`openai/openai-cookbook`, `huggingface/notebooks`).

#### New cells in `unformatted.ipynb`

**Cell 2**
```markdown
A markdown cell
```

**Cell 3**
```python
def some_function(foo, bar):
    pass
%matplotlib inline
```

**Cell 4**
```python
foo = %pwd
def some_function(foo,bar,):
	foo = %pwd
    print(foo
	)
```

fixes: #8204
2023-10-25 14:01:50 +00:00
Dhruv Manilawala
c2ec5f0bc9
Use source type to determine parser mode for formatting (#8205)
## Summary

This PR fixes the bug where if a Notebook contained IPython syntax, then
the format command would fail. This was because the correct mode was not
being used while parsing through the formatter code path.

## Test Plan

This PR isn't the only requirement for Notebook formatting to start
working with IPython escape commands. The following PR in the stack is
required as well.
2023-10-25 19:20:02 +05:30
Micha Reiser
fd07a12a52
Refine warning about incompatible isort settings (#8192) 2023-10-25 08:41:17 +01:00
Micha Reiser
e36afc3324
Avoid space around pow for None, True and False (#8189) 2023-10-25 07:24:06 +01:00
Micha Reiser
8b665f40c8
Avoid parenthesizing octal/hex or binary literals in object positions (#8160) 2023-10-24 15:12:52 +01:00
Dhruv Manilawala
2e81b9c391
Don't move type param opening parenthesis comment (#8163)
## Summary

This PR fixes the issue to avoid collapsing the type param declaration
if
there's a comment after the opening parenthesis. For example,

```python
type foo[  # comment
    A,
    B
] = int
```

Here, we'll preserve the comment on the same line as is being done for
other
similar type of nodes.

## Test Plan

Add a new test case for it, update the snapshots, and validate the
ecosystem
check.

### Formatter ecosystem

#### `main`

| project | similarity index | total files | changed files |

|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75803 | 1799 | 1647 |
| django | 0.99983 | 2772 | 34 |
| home-assistant | 0.99953 | 10596 | 186 |
| poetry | 0.99891 | 317 | 17 |
| transformers | 0.99966 | 2657 | 330 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99978 | 3669 | 20 |
| warehouse | 0.99977 | 654 | 13 |
| zulip | 0.99970 | 1459 | 22 |

#### `dhruv/type-params`

| project | similarity index | total files | changed files |

|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75803 | 1799 | 1647 |
| django | 0.99983 | 2772 | 34 |
| home-assistant | 0.99953 | 10596 | 186 |
| poetry | 0.99891 | 317 | 17 |
| transformers | 0.99966 | 2657 | 330 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99978 | 3669 | 20 |
| warehouse | 0.99977 | 654 | 13 |
| zulip | 0.99970 | 1459 | 22 |

fixes: #8162
2023-10-24 12:02:27 +00:00
Charlie Marsh
95702e408f
Respect parenthesized generators in has_own_parentheses (#8100)
## Summary

When analyzing:

```python
if "root" not in (
    long_tree_name_tree.split("/")[0]
    for long_tree_name_tree in really_really_long_variable_name
):
    msg = "Could not find root. Please try a different forest."
    raise ValueError(msg)
```

We missed that the generator expression is parenthesized, because the
parentheses are _part_ of the generator -- so
`is_expression_parenthesized` returns `False`. We needed to take into
account that generators and tuples may or may not be parenthesized when
determining whether we can omit parentheses while splitting an
expression.

Closes https://github.com/astral-sh/ruff/issues/8090.

## Test Plan

No changes in similarity.

Before:

| project | similarity index | total files | changed files |

|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75803 | 1799 | 1647 |
| django | 0.99983 | 2772 | 34 |
| home-assistant | 0.99953 | 10596 | 186 |
| poetry | 0.99891 | 317 | 17 |
| transformers | 0.99966 | 2657 | 330 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99978 | 3669 | 20 |
| warehouse | 0.99977 | 654 | 13 |
| zulip | 0.99970 | 1459 | 22 |

After:

| project | similarity index | total files | changed files |

|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75803 | 1799 | 1647 |
| django | 0.99983 | 2772 | 34 |
| home-assistant | 0.99953 | 10596 | 186 |
| poetry | 0.99891 | 317 | 17 |
| transformers | 0.99966 | 2657 | 330 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99978 | 3669 | 20 |
| warehouse | 0.99977 | 654 | 13 |
| zulip | 0.99970 | 1459 | 22 |
2023-10-22 19:58:25 -04:00
konsti
8f9753f58e
Comments outside expression parentheses (#7873)
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

Fixes https://github.com/astral-sh/ruff/issues/7448
Fixes https://github.com/astral-sh/ruff/issues/7892

I've removed automatic dangling comment formatting, we're doing manual
dangling comment formatting everywhere anyway (the
assert-all-comments-formatted ensures this) and dangling comments would
break the formatting there.

## Test Plan

New test file.

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2023-10-19 09:24:11 +00:00
Charlie Marsh
2729c4cacd
Skip over parentheses when detecting in keyword (#8054)
## Summary

Given an expression like `[x for (x) in y]`, we weren't skipping over
parentheses when searching for the `in` between `(x)` and `y`.

Closes https://github.com/astral-sh/ruff/issues/8053.
2023-10-18 19:13:58 -04:00
konsti
0c3123e07e
Insert newline after nested function or class statements (#7946)
**Summary** Insert a newline after nested function and class
definitions, unless there is a trailing own line comment.

We need to e.g. format
```python
if platform.system() == "Linux":
    if sys.version > (3, 10):
        def f():
            print("old")
    else:
        def f():
            print("new")
    f()
```
as
```python
if platform.system() == "Linux":
    if sys.version > (3, 10):

        def f():
            print("old")

    else:

        def f():
            print("new")

    f()
```
even though `f()` is directly preceded by an if statement, not a
function or class definition. See the comments and fixtures for trailing
own line comment handling.

**Test Plan** I checked that the new content of `newlines.py` matches
black's formatting.

---------

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
2023-10-18 09:45:58 +00:00
Micha Reiser
fe485d791c
Add [format|lint].exclude options (#8000) 2023-10-18 01:15:25 +00:00
Charlie Marsh
d685107638
Move {AnyNodeRef, AstNode} to ruff_python_ast crate root (#8030)
This is a do-over of https://github.com/astral-sh/ruff/pull/8011, which
I accidentally merged into a non-`main` branch. Sorry!
2023-10-18 00:01:18 +00:00
Charlie Marsh
aa6846c78c
Add trailing zero between dot and exponential (#7956)
Closes https://github.com/astral-sh/ruff/issues/7952.
2023-10-15 21:42:00 -04:00
Charlie Marsh
3d03e75a9d
Force parentheses for power operations in unary expressions (#7955)
## Summary

E.g., given `-10**100`, reformat as `-(10**100)`.

Black special cases this (https://github.com/psf/black/pull/909) and
it's currently a deviation.

Closes https://github.com/astral-sh/ruff/issues/7951.
2023-10-15 21:41:50 -04:00
konsti
3944c42d4c
Format comment before parameter default correctly (#7870)
**Summary** Handle comment before the default values of function
parameters correctly by inserting a line break instead of space after
the equals sign where required.

```python
def f(
    a = # parameter trailing comment; needs line break
    1,
    b =
    # default leading comment; needs line break
    2,
    c = ( # the default leading can only be end-of-line with parentheses; no line break
        3
    ),
    d = (
        # own line leading comment with parentheses; no line break
        4
    )
)
```

Fixes #7603

**Test Plan** Added the different cases and one more complex case as
fixtures.
2023-10-12 17:50:12 +02:00
konsti
0f759af3cf
Remove spaces from import statements (#7859)
**Summary** Remove spaces from import statements such as 

```python
import tqdm .  tqdm
from tqdm .    auto import tqdm
```

See also #7760 for a better solution.

**Test Plan** New fixtures
2023-10-11 11:35:41 +00:00
konsti
644011fb14
Formatter quoting for f-strings with triple quotes (#7826)
**Summary** Quoting of f-strings can change if they are triple quoted
and only contain single quotes inside.

Fixes #6841

**Test Plan** New fixtures

---------

Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
2023-10-11 11:30:34 +00:00
Dhruv Manilawala
a1509dfc7c
Use correct start location for class/function clause header (#7802)
## Summary

This PR fixes the bug where the formatter would panic if a class/function with
decorators had a suppression comment.

The fix is to use to correct start location to find the `async`/`def`/`class`
keyword when decorators are present which is the end of the last
decorator.

## Test Plan

Add test cases for the fix and update the snapshots.
2023-10-04 07:55:01 +00:00
Charlie Marsh
c71ff7eae1
Avoid printing continuations within import identifiers (#7744)
## Summary

It turns out that _some_ identifiers can contain newlines --
specifically, dot-delimited import identifiers, like:
```python
import foo\
    .bar
```

At present, we print all identifiers verbatim, which causes us to retain
the `\` in the formatted output. This also leads to violating some debug
assertions (see the linked issue, though that's a symptom of this
formatting failure).

This PR adds detection for import identifiers that contain newlines, and
formats them via `text` (slow) rather than `source_code_slice` (fast) in
those cases.

Closes https://github.com/astral-sh/ruff/issues/7734.

## Test Plan

`cargo test`
2023-10-02 09:51:07 -04:00
Charlie Marsh
d8a6279fe5
Remove string allocation in relative import formatting (#7743) 2023-10-01 18:15:43 +00:00
Charlie Marsh
b42a8972bf
Use Expr::is_* methods in more matches (#7714) 2023-09-29 17:28:50 +00:00
Micha Reiser
e2ec42539b
Attach dangling comments to the comprehension instead of the if or iter nodes (#7693) 2023-09-29 10:45:01 +01:00
Dhruv Manilawala
e62e245c61
Add support for PEP 701 (#7376)
## Summary

This PR adds support for PEP 701 in Ruff. This is a rollup PR of all the
other individual PRs. The separate PRs were created for logic separation
and code reviews. Refer to each pull request for a detail description on
the change.

Refer to the PR description for the list of pull requests within this PR.

## Test Plan

### Formatter ecosystem checks

Explanation for the change in ecosystem check:
https://github.com/astral-sh/ruff/pull/7597#issue-1908878183

#### `main`

```
| project      | similarity index  | total files       | changed files     |
|--------------|------------------:|------------------:|------------------:|
| cpython      |           0.76083 |              1789 |              1631 |
| django       |           0.99983 |              2760 |                36 |
| transformers |           0.99963 |              2587 |               319 |
| twine        |           1.00000 |                33 |                 0 |
| typeshed     |           0.99983 |              3496 |                18 |
| warehouse    |           0.99967 |               648 |                15 |
| zulip        |           0.99972 |              1437 |                21 |
```

#### `dhruv/pep-701`

```
| project      | similarity index  | total files       | changed files     |
|--------------|------------------:|------------------:|------------------:|
| cpython      |           0.76051 |              1789 |              1632 |
| django       |           0.99983 |              2760 |                36 |
| transformers |           0.99963 |              2587 |               319 |
| twine        |           1.00000 |                33 |                 0 |
| typeshed     |           0.99983 |              3496 |                18 |
| warehouse    |           0.99967 |               648 |                15 |
| zulip        |           0.99972 |              1437 |                21 |
```
2023-09-29 02:55:39 +00:00
Charlie Marsh
695dbbc539
Always prefer double quotes for docstrings and triple-quoted srings (#7680)
## Summary

At present, `quote-style` is used universally. However, [PEP
8](https://peps.python.org/pep-0008/) and [PEP
257](https://peps.python.org/pep-0257/) suggest that while either single
or double quotes are acceptable in general (as long as they're
consistent), docstrings and triple-quoted strings should always use
double quotes. In our research, the vast majority of Ruff users that
enable the `flake8-quotes` rules only enable them for inline strings
(i.e., non-triple-quoted strings).

Additionally, many Black forks (like Blue and Pyink) use double quotes
for docstrings and triple-quoted strings.

Our decision for now is to always prefer double quotes for triple-quoted
strings (which should include docstrings). Based on feedback, we may
consider adding additional options (e.g., a `"preserve"` mode, to avoid
changing quotes; or a `"multiline-quote-style"` to override this).

Closes https://github.com/astral-sh/ruff/issues/7615.

## Test Plan

`cargo test`
2023-09-28 15:11:33 -04:00
Charlie Marsh
f62b4c801f
Extend pragma comment cases (#7687)
## Summary

Extends the pragma comment detection in the formatter to support
case-insensitive `noqa` (as supposed by Ruff), plus a variety of other
pragmas (`isort:`, `nosec`, etc.).

Also extracts the detection out into the trivia crate so that we can
reuse it in the linter (see:
https://github.com/astral-sh/ruff/issues/7471).

## Test Plan

`cargo test`
2023-09-28 18:55:19 +00:00
Charlie Marsh
46b85ab0a9
Misc. follow-ups to single-element tuple patterns (#7698)
Just changes to internal comments and tests.

See comments in https://github.com/astral-sh/ruff/pull/7683.
2023-09-28 18:49:13 +00:00
Charlie Marsh
1c02fcd7ce
Avoid unnecessary comments check in maybe_parenthesize_expression (#7686)
## Summary

No-op refactor, but we can evaluate early if the first part of
`preserve_parentheses || has_comments` is `true`, and thus avoid looking
up the node comments.

## Test Plan

`cargo test`
2023-09-28 13:42:12 -04:00
Micha Reiser
f53c410ff8
Prefer preserving WithItem parentheses (#7694) 2023-09-28 14:42:40 +01:00
Charlie Marsh
8028de8956
Improve some comments in normalize_comment (#7688) 2023-09-28 03:08:25 +00:00
Charlie Marsh
a6d79c03b3
Break with on end-of-line trailing comments (#7685)
## Summary

Ensures that:

```python
with (
    a  # comment
):
    pass
```

Retains its parentheses.

Closes https://github.com/astral-sh/ruff/issues/6750.

## Test Plan

`cargo test`
2023-09-28 00:16:40 +00:00
Charlie Marsh
58b50a6290
Avoid expanding single-element tuple patterns (#7683)
## Summary

The formatting for tuple patterns is now intended to match that of `for`
loops:

- Always parenthesize single-element tuples.
- Don't break on the trailing comma in single-element tuples.
- For other tuples, preserve the parentheses, and insert if-breaks.

Closes https://github.com/astral-sh/ruff/issues/7681.

## Test Plan

`cargo test`
2023-09-27 23:57:18 +00:00
konsti
4d16e2308d
Formatter and parser refactoring (#7569)
I got confused and refactored a bit, now the naming should be more
consistent. This is the basis for the range formatting work.

Chages:
* `format_module` -> `format_module_source` (format a string)
* `format_node` -> `format_module_ast` (format a program parsed into an
AST)
* Added `parse_ok_tokens` that takes `Token` instead of `Result<Token>`
* Call the source code `source` consistently
* Added a `tokens_and_ranges` helper
* `python_ast` -> `module` (because that's the type)
2023-09-26 15:29:43 +02:00
Charlie Marsh
17ceb5dcb3
Preserve newlines after nested compound statements (#7608)
## Summary

Given:
```python
if True:
    if True:
        pass
    else:
        pass
        # a

        # b
        # c

else:
    pass
```

We want to preserve the newline after the `# c` (before the `else`).
However, the `last_node` ends at the `pass`, and the comments are
trailing comments on the `pass`, not trailing comments on the
`last_node` (the `if`). As such, when counting the trailing newlines on
the outer `if`, we abort as soon as we see the comment (`# a`).

This PR changes the logic to skip _all_ comments (even those with
newlines between them). This is safe as we know that there are no
"leading" comments on the `else`, so there's no risk of skipping those
accidentally.

Closes https://github.com/astral-sh/ruff/issues/7602.

## Test Plan

No change in compatibility.

Before:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| transformers | 0.99963 | 2587 | 319 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99979 | 3496 | 22 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |

After:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| transformers | 0.99963 | 2587 | 319 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99983 | 3496 | 18 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |
2023-09-25 14:21:44 +00:00
Charlie Marsh
865c89800e
Avoid searching for bracketed comments in unparenthesized generators (#7627)
Similar to tuples, a generator _can_ be parenthesized or
unparenthesized. Only search for bracketed comments if it contains its
own parentheses.

Closes https://github.com/astral-sh/ruff/issues/7623.
2023-09-24 02:08:44 +00:00
Charlie Marsh
1a4f2a9baf
Avoid reordering mixed-indent-level comments after branches (#7609)
## Summary

Given:

```python
if True:
    if True:
        if True:
            pass

        #a
            #b
        #c
else:
    pass
```

When determining the placement of the various comments, we compute the
indentation depth of each comment, and then compare it to the depth of
the previous statement. It turns out this can lead to reordering
comments, e.g., above, `#b` is assigned as a trailing comment of `pass`,
and so gets reordered above `#a`.

This PR modifies the logic such that when we compute the indentation
depth of `#b`, we limit it to at most the indentation depth of `#a`. In
other words, when analyzing comments at the end of branches, we don't
let successive comments go any _deeper_ than their preceding comments.

Closes https://github.com/astral-sh/ruff/issues/7602.

## Test Plan

`cargo test`

No change in similarity.

Before:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| transformers | 0.99963 | 2587 | 319 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99979 | 3496 | 22 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |

After:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| transformers | 0.99963 | 2587 | 319 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99979 | 3496 | 22 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |
2023-09-22 18:12:31 -04:00
Charlie Marsh
5174e8c926
Ignore blank lines between comments when counting newlines-after-imports (#7607)
## Summary

Given:

```python
# -*- coding: utf-8 -*-
import random

# Defaults for arguments are defined here
# args.threshold = None;


logger = logging.getLogger("FastProject")
```

We want to count the number of newlines after `import random`, to ensure
that there's _at least one_, but up to two.

Previously, we used the end range of the statement (then skipped
trivia); instead, we need to use the end of the _last comment_. This is
similar to #7556.

Closes https://github.com/astral-sh/ruff/issues/7604.
2023-09-22 17:49:39 +00:00
Micha Reiser
9d16e46129
Add most formatter options to ruff.toml / pyproject.toml (#7566) 2023-09-22 15:47:57 +00:00
Charlie Marsh
d7508af48d
Truncate to one empty line in stub files (#7558)
## Summary

This PR modifies a variety of sites in which we insert up to two empty
lines to instead truncate to at most one empty line in stub files. We
already enforce this in _some_ places, but not all.

## Test Plan

`cargo test`

No changes in similarity (as expected, since this only impacts
unformatted `.pyi` files).

Before:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| transformers | 0.99963 | 2587 | 323 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99979 | 3496 | 22 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |

After:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| transformers | 0.99963 | 2587 | 323 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99979 | 3496 | 22 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |
2023-09-21 16:24:42 -04:00
Charlie Marsh
4d6f5ff0a7
Remove Int wrapper type from parser (#7577)
## Summary

This is only used for the `level` field in relative imports (e.g., `from
..foo import bar`). It seems unnecessary to use a wrapper here, so this
PR changes to a `u32` directly.
2023-09-21 17:01:44 +00:00
Charlie Marsh
7f1456a2c9
Allow up to two newlines before trailing clause body comments (#7575)
## Summary

This is the peer to https://github.com/astral-sh/ruff/pull/7557, but for
"leading" clause comments, like:

```python
if True:
    pass


# comment
else:
    pass
```

In this case, we again want to allow up to two newlines at the top
level.

## Test Plan

`cargo test`

No changes.

Before:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| transformers | 0.99963 | 2587 | 323 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99979 | 3496 | 22 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |

After:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| transformers | 0.99963 | 2587 | 323 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99979 | 3496 | 22 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |
2023-09-21 14:52:38 +00:00
Charlie Marsh
2759db6604
Allow up to two newlines after trailing clause body comments (#7557)
## Summary

The number of newlines after a trailing comment in a clause body needs
to follow the usual rules -- so, up to two for top-level, up to one for
nested, etc.

For example, Black preserves both newlines after `# comment` here:

```python
if True:
    pass

    # comment


else:
    pass
```

But it truncates to one newline here:

```python
if True:
    if True:
        pass
        # comment


    else:
        pass
else:
    pass
```

## Test Plan

Significant improvement on `transformers`.

Before:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| transformers | 0.99957 | 2587 | 402 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99979 | 3496 | 22 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |


After:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| **transformers** | **0.99963** | **2587** | **323** |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99979 | 3496 | 22 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |
2023-09-21 14:04:49 +00:00
Charlie Marsh
124d95d246
Fix instability in trailing clause body comments (#7556)
## Summary

When we format the trailing comments on a clause body, we check if there
are any newlines after the last statement; if not, we insert one.

This logic didn't take into account that the last statement could itself
have trailing comments, as in:

```python
if True:
    pass

    # comment
else:
    pass
```

We were thus inserting a newline after the comment, like:

```python
if True:
    pass

    # comment

else:
    pass
```

In the context of function definitions, this led to an instability,
since we insert a newline _after_ a function, which would in turn lead
to the bug above appearing in the second formatting pass.

Closes https://github.com/astral-sh/ruff/issues/7465.

## Test Plan

`cargo test`

Small improvement in `transformers`, but no regressions.

Before:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| transformers | 0.99956 | 2587 | 404 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99983 | 3496 | 18 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |

After:

| project | similarity index | total files | changed files |

|--------------|------------------:|------------------:|------------------:|
| cpython | 0.76083 | 1789 | 1631 |
| django | 0.99983 | 2760 | 36 |
| **transformers** | **0.99957** | **2587** | **402** |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99983 | 3496 | 18 |
| warehouse | 0.99967 | 648 | 15 |
| zulip | 0.99972 | 1437 | 21 |
2023-09-21 13:32:16 +00:00
Micha Reiser
f8f1cd5016
Introduce FormatterSettings (#7545) 2023-09-21 08:01:24 +02:00
Charlie Marsh
5df0326bc8
Treat parameters-with-newline as empty in function formatting (#7550)
## Summary

If a function has no parameters (and no comments within the parameters'
`()`), we're supposed to wrap the return annotation _whenever_ it
breaks. However, our `empty_parameters` test didn't properly account for
the case in which the parameters include a newline (but no other
content), like:

```python
def get_dashboards_hierarchy(
) -> Dict[Type['BaseDashboard'], List[Type['BaseDashboard']]]:
    """Get hierarchy of dashboards classes.

    Returns:
        Dict of dashboards classes.
    """
    dashboards_hierarchy = {}
```

This PR fixes that detection. Instead of lexing, it now checks if the
parameters itself is empty (or if it contains comments).

Closes https://github.com/astral-sh/ruff/issues/7457.
2023-09-20 16:20:22 -04:00