Commit graph

373 commits

Author SHA1 Message Date
Dhruv Manilawala
77a72ecd38
Avoid multiline expression if format specifier is present (#11123)
## Summary

This PR fixes the bug where the formatter would format an f-string and
could potentially change the AST.

For a triple-quoted f-string, the element can't be formatted into
multiline if it has a format specifier because otherwise the newline
would be treated as part of the format specifier.

Given the following f-string:
```python
f"""aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbb ccccccccccc {
    variable:.3f} ddddddddddddddd eeeeeeee"""
```

The formatter sees that the f-string is already multiline so it assumes
that it can contain line breaks i.e., broken into multiple lines. But,
in this specific case we can't format it as:

```python
f"""aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbb ccccccccccc {
    variable:.3f
} ddddddddddddddd eeeeeeee"""
```
                     
Because the format specifier string would become ".3f\n", which is not
the original string (`.3f`).

If the original source code already contained a newline, they'll be
preserved. For example:
```python
f"""aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbb ccccccccccc {
    variable:.3f
} ddddddddddddddd eeeeeeee"""
```

The above will be formatted as:
```py
f"""aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbb ccccccccccc {variable:.3f
} ddddddddddddddd eeeeeeee"""
```

Note that the newline after `.3f` is part of the format specifier which
needs to be preserved.
The Python version is irrelevant in this case.

fixes: #10040 

## Test Plan

Add some test cases to verify this behavior.
2024-04-26 13:34:38 +00:00
Jelle Zijlstra
cd3e319538
Add support for PEP 696 syntax (#11120) 2024-04-26 09:47:29 +02:00
Dhruv Manilawala
13ffb5bc19
Replace LALRPOP parser with hand-written parser (#10036)
(Supersedes #9152, authored by @LaBatata101)

## Summary

This PR replaces the current parser generated from LALRPOP to a
hand-written recursive descent parser.

It also updates the grammar for [PEP
646](https://peps.python.org/pep-0646/) so that the parser outputs the
correct AST. For example, in `data[*x]`, the index expression is now a
tuple with a single starred expression instead of just a starred
expression.

Beyond the performance improvements, the parser is also error resilient
and can provide better error messages. The behavior as seen by any
downstream tools isn't changed. That is, the linter and formatter can
still assume that the parser will _stop_ at the first syntax error. This
will be updated in the following months.

For more details about the change here, refer to the PR corresponding to
the individual commits and the release blog post.

## Test Plan

Write _lots_ and _lots_ of tests for both valid and invalid syntax and
verify the output.

## Acknowledgements

- @MichaReiser for reviewing 100+ parser PRs and continuously providing
guidance throughout the project
- @LaBatata101 for initiating the transition to a hand-written parser in
#9152
- @addisoncrump for implementing the fuzzer which helped
[catch](https://github.com/astral-sh/ruff/pull/10903)
[a](https://github.com/astral-sh/ruff/pull/10910)
[lot](https://github.com/astral-sh/ruff/pull/10966)
[of](https://github.com/astral-sh/ruff/pull/10896)
[bugs](https://github.com/astral-sh/ruff/pull/10877)

---------

Co-authored-by: Victor Hugo Gomes <labatata101@linuxmail.org>
Co-authored-by: Micha Reiser <micha@reiser.io>
2024-04-18 17:57:39 +05:30
Micha Reiser
9d705a4414
Fix subscript comment placement with parenthesized value (#10496)
## Summary

This is a follow up on https://github.com/astral-sh/ruff/pull/10492 

I incorrectly assumed that `subscript.value.end()` always points past
the value. However, this isn't the case for parenthesized values where
the end "ends" before the parentheses.

## Test Plan

I added new tests for the parenthesized case.
2024-03-20 20:30:22 +00:00
Micha Reiser
954a48b129
Fix instable formatting for trailing subscribt end-of-line comment (#10492)
## Summary

This PR fixes an instability where formatting a subscribt 
where the `slice` is not an `ExprSlice` and it has a trailing
end-of-line comment after its opening `[` required two formatting passes
to be stable.

The fix is to associate the trailing end-of-line comment as dangling
comment on `[` to preserve its position, similar to how Ruff does it for
other parenthesized expressions.
This also matches how trailing end-of-line subscript comments are
handled when the `slice` is an `ExprSlice`.

Fixes https://github.com/astral-sh/ruff/issues/10355

## Versioning

Shipping this as part of a patch release is fine because:

* It fixes a stability issue
* It doesn't impact already formatted code because Ruff would already
have moved to the comment to the end of the line (instead of preserving
it)

## Test Plan

Added tests
2024-03-20 18:12:10 +01:00
Auguste Lalande
3ed707f245
Spellcheck & grammar (#10375)
## Summary

I used `codespell` and `gramma` to identify mispellings and grammar
errors throughout the codebase and fixed them. I tried not to make any
controversial changes, but feel free to revert as you see fit.
2024-03-13 02:34:23 +00:00
Micha Reiser
b64f2ea401
Formatter: Improve single-with item formatting for Python 3.8 or older (#10276)
## Summary

This PR changes how we format `with` statements with a single with item
for Python 3.8 or older. This change is not compatible with Black.

This is how we format a single-item with statement today 

```python
def run(data_path, model_uri):
    with pyspark.sql.SparkSession.builder.config(
        key="spark.python.worker.reuse", value=True
    ).config(key="spark.ui.enabled", value=False).master(
        "local-cluster[2, 1, 1024]"
    ).getOrCreate():
        # ignore spark log output
        spark.sparkContext.setLogLevel("OFF")
        print(score_model(spark, data_path, model_uri))
```

This is different than how we would format the same expression if it is
inside any other clause header (`while`, `if`, ...):

```python
def run(data_path, model_uri):
    while (
        pyspark.sql.SparkSession.builder.config(
            key="spark.python.worker.reuse", value=True
        )
        .config(key="spark.ui.enabled", value=False)
        .master("local-cluster[2, 1, 1024]")
        .getOrCreate()
    ):
        # ignore spark log output
        spark.sparkContext.setLogLevel("OFF")
        print(score_model(spark, data_path, model_uri))

```

Which seems inconsistent to me. 

This PR changes the formatting of the single-item with Python 3.8 or
older to match that of other clause headers.

```python
def run(data_path, model_uri):
    with (
        pyspark.sql.SparkSession.builder.config(
            key="spark.python.worker.reuse", value=True
        )
        .config(key="spark.ui.enabled", value=False)
        .master("local-cluster[2, 1, 1024]")
        .getOrCreate()
    ):
        # ignore spark log output
        spark.sparkContext.setLogLevel("OFF")
        print(score_model(spark, data_path, model_uri))
```

According to our versioning policy, this style change is gated behind a
preview flag.

## Test Plan

See added tests.

Added
2024-03-08 23:56:02 +00:00
Micha Reiser
4bce801065
Fix unstable with-items formatting (#10274)
## Summary

Fixes https://github.com/astral-sh/ruff/issues/10267

The issue with the current formatting is that the formatter flips
between the `SingleParenthesizedContextManager` and
`ParenthesizeIfExpands` or `SingleWithTarget` because the layouts use
incompatible formatting ( `SingleParenthesizedContextManager`:
`maybe_parenthesize_expression(context)` vs `ParenthesizeIfExpands`:
`parenthesize_if_expands(item)`, `SingleWithTarget`:
`optional_parentheses(item)`.

The fix is to ensure that the layouts between which the formatter flips
when adding or removing parentheses are the same. I do this by
introducing a new `SingleWithoutTarget` layout that uses the same
formatting as `SingleParenthesizedContextManager` if it has no target
and prefer `SingleWithoutTarget` over using `ParenthesizeIfExpands` or
`SingleWithTarget`.

## Formatting change

The downside is that we now use `maybe_parenthesize_expression` over
`parenthesize_if_expands` for expressions where
`can_omit_optional_parentheses` returns `false`. This can lead to stable
formatting changes. I only found one formatting change in our ecosystem
check and, unfortunately, this is necessary to fix the instability (and
instability fixes are okay to have as part of minor changes according to
our versioning policy)

The benefit of the change is that `with` items with a single context
manager and without a target are now formatted identically to how the
same expression would be formatted in other clause headers.

## Test Plan

I ran the ecosystem check locally
2024-03-08 23:48:47 +00:00
Micha Reiser
965adbed4b
Fix trailing kwargs end of line comment after slash (#10297)
## Summary

Fixes the handling end of line comments that belong to `**kwargs` when
the `**kwargs` come after a slash.

The issue was that we missed to include the `**kwargs` start position
when determining the start of the next node coming after the `/`.

Fixes https://github.com/astral-sh/ruff/issues/10281

## Test Plan

Added test
2024-03-08 14:45:26 +00:00
Micha Reiser
dcc92f50cf
Update black tests (#10166) 2024-02-29 10:00:51 +01:00
Micha Reiser
a6f32ddc5e
Ruff 2024.2 style (#9639) 2024-02-29 09:30:54 +01:00
Dhruv Manilawala
72bf1c2880
Preview minimal f-string formatting (#9642)
## Summary

_This is preview only feature and is available using the `--preview`
command-line flag._

With the implementation of [PEP 701] in Python 3.12, f-strings can now
be broken into multiple lines, can contain comments, and can re-use the
same quote character. Currently, no other Python formatter formats the
f-strings so there's some discussion which needs to happen in defining
the style used for f-string formatting. Relevant discussion:
https://github.com/astral-sh/ruff/discussions/9785

The goal for this PR is to add minimal support for f-string formatting.
This would be to format expression within the replacement field without
introducing any major style changes.

### Newlines

The heuristics for adding newline is similar to that of
[Prettier](https://prettier.io/docs/en/next/rationale.html#template-literals)
where the formatter would only split an expression in the replacement
field across multiple lines if there was already a line break within the
replacement field.

In other words, the formatter would not add any newlines unless they
were already present i.e., they were added by the user. This makes
breaking any expression inside an f-string optional and in control of
the user. For example,

```python
# We wouldn't break this
aaaaaaaaaaa = f"asaaaaaaaaaaaaaaaa { aaaaaaaaaaaa + bbbbbbbbbbbb + ccccccccccccccc } cccccccccc"

# But, we would break the following as there's already a newline
aaaaaaaaaaa = f"asaaaaaaaaaaaaaaaa {
	aaaaaaaaaaaa + bbbbbbbbbbbb + ccccccccccccccc } cccccccccc"
```


If there are comments in any of the replacement field of the f-string,
then it will always be a multi-line f-string in which case the formatter
would prefer to break expressions i.e., introduce newlines. For example,

```python
x = f"{ # comment
    a }"
```

### Quotes

The logic for formatting quotes remains unchanged. The existing logic is
used to determine the necessary quote char and is used accordingly.

Now, if the expression inside an f-string is itself a string like, then
we need to make sure to preserve the existing quote and not change it to
the preferred quote unless it's 3.12. For example,

```python
f"outer {'inner'} outer"

# For pre 3.12, preserve the single quote
f"outer {'inner'} outer"

# While for 3.12 and later, the quotes can be changed
f"outer {"inner"} outer"
```

But, for triple-quoted strings, we can re-use the same quote char unless
the inner string is itself a triple-quoted string.

```python
f"""outer {"inner"} outer"""  # valid
f"""outer {'''inner'''} outer"""  # preserve the single quote char for the inner string
```

### Debug expressions

If debug expressions are present in the replacement field of a f-string,
then the whitespace needs to be preserved as they will be rendered as it
is (for example, `f"{ x = }"`. If there are any nested f-strings, then
the whitespace in them needs to be preserved as well which means that
we'll stop formatting the f-string as soon as we encounter a debug
expression.

```python
f"outer {   x =  !s  :.3f}"
#                  ^^
#                  We can remove these whitespaces
```

Now, the whitespace doesn't need to be preserved around conversion spec
and format specifiers, so we'll format them as usual but we won't be
formatting any nested f-string within the format specifier.

### Miscellaneous

- The
[`hug_parens_with_braces_and_square_brackets`](https://github.com/astral-sh/ruff/issues/8279)
preview style isn't implemented w.r.t. the f-string curly braces.
- The
[indentation](https://github.com/astral-sh/ruff/discussions/9785#discussioncomment-8470590)
is always relative to the f-string containing statement

## Test Plan

* Add new test cases
* Review existing snapshot changes
* Review the ecosystem changes

[PEP 701]: https://peps.python.org/pep-0701/
2024-02-16 20:28:11 +05:30
Micha Reiser
edfe8421ec
Disable top-level docstring formatting for notebooks (#9957) 2024-02-12 18:14:02 +00:00
Micha Reiser
8657a392ff
Docstring formatting: Preserve tab indentation when using indent-style=tabs (#9915) 2024-02-12 16:09:13 +01:00
Micha Reiser
4946a1876f
Stabilize quote-style preserve (#9922) 2024-02-12 09:30:07 +00:00
Micha Reiser
fe7d965334
Reduce Result<Tok, LexicalError> size by using Box<str> instead of String (#9885) 2024-02-08 20:36:22 +00:00
Shaygan Hooshyari
b47f85eb69
Preview Style: Format module level docstring (#9725)
Co-authored-by: Micha Reiser <micha@reiser.io>
2024-02-05 15:03:34 +00:00
Micha Reiser
80fc02e7d5
Don't trim last empty line in docstrings (#9813) 2024-02-05 13:29:24 +00:00
Micha Reiser
4f7fb566f0
Range formatting: Fix invalid syntax after parenthesizing expression (#9751) 2024-02-02 17:56:25 +01:00
Micha Reiser
ce14f4dea5
Range formatting API (#9635) 2024-01-31 11:13:37 +01:00
Dhruv Manilawala
541aef4e6c
Implement blank_line_after_nested_stub_class preview style (#9155)
## Summary

This PR implements the `blank_line_after_nested_stub_class` preview
style in the formatter.

The logic is divided into 3 parts:
1. In between preceding and following nodes at top level and nested
suite
2. When there's a trailing comment after the class
3. When there is no following node from (1) which is the case when it's
the last or the only node in a suite

We handle (3) with `FormatLeadingAlternateBranchComments`.

## Test Plan

- Add new test cases and update existing snapshots
- Checked the `typeshed` diff

fixes: #8891
2024-01-31 00:09:38 +05:30
Micha Reiser
3c7fea769c
Show source-type in formatter snapshot tests with options (#9699) 2024-01-30 10:08:50 +00:00
Micha Reiser
0045032905
Set source type: Stub for black tests with options (#9674) 2024-01-29 15:54:30 +01:00
Micha Reiser
91046e4c81
Preserve indent around multiline strings (#9637) 2024-01-26 08:18:30 +01:00
Micha Reiser
395bf3dc98
Fix the input for black's line ranges test file (#9622) 2024-01-23 10:40:23 +00:00
Micha Reiser
58fcd96ac1
Update Black Tests (#9455) 2024-01-10 12:09:34 +00:00
Micha Reiser
ac02d3aedd
Hug multiline-strings preview style (#9243) 2024-01-10 12:47:34 +01:00
Charlie Marsh
ba71772d93
Parenthesize breaking named expressions in match guards (#9396)
## Summary

This is an attempt to solve
https://github.com/astral-sh/ruff/issues/9394 by avoiding breaks in
named expressions when invalid.
2024-01-08 14:47:01 +00:00
Charlie Marsh
60ba7a7c0d
Allow # fmt: skip with interspersed same-line comments (#9395)
## Summary

This is similar to https://github.com/astral-sh/ruff/pull/8876, but more
limited in scope:

1. It only applies to `# fmt: skip` (like Black). Like `# isort: on`, `#
fmt: on` needs to be on its own line (still).
2. It only delimits on `#`, so you can do `# fmt: skip # noqa`, but not
`# fmt: skip - some other content` or `# fmt: skip; noqa`.

If we want to support the `;`-delimited version, we should revisit
later, since we don't support that in the linter (so `# fmt: skip; noqa`
wouldn't register a `noqa`).

Closes https://github.com/astral-sh/ruff/issues/8892.
2024-01-04 19:39:37 -05:00
Dimitri Papadopoulos Orfanos
d04d49cc0e
Fix typos found by codespell (#9346)
## Summary

Fix typos found by
[codespell](https://github.com/codespell-project/codespell).

## Test Plan

CI tests.
2024-01-02 02:08:15 +00:00
Charlie Marsh
e80260a3c5
Remove source path from parser errors (#9322)
## Summary

I always found it odd that we had to pass this in, since it's really
higher-level context for the error. The awkwardness is further evidenced
by the fact that we pass in fake values everywhere (even outside of
tests). The source path isn't actually used to display the error; it's
only accessed elsewhere to _re-display_ the error in certain cases. This
PR modifies to instead pass the path directly in those cases.
2023-12-30 20:33:05 +00:00
Micha Reiser
5d4825b60f
Normalise Hex and unicode escape sequences in string (#9280) 2023-12-28 09:06:58 +08:00
Micha Reiser
9cc257ee7d
Improve dummy_implementations preview style formatting (#9240) 2023-12-22 03:44:14 +00:00
Micha Reiser
a06723da2b
Parenthesize multi-context managers (#9222) 2023-12-22 03:41:03 +00:00
Micha Reiser
fa2c37b411
Parenthesize long type annotations in annotated assignments (#9210) 2023-12-22 03:33:47 +00:00
Micha Reiser
d835b28d01
Show preview changes for tests with options (#9223) 2023-12-21 23:36:19 +00:00
Micha Reiser
c6d8076034
Set target versions in Black tests (#9221) 2023-12-21 04:20:17 +00:00
Micha Reiser
8cb7950102
Add target_version to formatter options (#9220) 2023-12-21 04:05:58 +00:00
Micha Reiser
ef4bd8d5ff
Fix: Avoid parenthesizing subscript targets and values (#9209) 2023-12-20 23:42:25 +00:00
Dhruv Manilawala
09296e3e3c
Implement no_blank_line_before_class_docstring preview style (#9154)
## Summary

This PR implements the `no_blank_line_before_class_docstring` preview
style.

## Test Plan

Update existing snapshots.

### Formatter ecosystem

`main`

| project | similarity index | total files | changed files |
|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75804 | 1799 | 1648 |
| django | 0.99984 | 2772 | 34 |
| home-assistant | 0.99955 | 10596 | 213 |
| poetry | 0.99905 | 321 | 15 |
| transformers | 0.99967 | 2657 | 324 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99980 | 3669 | 18 |
| warehouse | 0.99976 | 654 | 14 |
| zulip | 0.99958 | 1459 | 36 |

`dhruv/no-blank-line-docstring`

| project | similarity index | total files | changed files |
|----------------|------------------:|------------------:|------------------:|
| cpython | 0.75804 | 1799 | 1648 |
| django | 0.99984 | 2772 | 34 |
| home-assistant | 0.99955 | 10596 | 213 |
| poetry | 0.99905 | 321 | 15 |
| transformers | 0.99967 | 2657 | 324 |
| twine | 1.00000 | 33 | 0 |
| typeshed | 0.99980 | 3669 | 18 |
| warehouse | 0.99976 | 654 | 14 |
| zulip | 0.99958 | 1459 | 36 |

fixes: #8888
2023-12-19 00:43:20 -06:00
Micha Reiser
c8d6958d15
Add new with and match sequence test cases (#9128)
## Summary

Add new test cases for `with_item` and `match` sequence that demonstrate how long headers break. 

Removes one use of `optional_parentheses` in a position where it is know that the parentheses always need to be added.

## Test Plan

cargo test
2023-12-15 11:45:13 +09:00
Andrew Gallant
28b1aa201b
ruff_python_formatter: fix 'dynamic' mode with doctests (#9129)
This fixes a bug where the current indent level was not calculated
correctly for doctests. Namely, it didn't account for the extra indent
level (in terms of ASCII spaces) used by by the PS1 (`>>> `) and PS2
(`... `) prompts. As a result, lines could extend up to 4 spaces beyond
the configured line length limit.

We fix that by passing the `CodeExampleKind` to the `format` routine
instead of just the code itself. In this way, `format` can query whether
there will be any extra indent added _after_ formatting the code and
take that into account for its line length setting.

We add a few regression tests, taken directly from @stinodego's
examples.

Fixes #9126
2023-12-14 09:53:43 -05:00
Micha Reiser
7256b882b9
Fix can_omit_optional_parentheses for expressions with a right most fstring (#9124) 2023-12-14 04:58:17 +00:00
Andrew Gallant
b6fb972e6f
config: add new docstring-code-format knob (#8854)
This PR does the plumbing to make a new formatting option,
`docstring-code-format`, available in the configuration for end users.
It is disabled by default (opt-in). It is opt-in at least initially to
reflect a conservative posture. The intent is to make it opt-out at some
point in the future.

This was split out from #8811 in order to make #8811 easier to merge.
Namely, once this is merged, docstring code snippet formatting will
become available to end users. (See comments below for how we arrived at
the name.)

Closes #7146

## Test Plan

Other than the standard test suite, I ran the formatter over the CPython
and polars projects to ensure both that the result looked sensible and
that tests still passed. At time of writing, one issue that currently
appears is that reformatting code snippets trips the long line lint:
1905886802
2023-12-13 11:02:11 -05:00
Micha Reiser
45f603000d
prefer_splitting_right_hand_side_of_assignments preview style (#8943) 2023-12-13 03:43:23 +00:00
Andrew Gallant
b972455ac7
ruff_python_formatter: implement "dynamic" line width mode for docstring code formatting (#9098)
## Summary

This PR changes the internal `docstring-code-line-width` setting to
additionally accept a string value `dynamic`. When `dynamic` is set, the
line width is dynamically adjusted when reformatting code snippets in
docstrings based on the indent level of the docstring. The result is
that the reformatted lines from the code snippet should not exceed the
"global" line width configuration for the surrounding source.

This PR does not change the default behavior, although I suspect the
default should probably be `dynamic`.

## Test Plan

I added a new configuration to the existing docstring code tests and
also added a new set of tests dedicated to the new `dynamic` mode.
2023-12-12 09:58:07 -05:00
Andrew Gallant
07380e0657
ruff_python_formatter: add docstring-code-line-width internal setting (#9055)
## Summary

This does the light plumbing necessary to add a new internal option that
permits setting the line width of code examples in docstrings. The plan
is to add the corresponding user facing knob in #8854.

Note that this effectively removes the `same-as-global` configuration
style discussed [in this
comment](https://github.com/astral-sh/ruff/issues/8855#issuecomment-1847230440).
It replaces it with the `{integer}` configuration style only.

There are a lot of commits here, but they are each tiny to make review
easier because of the changes to snapshots.

## Test Plan

I added a new docstring test configuration that sets
`docstring-code-line-width = 60` and examined the differences.
2023-12-11 08:20:59 -05:00
Charlie Marsh
febc69ab48
Avoid trailing comma for single-argument with positional separator (#9076)
## Summary

In https://github.com/astral-sh/ruff/pull/8921, we changed our parameter
formatting behavior to add a trailing comma whenever a single-argument
function breaks. This introduced a deviation in the case that a function
contains a single argument, but _also_ includes a positional-only or
keyword-only separator.

Closes https://github.com/astral-sh/ruff/issues/9074.
2023-12-09 18:03:31 -05:00
Micha Reiser
d0d88d9375
Fix handling of trailing target comment (#9051) 2023-12-08 05:00:36 +00:00
Andrew Gallant
a224f19903
ruff_python_formatter: add test for extraneous info string text (#9050)
@ofek asked [about this][ref]. I did specifically add support for it,
but neglected to add a test. This PR adds a test.

[ref]:
https://github.com/astral-sh/ruff/pull/9030#issuecomment-1846054764
2023-12-07 19:52:14 -05:00