Summary
--
This updates the regex in `ruff-ecosystem` to catch syntax errors in an
effort to prevent bugs like #16874. This should catch `ParseError`s,
`UnsupportedSyntaxError`s, and the upcoming `SemanticSyntaxError`s.
Test Plan
--
I ran the ecosystem check locally comparing v0.11.0 and v0.11.1 and saw
a large number (2757!) of new syntax errors. I also manually tested the
regex on a few lines before that.
If we merge this before #16878, I'd expect to see that number decrease
substantially in that PR too, as another test.
Adds the ability to override `ruff.toml` or `pyproject.toml` settings
per-project during ecosystem checks.
Exploring this as a fix for the `setuptools` project error.
Also useful for including Jupyter Notebooks in the ecosystem checks, see
#9293
Note the remaining `sphinx` project error is resolved in #9294
Instead of displaying the total completed project count in the "changed"
section of a header, we now separately calculated the changed and
unchanged count to make the header message nice and clear.
e.g.
> ℹ️ ecosystem check **detected format changes**. (+1772 -1859 lines in
239 files in 26 projects; 6 project errors; 9 projects unchanged)
and
> ℹ️ ecosystem check **detected linter changes**. (+4598 -5023
violations, +0 -40 fixes in 13 projects; 4 project errors; 24 projects
unchanged)
Previously, it would have included the unchanged count in the first
project count.
Extends https://github.com/astral-sh/ruff/pull/8416 activating the
`black-and-ruff` and `black-then-ruff` formatter comparison modes for
ecosystem checks allowing us to compare changes to Black across the
ecosystem.
Until https://github.com/astral-sh/ruff/issues/8076 is ready, it seems
beneficial to get feedback on preview mode changes.
Tested locally, updated logs to output the flags passed to `ruff` and
verified `--preview` is used.
Fixes bug where `total_affected_rules` is empty, a division by zero
error can occur if there are only errors and no rule changes. Calculates
the maximum display per rule with the calculated project maximum as the
upper bound instead of 50, this should show more rule variety when
project maximums are lower.
This commit was meant to be in #8223 but I missed it.
Closes#7239
- Refactors `scripts/check_ecosystem.py` into a new Python project at
`python/ruff-ecosystem`
- Includes
[documentation](https://github.com/astral-sh/ruff/blob/zanie/ecosystem-format/python/ruff-ecosystem/README.md)
now
- Provides a `ruff-ecosystem` CLI
- Fixes bug where `ruff check` report included "fixable" summary line
- Adds truncation to `ruff check` reports
- Otherwise we often won't see the `ruff format` reports
- The truncation uses some very simple heuristics and could be improved
in the future
- Identifies diagnostic changes that occur just because a violation's
fix available changes
- We still show the diff for the line because it's could matter _where_
this changes, but we could improve this
- Similarly, we could improve detection of diagnostic changes where just
the message changes
- Adds support for JSON ecosystem check output
- I added this primarily for development purposes
- If there are no changes, only errors while processing projects, we
display a different summary message
- When caching repositories, we now checkout the requested ref
- Adds `ruff format` reports, which format with the baseline then the
use `format --diff` to generate a report
- Runs all CI jobs when the CI workflow is changed
## Known problems
- Since we must format the project to get a baseline, the permalink line
numbers do not exactly correspond to the correct range
- This looks... hard. I tried using `git diff` and some wonky hunk
matching to recover the original line numbers but it doesn't seem worth
it. I think we should probably commit the formatted changes to a fork or
something if we want great results here. Consequently, I've just used
the start line instead of a range for now.
- I don't love the comment structure — it'd be nice, perhaps, to have
separate headings for the linter and formatter.
- However, the `pr-comment` workflow is an absolute pain to change
because it runs _separately_ from this pull request so I if I want to
make edits to it I can only test it via manual workflow dispatch.
- Lines are not printed "as we go" which means they're all held in
memory, presumably this would be a problem for large-scale ecosystem
checks
- We are encountering a hard limit with the maximum comment length
supported by GitHub. We will need to move the bulk of the report
elsewhere.
## Future work
- Update `ruff-ecosystem` to support non-default projects and
`check_ecosystem_all.py` behavior
- Remove existing ecosystem check scripts
- Add preview mode toggle (#8076)
- Add a toggle for truncation
- Add hints for quick reproduction of runs locally
- Consider parsing JSON output of Ruff instead of using regex to parse
the text output
- Links to project repositories should use the commit hash we checked
against
- When caching repositories, we should pull the latest changes for the
ref
- Sort check diffs by path and rule code only (changes in messages
should not change order)
- Update check diffs to distinguish between new violations and changes
in messages
- Add "fix" diffs
- Remove existing formatter similarity reports
- On release pull request, compare to the previous tag instead
---------
Co-authored-by: konsti <konstin@mailbox.org>
