ruff/scripts
Zanie Blue fc94857a20
Rewrite ecosystem checks and add ruff format reports (#8223)
Closes #7239 

- Refactors `scripts/check_ecosystem.py` into a new Python project at
`python/ruff-ecosystem`
- Includes
[documentation](https://github.com/astral-sh/ruff/blob/zanie/ecosystem-format/python/ruff-ecosystem/README.md)
now
    - Provides a `ruff-ecosystem` CLI
- Fixes bug where `ruff check` report included "fixable" summary line
- Adds truncation to `ruff check` reports
    - Otherwise we often won't see the `ruff format` reports
- The truncation uses some very simple heuristics and could be improved
in the future
- Identifies diagnostic changes that occur just because a violation's
fix available changes
- We still show the diff for the line because it's could matter _where_
this changes, but we could improve this
- Similarly, we could improve detection of diagnostic changes where just
the message changes
- Adds support for JSON ecosystem check output
    - I added this primarily for development purposes
- If there are no changes, only errors while processing projects, we
display a different summary message
- When caching repositories, we now checkout the requested ref
- Adds `ruff format` reports, which format with the baseline then the
use `format --diff` to generate a report
- Runs all CI jobs when the CI workflow is changed

## Known problems

- Since we must format the project to get a baseline, the permalink line
numbers do not exactly correspond to the correct range
- This looks... hard. I tried using `git diff` and some wonky hunk
matching to recover the original line numbers but it doesn't seem worth
it. I think we should probably commit the formatted changes to a fork or
something if we want great results here. Consequently, I've just used
the start line instead of a range for now.
- I don't love the comment structure — it'd be nice, perhaps, to have
separate headings for the linter and formatter.
- However, the `pr-comment` workflow is an absolute pain to change
because it runs _separately_ from this pull request so I if I want to
make edits to it I can only test it via manual workflow dispatch.
- Lines are not printed "as we go" which means they're all held in
memory, presumably this would be a problem for large-scale ecosystem
checks
- We are encountering a hard limit with the maximum comment length
supported by GitHub. We will need to move the bulk of the report
elsewhere.

## Future work

- Update `ruff-ecosystem` to support non-default projects and
`check_ecosystem_all.py` behavior
- Remove existing ecosystem check scripts
- Add preview mode toggle (#8076)
- Add a toggle for truncation
- Add hints for quick reproduction of runs locally
- Consider parsing JSON output of Ruff instead of using regex to parse
the text output
- Links to project repositories should use the commit hash we checked
against
- When caching repositories, we should pull the latest changes for the
ref
- Sort check diffs by path and rule code only (changes in messages
should not change order)
- Update check diffs to distinguish between new violations and changes
in messages
- Add "fix" diffs
- Remove existing formatter similarity reports
- On release pull request, compare to the previous tag instead

---------

Co-authored-by: konsti <konstin@mailbox.org>
2023-10-27 17:28:01 -05:00
..
benchmarks Bump version to v0.1.3 (#8259) 2023-10-26 18:57:05 +00:00
_utils.py Use __future__ imports in scripts (#5301) 2023-06-22 11:40:16 -04:00
add_plugin.py Rename ruff crate to ruff_linter (#7529) 2023-09-20 08:38:27 +02:00
add_rule.py Rename RuleGroup::Unspecified to Stable (#7991) 2023-10-16 14:53:27 -05:00
check_docs_formatted.py [pylint] Implement PLR0916 (too-many-boolean-expressions) (#7975) 2023-10-17 04:44:25 +00:00
check_ecosystem.py Rewrite ecosystem checks and add ruff format reports (#8223) 2023-10-27 17:28:01 -05:00
Dockerfile.ecosystem Remove outdated feature flag from Dockerfile.ecosystem (#4620) 2023-05-24 08:19:08 +00:00
ecosystem_all_check.py Rename Autofix to Fix (#7657) 2023-09-28 10:53:05 +00:00
ecosystem_all_check.sh Rename Autofix to Fix (#7657) 2023-09-28 10:53:05 +00:00
ecosystem_all_check_entrypoint.sh Make ecosystem all check more generic (#4629) 2023-05-24 16:26:23 +02:00
formatter_ecosystem_checks.sh Add test and basic implementation for formatter preview mode (#8044) 2023-10-26 15:33:26 +00:00
generate_known_standard_library.py Remove HashMap and HashSet for known-standard-library detection (#5345) 2023-06-23 19:59:03 +00:00
generate_mkdocs.py Fix dead link in README (#8146) 2023-10-23 15:56:28 +00:00
pyproject.toml Check for Any in other types for ANN401 (#5601) 2023-07-13 18:19:27 +05:30
transform_readme.py Use __future__ imports in scripts (#5301) 2023-06-22 11:40:16 -04:00
update_ambiguous_characters.py Rename ruff crate to ruff_linter (#7529) 2023-09-20 08:38:27 +02:00
update_schemastore.py Use SSH clones in update_schemastore.py (#5322) 2023-06-23 09:50:10 -04:00