Commit graph

60 commits

Author SHA1 Message Date
Charlie Marsh
3711f8ad59
Expand SimpleTokenizer to all keywords and single-character tokens (#6518)
## Summary

For #6485, I need to be able to use the `SimpleTokenizer` to lex the
space between any two adjacent expressions (i.e., the space between a
preceding and following node). This requires that we support a wider
range of keywords (like `and`, to connect the pieces of `x and y`), and
some additional single-character tokens (like `-` and `>`, to support
`->`). Note that the `SimpleTokenizer` does not support multi-character
tokens, so the `->` in a function signature is lexed as a `-` followed
by a `>` -- but this is fine for our purposes.
2023-08-14 10:35:31 -04:00
Micha Reiser
9584f613b9
Remove allow(pedantic) from formatter (#6549) 2023-08-14 14:02:06 +02:00
Charlie Marsh
1e3fe67ca5
Refactor and rename skip_trailing_trivia (#6312)
Based on feedback here:
https://github.com/astral-sh/ruff/pull/6274#discussion_r1282747964.
2023-08-04 13:30:53 +00:00
Micha Reiser
debfca3a11
Remove Parse trait (#6235) 2023-08-01 18:35:03 +02:00
Charlie Marsh
646ff6497c
Ignore end-of-line file exemption comments (#6160)
## Summary

This PR protects against code like:

```python
from typing import Optional

import bar  # ruff: noqa
import baz

class Foo:
    x: Optional[str] = None
```

In which the user wrote `# ruff: noqa` to ignore a specific error, not
realizing that it was a file-level exemption that thus turned off all
lint rules.

Specifically, if a `# ruff: noqa` directive is not at the start of a
line, we now ignore it and warn, since this is almost certainly a
mistake.
2023-07-29 00:40:32 +00:00
Micha Reiser
40f54375cb
Pull in RustPython parser (#6099) 2023-07-27 09:29:11 +00:00
Micha Reiser
2cf00fee96
Remove parser dependency from ruff-python-ast (#6096) 2023-07-26 17:47:22 +02:00
Micha Reiser
d351761f5d
SimpleTokenizer: Fix infinite loop when lexing empty quotes (#5917) 2023-07-20 15:18:35 +02:00
Micha Reiser
76e9ce6dc0
Fix SimpleTokenizer's backward lexing of # (#5878) 2023-07-20 11:54:18 +02:00
Charlie Marsh
5f3da9955a
Rename ruff_python_whitespace to ruff_python_trivia (#5886)
## Summary

This crate now contains utilities for dealing with trivia more broadly:
whitespace, newlines, "simple" trivia lexing, etc. So renaming it to
reflect its increased responsibilities.

To avoid conflicts, I've also renamed `Token` and `TokenKind` to
`SimpleToken` and `SimpleTokenKind`.
2023-07-19 11:48:27 -04:00