ruff/crates
Charlie Marsh 86ccdcc9d9
Add support for multi-character operator tokens to SimpleTokenizer (#6563)
## Summary

Allows for proper lexing of tokens like `->`.

The main challenge is to ensure that our forward and backwards
representations are the same for cases like `===`. Specifically, we want
that to lex as `==` followed by `=` regardless of whether it's a
forwards or backwards lex. To do so, we identify the range of the
sequential characters (the full span of `===`), lex it forwards, then
return the last token.

## Test Plan

`cargo test`
2023-08-16 09:09:19 -04:00
..
flake8_to_ruff Bump version to 0.0.284 (#6453) 2023-08-09 13:32:33 -05:00
ruff Allow top-level await in Jupyter notebooks (#6607) 2023-08-15 23:59:05 -04:00
ruff_benchmark Remove some extraneous newlines in Cargo.toml (#6577) 2023-08-14 23:39:41 +00:00
ruff_cache Error on zero tab width (#6429) 2023-08-08 16:51:37 -04:00
ruff_cli Remove some extraneous newlines in Cargo.toml (#6577) 2023-08-14 23:39:41 +00:00
ruff_dev Use a faster diffing library for the formatter ecosystem checks (#6497) 2023-08-11 15:51:54 +02:00
ruff_diagnostics
ruff_formatter Replace inline(always) with inline (#6590) 2023-08-15 08:58:11 +02:00
ruff_index
ruff_macros
ruff_python_ast Replace inline(always) with inline (#6590) 2023-08-15 08:58:11 +02:00
ruff_python_codegen Remove Stmt::TryStar (#6566) 2023-08-14 13:39:44 -04:00
ruff_python_formatter Add support for multi-character operator tokens to SimpleTokenizer (#6563) 2023-08-16 09:09:19 -04:00
ruff_python_index Remove some extraneous newlines in Cargo.toml (#6577) 2023-08-14 23:39:41 +00:00
ruff_python_literal Remove allow(pedantic) from formatter (#6549) 2023-08-14 14:02:06 +02:00
ruff_python_parser Remove Stmt::TryStar (#6566) 2023-08-14 13:39:44 -04:00
ruff_python_resolver Replace .map_or(false, $closure) with .is_some_and(closure) (#6244) 2023-08-01 19:29:42 +02:00
ruff_python_semantic Remove Stmt::TryStar (#6566) 2023-08-14 13:39:44 -04:00
ruff_python_stdlib Replace .map_or(false, $closure) with .is_some_and(closure) (#6244) 2023-08-01 19:29:42 +02:00
ruff_python_trivia Add support for multi-character operator tokens to SimpleTokenizer (#6563) 2023-08-16 09:09:19 -04:00
ruff_shrinking Use Jupyter mode while parsing Notebook files (#5552) 2023-08-05 00:32:07 +00:00
ruff_source_file
ruff_text_size
ruff_wasm Set a default on PythonVersion (#6446) 2023-08-09 15:19:27 +00:00