language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-07-16 01:25:22 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
ruff/crates/ruff_python_parser/resources/inline/err
History
Dhruv Manilawala 6c1fa1d440
Use speculative parsing for with-items (#11770) 
## Summary

This PR updates the with-items parsing logic to use speculative parsing
instead.

### Existing logic

First, let's understand the previous logic:
1. The parser sees `(`, it doesn't know whether it's part of a
parenthesized with items or a parenthesized expression
2. Consider it a parenthesized with items and perform a hand-rolled
speculative parsing
3. Then, verify the assumption and if it's incorrect convert the parsed
with items into an appropriate expression which becomes part of the
first with item

Here, in (3) there are lots of edge cases which we've to deal with:
1. Trailing comma with a single element should be [converted to the
expression as
is](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2140-L2153))
2. Trailing comma with multiple elements should be [converted to a tuple
expression](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2155-L2178))
3. Limit the allowed expression based on whether it's
[(1)](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2144-L2152))
or
[(2)](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2157-L2171))
4. [Consider postfix
expressions](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2181-L2200))
after (3)
5. [Consider `if`
expressions](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2203-L2208))
after (3)
6. [Consider binary
expressions](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2210-L2228))
after (3)

Consider other cases like
* [Single generator
expression](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2020-L2035))
* [Expecting a
comma](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2122-L2130))

And, this is all possible only if we allow parsing these expressions in
the [with item parsing
logic](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2287-L2334)).

### Speculative parsing

With #11457 merged, we can simplify this logic by changing the step (3)
from above to just rewind the parser back to the `(` if our assumption
(parenthesized with-items) was incorrect and then continue parsing it
considering parenthesized expression.

This also behaves a lot similar to what a PEG parser does which is to
consider the first grammar rule and if it fails consider the second
grammar rule and so on.

resolves: #11639 

## Test Plan

- [x] Verify the updated snapshots
- [x] Run the fuzzer on around 3000 valid source code (locally)
2024-06-06 08:59:56 +00:00
..
ann_assign_stmt_invalid_annotation.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
ann_assign_stmt_invalid_target.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
ann_assign_stmt_invalid_value.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
ann_assign_stmt_missing_rhs.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
ann_assign_stmt_type_alias_annotation.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
assert_empty_msg.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
assert_empty_test.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
assert_invalid_msg_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
assert_invalid_test_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
assign_stmt_invalid_target.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
assign_stmt_invalid_value_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
assign_stmt_keyword_target.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
assign_stmt_missing_rhs.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
async_unexpected_token.py Maintain synchronicity between the lexer and the parser (#11457) 2024-06-03 18:23:50 +05:30
aug_assign_stmt_invalid_target.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
aug_assign_stmt_invalid_value.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
aug_assign_stmt_missing_rhs.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
case_expect_indented_block.py Expect indented case block instead of match stmt (#11033) 2024-04-19 16:46:15 +05:30
class_def_empty_body.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
class_def_missing_name.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
class_def_unclosed_type_param_list.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
clause_expect_indented_block.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
clause_expect_single_statement.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
comprehension_missing_for_after_async.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
decorator_invalid_expression.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
decorator_missing_expression.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
decorator_missing_newline.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
decorator_unexpected_token.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
del_incomplete_target.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
del_stmt_empty.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
dotted_name_multiple_dots.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
except_stmt_invalid_expression.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
except_stmt_missing_as_name.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
except_stmt_missing_exception.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
except_stmt_missing_exception_and_as_name.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
except_stmt_unparenthesized_tuple.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
f_string_empty_expression.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
f_string_invalid_conversion_flag_name_tok.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
f_string_invalid_conversion_flag_other_tok.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
f_string_invalid_starred_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
f_string_lambda_without_parentheses.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
f_string_unclosed_lbrace.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
f_string_unclosed_lbrace_in_format_spec.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
for_stmt_invalid_iter_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
for_stmt_invalid_target.py Add ExpressionContext for expression parsing (#11055) 2024-04-23 04:19:05 +00:00
for_stmt_invalid_target_binary_expr.py Add ExpressionContext for expression parsing (#11055) 2024-04-23 04:19:05 +00:00
for_stmt_invalid_target_in_keyword.py Add ExpressionContext for expression parsing (#11055) 2024-04-23 04:19:05 +00:00
for_stmt_missing_in_keyword.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
for_stmt_missing_iter.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
for_stmt_missing_target.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
from_import_dotted_names.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
from_import_empty_names.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
from_import_missing_module.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
from_import_missing_rpar.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
from_import_star_with_other_names.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
from_import_unparenthesized_trailing_comma.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
function_def_empty_body.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
function_def_invalid_return_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
function_def_missing_identifier.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
function_def_missing_return_type.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
function_def_unclosed_parameter_list.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
function_def_unclosed_type_param_list.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
function_def_unparenthesized_return_types.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
global_stmt_empty.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
global_stmt_expression.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
global_stmt_trailing_comma.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
if_stmt_elif_missing_colon.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
if_stmt_empty_body.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
if_stmt_invalid_elif_test_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
if_stmt_invalid_test_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
if_stmt_missing_colon.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
if_stmt_missing_test.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
if_stmt_misspelled_elif.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
implicitly_concatenated_unterminated_string.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
implicitly_concatenated_unterminated_string_multiline.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
import_alias_missing_asname.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
import_stmt_empty.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
import_stmt_parenthesized_names.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
import_stmt_star_import.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
import_stmt_trailing_comma.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
invalid_byte_literal.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
invalid_del_target.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
invalid_fstring_literal_element.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
invalid_string_literal.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
lambda_body_with_starred_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
lambda_body_with_yield_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
match_classify_as_keyword.py Maintain synchronicity between the lexer and the parser (#11457) 2024-06-03 18:23:50 +05:30
match_classify_as_keyword_or_identifier.py Maintain synchronicity between the lexer and the parser (#11457) 2024-06-03 18:23:50 +05:30
match_expected_colon.py Maintain synchronicity between the lexer and the parser (#11457) 2024-06-03 18:23:50 +05:30
match_stmt_expect_indented_block.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
match_stmt_expected_case_block.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
match_stmt_invalid_guard_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
match_stmt_invalid_subject_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
match_stmt_missing_guard_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
match_stmt_missing_pattern.py Maintain synchronicity between the lexer and the parser (#11457) 2024-06-03 18:23:50 +05:30
match_stmt_no_newline_before_case.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
match_stmt_single_starred_subject.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
mixed_bytes_and_non_bytes_literals.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
multiple_clauses_on_same_line.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
node_range_with_gaps.py Use empty range when there's "gap" in token source (#11032) 2024-04-19 11:36:26 +00:00
nonlocal_stmt_empty.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
nonlocal_stmt_expression.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
nonlocal_stmt_trailing_comma.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
param_missing_annotation.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
param_missing_default.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
param_with_invalid_annotation.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
param_with_invalid_default.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
param_with_invalid_star_annotation.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_duplicate_names.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_expected_after_star_separator.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_kwarg_after_star_separator.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_multiple_kwargs.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_multiple_slash_separator.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_multiple_star_separator.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_multiple_varargs.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_no_arg_before_slash.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_non_default_after_default.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_star_after_slash.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_star_separator_after_star_param.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_var_keyword_with_default.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
params_var_positional_with_default.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
raise_stmt_invalid_cause.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
raise_stmt_invalid_exc.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
raise_stmt_unparenthesized_tuple_cause.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
raise_stmt_unparenthesized_tuple_exc.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
return_stmt_invalid_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
simple_and_compound_stmt_on_same_line.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
simple_and_compound_stmt_on_same_line_in_block.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
simple_stmts_on_same_line.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
simple_stmts_on_same_line_in_block.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
try_stmt_invalid_order.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
try_stmt_missing_except_finally.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
try_stmt_misspelled_except.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
type_alias_incomplete_stmt.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
type_alias_invalid_value_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
type_param_invalid_bound_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
type_param_missing_bound.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
type_param_param_spec_bound.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
type_param_param_spec_invalid_default_expr.py Add support for PEP 696 syntax (#11120) 2024-04-26 09:47:29 +02:00
type_param_param_spec_missing_default.py Add support for PEP 696 syntax (#11120) 2024-04-26 09:47:29 +02:00
type_param_type_var_invalid_default_expr.py Add support for PEP 696 syntax (#11120) 2024-04-26 09:47:29 +02:00
type_param_type_var_missing_default.py Add support for PEP 696 syntax (#11120) 2024-04-26 09:47:29 +02:00
type_param_type_var_tuple_bound.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
type_param_type_var_tuple_invalid_default_expr.py Add support for PEP 696 syntax (#11120) 2024-04-26 09:47:29 +02:00
type_param_type_var_tuple_missing_default.py Add support for PEP 696 syntax (#11120) 2024-04-26 09:47:29 +02:00
unterminated_fstring_newline_recovery.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
while_stmt_invalid_test_expr.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
while_stmt_missing_colon.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
while_stmt_missing_test.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30
with_items_parenthesized_missing_colon.py Use speculative parsing for with-items (#11770) 2024-06-06 08:59:56 +00:00
with_items_parenthesized_missing_comma.py Replace LALRPOP parser with hand-written parser (#10036) 2024-04-18 17:57:39 +05:30