New AST nodes for f-string elements (#8835)

Rebase of #6365 authored by @davidszotten.

## Summary

This PR updates the AST structure for an f-string elements.

The main **motivation** behind this change is to have a dedicated node
for the string part of an f-string. Previously, the existing
`ExprStringLiteral` node was used for this purpose which isn't exactly
correct. The `ExprStringLiteral` node should include the quotes as well
in the range but the f-string literal element doesn't include the quote
as it's a specific part within an f-string. For example,

```python
f"foo {x}"
# ^^^^
# This is the literal part of an f-string
```

The introduction of `FStringElement` enum is helpful which represent
either the literal part or the expression part of an f-string.

### Rule Updates

This means that there'll be two nodes representing a string depending on
the context. One for a normal string literal while the other is a string
literal within an f-string. The AST checker is updated to accommodate
this change. The rules which work on string literal are updated to check
on the literal part of f-string as well.

#### Notes

1. The `Expr::is_literal_expr` method would check for
`ExprStringLiteral` and return true if so. But now that we don't
represent the literal part of an f-string using that node, this improves
the method's behavior and confines to the actual expression. We do have
the `FStringElement::is_literal` method.
2. We avoid checking if we're in a f-string context before adding to
`string_type_definitions` because the f-string literal is now a
dedicated node and not part of `Expr`.
3. Annotations cannot use f-string so we avoid changing any rules which
work on annotation and checks for `ExprStringLiteral`.

## Test Plan

- All references of `Expr::StringLiteral` were checked to see if any of
the rules require updating to account for the f-string literal element
node.
- New test cases are added for rules which check against the literal
part of an f-string.
- Check the ecosystem results and ensure it remains unchanged.

## Performance

There's a performance penalty in the parser. The reason for this remains
unknown as it seems that the generated assembly code is now different
for the `__reduce154` function. The reduce function body is just popping
the `ParenthesizedExpr` on top of the stack and pushing it with the new
location.

- The size of `FStringElement` enum is the same as `Expr` which is what
it replaces in `FString::format_spec`
- The size of `FStringExpressionElement` is the same as
`ExprFormattedValue` which is what it replaces

I tried reducing the `Expr` enum from 80 bytes to 72 bytes but it hardly
resulted in any performance gain. The difference can be seen here:
- Original profile: https://share.firefox.dev/3Taa7ES
- Profile after boxing some node fields:
https://share.firefox.dev/3GsNXpD

### Backtracking

I tried backtracking the changes to see if any of the isolated change
produced this regression. The problem here is that the overall change is
so small that there's only a single checkpoint where I can backtrack and
that checkpoint results in the same regression. This checkpoint is to
revert using `Expr` to the `FString::format_spec` field. After this
point, the change would revert back to the original implementation.

## Review process

The review process is similar to #7927. The first set of commits update
the node structure, parser, and related AST files. Then, further commits
update the linter and formatter part to account for the AST change.

---------

Co-authored-by: David Szotten <davidszotten@gmail.com>

crates/ruff_python_formatter/src/expression/mod.rs

@@ -36,7 +36,6 @@ pub(crate) mod expr_dict;
 pub(crate) mod expr_dict_comp;
 pub(crate) mod expr_ellipsis_literal;
 pub(crate) mod expr_f_string;
-pub(crate) mod expr_formatted_value;
 pub(crate) mod expr_generator_exp;
 pub(crate) mod expr_if_exp;
 pub(crate) mod expr_ipy_escape_command;
@@ -97,7 +96,6 @@ impl FormatRule<Expr, PyFormatContext<'_>> for FormatExpr {
             Expr::YieldFrom(expr) => expr.format().fmt(f),
             Expr::Compare(expr) => expr.format().fmt(f),
             Expr::Call(expr) => expr.format().fmt(f),
-            Expr::FormattedValue(expr) => expr.format().fmt(f),
             Expr::FString(expr) => expr.format().fmt(f),
             Expr::StringLiteral(expr) => expr.format().fmt(f),
             Expr::BytesLiteral(expr) => expr.format().fmt(f),
@@ -286,7 +284,6 @@ fn format_with_parentheses_comments(
         Expr::YieldFrom(expr) => FormatNodeRule::fmt_fields(expr.format().rule(), expr, f),
         Expr::Compare(expr) => FormatNodeRule::fmt_fields(expr.format().rule(), expr, f),
         Expr::Call(expr) => FormatNodeRule::fmt_fields(expr.format().rule(), expr, f),
-        Expr::FormattedValue(expr) => FormatNodeRule::fmt_fields(expr.format().rule(), expr, f),
         Expr::FString(expr) => FormatNodeRule::fmt_fields(expr.format().rule(), expr, f),
         Expr::StringLiteral(expr) => FormatNodeRule::fmt_fields(expr.format().rule(), expr, f),
         Expr::BytesLiteral(expr) => FormatNodeRule::fmt_fields(expr.format().rule(), expr, f),
@@ -488,7 +485,6 @@ impl NeedsParentheses for Expr {
             Expr::YieldFrom(expr) => expr.needs_parentheses(parent, context),
             Expr::Compare(expr) => expr.needs_parentheses(parent, context),
             Expr::Call(expr) => expr.needs_parentheses(parent, context),
-            Expr::FormattedValue(expr) => expr.needs_parentheses(parent, context),
             Expr::FString(expr) => expr.needs_parentheses(parent, context),
             Expr::StringLiteral(expr) => expr.needs_parentheses(parent, context),
             Expr::BytesLiteral(expr) => expr.needs_parentheses(parent, context),
@@ -746,7 +742,6 @@ impl<'input> CanOmitOptionalParenthesesVisitor<'input> {
             Expr::Tuple(_)
             | Expr::NamedExpr(_)
             | Expr::GeneratorExp(_)
-            | Expr::FormattedValue(_)
             | Expr::FString(_)
             | Expr::StringLiteral(_)
             | Expr::BytesLiteral(_)
@@ -1098,7 +1093,6 @@ pub(crate) fn is_expression_huggable(expr: &Expr, context: &PyFormatContext) ->
         | Expr::YieldFrom(_)
         | Expr::Compare(_)
         | Expr::Call(_)
-        | Expr::FormattedValue(_)
         | Expr::FString(_)
         | Expr::Attribute(_)
         | Expr::Subscript(_)

crates/ruff_python_formatter/src/expression/string/mod.rs

@@ -52,8 +52,11 @@ impl<'a> AnyString<'a> {
                     .trim_start_matches(|c| c != '"' && c != '\'');
                 let triple_quoted =
                     unprefixed.starts_with(r#"""""#) || unprefixed.starts_with(r"'''");
-                if f_string.value.elements().any(|value| match value {
-                    Expr::FormattedValue(ast::ExprFormattedValue { range, .. }) => {
+                if f_string.value.elements().any(|element| match element {
+                    ast::FStringElement::Expression(ast::FStringExpressionElement {
+                        range,
+                        ..
+                    }) => {
                         let string_content = locator.slice(*range);
                         if triple_quoted {
                             string_content.contains(r#"""""#) || string_content.contains("'''")
@@ -61,7 +64,7 @@ impl<'a> AnyString<'a> {
                             string_content.contains(['"', '\''])
                         }
                     }
-                    _ => false,
+                    ast::FStringElement::Literal(_) => false,
                 }) {
                     Quoting::Preserve
                 } else {

crates/ruff_python_formatter/src/generated.rs

@@ -1534,42 +1534,6 @@ impl<'ast> IntoFormat<PyFormatContext<'ast>> for ast::ExprCall {
     }
 }
 
-impl FormatRule<ast::ExprFormattedValue, PyFormatContext<'_>>
-    for crate::expression::expr_formatted_value::FormatExprFormattedValue
-{
-    #[inline]
-    fn fmt(&self, node: &ast::ExprFormattedValue, f: &mut PyFormatter) -> FormatResult<()> {
-        FormatNodeRule::<ast::ExprFormattedValue>::fmt(self, node, f)
-    }
-}
-impl<'ast> AsFormat<PyFormatContext<'ast>> for ast::ExprFormattedValue {
-    type Format<'a> = FormatRefWithRule<
-        'a,
-        ast::ExprFormattedValue,
-        crate::expression::expr_formatted_value::FormatExprFormattedValue,
-        PyFormatContext<'ast>,
-    >;
-    fn format(&self) -> Self::Format<'_> {
-        FormatRefWithRule::new(
-            self,
-            crate::expression::expr_formatted_value::FormatExprFormattedValue::default(),
-        )
-    }
-}
-impl<'ast> IntoFormat<PyFormatContext<'ast>> for ast::ExprFormattedValue {
-    type Format = FormatOwnedWithRule<
-        ast::ExprFormattedValue,
-        crate::expression::expr_formatted_value::FormatExprFormattedValue,
-        PyFormatContext<'ast>,
-    >;
-    fn into_format(self) -> Self::Format {
-        FormatOwnedWithRule::new(
-            self,
-            crate::expression::expr_formatted_value::FormatExprFormattedValue::default(),
-        )
-    }
-}
-
 impl FormatRule<ast::ExprFString, PyFormatContext<'_>>
     for crate::expression::expr_f_string::FormatExprFString
 {