New AST nodes for f-string elements (#8835)

Rebase of #6365 authored by @davidszotten.

## Summary

This PR updates the AST structure for an f-string elements.

The main **motivation** behind this change is to have a dedicated node
for the string part of an f-string. Previously, the existing
`ExprStringLiteral` node was used for this purpose which isn't exactly
correct. The `ExprStringLiteral` node should include the quotes as well
in the range but the f-string literal element doesn't include the quote
as it's a specific part within an f-string. For example,

```python
f"foo {x}"
# ^^^^
# This is the literal part of an f-string
```

The introduction of `FStringElement` enum is helpful which represent
either the literal part or the expression part of an f-string.

### Rule Updates

This means that there'll be two nodes representing a string depending on
the context. One for a normal string literal while the other is a string
literal within an f-string. The AST checker is updated to accommodate
this change. The rules which work on string literal are updated to check
on the literal part of f-string as well.

#### Notes

1. The `Expr::is_literal_expr` method would check for
`ExprStringLiteral` and return true if so. But now that we don't
represent the literal part of an f-string using that node, this improves
the method's behavior and confines to the actual expression. We do have
the `FStringElement::is_literal` method.
2. We avoid checking if we're in a f-string context before adding to
`string_type_definitions` because the f-string literal is now a
dedicated node and not part of `Expr`.
3. Annotations cannot use f-string so we avoid changing any rules which
work on annotation and checks for `ExprStringLiteral`.

## Test Plan

- All references of `Expr::StringLiteral` were checked to see if any of
the rules require updating to account for the f-string literal element
node.
- New test cases are added for rules which check against the literal
part of an f-string.
- Check the ecosystem results and ensure it remains unchanged.

## Performance

There's a performance penalty in the parser. The reason for this remains
unknown as it seems that the generated assembly code is now different
for the `__reduce154` function. The reduce function body is just popping
the `ParenthesizedExpr` on top of the stack and pushing it with the new
location.

- The size of `FStringElement` enum is the same as `Expr` which is what
it replaces in `FString::format_spec`
- The size of `FStringExpressionElement` is the same as
`ExprFormattedValue` which is what it replaces

I tried reducing the `Expr` enum from 80 bytes to 72 bytes but it hardly
resulted in any performance gain. The difference can be seen here:
- Original profile: https://share.firefox.dev/3Taa7ES
- Profile after boxing some node fields:
https://share.firefox.dev/3GsNXpD

### Backtracking

I tried backtracking the changes to see if any of the isolated change
produced this regression. The problem here is that the overall change is
so small that there's only a single checkpoint where I can backtrack and
that checkpoint results in the same regression. This checkpoint is to
revert using `Expr` to the `FString::format_spec` field. After this
point, the change would revert back to the original implementation.

## Review process

The review process is similar to #7927. The first set of commits update
the node structure, parser, and related AST files. Then, further commits
update the linter and formatter part to account for the AST change.

---------

Co-authored-by: David Szotten <davidszotten@gmail.com>
This commit is contained in:
Dhruv Manilawala 2023-12-07 10:28:05 -06:00 committed by GitHub
parent fcc08894cf
commit cdac90ef68
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
77 changed files with 1714 additions and 1925 deletions

View file

@ -59,7 +59,6 @@ pub(crate) fn assignment_target(target: &Expr) -> Result<(), LexicalError> {
YieldFrom(ref e) => Err(err(e.range.start())),
Compare(ref e) => Err(err(e.range.start())),
Call(ref e) => Err(err(e.range.start())),
FormattedValue(ref e) => Err(err(e.range.start())),
// FString is recursive, but all its forms are invalid as an
// assignment target, so we can reject it without exploring it.
FString(ref e) => Err(err(e.range.start())),

View file

@ -11,7 +11,7 @@ use crate::{
lexer::{LexicalError, LexicalErrorType},
function::{ArgumentList, parse_arguments, validate_pos_params, validate_arguments},
context::set_context,
string::{StringType, concatenated_strings, parse_fstring_middle, parse_string_literal},
string::{StringType, concatenated_strings, parse_fstring_literal_element, parse_string_literal},
token::{self, StringKind},
invalid,
};
@ -1611,23 +1611,23 @@ StringLiteral: StringType = {
};
FStringExpr: StringType = {
<location:@L> FStringStart <values:FStringMiddlePattern*> FStringEnd <end_location:@R> => {
<location:@L> FStringStart <elements:FStringMiddlePattern*> FStringEnd <end_location:@R> => {
StringType::FString(ast::FString {
values,
elements,
range: (location..end_location).into()
})
}
};
FStringMiddlePattern: ast::Expr = {
FStringMiddlePattern: ast::FStringElement = {
FStringReplacementField,
<location:@L> <fstring_middle:fstring_middle> <end_location:@R> =>? {
let (source, is_raw) = fstring_middle;
Ok(parse_fstring_middle(&source, is_raw, (location..end_location).into())?)
Ok(parse_fstring_literal_element(&source, is_raw, (location..end_location).into())?)
}
};
FStringReplacementField: ast::Expr = {
FStringReplacementField: ast::FStringElement = {
<location:@L> "{" <value:TestListOrYieldExpr> <debug:"="?> <conversion:FStringConversion?> <format_spec:FStringFormatSpecSuffix?> "}" <end_location:@R> =>? {
if value.expr.is_lambda_expr() && !value.is_parenthesized() {
return Err(LexicalError {
@ -1651,30 +1651,27 @@ FStringReplacementField: ast::Expr = {
}
});
Ok(
ast::ExprFormattedValue {
value: Box::new(value.into()),
ast::FStringElement::Expression(ast::FStringExpressionElement {
expression: Box::new(value.into()),
debug_text,
conversion: conversion.map_or(ast::ConversionFlag::None, |(_, conversion_flag)| {
conversion_flag
}),
format_spec: format_spec.map(Box::new),
range: (location..end_location).into(),
}
.into()
})
)
}
};
FStringFormatSpecSuffix: ast::Expr = {
FStringFormatSpecSuffix: ast::FStringFormatSpec = {
":" <format_spec:FStringFormatSpec> => format_spec
};
FStringFormatSpec: ast::Expr = {
<location:@L> <values:FStringMiddlePattern*> <end_location:@R> => {
ast::FString {
values,
range: (location..end_location).into()
}.into()
FStringFormatSpec: ast::FStringFormatSpec = {
<location:@L> <elements:FStringMiddlePattern*> <end_location:@R> => ast::FStringFormatSpec {
elements,
range: (location..end_location).into(),
},
};

File diff suppressed because it is too large Load diff

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..9,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..8,
value: StringLiteral(
expression: StringLiteral(
ExprStringLiteral {
range: 3..7,
value: StringLiteralValue {
@ -57,11 +57,11 @@ expression: parse_ast
FString(
FString {
range: 10..20,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 12..19,
value: Name(
expression: Name(
ExprName {
range: 13..16,
id: "foo",
@ -93,11 +93,11 @@ expression: parse_ast
FString(
FString {
range: 21..28,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 23..27,
value: Tuple(
expression: Tuple(
ExprTuple {
range: 24..26,
elts: [
@ -138,11 +138,11 @@ expression: parse_ast
FString(
FString {
range: 29..39,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 31..38,
value: Compare(
expression: Compare(
ExprCompare {
range: 32..36,
left: NumberLiteral(
@ -171,21 +171,10 @@ expression: parse_ast
debug_text: None,
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 37..37,
value: FStringValue {
inner: Single(
FString(
FString {
range: 37..37,
values: [],
},
),
),
},
},
),
FStringFormatSpec {
range: 37..37,
elements: [],
},
),
},
),
@ -209,11 +198,11 @@ expression: parse_ast
FString(
FString {
range: 40..55,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 42..54,
value: NumberLiteral(
expression: NumberLiteral(
ExprNumberLiteral {
range: 43..44,
value: Int(
@ -224,58 +213,39 @@ expression: parse_ast
debug_text: None,
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 45..53,
value: FStringValue {
inner: Single(
FString(
FString {
range: 45..53,
values: [
FormattedValue(
ExprFormattedValue {
range: 45..50,
value: StringLiteral(
ExprStringLiteral {
range: 46..49,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 46..49,
value: "}",
unicode: false,
},
),
},
},
),
debug_text: None,
conversion: None,
format_spec: None,
FStringFormatSpec {
range: 45..53,
elements: [
Expression(
FStringExpressionElement {
range: 45..50,
expression: StringLiteral(
ExprStringLiteral {
range: 46..49,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 46..49,
value: "}",
unicode: false,
},
),
StringLiteral(
ExprStringLiteral {
range: 50..53,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 50..53,
value: ">10",
unicode: false,
},
),
},
},
),
],
},
},
),
),
},
},
),
debug_text: None,
conversion: None,
format_spec: None,
},
),
Literal(
FStringLiteralElement {
range: 50..53,
value: ">10",
},
),
],
},
),
},
),
@ -299,11 +269,11 @@ expression: parse_ast
FString(
FString {
range: 56..71,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 58..70,
value: NumberLiteral(
expression: NumberLiteral(
ExprNumberLiteral {
range: 59..60,
value: Int(
@ -314,58 +284,39 @@ expression: parse_ast
debug_text: None,
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 61..69,
value: FStringValue {
inner: Single(
FString(
FString {
range: 61..69,
values: [
FormattedValue(
ExprFormattedValue {
range: 61..66,
value: StringLiteral(
ExprStringLiteral {
range: 62..65,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 62..65,
value: "{",
unicode: false,
},
),
},
},
),
debug_text: None,
conversion: None,
format_spec: None,
FStringFormatSpec {
range: 61..69,
elements: [
Expression(
FStringExpressionElement {
range: 61..66,
expression: StringLiteral(
ExprStringLiteral {
range: 62..65,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 62..65,
value: "{",
unicode: false,
},
),
StringLiteral(
ExprStringLiteral {
range: 66..69,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 66..69,
value: ">10",
unicode: false,
},
),
},
},
),
],
},
},
),
),
},
},
),
debug_text: None,
conversion: None,
format_spec: None,
},
),
Literal(
FStringLiteralElement {
range: 66..69,
value: ">10",
},
),
],
},
),
},
),
@ -389,11 +340,11 @@ expression: parse_ast
FString(
FString {
range: 72..86,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 74..85,
value: Name(
expression: Name(
ExprName {
range: 77..80,
id: "foo",
@ -430,11 +381,11 @@ expression: parse_ast
FString(
FString {
range: 87..107,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 89..106,
value: Name(
expression: Name(
ExprName {
range: 92..95,
id: "foo",
@ -449,36 +400,17 @@ expression: parse_ast
),
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 100..105,
value: FStringValue {
inner: Single(
FString(
FString {
range: 100..105,
values: [
StringLiteral(
ExprStringLiteral {
range: 100..105,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 100..105,
value: ".3f ",
unicode: false,
},
),
},
},
),
],
},
),
),
},
},
),
FStringFormatSpec {
range: 100..105,
elements: [
Literal(
FStringLiteralElement {
range: 100..105,
value: ".3f ",
},
),
],
},
),
},
),
@ -502,11 +434,11 @@ expression: parse_ast
FString(
FString {
range: 108..126,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 110..125,
value: Name(
expression: Name(
ExprName {
range: 113..116,
id: "foo",
@ -543,11 +475,11 @@ expression: parse_ast
FString(
FString {
range: 127..143,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 129..142,
value: Tuple(
expression: Tuple(
ExprTuple {
range: 132..136,
elts: [
@ -601,11 +533,11 @@ expression: parse_ast
FString(
FString {
range: 144..170,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 146..169,
value: FString(
expression: FString(
ExprFString {
range: 147..163,
value: FStringValue {
@ -613,11 +545,11 @@ expression: parse_ast
FString(
FString {
range: 147..163,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 149..162,
value: NumberLiteral(
expression: NumberLiteral(
ExprNumberLiteral {
range: 150..156,
value: Float(
@ -633,36 +565,17 @@ expression: parse_ast
),
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 158..161,
value: FStringValue {
inner: Single(
FString(
FString {
range: 158..161,
values: [
StringLiteral(
ExprStringLiteral {
range: 158..161,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 158..161,
value: ".1f",
unicode: false,
},
),
},
},
),
],
},
),
),
},
},
),
FStringFormatSpec {
range: 158..161,
elements: [
Literal(
FStringLiteralElement {
range: 158..161,
value: ".1f",
},
),
],
},
),
},
),
@ -676,36 +589,17 @@ expression: parse_ast
debug_text: None,
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 164..168,
value: FStringValue {
inner: Single(
FString(
FString {
range: 164..168,
values: [
StringLiteral(
ExprStringLiteral {
range: 164..168,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 164..168,
value: "*^20",
unicode: false,
},
),
},
},
),
],
},
),
),
},
},
),
FStringFormatSpec {
range: 164..168,
elements: [
Literal(
FStringLiteralElement {
range: 164..168,
value: "*^20",
},
),
],
},
),
},
),
@ -742,25 +636,17 @@ expression: parse_ast
FString(
FString {
range: 180..195,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 182..186,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 182..186,
value: "bar ",
unicode: false,
},
),
},
value: "bar ",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 186..193,
value: BinOp(
expression: BinOp(
ExprBinOp {
range: 187..192,
left: Name(
@ -785,18 +671,10 @@ expression: parse_ast
format_spec: None,
},
),
StringLiteral(
ExprStringLiteral {
Literal(
FStringLiteralElement {
range: 193..194,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 193..194,
value: " ",
unicode: false,
},
),
},
value: " ",
},
),
],
@ -925,25 +803,17 @@ expression: parse_ast
FString(
FString {
range: 300..317,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 302..303,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 302..303,
value: "\\",
unicode: false,
},
),
},
value: "\\",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 303..308,
value: Name(
expression: Name(
ExprName {
range: 304..307,
id: "foo",
@ -955,24 +825,16 @@ expression: parse_ast
format_spec: None,
},
),
StringLiteral(
ExprStringLiteral {
Literal(
FStringLiteralElement {
range: 308..309,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 308..309,
value: "\\",
unicode: false,
},
),
},
value: "\\",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 309..316,
value: Name(
expression: Name(
ExprName {
range: 310..313,
id: "bar",
@ -982,36 +844,17 @@ expression: parse_ast
debug_text: None,
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 314..315,
value: FStringValue {
inner: Single(
FString(
FString {
range: 314..315,
values: [
StringLiteral(
ExprStringLiteral {
range: 314..315,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 314..315,
value: "\\",
unicode: false,
},
),
},
},
),
],
},
),
),
},
},
),
FStringFormatSpec {
range: 314..315,
elements: [
Literal(
FStringLiteralElement {
range: 314..315,
value: "\\",
},
),
],
},
),
},
),
@ -1035,19 +878,11 @@ expression: parse_ast
FString(
FString {
range: 318..332,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 320..331,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 320..331,
value: "\\{foo\\}",
unicode: false,
},
),
},
value: "\\{foo\\}",
},
),
],
@ -1070,11 +905,11 @@ expression: parse_ast
FString(
FString {
range: 333..373,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 337..370,
value: Name(
expression: Name(
ExprName {
range: 343..346,
id: "foo",
@ -1084,36 +919,17 @@ expression: parse_ast
debug_text: None,
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 347..369,
value: FStringValue {
inner: Single(
FString(
FString {
range: 347..369,
values: [
StringLiteral(
ExprStringLiteral {
range: 347..369,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 347..369,
value: "x\n y\n z\n",
unicode: false,
},
),
},
},
),
],
},
),
),
},
},
),
FStringFormatSpec {
range: 347..369,
elements: [
Literal(
FStringLiteralElement {
range: 347..369,
value: "x\n y\n z\n",
},
),
],
},
),
},
),

View file

@ -22,11 +22,11 @@ expression: parse_ast
FString(
FString {
range: 7..15,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 9..14,
value: Name(
expression: Name(
ExprName {
range: 10..13,
id: "bar",
@ -81,11 +81,11 @@ expression: parse_ast
FString(
FString {
range: 36..44,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 38..43,
value: Name(
expression: Name(
ExprName {
range: 39..42,
id: "bar",
@ -140,11 +140,11 @@ expression: parse_ast
FString(
FString {
range: 66..74,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 68..73,
value: Name(
expression: Name(
ExprName {
range: 69..72,
id: "bar",
@ -199,25 +199,17 @@ expression: parse_ast
FString(
FString {
range: 97..116,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 99..103,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 99..103,
value: "bar ",
unicode: false,
},
),
},
value: "bar ",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 103..108,
value: Name(
expression: Name(
ExprName {
range: 104..107,
id: "baz",
@ -229,18 +221,10 @@ expression: parse_ast
format_spec: None,
},
),
StringLiteral(
ExprStringLiteral {
Literal(
FStringLiteralElement {
range: 108..115,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 108..115,
value: " really",
unicode: false,
},
),
},
value: " really",
},
),
],

View file

@ -14,19 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..14,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 2..13,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 2..13,
value: "Hello world",
unicode: false,
},
),
},
value: "Hello world",
},
),
],

View file

@ -86,25 +86,17 @@ expression: parse_ast
FString(
FString {
range: 62..81,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 64..71,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 64..71,
value: "caught ",
unicode: false,
},
),
},
value: "caught ",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 71..80,
value: Call(
expression: Call(
ExprCall {
range: 72..79,
func: Name(
@ -194,25 +186,17 @@ expression: parse_ast
FString(
FString {
range: 114..133,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 116..123,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 116..123,
value: "caught ",
unicode: false,
},
),
},
value: "caught ",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 123..132,
value: Call(
expression: Call(
ExprCall {
range: 124..131,
func: Name(

View file

@ -204,25 +204,17 @@ expression: parse_ast
FString(
FString {
range: 133..179,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 135..142,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 135..142,
value: "caught ",
unicode: false,
},
),
},
value: "caught ",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 142..151,
value: Call(
expression: Call(
ExprCall {
range: 143..150,
func: Name(
@ -252,24 +244,16 @@ expression: parse_ast
format_spec: None,
},
),
StringLiteral(
ExprStringLiteral {
Literal(
FStringLiteralElement {
range: 151..164,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 151..164,
value: " with nested ",
unicode: false,
},
),
},
value: " with nested ",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 164..178,
value: Attribute(
expression: Attribute(
ExprAttribute {
range: 165..177,
value: Name(
@ -351,25 +335,17 @@ expression: parse_ast
FString(
FString {
range: 213..259,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 215..222,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 215..222,
value: "caught ",
unicode: false,
},
),
},
value: "caught ",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 222..231,
value: Call(
expression: Call(
ExprCall {
range: 223..230,
func: Name(
@ -399,24 +375,16 @@ expression: parse_ast
format_spec: None,
},
),
StringLiteral(
ExprStringLiteral {
Literal(
FStringLiteralElement {
range: 231..244,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 231..244,
value: " with nested ",
unicode: false,
},
),
},
value: " with nested ",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 244..258,
value: Attribute(
expression: Attribute(
ExprAttribute {
range: 245..257,
value: Name(

View file

@ -14,25 +14,17 @@ expression: parse_ast
FString(
FString {
range: 0..22,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 2..5,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 2..5,
value: "aaa",
unicode: false,
},
),
},
value: "aaa",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 5..10,
value: Name(
expression: Name(
ExprName {
range: 6..9,
id: "bbb",
@ -44,24 +36,16 @@ expression: parse_ast
format_spec: None,
},
),
StringLiteral(
ExprStringLiteral {
Literal(
FStringLiteralElement {
range: 10..13,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 10..13,
value: "ccc",
unicode: false,
},
),
},
value: "ccc",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 13..18,
value: Name(
expression: Name(
ExprName {
range: 14..17,
id: "ddd",
@ -73,18 +57,10 @@ expression: parse_ast
format_spec: None,
},
),
StringLiteral(
ExprStringLiteral {
Literal(
FStringLiteralElement {
range: 18..21,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 18..21,
value: "eee",
unicode: false,
},
),
},
value: "eee",
},
),
],

View file

@ -14,25 +14,17 @@ expression: parse_ast
FString(
FString {
range: 0..8,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 2..4,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 2..4,
value: "\\",
unicode: false,
},
),
},
value: "\\",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 4..7,
value: Name(
expression: Name(
ExprName {
range: 5..6,
id: "x",

View file

@ -14,25 +14,17 @@ expression: parse_ast
FString(
FString {
range: 0..8,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 2..4,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 2..4,
value: "\n",
unicode: false,
},
),
},
value: "\n",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 4..7,
value: Name(
expression: Name(
ExprName {
range: 5..6,
id: "x",

View file

@ -14,25 +14,17 @@ expression: parse_ast
FString(
FString {
range: 0..9,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 3..5,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 3..5,
value: "\\\n",
unicode: false,
},
),
},
value: "\\\n",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 5..8,
value: Name(
expression: Name(
ExprName {
range: 6..7,
id: "x",

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..10,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..9,
value: Name(
expression: Name(
ExprName {
range: 3..7,
id: "user",

View file

@ -14,25 +14,17 @@ expression: parse_ast
FString(
FString {
range: 0..38,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 2..6,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 2..6,
value: "mix ",
unicode: false,
},
),
},
value: "mix ",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 6..13,
value: Name(
expression: Name(
ExprName {
range: 7..11,
id: "user",
@ -49,24 +41,16 @@ expression: parse_ast
format_spec: None,
},
),
StringLiteral(
ExprStringLiteral {
Literal(
FStringLiteralElement {
range: 13..28,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 13..28,
value: " with text and ",
unicode: false,
},
),
},
value: " with text and ",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 28..37,
value: Name(
expression: Name(
ExprName {
range: 29..35,
id: "second",

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..14,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..13,
value: Name(
expression: Name(
ExprName {
range: 3..7,
id: "user",
@ -33,36 +33,17 @@ expression: parse_ast
),
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 9..12,
value: FStringValue {
inner: Single(
FString(
FString {
range: 9..12,
values: [
StringLiteral(
ExprStringLiteral {
range: 9..12,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 9..12,
value: ">10",
unicode: false,
},
),
},
},
),
],
},
),
),
},
},
),
FStringFormatSpec {
range: 9..12,
elements: [
Literal(
FStringLiteralElement {
range: 9..12,
value: ">10",
},
),
],
},
),
},
),

View file

@ -14,25 +14,17 @@ expression: parse_ast
FString(
FString {
range: 0..11,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 4..5,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 4..5,
value: "\n",
unicode: false,
},
),
},
value: "\n",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 5..8,
value: Name(
expression: Name(
ExprName {
range: 6..7,
id: "x",

View file

@ -14,7 +14,7 @@ expression: "parse_suite(r#\"f\"\"\"#, \"<test>\").unwrap()"
FString(
FString {
range: 0..3,
values: [],
elements: [],
},
),
),

View file

@ -22,19 +22,11 @@ expression: parse_ast
FString(
FString {
range: 9..17,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 11..16,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 11..16,
value: "world",
unicode: false,
},
),
},
value: "world",
},
),
],

View file

@ -22,19 +22,11 @@ expression: parse_ast
FString(
FString {
range: 9..17,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 11..16,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 11..16,
value: "world",
unicode: false,
},
),
},
value: "world",
},
),
],

View file

@ -22,25 +22,17 @@ expression: parse_ast
FString(
FString {
range: 9..22,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 11..16,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 11..16,
value: "world",
unicode: false,
},
),
},
value: "world",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 16..21,
value: StringLiteral(
expression: StringLiteral(
ExprStringLiteral {
range: 17..20,
value: StringLiteralValue {

View file

@ -22,25 +22,17 @@ expression: parse_ast
FString(
FString {
range: 9..22,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 11..16,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 11..16,
value: "world",
unicode: false,
},
),
},
value: "world",
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 16..21,
value: StringLiteral(
expression: StringLiteral(
ExprStringLiteral {
range: 17..20,
value: StringLiteralValue {

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..18,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..5,
value: Name(
expression: Name(
ExprName {
range: 3..4,
id: "a",
@ -30,10 +30,10 @@ expression: parse_ast
format_spec: None,
},
),
FormattedValue(
ExprFormattedValue {
Expression(
FStringExpressionElement {
range: 5..10,
value: Name(
expression: Name(
ExprName {
range: 7..8,
id: "b",
@ -45,18 +45,10 @@ expression: parse_ast
format_spec: None,
},
),
StringLiteral(
ExprStringLiteral {
Literal(
FStringLiteralElement {
range: 10..17,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 10..17,
value: "{foo}",
unicode: false,
},
),
},
value: "{foo}",
},
),
],

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..13,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..12,
value: Compare(
expression: Compare(
ExprCompare {
range: 3..11,
left: NumberLiteral(

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..16,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..15,
value: Name(
expression: Name(
ExprName {
range: 3..6,
id: "foo",
@ -28,54 +28,43 @@ expression: parse_ast
debug_text: None,
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 7..14,
value: FStringValue {
inner: Single(
FString(
FString {
range: 7..14,
values: [
FormattedValue(
ExprFormattedValue {
range: 7..14,
value: StringLiteral(
ExprStringLiteral {
range: 8..13,
value: StringLiteralValue {
inner: Concatenated(
ConcatenatedStringLiteral {
strings: [
StringLiteral {
range: 8..10,
value: "",
unicode: false,
},
StringLiteral {
range: 11..13,
value: "",
unicode: false,
},
],
value: "",
},
),
},
FStringFormatSpec {
range: 7..14,
elements: [
Expression(
FStringExpressionElement {
range: 7..14,
expression: StringLiteral(
ExprStringLiteral {
range: 8..13,
value: StringLiteralValue {
inner: Concatenated(
ConcatenatedStringLiteral {
strings: [
StringLiteral {
range: 8..10,
value: "",
unicode: false,
},
),
debug_text: None,
conversion: None,
format_spec: None,
StringLiteral {
range: 11..13,
value: "",
unicode: false,
},
],
value: "",
},
),
],
},
},
),
),
},
},
),
debug_text: None,
conversion: None,
format_spec: None,
},
),
],
},
),
},
),

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..15,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..14,
value: Name(
expression: Name(
ExprName {
range: 3..6,
id: "foo",
@ -28,37 +28,26 @@ expression: parse_ast
debug_text: None,
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 7..13,
value: FStringValue {
inner: Single(
FString(
FString {
range: 7..13,
values: [
FormattedValue(
ExprFormattedValue {
range: 7..13,
value: Name(
ExprName {
range: 8..12,
id: "spec",
ctx: Load,
},
),
debug_text: None,
conversion: None,
format_spec: None,
},
),
],
FStringFormatSpec {
range: 7..13,
elements: [
Expression(
FStringExpressionElement {
range: 7..13,
expression: Name(
ExprName {
range: 8..12,
id: "spec",
ctx: Load,
},
),
),
},
},
),
debug_text: None,
conversion: None,
format_spec: None,
},
),
],
},
),
},
),

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..13,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..12,
value: Name(
expression: Name(
ExprName {
range: 3..6,
id: "foo",
@ -28,44 +28,33 @@ expression: parse_ast
debug_text: None,
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 7..11,
value: FStringValue {
inner: Single(
FString(
FString {
range: 7..11,
values: [
FormattedValue(
ExprFormattedValue {
range: 7..11,
value: StringLiteral(
ExprStringLiteral {
range: 8..10,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 8..10,
value: "",
unicode: false,
},
),
},
},
),
debug_text: None,
conversion: None,
format_spec: None,
FStringFormatSpec {
range: 7..11,
elements: [
Expression(
FStringExpressionElement {
range: 7..11,
expression: StringLiteral(
ExprStringLiteral {
range: 8..10,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 8..10,
value: "",
unicode: false,
},
),
],
},
},
),
),
},
},
),
debug_text: None,
conversion: None,
format_spec: None,
},
),
],
},
),
},
),

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..11,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..10,
value: Compare(
expression: Compare(
ExprCompare {
range: 3..9,
left: NumberLiteral(

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..13,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..12,
value: Name(
expression: Name(
ExprName {
range: 3..6,
id: "foo",
@ -28,36 +28,17 @@ expression: parse_ast
debug_text: None,
conversion: None,
format_spec: Some(
FString(
ExprFString {
range: 7..11,
value: FStringValue {
inner: Single(
FString(
FString {
range: 7..11,
values: [
StringLiteral(
ExprStringLiteral {
range: 7..11,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 7..11,
value: "spec",
unicode: false,
},
),
},
},
),
],
},
),
),
},
},
),
FStringFormatSpec {
range: 7..11,
elements: [
Literal(
FStringLiteralElement {
range: 7..11,
value: "spec",
},
),
],
},
),
},
),

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..10,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..9,
value: Name(
expression: Name(
ExprName {
range: 3..4,
id: "x",

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..10,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..9,
value: Name(
expression: Name(
ExprName {
range: 3..4,
id: "x",

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..10,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 2..9,
value: Yield(
expression: Yield(
ExprYield {
range: 3..8,
value: None,

View file

@ -22,19 +22,11 @@ expression: parse_ast
FString(
FString {
range: 10..18,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 12..17,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 12..17,
value: "world",
unicode: false,
},
),
},
value: "world",
},
),
],

View file

@ -22,19 +22,11 @@ expression: parse_ast
FString(
FString {
range: 10..18,
values: [
StringLiteral(
ExprStringLiteral {
elements: [
Literal(
FStringLiteralElement {
range: 12..17,
value: StringLiteralValue {
inner: Single(
StringLiteral {
range: 12..17,
value: "world",
unicode: false,
},
),
},
value: "world",
},
),
],

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..7,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 3..6,
value: Name(
expression: Name(
ExprName {
range: 4..5,
id: "x",

View file

@ -14,11 +14,11 @@ expression: parse_ast
FString(
FString {
range: 0..11,
values: [
FormattedValue(
ExprFormattedValue {
elements: [
Expression(
FStringExpressionElement {
range: 5..8,
value: Name(
expression: Name(
ExprName {
range: 6..7,
id: "x",

View file

@ -202,7 +202,7 @@ impl<'a> StringParser<'a> {
Ok(())
}
fn parse_fstring_middle(&mut self) -> Result<Expr, LexicalError> {
fn parse_fstring_middle(&mut self) -> Result<ast::FStringElement, LexicalError> {
let mut value = String::new();
while let Some(ch) = self.next_char() {
match ch {
@ -239,9 +239,8 @@ impl<'a> StringParser<'a> {
ch => value.push(ch),
}
}
Ok(Expr::from(ast::StringLiteral {
Ok(ast::FStringElement::Literal(ast::FStringLiteralElement {
value,
unicode: false,
range: self.range,
}))
}
@ -324,11 +323,11 @@ pub(crate) fn parse_string_literal(
StringParser::new(source, kind, start_location, range).parse()
}
pub(crate) fn parse_fstring_middle(
pub(crate) fn parse_fstring_literal_element(
source: &str,
is_raw: bool,
range: TextRange,
) -> Result<Expr, LexicalError> {
) -> Result<ast::FStringElement, LexicalError> {
let kind = if is_raw {
StringKind::RawString
} else {