Auto generate visit_source_order (#17180)

## Summary

part of: #15655 

I tried generating the source order function using code generation. I
tried a simple approach, but it is not enough to generate all of them
this way.

There is one good thing, that most of the implementations are fine with
this. We only have a few that are not. So one benefit of this PR could
be it eliminates a lot of the code, hence changing the AST structure
will only leave a few places to be fixed.

The `source_order` field determines if a node requires a source order
implementation. If it’s empty it means source order does not visit
anything.

Initially I didn’t want to repeat the field names. But I found two
things:
- `ExprIf` statement unlike other statements does not have the fields
defined in source order. This and also some fields do not need to be
included in the visit. So we just need a way to determine order, and
determine presence.
- Relying on the fields sounds more complicated to me. Maybe another
solution is to add a new attribute `order` to each field? I'm open to
suggestions.
But anyway, except for the `ExprIf` we don't need to write the field
names in order. Just knowing what fields must be visited are enough.

Some nodes had a more complex visitor:

`ExprCompare` required zipping two fields.

`ExprBoolOp` required a match over the fields.

`FstringValue` required a match, I created a new walk_ function that
does the match. and used it in code generation. I don’t think this
provides real value. Because I mostly moved the code from one file to
another. I was tried it as an option. I prefer to leave it in the code
as before.

Some visitors visit a slice of items. Others visit a single element. I
put a check on this in code generation to see if the field requires a
for loop or not. I think better approach is to have a consistent style.
So we can by default loop over any field that is a sequence.

For field types `StringLiteralValue` and `BytesLiteralValue` the types
are not a sequence in toml definition. But they implement `iter` so they
are iterated over. So the code generation does not properly identify
this. So in the code I'm checking for their types.

## Test Plan

All the tests should pass without any changes.
I checked the generated code to make sure it's the same as old code. I'm
not sure if there's a test for the source order visitor.
This commit is contained in:
Shaygan Hooshyari 2025-04-17 14:59:57 +02:00 committed by GitHub
parent bd89838212
commit 3ada36b766
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 1048 additions and 886 deletions

View file

@ -6,255 +6,6 @@ use crate::{
PatternKeyword,
};
impl ast::ModModule {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ModModule { body, range: _ } = self;
visitor.visit_body(body);
}
}
impl ast::ModExpression {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ModExpression { body, range: _ } = self;
visitor.visit_expr(body);
}
}
impl ast::StmtFunctionDef {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtFunctionDef {
parameters,
body,
decorator_list,
returns,
type_params,
range: _,
is_async: _,
name,
} = self;
for decorator in decorator_list {
visitor.visit_decorator(decorator);
}
visitor.visit_identifier(name);
if let Some(type_params) = type_params {
visitor.visit_type_params(type_params);
}
visitor.visit_parameters(parameters);
if let Some(expr) = returns {
visitor.visit_annotation(expr);
}
visitor.visit_body(body);
}
}
impl ast::StmtClassDef {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtClassDef {
arguments,
body,
decorator_list,
type_params,
name,
range: _,
} = self;
for decorator in decorator_list {
visitor.visit_decorator(decorator);
}
visitor.visit_identifier(name);
if let Some(type_params) = type_params {
visitor.visit_type_params(type_params);
}
if let Some(arguments) = arguments {
visitor.visit_arguments(arguments);
}
visitor.visit_body(body);
}
}
impl ast::StmtReturn {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtReturn { value, range: _ } = self;
if let Some(expr) = value {
visitor.visit_expr(expr);
}
}
}
impl ast::StmtDelete {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtDelete { targets, range: _ } = self;
for expr in targets {
visitor.visit_expr(expr);
}
}
}
impl ast::StmtTypeAlias {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtTypeAlias {
range: _,
name,
type_params,
value,
} = self;
visitor.visit_expr(name);
if let Some(type_params) = type_params {
visitor.visit_type_params(type_params);
}
visitor.visit_expr(value);
}
}
impl ast::StmtAssign {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtAssign {
targets,
value,
range: _,
} = self;
for expr in targets {
visitor.visit_expr(expr);
}
visitor.visit_expr(value);
}
}
impl ast::StmtAugAssign {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtAugAssign {
target,
op,
value,
range: _,
} = self;
visitor.visit_expr(target);
visitor.visit_operator(op);
visitor.visit_expr(value);
}
}
impl ast::StmtAnnAssign {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtAnnAssign {
target,
annotation,
value,
range: _,
simple: _,
} = self;
visitor.visit_expr(target);
visitor.visit_annotation(annotation);
if let Some(expr) = value {
visitor.visit_expr(expr);
}
}
}
impl ast::StmtFor {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtFor {
target,
iter,
body,
orelse,
range: _,
is_async: _,
} = self;
visitor.visit_expr(target);
visitor.visit_expr(iter);
visitor.visit_body(body);
visitor.visit_body(orelse);
}
}
impl ast::StmtWhile {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtWhile {
test,
body,
orelse,
range: _,
} = self;
visitor.visit_expr(test);
visitor.visit_body(body);
visitor.visit_body(orelse);
}
}
impl ast::StmtIf {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtIf {
test,
body,
elif_else_clauses,
range: _,
} = self;
visitor.visit_expr(test);
visitor.visit_body(body);
for clause in elif_else_clauses {
visitor.visit_elif_else_clause(clause);
}
}
}
impl ast::ElifElseClause {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
@ -272,215 +23,19 @@ impl ast::ElifElseClause {
}
}
impl ast::StmtWith {
impl ast::ExprDict {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtWith {
items,
body,
is_async: _,
range: _,
} = self;
let ast::ExprDict { items, range: _ } = self;
for with_item in items {
visitor.visit_with_item(with_item);
for ast::DictItem { key, value } in items {
if let Some(key) = key {
visitor.visit_expr(key);
}
visitor.visit_expr(value);
}
visitor.visit_body(body);
}
}
impl ast::StmtMatch {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtMatch {
subject,
cases,
range: _,
} = self;
visitor.visit_expr(subject);
for match_case in cases {
visitor.visit_match_case(match_case);
}
}
}
impl ast::StmtRaise {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtRaise {
exc,
cause,
range: _,
} = self;
if let Some(expr) = exc {
visitor.visit_expr(expr);
}
if let Some(expr) = cause {
visitor.visit_expr(expr);
}
}
}
impl ast::StmtTry {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtTry {
body,
handlers,
orelse,
finalbody,
is_star: _,
range: _,
} = self;
visitor.visit_body(body);
for except_handler in handlers {
visitor.visit_except_handler(except_handler);
}
visitor.visit_body(orelse);
visitor.visit_body(finalbody);
}
}
impl ast::StmtAssert {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtAssert {
test,
msg,
range: _,
} = self;
visitor.visit_expr(test);
if let Some(expr) = msg {
visitor.visit_expr(expr);
}
}
}
impl ast::StmtImport {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtImport { names, range: _ } = self;
for alias in names {
visitor.visit_alias(alias);
}
}
}
impl ast::StmtImportFrom {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtImportFrom {
range: _,
module,
names,
level: _,
} = self;
if let Some(module) = module {
visitor.visit_identifier(module);
}
for alias in names {
visitor.visit_alias(alias);
}
}
}
impl ast::StmtGlobal {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtGlobal { range: _, names } = self;
for name in names {
visitor.visit_identifier(name);
}
}
}
impl ast::StmtNonlocal {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtNonlocal { range: _, names } = self;
for name in names {
visitor.visit_identifier(name);
}
}
}
impl ast::StmtExpr {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtExpr { value, range: _ } = self;
visitor.visit_expr(value);
}
}
impl ast::StmtPass {
#[inline]
pub(crate) fn visit_source_order<'a, V>(&'a self, _visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtPass { range: _ } = self;
}
}
impl ast::StmtBreak {
#[inline]
pub(crate) fn visit_source_order<'a, V>(&'a self, _visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtBreak { range: _ } = self;
}
}
impl ast::StmtContinue {
#[inline]
pub(crate) fn visit_source_order<'a, V>(&'a self, _visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtContinue { range: _ } = self;
}
}
impl ast::StmtIpyEscapeCommand {
#[inline]
pub(crate) fn visit_source_order<'a, V>(&'a self, _visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::StmtIpyEscapeCommand {
range: _,
kind: _,
value: _,
} = self;
}
}
@ -509,227 +64,6 @@ impl ast::ExprBoolOp {
}
}
impl ast::ExprNamed {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprNamed {
target,
value,
range: _,
} = self;
visitor.visit_expr(target);
visitor.visit_expr(value);
}
}
impl ast::ExprBinOp {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprBinOp {
left,
op,
right,
range: _,
} = self;
visitor.visit_expr(left);
visitor.visit_operator(op);
visitor.visit_expr(right);
}
}
impl ast::ExprUnaryOp {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprUnaryOp {
op,
operand,
range: _,
} = self;
visitor.visit_unary_op(op);
visitor.visit_expr(operand);
}
}
impl ast::ExprLambda {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprLambda {
parameters,
body,
range: _,
} = self;
if let Some(parameters) = parameters {
visitor.visit_parameters(parameters);
}
visitor.visit_expr(body);
}
}
impl ast::ExprIf {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprIf {
test,
body,
orelse,
range: _,
} = self;
// `body if test else orelse`
visitor.visit_expr(body);
visitor.visit_expr(test);
visitor.visit_expr(orelse);
}
}
impl ast::ExprDict {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprDict { items, range: _ } = self;
for ast::DictItem { key, value } in items {
if let Some(key) = key {
visitor.visit_expr(key);
}
visitor.visit_expr(value);
}
}
}
impl ast::ExprSet {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprSet { elts, range: _ } = self;
for expr in elts {
visitor.visit_expr(expr);
}
}
}
impl ast::ExprListComp {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprListComp {
elt,
generators,
range: _,
} = self;
visitor.visit_expr(elt);
for comprehension in generators {
visitor.visit_comprehension(comprehension);
}
}
}
impl ast::ExprSetComp {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprSetComp {
elt,
generators,
range: _,
} = self;
visitor.visit_expr(elt);
for comprehension in generators {
visitor.visit_comprehension(comprehension);
}
}
}
impl ast::ExprDictComp {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprDictComp {
key,
value,
generators,
range: _,
} = self;
visitor.visit_expr(key);
visitor.visit_expr(value);
for comprehension in generators {
visitor.visit_comprehension(comprehension);
}
}
}
impl ast::ExprGenerator {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprGenerator {
elt,
generators,
range: _,
parenthesized: _,
} = self;
visitor.visit_expr(elt);
for comprehension in generators {
visitor.visit_comprehension(comprehension);
}
}
}
impl ast::ExprAwait {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprAwait { value, range: _ } = self;
visitor.visit_expr(value);
}
}
impl ast::ExprYield {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprYield { value, range: _ } = self;
if let Some(expr) = value {
visitor.visit_expr(expr);
}
}
}
impl ast::ExprYieldFrom {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprYieldFrom { value, range: _ } = self;
visitor.visit_expr(value);
}
}
impl ast::ExprCompare {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
@ -751,21 +85,6 @@ impl ast::ExprCompare {
}
}
impl ast::ExprCall {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprCall {
func,
arguments,
range: _,
} = self;
visitor.visit_expr(func);
visitor.visit_arguments(arguments);
}
}
impl ast::FStringFormatSpec {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
@ -852,181 +171,6 @@ impl ast::ExprBytesLiteral {
}
}
impl ast::ExprNumberLiteral {
#[inline]
pub(crate) fn visit_source_order<'a, V>(&'a self, _visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprNumberLiteral { range: _, value: _ } = self;
}
}
impl ast::ExprBooleanLiteral {
#[inline]
pub(crate) fn visit_source_order<'a, V>(&'a self, _visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprBooleanLiteral { range: _, value: _ } = self;
}
}
impl ast::ExprNoneLiteral {
#[inline]
pub(crate) fn visit_source_order<'a, V>(&'a self, _visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprNoneLiteral { range: _ } = self;
}
}
impl ast::ExprEllipsisLiteral {
#[inline]
pub(crate) fn visit_source_order<'a, V>(&'a self, _visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprEllipsisLiteral { range: _ } = self;
}
}
impl ast::ExprAttribute {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprAttribute {
value,
attr,
ctx: _,
range: _,
} = self;
visitor.visit_expr(value);
visitor.visit_identifier(attr);
}
}
impl ast::ExprSubscript {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprSubscript {
value,
slice,
ctx: _,
range: _,
} = self;
visitor.visit_expr(value);
visitor.visit_expr(slice);
}
}
impl ast::ExprStarred {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprStarred {
value,
ctx: _,
range: _,
} = self;
visitor.visit_expr(value);
}
}
impl ast::ExprName {
#[inline]
pub(crate) fn visit_source_order<'a, V>(&'a self, _visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprName {
range: _,
id: _,
ctx: _,
} = self;
}
}
impl ast::ExprList {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprList {
elts,
ctx: _,
range: _,
} = self;
for expr in elts {
visitor.visit_expr(expr);
}
}
}
impl ast::ExprTuple {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprTuple {
elts,
ctx: _,
range: _,
parenthesized: _,
} = self;
for expr in elts {
visitor.visit_expr(expr);
}
}
}
impl ast::ExprSlice {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprSlice {
lower,
upper,
step,
range: _,
} = self;
if let Some(expr) = lower {
visitor.visit_expr(expr);
}
if let Some(expr) = upper {
visitor.visit_expr(expr);
}
if let Some(expr) = step {
visitor.visit_expr(expr);
}
}
}
impl ast::ExprIpyEscapeCommand {
#[inline]
pub(crate) fn visit_source_order<'a, V>(&'a self, _visitor: &mut V)
where
V: SourceOrderVisitor<'a> + ?Sized,
{
let ast::ExprIpyEscapeCommand {
range: _,
kind: _,
value: _,
} = self;
}
}
impl ast::ExceptHandlerExceptHandler {
pub(crate) fn visit_source_order<'a, V>(&'a self, visitor: &mut V)
where