format doctests in docstrings (#8811)

## Summary

This PR adds opt-in support for formatting doctests in docstrings. This
reflects initial support and it is intended to add support for Markdown
and reStructuredText Python code blocks in the future. But I believe
this PR lays the groundwork, and future additions for Markdown and reST
should be less costly to add.

It's strongly recommended to review this PR commit-by-commit. The last
few commits in particular implement the bulk of the work here and
represent the denser portions.

Some things worth mentioning:

* The formatter is itself not perfect, and it is possible for it to
produce invalid Python code. Because of this, reformatted code snippets
are checked for Python validity. If they aren't valid, then we
(unfortunately silently) bail on formatting that code snippet.
* There are a couple places where it would be nice to at least warn the
user that doctest formatting failed, but it wasn't clear to me what the
best way to do that is.
* I haven't yet run this in anger on a real world code base. I think
that should happen before merging.

Closes #7146 

## Test Plan

* [x] Pass the local test suite.
* [x] Scrutinize ecosystem changes.
* [x] Run this formatter on extant code and scrutinize the results.
(e.g., CPython, numpy.)

crates/ruff_python_formatter/tests/normalizer.rs

@@ -1,4 +1,8 @@
-use itertools::Either::{Left, Right};
+use {
+    itertools::Either::{Left, Right},
+    once_cell::sync::Lazy,
+    regex::Regex,
+};
 
 use ruff_python_ast::visitor::transformer;
 use ruff_python_ast::visitor::transformer::Transformer;
@@ -12,6 +16,10 @@ use ruff_python_ast::{self as ast, Expr, Stmt};
 ///   between `class C: ...` and `class C(): ...`, which is part of our AST but not `CPython`'s.
 /// - Normalize strings. The formatter can re-indent docstrings, so we need to compare string
 ///   contents ignoring whitespace. (Black does the same.)
+/// - The formatter can also reformat code snippets when they're Python code, which can of
+///   course change the string in arbitrary ways. Black itself does not reformat code snippets,
+///   so we carve our own path here by stripping everything that looks like code snippets from
+///   string literals.
 /// - Ignores nested tuples in deletions. (Black does the same.)
 pub(crate) struct Normalizer;
 
@@ -52,8 +60,28 @@ impl Transformer for Normalizer {
     }
 
     fn visit_string_literal(&self, string_literal: &mut ast::StringLiteral) {
-        // Normalize a string by (1) stripping any leading and trailing space from each
-        // line, and (2) removing any blank lines from the start and end of the string.
+        static STRIP_CODE_SNIPPETS: Lazy<Regex> = Lazy::new(|| {
+            Regex::new(
+                r#"(?mx)
+                    # strip doctest PS1 prompt lines
+                    ^\s*>>>\s.*(\n|$)
+                    |
+                    # strip doctest PS2 prompt lines
+                    # Also handles the case of an empty ... line.
+                    ^\s*\.\.\.((\n|$)|\s.*(\n|$))
+                "#,
+            )
+            .unwrap()
+        });
+
+        // Start by (1) stripping everything that looks like a code
+        // snippet, since code snippets may be completely reformatted if
+        // they are Python code.
+        string_literal.value = STRIP_CODE_SNIPPETS
+            .replace_all(&string_literal.value, "")
+            .into_owned();
+        // Normalize a string by (2) stripping any leading and trailing space from each
+        // line, and (3) removing any blank lines from the start and end of the string.
         string_literal.value = string_literal
             .value
             .lines()