Add an option to bytecode compile during installation (#2086)

Add a `--compile` option to `pip install` and `pip sync`.

I chose to implement this as a separate pass over the entire venv. If we
wanted to compile during installation, we'd have to make sure that
writing is exclusive, to avoid concurrent processes writing broken
`.pyc` files. Additionally, this ensures that the entire site-packages
are bytecode compiled, even if there are packages that aren't from this
`uv` invocation. The disadvantage is that we do not update RECORD and
rely on this comment from [PEP 491](https://peps.python.org/pep-0491/):

> Uninstallers should be smart enough to remove .pyc even if it is not
mentioned in RECORD.

If this is a problem we can change it to run during installation and
write RECORD entries.

Internally, this is implemented as an async work-stealing subprocess
worker pool. The producer is a directory traversal over site-packages,
sending each `.py` file to a bounded async FIFO queue/channel. Each
worker has a long-running python process. It pops the queue to get a
single path (or exists if the channel is closed), then sends it to
stdin, waits until it's informed that the compilation is done through a
line on stdout, and repeat. This is fast, e.g. installing `jupyter
plotly` on Python 3.12 it processes 15876 files in 319ms with 32 threads
(vs. 3.8s with a single core). The python processes internally calls
`compileall.compile_file`, the same as pip.

Like pip, we ignore and silence all compilation errors
(https://github.com/astral-sh/uv/issues/1559). There is a 10s timeout to
handle the case when the workers got stuck. For the reviewers, please
check if i missed any spots where we could deadlock, this is the hardest
part of this PR.

I've added `uv-dev compile <dir>` and `uv-dev clear-compile <dir>`
commands, mainly for my own benchmarking. I don't want to expose them in
`uv`, they almost certainly not the correct workflow and we don't want
to support them.

Fixes #1788
Closes #1559
Closes #1928

crates/uv-dev/src/compile.rs

@@ -0,0 +1,37 @@
+use std::path::PathBuf;
+
+use clap::Parser;
+use platform_host::Platform;
+use tracing::info;
+use uv_cache::{Cache, CacheArgs};
+use uv_interpreter::PythonEnvironment;
+
+#[derive(Parser)]
+pub(crate) struct CompileArgs {
+    /// Compile all `.py` in this or any subdirectory to bytecode
+    root: PathBuf,
+    python: Option<PathBuf>,
+    #[command(flatten)]
+    cache_args: CacheArgs,
+}
+
+pub(crate) async fn compile(args: CompileArgs) -> anyhow::Result<()> {
+    let cache = Cache::try_from(args.cache_args)?;
+
+    let interpreter = if let Some(python) = args.python {
+        python
+    } else {
+        let platform = Platform::current()?;
+        let venv = PythonEnvironment::from_virtualenv(platform, &cache)?;
+        venv.python_executable().to_path_buf()
+    };
+
+    let files = uv_installer::compile_tree(
+        &fs_err::canonicalize(args.root)?,
+        &interpreter,
+        cache.root(),
+    )
+    .await?;
+    info!("Compiled {files} files");
+    Ok(())
+}