Whew this is a lot.
The user-facing changes are:
- `uv toolchain` to `uv python` e.g. `uv python find`, `uv python
install`, ...
- `UV_TOOLCHAIN_DIR` to` UV_PYTHON_INSTALL_DIR`
- `<UV_STATE_DIR>/toolchains` to `<UV_STATE_DIR>/python` (with
[automatic
migration](https://github.com/astral-sh/uv/pull/4735/files#r1663029330))
- User-facing messages no longer refer to toolchains, instead using
"Python", "Python versions" or "Python installations"
The internal changes are:
- `uv-toolchain` crate to `uv-python`
- `Toolchain` no longer referenced in type names
- Dropped unused `SystemPython` type (previously replaced)
- Clarified the type names for "managed Python installations"
- (more little things)
Restores the `PythonEnvironment::find` API which was removed a while
back in favor of `Toolchain::find`. As mentioned in #4416, I'm
attempting to separate the case where you want an active environment
from the case where you want an installed toolchain in order to create
environments.
I wanted to drop `EnvironmentPreference` from `Toolchain::find` and just
have us consistently consider (or not consider) virtual environments
when discovering toolchains for creating environments. Unfortunately
this caused a few things to break so I reverted that change and will
explore it separately. Because I was exploring that change, there are
some minor changes to the `Toolchain` API here.
Adds support for the toolchain discovery preferences outlined in
https://github.com/astral-sh/uv/issues/4198 but we don't expose this to
users yet, I'll do that next to make it easier to review.
I've made some refactors in the toolchain discovery implementation to
enable this behavior and move us towards clearer abstractions. There's
still remaining work here, but I'd prefer tackle things in follow-ups
instead of expanding this pull request. I plan on opening a couple
before merging this.
I'd like to shift the public toolchain API to focus on discovering
either an **environment** or a **toolchain**. The first would be used by
commands that operate on an environment, while the latter would be used
by commands that just need an interpreter to create environments. I
haven't changed this here, but some of the refactors are in preparation
for supporting this idea.
In brief:
- We now allow different ordering of installed toolchain discovery based
on a `ToolchainPreference` type. This is the type we will expose to
users.
- `SystemPython` was changed into an `EnvironmentPreference` which is
used to determine if we should prefer virtual or system Python
environments.
- We drop the whole `ToolchainSources` selection concept, it was
confusing and the error messages from it were awkward. Most of the
functionality is now captured by the preference enums, but you can't do
things like "only find a toolchain from the parent interpreter" as
easily anymore.
Extends #4120
Part of #2607
There should be no behavior changes here. Restructures the discovery API
to be focused on a toolchain first perspective in preparation for
exposing a `find_or_fetch` method for toolchains in
https://github.com/astral-sh/uv/pull/4138.
The architecture of uv does not necessarily match that of the python
interpreter (#2326). In cross compiling/testing scenarios the operating
system can also mismatch. To solve this, we move arch and os detection
to python, vendoring the relevant pypa/packaging code, preventing
mismatches between what the python interpreter was compiled for and what
uv was compiled for.
To make the scripts more manageable, they are now a directory in a
tempdir and we run them with `python -m` . I've simplified the
pypa/packaging code since we're still building the tags in rust. A
`Platform` is now instantiated by querying the python interpreter for
its platform. The pypa/packaging files are copied verbatim for easier
updates except a `lru_cache()` python 3.7 backport.
Error handling is done by a `"result": "success|error"` field that allow
passing error details to rust:
```console
$ uv venv --no-cache
× Can't use Python at `/home/konsti/projects/uv/.venv/bin/python3`
╰─▶ Unknown operation system `linux`
```
I've used the [maturin sysconfig
collection](855f6d2cb1/sysconfig)
as reference. I'm unsure how to test these changes across the wide
variety of platforms.
Fixes#2326
Add a `--compile` option to `pip install` and `pip sync`.
I chose to implement this as a separate pass over the entire venv. If we
wanted to compile during installation, we'd have to make sure that
writing is exclusive, to avoid concurrent processes writing broken
`.pyc` files. Additionally, this ensures that the entire site-packages
are bytecode compiled, even if there are packages that aren't from this
`uv` invocation. The disadvantage is that we do not update RECORD and
rely on this comment from [PEP 491](https://peps.python.org/pep-0491/):
> Uninstallers should be smart enough to remove .pyc even if it is not
mentioned in RECORD.
If this is a problem we can change it to run during installation and
write RECORD entries.
Internally, this is implemented as an async work-stealing subprocess
worker pool. The producer is a directory traversal over site-packages,
sending each `.py` file to a bounded async FIFO queue/channel. Each
worker has a long-running python process. It pops the queue to get a
single path (or exists if the channel is closed), then sends it to
stdin, waits until it's informed that the compilation is done through a
line on stdout, and repeat. This is fast, e.g. installing `jupyter
plotly` on Python 3.12 it processes 15876 files in 319ms with 32 threads
(vs. 3.8s with a single core). The python processes internally calls
`compileall.compile_file`, the same as pip.
Like pip, we ignore and silence all compilation errors
(https://github.com/astral-sh/uv/issues/1559). There is a 10s timeout to
handle the case when the workers got stuck. For the reviewers, please
check if i missed any spots where we could deadlock, this is the hardest
part of this PR.
I've added `uv-dev compile <dir>` and `uv-dev clear-compile <dir>`
commands, mainly for my own benchmarking. I don't want to expose them in
`uv`, they almost certainly not the correct workflow and we don't want
to support them.
Fixes#1788Closes#1559Closes#1928
