We were checking whether a path was an executable in a virtual
environment or the base directory of a virtual environment in multiple
places in the codebase. This PR consolidates this logic into one place.
Closes#13947.
> NOTE: The PRs that were merged into this feature branch have all been
independently reviewed. But it's also useful to see all of the changes
in their final form. I've added comments to significant changes
throughout the PR to aid discussion.
This PR introduces transparent Python version upgrades to uv, allowing
for a smoother experience when upgrading to new patch versions.
Previously, upgrading Python patch versions required manual updates to
each virtual environment. Now, virtual environments can transparently
upgrade to newer patch versions.
Due to significant changes in how uv installs and executes managed
Python executables, this functionality is initially available behind a
`--preview` flag. Once an installation has been made upgradeable through
`--preview`, subsequent operations (like `uv venv -p 3.10` or patch
upgrades) will work without requiring the flag again. This is
accomplished by checking for the existence of a minor version symlink
directory (or junction on Windows).
### Features
* New `uv python upgrade` command to upgrade installed Python versions
to the latest available patch release:
```
# Upgrade specific minor version
uv python upgrade 3.12 --preview
# Upgrade all installed minor versions
uv python upgrade --preview
```
* Transparent upgrades also occur when installing newer patch versions:
```
uv python install 3.10.8 --preview
# Automatically upgrades existing 3.10 environments
uv python install 3.10.18
```
* Support for transparently upgradeable Python `bin` installations via
`--preview` flag
```
uv python install 3.13 --preview
# Automatically upgrades the `bin` installation if there is a newer patch version available
uv python upgrade 3.13 --preview
```
* Virtual environments can still be tied to a patch version if desired
(ignoring patch upgrades):
```
uv venv -p 3.10.8
```
### Implementation
Transparent upgrades are implemented using:
* Minor version symlink directories (Unix) or junctions (Windows)
* On Windows, trampolines simulate paths with junctions
* Symlink directory naming follows Python build standalone format: e.g.,
`cpython-3.10-macos-aarch64-none`
* Upgrades are scoped to the minor version key (as represented in the
naming format: implementation-minor version+variant-os-arch-libc)
* If the context does not provide a patch version request and the
interpreter is from a managed CPython installation, the `Interpreter`
used by `uv python run` will use the full symlink directory executable
path when available, enabling transparently upgradeable environments
created with the `venv` module (`uv run python -m venv`)
New types:
* `PythonMinorVersionLink`: in a sense, the core type for this PR, this
is a representation of a minor version symlink directory (or junction on
Windows) that points to the highest installed managed CPython patch
version for a minor version key.
* `PythonInstallationMinorVersionKey`: provides a view into a
`PythonInstallationKey` that excludes the patch and prerelease. This is
used for grouping installations by minor version key (e.g., to find the
highest available patch installation for that minor version key) and for
minor version directory naming.
### Compatibility
* Supports virtual environments created with:
* `uv venv`
* `uv run python -m venv` (using managed Python that was installed or
upgraded with `--preview`)
* Virtual environments created within these environments
* Existing virtual environments from before these changes continue to
work but aren't transparently upgradeable without being recreated
* Supports both standard Python (`python3.10`) and freethreaded Python
(`python3.10t`)
* Support for transparently upgrades is currently only available for
managed CPython installations
Closes#7287Closes#7325Closes#7892Closes#9031Closes#12977
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
When working on support for reading global Python pins in tool
operations, I noticed that we weren't using the canonicalized Python
request in receipts — we were using the raw string provided by the user.
Since we'll need to compare these values, we should be using the
canonicalized string.
The `Tool` and `ToolReceipt` types have been updated to hold a
`PythonRequest` instead of a `String`, and `Serialize` was implemented
for `PythonRequest` so canonicalization can happen at the edge instead
of being the caller's responsibility.
In the case where we have platform information in a Python request, we
should filter managed Python distributions by it prior to querying them.
Closes https://github.com/astral-sh/uv/issues/13935
---------
Co-authored-by: Aria Desires <aria.desires@gmail.com>
Previously if you wanted to run e.g. PyPy via `uvx`, you had to spell it
like `uvx -p pypy python`. Now we reuse some of the
`PythonRequest::parse` machinery to handle the executable, so all of the
following examples work:
- `uvx python3.8`
- `uvx 'python>3.7,<3.9'`
- `uvx --from python3.8 python` (or e.g. `bash`)
- `uvx pypy38`
- `uvx graalpy@38`
The `python` (and on Windows only, `pythonw`) special cases are
retained, which normally aren't allowed values of `-p`/`--python`.
Closes https://github.com/astral-sh/uv/issues/13536.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Co-authored-by: konsti <konstin@mailbox.org>
In https://github.com/astral-sh/uv/pull/13721#issuecomment-2920530601 I
presumed that all the installation problems in
https://github.com/astral-sh/uv/pull/13722 were solved by
https://github.com/astral-sh/uv/pull/13709 but they were not because we
don't differentiate between implicit and explicit architecture requests
so a request for `aarch64` is considered satisfied by an existing
`x86-64` installation even if the user explicitly requested that
architecture.
Now, we track if it was explicit or implicit, requiring an exact match
in the former case, and a `supports` in the latter.
We considered doing this for other items in the request, like the
operating system but we don't have a `supports()` concept there. It
might make sense for libc in the future.
Previously you could get a confusing error message like this:
```
$ docker run ghcr.io/astral-sh/uv run python
error: Could not read ELF interpreter from any of the following paths: /bin/sh, /usr/bin/env, /bin/dash, /bin/ls
```
Now the error message is better:
```
error: Failed to discover managed Python installations
Caused by: Failed to determine the libc used on the current platform
Caused by: Failed to find any common binaries to determine libc from: /bin/sh, /usr/bin/env, /bin/dash, /bin/ls
```
See https://github.com/astral-sh/uv/issues/8635.
---------
Co-authored-by: konsti <konstin@mailbox.org>
Co-authored-by: Zanie Blue <contact@zanie.dev>
e.g., these are misleading cruft in the error message at
https://github.com/astral-sh/uv/pull/12168#discussion_r2078204601
```
❯ uv python find /foo/bar
error: No interpreter found for path `/foo/bar` in virtual environments, managed installations, or search path
❯ cargo run -q -- python find /foo/bar
error: No interpreter found at path `/foo/bar`
```
When removing a Python interpreter underneath an existing venv, uv
currently shows a not found error:
```
error: Failed to inspect Python interpreter from active virtual environment at `.venv/bin/python3`
Caused by: Python interpreter not found at `/home/konsti/projects/uv/.venv/bin/python3`
```
This is unintuitive, as the file for the Python interpreter does exist,
it is a broken symlink that needs to be replaced with `uv venv`.
I've been encountering those occasionally, and I expect users that
switch between versions a lot will, too, especially when they also use
pyenv or a similar Python manager.
The new error hints at this solution:
```
error: Failed to inspect Python interpreter from active virtual environment at `.venv/bin/python3`
Caused by: Broken symlink at `.venv/bin/python3`, was the underlying Python interpreter removed?
hint: To recreate the virtual environment, run `uv venv`
```
We have test coverage for this elsewhere, but managed Python versions
are a distinct case because we know the _full_ version before querying
the interpreter (whereas, when we find them on the `PATH`, we usually
only know `X.y` from the file name).
This pre-filter logic now matches our subsequent logic at
060be9cef1/crates/uv-python/src/discovery.rs (L2146-L2149)060be9cef1
shows the snapshot change.
## Summary
Before:
```console
$ uv python list py --managed-python
error: Interpreter discovery for `executable name `py`` requires `search path` but only only managed is allowed
```
After:
```console
$ uv python list py --managed-python
error: Interpreter discovery for `executable name `py`` requires `search path` but only `only managed` is allowed
```
## Summary
If we're looking at (e.g.) `python3.12`, and we have a `requires-python:
">=3.12.7, <3.13"`, then checking if the range includes `3.12` will
return `false`. Instead, we need to look at the lower- and upper-bound
major-minors of the `requires-python`.
Closes https://github.com/astral-sh/uv/issues/11825.
## Summary
* Upgrade the rust toolchain to 1.85.0. This does not increase the MSRV.
* Update windows trampoline to 1.86 nightly beta (previously in 1.85
nightly beta).
## Test Plan
Existing tests
Closes https://github.com/astral-sh/uv/issues/11288
I tested the reproduction there manually.
I'm a little uncertain about this behavior, it's not true to the spirit
of `--python <dir>` selecting a target environment but this method is
only used to see if an existing environment matches for the purpose of
invalidation in projects and tools where I think we always force a
separate environment anyway?
Closes https://github.com/astral-sh/uv/issues/11214
Special-cases the first Python executable we find on the `PATH`,
allowing it to be considered during searches for virtual environments.
For some context, there are two stages to Python interpreter discovery
1. We find possible Python executables in various sources
2. We query the executables to determine canonical metadata about the
interpreter
We can't really be "sure" if an executable is a complaint virtual
environment during (1), we need to query the interpreter first. This
means that if you're only allowed to installed into virtual
environments, we'll query every interpreter on your PATH. This is not
performant, and causes confusion for users. Notably, I recently improved
error messaging when we can't find any valid interpreters, by showing
the error message we encounter while querying an interpreter (if any).
However, this is problematic when there's an error for an interpreter
that is not relevant to your search. In
https://github.com/astral-sh/uv/pull/11143, I added filtering to avoid
querying additional interpreters, but that regressed some user
experiences where they were relying on us finding implicitly active
virtual environments via the PATH.
## Summary
In preview mode on windows, register und un-register the managed python build standalone installations in the Windows registry following PEP 514.
We write the values defined in the PEP plus the download URL and hash. We add an entry when installing a version, remove an entry when uninstalling and removing all values when uninstalling with `--all`. We update entries only by overwriting existing values, there is no "syncing" involved.
Since they are not official builds, pbs gets a prefix. `py -V:Astral/CPython3.13.1` works, `py -3.13` doesn't.
```
$ py --list-paths
-V:3.12 * C:\Users\Konsti\AppData\Local\Programs\Python\Python312\python.exe
-V:3.11.9 C:\Users\Konsti\.pyenv\pyenv-win\versions\3.11.9\python.exe
-V:3.11 C:\Users\micro\AppData\Local\Programs\Python\Python311\python.exe
-V:3.8 C:\Users\micro\AppData\Local\Programs\Python\Python38\python.exe
-V:Astral/CPython3.13.1 C:\Users\Konsti\AppData\Roaming\uv\data\python\cpython-3.13.1-windows-x86_64-none\python.exe
```
Registry errors are reported but not fatal, except for operations on the company key since it's not bound to any specific python interpreter.
On uninstallation, we prune registry entries that have no matching Python installation (i.e. broken entries).
The code uses the official `windows_registry` crate of the `winreg` crate.
Best reviewed commit-by-commit.
## Test Plan
We're reusing an existing system check to test different (un)installation scenarios.
Previously, these errors would only be visible in the debug logs as
"Skipping bad interpreter ..." which can lead us to making some
ridiculous claims like "There is no virtual environment" or "Python is
not installed" when really we just failed to query the interpreter for
some reason.
We show the first error, sort of arbitrarily — but I think it matches
user expectation, i.e., this would be the first Python on your PATH.
Related to #10713
See https://github.com/astral-sh/uv/issues/4204 for motivation
This doesn't really reach the user experience I'd expect — i.e., we end
up saying a virtual environment "does not exist" which is a little
silly. However, I think improving the error messaging on interpreter
queries in general should be solved separately. I did one small
"general" change in
89e11d0222
— otherwise we don't show the message at all.
---------
Co-authored-by: konsti <konstin@mailbox.org>
## Summary
This PR reimplements
[`sysconfigpatcher`](https://github.com/bluss/sysconfigpatcher) in Rust
and applies it to our Python installations at install-time, ensuring
that the `sysconfig` data is more likely to be correct.
For now, we only rewrite prefixes (i.e., any path that starts with
`/install` gets rewritten to the correct absolute path for the current
machine).
Unlike `sysconfigpatcher`, this PR does not yet do any of the following:
- Patch `pkginfo` files.
- Change `clang` references to `cc`.
A few things that we should do as follow-ups, in my opinion:
1. Rewrite
[`AR`](c1ebf8ab92/src/sysconfigpatcher.py (L61)).
2. Remove `-isysroot`, which we already do for newer builds.
## Summary
Fix#8075.
Invalid discovered environments in the working directory should be
filtered out.
## Test Plan
- Test python_find
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
This PR adds `--install-dir` argument for the following commands:
- `uv python install`
- `uv python uninstall`
The `UV_PYTHON_INSTALL_DIR` env variable can be used to set it
(previously it was also used internally).
Any more commands we would want to add this to?
## Test Plan
For now just manual test (works on my machine hehe)
```
❯ ./target/debug/uv python install --install-dir /tmp/pythons 3.8.12
Searching for Python versions matching: Python 3.8.12
Installed Python 3.8.12 in 4.31s
+ cpython-3.8.12-linux-x86_64-gnu
❯ /tmp/pythons/cpython-3.8.12-linux-x86_64-gnu/bin/python --help
usage: /tmp/pythons/cpython-3.8.12-linux-x86_64-gnu/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ...
```
Open to add some tests after the initial feedback.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
I'm not sure why this hasn't come up before... But it looks like this
method is only looking at `python.exe` and `python3.exe`? From the user
screenshots, the `python3.12.exe` and `python3.13.exe` are also present,
though.
Closes https://github.com/astral-sh/uv/issues/9667.
The `SysVersion` registry entry may or may not include the patch
version, so if we encounter a registry entry without a patch version, we
must not assume that the patch version is 0.
```
Name Property
---- --------
3.9 DisplayName : Python 3.9 (64-bit)
SupportUrl : https://www.python.org/
Version : 3.9.13
SysVersion : 3.9
SysArchitecture : 64bit
Hive: HKEY_CURRENT_USER\Software\Python\PythonCore\3.9
```
Confirmed the fix manually.
Fixes#9668
## Summary
A lot of good new lints, and most importantly, error stabilizations. I
tried to find a few usages of the new stabilizations, but I'm sure there
are more.
IIUC, this _does_ require bumping our MSRV.
## Summary
These were moved as part of a broader refactor to create a single
integration test module. That "single integration test module" did
indeed have a big impact on compile times, which is great! But we aren't
seeing any benefit from moving these tests into their own files (despite
the claim in [this blog
post](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html),
I see the same compilation pattern regardless of where the tests are
located). Plus, we don't have many of these, and same-file tests is such
a strong Rust convention.
This restores behavior previously removed in
https://github.com/astral-sh/uv/pull/7649.
I thought it'd be clearer (and simpler) to have a consistent Python
executable name ordering. However, we've seen some cases where this can
be surprising and, in combination with #8481, can result in incorrect
behavior. For example, see https://github.com/astral-sh/uv/issues/9046
where we prefer `python3` over `python3.12` in the same directory even
though `python3.12` was requested. While `python3` and `python3.12` both
point to valid Python 3.12 environments there, the expectation is that
when `python3.12` is requested that the `python3.12` executable is
preferred. This expectation may be less obvious if the user requests
`python@3.12`, but uv does not distinguish between these request forms.
Similarly, this may be surprising as by default uv prefers `python` over
`python3` but when requesting `python3.12` the preference will be
swapped.
Implement a full working version of local version semantics. The (AFAIA)
major move towards this was implemented in #2430. This added support
such that the version specifier `torch==2.1.0+cpu` would install
`torch@2.1.0+cpu` and consider `torch@2.1.0+cpu` a valid way to satisfy
the requirement `torch==2.1.0` in further dependency resolution.
In this feature, we more fully support local version semantics. Namely,
we now allow `torch==2.1.0` to install `torch@2.1.0+cpu` regardless of
whether `torch@2.1.0` (no local tag) actually exists.
We do this by adding an internal-only `Max` value to local versions that
compare greater to all other local versions. Then we can translate
`torch==2.1.0` into bounds: greater than 2.1.0 with no local tag and
less than 2.1.0 with the `Max` local tag.
Depends on https://github.com/astral-sh/packse/pull/227.
e.g.
```
❯ echo "anyio" | cargo run -q -- pip compile - -v
DEBUG uv 0.4.30 (107ab3d71 2024-11-07)
DEBUG Starting Python discovery for a default Python
DEBUG Looking for exact match for request a default Python
DEBUG Searching for default Python interpreter in virtual environments, managed installations, or search path
DEBUG Found `cpython-3.12.7-macos-aarch64-none` at `/Users/zb/workspace/uv/.venv/bin/python3` (virtual environment)
```
```
❯ cargo run -q -- pip install anyio -v
DEBUG uv 0.4.30 (107ab3d71 2024-11-07)
DEBUG Searching for default Python interpreter in virtual environments
DEBUG Found `cpython-3.12.7-macos-aarch64-none` at `/Users/zb/workspace/uv/.venv/bin/python3` (virtual environment)
```
vs
```
❯ uv pip install anyio -v
DEBUG uv 0.4.30 (61ed2a236 2024-11-04)
DEBUG Searching for default Python interpreter in system path
DEBUG Found `cpython-3.12.7-macos-aarch64-none` at `/Users/zb/workspace/uv/.venv/bin/python3` (virtual environment)
```
```
❯ echo "anyio" | uv pip compile - -v
DEBUG uv 0.4.30 (61ed2a236 2024-11-04)
DEBUG Starting Python discovery for a default Python
DEBUG Looking for exact match for request a default Python
DEBUG Searching for default Python interpreter in managed installations or system path
DEBUG Found `cpython-3.12.7-macos-aarch64-none` at `/Users/zb/workspace/uv/.venv/bin/python3` (virtual environment)
```
