I think this would give us better hygiene than a global flag. It makes
it easier for users to opt-in to overlapping features, such as Python
upgrades and Python bin installations and to disable warnings for
preview mode without opting in to a bunch of other features. In general,
I want to reduce the burden for putting something under preview.
The `--preview` and `--no-preview` flags are retained as global
overrides. A new `--preview-features` option is added which accepts
comma separated features or can be passed multiple times, e.g.,
`--preview-features add-bounds,pylock`. There's a `UV_PREVIEW_FEATURES`
environment variable for that option (I'm not sure if we should overload
`UV_PREVIEW`, but could be convinced).
Motivated by some code duplication highlighted in
https://github.com/astral-sh/uv/pull/14201, I noticed we weren't taking
advantage of the existing implementation for casting to a str here.
Unfortunately, we do need a special case for CPython still.
> NOTE: The PRs that were merged into this feature branch have all been
independently reviewed. But it's also useful to see all of the changes
in their final form. I've added comments to significant changes
throughout the PR to aid discussion.
This PR introduces transparent Python version upgrades to uv, allowing
for a smoother experience when upgrading to new patch versions.
Previously, upgrading Python patch versions required manual updates to
each virtual environment. Now, virtual environments can transparently
upgrade to newer patch versions.
Due to significant changes in how uv installs and executes managed
Python executables, this functionality is initially available behind a
`--preview` flag. Once an installation has been made upgradeable through
`--preview`, subsequent operations (like `uv venv -p 3.10` or patch
upgrades) will work without requiring the flag again. This is
accomplished by checking for the existence of a minor version symlink
directory (or junction on Windows).
### Features
* New `uv python upgrade` command to upgrade installed Python versions
to the latest available patch release:
```
# Upgrade specific minor version
uv python upgrade 3.12 --preview
# Upgrade all installed minor versions
uv python upgrade --preview
```
* Transparent upgrades also occur when installing newer patch versions:
```
uv python install 3.10.8 --preview
# Automatically upgrades existing 3.10 environments
uv python install 3.10.18
```
* Support for transparently upgradeable Python `bin` installations via
`--preview` flag
```
uv python install 3.13 --preview
# Automatically upgrades the `bin` installation if there is a newer patch version available
uv python upgrade 3.13 --preview
```
* Virtual environments can still be tied to a patch version if desired
(ignoring patch upgrades):
```
uv venv -p 3.10.8
```
### Implementation
Transparent upgrades are implemented using:
* Minor version symlink directories (Unix) or junctions (Windows)
* On Windows, trampolines simulate paths with junctions
* Symlink directory naming follows Python build standalone format: e.g.,
`cpython-3.10-macos-aarch64-none`
* Upgrades are scoped to the minor version key (as represented in the
naming format: implementation-minor version+variant-os-arch-libc)
* If the context does not provide a patch version request and the
interpreter is from a managed CPython installation, the `Interpreter`
used by `uv python run` will use the full symlink directory executable
path when available, enabling transparently upgradeable environments
created with the `venv` module (`uv run python -m venv`)
New types:
* `PythonMinorVersionLink`: in a sense, the core type for this PR, this
is a representation of a minor version symlink directory (or junction on
Windows) that points to the highest installed managed CPython patch
version for a minor version key.
* `PythonInstallationMinorVersionKey`: provides a view into a
`PythonInstallationKey` that excludes the patch and prerelease. This is
used for grouping installations by minor version key (e.g., to find the
highest available patch installation for that minor version key) and for
minor version directory naming.
### Compatibility
* Supports virtual environments created with:
* `uv venv`
* `uv run python -m venv` (using managed Python that was installed or
upgraded with `--preview`)
* Virtual environments created within these environments
* Existing virtual environments from before these changes continue to
work but aren't transparently upgradeable without being recreated
* Supports both standard Python (`python3.10`) and freethreaded Python
(`python3.10t`)
* Support for transparently upgrades is currently only available for
managed CPython installations
Closes#7287Closes#7325Closes#7892Closes#9031Closes#12977
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Previously you could get a confusing error message like this:
```
$ docker run ghcr.io/astral-sh/uv run python
error: Could not read ELF interpreter from any of the following paths: /bin/sh, /usr/bin/env, /bin/dash, /bin/ls
```
Now the error message is better:
```
error: Failed to discover managed Python installations
Caused by: Failed to determine the libc used on the current platform
Caused by: Failed to find any common binaries to determine libc from: /bin/sh, /usr/bin/env, /bin/dash, /bin/ls
```
See https://github.com/astral-sh/uv/issues/8635.
---------
Co-authored-by: konsti <konstin@mailbox.org>
Co-authored-by: Zanie Blue <contact@zanie.dev>
Rustfmt introduces a lot of formatting changes in the 2024 edition. To
not break everything all at once, we split out the set of formatting
changes compatible with both the 2021 and 2024 edition by first
formatting with the 2024 style, and then again with the currently used
2021 style.
Notable changes are the formatting of derive macro attributes and lines
with overly long strings and adding trailing semicolons after statements
consistently.
## Summary
This adds GraalPy download metadata so that `uv python install graalpy`
works. See https://github.com/astral-sh/uv/issues/13114
## Test Plan
The existing integration test was changed to test this functionality.
## Summary
In preview mode on windows, register und un-register the managed python build standalone installations in the Windows registry following PEP 514.
We write the values defined in the PEP plus the download URL and hash. We add an entry when installing a version, remove an entry when uninstalling and removing all values when uninstalling with `--all`. We update entries only by overwriting existing values, there is no "syncing" involved.
Since they are not official builds, pbs gets a prefix. `py -V:Astral/CPython3.13.1` works, `py -3.13` doesn't.
```
$ py --list-paths
-V:3.12 * C:\Users\Konsti\AppData\Local\Programs\Python\Python312\python.exe
-V:3.11.9 C:\Users\Konsti\.pyenv\pyenv-win\versions\3.11.9\python.exe
-V:3.11 C:\Users\micro\AppData\Local\Programs\Python\Python311\python.exe
-V:3.8 C:\Users\micro\AppData\Local\Programs\Python\Python38\python.exe
-V:Astral/CPython3.13.1 C:\Users\Konsti\AppData\Roaming\uv\data\python\cpython-3.13.1-windows-x86_64-none\python.exe
```
Registry errors are reported but not fatal, except for operations on the company key since it's not bound to any specific python interpreter.
On uninstallation, we prune registry entries that have no matching Python installation (i.e. broken entries).
The code uses the official `windows_registry` crate of the `winreg` crate.
Best reviewed commit-by-commit.
## Test Plan
We're reusing an existing system check to test different (un)installation scenarios.
## Summary
For example, `cargo run python install
cpython-3.12.8-linux-x86_64_v3-gnu` (on macOS) shouldn't attempt to
patch the dylib. At present, it leads to this warning:
```
warning: Failed to patch the install name of the dynamic library for /Users/crmarsh/.local/share/uv/python/cpython-3.12.8-linux-x86_64_v3-gnu/bin/python3.12. This may cause issues when building Python native extensions.
Underlying error: Failed to update the install name of the Python dynamic library located at `/Users/crmarsh/.local/share/uv/python/cpython-3.12.8-linux-x86_64_v3-gnu/lib/libpython3.12.dylib`
```
## Summary
Fixes#10598
## Test Plan
Looking for input here @zanieb. How/where would you include tests for
this?
More broadly: do we want a failure to perform the rename to be a hard
error? Or should it start out as a warning?
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
This PR reimplements
[`sysconfigpatcher`](https://github.com/bluss/sysconfigpatcher) in Rust
and applies it to our Python installations at install-time, ensuring
that the `sysconfig` data is more likely to be correct.
For now, we only rewrite prefixes (i.e., any path that starts with
`/install` gets rewritten to the correct absolute path for the current
machine).
Unlike `sysconfigpatcher`, this PR does not yet do any of the following:
- Patch `pkginfo` files.
- Change `clang` references to `cc`.
A few things that we should do as follow-ups, in my opinion:
1. Rewrite
[`AR`](c1ebf8ab92/src/sysconfigpatcher.py (L61)).
2. Remove `-isysroot`, which we already do for newer builds.
## Summary
This PR adds `--install-dir` argument for the following commands:
- `uv python install`
- `uv python uninstall`
The `UV_PYTHON_INSTALL_DIR` env variable can be used to set it
(previously it was also used internally).
Any more commands we would want to add this to?
## Test Plan
For now just manual test (works on my machine hehe)
```
❯ ./target/debug/uv python install --install-dir /tmp/pythons 3.8.12
Searching for Python versions matching: Python 3.8.12
Installed Python 3.8.12 in 4.31s
+ cpython-3.8.12-linux-x86_64-gnu
❯ /tmp/pythons/cpython-3.8.12-linux-x86_64-gnu/bin/python --help
usage: /tmp/pythons/cpython-3.8.12-linux-x86_64-gnu/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ...
```
Open to add some tests after the initial feedback.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Previously, we'd use the `--reinstall` flag to determine if we should
replace existing Python executables in the bin directory during an
install. There are a few problems with this:
- We replace executables we don't manage
- We can replace executables from other uv Python installations during
reinstall (surprising)
- We don't do the "right" thing when installing patch versions e.g.
installing `3.12.4` then `3.12.6` would fail without the reinstall flag
In `uv tool`, we have separate `--force` and `--reinstall` concepts.
Here we separate the flags (`--force` was previously just a
`--reinstall` alias) and add inspection of the existing executables to
inform a decision on replacement.
In brief, we will:
- Replace any executables with `--force`
- Replace executables for the same installation with `--reinstall`
- Replace executables for an older patch version by default
Incorporating #8637 into #8458
- Adds `python-managed` feature selection to Windows CI for `python
install` tests
- Adds trampoline sniffing utilities to `uv-trampoline-builder`
- Uses a trampoline to install Python executables into the `PATH` on
Windows
Updates `uv python install` to link `python3.x` in the executable
directory (i.e., `~/.local/bin`) to the the managed interpreter path.
Includes
- #8569
- #8571
Remaining work
- #8663
- #8650
- Add an opt-out setting and flag
- Update documentation
A temporary fix for https://github.com/astral-sh/uv/issues/8298 while we
wait for my slower upstream fix at
https://github.com/indygreg/python-build-standalone/pull/373
I think we'll want this machinery anyway to ensure that the various
executable names are available? Otherwise we need to special-case all
the `python` names in `uv run`?
We don't have unit test coverage of managed downloads, so I added an
[integration
test](3170395680)
similar to what we have for Linux.
## Summary
This PR declares and documents all environment variables that are used
in one way or another in `uv`, either internally, or externally, or
transitively under a common struct.
I think over time as uv has grown there's been many environment
variables introduced. Its harder to know which ones exists, which ones
are missing, what they're used for, or where are they used across the
code. The docs only documents a handful of them, for others you'd have
to dive into the code and inspect across crates to know which crates
they're used on or where they're relevant.
This PR is a starting attempt to unify them, make it easier to discover
which ones we have, and maybe unlock future posibilities in automating
generating documentation for them.
I think we can split out into multiple structs later to better organize,
but given the high influx of PR's and possibly new environment variables
introduced/re-used, it would be hard to try to organize them all now
into their proper namespaced struct while this is all happening given
merge conflicts and/or keeping up to date.
I don't think this has any impact on performance as they all should
still be inlined, although it may affect local build times on changes to
the environment vars as more crates would likely need a rebuild. Lastly,
some of them are declared but not used in the code, for example those in
`build.rs`. I left them declared because I still think it's useful to at
least have a reference.
Did I miss any? Are their initial docs cohesive?
Note, `uv-static` is a terrible name for a new crate, thoughts? Others
considered `uv-vars`, `uv-consts`.
## Test Plan
Existing tests
As we support more complex Python discovery behaviors such as:
- #7431
- #7335
- #7300
`Any` is no longer accurate, we actually are looking for a reasonable
default Python version to use which may exclude the first one we find.
Separately, we need the idea of `Any` to improve behavior when listing
versions (e.g., #7286) where we do actually want to match _any_ Python
version. As a first step, we'll rename `Any` to `Default`. Then, we'll
introduce a new `Any` that actually behaves as we'd expect.
This is achieved by updating the `LockedFile::acquire` API to be async —
as in some cases we were attempting to acquire the lock synchronously,
i.e., without yielding, which blocked the runtime.
Closes https://github.com/astral-sh/uv/issues/6691 — I tested with the
reproduction there and a local release build and no longer reproduce the
deadlock with these changes.
Some additional context in the [internal Discord
thread](1278478941)
As described in #4242, we're currently incorrectly downloading glibc
python-build-standalone on musl target, but we also can't fix this by
using musl python-build-standalone on musl targets since the musl builds
are effectively broken.
We reintroduce the libc detection previously removed in #2381, using it
to detect which libc is the current one before we have a python
interpreter. I changed the strategy a big to support an empty `PATH`
which we use in the tests.
For simplicity, i've decided to just filter out the musl
python-build-standalone archives from the list of available archive,
given this is temporary. This means we show the same error message as if
we don't have a build for the platform. We could also add a dedicated
error message for musl.
Fixes#4242
## Test Plan
Tested manually.
On my ubuntu host, python downloads continue to pass:
```
target/x86_64-unknown-linux-musl/debug/uv python install
```
On alpine, we fail:
```
$ docker run -it --rm -v .:/io alpine /io/target/x86_64-unknown-linux-musl/debug/uv python install
Searching for Python installations
error: No download found for request: cpython-any-linux-x86_64-musl
```
## Summary
Resolves#5139
`PythonInstallationKey` was sorted as a string, which caused `3.8` to
appear before `3.11`. This update changes the sorting of
`PythonInstallationKey` to be a descending order by version.
## Test Plan
```sh
$ cargo run -- python install 3.8 3.12
$ cargo run -- tool run -v python -V
DEBUG uv 0.2.25
warning: `uv tool run` is experimental and may change without warning.
DEBUG Searching for Python interpreter in managed installations, system path, or `py` launcher
DEBUG Searching for managed installations at `C:\Users\xx\AppData\Roaming\uv\data\python`
DEBUG Found managed Python `cpython-3.12.3-windows-x86_64-none`
DEBUG Found cpython 3.12.3 at `C:\Users\xx\AppData\Roaming\uv\data\python\cpython-3.12.3-windows-x86_64-none\install\python.exe` (managed installations)
DEBUG Using request timeout of 30s
DEBUG Using request timeout of 30s
DEBUG Acquired lock for `C:\Users\nigel\AppData\Roaming\uv\data\tools`
DEBUG Using existing environment for tool `httpx`: C:\Users\xx\AppData\Roaming\uv\data\tools\httpx
DEBUG Using existing tool `httpx`
DEBUG Running `httpx -v`
```
## Summary
Resolves#4834
## Test Plan
```sh
# 3.12.3 is a `install_only` archive
$ cargo run -- python install --preview --force 3.12.3
# 3.9.4 has only `full` archive
$ cargo run -- python install --preview --force 3.9.4
```
Whew this is a lot.
The user-facing changes are:
- `uv toolchain` to `uv python` e.g. `uv python find`, `uv python
install`, ...
- `UV_TOOLCHAIN_DIR` to` UV_PYTHON_INSTALL_DIR`
- `<UV_STATE_DIR>/toolchains` to `<UV_STATE_DIR>/python` (with
[automatic
migration](https://github.com/astral-sh/uv/pull/4735/files#r1663029330))
- User-facing messages no longer refer to toolchains, instead using
"Python", "Python versions" or "Python installations"
The internal changes are:
- `uv-toolchain` crate to `uv-python`
- `Toolchain` no longer referenced in type names
- Dropped unused `SystemPython` type (previously replaced)
- Clarified the type names for "managed Python installations"
- (more little things)
2024-07-03 07:44:29 -05:00
Renamed from crates/uv-toolchain/src/managed.rs (Browse further)
