## Summary
This is causing some cyclic dependencies issues for me, because these
can be used in virtually _any_ crate (like `uv-install-wheel`), which
then means that all of `uv-configuration` becomes a dependency, etc. I
think this should be a leaf crate so that we can safely depend on it
anywhere.
This fixes a regression from 0.8.0 from
https://github.com/astral-sh/uv/pull/7934 and follows
https://github.com/astral-sh/uv/pull/15059
The regression is from [this
change](https://github.com/astral-sh/uv/pull/7934/files#diff-c7a660ac39628d5e12f388b0cacc7360affa3d7bb21191184d7ee78489675e83),
which was made because we'd otherwise (with the other changes in that
pull request) _filter out_ managed Python interpreters found in virtual
environments.
When `--system` is used we'll convert the default Python preference of
`managed` to `system` which avoids things like `uv pip install --system`
targeting a managed Python installation.
The basic test is
```
uv python install
uv pip install --system anyio
```
Prior to this change, we'd read a managed interpreter from our managed
installation directory and target that. After this change, without
#15059, we'd read a managed interpreter from the PATH and target that.
Both of those experiences are bad, because the managed interpreters are
marked as externally managed. After this change, with #15059, we
properly target the system interpreter.
Since we use `system` instead of `only-system`, if there is not a system
interpreter we'll still retain our existing behavior and use a managed
interpreter. This should limit breakage from the change. Given the
source of the regression, we could probably use `only-system` here. I
don't feel strongly. I think the main benefit of doing so would be that
we'd omit the check for managed installations in error messages when an
interpreter cannot be found?
We can't really add test coverage here because the test suite always has
externally managed interpreters :)
I think this would give us better hygiene than a global flag. It makes
it easier for users to opt-in to overlapping features, such as Python
upgrades and Python bin installations and to disable warnings for
preview mode without opting in to a bunch of other features. In general,
I want to reduce the burden for putting something under preview.
The `--preview` and `--no-preview` flags are retained as global
overrides. A new `--preview-features` option is added which accepts
comma separated features or can be passed multiple times, e.g.,
`--preview-features add-bounds,pylock`. There's a `UV_PREVIEW_FEATURES`
environment variable for that option (I'm not sure if we should overload
`UV_PREVIEW`, but could be convinced).
> NOTE: The PRs that were merged into this feature branch have all been
independently reviewed. But it's also useful to see all of the changes
in their final form. I've added comments to significant changes
throughout the PR to aid discussion.
This PR introduces transparent Python version upgrades to uv, allowing
for a smoother experience when upgrading to new patch versions.
Previously, upgrading Python patch versions required manual updates to
each virtual environment. Now, virtual environments can transparently
upgrade to newer patch versions.
Due to significant changes in how uv installs and executes managed
Python executables, this functionality is initially available behind a
`--preview` flag. Once an installation has been made upgradeable through
`--preview`, subsequent operations (like `uv venv -p 3.10` or patch
upgrades) will work without requiring the flag again. This is
accomplished by checking for the existence of a minor version symlink
directory (or junction on Windows).
### Features
* New `uv python upgrade` command to upgrade installed Python versions
to the latest available patch release:
```
# Upgrade specific minor version
uv python upgrade 3.12 --preview
# Upgrade all installed minor versions
uv python upgrade --preview
```
* Transparent upgrades also occur when installing newer patch versions:
```
uv python install 3.10.8 --preview
# Automatically upgrades existing 3.10 environments
uv python install 3.10.18
```
* Support for transparently upgradeable Python `bin` installations via
`--preview` flag
```
uv python install 3.13 --preview
# Automatically upgrades the `bin` installation if there is a newer patch version available
uv python upgrade 3.13 --preview
```
* Virtual environments can still be tied to a patch version if desired
(ignoring patch upgrades):
```
uv venv -p 3.10.8
```
### Implementation
Transparent upgrades are implemented using:
* Minor version symlink directories (Unix) or junctions (Windows)
* On Windows, trampolines simulate paths with junctions
* Symlink directory naming follows Python build standalone format: e.g.,
`cpython-3.10-macos-aarch64-none`
* Upgrades are scoped to the minor version key (as represented in the
naming format: implementation-minor version+variant-os-arch-libc)
* If the context does not provide a patch version request and the
interpreter is from a managed CPython installation, the `Interpreter`
used by `uv python run` will use the full symlink directory executable
path when available, enabling transparently upgradeable environments
created with the `venv` module (`uv run python -m venv`)
New types:
* `PythonMinorVersionLink`: in a sense, the core type for this PR, this
is a representation of a minor version symlink directory (or junction on
Windows) that points to the highest installed managed CPython patch
version for a minor version key.
* `PythonInstallationMinorVersionKey`: provides a view into a
`PythonInstallationKey` that excludes the patch and prerelease. This is
used for grouping installations by minor version key (e.g., to find the
highest available patch installation for that minor version key) and for
minor version directory naming.
### Compatibility
* Supports virtual environments created with:
* `uv venv`
* `uv run python -m venv` (using managed Python that was installed or
upgraded with `--preview`)
* Virtual environments created within these environments
* Existing virtual environments from before these changes continue to
work but aren't transparently upgradeable without being recreated
* Supports both standard Python (`python3.10`) and freethreaded Python
(`python3.10t`)
* Support for transparently upgrades is currently only available for
managed CPython installations
Closes#7287Closes#7325Closes#7892Closes#9031Closes#12977
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Surprisingly, we weren't locking during `uv sync` so far, so running `uv
sync` in parallel could cause errors in filesystem races.
I've also added locks to `uv add` and `uv remove` which concurrently
modify `pyproject.toml`. These locks only apply after we determined the
interpreter, so they don't actually prevent computing the same thing
twice when running `uv add` in parallel.
All other subcommands that I checked were already locking (with no claim
to exhaustiveness)
Fixes#12751
# Test Plan
I don't have CI-sized reproducer for this.
```toml
[project]
name = "debug"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
"boto3>=1.38.30",
"fastapi>=0.115.12",
"numba>=0.61.2",
"polars>=1.30.0",
"protobuf>=6.31.1",
"pyarrow>=20.0.0",
"pydantic>=2.11.5",
"requests>=2.32.3",
"urllib3>=2.4.0",
"scikit-learn>=1.6.1",
"jupyter>=1.1.1",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
```
```
rm -rf .venv && parallel -n0 "uv sync -q" ::: {1..10}
```
Rustfmt introduces a lot of formatting changes in the 2024 edition. To
not break everything all at once, we split out the set of formatting
changes compatible with both the 2021 and 2024 edition by first
formatting with the 2024 style, and then again with the currently used
2021 style.
Notable changes are the formatting of derive macro attributes and lines
with overly long strings and adding trailing semicolons after statements
consistently.
I think this regressed in https://github.com/astral-sh/uv/pull/13027 — I
misunderstood what versions could be represented in the `pyvenv.cfg` (I
assumed they _never_ included pre-release components).
Closes#13233
When working on #13025 I noticed this message was lacking versions,
which seems frustrating if you're debugging things.
I refactored the general `matches_interpreter` utilities that were added
in https://github.com/astral-sh/uv/pull/12884 into a more purpose-fit
function that returns an `Option` with the versions if there's a
mismatch.
It was possible that a virtual environment became out of sync with the
interpreter it pointed to (for example, if a symlink was changed to an
updated Python version). In such a case, `pyvenv.cfg` and
`activate_this.py` would no longer be correct. This PR detects when the
`version` (`venv` module) or `version_info` (uv and `virtualenv`) field
in `pyvenv.cfg` is out of sync with the interpreter. In such a case, uv
recreates the virtual environment.
Closes#12461
## Summary
We use a similar strategy to the ephemeral overlay: set
`include-system-site-packages` in the `pyvenv.cfg`, and clear it
whenever we access a new environment.
Closes https://github.com/astral-sh/uv/issues/11829.
## Test Plan
Difficult to test because we don't really have support for system
packages in our test infrastructure. But...
```
> uv venv --system-site-packages
> ['', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python313.zip', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13/lib-dynload', '/Users/crmarsh/.cache/uv/archive-v0/AhKcORkaCdbBl31VweRtG/lib/python3.13/site-packages', '/Users/crmarsh/workspace/uv/foo/.venv/lib/python3.13/site-packages', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13/site-packages']
```
```
> uv venv
> ['', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python313.zip', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13/lib-dynload', '/Users/crmarsh/.cache/uv/archive-v0/AhKcORkaCdbBl31VweRtG/lib/python3.13/site-packages', '/Users/crmarsh/workspace/uv/foo/.venv/lib/python3.13/site-packages']
```
## Summary
This is a follow-on to #11347 to use a stable directory for remote and
stdin scripts. The annoying piece here was figuring out what to use as
the cache key. For remote scripts, I'm using the URL; for stdin scripts,
there isn't any identifying information, so I'm just using a hash of the
metadata.
Closes https://github.com/astral-sh/uv/issues/11048
This brings the `PythonEnvironment::from_root` behavior in-line with the
rest of uv Python discovery behavior (and in-line with pip). It's not
clear why we were canonicalizing the path in the first place here.
As discussed in https://github.com/astral-sh/uv/issues/9423, it's
confusing that we do not allow `uv sync` just because the `.venv`
directory _exists_. This change matches `uv venv`.
This PR adds support for upgrading the build environment of tools with
the addition of a ```--python``` argument to ```uv upgrade```, as
specified in #7471.
Some things to note:
- I added support for individual packages — I didn't think there was a
good reason for ```--python``` to only apply to all packages
- Upgrading with ```--python``` also upgrades the package itself — I
think this is fair as if a user wants to _strictly_ switch the version
of Python being used to build a tool's environment they can use ```uv
install```. This behavior can of course be modified if others don't
agree!
Closes https://github.com/astral-sh/uv/issues/6297.
Closes https://github.com/astral-sh/uv/issues/7471.
As we support more complex Python discovery behaviors such as:
- #7431
- #7335
- #7300
`Any` is no longer accurate, we actually are looking for a reasonable
default Python version to use which may exclude the first one we find.
Separately, we need the idea of `Any` to improve behavior when listing
versions (e.g., #7286) where we do actually want to match _any_ Python
version. As a first step, we'll rename `Any` to `Default`. Then, we'll
introduce a new `Any` that actually behaves as we'd expect.
This is achieved by updating the `LockedFile::acquire` API to be async —
as in some cases we were attempting to acquire the lock synchronously,
i.e., without yielding, which blocked the runtime.
Closes https://github.com/astral-sh/uv/issues/6691 — I tested with the
reproduction there and a local release build and no longer reproduce the
deadlock with these changes.
Some additional context in the [internal Discord
thread](1278478941)
## Summary
Adds a `--relocatable` CLI arg to `uv venv`. This flag does two things:
* ensures that the associated activation scripts do not rely on a
hardcoded
absolute path to the virtual environment (to the extent possible; `.csh`
and
`.nu` left as-is)
* persists a `relocatable` flag in `pyvenv.cfg`.
The flag in `pyvenv.cfg` in turn instructs the wheel `Installer` to
create script
entrypoints in a relocatable way (use `exec` trick + `dirname $0` on
POSIX;
use relative path to `python[w].exe` on Windows).
Fixes: #3863
## Test Plan
* Relocatable console scripts covered as additional scenarios in
existing test cases.
* Integration testing of boilerplate generation in `venv`.
* Manual testing of `uv venv` with and without `--relocatable`
Whew this is a lot.
The user-facing changes are:
- `uv toolchain` to `uv python` e.g. `uv python find`, `uv python
install`, ...
- `UV_TOOLCHAIN_DIR` to` UV_PYTHON_INSTALL_DIR`
- `<UV_STATE_DIR>/toolchains` to `<UV_STATE_DIR>/python` (with
[automatic
migration](https://github.com/astral-sh/uv/pull/4735/files#r1663029330))
- User-facing messages no longer refer to toolchains, instead using
"Python", "Python versions" or "Python installations"
The internal changes are:
- `uv-toolchain` crate to `uv-python`
- `Toolchain` no longer referenced in type names
- Dropped unused `SystemPython` type (previously replaced)
- Clarified the type names for "managed Python installations"
- (more little things)
2024-07-03 07:44:29 -05:00
Renamed from crates/uv-toolchain/src/environment.rs (Browse further)
