Part of #11834
Currently, all Python installation are a streaming download-and-extract.
With this PR, we add the `UV_PYTHON_CACHE_DIR` variable. When set, the
installation is split into downloading the interpreter into
`UV_PYTHON_CACHE_DIR` and extracting it there from a second step. If the
archive is already present in `UV_PYTHON_CACHE_DIR`, we skip the
download.
The feature can be used to speed up tests and CI. Locally for me, `cargo
test -p uv -- python_install` goes from 43s to 7s (1,7s in release mode)
when setting `UV_PYTHON_CACHE_DIR`. It can also be used for offline
installation of Python interpreter, by copying the archives to a
directory in the offline machine, while the path rewriting is still
performed on the target machine on installation.
When working on #13025 I noticed this message was lacking versions,
which seems frustrating if you're debugging things.
I refactored the general `matches_interpreter` utilities that were added
in https://github.com/astral-sh/uv/pull/12884 into a more purpose-fit
function that returns an `Option` with the versions if there's a
mismatch.
## Summary
Before:
```console
$ uv python list py --managed-python
error: Interpreter discovery for `executable name `py`` requires `search path` but only only managed is allowed
```
After:
```console
$ uv python list py --managed-python
error: Interpreter discovery for `executable name `py`` requires `search path` but only `only managed` is allowed
```
Fixes#12914.
When `PythonDownloadRequest` does not have the `implementation` set, do
not set it to CPython when calling `fill`, otherwise only CPython
interpreters are shown when listing interpreters available for download,
with `uv python list`.
It was possible that a virtual environment became out of sync with the
interpreter it pointed to (for example, if a symlink was changed to an
updated Python version). In such a case, `pyvenv.cfg` and
`activate_this.py` would no longer be correct. This PR detects when the
`version` (`venv` module) or `version_info` (uv and `virtualenv`) field
in `pyvenv.cfg` is out of sync with the interpreter. In such a case, uv
recreates the virtual environment.
Closes#12461
## Summary
Add an option to overwrite the list of available Python downloads from a
local JSON file by using the environment variable
`UV_PYTHON_DOWNLOADS_JSON_URL`
as an experimental support for providing custom sources for Python
distribution binaries #8015
related #10203
I probably should make the JSON to be fetched from a remote URL instead
of a local file.
please let me know what you think and I will modify the code
accordingly.
## Test Plan
### normal run
```
root@75c66494ba8b:/# /code/target/release/uv python list
cpython-3.14.0a4+freethreaded-linux-x86_64-gnu <download available>
cpython-3.14.0a4-linux-x86_64-gnu <download available>
cpython-3.13.1+freethreaded-linux-x86_64-gnu <download available>
cpython-3.13.1-linux-x86_64-gnu <download available>
cpython-3.12.8-linux-x86_64-gnu <download available>
cpython-3.11.11-linux-x86_64-gnu <download available>
cpython-3.10.16-linux-x86_64-gnu <download available>
cpython-3.9.21-linux-x86_64-gnu <download available>
cpython-3.8.20-linux-x86_64-gnu <download available>
cpython-3.7.9-linux-x86_64-gnu <download available>
pypy-3.10.14-linux-x86_64-gnu <download available>
pypy-3.9.19-linux-x86_64-gnu <download available>
pypy-3.8.16-linux-x86_64-gnu <download available>
pypy-3.7.13-linux-x86_64-gnu <download available>
```
### empty JSON file
```sh
root@75c66494ba8b:/# export UV_PYTHON_DOWNLOADS_JSON_URL=/code/crates/uv-python/my-download-metadata.json
root@75c66494ba8b:/# cat $UV_PYTHON_DOWNLOADS_JSON_URL
{}
root@75c66494ba8b:/# /code/target/release/uv python list
root@75c66494ba8b:/#
```
### JSON file with valid version
```sh
root@75c66494ba8b:/# export UV_PYTHON_DOWNLOADS_JSON_URL=/code/crates/uv-python/my-download-metadata.json
root@75c66494ba8b:/# cat $UV_PYTHON_DOWNLOADS_JSON_URL
{
"cpython-3.11.9-linux-x86_64-gnu": {
"name": "cpython",
"arch": {
"family": "x86_64",
"variant": null
},
"os": "linux",
"libc": "gnu",
"major": 3,
"minor": 11,
"patch": 9,
"prerelease": "",
"url": "20240814/cpython-3.11.9%2B20240814-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz",
"sha256": "daa487c7e73005c4426ac393273117cf0e2dc4ab9b2eeda366e04cd00eea00c9",
"variant": null
}
}
root@75c66494ba8b:/# /code/target/release/uv python list
cpython-3.11.9-linux-x86_64-gnu <download available>
root@75c66494ba8b:/#
```
### Remote Path
```sh
root@75c66494ba8b:/# export UV_PYTHON_DOWNLOADS_JSON_URL=http://a.com/file.json
root@75c66494ba8b:/# /code/target/release/uv python list
error: Remote python downloads JSON is not yet supported, please use a local path (without `file://` prefix)
```
---------
Co-authored-by: Aria Desires <aria.desires@gmail.com>
In an attempt to avoid reporting shims as their resolved
`sys.executable` path, we report the queried executable path instead.
This seems more correct for this command, broadly? Interestingly, it
changes the reported paths for Homebrew Python
<img width="1430" alt="Screenshot 2025-04-02 at 11 05 18 AM"
src="https://github.com/user-attachments/assets/0e1600e8-fb07-40c7-a6d6-56eaeb4b9293"
/>
Closes#9979
## Summary
This ended up being more involved than expected. The gist is that we
setup all the packages we want to reinstall upfront (they're passed in
on the command-line); but at that point, we don't have names for all the
packages that the user has specified. (Consider, e.g., `uv pip install
.` -- we don't have a name for `.`, so we can't add it to the list of
`Reinstall` packages.)
Now, `Reinstall` also accepts paths, so we can augment `Reinstall` based
on the user-provided paths.
Closes#12038.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
There were no GraalPy binary wheels were available when uv support was
added, and thus the abi tag was never tested against actual packages.
Now that GraalPy publishes binary wheels to
https://www.graalvm.org/python/wheels/ we noticed the abi tag was
incorrect and the version info incorrectly determined.
## Test Plan
I tested manually:
```
> target/debug/uv venv --python graalpy testvenv
Using GraalPy 3.11.7 interpreter at: /home/tim/.pyenv/versions/graalpy-24.1.1/bin/graalpy
Creating virtual environment at: testvenv
Activate with: source testvenv/bin/activate
> cat <<EOF> uv.toml
> [[index]]
> url = "https://www.graalvm.org/python/wheels/"
> EOF
> target/debug/uv -v pip install psutil
warning: Found both a `uv.toml` file and a `[tool.uv]` section in an adjacent `pyproject.toml`. The `[tool.uv]` section will be ignored in favor of the `uv.toml` file.
DEBUG uv 0.6.6+3 (be8725553 2025-03-13)
DEBUG Searching for default Python interpreter in virtual environments
DEBUG Found `graalpy-3.11.7-linux-x86_64-gnu` at `/home/tim/dev/uv/.venv/bin/python3` (virtual environment)
DEBUG Using Python 3.11.7 environment at: .venv
DEBUG Acquired lock for `.venv`
DEBUG At least one requirement is not satisfied: psutil
DEBUG Using request timeout of 30s
DEBUG Solving with installed Python version: 3.11.7
DEBUG Solving with target Python version: >=3.11.7
DEBUG Adding direct dependency: psutil*
DEBUG Found fresh response for: https://www.graalvm.org/python/wheels/psutil/
DEBUG Searching for a compatible version of psutil (*)
DEBUG Selecting: psutil==5.9.8 [compatible] (psutil-5.9.8-graalpy311-graalpy241_311_native-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_28_x86_64.whl)
DEBUG No cache entry for: https://gds.oracle.com/download/graalpy-wheels/psutil-5.9.8-graalpy311-graalpy241_311_native-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_28_x86_64.whl
DEBUG Tried 1 versions: psutil 1
DEBUG marker environment resolution took 0.968s
Resolved 1 package in 971ms
DEBUG Identified uncached distribution: psutil==5.9.8
DEBUG No cache entry for: https://gds.oracle.com/download/graalpy-wheels/psutil-5.9.8-graalpy311-graalpy241_311_native-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_28_x86_64.whl
Prepared 1 package in 268ms
Installed 1 package in 28ms
+ psutil==5.9.8
DEBUG Released lock at `/home/tim/dev/uv/.venv/.lock`
```
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
These changes add support for
```
uv python pin 3.12 --global
```
This adds the specified version to a `.python-version` file in the
user-level config directory. uv will now use the user-level version as a
fallback if no version is found in the project directory or its
ancestors.
Closes#4972
Following the upstream release and #12120, removes gating preventing
installation of the managed musl Python versions.
Of note
- The filtering of musl Python distributions has moved from the Rust
runtime to the metadata fetcher
- The filtering is now conditional on the PBS release date, removing all
old static musl distributions
- We could support the `+static` musl downloads in the future; right
now, they are deprioritized when selecting a variant
- I added test to CI which uses Alpine and installs numpy
## Summary
If we're looking at (e.g.) `python3.12`, and we have a `requires-python:
">=3.12.7, <3.13"`, then checking if the range includes `3.12` will
return `false`. Instead, we need to look at the lower- and upper-bound
major-minors of the `requires-python`.
Closes https://github.com/astral-sh/uv/issues/11825.
We prepend the interpreter discovery in a temporary path to `sys.path`,
which we have to strip to avoid the `sys.path` value containing a
then-deleted temp dir.
Fixes#11217
By default, a 64-bit uv does not see a 32-bit global (HKLM) installation
of Python in the registry
(https://github.com/astral-sh/uv/issues/11217). To work around this, we
manually request both 32-bit and 64-bit access using registry access
flags (https://peps.python.org/pep-0514/#sample-code). The flags have no
effect on 32-bit (https://stackoverflow.com/a/12796797/3549270).
This effect is that there is an asymmetry between discovery modes: For
the registry-based discovery using PEP 514, we discover both 32-bit and
64-bit Pythons, while for managed installations, we are stricter and
only discover those matching in bit-ness.
I tested this manually with an additional 32-bit installation of CPython
on a 64-bit machine and windows with 32-bit and 64-bit (x86_64 and i686)
builds of uv.
Three edition 2021 compatible sets of changes in preparation for the
edition 2025 split out from #11724.
In edition 2025, `gen` is a keyword, so we escape it as `r#gen`. `ref`
and `ref mut` are not allowed anymore for `&T` and `&mut T`, so we
remove them. `cargo fmt` now formats inside of macros, which the 2021
formatter doesn't undo.
## Summary
In. https://github.com/astral-sh/uv/issues/11857, we had a case of a
user that was seeing incorrect resolution results after upgrading to a
newer version of macOS, since we retained cache information about the
interpreter. This PR adds the OS name and version to the cache key for
the interpreter. This seems to be extremely cheap, and it's nice to make
this robust so that users don't run into the same confusion in the
future.
Closes https://github.com/astral-sh/uv/issues/11857.
## Summary
We use a similar strategy to the ephemeral overlay: set
`include-system-site-packages` in the `pyvenv.cfg`, and clear it
whenever we access a new environment.
Closes https://github.com/astral-sh/uv/issues/11829.
## Test Plan
Difficult to test because we don't really have support for system
packages in our test infrastructure. But...
```
> uv venv --system-site-packages
> ['', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python313.zip', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13/lib-dynload', '/Users/crmarsh/.cache/uv/archive-v0/AhKcORkaCdbBl31VweRtG/lib/python3.13/site-packages', '/Users/crmarsh/workspace/uv/foo/.venv/lib/python3.13/site-packages', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13/site-packages']
```
```
> uv venv
> ['', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python313.zip', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13', '/Users/crmarsh/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13/lib-dynload', '/Users/crmarsh/.cache/uv/archive-v0/AhKcORkaCdbBl31VweRtG/lib/python3.13/site-packages', '/Users/crmarsh/workspace/uv/foo/.venv/lib/python3.13/site-packages']
```
## Summary
This is the pattern I see in a variety of crates, and I believe this is
preferred if you don't _need_ an owned `String`, since you can avoid the
allocation. This could be pretty impactful for us?
## Summary
* Upgrade the rust toolchain to 1.85.0. This does not increase the MSRV.
* Update windows trampoline to 1.86 nightly beta (previously in 1.85
nightly beta).
## Test Plan
Existing tests
Revert #11601 for now
We run Python interpreter discovery with `-I` (#2500) which means these
environments variables are ignored when determining `sys.path`. Unless
we decide to remove the `-I` flag from the `sys.path` query, we
shouldn't release these changes to interpreter discovery caching.
I noticed that the latest two `sync-python-releases` jobs failed due to
`httpx.RemoteProtocolError: peer closed connection without sending
complete message body (incomplete chunked read)`.
For the current python-build-standalone release, each request page
(defaulting to 30 items per page) takes about 20 seconds and loads
around 32MB of data. This extensive data load might be causing the
request to frequently fail.
In this PR, I reduced number of items per page to 10 and added
`Accept-Encoding: gzip, deflate` to the request header. Now, it takes
about 6 seconds to load, and the compressed response size has been
reduced to 534KB. I hope this would addresses the request failure.
We want to use `sys.path` for package discovery (#2500, #9849). For
that, we need to know the correct value of `sys.path`. `sys.path` is a
runtime-changeable value, which gets influenced from a lot of different
sources: Environment variables, CLI arguments, `.pth` files with
scripting, `sys.path.append()` at runtime, a distributor patching
Python, etc. We cannot capture them all accurately, especially since
it's possible to change `sys.path` mid-execution. Instead, we do a best
effort attempt at matching the user's expectation.
The assumption is that package installation generally happens in venv
site-packages, system/user site-packages (including pypy shipping
packages with std), and `PYTHONPATH`. Specifically, we reuse
`PYTHONPATH` as dedicated way for users to tell uv to include specific
directories in package discovery.
A common way to influence `sys.path` that is not using venvs is setting
`PYTHONPATH`. To support this we're capturing `PYTHONPATH` as part of
the cache invalidation, i.e. we refresh the interpreter metadata if it
changed. For completeness, we're also capturing other environment
variables documented as influencing `sys.path` or other fields in the
interpreter info.
This PR does not include reading registry values for `sys.path`
additions on Windows as documented in
https://docs.python.org/3.11/using/windows.html#finding-modules. It
notably also does not include parsing of python CLI arguments, we only
consider their environment variable versions for package installation
and listing. We could try parsing CLI flags in `uv run python`, but we'd
still miss them when Python is launched indirectly through a script, and
it's more consistent to only consider uv's own arguments and environment
variables, similar to uv's behavior in other places.
Closes https://github.com/astral-sh/uv/issues/11288
I tested the reproduction there manually.
I'm a little uncertain about this behavior, it's not true to the spirit
of `--python <dir>` selecting a target environment but this method is
only used to see if an existing environment matches for the purpose of
invalidation in projects and tools where I think we always force a
separate environment anyway?
## Summary
This is a follow-on to #11347 to use a stable directory for remote and
stdin scripts. The annoying piece here was figuring out what to use as
the cache key. For remote scripts, I'm using the URL; for stdin scripts,
there isn't any identifying information, so I'm just using a hash of the
metadata.
A user reported a homebrew Python that would raise an exception in the
interpreter probing script because `platform.mac_ver()` returned `('',
('', '', ''), '')` on his installation due to
https://github.com/Homebrew/homebrew-core/issues/206778
This is easy enough to catch and show a proper error message instead of
the Python backtrace.
Previously, we patched pkg-config .pc files to have the absolute path to
the directory where we unpack a python-build-standalone release. As
discussed in #11028, we can use ${pcfiledir} in a .pc file to indicate
paths relative to the location of the file itself.
This change was implemented in astral-sh/python-build-standalone#507, so
for newer python-build-standalone releases, we don't need to do any
patching. Optimize this case by only modifying the .pc file if an actual
change is needed (which might be helpful down the line with hard links
or something). For older releases, change uv's patch to match what
python-build-standalone now does.
Includes https://pypy.org/posts/2025/02/pypy-v7318-release.html
These are labeled as betas in the post but not anywhere obvious to me?
I'm not sure we need to portray this to users.
Co-authored-by: zanieb <2586601+zanieb@users.noreply.github.com>
