While reviewing https://github.com/astral-sh/uv/pull/14107, @oconnor663
pointed out a bug where we allow `uv python pin --rm` to delete the
global pin without the `--global` flag. I think that shouldn't be
allowed? I'm not 100% certain though.
Clap does not perform global validation, so flag that are declared as
overriding can be set at the same time:
https://github.com/clap-rs/clap/issues/6049. This would previously cause
a panic. We work around this by choosing the yes-value always and
writing a warning.
An alternative would be erroring when both are set, but it's unclear to
me if this may break things we want to support. (`UV_OFFLINE=1 cargo run
-q pip --no-offline install tqdm --no-cache` is already banned).
Fixes https://github.com/astral-sh/uv/pull/14299
**Test Plan**
```
$ cargo run -q pip --offline install --no-offline tqdm --no-cache
warning: Boolean flags on different levels are not correctly supported (https://github.com/clap-rs/clap/issues/6049)
× No solution found when resolving dependencies:
╰─▶ Because tqdm was not found in the cache and you require tqdm, we can conclude that your requirements are unsatisfiable.
hint: Packages were unavailable because the network was disabled. When the network is disabled, registry packages may only be read from the cache.
```
## Summary
The basic idea here is that we can (should) reuse a build environment
across resolution (`prepare_metadata_for_build_wheel`) and installation.
This also happens to solve the build-PyTorch-from-source problem, since
we use a consistent build environment between the invocations.
Since `SourceDistributionBuilder` is stateless, we instead store the
builds on `BuildContext`, and we key them by various properties: the
underlying interpreter, the configuration settings, etc. This just
ensures that if we build the same package twice within a process, we
don't accidentally reuse an incompatible build (virtual) environment.
(Note that still drop build environments at the end of the command, and
don't attempt to reuse them across processes.)
Closes#14269.
If/when we see https://github.com/astral-sh/uv/issues/14171 again, this
should clarify whether our retry logic was skipped (i.e. a transient
error wasn't correctly identified as transient), or whether we exhausted
our retries. Previously, if you ran a local example fileserver as in
https://github.com/astral-sh/uv/issues/14171#issuecomment-3014580701 and
then you tried to install Python from it, you'd get:
```
$ export UV_TEST_NO_CLI_PROGRESS=1
$ uv python install 3.8.20 --mirror http://localhost:8000 2>&1 | cat
error: Failed to install cpython-3.8.20-linux-x86_64-gnu
Caused by: Failed to extract archive: cpython-3.8.20-20241002-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
Caused by: failed to unpack `/home/jacko/.local/share/uv/python/.temp/.tmpS4sHHZ/python/lib/libpython3.8.so.1.0`
Caused by: failed to unpack `python/lib/libpython3.8.so.1.0` into `/home/jacko/.local/share/uv/python/.temp/.tmpS4sHHZ/python/lib/libpython3.8.so.1.0`
Caused by: error decoding response body
Caused by: request or response body error
Caused by: error reading a body from connection
Caused by: Connection reset by peer (os error 104)
```
With this change you get:
```
error: Failed to install cpython-3.8.20-linux-x86_64-gnu
Caused by: Request failed after 3 retries
Caused by: Failed to extract archive: cpython-3.8.20-20241002-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
Caused by: failed to unpack `/home/jacko/.local/share/uv/python/.temp/.tmp4Ia24w/python/lib/libpython3.8.so.1.0`
Caused by: failed to unpack `python/lib/libpython3.8.so.1.0` into `/home/jacko/.local/share/uv/python/.temp/.tmp4Ia24w/python/lib/libpython3.8.so.1.0`
Caused by: error decoding response body
Caused by: request or response body error
Caused by: error reading a body from connection
Caused by: Connection reset by peer (os error 104)
```
At the same time, I'm updating the way we handle the retry count to
avoid nested retry loops exceeding the intended number of attempts, as I
mentioned at
https://github.com/astral-sh/uv/issues/14069#issuecomment-3020634281.
It's not clear to me whether we actually want this part of the change,
and I need feedback here.
## Summary
This PR ensures that we avoid cleaning up build directories until the
end of a resolve-and-install cycle. It's not bulletproof (since we could
still run into issues with `uv lock` followed by `uv sync` whereby a
build directory gets cleaned up that's still referenced in the `build`
artifacts), but it at least gets PyTorch building without error with `uv
pip install .`, which is a case that's been reported several times.
Closes https://github.com/astral-sh/uv/issues/14269.
The marker display code assumes that all versions are normalized, in
that all trailing zeroes are stripped. This is not the case for
tilde-equals and equals-star versions, where the trailing zeroes (before
the `.*`) are semantically relevant. This would cause path
dependent-behavior where we would get a different marker string
depending on whether a version with or without a trailing zero was added
to the cache first.
To handle both equals-star and tilde-equals when converting
`python_version` to `python_full_version` markers, we have to merge the
version normalization (i.e. trimming the trailing zeroes) and the
conversion both to `python_full_version` and to `Ranges`, while special
casing equals-star and tilde-equals.
To avoid churn in lockfiles, we only trim in the conversion to `Ranges`
for markers, but keep using untrimmed versions for requires-python.
(Note that this behavior is technically also path dependent, as versions
with and without trailing zeroes have the same Hash and Eq. E.q.,
`requires-python == ">= 3.10.0"` and `requires-python == ">= 3.10"` in
the same workspace could lead to either value in `uv.lock`, and which
one it is could change if we make unrelated (performance) changes.
Always trimming however definitely changes lockfiles, a churn I wouldn't
do outside another breaking or lockfile-changing change.) Nevertheless,
there is a change for users who have `requires-python = "~= 3.12.0"` in
their `pyproject.toml`, as this now hits the correct normalization path.
Fixes#14231Fixes#14270
## Summary
There's a good example of the downside of using verbatim URLs here:
https://github.com/astral-sh/uv/pull/14197#discussion_r2163599625 (we
show two relative paths that point to the same directory, but it's not
clear from the error message).
The diff:
```
2 2 │ ----- stdout -----
3 3 │
4 4 │ ----- stderr -----
5 5 │ error: Requirements contain conflicting URLs for package `library` in all marker environments:
6 │-- ../../library
7 │-- ./library
6 │+- file://[TEMP_DIR]/library
7 │+- file://[TEMP_DIR]/library (editable)
```
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
As explained in the [`codspeed-rust` v3 release
notes](https://github.com/CodSpeedHQ/codspeed-rust/releases/tag/v3.0.0),
the `v3` of the compatibility layers is now required to work with the
latest version(`v3`) of `cargo-codspeed`.
and prefer emulated x64 windows in its stead.
This is preparatory work for shipping support for uv downloading and
installing aarch64 (arm64) windows Pythons. We've [had builds for this
platform ready for a
while](https://github.com/astral-sh/python-build-standalone/pull/387),
but have held back on shipping them due to a fundamental problem:
**The Python packaging ecosystem does not have strong support for
aarch64 windows**, e.g., not many projects build aarch64 wheels yet. The
net effect of this is that, if we handed you an aarch64 python
interpreter on windows, you would have to build a lot more sdists, and
there's a high chance you will simply fail to build that sdist and be
sad.
Yes unfortunately, in this case a non-native Python interpreter simply
*works better* than the native one... in terms of working at all, today.
Of course, if the native interpreter works for your project, it should
presumably have better performance and platform compatibility.
We do not want to stand in the way of progress, as ideally this
situation is a temporary state of affairs as the ecosystem grows to
support aarch64 windows. To enable progress, on aarch64 Windows builds
of uv:
* We will still use a native python interpreter, e.g., if it's at the
front of your `PATH` or the only installed version.
* If we are choosing between equally good interpreters that differ in
architecture, x64 will be preferred.
* If the aarch64 version is newer, we will prefer the aarch64 one.
* We will emit a diagnostic on installation, and show the python request
to pass to uv to force aarch64 windows to be used.
* Will be shipping [aarch64 Windows Python
downloads](https://github.com/astral-sh/python-build-standalone/pull/387)
* Will probably add some kind of global override setting/env-var to
disable this behaviour.
* Will be shipping this behaviour in
[astral-sh/setup-uv](https://github.com/astral-sh/setup-uv)
We're coordinating with Microsoft, GitHub (for the `setup-python`
action), and the CPython team (for the `python.org` installers), to
ensure we're aligned on this default and the timing of toggling to
prefer native distributions in the future.
See discussion in
- https://github.com/astral-sh/uv/issues/12906
---
This is an alternative to
* #13719
which uses sorting rather than filtering, as discussed in
* #13721
This fixes an obscure cache collision in Python interpreter queries,
which we believe to be the root cause of CI flakes we've been seeing
where a project environment is invalidated and recreated.
This work follows from the logs in [this CI
run](4495059999)
which captured one of the flakes with tracing enabled. There, we can see
that the project environment is invalidated because the Python
interpreter in the environment has a different version than expected:
```
DEBUG Checking for Python environment at `.venv`
TRACE Cached interpreter info for Python 3.12.9, skipping probing: .venv/bin/python3
DEBUG The interpreter in the project environment has different version (3.12.9) than it was created with (3.9.21)
```
(this message is updated to reflect #14329)
The flow is roughly:
- We create an environment with 3.12.9
- We query the environment, and cache the interpreter version for
`.venv/bin/python`
- We create an environment for 3.9.12, replacing the existing one
- We query the environment, and read the cached information
The Python cache entries are keyed by the absolute path to the
interpreter, and rely on the modification time (ctime, nsec resolution)
of the canonicalized path to determine if the cache entry should be
invalidated. The key is a hex representation of a u64 sea hasher output
— which is very unlikely to collide.
After an audit of the Python query caching logic, we determined that the
most likely cause of a collision in cache entries is that the
modification times of underlying interpreters are identical. This seems
pretty feasible, especially if the file system does not support
nanosecond precision — though it appears that the GitHub runners do
support it.
The fix here is to include the canonicalized path in the cache key,
which ensures we're looking at the modification time of the _same_
underlying interpreter.
This will "invalidate" all existing interpreter cache entries but that's
not a big deal.
This should also have the effect of reducing cache churn for
interpreters in virtual environments. Now, when you change Python
versions, we won't invalidate the previous cache entry so if you change
_back_ to the old version we can re-use our cached information.
It's a bit speculative, since we don't have a deterministic reproduction
in CI, but this is the strongest candidate given the logs and should
increase correctness regardless.
Closes https://github.com/astral-sh/uv/issues/14160
Closes https://github.com/astral-sh/uv/issues/13744
Closes https://github.com/astral-sh/uv/issues/13745
Once it's confirmed the flakes are resolved, we should revert
- https://github.com/astral-sh/uv/pull/14275
- #13817
## Summary
In #14245, we started normalizing index URLs by dropping the trailing
slash in the lockfile. We added tests to ensure that this didn't cause
existing lockfiles to be invalidated, but we missed one of the
constructors (specifically, the path that's used with
`tool.uv.sources`).
Motivated by some code duplication highlighted in
https://github.com/astral-sh/uv/pull/14201, I noticed we weren't taking
advantage of the existing implementation for casting to a str here.
Unfortunately, we do need a special case for CPython still.
Close#7426
## Summary
Picking up on #8284, I noticed that the `requires_python` object already
has its specifiers canonicalized in the `intersection` method, meaning
`~=3.12` is converted to `>=3.12, <4`. To fix this, we check and warn in
`intersection`.
## Test Plan
Used the same tests from #8284.
## Summary
When the user provides a requirement like `==2.4.*`, we desugar that to
`>=2.4.dev0,<2.5.dev0`. These bounds then appear in error messages, and
worse, they also trick the error message reporter into thinking that the
user asked for a pre-release.
This PR adds logic to convert to the more-concise `==2.4.*`
representation when possible. We could probably do a similar thing for
the compatible release operator (`~=`).
Closes https://github.com/astral-sh/uv/issues/14177.
Co-authored-by: Zanie Blue <contact@zanie.dev>
Python `bin` installations installed with `uv python install --default
--preview` (no version specified) were not being installed as
upgradeable. Instead each link was pointed at the highest patch version
for a minor version. This change ensures that these preview default
installations are also treated as upgradeable.
The PR includes some updates to the related tests. First, it checks the
default install without specified version case. Second, since it's
adding more read link checks, it creates a new `read_link` helper method
to consolidate repeated logic and replace instances of
`#[cfg(unix/windows)` with `if cfg!(unix/windows)`.
Fixes#14247
This PR updates `IndexUrl` parsing to normalize non-file URLs by
removing trailing slashes. It also normalizes registry source URLs when
using them to validate the lockfile.
Prior to this change, when writing an index URL to the lockfile, uv
would use a trailing slash if present in the provided URL and no
trailing slash otherwise. This can cause surprising behavior. For
example, `uv lock --locked` will fail when a package is added with an
`--index` value without a trailing slash and then `uv lock --locked` is
run with a `pyproject.toml` version of the index URL that contains a
trailing slash. This PR fixes this and adds a test for the scenario.
It might be safe to normalize file URLs in the same way, but since
slashes have a well-defined meaning in the context of files and
directories, I chose not to normalize them here.
Closes#13707.
uv currently ignores URL-encoded credentials in a redirect location.
This PR adds a check for these credentials to the redirect handling
logic. If found, they are moved to the Authorization header in the
redirect request.
Closes#11097
Previously we were using the XDG data dir to avoid symlinks, but there's no
particular guarantee that that's not going to be a symlink too. Using the
canonicalized temp dir by default is also slightly nicer for a couple reasons:
It's sometimes faster (an in-memory tempfs on e.g. Arch), and it makes
overriding `$TMPDIR` or `%TMP%` sufficient to control where tests put temp
files, without needing to override `UV_INTERNAL__TEST_DIR` too.
