This PR redacts credentials in displayed URLs.
It mostly relies on a `redacted_url` function (and where possible
`IndexUrl::redacted`). This is a quick way to prevent leaked credentials
but it's prone to programmer error when adding new trace statements. A
better follow-on would use a `RedactedUrl` type with the appropriate
`Display` implementation. This would allow us to still extract
credentials from the URL while displaying it securely. On the plus side,
the sites where the `redacted_url` function are used serve as easy
signposts for where to use the new type in a future PR.
Closes#1714.
## Summary
The only thing that changed for #12175 relevant to the existing
downloads is the order of nesting, so we're checking all nested IO
errors instead of only the first one.
See #13238
## Test Plan
This is an educated guess based on what happens if I turn off the
network during a download.
```
Downloading cpython-3.13.3-linux-x86_64-gnu (download) (20.3MiB)
TRACE Considering retry of error: ExtractError("cpython-3.13.3-20250409-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz", Io(Custom { kind: Other, error: TarError { desc: "failed to unpack `/home/konsti/.local/share/uv/python/.temp/.tmpe3AIvt/python/lib/libpython3.13.so.1.0`", io: Custom { kind: Other, error: TarError { desc: "failed to unpack `python/lib/libpython3.13.so.1.0` into `/home/konsti/.local/share/uv/python/.temp/.tmpe3AIvt/python/lib/libpython3.13.so.1.0`", io: Custom { kind: Other, error: reqwest::Error { kind: Decode, source: reqwest::Error { kind: Body, source: TimedOut } } } } } } }))
TRACE Cannot retry IO error: not one of `ConnectionReset` or `UnexpectedEof`
TRACE Cannot retry IO error: not one of `ConnectionReset` or `UnexpectedEof`
TRACE Cannot retry error: not an IO error
error: Failed to install cpython-3.13.3-linux-x86_64-gnu
Caused by: Failed to extract archive: cpython-3.13.3-20250409-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
Caused by: failed to unpack `/home/konsti/.local/share/uv/python/.temp/.tmpe3AIvt/python/lib/libpython3.13.so.1.0`
Caused by: failed to unpack `python/lib/libpython3.13.so.1.0` into `/home/konsti/.local/share/uv/python/.temp/.tmpe3AIvt/python/lib/libpython3.13.so.1.0`
Caused by: error decoding response body
Caused by: request or response body error
Caused by: operation timed out
```
uv’s default index strategy was designed with dependency confusion
attacks in mind. [According to the
docs](https://docs.astral.sh/uv/configuration/indexes/#searching-across-multiple-indexes),
“if a package exists on an internal index, it should always be installed
from the internal index, and never from PyPI”. Unfortunately, this is
not true in the case where authentication fails on that internal index.
In that case, uv will simply try the next index (even on the
`first-index` strategy). This means that uv is not secure by default in
this common scenario.
This PR causes uv to stop searching for a package if it encounters an
authentication failure at an index. It is possible to opt out of this
behavior for an index with a new `pyproject.toml` option
`ignore-error-codes`. For example:
```
[[tool.uv.index]]
name = "my-index"
url = "<index-url>"
ignore-error-codes = [401, 403]
```
This will also enable users to handle idiosyncratic registries in a more
fine-grained way. For example, PyTorch registries return a 403 when a
package is not found. In this PR, we special-case PyTorch registries to
ignore 403s, but users can use `ignore-error-codes` to handle similar
behaviors if they encounter them on internal registries.
Depends on #12651Closes#9429Closes#12362
Some registries (like Azure Artifact) can require you to authenticate
separately for every package URL if you do not authenticate for the
/simple endpoint. These changes make the auth middleware aware of index
URL endpoints and attempts to fetch keyring credentials for such an
index URL when making a request to any URL it's a prefix of.
The current uv behavior is to cache credentials either at the request
URL or realm level. But with these changes, we also need to cache
credentials at the index level. Note that when uv does not detect an
index URL for a request URL, it will continue to apply the old behavior.
Addresses part of #4056Closes#4583Closes#11236Closes#11391Closes#11507
This PR restores #13041 and integrates two PRs from @zanieb:
* #13038
* #13040
It also adds tests for relative URI and fragment handling.
Closes#13037.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
uv was failing to authenticate on 302 redirects when credentials were
available. This was because it was relying on `reqwest_middleware`'s
default redirect behavior which bypasses the middleware pipeline when
trying the redirect request (and hence bypasses our authentication
middleware). This PR updates uv to retrigger the middleware pipeline
when handling a 302 redirect, correctly using credentials from the URL,
the keyring, or `.netrc`.
Closes#5595Closes#11097
## Summary
This PR extends `[[tool.uv.index]]` to support `--find-links`-style
"flat" indexes, so that users can point to such indexes without using
`--find-links` _and_ get access to the full functionality of
`[[tool.uv.index]]` (e.g., they can now pin packages to
`--find-links`-style indexes).
Note that, at present, `--find-links` indexes actually have some quirky
behavior, in that we combine them into a single entity and then merge
the discovered distributions into each Simple API-style index. The
motivation here, IIRC, was to match pip's behavior quite closely. I'm
interested in _removing_ that behavior, but it'd be breaking (and may
also be inconvenient for some use-cases). So, the behavior for indexes
passed in via `--find-links` remains completely unchanged. However,
`[[tool.uv.index]]` entries with `format = "flat"` are now treated
identically to those defined with `format = "simple"` (the default), in
that we stop after we find the first-matching index, etc.
Closes https://github.com/astral-sh/uv/issues/11634.
## Summary
I don't know if I actually want to commit this, but I did it on the
plane last time and just polished it off (got it to compile) while
waiting to board.
## Summary
This PR modifies the requirement source entities to store a (new)
container struct that wraps `IndexUrl`. This will allow us to store
user-defined metadata alongside `IndexUrl`, and propagate that metadata
throughout resolution.
Specifically, I need to store the "kind" of the index (Simple API vs.
`--find-links`), but I also ran into this problem when I tried to add
support for overriding `Cache-Control` headers on a per-index basis: at
present, we have no way to passing around metadata alongside an
`IndexUrl`.
## Summary
I want to use the flat index client from within the registry client, so
making them both depend on the same underlying primitives rather than
having the flat index client depend on the registry client.
## Summary
It's possible that the PyTorch version the user depends on isn't in the
latest index. These indexes are equally trusted, so we should override
the policy.
Closes#12357.
## Summary
This is a prototype that I'm considering shipping under `--preview`,
based on [`light-the-torch`](https://github.com/pmeier/light-the-torch).
`light-the-torch` patches pip to pull PyTorch packages from the PyTorch
indexes automatically. And, in particular, `light-the-torch` will query
the installed CUDA drivers to determine which indexes are compatible
with your system.
This PR implements equivalent behavior under `--torch-backend auto`,
though you can also set `--torch-backend cpu`, etc. for convenience.
When enabled, the registry client will fetch from the appropriate
PyTorch index when it sees a package from the PyTorch ecosystem (and
ignore any other configured indexes, _unless_ the package is explicitly
pinned to a different index).
Right now, this is only implemented in the `uv pip` CLI, since it
doesn't quite fit into the lockfile APIs given that it relies on feature
detection on the currently-running machine.
## Test Plan
On macOS, you can test this with (e.g.):
```shell
UV_TORCH_BACKEND=auto UV_CUDA_DRIVER_VERSION=450.80.2 cargo run \
pip install torch --python-platform linux --python-version 3.12
```
On a GPU-enabled EC2 machine:
```shell
ubuntu@ip-172-31-47-149:~/uv$ UV_TORCH_BACKEND=auto cargo run pip install torch -v
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.31s
Running `target/debug/uv pip install torch -v`
DEBUG uv 0.6.6 (e95ca063b 2025-03-14)
DEBUG Searching for default Python interpreter in virtual environments
DEBUG Found `cpython-3.13.0-linux-x86_64-gnu` at `/home/ubuntu/uv/.venv/bin/python3` (virtual environment)
DEBUG Using Python 3.13.0 environment at: .venv
DEBUG Acquired lock for `.venv`
DEBUG At least one requirement is not satisfied: torch
warning: The `--torch-backend` setting is experimental and may change without warning. Pass `--preview` to disable this warning.
DEBUG Detected CUDA driver version from `/sys/module/nvidia/version`: 550.144.3
...
```
## Summary
This ended up being more involved than expected. The gist is that we
setup all the packages we want to reinstall upfront (they're passed in
on the command-line); but at that point, we don't have names for all the
packages that the user has specified. (Consider, e.g., `uv pip install
.` -- we don't have a name for `.`, so we can't add it to the list of
`Reinstall` packages.)
Now, `Reinstall` also accepts paths, so we can augment `Reinstall` based
on the user-provided paths.
Closes#12038.
Adds a new optional key `auth-policy` to `[tool.uv.index]` that sets the
authentication policy for the index URL.
The default is `"auto"`, which attempts to authenticate when necessary.
`"always"` always attempts to authenticate and fails if the endpoint is
unauthenticated. `"never"` never attempts to authenticate.
These policy address two kinds of cases:
* Some indexes don’t fail on unauthenticated requests; instead they just
forward to the public PyPI. This can leave the user confused as to why
their package is missing. The "always" policy prevents this.
* "never" allows users to ensure their credentials couldn't be leaked to
an unexpected index, though it will only allow for successful requests
on an index that doesn't require credentials.
Closes#11600
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Closes#2410
<!-- What's the purpose of the change? What does it do, and why? -->
This changes the name of files in `wheels` bucket to use a hash instead
of the wheel name as to not exceed maximum file length limit on various
systems.
This only addresses the primary concern of #2410. It still does _not_
address:
- Path limit of 260 on windows:
https://github.com/astral-sh/uv/issues/2410#issuecomment-2062020882
To solve this we need to opt-in to longer path limits on windows
([ref](https://github.com/astral-sh/uv/issues/2410#issuecomment-2150532658)),
but I think that is a separate issue and should be a separate MR.
- Exceeding filename limit while building a wheel from source
distribution
As per my understanding, this is out of uv's control. Name of the output
wheel will be decided by build-backend used by the project. For wheels
built from source distribution, pip also uses the wheel names in cache.
So I have not touched `sdists` cache.
I have added a `filename: WheelFileName` field in `Archive`, so we can
use it while indexing instead of relying on the filename on disk.
Another way to do this was to read `.dist-info/WHEEL` and
`.dist-info/METADATA` and build `WheelFileName` but that seems less
robust and will be slower.
## Test Plan
<!-- How was it tested? -->
Tested by installing `yt-dlp`, `httpie` and `sqlalchemy` and verifying
that cache files in `wheels` bucket use hash.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
* Upgrade the rust toolchain to 1.85.0. This does not increase the MSRV.
* Update windows trampoline to 1.86 nightly beta (previously in 1.85
nightly beta).
## Test Plan
Existing tests
With the parallel simple index fetching, we would only acquire one
download concurrency token, meaning that we could in the worst case make
times the number of indexes more requests than the user requested limit.
We fix this by passing the semaphore down to the simple API method.
## Summary
This lets us drop a dependency entirely. `percent-encoding` is used by
`url` and so is already in the graph, whereas `urlencoding` isn't used
by anything else.
## Summary
I'm not a fan of registries including fragments here that aren't hashes,
but the spec doesn't expressly forbid it. I think it's reasonable to
ignore them.
Specifically, the spec is here:
https://packaging.python.org/en/latest/specifications/simple-repository-api/.
It says that:
> The URL **SHOULD** include a hash in the form of a URL fragment with
the following syntax: `#<hashname>=<hashvalue>`, where `<hashname>`he
lowercase name of the hash function (such as sha256) and `<hashvalue>`
is the hex encoded digest.
But it doesn't mention other fragments.
Closes https://github.com/astral-sh/uv/issues/7257.
## Summary
This PR migrates all of our PyTorch tests to use our own mirror, which
includes upload timestamps that we can use to enforce
`--excludes-newer`, making the tests far more stable over time. (Today,
if you checkout old versions of `uv`, many of the PyTorch tests will
fail, since the index contents drift over time.)
Some snapshots changed in this PR (see, e.g.,
`universal_nested_overlapping_local_requirement`). The underlying reason
is that I used the current timestamp when setting upload times in the
PyTorch mirror, but those tests read from both the PyTorch
`--find-links` index _and_ PyPI. I guess we don't omit `--find-links`
entries based on `--excludes-newer`? That might be a bug. But I had to
_increase_ the `--excludes-newer` to include the PyTorch mirror's
`--find-links`, which meant pulling in some newer packages from PyPI
too. This is fine: it's a one-time churn, and they'll be stable going
forward.
## Summary
On Windows, we have a lot of issues with atomic replacement and such.
There are a bunch of different failure modes, but they generally
involve: trying to persist a fail to a path at which the file already
exists, trying to replace or remove a file while someone else is reading
it, etc.
This PR adds locks to all of the relevant database paths. We already use
these advisory locks when building source distributions; now we use them
when unzipping wheels, storing metadata, etc.
Closes#11002.
## Test Plan
I ran the following script:
```shell
# Define the cache directory path
$cacheDir = "C:\Users\crmar\workspace\uv\cache"
# Clear the cache directory if it exists
if (Test-Path $cacheDir) {
Remove-Item -Recurse -Force $cacheDir
}
# Create the cache directory again
New-Item -ItemType Directory -Force -Path $cacheDir
# Define the command to run with --cache-dir flag
$command = {
param ($venvPath)
# Create a virtual environment in the specified path with --python
uv venv $venvPath
# Run the pip install command with --cache-dir flag
C:\Users\crmar\workspace\uv\target\profiling\uv.exe pip install flask==1.0.4 --no-binary flask --cache-dir C:\Users\crmar\workspace\uv\cache -v --python $venvPath
}
# Define the paths for the different virtual environments
$venv1 = "C:\Users\crmar\workspace\uv\venv1"
$venv2 = "C:\Users\crmar\workspace\uv\venv2"
$venv3 = "C:\Users\crmar\workspace\uv\venv3"
$venv4 = "C:\Users\crmar\workspace\uv\venv4"
$venv5 = "C:\Users\crmar\workspace\uv\venv5"
# Start the command in parallel five times using Start-Job, each with a different venv
$job1 = Start-Job -ScriptBlock $command -ArgumentList $venv1
$job2 = Start-Job -ScriptBlock $command -ArgumentList $venv2
$job3 = Start-Job -ScriptBlock $command -ArgumentList $venv3
$job4 = Start-Job -ScriptBlock $command -ArgumentList $venv4
$job5 = Start-Job -ScriptBlock $command -ArgumentList $venv5
# Wait for all jobs to complete
$jobs = @($job1, $job2, $job3, $job4, $job5)
$jobs | ForEach-Object { Wait-Job $_ }
# Retrieve the results (optional)
$jobs | ForEach-Object { Receive-Job -Job $_ }
# Clean up the jobs
$jobs | ForEach-Object { Remove-Job -Job $_ }
```
And ensured it succeeded in five straight invocations (whereas on
`main`, it consistently fails with a variety of different traces).
