I think `UV_PROJECT_ENVIRONMENT` is too complicated for use-cases where
the user wants to sync to the active environment. I don't see a
compelling reason not to make opt-in easier. I see a lot of questions
about how to deal with this warning in the issue tracker, but it seems
painful to collect them here for posterity.
A notable behavior here — we'll treat this as equivalent to
`UV_PROJECT_ENVIRONMENT` so... if you point us to a valid virtual
environment that needs to be recreated for some reason (e.g., new Python
version request), we'll happily delete it and start over.
## Summary
This PR removes the ephemeral `.pth` overlay when using a cached
environment. This solution isn't _completely_ safe, since we could
remove the `.pth` file just as another process is starting the
environment... But that risk already exists today, since we could
_overwrite_ the `.pth` file just as another process is starting the
environment, so I think what I've added here is a strict improvement.
Ideally, we wouldn't write this file at all, and we'd instead somehow
(e.g.) pass a file to the interpreter to run at startup? Or find some
other solution that doesn't require poisoning the cache like this.
Closes https://github.com/astral-sh/uv/issues/11117.
# Test Plan
Ran through the great reproduction steps from the linked issue.
Before:
After:
## Summary
I'm not sure that this has much of an effect in practice, but currently,
when we return a virtual environment, the `sys_base_executable ` of the
parent ends up being retained as `sys_base_executable` of the created
environment. But these can be, like, subtly different? If you have a
symlink to a Python, then for the symlink, `sys_base_executable` will be
equal to `sys_executable`. But when you create a virtual environment for
that interpreter, we'll set `home` to the resolved symlink, and so
`sys_base_executable` will be the resolved symlink too, in general.
Anyway, this means that we should now have a consistent value between
(1) returning `Virtualenv` from the creation routine and (2) querying
the created interpreter.
## Summary
It turns out that we were returning slightly different interpreter paths
on repeated `uv run --with` commands. This likely didn't affect many (or
any?) users, but it does affect our test suite, since in the test suite,
we use a symlinked interpreter.
The issue is that on first invocation, we create the virtual
environment, and that returns the path to the `python` executable in the
environment. On second invocation, we return the `python3` executable,
since that gets priority during discovery. This on its own is
potentially ok. The issue is that these resolve to different
`sys._base_executable` values in these flows... The latter gets the
correct value (since it's read from the `home` key), but the former gets
the incorrect value (since it's just the `base_executable` of the
executable that created the virtualenv, which is the symlink).
We now use the same logic to determine the "cached interpreter" as in
virtual environment creation, to ensure consistency between those paths.
Closes https://github.com/astral-sh/uv/issues/11214
Special-cases the first Python executable we find on the `PATH`,
allowing it to be considered during searches for virtual environments.
For some context, there are two stages to Python interpreter discovery
1. We find possible Python executables in various sources
2. We query the executables to determine canonical metadata about the
interpreter
We can't really be "sure" if an executable is a complaint virtual
environment during (1), we need to query the interpreter first. This
means that if you're only allowed to installed into virtual
environments, we'll query every interpreter on your PATH. This is not
performant, and causes confusion for users. Notably, I recently improved
error messaging when we can't find any valid interpreters, by showing
the error message we encounter while querying an interpreter (if any).
However, this is problematic when there's an error for an interpreter
that is not relevant to your search. In
https://github.com/astral-sh/uv/pull/11143, I added filtering to avoid
querying additional interpreters, but that regressed some user
experiences where they were relying on us finding implicitly active
virtual environments via the PATH.
In https://github.com/astral-sh/tokio-tar/pull/2, we accidentally
changed the `target_base` from the target base to the parent of the
file. This would cause hardlink unpacking to fail.
Example: A hardlink at `hardlinked-0.1.0/pyproject.toml` pointing to
`hardlinked-0.1.0/pyproject.toml.real` would try pointing to
`hardlinked-0.1.0/hardlinked-0.1.0/pyproject.toml.real` instead and fail
the unpacking.
The actual fix is in astral-tokio-tar, on the uv side there are only tests.
Fixes#11213
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
I got a bit confused when testing `[dependency-groups]` because uv's
error message had the same typo I did in my `pyproject.toml`.
I tried to fix it, as well as a few comment I found along the way.
These are noisy relative to the effect they have on the user. It seems
better to prioritize hints on poor resolutions. Notably, it seems hard
to make these "not noisy" ref #11091.
Does not include the "lowest" resolution mode, in which lower bounds are
critical.
With the parallel simple index fetching, we would only acquire one
download concurrency token, meaning that we could in the worst case make
times the number of indexes more requests than the user requested limit.
We fix this by passing the semaphore down to the simple API method.
Looks like the set based prioritize tracking from
https://github.com/pubgrub-rs/pubgrub/pull/313 is a slight speedup.
I assume the changed derivation tree in the error snapshot is due to
out-of-sync virtual package priorities, while the main package priority
defining the solution remains stable.
```
$ hyperfine --warmup 2 "./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal" "./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal"
Benchmark 1: ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 115.0 ms ± 4.8 ms [User: 131.0 ms, System: 113.6 ms]
Range (min … max): 108.1 ms … 125.8 ms 25 runs
Benchmark 2: ./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 105.4 ms ± 2.6 ms [User: 118.5 ms, System: 113.5 ms]
Range (min … max): 101.1 ms … 111.9 ms 28 runs
Summary
./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal ran
1.09 ± 0.05 times faster than ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
```
uv-install-wheel had the logic for laying out the installation and for
linking a directory in the same module. We split them up to isolate each
module's logic and tighten the crate's interface to only expose top
level members.
No logic changes, only moving code around.
As before, these are fine-grained PATs and will expire in 366 days.
They're generated by splitting the token into three parts (by `_`) and
base64 encoding.
## Summary
This lets us drop a dependency entirely. `percent-encoding` is used by
`url` and so is already in the graph, whereas `urlencoding` isn't used
by anything else.
## Summary
This PR adds an additional normalization step to `CanonicalUrl` whereby
we now percent-decode the path, to ensure that (e.g.)
`torch-2.5.1%2Bcpu.cxx11.abi-cp39-cp39-linux_x86_64.whl` and
`torch-2.5.1+cpu.cxx11.abi-cp39-cp39-linux_x86_64.whl` are considered
equal. Further, when generating the "reinstall" report, we use the
canonical URL rather than the verbatim URL.
In making this change, I also learned that we don't apply any of the
normalization passes to `file://` URLs. I inadvertently removed it in
93d606aba2,
since setting the password or URL on ` file://` URL errors -- but now
suppress those errors anyway.
Closes https://github.com/astral-sh/uv/issues/11082.
## Test Plan
- Downloaded a [PyTorch
wheel](https://download.pytorch.org/whl/cpu-cxx11-abi/torch-2.5.1%2Bcpu.cxx11.abi-cp39-cp39-linux_x86_64.whl)
- `python3.9 -m pip install
torch-2.5.1+cpu.cxx11.abi-cp39-cp39-linux_x86_64.whl --platform
linux_x86_64 --target foo --no-deps`
- `cargo run pip install
torch-2.5.1+cpu.cxx11.abi-cp39-cp39-linux_x86_64.whl --python-platform
linux --python-version 3.9 --target foo --no-deps`
- Verified that the package had the `~` symbol for the reinstall.
## Summary
We now show a custom error if (1) the file doesn't exist at all, or (2)
it's not a PEP 723 script.
In the future, `uv lock --script` should probably initialize the script,
but that requires a more extensive refactor. At present, we just
silently lock the project instead, which is pretty bad!
Closes https://github.com/astral-sh/uv/issues/10979.
## Summary
I'm not a fan of registries including fragments here that aren't hashes,
but the spec doesn't expressly forbid it. I think it's reasonable to
ignore them.
Specifically, the spec is here:
https://packaging.python.org/en/latest/specifications/simple-repository-api/.
It says that:
> The URL **SHOULD** include a hash in the form of a URL fragment with
the following syntax: `#<hashname>=<hashvalue>`, where `<hashname>`he
lowercase name of the hash function (such as sha256) and `<hashvalue>`
is the hex encoded digest.
But it doesn't mention other fragments.
Closes https://github.com/astral-sh/uv/issues/7257.
## Summary
If we fail to deserialize cached metadata in the cache, we should just
ignore it, rather than failing.
Ideally, this never happens. If it does, it means we missed a cache
version bump. But if it does happen, it should still be non-fatal.
Closes https://github.com/astral-sh/uv/issues/11043.
Closes https://github.com/astral-sh/uv/issues/11101.
## Test Plan
Prior to this PR, the following would fail:
- `uvx uv@0.5.25 venv --python 3.12 --cache-dir foo`
- `uvx uv@0.5.25 pip install ./scripts/packages/hatchling_dynamic
--no-deps --python 3.12 --cache-dir foo`
- `uvx uv@0.5.18 venv --python 3.12 --cache-dir foo`
- `uvx uv@0.5.18 pip install ./scripts/packages/hatchling_dynamic
--no-deps --python 3.12 --cache-dir foo`
We can't go back and fix 0.5.18, but this will prevent such regressions
in the future.
Closes https://github.com/astral-sh/uv/issues/11048
This brings the `PythonEnvironment::from_root` behavior in-line with the
rest of uv Python discovery behavior (and in-line with pip). It's not
clear why we were canonicalizing the path in the first place here.
## Summary
This PR migrates all of our PyTorch tests to use our own mirror, which
includes upload timestamps that we can use to enforce
`--excludes-newer`, making the tests far more stable over time. (Today,
if you checkout old versions of `uv`, many of the PyTorch tests will
fail, since the index contents drift over time.)
Some snapshots changed in this PR (see, e.g.,
`universal_nested_overlapping_local_requirement`). The underlying reason
is that I used the current timestamp when setting upload times in the
PyTorch mirror, but those tests read from both the PyTorch
`--find-links` index _and_ PyPI. I guess we don't omit `--find-links`
entries based on `--excludes-newer`? That might be a bug. But I had to
_increase_ the `--excludes-newer` to include the PyTorch mirror's
`--find-links`, which meant pulling in some newer packages from PyPI
too. This is fine: it's a one-time churn, and they'll be stable going
forward.
In #10875, I relaxed the error checking during resolution to permit
dependencies like `foo[x1]`, where `x1` was defined to be conflicting.
In exchange, the error was, roughly speaking, moved to installation
time. This was achieved by looking at the full set of enabled extras
and checking whether any conflicts occurred. If so, an error was
reported. This ends up being more expressive and permits more valid
configurations.
However, in so doing, there was a bug in how the accumulated extras
were being passed to conflict marker evaluation. Namely, we weren't
accounting for the fact that if `foo[x1]` was enabled, then that fact
should be carried through to all conflict marker evaluations. This is
because some of those will use things like `extra != 'x1'` to indicate
that it should only be included if an extra *isn't* enabled.
In #10985, this manifested with PyTorch where `torch==2.4.1` and
`torch==2.4.1+cpu` were being installed simultaneously. Namely, the
choice to install `torch==2.4.1` was not taking into account that
the `cpu` extra has been enabled. If it did, then it's conflict
marker would evaluate to `false`. Since it didn't, and since
`torch==2.4.1+cpu` was also being included, we ended up installing both
versions.
The approach I took in this PR was to add a second breadth first
traversal (which comes first) over the dependency tree to accumulate all
of the activated extras. Then, only in the second traversal do we
actually build up the resolution graph.
Unfortunately, I have no automatic regression test to include here. The
regression test we _ought_ to include involves `torch`. And while we are
generally find to use those in tests that only generate a lock file, the
regression test here actually requires running installation. And
downloading and installing `torch` in tests is bad juju. So adding a
regression test for this is blocked on better infrastructure for PyTorch
tests. With that said, I did manually verify that the test case in #10985
no longer installs multiple versions of `torch`.
Fixes#10985
