At certain points in the code, dependency groups are represented by
`DevGroups*` naming, probably as a historical artifact. This PR updates
the naming.
This includes renaming `uv-configuration/src/dev.rs` to
`uv-configuration/src/dependency_groups.rs`.
Three edition 2021 compatible sets of changes in preparation for the
edition 2025 split out from #11724.
In edition 2025, `gen` is a keyword, so we escape it as `r#gen`. `ref`
and `ref mut` are not allowed anymore for `&T` and `&mut T`, so we
remove them. `cargo fmt` now formats inside of macros, which the 2021
formatter doesn't undo.
## Summary
This is the pattern I see in a variety of crates, and I believe this is
preferred if you don't _need_ an owned `String`, since you can avoid the
allocation. This could be pretty impactful for us?
## Summary
Since we use `SmallString` internally, there's no benefit to passing an
owned string to the `PackageName` constructor (same goes for
`ExtraName`, etc.). I've kept them for now (maybe that will change in
the future, so it's useful to have clients passed own values if they
_can_), but removed a bunch of usages where we were casting from `&str`
to `String` needlessly to use the constructor.
## Summary
* Upgrade the rust toolchain to 1.85.0. This does not increase the MSRV.
* Update windows trampoline to 1.86 nightly beta (previously in 1.85
nightly beta).
## Test Plan
Existing tests
## Summary
Today, if you have a lockfile that includes conflict markers, we write
those markers out to `requirements.txt` in `uv export`. This is
problematic, since no tool will ever evaluate those markers correctly
downstream.
This PR adds handling for the conflict markers, though it's quite
involved. Specifically, we have a new reachability algorithm that
tracks, for each node, the reachable marker for that node _and_ the
marker conditions under which each conflict item is `true` (at that
node).
I'm slightly worried that this algorithm could be wrong for graphs with
cycles, but we only use this logic for lockfiles with conflicts anyway,
so I think it's a strict improvement over the status quo.
Closes https://github.com/astral-sh/uv/issues/11559.
Closes https://github.com/astral-sh/uv/issues/11548.
## Summary
We need to compute the set of activated groups prior to evaluating the
conflict markers on the groups' dependencies.
Closes https://github.com/astral-sh/uv/issues/11648.
Solving spent a chunk of its time just converting resolutions, the left
two blocks:
These blocks are `ResolverOutput::from_state` with 1.3% and
`ForkState::into_resolution` with 4.1% of resolver thread runtime for
apache airflow universal.
We reduce the overhead spent in those functions, to now 1.1% and 2.1% of
resolver time spend in those functions by:
Commit 1: Replace the hash set for the edges with a vec in
`ForkState::into_resolution`. We deduplicate edges anyway when
collecting them, and the hash-and-insert was slow.
Commit 2: Reduce the distribution clonign in
`ResolverOutput::from_state` by using an `Arc`.
The same profile excerpt for the resolver with the branch (note that
there is now an unrelated block between the two we optimized):
Wall times are noisy, but the profiles show those changes as
improvements.
```
$ hyperfine --warmup 2 "./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal" "./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal"
Benchmark 1: ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 99.1 ms ± 3.8 ms [User: 111.8 ms, System: 115.5 ms]
Range (min … max): 93.6 ms … 110.4 ms 29 runs
Benchmark 2: ./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 97.1 ms ± 4.3 ms [User: 114.8 ms, System: 112.0 ms]
Range (min … max): 90.9 ms … 112.4 ms 29 runs
Summary
./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal ran
1.02 ± 0.06 times faster than ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
```
The particular example I honed in on here was the `e3nn -> sympy 1.13.1`
and `e3nn -> sympy 1.13.3` dependency edges. In particular, while the
former correctly has a conflict marker, the latter's conflict marker was
getting simplified to `true`. This makes the edges trivially
overlapping, and results in both of them getting installed
simultaneously. (A similar problem happens for the `e3nn -> torch`
dependency edges.)
Why does this happen? Well, conflict marker simplification works by
detecting which extras are known to be enabled (and disabled) for each
node in the graph. This ends up being expressed as a set of sets, where
each inner set contains items corresponding to "extras is included" or
"extra is excluded."
The logic then is if _all_ of these sets are satisfied by the conflict
marker on the dependency edge, then this conflict marker can be
simplified by assuming all of the inclusions/exclusions to be true.
In this particular case, we run into an issue where the set of
assumptions discovered for `e3nn` is:
{test[sevennet]}, {}, {~test[m3gnet], ~test[alignn], test[all]}
And the corresponding conflict marker for `e3nn -> sympy 1.13.1` is:
extra == 'extra-4-test-all'
or extra == 'extra-4-test-chgnet'
or (extra != 'extra-4-test-alignn' and extra != 'extra-4-test-m3gnet')
And the conflict marker for `e3nn -> sympy 1.13.3` is:
extra == 'extra-4-test-alignn' or extra == 'extra-4-test-m3gnet'
Evaluating each of the sets above for `sympy 1.13.1`'s conflict
marker results in them all being true. Simplifying in turn results in
the marker being true. For `sympy 1.13.3`, not all of the sets are
satisfied, so this marker is not simplified.
I think the fundamental problem here is that our inferences aren't quite
rich enough to make these logical leaps. In particular, the conflict
marker for `e3nn -> sympy 1.13.3` is not satisfied by _any_ of our sets.
One might therefore conclude that this dependency edge is impossible.
But! The `test[sevennet]` set doesn't actually rule out `test[m3gnet]`
from being included, for example, because there is no conflict. So it is
actually possible for this marker to evaluate to true.
And I think this reveals the problem: for the `e3nn -> sympy 1.13.1`
conflict marker, the inferences don't capture the fact that
`test[sevennet]` _might_ have `test[m3gnet]` enabled, and that would in
turn result in the conflict marker evaluating to `false`. This directly
implies that our simplification here is inappropriate.
It would be nice to revisit how we build our inferences here so that
they are richer and enable us to make correct logical leaps. For now, we
fix this particular bug with a bit of a cop-out: we skip conflict marker
simplification when there are ambiguous dependency edges.
Fixes#11479
Initially, we were limiting Git schemes to HTTPS and SSH as only
supported schemes. We lost this validation in #3429. This incidentally
allowed file schemes, which apparently work with Git out of the box.
A caveat for this is that in tool.uv.sources, we parse the git field
always as URL. This caused a problem with #11425: repo = { git =
'c:\path\to\repo', rev = "xxxxx" } was parsed as a URL where c: is the
scheme, causing a bad error message down the line.
This PR:
* Puts Git URL validation back in place. It bans everything but HTTPS,
SSH, and file URLs. This could be a breaking change, if users were using
a git transport protocol were not aware of, even though never
intentionally supported.
* Allows file: URL in Git: This seems to be supported by Git and we were
supporting it albeit unintentionally, so it's reasonable to continue to
support it.
* It does not allow relative paths in the git field in tool.uv.sources.
Absolute file URLs are supported, whether we want relative file URLs for
Git too should be discussed separately.
Closes#3429: We reject the input with a proper error message, while
hinting the user towards file:. If there's still desire for relative
path support, we can keep it open.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
We want to build `uv-build` without depending on the network crates. In
preparation for that, we split uv-git into uv-git and uv-git-types,
where only uv-git depends on reqwest, so that uv-build can use
uv-git-types.
## Summary
This PR fixes a subtle issue arising from our propagation of
preferences. When we resolve a fork, we take the solution from that fork
and mark all the chosen versions as "preferred" as we move on to the
next fork.
In this specific case, the resolver ended up solving a macOS-specific
fork first, which led us to pick `2.6.0` rather than `2.6.0+cpu`. This
in itself is correct; but when we moved on to the next fork, we
preferred `2.6.0` over `2.6.0+cpu`, despite the fact that `2.6.0` _only_
includes macOS wheel, and that branch was focused on Linux.
Now, in preferences, we prefer local variants (if they exist). If the
local variant ends up not working, we'll presumedly backtrack to the
base version anyway.
Closes https://github.com/astral-sh/uv/issues/11406.
## Summary
This PR revives https://github.com/astral-sh/uv/pull/10017, which might
be viable now that we _don't_ enforce any platforms by default.
The basic idea here is that users can mark certain platforms as required
(empty, by default). When resolving, we ensure that the specified
platforms have wheel coverage, backtracking if not.
For example, to require that we include a version of PyTorch that
supports Intel macOS:
```toml
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = ["torch>1.13"]
[tool.uv]
required-platforms = [
"sys_platform == 'darwin' and platform_machine == 'x86_64'"
]
```
Other than that, the forking is identical to past iterations of this PR.
This would give users a way to resolve the tail of issues in #9711, but
with manual opt-in to supporting specific platforms.
## Summary
This is an alternative to the approach we took in #11063 whereby we
always included `provides-extra` and `requires-dist`, since we needed
some way to differentiate between "no extras" and "lockfile was
generated by a uv version that didn't include extras".
Instead, this PR adds a minor version (called a "revision") to the
lockfile that we can use to indicate support for this feature. While
lockfile version bumps are backwards-incompatible, older uv versions
_can_ read lockfiles with a later revision -- they just won't understand
all the data.
In a future major version bump, we could simplify things and change the
schema to use a (major, minor) format instead of these two separate
fields. But this is the only way to do it that's backwards-compatible
with existing uv versions.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Closes#10597.
Recreated https://github.com/astral-sh/uv/pull/10925 that got closed as
the base branch got merged.
Snapshot tests.
---------
Co-authored-by: Aria Desires <aria.desires@gmail.com>
## Summary
Today, scripts use `CachedEnvironment`, which results in a different
virtual environment path every time the interpreter changes _or_ the
project requirements change. This makes it impossible to provide users
with a stable path to the script that they can use for (e.g.) directing
their editor.
This PR modifies `uv run` to use a stable path for local scripts (we
continue to use `CachedEnvironment` for remote scripts and scripts from
`stdin`). The logic now looks a lot more like it does for projects: we
`get_or_init` an environment, etc.
For now, the path to the script is like:
`environments-v1/4485801245a4732f`, where `4485801245a4732f` is a SHA of
the absolute path to the script. But I'm not picky on that :)
## Summary
Now that `version` is an optional field, we shouldn't error if an
unambiguous package is lacking a version. We can still enforce the same
guarantees via `source`, since we always set version and source
together, if the package is unambiguous. I also retained the same error
for non-local packages that lack a version like this.
Closes https://github.com/astral-sh/uv/issues/11384.
The underlying cause here, I believe, was that we weren't accounting
for the case where an edge could be visited *without* any extras
enabled. Because of that, we got into situations where we thought
there was only one path to an edge when there were actually more
paths. This in turn lead to us erroneously doing simplification where
it actually isn't justified. And in turn lead to duplicate versions
of the same package being installed in the same environment.
The fix for this ends up being really simple: in the case where we
don't add any conflict items for a package during graph traversal,
we materialize an empty set of conflicts to mark the case of no
extras being enabled when visiting the child edges. This is enough
to propagate the knowledge of multiple paths to the same edge and
causes us to avoid doing improper simplifications.
This does fix the problem in the snapshot, but it does also I think
lead to other cases where simplifications are no longer possible
(hence the changes to the airflow snapshot). But this seems
expected, since we are doing strictly less simplification than we
were before. It's unclear if all of those cases were actual bugs
or not though.
Given an input in the shape:
```
foo[bar]==1.0.0; sys_platform == 'linux'
foo==1.0.0; sys_platform != 'linux'
```
We would write either
```
foo==1.0.0; sys_platform == 'linux'
```
or
```
foo==1.0.0
```
depending on the iteration order, as the first one is from the marker
proxy package and the second one from the package without marker.
The fix correctly merges graph entries when there are two nodes with
different extras and different markers.
I tried to write a packse test but it failed due to a different
iteration order showing the correct case directly instead of the failing
one we'd need.
Only `strip_extras` is affected, since `combine_extras` uses
`version_marker`.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
I got a bit confused when testing `[dependency-groups]` because uv's
error message had the same typo I did in my `pyproject.toml`.
I tried to fix it, as well as a few comment I found along the way.
With the parallel simple index fetching, we would only acquire one
download concurrency token, meaning that we could in the worst case make
times the number of indexes more requests than the user requested limit.
We fix this by passing the semaphore down to the simple API method.
Looks like the set based prioritize tracking from
https://github.com/pubgrub-rs/pubgrub/pull/313 is a slight speedup.
I assume the changed derivation tree in the error snapshot is due to
out-of-sync virtual package priorities, while the main package priority
defining the solution remains stable.
```
$ hyperfine --warmup 2 "./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal" "./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal"
Benchmark 1: ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 115.0 ms ± 4.8 ms [User: 131.0 ms, System: 113.6 ms]
Range (min … max): 108.1 ms … 125.8 ms 25 runs
Benchmark 2: ./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 105.4 ms ± 2.6 ms [User: 118.5 ms, System: 113.5 ms]
Range (min … max): 101.1 ms … 111.9 ms 28 runs
Summary
./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal ran
1.09 ± 0.05 times faster than ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
```
In #10875, I relaxed the error checking during resolution to permit
dependencies like `foo[x1]`, where `x1` was defined to be conflicting.
In exchange, the error was, roughly speaking, moved to installation
time. This was achieved by looking at the full set of enabled extras
and checking whether any conflicts occurred. If so, an error was
reported. This ends up being more expressive and permits more valid
configurations.
However, in so doing, there was a bug in how the accumulated extras
were being passed to conflict marker evaluation. Namely, we weren't
accounting for the fact that if `foo[x1]` was enabled, then that fact
should be carried through to all conflict marker evaluations. This is
because some of those will use things like `extra != 'x1'` to indicate
that it should only be included if an extra *isn't* enabled.
In #10985, this manifested with PyTorch where `torch==2.4.1` and
`torch==2.4.1+cpu` were being installed simultaneously. Namely, the
choice to install `torch==2.4.1` was not taking into account that
the `cpu` extra has been enabled. If it did, then it's conflict
marker would evaluate to `false`. Since it didn't, and since
`torch==2.4.1+cpu` was also being included, we ended up installing both
versions.
The approach I took in this PR was to add a second breadth first
traversal (which comes first) over the dependency tree to accumulate all
of the activated extras. Then, only in the second traversal do we
actually build up the resolution graph.
Unfortunately, I have no automatic regression test to include here. The
regression test we _ought_ to include involves `torch`. And while we are
generally find to use those in tests that only generate a lock file, the
regression test here actually requires running installation. And
downloading and installing `torch` in tests is bad juju. So adding a
regression test for this is blocked on better infrastructure for PyTorch
tests. With that said, I did manually verify that the test case in #10985
no longer installs multiple versions of `torch`.
Fixes#10985
## Summary
I'm open to not merging this -- I was kind of just interested in what
the API looked like. But the idea is: we can avoid hashing values twice
and unnecessarily cloning within the priority map by using the raw entry
API.
This collects ALL activated extras while traversing the lock file to
produce a `Resolution` for installation. If any two extras are activated
that are conflicting, then an error is produced.
We add a couple of tests to demonstrate the behavior. One case is
desirable (where we conditionally depend on `package[extra]`) and the
other case is undesirable (where we create an uninstallable lock file).
Fixes#9942, Fixes#10590
This will make `package[extra]` work even when `extra` is declared as a
conflicting extra.
Note that this isn't relevant for dependency groups since AFAIK those
can actually only be enabled on the CLI. There is no `package:group`
dependency syntax.
This removes the error that was causing folks problems.
This does result in some snapshot updates that are arguably wrong, or at
least sub-optimal. However, it's actually intended. Because the approach
we're going to take is going to permit the creation of uninstallable
lock files as a side effect. In the future, we will modify this test to
check that, while `uv lock` succeeds, `uv sync` will always fail.
