## Summary
Prefers, in order:
- The major-minor version of an interpreter discovered via `--python`.
- The `requires-python` from the workspace.
- The major-minor version of the default interpreter.
If the `--python` request is a version or a version range, we use that
without fetching an interpreter.
Closes https://github.com/astral-sh/uv/issues/5299.
The `RequirementsTxtComparator` was written assuming there is one
distribution per package name. This changed with the universal
resolution, which allows multiple versions or urls for the same package
name. The sorting we emitted for these new entries was incidental.
With this change, we properly sort these entries by name, version and
then url in universal mode.
This is an output format change for `--universal` users.
## Summary
Excellent find from @konstin. If we have a package that's included in
two forks at the same version, but with different URLs, we need to avoid
collapsing them in the lockfile.
Closes https://github.com/astral-sh/uv/issues/5294.
Looking at how to merge identical forks, i found that the `Resolution`
can be simplified by treating it as a nodes and edges store (plus pins,
they are separate since they are per name-version, not per
(virtual-)package-version). This should also make #5294 more apparent,
which i didn't touch here.
I additionally added some doc comments to the `Resolution` types.
Our everything-method `solve` tends to grow large, so before i'm adding
more logic, i'm moving some code and some logging statements around to
keep it manageable.
I made minor changes to the logging, otherwise no logic changes, only
refactoring.
## Summary
`dearpygui==1.9.1 has no wheels are available with a matching Python
ABI` is way better than `he requested Python version (>=3.12.3) does not
satisfy Python>=3.12.3`.
Closes https://github.com/astral-sh/uv/issues/5284.
This fixes resolving packages that publish an invalid stub to pypi, such
as tensorrt-llm.
## Summary
In https://github.com/astral-sh/uv/pull/3138 , we implemented
`unsafe-best-match`. However, it seems to not quite work as expected.
When multiple indices contain the same version, it's not clear which
index the current code uses. This PR fixes that to use the first index
the package is in.
## Test Plan
```console
$ echo 'tensorrt-llm==0.11.0' | ./target/debug/uv pip compile - --extra-index-url https://pypi.nvidia.com --python-version=3.10 --index-strategy=unsafe-best-match --annotation-style=line
```
## Summary
Given `Requires-Python: ">=3.12.3"`, we were rejecting wheels like
`dearpygui-1.11.1-cp312-cp312-win_amd64.whl`, since `3.12.0` is not
included in `>=3.12.3`. We instead need to test against the major-minor
version of `Requires-Python`.
The easiest way to do this, I think, is the use the `RequiresPython`
struct, which has a single bound that we can truncate the major-minor.
This also means that we now allow
`dearpygui-1.11.1-cp312-cp312-win_amd64.whl` for specifiers like
`Requires-Python: "==3.10.*"`. This is incorrect on the surface, but it
does match our semantics for `Requires-Python` elsewhere: we treat it as
a lower bound.
Closes https://github.com/astral-sh/uv/issues/5287.
## Summary
This PR modifies the lockfile to include the impactful resolution
settings, like the resolution and pre-release mode. If any of those
values change, we want to ignore the existing lockfile. Otherwise,
`--resolution lowest-direct` will typically have no effect, which is
really unintuitive.
Closes https://github.com/astral-sh/uv/issues/5226.
## Summary
This fixes a few bugs introduced by
https://github.com/astral-sh/uv/pull/5104. I previously thought we could
track conflicting locals the same way we track conflicting URLs in
forks, but it turns out that ends up being very tricky. URL forks work
because we prioritize directly URL requirements. We can't prioritize
locals in the same way without conflicting with the URL prioritization
(this may be possible but it's not trivial), so we run into issues where
a correct resolution depends on the order in which dependencies are
traversed.
Instead, we track local versions across all forks in `Locals`. When
applying a local version, we apply all locals with markers that
intersect with the current fork. This way we end up applying some local
versions without creating a fork. For example, given:
```
// pyproject.toml
dependencies = [
"torch==2.0.0+cu118 ; platform_machine == 'x86_64'",
]
// requirements.in
torch==2.0.0
.
```
We choose `2.0.0+cu118` in all cases. However, if a disjoint fork is
created based on local versions, the resolver will choose the most
compatible local when it narrows to a specific fork. Thus we correctly
respect local versions when forking:
```
// pyproject.toml
dependencies = [
"torch==2.0.0+cu118 ; platform_machine == 'x86_64'",
"torch==2.0.0+cpu ; platform_machine != 'x86_64'"
]
// requirements.in
torch==2.0.0
.
```
We should also be able to use a similar strategy for
https://github.com/astral-sh/uv/pull/5150.
## Test Plan
This fixes https://github.com/astral-sh/uv/issues/5220 locally for me,
as well as a few other bugs that were not reported yet.
## Summary
This ensures that we process Python installs and uninstalls as soon as
they complete, rather than waiting for them all to complete, then
processing them sequentially. In practice, it shouldn't be much of a
difference (since the processing is code is fairly light), but it
strikes me as more correct.
As user, you specify a list of extras. Internally, we decompose this
into one virtual package per extra. We currently leak this abstraction
by writing one entry per extra to the lockfile:
```toml
[[distribution]]
name = "foo"
version = "4.39.0.dev0"
source = { editable = "." }
dependencies = [
{ name = "pandas" },
{ name = "pandas", extra = "excel" },
{ name = "pandas", extra = "hdf5" },
{ name = "pandas", extra = "html", marker = "os_name != 'posix'" },
{ name = "pandas", extra = "output-formatting", marker = "os_name == 'posix'" },
{ name = "pandas", extra = "plot", marker = "os_name == 'posix'" },
]
```
Instead, we should merge the extras into a list of extras, creating a
more concise lockfile:
```toml
[[distribution]]
name = "foo"
version = "4.39.0.dev0"
source = { editable = "." }
dependencies = [
{ name = "pandas", extra = ["excel", "hdf5"] },
{ name = "pandas", extra = ["html"], marker = "os_name != 'posix'" },
{ name = "pandas", extra = ["output-formatting", "plot"], marker = "os_name == 'posix'" },
]
```
The base package is now implicitly included, as it is in PEP 508.
Fixes#4888
Warn when there is a direct dependency without a lower bound and
`--resolution lowest` is set.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
Hashes will be validated if present, but aren't required (since, e.g.,
some registries will omit them, as will Git dependencies and such).
Closes https://github.com/astral-sh/uv/issues/5168.
## Summary
We only need to store one hash -- it should be the "strongest" hash. In
practice, most registries (like PyPI) only serve one, and we only
compute a SHA256 hash for direct URLs.
Part of: https://github.com/astral-sh/uv/issues/4924
## Test Plan
I verified that changing:
```diff
diff --git a/crates/distribution-types/src/hash.rs b/crates/distribution-types/src/hash.rs
index 553a74f55..d36c62286 100644
--- a/crates/distribution-types/src/hash.rs
+++ b/crates/distribution-types/src/hash.rs
@@ -31,7 +31,7 @@ impl<'a> HashPolicy<'a> {
pub fn algorithms(&self) -> Vec<HashAlgorithm> {
match self {
Self::None => vec![],
- Self::Generate => vec![HashAlgorithm::Sha256],
+ Self::Generate => vec![HashAlgorithm::Sha256, HashAlgorithm::Sha512],
Self::Validate(hashes) => {
let mut algorithms = hashes.iter().map(HashDigest::algorithm).collect::<Vec<_>>();
algorithms.sort();
```
Then running `uv lock` with a URL gave me:
```toml
[[distribution]]
name = "iniconfig"
version = "2.0.0"
source = { url = "62565a6e1c/iniconfig-2.0.0-py3-none-any.whl" }
wheels = [
{ url = "62565a6e1c/iniconfig-2.0.0-py3-none-any.whl", hash = "sha512:44cc53a6c8dd7cf4d6d52bded308bcc4b4f85fff2ed081f60f7d4beaa86a7cde6d099e3976331232d4cbd472ad5d1781064725b0999c7cd3a2a4d42df687ee81" },
]
```
* Use a dedicated `ResolverMarkers` check in the fork state. This is
better than the `MarkerTree::And(Vec::new())` check.
* Report the timing correct naming universal resolution instead of two
spaces around an empty string when there are no markers.
* On resolution error, show the split that we're in. I'm not sure how to
word this, since we're doing a universal resolution until we fork, so
the trace may contain information from requirements that are not part of
this fork.
## Summary
Currently, the `Locals` type relies on there being a single local
version for a given package. With marker expressions this may not be
true, a similar problem to https://github.com/astral-sh/uv/pull/4435.
This changes the `Locals` type to `ForkLocals`, which tracks locals for
a given fork. Local versions are now tracked on `PubGrubRequirement`
before forking.
Resolves https://github.com/astral-sh/uv/issues/4580.
Specifically, this shows the resolution produced by the
resolver *before* constructing a resolution graph.
Unlike most trace messages, this is a multi-line message
that needs to do some small amount of work to build
itself. So we do an explicit gating on the log level here
instead of just relying on the `trace!` macro itself.
## Summary
The example in the linked issue doesn't quite work, but I think it has
to do with the existing filtering logic. Will follow-up separately.
Closes https://github.com/astral-sh/uv/issues/5012.
In some cases, it's possible for the marker expressions on conflicting
dependency specification to be disjoint but *incomplete*. That is, if
one unions the disjoint markers, the result is not the complete set of
marker environments possible. There may be some "gap" of marker
environments not covered by the markers.
This is a problem in practice because, before this commit, we only
created forks in the resolver for specific marker expressions. So if a
dependency happened to fall in a "gap," our resolver would never see it.
This commit fixes this by adding a new split covering the negation of
the union of all marker expressions in a set of forks for a specific
package.
Originally, I had planned on only creating this split when it was known
that the gap actually existed. That is, when the negation of the marker
expressions did *not* correspond to the empty set. After a lot of
thought, unfortunately, this (I believe) effectively boils down to 3SAT,
which is NP-complete.
Instead, what we do here is *always* create an extra split unless we can
definitively tell that it is empty. We look for a few cases, but
otherwise throw our hands up and potentially do wasted work.
This also updates the lock scenario tests to reflect the actual bug fix
here.
The only pubgrub error that can occur is a `NoSolutionError`, and the
only place it can occur is `unit_propagation`, all other variants if
`PubGrubError` are unreachable. By changing the return type on pubgrub's
side (https://github.com/astral-sh/pubgrub/pull/28), we can remove the
pattern matching and the `unreachable!()` asserts on `PubGrubError`.
Our pubgrub error wrapper used to have a two phased initialization,
first mostly stubs in `solve[_tracked]()` and then adding the actual
context in `resolve()`. When constructing the error in `solve` we
already have all this context, so we can unify this to a regular
constructor and remove the special casing in `resolve()` and `hints()`.
Currently, with
```toml
[project]
name = "transformers"
version = "4.39.0.dev0"
requires-python = ">=3.10"
dependencies = [
"torch==1.10.0"
]
```
i get
```
$ uv sync --preview
Resolved 3 packages in 7ms
error: found distribution torch==1.10.0 @ registry+https://pypi.org/simple with neither wheels nor source distribution
```
This error message is wrong, there are wheels, they are just not
compatible. I initially got this error message during `uv lock` (in a
build), so i also added that this is about installation, not about
locking.
We should reject this version immediately because with the current
requires python, it can never be installed, but even then we need to
change the error message because you can be on the correct python
version, but an unsupported platform.
## Summary
Use the lockfile to prefill the `InMemoryIndex` used by the resolver.
This enables us to resolve completely from the lockfile without making
any network requests/builds if the requirements are unchanged. It also
means that if new requirements are added we can still avoid most I/O
during resolution, partially addressing
https://github.com/astral-sh/uv/issues/3925.
The main limitation of this PR is that resolution from the lockfile can
fail if new versions are requested that are not present in the lockfile,
in which case we have to perform a fresh resolution. Fixing this would
likely require lazy version/metadata requests by `VersionMap` (this is
different from the lazy parsing we do, the list of versions in a
`VersionMap` is currently immutable).
Resolves https://github.com/astral-sh/uv/issues/3892.
## Test Plan
Added a `deterministic!` macro that ensures that a resolve from the
lockfile and a clean resolve result in the same lockfile output for all
our current tests.
## Summary
We currently store wheel URLs in an unparsed state because we don't have
a stable parsed representation to use with rykv. Unfortunately this
means we end up reparsing unnecessarily in a lot of places, especially
when constructing a `Lock`. This PR adds a `UrlString` type that lets us
avoid reparsing without losing the validity of the `Url`.
## Test Plan
Shaves off another ~10 ms from
https://github.com/astral-sh/uv/issues/4860.
```
➜ transformers hyperfine "../../uv/target/profiling/uv lock" "../../uv/target/profiling/baseline lock" --warmup 3
Benchmark 1: ../../uv/target/profiling/uv lock
Time (mean ± σ): 120.9 ms ± 2.5 ms [User: 126.0 ms, System: 80.6 ms]
Range (min … max): 116.8 ms … 125.7 ms 23 runs
Benchmark 2: ../../uv/target/profiling/baseline lock
Time (mean ± σ): 129.9 ms ± 4.2 ms [User: 127.1 ms, System: 86.1 ms]
Range (min … max): 123.4 ms … 141.2 ms 23 runs
Summary
../../uv/target/profiling/uv lock ran
1.07 ± 0.04 times faster than ../../uv/target/profiling/baseline lock
```
## Summary
Avoid serializing and writing the lockfile if a cheap comparison shows
that the contents have not changed.
## Test Plan
Shaves ~10ms off of https://github.com/astral-sh/uv/issues/4860 for me.
```
➜ transformers hyperfine "../../uv/target/profiling/uv lock" "../../uv/target/profiling/baseline lock" --warmup 3
Benchmark 1: ../../uv/target/profiling/uv lock
Time (mean ± σ): 130.5 ms ± 2.5 ms [User: 130.3 ms, System: 85.0 ms]
Range (min … max): 126.8 ms … 136.9 ms 23 runs
Benchmark 2: ../../uv/target/profiling/baseline lock
Time (mean ± σ): 140.5 ms ± 5.0 ms [User: 142.8 ms, System: 85.5 ms]
Range (min … max): 133.2 ms … 153.3 ms 21 runs
Summary
../../uv/target/profiling/uv lock ran
1.08 ± 0.04 times faster than ../../uv/target/profiling/baseline lock
```
This is an attempt to solve https://github.com/astral-sh/uv/issues/ by
applying the extra marker of the requirement to overrides and
constraints.
Say in `a` we have a requirements
```
b==1; python_version < "3.10"
c==1; extra == "feature"
```
and overrides
```
b==2; python_version < "3.10"
b==3; python_version >= "3.10"
c==2; python_version < "3.10"
c==3; python_version >= "3.10"
```
Our current strategy is to discard the markers in the original
requirements. This means that on 3.12 for `a` we install `b==3`, but it
also means that we add `c` to `a` without `a[feature]`, causing #4826.
With this PR, the new requirement become,
```
b==2; python_version < "3.10"
b==3; python_version >= "3.10"
c==2; python_version < "3.10" and extra == "feature"
c==3; python_version >= "3.10" and extra == "feature"
```
allowing to override markers while preserving optional dependencies as
such.
Fixes#4826
## Summary
In marker normalization, we now remove any markers that are redundant
with the `requires-python` specifier (i.e., always true for the given
Python requirement).
For example, given `iniconfig ; python_version >= '3.7'`, we can remove
the `python_version >= '3.7'` marker when resolving with
`--python-version 3.8`.
Closes#4852.
## Summary
Given `python_version != '3.8' and python_version < '3.10'`, the first
term was expanded to `python_version < '3.8'` and `python_version >
'3.8'`. We then AND'd all three terms together. We don't seem to have a
way to differentiate between the terms to AND and the terms to OR in the
normalization code (it all gets flattened together), so instead this PR
expands the expressions at the leaf level and then flattens them at the
level above when appropriate.
Closes https://github.com/astral-sh/uv/issues/4910.
## Summary
More marker simplification:
- Filters out redundant subtrees based on outer expressions, e.g. `a and (a or
b)` simplifies to `a`.
- Flattens nested trees internally, e.g. `(a and b) and c`
Resolves https://github.com/astral-sh/uv/issues/4536.
The PR #4707 introduced the notion of "version narrowing," where a
Requires-Python constraint was _possibly_ narrowed whenever the
universal resolver created a fork. The version narrowing would occur
when the fork was a result of a marker expression on `python_version`
that is *stricter* than the configured `Requires-Python` (via, say,
`pyproject.toml`).
The crucial conceptual change made by #4707 is therefore that
`Requires-Python` is no longer an invariant configuration of resolution,
but rather a mutable constraint that can vary from fork to fork. This in
turn can result in some cases, such as in #4885, where different
versions of dependencies are selected. We aren't sure whether we can fix
those or not, with version narrowing, so for now, we do this revert to
restore the previous behavior and we'll try to address the version
narrowing some other time.
This also adds the case from #4885 as a regression test, ensuring that
we don't break that in the future. I confirmed that with version
narrowing, this test outputs duplicate distributions. Without narrowing,
there are no duplicates.
Ref #4707, Fixes#4885
