mirrors/uv
1
0
Fork 0
mirror of https://github.com/astral-sh/uv.git synced 2025-08-04 19:08:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Rewrite resolver docs (#5723)

Browse source 
This PR rewrites the resolver concept and adds a resolver internals page
targeted at power users.

The new resolution concept documentation has three parts:
* An introduction for people how never heard of "resolution" before, and
a motivating example what it does. I've also shoved the part about
equally valid resolutions in there.
* Features you commonly use: Non-universal vs. universal resolution,
lowest resolution amd pre-releases.
* Expert features, we don't advertise them, you'll only need them in
complex cases when you already know and i kept them to the reference
points you need in this case: Constraints, overrides and exclude-newer.

I intentionally didn't lay out any detail of the resolution itself, the
idea is that users get a vague sense of "uv has to select fitting
versions", but then they learn the options they have to use and some
common failure points without ever coming near SAT or even graphs.

The resolution internals reference page is targeted at power users who
need to understand what is going on behind the scenes. It assumes ample
prior knowledge and exists to explain the uv-specific behaviors for
someone who already understands dependency resolution conceptually and
has interacted with their dependency tree before. I had a section on the
lockfile but removed it because it found the lockfile to be too
self-documenting.

I haven't touched the readme.

Closes #5603
Closes #5238
Closes #5237

---------

Co-authored-by: Zanie Blue <contact@zanie.dev>
This commit is contained in:
konsti 2024-08-03 15:47:11 +02:00 • committed by GitHub
parent 44a6dbfa53
commit 6e310f2702
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 219 additions and 76 deletions
Download patch file Download diff file Expand all files Collapse all files

10
BENCHMARKS.md
View file

@ -9,11 +9,11 @@ important caveats:
- Benchmark performance may vary dramatically depending on the set of packages being installed. For
example, a resolution that requires building a single intensive source distribution may appear
very similar across tools, since the bottleneck is tool-agnostic.
- Unlike Poetry, both uv and pip-tools do _not_ generate multi-platform lockfiles. As such, Poetry
is (by design) doing significantly more work than other tools in the resolution benchmarks. Poetry
is included for completeness, as many projects may not _need_ a multi-platform lockfile. However,
it's critical to understand that benchmarking uv's resolution time against Poetry is an unfair
comparison. (Benchmarking installation, however, _is_ a fair comparison.)
- Unlike Poetry, both uv and pip-tools do _not_ generate platform-independent lockfiles. As such,
Poetry is (by design) doing significantly more work than other tools in the resolution benchmarks.
Poetry is included for completeness, as many projects may not _need_ a platform-independent
lockfile. However, it's critical to understand that benchmarking uv's resolution time against
Poetry is an unfair comparison. (Benchmarking installation, however, _is_ a fair comparison.)
This document benchmarks against Trio's `docs-requirements.in`, as a representative example of a
real-world project.

2
README.md
View file

@ -459,7 +459,7 @@ While constraints are purely _additive_, and thus cannot _expand_ the set of acc
a package, overrides _can_ expand the set of acceptable versions for a package, providing an escape
hatch for erroneous upper version bounds.
### Multi-platform resolution
### Platform-independent resolution
By default, uv's `pip-compile` command produces a resolution that's known to be compatible with the
current platform and Python version. Unlike Poetry and PDM, uv does not yet produce a

166
docs/concepts/resolution.md
View file

@ -1,20 +1,88 @@
# Resolution
Dependency resolution is the process of taking your requirements and converting them to a list of
package versions that fulfil your requirements and the requirements of all included packages.
## Overview
Imagine you have the following dependency tree:
- Your project depends on `foo>=1,<3` and `bar>=1,<3`.
- `foo` has two versions, 1.0.0 and 2.0.0. `foo` 2.0.0 depends on `lib==2.0.0`, `foo` 1.0.0 has no
dependencies.
- `bar` has two versions, 1.0.0 and 2.0.0. `bar` 2.0.0 depends on `lib==1.0.0`, `bar` 1.0.0 has no
dependencies.
- `lib` has two versions, 1.0.0 and 2.0.0. Both versions have no dependencies.
We can't install both `foo` 2.0.0 and `bar` 2.0.0 because they conflict on the version of `lib`, so
the resolver will pick either `foo` 1.0.0 or `bar` 1.0.0. Both are valid solutions, at the resolvers
choice.
## Platform-specific and universal resolution
uv supports two modes of resolution: Platform-specific and universal (platform-independent).
Like `pip` and `pip-tools`, `uv pip compile` produces a resolution that's only known to be
compatible with the current operating system, architecture, Python version and Python interpreter.
`uv pip compile --universal` and the [project](../guides/projects.md) interface on the other hand
will solve to a host-agnostic universal resolution that can be used across platforms.
For universal resolution, you need to configure the minimum required python version. For
`uv pip compile --universal`, you can pass `--python-version`, otherwise the current Python version
will be treated as a lower bound. For example, `--universal --python-version 3.9` writes a universal
resolution for Python 3.9 and later. Project commands such as `uv sync` or `uv lock` read
`project.requires-python` from your `pyproject.toml`.
Setting the minimum Python version is important because all package versions we select have to be
compatible with the python range. For example, a universal resolution of `numpy<2` with
`--python-version 3.8` resolves to `numpy==1.24.4`, while `--python-version 3.9` resolves to
`numpy==1.26.4`, as `numpy` releases after 1.26.4 require at least Python 3.9. Note that we only
consider the lower bound of any Python requirement.
In platform-specific mode, the `uv pip` interface also supports resolving for specific alternate
platforms and Python versions with `--python-platform` and `--python-version`. For example, if
you're running Python 3.12 on macOS, but want to resolve for Linux with Python 3.10, you can run
`uv pip compile --python-platform linux --python-version 3.10 requirements.in` to produce a
`manylinux2014`-compatible resolution. In this mode, `--python-version` is the exact python version
to use, not a lower bound.
!!! note
Python's environment markers expose far more information about the current machine
than can be expressed by a simple `--python-platform` argument. For example, the `platform_version` marker
on macOS includes the time at which the kernel was built, which can (in theory) be encoded in
package requirements. uv's resolver makes a best-effort attempt to generate a resolution that is
compatible with any machine running on the target `--python-platform`, which should be sufficient for
most use cases, but may lose fidelity for complex package and platform combinations.
In universal mode, a package may be listed multiple times with different versions or URLs. In this
case, uv determined that we need different versions to be compatible different platforms, and the
markers decides on which platform we use which version. A universal resolution is often more
constrained than a platform-specific resolution, since we need to take the requirements for all
markers into account.
If an output file is used with `uv pip` or `uv.lock` exist with the project commands, we try to
resolve to the versions present there, considering them preferences in the resolution. The same
applies to version already installed to the active virtual environments. You can override this with
`--upgrade`.
## Resolution strategy
By default, uv follows the standard Python dependency resolution strategy of preferring the latest
compatible version of each package. For example, `uv pip install flask>=2.0.0` will install the
latest version of Flask (at time of writing: `3.0.0`).
By default, uv tries to use the latest version of each package. For example,
`uv pip install flask>=2.0.0` will install the latest version of Flask (at time of writing:
`3.0.0`). If you have `flask>=2.0.0` as a dependency of your library, you will only test `flask`
3.0.0 this way, but not if you are actually still compatible with `flask` 2.0.0.
However, uv's resolution strategy can be configured to support alternative workflows. With
`--resolution lowest`, uv will install the **lowest** compatible versions for all dependencies, both
**direct** and **transitive**. Alternatively, `--resolution lowest-direct` will opt for the
**lowest** compatible versions for all **direct** dependencies, while using the **latest**
compatible versions for all **transitive** dependencies. This distinction can be particularly useful
for library authors who wish to test against the lowest supported versions of direct dependencies
without restricting the versions of transitive dependencies.
With `--resolution lowest`, uv will install the lowest possible version for all dependencies, both
direct and indirect (transitive). Alternatively, `--resolution lowest-direct` will opt for the
lowest compatible versions for all direct dependencies, while using the latest compatible versions
for all other dependencies. uv will always use the latest versions for build dependencies.
For example, given the following `requirements.in` file:
For libraries, we recommend separately running tests with `--resolution lowest` or
`--resolution lowest-direct` in continuous integration to ensure compatibility with the declared
lower bounds.
As an example, given the following `requirements.in` file:
```text title="requirements.in"
flask>=2.0.0
@ -64,7 +132,7 @@ werkzeug==2.0.0
By default, uv will accept pre-release versions during dependency resolution in two cases:
1. If the package is a direct dependency, and its version markers include a pre-release specifier
1. If the package is a direct dependency, and its version specifiers include a pre-release specifier
(e.g., `flask>=2.0.0rc1`).
1. If _all_ published versions of a package are pre-releases.
@ -81,70 +149,31 @@ model, and are a frequent source of bugs in other packaging tools. uv's pre-rele
_intentionally_ limited and _intentionally_ requires user opt-in for pre-releases, to ensure
correctness.
For more, see ["Pre-release compatibility"](../pip/compatibility.md#pre-release-compatibility)
For more, see [Pre-release compatibility](../pip/compatibility.md#pre-release-compatibility).
## Dependency overrides
## Constraints
Historically, `pip` has supported "constraints" (`-c constraints.txt`), which allows users to narrow
the set of acceptable versions for a given package.
Like `pip`, uv supports constraints files (`--constraint constraints.txt`), which allows users to
narrow the set of acceptable versions for a given package. A constraint files is like a regular
requirements files, but it doesn't add packages, it only constrains their version range when they
are depended on by a regular requirement.
uv supports constraints, but also takes this concept further by allowing users to _override_ the
acceptable versions of a package across the dependency tree via overrides
(`--override overrides.txt`).
## Overrides
In short, overrides allow the user to lie to the resolver by overriding the declared dependencies of
a package. Overrides are a useful last resort for cases in which the user knows that a dependency is
compatible with a newer version of a package than the package declares, but the package has not yet
been updated to declare that compatibility.
Sometimes, the requirements in one of your (transitive) dependencies are too strict, and you want to
install a version of a package that you know to work, but wouldn't be allowed regularly. Overrides
allow you to lie to the resolver, replacing all other requirements for that package with the
override. They break the usual rules of version resolving and should only be used as last resort
measure.
For example, if a transitive dependency declares `pydantic>=1.0,<2.0`, but the user knows that the
package is compatible with `pydantic>=2.0`, the user can override the declared dependency with
`pydantic>=2.0,<3` to allow the resolver to continue.
While constraints are purely _additive_, and thus cannot _expand_ the set of acceptable versions for
a package, overrides _can_ expand the set of acceptable versions for a package, providing an escape
hatch for erroneous upper version bounds.
## Multi-platform resolution
By default, uv's `pip-compile` command produces a resolution that's known to be compatible with the
current platform and Python version.
uv also supports a machine agnostic resolution. uv supports writing multiplatform resolutions in
both a `requirements.txt` format and uv-specific (`uv.lock`) format.
If using uv's `pip compile`, the `--universal` flag will generate a resolution that is compatible
with all operating systems, architectures, and Python implementations. In universal mode, the
current Python version (or provided `--python-version`) will be treated as a lower bound. For
example, `--universal --python-version 3.7` would produce a universal resolution for Python 3.7 and
later.
If using uv's [project](../guides/projects.md) interface, the machine agnostic resolution will be
used automatically and a `uv.lock` file will be created. The lockfile can also be created with an
explicit `uv lock` invocation.
uv also supports resolving for specific alternate platforms and Python versions via the
`--python-platform` and `--python-version` command line arguments.
For example, if you're running uv on macOS, but want to resolve for Linux, you can run
`uv pip compile --python-platform linux requirements.in` to produce a `manylinux2014`-compatible
resolution.
Similarly, if you're running uv on Python 3.9, but want to resolve for Python 3.8, you can run
`uv pip compile --python-version 3.8 requirements.in` to produce a Python 3.8-compatible resolution.
The `--python-platform` and `--python-version` arguments can be combined to produce a resolution for
a specific platform and Python version, enabling users to generate multiple lockfiles for different
environments from a single machine.
!!! note
Python's environment markers expose far more information about the current machine
than can be expressed by a simple `--python-platform` argument. For example, the `platform_version` marker
on macOS includes the time at which the kernel was built, which can (in theory) be encoded in
package requirements. uv's resolver makes a best-effort attempt to generate a resolution that is
compatible with any machine running on the target `--python-platform`, which should be sufficient for
most use cases, but may lose fidelity for complex package and platform combinations.
Overrides are passed to `uv pip` as `--override` with an overrides file with the same syntax as
requirements or constraints files. In `pyproject.toml`, you can set `tool.uv.override-dependencies`
to a list of requirements. If you provide multiple overrides for the same package, we apply them
simultaneously, while markers are applied as usual.
## Time-restricted reproducible resolutions
@ -155,7 +184,8 @@ may be specified as an [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) t
Note the package index must support the `upload-time` field as specified in
[`PEP 700`](https://peps.python.org/pep-0700/). If the field is not present for a given
distribution, the distribution will be treated as unavailable.
distribution, the distribution will be treated as unavailable. PyPI provides `upload-time` for all
packages.
To ensure reproducibility, messages for unsatisfiable resolutions will not mention that
distributions were excluded due to the `--exclude-newer` flag — newer distributions will be treated

5
docs/index.md
View file

@ -160,9 +160,10 @@ See the [installing Python guide](./guides/install-python.md) to get started.
uv provides a drop-in replacement for common `pip`, `pip-tools`, and `virtualenv` commands. uv
extends their interfaces with advanced features, such as dependency version overrides,
multi-platform resolutions, reproducible resolutions, alternative resolution strategies, and more.
platform-independent resolutions, reproducible resolutions, alternative resolution strategies, and
more.
Compile requirements into a multi-platform requirements file:
Compile requirements into a platform-independent requirements file:
```console
$ uv pip compile docs/requirements.in \

111
docs/reference/resolution-internals.md Normal file
View file

@ -0,0 +1,111 @@
# Resolution internals
This page explains some of the internal workings of uv, its resolver and the lockfile. For using uv,
see [Resolution](../concepts/resolution.md).
## Dependency resolution with PubGrub
If you look into a textbook, it will tell you that finding a set of version to install from a given
set of requirements is equivalent to the
[SAT problem](https://en.wikipedia.org/wiki/Boolean_satisfiability_problem) and thereby NP-complete,
i.e., in the worst case you have to try all possible combinations of all versions of all packages
and there are no general fast algorithms. In practice, this is fairly misleading for a number of
reasons:
- The slowest part of uv is loading package and version metadata, even if it's cached.
- Certain solution are more preferable than others, for example we generally want to use latest
versions.
- Requirements follow lots of patterns: We use continuous versions ranges and not arbitrary boolean
inclusion/exclusions of versions, adjacent release have the same or similar requirements, etc.
- For the majority of resolutions, we wouldn't even need to backtrack, just picking versions
iteratively is sufficient. If we have preferences from a previous resolution we often barely need
to anything at all.
- We don't just need either a solution or a message that there is no solution (like for SAT), we
need an understandable error trace that tell you which packages are involved in away to allows you
to remove the conflict.
uv uses [pubgrub-rs](https://github.com/pubgrub-rs/pubgrub), the Rust implementation of
[PubGrub](https://nex3.medium.com/pubgrub-2fb6470504f), an incremental version solver. PubGrub in uv
works in the following steps:
- We have a partial solution that tells us for which packages we already picked versions and for
which we still need to decide.
- From the undecided packages we pick the one with the highest priority. Package with URLs
(including file, git, etc.) have the highest priority, then those with more exact specifiers (such
as `==`), then those with less strict specifiers. Inside each category, we order packages by when
we first saw them, making the resolution deterministic.
- For that package with the highest priority, pick a version that works with all specifiers from the
packages with versions in the partial solution and that is not yet marked as incompatible. We
prefer versions from a lockfile (`uv.lock` or `-o requirements.txt`) and installed versions, then
we go from highest to lowest (unless you changed the resolution mode). You can see this happening
by the `Selecting ...` messages in `uv lock -v`.
- Add all requirements of this version to pubgrub. Start prefetching their metadata in the
background.
- Now we either we repeat this process with the next package or we have a conflict. Let's say we
pick picked, among other packages, `a` 2 and then `b` 2, and those have requirements `a 2 -> c 1`
and `b 2 -> c 2`. When trying to pick a version for `c`, we see there is no version we can pick.
Using its internal incompatibilities store, PubGrub traces this back to `a 2` and `b 2` and adds
an incompatibility for `{a 2, b 2}`, meaning when either is picked we can't select the other. We
restore the state with `a` 2 before picking `b` 2 with the new learned incompatibility and pick a
new version for `b`.
Eventually, we either have picked compatible versions for all packages and get a successful
resolution, or we get an incompatibility for the virtual root package, that is whatever versions of
the root dependencies and their transitive dependencies we'd pick, we'll always get a conflict. From
the incompatibilities in PubGrub, we can trace which packages were involved and format an error
message. For more details on the PubGrub algorithm, see
[Internals of the PubGrub algorithm](https://pubgrub-rs-guide.pages.dev/internals/intro).
## Forking
Python historically didn't have backtracking version resolution, and even with version resolution,
it was usually limited to single environment, which one specific architecture, operating system,
python version and python implementation. Some packages use contradictory requirements for different
environments, something like:
```text
numpy>=2,<3 ; python_version >= "3.11"
numpy>=1.16,<2 ; python_version < "3.11"
```
Since Python only allows one version package, just version resolution would error here. Inspired by
[poetry](https://github.com/python-poetry/poetry), we instead use forking: Whenever there are
multiple requirements with different for one package name in the requirements of a package, we split
the resolution around these requirements. In this case, we take our partial solution and then once
solve the rest for `python_version >= "3.11"` and once for `python_version < "3.11"`. If some
markers overlap or are missing a part of the marker space, we add additional forks. There can be
more than 2 forks per package and we nest forks. You can see this in the log of `uv lock -v` by
looking for `Splitting resolution on ...`, `Solving split ... (requires-python: ...)` and
`Split ... resolution took ...`.
One problem is that where and how we split is dependent on the order we see packages, which is in
turn dependent on the preference you get e.g. from `uv.lock`. So it can happen that we solve your
requirements with specific forks, write this to the lockfile, and when you call `uv lock` again,
we'd do a different resolution even if nothing changed because the preferences cause us to use
different fork points. To avoid this we write the `environment-markers` of each fork and each
package that diverges between forks to the lockfile. When doing a new resolution, we start with the
forks from the lockfile and use fork-dependent preference (from the `environment-markers` on each
package) to keep the resolution stable. When requirements change, we may introduce new forks from
the saved forks. We also merge forks with identical packages to keep the number of forks low.
## Requires-python
To ensure that a resolution with `requires-python = ">=3.9"` can actually be installed for all those
python versions, uv requires that all dependency support at least that python version. We reject
package versions that declare e.g. `requires-python = ">=3.10"` because we already know that a
resolution with that version can't be installed on Python 3.9, while the user explicitly requested
including 3.9. For simplicity and forward compatibility, we do however only consider lower bounds
for requires-python. If a dependency declares `requires-python = ">=3.8,<4"`, we don't want to
propagate that `<4` marker.
## Wheel tags
While our resolution is universal with respect to requirement markers, this doesn't extend to wheel
tags. Wheel tags can encode Python version, Python interpreter, operating system and architecture,
e.g. `torch-2.4.0-cp312-cp312-manylinux2014_aarch64.whl` is only compatible with CPython 3.12 on
arm64 Linux with glibc >= 2.17 (the manylinux2014 policy), while `tqdm-4.66.4-py3-none-any.whl`
works with all Python 3 versions and interpreters on any operating system and architecture. Most
projects have a (universally compatible) source distribution we can fall back to when we try to
install a package version and there is no compatible wheel, but some, such as `torch`, don't have a
source distribution. In this case an installation on e.g. Python 3.13 or an uncommon operating
system or architecture will fail with a message about a missing matching wheel.

1
mkdocs.template.yml
View file

@ -122,6 +122,7 @@ nav:
- Reference:
- Commands: reference/cli.md
- Settings: reference/settings.md
- Resolution Internals: reference/resolution-internals.md
- Policies:
- Versioning: versioning.md
- Platform support: platforms.md