ffd18cc75d
## Summary This PR rewrites the `MarkerTree` type to use algebraic decision diagrams (ADD). This has many benefits: - The diagram is canonical for a given marker function. It is impossible to create two functionally equivalent marker trees that don't refer to the same underlying ADD. This also means that any trivially true or unsatisfiable markers are represented by the same constants. - The diagram can handle complex operations (conjunction/disjunction) in polynomial time, as well as constant-time negation. - The diagram can be converted to a simplified DNF form for user-facing output. The new representation gives us a lot more confidence in our marker operations and simplification, which is proving to be very important (see https://github.com/astral-sh/uv/pull/5733 and https://github.com/astral-sh/uv/pull/5163). Unfortunately, it is not easy to split this PR into multiple commits because it is a large rewrite of the `marker` module. I'd suggest reading through the `marker/algebra.rs`, `marker/simplify.rs`, and `marker/tree.rs` files for the new implementation, as well as the updated snapshots to verify how the new simplification rules work in practice. However, a few other things were changed: - [We now use release-only comparisons for `python_full_version`, where we previously only did for `python_version`](https://github.com/astral-sh/uv/blob/ibraheem/canonical-markers/crates/pep508-rs/src/marker/algebra.rs#L522). I'm unsure how marker operations should work in the presence of pre-release versions if we decide that this is incorrect. - [Meaningless marker expressions are now ignored](https://github.com/astral-sh/uv/blob/ibraheem/canonical-markers/crates/pep508-rs/src/marker/parse.rs#L502). This means that a marker such as `'x' == 'x'` will always evaluate to `true` (as if the expression did not exist), whereas we previously treated this as always `false`. It's negation however, remains `false`. - [Unsatisfiable markers are written as `python_version < '0'`](https://github.com/astral-sh/uv/blob/ibraheem/canonical-markers/crates/pep508-rs/src/marker/tree.rs#L1329). - The `PubGrubSpecifier` type has been moved to the new `uv-pubgrub` crate, shared by `pep508-rs` and `uv-resolver`. `pep508-rs` also depends on the `pubgrub` crate for the `Range` type, we probably want to move `pubgrub::Range` into a separate crate to break this, but I don't think that should block this PR (cc @konstin). There is still some remaining work here that I decided to leave for now for the sake of unblocking some of the related work on the resolver. - We still use `Option<MarkerTree>` throughout uv, which is unnecessary now that `MarkerTree::TRUE` is canonical. - The `MarkerTree` type is now interned globally and can potentially implement `Copy`. However, it's unclear if we want to add more information to marker trees that would make it `!Copy`. For example, we may wish to attach extra and requires-python environment information to avoid simplifying after construction. - We don't currently combine `python_full_version` and `python_version` markers. - I also have not spent too much time investigating performance and there is probably some low-hanging fruit. Many of the test cases I did run actually saw large performance improvements due to the markers being simplified internally, reducing the stress on the old `normalize` routine, especially for the extremely large markers seen in `transformers` and other projects. Resolves https://github.com/astral-sh/uv/issues/5660, https://github.com/astral-sh/uv/issues/5179. |
||
|---|---|---|
| .. | ||
| bench | ||
| cache-key | ||
| distribution-filename | ||
| distribution-types | ||
| install-wheel-rs | ||
| once-map | ||
| pep440-rs | ||
| pep508-rs | ||
| platform-tags | ||
| pypi-types | ||
| requirements-txt | ||
| uv | ||
| uv-auth | ||
| uv-build | ||
| uv-cache | ||
| uv-cli | ||
| uv-client | ||
| uv-configuration | ||
| uv-dev | ||
| uv-dispatch | ||
| uv-distribution | ||
| uv-extract | ||
| uv-fs | ||
| uv-git | ||
| uv-installer | ||
| uv-macros | ||
| uv-normalize | ||
| uv-options-metadata | ||
| uv-pubgrub | ||
| uv-python | ||
| uv-requirements | ||
| uv-resolver | ||
| uv-scripts | ||
| uv-settings | ||
| uv-shell | ||
| uv-state | ||
| uv-tool | ||
| uv-trampoline | ||
| uv-types | ||
| uv-version | ||
| uv-virtualenv | ||
| uv-warnings | ||
| uv-workspace | ||
| README.md | ||
Crates
bench
Functionality for benchmarking uv.
cache-key
Generic functionality for caching paths, URLs, and other resources across platforms.
distribution-filename
Parse built distribution (wheel) and source distribution (sdist) filenames to extract structured metadata.
distribution-types
Abstractions for representing built distributions (wheels) and source distributions (sdists), and the sources from which they can be downloaded.
install-wheel-rs
Install built distributions (wheels) into a virtual environment.]
once-map
A waitmap-like concurrent hash map for executing tasks
exactly once.
pep440-rs
Utilities for interacting with Python version numbers and specifiers.
pep508-rs
Utilities for interacting with PEP 508 dependency specifiers.
platform-host
Functionality for detecting the current platform (operating system, architecture, etc.).
platform-tags
Functionality for parsing and inferring Python platform tags as per PEP 425.
uv
Command-line interface for the uv package manager.
uv-build
A PEP 517-compatible build frontend for uv.
uv-cache
Functionality for caching Python packages and associated metadata.
uv-client
Client for interacting with PyPI-compatible HTTP APIs.
uv-dev
Development utilities for uv.
uv-dispatch
A centralized struct for resolving and building source distributions in isolated environments.
Implements the traits defined in uv-types.
uv-distribution
Client for interacting with built distributions (wheels) and source distributions (sdists). Capable of fetching metadata, distribution contents, etc.
uv-extract
Utilities for extracting files from archives.
uv-fs
Utilities for interacting with the filesystem.
uv-git
Functionality for interacting with Git repositories.
uv-installer
Functionality for installing Python packages into a virtual environment.
uv-python
Functionality for detecting and leveraging the current Python interpreter.
uv-normalize
Normalize package and extra names as per Python specifications.
uv-package
Types and functionality for working with Python packages, e.g., parsing wheel files.
uv-requirements
Utilities for reading package requirements from pyproject.toml and requirements.txt files.
uv-resolver
Functionality for resolving Python packages and their dependencies.
uv-shell
Utilities for detecting and manipulating shell environments.
uv-types
Shared traits for uv, to avoid circular dependencies.
pypi-types
General-purpose type definitions for types used in PyPI-compatible APIs.
uv-virtualenv
A venv replacement to create virtual environments in Rust.
uv-warnings
User-facing warnings for uv.
uv-workspace
Workspace abstractions for uv.
requirements-txt
Functionality for parsing requirements.txt files.