## Summary
We weren't including these in the cache key when constructing the
install plan. We likely still read them from the cache later, but we may
have reported the wrong number of prepares, etc.
If a user specifies `-e /path/to/dir` and `/path/to/dir` in a `uv pip
install` command, we want the editable to "win" (rather than erroring
due to conflicting URLs). Unfortunately, this behavior meant that when
you requested a package as editable and non-editable in conflicting
groups, the editable version was _always_ used. This PR modifies the
requisite types to use `Option<bool>` rather than `bool` for the
`editable` field, so we can determine whether a requirement was
explicitly requested as editable, explicitly requested as non-editable,
or not specified (as in the case of `/path/to/dir` in a
`requirements.txt` file). In the latter case, we allow editables to
override the "unspecified" requirement.
If a project includes a path dependency twice, once with `editable =
true` and once without any `editable` annotation, those are now
considered conflicting URLs, and lead to an error, so I've marked this
change as breaking.
Closes https://github.com/astral-sh/uv/issues/14139.
## Summary
When refactoring the addition PR I accidentally introduced a bug where
the debug message would not be output if the default value is used.
cc @zanieb
## Summary
When installing packages on _very_ slow/overloaded systems it'spossible
to trigger bytecode compilation timeouts, which tends to happen in
environments such as Qemu (especially without KVM/virtio), but also on
systems that are simply overloaded. I've seen this in my Nix builds if I
for example am compiling a Linux kernel at the same time as a few other
concurrent builds.
By making the bytecode compilation timeout adjustable you can work
around such issues. I plan to set `UV_COMPILE_BYTECODE_TIMEOUT=0` in the
[pyproject.nix
builders](https://pyproject-nix.github.io/pyproject.nix/build.html) to
make them more reliable.
- Related issues
* https://github.com/astral-sh/uv/issues/6105
## Test Plan
Only manual testing was applied in this instance. There is no existing
automated tests for bytecode compilation timeout afaict.
Prior to this PR, there were numerous places where uv would leak
credentials in logs. We had a way to mask credentials by calling methods
or a recently-added `redact_url` function, but this was not secure by
default. There were a number of other types (like `GitUrl`) that would
leak credentials on display.
This PR adds a `DisplaySafeUrl` newtype to prevent leaking credentials
when logging by default. It takes a maximalist approach, replacing the
use of `Url` almost everywhere. This includes when first parsing config
files, when storing URLs in types like `GitUrl`, and also when storing
URLs in types that in practice will never contain credentials (like
`DirectorySourceUrl`). The idea is to make it easy for developers to do
the right thing and for the compiler to support this (and to minimize
ever having to manually convert back and forth). Displaying credentials
now requires an active step. Note that despite this maximalist approach,
the use of the newtype should be zero cost.
One conspicuous place this PR does not use `DisplaySafeUrl` is in the
`uv-auth` crate. That would require new clones since there are calls to
`request.url()` that return a `&Url`. One option would have been to make
`DisplaySafeUrl` wrap a `Cow`, but this would lead to lifetime
annotations all over the codebase. I've created a separate PR based on
this one (#13576) that updates `uv-auth` to use `DisplaySafeUrl` with
one new clone. We can discuss the tradeoffs there.
Most of this PR just replaces `Url` with `DisplaySafeUrl`. The core is
`uv_redacted/lib.rs`, where the newtype is implemented. To make it
easier to review the rest, here are some points of note:
* `DisplaySafeUrl` has a `Display` implementation that masks
credentials. Currently, it will still display the username when there is
both a username and password. If we think is the wrong choice, it can
now be changed in one place.
* `DisplaySafeUrl` has a `remove_credentials()` method and also a
`.to_string_with_credentials()` method. This allows us to use it in a
variety of scenarios.
* `IndexUrl::redacted()` was renamed to
`IndexUrl::removed_credentials()` to make it clearer that we are not
masking.
* We convert from a `DisplaySafeUrl` to a `Url` when calling `reqwest`
methods like `.get()` and `.head()`.
* We convert from a `DisplaySafeUrl` to a `Url` when creating a
`uv_auth::Index`. That is because, as mentioned above, I will be
updating the `uv_auth` crate to use this newtype in a separate PR.
* A number of tests (e.g., in `pip_install.rs`) that formerly used
filters to mask tokens in the test output no longer need those filters
since tokens in URLs are now masked automatically.
* The one place we are still knowingly writing credentials to
`pyproject.toml` is when a URL with credentials is passed to `uv add`
with `--raw`. Since displaying credentials is no longer automatic, I
have added a `to_string_with_credentials()` method to the `Pep508Url`
trait. This is used when `--raw` is passed. Adding it to that trait is a
bit weird, but it's the simplest way to achieve the goal. I'm open to
suggestions on how to improve this, but note that because of the way
we're using generic bounds, it's not as simple as just creating a
separate trait for that method.
Rustfmt introduces a lot of formatting changes in the 2024 edition. To
not break everything all at once, we split out the set of formatting
changes compatible with both the 2021 and 2024 edition by first
formatting with the 2024 style, and then again with the currently used
2021 style.
Notable changes are the formatting of derive macro attributes and lines
with overly long strings and adding trailing semicolons after statements
consistently.
## Summary
I don't know if I actually want to commit this, but I did it on the
plane last time and just polished it off (got it to compile) while
waiting to board.
## Summary
This crate is for standards-compliant types, but this is explicitly a
type that's custom to uv. It's also strange because we kind of want to
reference `IndexUrl` on the registry type, but that's in a crate that
_depends_ on `uv-pypi-types`, which to me is a sign that this is off.
## Summary
I think this is reasonable to change. Right now, if you're on Python
3.11, the resolver returns `multiprocess-0.70.17-py311-none-any.whl`,
but `multiprocess-0.70.17-py310-none-any.whl` is in the cache, we'll
reuse `multiprocess-0.70.17-py310-none-any.whl` (since it _is_
compatible with Python 3.11).
Instead, we now _require_ the cached wheel to match the wheel returned
by the resolver.
Closes https://github.com/astral-sh/uv/issues/12273.
## Summary
This ended up being more involved than expected. The gist is that we
setup all the packages we want to reinstall upfront (they're passed in
on the command-line); but at that point, we don't have names for all the
packages that the user has specified. (Consider, e.g., `uv pip install
.` -- we don't have a name for `.`, so we can't add it to the list of
`Reinstall` packages.)
Now, `Reinstall` also accepts paths, so we can augment `Reinstall` based
on the user-provided paths.
Closes#12038.
## Summary
This PR adds support for `SitePackages::satisfies` with unnamed
overrides and requirements.
The main challenge here was cases like: you have a `requirements.in`
with `git+https://github.com/pallets/flask` in it, and an
`overrides.txt` with `flask==2.0.0` in it. You _need_ to include
`flask==2.0.0`, but you can't know that without resolving the unnamed
URL requirement (since overrides only take effect when the package is
included, like constraints).
We now make the assumption that any unnamed overrides _are_ relevant,
for the purpose of the satisfies check. This is conservative, but this
whole check is an optimization anyway.
## Summary
Small omission I noticed last night. This was overly strict (so, didn't
lead to any incorrect behavior; more that we did unnecessary work in
some cases).
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Closes#2410
<!-- What's the purpose of the change? What does it do, and why? -->
This changes the name of files in `wheels` bucket to use a hash instead
of the wheel name as to not exceed maximum file length limit on various
systems.
This only addresses the primary concern of #2410. It still does _not_
address:
- Path limit of 260 on windows:
https://github.com/astral-sh/uv/issues/2410#issuecomment-2062020882
To solve this we need to opt-in to longer path limits on windows
([ref](https://github.com/astral-sh/uv/issues/2410#issuecomment-2150532658)),
but I think that is a separate issue and should be a separate MR.
- Exceeding filename limit while building a wheel from source
distribution
As per my understanding, this is out of uv's control. Name of the output
wheel will be decided by build-backend used by the project. For wheels
built from source distribution, pip also uses the wheel names in cache.
So I have not touched `sdists` cache.
I have added a `filename: WheelFileName` field in `Archive`, so we can
use it while indexing instead of relying on the filename on disk.
Another way to do this was to read `.dist-info/WHEEL` and
`.dist-info/METADATA` and build `WheelFileName` but that seems less
robust and will be slower.
## Test Plan
<!-- How was it tested? -->
Tested by installing `yt-dlp`, `httpie` and `sqlalchemy` and verifying
that cache files in `wheels` bucket use hash.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
Solving spent a chunk of its time just converting resolutions, the left
two blocks:
These blocks are `ResolverOutput::from_state` with 1.3% and
`ForkState::into_resolution` with 4.1% of resolver thread runtime for
apache airflow universal.
We reduce the overhead spent in those functions, to now 1.1% and 2.1% of
resolver time spend in those functions by:
Commit 1: Replace the hash set for the edges with a vec in
`ForkState::into_resolution`. We deduplicate edges anyway when
collecting them, and the hash-and-insert was slow.
Commit 2: Reduce the distribution clonign in
`ResolverOutput::from_state` by using an `Arc`.
The same profile excerpt for the resolver with the branch (note that
there is now an unrelated block between the two we optimized):
Wall times are noisy, but the profiles show those changes as
improvements.
```
$ hyperfine --warmup 2 "./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal" "./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal"
Benchmark 1: ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 99.1 ms ± 3.8 ms [User: 111.8 ms, System: 115.5 ms]
Range (min … max): 93.6 ms … 110.4 ms 29 runs
Benchmark 2: ./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 97.1 ms ± 4.3 ms [User: 114.8 ms, System: 112.0 ms]
Range (min … max): 90.9 ms … 112.4 ms 29 runs
Summary
./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal ran
1.02 ± 0.06 times faster than ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
```
Instead of always using all available threads for bytecode compilation,
respect `UV_CONCURRENT_INSTALLS`, so the parallelism is configurable
instead of hardcoded. We reuse the install limit since bytecode
compilation only runs after install.
Initially, we were limiting Git schemes to HTTPS and SSH as only
supported schemes. We lost this validation in #3429. This incidentally
allowed file schemes, which apparently work with Git out of the box.
A caveat for this is that in tool.uv.sources, we parse the git field
always as URL. This caused a problem with #11425: repo = { git =
'c:\path\to\repo', rev = "xxxxx" } was parsed as a URL where c: is the
scheme, causing a bad error message down the line.
This PR:
* Puts Git URL validation back in place. It bans everything but HTTPS,
SSH, and file URLs. This could be a breaking change, if users were using
a git transport protocol were not aware of, even though never
intentionally supported.
* Allows file: URL in Git: This seems to be supported by Git and we were
supporting it albeit unintentionally, so it's reasonable to continue to
support it.
* It does not allow relative paths in the git field in tool.uv.sources.
Absolute file URLs are supported, whether we want relative file URLs for
Git too should be discussed separately.
Closes#3429: We reject the input with a proper error message, while
hinting the user towards file:. If there's still desire for relative
path support, we can keep it open.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
We want to build `uv-build` without depending on the network crates. In
preparation for that, we split uv-git into uv-git and uv-git-types,
where only uv-git depends on reqwest, so that uv-build can use
uv-git-types.
## Summary
We currently enforce that if you do `uv pip install
./dist/iniconfig-1.0.0.tar.gz`, the build _must_ produce a wheel like
`iniconfig-1.0.0-py3-none-any.whl` (i.e., the name and version must
match). It turns out some packages produce a wheel that has a local
suffix on it, like `vllm`. This PR makes the check a little more
permissive in that we now accept `1.0.0` or that version with a local
suffix (e.g., `1.0.0+cpu`). I don't love this practice, but we already
relaxed this check when _installing_ a wheel, so this seems reasonable:
5e15881dcc/crates/uv-install-wheel/src/install.rs (L50-L52)
Note that this is _still_ stricter than pip. pip seems to only require
that the package name is the same (i.e., `iniconfig` matches
`iniconfig`; but they'll happily install a wheel like
`iniconfig-2.0.0-py3-none-any.whl` given
`./dist/iniconfig-1.0.0.tar.gz`).
Closes https://github.com/astral-sh/uv/issues/11038.
uv-install-wheel had the logic for laying out the installation and for
linking a directory in the same module. We split them up to isolate each
module's logic and tighten the crate's interface to only expose top
level members.
No logic changes, only moving code around.
Log the file that failed to bytecode compile when encountering a timeout
for debugging #6105 better.
[sysinfo](https://lib.rs/crates/sysinfo) would give us the option to
report memory usage too, but i'm hesitant to add a dependency just for
the error path.
## Summary
When `--upgrade` is provided, we should retain already-installed
packages _if_ they're newer than whatever is available from the
registry.
Closes https://github.com/astral-sh/uv/issues/10089.
## Summary
Sort of undecided on this. These are already stored as `dyn Reporter` in
each struct, so we're already using dynamic dispatch in that sense. But
all the methods take `impl Reporter`. This is sometimes nice (the
callsites are simpler?), but it also means that in practice, you often
_can't_ pass `None` to these methods that accept `Option<impl
Reporter>`, because Rust can't infer the generic type.
Anyway, this adds more consistency and simplifies the setup by using
`Arc<dyn Reporter>` everywhere.
Build failures are one of the most common user facing failures that
aren't "obivous" errors (such as typos) or resolver errors. Currently,
they show more technical details than being focussed on this being an
error in a subprocess that is either on the side of the package or -
more likely - in the build environment, e.g. the user needs to install a
dev package or their python version is incompatible.
The new error message clearly delineates the part that's important (this
is a build backend problem) from the internals (we called this hook) and
is consistent about which part of the dist building stage failed. We
have to calibrate the exact wording of the error message some more. Most
of the implementation is working around the orphan rule, (this)error
rules and trait rules, so it came out more of a refactoring than
intended.
Example:
The resolver methods are already too large and complex, especially
`choose_version*`, so i wanted to shrink and simplify them a bit before
adding new methods to them.
I've split `MetadataResponse` into three variants: success, non-fatal
error (reported through pubgrub), fatal error (reported as error trace).
The resulting non-fatal `MetadataUnavailable` type is equivalent to the
`IncompletePackage` type, so they are now merged. (`UnavailableVersion`
is a bit different since, besides the extra `IncompatibleDist` variant,
it have no error source attached). This shows that the missing metadata
variant was unused, which I removed.
Tagging as error messages for the logging format changes.
Instead of modifying the error to replace a dummy derivation chain from
construction with the real one, build the error with the real derivation
chain directly.
This came up when trying to improve the build error reporting.
Introduces `DistErrorKind` to avoid error variants for each case that
are only different in one line of the message.
## Summary
This change introduces the `UV_NO_INSTALLER_METADATA` environment
variable
as a way to opt out of the extra installer metadata files that `uv` is
creating.
This is important to achieve reproducible builds in distribution
packaging, allowing to replace usage of
[installer](https://pypi.org/project/installer) with `uv pip install`.
At the time of writing these files are:
- `uv_cache.json`
Contains timestamps which are non-reproducible.
These hashes also leak in to the `RECORD` file.
- `direct_url.json`
Contains the path to the installed wheel.
While not non-reproducible it's not required for distribution packaging.
- `INSTALLER`
Again, not non-reproducible, but of no value in distribution packaging.
## Test Plan
Automated test added.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
When performing a noop sync, we don't need the rayon threadpool, yet we
pay for its initialization:
Be making the initialization lazy, we avoid that cost:
This code runs every time before user code in `uv run`.
This means that before calling rayon, one now needs to call
`LazyLock::force(&RAYON_INITIALIZE);`.
Performance mode (CPU 0 is a perf core):
```
$ taskset -c 0 hyperfine --warmup 5 -N "/home/konsti/projects/uv/uv-main sync" "/home/konsti/projects/uv/target/profiling/uv sync"
Benchmark 1: /home/konsti/projects/uv/uv-main sync
Time (mean ± σ): 4.5 ms ± 0.1 ms [User: 2.7 ms, System: 1.8 ms]
Range (min … max): 4.4 ms … 6.4 ms 640 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Benchmark 2: /home/konsti/projects/uv/target/profiling/uv sync
Time (mean ± σ): 4.4 ms ± 0.1 ms [User: 2.7 ms, System: 1.6 ms]
Range (min … max): 4.3 ms … 5.0 ms 679 runs
Summary
/home/konsti/projects/uv/target/profiling/uv sync ran
1.03 ± 0.04 times faster than /home/konsti/projects/uv/uv-main sync
```
Power saver mode:
```
$ hyperfine --warmup 5 -N "/home/konsti/projects/uv/uv-main sync" "/home/konsti/projects/uv/target/profiling/uv sync"
Benchmark 1: /home/konsti/projects/uv/uv-main sync
Time (mean ± σ): 28.1 ms ± 1.2 ms [User: 15.5 ms, System: 20.3 ms]
Range (min … max): 25.7 ms … 31.9 ms 102 runs
Benchmark 2: /home/konsti/projects/uv/target/profiling/uv sync
Time (mean ± σ): 24.0 ms ± 1.2 ms [User: 13.8 ms, System: 9.9 ms]
Range (min … max): 22.2 ms … 28.2 ms 122 runs
Summary
/home/konsti/projects/uv/target/profiling/uv sync ran
1.17 ± 0.08 times faster than /home/konsti/projects/uv/uv-main sync
```
## Summary
This PR adds context to our error messages to explain _why_ a given
package was included, if we fail to download or build it.
It's quite a large change, but it motivated some good refactors and
improvements along the way.
Closes https://github.com/astral-sh/uv/issues/8962.
## Summary
I need this for the derivation chain work
(https://github.com/astral-sh/uv/issues/8962), but it just seems
generally useful. You can't always get a version from a `Dist` (it could
be URL-based!), but when we create a `ResolvedDist`, we _do_ know the
version (and not just the URL). This PR preserves it.
## Summary
Shows similar diagnostics for failures that happen at install time,
rather than resolve time. This will ultimately feed into
https://github.com/astral-sh/uv/issues/8962 since we'll now have
consolidated handling for these kinds of failures.
