## Summary
We now initialize an HTTP client in advance for remote requirements
files. It turns out this adds a significant overhead, even for
operations like auditing the environment (at least on macOS).
This PR makes initialization lazy. After a lot of evaluation, I took the
easiest route, which is: we just pass in `Connectivity`, and then use
the default HTTP client. So we won't respect netrc files and anything
else that we get from our registry client. If we want to keep using the
registry client, we _can_, it's just way more ceremony to pass down a
closure.
See: https://github.com/astral-sh/uv/issues/2346.
## Test Plan
- Verified that `cargo run pip compile
https://raw.githubusercontent.com/ansible/ansible/f1ded0f41759235eb15a7d13dbc3c95dce5d5acd/requirements.txt`
completed without error.
- Verified that `cargo run pip compile
https://raw.githubusercontent.com/ansible/ansible/f1ded0f41759235eb15a7d13dbc3c95dce5d5acd/requirements.txt
--offline` failed with an error.
- Verified that `./target/release/uv pip install requests` completed in
0-2ms, rather than hundreds.
## Summary
In #2000, I shipped a regression whereby we stopped writing relative
paths for scripts within `data` directories. The net effect here is that
we aren't _uninstalling_ binaries in all cases. (This does _not_ apply
to entrypoints, only scripts in `data` directories.)
Closes https://github.com/astral-sh/uv/issues/2330.
## Test Plan
Most Python packages ship entrypoints, not binaries, so I don't know how
to test this cheaply. But I did test it locally by verifying that `uv`
is now removed from the `bin` directory after an uninstall.
## Summary
Some zip files can't be streamed; in particular, `rs-async-zip` doesn't
support data descriptors right now (though it may in the future). This
PR adds a fallback path for such zips that downloads the entire zip file
to disk, then unzips it from disk (which gives us `Seek`).
Closes https://github.com/astral-sh/uv/issues/2216.
## Test Plan
`cargo run pip install --extra-index-url https://buf.build/gen/python
hashb_foxglove_protocolbuffers_python==25.3.0.1.20240226043130+465630478360
--force-reinstall -n`
## Summary
The netrc middleware we added in
https://github.com/astral-sh/uv/pull/2241 has a slight problem. If you
include credentials in your index URL, _and_ in the netrc file, the
crate blindly adds the netrc credentials as a header. And given the
`ReqwestBuilder` API, this means you end up with _two_ `Authorization`
headers, which always leads to an invalid request, though the exact
failure can take different forms.
This PR removes the middleware crate in favor of our own middleware.
Instead of using the `RequestInitialiser` API, we have to use the
`Middleware` API, so that we can remove the header on the request
itself.
Closes https://github.com/astral-sh/uv/issues/2323.
## Test Plan
- Verified that running against a private index with credentials in the
URL (but no netrc file) worked without error.
- Verified that running against a private index with credentials in the
netrc file (but not the URL) worked without error.
- Verified that running against a private index with a mix of
credentials in both _also_ worked without error.
## Summary
We write this a few lines down with a value passed in by the caller. I
suspect I missed that this was already here (with a less accurate value)
when adding `INSTALLER`.
`packaging==24.0` came out which broke this test. It has to run without
`--exclude-newer` since it's testing an index that doesn't support it.
Instead, though, we can just disable dependencies, since the test still
exercises the same logic.
Preparing for #2058, i found it hard to follow where which discovery
function gets called. I moved all the discovery functions to a
`find_python` module (some exposed through `PythonEnvironment`) and
documented which subcommand uses which python discovery strategy.
No functional changes.
`install_extra_index_url_has_priority` started failing because
`packaging` had a new release. I'm not sure if this preserves the index
order check as intended, but it does unblock CI.
## Summary
No behavioral changes; just taking code that's duplicated between two
branches in `distribution_database.rs` and pulling it into its own
method.
## Summary
Addressing the extremely slow performance detailed in
https://github.com/astral-sh/uv/issues/2220. There are two changes to
increase download performance:
1. setting `accept-encoding: identity`, in the spirit of
https://github.com/pypa/pip/pull/1688
2. increasing buffer from 8KiB to 128KiB.
### 1. accept-encoding: identity
I think this related `pip` PR has a good explanation of what's going on:
https://github.com/pypa/pip/pull/1688
```
# We use Accept-Encoding: identity here because requests
# defaults to accepting compressed responses. This breaks in
# a variety of ways depending on how the server is configured.
# - Some servers will notice that the file isn't a compressible
# file and will leave the file alone and with an empty
# Content-Encoding
# - Some servers will notice that the file is already
# compressed and will leave the file alone and will add a
# Content-Encoding: gzip header
# - Some servers won't notice anything at all and will take
# a file that's already been compressed and compress it again
# and set the Content-Encoding: gzip header
```
The `files.pythonhosted.org` server is the 1st kind. Example debug log I
added in `uv` when installing against PyPI:
<img width="1459" alt="image"
src="ef10d758-46aa-4c8e-9dba-47f33437401b">
(there is no `content-encoding` header in this response, the `whl`
hasn't been compressed, and there is a content-length header)
Our internal mirror is the third case. It does seem sensible that our
mirror should be modified to act like the 1st kind. But `uv` should
handle all three cases like `pip` does.
### 2. buffer increase
In https://github.com/astral-sh/uv/issues/2220 I observed that `pip`'s
downloading was causing up-to 128KiB flushes in our mirror.
After fix 1, `uv` was still only causing up-to 8KiB flushes, and was
slower to download than `pip`. Increasing this buffer from the default
8KiB led to a download performance improvement against our mirror and
the expected observed 128KiB flushes.
## Test Plan
Ran benchmarking as instructed by @charliermarsh
<img width="1447" alt="image"
src="840d9c8d-4b98-4bfa-89f3-073a2dec1f23">
No performance improvement or regression.
## Summary
In #1813, we were failing to install `scikit-image==0.19.3` from source
in Python 3.11. Confusingly, though, the trace showed that the build
command exited with status 0...
The issue is that we get results from the PEP 517 hooks by reading from
`stdout` -- that is, we `print` at the end of the script, and parse the
printed output on the other side.
It turns out that for `scikit-image`, in this case, there was output
_after_ the wheel filename:
```
...
no previously-included directories found matching 'doc/gh-pages'
adding license file 'LICENSE.txt'
writing manifest file 'scikit_image.egg-info/SOURCES.txt'
Copying scikit_image.egg-info to build/bdist.macosx-12.6-arm64/wheel/scikit_image-0.19.3-py3.11.egg-info
running install_scripts
scikit_image-0.19.3-cp311-cp311-macosx_14_0_arm64.whl
INFO:
########### EXT COMPILER OPTIMIZATION ###########
INFO: Platform :
Architecture: aarch64
Compiler : clang
CPU baseline :
Requested : 'min'
Enabled : NEON NEON_FP16 NEON_VFPV4 ASIMD
Flags : none
Extra checks: none
CPU dispatch :
Requested : 'max -xop -fma4'
Enabled : ASIMDHP ASIMDDP ASIMDFHM
Generated : none
INFO: CCompilerOpt.cache_flush[864] : write cache to path -> /private/var/folders/nt/6gf2v7_s3k13zq_t3944rwz40000gn/T/.tmp5ZPIbv/built-wheels-v0/pypi/scikit-image/0.19.3/hLW_f7wWeGDOPRlSazQXw/scikit-image-0.19.3.tar.gz/build/temp.macosx-12.6-arm64-3.11/ccompiler_opt_cache_ext.py
```
We need the `scikit_image-0.19.3-cp311-cp311-macosx_14_0_arm64.whl`
line, but we were failing to find it due to all the extra output at the
end (presumedly, some kind of `atexit` logging).
This PR modifies the hooks to instead write their results to files that
are passed in by the parent. On the other end, we then read the results
back from disk. This makes it much more robust to "other" output in the
script.
Closes https://github.com/astral-sh/uv/issues/1813.
## Test Plan
Ran `cargo run pip install scikit-image==0.19.3 --reinstall
--no-cache-dir` on Python 3.11.
## Summary
It turns out that setuptools includes a shim to patch distutils. I'll
admit that I don't fully understand why or how it's different, but this
is the trick `pip` uses to ensure that it gets the "original" distutils.
We actually use distutils in two places: once for the system Python
scheme, and once for virtual environments. In virtualenv, they _do_ use
the patched distutils, so this could deviate in ways I don't understand.
Closes#2302.
Extends the "compatibility" types introduced in #1293 to apply to source
distributions as well as wheels.
- We now track the most-relevant incompatible source distribution
- Exclude newer, Python requirements, and yanked versions are all
tracked as incompatibilities in the new model (this lets us remove
`DistMetadata`!)
Since Python 3.7, deterministic pycs are possible (see [PEP
552](https://peps.python.org/pep-0552/))
To select the bytecode invalidation mode explicitly by env var:
PYC_INVALIDATION_MODE=UNCHECKED_HASH uv pip install --compile ...
Valid values are TIMESTAMP (default), CHECKED_HASH, and UNCHECKED_HASH.
The latter options are useful for reproducible builds.
---------
Co-authored-by: konstin <konstin@mailbox.org>
## Summary
PyPI now supports Metadata 2.2, which means distributions with Metadata
2.2-compliant metadata will start to appear. The upside is that if a
source distribution includes a `PKG-INFO` file with (1) a metadata
version of 2.2 or greater, and (2) no dynamic fields (at least, of the
fields we rely on), we can read the metadata from the `PKG-INFO` file
directly rather than running _any_ of the PEP 517 build hooks.
Closes https://github.com/astral-sh/uv/issues/2009.
This PR tweaks uv to support reading `requirements.txt` regardless of
whether it is encoded as UTF-8 or UTF-16. This is particularly relevant
on Windows where `uv pip freeze > requirements.txt` will likely write a
UTF-16 encoded `requirements.txt` file.
There is some discussion on #1666 where it's suggested that perhaps
we should explicitly not support this. I didn't see that until I
had already put this PR together, but even so, I think it's worth
considering this. UTF-16 is predominant on Windows. It is very easy
to produce a UTF-16 encoded file. Moreover, there is an easy and well
specified way to recognize and transcode UTF-16 encoded data to UTF-8.
I think the downside of this is that it could encourage the use UTF-16
encoded `requirements.txt` files *in addition* to UTF-8 encoded
files, and it would probably be nice to converge and standardize on
one encoding. One possible alternative to this PR is that we provide
a better error message. Another alternative is to ensure that a
`-o/--output` flag exists for all commands (neither `uv pip freeze` nor
`pip freeze` have such a flag) so that users can always write output
to a file without relying on their environment's piping behavior.
(Although this last alternative seems a little sad to me.)
It's also worth noting the [PEP-0508] doesn't seem to mention file
encoding at all. So I think from a "do the standards allow this"
perspective, this change is OK.
Finally, `pip` itself seems to work with UTF-16 encoded
`requirements.txt` files.
I think I personally overall lean towards supporting UTF-16 for
`requirements.txt` files. In part because I think it smoothes out the
UX a little bit, in part because there is no obvious specification
(that I'm aware of) that mandates that these files are UTF-8, and
finally in part because `pip` supports it too.
Fixes#1666, Fixes#2276
[PEP-0508]: https://peps.python.org/pep-0508/
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
refactor pip_list function to trim end of joined elements
fixed: https://github.com/astral-sh/uv/issues/2296
<!-- How was it tested? -->
## Summary
If a user provides a source distribution via a direct path, it can
either be an archive (like a `.tar.gz` or `.zip` file) or a directory.
If the former, we need to extract (e.g., unzip) the contents at some
point. Previously, this extraction was in `uv-build`; this PR lifts it
up to the distribution database.
The first benefit here is that various methods that take the
distribution are now simpler, as they can assume a directory.
The second benefit is that we no longer extract _multiple times_ when
working with a source distribution. (Previously, if we tried to get the
metadata, then fell back and built the wheel, we'd extract the wheel
_twice_.)
## Summary
This PR removes the URL conflict errors when the output of a `uv pip
compile` is used as a constraint to a subsequent `uv pip compile`.
If you run `uv pip compile`, the output file will contain your Git
dependencies, but pinned to a specific commit, like:
```
git+https://github.com/pallets/werkzeug@32e69512134c2f8183c6438b2b2e13fd24e9d19f
```
If you then use the output as a constraint to a subsequent resolution
(e.g., perhaps you require
`git+https://github.com/pallets/werkzeug@main`), we currently fail. I
think this is a reasonable workflow to support when all of these
requirements are coming from _your own_ dependencies. So we now assume
when resolving that the former is a more precise variant of the latter.
Closes https://github.com/astral-sh/uv/issues/1903.
Closes https://github.com/astral-sh/uv/issues/2266.
## Test Plan
`cargo test`
## Summary
This PR enables use of the Windows Store Pythons even with `py` is not
installed. Specifically, we need to ensure that the `python.exe` and
`python3.exe` executables installed into the
`C:\Users\crmar\AppData\Local\Microsoft\WindowsApp` directory _are_ used
when they're not "App execution aliases" (which merely open the Windows
Store, to help you install Python).
When `py` is installed, this isn't strictly necessary, since the
"resolved" executables are discovered via `py`. These look like
`C:\Users\crmar\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbs5n2kfra8p0\python.exe`.
Closes https://github.com/astral-sh/uv/issues/2264.
## Test Plan
- Removed all Python installations from my Windows machine.
- Uninstalled `py`.
- Enabled "App execution aliases".
- Verified that for both `cargo run venv --python python.exe` and `cargo
run venv --python python3.exe`, `uv` exited with a failure that no
Python could be found.
- Installed Python 3.10 via the Windows Store.
- Verified that the above commands succeeded without error.
- Verified that `cargo run venv --python python3.10.exe` _also_
succeeded.
- Now that `packse` is being published to PyPI we can install it from
there.
- Tweaks the tooling around scenario updates to manage a temporary
virtual environment for you.
- Makes use of a new index URL
- Includes local version segment scenarios (supersedes
https://github.com/astral-sh/uv/pull/2022)
Sometimes, the first time we read from the stdout of the bytecode
compiler python subprocess, we get an empty string back (no newline). If
we try to write to stdin, it will often be a broken pipe (#2245). After
we got an empty string the first time, we will get the same empty string
if we read a line again.
The details of this behavior are mysterious to me, but it seems that it
can be identified by the first empty string. We check by inserting
starting with a `Ready` message on the Python side. When we encounter
the broken state, we discard the interpreter and try again.
We have to introduce a third timeout check for the interpreter launch
itself.
Minimized test script:
```bash
#!/usr/bin/env bash
set -euo pipefail
while true; do
date --iso-8601=seconds # Progress indicator
rm -rf testenv
target/profiling/uv venv testenv -q --python 3.12
VIRTUAL_ENV=$PWD/testenv target/profiling/uv pip install -q --compile wheel==0.42.0
done
```
Run as
```
cargo build --profile profiling && bash compile_bug.sh
```
Fixes#2245
## Summary
In #2102, I did some refactor, and changed a method to return the Python
executable path rather than the parent directory path. But I missed this
one codepath for Conda on Windows.
Closes https://github.com/astral-sh/uv/issues/2269.
## Test Plan
- Installed micromamba on my Windows machine.
- Reproduced the failure in the linked issue.
- Verified that `python.exe` exists at `${CONDA_PREFIX}\python.exe`.
- Ran with this change; installed successfully.
## Summary
In the list of tags produced by `Tags::from_env`, higher-priority tags
are expected to come earlier in the list. Right now, though, we push
tags like `py38` before `py312`. So if you run `cargo run pip install
multiprocess -n --reinstall --verbose` on Python 3.12, you get the
`py38` wheel rather than the `py32` wheel.
Closes https://github.com/astral-sh/uv/issues/2261.
## Summary
This PR adds support for pip's `--no-build-isolation`. When enabled,
build requirements won't be installed during PEP 517-style builds, but
the source environment _will_ be used when executing the build steps
themselves.
Closes https://github.com/astral-sh/uv/issues/1715.
Fix parsing `pytest;'4.0'>=python_version`, where previously the
operator and the variable were incorrectly tokenized as one invalid
operator.
Fixes#2247
Running the pep508_rs tests was failing due to uv-fs depending on
`fs_err::tokio` even when not selected. But the function that used it is
unused anyway, so i removed it.
## Summary
It turns out that by keeping the `RECORD` file open, older versions of
Windows mark it for deletion, but don't allow it to be deleted until
it's closed. As such, we end up leaving the `.dist-info` directory
around, since it appears non-empty; but once the program terminates, we
_do_ delete `RECORD`, leaving it empty. This then creates the impression
that a package exists where it does not.
Closes https://github.com/astral-sh/uv/issues/2074.
## Summary
If we fallback to streaming the wheel (because the registry doesn't
support range requests), we currently don't cache the metadata at all.
This PR fixes that, ensuring that we cache based on the same HTTP
policies, etc.
## Summary
We're seeing reports that Sonatype Nexus isn't working with cached data.
Users are reporting 304 responses that show "Found modified response..."
path in the logs. I can't reproduce this on latest Sonatype Nexus, but
my best guess is that there's a 304 response that is failing our
validators, and we try to use that as if it's a "complete" response?
Closes https://github.com/astral-sh/uv/issues/1754.
