Commit graph

198 commits

Author SHA1 Message Date
samypr100
68abf85f0d
feat: mTLS support (#4171)
## Summary

Closes https://github.com/astral-sh/uv/issues/3626

This adds mTLS support to uv via the standard env var `SSL_CLIENT_CERT`.

## Test Plan

Tested locally using a [nginx proxy to
pypi](https://github.com/hauntsaninja/nginx_pypi_cache) using my own
self-signed ca + certs + client certs generated via
[mkcert](https://github.com/FiloSottile/mkcert). Used this proxy with
both uv and pip to make sure we have feature partity in mTLS
functionality.
2024-06-10 20:11:35 -05:00
Zanie Blue
f9ea304be4
Drop "registry" prefix from request timeout log (#4144)
We use this base client for more than registry requests
2024-06-07 16:56:32 -05:00
konsti
63c84ed4a6
Log transient network request failures (#3933)
We retry several kinds of network request failures, but it's often
unclear whether a request was retried or not
(https://github.com/astral-sh/uv/issues/3514#issuecomment-2105485773).
This PR adds a small intermediary layer that logs all transient request
failures, adding the `DEBUG Transient request failure` lines:

```
DEBUG Searching for Python interpreter in virtual environments
DEBUG Found CPython 3.12.3 at `/home/konsti/projects/uv/.venv/bin/python3` (active virtual environment)
DEBUG Using Python 3.12.3 environment at .venv/bin/python3
DEBUG Acquired lock for `.venv`
DEBUG At least one requirement is not satisfied: tqdm
DEBUG Using registry request timeout of 30s
DEBUG Solving with target Python version 3.12.3
DEBUG Adding direct dependency: tqdm*
DEBUG No cache entry for: https://pypi.org/simple/tqdm/
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: client error (Connect)
  Caused by: dns error: failed to lookup address information: Name or service not known
  Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: client error (Connect)
  Caused by: dns error: failed to lookup address information: Name or service not known
  Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: client error (Connect)
  Caused by: dns error: failed to lookup address information: Name or service not known
  Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: client error (Connect)
  Caused by: dns error: failed to lookup address information: Name or service not known
  Caused by: failed to lookup address information: Name or service not known
error: Could not connect, are you offline?
  Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: client error (Connect)
  Caused by: dns error: failed to lookup address information: Name or service not known
  Caused by: failed to lookup address information: Name or service not known
```

I decided for multi-line logging to show the complete error trace since
only `Transient request failure for https://pypi.org/simple/tqdm/,
retrying: Request error: error sending request for url
(https://pypi.org/simple/tqdm/)` doesn't tell you the actual problem (a
dns error).

Note that running with `-v` will not show messages about retry backoff
timing, but running with `RUST_LOG=debug` now shows a complete picture:

```
DEBUG starting new connection: https://pypi.org/
DEBUG resolving host="pypi.org"
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
  Caused by: client error (Connect)
  Caused by: dns error: failed to lookup address information: Name or service not known
  Caused by: failed to lookup address information: Name or service not known
WARN Retry attempt #2. Sleeping 528.728192ms before the next attempt
```

Fixes #3572
2024-06-04 15:39:16 +02:00
Charlie Marsh
cedd18e4c6
Remove some unused pub functions (#3872)
## Summary

I wrote a bad Python script to find these.
2024-05-28 15:58:13 +00:00
Zanie Blue
84afca2696
Add offline support to uv tool run and uv run (#3676)
Adds `--offline` support to `uv tool run` and `uv run` because I needed
it on the airplane today.

I think we should move `--offline` to the global settings like
`--native-tls`.
2024-05-21 15:58:15 -05:00
Charlie Marsh
558f628ef1
Propagate URL errors in verbatim parsing (#3720)
## Summary

Closes https://github.com/astral-sh/uv/issues/3715.

## Test Plan

```
❯ echo "/../test" | cargo run pip compile -
error: Couldn't parse requirement in `-` at position 0
  Caused by: path could not be normalized: /../test
/../test
^^^^^^^^

❯ echo "-e /../test" | cargo run pip compile -
error: Invalid URL in `-`: `/../test`
  Caused by: path could not be normalized: /../test
  Caused by: cannot normalize a relative path beyond the base directory
```
2024-05-21 19:58:59 +00:00
Charlie Marsh
cf997080b0
Rename DistInfoMetadata to CoreMetadata (#3699)
## Summary

This reflects the change codified in PEP 714.
2024-05-21 18:26:59 +00:00
Charlie Marsh
fee344db6f
Add PEP 714 support for HTML API client (#3697)
## Summary

If `data-core-metadata` is set, we need to respect that over
`data-dist-info-metadata` in the HTML client.

See: https://github.com/astral-sh/uv/issues/3689
2024-05-21 18:05:40 +00:00
Charlie Marsh
5205165d42
Add PEP 714 support for JSON API client (#3698)
## Summary

Closes https://github.com/astral-sh/uv/issues/3689.

## Test Plan

Manually verified we pick up `core-metadata` from PyPI if I remove the
aliases.
2024-05-21 15:52:37 +00:00
Charlie Marsh
963f2a778b
URL-decode hashes in HTML fragments (#3655)
## Summary

Closes https://github.com/astral-sh/uv/issues/3654
2024-05-18 22:19:55 -04:00
Andrew Gallant
018a7150d6
uv-distribution: include all wheels in distribution types (#3595)
Our current flow of data from "simple registry package" to "final
resolved distribution" goes through a number of types:

* `SimpleMetadata` is the API response from a registry that includes all
published versions for a package. Each version has an assortment of
metadata
associated with it.
* `VersionFiles` is the aforementioned metadata. It is split in two: a
group of files for source distributions and a group of files for wheels.
* `PrioritizedDist` collects a subset of the files from `VersionFiles`
to form a selection of the "best" sdist and the "best" wheel for the
current environment.
* `CompatibleDist` is created from a borrowed `PrioritizedDist` that,
perhaps among other things, encapsulates the decision of whether to pick
an sdist or a wheel. (This decision depends both on compatibility and
the action being performed. e.g., When doing installation, a
`CompatibleDist` will sometimes select an sdist over a wheel.)
* `ResolvedDistRef` is like a `ResolvedDist`, but borrows a `Dist`.
* `ResolvedDist` is the almost-final-form of a distribution in a
resolution and is created from a `ResolvedDistRef`.
* `AnnotatedResolvedDist` is a new data type that is the actual final
form of a distribution that a universal lock file cares about. It
bundles a `ResolvedDist` with some metadata needed to generate a lock
file.

One of the requirements of a universal lock file is that we include all
wheels (and maybe all source distributions? but at least one if it's
present) associated with a distribution. But the above flow of data (in
the step from `VersionFiles` to `PrioritizedDist`) drops all wheels
except for the best one.

To remedy this, in this PR, we rejigger `PrioritizedDist`,
`CompatibleDist` and `ResolvedDistRef` so that all wheel data is
preserved. And when a `ResolvedDistRef` is finally turned into a
`ResolvedDist`, we copy all of the wheel data. And finally, we adjust
the `Lock` constructor to read this new data and include it in the lock
file. To make this work, we also modify `RegistryBuiltDist` so that it
can contain one or more wheels instead of just one.

One shortcoming here (called out in the code as a FIXME) is that if a
source distribution is selected as the "best" thing to use (perhaps
there are no compatible wheels), then the wheels won't end up in the
lock file. I plan to fix this in a follow-up PR.

We also aren't totally consistent on source distribution naming.
Sometimes we use `sdist`. Sometimes `source`. Sometimes `source_dist`.
I think it'd be nice to just use `sdist` everywhere, but I do prefer
the type names to be `SourceDist`. And sometimes you want function
names to match the type names (i.e., `from_source_dist`), which in turn
leads to an appearance of inconsistency. I'm open to ideas.

Closes #3351
2024-05-15 15:07:28 -04:00
Dimitri Papadopoulos Orfanos
d2ee567fe7
Fix a few typos found by codespell (#3543)
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

Just fix typos.

While `alpha-numeric` is not really a misspelling:
- it is missing from mainstream curated dictionaries, all of them
suggest `alphanumeric`;
- it is less used than `alphanumeric` (more than ⨉10 less) according to
the Google [Ngram
Viewer](https://books.google.com/ngrams/graph?content=alpha-numeric%2Calphanumeric&year_start=1900&year_end=2019&corpus=en-2019);
- it is [missing from
SCOWL](http://app.aspell.net/lookup?dict=en_US-large;words=alpha-numeric).

## Test Plan

CI jobs.
2024-05-13 11:55:10 +00:00
Andrew Gallant
7d67b7bb49 pep508: un-export fields for MarkerEnvironment
We now use the getters and setters everywhere.

There were some places where we wanted to build a `MarkerEnvironment`
out of whole cloth, usually in tests. To facilitate those use cases, we
add a `MarkerEnvironmentBuilder` that provides a convenient constructor.
It's basically like a `MarkerEnvironment::new`, but with named
parameters. That's useful here because there are so many fields (and
they many have the same type).
2024-05-09 10:06:02 -04:00
Charlie Marsh
18d229e2bb
Upgrade async_http_range_reader to v0.8.0 (#3460)
## Summary

Closes #2025.
Closes https://github.com/astral-sh/uv/issues/3255.
Closes https://github.com/astral-sh/uv/pull/2843.
2024-05-08 10:54:08 -04:00
Ibraheem Ahmed
94cf604574
Remove unnecessary uses of DashMap and Arc (#3413)
## Summary

All of the resolver code is run on the main thread, so a lot of the
`Send` bounds and uses of `DashMap` and `Arc` are unnecessary. We could
also switch to using single-threaded versions of `Mutex` and `Notify` in
some places, but there isn't really a crate that provides those I would
be comfortable with using.

The `Arc` in `OnceMap` can't easily be removed because of the uv-auth
code which uses the
[reqwest-middleware](https://docs.rs/reqwest-middleware/latest/reqwest_middleware/trait.Middleware.html)
crate, that seems to adds unnecessary `Send` bounds because of
`async-trait`. We could duplicate the code and create a `OnceMapLocal`
variant, but I don't feel that's worth it.
2024-05-06 22:30:43 -04:00
Tim de Jager
9ae116f82b
fix: remove cache generic from builder (#3322)
Just a small fix, remove generic argument that I think was unused.
2024-04-30 08:27:55 -05:00
Charlie Marsh
eabefbf8a2
Ignore 401 errors with multiple indexes (#3292)
## Summary

It seems like Azure might return a 401 when you request a package that
doesn't exist (even with valid credentials)? But I admittedly haven't
tested this. (We already skip 403, and this seems similar?)

Closes https://github.com/astral-sh/uv/issues/3291.
2024-04-28 10:06:43 -04:00
Yorick
43181f1933
Implement --index-strategy unsafe-best-match (#3138)
## Summary

This index strategy resolves every package to the latest possible
version across indexes. If a version is in multiple indexes, the first
available index is selected.

Implements #3137 

This closely matches pip.

## Test Plan

Good question. I'm hesitant to use my certifi example here, since that
would inevitably break when torch removes this package. Please comment!
2024-04-27 01:24:54 +00:00
konsti
d10903f0a4
30s default http read timeout (#3182)
Since we're now using read timeouts and not total timeouts, we can use a
lower threshold, a single read shouldn't take 5 min (and not even 10s).

The 10s value is somewhat arbitrary.

Like #3144, this is a breaking change in some sense.
2024-04-22 19:05:44 -04:00
Charlie Marsh
9f2bc19eaf
Enforce HTTP timeouts on a per-read (rather than per-request) basis (#3144)
## Summary

This leverages the new `read_timeout` property, which ensures that (like
pip) our timeout is not applied to the _entire_ request, but rather, to
each individual read operation.

Closes: #1921.

See: #1912.
2024-04-19 16:49:53 -04:00
elbaro
ab74263cbc
Skip HEAD requests for Pypicloud with Private S3 (#3070) 2024-04-16 18:25:35 +00:00
Zanie Blue
c0efeeddf6
Rewrite uv-auth (#2976)
Closes 

- #2822 
- https://github.com/astral-sh/uv/issues/2563 (via #2984)

Partially address:

- https://github.com/astral-sh/uv/issues/2465
- https://github.com/astral-sh/uv/issues/2464

Supersedes:

- https://github.com/astral-sh/uv/pull/2947
- https://github.com/astral-sh/uv/pull/2570 (via #2984)

Some significant refactors to the whole `uv-auth` crate:

- Improving the API
- Adding test coverage
- Fixing handling of URL-encoded passwords
- Fixing keyring authentication
- Updated middleware (see #2984 for more)
2024-04-16 11:48:37 -05:00
Charlie Marsh
3dd673677a
Add --find-links source distributions to the registry cache (#2986)
## Summary

Source distributions in `--find-links` are now properly picked up in the
cache.

Closes https://github.com/astral-sh/uv/issues/2978.
2024-04-11 01:25:58 +00:00
Charlie Marsh
1f3b5bb093
Add hash-checking support to install and sync (#2945)
## Summary

This PR adds support for hash-checking mode in `pip install` and `pip
sync`. It's a large change, both in terms of the size of the diff and
the modifications in behavior, but it's also one that's hard to merge in
pieces (at least, with any test coverage) since it needs to work
end-to-end to be useful and testable.

Here are some of the most important highlights:

- We store hashes in the cache. Where we previously stored pointers to
unzipped wheels in the `archives` directory, we now store pointers with
a set of known hashes. So every pointer to an unzipped wheel also
includes its known hashes.
- By default, we don't compute any hashes. If the user runs with
`--require-hashes`, and the cache doesn't contain those hashes, we
invalidate the cache, redownload the wheel, and compute the hashes as we
go. For users that don't run with `--require-hashes`, there will be no
change in performance. For users that _do_, the only change will be if
they don't run with `--generate-hashes` -- then they may see some
repeated work between resolution and installation, if they use `pip
compile` then `pip sync`.
- Many of the distribution types now include a `hashes` field, like
`CachedDist` and `LocalWheel`.
- Our behavior is similar to pip, in that we enforce hashes when pulling
any remote distributions, and when pulling from our own cache. Like pip,
though, we _don't_ enforce hashes if a distribution is _already_
installed.
- Hash validity is enforced in a few different places:
1. During resolution, we enforce hash validity based on the hashes
reported by the registry. If we need to access a source distribution,
though, we then enforce hash validity at that point too, prior to
running any untrusted code. (This is enforced in the distribution
database.)
2. In the install plan, we _only_ add cached distributions that have
matching hashes. If a cached distribution is missing any hashes, or the
hashes don't match, we don't return them from the install plan.
3. In the downloader, we _only_ return distributions with matching
hashes.
4. The final combination of "things we install" are: (1) the wheels from
the cache, and (2) the downloaded wheels. So this ensures that we never
install any mismatching distributions.
- Like pip, if `--require-hashes` is provided, we require that _all_
distributions are pinned with either `==` or a direct URL. We also
require that _all_ distributions have hashes.

There are a few notable TODOs:

- We don't support hash-checking mode for unnamed requirements. These
should be _somewhat_ rare, though? Since `pip compile` never outputs
unnamed requirements. I can fix this, it's just some additional work.
- We don't automatically enable `--require-hashes` with a hash exists in
the requirements file. We require `--require-hashes`.

Closes #474.

## Test Plan

I'd like to add some tests for registries that report incorrect hashes,
but otherwise: `cargo test`
2024-04-10 19:09:03 +00:00
Charlie Marsh
48ba7df98a
Move FlatIndex into the uv-resolver crate (#2972)
## Summary

This lets us remove circular dependencies (in the future, e.g., #2945)
that arise from `FlatIndex` needing a bunch of resolver-specific
abstractions (like incompatibilities, required hashes, etc.) that aren't
necessary to _fetch_ the flat index entries.
2024-04-10 14:38:42 -04:00
Charlie Marsh
a01143980a
Upgrade reqwest to v0.12.3 (#2817)
## Summary

Closes #2814.
2024-04-10 11:20:44 -04:00
Charlie Marsh
38ab39c439
Strip query string when parsing filename from HTML index (#2961)
## Summary

Closes https://github.com/astral-sh/uv/issues/2958.
2024-04-10 09:25:29 -05:00
Charlie Marsh
13ae5ac8dc
Replace PyPI-internal Hashes representation with flat vector (#2925)
## Summary

Right now, we have a `Hashes` representation that looks like:

```rust
/// A dictionary mapping a hash name to a hex encoded digest of the file.
///
/// PEP 691 says multiple hashes can be included and the interpretation is left to the client.
#[derive(Debug, Clone, Eq, PartialEq, Default, Deserialize)]
pub struct Hashes {
    pub md5: Option<Box<str>>,
    pub sha256: Option<Box<str>>,
    pub sha384: Option<Box<str>>,
    pub sha512: Option<Box<str>>,
}
```

It stems from the PyPI API, which returns a dictionary of hashes.

We tend to pass these around as a vector of `Vec<Hashes>`. But it's a
bit strange because each entry in that vector could contain multiple
hashes. And it makes it difficult to ask questions like "Is
`sha256:ab21378ca980a8` in the set of hashes"?

This PR instead treats `Hashes` as the PyPI-internal type, and uses a
new `Vec<HashDigest>` everywhere in our own APIs.
2024-04-09 16:56:16 +00:00
Zanie Blue
1512e07a2e
Split configuration options out of uv-types (#2924)
Needed to prevent circular dependencies in my toolchain work (#2931). I
think this is probably a reasonable change as we move towards persistent
configuration too?

Unfortunately `BuildIsolation` needs to be in `uv-types` to avoid
circular dependencies still. We might be able to resolve that in the
future.
2024-04-09 11:35:53 -05:00
Zanie Blue
b535d252c9
Fix base client builder docstring reference (#2860) 2024-04-07 16:16:13 +00:00
Charlie Marsh
00934044aa
Backtrack on distributions with invalid metadata (#2834)
## Summary

Closes https://github.com/astral-sh/uv/issues/2821.
2024-04-05 18:00:48 -04:00
Charlie Marsh
a0b8d1a994
Clean up Error enum in Metadata23 (#2835)
## Summary

Rename, and remove a bunch of unused variants.
2024-04-05 14:40:33 +00:00
Charlie Marsh
2ac562b40d
Respect --no-build and --no-binary in --find-links (#2826)
## Summary

In working on `--require-hashes`, I noticed that we're missing some
incompatibility tracking for `--find-links` distributions. Specifically,
we don't respect `--no-build` or `--no-binary`, so if we select a wheel
due to `--find-links`, we then throw a hard error when trying to build
it later (if `--no-binary` is provided), rather than selecting the
source distribution instead.

Closes https://github.com/astral-sh/uv/issues/2827.
2024-04-05 02:00:39 +00:00
Charlie Marsh
dc2c289dff
Upgrade rs-async-zip to support data descriptors (#2809)
## Summary

Upgrading `rs-async-zip` enables us to support data descriptors in
streaming. This both greatly improves performance for indexes that use
data descriptors _and_ ensures that we support them in a few other
places (e.g., zipped source distributions created in Finder).

Closes #2808.
2024-04-04 01:31:40 +00:00
Charlie Marsh
34341bd6e9
Allow package lookups across multiple indexes via explicit opt-in (#2815)
## Summary

This partially revives https://github.com/astral-sh/uv/pull/2135 (with
some modifications) to enable users to opt-in to looking for packages
across multiple indexes.

The behavior is such that, in version selection, we take _any_
compatible version from a "higher-priority" index over the compatible
versions of a "lower-priority" index, even if that means we might accept
an "older" version.

Closes https://github.com/astral-sh/uv/issues/2775.
2024-04-03 23:23:37 +00:00
Charlie Marsh
dc957d7322
Respect --no-index with --find-links in pip sync (#2692)
## Summary

In `pip sync`, we weren't properly handling cases in which a package
_only_ existed in `--find-links` (e.g., the user passed `--offline` or
`--no-index`).

I plan to explore removing `Finder` entirely to avoid these mismatch
bugs between `pip sync` and other commands, but this is fine for now.

Closes https://github.com/astral-sh/uv/issues/2688.

## Test Plan

`cargo test`
2024-03-27 16:15:14 +00:00
Charlie Marsh
5d7d7dce24
Enable PEP 517 builds for unnamed requirements (#2600)
## Summary

This PR enables the source distribution database to be used with unnamed
requirements (i.e., URLs without a package name). The (significant)
upside here is that we can now use PEP 517 hooks to resolve unnamed
requirement metadata _and_ reuse any computation in the cache.

The changes to `crates/uv-distribution/src/source/mod.rs` are quite
extensive, but mostly mechanical. The core idea is that we introduce a
new `BuildableSource` abstraction, which can either be a distribution,
or an unnamed URL:

```rust
/// A reference to a source that can be built into a built distribution.
///
/// This can either be a distribution (e.g., a package on a registry) or a direct URL.
///
/// Distributions can _also_ point to URLs in lieu of a registry; however, the primary distinction
/// here is that a distribution will always include a package name, while a URL will not.
#[derive(Debug, Clone, Copy)]
pub enum BuildableSource<'a> {
    Dist(&'a SourceDist),
    Url(SourceUrl<'a>),
}
```

All the methods on the source distribution database now accept
`BuildableSource`. `BuildableSource` has a `name()` method, but it
returns `Option<&PackageName>`, and everything is required to work with
and without a package name.

The main drawback of this approach (which isn't a terrible one) is that
we can no longer include the package name in the cache. (We do continue
to use the package name for registry-based distributions, since those
always have a name.). The package name was included in the cache route
for two reasons: (1) it's nice for debugging; and (2) we use it to power
`uv cache clean flask`, to identify the entries that are relevant for
Flask.

To solve this, I changed the `uv cache clean` code to look one level
deeper. So, when we want to determine whether to remove the cache entry
for a given URL, we now look into the directory to see if there are any
wheels that match the package name. This isn't as nice, but it does work
(and we have test coverage for it -- all passing).

I also considered removing the package name from the cache routes for
non-registry _wheels_, for consistency... But, it would require a cache
bump, and it didn't feel important enough to merit that.
2024-03-21 22:46:39 -04:00
Zanie Blue
c6e181d233
Respect HTTP client options when reading remote requirements files (#2434)
Uses the base client introduced in #2431 to restore use of our fully
configured client when reading remote requirements files.

Closes https://github.com/astral-sh/uv/issues/2357

## Test plan

```
npx http-server --username user --password password
cargo run -- pip install -r http://user:password@127.0.0.1:8080/requirements.txt
```

Fails on main succeeds on branch.
2024-03-21 13:48:57 -05:00
Charlie Marsh
2979918320
Add support for unnamed Git and HTTP requirements (#2578)
## Summary

Enables, e.g., `uv pip install
git+https://github.com/pallets/flask.git`.

Part of: https://github.com/astral-sh/uv/issues/313.
2024-03-21 13:44:54 +00:00
veryyet
d6dad57fab
chore: fix some typos (#2581) 2024-03-21 04:09:37 +00:00
konsti
32b9eeb532
Use mac version from python for linehaul information (#2509)
See
https://github.com/astral-sh/uv/pull/2493#pullrequestreview-1942899151.

---------

Co-authored-by: Zanie Blue <contact@zanie.dev>
2024-03-20 10:55:50 +01:00
Charlie Marsh
659e00c4c1
Use Box<str> in Hashes to reduce size (#2536) 2024-03-19 02:51:46 +00:00
Charlie Marsh
80aa03dcba
Add SHA384 and SHA512 hash algorithms (#2534)
Closes #2533.
2024-03-19 02:23:16 +00:00
Charlie Marsh
d0789fc078
Preserve hashes for pinned packages (#2532)
## Summary

When a user runs with `--output-file` and `--generate-hashes`, we should
_only_ update the hashes if the pinned version itself changes.

Closes https://github.com/astral-sh/uv/issues/1530.
2024-03-19 01:02:18 +00:00
samypr100
42973cd9cb
feat: add linehaul info to uv-client (#2493)
## Summary

Closes #1958

This adds linehaul metadata to uv's user-agent when pep 508 markers are
provided to the RegistryClientBuilder. Thanks to #2381, we were able to
leverage most information from markers and avoid inconsistency.

Linehaul is meant to be accompanying metadata pip sends in it's user
agent when talking to registries. You can see this output by running
something like `python -c 'from pip._internal.network.session import
user_agent; print(user_agent())'`.
In PyPI, this metadata processed by the
[linehaul-cloud-function](https://github.com/pypi/linehaul-cloud-function).
More info about linehaul can be found in #1958.

Below are some examples from pip:

* Linux GHA: `pip/24.0
{"ci":true,"cpu":"x86_64","distro":{"id":"jammy","libc":{"lib":"glibc","version":"2.35"},"name":"Ubuntu","version":"22.04"},"implementation":{"name":"CPython","version":"3.12.2"},"installer":{"name":"pip","version":"24.0"},"openssl_version":"OpenSSL
3.0.2 15 Mar
2022","python":"3.12.2","rustc_version":"1.76.0","system":{"name":"Linux","release":"6.5.0-1016-azure"}}`
* Windows GHA: `pip/24.0
{"ci":true,"cpu":"AMD64","implementation":{"name":"CPython","version":"3.12.2"},"installer":{"name":"pip","version":"24.0"},"openssl_version":"OpenSSL
3.0.13 30 Jan
2024","python":"3.12.2","rustc_version":"1.76.0","system":{"name":"Windows","release":"2022Server"}}`
* OSX GHA: `pip/24.0
{"ci":true,"cpu":"arm64","distro":{"name":"macOS","version":"14.2.1"},"implementation":{"name":"CPython","version":"3.12.2"},"installer":{"name":"pip","version":"24.0"},"openssl_version":"OpenSSL
3.0.13 30 Jan
2024","python":"3.12.2","rustc_version":"1.76.0","system":{"name":"Darwin","release":"23.2.0"}}`



Here's how uv results look like (sorry for the keys not having the same
order):

* Linux GHA: `uv/0.1.21
{"installer":{"name":"uv","version":"0.1.21"},"python":"3.12.2","implementation":{"name":"CPython","version":"3.12.2"},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":"Linux","release":"6.5.0-1016-azure"},"cpu":"x86_64","openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`
* Windows GHA: `uv/0.1.21
{"installer":{"name":"uv","version":"0.1.21"},"python":"3.12.2","implementation":{"name":"CPython","version":"3.12.2"},"distro":null,"system":{"name":"Windows","release":"2022Server"},"cpu":"AMD64","openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`
* OSX GHA: `uv/0.1.21
{"installer":{"name":"uv","version":"0.1.21"},"python":"3.12.2","implementation":{"name":"CPython","version":"3.12.2"},"distro":{"name":"macOS","version":"14.2.1","id":null,"libc":null},"system":{"name":"Darwin","release":"23.2.0"},"cpu":"arm64","openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Distro information (such as the one pip uses `from pip._vendor import
distro` to retrieve instead of `platform` module) was not retrieved from
markers. Instead, the linux release codename/name/version uses
`sys-info` crate, adding about 50us of extra overhead on linux. The
distro osx version re-used the [mac_os version
implementation](99c992e38b/crates/platform-host/src/mac_os.rs)
from #2381 which adds about 20us of overhead on osx. I tried to use
other crates to avoid re-introducing `mac_os.rs` but most of them didn't
yield satisfactory performance (40ms-60ms~) or had the wrong values
needed (e.g. darwin version vs osx version).

I also didn't add libc retrieval or rustc retrieval as those seem to add
substantial overhead due to querying `ldd` or `rustc`. PyPy version
detection was also not added to avoid adding extra overhead to [support
PyPy for
linehaul](https://github.com/pypa/pip/blob/24.0/src/pip/_internal/network/session.py#L123).
All other behavior was kept 1-1 to match what pip's linehaul
implementation does (as of 24.0). This also aligns with what was
discussed in #1958.

## Test Plan

Added new integration test to uv-client.

---------

Co-authored-by: konstin <konstin@mailbox.org>
2024-03-18 10:46:58 +00:00
Zanie Blue
9c27f92203
Introduce a BaseClient for construction of canonical configured client (#2431)
In preparation for support of
https://github.com/astral-sh/uv/issues/2357 (see
https://github.com/astral-sh/uv/pull/2434)
2024-03-15 12:07:38 -05:00
Charlie Marsh
8463d6d672
Apply from-URL credentials in authentication middleware (#2449)
## Summary

Right now, the middleware doesn't apply credentials that were
_originally_ sourced from a URL. This requires that we call
`with_url_encoded_auth` whenever we create a request to ensure that any
credentials that were passed in as part of an index URL (for example)
are respected.

This PR modifies `uv-auth` to instead apply those credentials in the
middleware itself. This seems preferable to me. As far as I can tell, we
can _only_ add in-URL credentials to the store ourselves (since in-URL
credentials are converted to headers by the time they reach the
middleware). And if we ever _didn't_ apply those credentials to new
URLs, it'd be a bug in the logic that precedes the middleware (i.e., us
forgetting to call `with_url_encoded_auth`).

## Test Plan

`cargo run pip install` with an authenticated index.
2024-03-15 16:21:37 +00:00
Zanie Blue
22a52391be
Refactor AuthenticationStore to inline credentials (#2427) 2024-03-13 17:48:02 -05:00
Hans Baker
9159731792
Add support for retrieving credentials from keyring (#2254)
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

<!-- What's the purpose of the change? What does it do, and why? -->

Adds basic keyring auth support for `uv` commands. Adds clone of `pip`'s
`--keyring-provider subprocess` argument (using CLI `keyring` tool).

See issue: https://github.com/astral-sh/uv/issues/1520

## Test Plan

<!-- How was it tested? -->

Hard to write full-suite unit tests due to reliance on
`process::Command` for `keyring` cli

Manually tested end-to-end in a project with GCP artifact registry using
keyring password:
```bash
➜  uv pip uninstall watchdog
Uninstalled 1 package in 46ms
 - watchdog==4.0.0

➜  cargo run -- pip install --index-url https://<redacted>/python/simple/ --extra-index-url https://<redacted>/pypi-mirror/simple/ watchdog
    Finished dev [unoptimized + debuginfo] target(s) in 0.18s
     Running `target/debug/uv pip install --index-url 'https://<redacted>/python/simple/' --extra-index-url 'https://<redacted>/pypi-mirror/simple/' watchdog`
error: HTTP status client error (401 Unauthorized) for url (https://<redacted>/pypi-mirror/simple/watchdog/)

➜  cargo run -- pip install --keyring-provider subprocess --index-url https://<redacted>/python/simple/ --extra-index-url https://<redacted>/pypi-mirror/simple/ watchdog
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/uv pip install --keyring-provider subprocess --index-url 'https://<redacted>/python/simple/' --extra-index-url 'https://<redacted>/pypi-mirror/simple/' watchdog`
Resolved 1 package in 2.34s
Installed 1 package in 27ms
 + watchdog==4.0.0
```

`requirements.txt`
```
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    .bin/generate-requirements
#
--index-url https://<redacted>/python/simple/
--extra-index-url https://<redacted>/pypi-mirror/simple/

...
```

```bash
➜  cargo run -- pip install --keyring-provider subprocess -r requirements.txt
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
     Running `target/debug/uv pip install --keyring-provider subprocess -r requirements.txt`
Resolved 205 packages in 23.52s
   Built <redacted>
   ...
Downloaded 47 packages in 19.32s
Installed 195 packages in 276ms
 + <redacted>
  ...
```

---------

Co-authored-by: Thomas Gilgenast <thomas@vant.ai>
Co-authored-by: Zanie Blue <contact@zanie.dev>
2024-03-13 15:02:18 -05:00
samypr100
e0ac5b4e84
feat: keep backwards compatibility with SSL_CERT_FILE without requiring --native-tls (#2401)
## Summary

Small follow up to https://github.com/astral-sh/uv/pull/2362 to check if
`SSL_CERT_FILE` is set to enable `--native-tls` functionality. This
maintains backwards compatibility with `0.1.17` and below users
leveraging only `SSL_CERT_FILE`.

Closes https://github.com/astral-sh/uv/issues/2400

## Test Plan

<!-- How was it tested? -->
Assuming `SSL_CERT_FILE` is already working via `--native-tls`, this is
simply a shortcut to enable `--native-tls` functionality implicitly
while still being able to let `rustls-native-certs` handle the loading
of `SSL_CERT_FILE` instead of ourselves.

Edit: Manually tested by setting up own self-signed CA certificate
bundle and set `SSL_CERT_FILE` to this and confirmed the loading happens
without having to specify `--native-tls`.
2024-03-13 04:33:10 +00:00