Commit graph

327 commits

Author SHA1 Message Date
Zanie Blue
4c0590ff6f Add --group support to uv add and uv remove (#8108)
Part of #8090

Adds the ability to add and remove dependencies from arbitrary groups
using `uv add` and `uv remove`. Does not include resolving with the new
dependencies — tackling that in #8110.

Additionally, this does not yet resolve interactions with the existing
`dev` group — we'll tackle that separately as well. I probably won't
merge the stack until that design is resolved.
2024-10-25 13:27:37 -05:00
konsti
3eda248ef5
Always attach URL to network errors (#8444) 2024-10-25 09:10:18 +00:00
Charlie Marsh
cc734ea2b6
Allow dependency metadata entries for direct URL requirements (#7846)
## Summary

This is part of making
https://github.com/astral-sh/uv/issues/7299#issuecomment-2385286341
better. You can now use `tool.uv.dependency-metadata` for direct URL
requirements. Unfortunately, you _must_ include a version, since we need
one to perform resolution.
2024-10-22 22:01:23 -04:00
Charlie Marsh
ff3ed3b797
Add type-based validation for index names (#8464)
## Summary

Also documents the normalization scheme.
2024-10-22 16:10:20 +00:00
Charlie Marsh
7beb5ebe26
Improve sources deserialization errors (#8308) 2024-10-17 21:33:04 +00:00
Charlie Marsh
5e05a62004
Respect index priority when storing credentials (#8256)
## Summary

Closes https://github.com/astral-sh/uv/issues/8248.
2024-10-16 15:52:26 +00:00
Charlie Marsh
999b3f06a4
Respect relative paths in uv build sources (#8237)
## Summary

Right now, `uv build` will fail if a package depends on a local source
in `build-system.requires`.
2024-10-16 01:46:29 +00:00
Charlie Marsh
98d049407f
Ignore sources in build requirements for non-source trees (#8235)
## Summary

We shouldn't enforce sources when, e.g., you provide a `.tar.gz` file.

Closes https://github.com/astral-sh/uv/issues/8236.
2024-10-16 00:53:19 +00:00
Charlie Marsh
59003cb021
Avoid showing lower-bound warning outside of explicit lock and sync (#8234)
## Summary

We shouldn't show these in `uv add`, especially when the thing we're
adding is about to have a lower-bound put on it. Now, we only show these
when the user runs `uv lock` or `uv sync`.
2024-10-15 20:49:40 -04:00
Charlie Marsh
2153c6ac0d
Respect named --index and --default-index values in tool.uv.sources (#7910)
## Summary

If you pass a named index via the CLI, you can now reference it as a
named source. This required some surprisingly large refactors, since we
now need to be able to track whether a given index was provided on the
CLI vs. elsewhere (since, e.g., we don't want users to be able to
reference named indexes defined in global configuration).

Closes https://github.com/astral-sh/uv/issues/7899.
2024-10-15 23:56:24 +00:00
Charlie Marsh
a034a8b83b
Remove the flat index types (#7759)
## Summary

I think these really don't pull their weight.
2024-10-15 23:30:37 +00:00
Charlie Marsh
1925922770
Enable environment variable authentication for named indexes (#7741)
## Summary

This PR enables users to provide index credentials via named environment
variables.

For example, given an index named `internal` that requires a username
(`public`) and password
(`koala`), you can define the index (without credentials) in your
`pyproject.toml`:

```toml
[[tool.uv.index]]
name = "internal"
url = "https://pypi-proxy.corp.dev/simple"
```

Then set the `UV_INDEX_INTERNAL_USERNAME` and
`UV_INDEX_INTERNAL_PASSWORD`
environment variables, where `INTERNAL` is the uppercase version of the
index name:

```sh
export UV_INDEX_INTERNAL_USERNAME=public
export UV_INDEX_INTERNAL_PASSWORD=koala
```
2024-10-15 22:35:07 +00:00
Charlie Marsh
5b391770df
Add support for named and explicit indexes (#7481)
## Summary

This PR adds a first-class API for defining registry indexes, beyond our
existing `--index-url` and `--extra-index-url` setup.

Specifically, you now define indexes like so in a `uv.toml` or
`pyproject.toml` file:

```toml
[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"
```

You can also provide indexes via `--index` and `UV_INDEX`, and override
the default index with `--default-index` and `UV_DEFAULT_INDEX`.

### Index priority

Indexes are prioritized in the order in which they're defined, such that
the first-defined index has highest priority.

Indexes are also inherited from parent configuration (e.g., the
user-level `uv.toml`), but are placed after any indexes in the current
project, matching our semantics for other array-based configuration
values.

You can mix `--index` and `--default-index` with the legacy
`--index-url` and `--extra-index-url` settings; the latter two are
merely treated as unnamed `[[tool.uv.index]]` entries.

### Index pinning

If an index includes a name (which is optional), it can then be
referenced via `tool.uv.sources`:

```toml
[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"

[tool.uv.sources]
torch = { index = "pytorch" }
```

If an index is marked as `explicit = true`, it can _only_ be used via
such references, and will never be searched implicitly:

```toml
[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"
explicit = true

[tool.uv.sources]
torch = { index = "pytorch" }
```

Indexes defined outside of the current project (e.g., in the user-level
`uv.toml`) can _not_ be explicitly selected.

(As of now, we only support using a single index for a given
`tool.uv.sources` definition.)

### Default index

By default, we include PyPI as the default index. This remains true even
if the user defines a `[[tool.uv.index]]` -- PyPI is still used as a
fallback. You can mark an index as `default = true` to (1) disable the
use of PyPI, and (2) bump it to the bottom of the prioritized list, such
that it's used only if a package does not exist on a prior index:

```toml
[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"
default = true
```

### Name reuse

If a name is reused, the higher-priority index with that name is used,
while the lower-priority indexes are ignored entirely.

For example, given:

```toml
[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"

[[tool.uv.index]]
name = "pytorch"
url = "https://test.pypi.org/simple"
```

The `https://test.pypi.org/simple` index would be ignored entirely,
since it's lower-priority than `https://download.pytorch.org/whl/cu121`
but shares the same name.

Closes #171.

## Future work

- Users should be able to provide authentication for named indexes via
environment variables.
- `uv add` should automatically write `--index` entries to the
`pyproject.toml` file.
- Users should be able to provide multiple indexes for a given package,
stratified by platform:
```toml
[tool.uv.sources]
torch = [
  { index = "cpu", markers = "sys_platform == 'darwin'" },
  { index = "gpu", markers = "sys_platform != 'darwin'" },
]
```
- Users should be able to specify a proxy URL for a given index, to
avoid writing user-specific URLs to a lockfile:
```toml
[[tool.uv.index]]
name = "test"
url = "https://private.org/simple"
proxy = "http://<omitted>/pypi/simple"
```
2024-10-15 18:24:23 -04:00
Charlie Marsh
855c1917e1
Respect [tool.uv.sources] in build requirements (#7172)
## Summary

We weren't respecting `tool.uv.sources` for `build-requires`.

Closes https://github.com/astral-sh/uv/issues/7147.
2024-10-15 15:31:04 +00:00
Amos Wenger
715f28fd39
chore: Move all integration tests to a single binary (#8093)
As per
https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html

Before that, there were 91 separate integration tests binary.

(As discussed on Discord — I've done the `uv` crate, there's still a few
more commits coming before this is mergeable, and I want to see how it
performs in CI and locally).
2024-10-11 16:41:35 +02:00
konsti
ad638d7fa3
Use a higher timeout for publishing (#7923) 2024-10-04 15:52:23 +02:00
Charlie Marsh
14507a1793
Add uv- prefix to all internal crates (#7853)
## Summary

Brings more consistency to the repo and ensures that all crates
automatically show up in `--verbose` logging.
2024-10-01 20:15:32 -04:00
Charlie Marsh
f0f2f897de
Add detailed errors for tool.uv.sources deserialization failures (#7823)
## Summary

Closes https://github.com/astral-sh/uv/issues/7817.
2024-10-01 11:49:06 -04:00
Charlie Marsh
1602b5c8d7
Remove unnecessary index location methods (#7826) 2024-10-01 04:44:53 +00:00
Charlie Marsh
b6de417c94
Use serde-untagged to improve some untagged enum error messages (#7822)
## Summary

This is related to https://github.com/astral-sh/uv/issues/7817, but
doesn't close it.
2024-09-30 23:40:21 +00:00
Charlie Marsh
f67347e72c
Allow multiple source entries for each package in tool.uv.sources (#7745)
## Summary

This PR enables users to provide multiple source entries in
`tool.uv.sources`, e.g.:

```toml
[tool.uv.sources]
httpx = [
  { git = "https://github.com/encode/httpx", tag = "0.27.2", marker = "sys_platform == 'darwin'" },
  { git = "https://github.com/encode/httpx", tag = "0.24.1", marker = "sys_platform == 'linux'" },
]
```

The implementation is relatively straightforward: when we lower the
requirement, we now return an iterator rather than a single requirement.
In other words, the above is transformed into two requirements:

```txt
httpx @ git+https://github.com/encode/httpx@0.27.2 ; sys_platform == 'darwin'
httpx @ git+https://github.com/encode/httpx@0.24.1 ; sys_platform == 'linux'
```

We verify (at deserialization time) that the markers are
non-overlapping.

Closes https://github.com/astral-sh/uv/issues/3397.
2024-09-30 21:16:44 +00:00
Charlie Marsh
66d7ec541a
Avoid reusing cached downloaded binaries with --no-binary (#7772)
## Summary

Historically, we've allowed the use of wheels that were downloaded from
PyPI even when the user passes `--no-binary`, if the wheel exists in the
cache. This PR modifies the cache lookup code such that we respect
`--no-build` and `--no-binary` in those paths.

Closes https://github.com/astral-sh/uv/issues/2154.
2024-09-29 17:34:52 +00:00
Charlie Marsh
c415251aa2
Use file stem when parsing cached wheel names (#7773)
## Summary

I noticed that we were including `http` (the file extension) in the
platform tags when reading from the cache:

![Screenshot 2024-09-28 at 9 40
15 PM](https://github.com/user-attachments/assets/d80ed351-1257-42b5-8292-0b11a50c767d)

Probably harmless, but wrong.
2024-09-29 12:05:15 -04:00
konsti
f5601e2610
Clean up "performance allocators" and "performance flate2" backends (#7686)
Co-authored-by: Amos Wenger <amos@bearcove.net>
2024-09-25 15:41:40 +00:00
konsti
5da73a24cb
Rename MetadataResolver to ResolutionMetadata (#7661) 2024-09-24 16:25:19 +00:00
konsti
484717d42f
Split metadata parsing into a module (#7656) 2024-09-24 17:16:21 +02:00
Charlie Marsh
445d1c0d43
Avoid validating workspace members when --no-sources is provided (#7599)
## Summary

Closes https://github.com/astral-sh/uv/issues/7572.
2024-09-20 19:03:31 +00:00
Charlie Marsh
f9b882939f
Make uv cache prune robust to unreadable rkyv entries (#7561)
## Summary

We're robust to these in the rest of the CLI, but not in `uv cache
prune`.
2024-09-19 20:48:20 +00:00
Charlie Marsh
f3463b3d08
Heal cache entries with missing source distributions (#7559)
## Summary

`uv cache prune --ci` will remove the source distribution directory. If
we then need to build a _different_ wheel (e.g., you're building a
package that has Python minor version-specific wheels), we fail, because
we expect the source to be there.

Now, if the source is missing, we re-download it. It would be slightly
easier to just _ignore_ that revision, but that would mean we'd also
lose the already-built wheels -- so if you ran against many Python
versions, we'd continuously lose the cached data.

Closes https://github.com/astral-sh/uv/issues/7543.

## Test Plan

We can add tests, but they _need_ to build non-pure Python wheels, which
tends to be expensive...

For reference:

```console
$ cargo run venv --python 3.12
$ cargo run pip install mercurial==6.8.1 --verbose
$ cargo run cache prune --ci
$ cargo run venv --python 3.11
$ cargo run pip install mercurial==6.8.1 --verbose
```

I also did this with a local `.tar.gz` that I downloaded from PyPI.
2024-09-19 20:31:55 +00:00
Charlie Marsh
e36cc99b0d
Use portable paths when serializing sources (#7504)
## Summary

Closes https://github.com/astral-sh/uv/issues/7493.
2024-09-18 18:51:14 +00:00
Charlie Marsh
97ae811b86
Avoid fatal error when searching for egg-info with missing directory (#7498)
## Summary

Closes https://github.com/astral-sh/uv/issues/7485.

## Test Plan

```
$ cargo run cache clean
$ cargo run venv
$ cargo run pip install django-allauth==0.51.0
$ cargo run venv
$ cargo run pip install django-allauth==0.51.0
```
2024-09-18 09:33:11 -04:00
Charlie Marsh
fda227616c
Allow users to provide pre-defined metadata for resolution (#7442)
## Summary

This PR enables users to provide pre-defined static metadata for
dependencies. It's intended for situations in which the user depends on
a package that does _not_ declare static metadata (e.g., a
`setup.py`-only sdist), and that is expensive to build or even cannot be
built on some architectures. For example, you might have a Linux-only
dependency that can't be built on ARM -- but we need to build that
package in order to generate the lockfile. By providing static metadata,
the user can instruct uv to avoid building that package at all.

For example, to override all `anyio` versions:

```toml
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["anyio"]

[[tool.uv.dependency-metadata]]
name = "anyio"
requires-dist = ["iniconfig"]
```

Or, to override a specific version:

```toml
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["anyio"]

[[tool.uv.dependency-metadata]]
name = "anyio"
version = "3.7.0"
requires-dist = ["iniconfig"]
```

The current implementation uses `Metadata23` directly, so we adhere to
the exact schema expected internally and defined by the standards. Any
entries are treated similarly to overrides, in that we won't even look
for `anyio@3.7.0` metadata in the above example. (In a way, this also
enables #4422, since you could remove a dependency for a specific
package, though it's probably too unwieldy to use in practice, since
you'd need to redefine the _rest_ of the metadata, and do that for every
package that requires the package you want to omit.)

This is under-documented, since I want to get feedback on the core ideas
and names involved.

Closes https://github.com/astral-sh/uv/issues/7393.
2024-09-18 03:18:05 +00:00
Charlie Marsh
c87ce7aaf8
Run cargo upgrade (#7448)
Co-authored-by: konstin <konstin@mailbox.org>
2024-09-17 12:39:58 +02:00
Charlie Marsh
6a9ec9665c
Include dev-dependencies with --no-sources (#7408)
## Summary

Running `uv lock --no-sources` should still include dev dependencies,
since dev dependencies are defined separately from sources.

Closes https://github.com/astral-sh/uv/issues/7406.
2024-09-15 22:29:03 -04:00
Charlie Marsh
083ec2f1bf
Error when tool.uv.sources contains duplicate package names (#7383)
## Summary

Closes https://github.com/astral-sh/uv/issues/7229.
2024-09-13 23:37:23 -04:00
Charlie Marsh
613a7d5c5d
Make version ID optional for source builds (#7362)
## Summary

Fixes a bug in which source builds would fail at the top-level of a
container.

Closes https://github.com/astral-sh/uv/issues/7346.
2024-09-13 10:52:25 -04:00
Charlie Marsh
65d53a7474
Use globwalk for cache-keys matching (#7337)
## Summary

This should be more efficient as we can do a single traversal.

Closes https://github.com/astral-sh/uv/issues/7321.
2024-09-12 15:06:05 -04:00
konsti
2b3890f2b4
Extract METADATA reading into a crate (#7231)
This is preparatory work for the upload functionality, which needs to
read the METADATA file and attach its parsed contents to the POST
request: We move finding the `.dist-info` from `install-wheel-rs` and
`uv-client` to a new `uv-metadata` crate, so it can be shared with the
publish crate.

I don't properly know if its the right place since the upload code isn't
ready, but i'm PR-ing it now because it already had merge conflicts.
2024-09-10 13:31:01 +00:00
Charlie Marsh
a8bd0211e0
Invalidate cache when --config-settings change (#7139)
## Summary

If `--config-settings` are provided, we cache the built wheels under one
more subdirectory.

We _don't_ invalidate the actual source (i.e., trigger a re-download) or
metadata, though -- those can be reused even when `--config-settings`
change.

Closes https://github.com/astral-sh/uv/issues/7028.
2024-09-10 01:49:16 +00:00
Charlie Marsh
4f2349119c
Add support for dynamic cache keys (#7136)
## Summary

This PR adds a more flexible cache invalidation abstraction for uv, and
uses that new abstraction to improve support for dynamic metadata.

Specifically, instead of relying solely on a timestamp, we now pass
around a `CacheInfo` struct which (as of now) contains
`Option<Timestamp>` and `Option<Commit>`. The `CacheInfo` is saved in
`dist-info` as `uv_cache.json`, so we can test already-installed
distributions for cache validity (along with testing _cached_
distributions for cache validity).

Beyond the defaults (`pyproject.toml`, `setup.py`, and `setup.cfg`
changes), users can also specify additional cache keys, and it's easy
for us to extend support in the future. Right now, cache keys can either
be instructions to include the current commit (for `setuptools_scm` and
similar) or file paths (for `hatch-requirements-txt` and similar):

```toml
[tool.uv]
cache-keys = [{ file = "requirements.txt" }, { git = true }]
```

This change should be fully backwards compatible.

Closes https://github.com/astral-sh/uv/issues/6964.

Closes https://github.com/astral-sh/uv/issues/6255.

Closes https://github.com/astral-sh/uv/issues/6860.
2024-09-09 20:19:15 +00:00
Charlie Marsh
9a7262c360
Avoid batch prefetching for un-optimized registries (#7226)
## Summary

We now track the discovered `IndexCapabilities` for each `IndexUrl`. If
we learn that an index doesn't support range requests, we avoid doing
any batch prefetching.

Closes https://github.com/astral-sh/uv/issues/7221.
2024-09-09 15:46:19 -04:00
Charlie Marsh
6179b65e37
Avoid removing entries during read_dir (#7151)
I think this is the source of the test flakiness.
2024-09-07 02:10:28 +00:00
Charlie Marsh
93fe3e83be
Prune unused source distributions from the cache (#7112)
## Summary

This has bothered me for a while and should be fairly impactful for
users. It requires a weird implementation, since the
distribution-building crate depends on the cache, and so the prune
operation can't live in the cache, since it needs to access internals of
the distribution-building crate.

Closes https://github.com/astral-sh/uv/issues/7096.
2024-09-05 21:40:51 -04:00
Charlie Marsh
05ed4bc11d
Show build output by default in uv build (#6912)
## Summary

This is a big improvement IMO:

![Screenshot 2024-09-01 at 12 52
28 PM](https://github.com/user-attachments/assets/2d8b1370-3385-429a-9a1d-e1d44611a2b4)
2024-09-04 15:39:21 +00:00
Zanie Blue
e3d5d3d26d
Avoid deadlocks when multiple uv processes lock resources (#6790)
This is achieved by updating the `LockedFile::acquire` API to be async —
as in some cases we were attempting to acquire the lock synchronously,
i.e., without yielding, which blocked the runtime.

Closes https://github.com/astral-sh/uv/issues/6691 — I tested with the
reproduction there and a local release build and no longer reproduce the
deadlock with these changes.

Some additional context in the [internal Discord
thread](1278478941)
2024-08-29 11:16:14 -05:00
Charlie Marsh
56cc0c9b3c
Avoid using editable tag in lockfile for non-package dependencies (#6728)
## Summary

Use a dedicated source type for non-package requirements. Also enables
us to support non-package `path` dependencies _and_ removes the need to
have the member `pyproject.toml` files available when we sync _and_
makes it explicit which dependencies are virtual vs. not (as evidenced
by the snapshot changes). All good things!
2024-08-28 01:19:05 +00:00
Charlie Marsh
14074f8775
Avoid reading stale .egg-info from mutable sources (#6714)
## Summary

In theory this problem already existed for `PKG-INFO`, but `egg-info`
would be more common, I think, since it's built in the source tree.

Closes https://github.com/astral-sh/uv/issues/6712.
2024-08-27 19:23:26 +00:00
Charlie Marsh
a999303d2f
Use PathBuf types in Source enum (#6708) 2024-08-27 14:46:39 -04:00
Charlie Marsh
d86075fc1e
Add support for --trusted-host (#6591)
## Summary

This PR revives https://github.com/astral-sh/uv/pull/4944, which I think
was a good start towards adding `--trusted-host`. Last night, I tried to
add `--trusted-host` with a custom verifier, but we had to vendor a lot
of `reqwest` code and I eventually hit some private APIs. I'm not
confident that I can implement it correctly with that mechanism, and
since this is security, correctness is the priority.

So, instead, we now use two clients and multiplex between them.

Closes https://github.com/astral-sh/uv/issues/1339.

## Test Plan

Created self-signed certificate, and ran `python3 -m http.server --bind
127.0.0.1 4443 --directory . --certfile cert.pem --keyfile key.pem` from
the packse index directory.

Verified that `cargo run pip install
transitive-yanked-and-unyanked-dependency-a-0abad3b6 --index-url
https://127.0.0.1:8443/simple-html` failed with:

```
error: Request failed after 3 retries
  Caused by: error sending request for url (https://127.0.0.1:8443/simple-html/transitive-yanked-and-unyanked-dependency-a-0abad3b6/)
  Caused by: client error (Connect)
  Caused by: invalid peer certificate: Other(OtherError(CaUsedAsEndEntity))
```

Verified that `cargo run pip install
transitive-yanked-and-unyanked-dependency-a-0abad3b6 --index-url
'https://127.0.0.1:8443/simple-html' --trusted-host '127.0.0.1:8443'`
failed with the expected error (invalid resolution) and made valid
requests.

Verified that `cargo run pip install
transitive-yanked-and-unyanked-dependency-a-0abad3b6 --index-url
'https://127.0.0.1:8443/simple-html' --trusted-host '127.0.0.2' -n` also
failed.
2024-08-27 09:36:50 -04:00
Charlie Marsh
ce749591de
Read requirements from requires.txt when available (#6655)
## Summary

Allows us to avoid building setuptools-based packages at versions prior
to Metadata 2.2

Closes https://github.com/astral-sh/uv/issues/6647.
2024-08-27 13:02:26 +00:00