We retry several kinds of network request failures, but it's often
unclear whether a request was retried or not
(https://github.com/astral-sh/uv/issues/3514#issuecomment-2105485773).
This PR adds a small intermediary layer that logs all transient request
failures, adding the `DEBUG Transient request failure` lines:
```
DEBUG Searching for Python interpreter in virtual environments
DEBUG Found CPython 3.12.3 at `/home/konsti/projects/uv/.venv/bin/python3` (active virtual environment)
DEBUG Using Python 3.12.3 environment at .venv/bin/python3
DEBUG Acquired lock for `.venv`
DEBUG At least one requirement is not satisfied: tqdm
DEBUG Using registry request timeout of 30s
DEBUG Solving with target Python version 3.12.3
DEBUG Adding direct dependency: tqdm*
DEBUG No cache entry for: https://pypi.org/simple/tqdm/
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
error: Could not connect, are you offline?
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
```
I decided for multi-line logging to show the complete error trace since
only `Transient request failure for https://pypi.org/simple/tqdm/,
retrying: Request error: error sending request for url
(https://pypi.org/simple/tqdm/)` doesn't tell you the actual problem (a
dns error).
Note that running with `-v` will not show messages about retry backoff
timing, but running with `RUST_LOG=debug` now shows a complete picture:
```
DEBUG starting new connection: https://pypi.org/
DEBUG resolving host="pypi.org"
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
WARN Retry attempt #2. Sleeping 528.728192ms before the next attempt
```
Fixes#3572
The workspace test directories can be used both in tests and directly
for developing/debugging. In the latter, we shouldn't copy the venv and
the lockfile when running tests. Using the ignore crate over manual
recursion we exclude those files.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
See #3834 .
This PR adds a new namespace, `override-dependencies`, to
pyproject.toml/uv.toml.
This namespace assumes that the dependencies you want to override are
written in the form of `requirements.txt`.
a example of pyproject.toml
```toml
[project]
name = "example"
version = "0.0.0"
dependencies = [
"flask==3.0.0"
]
[tool.uv]
override-dependencies = [
"werkzeug==2.3.0"
]
```
This will improve usability by allowing you to override dependencies
without having to specify the --override option when running `uv pip
compile/install`.
## Test Plan
added test to `crates/uv/tests/pip_compile.rs`.
---------
Co-authored-by: konstin <konstin@mailbox.org>
Add a `--package` option that allows switching the current project in
the workspace. Wherever you are in a workspace, you should be able to
run with any other project as root. This is the uv equivalent of `cargo
run -p`.
I don't love the `--package` name, esp. since `-p` is already taken and
in general to many things start with p already.
Part of this change is moving the workspace discovery of
`ProjectWorkspace` to `Workspace` itself.
## Usage
In albatross-virtual-workspace:
```console
$ uv venv
$ uv run --preview --package bird-feeder python -c "import albatross"
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds
Built 2 editables in 167ms
Resolved 5 packages in 4ms
Installed 5 packages in 1ms
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds)
+ sniffio==1.3.1
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'albatross'
$ uv venv
$ uv run --preview --package albatross python -c "import albatross"
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/albatross
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds
Built 3 editables in 173ms
Resolved 7 packages in 6ms
Installed 7 packages in 1ms
+ albatross==0.1.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/albatross)
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds)
+ sniffio==1.3.1
+ tqdm==4.66.4
```
In albatross-root-workspace:
```console
$ uv venv
$ uv run --preview --package bird-feeder python -c "import albatross"
Using Python 3.12.3 interpreter at: /home/konsti/.local/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.10s
Running `/home/konsti/projects/uv/target/debug/uv run --preview --package bird-feeder python -c 'import albatross'`
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds Built 2 editables in 161ms
Resolved 5 packages in 4ms
Installed 5 packages in 1ms
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds)
+ sniffio==1.3.1
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'albatross'
$ uv venv
$ cargo run run --preview --package albatross python -c "import albatross"
Using Python 3.12.3 interpreter at: /home/konsti/.local/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.13s
Running `/home/konsti/projects/uv/target/debug/uv run --preview --package albatross python -c 'import albatross'`
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds
Built 3 editables in 168ms
Resolved 7 packages in 5ms
Installed 7 packages in 1ms
+ albatross==0.1.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace)
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds)
+ sniffio==1.3.1
+ tqdm==4.66.4
```
## Summary
This PR removes the static resolver map:
```rust
static RESOLVED_GIT_REFS: Lazy<Mutex<FxHashMap<RepositoryReference, GitSha>>> =
Lazy::new(Mutex::default);
```
With a `GitResolver` struct that we now pass around on the
`BuildContext`. There should be no behavior changes here; it's purely an
internal refactor with an eye towards making it cleaner for us to
"pre-populate" the list of resolved SHAs.
## Summary
This will help prevent bugs like #3934 by unifying the implementations
for editables and non-editable unnamed requirements. Specifically, both
of these now go through the same parsing paths and use the same struct
representations (with the exception that the editable flag is flipped in
the first case):
```
-e ./foo/bar
./foo/bar
```
We also now support PEP 508 in editable URLs. It turns out this is just
a limitation in pip, so it's correct to support it. For example, this
now works:
```
-e black[d] @ file://${PROJECT_ROOT}/scripts/packages/black_editable
```
Closes#3941.
Closes#3942.
With the change, we remove the special casing of workspace dependencies
and resolve `tool.uv` for all git and directory distributions. This
gives us support for non-editable workspace dependencies and path
dependencies in other workspaces. It removes a lot of special casing
around workspaces. These changes are the groundwork for supporting
`tool.uv` with dynamic metadata.
The basis for this change is moving `Requirement` from
`distribution-types` to `pypi-types` and the lowering logic from
`uv-requirements` to `uv-distribution`. This changes should be split out
in separate PRs.
I've included an example workspace `albatross-root-workspace2` where
`bird-feeder` depends on `a` from another workspace `ab`. There's a
bunch of failing tests and regressed error messages that still need
fixing. It does fix the audited package count for the workspace tests.
## Summary
In general, it's not quite right to filter preferences by `--reinstall`
-- we still want to respect existing versions, we just don't want to
respect _installed_ versions. But now that the installed versions and
preferences are decoupled, we can remove this (`--reinstall` is enforced
on the installed versions via the `Exclusions` struct that we pass to
the resolver).
While I was here, I also cleaned up the lockfile preference code to
better match the structure for `requirements.txt`.
## Summary
We currently rely on libgit2 for most git-related functionality.
However, libgit2 has long-standing performance issues, as well as lags
significantly behind git in terms of new features. For these reasons we
now use the git CLI by default for fetching repositories
(https://github.com/astral-sh/uv/pull/1781). This PR completely drops
libgit2 in favor of the git CLI for all git-related functionality, which
should allow us to use features such as partial clones and sparse
checkouts in the future for performance.
There is also a lot of technical debt in the current git code as it's
mostly taken from Cargo. Switching to the git CLI *vastly* simplifies
the `uv-git` codebase.
Eventually we might want to look into switching to
[`gitoxide`](https://github.com/Byron/gitoxide), but it's currently too
immature for our use case.
## Summary
This PR changes the lock-file format to use inline tables for wheels and
source distributions, which currently use separate tables that make the
file harder to follow.
```diff
[[distribution]]
name = "typing-extensions"
version = "4.10.0"
source = "registry+https://pypi.org/simple"
- [distribution.sdist]
- url = "0d26ce356c/typing_extensions-4.10.0.tar.gz"
- hash = "sha256:b0abd7c89e8fb96f98db18d86106ff1d90ab692004eb746cf6eda2682f91b3cb"
- size = 77558
-
- [[distribution.wheel]]
- url = "dc04a3ea60/typing_extensions-4.10.0-py3-none-any.whl"
- hash = "sha256:69b1a937c3a517342112fb4c6df7e72fc39a38e7891a5730ed4985b5214b5475"
- size = 33926
+ sdist = { url = "0d26ce356c/typing_extensions-4.10.0.tar.gz", hash = "sha256:b0abd7c89e8fb96f98db18d86106ff1d90ab692004eb746cf6eda2682f91b3cb", size = 77558 }
+ wheel = [{ url = "dc04a3ea60/typing_extensions-4.10.0-py3-none-any.whl", hash = "sha256:69b1a937c3a517342112fb4c6df7e72fc39a38e7891a5730ed4985b5214b5475", size = 33926 }]
```
The downside is that the inline-tables end up quite long and TOML
doesn't support line breaks in inline tables, yet.
Part of https://github.com/astral-sh/uv/issues/3611.
## Summary
There are a few behavior changes in here:
- We now enforce `--require-hashes` for editables, like pip. So if you
use `--require-hashes` with an editable requirement, we'll reject it. I
could change this if it seems off.
- We now treat source tree requirements, editable or not (e.g., both `-e
./black` and `./black`) as if `--refresh` is always enabled. This
doesn't mean that we _always_ rebuild them; but if you pass
`--reinstall`, then yes, we always rebuild them. I think this is an
improvement and is close to how editables work today.
Closes#3844.
Closes#2695.
Add workspace support when using `-r <path>/pyproject.toml` or `-e
<path>` in the pip interface. It is limited to all-editable
static-metadata workspaces, and tests only include a single main
workspace, ignoring path dependencies in another workspace. This can be
considered the MVP for workspace support: You can create a workspace,
you can install from it, but some options and conveniences are still
missing. I'll file follow-up tickets (support in lockfiles, support path
deps in other workspace, #3625)
There is also support in `uv run`, but we need
https://github.com/astral-sh/uv/issues/3700 first to properly support
using different current projects in the bluejay interface, currently the
resolution and therefore the lockfile depends on the current project.
I'd do this change first (it's big enough already), then #3700, and then
add workspace support properly to bluejay.
Fixes#3404
This bumps the versions of pep580 and pep440 to coincide with the
crates.io versions. While not strictly the same, the new types in uv us
an `Inner` struct. Practically I've found I'm still able to use the
patched versions, as can seen from the open PR here:
https://github.com/prefix-dev/pixi/pull/1436.
Would be great if this bump can be done so we can keep combining the
types :)
Extends #3726
Moves toolchain storage out of `UV_BOOTSTRAP_DIR` (`./bin`) into the
proper user data directory as defined by #3726.
Replaces `UV_BOOTSTRAP_DIR` with `UV_TOOLCHAIN_DIR` for customization.
Installed toolchains will be discovered without opt-in, but the idea is
still that these are not yet user-facing.
## Summary
This PR makes a variety of invalid states unrepresentable by changing
`Preference` to require a `PackageName` and `Version`, rather than
accepting a generic `Requirement`. There should be no meaningful
behavior changes.
## Summary
I haven't tested on Windows yet, but the idea here is that we should use
a portable representation when printing paths.
I decided to limit the scope here to paths that we write to output
files.
Closes https://github.com/astral-sh/uv/issues/3800.
When parsing requirements from any source, directly parse the url parts
(and reject unsupported urls) instead of parsing url parts at a later
stage. This removes a bunch of error branches and concludes the work
parsing url parts once and passing them around everywhere.
Many usages of the assembled `VerbatimUrl` remain, but these can be
removed incrementally.
Please review commit-by-commit.
## Summary
This seems to be one of the most consistent benchmark cases we have in
terms of standard deviation:
```
➜ hyperfine "target/profiling/main pip compile scripts/requirements/airflow.in" --runs 200
Benchmark 1: target/profiling/main pip compile scripts/requirements/airflow.in
Time (mean ± σ): 292.6 ms ± 6.6 ms [User: 414.1 ms, System: 194.2 ms]
Range (min … max): 282.7 ms … 320.1 ms 200 runs
```
For smaller benchmarks, scispacy and dtlssocket seem to be a bit more
consistent than our current jupyter benchmark, but it hasn't given us
any problems so I'll leave it for now.
## Summary
We now show yanks as part of the resolution diagnostics, so they now
appear for `sync`, `install`, `compile`, and any other operations.
Further, they'll also appear for cached packages (but not packages that
are _already_ installed).
Closes https://github.com/astral-sh/uv/issues/3768.
Closes#3766.
Updates our Python interpreter discovery to conform to the rules
described in #2386, please see that issue for a full description of the
behavior. Briefly, we now will search for interpreters that satisfy a
requested version without stopping at the first Python executable.
Additionally, if retrieving information about an interpreter fails we
will continue to search for a working interpreter. We also add the
plumbing necessary to request Python implementations other than CPython,
though we do not add support for other implementations at this time.
A major internal goal of this work is to prepare for user-facing managed
toolchains i.e. fetching a requested version during `uv run`. These APIs
are not introduced, but there is some managed toolchain handling as
required for our test suite.
Some noteworthy implementation changes:
- The `uv_interpreter::find_python` module has been removed in favor of
a `uv_interpreter::discovery` module.
- There are new types to help structure interpreter requests and track
sources
- Executable discovery is implemented as a big lazy iterator and is a
central authority for source precedence
- `uv_interpreter::Error` variants were split into scoped types in each
module
- There's much more unit test coverage, but not for Windows yet
Remaining work:
- [x] Write new test cases
- [x] Determine correct behavior around executables in the current
directory
- _Future_: Combine `PythonVersion` and `VersionRequest`
- _Future_: Consider splitting `ManagedToolchain` into local and remote
variants
- _Future_: Add Windows unit test coverage
- _Future_: Explore behavior around implementation precedence (i.e.
CPython over PyPy)
Refactors split into:
- #3329
- #3330
- #3331
- #3332Closes#2386
Add minimal support for workspace discovery, only used for determining
paths in the bluejay commands.
We can now discover the workspace structure, namely that the
`pyproject.toml` of a package belongs to a workspace `pyproject.toml`
with members and exclusion. The globbing logic is inspired by cargo. We
don't resolve `workspace = true` metadata declarations yet.
## Summary
Restore API-compatibility with pre-1.1.0 versions of the `zip` crate,
and pin the dependency to the 0.6 series, due to concerns discussed in
https://github.com/astral-sh/uv/issues/3642.
## Test Plan
```
cargo run -p uv-dev -- fetch-python
cargo test
```
