This commit prepares to make the `source` and `version` fields optional
in a `distribution.dependency` based on whether they have an unambiguous
value. e.g., When there is exactly one distribution with a matching
package name.
This refactor effectively defines "wire" types for most of the lock data
types (repeating the `WheelWire` and `LockWire` pattern) with one key
difference: we don't use serde's `TryFrom` integration. In this
refactor, we could have, and it would have worked. But in a subsequent
commit, we're going to be adding state to the `unwire()` calls that is
impossible to thread through a `TryFrom` implementation. This state will
tell us how to populate the `source` and `version` values on a
`Dependency` when they're missing.
The duplication of types here is unfortunate, but compiler should catch
any deviations. And the wire types are unexported, so they have a
limited blast radius on complexity.
Downstack PR: #4481
## Introduction
We support forking the dependency resolution to support conflicting
registry requirements for different platforms, say on package range is
required for an older python version while a newer is required for newer
python versions, or dependencies that are different per platform. We
need to extend this support to direct URL requirements.
```toml
dependencies = [
"iniconfig @ 62565a6e1c/iniconfig-2.0.0-py3-none-any.whl ; python_version >= '3.12'",
"iniconfig @ b3c12c6d70/iniconfig-1.1.1-py2.py3-none-any.whl ; python_version < '3.12'"
]
```
This did not work because `Urls` was built on the assumption that there
is a single allowed URL per package. We collect all allowed URL ahead of
resolution by following direct URL dependencies (including path
dependencies) transitively, i.e. a registry distribution can't require a
URL.
## The same package can have Registry and URL requirements
Consider the following two cases:
requirements.in:
```text
werkzeug==2.0.0
werkzeug @ 960bb4017c/Werkzeug-2.0.0-py3-none-any.whl
```
pyproject.toml:
```toml
dependencies = [
"iniconfig == 1.1.1 ; python_version < '3.12'",
"iniconfig @ git+https://github.com/pytest-dev/iniconfig@93f5930e668c0d1ddf4597e38dd0dea4e2665e7a ; python_version >= '3.12'",
]
```
In the first case, we want the URL to override the registry dependency,
in the second case we want to fork and have one branch use the registry
and the other the URL. We have to know about this in
`PubGrubRequirement::from_registry_requirement`, but we only fork after
the current method.
Consider the following case too:
a:
```
c==1.0.0
b @ https://b.zip
```
b:
```
c @ https://c_new.zip ; python_version >= '3.12'",
c @ https://c_old.zip ; python_version < '3.12'",
```
When we convert the requirements of `a`, we can't know the url of `c`
yet. The solution is to remove the `Url` from `PubGrubPackage`: The
`Url` is redundant with `PackageName`, there can be only one url per
package name per fork. We now do the following: We track the urls from
requirements in `PubGrubDependency`. After forking, we call
`add_package_version_dependencies` where we apply override URLs, check
if the URL is allowed and check if the url is unique in this fork. When
we request a distribution, we ask the fork urls for the real URL. Since
we prioritize url dependencies over registry dependencies and skip
packages with `Urls` entries in pre-visiting, we know that when fetching
a package, we know if it has a url or not.
## URL conflicts
pyproject.toml (invalid):
```toml
dependencies = [
"iniconfig @ e96292c7f7/iniconfig-1.1.0.tar.gz",
"iniconfig @ b3c12c6d70/iniconfig-1.1.1-py2.py3-none-any.whl ; python_version < '3.12'",
"iniconfig @ 62565a6e1c/iniconfig-2.0.0-py3-none-any.whl ; python_version >= '3.12'",
]
```
On the fork state, we keep `ForkUrls` that check for conflicts after
forking, rejecting the third case because we added two packages of the
same name with different URLs.
We need to flatten out the requirements before transformation into
pubgrub requirements to get the full list of other requirements which
may contain a URL, which was changed in a previous PR: #4430.
## Complex Example
a:
```toml
dependencies = [
# Force a split
"anyio==4.3.0 ; python_version >= '3.12'",
"anyio==4.2.0 ; python_version < '3.12'",
# Include URLs transitively
"b"
]
```
b:
```toml
dependencies = [
# Only one is used in each split.
"b1 ; python_version < '3.12'",
"b2 ; python_version >= '3.12'",
"b3 ; python_version >= '3.12'",
]
```
b1:
```toml
dependencies = [
"iniconfig @ b3c12c6d70/iniconfig-1.1.1-py2.py3-none-any.whl",
]
```
b2:
```toml
dependencies = [
"iniconfig @ 62565a6e1c/iniconfig-2.0.0-py3-none-any.whl",
]
```
b3:
```toml
dependencies = [
"iniconfig @ e96292c7f7/iniconfig-1.1.0.tar.gz",
]
```
In this example, all packages are url requirements (directory
requirements) and the root package is `a`. We first split on `a`, `b`
being in each split. In the first fork, we reach `b1`, the fork URLs are
empty, we insert the iniconfig 1.1.1 URL, and then we skip over `b2` and
`b3` since the mark is disjoint with the fork markers. In the second
fork, we skip over `b1`, visit `b2`, insert the iniconfig 2.0.0 URL into
the again empty fork URLs, then visit `b3` and try to insert the
iniconfig 1.1.0 URL. At this point we find a conflict for the iniconfig
URL and error.
## Closing
The git tests are slow, but they make the best example for different URL
types i could find.
Part of #3927. This PR does not handle `Locals` or pre-releases yet.
In the last PR (#4430), we flatten the requirements. In the next PR
(#4435), we want to pass `Url` around next to `PubGrubPackage` and
`Range<Version>` to keep track of which `Requirement`s added a url
across forking. This PR is a refactoring split out from #4435 that rolls
the dependency conversion into a single iterator and introduces a new
`PubGrubDependency` struct as abstraction over `(PubGrubPackage,
Range<Version>)` (or `(PubGrubPackage, Range<Version>,
VerbatimParsedUrl)` in the next PR), and it removes the now unnecessary
`PubGrubDependencies` abstraction.
This PR refactors the command creation in the test suite to remove the
duplication.
**1)** We add the same set of test stubbing args to almost any uv
invocation in the tests:
```rust
command
.arg("--cache-dir")
.arg(self.cache_dir.path())
.env("VIRTUAL_ENV", self.venv.as_os_str())
.env("UV_NO_WRAP", "1")
.env("HOME", self.home_dir.as_os_str())
.env("UV_TOOLCHAIN_DIR", "")
.env("UV_TEST_PYTHON_PATH", &self.python_path())
.current_dir(self.temp_dir.path());
if cfg!(all(windows, debug_assertions)) {
// TODO(konstin): Reduce stack usage in debug mode enough that the tests pass with the
// default windows stack of 1MB
command.env("UV_STACK_SIZE", (8 * 1024 * 1024).to_string());
}
```
Centralizing these into a `TestContext::add_shared_args` method removes
them from everywhere.
**2)** Prefix all `TextContext` methods of the pip interface with
`pip_`. This is now necessary due to `uv sync` vs. `uv pip sync`.
**3)** Move command creation in the various test files into dedicated
functions or methods to avoid repeating the arguments. Except for error
message tests, there should be at most one `Command::new(get_bin())`
call per test file. `EXCLUDE_NEWER` is exclusively used in
`TestContext`.
---
I'm considering adding a `TestCommand` on top of these changes (in
another PR) that holds a reference to the `TextContext`, has
`add_shared_args` as a method and uses `Fn(Self) -> Self` instead of
`Fn(&mut Self) -> Self` for methods to improved chaining.
Downstack PR: #4515 Upstack PR: #4481
Consider these two cases:
A:
```
werkzeug==2.0.0
werkzeug @ 960bb4017c/Werkzeug-2.0.0-py3-none-any.whl
```
B:
```toml
dependencies = [
"iniconfig == 1.1.1 ; python_version < '3.12'",
"iniconfig @ git+https://github.com/pytest-dev/iniconfig@93f5930e668c0d1ddf4597e38dd0dea4e2665e7a ; python_version >= '3.12'",
]
```
In the first case, `werkzeug==2.0.0` should be overridden by the url. In
the second case `iniconfig == 1.1.1` is in a different fork and must
remain a registry distribution.
That means the conversion from `Requirement` to `PubGrubPackage` is
dependent on the other requirements of the package. We can either look
into the other packages immediately, or we can move the forking before
the conversion to `PubGrubDependencies` instead of after. Either version
requires a flat list of `Requirement`s to use. This refactoring gives us
this list.
I'll add support for both of the above cases in the forking urls branch
before merging this PR. I also have to move constraints over to this.
While working on https://github.com/astral-sh/uv/pull/4492 I noticed
that `--reinstall-package` was not actually respected by
`update_environment`, it exited early due to satisfied requirements.
Before
```
❯ cargo run -q -- tool install black -v --reinstall-package tomli
...
DEBUG All requirements satisfied: black | click>=8.0.0 | mypy-extensions>=0.4.3 | packaging>=22.0 | pathspec>=0.9.0 | platformdirs>=2 | tomli>=1.1.0 ; python_version < '3.11' | typing-extensions>=4.0.1 ; python_version < '3.11'
```
After
```
❯ cargo run -q -- tool install black -v --reinstall-package tomli
...
Uninstalled 1 package in 0.99ms
Installed 1 package in 4ms
- tomli==2.0.1
+ tomli==2.0.1
```
## Summary
This is an intermediary change in enabling universal resolution for
`requirements.txt` files. To start, we need to be able to preserve
markers in the `requirements.txt` output _and_ propagate those markers,
such that if you have a dependency that's only included with a given
marker, the transitive dependencies respect that marker too.
Closes#1429.
## Summary
If the user puts their configuration in a `pyproject.toml` that _isn't_
a valid workspace root (e.g., it's a Poetry file), we won't discover it,
because we only look in `uv.toml` files in that case. I think this is
somewhat debatable... We could choose to _require_ `uv.toml` there, but
as a user I'd probably expect it to work?
Closes https://github.com/astral-sh/uv/issues/4521.
`junction::create` apparently will happily succeed but not create a link
to files? Since our symlink function does not indicate that it cannot
handle files, this was quite surprising.
Tested over in #4509 which previously failed on an assertion that
`black.exe` existed.
```
error: Failed to install entrypoint
Caused by: Cannot create a junction for [TEMP_DIR]/tools/black/Scripts/black.exe: is not a directory
```
We should file an issue upstream too, I think?
## Summary
We currently accept `--index-url /path/to/index` on the command line,
but confusingly, not in `requirements.txt`. This PR just brings the two
in sync.
## Test Plan
New snapshot tests.
## Summary
pip allows these with the following logic:
```python
if os.path.exists(location): # Is a local path.
url = path_to_url(location)
path = location
elif location.startswith("file:"): # A file: URL.
url = location
path = url_to_path(location)
elif is_url(location):
url = location
```
Closes https://github.com/astral-sh/uv/issues/4510.
## Test Plan
`cargo run pip install --index-url ../packse/index/simple-html/
example-a-961b4c22 --reinstall --no-cache --no-deps`
I was getting this test failure locally on my Archlinux system:
```
-old snapshot
+new results
0 0 │ success: true
1 1 │ exit_code: 0
2 2 │ ----- stdout -----
3 │-[PYTHON-3.12]
3 │+/usr/bin/python3
4 4 │
5 5 │ ----- stderr -----
```
Where I have `/usr/bin/python3` and `/usr/bin/python3.12`.
Thanks @zanieb for the help with figuring out the fix here!
So we can skip them when there are not code changes and still enforce
our required checks (xref #4438)
Otherwise, the names are dynamic and they are forever expected (see
#4426 for example)
## Summary
In #4085, support was implemented for the `--prefix` option. When using
this option, however, a lock is either acquired on the virtualenv or
globally, preventing multiple installs to different `--prefix`s from the
same interpreter.
In this change, acquire the lock on just the prefix in question.
## Test Plan
Ran a `uv pip install` with `--prefix` and `RUST_LOG=trace` and observed
that the lock was acquired in the prefix.
## Summary
It turns out that the Git fetch implementation is initializing its own
client, which can be really expensive on macOS (due to loading native
certificates) _and_ bypasses any of our middleware. This PR modifies the
Git implementation to accept a shared client.
## Summary
We unconditionally update the submodules in our Git code, but AFAICT it
shouldn't be necessary if we already have a complete, up-to-date fetch
available.
## Summary
PEP 517 build isolation https://github.com/astral-sh/uv/pull/843 has not
yet been mentioned in the PIP compatibility documentation. Add a section
for it.
## Test Plan
Visual inspection only
## Open Questions
> in most cases, swapping out `pip install` for `uv pip install` should
"just work".
In the first non-trivial case I tried, it worked for a short time and
then [started
failing](https://github.com/astral-sh/uv/issues/4069#issuecomment-2186762048).
Is there any data out there on how many top 100 or top 1000 packages
work with PEP 517 build isolation?
How can someone specify `--no-build-isolation` for just one package/line
in `requirements.txt`?
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
This does require cloning the settings, but I think it's fine. A better
solution would be to have owned and unowned settings structs, so that we
could convert `ResolverInstallerSettingsRef` to `InstallerSettingsRef`
without cloning, but that requires maintaining owned and unowned
variants.
Closes https://github.com/astral-sh/uv/issues/4455.
## Summary
I might be mistaken, but I think we need to read the header from the
response, not the request. The request would only contain headers that
we set.
I verified (with extra logging) that the request header is `None` while
PyPI returns a valid length in the response header.
## Summary
Ultimately decided to view this as part of `LockWire` normalization:
removing references to extras that don't exist. I think it would be nice
if the resolver avoided omitting these, but I don't know if it's fully
possible.
Closes https://github.com/astral-sh/uv/issues/4405.
I have to add yet another indentation level to the prerelease-available
check in `PubGrubReportFormatter::hints` for #4435, so i've broken the
code into methods and decreased indentation in this split out
refactoring-only change.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Resolves https://github.com/astral-sh/uv/issues/4439 partially.
Implements for `uv pip tree`:
- `--no-dedupe` flag, similar to `cargo tree --no-dedupe` .
- denote dependency cycles with `(#)` and add a footnote if there's a
cycle (using `(*)` would require keeping track of the cycle state, so
opted to do this instead).
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
The existing tests pass + added a couple of tests to validate
`--no-dedupe` behavior.
<!-- How was it tested? -->
## Summary
This will make `uv lock` read `override-dependencies` from the
`[tool.uv]` section of `pyproject.toml`.
Resolves#4108
This [other](https://github.com/astral-sh/uv/pull/4446) implementation
touches more code but seems more consistent.
## Test Plan
Unit test
Releasing 0.2.15 with a few additions over 0.2.14. Motivated by the
incorrect tagging of 0.2.14 (#4474).
Generated the changelog with a small patch to Rooster allowing me to
force the previous commit to be correct.
```diff
diff --git a/src/rooster/_cli.py b/src/rooster/_cli.py
index 2a4f61b..4ec1299 100644
--- a/src/rooster/_cli.py
+++ b/src/rooster/_cli.py
@@ -38,6 +38,7 @@ def release(
without_sections: list[str] = typer.Option(
[], help="Sections to exclude from the changelog"
),
+ previous_commit: str = None,
):
"""
Create a new release.
@@ -58,7 +59,11 @@ def release(
typer.echo("It looks like there are no version tags for this project.")
# Get the commits since the last release
- changes = list(get_commits_between(config, repo, last_version))
+ changes = list(
+ get_commits_between(
+ config, repo, last_version, force_first_commit=previous_commit
+ )
+ )
since = "since last release" if last_version else "in the project"
typer.echo(f"Found {len(changes)} commits {since}.")
diff --git a/src/rooster/_git.py b/src/rooster/_git.py
index 597bb88..66bc54e 100644
--- a/src/rooster/_git.py
+++ b/src/rooster/_git.py
@@ -29,12 +29,13 @@ def get_commits_between(
target: Path,
first_version: Version | None = None,
second_version: Version | None = None,
+ force_first_commit: str | None = None,
) -> Generator[git.Commit, None, None]:
"""
Yield all commits between two tags
"""
repo = git.repository.Repository(target.absolute())
- first_commit = (
+ first_commit = force_first_commit or (
repo.lookup_reference(
TAG_PREFIX + config.version_tag_prefix + str(first_version)
)
```
## Summary
The `--index-strategy` is linked to the index locations, which we
propagate to source distribution builds; so it makes sense to pass the
`--index-strategy` too.
While I was here, I made `exclude_newer` a required argument so that we
don't forget to set it via the `with_options` builder.
Closes https://github.com/astral-sh/uv/issues/4465.
## Summary
This PR moves all the CLI code into its own crate, separate from the
`uv` crate. The `uv` crate is iterated on frequently, and the CLI code
comprises a significant portion of it but rarely changes. Removing the
CLI code reduces the `uv` crate size from 1.4MiB to 1.0MiB.
