## Summary
This is an alternative to #14003 that takes advantage of the fact that
we already validate that the requirements are up-to-date when validating
the lockfile, and the requirements for pinned requirements include the
index itself -- so rather than collecting all the explicit indexes
upfront, we can just add them to the available list as we iterate over
the lockfile's dependency graph.
This gets all the tests passing from that PR, but with ~no performance
impact and a much less invasive change. It also gets the "circular
dependency" test passing, which is marked with a TODO in that PR.
Closes https://github.com/astral-sh/uv/issues/11419.
If a user specifies `-e /path/to/dir` and `/path/to/dir` in a `uv pip
install` command, we want the editable to "win" (rather than erroring
due to conflicting URLs). Unfortunately, this behavior meant that when
you requested a package as editable and non-editable in conflicting
groups, the editable version was _always_ used. This PR modifies the
requisite types to use `Option<bool>` rather than `bool` for the
`editable` field, so we can determine whether a requirement was
explicitly requested as editable, explicitly requested as non-editable,
or not specified (as in the case of `/path/to/dir` in a
`requirements.txt` file). In the latter case, we allow editables to
override the "unspecified" requirement.
If a project includes a path dependency twice, once with `editable =
true` and once without any `editable` annotation, those are now
considered conflicting URLs, and lead to an error, so I've marked this
change as breaking.
Closes https://github.com/astral-sh/uv/issues/14139.
Reverts:
- #14349
- #14346
- #14245
Retains the test cases. Includes a `find-links` test case.
Supersedes
- https://github.com/astral-sh/uv/pull/14387
- https://github.com/astral-sh/uv/pull/14503
We originally got a report at
https://github.com/astral-sh/uv/issues/13707 that inclusion of a
trailing slash on an index URL was causing lockfile churn despite having
no semantic meaning and resolved the issue by adding normalization that
stripped trailing slashes at parse time.
We then discovered that, while there are not semantic differences for
trailing slashes on Simple API index URLs, there are differences for
some flat (or find links) indexes. As reported in
https://github.com/astral-sh/uv/issues/14367, the change in
https://github.com/astral-sh/uv/pull/14245 caused a regression for at
least one user.
We attempted to fix the regression via a few approaches.
https://github.com/astral-sh/uv/pull/14387 attempted to differentiate
between Simple API and flat index URL parsing, but failed to account for
the `Deserialize` implementation, which always assumed Simple API-style
index URLs and incorrectly trimmed trailing slashes in various cases
where we deserialized the `IndexUrl` type from a file. I attempted to
resolve this by performing a larger refactor, but it ended up being
quite painful. In particular, the `Index` type was a blocker — we don't
know the `IndexUrl` variant until we've parsed the `IndexFormat` and
having a multi-stage deserializer is not appealing but adding a new
intermediate type (i.e., `RawIndex`) is painful due to the pervasiveness
of `Index`. Given that we've regressed behavior here and there's not a
straight-forward fix, we're reverting the normalization entirely.
https://github.com/astral-sh/uv/pull/14503 attempted to perform
normalization at compare-time, but that means we'd fail to invalidate
the lockfile when the a trailing slash was added or removed and given
that a trailing slash has semantic meaning for a find-links URL... we'd
have another correctness problem.
After this revert, we'll retain all index URLs verbatim. The downside to
this approach is that we'll be adding a bunch of trailing slashes back
to lockfiles that we previously normalized out, and we'll be reverting
our fix for users with inconsistent trailing slashes on their index
URLs. Users affected by the original motivating issue should use
consistent trailing slashes on their URLs, as they do frequently have
semantic meaning. We may want to revisit normalization and type aware
index URL parsing as part of a larger change.
Closes https://github.com/astral-sh/uv/issues/14367
This PR updates `IndexUrl` parsing to normalize non-file URLs by
removing trailing slashes. It also normalizes registry source URLs when
using them to validate the lockfile.
Prior to this change, when writing an index URL to the lockfile, uv
would use a trailing slash if present in the provided URL and no
trailing slash otherwise. This can cause surprising behavior. For
example, `uv lock --locked` will fail when a package is added with an
`--index` value without a trailing slash and then `uv lock --locked` is
run with a `pyproject.toml` version of the index URL that contains a
trailing slash. This PR fixes this and adds a test for the scenario.
It might be safe to normalize file URLs in the same way, but since
slashes have a well-defined meaning in the context of files and
directories, I chose not to normalize them here.
Closes#13707.
When using `uv lock --upgrade-package=python` after changing
`requires-python`, it was possible to get into a state where the fork
markers produced corresponded to the empty set. This in turn resulted in
an empty lock file.
There was already some infrastructure in place that I think was perhaps
intended to handle this. In particular, `Lock::check_marker_coverage`
checks whether the fork markers have some overlap with the supported
environments (including the `requires-python`). But there were two
problems with this.
First is that in lock validation, this marker coverage check came
_after_ a path that returned `Preferable` (meaning that the fork markers
should be kept) when `--upgrade-package` was used. Second is that the
marker coverage check used the `requires-python` in the lock file and
_not_ the `requires-python` in the now updated `pyproject.toml`.
We attempt to solve this conundrum by slightly re-arranging lock file
validation and by explicitly checking whether the *new*
`requires-python` is disjoint from the fork markers in the lock file. If
it is, then we return `Versions` from lock file validation (indicating
that the fork markers should be dropped).
Fixes#13951
This allows you to specify requires-python on individual dependency-groups,
with the intended usecase being "oh my dev-dependencies have a higher
requires-python than my actual project".
This includes a large driveby move of the RequiresPython type to
uv-distribution-types to allow us to generate the appropriate markers at
this point in the code. It also migrates RequiresPython from
pubgrub::Range to version_ranges::Ranges, and makes several pub(crate)
items pub, as it's no longer defined in uv_resolver.
Fixes#11606
For the case where there was no matching wheel on sync, we previously
added a note about which wheels are available vs. on which platform you
are on. We extend this error message to link directly towards
`tool.uv.required-environments`, which otherwise has a discovery
problem.
On Linux (Setting `tool.uv.required-environments` doesn't help here
either, but it's a clear example):
```
[project]
name = "debug"
version = "0.1.0"
requires-python = "==3.10.*"
dependencies = ["tensorflow-macos>=2.13.1"]
```
```
Resolved 41 packages in 24ms
error: Distribution `tensorflow-macos==2.16.2 @ registry+https://pypi.org/simple` can't be installed because it doesn't have a source distribution or wheel for the current platform
hint: You're on Linux (`manylinux_2_39_x86_64`), but there are no wheels for the current platform, consider configuring `tool.uv.required-environments`.
hint: `tensorflow-macos` (v2.16.2) only has wheels for the following platform: `macosx_12_0_arm64`.
```
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Prior to this PR, there were numerous places where uv would leak
credentials in logs. We had a way to mask credentials by calling methods
or a recently-added `redact_url` function, but this was not secure by
default. There were a number of other types (like `GitUrl`) that would
leak credentials on display.
This PR adds a `DisplaySafeUrl` newtype to prevent leaking credentials
when logging by default. It takes a maximalist approach, replacing the
use of `Url` almost everywhere. This includes when first parsing config
files, when storing URLs in types like `GitUrl`, and also when storing
URLs in types that in practice will never contain credentials (like
`DirectorySourceUrl`). The idea is to make it easy for developers to do
the right thing and for the compiler to support this (and to minimize
ever having to manually convert back and forth). Displaying credentials
now requires an active step. Note that despite this maximalist approach,
the use of the newtype should be zero cost.
One conspicuous place this PR does not use `DisplaySafeUrl` is in the
`uv-auth` crate. That would require new clones since there are calls to
`request.url()` that return a `&Url`. One option would have been to make
`DisplaySafeUrl` wrap a `Cow`, but this would lead to lifetime
annotations all over the codebase. I've created a separate PR based on
this one (#13576) that updates `uv-auth` to use `DisplaySafeUrl` with
one new clone. We can discuss the tradeoffs there.
Most of this PR just replaces `Url` with `DisplaySafeUrl`. The core is
`uv_redacted/lib.rs`, where the newtype is implemented. To make it
easier to review the rest, here are some points of note:
* `DisplaySafeUrl` has a `Display` implementation that masks
credentials. Currently, it will still display the username when there is
both a username and password. If we think is the wrong choice, it can
now be changed in one place.
* `DisplaySafeUrl` has a `remove_credentials()` method and also a
`.to_string_with_credentials()` method. This allows us to use it in a
variety of scenarios.
* `IndexUrl::redacted()` was renamed to
`IndexUrl::removed_credentials()` to make it clearer that we are not
masking.
* We convert from a `DisplaySafeUrl` to a `Url` when calling `reqwest`
methods like `.get()` and `.head()`.
* We convert from a `DisplaySafeUrl` to a `Url` when creating a
`uv_auth::Index`. That is because, as mentioned above, I will be
updating the `uv_auth` crate to use this newtype in a separate PR.
* A number of tests (e.g., in `pip_install.rs`) that formerly used
filters to mask tokens in the test output no longer need those filters
since tokens in URLs are now masked automatically.
* The one place we are still knowingly writing credentials to
`pyproject.toml` is when a URL with credentials is passed to `uv add`
with `--raw`. Since displaying credentials is no longer automatic, I
have added a `to_string_with_credentials()` method to the `Pep508Url`
trait. This is used when `--raw` is passed. Adding it to that trait is a
bit weird, but it's the simplest way to achieve the goal. I'm open to
suggestions on how to improve this, but note that because of the way
we're using generic bounds, it's not as simple as just creating a
separate trait for that method.
PackageMetadata, for whatever reason, does not have a mirrored Wire type
so it was easy to not realize that it contains markers that need to be
complexified.
Fixes#13614
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
Rustfmt introduces a lot of formatting changes in the 2024 edition. To
not break everything all at once, we split out the set of formatting
changes compatible with both the 2021 and 2024 edition by first
formatting with the 2024 style, and then again with the currently used
2021 style.
Notable changes are the formatting of derive macro attributes and lines
with overly long strings and adding trailing semicolons after statements
consistently.
## Summary
In https://github.com/astral-sh/uv/pull/12968, we added support for
upload time to `uv.lock`, but stylized as `upload_time`. The other keys
in `uv.lock` use kebab casing, as in common in Python formats, so this
really should've been `upload-time`. I want to change it ASAP to
minimize churn for users. Any users that already upgraded will of course
experience churn in their files a second time. But if we don't change it
now, we'll only increase the surface area of affected users.
So, this PR uses `upload-time` instead, but continues reading
`upload_time` to make it non-breaking.
## Summary
We accept `pylock.toml` as a requirements file (e.g., `uv sync
pylock.toml` or `uv pip install -r pylock.toml`). When you provide a
`pylock.toml` file, we don't allow you to provide other requirements, or
constraints, etc. And you can only provide one `pylock.toml` file, not
multiple.
We might want to remove this from `uv pip install` for now, since `pip`
may end up with a different interface (whereas `uv pip sync` is already
specific to uv), and most of the arguments aren't applicable (like
`--resolution`, etc.). Regardless, it's behind `--preview` for both
commands.
## Summary
This PR adds `uv export` support for [PEP
751](https://peps.python.org/pep-0751). We don't yet expose a way to
consume the generated lockfile, but it's a first step.
The logic to go from `uv.lock` to "flat set of packages to include, with
markers telling us when to include them" is all shared with the
`requirements.txt` export (and extracted in
https://github.com/astral-sh/uv/pull/12956). So most of the code is just
converting from our internal types to the PEP 751 schema.
## Summary
This is included in PEP 751, so we lose it when converting from
`uv.lock`. I think it's a good piece of information to include in the
`uv.lock` anyway.
## Summary
I don't know if I actually want to commit this, but I did it on the
plane last time and just polished it off (got it to compile) while
waiting to board.
## Summary
This PR modifies the requirement source entities to store a (new)
container struct that wraps `IndexUrl`. This will allow us to store
user-defined metadata alongside `IndexUrl`, and propagate that metadata
throughout resolution.
Specifically, I need to store the "kind" of the index (Simple API vs.
`--find-links`), but I also ran into this problem when I tried to add
support for overriding `Cache-Control` headers on a per-index basis: at
present, we have no way to passing around metadata alongside an
`IndexUrl`.
## Summary
This crate is for standards-compliant types, but this is explicitly a
type that's custom to uv. It's also strange because we kind of want to
reference `IndexUrl` on the registry type, but that's in a crate that
_depends_ on `uv-pypi-types`, which to me is a sign that this is off.
Reject lockfiles where the package version and the wheel versions are
incoherent. This implicitly checks that all wheel files have the same
version.
It does not check for the source dist version, since a source dist may
not contain a version in the filename and attempting to deserialize
source dist filenames we may not need is a performance overhead for
something that's already slow in `uv run`.
Fixes#12164
In #10669, a pyproject.toml with requires-python but no environment had
a lockfile covering only a subset of the requires-python space:
```toml
resolution-markers = [
"python_full_version >= '3.10' and platform_python_implementation == 'CPython'",
"python_full_version == '3.9.*'",
"python_full_version < '3.9'",
]
```
This marker set is invalid, we have to reject the lockfile. (We can
still use the versions though, to avoid churn).
Part 1/2 of #10669
## Summary
This is the pattern I see in a variety of crates, and I believe this is
preferred if you don't _need_ an owned `String`, since you can avoid the
allocation. This could be pretty impactful for us?
Initially, we were limiting Git schemes to HTTPS and SSH as only
supported schemes. We lost this validation in #3429. This incidentally
allowed file schemes, which apparently work with Git out of the box.
A caveat for this is that in tool.uv.sources, we parse the git field
always as URL. This caused a problem with #11425: repo = { git =
'c:\path\to\repo', rev = "xxxxx" } was parsed as a URL where c: is the
scheme, causing a bad error message down the line.
This PR:
* Puts Git URL validation back in place. It bans everything but HTTPS,
SSH, and file URLs. This could be a breaking change, if users were using
a git transport protocol were not aware of, even though never
intentionally supported.
* Allows file: URL in Git: This seems to be supported by Git and we were
supporting it albeit unintentionally, so it's reasonable to continue to
support it.
* It does not allow relative paths in the git field in tool.uv.sources.
Absolute file URLs are supported, whether we want relative file URLs for
Git too should be discussed separately.
Closes#3429: We reject the input with a proper error message, while
hinting the user towards file:. If there's still desire for relative
path support, we can keep it open.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
We want to build `uv-build` without depending on the network crates. In
preparation for that, we split uv-git into uv-git and uv-git-types,
where only uv-git depends on reqwest, so that uv-build can use
uv-git-types.
## Summary
This PR revives https://github.com/astral-sh/uv/pull/10017, which might
be viable now that we _don't_ enforce any platforms by default.
The basic idea here is that users can mark certain platforms as required
(empty, by default). When resolving, we ensure that the specified
platforms have wheel coverage, backtracking if not.
For example, to require that we include a version of PyTorch that
supports Intel macOS:
```toml
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = ["torch>1.13"]
[tool.uv]
required-platforms = [
"sys_platform == 'darwin' and platform_machine == 'x86_64'"
]
```
Other than that, the forking is identical to past iterations of this PR.
This would give users a way to resolve the tail of issues in #9711, but
with manual opt-in to supporting specific platforms.
## Summary
This is an alternative to the approach we took in #11063 whereby we
always included `provides-extra` and `requires-dist`, since we needed
some way to differentiate between "no extras" and "lockfile was
generated by a uv version that didn't include extras".
Instead, this PR adds a minor version (called a "revision") to the
lockfile that we can use to indicate support for this feature. While
lockfile version bumps are backwards-incompatible, older uv versions
_can_ read lockfiles with a later revision -- they just won't understand
all the data.
In a future major version bump, we could simplify things and change the
schema to use a (major, minor) format instead of these two separate
fields. But this is the only way to do it that's backwards-compatible
with existing uv versions.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Closes#10597.
Recreated https://github.com/astral-sh/uv/pull/10925 that got closed as
the base branch got merged.
Snapshot tests.
---------
Co-authored-by: Aria Desires <aria.desires@gmail.com>
