## Summary
Part of #8607. This is a pure refactor aimed at paving the way for
supporting the `default-extras` configuration in the `pyproject.toml`
file.
The `ExtraSpecification` struct has been refactored to align more
closely with the
[`DependencyGroups`](256b100a9e/crates/uv-configuration/src/dependency_groups.rs (L9))
struct.
## Test Plan
Existing tests.
This is a minimal redux of #10861 to be compatible with `uv pip`.
This implements the interface described in:
https://github.com/pypa/pip/pull/13065#issuecomment-2544000876 for `uv
pip install` and `uv pip compile`. Namely `--group <[path:]name>`, where
`path` when not defined defaults to `pyproject.toml`.
In that interface they add `--group` to `pip install`, `pip download`,
and `pip wheel`. Notably we do not define `uv pip download` and `uv pip
wheel`, so for parity we only need to implement `uv pip install`.
However, we also support `uv pip compile` which is not part of pip
itself, and `--group` makes sense there too.
----
The behaviour of `--group` for `uv pip` commands makes sense for the
cases upstream pip supports, but has confusing meanings in cases that
only we support (because reading pyproject.tomls is New Tech to them but
heavily supported by us). **Specifically case (h) below is a concerning
footgun, and case (e) below may get complaints from people who aren't
well-versed in dependency-groups-as-they-pertain-to-wheels.**
## Only Group Flags
Group flags on their own work reasonably and uncontroversially, except
perhaps that they don't do very clever automatic project discovery.
a) `uv pip install --group path/to/pyproject.toml:mygroup` pulls up
`path/to/project.toml` and installs all the packages listed by its
`mygroup` dependency-group (essentially treating it like another kind of
requirements.txt). In this regard it functions similarly to
`--only-group` in the rest of uv's interface.
b) `uv pip install --group mygroup` is just sugar for `uv pip install
--group pyproject.toml:mygroup` (**note that no project discovery
occurs**, upstream pip simply hardcodes the path "pyproject.toml" here
and we reproduce that.)
c) `uv pip install --group a/pyproject.toml:groupx --group
b/pyproject.toml:groupy`, and any other instance of multiple `--group`
flags, can be understood as completely independent requests for the
given groups at the given files.
## Groups With Named Packages
Groups being mixed with named packages also work in a fairly
unsurprising way, especially if you understand that things like
dependency-groups are not really supposed to exist on pypi, they're just
for local development.
d) `uv pip install mypackage --group path/to/pyproject.toml:mygroup`
much like multiple instances of `--group` the two requests here are
essentially completely independent: pleases install `mypackage`, and
please also install `path/to/pyproject.toml:mygroup`.
e) `uv pip install mypackage --group mygroup` is exactly the same, but
this is where it becomes possible for someone to be a little confused,
as you might think `mygroup` is supposed to refer to `mypackage` in some
way (it can't). But no, it's sourcing `pyproject.toml:mygroup` from the
current working directory.
## Groups With Requirements/Sourcetrees/Editables
Requirements and sourcetrees are where I expect users to get confused.
It behaves *exactly* the same as it does in the previous sections but
you would absolutely be forgiven for expecting a different behaviour.
*Especially* because `--group` with the rest of uv *does* do something
different.
f) `uv pip install -r a/pyproject.toml --group b/pyproject.toml:mygroup`
is again just two independent requests (install `a/pyproject.toml`'s
dependencies, and `b/pyproject.toml`'s `mygroup`).
g) `uv pip install -r pyproject.toml --group mygroup` is exactly like
the previous case but *incidentally* the two requests refer to the same
file. What the user wanted to happen is almost certainly happening, but
they are likely getting "lucky" here that they're requesting something
simple.
h) `uv pip install -r a/pyproject.toml --group mygroup` is again exactly
the same but the user is likely to get surprised and upset as this
invocation actually sources two different files (install
`a/pyproject.toml`'s dependencies, and `pyproject.toml`'s `mygroup`)! I
would expect most people to assume the `--group` flag here is covering
all applicable requirements/sourcetrees/editables, but no, it continues
to be a totally independent reference to a file with a hardcoded
relative path.
------
Fixes https://github.com/astral-sh/uv/issues/8590
Fixes https://github.com/astral-sh/uv/issues/8969
## Summary
This is the pattern I see in a variety of crates, and I believe this is
preferred if you don't _need_ an owned `String`, since you can avoid the
allocation. This could be pretty impactful for us?
## Summary
Since we use `SmallString` internally, there's no benefit to passing an
owned string to the `PackageName` constructor (same goes for
`ExtraName`, etc.). I've kept them for now (maybe that will change in
the future, so it's useful to have clients passed own values if they
_can_), but removed a bunch of usages where we were casting from `&str`
to `String` needlessly to use the constructor.
## Summary
This appears to be a consistent 1% performance improvement and should
also reduce memory quite a bit. We've also decided to use these for
markers, so it's nice to use the same optimization here.
```
❯ hyperfine "./uv pip compile --universal scripts/requirements/airflow.in" "./arcstr pip compile --universal scripts/requirements/airflow.in" --min-runs 50 --warmup 20
Benchmark 1: ./uv pip compile --universal scripts/requirements/airflow.in
Time (mean ± σ): 136.3 ms ± 4.0 ms [User: 139.1 ms, System: 241.9 ms]
Range (min … max): 131.5 ms … 149.5 ms 50 runs
Benchmark 2: ./arcstr pip compile --universal scripts/requirements/airflow.in
Time (mean ± σ): 134.9 ms ± 3.2 ms [User: 137.6 ms, System: 239.0 ms]
Range (min … max): 130.1 ms … 151.8 ms 50 runs
Summary
./arcstr pip compile --universal scripts/requirements/airflow.in ran
1.01 ± 0.04 times faster than ./uv pip compile --universal scripts/requirements/airflow.in
```
## Summary
These were moved as part of a broader refactor to create a single
integration test module. That "single integration test module" did
indeed have a big impact on compile times, which is great! But we aren't
seeing any benefit from moving these tests into their own files (despite
the claim in [this blog
post](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html),
I see the same compilation pattern regardless of where the tests are
located). Plus, we don't have many of these, and same-file tests is such
a strong Rust convention.
`ExtraName` did implement `AsRef<str>`, but that should generally
only be used in a context with an `AsRef<str>` generic bound. If
you just want a `&str` from a concrete `ExtraName`, then a specific
method for that purpose should be used.
Part of #8090
Adds the ability to add and remove dependencies from arbitrary groups
using `uv add` and `uv remove`. Does not include resolving with the new
dependencies — tackling that in #8110.
Additionally, this does not yet resolve interactions with the existing
`dev` group — we'll tackle that separately as well. I probably won't
merge the stack until that design is resolved.
As per
https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html
Before that, there were 91 separate integration tests binary.
(As discussed on Discord — I've done the `uv` crate, there's still a few
more commits coming before this is mergeable, and I want to see how it
performs in CI and locally).
## Summary
Random, but I noticed that we can remove a ton of serialize and
deserialize derives by using `rkyv` for the flat-index caches. (We
already use `rkyv` for these same structs in the registry cache.)
Recently, rkyv 0.8 was released. Its API is a fair bit simpler now for
higher level uses (like for us in `uv`) and results in us being able to
delete a fair bit of code. This also removes our last dependency on `syn
1.0`, and thus drops that dependency.
Performance (via testing on the `transformers` example) seems to remain
about the same, which is what was expected:
```
$ hyperfine -w5 -r100 'uv lock' 'uv-ag-rkyv-update lock'
Benchmark 1: uv lock
Time (mean ± σ): 55.6 ms ± 6.4 ms [User: 30.4 ms, System: 35.1 ms]
Range (min … max): 43.0 ms … 73.1 ms 100 runs
Benchmark 2: uv-ag-rkyv-update lock
Time (mean ± σ): 56.5 ms ± 7.2 ms [User: 30.5 ms, System: 36.3 ms]
Range (min … max): 39.1 ms … 71.5 ms 100 runs
Summary
uv lock ran
1.02 ± 0.18 times faster than uv-ag-rkyv-update lock
```
Closes#7415
Interestingly, the empty string appears to be valid for these
types. I'm not sure if that's intended, but having a Default
impl is useful for use with `std::mem::take`.
Specifically, this shows the resolution produced by the
resolver *before* constructing a resolution graph.
Unlike most trace messages, this is a multi-line message
that needs to do some small amount of work to build
itself. So we do an explicit gating on the log level here
instead of just relying on the `trace!` macro itself.
## Summary
Externally, development dependencies are currently structured as a flat
list of PEP 580-compatible requirements:
```toml
[tool.uv]
dev-dependencies = ["werkzeug"]
```
When locking, we lock all development dependencies; when syncing, users
can provide `--dev`.
Internally, though, we model them as dependency groups, similar to
Poetry, PDM, and [PEP 735](https://peps.python.org/pep-0735). This
enables us to change out the user-facing frontend without changing the
internal implementation, once we've decided how these should be exposed
to users.
A few important decisions encoded in the implementation (which we can
change later):
1. Groups are enabled globally, for all dependencies. This differs from
extras, which are enabled on a per-requirement basis. Note, however,
that we'll only discover groups for uv-enabled packages anyway.
2. Installing a group requires installing the base package. We rely on
this in PubGrub to ensure that we resolve to the same version (even
though we only expect groups to come from workspace dependencies anyway,
which are unique). But anyway, that's encoded in the resolver right now,
just as it is for extras.
In *some* places in our crates, `serde` (and `rkyv`) are optional
dependencies. I believe this was done out of reasons of "good sense,"
that is, it follows a Rust ecosystem pattern where serde integration
tends to be an opt-in crate feature. (And similarly for `rkyv`.)
However, ultimately, `uv` itself requires `serde` and `rkyv` to
function. Since our crates are strictly internal, there are limited
consumers for our crates without `serde` (and `rkyv`) enabled. I think
one possibility is that optional `serde` (and `rkyv`) integration means
that someone can do this:
cargo test -p pep440_rs
And this will run tests _without_ `serde` or `rkyv` enabled. That in
turn could lead to faster iteration time by reducing compile times. But,
I'm not sure this is worth supporting. The iterative compilation times
of
individual crates are probably fast enough in debug mode, even with
`serde` and `rkyv` enabled. Namely, `serde` and `rkyv` themselves
shouldn't need to be re-compiled in most cases. On `main`:
```
from-scratch: `cargo test -p pep440_rs --lib` 0.685
incremental: `cargo test -p pep440_rs --lib` 0.278s
from-scratch: `cargo test -p pep440_rs --features serde,rkyv --lib` 3.948s
incremental: `cargo test -p pep440_rs --features serde,rkyv --lib` 0.321s
```
So while a from-scratch build does take significantly longer, an
incremental build is about the same.
The benefit of doing this change is two-fold:
1. It brings out crates into alignment with "reality." In particular,
some crates were _implicitly_ relying on `serde` being enabled
without explicitly declaring it. This technically means that our
`Cargo.toml`s were wrong in some cases, but it is hard to observe it
because of feature unification in a Cargo workspace.
2. We no longer need to deal with the cognitive burden of writing
`#[cfg_attr(feature = "serde", ...)]` everywhere.
Address a few pedantic lints
lints are separated into separate commits so they can be reviewed
individually.
I've not added enforcement for any of these lints, but that could be
added if desirable.
First, replace all usages in files in-place. I used my editor for this.
If someone wants to add a one-liner that'd be fun.
Then, update directory and file names:
```
# Run twice for nested directories
find . -type d -print0 | xargs -0 rename s/puffin/uv/g
find . -type d -print0 | xargs -0 rename s/puffin/uv/g
# Update files
find . -type f -print0 | xargs -0 rename s/puffin/uv/g
```
Then add all the files again
```
# Add all the files again
git add crates
git add python/uv
# This one needs a force-add
git add -f crates/uv-trampoline
```
