Commit graph

130 commits

Author SHA1 Message Date
Andrew Gallant
563507edba uv-resolver: add support for incomplete markers
In some cases, it's possible for the marker expressions on conflicting
dependency specification to be disjoint but *incomplete*. That is, if
one unions the disjoint markers, the result is not the complete set of
marker environments possible. There may be some "gap" of marker
environments not covered by the markers.

This is a problem in practice because, before this commit, we only
created forks in the resolver for specific marker expressions. So if a
dependency happened to fall in a "gap," our resolver would never see it.

This commit fixes this by adding a new split covering the negation of
the union of all marker expressions in a set of forks for a specific
package.

Originally, I had planned on only creating this split when it was known
that the gap actually existed. That is, when the negation of the marker
expressions did *not* correspond to the empty set. After a lot of
thought, unfortunately, this (I believe) effectively boils down to 3SAT,
which is NP-complete.

Instead, what we do here is *always* create an extra split unless we can
definitively tell that it is empty. We look for a few cases, but
otherwise throw our hands up and potentially do wasted work.

This also updates the lock scenario tests to reflect the actual bug fix
here.
2024-07-15 10:09:01 -07:00
Andrew Gallant
8d6c49b36c pep508: add MarkerTree::negate
It does what you think it does, for the most part.
2024-07-12 04:37:36 -07:00
Andrew Gallant
da8a4a6faa pep508: write x.y.* when serializing MarkerExpression
It's unclear to me whether this was intentional or not, but
I realized that converting a MarkerExpression to a string
treated EqualStar and NotEqualStar as Equal and NotEqual,
respectively. I tweaked this to match the Display impl for
VersionSpecifier.

(Negation tests in the next commit cover this change.)
2024-07-12 04:37:36 -07:00
Charlie Marsh
23eb42deed
Allow constraints to be provided in --upgrade-package (#4952)
## Summary

Allows, e.g., `--upgrade-package flask<3.0.0`.

Closes https://github.com/astral-sh/uv/issues/1964.
2024-07-09 20:09:13 -07:00
konsti
53db63f6dd
Apply extra to overrides and constraints (#4829)
This is an attempt to solve https://github.com/astral-sh/uv/issues/ by
applying the extra marker of the requirement to overrides and
constraints.

Say in `a` we have a requirements
```
b==1; python_version < "3.10"
c==1; extra == "feature"
```

and overrides
```
b==2; python_version < "3.10"
b==3; python_version >= "3.10"
c==2; python_version < "3.10"
c==3; python_version >= "3.10"
```

Our current strategy is to discard the markers in the original
requirements. This means that on 3.12 for `a` we install `b==3`, but it
also means that we add `c` to `a` without `a[feature]`, causing #4826.
With this PR, the new requirement become,

```
b==2; python_version < "3.10"
b==3; python_version >= "3.10"
c==2; python_version < "3.10" and extra == "feature"
c==3; python_version >= "3.10" and extra == "feature"
```

allowing to override markers while preserving optional dependencies as
such.

Fixes #4826
2024-07-09 20:37:24 +02:00
Charlie Marsh
ac87fd4006
Disable Clippy's too-many-arguments rule (#4663)
## Summary

We allow this constantly, I think it's just too pedantic for us.
2024-06-30 19:30:38 +00:00
konsti
d9dbb8a4af
Support conflicting URL in separate forks (#4435)
Downstack PR: #4481

## Introduction

We support forking the dependency resolution to support conflicting
registry requirements for different platforms, say on package range is
required for an older python version while a newer is required for newer
python versions, or dependencies that are different per platform. We
need to extend this support to direct URL requirements.

```toml
dependencies = [
  "iniconfig @ 62565a6e1c/iniconfig-2.0.0-py3-none-any.whl ; python_version >= '3.12'",
  "iniconfig @ b3c12c6d70/iniconfig-1.1.1-py2.py3-none-any.whl ; python_version < '3.12'"
]
```

This did not work because `Urls` was built on the assumption that there
is a single allowed URL per package. We collect all allowed URL ahead of
resolution by following direct URL dependencies (including path
dependencies) transitively, i.e. a registry distribution can't require a
URL.

## The same package can have Registry and URL requirements

Consider the following two cases:

requirements.in:
```text
werkzeug==2.0.0
werkzeug @ 960bb4017c/Werkzeug-2.0.0-py3-none-any.whl
```
pyproject.toml:
```toml
dependencies = [
  "iniconfig == 1.1.1 ; python_version < '3.12'",
  "iniconfig @ git+https://github.com/pytest-dev/iniconfig@93f5930e668c0d1ddf4597e38dd0dea4e2665e7a ; python_version >= '3.12'",
]
```

In the first case, we want the URL to override the registry dependency,
in the second case we want to fork and have one branch use the registry
and the other the URL. We have to know about this in
`PubGrubRequirement::from_registry_requirement`, but we only fork after
the current method.

Consider the following case too:

a:
```
c==1.0.0
b @ https://b.zip
```
b:
```
c @ https://c_new.zip ; python_version >= '3.12'",
c @ https://c_old.zip ; python_version < '3.12'",
```

When we convert the requirements of `a`, we can't know the url of `c`
yet. The solution is to remove the `Url` from `PubGrubPackage`: The
`Url` is redundant with `PackageName`, there can be only one url per
package name per fork. We now do the following: We track the urls from
requirements in `PubGrubDependency`. After forking, we call
`add_package_version_dependencies` where we apply override URLs, check
if the URL is allowed and check if the url is unique in this fork. When
we request a distribution, we ask the fork urls for the real URL. Since
we prioritize url dependencies over registry dependencies and skip
packages with `Urls` entries in pre-visiting, we know that when fetching
a package, we know if it has a url or not.

## URL conflicts

pyproject.toml (invalid):
```toml
dependencies = [
  "iniconfig @ e96292c7f7/iniconfig-1.1.0.tar.gz",
  "iniconfig @ b3c12c6d70/iniconfig-1.1.1-py2.py3-none-any.whl ; python_version < '3.12'",
  "iniconfig @ 62565a6e1c/iniconfig-2.0.0-py3-none-any.whl ; python_version >= '3.12'",
]
```

On the fork state, we keep `ForkUrls` that check for conflicts after
forking, rejecting the third case because we added two packages of the
same name with different URLs.

We need to flatten out the requirements before transformation into
pubgrub requirements to get the full list of other requirements which
may contain a URL, which was changed in a previous PR: #4430.

## Complex Example

a:
```toml
dependencies = [
  # Force a split
  "anyio==4.3.0 ; python_version >= '3.12'",
  "anyio==4.2.0 ; python_version < '3.12'",
  # Include URLs transitively
  "b"
]
```
b:
```toml
dependencies = [
  # Only one is used in each split.
  "b1 ; python_version < '3.12'",
  "b2 ; python_version >= '3.12'",
  "b3 ; python_version >= '3.12'",
]
```
b1:
```toml
dependencies = [
  "iniconfig @ b3c12c6d70/iniconfig-1.1.1-py2.py3-none-any.whl",
]
```
b2:
```toml
dependencies = [
  "iniconfig @ 62565a6e1c/iniconfig-2.0.0-py3-none-any.whl",
]
```
b3:
```toml
dependencies = [
  "iniconfig @ e96292c7f7/iniconfig-1.1.0.tar.gz",
]
```

In this example, all packages are url requirements (directory
requirements) and the root package is `a`. We first split on `a`, `b`
being in each split. In the first fork, we reach `b1`, the fork URLs are
empty, we insert the iniconfig 1.1.1 URL, and then we skip over `b2` and
`b3` since the mark is disjoint with the fork markers. In the second
fork, we skip over `b1`, visit `b2`, insert the iniconfig 2.0.0 URL into
the again empty fork URLs, then visit `b3` and try to insert the
iniconfig 1.1.0 URL. At this point we find a conflict for the iniconfig
URL and error.

## Closing

The git tests are slow, but they make the best example for different URL
types i could find.

Part of #3927. This PR does not handle `Locals` or pre-releases yet.
2024-06-26 13:58:23 +02:00
Ibraheem Ahmed
7b72b55af8
Opt-out tool.uv.sources support for uv add (#4406)
## Summary

After this change, `uv add` will try to use `tool.uv.sources` for all
source requirements. If a source cannot be resolved, i.e. an ambiguous
Git reference is provided, it will error. Git references can be
specified with the `--tag`, `--branch`, or `--rev` arguments. Editables
are also supported with `--editable`.

Users can opt-out of `tool.uv.sources` support with the `--raw` flag,
which will force uv to use `project.dependencies`.

Part of https://github.com/astral-sh/uv/issues/3959.
2024-06-19 14:20:16 -04:00
Charlie Marsh
c996e8e3f3
Enable workspace lint configuration in remaining crates (#4329)
## Summary

We didn't have Clippy enabled (to match our workspace settings) in a few
crates.
2024-06-18 03:02:28 +00:00
Charlie Marsh
22795f85bc
Flatten ORs and ANDs in marker construction (#4260)
## Summary

If we're ORing an OR, we should just append rather than nesting in
another OR.

In my branch, this let us simplify:

```
python_version < '3.10' or python_version > '3.12' or (python_version < '3.8' or python_version > '3.12')
```

To:

```
python_version < '3.10' or python_version > '3.12
```
2024-06-11 21:44:49 -04:00
Charlie Marsh
656fc427b9
Add support for local directories with --index-url (#4226)
## Summary

Closes #4078.
2024-06-10 22:27:04 -04:00
Ibraheem Ahmed
7232c53718
Simplify marker expressions in lockfile (#4066)
## Summary

Simplify and normalize marker expressions in the lockfile. Right now
this does a simple analysis by only looking at related operators at the
same level of precedence. I think anything more complex would be out of
scope.

Resolves https://github.com/astral-sh/uv/issues/4002.
2024-06-07 16:14:24 -04:00
Andrew Gallant
5c30b39fe3
uv-resolver: normalize marker expressions (#4017)
This is a quick fix for some flaky tests where the output in the lock
file isn't stable because marker expressions can be combined in a
non-deterministic order.

I believe there is ongoing work to simplify marker expressions which
will help here, but I think some kind of normalization is still
ultimately needed to guarantee consistent output.

I first noticed the flaky test in:
https://github.com/astral-sh/uv/pull/4015
2024-06-04 13:45:54 -04:00
Di-Is
5c776939d2
Add override namespace to pyproject.toml/uv.toml (#3839)
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

See #3834 .

This PR adds a new namespace, `override-dependencies`, to
pyproject.toml/uv.toml.
This namespace assumes that the dependencies you want to override are
written in the form of `requirements.txt`.


a example of pyproject.toml
```toml
[project]
name = "example"
version = "0.0.0"
dependencies = [
  "flask==3.0.0"
]

[tool.uv]
override-dependencies = [
  "werkzeug==2.3.0"
]
```

This will improve usability by allowing you to override dependencies
without having to specify the --override option when running `uv pip
compile/install`.

## Test Plan

added test to `crates/uv/tests/pip_compile.rs`.

---------

Co-authored-by: konstin <konstin@mailbox.org>
2024-06-03 12:15:51 +02:00
Charlie Marsh
7b7da803bb
Unify editable and unnamed URL parsing (#3946)
## Summary

This will help prevent bugs like #3934 by unifying the implementations
for editables and non-editable unnamed requirements. Specifically, both
of these now go through the same parsing paths and use the same struct
representations (with the exception that the editable flag is flipped in
the first case):

```
-e ./foo/bar
./foo/bar
```

We also now support PEP 508 in editable URLs. It turns out this is just
a limitation in pip, so it's correct to support it. For example, this
now works:

```
-e black[d] @ file://${PROJECT_ROOT}/scripts/packages/black_editable
```

Closes #3941.

Closes #3942.
2024-05-31 21:08:00 +00:00
Charlie Marsh
8c11f99fdf
Discard fragments when parsing unnamed URLs (#3940)
## Summary

Closes https://github.com/astral-sh/uv/issues/3934.
2024-05-31 13:54:15 +00:00
konsti
081f20c53e
Add support for tool.uv into distribution building (#3904)
With the change, we remove the special casing of workspace dependencies
and resolve `tool.uv` for all git and directory distributions. This
gives us support for non-editable workspace dependencies and path
dependencies in other workspaces. It removes a lot of special casing
around workspaces. These changes are the groundwork for supporting
`tool.uv` with dynamic metadata.

The basis for this change is moving `Requirement` from
`distribution-types` to `pypi-types` and the lowering logic from
`uv-requirements` to `uv-distribution`. This changes should be split out
in separate PRs.

I've included an example workspace `albatross-root-workspace2` where
`bird-feeder` depends on `a` from another workspace `ab`. There's a
bunch of failing tests and regressed error messages that still need
fixing. It does fix the audited package count for the workspace tests.
2024-05-31 02:42:03 +00:00
Andrew Gallant
f5f330627b pep508: make 'simplify_extras' more flexible
This makes it so we can pass any function to determine whether an extra
is always true or not.

For example, `markers.simplify_extras_with(|_| true)` will remove all
extras in any marker expression. This wasn't possible to express
(without knowing all of the marker names) using the old API, but becomes
trivial to express with a predicate function.
2024-05-30 14:23:14 -04:00
Andrew Gallant
a74c2c6792 pep508: add and and or mutators to MarkerTree
While this could be done by callers since the representation
of `MarkerTree` is public, they are just annoying enough to do
that I think it makes sense to provide them on `MarkerTree`
itself.

These could also be improved in the future to do even more
flattening of conjunctions/disjunctions (or perhaps even
more robust simplification). But for now, some basic flattening
is good enough.

These routines will be used to combine marker expressions when
merging forked resolutions.
2024-05-30 14:23:14 -04:00
Charlie Marsh
cedd18e4c6
Remove some unused pub functions (#3872)
## Summary

I wrote a bad Python script to find these.
2024-05-28 15:58:13 +00:00
konsti
a89e146107
Initial workspace support (#3705)
Add workspace support when using `-r <path>/pyproject.toml` or `-e
<path>` in the pip interface. It is limited to all-editable
static-metadata workspaces, and tests only include a single main
workspace, ignoring path dependencies in another workspace. This can be
considered the MVP for workspace support: You can create a workspace,
you can install from it, but some options and conveniences are still
missing. I'll file follow-up tickets (support in lockfiles, support path
deps in other workspace, #3625)

There is also support in `uv run`, but we need
https://github.com/astral-sh/uv/issues/3700 first to properly support
using different current projects in the bluejay interface, currently the
resolution and therefore the lockfile depends on the current project.
I'd do this change first (it's big enough already), then #3700, and then
add workspace support properly to bluejay.

Fixes #3404
2024-05-28 07:41:53 +00:00
Tim de Jager
65b17f6e81
feat: bump pep580 and pep440 types (#3860)
This bumps the versions of pep580 and pep440 to coincide with the
crates.io versions. While not strictly the same, the new types in uv us
an `Inner` struct. Practically I've found I'm still able to use the
patched versions, as can seen from the open PR here:
https://github.com/prefix-dev/pixi/pull/1436.

Would be great if this bump can be done so we can keep combining the
types :)
2024-05-27 09:38:39 +02:00
konsti
4db468e27f
Use VerbatimParsedUrl in pep508_rs (#3758)
When parsing requirements from any source, directly parse the url parts
(and reject unsupported urls) instead of parsing url parts at a later
stage. This removes a bunch of error branches and concludes the work
parsing url parts once and passing them around everywhere.

Many usages of the assembled `VerbatimUrl` remain, but these can be
removed incrementally.

Please review commit-by-commit.
2024-05-23 19:52:47 +00:00
Charlie Marsh
558f628ef1
Propagate URL errors in verbatim parsing (#3720)
## Summary

Closes https://github.com/astral-sh/uv/issues/3715.

## Test Plan

```
❯ echo "/../test" | cargo run pip compile -
error: Couldn't parse requirement in `-` at position 0
  Caused by: path could not be normalized: /../test
/../test
^^^^^^^^

❯ echo "-e /../test" | cargo run pip compile -
error: Invalid URL in `-`: `/../test`
  Caused by: path could not be normalized: /../test
  Caused by: cannot normalize a relative path beyond the base directory
```
2024-05-21 19:58:59 +00:00
Charlie Marsh
0362918196
Evaluate arbitrary markers to false (#3681)
## Summary

See: https://github.com/astral-sh/uv/pull/3679#issuecomment-2121387428.

Closes: https://github.com/astral-sh/uv/issues/3675 (although I think we
have another improvement to make there -- will file separately).
2024-05-21 01:01:11 +00:00
Andrew Gallant
d0435ef20a pep508: add PartialOrd and Ord implementations to MarkerTree
Since we're adding a `Option<MarkerTree>` to `PubGrubPackage`, and since
we just make `PubGrubPackage` implement `Ord`, it follows that we want
`MarkerTree` to also implement `Ord`.
2024-05-20 19:56:24 -04:00
Andrew Gallant
976bc9ba0e uv-resolver: make PubGrubPackage orderable
It turns out that we use PubGrubPackage as the key in hashmaps in a fair
few places. And when we iterate over hashmaps, the order is unspecified.
This can in turn result in changes in output as a result of changes in
the PubGrubPackage definition, purely as a function of its changing
hash. This is confusing as there should be no semantic difference.

Thus, this is a precursor to introducing some more determinism to places
I found in the error reporting whose output depending on hashmap
iteration order.
2024-05-20 19:56:24 -04:00
Charlie Marsh
44fe0f6749
Add rustdoc links to marker.rs documentation (#3680) 2024-05-20 23:29:15 +00:00
Charlie Marsh
ed91b1d562
Parse and store extras on editables (#3629)
## Summary

As a follow-up to #3622, we now parse and store (but don't respect)
markers on editable requirements.
2024-05-16 17:12:52 -04:00
Ibraheem Ahmed
a962a65302
Add routine for verifying two marker trees are disjoint (#3583)
## Summary

Implements https://github.com/astral-sh/uv/issues/3355.
2024-05-15 13:01:05 -04:00
Ibraheem Ahmed
8ce9051296
Parse marker tree before evaluation (#3520)
## Summary

Parse `MarkerTree` expressions upfront, instead of lazily during
evaluation.

This makes implementing https://github.com/astral-sh/uv/issues/3355 a
lot easier.
2024-05-14 11:02:30 -04:00
Charlie Marsh
c2452957f9
Remove unused dependencies (#3527)
Surfaced with `cargo shear`.
2024-05-11 13:33:49 -04:00
konsti
5f8c3b7e45
Re-add dummy serde feature to pep440-rs and pep508-rs (#3501)
This keeps us in sync with the published pep440-rs and pep508-rs crates
for prefix.
2024-05-10 13:46:29 +00:00
Andrew Gallant
58d1cd2acb pep508: use Arc inside of MarkerEnvironment
Now that the type is fully encapsulated, we can pretty easily
migrate to using an Arc inside of a MarkerEnvironment.

It looks like the pyo3 macros can't deal with an Arc, so we
write out the getter methods by hand.
2024-05-09 10:06:02 -04:00
Andrew Gallant
7d67b7bb49 pep508: un-export fields for MarkerEnvironment
We now use the getters and setters everywhere.

There were some places where we wanted to build a `MarkerEnvironment`
out of whole cloth, usually in tests. To facilitate those use cases, we
add a `MarkerEnvironmentBuilder` that provides a convenient constructor.
It's basically like a `MarkerEnvironment::new`, but with named
parameters. That's useful here because there are so many fields (and
they many have the same type).
2024-05-09 10:06:02 -04:00
Andrew Gallant
be12cfb2b8 pep508: add getters and setters to MarkerEnvironment
This preps a change over to a world where MarkerEnvironment is fully
encapsulated. To facilitate this, we add getters and setters for each
field.
2024-05-09 10:06:02 -04:00
Andrew Gallant
342cac8b18 pep508: fix pyo3 related failing test
This test was failing on master. I guess we don't test
this crate with the pyo3 feature enabled? I think this
regression was due to a recent change in the error reporting
of the pep440 crate.
2024-05-09 10:06:02 -04:00
Charlie Marsh
b6bb2ac548
Remove Optional from with_origin API (#3482) 2024-05-09 13:40:51 +00:00
Charlie Marsh
3e4365301e
Track origin for setup.py files and friends (#3481)
## Summary

Ensures that we track the origins for requirements regardless of whether
they come from `pyproject.toml` or `setup.py` or `setup.cfg`.

Closes #3480.
2024-05-09 09:30:40 -04:00
Andrew Gallant
8b0fad3560 uv-resolver: make MarkerEnvironment optional
This commit touches a lot of code, but the conceptual change here is
pretty simple: make it so we can run the resolver without providing a
`MarkerEnvironment`. This also indicates that the resolver should run in
universal mode. That is, the effect of a missing marker environment is
that all marker expressions that reference the marker environment are
evaluated to `true`. That is, they are ignored. (The only markers we
evaluate in that context are extras, which are the only markers that
aren't dependent on the environment.)

One interesting change here is that a `Resolver` no longer needs an
`Interpreter`. Previously, it had only been using it to construct a
`PythonRequirement`, by filling in the installed version from the
`Interpreter` state. But we now construct a `PythonRequirement`
explicitly since its `target` Python version should no longer be tied to
the `MarkerEnvironment`. (Currently, the marker environment is mutated
such that its `python_full_version` is derived from multiple sources,
including the CLI, which I found a touch confusing.)

The change in behavior can now be observed through the
`--unstable-uv-lock-file` flag. First, without it:

```
$ cat requirements.in
anyio>=4.3.0 ; sys_platform == "linux"
anyio<4 ; sys_platform == "darwin"
$ cargo run -qp uv -- pip compile -p3.10 requirements.in
anyio==4.3.0
exceptiongroup==1.2.1
    # via anyio
idna==3.7
    # via anyio
sniffio==1.3.1
    # via anyio
typing-extensions==4.11.0
    # via anyio
```

And now with it:

```
$ cargo run -qp uv -- pip compile -p3.10 requirements.in --unstable-uv-lock-file
  x No solution found when resolving dependencies:
  `-> Because you require anyio>=4.3.0 and anyio<4, we can conclude that the requirements are unsatisfiable.
```

This is expected at this point because the marker expressions are being
explicitly ignored, *and* there is no forking done yet to account for
the conflict.
2024-05-09 09:24:37 -04:00
Andrew Gallant
21f5999b57 pep508: provide extra-only marker evaluation
We provide a new API on a `Requirement` that specifically
ignores the marker environment and only evaluates a requirement's
marker expression with respect to extras. Any marker expressions
that reference the marker environment automatically evaluate to
true.

Instead of duplicating the evaluation code, we just make a marker
environment optional on the lower level APIs. In theory, we could
just writer a separate evaluation routine that ignores everything
except extras, but the evaluator has a fair bit of other stuff in it
(such as emitting warnings) that would be good to keep DRY IMO.
2024-05-09 09:24:37 -04:00
Andrew Gallant
624f92b3f8 pep508: fix doc test
This doc test seems to fail due to the recent change making
`Requirement` generic on its URL type. While the type parameter
was given a default of `VerbatimUrl`, this default doesn't always
apply. For example, the `FromStr` impl on `Requirement` is still
generic on any URL type, and so callers must indicate the type
of the URL to return. (An alternative would be to define the
`FromStr` impl for just the default URL type.)
2024-05-09 09:24:37 -04:00
Charlie Marsh
f16cbfda7e
Add a dedicated struct for source annotations (#3478) 2024-05-09 04:40:35 +00:00
Tom Parker-Shemilt
bc963d13cb
Annotate sources of requirements (#3269)
## Summary

Fixes https://github.com/astral-sh/uv/issues/1343. This is kinda a first
draft at the moment, but does at least mostly work locally (barring some
bits of the test suite that seem to not work for me in general).

## Test Plan

Mostly running the existing tests and checking the revised output is
sane

## Outstanding issues

Most of these come down to "AFAIK, the existing tools don't support
these patterns, but `uv` does" and so I'm not sure there's an existing
good answer here! Most of the answers so far are "whatever was easiest
to build"

- [x] ~~Is "-r pyproject.toml" correct? Should it show something else or
get skipped entirely~~ No it wasn't. Fixed in
3044fa8b86
- [ ] If the requirements file is stdin, that just gets skipped. Should
it be recorded?
- [ ] Overrides get shown as "--override<override.txt>". Correct?
- [x] ~~Some of the tests (e.g.
`dependency_excludes_non_contiguous_range_of_compatible_versions`) make
assumptions about the order of package versions being outputted, which
this PR breaks. I'm not sure if the text is fairly arbitrary and can be
replaced or whether the behaviour needs fixing?~~ - fixed by removing
the custom pubgrub PartialEq/Hash
- [ ] Are all the `TrackedFromStr` et al changes needed, or is there an
easier way? I don't think so, I think it's necessary to track these sort
of things fairly comprehensively to make this feature work, and this
sort of invasive change feels necessary, but happy to be proved wrong
there :)
- [x] ~~If you have a requirement coming in from two or more different
requirements files only one turns up. I've got a closed-source example
for this (can go into more detail if needed), mostly consisting of a
complicated set of common deps creating a larger set. It's a rarer case,
but worth considering.~~ 042432b200
- [ ] Doesn't add annotations for `setup.py` yet
- This is pretty hard, as the correct location to insert the path is
`crates/pypi-types/src/metadata.rs`'s `parse_pkg_info`, which as it's
based off a source distribution has entirely thrown away such matters as
"where did this package requirement get built from". Could add "`built
package name`" as a dep, but that's a little odd.
2024-05-08 23:19:22 -04:00
Charlie Marsh
d665410d59
Move PyO3 derive down (#3434)
## Summary

For some reason the current order is breaking the IntelliJ LSP.
2024-05-07 18:16:28 +00:00
konsti
55f6e4e66b
Make Requirement generic over url type (#3253)
This change allows switching out the url type for requirements. The
original idea was to allow different types for different requirement
origins, so that core metadata reads can ban non-pep 508 requirements
while we only allow them for requirements.txt. This didn't work out
because we expect `&Requirement`s from all sources to match.

I also tried to split `pep508_rs` into a PEP 508 compliant crate and
into our extensions, but they are to tightly coupled.

I think this change is an improvement still as it reduces the hardcoded
dependence on `VerbatimUrl`.
2024-05-07 16:45:49 +02:00
konsti
b2f6b92e3b
Move pep508-rs Cursor into its own module (#3401) 2024-05-06 10:13:27 +00:00
konsti
4f87edbe66
Add basic tool.uv.sources support (#3263)
## Introduction

PEP 621 is limited. Specifically, it lacks
* Relative path support
* Editable support
* Workspace support
* Index pinning or any sort of index specification

The semantics of urls are a custom extension, PEP 440 does not specify
how to use git references or subdirectories, instead pip has a custom
stringly format. We need to somehow support these while still stying
compatible with PEP 621.

## `tool.uv.source`

Drawing inspiration from cargo, poetry and rye, we add `tool.uv.sources`
or (for now stub only) `tool.uv.workspace`:

```toml
[project]
name = "albatross"
version = "0.1.0"
dependencies = [
  "tqdm >=4.66.2,<5",
  "torch ==2.2.2",
  "transformers[torch] >=4.39.3,<5",
  "importlib_metadata >=7.1.0,<8; python_version < '3.10'",
  "mollymawk ==0.1.0"
]

[tool.uv.sources]
tqdm = { git = "https://github.com/tqdm/tqdm", rev = "cc372d09dcd5a5eabdc6ed4cf365bdb0be004d44" }
importlib_metadata = { url = "https://github.com/python/importlib_metadata/archive/refs/tags/v7.1.0.zip" }
torch = { index = "torch-cu118" }
mollymawk = { workspace = true }

[tool.uv.workspace]
include = [
  "packages/mollymawk"
]

[tool.uv.indexes]
torch-cu118 = "https://download.pytorch.org/whl/cu118"
```

See `docs/specifying_dependencies.md` for a detailed explanation of the
format. The basic gist is that `project.dependencies` is what ends up on
pypi, while `tool.uv.sources` are your non-published additions. We do
support the full range or PEP 508, we just hide it in the docs and
prefer the exploded table for easier readability and less confusing with
actual url parts.

This format should eventually be able to subsume requirements.txt's
current use cases. While we will continue to support the legacy `uv pip`
interface, this is a piece of the uv's own top level interface. Together
with `uv run` and a lockfile format, you should only need to write
`pyproject.toml` and do `uv run`, which generates/uses/updates your
lockfile behind the scenes, no more pip-style requirements involved. It
also lays the groundwork for implementing index pinning.

## Changes

This PR implements:
* Reading and lowering `project.dependencies`,
`project.optional-dependencies` and `tool.uv.sources` into a new
requirements format, including:
  * Git dependencies
  * Url dependencies
  * Path dependencies, including relative and editable
* `pip install` integration
* Error reporting for invalid `tool.uv.sources`
* Json schema integration (works in pycharm, see below)
* Draft user-level docs (see `docs/specifying_dependencies.md`)

It does not implement:
* No `pip compile` testing, deprioritizing towards our own lockfile
* Index pinning (stub definitions only)
* Development dependencies
* Workspace support (stub definitions only)
* Overrides in pyproject.toml
* Patching/replacing dependencies

One technically breaking change is that we now require user provided
pyproject.toml to be valid wrt to PEP 621. Included files still fall
back to PEP 517. That means `pip install -r requirements.txt` requires
it to be valid while `pip install -r requirements.txt` with `-e .` as
content falls back to PEP 517 as before.

## Implementation

The `pep508` requirement is replaced by a new `UvRequirement` (name up
for bikeshedding, not particularly attached to the uv prefix). The still
existing `pep508_rs::Requirement` type is a url format copied from pip's
requirements.txt and doesn't appropriately capture all features we
want/need to support. The bulk of the diff is changing the requirement
type throughout the codebase.

We still use `VerbatimUrl` in many places, where we would expect a
parsed/decomposed url type, specifically:
* Reading core metadata except top level pyproject.toml files, we fail a
step later instead if the url isn't supported.
* Allowed `Urls`.
* `PackageId` with a custom `CanonicalUrl` comparison, instead of
canonicalizing urls eagerly.
* `PubGrubPackage`: We eventually convert the `VerbatimUrl` back to a
`Dist` (`Dist::from_url`), instead of remembering the url.
* Source dist types: We use verbatim url even though we know and require
that these are supported urls we can and have parsed.

I tried to make improve the situation be replacing `VerbatimUrl`, but
these changes would require massive invasive changes (see e.g.
https://github.com/astral-sh/uv/pull/3253). A main problem is the ref
`VersionOrUrl` and applying overrides, which assume the same
requirement/url type everywhere. In its current form, this PR increases
this tech debt.

I've tried to split off PRs and commits, but the main refactoring is
still a single monolith commit to make it compile and the tests pass.

## Demo

Adding
d1ae3b85d5/pyproject.json
as json schema (v7) to pycharm for `pyproject.toml`, you can try the IDE
support already:


![pycharm](599082c7-6be5-41c1-a3cd-516092382f8d)


[dove.webm](c293c272-c80b-459d-8c95-8c46a8d198a1)
2024-05-03 21:10:50 +00:00
Andrew Gallant
1089abda3f
require serde and rkyv everywhere; remove optional serde and rkyv features (#3345)
In *some* places in our crates, `serde` (and `rkyv`) are optional
dependencies. I believe this was done out of reasons of "good sense,"
that is, it follows a Rust ecosystem pattern where serde integration
tends to be an opt-in crate feature. (And similarly for `rkyv`.)

However, ultimately, `uv` itself requires `serde` and `rkyv` to
function. Since our crates are strictly internal, there are limited
consumers for our crates without `serde` (and `rkyv`) enabled. I think
one possibility is that optional `serde` (and `rkyv`) integration means
that someone can do this:

    cargo test -p pep440_rs

And this will run tests _without_ `serde` or `rkyv` enabled. That in
turn could lead to faster iteration time by reducing compile times. But,
I'm not sure this is worth supporting. The iterative compilation times
of
individual crates are probably fast enough in debug mode, even with
`serde` and `rkyv` enabled. Namely, `serde` and `rkyv` themselves
shouldn't need to be re-compiled in most cases. On `main`:

```
from-scratch: `cargo test -p pep440_rs --lib` 0.685
incremental: `cargo test -p pep440_rs --lib` 0.278s
from-scratch: `cargo test -p pep440_rs --features serde,rkyv --lib` 3.948s
incremental: `cargo test -p pep440_rs --features serde,rkyv --lib` 0.321s
```

So while a from-scratch build does take significantly longer, an
incremental build is about the same.

The benefit of doing this change is two-fold:

1. It brings out crates into alignment with "reality." In particular,
   some crates were _implicitly_ relying on `serde` being enabled
   without explicitly declaring it. This technically means that our
   `Cargo.toml`s were wrong in some cases, but it is hard to observe it
   because of feature unification in a Cargo workspace.
2. We no longer need to deal with the cognitive burden of writing
   `#[cfg_attr(feature = "serde", ...)]` everywhere.
2024-05-03 10:21:03 -04:00
Zanie Blue
100dbe475c
Quote version parse errors consistently (#3325) 2024-04-30 12:20:14 -05:00