Commit graph

112 commits

Author SHA1 Message Date
Charlie Marsh
8587c440fe
Support file://localhost/ schemes (#2657)
## Summary

`uv` was failing to install requirements defined like:

```
file://localhost/Users/crmarsh/Downloads/iniconfig-2.0.0-py3-none-any.whl
```

Closes https://github.com/astral-sh/uv/issues/2652.
2024-03-25 19:23:26 +00:00
Hans Baker
a632d2485b
Don't error on multiple matching index URLs (#2627)
## Summary

Closes Issue:
- https://github.com/astral-sh/uv/issues/2626

## Test Plan

```
cargo run -- pip install -r dev-requirements.txt -r requirements.txt
```
where both requirements files have same `--index-url`
2024-03-22 23:37:02 +00:00
konsti
48bd02b8a8
Update miette v7, pubgrub and small Cargo.toml cleanup (#2610)
I was going through the output of `cargo tree --duplicate -p uv`, not
much success except these small cleanups.
2024-03-22 10:42:48 +00:00
Charlie Marsh
31743f2bd8
Use PEP 517 build hooks to resolve unnamed requirements (#2604) 2024-03-22 03:20:40 +00:00
Charlie Marsh
5d7d7dce24
Enable PEP 517 builds for unnamed requirements (#2600)
## Summary

This PR enables the source distribution database to be used with unnamed
requirements (i.e., URLs without a package name). The (significant)
upside here is that we can now use PEP 517 hooks to resolve unnamed
requirement metadata _and_ reuse any computation in the cache.

The changes to `crates/uv-distribution/src/source/mod.rs` are quite
extensive, but mostly mechanical. The core idea is that we introduce a
new `BuildableSource` abstraction, which can either be a distribution,
or an unnamed URL:

```rust
/// A reference to a source that can be built into a built distribution.
///
/// This can either be a distribution (e.g., a package on a registry) or a direct URL.
///
/// Distributions can _also_ point to URLs in lieu of a registry; however, the primary distinction
/// here is that a distribution will always include a package name, while a URL will not.
#[derive(Debug, Clone, Copy)]
pub enum BuildableSource<'a> {
    Dist(&'a SourceDist),
    Url(SourceUrl<'a>),
}
```

All the methods on the source distribution database now accept
`BuildableSource`. `BuildableSource` has a `name()` method, but it
returns `Option<&PackageName>`, and everything is required to work with
and without a package name.

The main drawback of this approach (which isn't a terrible one) is that
we can no longer include the package name in the cache. (We do continue
to use the package name for registry-based distributions, since those
always have a name.). The package name was included in the cache route
for two reasons: (1) it's nice for debugging; and (2) we use it to power
`uv cache clean flask`, to identify the entries that are relevant for
Flask.

To solve this, I changed the `uv cache clean` code to look one level
deeper. So, when we want to determine whether to remove the cache entry
for a given URL, we now look into the directory to see if there are any
wheels that match the package name. This isn't as nice, but it does work
(and we have test coverage for it -- all passing).

I also considered removing the package name from the cache routes for
non-registry _wheels_, for consistency... But, it would require a cache
bump, and it didn't feel important enough to merit that.
2024-03-21 22:46:39 -04:00
Tim de Jager
fc9761d020
feat: make direct_url public (#2584)
## Summary

Made the `direct_url` method public because it was not :) Can be merged
instead of in addition to: https://github.com/astral-sh/uv/pull/2510
2024-03-21 09:37:15 -04:00
veryyet
d6dad57fab
chore: fix some typos (#2581) 2024-03-21 04:09:37 +00:00
konsti
70e0967dbd
Avoid repeating paths of workspace packages (#2573)
Scott schafer got me the idea: We can avoid repeating the path for
workspaces dependencies everywhere if we declare them in the virtual
package once and treat them as workspace dependencies from there on.
2024-03-20 16:16:02 -04:00
Charlie Marsh
00fc44012c
Use relative paths for user display (#2559)
## Summary

This PR changes our user-facing representation for paths to use relative
paths, when the path is within the current working directory. This
mirrors what we do in Ruff. (If the path is _outside_ the current
working directory, we print an absolute path.)

Before:

```shell
❯ uv venv .venv2
Using Python 3.12.2 interpreter at: /Users/crmarsh/workspace/uv/.venv/bin/python3
Creating virtualenv at: .venv2
Activate with: source .venv2/bin/activate
```

After:

```shell
❯ cargo run venv .venv2
    Finished dev [unoptimized + debuginfo] target(s) in 0.15s
     Running `target/debug/uv venv .venv2`
Using Python 3.12.2 interpreter at: .venv/bin/python3
Creating virtualenv at: .venv2
Activate with: source .venv2/bin/activate
```

Note that we still want to use the existing `.simplified_display()`
anywhere that the path is being simplified, but _still_ intended for
machine consumption (e.g., when passing to `.current_dir()`).
2024-03-20 09:52:50 -04:00
Micha Reiser
acbee166c0
Remove unused dependencies (#2543)
## Summary

I tried out `cargo shear` to see if there are any unused dependencies
that `cargo udeps` isn't reporting. It turned out, there are a few. This
PR removes those dependencies.

## Test Plan

`cargo build`
2024-03-19 13:10:10 -04:00
Charlie Marsh
2b01d9f70b
Validate required package names against wheel package names (#2516)
Closes https://github.com/astral-sh/uv/issues/2484.
2024-03-18 17:35:15 +00:00
konsti
060a2fb80b
Shrink solve() a little by moving formatting out (#2506)
`solve()` is our main loop. Shrinking it down a little by moving
formatting out makes it easier to follow.
2024-03-18 12:27:24 +00:00
Charlie Marsh
8463d6d672
Apply from-URL credentials in authentication middleware (#2449)
## Summary

Right now, the middleware doesn't apply credentials that were
_originally_ sourced from a URL. This requires that we call
`with_url_encoded_auth` whenever we create a request to ensure that any
credentials that were passed in as part of an index URL (for example)
are respected.

This PR modifies `uv-auth` to instead apply those credentials in the
middleware itself. This seems preferable to me. As far as I can tell, we
can _only_ add in-URL credentials to the store ourselves (since in-URL
credentials are converted to headers by the time they reach the
middleware). And if we ever _didn't_ apply those credentials to new
URLs, it'd be a bug in the logic that precedes the middleware (i.e., us
forgetting to call `with_url_encoded_auth`).

## Test Plan

`cargo run pip install` with an authenticated index.
2024-03-15 16:21:37 +00:00
Charlie Marsh
f1aec3e779
Add in-URL credentials to store prior to creating requests (#2446)
## Summary

The authentication middleware extracts in-URL credentials from URLs that
pass through it; however, by the time a request reaches the store, the
credentials will have already been removed, and relocated to the header.
So we were never propagating in-URL credentials.

This PR adds an explicit pass wherein we pass in-URL credentials to the
store prior to doing any work.

Closes https://github.com/astral-sh/uv/issues/2444.

## Test Plan

`cargo run pip install` against an authenticated AWS registry.
2024-03-14 03:46:33 +00:00
Zanie Blue
22a52391be
Refactor AuthenticationStore to inline credentials (#2427) 2024-03-13 17:48:02 -05:00
Hans Baker
9159731792
Add support for retrieving credentials from keyring (#2254)
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

<!-- What's the purpose of the change? What does it do, and why? -->

Adds basic keyring auth support for `uv` commands. Adds clone of `pip`'s
`--keyring-provider subprocess` argument (using CLI `keyring` tool).

See issue: https://github.com/astral-sh/uv/issues/1520

## Test Plan

<!-- How was it tested? -->

Hard to write full-suite unit tests due to reliance on
`process::Command` for `keyring` cli

Manually tested end-to-end in a project with GCP artifact registry using
keyring password:
```bash
➜  uv pip uninstall watchdog
Uninstalled 1 package in 46ms
 - watchdog==4.0.0

➜  cargo run -- pip install --index-url https://<redacted>/python/simple/ --extra-index-url https://<redacted>/pypi-mirror/simple/ watchdog
    Finished dev [unoptimized + debuginfo] target(s) in 0.18s
     Running `target/debug/uv pip install --index-url 'https://<redacted>/python/simple/' --extra-index-url 'https://<redacted>/pypi-mirror/simple/' watchdog`
error: HTTP status client error (401 Unauthorized) for url (https://<redacted>/pypi-mirror/simple/watchdog/)

➜  cargo run -- pip install --keyring-provider subprocess --index-url https://<redacted>/python/simple/ --extra-index-url https://<redacted>/pypi-mirror/simple/ watchdog
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/uv pip install --keyring-provider subprocess --index-url 'https://<redacted>/python/simple/' --extra-index-url 'https://<redacted>/pypi-mirror/simple/' watchdog`
Resolved 1 package in 2.34s
Installed 1 package in 27ms
 + watchdog==4.0.0
```

`requirements.txt`
```
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    .bin/generate-requirements
#
--index-url https://<redacted>/python/simple/
--extra-index-url https://<redacted>/pypi-mirror/simple/

...
```

```bash
➜  cargo run -- pip install --keyring-provider subprocess -r requirements.txt
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
     Running `target/debug/uv pip install --keyring-provider subprocess -r requirements.txt`
Resolved 205 packages in 23.52s
   Built <redacted>
   ...
Downloaded 47 packages in 19.32s
Installed 195 packages in 276ms
 + <redacted>
  ...
```

---------

Co-authored-by: Thomas Gilgenast <thomas@vant.ai>
Co-authored-by: Zanie Blue <contact@zanie.dev>
2024-03-13 15:02:18 -05:00
Charlie Marsh
7220894ffb
Expand environment variables prior to detecting scheme (#2394)
## Summary

This PR ensures that we expand environment variables _before_ sniffing
for the URL scheme (e.g., `file://` vs. `https://` vs. something else).

Closes https://github.com/astral-sh/uv/issues/2375.
2024-03-12 19:17:41 -04:00
Zanie Blue
10c4effbd3
Refactor incompatiblity tracking for distributions (#1298)
Extends the "compatibility" types introduced in #1293 to apply to source
distributions as well as wheels.

- We now track the most-relevant incompatible source distribution
- Exclude newer, Python requirements, and yanked versions are all
tracked as incompatibilities in the new model (this lets us remove
`DistMetadata`!)
2024-03-08 11:02:31 -06:00
Charlie Marsh
2e9678e5d3
Add support for Metadata 2.2 (#2293)
## Summary

PyPI now supports Metadata 2.2, which means distributions with Metadata
2.2-compliant metadata will start to appear. The upside is that if a
source distribution includes a `PKG-INFO` file with (1) a metadata
version of 2.2 or greater, and (2) no dynamic fields (at least, of the
fields we rely on), we can read the metadata from the `PKG-INFO` file
directly rather than running _any_ of the PEP 517 build hooks.

Closes https://github.com/astral-sh/uv/issues/2009.
2024-03-08 16:02:32 +00:00
Andrew Gallant
5e351343da
tweak the order of index priority (#2083)
Previously, `uv` would always prioritize the index given by
`--index-url`. It would then try any indexes after that given by zero
or more `--extra-index-url` flags. This differed from `pip` in that any
priority was given at all, where `pip` doesn't guarantee any priority
ordering of indexes.

We could go in the direction of mimicing `pip`'s behavior here, but it
at present has issues with dependency confusion attacks where packages
may get installed from indexes you don't control. More specifically,
there is an issue of different trust levels. See discussion in #171 and
[PEP-0708] for more on the security impact.

In contrast, `uv` will only select versions for a package from a single
index. That is, even if `foo` is in indexes `a` and `b`, it will
only consider the versions from the index that it checks first. This
probably helps with respect to dependency confusion attacks, but also
means that `uv` doesn't quite cover all of the same use cases as `pip`.

In this PR, we retain the notion of prioritizing indexes, but
tweak it so that PyPI is preferred last as opposed to first. Or
more precisely, the `--index-url` flag specifies a fallback index,
not the primary index, and is deprioritized beneath every index
specified by `--extra-index-url`. The ordering among indexes given by
`--extra-index-url` remains the same: earlier indexes are prioritized
over later indexes.

While this tweak likely won't hit all use cases, I believe it will
resolve some of the most common pain points without exacerbating
dependency confusion problems.

Ref #171, Fixes #1377, Fixes #1451, Fixes #1600

[PEP-0708]: https://peps.python.org/pep-0708/
2024-02-29 11:57:07 -05:00
Charlie Marsh
ef15098288
Use Simplified instead of Normalized for path prefix stripping (#2071)
## Summary

This directly matches the naming of the `dunce` methods.
2024-02-29 01:44:50 +00:00
Charlie Marsh
9c63412526
Track wheel compatibility as a single field (#2054)
## Summary

Internal refactor to `PrioritizedDistribution` that I think should
reduce the size? Although the motivation here is simplicity, not perf.

Instead of storing:

```rust
/// The highest-priority, installable wheel for the package version.
compatible_wheel: Option<(DistMetadata, TagPriority)>,
/// The most-relevant, incompatible wheel for the package version.
incompatible_wheel: Option<(DistMetadata, IncompatibleWheel)>,
```

We now store:

```rust
wheel: Option<(DistMetadata, WheelCompatibility)>,
```

Where `WheelCompatibility` is an enum of `TagPriority` or
`IncompatibleWheel`.
2024-02-28 16:59:22 -05:00
Charlie Marsh
b873e3e991
Support environment variables in index URLs in requirements files (#2036)
## Summary

This also preserves the environment variables in the output file, e.g.:

```
Resolved 1 package in 216ms
# This file was autogenerated by uv via the following command:
#    uv pip compile requirements.in --emit-index-url
--index-url https://test.pypi.org/${SUFFIX}

requests==2.5.4.1
```

I'm torn on whether that's correct or undesirable here.

Closes #2035.
2024-02-28 19:36:20 +00:00
Simon Brugman
0f1377bb08
uv pip list (#1662)
Hi, love your work on `uv` 👋! 

Opening a Draft PR early to check if there are any existing rust table
formatting libs that I am unaware of (either already in `uv`/`ruff`, or
the rust ecosystem) before spending much time on inventing the wheel
myself and cleaning it up. Any other pointers are also welcome (e.g. on
the editable filtering).

Editable project locations in `uv pip list` include the file scheme
(`file://`), where they are omitted in `pip list`. Is this desired, or
should it replicate pip?

## Summary

Implementation for #1401 
`--editable` flag is implemented.

`--outdated` and `--uptodate` out of scope for this PR (requires latest
version information, and type wheel/sdist)

## Test Plan

Not yet implemented as I couldn't locate the tests for `uv pip freeze`.
We can compare to `pip` in
`scripts/compare_with_pip/compare_with_pip.py`?
2024-02-25 19:42:27 +00:00
danieleades
8d721830db
Clippy pedantic (#1963)
Address a few pedantic lints

lints are separated into separate commits so they can be reviewed
individually.

I've not added enforcement for any of these lints, but that could be
added if desirable.
2024-02-25 14:04:05 -05:00
Charlie Marsh
c69e270071
Gracefully handle virtual environments with conflicting packages (#1893)
## Summary

When a virtual environment contains multiple packages with the same
name, we no longer throw a hard error. Instead:

- In `uv pip freeze`, we list all versions.
- In `uv pip uninstall`, we uninstall all versions.
- In `uv pip install`, we uninstall all versions prior to installing a
new version.

Closes https://github.com/astral-sh/uv/issues/1848.
2024-02-23 19:14:52 +00:00
Zanie Blue
8a12b2ebf9
Ensure authentication is passed from the index url to distribution files (#1886)
Closes https://github.com/astral-sh/uv/issues/1709
Closes https://github.com/astral-sh/uv/issues/1371

Tested with the reproduction provided in #1709 which gets past the HTTP
401.

Reuses the same copying logic we introduced in
https://github.com/astral-sh/uv/pull/1874 to ensure authentication is
attached to file URLs with a realm that matches that of the index. I had
to move the authentication logic into a new crate so it could be used in
`distribution-types`.

We will want to something more robust in the future, like track all
realms with authentication in a central store and perform lookups there.
That's what `pip` does and it allows consolidation of logic like netrc
lookups. That refactor feels significant though, and I'd like to get
this fixed ASAP so this is a minimal fix.
2024-02-22 18:10:17 -06:00
Charlie Marsh
a2a1b2fb0f
Avoid enforcing URL correctness for installed distributions (#1793)
## Summary

Allows the corresponding `pypi_types` struct to use any URL, since other
installers can put those into the environment, and Poetry seems to write
invalid URLs.

If we see a distribution with an invalid URL, we just treat it as a
registry distribution, which isn't ideal, but is better than (1)
erroring, and (2) changing `Url` to `String` everywhere internally. (I'm
torn on this second option.)

Closes https://github.com/astral-sh/uv/issues/1744.

## Test Plan

- Added `flask = { git = "git@github.com:pallets/flask.git", rev =
"b90a4f1f4a370e92054b9cc9db0efcb864f87ebe" }` to
`scripts/editable-installs/poetry_editable/pyproject.toml`.
- Ran `poetry install`.
- Ran `cargo pip freeze`. Verified that it errored on `main`, but passed
here.
- Ran `cargo run pip install "flask==3.0.0"`. Verified that it
uninstalled the existing Flask, and installed a new version from the
registry.
2024-02-21 09:06:31 -05:00
Tim de Jager
8cbeab7107
feat: add installer to InstalledDist (#1762)
## Summary

Add `installer` method to `InstalledDist` to distinguish between
different installers. Might be nice to add an enum for all possible
installers, but this might be too hard to keep up to date :).

The `INSTALLER` file is a file that can be added to the `.dist-info`
folder with the installer name.

Closes: #1759 

## Test Plan

Not sure if there is a place I can automatically test it, if you have a
pointer I would be happy to add a test.
2024-02-20 17:07:08 +00:00
Charlie Marsh
c05080a3e3
Add support for absolute paths on Windows (#1725)
## Summary

The main change is that we need to have an explicit list of protocols we
_do_ support (like `https`), so that when we see a Windows absolute path
(`C:\...`), we don't treat the `C` as a protocol itself.

Closes https://github.com/astral-sh/uv/issues/1539.
2024-02-20 01:36:53 +00:00
Charlie Marsh
034f62b24f
Respect --index-url provided via requirements.txt (#1719)
## Summary

When we read `--index-url` from a `requirements.txt`, we attempt to
respect the `--index-url` provided by the CLI if it exists.
Unfortunately, `--index-url` from the CLI has a default value... so we
_never_ respect the `--index-url` in the requirements file.

This PR modifies the CLI to use `None`, and moves the default into logic
in the `IndexLocations `struct.

Closes https://github.com/astral-sh/uv/issues/1692.
2024-02-20 00:02:26 +00:00
Zanie Blue
563c636aa0
Improve tracing when encountering invalid requires-python values (#1568)
Unsure what the easiest way to test this is, it is hard to publish
invalid requires-python specifiers with hatchling
2024-02-17 07:40:13 -05:00
Charlie Marsh
f897ee3f88
Allow repeated dependencies when installing (#1558)
## Summary

It turns out that it's not uncommon to end up with repeated packages in
requirements files when running `pip-sync`, e.g., you might have
`anyio==4.0.0` specified multiple times. This PR relaxes our assertions
in the install plan to allow such repeated packages, as long as the
requirement markers are exactly the same (i.e., they are truly
duplicates).

Closes https://github.com/astral-sh/uv/issues/1552.
2024-02-17 01:33:40 +00:00
Charlie Marsh
c1eb6130e1
Support MD5 hashes (#1556)
## Summary

We can add other hashes if necessary, but I don't know that they're
really used in practice.

Closes https://github.com/astral-sh/uv/issues/1547.
2024-02-17 00:25:16 +00:00
Charlie Marsh
9e0336c28a
Remove URL encoding when determining file name (#1555)
## Summary

Closes https://github.com/astral-sh/uv/issues/1553.
2024-02-16 19:15:24 -05:00
Charlie Marsh
b4ea48955b
Use comparable representation for PackageId (#1543)
## Summary

By using the display representation of `Version` to form a `PackageId`,
we run the risk (as seen in the linked issue) of thinking that versions
like `2021.1` and `2021.1.0` are not equivalent.

Closes https://github.com/astral-sh/uv/issues/1536
2024-02-16 16:30:54 -05:00
Zanie Blue
2586f655bb
Rename to uv (#1302)
First, replace all usages in files in-place. I used my editor for this.
If someone wants to add a one-liner that'd be fun.

Then, update directory and file names:

```
# Run twice for nested directories
find . -type d -print0 | xargs -0 rename s/puffin/uv/g
find . -type d -print0 | xargs -0 rename s/puffin/uv/g

# Update files
find . -type f -print0 | xargs -0 rename s/puffin/uv/g
```

Then add all the files again

```
# Add all the files again
git add crates
git add python/uv

# This one needs a force-add
git add -f crates/uv-trampoline
```
2024-02-15 11:19:46 -06:00
Zanie Blue
e9e3e573a2
Report incompatible distributions to users (#1293)
Instead of dropping versions without a compatible distribution, we track
them as incompatibilities in the solver. This implementation follows
patterns established in https://github.com/astral-sh/puffin/pull/1290.

This required some significant refactoring of how we track incompatible
distributions. Notably:

- `Option<TagPriority>` is now `WheelCompatibility` which allows us to
track the reason a wheel is incompatible instead of just `None`.
- `Candidate` now has a `CandidateDist` with `Compatible` and
`Incompatibile` variants instead of just `ResolvableDist`; candidates
are not strictly compatible anymore
- `ResolvableDist` was renamed to `CompatibleDist`
- `IncompatibleWheel` was given an ordering implementation so we can
track the "most compatible" (but still incompatible) wheel. This allows
us to collapse the reason a version cannot be used to a single
incompatibility.
- The filtering in the `VersionMap` is retained, we still only store one
incompatible wheel per version. This is sufficient for error reporting.
- A `TagCompatibility` type was added for tracking which part of a wheel
tag is incompatible
- `Candidate::validate_python` moved to
`PythonRequirement::validate_dist`

I am doing more refactoring in #1298 — I think a couple passes will be
necessary to clarify the relationships of these types.

Includes improved error message snapshots for multiple incompatible
Python tag types from #1285 — we should add more scenarios for coverage
of behavior when multiple tags with different levels are present.
2024-02-15 10:48:15 -06:00
Zanie Blue
7fec2a311a
Refactor storage of distribution metadata needed in resolver (#1291)
Follows #1290 and https://github.com/astral-sh/puffin/pull/912 with some
minor clean-up.
2024-02-13 04:19:21 +00:00
Zanie Blue
b5dd8b7de2
Track yanked versions as incompatibilities (#1290)
Moves yanked version filtering from `VersionMap::from_metadata` to the
resolver and tracks it as a PubGrub unavailable incompatibility so
yanked versions are reflected in error messages.

e.g. before
```
╰─▶ Because only albatross<=0.1.0 is available and you require albatross>0.1.0, 
       we can conclude that the requirements are unsatisfiable.
```

after

```
╰─▶ Because only the following versions of albatross are available:
            albatross<=0.1.0
            albatross==1.0.0
      and albatross==1.0.0 is unusable because it was yanked, we can conclude that albatross>0.1.0 cannot be used.
      And because you require albatross>0.1.0, we can conclude that the requirements are unsatisfiable.
```
2024-02-12 22:01:17 -06:00
Zanie Blue
a37b08808e
Implement pip compatible --no-binary and --only-binary options (#1268)
Updates our `--no-binary` option and adds a `--only-binary` option for
compatibility with `pip` which uses `:all:`, `:none:` and `<name>` for
specifying packages.

This required adding support for `--only-binary <name>` into our
resolver, previously it was only a boolean toggle.

Retains`--no-build` which is equivalent to `--only-binary :all:`. This
is common enough for safety that I would prefer it is available without
pip's awkward `:all:` syntax.

---------

Co-authored-by: konsti <konstin@mailbox.org>
2024-02-11 19:31:41 -06:00
Charlie Marsh
d98b3c3070
Strip UNC prefix when setting working directory (#1277)
## Summary

For PEP 517 builds, the current working directory needs to be set to the
directory of the source distribution. It turns out that on Windows, if
you use a UNC path for the working directory, then relative paths are
interpreted relative to the root of the current drive
([source](https://www.fileside.app/blog/2023-03-17_windows-file-paths/#paths-relative-to-the-root-of-the-current-drive)).
So, when builds attempted to resolve relative paths, they always
errored...

This PR ensures that we remove the UNC prefix when setting the current
working directory.

Closes #1238.

## Test Plan

I tested this on my Windows machine by installing `ujson` with
`--no-binary ujson`. (I don't want to add that specific test, since it's
really slow to build.)
2024-02-12 00:51:36 +00:00
Andrew Gallant
96276d9e3e
puffin-resolver: simplify version map construction (#1267)
In the process of making VersionMap construction lazy, I realized this
refactoring would be useful to me. It also simplifies a fair bit of case
analysis and does fewer BTreeMap lookups during construction. With that
said, this doesn't seem to matter for perf:

```
$ hyperfine -w10 --runs 50 \
    "puffin-main pip compile --cache-dir ~/astral/tmp/cache-main ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null" \
    "puffin-test pip compile --cache-dir ~/astral/tmp/cache-test ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null"
Benchmark 1: puffin-main pip compile --cache-dir ~/astral/tmp/cache-main ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null
  Time (mean ± σ):     146.8 ms ±   4.1 ms    [User: 350.1 ms, System: 314.2 ms]
  Range (min … max):   140.7 ms … 158.0 ms    50 runs

Benchmark 2: puffin-test pip compile --cache-dir ~/astral/tmp/cache-test ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null
  Time (mean ± σ):     146.8 ms ±   4.5 ms    [User: 359.8 ms, System: 308.3 ms]
  Range (min … max):   138.2 ms … 160.1 ms    50 runs

Summary
  puffin-main pip compile --cache-dir ~/astral/tmp/cache-main ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null ran
    1.00 ± 0.04 times faster than puffin-test pip compile --cache-dir ~/astral/tmp/cache-test ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null
```

But the simplification is still nice, and will decrease the delta
between what we have now and a lazy version map.
2024-02-08 15:33:33 -05:00
konsti
809c6d676f
Use normalized display in tests and other small windows fixes (#1228)
Split out from the large test refactoring PR. Use `normalized_display`
in tests and two more thiserror derives to match snapshots and output,
and other small windows fixes.
2024-02-01 16:12:30 +01:00
Charlie Marsh
67a09649f2
Support parsing --find-links, --index-url, and --extra-index-url in requirements.txt (#1146)
## Summary

This PR adds support for `--find-links`, `--index-url`, and
`--extra-index-url` arguments when specified in a `requirements.txt`.

It's a mostly-straightforward change. The only uncertain piece is what
to do when multiple files include these flags, and/or when we include
them on the CLI and in other files.

In general:

- If _anything_ specifies `--no-index`, we respect it.
- We combine all `--extra-index-url` and `--find-links` across all
sources, since those are just vectors.
- If we see multiple `--index-url` in requirements files, we error.
- We respect the `--index-url` from the command line over any provided
in a requirements file.

(`pip-compile` seems to just pick one semi-arbitrarily when multiple are
provided.)

Closes https://github.com/astral-sh/puffin/issues/1143.
2024-01-29 15:06:40 +00:00
konsti
be48200642
Small instrumentation improvements (#1164)
Less verbose span fields for `Dist`s by using the display impl and no
more min length in the tracing durations plot config for comparability
(we lose spans due to a speedup otherwise). Both wait points in the
solver loop are now instrumented so we can inspect what we're waiting
for to progress in the solver.
2024-01-29 10:55:19 +00:00
Andrew Gallant
5219d37250
add initial rkyv support (#1135)
This PR adds initial support for [rkyv] to puffin. In particular,
the main aim here is to make puffin-client's `SimpleMetadata` type
possible to deserialize from a `&[u8]` without doing any copies. This
PR **stops short of actuallying doing that zero-copy deserialization**.
Instead, this PR is about adding the necessary trait impls to a variety
of types, along with a smattering of small refactorings to make rkyv
possible to use.

For those unfamiliar, rkyv works via the interplay of three traits:
`Archive`, `Serialize` and `Deserialize`. The usual flow of things is
this:

* Make a type `T` implement `Archive`, `Serialize` and `Deserialize`.
rkyv
helpfully provides `derive` macros to make this pretty painless in most
  cases.
* The process of implementing `Archive` for `T` *usually* creates an
entirely
new distinct type within the same namespace. One can refer to this type
without naming it explicitly via `Archived<T>` (where `Archived` is a
clever
  type alias defined by rkyv).
* Serialization happens from `T` to (conceptually) a `Vec<u8>`. The
serialization format is specifically designed to reflect the in-memory
layout
  of `Archived<T>`. Notably, *not* `T`. But `Archived<T>`.
* One can then get an `Archived<T>` with no copying (albeit, we will
likely
need to incur some cost for validation) from the previously created
`&[u8]`.
This is quite literally [implemented as a pointer cast][rkyv-ptr-cast].
* The problem with an `Archived<T>` is that it isn't your `T`. It's
something
  else. And while there is limited interoperability between a `T` and an
`Archived<T>`, the main issue is that the surrounding code generally
demands
a `T` and not an `Archived<T>`. **This is at the heart of the tension
for
  introducing zero-copy deserialization, and this is mostly an intrinsic
problem to the technique and not an rkyv-specific issue.** For this
reason,
  given an `Archived<T>`, one can get a `T` back via an explicit
deserialization step. This step is like any other kind of
deserialization,
although generally faster since no real "parsing" is required. But it
will
  allocate and create all necessary objects.

This PR largely proceeds by deriving the three aforementioned traits
for `SimpleMetadata`. And, of course, all of its type dependencies. But
we stop there for now.

The main issue with carrying this work forward so that rkyv is actually
used to deserialize a `SimpleMetadata` is figuring out how to deal
with `DataWithCachePolicy` inside of the cached client. Ideally, this
type would itself have rkyv support, but adding it is difficult. The
main difficulty lay in the fact that its `CachePolicy` type is opaque,
not easily constructable and is internally the tip of the iceberg of
a rat's nest of types found in more crates such as `http`. While one
"dumb"-but-annoying approach would be to fork both of those crates
and add rkyv trait impls to all necessary types, it is my belief that
this is the wrong approach. What we'd *like* to do is not just use
rkyv to deserialize a `DataWithCachePolicy`, but we'd actually like to
get an `Archived<DataWithCachePolicy>` and make actual decisions used
the archived type directly. Doing that will require some work to make
`Archived<DataWithCachePolicy>` directly useful.

My suspicion is that, after doing the above, we may want to mush
forward with a similar approach for `SimpleMetadata`. That is, we want
`Archived<SimpleMetadata>` to be as useful as possible. But right
now, the structure of the code demands an eager conversion (and thus
deserialization) into a `SimpleMetadata` and then into a `VersionMap`.
Getting rid of that eagerness is, I think, the next step after dealing
with `DataWithCachePolicy` to unlock bigger wins here.

There are many commits in this PR, but most are tiny. I still encourage
review to happen commit-by-commit.

[rkyv]: https://rkyv.org/
[rkyv-ptr-cast]:
https://docs.rs/rkyv/latest/src/rkyv/util/mod.rs.html#63-68
2024-01-28 12:14:59 -05:00
Charlie Marsh
15ca17a68d
Support relative file: paths for --find-links (#1147)
Just for consistency.
2024-01-27 03:48:25 +00:00
Charlie Marsh
addb94fbd6
Add support for emitting index URLs and --find-links (#1142)
Closes https://github.com/astral-sh/puffin/issues/1140.
2024-01-27 01:37:55 +00:00
Charlie Marsh
7755f986c3
Support extras in editable requirements (#1113)
## Summary

This PR adds support for requirements like `-e .[d]`.

Closes #1091.
2024-01-26 12:07:51 +00:00