Commit graph

24 commits

Author SHA1 Message Date
Charlie Marsh
47290f784e
Add fixup for invalid double quotes (#663)
Closes https://github.com/astral-sh/puffin/issues/658.
2023-12-15 18:11:22 +00:00
Charlie Marsh
4fd69c74b6
Use URL rather than String in direct URL types (#643) 2023-12-14 01:01:27 +00:00
Charlie Marsh
a24eb57e93
Make warnings user-facing (#628)
## Summary

Now, `puffin_warnings::warn_once` and `puffin_warnings::warn` will go to
`stderr`, as long as the user isn't running under `--quiet`. Previously,
these went through `tracing`, and so were only visible when running
under `--verbose`.
2023-12-12 21:24:38 -05:00
Charlie Marsh
edcb71b1be
Remove some unused fields from SimpleJson (#612) 2023-12-11 23:01:37 -05:00
konsti
9806901a16
Consolidate wheel caches (#524)
After this change, two wheel caches remain: `built-wheels-v0` and
`wheels-v0`, docs screenshots below. Each contains both the wheel
metadata, cache policy and zip or unzipped wheels under the same name.

The zipped/unzipped strategy is as follows: In `pip-compile`, when we
build a wheel, we store it zipped. When `pip-sync` or a source dist
build in `pip-compile` need to install the wheel, we unzip it, remove
the file and replace it with the unzipped wheel.

This removes `WheelCache` and `UrlIndex` in favor of `Cache` plus
`WheelCache`. The non-built wheel cache now considers index urls and the
url for url wheels.

I'm unsure if we need the `Unzipper` type, this could just be a
function.

I move `no_index` into `IndexUrls` and started using `IndexUrl` up to
the clap level.

I left a number of TODOs in the code, namely performing the actual
invalidation of unzipped wheels and making the `InstallPlan` understand
cache invalidation (i.e. uninstall wheels when their remote changed).


![image](c4d45979-485b-4954-848d-fd3347ee2510)
2023-12-01 20:16:33 +00:00
Charlie Marsh
9d35128840
Use Clippy lint table over Cargo config (#490)
Closes https://github.com/astral-sh/puffin/issues/482.
2023-11-22 15:10:27 +00:00
Charlie Marsh
443a0a9df2
Use a sparse Metadata 2.1 representation (#488)
This is an optimization to avoid parsing the entire Metadata 2.1 when we
only need a small subset of the fields.

Closes #175.
2023-11-22 13:25:35 +00:00
konsti
e1dafe7203
Allow applying multiple fixups for version specifiers (#486)
Allow applying multiple fixups for version specifiers, remove the
duplication from the code and add another test case.
2023-11-22 10:26:12 +00:00
konsti
ff1100a1ab
Fixup for >= '2.7' (#485)
Fixup to allow parsing
https://pypi.org/simple/shellingham/?format=application/vnd.pypi.simple.v1+json
2023-11-22 10:00:12 +00:00
Charlie Marsh
17228ba04e
Add support for path dependencies (#471)
## Summary

This PR adds support for local path dependencies. The approach mostly
just falls out of our existing approach and infrastructure for Git and
URL dependencies.

Closes https://github.com/astral-sh/puffin/issues/436. (We'll open a
separate issue for editable installs.)

## Test Plan

Added `pip-compile` tests that pre-download a wheel or source
distribution, then install it via local path.
2023-11-21 11:49:42 +00:00
konsti
f0841cdb6e
Wheel metadata refactor (#462)
A consistent cache structure for remote wheel metadata:

 * `<wheel metadata cache>/pypi/foo-1.0.0-py3-none-any.json`
* `<wheel metadata
cache>/<digest(index-url)>/foo-1.0.0-py3-none-any.json`
* `<wheel metadata cache>/url/<digest(url)>/foo-1.0.0-py3-none-any.json`

The source dist caching will use a similar structure (#468).
2023-11-20 17:26:36 +01:00
konsti
d3e9e1783f
Refactor lenient parsing (#467)
Deduplicate lenient parsing code between version specifiers and
Requirement. Use `warn_once!` since the warnings did show up multiple
times in my code. Fix the macro hygiene in `warn_once!`.
2023-11-20 15:35:38 +00:00
konsti
46bb18f06e
Track file index (#452)
Track the index (or at least its url) where we got a file from across
the source code.

Fixes #448
2023-11-20 08:48:16 +00:00
konsti
e41ec12239
Option to resolve at a fixed timestamp with pip-compile --exclude-newer YYYY-MM-DD (#434)
This works by filtering out files with a more recent upload time, so if
the index you use does not provide upload times, the results might be
inaccurate. pypi provides upload times for all files. This is, the field
is non-nullable in the warehouse schema, but the simple API PEP does not
know this field.

If you have only pypi dependencies, this means deterministic,
reproducible(!) resolution. We could try doing the same for git repos
but it doesn't seem worth the effort, i'd recommend pinning commits
since git histories are arbitrarily malleable and also if you care about
reproducibility and such you such not use git dependencies but a custom
index.

Timestamps are given either as RFC 3339 timestamps such as
`2006-12-02T02:07:43Z` or as UTC dates in the same format such as
`2006-12-02`. Dates are interpreted as including this day, i.e. until
midnight UTC that day. Date only is required to make this ergonomic and
midnight seems like an ergonomic choice.

In action for `pandas`:

```console
$ target/debug/puffin pip-compile --exclude-newer 2023-11-16 target/pandas.in
Resolved 6 packages in 679ms
# This file was autogenerated by Puffin v0.0.1 via the following command:
#    target/debug/puffin pip-compile --exclude-newer 2023-11-16 target/pandas.in
numpy==1.26.2
    # via pandas
pandas==2.1.3
python-dateutil==2.8.2
    # via pandas
pytz==2023.3.post1
    # via pandas
six==1.16.0
    # via python-dateutil
tzdata==2023.3
    # via pandas
$ target/debug/puffin pip-compile --exclude-newer 2022-11-16 target/pandas.in
Resolved 5 packages in 655ms
# This file was autogenerated by Puffin v0.0.1 via the following command:
#    target/debug/puffin pip-compile --exclude-newer 2022-11-16 target/pandas.in
numpy==1.23.4
    # via pandas
pandas==1.5.1
python-dateutil==2.8.2
    # via pandas
pytz==2022.6
    # via pandas
six==1.16.0
    # via python-dateutil
$ target/debug/puffin pip-compile --exclude-newer 2021-11-16 target/pandas.in
Resolved 5 packages in 594ms
# This file was autogenerated by Puffin v0.0.1 via the following command:
#    target/debug/puffin pip-compile --exclude-newer 2021-11-16 target/pandas.in
numpy==1.21.4
    # via pandas
pandas==1.3.4
python-dateutil==2.8.2
    # via pandas
pytz==2021.3
    # via pandas
six==1.16.0
    # via python-dateutil
```
2023-11-16 19:46:17 +00:00
konsti
751f7fa9c6
Improve PEP 691 compatibility (#428)
[PEP 691](https://peps.python.org/pep-0691/#project-detail) has slightly
different, more relaxed rules around file metadata. These changes are
now reflected in the `File` struct. This will make it easier to support
alternative indices.

I had expected that i need to introduce a separate type for that, so i'm
happy it's two `Option`s more and an alias.

Part of #412
2023-11-16 19:03:44 +01:00
konsti
bacf1dc911
Filter out yanked files (#413)
Implement two behaviors for yanked versions:

* During `pip-compile`, yanked versions are filtered out entirely, we
currently treat them is if they don't exist. This is leads to confusing
error messages because a version that does exist seems to have suddenly
disappeared.
* During `pip-sync`, we warn when we fetch a remote distribution and it
has been yanked. We currently don't warn on cached or installed
distributions that have been yanked.
2023-11-13 20:58:50 +00:00
Charlie Marsh
28ec4e79f0
Co-locate lenient requirement parsing (#418)
No behavior changes.
2023-11-13 15:46:21 -05:00
Charlie Marsh
437d4fb87e
Add trailing-comma fix to lenient requirements (#417)
Closes https://github.com/astral-sh/puffin/issues/408.
2023-11-13 20:20:57 +00:00
Charlie Marsh
582c94cec3
Add missing-dot fix to lenient requirements (#416)
Part of https://github.com/astral-sh/puffin/issues/408.
2023-11-13 20:17:01 +00:00
konsti
76a41066ac
Filter out incompatible dists (#398)
Filter out source dists and wheels whose `requires-python` from the
simple api is incompatible with the current python version.

This change showed an important problem: When we use a fake python
version for resolving, building source distributions breaks down because
we can only build with versions we actually have.

This change became surprisingly big. The tests now require python 3.7 to
be installed, but changing that would mean an even bigger change.

Fixes #388
2023-11-13 17:14:07 +01:00
Charlie Marsh
2c32bc5a86
Respect direct URLs in puffin installer (#345)
We now write the `direct_url.json` when installing, and _skip_
installing if we find a package installed via the direct URL that the
user is requesting.

A lot of TODOs, especially around cleaning up the `Source` abstraction
and its relationship to `DirectUrl`. I'm gonna keep working on these
today, but this works and makes the requirements clear.

Closes #332.
2023-11-07 09:11:27 -05:00
konsti
c883b123ac
Allow greater than star (torch (>=1.9.*)) in lenient requirement (#351)
This appeared in the pypi top 8k testing.
2023-11-07 11:37:23 +00:00
Charlie Marsh
b013ea9c93
Move DirectUrl into pypi-types (#343)
This needs to be reused elsewhere, and there's nothing specific to wheel
installation about it.
2023-11-06 18:26:33 +00:00
Charlie Marsh
24e30e6557
Split puffin-package into requirements.txt parser and pypi-types (#341)
There are only two things left in this crate and they don't really have
anything to do with one another.
2023-11-06 18:19:49 +00:00