Commit graph

513 commits

Author SHA1 Message Date
konsti
889f6173cc
Unify python interpreter abstractions (#178)
Previously, we had two python interpreter metadata structs, one in
gourgeist and one in puffin. Both would spawn a subprocess to query
overlapping metadata and both would appear in the cli crate, if you
weren't careful you could even have to different base interpreters at
once. This change unifies this to one set of metadata, queried and
cached once.

Another effect of this crate is proper separation of python interpreter
and venv. A base interpreter (such as `/usr/bin/python/`, but also pyenv
and conda installed python) has a set of metadata. A venv has a root and
inherits the base python metadata except for `sys.prefix`, which unlike
`sys.base_prefix`, gets set to the venv root. From the root and the
interpreter info we can compute the paths inside the venv. We can reuse
the interpreter info of the base interpreter when creating a venv
without having to query the newly created `python`.
2023-10-25 20:11:36 +00:00
konsti
1fbe328257
Build source distributions in the resolver (#138)
This is isn't ready, but it can resolve
`meine_stadt_transparent==0.2.14`.

The source distributions are currently being built serially one after
the other, i don't know if that is incidentally due to the resolution
order, because sdist building is blocking or because of something in the
resolver that could be improved.

It's a bit annoying that the thing that was supposed to do http requests
now suddenly also has to a whole download/unpack/resolve/install/build
routine, it messes up the type hierarchy. The much bigger problem though
is avoid recursive crate dependencies, it's the reason for the callback
and for splitting the builder into two crates (badly named atm)
2023-10-25 20:05:13 +00:00
konsti
b5c57ee6fe
Fix rustdoc warnings (#182)
Changes to make `cargo doc --all --all-features` pass without warnings.
2023-10-25 11:48:24 +00:00
Charlie Marsh
d0aeb2ac80
Remove vector allocation in WheelFilename (#177) 2023-10-24 01:23:14 +00:00
Charlie Marsh
21bb9c29cc
Add an additional requirements fixup (#174)
Also checking in a variety of different requirements inputs.
2023-10-23 19:50:39 -04:00
konstin
815c2117c8 Clippy 2023-10-23 13:54:31 +02:00
Charlie Marsh
0e097874f8
Add support for alternate index URLs (#169)
As elsewhere, we just use the `pip` and `pip-compile` APIs. So we
support `--index-url` to override PyPI, then `--extra-index-url` to add
_additional_ indexes, and `--no-index` to avoid hitting the index at
all.

Closes #156.
2023-10-23 03:18:30 +00:00
Charlie Marsh
49a27ff33c
Add support for parameterized link modes (#164)
Allows the user to select between clone, hardlink, and copy semantics
for installs. (The pnpm documentation has a decent description of what
these mean: https://pnpm.io/npmrc#package-import-method.)

Closes #159.
2023-10-22 04:35:50 +00:00
Charlie Marsh
9bcc7fe77a
Move venv command to miette (#162) 2023-10-22 04:17:16 +00:00
Charlie Marsh
370771b28c
Make clean non-async (#163) 2023-10-22 03:54:13 +00:00
Charlie Marsh
b665f1489a
Add tests for puffin sync (#161)
Closes #158.
2023-10-22 03:25:00 +00:00
Charlie Marsh
3072c3265e
Add support for lowest and lowest-direct resolution modes (#160)
Borrows terminology from pnpm by introducing three resolution modes:

- "Highest": always choose the highest compliant version (default).
- "Lowest": always choose the lowest compliant version.
- "LowestDirect": choose the lowest compliant version of direct
dependencies, and the highest compliant version of any transitive
dependencies. (This makes a bit more sense than "lowest".)

Closes https://github.com/astral-sh/puffin/issues/142.
2023-10-21 22:58:06 -04:00
konsti
ae9d1f7572
Add source distribution filename abstraction (#154)
The need for this became clear when working on the source distribution
integration into the resolver.

While at it i also switch the `WheelFilename` version to the parsed
`pep440_rs` version now that we have this crate.
2023-10-20 17:45:57 +02:00
Charlie Marsh
6f52b5ca4d
Use index instead of current selection (#155)
We can also use `swap_remove` because we're discarding the vector.
2023-10-20 14:02:24 +00:00
Charlie Marsh
4645f79237
Use FxHash (#151) 2023-10-20 05:26:06 +00:00
Charlie Marsh
8001c792e7
Show requirement sources in pip-compile output (#149)
Builds up a complete resolved graph from PubGrub, and shows the sources
that led to each package being included in the resolution, like
`pip-compile`.

Closes https://github.com/astral-sh/puffin/issues/60.
2023-10-20 05:14:59 +00:00
Charlie Marsh
e662fe341b
Short-circuit when a dependency has no matching versions (#148)
Kind of an oversight in my initial implementation. If we find that any
package has _no_ matching versions, we should select it! This lets us
short-circuit _immediately_ when top-level dependencies aren't
satisfiable.
2023-10-20 03:49:20 +00:00
Charlie Marsh
9b3405bf0e
Upgrade PubGrub to dev branch (#147)
Updates to `29c48fb9f3daa11bd02794edd55060d0b01ee705` from the
`pubgrub-rs` dev branch. This lets us reduce the number of changes we've
made to PubGrub itself (now, only changing visibility to export a few
things from the `solver.rs` module).
2023-10-20 03:23:26 +00:00
Charlie Marsh
bcd281eb1f
Remove async from some filesystem-only APIs (#146) 2023-10-20 01:08:51 +00:00
Charlie Marsh
03101c6a5c
Add an autogeneration header to pip-compile (#145)
Closes https://github.com/astral-sh/puffin/issues/132.
2023-10-19 20:57:27 -04:00
Charlie Marsh
0b60804db6
Add support for constraints during pip-compile resolution (#144)
Closes https://github.com/astral-sh/puffin/issues/130.
2023-10-20 00:24:05 +00:00
Charlie Marsh
d5105a76c5
Improve and test diagnostics for requirements-reading CLI commands (#143)
Also removes `owo_colors` because it was really painful to get it to
avoid printing colors during tests.
2023-10-19 18:13:40 -04:00
Charlie Marsh
ba181eacdd
Accept dependencies from pyproject.toml (#141)
Doesn't support extras yet. It's also supported for `pip uninstall`,
which `pip` itself doesn't support, but whatever.

Closes #127.
2023-10-19 18:42:05 +00:00
Charlie Marsh
385345807c
Accept multiple input files in pip-sync and pip-compile (#140)
Closes https://github.com/astral-sh/puffin/issues/126.
2023-10-19 18:17:27 +00:00
Charlie Marsh
7ef6c0315c
Unify site-packages into distribution enum (#136)
Gets rid of the custom `DistInfo` struct in the site-packages
abstraction in favor of a new kind of distribution
(`InstalledDistribution`). No change in behavior.
2023-10-19 04:37:52 +00:00
Charlie Marsh
bd01fb490e
Remove packages when syncing (#135)
`pip-sync` will now uninstall any packages that aren't necessary.

Closes https://github.com/astral-sh/puffin/issues/128.
2023-10-19 00:14:20 -04:00
Charlie Marsh
41ece4184b
Print to stderr by default (#134) 2023-10-18 23:30:07 -04:00
Charlie Marsh
20bb4c5c61
Avoid showing resolver progress bar when no resolution is required (#133) 2023-10-19 03:23:22 +00:00
Charlie Marsh
573f5832a3
Allow uninstall to take multiple packages and files (#125)
Moves the command to `puffin pip-uninstall` for now to separate from the
managed interface, and redoes the command output.
2023-10-18 22:30:11 -04:00
Charlie Marsh
4b91ae4769
Add CLI tests for add and remove commands (#124) 2023-10-19 01:06:48 +00:00
Charlie Marsh
e15b99b911
Rename commands to pip-sync and pip-compile (#123)
To free up the rest of the interface.
2023-10-18 21:15:20 +00:00
konsti
8cc4fe0d44
Install source distribution requirements with puffin itself instead of pip (#122)
This is also a lot faster. Unfortunately it copies a lot of code from
the sync cli since the `Printer` is private.

The first commit are some refactorings i made when i thought about how i
could reuse the existing code.
2023-10-18 19:11:17 +00:00
Charlie Marsh
7bc42ca2ce
Use owo_colors instead of colored (#121)
This is what `miette` uses so seems better to avoid two coloring crates.
2023-10-18 18:57:07 +00:00
Charlie Marsh
2d14c0647e
Add a puffin remove command (#120) 2023-10-18 18:50:08 +00:00
Charlie Marsh
1fc03780f9
Use miette for puffin add diagnostics (#119)
Experiment in using `miette` for better user-facing diagnostics in the
CLI crate:

<img width="710" alt="Screen Shot 2023-10-18 at 2 11 54 PM"
src="30299da0-da65-4972-944f-cb8cc5f72a77">

For now, only the `add` command has been migrated, and all the library
crates continue to use `anyhow`.
2023-10-18 14:24:09 -04:00
konsti
fec4ee2848
Support prepare_metadata_for_build_wheel (#106)
Support calling `prepare_metadata_for_build_wheel`, which can give you
the metadata without executing the actual build if the backend supports
it.

This makes the code a lot uglier since we effectively have a state
machine:

* Setup: Either venv plus requires (PEP 517) or just a venv (setup.py)
* Get metadata (optional step): None (setup.py) or
`prepare_metadata_for_build_wheel` and saving that result
* Build: `setup.py`, `build_wheel()` or
`build_wheel(metadata_directory=metadata_directory)`, but i think i got
general flow right.

@charliermarsh This is a "barely works but unblocks building on top"
implementation, say if you want more polishing (i'll look at this again
tomorrow)
2023-10-18 14:48:30 +02:00
Charlie Marsh
4c87a1d42c
Add a puffin add command (#117)
This needs far better error handling and user-facing feedback, but it
does the basic operation (and includes discovery of the `pyproject.toml`
file, etc.).
2023-10-18 00:51:20 -04:00
Charlie Marsh
339553e228
Mark --no-cache as global (#116) 2023-10-17 23:15:36 -04:00
Charlie Marsh
89db5d79bc
Add support for lenient parsing (#115)
This PR enables us to make "fixups" to bad metadata. I copied over the
one fixup that @konstin made in `monotrail-resolve`, and added a few
common ones for `Requires-Python`.
2023-10-17 22:03:16 -04:00
Charlie Marsh
0d90256151
Store all distributions rather than compatible wheels (#114)
This PR reverts #109 which is actually a performance _regression_ since
we need to iterate over a bunch of wheels that we could otherwise
entirely ignore.
2023-10-17 17:09:31 -04:00
Charlie Marsh
5b046a8102
Use select! instead of tokio::spawn for network thread (#110) 2023-10-16 15:41:25 -04:00
Charlie Marsh
1b433fdcee
Only store compatible wheels in the resolver (#109)
Rather than constantly iterating over all files and testing their
compatibility with the current platform, just store wheels we can
actually consider in the solver cache.
2023-10-16 19:21:07 +00:00
Charlie Marsh
5f5788e866
Surface PubGrub derivation trees (#108)
I think the derivation trees could be stronger but this exposes
PubGrub's proof-like error messages.

Closes #102.
2023-10-16 14:14:36 -04:00
Charlie Marsh
bae52d5edd
Surface request stream errors in the resolver (#107)
Closes https://github.com/astral-sh/puffin/issues/105.
2023-10-16 17:26:46 +00:00
Charlie Marsh
7e8ffeb2df
Use fs-err in more crates (#100)
Closes https://github.com/astral-sh/puffin/issues/88.
2023-10-16 13:37:58 +00:00
konsti
fa2fd14587
Add basic sdist builder (#104)
This adds a basic sdist builder that has been tested with two source
distributions, one with a PEP 517 backend and one with setup.py.

It uses pip for requirements installation atm, lacks testing in all
directions, lacks checks for recursive requirements, can't pass in
already resolved versions, doesn't support prepare metadata for build to
allow resolution to continue without doing the actual (native) build,
error messages are mediocre, etc.

```console
$ RUST_LOG=puffin_build=debug puffin-build --wheels wheels downloads/tqdm-4.66.1.tar.gz
2023-10-16T12:28:35.503182Z DEBUG build_sdist{path="downloads/tqdm-4.66.1.tar.gz" base_python="/usr/bin/python3"}: puffin_build: Building downloads/tqdm-4.66.1.tar.gz
2023-10-16T12:28:35.521780Z  INFO build_sdist{path="downloads/tqdm-4.66.1.tar.gz" base_python="/usr/bin/python3"}:extract_archive: puffin_build: close time.busy=18.4ms time.idle=16.7µs
2023-10-16T12:28:35.845096Z DEBUG build_sdist{path="downloads/tqdm-4.66.1.tar.gz" base_python="/usr/bin/python3"}:resolve_and_install: puffin_build: Calling pip to install build dependencies
2023-10-16T12:28:37.668660Z  INFO build_sdist{path="downloads/tqdm-4.66.1.tar.gz" base_python="/usr/bin/python3"}:resolve_and_install: puffin_build: close time.busy=1.82s time.idle=13.2µs
2023-10-16T12:28:37.668744Z DEBUG build_sdist{path="downloads/tqdm-4.66.1.tar.gz" base_python="/usr/bin/python3"}: puffin_build: Calling `setuptools.build_meta.get_requires_for_build_wheel()`
2023-10-16T12:28:38.159205Z  INFO build_sdist{path="downloads/tqdm-4.66.1.tar.gz" base_python="/usr/bin/python3"}:run_python_script{python_interpreter="/tmp/.tmpm4cTra/venv/bin/python"}: puffin_build: close time.busy=490ms time.idle=13.0µs
2023-10-16T12:28:38.159304Z DEBUG build_sdist{path="downloads/tqdm-4.66.1.tar.gz" base_python="/usr/bin/python3"}: puffin_build: Calling `setuptools.build_meta.build_wheel()`
2023-10-16T12:28:38.501732Z  INFO build_sdist{path="downloads/tqdm-4.66.1.tar.gz" base_python="/usr/bin/python3"}:run_python_script{python_interpreter="/tmp/.tmpm4cTra/venv/bin/python"}: puffin_build: close time.busy=342ms time.idle=15.2µs
2023-10-16T12:28:38.522700Z  INFO build_sdist{path="downloads/tqdm-4.66.1.tar.gz" base_python="/usr/bin/python3"}: puffin_build: close time.busy=3.02s time.idle=16.2µs
Wheel built to /home/konsti/projects/puffin/crates/puffin-build/wheels/tqdm-4.66.1-py3-none-any.whl
2023-10-16T12:28:38.522772Z DEBUG puffin_build: Took 3020ms
$ puffin-build --wheels wheels downloads/geoextract-0.3.1.tar.gz
2023-10-16T12:28:40.884622Z DEBUG build_sdist{path="downloads/geoextract-0.3.1.tar.gz" base_python="/usr/bin/python3"}: puffin_build: Building downloads/geoextract-0.3.1.tar.gz
2023-10-16T12:28:40.887743Z  INFO build_sdist{path="downloads/geoextract-0.3.1.tar.gz" base_python="/usr/bin/python3"}:extract_archive: puffin_build: close time.busy=2.97ms time.idle=12.6µs
2023-10-16T12:28:41.469738Z  INFO build_sdist{path="downloads/geoextract-0.3.1.tar.gz" base_python="/usr/bin/python3"}: puffin_build: close time.busy=585ms time.idle=15.3µs
Wheel built to /home/konsti/projects/puffin/crates/puffin-build/wheels/geoextract-0.3.1-py3-none-any.whl
2023-10-16T12:28:41.469814Z DEBUG puffin_build: Took 585ms
```
2023-10-16 12:43:31 +00:00
konsti
cb29c89424
Better error reporting (#95)
The main change is to print the whole error chain. We can combine this
with adding `.context` to distinct phases to be able to locate crashes
without having to use a debugger.
2023-10-16 02:15:10 +00:00
Charlie Marsh
471a1d657d
Migrate resolver proof-of-concept to PubGrub (#97)
## Summary

This PR enables the proof-of-concept resolver to backtrack by way of
using the `pubgrub-rs` crate.

Rather than using PubGrub as a _framework_ (implementing the
`DependencyProvider` trait, letting PubGrub call us), I've instead
copied over PubGrub's primary solver hook (which is only ~100 lines or
so) and modified it for our purposes (e.g., made it async).

There's a lot to improve here, but it's a start that will let us
understand PubGrub's appropriateness for this problem space. A few
observations:

- In simple cases, the resolver is slower than our current (naive)
resolver. I think it's just that the pipelining isn't as efficient as in
the naive case, where we can just stream package and version fetches
concurrently without any bottlenecks.
- A lot of the code here relates to bridging PubGrub with our own
abstractions -- so we need a `PubGrubPackage`, a `PubGrubVersion`, etc.
2023-10-15 22:05:44 -04:00
konsti
de9e85978b
Fix tempdir rename (#94)
This fixes two bugs on linux:

`/tmp` and `$HOME` are technically on two different partitions on my
machine, which means that rename-as-atomic-dir-write doesn't work. The
solution is to create the temp dir in the target directory.

zip files may contain directory entries, we can't create files for them
but need to create directories. We could skip them though because iirc
they are not in the RECORD so they won't be uninstalled.
2023-10-12 18:47:38 +00:00
konsti
530edb6e39
Add output file option to compile (#93)
`pip-compile` has the same option. I need this esp. since piping doesn't
work as we write to stdout.
2023-10-12 20:42:06 +02:00