2a846e76b7
This PR massively speeds up the case in which you need to install wheels that already exist in the global cache. The new strategy is as follows: - Download the wheel into the content-addressed cache. - Unzip the wheel into the cache, but ignore content-addressing. It turns out that writing to `cacache` for every file in the zip added a ton of overhead, and I don't see any actual advantages to doing so. Instead, we just unzip the contents into a directory at, e.g., `~/.cache/puffin/django-4.1.5`. - (The unzip itself is now parallelized with Rayon.) - When installing the wheel, we now support unzipping from a directory instead of a zip archive. This required duplicating and tweaking a few functions. - When installing the wheel, we now use reflinks (or copy-on-write links). These have a few fantastic properties: (1) they're extremely cheap to create (on macOS, they are allegedly faster than hard links); (2) they minimize disk space, since we avoid copying files entirely in the vast majority of cases; and (3) if the user then edits a file locally, the cache doesn't get polluted. Orogene, Bun, and soon pnpm all use reflinks. Puffin is now ~15x faster than `pip` for the common case of installing cached data into a fresh environment. Closes https://github.com/astral-sh/puffin/issues/21. Closes https://github.com/astral-sh/puffin/issues/39.
4.3 KiB
puffin
An experimental Python packaging tool.
Motivation
Puffin is an extremely fast (experimental) Python package resolver and installer, intended to
replace pip and pip-tools (pip-compile and pip-sync).
Puffin itself is not a complete "package manager", but rather a tool for locking dependencies
(similar to pip-compile) and installing them (similar to pip-sync). Puffin can be used to
generate a set of locked dependencies from a requirements.txt file, and then install those
locked dependencies into a virtual environment.
Puffin represents an intermediary goal in our pursuit of building a "Cargo for Python": a Python
package manager that is extremely fast, reliable, and easy to use -- capable of replacing not only
pip, but also pipx, pip-tools, virtualenv, tox, setuptools, and even pyenv, by way of
managing the Python installation itself.
Puffin's limited scope allows us to solve many of the low-level problems that are required to
build such a package manager (like package installation) while shipping an immediately useful tool
with a minimal barrier to adoption. Try it today in lieu of pip and pip-tools.
Limitations
Puffin does not yet support:
- Source distributions
- VCS dependencies
- URL dependencies
- Uninstalling packages
- ...
Like pip-compile, Puffin generates a platform-specific requirements.txt file (unlike, e.g.,
poetry, which generates a platform-agnostic poetry.lock file). As such, Puffin's
requirements.txt files are not portable across platforms and Python versions.
Usage
To resolve a requirements.in file:
cargo run -p puffin-cli -- compile requirements.in
To install from a resolved requirements.txt file:
cargo run -p puffin-cli -- sync requirements.txt
For more, see cargo run -p puffin-cli -- --help:
Usage: puffin-cli <COMMAND>
Commands:
compile Compile a `requirements.in` file to a `requirements.txt` file
sync Sync dependencies from a `requirements.txt` file
clean Clear the cache
freeze Enumerate the installed packages in the current environment
help Print this message or the help of the given subcommand(s)
Options:
-h, --help Print help
-V, --version Print version
Benchmarks
Resolution
To compare with a cold cache:
hyperfine --runs 10 --warmup 3 --prepare "rm -f /tmp/tmp.txt" \
"./target/release/puffin-cli compile requirements.txt --no-cache" \
"pip-compile requirements.txt --rebuild --pip-args '--no-cache-dir' -o /tmp/tmp.txt"
To compare with a warm cache:
hyperfine --runs 10 --warmup 3 --prepare "rm -f /tmp/tmp.txt" \
"./target/release/puffin-cli compile requirements.txt" \
"pip-compile requirements.txt -o /tmp/tmp.txt"
Installation
To compare with a cold cache:
hyperfine --runs 10 --warmup 3 --prepare "rm -rf .venv && virtualenv .venv && source .venv/bin/activate" \
"./target/release/puffin-cli sync requirements.txt --ignore-installed --no-cache" \
"pip install -r requirements.txt --ignore-installed --no-cache-dir --no-deps"
To compare with a warm cache:
hyperfine --runs 10 --warmup 3 --prepare "rm -rf .venv && virtualenv .venv && source .venv/bin/activate" \
"./target/release/puffin-cli sync requirements.txt --ignore-installed" \
"pip install -r requirements.txt --ignore-installed --no-deps"
To compare with all dependencies already installed:
hyperfine --runs 10 --warmup 3 --prepare "rm -rf .venv && virtualenv .venv && source .venv/bin/activate" \
"./target/release/puffin-cli sync requirements.txt" \
"pip install -r requirements.txt --no-deps"
License
Puffin is licensed under either of
- Apache License, Version 2.0, (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Puffin by you, as defined in the Apache-2.0 license, shall be dually licensed as above, without any additional terms or conditions.
