An extremely fast Python package and project manager, written in Rust.
Find a file
Charlie Marsh 62c474d880
Add support for Git dependencies (#283)
## Summary

This PR adds support for Git dependencies, like:

```
flask @ git+https://github.com/pallets/flask.git
```

Right now, they're only supported in the resolver (and not the
installer), since the installer doesn't yet support source distributions
at all.

The general approach here is based on Cargo's Git implementation.
Specifically, I adapted Cargo's
[`git`](23eb492cf9/src/cargo/sources/git/mod.rs)
module to perform the cloning, which is based on `libgit2`.

As compared to Cargo's implementation, I made the following changes:

- Removed any unnecessary code.
- Fixed any Clippy errors for our stricter ruleset.
- Removed the dependency on `curl`, in favor of `reqwest` which we use
elsewhere.
- Removed the ability to use `gix`. Cargo allows the use of `gix` as an
experimental flag, but it only supports a small subset of the
operations. When Cargo fully adopts `gix`, we should plan to do the
same.
- Removed Cargo's host key checking. We need to re-add this! I'll do it
shortly.
- Removed Cargo's progress bars. We should re-add this too, but we use
`indicatif` and Cargo had their own thing.

There are a few follow-ups to consider:

- Adding support in the installer.
- When we lock, we should write out the Git URL that includes the exact
SHA. This lets us cache in perpetuity and avoids dependencies changing
without re-locking.
- When we resolve, we should _always_ try to refresh Git dependencies.
(Right now, we skip if the wheel was already built.)

I'll work on the latter two in follow-up PRs.

Closes #202.
2023-11-02 15:14:55 +00:00
.cargo Add basic CI via GitHub Actions (#10) 2023-10-05 13:42:58 -04:00
.github/workflows Add tests for puffin sync (#161) 2023-10-22 03:25:00 +00:00
crates Add support for Git dependencies (#283) 2023-11-02 15:14:55 +00:00
scripts Implement mixed PEP 517 and setup.py build 2023-10-30 19:11:52 +01:00
vendor/pubgrub Add PubGrub's priority queue (#221) 2023-10-29 21:16:02 +00:00
workers/pypi-metadata Handle dist info casing mismatch in worker (#273) 2023-11-02 11:04:28 +00:00
.dockerignore Add docker builder (#238) 2023-11-02 12:03:56 +01:00
.gitignore Add docker builder (#238) 2023-11-02 12:03:56 +01:00
builder.dockerfile Add docker builder (#238) 2023-11-02 12:03:56 +01:00
Cargo.lock Add support for Git dependencies (#283) 2023-11-02 15:14:55 +00:00
Cargo.toml Add support for Git dependencies (#283) 2023-11-02 15:14:55 +00:00
CONTRIBUTING.md Add docker builder (#238) 2023-11-02 12:03:56 +01:00
LICENSE-APACHE Add README and LICENSE files 2023-10-05 12:45:38 -04:00
LICENSE-MIT Add README and LICENSE files 2023-10-05 12:45:38 -04:00
README.md Rename commands to pip-sync and pip-compile (#123) 2023-10-18 21:15:20 +00:00
ruff.toml Unify python interpreter abstractions (#178) 2023-10-25 20:11:36 +00:00
rust-toolchain.toml Rust 1.73 2023-10-23 13:52:57 +02:00

puffin

An experimental Python packaging tool.

Motivation

Puffin is an extremely fast (experimental) Python package resolver and installer, intended to replace pip and pip-tools (pip-compile and pip-sync).

Puffin itself is not a complete "package manager", but rather a tool for locking dependencies (similar to pip-compile) and installing them (similar to pip-sync). Puffin can be used to generate a set of locked dependencies from a requirements.txt file, and then install those locked dependencies into a virtual environment.

Puffin represents an intermediary goal in our pursuit of building a "Cargo for Python": a Python package manager that is extremely fast, reliable, and easy to use -- capable of replacing not only pip, but also pipx, pip-tools, virtualenv, tox, setuptools, and even pyenv, by way of managing the Python installation itself.

Puffin's limited scope allows us to solve many of the low-level problems that are required to build such a package manager (like package installation) while shipping an immediately useful tool with a minimal barrier to adoption. Try it today in lieu of pip and pip-tools.

Features

  • Extremely fast dependency resolution and installation: install dependencies in sub-second time.
  • Disk-space efficient: Puffin uses a global cache to deduplicate dependencies, and uses Copy-on-Write on supported filesystems to reduce disk usage.

Limitations

Puffin does not yet support:

  • Source distributions
  • VCS dependencies
  • URL dependencies
  • Windows
  • ...

Like pip-compile, Puffin generates a platform-specific requirements.txt file (unlike, e.g., poetry, which generates a platform-agnostic poetry.lock file). As such, Puffin's requirements.txt files are not portable across platforms and Python versions.

Usage

To resolve a requirements.in file:

cargo run -p puffin-cli -- pip-compile requirements.in

To install from a resolved requirements.txt file:

cargo run -p puffin-cli -- pip-sync requirements.txt

For more, see cargo run -p puffin-cli -- --help:

Usage: puffin-cli <COMMAND>

Commands:
  compile  Compile a `requirements.in` file to a `requirements.txt` file
  sync     Sync dependencies from a `requirements.txt` file
  clean    Clear the cache
  freeze   Enumerate the installed packages in the current environment
  help     Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

License

Puffin is licensed under either of

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Puffin by you, as defined in the Apache-2.0 license, shall be dually licensed as above, without any additional terms or conditions.