598 commits

Author	SHA1	Message	Date
konsti	76a41066ac	Filter out incompatible dists (#398) ... Filter out source dists and wheels whose `requires-python` from the simple api is incompatible with the current python version. This change showed an important problem: When we use a fake python version for resolving, building source distributions breaks down because we can only build with versions we actually have. This change became surprisingly big. The tests now require python 3.7 to be installed, but changing that would mean an even bigger change. Fixes #388	2023-11-13 17:14:07 +01:00
konsti	81c9cd0d4a	Print url for bad json error (#409) ... Split out from #382	2023-11-13 11:41:20 +00:00
konsti	fa423b8751	Backend path is supported (#405) ... The check is outdated now	2023-11-13 08:19:20 +00:00
Zanie Blue	beadd3274a	Improve debug log version display (#403) ... Follow-up to https://github.com/astral-sh/puffin/pull/346 for some debug messages	2023-11-10 17:07:29 -06:00
Charlie Marsh	06b312de7e	Overwrite existing files when hardlinking (#402) ... ## Summary Closes https://github.com/astral-sh/puffin/issues/390. ## Test Plan Installed `jupyter_core==5.5.0`, then removed the `jupyter_core` and `jupyter_core-5.5.0.dist-info` directories from my virtualenv manually, but left `jupyter.py`. I then re-ran `puffin pip-compile`, and verified that it errored on `main` but succeeded here.	2023-11-10 20:24:19 +00:00
Charlie Marsh	56a4b51eb6	Refactor hardlink fallback to use an enum (#401) ... Makes an invalid state unrepresentable (`first_try_hard_linking = true`, `use_copy_fallback` = true`).	2023-11-10 15:18:51 -05:00
Andrew Gallant	ff4d079dc9	pep508-rs: remove \x20 trailing whitespace hack (#400) ... ... we just remove the trailing whitespace from the input and that resolves things. Thanks @konstin for pointing this out! Ref https://github.com/astral-sh/puffin/pull/399#discussion_r1389854823	2023-11-10 15:11:29 -05:00
Charlie Marsh	e8108cb28b	Remove __pycache__ directories when uninstalling (#397) ... According to the [packaging documentation](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#binary-distribution-format), "uninstallers should be smart enough to remove .pyc even if it is not mentioned in RECORD". Previously, we weren't handling this case, so if you installed via Puffin, then imported a file (to trigger bytecode compilation), then uninstalled, we'd leave spare `__pycache__` directories around. Closes https://github.com/astral-sh/puffin/issues/395.	2023-11-10 14:55:33 -05:00
Andrew Gallant	63f7f65190	change global allocator to jemalloc (and mimalloc on Windows) (#399) ... This copies the allocator configuration used in the Ruff project. In particular, this gives us an instant 10% win when resolving the top 1K PyPI packages: $ hyperfine \ "./target/profiling/puffin-dev-main resolve-many --cache-dir cache-docker-no-build --no-build pypi_top_8k_flat.txt --limit 1000 2> /dev/null" \ "./target/profiling/puffin-dev resolve-many --cache-dir cache-docker-no-build --no-build pypi_top_8k_flat.txt --limit 1000 2> /dev/null" Benchmark 1: ./target/profiling/puffin-dev-main resolve-many --cache-dir cache-docker-no-build --no-build pypi_top_8k_flat.txt --limit 1000 2> /dev/null Time (mean ± σ): 974.2 ms ± 26.4 ms [User: 17503.3 ms, System: 2205.3 ms] Range (min … max): 943.5 ms … 1015.9 ms 10 runs Benchmark 2: ./target/profiling/puffin-dev resolve-many --cache-dir cache-docker-no-build --no-build pypi_top_8k_flat.txt --limit 1000 2> /dev/null Time (mean ± σ): 883.1 ms ± 23.3 ms [User: 14626.1 ms, System: 2542.2 ms] Range (min … max): 849.5 ms … 916.9 ms 10 runs Summary './target/profiling/puffin-dev resolve-many --cache-dir cache-docker-no-build --no-build pypi_top_8k_flat.txt --limit 1000 2> /dev/null' ran 1.10 ± 0.04 times faster than './target/profiling/puffin-dev-main resolve-many --cache-dir cache-docker-no-build --no-build pypi_top_8k_flat.txt --limit 1000 2> /dev/null' I was moved to do this because I noticed `malloc`/`free` taking up a fairly sizeable percentage of time during light profiling. As is becoming a pattern, it will be easier to review this commit-by-commit. Ref #396 (wouldn't call this issue fixed) ----- I did also try adding a `smallvec` optimization to the `Version::release` field, but it didn't bare any fruit. I still think there is more to explore since the results I observed don't quite line up with what I expect. (So probably either my mental model is off or my measurement process is flawed.) You can see that attempt with a little more explanation here: f9528b4ecd In the course of adding the `smallvec` optimization, I also shrunk the `Version` fields from a `usize` to a `u32`. They should at least be a fixed size integer since version numbers aren't used to index memory, and I shrunk it to `u32` since it seems reasonable to assume that all version numbers will be smaller than `2^32`.	2023-11-10 14:48:59 -05:00
konsti	d8408b1783	Add source to failing metadata parsing (#387) ... Before: ``` cargo run --bin puffin-dev -q -- resolve-cli "transformers[accelerate, agents, all, audio, codecarbon, deepspeed, deepspeed-testing, dev, dev-tensorflow, dev-torch, docs, docs_specific, flax, flax-speech, ftfy, integrations, ja, modelcreation, onnx, onnxruntime, optuna, quality, ray, retrieval, sagemaker, sentencepiece, serving, sigopt, sklearn, speech, testing, tf, tf-cpu, tf-speech, timm, tokenizers, torch, torch-speech, torch-vision, torchhub, video, vision]" puffin-dev failed Caused by: No solution found when resolving: transformers[accelerate,agents,all,audio,codecarbon,deepspeed,deepspeed-testing,dev,dev-tensorflow,dev-torch,docs,docs-specific,flax,flax-speech,ftfy,integrations,ja,modelcreation,onnx,onnxruntime,optuna,quality,ray,retrieval,sagemaker,sentencepiece,serving,sigopt,sklearn,speech,testing,tf,tf-cpu,tf-speech,timm,tokenizers,torch,torch-speech,torch-vision,torchhub,video,vision] Caused by: Not a valid package or extra name: ".none". Names must start and end with a letter or digit and may only contain -, _, ., and alphanumeric characters ``` After: ``` cargo run --bin puffin-dev -q -- resolve-cli "transformers[accelerate, agents, all, audio, codecarbon, deepspeed, deepspeed-testing, dev, dev-tensorflow, dev-torch, docs, docs_specific, flax, flax-speech, ftfy, integrations, ja, modelcreation, onnx, onnxruntime, optuna, quality, ray, retrieval, sagemaker, sentencepiece, serving, sigopt, sklearn, speech, testing, tf, tf-cpu, tf-speech, timm, tokenizers, torch, torch-speech, torch-vision, torchhub, video, vision]" puffin-dev failed Caused by: No solution found when resolving: transformers[accelerate,agents,all,audio,codecarbon,deepspeed,deepspeed-testing,dev,dev-tensorflow,dev-torch,docs,docs-specific,flax,flax-speech,ftfy,integrations,ja,modelcreation,onnx,onnxruntime,optuna,quality,ray,retrieval,sagemaker,sentencepiece,serving,sigopt,sklearn,speech,testing,tf,tf-cpu,tf-speech,timm,tokenizers,torch,torch-speech,torch-vision,torchhub,video,vision] Caused by: Couldn't parse metadata in fastapi-0.10.1-py3-none-any.whl (97ac91cb7c/fastapi-0.10.1-py3-none-any.whl) Caused by: Not a valid package or extra name: ".none". Names must start and end with a letter or digit and may only contain -, _, ., and alphanumeric characters ```	2023-11-10 18:33:49 +00:00
Charlie Marsh	b3edf7c2b2	Delete any directories listed in the RECORD file (#394) ... ## Summary It looks like, when you install `pip`, it includes a bunch of `__pycache__` directories in the RECORD file (although these directories don't exist until you run `pip`). Our uninstaller assumed that the RECORD file only contained _files_. Closes https://github.com/astral-sh/puffin/issues/389.	2023-11-10 18:17:52 +00:00
Charlie Marsh	6a15950cb5	Rename Distribution to Dist in all structs and traits (#384) ... We tend to avoid abbreviations, but this one is just so long and absolutely ubiquitous.	2023-11-10 14:55:11 +00:00
konsti	5cef40d87a	Add proper caching for pypi metadata fetching kinds (#368) ... I intend this to become the main form of caching for puffin: You can make http requests, you tranform the data to what you really need, you have control over the cache key, and the cache is always json (or anything else much faster we want to replace it with as long as it's serde!)	2023-11-10 11:03:40 +00:00
konsti	d1b57acaa8	Implement PEP 517 backend-path (#385) ... Closes #192	2023-11-10 11:54:23 +01:00
Charlie Marsh	a148f9d0be	Refactor distribution types to adhere to a clear hierarchy (#369) ... ## Summary This PR refactors our `RemoteDistribution` type such that it now follows a clear hierarchy that matches the actual variants, and encodes the differences between source and built distributions: ```rust pub enum Distribution { Built(BuiltDistribution), Source(SourceDistribution), } pub enum BuiltDistribution { Registry(RegistryBuiltDistribution), DirectUrl(DirectUrlBuiltDistribution), } pub enum SourceDistribution { Registry(RegistrySourceDistribution), DirectUrl(DirectUrlSourceDistribution), Git(GitSourceDistribution), } /// A built distribution (wheel) that exists in a registry, like `PyPI`. pub struct RegistryBuiltDistribution { pub name: PackageName, pub version: Version, pub file: File, } /// A built distribution (wheel) that exists at an arbitrary URL. pub struct DirectUrlBuiltDistribution { pub name: PackageName, pub url: Url, } /// A source distribution that exists in a registry, like `PyPI`. pub struct RegistrySourceDistribution { pub name: PackageName, pub version: Version, pub file: File, } /// A source distribution that exists at an arbitrary URL. pub struct DirectUrlSourceDistribution { pub name: PackageName, pub url: Url, } /// A source distribution that exists in a Git repository. pub struct GitSourceDistribution { pub name: PackageName, pub url: Url, } ``` Most of the PR just stems downstream from this change. There are no behavioral changes, so I'm largely relying on lint, tests, and the compiler for correctness.	2023-11-10 02:45:41 +00:00
Andrew Gallant	33c0901a28	distribution-filename: speed up is_compatible (#367) ... This PR tweaks the representation of `Tags` in order to offer a faster implementation of `WheelFilename::is_compatible`. We now use a nested map of tags that lets us avoid looping over every supported platform tag. As the code comments suggest, that is the essential gain. We still do not mind looping over the tags in each wheel name since they tend to be quite small. And pushing our thumb on that side of things can make things worse overall since it would likely slow down WheelFilename construction itself. For micro-benchmarks, we improve considerably for compatibility checking: $ critcmp base test3 group base test3 ----- ---- ----- build_platform_tags/burntsushi-archlinux 1.00 46.2±0.28µs ? ?/sec 2.48 114.8±0.45µs ? ?/sec wheelname_parsing/flyte-long-compatible 1.00 624.8±3.31ns 174.0 MB/sec 1.01 629.4±4.30ns 172.7 MB/sec wheelname_parsing/flyte-long-incompatible 1.00 743.6±4.23ns 165.4 MB/sec 1.00 746.9±4.62ns 164.7 MB/sec wheelname_parsing/flyte-short-compatible 1.00 526.7±4.76ns 54.3 MB/sec 1.01 530.2±5.81ns 54.0 MB/sec wheelname_parsing/flyte-short-incompatible 1.00 540.4±4.93ns 60.0 MB/sec 1.01 545.7±5.31ns 59.4 MB/sec wheelname_parsing_failure/flyte-long-extension 1.00 13.6±0.13ns 3.2 GB/sec 1.01 13.7±0.14ns 3.2 GB/sec wheelname_parsing_failure/flyte-short-extension 1.00 14.0±0.20ns 1160.4 MB/sec 1.01 14.1±0.14ns 1146.5 MB/sec wheelname_tag_compatibility/flyte-long-compatible 11.33 159.8±2.79ns 680.5 MB/sec 1.00 14.1±0.23ns 7.5 GB/sec wheelname_tag_compatibility/flyte-long-incompatible 237.60 1671.8±37.99ns 73.6 MB/sec 1.00 7.0±0.08ns 17.1 GB/sec wheelname_tag_compatibility/flyte-short-compatible 16.07 223.5±8.60ns 128.0 MB/sec 1.00 13.9±0.30ns 2.0 GB/sec wheelname_tag_compatibility/flyte-short-incompatible 149.83 628.3±2.13ns 51.6 MB/sec 1.00 4.2±0.10ns 7.6 GB/sec We do regress slightly on the time it takes for `Tags::new` to run, but this is somewhat expected. And in absolute terms, 114us is perfectly acceptable given that it's only executed ~once for each `puffin` invocation. Ad hoc benchmarks indicate an overall 25% perf improvement in `puffin pip-compile` times. This roughly corresponds with how much time `is_compatible` was taking. Indeed, profiling confirms that it has virtually disappeared from the profile. Fixes #157	2023-11-09 09:01:03 -05:00
konsti	bdb89b4072	Allow setting num tasks in puffin-dev parallel resolve (#374)	2023-11-09 13:12:03 +00:00
Charlie Marsh	6144de0a7e	Implement some minor optimizations to version match (#371) ... `Range::intersection` goes from 74.2% to 64.9%, and `sortable_tuple` goes from 2.3% to 1.5%.	2023-11-09 02:11:40 +00:00
Charlie Marsh	cfd84d6365	Support resolving for an alternate Python distribution (#364) ... ## Summary Low-priority but fun thing to end the day. You can now pass `--target-version py37`, and we'll generate a resolution for Python 3.7. See: https://github.com/astral-sh/puffin/issues/183.	2023-11-08 23:19:16 +00:00
konsti	d407bbbee6	Special case missing header build errors (on linux) (#354) ... One of the most common errors i observed are build failures due to missing header files. On ubuntu, this generally means that you need to install some `<...>-dev` package that the documentation tells you about, e.g. [mysqlclient](https://github.com/PyMySQL/mysqlclient#linux) needs `default-libmysqlclient-dev`, [some psycopg versions](https://www.psycopg.org/psycopg3/docs/basic/install.html#local-installation) (i remember that this was always required at some earlier point) require `libpq-dev` and pygraphviz wants `graphviz-dev`. This is quite common for many scientific packages (where conda has an advantage because they can provide those package as a dependency). The error message can be completely inscrutable if you're just a python programmer (or user) and not a c programmer (example: pygraphviz): ``` warning: no files found matching '*.png' under directory 'doc' warning: no files found matching '*.txt' under directory 'doc' warning: no files found matching '*.css' under directory 'doc' warning: no previously-included files matching '*~' found anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '.svn' found anywhere in distribution no previously-included directories found matching 'doc/build' pygraphviz/graphviz_wrap.c:3020:10: fatal error: graphviz/cgraph.h: No such file or directory 3020 | #include "graphviz/cgraph.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. error: command '/usr/bin/gcc' failed with exit code 1 ``` The only relevant part is `Fatal error: graphviz/cgraph.h: No such file or directory`. Why is this file not there and how do i get it to be there? This is even harder to spot in pip's output, where it's 11 lines above the last line:  I've special cased missing headers and made sure that the last line tells you the important information: We're missing some header, please check the documentation of {package} {version} for what to install:  Scrolling up:  The difference gets even clearer with a default ubuntu terminal with its 80 columns:  --- Note that the situation is better for a missing compiler, there i get: ``` [...] warning: no previously-included files matching '*~' found anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '.svn' found anywhere in distribution no previously-included directories found matching 'doc/build' error: command 'gcc' failed: No such file or directory --- ``` Putting the last line into google, the first two results tell me to `sudo apt-get install gcc`, the third even tells me about `sudo apt install build-essential`	2023-11-08 15:26:39 +00:00
konsti	2ebe40b986	Add --no-build (#358) ... By default, we will build source distributions for both resolving and installing, running arbitrary code. `--no-build` adds an option to ban this and only install from wheels, no source distributions or git builds allowed. We also don't fetch these and instead report immediately. I've heard from users for whom this is a requirement, i'm implementing it now because it's helpful for testing. I'm thinking about adding a shared `PuffinSharedArgs` struct so we don't have to repeat each option everywhere.	2023-11-08 10:05:15 -05:00
Charlie Marsh	4fe583257e	Use a custom PubGrub error type to always show resolution report (#365) ... Closes https://github.com/astral-sh/puffin/issues/356. The example from the issue now renders as: ``` ❯ cargo run --bin puffin-dev -q -- resolve-cli tensorflow-cpu-aws puffin-dev failed Caused by: No solution found when resolving build dependencies for source distribution: Caused by: Because there is no available version for tensorflow-cpu-aws and root depends on tensorflow-cpu-aws, version solving failed. ```	2023-11-08 09:57:26 -05:00
Charlie Marsh	3c24301193	Avoid removing progress bars (#362) ... This was dumb of me. We pass out indexes when adding progress bars, but were then removing entries on completion, so any outstanding indexes were now _invalid_. We just shouldn't remove them. The `MultiProgress` retains a reference anyway, IIUC. Closes https://github.com/astral-sh/puffin/issues/360.	2023-11-07 18:58:17 +00:00
Charlie Marsh	7abe141d3f	Add SSL to possible spurious errors (#361) ... \cc @konstin	2023-11-07 18:53:39 +00:00
Andrew Gallant	294955ecff	fix platform detection on Linux (#359) ... Rejigger Linux platform detection This change makes some very small improvements to the Linux platform detection logic. In particular, the existing logic did not work on my Archlinux machine since /lib64/ld-linux-x86-64.so.2 isn't a symlink. In that case, the detection logic should have fallen back to the slower `ldd --version` technique, but `read_link` fails outright when its argument isn't a symbolic link. So we tweak the logic to allow it to fail, and if it does, we still try the `ldd --version` approach instead of giving up completely. I also made some cosmetic improvements to the regex matching, as well as ensuring that the regexes are only compiled exactly once.	2023-11-07 11:39:35 -05:00
konsti	692d2eb26f	puffin-dev resolve many improvements (#357) ... Print the current step, the time for and also respect the cache dir arg.	2023-11-07 14:56:35 +00:00
Charlie Marsh	b0286a8939	Add user feedback when building source distributions in the resolver (#347) ... It looks like Cargo, notice the bold green lines at the top (which appear during the resolution, to indicate Git fetches and source distribution builds): <img width="868" alt="Screen Shot 2023-11-06 at 11 28 47 PM" src="9647a480-7be7-41e9-b1d3-69faefd054ae"> <img width="868" alt="Screen Shot 2023-11-06 at 11 28 51 PM" src="6bc491aa-5b51-4b37-9ee1-257f1bc1c049"> Closes https://github.com/astral-sh/puffin/issues/287 although we can do a lot more here.	2023-11-07 14:17:31 +00:00
Charlie Marsh	2c32bc5a86	Respect direct URLs in puffin installer (#345) ... We now write the `direct_url.json` when installing, and _skip_ installing if we find a package installed via the direct URL that the user is requesting. A lot of TODOs, especially around cleaning up the `Source` abstraction and its relationship to `DirectUrl`. I'm gonna keep working on these today, but this works and makes the requirements clear. Closes #332.	2023-11-07 09:11:27 -05:00
konsti	c11586f2f0	Fix index out of bounds in SourceDistributionFilename::parse (#353) ... Found this one in the top 8k pypi tests too	2023-11-07 11:44:40 +00:00
konsti	c883b123ac	Allow greater than star (torch (>=1.9.*)) in lenient requirement (#351) ... This appeared in the pypi top 8k testing.	2023-11-07 11:37:23 +00:00
konsti	fbe28d3b7c	Fix mastodon-py dist-info handling (#336) ... mastodon-py 1.5.1 uses a dot in its dist-info dir name, which we previously didn't handle, causing home-assistant to fail. The new implementation is based on 2f83540272/src/packaging/utils.py (L146-L172). Part of #199 ``` unzip -l Mastodon.py-1.5.1-py2.py3-none-any.whl Archive: Mastodon.py-1.5.1-py2.py3-none-any.whl Length Date Time Name --------- ---------- ----- ---- 153929 2020-02-29 17:39 mastodon/Mastodon.py 1029 2019-10-11 19:15 mastodon/__init__.py 7357 2019-10-11 20:24 mastodon/streaming.py 10 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/DESCRIPTION.rst 1398 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/metadata.json 9 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/top_level.txt 110 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/WHEEL 1543 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/METADATA 753 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/RECORD --------- ------- 166138 9 files ```	2023-11-07 12:36:11 +01:00
konsti	aac8ae997f	Rename source distribution build to source build (#334) ... This is less verbose and better reflects that we're building both source distributions and source trees passed into the function.	2023-11-07 03:55:23 +00:00
Charlie Marsh	620afc3caf	Avoid refreshing Git repo twice (#350) ... This was a bug in the Git code (that I wrote, not from Cargo) -- when we `precise` the reference, we should store the resolved commit.	2023-11-07 02:52:15 +00:00
Charlie Marsh	243549876c	Upgrade PubGrub (#349) ... Upgrades to `fe309ffb63b2f3ce9b35eb7746b2350cd704515e`, with our changes layered on top.	2023-11-07 02:00:57 +00:00
Charlie Marsh	2c114592bd	Only store small wheels in-memory (#348) ... Closes https://github.com/astral-sh/puffin/issues/246.	2023-11-07 00:50:00 +00:00
Zanie Blue	e952557bf1	Improve root message when version solving fails (#344) ... Matching description at https://github.com/dart-lang/pub/blob/master/doc/solver.md#linear-error-reporting	2023-11-06 20:07:50 +00:00
Zanie Blue	b0720ea5b2	Improve error message for dependencies with no versions available (#342) ... Partially addresses https://github.com/astral-sh/puffin/issues/310 Addresses case at https://github.com/astral-sh/puffin/issues/309#issuecomment-1793541558 Follow-up to #300 ensuring `PuffinExternal` is used consistently when formatting messages Example at https://github.com/astral-sh/puffin/pull/342/files#diff-5c74a74ef34ef1d6e7453de8d2d19134813156e8b6a657e6b5ed71fda5a3a870	2023-11-06 14:04:29 -06:00
Zanie Blue	1748cfb522	Display dependency versions in pip-like format during solve failure (#346) ... - Display `==` for exact version ranges - Remove space between dependency and version range	2023-11-06 13:53:15 -06:00
Charlie Marsh	a5e535f6fb	Remove virtualenv setup from gourgeist (#339) ... We now only support building bare environments.	2023-11-06 18:32:45 +00:00
Charlie Marsh	b013ea9c93	Move DirectUrl into pypi-types (#343) ... This needs to be reused elsewhere, and there's nothing specific to wheel installation about it.	2023-11-06 18:26:33 +00:00
Charlie Marsh	24e30e6557	Split puffin-package into requirements.txt parser and pypi-types (#341) ... There are only two things left in this crate and they don't really have anything to do with one another.	2023-11-06 18:19:49 +00:00
Charlie Marsh	1f447892f3	Rename PartitionedRequirements to InstallPlan (#340) ... @konstin named this file at some point and I like it, it feels appropriate for the struct itself too.	2023-11-06 12:44:35 -05:00
Charlie Marsh	d9bcfafa16	Write direct_url.json in wheel installer (#337) ... ## Summary This PR just adds the logic in `install-wheel-rs` to write `direct_url.json`. We're not actually taking advantage of it yet (or wiring it through) in Puffin. Part of https://github.com/astral-sh/puffin/issues/332.	2023-11-06 17:09:28 +00:00
konsti	9b077f3d0f	cargo upgrade --incompatible (#330) ... Ran `cargo upgrade --incompatible`, seems there are no changes required. From cacache 0.12.0: > BREAKING CHANGE: some signatures for copy have changed, and copy no longer automatically reflinks `which` 5.0.0 seems to have only error message changes.	2023-11-06 14:14:47 +00:00
konsti	d99ca3159b	Cache the setup.py resolution (#327) ... Cache the resolution for the setup.py requirements (`pip`, `setuptools`, `wheels`) across builds.	2023-11-06 14:14:24 +00:00
konsti	b2439b24a1	Fetch wheel metadata by async range requests on the remote wheel (#301) ... Use range requests and async zip to extract the METADATA file from a remote wheel. We currently only cache when the remote says the remote declares the resource as immutable, see https://github.com/06chaynes/http-cache/issues/57 and https://github.com/baszalmstra/async_http_range_reader/pull/1 . The cache is stored as json with the description omitted, this improve cache deserialization performance.	2023-11-06 15:06:49 +01:00
konsti	6f83a44fea	Improve error messages and make cache failures non fatal (#333)	2023-11-06 15:06:27 +01:00
konsti	3defe233e6	Use dist info name in cache again (#331) ... Fixup for the `PackageName`/`DistInfoName` refactor that would lead to invalid cache entries	2023-11-06 13:47:38 +00:00
Charlie Marsh	6d672b8951	Add source distribution support to pip-compile (#323) ... ## Summary This is a first-pass at adding source distribution support to the installer. The previous installation flow was: 1. Come up with a plan. 1. Find a distribution (specific file) for every package that we'll need to download. 1. Download those distributions. 1. Unzip them (since we assumed they were all wheels). 1. Install them into the virtual environment. Now, Step (3) downloads both wheels and source distributions, and we insert a step between Steps (3) and (4) to build any source distributions into zipped wheels. There are a bunch of TODOs, the most important (IMO) is that we basically have two implementations of downloading and building, between the stuff in `puffin_installer` and `puffin_resolver` (namely in `crates/puffin-resolver/src/distribution`). I didn't attempt to clean that up here -- it's already a problem, and it's related to the overall problem we need to solve around unified caching and resource management. Closes #243.	2023-11-06 08:22:36 -05:00
konsti	b79a15b458	Update pyproject-toml to 0.8.0 (#329)	2023-11-06 13:16:36 +00:00