47fc90d1b3
This is https://github.com/astral-sh/puffin/pull/947 again but this time merging into main instead of downstack, sorry for the noise. --- Windows has a default stack size of 1MB, which makes puffin often fail with stack overflows. The PR reduces stack size by three changes: * Boxing `File` in `Dist`, reducing the size from 496 to 240. * Boxing the largest futures. * Boxing `CachePolicy` ## Method Debugging happened on linux using https://github.com/astral-sh/puffin/pull/941 to limit the stack size to 1MB. Used ran the command below. ``` RUSTFLAGS=-Zprint-type-sizes cargo +nightly build -p puffin-cli -j 1 > type-sizes.txt && top-type-sizes -w -s -h 10 < type-sizes.txt > sizes.txt ``` The main drawback is top-type-sizes not saying what the `__awaitee` is, so it requires manually looking up with a future with matching size. When the `brotli` features on `reqwest` is active, a lot of brotli types show up. Toggling this feature however seems to have no effect. I assume they are false positives since the `brotli` crate has elaborate control about allocation. The sizes are therefore shown with the feature off. ## Results The largest future goes from 12208B to 6416B, the largest type (`PrioritizedDistribution`, see also #948) from 17448B to 9264B. Full diff: https://gist.github.com/konstin/62635c0d12110a616a1b2bfcde21304f For the second commit, i iteratively boxed the largest file until the tests passed, then with an 800KB stack limit looked through the backtrace of a failing test and added some more boxing. Quick benchmarking showed no difference: ```console $ hyperfine --warmup 2 "target/profiling/main-dev resolve meine_stadt_transparent" "target/profiling/puffin-dev resolve meine_stadt_transparent" Benchmark 1: target/profiling/main-dev resolve meine_stadt_transparent Time (mean ± σ): 49.2 ms ± 3.0 ms [User: 39.8 ms, System: 24.0 ms] Range (min … max): 46.6 ms … 63.0 ms 55 runs Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options. Benchmark 2: target/profiling/puffin-dev resolve meine_stadt_transparent Time (mean ± σ): 47.4 ms ± 3.2 ms [User: 41.3 ms, System: 20.6 ms] Range (min … max): 44.6 ms … 60.5 ms 62 runs Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options. Summary target/profiling/puffin-dev resolve meine_stadt_transparent ran 1.04 ± 0.09 times faster than target/profiling/main-dev resolve meine_stadt_transparent ``` |
||
|---|---|---|
| .. | ||
| bench | ||
| cache-key | ||
| distribution-filename | ||
| distribution-types | ||
| gourgeist | ||
| install-wheel-rs | ||
| once-map | ||
| pep440-rs | ||
| pep508-rs | ||
| platform-host | ||
| platform-tags | ||
| puffin | ||
| puffin-build | ||
| puffin-cache | ||
| puffin-client | ||
| puffin-dev | ||
| puffin-dispatch | ||
| puffin-distribution | ||
| puffin-extract | ||
| puffin-fs | ||
| puffin-git | ||
| puffin-installer | ||
| puffin-interpreter | ||
| puffin-normalize | ||
| puffin-resolver | ||
| puffin-traits | ||
| puffin-warnings | ||
| puffin-workspace | ||
| pypi-types | ||
| requirements-txt | ||
| README.md | ||
Crates
bench
Functionality for benchmarking Puffin.
cache-key
Generic functionality for caching paths, URLs, and other resources across platforms.
distribution-filename
Parse built distribution (wheel) and source distribution (sdist) filenames to extract structured metadata.
distribution-types
Abstractions for representing built distributions (wheels) and source distributions (sdists), and the sources from which they can be downloaded.
gourgeist
A venv replacement to create virtual environments in Rust.
install-wheel-rs
Install built distributions (wheels) into a virtual environment.]
once-map
A waitmap-like concurrent hash map for executing tasks
exactly once.
pep440-rs
Utilities for interacting with Python version numbers and specifiers.
pep508-rs
Utilities for interacting with PEP 508 dependency specifiers.
platform-host
Functionality for detecting the current platform (operating system, architecture, etc.).
platform-tags
Functionality for parsing and inferring Python platform tags as per PEP 425.
puffin
Command-line interface for the Puffin package manager.
puffin-build
A PEP 517-compatible build frontend for Puffin.
puffin-cache
Functionality for caching Python packages and associated metadata.
puffin-client
Client for interacting with PyPI-compatible HTTP APIs.
puffin-dev
Development utilities for Puffin.
puffin-dispatch
A centralized struct for resolving and building source distributions in isolated environments.
Implements the traits defined in puffin-traits.
puffin-distribution
Client for interacting with built distributions (wheels) and source distributions (sdists). Capable of fetching metadata, distribution contents, etc.
puffin-extract
Utilities for extracting files from archives.
puffin-fs
Utilities for interacting with the filesystem.
puffin-git
Functionality for interacting with Git repositories.
puffin-installer
Functionality for installing Python packages into a virtual environment.
puffin-interpreter
Functionality for detecting and leveraging the current Python interpreter.
puffin-normalize
Normalize package and extra names as per Python specifications.
puffin-package
Types and functionality for working with Python packages, e.g., parsing wheel files.
puffin-resolver
Functionality for resolving Python packages and their dependencies.
puffin-traits
Shared traits for Puffin, to avoid circular dependencies.
pypi-types
General-purpose type definitions for types used in PyPI-compatible APIs.
puffin-warnings
User-facing warnings for Puffin.
requirements-txt
Functionality for parsing requirements.txt files.