Commit graph

20 commits

Author SHA1 Message Date
Charlie Marsh
e109e02b12
Expand documentation in scripts/bench/__main__.py (#2875)
Closes https://github.com/astral-sh/uv/issues/1318.
2024-04-08 02:48:12 +00:00
konsti
79fbac7af5
Fast lint CI job: Rustfmt, Prettier, Ruff (#2406)
Add a single job for for fast lint tools. Rustfmt for rust, ruff for
python formatting and linting, prettier avoids inconsistent formatter
changes between pycharm and vscode.
2024-03-20 00:16:46 +00:00
Henry Schreiner
c6fd3d97fb
fix: remove uv version from uv pip compile header (#1716)
## Summary

This fixes https://github.com/astral-sh/uv/issues/1704 by removing the
version from the produced header.

## Test Plan

Checked with clippy, and tests are updated too.
2024-02-19 20:26:53 +00:00
Zanie Blue
ee2e7bb9ae
Change benchmarks to use 3.12.1 (#1322)
We can wait to merge this until later, but we explicitly bootstrap
3.12.1 and the benches should not pin 3.12.0
2024-02-17 12:00:50 -06:00
Zanie Blue
2586f655bb
Rename to uv (#1302)
First, replace all usages in files in-place. I used my editor for this.
If someone wants to add a one-liner that'd be fun.

Then, update directory and file names:

```
# Run twice for nested directories
find . -type d -print0 | xargs -0 rename s/puffin/uv/g
find . -type d -print0 | xargs -0 rename s/puffin/uv/g

# Update files
find . -type f -print0 | xargs -0 rename s/puffin/uv/g
```

Then add all the files again

```
# Add all the files again
git add crates
git add python/uv

# This one needs a force-add
git add -f crates/uv-trampoline
```
2024-02-15 11:19:46 -06:00
Charlie Marsh
51e8609ee8
Use Python 3.12 in benchmarks (#1215)
I originally used Python 3.10, since 3.10 and 3.11 are by far the most
common (at least for [Ruff](https://pypistats.org/packages/ruff)). But
3.12 should give Python tools the most favorable benchmarks.
2024-01-31 15:51:13 -05:00
Charlie Marsh
ee69fb51ea
Add PDM to benchmark script (#1214)
## Summary

Overall, similar to Poetry, with some simplifications (e.g., we don't
need to translate to Poetry's dependency syntax), and the need to adjust
how we manage the cache and virtual environment.
2024-01-31 20:31:45 +00:00
Charlie Marsh
c4bfb6efee
Add a BENCHMARKS.md with rendered benchmarks (#1211)
As a precursor to the release, I want to include a structured document
with detailed benchmarks.

Closes https://github.com/astral-sh/puffin/issues/1210.
2024-01-31 20:11:52 +00:00
Charlie Marsh
c6d36a161a
Make bench script requirements Poetry-only (#1093)
If you're just benchmarking Puffin, you can do so without setting up a
virtual environment.
2024-01-25 09:48:00 -05:00
Charlie Marsh
f852d986f3
Add an incremental resolution benchmark (#954)
## Summary

This adds a benchmark in which we reuse the lockfile, but add a new
dependency to the input requirements.

Running `python -m scripts.bench --poetry --puffin --pip-compile
scripts/requirements/trio.in --benchmark resolve-warm --benchmark
resolve-incremental`:

```text
Benchmark 1: pip-compile (resolve-warm)
  Time (mean ± σ):      1.169 s ±  0.023 s    [User: 0.675 s, System: 0.112 s]
  Range (min … max):    1.129 s …  1.198 s    10 runs

Benchmark 2: poetry (resolve-warm)
  Time (mean ± σ):     610.7 ms ±  10.4 ms    [User: 528.1 ms, System: 60.3 ms]
  Range (min … max):   599.9 ms … 632.6 ms    10 runs

Benchmark 3: puffin (resolve-warm)
  Time (mean ± σ):      19.3 ms ±   0.6 ms    [User: 13.5 ms, System: 13.1 ms]
  Range (min … max):    17.9 ms …  22.1 ms    122 runs

Summary
  'puffin (resolve-warm)' ran
   31.63 ± 1.19 times faster than 'poetry (resolve-warm)'
   60.53 ± 2.37 times faster than 'pip-compile (resolve-warm)'
Benchmark 1: pip-compile (resolve-incremental)
  Time (mean ± σ):      1.554 s ±  0.059 s    [User: 0.974 s, System: 0.130 s]
  Range (min … max):    1.473 s …  1.652 s    10 runs

Benchmark 2: poetry (resolve-incremental)
  Time (mean ± σ):     474.2 ms ±   2.4 ms    [User: 411.7 ms, System: 54.0 ms]
  Range (min … max):   470.6 ms … 477.7 ms    10 runs

Benchmark 3: puffin (resolve-incremental)
  Time (mean ± σ):      28.0 ms ±   1.1 ms    [User: 21.7 ms, System: 14.6 ms]
  Range (min … max):    26.7 ms …  34.4 ms    89 runs

Summary
  'puffin (resolve-incremental)' ran
   16.94 ± 0.67 times faster than 'poetry (resolve-incremental)'
   55.52 ± 3.02 times faster than 'pip-compile (resolve-incremental)'
```
2024-01-18 18:18:37 +00:00
Zanie Blue
4feef006e9
Fixup to benchmark requirement setup (#952) 2024-01-17 14:05:48 -05:00
Charlie Marsh
249ca10765
Move Puffin subcommands to a pip namespace (#921)
## Summary

This makes the separation clearer between the legacy `pip` API and the
API we'll add in the future for the package manager itself. It also
enables seamless `puffin pip` aliasing for those that want it.

Closes #918.
2024-01-15 16:36:45 +00:00
Andrew Gallant
1629141d67
bench: set log level to WARN, make --verbose raise it to INFO (#898)
I personally found the output by default somewhat noisy, especially for
large requirements files. Since --verbose is already a thing, I propose
making the extra output opt-in.
2024-01-12 10:07:58 -05:00
Andrew Gallant
cf62d296b3
bench: ignore empty requirements lines (#897)
In particular, this script previously choked on the `home-assistant.in`
requirements file because it contains many empty lines.
2024-01-12 09:39:48 -05:00
bojanserafimov
cd6265a66d
Fix typo in bench docs (#833) 2024-01-08 12:35:26 -05:00
Charlie Marsh
5f98210083
Use a single hyperfine command for each benchmark (#819)
## Summary

Refactors the benchmark script such that we use a single `hyperfine`
invocation per benchmark, and thus get the comparative summary, which is
_way_ nicer:

```
Benchmark 1: ./target/release/puffin (install-cold)
  Time (mean ± σ):     410.3 ms ±  19.9 ms    [User: 173.7 ms, System: 1314.5 ms]
  Range (min … max):   389.7 ms … 452.1 ms    10 runs

Benchmark 2: ./target/release/baseline (install-cold)
  Time (mean ± σ):     418.2 ms ±  14.4 ms    [User: 210.7 ms, System: 1246.0 ms]
  Range (min … max):   397.3 ms … 445.7 ms    10 runs

Summary
  './target/release/puffin (install-cold)' ran
    1.02 ± 0.06 times faster than './target/release/baseline (install-cold)'
```
2024-01-06 20:14:22 +00:00
Charlie Marsh
0817a0d0d4
Use a dedicated flag for each tool in bench script (#818)
Taking some of Zanie's suggestions to make the custom-path API simpler
in the benchmark script. Each tool is now a dedicated argument, like:

```
python -m scripts.bench --pip-sync --poetry requirements.in
```

To provide custom binaries:

```
python -m scripts.bench \
    --puffin-path ./target/release/puffin \
    --puffin-path ./target/release/baseline \
    requirements.in
```
2024-01-06 19:45:00 +00:00
Charlie Marsh
3b43515262
Misc. refactors to benchmark script (#814)
- Use separate suites for `pip-sync` and `pip-compile`
- DRY up some properties
- Improve documentation
- Reorder classes to match enum
2024-01-06 03:33:50 +00:00
Charlie Marsh
063dd00542
Enable self-benchmarking for Puffin branches in bench.py (#804)
## Summary

This PR enables the use of the `bench.py` script to benchmark Puffin
itself. This is something I often do by via a process like:

- Checkout the `main` branch (or any other baseline branch)
- Run: `cargo build --release`
- Run: `mv ./target/release/puffin ./target/release/baseline`
- Checkout a development branch
- Run: `cargo build --release`
- (New) Run: `python bench.py --tool puffin --path
./target/release/puffin --tool puffin --path ./target/release/baseline
requirements.in`
2024-01-06 03:23:19 +00:00
Charlie Marsh
d2d87db7a3
Add Poetry support to bench.py (#803)
## Summary

Enables benchmarking against Poetry for resolution and installation:

```
Benchmark 1: pip-tools (resolve-cold)
  Time (mean ± σ):     962.7 ms ± 241.9 ms    [User: 322.8 ms, System: 80.5 ms]
  Range (min … max):   714.9 ms … 1459.4 ms    10 runs

Benchmark 1: puffin (resolve-cold)
  Time (mean ± σ):     193.2 ms ±   8.2 ms    [User: 31.3 ms, System: 22.8 ms]
  Range (min … max):   179.8 ms … 206.4 ms    14 runs

Benchmark 1: poetry (resolve-cold)
  Time (mean ± σ):     900.7 ms ±  21.2 ms    [User: 371.6 ms, System: 92.1 ms]
  Range (min … max):   855.7 ms … 933.4 ms    10 runs

Benchmark 1: pip-tools (resolve-warm)
  Time (mean ± σ):     386.0 ms ±  19.1 ms    [User: 255.8 ms, System: 46.2 ms]
  Range (min … max):   368.7 ms … 434.5 ms    10 runs

Benchmark 1: puffin (resolve-warm)
  Time (mean ± σ):       8.1 ms ±   0.4 ms    [User: 4.4 ms, System: 5.1 ms]
  Range (min … max):     7.5 ms …  11.1 ms    183 runs

Benchmark 1: poetry (resolve-warm)
  Time (mean ± σ):     336.3 ms ±   0.6 ms    [User: 283.6 ms, System: 44.7 ms]
  Range (min … max):   335.0 ms … 337.3 ms    10 runs
```
2024-01-06 02:52:55 +00:00