Closes https://github.com/denoland/deno/issues/27229.
TODO:
- [x] Tests
- [x] Make some changes to `deno_cache_dir` so we can get the paths for
the local http cache
- [x] Right now this leaves the node modules setup cache in an incorrect
state (removes the symlinks, but doesn't update the setup cache)
- [ ] ~~Handle code cache and other sqlite caches?~~
Fixes two issues:
- If a cached packument was out of date and missing a version from the
lockfile, we would fail. Instead we should try again with a forced
re-fetch
- We weren't threading through the workspace patch packages correctly
With `nodeModulesDir: auto` (and `none`, to a lesser extent), we
frequently repeatedly try to cache npm packages during resolution. On a
(private) fresh project, I counted that `sync_resolution_with_fs` was
called 478 times over the course of a fresh build. This reduces the
number of calls to 8, and doing so speeds things up substantially.
```
❯ hyperfine --warmup 2 -N "deno task build" "../deno/target/release-lite/deno task build"
Benchmark 1: deno task build
Time (mean ± σ): 3.652 s ± 0.042 s [User: 3.285 s, System: 2.399 s]
Range (min … max): 3.587 s … 3.712 s 10 runs
Benchmark 2: ../deno/target/release-lite/deno task build
Time (mean ± σ): 1.356 s ± 0.007 s [User: 1.961 s, System: 1.108 s]
Range (min … max): 1.345 s … 1.372 s 10 runs
Summary
../deno/target/release-lite/deno task build ran
2.69 ± 0.03 times faster than deno task build
```
Fixes#27264. Fixes https://github.com/denoland/deno/issues/28161.
Currently the new lockfile version is gated behind an unstable flag
(`--unstable-lockfile-v5`) until the next minor release, where it will
become the default.
The main motivation here is that it improves startup performance when
using the global cache or `--node-modules-dir=auto`.
In a create-next-app project, running an empty file:
```
❯ hyperfine --warmup 25 -N --setup "rm -f deno.lock" "deno run --node-modules-dir=auto -A empty.js" "deno-this-pr run --node-modules-dir=auto -A empty.js" "deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js" "deno run --node-modules-dir=manual -A empty.js" "deno-this-pr run --node-modules-dir=manual -A empty.js"
Benchmark 1: deno run --node-modules-dir=auto -A empty.js
Time (mean ± σ): 247.6 ms ± 1.7 ms [User: 228.7 ms, System: 19.0 ms]
Range (min … max): 245.5 ms … 251.5 ms 12 runs
Benchmark 2: deno-this-pr run --node-modules-dir=auto -A empty.js
Time (mean ± σ): 169.8 ms ± 1.0 ms [User: 152.9 ms, System: 17.9 ms]
Range (min … max): 168.9 ms … 172.5 ms 17 runs
Benchmark 3: deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js
Time (mean ± σ): 16.2 ms ± 0.7 ms [User: 12.3 ms, System: 5.7 ms]
Range (min … max): 15.2 ms … 19.2 ms 185 runs
Benchmark 4: deno run --node-modules-dir=manual -A empty.js
Time (mean ± σ): 16.2 ms ± 0.8 ms [User: 11.6 ms, System: 5.5 ms]
Range (min … max): 14.9 ms … 19.7 ms 187 runs
Benchmark 5: deno-this-pr run --node-modules-dir=manual -A empty.js
Time (mean ± σ): 16.0 ms ± 0.9 ms [User: 12.0 ms, System: 5.5 ms]
Range (min … max): 14.8 ms … 22.3 ms 190 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Summary
deno-this-pr run --node-modules-dir=manual -A empty.js ran
1.01 ± 0.08 times faster than deno run --node-modules-dir=manual -A empty.js
1.01 ± 0.07 times faster than deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js
10.64 ± 0.60 times faster than deno-this-pr run --node-modules-dir=auto -A empty.js
15.51 ± 0.88 times faster than deno run --node-modules-dir=auto -A empty.js
```
When using the new lockfile version, this leads to a 15.5x faster
startup time compared to the current deno version.
Install times benefit as well, though to a lesser degree.
`deno install` on a create-next-app project, with everything cached
(just setting up node_modules from scratch):
```
❯ hyperfine --warmup 5 -N --prepare "rm -rf node_modules" --setup "rm -rf deno.lock" "deno i" "deno-this-pr i" "deno-this-pr i --unstable-lockfile-v5"
Benchmark 1: deno i
Time (mean ± σ): 464.4 ms ± 8.8 ms [User: 227.7 ms, System: 217.3 ms]
Range (min … max): 452.6 ms … 478.3 ms 10 runs
Benchmark 2: deno-this-pr i
Time (mean ± σ): 368.8 ms ± 22.0 ms [User: 150.8 ms, System: 198.1 ms]
Range (min … max): 344.8 ms … 397.6 ms 10 runs
Benchmark 3: deno-this-pr i --unstable-lockfile-v5
Time (mean ± σ): 211.9 ms ± 17.1 ms [User: 7.1 ms, System: 177.2 ms]
Range (min … max): 191.3 ms … 233.4 ms 10 runs
Summary
deno-this-pr i --unstable-lockfile-v5 ran
1.74 ± 0.17 times faster than deno-this-pr i
2.19 ± 0.18 times faster than deno i
```
With lockfile v5, a 2.19x faster install time compared to the current
deno.
Updates to use rust 1.85. Doesn't move to the 2024 edition, as that's a
fair bit more involved.
A nice side benefit is that the new rustc version seems to lead to a
slight reduction in binary size (at least on mac):
```
FILE SIZE
--------------
+4.3% +102Ki __DATA_CONST,__const
[NEW] +69.3Ki __TEXT,__literals
[NEW] +68.5Ki Rebase Info
+5.0% +39.9Ki __TEXT,__unwind_info
+57% +8.85Ki [__TEXT]
[NEW] +8.59Ki Lazy Binding Info
[NEW] +5.16Ki __TEXT,__stub_helper
[NEW] +3.58Ki Export Info
[NEW] +3.42Ki __DATA,__la_symbol_ptr
-0.1% -726 [12 Others]
-21.4% -3.10Ki [__DATA_CONST]
-95.8% -3.39Ki __DATA_CONST,__got
-20.9% -3.43Ki [__DATA]
-0.5% -4.52Ki Code Signature
-100.0% -11.6Ki [__LINKEDIT]
-1.0% -43.5Ki Symbol Table
-1.6% -44.0Ki __TEXT,__gcc_except_tab
-0.2% -48.1Ki __TEXT,__const
-3.3% -78.6Ki __TEXT,__eh_frame
-0.7% -320Ki __TEXT,__text
-1.5% -334Ki String Table
-0.5% -586Ki TOTAL
```
