Closes https://github.com/denoland/deno/issues/27229.
TODO:
- [x] Tests
- [x] Make some changes to `deno_cache_dir` so we can get the paths for
the local http cache
- [x] Right now this leaves the node modules setup cache in an incorrect
state (removes the symlinks, but doesn't update the setup cache)
- [ ] ~~Handle code cache and other sqlite caches?~~
Basically just update deno_lockfile, deno_npm, and eszip, and then adapt
to those changes. The main changes were the removal of the lockfile v4
resolution snapshot loading, and a terser formatting for the `os` and
`cpu` fields in the lockfile.
Fixes two issues:
- If a cached packument was out of date and missing a version from the
lockfile, we would fail. Instead we should try again with a forced
re-fetch
- We weren't threading through the workspace patch packages correctly
With `nodeModulesDir: auto` (and `none`, to a lesser extent), we
frequently repeatedly try to cache npm packages during resolution. On a
(private) fresh project, I counted that `sync_resolution_with_fs` was
called 478 times over the course of a fresh build. This reduces the
number of calls to 8, and doing so speeds things up substantially.
```
❯ hyperfine --warmup 2 -N "deno task build" "../deno/target/release-lite/deno task build"
Benchmark 1: deno task build
Time (mean ± σ): 3.652 s ± 0.042 s [User: 3.285 s, System: 2.399 s]
Range (min … max): 3.587 s … 3.712 s 10 runs
Benchmark 2: ../deno/target/release-lite/deno task build
Time (mean ± σ): 1.356 s ± 0.007 s [User: 1.961 s, System: 1.108 s]
Range (min … max): 1.345 s … 1.372 s 10 runs
Summary
../deno/target/release-lite/deno task build ran
2.69 ± 0.03 times faster than deno task build
```
Fixes#28891
We were checking if the node_modules entry for the package was present,
but then reading from the global cache.
Instead, read from the package.json in node_modules. As a fallback(which
in theory should only really happen if the node_modules dir is somehow
messed up), take the more expensive (but likely to work) path of reading
from the registry.json.
This is where deno_lockfile gets the info for the transform from 4->5.
So while we were doing this optimization on new v5 lockfiles, we weren't
doing it correctly for ones migrated from earlier versions.
Testing this is kinda hard because our tests don't use the default
registry, hmm
Apparently things like the `bin` field can appear in the version info
from the registry, but not the package's `package.json`. I'm still not
sure how you actually achieve this, but it's the case for
`esbuild-wasm`. This fixes the following panic:
```
❯ deno i --node-modules-dir npm:esbuild-wasm
Add npm:esbuild-wasm@0.25.2
Initialize ⣯ [00:00]
- esbuild-wasm@0.25.2
============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.
Platform: macos aarch64
Version: 2.2.8+58c6c0b
Args: ["deno", "i", "--node-modules-dir", "npm:esbuild-wasm"]
View stack trace at:
https://panic.deno.com/v2.2.8+58c6c0bc9c1b4ee08645be936ff9268f17028f0f/aarch64-apple-darwin/g4h6Jo393pB4k4kqBo-3kqBg6klqBogtyLg13yLw_t0Lw549Hgj8-Hgw__H428-F4yv_HgjkpKww7gIon4gIw54rKwi5MorzMw5y7G42g7Iw---I40s-I4vu4Jw2rEw8z7Dwnr6J4tp7Bo_vvK
thread 'main' panicked at cli/npm/installer/common/bin_entries.rs:108:30:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Fixes#27264. Fixes https://github.com/denoland/deno/issues/28161.
Currently the new lockfile version is gated behind an unstable flag
(`--unstable-lockfile-v5`) until the next minor release, where it will
become the default.
The main motivation here is that it improves startup performance when
using the global cache or `--node-modules-dir=auto`.
In a create-next-app project, running an empty file:
```
❯ hyperfine --warmup 25 -N --setup "rm -f deno.lock" "deno run --node-modules-dir=auto -A empty.js" "deno-this-pr run --node-modules-dir=auto -A empty.js" "deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js" "deno run --node-modules-dir=manual -A empty.js" "deno-this-pr run --node-modules-dir=manual -A empty.js"
Benchmark 1: deno run --node-modules-dir=auto -A empty.js
Time (mean ± σ): 247.6 ms ± 1.7 ms [User: 228.7 ms, System: 19.0 ms]
Range (min … max): 245.5 ms … 251.5 ms 12 runs
Benchmark 2: deno-this-pr run --node-modules-dir=auto -A empty.js
Time (mean ± σ): 169.8 ms ± 1.0 ms [User: 152.9 ms, System: 17.9 ms]
Range (min … max): 168.9 ms … 172.5 ms 17 runs
Benchmark 3: deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js
Time (mean ± σ): 16.2 ms ± 0.7 ms [User: 12.3 ms, System: 5.7 ms]
Range (min … max): 15.2 ms … 19.2 ms 185 runs
Benchmark 4: deno run --node-modules-dir=manual -A empty.js
Time (mean ± σ): 16.2 ms ± 0.8 ms [User: 11.6 ms, System: 5.5 ms]
Range (min … max): 14.9 ms … 19.7 ms 187 runs
Benchmark 5: deno-this-pr run --node-modules-dir=manual -A empty.js
Time (mean ± σ): 16.0 ms ± 0.9 ms [User: 12.0 ms, System: 5.5 ms]
Range (min … max): 14.8 ms … 22.3 ms 190 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Summary
deno-this-pr run --node-modules-dir=manual -A empty.js ran
1.01 ± 0.08 times faster than deno run --node-modules-dir=manual -A empty.js
1.01 ± 0.07 times faster than deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js
10.64 ± 0.60 times faster than deno-this-pr run --node-modules-dir=auto -A empty.js
15.51 ± 0.88 times faster than deno run --node-modules-dir=auto -A empty.js
```
When using the new lockfile version, this leads to a 15.5x faster
startup time compared to the current deno version.
Install times benefit as well, though to a lesser degree.
`deno install` on a create-next-app project, with everything cached
(just setting up node_modules from scratch):
```
❯ hyperfine --warmup 5 -N --prepare "rm -rf node_modules" --setup "rm -rf deno.lock" "deno i" "deno-this-pr i" "deno-this-pr i --unstable-lockfile-v5"
Benchmark 1: deno i
Time (mean ± σ): 464.4 ms ± 8.8 ms [User: 227.7 ms, System: 217.3 ms]
Range (min … max): 452.6 ms … 478.3 ms 10 runs
Benchmark 2: deno-this-pr i
Time (mean ± σ): 368.8 ms ± 22.0 ms [User: 150.8 ms, System: 198.1 ms]
Range (min … max): 344.8 ms … 397.6 ms 10 runs
Benchmark 3: deno-this-pr i --unstable-lockfile-v5
Time (mean ± σ): 211.9 ms ± 17.1 ms [User: 7.1 ms, System: 177.2 ms]
Range (min … max): 191.3 ms … 233.4 ms 10 runs
Summary
deno-this-pr i --unstable-lockfile-v5 ran
1.74 ± 0.17 times faster than deno-this-pr i
2.19 ± 0.18 times faster than deno i
```
With lockfile v5, a 2.19x faster install time compared to the current
deno.
This improves peer dependency resolution to be more relaxed and resolve
non-version matching ancestors similar to pnpm rather than introducing
duplicate dependencies. Deno will warn when this occurs. In the future,
we should look into introducing an option to have Deno error in this
scenario.
This adds support for using a local copy of an npm package.
```js
// deno.json
{
"patch": [
"../path/to/local_npm_package"
],
// required until Deno 2.3, but it will still be considered unstable
"unstable": ["npm-patch"]
}
```
1. Requires using a node_modules folder.
2. When using `"nodeModulesDir": "auto"`, it recreates the folder in the
node_modules directory on each run which will slightly increase startup
time.
3. When using the default with a package.json (`"nodeModulesDir":
"manual"`), updating the package requires running `deno install`. This
is to get the package into the node_modules directory of the current
workspace. This is necessary instead of linking because packages can
have multiple "copy packages" due to peer dep resolution.
Caveat: Specifying a local copy of an npm package or making changes to
its dependencies will purge npm packages from the lockfile. This might
cause npm resolution to resolve differently and it may end up not using
the local copy of the npm package. It's very difficult to only
invalidate resolution midway through the graph and then only rebuild
that part of the resolution, so this is just a first pass that can be
improved in the future. In practice, this probably won't be an issue for
most people.
Another limitation is this also requires the npm package name to exist
in the registry at the moment.
This adds support for installing `file:` dependencies in a local
package.json.
In order to use these, you must not set `--node-modules-dir=...` when
using a package.json and it should use the default of
`--node-modules-dir=manual`.
Closes https://github.com/denoland/deno/issues/18701
Updates to use rust 1.85. Doesn't move to the 2024 edition, as that's a
fair bit more involved.
A nice side benefit is that the new rustc version seems to lead to a
slight reduction in binary size (at least on mac):
```
FILE SIZE
--------------
+4.3% +102Ki __DATA_CONST,__const
[NEW] +69.3Ki __TEXT,__literals
[NEW] +68.5Ki Rebase Info
+5.0% +39.9Ki __TEXT,__unwind_info
+57% +8.85Ki [__TEXT]
[NEW] +8.59Ki Lazy Binding Info
[NEW] +5.16Ki __TEXT,__stub_helper
[NEW] +3.58Ki Export Info
[NEW] +3.42Ki __DATA,__la_symbol_ptr
-0.1% -726 [12 Others]
-21.4% -3.10Ki [__DATA_CONST]
-95.8% -3.39Ki __DATA_CONST,__got
-20.9% -3.43Ki [__DATA]
-0.5% -4.52Ki Code Signature
-100.0% -11.6Ki [__LINKEDIT]
-1.0% -43.5Ki Symbol Table
-1.6% -44.0Ki __TEXT,__gcc_except_tab
-0.2% -48.1Ki __TEXT,__const
-3.3% -78.6Ki __TEXT,__eh_frame
-0.7% -320Ki __TEXT,__text
-1.5% -334Ki String Table
-0.5% -586Ki TOTAL
```
Speeds up the caching part of this arbitrary `"nodeModulesDir": "auto"`
project by about 22%
We write the tags associated with a given npm package to the
`.initialized` file, so that byonm can correctly resolve tags. When
setting up the node modules dir, we read that file to see if we need to
update the tags.
If we don't have any tags associated with the package though, we can
just check for existence (which is a fair bit faster than trying to
`open` + `read` a file).
```
❯ hyperfine --warmup 3 "deno check src/**/*.ts" "../deno/target/release-lite/deno check src/**/*.ts"
Benchmark 1: deno check src/**/*.ts
Time (mean ± σ): 369.9 ms ± 5.5 ms [User: 286.9 ms, System: 128.9 ms]
Range (min … max): 361.7 ms … 377.7 ms 10 runs
Benchmark 2: ../deno/target/release-lite/deno check src/**/*.ts
Time (mean ± σ): 303.5 ms ± 5.9 ms [User: 210.9 ms, System: 124.5 ms]
Range (min … max): 292.7 ms … 315.0 ms 10 runs
Summary
../deno/target/release-lite/deno check src/**/*.ts ran
1.22 ± 0.03 times faster than deno check src/**/*.ts
```
Extracted out of https://github.com/denoland/deno/pull/27838/files
Reduces some allocations by accepting either a pathbuf or url for the
referrer for resolution and returning either a pathbuf or url at the
end, which the caller can then convert into to their preferred state.
This is about 4% faster when still converting the final result to a url
and 6% faster when keeping the result as a path in a benchmark I ran.
This slightly degrades the performance of CJS export analysis on
subsequent runs because I changed it to no longer cache in the DENO_DIR
with this PR (denort now properly has no idea about the DENO_DIR). We'll
have to change it to embed this data in the binary and that will also
allow us to get rid of swc in denort (will do that in a follow-up PR).
