Since `rust 1.87.0` reported `undefined symbol:
ring::pbkdf2::PBKDF2_HMAC_SHA1::*` in CI and it was difficult to debug
locally, use `rust 1.86.0` in CI tests for troubleshoot the errors
1. Renames `DenoResolver` to `RawDenoResolver`
2. Renames `DenoGraphResolver` to `DenoResolver`
3. Shifts down creation of `DenoGraphResolver` and
`HasPackageJsonDepFlag` to `ResolverFactory`
This removes the `moduleGraphX` data from the `<version>_meta.json`
files for jsr packages when copying from the global cache to the local
one. This property is not really useful to vendor because it's just a
performance optimization when downloading the files and someone may be
changing the data in the files leading to the `moduleGraph` data to be
out of date.
Closes https://github.com/denoland/deno/issues/27229.
TODO:
- [x] Tests
- [x] Make some changes to `deno_cache_dir` so we can get the paths for
the local http cache
- [x] Right now this leaves the node modules setup cache in an incorrect
state (removes the symlinks, but doesn't update the setup cache)
- [ ] ~~Handle code cache and other sqlite caches?~~
This is the release commit being forwarded back to main for 2.2.11
---------
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is the release commit being forwarded back to main for 2.2.10
Co-authored-by: nathanwhit <nathanwhit@users.noreply.github.com>
Co-authored-by: Nathan Whitaker <nathan@deno.com>
Fixes#27264. Fixes https://github.com/denoland/deno/issues/28161.
Currently the new lockfile version is gated behind an unstable flag
(`--unstable-lockfile-v5`) until the next minor release, where it will
become the default.
The main motivation here is that it improves startup performance when
using the global cache or `--node-modules-dir=auto`.
In a create-next-app project, running an empty file:
```
❯ hyperfine --warmup 25 -N --setup "rm -f deno.lock" "deno run --node-modules-dir=auto -A empty.js" "deno-this-pr run --node-modules-dir=auto -A empty.js" "deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js" "deno run --node-modules-dir=manual -A empty.js" "deno-this-pr run --node-modules-dir=manual -A empty.js"
Benchmark 1: deno run --node-modules-dir=auto -A empty.js
Time (mean ± σ): 247.6 ms ± 1.7 ms [User: 228.7 ms, System: 19.0 ms]
Range (min … max): 245.5 ms … 251.5 ms 12 runs
Benchmark 2: deno-this-pr run --node-modules-dir=auto -A empty.js
Time (mean ± σ): 169.8 ms ± 1.0 ms [User: 152.9 ms, System: 17.9 ms]
Range (min … max): 168.9 ms … 172.5 ms 17 runs
Benchmark 3: deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js
Time (mean ± σ): 16.2 ms ± 0.7 ms [User: 12.3 ms, System: 5.7 ms]
Range (min … max): 15.2 ms … 19.2 ms 185 runs
Benchmark 4: deno run --node-modules-dir=manual -A empty.js
Time (mean ± σ): 16.2 ms ± 0.8 ms [User: 11.6 ms, System: 5.5 ms]
Range (min … max): 14.9 ms … 19.7 ms 187 runs
Benchmark 5: deno-this-pr run --node-modules-dir=manual -A empty.js
Time (mean ± σ): 16.0 ms ± 0.9 ms [User: 12.0 ms, System: 5.5 ms]
Range (min … max): 14.8 ms … 22.3 ms 190 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Summary
deno-this-pr run --node-modules-dir=manual -A empty.js ran
1.01 ± 0.08 times faster than deno run --node-modules-dir=manual -A empty.js
1.01 ± 0.07 times faster than deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js
10.64 ± 0.60 times faster than deno-this-pr run --node-modules-dir=auto -A empty.js
15.51 ± 0.88 times faster than deno run --node-modules-dir=auto -A empty.js
```
When using the new lockfile version, this leads to a 15.5x faster
startup time compared to the current deno version.
Install times benefit as well, though to a lesser degree.
`deno install` on a create-next-app project, with everything cached
(just setting up node_modules from scratch):
```
❯ hyperfine --warmup 5 -N --prepare "rm -rf node_modules" --setup "rm -rf deno.lock" "deno i" "deno-this-pr i" "deno-this-pr i --unstable-lockfile-v5"
Benchmark 1: deno i
Time (mean ± σ): 464.4 ms ± 8.8 ms [User: 227.7 ms, System: 217.3 ms]
Range (min … max): 452.6 ms … 478.3 ms 10 runs
Benchmark 2: deno-this-pr i
Time (mean ± σ): 368.8 ms ± 22.0 ms [User: 150.8 ms, System: 198.1 ms]
Range (min … max): 344.8 ms … 397.6 ms 10 runs
Benchmark 3: deno-this-pr i --unstable-lockfile-v5
Time (mean ± σ): 211.9 ms ± 17.1 ms [User: 7.1 ms, System: 177.2 ms]
Range (min … max): 191.3 ms … 233.4 ms 10 runs
Summary
deno-this-pr i --unstable-lockfile-v5 ran
1.74 ± 0.17 times faster than deno-this-pr i
2.19 ± 0.18 times faster than deno i
```
With lockfile v5, a 2.19x faster install time compared to the current
deno.
This is the release commit being forwarded back to main for 2.2.7
Signed-off-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Fixes#28517.
The npm package info gets requested a bunch of times by deno_npm. Before
this PR, we were loading it from the FS and parsing it each and every
time. With a lot of dependencies (and large `registry.json` files), this
can lead to massive blowups in install times.
From the repro in #28517
before this PR:
```
Command being timed: "deno i"
User time (seconds): 538.54
System time (seconds): 56.49
Percent of CPU this job got: 198%
Elapsed (wall clock) time (h:mm:ss or m:ss): 4:59.45
Maximum resident set size (kbytes): 378976
```
this PR:
```
Command being timed: "deno-this-pr i"
User time (seconds): 1.29
System time (seconds): 1.56
Percent of CPU this job got: 68%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:04.16
Maximum resident set size (kbytes): 500864
```
So roughly an improvement from 339s to 4s. You can see that the max RSS
does increase a decent amount, which is the main downside. However, this
in memory cache is cleared once we're done caching npm packages, and IMO
the performance tradeoff is well worth it.
This also has a very noticable, though less drastic, effect on fresh
installs (no deno.lock) for smaller projects. Here's a clean nextJS
template project:
```
❯ hyperfine --warmup 5 --prepare "rm -rf node_modules deno.lock" "deno i" "deno-this-pr i"
Benchmark 1: deno
Time (mean ± σ): 765.0 ms ± 10.1 ms [User: 622.3 ms, System: 216.4 ms]
Range (min … max): 749.0 ms … 783.6 ms 10 runs
Benchmark 2: deno-this-pr
Time (mean ± σ): 357.2 ms ± 9.4 ms [User: 193.2 ms, System: 198.2 ms]
Range (min … max): 346.4 ms … 374.1 ms 10 runs
Summary
deno-this-pr ran
2.14 ± 0.06 times faster than deno
```
NOTE: Commit 27363d389 was incorrectly landed in main before the release
completed and is not included in v2.2.5. The official v2.2.5 release was made
from the v2.2 branch.
This adds support for using a local copy of an npm package.
```js
// deno.json
{
"patch": [
"../path/to/local_npm_package"
],
// required until Deno 2.3, but it will still be considered unstable
"unstable": ["npm-patch"]
}
```
1. Requires using a node_modules folder.
2. When using `"nodeModulesDir": "auto"`, it recreates the folder in the
node_modules directory on each run which will slightly increase startup
time.
3. When using the default with a package.json (`"nodeModulesDir":
"manual"`), updating the package requires running `deno install`. This
is to get the package into the node_modules directory of the current
workspace. This is necessary instead of linking because packages can
have multiple "copy packages" due to peer dep resolution.
Caveat: Specifying a local copy of an npm package or making changes to
its dependencies will purge npm packages from the lockfile. This might
cause npm resolution to resolve differently and it may end up not using
the local copy of the npm package. It's very difficult to only
invalidate resolution midway through the graph and then only rebuild
that part of the resolution, so this is just a first pass that can be
improved in the future. In practice, this probably won't be an issue for
most people.
Another limitation is this also requires the npm package name to exist
in the registry at the moment.
Adds support for re-exporting an ES module from a CJS one and then
importing the CJS module from ESM. Also fixes a bug where require esm
wasn't working in deno compile.
This adds support for installing `file:` dependencies in a local
package.json.
In order to use these, you must not set `--node-modules-dir=...` when
using a package.json and it should use the default of
`--node-modules-dir=manual`.
Closes https://github.com/denoland/deno/issues/18701
Updates to use rust 1.85. Doesn't move to the 2024 edition, as that's a
fair bit more involved.
A nice side benefit is that the new rustc version seems to lead to a
slight reduction in binary size (at least on mac):
```
FILE SIZE
--------------
+4.3% +102Ki __DATA_CONST,__const
[NEW] +69.3Ki __TEXT,__literals
[NEW] +68.5Ki Rebase Info
+5.0% +39.9Ki __TEXT,__unwind_info
+57% +8.85Ki [__TEXT]
[NEW] +8.59Ki Lazy Binding Info
[NEW] +5.16Ki __TEXT,__stub_helper
[NEW] +3.58Ki Export Info
[NEW] +3.42Ki __DATA,__la_symbol_ptr
-0.1% -726 [12 Others]
-21.4% -3.10Ki [__DATA_CONST]
-95.8% -3.39Ki __DATA_CONST,__got
-20.9% -3.43Ki [__DATA]
-0.5% -4.52Ki Code Signature
-100.0% -11.6Ki [__LINKEDIT]
-1.0% -43.5Ki Symbol Table
-1.6% -44.0Ki __TEXT,__gcc_except_tab
-0.2% -48.1Ki __TEXT,__const
-3.3% -78.6Ki __TEXT,__eh_frame
-0.7% -320Ki __TEXT,__text
-1.5% -334Ki String Table
-0.5% -586Ki TOTAL
```
Fixes https://github.com/denoland/deno/issues/27569.
Fixes https://github.com/denoland/deno/issues/27215.
This PR makes it so type resolution falls back to looking for definitely
typed packages (`@types/foo`) if a given NPM package does not contain
type declarations.
One complication is choosing _which_ version of the `@types/*` package
to use, if the project depends on multiple versions. The heuristic here
is to try to match the major and minor versions, falling back to the
latest version. So if you have
```
@types/foo: 0.1.0, 0.2.0, 3.1.0, 3.1.2, 4.0.0
foo: 3.1.0
```
we would choose `@types/foo@3.1.2` when resolving types for `foo`.
---
Note that this only uses `@types/` packages if you _already_ depend on
them. So a follow up to this PR could be to add a diagnostic and
quickfix to install `@types/foo` if we don't find types for `foo`.
