todo:
- [ ] cleanup cli, decide what flags we want to commit to
- [x] decide what to do about node addons - (you can mark them external
via `--external`)
- [x] move `esbuild_rs` to the `denoland` org
- [x] figure out the dynamic require issue
- [x] figure out how to test this
- [x] clean up / revert all the random changes
Since `rust 1.87.0` reported `undefined symbol:
ring::pbkdf2::PBKDF2_HMAC_SHA1::*` in CI and it was difficult to debug
locally, use `rust 1.86.0` in CI tests for troubleshoot the errors
This commit adds a signal handler for SIGUSR2 that helps reduce the
memory usage of both main worker and web worker by:
1. Triggering `malloc_trim(0)` to release memory back to the system
2. Invoking V8 isolate's `low_memory_notification` function
This is only available on Linux and enabled when `DENO_USR2_MEMORY_TRIM`
env var is specified.
---------
Co-authored-by: Yusuke Tanaka <yusuktan@maguro.dev>
This commit changes the `deno jupyter` subcommand:
- `deno jupyter` now accepts additional `--name` argument to
allow installing and maintaing multiple kernelsspec - useful when
one wants to install a stable kernel and a debug/canary kernel
- `deno jupyter --install` now accepts additional `--display`
argument to allow customizing display name of the kernel - the
default one is "Deno"
- `deno jupyter --install` no longer blindly installs kernelspec,
instead it first checks if a kernelspec already exists and if so,
returns an error suggesting to use `--force` flag
- `deno jupyter --help` no longer shows `--unstable` flag
Closes https://github.com/denoland/deno/issues/29219
Closes https://github.com/denoland/deno/issues/29220
This change configures V8 isolates to respect memory limits imposed by
cgroups on Linux.
It adds support for detecting both cgroups v1 and v2 memory limits,
enabling Deno to properly adapt to containerized environments with
memory constraints. When cgroups information is unavailable or not
applicable, it falls back to using the system's total memory as before.
Closes#29077
## Test
For testing, I created a ubuntu VM with 1Gi memory. Within this VM, set
up a cgroup with 512Mi memory limit, then ran the following script to
see how much heap size limit the V8 isolate had.
```js
import * as v8 from "node:v8";
console.log(v8.getHeapStatistics());
```
### Ubuntu 20.04
In this version of ubuntu, hybrid mode is enabled by default.
```
$ cat /proc/self/cgroup
12:rdma:/
11:blkio:/user.slice
10:devices:/user.slice
9:cpu,cpuacct:/user.slice
8:pids:/user.slice/user-1000.slice/session-3.scope
7:memory:/user.slice/user-1000.slice/session-3.scope
6:perf_event:/
5:freezer:/
4:net_cls,net_prio:/
3:hugetlb:/
2:cpuset:/
1:name=systemd:/user.slice/user-1000.slice/session-3.scope
0::/user.slice/user-1000.slice/session-3.scope
```
Create a new cgroup with 512Mi memory limit and run the above script in
this cgroup:
```
$ sudo cgcreate -g memory:/mygroup
$ sudo cgset -r memory.limit_in_bytes=$((512 * 1024 * 1024)) mygroup
$ sudo cgexec -g memory:mygroup ./deno run main.mjs
{
total_heap_size: 7745536,
total_heap_size_executable: 0,
total_physical_size: 7090176,
total_available_size: 266348216,
used_heap_size: 6276752,
heap_size_limit: 271581184,
malloced_memory: 303200,
peak_malloced_memory: 140456,
does_zap_garbage: 0,
number_of_native_contexts: 1,
number_of_detached_contexts: 0,
total_global_handles_size: 24576,
used_global_handles_size: 22432,
external_memory: 3232012
}
```
This indicates that the isolate was informed of cgroup-constrained
memory limit (512Mi) and hence got ~270M heap limit.
### Ubuntu 22.04
In this version of ubuntu, cgroup v2 is used.
```
$ cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-3.scope
```
Run the above script using `systemd-run`:
```
$ sudo systemd-run --property=MemoryMax=512M --pty bash -c '/home/ubuntu/deno run /home/ubuntu/main.mjs'
{
total_heap_size: 7745536,
total_heap_size_executable: 0,
total_physical_size: 7090176,
total_available_size: 266348184,
used_heap_size: 6276784,
heap_size_limit: 271581184,
malloced_memory: 303200,
peak_malloced_memory: 140456,
does_zap_garbage: 0,
number_of_native_contexts: 1,
number_of_detached_contexts: 0,
total_global_handles_size: 24576,
used_global_handles_size: 22432,
external_memory: 3232012
}
```
Again the isolate got ~270M heap limit properly.
Note that it should have had bigger heap limit if the entire system
memory, i.e. 1Gi, had been passed to V8. In fact, if we run the same
script outside the cgroup, it does display larger `heap_size_limit` like
below:
```
$ ./deno run main.mjs
{
total_heap_size: 7745536,
total_heap_size_executable: 0,
total_physical_size: 7090176,
total_available_size: 546580152,
used_heap_size: 6276752,
heap_size_limit: 551813120,
malloced_memory: 303200,
peak_malloced_memory: 140456,
does_zap_garbage: 0,
number_of_native_contexts: 1,
number_of_detached_contexts: 0,
total_global_handles_size: 24576,
used_global_handles_size: 22432,
external_memory: 3232012
}
```
---------
Signed-off-by: Yusuke Tanaka <wing0920@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Closes https://github.com/denoland/deno/issues/27229.
TODO:
- [x] Tests
- [x] Make some changes to `deno_cache_dir` so we can get the paths for
the local http cache
- [x] Right now this leaves the node modules setup cache in an incorrect
state (removes the symlinks, but doesn't update the setup cache)
- [ ] ~~Handle code cache and other sqlite caches?~~
Change:
Supported --open flag with deno serve -> (deno serve --open
somescript.ts/js).
The action that takes place is openning the browser on the address that
the server is running on.
Signed-off-by: HasanAlrimawi <141642411+HasanAlrimawi@users.noreply.github.com>
This commit adds two env vars:
- "DENO_CACHE_DB_MODE"
- "DENO_KV_DB_MODE"
Both of these env vars accept either "disk" or "memory" values and
control the modes of backing databases for Web Cache API and
"Deno.openKv()" API.
By default both APIs use disk backed DBs, but they can be changed to use
in-memory
DB, making them effectively ephemeral.
This commit adds "deno_features" crate that contains definitions of all
unstable features in Deno.
Based on these definitions, both Rust and JS code is generated ensuring
that the two are always in sync.
In addition some of flag handling was rewritten to use the generated
definitions, instead of hand rolling these flag definitions.
---------
Co-authored-by: snek <snek@deno.com>
This PR updates the behavior of `deno test --coverage` option. Now if
`--coverage` option is specified, `deno test` command automatically
shows summary report in the terminal, and generates the lcov report in
`$coverage_dir/lcov.info` and html report in `$coverage_dir/html/`
This change also adds `--coverage-raw-data-only` flag, which prevents
the above reports generated, instead only generates the raw json
coverage data (which is the same as current behavior)
Fixes#27264. Fixes https://github.com/denoland/deno/issues/28161.
Currently the new lockfile version is gated behind an unstable flag
(`--unstable-lockfile-v5`) until the next minor release, where it will
become the default.
The main motivation here is that it improves startup performance when
using the global cache or `--node-modules-dir=auto`.
In a create-next-app project, running an empty file:
```
❯ hyperfine --warmup 25 -N --setup "rm -f deno.lock" "deno run --node-modules-dir=auto -A empty.js" "deno-this-pr run --node-modules-dir=auto -A empty.js" "deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js" "deno run --node-modules-dir=manual -A empty.js" "deno-this-pr run --node-modules-dir=manual -A empty.js"
Benchmark 1: deno run --node-modules-dir=auto -A empty.js
Time (mean ± σ): 247.6 ms ± 1.7 ms [User: 228.7 ms, System: 19.0 ms]
Range (min … max): 245.5 ms … 251.5 ms 12 runs
Benchmark 2: deno-this-pr run --node-modules-dir=auto -A empty.js
Time (mean ± σ): 169.8 ms ± 1.0 ms [User: 152.9 ms, System: 17.9 ms]
Range (min … max): 168.9 ms … 172.5 ms 17 runs
Benchmark 3: deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js
Time (mean ± σ): 16.2 ms ± 0.7 ms [User: 12.3 ms, System: 5.7 ms]
Range (min … max): 15.2 ms … 19.2 ms 185 runs
Benchmark 4: deno run --node-modules-dir=manual -A empty.js
Time (mean ± σ): 16.2 ms ± 0.8 ms [User: 11.6 ms, System: 5.5 ms]
Range (min … max): 14.9 ms … 19.7 ms 187 runs
Benchmark 5: deno-this-pr run --node-modules-dir=manual -A empty.js
Time (mean ± σ): 16.0 ms ± 0.9 ms [User: 12.0 ms, System: 5.5 ms]
Range (min … max): 14.8 ms … 22.3 ms 190 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Summary
deno-this-pr run --node-modules-dir=manual -A empty.js ran
1.01 ± 0.08 times faster than deno run --node-modules-dir=manual -A empty.js
1.01 ± 0.07 times faster than deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js
10.64 ± 0.60 times faster than deno-this-pr run --node-modules-dir=auto -A empty.js
15.51 ± 0.88 times faster than deno run --node-modules-dir=auto -A empty.js
```
When using the new lockfile version, this leads to a 15.5x faster
startup time compared to the current deno version.
Install times benefit as well, though to a lesser degree.
`deno install` on a create-next-app project, with everything cached
(just setting up node_modules from scratch):
```
❯ hyperfine --warmup 5 -N --prepare "rm -rf node_modules" --setup "rm -rf deno.lock" "deno i" "deno-this-pr i" "deno-this-pr i --unstable-lockfile-v5"
Benchmark 1: deno i
Time (mean ± σ): 464.4 ms ± 8.8 ms [User: 227.7 ms, System: 217.3 ms]
Range (min … max): 452.6 ms … 478.3 ms 10 runs
Benchmark 2: deno-this-pr i
Time (mean ± σ): 368.8 ms ± 22.0 ms [User: 150.8 ms, System: 198.1 ms]
Range (min … max): 344.8 ms … 397.6 ms 10 runs
Benchmark 3: deno-this-pr i --unstable-lockfile-v5
Time (mean ± σ): 211.9 ms ± 17.1 ms [User: 7.1 ms, System: 177.2 ms]
Range (min … max): 191.3 ms … 233.4 ms 10 runs
Summary
deno-this-pr i --unstable-lockfile-v5 ran
1.74 ± 0.17 times faster than deno-this-pr i
2.19 ± 0.18 times faster than deno i
```
With lockfile v5, a 2.19x faster install time compared to the current
deno.
Deno.serve `Request` abort signals are aborted by default even when it
is finished successfully. This PR gates this behavior behind the
"legacy_abort" which is the default right now.
Turning the `no_legacy_abort` runtime option on is a **breaking change**
and will only abort request signals when there is a failure, thereby
cannot be used to determine if the request finished. This aligns with
`fetch` API.
Ref https://github.com/denoland/deno/issues/27005
Updates to use rust 1.85. Doesn't move to the 2024 edition, as that's a
fair bit more involved.
A nice side benefit is that the new rustc version seems to lead to a
slight reduction in binary size (at least on mac):
```
FILE SIZE
--------------
+4.3% +102Ki __DATA_CONST,__const
[NEW] +69.3Ki __TEXT,__literals
[NEW] +68.5Ki Rebase Info
+5.0% +39.9Ki __TEXT,__unwind_info
+57% +8.85Ki [__TEXT]
[NEW] +8.59Ki Lazy Binding Info
[NEW] +5.16Ki __TEXT,__stub_helper
[NEW] +3.58Ki Export Info
[NEW] +3.42Ki __DATA,__la_symbol_ptr
-0.1% -726 [12 Others]
-21.4% -3.10Ki [__DATA_CONST]
-95.8% -3.39Ki __DATA_CONST,__got
-20.9% -3.43Ki [__DATA]
-0.5% -4.52Ki Code Signature
-100.0% -11.6Ki [__LINKEDIT]
-1.0% -43.5Ki Symbol Table
-1.6% -44.0Ki __TEXT,__gcc_except_tab
-0.2% -48.1Ki __TEXT,__const
-3.3% -78.6Ki __TEXT,__eh_frame
-0.7% -320Ki __TEXT,__text
-1.5% -334Ki String Table
-0.5% -586Ki TOTAL
```
Supports `deno task --frozen=false some_task`, which is necessary
because with `"nodeModulesDir": "auto"` or `"global"` (the default with
no package.json) we do an auto-install of npm packages.
Closes https://github.com/denoland/deno/issues/28070
In particular this helps startup of the TSC isolate because
`00_typescript.js` can use the code cache.
Overall, this offsets a fair bit of the hit we took when we removed the
TSC snapshot.
```
❯ hyperfine --warmup 5 -p "rm -rf ~/Library/Caches/deno/check_cache_v2" "./deno-this-pr check main.ts" "./deno-no-snapshot check main.ts" "./deno-with-snapshot check main.ts"
Benchmark 1: ../../deno/target/release-lite/deno check main.ts
Time (mean ± σ): 145.7 ms ± 3.6 ms [User: 347.6 ms, System: 36.9 ms]
Range (min … max): 142.2 ms … 155.9 ms 19 runs
Benchmark 2: ./deno-no-snapshot check main.ts
Time (mean ± σ): 195.4 ms ± 3.3 ms [User: 397.7 ms, System: 34.9 ms]
Range (min … max): 192.1 ms … 206.0 ms 15 runs
Benchmark 3: ./deno-with-snapshot check main.ts
Time (mean ± σ): 109.0 ms ± 2.2 ms [User: 155.9 ms, System: 19.3 ms]
Range (min … max): 106.5 ms … 118.0 ms 26 runs
Summary
./deno-with-snapshot check main.ts ran
1.34 ± 0.04 times faster than ./deno-this-pr check main.ts
1.79 ± 0.05 times faster than ./deno-no-snapshot check main.ts
```
