This change configures V8 isolates to respect memory limits imposed by
cgroups on Linux.
It adds support for detecting both cgroups v1 and v2 memory limits,
enabling Deno to properly adapt to containerized environments with
memory constraints. When cgroups information is unavailable or not
applicable, it falls back to using the system's total memory as before.
Closes#29077
## Test
For testing, I created a ubuntu VM with 1Gi memory. Within this VM, set
up a cgroup with 512Mi memory limit, then ran the following script to
see how much heap size limit the V8 isolate had.
```js
import * as v8 from "node:v8";
console.log(v8.getHeapStatistics());
```
### Ubuntu 20.04
In this version of ubuntu, hybrid mode is enabled by default.
```
$ cat /proc/self/cgroup
12:rdma:/
11:blkio:/user.slice
10:devices:/user.slice
9:cpu,cpuacct:/user.slice
8:pids:/user.slice/user-1000.slice/session-3.scope
7:memory:/user.slice/user-1000.slice/session-3.scope
6:perf_event:/
5:freezer:/
4:net_cls,net_prio:/
3:hugetlb:/
2:cpuset:/
1:name=systemd:/user.slice/user-1000.slice/session-3.scope
0::/user.slice/user-1000.slice/session-3.scope
```
Create a new cgroup with 512Mi memory limit and run the above script in
this cgroup:
```
$ sudo cgcreate -g memory:/mygroup
$ sudo cgset -r memory.limit_in_bytes=$((512 * 1024 * 1024)) mygroup
$ sudo cgexec -g memory:mygroup ./deno run main.mjs
{
total_heap_size: 7745536,
total_heap_size_executable: 0,
total_physical_size: 7090176,
total_available_size: 266348216,
used_heap_size: 6276752,
heap_size_limit: 271581184,
malloced_memory: 303200,
peak_malloced_memory: 140456,
does_zap_garbage: 0,
number_of_native_contexts: 1,
number_of_detached_contexts: 0,
total_global_handles_size: 24576,
used_global_handles_size: 22432,
external_memory: 3232012
}
```
This indicates that the isolate was informed of cgroup-constrained
memory limit (512Mi) and hence got ~270M heap limit.
### Ubuntu 22.04
In this version of ubuntu, cgroup v2 is used.
```
$ cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-3.scope
```
Run the above script using `systemd-run`:
```
$ sudo systemd-run --property=MemoryMax=512M --pty bash -c '/home/ubuntu/deno run /home/ubuntu/main.mjs'
{
total_heap_size: 7745536,
total_heap_size_executable: 0,
total_physical_size: 7090176,
total_available_size: 266348184,
used_heap_size: 6276784,
heap_size_limit: 271581184,
malloced_memory: 303200,
peak_malloced_memory: 140456,
does_zap_garbage: 0,
number_of_native_contexts: 1,
number_of_detached_contexts: 0,
total_global_handles_size: 24576,
used_global_handles_size: 22432,
external_memory: 3232012
}
```
Again the isolate got ~270M heap limit properly.
Note that it should have had bigger heap limit if the entire system
memory, i.e. 1Gi, had been passed to V8. In fact, if we run the same
script outside the cgroup, it does display larger `heap_size_limit` like
below:
```
$ ./deno run main.mjs
{
total_heap_size: 7745536,
total_heap_size_executable: 0,
total_physical_size: 7090176,
total_available_size: 546580152,
used_heap_size: 6276752,
heap_size_limit: 551813120,
malloced_memory: 303200,
peak_malloced_memory: 140456,
does_zap_garbage: 0,
number_of_native_contexts: 1,
number_of_detached_contexts: 0,
total_global_handles_size: 24576,
used_global_handles_size: 22432,
external_memory: 3232012
}
```
---------
Signed-off-by: Yusuke Tanaka <wing0920@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
implement lazy(?) mode. an unconfigured jsruntime is created if
DENO_UNSTABLE_CONTROL_SOCK is present, and later passed into deno_runtime to be
configured and used.
Closes https://github.com/denoland/deno/issues/27229.
TODO:
- [x] Tests
- [x] Make some changes to `deno_cache_dir` so we can get the paths for
the local http cache
- [x] Right now this leaves the node modules setup cache in an incorrect
state (removes the symlinks, but doesn't update the setup cache)
- [ ] ~~Handle code cache and other sqlite caches?~~
This commit adds "deno_features" crate that contains definitions of all
unstable features in Deno.
Based on these definitions, both Rust and JS code is generated ensuring
that the two are always in sync.
In addition some of flag handling was rewritten to use the generated
definitions, instead of hand rolling these flag definitions.
---------
Co-authored-by: snek <snek@deno.com>
#28646
Edit:
Hi apologies for the delayed explanation @marvinhagemeister.
The reason the error message wasn't appearing was due to the
initialization order. The logging system was being initialized after the
telemetry system. This created a problem: if telemetry failed during its
own initialization, the logging system wasn't yet ready to capture and
report the error.
My solution was to simply move the log initialization to occur before
the telemetry initialization. This resolves the issue, ensuring any
telemetry setup errors are correctly logged.
Basically just update deno_lockfile, deno_npm, and eszip, and then adapt
to those changes. The main changes were the removal of the lockfile v4
resolution snapshot loading, and a terser formatting for the `os` and
`cpu` fields in the lockfile.
Fixes two issues:
- If a cached packument was out of date and missing a version from the
lockfile, we would fail. Instead we should try again with a forced
re-fetch
- We weren't threading through the workspace patch packages correctly
This PR updates the behavior of `deno test --coverage` option. Now if
`--coverage` option is specified, `deno test` command automatically
shows summary report in the terminal, and generates the lcov report in
`$coverage_dir/lcov.info` and html report in `$coverage_dir/html/`
This change also adds `--coverage-raw-data-only` flag, which prevents
the above reports generated, instead only generates the raw json
coverage data (which is the same as current behavior)
This commit enabled [explicit resource management]
(https://github.com/tc39/proposal-explicit-resource-management)
proposal for JavaScript code.
This is done by upgrading `deno_ast` to a version that no longer
transpiles TS files with `using` keyword, and instead enables
a V8 flag that provides native support.
Closes https://github.com/denoland/deno/issues/20821
---------
Co-authored-by: Luca Casonato <hello@lcas.dev>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Fixes#27264. Fixes https://github.com/denoland/deno/issues/28161.
Currently the new lockfile version is gated behind an unstable flag
(`--unstable-lockfile-v5`) until the next minor release, where it will
become the default.
The main motivation here is that it improves startup performance when
using the global cache or `--node-modules-dir=auto`.
In a create-next-app project, running an empty file:
```
❯ hyperfine --warmup 25 -N --setup "rm -f deno.lock" "deno run --node-modules-dir=auto -A empty.js" "deno-this-pr run --node-modules-dir=auto -A empty.js" "deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js" "deno run --node-modules-dir=manual -A empty.js" "deno-this-pr run --node-modules-dir=manual -A empty.js"
Benchmark 1: deno run --node-modules-dir=auto -A empty.js
Time (mean ± σ): 247.6 ms ± 1.7 ms [User: 228.7 ms, System: 19.0 ms]
Range (min … max): 245.5 ms … 251.5 ms 12 runs
Benchmark 2: deno-this-pr run --node-modules-dir=auto -A empty.js
Time (mean ± σ): 169.8 ms ± 1.0 ms [User: 152.9 ms, System: 17.9 ms]
Range (min … max): 168.9 ms … 172.5 ms 17 runs
Benchmark 3: deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js
Time (mean ± σ): 16.2 ms ± 0.7 ms [User: 12.3 ms, System: 5.7 ms]
Range (min … max): 15.2 ms … 19.2 ms 185 runs
Benchmark 4: deno run --node-modules-dir=manual -A empty.js
Time (mean ± σ): 16.2 ms ± 0.8 ms [User: 11.6 ms, System: 5.5 ms]
Range (min … max): 14.9 ms … 19.7 ms 187 runs
Benchmark 5: deno-this-pr run --node-modules-dir=manual -A empty.js
Time (mean ± σ): 16.0 ms ± 0.9 ms [User: 12.0 ms, System: 5.5 ms]
Range (min … max): 14.8 ms … 22.3 ms 190 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Summary
deno-this-pr run --node-modules-dir=manual -A empty.js ran
1.01 ± 0.08 times faster than deno run --node-modules-dir=manual -A empty.js
1.01 ± 0.07 times faster than deno-this-pr run --node-modules-dir=auto --unstable-lockfile-v5 empty.js
10.64 ± 0.60 times faster than deno-this-pr run --node-modules-dir=auto -A empty.js
15.51 ± 0.88 times faster than deno run --node-modules-dir=auto -A empty.js
```
When using the new lockfile version, this leads to a 15.5x faster
startup time compared to the current deno version.
Install times benefit as well, though to a lesser degree.
`deno install` on a create-next-app project, with everything cached
(just setting up node_modules from scratch):
```
❯ hyperfine --warmup 5 -N --prepare "rm -rf node_modules" --setup "rm -rf deno.lock" "deno i" "deno-this-pr i" "deno-this-pr i --unstable-lockfile-v5"
Benchmark 1: deno i
Time (mean ± σ): 464.4 ms ± 8.8 ms [User: 227.7 ms, System: 217.3 ms]
Range (min … max): 452.6 ms … 478.3 ms 10 runs
Benchmark 2: deno-this-pr i
Time (mean ± σ): 368.8 ms ± 22.0 ms [User: 150.8 ms, System: 198.1 ms]
Range (min … max): 344.8 ms … 397.6 ms 10 runs
Benchmark 3: deno-this-pr i --unstable-lockfile-v5
Time (mean ± σ): 211.9 ms ± 17.1 ms [User: 7.1 ms, System: 177.2 ms]
Range (min … max): 191.3 ms … 233.4 ms 10 runs
Summary
deno-this-pr i --unstable-lockfile-v5 ran
1.74 ± 0.17 times faster than deno-this-pr i
2.19 ± 0.18 times faster than deno i
```
With lockfile v5, a 2.19x faster install time compared to the current
deno.
- upgrade v8 to 13.4
- turbocall conversion for arraybuffers is now much more complex, so use
cranelift
- misc updates for deprecated fns
- v8 default stack size is too small now, causing stack overflow
exceptions in some tests
- add syscall shim to support new syscall in old sysroot
This commit changes the TS host implementation
in the LSP to use the same snapshot as the runtime worker
and web worker use.
This is due to upcoming V8 upgrade that might require
that all isolates in the same process use the exact same
snapshot.
Allows easily constructing a `DenoResolver` using the exact same logic
that we use in the CLI (useful for dnt and for external bundlers). This
code is then used in the CLI to ensure the logic is always up-to-date.
```rs
use std::rc::Rc;
use deno_resolver:🏭:ResolverFactory;
use deno_resolver:🏭:WorkspaceFactory;
use sys_traits::impls::RealSys;
let sys = RealSys;
let cwd = sys.env_current_dir()?;
let workspace_factory = Rc::new(WorkspaceFactory::new(sys, cwd, Default::default()));
let resolver_factory = ResolverFactory::new(workspace_factory.clone(), Default::default());
let deno_resolver = resolver_factory.deno_resolver().await?;
```
This slightly degrades the performance of CJS export analysis on
subsequent runs because I changed it to no longer cache in the DENO_DIR
with this PR (denort now properly has no idea about the DENO_DIR). We'll
have to change it to embed this data in the binary and that will also
allow us to get rid of swc in denort (will do that in a follow-up PR).
split up otel config into user configurable and runtime configurable
parts. user configurable part is now set via env vars parsed according
to the otel spec. otel is now enabled via `OTEL_DENO=true`, and
`--unstable-otel` only acts as a guard.
Fixes: https://github.com/denoland/deno/issues/27273
This commit adds support for `deno init --npm <package>`.
Running this will actually call to `npm:create-<package>` package that
is equivalent to running `npm create <package>`.
User will be prompted if they want to allow all permissions and
lifecycle scripts to be executed.
Closes https://github.com/denoland/deno/issues/26461
---------
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
This commit adds workspace support to "deno taks".
Two new flags were added:
- "--recursive" - allows to run a specified task in workspace members,
eg. "deno task --recursive dev"
- "--filter" - allows to run a specified task only in specific workspace members,
eg. "deno task --recursive --filter 'client/*' dev"
"--filter" flag implies "--recursive" flag.
Closes https://github.com/denoland/deno/issues/24991
---------
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Signed-off-by: David Sherret <dsherret@users.noreply.github.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Nathan Whitaker <17734409+nathanwhit@users.noreply.github.com>
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
Co-authored-by: David Sherret <dsherret@gmail.com>
Closes#20487
Currently spelled
```
deno outdated
```
and
```
deno outdated --update
```
Works across package.json and deno.json, and in workspaces.
There's a bit of duplicated code, I'll refactor to reduce this in follow
ups
## Currently supported:
### Printing outdated deps (current output below which basically mimics
pnpm, but requesting feedback / suggestions)
```
deno outdated
```
### Updating deps
semver compatible:
```
deno outdated --update
```
latest:
```
deno outdated --latest
```
current output is basic, again would love suggestions
#### Filters
```
deno outdated --update "@std/*"
deno outdated --update --latest "@std/* "!@std/fmt"
```
#### Update to specific versions
```
deno outdated --update @std/fmt@1.0.2 @std/cli@^1.0.3
```
### Include all workspace members
```
deno outdated --recursive
deno outdated --update --recursive
```
## Future work
- interactive update
- update deps in js/ts files
- better support for transitive deps
Known issues (to be fixed in follow ups):
- If no top level dependencies have changed, we won't update transitive
deps (even if they could be updated)
- Can't filter transitive deps, or update them to specific versions
## TODO (in this PR):
- ~~spec tests for filters~~
- ~~spec test for mixed workspace (have tested manually)~~
- tweak output
- suggestion when you try `deno update`
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds `--eval` flag to `deno task` subcommand.
This flag allows to evaluate provided "task name" as a task itself,
effectively allowing to use `deno_task_shell` from the command line.
Also fixes shebang parsing for `node_modules/.bin/` entries to handle
`#!/usr/bin/node -S node` in addition to `#!/usr/bin/node node`.
Closes https://github.com/denoland/deno/issues/26918
Improving the breadth of collected data, and ensuring that the collected
data is more likely to be successfully reported.
- Use `log` crate in more places
- Hook up `log` crate to otel
- Switch to process-wide otel processors
- Handle places that use `process::exit`
Also adds a more robust testing framework, with a deterministic tracing
setting.
Refs: https://github.com/denoland/deno/issues/26852
