This is a simple computation over the JSON structure of the schema,
so it doesn’t need pulling in another programming language and a
dependency on an external C program to interpret it.
This removes an external tool dependency, skips the complexities
of subprocessing, and avoids having to poke a hole in the macOS
Nix sandbox because of `reqwest` trying to read the system’s TLS
configuration.
Note that despite the `Cargo.lock` getting bigger, this is actually a
net reduction of dependencies, as Taplo is written in Rust and pulls
in not only `jsonschema` but much more besides.
Extracts the `"default"` values from the schema and creates a synthetic
TOML file holding all the defaults according to the schema. This is done
through some `jq` magic and is not perfect but rather a best effort.
If `jq` is not available, the test is skipped; in CI `jq` is required.
The test then run `jj config get` in the test env for each key in that
defaults file and compares the resulting value with the schema default.
For a few keys, there are actually no defaults known to `jj config get`,
because they are hard-coded or dynamic. These exceptions are intercepted
and explained in the test.
Latest rustc nightlies seem to have broken incremental compilation [0].
I apparently have the worst luck in this department, as I've gotten multiple
ICEs within the last few hours alone. So, I'm putting future me out of
her misery.
[0]: https://github.com/rust-lang/rust/issues/139110
This uses `zlib-rs`, a native Rust library that is comparable in
performance to `zlib-ng`. Since there’s no complicated C build
and gitoxide only has one hashing backend now, this lets us drop our
`packaging` feature without adding any awkward build requirements.
`zlib-rs` is generally faster at decompression than
`zlib-ng`, and faster at compression on levels 6 and 9; see
<https://trifectatech.org/blog/zlib-rs-is-faster-than-c/>
for details.
I couldn’t get reliable‐looking benchmark results out of my
temperamental laptop; `hyperfine` seemed to think that some random
`jj` workloads I tested might be slightly slower than with `zlib-ng`,
but it wasn’t unambiguously distinguishable from noise, so I’d
like to see measurements from others.
It’s certainly a lot faster than the previous default, and I
think it’s likely that `zlib-rs` will continue to get faster
and that it’s more than worth avoiding the headaches of a native
library with a CMake build dependency. (Though on the other hand,
if distributions move in the direction of shipping `zlib-ng` by
default, maybe there will be more motivation to make `libz-ng-sys`
support system libraries.)
Overall, building in the test profile should significantly speed up
the overall build pipeline because so many less cycles are spent (on
GHA runners that are certainly at high load). The goal here is to help
reduce CI flake outs due to things timing out; I suspect part of the
problem may be a lot of the ~15 minute time limit being used up just
compiling things.
This is a partial revert of b714592952, which removed this previous
override of the Flake `checks`.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Some of our builds have been timing out, and one angle I want to look at
is whether any tests are hanging. Nextest can help us keep track of slow
tests in CI where the underlying hosts will have significantly higher
load. In general this should also help keep our tests healthy, IMO.
I don't see any reason why any existing test should take over 20
seconds, but it should at least help control for really slow runners
that have huge latency spikes. we can start there and see how it goes
in practice.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Currently, the Git subprocess tests only work on Linux due to a default
path being used for the `git` executable when `$PATH` is unset. This can
break if Git isn't installed at the expected path. Also, I believe it is
currently necessary to set the `$TEST_GIT_EXECUTABLE_PATH` environment
variable on Windows for tests to pass. Instead, we should use the user's
`$PATH` to locate the `git` executable, as well as any other executables
that are needed. This also makes `$TEST_GIT_EXECUTABLE_PATH` no longer
necessary, so it can be removed.
bacon 3.6.0 -> 3.8.0, shows output with nextest 0.9.86
https://github.com/Canop/bacon/issues/280
Add python3 to the devshell to be able to run tools installed with uv
The `RUSTFLAGS` enviroment variable needs to be set in a `shellHook` to
allow the command to be interpreted by the shell. Otherwise, it will
just be passed as a string into the `rustc` binary:
```
cargo nextest run --workspace
error: failed to run `rustc` to learn about target-specific information
Caused by:
process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Zthreads=0 -C 'link-arg=--ld-path=$(unset' 'DEVELOPER_DIR;' /usr/bin/xcrun --find 'ld)' -C link-arg=-ld_new --target aarch64-apple-darwin --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 1)
--- stderr
error: Unrecognized option: 'find'
```
`zstd` is only used to write files in the native backend right now. For now,
jettison it, to unbundle some C code that we don't really need.
(Ideally, a future compression library would be pure Rust, but we'll cross that
bridge when we get to it...)
Signed-off-by: Austin Seipp <aseipp@pobox.com>
I noticed miniz_oxide appears in perf samples. While miniz_oxide is safer, I
think zlib-ng is pretty reliable.
https://docs.rs/gix/latest/gix/#performance
libz-ng-sys is downgraded to 1.1.16 due to the Windows linking issue. The
benchmark result is obtained with libz-ng-sys 1.1.20.
https://github.com/rust-lang/libz-sys/issues/225
```
% hyperfine --sort command --warmup 3 --runs 10 -L bin jj-0,jj-1 \
'target/release-with-debug/{bin} --ignore-working-copy log README.md'
Benchmark 1: target/release-with-debug/jj-0 ..
Time (mean ± σ): 256.6 ms ± 4.3 ms [User: 214.1 ms, System: 38.6 ms]
Range (min … max): 245.4 ms … 261.2 ms 10 runs
Benchmark 2: target/release-with-debug/jj-1 ..
Time (mean ± σ): 223.0 ms ± 4.2 ms [User: 174.7 ms, System: 44.4 ms]
Range (min … max): 212.4 ms … 225.8 ms 10 runs
```
As reported by @lilyball on Discord, the recent nixpkgs update this week brought
with it an entirely new Darwin stdenv, which had a behavioral change versus
the prior one, causing builds inside the developer shell to fail, as the new
super-duper parallel linker could not be used.
However, rather than giving up and throwing away *billions* of precious CPU
cycles, @emilazy and @lilyball instead doubled down and diagnosed the issue. The
result is a new impenetrable line of Nix code, which is fully explained by the
comments within.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Our docs are built with MkDocs, which requires Python and several deps.
Previously those deps were managed with Poetry, which is also written in Python.
This commit replaces Poetry with `uv`, a Rust-based Python
project/package manager, and thus removes several steps from the docs
build process.
Before:
<install Python>
<install pipx>
pipx install poetry
poetry install
poetry run -- mkdocs serve
After:
<install uv>
uv run mkdocs serve
The most relevant part (and the reason for this change) is the addition
of the `mainProgram` attribute. This allows getting the executable name
from inside nix expressions with ease:
```
# before
lib.getExe' jujutsu "jj"
# or
"${jujutsu}/bin/jj"
```
```
# now
lib.getExe jujutsu
```
In practice, `packages` sets the value for `nativeBuildInputs`. This has the following advantages:
- `buildInputs` is for runtime dependencies. `nativeBuildInputs` is for
build dependencies (this usually not an issue unless cross compiling)
- `shellHook`s will actually run, allowing packages to safely append to
variables like `XDG_DATA_DIRS`
- It's more idiomatic and how it's recommended to be used in the docs
See the included comment. This helps avoid a large dependency on the rustc
toolchain and helps reduce closure size for Nix users.
Partial revert of 749481e552.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
The new parallel macOS linker reduces link time for the debug `jj` binary from
3s to 0.7s on my M2 Macbook Air, which is a significant reduction for nearly
no cost at all. This only assumes that you have a new enough Xcode environment
as your default (where `/usr/bin/ld` resides.)
This change requires Sonoma and Xcode 15, but in theory I think we could target
a lower macOS SDK version in order to produce binaries that are more backwards
compatible, so the only real cost is that developers who also use Nix would
require Sonoma.
This is already included with the `ourRustVersion` expression since it includes
a complete Rust toolchain with all extensions.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
This cuts the size of a final debug binary in half (~250MiB -> 127MiB)
in my dev shell, and saves about 15% of total `target/` directory size
when a full build from scratch happens (2GiB -> 1.7GiB). Let's take an
easy free win.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Only active within `devShell`; the default `.#jujutsu` package is unaffected
by this change. This somewhat increases CPU utilization and has a marginal
improvement on my local compile times, but it's also nice if you have to
recompile from scratch too.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Change-Id: I07ab28991843ca3723185569db5f38f2ed076875
This is a change that does affect the nix built package, and is tested as part
of the CI Matrix, so there is now an actively maintained (by me) nightly build
that can be checked for regressions.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Change-Id: If9a134de7e1d496d47936259c59f73c4a62a341f
When the `jj-proc-macros` crate was introduced, it triggered an underlying
bug in `nextest`, which is the test harness we use in the Nix build. This is
upstream Nextest bug 267. The long and short of it is that `rustc` fails to
find needed libraries whenever the proc macros are loaded.
This can easily be worked around however, by setting
`DYLD_FALLBACK_LIBRARY_PATH` to an appropriate value in the devShell and in the
`preCheck` phase of the main expression.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
When running the `nix build`, the `buildRustPackage` function -- which builds
the `jj` crates -- calls `cargo build --release` with flags like `HOST_CXX`
set. This is called the `buildPhase`. Then, it runs the `checkPhase`, which
calls `cargo nextest`, in our case. However, it does not set `HOST_CXX`, for
some reason.
The intent of `buildRustPackage` is that the `buildPhase` and `checkPhase`
have their compilation options fully aligned so that they reuse the local cargo
`target/` cache directory, avoiding a full recompilation. However, the lack
of `HOST_CXX` above among others causes a cache miss, and a bunch of cascading
recompilations. The net impact is that we compile all of the codebase once in
`buildPhase`, then again in `checkPhase`, making the Nix CI build 2x slower on
average than the other Linux runners; 2-3 minutes versus 7 minutes, on average.
Instead, re-introduce a 'check' into the Flake directly, which overrides the
`jujustsu` package, but stubs out the `buildPhase`. This means it will only run
`checkPhase`, which is what we want. Then, modify the CI to run `nix flake check`
again, like it used to in the past.
Unfortunately, this doesn't fix the cache miss when running `nix build`
yourself, it recompiles from scratch in both phases still. That needs to be
fixed in the future, but it's tolerable for now.
This reverts most of 71a3045032.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
This matches the behavior of the actual `nix build` more closely, and might also
help Anton, since he was debugging some recompilation issues on his machine,
where `RUSTFLAGS` might have become inconsistent due to VS Code.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
