rcl/docs/using_ninja.md

# Using Ninja

As we saw [the previous chapter][generating-files], <abbr>RCL</abbr> can
abstract away repetition in files like GitHub Actions workflows and Kubernetes
manifests, and enable sharing data between tools that do not natively share data.
To do that, we still need to run `rcl` to generate the `.yml`, `.tf.json`,
`.json`, and `.toml` files that can be consumed by existing tools.

Updating those generated files is the job of a build system. In the previous
chapter we saw a lightweight solution that is built into <abbr>RCL</abbr>:
[`rcl build`](rcl_build.md). Using `rcl build` is easy and avoids the need to
bring in external tools, but it has two main limitations:

 * **It can only evaluate expressions, it does not call external programs.**
   Suppose one of the <abbr>RCL</abbr> files imports a <abbr>JSON</abbr> file
   that is generated by an external program. Then now we are back to having to
   run multiple commands to update all generated files.
 * **It rewrites files even when inputs did not change.**
   Unless your configuration is truly massive, <abbr>RCL</abbr> is probably fast
   enough that regenerating all files is not a problem, but it can still have
   downsides. For example, if you use the generated files in the next stage of a
   build system, the updated mtimes may cause unnecessary rebuilds, and rewrites
   break reflink sharing on copy-on-write file systems.

To get around those limitations, we can to switch to a proper build system,
such as Make, Bazel, or Ninja.

[generating-files]: generating_files.md

## Make

Updating generated files when inputs change is the role of a build tool.
We could use [Make][gnumake] and write a makefile:

```make
policies.json: policies.rcl
    rcl evaluate --format=json --output=$@ $<
```

Aside from the somewhat arcane syntax, this makefile has one big problem. If
`policies.rcl` imports an <abbr>RCL</abbr> file, say `users.rcl`, then
Make will not rebuild `policies.json` when we change `users.rcl`, because
we haven’t specified the dependency in the makefile. Manually listing all
transitive dependencies is tedious and prone to go out of date, which is why
Make [can automatically remake makefiles][make-dep]. Unfortunately the syntax
for achieving this is so vexing[^1] that it’s hard to seriously consider Make
when clearer alternatives exist.

[^1]: The <abbr>GNU</abbr> Make manual [recommends][make-dep] a pattern rule
      for generating prerequisite makefiles that contains this snippet:
      `$(CMD) $< > $@.$$$$; \ sed 's,\($*\)\.o[ :]*,\1.o $@ : ,g' < $@.$$$$ > $@;`.
      While this is an excellent demonstration of the Unix philosophy, newer
      build systems like Ninja feature significantly more readable build files.

[Ninja][ninja-build] is a different build tool that can solve this problem by
reading transitive dependencies from a [depfile][depfile], and [<abbr>RCL</abbr>
can write such a depfile][odepfile]. In the remainder of this chapter, we’ll
explore using Ninja as the build tool.

[make-dep]:    https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html
[gnumake]:     https://www.gnu.org/software/make/manual/html_node/index.html
[ninja-build]: https://ninja-build.org/
[depfile]:     https://ninja-build.org/manual.html#_depfile
[odepfile]:    rcl_evaluate.md#-output-depfile-depfile

## Ninja

[Ninja][ninja-build] is a fast and flexible build tool, but its build files are
low-level and intended to be _generated_, not written by hand. Let’s write one
by hand anyway, to better understand what we are working with.

In a Ninja file, we first define [a rule][ninja-rule] that specifies how
to invoke a program. This is also where we can tell Ninja to use a
[depfile][depfile].

```ninja
rule rcl
  description = Generating $out
  command = rcl eval --color=ansi --format=$format --output=$out --output-depfile=$out.d $in
  depfile = $out.d
  deps = gcc
```

Here `$in`, `$out`, and `$format` are variables. Ninja itself sets `$in` and
`$out`, and `$format` is one that we define because it varies per target. The
`deps = gcc` line is not required, but it makes Ninja store the depedency
information in `.ninja_deps` and then delete the generated depfile, instead of
reading it on demand. This is nice to keep the repository clean.

Next, we add a [build statement][ninja-stmt] that specifies how to build a file:

```ninja
build policies.json: rcl policies.rcl
  format = json
```

This is enough for Ninja to work. Save the file to `build.ninja` and then build
`policies.json`:

```console
$ ninja
[1/1] Generating policies.json

$ ninja
ninja: no work to do.

$ touch users.rcl
$ ninja
[1/1] Generating policies.json
```

[ninja-rule]: https://ninja-build.org/manual.html#_rules
[ninja-stmt]: https://ninja-build.org/manual.html#_build_statements

## Generating Ninja files

Okay, so we can write a Ninja file by hand, it’s quite readable even. But at some
point, we’re going to end up with lots of similar build statements, and wish we
had a way to abstract that. If only we had a tool that could abstract away this
repetition …

We can write a `ninja.rcl` that evaluates to a Ninja build file like so:

```rcl
#!/usr/bin/env -S rcl evaluate --output=build.ninja --format=raw

let ninja_prelude =
  """
  rule rcl
    description = Generating $out
    command = rcl eval --color=ansi --format=$format --output=$out --output-depfile=$out.d $in
    depfile = $out.d
    deps = gcc
  """;

let build_json = basename =>
  f"""
  build {basename}.json: rcl {basename}.rcl
    format = json
  """;

// File basenames that we want to generate build rules for.
// This is the part we need to edit when we add more files.
let basenames_json = [
  "policies",
];

let sections = [
  ninja_prelude,
  for basename in basenames_json: build_json(basename),
];

sections.join("\n")
```

Now we can generate the same build file that we previously wrote by hand, and
when we add more json target files, we only need to add one string to the list.
By adding a `#!`-line and making the file executable, we can even record how the
Ninja file is generated. Unfortunately, even with the `#!`-line we are back to
multiple build steps: first `./ninja.rcl`, and then `ninja`. Can we do better?

For bootstrapping `build.ninja`, that will always need a manual step. But after
we run `./ninja.rcl` once, Ninja can keep `build.ninja` up to date for us. We
just need to list it as a build target:

```rcl
let sections = [
  ninja_prelude,
  """
  build build.ninja: rcl ninja.rcl
    format = raw
  """,
  for basename in basenames_json: build_json(basename),
];
```

## Dynamic targets

Now that we generate our `build.nina` from `ninja.rcl`, we can import
<abbr>RCL</abbr> documents to dynamically create build tagets. For instance,
we can leverage [`rcl query`](rcl_query.md) to build all the keys of a document
`manifests.rcl` as separate files. We could do that as follows:

```rcl
#!/usr/bin/env -S rcl evaluate --format=raw --output=build.ninja

let command = [
  "rcl",
  "query",
  "--color=ansi",
  "--format=$format",
  "--output=$out",
  "--output-depfile=$out.d",
  "$in",
  "$query",
];
let ninja_prelude =
  f"""
  rule rcl
    description = Generating $out
    command = {command.join(" ")}
    depfile = $out.d
    deps = gcc
  """;

let build_raw = (target, src) =>
  f"""
  build {target}: rcl {src}
    query = input
    format = raw
  """;

let build_json_query = (target, src, query) =>
  f"""
  build {target}: rcl {src}
    format = json
    query = {query}
  """;

let manifests = import "manifests.rcl";

let sections = [
  ninja_prelude,
  build_raw("build.ninja", "ninja.rcl"),
  for key, _ in manifests:
  // Warning, this assumes that the key is both a valid filename
  // and RCL expression. Currently no built-in functions exist for
  // validating this.
  build_json_query(f"{key}.yml", "manifests.rcl", f"input.{key}"),
];

sections.join("\n")
```

**Warning:** Generating targets dynamically is powerful, but also a sure way
to make your build process intractable quickly! Use sparingly and with good
judgement!

## Conclusion

RCL enables sharing configuration between systems that do not natively share
data. To do so, you will likely need to generate files. Keeping those files up
to date is the job of a build tool. In this chapter we have seen how to use the
Ninja build tool, and how to use <abbr>RCL</abbr> to write Ninja build files.