rcl/docs/generating_files.md

# Generating files

RCL can abstract away repetition, for example in GitHub Actions workflows and
Kubernetes manifests. It also enables sharing configuration between systems
that do not natively share data. For example, you can have one file to define
users and groups, and import it into both your Tailscale configuration and into
your Hashicorp Vault configuration. However, none of these tools natively read
<abbr>RCL</abbr>. You still need to run `rcl` to generate the required `.yml`,
`.tf.json`, `.json`, and `.toml` files that can be consumed by your existing
tools.

If you have just a few files, it’s not so bad to run `rcl` yourself:

    rcl evaluate --format=json policies.rcl --output=policies.json

In a larger repository with many generated files, this gets tedious. It’s not
very discoverable either. Ideally we would have one command to update all
generated files. This is typically the job of a build system. RCL offers two
solutions here:

 * The built-in [`rcl build`](rcl_build.md) command acts as a simple build
   system. It has limitations, but avoids the need to bring in more tools, and
   it suffices for many use cases.
 * For more advanced use cases, <abbr>RCL</abbr> can integrate with external
   build systems through e.g. [depfile support][depfile]. See the
   [Ninja chapter][using-ninja] for an example of how to integrate with the
   [Ninja build system][ninja-build].

In this chapter we will look at the first case, [`rcl build`](rcl_build.md).

[using-ninja]: using_ninja.md
[depfile]:     rcl_evaluate.md#-output-depfile-depfile
[ninja-build]: https://ninja-build.org/

## Using rcl build

To start using `rcl build`, create a file named `build.rcl`:

```rcl
{
  "output.txt": {
    format = "raw",
    contents = "Hello, world.",
  }
}
```

Then, in the same directory, run `rcl build`. This will create the file
`output.txt` next to `build.rcl`, with the following contents:

```
Hello, world.
```

The format corresponds to one of [the output formats supported by
`rcl evaluate`][format], and the `contents` can be an arbitrary value. For
example, we can output as <abbr>JSON</abbr>:

```rcl
{
  "output.json": {
    format = "json",
    contents = {
      is-example = true,
      name = "rcl-build demo",
    }
  }
}
```

This would create `output.json` with the following contents:

```json
{"is-example": true, "name": "rcl-build demo"}
```

[format]: rcl_evaluate.md#-f-format-format

## Adding a banner

The files generated by <abbr>RCL</abbr> may be used in places where it is not
obvious that they are generated. To clarify that a file is generated, we can add
a [banner][banner]. This is a short message that gets prepended to the output.
For example:

```rcl
{
  "Cargo.toml": {
    format = "toml",
    banner = "# This file is generated from build.rcl.",
    contents = { package = { name = "rcl", edition = "2021" } },
  }
}
```

Running `rcl build` then creates the following `Cargo.toml`:

```toml
# This file is generated from build.rcl.
[package]
edition = "2021"
name = "rcl"
```

Inside `build.rcl`, the banner message is enclosed as a string literal, which
makes it clear that the message refers to the _generated_ file and not to
`build.rcl` itself. This is unlike e.g. standalone Jinja template files, where
if you want the banner in the output file, putting it in the input template can
create confusion about whether the comment refers to the template or to the
rendered output.

[banner]: rcl_evaluate.md#-banner-message

## Loading external data

The `build.rcl` file is a regular <abbr>RCL</abbr> document, which means it can
[import](imports.md) other documents. For example, say we have `users.rcl` with
the following contents:

```rcl
[
  {
    uid = 0,
    name = "Eldon Tyrell",
    email = "eldon@tyrell.com",
  },
  {
    uid = 7,
    name = "Rachael Tyrell",
    email = "rachael@tyrell.com",
  },
]
```

Then we can convert that to <abbr>YAML</abbr> using the following `build.rcl`:

```rcl
{
  "users.yaml": {
    // JSON documents are valid YAML.
    format = "json",
    banner = "# This file is generated from users.rcl.",
    contents = { users = import "users.rcl" },
  },
}
```

Which produces the following `users.yaml`:

```yaml
# This file is generated from users.rcl.
{
  "users": [
    {"email": "eldon@tyrell.com", "name": "Eldon Tyrell", "uid": 0},
    {"email": "rachael@tyrell.com", "name": "Rachael Tyrell", "uid": 7}
  ]
}
```

## Dynamic build targets

Because `build.rcl` is a regular <abbr>RCL</abbr> document, we can use the
normal features to _generate_ build targets. For example, let’s take the same
`users.rcl` as before, and generate one output file per user:

```rcl
let users = import "users.rcl";
{
  for user in users:
  let username = user.email.remove_suffix("@tyrell.com");
  f"users/{username}.toml": {
    format = "toml",
    banner = "# This file is generated from build.rcl.",
    contents = user,
  }
}
```

This will create a directory `users` with two files in it: `eldon.toml` and
`rachael.toml`. The contents of `eldon.toml` are as follows:

```toml
# This file is generated from build.rcl.
email = "eldon@tyrell.com"
name = "Eldon Tyrell"
uid = 0
```

## Recursing into subdirectories

In a larger repository, you may have several independent subdirectories that
each contain a `build.rcl` file. It would be nice to be able to build everything
with a single command, instead of having to run `rcl build` in every subdirectory.
We can do this through regular imports. Let's say we have subdirectory `a` and
`b`, then the top-level `build.rcl` could look like this:

```rcl
{
  for fname, target in import "a/build.rcl": f"a/{fname}": target,
  for fname, target in import "b/build.rcl": f"b/{fname}": target,
}
```

## Further reading

For full details, see the [`rcl build` docs](rcl_build.md), in particular the
section with [the build file specification][build-spec]. For integration with an
external build system, continue to [the next chapter about Ninja integration][using-ninja].

[build-spec]: rcl_build.md#build-files