b255b17db9
Problems
========
* Combining collections through comprehensions is too verbose.
I don’t want to write {for x in xs: x, for y in ys: y}.
* Pipe (|) for set and dict union is very ad-hoc in the typechecker,
and binary operators are awkward to format.
Solution
========
We can solve both problems by adding unpack: the ability to splice
a collection into another one. Like * and ** in Python, and ... in
Javascript.
* It’s a short way to splice in collections (of which concat is a
special case).
* Dict and set union are now completely regular and easily formattable.
* Arguably, the meaning of .. and ... is more self-evident than the
meaning of |, so it makes code more readable to newcomers.
Design space and alternatives not taken
=======================================
I would prefer to have just .. for unpack, and it would unpack both
single elements (scalars) from lists and sets, as well as key-value
pairs from dicts. However, that means that you can no longer
syntactically tell whether {..xs} is a dict or a set, which previously
was possible (except for the empty {}, which we defined to be a dict,
like Python). There are several ways to deal with this:
Distinct unpack syntax
Distinguish single-element and key-value unpack syntactically, e.g.
.. for scalar and ... for key-value. This is what Python does too,
with * vs. **.
Distinct set literal syntax
Use a distinct syntax for sets. { ... } would be reserved for dicts,
sets would use e.g. {| ... |}, set{ ... }, {{ }}, etc. It would
simplify many things, right now it’s kind of messy that we don't
know what a {}-literal is until we peek inside.
Delete sets entirely
As with using distinct syntax, this removes many complications,
including much of the near duplication between lists and sets. Who
needs sets anyway? We can just use lists instead, and add a few
convenience methods to make things unique. We can even distinguish
between “drop duplicates” and “assert that it’s already unique”,
which a set in RCL currently can’t do, constructing one can only
drop duplicates. Though nothing prevents adding a method on list
that asserts elements are unique and returns a set. Sets are a bit
like unsigned integers, they add some value by making invalid states
unrepresentable, they allow slightly more precise types, but we can
survive without them, and in practice not much is lost.
I did implement this option. It did simplify things a lot. It felt
like a shame to delete so much code, including some rather neat
parts, but that’s just sunk cost fallacy. But in the end, I think
that there is value in having sets. Many things are naturally sets
rather than lists. No other configuration language offers sets,
which gives RCL a unique advantage in this regard. Or maybe there is
a good reason none of the others have sets. Though Python does, and
it also reuses the {}-syntax for dicts and sets.
It also simplifies the type system tremendously, because now we
no longer have two collection types that have an inner element type
(list and set), and it sidesteps the mess around kind of needing a
Collection[T] supertype, but then how to deal with Collection[T]
vs. Union[List[T], Set[T]], how to document it and explain when it
occurs in type errors, etc.
So I tried it, but it felt too harsh, I want to try and make unpack
work without deleting sets. This dead end is not part of the history
included with this merge commit.
Give up on static typechecking
Maybe it’s fine if we don’t know whether {..xs} is a dict or a
set? That minimal example is pathological, but union is a common
operation, and losing the type for {..xs, ..ys} would be a shame.
In the end, the option implemented is to have the distinct unpack
syntax.
This was a big feature to write, primarily due to complications for the
typechecker, and the many cases to handle. Most of the dead ends are not
part of the history merged in this merge commit. There is an entire
other branch about adding a Collection[T] type that is now obsolete as
well.
|
||
|---|---|---|
| docs | ||
| examples | ||
| fuzz | ||
| golden | ||
| grammar | ||
| ideas | ||
| pyrcl | ||
| src | ||
| tools | ||
| wasm | ||
| website | ||
| .gitignore | ||
| .gitmodules | ||
| build.rcl | ||
| Cargo.lock | ||
| Cargo.rcl | ||
| Cargo.toml | ||
| CONTRIBUTING.md | ||
| flake.lock | ||
| flake.nix | ||
| LICENSE | ||
| mkdocs.yml | ||
| README.md | ||
| rust-toolchain.toml | ||
The RCL Configuration Language
Getting Started · Documentation · Changelog · Online Playground
RCL is a domain-specific language for generating configuration files and querying json documents. It is a superset of json that extends it into a simple, gradually typed, functional programming language that resembles Python and Nix.
RCL can be used through the rcl command-line tool that can export
documents to json, yaml, toml, and more. It can also be used through
a native Python module, with an interface similar to the json module.
About
RCL solves the following problems:
- Copy-pasted yaml blocks that differ by a single value.
- Broken configs due to whitespace and escaping footguns in templating engines.
- Drift between tools due to lack of a single source of truth.
- Struggling to remember
jqsyntax.
It does that as follows:
- A real language. Use variables, loops, imports, and functions to eliminate duplication.
- Familiar syntax. Have you used Python, TypeScript, or Rust before? Then you will find RCL obvious to read and natural to write.
- Generate rather than template. Manipulate data structures, not strings. Generate correct json, yaml, and toml.
- Built to integrate.
Generate configs for tools that do not natively talk to each other, all
from a single source of truth. Integrate with your existing build tools,
use the Python module, or the built-in
rcl buildto update generated files. - Gradual types. Add type annotations where they help to eliminate bugs and make code self-documenting, omit them in straightforward code.
- Built-in json queries. A language for manipulating structured data makes a pretty good query tool. Run map and filter pipelines straight from your terminal.
Example
Given this input:
{
// Generate backup buckets for each database and retention period.
backup_buckets = [
let retention_days = { hourly = 4, daily = 30, monthly = 365 };
for database in ["alpha", "bravo"]:
for period, days in retention_days:
{
name = f"{database}-{period}",
region = "eu-west",
lifecycle_policy = { delete_after_seconds = days * 24 * 3600 },
}
],
}
RCL generates:
{
"backup_buckets": [
{
"name": "alpha-hourly",
"region": "eu-west",
"lifecycle_policy": { "delete_after_seconds": 345600 }
},
{
"name": "alpha-daily",
"region": "eu-west",
"lifecycle_policy": { "delete_after_seconds": 2592000 }
},
// And 4 similar entries, omitted here for brevity.
]
}
For an interactive demo in your browser, see https://rcl-lang.org.
Getting started
After the interactive examples on the website, the manual is the best resource for further information. The most useful chapters to get started:
You may also find the examples in the examples directory instructive.
Some helpful commands after a clone:
# Build
cargo build --release
# Print usage
target/release/rcl
target/release/rcl eval --help
# Evaluate an RCL expression to json
target/release/rcl eval --format=json examples/tags.rcl
# Query an RCL or json document
target/release/rcl query examples/tags.rcl input.tags.ams01
# Autoformat and highlight an RCL expression (non-destructive, prints to stdout)
target/release/rcl fmt examples/tags.rcl
Status
RCL is a hobby project without stability promise. It is usable and useful, well-tested, and well-documented, but also still experimental, and it may have breaking changes. Syntax highlighting is available for major editors like Vim, Emacs, Helix, and Zed.
Support RCL
One thing that holds RCL back from being useful to more people is the lack of widespread support for syntax highlighting on platforms such as GitHub. If RCL is useful to you, you can help by using RCL publicly in a GitHub repository to demonstrate traction. Use it seriously of course, please don’t game the metric. Other things you can help with are getting RCL packaged for your favorite package manager, and developing syntax highlighting for your favorite editor if it is not yet supported.
Development
Run all tests and checks below in one command:
nix flake check
Run golden tests:
cargo build
golden/run.py
Run unit tests and lints:
cargo test
cargo clippy
Typecheck Python sources
mypy --strict --exclude pyrcl .
mypy --strict pyrcl
Check formatting:
cargo fmt
black .
View coverage of the golden tests:
nix build .#coverage --out-link result
xdg-open result/index.html
For how to run the fuzzers, see docs/testing.md.
Building WASM
See the readme in the wasm directory.
License
RCL is licensed under the Apache 2.0 license. It may be used in free software as well as closed-source applications, both for commercial and non-commercial use under the conditions given in the license. If you want to use RCL in your GPLv2-licensed software, you can add an exception to your copyright notice. Please do not open an issue if you disagree with the choice of license.