I want to add an unpack operator, and to be able to type the type
expectation, I need a collection supertype for List and Set. Maybe
once I have this, I can actually type the union operator in a more
elegant way as well and I no longer need unpack. But unpack seems
nice anyway, and a Collection supertype seems nice as well.
This moves one more place of duplication of builtins into
generate_keywords.py as a single source of truth, resolving
a to do in the smith fuzzer.
This does once more shuffle all of these around in the fuzzer, which
makes the existing fuzz corpus mostly meaningless. Fortunately, this
should be the last time that this happens: with the new approach we
can modify the builtins with minimal changes to the meaning of the
fuzz corpus, which is something that I wanted for a long time.
I regularly add new methods, and it's becoming tedious to have to
remember to update all the places that reference these, so let's
generate them and automate the process. For now, I'm choosing the
Pygments grammar as the source of truth, and the first target to
generate is the fuzz dictionary.
I'm leaving the Zed extension pointing to the older commit of the
Tree-sitter grammar, I'll update that after this version bump. It's
a bit awkward to do it this way around, but there are circular
dependencies that can't be avoided. Maybe with an attack on SHA1 it
can be done in theory, but let's not go there.
At first I also wanted to support rounding to a negative number of
decimals (so rounding to a positive power of 10), but scope creep,
complications ... I don't need it, and we can always add that later.
It found one issue right away, related to using an i16::MIN exponent,
which overflows the way we parsed. But then I realized there are a few
other bugs in the number parser ... I added a marker for one and fixed
handling of the implicit exponent offsets.
This enables us treat numbers with an exponent losslessly. We don't
conflate the decimal point with the exponent in case they get in the
way of each other.
It also greatly simplifies the formatting. We can mechanically format
the representation now, without having to use heuristics for when to
switch to scientific notation. The catch is of course that the
heuristics will need to move elsewhere. We'll have to normalize the
numbers after arithmetic operations.
RCL aims to be obvious to understand. Num might be cryptic for new users,
and although we also have "Int" rather than "Integer", that one is very
established, "Num" may be a bit too obscure. (We also have "String"
rather than "Str", consistency ...). It's a type that I expect has
little use for end-users, but it shows up in the negation error message,
so let's make it unambiguous and call it "Number".
This is only the start, but let's verify Decimal::cmp against f64::cmp.
It instantly finds an input where they disagree:
Compare {
a: NormalF64(
-0.16406250000007813,
),
b: NormalF64(
0.0,
),
}
This adds back the exception that was removed by allowing float parsing
imprecision, though in a more limited form initially because it only
affected exponents.
But after running the fuzzer for a bit longer, it also affects large
integers, so we are back to the start, overflow is just an intentional
incompatibility.
This removes one case of incompatibility with Serde. If you write a
float literal that is too precise to be represented exactly, then we now
silently round it rather than treating it as an overflow error. I think
this is acceptable because if you are in the case where you care about
numbers to 19 significant digits then probably RCL is not the best tool
for what you are doing, but the case where we encounter some arbitrary
json that we want to query with "rcl jq" and it happens to have some
humongous float in it, that is probably more likely. Python handles
float literals in this way too so I think it's okay.
The choice I went with is to have a 16-bit exponent, which gives RCL's
float/decimal type more range than a regular f64. Now the fuzzer can
generate an input with a large exponent, and RCL will happily echo it,
and it's technically syntactically valid json, but Serde rejects it with
"number out of range" (in the same way that RCL rejects some numbers as
overflow). So add an exception for this mismatch.
The one thing that prevents that right now is floats, and the fuzzer
discovered it within a few seconds:
╭──────╴ Opcode (hex)
│ ╭───╴ Argument (hex)
│ │ ╭╴ Operation, argument (decimal)
26 03 ExprPushInput, 3
take_str, 3 → "4e2"
e6 01 ModeJsonSuperset, 1
EvalJsonSuperset -->
4e2
I realized today that I want this. In particular, the API of my music
player Musium returns albums with a numeric playcount and discovery
score, and I want to sort on that. Finally that is possible now that I
am adding support for floats. But I need a way to sort on one field of
a dict! Arguably this is more important than the bare sort itself.
While I do this for lists, we can do the same for sets.
It started to get annoying to have to define it myself every time, so
let's just add it properly now. This also resolves the longstanding
issue in the RCL pretty-printer that we have no good way to print the
empty set -- now we do!
It means the fuzzer gets to explore less, actually, but we still have
the source-based fuzzer that will find the case where the colon is
missing, and which could hunt for non-idempotencies in the formatter and
such.
As expected, the golden tests fail to run under Nix because the test
directory is not writable. And it's better to not write in my opinion,
let's not hack that and have a dry run output mode.
For now the output format is not structured, this is good enough for the
thests. It could be nice to do structured output in RCL format, but we
can do that later if needed.
The fuzzer was unable to find the overflow case in the new List.sum
method even after several minutes. To help it a bit to find interesting
cases, let's add large integers that are close to overflowing to the
input through a shorcut. Previously they could get there, but only when
reading an 8-byte integer from the input, so first you need a mutation
to bump the number to 8, and then a mutation to have an integer in those
8 bytes that is close to overflowing, and those together are very
unlikely.
When using RCL as a jq replacement, often I have some pipeline and I
want to edit the last part of the query on the command line. I don't
want to have to move my cursor all the way back to wrap the entire
expression in braces. So even though comprehensions can already do map
and filter and flatmap, I still want to add those.
This is step one, adding map.
So I can highlight stuff on my blog as long as there is no highlighting
in Pandoc/Skylighting. With the change to MarkupString, this was really
easy to do!
This fixes a longstanding issue where reporting errors that we have to
blame on just the document's result in general got blamed on its full
span, which is often a comment and not the offending value. Now we blame
it on the inner body expression, which is more natural.
The autoformatter in "rcl format" pretty-prints the CST. But we can
also evaluate the document, and then it gets pretty-printed by the
value pretty printer. These two should match, but currently they do
not. Add a fuzzer to discover all the cases where they don't.
I am making the formatter more suitable for real use, and as a result
I want to add a flake/CI check that ensures that all the examples are
formatted correctly. But then I need this --check mode.
