The module name and path hack throws off Mypy, in different ways
depending on how you run it (outside Nix shell, inside Nix develop
shell, or as part of the flake check ...). Let's just forward-declare
it and sidestep problems.
Maybe I should make this script the source of truth and generate the
Pygments grammar from it. We could do that later, for now this works.
I should do it, and if I don't do it now then I probably will not do it
soon. On the other hand, no need to let the perfect be the enemy of the
good; calling the script manually is already much better than having to
update all the places manually. So let's merge it as it is.
This corrects one bug in the plugin there that went unnoticed: the
'enumerate' builtin was not highlighted previously. So that's a win
for generating things!
This moves one more place of duplication of builtins into
generate_keywords.py as a single source of truth, resolving
a to do in the smith fuzzer.
This does once more shuffle all of these around in the fuzzer, which
makes the existing fuzz corpus mostly meaningless. Fortunately, this
should be the last time that this happens: with the new approach we
can modify the builtins with minimal changes to the meaning of the
fuzz corpus, which is something that I wanted for a long time.
I regularly add new methods, and it's becoming tedious to have to
remember to update all the places that reference these, so let's
generate them and automate the process. For now, I'm choosing the
Pygments grammar as the source of truth, and the first target to
generate is the fuzz dictionary.
I want to make a Zed extension, and it looks like Zed does not support
Tree-sitter grammars that do not live at the repository root, so let's
just make a separate repository to dump a generated copy of the grammar
in. It will also make integration with other Tree-sitter consumers
easier.
I accidentally reused the mode letter for typecheck and for toml export.
So now is maybe a good time to create a copy of every input, reorder the
modes, then minimize the corpus again.
Without this, the inputs are immediately discarded by the fuzzer -- so
it turns out this seeding was not doing anything all the time! When I
run it now, it crashes instantly with a SIGSGEV. It is not caught by
libfuzzer itself so I can't see which input is crashing, but I suspect
it's my stack overflow example. Let's handle that and see if I can still
make the fuzzer crash.
The fuzzer is having a hard time discovering interesting inputs to
key_by. So far it discovered that it can do
[true].key_by({}.key_by)
which works, because {}.key_by(true) does not fail because the input is
empty, so it returns and empty dict, and then the output is {{}: true}.
But the fuzzer has not yet discovered that key_by is supposed to receive
a lambda, and it's not producing any interesting groupings.
But we do have a corpus of goldens that covers these cases. They are not
as small as the fuzz inputs, but they might be a good starting point. So
let's add a script to seed the fuzz corpus. So far I'm still not
impressed, but maybe I am just impatient and I need to let the fuzzer
run a bit longer.
