If a lambda set is non-recursive, but contains naked recursion pointers,
we should not fill those naked pointers in with the slot of the lambda
set during interning. Such naked pointers must belong to an encompassing
lambda set that is in fact recursive, and will be filled in later.
For example, `LambdaSet([Foo, LambdaSet(Bar, [<rec>])] as <rec>)` should
not have the inner lambda set's capture be filled in with itself.
Also, during reification of recursion pointers, we do not need to
traverse re-inserted lambda sets again, since they were just fixed-up.
Closes#5026
Adding a cache layer can only be done with a snapshot and rollback.
This is necessary to prevent extra variables just lying around on the
toplevel of the layout cache.
After unification, variable roots can change. So, when we invalidate
entries in the layout cache, we must compare for variable equivalence
relative to the current state of subs.
The `LambdaSet` struct is frequently used independently to examine how a
lambda set should be packed or unpacked. However, it is also often
converted into a full layout via `Layout::LambdaSet(LambdaSet)` to be a
part of function arguments, for example.
In preparing to intern all layouts, we need a way to cheaply go from a
`lambda_set` to an interned `Layout::LambdaSet(lambda_set)`, since this
is a very common operation. The proposed solution is to keep the wrapped
layout cached on `LambdaSet` itself, which this PR does.
The tricky bit of inserting a lambda set is we need to fill in the
interned `full_layout` only after the lambda set is inserted,
but we don't want to allocate a new interned slot if the same lambda set
layout has already been inserted with a different `full_layout` slot.
For example, if we insert `LambdaSet { set : [A] }` twice in two
different threads, we want the `full_layout` they map to to be the same.
So we nede to check if an interned representation with a full_layout
exists, before we allocate a new full_layout and insert a fresh lambda
set.
So,
- check if the "normalized" lambda set (with a void full_layout slot) maps to an
inserted lambda set in
- in a thread-local cache, or globally
- if so, use that one immediately
- otherwise, allocate a new (global) slot, intern the lambda set, and then fill the slot in
- save the interned layout and lambda set mapping thread-locally
I realized that we'll need to make the layout interner more complicated
to support things like recursive pointers pointing to their parents and
to support lambda set layout caching. Since the layout interner is the
only user of intern crate right now anyway, just inline the whole thing.
