If an entry in the layout cache contains recursive structures, the entry
is not reusable if the recursive structure is currently in the "seen"
set. The example elucidated in the source code is as follows:
Suppose we are constructing the layout of
```
[A, B (List r)] as r
```
and we have already constructed and cached the layout of `List r`, which would
be
```
List (Recursive [Unit, List RecursivePointer])
```
If we use the cached entry of `List r`, we would end up with the layout
```
Recursive [Unit, (List (Recursive [Unit, List RecursivePointer]))]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cached layout for `List r`
```
but this is not correct; the canonical layout of `[A, B (List r)] as r` is
```
Recursive [Unit, (List RecursivePointer)]
```
However, the current implementation only preserves this behavior for
structures that contain one recursive structure under them. In practice,
there can be structures that contain multiple recursive structures under
them, and we must be sure to record all those structures in the
layout-cache.
With a code like
```
thenDo = \x, callback ->
callback x
f = \{} ->
code = 10u16
bf = \{} ->
thenDo code \_ -> bf {}
bf {}
```
The lambda `\_ -> bf {}` must capture `bf`. Previously, this would not
happen correctly, because we assumed that mutually recursive functions
(including singleton recursive functions, like `bf` here) cannot capture
themselves.
Of course, that premise does not hold in general. Instead, we should have
mutually recursive functions capture the closure (haha, get it) of
values captured by all functions constituting the mutual recursion.
Then, any nested closures can capture outer recursive closures' values
appropriately.
There are times that multiple concrete types may appear in
unspecialized lambda sets that are being unified. The primary case is
during monomorphization, when unspecialized lambda sets join at the same
time that concrete types get instantiated. Since lambda set
specialization and compaction happens only after unifications are
complete, unifications that monomorphize can induce the above-described
situation.
In these cases,
- unspecialized lambda sets that are due to equivalent type variables
can be compacted, since they are in fact the same specialization.
- unspecialized lambda sets that are due to different type variables
cannot be compacted, even if their types unify, since they may point
to different specializations. For example, consider the unspecialized
lambda set `[[] + [A]:toEncoder:1 + [B]:toEncoder:1]` - this set wants
two encoders, one for `[A]` and one for `[B]`, which is materially
different from the set `[[] + [A, B]:toEncoder:1]`.
If a lambda set is non-recursive, but contains naked recursion pointers,
we should not fill those naked pointers in with the slot of the lambda
set during interning. Such naked pointers must belong to an encompassing
lambda set that is in fact recursive, and will be filled in later.
For example, `LambdaSet([Foo, LambdaSet(Bar, [<rec>])] as <rec>)` should
not have the inner lambda set's capture be filled in with itself.
Also, during reification of recursion pointers, we do not need to
traverse re-inserted lambda sets again, since they were just fixed-up.
Closes#5026
