The free threading build uses QSBR to delay the freeing of dictionary
keys and list arrays when the objects are accessed by multiple threads
in order to allow concurrent reads to proceed with holding the object
lock. The requests are processed in batches to reduce execution
overhead, but for large memory blocks this can lead to excess memory
usage.
Take into account the size of the memory block when deciding when to
process QSBR requests.
Also track the amount of memory being held by QSBR for mimalloc pages.
Advance the write sequence if this memory exceeds a limit. Advancing
the sequence will allow it to be freed more quickly.
Process the held QSBR items from the "eval breaker", rather than from
`_PyMem_FreeDelayed()`. This gives a higher chance that the global read
sequence has advanced enough so that items can be freed.
(cherry picked from commit 113de8545f)
Co-authored-by: Neil Schemenauer <nas-github@arctrix.com>
Co-authored-by: Sam Gross <colesbury@gmail.com>
This avoids a case where the interpreter's queue of memory to be freed
could grow rapidly if there are many short lived threads.
(cherry picked from commit 2f6e0e9f70)
The read of `shared->array` should happen under the lock to avoid a race.
(cherry picked from commit b4ff8b22b3)
Co-authored-by: Peter Hawkins <phawkins@google.com>
Due to a limitation in TSAN, all reads from `PyThreadState.state` must be
atomic to avoid reported races.
(cherry picked from commit 90ec19fd33)
Co-authored-by: Sam Gross <colesbury@gmail.com>
Release the GIL before calling `_Py_qsbr_unregister`.
The deadlock could occur when the GIL was enabled at runtime. The
`_Py_qsbr_unregister` call might block while holding the GIL because the
thread state was not active, but the GIL was still held.
(cherry picked from commit 078b8c8cf2)
Co-authored-by: Sam Gross <colesbury@gmail.com>
`_Py_qsbr_unregister` is called when the PyThreadState is already
detached, so the access to `tstate->qsbr` isn't safe without locking the
shared mutex. Grab the `struct _qsbr_shared` from the interpreter
instead.
(cherry picked from commit 33d20199af)
Co-authored-by: Alex Turner <alexturner@meta.com>
If a thread blocks while waiting on the `shared->mutex` lock, the array
of QSBR states may be reallocated. The `tstate->qsbr` values before the
lock is acquired may not be the same as the value after the lock is acquired.
This implements the delayed reuse of mimalloc pages that contain Python
objects in the free-threaded build.
Allocations of the same size class are grouped in data structures called
pages. These are different from operating system pages. For thread-safety, we
want to ensure that memory used to store PyObjects remains valid as long as
there may be concurrent lock-free readers; we want to delay using it for
other size classes, in other heaps, or returning it to the operating system.
When a mimalloc page becomes empty, instead of immediately freeing it, we tag
it with a QSBR goal and insert it into a per-thread state linked list of
pages to be freed. When mimalloc needs a fresh page, we process the queue and
free any still empty pages that are now deemed safe to be freed. Pages
waiting to be freed are still available for allocations of the same size
class and allocating from a page prevent it from being freed. There is
additional logic to handle abandoned pages when threads exit.
This adds a safe memory reclamation scheme based on FreeBSD's "GUS" and
quiescent state based reclamation (QSBR). The API provides a mechanism
for callers to detect when it is safe to free memory that may be
concurrently accessed by readers.
