bpo-32030: Rework memory allocators (#4625)

* Fix _PyMem_SetupAllocators("debug"): always restore allocators to
  the defaults, rather than only caling _PyMem_SetupDebugHooks().
* Add _PyMem_SetDefaultAllocator() helper to set the "default"
  allocator.
* Add _PyMem_GetAllocatorsName(): get the name of the allocators
* main() now uses debug hooks on memory allocators if Py_DEBUG is
  defined, rather than calling directly malloc()
* Document default memory allocators in C API documentation
* _Py_InitializeCore() now fails with a fatal user error if
  PYTHONMALLOC value is an unknown memory allocator, instead of
  failing with a fatal internal error.
* Add new tests on the PYTHONMALLOC environment variable
* Add support.with_pymalloc()
* Add the _testcapi.WITH_PYMALLOC constant and expose it as
   support.with_pymalloc().
* sysconfig.get_config_var('WITH_PYMALLOC') doesn't work on Windows, so
   replace it with support.with_pymalloc().
* pythoninfo: add _testcapi collector for pymem
This commit is contained in:
Victor Stinner 2017-11-29 17:20:38 +01:00 committed by GitHub
parent c15bb49d71
commit 5d39e04290
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
14 changed files with 405 additions and 171 deletions

View file

@ -100,9 +100,10 @@ The following function sets are wrappers to the system allocator. These
functions are thread-safe, the :term:`GIL <global interpreter lock>` does not
need to be held.
The default raw memory block allocator uses the following functions:
:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call
``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes.
The :ref:`default raw memory allocator <default-memory-allocators>` uses
the following functions: :c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc`
and :c:func:`free`; call ``malloc(1)`` (or ``calloc(1, 1)``) when requesting
zero bytes.
.. versionadded:: 3.4
@ -165,7 +166,8 @@ The following function sets, modeled after the ANSI C standard, but specifying
behavior when requesting zero bytes, are available for allocating and releasing
memory from the Python heap.
By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`.
The :ref:`default memory allocator <default-memory-allocators>` uses the
:ref:`pymalloc memory allocator <pymalloc>`.
.. warning::
@ -270,7 +272,8 @@ The following function sets, modeled after the ANSI C standard, but specifying
behavior when requesting zero bytes, are available for allocating and releasing
memory from the Python heap.
By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`.
The :ref:`default object allocator <default-memory-allocators>` uses the
:ref:`pymalloc memory allocator <pymalloc>`.
.. warning::
@ -326,6 +329,31 @@ By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`.
If *p* is *NULL*, no operation is performed.
.. _default-memory-allocators:
Default Memory Allocators
=========================
Default memory allocators:
=============================== ==================== ================== ===================== ====================
Configuration Name PyMem_RawMalloc PyMem_Malloc PyObject_Malloc
=============================== ==================== ================== ===================== ====================
Release build ``"pymalloc"`` ``malloc`` ``pymalloc`` ``pymalloc``
Debug build ``"pymalloc_debug"`` ``malloc`` + debug ``pymalloc`` + debug ``pymalloc`` + debug
Release build, without pymalloc ``"malloc"`` ``malloc`` ``malloc`` ``malloc``
Release build, without pymalloc ``"malloc_debug"`` ``malloc`` + debug ``malloc`` + debug ``malloc`` + debug
=============================== ==================== ================== ===================== ====================
Legend:
* Name: value for :envvar:`PYTHONMALLOC` environment variable
* ``malloc``: system allocators from the standard C library, C functions:
:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`
* ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>`
* "+ debug": with debug hooks installed by :c:func:`PyMem_SetupDebugHooks`
Customize Memory Allocators
===========================
@ -431,7 +459,8 @@ Customize Memory Allocators
displayed if :mod:`tracemalloc` is tracing Python memory allocations and the
memory block was traced.
These hooks are installed by default if Python is compiled in debug
These hooks are :ref:`installed by default <default-memory-allocators>` if
Python is compiled in debug
mode. The :envvar:`PYTHONMALLOC` environment variable can be used to install
debug hooks on a Python compiled in release mode.
@ -453,9 +482,9 @@ to 512 bytes) with a short lifetime. It uses memory mappings called "arenas"
with a fixed size of 256 KiB. It falls back to :c:func:`PyMem_RawMalloc` and
:c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes.
*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_MEM` (ex:
:c:func:`PyMem_Malloc`) and :c:data:`PYMEM_DOMAIN_OBJ` (ex:
:c:func:`PyObject_Malloc`) domains.
*pymalloc* is the :ref:`default allocator <default-memory-allocators>` of the
:c:data:`PYMEM_DOMAIN_MEM` (ex: :c:func:`PyMem_Malloc`) and
:c:data:`PYMEM_DOMAIN_OBJ` (ex: :c:func:`PyObject_Malloc`) domains.
The arena allocator uses the following functions: