gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters (gh-103084)

Sharing mutable (or non-immortal) objects between interpreters is generally not safe.  We can work around that but not easily. 
 There are two restrictions that are critical for objects that break interpreter isolation.

The first is that the object's state be guarded by a global lock.  For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL.

The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one.  This is because every interpreter has (or will have, see gh-101660) its own object allocator.  Deallocating an object with a different allocator can cause crashes.

The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements.  To do so, we do the following:

* add a mechanism for re-using a temporary thread state tied to the main interpreter in an arbitrary thread
   * add _PyRuntime.imports.extensions.main_tstate` 
   * add _PyThreadState_InitDetached() and _PyThreadState_ClearDetached() (pystate.c)
   * add _PyThreadState_BindDetached() and _PyThreadState_UnbindDetached() (pystate.c)
* make sure the cache dict (_PyRuntime.imports.extensions.dict) and its items are all owned by the main interpreter)
* add a placeholder using for a granular global lock

Note that the cache is only used for legacy extension modules and not for multi-phase init modules.

https://github.com/python/cpython/issues/100227
This commit is contained in:
Eric Snow 2023-03-29 17:15:43 -06:00 committed by GitHub
parent 121057aa36
commit dcd6f226d6
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 274 additions and 59 deletions

View file

@ -14,13 +14,19 @@ struct _import_runtime_state {
which is just about every time an extension module is imported.
See PyInterpreterState.modules_by_index for more info. */
Py_ssize_t last_module_index;
/* A dict mapping (filename, name) to PyModuleDef for modules.
Only legacy (single-phase init) extension modules are added
and only if they support multiple initialization (m_size >- 0)
or are imported in the main interpreter.
This is initialized lazily in _PyImport_FixupExtensionObject().
Modules are added there and looked up in _imp.find_extension(). */
PyObject *extensions;
struct {
/* A thread state tied to the main interpreter,
used exclusively for when the extensions dict is access/modified
from an arbitrary thread. */
PyThreadState main_tstate;
/* A dict mapping (filename, name) to PyModuleDef for modules.
Only legacy (single-phase init) extension modules are added
and only if they support multiple initialization (m_size >- 0)
or are imported in the main interpreter.
This is initialized lazily in _PyImport_FixupExtensionObject().
Modules are added there and looked up in _imp.find_extension(). */
PyObject *dict;
} extensions;
/* Package context -- the full module name for package imports */
const char * pkgcontext;
};

View file

@ -127,6 +127,11 @@ PyAPI_FUNC(void) _PyThreadState_Init(
PyThreadState *tstate);
PyAPI_FUNC(void) _PyThreadState_DeleteExcept(PyThreadState *tstate);
extern void _PyThreadState_InitDetached(PyThreadState *, PyInterpreterState *);
extern void _PyThreadState_ClearDetached(PyThreadState *);
extern void _PyThreadState_BindDetached(PyThreadState *);
extern void _PyThreadState_UnbindDetached(PyThreadState *);
static inline void
_PyThreadState_UpdateTracingState(PyThreadState *tstate)

View file

@ -41,6 +41,11 @@ extern PyTypeObject _PyExc_MemoryError;
in accordance with the specification. */ \
.autoTSSkey = Py_tss_NEEDS_INIT, \
.parser = _parser_runtime_state_INIT, \
.imports = { \
.extensions = { \
.main_tstate = _PyThreadState_INIT, \
}, \
}, \
.ceval = { \
.perf = _PyEval_RUNTIME_PERF_INIT, \
}, \