gh-134939: Fill Out the concurrent.interpreters Docs (gh-135902)

Doc/library/concurrent.interpreters.rst

@@ -13,17 +13,26 @@
 
 --------------
 
-
-Introduction
-------------
-
 The :mod:`!concurrent.interpreters` module constructs higher-level
 interfaces on top of the lower level :mod:`!_interpreters` module.
 
-.. XXX Add references to the upcoming HOWTO docs in the seealso block.
+The module is primarily meant to provide a basic API for managing
+interpreters (AKA "subinterpreters") and running things in them.
+Running mostly involves switching to an interpreter (in the current
+thread) and calling a function in that execution context.
+
+For concurrency, interpreters themselves (and this module) don't
+provide much more than isolation, which on its own isn't useful.
+Actual concurrency is available separately through
+:mod:`threads <threading>`  See `below <interp-concurrency_>`_
 
 .. seealso::
 
+   :class:`~concurrent.futures.InterpreterPoolExecutor`
+      combines threads with interpreters in a familiar interface.
+
+    .. XXX Add references to the upcoming HOWTO docs in the seealso block.
+
    :ref:`isolating-extensions-howto`
        how to update an extension module to support multiple interpreters
 
@@ -41,18 +50,155 @@ interfaces on top of the lower level :mod:`!_interpreters` module.
 Key details
 -----------
 
-Before we dive into examples, there are a small number of details
+Before we dive in further, there are a small number of details
 to keep in mind about using multiple interpreters:
 
-* isolated, by default
+* `isolated <interp-isolation_>`_, by default
 * no implicit threads
 * not all PyPI packages support use in multiple interpreters yet
 
 .. XXX Are there other relevant details to list?
 
-In the context of multiple interpreters, "isolated" means that
+
-different interpreters do not share any state.  In practice, there is some
+.. _interpreters-intro:
-process-global data they all share, but that is managed by the runtime.
+
+Introduction
+------------
+
+An "interpreter" is effectively the execution context of the Python
+runtime.  It contains all of the state the runtime needs to execute
+a program.  This includes things like the import state and builtins.
+(Each thread, even if there's only the main thread, has some extra
+runtime state, in addition to the current interpreter, related to
+the current exception and the bytecode eval loop.)
+
+The concept and functionality of the interpreter have been a part of
+Python since version 2.2, but the feature was only available through
+the C-API and not well known, and the `isolation <interp-isolation_>`_
+was relatively incomplete until version 3.12.
+
+.. _interp-isolation:
+
+Multiple Interpreters and Isolation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A Python implementation may support using multiple interpreters in the
+same process.  CPython has this support.  Each interpreter is
+effectively isolated from the others (with a limited number of
+carefully managed process-global exceptions to the rule).
+
+That isolation is primarily useful as a strong separation between
+distinct logical components of a program, where you want to have
+careful control of how those components interact.
+
+.. note::
+
+   Interpreters in the same process can technically never be strictly
+   isolated from one another since there are few restrictions on memory
+   access within the same process.  The Python runtime makes a best
+   effort at isolation but extension modules may easily violate that.
+   Therefore, do not use multiple interpreters in security-sensitive
+   situations, where they shouldn't have access to each other's data.
+
+Running in an Interpreter
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Running in a different interpreter involves switching to it in the
+current thread and then calling some function.  The runtime will
+execute the function using the current interpreter's state.  The
+:mod:`!concurrent.interpreters` module provides a basic API for
+creating and managing interpreters, as well as the switch-and-call
+operation.
+
+No other threads are automatically started for the operation.
+There is `a helper <interp-call-in-thread_>`_ for that though.
+There is another dedicated helper for calling the builtin
+:func:`exec` in an interpreter.
+
+When :func:`exec` (or :func:`eval`) are called in an interpreter,
+they run using the interpreter's :mod:`!__main__` module as the
+"globals" namespace.  The same is true for functions that aren't
+associated with any module.  This is the same as how scripts invoked
+from the command-line run in the :mod:`!__main__` module.
+
+
+.. _interp-concurrency:
+
+Concurrency and Parallelism
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As noted earlier, interpreters do not provide any concurrency
+on their own.  They strictly represent the isolated execution
+context the runtime will use *in the current thread*.  That isolation
+makes them similar to processes, but they still enjoy in-process
+efficiency, like threads.
+
+All that said, interpreters do naturally support certain flavors of
+concurrency, as a powerful side effect of that isolation.
+There's a powerful side effect of that isolation.  It enables a
+different approach to concurrency than you can take with async or
+threads.  It's a similar concurrency model to CSP or the actor model,
+a model which is relatively easy to reason about.
+
+You can take advantage of that concurrency model in a single thread,
+switching back and forth between interpreters, Stackless-style.
+However, this model is more useful when you combine interpreters
+with multiple threads.  This mostly involves starting a new thread,
+where you switch to another interpreter and run what you want there.
+
+Each actual thread in Python, even if you're only running in the main
+thread, has its own *current* execution context.  Multiple threads can
+use the same interpreter or different ones.
+
+At a high level, you can think of the combination of threads and
+interpreters as threads with opt-in sharing.
+
+As a significant bonus, interpreters are sufficiently isolated that
+they do not share the :term:`GIL`, which means combining threads with
+multiple interpreters enables full multi-core parallelism.
+(This has been the case since Python 3.12.)
+
+Communication Between Interpreters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In practice, multiple interpreters are useful only if we have a way
+to communicate between them.  This usually involves some form of
+message passing, but can even mean sharing data in some carefully
+managed way.
+
+With this in mind, the :mod:`!concurrent.interpreters` module provides
+a :class:`queue.Queue` implementation, available through
+:func:`create_queue`.
+
+.. _interp-object-sharing:
+
+"Sharing" Objects
+^^^^^^^^^^^^^^^^^
+
+Any data actually shared between interpreters loses the thread-safety
+provided by the :term:`GIL`.  There are various options for dealing with
+this in extension modules.  However, from Python code the lack of
+thread-safety means objects can't actually be shared, with a few
+exceptions.  Instead, a copy must be created, which means mutable
+objects won't stay in sync.
+
+By default, most objects are copied with :mod:`pickle` when they are
+passed to another interpreter.  Nearly all of the immutable builtin
+objects are either directly shared or copied efficiently.  For example:
+
+* :const:`None`
+* :class:`bool` (:const:`True` and :const:`False`)
+* :class:`bytes`
+* :class:`str`
+* :class:`int`
+* :class:`float`
+* :class:`tuple` (of similarly supported objects)
+
+There is a small number of Python types that actually share mutable
+data between interpreters:
+
+* :class:`memoryview`
+* :class:`Queue`
 
 
 Reference
@@ -73,12 +219,19 @@ This module defines the following functions:
 .. function:: get_main()
 
    Return an :class:`Interpreter` object for the main interpreter.
+   This is the interpreter the runtime created to run the :term:`REPL`
+   or the script given at the command-line.  It is usually the only one.
 
 .. function:: create()
 
    Initialize a new (idle) Python interpreter
    and return a :class:`Interpreter` object for it.
 
+.. function:: create_queue()
+
+   Initialize a new cross-interpreter queue and return a :class:`Queue`
+   object for it.
+
 
 Interpreter objects
 ^^^^^^^^^^^^^^^^^^^
@@ -94,7 +247,7 @@ Interpreter objects
 
       (read-only)
 
-      The interpreter's ID.
+      The underlying interpreter's ID.
 
    .. attribute:: whence
 
@@ -113,8 +266,10 @@ Interpreter objects
 
    .. method:: prepare_main(ns=None, **kwargs)
 
-      Bind "shareable" objects in the interpreter's
+      Bind objects in the interpreter's :mod:`!__main__` module.
-      :mod:`!__main__` module.
+
+      Some objects are actually shared and some are copied efficiently,
+      but most are copied via :mod:`pickle`.  See :ref:`interp-object-sharing`.
 
    .. method:: exec(code, /, dedent=True)
 
@@ -125,6 +280,8 @@ Interpreter objects
       Return the result of calling running the given function in the
       interpreter (in the current thread).
 
+   .. _interp-call-in-thread:
+
    .. method:: call_in_thread(callable, /, *args, **kwargs)
 
       Run the given function in the interpreter (in a new thread).
@@ -159,7 +316,36 @@ Exceptions
    an object cannot be sent to another interpreter.
 
 
-.. XXX Add functions for communicating between interpreters.
+Communicating Between Interpreters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. class:: Queue(id)
+
+   A wrapper around a low-level, cross-interpreter queue, which
+   implements the :class:`queue.Queue` interface.  The underlying queue
+   can only be created through :func:`create_queue`.
+
+   Some objects are actually shared and some are copied efficiently,
+   but most are copied via :mod:`pickle`.  See :ref:`interp-object-sharing`.
+
+   .. attribute:: id
+
+      (read-only)
+
+      The queue's ID.
+
+
+.. exception:: QueueEmptyError
+
+   This exception, a subclass of :exc:`queue.Empty`, is raised from
+   :meth:`!Queue.get` and :meth:`!Queue.get_nowait` when the queue
+   is empty.
+
+.. exception:: QueueFullError
+
+   This exception, a subclass of :exc:`queue.Full`, is raised from
+   :meth:`!Queue.put` and :meth:`!Queue.put_nowait` when the queue
+   is full.
 
 
 Basic usage
@@ -184,6 +370,12 @@ Creating an interpreter and running code in it::
         print('spam!')
         """))
 
+    def run(arg):
+        return arg
+
+    res = interp.call(run, 'spam!')
+    print(res)
+
     def run():
         print('spam!')
 
@@ -193,6 +385,3 @@ Creating an interpreter and running code in it::
 
     t = interp.call_in_thread(run)
     t.join()
-
-
-.. XXX Explain about object "sharing".