bpo-43795: PEP 652 user documentation (GH-25668)

- Reformat the C API and ABI Versioning page (and extend/clarify a bit)
- Rewrite the stable ABI docs into a general text on C API Compatibility
- Add a list of Limited API contents, and notes for the individual items. 
- Replace `Include/README.rst` with a link to a devguide page with the same info

Doc/c-api/apiabiversion.rst

@@ -6,34 +6,57 @@
 API and ABI Versioning
 ***********************
 
-``PY_VERSION_HEX`` is the Python version number encoded in a single integer.
+CPython exposes its version number in the following macros.
+Note that these correspond to the version code is **built** with,
+not necessarily the version used at **run time**.
 
-For example if the ``PY_VERSION_HEX`` is set to ``0x030401a2``, the underlying
+See :ref:`stable` for a discussion of API and ABI stability across versions.
-version information can be found by treating it as a 32 bit number in
-the following manner:
 
-   +-------+-------------------------+------------------------------------------------+
+.. c:macro:: PY_MAJOR_VERSION
-   | Bytes | Bits (big endian order) | Meaning                                        |
+
-   +=======+=========================+================================================+
+   The ``3`` in ``3.4.1a2``.
-   | ``1`` |       ``1-8``           |  ``PY_MAJOR_VERSION`` (the ``3`` in            |
+
-   |       |                         |  ``3.4.1a2``)                                  |
+.. c:macro:: PY_MINOR_VERSION
-   +-------+-------------------------+------------------------------------------------+
+
-   | ``2`` |       ``9-16``          |  ``PY_MINOR_VERSION`` (the ``4`` in            |
+   The ``4`` in ``3.4.1a2``.
-   |       |                         |  ``3.4.1a2``)                                  |
+
-   +-------+-------------------------+------------------------------------------------+
+.. c:macro:: PY_MICRO_VERSION
-   | ``3`` |       ``17-24``         |  ``PY_MICRO_VERSION`` (the ``1`` in            |
+
-   |       |                         |  ``3.4.1a2``)                                  |
+   The ``1`` in ``3.4.1a2``.
-   +-------+-------------------------+------------------------------------------------+
+
-   | ``4`` |       ``25-28``         |  ``PY_RELEASE_LEVEL`` (``0xA`` for alpha,      |
+.. c:macro:: PY_RELEASE_LEVEL
-   |       |                         |  ``0xB`` for beta, ``0xC`` for release         |
+
-   |       |                         |  candidate and ``0xF`` for final), in this     |
+   The ``a`` in ``3.4.1a2``.
-   |       |                         |  case it is alpha.                             |
+   This can be ``0xA`` for alpha, ``0xB`` for beta, ``0xC`` for release
-   +-------+-------------------------+------------------------------------------------+
+   candidate or ``0xF`` for final.
-   |       |       ``29-32``         |  ``PY_RELEASE_SERIAL`` (the ``2`` in           |
+
-   |       |                         |  ``3.4.1a2``, zero for final releases)         |
+.. c:macro:: PY_RELEASE_SERIAL
-   +-------+-------------------------+------------------------------------------------+
+
+   The ``2`` in ``3.4.1a2``. Zero for final releases.
+
+.. c:macro:: PY_VERSION_HEX
+
+   The Python version number encoded in a single integer.
+
+   The underlying version information can be found by treating it as a 32 bit
+   number in the following manner:
+
+   +-------+-------------------------+-------------------------+--------------------------+
+   | Bytes | Bits (big endian order) | Meaning                 | Value for ``3.4.1a2``    |
+   +=======+=========================+=========================+==========================+
+   |   1   |         1-8             |  ``PY_MAJOR_VERSION``   | ``0x03``                 |
+   +-------+-------------------------+-------------------------+--------------------------+
+   |   2   |         9-16            |  ``PY_MINOR_VERSION``   | ``0x04``                 |
+   +-------+-------------------------+-------------------------+--------------------------+
+   |   3   |         17-24           |  ``PY_MICRO_VERSION``   | ``0x01``                 |
+   +-------+-------------------------+-------------------------+--------------------------+
+   |   4   |         25-28           |  ``PY_RELEASE_LEVEL``   | ``0xA``                  |
+   +       +-------------------------+-------------------------+--------------------------+
+   |       |         29-32           |  ``PY_RELEASE_SERIAL``  | ``0x2``                  |
+   +-------+-------------------------+-------------------------+--------------------------+
+
+   Thus ``3.4.1a2`` is hexversion ``0x030401a2`` and ``3.10.0`` is
+   hexversion ``0x030a00f0``.
 
-Thus ``3.4.1a2`` is hexversion ``0x030401a2``.
 
 All the given macros are defined in :source:`Include/patchlevel.h`.
-

Doc/c-api/stable.rst

@@ -2,37 +2,157 @@
 
 .. _stable:
 
-***********************************
+***************
+C API Stability
+***************
+
+Python's C API is covered by the Backwards Compatibility Policy, :pep:`387`.
+While the C API will change with every minor release (e.g. from 3.9 to 3.10),
+most changes will be source-compatible, typically by only adding new API.
+Changing existing API or removing API is only done after a deprecation period
+or to fix serious issues.
+
+CPython's Application Binary Interface (ABI) is forward- and
+backwards-compatible across a minor release (if these are compiled the same
+way; see :ref:`stable-abi-platform` below).
+So, code compiled for Python 3.10.0 will work on 3.10.8 and vice versa,
+but will need to be compiled separately for 3.9.x and 3.10.x.
+
+Names prefixed by an underscore, such as ``_Py_InternalState``,
+are private API that can change without notice even in patch releases.
+
+
 Stable Application Binary Interface
-***********************************
+===================================
 
-Traditionally, the C API of Python will change with every release.  Most changes
+Python 3.2 introduced the *Limited API*, a subset of Python's C API.
-will be source-compatible, typically by only adding API, rather than changing
+Extensions that only use the Limited API can be
-existing API or removing API (although some interfaces do get removed after
+compiled once and work with multiple versions of Python.
-being deprecated first).
+Contents of the Limited API are :ref:`listed below <stable-abi-list>`.
 
-Unfortunately, the API compatibility does not extend to binary compatibility
+To enable this, Python provides a *Stable ABI*: a set of symbols that will
-(the ABI). The reason is primarily the evolution of struct definitions, where
+remain compatible across Python 3.x versions. The Stable ABI contains symbols
-addition of a new field, or changing the type of a field, might not break the
+exposed in the Limited API, but also other ones – for example, functions
-API, but can break the ABI.  As a consequence, extension modules need to be
+necessary to support older versions of the Limited API.
-recompiled for every Python release (although an exception is possible on Unix
-when none of the affected interfaces are used). In addition, on Windows,
-extension modules link with a specific pythonXY.dll and need to be recompiled to
-link with a newer one.
 
-Since Python 3.2, a subset of the API has been declared to guarantee a stable
+(For simplicity, this document talks about *extensions*, but the Limited API
-ABI. Extension modules wishing to use this API (called "limited API") need to
+and Stable ABI work the same way for all uses of the API – for example,
-define ``Py_LIMITED_API``. A number of interpreter details then become hidden
+embedding Python.)
-from the extension module; in return, a module is built that works on any 3.x
-version (x>=2) without recompilation.
 
-In some cases, the stable ABI needs to be extended with new functions.
+.. c:macro:: Py_LIMITED_API
-Extension modules wishing to use these new APIs need to set ``Py_LIMITED_API``
-to the ``PY_VERSION_HEX`` value (see :ref:`apiabiversion`) of the minimum Python
-version they want to support (e.g. ``0x03030000`` for Python 3.3). Such modules
-will work on all subsequent Python releases, but fail to load (because of
-missing symbols) on the older releases.
 
-As of Python 3.2, the set of functions available to the limited API is
+   Define this macro ``Py_LIMITED_API`` before including ``Python.h`` to
-documented in :pep:`384`.  In the C API documentation, API elements that are not
+   opt in to only use the Limited API.
-part of the limited API are marked as "Not part of the limited API."
+
+   Defining ``Py_LIMITED_API`` to ``3`` will limit the available API so that
+   the extension will work without recompilation with all Python 3.x releases
+   (x>=2) on the particular  :ref:`platform <stable-abi-platform>`.
+
+   Defining ``Py_LIMITED_API`` to a value of :c:data:`PY_VERSION_HEX` will
+   limit the available API so that the extension will work without
+   recompilation with all Python 3 releases from the specified one.
+   This will allow using additional API introduced up to this version,
+   but the extension will lose compatibility with earlier Python versions.
+   Rather than using the ``PY_VERSION_HEX`` macro directly, hardcode a minimum
+   minor version (e.g. ``0x030A0000`` for Python 3.10) for stability when
+   compiling with future Python versions.
+
+On Windows, extensions that use the Stable ABI should be linked against
+``python3.dll`` rather than a version-specific library such as
+``python39.dll``.
+
+On some platforms, Python will look for and load shared library files named
+with the ``abi3`` tag (e.g. ``mymodule.abi3.so``).
+It does not check if such extensions conform to a Stable ABI.
+The user (or their packaging tools) need to ensure that, for example,
+extensions built with the 3.10+ Limited API are not installed for lower
+versions of Python.
+
+All functions in the Stable ABI are present as functions in Python's shared
+library, not solely as macros. This makes them usable from languages that don't
+use the C preprocessor.
+
+
+Limited API Scope and Performance
+---------------------------------
+
+The goal for the Limited API is to allow everything that is possible with the
+full C API, but possibly with a performance penalty.
+
+For example, while :c:func:`PyList_GetItem` is available, its “unsafe” macro
+variant :c:func:`PyList_GET_ITEM` is not.
+The macro can be faster because it can rely on version-specific implementation
+details of the list object.
+
+Without ``Py_LIMITED_API`` defined, some C API functions are inlined or
+replaced by macros.
+Defining ``Py_LIMITED_API`` disables this inlining, allowing stability as
+Python's data structures are improved, but possibly reducing performance.
+
+By leaving out the ``Py_LIMITED_API`` definition, it is possible to compile
+a Limited API extension with a version-specific ABI. This can improve
+performance for that Python version, but will limit compatibility.
+Compiling with ``Py_LIMITED_API`` will then yield an extension that can be
+distributed where a version-specific one is not available – for example,
+for prereleases of an upcoming Python version.
+
+
+Limited API Caveats
+-------------------
+
+Note that compiling with ``Py_LIMITED_API`` is *not* a complete guarantee that
+code conforms to the Limited API or the Stable ABI. ``Py_LIMITED_API`` only
+covers definitions, but an API also includes other issues, such as expected
+semantics.
+
+One issue that ``Py_LIMITED_API`` does not guard against is calling a function
+with arguments that are invalid in a lower Python version.
+For example, consider a function that starts accepting ``NULL`` for an
+argument. In Python 3.9, ``NULL`` now selects a default behavior, but in
+Python 3.8, the argument will be used directly, causing a ``NULL`` dereference
+and crash. A similar argument works for fields of structs.
+
+Another issue is that some struct fields are currently not hidden when
+``Py_LIMITED_API`` is defined, even though they're part of the Limited API.
+
+For these reasons, we recommend testing an extension with *all* minor Python
+versions it supports, and preferably to build with the *lowest* such version.
+
+We also recommend reviewing documentation of all used API to check
+if it is explicitly part of the Limited API. Even with ``Py_LIMITED_API``
+defined, a few private declarations are exposed for technical reasons (or
+even unintentionally, as bugs).
+
+Also note that the Limited API is not necessarily stable: compiling with
+``Py_LIMITED_API`` with Python 3.8 means that the extension will
+run with Python 3.12, but it will not necessarily *compile* with Python 3.12.
+In particular, parts of the Limited API may be deprecated and removed,
+provided that the Stable ABI stays stable.
+
+
+.. _stable-abi-platform:
+
+Platform Considerations
+=======================
+
+ABI stability depends not only on Python, but also on the compiler used,
+lower-level libraries and compiler options. For the purposes of the Stable ABI,
+these details define a “platform”. They usually depend on the OS
+type and processor architecture
+
+It is the responsibility of each particular distributor of Python
+to ensure that all Python versions on a particular platform are built
+in a way that does not break the Stable ABI.
+This is the case with Windows and macOS releases from ``python.org`` and many
+third-party distributors.
+
+
+.. _stable-abi-list:
+
+Contents of Limited API
+=======================
+
+
+Currently, the Limited API includes the following items:
+
+.. limited-api-list::

Doc/conf.py

@@ -225,8 +225,9 @@ linkcheck_ignore = [r'https://bugs.python.org/(issue)?\d+',
 # Options for extensions
 # ----------------------
 
-# Relative filename of the reference count data file.
+# Relative filename of the data files
 refcount_file = 'data/refcounts.dat'
+stable_abi_file = 'data/stable_abi.dat'
 
 # Sphinx 2 and Sphinx 3 compatibility
 # -----------------------------------

Doc/tools/extensions/c_annotations.py

@@ -10,8 +10,10 @@
 
     * stable API annotations
 
-    Usage: Set the `refcount_file` config value to the path to the reference
+    Usage:
+    * Set the `refcount_file` config value to the path to the reference
     count data file.
+    * Set the `stable_abi_file` config value to the path to stable ABI list.
 
     :copyright: Copyright 2007-2014 by Georg Brandl.
     :license: Python license.
@@ -20,11 +22,23 @@
 from os import path
 from docutils import nodes
 from docutils.parsers.rst import directives
+from docutils.parsers.rst import Directive
+from docutils.statemachine import StringList
+import csv
 
 from sphinx import addnodes
 from sphinx.domains.c import CObject
 
 
+REST_ROLE_MAP = {
+    'function': 'func',
+    'var': 'data',
+    'type': 'type',
+    'macro': 'macro',
+    'type': 'type',
+}
+
+
 class RCEntry:
     def __init__(self, name):
         self.name = name
@@ -33,12 +47,10 @@ class RCEntry:
         self.result_refs = None
 
 
-class Annotations(dict):
+class Annotations:
-    @classmethod
+    def __init__(self, refcount_filename, stable_abi_file):
-    def fromfile(cls, filename):
+        self.refcount_data = {}
-        d = cls()
+        with open(refcount_filename, 'r') as fp:
-        fp = open(filename, 'r')
-        try:
             for line in fp:
                 line = line.strip()
                 if line[:1] in ("", "#"):
@@ -50,9 +62,9 @@ class Annotations(dict):
                 function, type, arg, refcount, comment = parts
                 # Get the entry, creating it if needed:
                 try:
-                    entry = d[function]
+                    entry = self.refcount_data[function]
                 except KeyError:
-                    entry = d[function] = RCEntry(function)
+                    entry = self.refcount_data[function] = RCEntry(function)
                 if not refcount or refcount == "null":
                     refcount = None
                 else:
@@ -64,27 +76,58 @@ class Annotations(dict):
                 else:
                     entry.result_type = type
                     entry.result_refs = refcount
-        finally:
+
-            fp.close()
+        self.stable_abi_data = {}
-        return d
+        with open(stable_abi_file, 'r') as fp:
+            for record in csv.DictReader(fp):
+                role = record['role']
+                name = record['name']
+                self.stable_abi_data[name] = record
 
     def add_annotations(self, app, doctree):
         for node in doctree.traverse(addnodes.desc_content):
             par = node.parent
             if par['domain'] != 'c':
                 continue
-            if par['stableabi']:
-                node.insert(0, nodes.emphasis(' Part of the stable ABI.',
-                                              ' Part of the stable ABI.',
-                                              classes=['stableabi']))
-            if par['objtype'] != 'function':
-                continue
             if not par[0].has_key('ids') or not par[0]['ids']:
                 continue
             name = par[0]['ids'][0]
             if name.startswith("c."):
                 name = name[2:]
-            entry = self.get(name)
+
+            objtype = par['objtype']
+
+            # Stable ABI annotation. These have two forms:
+            #   Part of the [Stable ABI](link).
+            #   Part of the [Stable ABI](link) since version X.Y.
+            record = self.stable_abi_data.get(name)
+            if record:
+                if record['role'] != objtype:
+                    raise ValueError(
+                        f"Object type mismatch in limited API annotation "
+                        f"for {name}: {record['role']!r} != {objtype!r}")
+                stable_added = record['added']
+                message = ' Part of the '
+                emph_node = nodes.emphasis(message, message,
+                                           classes=['stableabi'])
+                ref_node = addnodes.pending_xref(
+                    'Stable ABI', refdomain="std", reftarget='stable',
+                    reftype='ref', refexplicit="False")
+                ref_node += nodes.Text('Stable ABI')
+                emph_node += ref_node
+                if record['ifdef_note']:
+                    emph_node += nodes.Text(' ' + record['ifdef_note'])
+                if stable_added == '3.2':
+                    # Stable ABI was introduced in 3.2.
+                    emph_node += nodes.Text('.')
+                else:
+                    emph_node += nodes.Text(f' since version {stable_added}.')
+                node.insert(0, emph_node)
+
+            # Return value annotation
+            if objtype != 'function':
+                continue
+            entry = self.refcount_data.get(name)
             if not entry:
                 continue
             elif not entry.result_type.endswith("Object*"):
@@ -99,13 +142,36 @@ class Annotations(dict):
 
 
 def init_annotations(app):
-    refcounts = Annotations.fromfile(
+    annotations = Annotations(
-        path.join(app.srcdir, app.config.refcount_file))
+        path.join(app.srcdir, app.config.refcount_file),
-    app.connect('doctree-read', refcounts.add_annotations)
+        path.join(app.srcdir, app.config.stable_abi_file),
+    )
+    app.connect('doctree-read', annotations.add_annotations)
+
+    class LimitedAPIList(Directive):
+
+        has_content = False
+        required_arguments = 0
+        optional_arguments = 0
+        final_argument_whitespace = True
+
+        def run(self):
+            content = []
+            for record in annotations.stable_abi_data.values():
+                role = REST_ROLE_MAP[record['role']]
+                name = record['name']
+                content.append(f'* :c:{role}:`{name}`')
+
+            pnode = nodes.paragraph()
+            self.state.nested_parse(StringList(content), 0, pnode)
+            return [pnode]
+
+    app.add_directive('limited-api-list', LimitedAPIList)
 
 
 def setup(app):
     app.add_config_value('refcount_file', '', True)
+    app.add_config_value('stable_abi_file', '', True)
     app.connect('builder-inited', init_annotations)
 
     # monkey-patch C object...

Include/README.rst

@@ -3,66 +3,10 @@ The Python C API
 
 The C API is divided into three sections:
 
-1. ``Include/``
+1. ``Include/``: Limited API
-2. ``Include/cpython/``
+2. ``Include/cpython/``: CPython implementation details
-3. ``Include/internal/``
+3. ``Include/internal/``: The internal API
 
+Information on changing the C API is available `in the developer guide`_
 
-Include: Limited API
+.. _in the developer guide: https://devguide.python.org/c-api/
-====================
-
-``Include/``, excluding the ``cpython`` and ``internal`` subdirectories,
-contains the public Limited API (Application Programming Interface).
-The Limited API is a subset of the C API, designed to guarantee ABI
-stability across Python 3 versions, and is defined in :pep:`384`.
-
-Guidelines for expanding the Limited API:
-
-- Functions *must not* steal references
-- Functions *must not* return borrowed references
-- Functions returning references *must* return a strong reference
-- Macros should not expose implementation details
-- Please start a public discussion before expanding the API
-- Functions or macros with a ``_Py`` prefix do not belong in ``Include/``.
-
-It is possible to add a function or macro to the Limited API from a
-given Python version.  For example, to add a function to the Limited API
-from Python 3.10 and onwards, wrap it with
-``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A0000``.
-
-
-Include/cpython: CPython implementation details
-===============================================
-
-``Include/cpython/`` contains the public API that is excluded from the
-Limited API and the Stable ABI.
-
-Guidelines for expanding the public API:
-
-- Functions *must not* steal references
-- Functions *must not* return borrowed references
-- Functions returning references *must* return a strong reference
-
-
-Include/internal: The internal API
-==================================
-
-
-With PyAPI_FUNC or PyAPI_DATA
------------------------------
-
-Functions or structures in ``Include/internal/`` defined with
-``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are
-exposed only for specific use cases like debuggers and profilers.
-
-
-With the extern keyword
------------------------
-
-Functions in ``Include/internal/`` defined with the ``extern`` keyword
-*must not and can not* be used outside the CPython code base.  Only
-built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN``
-macro defined) can use such functions.
-
-When in doubt, new internal C functions should be defined in
-``Include/internal`` using the ``extern`` keyword.

Tools/scripts/stable_abi.py

@@ -21,6 +21,7 @@ import os
 import os.path
 import io
 import re
+import csv
 
 MISSING = object()
 
@@ -45,6 +46,11 @@ EXCLUDED_HEADERS = {
 MACOS = (sys.platform == "darwin")
 UNIXY = MACOS or (sys.platform == "linux")  # XXX should this be "not Windows"?
 
+IFDEF_DOC_NOTES = {
+    'MS_WINDOWS': 'on Windows',
+    'HAVE_FORK': 'on platforms with fork()',
+    'USE_STACKCHECK': 'on platforms with USE_STACKCHECK',
+}
 
 # The stable ABI manifest (Misc/stable_abi.txt) exists only to fill the
 # following dataclasses.
@@ -227,16 +233,31 @@ def gen_python3dll(manifest, args, outfile):
             key=sort_key):
         write(f'EXPORT_DATA({item.name})')
 
+REST_ROLES = {
+    'function': 'function',
+    'data': 'var',
+    'struct': 'type',
+    'macro': 'macro',
+    # 'const': 'const',  # all undocumented
+    'typedef': 'type',
+}
 
 @generator("doc_list", 'Doc/data/stable_abi.dat')
 def gen_doc_annotations(manifest, args, outfile):
     """Generate/check the stable ABI list for documentation annotations"""
-    write = partial(print, file=outfile)
+    writer = csv.DictWriter(
-    write("# Generated by Tools/scripts/stable_abi.py")
+        outfile, ['role', 'name', 'added', 'ifdef_note'], lineterminator='\n')
-    write()
+    writer.writeheader()
-    for item in manifest.select(ABIItem.KINDS, include_abi_only=False):
+    for item in manifest.select(REST_ROLES.keys(), include_abi_only=False):
-        write(item.name)
+        if item.ifdef:
-
+            ifdef_note = IFDEF_DOC_NOTES[item.ifdef]
+        else:
+            ifdef_note = None
+        writer.writerow({
+            'role': REST_ROLES[item.kind],
+            'name': item.name,
+            'added': item.added,
+            'ifdef_note': ifdef_note})
 
 def generate_or_check(manifest, args, path, func):
     """Generate/check a file with a single generator