[3.14] gh-139707: Add docs for optional modules (GH-140171) (GH-141204)

(cherry picked from commit d2ce6d708a)

Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Emma Smith <emma@emmatyping.dev>
Co-authored-by: Author: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Victor Stinner <vstinner@python.org>

Doc/glossary.rst

@@ -1025,6 +1025,15 @@ Glossary
       applied to all scopes, only those relying on a known set of local
       and nonlocal variable names are restricted to optimized scopes.
 
+   optional module
+      An :term:`extension module` that is part of the :term:`standard library`,
+      but may be absent in some builds of :term:`CPython`,
+      usually due to missing third-party libraries or because the module
+      is not available for a given platform.
+
+      See :ref:`optional-module-requirements` for a list of optional modules
+      that require third-party libraries.
+
    package
       A Python :term:`module` which can contain submodules or recursively,
       subpackages.  Technically, a package is a Python module with a

Doc/includes/optional-module.rst

@@ -0,0 +1,9 @@
+This is an :term:`optional module`.
+If it is missing from your copy of CPython,
+look for documentation from your distributor (that is,
+whoever provided Python to you).
+If you are the distributor, see :ref:`optional-module-requirements`.
+
+.. Similar notes appear in the docs of the modules:
+   - zipfile
+   - tarfile

Doc/library/bz2.rst

@@ -25,6 +25,8 @@ The :mod:`bz2` module contains:
 * The :func:`compress` and :func:`decompress` functions for one-shot
   (de)compression.
 
+.. include:: ../includes/optional-module.rst
+
 
 (De)compression of files
 ------------------------

Doc/library/compression.zstd.rst

@@ -33,6 +33,8 @@ The :mod:`!compression.zstd` module contains:
 * The :class:`CompressionParameter`, :class:`DecompressionParameter`, and
   :class:`Strategy` classes for setting advanced (de)compression parameters.
 
+.. include:: ../includes/optional-module.rst
+
 
 Exceptions
 ----------

Doc/library/ctypes.rst

@@ -14,6 +14,8 @@
 data types, and allows calling functions in DLLs or shared libraries.  It can be
 used to wrap these libraries in pure Python.
 
+.. include:: ../includes/optional-module.rst
+
 
 .. _ctypes-ctypes-tutorial:
 

Doc/library/curses.rst

@@ -23,6 +23,8 @@ Linux and the BSD variants of Unix.
 
 .. include:: ../includes/wasm-mobile-notavail.rst
 
+.. include:: ../includes/optional-module.rst
+
 .. note::
 
    Whenever the documentation mentions a *character* it can be specified

Doc/library/ensurepip.rst

@@ -30,6 +30,8 @@ when creating a virtual environment) or after explicitly uninstalling
    needed to bootstrap ``pip`` are included as internal parts of the
    package.
 
+.. include:: ../includes/optional-module.rst
+
 .. seealso::
 
    :ref:`installing-index`

Doc/library/gzip.rst

@@ -11,6 +11,8 @@
 This module provides a simple interface to compress and decompress files just
 like the GNU programs :program:`gzip` and :program:`gunzip` would.
 
+.. include:: ../includes/optional-module.rst
+
 The data compression is provided by the :mod:`zlib` module.
 
 The :mod:`gzip` module provides the :class:`GzipFile` class, as well as the

Doc/library/idle.rst

@@ -37,6 +37,10 @@ IDLE has the following features:
 
 * configuration, browsers, and other dialogs
 
+The IDLE application is implemented in the :mod:`idlelib` package.
+
+.. include:: ../includes/optional-module.rst
+
 Menus
 -----
 

Doc/library/lzma.rst

@@ -23,6 +23,8 @@ module. Note that :class:`LZMAFile` and :class:`bz2.BZ2File` are *not*
 thread-safe, so if you need to use a single :class:`LZMAFile` instance
 from multiple threads, it is necessary to protect it with a lock.
 
+.. include:: ../includes/optional-module.rst
+
 
 .. exception:: LZMAError
 

Doc/library/readline.rst

@@ -26,6 +26,8 @@ Readline library in general.
 
 .. include:: ../includes/wasm-mobile-notavail.rst
 
+.. include:: ../includes/optional-module.rst
+
 .. note::
 
   The underlying Readline library API may be implemented by

Doc/library/sqlite3.rst

@@ -31,7 +31,9 @@ PostgreSQL or Oracle.
 
 The :mod:`!sqlite3` module was written by Gerhard Häring.  It provides an SQL interface
 compliant with the DB-API 2.0 specification described by :pep:`249`, and
-requires SQLite 3.15.2 or newer.
+requires the third-party `SQLite <https://sqlite.org/>`_ library.
+
+.. include:: ../includes/optional-module.rst
 
 This document includes four main sections:
 

Doc/library/ssl.rst

@@ -18,8 +18,9 @@
 This module provides access to Transport Layer Security (often known as "Secure
 Sockets Layer") encryption and peer authentication facilities for network
 sockets, both client-side and server-side.  This module uses the OpenSSL
-library. It is available on all modern Unix systems, Windows, macOS, and
-probably additional platforms, as long as OpenSSL is installed on that platform.
+library.
+
+.. include:: ../includes/optional-module.rst
 
 .. note::
 

Doc/library/tarfile.rst

@@ -21,6 +21,14 @@ Some facts and figures:
 * reads and writes :mod:`gzip`, :mod:`bz2`, :mod:`compression.zstd`, and
   :mod:`lzma` compressed archives if the respective modules are available.
 
+  ..
+     The following paragraph should be similar to ../includes/optional-module.rst
+
+  If any of these :term:`optional modules <optional module>` are missing from
+  your copy of CPython, look for documentation from your distributor (that is,
+  whoever provided Python to you).
+  If you are the distributor, see :ref:`optional-module-requirements`.
+
 * read/write support for the POSIX.1-1988 (ustar) format.
 
 * read/write support for the GNU tar format including *longname* and *longlink*

Doc/library/tkinter.rst

@@ -36,6 +36,8 @@ details that are unchanged.
    Most documentation you will find online still uses the old API and
    can be woefully outdated.
 
+.. include:: ../includes/optional-module.rst
+
 .. seealso::
 
    * `TkDocs <https://tkdocs.com/>`_

Doc/library/turtle.rst

@@ -29,6 +29,8 @@ introduced in Logo <https://en.wikipedia.org/wiki/Turtle_
 (robot)>`_, developed by Wally Feurzeig, Seymour Papert and Cynthia Solomon
 in 1967.
 
+.. include:: ../includes/optional-module.rst
+
 
 Get started
 ===========

Doc/library/zipfile.rst

@@ -23,6 +23,16 @@ decryption of encrypted files in ZIP archives, but it currently cannot
 create an encrypted file.  Decryption is extremely slow as it is
 implemented in native Python rather than C.
 
+..
+   The following paragraph should be similar to ../includes/optional-module.rst
+
+Handling compressed archives requires :term:`optional modules <optional module>`
+such as :mod:`zlib`, :mod:`bz2`, :mod:`lzma`, and :mod:`compression.zstd`.
+If any of them are missing from your copy of CPython,
+look for documentation from your distributor (that is,
+whoever provided Python to you).
+If you are the distributor, see :ref:`optional-module-requirements`.
+
 The module defines the following items:
 
 .. exception:: BadZipFile

Doc/library/zlib.rst

@@ -8,11 +8,9 @@
 --------------
 
 For applications that require data compression, the functions in this module
-allow compression and decompression, using the zlib library. The zlib library
-has its own home page at https://www.zlib.net.   There are known
-incompatibilities between the Python module and versions of the zlib library
-earlier than 1.1.3; 1.1.3 has a `security vulnerability <https://zlib.net/zlib_faq.html#faq33>`_, so we recommend using
-1.1.4 or later.
+allow compression and decompression, using the `zlib library <https://www.zlib.net>`_.
+
+.. include:: ../includes/optional-module.rst
 
 zlib's functions have many options and often need to be used in a particular
 order.  This documentation doesn't attempt to cover all of the permutations;

Doc/using/configure.rst

@@ -4,10 +4,13 @@ Configure Python
 
 .. highlight:: sh
 
+
+.. _build-requirements:
+
 Build Requirements
 ==================
 
-Features and minimum versions required to build CPython:
+To build CPython, you will need:
 
 * A `C11 <https://en.cppreference.com/w/c/11>`_ compiler. `Optional C11
   features
@@ -22,87 +25,138 @@ Features and minimum versions required to build CPython:
 
 * Support for threads.
 
-To build optional modules:
-
-* `libbz2 <https://sourceware.org/bzip2/>`_ for the :mod:`bz2` module.
-
-* `libb2 <https://github.com/BLAKE2/libb2>`_ (:ref:`BLAKE2 <hashlib-blake2>`),
-  used by :mod:`hashlib` module.
-
-* `libffi <https://sourceware.org/libffi/>`_ 3.3.0 is the recommended
-  minimum version for the :mod:`ctypes` module.
-
-* ``liblzma``, for the :mod:`lzma` module.
-
-* `libmpdec <https://www.bytereef.org/mpdecimal/doc/libmpdec/>`_ 2.5.0
-  for the :mod:`decimal` module.
-
-* ``libncurses`` or ``libncursesw``,
-  for the :mod:`curses` module.
-
-* ``libpanel`` or ``libpanelw``,
-  for the :mod:`curses.panel` module.
-
-* `libreadline <https://tiswww.case.edu/php/chet/readline/rltop.html>`_ or
-  `libedit <https://www.thrysoee.dk/editline/>`_
-  for the :mod:`readline` module.
-
-* `libuuid <https://linux.die.net/man/3/libuuid>`_, for the :mod:`uuid` module.
-
-* `OpenSSL <https://www.openssl.org/>`_ 1.1.1 is the minimum version and
-  OpenSSL 3.0.18 is the recommended minimum version for the
-  :mod:`ssl` and :mod:`hashlib` extension modules.
-
-* `SQLite <https://sqlite.org/>`_ 3.15.2 for the :mod:`sqlite3` extension module.
-
-* `Tcl/Tk <https://www.tcl-lang.org/>`_ 8.5.12 for the :mod:`tkinter` module.
-
-* `zlib <https://www.zlib.net>`_ 1.1.4 is the reccomended minimum version for the
-  :mod:`zlib` module.
-
-* `zstd <https://facebook.github.io/zstd/>`_ 1.4.5 is the minimum version for
-  the :mod:`compression.zstd` module.
-
-For a full list of dependencies required to build all modules and how to install
-them, see the
-`devguide <https://devguide.python.org/getting-started/setup-building/#install-dependencies>`_.
-
-* Autoconf 2.72 and aclocal 1.16.5 are required to regenerate the
-  :file:`configure` script.
-
-.. versionchanged:: 3.1
-   Tcl/Tk version 8.3.1 is now required.
-
 .. versionchanged:: 3.5
    On Windows, Visual Studio 2015 or later is now required.
-   Tcl/Tk version 8.4 is now required.
 
 .. versionchanged:: 3.6
-   Selected C99 features are now required, like ``<stdint.h>`` and ``static
-   inline`` functions.
+   Selected C99 features, like ``<stdint.h>`` and ``static inline`` functions,
+   are now required.
 
 .. versionchanged:: 3.7
-   Thread support and OpenSSL 1.0.2 are now required.
-
-.. versionchanged:: 3.10
-   OpenSSL 1.1.1 is now required.
-   Require SQLite 3.7.15.
+   Thread support is now required.
 
 .. versionchanged:: 3.11
    C11 compiler, IEEE 754 and NaN support are now required.
    On Windows, Visual Studio 2017 or later is required.
-   Tcl/Tk version 8.5.12 is now required for the :mod:`tkinter` module.
-
-.. versionchanged:: 3.13
-   Autoconf 2.71, aclocal 1.16.5 and SQLite 3.15.2 are now required.
-
-.. versionchanged:: 3.14
-   Autoconf 2.72 is now required.
 
 See also :pep:`7` "Style Guide for C Code" and :pep:`11` "CPython platform
 support".
 
 
+.. _optional-module-requirements:
+
+Requirements for optional modules
+---------------------------------
+
+Some :term:`optional modules <optional module>` of the standard library
+require third-party libraries installed for development
+(for example, header files must be available).
+
+Missing requirements are reported in the ``configure`` output.
+Modules that are missing due to missing dependencies are listed near the end
+of the ``make`` output,
+sometimes using an internal name, for example, ``_ctypes`` for :mod:`ctypes`
+module.
+
+If you distribute a CPython interpreter without optional modules,
+it's best practice to advise users, who generally expect that
+standard library modules are available.
+
+Dependencies to build optional modules are:
+
+.. list-table::
+   :header-rows: 1
+   :align: left
+
+   * - Dependency
+     - Minimum version
+     - Python module
+   * - `libbz2 <https://sourceware.org/bzip2/>`_
+     -
+     - :mod:`bz2`
+   * - `libffi <https://sourceware.org/libffi/>`_
+     - 3.3.0 recommended
+     - :mod:`ctypes`
+   * - `liblzma <https://tukaani.org/xz/>`_
+     -
+     - :mod:`lzma`
+   * - `libmpdec <https://www.bytereef.org/mpdecimal/doc/libmpdec/>`_
+     - 2.5.0
+     - :mod:`decimal` [1]_
+   * - `libreadline <https://tiswww.case.edu/php/chet/readline/rltop.html>`_ or
+       `libedit <https://www.thrysoee.dk/editline/>`_ [2]_
+     -
+     - :mod:`readline`
+   * - `libuuid <https://linux.die.net/man/3/libuuid>`_
+     -
+     - ``_uuid`` [3]_
+   * - `ncurses <https://gnu.org/software/ncurses/ncurses.html>`_ [4]_
+     -
+     - :mod:`curses`
+   * - `OpenSSL <https://openssl-library.org/>`_
+     - | 3.0.18 recommended
+       | (1.1.1 minimum)
+     - :mod:`ssl`, :mod:`hashlib` [5]_
+   * - `SQLite <https://sqlite.org/>`_
+     - 3.15.2
+     - :mod:`sqlite3`
+   * - `Tcl/Tk <https://www.tcl-lang.org/>`_
+     - 8.5.12
+     - :mod:`tkinter`, :ref:`IDLE <idle>`, :mod:`turtle`
+   * - `zlib <https://www.zlib.net>`_
+     - 1.2.2.1
+     - :mod:`zlib`, :mod:`gzip`, :mod:`ensurepip`
+   * - `zstd <https://facebook.github.io/zstd/>`_
+     - 1.4.5
+     - :mod:`compression.zstd`
+
+.. [1] If *libmpdec* is not available, the :mod:`decimal` module will use
+   a pure-Python implementation.
+   See :option:`--with-system-libmpdec` for details.
+.. [2] See :option:`--with-readline` for choosing the backend for the
+   :mod:`readline` module.
+.. [3] The :mod:`uuid` module uses ``_uuid`` to generate "safe" UUIDs.
+   See the module documentation for details.
+.. [4] The :mod:`curses` module requires the ``libncurses`` or ``libncursesw``
+   library.
+   The :mod:`curses.panel` module additionally requires the ``libpanel`` or
+   ``libpanelw`` library.
+.. [5] If OpenSSL is not available, the :mod:`hashlib` module will use
+   bundled implementations of several hash functions.
+   See :option:`--with-builtin-hashlib-hashes` for *forcing* usage of OpenSSL.
+
+Note that the table does not include all optional modules; in particular,
+platform-specific modules like :mod:`winreg` are not listed here.
+
+.. seealso::
+
+   * The `devguide <https://devguide.python.org/getting-started/setup-building/#install-dependencies>`_
+     includes a full list of dependencies required to build all modules and
+     instructions on how to install them on common platforms.
+   * :option:`--with-system-expat` allows building with an external
+     `libexpat <https://libexpat.github.io/>`_ library.
+   * :ref:`configure-options-for-dependencies`
+
+.. versionchanged:: 3.1
+   Tcl/Tk version 8.3.1 is now required for :mod:`tkinter`.
+
+.. versionchanged:: 3.5
+   Tcl/Tk version 8.4 is now required for :mod:`tkinter`.
+
+.. versionchanged:: 3.7
+   OpenSSL 1.0.2 is now required for :mod:`hashlib` and :mod:`ssl`.
+
+.. versionchanged:: 3.10
+   OpenSSL 1.1.1 is now required for :mod:`hashlib` and :mod:`ssl`.
+   SQLite 3.7.15 is now required for :mod:`sqlite3`.
+
+.. versionchanged:: 3.11
+   Tcl/Tk version 8.5.12 is now required for :mod:`tkinter`.
+
+.. versionchanged:: 3.13
+   SQLite 3.15.2 is now required for :mod:`sqlite3`.
+
+
 Generated files
 ===============
 
@@ -129,8 +183,19 @@ The container is optional, the following command can be run locally::
 
     autoreconf -ivf -Werror
 
-The generated files can change depending on the exact ``autoconf-archive``,
-``aclocal`` and ``pkg-config`` versions.
+The generated files can change depending on the exact versions of the
+tools used.
+The container that CPython uses has
+`Autoconf <https://gnu.org/software/autoconf>`_ 2.72,
+``aclocal`` from `Automake <https://www.gnu.org/software/automake>`_ 1.16.5,
+and `pkg-config <https://www.freedesktop.org/wiki/Software/pkg-config/>`_ 1.8.1.
+
+.. versionchanged:: 3.13
+   Autoconf 2.71 and aclocal 1.16.5 and are now used to regenerate
+   :file:`configure`.
+
+.. versionchanged:: 3.14
+   Autoconf 2.72 is now used to regenerate :file:`configure`.
 
 
 .. _configure-options:
@@ -408,6 +473,8 @@ Linker options
    Name for machine-dependent library files.
 
 
+.. _configure-options-for-dependencies:
+
 Options for third-party dependencies
 ------------------------------------
 
@@ -430,12 +497,6 @@ Options for third-party dependencies
 
    C compiler and linker flags for ``gdbm``.
 
-.. option:: LIBB2_CFLAGS
-.. option:: LIBB2_LIBS
-
-   C compiler and linker flags for ``libb2`` (:ref:`BLAKE2 <hashlib-blake2>`),
-   used by :mod:`hashlib` module, overriding ``pkg-config``.
-
 .. option:: LIBEDIT_CFLAGS
 .. option:: LIBEDIT_LIBS
 
@@ -901,6 +962,13 @@ Libraries options
    .. versionchanged:: 3.13
       Default to using the installed ``mpdecimal`` library.
 
+   .. versionchanged:: 3.15
+
+      A bundled copy of the library will no longer be selected
+      implicitly if an installed ``mpdecimal`` library is not found.
+      In Python 3.15 only, it can still be selected explicitly using
+      ``--with-system-libmpdec=no`` or ``--without-system-libmpdec``.
+
    .. deprecated-removed:: 3.13 3.16
       A copy of the ``mpdecimal`` library sources will no longer be distributed
       with Python 3.16.

Doc/using/unix.rst

@@ -84,11 +84,17 @@ On FreeBSD and OpenBSD
 Building Python
 ===============
 
+.. seealso::
+
+   If you want to contribute to CPython, refer to the
+   `devguide <https://devguide.python.org/getting-started/setup-building/>`_,
+   which includes build instructions and other tips on setting up environment.
+
 If you want to compile CPython yourself, first thing you should do is get the
 `source <https://www.python.org/downloads/source/>`_. You can download either the
-latest release's source or just grab a fresh `clone
-<https://devguide.python.org/setup/#get-the-source-code>`_.  (If you want
-to contribute patches, you will need a clone.)
+latest release's source or grab a fresh `clone
+<https://devguide.python.org/setup/#get-the-source-code>`_.
+You will also need to install the :ref:`build requirements <build-requirements>`.
 
 The build process consists of the usual commands::