cpython/Include
Petr Viktorin 49f6beb56a
[3.12] gh-113993: Make interned strings mortal (GH-120520, GH-121364, GH-121903, GH-122303) (#123065)
This backports several PRs for gh-113993, making interned strings mortal so they can be garbage-collected when no longer needed.

* Allow interned strings to be mortal, and fix related issues (GH-120520)

  * Add an InternalDocs file describing how interning should work and how to use it.

  * Add internal functions to *explicitly* request what kind of interning is done:
    - `_PyUnicode_InternMortal`
    - `_PyUnicode_InternImmortal`
    - `_PyUnicode_InternStatic`

  * Switch uses of `PyUnicode_InternInPlace` to those.

  * Disallow using `_Py_SetImmortal` on strings directly.
    You should use `_PyUnicode_InternImmortal` instead:
    - Strings should be interned before immortalization, otherwise you're possibly
      interning a immortalizing copy.
    - `_Py_SetImmortal` doesn't handle the `SSTATE_INTERNED_MORTAL` to
      `SSTATE_INTERNED_IMMORTAL` update, and those flags can't be changed in
      backports, as they are now part of public API and version-specific ABI.

  * Add private `_only_immortal` argument for `sys.getunicodeinternedsize`, used in refleak test machinery.

   Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:
    - `_Py_ID`
    - `_Py_STR` (including the empty string)
    - one-character latin-1 singletons

    Now, when you intern a singleton, that exact singleton will be interned.

  * Add a `_Py_LATIN1_CHR` macro, use it instead of `_Py_ID`/`_Py_STR` for one-character latin-1 singletons everywhere (including Clinic).

  * Intern `_Py_STR` singletons at startup.

  * Beef up the tests. Cover internal details (marked with `@cpython_only`).

  * Add lots of assertions

* Don't immortalize in PyUnicode_InternInPlace; keep immortalizing in other API (GH-121364)

  * Switch PyUnicode_InternInPlace to _PyUnicode_InternMortal, clarify docs

  * Document immortality in some functions that take `const char *`

  This is PyUnicode_InternFromString;
  PyDict_SetItemString, PyObject_SetAttrString;
  PyObject_DelAttrString; PyUnicode_InternFromString;
  and the PyModule_Add convenience functions.

  Always point out a non-immortalizing alternative.

  * Don't immortalize user-provided attr names in _ctypes

* Immortalize names in code objects to avoid crash (GH-121903)

* Intern latin-1 one-byte strings at startup (GH-122303)

There are some 3.12-specific changes, mainly to allow statically allocated strings in deepfreeze. (In 3.13, deepfreeze switched to the general `_Py_ID`/`_Py_STR`.)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2024-09-27 13:28:48 -07:00
..
cpython [3.12] gh-116767: fix crash on 'async with' with many context managers (GH-118348) (#118477) 2024-05-01 18:23:29 +01:00
internal [3.12] gh-113993: Make interned strings mortal (GH-120520, GH-121364, GH-121903, GH-122303) (#123065) 2024-09-27 13:28:48 -07:00
abstract.h
bltinmodule.h
boolobject.h [3.12] gh-106560: Fix redundant declarations in Include/ (#112611) (#112650) 2023-12-03 11:45:32 +00:00
bytearrayobject.h
bytesobject.h
ceval.h
codecs.h
compile.h [3.12] gh-109596: Ensure repeated rules in the grammar are not allowed and fix incorrect soft keywords (GH-109606). (#109752) 2023-10-02 17:22:07 +02:00
complexobject.h
datetime.h
descrobject.h gh-103509: PEP 697 -- Limited C API for Extending Opaque Types (GH-103511) 2023-05-04 09:56:53 +02:00
dictobject.h
dynamic_annotations.h
enumobject.h
errcode.h [3.12] gh-107450: Check for overflow in the tokenizer and fix overflow test (GH-110832) (#110931) 2023-10-16 18:59:18 +02:00
exports.h
fileobject.h [3.12] gh-77782: Deprecate Py_HasFileSystemDefaultEncoding (GH-106272) (#106274) 2023-06-30 10:21:36 +00:00
fileutils.h
floatobject.h [3.12] gh-121905: Consistently use "floating-point" instead of "floating point" (GH-121907) (GH-122013) 2024-07-19 09:08:33 +00:00
frameobject.h
genericaliasobject.h
import.h
interpreteridobject.h [3.12] gh-101524: Only Use Public C-API in the _xxsubinterpreters Module (gh-105258) (gh-107303) 2023-07-27 13:15:47 -06:00
intrcheck.h
iterobject.h
listobject.h
longobject.h [3.12] gh-117021: Fix integer overflow in PyLong_AsPid() on non-Windows 64-bit platforms (GH-117064) (GH-117070) 2024-03-20 16:44:05 +02:00
marshal.h
memoryobject.h
methodobject.h
modsupport.h [3.12] gh-107226: PyModule_AddObjectRef() should only be in the limited API 3.10 (GH-107227) (GH-107260) 2023-07-25 23:01:18 +03:00
moduleobject.h [3.12] gh-111698: Restrict Py_mod_multiple_interpreters to 3.12+ Under Py_LIMITED_API (gh-111707) (gh-111787) 2023-11-27 19:42:27 -07:00
object.h [3.12] gh-116869: Make C API compatible with ISO C90 (GH-116950) (#117011) 2024-03-19 16:06:15 +00:00
objimpl.h
opcode.h gh-103865: add monitoring support to LOAD_SUPER_ATTR (#103866) 2023-05-16 10:29:00 -06:00
osdefs.h
osmodule.h
patchlevel.h Post 3.12.6 2024-09-07 00:22:27 +02:00
py_curses.h
pybuffer.h gh-102500: Implement PEP 688 (#102521) 2023-05-04 07:59:46 -07:00
pycapsule.h
pydtrace.d
pydtrace.h
pyerrors.h
pyexpat.h [3.12] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) (GH-115623) (GH-116248) 2024-03-06 22:01:45 +00:00
pyframe.h
pyhash.h
pylifecycle.h
pymacconfig.h [3.12] gh-110820: Disable test_signal.test_stress_modifying_handlers on macOS (GH-112834) 2023-12-09 15:53:16 +01:00
pymacro.h [3.12] gh-118124: fix assert related C++ checks on Solaris/Illumos (GH-121974) (#122109) 2024-07-22 07:45:17 +00:00
pymath.h gh-104263: Rely on Py_NAN and introduce Py_INFINITY (GH-104202) 2023-05-10 17:44:52 +01:00
pymem.h
pyport.h [3.12] gh-112536: Define _Py_THREAD_SANITIZER on GCC when TSan is enabled (GH-117702) (#117713) 2024-04-10 14:38:10 +00:00
pystate.h
pystats.h Remove useless symbol in pystats.h (#101864) 2023-04-09 14:13:21 +05:30
pystrcmp.h
pystrtod.h
Python.h [3.12] gh-117021: Fix integer overflow in PyLong_AsPid() on non-Windows 64-bit platforms (GH-117064) (GH-117070) 2024-03-20 16:44:05 +02:00
pythonrun.h
pythread.h
pytypedefs.h
rangeobject.h
README.rst
setobject.h
sliceobject.h
structmember.h
structseq.h
sysmodule.h gh-103295: expose API for writing perf map files (#103546) 2023-05-21 11:12:24 +01:00
traceback.h
tracemalloc.h GH-101520: Move tracemalloc functionality into core, leaving interface in Modules. (#104508) 2023-05-17 14:17:16 +01:00
tupleobject.h
typeslots.h
unicodeobject.h Fix typo in "expected" word in few source files (#104034) 2023-05-01 09:45:50 -06:00
warnings.h
weakrefobject.h

The Python C API
================

The C API is divided into these sections:

1. ``Include/``: Limited API
2. ``Include/cpython/``: CPython implementation details
3. ``Include/cpython/``, names with the ``PyUnstable_`` prefix: API that can
   change between minor releases
4. ``Include/internal/``, and any name with ``_`` prefix: The internal API

Information on changing the C API is available `in the developer guide`_

.. _in the developer guide: https://devguide.python.org/c-api/