mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-08-04 17:08:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 13

Merge branch 'main' into unicode-linebreaks

Browse source
This commit is contained in:
L. Grobol 2025-05-24 19:35:05 +02:00 • committed by GitHub
commit 286ab7ba6e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1256 changed files with 80530 additions and 25989 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.azure-pipelines/ci.yml
View file

@ -1,4 +1,4 @@
trigger: ['main', '3.13', '3.12', '3.11', '3.10', '3.9', '3.8']
trigger: ['main', '3.*']
jobs:
- job: Prebuild

4
.editorconfig
View file

@ -1,6 +1,6 @@
root = true
[*.{py,c,cpp,h,js,rst,md,yml}]
[*.{py,c,cpp,h,js,rst,md,yml,yaml}]
trim_trailing_whitespace = true
insert_final_newline = true
indent_style = space
@ -11,5 +11,5 @@ indent_size = 4
[*.rst]
indent_size = 3
[*.{js,yml}]
[*.{js,yml,yaml}]
indent_size = 2

2
.gitattributes vendored
View file

@ -10,6 +10,7 @@
*.ico binary
*.jpg binary
*.pck binary
*.pdf binary
*.png binary
*.psd binary
*.tar binary
@ -67,6 +68,7 @@ PCbuild/readme.txt dos
**/clinic/*.cpp.h generated
**/clinic/*.h.h generated
*_db.h generated
Doc/c-api/lifecycle.dot.svg generated
Doc/data/stable_abi.dat generated
Doc/library/token-list.inc generated
Include/internal/pycore_ast.h generated

19
.github/CODEOWNERS vendored
View file

@ -30,6 +30,7 @@ Modules/Setup* @erlend-aasland
Objects/set* @rhettinger
Objects/dict* @methane @markshannon
Objects/typevarobject.c @JelleZijlstra
Objects/unionobject.c @JelleZijlstra
Objects/type* @markshannon
Objects/codeobject.c @markshannon
Objects/frameobject.c @markshannon
@ -167,6 +168,9 @@ Include/internal/pycore_time.h @pganssle @abalkin
**/*imap* @python/email-team
**/*poplib* @python/email-team
# Exclude .mailmap from being owned by @python/email-team
/.mailmap
# Garbage collector
/Modules/gcmodule.c @pablogsal
/Doc/library/gc.rst @pablogsal
@ -184,7 +188,7 @@ Include/internal/pycore_time.h @pganssle @abalkin
# AST
Python/ast.c @isidentical @JelleZijlstra @eclips4
Python/ast_opt.c @isidentical @eclips4
Python/ast_preprocess.c @isidentical @eclips4
Parser/asdl.py @isidentical @JelleZijlstra @eclips4
Parser/asdl_c.py @isidentical @JelleZijlstra @eclips4
Lib/ast.py @isidentical @JelleZijlstra @eclips4
@ -294,7 +298,12 @@ Lib/test/test_interpreters/ @ericsnowcurrently
**/*-ios* @freakboy3742
# WebAssembly
/Tools/wasm/ @brettcannon @freakboy3742
Tools/wasm/config.site-wasm32-emscripten @freakboy3742
/Tools/wasm/README.md @brettcannon @freakboy3742
/Tools/wasm/wasi-env @brettcannon
/Tools/wasm/wasi.py @brettcannon
/Tools/wasm/emscripten @freakboy3742
/Tools/wasm/wasi @brettcannon
# SBOM
/Misc/externals.spdx.json @sethmlarson
@ -316,3 +325,9 @@ Lib/test/test__colorize.py @hugovk
# Fuzzing
Modules/_xxtestfuzz/ @ammaraskar
# t-strings
**/*interpolationobject* @lysnikolaou
**/*templateobject* @lysnikolaou
**/*templatelib* @lysnikolaou
**/*tstring* @lysnikolaou

1
.github/ISSUE_TEMPLATE/bug.yml vendored
View file

@ -40,6 +40,7 @@ body:
- "3.12"
- "3.13"
- "3.14"
- "3.15"
- "CPython main branch"
validations:
required: true

1
.github/ISSUE_TEMPLATE/crash.yml vendored
View file

@ -33,6 +33,7 @@ body:
- "3.12"
- "3.13"
- "3.14"
- "3.15"
- "CPython main branch"
validations:
required: true

10
.github/PULL_REQUEST_TEMPLATE.md vendored
View file

@ -7,10 +7,10 @@ Please read this comment in its entirety. It's quite important.
It should be in the following format:
```
gh-NNNNN: Summary of the changes made
gh-NNNNNN: Summary of the changes made
```
Where: gh-NNNNN refers to the GitHub issue number.
Where: gh-NNNNNN refers to the GitHub issue number.
Most PRs will require an issue number. Trivial changes, like fixing a typo, do not need an issue.
@ -20,11 +20,11 @@ If this is a backport PR (PR made against branches other than `main`),
please ensure that the PR title is in the following format:
```
[X.Y] <title from the original PR> (GH-NNNN)
[X.Y] <title from the original PR> (GH-NNNNNN)
```
Where: [X.Y] is the branch name, e.g. [3.6].
Where: [X.Y] is the branch name, for example: [3.13].
GH-NNNN refers to the PR number from `main`.
GH-NNNNNN refers to the PR number from `main`.
-->

3
.github/actionlint.yaml vendored
View file

@ -1,5 +1,6 @@
self-hosted-runner:
labels: ["windows-aarch64"]
# Pending https://github.com/rhysd/actionlint/issues/533
labels: ["windows-11-arm"]
config-variables: null

45
.github/workflows/build.yml vendored
View file

@ -15,7 +15,13 @@ permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}-reusable
# https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#concurrency
# 'group' must be a key uniquely representing a PR or push event.
# github.workflow is the workflow name
# github.actor is the user invoking the workflow
# github.head_ref is the source branch of the PR or otherwise blank
# github.run_id is a unique number for the current run
group: ${{ github.workflow }}-${{ github.actor }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
env:
@ -156,28 +162,18 @@ jobs:
strategy:
fail-fast: false
matrix:
os:
- windows-latest
arch:
- x64
- Win32
- arm64
free-threading:
- false
- true
include:
# Forks don't have access to Windows on Arm runners. These jobs are skipped below:
- os: ${{ github.repository_owner == 'python' && 'windows-aarch64' || 'windows-latest' }}
arch: arm64
free-threading: false
# Forks don't have access to Windows on Arm runners. These jobs are skipped below:
- os: ${{ github.repository_owner == 'python' && 'windows-aarch64' || 'windows-latest' }}
arch: arm64
free-threading: true
- os: windows-latest
arch: Win32
free-threading: false
exclude:
# Skip Win32 on free-threaded builds
- { arch: Win32, free-threading: true }
uses: ./.github/workflows/reusable-windows.yml
with:
os: ${{ matrix.os }}
arch: ${{ matrix.arch }}
free-threading: ${{ matrix.free-threading }}
@ -189,18 +185,12 @@ jobs:
strategy:
fail-fast: false
matrix:
os:
- windows-latest
arch:
- x86
- x64
include:
# Forks don't have access to Windows on Arm runners. These jobs are skipped below:
- os: ${{ github.repository_owner == 'python' && 'windows-aarch64' || 'windows-latest' }}
arch: arm64
- arm64
uses: ./.github/workflows/reusable-windows-msi.yml
with:
os: ${{ matrix.os }}
arch: ${{ matrix.arch }}
build-macos:
@ -280,7 +270,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-24.04]
openssl_ver: [3.0.15, 3.1.7, 3.2.3, 3.3.2, 3.4.0]
openssl_ver: [3.0.16, 3.1.8, 3.2.4, 3.3.3, 3.4.1]
# See Tools/ssl/make_ssl_data.py for notes on adding a new version
env:
OPENSSL_VER: ${{ matrix.openssl_ver }}
@ -347,7 +337,7 @@ jobs:
needs: build-context
if: needs.build-context.outputs.run-tests == 'true'
env:
OPENSSL_VER: 3.0.15
OPENSSL_VER: 3.0.16
PYTHONSTRICTEXTENSIONBUILD: 1
steps:
- uses: actions/checkout@v4
@ -438,8 +428,9 @@ jobs:
# failing when executed from inside a virtual environment.
"${VENV_PYTHON}" -m test \
-W \
-o \
--slowest \
-j4 \
--timeout 900 \
-x test_asyncio \
-x test_multiprocessing_fork \
-x test_multiprocessing_forkserver \
@ -466,7 +457,7 @@ jobs:
matrix:
os: [ubuntu-24.04]
env:
OPENSSL_VER: 3.0.15
OPENSSL_VER: 3.0.16
PYTHONSTRICTEXTENSIONBUILD: 1
ASAN_OPTIONS: detect_leaks=0:allocator_may_return_null=1:handle_segv=0
steps:

63
.github/workflows/jit.yml vendored
View file

@ -74,8 +74,7 @@ jobs:
runner: windows-latest
- target: aarch64-pc-windows-msvc/msvc
architecture: ARM64
# Forks don't have access to Windows on Arm runners. These jobs are skipped below:
runner: ${{ github.repository_owner == 'python' && 'windows-aarch64' || 'windows-latest' }}
runner: windows-11-arm
- target: x86_64-apple-darwin/clang
architecture: x86_64
runner: macos-13
@ -96,11 +95,10 @@ jobs:
with:
python-version: '3.11'
# PCbuild downloads LLVM automatically:
- name: Windows
# Forks don't have access to Windows on Arm runners. Skip those:
if: runner.os == 'Windows' && (matrix.architecture != 'ARM64' || github.repository_owner == 'python')
if: runner.os == 'Windows'
run: |
choco install llvm --allow-downgrade --no-progress --version ${{ matrix.llvm }}.1.0
./PCbuild/build.bat --experimental-jit ${{ matrix.debug && '-d' || '' }} -p ${{ matrix.architecture }}
./PCbuild/rt.bat ${{ matrix.debug && '-d' || '' }} -p ${{ matrix.architecture }} -q --multiprocess 0 --timeout 4500 --verbose2 --verbose3
@ -115,7 +113,7 @@ jobs:
find /usr/local/bin -lname '*/Library/Frameworks/Python.framework/*' -delete
brew install llvm@${{ matrix.llvm }}
export SDKROOT="$(xcrun --show-sdk-path)"
./configure --enable-experimental-jit ${{ matrix.debug && '--with-pydebug' || '' }}
./configure --enable-experimental-jit --enable-universalsdk --with-universal-archs=universal2 ${{ matrix.debug && '--with-pydebug' || '' }}
make all --jobs 4
./python.exe -m test --multiprocess 0 --timeout 4500 --verbose2 --verbose3
@ -128,29 +126,30 @@ jobs:
make all --jobs 4
./python -m test --multiprocess 0 --timeout 4500 --verbose2 --verbose3
jit-with-disabled-gil:
name: Free-Threaded (Debug)
needs: interpreter
runs-on: ubuntu-24.04
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
llvm:
- 19
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Build with JIT enabled and GIL disabled
run: |
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" ./llvm.sh ${{ matrix.llvm }}
export PATH="$(llvm-config-${{ matrix.llvm }} --bindir):$PATH"
./configure --enable-experimental-jit --with-pydebug --disable-gil
make all --jobs 4
- name: Run tests
run: |
./python -m test --multiprocess 0 --timeout 4500 --verbose2 --verbose3
# XXX: GH-133171
# jit-with-disabled-gil:
# name: Free-Threaded (Debug)
# needs: interpreter
# runs-on: ubuntu-24.04
# timeout-minutes: 90
# strategy:
# fail-fast: false
# matrix:
# llvm:
# - 19
# steps:
# - uses: actions/checkout@v4
# with:
# persist-credentials: false
# - uses: actions/setup-python@v5
# with:
# python-version: '3.11'
# - name: Build with JIT enabled and GIL disabled
# run: |
# sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" ./llvm.sh ${{ matrix.llvm }}
# export PATH="$(llvm-config-${{ matrix.llvm }} --bindir):$PATH"
# ./configure --enable-experimental-jit --with-pydebug --disable-gil
# make all --jobs 4
# - name: Run tests
# run: |
# ./python -m test --multiprocess 0 --timeout 4500 --verbose2 --verbose3

12
.github/workflows/mypy.yml vendored
View file

@ -8,15 +8,23 @@ on:
pull_request:
paths:
- ".github/workflows/mypy.yml"
- "Lib/_colorize.py"
- "Lib/_pyrepl/**"
- "Lib/test/libregrtest/**"
- "Lib/tomllib/**"
- "Misc/mypy/**"
- "Tools/build/compute-changes.py"
- "Tools/build/deepfreeze.py"
- "Tools/build/generate_sbom.py"
- "Tools/build/generate-build-details.py"
- "Tools/build/verify_ensurepip_wheels.py"
- "Tools/build/update_file.py"
- "Tools/build/umarshal.py"
- "Tools/cases_generator/**"
- "Tools/clinic/**"
- "Tools/jit/**"
- "Tools/peg_generator/**"
- "Tools/requirements-dev.txt"
- "Tools/wasm/**"
workflow_dispatch:
permissions:
@ -42,12 +50,12 @@ jobs:
target: [
"Lib/_pyrepl",
"Lib/test/libregrtest",
"Lib/tomllib",
"Tools/build",
"Tools/cases_generator",
"Tools/clinic",
"Tools/jit",
"Tools/peg_generator",
"Tools/wasm",
]
steps:
- uses: actions/checkout@v4

3
.github/workflows/reusable-context.yml vendored
View file

@ -97,6 +97,9 @@ jobs:
run: python Tools/build/compute-changes.py
env:
GITHUB_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
GITHUB_EVENT_NAME: ${{ github.event_name }}
CCF_TARGET_REF: ${{ github.base_ref || github.event.repository.default_branch }}
CCF_HEAD_REF: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Compute hash for config cache key
id: config-hash

4
.github/workflows/reusable-docs.yml vendored
View file

@ -66,7 +66,7 @@ jobs:
run: |
set -Eeuo pipefail
# Build docs with the nit-picky option; write warnings to file
make -C Doc/ PYTHON=../python SPHINXOPTS="--quiet --nitpicky --fail-on-warning --keep-going --warning-file sphinx-warnings.txt" html
make -C Doc/ PYTHON=../python SPHINXOPTS="--quiet --nitpicky --fail-on-warning --warning-file sphinx-warnings.txt" html
- name: 'Check warnings'
if: github.event_name == 'pull_request'
run: |
@ -101,4 +101,4 @@ jobs:
run: make -C Doc/ PYTHON=../python venv
# Use "xvfb-run" since some doctest tests open GUI windows
- name: 'Run documentation doctest'
run: xvfb-run make -C Doc/ PYTHON=../python SPHINXERRORHANDLING="--fail-on-warning --keep-going" doctest
run: xvfb-run make -C Doc/ PYTHON=../python SPHINXERRORHANDLING="--fail-on-warning" doctest

8
.github/workflows/reusable-windows-msi.yml vendored
View file

@ -3,10 +3,6 @@ name: Reusable Windows MSI
on:
workflow_call:
inputs:
os:
description: OS to run on
required: true
type: string
arch:
description: CPU architecture
required: true
@ -21,7 +17,7 @@ env:
jobs:
build:
name: installer for ${{ inputs.arch }}
runs-on: ${{ inputs.os }}
runs-on: ${{ inputs.arch == 'arm64' && 'windows-11-arm' || 'windows-latest' }}
timeout-minutes: 60
env:
ARCH: ${{ inputs.arch }}
@ -31,7 +27,5 @@ jobs:
with:
persist-credentials: false
- name: Build CPython installer
# Forks don't have access to Windows on Arm runners. Skip those:
if: inputs.arch != 'arm64' || github.repository_owner == 'python'
run: ./Tools/msi/build.bat --doc -"${ARCH}"
shell: bash

12
.github/workflows/reusable-windows.yml vendored
View file

@ -3,10 +3,6 @@ name: Reusable Windows
on:
workflow_call:
inputs:
os:
description: OS to run on
required: true
type: string
arch:
description: CPU architecture
required: true
@ -25,7 +21,7 @@ env:
jobs:
build:
name: Build and test (${{ inputs.arch }})
runs-on: ${{ inputs.os }}
runs-on: ${{ inputs.arch == 'arm64' && 'windows-11-arm' || 'windows-latest' }}
timeout-minutes: 60
env:
ARCH: ${{ inputs.arch }}
@ -37,8 +33,6 @@ jobs:
if: inputs.arch != 'Win32'
run: echo "::add-matcher::.github/problem-matchers/msvc.json"
- name: Build CPython
# Forks don't have access to Windows on Arm runners. Skip those:
if: inputs.arch != 'arm64' || github.repository_owner == 'python'
run: >-
.\\PCbuild\\build.bat
-e -d -v
@ -46,12 +40,8 @@ jobs:
${{ fromJSON(inputs.free-threading) && '--disable-gil' || '' }}
shell: bash
- name: Display build info
# Forks don't have access to Windows on Arm runners. Skip those:
if: inputs.arch != 'arm64' || github.repository_owner == 'python'
run: .\\python.bat -m test.pythoninfo
- name: Tests
# Forks don't have access to Windows on Arm runners. Skip those:
if: inputs.arch != 'arm64' || github.repository_owner == 'python'
run: >-
.\\PCbuild\\rt.bat
-p "${ARCH}"

4
.github/zizmor.yml vendored
View file

@ -4,3 +4,7 @@ rules:
dangerous-triggers:
ignore:
- documentation-links.yml
unpinned-uses:
config:
policies:
"*": ref-pin

3
.gitignore vendored
View file

@ -138,11 +138,12 @@ Tools/unicode/data/
# hendrikmuhs/ccache-action@v1
/.ccache
/cross-build/
/jit_stencils.h
/jit_stencils*.h
/platform
/profile-clean-stamp
/profile-run-stamp
/profile-bolt-stamp
/profile-gen-stamp
/pybuilddir.txt
/pyconfig.h
/python-config

1
.mailmap
View file

@ -1,3 +1,4 @@
# This file sets the canonical name for contributors to the repository.
# Documentation: https://git-scm.com/docs/gitmailmap
Willow Chargin <wchargin@gmail.com>
Amethyst Reese <amethyst@n7.gg> <john@noswap.com>

22
.pre-commit-config.yaml
View file

@ -1,6 +1,6 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.9.1
rev: v0.11.8
hooks:
- id: ruff
name: Run Ruff (lint) on Doc/
@ -11,9 +11,9 @@ repos:
args: [--exit-non-zero-on-fix]
files: ^Lib/test/
- id: ruff
name: Run Ruff (lint) on Tools/build/check_warnings.py
name: Run Ruff (lint) on Tools/build/
args: [--exit-non-zero-on-fix, --config=Tools/build/.ruff.toml]
files: ^Tools/build/check_warnings.py
files: ^Tools/build/
- id: ruff
name: Run Ruff (lint) on Argument Clinic
args: [--exit-non-zero-on-fix, --config=Tools/clinic/.ruff.toml]
@ -22,14 +22,14 @@ repos:
name: Run Ruff (format) on Doc/
args: [--check]
files: ^Doc/
- id: ruff-format
name: Run Ruff (format) on Tools/build/check_warnings.py
args: [--check, --config=Tools/build/.ruff.toml]
files: ^Tools/build/check_warnings.py
- repo: https://github.com/psf/black-pre-commit-mirror
rev: 24.10.0
rev: 25.1.0
hooks:
- id: black
name: Run Black on Tools/build/check_warnings.py
files: ^Tools/build/check_warnings.py
args: [--line-length=79]
- id: black
name: Run Black on Tools/jit/
files: ^Tools/jit/
@ -47,9 +47,11 @@ repos:
exclude: Lib/test/tokenizedata/coding20731.py
- id: trailing-whitespace
types_or: [c, inc, python, rst]
- id: trailing-whitespace
files: '\.(gram)$'
- repo: https://github.com/python-jsonschema/check-jsonschema
rev: 0.31.0
rev: 0.33.0
hooks:
- id: check-dependabot
- id: check-github-workflows
@ -61,7 +63,7 @@ repos:
- id: actionlint
- repo: https://github.com/woodruffw/zizmor-pre-commit
rev: v1.1.1
rev: v1.6.0
hooks:
- id: zizmor

12
.ruff.toml Normal file
View file

@ -0,0 +1,12 @@
# Default settings for Ruff in CPython
# PYTHON_FOR_REGEN
target-version = "py310"
# PEP 8
line-length = 79
# Enable automatic fixes by default.
# To override this, use ``fix = false`` in a subdirectory's config file
# or ``--no-fix`` on the command line.
fix = true

8
Android/README.md
View file

@ -25,11 +25,13 @@ it:
`android-sdk/cmdline-tools/latest`.
* `export ANDROID_HOME=/path/to/android-sdk`
The `android.py` script also requires the following commands to be on the `PATH`:
The `android.py` script will automatically use the SDK's `sdkmanager` to install
any packages it needs.
The script also requires the following commands to be on the `PATH`:
* `curl`
* `java` (or set the `JAVA_HOME` environment variable)
* `tar`
## Building
@ -97,7 +99,7 @@ similar to the `Android` directory of the CPython source tree.
The Python test suite can be run on Linux, macOS, or Windows:
* On Linux, the emulator needs access to the KVM virtualization interface, and
a DISPLAY environment variable pointing at an X server.
a DISPLAY environment variable pointing at an X server. Xvfb is acceptable.
The test suite can usually be run on a device with 2 GB of RAM, but this is
borderline, so you may need to increase it to 4 GB. As of Android

20
Android/android.py
View file

@ -138,19 +138,19 @@ def make_build_python(context):
run(["make", "-j", str(os.cpu_count())])
def unpack_deps(host):
def unpack_deps(host, prefix_dir):
deps_url = "https://github.com/beeware/cpython-android-source-deps/releases/download"
for name_ver in ["bzip2-1.0.8-2", "libffi-3.4.4-3", "openssl-3.0.15-4",
"sqlite-3.49.1-0", "xz-5.4.6-1"]:
filename = f"{name_ver}-{host}.tar.gz"
download(f"{deps_url}/{name_ver}/{filename}")
run(["tar", "-xf", filename])
shutil.unpack_archive(filename, prefix_dir)
os.remove(filename)
def download(url, target_dir="."):
out_path = f"{target_dir}/{basename(url)}"
run(["curl", "-Lf", "-o", out_path, url])
run(["curl", "-Lf", "--retry", "5", "--retry-all-errors", "-o", out_path, url])
return out_path
@ -162,8 +162,7 @@ def configure_host_python(context):
prefix_dir = host_dir / "prefix"
if not prefix_dir.exists():
prefix_dir.mkdir()
os.chdir(prefix_dir)
unpack_deps(context.host)
unpack_deps(context.host, prefix_dir)
os.chdir(host_dir)
command = [
@ -241,16 +240,15 @@ def setup_sdk():
# the Gradle wrapper is not included in the CPython repository. Instead, we
# extract it from the Gradle GitHub repository.
def setup_testbed():
# The Gradle version used for the build is specified in
# testbed/gradle/wrapper/gradle-wrapper.properties. This wrapper version
# doesn't need to match, as any version of the wrapper can download any
# version of Gradle.
version = "8.9.0"
paths = ["gradlew", "gradlew.bat", "gradle/wrapper/gradle-wrapper.jar"]
if all((TESTBED_DIR / path).exists() for path in paths):
return
# The wrapper version isn't important, as any version of the wrapper can
# download any version of Gradle. The Gradle version actually used for the
# build is specified in testbed/gradle/wrapper/gradle-wrapper.properties.
version = "8.9.0"
for path in paths:
out_path = TESTBED_DIR / path
out_path.parent.mkdir(exist_ok=True)

5
Doc/.ruff.toml
View file

@ -1,7 +1,6 @@
extend = "../.ruff.toml" # Inherit the project-wide settings
target-version = "py312" # Align with the version in oldest_supported_sphinx
fix = true
output-format = "full"
line-length = 79
extend-exclude = [
"includes/*",
# Temporary exclusions:

130
Doc/c-api/allocation.rst
View file

@ -16,7 +16,20 @@ Allocating Objects on the Heap
Initialize a newly allocated object *op* with its type and initial
reference. Returns the initialized object. Other fields of the object are
not affected.
not initialized. Despite its name, this function is unrelated to the
object's :meth:`~object.__init__` method (:c:member:`~PyTypeObject.tp_init`
slot). Specifically, this function does **not** call the object's
:meth:`!__init__` method.
In general, consider this function to be a low-level routine. Use
:c:member:`~PyTypeObject.tp_alloc` where possible.
For implementing :c:member:`!tp_alloc` for your type, prefer
:c:func:`PyType_GenericAlloc` or :c:func:`PyObject_New`.
.. note::
This function only initializes the object's memory corresponding to the
initial :c:type:`PyObject` structure. It does not zero the rest.
.. c:function:: PyVarObject* PyObject_InitVar(PyVarObject *op, PyTypeObject *type, Py_ssize_t size)
@ -24,38 +37,107 @@ Allocating Objects on the Heap
This does everything :c:func:`PyObject_Init` does, and also initializes the
length information for a variable-size object.
.. note::
This function only initializes some of the object's memory. It does not
zero the rest.
.. c:macro:: PyObject_New(TYPE, typeobj)
Allocate a new Python object using the C structure type *TYPE*
and the Python type object *typeobj* (``PyTypeObject*``).
Fields not defined by the Python object header are not initialized.
The caller will own the only reference to the object
(i.e. its reference count will be one).
The size of the memory allocation is determined from the
:c:member:`~PyTypeObject.tp_basicsize` field of the type object.
Allocates a new Python object using the C structure type *TYPE* and the
Python type object *typeobj* (``PyTypeObject*``) by calling
:c:func:`PyObject_Malloc` to allocate memory and initializing it like
:c:func:`PyObject_Init`. The caller will own the only reference to the
object (i.e. its reference count will be one).
Note that this function is unsuitable if *typeobj* has
:c:macro:`Py_TPFLAGS_HAVE_GC` set. For such objects,
use :c:func:`PyObject_GC_New` instead.
Avoid calling this directly to allocate memory for an object; call the type's
:c:member:`~PyTypeObject.tp_alloc` slot instead.
When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot,
:c:func:`PyType_GenericAlloc` is preferred over a custom function that
simply calls this macro.
This macro does not call :c:member:`~PyTypeObject.tp_alloc`,
:c:member:`~PyTypeObject.tp_new` (:meth:`~object.__new__`), or
:c:member:`~PyTypeObject.tp_init` (:meth:`~object.__init__`).
This cannot be used for objects with :c:macro:`Py_TPFLAGS_HAVE_GC` set in
:c:member:`~PyTypeObject.tp_flags`; use :c:macro:`PyObject_GC_New` instead.
Memory allocated by this macro must be freed with :c:func:`PyObject_Free`
(usually called via the object's :c:member:`~PyTypeObject.tp_free` slot).
.. note::
The returned memory is not guaranteed to have been completely zeroed
before it was initialized.
.. note::
This macro does not construct a fully initialized object of the given
type; it merely allocates memory and prepares it for further
initialization by :c:member:`~PyTypeObject.tp_init`. To construct a
fully initialized object, call *typeobj* instead. For example::
PyObject *foo = PyObject_CallNoArgs((PyObject *)&PyFoo_Type);
.. seealso::
* :c:func:`PyObject_Free`
* :c:macro:`PyObject_GC_New`
* :c:func:`PyType_GenericAlloc`
* :c:member:`~PyTypeObject.tp_alloc`
.. c:macro:: PyObject_NewVar(TYPE, typeobj, size)
Allocate a new Python object using the C structure type *TYPE* and the
Python type object *typeobj* (``PyTypeObject*``).
Fields not defined by the Python object header
are not initialized. The allocated memory allows for the *TYPE* structure
plus *size* (``Py_ssize_t``) fields of the size
given by the :c:member:`~PyTypeObject.tp_itemsize` field of
*typeobj*. This is useful for implementing objects like tuples, which are
able to determine their size at construction time. Embedding the array of
fields into the same allocation decreases the number of allocations,
improving the memory management efficiency.
Like :c:macro:`PyObject_New` except:
Note that this function is unsuitable if *typeobj* has
:c:macro:`Py_TPFLAGS_HAVE_GC` set. For such objects,
use :c:func:`PyObject_GC_NewVar` instead.
* It allocates enough memory for the *TYPE* structure plus *size*
(``Py_ssize_t``) fields of the size given by the
:c:member:`~PyTypeObject.tp_itemsize` field of *typeobj*.
* The memory is initialized like :c:func:`PyObject_InitVar`.
This is useful for implementing objects like tuples, which are able to
determine their size at construction time. Embedding the array of fields
into the same allocation decreases the number of allocations, improving the
memory management efficiency.
Avoid calling this directly to allocate memory for an object; call the type's
:c:member:`~PyTypeObject.tp_alloc` slot instead.
When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot,
:c:func:`PyType_GenericAlloc` is preferred over a custom function that
simply calls this macro.
This cannot be used for objects with :c:macro:`Py_TPFLAGS_HAVE_GC` set in
:c:member:`~PyTypeObject.tp_flags`; use :c:macro:`PyObject_GC_NewVar`
instead.
Memory allocated by this function must be freed with :c:func:`PyObject_Free`
(usually called via the object's :c:member:`~PyTypeObject.tp_free` slot).
.. note::
The returned memory is not guaranteed to have been completely zeroed
before it was initialized.
.. note::
This macro does not construct a fully initialized object of the given
type; it merely allocates memory and prepares it for further
initialization by :c:member:`~PyTypeObject.tp_init`. To construct a
fully initialized object, call *typeobj* instead. For example::
PyObject *list_instance = PyObject_CallNoArgs((PyObject *)&PyList_Type);
.. seealso::
* :c:func:`PyObject_Free`
* :c:macro:`PyObject_GC_NewVar`
* :c:func:`PyType_GenericAlloc`
* :c:member:`~PyTypeObject.tp_alloc`
.. c:function:: void PyObject_Del(void *op)

14
Doc/c-api/arg.rst
View file

@ -274,6 +274,9 @@ small to receive the value.
Convert a Python integer to a C :c:expr:`unsigned long` without
overflow checking.
.. versionchanged:: 3.14
Use :meth:`~object.__index__` if available.
``L`` (:class:`int`) [long long]
Convert a Python integer to a C :c:expr:`long long`.
@ -281,6 +284,9 @@ small to receive the value.
Convert a Python integer to a C :c:expr:`unsigned long long`
without overflow checking.
.. versionchanged:: 3.14
Use :meth:`~object.__index__` if available.
``n`` (:class:`int`) [:c:type:`Py_ssize_t`]
Convert a Python integer to a C :c:type:`Py_ssize_t`.
@ -374,10 +380,10 @@ Other objects
The *converter* for the ``O&`` format unit in *items* must not store
a borrowed buffer or a borrowed reference.
.. versionchanged:: next
.. versionchanged:: 3.14
:class:`str` and :class:`bytearray` objects no longer accepted as a sequence.
.. deprecated:: next
.. deprecated:: 3.14
Non-tuple sequences are deprecated if *items* contains format units
which store a borrowed buffer or a borrowed reference.
@ -390,7 +396,7 @@ Other objects
If the argument is not ``None``, it is parsed according to the specified
format unit.
.. versionadded:: next
.. versionadded:: 3.14
A few other characters have a meaning in a format string. These may not occur
inside nested parentheses. They are:
@ -669,6 +675,8 @@ Building values
``L`` (:class:`int`) [long long]
Convert a C :c:expr:`long long` to a Python integer object.
.. _capi-py-buildvalue-format-K:
``K`` (:class:`int`) [unsigned long long]
Convert a C :c:expr:`unsigned long long` to a Python integer object.

54
Doc/c-api/complex.rst
View file

@ -44,36 +44,12 @@ pointers. This is consistent throughout the API.
representation.
.. c:function:: Py_complex _Py_cr_sum(Py_complex left, double right)
Return the sum of a complex number and a real number, using the C :c:type:`Py_complex`
representation.
.. versionadded:: 3.14
.. c:function:: Py_complex _Py_c_diff(Py_complex left, Py_complex right)
Return the difference between two complex numbers, using the C
:c:type:`Py_complex` representation.
.. c:function:: Py_complex _Py_cr_diff(Py_complex left, double right)
Return the difference between a complex number and a real number, using the C
:c:type:`Py_complex` representation.
.. versionadded:: 3.14
.. c:function:: Py_complex _Py_rc_diff(double left, Py_complex right)
Return the difference between a real number and a complex number, using the C
:c:type:`Py_complex` representation.
.. versionadded:: 3.14
.. c:function:: Py_complex _Py_c_neg(Py_complex num)
Return the negation of the complex number *num*, using the C
@ -86,14 +62,6 @@ pointers. This is consistent throughout the API.
representation.
.. c:function:: Py_complex _Py_cr_prod(Py_complex left, double right)
Return the product of a complex number and a real number, using the C
:c:type:`Py_complex` representation.
.. versionadded:: 3.14
.. c:function:: Py_complex _Py_c_quot(Py_complex dividend, Py_complex divisor)
Return the quotient of two complex numbers, using the C :c:type:`Py_complex`
@ -103,28 +71,6 @@ pointers. This is consistent throughout the API.
:c:data:`errno` to :c:macro:`!EDOM`.
.. c:function:: Py_complex _Py_cr_quot(Py_complex dividend, double divisor)
Return the quotient of a complex number and a real number, using the C
:c:type:`Py_complex` representation.
If *divisor* is zero, this method returns zero and sets
:c:data:`errno` to :c:macro:`!EDOM`.
.. versionadded:: 3.14
.. c:function:: Py_complex _Py_rc_quot(double dividend, Py_complex divisor)
Return the quotient of a real number and a complex number, using the C
:c:type:`Py_complex` representation.
If *divisor* is zero, this method returns zero and sets
:c:data:`errno` to :c:macro:`!EDOM`.
.. versionadded:: 3.14
.. c:function:: Py_complex _Py_c_pow(Py_complex num, Py_complex exp)
Return the exponentiation of *num* by *exp*, using the C :c:type:`Py_complex`

3
Doc/c-api/float.rst
View file

@ -96,6 +96,9 @@ NaNs (if such things exist on the platform) isn't handled correctly, and
attempting to unpack a bytes string containing an IEEE INF or NaN will raise an
exception.
Note that NaNs type may not be preserved on IEEE platforms (silent NaN become
quiet), for example on x86 systems in 32-bit mode.
On non-IEEE platforms with more precision, or larger dynamic range, than IEEE
754 supports, not all values can be packed; on non-IEEE platforms with less
precision, or smaller dynamic range, not all values can be unpacked. What

63
Doc/c-api/gcsupport.rst
View file

@ -57,11 +57,49 @@ rules:
Analogous to :c:macro:`PyObject_New` but for container objects with the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag set.
Do not call this directly to allocate memory for an object; call the type's
:c:member:`~PyTypeObject.tp_alloc` slot instead.
When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot,
:c:func:`PyType_GenericAlloc` is preferred over a custom function that
simply calls this macro.
Memory allocated by this macro must be freed with
:c:func:`PyObject_GC_Del` (usually called via the object's
:c:member:`~PyTypeObject.tp_free` slot).
.. seealso::
* :c:func:`PyObject_GC_Del`
* :c:macro:`PyObject_New`
* :c:func:`PyType_GenericAlloc`
* :c:member:`~PyTypeObject.tp_alloc`
.. c:macro:: PyObject_GC_NewVar(TYPE, typeobj, size)
Analogous to :c:macro:`PyObject_NewVar` but for container objects with the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag set.
Do not call this directly to allocate memory for an object; call the type's
:c:member:`~PyTypeObject.tp_alloc` slot instead.
When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot,
:c:func:`PyType_GenericAlloc` is preferred over a custom function that
simply calls this macro.
Memory allocated by this macro must be freed with
:c:func:`PyObject_GC_Del` (usually called via the object's
:c:member:`~PyTypeObject.tp_free` slot).
.. seealso::
* :c:func:`PyObject_GC_Del`
* :c:macro:`PyObject_NewVar`
* :c:func:`PyType_GenericAlloc`
* :c:member:`~PyTypeObject.tp_alloc`
.. c:function:: PyObject* PyUnstable_Object_GC_NewWithExtraData(PyTypeObject *type, size_t extra_size)
Analogous to :c:macro:`PyObject_GC_New` but allocates *extra_size*
@ -73,6 +111,10 @@ rules:
The extra data will be deallocated with the object, but otherwise it is
not managed by Python.
Memory allocated by this function must be freed with
:c:func:`PyObject_GC_Del` (usually called via the object's
:c:member:`~PyTypeObject.tp_free` slot).
.. warning::
The function is marked as unstable because the final mechanism
for reserving extra data after an instance is not yet decided.
@ -136,6 +178,21 @@ rules:
Releases memory allocated to an object using :c:macro:`PyObject_GC_New` or
:c:macro:`PyObject_GC_NewVar`.
Do not call this directly to free an object's memory; call the type's
:c:member:`~PyTypeObject.tp_free` slot instead.
Do not use this for memory allocated by :c:macro:`PyObject_New`,
:c:macro:`PyObject_NewVar`, or related allocation functions; use
:c:func:`PyObject_Free` instead.
.. seealso::
* :c:func:`PyObject_Free` is the non-GC equivalent of this function.
* :c:macro:`PyObject_GC_New`
* :c:macro:`PyObject_GC_NewVar`
* :c:func:`PyType_GenericAlloc`
* :c:member:`~PyTypeObject.tp_free`
.. c:function:: void PyObject_GC_UnTrack(void *op)
@ -180,9 +237,9 @@ provided. In order to use this macro, the :c:member:`~PyTypeObject.tp_traverse`
must name its arguments exactly *visit* and *arg*:
.. c:function:: void Py_VISIT(PyObject *o)
.. c:macro:: Py_VISIT(o)
If *o* is not ``NULL``, call the *visit* callback, with arguments *o*
If the :c:expr:`PyObject *` *o* is not ``NULL``, call the *visit* callback, with arguments *o*
and *arg*. If *visit* returns a non-zero value, then return it.
Using this macro, :c:member:`~PyTypeObject.tp_traverse` handlers
look like::
@ -277,7 +334,7 @@ the garbage collector.
Type of the visitor function to be passed to :c:func:`PyUnstable_GC_VisitObjects`.
*arg* is the same as the *arg* passed to ``PyUnstable_GC_VisitObjects``.
Return ``0`` to continue iteration, return ``1`` to stop iteration. Other return
Return ``1`` to continue iteration, return ``0`` to stop iteration. Other return
values are reserved for now so behavior on returning anything else is undefined.
.. versionadded:: 3.12

13
Doc/c-api/import.rst
View file

@ -16,19 +16,6 @@ Importing Modules
This is a wrapper around :c:func:`PyImport_Import()` which takes a
:c:expr:`const char *` as an argument instead of a :c:expr:`PyObject *`.
.. c:function:: PyObject* PyImport_ImportModuleNoBlock(const char *name)
This function is a deprecated alias of :c:func:`PyImport_ImportModule`.
.. versionchanged:: 3.3
This function used to fail immediately when the import lock was held
by another thread. In Python 3.3 though, the locking scheme switched
to per-module locks for most purposes, so this function's special
behaviour isn't needed anymore.
.. deprecated-removed:: 3.13 3.15
Use :c:func:`PyImport_ImportModule` instead.
.. c:function:: PyObject* PyImport_ImportModuleEx(const char *name, PyObject *globals, PyObject *locals, PyObject *fromlist)

241
Doc/c-api/init.rst
View file

@ -77,10 +77,7 @@ The following functions can be safely called before Python is initialized:
Despite their apparent similarity to some of the functions listed above,
the following functions **should not be called** before the interpreter has
been initialized: :c:func:`Py_EncodeLocale`, :c:func:`Py_GetPath`,
:c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`,
:c:func:`Py_GetProgramFullPath`, :c:func:`Py_GetPythonHome`,
:c:func:`Py_GetProgramName`, :c:func:`PyEval_InitThreads`, and
been initialized: :c:func:`Py_EncodeLocale`, :c:func:`PyEval_InitThreads`, and
:c:func:`Py_RunMain`.
@ -145,9 +142,6 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2.
:c:member:`PyConfig.pathconfig_warnings` should be used instead, see
:ref:`Python Initialization Configuration <init-config>`.
Suppress error messages when calculating the module search path in
:c:func:`Py_GetPath`.
Private flag used by ``_freeze_module`` and ``frozenmain`` programs.
.. deprecated-removed:: 3.12 3.15
@ -203,7 +197,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2.
Set by the :option:`-i` option.
.. deprecated:: 3.12
.. deprecated-removed:: 3.12 3.15
.. c:var:: int Py_IsolatedFlag
@ -586,7 +580,6 @@ Process-wide parameters
.. index::
single: Py_Initialize()
single: main()
single: Py_GetPath()
This API is kept for backward compatibility: setting
:c:member:`PyConfig.program_name` should be used instead, see :ref:`Python
@ -596,7 +589,7 @@ Process-wide parameters
the first time, if it is called at all. It tells the interpreter the value
of the ``argv[0]`` argument to the :c:func:`main` function of the program
(converted to wide characters).
This is used by :c:func:`Py_GetPath` and some other functions below to find
This is used by some other functions below to find
the Python run-time libraries relative to the interpreter executable. The
default value is ``'python'``. The argument should point to a
zero-terminated wide character string in static storage whose contents will not
@ -609,146 +602,6 @@ Process-wide parameters
.. deprecated-removed:: 3.11 3.15
.. c:function:: wchar_t* Py_GetProgramName()
Return the program name set with :c:member:`PyConfig.program_name`, or the default.
The returned string points into static storage; the caller should not modify its
value.
This function should not be called before :c:func:`Py_Initialize`, otherwise
it returns ``NULL``.
.. versionchanged:: 3.10
It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
.. deprecated-removed:: 3.13 3.15
Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>`
(:data:`sys.executable`) instead.
.. c:function:: wchar_t* Py_GetPrefix()
Return the *prefix* for installed platform-independent files. This is derived
through a number of complicated rules from the program name set with
:c:member:`PyConfig.program_name` and some environment variables; for example, if the
program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The
returned string points into static storage; the caller should not modify its
value. This corresponds to the :makevar:`prefix` variable in the top-level
:file:`Makefile` and the :option:`--prefix` argument to the :program:`configure`
script at build time. The value is available to Python code as ``sys.base_prefix``.
It is only useful on Unix. See also the next function.
This function should not be called before :c:func:`Py_Initialize`, otherwise
it returns ``NULL``.
.. versionchanged:: 3.10
It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
.. deprecated-removed:: 3.13 3.15
Use :c:func:`PyConfig_Get("base_prefix") <PyConfig_Get>`
(:data:`sys.base_prefix`) instead. Use :c:func:`PyConfig_Get("prefix")
<PyConfig_Get>` (:data:`sys.prefix`) if :ref:`virtual environments
<venv-def>` need to be handled.
.. c:function:: wchar_t* Py_GetExecPrefix()
Return the *exec-prefix* for installed platform-*dependent* files. This is
derived through a number of complicated rules from the program name set with
:c:member:`PyConfig.program_name` and some environment variables; for example, if the
program name is ``'/usr/local/bin/python'``, the exec-prefix is
``'/usr/local'``. The returned string points into static storage; the caller
should not modify its value. This corresponds to the :makevar:`exec_prefix`
variable in the top-level :file:`Makefile` and the ``--exec-prefix``
argument to the :program:`configure` script at build time. The value is
available to Python code as ``sys.base_exec_prefix``. It is only useful on
Unix.
Background: The exec-prefix differs from the prefix when platform dependent
files (such as executables and shared libraries) are installed in a different
directory tree. In a typical installation, platform dependent files may be
installed in the :file:`/usr/local/plat` subtree while platform independent may
be installed in :file:`/usr/local`.
Generally speaking, a platform is a combination of hardware and software
families, e.g. Sparc machines running the Solaris 2.x operating system are
considered the same platform, but Intel machines running Solaris 2.x are another
platform, and Intel machines running Linux are yet another platform. Different
major revisions of the same operating system generally also form different
platforms. Non-Unix operating systems are a different story; the installation
strategies on those systems are so different that the prefix and exec-prefix are
meaningless, and set to the empty string. Note that compiled Python bytecode
files are platform independent (but not independent from the Python version by
which they were compiled!).
System administrators will know how to configure the :program:`mount` or
:program:`automount` programs to share :file:`/usr/local` between platforms
while having :file:`/usr/local/plat` be a different filesystem for each
platform.
This function should not be called before :c:func:`Py_Initialize`, otherwise
it returns ``NULL``.
.. versionchanged:: 3.10
It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
.. deprecated-removed:: 3.13 3.15
Use :c:func:`PyConfig_Get("base_exec_prefix") <PyConfig_Get>`
(:data:`sys.base_exec_prefix`) instead. Use
:c:func:`PyConfig_Get("exec_prefix") <PyConfig_Get>`
(:data:`sys.exec_prefix`) if :ref:`virtual environments <venv-def>` need
to be handled.
.. c:function:: wchar_t* Py_GetProgramFullPath()
.. index::
single: executable (in module sys)
Return the full program name of the Python executable; this is computed as a
side-effect of deriving the default module search path from the program name
(set by :c:member:`PyConfig.program_name`). The returned string points into
static storage; the caller should not modify its value. The value is available
to Python code as ``sys.executable``.
This function should not be called before :c:func:`Py_Initialize`, otherwise
it returns ``NULL``.
.. versionchanged:: 3.10
It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
.. deprecated-removed:: 3.13 3.15
Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>`
(:data:`sys.executable`) instead.
.. c:function:: wchar_t* Py_GetPath()
.. index::
triple: module; search; path
single: path (in module sys)
Return the default module search path; this is computed from the program name
(set by :c:member:`PyConfig.program_name`) and some environment variables.
The returned string consists of a series of directory names separated by a
platform dependent delimiter character. The delimiter character is ``':'``
on Unix and macOS, ``';'`` on Windows. The returned string points into
static storage; the caller should not modify its value. The list
:data:`sys.path` is initialized with this value on interpreter startup; it
can be (and usually is) modified later to change the search path for loading
modules.
This function should not be called before :c:func:`Py_Initialize`, otherwise
it returns ``NULL``.
.. XXX should give the exact rules
.. versionchanged:: 3.10
It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
.. deprecated-removed:: 3.13 3.15
Use :c:func:`PyConfig_Get("module_search_paths") <PyConfig_Get>`
(:data:`sys.path`) instead.
.. c:function:: const char* Py_GetVersion()
Return the version of this Python interpreter. This is a string that looks
@ -919,23 +772,6 @@ Process-wide parameters
.. deprecated-removed:: 3.11 3.15
.. c:function:: wchar_t* Py_GetPythonHome()
Return the default "home", that is, the value set by
:c:member:`PyConfig.home`, or the value of the :envvar:`PYTHONHOME`
environment variable if it is set.
This function should not be called before :c:func:`Py_Initialize`, otherwise
it returns ``NULL``.
.. versionchanged:: 3.10
It now returns ``NULL`` if called before :c:func:`Py_Initialize`.
.. deprecated-removed:: 3.13 3.15
Use :c:func:`PyConfig_Get("home") <PyConfig_Get>` or the
:envvar:`PYTHONHOME` environment variable instead.
.. _threads:
Thread State and the Global Interpreter Lock
@ -1083,8 +919,36 @@ Note that the ``PyGILState_*`` functions assume there is only one global
interpreter (created automatically by :c:func:`Py_Initialize`). Python
supports the creation of additional interpreters (using
:c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the
``PyGILState_*`` API is unsupported.
``PyGILState_*`` API is unsupported. This is because :c:func:`PyGILState_Ensure`
and similar functions default to :term:`attaching <attached thread state>` a
:term:`thread state` for the main interpreter, meaning that the thread can't safely
interact with the calling subinterpreter.
Supporting subinterpreters in non-Python threads
------------------------------------------------
If you would like to support subinterpreters with non-Python created threads, you
must use the ``PyThreadState_*`` API instead of the traditional ``PyGILState_*``
API.
In particular, you must store the interpreter state from the calling
function and pass it to :c:func:`PyThreadState_New`, which will ensure that
the :term:`thread state` is targeting the correct interpreter::
/* The return value of PyInterpreterState_Get() from the
function that created this thread. */
PyInterpreterState *interp = ThreadData->interp;
PyThreadState *tstate = PyThreadState_New(interp);
PyThreadState_Swap(tstate);
/* GIL of the subinterpreter is now held.
Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */
/* Destroy the thread state. No Python API allowed beyond this point. */
PyThreadState_Clear(tstate);
PyThreadState_DeleteCurrent();
.. _fork-and-threads:
@ -1131,7 +995,7 @@ Cautions regarding runtime finalization
In the late stage of :term:`interpreter shutdown`, after attempting to wait for
non-daemon threads to exit (though this can be interrupted by
:class:`KeyboardInterrupt`) and running the :mod:`atexit` functions, the runtime
is marked as *finalizing*: :c:func:`_Py_IsFinalizing` and
is marked as *finalizing*: :c:func:`Py_IsFinalizing` and
:func:`sys.is_finalizing` return true. At this point, only the *finalization
thread* that initiated finalization (typically the main thread) is allowed to
acquire the :term:`GIL`.
@ -1261,6 +1125,10 @@ code, or when embedding the Python interpreter:
.. seealso:
:c:func:`PyEval_ReleaseThread`
.. note::
Similar to :c:func:`PyGILState_Ensure`, this function will hang the
thread if the runtime is finalizing.
The following functions use thread-local storage, and are not compatible
with sub-interpreters:
@ -1287,10 +1155,10 @@ with sub-interpreters:
When the function returns, there will be an :term:`attached thread state`
and the thread will be able to call arbitrary Python code. Failure is a fatal error.
.. note::
Calling this function from a thread when the runtime is finalizing will
hang the thread until the program exits, even if the thread was not
created by Python. Refer to
.. warning::
Calling this function when the runtime is finalizing is unsafe. Doing
so will either hang the thread until the program ends, or fully crash
the interpreter in rare cases. Refer to
:ref:`cautions-regarding-runtime-finalization` for more details.
.. versionchanged:: 3.14
@ -1307,7 +1175,6 @@ with sub-interpreters:
Every call to :c:func:`PyGILState_Ensure` must be matched by a call to
:c:func:`PyGILState_Release` on the same thread.
.. c:function:: PyThreadState* PyGILState_GetThisThreadState()
Get the :term:`attached thread state` for this thread. May return ``NULL`` if no
@ -1315,20 +1182,30 @@ with sub-interpreters:
always has such a thread-state, even if no auto-thread-state call has been
made on the main thread. This is mainly a helper/diagnostic function.
.. seealso: :c:func:`PyThreadState_Get``
.. note::
This function does not account for :term:`thread states <thread state>` created
by something other than :c:func:`PyGILState_Ensure` (such as :c:func:`PyThreadState_New`).
Prefer :c:func:`PyThreadState_Get` or :c:func:`PyThreadState_GetUnchecked`
for most cases.
.. seealso: :c:func:`PyThreadState_Get``
.. c:function:: int PyGILState_Check()
Return ``1`` if the current thread is holding the :term:`GIL` and ``0`` otherwise.
This function can be called from any thread at any time.
Only if it has had its Python thread state initialized and currently is
holding the :term:`GIL` will it return ``1``.
Only if it has had its :term:`thread state <attached thread state>` initialized
via :c:func:`PyGILState_Ensure` will it return ``1``.
This is mainly a helper/diagnostic function. It can be useful
for example in callback contexts or memory allocation functions when
knowing that the :term:`GIL` is locked can allow the caller to perform sensitive
actions or otherwise behave differently.
.. note::
If the current Python process has ever created a subinterpreter, this
function will *always* return ``1``. Prefer :c:func:`PyThreadState_GetUnchecked`
for most cases.
.. versionadded:: 3.4
@ -1517,16 +1394,6 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
.. versionadded:: 3.8
.. c:function:: PyObject* PyUnstable_InterpreterState_GetMainModule(PyInterpreterState *interp)
Return a :term:`strong reference` to the ``__main__`` :ref:`module object <moduleobjects>`
for the given interpreter.
The caller must have an :term:`attached thread state`.
.. versionadded:: 3.13
.. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag)
Type of a frame evaluation function.

16
Doc/c-api/init_config.rst
View file

@ -320,7 +320,7 @@ Configuration Options
* - ``"cpu_count"``
- :c:member:`cpu_count <PyConfig.cpu_count>`
- ``int``
- Read-only
- Public
* - ``"dev_mode"``
- :c:member:`dev_mode <PyConfig.dev_mode>`
- ``bool``
@ -363,7 +363,7 @@ Configuration Options
- Read-only
* - ``"import_time"``
- :c:member:`import_time <PyConfig.import_time>`
- ``bool``
- ``int``
- Read-only
* - ``"inspect"``
- :c:member:`inspect <PyConfig.inspect>`
@ -619,6 +619,8 @@ Some options are read from the :mod:`sys` attributes. For example, the option
The caller must have an :term:`attached thread state`. The function cannot
be called before Python initialization nor after Python finalization.
.. audit-event:: cpython.PyConfig_Set name,value c.PyConfig_Set
.. versionadded:: 3.14
@ -1475,13 +1477,19 @@ PyConfig
.. c:member:: int import_time
If non-zero, profile import time.
If ``1``, profile import time.
If ``2``, include additional output that indicates
when an imported module has already been loaded.
Set the ``1`` by the :option:`-X importtime <-X>` option and the
Set by the :option:`-X importtime <-X>` option and the
:envvar:`PYTHONPROFILEIMPORTTIME` environment variable.
Default: ``0``.
.. versionchanged:: 3.14
Added support for ``import_time = 2``
.. c:member:: int inspect
Enter interactive mode after executing a script or a command.

28
Doc/c-api/intro.rst
View file

@ -148,7 +148,7 @@ complete listing.
.. c:macro:: Py_ALWAYS_INLINE
Ask the compiler to always inline a static inline function. The compiler can
ignore it and decides to not inline the function.
ignore it and decide to not inline the function.
It can be used to inline performance critical static inline functions when
building Python in debug mode with function inlining disabled. For example,
@ -779,20 +779,11 @@ found along :envvar:`PATH`.) The user can override this behavior by setting the
environment variable :envvar:`PYTHONHOME`, or insert additional directories in
front of the standard path by setting :envvar:`PYTHONPATH`.
.. index::
single: Py_GetPath (C function)
single: Py_GetPrefix (C function)
single: Py_GetExecPrefix (C function)
single: Py_GetProgramFullPath (C function)
The embedding application can steer the search by setting
:c:member:`PyConfig.program_name` *before* calling
:c:func:`Py_InitializeFromConfig`. Note that
:envvar:`PYTHONHOME` still overrides this and :envvar:`PYTHONPATH` is still
inserted in front of the standard path. An application that requires total
control has to provide its own implementation of :c:func:`Py_GetPath`,
:c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, and
:c:func:`Py_GetProgramFullPath` (all defined in :file:`Modules/getpath.c`).
inserted in front of the standard path.
.. index:: single: Py_IsInitialized (C function)
@ -826,14 +817,17 @@ frequently used builds will be described in the remainder of this section.
Compiling the interpreter with the :c:macro:`!Py_DEBUG` macro defined produces
what is generally meant by :ref:`a debug build of Python <debug-build>`.
:c:macro:`!Py_DEBUG` is enabled in the Unix build by adding
:option:`--with-pydebug` to the :file:`./configure` command.
It is also implied by the presence of the
not-Python-specific :c:macro:`!_DEBUG` macro. When :c:macro:`!Py_DEBUG` is enabled
in the Unix build, compiler optimization is disabled.
On Unix, :c:macro:`!Py_DEBUG` can be enabled by adding :option:`--with-pydebug`
to the :file:`./configure` command. This will also disable compiler optimization.
On Windows, selecting a debug build (e.g., by passing the :option:`-d` option to
:file:`PCbuild/build.bat`) automatically enables :c:macro:`!Py_DEBUG`.
Additionally, the presence of the not-Python-specific :c:macro:`!_DEBUG` macro,
when defined by the compiler, will also implicitly enable :c:macro:`!Py_DEBUG`.
In addition to the reference count debugging described below, extra checks are
performed, see :ref:`Python Debug Build <debug-build>`.
performed. See :ref:`Python Debug Build <debug-build>` for more details.
Defining :c:macro:`Py_TRACE_REFS` enables reference tracing
(see the :option:`configure --with-trace-refs option <--with-trace-refs>`).

156
Doc/c-api/lifecycle.dot Normal file
View file

@ -0,0 +1,156 @@
digraph "Life Events" {
graph [
fontnames="svg"
fontsize=12.0
id="life_events_graph"
layout="dot"
margin="0,0"
ranksep=0.25
stylesheet="lifecycle.dot.css"
]
node [
fontname="Courier"
fontsize=12.0
]
edge [
fontname="Times-Italic"
fontsize=12.0
]
"start" [fontname="Times-Italic" shape=plain label=< start > style=invis]
{
rank="same"
"tp_new" [href="typeobj.html#c.PyTypeObject.tp_new" target="_top"]
"tp_alloc" [href="typeobj.html#c.PyTypeObject.tp_alloc" target="_top"]
}
"tp_init" [href="typeobj.html#c.PyTypeObject.tp_init" target="_top"]
"reachable" [fontname="Times-Italic" shape=box]
"tp_traverse" [
href="typeobj.html#c.PyTypeObject.tp_traverse"
ordering="in"
target="_top"
]
"finalized?" [
fontname="Times-Italic"
label=<marked as<br/>finalized?>
ordering="in"
shape=diamond
tooltip="marked as finalized?"
]
"tp_finalize" [
href="typeobj.html#c.PyTypeObject.tp_finalize"
ordering="in"
target="_top"
]
"tp_clear" [href="typeobj.html#c.PyTypeObject.tp_clear" target="_top"]
"uncollectable" [
fontname="Times-Italic"
label=<uncollectable<br/>(leaked)>
shape=box
tooltip="uncollectable (leaked)"
]
"tp_dealloc" [
href="typeobj.html#c.PyTypeObject.tp_dealloc"
ordering="in"
target="_top"
]
"tp_free" [href="typeobj.html#c.PyTypeObject.tp_free" target="_top"]
"start" -> "tp_new" [
label=< type call >
]
"tp_new" -> "tp_alloc" [
label=< direct call > arrowhead=empty
labeltooltip="tp_new to tp_alloc: direct call"
tooltip="tp_new to tp_alloc: direct call"
]
"tp_new" -> "tp_init" [tooltip="tp_new to tp_init"]
"tp_init" -> "reachable" [tooltip="tp_init to reachable"]
"reachable" -> "tp_traverse" [
dir="back"
label=< not in a <br/> cyclic <br/> isolate >
labeltooltip="tp_traverse to reachable: not in a cyclic isolate"
tooltip="tp_traverse to reachable: not in a cyclic isolate"
]
"reachable" -> "tp_traverse" [
label=< periodic <br/> cyclic isolate <br/> detection >
labeltooltip="reachable to tp_traverse: periodic cyclic isolate detection"
tooltip="reachable to tp_traverse: periodic cyclic isolate detection"
]
"reachable" -> "tp_init" [tooltip="reachable to tp_init"]
"reachable" -> "tp_finalize" [
dir="back"
label=< resurrected <br/> (maybe remove <br/> finalized mark) >
labeltooltip="tp_finalize to reachable: resurrected (maybe remove finalized mark)"
tooltip="tp_finalize to reachable: resurrected (maybe remove finalized mark)"
]
"tp_traverse" -> "finalized?" [
label=< cyclic <br/> isolate >
labeltooltip="tp_traverse to finalized?: cyclic isolate"
tooltip="tp_traverse to finalized?: cyclic isolate"
]
"reachable" -> "finalized?" [
label=< no refs >
labeltooltip="reachable to finalized?: no refs"
tooltip="reachable to finalized?: no refs"
]
"finalized?" -> "tp_finalize" [
label=< no (mark <br/> as finalized) >
labeltooltip="finalized? to tp_finalize: no (mark as finalized)"
tooltip="finalized? to tp_finalize: no (mark as finalized)"
]
"finalized?" -> "tp_clear" [
label=< yes >
labeltooltip="finalized? to tp_clear: yes"
tooltip="finalized? to tp_clear: yes"
]
"tp_finalize" -> "tp_clear" [
label=< no refs or <br/> cyclic isolate >
labeltooltip="tp_finalize to tp_clear: no refs or cyclic isolate"
tooltip="tp_finalize to tp_clear: no refs or cyclic isolate"
]
"tp_finalize" -> "tp_dealloc" [
arrowtail=empty
dir="back"
href="lifecycle.html#c.PyObject_CallFinalizerFromDealloc"
style=dashed
label=< recommended<br/> call (see<br/> explanation)>
labeltooltip="tp_dealloc to tp_finalize: recommended call (see explanation)"
target="_top"
tooltip="tp_dealloc to tp_finalize: recommended call (see explanation)"
]
"tp_finalize" -> "tp_dealloc" [
label=< no refs >
labeltooltip="tp_finalize to tp_dealloc: no refs"
tooltip="tp_finalize to tp_dealloc: no refs"
]
"tp_clear" -> "tp_dealloc" [
label=< no refs >
labeltooltip="tp_clear to tp_dealloc: no refs"
tooltip="tp_clear to tp_dealloc: no refs"
]
"tp_clear" -> "uncollectable" [
label=< cyclic <br/> isolate >
labeltooltip="tp_clear to uncollectable: cyclic isolate"
tooltip="tp_clear to uncollectable: cyclic isolate"
]
"uncollectable" -> "tp_dealloc" [
style=invis
tooltip="uncollectable to tp_dealloc"
]
"reachable" -> "uncollectable" [
label=< cyclic <br/> isolate <br/> (no GC <br/> support) >
labeltooltip="reachable to uncollectable: cyclic isolate (no GC support)"
tooltip="reachable to uncollectable: cyclic isolate (no GC support)"
]
"reachable" -> "tp_dealloc" [
label=< no refs>
labeltooltip="reachable to tp_dealloc: no refs"
]
"tp_dealloc" -> "tp_free" [
arrowhead=empty
label=< direct call >
labeltooltip="tp_dealloc to tp_free: direct call"
tooltip="tp_dealloc to tp_free: direct call"
]
}

21
Doc/c-api/lifecycle.dot.css Normal file
View file

@ -0,0 +1,21 @@
#life_events_graph {
--svg-fgcolor: currentcolor;
--svg-bgcolor: transparent;
}
#life_events_graph a {
color: inherit;
}
#life_events_graph [stroke="black"] {
stroke: var(--svg-fgcolor);
}
#life_events_graph text,
#life_events_graph [fill="black"] {
fill: var(--svg-fgcolor);
}
#life_events_graph [fill="white"] {
fill: var(--svg-bgcolor);
}
#life_events_graph [fill="none"] {
/* On links, setting fill will make the entire shape clickable */
fill: var(--svg-bgcolor);
}

BIN
Doc/c-api/lifecycle.dot.pdf Normal file
View file

Binary file not shown.

374
Doc/c-api/lifecycle.dot.svg generated Normal file
View file

@ -0,0 +1,374 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet href="lifecycle.dot.css" type="text/css"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 12.2.0 (0)
-->
<!-- Title: Life Events Pages: 1 -->
<svg width="465pt" height="845pt"
viewBox="0.00 0.00 465.30 845.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="life_events_graph" class="graph" transform="scale(1 1) rotate(0) translate(4 841)">
<title>Life Events</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-841 461.3,-841 461.3,4 -4,4"/>
<!-- start -->
<!-- tp_new -->
<g id="life_events_graph_node2" class="node">
<title>tp_new</title>
<g id="a_life_events_graph_node2"><a xlink:href="typeobj.html#c.PyTypeObject.tp_new" xlink:title="tp_new" target="_top">
<ellipse fill="none" stroke="black" cx="192.8" cy="-772.5" rx="38.8" ry="18"/>
<text text-anchor="middle" x="192.8" y="-768.23" font-family="monospace,monospace" font-size="12.00">tp_new</text>
</a>
</g>
</g>
<!-- start&#45;&gt;tp_new -->
<g id="life_events_graph_edge1" class="edge">
<title>start&#45;&gt;tp_new</title>
<g id="a_life_events_graph_edge1"><a xlink:title="start to tp_new: type call">
<path fill="none" stroke="black" d="M192.8,-822.95C192.8,-817.85 192.8,-810.09 192.8,-802.22"/>
<polygon fill="black" stroke="black" points="196.3,-802.42 192.8,-792.42 189.3,-802.42 196.3,-802.42"/>
</a>
</g>
<g id="a_life_events_graph_edge1&#45;label"><a xlink:title="start to tp_new: type call">
<text text-anchor="start" x="192.8" y="-802.35" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;&#160;&#160;type call &#160;</text>
</a>
</g>
</g>
<!-- tp_alloc -->
<g id="life_events_graph_node3" class="node">
<title>tp_alloc</title>
<g id="a_life_events_graph_node3"><a xlink:href="typeobj.html#c.PyTypeObject.tp_alloc" xlink:title="tp_alloc" target="_top">
<ellipse fill="none" stroke="black" cx="373.8" cy="-772.5" rx="48.34" ry="18"/>
<text text-anchor="middle" x="373.8" y="-768.23" font-family="monospace,monospace" font-size="12.00">tp_alloc</text>
</a>
</g>
</g>
<!-- tp_new&#45;&gt;tp_alloc -->
<g id="life_events_graph_edge2" class="edge">
<title>tp_new&#45;&gt;tp_alloc</title>
<g id="a_life_events_graph_edge2"><a xlink:title="tp_new to tp_alloc: direct call">
<path fill="none" stroke="black" d="M232.07,-772.5C256,-772.5 287.05,-772.5 313.98,-772.5"/>
<polygon fill="none" stroke="black" points="313.73,-776 323.73,-772.5 313.73,-769 313.73,-776"/>
</a>
</g>
<g id="a_life_events_graph_edge2&#45;label"><a xlink:title="tp_new to tp_alloc: direct call">
<text text-anchor="start" x="240.65" y="-778.35" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;direct call &#160;</text>
</a>
</g>
</g>
<!-- tp_init -->
<g id="life_events_graph_node4" class="node">
<title>tp_init</title>
<g id="a_life_events_graph_node4"><a xlink:href="typeobj.html#c.PyTypeObject.tp_init" xlink:title="tp_init" target="_top">
<ellipse fill="none" stroke="black" cx="192.8" cy="-717.5" rx="43.57" ry="18"/>
<text text-anchor="middle" x="192.8" y="-713.23" font-family="monospace,monospace" font-size="12.00">tp_init</text>
</a>
</g>
</g>
<!-- tp_new&#45;&gt;tp_init -->
<g id="life_events_graph_edge3" class="edge">
<title>tp_new&#45;&gt;tp_init</title>
<g id="a_life_events_graph_edge3"><a xlink:title="tp_new to tp_init">
<path fill="none" stroke="black" d="M192.8,-754.15C192.8,-751.83 192.8,-749.42 192.8,-746.98"/>
<polygon fill="black" stroke="black" points="196.3,-747.23 192.8,-737.23 189.3,-747.23 196.3,-747.23"/>
</a>
</g>
</g>
<!-- reachable -->
<g id="life_events_graph_node5" class="node">
<title>reachable</title>
<polygon fill="none" stroke="black" points="230.8,-680.5 154.8,-680.5 154.8,-644.5 230.8,-644.5 230.8,-680.5"/>
<text text-anchor="middle" x="192.8" y="-658.23" font-family="serif,serif" font-style="italic" font-size="12.00">reachable</text>
</g>
<!-- tp_init&#45;&gt;reachable -->
<g id="life_events_graph_edge4" class="edge">
<title>tp_init&#45;&gt;reachable</title>
<g id="a_life_events_graph_edge4"><a xlink:title="tp_init to reachable">
<path fill="none" stroke="black" d="M186.44,-699.44C186.24,-697.12 186.11,-694.69 186.07,-692.24"/>
<polygon fill="black" stroke="black" points="189.56,-692.51 186.37,-682.41 182.56,-692.29 189.56,-692.51"/>
</a>
</g>
</g>
<!-- reachable&#45;&gt;tp_init -->
<g id="life_events_graph_edge7" class="edge">
<title>reachable&#45;&gt;tp_init</title>
<g id="a_life_events_graph_edge7"><a xlink:title="reachable to tp_init">
<path fill="none" stroke="black" d="M199.18,-680.89C199.37,-683.22 199.49,-685.65 199.53,-688.11"/>
<polygon fill="black" stroke="black" points="196.04,-687.81 199.2,-697.93 203.04,-688.05 196.04,-687.81"/>
</a>
</g>
</g>
<!-- tp_traverse -->
<g id="life_events_graph_node6" class="node">
<title>tp_traverse</title>
<g id="a_life_events_graph_node6"><a xlink:href="typeobj.html#c.PyTypeObject.tp_traverse" xlink:title="tp_traverse" target="_top">
<ellipse fill="none" stroke="black" cx="136.8" cy="-565.75" rx="62.65" ry="18"/>
<text text-anchor="middle" x="136.8" y="-561.48" font-family="monospace,monospace" font-size="12.00">tp_traverse</text>
</a>
</g>
</g>
<!-- reachable&#45;&gt;tp_traverse -->
<g id="life_events_graph_edge5" class="edge">
<title>reachable&#45;&gt;tp_traverse</title>
<g id="a_life_events_graph_edge5"><a xlink:title="tp_traverse to reachable: not in a cyclic isolate">
<path fill="none" stroke="black" d="M143.43,-658.77C108.3,-655.68 65.38,-649.16 54.05,-635.5 41.91,-620.88 42.8,-608.07 54.05,-592.75 60.55,-583.89 70.07,-577.97 80.37,-574.03"/>
<polygon fill="black" stroke="black" points="142.76,-662.23 153.01,-659.54 143.32,-655.25 142.76,-662.23"/>
</a>
</g>
<g id="a_life_events_graph_edge5&#45;label"><a xlink:title="tp_traverse to reachable: not in a cyclic isolate">
<text text-anchor="start" x="54.05" y="-624.1" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;not in a &#160;</text>
<text text-anchor="start" x="59.67" y="-609.85" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;cyclic &#160;</text>
<text text-anchor="start" x="57.05" y="-595.6" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;isolate &#160;</text>
</a>
</g>
</g>
<!-- reachable&#45;&gt;tp_traverse -->
<g id="life_events_graph_edge6" class="edge">
<title>reachable&#45;&gt;tp_traverse</title>
<g id="a_life_events_graph_edge6"><a xlink:title="reachable to tp_traverse: periodic cyclic isolate detection">
<path fill="none" stroke="black" d="M154.41,-650.07C147.94,-646.44 142.04,-641.69 138.05,-635.5 130.52,-623.82 129.57,-608.56 130.79,-595.38"/>
<polygon fill="black" stroke="black" points="134.25,-595.91 132.17,-585.52 127.31,-594.94 134.25,-595.91"/>
</a>
</g>
<g id="a_life_events_graph_edge6&#45;label"><a xlink:title="reachable to tp_traverse: periodic cyclic isolate detection">
<text text-anchor="start" x="154.17" y="-624.1" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;periodic &#160;</text>
<text text-anchor="start" x="138.05" y="-609.85" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;cyclic isolate &#160;&#160;</text>
<text text-anchor="start" x="151.17" y="-595.6" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;detection &#160;</text>
</a>
</g>
</g>
<!-- finalized? -->
<g id="life_events_graph_node7" class="node">
<title>finalized?</title>
<g id="a_life_events_graph_node7"><a xlink:title="marked as finalized?">
<polygon fill="none" stroke="black" points="191.8,-487 112.05,-450.5 191.8,-414 271.55,-450.5 191.8,-487"/>
<text text-anchor="start" x="159.92" y="-453.35" font-family="serif,serif" font-style="italic" font-size="12.00">marked as</text>
<text text-anchor="start" x="162.92" y="-439.1" font-family="serif,serif" font-style="italic" font-size="12.00">finalized?</text>
</a>
</g>
</g>
<!-- reachable&#45;&gt;finalized? -->
<g id="life_events_graph_edge10" class="edge">
<title>reachable&#45;&gt;finalized?</title>
<g id="a_life_events_graph_edge10"><a xlink:title="reachable to finalized?: no refs">
<path fill="none" stroke="black" d="M227.72,-644.32C230.51,-641.73 232.96,-638.8 234.8,-635.5 244.04,-618.9 235.48,-611.74 234.8,-592.75 233.24,-549.67 243.64,-536.1 227.8,-496 226.37,-492.38 224.53,-488.82 222.45,-485.4"/>
<polygon fill="black" stroke="black" points="225.47,-483.62 216.91,-477.39 219.72,-487.61 225.47,-483.62"/>
</a>
</g>
<g id="a_life_events_graph_edge10&#45;label"><a xlink:title="reachable to finalized?: no refs">
<text text-anchor="start" x="236.45" y="-561.48" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;no refs &#160;</text>
</a>
</g>
</g>
<!-- tp_finalize -->
<g id="life_events_graph_node8" class="node">
<title>tp_finalize</title>
<g id="a_life_events_graph_node8"><a xlink:href="typeobj.html#c.PyTypeObject.tp_finalize" xlink:title="tp_finalize" target="_top">
<ellipse fill="none" stroke="black" cx="122.8" cy="-321" rx="62.65" ry="18"/>
<text text-anchor="middle" x="122.8" y="-316.73" font-family="monospace,monospace" font-size="12.00">tp_finalize</text>
</a>
</g>
</g>
<!-- reachable&#45;&gt;tp_finalize -->
<g id="life_events_graph_edge8" class="edge">
<title>reachable&#45;&gt;tp_finalize</title>
<g id="a_life_events_graph_edge8"><a xlink:title="tp_finalize to reachable: resurrected (maybe remove finalized mark)">
<path fill="none" stroke="black" d="M142.86,-659.6C103.8,-656.96 53.97,-650.63 40.8,-635.5 -37.32,-545.75 69.61,-390.31 109.14,-338.99"/>
<polygon fill="black" stroke="black" points="142.62,-663.09 152.82,-660.2 143.05,-656.11 142.62,-663.09"/>
</a>
</g>
<g id="a_life_events_graph_edge8&#45;label"><a xlink:title="tp_finalize to reachable: resurrected (maybe remove finalized mark)">
<text text-anchor="start" x="33.68" y="-527.35" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;resurrected &#160;</text>
<text text-anchor="start" x="22.43" y="-513.1" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;(maybe remove &#160;</text>
<text text-anchor="start" x="23.18" y="-498.85" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;finalized mark) &#160;</text>
</a>
</g>
</g>
<!-- uncollectable -->
<g id="life_events_graph_node10" class="node">
<title>uncollectable</title>
<g id="a_life_events_graph_node10"><a xlink:title="uncollectable (leaked)">
<polygon fill="none" stroke="black" points="371.92,-159.75 275.67,-159.75 275.67,-123.25 371.92,-123.25 371.92,-159.75"/>
<text text-anchor="start" x="283.67" y="-144.35" font-family="serif,serif" font-style="italic" font-size="12.00">uncollectable</text>
<text text-anchor="start" x="299.42" y="-130.1" font-family="serif,serif" font-style="italic" font-size="12.00">(leaked)</text>
</a>
</g>
</g>
<!-- reachable&#45;&gt;uncollectable -->
<g id="life_events_graph_edge19" class="edge">
<title>reachable&#45;&gt;uncollectable</title>
<g id="a_life_events_graph_edge19"><a xlink:title="reachable to uncollectable: cyclic isolate (no GC support)">
<path fill="none" stroke="black" d="M231.2,-652.03C270.79,-639.69 326.8,-613.9 326.8,-566.75 326.8,-566.75 326.8,-566.75 326.8,-237.5 326.8,-215.3 325.97,-190.2 325.18,-171.37"/>
<polygon fill="black" stroke="black" points="328.68,-171.35 324.75,-161.52 321.69,-171.66 328.68,-171.35"/>
</a>
</g>
<g id="a_life_events_graph_edge19&#45;label"><a xlink:title="reachable to uncollectable: cyclic isolate (no GC support)">
<text text-anchor="start" x="335.05" y="-393.6" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;cyclic &#160;</text>
<text text-anchor="start" x="332.42" y="-379.35" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;isolate &#160;</text>
<text text-anchor="start" x="331.3" y="-365.1" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;(no GC &#160;</text>
<text text-anchor="start" x="326.8" y="-350.85" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;support) &#160;</text>
</a>
</g>
</g>
<!-- tp_dealloc -->
<g id="life_events_graph_node11" class="node">
<title>tp_dealloc</title>
<g id="a_life_events_graph_node11"><a xlink:href="typeobj.html#c.PyTypeObject.tp_dealloc" xlink:title="tp_dealloc" target="_top">
<ellipse fill="none" stroke="black" cx="200.8" cy="-86.25" rx="57.88" ry="18"/>
<text text-anchor="middle" x="200.8" y="-81.97" font-family="monospace,monospace" font-size="12.00">tp_dealloc</text>
</a>
</g>
</g>
<!-- reachable&#45;&gt;tp_dealloc -->
<g id="life_events_graph_edge20" class="edge">
<title>reachable&#45;&gt;tp_dealloc</title>
<path fill="none" stroke="black" d="M231.23,-661.18C293.08,-658.43 407.8,-643.03 407.8,-566.75 407.8,-566.75 407.8,-566.75 407.8,-140.5 407.8,-111.22 329.12,-97.8 268.77,-91.82"/>
<polygon fill="black" stroke="black" points="269.15,-88.34 258.87,-90.89 268.5,-95.31 269.15,-88.34"/>
<g id="a_life_events_graph_edge20&#45;label"><a xlink:title="reachable to tp_dealloc: no refs">
<text text-anchor="start" x="407.8" y="-316.73" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;no refs</text>
</a>
</g>
</g>
<!-- tp_traverse&#45;&gt;finalized? -->
<g id="life_events_graph_edge9" class="edge">
<title>tp_traverse&#45;&gt;finalized?</title>
<g id="a_life_events_graph_edge9"><a xlink:title="tp_traverse to finalized?: cyclic isolate">
<path fill="none" stroke="black" d="M145.15,-547.55C152.4,-532.62 163.18,-510.43 172.55,-491.13"/>
<polygon fill="black" stroke="black" points="175.7,-492.66 176.92,-482.14 169.4,-489.61 175.7,-492.66"/>
</a>
</g>
<g id="a_life_events_graph_edge9&#45;label"><a xlink:title="tp_traverse to finalized?: cyclic isolate">
<text text-anchor="start" x="171.85" y="-520.23" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;cyclic &#160;</text>
<text text-anchor="start" x="169.22" y="-505.98" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;isolate &#160;</text>
</a>
</g>
</g>
<!-- finalized?&#45;&gt;tp_finalize -->
<g id="life_events_graph_edge11" class="edge">
<title>finalized?&#45;&gt;tp_finalize</title>
<g id="a_life_events_graph_edge11"><a xlink:title="finalized? to tp_finalize: no (mark as finalized)">
<path fill="none" stroke="black" d="M172.89,-422.6C169.14,-416.89 165.34,-410.82 162.05,-405 151.89,-387.08 141.99,-366.11 134.68,-349.73"/>
<polygon fill="black" stroke="black" points="137.89,-348.35 130.66,-340.61 131.48,-351.17 137.89,-348.35"/>
</a>
</g>
<g id="a_life_events_graph_edge11&#45;label"><a xlink:title="finalized? to tp_finalize: no (mark as finalized)">
<text text-anchor="start" x="170.67" y="-379.35" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;no (mark &#160;</text>
<text text-anchor="start" x="162.05" y="-365.1" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;as finalized) &#160;</text>
</a>
</g>
</g>
<!-- tp_clear -->
<g id="life_events_graph_node9" class="node">
<title>tp_clear</title>
<g id="a_life_events_graph_node9"><a xlink:href="typeobj.html#c.PyTypeObject.tp_clear" xlink:title="tp_clear" target="_top">
<ellipse fill="none" stroke="black" cx="222.8" cy="-238.5" rx="48.34" ry="18"/>
<text text-anchor="middle" x="222.8" y="-234.22" font-family="monospace,monospace" font-size="12.00">tp_clear</text>
</a>
</g>
</g>
<!-- finalized?&#45;&gt;tp_clear -->
<g id="life_events_graph_edge12" class="edge">
<title>finalized?&#45;&gt;tp_clear</title>
<g id="a_life_events_graph_edge12"><a xlink:title="finalized? to tp_clear: yes">
<path fill="none" stroke="black" d="M227.56,-430.1C236.46,-423.41 244.86,-415.02 249.8,-405 277.22,-349.39 274.06,-322.55 249.8,-265.5 249.7,-265.27 249.6,-265.04 249.49,-264.81"/>
<polygon fill="black" stroke="black" points="252.41,-262.88 243.93,-256.52 246.6,-266.78 252.41,-262.88"/>
</a>
</g>
<g id="a_life_events_graph_edge12&#45;label"><a xlink:title="finalized? to tp_clear: yes">
<text text-anchor="start" x="269.2" y="-316.73" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;yes &#160;</text>
</a>
</g>
</g>
<!-- tp_finalize&#45;&gt;tp_clear -->
<g id="life_events_graph_edge13" class="edge">
<title>tp_finalize&#45;&gt;tp_clear</title>
<g id="a_life_events_graph_edge13"><a xlink:title="tp_finalize to tp_clear: no refs or cyclic isolate">
<path fill="none" stroke="black" d="M130.02,-302.72C135.75,-290.85 144.8,-275.49 156.8,-265.5 161.95,-261.21 167.9,-257.57 174.07,-254.49"/>
<polygon fill="black" stroke="black" points="175.46,-257.71 183.18,-250.45 172.62,-251.31 175.46,-257.71"/>
</a>
</g>
<g id="a_life_events_graph_edge13&#45;label"><a xlink:title="tp_finalize to tp_clear: no refs or cyclic isolate">
<text text-anchor="start" x="164.3" y="-282.6" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;no refs or &#160;&#160;</text>
<text text-anchor="start" x="156.8" y="-268.35" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;cyclic isolate &#160;</text>
</a>
</g>
</g>
<!-- tp_finalize&#45;&gt;tp_dealloc -->
<g id="life_events_graph_edge14" class="edge">
<title>tp_finalize&#45;&gt;tp_dealloc</title>
<g id="a_life_events_graph_edge14"><a xlink:href="lifecycle.html#c.PyObject_CallFinalizerFromDealloc" xlink:title="tp_dealloc to tp_finalize: recommended call (see explanation)" target="_top">
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M85.85,-298.52C42.09,-270.18 -21.4,-218.11 7.8,-168.75 36.22,-120.7 99.95,-100.97 146.42,-92.87"/>
<polygon fill="none" stroke="black" points="83.78,-301.35 94.11,-303.72 87.52,-295.43 83.78,-301.35"/>
</a>
</g>
<g id="a_life_events_graph_edge14&#45;label"><a xlink:href="lifecycle.html#c.PyObject_CallFinalizerFromDealloc" xlink:title="tp_dealloc to tp_finalize: recommended call (see explanation)" target="_top">
<text text-anchor="start" x="7.8" y="-200.1" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;recommended</text>
<text text-anchor="start" x="25.8" y="-185.85" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;call (see</text>
<text text-anchor="start" x="13.05" y="-171.6" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;explanation)</text>
</a>
</g>
</g>
<!-- tp_finalize&#45;&gt;tp_dealloc -->
<g id="life_events_graph_edge15" class="edge">
<title>tp_finalize&#45;&gt;tp_dealloc</title>
<g id="a_life_events_graph_edge15"><a xlink:title="tp_finalize to tp_dealloc: no refs">
<path fill="none" stroke="black" d="M123.03,-302.58C123.95,-273.77 128.08,-214.78 146.05,-168.75 153.95,-148.5 167.56,-128.2 179.24,-112.92"/>
<polygon fill="black" stroke="black" points="181.81,-115.32 185.25,-105.3 176.31,-110.98 181.81,-115.32"/>
</a>
</g>
<g id="a_life_events_graph_edge15&#45;label"><a xlink:title="tp_finalize to tp_dealloc: no refs">
<text text-anchor="start" x="146.05" y="-185.85" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;&#160;no refs &#160;</text>
</a>
</g>
</g>
<!-- tp_clear&#45;&gt;uncollectable -->
<g id="life_events_graph_edge17" class="edge">
<title>tp_clear&#45;&gt;uncollectable</title>
<g id="a_life_events_graph_edge17"><a xlink:title="tp_clear to uncollectable: cyclic isolate">
<path fill="none" stroke="black" d="M227.75,-220.38C232.99,-205 242.67,-182.74 258.05,-168.75 260.43,-166.58 263.02,-164.58 265.74,-162.73"/>
<polygon fill="black" stroke="black" points="267.27,-165.89 274.12,-157.81 263.73,-159.86 267.27,-165.89"/>
</a>
</g>
<g id="a_life_events_graph_edge17&#45;label"><a xlink:title="tp_clear to uncollectable: cyclic isolate">
<text text-anchor="start" x="260.67" y="-192.97" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;cyclic &#160;</text>
<text text-anchor="start" x="258.05" y="-178.72" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;isolate &#160;</text>
</a>
</g>
</g>
<!-- tp_clear&#45;&gt;tp_dealloc -->
<g id="life_events_graph_edge16" class="edge">
<title>tp_clear&#45;&gt;tp_dealloc</title>
<g id="a_life_events_graph_edge16"><a xlink:title="tp_clear to tp_dealloc: no refs">
<path fill="none" stroke="black" d="M219.7,-220.24C216.92,-204.51 212.83,-180.61 209.8,-159.75 207.7,-145.34 205.67,-129.26 204.07,-115.92"/>
<polygon fill="black" stroke="black" points="207.56,-115.59 202.91,-106.07 200.61,-116.41 207.56,-115.59"/>
</a>
</g>
<g id="a_life_events_graph_edge16&#45;label"><a xlink:title="tp_clear to tp_dealloc: no refs">
<text text-anchor="start" x="209.8" y="-137.22" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;no refs &#160;</text>
</a>
</g>
</g>
<!-- uncollectable&#45;&gt;tp_dealloc -->
<!-- tp_free -->
<g id="life_events_graph_node12" class="node">
<title>tp_free</title>
<g id="a_life_events_graph_node12"><a xlink:href="typeobj.html#c.PyTypeObject.tp_free" xlink:title="tp_free" target="_top">
<ellipse fill="none" stroke="black" cx="200.8" cy="-18" rx="43.57" ry="18"/>
<text text-anchor="middle" x="200.8" y="-13.72" font-family="monospace,monospace" font-size="12.00">tp_free</text>
</a>
</g>
</g>
<!-- tp_dealloc&#45;&gt;tp_free -->
<g id="life_events_graph_edge21" class="edge">
<title>tp_dealloc&#45;&gt;tp_free</title>
<g id="a_life_events_graph_edge21"><a xlink:title="tp_dealloc to tp_free: direct call">
<path fill="none" stroke="black" d="M200.8,-67.84C200.8,-61.63 200.8,-54.46 200.8,-47.56"/>
<polygon fill="none" stroke="black" points="204.3,-47.57 200.8,-37.57 197.3,-47.57 204.3,-47.57"/>
</a>
</g>
<g id="a_life_events_graph_edge21&#45;label"><a xlink:title="tp_dealloc to tp_free: direct call">
<text text-anchor="start" x="200.8" y="-47.85" font-family="serif,serif" font-style="italic" font-size="12.00"> &#160;&#160;&#160;direct call &#160;</text>
</a>
</g>
</g>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 21 KiB

273
Doc/c-api/lifecycle.rst Normal file
View file

@ -0,0 +1,273 @@
.. highlight:: c
.. _life-cycle:
Object Life Cycle
=================
This section explains how a type's slots relate to each other throughout the
life of an object. It is not intended to be a complete canonical reference for
the slots; instead, refer to the slot-specific documentation in
:ref:`type-structs` for details about a particular slot.
Life Events
-----------
The figure below illustrates the order of events that can occur throughout an
object's life. An arrow from *A* to *B* indicates that event *B* can occur
after event *A* has occurred, with the arrow's label indicating the condition
that must be true for *B* to occur after *A*.
.. only:: html and not epub
.. raw:: html
<style type="text/css">
.. raw:: html
:file: lifecycle.dot.css
.. raw:: html
</style>
.. raw:: html
:file: lifecycle.dot.svg
.. raw:: html
<script>
(() => {
const g = document.getElementById('life_events_graph');
const title = g.querySelector(':scope > title');
title.id = 'life-events-graph-title';
const svg = g.closest('svg');
svg.role = 'img';
svg.setAttribute('aria-describedby',
'life-events-graph-description');
svg.setAttribute('aria-labelledby', 'life-events-graph-title');
})();
</script>
.. only:: epub or not (html or latex)
.. image:: lifecycle.dot.svg
:align: center
:class: invert-in-dark-mode
:alt: Diagram showing events in an object's life. Explained in detail
below.
.. only:: latex
.. image:: lifecycle.dot.pdf
:align: center
:class: invert-in-dark-mode
:alt: Diagram showing events in an object's life. Explained in detail
below.
.. container::
:name: life-events-graph-description
Explanation:
* When a new object is constructed by calling its type:
#. :c:member:`~PyTypeObject.tp_new` is called to create a new object.
#. :c:member:`~PyTypeObject.tp_alloc` is directly called by
:c:member:`~PyTypeObject.tp_new` to allocate the memory for the new
object.
#. :c:member:`~PyTypeObject.tp_init` initializes the newly created object.
:c:member:`!tp_init` can be called again to re-initialize an object, if
desired. The :c:member:`!tp_init` call can also be skipped entirely,
for example by Python code calling :py:meth:`~object.__new__`.
* After :c:member:`!tp_init` completes, the object is ready to use.
* Some time after the last reference to an object is removed:
#. If an object is not marked as *finalized*, it might be finalized by
marking it as *finalized* and calling its
:c:member:`~PyTypeObject.tp_finalize` function. Python does
*not* finalize an object when the last reference to it is deleted; use
:c:func:`PyObject_CallFinalizerFromDealloc` to ensure that
:c:member:`~PyTypeObject.tp_finalize` is always called.
#. If the object is marked as finalized,
:c:member:`~PyTypeObject.tp_clear` might be called by the garbage collector
to clear references held by the object. It is *not* called when the
object's reference count reaches zero.
#. :c:member:`~PyTypeObject.tp_dealloc` is called to destroy the object.
To avoid code duplication, :c:member:`~PyTypeObject.tp_dealloc` typically
calls into :c:member:`~PyTypeObject.tp_clear` to free up the object's
references.
#. When :c:member:`~PyTypeObject.tp_dealloc` finishes object destruction,
it directly calls :c:member:`~PyTypeObject.tp_free` (usually set to
:c:func:`PyObject_Free` or :c:func:`PyObject_GC_Del` automatically as
appropriate for the type) to deallocate the memory.
* The :c:member:`~PyTypeObject.tp_finalize` function is permitted to add a
reference to the object if desired. If it does, the object is
*resurrected*, preventing its pending destruction. (Only
:c:member:`!tp_finalize` is allowed to resurrect an object;
:c:member:`~PyTypeObject.tp_clear` and
:c:member:`~PyTypeObject.tp_dealloc` cannot without calling into
:c:member:`!tp_finalize`.) Resurrecting an object may
or may not cause the object's *finalized* mark to be removed. Currently,
Python does not remove the *finalized* mark from a resurrected object if
it supports garbage collection (i.e., the :c:macro:`Py_TPFLAGS_HAVE_GC`
flag is set) but does remove the mark if the object does not support
garbage collection; either or both of these behaviors may change in the
future.
* :c:member:`~PyTypeObject.tp_dealloc` can optionally call
:c:member:`~PyTypeObject.tp_finalize` via
:c:func:`PyObject_CallFinalizerFromDealloc` if it wishes to reuse that
code to help with object destruction. This is recommended because it
guarantees that :c:member:`!tp_finalize` is always called before
destruction. See the :c:member:`~PyTypeObject.tp_dealloc` documentation
for example code.
* If the object is a member of a :term:`cyclic isolate` and either
:c:member:`~PyTypeObject.tp_clear` fails to break the reference cycle or
the cyclic isolate is not detected (perhaps :func:`gc.disable` was called,
or the :c:macro:`Py_TPFLAGS_HAVE_GC` flag was erroneously omitted in one
of the involved types), the objects remain indefinitely uncollectable
(they "leak"). See :data:`gc.garbage`.
If the object is marked as supporting garbage collection (the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag is set in
:c:member:`~PyTypeObject.tp_flags`), the following events are also possible:
* The garbage collector occasionally calls
:c:member:`~PyTypeObject.tp_traverse` to identify :term:`cyclic isolates
<cyclic isolate>`.
* When the garbage collector discovers a :term:`cyclic isolate`, it
finalizes one of the objects in the group by marking it as *finalized* and
calling its :c:member:`~PyTypeObject.tp_finalize` function, if it has one.
This repeats until the cyclic isolate doesn't exist or all of the objects
have been finalized.
* :c:member:`~PyTypeObject.tp_finalize` is permitted to resurrect the object
by adding a reference from outside the :term:`cyclic isolate`. The new
reference causes the group of objects to no longer form a cyclic isolate
(the reference cycle may still exist, but if it does the objects are no
longer isolated).
* When the garbage collector discovers a :term:`cyclic isolate` and all of
the objects in the group have already been marked as *finalized*, the
garbage collector clears one or more of the uncleared objects in the group
(possibly concurrently) by calling each's
:c:member:`~PyTypeObject.tp_clear` function. This repeats as long as the
cyclic isolate still exists and not all of the objects have been cleared.
Cyclic Isolate Destruction
--------------------------
Listed below are the stages of life of a hypothetical :term:`cyclic isolate`
that continues to exist after each member object is finalized or cleared. It
is a memory leak if a cyclic isolate progresses through all of these stages; it should
vanish once all objects are cleared, if not sooner. A cyclic isolate can
vanish either because the reference cycle is broken or because the objects are
no longer isolated due to finalizer resurrection (see
:c:member:`~PyTypeObject.tp_finalize`).
0. **Reachable** (not yet a cyclic isolate): All objects are in their normal,
reachable state. A reference cycle could exist, but an external reference
means the objects are not yet isolated.
#. **Unreachable but consistent:** The final reference from outside the cyclic
group of objects has been removed, causing the objects to become isolated
(thus a cyclic isolate is born). None of the group's objects have been
finalized or cleared yet. The cyclic isolate remains at this stage until
some future run of the garbage collector (not necessarily the next run
because the next run might not scan every object).
#. **Mix of finalized and not finalized:** Objects in a cyclic isolate are
finalized one at a time, which means that there is a period of time when the
cyclic isolate is composed of a mix of finalized and non-finalized objects.
Finalization order is unspecified, so it can appear random. A finalized
object must behave in a sane manner when non-finalized objects interact with
it, and a non-finalized object must be able to tolerate the finalization of
an arbitrary subset of its referents.
#. **All finalized:** All objects in a cyclic isolate are finalized before any
of them are cleared.
#. **Mix of finalized and cleared:** The objects can be cleared serially or
concurrently (but with the :term:`GIL` held); either way, some will finish
before others. A finalized object must be able to tolerate the clearing of
a subset of its referents. :pep:`442` calls this stage "cyclic trash".
#. **Leaked:** If a cyclic isolate still exists after all objects in the group
have been finalized and cleared, then the objects remain indefinitely
uncollectable (see :data:`gc.garbage`). It is a bug if a cyclic isolate
reaches this stage---it means the :c:member:`~PyTypeObject.tp_clear` methods
of the participating objects have failed to break the reference cycle as
required.
If :c:member:`~PyTypeObject.tp_clear` did not exist, then Python would have no
way to safely break a reference cycle. Simply destroying an object in a cyclic
isolate would result in a dangling pointer, triggering undefined behavior when
an object referencing the destroyed object is itself destroyed. The clearing
step makes object destruction a two-phase process: first
:c:member:`~PyTypeObject.tp_clear` is called to partially destroy the objects
enough to detangle them from each other, then
:c:member:`~PyTypeObject.tp_dealloc` is called to complete the destruction.
Unlike clearing, finalization is not a phase of destruction. A finalized
object must still behave properly by continuing to fulfill its design
contracts. An object's finalizer is allowed to execute arbitrary Python code,
and is even allowed to prevent the impending destruction by adding a reference.
The finalizer is only related to destruction by call order---if it runs, it runs
before destruction, which starts with :c:member:`~PyTypeObject.tp_clear` (if
called) and concludes with :c:member:`~PyTypeObject.tp_dealloc`.
The finalization step is not necessary to safely reclaim the objects in a
cyclic isolate, but its existence makes it easier to design types that behave
in a sane manner when objects are cleared. Clearing an object might
necessarily leave it in a broken, partially destroyed state---it might be
unsafe to call any of the cleared object's methods or access any of its
attributes. With finalization, only finalized objects can possibly interact
with cleared objects; non-finalized objects are guaranteed to interact with
only non-cleared (but potentially finalized) objects.
To summarize the possible interactions:
* A non-finalized object might have references to or from non-finalized and
finalized objects, but not to or from cleared objects.
* A finalized object might have references to or from non-finalized, finalized,
and cleared objects.
* A cleared object might have references to or from finalized and cleared
objects, but not to or from non-finalized objects.
Without any reference cycles, an object can be simply destroyed once its last
reference is deleted; the finalization and clearing steps are not necessary to
safely reclaim unused objects. However, it can be useful to automatically call
:c:member:`~PyTypeObject.tp_finalize` and :c:member:`~PyTypeObject.tp_clear`
before destruction anyway because type design is simplified when all objects
always experience the same series of events regardless of whether they
participated in a cyclic isolate. Python currently only calls
:c:member:`~PyTypeObject.tp_finalize` and :c:member:`~PyTypeObject.tp_clear` as
needed to destroy a cyclic isolate; this may change in a future version.
Functions
---------
To allocate and free memory, see :ref:`allocating-objects`.
.. c:function:: void PyObject_CallFinalizer(PyObject *op)
Finalizes the object as described in :c:member:`~PyTypeObject.tp_finalize`.
Call this function (or :c:func:`PyObject_CallFinalizerFromDealloc`) instead
of calling :c:member:`~PyTypeObject.tp_finalize` directly because this
function may deduplicate multiple calls to :c:member:`!tp_finalize`.
Currently, calls are only deduplicated if the type supports garbage
collection (i.e., the :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set); this may
change in the future.
.. c:function:: int PyObject_CallFinalizerFromDealloc(PyObject *op)
Same as :c:func:`PyObject_CallFinalizer` but meant to be called at the
beginning of the object's destructor (:c:member:`~PyTypeObject.tp_dealloc`).
There must not be any references to the object. If the object's finalizer
resurrects the object, this function returns -1; no further destruction
should happen. Otherwise, this function returns 0 and destruction can
continue normally.
.. seealso::
:c:member:`~PyTypeObject.tp_dealloc` for example code.

18
Doc/c-api/memory.rst
View file

@ -376,6 +376,24 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
If *p* is ``NULL``, no operation is performed.
Do not call this directly to free an object's memory; call the type's
:c:member:`~PyTypeObject.tp_free` slot instead.
Do not use this for memory allocated by :c:macro:`PyObject_GC_New` or
:c:macro:`PyObject_GC_NewVar`; use :c:func:`PyObject_GC_Del` instead.
.. seealso::
* :c:func:`PyObject_GC_Del` is the equivalent of this function for memory
allocated by types that support garbage collection.
* :c:func:`PyObject_Malloc`
* :c:func:`PyObject_Realloc`
* :c:func:`PyObject_Calloc`
* :c:macro:`PyObject_New`
* :c:macro:`PyObject_NewVar`
* :c:func:`PyType_GenericAlloc`
* :c:member:`~PyTypeObject.tp_free`
.. _default-memory-allocators:

50
Doc/c-api/object.rst
View file

@ -613,6 +613,38 @@ Object Protocol
.. versionadded:: 3.14
.. c:function:: int PyUnstable_Object_IsUniqueReferencedTemporary(PyObject *obj)
Check if *obj* is a unique temporary object.
Returns ``1`` if *obj* is known to be a unique temporary object,
and ``0`` otherwise. This function cannot fail, but the check is
conservative, and may return ``0`` in some cases even if *obj* is a unique
temporary object.
If an object is a unique temporary, it is guaranteed that the current code
has the only reference to the object. For arguments to C functions, this
should be used instead of checking if the reference count is ``1``. Starting
with Python 3.14, the interpreter internally avoids some reference count
modifications when loading objects onto the operands stack by
:term:`borrowing <borrowed reference>` references when possible, which means
that a reference count of ``1`` by itself does not guarantee that a function
argument uniquely referenced.
In the example below, ``my_func`` is called with a unique temporary object
as its argument::
my_func([1, 2, 3])
In the example below, ``my_func`` is **not** called with a unique temporary
object as its argument, even if its refcount is ``1``::
my_list = [1, 2, 3]
my_func(my_list)
See also the function :c:func:`Py_REFCNT`.
.. versionadded:: 3.14
.. c:function:: int PyUnstable_IsImmortal(PyObject *obj)
This function returns non-zero if *obj* is :term:`immortal`, and zero
@ -705,3 +737,21 @@ Object Protocol
caller must hold a :term:`strong reference` to *obj* when calling this.
.. versionadded:: 3.14
.. c:function:: int PyUnstable_Object_IsUniquelyReferenced(PyObject *op)
Determine if *op* only has one reference.
On GIL-enabled builds, this function is equivalent to
:c:expr:`Py_REFCNT(op) == 1`.
On a :term:`free threaded <free threading>` build, this checks if *op*'s
:term:`reference count` is equal to one and additionally checks if *op*
is only used by this thread. :c:expr:`Py_REFCNT(op) == 1` is **not**
thread-safe on free threaded builds; prefer this function.
The caller must hold an :term:`attached thread state`, despite the fact
that this function doesn't call into the Python interpreter. This function
cannot fail.
.. versionadded:: 3.14

1
Doc/c-api/objimpl.rst
View file

@ -12,6 +12,7 @@ object types.
.. toctree::
allocation.rst
lifecycle.rst
structures.rst
typeobj.rst
gcsupport.rst

9
Doc/c-api/refcounting.rst
View file

@ -23,6 +23,15 @@ of Python objects.
Use the :c:func:`Py_SET_REFCNT()` function to set an object reference count.
.. note::
On :term:`free threaded <free threading>` builds of Python, returning 1
isn't sufficient to determine if it's safe to treat *o* as having no
access by other threads. Use :c:func:`PyUnstable_Object_IsUniquelyReferenced`
for that instead.
See also the function :c:func:`PyUnstable_Object_IsUniqueReferencedTemporary()`.
.. versionchanged:: 3.10
:c:func:`Py_REFCNT()` is changed to the inline static function.

25
Doc/c-api/type.rst
View file

@ -151,14 +151,29 @@ Type Objects
.. c:function:: PyObject* PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems)
Generic handler for the :c:member:`~PyTypeObject.tp_alloc` slot of a type object. Use
Python's default memory allocation mechanism to allocate a new instance and
initialize all its contents to ``NULL``.
Generic handler for the :c:member:`~PyTypeObject.tp_alloc` slot of a type
object. Uses Python's default memory allocation mechanism to allocate memory
for a new instance, zeros the memory, then initializes the memory as if by
calling :c:func:`PyObject_Init` or :c:func:`PyObject_InitVar`.
Do not call this directly to allocate memory for an object; call the type's
:c:member:`~PyTypeObject.tp_alloc` slot instead.
For types that support garbage collection (i.e., the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag is set), this function behaves like
:c:macro:`PyObject_GC_New` or :c:macro:`PyObject_GC_NewVar` (except the
memory is guaranteed to be zeroed before initialization), and should be
paired with :c:func:`PyObject_GC_Del` in :c:member:`~PyTypeObject.tp_free`.
Otherwise, it behaves like :c:macro:`PyObject_New` or
:c:macro:`PyObject_NewVar` (except the memory is guaranteed to be zeroed
before initialization) and should be paired with :c:func:`PyObject_Free` in
:c:member:`~PyTypeObject.tp_free`.
.. c:function:: PyObject* PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds)
Generic handler for the :c:member:`~PyTypeObject.tp_new` slot of a type object. Create a
new instance using the type's :c:member:`~PyTypeObject.tp_alloc` slot.
Generic handler for the :c:member:`~PyTypeObject.tp_new` slot of a type
object. Creates a new instance using the type's
:c:member:`~PyTypeObject.tp_alloc` slot and returns the resulting object.
.. c:function:: int PyType_Ready(PyTypeObject *type)

470
Doc/c-api/typeobj.rst
View file

@ -79,7 +79,7 @@ Quick Reference
| :c:member:`~PyTypeObject.tp_setattro` | :c:type:`setattrofunc` | __setattr__, | X | X | | G |
| | | __delattr__ | | | | |
+------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
| :c:member:`~PyTypeObject.tp_as_buffer` | :c:type:`PyBufferProcs` * | | | | | % |
| :c:member:`~PyTypeObject.tp_as_buffer` | :c:type:`PyBufferProcs` * | :ref:`sub-slots` | | | | % |
+------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
| :c:member:`~PyTypeObject.tp_flags` | unsigned long | | X | X | | ? |
+------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
@ -325,9 +325,10 @@ sub-slots
+---------------------------------------------------------+-----------------------------------+---------------+
| |
+---------------------------------------------------------+-----------------------------------+---------------+
| :c:member:`~PyBufferProcs.bf_getbuffer` | :c:func:`getbufferproc` | |
| :c:member:`~PyBufferProcs.bf_getbuffer` | :c:func:`getbufferproc` | __buffer__ |
+---------------------------------------------------------+-----------------------------------+---------------+
| :c:member:`~PyBufferProcs.bf_releasebuffer` | :c:func:`releasebufferproc` | |
| :c:member:`~PyBufferProcs.bf_releasebuffer` | :c:func:`releasebufferproc` | __release_\ |
| | | buffer\__ |
+---------------------------------------------------------+-----------------------------------+---------------+
.. _slot-typedefs-table:
@ -611,7 +612,7 @@ and :c:data:`PyType_Type` effectively act as defaults.)
Note that the :c:member:`~PyVarObject.ob_size` field may later be used for
other purposes. For example, :py:type:`int` instances use the bits of
:c:member:`~PyVarObject.ob_size` in an implementation-defined
way; the underlying storage and its size should be acessed using
way; the underlying storage and its size should be accessed using
:c:func:`PyLong_Export`.
.. note::
@ -676,72 +677,122 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. c:member:: destructor PyTypeObject.tp_dealloc
A pointer to the instance destructor function. This function must be defined
unless the type guarantees that its instances will never be deallocated (as is
the case for the singletons ``None`` and ``Ellipsis``). The function signature is::
A pointer to the instance destructor function. The function signature is::
void tp_dealloc(PyObject *self);
The destructor function is called by the :c:func:`Py_DECREF` and
:c:func:`Py_XDECREF` macros when the new reference count is zero. At this point,
the instance is still in existence, but there are no references to it. The
destructor function should free all references which the instance owns, free all
memory buffers owned by the instance (using the freeing function corresponding
to the allocation function used to allocate the buffer), and call the type's
:c:member:`~PyTypeObject.tp_free` function. If the type is not subtypable
(doesn't have the :c:macro:`Py_TPFLAGS_BASETYPE` flag bit set), it is
permissible to call the object deallocator directly instead of via
:c:member:`~PyTypeObject.tp_free`. The object deallocator should be the one used to allocate the
instance; this is normally :c:func:`PyObject_Free` if the instance was allocated
using :c:macro:`PyObject_New` or :c:macro:`PyObject_NewVar`, or
:c:func:`PyObject_GC_Del` if the instance was allocated using
:c:macro:`PyObject_GC_New` or :c:macro:`PyObject_GC_NewVar`.
The destructor function should remove all references which the instance owns
(e.g., call :c:func:`Py_CLEAR`), free all memory buffers owned by the
instance, and call the type's :c:member:`~PyTypeObject.tp_free` function to
free the object itself.
If the type supports garbage collection (has the :c:macro:`Py_TPFLAGS_HAVE_GC`
flag bit set), the destructor should call :c:func:`PyObject_GC_UnTrack`
No guarantees are made about when an object is destroyed, except:
* Python will destroy an object immediately or some time after the final
reference to the object is deleted, unless its finalizer
(:c:member:`~PyTypeObject.tp_finalize`) subsequently resurrects the
object.
* An object will not be destroyed while it is being automatically finalized
(:c:member:`~PyTypeObject.tp_finalize`) or automatically cleared
(:c:member:`~PyTypeObject.tp_clear`).
CPython currently destroys an object immediately from :c:func:`Py_DECREF`
when the new reference count is zero, but this may change in a future
version.
It is recommended to call :c:func:`PyObject_CallFinalizerFromDealloc` at the
beginning of :c:member:`!tp_dealloc` to guarantee that the object is always
finalized before destruction.
If the type supports garbage collection (the :c:macro:`Py_TPFLAGS_HAVE_GC`
flag is set), the destructor should call :c:func:`PyObject_GC_UnTrack`
before clearing any member fields.
.. code-block:: c
It is permissible to call :c:member:`~PyTypeObject.tp_clear` from
:c:member:`!tp_dealloc` to reduce code duplication and to guarantee that the
object is always cleared before destruction. Beware that
:c:member:`!tp_clear` might have already been called.
static void foo_dealloc(foo_object *self) {
If the type is heap allocated (:c:macro:`Py_TPFLAGS_HEAPTYPE`), the
deallocator should release the owned reference to its type object (via
:c:func:`Py_DECREF`) after calling the type deallocator. See the example
code below.::
static void
foo_dealloc(PyObject *op)
{
foo_object *self = (foo_object *) op;
PyObject_GC_UnTrack(self);
Py_CLEAR(self->ref);
Py_TYPE(self)->tp_free((PyObject *)self);
}
Py_TYPE(self)->tp_free(self);
}
Finally, if the type is heap allocated (:c:macro:`Py_TPFLAGS_HEAPTYPE`), the
deallocator should release the owned reference to its type object
(via :c:func:`Py_DECREF`) after
calling the type deallocator. In order to avoid dangling pointers, the
recommended way to achieve this is:
:c:member:`!tp_dealloc` must leave the exception status unchanged. If it
needs to call something that might raise an exception, the exception state
must be backed up first and restored later (after logging any exceptions
with :c:func:`PyErr_WriteUnraisable`).
.. code-block:: c
Example::
static void foo_dealloc(foo_object *self) {
PyTypeObject *tp = Py_TYPE(self);
// free references and buffers here
tp->tp_free(self);
Py_DECREF(tp);
}
static void
foo_dealloc(PyObject *self)
{
PyObject *exc = PyErr_GetRaisedException();
.. warning::
if (PyObject_CallFinalizerFromDealloc(self) < 0) {
// self was resurrected.
goto done;
}
In a garbage collected Python, :c:member:`!tp_dealloc` may be called from
any Python thread, not just the thread which created the object (if the
object becomes part of a refcount cycle, that cycle might be collected by
a garbage collection on any thread). This is not a problem for Python
API calls, since the thread on which :c:member:`!tp_dealloc` is called
with an :term:`attached thread state`. However, if the object being
destroyed in turn destroys objects from some other C or C++ library, care
should be taken to ensure that destroying those objects on the thread
which called :c:member:`!tp_dealloc` will not violate any assumptions of
the library.
PyTypeObject *tp = Py_TYPE(self);
if (tp->tp_flags & Py_TPFLAGS_HAVE_GC) {
PyObject_GC_UnTrack(self);
}
// Optional, but convenient to avoid code duplication.
if (tp->tp_clear && tp->tp_clear(self) < 0) {
PyErr_WriteUnraisable(self);
}
// Any additional destruction goes here.
tp->tp_free(self);
self = NULL; // In case PyErr_WriteUnraisable() is called below.
if (tp->tp_flags & Py_TPFLAGS_HEAPTYPE) {
Py_CLEAR(tp);
}
done:
// Optional, if something was called that might have raised an
// exception.
if (PyErr_Occurred()) {
PyErr_WriteUnraisable(self);
}
PyErr_SetRaisedException(exc);
}
:c:member:`!tp_dealloc` may be called from
any Python thread, not just the thread which created the object (if the
object becomes part of a refcount cycle, that cycle might be collected by
a garbage collection on any thread). This is not a problem for Python
API calls, since the thread on which :c:member:`!tp_dealloc` is called
with an :term:`attached thread state`. However, if the object being
destroyed in turn destroys objects from some other C library, care
should be taken to ensure that destroying those objects on the thread
which called :c:member:`!tp_dealloc` will not violate any assumptions of
the library.
**Inheritance:**
This field is inherited by subtypes.
.. seealso::
:ref:`life-cycle` for details about how this slot relates to other slots.
.. c:member:: Py_ssize_t PyTypeObject.tp_vectorcall_offset
@ -1132,11 +1183,11 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. c:macro:: Py_TPFLAGS_HAVE_GC
This bit is set when the object supports garbage collection. If this bit
is set, instances must be created using :c:macro:`PyObject_GC_New` and
destroyed using :c:func:`PyObject_GC_Del`. More information in section
:ref:`supporting-cycle-detection`. This bit also implies that the
GC-related fields :c:member:`~PyTypeObject.tp_traverse` and :c:member:`~PyTypeObject.tp_clear` are present in
the type object.
is set, memory for new instances (see :c:member:`~PyTypeObject.tp_alloc`)
must be allocated using :c:macro:`PyObject_GC_New` or
:c:func:`PyType_GenericAlloc` and deallocated (see
:c:member:`~PyTypeObject.tp_free`) using :c:func:`PyObject_GC_Del`. More
information in section :ref:`supporting-cycle-detection`.
**Inheritance:**
@ -1416,8 +1467,9 @@ and :c:data:`PyType_Type` effectively act as defaults.)
:mod:`!_thread` extension module::
static int
local_traverse(localobject *self, visitproc visit, void *arg)
local_traverse(PyObject *op, visitproc visit, void *arg)
{
localobject *self = (localobject *) op;
Py_VISIT(self->args);
Py_VISIT(self->kw);
Py_VISIT(self->dict);
@ -1472,6 +1524,11 @@ and :c:data:`PyType_Type` effectively act as defaults.)
heap-allocated superclass).
If they do not, the type object may not be garbage-collected.
.. note::
The :c:member:`~PyTypeObject.tp_traverse` function can be called from any
thread.
.. versionchanged:: 3.9
Heap-allocated types are expected to visit ``Py_TYPE(self)`` in
@ -1491,28 +1548,110 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. c:member:: inquiry PyTypeObject.tp_clear
An optional pointer to a clear function for the garbage collector. This is only
used if the :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit is set. The signature is::
An optional pointer to a clear function. The signature is::
int tp_clear(PyObject *);
The :c:member:`~PyTypeObject.tp_clear` member function is used to break reference cycles in cyclic
garbage detected by the garbage collector. Taken together, all :c:member:`~PyTypeObject.tp_clear`
functions in the system must combine to break all reference cycles. This is
subtle, and if in any doubt supply a :c:member:`~PyTypeObject.tp_clear` function. For example,
the tuple type does not implement a :c:member:`~PyTypeObject.tp_clear` function, because it's
possible to prove that no reference cycle can be composed entirely of tuples.
Therefore the :c:member:`~PyTypeObject.tp_clear` functions of other types must be sufficient to
break any cycle containing a tuple. This isn't immediately obvious, and there's
rarely a good reason to avoid implementing :c:member:`~PyTypeObject.tp_clear`.
The purpose of this function is to break reference cycles that are causing a
:term:`cyclic isolate` so that the objects can be safely destroyed. A
cleared object is a partially destroyed object; the object is not obligated
to satisfy design invariants held during normal use.
:c:member:`!tp_clear` does not need to delete references to objects that
can't participate in reference cycles, such as Python strings or Python
integers. However, it may be convenient to clear all references, and write
the type's :c:member:`~PyTypeObject.tp_dealloc` function to invoke
:c:member:`!tp_clear` to avoid code duplication. (Beware that
:c:member:`!tp_clear` might have already been called. Prefer calling
idempotent functions like :c:func:`Py_CLEAR`.)
Any non-trivial cleanup should be performed in
:c:member:`~PyTypeObject.tp_finalize` instead of :c:member:`!tp_clear`.
.. note::
If :c:member:`!tp_clear` fails to break a reference cycle then the
objects in the :term:`cyclic isolate` may remain indefinitely
uncollectable ("leak"). See :data:`gc.garbage`.
.. note::
Referents (direct and indirect) might have already been cleared; they are
not guaranteed to be in a consistent state.
.. note::
The :c:member:`~PyTypeObject.tp_clear` function can be called from any
thread.
.. note::
An object is not guaranteed to be automatically cleared before its
destructor (:c:member:`~PyTypeObject.tp_dealloc`) is called.
This function differs from the destructor
(:c:member:`~PyTypeObject.tp_dealloc`) in the following ways:
* The purpose of clearing an object is to remove references to other objects
that might participate in a reference cycle. The purpose of the
destructor, on the other hand, is a superset: it must release *all*
resources it owns, including references to objects that cannot participate
in a reference cycle (e.g., integers) as well as the object's own memory
(by calling :c:member:`~PyTypeObject.tp_free`).
* When :c:member:`!tp_clear` is called, other objects might still hold
references to the object being cleared. Because of this,
:c:member:`!tp_clear` must not deallocate the object's own memory
(:c:member:`~PyTypeObject.tp_free`). The destructor, on the other hand,
is only called when no (strong) references exist, and as such, must
safely destroy the object itself by deallocating it.
* :c:member:`!tp_clear` might never be automatically called. An object's
destructor, on the other hand, will be automatically called some time
after the object becomes unreachable (i.e., either there are no references
to the object or the object is a member of a :term:`cyclic isolate`).
No guarantees are made about when, if, or how often Python automatically
clears an object, except:
* Python will not automatically clear an object if it is reachable, i.e.,
there is a reference to it and it is not a member of a :term:`cyclic
isolate`.
* Python will not automatically clear an object if it has not been
automatically finalized (see :c:member:`~PyTypeObject.tp_finalize`). (If
the finalizer resurrected the object, the object may or may not be
automatically finalized again before it is cleared.)
* If an object is a member of a :term:`cyclic isolate`, Python will not
automatically clear it if any member of the cyclic isolate has not yet
been automatically finalized (:c:member:`~PyTypeObject.tp_finalize`).
* Python will not destroy an object until after any automatic calls to its
:c:member:`!tp_clear` function have returned. This ensures that the act
of breaking a reference cycle does not invalidate the ``self`` pointer
while :c:member:`!tp_clear` is still executing.
* Python will not automatically call :c:member:`!tp_clear` multiple times
concurrently.
CPython currently only automatically clears objects as needed to break
reference cycles in a :term:`cyclic isolate`, but future versions might
clear objects regularly before their destruction.
Taken together, all :c:member:`~PyTypeObject.tp_clear` functions in the
system must combine to break all reference cycles. This is subtle, and if
in any doubt supply a :c:member:`~PyTypeObject.tp_clear` function. For
example, the tuple type does not implement a
:c:member:`~PyTypeObject.tp_clear` function, because it's possible to prove
that no reference cycle can be composed entirely of tuples. Therefore the
:c:member:`~PyTypeObject.tp_clear` functions of other types are responsible
for breaking any cycle containing a tuple. This isn't immediately obvious,
and there's rarely a good reason to avoid implementing
:c:member:`~PyTypeObject.tp_clear`.
Implementations of :c:member:`~PyTypeObject.tp_clear` should drop the instance's references to
those of its members that may be Python objects, and set its pointers to those
members to ``NULL``, as in the following example::
static int
local_clear(localobject *self)
local_clear(PyObject *op)
{
localobject *self = (localobject *) op;
Py_CLEAR(self->key);
Py_CLEAR(self->args);
Py_CLEAR(self->kw);
@ -1538,18 +1677,6 @@ and :c:data:`PyType_Type` effectively act as defaults.)
PyObject_ClearManagedDict((PyObject*)self);
Note that :c:member:`~PyTypeObject.tp_clear` is not *always* called
before an instance is deallocated. For example, when reference counting
is enough to determine that an object is no longer used, the cyclic garbage
collector is not involved and :c:member:`~PyTypeObject.tp_dealloc` is
called directly.
Because the goal of :c:member:`~PyTypeObject.tp_clear` functions is to break reference cycles,
it's not necessary to clear contained objects like Python strings or Python
integers, which can't participate in reference cycles. On the other hand, it may
be convenient to clear all contained Python objects, and write the type's
:c:member:`~PyTypeObject.tp_dealloc` function to invoke :c:member:`~PyTypeObject.tp_clear`.
More information about Python's garbage collection scheme can be found in
section :ref:`supporting-cycle-detection`.
@ -1562,6 +1689,10 @@ and :c:data:`PyType_Type` effectively act as defaults.)
:c:member:`~PyTypeObject.tp_clear` are all inherited from the base type if they are all zero in
the subtype.
.. seealso::
:ref:`life-cycle` for details about how this slot relates to other slots.
.. c:member:: richcmpfunc PyTypeObject.tp_richcompare
@ -1938,18 +2069,17 @@ and :c:data:`PyType_Type` effectively act as defaults.)
**Inheritance:**
This field is inherited by static subtypes, but not by dynamic
subtypes (subtypes created by a class statement).
Static subtypes inherit this slot, which will be
:c:func:`PyType_GenericAlloc` if inherited from :class:`object`.
:ref:`Heap subtypes <heap-types>` do not inherit this slot.
**Default:**
For dynamic subtypes, this field is always set to
:c:func:`PyType_GenericAlloc`, to force a standard heap
allocation strategy.
For heap subtypes, this field is always set to
:c:func:`PyType_GenericAlloc`.
For static subtypes, :c:data:`PyBaseObject_Type` uses
:c:func:`PyType_GenericAlloc`. That is the recommended value
for all statically defined types.
For static subtypes, this slot is inherited (see above).
.. c:member:: newfunc PyTypeObject.tp_new
@ -1997,20 +2127,27 @@ and :c:data:`PyType_Type` effectively act as defaults.)
void tp_free(void *self);
An initializer that is compatible with this signature is :c:func:`PyObject_Free`.
This function must free the memory allocated by
:c:member:`~PyTypeObject.tp_alloc`.
**Inheritance:**
This field is inherited by static subtypes, but not by dynamic
subtypes (subtypes created by a class statement)
Static subtypes inherit this slot, which will be :c:func:`PyObject_Free` if
inherited from :class:`object`. Exception: If the type supports garbage
collection (i.e., the :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set in
:c:member:`~PyTypeObject.tp_flags`) and it would inherit
:c:func:`PyObject_Free`, then this slot is not inherited but instead defaults
to :c:func:`PyObject_GC_Del`.
:ref:`Heap subtypes <heap-types>` do not inherit this slot.
**Default:**
In dynamic subtypes, this field is set to a deallocator suitable to
match :c:func:`PyType_GenericAlloc` and the value of the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag bit.
For :ref:`heap subtypes <heap-types>`, this slot defaults to a deallocator suitable to match
:c:func:`PyType_GenericAlloc` and the value of the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag.
For static subtypes, :c:data:`PyBaseObject_Type` uses :c:func:`PyObject_Free`.
For static subtypes, this slot is inherited (see above).
.. c:member:: inquiry PyTypeObject.tp_is_gc
@ -2137,29 +2274,138 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. c:member:: destructor PyTypeObject.tp_finalize
An optional pointer to an instance finalization function. Its signature is::
An optional pointer to an instance finalization function. This is the C
implementation of the :meth:`~object.__del__` special method. Its signature
is::
void tp_finalize(PyObject *self);
If :c:member:`~PyTypeObject.tp_finalize` is set, the interpreter calls it once when
finalizing an instance. It is called either from the garbage
collector (if the instance is part of an isolated reference cycle) or
just before the object is deallocated. Either way, it is guaranteed
to be called before attempting to break reference cycles, ensuring
that it finds the object in a sane state.
The primary purpose of finalization is to perform any non-trivial cleanup
that must be performed before the object is destroyed, while the object and
any other objects it directly or indirectly references are still in a
consistent state. The finalizer is allowed to execute
arbitrary Python code.
:c:member:`~PyTypeObject.tp_finalize` should not mutate the current exception status;
therefore, a recommended way to write a non-trivial finalizer is::
Before Python automatically finalizes an object, some of the object's direct
or indirect referents might have themselves been automatically finalized.
However, none of the referents will have been automatically cleared
(:c:member:`~PyTypeObject.tp_clear`) yet.
Other non-finalized objects might still be using a finalized object, so the
finalizer must leave the object in a sane state (e.g., invariants are still
met).
.. note::
After Python automatically finalizes an object, Python might start
automatically clearing (:c:member:`~PyTypeObject.tp_clear`) the object
and its referents (direct and indirect). Cleared objects are not
guaranteed to be in a consistent state; a finalized object must be able
to tolerate cleared referents.
.. note::
An object is not guaranteed to be automatically finalized before its
destructor (:c:member:`~PyTypeObject.tp_dealloc`) is called. It is
recommended to call :c:func:`PyObject_CallFinalizerFromDealloc` at the
beginning of :c:member:`!tp_dealloc` to guarantee that the object is
always finalized before destruction.
.. note::
The :c:member:`~PyTypeObject.tp_finalize` function can be called from any
thread, although the :term:`GIL` will be held.
.. note::
The :c:member:`!tp_finalize` function can be called during shutdown,
after some global variables have been deleted. See the documentation of
the :meth:`~object.__del__` method for details.
When Python finalizes an object, it behaves like the following algorithm:
#. Python might mark the object as *finalized*. Currently, Python always
marks objects whose type supports garbage collection (i.e., the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag is set in
:c:member:`~PyTypeObject.tp_flags`) and never marks other types of
objects; this might change in a future version.
#. If the object is not marked as *finalized* and its
:c:member:`!tp_finalize` finalizer function is non-``NULL``, the
finalizer function is called.
#. If the finalizer function was called and the finalizer made the object
reachable (i.e., there is a reference to the object and it is not a
member of a :term:`cyclic isolate`), then the finalizer is said to have
*resurrected* the object. It is unspecified whether the finalizer can
also resurrect the object by adding a new reference to the object that
does not make it reachable, i.e., the object is (still) a member of a
cyclic isolate.
#. If the finalizer resurrected the object, the object's pending destruction
is canceled and the object's *finalized* mark might be removed if
present. Currently, Python never removes the *finalized* mark; this
might change in a future version.
*Automatic finalization* refers to any finalization performed by Python
except via calls to :c:func:`PyObject_CallFinalizer` or
:c:func:`PyObject_CallFinalizerFromDealloc`. No guarantees are made about
when, if, or how often an object is automatically finalized, except:
* Python will not automatically finalize an object if it is reachable, i.e.,
there is a reference to it and it is not a member of a :term:`cyclic
isolate`.
* Python will not automatically finalize an object if finalizing it would
not mark the object as *finalized*. Currently, this applies to objects
whose type does not support garbage collection, i.e., the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag is not set. Such objects can still be
manually finalized by calling :c:func:`PyObject_CallFinalizer` or
:c:func:`PyObject_CallFinalizerFromDealloc`.
* Python will not automatically finalize any two members of a :term:`cyclic
isolate` concurrently.
* Python will not automatically finalize an object after it has
automatically cleared (:c:member:`~PyTypeObject.tp_clear`) the object.
* If an object is a member of a :term:`cyclic isolate`, Python will not
automatically finalize it after automatically clearing (see
:c:member:`~PyTypeObject.tp_clear`) any other member.
* Python will automatically finalize every member of a :term:`cyclic
isolate` before it automatically clears (see
:c:member:`~PyTypeObject.tp_clear`) any of them.
* If Python is going to automatically clear an object
(:c:member:`~PyTypeObject.tp_clear`), it will automatically finalize the
object first.
Python currently only automatically finalizes objects that are members of a
:term:`cyclic isolate`, but future versions might finalize objects regularly
before their destruction.
To manually finalize an object, do not call this function directly; call
:c:func:`PyObject_CallFinalizer` or
:c:func:`PyObject_CallFinalizerFromDealloc` instead.
:c:member:`~PyTypeObject.tp_finalize` should leave the current exception
status unchanged. The recommended way to write a non-trivial finalizer is
to back up the exception at the beginning by calling
:c:func:`PyErr_GetRaisedException` and restore the exception at the end by
calling :c:func:`PyErr_SetRaisedException`. If an exception is encountered
in the middle of the finalizer, log and clear it with
:c:func:`PyErr_WriteUnraisable` or :c:func:`PyErr_FormatUnraisable`. For
example::
static void
local_finalize(PyObject *self)
foo_finalize(PyObject *self)
{
/* Save the current exception, if any. */
// Save the current exception, if any.
PyObject *exc = PyErr_GetRaisedException();
/* ... */
// ...
/* Restore the saved exception. */
if (do_something_that_might_raise() != success_indicator) {
PyErr_WriteUnraisable(self);
goto done;
}
done:
// Restore the saved exception. This silently discards any exception
// raised above, so be sure to call PyErr_WriteUnraisable first if
// necessary.
PyErr_SetRaisedException(exc);
}
@ -2175,7 +2421,13 @@ and :c:data:`PyType_Type` effectively act as defaults.)
:c:macro:`Py_TPFLAGS_HAVE_FINALIZE` flags bit in order for this field to be
used. This is no longer required.
.. seealso:: "Safe object finalization" (:pep:`442`)
.. seealso::
* :pep:`442`: "Safe object finalization"
* :ref:`life-cycle` for details about how this slot relates to other
slots.
* :c:func:`PyObject_CallFinalizer`
* :c:func:`PyObject_CallFinalizerFromDealloc`
.. c:member:: vectorcallfunc PyTypeObject.tp_vectorcall

57
Doc/c-api/unicode.rst
View file

@ -33,8 +33,14 @@ Python:
.. c:var:: PyTypeObject PyUnicode_Type
This instance of :c:type:`PyTypeObject` represents the Python Unicode type. It
is exposed to Python code as :py:class:`str`.
This instance of :c:type:`PyTypeObject` represents the Python Unicode type.
It is exposed to Python code as :py:class:`str`.
.. c:var:: PyTypeObject PyUnicodeIter_Type
This instance of :c:type:`PyTypeObject` represents the Python Unicode
iterator type. It is used to iterate over Unicode string objects.
.. c:type:: Py_UCS4
@ -596,6 +602,14 @@ APIs:
Objects other than Unicode or its subtypes will cause a :exc:`TypeError`.
.. c:function:: PyObject* PyUnicode_FromOrdinal(int ordinal)
Create a Unicode Object from the given Unicode code point *ordinal*.
The ordinal must be in ``range(0x110000)``. A :exc:`ValueError` is
raised in the case it is not.
.. c:function:: PyObject* PyUnicode_FromEncodedObject(PyObject *obj, \
const char *encoding, const char *errors)
@ -622,7 +636,7 @@ APIs:
On error, set *\*p_left* to ``NULL`` and set an exception.
On sucess, set *\*p_left* to a new strong reference to the result.
On success, set *\*p_left* to a new strong reference to the result.
.. c:function:: void PyUnicode_AppendAndDel(PyObject **p_left, PyObject *right)
@ -631,6 +645,17 @@ APIs:
difference being that it decrements the reference count of *right* by one.
.. c:function:: PyObject* PyUnicode_BuildEncodingMap(PyObject* string)
Return a mapping suitable for decoding a custom single-byte encoding.
Given a Unicode string *string* of up to 256 characters representing an encoding
table, returns either a compact internal mapping object or a dictionary
mapping character ordinals to byte values. Raises a :exc:`TypeError` and
return ``NULL`` on invalid input.
.. versionadded:: 3.2
.. c:function:: const char* PyUnicode_GetDefaultEncoding(void)
Return the name of the default string encoding, ``"utf-8"``.
@ -666,6 +691,21 @@ APIs:
.. versionadded:: 3.3
.. c:function:: int PyUnicode_Resize(PyObject **unicode, Py_ssize_t length);
Resize a Unicode object *\*unicode* to the new *length* in code points.
Try to resize the string in place (which is usually faster than allocating
a new string and copying characters), or create a new string.
*\*unicode* is modified to point to the new (resized) object and ``0`` is
returned on success. Otherwise, ``-1`` is returned and an exception is set,
and *\*unicode* is left untouched.
The function doesn't check string content, the result may not be a
string in canonical representation.
.. c:function:: Py_ssize_t PyUnicode_Fill(PyObject *unicode, Py_ssize_t start, \
Py_ssize_t length, Py_UCS4 fill_char)
@ -1003,6 +1043,17 @@ generic ones are documented for simplicity.
Generic Codecs
""""""""""""""
The following macro is provided:
.. c:macro:: Py_UNICODE_REPLACEMENT_CHARACTER
The Unicode code point ``U+FFFD`` (replacement character).
This Unicode character is used as the replacement character during
decoding if the *errors* argument is set to "replace".
These are the generic codec APIs:

1
Doc/conf.py
View file

@ -308,7 +308,6 @@ nitpick_ignore += [
('py:attr', '__annotations__'),
('py:meth', '__missing__'),
('py:attr', '__wrapped__'),
('py:attr', 'decimal.Context.clamp'),
('py:meth', 'index'), # list.index, tuple.index, etc.
]

23
Doc/data/refcounts.dat
View file

@ -1093,9 +1093,6 @@ PyImport_ImportModuleLevelObject:PyObject*:locals:0:???
PyImport_ImportModuleLevelObject:PyObject*:fromlist:0:???
PyImport_ImportModuleLevelObject:int:level::
PyImport_ImportModuleNoBlock:PyObject*::+1:
PyImport_ImportModuleNoBlock:const char*:name::
PyImport_ReloadModule:PyObject*::+1:
PyImport_ReloadModule:PyObject*:m:0:
@ -2770,6 +2767,9 @@ PyUnicode_FromFormatV:PyObject*::+1:
PyUnicode_FromFormatV:const char*:format::
PyUnicode_FromFormatV:va_list:args::
PyUnicode_FromOrdinal:PyObject*::+1:
PyUnicode_FromOrdinal:int:ordinal::
PyUnicode_Append:void:::
PyUnicode_Append:PyObject**:p_left:0:
PyUnicode_Append:PyObject*:right::
@ -2778,6 +2778,9 @@ PyUnicode_AppendAndDel:void:::
PyUnicode_AppendAndDel:PyObject**:p_left:0:
PyUnicode_AppendAndDel:PyObject*:right:-1:
PyUnicode_BuildEncodingMap:PyObject*::+1:
PyUnicode_BuildEncodingMap:PyObject*:string:::
PyUnicode_GetDefaultEncoding:const char*:::
PyUnicode_GetDefaultEncoding::void::
@ -2791,6 +2794,10 @@ PyUnicode_CopyCharacters:PyObject*:from:0:
PyUnicode_CopyCharacters:Py_ssize_t:from_start::
PyUnicode_CopyCharacters:Py_ssize_t:how_many::
PyUnicode_Resize:int:::
PyUnicode_Resize:PyObject**:unicode:0:
PyUnicode_Resize:Py_ssize_t:length::
PyUnicode_Fill:Py_ssize_t:::
PyUnicode_Fill:PyObject*:unicode:0:
PyUnicode_Fill:Py_ssize_t:start::
@ -3000,18 +3007,8 @@ Py_GetCompiler:const char*:::
Py_GetCopyright:const char*:::
Py_GetExecPrefix:wchar_t*:::
Py_GetPath:wchar_t*:::
Py_GetPlatform:const char*:::
Py_GetPrefix:wchar_t*:::
Py_GetProgramFullPath:wchar_t*:::
Py_GetProgramName:wchar_t*:::
Py_GetVersion:const char*:::
Py_INCREF:void:::

14
Doc/data/stable_abi.dat generated
View file

@ -323,7 +323,6 @@ func,PyImport_ImportFrozenModuleObject,3.7,,
func,PyImport_ImportModule,3.2,,
func,PyImport_ImportModuleLevel,3.2,,
func,PyImport_ImportModuleLevelObject,3.7,,
func,PyImport_ImportModuleNoBlock,3.2,,
func,PyImport_ReloadModule,3.2,,
func,PyIndex_Check,3.8,,
type,PyInterpreterState,3.2,,opaque
@ -362,6 +361,7 @@ func,PyLong_AsLong,3.2,,
func,PyLong_AsLongAndOverflow,3.2,,
func,PyLong_AsLongLong,3.2,,
func,PyLong_AsLongLongAndOverflow,3.2,,
func,PyLong_AsNativeBytes,3.14,,
func,PyLong_AsSize_t,3.2,,
func,PyLong_AsSsize_t,3.2,,
func,PyLong_AsUInt32,3.14,,
@ -376,6 +376,7 @@ func,PyLong_FromInt32,3.14,,
func,PyLong_FromInt64,3.14,,
func,PyLong_FromLong,3.2,,
func,PyLong_FromLongLong,3.2,,
func,PyLong_FromNativeBytes,3.14,,
func,PyLong_FromSize_t,3.2,,
func,PyLong_FromSsize_t,3.2,,
func,PyLong_FromString,3.2,,
@ -383,6 +384,7 @@ func,PyLong_FromUInt32,3.14,,
func,PyLong_FromUInt64,3.14,,
func,PyLong_FromUnsignedLong,3.2,,
func,PyLong_FromUnsignedLongLong,3.2,,
func,PyLong_FromUnsignedNativeBytes,3.14,,
func,PyLong_FromVoidPtr,3.2,,
func,PyLong_GetInfo,3.2,,
data,PyLong_Type,3.2,,
@ -737,11 +739,7 @@ func,PyUnicode_Append,3.2,,
func,PyUnicode_AppendAndDel,3.2,,
func,PyUnicode_AsASCIIString,3.2,,
func,PyUnicode_AsCharmapString,3.2,,
func,PyUnicode_AsDecodedObject,3.2,,
func,PyUnicode_AsDecodedUnicode,3.2,,
func,PyUnicode_AsEncodedObject,3.2,,
func,PyUnicode_AsEncodedString,3.2,,
func,PyUnicode_AsEncodedUnicode,3.2,,
func,PyUnicode_AsLatin1String,3.2,,
func,PyUnicode_AsMBCSString,3.7,on Windows,
func,PyUnicode_AsRawUnicodeEscapeString,3.2,,
@ -859,13 +857,7 @@ func,Py_GetCompiler,3.2,,
func,Py_GetConstant,3.13,,
func,Py_GetConstantBorrowed,3.13,,
func,Py_GetCopyright,3.2,,
func,Py_GetExecPrefix,3.2,,
func,Py_GetPath,3.2,,
func,Py_GetPlatform,3.2,,
func,Py_GetPrefix,3.2,,
func,Py_GetProgramFullPath,3.2,,
func,Py_GetProgramName,3.2,,
func,Py_GetPythonHome,3.2,,
func,Py_GetRecursionLimit,3.2,,
func,Py_GetVersion,3.2,,
data,Py_HasFileSystemDefaultEncoding,3.2,,

24
Doc/deprecations/c-api-pending-removal-in-3.15.rst
View file

@ -2,7 +2,7 @@ Pending removal in Python 3.15
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* The bundled copy of ``libmpdecimal``.
* The :c:func:`PyImport_ImportModuleNoBlock`:
* The :c:func:`!PyImport_ImportModuleNoBlock`:
Use :c:func:`PyImport_ImportModule` instead.
* :c:func:`PyWeakref_GetObject` and :c:func:`PyWeakref_GET_OBJECT`:
Use :c:func:`PyWeakref_GetRef` instead. The `pythoncapi-compat project
@ -10,29 +10,39 @@ Pending removal in Python 3.15
:c:func:`PyWeakref_GetRef` on Python 3.12 and older.
* :c:type:`Py_UNICODE` type and the :c:macro:`!Py_UNICODE_WIDE` macro:
Use :c:type:`wchar_t` instead.
* :c:func:`!PyUnicode_AsDecodedObject`:
Use :c:func:`PyCodec_Decode` instead.
* :c:func:`!PyUnicode_AsDecodedUnicode`:
Use :c:func:`PyCodec_Decode` instead; Note that some codecs (for example, "base64")
may return a type other than :class:`str`, such as :class:`bytes`.
* :c:func:`!PyUnicode_AsEncodedObject`:
Use :c:func:`PyCodec_Encode` instead.
* :c:func:`!PyUnicode_AsEncodedUnicode`:
Use :c:func:`PyCodec_Encode` instead; Note that some codecs (for example, "base64")
may return a type other than :class:`bytes`, such as :class:`str`.
* Python initialization functions, deprecated in Python 3.13:
* :c:func:`Py_GetPath`:
* :c:func:`!Py_GetPath`:
Use :c:func:`PyConfig_Get("module_search_paths") <PyConfig_Get>`
(:data:`sys.path`) instead.
* :c:func:`Py_GetPrefix`:
* :c:func:`!Py_GetPrefix`:
Use :c:func:`PyConfig_Get("base_prefix") <PyConfig_Get>`
(:data:`sys.base_prefix`) instead. Use :c:func:`PyConfig_Get("prefix")
<PyConfig_Get>` (:data:`sys.prefix`) if :ref:`virtual environments
<venv-def>` need to be handled.
* :c:func:`Py_GetExecPrefix`:
* :c:func:`!Py_GetExecPrefix`:
Use :c:func:`PyConfig_Get("base_exec_prefix") <PyConfig_Get>`
(:data:`sys.base_exec_prefix`) instead. Use
:c:func:`PyConfig_Get("exec_prefix") <PyConfig_Get>`
(:data:`sys.exec_prefix`) if :ref:`virtual environments <venv-def>` need to
be handled.
* :c:func:`Py_GetProgramFullPath`:
* :c:func:`!Py_GetProgramFullPath`:
Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>`
(:data:`sys.executable`) instead.
* :c:func:`Py_GetProgramName`:
* :c:func:`!Py_GetProgramName`:
Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>`
(:data:`sys.executable`) instead.
* :c:func:`Py_GetPythonHome`:
* :c:func:`!Py_GetPythonHome`:
Use :c:func:`PyConfig_Get("home") <PyConfig_Get>` or the
:envvar:`PYTHONHOME` environment variable instead.

8
Doc/deprecations/c-api-pending-removal-in-future.rst
View file

@ -18,14 +18,6 @@ although there is currently no date scheduled for their removal.
Use :c:func:`PyOS_AfterFork_Child` instead.
* :c:func:`PySlice_GetIndicesEx`:
Use :c:func:`PySlice_Unpack` and :c:func:`PySlice_AdjustIndices` instead.
* :c:func:`!PyUnicode_AsDecodedObject`:
Use :c:func:`PyCodec_Decode` instead.
* :c:func:`!PyUnicode_AsDecodedUnicode`:
Use :c:func:`PyCodec_Decode` instead.
* :c:func:`!PyUnicode_AsEncodedObject`:
Use :c:func:`PyCodec_Encode` instead.
* :c:func:`!PyUnicode_AsEncodedUnicode`:
Use :c:func:`PyCodec_Encode` instead.
* :c:func:`PyUnicode_READY`:
Unneeded since Python 3.12
* :c:func:`!PyErr_Display`:

6
Doc/deprecations/index.rst
View file

@ -5,6 +5,10 @@ Deprecations
.. include:: pending-removal-in-3.16.rst
.. include:: pending-removal-in-3.17.rst
.. include:: pending-removal-in-3.19.rst
.. include:: pending-removal-in-future.rst
C API deprecations
@ -12,4 +16,6 @@ C API deprecations
.. include:: c-api-pending-removal-in-3.15.rst
.. include:: c-api-pending-removal-in-3.18.rst
.. include:: c-api-pending-removal-in-future.rst

2
Doc/deprecations/pending-removal-in-3.14.rst
View file

@ -78,7 +78,7 @@ Pending removal in Python 3.14
:meth:`~pathlib.PurePath.relative_to`: passing additional arguments is
deprecated.
* :mod:`pkgutil`: :func:`!pkgutil.find_loader` and :func:!pkgutil.get_loader`
* :mod:`pkgutil`: :func:`!pkgutil.find_loader` and :func:`!pkgutil.get_loader`
now raise :exc:`DeprecationWarning`;
use :func:`importlib.util.find_spec` instead.
(Contributed by Nikita Sobolev in :gh:`97850`.)

14
Doc/deprecations/pending-removal-in-3.15.rst
View file

@ -20,7 +20,7 @@ Pending removal in Python 3.15
* :mod:`http.server`:
* The obsolete and rarely used :class:`~http.server.CGIHTTPRequestHandler`
* The obsolete and rarely used :class:`!CGIHTTPRequestHandler`
has been deprecated since Python 3.13.
No direct replacement exists.
*Anything* is better than CGI to interface
@ -51,7 +51,7 @@ Pending removal in Python 3.15
* :mod:`platform`:
* :func:`~platform.java_ver` has been deprecated since Python 3.13.
* :func:`!platform.java_ver` has been deprecated since Python 3.13.
This function is only useful for Jython support, has a confusing API,
and is largely untested.
@ -85,6 +85,13 @@ Pending removal in Python 3.15
has been deprecated since Python 3.13.
Use the class-based syntax or the functional syntax instead.
* When using the functional syntax of :class:`~typing.TypedDict`\s, failing
to pass a value to the *fields* parameter (``TD = TypedDict("TD")``) or
passing ``None`` (``TD = TypedDict("TD", None)``) has been deprecated
since Python 3.13.
Use ``class TD(TypedDict): pass`` or ``TD = TypedDict("TD", {})``
to create a TypedDict with zero field.
* The :func:`typing.no_type_check_decorator` decorator function
has been deprecated since Python 3.13.
After eight years in the :mod:`typing` module,
@ -92,8 +99,7 @@ Pending removal in Python 3.15
* :mod:`wave`:
* The :meth:`~wave.Wave_read.getmark`, :meth:`!setmark`,
and :meth:`~wave.Wave_read.getmarkers` methods of
* The ``getmark()``, ``setmark()`` and ``getmarkers()`` methods of
the :class:`~wave.Wave_read` and :class:`~wave.Wave_write` classes
have been deprecated since Python 3.13.

15
Doc/deprecations/pending-removal-in-3.16.rst
View file

@ -32,7 +32,6 @@ Pending removal in Python 3.16
* :class:`asyncio.WindowsProactorEventLoopPolicy`
* :func:`asyncio.get_event_loop_policy`
* :func:`asyncio.set_event_loop_policy`
* :func:`asyncio.set_event_loop`
Users should use :func:`asyncio.run` or :class:`asyncio.Runner` with
*loop_factory* to use the desired event loop implementation.
@ -62,6 +61,20 @@ Pending removal in Python 3.16
* Calling the Python implementation of :func:`functools.reduce` with *function*
or *sequence* as keyword arguments has been deprecated since Python 3.14.
* :mod:`logging`:
Support for custom logging handlers with the *strm* argument is deprecated
and scheduled for removal in Python 3.16. Define handlers with the *stream*
argument instead. (Contributed by Mariusz Felisiak in :gh:`115032`.)
* :mod:`mimetypes`:
* Valid extensions start with a '.' or are empty for
:meth:`mimetypes.MimeTypes.add_type`.
Undotted extensions are deprecated and will
raise a :exc:`ValueError` in Python 3.16.
(Contributed by Hugo van Kemenade in :gh:`75223`.)
* :mod:`shutil`:
* The :class:`!ExecError` exception

10
Doc/deprecations/pending-removal-in-3.17.rst Normal file
View file

@ -0,0 +1,10 @@
Pending removal in Python 3.17
------------------------------
* :mod:`typing`:
- Before Python 3.14, old-style unions were implemented using the private class
``typing._UnionGenericAlias``. This class is no longer needed for the implementation,
but it has been retained for backward compatibility, with removal scheduled for Python
3.17. Users should use documented introspection helpers like :func:`typing.get_origin`
and :func:`typing.get_args` instead of relying on private implementation details.

8
Doc/deprecations/pending-removal-in-3.19.rst Normal file
View file

@ -0,0 +1,8 @@
Pending removal in Python 3.19
------------------------------
* :mod:`ctypes`:
* Implicitly switching to the MSVC-compatible struct layout by setting
:attr:`~ctypes.Structure._pack_` but not :attr:`~ctypes.Structure._layout_`
on non-Windows platforms.

7
Doc/deprecations/pending-removal-in-future.rst
View file

@ -13,8 +13,6 @@ although there is currently no date scheduled for their removal.
deprecated.
* The :class:`argparse.FileType` type converter is deprecated.
* :mod:`array`'s ``'u'`` format code (:gh:`57281`)
* :mod:`builtins`:
* ``bool(NotImplemented)``.
@ -49,6 +47,8 @@ although there is currently no date scheduled for their removal.
:data:`calendar.FEBRUARY`.
(Contributed by Prince Roshan in :gh:`103636`.)
* :mod:`codecs`: use :func:`open` instead of :func:`codecs.open`. (:gh:`133038`)
* :attr:`codeobject.co_lnotab`: use the :meth:`codeobject.co_lines` method
instead.
@ -153,3 +153,6 @@ although there is currently no date scheduled for their removal.
:class:`~xml.etree.ElementTree.Element` is deprecated. In a future release it
will always return ``True``. Prefer explicit ``len(elem)`` or
``elem is not None`` tests instead.
* :func:`sys._clear_type_cache` is deprecated:
use :func:`sys._clear_internal_caches` instead.

76
Doc/extending/newtypes.rst
View file

@ -70,22 +70,24 @@ object itself needs to be freed here as well. Here is an example of this
function::
static void
newdatatype_dealloc(newdatatypeobject *obj)
newdatatype_dealloc(PyObject *op)
{
free(obj->obj_UnderlyingDatatypePtr);
Py_TYPE(obj)->tp_free((PyObject *)obj);
newdatatypeobject *self = (newdatatypeobject *) op;
free(self->obj_UnderlyingDatatypePtr);
Py_TYPE(self)->tp_free(self);
}
If your type supports garbage collection, the destructor should call
:c:func:`PyObject_GC_UnTrack` before clearing any member fields::
static void
newdatatype_dealloc(newdatatypeobject *obj)
newdatatype_dealloc(PyObject *op)
{
PyObject_GC_UnTrack(obj);
Py_CLEAR(obj->other_obj);
newdatatypeobject *self = (newdatatypeobject *) op;
PyObject_GC_UnTrack(op);
Py_CLEAR(self->other_obj);
...
Py_TYPE(obj)->tp_free((PyObject *)obj);
Py_TYPE(self)->tp_free(self);
}
.. index::
@ -117,17 +119,19 @@ done. This can be done using the :c:func:`PyErr_Fetch` and
PyErr_Fetch(&err_type, &err_value, &err_traceback);
cbresult = PyObject_CallNoArgs(self->my_callback);
if (cbresult == NULL)
PyErr_WriteUnraisable(self->my_callback);
else
if (cbresult == NULL) {
PyErr_WriteUnraisable(self->my_callback);
}
else {
Py_DECREF(cbresult);
}
/* This restores the saved exception state */
PyErr_Restore(err_type, err_value, err_traceback);
Py_DECREF(self->my_callback);
}
Py_TYPE(obj)->tp_free((PyObject*)self);
Py_TYPE(self)->tp_free(self);
}
.. note::
@ -168,10 +172,11 @@ representation of the instance for which it is called. Here is a simple
example::
static PyObject *
newdatatype_repr(newdatatypeobject *obj)
newdatatype_repr(PyObject *op)
{
newdatatypeobject *self = (newdatatypeobject *) op;
return PyUnicode_FromFormat("Repr-ified_newdatatype{{size:%d}}",
obj->obj_UnderlyingDatatypePtr->size);
self->obj_UnderlyingDatatypePtr->size);
}
If no :c:member:`~PyTypeObject.tp_repr` handler is specified, the interpreter will supply a
@ -188,10 +193,11 @@ used instead.
Here is a simple example::
static PyObject *
newdatatype_str(newdatatypeobject *obj)
newdatatype_str(PyObject *op)
{
newdatatypeobject *self = (newdatatypeobject *) op;
return PyUnicode_FromFormat("Stringified_newdatatype{{size:%d}}",
obj->obj_UnderlyingDatatypePtr->size);
self->obj_UnderlyingDatatypePtr->size);
}
@ -329,16 +335,16 @@ method of a class would be called.
Here is an example::
static PyObject *
newdatatype_getattr(newdatatypeobject *obj, char *name)
newdatatype_getattr(PyObject *op, char *name)
{
if (strcmp(name, "data") == 0)
{
return PyLong_FromLong(obj->data);
newdatatypeobject *self = (newdatatypeobject *) op;
if (strcmp(name, "data") == 0) {
return PyLong_FromLong(self->data);
}
PyErr_Format(PyExc_AttributeError,
"'%.100s' object has no attribute '%.400s'",
Py_TYPE(obj)->tp_name, name);
Py_TYPE(self)->tp_name, name);
return NULL;
}
@ -349,7 +355,7 @@ example that simply raises an exception; if this were really all you wanted, the
:c:member:`~PyTypeObject.tp_setattr` handler should be set to ``NULL``. ::
static int
newdatatype_setattr(newdatatypeobject *obj, char *name, PyObject *v)
newdatatype_setattr(PyObject *op, char *name, PyObject *v)
{
PyErr_Format(PyExc_RuntimeError, "Read-only attribute: %s", name);
return -1;
@ -379,8 +385,10 @@ Here is a sample implementation, for a datatype that is considered equal if the
size of an internal pointer is equal::
static PyObject *
newdatatype_richcmp(newdatatypeobject *obj1, newdatatypeobject *obj2, int op)
newdatatype_richcmp(PyObject *lhs, PyObject *rhs, int op)
{
newdatatypeobject *obj1 = (newdatatypeobject *) lhs;
newdatatypeobject *obj2 = (newdatatypeobject *) rhs;
PyObject *result;
int c, size1, size2;
@ -399,8 +407,7 @@ size of an internal pointer is equal::
case Py_GE: c = size1 >= size2; break;
}
result = c ? Py_True : Py_False;
Py_INCREF(result);
return result;
return Py_NewRef(result);
}
@ -439,12 +446,14 @@ This function, if you choose to provide it, should return a hash number for an
instance of your data type. Here is a simple example::
static Py_hash_t
newdatatype_hash(newdatatypeobject *obj)
newdatatype_hash(PyObject *op)
{
newdatatypeobject *self = (newdatatypeobject *) op;
Py_hash_t result;
result = obj->some_size + 32767 * obj->some_number;
if (result == -1)
result = -2;
result = self->some_size + 32767 * self->some_number;
if (result == -1) {
result = -2;
}
return result;
}
@ -478,8 +487,9 @@ This function takes three arguments:
Here is a toy ``tp_call`` implementation::
static PyObject *
newdatatype_call(newdatatypeobject *obj, PyObject *args, PyObject *kwds)
newdatatype_call(PyObject *op, PyObject *args, PyObject *kwds)
{
newdatatypeobject *self = (newdatatypeobject *) op;
PyObject *result;
const char *arg1;
const char *arg2;
@ -490,7 +500,7 @@ Here is a toy ``tp_call`` implementation::
}
result = PyUnicode_FromFormat(
"Returning -- value: [%d] arg1: [%s] arg2: [%s] arg3: [%s]\n",
obj->obj_UnderlyingDatatypePtr->size,
self->obj_UnderlyingDatatypePtr->size,
arg1, arg2, arg3);
return result;
}
@ -563,12 +573,12 @@ The only further addition is that ``tp_dealloc`` needs to clear any weak
references (by calling :c:func:`PyObject_ClearWeakRefs`)::
static void
Trivial_dealloc(TrivialObject *self)
Trivial_dealloc(PyObject *op)
{
/* Clear weakrefs first before calling any destructors */
PyObject_ClearWeakRefs((PyObject *) self);
PyObject_ClearWeakRefs(op);
/* ... remainder of destruction code omitted for brevity ... */
Py_TYPE(self)->tp_free((PyObject *) self);
Py_TYPE(op)->tp_free(op);
}

10
Doc/faq/design.rst
View file

@ -420,10 +420,12 @@ strings representing the files in the current directory. Functions which
operate on this output would generally not break if you added another file or
two to the directory.
Tuples are immutable, meaning that once a tuple has been created, you can't
replace any of its elements with a new value. Lists are mutable, meaning that
you can always change a list's elements. Only immutable elements can be used as
dictionary keys, and hence only tuples and not lists can be used as keys.
Tuples are :term:`immutable`, meaning that once a tuple has been created, you can't
replace any of its elements with a new value. Lists are :term:`mutable`, meaning that
you can always change a list's elements. Only :term:`hashable` objects can
be used as dictionary keys. Most immutable types are hashable, which is why
tuples, but not lists, can be used as keys. Note, however, that a tuple is
only hashable if all of its elements are hashable.
How are lists implemented in CPython?

6
Doc/glossary.rst
View file

@ -355,6 +355,12 @@ Glossary
tasks (see :mod:`asyncio`) associate each task with a context which
becomes the current context whenever the task starts or resumes execution.
cyclic isolate
A subgroup of one or more objects that reference each other in a reference
cycle, but are not referenced by objects outside the group. The goal of
the :term:`cyclic garbage collector <garbage collection>` is to identify these groups and break the reference
cycles so that the memory can be reclaimed.
decorator
A function returning another function, usually applied as a function
transformation using the ``@wrapper`` syntax. Common examples for

7
Doc/howto/annotations.rst
View file

@ -248,4 +248,9 @@ quirks by using :func:`annotationlib.get_annotations` on Python 3.14+ or
:func:`inspect.get_annotations` on Python 3.10+. On earlier versions of
Python, you can avoid these bugs by accessing the annotations from the
class's :attr:`~type.__dict__`
(e.g., ``cls.__dict__.get('__annotations__', None)``).
(for example, ``cls.__dict__.get('__annotations__', None)``).
In some versions of Python, instances of classes may have an ``__annotations__``
attribute. However, this is not supported functionality. If you need the
annotations of an instance, you can use :func:`type` to access its class
(for example, ``annotationlib.get_annotations(type(myinstance))`` on Python 3.14+).

145
Doc/howto/free-threading-extensions.rst
View file

@ -23,6 +23,14 @@ You can use it to enable code that only runs under the free-threaded build::
/* code that only runs in the free-threaded build */
#endif
.. note::
On Windows, this macro is not defined automatically, but must be specified
to the compiler when building. The :func:`sysconfig.get_config_var` function
can be used to determine whether the current running interpreter had the
macro defined.
Module Initialization
=====================
@ -243,6 +251,141 @@ depend on your extension, but some common patterns include:
`thread-local storage <https://en.cppreference.com/w/c/language/storage_duration>`_.
Critical Sections
=================
.. _critical-sections:
In the free-threaded build, CPython provides a mechanism called "critical
sections" to protect data that would otherwise be protected by the GIL.
While extension authors may not interact with the internal critical section
implementation directly, understanding their behavior is crucial when using
certain C API functions or managing shared state in the free-threaded build.
What Are Critical Sections?
...........................
Conceptually, critical sections act as a deadlock avoidance layer built on
top of simple mutexes. Each thread maintains a stack of active critical
sections. When a thread needs to acquire a lock associated with a critical
section (e.g., implicitly when calling a thread-safe C API function like
:c:func:`PyDict_SetItem`, or explicitly using macros), it attempts to acquire
the underlying mutex.
Using Critical Sections
.......................
The primary APIs for using critical sections are:
* :c:macro:`Py_BEGIN_CRITICAL_SECTION` and :c:macro:`Py_END_CRITICAL_SECTION` -
For locking a single object
* :c:macro:`Py_BEGIN_CRITICAL_SECTION2` and :c:macro:`Py_END_CRITICAL_SECTION2`
- For locking two objects simultaneously
These macros must be used in matching pairs and must appear in the same C
scope, since they establish a new local scope. These macros are no-ops in
non-free-threaded builds, so they can be safely added to code that needs to
support both build types.
A common use of a critical section would be to lock an object while accessing
an internal attribute of it. For example, if an extension type has an internal
count field, you could use a critical section while reading or writing that
field::
// read the count, returns new reference to internal count value
PyObject *result;
Py_BEGIN_CRITICAL_SECTION(obj);
result = Py_NewRef(obj->count);
Py_END_CRITICAL_SECTION();
return result;
// write the count, consumes reference from new_count
Py_BEGIN_CRITICAL_SECTION(obj);
obj->count = new_count;
Py_END_CRITICAL_SECTION();
How Critical Sections Work
..........................
Unlike traditional locks, critical sections do not guarantee exclusive access
throughout their entire duration. If a thread would block while holding a
critical section (e.g., by acquiring another lock or performing I/O), the
critical section is temporarily suspended—all locks are released—and then
resumed when the blocking operation completes.
This behavior is similar to what happens with the GIL when a thread makes a
blocking call. The key differences are:
* Critical sections operate on a per-object basis rather than globally
* Critical sections follow a stack discipline within each thread (the "begin" and
"end" macros enforce this since they must be paired and within the same scope)
* Critical sections automatically release and reacquire locks around potential
blocking operations
Deadlock Avoidance
..................
Critical sections help avoid deadlocks in two ways:
1. If a thread tries to acquire a lock that's already held by another thread,
it first suspends all of its active critical sections, temporarily releasing
their locks
2. When the blocking operation completes, only the top-most critical section is
reacquired first
This means you cannot rely on nested critical sections to lock multiple objects
at once, as the inner critical section may suspend the outer ones. Instead, use
:c:macro:`Py_BEGIN_CRITICAL_SECTION2` to lock two objects simultaneously.
Note that the locks described above are only :c:type:`!PyMutex` based locks.
The critical section implementation does not know about or affect other locking
mechanisms that might be in use, like POSIX mutexes. Also note that while
blocking on any :c:type:`!PyMutex` causes the critical sections to be
suspended, only the mutexes that are part of the critical sections are
released. If :c:type:`!PyMutex` is used without a critical section, it will
not be released and therefore does not get the same deadlock avoidance.
Important Considerations
........................
* Critical sections may temporarily release their locks, allowing other threads
to modify the protected data. Be careful about making assumptions about the
state of the data after operations that might block.
* Because locks can be temporarily released (suspended), entering a critical
section does not guarantee exclusive access to the protected resource
throughout the section's duration. If code within a critical section calls
another function that blocks (e.g., acquires another lock, performs blocking
I/O), all locks held by the thread via critical sections will be released.
This is similar to how the GIL can be released during blocking calls.
* Only the lock(s) associated with the most recently entered (top-most)
critical section are guaranteed to be held at any given time. Locks for
outer, nested critical sections might have been suspended.
* You can lock at most two objects simultaneously with these APIs. If you need
to lock more objects, you'll need to restructure your code.
* While critical sections will not deadlock if you attempt to lock the same
object twice, they are less efficient than purpose-built reentrant locks for
this use case.
* When using :c:macro:`Py_BEGIN_CRITICAL_SECTION2`, the order of the objects
doesn't affect correctness (the implementation handles deadlock avoidance),
but it's good practice to always lock objects in a consistent order.
* Remember that the critical section macros are primarily for protecting access
to *Python objects* that might be involved in internal CPython operations
susceptible to the deadlock scenarios described above. For protecting purely
internal extension state, standard mutexes or other synchronization
primitives might be more appropriate.
Building Extensions for the Free-Threaded Build
===============================================
@ -253,7 +396,7 @@ The wheels, shared libraries, and binaries are indicated by a ``t`` suffix.
free-threaded build, with the ``t`` suffix, such as ``python3.13t``.
* `pypa/cibuildwheel <https://github.com/pypa/cibuildwheel>`_ supports the
free-threaded build if you set
`CIBW_FREE_THREADED_SUPPORT <https://cibuildwheel.pypa.io/en/stable/options/#free-threaded-support>`_.
`CIBW_ENABLE to cpython-freethreading <https://cibuildwheel.pypa.io/en/stable/options/#enable>`_.
Limited C API and Stable ABI
............................

2
Doc/howto/free-threading-python.rst
View file

@ -32,7 +32,7 @@ optionally support installing free-threaded Python binaries. The installers
are available at https://www.python.org/downloads/.
For information on other platforms, see the `Installing a Free-Threaded Python
<https://py-free-threading.github.io/installing_cpython/>`_, a
<https://py-free-threading.github.io/installing-cpython/>`_, a
community-maintained installation guide for installing free-threaded Python.
When building CPython from source, the :option:`--disable-gil` configure option

2
Doc/howto/functional.rst
View file

@ -372,8 +372,8 @@ have the form::
for expr2 in sequence2
if condition2
for expr3 in sequence3
...
if condition3
...
for exprN in sequenceN
if conditionN )

2
Doc/howto/index.rst
View file

@ -34,6 +34,7 @@ Python Library Reference.
mro.rst
free-threading-python.rst
free-threading-extensions.rst
remote_debugging.rst
General:
@ -66,3 +67,4 @@ Debugging and profiling:
* :ref:`gdb`
* :ref:`instrumentation`
* :ref:`perf_profiling`
* :ref:`remote-debugging`

15
Doc/howto/isolating-extensions.rst
View file

@ -215,21 +215,36 @@ multiple interpreters correctly. If this is not yet the case for your
module, you can explicitly make your module loadable only once per
process. For example::
// A process-wide flag
static int loaded = 0;
// Mutex to provide thread safety (only needed for free-threaded Python)
static PyMutex modinit_mutex = {0};
static int
exec_module(PyObject* module)
{
PyMutex_Lock(&modinit_mutex);
if (loaded) {
PyMutex_Unlock(&modinit_mutex);
PyErr_SetString(PyExc_ImportError,
"cannot load module more than once per process");
return -1;
}
loaded = 1;
PyMutex_Unlock(&modinit_mutex);
// ... rest of initialization
}
If your module's :c:member:`PyModuleDef.m_clear` function is able to prepare
for future re-initialization, it should clear the ``loaded`` flag.
In this case, your module won't support multiple instances existing
*concurrently*, but it will, for example, support being loaded after
Python runtime shutdown (:c:func:`Py_FinalizeEx`) and re-initialization
(:c:func:`Py_Initialize`).
Module State Access from Functions
----------------------------------

13
Doc/howto/logging-cookbook.rst
View file

@ -626,6 +626,19 @@ which, when run, will produce:
of each message with the handler's level, and only passes a message to a
handler if it's appropriate to do so.
.. versionchanged:: 3.14
The :class:`QueueListener` can be started (and stopped) via the
:keyword:`with` statement. For example:
.. code-block:: python
with QueueListener(que, handler) as listener:
# The queue listener automatically starts
# when the 'with' block is entered.
pass
# The queue listener automatically stops once
# the 'with' block is exited.
.. _network-logging:
Sending and receiving logging events across a network

31
Doc/howto/perf_profiling.rst
View file

@ -254,13 +254,28 @@ files in the current directory which are ELF images for all the JIT trampolines
that were created by Python.
.. warning::
Notice that when using ``--call-graph dwarf`` the ``perf`` tool will take
When using ``--call-graph dwarf``, the ``perf`` tool will take
snapshots of the stack of the process being profiled and save the
information in the ``perf.data`` file. By default the size of the stack dump
is 8192 bytes but the user can change the size by passing the size after
comma like ``--call-graph dwarf,4096``. The size of the stack dump is
important because if the size is too small ``perf`` will not be able to
unwind the stack and the output will be incomplete. On the other hand, if
the size is too big, then ``perf`` won't be able to sample the process as
frequently as it would like as the overhead will be higher.
information in the ``perf.data`` file. By default, the size of the stack dump
is 8192 bytes, but you can change the size by passing it after
a comma like ``--call-graph dwarf,16384``.
The size of the stack dump is important because if the size is too small
``perf`` will not be able to unwind the stack and the output will be
incomplete. On the other hand, if the size is too big, then ``perf`` won't
be able to sample the process as frequently as it would like as the overhead
will be higher.
The stack size is particularly important when profiling Python code compiled
with low optimization levels (like ``-O0``), as these builds tend to have
larger stack frames. If you are compiling Python with ``-O0`` and not seeing
Python functions in your profiling output, try increasing the stack dump
size to 65528 bytes (the maximum)::
$ perf record -F 9999 -g -k 1 --call-graph dwarf,65528 -o perf.data python -Xperf_jit my_script.py
Different compilation flags can significantly impact stack sizes:
- Builds with ``-O0`` typically have much larger stack frames than those with ``-O1`` or higher
- Adding optimizations (``-O1``, ``-O2``, etc.) typically reduces stack size
- Frame pointers (``-fno-omit-frame-pointer``) generally provide more reliable stack unwinding

5
Doc/howto/regex.rst
View file

@ -738,9 +738,12 @@ given location, they can obviously be matched an infinite number of times.
different: ``\A`` still matches only at the beginning of the string, but ``^``
may match at any location inside the string that follows a newline character.
``\Z``
``\z``
Matches only at the end of the string.
``\Z``
The same as ``\z``. For compatibility with old Python versions.
``\b``
Word boundary. This is a zero-width assertion that matches only at the
beginning or end of a word. A word is defined as a sequence of alphanumeric

545
Doc/howto/remote_debugging.rst Normal file
View file

@ -0,0 +1,545 @@
.. _remote-debugging:
Remote debugging attachment protocol
====================================
This section describes the low-level protocol that enables external tools to
inject and execute a Python script within a running CPython process.
This mechanism forms the basis of the :func:`sys.remote_exec` function, which
instructs a remote Python process to execute a ``.py`` file. However, this
section does not document the usage of that function. Instead, it provides a
detailed explanation of the underlying protocol, which takes as input the
``pid`` of a target Python process and the path to a Python source file to be
executed. This information supports independent reimplementation of the
protocol, regardless of programming language.
.. warning::
The execution of the injected script depends on the interpreter reaching a
safe evaluation point. As a result, execution may be delayed depending on
the runtime state of the target process.
Once injected, the script is executed by the interpreter within the target
process the next time a safe evaluation point is reached. This approach enables
remote execution capabilities without modifying the behavior or structure of
the running Python application.
Subsequent sections provide a step-by-step description of the protocol,
including techniques for locating interpreter structures in memory, safely
accessing internal fields, and triggering code execution. Platform-specific
variations are noted where applicable, and example implementations are included
to clarify each operation.
Locating the PyRuntime structure
================================
CPython places the ``PyRuntime`` structure in a dedicated binary section to
help external tools find it at runtime. The name and format of this section
vary by platform. For example, ``.PyRuntime`` is used on ELF systems, and
``__DATA,__PyRuntime`` is used on macOS. Tools can find the offset of this
structure by examining the binary on disk.
The ``PyRuntime`` structure contains CPython’s global interpreter state and
provides access to other internal data, including the list of interpreters,
thread states, and debugger support fields.
To work with a remote Python process, a debugger must first find the memory
address of the ``PyRuntime`` structure in the target process. This address
can’t be hardcoded or calculated from a symbol name, because it depends on
where the operating system loaded the binary.
The method for finding ``PyRuntime`` depends on the platform, but the steps are
the same in general:
1. Find the base address where the Python binary or shared library was loaded
in the target process.
2. Use the on-disk binary to locate the offset of the ``.PyRuntime`` section.
3. Add the section offset to the base address to compute the address in memory.
The sections below explain how to do this on each supported platform and
include example code.
.. rubric:: Linux (ELF)
To find the ``PyRuntime`` structure on Linux:
1. Read the process’s memory map (for example, ``/proc/<pid>/maps``) to find
the address where the Python executable or ``libpython`` was loaded.
2. Parse the ELF section headers in the binary to get the offset of the
``.PyRuntime`` section.
3. Add that offset to the base address from step 1 to get the memory address of
``PyRuntime``.
The following is an example implementation::
def find_py_runtime_linux(pid: int) -> int:
# Step 1: Try to find the Python executable in memory
binary_path, base_address = find_mapped_binary(
pid, name_contains="python"
)
# Step 2: Fallback to shared library if executable is not found
if binary_path is None:
binary_path, base_address = find_mapped_binary(
pid, name_contains="libpython"
)
# Step 3: Parse ELF headers to get .PyRuntime section offset
section_offset = parse_elf_section_offset(
binary_path, ".PyRuntime"
)
# Step 4: Compute PyRuntime address in memory
return base_address + section_offset
On Linux systems, there are two main approaches to read memory from another
process. The first is through the ``/proc`` filesystem, specifically by reading from
``/proc/[pid]/mem`` which provides direct access to the process's memory. This
requires appropriate permissions - either being the same user as the target
process or having root access. The second approach is using the
``process_vm_readv()`` system call which provides a more efficient way to copy
memory between processes. While ptrace's ``PTRACE_PEEKTEXT`` operation can also be
used to read memory, it is significantly slower as it only reads one word at a
time and requires multiple context switches between the tracer and tracee
processes.
For parsing ELF sections, the process involves reading and interpreting the ELF
file format structures from the binary file on disk. The ELF header contains a
pointer to the section header table. Each section header contains metadata about
a section including its name (stored in a separate string table), offset, and
size. To find a specific section like .PyRuntime, you need to walk through these
headers and match the section name. The section header then provides the offset
where that section exists in the file, which can be used to calculate its
runtime address when the binary is loaded into memory.
You can read more about the ELF file format in the `ELF specification
<https://en.wikipedia.org/wiki/Executable_and_Linkable_Format>`_.
.. rubric:: macOS (Mach-O)
To find the ``PyRuntime`` structure on macOS:
1. Call ``task_for_pid()`` to get the ``mach_port_t`` task port for the target
process. This handle is needed to read memory using APIs like
``mach_vm_read_overwrite`` and ``mach_vm_region``.
2. Scan the memory regions to find the one containing the Python executable or
``libpython``.
3. Load the binary file from disk and parse the Mach-O headers to find the
section named ``PyRuntime`` in the ``__DATA`` segment. On macOS, symbol
names are automatically prefixed with an underscore, so the ``PyRuntime``
symbol appears as ``_PyRuntime`` in the symbol table, but the section name
is not affected.
The following is an example implementation::
def find_py_runtime_macos(pid: int) -> int:
# Step 1: Get access to the process's memory
handle = get_memory_access_handle(pid)
# Step 2: Try to find the Python executable in memory
binary_path, base_address = find_mapped_binary(
handle, name_contains="python"
)
# Step 3: Fallback to libpython if the executable is not found
if binary_path is None:
binary_path, base_address = find_mapped_binary(
handle, name_contains="libpython"
)
# Step 4: Parse Mach-O headers to get __DATA,__PyRuntime section offset
section_offset = parse_macho_section_offset(
binary_path, "__DATA", "__PyRuntime"
)
# Step 5: Compute the PyRuntime address in memory
return base_address + section_offset
On macOS, accessing another process's memory requires using Mach-O specific APIs
and file formats. The first step is obtaining a ``task_port`` handle via
``task_for_pid()``, which provides access to the target process's memory space.
This handle enables memory operations through APIs like
``mach_vm_read_overwrite()``.
The process memory can be examined using ``mach_vm_region()`` to scan through the
virtual memory space, while ``proc_regionfilename()`` helps identify which binary
files are loaded at each memory region. When the Python binary or library is
found, its Mach-O headers need to be parsed to locate the ``PyRuntime`` structure.
The Mach-O format organizes code and data into segments and sections. The
``PyRuntime`` structure lives in a section named ``__PyRuntime`` within the
``__DATA`` segment. The actual runtime address calculation involves finding the
``__TEXT`` segment which serves as the binary's base address, then locating the
``__DATA`` segment containing our target section. The final address is computed by
combining the base address with the appropriate section offsets from the Mach-O
headers.
Note that accessing another process's memory on macOS typically requires
elevated privileges - either root access or special security entitlements
granted to the debugging process.
.. rubric:: Windows (PE)
To find the ``PyRuntime`` structure on Windows:
1. Use the ToolHelp API to enumerate all modules loaded in the target process.
This is done using functions such as `CreateToolhelp32Snapshot
<https://learn.microsoft.com/en-us/windows/win32/api/tlhelp32/nf-tlhelp32-createtoolhelp32snapshot>`_,
`Module32First
<https://learn.microsoft.com/en-us/windows/win32/api/tlhelp32/nf-tlhelp32-module32first>`_,
and `Module32Next
<https://learn.microsoft.com/en-us/windows/win32/api/tlhelp32/nf-tlhelp32-module32next>`_.
2. Identify the module corresponding to :file:`python.exe` or
:file:`python{XY}.dll`, where ``X`` and ``Y`` are the major and minor
version numbers of the Python version, and record its base address.
3. Locate the ``PyRuntim`` section. Due to the PE format's 8-character limit
on section names (defined as ``IMAGE_SIZEOF_SHORT_NAME``), the original
name ``PyRuntime`` is truncated. This section contains the ``PyRuntime``
structure.
4. Retrieve the section’s relative virtual address (RVA) and add it to the base
address of the module.
The following is an example implementation::
def find_py_runtime_windows(pid: int) -> int:
# Step 1: Try to find the Python executable in memory
binary_path, base_address = find_loaded_module(
pid, name_contains="python"
)
# Step 2: Fallback to shared pythonXY.dll if the executable is not
# found
if binary_path is None:
binary_path, base_address = find_loaded_module(
pid, name_contains="python3"
)
# Step 3: Parse PE section headers to get the RVA of the PyRuntime
# section. The section name appears as "PyRuntim" due to the
# 8-character limit defined by the PE format (IMAGE_SIZEOF_SHORT_NAME).
section_rva = parse_pe_section_offset(binary_path, "PyRuntim")
# Step 4: Compute PyRuntime address in memory
return base_address + section_rva
On Windows, accessing another process's memory requires using the Windows API
functions like ``CreateToolhelp32Snapshot()`` and ``Module32First()/Module32Next()``
to enumerate loaded modules. The ``OpenProcess()`` function provides a handle to
access the target process's memory space, enabling memory operations through
``ReadProcessMemory()``.
The process memory can be examined by enumerating loaded modules to find the
Python binary or DLL. When found, its PE headers need to be parsed to locate the
``PyRuntime`` structure.
The PE format organizes code and data into sections. The ``PyRuntime`` structure
lives in a section named "PyRuntim" (truncated from "PyRuntime" due to PE's
8-character name limit). The actual runtime address calculation involves finding
the module's base address from the module entry, then locating our target
section in the PE headers. The final address is computed by combining the base
address with the section's virtual address from the PE section headers.
Note that accessing another process's memory on Windows typically requires
appropriate privileges - either administrative access or the ``SeDebugPrivilege``
privilege granted to the debugging process.
Reading _Py_DebugOffsets
========================
Once the address of the ``PyRuntime`` structure has been determined, the next
step is to read the ``_Py_DebugOffsets`` structure located at the beginning of
the ``PyRuntime`` block.
This structure provides version-specific field offsets that are needed to
safely read interpreter and thread state memory. These offsets vary between
CPython versions and must be checked before use to ensure they are compatible.
To read and check the debug offsets, follow these steps:
1. Read memory from the target process starting at the ``PyRuntime`` address,
covering the same number of bytes as the ``_Py_DebugOffsets`` structure.
This structure is located at the very start of the ``PyRuntime`` memory
block. Its layout is defined in CPython’s internal headers and stays the
same within a given minor version, but may change in major versions.
2. Check that the structure contains valid data:
- The ``cookie`` field must match the expected debug marker.
- The ``version`` field must match the version of the Python interpreter
used by the debugger.
- If either the debugger or the target process is using a pre-release
version (for example, an alpha, beta, or release candidate), the versions
must match exactly.
- The ``free_threaded`` field must have the same value in both the debugger
and the target process.
3. If the structure is valid, the offsets it contains can be used to locate
fields in memory. If any check fails, the debugger should stop the operation
to avoid reading memory in the wrong format.
The following is an example implementation that reads and checks
``_Py_DebugOffsets``::
def read_debug_offsets(pid: int, py_runtime_addr: int) -> DebugOffsets:
# Step 1: Read memory from the target process at the PyRuntime address
data = read_process_memory(
pid, address=py_runtime_addr, size=DEBUG_OFFSETS_SIZE
)
# Step 2: Deserialize the raw bytes into a _Py_DebugOffsets structure
debug_offsets = parse_debug_offsets(data)
# Step 3: Validate the contents of the structure
if debug_offsets.cookie != EXPECTED_COOKIE:
raise RuntimeError("Invalid or missing debug cookie")
if debug_offsets.version != LOCAL_PYTHON_VERSION:
raise RuntimeError(
"Mismatch between caller and target Python versions"
)
if debug_offsets.free_threaded != LOCAL_FREE_THREADED:
raise RuntimeError("Mismatch in free-threaded configuration")
return debug_offsets
.. warning::
**Process suspension recommended**
To avoid race conditions and ensure memory consistency, it is strongly
recommended that the target process be suspended before performing any
operations that read or write internal interpreter state. The Python runtime
may concurrently mutate interpreter data structures—such as creating or
destroying threads—during normal execution. This can result in invalid
memory reads or writes.
A debugger may suspend execution by attaching to the process with ``ptrace``
or by sending a ``SIGSTOP`` signal. Execution should only be resumed after
debugger-side memory operations are complete.
.. note::
Some tools, such as profilers or sampling-based debuggers, may operate on
a running process without suspension. In such cases, tools must be
explicitly designed to handle partially updated or inconsistent memory.
For most debugger implementations, suspending the process remains the
safest and most robust approach.
Locating the interpreter and thread state
=========================================
Before code can be injected and executed in a remote Python process, the
debugger must choose a thread in which to schedule execution. This is necessary
because the control fields used to perform remote code injection are located in
the ``_PyRemoteDebuggerSupport`` structure, which is embedded in a
``PyThreadState`` object. These fields are modified by the debugger to request
execution of injected scripts.
The ``PyThreadState`` structure represents a thread running inside a Python
interpreter. It maintains the thread’s evaluation context and contains the
fields required for debugger coordination. Locating a valid ``PyThreadState``
is therefore a key prerequisite for triggering execution remotely.
A thread is typically selected based on its role or ID. In most cases, the main
thread is used, but some tools may target a specific thread by its native
thread ID. Once the target thread is chosen, the debugger must locate both the
interpreter and the associated thread state structures in memory.
The relevant internal structures are defined as follows:
- ``PyInterpreterState`` represents an isolated Python interpreter instance.
Each interpreter maintains its own set of imported modules, built-in state,
and thread state list. Although most Python applications use a single
interpreter, CPython supports multiple interpreters in the same process.
- ``PyThreadState`` represents a thread running within an interpreter. It
contains execution state and the control fields used by the debugger.
To locate a thread:
1. Use the offset ``runtime_state.interpreters_head`` to obtain the address of
the first interpreter in the ``PyRuntime`` structure. This is the entry point
to the linked list of active interpreters.
2. Use the offset ``interpreter_state.threads_main`` to access the main thread
state associated with the selected interpreter. This is typically the most
reliable thread to target.
3. Optionally, use the offset ``interpreter_state.threads_head`` to iterate
through the linked list of all thread states. Each ``PyThreadState`` structure
contains a ``native_thread_id`` field, which may be compared to a target thread
ID to find a specific thread.
1. Once a valid ``PyThreadState`` has been found, its address can be used in
later steps of the protocol, such as writing debugger control fields and
scheduling execution.
The following is an example implementation that locates the main thread state::
def find_main_thread_state(
pid: int, py_runtime_addr: int, debug_offsets: DebugOffsets,
) -> int:
# Step 1: Read interpreters_head from PyRuntime
interp_head_ptr = (
py_runtime_addr + debug_offsets.runtime_state.interpreters_head
)
interp_addr = read_pointer(pid, interp_head_ptr)
if interp_addr == 0:
raise RuntimeError("No interpreter found in the target process")
# Step 2: Read the threads_main pointer from the interpreter
threads_main_ptr = (
interp_addr + debug_offsets.interpreter_state.threads_main
)
thread_state_addr = read_pointer(pid, threads_main_ptr)
if thread_state_addr == 0:
raise RuntimeError("Main thread state is not available")
return thread_state_addr
The following example demonstrates how to locate a thread by its native thread
ID::
def find_thread_by_id(
pid: int,
interp_addr: int,
debug_offsets: DebugOffsets,
target_tid: int,
) -> int:
# Start at threads_head and walk the linked list
thread_ptr = read_pointer(
pid,
interp_addr + debug_offsets.interpreter_state.threads_head
)
while thread_ptr:
native_tid_ptr = (
thread_ptr + debug_offsets.thread_state.native_thread_id
)
native_tid = read_int(pid, native_tid_ptr)
if native_tid == target_tid:
return thread_ptr
thread_ptr = read_pointer(
pid,
thread_ptr + debug_offsets.thread_state.next
)
raise RuntimeError("Thread with the given ID was not found")
Once a valid thread state has been located, the debugger can proceed with
modifying its control fields and scheduling execution, as described in the next
section.
Writing control information
===========================
Once a valid ``PyThreadState`` structure has been identified, the debugger may
modify control fields within it to schedule the execution of a specified Python
script. These control fields are checked periodically by the interpreter, and
when set correctly, they trigger the execution of remote code at a safe point
in the evaluation loop.
Each ``PyThreadState`` contains a ``_PyRemoteDebuggerSupport`` structure used
for communication between the debugger and the interpreter. The locations of
its fields are defined by the ``_Py_DebugOffsets`` structure and include the
following:
- ``debugger_script_path``: A fixed-size buffer that holds the full path to a
Python source file (``.py``). This file must be accessible and readable by
the target process when execution is triggered.
- ``debugger_pending_call``: An integer flag. Setting this to ``1`` tells the
interpreter that a script is ready to be executed.
- ``eval_breaker``: A field checked by the interpreter during execution.
Setting bit 5 (``_PY_EVAL_PLEASE_STOP_BIT``, value ``1U << 5``) in this
field causes the interpreter to pause and check for debugger activity.
To complete the injection, the debugger must perform the following steps:
1. Write the full script path into the ``debugger_script_path`` buffer.
2. Set ``debugger_pending_call`` to ``1``.
3. Read the current value of ``eval_breaker``, set bit 5
(``_PY_EVAL_PLEASE_STOP_BIT``), and write the updated value back. This
signals the interpreter to check for debugger activity.
The following is an example implementation::
def inject_script(
pid: int,
thread_state_addr: int,
debug_offsets: DebugOffsets,
script_path: str
) -> None:
# Compute the base offset of _PyRemoteDebuggerSupport
support_base = (
thread_state_addr +
debug_offsets.debugger_support.remote_debugger_support
)
# Step 1: Write the script path into debugger_script_path
script_path_ptr = (
support_base +
debug_offsets.debugger_support.debugger_script_path
)
write_string(pid, script_path_ptr, script_path)
# Step 2: Set debugger_pending_call to 1
pending_ptr = (
support_base +
debug_offsets.debugger_support.debugger_pending_call
)
write_int(pid, pending_ptr, 1)
# Step 3: Set _PY_EVAL_PLEASE_STOP_BIT (bit 5, value 1 << 5) in
# eval_breaker
eval_breaker_ptr = (
thread_state_addr +
debug_offsets.debugger_support.eval_breaker
)
breaker = read_int(pid, eval_breaker_ptr)
breaker |= (1 << 5)
write_int(pid, eval_breaker_ptr, breaker)
Once these fields are set, the debugger may resume the process (if it was
suspended). The interpreter will process the request at the next safe
evaluation point, load the script from disk, and execute it.
It is the responsibility of the debugger to ensure that the script file remains
present and accessible to the target process during execution.
.. note::
Script execution is asynchronous. The script file cannot be deleted
immediately after injection. The debugger should wait until the injected
script has produced an observable effect before removing the file.
This effect depends on what the script is designed to do. For example,
a debugger might wait until the remote process connects back to a socket
before removing the script. Once such an effect is observed, it is safe to
assume the file is no longer needed.
Summary
=======
To inject and execute a Python script in a remote process:
1. Locate the ``PyRuntime`` structure in the target process’s memory.
2. Read and validate the ``_Py_DebugOffsets`` structure at the beginning of
``PyRuntime``.
3. Use the offsets to locate a valid ``PyThreadState``.
4. Write the path to a Python script into ``debugger_script_path``.
5. Set the ``debugger_pending_call`` flag to ``1``.
6. Set ``_PY_EVAL_PLEASE_STOP_BIT`` in the ``eval_breaker`` field.
7. Resume the process (if suspended). The script will execute at the next safe
evaluation point.

271
Doc/library/annotationlib.rst
View file

@ -40,7 +40,7 @@ The :func:`get_annotations` function is the main entry point for
retrieving annotations. Given a function, class, or module, it returns
an annotations dictionary in the requested format. This module also provides
functionality for working directly with the :term:`annotate function`
that is used to evaluate annotations, such as :func:`get_annotate_function`
that is used to evaluate annotations, such as :func:`get_annotate_from_class_namespace`
and :func:`call_annotate_function`, as well as the
:func:`call_evaluate_function` function for working with
:term:`evaluate functions <evaluate function>`.
@ -127,25 +127,8 @@ Classes
Values are the result of evaluating the annotation expressions.
.. attribute:: FORWARDREF
:value: 2
Values are real annotation values (as per :attr:`Format.VALUE` format)
for defined values, and :class:`ForwardRef` proxies for undefined
values. Real objects may contain references to, :class:`ForwardRef`
proxy objects.
.. attribute:: STRING
:value: 3
Values are the text string of the annotation as it appears in the
source code, up to modifications including, but not restricted to,
whitespace normalizations and constant values optimizations.
The exact values of these strings may change in future versions of Python.
.. attribute:: VALUE_WITH_FAKE_GLOBALS
:value: 4
:value: 2
Special value used to signal that an annotate function is being
evaluated in a special environment with fake globals. When passed this
@ -155,6 +138,23 @@ Classes
This format is only used internally and should not be passed to
the functions in this module.
.. attribute:: FORWARDREF
:value: 3
Values are real annotation values (as per :attr:`Format.VALUE` format)
for defined values, and :class:`ForwardRef` proxies for undefined
values. Real objects may contain references to :class:`ForwardRef`
proxy objects.
.. attribute:: STRING
:value: 4
Values are the text string of the annotation as it appears in the
source code, up to modifications including, but not restricted to,
whitespace normalizations and constant values optimizations.
The exact values of these strings may change in future versions of Python.
.. versionadded:: 3.14
.. class:: ForwardRef
@ -172,14 +172,21 @@ Classes
:class:`~ForwardRef`. The string may not be exactly equivalent
to the original source.
.. method:: evaluate(*, owner=None, globals=None, locals=None, type_params=None)
.. method:: evaluate(*, owner=None, globals=None, locals=None, type_params=None, format=Format.VALUE)
Evaluate the forward reference, returning its value.
This may throw an exception, such as :exc:`NameError`, if the forward
If the *format* argument is :attr:`~Format.VALUE` (the default),
this method may throw an exception, such as :exc:`NameError`, if the forward
reference refers to a name that cannot be resolved. The arguments to this
method can be used to provide bindings for names that would otherwise
be undefined.
be undefined. If the *format* argument is :attr:`~Format.FORWARDREF`,
the method will never throw an exception, but may return a :class:`~ForwardRef`
instance. For example, if the forward reference object contains the code
``list[undefined]``, where ``undefined`` is a name that is not defined,
evaluating it with the :attr:`~Format.FORWARDREF` format will return
``list[ForwardRef('undefined')]``. If the *format* argument is
:attr:`~Format.STRING`, the method will return :attr:`~ForwardRef.__forward_arg__`.
The *owner* parameter provides the preferred mechanism for passing scope
information to this method. The owner of a :class:`~ForwardRef` is the
@ -214,7 +221,7 @@ Functions
Convert an annotations dict containing runtime values to a
dict containing only strings. If the values are not already strings,
they are converted using :func:`value_to_string`.
they are converted using :func:`type_repr`.
This is meant as a helper for user-provided
annotate functions that support the :attr:`~Format.STRING` format but
do not have access to the code creating the annotations.
@ -300,15 +307,13 @@ Functions
.. versionadded:: 3.14
.. function:: get_annotate_function(obj)
.. function:: get_annotate_from_class_namespace(namespace)
Retrieve the :term:`annotate function` for *obj*. Return :const:`!None`
if *obj* does not have an annotate function.
This is usually equivalent to accessing the :attr:`~object.__annotate__`
attribute of *obj*, but direct access to the attribute may return the wrong
object in certain situations involving metaclasses. This function should be
used instead of accessing the attribute directly.
Retrieve the :term:`annotate function` from a class namespace dictionary *namespace*.
Return :const:`!None` if the namespace does not contain an annotate function.
This is primarily useful before the class has been fully created (e.g., in a metaclass);
after the class exists, the annotate function can be retrieved with ``cls.__annotate__``.
See :ref:`below <annotationlib-metaclass>` for an example using this function in a metaclass.
.. versionadded:: 3.14
@ -317,11 +322,22 @@ Functions
Compute the annotations dict for an object.
*obj* may be a callable, class, module, or other object with
:attr:`~object.__annotate__` and :attr:`~object.__annotations__` attributes.
Passing in an object of any other type raises :exc:`TypeError`.
:attr:`~object.__annotate__` or :attr:`~object.__annotations__` attributes.
Passing any other object raises :exc:`TypeError`.
The *format* parameter controls the format in which annotations are returned,
and must be a member of the :class:`Format` enum or its integer equivalent.
The different formats work as follows:
* VALUE: :attr:`!object.__annotations__` is tried first; if that does not exist,
the :attr:`!object.__annotate__` function is called if it exists.
* FORWARDREF: If :attr:`!object.__annotations__` exists and can be evaluated successfully,
it is used; otherwise, the :attr:`!object.__annotate__` function is called. If it
does not exist either, :attr:`!object.__annotations__` is tried again and any error
from accessing it is re-raised.
* STRING: If :attr:`!object.__annotate__` exists, it is called first;
otherwise, :attr:`!object.__annotations__` is used and stringified
using :func:`annotations_to_string`.
Returns a dict. :func:`!get_annotations` returns a new dict every time
it's called; calling it twice on the same object will return two
@ -382,7 +398,7 @@ Functions
.. versionadded:: 3.14
.. function:: value_to_string(value)
.. function:: type_repr(value)
Convert an arbitrary Python value to a format suitable for use by the
:attr:`~Format.STRING` format. This calls :func:`repr` for most
@ -396,3 +412,190 @@ Functions
.. versionadded:: 3.14
Recipes
-------
.. _annotationlib-metaclass:
Using annotations in a metaclass
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A :ref:`metaclass <metaclasses>` may want to inspect or even modify the annotations
in a class body during class creation. Doing so requires retrieving annotations
from the class namespace dictionary. For classes created with
``from __future__ import annotations``, the annotations will be in the ``__annotations__``
key of the dictionary. For other classes with annotations,
:func:`get_annotate_from_class_namespace` can be used to get the
annotate function, and :func:`call_annotate_function` can be used to call it and
retrieve the annotations. Using the :attr:`~Format.FORWARDREF` format will usually
be best, because this allows the annotations to refer to names that cannot yet be
resolved when the class is created.
To modify the annotations, it is best to create a wrapper annotate function
that calls the original annotate function, makes any necessary adjustments, and
returns the result.
Below is an example of a metaclass that filters out all :class:`typing.ClassVar`
annotations from the class and puts them in a separate attribute:
.. code-block:: python
import annotationlib
import typing
class ClassVarSeparator(type):
def __new__(mcls, name, bases, ns):
if "__annotations__" in ns: # from __future__ import annotations
annotations = ns["__annotations__"]
classvar_keys = {
key for key, value in annotations.items()
# Use string comparison for simplicity; a more robust solution
# could use annotationlib.ForwardRef.evaluate
if value.startswith("ClassVar")
}
classvars = {key: annotations[key] for key in classvar_keys}
ns["__annotations__"] = {
key: value for key, value in annotations.items()
if key not in classvar_keys
}
wrapped_annotate = None
elif annotate := annotationlib.get_annotate_from_class_namespace(ns):
annotations = annotationlib.call_annotate_function(
annotate, format=annotationlib.Format.FORWARDREF
)
classvar_keys = {
key for key, value in annotations.items()
if typing.get_origin(value) is typing.ClassVar
}
classvars = {key: annotations[key] for key in classvar_keys}
def wrapped_annotate(format):
annos = annotationlib.call_annotate_function(annotate, format, owner=typ)
return {key: value for key, value in annos.items() if key not in classvar_keys}
else: # no annotations
classvars = {}
wrapped_annotate = None
typ = super().__new__(mcls, name, bases, ns)
if wrapped_annotate is not None:
# Wrap the original __annotate__ with a wrapper that removes ClassVars
typ.__annotate__ = wrapped_annotate
typ.classvars = classvars # Store the ClassVars in a separate attribute
return typ
Limitations of the ``STRING`` format
------------------------------------
The :attr:`~Format.STRING` format is meant to approximate the source code
of the annotation, but the implementation strategy used means that it is not
always possible to recover the exact source code.
First, the stringifier of course cannot recover any information that is not present in
the compiled code, including comments, whitespace, parenthesization, and operations that
get simplified by the compiler.
Second, the stringifier can intercept almost all operations that involve names looked
up in some scope, but it cannot intercept operations that operate fully on constants.
As a corollary, this also means it is not safe to request the ``STRING`` format on
untrusted code: Python is powerful enough that it is possible to achieve arbitrary
code execution even with no access to any globals or builtins. For example:
.. code-block:: pycon
>>> def f(x: (1).__class__.__base__.__subclasses__()[-1].__init__.__builtins__["print"]("Hello world")): pass
...
>>> annotationlib.get_annotations(f, format=annotationlib.Format.SOURCE)
Hello world
{'x': 'None'}
.. note::
This particular example works as of the time of writing, but it relies on
implementation details and is not guaranteed to work in the future.
Among the different kinds of expressions that exist in Python,
as represented by the :mod:`ast` module, some expressions are supported,
meaning that the ``STRING`` format can generally recover the original source code;
others are unsupported, meaning that they may result in incorrect output or an error.
The following are supported (sometimes with caveats):
* :class:`ast.BinOp`
* :class:`ast.UnaryOp`
* :class:`ast.Invert` (``~``), :class:`ast.UAdd` (``+``), and :class:`ast.USub` (``-``) are supported
* :class:`ast.Not` (``not``) is not supported
* :class:`ast.Dict` (except when using ``**`` unpacking)
* :class:`ast.Set`
* :class:`ast.Compare`
* :class:`ast.Eq` and :class:`ast.NotEq` are supported
* :class:`ast.Lt`, :class:`ast.LtE`, :class:`ast.Gt`, and :class:`ast.GtE` are supported, but the operand may be flipped
* :class:`ast.Is`, :class:`ast.IsNot`, :class:`ast.In`, and :class:`ast.NotIn` are not supported
* :class:`ast.Call` (except when using ``**`` unpacking)
* :class:`ast.Constant` (though not the exact representation of the constant; for example, escape
sequences in strings are lost; hexadecimal numbers are converted to decimal)
* :class:`ast.Attribute` (assuming the value is not a constant)
* :class:`ast.Subscript` (assuming the value is not a constant)
* :class:`ast.Starred` (``*`` unpacking)
* :class:`ast.Name`
* :class:`ast.List`
* :class:`ast.Tuple`
* :class:`ast.Slice`
The following are unsupported, but throw an informative error when encountered by the
stringifier:
* :class:`ast.FormattedValue` (f-strings; error is not detected if conversion specifiers like ``!r``
are used)
* :class:`ast.JoinedStr` (f-strings)
The following are unsupported and result in incorrect output:
* :class:`ast.BoolOp` (``and`` and ``or``)
* :class:`ast.IfExp`
* :class:`ast.Lambda`
* :class:`ast.ListComp`
* :class:`ast.SetComp`
* :class:`ast.DictComp`
* :class:`ast.GeneratorExp`
The following are disallowed in annotation scopes and therefore not relevant:
* :class:`ast.NamedExpr` (``:=``)
* :class:`ast.Await`
* :class:`ast.Yield`
* :class:`ast.YieldFrom`
Limitations of the ``FORWARDREF`` format
----------------------------------------
The :attr:`~Format.FORWARDREF` format aims to produce real values as much
as possible, with anything that cannot be resolved replaced with
:class:`ForwardRef` objects. It is affected by broadly the same Limitations
as the :attr:`~Format.STRING` format: annotations that perform operations on
literals or that use unsupported expression types may raise exceptions when
evaluated using the :attr:`~Format.FORWARDREF` format.
Below are a few examples of the behavior with unsupported expressions:
.. code-block:: pycon
>>> from annotationlib import get_annotations, Format
>>> def zerodiv(x: 1 / 0): ...
>>> get_annotations(zerodiv, format=Format.STRING)
Traceback (most recent call last):
...
ZeroDivisionError: division by zero
>>> get_annotations(zerodiv, format=Format.FORWARDREF)
Traceback (most recent call last):
...
ZeroDivisionError: division by zero
>>> def ifexp(x: 1 if y else 0): ...
>>> get_annotations(ifexp, format=Format.STRING)
{'x': '1'}

4
Doc/library/archiving.rst
View file

@ -5,13 +5,15 @@ Data Compression and Archiving
******************************
The modules described in this chapter support data compression with the zlib,
gzip, bzip2 and lzma algorithms, and the creation of ZIP- and tar-format
gzip, bzip2, lzma, and zstd algorithms, and the creation of ZIP- and tar-format
archives. See also :ref:`archiving-operations` provided by the :mod:`shutil`
module.
.. toctree::
compression.rst
compression.zstd.rst
zlib.rst
gzip.rst
bz2.rst

38
Doc/library/argparse.rst
View file

@ -74,7 +74,7 @@ ArgumentParser objects
prefix_chars='-', fromfile_prefix_chars=None, \
argument_default=None, conflict_handler='error', \
add_help=True, allow_abbrev=True, exit_on_error=True, \
suggest_on_error=False)
*, suggest_on_error=False, color=False)
Create a new :class:`ArgumentParser` object. All parameters should be passed
as keyword arguments. Each parameter has its own more detailed description
@ -111,7 +111,7 @@ ArgumentParser objects
* add_help_ - Add a ``-h/--help`` option to the parser (default: ``True``)
* allow_abbrev_ - Allows long options to be abbreviated if the
abbreviation is unambiguous. (default: ``True``)
abbreviation is unambiguous (default: ``True``)
* exit_on_error_ - Determines whether or not :class:`!ArgumentParser` exits with
error info when an error occurs. (default: ``True``)
@ -119,6 +119,7 @@ ArgumentParser objects
* suggest_on_error_ - Enables suggestions for mistyped argument choices
and subparser names (default: ``False``)
* color_ - Allow color output (default: ``False``)
.. versionchanged:: 3.5
*allow_abbrev* parameter was added.
@ -130,6 +131,9 @@ ArgumentParser objects
.. versionchanged:: 3.9
*exit_on_error* parameter was added.
.. versionchanged:: 3.14
*suggest_on_error* and *color* parameters were added.
The following sections describe how each of these are used.
@ -594,7 +598,8 @@ subparser names, the feature can be enabled by setting ``suggest_on_error`` to
``True``. Note that this only applies for arguments when the choices specified
are strings::
>>> parser = argparse.ArgumentParser(description='Process some integers.', suggest_on_error=True)
>>> parser = argparse.ArgumentParser(description='Process some integers.',
suggest_on_error=True)
>>> parser.add_argument('--action', choices=['sum', 'max'])
>>> parser.add_argument('integers', metavar='N', type=int, nargs='+',
... help='an integer for the accumulator')
@ -612,6 +617,33 @@ keyword argument::
.. versionadded:: 3.14
color
^^^^^
By default, the help message is printed in plain text. If you want to allow
color in help messages, you can enable it by setting ``color`` to ``True``::
>>> parser = argparse.ArgumentParser(description='Process some integers.',
... color=True)
>>> parser.add_argument('--action', choices=['sum', 'max'])
>>> parser.add_argument('integers', metavar='N', type=int, nargs='+',
... help='an integer for the accumulator')
>>> parser.parse_args(['--help'])
Even if a CLI author has enabled color, it can be
:ref:`controlled using environment variables <using-on-controlling-color>`.
If you're writing code that needs to be compatible with older Python versions
and want to opportunistically use ``color`` when it's available, you
can set it as an attribute after initializing the parser instead of using the
keyword argument::
>>> parser = argparse.ArgumentParser(description='Process some integers.')
>>> parser.color = True
.. versionadded:: 3.14
The add_argument() method
-------------------------

73
Doc/library/ast.rst
View file

@ -1,4 +1,4 @@
:mod:`!ast` --- Abstract Syntax Trees
:mod:`!ast` --- Abstract syntax trees
=====================================
.. module:: ast
@ -29,7 +29,7 @@ compiled into a Python code object using the built-in :func:`compile` function.
.. _abstract-grammar:
Abstract Grammar
Abstract grammar
----------------
The abstract grammar is currently defined as follows:
@ -1190,7 +1190,7 @@ Control flow
.. doctest::
>> print(ast.dump(ast.parse("""
>>> print(ast.dump(ast.parse("""
... while x:
... ...
... else:
@ -1761,6 +1761,43 @@ Pattern matching
.. versionadded:: 3.10
Type annotations
^^^^^^^^^^^^^^^^
.. class:: TypeIgnore(lineno, tag)
A ``# type: ignore`` comment located at *lineno*.
*tag* is the optional tag specified by the form ``# type: ignore <tag>``.
.. doctest::
>>> print(ast.dump(ast.parse('x = 1 # type: ignore', type_comments=True), indent=4))
Module(
body=[
Assign(
targets=[
Name(id='x', ctx=Store())],
value=Constant(value=1))],
type_ignores=[
TypeIgnore(lineno=1, tag='')])
>>> print(ast.dump(ast.parse('x: bool = 1 # type: ignore[assignment]', type_comments=True), indent=4))
Module(
body=[
AnnAssign(
target=Name(id='x', ctx=Store()),
annotation=Name(id='bool', ctx=Load()),
value=Constant(value=1),
simple=1)],
type_ignores=[
TypeIgnore(lineno=1, tag='[assignment]')])
.. note::
:class:`!TypeIgnore` nodes are not generated when the *type_comments* parameter
is set to ``False`` (default). See :func:`ast.parse` for more details.
.. versionadded:: 3.8
.. _ast-type-params:
Type parameters
@ -2119,10 +2156,10 @@ Async and await
of :class:`ast.operator`, :class:`ast.unaryop`, :class:`ast.cmpop`,
:class:`ast.boolop` and :class:`ast.expr_context`) on the returned tree
will be singletons. Changes to one will be reflected in all other
occurrences of the same value (e.g. :class:`ast.Add`).
occurrences of the same value (for example, :class:`ast.Add`).
:mod:`ast` Helpers
:mod:`ast` helpers
------------------
Apart from the node classes, the :mod:`ast` module defines these utility functions
@ -2447,7 +2484,7 @@ and classes for traversing abstract syntax trees:
.. _ast-compiler-flags:
Compiler Flags
Compiler flags
--------------
The following flags may be passed to :func:`compile` in order to change
@ -2496,7 +2533,7 @@ effects on the compilation of a program:
.. _ast-cli:
Command-Line Usage
Command-line usage
------------------
.. versionadded:: 3.9
@ -2535,6 +2572,28 @@ The following options are accepted:
Indentation of nodes in AST (number of spaces).
.. option:: --feature-version <version>
Python version in the format 3.x (for example, 3.10). Defaults to the
current version of the interpreter.
.. versionadded:: 3.14
.. option:: -O <level>
--optimize <level>
Optimization level for parser. Defaults to no optimization.
.. versionadded:: 3.14
.. option:: --show-empty
Show empty lists and fields that are ``None``. Defaults to not showing empty
objects.
.. versionadded:: 3.14
If :file:`infile` is specified its contents are parsed to AST and dumped
to stdout. Otherwise, the content is read from stdin.

4
Doc/library/asyncio-dev.rst
View file

@ -46,10 +46,6 @@ In addition to enabling the debug mode, consider also:
When the debug mode is enabled:
* asyncio checks for :ref:`coroutines that were not awaited
<asyncio-coroutine-not-scheduled>` and logs them; this mitigates
the "forgotten await" pitfall.
* Many non-threadsafe asyncio APIs (such as :meth:`loop.call_soon` and
:meth:`loop.call_at` methods) raise an exception if they are called
from a wrong thread.

43
Doc/library/asyncio-eventloop.rst
View file

@ -65,18 +65,14 @@ an event loop:
.. note::
The :mod:`!asyncio` policy system is deprecated and will be removed
in Python 3.16; from there on, this function will always return the
running event loop.
in Python 3.16; from there on, this function will return the current
running event loop if present else it will return the
loop set by :func:`set_event_loop`.
.. function:: set_event_loop(loop)
Set *loop* as the current event loop for the current OS thread.
.. deprecated:: 3.14
The :func:`set_event_loop` function is deprecated and will be removed
in Python 3.16.
.. function:: new_event_loop()
Create and return a new event loop object.
@ -365,7 +361,7 @@ Creating Futures and Tasks
.. versionadded:: 3.5.2
.. method:: loop.create_task(coro, *, name=None, context=None)
.. method:: loop.create_task(coro, *, name=None, context=None, eager_start=None, **kwargs)
Schedule the execution of :ref:`coroutine <coroutine>` *coro*.
Return a :class:`Task` object.
@ -374,6 +370,10 @@ Creating Futures and Tasks
for interoperability. In this case, the result type is a subclass
of :class:`Task`.
The full function signature is largely the same as that of the
:class:`Task` constructor (or factory) - all of the keyword arguments to
this function are passed through to that interface.
If the *name* argument is provided and not ``None``, it is set as
the name of the task using :meth:`Task.set_name`.
@ -381,12 +381,27 @@ Creating Futures and Tasks
custom :class:`contextvars.Context` for the *coro* to run in.
The current context copy is created when no *context* is provided.
An optional keyword-only *eager_start* argument allows specifying
if the task should execute eagerly during the call to create_task,
or be scheduled later. If *eager_start* is not passed the mode set
by :meth:`loop.set_task_factory` will be used.
.. versionchanged:: 3.8
Added the *name* parameter.
.. versionchanged:: 3.11
Added the *context* parameter.
.. versionchanged:: 3.13.3
Added ``kwargs`` which passes on arbitrary extra parameters, including ``name`` and ``context``.
.. versionchanged:: 3.13.4
Rolled back the change that passes on *name* and *context* (if it is None),
while still passing on other arbitrary keyword arguments (to avoid breaking backwards compatibility with 3.13.3).
.. versionchanged:: 3.14
All *kwargs* are now passed on. The *eager_start* parameter works with eager task factories.
.. method:: loop.set_task_factory(factory)
Set a task factory that will be used by
@ -398,6 +413,16 @@ Creating Futures and Tasks
event loop, and *coro* is a coroutine object. The callable
must pass on all *kwargs*, and return a :class:`asyncio.Task`-compatible object.
.. versionchanged:: 3.13.3
Required that all *kwargs* are passed on to :class:`asyncio.Task`.
.. versionchanged:: 3.13.4
*name* is no longer passed to task factories. *context* is no longer passed
to task factories if it is ``None``.
.. versionchanged:: 3.14
*name* and *context* are now unconditionally passed on to task factories again.
.. method:: loop.get_task_factory()
Return a task factory or ``None`` if the default one is in use.
@ -1444,6 +1469,8 @@ Allows customizing how exceptions are handled in the event loop.
* 'protocol' (optional): :ref:`Protocol <asyncio-protocol>` instance;
* 'transport' (optional): :ref:`Transport <asyncio-transport>` instance;
* 'socket' (optional): :class:`socket.socket` instance;
* 'source_traceback' (optional): Traceback of the source;
* 'handle_traceback' (optional): Traceback of the handle;
* 'asyncgen' (optional): Asynchronous generator that caused
the exception.

7
Doc/library/asyncio-future.rst
View file

@ -51,12 +51,13 @@ Future Functions
.. important::
See also the :func:`create_task` function which is the
preferred way for creating new Tasks.
Save a reference to the result of this function, to avoid
a task disappearing mid-execution.
See also the :func:`create_task` function which is the
preferred way for creating new tasks or use :class:`asyncio.TaskGroup`
which keeps reference to the task internally.
.. versionchanged:: 3.5.1
The function accepts any :term:`awaitable` object.

26
Doc/library/asyncio-task.rst
View file

@ -238,18 +238,24 @@ Creating Tasks
-----------------------------------------------
.. function:: create_task(coro, *, name=None, context=None)
.. function:: create_task(coro, *, name=None, context=None, eager_start=None, **kwargs)
Wrap the *coro* :ref:`coroutine <coroutine>` into a :class:`Task`
and schedule its execution. Return the Task object.
If *name* is not ``None``, it is set as the name of the task using
:meth:`Task.set_name`.
The full function signature is largely the same as that of the
:class:`Task` constructor (or factory) - all of the keyword arguments to
this function are passed through to that interface.
An optional keyword-only *context* argument allows specifying a
custom :class:`contextvars.Context` for the *coro* to run in.
The current context copy is created when no *context* is provided.
An optional keyword-only *eager_start* argument allows specifying
if the task should execute eagerly during the call to create_task,
or be scheduled later. If *eager_start* is not passed the mode set
by :meth:`loop.set_task_factory` will be used.
The task is executed in the loop returned by :func:`get_running_loop`,
:exc:`RuntimeError` is raised if there is no running loop in
current thread.
@ -290,6 +296,9 @@ Creating Tasks
.. versionchanged:: 3.11
Added the *context* parameter.
.. versionchanged:: 3.14
Added the *eager_start* parameter by passing on all *kwargs*.
Task Cancellation
=================
@ -330,7 +339,7 @@ and reliable way to wait for all tasks in the group to finish.
.. versionadded:: 3.11
.. method:: create_task(coro, *, name=None, context=None)
.. method:: create_task(coro, *, name=None, context=None, eager_start=None, **kwargs)
Create a task in this task group.
The signature matches that of :func:`asyncio.create_task`.
@ -342,6 +351,10 @@ and reliable way to wait for all tasks in the group to finish.
Close the given coroutine if the task group is not active.
.. versionchanged:: 3.14
Passes on all *kwargs* to :meth:`loop.create_task`
Example::
async def main():
@ -1381,7 +1394,10 @@ Task Object
Request the Task to be cancelled.
This arranges for a :exc:`CancelledError` exception to be thrown
If the Task is already *done* or *cancelled*, return ``False``,
otherwise, return ``True``.
The method arranges for a :exc:`CancelledError` exception to be thrown
into the wrapped coroutine on the next cycle of the event loop.
The coroutine then has a chance to clean up or even deny the

49
Doc/library/audit_events.rst
View file

@ -23,25 +23,30 @@ information on handling these events.
The following events are raised internally and do not correspond to any
public API of CPython:
+--------------------------+-------------------------------------------+
| Audit event | Arguments |
+==========================+===========================================+
| _winapi.CreateFile | ``file_name``, ``desired_access``, |
| | ``share_mode``, ``creation_disposition``, |
| | ``flags_and_attributes`` |
+--------------------------+-------------------------------------------+
| _winapi.CreateJunction | ``src_path``, ``dst_path`` |
+--------------------------+-------------------------------------------+
| _winapi.CreateNamedPipe | ``name``, ``open_mode``, ``pipe_mode`` |
+--------------------------+-------------------------------------------+
| _winapi.CreatePipe | |
+--------------------------+-------------------------------------------+
| _winapi.CreateProcess | ``application_name``, ``command_line``, |
| | ``current_directory`` |
+--------------------------+-------------------------------------------+
| _winapi.OpenProcess | ``process_id``, ``desired_access`` |
+--------------------------+-------------------------------------------+
| _winapi.TerminateProcess | ``handle``, ``exit_code`` |
+--------------------------+-------------------------------------------+
| ctypes.PyObj_FromPtr | ``obj`` |
+--------------------------+-------------------------------------------+
+----------------------------+-------------------------------------------+
| Audit event | Arguments |
+============================+===========================================+
| _winapi.CreateFile | ``file_name``, ``desired_access``, |
| | ``share_mode``, ``creation_disposition``, |
| | ``flags_and_attributes`` |
+----------------------------+-------------------------------------------+
| _winapi.CreateJunction | ``src_path``, ``dst_path`` |
+----------------------------+-------------------------------------------+
| _winapi.CreateNamedPipe | ``name``, ``open_mode``, ``pipe_mode`` |
+----------------------------+-------------------------------------------+
| _winapi.CreatePipe | |
+----------------------------+-------------------------------------------+
| _winapi.CreateProcess | ``application_name``, ``command_line``, |
| | ``current_directory`` |
+----------------------------+-------------------------------------------+
| _winapi.OpenProcess | ``process_id``, ``desired_access`` |
+----------------------------+-------------------------------------------+
| _winapi.TerminateProcess | ``handle``, ``exit_code`` |
+----------------------------+-------------------------------------------+
| _posixsubprocess.fork_exec | ``exec_list``, ``args``, ``env`` |
+----------------------------+-------------------------------------------+
| ctypes.PyObj_FromPtr | ``obj`` |
+----------------------------+-------------------------------------------+
.. versionadded:: 3.14
The ``_posixsubprocess.fork_exec`` internal audit event.

48
Doc/library/base64.rst
View file

@ -15,14 +15,9 @@
This module provides functions for encoding binary data to printable
ASCII characters and decoding such encodings back to binary data.
It provides encoding and decoding functions for the encodings specified in
:rfc:`4648`, which defines the Base16, Base32, and Base64 algorithms,
and for the de-facto standard Ascii85 and Base85 encodings.
The :rfc:`4648` encodings are suitable for encoding binary data so that it can be
safely sent by email, used as parts of URLs, or included as part of an HTTP
POST request. The encoding algorithm is not the same as the
:program:`uuencode` program.
This includes the :ref:`encodings specified in <base64-rfc-4648>`
:rfc:`4648` (Base64, Base32 and Base16)
and the non-standard :ref:`Base85 encodings <base64-base-85>`.
There are two interfaces provided by this module. The modern interface
supports encoding :term:`bytes-like objects <bytes-like object>` to ASCII
@ -30,7 +25,7 @@ supports encoding :term:`bytes-like objects <bytes-like object>` to ASCII
strings containing ASCII to :class:`bytes`. Both base-64 alphabets
defined in :rfc:`4648` (normal, and URL- and filesystem-safe) are supported.
The legacy interface does not support decoding from strings, but it does
The :ref:`legacy interface <base64-legacy>` does not support decoding from strings, but it does
provide functions for encoding and decoding to and from :term:`file objects
<file object>`. It only supports the Base64 standard alphabet, and it adds
newlines every 76 characters as per :rfc:`2045`. Note that if you are looking
@ -46,7 +41,15 @@ package instead.
Any :term:`bytes-like objects <bytes-like object>` are now accepted by all
encoding and decoding functions in this module. Ascii85/Base85 support added.
The modern interface provides:
.. _base64-rfc-4648:
RFC 4648 Encodings
------------------
The :rfc:`4648` encodings are suitable for encoding binary data so that it can be
safely sent by email, used as parts of URLs, or included as part of an HTTP
POST request.
.. function:: b64encode(s, altchars=None)
@ -181,6 +184,26 @@ The modern interface provides:
incorrectly padded or if there are non-alphabet characters present in the
input.
.. _base64-base-85:
Base85 Encodings
-----------------
Base85 encoding is not formally specified but rather a de facto standard,
thus different systems perform the encoding differently.
The :func:`a85encode` and :func:`b85encode` functions in this module are two implementations of
the de facto standard. You should call the function with the Base85
implementation used by the software you intend to work with.
The two functions present in this module differ in how they handle the following:
* Whether to include enclosing ``<~`` and ``~>`` markers
* Whether to include newline characters
* The set of ASCII characters used for encoding
* Handling of null bytes
Refer to the documentation of the individual functions for more information.
.. function:: a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False)
@ -262,7 +285,10 @@ The modern interface provides:
.. versionadded:: 3.13
The legacy interface:
.. _base64-legacy:
Legacy Interface
----------------
.. function:: decode(input, output)

4
Doc/library/cmdline.rst
View file

@ -1,3 +1,5 @@
.. _library-cmdline:
++++++++++++++++++++++++++++++++++++
Modules command-line interface (CLI)
++++++++++++++++++++++++++++++++++++
@ -27,7 +29,7 @@ The following modules have a command-line interface.
* :mod:`pdb`
* :ref:`pickle <pickle-cli>`
* :ref:`pickletools <pickletools-cli>`
* :mod:`platform`
* :ref:`platform <platform-cli>`
* :mod:`poplib`
* :ref:`profile <profile-cli>`
* :mod:`pstats`

6
Doc/library/code.rst
View file

@ -22,6 +22,12 @@ build applications which provide an interactive interpreter prompt.
it defaults to a newly created dictionary with key ``'__name__'`` set to
``'__console__'`` and key ``'__doc__'`` set to ``None``.
Note that functions and classes objects created under an
:class:`!InteractiveInterpreter` instance will belong to the namespace
specified by *locals*.
They are only pickleable if *locals* is the namespace of an existing
module.
.. class:: InteractiveConsole(locals=None, filename="<console>", local_exit=False)

22
Doc/library/codecs.rst
View file

@ -55,7 +55,7 @@ any codec:
The full details for each codec can also be looked up directly:
.. function:: lookup(encoding)
.. function:: lookup(encoding, /)
Looks up the codec info in the Python codec registry and returns a
:class:`CodecInfo` object as defined below.
@ -156,7 +156,7 @@ these additional functions which use :func:`lookup` for the codec lookup:
Custom codecs are made available by registering a suitable codec search
function:
.. function:: register(search_function)
.. function:: register(search_function, /)
Register a codec search function. Search functions are expected to take one
argument, being the encoding name in all lower case letters with hyphens
@ -168,7 +168,7 @@ function:
Hyphens and spaces are converted to underscore.
.. function:: unregister(search_function)
.. function:: unregister(search_function, /)
Unregister a codec search function and clear the registry's cache.
If the search function is not registered, do nothing.
@ -208,6 +208,10 @@ wider range of codecs when working with binary files:
.. versionchanged:: 3.11
The ``'U'`` mode has been removed.
.. deprecated:: 3.14
:func:`codecs.open` has been superseded by :func:`open`.
.. function:: EncodedFile(file, data_encoding, file_encoding=None, errors='strict')
@ -416,7 +420,7 @@ In addition, the following error handler is specific to the given codecs:
The set of allowed values can be extended by registering a new named error
handler:
.. function:: register_error(name, error_handler)
.. function:: register_error(name, error_handler, /)
Register the error handling function *error_handler* under the name *name*.
The *error_handler* argument will be called during encoding and decoding
@ -442,7 +446,7 @@ handler:
Previously registered error handlers (including the standard error handlers)
can be looked up by name:
.. function:: lookup_error(name)
.. function:: lookup_error(name, /)
Return the error handler previously registered under the name *name*.
@ -1107,7 +1111,7 @@ particular, the following variants typically exist:
+-----------------+--------------------------------+--------------------------------+
| cp852 | 852, IBM852 | Central and Eastern Europe |
+-----------------+--------------------------------+--------------------------------+
| cp855 | 855, IBM855 | Bulgarian, Byelorussian, |
| cp855 | 855, IBM855 | Belarusian, Bulgarian, |
| | | Macedonian, Russian, Serbian |
+-----------------+--------------------------------+--------------------------------+
| cp856 | | Hebrew |
@ -1155,7 +1159,7 @@ particular, the following variants typically exist:
+-----------------+--------------------------------+--------------------------------+
| cp1250 | windows-1250 | Central and Eastern Europe |
+-----------------+--------------------------------+--------------------------------+
| cp1251 | windows-1251 | Bulgarian, Byelorussian, |
| cp1251 | windows-1251 | Belarusian, Bulgarian, |
| | | Macedonian, Russian, Serbian |
+-----------------+--------------------------------+--------------------------------+
| cp1252 | windows-1252 | Western Europe |
@ -1220,7 +1224,7 @@ particular, the following variants typically exist:
+-----------------+--------------------------------+--------------------------------+
| iso8859_4 | iso-8859-4, latin4, L4 | Baltic languages |
+-----------------+--------------------------------+--------------------------------+
| iso8859_5 | iso-8859-5, cyrillic | Bulgarian, Byelorussian, |
| iso8859_5 | iso-8859-5, cyrillic | Belarusian, Bulgarian, |
| | | Macedonian, Russian, Serbian |
+-----------------+--------------------------------+--------------------------------+
| iso8859_6 | iso-8859-6, arabic | Arabic |
@ -1257,7 +1261,7 @@ particular, the following variants typically exist:
| | | |
| | | .. versionadded:: 3.5 |
+-----------------+--------------------------------+--------------------------------+
| mac_cyrillic | maccyrillic | Bulgarian, Byelorussian, |
| mac_cyrillic | maccyrillic | Belarusian, Bulgarian, |
| | | Macedonian, Russian, Serbian |
+-----------------+--------------------------------+--------------------------------+
| mac_greek | macgreek | Greek |

18
Doc/library/compression.rst Normal file
View file

@ -0,0 +1,18 @@
The :mod:`!compression` package
===============================
.. versionadded:: 3.14
The :mod:`!compression` package contains the canonical compression modules
containing interfaces to several different compression algorithms. Some of
these modules have historically been available as separate modules; those will
continue to be available under their original names for compatibility reasons,
and will not be removed without a deprecation cycle. The use of modules in
:mod:`!compression` is encouraged where practical.
* :mod:`!compression.bz2` -- Re-exports :mod:`bz2`
* :mod:`!compression.gzip` -- Re-exports :mod:`gzip`
* :mod:`!compression.lzma` -- Re-exports :mod:`lzma`
* :mod:`!compression.zlib` -- Re-exports :mod:`zlib`
* :mod:`compression.zstd` -- Wrapper for the Zstandard compression library

840
Doc/library/compression.zstd.rst Normal file
View file

@ -0,0 +1,840 @@
:mod:`!compression.zstd` --- Compression compatible with the Zstandard format
=============================================================================
.. module:: compression.zstd
:synopsis: Low-level interface to compression and decompression routines in
the zstd library.
.. versionadded:: 3.14
**Source code:** :source:`Lib/compression/zstd/__init__.py`
--------------
This module provides classes and functions for compressing and decompressing
data using the Zstandard (or *zstd*) compression algorithm. The
`zstd manual <https://facebook.github.io/zstd/doc/api_manual_latest.html>`__
describes Zstandard as "a fast lossless compression algorithm, targeting
real-time compression scenarios at zlib-level and better compression ratios."
Also included is a file interface that supports reading and writing the
contents of ``.zst`` files created by the :program:`zstd` utility, as well as
raw zstd compressed streams.
The :mod:`!compression.zstd` module contains:
* The :func:`.open` function and :class:`ZstdFile` class for reading and
writing compressed files.
* The :class:`ZstdCompressor` and :class:`ZstdDecompressor` classes for
incremental (de)compression.
* The :func:`compress` and :func:`decompress` functions for one-shot
(de)compression.
* The :func:`train_dict` and :func:`finalize_dict` functions and the
:class:`ZstdDict` class to train and manage Zstandard dictionaries.
* The :class:`CompressionParameter`, :class:`DecompressionParameter`, and
:class:`Strategy` classes for setting advanced (de)compression parameters.
Exceptions
----------
.. exception:: ZstdError
This exception is raised when an error occurs during compression or
decompression, or while initializing the (de)compressor state.
Reading and writing compressed files
------------------------------------
.. function:: open(file, /, mode='rb', *, level=None, options=None, \
zstd_dict=None, encoding=None, errors=None, newline=None)
Open a Zstandard-compressed file in binary or text mode, returning a
:term:`file object`.
The *file* argument can be either a file name (given as a
:class:`str`, :class:`bytes` or :term:`path-like <path-like object>`
object), in which case the named file is opened, or it can be an existing
file object to read from or write to.
The mode argument can be either ``'rb'`` for reading (default), ``'wb'`` for
overwriting, ``'ab'`` for appending, or ``'xb'`` for exclusive creation.
These can equivalently be given as ``'r'``, ``'w'``, ``'a'``, and ``'x'``
respectively. You may also open in text mode with ``'rt'``, ``'wt'``,
``'at'``, and ``'xt'`` respectively.
When reading, the *options* argument can be a dictionary providing advanced
decompression parameters; see :class:`DecompressionParameter` for detailed
information about supported
parameters. The *zstd_dict* argument is a :class:`ZstdDict` instance to be
used during decompression. When reading, if the *level*
argument is not None, a :exc:`!TypeError` will be raised.
When writing, the *options* argument can be a dictionary
providing advanced decompression parameters; see
:class:`CompressionParameter` for detailed information about supported
parameters. The *level* argument is the compression level to use when
writing compressed data. Only one of *level* or *options* may be non-None.
The *zstd_dict* argument is a :class:`ZstdDict` instance to be used during
compression.
In binary mode, this function is equivalent to the :class:`ZstdFile`
constructor: ``ZstdFile(file, mode, ...)``. In this case, the
*encoding*, *errors*, and *newline* parameters must not be provided.
In text mode, a :class:`ZstdFile` object is created, and wrapped in an
:class:`io.TextIOWrapper` instance with the specified encoding, error
handling behavior, and line endings.
.. class:: ZstdFile(file, /, mode='rb', *, level=None, options=None, \
zstd_dict=None)
Open a Zstandard-compressed file in binary mode.
A :class:`ZstdFile` can wrap an already-open :term:`file object`, or operate
directly on a named file. The *file* argument specifies either the file
object to wrap, or the name of the file to open (as a :class:`str`,
:class:`bytes` or :term:`path-like <path-like object>` object). If
wrapping an existing file object, the wrapped file will not be closed when
the :class:`ZstdFile` is closed.
The *mode* argument can be either ``'rb'`` for reading (default), ``'wb'``
for overwriting, ``'xb'`` for exclusive creation, or ``'ab'`` for appending.
These can equivalently be given as ``'r'``, ``'w'``, ``'x'`` and ``'a'``
respectively.
If *file* is a file object (rather than an actual file name), a mode of
``'w'`` does not truncate the file, and is instead equivalent to ``'a'``.
When reading, the *options* argument can be a dictionary
providing advanced decompression parameters; see
:class:`DecompressionParameter` for detailed information about supported
parameters. The *zstd_dict* argument is a :class:`ZstdDict` instance to be
used during decompression. When reading, if the *level*
argument is not None, a :exc:`!TypeError` will be raised.
When writing, the *options* argument can be a dictionary
providing advanced decompression parameters; see
:class:`CompressionParameter` for detailed information about supported
parameters. The *level* argument is the compression level to use when
writing compressed data. Only one of *level* or *options* may be passed. The
*zstd_dict* argument is a :class:`ZstdDict` instance to be used during
compression.
:class:`!ZstdFile` supports all the members specified by
:class:`io.BufferedIOBase`, except for :meth:`~io.BufferedIOBase.detach`
and :meth:`~io.IOBase.truncate`.
Iteration and the :keyword:`with` statement are supported.
The following method and attributes are also provided:
.. method:: peek(size=-1)
Return buffered data without advancing the file position. At least one
byte of data will be returned, unless EOF has been reached. The exact
number of bytes returned is unspecified (the *size* argument is ignored).
.. note:: While calling :meth:`peek` does not change the file position of
the :class:`ZstdFile`, it may change the position of the underlying
file object (for example, if the :class:`ZstdFile` was constructed by
passing a file object for *file*).
.. attribute:: mode
``'rb'`` for reading and ``'wb'`` for writing.
.. attribute:: name
The name of the Zstandard file. Equivalent to the :attr:`~io.FileIO.name`
attribute of the underlying :term:`file object`.
Compressing and decompressing data in memory
--------------------------------------------
.. function:: compress(data, level=None, options=None, zstd_dict=None)
Compress *data* (a :term:`bytes-like object`), returning the compressed
data as a :class:`bytes` object.
The *level* argument is an integer controlling the level of
compression. *level* is an alternative to setting
:attr:`CompressionParameter.compression_level` in *options*. Use
:meth:`~CompressionParameter.bounds` on
:attr:`~CompressionParameter.compression_level` to get the values that can
be passed for *level*. If advanced compression options are needed, the
*level* argument must be omitted and in the *options* dictionary the
:attr:`!CompressionParameter.compression_level` parameter should be set.
The *options* argument is a Python dictionary containing advanced
compression parameters. The valid keys and values for compression parameters
are documented as part of the :class:`CompressionParameter` documentation.
The *zstd_dict* argument is an instance of :class:`ZstdDict`
containing trained data to improve compression efficiency. The
function :func:`train_dict` can be used to generate a Zstandard dictionary.
.. function:: decompress(data, zstd_dict=None, options=None)
Decompress *data* (a :term:`bytes-like object`), returning the uncompressed
data as a :class:`bytes` object.
The *options* argument is a Python dictionary containing advanced
decompression parameters. The valid keys and values for compression
parameters are documented as part of the :class:`DecompressionParameter`
documentation.
The *zstd_dict* argument is an instance of :class:`ZstdDict`
containing trained data used during compression. This must be
the same Zstandard dictionary used during compression.
If *data* is the concatenation of multiple distinct compressed frames,
decompress all of these frames, and return the concatenation of the results.
.. class:: ZstdCompressor(level=None, options=None, zstd_dict=None)
Create a compressor object, which can be used to compress data
incrementally.
For a more convenient way of compressing a single chunk of data, see the
module-level function :func:`compress`.
The *level* argument is an integer controlling the level of
compression. *level* is an alternative to setting
:attr:`CompressionParameter.compression_level` in *options*. Use
:meth:`~CompressionParameter.bounds` on
:attr:`~CompressionParameter.compression_level` to get the values that can
be passed for *level*. If advanced compression options are needed, the
*level* argument must be omitted and in the *options* dictionary the
:attr:`!CompressionParameter.compression_level` parameter should be set.
The *options* argument is a Python dictionary containing advanced
compression parameters. The valid keys and values for compression parameters
are documented as part of the :class:`CompressionParameter` documentation.
The *zstd_dict* argument is an optional instance of :class:`ZstdDict`
containing trained data to improve compression efficiency. The
function :func:`train_dict` can be used to generate a Zstandard dictionary.
.. method:: compress(data, mode=ZstdCompressor.CONTINUE)
Compress *data* (a :term:`bytes-like object`), returning a :class:`bytes`
object with compressed data if possible, or otherwise an empty
:class:`!bytes` object. Some of *data* may be buffered internally, for
use in later calls to :meth:`!compress` and :meth:`~.flush`. The returned
data should be concatenated with the output of any previous calls to
:meth:`~.compress`.
The *mode* argument is a :class:`ZstdCompressor` attribute, either
:attr:`~.CONTINUE`, :attr:`~.FLUSH_BLOCK`,
or :attr:`~.FLUSH_FRAME`.
When all data has been provided to the compressor, call the
:meth:`~.flush` method to finish the compression process. If
:meth:`~.compress` is called with *mode* set to :attr:`~.FLUSH_FRAME`,
:meth:`~.flush` should not be called, as it would write out a new empty
frame.
.. method:: flush(mode=ZstdCompressor.FLUSH_FRAME)
Finish the compression process, returning a :class:`bytes` object
containing any data stored in the compressor's internal buffers.
The *mode* argument is a :class:`ZstdCompressor` attribute, either
:attr:`~.FLUSH_BLOCK`, or :attr:`~.FLUSH_FRAME`.
.. attribute:: CONTINUE
Collect more data for compression, which may or may not generate output
immediately. This mode optimizes the compression ratio by maximizing the
amount of data per block and frame.
.. attribute:: FLUSH_BLOCK
Complete and write a block to the data stream. The data returned so far
can be immediately decompressed. Past data can still be referenced in
future blocks generated by calls to :meth:`~.compress`,
improving compression.
.. attribute:: FLUSH_FRAME
Complete and write out a frame. Future data provided to
:meth:`~.compress` will be written into a new frame and
*cannot* reference past data.
.. class:: ZstdDecompressor(zstd_dict=None, options=None)
Create a decompressor object, which can be used to decompress data
incrementally.
For a more convenient way of decompressing an entire compressed stream at
once, see the module-level function :func:`decompress`.
The *options* argument is a Python dictionary containing advanced
decompression parameters. The valid keys and values for compression
parameters are documented as part of the :class:`DecompressionParameter`
documentation.
The *zstd_dict* argument is an instance of :class:`ZstdDict`
containing trained data used during compression. This must be
the same Zstandard dictionary used during compression.
.. note::
This class does not transparently handle inputs containing multiple
compressed frames, unlike the :func:`decompress` function and
:class:`ZstdFile` class. To decompress a multi-frame input, you should
use :func:`decompress`, :class:`ZstdFile` if working with a
:term:`file object`, or multiple :class:`!ZstdDecompressor` instances.
.. method:: decompress(data, max_length=-1)
Decompress *data* (a :term:`bytes-like object`), returning
uncompressed data as bytes. Some of *data* may be buffered
internally, for use in later calls to :meth:`!decompress`.
The returned data should be concatenated with the output of any previous
calls to :meth:`!decompress`.
If *max_length* is non-negative, the method returns at most *max_length*
bytes of decompressed data. If this limit is reached and further
output can be produced, the :attr:`~.needs_input` attribute will
be set to ``False``. In this case, the next call to
:meth:`~.decompress` may provide *data* as ``b''`` to obtain
more of the output.
If all of the input data was decompressed and returned (either
because this was less than *max_length* bytes, or because
*max_length* was negative), the :attr:`~.needs_input` attribute
will be set to ``True``.
Attempting to decompress data after the end of a frame will raise a
:exc:`ZstdError`. Any data found after the end of the frame is ignored
and saved in the :attr:`~.unused_data` attribute.
.. attribute:: eof
``True`` if the end-of-stream marker has been reached.
.. attribute:: unused_data
Data found after the end of the compressed stream.
Before the end of the stream is reached, this will be ``b''``.
.. attribute:: needs_input
``False`` if the :meth:`.decompress` method can provide more
decompressed data before requiring new compressed input.
Zstandard dictionaries
----------------------
.. function:: train_dict(samples, dict_size)
Train a Zstandard dictionary, returning a :class:`ZstdDict` instance.
Zstandard dictionaries enable more efficient compression of smaller sizes
of data, which is traditionally difficult to compress due to less
repetition. If you are compressing multiple similar groups of data (such as
similar files), Zstandard dictionaries can improve compression ratios and
speed significantly.
The *samples* argument (an iterable of :class:`bytes` objects), is the
population of samples used to train the Zstandard dictionary.
The *dict_size* argument, an integer, is the maximum size (in bytes) the
Zstandard dictionary should be. The Zstandard documentation suggests an
absolute maximum of no more than 100 KB, but the maximum can often be smaller
depending on the data. Larger dictionaries generally slow down compression,
but improve compression ratios. Smaller dictionaries lead to faster
compression, but reduce the compression ratio.
.. function:: finalize_dict(zstd_dict, /, samples, dict_size, level)
An advanced function for converting a "raw content" Zstandard dictionary into
a regular Zstandard dictionary. "Raw content" dictionaries are a sequence of
bytes that do not need to follow the structure of a normal Zstandard
dictionary.
The *zstd_dict* argument is a :class:`ZstdDict` instance with
the :attr:`~ZstdDict.dict_content` containing the raw dictionary contents.
The *samples* argument (an iterable of :class:`bytes` objects), contains
sample data for generating the Zstandard dictionary.
The *dict_size* argument, an integer, is the maximum size (in bytes) the
Zstandard dictionary should be. See :func:`train_dict` for
suggestions on the maximum dictionary size.
The *level* argument (an integer) is the compression level expected to be
passed to the compressors using this dictionary. The dictionary information
varies for each compression level, so tuning for the proper compression
level can make compression more efficient.
.. class:: ZstdDict(dict_content, /, *, is_raw=False)
A wrapper around Zstandard dictionaries. Dictionaries can be used to improve
the compression of many small chunks of data. Use :func:`train_dict` if you
need to train a new dictionary from sample data.
The *dict_content* argument (a :term:`bytes-like object`), is the already
trained dictionary information.
The *is_raw* argument, a boolean, is an advanced parameter controlling the
meaning of *dict_content*. ``True`` means *dict_content* is a "raw content"
dictionary, without any format restrictions. ``False`` means *dict_content*
is an ordinary Zstandard dictionary, created from Zstandard functions,
for example, :func:`train_dict` or the external :program:`zstd` CLI.
When passing a :class:`!ZstdDict` to a function, the
:attr:`!as_digested_dict` and :attr:`!as_undigested_dict` attributes can
control how the dictionary is loaded by passing them as the ``zstd_dict``
argument, for example, ``compress(data, zstd_dict=zd.as_digested_dict)``.
Digesting a dictionary is a costly operation that occurs when loading a
Zstandard dictionary. When making multiple calls to compression or
decompression, passing a digested dictionary will reduce the overhead of
loading the dictionary.
.. list-table:: Difference for compression
:widths: 10 14 10
:header-rows: 1
* -
- Digested dictionary
- Undigested dictionary
* - Advanced parameters of the compressor which may be overridden by
the dictionary's parameters
- ``window_log``, ``hash_log``, ``chain_log``, ``search_log``,
``min_match``, ``target_length``, ``strategy``,
``enable_long_distance_matching``, ``ldm_hash_log``,
``ldm_min_match``, ``ldm_bucket_size_log``, ``ldm_hash_rate_log``,
and some non-public parameters.
- None
* - :class:`!ZstdDict` internally caches the dictionary
- Yes. It's faster when loading a digested dictionary again with the
same compression level.
- No. If you wish to load an undigested dictionary multiple times,
consider reusing a compressor object.
If passing a :class:`!ZstdDict` without any attribute, an undigested
dictionary is passed by default when compressing and a digested dictionary
is generated if necessary and passed by default when decompressing.
.. attribute:: dict_content
The content of the Zstandard dictionary, a ``bytes`` object. It's the
same as the *dict_content* argument in the ``__init__`` method. It can
be used with other programs, such as the ``zstd`` CLI program.
.. attribute:: dict_id
Identifier of the Zstandard dictionary, a non-negative int value.
Non-zero means the dictionary is ordinary, created by Zstandard
functions and following the Zstandard format.
``0`` means a "raw content" dictionary, free of any format restriction,
used for advanced users.
.. note::
The meaning of ``0`` for :attr:`!ZstdDict.dict_id` is different
from the ``dictionary_id`` attribute to the :func:`get_frame_info`
function.
.. attribute:: as_digested_dict
Load as a digested dictionary.
.. attribute:: as_undigested_dict
Load as an undigested dictionary.
Advanced parameter control
--------------------------
.. class:: CompressionParameter()
An :class:`~enum.IntEnum` containing the advanced compression parameter
keys that can be used when compressing data.
The :meth:`~.bounds` method can be used on any attribute to get the valid
values for that parameter.
Parameters are optional; any omitted parameter will have it's value selected
automatically.
Example getting the lower and upper bound of :attr:`~.compression_level`::
lower, upper = CompressionParameter.compression_level.bounds()
Example setting the :attr:`~.window_log` to the maximum size::
_lower, upper = CompressionParameter.window_log.bounds()
options = {CompressionParameter.window_log: upper}
compress(b'venezuelan beaver cheese', options=options)
.. method:: bounds()
Return the tuple of int bounds, ``(lower, upper)``, of a compression
parameter. This method should be called on the attribute you wish to
retrieve the bounds of. For example, to get the valid values for
:attr:`~.compression_level`, one may check the result of
``CompressionParameter.compression_level.bounds()``.
Both the lower and upper bounds are inclusive.
.. attribute:: compression_level
A high-level means of setting other compression parameters that affect
the speed and ratio of compressing data. Setting the level to zero uses
:attr:`COMPRESSION_LEVEL_DEFAULT`.
.. attribute:: window_log
Maximum allowed back-reference distance the compressor can use when
compressing data, expressed as power of two, ``1 << window_log`` bytes.
This parameter greatly influences the memory usage of compression. Higher
values require more memory but gain better compression values.
A value of zero causes the value to be selected automatically.
.. attribute:: hash_log
Size of the initial probe table, as a power of two. The resulting memory
usage is ``1 << (hash_log+2)`` bytes. Larger tables improve compression
ratio of strategies <= :attr:`~Strategy.dfast`, and improve compression
speed of strategies > :attr:`~Strategy.dfast`.
A value of zero causes the value to be selected automatically.
.. attribute:: chain_log
Size of the multi-probe search table, as a power of two. The resulting
memory usage is ``1 << (chain_log+2)`` bytes. Larger tables result in
better and slower compression. This parameter has no effect for the
:attr:`~Strategy.fast` strategy. It's still useful when using
:attr:`~Strategy.dfast` strategy, in which case it defines a secondary
probe table.
A value of zero causes the value to be selected automatically.
.. attribute:: search_log
Number of search attempts, as a power of two. More attempts result in
better and slower compression. This parameter is useless for
:attr:`~Strategy.fast` and :attr:`~Strategy.dfast` strategies.
A value of zero causes the value to be selected automatically.
.. attribute:: min_match
Minimum size of searched matches. Larger values increase compression and
decompression speed, but decrease ratio. Note that Zstandard can still
find matches of smaller size, it just tweaks its search algorithm to look
for this size and larger. For all strategies < :attr:`~Strategy.btopt`,
the effective minimum is ``4``; for all strategies
> :attr:`~Strategy.fast`, the effective maximum is ``6``.
A value of zero causes the value to be selected automatically.
.. attribute:: target_length
The impact of this field depends on the selected :class:`Strategy`.
For strategies :attr:`~Strategy.btopt`, :attr:`~Strategy.btultra` and
:attr:`~Strategy.btultra2`, the value is the length of a match
considered "good enough" to stop searching. Larger values make
compression ratios better, but compresses slower.
For strategy :attr:`~Strategy.fast`, it is the distance between match
sampling. Larger values make compression faster, but with a worse
compression ratio.
A value of zero causes the value to be selected automatically.
.. attribute:: strategy
The higher the value of selected strategy, the more complex the
compression technique used by zstd, resulting in higher compression
ratios but slower compression.
.. seealso:: :class:`Strategy`
.. attribute:: enable_long_distance_matching
Long distance matching can be used to improve compression for large
inputs by finding large matches at greater distances. It increases memory
usage and window size.
``True`` or ``1`` enable long distance matching while ``False`` or ``0``
disable it.
Enabling this parameter increases default
:attr:`~CompressionParameter.window_log` to 128 MiB except when expressly
set to a different value. This setting is enabled by default if
:attr:`!window_log` >= 128 MiB and the compression
strategy >= :attr:`~Strategy.btopt` (compression level 16+).
.. attribute:: ldm_hash_log
Size of the table for long distance matching, as a power of two. Larger
values increase memory usage and compression ratio, but decrease
compression speed.
A value of zero causes the value to be selected automatically.
.. attribute:: ldm_min_match
Minimum match size for long distance matcher. Larger or too small values
can often decrease the compression ratio.
A value of zero causes the value to be selected automatically.
.. attribute:: ldm_bucket_size_log
Log size of each bucket in the long distance matcher hash table for
collision resolution. Larger values improve collision resolution but
decrease compression speed.
A value of zero causes the value to be selected automatically.
.. attribute:: ldm_hash_rate_log
Frequency of inserting/looking up entries into the long distance matcher
hash table. Larger values improve compression speed. Deviating far from
the default value will likely result in a compression ratio decrease.
A value of zero causes the value to be selected automatically.
.. attribute:: checksum_flag
A four-byte checksum using XXHash64 of the uncompressed content is
written at the end of each frame. Zstandard's decompression code verifies
the checksum. If there is a mismatch a :class:`ZstdError` exception is
raised.
``True`` or ``1`` enable checksum generation while ``False`` or ``0``
disable it.
.. attribute:: dict_id_flag
When compressing with a :class:`ZstdDict`, the dictionary's ID is written
into the frame header.
``True`` or ``1`` enable storing the dictionary ID while ``False`` or
``0`` disable it.
.. attribute:: nb_workers
Select how many threads will be spawned to compress in parallel. When
:attr:`!nb_workers` > 0, enables multi-threaded compression, a value of
``1`` means "one-thread multi-threaded mode". More workers improve speed,
but also increase memory usage and slightly reduce compression ratio.
A value of zero disables multi-threading.
.. attribute:: job_size
Size of a compression job, in bytes. This value is enforced only when
:attr:`~CompressionParameter.nb_workers` >= 1. Each compression job is
completed in parallel, so this value can indirectly impact the number of
active threads.
A value of zero causes the value to be selected automatically.
.. attribute:: overlap_log
Sets how much data is reloaded from previous jobs (threads) for new jobs
to be used by the look behind window during compression. This value is
only used when :attr:`~CompressionParameter.nb_workers` >= 1. Acceptable
values vary from 0 to 9.
* 0 means dynamically set the overlap amount
* 1 means no overlap
* 9 means use a full window size from the previous job
Each increment halves/doubles the overlap size. "8" means an overlap of
``window_size/2``, "7" means an overlap of ``window_size/4``, etc.
.. class:: DecompressionParameter()
An :class:`~enum.IntEnum` containing the advanced decompression parameter
keys that can be used when decompressing data. Parameters are optional; any
omitted parameter will have it's value selected automatically.
The :meth:`~.bounds` method can be used on any attribute to get the valid
values for that parameter.
Example setting the :attr:`~.window_log_max` to the maximum size::
data = compress(b'Some very long buffer of bytes...')
_lower, upper = DecompressionParameter.window_log_max.bounds()
options = {DecompressionParameter.window_log_max: upper}
decompress(data, options=options)
.. method:: bounds()
Return the tuple of int bounds, ``(lower, upper)``, of a decompression
parameter. This method should be called on the attribute you wish to
retrieve the bounds of.
Both the lower and upper bounds are inclusive.
.. attribute:: window_log_max
The base-two logarithm of the maximum size of the window used during
decompression. This can be useful to limit the amount of memory used when
decompressing data. A larger maximum window size leads to faster
decompression.
A value of zero causes the value to be selected automatically.
.. class:: Strategy()
An :class:`~enum.IntEnum` containing strategies for compression.
Higher-numbered strategies correspond to more complex and slower
compression.
.. note::
The values of attributes of :class:`!Strategy` are not necessarily stable
across zstd versions. Only the ordering of the attributes may be relied
upon. The attributes are listed below in order.
The following strategies are available:
.. attribute:: fast
.. attribute:: dfast
.. attribute:: greedy
.. attribute:: lazy
.. attribute:: lazy2
.. attribute:: btlazy2
.. attribute:: btopt
.. attribute:: btultra
.. attribute:: btultra2
Miscellaneous
-------------
.. function:: get_frame_info(frame_buffer)
Retrieve a :class:`FrameInfo` object containing metadata about a Zstandard
frame. Frames contain metadata related to the compressed data they hold.
.. class:: FrameInfo
Metadata related to a Zstandard frame.
.. attribute:: decompressed_size
The size of the decompressed contents of the frame.
.. attribute:: dictionary_id
An integer representing the Zstandard dictionary ID needed for
decompressing the frame. ``0`` means the dictionary ID was not
recorded in the frame header. This may mean that a Zstandard dictionary
is not needed, or that the ID of a required dictionary was not recorded.
.. attribute:: COMPRESSION_LEVEL_DEFAULT
The default compression level for Zstandard: ``3``.
.. attribute:: zstd_version_info
Version number of the runtime zstd library as a tuple of integers
(major, minor, release).
Examples
--------
Reading in a compressed file:
.. code-block:: python
from compression import zstd
with zstd.open("file.zst") as f:
file_content = f.read()
Creating a compressed file:
.. code-block:: python
from compression import zstd
data = b"Insert Data Here"
with zstd.open("file.zst", "w") as f:
f.write(data)
Compressing data in memory:
.. code-block:: python
from compression import zstd
data_in = b"Insert Data Here"
data_out = zstd.compress(data_in)
Incremental compression:
.. code-block:: python
from compression import zstd
comp = zstd.ZstdCompressor()
out1 = comp.compress(b"Some data\n")
out2 = comp.compress(b"Another piece of data\n")
out3 = comp.compress(b"Even more data\n")
out4 = comp.flush()
# Concatenate all the partial results:
result = b"".join([out1, out2, out3, out4])
Writing compressed data to an already-open file:
.. code-block:: python
from compression import zstd
with open("myfile", "wb") as f:
f.write(b"This data will not be compressed\n")
with zstd.open(f, "w") as zstf:
zstf.write(b"This *will* be compressed\n")
f.write(b"Not compressed\n")
Creating a compressed file using compression parameters:
.. code-block:: python
from compression import zstd
options = {
zstd.CompressionParameter.checksum_flag: 1
}
with zstd.open("file.zst", "w", options=options) as f:
f.write(b"Mind if I squeeze in?")

5
Doc/library/concurrent.futures.rst
View file

@ -6,8 +6,9 @@
.. versionadded:: 3.2
**Source code:** :source:`Lib/concurrent/futures/thread.py`
and :source:`Lib/concurrent/futures/process.py`
**Source code:** :source:`Lib/concurrent/futures/thread.py`,
:source:`Lib/concurrent/futures/process.py`,
and :source:`Lib/concurrent/futures/interpreter.py`
--------------

40
Doc/library/ctypes.rst
View file

@ -1779,7 +1779,8 @@ in :mod:`!ctypes`) which inherits from the private :class:`_CFuncPtr` class:
.. audit-event:: ctypes.call_function func_pointer,arguments foreign-functions
Some ways to invoke foreign function calls may raise an auditing event
Some ways to invoke foreign function calls as well as some of the
functions in this module may raise an auditing event
``ctypes.call_function`` with arguments ``function pointer`` and ``arguments``.
.. _ctypes-function-prototypes:
@ -2171,10 +2172,20 @@ Utility functions
.. function:: POINTER(type, /)
Create and return a new ctypes pointer type. Pointer types are cached and
Create or return a ctypes pointer type. Pointer types are cached and
reused internally, so calling this function repeatedly is cheap.
*type* must be a ctypes type.
.. impl-detail::
The resulting pointer type is cached in the ``__pointer_type__``
attribute of *type*.
It is possible to set this attribute before the first call to
``POINTER`` in order to set a custom pointer type.
However, doing this is discouraged: manually creating a suitable
pointer type is difficult without relying on implementation
details that may change in future Python versions.
.. function:: pointer(obj, /)
@ -2339,6 +2350,16 @@ Data types
library. *name* is the name of the symbol that exports the data, *library*
is the loaded shared library.
Common class variables of ctypes data types:
.. attribute:: __pointer_type__
The pointer type that was created by calling
:func:`POINTER` for corresponding ctypes data type. If a pointer type
was not yet created, the attribute is missing.
.. versionadded:: 3.14
Common instance variables of ctypes data types:
.. attribute:: _b_base_
@ -2733,6 +2754,16 @@ fields, or any other data types containing pointer type fields.
when :attr:`_fields_` is assigned, otherwise it will have no effect.
Setting this attribute to 0 is the same as not setting it at all.
This is only implemented for the MSVC-compatible memory layout.
.. deprecated-removed:: 3.14 3.19
For historical reasons, if :attr:`!_pack_` is non-zero,
the MSVC-compatible layout will be used by default.
On non-Windows platforms, this default is deprecated and is slated to
become an error in Python 3.19.
If it is intended, set :attr:`~Structure._layout_` to ``'ms'``
explicitly.
.. attribute:: _align_
@ -2761,12 +2792,15 @@ fields, or any other data types containing pointer type fields.
Currently the default will be:
- On Windows: ``"ms"``
- When :attr:`~Structure._pack_` is specified: ``"ms"``
- When :attr:`~Structure._pack_` is specified: ``"ms"``.
(This is deprecated; see :attr:`~Structure._pack_` documentation.)
- Otherwise: ``"gcc-sysv"``
:attr:`!_layout_` must already be defined when
:attr:`~Structure._fields_` is assigned, otherwise it will have no effect.
.. versionadded:: 3.14
.. attribute:: _anonymous_
An optional sequence that lists the names of unnamed (anonymous) fields.

37
Doc/library/curses.rst
View file

@ -68,6 +68,21 @@ The module :mod:`curses` defines the following exception:
The module :mod:`curses` defines the following functions:
.. function:: assume_default_colors(fg, bg, /)
Allow use of default values for colors on terminals supporting this feature.
Use this to support transparency in your application.
* Assign terminal default foreground/background colors to color number ``-1``.
So ``init_pair(x, COLOR_RED, -1)`` will initialize pair *x* as red
on default background and ``init_pair(x, -1, COLOR_BLUE)`` will
initialize pair *x* as default foreground on blue.
* Change the definition of the color-pair ``0`` to ``(fg, bg)``.
.. versionadded:: 3.14
.. function:: baudrate()
Return the output speed of the terminal in bits per second. On software
@ -290,9 +305,11 @@ The module :mod:`curses` defines the following functions:
Change the definition of a color-pair. It takes three arguments: the number of
the color-pair to be changed, the foreground color number, and the background
color number. The value of *pair_number* must be between ``1`` and
``COLOR_PAIRS - 1`` (the ``0`` color pair is wired to white on black and cannot
be changed). The value of *fg* and *bg* arguments must be between ``0`` and
``COLORS - 1``, or, after calling :func:`use_default_colors`, ``-1``.
``COLOR_PAIRS - 1`` (the ``0`` color pair can only be changed by
:func:`use_default_colors` and :func:`assume_default_colors`).
The value of *fg* and *bg* arguments must be between ``0`` and
``COLORS - 1``, or, after calling :func:`!use_default_colors` or
:func:`!assume_default_colors`, ``-1``.
If the color-pair was previously initialized, the screen is
refreshed and all occurrences of that color-pair are changed to the new
definition.
@ -678,11 +695,7 @@ The module :mod:`curses` defines the following functions:
.. function:: use_default_colors()
Allow use of default values for colors on terminals supporting this feature. Use
this to support transparency in your application. The default color is assigned
to the color number ``-1``. After calling this function, ``init_pair(x,
curses.COLOR_RED, -1)`` initializes, for instance, color pair *x* to a red
foreground color on the default background.
Equivalent to ``assume_default_colors(-1, -1)``.
.. function:: wrapper(func, /, *args, **kwargs)
@ -975,6 +988,10 @@ the following methods and attributes:
window.getstr(y, x, n)
Read a bytes object from the user, with primitive line editing capacity.
The maximum value for *n* is 2047.
.. versionchanged:: 3.14
The maximum value for *n* was increased from 1023 to 2047.
.. method:: window.getyx()
@ -1066,6 +1083,10 @@ the following methods and attributes:
current cursor position, or at *y*, *x* if specified. Attributes are stripped
from the characters. If *n* is specified, :meth:`instr` returns a string
at most *n* characters long (exclusive of the trailing NUL).
The maximum value for *n* is 2047.
.. versionchanged:: 3.14
The maximum value for *n* was increased from 1023 to 2047.
.. method:: window.is_linetouched(line)

17
Doc/library/dataclasses.rst
View file

@ -304,9 +304,9 @@ Module contents
.. versionadded:: 3.10
- ``doc``: optional docstring for this field.
- *doc*: optional docstring for this field.
.. versionadded:: 3.13
.. versionadded:: 3.14
If the default value of a field is specified by a call to
:func:`!field`, then the class attribute for this field will be
@ -344,6 +344,15 @@ Module contents
Other attributes may exist, but they are private and must not be
inspected or relied on.
.. class:: InitVar
``InitVar[T]`` type annotations describe variables that are :ref:`init-only
<dataclasses-init-only-variables>`. Fields annotated with :class:`!InitVar`
are considered pseudo-fields, and thus are neither returned by the
:func:`fields` function nor used in any way except adding them as
parameters to :meth:`~object.__init__` and an optional
:meth:`__post_init__`.
.. function:: fields(class_or_instance)
Returns a tuple of :class:`Field` objects that define the fields for this
@ -600,8 +609,8 @@ Init-only variables
Another place where :func:`@dataclass <dataclass>` inspects a type annotation is to
determine if a field is an init-only variable. It does this by seeing
if the type of a field is of type ``dataclasses.InitVar``. If a field
is an ``InitVar``, it is considered a pseudo-field called an init-only
if the type of a field is of type :class:`InitVar`. If a field
is an :class:`InitVar`, it is considered a pseudo-field called an init-only
field. As it is not a true field, it is not returned by the
module-level :func:`fields` function. Init-only fields are added as
parameters to the generated :meth:`~object.__init__` method, and are passed to

30
Doc/library/datetime.rst
View file

@ -261,6 +261,22 @@ A :class:`timedelta` object represents a duration, the difference between two
>>> (d.days, d.seconds, d.microseconds)
(-1, 86399, 999999)
Since the string representation of :class:`!timedelta` objects can be confusing,
use the following recipe to produce a more readable format:
.. code-block:: pycon
>>> def pretty_timedelta(td):
... if td.days >= 0:
... return str(td)
... return f'-({-td!s})'
...
>>> d = timedelta(hours=-1)
>>> str(d) # not human-friendly
'-1 day, 23:00:00'
>>> pretty_timedelta(d)
'-(1:00:00)'
Class attributes:
@ -1486,11 +1502,11 @@ Instance methods:
returned by :func:`time.time`.
Naive :class:`.datetime` instances are assumed to represent local
time and this method relies on the platform C :c:func:`mktime`
function to perform the conversion. Since :class:`.datetime`
supports wider range of values than :c:func:`mktime` on many
platforms, this method may raise :exc:`OverflowError` or :exc:`OSError`
for times far in the past or far in the future.
time and this method relies on platform C functions to perform
the conversion. Since :class:`.datetime` supports a wider range of
values than the platform C functions on many platforms, this
method may raise :exc:`OverflowError` or :exc:`OSError` for times
far in the past or far in the future.
For aware :class:`.datetime` instances, the return value is computed
as::
@ -1503,6 +1519,10 @@ Instance methods:
The :meth:`timestamp` method uses the :attr:`.fold` attribute to
disambiguate the times during a repeated interval.
.. versionchanged:: 3.6
This method no longer relies on the platform C :c:func:`mktime`
function to perform conversions.
.. note::
There is no method to obtain the POSIX timestamp directly from a

107
Doc/library/decimal.rst
View file

@ -2,7 +2,7 @@
=====================================================================
.. module:: decimal
:synopsis: Implementation of the General Decimal Arithmetic Specification.
:synopsis: Implementation of the General Decimal Arithmetic Specification.
.. moduleauthor:: Eric Price <eprice at tjhsst.edu>
.. moduleauthor:: Facundo Batista <facundo at taniquetil.com.ar>
@ -121,7 +121,7 @@ reset them before monitoring a calculation.
.. _decimal-tutorial:
Quick-start Tutorial
Quick-start tutorial
--------------------
The usual start to using decimals is importing the module, viewing the current
@ -367,6 +367,8 @@ Decimal objects
appears above. These include decimal digits from various other
alphabets (for example, Arabic-Indic and Devanāgarī digits) along
with the fullwidth digits ``'\uff10'`` through ``'\uff19'``.
Case is not significant, so, for example, ``inf``, ``Inf``, ``INFINITY``,
and ``iNfINity`` are all acceptable spellings for positive infinity.
If *value* is a :class:`tuple`, it should have three components, a sign
(``0`` for positive or ``1`` for negative), a :class:`tuple` of
@ -1029,6 +1031,14 @@ function to temporarily change the active context.
.. versionchanged:: 3.11
:meth:`localcontext` now supports setting context attributes through the use of keyword arguments.
.. function:: IEEEContext(bits)
Return a context object initialized to the proper values for one of the
IEEE interchange formats. The argument must be a multiple of 32 and less
than :const:`IEEE_CONTEXT_MAX_BITS`.
.. versionadded:: 3.14
New contexts can also be created using the :class:`Context` constructor
described below. In addition, the module provides three pre-made contexts:
@ -1086,40 +1096,52 @@ In addition to the three supplied contexts, new contexts can be created with the
default values are copied from the :const:`DefaultContext`. If the *flags*
field is not specified or is :const:`None`, all flags are cleared.
*prec* is an integer in the range [``1``, :const:`MAX_PREC`] that sets
the precision for arithmetic operations in the context.
.. attribute:: prec
The *rounding* option is one of the constants listed in the section
`Rounding Modes`_.
An integer in the range [``1``, :const:`MAX_PREC`] that sets
the precision for arithmetic operations in the context.
The *traps* and *flags* fields list any signals to be set. Generally, new
contexts should only set traps and leave the flags clear.
.. attribute:: rounding
The *Emin* and *Emax* fields are integers specifying the outer limits allowable
for exponents. *Emin* must be in the range [:const:`MIN_EMIN`, ``0``],
*Emax* in the range [``0``, :const:`MAX_EMAX`].
One of the constants listed in the section `Rounding Modes`_.
The *capitals* field is either ``0`` or ``1`` (the default). If set to
``1``, exponents are printed with a capital ``E``; otherwise, a
lowercase ``e`` is used: ``Decimal('6.02e+23')``.
.. attribute:: traps
flags
The *clamp* field is either ``0`` (the default) or ``1``.
If set to ``1``, the exponent ``e`` of a :class:`Decimal`
instance representable in this context is strictly limited to the
range ``Emin - prec + 1 <= e <= Emax - prec + 1``. If *clamp* is
``0`` then a weaker condition holds: the adjusted exponent of
the :class:`Decimal` instance is at most :attr:`~Context.Emax`. When *clamp* is
``1``, a large normal number will, where possible, have its
exponent reduced and a corresponding number of zeros added to its
coefficient, in order to fit the exponent constraints; this
preserves the value of the number but loses information about
significant trailing zeros. For example::
Lists of any signals to be set. Generally, new contexts should only set
traps and leave the flags clear.
>>> Context(prec=6, Emax=999, clamp=1).create_decimal('1.23e999')
Decimal('1.23000E+999')
.. attribute:: Emin
Emax
A *clamp* value of ``1`` allows compatibility with the
fixed-width decimal interchange formats specified in IEEE 754.
Integers specifying the outer limits allowable for exponents. *Emin* must
be in the range [:const:`MIN_EMIN`, ``0``], *Emax* in the range
[``0``, :const:`MAX_EMAX`].
.. attribute:: capitals
Either ``0`` or ``1`` (the default). If set to
``1``, exponents are printed with a capital ``E``; otherwise, a
lowercase ``e`` is used: ``Decimal('6.02e+23')``.
.. attribute:: clamp
Either ``0`` (the default) or ``1``. If set to ``1``, the exponent ``e``
of a :class:`Decimal` instance representable in this context is strictly
limited to the range ``Emin - prec + 1 <= e <= Emax - prec + 1``.
If *clamp* is ``0`` then a weaker condition holds: the adjusted exponent of
the :class:`Decimal` instance is at most :attr:`~Context.Emax`. When *clamp* is
``1``, a large normal number will, where possible, have its
exponent reduced and a corresponding number of zeros added to its
coefficient, in order to fit the exponent constraints; this
preserves the value of the number but loses information about
significant trailing zeros. For example::
>>> Context(prec=6, Emax=999, clamp=1).create_decimal('1.23e999')
Decimal('1.23000E+999')
A *clamp* value of ``1`` allows compatibility with the
fixed-width decimal interchange formats specified in IEEE 754.
The :class:`Context` class defines several general purpose methods as well as
a large number of methods for doing arithmetic directly in a given context.
@ -1550,18 +1572,19 @@ Constants
The constants in this section are only relevant for the C module. They
are also included in the pure Python version for compatibility.
+---------------------+---------------------+-------------------------------+
| | 32-bit | 64-bit |
+=====================+=====================+===============================+
| .. data:: MAX_PREC | ``425000000`` | ``999999999999999999`` |
+---------------------+---------------------+-------------------------------+
| .. data:: MAX_EMAX | ``425000000`` | ``999999999999999999`` |
+---------------------+---------------------+-------------------------------+
| .. data:: MIN_EMIN | ``-425000000`` | ``-999999999999999999`` |
+---------------------+---------------------+-------------------------------+
| .. data:: MIN_ETINY | ``-849999999`` | ``-1999999999999999997`` |
+---------------------+---------------------+-------------------------------+
+---------------------------------+---------------------+-------------------------------+
| | 32-bit | 64-bit |
+=================================+=====================+===============================+
| .. data:: MAX_PREC | ``425000000`` | ``999999999999999999`` |
+---------------------------------+---------------------+-------------------------------+
| .. data:: MAX_EMAX | ``425000000`` | ``999999999999999999`` |
+---------------------------------+---------------------+-------------------------------+
| .. data:: MIN_EMIN | ``-425000000`` | ``-999999999999999999`` |
+---------------------------------+---------------------+-------------------------------+
| .. data:: MIN_ETINY | ``-849999999`` | ``-1999999999999999997`` |
+---------------------------------+---------------------+-------------------------------+
| .. data:: IEEE_CONTEXT_MAX_BITS | ``256`` | ``512`` |
+---------------------------------+---------------------+-------------------------------+
.. data:: HAVE_THREADS
@ -1758,7 +1781,7 @@ The following table summarizes the hierarchy of signals::
.. _decimal-notes:
Floating-Point Notes
Floating-point notes
--------------------

12
Doc/library/dis.rst
View file

@ -1354,9 +1354,6 @@ iterations of the loop.
If ``STACK[-1]`` is not ``None``, increments the bytecode counter by *delta*.
``STACK[-1]`` is popped.
This opcode is a pseudo-instruction, replaced in final bytecode by
the directed versions (forward/backward).
.. versionadded:: 3.11
.. versionchanged:: 3.12
@ -1368,9 +1365,6 @@ iterations of the loop.
If ``STACK[-1]`` is ``None``, increments the bytecode counter by *delta*.
``STACK[-1]`` is popped.
This opcode is a pseudo-instruction, replaced in final bytecode by
the directed versions (forward/backward).
.. versionadded:: 3.11
.. versionchanged:: 3.12
@ -1673,7 +1667,7 @@ iterations of the loop.
* ``oparg == 2``: call :func:`repr` on *value*
* ``oparg == 3``: call :func:`ascii` on *value*
Used for implementing formatted literal strings (f-strings).
Used for implementing formatted string literals (f-strings).
.. versionadded:: 3.13
@ -1686,7 +1680,7 @@ iterations of the loop.
result = value.__format__("")
STACK.append(result)
Used for implementing formatted literal strings (f-strings).
Used for implementing formatted string literals (f-strings).
.. versionadded:: 3.13
@ -1699,7 +1693,7 @@ iterations of the loop.
result = value.__format__(spec)
STACK.append(result)
Used for implementing formatted literal strings (f-strings).
Used for implementing formatted string literals (f-strings).
.. versionadded:: 3.13

4
Doc/library/exceptions.rst
View file

@ -428,6 +428,7 @@ The following exceptions are the exceptions that are usually raised.
:exc:`PythonFinalizationError` during the Python finalization:
* Creating a new Python thread.
* :meth:`Joining <threading.Thread.join>` a running daemon thread.
* :func:`os.fork`.
See also the :func:`sys.is_finalizing` function.
@ -435,6 +436,9 @@ The following exceptions are the exceptions that are usually raised.
.. versionadded:: 3.13
Previously, a plain :exc:`RuntimeError` was raised.
.. versionchanged:: 3.14
:meth:`threading.Thread.join` can now raise this exception.
.. exception:: RecursionError

40
Doc/library/faulthandler.rst
View file

@ -66,10 +66,41 @@ Dumping the traceback
Added support for passing file descriptor to this function.
Dumping the C stack
-------------------
.. versionadded:: 3.14
.. function:: dump_c_stack(file=sys.stderr)
Dump the C stack trace of the current thread into *file*.
If the Python build does not support it or the operating system
does not provide a stack trace, then this prints an error in place
of a dumped C stack.
.. _c-stack-compatibility:
C Stack Compatibility
*********************
If the system does not support the C-level :manpage:`backtrace(3)`
or :manpage:`dladdr1(3)`, then C stack dumps will not work.
An error will be printed instead of the stack.
Additionally, some compilers do not support :term:`CPython's <CPython>`
implementation of C stack dumps. As a result, a different error may be printed
instead of the stack, even if the the operating system supports dumping stacks.
.. note::
Dumping C stacks can be arbitrarily slow, depending on the DWARF level
of the binaries in the call stack.
Fault handler state
-------------------
.. function:: enable(file=sys.stderr, all_threads=True)
.. function:: enable(file=sys.stderr, all_threads=True, c_stack=True)
Enable the fault handler: install handlers for the :const:`~signal.SIGSEGV`,
:const:`~signal.SIGFPE`, :const:`~signal.SIGABRT`, :const:`~signal.SIGBUS`
@ -81,6 +112,10 @@ Fault handler state
The *file* must be kept open until the fault handler is disabled: see
:ref:`issue with file descriptors <faulthandler-fd>`.
If *c_stack* is ``True``, then the C stack trace is printed after the Python
traceback, unless the system does not support it. See :func:`dump_c_stack` for
more information on compatibility.
.. versionchanged:: 3.5
Added support for passing file descriptor to this function.
@ -95,6 +130,9 @@ Fault handler state
Only the current thread is dumped if the :term:`GIL` is disabled to
prevent the risk of data races.
.. versionchanged:: 3.14
The dump now displays the C stack trace if *c_stack* is true.
.. function:: disable()
Disable the fault handler: uninstall the signal handlers installed by

92
Doc/library/fcntl.rst
View file

@ -79,68 +79,95 @@ descriptor.
On macOS and NetBSD, the :mod:`!fcntl` module exposes the ``F_GETNOSIGPIPE``
and ``F_SETNOSIGPIPE`` constant.
.. versionchanged:: 3.14
On Linux >= 6.1, the :mod:`!fcntl` module exposes the ``F_DUPFD_QUERY``
to query a file descriptor pointing to the same file.
The module defines the following functions:
.. function:: fcntl(fd, cmd, arg=0)
.. function:: fcntl(fd, cmd, arg=0, /)
Perform the operation *cmd* on file descriptor *fd* (file objects providing
a :meth:`~io.IOBase.fileno` method are accepted as well). The values used
for *cmd* are operating system dependent, and are available as constants
in the :mod:`fcntl` module, using the same names as used in the relevant C
header files. The argument *arg* can either be an integer value, or a
:class:`bytes` object. With an integer value, the return value of this
function is the integer return value of the C :c:func:`fcntl` call. When
the argument is bytes it represents a binary structure, e.g. created by
:func:`struct.pack`. The binary data is copied to a buffer whose address is
header files. The argument *arg* can either be an integer value, a
:term:`bytes-like object`, or a string.
The type and size of *arg* must match the type and size of
the argument of the operation as specified in the relevant C documentation.
When *arg* is an integer, the function returns the integer
return value of the C :c:func:`fcntl` call.
When the argument is bytes-like object, it represents a binary structure,
for example, created by :func:`struct.pack`.
A string value is encoded to binary using the UTF-8 encoding.
The binary data is copied to a buffer whose address is
passed to the C :c:func:`fcntl` call. The return value after a successful
call is the contents of the buffer, converted to a :class:`bytes` object.
The length of the returned object will be the same as the length of the
*arg* argument. This is limited to 1024 bytes. If the information returned
in the buffer by the operating system is larger than 1024 bytes, this is
most likely to result in a segmentation violation or a more subtle data
corruption.
*arg* argument.
If the :c:func:`fcntl` call fails, an :exc:`OSError` is raised.
.. note::
If the type or size of *arg* does not match the type or size
of the operation's argument (for example, if an integer is
passed when a pointer is expected, or the information returned in
the buffer by the operating system is larger than the size of *arg*),
this is most likely to result in a segmentation violation or
a more subtle data corruption.
.. audit-event:: fcntl.fcntl fd,cmd,arg fcntl.fcntl
.. versionchanged:: 3.14
Add support of arbitrary :term:`bytes-like objects <bytes-like object>`,
not only :class:`bytes`.
.. function:: ioctl(fd, request, arg=0, mutate_flag=True)
.. versionchanged:: next
The size of bytes-like objects is no longer limited to 1024 bytes.
.. function:: ioctl(fd, request, arg=0, mutate_flag=True, /)
This function is identical to the :func:`~fcntl.fcntl` function, except
that the argument handling is even more complicated.
The *request* parameter is limited to values that can fit in 32-bits.
The *request* parameter is limited to values that can fit in 32-bits
or 64-bits, depending on the platform.
Additional constants of interest for use as the *request* argument can be
found in the :mod:`termios` module, under the same names as used in
the relevant C header files.
The parameter *arg* can be one of an integer, an object supporting the
read-only buffer interface (like :class:`bytes`) or an object supporting
the read-write buffer interface (like :class:`bytearray`).
The parameter *arg* can be an integer, a :term:`bytes-like object`,
or a string.
The type and size of *arg* must match the type and size of
the argument of the operation as specified in the relevant C documentation.
In all but the last case, behaviour is as for the :func:`~fcntl.fcntl`
If *arg* does not support the read-write buffer interface or
the *mutate_flag* is false, behavior is as for the :func:`~fcntl.fcntl`
function.
If a mutable buffer is passed, then the behaviour is determined by the value of
the *mutate_flag* parameter.
If it is false, the buffer's mutability is ignored and behaviour is as for a
read-only buffer, except that the 1024 byte limit mentioned above is avoided --
so long as the buffer you pass is at least as long as what the operating system
wants to put there, things should work.
If *mutate_flag* is true (the default), then the buffer is (in effect) passed
to the underlying :func:`ioctl` system call, the latter's return code is
If *arg* supports the read-write buffer interface (like :class:`bytearray`)
and *mutate_flag* is true (the default), then the buffer is (in effect) passed
to the underlying :c:func:`!ioctl` system call, the latter's return code is
passed back to the calling Python, and the buffer's new contents reflect the
action of the :func:`ioctl`. This is a slight simplification, because if the
action of the :c:func:`ioctl`. This is a slight simplification, because if the
supplied buffer is less than 1024 bytes long it is first copied into a static
buffer 1024 bytes long which is then passed to :func:`ioctl` and copied back
into the supplied buffer.
If the :c:func:`ioctl` call fails, an :exc:`OSError` exception is raised.
.. note::
If the type or size of *arg* does not match the type or size
of the operation's argument (for example, if an integer is
passed when a pointer is expected, or the information returned in
the buffer by the operating system is larger than the size of *arg*),
this is most likely to result in a segmentation violation or
a more subtle data corruption.
An example::
>>> import array, fcntl, struct, termios, os
@ -156,8 +183,15 @@ The module defines the following functions:
.. audit-event:: fcntl.ioctl fd,request,arg fcntl.ioctl
.. versionchanged:: 3.14
The GIL is always released during a system call.
System calls failing with EINTR are automatically retried.
.. function:: flock(fd, operation)
.. versionchanged:: next
The size of not mutated bytes-like objects is no longer
limited to 1024 bytes.
.. function:: flock(fd, operation, /)
Perform the lock operation *operation* on file descriptor *fd* (file objects providing
a :meth:`~io.IOBase.fileno` method are accepted as well). See the Unix manual
@ -169,7 +203,7 @@ The module defines the following functions:
.. audit-event:: fcntl.flock fd,operation fcntl.flock
.. function:: lockf(fd, cmd, len=0, start=0, whence=0)
.. function:: lockf(fd, cmd, len=0, start=0, whence=0, /)
This is essentially a wrapper around the :func:`~fcntl.fcntl` locking calls.
*fd* is the file descriptor (file objects providing a :meth:`~io.IOBase.fileno`

2
Doc/library/fnmatch.rst
View file

@ -111,7 +111,7 @@ functions: :func:`fnmatch`, :func:`fnmatchcase`, :func:`.filter`.
>>>
>>> regex = fnmatch.translate('*.txt')
>>> regex
'(?s:.*\\.txt)\\Z'
'(?s:.*\\.txt)\\z'
>>> reobj = re.compile(regex)
>>> reobj.match('foobar.txt')
<re.Match object; span=(0, 10), match='foobar.txt'>

62
Doc/library/functions.rst
View file

@ -1154,44 +1154,44 @@ are always available. They are listed here in alphabetical order.
.. function:: locals()
Return a mapping object representing the current local symbol table, with
variable names as the keys, and their currently bound references as the
values.
Return a mapping object representing the current local symbol table, with
variable names as the keys, and their currently bound references as the
values.
At module scope, as well as when using :func:`exec` or :func:`eval` with
a single namespace, this function returns the same namespace as
:func:`globals`.
At module scope, as well as when using :func:`exec` or :func:`eval` with
a single namespace, this function returns the same namespace as
:func:`globals`.
At class scope, it returns the namespace that will be passed to the
metaclass constructor.
At class scope, it returns the namespace that will be passed to the
metaclass constructor.
When using ``exec()`` or ``eval()`` with separate local and global
arguments, it returns the local namespace passed in to the function call.
When using ``exec()`` or ``eval()`` with separate local and global
arguments, it returns the local namespace passed in to the function call.
In all of the above cases, each call to ``locals()`` in a given frame of
execution will return the *same* mapping object. Changes made through
the mapping object returned from ``locals()`` will be visible as assigned,
reassigned, or deleted local variables, and assigning, reassigning, or
deleting local variables will immediately affect the contents of the
returned mapping object.
In all of the above cases, each call to ``locals()`` in a given frame of
execution will return the *same* mapping object. Changes made through
the mapping object returned from ``locals()`` will be visible as assigned,
reassigned, or deleted local variables, and assigning, reassigning, or
deleting local variables will immediately affect the contents of the
returned mapping object.
In an :term:`optimized scope` (including functions, generators, and
coroutines), each call to ``locals()`` instead returns a fresh dictionary
containing the current bindings of the function's local variables and any
nonlocal cell references. In this case, name binding changes made via the
returned dict are *not* written back to the corresponding local variables
or nonlocal cell references, and assigning, reassigning, or deleting local
variables and nonlocal cell references does *not* affect the contents
of previously returned dictionaries.
In an :term:`optimized scope` (including functions, generators, and
coroutines), each call to ``locals()`` instead returns a fresh dictionary
containing the current bindings of the function's local variables and any
nonlocal cell references. In this case, name binding changes made via the
returned dict are *not* written back to the corresponding local variables
or nonlocal cell references, and assigning, reassigning, or deleting local
variables and nonlocal cell references does *not* affect the contents
of previously returned dictionaries.
Calling ``locals()`` as part of a comprehension in a function, generator, or
coroutine is equivalent to calling it in the containing scope, except that
the comprehension's initialised iteration variables will be included. In
other scopes, it behaves as if the comprehension were running as a nested
function.
Calling ``locals()`` as part of a comprehension in a function, generator, or
coroutine is equivalent to calling it in the containing scope, except that
the comprehension's initialised iteration variables will be included. In
other scopes, it behaves as if the comprehension were running as a nested
function.
Calling ``locals()`` as part of a generator expression is equivalent to
calling it in a nested generator function.
Calling ``locals()`` as part of a generator expression is equivalent to
calling it in a nested generator function.
.. versionchanged:: 3.12
The behaviour of ``locals()`` in a comprehension has been updated as

3
Doc/library/functools.rst
View file

@ -403,8 +403,7 @@ The :mod:`functools` module defines the following functions:
>>> remove_first_dear(message)
'Hello, dear world!'
:data:`!Placeholder` has no special treatment when used in a keyword
argument to :func:`!partial`.
:data:`!Placeholder` cannot be passed to :func:`!partial` as a keyword argument.
.. versionchanged:: 3.14
Added support for :data:`Placeholder` in positional arguments.

5
Doc/library/gc.rst
View file

@ -128,6 +128,11 @@ The :mod:`gc` module provides the following functions:
starts. For each collection, all the objects in the young generation and some
fraction of the old generation is collected.
In the free-threaded build, the increase in process memory usage is also
checked before running the collector. If the memory usage has not increased
by 10% since the last collection and the net number of object allocations
has not exceeded 40 times *threshold0*, the collection is not run.
The fraction of the old generation that is collected is **inversely** proportional
to *threshold1*. The larger *threshold1* is, the slower objects in the old generation
are collected.

11
Doc/library/getpass.rst
View file

@ -16,7 +16,7 @@
The :mod:`getpass` module provides two functions:
.. function:: getpass(prompt='Password: ', stream=None)
.. function:: getpass(prompt='Password: ', stream=None, *, echo_char=None)
Prompt the user for a password without echoing. The user is prompted using
the string *prompt*, which defaults to ``'Password: '``. On Unix, the
@ -25,6 +25,12 @@ The :mod:`getpass` module provides two functions:
(:file:`/dev/tty`) or if that is unavailable to ``sys.stderr`` (this
argument is ignored on Windows).
The *echo_char* argument controls how user input is displayed while typing.
If *echo_char* is ``None`` (default), input remains hidden. Otherwise,
*echo_char* must be a printable ASCII string and each typed character
is replaced by it. For example, ``echo_char='*'`` will display
asterisks instead of the actual input.
If echo free input is unavailable getpass() falls back to printing
a warning message to *stream* and reading from ``sys.stdin`` and
issuing a :exc:`GetPassWarning`.
@ -33,6 +39,9 @@ The :mod:`getpass` module provides two functions:
If you call getpass from within IDLE, the input may be done in the
terminal you launched IDLE from rather than the idle window itself.
.. versionchanged:: 3.14
Added the *echo_char* parameter for keyboard feedback.
.. exception:: GetPassWarning
A :exc:`UserWarning` subclass issued when password input may be echoed.

2
Doc/library/glob.rst
View file

@ -134,7 +134,7 @@ The :mod:`glob` module defines the following functions:
>>>
>>> regex = glob.translate('**/*.txt', recursive=True, include_hidden=True)
>>> regex
'(?s:(?:.+/)?[^/]*\\.txt)\\Z'
'(?s:(?:.+/)?[^/]*\\.txt)\\z'
>>> reobj = re.compile(regex)
>>> reobj.match('foo/bar/baz.txt')
<re.Match object; span=(0, 15), match='foo/bar/baz.txt'>

Some files were not shown because too many files have changed in this diff Show more