mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-08-04 00:48:58 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

[3.12] gh-108740: Fix "make regen-all" race condition (#108741) (#109019)

Browse source 
gh-108740: Fix "make regen-all" race condition (#108741)

Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script.
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
  It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are now outdated, these commands are now
  safe to use.
* Document generates files in Doc/using/configure.rst.

Co-authored-by: Erlend E. Aasland <erlend@python.org>
(cherry picked from commit db1ee6a19a)
This commit is contained in:
Victor Stinner 2023-09-08 15:19:20 +02:00 • committed by GitHub
parent 55d7e8fef1
commit de0202e937
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
4 changed files with 38 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

23
Doc/using/configure.rst
View file

@ -43,6 +43,29 @@ See also :pep:`7` "Style Guide for C Code" and :pep:`11` "CPython platform
support".
Generated files
===============
To reduce build dependencies, Python source code contains multiple generated
files. Commands to regenerate all generated files::
make regen-all
make regen-stdlib-module-names
make regen-limited-abi
make regen-configure
The ``Makefile.pre.in`` file documents generated files, their inputs, and tools used
to regenerate them. Search for ``regen-*`` make targets.
The ``make regen-configure`` command runs `tiran/cpython_autoconf
<https://github.com/tiran/cpython_autoconf>`_ container for reproducible build;
see container ``entry.sh`` script. The container is optional, the following
command can be run locally, the generated files depend on autoconf and aclocal
versions::
autoreconf -ivf -Werror
.. _configure-options:
Configure Options

19
Makefile.pre.in
View file

@ -487,6 +487,7 @@ OBJECT_OBJS= \
Objects/weakrefobject.o \
@PERF_TRAMPOLINE_OBJ@
DEEPFREEZE_C = Python/deepfreeze/deepfreeze.c
DEEPFREEZE_OBJS = Python/deepfreeze/deepfreeze.o
##########################################################################
@ -774,7 +775,6 @@ coverage-report: regen-token regen-frozen
.PHONY: clinic
clinic: check-clean-src $(srcdir)/Modules/_blake2/blake2s_impl.c
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/clinic/clinic.py --make --srcdir $(srcdir)
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/build/generate_global_objects.py
# Build the interpreter
$(BUILDPYTHON): Programs/python.o $(LINK_PYTHON_DEPS)
@ -1245,12 +1245,12 @@ regen-frozen: Tools/build/freeze_modules.py $(FROZEN_FILES_IN)
# Deepfreeze targets
.PHONY: regen-deepfreeze
regen-deepfreeze: $(DEEPFREEZE_OBJS)
regen-deepfreeze: $(DEEPFREEZE_C)
DEEPFREEZE_DEPS=$(srcdir)/Tools/build/deepfreeze.py $(FREEZE_MODULE_DEPS) $(FROZEN_FILES_OUT)
# BEGIN: deepfreeze modules
Python/deepfreeze/deepfreeze.c: $(DEEPFREEZE_DEPS)
$(DEEPFREEZE_C): $(DEEPFREEZE_DEPS)
$(PYTHON_FOR_FREEZE) $(srcdir)/Tools/build/deepfreeze.py \
Python/frozen_modules/importlib._bootstrap.h:importlib._bootstrap \
Python/frozen_modules/importlib._bootstrap_external.h:importlib._bootstrap_external \
@ -1277,8 +1277,6 @@ Python/deepfreeze/deepfreeze.c: $(DEEPFREEZE_DEPS)
Python/frozen_modules/frozen_only.h:frozen_only \
-o Python/deepfreeze/deepfreeze.c
# END: deepfreeze modules
@echo "Note: Deepfreeze may have added some global objects,"
@echo " so run 'make regen-global-objects' if necessary."
# We keep this renamed target around for folks with muscle memory.
.PHONY: regen-importlib
@ -1287,11 +1285,12 @@ regen-importlib: regen-frozen
############################################################################
# Global objects
# Dependencies which can add and/or remove _Py_ID() identifiers:
# - deepfreeze.c
# - "make clinic"
.PHONY: regen-global-objects
regen-global-objects: $(srcdir)/Tools/build/generate_global_objects.py
regen-global-objects: $(srcdir)/Tools/build/generate_global_objects.py $(DEEPFREEZE_C) clinic
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/build/generate_global_objects.py
@echo "Note: Global objects can be added or removed by other tools (e.g. deepfreeze), "
@echo " so be sure to re-run regen-global-objects after those tools."
############################################################################
# ABI
@ -1313,9 +1312,10 @@ regen-limited-abi: all
############################################################################
# Regenerate all generated files
# "clinic" is regenerated implicitly via "regen-global-objects".
.PHONY: regen-all
regen-all: regen-cases regen-opcode regen-opcode-targets regen-typeslots \
regen-token regen-ast regen-keyword regen-sre regen-frozen clinic \
regen-token regen-ast regen-keyword regen-sre regen-frozen \
regen-pegen-metaparser regen-pegen regen-test-frozenmain \
regen-test-levenshtein regen-global-objects
@echo
@ -2586,6 +2586,7 @@ recheck:
autoconf:
(cd $(srcdir); autoreconf -ivf -Werror)
# See https://github.com/tiran/cpython_autoconf container
.PHONY: regen-configure
regen-configure:
@if command -v podman >/dev/null; then RUNTIME="podman"; else RUNTIME="docker"; fi; \

4
Misc/NEWS.d/next/Build/2023-09-01-01-39-26.gh-issue-108740.JHExAQ.rst Normal file
View file

@ -0,0 +1,4 @@
Fix a race condition in ``make regen-all``. The ``deepfreeze.c`` source and
files generated by Argument Clinic are now generated or updated before
generating "global objects". Previously, some identifiers may miss depending
on the order in which these files were generated. Patch by Victor Stinner.

2
Tools/build/freeze_modules.py
View file

@ -585,7 +585,7 @@ def regen_makefile(modules):
pyfiles = []
frozenfiles = []
rules = ['']
deepfreezerules = ["Python/deepfreeze/deepfreeze.c: $(DEEPFREEZE_DEPS)",
deepfreezerules = ["$(DEEPFREEZE_C): $(DEEPFREEZE_DEPS)",
"\t$(PYTHON_FOR_FREEZE) $(srcdir)/Tools/build/deepfreeze.py \\"]
for src in _iter_sources(modules):
frozen_header = relpath_for_posix_display(src.frozenfile, ROOT_DIR)