The change in gh-118157 (b2cd54a) should have also updated clear_singlephase_extension() but didn't. We fix that here. Note that clear_singlephase_extension() (AKA _PyImport_ClearExtension()) is only used in tests.
(cherry picked from commit 15d48aea02, AKA gh-121503)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
The `_PySeqLock_EndRead` function needs an acquire fence to ensure that
the load of the sequence happens after any loads within the read side
critical section. The missing fence can trigger bugs on macOS arm64.
Additionally, we need a release fence in `_PySeqLock_LockWrite` to
ensure that the sequence update is visible before any modifications to
the cache entry.
(cherry picked from commit 1d3cf79a50)
Co-authored-by: Sam Gross <colesbury@gmail.com>
gh-120837: Update _Py_DumpExtensionModules to be async-signal-safe (gh-121051)
(cherry picked from commit 1a2e7a7475)
Co-authored-by: Donghee Na <donghee.na@python.org>
This change makes things a little less painful for some users. It also fixes a failing assert (gh-120765), by making sure all subinterpreters are destroyed before the main interpreter. As part of that, we make sure Py_Finalize() always runs with the main interpreter active.
(cherry picked from commit 4be1f37b20, AKA gh-121060)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
We also add _PyThreadState_NewBound() and drop _PyThreadState_SetWhence().
This change only affects internal API.
(cherry picked from commit a905721b9c, AKA gh-121010)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
* Add an InternalDocs file describing how interning should work and how to use it.
* Add internal functions to *explicitly* request what kind of interning is done:
- `_PyUnicode_InternMortal`
- `_PyUnicode_InternImmortal`
- `_PyUnicode_InternStatic`
* Switch uses of `PyUnicode_InternInPlace` to those.
* Disallow using `_Py_SetImmortal` on strings directly.
You should use `_PyUnicode_InternImmortal` instead:
- Strings should be interned before immortalization, otherwise you're possibly
interning a immortalizing copy.
- `_Py_SetImmortal` doesn't handle the `SSTATE_INTERNED_MORTAL` to
`SSTATE_INTERNED_IMMORTAL` update, and those flags can't be changed in
backports, as they are now part of public API and version-specific ABI.
* Add private `_only_immortal` argument for `sys.getunicodeinternedsize`, used in refleak test machinery.
* Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:
- `_Py_ID`
- `_Py_STR` (including the empty string)
- one-character latin-1 singletons
Now, when you intern a singleton, that exact singleton will be interned.
* Add a `_Py_LATIN1_CHR` macro, use it instead of `_Py_ID`/`_Py_STR` for one-character latin-1 singletons everywhere (including Clinic).
* Intern `_Py_STR` singletons at startup.
* For free-threaded builds, intern `_Py_LATIN1_CHR` singletons at startup.
* Beef up the tests. Cover internal details (marked with `@cpython_only`).
* Add lots of assertions
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
This makes the following macros public as part of the non-limited C-API for
locking a single object or two objects at once.
* `Py_BEGIN_CRITICAL_SECTION(op)` / `Py_END_CRITICAL_SECTION()`
* `Py_BEGIN_CRITICAL_SECTION2(a, b)` / `Py_END_CRITICAL_SECTION2()`
The supporting functions and structs used by the macros are also exposed for
cases where C macros are not available.
(cherry picked from commit 8f17d69b7b)
gh-120726: Fix compiler warnings on is_core_module() (GH-120727)
Fix compiler warnings on is_core_module() and
check_interpreter_whence(): only define them when
assertions are built.
(cherry picked from commit a816cd67f4)
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Fix a race in `PyMember_GetOne` and `PyMember_SetOne` for `Py_T_OBJECT_EX`.
These functions implement `__slots__` accesses for Python objects.
(cherry picked from commit 362cd2680b)
Co-authored-by: Daniele Parmeggiani <8658291+dpdani@users.noreply.github.com>
gh-115649: Copy the filename into main interpreter before intern in import.c (GH-120315)
(cherry picked from commit 28140d1f2d)
Co-authored-by: AN Long <aisk@users.noreply.github.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
map() requires at least one iterable arg.
(cherry picked from commit d4039d3f6f)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Co-authored-by: Adam Williamson <adam@blueradius.ca>
In gh-120009 I used an atexit hook to finalize the _datetime module's static types at interpreter shutdown. However, atexit hooks are executed very early in finalization, which is a problem in the few cases where a subclass of one of those static types is still alive until the final GC collection. The static builtin types don't have this probably because they are finalized toward the end, after the final GC collection. To avoid the problem for _datetime, I have applied a similar approach here.
Also, credit goes to @mgorny and @neonene for the new tests.
FYI, I would have liked to take a slightly cleaner approach with managed static types, but wanted to get a smaller fix in first for the sake of backporting. I'll circle back to the cleaner approach with a future change on the main branch.
(cherry picked from commit b2e71ff4f8, AKA gh-120182)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
gh-120400 ：Support Linux perf profile to see Python calls on RISC-V architecture (GH-120089)
(cherry picked from commit 4b1e85bafc)
Co-authored-by: ixgbe00 <yangwang@iscas.ac.cn>
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
gh-117657: Fix some simple races in instrumentation.c (GH-120118)
* stop the world when setting local events
(cherry picked from commit b1b61dc4ce)
Co-authored-by: Ken Jin <kenjin@python.org>
gh-93691: fix too broad source locations of for statement iterators (GH-120330)
(cherry picked from commit 97b69db167)
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
gh-120225: fix crash in compiler on empty block at end of exception handler (GH-120235)
(cherry picked from commit 4fc82b6d3b)
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
This adds a `_PyRecursiveMutex` type based on `PyMutex` and uses that
for the import lock. This fixes some data races in the free-threaded
build and generally simplifies the import lock code.
(cherry picked from commit e21057b999)
Co-authored-by: Sam Gross <colesbury@gmail.com>
The `_PyThreadState_Bind()` function is called before the first
`PyEval_AcquireThread()` so it's not synchronized with the stop the
world GC. We had a race where `gc_visit_heaps()` might visit a thread's
heap while it's being initialized.
Use a simple atomic int to avoid visiting heaps for threads that are not
yet fully initialized (i.e., before `tstate_mimalloc_bind()` is called).
The race was reproducible by running:
`python Lib/test/test_importlib/partial/pool_in_threads.py`.
(cherry picked from commit e69d068ad0)
Co-authored-by: Sam Gross <colesbury@gmail.com>
gh-111499: Fix PYTHONMALLOCSTATS at Python exit (GH-120021)
Call _PyObject_DebugMallocStats() earlier in Py_FinalizeEx(), before
the interpreter is deleted.
(cherry picked from commit 5a1205b641)
Co-authored-by: Victor Stinner <vstinner@python.org>
We make use of the same mechanism that we use for the static builtin types. This required a few tweaks.
This change is the final piece needed to make _datetime support multiple interpreters. I've updated the module slot accordingly.
(cherry picked from commit 105f22ea46, AKA gh-119929)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
The free-threaded build currently immortalizes objects that use deferred
reference counting (see gh-117783). This typically happens once the
first non-main thread is created, but the behavior can be suppressed for
tests, in subinterpreters, or during a compile() call.
This fixes a race condition involving the tracking of whether the
behavior is suppressed.
(cherry picked from commit 47fb4327b5)
The `sem_clockwait` function is not currently instrumented, which leads
to false positives.
(cherry picked from commit 41c1cefbae)
Co-authored-by: Sam Gross <colesbury@gmail.com>
Due to a limitation in TSAN, all reads from `PyThreadState.state` must be
atomic to avoid reported races.
(cherry picked from commit 90ec19fd33)
Co-authored-by: Sam Gross <colesbury@gmail.com>
Only call `gc_restore_tid()` from stop-the-world contexts.
`worklist_pop()` can be called while other threads are running, so use a
relaxed atomic to modify `ob_tid`.
(cherry picked from commit 60593b2052)
Co-authored-by: Sam Gross <colesbury@gmail.com>
dSupport non-dict globals in LOAD_FROM_DICT_OR_GLOBALS
The implementation basically copies LOAD_GLOBAL. Possibly it could be deduplicated,
but that seems like it may get hairy since the two operations have different operands.
This is important to fix in 3.14 for PEP 649, but it's a bug in earlier versions too,
and we should backport to 3.13 and 3.12 if possible.
(cherry picked from commit 80a4e38994)
* Add docs for new APIs
* Add soft-deprecation notices
* Add What's New porting entries
* Update comments referencing `PyFrame_LocalsToFast()` to mention the proxy instead
* Other related cleanups found when looking for refs to the deprecated APIs
(cherry picked from commit 3859e09e3d)
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
Release the GIL before calling `_Py_qsbr_unregister`.
The deadlock could occur when the GIL was enabled at runtime. The
`_Py_qsbr_unregister` call might block while holding the GIL because the
thread state was not active, but the GIL was still held.
(cherry picked from commit 078b8c8cf2)
Co-authored-by: Sam Gross <colesbury@gmail.com>
Make sure that `gilstate_counter` is not zero in when calling
`PyThreadState_Clear()`. A destructor called from `PyThreadState_Clear()` may
call back into `PyGILState_Ensure()` and `PyGILState_Release()`. If
`gilstate_counter` is zero, it will try to create a new thread state before
the current active thread state is destroyed, leading to an assertion failure
or crash.
(cherry picked from commit bcc1be39cb)
Co-authored-by: Sam Gross <colesbury@gmail.com>
- Cache line object to avoid creating a Unicode object
for all of the tokens in the same line.
- Speed up byte offset to column offset conversion by using the
smallest buffer possible to measure the difference.
(cherry picked from commit d87b015106)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
