Commit graph

523 commits

Author SHA1 Message Date
Mark Shannon
48be46ec1f
bpo-46072: Add some object layout and allocation stats (GH-31051) 2022-02-01 15:05:18 +00:00
Mark Shannon
90ab138bbd
bpo-46072: Add simple stats for Python calls. (GH-30989) 2022-01-28 15:20:33 +00:00
Mark Shannon
89fd7c3452
bpo-46329: Split calls into precall and call instructions. (GH-30855)
* Add PRECALL_FUNCTION opcode.

* Move 'call shape' varaibles into struct.

* Replace CALL_NO_KW and CALL_KW with KW_NAMES and CALL instructions.

* Specialize for builtin methods taking using the METH_FASTCALL | METH_KEYWORDS protocol.

* Allow kwnames for specialized calls to builtin types.

* Specialize calls to tuple(arg) and str(arg).
2022-01-28 12:42:30 +00:00
Victor Stinner
18ea973c21
bpo-40170: Remove PyHeapType_GET_MEMBERS() macro (GH-30942)
Remove the PyHeapType_GET_MEMBERS() macro. It was exposed in the
public C API by mistake, it must only be used by Python internally.
Use the PyTypeObject.tp_members member instead.

Rename PyHeapType_GET_MEMBERS() to _PyHeapType_GET_MEMBERS() and move
it to the internal C API.
2022-01-28 00:39:52 +01:00
Victor Stinner
0575551f69
bpo-40170: Move _Py_GetAllocatedBlocks() to pycore_pymem.h (GH-30943)
Move _Py_GetAllocatedBlocks() and _PyObject_DebugMallocStats()
declarations to pycore_pymem.h. These functions are related to memory
allocators, not to the PyObject structure.
2022-01-27 21:23:22 +01:00
Christian Heimes
26b0482393
bpo-46476: Simplify and fix _PyStaticCode_Dealloc (GH-30965) 2022-01-27 19:32:12 +00:00
Eric Snow
247480a21c
bpo-46541: Generate the global objects initializer. (gh-30941)
This change is a prerequisite for generating code for other global objects (like strings in gh-30928).

(We borrowed some code from Tools/scripts/deepfreeze.py.)

https://bugs.python.org/issue46541
2022-01-27 11:06:09 -07:00
Kumar Aditya
c7f810b34d
bpo-46476: Fix memory leak in code objects generated by deepfreeze (GH-30853)
Add _Py_Deepfreeze_Fini() and _PyStaticCode_Dealloc() functions.
2022-01-27 14:03:47 +01:00
Victor Stinner
6b491b9dc0
bpo-40170: Remove _Py_GetAllocatedBlocks() function (GH-30940)
Move _Py_GetAllocatedBlocks() and _PyObject_DebugMallocStats()
private functions to the internal C API.
2022-01-27 03:35:51 +01:00
Victor Stinner
af32b3ef1f
bpo-40170: PyType_SUPPORTS_WEAKREFS() becomes a regular function (GH-30938)
Convert the PyType_SUPPORTS_WEAKREFS() macro to a regular function.
It no longer access the PyTypeObject.tp_weaklistoffset member
directly.

Add _PyType_SUPPORTS_WEAKREFS() static inline functions, used
internally by Python for best performance.
2022-01-27 03:00:55 +01:00
Mark Dickinson
83a0ef2162
bpo-29882: Fix portability bug introduced in GH-30774 (#30794) 2022-01-23 09:59:34 +00:00
Victor Stinner
12f4ac3bc8
bpo-46417: Clear symtable identifiers at exit (GH-30809)
Add _PySymtable_Fini() function, called by finalize_interp_clear().

Update test_cmd_line.test_showrefcount() to tolerate negative
reference count.
2022-01-23 00:06:56 +01:00
Victor Stinner
1626bf4ac7
bpo-46417: Clear Unicode static types at exit (GH-30806)
Add _PyUnicode_FiniTypes() function, called by
finalize_interp_types(). It clears these static types:

* EncodingMapType
* PyFieldNameIter_Type
* PyFormatterIter_Type

_PyStaticType_Dealloc() now does nothing if tp_subclasses
is not NULL.
2022-01-22 22:55:39 +01:00
Victor Stinner
f1bcdeaca6
bpo-46417: Factorize _PyExc_InitTypes() code (GH-30804)
Add 'static_exceptions' list to factorize code between
_PyExc_InitTypes() and _PyBuiltins_AddExceptions().

_PyExc_InitTypes() does nothing if it's not the main interpreter.

Sort exceptions in Lib/test/exception_hierarchy.txt.
2022-01-22 21:48:56 +01:00
Victor Stinner
500c146387
bpo-46417: Clear more static types (GH-30796)
* Move PyContext static types into object.c static_types list.
* Rename PyContextTokenMissing_Type to _PyContextTokenMissing_Type
  and declare it in pycore_context.h.
* _PyHamtItems types are no long exported: replace PyAPI_DATA() with
  extern.
2022-01-22 18:55:48 +01:00
Victor Stinner
cd8de40b3b
bpo-29882: _Py_popcount32() doesn't need 64x64 multiply (GH-30774)
32x32 bits multiply is enough for _Py_popcount32().
2022-01-22 00:54:42 +01:00
Victor Stinner
8ee07dda13
bpo-46417: Add _PyType_GetSubclasses() function (GH-30761)
Add a new _PyType_GetSubclasses() function to get type's subclasses.

_PyType_GetSubclasses(type) returns a list which holds strong
refererences to subclasses. It is safer than iterating on
type->tp_subclasses which yields weak references and can be modified
in the loop.

_PyType_GetSubclasses(type) now holds a reference to the tp_subclasses
dict while creating the list of subclasses.

set_collection_flag_recursive() of _abc.c now uses
_PyType_GetSubclasses().
2022-01-21 23:29:10 +01:00
Victor Stinner
595225e86d
bpo-46417: Py_Finalize() clears static types (GH-30743)
Add _PyTypes_FiniTypes() best-effort function to clear static types:
don't deallocate a type if it still has subclasses.

remove_subclass() now sets tp_subclasses to NULL when removing the
last subclass.
2022-01-21 13:06:34 +01:00
Victor Stinner
1781d55eb3
bpo-46417: _curses uses PyStructSequence_NewType() (GH-30736)
The _curses module now creates its ncurses_version type as a heap
type using PyStructSequence_NewType(), rather than using a static
type.

* Move _PyStructSequence_FiniType() definition to pycore_structseq.h.
* test.pythoninfo: log curses.ncurses_version.
2022-01-21 03:30:20 +01:00
Victor Stinner
e9e3eab0b8
bpo-46417: Finalize structseq types at exit (GH-30645)
Add _PyStructSequence_FiniType() and _PyStaticType_Dealloc()
functions to finalize a structseq static type in Py_Finalize().
Currrently, these functions do nothing if Python is built in release
mode.

Clear static types:

* AsyncGenHooksType: sys.set_asyncgen_hooks()
* FlagsType: sys.flags
* FloatInfoType: sys.float_info
* Hash_InfoType: sys.hash_info
* Int_InfoType: sys.int_info
* ThreadInfoType: sys.thread_info
* UnraisableHookArgsType: sys.unraisablehook
* VersionInfoType: sys.version
* WindowsVersionType: sys.getwindowsversion()
2022-01-21 01:42:25 +01:00
Mark Shannon
b04dfbbe4b
bpo-46409: Make generators in bytecode (GH-30633)
* Add RETURN_GENERATOR and JUMP_NO_INTERRUPT opcodes.

* Trim frame and generator by word each.

* Minor refactor of frame.c

* Update test.test_sys to account for smaller frames.

* Treat generator functions as normal functions when evaluating and specializing.
2022-01-20 11:46:39 +00:00
Eric Snow
322f962f3e
bpo-45953: Statically initialize all the non-object PyInterpreterState fields we can. (gh-30589)
https://bugs.python.org/issue45953
2022-01-13 17:17:28 -07:00
Eric Snow
324908ba93
bpo-45953: Statically initialize all the PyThreadState fields we can. (gh-30590)
https://bugs.python.org/issue45953
2022-01-13 17:09:24 -07:00
neonene
d4e64cd4b0
bpo-46362: Ensure ntpath.abspath() uses the Windows API correctly (GH-30571)
This makes ntpath.abspath()/getpath_abspath() follow normpath(), since some WinAPIs such as PathCchSkipRoot() require backslashed paths.
2022-01-13 23:35:42 +00:00
Eric Snow
b8ddf7e794
Statically initialize _PyRuntimeState fields. (gh-30588)
https://bugs.python.org/issue45953
2022-01-13 16:33:40 -07:00
Eric Snow
bc02eac9d2
bpo-46370: Move the static initializer for _PyRuntime to its own header file. (gh-30587)
https://bugs.python.org/issue46370
2022-01-13 15:54:36 -07:00
Christian Heimes
a6ca8eee22
bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507) 2022-01-13 09:46:04 +01:00
Eric Snow
ed57b36c32
bpo-45953: Statically allocate the main interpreter (and initial thread state). (gh-29883)
Previously, the main interpreter was allocated on the heap during runtime initialization.  Here we instead embed it into _PyRuntimeState, which means it is statically allocated as part of the _PyRuntime global.  The same goes for the initial thread state (of each interpreter, including the main one).  Consequently there are fewer allocations during runtime/interpreter init, fewer possible failures, and better memory locality.

FYI, this also helps efforts to consolidate globals, which in turns helps work on subinterpreter isolation.

https://bugs.python.org/issue45953
2022-01-12 16:28:46 -07:00
Eric Snow
cf496d657a
bpo-45953: Statically allocate and initialize global bytes objects. (gh-30096)
The empty bytes object (b'') and the 256 one-character bytes objects were allocated at runtime init.  Now we statically allocate and initialize them.

https://bugs.python.org/issue45953
2022-01-11 09:37:24 -07:00
Victor Stinner
ea1a54506b
bpo-46303: Move fileutils.h private functions to internal C API (GH-30484)
Move almost all private functions of Include/cpython/fileutils.h to
the internal C API Include/internal/pycore_fileutils.h.

Only keep _Py_fopen_obj() in Include/cpython/fileutils.h, since it's
used by _testcapi which must not use the internal C API.

Move EncodeLocaleEx() and DecodeLocaleEx() functions from _testcapi
to _testinternalcapi, since the C API moved to the internal C API.
2022-01-11 11:56:16 +01:00
Victor Stinner
35d6540c90
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (GH-20085)" (GH-30422)
This reverts commit ea251806b8.

Keep "assert(interned == NULL);" in _PyUnicode_Fini(), but only for
the main interpreter.

Keep _PyUnicode_ClearInterned() changes avoiding the creation of a
temporary Python list object.
2022-01-06 08:53:44 +01:00
Brandt Bucher
332e6b9725
bpo-45256: Don't track the exact depth of each InterpreterFrame (GH-30372) 2022-01-05 11:30:26 +00:00
Irit Katriel
65e7c1f90e
bpo-46219, 46221: simplify except* implementation following exc_info changes. Move helpers to exceptions.c. Do not assume that exception groups are truthy. (GH-30289) 2022-01-02 23:22:42 +00:00
Brandt Bucher
77195cd44b
bpo-46090: Allow PyThreadState.datastack_* members to be NULL (GH-30234) 2021-12-28 09:49:48 -08:00
Mark Shannon
efd6236d36
bpo-46072: Add top level stats struct (GH-30169) 2021-12-17 14:48:01 +00:00
Irit Katriel
396b58345f
bpo-45711: Remove type and traceback from exc_info (GH-30122)
* Do not PUSH/POP traceback or type to the stack as part of exc_info

* Remove exc_traceback and exc_type from _PyErr_StackItem

* Add to what's new, because this change breaks things like Cython
2021-12-17 14:46:22 +00:00
Mark Shannon
342b93f9f2
bpo-46072: Add --with-pystats configure option to simplify gathering of VM stats (GH-30116)
* Simplify specialization stats collection macros.

* Add --enable-pystats option to configure.

* Update specialization summary script to handle larger number of kinds
2021-12-15 15:32:32 +00:00
Mark Shannon
0b50a4f0cd
bpo-46039: Split yield from in two (GH-30035)
* Split YIELD_FROM opcode into SEND and JUMP_ABSOLUTE.

* Remove YIELD_FROM opcode.
2021-12-15 10:30:09 +00:00
Mark Shannon
9f8f45144b
bpo-44525: Split calls into PRECALL and CALL (GH-30011)
* Add 3 new opcodes for calls: PRECALL_METHOD, CALL_NO_KW, CALL_KW.

* Update specialization to handle new CALL opcodes.

* Specialize call to method descriptors.

* Remove old CALL opcodes: CALL_FUNCTION, CALL_METHOD, CALL_METHOD_KW, CALL_FUNCTION_KW.
2021-12-14 18:22:44 +00:00
Irit Katriel
d60457a667
bpo-45292: [PEP-654] add except* (GH-29581) 2021-12-14 16:48:15 +00:00
Eric Snow
121f1f893a
bpo-45953: Statically initialize the small ints. (gh-30092)
The array of small PyLong objects has been statically declared. Here I also statically initialize them. Consequently they are no longer initialized dynamically during runtime init.

I've also moved them under a new sub-struct in _PyRuntimeState, in preparation for static allocation and initialization of other global objects.

https://bugs.python.org/issue45953
2021-12-13 18:04:05 -07:00
Pablo Galindo Salgado
59435eea08
bpo-46042: Improve SyntaxError locations in the symbol table (GH-30059) 2021-12-11 21:28:24 +00:00
Eric Snow
c8749b5783
bpo-46008: Make runtime-global object/type lifecycle functions and state consistent. (gh-29998)
This change is strictly renames and moving code around.  It helps in the following ways:

* ensures type-related init functions focus strictly on one of the three aspects (state, objects, types)
* passes in PyInterpreterState * to all those functions, simplifying work on moving types/objects/state to the interpreter
* consistent naming conventions help make what's going on more clear
* keeping API related to a type in the corresponding header file makes it more obvious where to look for it

https://bugs.python.org/issue46008
2021-12-09 12:59:26 -07:00
Eric Snow
32a67246b0
bpo-46008: Move Py*State init into distinct functions. (gh-29977)
Previously, basic initialization of PyInterprterState happened in PyInterpreterState_New() (along with allocation and adding the new interpreter to the runtime state). This prevented us from initializing interpreter states that were allocated separately (e.g. statically or in a free list). We've addressed that here by factoring out a separate function just for initialization. We've done the same for PyThreadState. _PyRuntimeState was sorted out when we added it since _PyRuntime is statically allocated. However, here we update the existing init code to line up with the functions for PyInterpreterState and PyThreadState.

https://bugs.python.org/issue46008
2021-12-07 18:59:49 -07:00
Eric Snow
758b74e71e
bpo-46008: Add _PyInterpreterState_Main(). (gh-29978)
PyInterpreterState_Main() is a plain function exposed in the public C-API. For internal usage we can take the more efficient approach in this PR.

https://bugs.python.org/issue46008
2021-12-07 18:56:06 -07:00
Eric Snow
1f384e3184
bpo-46008: Stop calling _PyThreadState_Init() in new_threadstate(). (gh-29973)
This simplifies new_threadstate().  We also rename _PyThreadState_Init() to _PyThreadState_SetCurrent() to reflect what it actually does.

https://bugs.python.org/issue46008
2021-12-07 17:26:29 -07:00
Eric Snow
313f92a57b
bpo-46008: Move thread-related interpreter state into a sub-struct. (gh-29971)
This parallels _PyRuntimeState.interpreters.  Doing this helps make it more clear what part of PyInterpreterState relates to its threads.

https://bugs.python.org/issue46008
2021-12-07 14:03:47 -07:00
Eric Snow
8262c96bcc
bpo-46008: Return void from _PyEval_InitState(). (gh-29970)
This falls into the category of keep-allocation-and-initialization separate. It also allows us to use _PyEval_InitState() safely in functions that return void.

https://bugs.python.org/issue46008
2021-12-07 14:02:17 -07:00
Mark Shannon
8319114fee
bpo-45947: Place dict and values pointer at fixed (negative) offset just before GC header. (GH-29879)
* Place __dict__ immediately before GC header for plain Python objects.

* Fix up lazy dict creation logic to use managed dict pointers.

* Manage values pointer, placing them directly before managed dict pointers.

* Convert hint-based load/store attr specialization target managed dict classes.

* Specialize LOAD_METHOD for managed dict objects.

* Remove unsafe _PyObject_GC_Calloc function.

* Remove unsafe _PyObject_GC_Malloc() function.

* Add comment explaning use of Py_TPFLAGS_MANAGED_DICT.
2021-12-07 16:02:53 +00:00
Mark Shannon
299483c95d
bpo-45963: Make space for the InterpreterFrame of a generator in that generator. (GH-29891)
* Make generator, coroutine and async gen structs all the same size.

* Store interpreter frame in generator (and coroutine). Reduces the number of allocations neeeded for a generator from two to one.
2021-12-06 10:13:49 +00:00