regrtest test runner: Add XML support to the refleak checker
(-R option).
* run_unittest() now stores XML elements as string, rather than
objects, in support.junit_xml_list.
* runtest_refleak() now saves/restores XML strings before/after
checking for reference leaks. Save XML into a temporary file.
This fixes XML unittest fallout from the https://github.com/python/cpython/issues/115398 security fix. When configured using `--with-system-expat` on systems with older pre 2.6.0 versions of libexpat, our unittests were failing.
* sax|etree: Simplify Expat version guard where simplifiable
Idea by Matěj Cepl
* sax|etree: Fix reparse deferral tests for vanilla Expat <2.6.0
This *does not fix* the case of distros with an older version of libexpat with the 2.6.0 feature backported as a security fix. (Ubuntu is a known example of this with its libexpat1 2.5.0-2ubunutu0.1 package)
* Ensure importlib.metadata tests do not leak references in sys.modules.
* Move importlib.metadata tests to their own package for easier syncing with importlib_metadata.
* Update owners and makefile for new directories.
* Add blurb
gh-116307: Create a new import helper 'isolated modules' and use that instead of 'Clean Import' to ensure that tests from importlib_resources don't leave modules in sys.modules.
Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks
This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.
Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.
Test *failure* is still determined by the existing heuristic.
* test.bisect_cmd now exit with code 0 on success, and code 1 on
failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
-X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
methods.
This change essentially replaces usage of `%1` with `%~1`, which removes
quotes, if any. Without this change, the if statements fail due to
the quotes mangling the syntax.
Additionally, this change works around comma being treated as a parameter
delimiter in test.bat by escaping commas at time of parsing. Tested
combinations of rt and regrtest arguments, all seems to work as before
but now you can specify commas in arguments like "-uall,extralargefile".
Add test annotations required to run the test suite on iOS (PEP 730).
The majority of the change involve annotating tests that use subprocess,
but are skipped on Emscripten/WASI for other reasons, and including
iOS/tvOS/watchOS under the same umbrella as macOS/darwin checks.
`is_apple` and `is_apple_mobile` test helpers have been added to
identify *any* Apple platform, and "any Apple platform except macOS",
respectively.
On Windows, `os.path.isabs()` now returns `False` when given a path that
starts with exactly one (back)slash. This is more compatible with other
functions in `os.path`, and with Microsoft's own documentation.
Also adjust `pathlib.PureWindowsPath.is_absolute()` to call
`ntpath.isabs()`, which corrects its handling of partial UNC/device paths
like `//foo`.
Co-authored-by: Jon Foster <jon@jon-foster.co.uk>
On recentish macOS versions the system tar
command includes system metadata (ACLs, extended attributes
and resource forks) in the tar archive, which
shutil.make_archive will not do. This can cause
spurious test failures.
Regression test that vfork is used when expected by subprocess.
This is written integration test style, it uses strace if it is present and appears to work to find out what system call actually gets used in different scenarios.
Test coverage is added for the default behavior and that of each of the specific arguments that must disable the use of vfork. obviously not an entire test matrix, but it covers the most important aspects.
If there are ever issues with this test being flaky or failing on new platforms, rather than try and adapt it for all possible platforms, feel free to narrow the range it gets tested on when appropriate. That is not likely to reduce coverage.
Now all results from worker processes are aggregated and
displayed together as a summary at the end of a regrtest run.
The traditional trace is left in place for use with sequential
in-process test runs but now raises a warning that those
numbers are not precise.
`-T -j` requires `--with-pydebug` as it relies on `-Xpresite=`.
Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
Fix detection of gdb built without Python scripting support.
* check_usable_gdb() doesn't check gdb exit code when calling
run_gdb().
* Use shutil.which() to get the path to the gdb program.
