* gh-108834: regrtest reruns failed tests in subprocesses (#108839)
When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.
Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.
Changes:
* Fix and enhance statistics in regrtest summary. Add "(filtered)"
when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.
(cherry picked from commit 31c2945f14)
* gh-108834: regrtest --fail-rerun exits with code 5 (#108896)
When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
(cherry picked from commit 1170d5a292)
* gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') (GH-108480)
(cherry picked from commit 1e0d62793a)
* Manually sync Lib/test/libregrtest/ from main
---------
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
atexit._run_exitfuncs() now logs callback exceptions using
sys.unraisablehook, rather than logging them directly into
sys.stderr and raising the last exception.
Run GeneralTest of test_atexit in a subprocess since it calls
atexit._clear() which clears all atexit callbacks.
_PyAtExit_Fini() sets state->callbacks to NULL.
* Add run_test_script() function to test.support.script_helper.
* Rename Lib/test/eintrdata/eintr_tester.py to
Lib/test/_test_eintr.py.
* test_eintr.py uses run_test_script().
try to debug a hang on the FreeBSD 9 buildbot.
Run also eintr_tester.py with python "-u" command line option to try to get the
full output on hang/crash.
"./python -m test -W test_eintr" wrote Lib/test/eintrdata/eintr_tester.py
output to stdout which was not expected.
Since test_eintr doesn't hang anymore, remove the verbose mode instead.
Skip test_open() and test_os_open(): both tests uses a FIFO and signals, but
there is a bug in the FreeBSD kernel which blocks the test. Skip the tests
until the bug is fixed in FreeBSD kernel.
Remove also debug traces from test_eintr:
* stop using faulthandler to have a timeout
* remove print()
Write also open and close on two lines in test_open() and test_os_open()
tests. If these tests block again, we can know if the test is stuck at open or
close.
test_eintr: don't always run the test in debug mode.
* Add verbose mode to test_eintr
* Always enable verbose mode in test_eintr
* Use faulthandler.dump_traceback_later() with a timeout of 15 minutes in
eintr_tester.py
