* gh-115122: Add --bisect option to regrtest (#115123)
* test.bisect_cmd now exit with code 0 on success, and code 1 on
failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
-X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
methods.
(cherry picked from commit 1e5719a663)
* gh-115720: Show number of leaks in huntrleaks progress reports (GH-115726)
Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks
This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.
Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.
Test *failure* is still determined by the existing heuristic.
(cherry picked from commit af5f9d682c)
* gh-83434: Disable XML in regrtest when -R option is used (#117232)
(cherry picked from commit d52bdfb19f)
---------
Co-authored-by: Petr Viktorin <encukou@gmail.com>
* gh-108834: regrtest reruns failed tests in subprocesses (#108839)
When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.
Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.
Changes:
* Fix and enhance statistics in regrtest summary. Add "(filtered)"
when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.
(cherry picked from commit 31c2945f14)
* gh-108834: regrtest --fail-rerun exits with code 5 (#108896)
When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
(cherry picked from commit 1170d5a292)
* gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') (GH-108480)
(cherry picked from commit 1e0d62793a)
* Manually sync Lib/test/libregrtest/ from main
---------
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Rename test.bisect module to test.bisect_cmd to avoid conflict with
bisect module when running directly a test like
"./python Lib/test/test_xmlrpc.py".
