Set sys.stdout encoder error handler to backslashreplace in regrtest
workers to avoid UnicodeEncodeError when printing a traceback
or any other non-encodable character.
Move the code from the Regrtest class to setup_process().
Call setup_process() earlier, before displaying regrtest headers.
Run each example as a subtest in unit tests synthesized by
doctest.DocFileSuite() and doctest.DocTestSuite().
Add the doctest.DocTestRunner.report_skip() method.
Use "backslashreplace" error handler to decode stdout and stderr.
Example:
vstinner@WIN C:\victor\python\main\build\test_python_worker_8360\x91>
"C:\victor\python\main\PCbuild\amd64\python_d.exe" -m test
--fast-ci --slow-ci --testdir
C:\Users\vstinner\AppData\Local\Temp\tmp0t59e8da
test_regrtest_noop1 test_regrtest_noop2 test_regrtest_noop3
test_regrtest_noop4
Notice the "\x91" byte at the end of the first line: it's the
non-ASCII U+00E6 character encoded to the OEM cp437 code page.
First, write the test name without color. Then, write the test name
and the result with color. Each test is displayed twice.
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Remove _PyInterpreterState_GetConfigCopy() and
_PyInterpreterState_SetConfig() private functions. PEP 741 "Python
Configuration C API" added a better public C API: PyConfig_Get() and
PyConfig_Set().
* Move get_signal_name() from test.libregrtest to test.support.
* Use get_signal_name() in support.script_helper.
* support.script_helper now decodes stdout and stderr from UTF-8,
instead of ASCII, if a command failed.
When creating the JUnit XML file, regrtest now escapes characters
which are invalid in XML, such as the chr(27) control character used
in ANSI escape sequences.
regrtest test runner: Add XML support to the refleak checker
(-R option).
* run_unittest() now stores XML elements as string, rather than
objects, in support.junit_xml_list.
* runtest_refleak() now saves/restores XML strings before/after
checking for reference leaks. Save XML into a temporary file.
test_match_tests now saves and restores patterns.
Add get_match_tests() function to libregrtest.filter.
Previously, running test_regrtest multiple times in a row only ran
tests once: "./python -m test test_regrtest -R 3:3.
Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks
This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.
Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.
Test *failure* is still determined by the existing heuristic.
* test.bisect_cmd now exit with code 0 on success, and code 1 on
failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
-X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
methods.
Now all results from worker processes are aggregated and
displayed together as a summary at the end of a regrtest run.
The traditional trace is left in place for use with sequential
in-process test runs but now raises a warning that those
numbers are not precise.
`-T -j` requires `--with-pydebug` as it relies on `-Xpresite=`.
Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
When using worker processes (-jN) with --verbose3 option, regrtest
can now display the worker output even if a worker process does
crash. Previously, sys.stdout and sys.stderr were replaced and so
the worker output was lost on a crash.
* _add_python_opts() now handles cross compilation and HOSTRUNNER.
* display_header() now tells if Python is cross-compiled, display
HOSTRUNNER, and get the host platform.
* Remove Tools/scripts/run_tests.py script.
* Remove "make hostrunnertest": use "make buildbottest"
or "make test" instead.
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.
Change also TestResults.display_result() order: display failed tests
at the end, the important important information.
WorkerThread now tries to get the signal name for negative exit code.
WASM and WASI buildbots use multiple PYTHON environment variables
such as PYTHONPATH and _PYTHON_HOSTRUNNER. Don't use -E if the
--python=COMMAND option is used.
* On Windows, use subprocess.run() instead of os.execv().
* Only add needed options
* Rename reexec parameter to _add_python_opts.
* Rename --no-reexec option to --dont-add-python-opts.
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.
Changes:
* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
"-u -W default -bb -E" options to Python: it's now done by
regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
process. Previously, buffered messages were lost.
* Add --fast-ci and --slow-ci options to libregrtest:
* --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
(skip slowest tests).
* --slow-ci uses a default timeout of 20 minues and "-u all" (run
all tests).
* regrtest header now lists test resources.
* Makefile changes:
* "make test", "make hostrunnertest" and "make coverage-report" now
use --fast-ci option and TESTTIMEOUT variable.
* "make buildbottest" now uses "--slow-ci". Remove options which
became redundant with "--slow-ci".
* "make testall" and "make testuniversal" now use --slow-ci option
and TESTTIMEOUT variable.
* "make testall" now uses "find -exec rm ..." instead of
"find ... -print|xargs rm ...", same as "make clean".
* GitHub Actions workflow:
* Ubuntu and Address Sanitizer jobs now use "make test". Remove
options which became redundant with "--fast-ci".
* Windows jobs now use --fast-ci option.
* Use -j0 to detect the number of CPUs.
* Set Makefile TESTTIMEOUT default to an empty string, since
--slow-ci and --fast-ci use different default timeout. It's now
accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
--timeout=1200 option, redundant with --slow-ci.
* Factorize code listing "bad / env changed / ..." tests.
* Add TestResults.is_all_good() method.
* Move "All 400 tests OK." to the end
* Move "Test suite interrupted by signal SIGINT." to the end.
libregrtest now decodes stdout of test worker processes with the
"backslashreplace" error handler to log corrupted stdout, instead of
failing with an error and not logging the stdout.
On Emscripten and WASI platforms, or if --python command line option
is used, libregrtest now uses a filename for the JSON file.
Emscripten and WASI buildbot workers run the main test process with a
different Python (Linux) which spawns Emscripten/WASI processes using
the command specified in --python command line option. Passing a file
descriptor from the parent process to the child process doesn't work
in this case.
* Add JsonFile and JsonFileType classes
* Add RunTests.json_file_use_filename() method.
* Add a test in test_regrtest on the --python command line option.
* test_regrtest: add parallel=False parameter.
* Split long RunWorkers._runtest() function into sub-functions.
