gh-126925: Modify how iOS test results are gathered (#127592)

Adds a `use_system_log` config item to enable stdout/stderr redirection for
Apple platforms. This log streaming is then used by a new iOS test runner
script, allowing the display of test suite output at runtime. The iOS test
runner script can be used by any Python project, not just the CPython test
suite.
This commit is contained in:
Russell Keith-Magee 2024-12-09 13:28:57 +08:00 committed by GitHub
parent d8d12b37b5
commit 2041a95e68
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
18 changed files with 792 additions and 58 deletions

View file

@ -285,52 +285,42 @@ This will:
* Install the Python iOS framework into the copy of the testbed project; and
* Run the test suite on an "iPhone SE (3rd generation)" simulator.
While the test suite is running, Xcode does not display any console output.
After showing some Xcode build commands, the console output will print ``Testing
started``, and then appear to stop. It will remain in this state until the test
suite completes. On a 2022 M1 MacBook Pro, the test suite takes approximately 12
minutes to run; a couple of extra minutes is required to boot and prepare the
iOS simulator.
On success, the test suite will exit and report successful completion of the
test suite. No output of the Python test suite will be displayed.
On failure, the output of the Python test suite *will* be displayed. This will
show the details of the tests that failed.
test suite. On a 2022 M1 MacBook Pro, the test suite takes approximately 12
minutes to run; a couple of extra minutes is required to compile the testbed
project, and then boot and prepare the iOS simulator.
Debugging test failures
-----------------------
The easiest way to diagnose a single test failure is to open the testbed project
in Xcode and run the tests from there using the "Product > Test" menu item.
Running ``make test`` generates a standalone version of the ``iOS/testbed``
project, and runs the full test suite. It does this using ``iOS/testbed``
itself - the folder is an executable module that can be used to create and run
a clone of the testbed project.
To test in Xcode, you must ensure the testbed project has a copy of a compiled
framework. If you've configured your build with the default install location of
``iOS/Frameworks``, you can copy from that location into the test project. To
test on an ARM64 simulator, run::
You can generate your own standalone testbed instance by running::
$ rm -rf iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator/*
$ cp -r iOS/Frameworks/arm64-iphonesimulator/* iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator
$ python iOS/testbed clone --framework iOS/Frameworks/arm64-iphonesimulator my-testbed
To test on an x86-64 simulator, run::
This invocation assumes that ``iOS/Frameworks/arm64-iphonesimulator`` is the
path to the iOS simulator framework for your platform (ARM64 in this case);
``my-testbed`` is the name of the folder for the new testbed clone.
$ rm -rf iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator/*
$ cp -r iOS/Frameworks/x86_64-iphonesimulator/* iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator
You can then use the ``my-testbed`` folder to run the Python test suite,
passing in any command line arguments you may require. For example, if you're
trying to diagnose a failure in the ``os`` module, you might run::
To test on a physical device::
$ python my-testbed run -- test -W test_os
$ rm -rf iOS/testbed/Python.xcframework/ios-arm64/*
$ cp -r iOS/Frameworks/arm64-iphoneos/* iOS/testbed/Python.xcframework/ios-arm64
This is the equivalent of running ``python -m test -W test_os`` on a desktop
Python build. Any arguments after the ``--`` will be passed to testbed as if
they were arguments to ``python -m`` on a desktop machine.
Alternatively, you can configure your build to install directly into the
testbed project. For a simulator, use::
You can also open the testbed project in Xcode by running::
--enable-framework=$(pwd)/iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator
For a physical device, use::
--enable-framework=$(pwd)/iOS/testbed/Python.xcframework/ios-arm64
$ open my-testbed/iOSTestbed.xcodeproj
This will allow you to use the full Xcode suite of tools for debugging.
Testing on an iOS device
^^^^^^^^^^^^^^^^^^^^^^^^

365
iOS/testbed/__main__.py Normal file
View file

@ -0,0 +1,365 @@
import argparse
import asyncio
import json
import plistlib
import shutil
import subprocess
import sys
from contextlib import asynccontextmanager
from datetime import datetime
from pathlib import Path
DECODE_ARGS = ("UTF-8", "backslashreplace")
# Work around a bug involving sys.exit and TaskGroups
# (https://github.com/python/cpython/issues/101515).
def exit(*args):
raise MySystemExit(*args)
class MySystemExit(Exception):
pass
# All subprocesses are executed through this context manager so that no matter
# what happens, they can always be cancelled from another task, and they will
# always be cleaned up on exit.
@asynccontextmanager
async def async_process(*args, **kwargs):
process = await asyncio.create_subprocess_exec(*args, **kwargs)
try:
yield process
finally:
if process.returncode is None:
# Allow a reasonably long time for Xcode to clean itself up,
# because we don't want stale emulators left behind.
timeout = 10
process.terminate()
try:
await asyncio.wait_for(process.wait(), timeout)
except TimeoutError:
print(
f"Command {args} did not terminate after {timeout} seconds "
f" - sending SIGKILL"
)
process.kill()
# Even after killing the process we must still wait for it,
# otherwise we'll get the warning "Exception ignored in __del__".
await asyncio.wait_for(process.wait(), timeout=1)
async def async_check_output(*args, **kwargs):
async with async_process(
*args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs
) as process:
stdout, stderr = await process.communicate()
if process.returncode == 0:
return stdout.decode(*DECODE_ARGS)
else:
raise subprocess.CalledProcessError(
process.returncode,
args,
stdout.decode(*DECODE_ARGS),
stderr.decode(*DECODE_ARGS),
)
# Return a list of UDIDs associated with booted simulators
async def list_devices():
# List the testing simulators, in JSON format
raw_json = await async_check_output(
"xcrun", "simctl", "--set", "testing", "list", "-j"
)
json_data = json.loads(raw_json)
# Filter out the booted iOS simulators
return [
simulator["udid"]
for runtime, simulators in json_data["devices"].items()
for simulator in simulators
if runtime.split(".")[-1].startswith("iOS") and simulator["state"] == "Booted"
]
async def find_device(initial_devices):
while True:
new_devices = set(await list_devices()).difference(initial_devices)
if len(new_devices) == 0:
await asyncio.sleep(1)
elif len(new_devices) == 1:
udid = new_devices.pop()
print(f"{datetime.now():%Y-%m-%d %H:%M:%S}: New test simulator detected")
print(f"UDID: {udid}")
return udid
else:
exit(f"Found more than one new device: {new_devices}")
async def log_stream_task(initial_devices):
# Wait up to 5 minutes for the build to complete and the simulator to boot.
udid = await asyncio.wait_for(find_device(initial_devices), 5 * 60)
# Stream the iOS device's logs, filtering out messages that come from the
# XCTest test suite (catching NSLog messages from the test method), or
# Python itself (catching stdout/stderr content routed to the system log
# with config->use_system_logger).
args = [
"xcrun",
"simctl",
"--set",
"testing",
"spawn",
udid,
"log",
"stream",
"--style",
"compact",
"--predicate",
(
'senderImagePath ENDSWITH "/iOSTestbedTests.xctest/iOSTestbedTests"'
' OR senderImagePath ENDSWITH "/Python.framework/Python"'
),
]
async with async_process(
*args,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
) as process:
suppress_dupes = False
while line := (await process.stdout.readline()).decode(*DECODE_ARGS):
# The iOS log streamer can sometimes lag; when it does, it outputs
# a warning about messages being dropped... often multiple times.
# Only print the first of these duplicated warnings.
if line.startswith("=== Messages dropped "):
if not suppress_dupes:
suppress_dupes = True
sys.stdout.write(line)
else:
suppress_dupes = False
sys.stdout.write(line)
async def xcode_test(location, simulator):
# Run the test suite on the named simulator
args = [
"xcodebuild",
"test",
"-project",
str(location / "iOSTestbed.xcodeproj"),
"-scheme",
"iOSTestbed",
"-destination",
f"platform=iOS Simulator,name={simulator}",
"-resultBundlePath",
str(location / f"{datetime.now():%Y%m%d-%H%M%S}.xcresult"),
"-derivedDataPath",
str(location / "DerivedData"),
]
async with async_process(
*args,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
) as process:
while line := (await process.stdout.readline()).decode(*DECODE_ARGS):
sys.stdout.write(line)
status = await asyncio.wait_for(process.wait(), timeout=1)
exit(status)
def clone_testbed(
source: Path,
target: Path,
framework: Path,
apps: list[Path],
) -> None:
if target.exists():
print(f"{target} already exists; aborting without creating project.")
sys.exit(10)
if framework is None:
if not (source / "Python.xcframework/ios-arm64_x86_64-simulator/bin").is_dir():
print(
f"The testbed being cloned ({source}) does not contain "
f"a simulator framework. Re-run with --framework"
)
sys.exit(11)
else:
if not framework.is_dir():
print(f"{framework} does not exist.")
sys.exit(12)
elif not (
framework.suffix == ".xcframework"
or (framework / "Python.framework").is_dir()
):
print(
f"{framework} is not an XCframework, "
f"or a simulator slice of a framework build."
)
sys.exit(13)
print("Cloning testbed project...")
shutil.copytree(source, target)
if framework is not None:
if framework.suffix == ".xcframework":
print("Installing XCFramework...")
xc_framework_path = target / "Python.xcframework"
shutil.rmtree(xc_framework_path)
shutil.copytree(framework, xc_framework_path)
else:
print("Installing simulator Framework...")
sim_framework_path = (
target / "Python.xcframework" / "ios-arm64_x86_64-simulator"
)
shutil.rmtree(sim_framework_path)
shutil.copytree(framework, sim_framework_path)
else:
print("Using pre-existing iOS framework.")
for app_src in apps:
print(f"Installing app {app_src.name!r}...")
app_target = target / f"iOSTestbed/app/{app_src.name}"
if app_target.is_dir():
shutil.rmtree(app_target)
shutil.copytree(app_src, app_target)
print(f"Testbed project created in {target}")
def update_plist(testbed_path, args):
# Add the test runner arguments to the testbed's Info.plist file.
info_plist = testbed_path / "iOSTestbed" / "iOSTestbed-Info.plist"
with info_plist.open("rb") as f:
info = plistlib.load(f)
info["TestArgs"] = args
with info_plist.open("wb") as f:
plistlib.dump(info, f)
async def run_testbed(simulator: str, args: list[str]):
location = Path(__file__).parent
print("Updating plist...")
update_plist(location, args)
# Get the list of devices that are booted at the start of the test run.
# The simulator started by the test suite will be detected as the new
# entry that appears on the device list.
initial_devices = await list_devices()
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(log_stream_task(initial_devices))
tg.create_task(xcode_test(location, simulator))
except* MySystemExit as e:
raise SystemExit(*e.exceptions[0].args) from None
except* subprocess.CalledProcessError as e:
# Extract it from the ExceptionGroup so it can be handled by `main`.
raise e.exceptions[0]
def main():
parser = argparse.ArgumentParser(
description=(
"Manages the process of testing a Python project in the iOS simulator."
),
)
subcommands = parser.add_subparsers(dest="subcommand")
clone = subcommands.add_parser(
"clone",
description=(
"Clone the testbed project, copying in an iOS Python framework and"
"any specified application code."
),
help="Clone a testbed project to a new location.",
)
clone.add_argument(
"--framework",
help=(
"The location of the XCFramework (or simulator-only slice of an "
"XCFramework) to use when running the testbed"
),
)
clone.add_argument(
"--app",
dest="apps",
action="append",
default=[],
help="The location of any code to include in the testbed project",
)
clone.add_argument(
"location",
help="The path where the testbed will be cloned.",
)
run = subcommands.add_parser(
"run",
usage="%(prog)s [-h] [--simulator SIMULATOR] -- <test arg> [<test arg> ...]",
description=(
"Run a testbed project. The arguments provided after `--` will be "
"passed to the running iOS process as if they were arguments to "
"`python -m`."
),
help="Run a testbed project",
)
run.add_argument(
"--simulator",
default="iPhone SE (3rd Generation)",
help="The name of the simulator to use (default: 'iPhone SE (3rd Generation)')",
)
try:
pos = sys.argv.index("--")
testbed_args = sys.argv[1:pos]
test_args = sys.argv[pos + 1 :]
except ValueError:
testbed_args = sys.argv[1:]
test_args = []
context = parser.parse_args(testbed_args)
if context.subcommand == "clone":
clone_testbed(
source=Path(__file__).parent,
target=Path(context.location),
framework=Path(context.framework) if context.framework else None,
apps=[Path(app) for app in context.apps],
)
elif context.subcommand == "run":
if test_args:
if not (
Path(__file__).parent / "Python.xcframework/ios-arm64_x86_64-simulator/bin"
).is_dir():
print(
f"Testbed does not contain a compiled iOS framework. Use "
f"`python {sys.argv[0]} clone ...` to create a runnable "
f"clone of this testbed."
)
sys.exit(20)
asyncio.run(
run_testbed(
simulator=context.simulator,
args=test_args,
)
)
else:
print(f"Must specify test arguments (e.g., {sys.argv[0]} run -- test)")
print()
parser.print_help(sys.stderr)
sys.exit(21)
else:
parser.print_help(sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()

View file

@ -263,6 +263,7 @@
runOnlyForDeploymentPostprocessing = 0;
shellPath = /bin/sh;
shellScript = "set -e\n\nmkdir -p \"$CODESIGNING_FOLDER_PATH/python/lib\"\nif [ \"$EFFECTIVE_PLATFORM_NAME\" = \"-iphonesimulator\" ]; then\n echo \"Installing Python modules for iOS Simulator\"\n rsync -au --delete \"$PROJECT_DIR/Python.xcframework/ios-arm64_x86_64-simulator/lib/\" \"$CODESIGNING_FOLDER_PATH/python/lib/\" \nelse\n echo \"Installing Python modules for iOS Device\"\n rsync -au --delete \"$PROJECT_DIR/Python.xcframework/ios-arm64/lib/\" \"$CODESIGNING_FOLDER_PATH/python/lib/\" \nfi\n";
showEnvVarsInLog = 0;
};
607A66562B0F06200010BFC8 /* Prepare Python Binary Modules */ = {
isa = PBXShellScriptBuildPhase;
@ -282,6 +283,7 @@
runOnlyForDeploymentPostprocessing = 0;
shellPath = /bin/sh;
shellScript = "set -e\n\ninstall_dylib () {\n INSTALL_BASE=$1\n FULL_EXT=$2\n\n # The name of the extension file\n EXT=$(basename \"$FULL_EXT\")\n # The location of the extension file, relative to the bundle\n RELATIVE_EXT=${FULL_EXT#$CODESIGNING_FOLDER_PATH/} \n # The path to the extension file, relative to the install base\n PYTHON_EXT=${RELATIVE_EXT/$INSTALL_BASE/}\n # The full dotted name of the extension module, constructed from the file path.\n FULL_MODULE_NAME=$(echo $PYTHON_EXT | cut -d \".\" -f 1 | tr \"/\" \".\"); \n # A bundle identifier; not actually used, but required by Xcode framework packaging\n FRAMEWORK_BUNDLE_ID=$(echo $PRODUCT_BUNDLE_IDENTIFIER.$FULL_MODULE_NAME | tr \"_\" \"-\")\n # The name of the framework folder.\n FRAMEWORK_FOLDER=\"Frameworks/$FULL_MODULE_NAME.framework\"\n\n # If the framework folder doesn't exist, create it.\n if [ ! -d \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER\" ]; then\n echo \"Creating framework for $RELATIVE_EXT\" \n mkdir -p \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER\"\n cp \"$CODESIGNING_FOLDER_PATH/dylib-Info-template.plist\" \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/Info.plist\"\n plutil -replace CFBundleExecutable -string \"$FULL_MODULE_NAME\" \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/Info.plist\"\n plutil -replace CFBundleIdentifier -string \"$FRAMEWORK_BUNDLE_ID\" \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/Info.plist\"\n fi\n \n echo \"Installing binary for $FRAMEWORK_FOLDER/$FULL_MODULE_NAME\" \n mv \"$FULL_EXT\" \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/$FULL_MODULE_NAME\"\n # Create a placeholder .fwork file where the .so was\n echo \"$FRAMEWORK_FOLDER/$FULL_MODULE_NAME\" > ${FULL_EXT%.so}.fwork\n # Create a back reference to the .so file location in the framework\n echo \"${RELATIVE_EXT%.so}.fwork\" > \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/$FULL_MODULE_NAME.origin\" \n}\n\nPYTHON_VER=$(ls -1 \"$CODESIGNING_FOLDER_PATH/python/lib\")\necho \"Install Python $PYTHON_VER standard library extension modules...\"\nfind \"$CODESIGNING_FOLDER_PATH/python/lib/$PYTHON_VER/lib-dynload\" -name \"*.so\" | while read FULL_EXT; do\n install_dylib python/lib/$PYTHON_VER/lib-dynload/ \"$FULL_EXT\"\ndone\necho \"Install app package extension modules...\"\nfind \"$CODESIGNING_FOLDER_PATH/app_packages\" -name \"*.so\" | while read FULL_EXT; do\n install_dylib app_packages/ \"$FULL_EXT\"\ndone\necho \"Install app extension modules...\"\nfind \"$CODESIGNING_FOLDER_PATH/app\" -name \"*.so\" | while read FULL_EXT; do\n install_dylib app/ \"$FULL_EXT\"\ndone\n\n# Clean up dylib template \nrm -f \"$CODESIGNING_FOLDER_PATH/dylib-Info-template.plist\"\necho \"Signing frameworks as $EXPANDED_CODE_SIGN_IDENTITY_NAME ($EXPANDED_CODE_SIGN_IDENTITY)...\"\nfind \"$CODESIGNING_FOLDER_PATH/Frameworks\" -name \"*.framework\" -exec /usr/bin/codesign --force --sign \"$EXPANDED_CODE_SIGN_IDENTITY\" ${OTHER_CODE_SIGN_FLAGS:-} -o runtime --timestamp=none --preserve-metadata=identifier,entitlements,flags --generate-entitlement-der \"{}\" \\; \n";
showEnvVarsInLog = 0;
};
/* End PBXShellScriptBuildPhase section */

View file

@ -50,6 +50,8 @@
// Enforce UTF-8 encoding for stderr, stdout, file-system encoding and locale.
// See https://docs.python.org/3/library/os.html#python-utf-8-mode.
preconfig.utf8_mode = 1;
// Use the system logger for stdout/err
config.use_system_logger = 1;
// Don't buffer stdio. We want output to appears in the log immediately
config.buffered_stdio = 0;
// Don't write bytecode; we can't modify the app bundle