gh-137973: Add a non-parallel test plan to the iOS testbed project (#138018)

Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.

.gitignore

@@ -80,7 +80,6 @@ iOS/testbed/Python.xcframework/ios-*/lib
 iOS/testbed/Python.xcframework/ios-*/Python.framework
 iOS/testbed/iOSTestbed.xcodeproj/project.xcworkspace
 iOS/testbed/iOSTestbed.xcodeproj/xcuserdata
-iOS/testbed/iOSTestbed.xcodeproj/xcshareddata
 Mac/Makefile
 Mac/PythonLauncher/Info.plist
 Mac/PythonLauncher/Makefile

Doc/using/ios.rst

@@ -374,6 +374,17 @@ You can also open the testbed project in Xcode by running:
 
 This will allow you to use the full Xcode suite of tools for debugging.
 
+The arguments used to run the test suite are defined as part of the test plan.
+To modify the test plan, select the test plan node of the project tree (it
+should be the first child of the root node), and select the "Configurations"
+tab. Modify the "Arguments Passed On Launch" value to change the testing
+arguments.
+
+The test plan also disables parallel testing, and specifies the use of the
+``iOSTestbed.lldbinit`` file for providing configuration of the debugger. The
+default debugger configuration disables automatic breakpoints on the
+``SIGINT``, ``SIGUSR1``, ``SIGUSR2``, and ``SIGXFSZ`` signals.
+
 App Store Compliance
 ====================
 

Misc/NEWS.d/next/Tools-Demos/2025-08-21-14-04-50.gh-issue-137873.qxffLt.rst

@@ -0,0 +1,3 @@
+The iOS test runner has been simplified, resolving some issues that have
+been observed using the runner in GitHub Actions and Azure Pipelines test
+environments.

iOS/README.rst

@@ -293,7 +293,7 @@ project, and then boot and prepare the iOS simulator.
 Debugging test failures
 -----------------------
 
-Running ``make test`` generates a standalone version of the ``iOS/testbed``
+Running ``make testios`` generates a standalone version of the ``iOS/testbed``
 project, and runs the full test suite. It does this using ``iOS/testbed``
 itself - the folder is an executable module that can be used to create and run
 a clone of the testbed project.
@@ -316,12 +316,26 @@ This is the equivalent of running ``python -m test -W test_os`` on a desktop
 Python build. Any arguments after the ``--`` will be passed to testbed as if
 they were arguments to ``python -m`` on a desktop machine.
 
+Testing in Xcode
+^^^^^^^^^^^^^^^^
+
 You can also open the testbed project in Xcode by running::
 
     $ open my-testbed/iOSTestbed.xcodeproj
 
 This will allow you to use the full Xcode suite of tools for debugging.
 
+The arguments used to run the test suite are defined as part of the test plan.
+To modify the test plan, select the test plan node of the project tree (it
+should be the first child of the root node), and select the "Configurations"
+tab. Modify the "Arguments Passed On Launch" value to change the testing
+arguments.
+
+The test plan also disables parallel testing, and specifies the use of the
+``iOSTestbed.lldbinit`` file for providing configuration of the debugger. The
+default debugger configuration disables automatic breakpoints on the
+``SIGINT``, ``SIGUSR1``, ``SIGUSR2``, and ``SIGXFSZ`` signals.
+
 Testing on an iOS device
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -336,40 +350,3 @@ select the root node of the project tree (labeled "iOSTestbed"), then the
 (this will likely be your own name), and plug in a physical device to your
 macOS machine with a USB cable. You should then be able to select your physical
 device from the list of targets in the pulldown in the Xcode titlebar.
-
-Running specific tests
-^^^^^^^^^^^^^^^^^^^^^^
-
-As the test suite is being executed on an iOS simulator, it is not possible to
-pass in command line arguments to configure test suite operation. To work
-around this limitation, the arguments that would normally be passed as command
-line arguments are configured as part of the ``iOSTestbed-Info.plist`` file
-that is used to configure the iOS testbed app. In this file, the ``TestArgs``
-key is an array containing the arguments that would be passed to ``python -m``
-on the command line (including ``test`` in position 0, the name of the test
-module to be executed).
-
-Disabling automated breakpoints
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-By default, Xcode will inserts an automatic breakpoint whenever a signal is
-raised. The Python test suite raises many of these signals as part of normal
-operation; unless you are trying to diagnose an issue with signals, the
-automatic breakpoints can be inconvenient. However, they can be disabled by
-creating a symbolic breakpoint that is triggered at the start of the test run.
-
-Select "Debug > Breakpoints > Create Symbolic Breakpoint" from the Xcode menu, and
-populate the new brewpoint with the following details:
-
-* **Name**: IgnoreSignals
-* **Symbol**: UIApplicationMain
-* **Action**: Add debugger commands for:
-  - ``process handle SIGINT -n true -p true -s false``
-  - ``process handle SIGUSR1 -n true -p true -s false``
-  - ``process handle SIGUSR2 -n true -p true -s false``
-  - ``process handle SIGXFSZ -n true -p true -s false``
-* Check the "Automatically continue after evaluating" box.
-
-All other details can be left blank. When the process executes the
-``UIApplicationMain`` entry point, the breakpoint will trigger, run the debugger
-commands to disable the automatic breakpoints, and automatically resume.

iOS/testbed/__main__.py

@@ -1,134 +1,29 @@
 import argparse
-import asyncio
-import fcntl
 import json
-import os
-import plistlib
 import re
 import shutil
 import subprocess
 import sys
-import tempfile
-from contextlib import asynccontextmanager
-from datetime import datetime
 from pathlib import Path
 
 
 DECODE_ARGS = ("UTF-8", "backslashreplace")
 
 # The system log prefixes each line:
-#   2025-01-17 16:14:29.090 Df iOSTestbed[23987:1fd393b4] (Python) ...
+#   2025-01-17 16:14:29.093742+0800 iOSTestbed[23987:1fd393b4] ...
-#   2025-01-17 16:14:29.090 E  iOSTestbed[23987:1fd393b4] (Python) ...
+#   2025-01-17 16:14:29.093742+0800 iOSTestbed[23987:1fd393b4] ...
 
 LOG_PREFIX_REGEX = re.compile(
     r"^\d{4}-\d{2}-\d{2}"  # YYYY-MM-DD
-    r"\s+\d+:\d{2}:\d{2}\.\d+"  # HH:MM:SS.sss
+    r"\s+\d+:\d{2}:\d{2}\.\d+\+\d{4}"  # HH:MM:SS.ssssss+ZZZZ
-    r"\s+\w+"  # Df/E
     r"\s+iOSTestbed\[\d+:\w+\]"  # Process/thread ID
-    r"\s+\(Python\)\s"  # Logger name
-)
-
-
-# Work around a bug involving sys.exit and TaskGroups
-# (https://github.com/python/cpython/issues/101515).
-def exit(*args):
-    raise MySystemExit(*args)
-
-
-class MySystemExit(Exception):
-    pass
-
-
-class SimulatorLock:
-    # An fcntl-based filesystem lock that can be used to ensure that
-    def __init__(self, timeout):
-        self.filename = Path(tempfile.gettempdir()) / "python-ios-testbed"
-        self.timeout = timeout
-
-        self.fd = None
-
-    async def acquire(self):
-        # Ensure the lockfile exists
-        self.filename.touch(exist_ok=True)
-
-        # Try `timeout` times to acquire the lock file, with a 1 second pause
-        # between each attempt. Report status every 10 seconds.
-        for i in range(0, self.timeout):
-            try:
-                fd = os.open(self.filename, os.O_RDWR | os.O_TRUNC, 0o644)
-                fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
-            except OSError:
-                os.close(fd)
-                if i % 10 == 0:
-                    print("... waiting", flush=True)
-                await asyncio.sleep(1)
-            else:
-                self.fd = fd
-                return
-
-        # If we reach the end of the loop, we've exceeded the allowed number of
-        # attempts.
-        raise ValueError("Unable to obtain lock on iOS simulator creation")
-
-    def release(self):
-        # If a lock is held, release it.
-        if self.fd is not None:
-            # Release the lock.
-            fcntl.flock(self.fd, fcntl.LOCK_UN)
-            os.close(self.fd)
-            self.fd = None
-
-
-# All subprocesses are executed through this context manager so that no matter
-# what happens, they can always be cancelled from another task, and they will
-# always be cleaned up on exit.
-@asynccontextmanager
-async def async_process(*args, **kwargs):
-    process = await asyncio.create_subprocess_exec(*args, **kwargs)
-    try:
-        yield process
-    finally:
-        if process.returncode is None:
-            # Allow a reasonably long time for Xcode to clean itself up,
-            # because we don't want stale emulators left behind.
-            timeout = 10
-            process.terminate()
-            try:
-                await asyncio.wait_for(process.wait(), timeout)
-            except TimeoutError:
-                print(
-                    f"Command {args} did not terminate after {timeout} seconds "
-                    f" - sending SIGKILL"
-                )
-                process.kill()
-
-                # Even after killing the process we must still wait for it,
-                # otherwise we'll get the warning "Exception ignored in __del__".
-                await asyncio.wait_for(process.wait(), timeout=1)
-
-
-async def async_check_output(*args, **kwargs):
-    async with async_process(
-        *args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs
-    ) as process:
-        stdout, stderr = await process.communicate()
-        if process.returncode == 0:
-            return stdout.decode(*DECODE_ARGS)
-        else:
-            raise subprocess.CalledProcessError(
-                process.returncode,
-                args,
-                stdout.decode(*DECODE_ARGS),
-                stderr.decode(*DECODE_ARGS),
 )
 
 
 # Select a simulator device to use.
-async def select_simulator_device():
+def select_simulator_device():
     # List the testing simulators, in JSON format
-    raw_json = await async_check_output(
+    raw_json = subprocess.check_output(["xcrun", "simctl", "list", "-j"])
-        "xcrun", "simctl", "list", "-j"
-    )
     json_data = json.loads(raw_json)
 
     # Any device will do; we'll look for "SE" devices - but the name isn't
@@ -145,7 +40,10 @@ async def select_simulator_device():
         for devicetype in json_data["devicetypes"]
         if devicetype["productFamily"] == "iPhone"
         and (
-            ("iPhone " in devicetype["name"] and devicetype["name"].endswith("e"))
+            (
+                "iPhone " in devicetype["name"]
+                and devicetype["name"].endswith("e")
+            )
             or "iPhone SE " in devicetype["name"]
         )
     )
@@ -153,126 +51,41 @@ async def select_simulator_device():
     return se_simulators[-1][1]
 
 
-# Return a list of UDIDs associated with booted simulators
+def xcode_test(location, simulator, verbose):
-async def list_devices():
+    # Build and run the test suite on the named simulator.
-    try:
-        # List the testing simulators, in JSON format
-        raw_json = await async_check_output(
-            "xcrun", "simctl", "--set", "testing", "list", "-j"
-        )
-        json_data = json.loads(raw_json)
-
-        # Filter out the booted iOS simulators
-        return [
-            simulator["udid"]
-            for runtime, simulators in json_data["devices"].items()
-            for simulator in simulators
-            if runtime.split(".")[-1].startswith("iOS") and simulator["state"] == "Booted"
-        ]
-    except subprocess.CalledProcessError as e:
-        # If there's no ~/Library/Developer/XCTestDevices folder (which is the
-        # case on fresh installs, and in some CI environments), `simctl list`
-        # returns error code 1, rather than an empty list. Handle that case,
-        # but raise all other errors.
-        if e.returncode == 1:
-            return []
-        else:
-            raise
-
-
-async def find_device(initial_devices, lock):
-    while True:
-        new_devices = set(await list_devices()).difference(initial_devices)
-        if len(new_devices) == 0:
-            await asyncio.sleep(1)
-        elif len(new_devices) == 1:
-            udid = new_devices.pop()
-            print(f"{datetime.now():%Y-%m-%d %H:%M:%S}: New test simulator detected")
-            print(f"UDID: {udid}", flush=True)
-            lock.release()
-            return udid
-        else:
-            exit(f"Found more than one new device: {new_devices}")
-
-
-async def log_stream_task(initial_devices, lock):
-    # Wait up to 5 minutes for the build to complete and the simulator to boot.
-    udid = await asyncio.wait_for(find_device(initial_devices, lock), 5 * 60)
-
-    # Stream the iOS device's logs, filtering out messages that come from the
-    # XCTest test suite (catching NSLog messages from the test method), or
-    # Python itself (catching stdout/stderr content routed to the system log
-    # with config->use_system_logger).
     args = [
-        "xcrun",
-        "simctl",
-        "--set",
-        "testing",
-        "spawn",
-        udid,
-        "log",
-        "stream",
-        "--style",
-        "compact",
-        "--predicate",
-        (
-            'senderImagePath ENDSWITH "/iOSTestbedTests.xctest/iOSTestbedTests"'
-            ' OR senderImagePath ENDSWITH "/Python.framework/Python"'
-        ),
-    ]
-
-    async with async_process(
-        *args,
-        stdout=subprocess.PIPE,
-        stderr=subprocess.STDOUT,
-    ) as process:
-        suppress_dupes = False
-        while line := (await process.stdout.readline()).decode(*DECODE_ARGS):
-            # Strip the prefix from each log line
-            line = LOG_PREFIX_REGEX.sub("", line)
-            # The iOS log streamer can sometimes lag; when it does, it outputs
-            # a warning about messages being dropped... often multiple times.
-            # Only print the first of these duplicated warnings.
-            if line.startswith("=== Messages dropped "):
-                if not suppress_dupes:
-                    suppress_dupes = True
-                    sys.stdout.write(line)
-            else:
-                suppress_dupes = False
-                sys.stdout.write(line)
-            sys.stdout.flush()
-
-
-async def xcode_test(location, simulator, verbose):
-    # Run the test suite on the named simulator
-    print("Starting xcodebuild...", flush=True)
-    args = [
-        "xcodebuild",
-        "test",
         "-project",
         str(location / "iOSTestbed.xcodeproj"),
         "-scheme",
         "iOSTestbed",
         "-destination",
         f"platform=iOS Simulator,name={simulator}",
-        "-resultBundlePath",
-        str(location / f"{datetime.now():%Y%m%d-%H%M%S}.xcresult"),
         "-derivedDataPath",
         str(location / "DerivedData"),
     ]
-    if not verbose:
+    verbosity_args = [] if verbose else ["-quiet"]
-        args += ["-quiet"]
 
-    async with async_process(
+    print("Building test project...")
-        *args,
+    subprocess.run(
+        ["xcodebuild", "build-for-testing"] + args + verbosity_args,
+        check=True,
+    )
+
+    print("Running test project...")
+    # Test execution *can't* be run -quiet; verbose mode
+    # is how we see the output of the test output.
+    process = subprocess.Popen(
+        ["xcodebuild", "test-without-building"] + args,
         stdout=subprocess.PIPE,
         stderr=subprocess.STDOUT,
-    ) as process:
+    )
-        while line := (await process.stdout.readline()).decode(*DECODE_ARGS):
+    while line := (process.stdout.readline()).decode(*DECODE_ARGS):
+        # Strip the timestamp/process prefix from each log line
+        line = LOG_PREFIX_REGEX.sub("", line)
         sys.stdout.write(line)
         sys.stdout.flush()
 
-        status = await asyncio.wait_for(process.wait(), timeout=1)
+    status = process.wait(timeout=5)
     exit(status)
 
 
@@ -310,7 +123,7 @@ def clone_testbed(
             sys.exit(13)
 
     print("Cloning testbed project:")
-    print(f"  Cloning {source}...", end="", flush=True)
+    print(f"  Cloning {source}...", end="")
     shutil.copytree(source, target, symlinks=True)
     print(" done")
 
@@ -318,7 +131,7 @@ def clone_testbed(
     sim_framework_path = xc_framework_path / "ios-arm64_x86_64-simulator"
     if framework is not None:
         if framework.suffix == ".xcframework":
-            print("  Installing XCFramework...", end="", flush=True)
+            print("  Installing XCFramework...", end="")
             if xc_framework_path.is_dir():
                 shutil.rmtree(xc_framework_path)
             else:
@@ -328,7 +141,7 @@ def clone_testbed(
             )
             print(" done")
         else:
-            print("  Installing simulator framework...", end="", flush=True)
+            print("  Installing simulator framework...", end="")
             if sim_framework_path.is_dir():
                 shutil.rmtree(sim_framework_path)
             else:
@@ -344,10 +157,9 @@ def clone_testbed(
         ):
             # XCFramework is a relative symlink. Rewrite the symlink relative
             # to the new location.
-            print("  Rewriting symlink to XCframework...", end="", flush=True)
+            print("  Rewriting symlink to XCframework...", end="")
             orig_xc_framework_path = (
-                source
+                source / xc_framework_path.readlink()
-                / xc_framework_path.readlink()
             ).resolve()
             xc_framework_path.unlink()
             xc_framework_path.symlink_to(
@@ -360,13 +172,11 @@ def clone_testbed(
             sim_framework_path.is_symlink()
             and not sim_framework_path.readlink().is_absolute()
         ):
-            print("  Rewriting symlink to simulator framework...", end="", flush=True)
+            print("  Rewriting symlink to simulator framework...", end="")
             # Simulator framework is a relative symlink. Rewrite the symlink
             # relative to the new location.
             orig_sim_framework_path = (
-                source
+                source / "Python.XCframework" / sim_framework_path.readlink()
-                / "Python.XCframework"
-                / sim_framework_path.readlink()
             ).resolve()
             sim_framework_path.unlink()
             sim_framework_path.symlink_to(
@@ -379,7 +189,7 @@ def clone_testbed(
             print("  Using pre-existing iOS framework.")
 
     for app_src in apps:
-        print(f"  Installing app {app_src.name!r}...", end="", flush=True)
+        print(f"  Installing app {app_src.name!r}...", end="")
         app_target = target / f"iOSTestbed/app/{app_src.name}"
         if app_target.is_dir():
             shutil.rmtree(app_target)
@@ -389,54 +199,31 @@ def clone_testbed(
     print(f"Successfully cloned testbed: {target.resolve()}")
 
 
-def update_plist(testbed_path, args):
+def update_test_plan(testbed_path, args):
-    # Add the test runner arguments to the testbed's Info.plist file.
+    # Modify the test plan to use the requested test arguments.
-    info_plist = testbed_path / "iOSTestbed" / "iOSTestbed-Info.plist"
+    test_plan_path = testbed_path / "iOSTestbed.xctestplan"
-    with info_plist.open("rb") as f:
+    with test_plan_path.open("r", encoding="utf-8") as f:
-        info = plistlib.load(f)
+        test_plan = json.load(f)
 
-    info["TestArgs"] = args
+    test_plan["defaultOptions"]["commandLineArgumentEntries"] = [
+        {"argument": arg} for arg in args
+    ]
 
-    with info_plist.open("wb") as f:
+    with test_plan_path.open("w", encoding="utf-8") as f:
-        plistlib.dump(info, f)
+        json.dump(test_plan, f, indent=2)
 
 
-async def run_testbed(simulator: str | None, args: list[str], verbose: bool=False):
+def run_testbed(simulator: str | None, args: list[str], verbose: bool = False):
     location = Path(__file__).parent
-    print("Updating plist...", end="", flush=True)
+    print("Updating test plan...", end="")
-    update_plist(location, args)
+    update_test_plan(location, args)
-    print(" done.", flush=True)
+    print(" done.")
 
     if simulator is None:
-        simulator = await select_simulator_device()
+        simulator = select_simulator_device()
-    print(f"Running test on {simulator}", flush=True)
+    print(f"Running test on {simulator}")
 
-    # We need to get an exclusive lock on simulator creation, to avoid issues
+    xcode_test(location, simulator=simulator, verbose=verbose)
-    # with multiple simulators starting and being unable to tell which
-    # simulator is due to which testbed instance. See
-    # https://github.com/python/cpython/issues/130294 for details. Wait up to
-    # 10 minutes for a simulator to boot.
-    print("Obtaining lock on simulator creation...", flush=True)
-    simulator_lock = SimulatorLock(timeout=10*60)
-    await simulator_lock.acquire()
-    print("Simulator lock acquired.", flush=True)
-
-    # Get the list of devices that are booted at the start of the test run.
-    # The simulator started by the test suite will be detected as the new
-    # entry that appears on the device list.
-    initial_devices = await list_devices()
-
-    try:
-        async with asyncio.TaskGroup() as tg:
-            tg.create_task(log_stream_task(initial_devices, simulator_lock))
-            tg.create_task(xcode_test(location, simulator=simulator, verbose=verbose))
-    except* MySystemExit as e:
-        raise SystemExit(*e.exceptions[0].args) from None
-    except* subprocess.CalledProcessError as e:
-        # Extract it from the ExceptionGroup so it can be handled by `main`.
-        raise e.exceptions[0]
-    finally:
-        simulator_lock.release()
 
 
 def main():
@@ -488,12 +275,16 @@ def main():
     run.add_argument(
         "--simulator",
         help=(
-            "The name of the simulator to use (eg: 'iPhone 16e'). Defaults to ",
+            "The name of the simulator to use (eg: 'iPhone 16e'). Defaults to "
-            "the most recently released 'entry level' iPhone device."
+            "the most recently released 'entry level' iPhone device. Device "
-        )
+            "architecture and OS version can also be specified; e.g., "
+            "`--simulator 'iPhone 16 Pro,arch=arm64,OS=26.0'` would run on "
+            "an ARM64 iPhone 16 Pro simulator running iOS 26.0."
+        ),
     )
     run.add_argument(
-        "-v", "--verbose",
+        "-v",
+        "--verbose",
         action="store_true",
         help="Enable verbose output",
     )
@@ -512,13 +303,16 @@ def main():
         clone_testbed(
             source=Path(__file__).parent.resolve(),
             target=Path(context.location).resolve(),
-            framework=Path(context.framework).resolve() if context.framework else None,
+            framework=Path(context.framework).resolve()
+            if context.framework
+            else None,
             apps=[Path(app) for app in context.apps],
         )
     elif context.subcommand == "run":
         if test_args:
             if not (
-                Path(__file__).parent / "Python.xcframework/ios-arm64_x86_64-simulator/bin"
+                Path(__file__).parent
+                / "Python.xcframework/ios-arm64_x86_64-simulator/bin"
             ).is_dir():
                 print(
                     f"Testbed does not contain a compiled iOS framework. Use "
@@ -527,15 +321,15 @@ def main():
                 )
                 sys.exit(20)
 
-            asyncio.run(
             run_testbed(
                 simulator=context.simulator,
                 verbose=context.verbose,
                 args=test_args,
             )
-            )
         else:
-            print(f"Must specify test arguments (e.g., {sys.argv[0]} run -- test)")
+            print(
+                f"Must specify test arguments (e.g., {sys.argv[0]} run -- test)"
+            )
             print()
             parser.print_help(sys.stderr)
             sys.exit(21)

iOS/testbed/iOSTestbed.lldbinit

@@ -0,0 +1,4 @@
+process handle SIGINT -n true -p true -s false
+process handle SIGUSR1 -n true -p true -s false
+process handle SIGUSR2 -n true -p true -s false
+process handle SIGXFSZ -n true -p true -s false

iOS/testbed/iOSTestbed.xcodeproj/project.pbxproj

@@ -70,6 +70,7 @@
 		607A66592B0F08600010BFC8 /* iOSTestbed-Info.plist */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.plist.xml; path = "iOSTestbed-Info.plist"; sourceTree = "<group>"; };
 		608619532CB77BA900F46182 /* app_packages */ = {isa = PBXFileReference; lastKnownFileType = folder; path = app_packages; sourceTree = "<group>"; };
 		608619552CB7819B00F46182 /* app */ = {isa = PBXFileReference; lastKnownFileType = folder; path = app; sourceTree = "<group>"; };
+		60FE0EFB2E56BB6D00524F87 /* iOSTestbed.xctestplan */ = {isa = PBXFileReference; lastKnownFileType = text; path = iOSTestbed.xctestplan; sourceTree = "<group>"; };
 /* End PBXFileReference section */
 
 /* Begin PBXFrameworksBuildPhase section */
@@ -95,6 +96,7 @@
 		607A66092B0EFA380010BFC8 = {
 			isa = PBXGroup;
 			children = (
+				60FE0EFB2E56BB6D00524F87 /* iOSTestbed.xctestplan */,
 				607A664A2B0EFB310010BFC8 /* Python.xcframework */,
 				607A66142B0EFA380010BFC8 /* iOSTestbed */,
 				607A66302B0EFA3A0010BFC8 /* iOSTestbedTests */,
@@ -379,7 +381,7 @@
 				GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE;
 				GCC_WARN_UNUSED_FUNCTION = YES;
 				GCC_WARN_UNUSED_VARIABLE = YES;
-				IPHONEOS_DEPLOYMENT_TARGET = 12.0;
+				IPHONEOS_DEPLOYMENT_TARGET = 13.0;
 				LOCALIZATION_PREFERS_STRING_CATALOGS = YES;
 				MTL_ENABLE_DEBUG_INFO = INCLUDE_SOURCE;
 				MTL_FAST_MATH = YES;
@@ -434,7 +436,7 @@
 				GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE;
 				GCC_WARN_UNUSED_FUNCTION = YES;
 				GCC_WARN_UNUSED_VARIABLE = YES;
-				IPHONEOS_DEPLOYMENT_TARGET = 12.0;
+				IPHONEOS_DEPLOYMENT_TARGET = 13.0;
 				LOCALIZATION_PREFERS_STRING_CATALOGS = YES;
 				MTL_ENABLE_DEBUG_INFO = NO;
 				MTL_FAST_MATH = YES;
@@ -460,7 +462,7 @@
 				INFOPLIST_KEY_UIMainStoryboardFile = Main;
 				INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
 				INFOPLIST_KEY_UISupportedInterfaceOrientations_iPhone = "UIInterfaceOrientationPortrait UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
-				IPHONEOS_DEPLOYMENT_TARGET = 12.0;
+				IPHONEOS_DEPLOYMENT_TARGET = 13.0;
 				LD_RUNPATH_SEARCH_PATHS = (
 					"$(inherited)",
 					"@executable_path/Frameworks",
@@ -491,7 +493,7 @@
 				INFOPLIST_KEY_UIMainStoryboardFile = Main;
 				INFOPLIST_KEY_UISupportedInterfaceOrientations_iPad = "UIInterfaceOrientationPortrait UIInterfaceOrientationPortraitUpsideDown UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
 				INFOPLIST_KEY_UISupportedInterfaceOrientations_iPhone = "UIInterfaceOrientationPortrait UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight";
-				IPHONEOS_DEPLOYMENT_TARGET = 12.0;
+				IPHONEOS_DEPLOYMENT_TARGET = 13.0;
 				LD_RUNPATH_SEARCH_PATHS = (
 					"$(inherited)",
 					"@executable_path/Frameworks",
@@ -514,7 +516,7 @@
 				DEVELOPMENT_TEAM = 3HEZE76D99;
 				GENERATE_INFOPLIST_FILE = YES;
 				HEADER_SEARCH_PATHS = "\"$(BUILT_PRODUCTS_DIR)/Python.framework/Headers\"";
-				IPHONEOS_DEPLOYMENT_TARGET = 12.0;
+				IPHONEOS_DEPLOYMENT_TARGET = 13.0;
 				MARKETING_VERSION = 1.0;
 				PRODUCT_BUNDLE_IDENTIFIER = org.python.iOSTestbedTests;
 				PRODUCT_NAME = "$(TARGET_NAME)";
@@ -534,7 +536,7 @@
 				DEVELOPMENT_TEAM = 3HEZE76D99;
 				GENERATE_INFOPLIST_FILE = YES;
 				HEADER_SEARCH_PATHS = "\"$(BUILT_PRODUCTS_DIR)/Python.framework/Headers\"";
-				IPHONEOS_DEPLOYMENT_TARGET = 12.0;
+				IPHONEOS_DEPLOYMENT_TARGET = 13.0;
 				MARKETING_VERSION = 1.0;
 				PRODUCT_BUNDLE_IDENTIFIER = org.python.iOSTestbedTests;
 				PRODUCT_NAME = "$(TARGET_NAME)";

iOS/testbed/iOSTestbed.xcodeproj/xcshareddata/xcschemes/iOSTestbed.xcscheme

@@ -0,0 +1,97 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<Scheme
+   LastUpgradeVersion = "1640"
+   version = "1.7">
+   <BuildAction
+      parallelizeBuildables = "YES"
+      buildImplicitDependencies = "YES"
+      buildArchitectures = "Automatic">
+      <BuildActionEntries>
+         <BuildActionEntry
+            buildForTesting = "YES"
+            buildForRunning = "YES"
+            buildForProfiling = "YES"
+            buildForArchiving = "YES"
+            buildForAnalyzing = "YES">
+            <BuildableReference
+               BuildableIdentifier = "primary"
+               BlueprintIdentifier = "607A66112B0EFA380010BFC8"
+               BuildableName = "iOSTestbed.app"
+               BlueprintName = "iOSTestbed"
+               ReferencedContainer = "container:iOSTestbed.xcodeproj">
+            </BuildableReference>
+         </BuildActionEntry>
+      </BuildActionEntries>
+   </BuildAction>
+   <TestAction
+      buildConfiguration = "Debug"
+      selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
+      selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
+      customLLDBInitFile = "/Users/rkm/projects/pyspamsum/localtest/iOSTestbed.lldbinit"
+      shouldUseLaunchSchemeArgsEnv = "YES">
+      <TestPlans>
+         <TestPlanReference
+            reference = "container:iOSTestbed.xctestplan"
+            default = "YES">
+         </TestPlanReference>
+      </TestPlans>
+      <Testables>
+         <TestableReference
+            skipped = "NO"
+            parallelizable = "NO">
+            <BuildableReference
+               BuildableIdentifier = "primary"
+               BlueprintIdentifier = "607A662C2B0EFA3A0010BFC8"
+               BuildableName = "iOSTestbedTests.xctest"
+               BlueprintName = "iOSTestbedTests"
+               ReferencedContainer = "container:iOSTestbed.xcodeproj">
+            </BuildableReference>
+         </TestableReference>
+      </Testables>
+   </TestAction>
+   <LaunchAction
+      buildConfiguration = "Debug"
+      selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
+      selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
+      launchStyle = "0"
+      useCustomWorkingDirectory = "NO"
+      ignoresPersistentStateOnLaunch = "NO"
+      debugDocumentVersioning = "YES"
+      debugServiceExtension = "internal"
+      allowLocationSimulation = "YES">
+      <BuildableProductRunnable
+         runnableDebuggingMode = "0">
+         <BuildableReference
+            BuildableIdentifier = "primary"
+            BlueprintIdentifier = "607A66112B0EFA380010BFC8"
+            BuildableName = "iOSTestbed.app"
+            BlueprintName = "iOSTestbed"
+            ReferencedContainer = "container:iOSTestbed.xcodeproj">
+         </BuildableReference>
+      </BuildableProductRunnable>
+   </LaunchAction>
+   <ProfileAction
+      buildConfiguration = "Release"
+      shouldUseLaunchSchemeArgsEnv = "YES"
+      savedToolIdentifier = ""
+      useCustomWorkingDirectory = "NO"
+      debugDocumentVersioning = "YES">
+      <BuildableProductRunnable
+         runnableDebuggingMode = "0">
+         <BuildableReference
+            BuildableIdentifier = "primary"
+            BlueprintIdentifier = "607A66112B0EFA380010BFC8"
+            BuildableName = "iOSTestbed.app"
+            BlueprintName = "iOSTestbed"
+            ReferencedContainer = "container:iOSTestbed.xcodeproj">
+         </BuildableReference>
+      </BuildableProductRunnable>
+   </ProfileAction>
+   <AnalyzeAction
+      buildConfiguration = "Debug">
+   </AnalyzeAction>
+   <ArchiveAction
+      buildConfiguration = "Release"
+      revealArchiveInOrganizer = "YES">
+   </ArchiveAction>
+</Scheme>

iOS/testbed/iOSTestbed.xctestplan

@@ -0,0 +1,46 @@
+{
+  "configurations" : [
+    {
+      "id" : "F5A95CE4-1ADE-4A6E-A0E1-CDBAE26DF0C5",
+      "name" : "Test Scheme Action",
+      "options" : {
+
+      }
+    }
+  ],
+  "defaultOptions" : {
+    "commandLineArgumentEntries" : [
+      {
+        "argument" : "test"
+      },
+      {
+        "argument" : "-uall"
+      },
+      {
+        "argument" : "--single-process"
+      },
+      {
+        "argument" : "--rerun"
+      },
+      {
+        "argument" : "-W"
+      }
+    ],
+    "targetForVariableExpansion" : {
+      "containerPath" : "container:iOSTestbed.xcodeproj",
+      "identifier" : "607A66112B0EFA380010BFC8",
+      "name" : "iOSTestbed"
+    }
+  },
+  "testTargets" : [
+    {
+      "parallelizable" : false,
+      "target" : {
+        "containerPath" : "container:iOSTestbed.xcodeproj",
+        "identifier" : "607A662C2B0EFA3A0010BFC8",
+        "name" : "iOSTestbedTests"
+      }
+    }
+  ],
+  "version" : 1
+}

iOS/testbed/iOSTestbed/iOSTestbed-Info.plist

@@ -40,18 +40,6 @@
 		<string>UIInterfaceOrientationPortraitUpsideDown</string>
 		<string>UIInterfaceOrientationLandscapeLeft</string>
 		<string>UIInterfaceOrientationLandscapeRight</string>
-	</array>
-	<key>TestArgs</key>
-	<array>
-		<string>test</string> <!-- Invoke "python -m test" -->
-        <string>-uall</string> <!-- Enable all resources -->
-        <string>--single-process</string> <!-- always run all tests sequentially in a single process -->
-        <string>--rerun</string> <!-- Re-run failed tests in verbose mode -->
-        <string>-W</string> <!-- Display test output on failure -->
-		<!-- To run a subset of tests, add the test names below; e.g.,
-        <string>test_os</string>
-        <string>test_sys</string>
-		-->
 	</array>
 	<key>UIApplicationSceneManifest</key>
 	<dict>

iOS/testbed/iOSTestbedTests/iOSTestbedTests.m

@@ -38,16 +38,20 @@
     // Arguments to pass into the test suite runner.
     // argv[0] must identify the process; any subsequent arg
     // will be handled as if it were an argument to `python -m test`
-    test_args = [[NSBundle mainBundle] objectForInfoDictionaryKey:@"TestArgs"];
+    // The processInfo arguments contain the binary that is running,
+    // followed by the arguments defined in the test plan. This means:
+    //    run_module = test_args[1]
+    //    argv = ["iOSTestbed"] + test_args[2:]
+    test_args = [[NSProcessInfo processInfo] arguments];
     if (test_args == NULL) {
         NSLog(@"Unable to identify test arguments.");
     }
-    argv = malloc(sizeof(char *) * ([test_args count] + 1));
+    NSLog(@"Test arguments: %@", test_args);
+    argv = malloc(sizeof(char *) * ([test_args count] - 1));
     argv[0] = "iOSTestbed";
-    for (int i = 1; i < [test_args count]; i++) {
+    for (int i = 1; i < [test_args count] - 1; i++) {
-        argv[i] = [[test_args objectAtIndex:i] UTF8String];
+        argv[i] = [[test_args objectAtIndex:i+1] UTF8String];
     }
-    NSLog(@"Test command: %@", test_args);
 
     // Generate an isolated Python configuration.
     NSLog(@"Configuring isolated Python...");
@@ -68,7 +72,7 @@
     // Ensure that signal handlers are installed
     config.install_signal_handlers = 1;
     // Run the test module.
-    config.run_module = Py_DecodeLocale([[test_args objectAtIndex:0] UTF8String], NULL);
+    config.run_module = Py_DecodeLocale([[test_args objectAtIndex:1] UTF8String], NULL);
     // For debugging - enable verbose mode.
     // config.verbose = 1;
 
@@ -101,7 +105,7 @@
     }
 
     NSLog(@"Configure argc/argv...");
-    status = PyConfig_SetBytesArgv(&config, [test_args count], (char**) argv);
+    status = PyConfig_SetBytesArgv(&config, [test_args count] - 1, (char**) argv);
     if (PyStatus_Exception(status)) {
         XCTFail(@"Unable to configure argc/argv: %s", status.err_msg);
         PyConfig_Clear(&config);