mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-07-07 19:35:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

gh-131531: Add android.py package command (#131532)

Browse source 
Adds a `package` entry point to the `android.py` build script to support
creating an Android distribution artefact.
This commit is contained in:
Malcolm Smith 2025-04-01 01:46:29 +01:00 • committed by GitHub
parent 45a3ab5a81
commit fe5c4c53e7
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 287 additions and 124 deletions
Download patch file Download diff file Expand all files Collapse all files

70
Android/README.md
View file

@ -1,19 +1,22 @@
# Python for Android
These instructions are only needed if you're planning to compile Python for
Android yourself. Most users should *not* need to do this. Instead, use one of
the tools listed in `Doc/using/android.rst`, which will provide a much easier
experience.
If you obtained this README as part of a release package, then the only
applicable sections are "Prerequisites", "Testing", and "Using in your own app".
If you obtained this README as part of the CPython source tree, then you can
also follow the other sections to compile Python for Android yourself.
However, most app developers should not need to do any of these things manually.
Instead, use one of the tools listed
[here](https://docs.python.org/3/using/android.html), which will provide a much
easier experience.
## Prerequisites
First, make sure you have all the usual tools and libraries needed to build
Python for your development machine.
Second, you'll need an Android SDK. If you already have the SDK installed,
export the `ANDROID_HOME` environment variable to point at its location.
Otherwise, here's how to install it:
If you already have an Android SDK installed, export the `ANDROID_HOME`
environment variable to point at its location. Otherwise, here's how to install
it:
* Download the "Command line tools" from <https://developer.android.com/studio>.
* Create a directory `android-sdk/cmdline-tools`, and unzip the command line
@ -27,15 +30,16 @@ The `android.py` script also requires the following commands to be on the `PATH`
* `curl`
* `java` (or set the `JAVA_HOME` environment variable)
* `tar`
* `unzip`
## Building
Python can be built for Android on any POSIX platform supported by the Android
development tools, which currently means Linux or macOS. This involves doing a
cross-build where you use a "build" Python (for your development machine) to
help produce a "host" Python for Android.
development tools, which currently means Linux or macOS.
First we'll make a "build" Python (for your development machine), then use it to
help produce a "host" Python for Android. So make sure you have all the usual
tools and libraries needed to build Python for your development machine.
The easiest way to do a build is to use the `android.py` script. You can either
have it perform the entire build process from start to finish in one step, or
@ -60,8 +64,8 @@ To do all steps in a single command, run:
./android.py build HOST
```
In the end you should have a build Python in `cross-build/build`, and an Android
build in `cross-build/HOST`.
In the end you should have a build Python in `cross-build/build`, and a host
Python in `cross-build/HOST`.
You can use `--` as a separator for any of the `configure`-related commands –
including `build` itself – to pass arguments to the underlying `configure`
@ -73,14 +77,27 @@ call. For example, if you want a pydebug build that also caches the results from
```
## Packaging
After building an architecture as described in the section above, you can
package it for release with this command:
```sh
./android.py package HOST
```
`HOST` is defined in the section above.
This will generate a tarball in `cross-build/HOST/dist`, whose structure is
similar to the `Android` directory of the CPython source tree.
## Testing
The test suite can be run on Linux, macOS, or Windows:
The Python test suite can be run on Linux, macOS, or Windows:
* On Linux, the emulator needs access to the KVM virtualization interface, and
a DISPLAY environment variable pointing at an X server.
* On Windows, you won't be able to do the build on the same machine, so you'll
have to copy the `cross-build/HOST` directory from somewhere else.
The test suite can usually be run on a device with 2 GB of RAM, but this is
borderline, so you may need to increase it to 4 GB. As of Android
@ -90,9 +107,16 @@ and find `hw.ramSize` in both config.ini and hardware-qemu.ini. Either set these
manually to the same value, or use the Android Studio Device Manager, which will
update both files.
Before running the test suite, follow the instructions in the previous section
to build the architecture you want to test. Then run the test script in one of
the following modes:
You can run the test suite either:
* Within the CPython repository, after doing a build as described above. On
Windows, you won't be able to do the build on the same machine, so you'll have
to copy the `cross-build/HOST/prefix` directory from somewhere else.
* Or by taking a release package built using the `package` command, extracting
it wherever you want, and using its own copy of `android.py`.
The test script supports the following modes:
* In `--connected` mode, it runs on a device or emulator you have already
connected to the build machine. List the available devices with
@ -133,4 +157,4 @@ until you re-run `android.py make-host` or `build`.
## Using in your own app
See `Doc/using/android.rst`.
See https://docs.python.org/3/using/android.html.

187
Android/android.py
View file

@ -2,7 +2,6 @@
import asyncio
import argparse
from glob import glob
import os
import re
import shlex
@ -13,6 +12,8 @@ import sys
import sysconfig
from asyncio import wait_for
from contextlib import asynccontextmanager
from datetime import datetime, timezone
from glob import glob
from os.path import basename, relpath
from pathlib import Path
from subprocess import CalledProcessError
@ -20,11 +21,12 @@ from tempfile import TemporaryDirectory
SCRIPT_NAME = Path(__file__).name
CHECKOUT = Path(__file__).resolve().parent.parent
ANDROID_DIR = CHECKOUT / "Android"
ANDROID_DIR = Path(__file__).resolve().parent
CHECKOUT = ANDROID_DIR.parent
TESTBED_DIR = ANDROID_DIR / "testbed"
CROSS_BUILD_DIR = CHECKOUT / "cross-build"
HOSTS = ["aarch64-linux-android", "x86_64-linux-android"]
APP_ID = "org.python.testbed"
DECODE_ARGS = ("UTF-8", "backslashreplace")
@ -58,12 +60,10 @@ def delete_glob(pattern):
path.unlink()
def subdir(name, *, clean=None):
path = CROSS_BUILD_DIR / name
if clean:
delete_glob(path)
def subdir(*parts, create=False):
path = CROSS_BUILD_DIR.joinpath(*parts)
if not path.exists():
if clean is None:
if not create:
sys.exit(
f"{path} does not exist. Create it by running the appropriate "
f"`configure` subcommand of {SCRIPT_NAME}.")
@ -123,7 +123,9 @@ def build_python_path():
def configure_build_python(context):
os.chdir(subdir("build", clean=context.clean))
if context.clean:
clean("build")
os.chdir(subdir("build", create=True))
command = [relpath(CHECKOUT / "configure")]
if context.args:
@ -153,18 +155,17 @@ def download(url, target_dir="."):
def configure_host_python(context):
host_dir = subdir(context.host, clean=context.clean)
if context.clean:
clean(context.host)
host_dir = subdir(context.host, create=True)
prefix_dir = host_dir / "prefix"
if not prefix_dir.exists():
prefix_dir.mkdir()
os.chdir(prefix_dir)
unpack_deps(context.host)
build_dir = host_dir / "build"
build_dir.mkdir(exist_ok=True)
os.chdir(build_dir)
os.chdir(host_dir)
command = [
# Basic cross-compiling configuration
relpath(CHECKOUT / "configure"),
@ -193,11 +194,10 @@ def make_host_python(context):
# the build.
host_dir = subdir(context.host)
prefix_dir = host_dir / "prefix"
delete_glob(f"{prefix_dir}/include/python*")
delete_glob(f"{prefix_dir}/lib/libpython*")
delete_glob(f"{prefix_dir}/lib/python*")
for pattern in ("include/python*", "lib/libpython*", "lib/python*"):
delete_glob(f"{prefix_dir}/{pattern}")
os.chdir(host_dir / "build")
os.chdir(host_dir)
run(["make", "-j", str(os.cpu_count())], host=context.host)
run(["make", "install", f"prefix={prefix_dir}"], host=context.host)
@ -209,8 +209,13 @@ def build_all(context):
step(context)
def clean(host):
delete_glob(CROSS_BUILD_DIR / host)
def clean_all(context):
delete_glob(CROSS_BUILD_DIR)
for host in HOSTS + ["build"]:
clean(host)
def setup_sdk():
@ -234,31 +239,27 @@ def setup_sdk():
# To avoid distributing compiled artifacts without corresponding source code,
# the Gradle wrapper is not included in the CPython repository. Instead, we
# extract it from the Gradle release.
# extract it from the Gradle GitHub repository.
def setup_testbed():
if all((TESTBED_DIR / path).exists() for path in [
"gradlew", "gradlew.bat", "gradle/wrapper/gradle-wrapper.jar",
]):
# The Gradle version used for the build is specified in
# testbed/gradle/wrapper/gradle-wrapper.properties. This wrapper version
# doesn't need to match, as any version of the wrapper can download any
# version of Gradle.
version = "8.9.0"
paths = ["gradlew", "gradlew.bat", "gradle/wrapper/gradle-wrapper.jar"]
if all((TESTBED_DIR / path).exists() for path in paths):
return
ver_long = "8.7.0"
ver_short = ver_long.removesuffix(".0")
for filename in ["gradlew", "gradlew.bat"]:
out_path = download(
f"https://raw.githubusercontent.com/gradle/gradle/v{ver_long}/{filename}",
TESTBED_DIR)
for path in paths:
out_path = TESTBED_DIR / path
out_path.parent.mkdir(exist_ok=True)
download(
f"https://raw.githubusercontent.com/gradle/gradle/v{version}/{path}",
out_path.parent,
)
os.chmod(out_path, 0o755)
with TemporaryDirectory(prefix=SCRIPT_NAME) as temp_dir:
bin_zip = download(
f"https://services.gradle.org/distributions/gradle-{ver_short}-bin.zip",
temp_dir)
outer_jar = f"gradle-{ver_short}/lib/plugins/gradle-wrapper-{ver_short}.jar"
run(["unzip", "-d", temp_dir, bin_zip, outer_jar])
run(["unzip", "-o", "-d", f"{TESTBED_DIR}/gradle/wrapper",
f"{temp_dir}/{outer_jar}", "gradle-wrapper.jar"])
# run_testbed will build the app automatically, but it's useful to have this as
# a separate command to allow running the app outside of this script.
@ -538,6 +539,73 @@ async def run_testbed(context):
raise e.exceptions[0]
def package_version(prefix_dir):
patchlevel_glob = f"{prefix_dir}/include/python*/patchlevel.h"
patchlevel_paths = glob(patchlevel_glob)
if len(patchlevel_paths) != 1:
sys.exit(f"{patchlevel_glob} matched {len(patchlevel_paths)} paths.")
for line in open(patchlevel_paths[0]):
if match := re.fullmatch(r'\s*#define\s+PY_VERSION\s+"(.+)"\s*', line):
version = match[1]
break
else:
sys.exit(f"Failed to find Python version in {patchlevel_paths[0]}.")
# If not building against a tagged commit, add a timestamp to the version.
# Follow the PyPA version number rules, as this will make it easier to
# process with other tools.
if version.endswith("+"):
version += datetime.now(timezone.utc).strftime("%Y%m%d.%H%M%S")
return version
def package(context):
prefix_dir = subdir(context.host, "prefix")
version = package_version(prefix_dir)
with TemporaryDirectory(prefix=SCRIPT_NAME) as temp_dir:
temp_dir = Path(temp_dir)
# Include all tracked files from the Android directory.
for line in run(
["git", "ls-files"],
cwd=ANDROID_DIR, capture_output=True, text=True, log=False,
).stdout.splitlines():
src = ANDROID_DIR / line
dst = temp_dir / line
dst.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(src, dst, follow_symlinks=False)
# Include anything from the prefix directory which could be useful
# either for embedding Python in an app, or building third-party
# packages against it.
for rel_dir, patterns in [
("include", ["openssl*", "python*", "sqlite*"]),
("lib", ["engines-3", "libcrypto*.so", "libpython*", "libsqlite*",
"libssl*.so", "ossl-modules", "python*"]),
("lib/pkgconfig", ["*crypto*", "*ssl*", "*python*", "*sqlite*"]),
]:
for pattern in patterns:
for src in glob(f"{prefix_dir}/{rel_dir}/{pattern}"):
dst = temp_dir / relpath(src, prefix_dir.parent)
dst.parent.mkdir(parents=True, exist_ok=True)
if Path(src).is_dir():
shutil.copytree(
src, dst, symlinks=True,
ignore=lambda *args: ["__pycache__"]
)
else:
shutil.copy2(src, dst, follow_symlinks=False)
dist_dir = subdir(context.host, "dist", create=True)
package_path = shutil.make_archive(
f"{dist_dir}/python-{version}-{context.host}", "gztar", temp_dir
)
print(f"Wrote {package_path}")
# Handle SIGTERM the same way as SIGINT. This ensures that if we're terminated
# by the buildbot worker, we'll make an attempt to clean up our subprocesses.
def install_signal_handler():
@ -550,6 +618,8 @@ def install_signal_handler():
def parse_args():
parser = argparse.ArgumentParser()
subcommands = parser.add_subparsers(dest="subcommand")
# Subcommands
build = subcommands.add_parser("build", help="Build everything")
configure_build = subcommands.add_parser("configure-build",
help="Run `configure` for the "
@ -561,25 +631,27 @@ def parse_args():
make_host = subcommands.add_parser("make-host",
help="Run `make` for Android")
subcommands.add_parser(
"clean", help="Delete the cross-build directory")
"clean", help="Delete all build and prefix directories")
subcommands.add_parser(
"build-testbed", help="Build the testbed app")
test = subcommands.add_parser(
"test", help="Run the test suite")
package = subcommands.add_parser("package", help="Make a release package")
# Common arguments
for subcommand in build, configure_build, configure_host:
subcommand.add_argument(
"--clean", action="store_true", default=False, dest="clean",
help="Delete any relevant directories before building")
for subcommand in build, configure_host, make_host:
help="Delete the relevant build and prefix directories first")
for subcommand in [build, configure_host, make_host, package]:
subcommand.add_argument(
"host", metavar="HOST",
choices=["aarch64-linux-android", "x86_64-linux-android"],
"host", metavar="HOST", choices=HOSTS,
help="Host triplet: choices=[%(choices)s]")
for subcommand in build, configure_build, configure_host:
subcommand.add_argument("args", nargs="*",
help="Extra arguments to pass to `configure`")
subcommands.add_parser(
"build-testbed", help="Build the testbed app")
test = subcommands.add_parser(
"test", help="Run the test suite")
# Test arguments
test.add_argument(
"-v", "--verbose", action="count", default=0,
help="Show Gradle output, and non-Python logcat messages. "
@ -608,14 +680,17 @@ def main():
stream.reconfigure(line_buffering=True)
context = parse_args()
dispatch = {"configure-build": configure_build_python,
"make-build": make_build_python,
"configure-host": configure_host_python,
"make-host": make_host_python,
"build": build_all,
"clean": clean_all,
"build-testbed": build_testbed,
"test": run_testbed}
dispatch = {
"configure-build": configure_build_python,
"make-build": make_build_python,
"configure-host": configure_host_python,
"make-host": make_host_python,
"build": build_all,
"clean": clean_all,
"build-testbed": build_testbed,
"test": run_testbed,
"package": package,
}
try:
result = dispatch[context.subcommand](context)

17
Android/testbed/.gitignore vendored
View file

@ -1,18 +1,19 @@
# The Gradle wrapper should be downloaded by running `../android.py setup-testbed`.
# The Gradle wrapper can be downloaded by running the `test` or `build-testbed`
# commands of android.py.
/gradlew
/gradlew.bat
/gradle/wrapper/gradle-wrapper.jar
# The repository's top-level .gitignore file ignores all .idea directories, but
# we want to keep any files which can't be regenerated from the Gradle
# configuration.
!.idea/
/.idea/*
!/.idea/inspectionProfiles
*.iml
.gradle
/local.properties
/.idea/caches
/.idea/deploymentTargetDropdown.xml
/.idea/libraries
/.idea/modules.xml
/.idea/workspace.xml
/.idea/navEditor.xml
/.idea/assetWizardSettings.xml
.DS_Store
/build
/captures

8
Android/testbed/.idea/inspectionProfiles/Project_Default.xml generated Normal file
View file

@ -0,0 +1,8 @@
<component name="InspectionProjectProfileManager">
<profile version="1.0">
<option name="myName" value="Project Default" />
<inspection_tool class="AndroidLintGradleDependency" enabled="true" level="WEAK WARNING" enabled_by_default="true" editorAttributes="INFO_ATTRIBUTES" />
<inspection_tool class="AndroidLintOldTargetApi" enabled="true" level="WEAK WARNING" enabled_by_default="true" editorAttributes="INFO_ATTRIBUTES" />
<inspection_tool class="UnstableApiUsage" enabled="true" level="WEAK WARNING" enabled_by_default="true" editorAttributes="INFO_ATTRIBUTES" />
</profile>
</component>

110
Android/testbed/app/build.gradle.kts
View file

@ -6,28 +6,71 @@ plugins {
id("org.jetbrains.kotlin.android")
}
val PYTHON_DIR = file("../../..").canonicalPath
val PYTHON_CROSS_DIR = "$PYTHON_DIR/cross-build"
val ANDROID_DIR = file("../..")
val PYTHON_DIR = ANDROID_DIR.parentFile!!
val PYTHON_CROSS_DIR = file("$PYTHON_DIR/cross-build")
val inSourceTree = (
ANDROID_DIR.name == "Android" && file("$PYTHON_DIR/pyconfig.h.in").exists()
)
val ABIS = mapOf(
"arm64-v8a" to "aarch64-linux-android",
"x86_64" to "x86_64-linux-android",
).filter { file("$PYTHON_CROSS_DIR/${it.value}").exists() }
if (ABIS.isEmpty()) {
val KNOWN_ABIS = mapOf(
"aarch64-linux-android" to "arm64-v8a",
"x86_64-linux-android" to "x86_64",
)
// Discover prefixes.
val prefixes = ArrayList<File>()
if (inSourceTree) {
for ((triplet, _) in KNOWN_ABIS.entries) {
val prefix = file("$PYTHON_CROSS_DIR/$triplet/prefix")
if (prefix.exists()) {
prefixes.add(prefix)
}
}
} else {
// Testbed is inside a release package.
val prefix = file("$ANDROID_DIR/prefix")
if (prefix.exists()) {
prefixes.add(prefix)
}
}
if (prefixes.isEmpty()) {
throw GradleException(
"No Android ABIs found in $PYTHON_CROSS_DIR: see Android/README.md " +
"for building instructions."
"No Android prefixes found: see README.md for testing instructions"
)
}
val PYTHON_VERSION = file("$PYTHON_DIR/Include/patchlevel.h").useLines {
for (line in it) {
val match = """#define PY_VERSION\s+"(\d+\.\d+)""".toRegex().find(line)
if (match != null) {
return@useLines match.groupValues[1]
// Detect Python versions and ABIs.
lateinit var pythonVersion: String
var abis = HashMap<File, String>()
for ((i, prefix) in prefixes.withIndex()) {
val libDir = file("$prefix/lib")
val version = run {
for (filename in libDir.list()!!) {
"""python(\d+\.\d+)""".toRegex().matchEntire(filename)?.let {
return@run it.groupValues[1]
}
}
throw GradleException("Failed to find Python version in $libDir")
}
throw GradleException("Failed to find Python version")
if (i == 0) {
pythonVersion = version
} else if (pythonVersion != version) {
throw GradleException(
"${prefixes[0]} is Python $pythonVersion, but $prefix is Python $version"
)
}
val libPythonDir = file("$libDir/python$pythonVersion")
val triplet = run {
for (filename in libPythonDir.list()!!) {
"""_sysconfigdata__android_(.+).py""".toRegex().matchEntire(filename)?.let {
return@run it.groupValues[1]
}
}
throw GradleException("Failed to find Python triplet in $libPythonDir")
}
abis[prefix] = KNOWN_ABIS[triplet]!!
}
@ -53,10 +96,16 @@ android {
versionCode = 1
versionName = "1.0"
ndk.abiFilters.addAll(ABIS.keys)
ndk.abiFilters.addAll(abis.values)
externalNativeBuild.cmake.arguments(
"-DPYTHON_CROSS_DIR=$PYTHON_CROSS_DIR",
"-DPYTHON_VERSION=$PYTHON_VERSION",
"-DPYTHON_PREFIX_DIR=" + if (inSourceTree) {
// AGP uses the ${} syntax for its own purposes, so use a Jinja style
// placeholder.
"$PYTHON_CROSS_DIR/{{triplet}}/prefix"
} else {
prefixes[0]
},
"-DPYTHON_VERSION=$pythonVersion",
"-DANDROID_SUPPORT_FLEXIBLE_PAGE_SIZES=ON",
)
@ -133,24 +182,25 @@ dependencies {
// Create some custom tasks to copy Python and its standard library from
// elsewhere in the repository.
androidComponents.onVariants { variant ->
val pyPlusVer = "python$PYTHON_VERSION"
val pyPlusVer = "python$pythonVersion"
generateTask(variant, variant.sources.assets!!) {
into("python") {
// Include files such as pyconfig.h are used by some of the tests.
into("include/$pyPlusVer") {
for (triplet in ABIS.values) {
from("$PYTHON_CROSS_DIR/$triplet/prefix/include/$pyPlusVer")
for (prefix in prefixes) {
from("$prefix/include/$pyPlusVer")
}
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}
into("lib/$pyPlusVer") {
// To aid debugging, the source directory takes priority.
from("$PYTHON_DIR/Lib")
// The cross-build directory provides ABI-specific files such as
// sysconfigdata.
for (triplet in ABIS.values) {
from("$PYTHON_CROSS_DIR/$triplet/prefix/lib/$pyPlusVer")
// To aid debugging, the source directory takes priority when
// running inside a CPython source tree.
if (inSourceTree) {
from("$PYTHON_DIR/Lib")
}
for (prefix in prefixes) {
from("$prefix/lib/$pyPlusVer")
}
into("site-packages") {
@ -164,9 +214,9 @@ androidComponents.onVariants { variant ->
}
generateTask(variant, variant.sources.jniLibs!!) {
for ((abi, triplet) in ABIS.entries) {
for ((prefix, abi) in abis.entries) {
into(abi) {
from("$PYTHON_CROSS_DIR/$triplet/prefix/lib")
from("$prefix/lib")
include("libpython*.*.so")
include("lib*_python.so")
}

11
Android/testbed/app/src/main/c/CMakeLists.txt
View file

@ -1,9 +1,14 @@
cmake_minimum_required(VERSION 3.4.1)
project(testbed)
set(PREFIX_DIR ${PYTHON_CROSS_DIR}/${CMAKE_LIBRARY_ARCHITECTURE}/prefix)
include_directories(${PREFIX_DIR}/include/python${PYTHON_VERSION})
link_directories(${PREFIX_DIR}/lib)
# Resolve variables from the command line.
string(
REPLACE {{triplet}} ${CMAKE_LIBRARY_ARCHITECTURE}
PYTHON_PREFIX_DIR ${PYTHON_PREFIX_DIR}
)
include_directories(${PYTHON_PREFIX_DIR}/include/python${PYTHON_VERSION})
link_directories(${PYTHON_PREFIX_DIR}/lib)
link_libraries(log python${PYTHON_VERSION})
add_library(main_activity SHARED main_activity.c)