Windows launchers using posy trampolines (#1092)

## Background

In virtual environments, we want to install python programs as console
commands, e.g. `black .` over `python -m black .`. They may be called
[entrypoints](https://packaging.python.org/en/latest/specifications/entry-points/)
or scripts. For entrypoints, we're given a module name and function to
call in that module.

On Unix, we generate a minimal python script launcher. Text files are
runnable on unix by adding a shebang at their top, e.g.

```python
#!/usr/bin/env python
```

will make the operating system run the file with the current python
interpreter. A venv launcher for black in `/home/ferris/colorize/.venv`
(module name: `black`, function to call: `patched_main`) would look like
this:

```python
#!/home/ferris/colorize/.venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from black import patched_main
if __name__ == "__main__":
    sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
    sys.exit(patched_main())
```

On windows, this doesn't work, we can only rely on launching `.exe`
files.

## Summary

We use posy's rust implementation of a trampoline, which is based on
distlib's c++ implementation. We pre-build a minimal exe and append the
launcher script as stored zip archive behind it. The exe will look for
the venv python interpreter next to it and use it to execute the
appended script.

The changes in this PR make the `black` entrypoint work:

```powershell
cargo run -- venv .venv
cargo run -q -- pip install black
.\.venv\Scripts\black --version
```

Integration with our existing tests will be done in follow-up PRs.

## Implementation and Details

I've vendored the posy trampoline crate. It is a formatted, renamed and
slightly changed for embedding version of
https://github.com/njsmith/posy/pull/28.

The posy launchers are smaller than the distlib launchers, 16K vs 106K
for black. Currently only `x86_64-pc-windows-msvc` is supported. The
crate requires a nightly compiler for its no-std binary size tricks.

On windows, an application can be launched with a console or without (to
create windows instead), which needs two different launchers. The gui
launcher will subsequently use `pythonw.exe` while the console launcher
uses `python.exe`.
This commit is contained in:
konsti 2024-01-26 14:54:11 +01:00 committed by GitHub
parent f1d3b08c12
commit 39021263dd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
24 changed files with 934 additions and 42 deletions

View file

@ -119,13 +119,17 @@ pub fn create_bare_venv(location: &Utf8Path, interpreter: &Interpreter) -> io::R
{
// https://github.com/python/cpython/blob/d457345bbc6414db0443819290b04a9a4333313d/Lib/venv/__init__.py#L261-L267
// https://github.com/pypa/virtualenv/blob/d9fdf48d69f0d0ca56140cf0381edbb5d6fe09f5/src/virtualenv/create/via_global_ref/builtin/cpython/cpython3.py#L78-L83
let shim = interpreter
.stdlib()
.join("venv")
.join("scripts")
.join("nt")
.join("python.exe");
fs_err::copy(shim, bin_dir.join("python.exe"))?;
// There's two kinds of applications on windows: Those that allocate a console (python.exe) and those that
// don't because they use window(s) (pythonw.exe).
for python_exe in ["python.exe", "pythonw.exe"] {
let shim = interpreter
.stdlib()
.join("venv")
.join("scripts")
.join("nt")
.join(python_exe);
fs_err::copy(shim, bin_dir.join(python_exe))?;
}
}
#[cfg(not(any(unix, windows)))]
{
@ -133,7 +137,8 @@ pub fn create_bare_venv(location: &Utf8Path, interpreter: &Interpreter) -> io::R
}
// Add all the activate scripts for different shells
// TODO(konstin): That's unix!
// TODO(konstin): RELATIVE_SITE_PACKAGES is currently only the unix path. We should ensure that all launchers work
// cross-platform.
for (name, template) in ACTIVATE_TEMPLATES {
let activator = template
.replace("{{ VIRTUAL_ENV_DIR }}", location.as_str())

View file

@ -67,8 +67,8 @@ pub enum Error {
RecordCsv(#[from] csv::Error),
#[error("Broken virtualenv: {0}")]
BrokenVenv(String),
#[error("Failed to detect the operating system version: {0}")]
OsVersionDetection(String),
#[error("Unable to create Windows launch for {0} (only x64_64 is supported)")]
UnsupportedWindowsArch(&'static str),
#[error("Failed to detect the current platform")]
PlatformInfo(#[source] PlatformInfoError),
#[error("Invalid version specification, only none or == is supported")]

View file

@ -99,8 +99,14 @@ pub fn install_wheel(
debug!(name, "Writing entrypoints");
let (console_scripts, gui_scripts) = parse_scripts(&wheel, &dist_info_prefix, None)?;
write_script_entrypoints(&site_packages, location, &console_scripts, &mut record)?;
write_script_entrypoints(&site_packages, location, &gui_scripts, &mut record)?;
write_script_entrypoints(
&site_packages,
location,
&console_scripts,
&mut record,
false,
)?;
write_script_entrypoints(&site_packages, location, &gui_scripts, &mut record, true)?;
let data_dir = site_packages.join(format!("{dist_info_prefix}.data"));
// 2.a Unpacked archive includes distribution-1.0.dist-info/ and (if there is data) distribution-1.0.data/.

View file

@ -34,9 +34,10 @@ use crate::{find_dist_info, Error};
/// `#!/usr/bin/env python`
pub const SHEBANG_PYTHON: &str = "#!/usr/bin/env python";
pub(crate) const LAUNCHER_T32: &[u8] = include_bytes!("../windows-launcher/t32.exe");
pub(crate) const LAUNCHER_T64: &[u8] = include_bytes!("../windows-launcher/t64.exe");
pub(crate) const LAUNCHER_T64_ARM: &[u8] = include_bytes!("../windows-launcher/t64-arm.exe");
pub(crate) const LAUNCHER_X86_64_GUI: &[u8] =
include_bytes!("../../puffin-trampoline/trampolines/puffin-trampoline-gui.exe");
pub(crate) const LAUNCHER_X86_64_CONSOLE: &[u8] =
include_bytes!("../../puffin-trampoline/trampolines/puffin-trampoline-console.exe");
/// Wrapper script template function
///
@ -283,35 +284,34 @@ pub(crate) fn get_shebang(location: &InstallLocation<impl AsRef<Path>>) -> Strin
format!("#!{path}")
}
/// To get a launcher on windows we write a minimal .exe launcher binary and then attach the actual
/// python after it.
///
/// TODO pyw scripts
///
/// TODO: a nice, reproducible-without-distlib rust solution
/// A windows script is a minimal .exe launcher binary with the python entrypoint script appended as stored zip file.
/// The launcher will look for `python[w].exe` adjacent to it in the same directory to start the embedded script.
///
/// <https://github.com/pypa/pip/blob/fd0ea6bc5e8cb95e518c23d901c26ca14db17f89/src/pip/_vendor/distlib/scripts.py#L248-L262>
pub(crate) fn windows_script_launcher(launcher_python_script: &str) -> Result<Vec<u8>, Error> {
pub(crate) fn windows_script_launcher(
launcher_python_script: &str,
is_gui: bool,
) -> Result<Vec<u8>, Error> {
let launcher_bin = match env::consts::ARCH {
"x84" => LAUNCHER_T32,
"x86_64" => LAUNCHER_T64,
"aarch64" => LAUNCHER_T64_ARM,
"x86_64" => {
if is_gui {
LAUNCHER_X86_64_GUI
} else {
LAUNCHER_X86_64_CONSOLE
}
}
arch => {
let error = format!(
"Don't know how to create windows launchers for script for {arch}, \
only x86, x86_64 and aarch64 (64-bit arm) are supported"
);
return Err(Error::OsVersionDetection(error));
return Err(Error::UnsupportedWindowsArch(arch));
}
};
let mut stream: Vec<u8> = Vec::new();
let mut payload: Vec<u8> = Vec::new();
{
// We're using the zip writer, but it turns out we're not actually deflating apparently
// we're just using an offset
// We're using the zip writer, but with stored compression
// https://github.com/njsmith/posy/blob/04927e657ca97a5e35bb2252d168125de9a3a025/src/trampolines/mod.rs#L75-L82
// https://github.com/pypa/distlib/blob/8ed03aab48add854f377ce392efffb79bb4d6091/PC/launcher.c#L259-L271
let stored = FileOptions::default().compression_method(zip::CompressionMethod::Stored);
let mut archive = ZipWriter::new(Cursor::new(&mut stream));
let mut archive = ZipWriter::new(Cursor::new(&mut payload));
let error_msg = "Writing to Vec<u8> should never fail";
archive.start_file("__main__.py", stored).expect(error_msg);
archive
@ -320,8 +320,9 @@ pub(crate) fn windows_script_launcher(launcher_python_script: &str) -> Result<Ve
archive.finish().expect(error_msg);
}
let mut launcher: Vec<u8> = launcher_bin.to_vec();
launcher.append(&mut stream);
let mut launcher: Vec<u8> = Vec::with_capacity(launcher_bin.len() + payload.len());
launcher.extend_from_slice(launcher_bin);
launcher.extend_from_slice(&payload);
Ok(launcher)
}
@ -335,6 +336,7 @@ pub(crate) fn write_script_entrypoints(
location: &InstallLocation<impl AsRef<Path>>,
entrypoints: &[Script],
record: &mut Vec<RecordEntry>,
is_gui: bool,
) -> Result<(), Error> {
for entrypoint in entrypoints {
let entrypoint_relative = if cfg!(windows) {
@ -356,7 +358,7 @@ pub(crate) fn write_script_entrypoints(
&get_shebang(location),
);
if cfg!(windows) {
let launcher = windows_script_launcher(&launcher_python_script)?;
let launcher = windows_script_launcher(&launcher_python_script, is_gui)?;
write_file_recorded(site_packages, &entrypoint_relative, &launcher, record)?;
} else {
write_file_recorded(
@ -1009,8 +1011,14 @@ pub fn install_wheel(
debug!(name = name.as_str(), "Writing entrypoints");
let (console_scripts, gui_scripts) = parse_scripts(&mut archive, &dist_info_prefix, None)?;
write_script_entrypoints(&site_packages, location, &console_scripts, &mut record)?;
write_script_entrypoints(&site_packages, location, &gui_scripts, &mut record)?;
write_script_entrypoints(
&site_packages,
location,
&console_scripts,
&mut record,
false,
)?;
write_script_entrypoints(&site_packages, location, &gui_scripts, &mut record, true)?;
let data_dir = site_packages.join(format!("{dist_info_prefix}.data"));
// 2.a Unpacked archive includes distribution-1.0.dist-info/ and (if there is data) distribution-1.0.data/.
@ -1135,7 +1143,9 @@ mod test {
use indoc::{formatdoc, indoc};
use crate::wheel::{read_record_file, relative_to};
use crate::wheel::{
read_record_file, relative_to, LAUNCHER_X86_64_CONSOLE, LAUNCHER_X86_64_GUI,
};
use crate::{parse_key_value_file, Script};
use super::parse_wheel_version;
@ -1264,4 +1274,19 @@ mod test {
})
);
}
#[test]
fn test_launchers_are_small() {
// At time of writing, they are 15872 bytes.
assert!(
LAUNCHER_X86_64_GUI.len() < 20 * 1024,
"GUI launcher: {}",
LAUNCHER_X86_64_GUI.len()
);
assert!(
LAUNCHER_X86_64_CONSOLE.len() < 20 * 1024,
"CLI launcher: {}",
LAUNCHER_X86_64_CONSOLE.len()
);
}
}

147
crates/puffin-trampoline/Cargo.lock generated Normal file
View file

@ -0,0 +1,147 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
[[package]]
name = "embed-manifest"
version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41cd446c890d6bed1d8b53acef5f240069ebef91d6fae7c5f52efe61fe8b5eae"
[[package]]
name = "proc-macro2"
version = "1.0.78"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e2422ad645d89c99f8f3e6b88a9fdeca7fabeac836b1002371c4367c8f984aae"
dependencies = [
"unicode-ident",
]
[[package]]
name = "puffin-trampoline"
version = "0.1.0"
dependencies = [
"embed-manifest",
"ufmt",
"ufmt-write",
"windows-sys",
]
[[package]]
name = "quote"
version = "1.0.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "291ec9ab5efd934aaf503a6466c5d5251535d108ee747472c3977cc5acc868ef"
dependencies = [
"proc-macro2",
]
[[package]]
name = "syn"
version = "1.0.109"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "ufmt"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a64846ec02b57e9108d6469d98d1648782ad6bb150a95a9baac26900bbeab9d"
dependencies = [
"ufmt-macros",
"ufmt-write",
]
[[package]]
name = "ufmt-macros"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d337d3be617449165cb4633c8dece429afd83f84051024079f97ad32a9663716"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "ufmt-write"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e87a2ed6b42ec5e28cc3b94c09982969e9227600b2e3dcbc1db927a84c06bd69"
[[package]]
name = "unicode-ident"
version = "1.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b"
[[package]]
name = "windows-sys"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [
"windows-targets",
]
[[package]]
name = "windows-targets"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a18201040b24831fbb9e4eb208f8892e1f50a37feb53cc7ff887feb8f50e7cd"
dependencies = [
"windows_aarch64_gnullvm",
"windows_aarch64_msvc",
"windows_i686_gnu",
"windows_i686_msvc",
"windows_x86_64_gnu",
"windows_x86_64_gnullvm",
"windows_x86_64_msvc",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb7764e35d4db8a7921e09562a0304bf2f93e0a51bfccee0bd0bb0b666b015ea"
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbaa0368d4f1d2aaefc55b6fcfee13f41544ddf36801e793edbbfd7d7df075ef"
[[package]]
name = "windows_i686_gnu"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a28637cb1fa3560a16915793afb20081aba2c92ee8af57b4d5f28e4b3e7df313"
[[package]]
name = "windows_i686_msvc"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ffe5e8e31046ce6230cc7215707b816e339ff4d4d67c65dffa206fd0f7aa7b9a"
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d6fa32db2bc4a2f5abeacf2b69f7992cd09dca97498da74a151a3132c26befd"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a657e1e9d3f514745a572a6846d3c7aa7dbe1658c056ed9c3344c4109a6949e"
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dff9641d1cd4be8d1a070daf9e3773c5f67e78b4d9d42263020c057706765c04"

View file

@ -0,0 +1,59 @@
[package]
name = "puffin-trampoline"
version = "0.1.0"
authors = ["Nathaniel J. Smith <njs@pobox.com>"]
license = "MIT OR Apache-2.0"
edition = "2021"
autotests = false
# Need to optimize etc. or else build fails
[profile.dev]
lto = true
codegen-units = 1
opt-level = 1
panic = "abort"
debug-assertions = false
overflow-checks = false
debug = true
[profile.release]
lto = true
codegen-units = 1
opt-level = "z"
panic = "abort"
debug = false
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
windows-sys = { version = "0.52.0", features = [
"Win32_Foundation",
"Win32_Security",
"Win32_Storage_FileSystem",
"Win32_System_Console",
"Win32_System_Diagnostics_Debug",
"Win32_System_Environment",
"Win32_System_IO",
"Win32_System_JobObjects",
"Win32_System_JobObjects",
"Win32_System_LibraryLoader",
"Win32_System_Memory",
"Win32_System_Threading",
"Win32_System_WindowsProgramming",
"Win32_UI_WindowsAndMessaging",
] }
# This provides implementations of memcpy, memset, etc., which the compiler assumes
# are available. But there's also a hidden copy of this crate inside `core`/`alloc`,
# and they may or may not conflict depending on how clever the linker is feeling.
# The issue is that the hidden copy doesn't have the "mem" feature enabled, and we
# need it. So two options:
# - Uncomment this, and cross fingers that it doesn't cause conflicts
# - Use -Zbuild-std=... -Zbuild-std-features=compiler-builtins-mem, which enables
# the mem feature on the built-in builtins.
#compiler_builtins = { version = "*", features = ["mem"]}
ufmt-write = "0.1.0"
ufmt = "0.2.0"
[build-dependencies]
embed-manifest = "1.4.0"

View file

@ -0,0 +1,114 @@
# Windows trampolines
This is a fork of [posy trampolines](https://github.com/njsmith/posy/tree/dda22e6f90f5fefa339b869dd2bbe107f5b48448/src/trampolines/windows-trampolines/posy-trampoline).
# What is this?
Sometimes you want to run a tool on Windows that's written in Python, like
`black` or `mypy` or `jupyter` or whatever. But, Windows does not know how to
run Python files! It knows how to run `.exe` files. So we need to somehow
convert our Python file a `.exe` file.
That's what this does: it's a generic "trampoline" that lets us generate custom
`.exe`s for arbitrary Python scripts, and when invoked it bounces to invoking
`python <the script>` instead.
# How do you use it?
Basically, this looks up `python.exe` (for console programs) or
`pythonw.exe` (for GUI programs) in the adjacent directory, and invokes
`python[w].exe path\to\the\<the .exe>`.
The intended use is: take your Python script, name it `__main__.py`, and pack it
into a `.zip` file. Then concatenate that `.zip` file onto the end of one of our
prebuilt `.exe`s.
Then when you run `python` on the `.exe`, it will see the `.zip` trailer at the
end of the `.exe`, and automagically look inside to find and execute
`__main__.py`. Easy-peasy.
(TODO: we should probably make the Python-finding logic slightly more flexible
at some point -- in particular to support more conventional venv-style
installation where you find `python` by looking in the directory next to the
trampoline `.exe` -- but this is good enough to get started.)
# Why does this exist?
I probably could have used Vinay's C++ implementation from `distlib`, but what's
the fun in that? In particular, optimizing for binary size was entertaining
(these are ~7x smaller than the distlib, which doesn't matter much, but does a
little bit, considering that it gets added to every Python script). There are
also some minor advantages, like I think the Rust code is easier to understand
(multiple files!) and it's convenient to be able to straightforwardly code the
Python-finding logic we want. But mostly it was just an interesting challenge.
This does owe a *lot* to the `distlib` implementation though. The overall logic
is copied more-or-less directly.
# Anything I should know for hacking on this?
In order to minimize binary size, this uses `#![no_std]`, `panic="abort"`, and
carefully avoids using `core::fmt`. This removes a bunch of runtime overhead: by
default, Rust "hello world" on Windows is ~150 KB! So these binaries are ~10x
smaller.
Of course the tradeoff is that `#![no_std]` is an awkward super-limited
environment. No C runtime, no platform APIs, very few features... you don't even
get `Vec` or memory allocation or panicking support by default. To work around
this:
- We use `windows-sys` to access Win32 APIs directly. Who needs a C runtime?
Though uh, this does mean that literally all of our code is `unsafe`. Sorry!
- `runtime.rs` has the core glue to get panicking, heap allocation, and linking
working.
- `diagnostics.rs` uses `ufmt` and some cute Windows tricks to get a convenient
version of `eprintln!` that works without `std`, and automatically prints to
either the console if available or pops up a message box if not.
- All the meat is in `bounce.rs`.
Miscellaneous tips:
- `cargo-bloat` is a useful tool for checking what code is ending up in the
final binary and how much space it's taking. (It makes it very obvious whether
you've pulled in `core::fmt`!)
- Lots of Rust built-in panicking checks will pull in `core::fmt`, e.g., if you
ever use `.unwrap()` then suddenly our binaries double in size, because the
`if foo.is_none() { panic!(...) }` that's hidden inside `.unwrap()` will
invoke `core::fmt`, even if the unwrap will actually never fail.
`.unwrap_unchecked()` avoids this. Similar for `slice[idx]` vs
`slice.get_unchecked(idx)`.
# How do you build this stupid thing?
Building this can be frustrating, because the low-level compiler/runtime
machinery have a bunch of implicit assumptions about the environment they'll run
in, and the facilities it provides for things like `memcpy`, unwinding, etc.
With `#![no_std]` most of this machinery is missing. So we need to replace the
bits that we actually need, and which bits we need can change depending on stuff
like optimization options. For example: we use `panic="abort"`, so we don't
actually need unwinding support, but at lower optimization levels the compiler
might not realize that, and still emit references to the unwinding helper
`__CxxFrameHandler3`. And then the linker blows up because that symbol doesn't
exist.
Two approaches that are reasonably likely to work:
- Uncomment `compiler-builtins` in `Cargo.toml`, and build normally: `cargo
build --profile release`.
- Leave `compiler-builtins` commented-out, and build like: `cargo build
--release -Z build-std=core,panic_abort,alloc -Z
build-std-features=compiler-builtins-mem --target x86_64-pc-windows-msvc`
Hopefully in the future as `#![no_std]` develops, this will get smoother.
Also, sometimes it helps to fiddle with optimization levels.

View file

@ -0,0 +1,16 @@
// This embeds a "manifest" - a special XML document - into our built binary.
// The main things it does is tell Windows that we want to use the magic
// utf8 codepage, so we can use the *A versions of Windows API functions and
// don't have to mess with utf-16.
use embed_manifest::{embed_manifest, new_manifest};
fn main() {
if std::env::var_os("CARGO_CFG_WINDOWS").is_some() {
let manifest = new_manifest("Puffin.Trampoline")
.remove_dependency("Microsoft.Windows.Common-Controls");
embed_manifest(manifest).expect("unable to embed manifest");
println!("cargo:rustc-link-arg=/ENTRY:entry");
println!("cargo:rustc-link-arg=/LTCG");
println!("cargo:rerun-if-changed=build.rs");
}
}

View file

@ -0,0 +1,2 @@
[toolchain]
channel = "nightly-2024-01-23"

View file

@ -0,0 +1,9 @@
#![no_std]
#![no_main]
#![windows_subsystem = "console"]
// build.rs passes a custom linker flag to make this the entrypoint to the executable
#[no_mangle]
pub extern "C" fn entry() -> ! {
puffin_trampoline::bounce::bounce(false)
}

View file

@ -0,0 +1,9 @@
#![no_std]
#![no_main]
#![windows_subsystem = "windows"]
// build.rs passes a custom linker flag to make this the entrypoint to the executable
#[no_mangle]
pub extern "C" fn entry() -> ! {
puffin_trampoline::bounce::bounce(true)
}

View file

@ -0,0 +1,276 @@
use alloc::{ffi::CString, vec, vec::Vec};
use core::mem::MaybeUninit;
use core::{
ffi::CStr,
ptr::{addr_of, addr_of_mut, null, null_mut},
};
use windows_sys::Win32::{
Foundation::*,
System::{
Console::*,
Environment::{GetCommandLineA, GetEnvironmentVariableA, SetCurrentDirectoryA},
JobObjects::*,
LibraryLoader::GetModuleFileNameA,
Threading::*,
},
UI::WindowsAndMessaging::*,
};
use crate::helpers::SizeOf;
use crate::{c, check, eprintln};
fn getenv(name: &CStr) -> Option<CString> {
unsafe {
let count = GetEnvironmentVariableA(name.as_ptr() as _, null_mut(), 0);
if count == 0 {
return None;
}
let mut value = Vec::<u8>::with_capacity(count as usize);
GetEnvironmentVariableA(
name.as_ptr() as _,
value.as_mut_ptr(),
value.capacity() as u32,
);
value.set_len(count as usize);
Some(CString::from_vec_with_nul_unchecked(value))
}
}
fn make_child_cmdline(is_gui: bool) -> Vec<u8> {
unsafe {
let python_exe = find_python_exe(is_gui);
let my_cmdline = CStr::from_ptr(GetCommandLineA() as _);
let mut child_cmdline = Vec::<u8>::new();
child_cmdline.push(b'"');
for byte in python_exe.as_bytes() {
if *byte == b'"' {
// 3 double quotes: one to end the quoted span, one to become a literal double-quote,
// and one to start a new quoted span.
child_cmdline.extend(br#"""""#);
} else {
child_cmdline.push(*byte);
}
}
child_cmdline.extend(br#"" "#);
child_cmdline.extend(my_cmdline.to_bytes_with_nul());
//eprintln!("new_cmdline: {}", core::str::from_utf8_unchecked(new_cmdline.as_slice()));
child_cmdline
}
}
/// The scripts are in the same directory as the Python interpreter, so we can find Python by getting the locations of
/// the current .exe and replacing the filename with `python[w].exe`.
fn find_python_exe(is_gui: bool) -> CString {
unsafe {
// MAX_PATH is a lie, Windows paths can be longer.
// https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation
// But it's a good first guess, usually paths are short and we should only need a single attempt.
let mut buffer: Vec<u8> = vec![0; MAX_PATH as usize];
loop {
// Call the Windows API function to get the module file name
let len = GetModuleFileNameA(0, buffer.as_mut_ptr(), buffer.len() as u32);
// That's the error condition because len doesn't include the trailing null byte
if len as usize == buffer.len() {
let last_error = GetLastError();
match last_error {
ERROR_INSUFFICIENT_BUFFER => {
SetLastError(ERROR_SUCCESS);
// Try again with twice the size
buffer.resize(buffer.len() * 2, 0);
}
err => {
eprintln!("Failed to get executable name: code {}", err);
ExitProcess(1);
}
}
} else {
buffer.truncate(len as usize + b"\0".len());
break;
}
}
// Replace the filename (the last segment of the path) with "python.exe"
// Assumption: We are not in an encoding where a backslash byte can be part of a larger character.
let Some(last_backslash) = buffer.iter().rposition(|byte| *byte == b'\\') else {
eprintln!(
"Invalid current exe path (missing backslash): `{}`",
CString::from_vec_with_nul_unchecked(buffer)
.to_string_lossy()
.as_ref()
);
ExitProcess(1);
};
buffer.truncate(last_backslash + 1);
buffer.extend_from_slice(if is_gui {
b"pythonw.exe\0"
} else {
b"python.exe\0"
});
CString::from_vec_with_nul_unchecked(buffer)
}
}
fn make_job_object() -> HANDLE {
unsafe {
let job = CreateJobObjectW(null(), null());
let mut job_info = MaybeUninit::<JOBOBJECT_EXTENDED_LIMIT_INFORMATION>::uninit();
let mut retlen = 0u32;
check!(QueryInformationJobObject(
job,
JobObjectExtendedLimitInformation,
job_info.as_mut_ptr() as *mut _,
job_info.size_of(),
&mut retlen as *mut _,
));
let mut job_info = job_info.assume_init();
job_info.BasicLimitInformation.LimitFlags |= JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE;
job_info.BasicLimitInformation.LimitFlags |= JOB_OBJECT_LIMIT_SILENT_BREAKAWAY_OK;
check!(SetInformationJobObject(
job,
JobObjectExtendedLimitInformation,
addr_of!(job_info) as *const _,
job_info.size_of(),
));
job
}
}
fn spawn_child(si: &STARTUPINFOA, child_cmdline: &mut [u8]) -> HANDLE {
unsafe {
if si.dwFlags & STARTF_USESTDHANDLES != 0 {
// ignore errors from these -- if the handle's not inheritable/not valid, then nothing
// we can do
SetHandleInformation(si.hStdInput, HANDLE_FLAG_INHERIT, HANDLE_FLAG_INHERIT);
SetHandleInformation(si.hStdOutput, HANDLE_FLAG_INHERIT, HANDLE_FLAG_INHERIT);
SetHandleInformation(si.hStdError, HANDLE_FLAG_INHERIT, HANDLE_FLAG_INHERIT);
}
let mut child_process_info = MaybeUninit::<PROCESS_INFORMATION>::uninit();
check!(CreateProcessA(
null(),
// Why does this have to be mutable? Who knows. But it's not a mistake --
// MS explicitly documents that this buffer might be mutated by CreateProcess.
child_cmdline.as_mut_ptr(),
null(),
null(),
1,
0,
null(),
null(),
addr_of!(*si),
child_process_info.as_mut_ptr(),
));
let child_process_info = child_process_info.assume_init();
CloseHandle(child_process_info.hThread);
child_process_info.hProcess
}
}
// Apparently, the Windows C runtime has a secret way to pass file descriptors into child
// processes, by using the .lpReserved2 field. We want to close those file descriptors too.
// The UCRT source code has details on the memory layout (see also initialize_inherited_file_handles_nolock):
// https://github.com/huangqinjin/ucrt/blob/10.0.19041.0/lowio/ioinit.cpp#L190-L223
fn close_handles(si: &STARTUPINFOA) {
unsafe {
for handle in [STD_INPUT_HANDLE, STD_OUTPUT_HANDLE] {
CloseHandle(GetStdHandle(handle));
SetStdHandle(handle, INVALID_HANDLE_VALUE);
}
if si.cbReserved2 == 0 || si.lpReserved2.is_null() {
return;
}
let crt_magic = si.lpReserved2 as *const u32;
let handle_count = crt_magic.read_unaligned() as isize;
let handle_start = crt_magic.offset(1 + handle_count);
for i in 0..handle_count {
CloseHandle(handle_start.offset(i).read_unaligned() as HANDLE);
}
}
}
/*
I don't really understand what this function does. It's a straight port from
https://github.com/pypa/distlib/blob/master/PC/launcher.c, which has the following
comment:
End the launcher's "app starting" cursor state.
When Explorer launches a Windows (GUI) application, it displays
the "app starting" (the "pointer + hourglass") cursor for a number
of seconds, or until the app does something UI-ish (eg, creating a
window, or fetching a message). As this launcher doesn't do this
directly, that cursor remains even after the child process does these
things. We avoid that by doing the stuff in here.
See http://bugs.python.org/issue17290 and
https://github.com/pypa/pip/issues/10444#issuecomment-973408601
Why do we call `PostMessage`/`GetMessage` at the start, before waiting for the
child? (Looking at the bpo issue above, this was originally the *whole* fix.)
Is creating a window and calling PeekMessage the best way to do this? idk.
*/
fn clear_app_starting_state(child_handle: HANDLE) {
unsafe {
PostMessageA(0, 0, 0, 0);
let mut msg = MaybeUninit::<MSG>::uninit();
GetMessageA(msg.as_mut_ptr(), 0, 0, 0);
WaitForInputIdle(child_handle, INFINITE);
let hwnd = CreateWindowExA(
0,
c!("STATIC").as_ptr() as *const _,
c!("Puffin Python Trampoline").as_ptr() as *const _,
0,
0,
0,
0,
0,
HWND_MESSAGE,
0,
0,
null(),
);
PeekMessageA(msg.as_mut_ptr(), hwnd, 0, 0, 0);
DestroyWindow(hwnd);
}
}
pub fn bounce(is_gui: bool) -> ! {
unsafe {
let mut child_cmdline = make_child_cmdline(is_gui);
let job = make_job_object();
let mut si = MaybeUninit::<STARTUPINFOA>::uninit();
GetStartupInfoA(si.as_mut_ptr());
let si = si.assume_init();
let child_handle = spawn_child(&si, child_cmdline.as_mut_slice());
check!(AssignProcessToJobObject(job, child_handle));
// (best effort) Close all the handles that we can
close_handles(&si);
// (best effort) Switch to some innocuous directory so we don't hold the original
// cwd open.
if let Some(tmp) = getenv(c!("TEMP")) {
SetCurrentDirectoryA(tmp.as_ptr() as *const _);
} else {
SetCurrentDirectoryA(c!("c:\\").as_ptr() as *const _);
}
// We want to ignore control-C/control-Break/logout/etc.; the same event will
// be delivered to the child, so we let them decide whether to exit or not.
unsafe extern "system" fn control_key_handler(_: u32) -> BOOL {
1
}
SetConsoleCtrlHandler(Some(control_key_handler), 1);
if is_gui {
clear_app_starting_state(child_handle);
}
WaitForSingleObject(child_handle, INFINITE);
let mut exit_code = 0u32;
check!(GetExitCodeProcess(child_handle, addr_of_mut!(exit_code)));
ExitProcess(exit_code);
}
}

View file

@ -0,0 +1,67 @@
extern crate alloc;
use alloc::{ffi::CString, string::String};
use core::{
convert::Infallible,
ptr::{addr_of_mut, null, null_mut},
};
use ufmt_write::uWrite;
use windows_sys::Win32::{
Storage::FileSystem::WriteFile,
System::Console::{GetStdHandle, STD_ERROR_HANDLE},
UI::WindowsAndMessaging::MessageBoxA,
};
pub struct DiagnosticBuffer {
buffer: String,
}
impl DiagnosticBuffer {
pub fn new() -> DiagnosticBuffer {
DiagnosticBuffer {
buffer: String::new(),
}
}
pub fn display(self) {
unsafe {
let handle = GetStdHandle(STD_ERROR_HANDLE);
let mut written: u32 = 0;
let mut remaining = self.buffer.as_str();
while !remaining.is_empty() {
let ok = WriteFile(
handle,
remaining.as_ptr(),
remaining.len() as u32,
addr_of_mut!(written),
null_mut(),
);
if ok == 0 {
let nul_terminated = CString::new(self.buffer.as_bytes()).unwrap_unchecked();
MessageBoxA(0, nul_terminated.as_ptr() as *const _, null(), 0);
return;
}
remaining = &remaining.get_unchecked(written as usize..);
}
}
}
}
impl uWrite for DiagnosticBuffer {
type Error = Infallible;
fn write_str(&mut self, s: &str) -> Result<(), Self::Error> {
self.buffer.push_str(s);
Ok(())
}
}
#[macro_export]
macro_rules! eprintln {
($($tt:tt)*) => {{
let mut d = $crate::diagnostics::DiagnosticBuffer::new();
_ = ufmt::uwriteln!(&mut d, $($tt)*);
d.display();
}}
}

View file

@ -0,0 +1,58 @@
use core::mem::size_of;
pub trait SizeOf {
fn size_of(&self) -> u32;
}
impl<T: Sized> SizeOf for T {
fn size_of(&self) -> u32 {
size_of::<T>() as u32
}
}
// Check result of win32 API call that returns BOOL
#[macro_export]
macro_rules! check {
($e:expr) => {
if $e == 0 {
use windows_sys::Win32::{
Foundation::*,
System::{
Diagnostics::Debug::{
FormatMessageA, FORMAT_MESSAGE_ALLOCATE_BUFFER, FORMAT_MESSAGE_FROM_SYSTEM,
FORMAT_MESSAGE_IGNORE_INSERTS,
},
}
};
let err = GetLastError();
let mut msg_ptr: *mut u8 = core::ptr::null_mut();
let size = FormatMessageA(
FORMAT_MESSAGE_ALLOCATE_BUFFER
| FORMAT_MESSAGE_FROM_SYSTEM
| FORMAT_MESSAGE_IGNORE_INSERTS,
null(),
err,
0,
// Weird calling convention: this argument is typed as *mut u16,
// but if you pass FORMAT_MESSAGE_ALLOCATE_BUFFER then you have to
// *actually* pass in a *mut *mut u16 and just lie about the type.
// Getting Rust to do this requires some convincing.
core::ptr::addr_of_mut!(msg_ptr) as *mut _ as _,
0,
core::ptr::null(),
);
let msg = core::slice::from_raw_parts(msg_ptr, size as usize);
let msg = core::str::from_utf8_unchecked(msg);
$crate::eprintln!("Error: {} (from {})", msg, stringify!($e));
ExitProcess(1);
}
}
}
// CStr literal: c!("...")
#[macro_export]
macro_rules! c {
($s:literal) => {
core::ffi::CStr::from_bytes_with_nul_unchecked(concat!($s, "\0").as_bytes())
};
}

View file

@ -0,0 +1,9 @@
#![feature(panic_info_message)]
#![no_std]
extern crate alloc;
pub mod bounce;
mod diagnostics;
mod helpers;
mod runtime;

View file

@ -0,0 +1,63 @@
// Nothing in this file is directly imported anywhere else; it just fills in
// some of the no_std gaps.
extern crate alloc;
use alloc::alloc::{GlobalAlloc, Layout};
use core::ffi::c_void;
use windows_sys::Win32::System::{
Memory::{GetProcessHeap, HeapAlloc, HeapFree, HeapReAlloc, HEAP_ZERO_MEMORY},
Threading::ExitProcess,
};
use crate::eprintln;
// Windows wants this symbol. It has something to do with floating point usage?
// idk, defining it gets rid of link errors.
#[no_mangle]
#[used]
static _fltused: i32 = 0;
struct SystemAlloc;
#[global_allocator]
static SYSTEM_ALLOC: SystemAlloc = SystemAlloc;
unsafe impl Sync for SystemAlloc {}
unsafe impl GlobalAlloc for SystemAlloc {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
HeapAlloc(GetProcessHeap(), 0, layout.size()) as *mut u8
}
unsafe fn dealloc(&self, ptr: *mut u8, _layout: Layout) {
HeapFree(GetProcessHeap(), 0, ptr as *const c_void);
}
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, layout.size()) as *mut u8
}
unsafe fn realloc(&self, ptr: *mut u8, _layout: Layout, new_size: usize) -> *mut u8 {
HeapReAlloc(GetProcessHeap(), 0, ptr as *const c_void, new_size) as *mut u8
}
}
#[panic_handler]
fn panic(info: &core::panic::PanicInfo) -> ! {
if let Some(location) = info.location() {
let mut msg = "(couldn't format message)";
if let Some(msg_args) = info.message() {
if let Some(msg_str) = msg_args.as_str() {
msg = msg_str;
}
}
eprintln!(
"panic at {}:{} (column {}): {}",
location.file(),
location.line(),
location.column(),
msg,
);
}
unsafe {
ExitProcess(128);
}
}