mirrors/uv
1
0
Fork 0
mirror of https://github.com/astral-sh/uv.git synced 2025-08-04 19:08:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Add bootstrapping and isolation of development Python versions (#1105)

Browse source 
Replaces https://github.com/astral-sh/puffin/pull/1068 and #1070 which
were more complicated than I wanted.

- Introduces a `.python-versions` file which defines the Python versions
needed for development
- Adds a Bash script at `scripts/bootstrap/install` which installs the
required Python versions from `python-build-standalone` to `./bin`
- Checks in a `versions.json` file with metadata about available
versions on each platform and a `fetch-version` Python script derived
from `rye` for updating the versions
- Updates CI to use these Python builds instead of the `setup-python`
action
- Updates to the latest packse scenarios which require Python 3.8+
instead of 3.7+ since we cannot use 3.7 anymore and includes new test
coverage of patch Python version requests
- Adds a `PUFFIN_PYTHON_PATH` variable to prevent lookup of system
Python versions for isolation during development

Tested on Linux (via CI) and macOS (locally) — presumably it will be a
bit more complicated to do proper Windows support.
This commit is contained in:
Zanie Blue 2024-01-26 12:12:48 -06:00 • committed by GitHub
parent cc0e211074
commit 21577ad002
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
16 changed files with 5789 additions and 444 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.envrc Normal file
View file

@ -0,0 +1,3 @@
PATH=$PWD/bin:$PATH
export PUFFIN_PYTHON_PATH=$PWD/bin

18
.github/workflows/ci.yml vendored
View file

@ -54,16 +54,11 @@ jobs:
name: "cargo test | ${{ matrix.os }}"
steps:
- uses: actions/checkout@v4
- name: "Install Pythons"
uses: actions/setup-python@v4
with:
python-version: |
3.7
3.8
3.9
3.10
3.11
3.12
- name: "Install required Python versions"
run: |
sudo apt install direnv
scripts/bootstrap/install.sh
direnv allow .envrc
- name: "Install Rust toolchain"
run: rustup show
- uses: rui314/setup-mold@v1
@ -75,7 +70,8 @@ jobs:
with:
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: "Tests"
run: cargo nextest run --workspace --all-features --status-level skip --failure-output immediate-final --no-fail-fast -j 12
run: |
direnv exec . cargo nextest run --all --all-features --status-level skip --failure-output immediate-final --no-fail-fast -j 12
# TODO(konstin): Merge with the cargo-test job once the tests pass
windows:

3
.gitignore vendored
View file

@ -3,6 +3,9 @@
debug/
target/
# Bootstrapped Python versions
bin/
# These are backup files generated by rustfmt
**/*.rs.bk

6
.python-versions Normal file
View file

@ -0,0 +1,6 @@
3.8.12
3.8.18
3.9.18
3.10.13
3.11.7
3.12.1

49
CONTRIBUTING.md
View file

@ -2,13 +2,12 @@
## Setup
You need [Rust](https://rustup.rs/), a C compiler, and CMake to build Puffin. To run the tests, you need Python 3.8, 3.9 and 3.12.
[Rust](https://rustup.rs/), a C compiler, and CMake are required to build Puffin.
To run the tests we recommend [nextest](https://nexte.st/). Make sure to run the tests with `--all-features`, otherwise you'll miss most of our integration tests.
Testing Puffin requires multiple specific Python versions. We provide a script to bootstrap development by downloading the required versions.
### Linux
We recommend [pyenv](https://github.com/pyenv/pyenv) to manage multiple Python versions.
On Ubuntu and other Debian-based distributions, you can install the C compiler and CMake with
@ -16,10 +15,53 @@ On Ubuntu and other Debian-based distributions, you can install the C compiler a
sudo apt install build-essential cmake
```
### macOS
CMake may be installed with Homebrew:
```
brew install cmake
```
The Python bootstrapping script requires `coreutils` and `zstd`; we recommend installing them with Homebrew:
```
brew install coreutils zstd
```
See the [Python](#python) section for instructions on installing the Python versions.
### Windows
You can install CMake from the [installers](https://cmake.org/download/) or with `pipx install cmake` (make sure that the pipx install path is in `PATH`, pipx complains if it isn't).
### Python
Install required Python versions with the bootstrapping script:
```
scripts/bootstrap/install.sh
```
Then add the Python binaries to your path:
```
export PATH=$PWD/bin:$PATH
```
We also strongly recommend setting the `PUFFIN_PYTHON_PATH` variable during development; this will prevent your
system Python versions from being found during tests:
```
export PUFFIN_PYTHON_PATH=$PWD/bin
```
If you use [direnv](https://direnv.net/), these variables will be exported automatically after you run `direnv allow`.
## Testing
To run the tests we recommend [nextest](https://nexte.st/). Make sure to run the tests with `--all-features`, otherwise you'll miss most of our integration tests.
## Running inside a docker container
Source distributions can run arbitrary code on build and can make unwanted modifications to your system (https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subnormals.html, https://pypi.org/project/nvidia-pyindex/), which can even occur when just resolving requirements. To prevent this, there's a Docker container you can run commands in:
@ -32,3 +74,4 @@ docker run --rm -it -v $(pwd):/app puffin-builder /app/target/x86_64-unknown-lin
```
We recommend using this container if you don't trust the dependency tree of the package(s) you are trying to resolve or install.

29
crates/puffin-interpreter/src/interpreter.rs
View file

@ -1,3 +1,4 @@
use std::ffi::{OsStr, OsString};
use std::path::{Path, PathBuf};
use std::process::Command;
@ -136,6 +137,10 @@ impl Interpreter {
/// - If a python version is given: `pythonx.y`
/// - `python3` (unix) or `python.exe` (windows)
///
/// If `PUFFIN_PYTHON_PATH` is set, we will not check for Python versions in the
/// global PATH, instead we will search using the provided path. Virtual environments
/// will still be respected.
///
/// If a version is provided and an interpreter cannot be found with the given version,
/// we will return [`None`].
pub fn find_version(
@ -170,7 +175,8 @@ impl Interpreter {
python_version.major(),
python_version.minor()
);
if let Ok(executable) = which::which(&requested) {
if let Ok(executable) = Interpreter::find_executable(&requested) {
debug!("Resolved {requested} to {}", executable.display());
let interpreter = Interpreter::query(&executable, &platform.0, cache)?;
if version_matches(&interpreter) {
@ -179,7 +185,7 @@ impl Interpreter {
}
}
if let Ok(executable) = which::which("python3") {
if let Ok(executable) = Interpreter::find_executable("python3") {
debug!("Resolved python3 to {}", executable.display());
let interpreter = Interpreter::query(&executable, &platform.0, cache)?;
if version_matches(&interpreter) {
@ -198,7 +204,7 @@ impl Interpreter {
}
}
if let Ok(executable) = which::which("python.exe") {
if let Ok(executable) = Interpreter::find_executable("python.exe") {
let interpreter = Interpreter::query(&executable, &platform.0, cache)?;
if version_matches(&interpreter) {
return Ok(Some(interpreter));
@ -211,6 +217,23 @@ impl Interpreter {
Ok(None)
}
pub fn find_executable<R: AsRef<OsStr> + Into<OsString> + Copy>(
requested: R,
) -> Result<PathBuf, Error> {
if let Some(isolated) = std::env::var_os("PUFFIN_PYTHON_PATH") {
if let Ok(cwd) = std::env::current_dir() {
which::which_in(requested, Some(isolated), cwd)
.map_err(|err| Error::Which(requested.into(), err))
} else {
which::which_in_global(requested, Some(isolated))
.map_err(|err| Error::Which(requested.into(), err))
.and_then(|mut paths| paths.next().ok_or(Error::PythonNotFound))
}
} else {
which::which(requested).map_err(|err| Error::Which(requested.into(), err))
}
}
/// Returns the path to the Python virtual environment.
#[inline]
pub fn platform(&self) -> &Platform {

7
crates/puffin-interpreter/src/lib.rs
View file

@ -1,3 +1,4 @@
use std::ffi::OsString;
use std::io;
use std::path::PathBuf;
use std::time::SystemTimeError;
@ -49,6 +50,8 @@ pub enum Error {
NoPythonInstalledUnix,
#[error("Could not find `python.exe` in PATH and `py --list-paths` did not list any Python versions. Do you need to install Python?")]
NoPythonInstalledWindows,
#[error("Patch versions cannot be requested on Windows")]
PatchVersionRequestedWindows,
#[error("{message}:\n--- stdout:\n{stdout}\n--- stderr:\n{stderr}\n---")]
PythonSubcommandOutput {
message: String,
@ -63,6 +66,6 @@ pub enum Error {
Encode(#[from] rmp_serde::encode::Error),
#[error("Failed to parse pyvenv.cfg")]
Cfg(#[from] cfg::Error),
#[error("Couldn't find `{0}` in PATH")]
Which(PathBuf, #[source] which::Error),
#[error("Couldn't find `{}` in PATH", _0.to_string_lossy())]
Which(OsString, #[source] which::Error),
}

29
crates/puffin-interpreter/src/python_query.rs
View file

@ -7,7 +7,7 @@ use once_cell::sync::Lazy;
use regex::Regex;
use tracing::{info_span, instrument};
use crate::Error;
use crate::{Error, Interpreter};
/// ```text
/// -V:3.12 C:\Users\Ferris\AppData\Local\Programs\Python\Python312\python.exe
@ -27,23 +27,30 @@ static PY_LIST_PATHS: Lazy<Regex> = Lazy::new(|| {
/// * `-p /home/ferris/.local/bin/python3.10` uses this exact Python.
#[instrument]
pub fn find_requested_python(request: &str) -> Result<PathBuf, Error> {
let major_minor = request
.split_once('.')
.and_then(|(major, minor)| Some((major.parse::<u8>().ok()?, minor.parse::<u8>().ok()?)));
if let Some((major, minor)) = major_minor {
// `-p 3.10`
let versions = request
.splitn(3, '.')
.map(str::parse::<u8>)
.collect::<Result<Vec<_>, _>>();
if let Ok(versions) = versions {
// `-p 3.10` or `-p 3.10.1`
if cfg!(unix) {
let formatted = PathBuf::from(format!("python{major}.{minor}"));
which::which(&formatted).map_err(|err| Error::Which(formatted, err))
let formatted = PathBuf::from(format!("python{request}"));
Interpreter::find_executable(&formatted)
} else if cfg!(windows) {
find_python_windows(major, minor)?.ok_or(Error::NoSuchPython { major, minor })
if let [major, minor] = versions.as_slice() {
find_python_windows(*major, *minor)?.ok_or(Error::NoSuchPython {
major: *major,
minor: *minor,
})
} else {
Err(Error::PatchVersionRequestedWindows)
}
} else {
unimplemented!("Only Windows and Unix are supported")
}
} else if !request.contains(std::path::MAIN_SEPARATOR) {
// `-p python3.10`; Generally not used on windows because all Python are `python.exe`.
let request = PathBuf::from(request);
which::which(&request).map_err(|err| Error::Which(request, err))
Interpreter::find_executable(request)
} else {
// `-p /home/ferris/.local/bin/python3.10`
Ok(fs_err::canonicalize(request)?)

1
crates/puffin/tests/pip_compile.rs
View file

@ -696,6 +696,7 @@ fn compile_python_37() -> Result<()> {
----- stdout -----
----- stderr -----
warning: The requested Python version 3.7 is not available; 3.12.1 will be used to build dependencies instead.
× No solution found when resolving dependencies:
╰─▶ Because the requested Python version (3.7) does not satisfy Python>=3.8
and black==23.10.1 depends on Python>=3.8, we can conclude that

173
crates/puffin/tests/pip_compile_scenarios.rs
View file

@ -1,7 +1,7 @@
//! DO NOT EDIT
//!
//! Generated with ./scripts/scenarios/update.py
//! Scenarios from <https://github.com/zanieb/packse/tree/78f34eec66acfba9c723285764dc1f4b841f4961/scenarios>
//! Scenarios from <https://github.com/zanieb/packse/tree/e944cb4c8f5d68457d0462ee19106509f63b8d34/scenarios>
//!
#![cfg(all(feature = "python", feature = "pypi"))]
@ -23,7 +23,7 @@ mod common;
/// resolution.
///
/// ```text
/// 818d78ce
/// 006fed96
/// ├── environment
/// │ └── python3.9
/// ├── root
@ -41,11 +41,11 @@ fn requires_incompatible_python_version_compatible_override() -> Result<()> {
// In addition to the standard filters, swap out package names for more realistic messages
let mut filters = INSTA_FILTERS.to_vec();
filters.push((r"a-818d78ce", "albatross"));
filters.push((r"-818d78ce", ""));
filters.push((r"a-006fed96", "albatross"));
filters.push((r"-006fed96", ""));
let requirements_in = temp_dir.child("requirements.in");
requirements_in.write_str("a-818d78ce==1.0.0")?;
requirements_in.write_str("a-006fed96==1.0.0")?;
insta::with_settings!({
filters => filters
@ -83,7 +83,7 @@ fn requires_incompatible_python_version_compatible_override() -> Result<()> {
/// request an incompatible Python version for package resolution.
///
/// ```text
/// e94b8bc2
/// 8c1b0389
/// ├── environment
/// │ └── python3.11
/// ├── root
@ -101,11 +101,11 @@ fn requires_compatible_python_version_incompatible_override() -> Result<()> {
// In addition to the standard filters, swap out package names for more realistic messages
let mut filters = INSTA_FILTERS.to_vec();
filters.push((r"a-e94b8bc2", "albatross"));
filters.push((r"-e94b8bc2", ""));
filters.push((r"a-8c1b0389", "albatross"));
filters.push((r"-8c1b0389", ""));
let requirements_in = temp_dir.child("requirements.in");
requirements_in.write_str("a-e94b8bc2==1.0.0")?;
requirements_in.write_str("a-8c1b0389==1.0.0")?;
insta::with_settings!({
filters => filters
@ -143,7 +143,7 @@ fn requires_compatible_python_version_incompatible_override() -> Result<()> {
/// source distributions available for the package.
///
/// ```text
/// 367303df
/// b8ee1c03
/// ├── environment
/// │ └── python3.9
/// ├── root
@ -161,11 +161,11 @@ fn requires_incompatible_python_version_compatible_override_no_wheels() -> Resul
// In addition to the standard filters, swap out package names for more realistic messages
let mut filters = INSTA_FILTERS.to_vec();
filters.push((r"a-367303df", "albatross"));
filters.push((r"-367303df", ""));
filters.push((r"a-b8ee1c03", "albatross"));
filters.push((r"-b8ee1c03", ""));
let requirements_in = temp_dir.child("requirements.in");
requirements_in.write_str("a-367303df==1.0.0")?;
requirements_in.write_str("a-b8ee1c03==1.0.0")?;
// Since there are no wheels for the package and it is not compatible with the
// local installation, we cannot build the source distribution to determine its
@ -207,7 +207,7 @@ fn requires_incompatible_python_version_compatible_override_no_wheels() -> Resul
/// wheel available for the package, but it does not have a compatible tag.
///
/// ```text
/// 7d66d27e
/// c0ea406a
/// ├── environment
/// │ └── python3.9
/// ├── root
@ -225,11 +225,11 @@ fn requires_incompatible_python_version_compatible_override_no_compatible_wheels
// In addition to the standard filters, swap out package names for more realistic messages
let mut filters = INSTA_FILTERS.to_vec();
filters.push((r"a-7d66d27e", "albatross"));
filters.push((r"-7d66d27e", ""));
filters.push((r"a-c0ea406a", "albatross"));
filters.push((r"-c0ea406a", ""));
let requirements_in = temp_dir.child("requirements.in");
requirements_in.write_str("a-7d66d27e==1.0.0")?;
requirements_in.write_str("a-c0ea406a==1.0.0")?;
// Since there are no compatible wheels for the package and it is not compatible
// with the local installation, we cannot build the source distribution to
@ -249,12 +249,15 @@ fn requires_incompatible_python_version_compatible_override_no_compatible_wheels
.env("VIRTUAL_ENV", venv.as_os_str())
.env("PUFFIN_NO_WRAP", "1")
.current_dir(&temp_dir), @r###"
success: false
exit_code: 2
success: true
exit_code: 0
----- stdout -----
# This file was autogenerated by Puffin v[VERSION] via the following command:
# puffin pip compile requirements.in --python-version=3.11 --extra-index-url https://test.pypi.org/simple --cache-dir [CACHE_DIR]
albatross==1.0.0
----- stderr -----
error: Package `albatross` was not found in the registry.
Resolved 1 package in [TIME]
"###);
});
@ -269,7 +272,7 @@ fn requires_incompatible_python_version_compatible_override_no_compatible_wheels
/// there is an incompatible version with a wheel available.
///
/// ```text
/// 47c905cb
/// 08a4e843
/// ├── environment
/// │ └── python3.9
/// ├── root
@ -290,11 +293,11 @@ fn requires_incompatible_python_version_compatible_override_other_wheel() -> Res
// In addition to the standard filters, swap out package names for more realistic messages
let mut filters = INSTA_FILTERS.to_vec();
filters.push((r"a-47c905cb", "albatross"));
filters.push((r"-47c905cb", ""));
filters.push((r"a-08a4e843", "albatross"));
filters.push((r"-08a4e843", ""));
let requirements_in = temp_dir.child("requirements.in");
requirements_in.write_str("a-47c905cb")?;
requirements_in.write_str("a-08a4e843")?;
// Since there are no wheels for the version of the package compatible with the
// target and it is not compatible with the local installation, we cannot build the
@ -329,3 +332,125 @@ fn requires_incompatible_python_version_compatible_override_other_wheel() -> Res
Ok(())
}
/// requires-python-patch-version-override-no-patch
///
/// The user requires a package which requires a Python version with a patch version
/// and the user provides a target version without a patch version.
///
/// ```text
/// 2e1edfd6
/// ├── environment
/// │ └── python3.8.18
/// ├── root
/// │ └── requires a==1.0.0
/// │ └── satisfied by a-1.0.0
/// └── a
/// └── a-1.0.0
/// └── requires python>=3.8.4
/// ```
#[test]
fn requires_python_patch_version_override_no_patch() -> Result<()> {
let temp_dir = assert_fs::TempDir::new()?;
let cache_dir = assert_fs::TempDir::new()?;
let venv = create_venv(&temp_dir, &cache_dir, "3.8.18");
// In addition to the standard filters, swap out package names for more realistic messages
let mut filters = INSTA_FILTERS.to_vec();
filters.push((r"a-2e1edfd6", "albatross"));
filters.push((r"-2e1edfd6", ""));
let requirements_in = temp_dir.child("requirements.in");
requirements_in.write_str("a-2e1edfd6==1.0.0")?;
// Since the resolver is asked to solve with 3.8, the minimum compatible Python
// requirement is treated as 3.8.0.
insta::with_settings!({
filters => filters
}, {
assert_cmd_snapshot!(Command::new(get_cargo_bin(BIN_NAME))
.arg("pip")
.arg("compile")
.arg("requirements.in")
.arg("--python-version=3.8")
.arg("--extra-index-url")
.arg("https://test.pypi.org/simple")
.arg("--cache-dir")
.arg(cache_dir.path())
.env("VIRTUAL_ENV", venv.as_os_str())
.env("PUFFIN_NO_WRAP", "1")
.current_dir(&temp_dir), @r###"
success: false
exit_code: 1
----- stdout -----
----- stderr -----
× No solution found when resolving dependencies:
╰─▶ Because the requested Python version (3.8) does not satisfy Python>=3.8.4 and albatross==1.0.0 depends on Python>=3.8.4, we can conclude that albatross==1.0.0 cannot be used.
And because you require albatross==1.0.0, we can conclude that the requirements are unsatisfiable.
"###);
});
Ok(())
}
/// requires-python-patch-version-override-patch-compatible
///
/// The user requires a package which requires a Python version with a patch version
/// and the user provides a target version with a compatible patch version.
///
/// ```text
/// 844899bd
/// ├── environment
/// │ └── python3.8.18
/// ├── root
/// │ └── requires a==1.0.0
/// │ └── satisfied by a-1.0.0
/// └── a
/// └── a-1.0.0
/// └── requires python>=3.8.0
/// ```
#[test]
fn requires_python_patch_version_override_patch_compatible() -> Result<()> {
let temp_dir = assert_fs::TempDir::new()?;
let cache_dir = assert_fs::TempDir::new()?;
let venv = create_venv(&temp_dir, &cache_dir, "3.8.18");
// In addition to the standard filters, swap out package names for more realistic messages
let mut filters = INSTA_FILTERS.to_vec();
filters.push((r"a-844899bd", "albatross"));
filters.push((r"-844899bd", ""));
let requirements_in = temp_dir.child("requirements.in");
requirements_in.write_str("a-844899bd==1.0.0")?;
insta::with_settings!({
filters => filters
}, {
assert_cmd_snapshot!(Command::new(get_cargo_bin(BIN_NAME))
.arg("pip")
.arg("compile")
.arg("requirements.in")
.arg("--python-version=3.8.0")
.arg("--extra-index-url")
.arg("https://test.pypi.org/simple")
.arg("--cache-dir")
.arg(cache_dir.path())
.env("VIRTUAL_ENV", venv.as_os_str())
.env("PUFFIN_NO_WRAP", "1")
.current_dir(&temp_dir), @r###"
success: true
exit_code: 0
----- stdout -----
# This file was autogenerated by Puffin v[VERSION] via the following command:
# puffin pip compile requirements.in --python-version=3.8.0 --extra-index-url https://test.pypi.org/simple --cache-dir [CACHE_DIR]
albatross==1.0.0
----- stderr -----
warning: The requested Python version 3.8.0 is not available; 3.8.18 will be used to build dependencies instead.
Resolved 1 package in [TIME]
"###);
});
Ok(())
}

829
crates/puffin/tests/pip_install_scenarios.rs
View file

File diff suppressed because it is too large Load diff

38
crates/puffin/tests/venv.rs
View file

@ -184,7 +184,7 @@ fn create_venv_unknown_python_patch() -> Result<()> {
----- stdout -----
----- stderr -----
× Couldn't find `3.8.0` in PATH
× Couldn't find `python3.8.0` in PATH
╰─▶ cannot find binary path
"###);
});
@ -193,3 +193,39 @@ fn create_venv_unknown_python_patch() -> Result<()> {
Ok(())
}
#[test]
fn create_venv_python_patch() -> Result<()> {
let temp_dir = assert_fs::TempDir::new()?;
let cache_dir = assert_fs::TempDir::new()?;
let venv = temp_dir.child(".venv");
let filter_venv = regex::escape(&venv.display().to_string());
insta::with_settings!({
filters => vec![
(r"interpreter at .+", "interpreter at [PATH]"),
(&filter_venv, "/home/ferris/project/.venv"),
]
}, {
assert_cmd_snapshot!(Command::new(get_cargo_bin(BIN_NAME))
.arg("venv")
.arg(venv.as_os_str())
.arg("--python")
.arg("3.12.1")
.arg("--cache-dir")
.arg(cache_dir.path())
.current_dir(&temp_dir), @r###"
success: true
exit_code: 0
----- stdout -----
----- stderr -----
Using Python 3.12.1 interpreter at [PATH]
Creating virtual environment at: /home/ferris/project/.venv
"###);
});
venv.assert(predicates::path::is_dir());
Ok(())
}

273
scripts/bootstrap/fetch-version-metadata.py Executable file
View file

@ -0,0 +1,273 @@
#!/usr/bin/env python3.12
"""
Fetch Python version metadata.
Generates the bootstrap `versions.json` file.
Installation:
pip install requests==2.31.0
Usage:
scripts/bootstrap/fetch-versions
Acknowledgements:
Derived from https://github.com/mitsuhiko/rye/tree/f9822267a7f00332d15be8551f89a212e7bc9017
Originally authored by Armin Ronacher under the MIT license
"""
import argparse
import hashlib
import json
import logging
import os
import re
import sys
from itertools import chain
from pathlib import Path
from urllib.parse import unquote
try:
import requests
except ImportError:
print("ERROR: requests is required; install with `pip install requests==2.31.0`")
sys.exit(1)
SELF_DIR = Path(__file__).parent
RELEASE_URL = "https://api.github.com/repos/indygreg/python-build-standalone/releases"
HEADERS = {
"X-GitHub-Api-Version": "2022-11-28",
}
VERSIONS_FILE = SELF_DIR / "versions.json"
FLAVOR_PREFERENCES = [
"shared-pgo",
"shared-noopt",
"shared-noopt",
"static-noopt",
"gnu-pgo+lto",
"gnu-lto",
"gnu-pgo",
"pgo+lto",
"lto",
"pgo",
]
HIDDEN_FLAVORS = [
"debug",
"noopt",
"install_only",
]
SPECIAL_TRIPLES = {
"macos": "x86_64-apple-darwin",
"linux64": "x86_64-unknown-linux",
"windows-amd64": "x86_64-pc-windows",
"windows-x86": "i686-pc-windows",
"linux64-musl": "x86_64-unknown-linux",
}
_filename_re = re.compile(
r"""(?x)
^
cpython-(?P<ver>\d+\.\d+\.\d+?)
(?:\+\d+)?
-(?P<triple>.*?)
(?:-[\dT]+)?\.tar\.(?:gz|zst)
$
"""
)
_suffix_re = re.compile(
r"""(?x)^(.*?)-(%s)$"""
% (
"|".join(
map(
re.escape,
sorted(FLAVOR_PREFERENCES + HIDDEN_FLAVORS, key=len, reverse=True),
)
)
)
)
# to match the output of the `arch` command
ARCH_MAP = {"aarch64": "arm64"}
def parse_filename(filename):
match = _filename_re.match(filename)
if match is None:
return
version, triple = match.groups()
if triple.endswith("-full"):
triple = triple[:-5]
match = _suffix_re.match(triple)
if match is not None:
triple, suffix = match.groups()
else:
suffix = None
return (version, triple, suffix)
def normalize_triple(triple):
if "-musl" in triple or "-static" in triple:
logging.debug("Skipping %r: unknown triple", triple)
return
triple = SPECIAL_TRIPLES.get(triple, triple)
pieces = triple.split("-")
try:
arch = pieces[0]
# Normalize
arch = ARCH_MAP.get(arch, arch)
platform = pieces[2]
except IndexError:
logging.debug("Skipping %r: unknown triple", triple)
return
return "%s-%s" % (arch, platform)
def read_sha256(session, url):
resp = session.get(url + ".sha256")
if not resp.ok:
return None
return resp.text.strip()
def sha256(path):
h = hashlib.sha256()
with open(path, "rb") as file:
while True:
# Reading is buffered, so we can read smaller chunks.
chunk = file.read(h.block_size)
if not chunk:
break
h.update(chunk)
return h.hexdigest()
def _sort_key(info):
triple, flavor, url = info
try:
pref = FLAVOR_PREFERENCES.index(flavor)
except ValueError:
pref = len(FLAVOR_PREFERENCES) + 1
return pref
def get_session() -> requests.Session:
session = requests.Session()
session.headers = HEADERS.copy()
token = os.environ.get("GITHUB_TOKEN")
if token:
session.headers["Authorization"] = "Bearer " + token
else:
logging.warning(
"An authentication token was not found at `GITHUB_TOKEN`, rate limits may be encountered.",
)
return session
def find(args):
"""
Find available Python versions and write metadata to a file.
"""
results = {}
session = get_session()
for page in range(1, 100):
logging.debug("Reading release page %s...", page)
resp = session.get("%s?page=%d" % (RELEASE_URL, page))
rows = resp.json()
if not rows:
break
for row in rows:
for asset in row["assets"]:
url = asset["browser_download_url"]
base_name = unquote(url.rsplit("/")[-1])
if base_name.endswith(".sha256"):
continue
info = parse_filename(base_name)
if info is None:
continue
py_ver, triple, flavor = info
if "-static" in triple or (flavor and "noopt" in flavor):
continue
triple = normalize_triple(triple)
if triple is None:
continue
results.setdefault(py_ver, []).append((triple, flavor, url))
cpython_results = {}
for py_ver, choices in results.items():
choices.sort(key=_sort_key)
urls = {}
for triple, flavor, url in choices:
triple = tuple(triple.split("-"))
if triple in urls:
continue
urls[triple] = url
cpython_results[tuple(map(int, py_ver.split(".")))] = urls
final_results = {}
for interpreter, py_ver, choices in sorted(
chain(
(("cpython",) + x for x in cpython_results.items()),
),
key=lambda x: x[:2],
reverse=True,
):
for (arch, platform), url in sorted(choices.items()):
key = "%s-%s.%s.%s-%s-%s" % (interpreter, *py_ver, platform, arch)
logging.info("Found %s", key)
sha256 = read_sha256(session, url)
final_results[key] = {
"name": interpreter,
"arch": arch,
"os": platform,
"major": py_ver[0],
"minor": py_ver[1],
"patch": py_ver[2],
"url": url,
"sha256": sha256,
}
VERSIONS_FILE.parent.mkdir(parents=True, exist_ok=True)
VERSIONS_FILE.write_text(json.dumps(final_results, indent=2))
def main():
parser = argparse.ArgumentParser(description="Fetch Python version metadata.")
parser.add_argument(
"-v",
"--verbose",
action="store_true",
help="Enable debug logging",
)
parser.add_argument(
"-q",
"--quiet",
action="store_true",
help="Disable logging",
)
args = parser.parse_args()
if args.quiet:
log_level = logging.CRITICAL
elif args.verbose:
log_level = logging.DEBUG
else:
log_level = logging.INFO
logging.basicConfig(
level=log_level,
format="%(asctime)s %(levelname)s %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
find(args)
if __name__ == "__main__":
main()

118
scripts/bootstrap/install.sh Executable file
View file

@ -0,0 +1,118 @@
#!/usr/bin/env bash
#
# Download required Python versions and install to `bin`
# Uses prebuilt Python distributions from indygreg/python-build-standalone
#
# Requirements
#
# macOS
#
# brew install zstd jq coreutils
#
# Ubuntu
#
# apt install zstd jq
#
# Arch Linux
#
# pacman -S zstd jq
#
# Windows
#
# winget install jqlang.jq
#
# Usage
#
# ./scripts/bootstrap/install.sh
#
# The Python versions are installed from `.python_versions`.
# Python versions are linked in-order such that the _last_ defined version will be the default.
#
# Version metadata can be updated with `fetch-version-metadata.py` which requires Python 3.12
set -euo pipefail
# Convenience function for displaying URLs
function urldecode() { : "${*//+/ }"; echo -e "${_//%/\\x}"; }
# Convenience function for checking that a command exists.
requires() {
cmd="$1"
if ! command -v "$cmd" > /dev/null 2>&1; then
echo "DEPENDENCY MISSING: $(basename $0) requires $cmd to be installed" >&2
exit 1
fi
}
requires jq
requires zstd
# Setup some file paths
this_dir=$(realpath "$(dirname "$0")")
root_dir=$(dirname "$(dirname "$this_dir")")
bin_dir="$root_dir/bin"
install_dir="$bin_dir/versions"
versions_file="$root_dir/.python-versions"
versions_metadata="$this_dir/versions.json"
# Determine system metadata
os=$(uname -s | tr '[:upper:]' '[:lower:]')
arch=$(uname -m)
interpreter='cpython'
# On macOS, we need a newer version of `realpath` for `--relative-to` support
realpath="$(which grealpath || which realpath)"
# Read requested versions into an array
readarray -t versions < "$versions_file"
# Install each version
for version in "${versions[@]}"; do
key="$interpreter-$version-$os-$arch"
echo "Installing $key"
url=$(jq --arg key "$key" '.[$key] | .url' -r < "$versions_metadata")
if [ "$url" == 'null' ]; then
echo "No matching download for $key"
exit 1
fi
filename=$(basename "$url")
echo "Downloading $(urldecode "$filename")"
curl -L --progress-bar -o "$filename" "$url" --output-dir "$this_dir"
expected_sha=$(jq --arg key "$key" '.[$key] | .sha256' -r < "$versions_metadata")
if [ "$expected_sha" == 'null' ]; then
echo "WARNING: no checksum for $key"
else
echo -n "Verifying checksum..."
echo "$expected_sha $this_dir/$filename" | sha256sum -c --quiet
echo " OK"
fi
install_key="$install_dir/$interpreter@$version"
rm -rf "$install_key"
echo "Extracting to $($realpath --relative-to="$root_dir" "$install_key")"
mkdir -p "$install_key"
zstd -d "$this_dir/$filename" --stdout | tar -x -C "$install_key"
# Setup the installation
mv "$install_key/python/"* "$install_key"
# Use relative paths for links so if the bin is moved they don't break
link=$($realpath --relative-to="$bin_dir" "$install_key/install/bin/python3")
minor=$(jq --arg key "$key" '.[$key] | .minor' -r < "$versions_metadata")
# Link as all version tuples, later versions in the file will take precedence
ln -sf "./$link" "$bin_dir/python$version"
ln -sf "./$link" "$bin_dir/python3.$minor"
ln -sf "./$link" "$bin_dir/python3"
ln -sf "./$link" "$bin_dir/python"
echo "Installed as python$version"
# Cleanup
rmdir "$install_key/python/"
rm "$this_dir/$filename"
done
echo "Done!"

4652
scripts/bootstrap/versions.json Normal file
View file

File diff suppressed because it is too large Load diff

5
scripts/scenarios/update.py
View file

@ -39,14 +39,13 @@ Requirements:
import json
import shutil
import os
import subprocess
import sys
import textwrap
from pathlib import Path
PACKSE_COMMIT = "78f34eec66acfba9c723285764dc1f4b841f4961"
PACKSE_COMMIT = "e944cb4c8f5d68457d0462ee19106509f63b8d34"
TOOL_ROOT = Path(__file__).parent
TEMPLATES = TOOL_ROOT / "templates"
INSTALL_TEMPLATE = TEMPLATES / "install.mustache"
@ -149,6 +148,8 @@ print("Loading scenario metadata...", file=sys.stderr)
data = json.loads(
subprocess.check_output(
[
sys.executable,
"-m",
"packse",
"inspect",
"--short-names",