Third attempt at documenting an installation scheme; this one feels like

it could work--reasonably short and straightforward to document, but
flexible enough that it will be able to handle more than just modules
and extensions in the future.
This commit is contained in:
Greg Ward 2000-03-18 15:11:50 +00:00
parent bae57a88a6
commit 295765630a

View file

@ -30,12 +30,12 @@
\newcommand{\option}[1]{\textsf{\small{#1}}}
\newcommand{\filevar}[1]{{\textsl{\filenq{#1}}}}
\newcommand{\homefile}[1]{\file{\tilde/#1}}
\newcommand{\comingsoon}{\emph{Coming soon$\ \ldots$}}
\newcommand{\comingsoon}{\emph{Coming soon...}}
% And how about these? Very handy for writing pathnames (tilde for
% Unix, backslash for DOS/Windows).
\renewcommand{\tilde}{\raisebox{-0.5ex}{\symbol{126}}}
\newcommand{\bslash}{\symbol{92}}
\newcommand{\bslash}{\symbol{92}} % XXX only works in tt fonts!
\begin{document}
@ -52,8 +52,6 @@
\section{Introduction}
\label{sec:intro}
\comingsoon
\subsection{The new way: Distutils}
\label{sec:new-way}
@ -69,9 +67,10 @@
\section{Normal Build and Install}
\section{Standard Build and Install}
\label{sec:normal-install}
% This will cover:
% * setup.py install (the usual thing)
% * setup.py build (if you like doing things one-at-a-time)
@ -82,222 +81,334 @@
\comingsoon
\section{Custom Extension Building}
\label{sec:custom-ext}
% This will cover:
% * normal extension build -- stress that it doesn't matter, you
% do the same thing whether there are extensions or not
% * what you might want to customize: compiler and compiler
% flags (warn of the dangers); per-file compiler flags
% (not handled yet!)
% * when things go wrong: I don't know! (and I don't know what
% to do, either!)
\comingsoon
% takes eight args (four pairs):
% pure module distribution base + directory
% non-pure module distribution base + directory
% script base + directory
% data base + directory
% ...and will no doubt take more args in future!
\newcommand{\installscheme}[8]
{\begin{tableiii}{lll}{textrm}
{Type of file}
{Installation Directory}
{Override option}
\lineiii{pure module distribution}
{\filevar{#1}\filenq{#2}}
{\option{install-purelib}}
\lineiii{non-pure module distribution}
{\filevar{#3}\filenq{#4}}
{\option{install-platlib}}
\lineiii{scripts}
{\filevar{#5}\filenq{#6}}
{\option{install-scripts}}
\lineiii{data}
{\filevar{#7}\filenq{#8}}
{\option{install-data}}
\end{tableiii}}
\section{Custom Installation (\UNIX)}
\label{sec:custom-install-unix}
% XXX probably should banish mentions of Windows here to the
% separate "Non-standard installation (Windows)" section.
A \dfn{custom installation} is where you install modules to a location
that's not in Python's default module search path. There are a couple
of reasons you might want to do this; the most typical is simply that
you don't have permission to write to the standard Python library
directory. Or, even if you do have write permission to the standard
library, you might wish to install a module distribution into a
non-standard place for testing or experimentation. (This is especially
useful when upgrading an existing module distribution: you might want to
make sure that your existing scripts continue to work as before, and
only then install the upgrade ``for real.'')
(XXX terminology: I keep saying ``standard Python library directory''
when I really mean ``the site-packages directory under the standard
Python library directory''. Is there a better way?)
In any event, you can easily install to non-standard locations with a
couple of options to the \command{install} command:
\begin{tableii}{ll}{option}{Option}{Description}
\lineii {install-lib}
{install directory for modules from pure Python distributions}
\lineii {install-platlib}
{install directory for modules from distributions with extensions}
\lineii {prefix}
{override \code{sys.prefix}:
point to an alternate Python installation}
\lineii {exec-prefix}
{override \code{sys.exec_prefix}:
point to an alternate Python installation}
\lineii {install-path}
{extra sub-path to append to \option{install-lib} (for
non-package-ized distributions)}
\end{tableii}
Of these, the most commonly used will probably be \option{install-lib}
and \option{install-platlib}: you use them to point module installation
right at a particular directory. (You'll only need
\option{install-platlib} if you maintain a multi-platform installation,
which is often done on \UNIX{} networks with different architectures and
operating systems.) The two prefix options are intended for the
somewhat arcane purpose of installing modules into a different Python
installation than the Python binary used to perform the installation.
The last, \option{install-path}, is mainly used for module developers to
ensure that their module will go into a directory of their own, but it
may occasionally be useful to you as a module installer.
\subsection{Directly specifying installation directories}
\label{sec:install-dirs}
\section{Alternate Installation}
\label{sec:alt-install}
The most common type of custom module installation is where you maintain
a personal stash of Python modules under your home directory, say in
\homefile{lib/python}. If you only care about a single platform
there, then you only need to specify the \option{install-lib} option and
can forget about \option{install-platlib}:
Often, it is necessary or desirable to install modules to a location
other than the standard location for third-party Python modules. For
example, on a Unix system you might not have permission to write to the
standard third-party module directory. Or you might wish to try out a
module before making it a standard part of your local Python
installation; this is especially true when upgrading a distribution
already present: you want to make sure your existing base of scripts
still works with the new version before actually upgrading.
The Distutils \command{install} command is designed to make installing
module distributions to an alternate location simple and painless. The
basic idea is that you supply a base directory for the installation, and
the \command{install} command picks a set of directories (called an
\emph{installation scheme}) under this base directory in which to
install files. The details differ across platforms, so read whichever
of the following section applies to you.
\subsection{Alternate installation: Unix (the home scheme)}
\label{sec:alt-unix-prefix}
Under Unix, there are two ways to perform an alternate installation.
The ``prefix scheme'' is similar to how alternate installation works
under Windows and Mac OS, but is not necessarily the most useful way to
maintain a personal Python library. Hence, we document the more
convenient and commonly useful ``home scheme'' first.
The idea behind the ``home scheme'' is that you are building and
maintaining a personal stash of Python modules, probably under your home
directory. Installing a new module distribution is as simple as
\begin{verbatim}
python setup.py install --install-lib=~/lib/python
python setup.py install --home
\end{verbatim}
You can, of course, supply whatever directory you like in place of
\homefile{lib/python}. More importantly, you can specify this
directory permanently in your personal configuration file (XXX
filename?):
or
\begin{verbatim}
python setup.py install --home=<dir>
\end{verbatim}
where you can supply any directory you like for the \option{home}
option. If you don't supply a directory (as in the first example
above), the \command{install} command uses the \code{HOME} environment
variable (or your official home directory as supplied by the password
file, if \code{HOME} is not defined).
The \option{home} option defines the installation base directory. Files
are installed to the following directories under the installation base
as follows:
\installscheme{home}{/lib/python}
{home}{/lib/python}
{home}{/bin}
{home}{/share}
\subsection{Alternate installation: Unix (the prefix scheme)}
\label{sec:alt-unix-home}
The ``prefix scheme'' is useful when you wish to use one Python
installation to perform the build/install (i.e., to run the setup
script), but install modules into the third-party module directory of a
different Python installation (or something that looks like a different
Python installation). If this sounds a trifle unusual, it is---that's
why the ``home scheme'' comes first. However, there are at least two
known cases where the prefix scheme will be useful.
First, consider that many Linux distribution put Python in \file{/usr},
rather than the more traditional \file{/usr/local}. This is entirely
appropriate, since in those cases Python is part of ``the system''
rather than a local add-on. However, if you are installing Python
modules from source, you probably want them to go in
\file{/usr/local/lib/python1.\filevar{X}} rather than
\file{/usr/lib/python1.\filevar{X}}. This can be done with
\begin{verbatim}
/usr/bin/python setup.py install --prefix=/usr/local
\end{verbatim}
Another possibility is a network filesystem where the name used to write
to a remote directory is different from the name used to read it: for
example, the Python interpreter accessed as \file{/usr/local/bin/python}
might search for modules in \file{/usr/local/lib/python1.\filevar{X}},
but those modules would have to be installed to, say,
\file{/mnt/\filevar{@server}/export/lib/python1.\filevar{X}}. This
could be done with
\begin{verbatim}
/usr/local/bin/python setup.py install --prefix=/mnt/@server/export
\end{verbatim}
In either case, the \option{prefix} option defines the installation
base, and the \option{exec-prefix} option defines the platform-specific
installation base, which is used for platform-specific files.
(Currently, this just means non-pure module distributions, but could be
expanded to C libraries, binary executables, etc.) If
\option{exec-prefix} is not supplied, it defaults to \option{prefix}.
Files are installed as follows:
\installscheme{prefix}{/lib/python1.\filevar{X}/site-packages}
{exec-prefix}{/lib/python1.\filevar{X}/site-packages}
{prefix}{/bin}
{prefix}{/share}
There is no requirement that \option{prefix} or \option{exec-prefix}
actually point to an alternate Python installation; if the directories
listed above do not already exist, they are created at installation
time.
Incidentally, the real reason the prefix scheme is important is simply
that a standard Unix installation uses the prefix scheme, but with
\option{prefix} and \option{exec-prefix} supplied by Python itself (as
\code{sys.prefix} and \code{sys.exec\_prefix}). Thus, you might think
you'll never use the prefix scheme, but every time you run \code{python
setup.py install} without any other options, you're using it.
Note that installing extensions to an alternate Python installation has
no effect on how those extensions are built: in particular, the Python
header files (\file{Python.h} and friends) installed with the Python
interpreter used to run the setup script will be used in compiling
extensions. It is your responsibility to ensure that the interpreter
used to run extensions installed in this way is compatibile with the
interpreter used to build them. The best way to ensure this is that the
two interpreters are the same version of Python (possibly different
builds, or possibly copies of the same build). (Of course, if your
\option{prefix} and \option{exec-prefix} don't even point to an
alternate Python installation, this is immaterial.)
\subsection{Alternate installation: Windows}
\label{sec:alt-windows}
Since Windows has no conception of a user's home directory, and since
the standard Python installation under Windows is simpler than that
under Unix, there's no point in having separate \option{prefix} and
\option{home} options. Just use the \option{prefix} option to specify
a base directory, e.g.
\begin{verbatim}
python setup.py install --base="\Temp\Python"
\end{verbatim}
to install modules to the \file{\bslash{}Temp} directory on the current
drive.
The installation base is defined by the \option{prefix} option; the
\option{exec-prefix} option is not supported under Windows. Files are
installed as follows:
\installscheme{prefix}{}
{prefix}{}
{prefix}{\bslash{}Scripts}
{prefix}{\bslash{}Data}
\subsection{Alternate installation: Mac OS}
\label{sec:alt-macos}
Like Windows, Mac OS has no notion of home directories (or even of
users), and a fairly simple standard Python installation. Thus, only a
\option{prefix} option is needed. It defines the installation base, and
files are installed under it as follows:
XXX how do MacPython users run the interpreter with command-line args?
\installscheme{prefix}{:Lib}
{prefix}{:Mac:PlugIns}
{prefix}{}
{prefix}{}
XXX Corran Webster says: ``Modules are found in either \file{:Lib} or
\file{:Mac:Lib}, while extensions usually go in
\file{:Mac:PlugIns}''---does this mean that non-pure distributions should
be divided between \file{:Mac:PlugIns} and \file{:Mac:Lib}? If so, that
changes the granularity at which we care about modules: instead of
``modules from pure distributions'' and ``modules from non-pure
distributions'', it becomes ``modules from pure distributions'',
``Python modules from non-pure distributions'', and ``extensions from
non-pure distributions''. Is this necessary?!?
\section{Custom Installation}
\label{sec:custom-install}
Sometimes, the alternate installation schemes described in
section~\ref{sec:alt-install} just don't do what you want. You might
want to tweak just one or two directories while keeping everything under
the same base directory, or you might want to completely redefine the
installation scheme. In either case, you're creating a \emph{custom
installation scheme}.
You probably noticed the column of ``override options'' in the tables
describing the alternate installation schemes above. Those options are
how you define a custom installation scheme. These override options can
be relative, absolute, or explicitly defined in terms of one of the
installation base directories. (There are two installation base
directories, and they are normally the same---they only differ when you
use the Unix ``prefix scheme'' and supply different \option{prefix} and
\option{exec-prefix} options.)
For example, say you're installing a module distribution to your home
directory under Unix---but you want scripts to go in
\file{\tilde/scripts} rather than \file{\tilde/bin}. As you might
expect, you can override this directory with the
\option{install-scripts} option; in this case, it makes most sense to
supply a relative path, which will be interpreted relative to the
installation base directory (your home directory, in this case):
\begin{verbatim}
python setup.py install --home --install-scripts=scripts
\end{verbatim}
Another Unix example: suppose your Python installation was built and
installed with a prefix of \file{/usr/local/python}, so under a standard
installation scripts will wind up in \file{/usr/local/python/bin}. If
you want them in \file{/usr/local/bin} instead, you would supply this
absolute directory for the \option{install-scripts} option:
\begin{verbatim}
python setup.py install --install-scripts=/usr/local/bin
\end{verbatim}
(This performs an installation using the ``prefix scheme,'' where the
prefix is whatever your Python interpreter was installed with---
\file{/usr/local/python} in this case.)
If you maintain Python on Windows, you might want third-party modules to
live in a subdirectory of \filevar{prefix}, rather than right in
\filevar{prefix} itself. This is almost as easy as customizing the
script installation directory---you just have to remember that there are
two types of modules to worry about, pure modules and non-pure modules
(i.e., modules from a non-pure distribution). For example:
\begin{verbatim}
python setup.py install --install-purelib=Site --install-platlib=Site
\end{verbatim}
The specified installation directories are relative to \filevar{prefix}.
Of course, you also have to ensure that these directories are in
Python's module search path, e.g. by putting a \file{.pth} file in
\filevar{prefix} (XXX should have a section describing .pth files and
cross-ref it here).
If you want to define an entire installation scheme, you just have to
supply all of the installation directory options. The recommended way
to do this is to supply relative paths; for example, if want to maintain
all Python module-related files under \file{python} in your home
directory, and you want a separate directory for each platform that you
use your home directory from, you might define the following
installation scheme:
\begin{verbatim}
python setup.py install --home \
--install-purelib=python/lib \
--install-platlib=python/lib.$PLAT \
--install-scripts=python/scripts
--install-data=python/data
\end{verbatim}
or, equivalently,
\begin{verbatim}
python setup.py install --home=~/python \
--install-purelib=lib \
--install-platlib=lib.$PLAT \
--install-scripts=scripts
--install-data=data
\end{verbatim}
\code{\$PLAT} is not (necessarily) an environment variable---it will be
expanded by the Distutils as it parses your command line options (just
as it does when parsing your configuration file(s)).
Obviously, specifying the entire installation scheme every time you
install a new module distribution would be very tedious. Thus, you can
put these options into your Distutils config file (see
section~\ref{sec:config-files}):
\begin{verbatim}
[install]
install-lib=~/lib/python
install-base=$HOME
install-purelib=python/lib
install-platlib=python/lib.$PLAT
install-scripts=python/scripts
install-data=python/data
\end{verbatim}
Note that use of shell-style tilde and environment variable expansion is
supported both on the command line and in configuration files. (See
section~\ref{sec:config-files} for more information on configuration
files.)
Of course, in order for this personal Python library scheme to work, you
have to ensure that \homefile{lib/python} is in \code{sys.path} when you
run Python. The easiest way to do this under \UNIX{} is to add it to
your \code{PYTHONPATH} environment variable when you login. For
example, if you use a Bourne shell derivative such as bash, zsh, or ksh,
add the following to your \homefile{.profile} (or \homefile{.bashrc}, or
\homefile{.zshenv}, depending on your shell and personal preferences):
\begin{verbatim}
export PYTHONPATH=$HOME/lib/python
\end{verbatim}
If you use a csh-derivative such as tcsh, add the following to your
\homefile{.cshrc}:
\begin{verbatim}
setenv PYTHONPATH $HOME/lib/python
\end{verbatim}
If you use multiple platforms (architectures and/or operating systems)
from the same home directory, then you probably want to maintain a
multi-platform personal Python library. One possible scheme is to put
platform-neutral (pure Python) distributions in \homefile{lib/python}
and platform-specific distributions (any that containe extension
modules) in \homefile{lib/python.\filevar{plat}}:
\begin{verbatim}
python setup.py install --install-lib=~/lib/python \
--install-lib-plat=~/lib/python.plat \
\end{verbatim}
On the command line, of course, you can just type in the current
platform in place of \filevar{plat}: \file{linux-x86},
\file{solaris-sparc}, \file{linux-alpha}, whatever. That's not an
option in a configuration file, though---the same file has to cover all
platforms for which you maintain a personal Python library. So the
Distutils provide a \code{PLAT} environment variable which will expand
to the current platform name:
or, equivalently,
\begin{verbatim}
[install]
install-lib=~/lib/python
install-platlib=~/lib/python.$PLAT
install-base=$HOME/python
install-purelib=lib
install-platlib=lib.$PLAT
install-scripts=scripts
install-data=data
\end{verbatim}
(If \code{PLAT} is already defined in your environment, the Distutils
won't override it: that way you can maintain consistency with other
applications that look for a \code{PLAT} variable; this is especially
useful when you refer to \code{PLAT} in your login scripts, as explained
below.)
(XXX danger danger! this environment-variable-in-config-file thing is
frighteningly make-like: is there any way to avoid it?)
Again, you have to make sure that your personal Python library appears
in \code{sys.path}, and again the easiest way to do this is to set
\code{PYTHONPATH} in your login scripts. This time, though, you have to
be sure to set \emph{both} directories (platform-neutral and the current
platform-specific directory). For Bourne-shell derivatives:
Note that these two are \emph{not} equivalent if you supply a different
installation base directory when you run the setup script. For example,
\begin{verbatim}
export PYTHONPATH=$HOME/lib/python:$HOME/lib/python.$PLAT
python setup.py --install-base=/tmp
\end{verbatim}
and for csh-derivatives:
\begin{verbatim}
setenv PYTHONPATH $HOME/lib/python:$HOME/lib/python.$PLAT
\end{verbatim}
Note that it is your responsibility to set the \code{PATH} environment
variable (unless your system administrator has kindly taken care of it
in the system-wide login scripts, which is a wise thing to do on
multi-platform networks). One way to do this is with the \code{uname}
command:
\begin{verbatim}
export PLAT=`uname -sm | tr 'A-Z ' 'a-z-'`
\end{verbatim}
(XXX check that this works well on other Unices: on Linux, \code{-m}
becomes eg. \code{i586}, which is not the \emph{machine} but the
\emph{processor}. Arggh!)
would install pure modules to \filevar{/tmp/python/lib} in the first
case, and to \filevar{/tmp/lib} in the second case. (For the second
case, you probably want to supply an installation base of
\file{/tmp/python}.)
Of course, there are more reasons to do custom installation than
maintaining a personal Python library. Even if you have write access to
the system-wide directories for third-party modules
(\file{\filevar{prefix}/lib/python1.\filevar{x}/site-packages} and
\file{\filevar{exec-prefix}/lib/python1.\filevar{x}/site-packages}), you
might want to install new module distributions---especially upgrades of
modules that are crucial to your local infrastructure---to a temporary
location, in order to test them before installing them ``for real.''
This is fundamentally no different from installing to your home
directory, except that you probably won't bother to set
\code{PYTHONPATH} permanently. For example, to install a module
distribution to \file{/tmp/pylib}:
\begin{verbatim}
python setup.py install --install-lib=/tmp/pylib
\end{verbatim}
Then, of course, you'll want to run some script that depends on these
modules to make sure that they still work with your installed base of
code:
\begin{verbatim}
env PYTHONPATH=/tmp/pylib python /usr/local/bin/crucial_script ...
\end{verbatim}
You probably noticed the use of \code{\$HOME} and \code{\$PLAT} in the
sample configuration file input. These are Distutils configuration
variables, which bear a strong resemblance to environment variables. In
fact, you can use environment variables in config files, but the
Distutils additionally define a few extra variables that may not be in
your environment, such as \code{\$PATH}. See
section~\ref{sec:config-files} for details.
Of course, you can do this temporary installation with separate
\option{install-lib} and \option{install-platlib} options. If you're
doing this to a network-wide directory, not \file{/tmp}, this might be
essential. As you might have guessed, it's not too hard:
\begin{verbatim}
python setup.py install --install-lib=/scratch/pylib \
--install-platlib=/scratch/pylib.plat
\end{verbatim}
and then, testing your crucial scripts on multiple platforms:
\begin{verbatim}
env PYTHONPATH=/scratch/pylib:/scratch/pylib.plat \
python /usr/local/bin/crucial_script ...
\end{verbatim}
XXX need some Windows and Mac OS examples---when would custom
installation schemes be needed on those platforms?
However you do the testing, once you're satisfied that the new version
doesn't break anything, you can install it to the system-wide
third-party module directory as usual:
\begin{verbatim}
python setup.py install
\end{verbatim}
\subsection{Indirect specification: prefix directories}
\label{sec:prefix-dirs}
Occasionally, you may want to install a module distribution
\section{Custom Installation (Windows)}
\label{sec:custom-install-windows}
\comingsoon
\section{Configuration Files}
@ -305,6 +416,4 @@ Occasionally, you may want to install a module distribution
\comingsoon
\end{document}