Lots of things added. README written (mostly).

Tools/freeze/README

@@ -0,0 +1,161 @@
+THE FREEZE SCRIPT
+=================
+
+
+What is Freeze?
+---------------
+
+Freeze make it possible to ship arbitrary Python programs to people
+who don't have Python.  The shipped file (called a "frozen" version of
+your Python program) is an executable, so this only works if your
+platform is compatible with that on the receiving end (this is usually
+a matter of having the same major operating system revision and CPU
+type).
+
+The shipped file contains a Python interpreter and large portions of
+the Python run-time.  Some measures have been taken to avoid linking
+unneeded modules, but the resulting binary is usually not small.
+
+The Python source code of your program (and of the library modules
+written in Python that it uses) is not included in the binary --
+instead, the compiled byte-code (the instruction stream used
+internally by the interpreter) is incorporated.  This gives some
+protection of your Python source code, though not much -- a
+disassembler for Python byte-code is available in the standard Python
+library.  At least someone running "strings" on your binary won't see
+the source.
+
+
+How does Freeze know which modules to include?
+----------------------------------------------
+
+Freeze uses a pretty simple-minded algorithm to find the modules that
+your program uses: given a file containing Python source code, it
+scans for lines beginning with the word "import" or "from" (possibly
+preceded by whitespace) and then it knows where to find the module
+name(s) in those lines.  It then recursively scans the source for
+those modules (if found, and not already processed) in the same way.
+
+Freeze will not see import statements hidden behind another statement,
+like this:
+
+	if some_test: import M  # M not seen
+
+or like this:
+
+	import A; import B; import C  # B and C not seen
+
+nor will it see import statements constructed using string
+operations and passed to 'exec', like this:
+
+	exec "import %s" % "M"  # M not seen
+
+On the other hand, Freeze will think you are importing a module even
+if the import statement it sees will never be executed, like this:
+
+	if 0:
+		import M  # M is seen
+
+One tricky issue: Freeze assumes that the Python interpreter and
+environment you're using to run Freeze is the same one that would be
+used to run your program, which should also be the same whose sources
+and installed files you will learn about in the next section.  In
+particular, your PYTHONPATH setting should be the same as for running
+your program locally.  (Tip: if the program doesn't run when you type
+"python hello.py" there's little chance of getting the frozen version
+to run.)
+
+
+How do I use Freeze?
+--------------------
+
+Ideally, you should be able to use it as follows:
+
+	python freeze.py hello.py
+
+where hello.py is your program and freeze.py is the main file of
+Freeze (in actuality, you'll probably specify an absolute pathname
+such as /ufs/guido/src/python/Demo/freeze/freeze.py).
+
+Unfortunately, this doesn't work.  Well, it might, but somehow it's
+extremely unlikely that it'll work on the first try.  (If it does,
+skip to the next section.)  Most likely you'll get this error message:
+
+	needed directory /usr/local/lib/python/lib not found
+
+The reason is that Freeze require that some files that are normally
+kept inside the Python build tree are installed, and it searches for
+it in the default install location.  (The default install prefix is
+/usr/local; these particular files are installed at lib/python/lib
+under the install prefix.)
+
+The particular set of files needed is installed only if you run "make
+libainstall" (note: "liba", not "lib") in the Python build tree (which
+is the tree where you build Python -- often, but not necessarily, this
+is also the Python source tree).  If you have in fact done a "make
+libainstall" but used a different prefix, all you need to do is pass
+that same prefix to Freeze with the -p option:
+
+	python freeze.py -p your-prefix hello.py
+
+(If you haven't run "make libainstall" yet, go and do it now and don't
+come back until you've done it.)
+
+
+How do I configure Freeze?
+--------------------------
+
+It's a good idea to change the line marked with XXX in freeze.py (an
+assignment to variable PACK) to point to the absolute pathname of the
+directory where Freeze lives (Demo/freeze in the Python source tree.)
+This makes it possible to call Freeze from other directories.
+
+You can also edit the assignment to variable PREFIX -- this saves a
+lot of -p options.
+
+
+How do I use Freeze with extensions modules?
+--------------------------------------------
+
+XXX to be written.  (In short: pass -e extensionbuilddir.)
+
+
+How do I use Freeze with dynamically loaded extension modules?
+--------------------------------------------------------------
+
+XXX to be written.  (In short: pass -e modulebuilddir -- this even
+works if you built the modules in Python's own Modules directory.)
+
+
+
+What do I do next?
+------------------
+
+Freeze creates three files: frozen.c, config.c and Makefile.  To
+produce the frozen version of your program, you can simply type
+"make".  This should produce a binary file.  If the filename argument
+to Freeze was "hello.py", the binary will be called "hello".  On the
+other hand, if the argument was "hello", the binary will be called
+"hello.bin".  If you passed any other filename, all bets are off. :-)
+In any case, the name of the file will be printed as the last message
+from Freeze.
+
+
+Help!  I've tried everything but it doesn't work!
+-------------------------------------------------
+
+Freeze is currently beta software.  You could email me a bug report.
+Please give as much context as possible -- "Freeze doesn't work" is
+not going to get much sympathy.  You could fix the bug and send me a
+patch.  You could learn Tcl.
+
+If you are thinking about debugging Freeze, start playing with a
+really simple program first (like "print 'hello world'").  If you
+can't get that to work there's something fundamentally wrong with your
+environment (or with your understanding of it).  Gradually build it up
+to use more modules and extensions until you find where it stops
+working.  After that, you're on your own -- happy hacking!
+
+
+--Guido van Rossum, CWI, Amsterdam <mailto:Guido.van.Rossum@cwi.nl>
+<http://www.cwi.nl/cwi/people/Guido.van.Rossum.html>

Tools/freeze/checkextensions.py

@@ -0,0 +1,87 @@
+# Check for a module in a set of extension directories.
+# An extension directory should contain a Setup file
+# and one or more .o files or a lib.a file.
+
+import os
+import string
+import parsesetup
+
+def checkextensions(unknown, extensions):
+	files = []
+	modules = []
+	edict = {}
+	for e in extensions:
+		setup = os.path.join(e, 'Setup')
+		liba = os.path.join(e, 'lib.a')
+		if not os.path.isfile(liba):
+			liba = None
+		edict[e] = parsesetup.getsetupinfo(setup), liba
+	for mod in unknown:
+		for e in extensions:
+			(mods, vars), liba = edict[e]
+			if not mods.has_key(mod):
+				continue
+			modules.append(mod)
+			if liba:
+				# If we find a lib.a, use it, ignore the
+				# .o files, and use *all* libraries for
+				# *all* modules in the Setup file
+				if liba in files:
+					break
+				files.append(liba)
+				for m in mods.keys():
+					files = files + select(e, mods, vars,
+							       m, 1)
+				break
+			files = files + select(e, mods, vars, mod, 0)
+			break
+	return files, modules
+
+def select(e, mods, vars, mod, skipofiles):
+	files = []
+	for w in mods[mod]:
+		w = treatword(w)
+		if not w:
+			continue
+		w = expandvars(w, vars)
+		if skipofiles and w[-2:] == '.o':
+			continue
+		if w[0] != '-' and w[-2:] in ('.o', '.a'):
+			w = os.path.join(e, w)
+		files.append(w)
+	return files
+
+cc_flags = ['-I', '-D', '-U']
+cc_exts = ['.c', '.C', '.cc', '.c++']
+
+def treatword(w):
+	if w[:2] in cc_flags:
+		return None
+	if w[:1] == '-':
+		return w # Assume loader flag
+	head, tail = os.path.split(w)
+	base, ext = os.path.splitext(tail)
+	if ext in cc_exts:
+		tail = base + '.o'
+		w = os.path.join(head, tail)
+	return w
+
+def expandvars(str, vars):
+	i = 0
+	while i < len(str):
+		i = k = string.find(str, '$', i)
+		if i < 0:
+			break
+		i = i+1
+		var = str[i:i+1]
+		i = i+1
+		if var == '(':
+			j = string.find(str, ')', i)
+			if j < 0:
+				break
+			var = str[i:j]
+			i = j+1
+		if vars.has_key(var):
+			str = str[:k] + vars[var] + str[i:]
+			i = k
+	return str

Tools/freeze/findmodules.py

@@ -21,6 +21,7 @@ def findmodules(scriptfile, modules = [], path = sys.path):
 	for name in modules:
 		mod = os.path.basename(name)
 		if mod[-3:] == '.py': mod = mod[:-3]
+		elif mod[-4:] == '.pyc': mod = mod[:-4]
 		todo[mod] = name
 	done = closure(todo)
 	return done
@@ -94,7 +95,6 @@ def scanfile(filename):
 # Return filename, or '<builtin>', or '<unknown>'.
 
 builtins = sys.builtin_module_names
-if 'sys' not in builtins: builtins.append('sys')
 tails = ['.py', '.pyc']
 
 def findmodule(modname, path = sys.path):

Tools/freeze/freeze.py

@@ -1,31 +1,49 @@
 #! /usr/local/bin/python
 
 # "Freeze" a Python script into a binary.
-# Usage: see first function below (before the imports!)
+# Usage: see variable usage_msg below (before the imports!)
 
 # HINTS:
-# - Edit the line at XXX below before running!
+# - Edit the lines marked XXX below to localize.
 # - You must have done "make inclinstall libainstall" in the Python
 #   build directory.
 # - The script should not use dynamically loaded modules
 #   (*.so on most systems).
 
 
-# XXX Change the following line to point to your Demo/freeze directory!
+# Usage message
-pack = '/ufs/guido/src/python/Demo/freeze'
+
+usage_msg = """
+usage: freeze [-p prefix] [-e extension] ... script [module] ...
+
+-p prefix:    This is the prefix used when you ran
+              'Make inclinstall libainstall' in the Python build directory.
+              (If you never ran this, freeze won't work.)
+               The default is /usr/local.
+
+-e extension: A directory containing additional .o files that
+              may be used to resolve modules.  This directory
+              should also have a Setup file describing the .o files.
+              More than one -e option may be given.
+
+script:       The Python script to be executed by the resulting binary.
+
+module ...:   Additional Python modules (referenced by pathname)
+              that will be included in the resulting binary.  These
+              may be .py or .pyc files.
+"""
 
 
-# Print usage message and exit
+# XXX Change the following line to point to your Demo/freeze directory
+PACK = '/ufs/guido/src/python/Demo/freeze'
 
-def usage(msg = None):
+# XXX Change the following line to point to your install prefix
-	if msg:
+PREFIX = '/usr/local'
-		sys.stderr.write(str(msg) + '\n')
-	sys.stderr.write('usage: freeze [-p prefix] script [module] ...\n')
-	sys.exit(2)
 
 
 # Import standard modules
 
+import cmp
 import getopt
 import os
 import string
@@ -38,29 +56,27 @@ import addpack
 dir = os.path.dirname(sys.argv[0])
 if dir:
 	pack = dir
+else:
+	pack = PACK
 addpack.addpack(pack)
 
 
 # Import the freeze-private modules
 
+import checkextensions
 import findmodules
 import makeconfig
 import makefreeze
 import makemakefile
 import parsesetup
 
-hint = """
-Use the '-p prefix' command line option to specify the prefix used
-when you ran 'Make inclinstall libainstall' in the Python build directory.
-(Please specify an absolute path.)
-"""
-
 
 # Main program
 
 def main():
 	# overridable context
-	prefix = '/usr/local'		# settable with -p option
+	prefix = PREFIX			# settable with -p option
+	extensions = []
 	path = sys.path
 
 	# output files
@@ -71,14 +87,14 @@ def main():
 
 	# parse command line
 	try:
-		opts, args = getopt.getopt(sys.argv[1:], 'p:')
+		opts, args = getopt.getopt(sys.argv[1:], 'e:p:')
-		if not args:
-			raise getopt.error, 'not enough arguments'
 	except getopt.error, msg:
 		usage('getopt error: ' + str(msg))
 
 	# proces option arguments
 	for o, a in opts:
+		if o == '-e':
+			extensions.append(a)
 		if o == '-p':
 			prefix = a
 
@@ -89,13 +105,13 @@ def main():
 	frozenmain_c = os.path.join(binlib, 'frozenmain.c')
 	makefile_in = os.path.join(binlib, 'Makefile')
 	defines = ['-DHAVE_CONFIG_H', '-DUSE_FROZEN', '-DNO_MAIN',
-		   '-DPTHONPATH=\\"$(PYTHONPATH)\\"']
+		   '-DPYTHONPATH=\\"$(PYTHONPATH)\\"']
 	includes = ['-I' + incldir, '-I' + binlib]
 
-	# sanity check of locations
+	# sanity check of directories and files
-	for dir in prefix, binlib, incldir:
+	for dir in [prefix, binlib, incldir] + extensions:
 		if not os.path.exists(dir):
-			usage('needed directory %s not found' % dir + hint)
+			usage('needed directory %s not found' % dir)
 		if not os.path.isdir(dir):
 			usage('%s: not a directory' % dir)
 	for file in config_c_in, makefile_in, frozenmain_c:
@@ -103,6 +119,16 @@ def main():
 			usage('needed file %s not found' % file)
 		if not os.path.isfile(file):
 			usage('%s: not a plain file' % file)
+	for dir in extensions:
+		setup = os.path.join(dir, 'Setup')
+		if not os.path.exists(setup):
+			usage('needed file %s not found' % setup)
+		if not os.path.isfile(setup):
+			usage('%s: not a plain file' % setup)
+
+	# check that enough arguments are passed
+	if not args:
+		usage('at least one filename argument required')
 
 	# check that file arguments exist
 	for arg in args:
@@ -128,30 +154,61 @@ def main():
 
 	dict = findmodules.findmodules(scriptfile, modules, path)
 
+	backup = frozen_c + '~'
+	try:
+		os.rename(frozen_c, backup)
+	except os.error:
+		backup = None
+	outfp = open(frozen_c, 'w')
+	try:
+		makefreeze.makefreeze(outfp, dict)
+	finally:
+		outfp.close()
+	if backup:
+		if cmp.cmp(backup, frozen_c):
+			sys.stderr.write('%s not changed, not written\n' %
+					 frozen_c)
+			os.rename(backup, frozen_c)
+
 	builtins = []
+	unknown = []
 	mods = dict.keys()
 	mods.sort()
 	for mod in mods:
 		if dict[mod] == '<builtin>':
 			builtins.append(mod)
 		elif dict[mod] == '<unknown>':
-			sys.stderr.write(
+			unknown.append(mod)
-				'Warning: module %s not found anywhere\n' %
-				mod)
 
-	outfp = open(frozen_c, 'w')
+	addfiles = []
-	try:
+	if unknown:
-		makefreeze.makefreeze(outfp, dict)
+		addfiles, addmods = \
-	finally:
+			  checkextensions.checkextensions(unknown, extensions)
-		outfp.close()
+		for mod in addmods:
+			unknown.remove(mod)
+		builtins = builtins + addmods
+	if unknown:
+		sys.stderr.write('Warning: unknown modules remain: %s\n' %
+				 string.join(unknown))
 
+	builtins.sort()
 	infp = open(config_c_in)
+	backup = config_c + '~'
+	try:
+		os.rename(config_c, backup)
+	except os.error:
+		backup = None
 	outfp = open(config_c, 'w')
 	try:
 		makeconfig.makeconfig(infp, outfp, builtins)
 	finally:
 		outfp.close()
 	infp.close()
+	if backup:
+		if cmp.cmp(backup, config_c):
+			sys.stderr.write('%s not changed, not written\n' %
+					 config_c)
+			os.rename(backup, config_c)
 
 	cflags = defines + includes + ['$(OPT)']
 	libs = []
@@ -166,7 +223,8 @@ def main():
 		somevars[key] = makevars[key]
 
 	somevars['CFLAGS'] = string.join(cflags) # override
-	files = ['$(OPT)', config_c, frozenmain_c] + libs + \
+	files = ['$(OPT)', config_c, frozen_c, frozenmain_c] + \
+		addfiles + libs + \
 		['$(MODLIBS)', '$(LIBS)', '$(SYSLIBS)']
 
 	outfp = open(makefile, 'w')
@@ -179,4 +237,14 @@ def main():
 
 	print 'Now run make to build the target:', target
 
+
+# Print usage message and exit
+
+def usage(msg = None):
+	if msg:
+		sys.stderr.write(str(msg) + '\n')
+	sys.stderr.write(usage_msg)
+	sys.exit(2)
+
+
 main()