mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-07-24 11:44:31 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

remove hotshot profiler from Py3k

Browse source
This commit is contained in:
Fred Drake 2007-10-11 18:01:43 +00:00
parent b62e8a8062
commit 0e474a801a
13 changed files with 14 additions and 2383 deletions
Download patch file Download diff file Expand all files Collapse all files

1
Doc/library/debug.rst
View file

@ -12,6 +12,5 @@ allowing you to identify bottlenecks in your programs.
bdb.rst bdb.rst
pdb.rst pdb.rst
profile.rst profile.rst
hotshot.rst
timeit.rst timeit.rst
trace.rst trace.rst

144
Doc/library/hotshot.rst
View file

@ -1,144 +0,0 @@
:mod:`hotshot` --- High performance logging profiler
====================================================
.. module:: hotshot
:synopsis: High performance logging profiler, mostly written in C.
.. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
.. sectionauthor:: Anthony Baxter <anthony@interlink.com.au>
This module provides a nicer interface to the :mod:`_hotshot` C module. Hotshot
is a replacement for the existing :mod:`profile` module. As it's written mostly
in C, it should result in a much smaller performance impact than the existing
:mod:`profile` module.
.. note::
The :mod:`hotshot` module focuses on minimizing the overhead while profiling, at
the expense of long data post-processing times. For common usages it is
recommended to use :mod:`cProfile` instead. :mod:`hotshot` is not maintained and
might be removed from the standard library in the future.
.. warning::
The :mod:`hotshot` profiler does not yet work well with threads. It is useful to
use an unthreaded script to run the profiler over the code you're interested in
measuring if at all possible.
.. class:: Profile(logfile[, lineevents[, linetimings]])
The profiler object. The argument *logfile* is the name of a log file to use for
logged profile data. The argument *lineevents* specifies whether to generate
events for every source line, or just on function call/return. It defaults to
``0`` (only log function call/return). The argument *linetimings* specifies
whether to record timing information. It defaults to ``1`` (store timing
information).
.. _hotshot-objects:
Profile Objects
---------------
Profile objects have the following methods:
.. method:: Profile.addinfo(key, value)
Add an arbitrary labelled value to the profile output.
.. method:: Profile.close()
Close the logfile and terminate the profiler.
.. method:: Profile.fileno()
Return the file descriptor of the profiler's log file.
.. method:: Profile.run(cmd)
Profile an :func:`exec`\ -compatible string in the script environment. The
globals from the :mod:`__main__` module are used as both the globals and locals
for the script.
.. method:: Profile.runcall(func, *args, **keywords)
Profile a single call of a callable. Additional positional and keyword arguments
may be passed along; the result of the call is returned, and exceptions are
allowed to propagate cleanly, while ensuring that profiling is disabled on the
way out.
.. method:: Profile.runctx(cmd, globals, locals)
Profile an :func:`exec`\ -compatible string in a specific environment. The
string is compiled before profiling begins.
.. method:: Profile.start()
Start the profiler.
.. method:: Profile.stop()
Stop the profiler.
Using hotshot data
------------------
.. module:: hotshot.stats
:synopsis: Statistical analysis for Hotshot
This module loads hotshot profiling data into the standard :mod:`pstats` Stats
objects.
.. function:: load(filename)
Load hotshot data from *filename*. Returns an instance of the
:class:`pstats.Stats` class.
.. seealso::
Module :mod:`profile`
The :mod:`profile` module's :class:`Stats` class
.. _hotshot-example:
Example Usage
-------------
Note that this example runs the python "benchmark" pystones. It can take some
time to run, and will produce large output files. ::
>>> import hotshot, hotshot.stats, test.pystone
>>> prof = hotshot.Profile("stones.prof")
>>> benchtime, stones = prof.runcall(test.pystone.pystones)
>>> prof.close()
>>> stats = hotshot.stats.load("stones.prof")
>>> stats.strip_dirs()
>>> stats.sort_stats('time', 'calls')
>>> stats.print_stats(20)
850004 function calls in 10.090 CPU seconds
Ordered by: internal time, call count
ncalls tottime percall cumtime percall filename:lineno(function)
1 3.295 3.295 10.090 10.090 pystone.py:79(Proc0)
150000 1.315 0.000 1.315 0.000 pystone.py:203(Proc7)
50000 1.313 0.000 1.463 0.000 pystone.py:229(Func2)
.
.
.

8
Doc/library/profile.rst
View file

@ -57,7 +57,7 @@ This profiler provides :dfn:`deterministic profiling` of any Python programs.
It also provides a series of report generation tools to allow users to rapidly It also provides a series of report generation tools to allow users to rapidly
examine the results of a profile operation. examine the results of a profile operation.
The Python standard library provides three different profilers: The Python standard library provides two different profilers:
#. :mod:`profile`, a pure Python module, described in the sequel. Copyright © #. :mod:`profile`, a pure Python module, described in the sequel. Copyright ©
1994, by InfoSeek Corporation. 1994, by InfoSeek Corporation.
@ -66,15 +66,11 @@ The Python standard library provides three different profilers:
it suitable for profiling long-running programs. Based on :mod:`lsprof`, it suitable for profiling long-running programs. Based on :mod:`lsprof`,
contributed by Brett Rosen and Ted Czotter. contributed by Brett Rosen and Ted Czotter.
#. :mod:`hotshot`, a C module focusing on minimizing the overhead while
profiling, at the expense of long data post-processing times.
The :mod:`profile` and :mod:`cProfile` modules export the same interface, so The :mod:`profile` and :mod:`cProfile` modules export the same interface, so
they are mostly interchangeables; :mod:`cProfile` has a much lower overhead but they are mostly interchangeables; :mod:`cProfile` has a much lower overhead but
is not so far as well-tested and might not be available on all systems. is not so far as well-tested and might not be available on all systems.
:mod:`cProfile` is really a compatibility layer on top of the internal :mod:`cProfile` is really a compatibility layer on top of the internal
:mod:`_lsprof` module. The :mod:`hotshot` module is reserved to specialized :mod:`_lsprof` module.
usages.
.. % \section{How Is This Profiler Different From The Old Profiler?} .. % \section{How Is This Profiler Different From The Old Profiler?}
.. % \nodename{Profiler Changes} .. % \nodename{Profiler Changes}

76
Lib/hotshot/__init__.py
View file

@ -1,76 +0,0 @@
"""High-perfomance logging profiler, mostly written in C."""
import _hotshot
from _hotshot import ProfilerError
class Profile:
def __init__(self, logfn, lineevents=0, linetimings=1):
self.lineevents = lineevents and 1 or 0
self.linetimings = (linetimings and lineevents) and 1 or 0
self._prof = p = _hotshot.profiler(
logfn, self.lineevents, self.linetimings)
# Attempt to avoid confusing results caused by the presence of
# Python wrappers around these functions, but only if we can
# be sure the methods have not been overridden or extended.
if self.__class__ is Profile:
self.close = p.close
self.start = p.start
self.stop = p.stop
self.addinfo = p.addinfo
def close(self):
"""Close the logfile and terminate the profiler."""
self._prof.close()
def fileno(self):
"""Return the file descriptor of the profiler's log file."""
return self._prof.fileno()
def start(self):
"""Start the profiler."""
self._prof.start()
def stop(self):
"""Stop the profiler."""
self._prof.stop()
def addinfo(self, key, value):
"""Add an arbitrary labelled value to the profile log."""
self._prof.addinfo(key, value)
# These methods offer the same interface as the profile.Profile class,
# but delegate most of the work to the C implementation underneath.
def run(self, cmd):
"""Profile an exec-compatible string in the script
environment.
The globals from the __main__ module are used as both the
globals and locals for the script.
"""
import __main__
dict = __main__.__dict__
return self.runctx(cmd, dict, dict)
def runctx(self, cmd, globals, locals):
"""Evaluate an exec-compatible string in a specific
environment.
The string is compiled before profiling begins.
"""
code = compile(cmd, "<string>", "exec")
self._prof.runcode(code, globals, locals)
return self
def runcall(self, func, *args, **kw):
"""Profile a single call of a callable.
Additional positional and keyword arguments may be passed
along; the result of the call is returned, and exceptions are
allowed to propogate cleanly, while ensuring that profiling is
disabled on the way out.
"""
return self._prof.runcall(func, args, kw)

192
Lib/hotshot/log.py
View file

@ -1,192 +0,0 @@
import _hotshot
import os.path
import parser
import symbol
import sys
from _hotshot import \
WHAT_ENTER, \
WHAT_EXIT, \
WHAT_LINENO, \
WHAT_DEFINE_FILE, \
WHAT_DEFINE_FUNC, \
WHAT_ADD_INFO
__all__ = ["LogReader", "ENTER", "EXIT", "LINE"]
ENTER = WHAT_ENTER
EXIT = WHAT_EXIT
LINE = WHAT_LINENO
class LogReader:
def __init__(self, logfn):
# fileno -> filename
self._filemap = {}
# (fileno, lineno) -> filename, funcname
self._funcmap = {}
self._reader = _hotshot.logreader(logfn)
self._nextitem = self._reader.__next__
self._info = self._reader.info
if 'current-directory' in self._info:
self.cwd = self._info['current-directory']
else:
self.cwd = None
# This mirrors the call stack of the profiled code as the log
# is read back in. It contains tuples of the form:
#
# (file name, line number of function def, function name)
#
self._stack = []
self._append = self._stack.append
self._pop = self._stack.pop
def close(self):
self._reader.close()
def fileno(self):
"""Return the file descriptor of the log reader's log file."""
return self._reader.fileno()
def addinfo(self, key, value):
"""This method is called for each additional ADD_INFO record.
This can be overridden by applications that want to receive
these events. The default implementation does not need to be
called by alternate implementations.
The initial set of ADD_INFO records do not pass through this
mechanism; this is only needed to receive notification when
new values are added. Subclasses can inspect self._info after
calling LogReader.__init__().
"""
pass
def get_filename(self, fileno):
try:
return self._filemap[fileno]
except KeyError:
raise ValueError("unknown fileno")
def get_filenames(self):
return self._filemap.values()
def get_fileno(self, filename):
filename = os.path.normcase(os.path.normpath(filename))
for fileno, name in self._filemap.items():
if name == filename:
return fileno
raise ValueError("unknown filename")
def get_funcname(self, fileno, lineno):
try:
return self._funcmap[(fileno, lineno)]
except KeyError:
raise ValueError("unknown function location")
# Iteration support:
# This adds an optional (& ignored) parameter to next() so that the
# same bound method can be used as the __getitem__() method -- this
# avoids using an additional method call which kills the performance.
def __next__(self, index=0):
while 1:
# This call may raise StopIteration:
what, tdelta, fileno, lineno = self._nextitem()
# handle the most common cases first
if what == WHAT_ENTER:
filename, funcname = self._decode_location(fileno, lineno)
t = (filename, lineno, funcname)
self._append(t)
return what, t, tdelta
if what == WHAT_EXIT:
return what, self._pop(), tdelta
if what == WHAT_LINENO:
filename, firstlineno, funcname = self._stack[-1]
return what, (filename, lineno, funcname), tdelta
if what == WHAT_DEFINE_FILE:
filename = os.path.normcase(os.path.normpath(tdelta))
self._filemap[fileno] = filename
elif what == WHAT_DEFINE_FUNC:
filename = self._filemap[fileno]
self._funcmap[(fileno, lineno)] = (filename, tdelta)
elif what == WHAT_ADD_INFO:
# value already loaded into self.info; call the
# overridable addinfo() handler so higher-level code
# can pick up the new value
if tdelta == 'current-directory':
self.cwd = lineno
self.addinfo(tdelta, lineno)
else:
raise ValueError("unknown event type")
def __iter__(self):
return self
#
# helpers
#
def _decode_location(self, fileno, lineno):
try:
return self._funcmap[(fileno, lineno)]
except KeyError:
#
# This should only be needed when the log file does not
# contain all the DEFINE_FUNC records needed to allow the
# function name to be retrieved from the log file.
#
if self._loadfile(fileno):
filename = funcname = None
try:
filename, funcname = self._funcmap[(fileno, lineno)]
except KeyError:
filename = self._filemap.get(fileno)
funcname = None
self._funcmap[(fileno, lineno)] = (filename, funcname)
return filename, funcname
def _loadfile(self, fileno):
try:
filename = self._filemap[fileno]
except KeyError:
print("Could not identify fileId", fileno)
return 1
if filename is None:
return 1
absname = os.path.normcase(os.path.join(self.cwd, filename))
try:
fp = open(absname)
except IOError:
return
st = parser.suite(fp.read())
fp.close()
# Scan the tree looking for def and lambda nodes, filling in
# self._funcmap with all the available information.
funcdef = symbol.funcdef
lambdef = symbol.lambdef
stack = [st.totuple(1)]
while stack:
tree = stack.pop()
try:
sym = tree[0]
except (IndexError, TypeError):
continue
if sym == funcdef:
self._funcmap[(fileno, tree[2][2])] = filename, tree[2][1]
elif sym == lambdef:
self._funcmap[(fileno, tree[1][2])] = filename, "<lambda>"
stack.extend(list(tree[1:]))

93
Lib/hotshot/stats.py
View file

@ -1,93 +0,0 @@
"""Statistics analyzer for HotShot."""
import profile
import pstats
import hotshot.log
from hotshot.log import ENTER, EXIT
def load(filename):
return StatsLoader(filename).load()
class StatsLoader:
def __init__(self, logfn):
self._logfn = logfn
self._code = {}
self._stack = []
self.pop_frame = self._stack.pop
def load(self):
# The timer selected by the profiler should never be used, so make
# sure it doesn't work:
p = Profile()
p.get_time = _brokentimer
log = hotshot.log.LogReader(self._logfn)
taccum = 0
for event in log:
what, (filename, lineno, funcname), tdelta = event
if tdelta > 0:
taccum += tdelta
# We multiply taccum to convert from the microseconds we
# have to the seconds that the profile/pstats module work
# with; this allows the numbers to have some basis in
# reality (ignoring calibration issues for now).
if what == ENTER:
frame = self.new_frame(filename, lineno, funcname)
p.trace_dispatch_call(frame, taccum * .000001)
taccum = 0
elif what == EXIT:
frame = self.pop_frame()
p.trace_dispatch_return(frame, taccum * .000001)
taccum = 0
# no further work for line events
assert not self._stack
return pstats.Stats(p)
def new_frame(self, *args):
# args must be filename, firstlineno, funcname
# our code objects are cached since we don't need to create
# new ones every time
try:
code = self._code[args]
except KeyError:
code = FakeCode(*args)
self._code[args] = code
# frame objects are create fresh, since the back pointer will
# vary considerably
if self._stack:
back = self._stack[-1]
else:
back = None
frame = FakeFrame(code, back)
self._stack.append(frame)
return frame
class Profile(profile.Profile):
def simulate_cmd_complete(self):
pass
class FakeCode:
def __init__(self, filename, firstlineno, funcname):
self.co_filename = filename
self.co_firstlineno = firstlineno
self.co_name = self.__name__ = funcname
class FakeFrame:
def __init__(self, code, back):
self.f_back = back
self.f_code = code
def _brokentimer():
raise RuntimeError("this timer should not be called")

31
Lib/hotshot/stones.py
View file

@ -1,31 +0,0 @@
import errno
import hotshot
import hotshot.stats
import os
import sys
import test.pystone
def main(logfile):
p = hotshot.Profile(logfile)
benchtime, stones = p.runcall(test.pystone.pystones)
p.close()
print("Pystone(%s) time for %d passes = %g" % \
(test.pystone.__version__, test.pystone.LOOPS, benchtime))
print("This machine benchmarks at %g pystones/second" % stones)
stats = hotshot.stats.load(logfile)
stats.strip_dirs()
stats.sort_stats('time', 'calls')
try:
stats.print_stats(20)
except IOError as e:
if e.errno != errno.EPIPE:
raise
if __name__ == '__main__':
if sys.argv[1:]:
main(sys.argv[1])
else:
import tempfile
main(tempfile.NamedTemporaryFile().name)

132
Lib/test/test_hotshot.py
View file

@ -1,132 +0,0 @@
import hotshot
import hotshot.log
import os
import pprint
import unittest
from test import test_support
from hotshot.log import ENTER, EXIT, LINE
def shortfilename(fn):
# We use a really shortened filename since an exact match is made,
# and the source may be either a Python source file or a
# pre-compiled bytecode file.
if fn:
return os.path.splitext(os.path.basename(fn))[0]
else:
return fn
class UnlinkingLogReader(hotshot.log.LogReader):
"""Extend the LogReader so the log file is unlinked when we're
done with it."""
def __init__(self, logfn):
self.__logfn = logfn
hotshot.log.LogReader.__init__(self, logfn)
def next(self, index=None):
try:
return hotshot.log.LogReader.next(self)
except StopIteration:
self.close()
os.unlink(self.__logfn)
raise
class HotShotTestCase(unittest.TestCase):
def new_profiler(self, lineevents=0, linetimings=1):
self.logfn = test_support.TESTFN
return hotshot.Profile(self.logfn, lineevents, linetimings)
def get_logreader(self):
return UnlinkingLogReader(self.logfn)
def get_events_wotime(self):
L = []
for event in self.get_logreader():
what, (filename, lineno, funcname), tdelta = event
L.append((what, (shortfilename(filename), lineno, funcname)))
return L
def check_events(self, expected):
events = self.get_events_wotime()
if events != expected:
self.fail(
"events did not match expectation; got:\n%s\nexpected:\n%s"
% (pprint.pformat(events), pprint.pformat(expected)))
def run_test(self, callable, events, profiler=None):
if profiler is None:
profiler = self.new_profiler()
self.failUnless(not profiler._prof.closed)
profiler.runcall(callable)
self.failUnless(not profiler._prof.closed)
profiler.close()
self.failUnless(profiler._prof.closed)
self.check_events(events)
def test_addinfo(self):
def f(p):
p.addinfo("test-key", "test-value")
profiler = self.new_profiler()
profiler.runcall(f, profiler)
profiler.close()
log = self.get_logreader()
info = log._info
list(log)
self.assertEqual(info["test-key"], ["test-value"])
def test_line_numbers(self):
def f():
y = 2
x = 1
def g():
f()
f_lineno = f.__code__.co_firstlineno
g_lineno = g.__code__.co_firstlineno
events = [(ENTER, ("test_hotshot", g_lineno, "g")),
(LINE, ("test_hotshot", g_lineno+1, "g")),
(ENTER, ("test_hotshot", f_lineno, "f")),
(LINE, ("test_hotshot", f_lineno+1, "f")),
(LINE, ("test_hotshot", f_lineno+2, "f")),
(EXIT, ("test_hotshot", f_lineno, "f")),
(EXIT, ("test_hotshot", g_lineno, "g")),
]
self.run_test(g, events, self.new_profiler(lineevents=1))
def test_start_stop(self):
# Make sure we don't return NULL in the start() and stop()
# methods when there isn't an error. Bug in 2.2 noted by
# Anthony Baxter.
profiler = self.new_profiler()
profiler.start()
profiler.stop()
profiler.close()
os.unlink(self.logfn)
def test_bad_sys_path(self):
import sys
import os
orig_path = sys.path
coverage = hotshot._hotshot.coverage
try:
# verify we require a list for sys.path
sys.path = 'abc'
self.assertRaises(RuntimeError, coverage, test_support.TESTFN)
# verify that we require sys.path exists
del sys.path
self.assertRaises(RuntimeError, coverage, test_support.TESTFN)
finally:
sys.path = orig_path
if os.path.exists(test_support.TESTFN):
os.remove(test_support.TESTFN)
def test_main():
test_support.run_unittest(HotShotTestCase)
if __name__ == "__main__":
test_main()

2
Makefile.pre.in
View file

@ -737,7 +737,7 @@ PLATMACDIRS= plat-mac plat-mac/Carbon plat-mac/lib-scriptpackages \
PLATMACPATH=:plat-mac:plat-mac/lib-scriptpackages PLATMACPATH=:plat-mac:plat-mac/lib-scriptpackages
LIBSUBDIRS= lib-tk site-packages test test/output test/data \ LIBSUBDIRS= lib-tk site-packages test test/output test/data \
test/decimaltestdata \ test/decimaltestdata \
encodings hotshot \ encodings \
email email/mime email/test email/test/data \ email email/mime email/test email/test/data \
sqlite3 sqlite3/test \ sqlite3 sqlite3/test \
logging bsddb bsddb/test csv wsgiref \ logging bsddb bsddb/test csv wsgiref \

10
Misc/NEWS
View file

@ -4,6 +4,16 @@ Python News
(editors: check NEWS.help for information about editing NEWS using ReST.) (editors: check NEWS.help for information about editing NEWS using ReST.)
What's New in Python 3.0a2?
*Unreleased*
Extension Modules
-----------------
- The `hotshot` profiler has been removed; use `cProfile` instead.
What's New in Python 3.0a1? What's New in Python 3.0a1?
========================== ==========================

1645
Modules/_hotshot.c
View file

File diff suppressed because it is too large Load diff

60
Tools/scripts/hotshotmain.py
View file

@ -1,60 +0,0 @@
#!/usr/bin/env python
# -*- coding: iso-8859-1 -*-
"""
Run a Python script under hotshot's control.
Adapted from a posting on python-dev by Walter Dörwald
usage %prog [ %prog args ] filename [ filename args ]
Any arguments after the filename are used as sys.argv for the filename.
"""
import sys
import optparse
import os
import hotshot
import hotshot.stats
PROFILE = "hotshot.prof"
def run_hotshot(filename, profile, args):
prof = hotshot.Profile(profile)
sys.path.insert(0, os.path.dirname(filename))
sys.argv = [filename] + args
fp = open(filename)
try:
script = fp.read()
finally:
fp.close()
prof.run("exec(%r)" % script)
prof.close()
stats = hotshot.stats.load(profile)
stats.sort_stats("time", "calls")
# print_stats uses unadorned print statements, so the only way
# to force output to stderr is to reassign sys.stdout temporarily
save_stdout = sys.stdout
sys.stdout = sys.stderr
stats.print_stats()
sys.stdout = save_stdout
return 0
def main(args):
parser = optparse.OptionParser(__doc__)
parser.disable_interspersed_args()
parser.add_option("-p", "--profile", action="store", default=PROFILE,
dest="profile", help='Specify profile file to use')
(options, args) = parser.parse_args(args)
if len(args) == 0:
parser.print_help("missing script to execute")
return 1
filename = args[0]
return run_hotshot(filename, options.profile, args[1:])
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))

3
setup.py
View file

@ -413,8 +413,7 @@ class PyBuildExt(build_ext):
exts.append( Extension("atexit", ["atexitmodule.c"]) ) exts.append( Extension("atexit", ["atexitmodule.c"]) )
# Python C API test module # Python C API test module
exts.append( Extension('_testcapi', ['_testcapimodule.c']) ) exts.append( Extension('_testcapi', ['_testcapimodule.c']) )
# profilers (_lsprof is for cProfile.py) # profiler (_lsprof is for cProfile.py)
exts.append( Extension('_hotshot', ['_hotshot.c']) )
exts.append( Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']) ) exts.append( Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']) )
# static Unicode character database # static Unicode character database
exts.append( Extension('unicodedata', ['unicodedata.c']) ) exts.append( Extension('unicodedata', ['unicodedata.c']) )