mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-11-02 19:12:55 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

#7499: Review of Library FAQ by Florent Xicluna.

Browse source
This commit is contained in:
Georg Brandl 2009-12-19 17:57:51 +00:00
parent 62eaaf62ff
commit 9e4ff75325
2 changed files with 141 additions and 144 deletions
Download patch file Download diff file Expand all files Collapse all files

283
Doc/faq/library.rst
View file

@ -38,7 +38,7 @@ There are (at least) three kinds of modules in Python:
type:: type::
import sys import sys
print sys.builtin_module_names print(sys.builtin_module_names)
How do I make a Python script executable on Unix? How do I make a Python script executable on Unix?
@ -187,8 +187,11 @@ How do I get a single keypress at a time?
----------------------------------------- -----------------------------------------
For Unix variants: There are several solutions. It's straightforward to do this For Unix variants: There are several solutions. It's straightforward to do this
using curses, but curses is a fairly large module to learn. Here's a solution using curses, but curses is a fairly large module to learn.
without curses::
.. XXX this doesn't work out of the box, some IO expert needs to check why
Here's a solution without curses::
import termios, fcntl, sys, os import termios, fcntl, sys, os
fd = sys.stdin.fileno() fd = sys.stdin.fileno()
@ -202,23 +205,24 @@ without curses::
fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK) fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK)
try: try:
while 1: while True:
try: try:
c = sys.stdin.read(1) c = sys.stdin.read(1)
print "Got character", `c` print("Got character", repr(c))
except IOError: pass except IOError:
pass
finally: finally:
termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm) termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm)
fcntl.fcntl(fd, fcntl.F_SETFL, oldflags) fcntl.fcntl(fd, fcntl.F_SETFL, oldflags)
You need the :mod:`termios` and the :mod:`fcntl` module for any of this to work, You need the :mod:`termios` and the :mod:`fcntl` module for any of this to
and I've only tried it on Linux, though it should work elsewhere. In this code, work, and I've only tried it on Linux, though it should work elsewhere. In
characters are read and printed one at a time. this code, characters are read and printed one at a time.
:func:`termios.tcsetattr` turns off stdin's echoing and disables canonical mode. :func:`termios.tcsetattr` turns off stdin's echoing and disables canonical
:func:`fcntl.fnctl` is used to obtain stdin's file descriptor flags and modify mode. :func:`fcntl.fnctl` is used to obtain stdin's file descriptor flags
them for non-blocking mode. Since reading stdin when it is empty results in an and modify them for non-blocking mode. Since reading stdin when it is empty
:exc:`IOError`, this error is caught and ignored. results in an :exc:`IOError`, this error is caught and ignored.
Threads Threads
@ -247,13 +251,13 @@ all the threads to finish::
import threading, time import threading, time
def thread_task(name, n): def thread_task(name, n):
for i in range(n): print name, i for i in range(n): print(name, i)
for i in range(10): for i in range(10):
T = threading.Thread(target=thread_task, args=(str(i), i)) T = threading.Thread(target=thread_task, args=(str(i), i))
T.start() T.start()
time.sleep(10) # <----------------------------! time.sleep(10) # <---------------------------!
But now (on many platforms) the threads don't run in parallel, but appear to run But now (on many platforms) the threads don't run in parallel, but appear to run
sequentially, one at a time! The reason is that the OS thread scheduler doesn't sequentially, one at a time! The reason is that the OS thread scheduler doesn't
@ -262,8 +266,8 @@ start a new thread until the previous thread is blocked.
A simple fix is to add a tiny sleep to the start of the run function:: A simple fix is to add a tiny sleep to the start of the run function::
def thread_task(name, n): def thread_task(name, n):
time.sleep(0.001) # <---------------------! time.sleep(0.001) # <--------------------!
for i in range(n): print name, i for i in range(n): print(name, i)
for i in range(10): for i in range(10):
T = threading.Thread(target=thread_task, args=(str(i), i)) T = threading.Thread(target=thread_task, args=(str(i), i))
@ -289,28 +293,28 @@ once.
Here's a trivial example:: Here's a trivial example::
import threading, Queue, time import threading, queue, time
# The worker thread gets jobs off the queue. When the queue is empty, it # The worker thread gets jobs off the queue. When the queue is empty, it
# assumes there will be no more work and exits. # assumes there will be no more work and exits.
# (Realistically workers will run until terminated.) # (Realistically workers will run until terminated.)
def worker (): def worker ():
print 'Running worker' print('Running worker')
time.sleep(0.1) time.sleep(0.1)
while True: while True:
try: try:
arg = q.get(block=False) arg = q.get(block=False)
except Queue.Empty: except queue.Empty:
print 'Worker', threading.currentThread(), print('Worker', threading.currentThread(), end=' ')
print 'queue empty' print('queue empty')
break break
else: else:
print 'Worker', threading.currentThread(), print('Worker', threading.currentThread(), end=' ')
print 'running with argument', arg print('running with argument', arg)
time.sleep(0.5) time.sleep(0.5)
# Create queue # Create queue
q = Queue.Queue() q = queue.Queue()
# Start a pool of 5 workers # Start a pool of 5 workers
for i in range(5): for i in range(5):
@ -322,10 +326,10 @@ Here's a trivial example::
q.put(i) q.put(i)
# Give threads time to run # Give threads time to run
print 'Main thread sleeping' print('Main thread sleeping')
time.sleep(5) time.sleep(5)
When run, this will produce the following output: When run, this will produce the following output::
Running worker Running worker
Running worker Running worker
@ -333,12 +337,12 @@ When run, this will produce the following output:
Running worker Running worker
Running worker Running worker
Main thread sleeping Main thread sleeping
Worker <Thread(worker 1, started)> running with argument 0 Worker <Thread(worker 1, started 130283832797456)> running with argument 0
Worker <Thread(worker 2, started)> running with argument 1 Worker <Thread(worker 2, started 130283824404752)> running with argument 1
Worker <Thread(worker 3, started)> running with argument 2 Worker <Thread(worker 3, started 130283816012048)> running with argument 2
Worker <Thread(worker 4, started)> running with argument 3 Worker <Thread(worker 4, started 130283807619344)> running with argument 3
Worker <Thread(worker 5, started)> running with argument 4 Worker <Thread(worker 5, started 130283799226640)> running with argument 4
Worker <Thread(worker 1, started)> running with argument 5 Worker <Thread(worker 1, started 130283832797456)> running with argument 5
... ...
Consult the module's documentation for more details; the ``Queue`` class Consult the module's documentation for more details; the ``Queue`` class
@ -351,7 +355,7 @@ What kinds of global value mutation are thread-safe?
A global interpreter lock (GIL) is used internally to ensure that only one A global interpreter lock (GIL) is used internally to ensure that only one
thread runs in the Python VM at a time. In general, Python offers to switch thread runs in the Python VM at a time. In general, Python offers to switch
among threads only between bytecode instructions; how frequently it switches can among threads only between bytecode instructions; how frequently it switches can
be set via :func:`sys.setcheckinterval`. Each bytecode instruction and be set via :func:`sys.setswitchinterval`. Each bytecode instruction and
therefore all the C implementation code reached from each instruction is therefore all the C implementation code reached from each instruction is
therefore atomic from the point of view of a Python program. therefore atomic from the point of view of a Python program.
@ -443,7 +447,7 @@ How do I delete a file? (And other file questions...)
----------------------------------------------------- -----------------------------------------------------
Use ``os.remove(filename)`` or ``os.unlink(filename)``; for documentation, see Use ``os.remove(filename)`` or ``os.unlink(filename)``; for documentation, see
the :mod:`os` module. The two functions are identical; :func:`unlink` is simply the :mod:`os` module. The two functions are identical; :func:`~os.unlink` is simply
the name of the Unix system call for this function. the name of the Unix system call for this function.
To remove a directory, use :func:`os.rmdir`; use :func:`os.mkdir` to create one. To remove a directory, use :func:`os.rmdir`; use :func:`os.mkdir` to create one.
@ -505,81 +509,83 @@ to read n bytes from a pipe p created with :func:`os.popen`, you need to use
``p.read(n)``. ``p.read(n)``.
How do I run a subprocess with pipes connected to both input and output? .. XXX update to use subprocess. See the :ref:`subprocess-replacements` section.
------------------------------------------------------------------------
.. XXX update to use subprocess How do I run a subprocess with pipes connected to both input and output?
------------------------------------------------------------------------
Use the :mod:`popen2` module. For example:: Use the :mod:`popen2` module. For example::
import popen2 import popen2
fromchild, tochild = popen2.popen2("command") fromchild, tochild = popen2.popen2("command")
tochild.write("input\n") tochild.write("input\n")
tochild.flush() tochild.flush()
output = fromchild.readline() output = fromchild.readline()
Warning: in general it is unwise to do this because you can easily cause a Warning: in general it is unwise to do this because you can easily cause a
deadlock where your process is blocked waiting for output from the child while deadlock where your process is blocked waiting for output from the child
the child is blocked waiting for input from you. This can be caused because the while the child is blocked waiting for input from you. This can be caused
parent expects the child to output more text than it does, or it can be caused because the parent expects the child to output more text than it does, or it
by data being stuck in stdio buffers due to lack of flushing. The Python parent can be caused by data being stuck in stdio buffers due to lack of flushing.
can of course explicitly flush the data it sends to the child before it reads The Python parent can of course explicitly flush the data it sends to the
any output, but if the child is a naive C program it may have been written to child before it reads any output, but if the child is a naive C program it
never explicitly flush its output, even if it is interactive, since flushing is may have been written to never explicitly flush its output, even if it is
normally automatic. interactive, since flushing is normally automatic.
Note that a deadlock is also possible if you use :func:`popen3` to read stdout Note that a deadlock is also possible if you use :func:`popen3` to read
and stderr. If one of the two is too large for the internal buffer (increasing stdout and stderr. If one of the two is too large for the internal buffer
the buffer size does not help) and you ``read()`` the other one first, there is (increasing the buffer size does not help) and you ``read()`` the other one
a deadlock, too. first, there is a deadlock, too.
Note on a bug in popen2: unless your program calls ``wait()`` or ``waitpid()``, Note on a bug in popen2: unless your program calls ``wait()`` or
finished child processes are never removed, and eventually calls to popen2 will ``waitpid()``, finished child processes are never removed, and eventually
fail because of a limit on the number of child processes. Calling calls to popen2 will fail because of a limit on the number of child
:func:`os.waitpid` with the :data:`os.WNOHANG` option can prevent this; a good processes. Calling :func:`os.waitpid` with the :data:`os.WNOHANG` option can
place to insert such a call would be before calling ``popen2`` again. prevent this; a good place to insert such a call would be before calling
``popen2`` again.
In many cases, all you really need is to run some data through a command and get In many cases, all you really need is to run some data through a command and
the result back. Unless the amount of data is very large, the easiest way to do get the result back. Unless the amount of data is very large, the easiest
this is to write it to a temporary file and run the command with that temporary way to do this is to write it to a temporary file and run the command with
file as input. The standard module :mod:`tempfile` exports a ``mktemp()`` that temporary file as input. The standard module :mod:`tempfile` exports a
function to generate unique temporary file names. :: ``mktemp()`` function to generate unique temporary file names. ::
import tempfile import tempfile
import os import os
class Popen3: class Popen3:
""" """
This is a deadlock-safe version of popen that returns This is a deadlock-safe version of popen that returns
an object with errorlevel, out (a string) and err (a string). an object with errorlevel, out (a string) and err (a string).
(capturestderr may not work under windows.) (capturestderr may not work under windows.)
Example: print Popen3('grep spam','\n\nhere spam\n\n').out Example: print(Popen3('grep spam','\n\nhere spam\n\n').out)
""" """
def __init__(self,command,input=None,capturestderr=None): def __init__(self,command,input=None,capturestderr=None):
outfile=tempfile.mktemp() outfile=tempfile.mktemp()
command="( %s ) > %s" % (command,outfile) command="( %s ) > %s" % (command,outfile)
if input: if input:
infile=tempfile.mktemp() infile=tempfile.mktemp()
open(infile,"w").write(input) open(infile,"w").write(input)
command=command+" <"+infile command=command+" <"+infile
if capturestderr: if capturestderr:
errfile=tempfile.mktemp() errfile=tempfile.mktemp()
command=command+" 2>"+errfile command=command+" 2>"+errfile
self.errorlevel=os.system(command) >> 8 self.errorlevel=os.system(command) >> 8
self.out=open(outfile,"r").read() self.out=open(outfile,"r").read()
os.remove(outfile) os.remove(outfile)
if input: if input:
os.remove(infile) os.remove(infile)
if capturestderr: if capturestderr:
self.err=open(errfile,"r").read() self.err=open(errfile,"r").read()
os.remove(errfile) os.remove(errfile)
Note that many interactive programs (e.g. vi) don't work well with pipes Note that many interactive programs (e.g. vi) don't work well with pipes
substituted for standard input and output. You will have to use pseudo ttys substituted for standard input and output. You will have to use pseudo ttys
("ptys") instead of pipes. Or you can use a Python interface to Don Libes' ("ptys") instead of pipes. Or you can use a Python interface to Don Libes'
"expect" library. A Python extension that interfaces to expect is called "expy" "expect" library. A Python extension that interfaces to expect is called
and available from http://expectpy.sourceforge.net. A pure Python solution that "expy" and available from http://expectpy.sourceforge.net. A pure Python
works like expect is `pexpect <http://pypi.python.org/pypi/pexpect/>`_. solution that works like expect is `pexpect
<http://pypi.python.org/pypi/pexpect/>`_.
How do I access the serial (RS232) port? How do I access the serial (RS232) port?
@ -601,7 +607,7 @@ Python file objects are a high-level layer of abstraction on top of C streams,
which in turn are a medium-level layer of abstraction on top of (among other which in turn are a medium-level layer of abstraction on top of (among other
things) low-level C file descriptors. things) low-level C file descriptors.
For most file objects you create in Python via the builtin ``file`` constructor, For most file objects you create in Python via the builtin ``open`` constructor,
``f.close()`` marks the Python file object as being closed from Python's point ``f.close()`` marks the Python file object as being closed from Python's point
of view, and also arranges to close the underlying C stream. This also happens of view, and also arranges to close the underlying C stream. This also happens
automatically in f's destructor, when f becomes garbage. automatically in f's destructor, when f becomes garbage.
@ -645,41 +651,29 @@ How can I mimic CGI form submission (METHOD=POST)?
I would like to retrieve web pages that are the result of POSTing a form. Is I would like to retrieve web pages that are the result of POSTing a form. Is
there existing code that would let me do this easily? there existing code that would let me do this easily?
Yes. Here's a simple example that uses httplib:: Yes. Here's a simple example that uses urllib.request::
#!/usr/local/bin/python #!/usr/local/bin/python
import httplib, sys, time import urllib.request
### build the query string ### build the query string
qs = "First=Josephine&MI=Q&Last=Public" qs = "First=Josephine&MI=Q&Last=Public"
### connect and send the server a path ### connect and send the server a path
httpobj = httplib.HTTP('www.some-server.out-there', 80) req = urllib.request.urlopen('http://www.some-server.out-there'
httpobj.putrequest('POST', '/cgi-bin/some-cgi-script') '/cgi-bin/some-cgi-script', data=qs)
### now generate the rest of the HTTP headers... msg, hdrs = req.read(), req.info()
httpobj.putheader('Accept', '*/*')
httpobj.putheader('Connection', 'Keep-Alive')
httpobj.putheader('Content-type', 'application/x-www-form-urlencoded')
httpobj.putheader('Content-length', '%d' % len(qs))
httpobj.endheaders()
httpobj.send(qs)
### find out what the server said in response...
reply, msg, hdrs = httpobj.getreply()
if reply != 200:
sys.stdout.write(httpobj.getfile().read())
Note that in general for URL-encoded POST operations, query strings must be Note that in general for URL-encoded POST operations, query strings must be
quoted by using :func:`urllib.quote`. For example to send name="Guy Steele, quoted by using :func:`urllib.parse.urlencode`. For example to send name="Guy Steele,
Jr.":: Jr."::
>>> from urllib import quote >>> import urllib.parse
>>> x = quote("Guy Steele, Jr.") >>> urllib.parse.urlencode({'name': 'Guy Steele, Jr.'})
>>> x 'name=Guy+Steele%2C+Jr.'
'Guy%20Steele,%20Jr.'
>>> query_string = "name="+x .. seealso:: :ref:`urllib-howto` for extensive examples.
>>> query_string
'name=Guy%20Steele,%20Jr.'
What module should I use to help with generating HTML? What module should I use to help with generating HTML?
@ -712,9 +706,9 @@ work on any host that supports an SMTP listener. ::
import sys, smtplib import sys, smtplib
fromaddr = raw_input("From: ") fromaddr = input("From: ")
toaddrs = raw_input("To: ").split(',') toaddrs = input("To: ").split(',')
print "Enter message, end with ^D:" print("Enter message, end with ^D:")
msg = '' msg = ''
while True: while True:
line = sys.stdin.readline() line = sys.stdin.readline()
@ -732,17 +726,17 @@ varies between systems; sometimes it is ``/usr/lib/sendmail``, sometime
``/usr/sbin/sendmail``. The sendmail manual page will help you out. Here's ``/usr/sbin/sendmail``. The sendmail manual page will help you out. Here's
some sample code:: some sample code::
SENDMAIL = "/usr/sbin/sendmail" # sendmail location SENDMAIL = "/usr/sbin/sendmail" # sendmail location
import os import os
p = os.popen("%s -t -i" % SENDMAIL, "w") p = os.popen("%s -t -i" % SENDMAIL, "w")
p.write("To: receiver@example.com\n") p.write("To: receiver@example.com\n")
p.write("Subject: test\n") p.write("Subject: test\n")
p.write("\n") # blank line separating headers from body p.write("\n") # blank line separating headers from body
p.write("Some text\n") p.write("Some text\n")
p.write("some more text\n") p.write("some more text\n")
sts = p.close() sts = p.close()
if sts != 0: if sts != 0:
print "Sendmail exit status", sts print("Sendmail exit status", sts)
How do I avoid blocking in the connect() method of a socket? How do I avoid blocking in the connect() method of a socket?
@ -759,7 +753,7 @@ have to check what's returned on your system.
You can use the ``connect_ex()`` method to avoid creating an exception. It will You can use the ``connect_ex()`` method to avoid creating an exception. It will
just return the errno value. To poll, you can call ``connect_ex()`` again later just return the errno value. To poll, you can call ``connect_ex()`` again later
-- 0 or ``errno.EISCONN`` indicate that you're connected -- or you can pass this -- ``0`` or ``errno.EISCONN`` indicate that you're connected -- or you can pass this
socket to select to check if it's writable. socket to select to check if it's writable.
@ -798,22 +792,16 @@ than a third of a second. This often beats doing something more complex and
general such as using gdbm with pickle/shelve. general such as using gdbm with pickle/shelve.
Why is cPickle so slow?
-----------------------
.. XXX update this, default protocol is 2/3
The default format used by the pickle module is a slow one that results in
readable pickles. Making it the default, but it would break backward
compatibility::
largeString = 'z' * (100 * 1024)
myPickle = cPickle.dumps(largeString, protocol=1)
If my program crashes with a bsddb (or anydbm) database open, it gets corrupted. How come? If my program crashes with a bsddb (or anydbm) database open, it gets corrupted. How come?
------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------
.. XXX move this FAQ entry elsewhere?
.. note::
The bsddb module is now available as a standalone package `pybsddb
<http://www.jcea.es/programacion/pybsddb.htm>`_.
Databases opened for write access with the bsddb module (and often by the anydbm Databases opened for write access with the bsddb module (and often by the anydbm
module, since it will preferentially use bsddb) must explicitly be closed using module, since it will preferentially use bsddb) must explicitly be closed using
the ``.close()`` method of the database. The underlying library caches database the ``.close()`` method of the database. The underlying library caches database
@ -827,6 +815,13 @@ encounter an exception the next time the file is opened.
I tried to open Berkeley DB file, but bsddb produces bsddb.error: (22, 'Invalid argument'). Help! How can I restore my data? I tried to open Berkeley DB file, but bsddb produces bsddb.error: (22, 'Invalid argument'). Help! How can I restore my data?
---------------------------------------------------------------------------------------------------------------------------- ----------------------------------------------------------------------------------------------------------------------------
.. XXX move this FAQ entry elsewhere?
.. note::
The bsddb module is now available as a standalone package `pybsddb
<http://www.jcea.es/programacion/pybsddb.htm>`_.
Don't panic! Your data is probably intact. The most frequent cause for the error Don't panic! Your data is probably intact. The most frequent cause for the error
is that you tried to open an earlier Berkeley DB file with a later version of is that you tried to open an earlier Berkeley DB file with a later version of
the Berkeley DB library. the Berkeley DB library.

2
Doc/howto/urllib2.rst
View file

@ -1,3 +1,5 @@
.. _urllib-howto:
*********************************************************** ***********************************************************
HOWTO Fetch Internet Resources Using The urllib Package HOWTO Fetch Internet Resources Using The urllib Package
*********************************************************** ***********************************************************