mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-08-04 17:08:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 13

bpo-31945: Configurable blocksize in HTTP(S)Connection (#4279)

Browse source 
blocksize was hardcoded to 8192, preventing efficient upload when using
file-like body. Add blocksize argument to __init__, so users can
configure the blocksize to fit their needs.

I tested this uploading data from /dev/zero to a web server dropping the
received data, to test the overhead of the HTTPConnection.send() with a
file-like object.

Here is an example 10g upload with the default buffer size (8192):

$ time ~/src/cpython/release/python upload-httplib.py 10 https://localhost:8000/
Uploaded 10.00g in 17.53 seconds (584.00m/s)

real	0m17.574s
user	0m8.887s
sys	0m5.971s

Same with 512k blocksize:

$ time ~/src/cpython/release/python upload-httplib.py 10 https://localhost:8000/
Uploaded 10.00g in 6.60 seconds (1551.15m/s)

real	0m6.641s
user	0m3.426s
sys	0m2.162s

In real world usage the difference will be smaller, depending on the
local and remote storage and the network.

See https://github.com/nirs/http-bench for more info.
This commit is contained in:
Nir Soffer 2017-11-06 23:16:37 +02:00 • committed by Victor Stinner
parent 30f4fa456e
commit ad455cd924
5 changed files with 56 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

18
Doc/library/http.client.rst
View file

@ -31,7 +31,8 @@ HTTPS protocols. It is normally not used directly --- the module
The module provides the following classes:
.. class:: HTTPConnection(host, port=None[, timeout], source_address=None)
.. class:: HTTPConnection(host, port=None[, timeout], source_address=None, \
blocksize=8192)
An :class:`HTTPConnection` instance represents one transaction with an HTTP
server. It should be instantiated passing it a host and optional port
@ -42,6 +43,8 @@ The module provides the following classes:
(if it is not given, the global default timeout setting is used).
The optional *source_address* parameter may be a tuple of a (host, port)
to use as the source address the HTTP connection is made from.
The optional *blocksize* parameter sets the buffer size in bytes for
sending a file-like message body.
For example, the following calls all create instances that connect to the server
at the same host and port::
@ -58,11 +61,14 @@ The module provides the following classes:
The *strict* parameter was removed. HTTP 0.9-style "Simple Responses" are
not longer supported.
.. versionchanged:: 3.7
*blocksize* parameter was added.
.. class:: HTTPSConnection(host, port=None, key_file=None, \
cert_file=None[, timeout], \
source_address=None, *, context=None, \
check_hostname=None)
check_hostname=None, blocksize=8192)
A subclass of :class:`HTTPConnection` that uses SSL for communication with
secure servers. Default port is ``443``. If *context* is specified, it
@ -338,6 +344,14 @@ HTTPConnection Objects
Close the connection to the server.
.. attribute:: HTTPConnection.blocksize
Buffer size in bytes for sending a file-like message body.
.. versionadded:: 3.7
As an alternative to using the :meth:`request` method described above, you can
also send your request step by step, by using the four functions below.