mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-08-04 08:59:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 11

bpo-31993: Do not allocate large temporary buffers in pickle dump. (#4353)

Browse source 
The picklers do no longer allocate temporary memory when dumping large
bytes and str objects into a file object. Instead the data is
directly streamed into the underlying file object.

Previously the C implementation would buffer all content and issue a
single call to file.write() at the end of the dump. With protocol 4
this behavior has changed to issue one call to file.write() per frame.

The Python pickler with protocol 4 now dumps each frame content as a
memoryview to an IOBytes instance that is never reused and the
memoryview is no longer released after the call to write. This makes it
possible for the file object to delay access to the memoryview of
previous frames without forcing any additional memory copy as was
already possible with the C pickler.
This commit is contained in:
Olivier Grisel 2018-01-06 16:18:54 +01:00 • committed by Serhiy Storchaka
parent 85ac726a40
commit 3cd7c6e6eb
6 changed files with 297 additions and 50 deletions
Download patch file Download diff file Expand all files Collapse all files

3
Lib/test/test_pickletools.py
View file

@ -15,6 +15,9 @@ class OptimizedPickleTests(AbstractPickleTests):
# Test relies on precise output of dumps()
test_pickle_to_2x = None
# Test relies on writing by chunks into a file object.
test_framed_write_sizes_with_delayed_writer = None
def test_optimize_long_binget(self):
data = [str(i) for i in range(257)]
data.append(data[-1])