mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-08-10 20:08:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 11

[3.12] gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig (GH-112615) (#112724)

Browse source 
gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig (GH-112615)

Work around a macOS bug, limit zlib crc32 calls to 1GiB.

Without this, `zlib.crc32` and `binascii.crc32` could produce incorrect
results on multi-gigabyte inputs depending on the macOS version's Apple
supplied zlib implementation.
(cherry picked from commit 4eddb4c9d9)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
This commit is contained in:
Miss Islington (bot) 2023-12-04 21:29:23 +01:00 • committed by GitHub
parent 8d21242bd1
commit 88ec2a4fef
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3 changed files with 30 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

4
Misc/NEWS.d/next/Core and Builtins/2023-12-01-19-02-21.gh-issue-105967.Puq5Cn.rst Normal file
View file

@ -0,0 +1,4 @@
Workaround a bug in Apple's macOS platform zlib library where
:func:`zlib.crc32` and :func:`binascii.crc32` could produce incorrect results
on multi-gigabyte inputs. Including when using :mod:`zipfile` on zips
containing large data.

18
Modules/binascii.c
View file

@ -780,12 +780,20 @@ binascii_crc32_impl(PyObject *module, Py_buffer *data, unsigned int crc)
Py_BEGIN_ALLOW_THREADS
/* Avoid truncation of length for very large buffers. crc32() takes
length as an unsigned int, which may be narrower than Py_ssize_t. */
while ((size_t)len > UINT_MAX) {
crc = crc32(crc, buf, UINT_MAX);
buf += (size_t) UINT_MAX;
len -= (size_t) UINT_MAX;
length as an unsigned int, which may be narrower than Py_ssize_t.
We further limit size due to bugs in Apple's macOS zlib.
See https://github.com/python/cpython/issues/105967
*/
#define ZLIB_CRC_CHUNK_SIZE 0x40000000
#if ZLIB_CRC_CHUNK_SIZE > INT_MAX
# error "unsupported less than 32-bit platform?"
#endif
while ((size_t)len > ZLIB_CRC_CHUNK_SIZE) {
crc = crc32(crc, buf, ZLIB_CRC_CHUNK_SIZE);
buf += (size_t) ZLIB_CRC_CHUNK_SIZE;
len -= (size_t) ZLIB_CRC_CHUNK_SIZE;
}
#undef ZLIB_CRC_CHUNK_SIZE
crc = crc32(crc, buf, (unsigned int)len);
Py_END_ALLOW_THREADS
} else {

18
Modules/zlibmodule.c
View file

@ -1890,12 +1890,20 @@ zlib_crc32_impl(PyObject *module, Py_buffer *data, unsigned int value)
Py_BEGIN_ALLOW_THREADS
/* Avoid truncation of length for very large buffers. crc32() takes
length as an unsigned int, which may be narrower than Py_ssize_t. */
while ((size_t)len > UINT_MAX) {
value = crc32(value, buf, UINT_MAX);
buf += (size_t) UINT_MAX;
len -= (size_t) UINT_MAX;
length as an unsigned int, which may be narrower than Py_ssize_t.
We further limit size due to bugs in Apple's macOS zlib.
See https://github.com/python/cpython/issues/105967.
*/
#define ZLIB_CRC_CHUNK_SIZE 0x40000000
#if ZLIB_CRC_CHUNK_SIZE > INT_MAX
# error "unsupported less than 32-bit platform?"
#endif
while ((size_t)len > ZLIB_CRC_CHUNK_SIZE) {
value = crc32(value, buf, ZLIB_CRC_CHUNK_SIZE);
buf += (size_t) ZLIB_CRC_CHUNK_SIZE;
len -= (size_t) ZLIB_CRC_CHUNK_SIZE;
}
#undef ZLIB_CRC_CHUNK_SIZE
value = crc32(value, buf, (unsigned int)len);
Py_END_ALLOW_THREADS
} else {