closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)

Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.
This commit is contained in:
Benjamin Peterson 2017-12-09 10:26:52 -08:00 committed by GitHub
parent 28d8d14013
commit 42aa93b8ff
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
33 changed files with 3364 additions and 2505 deletions

View file

@ -210,6 +210,20 @@ Miscellaneous options
import of source modules. See also :envvar:`PYTHONDONTWRITEBYTECODE`.
.. cmdoption:: --check-hash-based-pycs default|always|never
Control the validation behavior of hash-based ``.pyc`` files. See
:ref:`pyc-invalidation`. When set to ``default``, checked and unchecked
hash-based bytecode cache files are validated according to their default
semantics. When set to ``always``, all hash-based ``.pyc`` files, whether
checked or unchecked, are validated against their corresponding source
file. When set to ``never``, hash-based ``.pyc`` files are not validated
against their corresponding source files.
The semantics of timestamp-based ``.pyc`` files are unaffected by this
option.
.. cmdoption:: -d
Turn on parser debugging output (for expert only, depending on compilation