This adds two new methods to `multiprocessing`'s `ProcessPoolExecutor`:
- **`terminate_workers()`**: forcefully terminates worker processes using `Process.terminate()`
- **`kill_workers()`**: forcefully kills worker processes using `Process.kill()`
These methods provide users with a direct way to stop worker processes without `shutdown()` or relying on implementation details, addressing situations where immediate termination is needed.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Sam Gross @colesbury
Commit-message-mostly-authored-by: Claude Sonnet 3.7 (because why not -greg)
Add support for generating UUIDv7 objects according to RFC 9562, §5.7 [1].
The functionality is provided by the `uuid.uuid7()` function. The implementation
is based on a 42-bit counter as described by Method 1, §6.2 [2] and guarantees
monotonicity within the same millisecond.
[1]: https://www.rfc-editor.org/rfc/rfc9562.html#section-5.7
[2]: https://www.rfc-editor.org/rfc/rfc9562.html#section-6.2
---------
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Éric <merwok@netwok.org>
This adds two new methods to `multiprocessing`'s `ProcessPoolExecutor`:
- **`terminate_workers()`**: forcefully terminates worker processes using `Process.terminate()`
- **`kill_workers()`**: forcefully kills worker processes using `Process.kill()`
These methods provide users with a direct way to stop worker processes without `shutdown()` or relying on implementation details, addressing situations where immediate termination is needed.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Commit-message-mostly-authored-by: Claude Sonnet 3.7 (because why not -greg)
Add support for generating UUIDv6 objects according to RFC 9562, §5.6 [1].
The functionality is provided by the `uuid.uuid6()` function which takes as inputs an optional 48-bit
hardware address and an optional 14-bit clock sequence. The UUIDv6 temporal fields are ordered
differently than those of UUIDv1, thereby providing improved database locality.
[1]: https://www.rfc-editor.org/rfc/rfc9562.html#section-5.6
---------
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
This broke tests on the 'aarch64 Fedora Stable Clang Installed 3.x' and
'AMD64 Fedora Stable Clang Installed 3.x' build bots.
This reverts commit da4899b94a.
## Filtered recursive walk
Expanding a recursive `**` segment entails walking the entire directory
tree, and so any subsequent pattern segments (except special segments) can
be evaluated by filtering the expanded paths through a regex. For example,
`glob.glob("foo/**/*.py", recursive=True)` recursively walks `foo/` with
`os.scandir()`, and then filters paths through a regex based on "`**/*.py`,
with no further filesystem access needed.
This fixes an issue where `glob()` could return duplicate results.
## Tracking path existence
We store a flag alongside each path indicating whether the path is
guaranteed to exist. As we process the pattern:
- Certain special pattern segments (`""`, `"."` and `".."`) leave the flag
unchanged
- Literal pattern segments (e.g. `foo/bar`) set the flag to false
- Wildcard pattern segments (e.g. `*/*.py`) set the flag to true (because
children are found via `os.scandir()`)
- Recursive pattern segments (e.g. `**`) leave the flag unchanged for the
root path, and set it to true for descendants discovered via
`os.scandir()`.
If the flag is false at the end, we call `lstat()` on each path to filter
out missing paths.
## Minor speed-ups
- Exclude paths that don't match a non-terminal non-recursive wildcard
pattern _prior_ to calling `is_dir()`.
- Use a stack rather than recursion to implement recursive wildcards.
- This fixes a recursion error when globbing deep trees.
- Pre-compile regular expressions and pre-join literal pattern segments.
- Convert to/from `bytes` (a minor use-case) in `iglob()` rather than
supporting `bytes` throughout. This particularly simplifies the code
needed to handle relative bytes paths with `dir_fd`.
- Avoid calling `os.path.join()`; instead we keep paths in a normalized
form and append trailing slashes when needed.
- Avoid calling `os.path.normcase()`; instead we use case-insensitive regex
matching.
## Implementation notes
Much of this functionality is already present in pathlib's implementation
of globbing. The specific additions we make are:
1. Support for `dir_fd`
2. Support for `include_hidden`
3. Support for generating paths relative to `root_dir`
This unifies the implementations of globbing in the `glob` and `pathlib`
modules.
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
The SyncManager provided support for various data structures such as dict, list, and queue, but oddly, not set.
This introduces support for set by defining SetProxy and registering it with SyncManager.
---
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Deprecate private C API functions:
* _PyUnicodeWriter_Init()
* _PyUnicodeWriter_Finish()
* _PyUnicodeWriter_Dealloc()
* _PyUnicodeWriter_WriteChar()
* _PyUnicodeWriter_WriteStr()
* _PyUnicodeWriter_WriteSubstring()
* _PyUnicodeWriter_WriteASCIIString()
* _PyUnicodeWriter_WriteLatin1String()
These functions are not deprecated in the internal C API (if the
Py_BUILD_CORE macro is defined).
Add function to list the currently loaded libraries to ctypes.util
The dllist() function calls platform-specific APIs in order to
list the runtime libraries loaded by Python and any imported modules.
On unsupported platforms the function may be missing.
Co-authored-by: Eryk Sun <eryksun@gmail.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Add `pathlib.Path.info` attribute, which stores an object implementing the `pathlib.types.PathInfo` protocol (also new). The object supports querying the file type and internally caching `os.stat()` results. Path objects generated by `Path.iterdir()` are initialised with status information from `os.DirEntry` objects, which is gleaned from scanning the parent directory.
The `PathInfo` protocol has four methods: `exists()`, `is_dir()`, `is_file()` and `is_symlink()`.
* gh-55454: Add IMAP4 IDLE support to imaplib
This extends imaplib with support for the rfc2177 IMAP IDLE command,
as requested in #55454. It allows events to be pushed to a client as
they occur, rather than having to continually poll for mailbox changes.
The interface is a new idle() method, which returns an iterable context
manager. Entering the context starts IDLE mode, during which events
(untagged responses) can be retrieved using the iteration protocol.
Exiting the context sends DONE to the server, ending IDLE mode.
An optional time limit for the IDLE session is supported, for use with
servers that impose an inactivity timeout.
The context manager also offers a burst() method, designed for programs
wishing to process events in batch rather than one at a time.
Notable differences from other implementations:
- It's an extension to imaplib, rather than a replacement.
- It doesn't introduce additional threads.
- It doesn't impose new requirements on the use of imaplib's existing methods.
- It passes the unit tests in CPython's test/test_imaplib.py module
(and adds new ones).
- It works on Windows, Linux, and other unix-like systems.
- It makes IDLE available on all of imaplib's client variants
(including IMAP4_stream).
- The interface is pythonic and easy to use.
Caveats:
- Due to a Windows limitation, the special case of IMAP4_stream running
on Windows lacks a duration/timeout feature. (This is the stdin/stdout
pipe connection variant; timeouts work fine for socket-based
connections, even on Windows.) I have documented it where appropriate.
- The file-like imaplib instance attributes are changed from buffered to
unbuffered mode. This could potentially break any client code that
uses those objects directly without expecting partial reads/writes.
However, these attributes are undocumented. As such, I think (and
PEP 8 confirms) that they are fair game for changes.
https://peps.python.org/pep-0008/#public-and-internal-interfaces
Usage examples:
https://github.com/python/cpython/issues/55454#issuecomment-2227543041
Original discussion:
https://discuss.python.org/t/gauging-interest-in-my-imap4-idle-implementation-for-imaplib/59272
Earlier requests and suggestions:
https://github.com/python/cpython/issues/55454https://mail.python.org/archives/list/python-ideas@python.org/thread/C4TVEYL5IBESQQPPS5GBR7WFBXCLQMZ2/
* gh-55454: Clarify imaplib idle() docs
- Add example idle response tuples, to make the minor difference from other
imaplib response tuples more obvious.
- Merge the idle context manager's burst() method docs with the IMAP
object's idle() method docs, for easier understanding.
- Upgrade the Windows note regarding lack of pipe timeouts to a warning.
- Rephrase various things for clarity.
* docs: words instead of <=
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* docs: improve style in an example
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* docs: grammatical edit
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* docs consistency
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* comment -> docstring
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* docs: refer to imaplib as "this module"
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* imaplib: simplify & clarify idle debug message
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* imaplib: elaborate in idle context manager comment
* imaplib: re-raise BaseException instead of bare except
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* imaplib: convert private doc string to comment
* docs: correct mistake in imaplib example
This is a correction to 8077f2eab2, which
changed a variable name in only one place and broke the subsequent
reference to it, departed from the naming convention used in the rest of
the module, and shadowed the type() builtin along the way.
* imaplib: simplify example code in doc string
This is for consistency with the documentation change in 8077f2eab2
and subsequent correction in 013bbf18fc.
* imaplib: rename _Idler to Idler, update its docs
* imaplib: add comment in Idler._pop()
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* imaplib: remove unnecessary blank line
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* imaplib: comment on use of unbuffered pipes
* docs: imaplib: use the reStructuredText :class: role
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Revert "docs: imaplib: use the reStructuredText :class: role"
This reverts commit f385e441df, because it
triggers CI failures in the docs by referencing a class that is
(deliberately) undocumented.
* docs: imaplib: use the reST :class: role, escaped
This is a different approach to f385e441df, which was reverted for
creating dangling link references.
By prefixing the reStructuredText role target with a ! we disable
conversion to a link, thereby passing continuous integration checks
even though the referenced class is deliberately absent from the
documentation.
* docs: refer to IMAP4 IDLE instead of just IDLE
This clarifies that we are referring to the email protocol, not the editor with the same name.
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
* imaplib: IDLE -> IMAP4 IDLE in exception message
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* docs: imaplib idle() phrasing and linking tweaks
* docs: imaplib: avoid linking to an invalid target
This reverts and rephrases part of a3f21cd75b
which created links to a method on a deliberately undocumented class.
The links didn't work consistently, and caused sphinx warnings that
broke cpython's continuous integration tests.
* imaplib: update test after recent exception change
This fixes a test that was broken by changing an exception in
b01de95171
* imaplib: rename idle() dur argument to duration
* imaplib: bytes.index() -> bytes.find()
This makes it more obvious which statement triggers the branch.
* imaplib: remove no-longer-necessary statement
Co-authored-by: Martin Panter <vadmium@users.noreply.github.com>
* docs: imaplib: concise & valid method links
The burst() method is a little tricky to link in restructuredText, due
to quirks of its parent class. This syntax allows sphinx to generate
working links without generating warnings (which break continuous
integration) and without burdening the reader with unimportant namespace
qualifications. It makes the reST source ugly, but few people read
the reST source, so it's a tolerable tradeoff.
* imaplib: note data types present in IDLE responses
* docs: imaplib: add comma to reST changes header
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* imaplib: sync doc strings with reST docs
* docs: imaplib: minor Idler clarifications
* imaplib: idle: emit (type, [data, ...]) tuples
This allows our iterator to emit untagged responses that contain literal
strings in the same way that imaplib's existing methods do, while still
emitting exactly one whole response per iteration.
* imaplib: while/yield instead of yield from iter()
* imaplib: idle: use deadline idiom when iterating
This simplifies the code, and avoids idle duration drift from time spent
processing each iteration.
* docs: imaplib: state duration/interval arg types
* docs: imaplib: minor rephrasing of a sentence
* docs: imaplib: reposition a paragraph
This might improve readability, especially when encountering Idler.burst()
for the first time.
* docs: imaplib: wrap long lines in idle() section
* docs: imaplib: note: Idler objects require 'with'
* docs: imaplib: say that 29 minutes is 1740 seconds
* docs: imaplib: mark a paragraph as a 'tip'
* docs: imaplib: rephrase reference to MS Windows
* imaplib: end doc string titles with a period
* imaplib: idle: socket timeouts instead of select()
IDLE timeouts were originally implemented using select() after
checking for the presence of already-buffered data.
That allowed timeouts on pipe connetions like IMAP4_stream.
However, it seemed possible that SSL data arriving without any
IMAP data afterward could cause select() to indicate available
application data when there was none, leading to a read() call
that would block with no timeout. It was unclear under what
conditions this would happen in practice. This change switches
to socket timeouts instead of select(), just to be safe.
This also reverts IMAP4_stream changes that were made to support IDLE
timeouts, since our new implementation only supports socket connections.
* imaplib: Idler: rename private state attributes
* imaplib: rephrase a comment in example code
* docs: imaplib: idle: use Sphinx code-block:: pycon
* docs: whatsnew: imaplib: reformat IMAP4.idle entry
* imaplib: idle: make doc strings brief
Since we generally rely on the reST/html documentation for details, we
can keep these doc strings short. This matches the module's existing doc
string style and avoids having to sync small changes between two files.
* imaplib: Idler: split assert into two statements
* imaplib: Idler: move assignment out of try: block
* imaplib: Idler: move __exit__() for readability
* imaplib: Idler: move __next__() for readability
* imaplib: test: make IdleCmdHandler a global class
* docs: imaplib: idle: collapse double-spaces
* imaplib: warn on use of undocumented 'file' attr
* imaplib: revert import reformatting
Since we no longer import platform or selectors, the original import
statement style can be restored, reducing the footprint of PR #122542.
* imaplib: restore original exception msg formatting
This reduces the footprint of PR #122542.
* docs: imaplib: idle: versionadded:: next
* imaplib: move import statement to where it's used
This import is only needed if external code tries to use an attribute
that it shouldn't be using. Making it a local import reduces module
loading time in supported cases.
* imaplib test: RuntimeWarning on IMAP4.file access
* imaplib: use stacklevel=2 in warnings.warn()
* imaplib test: simplify IMAP4.file warning test
* imaplib test: pre-idle-continuation response
* imaplib test: post-done untagged response
* imaplib: downgrade idle-denied exception to error
This makes it easier for client code to distinguish a temporary
rejection of the IDLE command from a server responding incorrectly to
IDLE.
* imaplib: simplify check for socket object
* imaplib: narrow the scope of IDLE socket timeouts
If an IDLE duration or burst() was in use, and an unsolicited response
contained a literal string, and crossed a packet boundary, and the
subsequent packet was delayed beyond the IDLE feature's time limit, the
timeout would leave the incoming protocol stream in a bad state (with
the tail of that response appearing where the start of a response is
expected).
This change moves the IDLE socket timeout to cover only the start
of a response, so it can no longer cause that problem.
* imaplib: preserve partial reads on exception
This ensures that short IDLE durations / burst() intervals
won't risk corrupting response lines that span multiple packets.
* imaplib: read/readline: save multipart buffer tail
For resilience if read() or readline() ever complete with more than one
bytes object remaining in the buffer. This is not expected to happen,
but it seems wise to be prepared for a future change making it possible.
* imaplib: use TimeoutError subclass only if needed
* doc: imaplib: elaborate on IDLE response delivery
* doc: imaplib: elaborate in note re: IMAP4.response
* imaplib: comment on benefit of reading in chunks
Our read() implementation designed to support IDLE replaces the one from
PR #119514, fixing the same problem it was addressing. The tests that it
added are preserved.
* imaplib: readline(): treat ConnectionError as EOF
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
Co-authored-by: Martin Panter <vadmium@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
