[3.13] gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally (GH-133222) (GH-135343)
Some checks are pending
Tests / Windows MSI (push) Blocked by required conditions
Tests / (push) Blocked by required conditions
Tests / CIFuzz (push) Blocked by required conditions
Tests / Check if the ABI has changed (push) Blocked by required conditions
Tests / Check if Autoconf files are up to date (push) Blocked by required conditions
Tests / Change detection (push) Waiting to run
Tests / Docs (push) Blocked by required conditions
Tests / Check if generated files are up to date (push) Blocked by required conditions
Tests / Ubuntu SSL tests with OpenSSL (push) Blocked by required conditions
Tests / WASI (push) Blocked by required conditions
Tests / Hypothesis tests on Ubuntu (push) Blocked by required conditions
Tests / Address sanitizer (push) Blocked by required conditions
Tests / All required checks pass (push) Blocked by required conditions
Lint / lint (push) Waiting to run

gh-132969:  Fix error/hang when shutdown(wait=False) and task exited abnormally (GH-133222)

When shutdown is called with wait=False, the executor thread keeps running
even after the ProcessPoolExecutor's state is reset. The executor then tries
to replenish the worker processes pool resulting in an error and a potential hang
when it comes across a worker that has died. Fixed the issue by having
_adjust_process_count() return without doing anything if the ProcessPoolExecutor's
state has been reset.

Added unit tests to validate two scenarios:
max_workers < num_tasks (exception)
max_workers > num_tasks (exception + hang)
(cherry picked from commit 598aa7cc98)

Co-authored-by: Ajay Kamdar <140011370+ogbiggles@users.noreply.github.com>
This commit is contained in:
Miss Islington (bot) 2025-06-16 14:08:03 +02:00 committed by GitHub
parent 3bd2818664
commit 404e8aa9f3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 70 additions and 0 deletions

View file

@ -747,6 +747,11 @@ class ProcessPoolExecutor(_base.Executor):
self._executor_manager_thread_wakeup
def _adjust_process_count(self):
# gh-132969: avoid error when state is reset and executor is still running,
# which will happen when shutdown(wait=False) is called.
if self._processes is None:
return
# if there's an idle process, we don't need to spawn a new one.
if self._idle_worker_semaphore.acquire(blocking=False):
return