Commit graph

463 commits

Author SHA1 Message Date
Victor Stinner
c7b01e3c32
[3.11] gh-90872: Fix subprocess.Popen.wait() for negative timeout (#116989) (#117003)
gh-90872: Fix subprocess.Popen.wait() for negative timeout (#116989)

On Windows, subprocess.Popen.wait() no longer calls
WaitForSingleObject() with a negative timeout: pass 0 ms if the
timeout is negative.

(cherry picked from commit 27cf3ed00c)
2024-03-19 14:21:42 +00:00
Miss Islington (bot)
363dddba14
[3.11] gh-104522: Fix test_subprocess failure when build Python in the root home directory (GH-114236) (GH-114245)
EPERM is raised when setreuid() fails.
EACCES is set in execve() when the test user has not access to sys.executable.
(cherry picked from commit 311d1e2701)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-01-18 13:19:18 +01:00
Serhiy Storchaka
2c9872428e
[3.11] gh-104522: Fix OSError raised when run a subprocess (GH-114195) (GH-114243)
Only set filename to cwd if it was caused by failed chdir(cwd).

_fork_exec() now returns "noexec:chdir" for failed chdir(cwd).

(cherry picked from commit e2c097ebde)

Co-authored-by: Robert O'Shea <PurityLake@users.noreply.github.com>
2024-01-18 11:32:57 +00:00
Miss Islington (bot)
d15e1ac828
gh-87868: Sort and remove duplicates in getenvironment() (GH-102731)
(cherry picked from commit c31be58da8)

Co-authored-by: AN Long <aisk@users.noreply.github.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2024-01-11 22:27:12 +00:00
Miss Islington (bot)
49a45f6cd8
[3.11] gh-110184: Fix subprocess test_pipesize_default() (GH-110465) (#110472)
gh-110184: Fix subprocess test_pipesize_default() (GH-110465)

For proc.stdin, get the size of the read end of the test pipe.

Use subprocess context manager ("with proc:").
(cherry picked from commit d023d4166b)

Co-authored-by: Victor Stinner <vstinner@python.org>
2023-10-06 14:11:51 +00:00
Serhiy Storchaka
cf19e8ea3a
[3.11] gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') (GH-108480) (GH-108924)
(cherry picked from commit 1e0d62793a)
2023-09-05 15:27:55 +00:00
Serhiy Storchaka
76f52196b9
[3.11] gh-108416: Mark slow test methods with @requires_resource('cpu') (GH-108421) (GH-108799)
Only mark tests which spend significant system or user time,
by itself or in subprocesses.
(cherry picked from commit f3ba0a74cd)
2023-09-03 09:34:30 +03:00
Miss Islington (bot)
c2e51a9985
[3.11] gh-105436: Ignore unrelated errors when checking empty env (GH-105742) (#105757)
gh-105436: Ignore unrelated errors when checking empty env (GH-105742)
(cherry picked from commit 4cefe3cf10)

Co-authored-by: Steve Dower <steve.dower@python.org>
2023-06-13 23:39:56 +00:00
Miss Islington (bot)
261d0b4f0e
[3.11] gh-105436: The environment block should end with two null wchar_t values (GH-105495) (#105701)
gh-105436: The environment block should end with two null wchar_t values (GH-105495)
(cherry picked from commit 4f7d3b602d)

Co-authored-by: Dora203 <66343334+sku2000@users.noreply.github.com>
2023-06-12 16:49:57 +00:00
Miss Islington (bot)
2801b3f8f7
[3.11] gh-104472: Skip test_subprocess.ProcessTestCase.test_empty_env if ASAN is enabled (GH-104667) (#104673)
gh-104472: Skip `test_subprocess.ProcessTestCase.test_empty_env` if ASAN is enabled (GH-104667)

Skip test_subprocess.ProcessTestCase.test_empty_env if ASAN is enabled.
(cherry picked from commit c3f43bfb4b)

Co-authored-by: chgnrdv <52372310+chgnrdv@users.noreply.github.com>
2023-05-19 21:48:07 +00:00
Miss Islington (bot)
5d47150eac
[3.11] gh-95672 fix typo SkitTest to SkipTest (gh-102119) (#102121)
gh-95672 fix typo SkitTest to SkipTest (gh-102119)
(cherry picked from commit d5c7954d0c)

Co-authored-by: HyunKyun Moon <hyunkyun.moon@linecorp.com>
2023-02-22 10:34:51 +09:00
Miss Islington (bot)
5533cf67e7
gh-100005: Skip test_script_as_dev_fd() on FreeBSD (GH-100006)
On FreeBSD, skip test_script_as_dev_fd() of test_cmd_line_script if
fdescfs is not mounted (at /dev/fd).
(cherry picked from commit 038b151963)

Co-authored-by: Victor Stinner <vstinner@python.org>
2022-12-05 06:07:48 -08:00
Miss Islington (bot)
7aa87bba05
gh-60203: Always pass True/False as boolean arguments in tests (GH-99983)
Unless we explicitly test non-bool values.
(cherry picked from commit 76f43fc09a)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2022-12-04 04:50:46 -08:00
Miss Islington (bot)
9133aabc70
gh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (GH-97826)
* fix AttributeError, add unit test
(cherry picked from commit db64fb9bbe)

Co-authored-by: andrei kulakov <andrei.avk@gmail.com>
2022-10-04 20:29:40 -07:00
Gregory P. Smith
f6dd14c653
gh-82616: Add process_group support to subprocess.Popen (#23930)
One more thing that can help prevent people from using `preexec_fn`.

Also adds conditional skips to two tests exposing ASAN flakiness on the Ubuntu 20.04 Address Sanitizer Github CI system. When that build is run on more modern systems the "problem" does not show up. It seems ASAN implementation related.

Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2022-05-05 16:22:32 -07:00
Dennis Sweeney
b9636180b3
gh-91954: Use shell=True in test_subprocess.test_encoding_warning (GH-92090) 2022-04-30 20:38:19 -04:00
Inada Naoki
354ace8b07
gh-91954: Emit EncodingWarning from locale and subprocess (GH-91977)
locale.getpreferredencoding() and subprocess.Popen() emit EncodingWarning
2022-04-30 15:53:29 +09:00
Gregory P. Smith
cd5726fe67
gh-91401: Add a failsafe way to disable vfork. (#91490)
Just in case there is ever an issue with _posixsubprocess's use of
vfork() due to the complexity of using it properly and potential
directions that Linux platforms where it defaults to on could take, this
adds a failsafe so that users can disable its use entirely by setting
a global flag.

No known reason to disable it exists. But it'd be a shame to encounter
one and not be able to use CPython without patching and rebuilding it.

See the linked issue for some discussion on reasoning.

Also documents the existing way to disable posix_spawn.
2022-04-25 16:19:39 -07:00
Christian Heimes
082d3495d0
bpo-40280: Emscripten fork_exec now fails early (GH-32224) 2022-04-01 21:20:56 +02:00
Christian Heimes
8464fbc42e
bpo-40280: Skip subprocess-based tests on wasm32-emscripten (GH-30615) 2022-01-25 08:09:06 +01:00
Rémi Lapeyre
4d2957c1b9
bpo-40497: Fix handling of check in subprocess.check_output() (GH-19897)
Co-authored-by: Tal Einat <taleinat@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-09-20 17:09:05 +02:00
Serhiy Storchaka
2a8127cafe
bpo-25130: Add calls of gc.collect() in tests to support PyPy (GH-28005) 2021-08-29 14:04:40 +03:00
Karthikeyan Singaravelan
7179930ab5
bpo-43826: Fix resource warning due to unclosed objects. (GH-25381) 2021-08-24 16:13:46 +05:30
Victor Stinner
b1f413e6cf
bpo-28254: Cleanup test_subprocess.test_preexec_gc_module_failure() (GH-25709)
Saving/restoring gc.disable and gc.isenabled is no longer needed.
2021-04-29 10:26:53 +02:00
Victor Stinner
103d5e420d
bpo-28254: _posixsubprocess uses PyGC_Enable/PyGC_Disable (GH-25693) 2021-04-28 19:09:29 +02:00
M. Kocher
db0c5b786d
bpo-43776: Remove list call from args in Popen repr (GH-25338)
Removes the `list` call in the Popen `repr`.

Current implementation:

For cmd = `python --version`,  with `shell=True`.

```bash
<Popen: returncode: None args: ['p', 'y', 't', 'h', 'o', 'n', ' ', '-', '-',...>
```

For `shell=False` and args=`['python', '--version']`, the output is correct:

```bash
<Popen: returncode: None args: ['python', '--version']>
```

With the new changes the `repr`  yields:

For cmd = `python --version`,  with `shell=True`:

```bash
<Popen: returncode: None args: 'python --version'>
```

For `shell=False` and args=`['python', '--version']`, the output:

```bash
<Popen: returncode: None args: ['python', '--version']>
```

Automerge-Triggered-By: GH:gpshead
2021-04-28 01:16:38 -07:00
Gregory P. Smith
64abf37344
bpo-42388: Fix subprocess.check_output input=None when text=True (GH-23467)
When the modern text= spelling of the universal_newlines= parameter was added
for Python 3.7, check_output's special case around input=None was overlooked.
So it behaved differently with universal_newlines=True vs text=True.  This
reconciles the behavior to be consistent and adds a test to guarantee it.

Also clarifies the existing check_output documentation.

Co-authored-by: Alexey Izbyshev <izbyshev@ispras.ru>
2020-12-24 20:57:21 -08:00
Filipe Laíns
01a202ab6b
bpo-40550: Fix time-of-check/time-of-action issue in subprocess.Popen.send_signal. (GH-20010)
send_signal() now swallows the exception if the process it thought was still alive winds up not to exist anymore (always a plausible race condition despite the checks).

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2020-11-21 01:22:08 -08:00
Alexey Izbyshev
c0590c0033
bpo-42146: Fix memory leak in subprocess.Popen() in case of uid/gid overflow (GH-22966)
Fix memory leak in subprocess.Popen() in case of uid/gid overflow

Also add a test that would catch this leak with `--huntrleaks`.

Alas, the test for `extra_groups` also exposes an inconsistency
in our error reporting: we use a custom ValueError for `extra_groups`,
but propagate OverflowError for `user` and `group`.
2020-10-25 17:09:32 -07:00
Gregory P. Smith
786addd9d0
bpo-41586: Attempt to make the pipesize tests more robust. (GH-22839)
Several buildbots are failing on these, likely due to an inability to
set the pipe size to the desired test value.
2020-10-20 17:37:20 -07:00
Ruben Vorderman
23c0fb8edd
bpo-41586: Add pipesize parameter to subprocess & F_GETPIPE_SZ and F_SETPIPE_SZ to fcntl. (GH-21921)
* Add F_SETPIPE_SZ and F_GETPIPE_SZ to fcntl module
* Add pipesize parameter for subprocess.Popen class

This will allow the user to control the size of the pipes.
On linux the default is 64K. When a pipe is full it blocks for writing.
When a pipe is empty it blocks for reading. On processes that are
very fast this can lead to a lot of wasted CPU cycles. On a typical
Linux system the max pipe size is 1024K which is much better.
For high performance-oriented libraries such as xopen it is nice to
be able to set the pipe size.

The workaround without this feature is to use my_popen_process.stdout.fileno() in
conjuction with fcntl and 1031 (value of F_SETPIPE_SZ) to acquire this behavior.
2020-10-19 16:30:02 -07:00
Serhiy Storchaka
b1a87300a0
bpo-41385: Fix test_executable_without_cwd on Windows (GH-21608) 2020-07-26 10:21:39 +03:00
Hai Shi
0c4f0f3b29
bpo-40275: Use new test.support helper submodules in tests (GH-21169) 2020-06-30 15:46:31 +02:00
Guido van Rossum
48b069a003
bpo-39481: Implementation for PEP 585 (#18239)
This implements things like `list[int]`,
which returns an object of type `types.GenericAlias`.
This object mostly acts as a proxy for `list`,
but has attributes `__origin__` and `__args__`
that allow recovering the parts (with values `list` and `(int,)`.

There is also an approximate notion of type variables;
e.g. `list[T]` has a `__parameters__` attribute equal to `(T,)`.
Type variables are objects of type `typing.TypeVar`.
2020-04-07 09:50:06 -07:00
Victor Stinner
278c1e159c
bpo-40094: Add test.support.wait_process() (GH-19254)
Moreover, the following tests now check the child process exit code:

* test_os.PtyTests
* test_mailbox.test_lock_conflict()
* test_tempfile.test_process_awareness()
* test_uuid.testIssue8621()
* multiprocessing resource tracker tests
2020-03-31 20:08:12 +02:00
Ronald Oussoren
044cf94f61
bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch (GH-9516)
* bpo-22490: Remove "__PYVENV_LAUNCHER__" from the shell environment on macOS

This changeset removes the environment varialbe "__PYVENV_LAUNCHER__"
during interpreter launch as it is only needed to communicate between
the stub executable in framework installs and the actual interpreter.

Leaving the environment variable present may lead to misbehaviour when
launching other scripts.

* Actually commit the changes for issue 22490...

* Correct typo

Co-Authored-By: Nicola Soranzo <nicola.soranzo@gmail.com>

* Run make patchcheck

Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
2020-03-22 14:31:46 -04:00
Victor Stinner
f7b5d419bf
bpo-39855: Fix test_subprocess if nobody user doesn't exist (GH-18781)
test_subprocess.test_user() now skips the test on an user name if the
user name doesn't exist. For example, skip the test if the user
"nobody" doesn't exist on Linux.
2020-03-05 14:28:40 +01:00
Alex Rebert
d3ae95e1e9 bpo-35182: fix communicate() crash after child closes its pipes (GH-17020) (GH-18117)
When communicate() is called in a loop, it crashes when the child process
has already closed any piped standard stream, but still continues to be running

Co-authored-by: Andriy Maletsky <andriy.maletsky@gmail.com>
2020-01-22 15:28:31 -08:00
Victor Stinner
e85a305503
bpo-38630: Fix subprocess.Popen.send_signal() race condition (GH-16984)
On Unix, subprocess.Popen.send_signal() now polls the process status.
Polling reduces the risk of sending a signal to the wrong process if
the process completed, the Popen.returncode attribute is still None,
and the pid has been reassigned (recycled) to a new different
process.
2020-01-15 17:38:55 +01:00
Batuhan Taşkaya
4dc5a9df59 bpo-39019: Implement missing __class_getitem__ for subprocess classes (GH-17558) 2019-12-30 16:02:04 +00:00
Victor Stinner
0d63bacefd
bpo-38614: Use test.support.SHORT_TIMEOUT constant (GH-17566)
Replace hardcoded timeout constants in tests with SHORT_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.

SHORT_TIMEOUT is 30 seconds by default, but it can be longer
depending on --timeout command line option.

The change makes almost all timeouts longer, except
test_reap_children() of test_support which is made 2x shorter:
SHORT_TIMEOUT should be enough. If this test starts to fail,
LONG_TIMEOUT should be used instead.

Uniformize also "from test import support" import in some test files.
2019-12-11 11:30:03 +01:00
Victor Stinner
6cac113666
bpo-38991: Remove test.support.strip_python_stderr() (GH-17490)
test.support: run_python_until_end(), assert_python_ok() and
assert_python_failure() functions no longer strip whitespaces from
stderr.
2019-12-08 08:38:16 +01:00
Andrey Doroschenko
645005e947 bpo-38724: Implement subprocess.Popen.__repr__ (GH-17151) 2019-11-17 16:08:31 +02:00
Pablo Galindo
46113e0cf3
bpo-38456: Handle the case when there is no 'true' command (GH-16739) 2019-10-13 02:40:24 +01:00
Gregory P. Smith
67b93f80c7
bpo-38456: Use /bin/true in test_subprocess (GH-16736)
* bpo-38456: Use /bin/true in test_subprocess.

Instead of sys.executable, "-c", "pass" or "import sys; sys.exit(0)"
use /bin/true when it is available.  On a reasonable machine this
shaves up to two seconds wall time off the otherwise ~40sec execution
on a --with-pydebug build.  It should be more notable on many
buildbots or overloaded slower I/O systems (CI, etc).
2019-10-12 16:35:53 -07:00
Gregory P. Smith
f3751efb5c
bpo-38417: Add umask support to subprocess (GH-16726)
On POSIX systems, allow the umask to be set in the child process before we exec.
2019-10-12 13:24:56 -07:00
Victor Stinner
faca855342 bpo-36046: posix_spawn() doesn't support uid/gid (GH-16384)
* subprocess.Popen now longer uses posix_spawn() if uid, gid or gids are set.
* test_subprocess: add "nobody" and "nfsnobody" group names for test_group().
* test_subprocess: test_user() and test_group() are now also tested with close_fds=False.
2019-09-25 15:52:49 +02:00
Gregory P. Smith
693aa80a43 bpo-36046: Fix buildbot failures (GH-16091)
Varying user/group/permission check needs on platforms.
2019-09-13 14:43:35 +01:00
Patrick McLean
2b2ead7438 bpo-36046: Add user and group parameters to subprocess (GH-11950)
* subprocess: Add user, group and extra_groups paremeters to subprocess.Popen

This adds a `user` parameter to the Popen constructor that will call
setreuid() in the child before calling exec(). This allows processes
running as root to safely drop privileges before running the subprocess
without having to use a preexec_fn.

This also adds a `group` parameter that will call setregid() in
the child process before calling exec().

Finally an `extra_groups` parameter was added that will call
setgroups() to set the supplimental groups.
2019-09-12 18:15:44 +01:00
Gregory P. Smith
580d2782f7 bpo-37424: Avoid a hang in subprocess.run timeout output capture (GH-14490)
Fixes a possible hang when using a timeout on subprocess.run() while
capturing output. If the child process spawned its own children or otherwise
connected its stdout or stderr handles with another process, we could hang
after the timeout was reached and our child was killed when attempting to read
final output from the pipes.
2019-09-11 02:23:05 -07:00