mirror of
				https://github.com/python/cpython.git
				synced 2025-11-03 19:34:08 +00:00 
			
		
		
		
	bpo-33649: Fix asyncio-dev (GH-9324)
This commit is contained in:
		
							parent
							
								
									afde1c1a05
								
							
						
					
					
						commit
						805e27eff6
					
				
					 6 changed files with 175 additions and 348 deletions
				
			
		| 
						 | 
				
			
			@ -15,7 +15,7 @@ Utilities to run asyncio programs, create Tasks, and
 | 
			
		|||
await on multiple things with timeouts.
 | 
			
		||||
 | 
			
		||||
.. list-table::
 | 
			
		||||
    :widths: 30 70
 | 
			
		||||
    :widths: 50 50
 | 
			
		||||
 | 
			
		||||
    * - :func:`run`
 | 
			
		||||
      - Create event loop, run a coroutine, close the loop.
 | 
			
		||||
| 
						 | 
				
			
			@ -71,7 +71,7 @@ implement connection pools, and pub/sub patterns.
 | 
			
		|||
 | 
			
		||||
 | 
			
		||||
.. list-table::
 | 
			
		||||
    :widths: 30 70
 | 
			
		||||
    :widths: 50 50
 | 
			
		||||
 | 
			
		||||
    * - :class:`Queue`
 | 
			
		||||
      - A FIFO queue.
 | 
			
		||||
| 
						 | 
				
			
			@ -97,7 +97,7 @@ Subprocesses
 | 
			
		|||
Utilities to spawn subprocesses and run shell commands.
 | 
			
		||||
 | 
			
		||||
.. list-table::
 | 
			
		||||
    :widths: 30 70
 | 
			
		||||
    :widths: 50 50
 | 
			
		||||
 | 
			
		||||
    * - ``await`` :func:`create_subprocess_exec`
 | 
			
		||||
      - Create a subprocess.
 | 
			
		||||
| 
						 | 
				
			
			@ -120,7 +120,7 @@ Streams
 | 
			
		|||
High-level APIs to work with network IO.
 | 
			
		||||
 | 
			
		||||
.. list-table::
 | 
			
		||||
    :widths: 30 70
 | 
			
		||||
    :widths: 50 50
 | 
			
		||||
 | 
			
		||||
    * - ``await`` :func:`open_connection`
 | 
			
		||||
      -  Establish a TCP connection.
 | 
			
		||||
| 
						 | 
				
			
			@ -155,7 +155,7 @@ Synchronization
 | 
			
		|||
Threading-like synchronization primitives that can be used in Tasks.
 | 
			
		||||
 | 
			
		||||
.. list-table::
 | 
			
		||||
    :widths: 30 70
 | 
			
		||||
    :widths: 50 50
 | 
			
		||||
 | 
			
		||||
    * - :class:`Lock`
 | 
			
		||||
      - A mutex lock.
 | 
			
		||||
| 
						 | 
				
			
			@ -185,7 +185,7 @@ Exceptions
 | 
			
		|||
==========
 | 
			
		||||
 | 
			
		||||
.. list-table::
 | 
			
		||||
    :widths: 30 70
 | 
			
		||||
    :widths: 50 50
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
    * - :exc:`asyncio.TimeoutError`
 | 
			
		||||
| 
						 | 
				
			
			
 | 
			
		|||
| 
						 | 
				
			
			@ -2,415 +2,237 @@
 | 
			
		|||
 | 
			
		||||
.. _asyncio-dev:
 | 
			
		||||
 | 
			
		||||
Develop with asyncio
 | 
			
		||||
====================
 | 
			
		||||
=======================
 | 
			
		||||
Developing with asyncio
 | 
			
		||||
=======================
 | 
			
		||||
 | 
			
		||||
Asynchronous programming is different than classical "sequential" programming.
 | 
			
		||||
This page lists common traps and explains how to avoid them.
 | 
			
		||||
Asynchronous programming is different from classical "sequential"
 | 
			
		||||
programming.
 | 
			
		||||
 | 
			
		||||
This page lists common mistakes and traps and explains how
 | 
			
		||||
to avoid them.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _asyncio-debug-mode:
 | 
			
		||||
 | 
			
		||||
Debug mode of asyncio
 | 
			
		||||
---------------------
 | 
			
		||||
Debug Mode
 | 
			
		||||
==========
 | 
			
		||||
 | 
			
		||||
The implementation of :mod:`asyncio` has been written for performance.
 | 
			
		||||
In order to ease the development of asynchronous code, you may wish to
 | 
			
		||||
enable *debug mode*.
 | 
			
		||||
By default asyncio runs in production mode.  In order to ease
 | 
			
		||||
the development asyncio has a *debug mode*.
 | 
			
		||||
 | 
			
		||||
To enable all debug checks for an application:
 | 
			
		||||
To enable debugging for an application:
 | 
			
		||||
 | 
			
		||||
* Enable the asyncio debug mode globally by setting the environment variable
 | 
			
		||||
  :envvar:`PYTHONASYNCIODEBUG` to ``1``, using ``-X dev`` command line option
 | 
			
		||||
  (see the :option:`-X` option), or by calling
 | 
			
		||||
  :meth:`loop.set_debug`.
 | 
			
		||||
* Set the log level of the :ref:`asyncio logger <asyncio-logger>` to
 | 
			
		||||
  :py:data:`logging.DEBUG`. For example, call
 | 
			
		||||
  ``logging.basicConfig(level=logging.DEBUG)`` at startup.
 | 
			
		||||
* Configure the :mod:`warnings` module to display :exc:`ResourceWarning`
 | 
			
		||||
  warnings. For example, use the ``-Wdefault`` command line option of Python to
 | 
			
		||||
  display them.
 | 
			
		||||
* Enable the debug mode globally by setting the environment variable
 | 
			
		||||
  :envvar:`PYTHONASYNCIODEBUG` to ``1``.
 | 
			
		||||
 | 
			
		||||
Examples debug checks:
 | 
			
		||||
* Alternatively, the debug mode can be enabled by using the ``-X dev``
 | 
			
		||||
  command line option for Python (see the :option:`-X` option).
 | 
			
		||||
 | 
			
		||||
* Log :ref:`coroutines defined but never "yielded from"
 | 
			
		||||
  <asyncio-coroutine-not-scheduled>`
 | 
			
		||||
* :meth:`loop.call_soon` and :meth:`loop.call_at` methods
 | 
			
		||||
  raise an exception if they are called from the wrong thread.
 | 
			
		||||
* Log the execution time of the selector
 | 
			
		||||
* Log callbacks taking more than 100 ms to be executed. The
 | 
			
		||||
* Yet another way to enable the debug mode is by calling
 | 
			
		||||
  :meth:`loop.set_debug` or by passing ``debug=True`` to
 | 
			
		||||
  :func:`asyncio.run`.
 | 
			
		||||
 | 
			
		||||
In addition to enabling debug mode, consider also:
 | 
			
		||||
 | 
			
		||||
* setting the log level of the :ref:`asyncio logger <asyncio-logger>` to
 | 
			
		||||
  :py:data:`logging.DEBUG`, for example the following snippet of code
 | 
			
		||||
  can be run at startup of the application::
 | 
			
		||||
 | 
			
		||||
    logging.basicConfig(level=logging.DEBUG)
 | 
			
		||||
 | 
			
		||||
* configuring the :mod:`warnings` module to display
 | 
			
		||||
  :exc:`ResourceWarning` warnings.  One way of doing that is by
 | 
			
		||||
  using the ``-Wdefault`` command line option.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
In asyncio debug mode the following checks are performed:
 | 
			
		||||
 | 
			
		||||
* Log :ref:`coroutines that were not awaited
 | 
			
		||||
  <asyncio-coroutine-not-scheduled>`; this mitigates the "forgotten
 | 
			
		||||
  await" pitfall.
 | 
			
		||||
 | 
			
		||||
* Many non-treadsafe asyncio APIs (such as :meth:`loop.call_soon` and
 | 
			
		||||
  :meth:`loop.call_at` methods) raise an exception if they are called
 | 
			
		||||
  from a wrong thread.
 | 
			
		||||
 | 
			
		||||
* Log the execution time of the IO selector if it takes too long to
 | 
			
		||||
  perform an IO operation.
 | 
			
		||||
 | 
			
		||||
* Log callbacks taking longer than 100 ms to be executed.  The
 | 
			
		||||
  :attr:`loop.slow_callback_duration` attribute is the minimum
 | 
			
		||||
  duration in seconds of "slow" callbacks.
 | 
			
		||||
* :exc:`ResourceWarning` warnings are emitted when transports and event loops
 | 
			
		||||
  are :ref:`not closed explicitly <asyncio-close-transports>`.
 | 
			
		||||
 | 
			
		||||
.. versionchanged:: 3.7
 | 
			
		||||
 | 
			
		||||
   The new ``-X dev`` command line option can now also be used to enable
 | 
			
		||||
   the debug mode.
 | 
			
		||||
 | 
			
		||||
.. seealso::
 | 
			
		||||
 | 
			
		||||
   The :meth:`loop.set_debug` method and the :ref:`asyncio logger
 | 
			
		||||
   <asyncio-logger>`.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
Cancellation
 | 
			
		||||
------------
 | 
			
		||||
 | 
			
		||||
Cancellation of tasks is not common in classic programming. In asynchronous
 | 
			
		||||
programming, not only is it something common, but you have to prepare your
 | 
			
		||||
code to handle it.
 | 
			
		||||
 | 
			
		||||
Futures and tasks can be cancelled explicitly with their :meth:`Future.cancel`
 | 
			
		||||
method. The :func:`wait_for` function cancels the waited task when the timeout
 | 
			
		||||
occurs. There are many other cases where a task can be cancelled indirectly.
 | 
			
		||||
 | 
			
		||||
Don't call :meth:`~Future.set_result` or :meth:`~Future.set_exception` method
 | 
			
		||||
of :class:`Future` if the future is cancelled: it would fail with an exception.
 | 
			
		||||
For example, write::
 | 
			
		||||
 | 
			
		||||
    if not fut.cancelled():
 | 
			
		||||
        fut.set_result('done')
 | 
			
		||||
 | 
			
		||||
Don't schedule directly a call to the :meth:`~Future.set_result` or the
 | 
			
		||||
:meth:`~Future.set_exception` method of a future with
 | 
			
		||||
:meth:`loop.call_soon`: the future can be cancelled before its method
 | 
			
		||||
is called.
 | 
			
		||||
 | 
			
		||||
If you wait for a future, you should check early if the future was cancelled to
 | 
			
		||||
avoid useless operations. Example::
 | 
			
		||||
 | 
			
		||||
    async def slow_operation(fut):
 | 
			
		||||
        if fut.cancelled():
 | 
			
		||||
            return
 | 
			
		||||
        # ... slow computation ...
 | 
			
		||||
        await fut
 | 
			
		||||
        # ...
 | 
			
		||||
 | 
			
		||||
The :func:`shield` function can also be used to ignore cancellation.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _asyncio-multithreading:
 | 
			
		||||
 | 
			
		||||
Concurrency and multithreading
 | 
			
		||||
------------------------------
 | 
			
		||||
Concurrency and Multithreading
 | 
			
		||||
==============================
 | 
			
		||||
 | 
			
		||||
An event loop runs in a thread (typically the main thread) and executes
 | 
			
		||||
all callbacks and tasks in its thread.  While a task is running in the
 | 
			
		||||
event loop, no other tasks may run in the same thread. When a task
 | 
			
		||||
executes an ``await`` expression, the running task gets suspended, and the
 | 
			
		||||
event loop executes the next task. Prior to suspending the task, the awaiting
 | 
			
		||||
chain is checked, and if the chain ends with a future, the running task is
 | 
			
		||||
not suspended.
 | 
			
		||||
all callbacks and Tasks in its thread.  While a Task is running in the
 | 
			
		||||
event loop, no other Tasks can run in the same thread.  When a Task
 | 
			
		||||
executes an ``await`` expression, the running Task gets suspended, and
 | 
			
		||||
the event loop executes the next Task.
 | 
			
		||||
 | 
			
		||||
To schedule a callback from a different thread, the
 | 
			
		||||
To schedule a callback from a different OS thread, the
 | 
			
		||||
:meth:`loop.call_soon_threadsafe` method should be used. Example::
 | 
			
		||||
 | 
			
		||||
    loop.call_soon_threadsafe(callback, *args)
 | 
			
		||||
 | 
			
		||||
Most asyncio objects are not thread safe. You should only worry if you access
 | 
			
		||||
objects outside the event loop. For example, to cancel a future, don't call
 | 
			
		||||
directly its :meth:`Future.cancel` method, but::
 | 
			
		||||
Almost all asyncio objects are not thread safe, which is typically
 | 
			
		||||
not a problem unless there is code that works with them from outside
 | 
			
		||||
of a Task or a callback.  If there's a need for such code to call a
 | 
			
		||||
low-level asyncio API, the :meth:`loop.call_soon_threadsafe` method
 | 
			
		||||
should be used, e.g.::
 | 
			
		||||
 | 
			
		||||
    loop.call_soon_threadsafe(fut.cancel)
 | 
			
		||||
 | 
			
		||||
To handle signals and to execute subprocesses, the event loop must be run in
 | 
			
		||||
the main thread.
 | 
			
		||||
 | 
			
		||||
To schedule a coroutine object from a different thread, the
 | 
			
		||||
To schedule a coroutine object from a different OS thread, the
 | 
			
		||||
:func:`run_coroutine_threadsafe` function should be used. It returns a
 | 
			
		||||
:class:`concurrent.futures.Future` to access the result::
 | 
			
		||||
 | 
			
		||||
     async def coro_func():
 | 
			
		||||
          return await asyncio.sleep(1, 42)
 | 
			
		||||
 | 
			
		||||
     # Later in another OS thread:
 | 
			
		||||
 | 
			
		||||
     future = asyncio.run_coroutine_threadsafe(coro_func(), loop)
 | 
			
		||||
     result = future.result(timeout)  # Wait for the result with a timeout
 | 
			
		||||
     # Wait for the result:
 | 
			
		||||
     result = future.result()
 | 
			
		||||
 | 
			
		||||
To handle signals and to execute subprocesses, the event loop must be
 | 
			
		||||
run in the main thread.
 | 
			
		||||
 | 
			
		||||
The :meth:`loop.run_in_executor` method can be used with a
 | 
			
		||||
:class:`concurrent.futures.ThreadPoolExecutor` to execute a callback in
 | 
			
		||||
different thread so as not to block the event loop's main thread.
 | 
			
		||||
 | 
			
		||||
.. seealso::
 | 
			
		||||
 | 
			
		||||
   The :ref:`Synchronization primitives <asyncio-sync>` section describes ways
 | 
			
		||||
   to synchronize tasks.
 | 
			
		||||
 | 
			
		||||
   The :ref:`Subprocess and threads <asyncio-subprocess-threads>` section lists
 | 
			
		||||
   asyncio limitations to run subprocesses from different threads.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
:class:`concurrent.futures.ThreadPoolExecutor` to execute
 | 
			
		||||
blocking code in a different OS thread without blocking the OS thread
 | 
			
		||||
that the event loop runs in.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _asyncio-handle-blocking:
 | 
			
		||||
 | 
			
		||||
Handle blocking functions correctly
 | 
			
		||||
-----------------------------------
 | 
			
		||||
Running Blocking Code
 | 
			
		||||
=====================
 | 
			
		||||
 | 
			
		||||
Blocking functions should not be called directly. For example, if a function
 | 
			
		||||
blocks for 1 second, other tasks are delayed by 1 second which can have an
 | 
			
		||||
important impact on reactivity.
 | 
			
		||||
Blocking (CPU-bound) code should not be called directly.  For example,
 | 
			
		||||
if a function performs a CPU-intensive calculation for 1 second,
 | 
			
		||||
all concurrent asyncio Tasks and IO operations would be delayed
 | 
			
		||||
by 1 second.
 | 
			
		||||
 | 
			
		||||
For networking and subprocesses, the :mod:`asyncio` module provides high-level
 | 
			
		||||
APIs like :ref:`protocols <asyncio-protocol>`.
 | 
			
		||||
 | 
			
		||||
An executor can be used to run a task in a different thread or even in a
 | 
			
		||||
different process, to not block the thread of the event loop. See the
 | 
			
		||||
:meth:`loop.run_in_executor` method.
 | 
			
		||||
 | 
			
		||||
.. seealso::
 | 
			
		||||
 | 
			
		||||
   The :ref:`Delayed calls <asyncio-delayed-calls>` section details how the
 | 
			
		||||
   event loop handles time.
 | 
			
		||||
An executor can be used to run a task in a different thread or even in
 | 
			
		||||
a different process to avoid blocking block the OS thread with the
 | 
			
		||||
event loop.  See the :meth:`loop.run_in_executor` method for more
 | 
			
		||||
details.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _asyncio-logger:
 | 
			
		||||
 | 
			
		||||
Logging
 | 
			
		||||
-------
 | 
			
		||||
=======
 | 
			
		||||
 | 
			
		||||
The :mod:`asyncio` module logs information with the :mod:`logging` module in
 | 
			
		||||
the logger ``'asyncio'``.
 | 
			
		||||
asyncio uses the :mod:`logging` module and all logging is performed
 | 
			
		||||
via the ``"asyncio"`` logger.
 | 
			
		||||
 | 
			
		||||
The default log level for the :mod:`asyncio` module is :py:data:`logging.INFO`.
 | 
			
		||||
For those not wanting such verbosity from :mod:`asyncio` the log level can
 | 
			
		||||
be changed.  For example, to change the level to :py:data:`logging.WARNING`:
 | 
			
		||||
The default log level is :py:data:`logging.INFO`, which can easily be
 | 
			
		||||
adjusted::
 | 
			
		||||
 | 
			
		||||
.. code-block:: none
 | 
			
		||||
 | 
			
		||||
   logging.getLogger('asyncio').setLevel(logging.WARNING)
 | 
			
		||||
   logging.getLogger("asyncio").setLevel(logging.WARNING)
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _asyncio-coroutine-not-scheduled:
 | 
			
		||||
 | 
			
		||||
Detect coroutine objects never scheduled
 | 
			
		||||
----------------------------------------
 | 
			
		||||
Detect never awaited coroutines
 | 
			
		||||
===============================
 | 
			
		||||
 | 
			
		||||
When a coroutine function is called and its result is not passed to
 | 
			
		||||
:meth:`asyncio.create_task` the execution of the coroutine object will
 | 
			
		||||
never be scheduled which is probably a bug. Using ``asyncio.create_task`` is
 | 
			
		||||
preferred to the low level :func:`ensure_future` and :meth:`loop.create_task`
 | 
			
		||||
methods. :ref:`Enable the debug mode of asyncio <asyncio-debug-mode>`
 | 
			
		||||
to :ref:`log a warning <asyncio-logger>` to detect it.
 | 
			
		||||
 | 
			
		||||
Example with the bug::
 | 
			
		||||
When a coroutine is called (e.g. ``coro()`` instead of ``await coro()``)
 | 
			
		||||
the call is not wrapped with :meth:`asyncio.create_task`, the execution
 | 
			
		||||
of the coroutine object will never be scheduled.  For example::
 | 
			
		||||
 | 
			
		||||
    import asyncio
 | 
			
		||||
 | 
			
		||||
    async def test():
 | 
			
		||||
        print("never scheduled")
 | 
			
		||||
 | 
			
		||||
    async def main():
 | 
			
		||||
        test()
 | 
			
		||||
 | 
			
		||||
    asyncio.run(main())
 | 
			
		||||
 | 
			
		||||
Output::
 | 
			
		||||
 | 
			
		||||
  test.py:7: RuntimeWarning: coroutine 'test' was never awaited
 | 
			
		||||
    test()
 | 
			
		||||
 | 
			
		||||
Output in debug mode::
 | 
			
		||||
 | 
			
		||||
    Coroutine test() at test.py:3 was never yielded from
 | 
			
		||||
    Coroutine object created at (most recent call last):
 | 
			
		||||
      File "test.py", line 7, in <module>
 | 
			
		||||
  test.py:7: RuntimeWarning: coroutine 'test' was never awaited
 | 
			
		||||
  Coroutine created at (most recent call last)
 | 
			
		||||
    File "../t.py", line 9, in <module>
 | 
			
		||||
      asyncio.run(main(), debug=True)
 | 
			
		||||
 | 
			
		||||
    < .. >
 | 
			
		||||
 | 
			
		||||
    File "../t.py", line 7, in main
 | 
			
		||||
      test()
 | 
			
		||||
    test()
 | 
			
		||||
 | 
			
		||||
The fix is to call the :meth:`asyncio.create_task` function. Using
 | 
			
		||||
``asyncio.create_task`` is preferred to the low level :func:`ensure_future` and
 | 
			
		||||
:meth:`loop.create_task` methods.
 | 
			
		||||
The usual fix is to either await the coroutine or call the
 | 
			
		||||
:meth:`asyncio.create_task` function::
 | 
			
		||||
 | 
			
		||||
.. seealso::
 | 
			
		||||
 | 
			
		||||
   :ref:`Pending task destroyed <asyncio-pending-task-destroyed>`.
 | 
			
		||||
    async def main():
 | 
			
		||||
        await test()
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
Detect exceptions never consumed
 | 
			
		||||
--------------------------------
 | 
			
		||||
Detect never consumed exceptions
 | 
			
		||||
================================
 | 
			
		||||
 | 
			
		||||
Python usually calls :func:`sys.excepthook` on unhandled exceptions. If
 | 
			
		||||
:meth:`Future.set_exception` is called, but the exception is never consumed,
 | 
			
		||||
:func:`sys.excepthook` is not called. Instead, :ref:`a log is emitted
 | 
			
		||||
<asyncio-logger>` when the future is deleted by the garbage collector, with the
 | 
			
		||||
traceback where the exception was raised.
 | 
			
		||||
If a :meth:`Future.set_exception` is called but the Future object is
 | 
			
		||||
never awaited on, the exception would never be propagated to the
 | 
			
		||||
user code.  In this case, asyncio would emit a log message when the
 | 
			
		||||
Future object is garbage collected.
 | 
			
		||||
 | 
			
		||||
Example of unhandled exception::
 | 
			
		||||
Example of an unhandled exception::
 | 
			
		||||
 | 
			
		||||
    import asyncio
 | 
			
		||||
 | 
			
		||||
    @asyncio.coroutine
 | 
			
		||||
    def bug():
 | 
			
		||||
    async def bug():
 | 
			
		||||
        raise Exception("not consumed")
 | 
			
		||||
 | 
			
		||||
    loop = asyncio.get_event_loop()
 | 
			
		||||
    asyncio.ensure_future(bug())
 | 
			
		||||
    loop.run_forever()
 | 
			
		||||
    loop.close()
 | 
			
		||||
    async def main():
 | 
			
		||||
        asyncio.create_task(bug())
 | 
			
		||||
 | 
			
		||||
    asyncio.run(main())
 | 
			
		||||
 | 
			
		||||
Output::
 | 
			
		||||
 | 
			
		||||
    Task exception was never retrieved
 | 
			
		||||
    future: <Task finished coro=<coro() done, defined at asyncio/coroutines.py:139> exception=Exception('not consumed',)>
 | 
			
		||||
    future: <Task finished coro=<bug() done, defined at test.py:3>
 | 
			
		||||
      exception=Exception('not consumed')>
 | 
			
		||||
 | 
			
		||||
    Traceback (most recent call last):
 | 
			
		||||
      File "asyncio/tasks.py", line 237, in _step
 | 
			
		||||
        result = next(coro)
 | 
			
		||||
      File "asyncio/coroutines.py", line 141, in coro
 | 
			
		||||
        res = func(*args, **kw)
 | 
			
		||||
      File "test.py", line 5, in bug
 | 
			
		||||
      File "test.py", line 4, in bug
 | 
			
		||||
        raise Exception("not consumed")
 | 
			
		||||
    Exception: not consumed
 | 
			
		||||
 | 
			
		||||
:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the
 | 
			
		||||
traceback where the task was created. Output in debug mode::
 | 
			
		||||
:ref:`Enable the debug mode <asyncio-debug-mode>` to get the
 | 
			
		||||
traceback where the task was created::
 | 
			
		||||
 | 
			
		||||
    asyncio.run(main(), debug=True)
 | 
			
		||||
 | 
			
		||||
Output in debug mode::
 | 
			
		||||
 | 
			
		||||
    Task exception was never retrieved
 | 
			
		||||
    future: <Task finished coro=<bug() done, defined at test.py:3> exception=Exception('not consumed',) created at test.py:8>
 | 
			
		||||
    future: <Task finished coro=<bug() done, defined at test.py:3>
 | 
			
		||||
        exception=Exception('not consumed') created at asyncio/tasks.py:321>
 | 
			
		||||
 | 
			
		||||
    source_traceback: Object created at (most recent call last):
 | 
			
		||||
      File "test.py", line 8, in <module>
 | 
			
		||||
        asyncio.ensure_future(bug())
 | 
			
		||||
      File "../t.py", line 9, in <module>
 | 
			
		||||
        asyncio.run(main(), debug=True)
 | 
			
		||||
 | 
			
		||||
    < .. >
 | 
			
		||||
 | 
			
		||||
    Traceback (most recent call last):
 | 
			
		||||
      File "asyncio/tasks.py", line 237, in _step
 | 
			
		||||
        result = next(coro)
 | 
			
		||||
      File "asyncio/coroutines.py", line 79, in __next__
 | 
			
		||||
        return next(self.gen)
 | 
			
		||||
      File "asyncio/coroutines.py", line 141, in coro
 | 
			
		||||
        res = func(*args, **kw)
 | 
			
		||||
      File "test.py", line 5, in bug
 | 
			
		||||
      File "../t.py", line 4, in bug
 | 
			
		||||
        raise Exception("not consumed")
 | 
			
		||||
    Exception: not consumed
 | 
			
		||||
 | 
			
		||||
There are different options to fix this issue. The first option is to chain the
 | 
			
		||||
coroutine in another coroutine and use classic try/except::
 | 
			
		||||
 | 
			
		||||
    async def handle_exception():
 | 
			
		||||
        try:
 | 
			
		||||
            await bug()
 | 
			
		||||
        except Exception:
 | 
			
		||||
            print("exception consumed")
 | 
			
		||||
 | 
			
		||||
    loop = asyncio.get_event_loop()
 | 
			
		||||
    asyncio.ensure_future(handle_exception())
 | 
			
		||||
    loop.run_forever()
 | 
			
		||||
    loop.close()
 | 
			
		||||
 | 
			
		||||
Another option is to use the :meth:`asyncio.run` function::
 | 
			
		||||
 | 
			
		||||
    asyncio.run(bug())
 | 
			
		||||
 | 
			
		||||
.. seealso::
 | 
			
		||||
 | 
			
		||||
   The :meth:`Future.exception` method.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
Chain coroutines correctly
 | 
			
		||||
--------------------------
 | 
			
		||||
 | 
			
		||||
When a coroutine function calls other coroutine functions and tasks, they
 | 
			
		||||
should be chained explicitly with ``await``. Otherwise, the execution is
 | 
			
		||||
not guaranteed to be sequential.
 | 
			
		||||
 | 
			
		||||
Example with different bugs using :func:`asyncio.sleep` to simulate slow
 | 
			
		||||
operations::
 | 
			
		||||
 | 
			
		||||
    import asyncio
 | 
			
		||||
 | 
			
		||||
    async def create():
 | 
			
		||||
        await asyncio.sleep(3.0)
 | 
			
		||||
        print("(1) create file")
 | 
			
		||||
 | 
			
		||||
    async def write():
 | 
			
		||||
        await asyncio.sleep(1.0)
 | 
			
		||||
        print("(2) write into file")
 | 
			
		||||
 | 
			
		||||
    async def close():
 | 
			
		||||
        print("(3) close file")
 | 
			
		||||
 | 
			
		||||
    async def test():
 | 
			
		||||
        asyncio.ensure_future(create())
 | 
			
		||||
        asyncio.ensure_future(write())
 | 
			
		||||
        asyncio.ensure_future(close())
 | 
			
		||||
        await asyncio.sleep(2.0)
 | 
			
		||||
        loop.stop()
 | 
			
		||||
 | 
			
		||||
    loop = asyncio.get_event_loop()
 | 
			
		||||
    asyncio.ensure_future(test())
 | 
			
		||||
    loop.run_forever()
 | 
			
		||||
    print("Pending tasks at exit: %s" % asyncio.Task.all_tasks(loop))
 | 
			
		||||
    loop.close()
 | 
			
		||||
 | 
			
		||||
Expected output:
 | 
			
		||||
 | 
			
		||||
.. code-block:: none
 | 
			
		||||
 | 
			
		||||
    (1) create file
 | 
			
		||||
    (2) write into file
 | 
			
		||||
    (3) close file
 | 
			
		||||
    Pending tasks at exit: set()
 | 
			
		||||
 | 
			
		||||
Actual output:
 | 
			
		||||
 | 
			
		||||
.. code-block:: none
 | 
			
		||||
 | 
			
		||||
    (3) close file
 | 
			
		||||
    (2) write into file
 | 
			
		||||
    Pending tasks at exit: {<Task pending create() at test.py:7 wait_for=<Future pending cb=[Task._wakeup()]>>}
 | 
			
		||||
    Task was destroyed but it is pending!
 | 
			
		||||
    task: <Task pending create() done at test.py:5 wait_for=<Future pending cb=[Task._wakeup()]>>
 | 
			
		||||
 | 
			
		||||
The loop stopped before the ``create()`` finished, ``close()`` has been called
 | 
			
		||||
before ``write()``, whereas coroutine functions were called in this order:
 | 
			
		||||
``create()``, ``write()``, ``close()``.
 | 
			
		||||
 | 
			
		||||
To fix the example, tasks must be marked with ``await``::
 | 
			
		||||
 | 
			
		||||
    async def test():
 | 
			
		||||
        await asyncio.ensure_future(create())
 | 
			
		||||
        await asyncio.ensure_future(write())
 | 
			
		||||
        await asyncio.ensure_future(close())
 | 
			
		||||
        await asyncio.sleep(2.0)
 | 
			
		||||
        loop.stop()
 | 
			
		||||
 | 
			
		||||
Or without ``asyncio.ensure_future()``::
 | 
			
		||||
 | 
			
		||||
    async def test():
 | 
			
		||||
        await create()
 | 
			
		||||
        await write()
 | 
			
		||||
        await close()
 | 
			
		||||
        await asyncio.sleep(2.0)
 | 
			
		||||
        loop.stop()
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _asyncio-pending-task-destroyed:
 | 
			
		||||
 | 
			
		||||
Pending task destroyed
 | 
			
		||||
----------------------
 | 
			
		||||
 | 
			
		||||
If a pending task is destroyed, the execution of its wrapped :ref:`coroutine
 | 
			
		||||
<coroutine>` did not complete. It is probably a bug and so a warning is logged.
 | 
			
		||||
 | 
			
		||||
Example of log:
 | 
			
		||||
 | 
			
		||||
.. code-block:: none
 | 
			
		||||
 | 
			
		||||
    Task was destroyed but it is pending!
 | 
			
		||||
    task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()]>>
 | 
			
		||||
 | 
			
		||||
:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the
 | 
			
		||||
traceback where the task was created. Example of log in debug mode:
 | 
			
		||||
 | 
			
		||||
.. code-block:: none
 | 
			
		||||
 | 
			
		||||
    Task was destroyed but it is pending!
 | 
			
		||||
    source_traceback: Object created at (most recent call last):
 | 
			
		||||
      File "test.py", line 15, in <module>
 | 
			
		||||
        task = asyncio.ensure_future(coro, loop=loop)
 | 
			
		||||
    task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()] created at test.py:7> created at test.py:15>
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. seealso::
 | 
			
		||||
 | 
			
		||||
   :ref:`Detect coroutine objects never scheduled <asyncio-coroutine-not-scheduled>`.
 | 
			
		||||
 | 
			
		||||
.. _asyncio-close-transports:
 | 
			
		||||
 | 
			
		||||
Close transports and event loops
 | 
			
		||||
--------------------------------
 | 
			
		||||
 | 
			
		||||
When a transport is no longer needed, call its ``close()`` method to release
 | 
			
		||||
resources. Event loops must also be closed explicitly.
 | 
			
		||||
 | 
			
		||||
If a transport or an event loop is not closed explicitly, a
 | 
			
		||||
:exc:`ResourceWarning` warning will be emitted in its destructor. By default,
 | 
			
		||||
:exc:`ResourceWarning` warnings are ignored. The :ref:`Debug mode of asyncio
 | 
			
		||||
<asyncio-debug-mode>` section explains how to display them.
 | 
			
		||||
| 
						 | 
				
			
			
 | 
			
		|||
| 
						 | 
				
			
			@ -1075,6 +1075,11 @@ Enabling debug mode
 | 
			
		|||
 | 
			
		||||
   Set the debug mode of the event loop.
 | 
			
		||||
 | 
			
		||||
   .. versionchanged:: 3.7
 | 
			
		||||
 | 
			
		||||
      The new ``-X dev`` command line option can now also be used
 | 
			
		||||
      to enable the debug mode.
 | 
			
		||||
 | 
			
		||||
.. seealso::
 | 
			
		||||
 | 
			
		||||
   The :ref:`debug mode of asyncio <asyncio-debug-mode>`.
 | 
			
		||||
| 
						 | 
				
			
			
 | 
			
		|||
| 
						 | 
				
			
			@ -121,6 +121,16 @@ Future Object
 | 
			
		|||
      or an exception set with :meth:`set_result` or
 | 
			
		||||
      :meth:`set_exception` calls.
 | 
			
		||||
 | 
			
		||||
   .. method:: cancelled()
 | 
			
		||||
 | 
			
		||||
      Return ``True`` if the Future was *cancelled*.
 | 
			
		||||
 | 
			
		||||
      The method is usually used to check if a Future is not
 | 
			
		||||
      *cancelled* before setting a result or an exception for it::
 | 
			
		||||
 | 
			
		||||
          if not fut.cancelled():
 | 
			
		||||
              fut.set_result(42)
 | 
			
		||||
 | 
			
		||||
   .. method:: add_done_callback(callback, *, context=None)
 | 
			
		||||
 | 
			
		||||
      Add a callback to be run when the Future is *done*.
 | 
			
		||||
| 
						 | 
				
			
			@ -180,10 +190,6 @@ Future Object
 | 
			
		|||
 | 
			
		||||
      .. versionadded:: 3.7
 | 
			
		||||
 | 
			
		||||
   .. method:: cancelled()
 | 
			
		||||
 | 
			
		||||
      Return ``True`` if the Future was *cancelled*.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
This example creates a Future object, creates and schedules an
 | 
			
		||||
asynchronous Task to set result for the Future, and waits until
 | 
			
		||||
| 
						 | 
				
			
			
 | 
			
		|||
| 
						 | 
				
			
			@ -32,8 +32,9 @@ asyncio provides a set of **high-level** APIs to:
 | 
			
		|||
as well as **low-level** APIs for *library and framework developers* to:
 | 
			
		||||
 | 
			
		||||
* create and manage :ref:`event loops <asyncio-event-loop>`, which
 | 
			
		||||
  provide asynchronous APIs for networking, subprocesses, OS signals,
 | 
			
		||||
  etc;
 | 
			
		||||
  provide asynchronous APIs for :meth:`networking <loop.create_server>`,
 | 
			
		||||
  running :meth:`subprocesses <loop.subprocess_exec>`,
 | 
			
		||||
  handling :meth:`OS signals <loop.add_signal_handler>`, etc;
 | 
			
		||||
 | 
			
		||||
* implement efficient protocols using
 | 
			
		||||
  :ref:`transports <asyncio-transports-protocols>`;
 | 
			
		||||
| 
						 | 
				
			
			@ -75,11 +76,3 @@ Reference
 | 
			
		|||
 | 
			
		||||
   asyncio-api-index.rst
 | 
			
		||||
   asyncio-dev.rst
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. seealso::
 | 
			
		||||
 | 
			
		||||
   The :mod:`asyncio` module was proposed in :PEP:`3156`.
 | 
			
		||||
   Since the acceptance of the PEP many new APIs were added and many
 | 
			
		||||
   original APIs were altered.  The PEP should be treated as a
 | 
			
		||||
   historical document.
 | 
			
		||||
| 
						 | 
				
			
			
 | 
			
		|||
| 
						 | 
				
			
			@ -1,11 +1,11 @@
 | 
			
		|||
.. _ipc:
 | 
			
		||||
 | 
			
		||||
*****************************************
 | 
			
		||||
Interprocess Communication and Networking
 | 
			
		||||
Networking and Interprocess Communication
 | 
			
		||||
*****************************************
 | 
			
		||||
 | 
			
		||||
The modules described in this chapter provide mechanisms for different processes
 | 
			
		||||
to communicate.
 | 
			
		||||
The modules described in this chapter provide mechanisms for
 | 
			
		||||
networking and inter-processes communication.
 | 
			
		||||
 | 
			
		||||
Some modules only work for two processes that are on the same machine, e.g.
 | 
			
		||||
:mod:`signal` and :mod:`mmap`.  Other modules support networking protocols
 | 
			
		||||
| 
						 | 
				
			
			@ -15,6 +15,7 @@ The list of modules described in this chapter is:
 | 
			
		|||
 | 
			
		||||
 | 
			
		||||
.. toctree::
 | 
			
		||||
   :maxdepth: 1
 | 
			
		||||
 | 
			
		||||
   asyncio.rst
 | 
			
		||||
   socket.rst
 | 
			
		||||
| 
						 | 
				
			
			
 | 
			
		|||
		Loading…
	
	Add table
		Add a link
		
	
		Reference in a new issue