Fixed #31224 -- Added support for asynchronous views and middleware.

This implements support for asynchronous views, asynchronous tests,
asynchronous middleware, and an asynchronous test client.

docs/index.txt

@@ -110,8 +110,7 @@ manipulating the data of your Web application. Learn more about it below:
   :doc:`Custom lookups <howto/custom-lookups>` |
   :doc:`Query Expressions <ref/models/expressions>` |
   :doc:`Conditional Expressions <ref/models/conditional-expressions>` |
-  :doc:`Database Functions <ref/models/database-functions>` |
-  :doc:`Asynchronous Support <topics/async>`
+  :doc:`Database Functions <ref/models/database-functions>`
 
 * **Other:**
   :doc:`Supported databases <ref/databases>` |
@@ -131,7 +130,8 @@ to know about views via the links below:
   :doc:`URLconfs <topics/http/urls>` |
   :doc:`View functions <topics/http/views>` |
   :doc:`Shortcuts <topics/http/shortcuts>` |
-  :doc:`Decorators <topics/http/decorators>`
+  :doc:`Decorators <topics/http/decorators>` |
+  :doc:`Asynchronous Support <topics/async>`
 
 * **Reference:**
   :doc:`Built-in Views <ref/views>` |

docs/ref/utils.txt

@@ -210,6 +210,31 @@ The functions defined in this module share the following properties:
          def my_view(request):
              pass
 
+.. function:: sync_only_middleware(middleware)
+
+    .. versionadded:: 3.1
+
+    Marks a middleware as :ref:`synchronous-only <async-middleware>`. (The
+    default in Django, but this allows you to future-proof if the default ever
+    changes in a future release.)
+
+.. function:: async_only_middleware(middleware)
+
+    .. versionadded:: 3.1
+
+    Marks a middleware as :ref:`asynchronous-only <async-middleware>`. Django
+    will wrap it in an asynchronous event loop when it is called from the WSGI
+    request path.
+
+.. function:: sync_and_async_middleware(middleware)
+
+    .. versionadded:: 3.1
+
+    Marks a middleware as :ref:`sync and async compatible <async-middleware>`,
+    this allows to avoid converting requests. You must implement detection of
+    the current request type to use this decorator. See :ref:`asynchronous
+    middleware documentation <async-middleware>` for details.
+
 ``django.utils.encoding``
 =========================
 

docs/releases/3.1.txt

@@ -27,6 +27,43 @@ officially support the latest release of each series.
 What's new in Django 3.1
 ========================
 
+Asynchronous views and middleware support
+-----------------------------------------
+
+Django now supports a fully asynchronous request path, including:
+
+* :ref:`Asynchronous views <async-views>`
+* :ref:`Asynchronous middleware <async-middleware>`
+* :ref:`Asynchronous tests and test client <async-tests>`
+
+To get started with async views, you need to declare a view using
+``async def``::
+
+    async def my_view(request):
+        await asyncio.sleep(0.5)
+        return HttpResponse('Hello, async world!')
+
+All asynchronous features are supported whether you are running under WSGI or
+ASGI mode. However, there will be performance penalties using async code in
+WSGI mode. You can read more about the specifics in :doc:`/topics/async`
+documentation.
+
+You are free to mix async and sync views, middleware, and tests as much as you
+want. Django will ensure that you always end up with the right execution
+context. We expect most projects will keep the majority of their views
+synchronous, and only have a select few running in async mode - but it is
+entirely your choice.
+
+Django's ORM, cache layer, and other pieces of code that do long-running
+network calls do not yet support async access. We expect to add support for
+them in upcoming releases. Async views are ideal, however, if you are doing a
+lot of API or HTTP calls inside your view, you can now natively do all those
+HTTP calls in parallel to considerably speed up your view's execution.
+
+Asynchronous support should be entirely backwards-compatible and we have tried
+to ensure that it has no speed regressions for your existing, synchronous code.
+It should have no noticeable effect on any existing Django projects.
+
 Minor features
 --------------
 

docs/spelling_wordlist

@@ -144,6 +144,7 @@ databrowse
 datafile
 dataset
 datasets
+datastores
 datatype
 datetimes
 Debian

docs/topics/async.txt

@@ -6,13 +6,106 @@ Asynchronous support
 
 .. currentmodule:: asgiref.sync
 
-Django has developing support for asynchronous ("async") Python, but does not
-yet support asynchronous views or middleware; they will be coming in a future
-release.
+Django has support for writing asynchronous ("async") views, along with an
+entirely async-enabled request stack if you are running under
+:doc:`ASGI </howto/deployment/asgi/index>` rather than WSGI. Async views will
+still work under WSGI, but with performance penalties, and without the ability
+to have efficient long-running requests.
 
-There is limited support for other parts of the async ecosystem; namely, Django
-can natively talk :doc:`ASGI </howto/deployment/asgi/index>`, and some async
-safety support.
+We're still working on asynchronous support for the ORM and other parts of
+Django; you can expect to see these in future releases. For now, you can use
+the :func:`sync_to_async` adapter to interact with normal Django, as well as
+use a whole range of Python asyncio libraries natively. See below for more
+details.
+
+.. versionchanged:: 3.1
+
+    Support for async views was added.
+
+Async views
+===========
+
+.. versionadded:: 3.1
+
+Any view can be declared async by making the callable part of it return a
+coroutine - commonly, this is done using ``async def``. For a function-based
+view, this means declaring the whole view using ``async def``. For a
+class-based view, this means making its ``__call__()`` method an ``async def``
+(not its ``__init__()`` or ``as_view()``).
+
+.. note::
+
+    Django uses ``asyncio.iscoroutinefunction`` to test if your view is
+    asynchronous or not. If you implement your own method of returning a
+    coroutine, ensure you set the ``_is_coroutine`` attribute of the view
+    to ``asyncio.coroutines._is_coroutine`` so this function returns ``True``.
+
+Under a WSGI server, asynchronous views will run in their own, one-off event
+loop. This means that you can do things like parallel, async HTTP calls to APIs
+without any issues, but you will not get the benefits of an asynchronous
+request stack.
+
+If you want these benefits - which are mostly around the ability to service
+hundreds of connections without using any Python threads (enabling slow
+streaming, long-polling, and other exciting response types) - you will need to
+deploy Django using :doc:`ASGI </howto/deployment/asgi/index>` instead.
+
+.. warning::
+
+    You will only get the benefits of a fully-asynchronous request stack if you
+    have *no synchronous middleware* loaded into your site; if there is a piece
+    of synchronous middleware, then Django must use a thread per request to
+    safely emulate a synchronous environment for it.
+
+    Middleware can be built to support :ref:`both sync and async
+    <async-middleware>` contexts. Some of Django's middleware is built like
+    this, but not all. To see what middleware Django has to adapt, you can turn
+    on debug logging for the ``django.request`` logger and look for log
+    messages about *`"Synchronous middleware ... adapted"*.
+
+In either ASGI or WSGI mode, though, you can safely use asynchronous support to
+run code in parallel rather than serially, which is especially handy when
+dealing with external APIs or datastores.
+
+If you want to call a part of Django that is still synchronous (like the ORM)
+you will need to wrap it in a :func:`sync_to_async` call, like this::
+
+    from asgiref.sync import sync_to_async
+
+    results = sync_to_async(MyModel.objects.get)(pk=123)
+
+You may find it easier to move any ORM code into its own function and call that
+entire function using :func:`sync_to_async`. If you accidentally try to call
+part of Django that is still synchronous-only from an async view, you will
+trigger Django's :ref:`asynchronous safety protection <async-safety>` to
+protect your data from corruption.
+
+Performance
+-----------
+
+When running in a mode that does not match the view (e.g. an async view under
+WSGI, or a traditional sync view under ASGI), Django must emulate the other
+call style to allow your code to run. This context-switch causes a small
+performance penalty of around a millisecond.
+
+This is true of middleware as well, however. Django will attempt to minimize
+the number of context-switches. If you have an ASGI server, but all your
+middleware and views are synchronous, it will switch just once, before it
+enters the middleware stack.
+
+If, however, you put synchronous middleware between an ASGI server and an
+asynchronous view, it will have to switch into sync mode for the middleware and
+then back to asynchronous mode for the view, holding the synchronous thread
+open for middleware exception propagation. This may not be noticeable, but bear
+in mind that even adding a single piece of synchronous middleware can drag your
+whole async project down to running with one thread per request, and the
+associated performance penalties.
+
+You should do your own performance testing to see what effect ASGI vs. WSGI has
+on your code. In some cases, there may be a performance increase even for
+purely-synchronous codebase under ASGI because the request-handling code is
+still all running asynchronously. In general, though, you will only want to
+enable ASGI mode if you have asynchronous code in your site.
 
 .. _async-safety:
 

docs/topics/http/middleware.txt

@@ -71,6 +71,10 @@ method from the handler which takes care of applying :ref:`view middleware
 applying :ref:`template-response <template-response-middleware>` and
 :ref:`exception <exception-middleware>` middleware.
 
+Middleware can either support only synchronous Python (the default), only
+asynchronous Python, or both. See :ref:`async-middleware` for details of how to
+advertise what you support, and know what kind of request you are getting.
+
 Middleware can live anywhere on your Python path.
 
 ``__init__(get_response)``
@@ -282,6 +286,81 @@ if the very next middleware in the chain raises an
 that exception; instead it will get an :class:`~django.http.HttpResponse`
 object with a :attr:`~django.http.HttpResponse.status_code` of 404.
 
+.. _async-middleware:
+
+Asynchronous support
+====================
+
+.. versionadded:: 3.1
+
+Middleware can support any combination of synchronous and asynchronous
+requests. Django will adapt requests to fit the middleware's requirements if it
+cannot support both, but at a performance penalty.
+
+By default, Django assumes that your middleware is capable of handling only
+synchronous requests. To change these assumptions, set the following attributes
+on your middleware factory function or class:
+
+* ``sync_capable`` is a boolean indicating if the middleware can handle
+  synchronous requests. Defaults to ``True``.
+
+* ``async_capable`` is a boolean indicating if the middleware can handle
+  asynchronous requests. Defaults to ``False``.
+
+If your middleware has both ``sync_capable = True`` and
+``async_capable = True``, then Django will pass it the request in whatever form
+it is currently in. You can work out what type of request you have by seeing
+if the ``get_response`` object you are passed is a coroutine function or not
+(using :py:func:`asyncio.iscoroutinefunction`).
+
+The ``django.utils.decorators`` module contains
+:func:`~django.utils.decorators.sync_only_middleware`,
+:func:`~django.utils.decorators.async_only_middleware`, and
+:func:`~django.utils.decorators.sync_and_async_middleware` decorators that
+allow you to apply these flags to middleware factory functions.
+
+The returned callable must match the sync or async nature of the
+``get_response`` method. If you have an asynchronous ``get_response``, you must
+return a coroutine function (``async def``).
+
+``process_view``, ``process_template_response`` and ``process_exception``
+methods, if they are provided, should also be adapted to match the sync/async
+mode. However, Django will individually adapt them as required if you do not,
+at an additional performance penalty.
+
+Here's an example of how to detect and adapt your middleware if it supports
+both::
+
+    import asyncio
+    from django.utils.decorators import sync_and_async_middleware
+
+    @sync_and_async_middleware
+    def simple_middleware(get_response):
+        # One-time configuration and initialization goes here.
+        if asyncio.iscoroutinefunction(get_response):
+            async def middleware(request):
+                # Do something here!
+                response = await get_response(request)
+                return response
+
+        else:
+            def middleware(request):
+                # Do something here!
+                response = get_response(request)
+                return response
+
+        return middleware
+
+.. note::
+
+    If you declare a hybrid middleware that supports both synchronous and
+    asynchronous calls, the kind of call you get may not match the underlying
+    view. Django will optimize the middleware call stack to have as few
+    sync/async transitions as possible.
+
+    Thus, even if you are wrapping an async view, you may be called in sync
+    mode if there is other, synchronous middleware between you and the view.
+
 .. _upgrading-middleware:
 
 Upgrading pre-Django 1.10-style middleware
@@ -292,8 +371,8 @@ Upgrading pre-Django 1.10-style middleware
 
 Django provides ``django.utils.deprecation.MiddlewareMixin`` to ease creating
 middleware classes that are compatible with both :setting:`MIDDLEWARE` and the
-old ``MIDDLEWARE_CLASSES``. All middleware classes included with Django
-are compatible with both settings.
+old ``MIDDLEWARE_CLASSES``, and support synchronous and asynchronous requests.
+All middleware classes included with Django are compatible with both settings.
 
 The mixin provides an ``__init__()`` method that requires a ``get_response``
 argument and stores it in ``self.get_response``.
@@ -345,3 +424,7 @@ These are the behavioral differences between using :setting:`MIDDLEWARE` and
    HTTP response, and then the next middleware in line will see that
    response. Middleware are never skipped due to a middleware raising an
    exception.
+
+.. versionchanged:: 3.1
+
+    Support for asynchronous requests was added to the ``MiddlewareMixin``.

docs/topics/http/views.txt

@@ -202,3 +202,28 @@ in a test view. For example::
             response = self.client.get('/403/')
             # Make assertions on the response here. For example:
             self.assertContains(response, 'Error handler content', status_code=403)
+
+.. _async-views:
+
+Asynchronous views
+==================
+
+.. versionadded:: 3.1
+
+As well as being synchronous functions, views can also be asynchronous
+functions (``async def``). Django will automatically detect these and run them
+in an asynchronous context. You will need to be using an asynchronous (ASGI)
+server to get the full power of them, however.
+
+Here's an example of an asynchronous view::
+
+    from django.http import HttpResponse
+    import datetime
+
+    async def current_datetime(request):
+        now = datetime.datetime.now()
+        html = '<html><body>It is now %s.</body></html>' % now
+        return HttpResponse(html)
+
+You can read more about Django's asynchronous support, and how to best use
+asynchronous views, in :doc:`/topics/async`.

docs/topics/testing/advanced.txt

@@ -67,6 +67,17 @@ The following is a unit test using the request factory::
             response = MyView.as_view()(request)
             self.assertEqual(response.status_code, 200)
 
+AsyncRequestFactory
+-------------------
+
+``RequestFactory`` creates WSGI-like requests. If you want to create ASGI-like
+requests, including having a correct ASGI ``scope``, you can instead use
+``django.test.AsyncRequestFactory``.
+
+This class is directly API-compatible with ``RequestFactory``, with the only
+difference being that it returns ``ASGIRequest`` instances rather than
+``WSGIRequest`` instances. All of its methods are still synchronous callables.
+
 Testing class-based views
 =========================
 

docs/topics/testing/tools.txt

@@ -1755,6 +1755,62 @@ You can also exclude tests by tag. To run core tests if they are not slow:
 test has two tags and you select one of them and exclude the other, the test
 won't be run.
 
+.. _async-tests:
+
+Testing asynchronous code
+=========================
+
+.. versionadded:: 3.1
+
+If you merely want to test the output of your asynchronous views, the standard
+test client will run them inside their own asynchronous loop without any extra
+work needed on your part.
+
+However, if you want to write fully-asynchronous tests for a Django project,
+you will need to take several things into account.
+
+Firstly, your tests must be ``async def`` methods on the test class (in order
+to give them an asynchronous context). Django will automatically detect
+any ``async def`` tests and wrap them so they run in their own event loop.
+
+If you are testing from an asynchronous function, you must also use the
+asynchronous test client. This is available as ``django.test.AsyncClient``,
+or as ``self.async_client`` on any test.
+
+With the exception of the ``follow`` parameter, which is not supported,
+``AsyncClient`` has the same methods and signatures as the synchronous (normal)
+test client, but any method that makes a request must be awaited::
+
+    async def test_my_thing(self):
+        response = await self.async_client.get('/some-url/')
+        self.assertEqual(response.status_code, 200)
+
+The asynchronous client can also call synchronous views; it runs through
+Django's :doc:`asynchronous request path </topics/async>`, which supports both.
+Any view called through the ``AsyncClient`` will get an ``ASGIRequest`` object
+for its ``request`` rather than the ``WSGIRequest`` that the normal client
+creates.
+
+.. warning::
+
+    If you are using test decorators, they must be async-compatible to ensure
+    they work correctly. Django's built-in decorators will behave correctly, but
+    third-party ones may appear to not execute (they will "wrap" the wrong part
+    of the execution flow and not your test).
+
+    If you need to use these decorators, then you should decorate your test
+    methods with :func:`~asgiref.sync.async_to_sync` *inside* of them instead::
+
+        from asgiref.sync import async_to_sync
+        from django.test import TestCase
+
+        class MyTests(TestCase):
+
+            @mock.patch(...)
+            @async_to_sync
+            def test_my_thing(self):
+                ...
+
 .. _topics-testing-email:
 
 Email services