Fixed #20869 -- made CSRF tokens change every request by salt-encrypting them

Note that the cookie is not changed every request, just the token retrieved
by the `get_token()` method (used also by the `{% csrf_token %}` tag).

While at it, made token validation strict: Where, before, any length was
accepted and non-ASCII chars were ignored, we now treat anything other than
`[A-Za-z0-9]{64}` as invalid (except for 32-char tokens, which, for
backwards-compatibility, are accepted and replaced by 64-char ones).

Thanks Trac user patrys for reporting, github user adambrenecki
for initial patch, Tim Graham for help, and Curtis Maloney,
Collin Anderson, Florian Apolloner, Markus Holtermann & Jon Dufresne
for reviews.

docs/ref/csrf.txt

@@ -218,20 +218,25 @@ How it works
 
 The CSRF protection is based on the following things:
 
-1. A CSRF cookie that is set to a random value (a session independent nonce, as
-   it is called), which other sites will not have access to.
+1. A CSRF cookie that is based on a random secret value, which other sites
+   will not have access to.
 
-   This cookie is set by ``CsrfViewMiddleware``.  It is meant to be permanent,
-   but since there is no way to set a cookie that never expires, it is sent with
-   every response that has called ``django.middleware.csrf.get_token()``
-   (the function used internally to retrieve the CSRF token).
+   This cookie is set by ``CsrfViewMiddleware``. It is sent with every
+   response that has called ``django.middleware.csrf.get_token()`` (the
+   function used internally to retrieve the CSRF token), if it wasn't already
+   set on the request.
 
-   For security reasons, the value of the CSRF cookie is changed each time a
+   In order to protect against `BREACH`_ attacks, the token is not simply the
+   secret; a random salt is prepended to the secret and used to scramble it.
+
+   For security reasons, the value of the secret is changed each time a
    user logs in.
 
 2. A hidden form field with the name 'csrfmiddlewaretoken' present in all
-   outgoing POST forms.  The value of this field is the value of the CSRF
-   cookie.
+   outgoing POST forms. The value of this field is, again, the value of the
+   secret, with a salt which is both added to it and used to scramble it. The
+   salt is regenerated on every call to ``get_token()`` so that the form field
+   value is changed in every such response.
 
    This part is done by the template tag.
 
@@ -239,6 +244,11 @@ The CSRF protection is based on the following things:
    TRACE, a CSRF cookie must be present, and the 'csrfmiddlewaretoken' field
    must be present and correct. If it isn't, the user will get a 403 error.
 
+   When validating the 'csrfmiddlewaretoken' field value, only the secret,
+   not the full token, is compared with the secret in the cookie value.
+   This allows the use of ever-changing tokens. While each request may use its
+   own token, the secret remains common to all.
+
    This check is done by ``CsrfViewMiddleware``.
 
 4. In addition, for HTTPS requests, strict referer checking is done by
@@ -247,7 +257,7 @@ The CSRF protection is based on the following things:
    application since that request won't come from your own exact domain.
 
    This also addresses a man-in-the-middle attack that's possible under HTTPS
-   when using a session independent nonce, due to the fact that HTTP
+   when using a session independent secret, due to the fact that HTTP
    ``Set-Cookie`` headers are (unfortunately) accepted by clients even when
    they are talking to a site under HTTPS. (Referer checking is not done for
    HTTP requests because the presence of the ``Referer`` header isn't reliable
@@ -283,6 +293,13 @@ vulnerability allows and much worse).
 
     Checking against the :setting:`CSRF_COOKIE_DOMAIN` setting was added.
 
+.. versionchanged:: 1.10
+
+   Added salting to the token and started changing it with each request
+   to protect against `BREACH`_ attacks.
+
+.. _BREACH: http://breachattack.com/
+
 Caching
 =======
 
@@ -499,15 +516,6 @@ No, this is by design. Not linking CSRF protection to a session allows using
 the protection on sites such as a `pastebin` that allow submissions from
 anonymous users which don't have a session.
 
-Why not use a new token for each request?
------------------------------------------
-
-Generating a new token for each request is problematic from a UI perspective
-because it invalidates all previous forms. Most users would be very unhappy to
-find that opening a new tab on your site has invalidated the form they had
-just spent time filling out in another tab or that a form they accessed via
-the back button could not be filled out.
-
 Why might a user encounter a CSRF validation failure after logging in?
 ----------------------------------------------------------------------
 

docs/ref/middleware.txt

@@ -118,13 +118,12 @@ GZip middleware
 .. warning::
 
     Security researchers recently revealed that when compression techniques
-    (including ``GZipMiddleware``) are used on a website, the site becomes
-    exposed to a number of possible attacks. These approaches can be used to
-    compromise, among other things, Django's CSRF protection. Before using
-    ``GZipMiddleware`` on your site, you should consider very carefully whether
-    you are subject to these attacks. If you're in *any* doubt about whether
-    you're affected, you should avoid using ``GZipMiddleware``. For more
-    details, see the `the BREACH paper (PDF)`_ and `breachattack.com`_.
+    (including ``GZipMiddleware``) are used on a website, the site may become
+    exposed to a number of possible attacks. Before using ``GZipMiddleware`` on
+    your site, you should consider very carefully whether you are subject to
+    these attacks. If you're in *any* doubt about whether you're affected, you
+    should avoid using ``GZipMiddleware``. For more details, see the `the BREACH
+    paper (PDF)`_ and `breachattack.com`_.
 
     .. _the BREACH paper (PDF): http://breachattack.com/resources/BREACH%20-%20SSL,%20gone%20in%2030%20seconds.pdf
     .. _breachattack.com: http://breachattack.com
@@ -147,6 +146,12 @@ It will NOT compress content if any of the following are true:
 You can apply GZip compression to individual views using the
 :func:`~django.views.decorators.gzip.gzip_page()` decorator.
 
+.. versionchanged:: 1.10
+
+    In older versions, Django's CSRF protection mechanism was vulnerable to
+    BREACH attacks when compression was used. This is no longer the case, but
+    you should still take care not to compromise your own secrets this way.
+
 Conditional GET middleware
 --------------------------