mirrors/django
1
0
Fork 0
mirror of https://github.com/django/django.git synced 2025-07-24 05:36:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

Fixed #12990, Refs #27694 -- Added JSONField model field.

Browse source 
Thanks to Adam Johnson, Carlton Gibson, Mariusz Felisiak, and Raphael
Michel for mentoring this Google Summer of Code 2019 project and
everyone else who helped with the patch.

Special thanks to Mads Jensen, Nick Pope, and Simon Charette for
extensive reviews.

Co-authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com>
This commit is contained in:
sage 2019-06-09 07:56:37 +07:00 • committed by Mariusz Felisiak
parent f97f71f592
commit 6789ded0a6
54 changed files with 2240 additions and 981 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/internals/deprecation.txt
View file

@ -83,6 +83,13 @@ details on these changes.
* ``django.conf.urls.url()`` will be removed.
* The model ``django.contrib.postgres.fields.JSONField`` will be removed. A
stub field will remain for compatibility with historical migrations.
* ``django.contrib.postgres.forms.JSONField``,
``django.contrib.postgres.fields.jsonb.KeyTransform``, and
``django.contrib.postgres.fields.jsonb.KeyTextTransform`` will be removed.
See the :ref:`Django 3.1 release notes <deprecated-features-3.1>` for more
details on these changes.

4
docs/ref/checks.txt
View file

@ -190,6 +190,7 @@ Model fields
``<field data type>`` columns.
* **fields.E170**: ``BinaryField``’s ``default`` cannot be a string. Use bytes
content instead.
* **fields.E180**: ``<database>`` does not support ``JSONField``\s.
* **fields.E900**: ``IPAddressField`` has been removed except for support in
historical migrations.
* **fields.W900**: ``IPAddressField`` has been deprecated. Support for it
@ -204,6 +205,9 @@ Model fields
Django 3.1. *This check appeared in Django 2.2 and 3.0*.
* **fields.W903**: ``NullBooleanField`` is deprecated. Support for it (except
in historical migrations) will be removed in Django 4.0.
* **fields.W904**: ``django.contrib.postgres.fields.JSONField`` is deprecated.
Support for it (except in historical migrations) will be removed in Django
4.0.
File fields
~~~~~~~~~~~

95
docs/ref/contrib/postgres/fields.txt
View file

@ -16,8 +16,7 @@ Indexes such as :class:`~django.contrib.postgres.indexes.GinIndex` and
:class:`~django.contrib.postgres.indexes.GistIndex` are better suited, though
the index choice is dependent on the queries that you're using. Generally, GiST
may be a good choice for the :ref:`range fields <range-fields>` and
:class:`HStoreField`, and GIN may be helpful for :class:`ArrayField` and
:class:`JSONField`.
:class:`HStoreField`, and GIN may be helpful for :class:`ArrayField`.
``ArrayField``
==============
@ -517,96 +516,14 @@ using in conjunction with lookups on
of the JSON which allows indexing. The trade-off is a small additional cost
on writing to the ``jsonb`` field. ``JSONField`` uses ``jsonb``.
.. deprecated:: 3.1
Use :class:`django.db.models.JSONField` instead.
Querying ``JSONField``
----------------------
We will use the following example model::
from django.contrib.postgres.fields import JSONField
from django.db import models
class Dog(models.Model):
name = models.CharField(max_length=200)
data = JSONField()
def __str__(self):
return self.name
.. fieldlookup:: jsonfield.key
Key, index, and path lookups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To query based on a given dictionary key, use that key as the lookup name::
>>> Dog.objects.create(name='Rufus', data={
... 'breed': 'labrador',
... 'owner': {
... 'name': 'Bob',
... 'other_pets': [{
... 'name': 'Fishy',
... }],
... },
... })
>>> Dog.objects.create(name='Meg', data={'breed': 'collie', 'owner': None})
>>> Dog.objects.filter(data__breed='collie')
<QuerySet [<Dog: Meg>]>
Multiple keys can be chained together to form a path lookup::
>>> Dog.objects.filter(data__owner__name='Bob')
<QuerySet [<Dog: Rufus>]>
If the key is an integer, it will be interpreted as an index lookup in an
array::
>>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy')
<QuerySet [<Dog: Rufus>]>
If the key you wish to query by clashes with the name of another lookup, use
the :lookup:`jsonfield.contains` lookup instead.
If only one key or index is used, the SQL operator ``->`` is used. If multiple
operators are used then the ``#>`` operator is used.
To query for ``null`` in JSON data, use ``None`` as a value::
>>> Dog.objects.filter(data__owner=None)
<QuerySet [<Dog: Meg>]>
To query for missing keys, use the ``isnull`` lookup::
>>> Dog.objects.create(name='Shep', data={'breed': 'collie'})
>>> Dog.objects.filter(data__owner__isnull=True)
<QuerySet [<Dog: Shep>]>
.. warning::
Since any string could be a key in a JSON object, any lookup other than
those listed below will be interpreted as a key lookup. No errors are
raised. Be extra careful for typing mistakes, and always check your queries
work as you intend.
Containment and key operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. fieldlookup:: jsonfield.contains
.. fieldlookup:: jsonfield.contained_by
.. fieldlookup:: jsonfield.has_key
.. fieldlookup:: jsonfield.has_any_keys
.. fieldlookup:: jsonfield.has_keys
:class:`~django.contrib.postgres.fields.JSONField` shares lookups relating to
containment and keys with :class:`~django.contrib.postgres.fields.HStoreField`.
- :lookup:`contains <hstorefield.contains>` (accepts any JSON rather than
just a dictionary of strings)
- :lookup:`contained_by <hstorefield.contained_by>` (accepts any JSON
rather than just a dictionary of strings)
- :lookup:`has_key <hstorefield.has_key>`
- :lookup:`has_any_keys <hstorefield.has_any_keys>`
- :lookup:`has_keys <hstorefield.has_keys>`
See :ref:`querying-jsonfield` for details.
.. _range-fields:

8
docs/ref/contrib/postgres/forms.txt
View file

@ -164,8 +164,8 @@ Fields
.. class:: JSONField
A field which accepts JSON encoded data for a
:class:`~django.contrib.postgres.fields.JSONField`. It is represented by an
HTML ``<textarea>``.
:class:`~django.db.models.JSONField`. It is represented by an HTML
``<textarea>``.
.. admonition:: User friendly forms
@ -173,6 +173,10 @@ Fields
it is a useful way to format data from a client-side widget for
submission to the server.
.. deprecated:: 3.1
Use :class:`django.forms.JSONField` instead.
Range Fields
------------

16
docs/ref/databases.txt
View file

@ -783,6 +783,22 @@ iterator. Your code must handle this.
.. _`Isolation in SQLite`: https://sqlite.org/isolation.html
.. _sqlite-json1:
Enabling JSON1 extension on SQLite
----------------------------------
To use :class:`~django.db.models.JSONField` on SQLite, you need to enable the
`JSON1 extension`_ on Python's :py:mod:`sqlite3` library. If the extension is
not enabled on your installation, a system error (``fields.E180``) will be
raised.
To enable the JSON1 extension you can follow the instruction on
`the wiki page`_.
.. _JSON1 extension: https://www.sqlite.org/json1.html
.. _the wiki page: https://code.djangoproject.com/wiki/JSON1Extension
.. _oracle-notes:
Oracle notes

54
docs/ref/forms/fields.txt
View file

@ -776,6 +776,60 @@ For each field, we describe the default widget used if you don't specify
These control the range of values permitted in the field.
``JSONField``
-------------
.. class:: JSONField(encoder=None, decoder=None, **kwargs)
.. versionadded:: 3.1
A field which accepts JSON encoded data for a
:class:`~django.db.models.JSONField`.
* Default widget: :class:`Textarea`
* Empty value: ``''`` (an empty string)
* Normalizes to: A Python representation of the JSON value (usually as a
``dict``, ``list``, or ``None``), depending on :attr:`JSONField.decoder`.
* Validates that the given value is a valid JSON.
* Error message keys: ``required``, ``invalid``
Takes two optional arguments:
.. attribute:: encoder
A :py:class:`json.JSONEncoder` subclass to serialize data types not
supported by the standard JSON serializer (e.g. ``datetime.datetime``
or :class:`~python:uuid.UUID`). For example, you can use the
:class:`~django.core.serializers.json.DjangoJSONEncoder` class.
Defaults to ``json.JSONEncoder``.
.. attribute:: decoder
A :py:class:`json.JSONDecoder` subclass to deserialize the input. Your
deserialization may need to account for the fact that you can't be
certain of the input type. For example, you run the risk of returning a
``datetime`` that was actually a string that just happened to be in the
same format chosen for ``datetime``\s.
The ``decoder`` can be used to validate the input. If
:py:class:`json.JSONDecodeError` is raised during the deserialization,
a ``ValidationError`` will be raised.
Defaults to ``json.JSONDecoder``.
.. note::
If you use a :class:`ModelForm <django.forms.ModelForm>`, the
``encoder`` and ``decoder`` from :class:`~django.db.models.JSONField`
will be used.
.. admonition:: User friendly forms
``JSONField`` is not particularly user friendly in most cases. However,
it is a useful way to format data from a client-side widget for
submission to the server.
``GenericIPAddressField``
-------------------------

69
docs/ref/models/fields.txt
View file

@ -348,7 +348,7 @@ The default can't be a mutable object (model instance, ``list``, ``set``, etc.),
as a reference to the same instance of that object would be used as the default
value in all new model instances. Instead, wrap the desired default in a
callable. For example, if you want to specify a default ``dict`` for
:class:`~django.contrib.postgres.fields.JSONField`, use a function::
:class:`~django.db.models.JSONField`, use a function::
def contact_default():
return {"email": "to1@example.com"}
@ -1175,6 +1175,73 @@ are converted to lowercase.
If you allow for blank values, you have to allow for null values since blank
values are stored as null.
``JSONField``
-------------
.. class:: JSONField(encoder=None, decoder=None, **options)
.. versionadded:: 3.1
A field for storing JSON encoded data. In Python the data is represented in its
Python native format: dictionaries, lists, strings, numbers, booleans and
``None``.
``JSONField`` is supported on MariaDB 10.2.7+, MySQL 5.7.8+, Oracle,
PostgreSQL, and SQLite 3.9.0+ (with the :ref:`JSON1 extension enabled
<sqlite-json1>`).
.. attribute:: JSONField.encoder
An optional :py:class:`json.JSONEncoder` subclass to serialize data types
not supported by the standard JSON serializer (e.g. ``datetime.datetime``
or :class:`~python:uuid.UUID`). For example, you can use the
:class:`~django.core.serializers.json.DjangoJSONEncoder` class.
Defaults to ``json.JSONEncoder``.
.. attribute:: JSONField.decoder
An optional :py:class:`json.JSONDecoder` subclass to deserialize the value
retrieved from the database. The value will be in the format chosen by the
custom encoder (most often a string). Your deserialization may need to
account for the fact that you can't be certain of the input type. For
example, you run the risk of returning a ``datetime`` that was actually a
string that just happened to be in the same format chosen for
``datetime``\s.
Defaults to ``json.JSONDecoder``.
If you give the field a :attr:`~django.db.models.Field.default`, ensure it's an
immutable object, such as a ``str``, or a callable object that returns a fresh
mutable object each time, such as ``dict`` or a function. Providing a mutable
default object like ``default={}`` or ``default=[]`` shares the one object
between all model instances.
To query ``JSONField`` in the database, see :ref:`querying-jsonfield`.
.. admonition:: Indexing
:class:`~django.db.models.Index` and :attr:`.Field.db_index` both create a
B-tree index, which isn't particularly helpful when querying ``JSONField``.
On PostgreSQL only, you can use
:class:`~django.contrib.postgres.indexes.GinIndex` that is better suited.
.. admonition:: PostgreSQL users
PostgreSQL has two native JSON based data types: ``json`` and ``jsonb``.
The main difference between them is how they are stored and how they can be
queried. PostgreSQL's ``json`` field is stored as the original string
representation of the JSON and must be decoded on the fly when queried
based on keys. The ``jsonb`` field is stored based on the actual structure
of the JSON which allows indexing. The trade-off is a small additional cost
on writing to the ``jsonb`` field. ``JSONField`` uses ``jsonb``.
.. admonition:: Oracle users
Oracle Database does not support storing JSON scalar values. Only JSON
objects and arrays (represented in Python using :py:class:`dict` and
:py:class:`list`) are supported.
``NullBooleanField``
--------------------

48
docs/releases/3.1.txt
View file

@ -64,6 +64,21 @@ Asynchronous support should be entirely backwards-compatible and we have tried
to ensure that it has no speed regressions for your existing, synchronous code.
It should have no noticeable effect on any existing Django projects.
JSONField for all supported database backends
---------------------------------------------
Django now includes the :class:`.models.JSONField` and
:class:`forms.JSONField <django.forms.JSONField>` that can be used on all
supported database backends. Both fields support the use of custom JSON
encoders and decoders. The model field supports the introspection, lookups, and
transforms that were previously PostgreSQL-only.
If your project uses ``django.contrib.postgres.fields.JSONField``, plus the
related form field and transforms, you should adjust to use the new fields,
and generate and apply a database migration. For now, the old fields and
transforms are left as a reference to the new ones and are :ref:`deprecated as
of this release <deprecated-jsonfield>`.
Minor features
--------------
@ -549,6 +564,15 @@ backends.
``DatabaseOperations.execute_sql_flush()`` is removed. The method now uses
the database of the called instance.
* Third-party database backends must implement support for ``JSONField`` or set
``DatabaseFeatures.supports_json_field`` to ``False``. If storing primitives
is not supported, set ``DatabaseFeatures.supports_primitives_in_json_field``
to ``False``. If there is a true datatype for JSON, set
``DatabaseFeatures.has_native_json_field`` to ``True``.
* Third party database backends must implement introspection for ``JSONField``
or set ``can_introspect_json_field`` to ``False``.
Dropped support for MariaDB 10.1
--------------------------------
@ -693,11 +717,35 @@ Miscellaneous
* The minimum supported version of ``mysqlclient`` is increased from 1.3.13 to
1.4.0.
* The undocumented ``django.contrib.postgres.forms.InvalidJSONInput`` and
``django.contrib.postgres.forms.JSONString`` are moved to
``django.forms.fields``.
* The undocumented ``django.contrib.postgres.fields.jsonb.JsonAdapter`` class
is removed.
.. _deprecated-features-3.1:
Features deprecated in 3.1
==========================
.. _deprecated-jsonfield:
PostgreSQL ``JSONField``
------------------------
``django.contrib.postgres.fields.JSONField`` and
``django.contrib.postgres.forms.JSONField`` are deprecated in favor of
:class:`.models.JSONField` and
:class:`forms.JSONField <django.forms.JSONField>`.
The undocumented ``django.contrib.postgres.fields.jsonb.KeyTransform`` and
``django.contrib.postgres.fields.jsonb.KeyTextTransform`` are also deprecated
in favor of the transforms in ``django.db.models.fields.json``.
The new ``JSONField``\s, ``KeyTransform``, and ``KeyTextTransform`` can be used
on all supported database backends.
Miscellaneous
-------------

230
docs/topics/db/queries.txt
View file

@ -798,6 +798,236 @@ being evaluated and therefore populate the cache::
Simply printing the queryset will not populate the cache. This is because
the call to ``__repr__()`` only returns a slice of the entire queryset.
.. _querying-jsonfield:
Querying ``JSONField``
======================
Lookups implementation is different in :class:`~django.db.models.JSONField`,
mainly due to the existence of key transformations. To demonstrate, we will use
the following example model::
from django.db import models
class Dog(models.Model):
name = models.CharField(max_length=200)
data = models.JSONField(null=True)
def __str__(self):
return self.name
Storing and querying for ``None``
---------------------------------
As with other fields, storing ``None`` as the field's value will store it as
SQL ``NULL``. While not recommended, it is possible to store JSON scalar
``null`` instead of SQL ``NULL`` by using :class:`Value('null')
<django.db.models.Value>`.
Whichever of the values is stored, when retrieved from the database, the Python
representation of the JSON scalar ``null`` is the same as SQL ``NULL``, i.e.
``None``. Therefore, it can be hard to distinguish between them.
This only applies to ``None`` as the top-level value of the field. If ``None``
is inside a :py:class:`list` or :py:class:`dict`, it will always be interpreted
as JSON ``null``.
When querying, ``None`` value will always be interpreted as JSON ``null``. To
query for SQL ``NULL``, use :lookup:`isnull`::
>>> Dog.objects.create(name='Max', data=None) # SQL NULL.
<Dog: Max>
>>> Dog.objects.create(name='Archie', data=Value('null')) # JSON null.
<Dog: Archie>
>>> Dog.objects.filter(data=None)
<QuerySet [<Dog: Archie>]>
>>> Dog.objects.filter(data=Value('null'))
<QuerySet [<Dog: Archie>]>
>>> Dog.objects.filter(data__isnull=True)
<QuerySet [<Dog: Max>]>
>>> Dog.objects.filter(data__isnull=False)
<QuerySet [<Dog: Archie>]>
Unless you are sure you wish to work with SQL ``NULL`` values, consider setting
``null=False`` and providing a suitable default for empty values, such as
``default=dict``.
.. note::
Storing JSON scalar ``null`` does not violate :attr:`null=False
<django.db.models.Field.null>`.
.. fieldlookup:: jsonfield.key
Key, index, and path transforms
-------------------------------
To query based on a given dictionary key, use that key as the lookup name::
>>> Dog.objects.create(name='Rufus', data={
... 'breed': 'labrador',
... 'owner': {
... 'name': 'Bob',
... 'other_pets': [{
... 'name': 'Fishy',
... }],
... },
... })
<Dog: Rufus>
>>> Dog.objects.create(name='Meg', data={'breed': 'collie', 'owner': None})
<Dog: Meg>
>>> Dog.objects.filter(data__breed='collie')
<QuerySet [<Dog: Meg>]>
Multiple keys can be chained together to form a path lookup::
>>> Dog.objects.filter(data__owner__name='Bob')
<QuerySet [<Dog: Rufus>]>
If the key is an integer, it will be interpreted as an index transform in an
array::
>>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy')
<QuerySet [<Dog: Rufus>]>
If the key you wish to query by clashes with the name of another lookup, use
the :lookup:`contains <jsonfield.contains>` lookup instead.
To query for missing keys, use the ``isnull`` lookup::
>>> Dog.objects.create(name='Shep', data={'breed': 'collie'})
<Dog: Shep>
>>> Dog.objects.filter(data__owner__isnull=True)
<QuerySet [<Dog: Shep>]>
.. note::
The lookup examples given above implicitly use the :lookup:`exact` lookup.
Key, index, and path transforms can also be chained with:
:lookup:`contains`, :lookup:`icontains`, :lookup:`endswith`,
:lookup:`iendswith`, :lookup:`iexact`, :lookup:`regex`, :lookup:`iregex`,
:lookup:`startswith`, :lookup:`istartswith`, :lookup:`lt`, :lookup:`lte`,
:lookup:`gt`, and :lookup:`gte` lookups.
.. warning::
Since any string could be a key in a JSON object, any lookup other than
those listed below will be interpreted as a key lookup. No errors are
raised. Be extra careful for typing mistakes, and always check your queries
work as you intend.
.. admonition:: MariaDB and Oracle users
Using :meth:`~django.db.models.query.QuerySet.order_by` on key, index, or
path transforms will sort the objects using the string representation of
the values. This is because MariaDB and Oracle Database do not provide a
function that converts JSON values into their equivalent SQL values.
.. admonition:: Oracle users
On Oracle Database, using ``None`` as the lookup value in an
:meth:`~django.db.models.query.QuerySet.exclude` query will return objects
that do not have ``null`` as the value at the given path, including objects
that do not have the path. On other database backends, the query will
return objects that have the path and the value is not ``null``.
.. admonition:: PostgreSQL users
On PostgreSQL, if only one key or index is used, the SQL operator ``->`` is
used. If multiple operators are used then the ``#>`` operator is used.
Containment and key operations
------------------------------
.. fieldlookup:: jsonfield.contains
``contains``
~~~~~~~~~~~~
The :lookup:`contains` lookup is overridden on ``JSONField``. The returned
objects are those where the given ``dict`` of key-value pairs are all
contained in the top-level of the field. For example::
>>> Dog.objects.create(name='Rufus', data={'breed': 'labrador', 'owner': 'Bob'})
<Dog: Rufus>
>>> Dog.objects.create(name='Meg', data={'breed': 'collie', 'owner': 'Bob'})
<Dog: Meg>
>>> Dog.objects.create(name='Fred', data={})
<Dog: Fred>
>>> Dog.objects.filter(data__contains={'owner': 'Bob'})
<QuerySet [<Dog: Rufus>, <Dog: Meg>]>
>>> Dog.objects.filter(data__contains={'breed': 'collie'})
<QuerySet [<Dog: Meg>]>
.. fieldlookup:: jsonfield.contained_by
``contained_by``
~~~~~~~~~~~~~~~~
This is the inverse of the :lookup:`contains <jsonfield.contains>` lookup - the
objects returned will be those where the key-value pairs on the object are a
subset of those in the value passed. For example::
>>> Dog.objects.create(name='Rufus', data={'breed': 'labrador', 'owner': 'Bob'})
<Dog: Rufus>
>>> Dog.objects.create(name='Meg', data={'breed': 'collie', 'owner': 'Bob'})
<Dog: Meg>
>>> Dog.objects.create(name='Fred', data={})
<Dog: Fred>
>>> Dog.objects.filter(data__contained_by={'breed': 'collie', 'owner': 'Bob'})
<QuerySet [<Dog: Meg>, <Dog: Fred>]>
>>> Dog.objects.filter(data__contained_by={'breed': 'collie'})
<QuerySet [<Dog: Fred>]>
.. admonition:: Oracle
``contained_by`` is not supported on Oracle.
.. fieldlookup:: jsonfield.has_key
``has_key``
~~~~~~~~~~~
Returns objects where the given key is in the top-level of the data. For
example::
>>> Dog.objects.create(name='Rufus', data={'breed': 'labrador'})
<Dog: Rufus>
>>> Dog.objects.create(name='Meg', data={'breed': 'collie', 'owner': 'Bob'})
<Dog: Meg>
>>> Dog.objects.filter(data__has_key='owner')
<QuerySet [<Dog: Meg>]>
.. fieldlookup:: jsonfield.has_any_keys
``has_keys``
~~~~~~~~~~~~
Returns objects where all of the given keys are in the top-level of the data.
For example::
>>> Dog.objects.create(name='Rufus', data={'breed': 'labrador'})
<Dog: Rufus>
>>> Dog.objects.create(name='Meg', data={'breed': 'collie', 'owner': 'Bob'})
<Dog: Meg>
>>> Dog.objects.filter(data__has_keys=['breed', 'owner'])
<QuerySet [<Dog: Meg>]>
.. fieldlookup:: jsonfield.has_keys
``has_any_keys``
~~~~~~~~~~~~~~~~
Returns objects where any of the given keys are in the top-level of the data.
For example::
>>> Dog.objects.create(name='Rufus', data={'breed': 'labrador'})
<Dog: Rufus>
>>> Dog.objects.create(name='Meg', data={'owner': 'Bob'})
<Dog: Meg>
>>> Dog.objects.filter(data__has_any_keys=['owner', 'breed'])
<QuerySet [<Dog: Rufus>, <Dog: Meg>]>
.. _complex-lookups-with-q:
Complex lookups with ``Q`` objects