feat: allow to configure media cache (for JS and CSS files) (#946)

This commit is contained in:
Juro Oravec 2025-02-03 21:24:26 +01:00 committed by GitHub
parent bb61ff42eb
commit 48bae51ab9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
14 changed files with 305 additions and 73 deletions

View file

@ -1,5 +1,16 @@
# Release notes
## v0.128
#### Feat
- Configurable cache - Set [`COMPONENTS.cache`](https://django-components.github.io/django-components/0.128/reference/settings/#django_components.app_settings.ComponentsSettings.cache) to change where and how django-components caches JS and CSS files.
Read more on [Caching](https://django-components.github.io/django-components/0.128/guides/setup/caching).
#### Perf
- Component input validation is now 6-7x faster on CPython and PyPy. This previously made up 10-30% of the total render time. ([#945](https://github.com/django-components/django-components/pull/945))
## v0.127
#### Fix

View file

@ -0,0 +1,47 @@
---
title: Caching
weight: 2
---
This page describes the kinds of assets that django-components caches and how to configure the cache backends.
## Component's JS and CSS files
django-components caches the JS and CSS files associated with your components. This enables components to be rendered as HTML fragments and still having the associated JS and CSS files loaded with them.
This includes:
- Inlined JS/CSS defined via [`Component.js`](../../reference/api.md#django_components.Component.js) and [`Component.css`](../../reference/api.md#django_components.Component.css)
- JS/CSS variables generated from [`get_js_data()`](../../reference/api.md#django_components.Component.get_js_data) and [`get_css_data()`](../../reference/api.md#django_components.Component.get_css_data)
By default, django-components uses Django's local memory cache backend to store these assets. You can configure it to use any of your Django cache backends by setting the [`COMPONENTS.cache`](../../reference/settings.md#django_components.app_settings.ComponentsSettings.cache) option in your settings:
```python
COMPONENTS = {
# Name of the cache backend to use
"cache": "my-cache-backend",
}
```
The value should be the name of one of your configured cache backends from Django's [`CACHES`](https://docs.djangoproject.com/en/stable/ref/settings/#std-setting-CACHES) setting.
For example, to use Redis for caching component assets:
```python
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.locmem.LocMemCache",
},
"component-media": {
"BACKEND": "django.core.cache.backends.redis.RedisCache",
"LOCATION": "redis://127.0.0.1:6379/1",
}
}
COMPONENTS = {
# Use the Redis cache backend
"cache": "component-media",
}
```
See [`COMPONENTS.cache`](../../reference/settings.md#django_components.app_settings.ComponentsSettings.cache) for more details about this setting.

View file

@ -1,6 +1,6 @@
---
title: Running with development server
weight: 2
title: Development server
weight: 3
---
### Reload dev server on component file changes

View file

@ -1,5 +1,6 @@
---
title: Syntax highlighting
weight: 1
---
## VSCode

View file

@ -26,10 +26,14 @@ weight: 3
BASE_DIR = Path(__file__).resolve().parent.parent
```
4. Set [`COMPONENTS.dirs`](../reference/settings.md#django_components.app_settings.ComponentsSettings.dirs)
4. _Optional._ Set [`COMPONENTS.dirs`](../reference/settings.md#django_components.app_settings.ComponentsSettings.dirs)
and/or [`COMPONENTS.app_dirs`](../reference/settings.md#django_components.app_settings.ComponentsSettings.app_dirs)
so django_components knows where to find component HTML, JS and CSS files:
If [`COMPONENTS.dirs`](../reference/settings.md#django_components.app_settings.ComponentsSettings.dirs)
is omitted, django-components will by default look for a top-level `/components` directory,
`{BASE_DIR}/components`.
```python
from django_components import ComponentsSettings
@ -41,10 +45,6 @@ weight: 3
)
```
If [`COMPONENTS.dirs`](../reference/settings.md#django_components.app_settings.ComponentsSettings.dirs)
is omitted, django-components will by default look for a top-level `/components` directory,
`{BASE_DIR}/components`.
In addition to [`COMPONENTS.dirs`](../reference/settings.md#django_components.app_settings.ComponentsSettings.dirs),
django_components will also load components from app-level directories, such as `my-app/components/`.
The directories within apps are configured with
@ -65,7 +65,7 @@ weight: 3
which has the same effect.
- Add `loaders` to `OPTIONS` list and set it to following value:
This allows Django load component HTML files as Django templates.
This allows Django to load component HTML files as Django templates.
```python
TEMPLATES = [
@ -152,8 +152,15 @@ If you want to use JS or CSS with components, you will need to:
</html>
```
5. _Optional._ By default, components' JS and CSS files are cached in memory.
If you want to change the cache backend, set the [`COMPONENTS.cache`](../reference/settings.md#django_components.app_settings.ComponentsSettings.cache) setting.
Read more in [Caching](../../guides/setup/caching).
## Optional
### Builtin template tags
To avoid loading the app in each template using `{% load component_tags %}`, you can add the tag as a 'builtin' in settings.py
```python
@ -169,4 +176,6 @@ TEMPLATES = [
]
```
Read on to find out how to build your first component!
---
Now you're all set! Read on to find out how to build your first component.

View file

@ -33,6 +33,7 @@ Here's overview of all available settings and their defaults:
```py
defaults = ComponentsSettings(
autodiscover=True,
cache=None,
context_behavior=ContextBehavior.DJANGO.value, # "django" | "isolated"
# Root-level "components" dirs, e.g. `/path/to/proj/components/`
dirs=[Path(settings.BASE_DIR) / "components"],
@ -85,6 +86,16 @@ defaults = ComponentsSettings(
show_if_no_docstring: true
show_labels: false
::: django_components.app_settings.ComponentsSettings.cache
options:
show_root_heading: true
show_signature: true
separate_signature: true
show_symbol_type_heading: false
show_symbol_type_toc: false
show_if_no_docstring: true
show_labels: false
::: django_components.app_settings.ComponentsSettings.context_behavior
options:
show_root_heading: true

View file

@ -20,7 +20,7 @@ Import as
<a href="https://github.com/django-components/django-components/tree/master/src/django_components/templatetags/component_tags.py#L1066" target="_blank">See source code</a>
<a href="https://github.com/django-components/django-components/tree/master/src/django_components/templatetags/component_tags.py#L1053" target="_blank">See source code</a>
@ -43,7 +43,7 @@ If you insert this tag multiple times, ALL CSS links will be duplicately inserte
<a href="https://github.com/django-components/django-components/tree/master/src/django_components/templatetags/component_tags.py#L1088" target="_blank">See source code</a>
<a href="https://github.com/django-components/django-components/tree/master/src/django_components/templatetags/component_tags.py#L1075" target="_blank">See source code</a>
@ -67,7 +67,7 @@ If you insert this tag multiple times, ALL JS scripts will be duplicately insert
<a href="https://github.com/django-components/django-components/tree/master/src/django_components/templatetags/component_tags.py#L1436" target="_blank">See source code</a>
<a href="https://github.com/django-components/django-components/tree/master/src/django_components/templatetags/component_tags.py#L1434" target="_blank">See source code</a>

View file

@ -212,6 +212,25 @@ class ComponentsSettings(NamedTuple):
```
"""
cache: Optional[str] = None
"""
Name of the [Django cache](https://docs.djangoproject.com/en/5.1/topics/cache/)
to be used for storing component's JS and CSS files.
If `None`, a [`LocMemCache`](https://docs.djangoproject.com/en/5.1/topics/cache/#local-memory-caching)
is used with default settings.
Defaults to `None`.
Read more about [caching](../../guides/setup/caching).
```python
COMPONENTS = ComponentsSettings(
cache="my_cache",
)
```
"""
context_behavior: Optional[ContextBehaviorType] = None
"""
Configure whether, inside a component template, you can use variables from the outside
@ -383,7 +402,7 @@ class ComponentsSettings(NamedTuple):
[`COMPONENTS.app_dirs`](../settings/#django_components.app_settings.ComponentsSettings.app_dirs)
change.
See [Reload dev server on component file changes](../../guides/setup/dev_server_setup/#reload-dev-server-on-component-file-changes).
See [Reload dev server on component file changes](../../guides/setup/development_server/#reload-dev-server-on-component-file-changes).
Defaults to `False`.
@ -617,6 +636,7 @@ class Dynamic(Generic[T]):
# --snippet:defaults--
defaults = ComponentsSettings(
autodiscover=True,
cache=None,
context_behavior=ContextBehavior.DJANGO.value, # "django" | "isolated"
# Root-level "components" dirs, e.g. `/path/to/proj/components/`
dirs=Dynamic(lambda: [Path(settings.BASE_DIR) / "components"]), # type: ignore[arg-type]
@ -661,6 +681,10 @@ class InternalSettings:
def AUTODISCOVER(self) -> bool:
return default(self._settings.autodiscover, cast(bool, defaults.autodiscover))
@property
def CACHE(self) -> Optional[str]:
return default(self._settings.cache, defaults.cache)
@property
def DIRS(self) -> Sequence[Union[str, PathLike, Tuple[str, str], Tuple[str, PathLike]]]:
# For DIRS we use a getter, because default values uses Django settings,

View file

@ -0,0 +1,46 @@
from typing import Optional
from django.core.cache import BaseCache, caches
from django.core.cache.backends.locmem import LocMemCache
from django_components.app_settings import app_settings
from django_components.util.cache import LRUCache
# This stores the parsed Templates. This is strictly local for now, as it stores instances.
# NOTE: Lazily initialized so it can be configured based on user-defined settings.
#
# TODO: Once we handle whole template parsing ourselves, this could store just
# the parsed template AST (+metadata) instead of Template instances. In that case
# we could open this up to be stored non-locally and shared across processes.
# This would also allow us to remove our custom `LRUCache` implementation.
template_cache: Optional[LRUCache] = None
# This stores the inlined component JS and CSS files (e.g. `Component.js` and `Component.css`).
# We also store here the generated JS and CSS scripts that inject JS / CSS variables into the page.
component_media_cache: Optional[BaseCache] = None
def get_template_cache() -> LRUCache:
global template_cache
if template_cache is None:
template_cache = LRUCache(maxsize=app_settings.TEMPLATE_CACHE_SIZE)
return template_cache
def get_component_media_cache() -> BaseCache:
global component_media_cache
if component_media_cache is None:
if app_settings.CACHE is not None:
component_media_cache = caches[app_settings.CACHE]
else:
component_media_cache = LocMemCache(
"django-components-media",
{
"TIMEOUT": None, # No timeout
"MAX_ENTRIES": None, # No max size
"CULL_FREQUENCY": 3,
},
)
return component_media_cache

View file

@ -1081,17 +1081,15 @@ class Component(
# Allow to access component input and metadata like component ID from within these hook
with self._with_metadata(metadata):
context_data = self.get_context_data(*args, **kwargs)
# TODO - enable JS and CSS vars
# js_data = self.get_js_data(*args, **kwargs)
# css_data = self.get_css_data(*args, **kwargs)
# TODO - enable JS and CSS vars - EXPOSE AND DOCUMENT AND MAKE NON-NULL
js_data = self.get_js_data(*args, **kwargs) if hasattr(self, "get_js_data") else {} # type: ignore
css_data = self.get_css_data(*args, **kwargs) if hasattr(self, "get_css_data") else {} # type: ignore
self._validate_outputs(data=context_data)
# Process Component's JS and CSS
js_data: Dict = {} # TODO
cache_component_js(self.__class__)
js_input_hash = cache_component_js_vars(self.__class__, js_data) if js_data else None
css_data: Dict = {} # TODO
cache_component_css(self.__class__)
css_input_hash = cache_component_css_vars(self.__class__, css_data) if css_data else None

View file

@ -4,7 +4,6 @@ import base64
import json
import re
import sys
from abc import ABC, abstractmethod
from hashlib import md5
from typing import (
TYPE_CHECKING,
@ -34,6 +33,7 @@ from django.utils.decorators import sync_and_async_middleware
from django.utils.safestring import SafeString, mark_safe
from djc_core_html_parser import set_html_attributes
from django_components.cache import get_component_media_cache
from django_components.node import BaseNode
from django_components.util.misc import is_nonempty_str
@ -46,40 +46,17 @@ RenderType = Literal["document", "fragment"]
#########################################################
# 1. Cache the inlined component JS and CSS scripts,
# so they can be referenced and retrieved later via
# an ID.
# 1. Cache the inlined component JS and CSS scripts (`Component.js` and `Component.css`).
#
# To support HTML fragments, when a fragment is loaded on a page,
# we on-demand request the JS and CSS files of the components that are
# referenced in the fragment.
#
# Thus, we need to persist the JS and CSS files across requests. These are then accessed
# via `cached_script_view` endpoint.
#########################################################
class ComponentMediaCacheABC(ABC):
@abstractmethod
def get(self, key: str) -> Optional[str]: ... # noqa: #704
@abstractmethod
def has(self, key: str) -> bool: ... # noqa: #704
@abstractmethod
def set(self, key: str, value: str) -> None: ... # noqa: #704
class InMemoryComponentMediaCache(ComponentMediaCacheABC):
def __init__(self) -> None:
self._data: Dict[str, str] = {}
def get(self, key: str) -> Optional[str]:
return self._data.get(key, None)
def has(self, key: str) -> bool:
return key in self._data
def set(self, key: str, value: str) -> None:
self._data[key] = value
comp_media_cache = InMemoryComponentMediaCache()
# NOTE: Initially, we fetched components by their registered name, but that didn't work
# for multiple registries and unregistered components.
#
@ -101,7 +78,9 @@ else:
comp_hash_mapping: WeakValueDictionary[str, Type["Component"]] = WeakValueDictionary()
# Convert Component class to something like `TableComp_a91d03`
# Generate keys like
# `__components:MyButton_a78y37:js:df7c6d10`
# `__components:MyButton_a78y37:css`
def _gen_cache_key(
comp_cls_hash: str,
script_type: ScriptType,
@ -119,7 +98,8 @@ def _is_script_in_cache(
input_hash: Optional[str],
) -> bool:
cache_key = _gen_cache_key(comp_cls._class_hash, script_type, input_hash)
return comp_media_cache.has(cache_key)
cache = get_component_media_cache()
return cache.has_key(cache_key)
def _cache_script(
@ -141,7 +121,8 @@ def _cache_script(
# NOTE: By setting the script in the cache, we will be able to retrieve it
# via the endpoint, e.g. when we make a request to `/components/cache/MyComp_ab0c2d.js`.
comp_media_cache.set(cache_key, script.strip())
cache = get_component_media_cache()
cache.set(cache_key, script.strip())
def cache_component_js(comp_cls: Type["Component"]) -> None:
@ -187,7 +168,7 @@ def cache_component_js_vars(comp_cls: Type["Component"], js_vars: Dict) -> Optio
if not _is_script_in_cache(comp_cls, "js", input_hash):
_cache_script(
comp_cls=comp_cls,
script="", # TODO
script="", # TODO - enable JS and CSS vars
script_type="js",
input_hash=input_hash,
)
@ -195,7 +176,7 @@ def cache_component_js_vars(comp_cls: Type["Component"], js_vars: Dict) -> Optio
return input_hash
def wrap_component_js(comp_cls: Type["Component"], content: str) -> SafeString:
def wrap_component_js(comp_cls: Type["Component"], content: str) -> str:
if "</script" in content:
raise RuntimeError(
f"Content of `Component.js` for component '{comp_cls.__name__}' contains '</script>' end tag. "
@ -237,7 +218,7 @@ def cache_component_css_vars(comp_cls: Type["Component"], css_vars: Dict) -> Opt
if not _is_script_in_cache(comp_cls, "css", input_hash):
_cache_script(
comp_cls=comp_cls,
script="", # TODO
script="", # TODO - enable JS and CSS vars
script_type="css",
input_hash=input_hash,
)
@ -245,7 +226,7 @@ def cache_component_css_vars(comp_cls: Type["Component"], css_vars: Dict) -> Opt
return input_hash
def wrap_component_css(comp_cls: Type["Component"], content: str) -> SafeString:
def wrap_component_css(comp_cls: Type["Component"], content: str) -> str:
if "</style" in content:
raise RuntimeError(
f"Content of `Component.css` for component '{comp_cls.__name__}' contains '</style>' end tag. "
@ -822,9 +803,10 @@ def get_script_content(
script_type: ScriptType,
comp_cls: Type["Component"],
input_hash: Optional[str],
) -> SafeString:
) -> Optional[str]:
cache = get_component_media_cache()
cache_key = _gen_cache_key(comp_cls._class_hash, script_type, input_hash)
script = comp_media_cache.get(cache_key)
script = cache.get(cache_key)
return script
@ -833,8 +815,13 @@ def get_script_tag(
script_type: ScriptType,
comp_cls: Type["Component"],
input_hash: Optional[str],
) -> SafeString:
) -> str:
content = get_script_content(script_type, comp_cls, input_hash)
if content is None:
raise RuntimeError(
f"Could not find {script_type.upper()} for component '{comp_cls.__name__}' "
f"(hash: {comp_cls._class_hash})"
)
if script_type == "js":
content = wrap_component_js(comp_cls, content)

View file

@ -2,17 +2,12 @@ from typing import Any, Optional, Type, TypeVar
from django.template import Origin, Template
from django_components.app_settings import app_settings
from django_components.util.cache import LRUCache
from django_components.cache import get_template_cache
from django_components.util.misc import get_import_path
TTemplate = TypeVar("TTemplate", bound=Template)
# Lazily initialize the cache
template_cache: Optional[LRUCache[Template]] = None
# Central logic for creating Templates from string, so we can cache the results
def cached_template(
template_string: str,
@ -54,16 +49,14 @@ def cached_template(
)
```
""" # noqa: E501
global template_cache
if template_cache is None:
template_cache = LRUCache(maxsize=app_settings.TEMPLATE_CACHE_SIZE)
template_cache = get_template_cache()
template_cls = template_cls or Template
template_cls_path = get_import_path(template_cls)
engine_cls_path = get_import_path(engine.__class__) if engine else None
cache_key = (template_cls_path, template_string, engine_cls_path)
maybe_cached_template = template_cache.get(cache_key)
maybe_cached_template: Optional[Template] = template_cache.get(cache_key)
if maybe_cached_template is None:
template = template_cls(template_string, origin=origin, name=name, engine=engine)
template_cache.set(cache_key, template)

View file

@ -1,6 +1,8 @@
from django.test import TestCase
from django.test import TestCase, override_settings
from django.core.cache.backends.locmem import LocMemCache
from django_components.util.cache import LRUCache
from django_components import Component, register
from .django_test_setup import setup_test_config
@ -40,3 +42,100 @@ class CacheTests(TestCase):
self.assertEqual(cache.get("d"), None)
self.assertEqual(cache.get("e"), None)
self.assertEqual(cache.get("f"), None)
class ComponentMediaCacheTests(TestCase):
def setUp(self):
# Create a custom locmem cache for testing
self.test_cache = LocMemCache(
"test-cache",
{
"TIMEOUT": None, # No timeout
"MAX_ENTRIES": None, # No max size
"CULL_FREQUENCY": 3,
},
)
@override_settings(COMPONENTS={"cache": "test-cache"})
def test_component_media_caching(self):
@register("test_simple")
class TestSimpleComponent(Component):
template = """
<div>Template only component</div>
"""
def get_js_data(self):
return {}
def get_css_data(self):
return {}
@register("test_media_no_vars")
class TestMediaNoVarsComponent(Component):
template = """
<div>Template and JS component</div>
{% component "test_simple" / %}
"""
js = "console.log('Hello from JS');"
css = ".novars-component { color: blue; }"
def get_js_data(self):
return {}
def get_css_data(self):
return {}
class TestMediaAndVarsComponent(Component):
template = """
<div>Full component</div>
{% component "test_media_no_vars" / %}
"""
js = "console.log('Hello from full component');"
css = ".full-component { color: blue; }"
def get_js_data(self):
return {"message": "Hello"}
def get_css_data(self):
return {"color": "blue"}
# Register our test cache
from django.core.cache import caches
caches["test-cache"] = self.test_cache
# Render the components to trigger caching
TestMediaAndVarsComponent.render()
# Check that JS/CSS is cached for components that have them
self.assertTrue(self.test_cache.has_key(f"__components:{TestMediaAndVarsComponent._class_hash}:js"))
self.assertTrue(self.test_cache.has_key(f"__components:{TestMediaAndVarsComponent._class_hash}:css"))
self.assertTrue(self.test_cache.has_key(f"__components:{TestMediaNoVarsComponent._class_hash}:js"))
self.assertTrue(self.test_cache.has_key(f"__components:{TestMediaNoVarsComponent._class_hash}:css"))
self.assertFalse(self.test_cache.has_key(f"__components:{TestSimpleComponent._class_hash}:js"))
self.assertFalse(self.test_cache.has_key(f"__components:{TestSimpleComponent._class_hash}:css"))
# Check that we cache `Component.js` / `Component.css`
self.assertEqual(
self.test_cache.get(f"__components:{TestMediaNoVarsComponent._class_hash}:js").strip(),
"console.log('Hello from JS');",
)
self.assertEqual(
self.test_cache.get(f"__components:{TestMediaNoVarsComponent._class_hash}:css").strip(),
".novars-component { color: blue; }",
)
# Check that we cache JS / CSS scripts generated from `get_js_data` / `get_css_data`
# NOTE: The hashes is generated from the data.
js_vars_hash = "216ecc"
css_vars_hash = "d039a3"
# TODO - Update once JS and CSS vars are enabled
self.assertEqual(
self.test_cache.get(f"__components:{TestMediaAndVarsComponent._class_hash}:js:{js_vars_hash}").strip(),
"",
)
self.assertEqual(
self.test_cache.get(f"__components:{TestMediaAndVarsComponent._class_hash}:css:{css_vars_hash}").strip(),
"",
)

View file

@ -30,12 +30,15 @@ class BaseTestCase(SimpleTestCase):
super().tearDown()
registry.clear()
from django_components.template import template_cache
from django_components.cache import component_media_cache, template_cache
# NOTE: There are 1-2 tests which check Templates, so we need to clear the cache
if template_cache:
template_cache.clear()
if component_media_cache:
component_media_cache.clear()
from django_components.component import component_node_subclasses_by_name
component_node_subclasses_by_name.clear()
@ -180,11 +183,14 @@ def parametrize_context_behavior(cases: List[ContextBehParam], settings: Optiona
self._start_gen_id_patch()
# Reset template cache
from django_components.template import template_cache
from django_components.cache import component_media_cache, template_cache
if template_cache: # May be None if the cache was not initialized
template_cache.clear()
if component_media_cache:
component_media_cache.clear()
from django_components.component import component_node_subclasses_by_name
component_node_subclasses_by_name.clear()