DjangoTricks blog

Wagtail Routable Pages and Layout Configuration

Sat, 21 Mar 2026 12:00:00 +0000

If you are familiar with Wagtail CMS for Django, you know that you can create Wagtail pages and control their content and layout with blocks inside of stream fields. But what if you have entries coming from normal Django models through a routable page? In this article, I will explore how you can control the dynamic layout of a detail view in a routable page.

Routable pages in Wagtail are dynamic pages of your CMS page tree that can have their own URL subpaths and views. You can use them for filtered list and detail views, multi-step forms, multiple formats for the same data, etc. Here I will show you a routable ArticleIndexPage with a list and detail views for Article instances rendering the detail views based on the block layout in a detail_layout stream field.

1. Project Setup

Create a Wagtail project myproject and articles app:

pip install wagtail
wagtail start myproject
cd myproject
python manage.py startapp articles

Add to INSTALLED_APPS in your Django project settings:

INSTALLED_APPS = [
    ...
    "wagtail.contrib.routable_page",  # required for RoutablePage
    "myproject.apps.articles",
]

2. File Structure

The articles app:

myproject/apps/articles/
├── __init__.py
├── apps.py
├── models.py          # Article, Category, ArticleIndexPage
├── blocks.py          # All StreamField block definitions
└── admin.py           # Register Article and Category in Django admin

The articles templates:

myproject/templates/articles/
├── article_list.html           # List view
├── article_detail.html         # Detail view
└── blocks/
    ├── cover_image_block.html
    ├── description_block.html
    └── related_articles_block.html

3. Models

`myproject/apps/articles/models.py`

Create the Category and Article Django models, and the ArticleIndexPage routable Wagtail page with article list and detail views:

from django.core.paginator import EmptyPage, PageNotAnInteger, Paginator
from django.db import models
from django.shortcuts import get_object_or_404
from django.utils.translation import gettext_lazy as _

from wagtail.admin.panels import FieldPanel, ObjectList, TabbedInterface
from wagtail.contrib.routable_page.models import RoutablePageMixin, path
from wagtail.fields import StreamField
from wagtail.models import Page

from .blocks import article_detail_layout_blocks


class Category(models.Model):
    name = models.CharField(max_length=100, verbose_name=_("name"))
    slug = models.SlugField(unique=True, verbose_name=_("slug"))

    class Meta:
        verbose_name = _("category")
        verbose_name_plural = _("categories")

    def __str__(self):
        return self.name


class Article(models.Model):
    title = models.CharField(max_length=255, verbose_name=_("title"))
    slug = models.SlugField(unique=True, verbose_name=_("slug"))
    category = models.ForeignKey(
        Category,
        null=True,
        blank=True,
        on_delete=models.SET_NULL,
        related_name="articles",
        verbose_name=_("category"),
    )
    cover_image = models.ForeignKey(
        "wagtailimages.Image",
        null=True,
        blank=True,
        on_delete=models.SET_NULL,
        related_name="+",
        verbose_name=_("cover image"),
    )
    description = models.TextField(blank=True, verbose_name=_("description"))
    created_at = models.DateTimeField(auto_now_add=True, verbose_name=_("created at"))

    class Meta:
        verbose_name = _("article")
        verbose_name_plural = _("articles")

    def __str__(self):
        return self.title


class ArticleIndexPage(RoutablePageMixin, Page):
    """
    A single Wagtail page that owns:
      - /articles/          → paginated list of all Articles
      - /articles/<slug>/   → detail view for one Article

    The StreamField is edited once in the Wagtail admin and
    defines the layout for every detail view.
    """

    articles_per_page = models.IntegerField(default=10, verbose_name=_("articles per page"))

    detail_layout = StreamField(
        article_detail_layout_blocks(),
        blank=True,
        use_json_field=True,
        verbose_name=_("detail layout"),
        help_text=_(
            "Configure the layout for all article detail pages. "
            "Add, remove, and reorder blocks to change what appears "
            "on every article detail view."
        ),
    )

    # TabbedInterface gives List View and Detail View their own tabs.
    # promote_panels and settings_panels must be added explicitly here
    # because edit_handler takes full ownership of the admin UI structure.
    edit_handler = TabbedInterface([
        ObjectList(Page.content_panels + [FieldPanel("articles_per_page")], heading=_("List View")),
        ObjectList([FieldPanel("detail_layout")], heading=_("Detail View")),
        ObjectList(Page.promote_panels, heading=_("SEO / Promote")),
        ObjectList(Page.settings_panels, heading=_("Settings")),
    ])

    class Meta:
        verbose_name = _("article index page")
        verbose_name_plural = _("article index pages")

    @path("")
    def article_list(self, request):
        all_articles = Article.objects.select_related("category", "cover_image").order_by("-created_at")
        paginator = Paginator(all_articles, self.articles_per_page)
        page_number = request.GET.get("page")

        try:
            articles = paginator.page(page_number)
        except PageNotAnInteger:
            articles = paginator.page(1)
        except EmptyPage:
            articles = paginator.page(paginator.num_pages)

        return self.render(
            request,
            context_overrides={"articles": articles, "paginator": paginator},
            template="articles/article_list.html",
        )

    @path("<slug:article_slug>/")
    def article_detail(self, request, article_slug):
        article = get_object_or_404(
            Article.objects.select_related("category", "cover_image"),
            slug=article_slug,
        )
        return self.render(
            request,
            context_overrides={"article": article},
            template="articles/article_detail.html",
        )

4. StreamField Blocks

`myproject/apps/articles/blocks.py`

Create Wagtail stream-field blocks for the cover image, description, and the related articles of an actual article. Each block can have some settings on how to represent the content of the block.

from django.utils.translation import gettext_lazy as _

from wagtail import blocks


class CoverImageBlock(blocks.StructBlock):
    aspect_ratio = blocks.ChoiceBlock(
        choices=[
            ("16-9", _("16:9 Widescreen")),
            ("4-3", _("4:3 Standard")),
            ("1-1", _("1:1 Square")),
            ("3-1", _("3:1 Banner")),
        ],
        default="16-9",
        label=_("Aspect ratio"),
        help_text=_("Controls the cropping of the cover image."),
    )

    class Meta:
        template = "articles/blocks/cover_image_block.html"
        icon = "image"
        label = _("Cover Image")


class DescriptionBlock(blocks.StructBlock):
    max_lines = blocks.IntegerBlock(
        min_value=0,
        default=0,
        label=_("Maximum lines"),
        help_text=_("Clamp the description to this many lines. Set to 0 to show all."),
        required=False,
    )

    class Meta:
        template = "articles/blocks/description_block.html"
        icon = "pilcrow"
        label = _("Description")


class RelatedArticlesBlock(blocks.StructBlock):
    sort_order = blocks.ChoiceBlock(
        choices=[
            ("newest", _("Newest first")),
            ("oldest", _("Oldest first")),
            ("title_asc", _("Title A → Z")),
            ("title_desc", _("Title Z → A")),
        ],
        default="newest",
        label=_("Sort order"),
        help_text=_("Order in which related articles are listed."),
    )

    def get_context(self, value, parent_context=None):
        context = super().get_context(value, parent_context=parent_context)
        article = (parent_context or {}).get("article")
        if not article or not article.category_id:
            context["related_articles"] = []
            return context

        from .models import Article

        sort_map = {
            "newest": "-created_at",
            "oldest": "created_at",
            "title_asc": "title",
            "title_desc": "-title",
        }

        context["related_articles"] = (
            Article.objects.select_related("category", "cover_image")
            .filter(category=article.category)
            .exclude(pk=article.pk)
            .order_by(sort_map.get(value["sort_order"], "-created_at"))[:3]
        )
        return context

    class Meta:
        template = "articles/blocks/related_articles_block.html"
        icon = "list-ul"
        label = _("Related Articles")


def article_detail_layout_blocks():
    """
    Returns the list of (name, block) tuples used in ArticleIndexPage.detail_layout.
    Defined as a function so models.py can import it without circular issues.
    """
    return [
        ("cover_image", CoverImageBlock()),
        ("description", DescriptionBlock()),
        ("related_articles", RelatedArticlesBlock()),
    ]

The RelatedArticlesBlock here also has a customized context where we pass related_articles variable with 3 other articles of the same category sorted by the sorting order defined in the block.

5. Templates

`articles/article_list.html`

This will be the template for the paginated article list. Later you could augment it with a search form and filters.

{% extends "base.html" %}
{% load wagtailcore_tags wagtailimages_tags i18n wagtailroutablepage_tags %}

{% block content %}
<main class="article-index">
  <h1>{{ page.title }}</h1>
  <ul class="article-list">
    {% for article in articles %}
      <li class="article-card">
        {% if article.cover_image %}{% image article.cover_image width-400 as img %}
          <img src="{{ img.url }}" alt="{{ article.title }}">
        {% endif %}
        <h2>
          <a href="{% routablepageurl page "article_detail" article.slug %}">{{ article.title }}</a>
        </h2>
        {% if article.category %}<span class="badge">{{ article.category.name }}</span>{% endif %}
        <p>{{ article.description|truncatewords:30 }}</p>
      </li>
    {% empty %}
      <li>{% trans "No articles yet." %}</li>
    {% endfor %}
  </ul>
  {% if articles.has_other_pages %}
    <nav class="pagination" aria-label="{% trans 'Article pagination' %}">
      {% if articles.has_previous %}
        <a href="?page={{ articles.previous_page_number }}">{% trans "← Previous" %}</a>
      {% endif %}
      <span>{% blocktrans with num=articles.number total=articles.paginator.num_pages %}Page {{ num }} of {{ total }}{% endblocktrans %}</span>
      {% if articles.has_next %}
        <a href="?page={{ articles.next_page_number }}">{% trans "Next →" %}</a>
      {% endif %}
    </nav>
  {% endif %}
</main>
{% endblock %}

`articles/article_detail.html`

The detail page would use the {% include_block page.detail_layout with article=article page=page %} to pass the article to the context of each block:

{% extends "base.html" %}
{% load i18n wagtailcore_tags wagtailroutablepage_tags %}

{% block content %}
<article class="article-detail">
  <header>
    <h1>{{ article.title }}</h1>
    {% if article.category %}<span class="badge">{{ article.category.name }}</span>{% endif %}
  </header>
  {% include_block page.detail_layout with article=article page=page %}
  <p>
    <a href="{% routablepageurl page "article_list" %}">{% trans "← Back to all articles" %}</a>
  </p>
</article>
{% endblock %}

`articles/blocks/cover_image_block.html`

Cover image block would show the article cover image with the aspect ratio set in the block:

{% load wagtailimages_tags %}

{% if article.cover_image %}
  <div class="cover-image cover-image--{{ value.aspect_ratio }}">
    {% image article.cover_image width-1200 as img %}
    <img src="{{ img.url }}" alt="{{ article.title }}">
  </div>
{% endif %}

`articles/blocks/description_block.html`

Description block would hide the article description text overflow based on the max lines set in the block:

<section class="article-description">
  <p{% if value.max_lines > 0 %} class="line-clamp" style="-webkit-line-clamp: {{ value.max_lines }};"{% endif %}>
    {{ article.description }}
  </p>
</section>

`articles/blocks/related_articles_block.html`

The related articles block would list the related articles as defined in the extra context of the block:

{% load i18n wagtailimages_tags wagtailroutablepage_tags %}

{% if related_articles %}
  <section class="related-articles">
    <h2>{% trans "Related Articles" %}</h2>
    <ul class="related-articles__list">
      {% for rel in related_articles %}
        <li class="related-card">
          {% if rel.cover_image %}{% image rel.cover_image width-400 as img %}
            <img src="{{ img.url }}" alt="{{ rel.title }}">
          {% endif %}
          <div class="related-card__body">
            {% if rel.category %}<span class="badge">{{ rel.category.name }}</span>{% endif %}
            <h3>
              <a href="{% routablepageurl page "article_detail" rel.slug %}">{{ rel.title }}</a>
            </h3>
            <p>{{ rel.description|truncatewords:20 }}</p>
          </div>
        </li>
      {% endfor %}
    </ul>
  </section>
{% endif %}

6. Django Admin Registration

`articles/admin.py`

Let's not forget to register admin views for the categories and articles so that we can add some data there:

from django.contrib import admin
from .models import Article, Category


@admin.register(Category)
class CategoryAdmin(admin.ModelAdmin):
    list_display = ("name", "slug")
    prepopulated_fields = {"slug": ("name",)}


@admin.register(Article)
class ArticleAdmin(admin.ModelAdmin):
    list_display = ("title", "category", "created_at")
    list_filter = ("category",)
    search_fields = ("title", "description")
    prepopulated_fields = {"slug": ("title",)}

7. Migrations and Initial Data

python manage.py makemigrations articles
python manage.py migrate
python manage.py createsuperuser
python manage.py runserver

8. Wagtail Admin Setup

Open http://localhost:8000/cms/ and log in.
In the Pages explorer, create an Article Index Page as a child of the root page.
- Set the Slug to articles.
On the List View tab, set Articles per page (e.g. 24).
On the Detail View tab, open the Detail Layout StreamField and add blocks in your preferred order:
- Cover Image — choose an aspect ratio.
- Description — optionally set a maximum line count to clamp long descriptions.
- Related Articles — choose the sort order for the three related articles shown.
Publish the page.
In the Django admin (/django-admin/), create some Categories and Articles with cover images and descriptions.
Visit http://localhost:8000/articles/ for the paginated list.
Click any article to see the detail view rendered using the StreamField layout you configured in step 4.

Final words

Using stream fields we can render not only editorial content, for example, images or rich-text descriptions, but also dynamic content based on values from other models and/or the context of the given template.

The approach illustrated in this article allows us to create Wagtail pages where content editors have freedom to adjust the layouts of the pages or insert blocks, such as ads or info texts, into specific places based on real-time events.

How to Show a Waitlist Until Your Wagtail Site Is Ready

Fri, 20 Mar 2026 12:00:00 +0000

This year, I want to bring my centralized gamified donation platform www.make-impact.org to life (at least technically). Earlier I had the version I was developing separate from the waiting list, but I decided to merge them and have a switch between the waitlist and an early preview.

This allows me to have no data duplication, the possibility to create user accounts immediately, and saves hosting and maintenance costs.

This guide walks through a pattern that lets you ship a temporary waitlist page while your Wagtail site is still being built, with the ability to show your progress to chosen people. If you are building a Software as a Service (SaaS) or a web platform with Django, this article is for you.

The Concept

A custom start page view will check for a specific cookie value. If it is unset, the visitor will be redirected to a waitlist form at /waitlist/. If it is set, the visitor will be served the Wagtail home page.

All views under development will have a decorator that checks the cookie value and redirects to the start page if it is unset.

There will be a special view at /preview-access/ with a passphrase form that allows the visitor to gain preview access by setting the mentioned cookie. This view will also allow preview access to be deactivated.

These are the steps to implement this:

1. Generate and store two secrets

You will need two secret values, either set manually or generated with a cryptographically secure random generator (e.g. Python's secrets module):

PREVIEW_ACCESS_PASSPHRASE — the human-readable passphrase typed into the form. Share this with the people who need site access.
PREVIEW_ACCESS_TOKEN — the opaque random value stored in the cookie. Never exposed to users; only the server compares against it.

>>> import secrets
>>> print(secrets.token_urlsafe(16))  # passphrase
dI5nGNftZOBx8m-r0m6glg
>>> print(secrets.token_hex(32))      # cookie token
c1b7a76e3ad5cbfb1657fa4e9885a3c8baa6a5a869f49a136abd0e873a9be9ee

Add both to environment variables or a secrets file untracked by Git, and load them in the Django project settings:

# myproject/settings/_base.py
PREVIEW_ACCESS_PASSPHRASE = get_secret("PREVIEW_ACCESS_PASSPHRASE")
PREVIEW_ACCESS_TOKEN = get_secret("PREVIEW_ACCESS_TOKEN")

The get_secret() here is my custom function to retrieve a secret from the secrets source.

2. Create the access-control decorator

Create myproject/apps/misc/decorators.py. Every protected view will import from here.

# myproject/apps/misc/decorators.py
from functools import wraps

from django.conf import settings
from django.shortcuts import redirect


def preview_access_required(view_func):
    @wraps(view_func)
    def wrapper(request, *args, **kwargs):
        if request.COOKIES.get("preview_access") == settings.PREVIEW_ACCESS_TOKEN:
            return view_func(request, *args, **kwargs)
        return redirect("misc:home_page")
    return wrapper

The decorator compares the cookie against the opaque unguessable token from settings, so unless the token value is known, a random attacker cannot gain access by setting the cookie manually in DevTools.

3. Create the passphrase form

Create myproject/apps/misc/forms.py. The form will have a single required password field. Validation will reject any value that does not match the setting.

# myproject/apps/misc/forms.py
from django import forms
from django.conf import settings
from django.utils.translation import gettext_lazy as _


class PreviewAccessForm(forms.Form):
    passphrase = forms.CharField(
        label=_("Passphrase"),
        widget=forms.PasswordInput(
            attrs={"autocomplete": "current-password"}
        ),
        required=True,
    )

    def clean_passphrase(self):
        value = self.cleaned_data["passphrase"]
        if value != settings.PREVIEW_ACCESS_PASSPHRASE:
            raise forms.ValidationError(
                _("Incorrect passphrase.")
            )
        return value

4. Build the cookie toggle view

Point your browser to /preview-access/. When access is off it shows a passphrase form; when access is on it shows a disable button.

# myproject/apps/misc/views.py
from django.conf import settings
from django.shortcuts import redirect, render

from .forms import PreviewAccessForm


def preview_access(request):
    has_access = request.COOKIES.get("preview_access") == settings.PREVIEW_ACCESS_TOKEN

    if request.method == "POST":
        if has_access:
            response = redirect("misc:home_page")
            response.delete_cookie("preview_access")
            return response

        form = PreviewAccessForm(request.POST)
        if form.is_valid():
            response = redirect("misc:home_page")
            response.set_cookie(
                "preview_access",
                settings.PREVIEW_ACCESS_TOKEN,
                httponly=True,
                samesite="Strict",
            )
            return response
    else:
        form = PreviewAccessForm()

    return render(
        request, 
        "preview_access/preview_access.html", 
        {"has_access": has_access, "form": form}
    )

Key points: - Disabling never requires the passphrase — the cookie is already proof of prior access. - The cookie is set with httponly=True (not readable by JavaScript) and samesite="Strict" (not sent on cross-site requests). - The cookie value is the opaque token, not "1", so it cannot be guessed.

The template renders the passphrase input only when not has_access, and shows field-level errors from the form if the passphrase is wrong.

5. Wrap the Wagtail catch-all with the decorator

Replace the default Wagtail catch-all route handler with a thin wrapper that enforces the same cookie check.

# myproject/apps/misc/views.py
from myproject.apps.misc.decorators import preview_access_required
from wagtail.views import serve as wagtail_serve


@preview_access_required
def serve_wagtail_page(request, path=""):
    return wagtail_serve(request, path)

Without this, a visitor who knows any Wagtail page URL could bypass the gate by typing it directly into the browser.

6. Build the proxy home page view

This view is the only entry point to the site. It decides what every visitor sees first.

# myproject/apps/misc/views.py
from django.conf import settings

from wagtail.models import Site
from wagtail.views import serve as wagtail_serve


def home_page(request):
    if request.COOKIES.get("preview_access") == settings.PREVIEW_ACCESS_TOKEN:
        # serve Wagtail directly
        site = Site.find_for_request(request)
        return wagtail_serve(request, "")  

    # redirect to waiting list
    return redirect("waiting_list")

Key point: the waiting_list view and a Wagtail Site and page must exist and be matched to the request domain before wagtail_serve is called.

7. Wire up the URLs

Django project URL rules:

# myproject/urls.py
from django.conf.urls.i18n import i18n_patterns
from django.urls import re_path

from wagtail.coreutils import WAGTAIL_APPEND_SLASH

from myproject.apps.misc import views as misc_views

if WAGTAIL_APPEND_SLASH:
    wagtail_serve_pattern = r"^((?:[\w\-]+/)*)$"
else:
    wagtail_serve_pattern = r"^([\w\-/]*)$"

urlpatterns += i18n_patterns(
    # ... all your other app URLs above ...

    # Catch-all — must be last
    re_path(
        wagtail_serve_pattern,
        misc_views.serve_wagtail_page,
        name="wagtail_serve"
    ),
)

The misc app URLs:

# myproject/apps/misc/urls.py
from django.urls import path

from . import views

app_name = "misc"

urlpatterns = [
    path("", views.home_page, name="home_page"),
    path("preview-access/", views.preview_access, name="preview_access"),
]

The waiting_list app URLs:

# myproject/apps/waiting_list/urls.py
from django.urls import path

from . import views

urlpatterns = [
    path("waitlist/", views.show_waiting_list_form, name="waiting_list"),
]

8. Protect every other app view

Import and apply @preview_access_required to every view that belongs to the real site. Class-based views can be wrapped at assignment time:

from myproject.apps.misc.decorators import preview_access_required

# Function-based view
@preview_access_required
def event_list(request): 
    ...

# Class-based view
event_list = preview_access_required(
    EventListView.as_view()
)

Waiting-list views, API views, social authentication views, and static/legal pages (/imprint/, /privacy/, etc.) must not receive this decorator — they need to remain publicly accessible.

Final words

You get a lot of benefits from this setup. The waitlist measures demand for your website while you are still building. Invited test users can evaluate your progress at any time. While you are developing the website, you do not necessarily need multiple servers. Launching later is also easier — no hassle or delays with domain IP updates and SSL certificates.

Using tox to Test a Django App Across Multiple Django Versions

Fri, 27 Feb 2026 12:00:00 +0000

Recently, I developed a reusable Django app django-clearplaintext for normalizing plain text in Django templates. And to package and test it properly, I had a fresh look at Tox.

Tox is the standard testing tool that creates isolated virtual environments, installs the exact dependencies you specify, and runs your test suite in each one — all from a single command.

This post walks through a complete, working setup using a minimal example app called django-shorturl.

The Example App: django-shorturl

django-shorturl is a self-contained Django app with one model and one view.

shorturl/models.py

from django.db import models
from django.utils.translation import gettext_lazy as _

class ShortLink(models.Model):
    slug = models.SlugField(_("slug"), unique=True)
    target_url = models.URLField(_("target URL"))
    created_at = models.DateTimeField(_("created at"), auto_now_add=True)

    class Meta:
        verbose_name = _("short link")
        verbose_name_plural = _("short links")

    def __str__(self):
        return self.slug

shorturl/views.py

from django.shortcuts import get_object_or_404, redirect

from .models import ShortLink

def redirect_link(request, slug):
    link = get_object_or_404(ShortLink, slug=slug)
    return redirect(link.target_url)

shorturl/urls.py

from django.urls import path

from . import views

urlpatterns = [
    path("<slug:slug>/", views.redirect_link, name="redirect_link"),
]

shorturl/admin.py

from django.contrib import admin
from .models import ShortLink

admin.site.register(ShortLink)

Project Layout

django-shorturl/
├── src/
│   └── shorturl/
│       ├── __init__.py
│       ├── admin.py
│       ├── models.py
│       ├── views.py
│       └── urls.py
├── tests/
│   ├── __init__.py
│   └── test_views.py
├── pyproject.toml
├── test_settings.py
└── tox.ini

The source lives under src/ and the tests are at the top level, separate from the package. This separation prevents the tests from accidentally being shipped inside the installed package.

Packaging: pyproject.toml

Tox needs a properly packaged app to install into each environment. With isolated_build = true (more on that below), Tox builds a wheel from your pyproject.toml before running any tests.

pyproject.toml

[project]
name = "django-shorturl"
version = "1.0.0"
requires-python = ">=3.8"
dependencies = [
    "Django>=4.2",
]

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[tool.setuptools.packages.find]
where = ["src"]

The dependencies list here declares the runtime minimum — your app needs Django, but you don't pin a specific version because that is Tox's job during testing.

For the [build-system] section, we can also use uv_build to gain some performance improvements:

[build-system]
requires = ["uv_build >= 0.10.0, <0.11.0"]
build-backend = "uv_build"

[tool.uv.build-backend]
module-name = "shorturl"

Here module-name lets uv_build not to get confused between django-shorturl and shorturl.

Test Settings: test_settings.py

Django requires a settings module to run. As we don't have an associated project, we have to create a minimal one by defining project settings in the project's settings, create a minimal one dedicated to testing. It lives at the repo root so it's easy to point to from anywhere.

test_settings.py

SECRET_KEY = "test"

INSTALLED_APPS = [
    "shorturl",
]

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.sqlite3",
        "NAME": ":memory:",
    }
}

ROOT_URLCONF = "shorturl.urls"

DEFAULT_AUTO_FIELD = "django.db.models.AutoField"

A few deliberate choices here:

SECRET_KEY = "test" — A fixed value, fine for tests, but never use this in production.
INSTALLED_APPS — Only include apps that your tests actually need. No django.contrib.admin, no auth, nothing extra.
SQLite in-memory database — ":memory:" means the database is created fresh for every test run and disappears when the process exits. No files left behind, no teardown needed, and it is fast.
ROOT_URLCONF — The test client resolves URLs through this setting. Without it, reverse() raises NoReverseMatch and the test client has no URL configuration to dispatch against. Point it at your app's urls.py.
DEFAULT_AUTO_FIELD — Suppresses Django's system check warning about the implicit primary key type. Setting it explicitly keeps the test output clean and makes the expectation clear.

The Core: tox.ini

This is where Tox is configured.

tox.ini

[tox]
envlist =
    py{38,39,310,311,312}-django42,
    py{310,311,312}-django50,
    py{310,311,312,313}-django51,
    py{310,311,312,313,314}-django52,
    py{312,313,314}-django60

isolated_build = true

[testenv]
deps =
    django42: Django>=4.2,<4.3
    django50: Django>=5.0,<5.1
    django51: Django>=5.1,<5.2
    django52: Django>=5.2,<6.0
    django60: Django>=6.0,<6.1
commands =
    python -m django test
setenv =
    DJANGO_SETTINGS_MODULE = test_settings

envlist — the matrix

py{38,39,310,311,312}-django42 is a shortcut used in Tox.

The numbers inside {} are expanded automatically. Tox combines each Python version with django42, creating 5 environments:

py38-django42
py39-django42
py310-django42
py311-django42
py312-django42

The full envlist simply lists all Python and Django combinations you want to test, so you can check that your project works in each setup.

Each part separated by a dash in an environment name is called a "factor". You can have as many factors as you like, and they can be named anything. py* factors are a convention for Python versions. Others need to be defined in the [testenv] deps section.

isolated_build = true

This tells tox to build a proper wheel from your pyproject.toml before installing into each environment. Without it, tox would try to install your package with pip install -e ., which bypasses the build system and can hide packaging bugs. With it, each environment tests the package exactly as a user would receive it after pip install django-shorturl.

deps — conditional dependencies

The django42: prefix is a Tox factor condition: the dependency on that line is only installed when the environment name contains the django42 factor. This is how a single [testenv] block handles all Django versions without needing a separate section for each one.

Tox also installs your package itself into each environment (because of isolated_build), so you don't need to list it here.

commands

commands =
    python -m django test

python -m django test is Django's built-in test runner. It discovers tests by looking for files matching test*.py under the current directory, which picks up everything in your tests/ folder automatically.

setenv

setenv =
    DJANGO_SETTINGS_MODULE = test_settings

Django refuses to run without a settings module. This environment variable tells it where to find yours. Because test_settings.py is at the repo root and tox runs from the repo root, the module name test_settings resolves correctly without any path manipulation.

Writing the Tests

Create test cases for each (critical) component of your app. For example, if you have models, views, and template tags, create tests/test_models.py, tests/test_views.py, and tests/test_templatetags.py.

tests/test_views.py

from django.test import TestCase
from django.urls import reverse

from shorturl.models import ShortLink


class RedirectLinkViewTest(TestCase):
    def setUp(self):
        ShortLink.objects.create(
            slug="dt",
            target_url="https://www.djangotricks.com",
        )

    def test_redirects_to_target_url(self):
        response = self.client.get(
            reverse(
                "redirect_link", kwargs={"slug": "dt"}
            )
        )
        self.assertRedirects(
            response,
            "https://www.djangotricks.com",
            fetch_redirect_response=False,
        )

    def test_returns_404_for_unknown_slug(self):
        response = self.client.get(
            reverse(
                "redirect_link", kwargs={"slug": "nope"}
            )
        )
        self.assertEqual(response.status_code, 404)

Installing Python Versions with pyenv

Tox needs the actual Python binaries for every version in your envlist. If you try to run tox without them installed, it will fail immediately with an InterpreterNotFound error. pyenv is the standard way to install and manage multiple Python versions side by side.

Install pyenv

Use Homebrew on macOS (or follow the official instructions for Linux):

brew install pyenv

Add the following to your shell config (~/.zshrc, ~/.bashrc, etc.) and restart your shell:

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Install each Python version

Install every version that appears in your envlist:

pyenv install 3.8
pyenv install 3.9
pyenv install 3.10
pyenv install 3.11
pyenv install 3.12
pyenv install 3.13
pyenv install 3.14

Make them all reachable at once

Tox resolves py312 by looking for a binary named python3.12 on PATH. The trick is pyenv global, which accepts multiple versions and places all of their binaries on your PATH simultaneously:

pyenv global 3.14 3.13 3.12 3.11 3.10 3.9 3.8

List the first (the one python3 and python resolve to) and work downward. After running this, confirm every interpreter is visible:

python3.8 --version   # Python 3.8.x
python3.9 --version   # Python 3.9.x
python3.10 --version   # Python 3.10.x
python3.11 --version   # Python 3.11.x
python3.12 --version   # Python 3.12.x
python3.13 --version   # Python 3.13.x
python3.14 --version   # Python 3.14.x

Now tox can find all of them and the full matrix will run without InterpreterNotFound errors.

Running tox

Run the full matrix:

tox

Or run a single environment:

tox -e py312-django52

tox will print a summary at the end showing which environments passed and which failed.

  py38-django42: OK (3.25=setup[2.32]+cmd[0.93] seconds)
  py39-django42: OK (2.88=setup[2.16]+cmd[0.72] seconds)
  py310-django42: OK (2.61=setup[2.02]+cmd[0.59] seconds)
  py311-django42: OK (2.70=setup[2.09]+cmd[0.61] seconds)
  py312-django42: OK (3.28=setup[2.46]+cmd[0.82] seconds)
  py310-django50: OK (2.67=setup[2.09]+cmd[0.58] seconds)
  py311-django50: OK (2.61=setup[2.02]+cmd[0.59] seconds)
  py312-django50: OK (2.85=setup[2.25]+cmd[0.60] seconds)
  py310-django51: OK (2.81=setup[2.27]+cmd[0.54] seconds)
  py311-django51: OK (2.85=setup[2.30]+cmd[0.55] seconds)
  py312-django51: OK (2.70=setup[2.09]+cmd[0.61] seconds)
  py313-django51: OK (2.97=setup[2.29]+cmd[0.68] seconds)
  py310-django52: OK (3.03=setup[2.31]+cmd[0.72] seconds)
  py311-django52: OK (2.88=setup[2.22]+cmd[0.66] seconds)
  py312-django52: OK (2.80=setup[2.13]+cmd[0.67] seconds)
  py313-django52: OK (4.70=setup[3.66]+cmd[1.04] seconds)
  py314-django52: OK (6.41=setup[5.18]+cmd[1.23] seconds)
  py312-django60: OK (5.13=setup[4.06]+cmd[1.07] seconds)
  py313-django60: OK (5.35=setup[4.15]+cmd[1.21] seconds)
  py314-django60: OK (6.01=setup[4.65]+cmd[1.37] seconds)
  congratulations :) (70.59 seconds)

Final Words

What makes this setup robust?

No shared state between environments. Each Tox environment is its own virtualenv with its own Django installation.
The package is built, not symlinked. isolated_build = true catches packaging mistakes before they reach users.
The database never persists between runs. SQLite in-memory means no stale data, no cleanup scripts, no CI-specific teardown.
The test settings are minimal by design. Fewer installed apps means faster startup, fewer implicit dependencies, and tests that fail for clear, local reasons rather than configuration noise from elsewhere in the project.

This setup is not the only way to test a Django app with Tox, but it is a solid starting point that balances comprehensiveness with maintainability. With a little effort upfront, you can ensure your app works across a wide range of Python and Django versions — and catch packaging bugs before they hit real users.

Understanding the Different POST Content Types

Sun, 16 Nov 2025 12:00:00 +0000

After more than 20 years of building for the web, this topic somehow kept slipping past me. It always felt obvious, so I never looked deeper. Recently I finally took the time to explore it properly, did some quick research, and now I’m sharing the results. Here’s a simple walkthrough of the different content types you can send in POST requests.

Standard Form Data

When you submit a basic HTML form like <form method="post" action=""></form>, for example a login form, the browser sends the data using the application/x-www-form-urlencoded content type. The body of the request looks like a URL-encoded query string, the same format typically used in GET requests.

Example: username=john_doe&password=pass123.

A POST request with this content type using the fetch API looks like this:

async function sendURLEncoded() {
    const params = new URLSearchParams();
    params.append('username', 'john_doe');
    params.append('email', 'john@example.com');
    params.append('password', 'Secret123');
    params.append('bio', 'This is a multi-line\nbio text.\nIt supports newlines!');

    const response = await fetch('/api/urlencoded/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/x-www-form-urlencoded',
            'X-CSRFToken': getCSRFToken()
        },
        body: params
    });
    const result = await response.json();
    console.log(result);
}

On the Django side, validation with a Django form is the usual approach:

from django import forms
from django.http import JsonResponse
from django.views.decorators.http import require_http_methods
from .forms import URLEncodedForm


class URLEncodedForm(forms.Form):
    username = forms.CharField(
        max_length=150,
        required=True,
        min_length=3,
    )
    email = forms.EmailField(
        required=True,
        error_messages={
            "required": "Email is required",
            "invalid": "Please enter a valid email address",
        },
    )
    password = forms.CharField(
        required=True,
        min_length=8,
    )
    bio = forms.CharField(
        required=False,
        max_length=500,
        widget=forms.Textarea,
        error_messages={"max_length": "Bio cannot exceed 500 characters"},
    )


@require_http_methods(["POST"])
def handle_urlencoded(request):
    """Handle application/x-www-form-urlencoded requests"""
    form = URLEncodedForm(request.POST)

    if not form.is_valid():
        return JsonResponse(
            {
                "status": "error",
                "errors": form.errors,
                "content_type": request.content_type,
            },
            status=400,
        )

    return JsonResponse(
        {
            "status": "success",
            "form_data": form.cleaned_data,
            "content_type": request.content_type,
        }
    )

Form Data with Files

If your form includes file uploads, the browser switches to multipart/form-data. You enable it in HTML like this:

<form method="post" action="" enctype="multipart/form-data">

A fetch-based upload looks like:

async function sendMultipart() {
    const formData = new FormData();
    formData.append('username', 'john_doe');
    formData.append('email', 'john@example.com');

    const fileInput = document.getElementById('file-input');
    if (fileInput.files[0]) {
        formData.append('avatar', fileInput.files[0]);
    }

    const response = await fetch('/api/multipart/', {
        method: 'POST',
        headers: {
            'X-CSRFToken': getCSRFToken()
        },
        body: formData
    });
    const result = await response.json();
    console.log(result);
}

The Django view works the same way, just with request.FILES added.

from django import forms
from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


class MultipartForm(forms.Form):
    username = forms.CharField(
        max_length=150,
        required=True,
        min_length=3,
    )
    email = forms.EmailField(
        required=True,
    )
    avatar = forms.FileField(
        required=False,
    )

    def clean_avatar(self):
        avatar = self.cleaned_data.get("avatar")
        if avatar:
            if avatar.size > 5 * 1024 * 1024:
                raise forms.ValidationError("File size cannot exceed 5MB")

            allowed_types = ["image/jpeg", "image/png", "image/gif", "image/webp"]
            if avatar.content_type not in allowed_types:
                raise forms.ValidationError(
                    f'Invalid file type. Allowed types: {", ".join(allowed_types)}'
                )
        return avatar



@require_http_methods(["POST"])
def handle_multipart(request):
    """Handle multipart/form-data requests (forms with files)"""
    form = MultipartForm(request.POST, request.FILES)

    if not form.is_valid():
        return JsonResponse(
            {
                "status": "error",
                "errors": form.errors,
                "content_type": request.content_type,
            },
            status=400,
        )

    form_data = {}

    for key, value in form.cleaned_data.items():
        if hasattr(value, "read"):
            form_data[key] = {
                "name": value.name,
                "size": value.size,
                "content_type": value.content_type,
            }
        else:
            form_data[key] = value

    return JsonResponse(
        {
            "status": "success",
            "form_data": form_data,
            "content_type": request.content_type,
        }
    )

JSON String

A very common modern content type is application/json. Single-Page Applications and most JavaScript-heavy frontends rely on it.

Frontend example:

async function sendJSON() {
    const data = { 
        name: "John Doe", 
        email: "john@example.com", 
        age: 30
    };
    const response = await fetch('/api/json/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'X-CSRFToken': getCSRFToken()
        },
        body: JSON.stringify(data)
    });
    const result = await response.json();
    console.log(result)
}

For validation, you can use Pydantic, which is a nice alternative to Django-REST-Framework serializers.

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods
from pydantic import BaseModel, EmailStr, Field, ConfigDict, ValidationError


class JSONDataSchema(BaseModel):
    model_config = ConfigDict(
        json_schema_extra={
            "example": {
                "name": "John Doe", 
                "email": "john@example.com", 
                "age": 30,
            }
        }
    )

    name: str = Field(min_length=2, max_length=100, description="User's full name")
    email: EmailStr = Field(description="Valid email address")
    age: int = Field(ge=0, le=150, description="User's age")


@require_http_methods(["POST"])
def handle_json(request):
    """Handle application/json requests"""
    try:
        data = json.loads(request.body)
        validated = JSONDataSchema(**data)

        return JsonResponse(
            {
                "status": "success",
                "received": validated.model_dump(),
                "content_type": request.content_type,
            }
        )
    except json.JSONDecodeError:
        return JsonResponse(
            {
                "status": "error",
                "error": "Invalid JSON",
                "content_type": request.content_type,
            },
            status=400,
        )
    except ValidationError as e:
        return JsonResponse(
            {
                "status": "error",
                "errors": e.errors(),
                "content_type": request.content_type,
            },
            status=400,
        )

Newline-Delimited JSON

application/x-ndjson (or simply NDJSON) is an experimental but handy format where each line is a separate JSON object. It’s useful for bulk imports – logs, analytics, and other large datasets.

async function sendNDJSON() {
    const ndjsonData = (
`{"name":"John","age":30}
{"name":"Jane","age":25}
{"name":"Bob","age":35}`
);
    const response = await fetch('/api/ndjson/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/x-ndjson',
            'X-CSRFToken': getCSRFToken()
        },
        body: ndjsonData
    });
    const result = await response.json();
    console.log(result);
}

The processing logic is standard: split, parse, validate.

import json
from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_ndjson(request):
    """Handle application/x-ndjson requests"""
    try:
        ndjson_content = request.body.decode("utf-8")
        lines = [
            json.loads(line) for line in ndjson_content.strip().split("\n") if line
        ]

        return JsonResponse(
            {
                "status": "success",
                "lines_count": len(lines),
                "data": lines,
                "content_type": request.content_type,
            }
        )
    except json.JSONDecodeError:
        return JsonResponse({"error": "Invalid NDJSON"}, status=400)

Plain Text

Some systems accept plain text via text/plain, especially object storage services or endpoints meant for raw logs or unstructured content.

async function sendTextPlain() {
    const textData = `This is plain text content.
It can span multiple lines.
Line 3 of the content.`;

    const response = await fetch('/api/text/', {
        method: 'POST',
        headers: {
            'Content-Type': 'text/plain',
            'X-CSRFToken': getCSRFToken()
        },
        body: textData
    });
    const result = await response.json();
    console.log(result);
}

The Django view is minimal and functional:

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_text_plain(request):
    """Handle text/plain requests"""
    text_content = request.body.decode("utf-8")
    return JsonResponse(
        {
            "status": "success",
            "received": text_content,
            "length": len(text_content),
            "content_type": request.content_type,
        }
    )

HTML Text

You can also POST HTML content using text/html, which can be useful for wiki-style pages or HTML editors that save full documents.

async function sendHTML() {
    const htmlData = `<!DOCTYPE html>
<html>
<head><title>Example</title></head>
<body><h1>Hello World!</h1></body>
</html>`;
    const response = await fetch('/api/html/', {
        method: 'POST',
        headers: {
            'Content-Type': 'text/html',
            'X-CSRFToken': getCSRFToken()
        },
        body: htmlData
    });
    const result = await response.json();
    console.log(result);
}

The Django view would accept the data like this (add your own validation):

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_html(request):
    """Handle text/html requests"""
    html_content = request.body.decode("utf-8")
    return JsonResponse(
        {
            "status": "success",
            "received": html_content,
            "length": len(html_content),
            "content_type": request.content_type,
        }
    )

XML Data

Older or legacy integrations (especially SOAP-based ones) still rely on application/xml.

Here's a JavaScript example:

async function sendXML() {
    const xmlData = `<?xml version="1.0" encoding="UTF-8"?>
<user>
<name>John Doe</name>
<email>john@example.com</email>
<age>30</age>
</user>`;

    const response = await fetch('/api/xml/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/xml',
            'X-CSRFToken': getCSRFToken()
        },
        body: xmlData
    });
    const result = await response.json();
    console.log(result);
}

And the Django view can parse the XML data into a dictionary like this:

import xml.etree.ElementTree as ET
from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_xml(request):
    """Handle application/xml requests"""
    try:
        xml_content = request.body.decode("utf-8")
        root = ET.fromstring(xml_content)

        data = {
            child.tag: child.text 
            for child in root
        }

        return JsonResponse(
            {
                "status": "success",
                "root_tag": root.tag,
                "data": data,
                "content_type": request.content_type,
            }
        )
    except ET.ParseError:
        return JsonResponse({"error": "Invalid XML"}, status=400)

SVG Image

SVG graphics can be sent as image/svg+xml, since they are XML-based.

async function sendSVG() {
    const svgData = `<svg xmlns="http://www.w3.org/2000/svg" width="100" height="100">
<circle cx="50" cy="50" r="40" stroke="black" stroke-width="3" fill="red" />
</svg>`;
    const response = await fetch('/api/svg/', {
        method: 'POST',
        headers: {
            'Content-Type': 'image/svg+xml',
            'X-CSRFToken': getCSRFToken()
        },
        body: svgData
    });
    const result = await response.json();
    console.log(result);
}

The Django view (add your own validation):

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_svg(request):
    """Handle image/svg+xml requests"""
    svg_content = request.body.decode("utf-8")
    return JsonResponse(
        {
            "status": "success",
            "received": svg_content,
            "length": len(svg_content),
            "content_type": request.content_type,
        }
    )

Binary Data

For raw binary streams – images, audio, video, PDF, or any unknown byte sequence – the fallback is application/octet-stream.

Here is a JavaScript example of how to post it:

async function sendBinary() {
    const binaryData = new Uint8Array([72, 101, 108, 108, 111]);
    const response = await fetch('/api/binary/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/octet-stream',
            'X-CSRFToken': getCSRFToken()
        },
        body: binaryData
    });
    const result = await response.json();
    console.log(result);
}

The Django view would be the most straightforward, because request.body is already coming as bytestring:

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_binary(request):
    """Handle application/octet-stream requests (raw binary data)"""
    binary_data = request.body
    return JsonResponse(
        {
            "status": "success",
            "size": len(binary_data),
            "content_type": request.content_type,
            "first_bytes": list(binary_data[:10]),
        }
    )

Protocol Buffers

Protocol Buffers use application/x-protobuf, and compared to JSON they’re usually around 50% smaller and faster to parse. I won't cover the code in this article, but do your research if you need speed.

Code to Play Around with

The rough distribution of content-type usage in real-world APIs is this:

JSON: ~85–90%
Multipart: ~10–15%
Everything else: <5%

If you want to experiment with all the content types, and see the code in context, here's a repo as a great playground: https://github.com/archatas/django-post-content-types

Moving a Domain to Another Registrar

Fri, 17 Oct 2025 12:00:00 +0000

The Situation

The domain for my first SaaS project 1st-things-1st.com was registered with GoDaddy. Even though the whole project was already running under my company’s name, I never really bothered to move the domain to my company’s account at Namecheap.

Last week I noticed that the domain was about to expire, and I thought, alright, time to finally do it.

I had never transferred a domain before, so I wasn’t sure how it would go or whether I could pull it off without any downtime. Here’s how it went.

The Story

Namecheap has this feature called “Transfer to Us.” You just follow a few simple steps: request a transfer for your domain, enter a one-time Auth code (also called as EPP - Extensible Provisioning Protocol - code) from another registrar to confirm you’re the owner, and pay for another year.

At GoDaddy’s side, there was a whole confirmation process to make sure I really wanted to transfer the domain and understood it couldn’t be undone. Once that was done, I got the code I needed for Namecheap.

Then came the waiting part. It took almost a week to get the confirmation that the transfer was successful, which was expected according to their help docs. As soon as I got the notification, I jumped into my Namecheap settings to check if everything looked right.

Unfortunately, the domain records weren’t migrated automatically. Only the ownership was transferred. The nameserver settings in Namecheap were still pointing to GoDaddy’s servers.

And since GoDaddy no longer listed the domain or its DNS records, I had to recover those values myself.

I needed to extract the following records for each subdomain:

A – IPv4 Address
CNAME – Canonical Name
MX – Mail Exchange
TXT – Text

I did it from the command line like this:

$ dig 1st-things-1st.com ANY > 1st-things-1st.txt
$ dig www.1st-things-1st.com ANY >> 1st-things-1st.txt
$ dig my.1st-things-1st.com ANY >> 1st-things-1st.txt
$ dig our.1st-things-1st.com ANY >> 1st-things-1st.txt
$ dig apps.1st-things-1st.com ANY >> 1st-things-1st.txt
$ dig analytics.1st-things-1st.com ANY >> 1st-things-1st.txt

Alternatively, you can look up WHOIS information on sites like who.is.

Once I had all the records, I switched to Namecheap’s nameservers (Namecheap BasicDNS) and added everything manually.

The website was down for about 30 to 60 minutes, depending on where in the world the visitor was. Some downtime was unavoidable, but overall, the transfer went through successfully.

The Learnings

The whole thing could have been smoother if I had known what to expect. Here’s how I’d do it next time:

Save all domain records to a local file before starting.
Get the Auth/EPP code from the old registrar.
Request the transfer at the new registrar.
Wait for the transfer to complete.
Pick the best time for downtime and let users know in advance.
Switch to the new nameservers and add all records manually.

Per-object Permissions for Elasticsearch Lists in Django Websites

Sat, 27 Sep 2025 12:00:00 +0000

The Challenge

Elasticsearch improves the performance of filterable and searchable list views, reducing load times from several seconds to about half a second. It stores list view details from various relations denormalized in a JSON-like structure.

In the Django-based system I’ve been working on, we use django-guardian to set per-object permissions for users or roles managing various items. This means you not only need to check whether a user has general permission to view, change, or delete a type of object, but also whether they have permission to access specific objects.

A major challenge arises when using Elasticsearch for authorized views based on user permissions – how can we check permissions without slowing down the listing too much?

Things We Considered

Here are a few options I compared:

Check all object UUIDs the user can access via django-guardian, then pass those UUIDs to the Elasticsearch search query. This might work with fewer than 100 items, but it doesn’t scale.
Filter the Elasticsearch list first, and then check each item’s UUID against user permissions. With thousands of search results, permission checks become too slow. If I check permissions only for the first page, pagination data becomes inaccurate.
Create a user-permission Elasticsearch index with all item UUIDs accessible to the user, and filter the list by looking up those UUIDs. This makes updating the index tricky, especially for admins and superusers.
For each item, store the list of user IDs and group IDs that can view it, then check the current user against those IDs in the list view. This is the approach I chose, since typically only a handful of users and groups need access to any given item.

Below is the code snippet that implements the last approach.

Our Chosen Approach

We use django-elasticsearch-dsl for indexing Django models in Elasticsearch. The Elasticsearch index document for an item with user IDs and group IDs can look like this:

# items/documents.py
from django.conf import settings
from django_elasticsearch_dsl.registries import registry
from django_elasticsearch_dsl import Document, fields
from guardian.shortcuts import get_users_with_perms, get_groups_with_perms

from .models import Item


@registry.register_document
class ItemDocument(Document):
    users_can_view = fields.KeywordField(multi=True)
    users_can_change = fields.KeywordField(multi=True)
    users_can_delete = fields.KeywordField(multi=True)
    groups_can_view = fields.KeywordField(multi=True)
    groups_can_change = fields.KeywordField(multi=True)
    groups_can_delete = fields.KeywordField(multi=True)

    class Index:
        name = "items"
        settings = {
            "number_of_shards": 1,
            "number_of_replicas": 0,
        }

    class Django:
        model = Item

        fields = [
            "uuid",
            "title",
            "intro",
            "created_at",
            "updated_at",
        ]

        queryset_pagination = 5000

    def prepare(self, instance):
        data = super().prepare(instance)

        data["users_can_view"] = []
        data["users_can_change"] = []
        data["users_can_delete"] = []
        for user, permissions in get_users_with_perms(
            item,
            attach_perms=True,
            with_superusers=True,
            with_group_users=False,
            only_with_perms_in=["view_item", "change_item", "delete_item"],
        ).items():
            if "view_item" in permissions:
                data["users_can_view"].append(user.pk)
            if "change_item" in permissions:
                data["users_can_change"].append(user.pk)
            if "delete_item" in permissions:
                data["users_can_delete"].append(user.pk)

        data["groups_can_view"] = []
        data["groups_can_change"] = []
        data["groups_can_delete"] = []
        for group, permissions in get_groups_with_perms(
            item, attach_perms=True
        ).items():
            for perm in permissions:
                if perm == "view_item":
                    data["groups_can_view"].append(group.pk)
                elif perm == "change_item":
                    data["groups_can_change"].append(group.pk)
                elif perm == "delete_item":
                    data["groups_can_delete"].append(group.pk)

        return data

Next, we need a utility class for paginating Elasticsearch indexes in a way that’s compatible with Django’s default queryset pagination:

# items/utils.py

class ElasticsearchPage:
    """
    Django Paginator-compatible interface for Elasticsearch search results.
    """

    def __init__(self, results, total_count, page_number, items_per_page):
        self.object_list = results
        self.total_count = total_count
        self.number = page_number
        self.paginator = type(
            "Paginator",
            (),
            {
                "count": total_count,
                "num_pages": (total_count + items_per_page - 1) // items_per_page,
                "per_page": items_per_page,
            },
        )()

    def has_previous(self):
        return self.number > 1

    def has_next(self):
        return self.number < self.paginator.num_pages

    def has_other_pages(self):
        return self.paginator.num_pages > 1

    def previous_page_number(self):
        return self.number - 1 if self.has_previous() else None

    def next_page_number(self):
        return self.number + 1 if self.has_next() else None

Finally, the list view checks user IDs and group IDs in the index against the current user’s ID and group memberships:

# items/views.py
from .utils import ElasticsearchPage
from .documents import ItemDocument

@login_required
def item_list(request):
    user_group_pks = list(request.user.groups.values_list("pk", flat=True))

    search_obj = ItemDocument.search()

    perm_filter = Q(
        "bool",
        should=[
            Q("term", users_can_view=request.user.pk),
            Q("terms", groups_can_view=user_group_pks),
        ],
        minimum_should_match=1,
    )

    search_obj = search_obj.query("bool", must=[perm_filter])

    # more search and filtering go here...

    items_per_page = int(request.GET.get("items_per_page", 24))
    page_number = int(request.GET.get("page", 1))
    offset = (page_number - 1) * items_per_page

    total_count = search_obj.count()

    search_obj = search_obj[offset : offset + items_per_page]
    search_results = search_obj.execute()

    page = ElasticsearchPage(
        results=search_results,
        total_count=total_count,
        page_number=page_number,
        items_per_page=items_per_page,
    )
    context = {
        "page": page,
        "items_per_page": items_per_page,
    }
    return render(request, "items/item_list.html", context)

At this point, it’s important to update the index not only when item details change, but also when permissions change.

This can be done by calling the following in the relevant views or form save methods:

from django_elasticsearch_dsl.registries import registry

registry.update(item)

Final Thoughts

Using django-guardian to pre-process or post-process Elasticsearch-filtered lists is inefficient. Instead, permissions should exist directly in the Elasticsearch index. Storing user IDs and group IDs in the items themselves is more practical. Just make sure Elasticsearch is properly secured with SSL/TLS and authentication (username and password) to protect the data from tampering.

User Timezones in Django

Mon, 28 Jul 2025 12:00:00 +0000

When you create a local website, the local time usually matches your country’s timezone, and all visitors see times in that timezone. That’s not a big issue if your country has only one timezone and your audience is local.

But when building a social platform like pybazaar.com, users are international and need to see times in their timezones. In this article, I’ll show you how to handle that in Django.

Time Zone Database

Since version 4.0, Django has used the zoneinfo library for managing timezones, and it used pytz up to version 3.2. Both rely on the IANA Time Zone Database (tzdata). IANA is the same organization that manages the DNS root zone, IP addresses, and other global internet resources.

Install tzdata in your virtual environment as usual:

(venv)$ pip install --upgrade tzdata

Timezone Changes

Timezone information changes several times a year due to:

Daylight Saving Time (DST) adjustments
Political and border changes
Shifts in standard time offset

Daylight Saving Time (DST) was first introduced in 1914 in Canada and later standardized in the U.S. in 1966. When dealing with historic dates before 1966—or future dates with uncertain timezone rules—precise time calculations can be unreliable.

# Before U.S. DST standardization:
old_date = datetime(1960, 6, 15, 12, 0)  

# DST rules may change in the future:
future_date = datetime(2030, 6, 15, 12, 0)

Some timezone changes are driven by politics:

Country splits or mergers — new countries may adopt different timezones.
Regional preferences — states or provinces may change timezone alignment.
Symbolic actions — e.g., North Korea introduced "Pyongyang Time" in 2015 by shifting 30 minutes back to symbolically break from Japan’s colonial legacy.

And countries sometimes adjust their UTC offsets:

Russia — changed its timezone policy multiple times
Venezuela — changed from UTC-4 to UTC-4:30 in 2007, then back in 2016
Samoa — jumped from UTC-11 to UTC+13 in 2011, skipping Dec 30 entirely

Best Practices for Django

Use named timezones, not fixed UTC offsets.
Update tzdata monthly or quarterly.
Test with historic and future dates.
Handle conversion errors gracefully, falling back to UTC.
Store all times in UTC internally.
Convert to user’s timezone only in the UI.
Include tzdata updates in deployment (Docker, Ansible, etc.).

Timezone Management for a Social Platform

For platforms with global users:

Store all datetimes in UTC.
Store each user's preferred timezone.
Convert times on input/output according to the user’s timezone.

1. Enable Timezone Support in Django Settings

Set the default timezone to UTC:

# settings.py
USE_TZ = True
TIME_ZONE = "UTC"  # Store everything in UTC

2. Add a `timezone` Field to the Custom `User` Model

Use a function for dynamic timezone choices, so you don’t need new migrations when the list changes.

def get_timezone_choices():
    import zoneinfo
    return [(tz, tz) for tz in sorted(zoneinfo.available_timezones())]

class User(AbstractUser):
    # ...
    timezone = models.CharField(
        _("Timezone"), max_length=50, choices=get_timezone_choices, default="UTC"
    )

3. Detect Timezone on the Frontend

Add hidden fields in your Login and Signup forms to capture the user’s timezone from their browser:

document.addEventListener('DOMContentLoaded', function () {
    const userTimezone = Intl.DateTimeFormat().resolvedOptions().timeZone;
    const timezoneInput = document.getElementById('id_timezone');
    if (timezoneInput) {
        timezoneInput.value = userTimezone;
    }
});

You can also let users change their timezone manually in account settings.

4. Use a Custom DateTime Field in Forms

This field will convert datetimes between UTC and the user’s local timezone:

import datetime
from zoneinfo import ZoneInfo
from django import forms
from django.utils import timezone
from django.utils.dateparse import parse_datetime

class TimezoneAwareDateTimeField(forms.DateTimeField):
    widget = forms.DateTimeInput(attrs={"type": "datetime-local"})

    def __init__(self, user_timezone=None, *args, **kwargs):
        self.user_timezone = user_timezone
        super().__init__(*args, **kwargs)

    def prepare_value(self, value):
        if value and self.user_timezone:
            try:
                user_tz = ZoneInfo(self.user_timezone)
                if timezone.is_aware(value):
                    value = value.astimezone(user_tz)
            except Exception:
                pass
        return value

    def to_python(self, value):
        if value in self.empty_values:
            return None
        if isinstance(value, datetime.datetime):
            result = value
        elif isinstance(value, datetime.date):
            result = datetime.datetime(value.year, value.month, value.day)
        else:
            try:
                result = parse_datetime(value.strip())
            except ValueError:
                raise forms.ValidationError(
                    self.error_messages["invalid"], code="invalid"
                )
        if not result:
            result = super(forms.DateTimeField).to_python(value)
        if result and self.user_timezone:
            try:
                user_tz = ZoneInfo(self.user_timezone)
                if timezone.is_naive(result):
                    result = result.replace(tzinfo=user_tz)
                result = result.astimezone(ZoneInfo("UTC"))
            except Exception:
                pass
        return result

The type="datetime-local" widget uses the browser’s native date/time picker.

Use the custom field like this:

from django import forms
from django.utils.translation import gettext_lazy as _
from myproject.apps.core.form_fields import TimezoneAwareDateTimeField
from .models import Post

class PostForm(forms.ModelForm):
    class Meta:
        model = Post
        fields = ["title", "content", "published_from"]

    def __init__(self, request, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.request = request
        self.fields["published_from"] = TimezoneAwareDateTimeField(
            label=_("Published from"),
            help_text=_("Enter date and time in your local timezone."),
            required=False,
            user_timezone=self.request.user.timezone,
        )

5. Output Dates and Times in User's Timezone

{% load tz %}
{% with user_timezone=request.user.timezone|default:"UTC" %}
    {{ post.published_from|timezone:user_timezone|date:"j M, Y H:i" }}
{% endwith %}

Other Options

You can also detect the visitor’s timezone in JavaScript and send it via Ajax to be saved in the Django session. Then you can use it even for anonymous users.

Final Words

Timezones aren’t so scary if you follow Django’s best practices:

Store all times in UTC.
Update tzdata regularly.
Use the user's timezone only at input/output stages.
Detect the user’s timezone via JavaScript—no need to ask them manually.

This keeps your website accurate, user-friendly, and ready for global audiences.

Cover photo by Andrey Grushnikov

Using Google Consent Mode v2 on a Django Website

Sun, 06 Jul 2025 12:00:00 +0000

A decade ago, adding website analytics was simple: you’d just paste a JavaScript snippet from Google Analytics, and that was it.

But things changed. As people became more concerned about their privacy, countries introduced strict privacy laws—like GDPR in the European Union, PIPEDA in Canada, and APPI in Japan. These laws gave users more control over their data and placed new responsibilities on us developers.

One of those responsibilities is showing Cookie Consent banners and respecting users’ choices before loading any tracking scripts.

Today, if you want to add Google Analytics to your website, it’s not just about copying a script. You need to understand Google Tag Manager (GTM)—a tool that lets you manage what scripts run on your site and when, using a web-based dashboard.

When you add Google Analytics as a Google tag through GTM, it doesn’t automatically start collecting data. It waits for a signal called Consent Mode, which tells Google whether the user has accepted or denied tracking. This signal must be sent from your website to GTM as an event.

That’s where your Cookie Consent widget comes in. For Django websites, I created Django GDPR Cookie Consent, which lets you manage user consent and send the proper signals to Google.

In this article, I’ll show you how to make all three—Django GDPR Cookie Consent, Google Tag Manager, and Google Consent Mode v2—work together smoothly.

1. Set up Google Analytics

Go to Google Analytics and set up a property for your website. You'll need to know your Google Analytics 4 Measurement ID, e.g., G-XXXXXXX for Google Tag Manager.

2. Set up Google Tag Manager

You’ll use the Google Tag Manager (GTM) web interface — to add and manage all your tracking scripts (called tags). You won’t have to edit your website code every time you want to change something — GTM handles it all from one place.

In your container settings, check Enable consent overview to make sure that all your tags have cookie consent setting configured.

Check the GTM container id bounded to your website, e.g., GTM-XXXXXXX. You'll need it later for the scripts.

Here’s how to configure GTM to load scripts only after the user gives consent, using the signals from Google Consent Mode.

GA4 Configuration Tag

This tag initializes Google Analytics 4 (GA4) tracking.

Tag Type: Google Tag
Measurement ID: Your GA4 ID (G-XXXXXXX)
Consent Settings: Require additional consent for tag to fire: analytics_storage
Trigger: Consent Initialization – All Pages (this runs very early, before other tags)

This setup allows GA4 to start in a consent-aware way. It reads the default denied state at page load and will automatically adjust when the user accepts cookies later.

GA4 Event Tags

These are additional GA4 tags to track specific actions (like form submissions or button clicks).

Tag Type: Google Analytics: GA4 Event
Consent Settings: Require additional consent for tag to fire: analytics_storage
Trigger: Choose based on the action (e.g., form submit, button click)

You don’t need to check for consent manually here — GA4 automatically tracks or holds data based on the user's consent provided through Google Consent Mode.

Google Ads Tags

If you are using Google Ads, set these for conversion tracking and remarketing (showing ads to users who visited your site):

Tag Type: Google Ads Conversion Tracking or Google Ads Remarketing
Consent Settings: Require additional consent for tag to fire: ad_storage, ad_personalization, ad_user_data
Trigger: Set this to fire after a successful action (like a purchase or sign-up)

These tags respect consent choices, like whether the user allowed ad_storage.

3. Set up Django GDPR Cookie Consent

Download and install the package

Get Django GDPR Cookie Consent from Gumroad.

Create a directory private_wheels/ in your project's repository and add the wheel file of the app there.

Link to this file in your requirements.txt:

Django==5.2
file:./private_wheels/django_gdpr_cookie_consent-4.1.2-py2.py3-none-any.whl

Install the pip requirements from the requirements.txt file into your project's virtual environment:

(venv)$ pip install -r requirements.txt

Add the app to INSTALLED_APPS

INSTALLED_APPS = [
    # …
    "gdpr_cookie_consent",
    # …
]

Add the context processor

Add gdpr_cookie_consent to context processors in your Django settings:

TEMPLATES = [
    {
        # …
        "OPTIONS": {
            "context_processors": [
                # …
                "gdpr_cookie_consent.context_processors.gdpr_cookie_consent",
            ],
        },
    },
]

Add URL path to urlpatterns

from django.urls import path, include

urlpatterns = [
    # …
    path(
        "cookies/",
        include("gdpr_cookie_consent.urls", namespace="cookie_consent"),
    ),
    # …
]

Prepare cookie consent configuration

Create COOKIE_CONSENT_SETTINGS configuration in your Django project settings with these cookie sections:

Essential (strictly necessary) for cookies related to Django sessionid, CSRF token, and cookie consent,
Analytics (optional) for website usage statistics with Google Analytics,
Marketing (optional) for tracking cross-website ad statistics with Google Ads.

from django.utils.translation import gettext_lazy as _

COOKIE_CONSENT_SETTINGS = {
    "base_template_name": "base.html",
    "description_template_name": "gdpr_cookie_consent/descriptions/what_are_cookies.html",
    "dialog_position": "center",
    "consent_cookie_max_age": 60 * 60 * 24 * 30 * 6,
    "sections": [
        {
            "slug": "essential",
            "title": _("Essential Cookies"),
            "required": True,
            "preselected": True,
            "summary": _(
                "These cookies are always on, as they’re essential for making this website work, and making it safe. Without these cookies, services you’ve asked for can’t be provided."),
            "description": _(
                "These cookies are always on, as they’re essential for making this website work, and making it safe. Without these cookies, services you’ve asked for can’t be provided."),
            "providers": [
                {
                    "title": _("This website"),
                    "cookies": [
                        {
                            "cookie_name": "sessionid",
                            "duration": _("2 Weeks"),
                            "description": _(
                                "Session ID used to authenticate you and give permissions to use the site."),
                            "domain": ".example.com",
                        },
                        {
                            "cookie_name": "csrftoken",
                            "duration": _("Session"),
                            "description": _(
                                "Security token used to ensure that no hackers are posting forms on your behalf."),
                            "description_template_name": "",
                            "domain": ".example.com",
                        },
                        {
                            "cookie_name": "cookie_consent",
                            "duration": _("6 Years"),
                            "description": _("Settings of Cookie Consent preferences."),
                            "description_template_name": "",
                            "domain": ".example.com",
                        },
                    ]
                },
            ],
        },
        {
            "slug": "analytics",
            "title": _("Analytics Cookies"),
            "required": False,
            "summary": _(
                "These cookies help us analyse how many people are using this website, where they come from and how they're using it. If you opt out of these cookies, we can’t get feedback to make this website better for you and all our users."),
            "description": _(
                "These cookies help us analyse how many people are using this website, where they come from and how they're using it. If you opt out of these cookies, we can’t get feedback to make this website better for you and all our users."),
            "providers": [
                {
                    "title": _("Google Analytics"),
                    "description": _("Google Analytics is used to track website usage statistics."),
                    "description_template_name": "",
                    "cookies": [
                        {
                            "cookie_name": "_ga",
                            "duration": _("2 Years"),
                            "description": _("Used to distinguish users."),
                            "description_template_name": "",
                            "domain": ".example.com",
                        },
                        {
                            "cookie_name": "_gid",
                            "duration": _("24 Hours"),
                            "description": _("Used to distinguish users."),
                            "description_template_name": "",
                            "domain": ".example.com",
                        },
                        {
                            "cookie_name": "_ga_*",
                            "duration": _("2 Years"),
                            "description": _("Used to persist session state."),
                            "description_template_name": "",
                            "domain": ".example.com",
                        },
                        {
                            "cookie_name": "_gat_gtag_UA_*",
                            "duration": _("1 Minute"),
                            "description": _("Stores unique user ID."),
                            "description_template_name": "",
                            "domain": ".example.com",
                        },
                    ]
                },
            ],
        },
        {
            "slug": "marketing",
            "title": _("Marketing Cookies"),
            "required": False,
            "summary": _(
                "These cookies are set by our advertising partners to track your activity and show you relevant ads on other sites as you browse the internet."),
            "description": _(
                "These cookies are set by our advertising partners to track your activity and show you relevant ads on other sites as you browse the internet."),
            "providers": [
                {
                    "title": _("Google Ads"),
                    "description": _("These cookies are related to Google Ads conversion tracking."),
                    "description_template_name": "",
                    "cookies": [
                        {
                            "cookie_name": "_gac_gb_*",
                            "duration": _("90 Days"),
                            "description": _(
                                "Contains campaign related information. If you have linked your Google Analytics and Google Ads accounts, Google Ads website conversion tags will read this cookie unless you opt-out."),
                            "description_template_name": "",
                            "domain": ".example.com",
                        },
                    ]
                },
            ],
        },
    ]
}

Include the widget to your `base.html` template

Load the CSS somewhere in the <head> section:

<link href="{% static 'gdpr-cookie-consent/css/gdpr-cookie-consent.css' %}" rel="stylesheet" />

Include the widget just before the closing </body> tag:

{% include "gdpr_cookie_consent/includes/cookie_consent.html" %}

Link to the cookie management view, for example, in the website's footer:

{% url "cookie_consent:cookies_management" as cookie_management_url %}
<a href="{{ cookie_management_url }}" rel="nofollow">
    {% trans "Manage Cookies" %}
</a>

Add GTM and Google Consent Mode snippets to your `base.html` template:

Add these two scripts in the <head> section:

<script nonce="{{ request.csp_nonce }}">
   window.dataLayer = window.dataLayer || [];
   function gtag(){dataLayer.push(arguments);}

   gtag('consent', 'default', {
      'analytics_storage': '{% if "analytics" in cookie_consent_controller.checked_sections %}granted{% else %}denied{% endif %}',
      'ad_storage': '{% if "marketing" in cookie_consent_controller.checked_sections %}granted{% else %}denied{% endif %}',
      'ad_user_data': '{% if "marketing" in cookie_consent_controller.checked_sections %}granted{% else %}denied{% endif %}',
      'ad_personalization': '{% if "marketing" in cookie_consent_controller.checked_sections %}granted{% else %}denied{% endif %}',
      'functionality_storage': 'denied',
      'personalization_storage': 'denied',
      'security_storage': 'granted'  // usually okay to grant by default
   });
</script>

<!-- Google Tag Manager (head) -->
<script nonce="{{ request.csp_nonce }}">(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer','GTM-XXXXXXX');</script>

Add this iframe in the beginning of the <body> tag:

<!-- Google Tag Manager (body) -->
<noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-XXXXXXX"
height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>

Replace GTM-XXXXXXX with your real GTM container ID.

Create JavaScript custom event handlers for granting and denying consent

Create a custom JavaScript file main.js and include it in the base.html template somewhere before the closing </body> tag:

<script src="{% static 'site/js/main.js' %}"></script>

In the JavaScript file, add these custom event handlers that will grant or deny cookie consent at Google–that's a new feature in Django GDPR Cookie Consent v4:

document.addEventListener('grantGDPRCookieConsent', (e) => {
    console.log(`${e.detail.section} cookies granted`);
    if (e.detail.section === 'analytics') {
        gtag('consent', 'update', {
            'analytics_storage': 'granted'
        });
        dataLayer.push({'event': 'analytics_consent_granted'});
    } else if (e.detail.section === 'marketing') {
        gtag('consent', 'update', {
            'ad_storage': 'granted',
            'ad_user_data': 'granted',
            'ad_personalization': 'granted'
        });
        dataLayer.push({'event': 'marketing_consent_granted'});
    }
});

document.addEventListener('denyGDPRCookieConsent', (e) => {
    console.log(`${e.detail.section} cookies denied`);
    if (e.detail.section === 'analytics') {
        gtag('consent', 'update', {
            'analytics_storage': 'denied'
        });
        dataLayer.push({'event': 'analytics_consent_denied'});
    } else if (e.detail.section === 'marketing') {
        gtag('consent', 'update', {
            'ad_storage': 'denied',
            'ad_user_data': 'denied',
            'ad_personalization': 'denied'
        });
        dataLayer.push({'event': 'marketing_consent_denied'});
    }
});

Check if your setup is correct

Check the correctness of your configuration with the following:

(venv)$ python manage.py check gdpr_cookie_consent

4. Test and debug with GTM preview mode

Deploy your project to production.
Install Tag Assistant Chrome extension.
Enable Preview in Google Tag Manager dashboard.
Visit your website via the preview mode.
Accept cookies.
You should see these events:
- analytics_consent_granted – GA4 tag should fire.
- marketing_consent_granted – Google Ads tag should fire.
Check which tags fired — they should match the user’s choices.
The Consent tab of each event should show the correct preferred consent choices.
Google Analytics realtime report should track you only when the consent was given.

How it works

When a user visits your website, the default cookie consent mode is set based on their previously saved preferences. If it's their first visit, consent will default to “denied.”

When the user sets or updates their cookie preferences—via the modal dialog or the preferences form—your consent widget will fire custom JavaScript events grantGDPRCookieConsent or denyGDPRCookieConsent available since Django GDPR Cookie Consent v4.

Your JavaScript handler will listen for these events, update the Google Consent Mode accordingly, and send the updated values to Google Tag Manager.

Based on those values, Google Tag Manager will decide whether to activate tracking tags such as Google Analytics and Google Ads. These tags can then track usage statistics and, if allowed, ad-related cross-site behavior.

Final words

Now you should be all set. Google Analytics should respect user's privacy based on the choices in Cookie Consent widget, provided by Django GDPR Cookie Consent. Your website will be compliant with Google Consent Mode and will fire analytics and marketing tags only after consent.

Rate Limiting for Django Websites

Thu, 03 Jul 2025 12:00:00 +0000

Sometimes, certain pages of a Django website might receive unwanted traffic from crawlers or malicious bots. These traffic spikes consume server resources and can make the website unusable for legitimate users. In this article, I will explore Nginx’s rate-limiting capabilities to prevent such performance issues.

What is rate limiting, and why use it?

A typical Django website is deployed using a Gunicorn (or Uvicorn for ASGI) application server, with an Nginx web server in front of it. When a request comes to Nginx, it goes through various checks, gets filtered by domain, protocol, and path, and is finally passed to Gunicorn. Gunicorn then runs Django, which parses the URL and returns a response from the appropriate view.

Nginx rate limiting allows you to limit how often certain pages (based on URL paths) or all Django endpoints can be accessed. It prevents quick reloading or scripted attacks that flood your site with requests. This is especially useful for pages with forms, faceted list views, REST APIs, or GraphQL endpoints.

A typical rate-limiting configuration in Nginx defines a zone with a specific memory size (in megabytes), a rate (requests per second or per minute), and one or more locations that apply that zone using options like burst and nodelay. When the rate limit is exceeded, Nginx returns a 429 Too Many Requests response.

You can imagine the burst as a funnel. The requests are like small balls coming in quickly. The first request passes through immediately, and the others get stacked in the funnel. Any requests that don’t fit into the funnel are rejected with a 429 response. The rest are either delayed or processed immediately, depending on the configuration. If more requests come in and there’s space in the funnel, they’re stacked and processed according to the defined rate.

Practical example

Here is an example configuration that limits list views to 1 request per second, allowing up to 2 additional requests in a short burst before rejecting further ones. It also limits POST requests to authentication paths to 10 per minute, allowing bursts of up to 5 requests at a time. And a generous fallback for other URL paths. All excess requests beyond these limits will be rejected. Each zone has 10 MB of storage for metadata such as IP addresses, timestamps, and the current state of the funnel.

map $request_method $limit_key {
    POST $binary_remote_addr;
    default "";
}

limit_req_zone $binary_remote_addr zone=list_pages:10m rate=1r/s;
limit_req_zone $limit_key zone=auth_conditional:10m rate=10r/m;
limit_req_zone $binary_remote_addr zone=global_fallback:10m rate=20r/s;


server {
    listen 443 ssl;
    server_name www.pybazaar.com;
    charset utf-8;

    # ...

    location ~* "^/(|resources|profiles|opportunities|buzz)/?$" {
        limit_req zone=list_pages burst=2 nodelay;
        limit_req_status 429;

        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_buffering off;
        proxy_redirect off;
        proxy_read_timeout 300s;
    }

    location ~* "^/(signup|login)/?$" {
        limit_req zone=auth_conditional burst=5 nodelay;
        limit_req_status 429;
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_buffering off;
        proxy_redirect off;
        proxy_read_timeout 300s;
    }

    location / {
        limit_req zone=global_fallback burst=40 nodelay;
        limit_req_status 429;
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_buffering off;
        proxy_redirect off;
        proxy_read_timeout 300s;
    }

    # ...
}

The $binary_remote_addr setting limits requests per client IP address. Alternatively, you can use $server_name to apply rate limiting globally. Other variables can also be used.

Checking if it worked

Once you have this set up, you can test the rate limiting using the following bash script:

#!/usr/bin/env bash

for i in {1..20}; do
  {
    status=$(curl -w "%{http_code}" -o /dev/null -s \
      https://www.pybazaar.com/opportunities/)
    echo "Request $i: $status"
  } &
done
wait

This script will output the HTTP status codes for 20 requests sent simultaneously, for example:

Request 7: 429
Request 4: 429
Request 8: 429
Request 3: 429
Request 19: 429
Request 15: 429
Request 9: 429
Request 13: 429
Request 6: 429
Request 20: 429
Request 18: 429
Request 10: 429
Request 17: 429
Request 11: 429
Request 16: 429
Request 14: 429
Request 12: 429
Request 5: 200
Request 1: 200
Request 2: 200

Keep in mind that the order of the requests and which ones succeed or fail won’t be in a simple sequence because Internet traffic takes different paths on its way to and from the server.

Next steps

The next step could be setting up the fail2ban service to ban IP addresses that receive the 429 response code too frequently. Should I write an article about that?

Conclusion

Rate limiting with Nginx is an effective way to protect your Django website from traffic spikes caused by crawlers or malicious bots. By setting limits on requests per second or minute—using zones and bursts—you can control how many requests each IP can make to different parts of your site. This helps prevent server overload and keeps the website responsive for real users.

Algorithms: Learning One's Learnings

Thu, 01 May 2025 12:00:00 +0000

When I studied algorithms more than twenty years ago, I saw their general usefulness, but since then, I have never had to use them in the real world because they had already been implemented by database management services or built-in functionalities of most programming languages that I used.

Interviewer: Can you sort this list by size?

Interviewee: Absolutely.
/types confidently.../
l.sort()

There came a time when I had to remind myself of sorting algorithms. At 1st things 1st, I was about to implement a feature that lets users prioritize their options by comparing them two at a time. I thought that the Bubble Sort algorithm would fit perfectly.

So, I built the UI using ReactJS and Django for the backend to let users compare options and sort their priorities.

However, I soon noticed a problem: for longer lists, the process became quite time-consuming for users.

One day, I remembered something about algorithm complexity (also known as Big O Notation) and realized that Bubble Sort isn't the most efficient choice for this task.

In computer science, Big O notation describes how the performance of an algorithm grows relative to the size of the input. It's usually written like O(value), where value is a function of n, and n is the number of items the algorithm processes.

For example:

Finding the first item in a list or a key in a dictionary of n items is O(1).
Summing all items by iterating through a list is O(n).
Using a triple nested loop over the list would result in O(n³).

Here is an overview of most common complexities:

Complexity	Name	Meaning and Example
`O(1)`	Constant	Always the same time, no matter input size, e.g. accessing an array item
`O(log n)`	Logarithmic	Doubles size, just +1 step, e.g. Binary Search
`O(n)`	Linear	Time grows with size, e.g. loop over list
`O(n log n)`	Linearithmic	"Almost linear", but a bit slower, e.g. Merge Sort
`O(n²)`	Quadratic	Time explodes with size, e.g. nested loops as in Bubble sort
`O(2ⁿ)`	Exponential	Be careful — grows crazy fast, e.g. Recursive Fibonacci
`O(n!)`	Factorial	Nightmare mode, e.g. solving Traveling Salesman by brute force

Understanding the Big O practically

Bubble Sort is an algorithm with a time complexity of O(n²). That means: if you have a list of items, the algorithm needs to compare each item against many others, using two nested loops.

To sort a list of 5 items, the user might need up to 25 comparisons.
To sort 10 items, up to 100 comparisons.
To sort 50 items, up to 2,500 comparisons.

Not ideal.

I asked ChatGPT to help evaluate sorting algorithms based on their complexity, and Quick Sort came out as a much better choice. Its complexity is O(n log n), where log n (base 2) roughly means "how many times you can divide n by 2 until you reach 1."

To sort 5 items, Quick Sort would need around 12 comparisons.
To sort 10 items, around 34 comparisons.
To sort 50 items, around 282 comparisons.

That's still some work, but much more manageable.

From a usability perspective, though, it’s even better to let users drag and drop items manually or to offer AI-suggested orderings. So I implemented those options too.

Algorithm complexities in action

Below is a quick cheat sheet with complexities from the fastest to the slowest:

Sorting algorithms

Algorithm	Best Case	Average Case	Worst Case	Notes
Timsort	`O(n)`	`O(n log n)`	`O(n log n)`	Used in Python, Java
Merge Sort	`O(n log n)`	`O(n log n)`	`O(n log n)`	Stable, reliable
Heap Sort	`O(n log n)`	`O(n log n)`	`O(n log n)`	In-place
Quick Sort	`O(n log n)`	`O(n log n)`	`O(n²)`	Fastest on average
Shell Sort	`O(n log n)`	`O(n¹·⁵)`	`O(n²)`	Improved insertion
Insertion Sort	`O(n)`	`O(n²)`	`O(n²)`	Good for small/mostly sorted data
Selection Sort	`O(n²)`	`O(n²)`	`O(n²)`	Simple but slow
Bubble Sort	`O(n)`	`O(n²)`	`O(n²)`	Only good for tiny lists

Search algorithms

Algorithm	Best Case	Average Case	Worst Case	Notes
Hash Table Search	`O(1)`	`O(1)`	`O(n)`	Worst case rare (collisions)
Binary Search	`O(1)`	`O(log n)`	`O(log n)`	Needs sorted data
Jump Search	`O(1)`	`O(√n)`	`O(√n)`	For sorted arrays
Ternary Search	`O(1)`	`O(log n)`	`O(log n)`	Split into three parts
Linear Search	`O(1)`	`O(n)`	`O(n)`	No sorting needed, but slow

Encryption algorithms

Algorithm	Complexity	Notes
AES (Advanced Encryption Standard)	`O(1)` per block	Very fast, symmetric key
ChaCha20	`O(1)` per block	Modern stream cipher, very fast
RSA Encryption	`O(n³)`	Public key, slow for big data
Elliptic Curve Cryptography (ECC)	`O(n²)`	Faster than RSA at similar security
Diffie-Hellman Key Exchange	`O(n³)`	Secure key exchange, slower

Just for the hunger of interest

Python, MySQL, and PostgreSQL use these algorithms behind the scenes:

System	Sorting Algorithm	Notes
Python `sort()` and `sorted()`	Timsort	Hybrid of Merge Sort + Insertion Sort
MySQL `ORDER BY`	Quicksort (small datasets), Merge Sort or Filesort (large datasets)	Depends on storage engine (InnoDB, etc.)
PostgreSQL `ORDER BY`	Quicksort (small data), Heapsort + Mergesort for bigger sets	Intelligent memory-aware switching

Final words

So if you ever have to solve a development issue algorithmically instead of using default libraries — or even when choosing an algorithm from a library — remember the Big O notation. Sometimes the difference can save milliseconds or seconds of time, and sometimes, when users' input is involved, it can save minutes or even hours.

Cover picture by Steve Johnson

How to Use GraphQL in Django with Elasticsearch

Sun, 26 Jan 2025 22:40:32 +0000

When you need to use GraphQL with Django, a common practice is to use Graphene-Django, which allows you to query Django models. However, directly querying Django models might be too slow, especially when you have many complex relations. To speed that up, you can add one more layer of abstraction and index your Django models to an Elasticsearch server. In this article, I will show you how to create a GraphQL interface for querying posts from the Elasticsearch server.

Elasticsearch Server

The easiest way to install the Elasticsearch server locally is using EVM - Elasticsearch Version Manager:

$ sudo curl -o /usr/local/bin/evm https://raw.githubusercontent.com/duydo/evm/master/evm
$ sudo chmod +x /usr/local/bin/evm

Then you can install Elasticsearch 7.17.27 (or other version) with EVM, as follows:

$ evm install 7.17.27

By default, the Elasticsearch server will raise a warning if you don't have HTTPS or basic authentication. You can change one setting to turn off those warnings for localhost. To do so, add a line to ~/.evm/elasticsearch-7.17.27/config/elasticsearch.yml:

xpack.security.enabled: false

Now, you can start the server with:

$ evm start

If you need more Elasticsearch versions on the same computer, you can easily install and switch between them.

Python Packages

Install the latest packages of Django Elasticsearch DSL for connecting to Elasticsearch server, and Graphene-Django for GraphQL implementation:

(venv)$ pip install Django
(venv)$ pip install django-elasticsearch-dsl
(venv)$ pip install graphene-django

Django Settings

Add the installed apps to INSTALLED_APPS in the myproject/settings.py. Also, add a few more settings that we will need in this exercise:

INSTALLED_APPS = [
    # Contributed Django apps
    # ...

    # Third-party apps
    "django_elasticsearch_dsl",
    "graphene_django",

    # Local apps
    "posts",
]

PROJECT_NAME = "myproject"
ENVIRONMENT = "dev"

ELASTICSEARCH_DSL = {
    "default": {
        "hosts": "http://localhost:9200",
    }
}

GRAPHENE = {
    "SCHEMA": "elasticsearch_graphql.schema.schema",
}

Models and Administration for the Posts App

Let's create a posts app with two models: Category and Post:

The posts/models.py file will look like this:

from django.db import models


class Category(models.Model):
    title = models.CharField("Title", max_length=255)
    slug = models.SlugField("Slug", max_length=255)

    class Meta:
        verbose_name = "Category"
        verbose_name_plural = "Categories"
        ordering = ["title"]

    def __str__(self):
        return self.title


class Post(models.Model):
    title = models.CharField("Title", max_length=255)
    slug = models.SlugField("Slug", max_length=255)
    content = models.TextField("Content")
    categories = models.ManyToManyField(Category)

    class Meta:
        verbose_name = "Post"
        verbose_name_plural = "Posts"
        ordering = ["title"]

    def __str__(self):
        return self.title

Also let's create administration posts/admin.py to be able to add some entries easily:

from django.contrib import admin
from .models import Category, Post


@admin.register(Category)
class CategoryAdmin(admin.ModelAdmin):
    list_display = ["title", "slug"]
    prepopulated_fields = {"slug": ["title"]}


@admin.register(Post)
class Post(admin.ModelAdmin):
    list_display = ["title", "slug"]
    list_filter = ["categories"]
    filter_horizontal = ["categories"]
    prepopulated_fields = {"slug": ["title"]}

Create the database schema and the superuser, then run the local development server with the following Django management commands:

(venv)$ python makemigrations
(venv)$ python migrate
(venv)$ python createsuperuser
(venv)$ python runserver

Add some posts and categories in the administration:

http://localhost:8000/admin/

Elasticsearch Documents for the Posts

Let's create PostDocument definition for the Elasticsearch index of the Post model and its categories at posts/documents.py:

import graphene

from django.conf import settings
from django_elasticsearch_dsl import Document
from django_elasticsearch_dsl.registries import registry
from django_elasticsearch_dsl import fields
from .models import Post


@registry.register_document
class PostDocument(Document):
    title = fields.TextField()
    slug = fields.KeywordField()
    content = fields.TextField()
    categories = fields.NestedField(
        properties={
            "title": fields.TextField(),
            "slug": fields.KeywordField(),
        }
    )

    class Index:
        name = f"{settings.PROJECT_NAME}_{settings.ENVIRONMENT}_posts"
        settings = {"number_of_shards": 1, "number_of_replicas": 0}

    class Django:
        model = Post

        ignore_signals = False
        auto_refresh = True
        queryset_pagination = 5000

    def prepare(self, instance):
        data = super().prepare(instance)

        # Prepare categories field
        data["categories"] = [
            {
                "title": category.title,
                "slug": category.slug,
            }
            for category in instance.categories.all()
        ]
        return data

To index the posts on the Elasticsearch server, run the following:

(venv)$ python manage.py search_index --rebuild

Let's see if the entries were indexed and can be retrieved.

At first, run the Django shell:

(venv)$ python manage.py shell

Then try to list out items from the Elasticsearch index:

>>> from posts.documents import PostDocument
>>> for doc in PostDocument.search():
...    print(doc.title)

It will list up to 10 posts (that's the default pagination) from the index:

Creating AI-based Summaries in a Django Website
Creating Open Graph Images in Django for Improved Social Media Sharing
HTTPS for Django Development Environment

GraphQL Types and Queries for the Posts App

Now, create the GraphQL schema posts/schema.py for the categories and posts, as follows:

import operator
from functools import reduce

import graphene
from .documents import PostDocument


class ElasticsearchObjectType(graphene.ObjectType):
    @classmethod
    def from_elasticsearch(cls, elasticsearch_document):
        instance = cls()
        instance.elasticsearch_document = elasticsearch_document
        return instance


class ElasticsearchCategoryType(ElasticsearchObjectType):
    title = graphene.String()
    slug = graphene.String()

    def resolve_title(self, info):
        return self.elasticsearch_document.title

    def resolve_slug(self, info):
        return self.elasticsearch_document.slug


class ElasticsearchPostType(ElasticsearchObjectType):
    title = graphene.String()
    slug = graphene.String()
    content = graphene.String()
    categories = graphene.List(of_type=ElasticsearchCategoryType)

    def resolve_title(self, info):
        return self.elasticsearch_document.title

    def resolve_slug(self, info):
        return self.elasticsearch_document.slug

    def resolve_content(self, info):
        return self.elasticsearch_document.content

    def resolve_categories(self, info):
        return [
            ElasticsearchCategoryType.from_elasticsearch(
                elasticsearch_document=category
            )
            for category in self.elasticsearch_document.categories or []
        ]


class Query(object):
    elasticsearch_posts = graphene.List(
        ElasticsearchPostType,
        limit=graphene.Int(),
        offset=graphene.Int(),
        search=graphene.String(),
        categories=graphene.List(of_type=graphene.String),
    )

    def resolve_elasticsearch_posts(
        self,
        info,
        limit=10,
        offset=0,
        search=None,
        categories=None,
    ):
        from elasticsearch_dsl.search import Q

        search_obj = PostDocument.search()

        queries = []
        if search:
            queries.append(
                Q(
                    "multi_match",
                    query=search,
                    fields=[
                        "title^5",
                        "slug^4",
                        "content^3",
                    ],
                    type="best_fields",
                    tie_breaker=0.3,
                    fuzziness="AUTO",
                    fuzzy_transpositions=True,
                    lenient=True,
                )
            )

        if categories:
            queries.append(
                Q(
                    "nested",
                    path="categories",
                    query=Q("terms", categories__slug=categories),
                )
            )

        if queries:
            # All filters AND-ed
            search_obj = search_obj.query(reduce(operator.iand, queries))

        search_obj = search_obj[offset : offset + limit if limit else None]

        return [
            ElasticsearchPostType.from_elasticsearch(elasticsearch_document=result)
            for result in search_obj.execute()
        ]

Here, we have an ElasticsearchObjectType that provides a custom method from_elasticsearch to initiate GraphQL objects from Elasticsearch instead of Django models.

The query for elasticsearch_posts has two filters: by a search string and by a list of OR-ed category slugs. The search query looks into title, slug, and content fields with different boosting weights and searches for the best match. The fuzziness set to "AUTO" allows mistakes in the search string by two letters.

At the project level, create Graphene Relay - a schema class that uses a Query class inheriting from all the Query classes from all Django apps. The file can be located at myproject/schema.py and have this content:

import graphene
import posts.schema

class Query(posts.schema.Query, graphene.ObjectType):
    pass

schema = graphene.Schema(query=Query)

Finally, enable the GraphQL endpoint in the myproject/urls.py:

from django.contrib import admin
from django.urls import path
from django.views.decorators.csrf import csrf_exempt
from graphene_django.views import GraphQLView

urlpatterns = [
    path("graphql/", csrf_exempt(GraphQLView.as_view(graphiql=True))),
    path("admin/", admin.site.urls),
]

Checking the GraphQL Queries

Navigate to http://localhost:8000/graphql/

Graphene-Django comes with a GraphiQL user interface, which, when accessed from a browser, allows the user to query GraphQL and see results in the same window:

A search query for "AI" with categories "Intermediate" or "Advanced" can look like:

query FilteredPosts {
  elasticsearchPosts(
    search: "AI"
    categories: ["intermediate", "advanced"], 
    limit: 10, 
    offset: 0,
  ) {
    title
    slug
    categories {
      title
      slug
    }
  }
}

It will produce results in JSON just like this:

{
  "data": {
    "elasticsearchPosts": [
      {
        "title": "Creating AI-based Summaries in a Django Website",
        "slug": "creating-ai-based-summaries-in-a-django-website",
        "categories": [
          {
            "title": "Basics",
            "slug": "basics"
          },
          {
            "title": "Django",
            "slug": "django"
          },
          {
            "title": "Gemini",
            "slug": "gemini"
          },
          {
            "title": "Intermediate",
            "slug": "intermediate"
          },
          {
            "title": "Large Language Models",
            "slug": "large-language-models"
          },
          {
            "title": "PyBazaar",
            "slug": "pybazaar"
          },
          {
            "title": "Simplemind",
            "slug": "simplemind"
          }
        ]
      },
      {
        "title": "HTTPS for Django Development Environment",
        "slug": "https-for-django-development-environment",
        "categories": [
          {
            "title": "DevOps",
            "slug": "devops"
          },
          {
            "title": "Development",
            "slug": "development"
          },
          {
            "title": "Django",
            "slug": "django"
          },
          {
            "title": "HTTPS",
            "slug": "https"
          },
          {
            "title": "Intermediate",
            "slug": "intermediate"
          },
          {
            "title": "SSL",
            "slug": "ssl"
          },
          {
            "title": "TLS",
            "slug": "tls"
          }
        ]
      }
    ]
  }
}

Conclusion

Using Elasticsearch with GraphQL is not trivial, but it provides excellent performance, especially for complex data structures. The best way to implement that in Django that I know of is combining Django Elasticsearch DSL and Graphene-Django.

Cover photo Angela Roma

Creating Open Graph Images in Django for Improved Social Media Sharing

Sat, 11 Jan 2025 01:19:41 +0000

Although social media algorithms usually discourage posting links so that users stay as long as possible on the network, people often still post links below an introductory post as a comment or reply. Normal links to websites on social media look pretty dull unless you add open-graph images representing that link.

In this article, I will show you how you can generate open-graph images for a Django website using web rendering from HTML and CSS. I rely on this technique to generate Open Graph previews for links from DjangoTricks, 1st things 1st, and PyBazaar.

What is an Open Graph?

Facebook created the Open Graph protocol to allow websites to provide rich representation of any web page. Although it has specifics for websites, articles, profiles, music, and video, the common use case is to have a preview image with a title for social feeds. Open Graph previews work with most well-known social networks, including Facebook, Threads, LinkedIn, Mastodon, and Blue Sky. Open Graph tags are HTML meta tags that you put in the HEAD section, e.g.:

<meta property="og:type" content="website" />
<meta property="og:url" content="{{ WEBSITE_URL }}{{ request.path }}" />
<meta property="og:title" content="{{ profile.user.get_full_name }}" />
{% if profile.open_graph_image %}
    <meta property="og:image" content="{{ profile.open_graph_image.url }}?t={{ profile.modified|date:'U' }}" />
{% endif %}
<meta property="og:description" content="{{ profile.summary }}" />
<meta property="og:site_name" content="PyBazaar" />
<meta property="og:locale" content="en_US" />

X (formerly known as Twitter) also has basic support for Open Graph, but it is better to add Twitter-Card-specific meta tags to make the preview more prominent. Here's an example:

<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:url" content="{{ WEBSITE_URL }}{{ request.path }}">
<meta name="twitter:title" content="{{ profile.user.get_full_name }}">
<meta name="twitter:description" content="{{ profile.summary }}">
{% if profile.open_graph_image %}
    <meta name="twitter:image" content="{{ profile.open_graph_image.url }}?t={{ profile.modified|date:'U' }}">
{% endif %}

Use both Open Graph and Twitter Card meta tags for cross-platform support.

The dimensions of the preview image

The Open Graph Preview image dimensions will vary across different social networks. Many sources online recommend using 1200 x 630 px, but I have also successfully used smaller dimensions.

If the image is too big, the social network might take longer to process it for the preview, which slows down the link publishing.

On the other hand, if it's too small, its text might be too blurry.

Using the browser testing tools for the screenshots

I have used Playwright and Selenium to generate preview images in a background task. Since Playwright has a built-in mechanism to download browser binaries and is much more advanced in testing capabilities, it is also my first choice for screenshots.

You can install Playwright with:

(venv)$ pip install playwright
(venv)$ playwright install

Ensuring to have the necessary settings

To show the proof of concept, you would need WEBSITE_URL and Open Graph Image dimensions in your settings:

WEBSITE_URL = "https://www.pybazaar.com"

OPEN_GRAPH_IMAGE_WIDTH = 800
OPEN_GRAPH_IMAGE_HEIGHT = 418

Passing the `WEBSITE_URL` to the templates

It's easiest to pass the WEBSITE_URL and other settings to all the templates by having a custom context processor:

def website_settings(request):
    from django.conf import settings

    return {
        "WEBSITE_URL": settings.WEBSITE_URL,
    }

Set it in the settings as follows:

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [os.path.join(BASE_DIR, "templates")],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
                "pybazaar.apps.core.context_processors.website_settings",
            ],
        },
    },
]

Preparing the models

Each model that should have an Open Graph preview image should get an image field open_graph_image:

import os
from django.db import models
from django.utils import timezone


def upload_images_to(instance, filename):
    now = timezone.now()
    filename_base, filename_ext = os.path.splitext(filename)
    return "profiles/{user_id}/{filename}{ext}".format(
        user_id=instance.user.pk,
        filename=now.strftime("%Y%m%d%H%M%S"),
        ext=filename_ext.lower(),
    )

class Profile(models.Model):
    user = models.OneToOneField(
        "accounts.User", verbose_name="User", on_delete=models.CASCADE
    )
    open_graph_image = models.ImageField(
        "Open-graph image", upload_to=upload_images_to, blank=True
    )

    def generate_open_graph_image(self):
        from .tasks import generate_profile_open_graph_image

        if self.open_graph_image:
            self.open_graph_image.delete()
        generate_profile_open_graph_image(profile_id=self.pk)

Creating a Django view for the Open Graph preview

To define the preview layout, we'll create a custom HTML view:

from django.views.decorators.cache import never_cache
from django.shortcuts import render

@never_cache
def profile_open_graph_preview(request, username):
    profile = get_object_or_404(Profile, user__username=username)
    context = {
        "profile": profile,
    }
    return render(request, "profiles/profile_open_graph_preview.html", context)

The Django view used for the Open Graph preview image can use CSS frameworks, such as TailwindCSS, custom fonts, and Javascript enhancements for layout. This allows you to have consistent typography and styling with the website.

Plugging the view into the URLs

The URL rule for the view is pretty straightforward:

from django.urls import path
from . import views

app_name = "profiles"

urlpatterns = [
    #...
    path(
        "<str:username>/_open-graph-preview/",
        views.profile_open_graph_preview,
        name="profile_open_graph_preview",
    ),
]

Creating a background task to make screenshots

I use Huey for background tasks. The background task that generates screenshots looks like this (Celery task would be analogous):

from huey.contrib.djhuey import db_task


@db_task()
def generate_profile_open_graph_image(profile_id):
    import os
    from playwright.sync_api import sync_playwright
    from PIL import Image
    from django.conf import settings
    from django.core.files.base import ContentFile
    from django.core.files.storage import default_storage
    from django.urls import reverse
    from io import BytesIO
    from .models import Profile

    os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"  # for Playwright
    profile = Profile.objects.get(pk=profile_id)

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(
            viewport={
                "width": settings.OPEN_GRAPH_IMAGE_WIDTH,  # Changed to target dimensions
                "height": settings.OPEN_GRAPH_IMAGE_HEIGHT,
            },
            ignore_https_errors=True,
        )
        page = context.new_page()
        page.goto(
            settings.WEBSITE_URL
            + reverse(
                "profiles:profile_open_graph_preview",
                kwargs={"username": profile.user.username},
            )
        )
        page.wait_for_load_state("networkidle")
        screenshot_bytes = page.screenshot()
        browser.close()

    # Open and resize if needed
    image = Image.open(BytesIO(screenshot_bytes))
    if image.size != (settings.OPEN_GRAPH_IMAGE_WIDTH, settings.OPEN_GRAPH_IMAGE_HEIGHT):
        image.thumbnail(
            (settings.OPEN_GRAPH_IMAGE_WIDTH, settings.OPEN_GRAPH_IMAGE_HEIGHT),
            resample=Image.LANCZOS
        )

    # Save the image
    final_image_io = BytesIO()
    image.save(final_image_io, format="PNG")
    final_image_io.seek(0)

    rel_file_path = Profile._meta.get_field("open_graph_image").upload_to(
        profile, "profile.png"
    )
    default_storage.save(rel_file_path, ContentFile(final_image_io.getvalue()))
    Profile.objects.filter(pk=profile.pk).update(open_graph_image=rel_file_path)

Triggering the generation of the Open Graph preview image

Call the image generation method after saving an entry and its relations in a change view, administration view, or management commands:

profile.generate_open_graph_image()

The method runs the background task that launches Chromium, opens the Open Graph preview page, takes a screenshot, resizes it if necessary, and then saves the image. All that takes up to 15 seconds.

Final words

If you want the pages you share on social feeds to have a branded look and feel, generating Open Graph images with Playwright could be the right approach. You can use Playwright to make a screenshot of a web page in Chromium. The page can be styled with the same quality as your website. JavaScripts can be applied to highlight code syntax or resize the texts to fit into the given container, too.

Cover photo by Zeeshaan Shabbir

HTTPS for Django Development Environment

Sat, 14 Dec 2024 15:06:05 +0000

Certain modern website features require HTTPS, including Progressive Web Apps (PWA), WebSockets, camera and microphone usage, and geolocation detection. However, the default Django runserver command can only run the web server under HTTP. Is it possible to have HTTPS in your local development environment? Surely, and in this article, I will guide you through how to set it up.

How TLS/SSL certificates work

First of all, let's have a common ground on how the HTTPS work.

HTTPS is what makes websites secure, protecting transferred visitors' data (like passwords or credit card info).

TLS (Transport Layer Security), which replaced SSL (Secure Sockets Layer), is the security tool that works behind the scenes to create the safe connection between a client (like a browser) and a server by encrypting the data exchanged.

They work using a public key (shared via the certificate) and a private key (kept secret by the server) to establish encrypted connections. These keys are often stored in BASE64-encoded files with extensions like .pem.

A Certificate Authority (CA) is a trusted organization or entity responsible for issuing and managing digital certificates used in TLS/SSL encryption. Trusted Certificate Authorities verify the identity of website owners and sign their certificates to ensure authenticity.

A trusted Certificate Authority signs the certificate to verify the server’s identity, and clients (like the browsers) validate it against the Certificate Authority’s root certificate to ensure the connection is authentic and secure. The CA root certificates are typically stored in the operating system or the browser and updated regularly with software updates.

Some well-known Certificate Authorities include:

Let’s Encrypt (free, automated certificates),
DigiCert,
GlobalSign,
Sectigo (formerly Comodo).

For local development, one can use a simple tool mkcert to create locally trusted TLS certificates to secure development environments without needing external Certificate Authorities. Soon, you'll see how to do that.

TLS/SSL certificates are used not only for HTTPS connections but also to secure other communication protocols, including WebSockets (wss://), FTPS, database connections, email protocols (SMTP, IMAP, POP3), and message brokers (e.g., Redis, RabbitMQ, Kafka).

1. The /etc/hosts file

I usually create local hostnames for my development environments to access them by domains, such as djangotricks.local instead of localhost or 127.0.0.1. This can be done by adding a mapping line to your /etc/hosts file:

127.0.0.1 djangotricks.local

2. Install mkcert

Let's install mkcert to your development environment. I used Homebrew with the following commands on macOS:

$ brew install mkcert
$ brew install nss  # for Firefox
$ mkcert -install

Installation instructions for other operating systems are available at the mkcert GitHub page.

3. Create certificates for your local domain

I made a directory certs in my Django project, changed the current directory to it, and then ran:

mkcert djangotricks.local

Afterwards, I created .gitignore file in it with the following content that ignores everything but itself:

*
!.gitignore

Finally, my project's file structure looked like this:

djangotricks/
├── certs/
│   ├── .gitignore
│   ├── djangotricks.local.pem  - public certificate
│   ├── djangotricks.local-key.pem  - private key
├── manage.py

4. Django settings

I added a few TLS-related settings to my development-environment settings file at djangotricks/settings/dev.py:

CSRF_TRUSTED_ORIGINS = [
    "https://djangotricks.local:8000",
    "http://djangotricks.local:8000",
]

CSRF_COOKIE_SECURE = True
SESSION_COOKIE_SECURE = True

5.a) Create a custom script for the WSGI server

The next step was to prepare scripts for running the web server under HTTPS. After a few iterations of chatting with Claude AI, I created a file run_https_wsgi_server.py with the following content:

import os
import ssl
from pathlib import Path
from django.core.servers.basehttp import get_internal_wsgi_application
from wsgiref.simple_server import make_server
from watchfiles import run_process

def run_server():
    # Set Django environment
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "djangotricks.settings.dev")
    application = get_internal_wsgi_application()

    # Set certificate paths
    cert_path = Path(__file__).parent / "certs"
    ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    ssl_context.load_cert_chain(
        certfile=cert_path / "djangotricks.local.pem",
        keyfile=cert_path / "djangotricks.local-key.pem",
    )

    # Run server with TLS
    with make_server("djangotricks.local", 8000, application) as httpd:
        httpd.socket = ssl_context.wrap_socket(httpd.socket, server_side=True)
        print("Serving on https://djangotricks.local:8000")
        try:
            httpd.serve_forever()
        except KeyboardInterrupt:
            print("Server stopped.")

if __name__ == "__main__":
    # Watch directories for changes
    watched_dirs = [
        str(Path(__file__).parent), # Current directory
    ]

    # Start watching and run the server with auto-reload
    run_process(
        *watched_dirs,            # Paths to watch
        target=run_server,        # Function to run
        debounce=1600,            # Debounce changes (milliseconds)
        recursive=True,           # Watch directories recursively
        debug=True                # Enable debug logging
    )

This Python script runs Python's default web server with the Django WSGI under HTTPS, watches the project files for changes, and restarts the server whenever something is updated.

Then I installed the dependency watchfiles, which is a Rust-powered watcher for file changes:

(venv)$ pip install watchfiles

That allowed me to run the web server script with:

(venv)$ python run_https_wsgi_server.py

5.b) Create a custom script for the ASGI server

Analogically, I created a script for running the web server under HTTPS with ASGI support. It uses uvicorn for the ASGI protocol and watchfiles for automatic reloading:

(venv)$ pip install uvicorn
(venv)$ pip install watchfiles

I named the script run_https_asgi_server.py and it had the following content:

import os
from pathlib import Path
import uvicorn

if __name__ == "__main__":
    # Set Django environment
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "djangotricks.settings.dev")

    # Set certificate paths
    project_root = Path(__file__).parent
    ssl_keyfile = project_root / "certs" / "djangotricks.local-key.pem"
    ssl_certfile = project_root / "certs" / "djangotricks.local.pem"

    # Run server with TLS
    uvicorn.run(
        "djangotricks.asgi:application",
        host="djangotricks.local",
        port=8000,
        ssl_keyfile=str(ssl_keyfile),
        ssl_certfile=str(ssl_certfile),
        reload=True,  # Enable autoreload
        reload_dirs=[
            str(project_root),
        ],
        reload_includes=["*.py", "*.html", "*.js", "*.css", "*.txt"],
        workers=1,
    )

Uvicorn's core features include integration of watchfiles and TLS/SSL support. Developers can define which directories and file types to watch for autoloading.

Lastly, I could run the script with:

(venv)$ python run_https_asgi_server.py

6. Test your server

To test my results, I navigated to https://djangotricks.local:8000, browsed all the main pages, tried to submit a form, tried an asynchronous view, tried one of the modern JavaScript features, and tried changing some HTML markup to see how the web server automatically restarts for the changes to take effect immediately.

Conclusion

HTTPS is an essential feature of modern web development, and it's absolutely necessary to be able to set it up in development environments.

Tools like mkcert make it possible to use TLS certificates locally.

Although Django's runserver doesn't yet support TLS/SSL certificates, you can create custom Python (or Bash) scripts to run the web servers with HTTPS and autoloading. Also, it is quite straightforward to remove the autoloading if you don't need it.

Now, it's time for me to update all my local development environments. For you, too?

Cover picture by Laura Gigch

Creating AI-based Summaries in a Django Website

Sat, 23 Nov 2024 12:00:00 +0000

Summarizing lengthy text can be tedious, especially on platforms like PyBazaar, where concise summaries improve user experience. In this post, I'll share how I used Simplemind and Gemini to automate this process in my Django-based project.

Background Info

Recently, I launched PyBazaar.com, a website for Python developers to show their skills, find job offers, and post and find development resources. Its purpose is to have a central place where Python developers can market their services, products, or projects.

PyBazaar shows lengthy descriptions of career opportunities and resources in the detail views and short summaries in the list views. Summaries help users quickly grasp the content of resources and career opportunities without opening each detailed view, enhancing the overall browsing experience on PyBazaar. To make the editing smoother, I introduced automatic summarization based on AI.

Choosing Simplemind for Communication with LLMs

Kenneth Reitz, the author of the famous package requests, recently published his newest creation—Simplemind—which improves the developer experience with the APIs of large language models (LLMs). I thought it would be a good opportunity to try integrating his package into PyBazaar.

While I chose Google Gemini for its free tier, Simplemind's support for providers like OpenAI or Claude means developers can scale up for more advanced features or more precise results if needed.

Setting Up API Keys

At first, I had to get an API Key at Google AI Studio.

Then I installed Simplemind:

(venv)$ pip install 'simplemind[full]'

However, there are two notes on that:

While waiting for one of the dependencies (grpcio) to compile on my Mac, I had time for an energy drink and enough time to scroll through half my social media feed.
Simplemind depends on 49 nested dependencies where some of them might have conflicts with your other modules, as it happened with google-api-python-client and api-client on one of my projects.

Simplemind expects the LLM API keys to be defined in the environment variables. In my Django projects, I store the secrets in JSON files, which Git ignores, and I read those values with a utility function I wrote, get_secret().

So, I added these lines in the Django settings:

import os
os.environ["GEMINI_API_KEY"] = get_secret("GEMINI_API_KEY")
DEFAULT_LLM_PROVIDER = "gemini"

Django Integration

I created a straightforward view that takes posted HTML content, asks LLM to summarize it, and returns the summary to the user:

import json
import simplemind
from django.contrib.auth.decorators import login_required
from django.conf import settings
from django.http import JsonResponse
from django.utils.html import strip_tags


@login_required
def summarize(request):
    summary = ""
    try:
        if (
            request.method == "POST"
            and (data := json.loads(request.body))
            and (content := data.get("content"))
            and (text := strip_tags(content).strip())
        ):
            summary = simplemind.generate_text(
                prompt=f"Condense the following information in 2 sentences:\n\n{text}",
                llm_provider=settings.DEFAULT_LLM_PROVIDER,
            ).strip()
    except json.JSONDecodeError:
        pass
    data = {"summary": summary}
    return JsonResponse(data)

As you can see, Simplemind is as elegant as the requests app. I could easily switch to OpenAI or Claude if I needed more advanced results or smarter queries.

I used strip_tags() to reduce the token count and strip() to remove leading and trailing whitespaces.

To improve the view's performance, I could also use ASGI or a background task, but that's something to consider when there are more users at PyBazaar.

The summarization button had its template, which I included in my Django Crispy Forms layout with layout.HTML("""{% include "summarizer/includes/summarize_button.html" %}"""):

{% load i18n static %}
<button type="button" class="summarize btn btn-secondary mb-3" data-url="{% url 'summarize' %}">
    {% translate "Summarize by AI" %}
</button>
<script src="{% static 'site/js/summarize.js' %}"></script>

Javascript Handling

And finally, the summarize.js looked like this:

document.addEventListener('DOMContentLoaded', function() {
    const summarizeButtons = document.querySelectorAll('.summarize');

    summarizeButtons.forEach(button => {
        button.addEventListener('click', async function(e) {
            const btn = e.currentTarget;
            const url = btn.dataset.url;
            const originalText = btn.textContent;

            try {
                // Disable button and show loading state
                btn.disabled = true;
                btn.textContent = 'Summarizing...';

                // Get HTML content from Quill
                const contentField = document.getElementById('quill-input-id_description');
                const quillData = JSON.parse(contentField.value);
                const contentHtml = quillData.html;

                const csrfToken = document.querySelector('[name="csrfmiddlewaretoken"]').value;

                const response = await fetch(url, {
                    method: 'POST',
                    headers: {
                        'Content-Type': 'application/json',
                        'X-CSRFToken': csrfToken,
                    },
                    body: JSON.stringify({
                        content: contentHtml
                    })
                });

                if (!response.ok) {
                    throw new Error(`HTTP error! status: ${response.status}`);
                }

                const data = await response.json();

                // Get the summary field and update its value
                const summaryField = document.getElementById('id_summary');
                summaryField.value = data.summary;
                // Trigger change event in case there are any listeners
                summaryField.dispatchEvent(new Event('change'));

            } catch (error) {
                console.error('Summarization failed:', error);
                alert('Failed to generate summary. Please try again.');
            } finally {
                // Reset button state
                btn.disabled = false;
                btn.textContent = originalText;
            }
        });
    });
});

When a user clicks on the "Summarize by AI" button, the Javascript temporarily disables the button, changes its text to "Summarizing...", reads the HTML value from the QuillJS field, and posts it as {"content": "..."} to the summarize view. After receiving the summary as {"summary": "..."}, the Javascript fills in the summary textarea and makes the button clickable again.

Conclusion

Simplemind makes working with LLMs easier using smart defaults, so developers don't have to adjust complicated settings like temperature or max_tokens.

Gemini LLM can be used for free, and that seems good enough for simple features like this with a moderate number of active users.

I implemented this summarization feature at PyBazaar in just half a day, and I could easily adapt this integration to generate meta descriptions, email drafts, or personalized recommendations.

If you're a Python developer looking to showcase your skills, share resources, or find opportunities, visit PyBazaar.com today!

Cover photo by Caio

Setting up DigitalOcean Spaces for Django Media

Mon, 29 Jul 2024 12:00:00 +0000

At DigitalOcean, when you need large amounts of static data (images, documents, or videos), you have two options: Volumes Block Storage and Spaces Object Storage. In case of Volumes, you mount an extra hard drive to your server and use the file system to manage your files. Whereas with Spaces, you store files in the cloud and use a special API to create, read and delete files there.

Spaces Object Storage is comparable with AWS S3 Cloud Object Storage. It even supports the same API for dealing with data there.

Here are some benefits of using Spaces:

The price of Spaces is at least twice as low as the one of Volumes.
Content Delivery Network (CDN) is available for media file caching.
The configuration of Spaces is easier and the user interface is more user-friendly than the one of AWS S3.
It is relatively easy to start using Spaces for media files with the django-storages package.

I will walk you through setting up Spaces for your media files.

Create Spaces Object Storage at DigitalOcean

When creating Spaces Object Storage at DigitalOcean, you will be asked for these values:

Data center - choose the one closest to your business for legal reasons. For example, I chose Frankfurt for PyBazaar.
Enable CDN for caching - enable it for server-side caching.
Spaces bucket name - lowercase name for your bucket (e.g. "pybazaar")
Select a project - Project name for grouping your DigitalOcean resources (e.g. "PyBazaar")

For the created Spaces instance, I keep the settings unchanged:

File Listing: Restricted
CDN: Enabled
CORS Configurations: Unset

This will create an instance, which resources, aka media files, can be accessed under https://pybazaar.fra1.digitaloceanspaces.com and https://pybazaar.fra1.cdn.digitaloceanspaces.com

Create API keys for spaces at DigitalOcean

Now go to API ➔ Spaces Keys and choose Generate New Key.

There you'll generate an Access Key and Secret Key. You'll need those in the Django settings and for example Transit app for easy mass file management.

Connect Django to Spaces

Install django-storages and boto3 to your Django project.

(venv)$ pip install boto3
(venv)$ pip install django-storages[s3]

Add the STORAGES setting:

STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "bucket_name": "pybazaar",
            "access_key": get_secret("SPACES_ACCESS_KEY"),
            "secret_key": get_secret("SPACES_SECRET_KEY"),
            "region_name": "fra1",
            "endpoint_url": "https://pybazaar.fra1.digitaloceanspaces.com",
            "default_acl": "public-read",
            "location": "media",
            # required for the correct storage.exists() functioning
            "file_overwrite": False,
            # don't append any authentication parameters to the files.
            "querystring_auth": False,  
        },
    },
    "staticfiles": {
        # For static files, use file-system storage 
        "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
        # Or Whitenoise storage
        # "BACKEND": "whitenoise.storage.CompressedStaticFilesStorage",
    },
}
MEDIA_URL = "https://pybazaar.fra1.cdn.digitaloceanspaces.com/pybazaar/media/"

The implementation of get_secrets() depends on your needs. You can import values from environment variables or from a JSON or INI file. Here's a version I am using in my projects. Just make sure not to include your secrets in the Git repository.

MEDIA_ROOT is not participating in the game of django-storages.

For static files, I recommend using File System or Whitenoise, althought theoretically you could also use django-storages for them too.

Try connecting your local environment to spaces at first, then check it remotely.

If you use a rich-text editor supporting images or links to documents, you should ensure that the media paths don't change when you dump your pages from production to development. Otherwise, the images and links will be broken. To ensure that, set MEDIA_URL to "/pybazaar/media/" locally when you use the file system storage locally again.

Use the Storage API for all your file management

Use the Django Storage operations instead of file-system operations for all your media file operations: listing directory contents, creating a file, reading a file, updating file content, deleting a file, or checking the existence of a file.

For example, instead of this code:

import os
from django.conf import settings

with open(os.path.join(settings.MEDIA_ROOT, "README.txt"), "w") as f: 
    f.write("Hello, World!")

use this one:

from django.core.files.storage import default_storage
from django.core.files.base import ContentFile

default_storage.save("README.txt", ContentFile("Hello, World!"))

Ensure that django-imagekit works well

If you are using django-imagekit or another image manipulation library, make sure that you use the latest version that supports the STORAGES setting.

For older Django versions (< 4.2), you might need to set AWS_* settings instead of the STORAGES dictionary. See the django-packages docs here.

Upload some files

You can upload files for Spaces one-by-one at DigitalOcean console. Or better, use Transit v5 on MacOS.

Add new connection and choose Amazon S3

Address: fra1.digitaloceanspaces.com
Access Key ID: [your access key]
Secret: [your secret key]
Remote Path: /pybazaar/pybazaar/media

Upload some files there. Check if they are accessible from the CDN endpoint URL.

Set up a subdomain of yours

To have a dedicated subdomain of yours pointing to the spaces, e.g. https://media.pybazaar.com, you can follow the instructions in this article.

If you manage your DNS settings on DigitalOcean, it's not a big deal: you just click a few buttons to set the CNAME record and enable Let's encrypt certificates.

But if you manage your DNS elsewhere, you will have to manually set and regularly update your SSL settings, and set a CNAME record to point your subdomain (e.g. media.pybazaar.com) to their CDN endpoint (e.g. pybazaar.fra1.cdn.digitaloceanspaces.com).

Final words

Using django-storages you can relatively simply replace your file system with cloud-based storage such as Spaces Object Storage at DigitalOcean.

DigitalOcean gives 200 USD in credit over 60 days for everyone who will use my affiliate link to signup. If you are planning a website with lots of media files, it's worthy to have a try.

Cover photo by gdtography

Generating Fake Django Model Instances with Factory Boy

Wed, 01 May 2024 12:00:00 +0000

As you might know, I am developing PyBazaar, a Python Developer Marketplace. For a project of that scope, I need to create hundreds or thousands of data entries to ensure that everything works as expected. Factory Boy is a tool that allows me to create model instances in batches, and this blog post is about it.

The benefits of using Factory Boy

By creating a bunch of fake entries, I can achieve the following:

Work on list and detail representation and styling.
Work on and try functionality like filters, sorting, and pagination.
Check and improve performance with loads of data entries.
Create dummy data for unit or functional tests.

Factory Boy seemed like a pretty complex package, so I want to simplify things and introduce you to all the necessary parts for creating a fake model instances.

Model preparation

At PyBazaar, I have users with profiles, job offers, and resources that can be faked in batch. The related categories are predefined and don't need to be faked.

To make it possible to distinguish between real and fake entries, I added a new boolean field is_fake to all those models that I can create in batch:

# For testing and debugging
is_fake = models.BooleanField(_("Fake"), default=False)

Here is what the list of profiles can look like when created with Factory Boy:

The setup

The installation is pretty straightforward:

(venv)$ pip install factory-boy==3.3.0

And then in each app where you need to create fake entries, create a file factories.py with factory classes, e.g.:

import random
import factory
from pybazaar.apps.accounts.models import User

class UserFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = User

    first_name = factory.Faker("first_name")
    last_name = factory.Faker("last_name")

    # ...

    is_fake = True

For factory classes, I also add custom class methods delete_fake() and recreate_batch() so that I can quickly create entries or delete them:

class ProfileFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Profile

    user = factory.SubFactory(UserFactory)

    # ...

    is_fake = True

    @classmethod
    def recreate_batch(cls, size, **kwargs):
        cls.delete_fake()
        cls.create_batch(size=size, **kwargs)

    @classmethod
    def delete_fake(cls):
        for profile in Profile.objects.filter(is_fake=True):
            profile.delete()
        for user in User.objects.filter(is_fake=True):
            user.delete()

Factory class attributes tell the system what values to assign to the models when creating instances. Let's explore multiple cases that we can use as values.

Assigning a static value

If it's a simple static value, you can just assign it. It will be the same for all fake entries:

is_fake = True
publishing_status = Profile.PublishingStatusChoices.PUBLISHED

Assigning a value from a list

If it's a value from a list, use the Iterator class:

title = factory.Iterator([
    "Developer",
    "Software Engineer",
    "Programmer",
])

experience_level = factory.Iterator(
    Profile.ExperienceLevelChoices.values
)

Assigning generated value of a certain type

Factory Boy uses the Faker package to allow the creating of fake names, paragraphs, or locations. You can use those as follows:

first_name = factory.Faker("first_name")
last_name = factory.Faker("last_name")
summary = factory.Faker("paragraph")
city = factory.Faker("city")
state = factory.Faker("state")
country = factory.Faker("country_code")

Assigning an instance

If it's a foreign key and you want a random value, use this:

resource_type = factory.LazyAttribute(
    lambda o: ResourceType.objects.order_by("?").first()
)

Assigning a value from a function

Similarly, you can assign a value from a function:

description = factory.LazyAttribute(
    lambda o: generate_quill_content()
)

Assigning a random value

Or a random value:

is_available_for_work = factory.LazyAttribute(
    lambda o: random.choice([True, False])
)

Assigning a value based on attributes or methods of the model instance

Once you define attributes like first_name or last_name, you can set other values depending on those:

username = factory.LazyAttribute(
    lambda o: f"{o.first_name}_{o.last_name}".lower()
)
email = factory.LazyAttribute(
    lambda o: f"{o.first_name}_{o.last_name}@example.com".lower()
)

Assigning a password

There is a special django.Password class for generating password values:

password = factory.django.Password("Pa$$w0rd")

Assigning dummy images

Here is how to create and assign a dummy single-color image:

avatar = factory.django.ImageField(
    width=200, height=200, color="rgb(2,132,199)"
)

Having two factories depending on each other

As we have profiles depending on users, we can define the codependence with SubFactory class.

class ProfileFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Profile

    user = factory.SubFactory(UserFactory)

Then, when creating profiles, the users will be created automatically for them, too.

Attaching many-to-many relations

In Django, many-to-many relationships must be added after creating a model instance. We can achieve that with the PostGeneration class:

def attach_categories(obj, create, extracted, **kwargs):
    obj.specializations.add(
        *list(Specialization.objects.order_by("?")[: random.randint(3, 7)])
    )

class ProfileFactory(factory.django.DjangoModelFactory):
    # ...
    do_afterwards = factory.PostGeneration(attach_categories)

How you call this attribute doesn't matter - it should just not clash with other field names or attributes.

A complete example

So the final factories.py file could look like this:

import random
import factory
import json
from pybazaar.apps.accounts.models import User
from pybazaar.apps.profiles.models import Profile
from pybazaar.apps.categories.models import Specialization

class UserFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = User

    first_name = factory.Faker("first_name")
    last_name = factory.Faker("last_name")
    username = factory.LazyAttribute(
        lambda o: f"{o.first_name}_{o.last_name}".lower()
    )
    email = factory.LazyAttribute(
        lambda o: f"{o.first_name}_{o.last_name}@example.com".lower()
    )
    password = factory.django.Password("Pa$$w0rd")
    is_fake = True

def generate_quill_content():
    return json.dumps(
        {
            "delta": '',
            "html": "<p>Hey there</p>",
        }
    )

def attach_categories(obj, create, extracted, **kwargs):
    obj.specializations.add(
        *list(Specialization.objects.order_by("?")[: random.randint(3, 7)])
    )

class ProfileFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Profile

    user = factory.SubFactory(UserFactory)
    title = factory.Iterator(
        [
            "Developer",
            "Software Engineer",
            "Programmer",
        ]
    )
    avatar = factory.django.ImageField(
        width=200, height=200, color="rgb(2,132,199)"
    )
    bio = factory.LazyAttribute(lambda o: generate_quill_content())
    city = factory.Faker("city")
    state = factory.Faker("state")
    country = factory.Faker("country_code")
    is_available_for_work = factory.LazyAttribute(
        lambda o: random.choice([True, False])
    )
    experience_level = factory.Iterator(
        Profile.ExperienceLevelChoices.values
    )
    publishing_status = Profile.PublishingStatusChoices.PUBLISHED
    is_fake = True

    do_afterwards = factory.PostGeneration(attach_categories)

    @classmethod
    def recreate_batch(cls, size, **kwargs):
        cls.delete_fake()
        cls.create_batch(size=size, **kwargs)

    @classmethod
    def delete_fake(cls):
        for profile in Profile.objects.filter(is_fake=True):
            profile.delete()
        for user in User.objects.filter(is_fake=True):
            user.delete()

Creating fake entries

Lastly, I can create the fake entries from the Django shell as follows:

>>> from pybazaar.apps.profiles.factories import ProfileFactory
>>> ProfileFactory.recreate_batch(100)

And later, when I don't need those anymore:

>>> from pybazaar.apps.profiles.factories import ProfileFactory
>>> ProfileFactory.delete_fake()

Whenever I add new fields to the models, I can easily tweak the factories and recreate the whole bunch of models in one step.

Final words

Factory Boy doesn't guarantee data validation and integrity. For example, city, state, and country will be three separate random values that don't match a real location. However, that is sufficient to test your website's basic look and feel or performance.

Cover image by Google DeepMind

Django Project on NGINX Unit

Sat, 24 Feb 2024 12:00:00 +0000

Recently, I learned about the NGINX Unit and decided to try it on my DjangoTricks website. Unit is a web server developed by people from NGINX, with pluggable support for Python (WSGI and ASGI), Ruby, Node.js, PHP, and a few other languages. I wanted to see whether it's really easy to set it up, have it locally on my Mac and the remote Ubuntu server, and try out the ASGI features of Django, allowing real-time communication. Also, I wanted to see whether Django is faster with Unit than with NGINX and Gunicorn. This article is about my findings.

My observations

Unit service uses HTTP requests to read and update its configuration. The configuration is a single JSON file that you can upload to the Unit service via a command line from the same computer or modify its values by keys in the JSON structure.

Normally, the docs suggest using the curl command to update the configuration. However, as I am using Ansible to deploy my Django websites, I wanted to create a script I could later copy to other projects. I used Google Gemini to convert bash commands from the documentation to Ansible directives and corrected its mistakes.

The trickiest part for me was to figure out how to use Let's Encrypt certificates in the simplest way possible. The docs are extensive and comprehensible, but sometimes, they dig into technical details that are unnecessary for a common Django developer.

Also, it's worth mentioning that the Unit plugin version must match your Python version in the virtual environment. It was unexpected for me when Brew installed Python 3.12 with unit-python3 and then required my project to use Python 3.12 instead of Python 3.10 (which I used for the DjangoTricks website). So I had to recreate my virtual environment and probably will have problems later with pip-compile-multi when I prepare packages for the production server, still running Python 3.10.

Below are the instructions I used to set up the NGINX Unit with my existing DjangoTricks website on Ubuntu 22.04. For simplicity, I am writing plain Terminal commands instead of analogous Ansible directives.

1. Install Unit service to your server

Follow the installation instructions from documentation to install unit, unit-dev, unit-python3.10, and whatever other plugins you want. Make sure the service is running.

2. Prepare Let's Encrypt certificates

Create a temporary JSON configuration file /var/webapps/djangotricks/unit-config/unit-config-pre.json, which will allow Let's Encrypt certbot to access the .well-known directory for domain confirmation:

{
  "listeners": {
    "*:80": {
      "pass": "routes/acme"
    }
  },
  "routes": {
    "acme": [
      {
        "match": {
          "uri": "/.well-known/acme-challenge/*"
        },
        "action": {
          "share": "/var/www/letsencrypt"
        }
      }
    ]
  }
}

Install it to Unit:

$ curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/unit-config-pre.json \
--unix-socket /var/run/control.unit.sock http://localhost/config

If you make any mistakes in the configuration, it will be rejected with an error message and not executed.

Create Let's Encrypt certificates:

$ certbot certonly -n -w /var/www/letsencrypt -m hello@djangotricks.com \
--agree-tos --no-verify-ssl -d djangotricks.com -d www.djangotricks.com

Create a bundle that is required by the NGINX Unit:

cat /etc/letsencrypt/live/djangotricks.com/fullchain.pem \
/etc/letsencrypt/live/djangotricks.com/privkey.pem > \
/var/webapps/djangotricks/unit-config/bundle1.pem

Install certificate to NGINX Unit as certbot1:

curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/bundle1.pem \
--unix-socket /var/run/control.unit.sock http://localhost/certificates/certbot1

3. Install Django project configuration

Create a JSON configuration file /var/webapps/djangotricks/unit-config/unit-config.json which will use your SSL certificate and will serve your Django project:

{
  "listeners": {
    "*:80": {
      "pass": "routes/acme"
    },
    "*:443": {
      "pass": "routes/main",
      "tls": {
        "certificate": "certbot1"
      }
    }
  },
  "routes": {
    "main": [
      {
        "match": {
          "host": [
            "djangotricks.com",
            "www.djangotricks.com"
          ],
          "scheme": "https"
        },
        "action": {
          "pass": "applications/django"
        }
      },
      {
        "action": {
          "return": 444
        }
      }
    ],
    "acme": [
      {
        "match": {
          "host": [
            "djangotricks.com",
            "www.djangotricks.com"
          ],
          "uri": "/.well-known/acme-challenge/*"
        },
        "action": {
          "share": "/var/www/letsencrypt"
        }
      },
      {
        "action": {
          "return": 444
        }
      }
    ]
  },
  "applications": {
    "django": {
      "type": "python",
      "path": "/var/webapps/djangotricks/project/djangotricks",
      "home": "/var/webapps/djangotricks/venv/",
      "module": "djangotricks.wsgi",
      "environment": {
        "DJANGO_SETTINGS_MODULE": "djangotricks.settings.production"
      },
      "user": "djangotricks",
      "group": "users"
    }
  }
}

In this configuration, HTTP requests can only be used for certification validation, and HTTPS requests point to the Django project if the domain used is correct. In other cases, the status "444 - No Response" is returned. (It's for preventing access for hackers who point their domains to your IP address).

In the NGINX Unit, switching between WSGI and ASGI is literally a matter of changing one letter from "w" to "a" in the line about the Django application module, from:

"module": "djangotricks.wsgi",

to:

"module": "djangotricks.asgi",

I could have easily served the static files in this configuration here, too, but my STATIC_URL contains a dynamic part to force retrieval of new files from the server instead of the browser cache. So, I used WhiteNoise to serve the static files.

For redirection from djangotricks.com to www.djangotricks.com, I also chose to use PREPEND_WWW = True setting instead of Unit directives.

And here, finally, installing it to Unit (it will overwrite the previous configuration):

$ curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/unit-config.json \
--unix-socket /var/run/control.unit.sock http://localhost/config

How it performed

DjangoTricks is a pretty small website; therefore, I couldn't do extensive benchmarks, but I checked two cases: how a filtered list view performs with NGINX and Gunicorn vs. NGINX Unit, and how you can replace NGINX, Gunicorn, and Huey background tasks with ASGI requests using NGINX Unit.

First of all, the https://www.djangotricks.com/tricks/?categories=development&technologies=django-4-2 returned the HTML result on average in 139 ms on NGINX with Gunicorn, whereas it was on average 140 ms with NGINX Unit using WSGI and 149 ms with NGINX Unit using ASGI. So, the NGINX Unit with WSGI is 0.72% slower than NGINX with Gunicorn, and the NGINX Unit with ASGI is 7.19% slower than NGINX with Gunicorn.

However, when I checked https://www.djangotricks.com/detect-django-version/ how it performs with background tasks and continuous Ajax requests until the result is retrieved vs. asynchronous checking using ASGI, I went on average from 6.62 s to 0.75 s. Of course, it depends on the timeout of the continuous Ajax request, but generally, a real-time ASGI setup can improve the user experience significantly.

Final words

Although NGINX Unit with Python is unnoticeably slower than NGINX with Gunicorn, it allows Django developers to use asynchronous requests and implement real-time user experience. Also, you could probably have a Django website and Matomo analytics or WordPress blog on the same server. The NGINX Unit configuration is relatively easy to understand, and you can script the process for reusability.

Cover Image by Volker Meyer.

Renewing Let's Encrypt Certificates with NGINX Unit

Mon, 05 Feb 2024 12:00:00 +0000

Recently, I moved the DjangoTricks website and started PyBazaar on servers with Nginx Unit. One thing that was left undone was SSL certificate renewals. Let's Encrypt has special certbot parameters for renewing certificates for websites on Apache or Nginx servers, but they don't work out of the box with the Nginx Unit. In this blog post, I will tell you how to do that.

The certificate bundle

Nginx Unit doesn't use the fullchain.pem and privkey.pem generated by certbot directly from the location where they were generated. Instead, one has to create a bundle (like bundle1.pem) by concatenating them and then uploading it to the Nginx Unit configuration endpoint.

The bash script

For that, I created a bash script:

#!/usr/bin/env bash
SECONDS=0
CRON_LOG_FILE=/var/webapps/pybazaar/logs/renew_certificate.log

echo "=== Renewing Letsencrypt Certificate ===" > ${CRON_LOG_FILE}
date >> ${CRON_LOG_FILE}

echo "Renewing certificate..." >> ${CRON_LOG_FILE}
certbot --renew-by-default certonly -n --webroot -w /var/www/letsencrypt/ -m hello@pybazaar.com --agree-tos --no-verify-ssl -d pybazaar.com -d www.pybazaar.com

echo "Creating bundle..." >> ${CRON_LOG_FILE}
cat /etc/letsencrypt/live/pybazaar.com/fullchain.pem /etc/letsencrypt/live/pybazaar.com/privkey.pem > /var/webapps/pybazaar/unit-config/bundle1.pem

echo "Temporarily switching the Unit configuration to a dummy one..." >> ${CRON_LOG_FILE}
curl -X PUT --data-binary @/var/webapps/pybazaar/unit-config/unit-config-pre.json --unix-socket /var/run/control.unit.sock http://localhost/config

echo "Deleting old certificate from Nginx Unit..." >> ${CRON_LOG_FILE}
curl -X DELETE --unix-socket /var/run/control.unit.sock http://localhost/certificates/certbot1

echo "Installing new certificate to Nginx Unit..." >> ${CRON_LOG_FILE}
curl -X PUT --data-binary @/var/webapps/pybazaar/unit-config/bundle1.pem --unix-socket /var/run/control.unit.sock http://localhost/certificates/certbot1

echo "Switching the Unit configuration to the correct one..." >> ${CRON_LOG_FILE}
curl -X PUT --data-binary @/var/webapps/pybazaar/unit-config/unit-config.json --unix-socket /var/run/control.unit.sock http://localhost/config

echo "Finished." >> ${CRON_LOG_FILE}
duration=$SECONDS
echo "$(($duration / 60)) minutes and $(($duration % 60)) seconds elapsed." >> ${CRON_LOG_FILE}

Once you have adapted the script, you can run it manually as a root user to test it:

$ chmod +x renew_certificate.sh
$ ./renew_certificate.sh

Note that the certbot command will try to validate your website's URL by attempting to reach a temporary file that it will create on http://example.com/.well-known/acme-challenge/, so make sure that this location is accessible and serving the static files.

For more details about the Nginx Unit, check my previous blog post.

The cron job

If everything works as expected, you can add it to the root user's cron jobs to be executed weekly.

Export the current root cron jobs to a crontab.txt:

$ crontab -l > crontab.txt

Then edit it and add the weekly script to update the SSL certificate:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
SHELL=/bin/bash
MAILTO=""
@weekly /var/webapps/pybazaar/unit-config/renew_certificate.sh

Then run the following as the root user to apply it:

$ crontab crontab.txt

The good thing about not editing the cron job with crontab -e is that you can choose the editor and even put the crontab.txt under Git version control.

Happy web development with WSGI or ASGI!

Cover picture by Gotta Be Worth It

How to Handle Django Forms within Modal Dialogs

Fri, 21 Oct 2022 12:00:00 +0000

I like django-crispy-forms. You can use it for stylish uniform HTML forms with Bootstrap, TailwindCSS, or even your custom template pack. But when it comes to custom widgets and dynamic form handling, it was always a challenge. Recently I discovered htmx. It's a JavaScript framework that handles Ajax communication based on custom HTML attributes. In this article, I will explore how you can use django-crispy-forms with htmx to provide a form with server-side validation in a modal dialog.

The setup

For this experiment, I will be using these PyPI packages:

Django - my beloved Python web framework.
django-crispy-forms - library for stylized forms.
crispy-bootstrap5 - Bootstrap 5 template pack for django-crispy-forms.
django-htmx - some handy htmx helpers for Django projects.

Also, I will use the CDN versions of Bootstrap5 and htmx.

The form

I decided to add some crispy style to the login form by extending Django's authentication form and attaching a crispy helper to it.

from django.contrib.auth.forms import AuthenticationForm
from crispy_forms.helper import FormHelper
from crispy_forms.layout import Layout
from crispy_bootstrap5 import bootstrap5


class LoginForm(AuthenticationForm):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.helper = FormHelper()
        self.helper.form_tag = False
        self.helper.include_media = False
        self.helper.layout = Layout(
            bootstrap5.FloatingField("username", autocomplete="username"),
            bootstrap5.FloatingField("password", autocomplete="current-password"),
        )

Here I set form_tag to False to skip the <form> tag from the rendered form elements because I want to customize it in the templates. And I set include_media to False because I want to manually handle the media inclusion, as you will see later, instead of automatically including them in the form.

The views

I will have two views:

The home view will have a button to open a modal dialog for login.
The login form view will handle the form and return the HTML markup for the dialog.

from django.shortcuts import render, redirect
from django.contrib.auth import login as auth_login, logout as auth_logout
from django_htmx.http import HttpResponseClientRefresh
from .forms import LoginForm


def home(request):
    if request.user.is_authenticated:
        return render(request, "dashboard.html")
    return render(request, "home.html", context)


def login(request):
    if request.method == "POST":
        form = LoginForm(request=request, data=request.POST, prefix="login")
        template_name = "login_form.html"
        if form.is_valid():
            user = form.get_user()
            auth_login(request, user)
            return HttpResponseClientRefresh()
    else:
        form = LoginForm(request=request, prefix="login")
        template_name = "login_dialog.html"
    context = {"form": form}
    return render(request, template_name, context)

The home view just returns different templates based on whether the user is logged in or not.

The login view handles a login form and renders different templates based on whether the view was accessed by GET or POST method. If the login succeeds, a special HttpResponseClientRefresh response is returned, which tells htmx to refresh the page from where the login form was loaded and submitted. It's an empty response with the HX-Refresh: true header.

Then I plug those two views into my urls.py rules.

The templates and javascript

At the end of the base.html template, I include htmx from CDN and my custom dialog.js:

<script src="https://unpkg.com/htmx.org@1.8.2" integrity="sha384-+8ISc/waZcRdXCLxVgbsLzay31nCdyZXQxnsUy++HJzJliTzxKWr0m1cIEMyUzQu" crossorigin="anonymous"></script>
<script src="{% static 'js/dialog.js' %}"></script>

In the home.html template, I add a button which will open the dialog:

<button
    data-hx-get="{% url 'login' %}"
    data-hx-target="main"
    data-hx-swap="beforeend"
    type="button"
    class="btn btn-primary"
>Log in</button>

Note that htmx allows either hx-* syntax or data-hx-* for its HTML attributes and I prefer the latter because data-* attributes make a valid HTML document.

In the snippet above, I tell htmx to load the login page on the button click and insert it before the end of the <main> HTML tag.

Let's have a look at my dialog.js file:

function init_widgets_for_htmx_element(target) {
    // init other widgets

    // init modal dialogs
    if (target.tagName === 'DIALOG') {
        target.showModal();
        htmx.on('.close-dialog', 'click', function(event) {
            var dialog = htmx.find('dialog[open]');
            dialog.close();
            htmx.remove(dialog);
        });
    }
}

htmx.onLoad(init_widgets_for_htmx_element);

Here, when a page loads or a htmx inserts a snippet, the init_widgets_for_htmx_element function will be called with the <body> or the inserted element as the target. One can use this function to initialize widgets, such as rich text fields, autocompletes, tabs, custom frontend validators, etc. Also, I use this function to open the loaded modal dialogs and add event handlers to close them.

Now the login_dialog.html template looks like this (I just stripped the styling markup):

<dialog id="htmx-dialog">
    <h1>Login</h1>
    <button type="button" class="close-dialog btn-close" aria-label="Close"></button>
    {% include "login_form.html" %}
    <button type="submit" form="htmx-dialog-form">Log in</button>
</dialog>

And the login_form.html looks like this:

<form
    id="htmx-dialog-form"
    novalidate
    data-hx-post="{% url 'login' %}"
    data-hx-swap="outerHTML"
>
    {% load crispy_forms_tags %}
    {% crispy form %}
</form>

The htmx attributes tell htmx to submit the form data to the login view by Ajax and replace the <form> HTML tag with the response received. That is used for form validation. If the response has the HX-Refresh: true header, as mentioned before, then the home page is refreshed.

Here is what the result looks like:

The form media

If you noticed before, we excluded the media from the form. Otherwise, the home page would load the media of its own forms (for example, a search form) and the media of the dialog forms. And that would cause double executions of shared scripts, for example, the ones for autocompletes and rich text fields.

You can nicely combine media from different forms by concatenating the media instances. This way, each CSS and Javascript file is included just once.

As the login form isn't part of the home page but instead included on demand, I need to have its media in the home view or any other view where the login button is shown.

Here comes this custom context processor for help:

def login_dialog(request):
    from .forms import LoginForm

    if not request.user.is_authenticated:
        form = LoginForm(request=request)
        if hasattr(request, "combined_media"):
            request.combined_media += form.media
        else:
            request.combined_media = form.media
    return {}

It checks for any request.combined_media and attaches the form media from the login form.

Lastly, I attach this context processor to the template settings and render the value of combined media before </body>:

{{ request.combined_media }}

The final words

Get the code to play with from Github. As you can see, htmx makes Ajax communications pretty straightforward, even when it's about dynamically loading and reloading parts of the content or initializing custom widgets.

Cover photo by Pixabay

How to Rename a Django App

Tue, 04 Oct 2022 12:00:00 +0000

When I initially created my MVP (minimal viable product) for 1st things 1st, I considered the whole Django project to be about prioritization. After a few years, I realized that the Django project is about SaaS (software as a service), and prioritization is just a part of all functionalities necessary for a SaaS to function. I ended up needing to rename apps to have clean and better-organized code. Here is how I did that.

0. Get your code and database up to date

Ensure you have the latest git pull and execute all database migrations.

1. Install django-rename-app

Put django-rename-app into pip requirements and install them or just run:

(venv)$ pip install django-rename-app

Put the app into INSTALLED_APPS in your settings:

INSTALLED_APPS = [
    # …
    "django_rename_app",
]

2. Rename the app directories

Rename the oldapp as newapp in your apps and templates.

3. Rename the app name occurrences in the code

Rename the app in all your imports, relations, migrations, and template paths.

You can do a global search for oldapp and then check case by case where you need to rename that term to newapp, and where not.

4. Run the management command rename_app

Run the management command rename_app:

(env)$ python manage.py rename_app oldapp newapp

This command renames the app prefix the app tables and the records in django_content_type and django_migrations tables.

If you plan to update staging or production servers, add the rename_app command before running migrations in your deployment scripts (Ansible, Docker, etc.)

5. Update indexes and constraints

Lastly, create an empty database migration for the app with custom code to update indexes and foreign-key constraints.

(env)$ python manage.py makemigrations newapp --empty --name rename_indexes

Fill the migration with the following code:

# newapp/migrations/0002_rename_indexes.py
from django.db import migrations


def named_tuple_fetch_all(cursor):
    "Return all rows from a cursor as a namedtuple"
    from collections import namedtuple

    desc = cursor.description
    Result = namedtuple("Result", [col[0] for col in desc])
    return [Result(*row) for row in cursor.fetchall()]


def rename_indexes(apps, schema_editor):
    from django.db import connection

    with connection.cursor() as cursor:
        cursor.execute(
            """SELECT indexname FROM pg_indexes 
            WHERE tablename LIKE 'newapp%'"""
        )
        for result in named_tuple_fetch_all(cursor):
            old_index_name = result.indexname
            new_index_name = old_index_name.replace(
                "oldapp_", "newapp_", 1
            )
            cursor.execute(
                f"""ALTER INDEX IF EXISTS {old_index_name} 
                RENAME TO {new_index_name}"""
            )


def rename_foreignkeys(apps, schema_editor):
    from django.db import connection

    with connection.cursor() as cursor:
        cursor.execute(
            """SELECT table_name, constraint_name 
            FROM information_schema.key_column_usage
            WHERE constraint_catalog=CURRENT_CATALOG 
            AND table_name LIKE 'newapp%'
            AND position_in_unique_constraint notnull"""
        )
        for result in named_tuple_fetch_all(cursor):
            table_name = result.table_name
            old_foreignkey_name = result.constraint_name
            new_foreignkey_name = old_foreignkey_name.replace(
                "oldapp_", "newapp_", 1
            )
            cursor.execute(
                f"""ALTER TABLE {table_name} 
                RENAME CONSTRAINT {old_foreignkey_name} 
                TO {new_foreignkey_name}"""
            )


class Migration(migrations.Migration):

    dependencies = [
        ("newapp", "0001_initial"),
    ]

    operations = [
        migrations.RunPython(rename_indexes, migrations.RunPython.noop),
        migrations.RunPython(rename_foreignkeys, migrations.RunPython.noop),
    ]

Run the migrations:

(env)$ python manage.py migrate

If something doesn't work as wanted, migrate back, fix the code, and migrate again. You can unmigrate by migrating to one step before the last migration, for example:

(env)$ python manage.py migrate 0001

6. Cleanup

After applying the migration in all necessary environments, you can clean them up by removing django-rename-app from your pip requirements and deployment scripts.

Final words

It's rarely possible to build a system that meets all your needs from the beginning. Proper systems always require continuous improvement and refactoring. Using a combination of Django migrations and django-rename-app, you can work on your websites in an Agile, clean, and flexible way.

Happy coding!

Cover photo by freestocks.

How I Integrated Zapier into my Django Project

Mon, 18 Apr 2022 12:00:00 +0000

As you might know, I have been developing, providing, and supporting the prioritization tool 1st things 1st. One of the essential features to implement was exporting calculated priorities to other productivity tools. Usually, building an export from one app to another takes 1-2 weeks for me. But this time, I decided to go a better route and use Zapier to export priorities to almost all possible apps in a similar amount of time. Whaaat!?? In this article, I will tell you how.

What is Zapier and how it works?

The no-code tool Zapier takes input from a wide variety of web apps and outputs it to many other apps. Optionally you can filter the input based on conditions. Or format the input differently (for example, convert HTML to Markdown). In addition, you can stack the output actions one after the other. Usually, people use 2-3 steps for their automation, but there are power users who create 50-step workflows.

The input is managed by Zapier's triggers. The output is controlled by Zapier's actions. These can be configured at the website UI or using a command-line tool. I used the UI as this was my first integration. Trigger events accept a JSON feed of objects with unique IDs. Each new item there is treated as a new input item. With a free tier, the triggers are checked every 15 minutes. Multiple triggers are handled in parallel, and the sorting order of execution is not guaranteed. As it is crucial to have the sorting order correct for 1st things 1st priorities, people from Zapier support suggested providing each priority with a 1-minute interval to make sure the priorities get listed in the target app sequentially.

The most challenging part of Zapier integration was setting up OAuth 2.0 provider. Even though I used a third-party Django app django-oauth-toolkit for that. Zapier accepts other authentication options too, but this one is the least demanding for the end-users.

Authentication

OAuth 2.0 allows users of one application to use specific data of another application while keeping private information private. You might have used the OAuth 2.0 client directly or via a wrapper for connecting to Twitter apps. For Zapier, one has to set OAuth 2.0 provider.

The official tutorial for setting up OAuth 2.0 provider with django-oauth-toolkit is a good start. However, one problem with it is that by default, any registered user can create OAuth 2.0 applications at your Django website, where in reality, you need just one global application.

First of all, I wanted to allow OAuth 2.0 application creation only for superusers.

For that, I created a new Django app oauth2_provider_adjustments with modified views and URLs to use instead of the ones from django-oauth-toolkit.

The views related to OAuth 2.0 app creation extended this SuperUserOnlyMixin instead of LoginRequiredMixin:

from django.contrib.auth.mixins import AccessMixin

class SuperUserOnlyMixin(AccessMixin):
    def dispatch(self, request, *args, **kwargs):
        if not request.user.is_superuser:
            return self.handle_no_permission()
        return super().dispatch(request, *args, **kwargs)

Then I replaced the default oauth2_provider URLs:

urlpatterns = [
    # …
    path("o/", include("oauth2_provider.urls", namespace="oauth2_provider")),
]

with my custom ones:

urlpatterns = [
    # …
    path("o/", include("oauth2_provider_adjustments.urls", namespace="oauth2_provider")),
]

I set the new OAuth 2.0 application by going to /o/applications/register/ and filling in this info:

Name: Zapier
Client type: Confidential
Authorization grant type: Authorization code
Redirect uris: https://zapier.com/dashboard/auth/oauth/return/1stThings1stCLIAPI/ (copied from Zapier)
Algorithm: No OIDC support

If you have some expertise in the setup choices and see any flaws, let me know.

Zapier requires creating a test view that will return anything to check if there are no errors authenticating a user with OAuth 2.0. So I made a simple JSON view like this:

from django.http.response import JsonResponse


def user_info(request, *args, **kwargs):
    if not request.user.is_authenticated:
        return JsonResponse(
            {
                "error": "User not authenticated",
            },
            status=200,
        )
    return JsonResponse(
        {
            "first_name": request.user.first_name,
            "last_name": request.user.last_name,
        },
        status=200,
    )

Also, I had to have login and registration views for those cases when the user's session was not present.

Lastly, at Zapier, I had to set these values for OAuth 2.0:

Client ID: The Client ID from registered app
Client Secret: The Client Secret from registered app

Authorization URL: https://apps.1st-things-1st.com/o/authorize/
Scope: read write
Access Token Request: https://apps.1st-things-1st.com/o/token/
Refresh Token Request: https://apps.1st-things-1st.com/o/token/
I want to automatically refresh on unauthorized error: Checked
Test: https://apps.1st-things-1st.com/user-info/
Connection Label: {{first_name}} {{last_name}}

Trigger implementation

There are two types of triggers in Zapier:

(A) Ones for providing new things to other apps, for example, sending priorities from 1st things 1st to other productivity apps.
(B) Ones for listing things in drop boxes at the former triggers, for example, letting Zapier users choose the 1st things 1st project from which to import priorities.

The feeds for triggers should (ideally) be paginated. But without meta information for the item count, page number, following page URL, etc., you would usually have with django-rest-framework or other REST frameworks. Provide only an array of objects with unique IDs for each page. The only field name that matters is "id" – others can be anything. Here is an example:

[
    {
        "id": "39T7NsgQarYf",
        "project": "5xPrQbPZNvJv",
        "title": "01. Custom landing pages for several project types (83%)",
        "plain_title": "Custom landing pages for several project types",
        "description": "",
        "score": 83,
        "priority": 1,
        "category": "Choose"
    },
    {
        "id": "4wBSgq3spS49",
        "project": "5xPrQbPZNvJv",
        "title": "02. Zapier integration (79%)",
        "plain_title": "Zapier integration",
        "description": "",
        "score": 79,
        "priority": 2,
        "category": "Choose"
    },
    {
        "id": "6WvwwB7QAnVS",
        "project": "5xPrQbPZNvJv",
        "title": "03. Electron.js desktop app for several project types (42%)",
        "plain_title": "Electron.js desktop app for several project types",
        "description": "",
        "score": 41,
        "priority": 3,
        "category": "Consider"
    }
]

The feeds should list items in reverse order for the (A) type of triggers: the newest things go at the beginning. The pagination is only used to cut the number of items: the second and further pages of the paginated list are ignored by Zapier.

In my specific case of priorities, the order matters, and no items should be lost in the void. So I listed the priorities sequentially (not newest first) and set the number of items per page unrealistically high so that you basically get all the things on the first page of the feed.

The feeds for the triggers of (B) type are normally paginated from the first page until the page returns empty results. The order should be alphabetical, chronological, or by sorting order field, whatever makes sense. There you need just two fields, the ID and the title of the item (but more fields are allowed too), for example:

[
    {
        "id": "5xPrQbPZNvJv",
        "title": "1st things 1st",
        "owner": "Aidas Bendoraitis"
    },
    {
        "id": "VEXGzThxL6Sr",
        "title": "Make Impact",
        "owner": "Aidas Bendoraitis"
    },
    {
        "id": "WoqQbuhdUHGF",
        "title": "DjangoTricks website",
        "owner": "Aidas Bendoraitis"
    },
]

I used django-rest-framework to implement the API because of the batteries included, such as browsable API, permissions, serialization, pagination, etc.

For the specific Zapier requirements, I had to write a custom pagination class, SimplePagination, to use with my API lists. It did two things: omitted the meta section and showed an empty list instead of a 404 error for pages that didn't have any results:

from django.core.paginator import InvalidPage

from rest_framework.pagination import PageNumberPagination
from rest_framework.response import Response


class SimplePagination(PageNumberPagination):
    page_size = 20

    def get_paginated_response(self, data):
        return Response(data)  # <-- Simple pagination without meta

    def get_paginated_response_schema(self, schema):
        return schema  # <-- Simple pagination without meta

    def paginate_queryset(self, queryset, request, view=None):
        """
        Paginate a queryset if required, either returning a
        page object, or `None` if pagination is not configured for this view.
        """
        page_size = self.get_page_size(request)
        if not page_size:
            return None

        paginator = self.django_paginator_class(queryset, page_size)
        page_number = self.get_page_number(request, paginator)

        try:
            self.page = paginator.page(page_number)
        except InvalidPage as exc:
            msg = self.invalid_page_message.format(
                page_number=page_number, message=str(exc)
            )
            return []  # <-- If no items found, don't raise NotFound error

        if paginator.num_pages > 1 and self.template is not None:
            # The browsable API should display pagination controls.
            self.display_page_controls = True

        self.request = request
        return list(self.page)

To preserve the order of items, I had to make the priorities appear one by one at 1-minute intervals. I did that by having a Boolean field exported_to_zapier at the priorities. The API showed priorities only if that field was set to True, which wasn't the case by default. Then, background tasks were scheduled 1 minute after each other, triggered by a button click at 1st things 1st, which set the exported_to_zapier to True for each next priority. I was using huey, but the same can be achieved with Celery, cron jobs, or other background task manager:

# zapier_api/tasks.py
from django.conf import settings
from django.utils.translation import gettext
from huey.contrib.djhuey import db_task


@db_task()
def export_next_initiative_to_zapier(project_id):
    from evaluations.models import Initiative

    next_initiatives = Initiative.objects.filter(
        project__pk=project_id,
        exported_to_zapier=False,
    ).order_by("-total_weight", "order")
    count = next_initiatives.count()
    if count > 0:
        next_initiative = next_initiatives.first()
        next_initiative.exported_to_zapier = True
        next_initiative.save(update_fields=["exported_to_zapier"])

        if count > 1:
            result = export_next_initiative_to_zapier.schedule(
                kwargs={"project_id": project_id},
                delay=settings.ZAPIER_EXPORT_DELAY,
            )
            result(blocking=False)

One gotcha: Zapier starts pagination from 0, whereas django-rest-framework starts pagination from 1. To make them work together, I had to modify the API request (written in JavaScript) at Zapier trigger configuration:

const options = {
  url: 'https://apps.1st-things-1st.com/api/v1/projects/',
  method: 'GET',
  headers: {
    'Accept': 'application/json',
    'Authorization': `Bearer ${bundle.authData.access_token}`
  },
  params: {
    'page': bundle.meta.page + 1  // <-- The custom line for pagination
  }
}

return z.request(options)
  .then((response) => {
    response.throwForStatus();
    const results = response.json;

    // You can do any parsing you need for results here before returning them

    return results;
  });

Final Words

For the v1 of Zapier integration, I didn't need any Zapier actions, so they are yet something to explore, experiment with, and learn about. But the Zapier triggers seem already pretty helpful and a big win compared to individual exports without this tool.

If you want to try the result, do this:

Create an account and a project at 1st things 1st
Prioritize something
Head to Zapier integrations and connect your prioritization project to a project of your favorite to-do list or project management app
Then click on "Export via Zapier" at 1st things 1st.

Cover photo by Anna Nekrashevich

Generic Functionality without Generic Relations

Sat, 09 Apr 2022 12:00:00 +0000

When you have some generic functionality like anything commentable, likable, or upvotable, it’s common to use Generic Relations in Django. The problem with Generic Relations is that they create the relationships at the application level instead of the database level, and that requires a lot of database queries if you want to aggregate content that shares the generic functionality. There is another way that I will show you in this article.

I learned this technique at my first job in 2002 and then rediscovered it again with Django a few years ago. The trick is to have a generic Item model where every other autonomous model has a one-to-one relation to the Item. Moreover, the Item model has an item_type field, allowing you to recognize the backward one-to-one relationship.

Then whenever you need to have some generic categories, you link them to the Item. Whenever you create generic functionality like media gallery, comments, likes, or upvotes, you attach them to the Item. Whenever you need to work with permissions, publishing status, or workflows, you deal with the Item. Whenever you need to create a global search or trash bin, you work with the Item instances.

Let’s have a look at some code.

Items

First, I'll create the items app with two models: the previously mentioned Item and the abstract model ItemBase with the one-to-one relation for various models to inherit:

# items/models.py
import sys

from django.db import models
from django.apps import apps

if "makemigrations" in sys.argv:
    from django.utils.translation import gettext_noop as _
else:
    from django.utils.translation import gettext_lazy as _


class Item(models.Model):
    """
    A generic model for all autonomous models to link to.

    Currently these autonomous models are available:
    - content.Post
    - companies.Company
    - accounts.User
    """
    ITEM_TYPE_CHOICES = (
        ("content.Post", _("Post")),
        ("companies.Company", _("Company")),
        ("accounts.User", _("User")),
    )
    item_type = models.CharField(
        max_length=200, choices=ITEM_TYPE_CHOICES, editable=False, db_index=True
    )

    class Meta:
        verbose_name = _("Item")
        verbose_name_plural = _("Items")

    def __str__(self):
        content_object_title = (
            str(self.content_object) if self.content_object else "BROKEN REFERENCE"
        )
        return (
            f"{content_object_title} ({self.get_item_type_display()})"
        )

    @property
    def content_object(self):
        app_label, model_name = self.item_type.split(".")
        model = apps.get_model(app_label, model_name)
        return model.objects.filter(item=self).first()


class ItemBase(models.Model):
    """
    An abstract model for the autonomous models that will link to the Item.
    """
    item = models.OneToOneField(
        Item,
        verbose_name=_("Item"),
        editable=False,
        blank=True,
        null=True,
        on_delete=models.CASCADE,
        related_name="%(app_label)s_%(class)s",
    )

    class Meta:
        abstract = True

    def save(self, *args, **kwargs):
        if not self.item:
            model = type(self)
            item = Item.objects.create(
                item_type=f"{model._meta.app_label}.{model.__name__}"
            )
            self.item = item
        super().save()

    def delete(self, *args, **kwargs):
        if self.item:
            self.item.delete()
        super().delete(*args, **kwargs)

Then let's create some autonomous models that will have one-to-one relations with the Item. By "autonomous models," I mean those which are enough by themselves, such as posts, companies, or accounts. Models like types, categories, tags, or likes, wouldn't be autonomous.

Posts

Second, I create the content app with the Post model. This model extends ItemBase which will create the one-to-one relation on save, and will define the item_type as content.Post:

# content/models.py
import sys

from django.contrib.auth.base_user import BaseUserManager
from django.db import models
from django.contrib.auth.models import AbstractUser

if "makemigrations" in sys.argv:
    from django.utils.translation import gettext_noop as _
else:
    from django.utils.translation import gettext_lazy as _

from items.models import ItemBase


class Post(ItemBase):
    title = models.CharField(_("Title"), max_length=255)
    slug = models.SlugField(_("Slug"), max_length=255)
    content = models.TextField(_("Content"))

    class Meta:
        verbose_name = _("Post")
        verbose_name_plural = _("Posts")

Companies

Third, I create the companies app with the Company model. This model also extends ItemBase which will create the one-to-one relation on save, and will define the item_type as companies.Company:

# companies/models.py
import sys

from django.contrib.auth.base_user import BaseUserManager
from django.db import models
from django.contrib.auth.models import AbstractUser

if "makemigrations" in sys.argv:
    from django.utils.translation import gettext_noop as _
else:
    from django.utils.translation import gettext_lazy as _

from items.models import ItemBase


class Company(ItemBase):
    name = models.CharField(_("Name"), max_length=255)
    slug = models.SlugField(_("Slug"), max_length=255)
    description = models.TextField(_("Description"))

    class Meta:
        verbose_name = _("Company")
        verbose_name_plural = _("Companies")

Accounts

Fourth, I'll have a more extensive example with the accounts app containing the User model. This model extends AbstractUser from django.contrib.auth as well as ItemBase for the one-to-one relation. The item_type set at the Item model will be accounts.User:

# accounts/models.py
import sys

from django.db import models
from django.contrib.auth.base_user import BaseUserManager
from django.contrib.auth.models import AbstractUser

if "makemigrations" in sys.argv:
    from django.utils.translation import gettext_noop as _
else:
    from django.utils.translation import gettext_lazy as _

from items.models import ItemBase


class UserManager(BaseUserManager):
    def create_user(self, username="", email="", password="", **extra_fields):
        if not email:
            raise ValueError("Enter an email address")
        email = self.normalize_email(email)
        user = self.model(username=username, email=email, **extra_fields)
        user.set_password(password)
        user.save(using=self._db)
        return user

    def create_superuser(self, username="", email="", password=""):
        user = self.create_user(email=email, password=password, username=username)
        user.is_superuser = True
        user.is_staff = True
        user.save(using=self._db)
        return user


class User(AbstractUser, ItemBase):
    # change username to non-editable non-required field
    username = models.CharField(
        _("Username"), max_length=150, editable=False, blank=True
    )
    # change email to unique and required field
    email = models.EmailField(_("Email address"), unique=True)
    bio = models.TextField(_("Bio"))

    USERNAME_FIELD = "email"
    REQUIRED_FIELDS = []

    objects = UserManager()

Creating new items

I will use the Django shell to create several autonomous model instances and the related Items too:

>>> from content.models import Post
>>> from companies.models import Company
>>> from accounts.models import User
>>> from items.models import Item
>>> post = Post.objects.create(
...     title="Hello, World!",
...     slug="hello-world",
...     content="Lorem ipsum…",
... )
>>> company = Company.objects.create(
...     name="Aidas & Co",
...     slug="aidas-co",
...     description="Lorem ipsum…",
... )
>>> user = User.objects.create_user(
...     username="aidas",
...     email="aidas@example.com",
...     password="jdf234oha&6sfhasdfh",
... )
>>> Item.objects.count()
3

Aggregating content from all those relations

Lastly, here is an example of having posts, companies, and users in a single view. For that, we will use the Item queryset with annotations:

from django import forms
from django.db import models
from django.shortcuts import render
from django.utils.translation import gettext, gettext_lazy as _

from .models import Item


class SearchForm(forms.Form):
    q = forms.CharField(label=_("Search"), required=False)


def all_items(request):
    qs = Item.objects.annotate(
        title=models.Case(
            models.When(
                item_type="content.Post", 
                then="content_post__title",
            ),
            models.When(
                item_type="companies.Company", 
                then="companies_company__name",
            ),
            models.When(
                item_type="accounts.User",
                then="accounts_user__email",
            ),
            default=models.Value(gettext("<Untitled>")),
        ),
        description=models.Case(
            models.When(
                item_type="content.Post",
                then="content_post__content",
            ),
            models.When(
                item_type="companies.Company",
                then="companies_company__description",
            ),
            models.When(
                item_type="accounts.User", 
                then="accounts_user__bio",
            ),
            default=models.Value(""),
        ),
    )

    form = SearchForm(data=request.GET, prefix="search")
    if form.is_valid():
        query = form.cleaned_data["q"]
        if query:
            qs = qs.annotate(
                search=SearchVector(
                    "title",
                    "description",
                )
            ).filter(search=query)

    context = {
        "queryset": qs,
        "search_form": form,
    }
    return render(request, "items/all_items.html", context)

Final words

You can have generic functionality and still avoid multiple hits to the database by using the Item one-to-one approach instead of generic relations.

The name of the Item model can be different, and you can even have multiple such models for various purposes, for example, TaggedItem for tags only.

Do you use anything similar in your projects?

Do you see how this approach could be improved?

Let me know in the comments!

Cover picture by Pixabay

17 Django Project Ideas that can Make a Positive Impact around You

Tue, 09 Nov 2021 12:00:00 +0000

For more than a decade, I was focused only on the technical part of website building with Django. In the process, I have built a bunch of interesting cultural websites. But I always felt that those sleepless nights were not worthy of the impact.

They say, "Don’t work hard, work smart!" I agree with that phrase, and for me it's not about working less hours. For me, it's working as much as necessary, but on things that matter most.

So after years of collecting facts about life, I connected the dots and came up with make-impact.org – a social donation platform, which became one of the most important long-term projects. All my planning goes around this project.

And I believe I am not the only programmer who sometimes feels that they want to make a positive impact with their skills. So I brainstormed 17 Django project ideas. You can choose one and realize it as a hobby project, open-source platform, startup, or non-profit organization; alone, with a team of developers, or collaborating with some non-technical people.

Idea #1: Low Qualification Job Search

The job market is pretty competitive, and not all people can keep up with the train. You could build a job search website for jobs that don't require high education or lots of working experience. It could be helpful for people with language barriers, harsh living conditions, or those who are very young or very old. You could build it for your city, region, or country.

Idea #2: Discounted Meals and Products

Get inspired from Too Good To Go and build a progressive web app for your city about discounted restaurant meals and shop products whose expiration date is close to the end, but they are still good to eat.

Idea #3: Personal Health Advisor and Tracker

Build a website for setting your personal health improvement goals and tracking the progress. For example, maybe one wants to start eating more particular vegetables every week, jogging daily, lose or gain weight, or get rid of unhealthy addictions. Let people choose their health goals and check in with each progressive step. Allow using the website anonymously.

Idea #4: Online Primary and Elementary School Materials

Some people don't have access to schools in general or miss some classes because of illnesses. You could build a global and open wiki-based primary and elementary school education website for children and adults. It should be translatable and localizable. It would also be interesting to compare the same subject teachings in different countries side-by-side.

Idea #5: Psychological Support for Women

You could build a website with a video chat providing psychological support to discriminated or violently abused women. The help could be given by professionals or emphatic volunteers. The technical part can be implemented using django-channels, WebSockets, and WebRTC.

Idea #6: Rain-harvesting Companies around the World

Rain harvesting is one of the available ways to solve the problem of the lack of drinking water. There could be a platform comparing rain-harvesting companies all around the world. What are the installation prices? What are the countries they are working with? How many people have they saved? This website would allow people to find the most optimal company to build a rain harvesting system for them.

Idea #7: Closest Electric Car Charging Stations

Use the Open Charge Map API and create a progressive web app that shows the nearest electric car charging station and how to get there.

Idea #8: Escrow-based Remote Job Search

As remote jobs are getting more and more popular, there is still a matter of trust between the employees and employers. "Will the job taker complete their job in a good quality?" "Will the company pay the employee on time?" There are Escrow services to fix this issue. These are third parties that take and hold the money until the job is done. You could build a remote job search website promoting the usage of Escrow.com or another escrow service provider.

Idea #9: Open Work Locations

You could build a website listing coworking spaces and cafes with free wifi in your city. It should include the map, price ranges, details if registration is required, and other information necessary for remote workers.

Idea #10: Most Admired Companies

There could be a social website listing the most admired companies to work for in your country. Companies could be rated by working conditions, salary equality, growth opportunities, work relations, and other criteria. Anyone could suggest such a company, and they would be rated by their current and former employees anonymously.

Idea #11: Tiny Houses

The cost of accommodation is a critical problem in many locations of the world. You could develop a website that lists examples of tiny houses and their building schemas and instructions.

Idea #12: Catalog of Recycled Products

You could work on a product catalog with links to online shops, selling things produced from collected plastic. For example, these sunglasses are made of plastic collected from the ocean. Where available, you could use affiliate marketing links.

Idea #13: Information for Climate-change Migrants

You could work on a website for climate-change migrants with information about getting registered, housing, education, and jobs in a new city or country with better climate conditions.

Idea #14: Fishes, Fishing, and Overfishing

Scrape parts of FishBase and create a website about fishes, fishing, and overfishing in your region or the world. Engage people about the marine world and inform them about the damage done by overfishing.

Idea #15: Plant Trees

Create an E-commerce shop or Software as a Service and integrate RaaS (Reforestation as a Service). Let a tree be planted for every sale.

Idea #16: Positive Parenting

Create a progressive web app about positive parenting. For inspiration and information check this article.

Idea #17: Constructive Forum

Create a forum with topic voting and automatic hate speech detection and flagging. For example, maybe you could use a combination of Sentiment analysis from text-processing.com and usage of profanity words to find negativity in forum posts.

It's your turn

I hope this post inspired you. If you decided to start a startup with one of those ideas, don't forget to do your research at first. What are the competitors in your area? What would be your unique selling point? Etc.

Also, it would be interesting to hear your thoughts. Which of the projects would seem to you the most crucial? Which of them would you like to work on?

Cover photo by Joshua Fuller

How to Use Semantic Versioning for Shared Django Apps

Sat, 06 Nov 2021 12:00:00 +0000

When you are building websites with Django, there is no need to track their software versions. You can just have a stable branch in Git repository and update the production environment whenever it makes sense. However, when you create a Django app or package shared among various websites and maybe with multiple people, it is vital to keep track of the package versions.

The Benefits

Versioning allows you to identify a specific state of your package and has these benefits:

Developers can be aware of which package version works with their websites together flawlessly.
You can track which versions had which bugs or certain features when communicating with open-source communities or technical support.
In the documentation, you can clearly see which version of the software it is referring to.
When fixing bugs of a particular version, developers of the versioned package have a narrower scope of code to check at version control commits.
Just from the version number, it's clear if the upgrade will only fix the bugs or if it requires more attention to make your software compatible.
When talking to other developers about a particular package, you can clearly state what you are talking about (yes, developers talk about software versions from time to time 😅).

Semantic Versioning

The most popular versioning scheme is called semantic versioning. The version consists of three numbers separated by dots. Django framework is using it too. For example, the latest stable version at the time of writing is 3.2.9. In semantic versioning, the first number is the major version, the second is the minor version, and the third is a patch version: MAJOR.MINOR.PATCH.

MAJOR version introduces backward incompatible features.
MINOR version adds functionality that is backward compatible.
PATCH version is for backward-compatible bug fixes.

There can also be some versioning outliers for alpha, beta, release candidates rc1, rc2, etc., but we will not cover them in this post.

By convention, the package's version is saved in its myapp/__init__.py file as __version__ variable, for example:

__version__ = "0.2.4"

In addition, it is saved in setup.py, README.md, CHANGELOG.md, and probably some more files.

Changelog

When you develop a Django app or another Python package that you will share with other developers, it is also recommended to have a changelog. Usually, it's a markdown-based document with the list of features, bugs, and deprecations that were worked on between specific versions of your package.

A good changelog starts with the "Unreleased" section and is followed by sections for each version in reverse order. In each of those sections, there are lists of changes grouped into categories:

Added
Changed
Deprecated
Removed
Fixed
Security

This could be the starting template for CHANGELOG.md:

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

<!--
### Added
### Changed
### Deprecated
### Removed
### Fixed
### Security
-->

As the new version is released, it will replace the Unreleased section while creating a new Unreleased section above it. For example, this:

## [Unreleased]

### Added

- Logging the cookie consent choices in the database because of the legal requirements.

### Fixed

- Disabling the buttons while saving the Cookie Choices so that they are not triggered more than once with slow Internet connections.

would become this:

## [Unreleased]

## [v0.2.0] - 2021-10-27

### Added

- Logging the cookie consent choices in the database because of the legal requirements.

### Fixed

- Disabling the buttons while saving the Cookie Choices so that they are not triggered more than once with slow Internet connections.

To keep track of the versions manually would be pretty tedious work, with the likelihood of forgetting one or more files or mismatching the version numbers. Gladly, there is a utility tool to do that for you, which is called bump-my-version (successor of bump2version).

Using bump-my-version

Installation

Install the bump-my-version utility as an independent tool using uv:

$ uv tool install bump-my-version

or to your virtual environment the standard way with pip:

(env)$ pip install bump-my-version

Preparation

In your package's __init__.py file, set the version to 0.0.0:

__version__ = "0.0.0"

Set the version to 0.0.0 in all other files, where the version needs to be mentioned, for example, README.md and setup.py.

Then create a default pyproject.toml file for the configuration using this command:

$ bump-my-version sample-config --no-prompt --destination pyproject.toml

You will get a file with the content like this:

[tool.bumpversion]
current_version = "0.13.0"
parse = "(?P<major>\\d+)\\.(?P<minor>\\d+)\\.(?P<patch>\\d+)"
serialize = ["{major}.{minor}.{patch}"]
search = "{current_version}"
replace = "{new_version}"
regex = false
ignore_missing_version = false
ignore_missing_files = false
tag = false
sign_tags = false
tag_name = "v{new_version}"
tag_message = "Bump version: {current_version} → {new_version}"
allow_dirty = false
commit = false
message = "Bump version: {current_version} → {new_version}"
moveable_tags = []
commit_args = ""
setup_hooks = []
pre_commit_hooks = []
post_commit_hooks = []

We'll make two changes here:

Change the properties commit and tag to true so that the newly created versions are committed to Git and tagged there.
Add rules for version replacements in your setup.py, __init__.py, README.md, CHANGELOG.md.

The final content of the pyproject.toml will look like this:

[tool.bumpversion]
current_version = "0.13.0"
parse = "(?P<major>\\d+)\\.(?P<minor>\\d+)\\.(?P<patch>\\d+)"
serialize = ["{major}.{minor}.{patch}"]
search = "{current_version}"
replace = "{new_version}"
regex = false
ignore_missing_version = false
ignore_missing_files = false
tag = true
sign_tags = false
tag_name = "v{new_version}"
tag_message = "Bump version: {current_version} → {new_version}"
allow_dirty = false
commit = true
message = "Bump version: {current_version} → {new_version}"
moveable_tags = []
commit_args = ""
setup_hooks = []
pre_commit_hooks = []
post_commit_hooks = []

[[tool.bumpversion.files]]
filename = "setup.py"
search = 'version="{current_version}"'
replace = 'version="{new_version}"'

[[tool.bumpversion.files]]
filename = "myapp/__init__.py"
search = '__version__ = "{current_version}"'
replace = '__version__ = "{new_version}"'

[[tool.bumpversion.files]]
filename = "README.md"
search = 'django_myapp-{current_version}-py3-none-any.whl'
replace = 'django_myapp-{new_version}-py3-none-any.whl'

[[tool.bumpversion.files]]
filename = "CHANGELOG.md"
search = '## [Unreleased]'
replace = '''## [Unreleased]

## [v{new_version}] - {utcnow:%Y-%m-%d}'''

Replace "django_myapp" and "myapp" with your Django app there. If you have more files, mentioning the package version, make sure to have analogous version replacements in the pyproject.toml.

Commit and push your changes to the Git repository.

Usage

Every time you want to create a new version, type this in the shell:

$ bump-my-version bump patch

$ bump-my-version bump minor

$ bump-my-version bump major

followed by the command to build the package:

$ python3 setup.py sdist bdist_wheel

As mentioned before, patch is for the bug fixes, minor is for backward-compatible changes, and major is for backward-incompatible changes.

The bump-my-version bump command will use the configuration at pyproject.toml and will do these things:

It will increment the current version number depending on the parameter passed to it.
It will replace the old version with the new one in the setup.py, myapp/__init__.py, and README.md.
It will take care of correct versioning in the CHANGELOG.md.
Then, it will commit the changes and create a tag according to the pattern vMAJOR.MINOR.PATCH there.

Some further details

Instead of using pyproject.toml, you can use .bumpversion.toml, but that file is hidden, so I recommend sticking with pyproject.toml.

Final words

You can see this type of semantic versioning configuration in the paid Django GDPR Cookie Consent app I recently published on Gumroad.

Happy programming!

Cover photo by Jacob Campbell

How I Tested ReactJS-based Webapp with Selenium

Fri, 10 Apr 2020 12:00:00 +0000

For quite some time, I have been building a SaaS product - strategic prioritizer 1st things 1st. It's using Django in the backend and ReactJS in the frontend and communicating between those ends by REST API. Every week I try to make progress with this project, be it a more prominent feature, some content changes, or small styling tweaks. In the past week, I implemented frontend testing with Selenium, and I want to share my journey with you.

What can you do with 1st things 1st

1st things 1st allows you to evaluate a list of items by multiple criteria and calculates priorities for you to follow and take action. The service has 4 main steps:

Defining criteria.
Listing out things.
Evaluating things by each criterion.
Exploring the priorities.

Selenium is a testing tool that mimics user interaction in the browser: you can fill in fields, trigger events, or read out information from the HTML tags. To test the frontend of 1st things 1st with Selenium, I had to

enter the user credentials and login,
create a project from a blank project template,
add some criteria,
add some things to do,
evaluate each thing by each criterion, and
see if the generated list of priorities was correct.

Let's see how I did it.

Preparation

In 2020, Chrome is the most popular browser, and it's my default browser, so I decided to develop tests using it.

I had to install Selenium with pip into my virtual environment:

(venv)$ pip install selenium

Also, I needed a binary chromedriver, which makes Selenium talk to your Chrome browser. I downloaded it and placed it under myproject/drivers/chromedriver.

In the Django project configuration, I needed a couple of settings. I usually have separate settings-file for each of the environments, such as:

myproject.settings.local for the local development,
myproject.settings.staging for the staging server,
myproject.settings.test for testing, and
myproject.settings.production for production.

All of them import defaults from a common base, and I have to set only the differences for each environment.

In the myproject.settings.test I added these settings:

WEBSITE_URL = 'http://my.1st-things-1st.127.0.0.1.xip.io:8080'  # no trailing slash

TESTS_SHOW_BROWSER = True

Here for the WEBSITE_URL, I was using the xip.io service. It allows you to create domains dynamically pointing to the localhost or any other IP. The Selenium tests will use this URL.

The TEST_SHOW_BROWSER was my custom setting, telling whether to show a browser while testing the frontend or just to run the tests in the background.

The test case

In one of my apps, myproject.apps.evaluations, I created a tests package, and there I placed a test case test_evaluations_frontend.py with the following content:

import os
from time import sleep
from datetime import timedelta

from django.conf import settings
from django.test import LiveServerTestCase
from django.test import override_settings
from django.contrib.auth import get_user_model
from django.utils import timezone

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait


User = get_user_model()

SHOW_BROWSER = getattr(settings, "TESTS_SHOW_BROWSER", False)


@override_settings(DEBUG=True)
class EvaluationTest(LiveServerTestCase):
    host = settings.WEBSITE_URL.rsplit(":", 1)[0].replace(
        "http://", ""
    )  # domain before port
    port = int(settings.WEBSITE_URL.rsplit(":", 1)[1])  # port
    USER1_USERNAME = "user1"
    USER1_FIRST_NAME = "user1"
    USER1_LAST_NAME = "user1"
    USER1_EMAIL = "user1@example.com"
    USER1_PASSWORD = "change-me"

    @classmethod
    def setUpClass(cls):
        # …

    @classmethod
    def tearDownClass(cls):
        # …

    def wait_until_element_found(self, xpath):
        # …

    def wait_a_little(self, seconds=2):
        # …

    def test_evaluations(self):
        # …

It's a live-server test case, which runs a Django development server under the specified IP and port and then runs the Chrome browser via Selenium and navigates through the DOM and fills in forms.

By default, the LiveServerTestCase runs in non-debug mode, but I want to have the debug mode on so that I could see any causes of server errors. With the @override_settings decorator, I could change the DEBUG setting to True.

The host and port attributes define on which host and port the test server will be running (instead of a 127.0.0.1 and a random port). I extracted those values from the WEBSITE_URL setting.

The test case also had some attributes for the user who will be navigating through the web app.

Let's dig deeper into the code for each method.

Test-case setup and teardown

Django test cases can have class-level setup and teardown, which run before and after all methods whose names start with test_:

    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        cls.user1 = User.objects.create_user(
            cls.USER1_USERNAME, cls.USER1_EMAIL, cls.USER1_PASSWORD
        )
        # … add subscription for this new user …

        driver_path = os.path.join(settings.BASE_DIR, "drivers", "chromedriver")
        chrome_options = Options()
        if not SHOW_BROWSER:
            chrome_options.add_argument("--headless")
        chrome_options.add_argument("--window-size=1200,800")

        cls.browser = webdriver.Chrome(
            executable_path=driver_path, options=chrome_options
        )
        cls.browser.delete_all_cookies()

    @classmethod
    def tearDownClass(cls):
        super().tearDownClass()
        cls.browser.quit()
        # … delete subscription for the user …
        cls.user1.delete()

In the setup, I created a new user, added a subscription to them, and prepared the Chrome browser to use.

If the TEST_SHOW_BROWSER setting was False, Chrome was running headless, that is, in the background without displaying a browser window.

When the tests were over, the browser closed, and the subscription, as well as the user, were deleted.

Utility methods

I created two utility methods for my Selenium test: wait_until_element_found() and wait_a_little():

    def wait_until_element_found(self, xpath):
        WebDriverWait(self.browser, timeout=10).until(
            lambda x: self.browser.find_element_by_xpath(xpath)
        )

    def wait_a_little(self, seconds=2):
        if SHOW_BROWSER:
            sleep(seconds)

I used the wait_until_element_found(xpath) method to keep the test running while pages switched.

I used the wait_a_little(seconds) method to stop the execution for 2 or more seconds so that I could follow what's on the screen, make some screenshots, or even inspect the DOM in the Web Developer Inspector.

XPath

Selenium allows to select DOM elements by ID, name, CSS class, tag name, and other ways, but the most flexible approach, in my opinion, is selecting elements by XPath (XML Path Language).

Contrary to jQuery, ReactJS doesn't use IDs or CSS classes in the markup to update the contents of specific widgets. So the straightforward Selenium's methods for finding elements by IDs or classes won't always work.

XPath is a very flexible and powerful tool. For example, you can:

Select elements by ID: "//input[@id='id_title']"
Select elements by any other attribute: "//div[@aria-label='Blank']"
Select elements by innerText: "//button[.='Save']"
Select elements by CSS class and innerText: "//button[contains(@class,'btn-primary')][.='Save']"
Select the first element by innerText: "(//button[.='yes'])[1]"

You can try out XPath syntax and capabilities in Web Developer Console in Chrome and Firefox, using the $x() function, for example:

»  $x("//h1[.='Projects']")
←  Array [ h1.display-4.mb-4 ]

Login and adding a project

I started with opening a login page, dismissing cookie consent notification, filling in user credentials into the login form, creating a new project from a blank template, setting title and description, etc.

    def test_evaluations(self):
        self.browser.get(f"{self.live_server_url}/")
        self.wait_until_element_found("//h1[.='Log in or Sign up']")
        # Accept Cookie Consent
        self.wait_until_element_found("//a[.='Got it!']")
        self.browser.find_element_by_xpath("//a[.='Got it!']").click()
        # Log in
        self.browser.find_element_by_id("id_email").send_keys(self.USER1_EMAIL)
        self.browser.find_element_by_id("id_password").send_keys(self.USER1_PASSWORD)
        self.browser.find_element_by_xpath('//button[text()="Log in"]').send_keys(
            "\n"
        )  # submit the form

        self.wait_until_element_found("//h1[.='Projects']")

        # Click on "Add new project"
        self.wait_until_element_found("//a[.='Add new project']")

        self.wait_a_little()
        self.browser.find_element_by_xpath("//a[.='Add new project']").send_keys("\n")

        self.wait_until_element_found("//div[@aria-label='Blank']")

        # Create a project from the project template "Blank"
        self.wait_a_little()
        self.browser.find_element_by_xpath("//div[@aria-label='Blank']").send_keys("\n")

        # Enter project title and description
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Urgent and Important Activities"
        )
        self.browser.find_element_by_xpath(
            "//textarea[@id='id_description']"
        ).send_keys("I want to find which things to do and which to skip.")
        self.browser.find_element_by_xpath("//button[.='Next']").send_keys("\n")

        # Keep the default verbose names for the criteria and initiatives
        self.wait_until_element_found("//input[@id='id_initiative_verbose_name_plural']")
        self.wait_a_little()
        self.browser.find_element_by_xpath("//button[.='Next']").send_keys("\n")

If TESTS_SHOW_BROWSER was set to True, we would see all this workflow in an opened browser window.

I was creating the test by carefully inspecting the markup in Web Developer Inspector and creating appropriate DOM navigation with XPath. For most of the navigation, I was using send_keys() method, which triggers keyboard events. During the testing, I also noticed that my cookie consent only worked with a mouse click, and I couldn't approve it by the keyboard. That's some room for improving accessibility.

I ran the test with the following command each time I added some more lines:

(venv)$ python manage.py test myproject.apps.evaluations --settings=myproject.settings.test

The test case failed if any command in the test failed. I didn't even need asserts.

Adding criteria

Now it was time to add some criteria:

        self.wait_until_element_found("//h2[.='Criteria']")

        # Add new criterion "Urgent" with the evaluation type Yes/No/Maybe
        self.wait_until_element_found("//a[.='Add new criterion']")
        self.browser.find_element_by_xpath("//a[.='Add new criterion']").send_keys("\n")
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Urgent"
        )
        self.browser.find_element_by_xpath("//input[@id='widget_y']").send_keys(" ")
        self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")

        # Add new criterion "Important" with the evaluation type Yes/No/Maybe
        self.wait_until_element_found("//a[.='Add new criterion']")
        self.browser.find_element_by_xpath("//a[.='Add new criterion']").send_keys("\n")
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Important"
        )
        self.browser.find_element_by_xpath("//input[@id='widget_y']").send_keys(" ")
        self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")

        # Click on the button "Done"
        self.wait_until_element_found("//a[.='Done']")
        self.browser.find_element_by_xpath("//a[.='Done']").send_keys("\n")

I added two criteria, "Urgent" and "Important", with evaluation type "Yes/No/Maybe".

Adding things

Then I created some activities to evaluate:

        self.wait_until_element_found("//h2[.='Things']")

        # Add new thing "Write a blog post"
        self.wait_until_element_found("//a[.='Add new thing']")
        self.browser.find_element_by_xpath("//a[.='Add new thing']").send_keys("\n")
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Write a blog post"
        )
        self.browser.find_element_by_xpath("//textarea[@id='id_description']").send_keys(
            "I have an idea of a blog post that I want to write."
        )
        self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")

        # Add new thing "Fix a bug"
        self.wait_until_element_found("//a[.='Add new thing']")
        self.browser.find_element_by_xpath("//a[.='Add new thing']").send_keys("\n")
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Fix a bug"
        )
        self.browser.find_element_by_xpath("//textarea[@id='id_description']").send_keys(
            "There is a critical bug that bothers our clients."
        )
        self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")

        # Add new thing "Binge-watch a series"
        self.wait_until_element_found("//a[.='Add new thing']")
        self.browser.find_element_by_xpath("//a[.='Add new thing']").send_keys("\n")
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Binge-watch a series"
        )
        self.browser.find_element_by_xpath("//textarea[@id='id_description']").send_keys(
            "There is an exciting series that I would like to watch."
        )
        self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")

        # Click on the button "Done"
        self.wait_until_element_found("//a[.='Done']")
        self.browser.find_element_by_xpath("//a[.='Done']").send_keys("\n")

These were three activities: "Write a blog post", "Fix a bug", and "Binge-watch a series" with their descriptions:

Evaluating things

In this step, there was a list of widgets to evaluate each thing by each criterion with answers "No", "Maybe", or "Yes". The buttons for those answers had no specific id or CSS class, but I could target them by the text on the button using XPath like "//button[.='maybe']":

        self.wait_until_element_found("//h2[.='Evaluations']")
        self.wait_until_element_found("//button[.='maybe']")

        # Evaluate all things by Urgency
        self.browser.find_element_by_xpath("(//button[.='no'])[1]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")
        self.browser.find_element_by_xpath("(//button[.='yes'])[2]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")
        self.browser.find_element_by_xpath("(//button[.='no'])[3]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")

        # Evaluate all things by Importance
        self.browser.find_element_by_xpath("(//button[.='yes'])[4]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")
        self.browser.find_element_by_xpath("(//button[.='yes'])[5]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")
        self.browser.find_element_by_xpath("(//button[.='maybe'])[6]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")

        # Click on the button "Done"
        self.browser.find_element_by_xpath("//a[.='Done']").send_keys("\n")

These were my evaluations:

"Write a blog post" was not urgent, but important.
"Fix a bug" was urgent and important.
"Binge-watch a series" was not urgent and maybe important (because one has to have rest and feed imagination too).

Checking priorities

So in the last step, I got the calculated priorities:

        self.wait_until_element_found("//h2[.='Priorities']")

        self.wait_until_element_found("//h5[.='1. Fix a bug (100%)']")
        self.wait_until_element_found("//h5[.='2. Write a blog post (50%)']")
        self.wait_until_element_found("//h5[.='3. Binge-watch a series (25%)']")
        self.wait_a_little()

The results looked correct:

"Fix a bug" was of the 100% priority.
"Write a blog post" was of the 50% priority.
"Binge-watch a series was of the 25% priority.

Final words

Selenium needs a binary browser driver that lets you manipulate DOM in the browser from Python.
You can set a specific host and port for a LiveServerTestCase.
The Chrome browser can be displayed or executed in the background, depending on your settings.
XPath is a flexible and powerful tool to address DOM elements by any attributes or even inner text.
Selenium can trigger keyboard or mouse events that are handled by JavaScript functions.

I hope that my journey was useful to you too.

Happy coding!

Thanks a lot to Adam Johnson for the review.
Cover photo by Science in HD.

DjangoTricks blog

Wagtail Routable Pages and Layout Configuration

1. Project Setup

2. File Structure

3. Models

myproject/apps/articles/models.py

4. StreamField Blocks

myproject/apps/articles/blocks.py

5. Templates

articles/article_list.html

articles/article_detail.html

articles/blocks/cover_image_block.html

articles/blocks/description_block.html

articles/blocks/related_articles_block.html

6. Django Admin Registration

articles/admin.py

7. Migrations and Initial Data

8. Wagtail Admin Setup

Final words

How to Show a Waitlist Until Your Wagtail Site Is Ready

The Concept

1. Generate and store two secrets

2. Create the access-control decorator

3. Create the passphrase form

4. Build the cookie toggle view

5. Wrap the Wagtail catch-all with the decorator

6. Build the proxy home page view

7. Wire up the URLs

8. Protect every other app view

Final words

Using tox to Test a Django App Across Multiple Django Versions

The Example App: django-shorturl

Project Layout

Packaging: pyproject.toml

Test Settings: test_settings.py

The Core: tox.ini

envlist — the matrix

isolated_build = true

deps — conditional dependencies

commands

setenv

Writing the Tests

Installing Python Versions with pyenv

Install pyenv

Install each Python version

Make them all reachable at once

Running tox

Final Words

Understanding the Different POST Content Types

Standard Form Data

Form Data with Files

JSON String

Newline-Delimited JSON

Plain Text

HTML Text

XML Data

SVG Image

Binary Data

Protocol Buffers

Code to Play Around with

Moving a Domain to Another Registrar

The Situation

The Story

The Learnings

Per-object Permissions for Elasticsearch Lists in Django Websites

The Challenge

Things We Considered

Our Chosen Approach

Final Thoughts

User Timezones in Django

Time Zone Database

Timezone Changes

Best Practices for Django

Timezone Management for a Social Platform

1. Enable Timezone Support in Django Settings

2. Add a timezone Field to the Custom User Model

3. Detect Timezone on the Frontend

4. Use a Custom DateTime Field in Forms

5. Output Dates and Times in User's Timezone

Other Options

`myproject/apps/articles/models.py`

`myproject/apps/articles/blocks.py`

`articles/article_list.html`

`articles/article_detail.html`

`articles/blocks/cover_image_block.html`

`articles/blocks/description_block.html`

`articles/blocks/related_articles_block.html`

`articles/admin.py`

2. Add a `timezone` Field to the Custom `User` Model

Include the widget to your `base.html` template

Add GTM and Google Consent Mode snippets to your `base.html` template:

Passing the `WEBSITE_URL` to the templates