Django Advent Articles

Small Things

2010-03-05T00:00:00-11:00

Size doesn't matter

Throughout the advent, we've been covering some of the big new features in Django 1.2. Those are all awesome, of course, but some of my favorites parts of Django 1.2 aren't big. There's a bunch of really sweet little improvements that taken together really add up.

Here are some of my favorites:

Email backends

Django 1.2 gives you complete control over how Django sends email.

Django ships with a handful of standard backends, including the default SMTP backend and backends that log emails to files, the console, or memory.

We added this feature primarily to support those running Django on sandboxed hosting environments like Google's App Engine, but I think it'll prove useful in a bunch of other situations.

The API's very simple: an email backend is a class that implements a handful of methods:

send_messages(email_messages), which sends a list of EmailMessage objects.

open() (optional), which starts an e-mail-sending connection.

close() (optional), which closes the current connection.

For example, you could easily redirect email to a RESTful API with something like the following (which uses the httplib2 library):

import httplib2
from django.conf import settings
from django.utils import simplejson

class HttpPostEmailBackend(object):
    def send_messages(self, messages):
        http = httplib2.Http()

        # Convert the list of messages to JSON to posting
        body = simplejson.dumps([m.__dict__ for m in messages])
        headers = {'content-type': 'application/json'}

        resp, content = h.request(settings.EMAIL_TARGET_URL, "POST", body=body, headers=headers)

        # In Real Life we'd have error handling here — the request could
        # fail or time out. That's left as an exercise for the reader.

        # Return the number of successfully sent messages.
        return len(messages)

Then just wire it up in your settings file:

EMAIL_BACKEND = 'path.to.HttpPostEmailBackend'
EMAIL_TARGET_URL = 'http://example.com/email'

Read-only admin fields

You can now make fields read-only in the admin with the readonly_fields option:

class PersonAdmin(admin.ModelAdmin):
    readonly_fields = ['age']

These fields will show up in the admin on the add/change pages, but won't be editable. I see this being most useful when used with fields containing calculated data, or other such de-normalized information.

Smarter if tag

Designers and template developers, this one's for you:

One of Django's core design philosophies is that template languages shouldn't try to be programming languages. To that end, we've kept the template language deliberately simplisitic so that you'd be more or less forced to push advanced logic up into the view.

It's a delicate line, though, and over the years it's become clear that the if tag took "simplicity" a bit too far. Thus, Django 1.2's if tag has been made much more powerful. It now supports comparison operators, meaning you can do things like:

{% if user.username == "jacob" %}
  ...
{% endif %}

{% if driver.bac >= 0.8 %}
  ...
{% endif %}

{% if athlete in team.roster %}
  ...
{% endif %}

The tag supports ==, !=, <, >, <=, >= and in, all of which work like their Python equivalents. You can chain logic together with and, or and not.

This means that ifequal and ifnotequal are no longer necessary, though they're still available for backwards compatibility.

CSS classes on required/erroneous form rows

It's pretty common to style form rows and fields that are required or have errors. For example, you might want to present required form rows in bold and highlight errors in red.

Django 1.2 now makes it easy to add class attributes to the generated HTML for form fields and rows; you can then use CSS to style them accordingly. You'll specify these class names in your form definitions:

class ContactForm(Form):
    error_css_class = 'error'
    required_css_class = 'required'

    # ... and the rest of your fields here.

This'll make the generated HTML contain error or required classes as necessary. That is, if you use {{ form.as_table }}, the generated HTML might look something like:

<tr class="required"><th><label for="id_subject">Subject:</label> ... </th></tr>
<tr class="required"><th><label for="id_message">Message:</label> ... </th></tr>
<tr class="required error"><th><label for="id_sender">Sender:</label> ... </th></tr>
<tr><th><label for="id_cc_myself">Cc myself:</label> ... </th></tr>

And you can then target those rows in CSS:

tr.required { font-weight: bold; }
tr.error { background-color: red; }

Cached template loading

Quick, pop quiz:

Given these settings:

INSTALLED_APPS = ['app1', 'app2']
TEMPLATE_DIRS = ['/dir1/', '/dir2/']

and this line of code:

return render_to_response('example.html')

and an example.html containing:

{% extends base.html %}

{% block content %} Hi! {% endblock %}

what's the worst-case number of many disk operations Django must perform to render the response?

...

The answer's eight. Why?

Well, Django has to find example.html. To do so, it'll consult the template loaders. By default, these will first look in each application's directory for the template, and then in each of the TEMPLATE_DIRS. So, to find example.html, Django will try to load:

<app1's path>/templates/example.html

<app2's path>/templates/example.html

/dir1/example.html

/dir2/example.html

Next, because example.html uses inheritance, Django must repeat the exercise for base.html.

These disk accesses are fairly cheap, but not free. As sites group, you'll have many more applications, more template directories, and probably more complex template inheritance. It's not unusual to see a single template load operation force over 100 disk I/O requests.

Yuck.

To help mitigate this problem, Django 1.2 now ships with a cached template loader. This wraps existing template loaders, so unknown templates are looked up normally. However, once the template's been found, the cached loader stores the compiled template in memory, returning it directly for subsequent requests for the same template. That is, each subsequent request doesn't even need to touch the disk.

To enable the cached template loader, you'll put something like this in your settings:

TEMPLATE_LOADERS = (
    ('django.template.loaders.cached.Loader', (
        'django.template.loaders.filesystem.Loader',
        'django.template.loaders.app_directories.Loader',
    )),
)

If you've got other custom template loaders you can add 'em to the inner list and they'll be wrapped by the cached loader as well.

Warning

All of the built-in Django template tags are safe to use with the cached loader, but if you're using custom template tags that come from third party packages, or that you wrote yourself, you'll need to ensure that the implementation for each tag is thread-safe.

Et cetera

These are just my favorite small changes coming in Django 1.2. There are many other small improvements, including:

A new big integer field.

Customizable syntax highlighting for manage.py output, including highlighting of the development server output.

A new --failfast flag to manage.py test that'll stop the running tests and report immediately after the first test failure.

And many more.

I18N/L10N improvements

2010-03-04T00:00:00-11:00

Django 1.2 comes with a bunch of improvements for the internationalization and localization machinery, some of which I'd like to introduce here briefly.

Formats

Until 1.2, Django had limited support for handling form input and displaying model data for locales other than US-English. Try to display a datetime value in the format appropriate to your site's locale (e.g. decimal and thousand separators) and you'd soon discover you were out of luck.

Even though Django can handle strings and a few technical messages (e.g. DATETIME_FORMAT), it requires the use of 3rd party tools like the (fabulous) Babel to create fully localized sites.

Thankfully, Marc Garcia worked on that problem during his Summer of Code in 2009 and laid the foundations for Django's own formatting engine, which can handle input and output based on the currently active language. Among other features, this includes transparent support for the admin widgets and the template tags (e.g. {% date %}). For example, with the German locale activated, datetime fields are rendered in the admin like this:

Note how the datetime fields are actually rendered with a format common in Germany, 30.03.2010 instead of 03/30/2010.

The same applies to numeric fields, e.g. a DecimalField formatted with decimal and thousands separators:

Custom formats

Django provides format definitions for many locales based on imported data from the CLDR (Unicode Common Locale Data Repository), but sometimes you might want to create your own, because a format file doesn't exist for your locale, or because you want to overwrite some of the values.

To use custom formats, specify the path where you'll place format files by setting FORMAT_MODULE_PATH to the Python path where format files will exists.

The format files are not placed directly in this directory, but in a subdirectory with the name of the locale, and they must be named formats.py.

To customize the English formats in your project mysite, set FORMAT_MODULE_PATH to mysite.formats` and create a dictory structure like this:

mysite/
    formats/
        __init__.py
        en/
            __init__.py
            formats.py

Available formats

You are probably already familiar with some of the formats because they were avaibable in earlier Django releases but haven't been hooked up with a formatting mechanism.

Valid format strings in the date and time formats below are of the same format as the {% now %} template tag.

Formatting

DATE_FORMAT

Default: 'N j, Y' (e.g. Feb. 15, 2010)

Default formatting of date fields.

Default:	`'N j, Y'` (e.g. Feb. 15, 2010)

TIME_FORMAT

Default: 'P' (e.g. 4 p.m.)

Default formatting of time fields.

Default:	`'P'` (e.g. 4 p.m.)

DATETIME_FORMAT

Default: 'N j, Y, P' (e.g. Feb. 15, 2010, 4 p.m.)

Default formatting of datetime fields.

Default:	`'N j, Y, P'` (e.g. Feb. 15, 2010, 4 p.m.)

YEAR_MONTH_FORMAT

Default: 'F Y' (e.g. February 2010)

Default formatting of date fields, displaying only year and month. Used in the admin changelist, et al.

Default:	`'F Y'` (e.g. February 2010)

MONTH_DAY_FORMAT

Default: 'F j' (e.g. February 15)

Default formatting of date fields, only displaying month and day. Used in the admin changelist, et al.

Default:	`'F j'` (e.g. February 15)

SHORT_DATE_FORMAT

Default: 'm/d/Y' (e.g. 02/15/2010)

A short formatting of date fields.

Default:	`'m/d/Y'` (e.g. 02/15/2010)

SHORT_DATETIME_FORMAT

Default: 'm/d/Y P' (e.g. 02/15/2010 4 p.m.)

A short formatting of datetime fields.

Default:	`'m/d/Y P'` (e.g. 02/15/2010 4 p.m.)

FIRST_DAY_OF_WEEK

Default: 0 (Sunday)

Number representing the first day of the week. Used in the admin calendar widget.

Default:	`0` (Sunday)

DECIMAL_SEPARATOR

Default: '.' (Dot)

Default decimal separator used when formatting decimal numbers.

Default:	`'.'` (Dot)

THOUSAND_SEPARATOR

Default: ',' (Comma)

Default thousands separator used when formatting numbers.

Default:	`','` (Comma)

NUMBER_GROUPING

Default: 0

Number of digits grouped together on the integer part of a number. Common use is to display a thousands separator. If this setting is 0, then, no grouping will be applied to the number. If this setting is greater than 0 then the setting THOUSAND_SEPARATOR will be used as the separator between those groups.

Default:	`0`

Note

If enabled, the English locale has a default of 3.

Handling input

There are three separate format strings that will be accepted when inputting data on a date, time or datetime field. Formats will be tried in order, using the first valid match. These format strings are specified in Python's datetime module syntax, which is different from the one used by Django for formatting dates to be displayed.

Use DATE_INPUT_FORMATS for date fields, TIME_INPUT_FORMATS, for time fields and DATETIME_INPUT_FORMATS for datetime fields.

Also, have a look at the English format file to see the default values.

Smaller bugfixes worth noting

The `makemessages` command

`--ignore`

The makemessages command now features a --ignore option. It allows you to specify a glob pattern of paths it should ignore while traversing the filesystem in search of marked strings. This is most useful if you have a bunch of apps in your "project" directory that you don't want to translate in a monolithic project wide translation catalog but inside their own directories.

A simple example that will conveniently ignore all directories in the app directory

$ [path/to/myproject] django-admin.py makemessages -l de --ignore=apps/*

`--symlinks`

Additionally to the --ignore option there is now a --symlinks option, that will tell makemessages to follow symlinks on its quest to find files containing strings marked for translation — very useful if you keep your apps or templates in a central filesystem location.

Plural forms

The plural forms of the default Django translations are now automatically copied to newly created translation files (changeset 12445).

Order of precedence of translation catalogs

r12447 fixed a bug in Django that prevented translation catalogs located in the "project" directory from sucessfully overriding translations stored in apps.

Template filter in `{% trans %}`

r12472 introduced support for template filters in the trans template tag. The filters will be applied to the result of the translation, e.g.

{% load i18n %}

<h1>{% trans "Welcome!"|upper %}</h1>

Suppose the locale is set to French the template would be rendered as

<h1>BIENVENUE!</h1>

Support for IDN (Internationalized Domain Names)

Internationlized Domain Names are domain names that contain characters in a language-specific script, like Chinese or the Latin-based languages with diacritics, such as French, that are encoded with Unicode and saved in the Domain Name System as ASCII strings using the Punycode transcription.¹

URLField and EMailField got support for IDNs in r12474 and r12620 and can now handle domains like http://新宿駅.jp and http://räksmörgås.josefsson.org.

Current locale in cache

When the internationalization mechanism is enabled (if the USE_I18N setting is True), Django will append the currently selected language (e.g. en-us) to the generated cache key, making it easier for multilingual sites to use the cache middleware and decorator (r12546 and r12624).

First day of week

The admin calendar widget now automatically looks up the first day of the week in the format modules before rendering. In the formats.py simply set FIRST_DAY_OF_WEEK in formats.py to a value from 0 to 6, where 0 means Sunday, 1 means Monday, and so forth.

Internationalization/localization documentation

Luckily, Ramiro Morales refactored the internationalization and localization documentation (r12440), so have a look at the new i18n landing page to learn more about the intricacies of internationalization.

[1]	http://en.wikipedia.org/wiki/Internationalized_domain_name

Syndication Gets Classy

2010-03-03T00:00:00-11:00

Since its original release¹ Django has included support for generating RSS feeds for content. RSS, often expanded as "Really Simple Syndication", is an XML format used to publish frequently updated content in a standardized format. Similarly, Django also includes support for generating the Atom Syndication Format², which is also a standardized format for web feeds that offers a few expanded capabilities over RSS.

[1]	As of revision 3, from July 2005, close to 5 years ago.

[2]	Atom feeds were added in revision 1194, from November 2005.

Prior to Django 1.2

While Django's syndication framework offered a very easy way to generate valid XML, often with just a few lines of Python, it had a number of reported issues:

Feeds had limited flexibility in the various ways developers wanted to hook them into urls.py. For example, if your content is tagged and has HTML views at urls such as /tag/django/, it was not possible to append an rss/, for example, to the url to have /tag/django/rss/ — the URL had to adopt the pattern of /rss/tag/django/. This is because the URL was constructed in such a way that it pattern matches a prefix plus the rest of the url:
```
(r'^rss/(?P<url>.*)/$', ...),
```
For more complex url patterns, as above, the URL parsing was up to the developer as the feed class simply split the URL based on the remaining URL string. For example, the above URL regular expression would match both /rss/tag/django/ and /rss/tag/django/other/junk/, and it was up to the developer to catch this and deal with this in the Feed class. The problem here is it leads to URL parsing being in more than one place (i.e. somewhere other than the urls.py) and can easily be overlooked.
There was a little wart where a content item's title and description had to be defined in a Django template rather than in the feed class, whereas all other feed attributes could be defined in Python. Often these templates consisted of {{ obj.title }} and/or {{ obj.description }}, which made no sense to drop into template rendering.
Plus various other smaller bugs and warts and inconsistencies.

All of the above issues have been fixed in Django 1.2. In total, changeset 12338 closed 10 tickets against the syndication framework.

Class-based Views

Besides the above fixes, the change brought with it the introduction of class-based views. Previously, while the developer did create a Feed class, it was called via function-based view (django.contrib.syndication.views.feed), which split the URL to look for the key in the supplied feed_dict:

# Django 1.1 and prior
from django.conf.urls.defaults import *
from myproject.feeds import LatestEntries, LatestEntriesByCategory

feeds = {
    'latest': LatestEntries,
    'categories': LatestEntriesByCategory,
}

urlpatterns = patterns('',
    # ...
    (r'^feeds/(?P<url>.*)/$', 'django.contrib.syndication.views.feed',
        {'feed_dict': feeds}),
)

As of Django 1.2, the Feed subclass instance is itself a view which results in a cleaner url pattern and removes the need for a feed dictionary to be defined. It also removes the need to wildcard match the remaining URL, bringing the URL pattern matching back into urls.py and out of the Feed class:

from django.conf.urls.defaults import *
from myproject.feeds import LatestEntries, LatestEntriesByCategory

urlpatterns = patterns('',
    # ...
    (r'^feeds/latest/$', LatestEntries()),
    (r'^feeds/categories/(?P<category_id>\d+)/$', LatestEntriesByCategory()),
)

If you wanted to flip the URL around, as in the tag example above, you can now put the named pattern wherever you'd like:

urlpatterns = patterns('',
    # ...
    (r'^/categories/(?P<category_id>\d+)/feed/$', LatestEntriesByCategory()),
)

The Feed class gets a category_id passed to the get_object method of the class. The get_object method exists for feeds that publish different data given different URL parameters. The items method then takes the object returned by get_object and returns a list of objects to publish. In the example below, this is the last 30 entries in the given category:

from django.contrib.syndication.views import FeedDoesNotExist
from django.shortcuts import get_object_or_404

class LatestEntriesByCategory(Feed):

    def get_object(self, request, category_id):
        return get_object_or_404(Category, pk=category_id)

    def items(self, obj):
        return Entry.objects.filter(category=obj).order_by('-post_date')[:30]

    def title(self, obj):
        return "Entries for category %s" % (obj.category,)

    def link(self, obj):
        return obj.get_absolute_url()

    def description(self, obj):
        return "Recent entries in category %s" % (obj.category,)

For more details on each method that can be provided, see the feed class reference.

Using the New Syndication Framework

The function-based syndication feeds will be around until 1.4, but if you'd like to take advantage of some of the fixes class-based syndication feeds bring, upgrading is pretty straight-forward and the Django 1.2 release notes are probably the best source for explaining how to upgrade.

If you're not already using feeds, the syndication feed framework docs offer all you need to get started, including the aforementioned feed class reference for all the methods that are available to override.

Improved Bash Completion

2010-03-02T00:00:00-11:00

Django 1.2 comes with an improved Bash completion script that enables tab completion for custom management commands and command options.

The main problem with the old Bash completion script was that the completions were hardcoded. This meant that only built-in management commands (like runserver or dumpdata) could be tab completed. Developers, however, commonly extend Django through custom management commands, which made the Bash completion script inconsistent. Tab completion worked for built-in management commands, but not for custom ones.

The Bash completion script in Django 1.2 changes that. Instead of a hardcoded list of completions, the Bash script was modified to call a Python function in Django. This function dynamically generates a list of possible completions based on the actual project settings. It will take into account the custom management commands from applications listed in the INSTALLED_APPS setting.

Installation

To get Django's Bash completion working, you'll first have to install the bash-completion package. It's a set of functions that enhances Bash's, otherwise very basic, programmable completion. You can install the package manually (check the "Installation" section of the projects README file) or through a package manager that is available on your operating system. MacPorts and all major Linux distributions have it available in their package repositories (it is usually called "bash-completion"). Ubuntu already ships with the package pre-installed.

Next up, you'll have to source the django_bash_completion file. It can be found in the extras directory of an SVN checkout.

source /path/to/django_bash_completion

If you installed Django through pip or easy_install, the extras directory will not be available. In that case, you'll need to manually download the file.

The script can be sourced automatically at every login by writing the command in the ~/.bash_profile file.

Usage

Once installed, Bash will generate completions for the django-admin.py, django-admin and manage.py commands. To trigger the completion you'll have to press the tab key twice after the respective command. For django-admin.py, the output should look like this:

$ django-admin.py <TAB><TAB>
cleanup           inspectdb         sqlall            startapp
compilemessages   loaddata          sqlclear          startproject
createcachetable  makemessages      sqlcustom         syncdb
dbshell           reset             sqlflush          test
diffsettings      runfcgi           sqlindexes        testserver
dumpdata          runserver         sqlinitialdata    validate
flush             shell             sqlreset
help              sql               sqlsequencereset

The same applies to the manage.py command of a project. Note that the manage.py file must be executable for this to work. If this isn't the case, you can run chmod u+x manage.py to make it executable.

$ ./manage.py <TAB><TAB>
changepassword    flush             shell             sqlreset
cleanup           help              sql               sqlsequencereset
compilemessages   inspectdb         sqlall            startapp
createcachetable  loaddata          sqlclear          syncdb
createsuperuser   makemessages      sqlcustom         test
dbshell           reset             sqlflush          testserver
diffsettings      runfcgi           sqlindexes        validate
dumpdata          runserver         sqlinitialdata

As you might've noticed, there's a small difference between the generated completions. The manage.py completion will take your project settings into account. In the above example, the changepassword and createsuperuser commands were added because of the installed django.contrib.auth application.

Another example is the django-extensions application, which adds some useful commands to your Django project. Once you've installed it and added 'django_extensions' to your INSTALLED_APPS setting, the Bash completion will include the new commands:

$ ./manage.py <TAB><TAB>
changepassword          help                    shell_plus
clean_pyc               inspectdb               show_urls
cleanup                 loaddata                sql
compile_pyc             mail_debug              sqlall
compilemessages         makemessages            sqlclear
create_app              passwd                  sqlcustom
create_command          print_user_for_session  sqldiff
create_jobs             reset                   sqlflush
createcachetable        reset_db                sqlindexes
createsuperuser         runfcgi                 sqlinitialdata
dbshell                 runjob                  sqlreset
describe_form           runjobs                 sqlsequencereset
diffsettings            runprofileserver        startapp
dumpdata                runscript               sync_media_s3
dumpscript              runserver               syncdata
export_emails           runserver_plus          syncdb
flush                   set_fake_emails         test
generate_secret_key     set_fake_passwords      testserver
graph_models            shell                   validate

Something that wasn't possible with the old completion script is option completion. You can press the tab key twice after a command to view all of it's options. For the runserver command, the output should look like this:

$ ./manage.py runserver <TAB><TAB>
--adminmedia=  --noreload     --settings=    --verbosity=
--help         --pythonpath=  --traceback

The equal sign after an option name indicates that a value must be specified.

When using the option completion you'll probably notice that a space character will be added after each completed word. This can be a little annoying for options that require a value, as you'll always have to press the backspace key to set the cursor to the correct position. Sadly, there's no solution for this. A Bash completion script is initialized with an option that either outputs a space character after each completed word, or doesn't. There's no backward compatible solution to change the option after the script was sourced. Even with the option disabled, you'd have to manually enter a space character after each completed word. Bash 4.0 fixed this by introducing a new 'compopt' builtin that allows completion functions to modify completion options dynamically. But since some operating systems ship with an old Bash version (yes, I'm looking at you Apple) this feature couldn't be used.

Tip: Enable single-tab completion

If you have the Readline library installed, you can tell Bash to display the list of possible completions by pressing the tab key once instead of twice. This makes the tab completion even more pleasant to use. To change it, write the following in your ~/.inputrc:

# enable single tab completion
set show-all-if-ambiguous on

End

I hope this article has helped you to set up and understand the Bash completion script for Django 1.2. Hopefully it'll save you some keystrokes in the future.

Django 1.2 and CSRF

2010-03-01T00:00:00-11:00

CSRF, or Cross-Site Request Forgery, is perhaps one of the most overlooked classes of security vulnerabilities around. SQL injection attacks and cross-site scripting attacks are generally well-understood by any web developer worth their salt, but CSRF attacks often catch them unawares.

For those unfamiliar with web security in general, CSRF attacks essentially work by using the user's browser, and their pre-existing sessions and cookies, to call other sites with malicious data¹. They're not a server exploit, per se; they can only affect data on the target site that the user has access to.

Don't let that fool you, though; there's a large range of nasty things you can do with CSRF, from changing a user's profile to include spam links, to using an administrator's account to delete pages, or slip in backdoors. There have even been remote code execution exploits; it essentially allows the attacker access to anything you have behind authentication or permissions.

Thankfully, due to the prescience of the Django developers, Django has shipped with a CSRFMiddleware for quite a while now, albeit a slightly broken and ill-designed one. To understand more, you first need to understand the basic vector of a CSRF attack.

Anatomy of a CSRF Attack

CSRF works by making the user's browser submit a request (either GET or POST, depending on the target) to the site being attacked, with the content of the request being chosen by the attacker.

For example, imagine we have an online bank, with a form that allows us to transfer our money. I want to send some money to Alice (account number 00000001), so I fill out the form, and my browser submits this:²

POST /banking/transfer/ HTTP/1.1
Host: www.andrewsbank.com

amount=100&recipient=00000001

Now, imagine I don't log out of my banking session³, and browse to a malicious site. The site presents me with a button marked "Continue"; however, this is the HTML

<form action="http://www.andrewsbank.com/banking/transfer/" method="POST">
    <input type="hidden" name="amount" value="100" />
    <input type="hidden" name="recipient" value="00000002" />
    <input type="submit" value="Continue" />
</form>

As you can see, while the button might look innocuous, it's actually sending money to Bob's account (00000002).

This isn't a particularly sophisticated attack, nor is it a very sophisticated bank (I'd recommend Alice move her money elsewhere). Still, it should hopefully get you thinking about these kinds of attacks, and what they're capable of⁴.

There's plenty of resources about CSRF attacks on the web if you want to know more; the Wikipedia article gives a good grounding, and there's a good article by Jeff Atwood.

Protecting against CSRF

So, how does one stop CSRF attacks? The underlying method is to make sure every request originated from a real use of your site, and isn't spoofed.

The general method is to generate a token for every form (a nonce in security parlance), tied to the user's session, and to check the user's REFERER header. Tying it to the session is crucial; without it, the attacking site could simply perform a request to your site itself and obtain a valid token.

However, if you do check the session, and make sure you issued the token relatively recently, you can be reasonably sure it's a valid request. There's never really any certainty in computer security, and new attacks are being discovered all the time⁵, but you're still a lot better off.

Django and CSRF

Now, back to the aforementioned CSRFMiddleware. The version of this shipped with Django 1.1 was lacking, at best; it took the output of your site, ran a regular expression over it that looked for <form> elements, and then added an <input> element containing the CSRF token, which incoming POST requests were checked for.

As well as the obvious issue of it just running regular expressions over and inserting content into your response stream, it also added tokens to every POSTable form on the page - including ones which submitted to a third-party site, sending a valid CSRF token along with the other data. Armed with that token, the third-party site could then have used it to mount a CSRF attack on that user.

There were also other problems with the old CSRF protection; for example, it required SessionMiddleware to be enabled, and it wasn't really possible to enable it only for one app, as middleware is global.

What's Changed?

In Django 1.2, the regex-happy CsrfMiddleware has gone, and a new one has appeared in its place, sporting more features along with a distinct lack of regular expressions.

The new CSRF protection also requires that you insert the {% csrf_token %} tag inside any <form> elements you want to have CSRF protection, rather than just adding it to all forms. While this means you have to remember to add the tag to all forms which submit to CSRF-protected views, it also means that you won't accidentally send valid tokens off to third-party sites, like before.

In addition, the new protection also doesn't need to be enabled globally to be effective; there's a new django.views.decorators.csrf.csrf_protect decorator that you can use to enable CSRF on a per-view basis, and which all of the core Django apps now use, meaning your admin pages are secure even if you've turned off the middleware.

CSRF protection has also been promoted from being a contrib app to being part of Django core; this is because we believe that, much like auto-escaping, CSRF is a vital part of what a web framework should provide; it's very easy to get wrong if you implement it yourself.

Finally, the new protection uses its own cookie directly, rather than relying on Django's built-in sessions, so if you're using a different way of storing user state, such as signed cookies, it will still work.

(If you were an avid user of the old CsrfMiddleware, don't worry; it's been preserved as django.middleware.csrf.CsrfResponseMiddleware, and will stay around until Django 1.4.)

Gotchas

Any reasonably magic solution has 'gotchas', and even the new CSRF protection is no exception.

Firstly, due to cookie subdomain rules, the CSRF protection does not protect you against any malicious subdomains on the same domain as you; they can easily retrieve the cookie's value, and use it to send requests. The obvious solution to this is to make sure your subdomains are all trusted, although that can be more difficult than it initially sounds⁶.

Secondly, this protection won't work on outgoing AJAX requests, or custom content renderers. Incoming AJAX requests are not an issue; browsers have a same-origin policy, and so the CSRF protection won't try to check them (they can be identified by their headers).

However, if you're sending HTML out manually as strings, or via a different template library, you'll need to make sure you manually embed the token yourself - you can use django.middleware.csrf.get_token() to get a valid token to send.

Conclusion

CSRF is a hard problem, and one that isn't fully solved yet in Django (and may well never be). However, much like auto-escaping, the aim is to give you a solid, reasonably secure base to work from, and on this front, the new CSRF protection is a good move forward - it's now up to you to make sure you use it!

[1]	As Wikipedia puts it, it "exploits the site's trust in the browser". By contrast, XSS explots the browser's trust in the site.

[2]	This isn't valid HTTP, so please don't send angry letters. I'm trying to illustrate a point here.

[3]	A lot of people would just close the tab, or browse to another page, or hit back. If there's one thing that web users are good at, it's doing things in a completely illogical manner.

[4]	CSRF vulnerabilites have actually been discovered in online banking interfaces; see Jeff Atwood's article, linked below.

[5]	Such as click-through attacks, where your site is loaded in a mostly-invisible `<iframe>`, and a crucial button (such as 'delete') is then placed so you click on it, thinking it's part of the attacking site.

[6]	If any subdomain allows running arbitary external code in its context, you've got a security hole, for example. This may not seem bad at first, but lots of sites simply embed third-party JavaScript snippets, like Google Analytics, without a second thought.

Django Pony: A Retrospective

2010-02-26T00:00:00-11:00

In recent years, we have seen the meteoric rise to fame of Django's mascot, the much-loved Django Pony; with universal acclaim, she has been a guiding symbol, uniting Django developers worldwide.

Yet the turbulent history of this well-known figure is relatively unknown; her past is shrouded in mystery so deep that even our highly underpaid researchers had trouble unmasking it, even with the help of our anonymous source.¹

Django Pony's formative years are the most mysterious of all; there are no birth records for any member of Equus Volatilis from that period; given the rare breed, this is very odd, even taking the species' magical powers into consideration.

Indeed, there have been rumours that Django Pony is not all she appears to be; in a leaked document from late 2008, there are claims that Django Pony is in fact a mechanically-operated exoskeletal animatronic body - while this may seem implausible, after extensive research² we've determined it is indeed possible, provided you have enough funding, and at least two (but possibly three) mad scientists.

(Our attempts to replicate the effect with a cat, cardboard tubes, and fake fur were unsuccessful, although the research department believes this is due to an insufficient number of captions on the cat³.)

To try and verify this rumour, we sent one of our best researchers undercover to try and obtain a DNA sample for analysis, during one of Django Pony's few public appearances. Unfortunately, he returned with only a autographed hoofprint and a moderately brighter outlook on life; we suspect that Django Pony must be equipped with a lower-power version of Steve Jobs' Reality Distortion Field.⁴

However, while her early years are still mostly unknown, a great deal more is known about her more recent activities. In 2007, Django Pony moved to California in search of a new life. With an image that appealed to the younger generation, she was quickly doing photoshoots for small businesses, and was the logo for My Little Pony's short-lived "My Little Glue Factory" set.

Unsatisfied with small-time adverts and minor branding, she was looking for a big break - a major branding opportunity. Luckily, news came through that the Django community (well known as one of the world's finest social groups) was holding a conference nearby, and that local speaker Cal Henderson was to give one of the keynotes.

Cal is, of course, a master of deception and subliminal messaging. In exchange for a bribe⁵, he wove the idea of the Django Pony into his presentation, leaving subtle⁶ hints everywhere. One man - a certain Bryan Veloso - was particularly affected by Cal's psychological onslought, and succumbed, spearheading the overwhelmingly successful appointment of Django Pony as community mascot.

Drunk with her newfound fame, and pleased at how quickly and easily the community had succumbed, Django Pony toured the country, meeting Django developers everywhere, helping with bug reports and code reviews; the effect was overwhelmingly beneficial, and as the new year came around, a revitalized and energetic Django community headed for PyCon, led by their new figurehead.

Determined to make her first major public appearance holding the new position of Django mascot, she arrived in Chicago buoyed by optimism. However, things quickly turned ugly; a gang of rogue Django developers known only as 'Pinax' decided to kidnap Django Pony for their own nefarious purposes. Thankfully, Django Pony managed to escape with only a minimum of casualties.⁷

Shocked by her hostage experience, Django Pony turned to both alcohol and even one night stands to try and forget the experience, ending up wandering the streets of Chicago in a drug-induced haze. Luckily, she was found by a group of core Python contributors, out hunting the GIL⁸, and taken back to the hotel.

Both hungover and humbled, Django Pony realised the error of her ways, and decided to try and help the community, ushering attendees into Django talks, erasing mentions of Zope and Pylons from the programme, and even making up with a (now reassembled) James Tauber, as well as many other Django developers.

She went on to make appearances at EuroDjangoCon 2009 and then DjangoCon 2009, clearly a reformed character; she was crucial in finally getting Django 1.1 out the door, even flying down to Australia with a copy of the Trac database, allowing Russ to finally download more than five tickets an hour over his cutting-edge Australian internet.

However, there are still many unanswered questions about the Django Pony. How did she get her name? How do those wings fold away when she's not flying? More importantly, is there any substance to the rumour that she stole the magic removed from Django itself four years ago?⁹

Even with these mysteries, though, we can still conclude that Django Pony has definitely been of benefit to the Django community; in a year where so many other technologies were competing, Django has steamed ahead, with new features, new committers, and new releases, and if Django Pony has anything to do with it, this coming year will be even better.¹⁰

[1]	He went only by the codename 'Jake Obyan'.

[2]	We checked it using The Google, and it got lots of hits.

[3]	Of course, it's a well-known fact that captions slow down cats; they're often employed so a picture can be taken without motion blur.

[4]	Fortunately, Django Pony's version seems to lack the original's ability to cause mass hysteria and inflate stock prices.

[5]	The bribe was later confirmed to be approximately three hundred t-shirts.

[6]	We suspect this image may contain some kind of hidden message, although our analysts are still working on it.

[7]	Pinax team members are extremely modular and reuseable; the team was quickly reassembled, although they never did find James' spleen.

[8]	Guido van Rossum, supreme overlord of Python, has offered a significant prize for anyone who can kill the GIL; nobody has yet succeeded.

[9]	Simon claims it was out of his sight for only a few minutes; whoever took it was clearly either a professional, or distracted him with a minature zeppelin.

[10]	Unless Django Pony's evil twin has anything to do with it; she's on a personal vendetta against multi-db support. Alex Gaynor will soon be taken into protective custody.

Scaling Django

2010-02-25T00:00:00-11:00

I started using Django 0.96 about two years ago while working on the social messaging website Pownce. At that time, Pownce was one of the largest Django sites on the net. A lot of work had gone into making Django a robust platform for general web development, but the Django community hadn't put much effort into making the framework operate at "web scale." In other words, there wasn't much interest in providing facilities for running a distributed, high availability system like those found at Facebook, Google, and Amazon — And it showed.

Since then, a lot of work has gone into balancing the needs of the thousands of small-scale Django sites with the demands of the relatively few large Django installs. There's still a lot of custom code required to use Django at scale, but that's to be expected — scaling is very application specific. The best we can hope for is a framework that provides the basic tools we need, then gets out of our way. And, for the most part, Django does just that.

Scalability

Scalability is a property that indicates a system's ability to accommodate increased usage or increased data volume. There are lots of things that will need to scale as your site grows: your team, your business model, your issue tracking system, etc. But I'm going to focus on the technical bits. A scalable Django application should be able to handle more traffic without requiring changes to the code base or architecture.

Turns out, making a scalable Django app isn't all that different from making a scalable app using any other language or framework. That said, I will try to tie things back to Django where appropriate.

Attention!

Caution: scalability and performance are often confused. They are two very different things. If my website can respond to your request in 10ms it's fast. If it can respond to your request and millions of others simultaneously without failing, it's scalable. The two concepts are related, but not interchangeable.

Gathering Stats

Let's pretend we just launched a new Django powered website. As data starts pouring into our system we start finding pages that used to load instantly now take five or ten seconds to materialize. Don't panic. The first step at this point is to gather information. Unless we're running some terribly inefficient algorithm, chances are the database is our bottleneck (this will be a common theme as a site grows, by the way). So we need to figure out what queries are running, and hope we come across some low-hanging fruit.

Screwing around in a production environment is generally a bad idea, so hopefully there's a staging or development environment that we can work with. Using a staging environment is an accepted best practice, but it's also important that this environment has a dataset that's similar in size and composition to production. Joining two tables with a few thousand rows is practically free. Try the same join on two tables with millions of rows each and you'll get quite different results. A replica of the production database should do just fine.

The Django ORM is awesome. It makes it really easy to interact with a relational database and generate complex SQL in a clean, pythonic way. Unfortunately, it also makes it easy to generate inefficient queries that join across dozens of tables, or run thousands of queries for a single request. As our site grows, we'll need lots of insight into how our database is being queried so we can add indexes and tweak our configuration for optimal performance. But the ORM abstraction also makes the inner workings of the underlying database fairly opaque. Thankfully, there are a couple of tools and tricks that can help out.

The Django Debug Toolbar provides a ton of useful information. It captures every query executed during a request, and injects a debug panel into your output HTML with detailed statistics including query execution time, total request time, and logging output.

If you don't want to install the debug toolbar, you can roll your own tools by poking around at the django.db.connection object (or django.db.connections dictionary, if you're using multi-db). When debug mode is turned on, Django records query execution times in the queries attribute of the connection object:

>>> from foo.models import Bar
>>> Bar.objects.get(pk=1)
<Bar: Bar object>
>>> from django.db import connection
>>> connection.queries
[{'time': '0.071', 'sql': u'SELECT "foo_bar"."id", "foo_bar"."name", "foo_bar"."age", "foo_bar"."created" FROM "foo_bar" WHERE "foo_bar"."id" = 1 '}]

If you're just wondering what SQL a particular operation generates, the ORM can tell you that too (note that this functionality has changed in Django 1.2):

>>> from foo.models import Bar
>>> str(Bar.objects.filter(name='Mike').query)
'SELECT "foo_bar"."id", "foo_bar"."name", "foo_bar"."age", "foo_bar"."created" FROM "foo_bar" WHERE "foo_bar"."name" = Mike '

You can take this output and gain further insight using the explain statement on your database's shell (available on MySQL and PostgreSQL, at least). This should give you enough information to speed things up by adding indexes or adjusting the query.

Aside from simply paying attention to queries during development, you should use a monitoring package like Munin or Ganglia to track database operations, web requests, and other interesting technical and business metrics. If you see an increase in database queries you should be able to correlate it with a business metric like posts per second. If you can't, you may have a bug. Dig deeper.

Caching

So we've removed some joins, added some indexes, and done a bit of denormalization. But there are still a couple of big hairy queries that take too long and simply can't be eliminated. One of the easiest ways to increase throughput for this sort of expensive operation is to do them less often and cache the result. Don't use cache as a crutch — an app shouldn't completely fail when its cache is invalidated — but intelligent caching can dramatically improve the performance and scalability of a web app.

Django's cache abstraction can be configured to use a couple of different "backends." The only option that's appropriate for production use is memcached. There are also a number of high-level cache abstractions like the per-site cache and per-view cache. Unfortunately, these are pretty useless if you're serving up highly personalized pages. Rather than caching rendered HTML, I typically cache raw data using the low-level cache API.

The basic cache pattern is pretty simple, here's an example from the Pownce code base:

from django.core.cache import cache
class UserProfile(models.Model):
    def get_social_network_profiles(self):
        cache_key = 'networks_for_%s' % (self.user.id,)
        profiles = cache.get(cache_key)
        if profiles is None:
            profiles = self.user.social_network_profiles.all()
            cache.set(cache_key, profiles)
        return profiles

But caching is the easy part. As Phil Karlton famously said, "there are only two hard problems in Computer Science: cache invalidation, and naming things." So how do we invalidate the cache and keep from serving up stale information when our User record is updated? Fortunately, Django signals make this easy too:

from django.core.cache import cache
from django.db.models import signals
def nuke_social_network_cache(self, instance, **kwargs):
    cache_key = 'networks_for_%s' % (self.instance.user_id,)
    cache.delete(cache_key)
signals.post_save.connect(nuke_social_network_cache, sender=SocialNetworkProfile)
signals.post_delete.connect(nuke_social_network_cache, sender=SocialNetworkProfile)

Keeping cached counters is another great way to reduce database load. Count how many words are in this sentence. Unless you're rain man, you had to look at each word to come up with an answer, right? The same is true when you ask a database to count how many messages a user sent yesterday. The database may get some help from indexes, but it basically has to perform the query to get the result. And that's not cheap.

Luckily, as of Django 1.1 memcached's atomic increment and decrement operations have been exposed by the Django cache API. Again, the pattern is pretty simple:

try:
    count = cache.incr(cache_key, delta)
except ValueError: # nonexistent key raises ValueError
    count = count_the_hard_way()
    cache.set(cache_key, count)
return count

And again, signal handlers are an excellent way to keep counters up to date without peppering your code with increment and decrement operations.

Speaking of counts, Django's built-in pagination functionality is often the culprit of extraneous count operations. If you're using the Django object paginator you may want to toy with one of the several countless pagination implementations floating around the web.

There are a couple of features that the Django cache abstraction is missing. There's no way to cache an object without an expiration time (e.g., cache an object until it's explicitly invalidated), which is useful for objects that are fetched frequently, but rarely change, like user profiles. And the cache compression functionality that's part of the Python memcache library isn't exposed. If you're running out of memory, but have CPU cycles to spare, compressing cached objects can be a big win. Luckily, the Django cache infrastructure allows you to specify your own custom cache backend. So it's pretty easy to write a backend that exposes both of these features (full version on GitHub):

from django.core.cache.backends import memcached
from django.utils.encoding import smart_unicode, smart_str
class CacheClass(memcached.CacheClass):
    def add(self, key, value, timeout=None, min_compress_len=150000):
        if isinstance(value, unicode):
            value = value.encode('utf-8')
        if timeout is None:
            timeout = self.default_timeout
        return self._cache.add(smart_str(key), value, timeout, min_compress_len)

To enable our custom backend, simply use the full module and class name for the CACHE_BACKEND setting in settings.py:

CACHE_BACKEND = 'package.mymemcached://127.0.0.1:11211/'

Finally, If you find yourself caching lots of individual model objects, you may want to handle cache fetches and invalidation transparently to reduce code duplication and keep the Django ORM interface. A custom QuerySet with an overridden get method can serve lookups by primary key from cache, and invalidation can be handled via signals. This is a technique we used at Pownce, and it worked wonderfully. If this sounds useful, feel free to grab the automatic model object caching code I've posted on GitHub.

Load Balancing

Once we've got our queries optimized and cache infrastructure in place our next bottleneck is probably going to be our app server. That is, the actual machine running our Django app. At some point it will simply not have the capacity to serve additional requests, it will start backlogging connections, and eventually become unresponsive.

Luckily, Django uses a shared-nothing architecture out of the box. As long as you're using the cache or database session backends, there is no single point of contention at the app server level. Responsibility for maintaining a consistent application state is pushed down the stack (to the database) leaving the application tier free to scale horizontally. So, when one app server reaches capacity, simply add another.

Of course it's slightly more complicated than that. In order to spread load between multiple application servers we need to use a load balancer. There are several varieties of load balancers available. They can be hardware or software, and can operate at different levels of the OSI stack (usually layer 4 or layer 7).

Hardware balancers are highly efficient, highly reliable, and extremely expensive. A common setup for a large operation is to use redundant layer 4 hardware balancers (e.g., Big-IPs) in front of a pool of layer 7 software balancers. But those two hardware balancers, with a maintenance contract, can cost in the neighborhood of $100,000. If you don't have that kind of money, fear not. A good open source software balancer like Perlbal, nginx, or HAProxy will get the job done, and cost no more than the hardware they run on (which, by the way, doesn't have to be that powerful — Pownce had a single Perlbal balancer that ran with a pretty steady 0.00 load).

But the best way to reduce load on your app servers is much simpler — don't use them to do hard stuff.

Queuing

A queue is simply a bucket that holds messages until they are removed for processing by a client. Any expensive operation that an app performs should be queued and completed asynchronously. A simple example is note distribution on Pownce: like Twitter, Pownce had to distribute any note you sent to all of your followers. Notes were associated with recipients using a simple join table. So note distribution required an insert per follower. Since some users had thousands, or tens of thousands of followers, this operation could be quite expensive.

So, rather than distributing notes immediately, we'd stick them in a queue and send them to your followers out of band. Since you would notice if the message you just sent didn't show up in your own inbox, we'd do that insert synchronously. The rest of your followers would see your note as soon as a consumer got around to distributing it — typically this would happen in a few seconds to a minute.

Like load balancers, there are tons of open source queue packages. Some of my favorites are Gearman, RabbitMQ, ActiveMQ, and ZeroMQ. These tools provide lots of fancy features: brokers, exchanges, routing keys, bindings, etc. (You can read more here.) But don't let that stuff distract from the fact that this is a pretty simple concept. It's a to-do list, nothing more.

Assuming you want to use the ORM, the clients that consume queue jobs will likely be standalone Django scripts. These can be rather confusing to write if you're just getting started with the framework. Luckily, James Bennett has an excellent article on writing standalone scripts that should demystify the process. The basic pattern is pretty straightforward though. Just start your scripts thusly:

from django.core.management import setup_environ
from mysite import settings
setup_environ(settings)

There are also a number of interesting projects that aim to simplify queuing and provide a more Pythonic interface that integrates nicely with Django. Honestly, I haven't used any of these myself, but Carrot, Celery, and Flopsy all look useful.

The Database

With a couple of app servers, liberal use of caching, and an asynchronous queue mechanism for expensive operations we should be able to grow our site to a respectable size. However, up until this point we've been ignoring the one component of the typical Django app's architecture that is truly difficult to scale: the database. Like the app tier, in order to scale our data tier we'll need to add more machines. But in this case it turns out to be quite a bit trickier.

The simplest way to add capacity and fault tolerance to the data tier is to use replication. And the simplest form of replication is master-slave replication. With MySQL master-slave replication, for example, every data manipulation or data definition query is executed on the master then sent over the network to be executed on the slave, or "replica." Thus, write operations are performed by both machines. Since the replica maintains an up-to-date copy of the master's data set, read operations can be distributed between the two nodes. And since 80 to 90% of the database operations executed by a typical web app are reads, adding read capacity via replication can get you a long way. For what it's worth, Pownce never grew past this form of replication.

Tip

Tip: It's important to ensure that data is never written to your slave database. One simple way to achieve this is to configure Django to use an account with read-only access to your slave database.

Prior to Django 1.2 you had to jump through hoops to work with multiple databases. But, thanks to the tremendous efforts of Alex Gaynor and Russell Keith-Magee Django now has robust multi-db support built in. With a 'default' and 'replica' database configured, you can easily perform a read operation on your replica with the 'using' method:

User.objects.using('replica').get(username='mmalone')

Master-slave replication is pretty trivial to setup, but it has its own peculiarities, the most insidious being slave lag. Again, write operations are committed to the master database before they're sent along to the replica asynchronously. That means there is a period of time after a successful write operation, but before the replica has received the update, where the replica will be out of sync.

Initially, it seems like slave lag shouldn't be much of an issue. Lag is usually only a few hundred milliseconds. But with the common POST/Redirect/GET pattern used on the web, lag can be painfully obvious. Since GET operations often follow immediately after a POST, there's a good chance the data a user POSTs won't be replicated in time. The simplest solution to this problem is to force read operations to go to the master for a little while after a user performs a write operation. At Pownce we flagged users in memcache after a successful write and pinned them to the master database until the flag expired (usually a few seconds). It's not the prettiest solution in the world, but it worked.

More sophisticated solutions exist. MySQL can actually report the current slave lag, so you could generate a more accurate lag estimate and use that to adjust the period of time users are pinned to the master. Unfortunately, lag is likely to spike when your app is under heavy load. And if you're experiencing heavy load you probably don't want to start sending more queries to your master database. So you need to decide: would you rather guarantee consistency, send queries to the master, and risk overloading it and taking your whole site down; or would you prefer to send queries to your slave database despite lag, and risk returning inconsistent results to your users. Keep this dichotomy in mind, we'll be revisiting it shortly.

Since every replica has to perform every write operation, you'll start to see decreasing returns as the app's write volume picks up. At some point write volume will be too much for a single machine to handle. Assuming you decide to stick with a relational database, the only option is to start splitting your dataset into pieces. This process is called "sharding" or "federation", and it's a technique that's employed by most large scale web properties including livejournal, Flickr, Digg, Twitter, Facebook, YouTube, and Wikipedia, among many others.

Conceptually, federation isn't that hard: you take a large database, and you split it up into a cluster of smaller databases running on different machines. You can partition your data vertically, moving entire tables onto their own nodes, or horizontally, splitting a large table onto several nodes. More likely, you'll combine these approaches to suit your particular needs. Throw in a bit of consistent hashing, or a central records database to point you to the correct shard, along with some mechanism for generating primary keys (autoincrement won't work when you have multiple databases) and you're in business until it's time to rebalance nodes. Of course in practice all of this is far more complicated than I'm letting on. Regardless, the new Django multi-db functionality should give you all of the tools you need to implement your sharding scheme.

Once you've federated your database, you'll lose the ability to join across shards and will no longer be able to rely on transactions to guarantee the integrity of your data. You can design an architecture that retains some of the atomicity, isolation, and consistency characteristics of a typical relational datastore. But at a certain scale you'll have to relax some of these constraints. Here's why: there are three important characteristics of shared data systems that are deployed in distributed environments like the web:

Consistency: every node in the system contains the same data (e.g., replicas are never out of date).
Availability: every request to a non-failing node in the system returns a response.
Partition Tolerance: system properties (consistency and/or availability) hold even when the system is partitioned and messages are lost.

But here's the rub: you can only have two. Eric Brewer popularized this theory in 2000 at Principles of Distributed Computing, and it was later formally proven and dubbed the CAP Theorem. It's simple enough to prove to yourself without the rigor of an academic publication — assume we have two nodes: A, and B. They're a master-master pair that replicate one another's data. Now suppose a write occurs on node A. Ordinarily, node A would immediately pass the new value on to node B, and wouldn't return a successful response to the client until it had done so (probably using a two-phase commit protocol). But if there's a network partition that splits nodes A and B this is impossible. Either node A fails to record the new data, which means the system is no longer available, or it updates its local copy and returns success without notifying node B, which means the system is no longer consistent. Them's the breaks.

The relational database systems we're all used to were built with consistency as their primary goal. But at scale our system needs to have high availability. And as we add more servers the possibility of a network partition becomes an inevitability. So how do we reconcile this situation? Well, we've already discussed several methods of loosening the consistency constraints imposed by traditional database systems: caching, replication, and sharding are all essentially kludges that trade consistency for availability and partition tolerance. It'd be nice if we didn't have to re-invent the wheel though.

NoSQL

Over the last couple of years a number of specialized databases have emerged: graph databases, document databases, key-value stores, and various combinations. Collectively, this new class of non-relational datastore has been dubbed "NoSQL." It's a broad category, and the name is not entirely accurate since SQL is rather orthogonal to their goals, but I digress. Many of these systems are based on some combination of Google's Bigtable and Amazon's Dynamo, which are well documented (albeit closed-source and proprietary) examples of the sort of operation that a horizontally scalable data store can power.

Instead of federating your MySQL database, it might make sense to move some data to a Cassandra cluster, for example. If you're using a relational database to store data blobs that are only fetched by key, something like Redis or Tokyo Tyrant should improve throughput considerably. And if you'd like to continue querying your dataset, but want the improved availability and increased capacity of a distributed system, you might check out CouchDB.

Unfortunately, none of these solutions integrate tightly with Django. The Django ORM was designed specifically to interact with relational databases. In fact, the Python Database API Specification assumes a datastore that is relational and supports SQL. So taking advantage of these new tools means you can't use the ORM, and you'll probably have to rewrite any reusable apps that you're using.

The Future

As Django continues to mature, I'd love to see a low-level database API develop that doesn't assume a relational datastore. I'm not sure exactly what this would look like, but get, save, and delete operations by key are pretty universal. So a basic API wrapping that functionality would be a great start. It'd be terrific if models built on this API integrated with the admin, and core reusable Django apps like contrib.auth utilized it, instead of the less universal ORM.

At Euro DjangoCon 2009, Joe Stump gave a keynote presentation about "rethinking the stack." During that presentation, he encouraged Django developers to stop thinking of Django as a framework, and to start thinking of it as "command and control" for your web application. I think Django has a leg-up on the competition in this area. The Django admin is a powerful tool without a corollary in other environments. In the future, it would be great have core apps that integrate with, and provide insight into, mail delivery, queuing, and remote web services too.

The End

So there's no magic here: building a really big application is a really big commitment. You'll need to think carefully about your architecture and plan for future needs. But as long as you keep an eye on your stats, keep your caches primed, your queues short, and your data tier distributed, your site should be able to handle whatever the net can throw at it. Guess it's time to start working on that business model.

Honestly though, it's been a pleasure watching the Django community grow and thrive over the last two years. And I'm extremely excited to see what's in store for the next two. There's one thing I'm sure of though: it's going to be a nightmare keeping my "scaling Django" notes up to date.

I am happy report that I can finally say, without stipulation or hesitation, something I've been unable to say up until this point: Django 1.2 is capable of operating at web-scale.

Django Testing Improvements

2010-02-24T00:00:00-11:00

The testing realm in Django is mostly composed of two audiences: contributors to Django itself (running the substantial existing suite of tests) and application developers with their own suite of tests. Django 1.2 brings a few improvements to the testing infrastructure aimed at making the tester's life a little bit easier in both of these scenarios.

Multiple Databases

What Changed

The introduction of multiple database support touched many parts of Django, and the testing framework is no exception.

Django Developers

Django's tests now require a more detailed settings file in order to pass. If you run the test suite with your old settings file, which simply defined a single database, the MultiDB tests will fail (22 of them, to be exact). An example of the minimum settings file needed is supplied below. As of the Django Pycon sprint, this file is actually shipped with Django itself. So in order to run the Django test suite on SQLite, you don't have to touch your filesystem!

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
    },
    'other': {
        'ENGINE': 'django.db.backends.sqlite3',
        'TEST_NAME': 'other_db',
    }
}

To run the Test Suite, cd into the tests directory of your Django checkout, and run the following command:

./runtests.py --settings=test_sqlite

This will run with the supplied test_sqlite.py file, which is identical to the above settings file.

App Developers

This only really affects you if your application expects to be deployed against a multiple database environment. The Django documentation on testing has specific topics about Testing Master/Slave setups, which allows you to use the TEST_MIRROR setting on a slave database and have it mirror the master for tests. The Django TestCase also added the multidb attribute, which will flush all databases before running the test cases inside. By default, Django only flushes the default database, as an optimization.

However, unless you are specifically writing against a multible database environment, you should be able to safely ignore these changes.

Fail fast and fail early

If you are running the test suite, it is presumably because you expect it to pass. So when you run the suite and see that a test has failed, more often than not, that is all the information that you need to know. If you have a long suite, you don't want to have to wait for it to finish running before you see the error, since that error is what you care about.

What Changed

As of Revision 11843, Django added a --failfast option to the test runner. When this option is selected, Django will exit the test run once it encounters a failure. This is useful for certain work flows, including some forms of Test Driven Development, where you expect tests to fail often, and don't want to run through the entire test to see what has failed.

In Django 1.1 and before, if you were to <Ctrl+C> and cancel a test run, you would be treated with a nice ugly traceback with no information.

............^CTraceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_manager(settings)
  File "/Users/eric/svn-clones/django/template/__init__.py", line 285, in parse
    compiled_result = compile_func(self, token)
KeyboardInterrupt

When you try the same trick as of Revision 12034, the current test will finish running and exit gracefully.

.....................^C <Test run halted by Ctrl-C> .
----------------------------------------------------------------------
Ran 22 tests in 1.308s

OK

This allows you to get the same benefits from the --failfast option, even if you forget to set it at the beginning. Once you see a test fail, you can simply Ctrl+C, and you will get your failing test output back out.

Django Developers

This is important for developing against the Django source tree because the Django test suite takes a long time to run. The full suite can take upwards of 10 minutes, especially for bigger databases like Oracle. Previously, if you saw a failure you would just have to wait until the tests had finished running to be able to see the actual error. Breaking out early would result in the loss of the data you were after! Now you will be able to easily bail out on the first failure, or if you're forgotten the --failfast option, a simple Ctrl+C will do it for you.

App Developers

This is important for integrating testing into your work flow. One of the main ideas behind testing is that you should be running your tests while you are developing, and see if you are making errors while you are writing the code.

Adding the --failfast argument to whatever mechanism you have to run your tests after changes, will allow you to work faster when things break. You will be alerted right away to breakages in your code, and be able to fix them.

There are a number of other features that would be nice to have in this realm as well. Being able to run your full test suite, and then only run the failing tests from the previous run is another optimization here. This will hopefully make it into Django 1.3, or be realized as a stand-alone app before then.

There are also a number of efforts to integrate your Django tests with nose. So if you haven't started writing your test suite, looking at django-nose or django-sane-testing might make sense, as nose already has a lot of these features built in.

Keepin’ it classy

The (arguably) largest improvement to Django's testing infrastructure in 1.2 is class-based test runners. This is a big deal in that it allows for easier subclassing of the current test runner; anyone who wants to change how Django's tests are run has a much easier job to do.

What Changed

As of Revision 12255, the default test runner for Django is now a class. There is code that will detect if you are trying to pass in a 1.1-style function based runner, and do the appropriate thing. However, you should try to update custom test runners to the new style, because it will result in being less code and more maintanable.

This is only really applicable to application developers, because if you are running the Django test suite, you should be using the official Django test runner.

An example

In my Pony Utils project, I wanted to create my own test runner, and pre-1.2, that required me to copy 60 lines of code, only to edit a few (line 60). The run_tests function in Django is now 7 lines of code, which calls out to other methods. These other methods are easy to subclass in your own test runner, overriding only the parts needed to get the desired behavior.

The diff of the new and old show the changed lines of code. However, this really should be done in the suite_result function on the test runner class, since I only want to modify how the results of the tests are sent. This shows a place where the design could be improved.

The early Djangonaut gets the Pony

Testing pre-release software allows users to catch bugs and find use cases that the core developers haven't thought about — and address them before they are baked into a production release that must be supported. If a new feature isn't well tested before release, Django is stuck supporting that functionality for at least three releases going forward, and everyone gets suboptimal behavior.

When I was writing this article, I found a way that the new class based test runner could be better used. The suite_result function of the test runner wasn't being passed the actual suite to report the results on. The default Django functionality here is just to report the number of errors and exit, but I wanted to do more, and needed the suite. So I filed a ticket, with my example use case that wasn't supported, and hoped for the best.

Luckily this happened during the Pycon sprints, and Russell Keith-Magee accepted the ticket and patched Django within a day! This case is a great example of how using the pre-release version of Django can lead to making the software better.

Thanks for reading

I hope this article has made testing more enlightening for you, and has helped highlight the new features in the 1.2 release. Now go out and use them to improve the quality of your code, and Django itself!

jQuery in the Admin

2010-02-23T00:00:00-11:00

Changeset r12297 dropped a bomb on everyone: a new file in the django/contrib/admin/media/js directory called jquery.js. This change ushers in a new era for the Django admin; an era of fancy new features, pretty widgets, and better usability.

The case for jQuery

Despite being discussed in the past, integration of jQuery into the admin wasn't on the map until the GSoC admin-ui proposal this summer, which suggested some UI-heavy features that would be difficult without the use of a framework.

It’s easy to see the need for a Javascript framework in the admin. Raw Javascript in any decently sized project slowly approaches the size and complexity of a framework. That process becomes a minefield of dealing with cross-browser issues, workarounds, and normalization; a framework short-circuits that pain and lets developers focus on building cool stuff. We live in the future, and there’s just no good reason to write Javascript straight to the DOM APIs when we could be adding useful functionality (with fewer bugs) instead.

It’s also easy to see why jQuery is the framework of choice: jQuery has soundly won the JS framework wars. Almost 30% of the sites tracked by builtwith.com use jQuery. Picking something else would mean fewer knowledgeable devs, less available code, and a smaller community of support.

This doesn’t mean Django endorses jQuery

There's some concern in the community that the inclusion of jQuery appears to be an endorsement of one framework over another — that's not the case at all.

Django proper remains blissfully agnostic to toolkit choice. The admin is an optional app designed for end-users, not other developers. Data entry clerks are unaffected by the admin’s tooling, and developers are not constrained by it.

If you're modifying the Django admin, jQuery stays out of your way. For example, the admin uses jQuery.noConflict() to make sure the $ variable isn’t polluting your namespace, so you can use Prototype’s $() in your own widgets without any conflicts. Also, jQuery is only loaded on pages with widgets that require it.

Your apps can still use whatever makes sense for you. For the Django admin, including a well-tested, proven framework makes more sense than to add even more undocumented, unvetted spaghetti Javascript.

This does mean Django’s admin will grow much faster

The front-end of the admin has stagnated a bit. A perfect storm is brewing for some kick-ass new front-end features: jQuery, a newly (re)appointed design czar, and a general focus in the community on better usability.

Django 1.2 mainly focused on laying the groundwork for future admin-ui improvements and most visible changes won't be felt until Django 1.3 and later. However, a few things did make it in.

New features

One of the first jQuery features in the admin is the dynamic addition of new inlines. For a demo of this functionality, check out this screencast:

A lot of implemented features missed the cut for Django 1.2, like drag-and-drop reordering of inlines for models with an ordering field, and an autocomplete widget for Foreign Key and M2M relations to be used instead of the select drop-down (or raw_id_fields). Those features (and others) will most likely land in Django 1.3.

These are exciting times for the users of the Django admin. There’s already some admin hotness in 1.2, and it’s only going to get better by the time 1.3 ships. jQuery it up!

If Gets Smart

2010-02-22T00:00:00-11:00

Ah, the humble {% if %} tag. The cornerstone of Django template display logic.

Every beginner working with Django 1.1 or below will most likely have tripped over the fact that the {% if %} template tag only supports basic boolean logic.

Restricting the tag to only evaluate if a variable is "true" (i.e. exists, is not empty, and is not a false boolean value) was a purposeful restriction to the template system to encourage users to separate their presentation and business logic.

As a weekend project, I challenged myself to write a smarter if tag and then released it as a django snippet, which quickly gained popularity. About eight months later, replacing the if tag was proposed for Django 1.2 and championed by Luke Plant.

So what's new, Mr If?

From Django 1.2, the {% if %} template tag has been extended to support basic Pythonic logic, giving you the ability do the following types of things:

{% if you.friends.count > 5 %}You're popular!{% endif %}
{% if country != "NZ" %}Come and visit New Zealand!{% endif %}

You can still use template filters with this new logic. This gives you the ability for useful statements like {% if messages|length > 3 %}...{% endif %}.

If not Python

Remember this isn't Python: you won't have access to the None builtin or any python builtin functions.

{% if movie == None %} won't work. But if you need more than {% if not movie %} then you're doing it wrong anyway.

Mixing boolean logic

The tag also now handles mixed and and or boolean logic. But I'd encourage you to avoid using it regularly - it quickly leads to confusion. For example:

{% if staff or author and not expired %}
    <a href="{{ edit_url }}">Edit this</a>
{% endif %}

What happens under certain circumstances would be unclear to designers ("if the user is staff, but has also expired, what happens then?"). It would be much clearer calculating this in the view and passing through an editable context variable. Lets come back to this in a moment.

If you need a different order of evaluation than Python's operator precedence, use multiple if tags — using parentheses are not supported.

Missing variables

An important consideration is what happens when a context variable does not exist. It has been left simple: it equates to None.

Digging deep

Let's have a look at what's going on behind the scenes. If you're not one of those people that likes to pull apart radios, you can skip this section.

The parser in the snippet was simple; it worked but it wasn't very deep. However, the only major bug found in the snippet was with boolean operators not having precidence over comparison:

x or x == 0 was being parsed as (x or x) == 0 instead of x or (x == 0).

While this was able to be fixed, I was never really happy with the parser and Russell Keith-Magee noted this in his review of the submission branch. Luke Plant followed up with a link to an article on a parser implementation titled Simple Top-down Parsing in Python by Fredrik Lundh.

The snippet still takes a naive approach to parsing complex logical expressions. In the snippet, "A or B and C" is parsed as "(A or B) and C", not the pythonic way "A or (B and C)" (giving operator precedence to AND over OR).

This better parser implementation was used for the patch with the benefits of correctly implementing full operator precedence. It weights operators (starting from highest precidence) in the same order as Python:

or

and

not

in

comparisons (==, !=, <, >,``<=``, >=)

"Too much time?" challenge

Read through that Simple Top-down Parsing in Python article once or twice, trying to understand exactly what's going on and then... build the parser yourself.

If you need to, come back to the article again to give yourself some tips. The challenge is to re-implement the functionality from the overall ideas rather than the specific implementation given there.

Being a bit of a sadist, I gave myself this challenge after reading the first paragraph. Here's my take.

Just because you can...

There's a time and a place for using the new functionality of the tag. Keep your presentation and business logic separated. Perhaps that comparison would be better pushed to your view? Or a model method?

A lot has been written about the importance of logic separation, but here are a few reasons:

Being nice to designers

Keeping complex logic out of your templates allows your designers to focus on what they are good at without having to worry about complexities like operator precedence or advanced boolean logic.

Separation of thought, and providing readable code

It is easier to read business logic that is not intermingled with presentation logic. Similarly, it is easier to look at the presentation of a screen or read html code without having to sift through the database and security logic.

endif

There's only so much I can ramble on about a basic logic tag, so let us finish here. Thanks to Luke Plant for his hard work in getting this ready for Django 1.2.

Remember, the official documentation covers all new functionality. Here's a link to the if tag section for you.

Deploying a Django Site using FastCGI

2010-02-19T00:00:00-11:00

When developing a site with Django, it's so easy to simply pop open a console and type:

python manage.py runserver

With that single management command, our admin media files are served properly, the Python path is set to properly include our project root, and a simple autoreloading webserver is started on the port that we specify. It's all so easy!

It's no wonder that people are frustrated when it comes to putting their site into production: there are so many steps in the process, and that makes it difficult to learn and to get right. Unsurprisingly, this difficulty has lead to many articles being written about deploying a Django website. But almost all of those articles focus on how to deploy a site using Apache and mod_wsgi or mod_python.

There are some times, however, where Apache is not the ideal solution. Maybe it's that our VPS only has 256MB of RAM. Maybe it's that we want to avoid the added complexity of Apache in our setup. Or maybe it's that we simply don't like Apache. For these reasons, we might turn our attention to FastCGI.

First Things First

Before we can get started on doing this deployment, we need to make sure that we've gotten the base of our system set up. Firstly, we'll need a server on which to deploy our app. This can really be any server and operating system, but for the sake of simplicity we'll be assuming an Ubuntu-flavored Linux as a target.

Let's get the basics out of the way, including some compilers, development headers for Python, and setuptools:

sudo apt-get install build-essential python-dev python-setuptools

Nginx should also be installed, as it will act as our webserver. We can do this with:

sudo apt-get install nginx

We should also have daemontools installed, which is a collection of tools for managing services. We're going to be using it to ensure that our service stays up and running (or at least comes back to life), even in the event of failure or server restart. To install daemontools, type:

sudo apt-get install daemontools

Unfortunately, the package for daemontools in Ubuntu could use a bit of work, so we have to do a bit more ourselves to finish the installation so that it automatically runs at boot time. First, create a file at /etc/event.d/svscanboot with these contents:

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /usr/bin/svscanboot

Now make a directory at /etc/service by running this command:

sudo mkdir /etc/service

Finally, we kick off the daemontools process by running this command:

sudo initctl start svscanboot

We're also going to create a new user for our site:

adduser mysite

Since we want to be able to use the sudo command with our user, we'll also edit /etc/sudoers. Find the line where it says root ALL=(ALL) ALL, and underneath that add the line:

mysite ALL=(ALL) ALL

Now we'll switch to our new user:

su - mysite

That's it for the base of our system. We're purposefully not covering database, memcached, mail servers, source control, and other various other base services because they can vary so much depending on personal preference.

Setting up our Python Virtual Environment

Now that we have the base of our system installed, we can focus on the fun stuff. First we're going to install virtualenv, which is a tool for creating isolated Python environments. Then we'll use virtualenv to create an isolated environment for our app.

sudo easy_install virtualenv

With our newly-installed copy of virtualenv, we can go ahead and set up a new virtual environment:

mkdir ~/virtualenvs
virtualenv ~/virtualenvs/mysite

We've created a directory in our home by the name of virtualenvs, and inside of that we've created a virtualenv called mysite. Now let's start using that, and install pip to make installing packages easier:

source ~/virtualenvs/mysite/bin/activate
easy_install pip

Now we need to make sure that we have a package called Flup installed. This package is a set of useful tools for dealing with WSGI applications. Included in these tools is an adapter for taking a WSGI application and serving it as FastCGI (and SCGI and AJP...but that's beyond the scope of this article.) Django requires this to be installed before you can use the runfcgi management command. Using pip we can easily install this:

pip install flup

If we wanted to use database adapters, imaging libraries, or xml parsers installed into the system Python path, we would also want to make sure that those are accessible from our virtualenv by adding a .pth file in the virtualenv's site-packages directory:

echo "/usr/lib/python2.6/dist-packages/" > ~/virtualenvs/mysite/lib/python2.6/site-packages/fix.pth

The next step is to check out our Django code on the server (obviously git can be replaced with mercurial, svn, or even something like rsync):

git clone http://github.com/myusername/mysite.git

If your project has a pip requirements file, you can make use of that now:

pip install -U -r mysite/requirements.txt

Or if you don't have a requirements file, you can install the dependencies manually as needed (the following is an example):

pip install -U Django simplejson python-memcached

Choosing options for our FastCGI Server

Whew! We've come a long way in getting our system set up, and we haven't even gotten to the FastCGI part yet. Never fear, we're ready to do that now. Let's decide on what kinds of options we want to set when we start our FastCGI server.

The first choice to be made is which concurrency method we want to use:

threaded:: Running a threaded server will run one single process for all of your HTTP requests, which saves a lot on memory, but all the threads are subject to a single Global Interpreter Lock (GIL). This means that performance can be hampered on some CPU-intensive loads. Note that IO takes place outside of the GIL, so IO-intensive loads shouldn't be affected by it. Also some Python extensions are deemed to not be "thread safe", which means that they cannot be used with this concurrency choice.
prefork:: Running a preforking server will spawn a pool of processes, each with their own separate copy of Django and Python loaded into memory. This means that it will necessarily use more memory, but it's not subject to the aforementioned problems with the GIL or thread safety.

Let's assume we're interested in FastCGI because we're on a server without a lot of memory. Since the prefork method will use more memory, we'll choose the threaded method instead.

Now we get to choose some more options about how the server should act under load:

minspare:: How many spare processes/threads should the server keep around, at minimum, to be ready and waiting for future requests?
maxspare:: How many spare processes/threads should the server keep around, at maximum, to be ready and waiting for future requests?
maxrequests (prefork only):: How many requests will each process serve before it's killed and re-created? To prevent memory leaks from becoming a problem, it's a good idea to set this.
maxchildren (prefork only):: How many child processes may be handling requests at any given time?

We're running on a small 256MB VPS, so we'll choose some very modest settings of 2 for minspare, 4 for maxspare, 6 for maxchildren, and 500 for maxrequests.

Finally, we choose our last few settings:

host: What is the hostname on which to listen for incoming connections?
port: On which port should we listen for incoming connections?
pidfile: When the FastCGI server starts up, it will write a file whose contents are a process ID. This process ID is the pid of the master thread/process. This is the process which will handle OS signals like SIGHUP. This option specifies the location of that file.

After we've made our choices, we can start our server by running the runfcgi command:

python manage.py runfcgi method=threaded host=127.0.0.1 port=8080 pidfile=mysite.pid minspare=4 maxspare=30 daemonize=false

Note that we've added a daemonize=false flag--this should always be set (in this author's opinion, having a daemonize option at all is a design flaw in the runfcgi command.) Also note that running this command will result in a mysite.pid file being written out in your project directory, so it's a good idea to ensure that your source control ignores that file.

Now that we've verified that our FastCGI server starts up properly, let's quit out and move on to the next step: using daemontools to run this command and keep it running in the background at all times.

Having Daemontools Run our FastCGI Server

Daemontools will look inside all of the subdirectories in the /etc/service directory, and in each one it will look for an executable file named run. If it finds one, it will run that executable and restart it if it dies. So let's set up a mysite directory:

sudo mkdir /etc/service/mysite

Now let's make a little script to run our fastcgi server, use your editor of choice to write these contents to /etc/service/mysite/run:

#!/usr/bin/env bash

source /home/mysite/virtualenvs/mysite/bin/activate
cd /home/mysite/mysite
exec envuidgid mysite python manage.py runfcgi method=threaded host=127.0.0.1 port=8080 pidfile=mysite.pid minspare=4 maxspare=30 daemonize=false

There's nothing tricky about this--first we make sure we're in the correct virtualenv, then we change directory to the mysite project directory, and then we run the runfcgi command that we discussed earlier. The envuidgid mysite part simply ensures that the following command should be run by the mysite user instead of root.

The script has to be executable for daemontools to recognize it, so let's make sure it is:

sudo chmod +x /etc/service/mysite/run

Now we can verify that it's running by using the svstat command:

sudo svstat /etc/service/mysite/

The results should look something like this:

/etc/service/mysite/: up (pid 3610) 33 seconds

That means that the process is up, it's got a process identifier of 3610, and it's been up for 33 seconds. You can use the svc command to take the service down like so:

sudo svc -d /etc/service/mysite/

Then, if you run svstat on it again, you should get this output:

/etc/service/mysite/: down 4 seconds, normally up

To bring it back up, simply run this command:

sudo svc -u /etc/service/mysite/

A full list of svc commands can be found on online, and is essential reading if you're going to dive deeper into daemontools.

Configuring Nginx to Talk to our Server

We're nearing the finish line, all we have left to do is configure nginx to talk to our FastCGI server to get its HTTP responses and serve those to the user.

Ubuntu comes with a helpful file at /etc/nginx/fastcgi_params, unfortunately it's not quite right. It encodes a parameter named SCRIPT_NAME, but what our server really wants is PATH_INFO. You can either do a search and replace, or copy the contents below into the /etc/nginx/fastcgi_params file:

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  PATH_INFO          $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

Now we're going to create the definition for our site, let's use our editor and create the file /etc/nginx/sites-available/mysite with the following contents:

server {
    listen 80;
    server_name mysite.com www.mysite.com;
    access_log /var/log/nginx/mysite.access.log;

    location /media {
        autoindex on;
        index index.html;
        root /home/mysite/mysite;
        break;
    }
    location / {
        include /etc/nginx/fastcgi_params;
        fastcgi_pass 127.0.0.1:8080;
        break;
    }
}

This says to listen on port 80 (the standard for HTTP) for http://mysite.com/ and http://www.mysite.com/. It says that requests for /media should be served directly from disk from the /home/mysite/mysite/media directory. And most importantly, it says that anything else should be passed via FastCGI to our server.

Now let's hook this up via symlink:

sudo ln -s /etc/nginx/sites-available/mysite /etc/nginx/sites-enabled/mysite

And finally we can restart nginx to have the new settings take effect:

sudo /etc/init.d/nginx restart

Conclusions

We have now set up a very minimal server, using nginx to serve media at blazing fast speeds, and a pure-python FastCGI server to serve dynamic requests. Nginx speaks directly to the FastCGI server without any layers in-between. Using daemontools, we have complete control over that FastCGI process, and we can stop it, restart it, or change its settings at any time.

The really interesting thing is that with just a few small tweaks, this exact same stack could be used for a gunicorn, spawning, or paste solution. Instead of doing fastcgi_pass, we would simply use proxy_pass. We'd still use daemontools to keep the process running and control it. Almost every single other step of this article would apply.

This is a very viable alternative to the oft-touted stack of Apache/mod_wsgi and hopefully after reading this article, more people will consider it as a deployment method.

History of Model Validation

2010-02-18T00:00:00-11:00

Motivation

When working with Django's awesome (new)forms framework I always felt a slight pang of pain that something similarly awesome isn't available for Django's model layer.

Sure, you can create a ModelForm which is very cool and usually good enough when you are only dealing with user input, especially after newforms admin landed and enabled people to supply their own form class. But if you are not dealing with users, don't have a single origin of your model or are dealing with other developers instead, things can get out of hand pretty quickly. If, on the other hand, you could define your validation in one place and have it propagated to all necessary places (ModelForm, admin), that's where you can really be sure that you have done enough to keep your data in a consistent state.

First draft

I first started working on model-validation during PyCON in 2008. It was my first python conference and first time meeting other Djangonauts that I previously only communicated with over email. It was a very stimulating atmosphere which inspired me to try and tackle this problem.

At the end of the sprint I had an almost working prototype of model validation. Very naive and missing most of the features it has in 1.2 or even some that ModelForms have in 1.1 (for example validate_unique). I continued working on it throughout the year, especially during conferences. Right before 1.0 was out of the door I even got into a discussion with some core devs about how we can merge it. Unfortunately for me, but not for Django, newforms-admin was ready earlier and had a higher priority. After newforms-admin landed, my patch lied in ruins — it was easier to rewrite it from scratch then to try and apply the model-validation merge.

The came another conference (Djangocon 2008) where I got two legendary core developers to sit down and spare a few minutes discussing what model-validation should look like, especially focusing on validators. I tried to incorporate their ideas and failed horribly, ending up with another almost working patch in desperate need of some attention. Fortunately¹ during yet another conference I got to talking with yet another great Djangonaut of German origin who inspired me to apply for the GSOC program. After I answered all the "How come you are still a student?" questions, I got accepted and thus ran out of excuses why I never finished the patch and became the champion of almost working.

GSOC

Google Summer of Code is a great program by Google to encourage students to work on open source projects. Since I was (and still am) a student, managed to write a feature proposal before the deadline (with full four hours to spare) and showed that I am stubborn enough to potentially finish it, I got accepted with Joseph Kocherhans as my mentor. I spent the summer working on the patch, moving a lot of code around — from formfields to validators and modelfields, from modelforms to models and, finally, from old patch to new SVN branch.

At the end of the summer I got, what I thought was, a clean implementation of model validation. However, as I thought about it — it was very hard to create a model that would pass the validation and then fail to save itself into the database. In my eagerness to finish the patch and provide a clean solution I forgot one minor detail — people don't work with clean models, most of the time, when manipulating a model instance, you work with a partial model, often constructed by a ModelField with fields excluded.

Back to reality

It became apparent when Joseph merged the branch into trunk that this oversight on my part is something that will upset many people and, as such, is not acceptable to be a part of the next release. After a brief panic attack we dove into the code rethinking our approach to the patch, especially regarding ModelForms. Where our priority originally was to create a bullet-proof system that would make it very hard for people to corrupt their models, our new priority became to keep everything as before the merge and provide a clear migration path for people wanting to do more validation.

So, essentially after the fourth rethinking of our approach, we arrived at the API you see today in trunk, which you can read about in the Validators and the Validating Objects documentation.

[1]	I would like to think that this was fortunate for both Django and myself.

Django Template Improvements

2010-02-17T00:00:00-11:00

init

Almost every request to your Django site renders a template. Django handles these pretty snapilly — after all it's not so expensive to load the template file and compile it before rendering it each request. But what happens if your site renders hundreds of templates per request? If you're rendering 500 templates per request, even if they only take one millisecond each you've added half a second of overhead to each request. It turns out this is exactly the problem the developers at TypePad encountered when developing their Motion software. The result of this was a set of improvements to the templating system to allow for caching of compiled template objects, however this set of improvements also makes the template loader system more flexible in a few other ways.

Caching

The most obvious improvement changeset 11862 brings is the inclusion of a new cached template loader. This loader is instantiated with a list of other template loaders, and then it loads templates using those loaders, but caching the compiled template objects. This is helpful because, previously, the template was reparsed and compiled on each render. If you were rendering a lot of templates on each request (such as including another template inside of a loop) this could incur a lot of overhead.

If you want to update your site to take advantage of this loader it's just a matter of changing your settings to use this new loader. The cached template loader is a class-based loader that you configure with a list of other loaders that it should wrap. The wrapped loaders are used to locate unknown templates when they are first encountered. The cached loader then stores the compiled Template in memory. The cached Template instance is returned for subsequent requests to load the same template.

For example, to enable template caching with the filesystem and app_directories template loaders you might use the following settings:

TEMPLATE_LOADERS = (
    ('django.template.loaders.cached.Loader', (
        'django.template.loaders.filesystem.Loader',
        'django.template.loaders.app_directories.Loader',
    )),
)

Caveats

Most environments that Django is deployed in use multiple threads to serve Django requests. This presented one of the major challenges to supporting cached templates in Django. This is because a cached template means it could be used used from multiple threads simultaneously. Before this patch landed, Django template objects were not thread safe, rendering the same template object from multiple threads could produce incorrect results. However, with this patch, Django's template objects became threadsafe, including all of the builtin template tags (specifically {% block %} and {% cycle %}).

Before it is safe to use the cached template loader with your own applications you need to make sure that all of your custom template tags are threadsafe. Fortunately the Django documentation provides an excellent guide on thread safety considerations.

Bonus

Previously, template loaders were required to return the string of the template, rather than the template object. This was another problem the aforementioned patch solved. This was a problem because the entire point of a cached template loader is that it returns a precompiled template object. Therefore this patch allows template loaders to return template objects¹. However, this also allows one to build a template loader that works with other template languages (such as Jinja, Cheeta, Mako, or any other language that works by taking a string and a dictionary). Before if you wanted to use another template language with Django you either had to monkey patch Django extensively, or replace every call to something like render_to_response() with your own function that used your alternate template language. Now, in Django 1.2, all you need to do is write a template loader with a load_template() method that returns a template object.

Conclusion

This patch provides several important improvements to Django's templating systems' performance and flexibility. As always the documentation is an excellent resource when it comes to taking advantage of these features of the templating system.

[1]	Defined as something with a `render()` method.

Natural Keys

2010-02-16T00:00:00-11:00

One of the features introduced in Django 1.2 is really little more than a fix for a long standing bug. Natural Keys were introduced as a way to solve a very specific problem with fixture loading.

However, there's a little bit more to this feature than a simple bugfix. If you look outside the box, Natural Keys are a handy utility for simplifying the way you access certain types of models.

In order to understand when natural keys are useful, it helps to know little history.

In the beginning ...

When Django's test framework was first introduced, it didn't include any support for fixtures. If you wanted a test case to have data, you used the ORM to define those data in the setUp() method of each TestCase.

This worked, but it wasn't especially easy to use. Writing large collections of test data was needlessly complex. There was no way to create test data using a running Django project, and then dump that data in a way that test framework could use it. Reusing test data between test cases required the creation of libraries of code that set up certain test objects.

So - a really simple test case might look something like:

from django.test import TestCase
from library.models import Book, Author
from library.utils import is_good_book

class MyTestCase(TestCase):
    def setUp(self):
        self.a1 = Author()
        self.a1.first_name = 'Douglas'
        self.a1.last_name = 'Douglas'
        self.a1.save()

        self.b1 = Book(title='Mostly Harmless')
        self.b1.author = self.a1
        self.b1.save()

    def test_stuff(self):
        self.assertTrue(is_good_book(self.b1), True)

Functional? Yes. Pretty? Not especially.

Let there be fixtures!

Fixture support for Django tests arrived in January 2007. There wasn't anything especially revolutionary about Django's test fixtures — Django's serialization framework had been part of Django for a while. Test fixtures just leveraged that serialization support in a new and interesting way.

With the introduction of fixtures, you could put all the data definition into a separate file, in a format that was suited to data definition — JSON, XML, YAML, or any other format for which a serializer existed. As a result, the test code becomes a lot clearer:

from django.test import TestCase
from library.models import Book, Author
from library.utils import is_good_book

class MyTestCase(TestCase):
    fixtures = ['test_library.json']

    def test_stuff(self):
        b1 = Book.objects.get(name="Mostly Harmless")
        self.assertTrue(is_good_book(b1), True)

The data for the test is then defined in a separate test fixture file:

[
    {
        "id": 1,
        "model": "library.author",
        "fields": {
            "first_name": "Douglas",
            "last_name": "Adams"
        }
    },
    {
        "id": 1,
        "model": "library.book",
        "fields": {
            "title": "Mostly Harmless",
            "author": 1
        }
    }
]

This test fixture can be reused between tests without the need to create test data libraries. The production of test fixtures can also be semi-automated — if you have a running project, you can use that project to produce test data, and then use the dumpdata management command to produce test data that can be used as a test fixture.

Houston, we have a problem

The test fixture format is an improvement over programmatic definition of test data, but it isn't without flaws. For example, the test fixture uses primary keys to reference other objects — you define the author of a book by specifying "author" : 1 in your fixture.

This isn't an especially natural way of referencing objects — but, test fixture production can be automated, so unless you are manually writing fixtures, this inconvenience doesn't really affect you.

However, the problem is slightly more than an inconvenience for certain types of data.

The post_syncdb handler

Django provides a lot of runtime metadata about the objects under it's control. The contrib.ContentTypes app provides a unique identifier for every data type that is registered as a Django object. The contrib.Auth app automatically creates access permissions for every model that is defined. All this meta data is stored in normal Django models; these models are populated using a post_syncdb handler.

Whenever you run ./manage.py syncdb, Django emits a post_syncdb signal. By registering a listener for this signal, it is possible for your application to to execute code in response to a syncdb. In the case of contenttypes, this means creating a new content type entry; in the case of the auth framework, it means creating permissions.

So what's the problem?

The consequence of this signal-listening is that new objects will be produced in the database as the result of a call to syncdb. The handler is told the names of the new models that have been created, and creates content types, permissions, and other database objects to reflect those additions.

But what are the primary keys of these new objects?

The primary keys that are allocated are determined at runtime, and can't be predicted. The keys allocated during normal execution aren't necessarily the primary keys that will be allocated on a test database during test execution. The keys allocated on my database won't necessarily match those allocated on your database.

So, if you have a model that references content types — for example, if you have an object that uses generic foreign keys — then you can't create a fixture that will consistently reproduce that object. There is literally no primary key value you can use that will be guaranteed to uniquely identify a content type (or any other dynamically created data object for that matter).

This problem was the nub of ticket #7052. It wasn't just a problem for content types and permissions, either — although these two models in particular were the most frequently reported manifestations of the problem. Any model that had dynamically defined data would be prone to the same issues.

Ironically, this problem didn't actually exist in the pre-fixture world. When you programatically define a fixture, you aren't limited to using primary keys to reference objects (although you could if you wanted to). You could use any attribute or combination of attributes that uniquely reference an object.

What we really need is a way to programmatically look up a dynamic content type in a fixture instead of hard coding a primary key value.

Natural Keys to the rescue!

This is what natural keys are for. Instead of using primary key values to reference an object, you can use a natural key. So, if your fixture previously had a primary key reference for a content type:

"content_type": 37

... you can replace that primary key value with a natural key:

"content_type": ["library", "book"]

The natural key itself is just a list of data values that can be used to uniquely identify an object in the database. In the case of a content type, the natural key is composed out of the app_label and model that forms the content type record.

This natural key will be resolved into a primary key value when the fixture is loaded. The content type model itself provides the method that defines how to resolve the natural key reference into an actual database object.

Defining a natural key

So, how do you declare that your model can be loaded using natural keys? Easy — you define a get_by_natural_key() method on the default manager for the model. This get_by_natural_key() method accepts a list of arguments; those arguments are the elements of the list that are defined in the fixture. In the case of content types, the get_by_natural_key() method looks something like this:

from django.db import models

class ContentTypeManager(models.Manager):
    def get_by_natural_key(self, app_label, model):
        return self.get(app_label=app_label, model=model)

That's all there is to it. The get_by_natural_key() method is just a method that does a specific model lookup, provided under a known name so the serializers know how to find it.

Producing a natural key

Well... there is a little more. If all you want to do is load natural keys, then all you need is the get_by_natural_key() method. However, if you're going to use dumpdata to produce fixtures, it would be nice to have those fixtures produced with natural keys in place of primary key references.

To close the loop, you need to define one more method, this time on the model itself. If your model defines a natural_key() method, Django knows that references to objects of this type can be output as natural key references. The natural_key() method accepts no arguments; it returns the list of values that can be used to uniquely identify the object.

For content types, the definition of natural_key() looks like this:

from django.db import models

class ContentType(models.Model):
    # ...
    def natural_key(self):
        return [self.app_label, self.model]

By default, Django doesn't use natural keys when it dumps fixtures. If you want a fixture to use natural keys, you must specify the --natural option when you call dumpdata (or the use_natural_keys=True argument if you are programatically calling the serializers).

For example, the following would dump all the objects defined in the library application in JSON format, using 2 character indents, using natural keys whenever they are available:

./manage.py dumpdata --format=json --indent=2 --natural library

Some caveats and suggestions

Any object can define a natural key by providing the get_by_natural_key() and natural_key() methods. However, that natural key must be composed out of one or more attributes that are guaranteed to be unique in the database.

The easiest way to ensure this uniqueness is to define attributes to be unique_together in the Meta options of the model. This enforces uniqueness at the database level; it also adds an index to the database which will speed up the retrieval of objects by natural key.

Uniqueness doesn't need to be enforced at the database level, but the attributes you use in a natural key must be unique in practice. If you choose not to define the attributes as unique_together, you might find it beneficial to add indexes to the individual columns (or define a multi-column index). This will speed up the natural key lookups.

When using natural keys in a fixture, you need to be careful of the order in which you define fixture objects. Natural keys are resolved by performing a lookup on the database — which means you can't reference a natural key in a fixture before the object it references has been defined.

This means that the object that is referenced must already exist in the database before you load the fixture. If you call dumpdata, Django will dump objects in an order that contains no ambiguity. However, if you're manually defining fixtures, you need to be careful to ensure that the object order contains no forward natural key references.

Back to the books...

So, natural keys can be used to avoid a specific problem with test fixtures. Is that all they are good for? In short — no.

What is a natural key really? A natural key is a composite database key that can be used to refer to a specific object in a database. Although dynamic lookups in a fixture are one use for a natural key, they can also be used in day-to-day code, whenever you need to refer to an object in a 'natural' way.

Returning to our Author example — an obvious natural key for an author is the first name and last name of the author. By adding the two natural key methods to our Author model:

from django.db import models

class AuthorManager(models.Manager):
    def get_by_natural_key(self, first_name, last_name)
        return self.get(first_name=first_name, last_name=last_name)

class Author(models.Model):
    objects = AuthorManager()

    first_name = models.CharField(max_length=50)
    last_name = models.CharField(max_length=50)

    def __unicode__(self):
        return u'%s %s' % (self.first_name, self.last_name)

    def natural_key(self):
        return [self.first_name, self.last_name]

...the lookup of an author:

>>> Author.objects.get(first_name='Douglas', last_name='Adams')

...can be replaced with a natural key lookup:

>>> Author.objects.get_by_natural_key('Douglas', 'Adams')

Errr... So what?

Ok — a 6 character gain isn't really a big advantage. The elegance of using a natural key for object lookup only becomes apparent when you have a more complex natural key.

Consider the case of Permissions. The natural key for a permission is composed out of the code name for the permission, and the natural key for the content type that the permission relates to. So, the following permission lookup:

>>> Permission.objects.get(code_name='add', content_type=ContentType.objects.get(app_label='library',model='book'))

...can be replaced by the much more elegant call:

>>> Permission.objects.get_by_natural_key('add', 'library', 'book')

Taking this a step further, natural keys don't need to map directly onto database attributes at all. Natural keys can be composed out of any unique value that can be converted into a unique query for an object instance. So - if you can pass in a value (or list of values), and parse those values in some meaningful way into a unique object lookup, you can use those values as a natural key.

Conclusion

Natural keys may not be as complex as multiple database support or model validation, but it is a really important improvement introduced by Django 1.2. The bug that is fixed by introducing natural keys is a long standing problem that has caused more than one headache. There are also benefits to be had outside of the test framework.

So - time to upgrade and enjoy your natural keys!

Object Permissions

2010-02-15T00:00:00-11:00

Introduction

Nearly every web application provides authorization support to limit access to certain pages to a privileged group of users. Since the patterns for permission checks are usually the same for every application, it makes sense for Django to provide an API for it. Even better, Django does it in a clever way, so developers can use this API in their applications and, if they like, write backends for the API itself. This way end-users can switch those backends to perform permission checks against other databases or servers, simply by choosing another backend in the settings file.

Django has had support for authentication backends for quite some time. In Django 1.0 support for authorization backends got added in r6375. The code which previously checked the permission tables was moved from the user model into the ModelBackend and the user objects delegated their checks to the backend. Django 1.2 adds object permission checks to the existing backends in r11807. For now it is really just a foundation — the default backend does not support it, nor does the admin application take advantage of object permissions. Nevertheless we can easily enhance the admin application to perform the necessary checks and write our own backend.

Before we start writing our own backend it is important to know which possibilities do exist. Django supports multiple authentication/authorization backends at the same time, which allows us to implement the stuff we want to support and pass the remaining parts (eg. authentication) to default backends like ModelBackend and RemoteUserBackend. There are many ways to write such a backend — we could use a LDAP server and query it for permissions or use the database of another application (like Drupal, phpBB, etc.) — but to keep this article short and readable we will use a simple Model and store the data in our own database.

Writing an object aware authorization backend

Let's start simple

Django installs three default permissions for every model: add, change and delete. In our case only the change and delete permissions make sense, as the add permission is not related to an object but the model. Instead we will add a view permission. We will use the content types framework to allow greater reuse-ability. Enough of the talking, start with the model:

from django.db import models
from django.contrib.auth.models import User
from django.contrib.contenttypes models import ContentType

class ObjectPermission(models.Model):
    user = models.ForeignKey(User)
    can_view = models.BooleanField()
    can_change = models.BooleanField()
    can_delete = models.BooleanField()

    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()

There should be nothing new about this model, __unicode__, Meta and translations are left out for the sake of simplicity. As we used generic relations we will be able to display this model as inlines in the admin application everywhere we want permission checks. Before we start writing the actual admin integration, here is the code for the backend:

from django.conf import settings
from django.contrib.contenttypes.models import ContentType
from django.contrib.auth.models import User

from objperms.models import ObjectPermission

class ObjectPermBackend(object):
    supports_object_permissions = True
    supports_anonymous_user = True

    def authenticate(self, username, password):
        return None

    def has_perm(self, user_obj, perm, obj=None):
        if not user_obj.is_authenticated():
            user_obj = User.objects.get(pk=settings.ANONYMOUS_USER_ID)

        if obj is None:
            return False

        ct = ContentType.objects.get_for_model(obj)

        try:
            perm = perm.split('.')[-1].split('_')[0]
        except IndexError:
            return False

        p = ObjectPermission.objects.filter(content_type=ct,
                                            object_id=obj.id,
                                            user=user_obj)
        return p.filter(**{'can_%s' % perm: True}).exists()

Now what is new in this snippet? Compared to Django versions before 1.2 we added an obj parameter to the has_perm method and set the supports_* attributes to True. The supports_* attributes got added in 1.2 to allow new backends to advertise their capabilities. These attributes are needed to allow older backends to work unmodified in Django 1.2. As mentioned in the deprecation timeline, in Django 1.4 the attributes will become redundant and backends have to support obj and anonymous users. Django currently requires authenticate to be implemented, so we implement it and simply return None as we don't care about authentication. The only remaining code is the implementation for the has_perm method. What about other methods? There are none, at least none which we will need to write: As we don't support authentication we don't need to implement authenticate or get_user. has_module_perms does not make sense in our case and does not support the obj parameter either. We could implement get_all_permissions or get_group_permissions but we won't gain anything from it; neither does Django use them internally nor do we need them in this example. That said, those last two methods are the only two you might want to implement (in addition to has_perm) when writing your own backend. The code above should be pretty straightforward: We ignore permission checks where the object is None — ModelBackend can take care of them. Then we check if the user is authenticated and replace the AnonymousUser instance with a real user if he is not authenticated.¹ After that we extract the permission name from the perm parameter; this parameter is partially redundant as it also contains the name of the model and the application, which we don't need because the obj provides the same info already. But the Django admin application will pass such names in and therefore we will stick to this convention.² One important detail about this backend is that we don't inherit from ModelBackend, which means we will break Django if we use our backend as the only one, but luckily we can pass a list of backends to AUTHENTICATION_BACKENDS in settings.py:

AUTHENTICATION_BACKENDS = (
    'django.contrib.auth.backends.ModelBackend',
    'objperms.backend.ObjectPermBackend',
)

Now, as the needed code is in place, it is a good time to test it:³

>>> page = FlatPage.objects.get(pk=1)
>>> ct = ContentType.objects.get_for_model(page)
>>> user = User.objects.get(username='apo') # Don't use a superuser here!
>>> ObjectPermission.objects.create(user=user, can_view=True,
...                                 can_change=True, can_delete=False,
...                                 content_type=ct, object_id=page.id)
<ObjectPermission: >
>>> user.has_perm('flatpages.delete_flatpage', page)
False
>>> user.has_perm('flatpages.view_flatpage', page)
True
>>> # As mentioned above we could also use:
>>> user.has_perm('view', page)
True

Apparently everything is working fine.

Wrapping the admin application

Now that the backend is working it is time to take a look at the admin application. Luckily there are only 2 methods to provide: has_change_permission and has_delete_permission, so we will write a simple mix-in class. If you intend to change more, a subclass of ModelAdmin from which the other admin classes inherit might be more adequate:

from django.contrib import admin
from django.contrib.contenttypes.generic import GenericTabularInline
from django.contrib.flatpages.models import FlatPage
from django.contrib.flatpages.admin import FlatPageAdmin as FPAdmin

from objperms.models import ObjectPermission

class ObjectPermissionInline(GenericTabularInline):
    model = ObjectPermission
    raw_id_fields = ['user']

class ObjectPermissionMixin(object):
    def has_change_permission(self, request, obj=None):
        opts = self.opts
        return request.user.has_perm(opts.app_label + '.' + opts.get_change_permission(), obj)

    def has_delete_permission(self, request, obj=None):
        opts = self.opts
        return request.user.has_perm(opts.app_label + '.' + opts.get_delete_permission(), obj)

class FlatPageAdmin(ObjectPermissionMixin, FPAdmin):
    inlines = FPAdmin.inlines + [ObjectPermissionInline]

admin.site.unregister(FlatPage)
admin.site.register(FlatPage, FlatPageAdmin)

There is nothing really new in this snippet — we just override the has_perm checks to include the object and add an inline to the admin, so the user can assign permissions while adding a new flatpage. The last thing we are going to fix is the changelist view, which currently lists every object, even those we don't have access to. We could modify the queryset method to return only the items the user can edit, but this assumes that a database backend is used and that is not necessarily the case (e.g. LDAP, etc.). Instead we wrap the change view, catch the PermissionDenied error, tell the user what happened and redirect back to the change list:

# This code goes into ``FlatPageAdmin``
def change_view(self, request, *args, **kwargs):
    try:
        return super(FlatPageAdmin, self).change_view(request, *args, **kwargs)
    except PermissionDenied, e:
        messages.add_message(request, messages.ERROR, u"You don't have the necessary permissions!")
        return HttpResponseRedirect(reverse('admin:flatpages_flatpage_changelist'))

The above code works fine, but the message gets displayed with a green success icon, which is not really intuitive. For the sake of simplicity, tweaking the admin template is left as an exercise for the reader.

[1]	You can specify the id of the user you want to represent anonymous users in your settings file using `ANONYMOUS_USER_ID`.

[2]	Although the code supports a simpler form containing just the permission name too, but that might break other backends relying on that convention.

[3]	I will use the flatpages application to test them.

Everything I hate about Mingus

2010-02-12T00:00:00-11:00

Overview

Mingus is a small Django project, created as an experiment in practicing one of the key features of Django — reusable apps. Mingus defines no models itself, but currently leverages 30+ reusable apps to provide one complete blog engine project.

This article is about the obstacles faced and lessons learned managing an application that relies on so many reusable apps, and experiences in managing a small, open source project.¹ It has nothing to do specifically with all the awesome that exists in Django 1.2.

What is Mingus?

Mingus is a blog engine. The act of developing yet another blog engine from the ground up is a chore really. Mingus exists to do that chore for you. How nice of Mingus, no?

So, why is the project a (very) small success? Four key reasons:

The concept is simple — a blog engine
The concept is intriguing — use only reusable apps to provide all of its features (blogging, commenting via Disqus, admin image cropping, inlines, WYSIWYG editor, debugging, contact form, etc.)
It's a learning tool on how developers can leverage reusable apps in their project, and a forced introduction to virtualenv and pip.
It has a minimal, attractive template system.

There is one thing that Mingus really strives to do, and that is to take all of the combined complexity of 30+ reusable apps, and reduce it into one project that's brain dead simple to get up and running. Literally, this is what it takes to get started:

mkvirtualenv myblog —-no-site-packages
workon myblog
cdvirtualenv
git clone git://github.com/montylounge/django-mingus.git
cd django-mingus/mingus
pip install -r stable-requirements.txt
cp local_settings.py.template local_settings.py
./manage.py syncdb
./manage.py loaddata test_data.json
./manage.py runserver

For those not familiar, the above commands mkvirtualenv, workon, and cdvirtualenv requires you have virtualenvwrapper installed. I included the reference to mkvirtualenv and the others as a dependency in the documentation as a way to introduce developers to the virtualenvwrapper project if they were not already familiar.² I believe it's a terrific tool that every Python developer should at least be familiar with, and I snuck in the reference so that maybe it would introduce a new user to the project. You'll see that tactic used quite often if you take a look throughout the code base.

Now we know it's really easy to get started with Mingus, so let's dive in...

Be “that guy” (or girl)

If you ever started an open source project that gained even a tiny bit of momentum, then you've received feature requests of varying kinds. Eventually someone will come along who is using your project but they need a few extra features that would help him/her along. Their requests end up in your email inbox and them on your issue list quite often.

As a project maintainer, this is where you need to start making some tough decisions. Are the features requested within the scope of your project's mission? Do they help make your project more complete? Or are these requests solely for the benefit of this one individual? Learning when to say "no" to a request is essential to the successful management of any project. But if the requests are practical, and if you care, then you are sensitive to things that should be done correctly and now you've added another action item to add to your to-do list.

So you're a good project lead and you fix the bug or add that feature. And if you really care, while you are refactoring code and you're using your project daily³, then you start thinking like "that guy" and you start seeing ways to make your project better. So you spend more time adding new features that you know are reasonable, and relative to your project's focus. Features that you may have first overlooked, or simply shrugged off.

I hate that guy because he adds to my to-do list, but I love that guy because he's making me a better developer, and he's making my project better. He's my QA. He's gathering my requirements. Mingus wouldn't be the tiny success that it is if it wasn't for that guy(s).

My advice to you — be that guy (or girl) and make me hate you for it.

Don't make me restart Apache

If there was one feature that I believe the Django admin is missing I would say application wide settings management.

The one request I receive on almost every single project I have every been on is the ability to manage application settings via the admin. I don't care if it's a Django project or a ASP.Net project — every single admin user has needed the ability to manage application wide settings via the admin UI. But there's no current best practice in managing application wide settings in Django. We need one, and it needs to be available via the admin.

Well, actually, there is a best practice out there, but it ends up being a pain point. The current best practice is to place application settings in the settings.py file of your project. Here's an example of how to best implement retrieving that value:

from django.conf import settings
post_list_count = getattr(settings, "post_list_count", 20)

The above uses getattr to retrieve the post_list_count value from the settings instance. If one doesn't exist it defaults to 20. This is terrific and follows the best practice of "sane defaults".

Now, take this example, and lets assume every app in your project (Mingus for example) requires that one setting is defined per app in your settings.py — in Mingus' case you would now have 30 additional values to manage in settings.py. It's not horrible, but it could lead to an extremely complex settings.py file when we're all looking to minimize our settings files, aren't we?

When following the above convention, if a settings value change is requested, a developer/sysadmin then needs access to the server to update the settings.py file themselves. Moreover, whoever handles the change request also needs to restart the web application server for the change to take affect. This is less than ideal.

Lucky for us, two reusable apps attempt to provide that solution:

I'm not sure which, but one of these solutions (or some of their shared concepts) should become the convention and ideally an approved contrib application.

Sure there are simple settings value:

BLOG_PAGE_SIZE = 20

And there are more advanced:

DEBUG_TOOLBAR_PANELS = (
    'debug_toolbar.panels.version.VersionDebugPanel',
    'debug_toolbar.panels.timer.TimerDebugPanel',
    'debug_toolbar.panels.settings_vars.SettingsVarsDebugPanel',
    'debug_toolbar.panels.headers.HeaderDebugPanel',
    'debug_toolbar.panels.request_vars.RequestVarsDebugPanel',
    'debug_toolbar.panels.template.TemplateDebugPanel',
    'debug_toolbar.panels.sql.SQLDebugPanel',
    'debug_toolbar.panels.signals.SignalDebugPanel',
    'debug_toolbar.panels.logger.LoggingPanel',
)

The solution? Beyond a simple key/value store each application could provide a handler. Django would provide default handlers of course... IntegerSettingHandler, StringSettingHandler, etc. But an application like django-debug-toolbar would provide a DebugToolBarHandler. This will allow the Settings app to have a standard API that any application can interface with via the Settings API but each custom application provides it's own custom handler logic to execute its rules on its own. And maybe I'm just crazy?

There's also the extra query factor for retrieving these values if they exist in the backend store. So a sensitivity towards performance is required.

Right now if I want to add an application to Mingus I try to think about the needs of the non-technical end user. Would they want to be able to change this setting via the admin? If yes, and the reusable app defines a settings.py required value, I have a tough decision to make. Do I fork that app and add the setting to a model which can be updated in the admin⁴ or do I just fold and give in, including the app and dropping another value into the project's settings.py?

Having a contrib app that resolves these issues would reduce complexity, maintenance, needless forks of code bases, and improve app flexibility and integration.

You can get with this Setting or that Setting

As if the previous global settings management discussion wasn't exciting enough, it's now time to talk about managing those settings files.

In Mingus I package two settings files:

settings.py
local_settings.py

The former maintains all the application wide settings. The latter is the override, allowing the developer to override various settings on her machine, and allowing the various stages of your environment (dev, staging, production) to have their own local_settings.py file defining environment specific setting values (think database settings, filesystem settings, debugging, etc).

This has been a convention I've come across a few times before when looking at other projects, so I stuck with this basic pattern.

But it's not the final answer. If you were to take a look at Daniel Lindsey's blog post Better Local Settings you'll see one proposed solution. Then read the comments of his post and you'll see a few other solutions, highlighting the fact that we need a standard. In fact, the popular DjangoDose podcast proposed their solution in their Handling Development, Staging, and Production Environments using the FLAVOR concept. But again, take a peek at the comments and you'll see another handful of alternative solutions used by other developers.

A contributer to Mingus suggested I take a look at the Transifix team's documentation Using a list of conf files on how they manage their settings files as a best practice, which looks interesting as well. The simple fact that wiki page for various solutions in managing your settings files even exists highlights the need for a standard.

One project that recently found its way on my radar is Django-Config from Nowell Strite, Shawn Rider and now supported by Tareque Hossain. Strite and Rider both work at PBS and recently detailed the obstacles they run into supporting the various projects and reusable apps across their infrastructure with their Pluggable, Reusable Django Apps: A Use Case and Proposed Solution presentation at DjangoCon 2009.

Django-Config defines itself as "...an easy way to maintain multiple configurations for django. It relies on the concept of having a shared configuration file (base) and a per user/ server custom configuration file (dev1/ dev2/ local/ staging). settings.py combines the base & custom configuration and loads it up." I have yet to give the project a run myself but assuming the complexity of the infrastructure that PBS maintains I'm going to believe that there's a few nuggets of tested and refined goodies in there.

So I'm left not knowing what to do. For now, I'll keep with the basic implementation.

Static Media? No you didn’t!

Anyone who has ever authored a Django reusable app has asked themselves the question, where do I put the static media? What do I name the directory? Do I name it /media/ or do I name it /static/? Where do I place it on my file system?

A perfect example is Simon Willison's django-cropper reusable app I recently integrated into Mingus. Willison recently left this git commit message, "Finally managed to get the package to include the template... no idea what I should do with the static file dependencies though". It's a good question. What does a developer do with static media dependencies? Do they include the files in their project? Do they tell the user to go download them from XYZ? If they do include the files, where do they place them in their app?

I believe there is an answer... Django-StaticFiles.

The project stems from the Pinax project that faces this same obstacle in a much larger scale. So if anyone knows a solution, the Pinax crew would. I'm not going to dive into the finer details of the project, as it provides a terrific set of features and functionality, but what it outlines in its implementation is an easy to follow standard for reusable apps and static media management. Maybe it should become a contrib app, or at least the convention we all look to?

Upload this pal

While we're here talking about media management, let's also talk about files uploaded via the Django admin. I believe we should also have a default convention here too — the /uploads/ directory off MEDIA_ROOT. Far too often I'll grab an application that has this:

photo = models.ImageField(upload_to="/images/")

That helps no one. The convention should be MEDIA_ROOT + "/uploads/app_name/" as the default, and any directory defined in the upload_to parameter is appended to the default, like so (in my photobooth app):

photo = models.ImageField(upload_to="images")

By default this would result in MEDIA_ROOT + /uploads/photobooth/images/ file path.

I'm simplifying the underlying complexity, obviously, but I do believe a sane default would provide better asset management, and again make reusable app integration less invasive for these cases.

i18n gets no respect

I'll make this as short and sweet as possible. The internationalization features in Django are amazing. Having built one multi-lingual site from the ground up, and benefitting from the features Django provides out of the box for this i18n, it's a damn shame more reusable apps don't internationalize their app from the start (and I'm to blame here myself — let's just be honest).

Here's the two simplest ways to at least lay the groundwork for internationalizing your application. Let's take a models.py for this example. All you need to do is this:

from django.utils.translation import ugettext_lazy as _
...

class Post(models.Model)
    description = models.TextField(_('description), help_text=_('The description of your post'))

    class Meta:
        verbose_name = _('post')
        verbose_name_plural = _('posts')
...

Now in your templates all you need to do for the text laying around is this, example landing.html:

{% load i18n %}

<h4>{% trans "Blog roll" %}</h4>

And that's about it. As always, there's a little more under the hood, so make sure to read the docs which covers everything you need to know in getting started, but for the most part the above gets your app 80-90% of the way there.

And if you are deploying a multi-lingual application you will want to take a look at these apps:

And take a look at this excellant article which reviews a handful of reusable apps to help you with i18n integration — Dynamic Translation Apps for Django. The fact is that if you are not internationalizing your app then you are a bad person. No, but seriously, if you aren't internationalizing your app you are creating a headache for another developer, and more importantly you are also limiting the potential adoption of your project. So be a good person and internationalize that bad boy.

Ain’t nothing but a Migrations thing

We have to start including South migrations in all our reusable apps we publish. Or we need my pony request to be fulfilled (discussed below). I've pitched this pony request once before in my South and Reusable Apps post but I wanted to reiterate the importance of the community selecting a migration tool.

As we discussed on the Reusable Apps in Django Panel on DjangoDose the current best-in-show migration tool is South. There is currently no easy way to migrate a collection of reusable apps since migration management isn't a discipline I've found practiced in most apps. And again, I'm to blame for this as well. But no more. Moving forward I'm putting my eggs in the South basket.

Now, South could provide a feature that makes this rather easy for us developers. That is the proposition I made in the aforementioned blog post. Andrew Godwin, the author of South, commented that the solution for migrating reusable apps that don't employ South themselves is already in the works in a forthcoming version of South. So all hope is not lost. This feature would allow us developers to generate South migrations for the reusable apps we leverage even if they don't make use of South themselves. Terrific!

Right now Mingus provides only one migration (raw sql) and that's because it wasn't until recently that people started using Mingus as their blog engine. And knowing this it would be negligent of me to not provide migrations for these users looking to upgrade to the next Mingus release. So at least they have raw sql to work with, but it's not the right answer. The right answer is a standard migration tool we all use.

Cache Keys Rule Everything Around Me

The Django cache framework provides a tremendous amount of caching functionality and flexibility. The one thing I often hear developers reiterating is to make sure your cache keys are named properly so to avoid cache key conflicts. You want unique cache keys that can be recalled easily for cache validation/invalidation, querying, etc.

So why not include a helper to ease this? That exactly what I did when I added create_cache_key to django-sugar. The method actually combines the code of one blog post and a reusable app:

Here's a look at the api:

from blog.models import Post
slug_val = 'some-slug'
mykey = create_cache_key(Post, 'slug', slug_val)
obj = cache.get(mykey)

What the above create_cache_key does is accept either a Model or Manager as its first argument, the field you are interested in as its 2nd argument, and the field value as its 3rd argument. Based on that it can generate, and regenerate a cache key. The benefit here is that it isolates the logic for remembering cache key names. It handles the construction for you.

This may not be the best solution possible or the most complete solution, but it's a solution that begs the question: why don't we have a similar utility method in Django itself that we reference by default so we don't have to be concerned about clashing cache key values in our apps?

Conclusion

The reason I love hacking on Mingus is Django... I love Django, and Python. So the above "hates" are really just small bumps in the road of an amazingly smooth ride that Django provides.

The future of Mingus is a final 1.0 release which will include any bug fixes that pop up, more documentation, more tests, and other than that I don't think there's much left to add to something that's not really anything more than a concept project. I believe the current feature set is final.

For those who manage any open source project, big or small, I tip my hat to you and thank you. Just like you I'm excited for all the amazing things coming in Django 1.2 and as a consumer of such a terrific open source project, I feel lucky that I get to work with Django daily. In ending this I just realized I was "that guy" for most of this article and I hate myself for it.

[1]	Even if it's an itsy bitsy one.

[2]	Since my docs reference these, I assume that by following the docs you forced yourself to play with these excellent tools.

[3]	Eating your own dog food.

[4]	At this point it's arguably no longer reusable, or actually maybe more reusable now that I think about it.

Smoothing The Curve

2010-02-11T00:00:00-11:00

For nearly all of the tools provided by Django, there are hooks provided to allow for incremental modification and extension. Very rarely do you ever need to rewrite a significant amount of functionality within Django, simply to change the behavior of a tool. For example, if you want change the way a form will look, you can change a widget from the default, create a custom field, or just use your own HTML. With each option, you can override a bit more functionality but still take advantage of everything else the forms library provides.

You can visualize these incremental options as a curve. At one end you have a stock form which is simple to write but provides limited control. At the other end, you have a custom form class with static HTML which provides much more control but is more complicated to create. In the case of forms this is a pretty smooth curve since there are options to incrementally replace all of the functionality of the library.

Before Django 1.2 the ORM had a similar curve with one exception — it had a bit of a cliff at the end. In particular, if you got to the point where you needed to just write a custom SQL query you needed to go completely outside the ORM. While this wasn't horrible, there was functionality which one would still want from the ORM which one had to now rebuild on their own. With Django 1.2, a Model.objects.raw() method has been added to address this problem and smooth out the ORM's curve.

The Old Way

Before Django 1.2 if you needed to do a raw SQL query, you needed to do something like:

from django.db import connection
from library.models import Author

cursor = connection.cursor()
query = "SELECT * FROM library_author"
cursor.execute(query)
results = cursor.fetchall()

authors = []
for result in results:
    author = Author(*result)
    authors.append(author)

Now this isn't horrible, but we have lost access to functionality in the ORM beyond the ability to generate SQL queries. In particular, we've lost the automatic transformation of the results of our query into model instances. There are ways we could replicate the lost functionality without too much work but we'd be reinventing the wheel.

The New Way

In Django 1.2 to perform a raw SQL query, you can do something like:

from library.models import Author

query = "SELECT * FROM library_author"
authors = Author.objects.raw(query)

The result here, authors, is a RawQuerySet. RawQuerySet is much like QuerySet in that it is an iterable object which returns a model instance from the result set with each iteration. It is not like a QuerySet in that it cannot be chained. Since the query isn't being built programatically anymore, chaining doesn't make sense.

Similar to a raw database cursor, we can provide a collection of parameters to the raw method and Django will safely quote/escape them.

query = "SELECT * FROM library_author WHERE first_name = %s"
params = ('bob',)
authors = Author.objects.raw(query, params)

So this is great! We're not reinventing the wheel, we're protected from SQL injection attacks, and we have the model instances we want.

“But wait, there's more!”

Like most things in Django, the raw() method offers some additional functionality to help with corner cases or particularly complex queries:

Field order independence

Model.objects.raw() doesn't care what order fields are returned in by the query, all that matters is that the query field names match up to a field on the model.

# All of these queries will work the same
Author.objects.raw("SELECT * FROM library_author")
Author.objects.raw("SELECT id, first_name, last_name FROM library_author")
Author.objects.raw("SELECT last_name, id, first_name FROM library_author")

Annotations

If a query returns any fields which do not exist in the model class, they are added as annotations to the model instances returned by the RawQueryset. This allows you to easily take advantage of operations or calculations which are more efficient to perform within the database.¹

>>> authors = Author.objects.raw("SELECT *, age(birth_date) as age FROM library_authors")
>>> for author in authors:
...     print "%s is %s." % (author.first_name, author.age)
John is 37.
Jane is 42.
...

Field Mappings

If for whatever reason, your query field names cannot exactly match your model field names, Model.objects.raw() provides a facility for mapping query fields to model fields.²

To map query fields to model fields, one simply needs to pass a dictionary containing the translations to the raw() method. Only fields which don't match model fields need to have translations provided.

field_map = {'first': 'first_name', 'last': 'last_name'}
query = 'SELECT id, first_name AS first, last_name as last FROM library_author'
authors = Author.objects.raw(query, translations=field_map)

Deferred Fields

Any fields which are expected by the model, but are not returned by the query, are marked as deferred. Deferred fields are only fetched when the model instance's field is accessed. This is useful in cases where you may not be pulling data from the “real” table for the model or when you have very large tables. Be aware that primary keys cannot be deferred and must be returned by all queries. If a query doesn't return a primary key, an InvalidQuery exception will be raised.

Limitations

There are a few limitations placed on what raw() can do. The biggest of which is that raw() will only allow SELECT queries. If any other type of query is attempted via raw(), an InvalidQuery exception will be raised. This is done partially because it doesn't make sense to return model instances for anything other than SELECT queries but it is primarily done as a deterrent. Modifying data with raw SQL is very much something which should be an absolute last resort in Django. Accordingly we didn't want to encourage the practice by making it any easier to do so. If you really need to perform raw SQL queries which are not SELECT queries, you can still get a raw database cursor and go from here.

That's all folks

There you have it. Now in Django 1.2, you can much more easily perform raw SQL queries when you need to. The curve has been smoothed. Official documentation for this new feature can be found on the raw SQL page.³

[1]	Example heavily stolen from the django docs

[2] It's worth noting here that when the term "model fields" is used, it means the database field name that the Django ORM is expecting to exist in the database, not necessarily the name of the python attribute on the model class. If you've overridden a field name using db_column, the override name is what the raw() method will be expecting.

[3]	Thanks to Jacob Kaplan-Moss for finishing the `raw()` work where I left it off and to Russell Keith-Magee for contributing the code to handle deferred fields.

Messages for the rest of us

2010-02-10T00:00:00-11:00

In today's world of web applications, it is frequently necessary to send notifications to visitors. From “Your comment has been received, and is awaiting moderation,” to “Thanks for your interest, we'll send you an invite when we launch,” these little messages show up all the time, and it’s nice to have a consistent interface to sending and displaying them for your users.

Django's bundled authentication and authorization app (django.contrib.auth) has always included basic functionality to send simple messages to users, but it had some pretty serious knocks against it. Django 1.2 now includes a brand new messaging framework, written primarily by Tobias McNulty, that makes sending these types of notifications painless and flexible.

What was wrong with the old way?

Because the legacy messaging system is part of django.contrib.auth, you must install that app in order to use it. But there are plenty of instances in which you may not need authentication and authorization, but you do need messaging. Or perhaps you prefer to use your own auth app, instead of the one included with Django. Or, you want to use django.contrib.auth but don't want to use messages. Django 1.2's messaging framework allows for these configurations.

Additionally, the legacy system stores and retrieves these messages from the database. While this is fine for many projects, it does mean that most every page view incurs an additional database hit — which is simply unnecessary, in most cases. Django 1.2 provides options for storing messages outside the database to avoid this performance hit.

What's more, in previous versions of Django all messages are treated equally. There is no way to indicate a “type” or “level” for a message — for example, to differentiate between an error message and a success message. The new messaging framework provides this feature.

Finally, Django 1.1's messaging system associates each message with a particular user. Therefore, you are limited to sending messages to logged-in users. In Django 1.2, you're able to send messages to visitors regardless of whether or not they're signed in.

Getting started with the new messaging framework

Messaging is enabled by default for new projects created with django-admin.py startproject. So if you're just starting your project with Django 1.2, messaging is already enabled! If you've got an existing project, or create your projects another way, enabling messaging functionality is very simple. Just complete these three steps:

Add ‘django.contrib.messages.middleware.MessageMiddleware’ to MIDDLEWARE_CLASSES in your settings file.
Add ‘django.contrib.messages.context_processors.messages’ to TEMPLATE_CONTEXT_PROCESSORS in your settings file.
Add ‘django.contrib.messages’ to INSTALLED_APPS in your settings file.

Sending messages

Sending messages to visitors with the new framework is also simple, especially if you want to use one of Django's five built-in message levels (DEBUG, INFO, SUCCESS, WARNING, and ERROR. More on message level later).

To send a success message to a visitor's request, simply call:

from django.contrib import messages
messages.success(request, "Skadoosh! You've updated your profile!")

Likewise for the other default message levels:

messages.info(request, 'Yo! There are new comments on your photo!')
messages.error(request, 'Doh! Something went wrong.')
messages.debug(request, 'Bam! %s objects were modified.' % modified_count)
messages.warning(request, 'Uh-oh. Your account expires in %s days.' % expiration_days)

Since messages are added to the request, you'll need to have access to a request object (as you do in every Django view).

If you're upgrading code which used the legacy messaging system, there are two steps:

In each view that sends messages, add from django.contrib import messages to the top of the file.
Replace each instance of request.user.message_set.create(message=message) with the new API, such as message.error(request, message).

Displaying messages

To display messages in a template, use code along these lines:

{% if messages %}
    <ul class="messages">
        {% for message in messages %}
            <li{% if message.tags %} class="{{ message.tags }}"{% endif %}>{{ message }}</li>
        {% endfor %}
    </ul>
{% endif %}

It usually makes sense to put this message template code in your base template, so all templates that extend it will display messages.

The code is nearly identical to what you may have been using with Django 1.1, but you'll note the new tags attribute for each message. Django 1.2 associates each message level with a string representation for use in your HTML template¹. Here, we output the tags as classes, which can then be used for individual styling of different message levels:

.messages li.error { background-color: red; }
.messages li.success { background-color: green; }

If all you want to do is send some basic messages, you don't have to read any farther. That's really all there is to it. It’s simple and very flexible.

The message storage engine

Django provides a handful of backend storage systems for messages — and it’s also easy to write your own. The LegacyFallbackStorage engine Django uses as the default is sensible and appropriate for most projects. However, there are reasons why you may want to switch engines, so it’s good to understand the options:

django.contrib.messages.storage.session.SessionStorage: This engine stores all messages in the request's session. Therefore, it requires the django.contrib.sessions app be installed (since it’s enabled by default, most projects are already using it). By default, sessions are stored in the database, so using this storage option does require a database hit each time you check for messages in a template (i.e. {% if messages %}).
django.contrib.messages.storage.cookie.CookieStorage: This engine stores all messages in a cookie. Therefore, it does not require a hit to the database, providing better performance than SessionStorage. However, cookies can only be 4096 bytes in length, so when a user's message exceed 4096 bytes, messages will not be delivered.
django.contrib.messages.storage.fallback.FallbackStorage: This engine first uses CookieStorage, but falls back to SessionStorage in the event all message could not fit in a single cookie.
django.contrib.messages.storage.user_messages.LegacyFallbackStorage: This engine is provided for backwards compatibility with Django 1.1 and earlier. It works exactly like FallbackStorage, but also retrieves messages from the legacy messaging system in django.contrib.auth. This provides compatibility with any applications that haven't yet been updated to use the new messages framework. Like the messaging system in django.contrib.auth, this engine is deprecated, and will be removed in Django 1.4. In the meantime, it’s the default storage method for the new messages framework. When it’s removed in 1.4, FallbackStorage will become the new default.

Message levels and tags

As noted earlier, Django includes five common message levels by default. Each level is an integer. You can easily extend or modify these for your own needs. The default message levels map to the following integers:

DEBUG: 10
INFO: 20
SUCCESS: 25
WARNING: 30
ERROR: 40

To add your own message level, define a constant and then use the add_message() method to send a message using it:

CRITICAL = 50
messages.add_message(request, CRITICAL, 'OH NOES! A critical error occurred.')

If you want to use the message level in HTML or CSS, as we did above, you'll also want to add MESSAGE_TAGS to your settings file to provide a mapping between your levels and the tag you want outputted in your templates:

MESSAGE_TAGS: { 50: 'critical' }

In conclusion

The new messaging framework included in Django 1.2 is not a particularly complex piece, but it does provide functionality that is absolutely essential in today's word of web apps, and it does so in an elegant, clean, and simple way. What's more, it’s completely backwards-compatible with the legacy system, so you don't have to worry about old code or third-party apps breaking when you upgrade to 1.2. Be aware, though, that this compatibility layer will be removed in 1.4 — slated for release sometime in 2027.²

Joking. Enjoy Django 1.2, everyone. It really is a great update to our favorite framework.

[1]	Ostensibly for CSS and JavaScript hooks.

[2]	Late 2027, if I had to guess.

Multiple Database Support

2010-02-09T00:00:00-11:00

Since Django's creation there's been an implicit restriction to a single database (this is as systemic as things like the DATABASE_* family of settings), and for almost as long support for multiple databases has been requested. This summer, as a part of the Google Summer of Code, multiple database support was implemented, and as a part of the 1.2 process it was merged into trunk. These changes involve a ton of changes to the internals, and a few well placed extensions to the existing public API.

The Multi-DB Public API

The most obvious change to Django is that instead of defining your database settings like:

DATABASE_ENGINE = "postgresql_psycopg2"
DATABASE_NAME = "my_big_project"
DATABASE_USER = "mario"
DATABASE_PASSWORD = "princess_peach"

You instead write something like:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
        "NAME": "my_big_project",
        "USER": "mario",
        "PASSWORD": "princess_peach",
    },
    "credentials": {
        "ENGINE": "django.db.backends.oracle",
        "NAME": "users",
    }
}

All projects will eventually need to be migrated these new settings, although the old style will continue to work until Django 1.4. The only key with special significance in the DATABASES dictionary is "default" — if you're using databases you need to define a "default" database.

Now that you've told Django about all your databases you need a way to tell Django when to use them. The first addition in this arena is the using() method on QuerySets. using() takes a single parameter, a database alias (alises are the keys of the DATABASES dictionary), and it binds the QuerySet to that database. Like every other QuerySet method it can be chained at will (like the order_by() method calling it a second time it overides the first call). This basically gives you complete control to define where models are read from (and if you get tricky with create(), delete() and update() even the ability to control writes):

>>> User.objects.filter(username__startswith="admin").using("credentials")

In addition to this new QuerySet method delete() and save() on models takes a new using parameter, which is, again, a database alias.

There is also a new method on Managers, db_manager(), this again takes a database alias and what it does is very similar to using(). The key difference is that instead of returning a QuerySet it returns a new Manager. The use case for this is being able to chain it with methods on Managers that don't return a QuerySet, such as create_user() on the UserManager.

Database Routers

By using all of these methods you can implement any sort of multiple database system you want, master-slave replication, partitioning, sharding, or anything else. However, it wouldn't necessarily be convenient. You'd have to litter your codebase with calls to using(), and that doesn't play nicely with Django's reusable application philosophy (it was for this reason that a using option was removed from the Meta class in Django models). In addition there are some places where you don't have an explicit QuerySet, for example my_obj.user will create a QuerySet behind the scenes to access the User model, but you don't have any place to call using() there. For these reason the concept of a "database router" was introduced. Database routers take whatever is available about the query you want to perform, and they can say what database it should go to. Database routers are setup in your settings file with a new DATABASE_ROUTERS setting, which is a list of database routers:

DATABASE_ROUTERS = [
    "path.to.AuthRouter",
    "path.to.MasterSlaveRouter",
]

DATABASE_ROUTERS is a list because at any stage a router can return None and then Django will fall back to the next router in the list. These routers can define a few different methods:

db_for_read
db_for_write
allow_relation
allow_syncdb

The first two are fairly self-explanitory, they return the alias that that query should be performed against. The allow_relation method exists to provide a sanity check. Django doesn't want to let you assign cross-database relations if it's going to fail. Therefore when you're trying to create a relationship between two models on different databases (for either a ForeignKey or ManyToManyField) Django will call this method so that you can provide the appropriate validation. The final method, allow_syncdb, provides a way for you to let Django know which models should be sync'd (and therefore available) on which database.

As always the Django documentation on multiple databases provides great examples of how to use all these things, including examples of how they're used, and how to get started implementing some common patterns with database routers. The addition of multiple databases should provide a tremendous boon for the Django community, allowing Django to be used in yet more enviroments.

Welcome to Django-Advent

2010-02-08T00:00:00-11:00

I'm incredibly excited about the release of Django 1.2. Looking back, there've been a few moments that turned out to be great inflection points for Django as a project and for our community. The landing of the "magic-removal" branch and the subsequent 0.96 release was the first of these points: we released a vastly improved Django, and our community began to grow by leaps and bounds. And of course the release of Django 1.0 was another such moment: the maturity of Django, and the "1.0" label proudly proclaiming our confidence, really propelled Django forward.

I'm fairly certain that 1.2 will be another such inflection point.

The new features — multiple database support, model validation, vastly improved CSRF protection, improvements to the admin UI — are some of the most hotly anticipated in Django's history. These features will let our users take Django to new levels, and I can't wait to see what y'all will be able to pull off.

Over the next few weeks you'll be treated to articles discussing these new features in depth, many written by the implementors of these features. So as to not steal their thunder, I'll try to look at the big picture of how they all fit together, and how they'll lay the groundwork for Django 1.3, 1.4, and beyond.

Multiple database support is shaping up to be Django 1.2's killer feature. It's been an interesting challenge adding features of this nature: there's a natural tension between the needs of new and small-scale developers — who value simplicity and ease of use above all else — and the increasing number of large-scale Django developers — who need much more control, and hence more complexity.

I think this tension between simplicity and configurability, between ease-of-use and scale, will continue to come up. As Django matures, users will use it to tackle tougher and tougher tasks, and we'll want Django to scale to the challenge. At the same time, though, there'll always be many users new to Django, and if we swamp them with complexity they'll look elsewhere, and our community will stagnate.

In Django's new multiple database support I see that we've walked the line quite closely. I'm proud of the job we've done balancing these competing needs: getting started with Django 1.2 is as easy as ever, but we've vastly expanded the "upper end" of where you'll be able to take Django. It's my hope that we'll be able to strike similar best-of-both-worlds balances.

Beyond features, though, I'm most excited by the changes I'm seeing in our development community. Over the Django 1.2 release cycle we've seen a lot of new blood in our development community. Even better, as existing contributors' time has waned, new developers have stepped up, and work's still getting done.

Much of this new blood came to use via Google's Summer of Code 2009. The year we accepted six projects:

Multiple database support (Alex Gaynor).

HTTP and WSGI improvements (Chris Cahoon).

Improved localization features (Marc Garcia).

Model validation (Honza Král).

Test framework additions (Kevin Kubasic).

Admin UI improvements (Zain Memon).

The program was a smashing success: all of the projects produced good, working code. Some has found its way into Django 1.2, and some will continue to trickle back into trunk over the 1.3 release cycle. Many of these projects were inspired by long-wished-for features, and the work done during the Summer of Code ended up being what we needed to get these features over the finish line.

We've also added a couple of new committers — Jannis Leidel and James Tauber -- and I'd wager that we'll see a few more folks get the commit bit before the final release.

Speaking of that final release — what of it?

The planned release date is March 9, 2010. We're on track for an on-time release... but we'll never make it without your help!

Every release of Django is truly the work of thousands. We need users who can try out our beta releases in test environments and report back to us. If you find bugs, you can report them on our ticket tracker.

Even better, we'd love to have your help fixing bugs. Contributions on any level — developing code, writing documentation or simply triaging tickets and helping to test proposed bugfixes — are always welcome and greatly appreciated.

Django's online documentation includes pointers on how to contribute to Django

We'll also be holding development sprints for Django 1.2 at PyCon US 2010. We'll hold four days of sprints (February 22 - 25), and we'd love to have you join us in person in Atlanta, or virtually in IRC (#django-dev on irc.freenode.net) or on our mailing list

I hope you'll enjoy the rest of the series, and I hope you'll end up as excited about the release of Django 1.2 as I am!

Django Advent Articles

Small Things

Size doesn't matter

Email backends

Read-only admin fields

Smarter if tag

CSS classes on required/erroneous form rows

Cached template loading

Et cetera

I18N/L10N improvements

Formats

Custom formats

Available formats

Formatting

Handling input

Smaller bugfixes worth noting

The makemessages command

--ignore

--symlinks

Plural forms

Order of precedence of translation catalogs

Template filter in {% trans %}

Support for IDN (Internationalized Domain Names)

Current locale in cache

First day of week

Internationalization/localization documentation

Syndication Gets Classy

Prior to Django 1.2

Class-based Views

Using the New Syndication Framework

Improved Bash Completion

Installation

Usage

Tip: Enable single-tab completion

End

Django 1.2 and CSRF

Anatomy of a CSRF Attack

Protecting against CSRF

Django and CSRF

What's Changed?

Gotchas

Conclusion

Django Pony: A Retrospective

Scaling Django

Scalability

Gathering Stats

Caching

Load Balancing

Queuing

The Database

NoSQL

The Future

The End

Django Testing Improvements

Multiple Databases

What Changed

Django Developers

App Developers

Fail fast and fail early

What Changed

Django Developers

App Developers

Keepin’ it classy

What Changed

An example

The early Djangonaut gets the Pony

Thanks for reading

jQuery in the Admin

The case for jQuery

This doesn’t mean Django endorses jQuery

This does mean Django’s admin will grow much faster

New features

If Gets Smart

So what's new, Mr If?

If not Python

Mixing boolean logic

Missing variables

Digging deep

"Too much time?" challenge

Just because you can...

The `makemessages` command

`--ignore`

`--symlinks`

Template filter in `{% trans %}`

init