Surfing in Kansas feed

Large Problems in Django, Mostly Solved: Database Migrations

Fri, 06 Nov 2009 22:03:11 +0000

Continuing in the series of big problems that are mostly solved, we have database migrations. A couple days ago I talked about Search.

Database Migrations

Database Migrations are an interesting piece of the Django community. Rails has the functionality built in, but Django currently relies on third party apps for this functionality. One of the core philosophies about not including apps in the Django core is that ideas percolate better in the fast release environment outside of the core. When something goes into core, it is automatically seen as blessed, and will certainly become the defacto answer to a problem. Leaving things outside allows multiple different implementations to develop (as they did), and for one to become the standard (which it has). Along the way it has picked up ideas from others, and now provides a good answer to migrations.

South

South has emerged as the obvious choice for database migrations in the Django community. We use it in production at work at the Journal World, and it has served us well.

I have talked about south in the past, using it to migrate test fixtures. This serves as a basic tutorial and introduction into south as well.

Main Features

Automatic Migrations

Most of the migrations that I write, I don't write a single line of code. South has the ability to how you model looked at the end of your last migration, and then extrapolate what has changed (in most simple and modestly complex cases). There are obviously times that it falls down, but for simple addition, deletion, and modification of fields it has worked almost flawlessly for me. With a simple command, it will do all your work for you.

django-admin.py migrate app_name --auto

It has problems with Generic Foreign Keys and a couple of other more complex models. However, I would say that it absolutely nails the 80% case that most migrations fall in to.

Fake ORM ("ORM Freezing")

This is a feature that South has grown from it's Migratory roots. I think it is one of the best conceptual features for migrations. It allows you to use a Fake ORM (the real ORM, applied to the aforementioned fake models), to do data transformation in your migrations. This example from the tutorial shows the value:

def forwards(self, orm):
    for adopter in orm.Adopter.objects.all():
        try:
            adopter.first_name, adopter.last_name = adopter.name.split(" ", 1)
        except ValueError:
            adopter.first_name, adopter.last_name = adopter.name, ""
        adopter.save()

Database Independent

This sounds like an obvious feature, but a lot of the approaches for migrations were only viable on one database. The support for SQLite is still lacking, but that is because of fundamental limitations in the way SQLite works. Most people using SQLite can just wipe their database and start over, if not, you should probably be using another database.

It knows when you've been naughty

South keeps track of all the migrations that you have run, and it intelligently informs you if you have missed a migration. It also supports inter-dependencies on migrations. This allows you to be safe in your knowledge that your migrations will be run properly, and that state is maintained. This sounds like a hand-wavey feature, but when you're migrating your data, knowing when things aren't quite right is a nice feeling!

South also keeps track of the migrations that are on disk, and won't let you migrate if they are different than previous runs. This makes sure that you aren't running against a different version of the code; allowing you to be sure that the migrations being run are correct.

Conclusion

Overall, south solves a lot of the problems about migrations in a good way. There have been other solutions to the migration problem, and I think that south has taken most of the good ideas and combined them in one place. It has some drawbacks still, but overall it is the best of breed in Django for Database Migrations. If you are looking for a migration tool for Django, this is your best bet.

There aren't a lot of flashy features in the migration realm I feel. Mostly you just want something that keeps your data safe, and allows you to write migrations as simply and foolproof as possible. South lets you do that, so I consider it a win.

I view migrations somewhere along the lines of testing. It is one of those things that once you have, you don't see how you ever lived without it. Being able to immediately see the state of your database, what migrations haven't been run, and what all needs to happen is incredibly useful. Having a safety net of repeatable migrations also ensures that your databases are all the same, across many installations and machines. The value of database migrations are many, and South brings them to you in a nice package.

Adding testing to pip

Thu, 05 Nov 2009 18:22:38 +0000

Python packaging has been in a bit of a state of disarray for as long as I've been using it. Pip has come along to make installing python packages easier. It has a lot of features that are useful, but they have been talked about in many other blog posts.

Today I want to talk about adding testing to pip. If you are familiar with the Perl community, then you probably know about CPAN. It is basically Pypi for Perl. They have a command, called cpan, which allows you to install packages in a similar way to pip.

One of the steps that a package goes through before being installed on your system is that the tests are run. This allows you to know if the package that you have installed is actually going to work on your system. It may be broken on your platform, or you may be missing a library that it thought you had. Currently, pip has no way to test packages when they are being installed. I went looking for a way to make that happen.

It should be noted that pip is based on setuptools. Setuptools is what parses and understands most of the logic inside of your setup() function in the setup.py for your project (which you have, right?). Setuptools has an option called test_suite, which allows you to run setup.py test on your package, and have it run the unit tests. This is done by calling whatever python function is defined in test_suite.

I added the ability for pip to run setup.py test on a package that it is installing. It is executed by running pip install --test <package>. The implementation is on a ticket on bitbucket, and in a repository on github.

If you want to check it out, go ahead and clone my repo and check out the test_command branch. Then you can simply run

python pip.py install --test wsgiref

for an example package. If a package doesn't have a test_suite, then it simply doesn't run anything.

Note that if the tests fail, it doesn't impact the installation of the package. The python community's tests aren't quite good enough,and almost any Django package you try this on will not have any tests. I wrote about how to add testing to your django package, but the process is long and involved. I'm working to improve the situation for Django and hopefully having the ability to run tests in the package management tool will spur people to add testing ability to their setup scripts!

Nose makes this really easy, by simply adding test_suite = 'nose.collector' to your setup.py, nose will run your tests correctly. This is the level of support that I am hoping to implement for Django.

On a side node, I talked to Ian Bicking about this, and he suggested writing the test command as a separate command, so you would be able to do pip test wsgiref, if it was installed. This has some other problems, which I will talk about after I have implemented this functionality.

I would love to hear feedback, or if anyone has ideas for improving testing in the python and django communities. I have lots of ideas, and I will be writing more of them up over the following weeks.

Making Template Tag Parsing Easier

Tue, 03 Nov 2009 23:55:35 +0000

In my previous post about template tags, I discussed the two steps required for template tags. Today I will be focusing on Parsing of template tags, and how they may be improved in the framework of Class Based Template Tags from yesterday. I have talked about problems with template parsing in the past as well. This post will offer 2 different approaches to making parsing better.

I would like to thank Cody and Chris who were involved in a slightly drunken conversation that led to these tags. Chris actually wrote the other neat parsing implementation that I will talk about today. Cody wrote the underpinnings of that implementation as well.

Note: Both of these approaches are more Proof of Concepts, and the code probably shows. Please don't knock implementations, and just think about the ideas housed within.

Parsing from above - A DSL approach

I'm going to go ahead and start talking about an approach to parsing template tags that was pointed out in yesterday's comments. It takes surlex which is made for easily parsing URL's, and applies it to the concept of parsing template tags.

In the tag_utils package, I looked at the tests, because they make great documentation. Here is an example of a tag definition.

p = ParsedNode('test', '<arg1:int> <arg2:string> <kw:kwarg>', test_expected) 
register.tag('test', p)

This defines a tag called test, which parses an int, string, and kwarg from a surlex expression. The third argument is a function that is executed on the arguments on rendering.

This allows you in your test_expected function, to act on the arguments that are defined inside of the surlex expression. A trivial example of the test_expected function is:

def test_expected(context, arg1, arg2, kw=None):
    print "Got %s and %s" % (arg1, arg2)

So if you called the tag {% test 1 racoon %}, it would print out Got 1 and racoon.

This is an interesting way to provide a sort of DSL on top of the current mess that is parsing of template tags. I really like how it reuses Surlex, which was made for parsing URLs. However, parsing template tags is a similar task, and it works well here too!

I could imagine this easily being bolted on to the approach from yesterday, which might allow for easier subclassing and reuse of the parsing functions.

Parsing based on keywords

An approach that I have talked about in the past is basically a subset of the above idea. It allows you to define kwarg type arguments for your tags, and have them magically parsed out for you. An example of this is my own SelfParsingTag. The following lines allow you to specify what arguments your tag will accept.

    def __init__(self, required_tags=[]):
        if not required_tags:
            self.required_tags = self._get_tags()
        else:
            self.required_tags = required_tags

    def _get_tags(self):
        return []

So you can either define the _get_tags function, or pass the allowed tags into the call when you make the tag. The following 2 bits of code are equivalent.

class GetContentTag(SelfParsingNode):
    def _get_tags(self):
        return ['as', 'for', 'limit']
register.tag('get_latest_content', GetContentTag())

#Is the same as the following:

class GetContentTag(SelfParsingNode):
    pass
register.tag('get_latest_content', GetContentTag(['as', 'for', 'limit']))

Once the Tag knows what it arguments it will be accepting, it parses them.

def parse_content(self, parser, token):
    parsed = parse_ttag(token, self.required_tags)
    for tag, val in parsed.items():
        setattr(self, '_' + tag, val)
    return parsed

This effectively sets a private varible on the tag to the value of the arg. So for example, if the tag was called {% sweet_tag for news.story as my_stories limit 10 %}, then self._for would equal news.story, and so on. It also returns the parsed values as a dictionary. There are a lot of improvements that could be made to parse_ttag, but it works as a basic implementation.

This approach allows us to implement a tag really easily. If you want a (silly) tag that just updated the context with whatever value you input, you could make a simple tag. It would be used {% my_tag with "awesome text" as context_var %}

class SimpleContextTag(SelfParsingTag):
    def _get_tags(self):
        return ['with', 'as']

    def render_content(self, tags, context):
        for tag in self.required_tags:
            context.update({tag['as']: tags['with']})

register.tag('my_tag', SimpleContextTag())

To implement the get_latest_object code from yesterday, we can skip all of the parsing steps.

class GetContentTag(SelfParsingTag):
    def _get_tags(self):
        return ['as', 'for', 'limit']

    def render_content(self, context):
        self.model = get_model(*self._for.split('.'))
        if self.model is None:
            raise template.TemplateSyntaxError("Generic content tag got invalid model: %s" % model)
        query_set = self.model._default_manager.all()
        context[self._as] = list(query_set[:self._limit])

register.tag('get_latest_object', GetContentTag())

Which is better?

To be truthful, I like the Surlex approach better than my own. It seems to have a lot of the benefits of mine, but with added flexibility. However, that does come with the implementation being a bit more complex. It brings some really neat ideas forward about how template tags might be handled differently. It allows for optional arguments, does basic type checking (based on it's regex nature), and ensures that the order of the arguments is the same.

I could imagine some kind of dispatch based template tag scheme that has a list of URLs, basically like the URLConf and view structure. I think that this problem has a lot more depth to it, and hopefully by pointing out a couple of different ways of solving it, and looking at it, we can improve the situation.

Class Based Template Tags

Tue, 03 Nov 2009 18:32:07 +0000

The problem

In Django, template tags currently are separated between a Node class and a "parsing function". The parsing function takes the tag, represented as a string, parses the input, and passes the correct arguments to a Node class. The Node class then does whatever rendering it does, or updating of the context, and then renders itself in a form suitable for the template.

This is mainly by convention that there is a separation here between the parsing and the Node. As I see it, there is no particular reason that the Tag can't be responsible for the parsing and rendering itself. A lot of the time I find the parsing function and the Node separated by hundreds of lines in a file, making it hard to understand.

The proposed solution

We can combine the parsing and rendering of a node in a similar way in something I call Class Based Template Tags. This allows the template tag to be able to parse and render itself.

I have an example in my playground over at github. They are based around a lot of the ideas in django-template-utils. Specifically, this example will be recreating the get_latest_objects tag from that package.

class ClassBasedTag(template.Node):
    """
    Tag that combined parsing and rendering

    Subclasses should define ``render_content()`` and ``parse_content()``.
    """

    def __call__(self, parser, token):
        self.token = token
        self.parser = parser
        return self

    def render(self, context):
        self.context = context
        self.parsed = self.parse_content(self.parser, self.token)
        return self.render_content(context)

    def parse_content(self, parser, token):
        """
        This is called to parse the incoming context.

        It's return value will be set to self.parsed
        """
        raise NotImplementedError

    def render_content(self, context):
        """
        This is called to return a node to the template.

        It should return set things in the context or return
        whatever representation is appropriate for the template.
        """
        raise NotImplementedError

As you can see, this tag combined the concepts of Parsing and Rendering a tag into the same place. The parse_content and render_content are equivalent to the current Django way of doing a parsing function, and Node class render function. Currently the render function depends on self.parsed being there, and not being passed in, this is to keep the function arguments the same as previous render functions. The code isn't meant to be production quality, more of a proof of concept.

A couple of gains are made from combining things together. First of all is the fact that the code is right next to each other, as mentioned earlier. However, it also allows you to subclass these classes, and provide functionality that makes people's lives easier. Having the rendering and parsing in the same class also allows for some trickery with passing around data, like mentioned, which may be a good or a bad thing.

Let's go ahead and show an example of an implementation of this type of tag.

class GetContentTag(ClassBasedTag):

    def parse_content(self, parser, token):
        bits = token.contents.split()
        return (bits[1], 1, bits[3])

    def render_content(self, context):
        model, pk, varname = self.parsed
        self.pk = template.Variable(pk)
        self.varname = varname
        self.model = get_model(*model.split('.'))
        context[self.varname] = self.model._default_manager.get(pk=self.pk.resolve(context))

register.tag('get_latest_content', GetContentTag())

This tag is used in the following manner:

{% get_latest_content news.story as latest_story %}

As you can see, I think it makes it nice and concise to be able to have the parsing and the rendering of a tag right there in the same place.

This code is a very simplified use case for the idea. It is basically the simplest possible thing that could work. I will expand on the ways that this idea gives us a lot of power and flexibility over our Template Tags in the future, but I think this idea stands well on it's own.

Large Problems in Django, Mostly Solved: Search

Mon, 02 Nov 2009 20:34:04 +0000

It's been a little over a year since I started doing Django development full-time, for one of them real jobs. Around that time, there were a few large problems in the community that hadn't been solved yet. They were kind of blemishes when you would talk to people about Django, and I'm happy that most of them have been solved.

This will be a series of posts that talk about the different big problems that have been solved, and how they have been addressed in the community.

Search

Search was probably the biggest annoyance for Django. It is something that every site needs, and something that just wasn't really happening in the Django community. 2008's Summer Of Code ended with djangosearch as a half-finished shell, that needed to be better architected; but it did show a good push in the realm of search.

Out of those ashes, comes an awesome solution to the search problem in Django. Haystack is something I am more familiar with (we use it in production at work), and is the brain child of the ever modest, house rocking Daniel Lindsley. It provides a number of Django patterns applied to search, which makes it easier to internalize.

Note: Another approach to search is available, which patches django's ORM. This uses the existing full-text search in your database.

Useful Haystack Patterns

Registration

The registration pattern of the admin allows you to unobtrusively make models searchable (including code you don't have access to). This allows you to register Django Comments as searchable for example, without forking the code base. This will look similar to the admin:

from haystack import site
site.register(Note, NoteIndex)

Search Querysets

Haystack provides an interface familiar to the Django ORM Queryset API. This gives you most of the commonly used functions from the ORM, but allowing you to use them on searches!

unfriendly_results = SearchQuerySet().exclude(content='hello').filter(content='world')
unfriendly_results.order_by('-pub_date')[:5]

It also gives you Search specific methods such as boost and facet.

Class Based Views

In 1.2, hopefully generic views will be class based. Haystack has an implementation of these as well. Like any other kind of class, it provides the easy ability to override functionality through subclassing.

This (simplified) example from the source shows how easy it is to provide extra context to a search view.

class FacetedSearchView(SearchView):
    def extra_context(self):
        extra = super(FacetedSearchView, self).extra_context()
        extra['facets'] = self.results.facet_counts()
        return extra

Pluggable Backends

Haystack currently supports three different search backends: Whoosh, Xapian, and Solr. With a publicly documented backend API, you can enjoy all of the power of the search engine of your choice, by providing a backend for it!

HAYSTACK_SEARCH_ENGINE = 'solr'

Documentation

A shining light in the Django world is the documentation. It is often talked about as being the biggest factor for how people learn Django and love it is the documentation. Haystack is another package with fantastic documentation. Here are a couple of little gems that really show the quality and thought that has been put into them:

The docs cover ways to improve your search and make it awesome, as well as just helping you get the software set up and running. The information is invaluable, and will help you make the search on your site really great!

Are you using Haystack?

If I'm preaching to the choir and you already use Haystack, there is a growing list of users that are using haystack. Daniel would love for you to contact him, and get yourself added to the list.

Did I miss anything?

Let me know what you love (or hate) about Haystack. I think it reuses a lot of the good patterns in Django, allowing people to take knowledge they already have, and apply it to a new problem domain easily. Is there anything that you don't like, or something that you love that I missed? Let me know in the comments!

Easily Running the Django Test Suite

Fri, 16 Oct 2009 02:12:10 +0000

Alex Gaynor had a write up about Running the Django Test Suite, which is a quick overview of how to run the suite. The official docs also have a simple mention of how to run them. This post will be more step by step, walking you through the steps to run the tests for Django. This is a really important first step in writing patches against Django. It is easy, but something that a lot of people have a question about when they start.

Step 1: Grab the Django Source

To test Django, you need the code, so go ahead and grab the source.

svn co http://code.djangoproject.com/svn/django/trunk/ django_src

Step 2: Settings

In order to run the Django test suite, you need to have a settings file. Usually for testing, you run the Django test suite under SQLite. This is the easiest and fastest way to run the tests. If you writing code against something that touches parts of the ORM or Database code in general, running it against another database that you have at your disposal if generally a good idea as well.

To run the SQLite tests, you simply need a settings file with one line in it:

DATABASE_ENGINE = 'sqlite3'

Go ahead and put this command in the top-level of your django checkout (the one with the tests directory in it).

Step 3: Run the tests

Now you can run the tests. Your checkout should look something like this:

django_src/
    docs
    django
    examples
    setup.py
    tests
    settings.py
    ...

We need to make sure that Django is on your PYTHONPATH, this allows Python and thus Django to see the django module that we want it to test. You can set this inline, and then run the tests with the correct settings file. We can do that in a single command like so:

 PYTHONPATH=`pwd` ./tests/runtests.py --settings=settings

The final commands, which you should be able to copy and paste into a shell to check out the code and run the tests is as follows:

svn co http://code.djangoproject.com/svn/django/trunk/ django_src
cd django_src
echo "DATABASE_ENGINE = 'sqlite3'" > settings.py
PYTHONPATH=`pwd` ./tests/runtests.py --settings=settings -v1

The -v1 will set the verbosity to 1, which gives you the dots that everyone knows and loves.

Now that you have a django source tree with running (and hopefully passing) tests, you can apply a patch or go ahead and develop on this code and be able to test it easily!

Hacker Book Club

Sat, 26 Sep 2009 13:12:09 +0000

At LPDN, our weekly programmer drinkup, we have been talking for a while about watching the SICP lectures from MIT as a fun thing to do. I then got to thinking about how it would be neat to involve more than just the few of us in Lawrence. Everything is more fun on a larger scale, and having compatriots makes you more likely to finish it. Somewhere along the lines of the Infinite Summer, I was thinking about having some kind of Hacker Book Club.

SICP may be a bit intense for the beginning of the club, but it's something that I think that everyone can learn and have fun with. Along with the typical book club style things, seeing the audience, I think we could make some really neat additions to the idea. A few have come to mind, but I'm sure there are lots more.

I would imagine having an IRC channel where we can all hang out and ask each other questions. I'm sure the #scheme channel will be a resource as well, but it could be neat to have a focused group of people focused on learning the same material. It feels a bit like an open source classroom, where disperate people can come together to learn and share in the open source mindset. Sharing is caring, not cheating.

Since SICP is available for free online, we could suck the book into a database of some sort and allow a commenting/discussion medium around the text as well. The Django Book is a good example of that. I think it would be interesting trying to tie in discussion on the site with IRC. Perhaps have some kind of encoding that allows you to reference pages in the book on IRC and have that inline the log transcripts. My friend Nathan also runs Readernaut, and already has a basic syntax established for tracking book progress on Twitter. We could throw this on IRC and twitter as well, where each user could register their account and keep progress.

I think it would be a really interesting way to combine a lot of the conversations that happens across the web in one place, with context, about a book. I have also wanted similar abilities in other things (Think your work IRC channel and code instead of Book pages). This could spawn a neat open source project, if we were so motivated.

Let me know what you think!

Pretty Django Error Pages

Wed, 23 Sep 2009 18:14:29 +0000

Continuing on with the simple tricks that make everyone's life a little bit better, I know a lot of people hate that Django's 500 pages don't get rendered as a RequestContext. This means that if you have context processors (like one that sets a MEDIA_URL), they don't get called. This was causing our 500 pages not only to make users sad because something broke, but knock them out of context becaue our entire design blew up.

Luckily, Django makes it incredibly simple to redefine your 500 handler in your URLConf. Most pythonistas know that import * is a bad thing, but it is standard in the Django community in your URLConf to do a from django.conf.urls.defaults import *. This has the effect of pulling in Django's default handler500 function. So if you want to override Django's default, you simply set it up like so.

from django.conf.urls.defaults import *
handler500 = 'path.to.my.sweet.views.server_error'

Then you simply define a server_error view that renders the error page with a RequestContext.

from django.shortcuts import render_to_response
from django.template import RequestContext

def server_error(request, template_name='500.html'):
    """
    500 error handler.

    Templates: `500.html`
    Context: None
    """
    return render_to_response(template_name,
        context_instance = RequestContext(request)
    )

If you're feeling extra special, you can even change the template rendered. Note that you can also do this for the 404 handler by defining a 404handler in your URLConf in the same fashion. Then you can get pretty error pages!

Token Testing Talk Slides: Djangocon 2009

Tue, 08 Sep 2009 08:26:49 +0000

There are the slides to my Token Testing Talk from Djangocon. I'm hoping the videos will be posted soon, but I think that it went well. There were a lot of good questions, and I need to put some recap posts up, but for now here is a copy of the slides. PDF

If you have any questions or comments, feel free to leave them below.

Token Testing Slides

View more documents from ericholscher.

Debugging Django in Production Revisited

Sat, 05 Sep 2009 10:04:43 +0000

In a previous post I talked about a neat middleware to debug production environments in Django. It basically checked to see if you were a superuser, or if you were in settings.INTERNAL_IPS, and if so, then it displayed a technical 500 page for you (The yellow one you know and love). Anyway, at that point it was more of a simple idea, and not really used in production.

At work the other day I was working on a bug that was only showing up in production, and not on staging. I remember back to this middleware and thought it would be perfect. Anyway, at work we have a lot of non-technical people that are superusers (think my bosses boss). We also all have the same external IP's when at work, so none of the previous methods I had would work for this.

Thinking about it, and talking to my co-worker Ben Spaulding, we thought that Django has Groups built in, so why not use that? So I went ahead and re-jiggered the middleware to be based around groups.

from django.views.debug import technical_500_response
from django.contrib.auth.models import Group
from django.core.cache import cache
import sys

class UserBasedExceptionMiddleware(object):
    def process_exception(self, request, exception):
        users = cache.get('technical_error_users')
        if not users:
            skip = cache.get('no_technical_error_users')
            if skip:
                return None
            try:
                g = Group.objects.get(name='Technical Errors')
                users = g.user_set.all()
                cache.set('technical_error_users', users, 60)
            except Group.DoesNotExist:
                cache.set('no_technical_error_users', True, 60*60)
                return None
        if request.user in users and request.user.is_superuser:
            return technical_500_response(request, *sys.exc_info())
        return None

Since it is middleware, I went ahead and decided to use the cache framework to make sure that we weren't doing a DB query on every request. Also, I had to account for the case when the group hasn't been added yet, so when that happens, it caches the fact and doesn't check again for another hour. If the Technical Errors group exists, it caches the members that are in it for a minute. This means that a DB query only happens every minute, which is fine.

I'd be curious how other people might improve this, as it seems a little bit janky still. However, it works for us, and is incredibly useful when debugging. Instead of getting a link to a broken page, you go to the page and get a nice 500, telling you exactly what went wrong.

I can think of one basic improvement in just writing this post, which would be to import settings and to just return None if DEBUG was True, or if the CACHE_BACKEND was set to None. This would allow it to stay out of the way if there was no caching, or the Technical 500 was already going to be raised.

I do think that this middleware removes a lot of the reason to run a site under DEBUG=True, so hopefully it will result in less sites launching with DEBUG on.