Presskit'n Hits

Users of Presskit’n have been asking for traffic statistics on their press releases so I decided I would get them the most recent data possible. At first I was parsing the access log once a minute and when I was testing that I decided it wasn’t updating fast enough. I’ve gotten used to everything being instant on the internet and I didn’t want to wait a minute to see how many more views there were. In this post I show how I got it to update on page load using Apache, python, Django, and memcached.

Apache Access Logs

Apache is installed with rotatelogs. This program can be used to rotate the logs after they get too large. However I wanted a few more features. Cronolog will update a symlink everytime it creates a new log file so that you can always have the most recent stats.

CustomLog "|/usr/bin/cronolog --symlink=/path/to/access /path/to/%Y/%m/%d/access.log" combined
ErrorLog "|/usr/bin/cronolog --symlink=/path/to/error /path/to/%Y/%m/%d/error.log"

CustomLog and ErrorLog directives in apache will let you pipe output to a command. So I put the full path to cronolog and then specified the parameters to cronolog. –symlink will point the named symlink to the most recent log created with cronolog. After the options, the path to the log location is specified and date formats can be used. I decided to break mine up by day.

Piping Apache Log info to a Python Script

Apache can have multiple log locations and log multiple times. So I wrote my own logging script in python that would insert into memcached. Here is the extra line I added to apache:

CustomLog "|/path/to/python /path/to/log_cache.py" combined

And this is log_cache.py:

#!/usr/bin/env python
 
import os
import sys
import re
from datetime import date
 
sys.path = ['/path/to/project',] + sys.path
os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'
 
from django.core.cache import cache
 
r = re.compile(r'"GET (?P\S+) ')
 
def be_parent(child_pid):
    exit_status = os.waitpid(child_pid, 0)
    if exit_status: # if there's an error, restart the child.
        pid = os.fork()
        if not pid:
            be_child()
        else:
            be_parent(pid)
    return
 
def be_child():
    while True:
        line = sys.stdin.readline() # wait for apache log data
        if not line:
            return # without error code so everything stops
        log_data(line)
 
def log_data(data):
    page = r.search(data)
    if page:
        key = '%s%s' % (date.today(), page.group('url'))
        try:
            cache.incr(key)
        except ValueError:
            # add it to the cache for 24 hours
            cache.set(key, 1, 24*60*60)
    return
 
pid = os.fork()
if not pid:
    be_child()
else:
    be_parent(pid)

A blog post about using python to store access records in postgres helped me out a lot. The parent/child processing came from that and fixed a lot of problems I was having before.

The page views are being added to memcached (with cache.incr() which is new in django 1.1) for quick retrieval and the logs will still be created by cronolog so no data will be lost when the cache expires. Those logs are used in the next part.

Parsing the Logs

The hit counts will expire from the cache after 24 hours so I parse the logs once a day and put that information into my database. For this I wrote a django management command (I didn’t do a management command before because I wasn’t sure how it would handle the parent and child processes). This command is called by ./manage.py parse_log

from django.conf import settings
from django.contrib.contenttypes.models import ContentType
from django.core.cache import cache
from django.core.management.base import BaseCommand
from django.core.urlresolvers import resolve, Resolver404
import datetime
# found on page linked above
from apachelogs import ApacheLogFile
from app.models import Model_being_hit
from metrics.models import Hits
 
def save_log(alf, date):
    hits = {}
    # loop to sum hits
    for log_line in alf:
        request = log_line.request_line
        request_parts = request.split(' ')
        hits[request_parts[1]] = hits.get(request_parts[1], 0) + 1
    for page, views in hits.iteritems():
        try:
            view, args, kwargs = resolve(page)
            # I check kwargs for something only passed to one app
            if 'param' in kwargs:
                a = Model_being_hit.objects.get(id=kwargs['id'])
                try:
                    content_type = ContentType.objects.get_for_model(a)
                    hit = Hits.objects.get(
                        date=date,
                        content_type=content_type,
                        object_id=a.id,
                    )
                    hit.views = views
                except Hits.DoesNotExist:
                    hit = Hits(date=date, views=views, content_object=a)
                hit.save()
        except:
            # something not in urls file like static files
            pass
class Command(BaseCommand):
    def handle(self, *args, **options):
        day = datetime.date.today()
        day = day - datetime.timedelta(days=1)
        alf = ApacheLogFile('%s/%s/%s/%s/access.log' % (
            settings.ACCESS_LOG_LOCATION,
            day.year,
            day.strftime('%m'), #month
            day.strftime('%d'), #day
        ))
        save_log(alf, day)

I use django.core.urlresolvers.resolve so that I can use my urls file and I don’t have to repeat myself.

Hits is a django model I created with a few fields for storing date and views. It uses the content types framework so that it can be tied to any of my django models.

from django.contrib.contenttypes        import generic
from django.contrib.contenttypes.models import ContentType
from django.db import models
 
class Hits(models.Model):
    date        = models.DateField()
    views       = models.IntegerField()
    # to add to any model
    content_type   = models.ForeignKey(ContentType)
    object_id      = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey('content_type', 'object_id')
 
    def __unicode__(self):
        return "%s hits on %s" % (self.views, self.date)

This was added to my cron with crontab -e

#every morning on the first minute
1 0 * * * /path/to/python /path/to/manage.py parse_log > /dev/null

Displaying the Hits

On my models I added a couple methods that would look up the info in the cache or database.

    @property
    def hits_today(self):
        from datetime import date
        from django.core.cache import cache
        key = '%s%s' % (date.today(), self.get_absolute_url())
        return cache.get(key)
 
    @property
    def hits(self):
        from metrics.models import Hits
        from django.contrib.contenttypes.models import ContentType
        content_type = ContentType.objects.get_for_model(self)
        hits = Hits.objects.filter(
            content_type=content_type,
            object_id=self.id,
        ).order_by('-date')
        return hits

The hits_today method requires that you define get_absolute_url which is useful in other places as well. @property is a decorator that makes it possible to access the data with object.hits and leave off the parenthesis.

The hits method uses the content type framework again to look up the hits in the database.

Just the Basics

There is a lot more that can be done with this. This barely touches the raw data available in the logs. A few ways I’ve already started improving this is to not include known bots as hits, check the referrer to see where traffic is coming from, and save the keywords used in search engines.

Related posts:

1. How to Speed up Your Django Sites with NginX, Memcached, and django-compress A lot of these steps will speed up any kind...
2. Python Projects in Users’ Home Directories with wsgi Letting users put static files and php files in a...
3. How to Write Reusable Apps for Pinax and Django Pinax is a collection of reusable django apps that...

In the apache configuration files I created some regular expression patterns that will look for a wsgi file based on the url requested. To serve this url: http://domain/~user/p/myproject, the server will look for this wsgi file: /home/user/public_python/myproject/deploy/myproject.wsgi

It is set up to run wsgi in daemon mode so that each user can touch their own wsgi file to restart their project instead of needing to reload the apache config and inconvenience everyone.

This is the code I added to the apache configuration (in a virtual host, other configs might be different):

RewriteEngine On
RewriteCond %{REQUEST_URI} ^/~(\w+)/p/(\w+)/(.*)
RewriteRule . - [E=python_project_name:%2]
 
WSGIScriptAliasMatch ^/~(\w+)/p/(\w+)  /home/$1/public_python/$2/deploy/$2.wsgi
WSGIDaemonProcess wsgi_processes.%{ENV:python_project_name}
processes=2 threads=15
WSGIProcessGroup wsgi_processes.%{ENV:python_project_name}
 
AliasMatch ^/~(\w+)/p/(\w+)/files(.*) /home/$1/public_python/$2/files$3
<LocationMatch ^/~(\w+)/p/(\w+)/files(.*)>
       SetHandler none
</LocationMatch>
 
AliasMatch ^/~(\w+)/p/(\w+)/media(.*) /home/$1/public_python/$2/media$3
<LocationMatch ^/~(\w+)/p/(\w+)/media(.*)>
       SetHandler none
</LocationMatch>

This will also serve two directories statically for images, css, and javascript. For one of them, I always make a symbolic link to the django admin media and tell my settings file to use that.

ln -s /path/to/django/contrib/admin/media media

To use this for a django project

This is a sample wsgi file to use for a django project. Username and project_name will need to be replaced. I’m also adding an apps folder to the path following the style I mention in my reusable apps post.

import os
import sys
 
sys.path = ['/home/username/public_python/', '/home/username/public_python/project_name/apps'] + sys.path
from django.core.handlers.wsgi import WSGIHandler
 
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.settings'
application = WSGIHandler()

I’ve been using this for a couple weeks and it’s working great for me. If you use it, I’d like to know how it works out for you. Let me know in the comments.

Related posts:

1. How to Add Locations to Python Path for Reusable Django Apps In my previous post I talk about reusable apps, but...
2. Getting Basecamp API Working with Python I found one library that was linked everywhere, but it...
3. Setting up Apache2, mod_python, MySQL, and Django on Debian Lenny or Ubuntu Hardy Heron Both Debian and Ubuntu make it really simple to get...

Django allows developers to create model managers that can change how the models work. The code below was written to return only the undeleted records by default. I added two new methods in case I need to get some of the deleted records.

from django.db import models
 
class SoftDeleteManager(models.Manager):
    ''' Use this manager to get objects that have a deleted field '''
    def get_query_set(self):
        return super(SoftDeleteManager, self).get_query_set().filter(deleted=False)
    def all_with_deleted(self):
        return super(SoftDeleteManager, self).get_query_set()
    def deleted_set(self):
        return super(SoftDeleteManager, self).get_query_set().filter(deleted=True)

This is usable by many models by adding this line to the model (it needs a deleted field) objects = SoftDeleteManager()

This will hide deleted records from django completely, even the django admin and even if you specify the id directly. The only way to find it is through the database itself or an app like phpMyAdmin. This might be good for some cases, but I went a step further to make it possible to undelete things in the django admin.

Django has a lot of customization options for the admin interface ( this article has some more info on customizing the django admin). I wanted the queryset to be different in the admin, so I created a ModelAdmin to customize what is displayed. First I set it up to show a few more columns than just __unicode__ on the list of items and added a filter to help easily separate the deleted from the active.

from django.contrib import admin
 
class SoftDeleteAdmin(admin.ModelAdmin):
    list_display = ('id', '__unicode__', 'deleted',)
    list_filter = ('deleted',)
# this requires __unicode__ to be defined in your model

This can also be used by many models by adding this at the bottom of the models.py file:

from django.contrib import admin
from wherever import SoftDeleteAdmin
admin.site.register(MyModel, SoftDeleteAdmin)

The next thing to do was override the queryset method in the default ModelAdmin. I copied the code from the django source and changed it from using get_query_set to make it use all_with_deleted() which was a method added to the ModelManager. The following code was added to SoftDeleteAdmin.

    def queryset(self, request):
        """ Returns a QuerySet of all model instances that can be edited by the
        admin site. This is used by changelist_view. """
        # Default: qs = self.model._default_manager.get_query_set()
        qs = self.model._default_manager.all_with_deleted()
        # TODO: this should be handled by some parameter to the ChangeList.
        ordering = self.ordering or () # otherwise we might try to *None, which is bad ;)
        if ordering:
            qs = qs.order_by(*ordering)
        return qs

The list of objects in the admin will start to look like this.

A screenshot of the django admin interface

They are showing up there now, but won’t be editable yet because django is using get_query_set to find them. There are two methods I added to SoftDeleteManager so that django can find the deleted records.

    def get(self, *args, **kwargs):
        ''' if a specific record was requested, return it even if it's deleted '''
        return self.all_with_deleted().get(*args, **kwargs)
 
    def filter(self, *args, **kwargs):
        ''' if pk was specified as a kwarg, return even if it's deleted '''
        if 'pk' in kwargs:
            return self.all_with_deleted().filter(*args, **kwargs)
        return self.get_query_set().filter(*args, **kwargs)

With those updated methods, django will be able to find records if the primary key is specified, not only in the admin section, but everywhere in the project. Lists of objects will only return deleted records in the admin section still.

This code can be applied to a bunch of models and easily allow soft deletes of records to prevent loss of accidentally deleted objects.

Related posts:

1. How to Display Realtime Traffic Analytics Users of Presskit’n have been asking for traffic statistics on...
2. How to Write Django Template Tags Template tags can be useful for making your applications more...
3. How to Write Reusable Apps for Pinax and Django Pinax is a collection of reusable django apps that...

With the redirect option I could redirect users to the main domain, check for cookies, and redirect them back so that they could get new cookies for the additional domain. The downside to this method is it will increase traffic for every pageload from a new visitor even if they will never need to log in. And since the sites this was for will have pages being viewed many more times than there will be logged in users, it wasn’t worth all of the extra traffic. It might be possible to minimize this traffic by only redirecting on login pages, but if the login form is at the top of all pages then it doesn’t help much.

Facebook uses a javascript method on all of the sites where you see facebook connect so you can use your facebook credentials to comment on blogs and other things. This method may be fine for their case, but again it will cause the extra traffic since the javascript is still connecting to the main server to get cookie info. I also don’t want to rely on javascript for my sessions.

I wanted a solution where it would only keep users logged in when they needed to be kept logged in. One way of knowing if they need to be kept logged in is: they are on one domain and click a link to go over to the other domain. Using a single-sign-on link to the other domain, the user would stay logged in at the new domain. The only use case that this doesn’t account for is someone is logged in at one domain and then types the other domain into the address bar. However that is a minimal case and I think the sso link will be the best way to keep users logged in most of the time and keep the overhead down.

I plan on open sourcing the django sso code so that other people can use it in their projects. It will allow a django site to accept single sign on requests and it will also help to create single sign on links to other sites. Both ends of the process don’t need to be a django site since it should work with other applications that use this type of process to authenticate users.

I’ll write a post on here about how to use the code once I get it set up at google code so if you are interested in that, you should probably subscribe to the rss so you don’t miss it.

Related posts:

1. Django Settings Site Domain example.com It took me a while to figure out how to...
2. OpenID Enabled If you haven’t stumbled upon any sites that use OpenID...
3. Python Projects in Users’ Home Directories with wsgi Letting users put static files and php files in a...

These three simple steps will speed up your server and allow it to handle more traffic.

• Reducing the Number of HTTP Requests
• Caching
• Using a lightweight front end server

Reducing the Number of HTTP Requests

Yahoo has developed a firefox extension called YSlow. It analyzes all of the traffic from a website and gives a score on a few categories where improvements can be made.

It recommends reducing all of your css files into one file and all of your js files into one file or as few as possible. There is a pluggable, open source django application available to help with that task. After setting up django-compress, a website will have css and js files that are minified (excess white space and characters are removed to reduce file size). The application will also give the files version numbers so that they can be cached by the web browser and won’t need to be downloaded again until a change is made and a new version of the file is created. How to setup the server to set a far future expiration is shown below in the lightweight server section.

Setting up Memcached

Django makes it really simple to set up caching backends and memcached is easy to install.

sudo aptitude install memcached, python-setuptools

We will need setuptools so that we can do the following command.

sudo easy_install python-memcached

Once that is done you can start the memcached server by doing the following:

sudo memcached -d -u www-data -p 11211 -m 64

-d will start it in daemon mode, -u is the user for it to run as, -p is the port, and -m is the maximum number of megabytes of memory to use.

Now open up the settings.py file for your project and add the following line:

CACHE_BACKEND = 'memcached://127.0.0.1:11211/'

Find the MIDDLEWARE_CLASSES section and add this to the beginning of the list:

    'django.middleware.cache.UpdateCacheMiddleware',

and this to the end of the list:

    'django.middleware.cache.FetchFromCacheMiddleware',

For more about caching with django see the django docs on caching. You can reload the server now to try it out.

sudo /etc/init.d/apache2 reload

To make sure that memcached is set up correctly you can telnet into it and get some statistics.

telnet localhost 11211

Once you are in type stats and it will show some information (press ctrl ] and then ctrl d to exit). If there are too many zeroes, it either isn’t working or you haven’t visited your site since the caching was set up. See the memcached site for more information.

Don’t Use Apache for Static Files

Apache has some overhead involved that makes it good for serving php, python, or ruby applications, but you do not need that for static files like your images, style sheets, and javascript. There are a few options for lightweight servers that you can put in front of apache to handle the static files. Lighttpd (lighty) and nginx (engine x) are two good options. Adding this layer in front of your application will act as an application firewall so there is a security bonus to the speed bonus.

There is this guide to install a django setup with nginx and apache from scratch. If you followed my guide to set up your server or already have apache set up for your application, then there are a few steps to get nginx handling your static files.

sudo aptitude install nginx

Edit the config file for your site (sudo nano /etc/apache2/sites-available/default) and change the port from 80 to 8080 and change the ip address (might be *) to 127.0.0.1. The lines will look like the following

NameVirtualHost 127.0.0.1:8080
<VirtualHost 127.0.0.1:8080>

Also edit the ports.conf file (sudo nano /etc/apache2/ports.conf) so that it will listen on 8080.

Listen 8080

Don’t restart the server yet, you want to configure nginx first. Edit the default nginx config file (sudo nano /etc/nginx/sites-available/default) and find where it says

        location / {
               root   /var/www/nginx-default;
               index  index.html index.htm;
        }

and replace it with

location / {
    proxy_pass http://192.168.0.180:8080;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    client_max_body_size 10m;
    client_body_buffer_size 128k;
    proxy_connect_timeout 90;
    proxy_send_timeout 90;
    proxy_read_timeout 90;
    proxy_buffer_size 4k;
    proxy_buffers 4 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k; 
}
location /files/ {
    root /var/www/myproject/;
    expires max;
}

/files/ is where I’ve stored all of my static files and /var/www/myproject/ is where my project lives and it contains the files directory.

Set static files to expire far in the future

expires max; will tell your users’ browsers to cache the files from that directory for a long time. Only use that if you are use those files won’t change. You can use expires 24h; if you aren’t sure.

Configure gzip

Edit the nginx configuration to use gzip on all of your static files (sudo nano /etc/nginx/nginx.conf). Where it says gzip on; make sure it looks like the following:

    gzip  on;
    gzip_comp_level 2;
    gzip_proxied any;
    gzip_types      text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;

The servers should be ready to be restarted.

sudo /etc/init.d/apache2 reload
sudo /etc/init.d/nginx reload

If you are having any problems I suggest reading through this guide and seeing if you have something set up differently.

Speedy Django Sites

Those three steps should speed up your server and allow for more simultaneous visitors. There is a lot more that can be done, but getting these three easy things out of the way first is a good start.

Related posts:

1. Static Files in Django on Production and Development Update 2009-03-25 I realize why this isn’t needed. If your...
2. How to Display Realtime Traffic Analytics Users of Presskit’n have been asking for traffic statistics on...
3. Python Projects in Users’ Home Directories with wsgi Letting users put static files and php files in a...

from appname.filename import foo

There are a few places you might need to add an apps folder to the pythonpath.

Add to settings.py

This will add the apps directory in your project directory to the beginning of the path list. This will allow manage.py syncdb and manage.py runserver to know that the apps folder should be added.

import os
import sys
PROJECT_ROOT = os.path.dirname(__file__)
sys.path.insert(0, os.path.join(PROJECT_ROOT, "apps"))

That should be all you need to do to get most django projects working with your reusable apps, but if for any reason you need add to the path with mod python or mod wsgi, the following should work.

Apache mod_python

In the setting-up-everything post I show an example httpd.conf file. In your apache configuration you will probably see something similar to what is below. To add the location /var/www/myproject/apps to the PythonPath I added it in the list.

SetHandler python-program
PythonHandler django.core.handlers.modpython
SetEnv DJANGO_SETTINGS_MODULE myproject.settings
PythonOption django.root /myproject
PythonDebug On
PythonPath "['/var/www','/var/www/myproject/apps'] + sys.path"

Apache mod_wsgi

If you use mod wsgi instead of mod python, your apache config will be loading a wsgi file with a line like this WSGIScriptAlias /var/www/myproject/myproject.wsgi. You will need to edit that file to add to the path (django’s site has an example file).

sys.path.append('/var/www/myproject/apps')

You never know when you might want to use an app in another project, so always try to keep from mentioning the project name anywhere in the applications.

Related posts:

1. Python Projects in Users’ Home Directories with wsgi Letting users put static files and php files in a...
2. How to Write Reusable Apps for Pinax and Django Pinax is a collection of reusable django apps that...
3. Getting Basecamp API Working with Python I found one library that was linked everywhere, but it...

import urllib2
 
protocol = 'https://'
url = 'example.com'
command = '/projects.xml'
headers = {'Accept' : 'application/xml', 
'Content-type' : 'applications/xml'}
username = 'x'
password = 'y'
 
# Setup password stuff
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
 
# Send the request and get response
req = urllib2.Request(protocol + url + command, None, headers)
response = urllib2.urlopen(req)
results = response.read()
 
print results

I thought it was a problem with how the authorization was formed so based on the above code I modified the old basecamp.py file and I was able to get a response. The following is what I changed.

Around line 64

    def __init__(self, username, password, protocol, url):
        self.baseURL = protocol+url
        if self.baseURL[-1] == '/':
            self.baseURL = self.baseURL[:-1]
 
        passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
        passman.add_password(None, url, username, password)
        authhandler = urllib2.HTTPBasicAuthHandler(passman)
 
        self.opener = urllib2.build_opener(authhandler)

And around line 142

path = '/projects.xml'

With that I was able to use basecamp.py to retrieve a list of projects. Other modifications may be needed for other features, but that was all I planned on using.

Here is an example of using ElementTree to parse the XML response to get the names of all of the projects returned from basecamp.

import elementtree.ElementTree as ET
from basecamp import Basecamp
 
protocol = 'https://'
url = 'example.com'
username = 'x'
password = 'y'
 
bc = Basecamp(username, password, protocol, url)
projects = bc.projects()
tree = ET.fromstring(projects)
tags = tree.getiterator(tag='project')
 
for t in tags:
    project_name = t.findtext('name')

Related posts:

1. Python Projects in Users’ Home Directories with wsgi Letting users put static files and php files in a...
2. How to Add Locations to Python Path for Reusable Django Apps In my previous post I talk about reusable apps, but...
3. Setting up Apache2, mod_python, MySQL, and Django on Debian Lenny or Ubuntu Hardy Heron Both Debian and Ubuntu make it really simple to get...

The evening events were a great chance to meet people that are doing awesome things. There was the unclasses guy and the awe.sm guy that I met through Tim. And then there was someone doing something more awesome than I ever could. He is a professional contest winner. He enters funny videos into contests and wins things like trips around the world where he will make more videos and win more contests.

I met a lot of the Orlando crowd while in another city. I met and followed Eric a few months ago and through him I was able to meet some of the people that go to Florida Creatives happy hour. So that was a great chance to get to know everyone better.

There was also a presentation from the rails envy podcast guy, Gregg Pollack from Orlando. Gregg’s was the only one I went to that wasn’t a panel. The panels could be a bit boring sometimes, but they gave a chance to hear from about 5 people with experience on the topic. The higher education panel started bad just like some of the other panels, but it got better as it went since they got into the important stuff. It would seem that I am still passionate about improving education and I hope that I will be able to work on something at UCF soon that will do just that.

There were a few other panels that weren’t that bad like How to Rawk SXSW where the panelists were experienced SXSWers and gave advice to the noobs. After my visit, the advice I would give is “Be ready to network” because that is the most important part of the conference. And the last panel I attended was how to rawk the rest of the year which had some advice like keep in contact with the people that you’ve met here. One easy way of doing that is finding out who they are on twitter (everyone at the event was on twitter by the end). One other piece of advice from the panel was the parties aren’t the only place to meet people, hallways and streets work well too. Maybe that would have been better advice at the beginning, but it arose from a question.

Hopefully I will get to run into a lot of the Orlando people again at the happy hours and of course I’ll be at BarCampOrlando on April 18th.

Related posts:

1. Keep Friends Posted While on the Road Going on an amazing road trip used to mean that...
2. Just Heard About an Open-Source Database Conference Baron Schwartz, the lead author of High Performance MySQL 2nd...

Start off by creating a folder called templatetags in your app directory and create two files in it. The first one named __init__.py and the second book_tags.py. There’s 3 things that we need to accomplish with our template tags. The first is to create a tag that will output the url for the action of the form. For example, {% get_book_form_url foo_object %}Next we need to get the form and assign it to a template variable that can be specified by the template variable. For example, {% book_form as bar_var %}. And the third template tag will get the books for an object and place in a template variable. For example, {% books_for_object foo_object as bar_var %}.

from django.template import Library, Node, TemplateSyntaxError
from django.template import Variable, resolve_variable
from django.utils.translation import ugettext as _
from django.contrib.contenttypes.models import ContentType
from django.core.urlresolvers import reverse
from books.models import Book
 
register = Library()
 
def get_contenttype_kwargs(content_object):
    """
    Gets the basic kwargs necessary for form submission url
    """
    kwargs = {'content_type_id':
        ContentType.objects.get_for_model(content_object).id,
    'object_id':
        getattr(content_object, 'pk',
            getattr(content_object, 'id')),
    }
    return kwargs
 
def get_book_form_url(content_object):
    """
    prints url for form action
    """
    kwargs = get_contenttype_kwargs(content_object)
    return reverse('new_book', kwargs=kwargs)
 
class BooksForObjectsNode(Node):
    """
    Get the books and add to the context
    """
    def __init__(self, obj, context_var):
        self.obj = Variable(obj)
        self.context_var = context_var
 
    def render(self, context):
        content_type = ContentType.objects.get_for_model(
            self.obj.resolve(context))
        # create the template var by adding to context
        context[self.context_var] = \
            Book.objects.filter( # find all books for object
                content_type__pk = content_type.id,
                object_id = self.obj.resolve(context).id
            )
        return ''
 
def books_for_object(parser, token):
    """
    Retrieves a list of books for given object
    {% books_for_object foo_object as book_list %}
    """
    try:
        bits = token.split_contents()
    except ValueError:
        raise TemplateSyntaxError(
            _('tag requires exactly two arguments')
    if len(bits) != 4:
        raise TemplateSyntaxError(
            _('tag requires exactly three arguments')
    if bits[2] != 'as':
        raise TemplateSyntaxError(
            _("second argument to tag must be 'as'")
    return BooksForObjectsNode(bits[1], bits[3])
 
def book_form(parser, token):
    """
    Adds a form to the context as given variable
    {% book_form as form %}
    """
    # take steps to ensure template var was formatted properly
    try:
        bits = token.split_contents()
    except ValueError:
        raise TemplateSyntaxError(
            _('tag requires exactly two arguments')
    if bits[1] != 'as':
        raise TemplateSyntaxError(
            _("second argument to tag must be 'as'")
    if len(bits) != 3:
        raise TemplateSyntaxError(
            _('tag requires exactly two arguments')
    # get the form
    return BookFormNode(bits[2])
 
class BookFormNode(Node):
    """
    Get the form and add it to the context
    """
    def __init__(self, context_name):
        self.context_name = context_name
    def render(self, context):
        from books.forms import NewBookForm
        form = NewBookForm()
        # create the template var by adding to context
        context[self.context_name] = form
        return ''
 
# register these tags for use in template files
register.tag('books_for_object', books_for_object)
register.tag('book_form', book_form)
register.simple_tag(get_book_form_url)

Add this to your template

To start adding books to an object, add this code to your template and change my_awesome_object_here to the template variable name of your object.

<h2>Books</h2>
{% load book_tags %}
 
{% books_for_object my_awesome_object_here as books %}
{% for book in books %}
 
<a href="{{ book.get_absolute_url }}">{{ book }}</a> -
        {{ book.description }}
 
{% endfor %}
<h2>Add a book</h2>
<form action="{% get_book_form_url my_awesome_object_here %}" method="post">
{% book_form as form %}
{{ form }}
<input type="submit" value="Go" />
</form>

You can get the template tags source code and the code from the previous post at the google code project page or by doing

svn co http://django-books.googlecode.com/svn/trunk books

in a directory on the python path.

Related posts:

1. How to Write Reusable Apps for Pinax and Django Pinax is a collection of reusable django apps that...
2. Django RequestContext Example Browsing other peoples’ code is a great way to learn...
3. Quick Thumbnails in Django I normally like to write code myself instead of installing...

My example will be to create a list of books and allow them to be tied to any object using Django’s ContentType framework. The books could be recommended reading for the members of a tribe (pinax group), a class, or anything in your project and will include title, description, and tags (requires django-tagging). In another post I’ve shown how to create template tags to make it easy to show the list of books and a form to add a book. Obviously, there is a lot more that could be done with this app, but I will leave it out of the example to keep it simple.

Starting the App

Create a folder in the apps directory or any place that is on the python path (ex. /path/to/pinax/projects/complete_project/apps/books/) and include these files:

• __init__.py even though it might be empty, it is required
• forms.py
• models.py
• urls.py
• views.py

models.py

I will start with creating the model for the project. Below is all of the code I am placing in the file. I’ve added a lot of comments to explain everything that is happening.

#import all of the things we will be using
from django.db                          import models
from tagging.fields                     import TagField
# to help with translation of field names
from django.utils.translation  import ugettext_lazy as _
# to have a generic foreign key for any model
from django.contrib.contenttypes        import generic
# stores model info so this can be applied to any model
from django.contrib.contenttypes.models import ContentType
 
class Book(models.Model):
    """
    The details of a Book
    """
    # fields that describe this book
    name        = models.CharField(_('name'), max_length=48)
    description = models.TextField(_('description'))
 
    # to add to any model
    content_type   = models.ForeignKey(ContentType)
    object_id      = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey('content_type',
        'object_id')
 
    # for the list of tags for this book
    tags        = TagField()
 
    # misc fields
    deleted     = models.BooleanField(default=0)
    created     = models.DateTimeField(auto_now_add=True)
    # so that {{book.get_absolute_url}} outputs the whole url
    @models.permalink
    def get_absolute_url(self):
        return ("book_details", [self.pk])
    # outputs name when printing this object as a string
    def __unicode__(self):
        return self.name

forms.py

Use Django’s ModelForm to create a form for our book model.

from django import forms
from books.models import Book
 
class NewBookForm(forms.ModelForm):
    class Meta:
        model = Book
        exclude = ('deleted', 'content_type',
            'object_id', 'created')

views.py

In this file we create a view to show the details of a book and a view to create a new book for an object.

from django.shortcuts import render_to_response
from django.shortcuts import get_object_or_404
from django.http import HttpResponseRedirect
from django.template import RequestContext
from django.core.urlresolvers import reverse
from django.utils.translation import ugettext_lazy as _
from django.contrib.auth.decorators import login_required
 
from tribes.models import Tribe
from books.models import Book
from django.contrib.contenttypes.models import ContentType
 
@login_required
def new(request, content_type_id, object_id,
            template_name="books/new.html"):
    """
    creates a new book
    """
    from books.forms import NewBookForm
 
    # if a new book was posted
    if request.method == 'POST':
        book_form = NewBookForm(request.POST)
        if book_form.is_valid():
            # create it
            book = book_form.save(commit=False)
            content_type        = \
                ContentType.objects.get(id=content_type_id)
            content_object      = \
                content_type.get_object_for_this_type(
                id=object_id)
            book.content_object = content_object
            book.save()
            request.user.message_set.create(
                message=
                _("Successfully created book '%s'")
                % book.name)
            # send to object page or book page
            try:
                return HttpResponseRedirect(
                    content_object.get_absolute_url()
                )
            except:
                return HttpResponseRedirect(reverse(
                    'book_details', args=(book.id,)))
        # if invalid, it gets displayed below
    else:
        book_form = NewBookForm()
 
    return render_to_response(template_name, {
        'book_form': book_form,
    }, context_instance=RequestContext(request))
 
@login_required
def details(request, book_id,
    template_name="books/details.html"):
    """
    displays details of a book
    """
    book = get_object_or_404(Book, id=book_id)
    return render_to_response(template_name, {
        'book': book,
    }, context_instance=RequestContext(request))

urls.py

To tie our views to some urls, add this to the urls.py file.

from django.conf.urls.defaults import *
from django.conf.urls.defaults import *
 
urlpatterns = patterns('',    
    # new book for object
    url(r'^new/(?P<content_type_id>\d+)/(?P<object_id>\d+)', 
        'books.views.new', name="new_book"),
    # display details of a book
    url(r'^details/(?P<book_id>\d+)$', 'books.views.details', 
        name="book_details"),
)

More Features

The rest of the application is described in the post titled: How to Write Django Template Tags. You can also check out all of the code from the google project by doing the following command:

svn co http://django-books.googlecode.com/svn/trunk books

in a directory on the python path.

Related posts:

1. How to Write Django Template Tags Template tags can be useful for making your applications more...
2. How to Add Locations to Python Path for Reusable Django Apps In my previous post I talk about reusable apps, but...
3. How to Display Realtime Traffic Analytics Users of Presskit’n have been asking for traffic statistics on...