oebfare: Latest Entries

Gitify Python SVN

Mon, 08 Dec 2008 10:12:28 -0600

I love git. I try to do as much as I can using it. I have converted many SVN based projects over to git so that I can interact with them more naturally. I wanted a bit more of a challenge than just using git svn clone to make the repositories.

I love Python. I use it all over the place such as web applications for one. I want to some day be able to give back to Python in some form of a contribution. Perhaps this is a first step. I tasked myself with moving Python's SVN into git. Python has a pretty long history which makes it a bit more tricky.

Going local

One approach that I can take is to use git svn clone on the public SVN URL, but that would take a very long time. It would be much better to have access to actual repository and do a clone locally. Fortnately Python provides the full repository:

wget http://svn.python.org/snapshots/projects-svn-tarball.tar.bz2
tar xvfj projects-svn-tarball.tar.bz2

This now gives us a diretory named projects that is a bonafide SVN repository. You can verify this by running:

svn ls file:///Users/brian/code/python-svn/projects

Note

The SVN path above is critical. The absolute path above is /Users/brian/code/python-svn/projects. This may be referenced in other places, be sure to know what yours is if you are trying accomplish something similar.

Gitify

Lets go ahead and begin the process of moving everything from SVN into git. We simply use git svn to handle this for us. It does a great job.

git svn clone -s --prefix=svn/ file:///Users/brian/code/python-svn/projects/python

We pass it -s which indicates that the tree at the given URL follows the standard SVN layout with trunk, branches and tags. This helps git see those and make them native. I always use --prefix=svn/ so that the ref names for the SVN stuff gets prefixed and will avoid name clashing of possible local branches.

The above command will take a very long time to complete in this case. If the SVN repository is small enough it will be relatively quick. It took about five hours on my Macbook Pro.

Getting our hands dirty

Now we have a git repository of Python's SVN! However, we are not left with some issues. We pulled down a nightly snapshot of the Python SVN repository. We want to be able to keep it in sync the the on-going changes upstream.

git svn stores some of its information in .git/config:

[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
    ignorecase = true
[svn-remote "svn"]
    url = file:///Users/brian/code/python-svn/projects
    fetch = python/trunk:refs/remotes/svn/trunk
    branches = python/branches/*:refs/remotes/svn/*
    tags = python/tags/*:refs/remotes/svn/tags/*

It seems logical that I can adjust the repository URL above and be able to git svn fetch without an issue. I have attempted this process several times all to which failed in some sort of way giving me:

Unable to determine upstream SVN information from working tree history

However, this one time it actually worked. The difference I am aware of between this attempt and the previous attempts was the version of git I am using. I am using git 1.6.0.4 which seems to handle this better. Perhaps someone knows if there was indeed some fixes in regard to this.

In my digging I found out that the reason why it would fail with the above message is that git svn is unable to automatically detect the repository information from the commit messages. If you look at git log you will see:

commit a3e5b71ea1f27e29285c7a3e686e6084d10231a7
Author: benjamin.peterson <benjamin.peterson@6015fed2-1504-0410-9fe1-9d1591cc4771>
Date:   Sun Nov 23 02:09:41 2008 +0000

    raise a better error

    git-svn-id: file:///Users/brian/code/python-svn/projects/python/trunk@67348 6015fed2-1504-0410-9fe1-9d1591cc4771

Notice the git-svn-id part. This has meaning to git svn. Rememeber before how I told you that the full path was important? My end goal for this task was to make this repository available. I need to change this in all commits.

`git filter-branch` is really poweful

Maybe too powerful? git comes with filter-branch that enables you to completely rewrite a branch commit history. This is pretty insane. I wanted to adjust the commit messages to contain the correct path in git-svn-id.

Lets break out some sed and awk awesomeness here:

git filter-branch \
    --msg-filter \
        'sed -e "s/^git-svn-id: file:\/\/\/Users\/brian\/code\/python-svn\//git-svn-id: http:\/\/svn.python.org\//g"' \
    $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')

This will rewrite the entire repository history adjusting the commit messages by changing file:///Users/brian/code/python-svn/ to http://svn.python.org/. Credit goes to this blog where I ripped the above command off. That blogger goes into more detail and I recommend the read.

Once again, the above command is going to take a very long time in this case.

Ta da!

The very last step in this process is to kill .git/svn. I figured at first that I didn't need to do this, but git svn rebase was taking forever. Did this:

rm -rf .git/svn
git svn rebase --all

And viola! All is working now.

I worked with Jannis Leidel to get the repository auto-updating and publically available. You can clone it from GitHub:

git clone git://github.com/python-git/python.git

I hope I was able to help out someone and provide a place where more people can contribute back to Python using git. Do keep in mind this git mirror is unofficial and all patches should be directed upstream to the Python developers.

Git Tab Complete

Fri, 14 Nov 2008 11:16:22 -0600

One thing that has bothered my about git was the lack of tab complete. Well, this was completely my fault. It turns out that in the git source tree in their contrib directory is tab complete bash script! Here is how I installed it and made my life 75.6 times easier:

cd GIT_SRC/contrib/completion
cp git-completion.bash ~/bin
echo "source ~/bin/git-completion.bash" >> ~/.profile
source ~/.profile

Viola.

Book Memery

Wed, 12 Nov 2008 13:11:05 -0600

Doing this thing.

The guard GuardExpr1, GuardExpr2, ..., GuardExprN is true if all the guard expressions -- GuardExpr1, GuardExpr2, ... -- evaluate to true.

Via Programming Erlang

Grab the nearest book.

Open it to page 56.

Find the fifth sentence.

Post the text of the sentence in your journal along with these instructions.

Don't dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

Saw this on Justin Lilly's and Greg Newman's blog.

Multiple Foreign Keys to Same Model for Inlines

Mon, 10 Nov 2008 23:26:39 -0600

When you are working with a model that has more than one foreign key to the same model and you want to inline that model you will run into problems. The reason why is due to the way inlineformset_factory works. By default it can work out what foreign key on the given model is referenced to the parent. Lets look how you might call inlineformset_factory:

BookFormSet = inlineformset_factory(Author, Book)

The above case is simple. In all likelyhood you will only have one foriegn key to Author. Lets look at a more complex scenerio:

class Person(models.Model):
        name = models.CharField(max_length=100)

class Friendship(models.Model):
        from_friend = models.ForeignKey(Person, related_name="friend_set")
        to_friend = models.ForeignKey(Person, related_name="otherside")

The models above show a more complex case of something you might want to inline. The above Friendship model would likely have more fields to make the inline worth something. Now lets look at how you might construct the inline formset for this:

FriendshipInline = inlineformset_factory(Person, Friendship, fk_name="from_friend")

We have explicitly given a fk_name because by default inlineformset_factory would not have known which foriegn key you intend to use to Person.

Side Note

I did have a two day hiatus in my blogging, but I will make up for those, I promise.

App Settings

Fri, 07 Nov 2008 23:01:43 -0600

In my recent reusable apps conventions post there was a comment by Devin Naquin who brought up app settings. I replied saying that it was one thing I didn't discuss, but wanted to revisit in another blog post. This will be that post.

There are times where an app needs to have some value defined by the user. However obtaining this value is somewhat not very clear. Django projects have a settings.py module that stores settings that a site developer might want to adjust based on the site or server. I believe this is the best place to store app settings. There has been some discussion in the Pinax community regarding the need of multiple levels of settings files. This would allow a seperation between global, site-specific and installation specific settings.

Looking at project settings

Lets step back a little and take a look at the settings.py file that you use to modify the project's settings. This file can really be divided it up. It is ideal that the file straight out of startapp is not committed into your own version control. The reasoning is due to some settings having either sensitive info (if it is a public repository) or they are almost always going to be changed in different checkouts. One of the common ways of dealing with this is to break out the installation specific settings from the more global ones that are more project level. Most users will simply add this bit to the end of the settings.py file:

try:
        from local_settings import *
except ImportError:
        pass

This enables a file named local_settings.py to contain setting overrides to the main settings. This is done at the end of the file to allow for the overriding of global variables. This works well and can go a long way.

The reason why this is important to app settings is that if you are distributing your app as reusable you might want to encourage this way of handling settings. Users can add settings that may be installation specific such as api keys, file system paths, etc.

Naming conventions

The way I normally handle this is to prefix the setting with the app name. For example in django-mailer users have the ability to control how long it sleeps in a daemon mode before querying the database. It is named MAILER_QUEUE_SLEEP. This helps to easily distiquish this setting from others ones.

Getting the values and dealing with optional settings

When it comes time to needing the value of a particular setting you will need to get it. Django provides a proxy class between the settings.py and the code that references them. Here is how you might grab the value of the setting I mentioned above:

from django.conf import settings
QUEUE_SLEEP = getattr(settings, "MAILER_QUEUE_SLEEP")

The other bit of naming conventions is that your name the variable that you bind the data to locally as the setting name without the app name to prefix it since it is implied. When the user does not provide MAILER_QUEUE_SLEEP this will raise an AttributeError exception. I see this has an expected error if the setting is required.

If you need the ability to make the setting optional where you provide a sensible default you can do that very easily:

QUEUE_SLEEP = getattr(settings, "MAILER_QUEUE_SLEEP", 30)

getattr takes an optional third parameter that will be returned in the event the second parameter is not found on the first parameter. This makes it very simple to give a sensible default when not provided, but also allow the user to override the default through the standard way.

Wrapping up

I feel this is the best way to deal with app settings. This is drop dead simple and effective. However, there are places where things could be improved. What if an app wanted to allow the end-user to define different parameters? Good question and it sounds like it would make an excellent reusable app to handle that.

Essentials

Thu, 06 Nov 2008 22:18:44 -0600

I am going to cheat a little today and go for a simple post. I have some good content coming this weekend. This post will cover the tools I use everyday.

OS X 10.5.5

Colloquy

Safari

Twitterific

Textmate

Skype

Skitch

Cord

Adium

iTunes

rsync

git

VMWare Fusion

NetNewsWire

Those are the pretty typical OS X apps one would expect. Just to juice up this post a bit let me share a bit from my ~/.profile. Thanks to Justin Lilly who cooked up this gem:

NORMAL='\[\033[00m\]'
BGREEN='\[\033[1;32m\]'
BBLUE='\[\033[1;34m\]'
GREEN='\[\033[0;32m\]'

parse_git_branch() {
    git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/\1 /';
}
parse_svn_rev() {
    svn info 2> /dev/null | grep "Revision" | sed 's/Revision: \(.*\)/r\1 /';
}

export PS1="${BGREEN}\u@\h ${GREEN}\$(parse_git_branch)\$(parse_svn_rev)${BBLUE}[${NORMAL}\w${BBLUE}]\n${BBLUE}$ ${NORMAL}"

This will render your PS1 as:

brian@brosner-osx master [~/Development/twid/twid]
$

Colors won't show, but you get the idea. Also here are some aliases I use:

alias gst='git status'
alias gl='git pull'
alias gp='git push'
alias gd='git diff | mate'
alias gc='git commit -v'
alias gca='git commit -v -a'
alias gb='git branch'
alias gba='git branch -a'
alias gsr='git svn rebase'

alias gho='$(git remote -v | grep github | sed -e "s/.*git\:\/\/\([a-z]\.\)*/\1/" -e "s/\.git$//g" -e "s/.*@\(.*\)$/\1/g" | tr ":" "/" | tr -d "\011" | sed -e "s/^/open http:\/\//g")'

alias delpyc='find . -name "*.pyc" -exec rm -fv {} \;'
alias deljyc='find . -name "*$py.class" -exec rm -v {} \;'

The gho alias was another one from Justin Lilly. I haven't actually used it a ton, but in theory will open a GitHub repository page for the given repository you are in.

Moving away from Google Code

Wed, 05 Nov 2008 21:28:46 -0600

Over in Pinax land we have recently moved away from using Google Code. Google provides project hosting for open source projects. It really is a valuable service for anyone with an open source project. They provide a web interface that includes a wiki, issue tracking, SVN browser and a bit more.

Pinax lived under the project name django-hotclub on Google Code. It mostly still does. We were sort of outgrowing the features Google Code was providing us. Utlimately we wanted to write our own software development platform using Pinax so it seemed to be the best time, early in the project, to move the version code system elsewhere.

Step One

The first step in this process is to get a local copy of the Subversion repository. I first created a local SVN repository:

svnadmin create /home/brian/svn/pinax

SVN has a handy tool named svnsync. The first thing I did was initialize the repository for using svnsync. I simply executed:

svnsync init file:///home/brian/svn/pinax/ https://django-hotclub.googlecode.com/svn/

This fails with:

svnsync: Repository has not been enabled to accept revision propchanges;
ask the administrator to create a pre-revprop-change hook

I enabled this hook by simply moving the template hook to a real hook and making it executable:

mv /home/brian/pinax/hooks/pre-revprop-change.tmpl /home/brian/pinax/hooks/pre-revprop-change
chmod +x /home/brian/pinax/hooks/pre-revprop-change

The Pinax repository sets more props than what the hook allows (this is my best guess as to why) so I needed to modify the hook a tiny bit to allow any revprop change. I simply changed the the exit value in the one error case to return a 0 status code so svnsync thinks it works.

Once this is all set I was able to run the sync:

svnsync sync file:///home/brian/svn/pinax/

It was on a roll.

Step Two

Step two was to remap the usernames that were in Google Code to ones the committers normally go by. How on earth am I going to do this? I searched high and low on the Internet to see if there was some simple way. I could not find anything. James Tauber mentioned that the dump file you can get from SVN is plain text. This sparked an idea. Why not handle this before re-importing it. James said that the only way to set it up correctly on Webfaction was to dump from my synced repository and re-import it. This made the dump/import a requirement.

The next step was to learn how the file format is layed out. While I was searching the web I happened to come across a Python script that can read/write dump files. This was perfect. I don't have this source available, but once I find it I will give proper credit. It was a bit dated in terms of Python code so I rewrote it. It was a good opportunity to actually learn how it works.

After spending two/three hours rewriting it and understanding how it works I was successfully able to parse the dump file and remap the usernames to what we desired.

Here is the final script:

import sys
import copy

from cStringIO import StringIO

username_mapping = {
    "leidel": "jezdez",
    "floguy": "ericflo",
    "gregoryjnewman": "newman",
}

class Lump(object):
    def __init__(self):
        self.hdrlist = []
        self.hdrdict = {}
        self.proplist = []
        self.propdict = {}
        self.text_data = ""

    def sethdr(self, key, value):
        if key not in self.hdrdict:
            self.hdrlist.append(key)
        self.hdrdict[key] = value

    def delhdr(self, key):
        if key in self.hdrdict:
            del self.hdrdict[key]
            self.hdrlist.remove(key)

def read_rfc822_headers(fp):
    lump = Lump()
    while 1:
        line = fp.readline()
        if line == "":
            return None # eof
        if line == "\n":
            if len(lump.hdrlist) > 0:
                break # newline after headers ends them
            else:
                continue # newline before headers is simply ignored
        if line[-1:] == "\n":
            line = line[:-1]
        colon = line.find(":")
        assert colon > 0
        assert line[colon:colon+2] == ": "
        key, value = line[:colon], line[colon+2:]
        lump.sethdr(key, value)
    return lump

def props_parse(bytes):
    i = 0
    proplist = []
    propdict = {}
    while 1:
        if bytes[i:i+2] == "K ":
            need_value = True
        elif bytes[i:i+2] == "D ":
            need_value = False
        elif bytes[i:i+9] == "PROPS-END":
            break
        else:
            raise Exception, "Unrecognized record in props section"

        # the position of the \n character
        nl = bytes.find("\n", i)
        assert nl > 0
        # length of the key
        kl = int(bytes[i+2:nl])
        assert bytes[nl+1+kl] == "\n"
        # key value
        key = bytes[nl+1:nl+1+kl]
        # move the index position
        i = nl + 2 + kl

        if need_value:
            assert bytes[i:i+2] == "V "
            nl = bytes.find("\n", i)
            assert nl > 0
            vl = int(bytes[i+2:nl])
            assert bytes[nl+1+vl] == "\n"
            value = bytes[nl+1:nl+1+vl]
            i = nl + 2 + vl
        else:
            value = None
        proplist.append(key)
        propdict[key] = value
    return proplist, propdict

def read_lump(fp):
    lump = read_rfc822_headers(fp)
    if lump is None:
        return None
    pcl = int(lump.hdrdict.get("Prop-content-length", 0))
    tcl = int(lump.hdrdict.get("Text-content-length", 0))
    if pcl > 0:
        lump.proplist, lump.propdict = props_parse(fp.read(pcl))
    if tcl > 0:
        lump.text_data = fp.read(tcl)
    return lump

def write_lump(stream, lump):
    """
    """
    lump = copy.deepcopy(lump)
    prop_stream = StringIO()
    if lump.proplist:
        for key in lump.proplist:
            value = lump.propdict[key]
            if value is None:
                prop_stream.write("D %d\n" % len(key))
                prop_stream.write("%s\n" % key)
            else:
                # write key out
                prop_stream.write("K %d\n" % len(key))
                prop_stream.write("%s\n" % key)

                # write value out
                prop_stream.write("V %d\n" % len(value))
                prop_stream.write("%s\n" % value)
        prop_stream.write("PROPS-END\n")
    prop_data = prop_stream.getvalue()

    if len(prop_data) > 0:
        lump.sethdr("Prop-content-length", str(len(prop_data)))
    else:
        lump.delhdr("Prop-content-length")

    # TODO: if lump.text_data ever changes we need to write out a new
    # Text-content-md5
    if len(lump.text_data) > 0:
        lump.sethdr("Text-content-length", str(len(lump.text_data)))
    else:
        lump.delhdr("Text-content-length")

    if len(prop_data) > 0 or len(lump.text_data) > 0:
        lump.sethdr("Content-length", str(len(prop_data) + len(lump.text_data)))
    else:
        lump.delhdr("Content-length")

    # TODO: need a better data strucuture here for a sorted dict
    for key in lump.hdrlist:
        value = lump.hdrdict[key]
        stream.write("%s: %s\n" % (key, value))
    stream.write("\n")
    # write prop data
    stream.write(prop_data)
    # the text data was not modified so just write it to the stream
    stream.write(lump.text_data)
    if "Prop-content-length" in lump.hdrdict or \
         "Text-content-length" in lump.hdrdict or \
         "Content-length" in lump.hdrdict:
        stream.write("\n")

def convert_username(lump):
    if "svn:author" in lump.propdict:
        if lump.propdict["svn:author"] in username_mapping:
            committer = username_mapping[lump.propdict["svn:author"]]
        else:
            committer = lump.propdict["svn:author"]
        lump.propdict["svn:author"] = committer
    return lump

if __name__ == "__main__":
    fp = sys.stdin
    new_fp = sys.stdout

    # read headers up until a revision
    lump = read_lump(fp)
    while "Revision-number" not in lump.hdrdict:
        # write the lump to disk
        write_lump(new_fp, lump)
        lump = read_lump(fp)
        revhdr = lump

    while revhdr is not None:
        # read revision header
        assert "Revision-number" in revhdr.hdrdict
        print >> sys.stderr, "processing r%s" % (revhdr.hdrdict["Revision-number"],)
        revhdr = convert_username(revhdr)
        contents = []
        # read revision contents
        while 1:
            lump = read_lump(fp)
            if lump is None or "Revision-number" in lump.hdrdict:
                newrevhdr = lump
                break
            contents.append(lump)
        # write out the revision
        write_lump(new_fp, revhdr)
        for lump in contents:
            write_lump(new_fp, lump)
        revhdr = newrevhdr

    fp.close()
    new_fp.close()

`svn commit blog-post`

I hope someone out there finds this useful. Rewriting to understand the script above was an absolute blast. Maybe you can provide ways to improve it. I didn't spend too much time thinking about improving it.

Reusable App Conventions

Tue, 04 Nov 2008 22:04:52 -0600

One of the great strengths of Django is the vast ecosystem of reusable apps. People saw common patterns in what they were making and abstracted those out in to an app that anyone else can use in their own projects. The Pinax project, in which I am a committer, is attempting to take the ecosystem and bundle up these reusable apps into more of a deliverable. We are working on having different flavors of Pinax that will suit what you are trying to do. This also doesn't mean you couldn't pick and choose your own apps.

I want to take the time in this post to go over conventions that should be established for reusable apps and Django apps in general.

Reusable vs Non-Reusable Apps

A reusable app is meant to do one thing and do it well. The second it cross the boundary of doing more than one thing you need to question yourself whether or not what you are adding is appropriate. Should that one new feature really be contained in its own app to flourish and grow on its own?

A Story

A good case of where I really began to question myself was during the development of django-categories. I started the project a little over a year ago. This was literally just before Jonathan Buchanan started to take some ideas on the Django wiki and dump them into what we now know as django-mptt. At the time I was just learning about MPTT and how it can benefit me in storing trees in a relational database. I was tasked with the need to create categories at work so it felt like the right thing to use.

I started to write django-categories in a reusable way. I used a parent foreign key to deal with the hierarchy. It quickly became apparent that the data set I had was a bit too large for some of queries I was doing using a simple parent foreign key. I started to re-work django-categories to include the denormalization that MPTT provides. Right then django-mptt came out. I was simply attonished by how well implemented it was. It stayed out of your way.

At this point I had two choices. One, use django-mptt having a dependancy on another reusable app. Two, continue on having on my own implementation of MPTT that was really lacking. I can't say for sure that this is a case of doing more than one thing, but it certainly broad sided me (in a good way) in understanding where lines can be drawn.

I ultimately just closed the project due to my particular use case and pointed people to django-mptt. However, recently I have come across a few cases where having an app that provided a Category model, a URLconf, views and some feeds around generically relating objects to a tree is a good idea. I have since reopened the project and moving forward with it.

Drawing the lines

The case I show above was specific to my needs, these things need to be thought of on your own. If you are working on a project using Django what is out there? Does it do what I need? It is always best to just start looking.

You may already have reasons for not making an app resuable. In the Pinax project we have a few of those cases. We have three levels of apps.

External apps

Third party apps provided by the community. They are svn:external'ed in the repository.

Local apps

These are apps that we have written to fill a specific feature on our site. They are ones that can be written with many assumptions but the ideas and features they imploy can be made reusable in the future.

Project specific apps

These are esstenially "core" apps. They live at the project level. Currently, that is either in the basic_project or complete_project. These have dependancies in such way that can not make them resuable in a generic sense. They are glue between external and local apps to make up certain features.

This is ultimately going to be a decision you will need to make. I want to clarify that a resuable app does not mean it is open sourced and freely available. They are primarily a clean implementation and proper separation in your Django project.

Make an app

Creating an app in Django is simple:

manage.py startapp appname

This will give you a skeleton app that you then go and code. One thing that is left to be desired are many of the modules most apps you see have. For example some apps come with forms that you can use in your views or templates. Many people get confused on where these things should go. Some argue that Django provides a horrible starting point. There are many things that should be there, but isn't. This has been argued for/against in the past. I think the starting point Django provides you is an excellent balance between giving you everything under the sun that you might use and nothing at all. I am certainly open to improving startapp, but the default behavior is good enough. The documentation makes up for the rest. You are reading it right?

Models

Models are the foundation around accessing your data in your database. Django will create you a models.py file for you. This is where models go.

One thing that some people don't know is models can be seperated out. If you make a models module (a models directory with an __init__.py in it) you can create many other Python modules with models that belong to that app. You must specify the app name in app_label of the Meta class in all the models for this to work. I typically recommend against this. The first reason is because of the idea behind a resuable app I discussed above. If you have this many models in one app, you doing something wrong. Another issue is you begin to have too many interdepent models that end up causing circular imports.

Forms

Forms are commonly placed in a forms.py module. This includes regular forms and model forms. Putting model forms in your models.py is just the wrong placement. They are just Python classes that could live anywhere though, but that would defeat the purpose of this article.

Admin Inteface

The admin interface is not enabled by default. However, many people use it. It went through a complete refactor during its newforms-admin days. Originally the only place where you specified the admin bits was in your models. NFA fixed that by requiring you to put the new ModelAdmin declarations in an admin.py module.

Note

People who used the newforms-admin branch had to deal with many issues with the placement of ModelAdmin classes. It mostly came down to where the register calls were happening.

If you are defining ModelAdmin and making admin.site.register calls in your models.py or even explicitly importing an admin.py in the app's __init__.py stop doing that. Now.

The admin app comes with a handy function that will properly import all of your admin.py modules for you. It is called autodiscover. I have gone into depth about this in my newforms-admin migration post.

Views

Views are what deal with the request once it is routed. Django provides you with a views.py module where you have your view functions. One thing to note is that these are simply callables. What I mean in the general sense is that you are not tied to using a Python function. One thing that the admin does is it uses a class that represents many methods (aka views) that deal with the HttpRequest. This is commonly refered to as class-based views. Here is a quick example:

class MyView(object):
    def __call__(self, request):
        return HttpResponse(self.get_text())

    def get_text(self):
        return "hello world"

The above is a horrible view, but a good example. In a resuable app you can easily provide overridable behavior for users of the views. There is proposal going around in the Django world about using this approach for generic views. I am personally a big +1 to that idea. It really stemmed out of the many use cases we saw during the newforms-admin period. Views would become their own classes where they can inherit base functionality and everything is overriable.

Templates

This deserves its own post. I have one in the works about I how feel in this area and what is acceptable in a reusable app.

Template tags

Template tags are awesome. They already have a pretty well established convention due to the fact its structure is well defined:

my_super_sweet_app/
    __init__.py
    models.py
    templatetags/
        __init__.py
        my_super_sweet_app_tags.py

my_super_sweet_app/templatetags/my_super_sweet_app_tags.py is where you define your template tags. Be it inclusion tags, filters, custom tags, etc...

One thing that I have not seen addressed well enough is custom template tag syntax. There is sort of convention, but nothing very well established. Let me describe how I typically handle this:

{% images_for_object [obj] as [varname] %}

The structure is that you pluralize what the model is. In this case I have an Image model and it contains a generic foreign key. This template tag takes in an object and then builds a new context variable for you containing the data you are looking for.

The convention I am trying to establish is that template tags read nicely. The above tag uses the word 'as' to indicate that the left hand side is going to be stored in its right hand right. There are other cases where other keywords can be used to show how a template tag works:

{% latest_flights 5 for profile.user as flights %}

In the tag above we are grabbing 5 latest flights for a given user and storing them in the context variable flights. I am not saying we can make a good convention around every template tag there is, but certainly I wanted to bring it up so you too can think about how this might be good in some places of your own template tags.

You are cleared for landing

This blog post has been mostly a brain dump of many things I see through out my apps. I encourage you to implement some of these ideas and take others opinions into consideration. The Django reusable app ecosystem is going to continually get stronger. There are a couple of things I have left off this post only because I am just beginning to learn more about them myself. I will be certain to post more things about reusable apps in the near future.

Writing a custom management command

Mon, 03 Nov 2008 15:42:25 -0600

One of the neat things you can do in Django 1.0 is write custom management commands. Django gives you an API that you can use to easily write code that you can execute on the command line. We are familiar with some of the built-in ones such as syncdb and runserver. This post will cover how to create our own.

Background

Today I was tasked with the need to deploy an application in the wild on my local intranet. It was a stupid simple Django application that basically used the admin app to browse some data in a MySQL database. All this will eventually be in a PostgreSQL database running on a nice Django project. However, in the mean time I needed something quick and working.

I could use runserver and daemonize it. Ideally I wanted something that could handle multiple requests at time because there were cases when a couple users would be using this heavily at the same time. I turned to the trusty WSGI server that CherryPy provides. It is multi-threaded and super simple to setup.

Getting started

The reason why I want to blog about this is because the Django documentation on management commands is pretty sparse. It does show you the basics of where the command must go to be recongized. Inside your app you will want to have a directory sturcture similiar to this:

wsgi/
    __init__.py
    models.py
    management/
        __init__.py
        commands/
            ___init__.py
            runwsgi.py

Above is what my app looks like for the command we are going to write. As you can see there is wsgi/management/commands/runwsgi.py. This is where your command code will live.

Django provides you with some base classes you can dervive your Command from. All commands ultimately derive from django.core.management.base.BaseCommand and in this case ours will be too. However, if you have a specific task there may be a helper base class to do some work for you.

django.core.management.base.BaseCommand

django.core.management.base.AppCommand

django.core.management.base.LabelCommand

django.core.management.base.NoArgsCommand

Most of the command that I write extend from either BaseCommand or NoArgsCommand.

Writing the management command

Now that we know where to put our command in our app and what the command can extend from lets actually see something in action. Here is some code to get you started on something:

from django.core.management.base import BaseCommand

class Command(BaseCommand):
    def handle(self, *args, **options):
        # do something interesting here

The code above in the case of the layout shown before lives in runwsgi.py. The name of the file is significant by the fact it is used in how you call your management command.

Note

Your app must be in INSTALLED_APPS for Django to see the management command.

This command can be executed by running in the same directory as manage.py:

./manage.py runwsgi

Seriously, do something interesting

Ok, lets cut the crap and get down to something useful. We are writing code for the runwsgi management command. Lets see some working code here:

from django.core.management.base import BaseCommand
from django.core.handlers.wsgi import WSGIHandler

try:
    from cherrypy.wsgiserver import CherryPyWSGIServer
except ImportError:
    from wsgiserver import CherryPyWSGIServer

class Command(BaseCommand):
    def handle(self, *args, **options):
        server = CherryPyWSGIServer(("127.0.0.1", 9001), WSGIHandler())
        try:
            server.start()
        except KeyboardInterrupt:
            server.stop()

Oops. I did say CherryPy's WSGI server was simple right? The above code hooks up Django right into the CherryPy multi-threaded WSGI server and runs it. One thing that is interesting, is based on that code, we can see so many places where we can let the user override values and behavior of the process.

Accepting and using user options

One thing that I find the best part of how Django's management command are implemented is it provides you with a nice way of specifying options. These options will come in through the command line that can adjust values based on how the user wants the process to behave. Based on the code shown just earlier we can see that we might want to the user to provide the host, port and specify the ability to daemonize itself. Lets go ahead and add these options that the user can specify:

from django.core.management.base import BaseCommand
from django.core.handlers.wsgi import WSGIHandler

from optparse import OptionParser, make_option

try:
    from cherrypy.wsgiserver import CherryPyWSGIServer
except ImportError:
    from wsgiserver import CherryPyWSGIServer

class Command(BaseCommand):
    option_list = BaseCommand.option_list + (
        make_option("-h", "--host", dest="host", default="127.0.0.1"),
        make_option("-p", "--port", dest="port", default=9001),
        make_option("-d", "--daemon", dest="daemonize", action="store_true"),
    )

    def handle(self, *args, **options):
        server = CherryPyWSGIServer((options["host"], options["port"]), WSGIHandler())
        try:
            server.start()
        except KeyboardInterrupt:
            server.stop()

Django provides a quick a simple way to use Python's optparse module to define how you want your command to accept options. We have added options for host, port and daemonize. I have hooked up the host and port when we create the CherryPyWSGIServer. I will touch on daemonize later.

One thing I want to draw some attension to is specifically the -h option. optparse defines -h for help. If you try to run the above code you will get an exception indicating there is a clash. To fix this you need take control of how Django create the option parser:

from django.core.management.base import BaseCommand
from optparse import OptionParser, make_option

class Command(BaseCommand):
    # ...

    def create_parser(self, prog_name, subcommand):
        """
        Create and return the ``OptionParser`` which will be used to
        parse the arguments to this command.
        """
        return OptionParser(prog=prog_name, usage=self.usage(subcommand),
            version = self.get_version(),
            option_list = self.option_list,
            conflict_handler = "resolve")

    # ...

The part of significance above is that we are passing in conflict_handler to OptionParser. This lets the parser handle the collision intelligently. You can read about this on Python documentation for conflicts between options.

Daemonizing the process

One small thing I want to touch on is how you can daemonize this Python process. Right now it will be attached to the shell and wont be very useful for long periods of time. I have spent some time with Twisted so I came across this neat snippet of code:

def daemonize():
    """
    Detach from the terminal and continue as a daemon.
    """
    # swiped from twisted/scripts/twistd.py
    # See http://www.erlenstar.demon.co.uk/unix/faq_toc.html#TOC16
    if os.fork():   # launch child and...
        os._exit(0) # kill off parent
    os.setsid()
    if os.fork():   # launch child and...
        os._exit(0) # kill off parent again.
    os.umask(077)
    null = os.open("/dev/null", os.O_RDWR)
    for i in range(3):
        try:
            os.dup2(null, i)
        except OSError, e:
            if e.errno != errno.EBADF:
                raise
    os.close(null)

Just put that in your runwsgi.py file and call it when you need it. You can do this conditionally based on the option being passed it as well. I will leave that bit as an exercise to you.

Coming to an end

I have pretty much covered everything I did to write this management command. It was really simple and we got some usable code out of it. I did leave you with an exercise if you did want to implement this command, but I like pulling code from others just as much as you probably do it is available on my blog's source code. You can find it here. Please ask questions or provide better examples if you so wish. I would love to hear your opinions and see your code.

Standalone Django App Testing

Sun, 02 Nov 2008 14:46:38 -0600

Yesterday I found myself hacking on a Django app. I am working on reviving my old django-categories app. More about that later. One thing I came across pretty quickly was the need to test the app. I had written tests way back when and wanted to ensure my new code worked with the API users were expecting. That is if there are actually any users, but mostly because this is the API I am wanting.

To test the app I basically needed a settings.py. I have a simple one living on my PYTHONPATH that I use regularly for running the Django test suite. However, it is not sufficient enough. It simply sets the DATABASE_ENGINE and ROOT_URLCONF settings. The latter setting is a requirement by Django. The requirement for running these tests involves having the app in INSTALLED_APPS which would defeat the purpose of the simple settings for running the test suite. I needed something to be a bit more robust.

Solving the problem

I was talking to Eric Holscher on IRC and I came up with the idea of having a standalone script that allowed some settings values to be passed in over the command line. The goal was to test the app. You can point it to the app and it will setup sys.path and INSTALLED_APPS for you. To point it at the app you would provide a path to the app:

app_test_runner.py /path/to/my/app

This is nice because you don't have to make sure the app lives on the PYTHONPATH allowing for quick testing. What you do need to ensure is that any dependancies the app has lives on the PYTHONPATH.

The other bit that is of importance is the ability to easily change some common Django settings. You can easily change the database settings right over the command line:

app_test_runner.py --DATABASE_ENGINE=postgresql_psycopg2 \
    --DATABASE_NAME=dbname \
    --DATABASE_USER=joe \
    --DATABASE_PASSWORD=password \
    /path/to/my/app

Pretty simple!

Give me some code

The code is available on GitHub. I did this so that it can continue to get better. The only thing I ask of future contributors is that it stays super simple. The script is pretty small so I would like to paste it here for convenience:

import os
import sys

from optparse import OptionParser

from django.conf import settings
from django.core.management import call_command

def main():
    """
    The entry point for the script. This script is fairly basic. Here is a
    quick example of how to use it::

        app_test_runner.py [path-to-app]

    You must have Django on the PYTHONPATH prior to running this script. This
    script basically will bootstrap a Django environment for you.

    By default this script with use SQLite and an in-memory database. If you
    are using Python 2.5 it will just work out of the box for you.
    """
    parser = OptionParser()
    parser.add_option("--DATABASE_ENGINE", dest="DATABASE_ENGINE", default="sqlite3")
    parser.add_option("--DATABASE_NAME", dest="DATABASE_NAME", default="")
    parser.add_option("--DATABASE_USER", dest="DATABASE_USER", default="")
    parser.add_option("--DATABASE_PASSWORD", dest="DATABASE_PASSWORD", default="")
    parser.add_option("--SITE_ID", dest="SITE_ID", type="int", default=1)

    options, args = parser.parse_args()

    # check for app in args
    try:
        app_path = args[0]
    except IndexError:
        print "You did not provide an app path."
        raise SystemExit
    else:
        if app_path.endswith("/"):
            app_path = app_path[:-1]
        parent_dir, app_name = os.path.split(app_path)
        sys.path.insert(0, parent_dir)

    settings.configure(**{
        "DATABASE_ENGINE": options.DATABASE_ENGINE,
        "DATABASE_NAME": options.DATABASE_NAME,
        "DATABASE_USER": options.DATABASE_USER,
        "DATABASE_PASSWORD": options.DATABASE_PASSWORD,
        "SITE_ID": options.SITE_ID,
        "ROOT_URLCONF": "",
        "TEMPLATE_LOADERS": (
            "django.template.loaders.filesystem.load_template_source",
            "django.template.loaders.app_directories.load_template_source",
        ),
        "TEMPLATE_DIRS": (
            os.path.join(os.path.dirname(__file__), "templates"),
        ),
        "INSTALLED_APPS": (
            # HACK: the admin app should *not* be required. Need to spend some
            # time looking into this. Django #8523 has a patch for this issue,
            # but was wrongly attached to that ticket. It should have its own
            # ticket.
            "django.contrib.admin",
            "django.contrib.auth",
            "django.contrib.contenttypes",
            "django.contrib.sessions",
            "django.contrib.sites",
            app_name,
        ),
    })
    call_command("test")

if __name__ == "__main__":
    main()

Known issues

One issue I came across while working on this bit of code was likely a bug in Django. In Django there is a unit test called ChangePasswordTest that uses setUp and tearDown to setup the TEMPLATE_DIRS to get the login.html template. Well, this overrides the whole TEMPLATE_DIRS and doesn't allow user defined values there. This creates a case where when django.contrib.admin is not in INSTALLED_APPS you will get test failures. I haven't had a terrible amount of time to look into this. I have temporarily patched this by including the admin app in INSTALLED_APPS.

oebfare: Latest Entries

Gitify Python SVN

Going local

Gitify

Getting our hands dirty

git filter-branch is really poweful

Ta da!

Git Tab Complete

Book Memery

Multiple Foreign Keys to Same Model for Inlines

Side Note

App Settings

Looking at project settings

Naming conventions

Getting the values and dealing with optional settings

Wrapping up

Essentials

Moving away from Google Code

Step One

Step Two

svn commit blog-post

Reusable App Conventions

Reusable vs Non-Reusable Apps

A Story

Drawing the lines

External apps

Local apps

Project specific apps

Make an app

Models

Forms

Admin Inteface

Views

Templates

Template tags

You are cleared for landing

Writing a custom management command

Background

Getting started

Writing the management command

Seriously, do something interesting

Accepting and using user options

Daemonizing the process

Coming to an end

Standalone Django App Testing

Solving the problem

Give me some code

Known issues

`git filter-branch` is really poweful

`svn commit blog-post`