Good job, Brian Rosner et al.

At first, I thought they’d eventually get around to it and that is the best I could hope for as surely they are too busy to respond to a non-paying customer like me. Then I learned they the used the Lighthouse ticket system to manage their project and took public submissions for features.

Before logging a feature request I did some searching and found a ticket already out there requesting exactly this same feature that I wanted. So I just added my own comments to the ticket to give it a +1.

After a little back and forth I learned it was implemented tonight (just a week after my first comment). Good to see such responsive development.

An example of this feature can be found in my django-aws project which is simple app to provide some template tags to provide Amazon Web Services information a la boto.

The Huck PAC is continuing with Mike Huckabee’s brand of positive, issues-based, politics, in promoting Vertical Day on July 23rd. Candidates such as John Cornyn, Elizabeth Dole, Lamar Alexander, Jim Inhofe, Roy Brown, Steve Daines, Gilbert Baker and Bob Clegg, have chosen to participate. They’ll blog about issues on July 23rd in a positive manner that promotes good healthy exchange of ideas instead of the typical partisan bickering.

As a coordinated effort on a single day, it should bring a more attention to Vertical Politics than having this effort scattered about. That being said, it would be nice if this set a tone for encouraging more candidates to seek to solve problems than to merely beat up the other guy for votes.

We have a ton of problems to solve as a nation. We don’t have time to deal with the status quo of politics in America. We need a fundamental shift in attitude and thinking. I am looking forward to Vertical Day and hopefully one day a Vertical Nation.

I had thought that there was but it turns out I was just mistaken in the object hierarchy. InlineModelAdmin inherits from BaseModelAdmin the same as ModelAdmin — I was thinking that InlineModelAdmin inherited from ModelAdmin.

Therefore, I had to determine a way to accomplish this via some spelunking through the code. At first, I thought I’d just create my own template by inheriting or copying the tabular.html template. After looking at it, I determined that I didn’t want to figure out how to do a regroup on inlineadminformset or if even that was the proper way to try to order things in that template.

After a few minutes in the code I realized that I could just subclass BaseInlineFormset and specify the fields that I wanted to order by (in this example I will use starttime and endtime:

from django.contrib import admin
from django.newforms.models import BaseInlineFormset

class MyOrderedFormset(BaseInlineFormset):
    def get_queryset(self):
        qs = super(SessionInlineFormset, self).get_queryset()
        return qs.order_by('start_time', 'end_time')

class MyOrderedInline(admin.TabularInline):
    model = MyModel
    extra = 1
    formset = MyOrderedFormset

That’s it. Now after that section of inlines are ordered by starttime and endtime.

Managing database changes in a team environment working on a django project can be complicated. I would imagine that there is no one size fits all solution and it would depend on team size and configuration, production database size, etc.

What I will outline here is a solution I developed for a small team that I work with on a django based web application. So far it has worked really well for allowing us to streamline database changes so we can stay in sync, deploy easily, and add small tweaks to the database (indexing, data manipulation, etc.) that is semi-automated.

The idea is based partly on Rails database migrations (though not has sleek and well put together) and a database versioning system that was used on a team that I have worked on previously using MS SQL in a corporate environment and a fairly decent team size. In one sense it is not nearly as well put together as either of these two solutions, but at the same time, it is pretty much hands off and has been working well for us for a number of months now.

Why not just use syncdb?

As nice as python manage.py syncdb is when the project is getting started and is still in early phases, it becomes less and less useful, mostly in handling schema changes. Since syncdb only creates a model if it doesn’t already exist in the database, we found ourselves outputting sql with python manage.py sql APP_NAME and then making adhoc ALTER TABLE scripts from the CREATE statements and passing them around to each other and then trying to remember the order to apply them in production.

Furthermore, syncdb left us without a way to alter data, add indexes, remove indexes, etc. Granted that wasn’t it’s intent and I am certainly not trying to beat up syncdb — it does what it does well.

The Solution

In order to address this, it seemed the most natural thing was to have some structured and ordered way to write SQL scripts that were applied in a uniform manner so that it was repeatable. This way we could run on our development databases, on test databases, and in deployment on production all the same way.

First step to achieve this was to version the database. This is accomplished with a simple table that will keep track of what it has executed along with some other metadata primarily for reference purposes.

CREATE TABLE `versions` (
    `version` VARCHAR(200) NOT NULL, 
    `date_created` DATETIME NOT NULL,
    `sql_executed` LONGTEXT NULL,
    `svn_version` int null
);

In order to populate this table and keep versions consistent, the naming scheme of the sql scripts should follow something that lends itself to easy sorting:

YYYYMMDD-##.sql

where pound signs are zero-padded integers of the changes for the day. In a team environment, it is probable that two people might be working on a database change at the same time and therefore would be use 01. Whoever committed first would be able to keep the 01, the other developer would get a conflict message and need to rename his script to 02.

These scripts live in a db/ folder in our project. The name of this folder is not important, but it is important to keep all the sql scripts in a folder not mixed in with other files.

To tie it all together there is s simple python script that gets the list of applied changes from the database, gets a list of files in the current directory in order, filters out the ones that have been applied and then processes the remaining scripts. In processing the scripts, it opens each script and splits script down into individual statements (splitting on the semicolon). After execution of an entire script file is complete, the versions table is updated to reflect the applied version.

Running python upgrade.py by itself will simply print to standard output all the statements it plans to execute, so that they can be reviewed. Running python upgrade.py --execute will actually execute the scripts.

If an error was found in executing a statement, processing stops immediately.

You can find upgrade.py in Django Snippet 849.

Improvements

There are lots of improvements to this script that I’d like to add if I got around to it. The ones I’d like to see most is support for “rollbacks”. Comments and suggestions are most welcome!

I’ve recently (past week or so) been digging into the newforms-admin branch of django. I am really looking forward to this code line getting merged into trunk as there are tons of great work in this branch. I especially like the separation of the admin code from the models.

Well, I’ll get to my point.

The Problem

I needed to be able to edit child records in the admin that were children through the use of GenericRelations. I was told about a solution on Google Code called django-genericadmin, but it seemed dead and I could not get it to work.

The thing that seemed the closest to working was a patch on ticket 4667 by Honza Kral and a variation on this patch in the Django Snippet 765. Neither of these worked for r7771 of branches/newforms-admin.

The Solution

I put the 765 snippet in a file called generic.py in the root of the django project I was working on and fiddled with it until I got it working. I ended up needing to change an import, fix some variable names, and change the argument list order in one of the init methods. I also added the canorder and candelete options to the GenericInlineModelAdmin class.

After I got it working isolated in generic.py, I then added the code to the django/contrib/admin/options.py file and created a new patch (after testing locally of course). This patch has since been added to ticket 4667. I also posted the full version that I had running in generic.py in snippet 832 in case someone wants to use it without patching their newforms-admin branch.

Example of How to Use

In order to get a sense for how to make use of this functionality, imagine you have a songs app that you use to model and record details about songs, you have built this generically so you can reuse it wisely in a number of different projects. One project in particular has a need to to reference song data.

You are building a home media inventory system and decide you want to catalog both your albums as well as your DVD collection, however, you want the ability to record what songs exist on both your albums as well as your DVDs as you are a fanatic about soundtracks.

So you have the following models (abbreviated for clarity):

# Media Collection App
class Album(models.Model):
    ...

class DVD(models.Model):
    ...

# Song App
class Song(models.Model):
    ...

Since you want to enable your Song model to relate to two other models generically, you decide to use Generic Relations like so:

from django.contrib.contenttypes.models import ContentType
from django.contrib.contenttypes import generic

class Song(models.Model):
    ...
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey()

and then on the models that you want to have 0 to Many child records that are songs:

from myproject.song.models import Song
from django.contrib.contenttypes import generic

class Album(models.Model):
    ...
    songs = generic.GenericRelation(Song)

class DVD(models.Model):
    ...
    songs = generic.GenericRelation(Song)

Now that’s all you need to do if you didn’t care about editing the data in the admin, however, since that is not the point of the article, here is what you need to do in your admin.py in the app that contains Album and DVD:

...
from myproject.song.models import Song
from myproject.generic import GenericTabularInline
### OR ###
### from django.contrib.admin.options import GenericTabularInline

class SongInline(GenericTabularInline):
    model = Song
    extra = 2 
    ct_field_name = 'content_type'
    id_field_name = 'object_id'

class AlbumOptions(admin.ModelAdmin):
    model = Album
    inlines = (SongInline,)

class DVDOptions(admin.ModelAdmin):
    model = DVD
    inlines = (SongInline,)

admin.site.register(Album, AlbumOptions)
admin.site.register(DVD, DVDOptions)

Now you should be able to add/remove songs when editing an Album or DVD in your admin.

Update: Fixed typo in code sample above in setting the GenericForeignKey() — thanks FunkyBob!

Because at the end of the day, as much as I want to play like Tiger, I want to be like Rocco.

Here’s to hoping the drive to be the best doesn’t compromise the need to be ourselves.

I completely agree and thought that was well put. Read the rest of his post to put a little context to that quote. I admire both qualities — to be the very best in those things you find yourself doing, as well as, the character of humbleness and complete self-awareness.

While Gore is raking in the awards and cash, he apparently doesn’t practice what he is preaching to everyone else. He was busted last year on his energy consumption levels so he raced out and renovated his house to be more green. After that investment last year, he actually is pulling 10% more energy off the grid than he was before the solar panels. So much so, it comes out to 20 times the average household.

Yes, I worked with the svnmerge.py hack for a couple of months, but it was introducing strange merge bugs and just generally left me uncomfortable trusting to do all its magic (relying on certain property values to guide its merge operations).

In light of going back to the old style svn merge, where I relied on comments to tell me the last merge point, I wanted to make sure that the log messages were standardized. Without some tool to help me do this, I was spending most of the merge time formatting comments and they ended up not being a reliable standard.

Taking advantage of the ability to output Subversion log messages into xml, I wrote a quick python script to generate single line summaries of the revision history:

# slfmt.py LOGFILE
from xml.dom.minidom import parse
import sys

doc = parse(sys.argv[1])

revisions = doc.getElementsByTagName('logentry')

for rev in revisions:
    r = rev.getAttribute('revision')
    try:
        msg = rev.getElementsByTagName('msg')[0].childNodes[0].data.split('\n')[0]
        print 'r%s - %s' % (r, msg)
    except IndexError:
        print 'r%s - NO LOG MESSAGE' % r

Then I wrapped up several other commands I find myself constantly running into a single bash script:

#!/bin/bash
# prepmerge REV BRANCH
REPO_ROOT=REPLACE_WITH_YOURS

svn merge -r$1:HEAD $REPO_ROOT/$2
echo "Merged r$1:HEAD from $2 into "`svn info | grep URL | awk '{ print $2 }'` > .merge-comment
echo "" >> .merge-comment
echo "  svn info:" >> .merge-comment
svn info $REPO_ROOT/$2 | awk '{ print "\t\t"$0 }' >> .merge-comment
echo "  change log:" >> .merge-comment
svn log --xml -r$1:HEAD $REPO_ROOT/$2 > .log.xml
python ~/bin/slfmt.py .log.xml | awk '{ print "\t\t"$0 }' >> .merge-comment
echo "" >> .merge-comment
echo "  files:" >> .merge-comment
svn st | grep -v "?" | awk '{ print "\t\t"$0 }' >> .merge-comment

With these two scripts in my ~/bin directory and assuming that my repo follows the standard /trunk, /branches, /tags, model, I can now do merges in three simple steps. Assuming you would like to merge changes from a branch called experimental from HEAD into trunk, you would need a local working copy of trunk and be in the the root of that local path and then:

• ~/bin/prepmerge HEAD branches/experimental
• review changes, resolve conflicts
• svn commit -F .merge-comment

That’s it. I am sure there are better/cleaner methods to this, but this now allows me to focus solely on resolving conflicts that emerge instead of fiddling with comments. Of course, if everything I worked on was on git, I wouldn’t have to fool with these types of scripts.

Hanoi crazy night traffic from v!Nc3sl4s on Vimeo.