Justin Lilly: Latest Entries

Piping stdout when using Buildbot

Wed, 08 Jul 2009 12:31:23 -0400

Just a quick tidbit for anyone googling like I have been all morning:

If you're attempting to pipe a command in a buildbot script, you need to issue it as a single string, not a list of strings. By using a single string, buildbot uses a shell, otherwise it doesn't. The > necessary for output redirection is a shell thing, which will only work in this method.

To recap:

f1.addStep(ShellCommand(command=['/home/www/bin/django', 'build_solr_schema', '>', '/etc/solr/conf/schema.xml'],
                    description=['building', 'schema'], descriptionDone=['schema','built']))

becomes:

f1.addStep(ShellCommand(command="/home/www/bin/django build_solr_schema > /etc/solr/conf/schema.xml",
                    description=['building', 'schema'], descriptionDone=['schema','built']))

6 things I learned about setuptools

Sun, 05 Jul 2009 13:29:31 -0400

While building pyvcs and django-vcs over 4th of July weekend with Alex Gaynor [ announcement ], there were a few interesting lessons learned, mostly revolving around how setuptools works. As Jacob mentions in his post on buildout, you mostly have to cobble together the internals of setuptools from other setups. This works well at about 90% of the case. There are a few gotchas, however.

find_packages('directory') will look under directory for a list of python modules to install. For our applications, we actually just wanted find_packages()
You can't easily specify multiple authors for a project. We settled on author = "First Author, Second Author"
Its a good idea to version target your requirements. Don't require "pyvcs", require "pyvcs>=0.1".
If your project contains django templates, set the zip_safe property to False.
The setup.py commands of register or sdist upload don't actually check that your project installs. For that, its likely better to use a virtualenv and test the installation.
package_data is used to includes docs (or in our case, templates) with the project.

This serves mostly as future reference for me, but hopefully helps someone else navigating the maze that is setuptools.

Using Database Views in Django

Sat, 20 Jun 2009 11:35:31 -0400

Database Views will make their appearance in the upcoming Django 1.1 release. Having used them for my latest project, there are a few gotchas when it comes to using them.

In our project, we were using South for our database migrations. To ensure our database views were created, we added them to a south migration using db.execute. This worked very well for 90% of our usecase. The biggest hurdle for this ended up being testing. When syncdb runs, your unmanaged models (ie: database views) aren't generated. There were a few options available to us at this point.

Provide a replacement for django.test.TestCase which handles our DB View setup / teardown. The biggest benefit of this approach is it stays neatly out of the way. It also allows us to further alter TestCase if we have a need. In the end, we chose not to do this as it felt a bit sloppy and we ran the risk of breaking something if django TestCase were to change.

Another option was to use the post-syncdb signal to create the models. This approach also stays neatly out of the way. The downsides here were that it would conflict with South already generating our database view as well as it was a bit bigger blanket than we needed. Our problem was only for running tests. That said, if we weren't using South to manager our database options, I think this would have been the route we'd have gone.

Instead, we opted to provide a function for setup & teardown which handles the view creation. These functions are explicit and fix the issue exactly where we need it. The downside here is it seems like an extra step when creating tests for an app which would be nice not to have to go through.

The other big gotcha for database views is that you can't delete from them. This means that due to django's cascading delete's, you may recieve errors when attempting to delete an object that has references to a database view. The proper way to handle this is to invalidate the foreign keys that an object has to this database view before deleting it. Setting the ForeignKey to None before deletion will ensure that it actually works. You can follow the process of the resulting ticket for this at #10829.

Overall, I feel the work that has gone into db views has been amazing. When you need one, they're exactly what you need. Here's hoping the above helps you avoid those gotchas.

EuroDjangoCon: Django @ 30,000 feet

Mon, 04 May 2009 11:37:23 -0400

So I gave my talk related to django from the ideas of a manager at EuroDjangoCon. It was a blast and thanks to all of those who were present and asked questions. You can find my slides below.

Customizing Django Forms with Multiwidget

Mon, 27 Apr 2009 18:11:16 -0400

Before my talk at django-nyc tonight, I thought it might be good to post some slides so I can point people at where to find the code and slides from tonight. I'll come back and flesh out a text version of the talk, but for now, you can find a code repository on github and the slides on google.

Virtualenv Wrapper Helper

Sat, 28 Mar 2009 23:57:19 -0400

A bit inspired by all the awesome stuff happening at PyCon, I've done a bit of hacking on my shell to improve the use of virtualenv_wrapper. Remembering to checkout the correct virtualenv is a bit of a pain, especially when you move around your filesystem as much as I do. To fix this, I've written a bit of code for my bash prompt to ease the pain.

It works by looking for the existance of a .venv file every time you cd into a directory. If it exists, it will attempt to workon the contents of the file, which should be the project name. For instance, for my oebfare project (more news forthcoming on that), the file contains the words "oebfare" as that's the name of my virtualenv. Due to this, you can get general awesomeness such as the following:

# prompt edited to save space
12:02 AM: (~/Code/django) jlilly$ cd baconfile/
(baconfile):12:03 AM: (~/Code/django/baconfile) (master) jlilly$ cd ..
(baconfile):12:03 AM: (~/Code/django) jlilly$ cd oebfare/
(oebfare):12:03 AM: (~/Code/django/oebfare) (master) jlilly$

When we cd into the baconfile dir, we get the virtualenv for baconfile. We can cd out of that directory and about the filesystem, but when we enter a directory that contains a different .venv file, we get a different virtualenv.

NameError: global name 'log' is not defined

Sun, 22 Mar 2009 00:49:35 -0400

I've spent a portion of this evening trying to get caught up with the crazy amounts of development that have been going on with the oebfare blogging engine over the past few weeks. I have a nice little ubuntu VM setup that works really nicely, but when gathering the requirements for the projects, I hit a snag. I apparently had a borked setuptools (installed via apt) which resulted in the error:

NameError: global name 'log' is not defined

To fix this, find your setuptools location. For me, it was relatively easy as I'm using virtualenvwrapper so it was located in ~/.virtualenvs/oebfare/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg

After a brief google search, I came across a patch which seems to fix the issue, which you can find here. To apply the patch, rename your .egg file to a .zip, extract it, patch the repository, zip it back up, rename it, and you should be set. Here are the commands I ran to do that.

321  mv setuptools-0.6c8-py2.5.egg setuptools-0.6c8-py2.5.zip
322  extract setuptools-0.6c8-py2.5.zip   # alias to extract an archived file
323  cd setuptools/
324  wget http://bugs.python.org/file10198/setuptools_patch.txt
325  patch -p0 < setuptools_patch.txt # this one didn't work. I was off by 1
326  patch -p1 < setuptools_patch.txt
327  ..  # alias to cd ..
328  rm setuptools-0.6c8-py2.5.zip
329  zip setuptools setuptools-0.6c8-py2.5.zip  # I never remember the proper order
330  zip setuptools-0.6c8-py2.5.zip setuptools
331  mv setuptools-0.6c8-py2.5.zip setuptools-0.6c8-py2.5.egg

Hope it helps someone!

psycopg2 InternalError ?

Fri, 13 Mar 2009 11:28:38 -0400

For those who've hit the following complaint with django (like I just did):

psycopg2.InternalError: current transaction is aborted, commands ignored until end of transaction block

You need to set up PostGIS and ensure that the table you're looking at has been created from the template_postgis template. Just a warning, check your database logs for more info.

A better look for Comments

Tue, 10 Feb 2009 00:43:24 -0400

I've been in the process of unforking my code from brosner's oebfare blogging engine. In doing so, I've decided to remove some of the pain points along the way.

For me, a big one is how comments are represented in the admin. The content-type, object-id paradigm make a ton of sense on the back end of things, but when it comes to seeing which comments map to which posts, it leaves a lot to be desired. By overriding the ModelAdmin for the comments app, its simple to include the name of the object being commented on.

admin.site.unregister(Comment)

class CommentsDisplayGenericObjectAdmin(CommentsAdmin):
    list_display = ('name', 'commented_object', 'ip_address',
                    'submit_date', 'is_public', 'is_removed')

    def commented_object(self, obj):
        return '<a href="%s">%s</a>' % (self.commented_obj_url(obj.content_object),
                                        obj.content_object)
    commented_object.allow_tags=True

    def commented_obj_url(self, obj):
        # hackish, needs to actually do a reverse lookup
        return '/admin/%s/%s/%s/' % (obj._meta.app_label,
                                     obj._meta.module_name,
                                     obj.id)

admin.site.register(Comment, CommentsDisplayGenericObjectAdmin)

And there you have it. A much simpler view in the admin.

UPDATE: Missed an attribute on the commented object which shortens code a bit. Thanks David

Django App Engine Doc Fix

Fri, 06 Feb 2009 23:01:09 -0400

For those setting up Django on Google App Engine on version after the signals refactor, the following fix is needed for the code supplied by Google.

# Log errors.
django.dispatch.dispatcher.connect(
   log_exception, django.core.signals.got_request_exception)

# Unregister the rollback event handler.
django.dispatch.dispatcher.disconnect(
    django.db._rollback_on_exception,
    django.core.signals.got_request_exception)

becomes:

# Log errors.
django.dispatch.Signal.connect(
   django.core.signals.got_request_exception, log_exception)

# Unregister the rollback event handler.
django.dispatch.Signal.disconnect(
    django.core.signals.got_request_exception,
    django.db._rollback_on_exception)