Justin Lilly: Latest Entries

A fix for your fixtures and your models subdir

Tue, 27 Oct 2009 21:28:32 -0500

For those of you who keep your models in a subdirectory, you may have noticed that you can't properly load fixtures. This is because Django checks for the location of your models file to determine your app directory. To fix this, move your models directory to something like modeldir and create a models.py which imports the models from inside.

Before:

myapp
|__ views.py
|__ __init__.py
|__ models/
|____ __init__.py
|____ mymodel1.py
|____ mymodel2.py

After:

myapp
|__ views.py
|__ __init__.py
|__ modeldir/
|____ __init__.py
|____ mymodel1.py
|____ mymodel2.py
|__ models.py

Where the contents of models.py is:

from myapp.modeldir.mymodel1 import *
from myapp.modeldir.mymodel2 import *

Some folks say that if you have too much code for a single models file, you need to split it into an entirely different application. I think they have a point, but for my particular use case, it just happened that we have one model with a bunch of business rules attached which makes for difficult to scroll code.

UPDATE: A few people asked why not just put the imports into the init.py? That was my original approach, which led to the issues. For fixture loading, django attempts to import models, which returns myapp/models/init.py. It then gets the directory of this module, myapp/models/ and attempts to find a fixtures directory below that, or myapp/models/fixtures. To circumvent this, you have to make a models.py file which imports from the other directory.

A better way to handle settings files

Fri, 23 Oct 2009 17:12:19 -0500

At work, we tend to have several people working on a project at once. When its time to deploy, we deploy to several servers at once. One pain point that we've come across is the ability to manage all of the settings files for these servers. My fix for it is here.

The main piece of this is centered around the HOSTMAP dict. It maps a role to a hostname. Roles as we currently have them defined are based on their locations. Staging servers have a specific "recipe", local development has a different one. I could also see these being used for features as well. We tend not to go through the hassle of setting up SOLR for the client-side developers. For that, we could set up a 'haystack_enabled' role and include the hostnames of the folks w/ SOLR installed.

One decision here was whether to list the roles as keys or the hostnames as keys. I thought it was more important to see who has a particular configuration rather than what configuration a single person has.

Google account suspended: A post mortem

Fri, 07 Aug 2009 00:11:12 -0500

A few weeks back, I had my Google account disabled. If you're anything like me, your first reaction is a lackadaisical, "That sucks." Then the realization of what that really means starts to creep in to your gut, making you nauseous. No more email. No more iGoogle homepage. No more feed reader. Next year's vacation spots you had mapped out in a comparison chart on Google Docs? Yep, those are gone too. More Nausea.

This was a window into what I was feeling on July 13th when my Google account was suspended. I made a few shouts on Twitter [1][2][3], messaged a few friends on IRC to see if they could help. After an amazing outpour of support from friends both known and otherwise, my account was reactivated on July 15th.

For those keeping track, that's 2 days without access to personal email. Even worse than that, any attempts to email my address returned an error stating my account was suspended. This wasn't a huge deal for me, but it was definitely a time to do a gut check.

For those of you who, like me, rely on Google for a large number of your daily activities, there are a few things you should really check into, so you aren't wondering "What's Next?" like I was.

Are you backing up your email?

While gmail isn't likely to go under, when your account is disabled, you no longer have access to your 6 years (in my cases) of email. I think we often forget just how much information is in old emails. The document you're supposed to get notarized for health insurance is backlogged and starred from 2 months ago. The proof that people owed me money for some service rendered. Correspondence with a family member who's died and you'll not talk to again. poof All gone.

What about the rest of your stuff?

If you keep pertinent information in Google Docs, Google Reader, Google Analytic, AdSense, etc.. you should ensure you're keeping a separate record of it all, else you could possibly lose access to that as well.

Set up an email account outside of gmail.

If you want, this can merely serve as a front to your gmail account. The important thing here is that if something goes wrong and you lose access to gmail, you can redirect the email somewhere where you can access it.

The basic idea here is that you likely depend on google for a lot. For those of you who rely on the new Voice service, even more so. Its worth taking an evaluative look at which services you depend on and either make peace with the fact that Google owns your life, or provide backup and alternatives... or both.

While there was no mention from Google as to why my account was suspended, I've surmised that it was due to a typo in a fetchmail script which was causing it to update too often.

My first time using TipJoy

Tue, 21 Jul 2009 11:26:47 -0500

So I just used tipjoy for the first time. A few thoughts.

I like that they allow me to pay people without first having to dole out my credit card. This encourages me to use the service without having to pull my card out each time. I think this could lead to more (compulsive) spending.

I don't like that their "cash-out" system was down.

I like that they immediately followed me on twitter. It seems like a legitimate way to interact with customers when they link their twitter accounts.

I like that they didn't require I input an email. I linked my twitter account using OAuth (which I also like), and they are willing to accept that as the primary method of contact.

I could definitely see this as being a service for small goals. "$0.25 every time you come late to a meeting" or something along those lines. Even for weightloss or similar things. $1 per pound per week to the person you're competing with. Overall, I really like this service and could definitely see using it more.

Piping stdout when using Buildbot

Wed, 08 Jul 2009 12:31:23 -0500

Just a quick tidbit for anyone googling like I have been all morning:

If you're attempting to pipe a command in a buildbot script, you need to issue it as a single string, not a list of strings. By using a single string, buildbot uses a shell, otherwise it doesn't. The > necessary for output redirection is a shell thing, which will only work in this method.

To recap:

f1.addStep(ShellCommand(command=['/home/www/bin/django', 'build_solr_schema', '>', '/etc/solr/conf/schema.xml'],
                    description=['building', 'schema'], descriptionDone=['schema','built']))

becomes:

f1.addStep(ShellCommand(command="/home/www/bin/django build_solr_schema > /etc/solr/conf/schema.xml",
                    description=['building', 'schema'], descriptionDone=['schema','built']))

6 things I learned about setuptools

Sun, 05 Jul 2009 13:29:31 -0500

While building pyvcs and django-vcs over 4th of July weekend with Alex Gaynor [ announcement ], there were a few interesting lessons learned, mostly revolving around how setuptools works. As Jacob mentions in his post on buildout, you mostly have to cobble together the internals of setuptools from other setups. This works well at about 90% of the case. There are a few gotchas, however.

find_packages('directory') will look under directory for a list of python modules to install. For our applications, we actually just wanted find_packages()
You can't easily specify multiple authors for a project. We settled on author = "First Author, Second Author"
Its a good idea to version target your requirements. Don't require "pyvcs", require "pyvcs>=0.1".
If your project contains django templates, set the zip_safe property to False.
The setup.py commands of register or sdist upload don't actually check that your project installs. For that, its likely better to use a virtualenv and test the installation.
package_data is used to includes docs (or in our case, templates) with the project.

This serves mostly as future reference for me, but hopefully helps someone else navigating the maze that is setuptools.

Using Database Views in Django

Sat, 20 Jun 2009 11:35:31 -0500

Database Views will make their appearance in the upcoming Django 1.1 release. Having used them for my latest project, there are a few gotchas when it comes to using them.

In our project, we were using South for our database migrations. To ensure our database views were created, we added them to a south migration using db.execute. This worked very well for 90% of our usecase. The biggest hurdle for this ended up being testing. When syncdb runs, your unmanaged models (ie: database views) aren't generated. There were a few options available to us at this point.

Provide a replacement for django.test.TestCase which handles our DB View setup / teardown. The biggest benefit of this approach is it stays neatly out of the way. It also allows us to further alter TestCase if we have a need. In the end, we chose not to do this as it felt a bit sloppy and we ran the risk of breaking something if django TestCase were to change.

Another option was to use the post-syncdb signal to create the models. This approach also stays neatly out of the way. The downsides here were that it would conflict with South already generating our database view as well as it was a bit bigger blanket than we needed. Our problem was only for running tests. That said, if we weren't using South to manager our database options, I think this would have been the route we'd have gone.

Instead, we opted to provide a function for setup & teardown which handles the view creation. These functions are explicit and fix the issue exactly where we need it. The downside here is it seems like an extra step when creating tests for an app which would be nice not to have to go through.

The other big gotcha for database views is that you can't delete from them. This means that due to django's cascading delete's, you may recieve errors when attempting to delete an object that has references to a database view. The proper way to handle this is to invalidate the foreign keys that an object has to this database view before deleting it. Setting the ForeignKey to None before deletion will ensure that it actually works. You can follow the process of the resulting ticket for this at #10829.

Overall, I feel the work that has gone into db views has been amazing. When you need one, they're exactly what you need. Here's hoping the above helps you avoid those gotchas.

EuroDjangoCon: Django @ 30,000 feet

Mon, 04 May 2009 11:37:23 -0500

So I gave my talk related to django from the ideas of a manager at EuroDjangoCon. It was a blast and thanks to all of those who were present and asked questions. You can find my slides below.

Customizing Django Forms with Multiwidget

Mon, 27 Apr 2009 18:11:16 -0500

Before my talk at django-nyc tonight, I thought it might be good to post some slides so I can point people at where to find the code and slides from tonight. I'll come back and flesh out a text version of the talk, but for now, you can find a code repository on github and the slides on google.

Virtualenv Wrapper Helper

Sat, 28 Mar 2009 23:57:19 -0500

A bit inspired by all the awesome stuff happening at PyCon, I've done a bit of hacking on my shell to improve the use of virtualenv_wrapper. Remembering to checkout the correct virtualenv is a bit of a pain, especially when you move around your filesystem as much as I do. To fix this, I've written a bit of code for my bash prompt to ease the pain.

It works by looking for the existance of a .venv file every time you cd into a directory. If it exists, it will attempt to workon the contents of the file, which should be the project name. For instance, for my oebfare project (more news forthcoming on that), the file contains the words "oebfare" as that's the name of my virtualenv. Due to this, you can get general awesomeness such as the following:

# prompt edited to save space
12:02 AM: (~/Code/django) jlilly$ cd baconfile/
(baconfile):12:03 AM: (~/Code/django/baconfile) (master) jlilly$ cd ..
(baconfile):12:03 AM: (~/Code/django) jlilly$ cd oebfare/
(oebfare):12:03 AM: (~/Code/django/oebfare) (master) jlilly$

When we cd into the baconfile dir, we get the virtualenv for baconfile. We can cd out of that directory and about the filesystem, but when we enter a directory that contains a different .venv file, we get a different virtualenv.