Shifting Bits by Patrick Altman: Entries tagged with "django"

Nashvegas Issues

Fri, 17 Apr 2009 10:20:42 -0500

GitHub has added integrated issue tracking:

So, in an effort to sample this awesomeness, I have converted my TODO file to issues for Nashvegas. Please vote on the ones you care about and enter new ones that would be helpful for you.

Django Compress: New CSS Filter

Mon, 23 Mar 2009 09:25:04 -0500

I added a new filter today for django-compress that allows for arbitrary string find/replace during the compile process on your css groups.

I had a specific use case for the new filter: my css had a lot of root relative urls for background images. I wanted to host my compiled media on a different host than where my images were being hosted from but this tied the two together.

Furthermore, I didn't want to hard code the absolute urls as I want to be able to test the full environment on my laptop, test environments and finally be able to deploy somewhere else in production. I want the process of changing environments to be simple settings file change.

Therefore, my approach was to create a filter that at compile time execute a string find and replace. It's super simple, but useful, at least to me.

I also found what I think were a couple of bugs when you had some groups with external_urls defined. When they were missing source_filenames and output_filenames definitions, exceptions were raised when running the management command to compile the source. I put some simple checks in place to safe guard against these exceptions.

Patch to Django Extensions

Thu, 19 Mar 2009 16:11:38 -0500

I am really loving the django-extensions project, especially the sync_media_s3 command to move my media to be hosted on Amazon S3.

I find it increasingly useful when combined with django-compress. However, I ran into a snag today where I wanted to only sync to s3 my compiled scripts from django-compress, instead of my entire MEDIA_ROOT.

Therefore, I added the [a patch to add a --dir optional parameter to the command](http://github.com/paltman/django-extensions/commit/4d1dbb14fb85ee8863cbbef2ef214ffcfcd53758).

New Django Site: Brite Revolution

Sun, 15 Mar 2009 02:03:31 -0500

Josh Kersey announced tonight a new site built by the folks at Cabedge and it appears to be done in Django.

I love seeing fresh ideas, especially concerning digital media. I have long thought the music industry just doesn't get the last 5 to 10 years of technological innovation and public mood shift.

Now, I am not industry expert, but I don't think it takes a wizard to see that Apple has single handedly been disruptive in this area. Couple that with the ability for individuals to produce very high quality recordings in their basement cheaply and then leverage social media and the Internet to distribute their music. The game has changed.

Brite Revolution is the first thing I have seen that is formerly recognizing that and building a business model around it. From their About page:

Artists need to be able to make music again without going into debt or losing ownership of their music. Fans need more music from their favorite artists on a regular basis and feel like a part of an event again. Philanthropic organizations need more opportunities to connect directly with people to make a difference in the world.

To top it all off, they are committing to supporting non-profit charities to the tune 10% of gross subscriptions. I will certainly be subscribing to this.

My Django-Compress Tasks

Tue, 10 Mar 2009 12:07:00 -0500

I am thoroughly impressed by django-compress as mentioned in yesterday's post.

There are number of things that I'll be looking to add over the next couple of days that hopefully will make it even more useful for more than just me. I'll be committing these to my fork and submitting pull requests upstream to hopefully have them committing into the main repository.

For output files, create the intermediary folders if they don't exist.
Add a gzip option flag to the command to create gzip versions of the output files.
Add a new css filter for replacing url() property contents with prefixes defined in configuration (allowing for the serving of content locally during development, and then off of CDN servers when deployed).
Add a new version strategy, that will default to a prefix folder using today's timestamp (YYYYMMDD). Also allow for an optional parameter to the command line to override this default prefix.
Add another "extra_context" variable called ABSOLUTE_URL to enable an override of both the PREFIX variable I included yesterday as well as the default MEDIA_URL. I need this so as to be able to put things like Google hosted libraries like jQuery, but also to have a quick and easy way to revert to hosting it myself in production.

My goal for these additions is to keep them isolated and non-obstrusive so that they exist if others want to use them but don't get in the way of anything or otherwise clutter up the code.

I'd love to hear any feedback about these or other related ideas.

Scratching Django-Compact for Django-Compress

Mon, 09 Mar 2009 17:25:58 -0500

Last week I talked about a project I had started to build called, django-compact, whose goal it was to provide a build process for all the various CSS and Javascript files in a given project.

I was quickly notified in via Will Larson on Twitter that I should checkout django-compress. I did. To say that I was following the same approach would be an understatement. Furthermore, this project was much further along than I was. I spent the better part of this morning getting a subsection of my project wired up to use it and am well satisfied.

There was a small additional feature I wanted to add, so I forked the project and made the additions. I feel like the changes are non-breaking and generally useful and therefore hope to get them pushed upstream.

The feature in question involves being able to have multiple MEDIA_URLs for different outputs. I call them PREFIXes. My approach was to patch the media_url function in compress.utils to accept defaulted argument called prefix. If prefix exists in a non-empty state, then it concatenates itself with the url argument, otherwise it concatenates with the settings.MEDIA_URL setting.

Upstream in render_common I look for a dictionary key called prefix in the extra_context optional dictionary. I then pass this value, that i default to NONE, to the media_url function that I updated.

Finally, in the settings.py, I can now add the parameter when I want to serve the media off of different url paths. This is primarily useful when there ends up being more than a few connections to media elements (images, css, js) on a single page request. Being that most browsers by default will only open up a few connections in parallel per hostname, we can speed things up by adding additional hostnames, especially when these hostnames are part of a CDN (at the cost of additional DNS lookups).

Patch

diff --git a/compress/templatetags/compressed.py b/compress/templatetags/compressed.py
index 0b236e3..f9c55f9 100644
--- a/compress/templatetags/compressed.py
+++ b/compress/templatetags/compressed.py
@@ -14,10 +14,11 @@ def render_common(template_name, obj, filename, version):
         filename = get_output_filename(filename, version)

     context = obj.get('extra_context', {})
+    prefix = context.get('prefix', None)
     if filename.startswith('http://'):
         context['url'] = filename
     else:
-        context['url'] = media_url(filename)
+        context['url'] = media_url(filename, prefix)

     return template.loader.render_to_string(template_name, context)

diff --git a/compress/utils.py b/compress/utils.py
index 6fd21c1..fba6075 100644
--- a/compress/utils.py
+++ b/compress/utils.py
@@ -60,7 +60,9 @@ def media_root(filename):
     """
     return os.path.join(django_settings.MEDIA_ROOT, filename)

-def media_url(url):
+def media_url(url, prefix=None):
+    if prefix:
+        return prefix + urlquote(url)
     return django_settings.MEDIA_URL + urlquote(url)

 def concat(filenames, separator=''):

Django-Compact Updates

Fri, 06 Mar 2009 16:30:50 -0500

Getting back into the swing of things on django-compact today:

Today's Commits

Got concatenation working for both js and css
Added minification support via YUI Compressor

This second item in the list leads me to a question about best practices when creating a re-distributable Django app. Minification support is an optional flag passed at run time to build your css/js:

./manage.py compact --minify

If specified, it will run the concatenated files through the YUI Compressor minifier. Since this tool is java based, and not python :(, I decided to include the jar file in a lib folder under the application folder.

I then created a module called yui.py and encapsulated the shell out via Popen in a python class called Compressor. This does assume that the java executable is in your path and is of a version greater than 1.4 (YUI Compressor requires at least 1.4).

The question I have is, is there a best practice of where and when to include binaries such as this? I feel that this is pretty clean in that I construct the path to the jar dynamically at run time so i don't assume any particular install location. I could do more up front checking for the existence of java and the version number.

Any thoughts?

Nashvegas: A Simple Django Migration Tool

Tue, 24 Feb 2009 10:17:41 -0500

Introduction

Back in July I discussed how I built a tool to help manage database changes across a team and through deployment for our Django based project.

And yes, I am aware of other approaches/projects tackling this same problem. None of them seemed to work how I wanted them to as they all seemed to focus on trying to keep you from writing the raw sql yourself.

I am comfortable with writing SQL and for something that will execute automatically against production databases, I prefer to have that control in knowing exactly what will get executed. Yes, many of these tools have options to allow me to preview the generated SQL, but it still gave me the feeling of being boxed in.

For your reference, here are the ones I reviewed:

So I abstracted out the script and it's assumptions about our environment to function as a management command in a stand alone reusable app.

My Approach

It's extremely simple and is based purely on executing SQL statements in order based on it's file naming scheme. Basically, you create a folder somewhere in your project -- I call mine "db". Within this folder you put the changes to the database into SQL scripts named using a convention that will list them in order. I use:

YYYYMMDD-##.sql

This allows members of the team to know at a glance when the script was created and gives some reasonable avoidance of name collisions. There have been instances where we've had a name collision (I try to push in a commit that has 20090224-01.sql added in my commit index and it already exists in the remote branch, so I just move the file to 20090224-02.sql, commit, pull and then push again).

To sync up your database with changes from the rest of the team you would just execute:

./manage.py upgradedb --execute

If you wanted to just see the output that will be executed you can run it without the --execute flag to just print out the contents to standard output:

./manage.py upgradedb

By default, it looks for a folder with scripts in your current working directory called "db". If your scripts are located elsewhere, just use the --path parameter:

./manage.py upgradedb --execute --path /path/to/db/folder

Where to Get It

I have released v0.1 of the app as it has the bare minimum now to be usable. But there are handful of other additions that I think will make it generally more useful. I do welcome any and all feedback on this project, but want to restate this isn't meant as a replacement or competition for the other migration projects listed above. I have reviewed all of them, and none fit how I preferred to work.

I also welcome any patches, so head over to the project and fork it!

I figured since they really didn't fit my needs and/or what I wanted to do, then there might be others in the same boat and might find what I have done to be useful.

To FogBugz With Python

Fri, 30 Jan 2009 16:28:07 -0500

A couple of weeks ago, I announced the pyfogbugz project that will, when complete, serve as a python wrapper around the FogBugz API. I have had very little time to spend on it recently so it has not progressed much at all. However, one feature that I have added recently is the ability to use the simple BugzScout interface for submitting bug reports.

The one feature about this interface that solves an immediate problem for me is being able to deal with duplicate exception reports. One of my major annoyances with picking up exception reports from an email inbox that FogBugz creates tickets from is that it can't deal with duplicates, so I end up spending a lot of time gardening tickets that are duplicates of each other.

BugzScout will solve this problem by automatically appending to a ticket that has an identical title. I find this extremely useful.

While the library is not comlete, the BugzScout part of it is usable. To install and use:

git clone git://github.com/paltman/pyfogbugz.git
cd pyfogbugz 
pylink pyfogbugz   ## See link [1] to Eric Florenzano's post

Now in your python script:

from pyfogbuz.scout import post_report
post_report('title of the report', 'body of the report -- what wil be appended to ticket if dupe title')

You may be wondering, how does it know where to submit my tickets. Good question! See the README for a sample configuration file. Put configuration options in a file at one of the following two places:

/etc/pyfogbugz.cfg
~/.pyfogbugz

A then correct the sample settings to match your own.

Next step for me will be to wire up this function call somewhere in the Django stack to send exception reports using this script in addition to the email (or maybe I just setup a script to read the email inbox) and then turn off email reading from FogBugz.

[1] Eric Florenzano's pylink

Mobile Django Admin Patches

Thu, 01 Jan 2009 01:31:01 -0500

In addition to adding a couple of minor, but useful (at least for me), patches to to the awesome django-mobileadmin project by Jannis Leidel, I learned something new about how Python works.

The first patch was simply removing an extra slash from a URL in a template. No big deal:

commit 599f172df85586742b5b1ef2bd5b7a24298a0839
Author: Patrick Altman <patrick@###.com>
Date:   Wed Dec 31 23:13:57 2008 -0600

    fixed bug where app links had an extra trailing slash

diff --git a/mobileadmin/templates/mobileadmin/mobile_safari/index.html b/mobileadmin/templates/mobileadmin/mobile_safar
index 046a39d..1d66f59 100644
--- a/mobileadmin/templates/mobileadmin/mobile_safari/index.html
+++ b/mobileadmin/templates/mobileadmin/mobile_safari/index.html
@@ -10,7 +10,7 @@
 {% if app_list %}
     <ul id="applist" selected="true">
     {% for app in app_list %}
-    <li><a href="{{ app.app_url }}/" id="{{ app.name|slugify }}_app">{% blocktrans with app.name as name %}{{ name }}{%
+    <li><a href="{{ app.app_url }}" id="{{ app.name|slugify }}_app">{% blocktrans with app.name as name %}{{ name }}{% 
     <script type="text/javascript">truncate({'{{ app.name|slugify }}_app': 35});</script>
     {% endfor %}
     </ul>

The second patch took a bit more research to figure out what was going on. The symptom I was witnessing was that the mobile interface looked fine in my iPhone until I tried to view a model's list of objects. It kept showing me the default admin view (as you would expect to on the desktop) and it didn't jive with the (screenshots I saw on Flickr)[http://www.flickr.com/photos/jannis/sets/72157607145456697/] that Jannis had posted showing off the app.

After seeing some drastic differences between the output of a dir command on the admin_site variable and the contents of admin_site.__dict__, it quickly became apparent that __dict__ on the object wasn't showing the whole picture and thus there was a bug in the autoregister function.

I then found Keith Waclena's excellent course documentation describing the problem perfectly. It turns out that __dict__ only reveals the members of the class or instance and NOT the inherited members, so one must recurse down through base classes to be complete. So, I borrowed his recursive lookup function, and rewrote a couple lines to approach the lookup differently but accomplishing the same intent (at least I hope so -- it seems to work as intended for me):

commit 5945491fe1216239471d41e590425bfe384d3d99
Author: Patrick Altman <patrick@###.com>
Date:   Thu Jan 1 00:52:46 2009 -0600

    fixed bug in autoregister where attribute searching didn't search deep enough (didn't search base classes)

diff --git a/mobileadmin/__init__.py b/mobileadmin/__init__.py
index f8658ac..77e0de1 100644
--- a/mobileadmin/__init__.py
+++ b/mobileadmin/__init__.py
@@ -4,6 +4,21 @@ from django.core.exceptions import ImproperlyConfigured
 from mobileadmin import decorators, views
 from mobileadmin.conf import settings

+### From http://www2.lib.uchicago.edu/keith/courses/python/class/5/#attrref
+def classlookup(C, name):
+       if C.__dict__.has_key(name):
+               return (1, C.__dict__[name])
+       else:
+               for b in C.__bases__:
+                   success, value = classlookup(b, name)
+                   if success:
+                           return (1, value)
+                   else:
+                           pass
+               else:
+                   return (0, None)
+
+
 def autoregister():
     """
     Auto-register all ModelAdmin instances of the default AdminSite with the
@@ -14,11 +29,11 @@ def autoregister():

     for model, modeladmin in main_site._registry.iteritems():
         admin_class = modeladmin.__class__
-
-        for name, value in admin_class.__dict__.iteritems():
-            if name in settings.TEMPLATE_MAPPING:
+        for name in settings.TEMPLATE_MAPPING:
+            (found, value) = classlookup(admin_class, name)
+            if found:
                 setattr(admin_class, name, decorators.mobile_templates(value))
-
+                
         if admin_class == UserAdmin:
             setattr(admin_class, 'add_view', views.auth_add_view)

I have pushed in these changes to my fork of the git project and sent pull requests to Jannis. Hopefully, he'll integrate them into his project soon, but in the meantime, my fork is available.

Being that this occurred shortly after the stroke of midnight, it is officially the first thing that I have done in 2009!

Happy New Year!

Shifting Bits: Twitter Integration Added

Tue, 04 Nov 2008 10:25:16 -0500

I just added integration to Twitter so that when making a blog post active in Shifting Bits a post is sent to the Twitter account configured in the settings.py file.

As part of the post to Twitter, the url for the post is passed through urlzen to shorten the link so everything is nice an tidy.

This post will serve as it's first public test (so hopefully it works!).

CivicSquared

Thu, 30 Oct 2008 17:09:12 -0500

This is a call to action, folks.

Please go vote for CivicSquared to get funding to bring such a great idea to reality.

The Knight News Challenge is a contest where $5 million worth of projects will be funded. The projects are those with a focus "on neighborhood and community-focused projects, services, and programs".

The CivicSquared project goal is essentially opening up government:

"The same prudence, which, in private life, would forbid our paying our money for unexplained projects, forbids it in the disposition of public moneys." -- Thomas Jefferson

CivicSquared is social software that helps communities obtain and discuss information about local legislation, tax allocation, and elected officials' voting records. It will be released as an open source Django application.

Most citizens do not know how their local taxes are spent. While it is true that citizens can voice opinions at the public meetings where these decisions are made, there are enormous barriers to effective participation. Some of these barriers include advance access to agendas, to information about issues, and to past deliberations on those issues. Even when agendas are posted in advance, there are often questions, and the impact of legislation on a community is often unclear without a public dialogue. One cannot assume that local media will offer multiple perspectives and unbiased views due to a lack of diversity in media ownership at the local level. Something needs to be done to make the process more inclusive.

CivicSquared seeks to address these problems via services far beyond basic access to public meeting records. Our goal is to provide communities the simple, powerful tools to be aware of public meetings, comment on upcoming legislation, track that legislation and who voted for it, and see how their tax dollars are being spent.

With CivicSquared, we aim to address all the principles of Open Government and will work with the OpenGov community to follow, implement, and produce standard formats for storing and sharing this data in a structured and semantic way.

This is beyond really cool. It's vitally important for our democracy. So with voting on everyone's mind (at least here in the US), please go vote for this awesome project.

JSONField for Django Models

Tue, 21 Oct 2008 13:28:33 -0500

I was needing to store semi-structured data on a model for a project I am working on, but didn't want to parse strings or create a foreign key to a key/value type table.

The use of JSON came to mind via the simplejson library. I thought leveraging Python's dict and list data structures would provide ample flexibility will minimizing any parsing of text (at least directly by me). Before I set out to write a JSON field type, I thought I'd give Google a try and sure enough someone (deadwisdom) had beat me to it.

This snippet is really cool in that now I can declare fields of type JSONField() and simple interact with them as python data types however, they get saved to the database as simplejson encoded strings and rehydrated back from strings into their original data structure, leaving you without having to think about how the data will be stored.

Shifting Bits: My Django Blog is Live

Sun, 19 Oct 2008 21:46:57 -0500

Just a short announcement to say that I am finally free of Wordpress and mod_php on my server. This site is Django-powered!

Still lots more that I want to do with it, but my basic requirements are complete (comments, comment moderation, xmlrpc interface for MarsEdit compatibility, rss feeds, and tagging).

The Django Toolbar: Updated

Thu, 16 Oct 2008 03:12:17 -0500

So after some late night code slinging to scratch a wild itch of an idea, I rethought my approach in how I forked the code for the Django Debug Toolbar project. Partly this was prompted by some folks that took some time to leave a few comments on the previous post announcing the release -- many thanks!

Anyway, I just committed changes to a new fork of the Django Toolbar project that I hope is considering merge worthy by Rob Hudson to get back into the project and my fork can live a short but useful life until my pull request is incorporated.

The basics of this change is that you can control whether or not server side logging occurs as well as whether or not the toolbar is displayed via some configuration settings that are documented in the README.rst.

Check it out and let me know what you think. I'd like to build some dashboard type tools for browsing the server side logged data. But beyond that I think it is probably ready to start using.

The Django Debug Un-Toolbar

Mon, 13 Oct 2008 05:53:53 -0500

When I caught wind of the Django Debug Toolbar a couple weeks ago, I was really excited to put it to use. However, when i wired it up, I discovered that it wasn't going to play nicely with MooTools since it is based on jQuery.

Not being a javascript expert, I am assuming there is some global namespace collisions in how the two libraries use shortcut functions like the dollar sign selectors (e.g. $('some-dom-element')).

Anyway, the more I thought about it, I figured it probably best to not manipulate the DOM at all. In some discussion with others, it was determined a very useful adaptation to the project would be to serialize off all the data collected in the middleware on the server and set a cookie to a special path so that it could be retrieved via something like a bookmarklet.

So that's what I did. I forked the Django Debug Toolbar project and renamed it to Django Debug Un-Toolbar, real original, I know. While I don't expect it to be perfect yet (I spent about 2 hours this evening making sweeping and hurried changes to the code base), it does work as intended.

There are a number of things I want to add/change before I dare think about using in production. Yes, I think it would be useful to have in production, and triggered to turn on in a number of different ways -- either through changing a settings parameter to passing it a special query string parameter, or working for certain users.

By having the debug "toolbar" work in this manner it will be possible to collect all this good debug information from customers who's screens are unavailable to see when things inevitable go wrong and you need to troubleshoot things quickly.

Update

I am closing the Un-Toolbar fork in favor of the other fork that i created that is more upstream compatible. Hopefully that fork is only temporary as well as I'd hope to get it worthy of merging back to Rob's repository.

Shifting Bits Makes Progress

Mon, 13 Oct 2008 05:43:23 -0500

I have done a little more hacking on Shifting Bits this weekend and have launched a beta version of it:

http://beta.paltman.com

In order to keep crawlers and indexers out of it and not leaving it up to trusting that they obey a good robots.txt instruction set, I have put some simple low-tech authentication on it (HTTP Basic).

The username and password for this beta site is paltman/test.

I'd love to hear any feedback about what's there.

This was mainly a test to get my environment where it needed to be to host the blog as well as to exercise my Wordpress import tool somewhere other than my laptop.

DjangoCon Sessions: Video

Tue, 16 Sep 2008 20:34:54 -0500

The DjangoCon sessions are now up on YouTube.

Shifting Bits: Another Django Blog Engine

Mon, 08 Sep 2008 03:47:24 -0500

Tonight I released/open-sourced ShiftingBits. This blog is still on WordPress, but I believe I am close to having the all the features I need/require to make the migration possible.

I pretty much stole a bunch from Brian Rosner's oebfare code base so if any of it looks familiar, that's where I copied a lot of this from. Thanks, Brian!

If you are wanting to move from a WordPress blog to your own Django-based site, you might be able to save some time with this blog engine. This is exactly the path I am taking.

I am using:

mootools 1.2 -- JavaScript library awesomeness
blueprint -- CSS framework
wmd -- Live markdown preview for comments
prettify -- Really cool syntax highlighter

Feature Summary:

Basic blog post mode driven using generic date based views with redirects so that the WordPress url date format (numbered month versus the prettier and django default of short month name) get mapped properly so that none of your links break.
WordPress to Django Migration Script
Tagging via django-tagging
Comments via the new django.contrib.comments and using the live markdown preview editor, wmd.
Full RSS feeds (both latest feeds and by tag)

Left to To:

Implement an XML-RPC interface that Mars Edit can communicate with.
Add Google Analytics
Clean up CSS so that the templates look better.
Handle Trackbacks/Pingbacks

This is the code that I am planning to use shortly here at paltman.com, including the css/templates. I would love to hear feedback from the community on the code and/or approach taken.

Django Blog in the Works

Wed, 03 Sep 2008 04:59:06 -0500

I am previously stated my desire about rebuilding Shifting Bits into a Django-based blog, but it has been really slow going due to other time commitments.

Over the weekend, I started cutting out a fresh new template and a migration script to get my data export from WordPress into my Post model (borrowing heavily from Brian Rosner's oebfare -- why invent when you can steal).

Here is a preview of what is to come, hopefully sooner rather than later:

Django Group in Nashville

Sat, 30 Aug 2008 03:53:17 -0500

I was pretty pumped to get the invite from Trey Piepmeier tonight about trying to start a local (Nashville, TN) Django meetup/group.

If you are in the area, head on over and sign up!

New App: Django and AWS

Tue, 26 Aug 2008 03:25:40 -0500

I have started a new app over on github called django-aws. It's pretty bare bones right now but wanted to announce it earlier rather than later as I am hoping to:

Get some folks who might be interested to help me out.
Get some eyeballs on it for recommendations for how to (or how to not) do certain things.

I would appreciate any feedback or help that I might be able to scrounge up! This app will have a strict dependency on boto which is a really cool project provides the de facto way for interacting with Amazon Web Services in the Python language.

Changeset 7967: NFA Branch Merged into Trunk

Sat, 19 Jul 2008 04:00:10 -0500

I and a host of other folks have been waiting anxiously for this merge for a while now. Excited to see it happen tonight.

Good job, Brian Rosner et al.

GitHub Enables Markdown Documentation

Tue, 15 Jul 2008 03:11:19 -0500

I have been desiring to have the documentation that I write for my various projects on GitHub render to HTML when using a .markdown extension, just like they support doing for special files like README.markdown.

At first, I thought they'd eventually get around to it and that is the best I could hope for as surely they are too busy to respond to a non-paying customer like me. Then I learned they the used the Lighthouse ticket system to manage their project and took public submissions for features.

Before logging a feature request I did some searching and found a ticket already out there requesting exactly this same feature that I wanted. So I just added my own comments to the ticket to give it a +1.

After a little back and forth I learned it was implemented tonight (just a week after my first comment). Good to see such responsive development.

An example of this feature can be found in my django-aws project which is simple app to provide some template tags to provide Amazon Web Services information a la boto.

Ordering Edit Inlines

Thu, 10 Jul 2008 03:37:06 -0500

In my continued experimentation with the newforms-admin branch of Django, I wanted to figure out how to order fields of an inline. Looking at the documentation for inlines I saw there was not an ordering field.

I had thought that there was but it turns out I was just mistaken in the object hierarchy. InlineModelAdmin inherits from BaseModelAdmin the same as ModelAdmin -- I was thinking that InlineModelAdmin inherited from ModelAdmin.

Therefore, I had to determine a way to accomplish this via some spelunking through the code. At first, I thought I'd just create my own template by inheriting or copying the tabular.html template. After looking at it, I determined that I didn't want to figure out how to do a regroup on inline_admin_formset or if even that was the proper way to try to order things in that template.

After a few minutes in the code I realized that I could just subclass BaseInlineFormset and specify the fields that I wanted to order by (in this example I will use start_time and end_time:

from django.contrib import admin
from django.newforms.models import BaseInlineFormset

class MyOrderedFormset(BaseInlineFormset):
    def get_queryset(self):
        qs = super(SessionInlineFormset, self).get_queryset()
        return qs.order_by('start_time', 'end_time')

class MyOrderedInline(admin.TabularInline):
    model = MyModel
    extra = 1
    formset = MyOrderedFormset

That's it. Now after that section of inlines are ordered by start_time and end_time.

Managing Database Changes in Django

Thu, 03 Jul 2008 17:24:01 -0500

Introduction

Managing database changes in a team environment working on a django project can be complicated. I would imagine that there is no one size fits all solution and it would depend on team size and configuration, production database size, etc.

What I will outline here is a solution I developed for a small team that I work with on a django based web application. So far it has worked really well for allowing us to streamline database changes so we can stay in sync, deploy easily, and add small tweaks to the database (indexing, data manipulation, etc.) that is semi-automated.

The idea is based partly on Rails database migrations (though not has sleek and well put together) and a database versioning system that was used on a team that I have worked on previously using MS SQL in a corporate environment and a fairly decent team size. In one sense it is not nearly as well put together as either of these two solutions, but at the same time, it is pretty much hands off and has been working well for us for a number of months now.

Why not just use syncdb?

As nice as python manage.py syncdb is when the project is getting started and is still in early phases, it becomes less and less useful, mostly in handling schema changes. Since syncdb only creates a model if it doesn't already exist in the database, we found ourselves outputting sql with python manage.py sql APP_NAME and then making adhoc ALTER TABLE scripts from the CREATE statements and passing them around to each other and then trying to remember the order to apply them in production.

Furthermore, syncdb left us without a way to alter data, add indexes, remove indexes, etc. Granted that wasn't it's intent and I am certainly not trying to beat up syncdb -- it does what it does well.

The Solution

In order to address this, it seemed the most natural thing was to have some structured and ordered way to write SQL scripts that were applied in a uniform manner so that it was repeatable. This way we could run on our development databases, on test databases, and in deployment on production all the same way.

First step to achieve this was to version the database. This is accomplished with a simple table that will keep track of what it has executed along with some other metadata primarily for reference purposes.

CREATE TABLE `versions` (
    `version` VARCHAR(200) NOT NULL, 
    `date_created` DATETIME NOT NULL,
    `sql_executed` LONGTEXT NULL,
    `svn_version` int null
);

In order to populate this table and keep versions consistent, the naming scheme of the sql scripts should follow something that lends itself to easy sorting:

YYYYMMDD-##.sql

where pound signs are zero-padded integers of the changes for the day. In a team environment, it is probable that two people might be working on a database change at the same time and therefore would be use 01. Whoever committed first would be able to keep the 01, the other developer would get a conflict message and need to rename his script to 02.

These scripts live in a db/ folder in our project. The name of this folder is not important, but it is important to keep all the sql scripts in a folder not mixed in with other files.

To tie it all together there is s simple python script that gets the list of applied changes from the database, gets a list of files in the current directory in order, filters out the ones that have been applied and then processes the remaining scripts. In processing the scripts, it opens each script and splits script down into individual statements (splitting on the semicolon). After execution of an entire script file is complete, the versions table is updated to reflect the applied version.

Running python upgrade.py by itself will simply print to standard output all the statements it plans to execute, so that they can be reviewed. Running python upgrade.py --execute will actually execute the scripts.

If an error was found in executing a statement, processing stops immediately.

You can find upgrade.py in Django Snippet 849.

Improvements

There are lots of improvements to this script that I'd like to add if I got around to it. The ones I'd like to see most is support for "rollbacks". Comments and suggestions are most welcome!

Edit Inline Support for Generic Relations

Mon, 30 Jun 2008 04:12:33 -0500

Introduction

I've recently (past week or so) been digging into the newforms-admin branch of django. I am really looking forward to this code line getting merged into trunk as there are tons of great work in this branch. I especially like the separation of the admin code from the models.

Well, I'll get to my point.

The Problem

I needed to be able to edit child records in the admin that were children through the use of GenericRelations. I was told about a solution on Google Code called django-genericadmin, but it seemed dead and I could not get it to work.

The thing that seemed the closest to working was a patch on ticket 4667 by Honza Kral and a variation on this patch in the Django Snippet 765. Neither of these worked for r7771 of branches/newforms-admin.

The Solution

I put the 765 snippet in a file called generic.py in the root of the django project I was working on and fiddled with it until I got it working. I ended up needing to change an import, fix some variable names, and change the argument list order in one of the init methods. I also added the can_order and can_delete options to the GenericInlineModelAdmin class.

After I got it working isolated in generic.py, I then added the code to the django/contrib/admin/options.py file and created a new patch (after testing locally of course). This patch has since been added to ticket 4667. I also posted the full version that I had running in generic.py in snippet 832 in case someone wants to use it without patching their newforms-admin branch.

Example of How to Use

In order to get a sense for how to make use of this functionality, imagine you have a songs app that you use to model and record details about songs, you have built this generically so you can reuse it wisely in a number of different projects. One project in particular has a need to to reference song data.

You are building a home media inventory system and decide you want to catalog both your albums as well as your DVD collection, however, you want the ability to record what songs exist on both your albums as well as your DVDs as you are a fanatic about soundtracks.

So you have the following models (abbreviated for clarity):

# Media Collection App

class Album(models.Model):
    ...

class DVD(models.Model):
    ...

# Song App

class Song(models.Model):
    ...

Since you want to enable your Song model to relate to two other models generically, you decide to use Generic Relations like so:

from django.contrib.contenttypes.models import ContentType
from django.contrib.contenttypes import generic

class Song(models.Model):
    ...
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey()

and then on the models that you want to have 0 to Many child records that are songs:

from myproject.song.models import Song
from django.contrib.contenttypes import generic

class Album(models.Model):
    ...
    songs = generic.GenericRelation(Song)

class DVD(models.Model):
    ...
    songs = generic.GenericRelation(Song)

Now that's all you need to do if you didn't care about editing the data in the admin, however, since that is not the point of the article, here is what you need to do in your admin.py in the app that contains Album and DVD:

...
from myproject.song.models import Song
from myproject.generic import GenericTabularInline
### OR ###

### from django.contrib.admin.options import GenericTabularInline



class SongInline(GenericTabularInline):
    model = Song
    extra = 2 
    ct_field_name = 'content_type'
    id_field_name = 'object_id'

class AlbumOptions(admin.ModelAdmin):
    model = Album
    inlines = (SongInline,)

class DVDOptions(admin.ModelAdmin):
    model = DVD
    inlines = (SongInline,)

admin.site.register(Album, AlbumOptions)
admin.site.register(DVD, DVDOptions)

Now you should be able to add/remove songs when editing an Album or DVD in your admin.

Update: Fixed typo in code sample above in setting the GenericForeignKey() -- thanks FunkyBob!

Django OperationalError Debugging

Sun, 11 May 2008 06:05:16 -0500

So on one of our test servers I was doing some heavy performance testing when I noticed a strange OperationalError exception message in the Debug console for Django:

I went digging around and sure enough I had all kinds of locks when I attempted:

mysql> SHOW PROCESSLIST;

So next I checked the status of the free space available to learn than the partition that the mysql data was on was completely full. So, shutdown mysql and relocated the data to another partition with plenty of room:

mysqladmin shutdown
mv /usr/local/apache/var/* /new/location/for/files/

Then I changed /etc/my.cnf so that it had the following lines edited or amended:

[mysqld_safe]
datadir = /new/location/for/files/

Make sure that the ib_logfile files don't remain in the new location:

rm /new/location/for/files/ib_logfile*

Now restart mysql and verify that everything is working:

mysqld_safe &
echo "create database testdatabase;" | mysql
echo "show databases;" | mysql

A Default Bug in Django

Wed, 07 May 2008 22:54:25 -0500

Not to say that this is a bug in django but rather a bug in how I had written some of my models.

Just wanted to point out a quick tip for anyone else who may have done the same thing as me as well as for any of the django experts out there that might suggest a better way.

What was happening was that I began to notice a wild discreptancy between two datetime columns in several models. They should have been identical, however, they were both getting their dates differently:

...
date_column_one = models.DateTimeField(default=datetime.now())
date_column_two = models.DateTimeField(auto_now_add=True)
...

What seems to be happening with this configuration is that datetime.now() is executed once when the server starts (and/or the module is imported) instead of what I thought would happen when I initially wrote these and that datetime.now() would get executed anytime an instance of the model was created and saved as a new record.

My solution was to override the save method:

...
date_column_one = models.DateTimeField()
...
def save(self):
    if not self.id:
        self.date_column_one = datetime.now()
    super(MyModel, self).save()
...

I am sure there is probably a better way and if you know of one, please leave a comment or send me an email and let me know!

Keeping ContentTypes and Permissions Updated without syncdb

Fri, 11 Apr 2008 16:16:41 -0500

In managing releases to a production django project, I find myself using python manage.py sqlall APPNAME to put the sql needed in a script for any new models, and then writing the ALTER or DROP statements by hand to also go into the sql scripts.

One thing that not running syncdb on production leaves out are the creation of the content type and permission records (if using django.contrib.auth and django.contrib.contenttypes). It was recommended to me on #django that I look at each of these apps management.py module to see where they hook into the post_syncdb signal.

In doing so I discovered some methods that I could just call from python and keep both my permissions and content types up to date:

from django.core.management import setup_environ
try:
    import settings
except ImportError:
    import sys
    sys.stderr.write("Couldn't find the settings.py module.")
    sys.exit(1)

setup_environ(settings)

# Add any missing content types
from django.contrib.contenttypes.management \
    import create_all_contenttypes 
create_all_contenttypes()

# Add any missing permissions
from django.contrib.auth.management import create_permissions
from django.db.models import get_apps
for app in get_apps():
    create_permissions(app, None, 2)

Enjoy.

I Want to Move My Blog to Django

Sun, 03 Feb 2008 02:08:05 -0500

Oh how I long to move this blog off of WordPress and onto something I build (or cull together) on top of the Django platform. It feels like it is mostly pulling together a host of different apps that are already written:

That leaves a core part of the blog to write -- something to store the actual entries and then a URL scheme to display different levels of archives and individual posts. Of course, this schemes needs not to break current links.

This led me to check out what already might exist on Google Code and found these (limited to the most active):

All are basically the same, with slightly different inclusions and approaches to this core "Blog" feature.

Lastly, a feature I'd want to implement (if there isn't already an app out there supporting it that I can integrate) is a MetaWeblog API interface so that I can use MarsEdit, my favorite blog publish. Then all that would be left would be figuring out the data structure of WordPress and migrate over all my content (this is the only part that I am not looking forward to).

Django People Rocks

Wed, 30 Jan 2008 04:08:14 -0500

This is a really cool site for getting a sense for the django community in your community.

I just put my profile up. We need a micro-format (maybe hcard would work) for "profiles" so the same set of metadata about ourselves can just be reused at the different "profile sites" that we find ourselves joining.

Breaking Apart Models in Django

Tue, 29 Jan 2008 23:48:55 -0500

Depending on the size of your models.py, you may find it becoming unwieldy, especially if you are on a team with multiple developers are editing the same file, increasing the odds for merge conflicts.

There are two parts to breaking up a large module in Django. The first is a purely Python convention where you break apart the file into multiple files and create a subdirectory for the name of the original module so your models.py file becomes:

models/__init__.py
models/module1.py
models/modlue2.py

Inside your __init__.py you'll want to import from your module1.py and module2.py modules and add those imported objects to __all__:

# __init__.py
from app.models.module1 import MyClass
from app.models.module2 import AnotherClass, function_one
__all__ = ['MyClass', 'AnotherClass', 'function_one']

Now that you have taken care of the Python part of this exercise, you'll need to do a couple tricks to get your models importing properly in the context of your django project.

It involves adding a couple o attributes to the Meta inner class. Thanks to Collin Grady (aka Magus on #django on irc.freenode.net) for pointing these tips out to me:

class MyClass:
    ...
    class Meta:
        app_name = 'myappname'
        db_table = 'myappname_myclass'

Now you should have a better code base that causes less contention and makes it faster to locate the models you are looking for.

Update

As has been pointed out to me in the comments as well as one #djano today, I got "Empty" and "Magus" mixed up with their real names. Much apologies for any and all confusion this might have caused.

Update 2

As pointed out in some of the recent comments below, this post was corrupted when I enabled the markdown plugin for wordpress. It is treating all posts I have ever written as markdown which is not right but I have not the energy nor the inclination to go back in time and update all my old posts. That being said, I really love markdown for my posting so I'll keep the plugin activated.

Django Flatpages with Debug Set to False

Fri, 11 Jan 2008 03:34:00 -0500

After deploying a site I am working on to a preview server running apache/mod_python, it was the first time serving up the site in something other than python manage.py runserver.

I worked through a number of issues that I found quick answers for on the FAQ site for django, but one in particular I couldn't figure out, but eventually got pointed to the obvious in the irc channel for django.

The site has a number of flatpages, which I have come to learn is a middleware trick where when a url doesn't match anything and right before a 404 normally would be thrown giving the Page Not Found error, it checks against any flatpage "urls" you have registered the database with static content. If it finds something it will return that content, if not the 404 bubbles back out of the middleware and goes to the user.

Everything worked fine with DEBUG=True in my settings.py file. However, when I changed it to DEBUG=False, I got a 500 Internal Server Error but then when I looked at my error.log, I could see a django exception saying it couldn't find the 404.html.

This made no sense to me at all. Why would it be looking for a 404.html when it doesn't need it yet (the flatpage middleware should have intercepted it before it looked for a 404.html template, was my thinking). It was suggested to me in the irc channel that I create a 404.html (for more reasons that just fixing this problem).

So to just test that theory, I ran touch templates/404.html -- it worked!

Django Custom Filter Registration

Thu, 10 Jan 2008 16:57:41 -0500

I wrestled with this one for about an hour last night before realizing it was rather simple. Following the instructions to create a custom template filter, I thought everything was simple and made since until when I tried to load the page to test it.

The page spewed a stack trace telling me that it couldn't find the custom filter I was trying lo load with:

{% load website.app.custom_filters %} {% filter htmlcompress %} ... {% endfilter %}

I then realized that I had registered it like so:




website/app/template_tags/custom_filters.py

from django import template register = template.Library() @register.filter("htmlcompress")

It seems that there is some magic that occurs that loads the custom_filters.py module in a global namespace so that doing this in my base template works:

{% load custom_filters %} {% filter htmlcompress %} ... {% endfilter %}

Thought I'd post this in case I am not the only new person to django wrestling with this.

Running Multiple Django Versions in Development

Tue, 08 Jan 2008 17:58:20 -0500

I have needed to use different revisions of the django trunk at different points in time (in testing locally a new revision before upgraded in production, or working on two different projects that deploy to two different production environments, etc).

There may be a better way, but how I have handled it pretty painlessly is by running this script:




!/bin/bash

rm -rf /[YOUR PATH]/$1 svn co -r $1 http://code.djangoproject.com/svn/django/trunk/django /[YOUR PATH]/$1 sudo rm -rf /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django sudo rm /usr/bin/django-admin sudo ln -s /[YOUR PATH]/$1/ /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django sudo ln -s /[YOUR PATH]/$1/bin/django-admin.py /usr/bin/django-admin echo "Switched to" $1

It takes a single parameter, a revision number.

./switch.sh 7000

I run this script from within a directory in my home directory called "django".

You will likely need to change the paths in the script above if you are not running Python 2.5 on Mac OSX. Also, remember to substitute "[YOUR PATH]" for the base location of where you want to store the django code.

Cool Django Apps / Frameworks / Add-Ons

Mon, 07 Jan 2008 04:06:48 -0500

Ran across three really cool looking extensions to django:

databrowse
Databrowse is a Django application that lets you browse your data.

As the Django admin dynamically creates an admin interface by introspecting your models, Databrowse dynamically creates a rich, browsable Web site by introspecting your models.
sitemap

Django comes with a high-level sitemap-generating framework that makes creating sitemap XML files easy.
webdesign

The django.contrib.webdesign package, part of the “django.contrib” add-ons, provides various Django helpers that are particularly useful to Web designers (as opposed to developers).

I am anxious to given these items a spin, especially databrowse.

Django Fixtures and Flatpage Deployment

Mon, 07 Jan 2008 03:59:23 -0500

I was introduced today to the concept and django feature known as Fixtures, while reading my dead-tree version of the new django book.

I am currently building a small site, fairly simple and mostly static. I have written an app that is my own take on the flatpages that are in django.contrib (a post on this later) -- I basically needed a little more flexibility for different islands of content.

In building this site and setting up the appropriate flatpages in my development, I realized that I am going to need to seed the database when I release to production or else there are going to be a lot of 404 errors. Also, since the pages are referred to in my menus, I need for the url on the flatpage model to be exact. It's not that many pages, but still, I thought, there had to be a better way that either manually hand entering all the data, troubleshooting 404s for typos in the url field, or doing any manual exporting of data at the mysql prompt.

As if by some divine revelation, I happened across the chapter in the book talking about django-admin dumpdata/loaddata. What's cool about this is by dumping the data into a file in source control, you can revision your flatpages (or any other data that is).

Here's what I did:




mkdir fixtures

django-admin dumpdata flatpages --indent=4 > fixtures/flatpages.json

Remember, I have a local app in my project called flatpages that is my replacement for the contrib version.

I tested that this data was usable by changing my setting.py file to use a different database (a new one):




mysql

mysql> create database test_import; msyql> exit; django-admin loaddata fixtures/flatpages.json python manage.py runserver

Then fired up the browser and there was all my flatpages, perfect!