Brandon Konkle's Blog - Latest Posts

4 Day Work Week

Fri, 15 Jul 2011 12:07:41 -0500

Since I began working with Lincoln Loop, I’ve been absolutely loving the flexibility. What has surprised me, though, is how hard it has been to consistently put in 40 hours a week. There were several weeks when it was even difficult for me to get to 32 hours, which is my absolute minimum. I decided that it was time to take advantage of this newfound flexibility, and try a new work schedule. So, six weeks ago I shifted to a 4 day work week. So far, it has been great! There are definitely positives and negatives about the arrangement, but overall it’s been very good for me and it’s made it much easier for me to hit my 40 hour goal each week.

I work Monday to Thursday, 8 hours during the day, and 2 hours in the evening. Splitting the time up that way helps me avoid getting to burnt out each day, and gives me a chance to spend some time with my wife and son before the little guy goes to bed. I usually jump back on the computer by 8, and finish up for the evening by 10. This leaves me a couple of hours for some reading (or Minecraft) before bed.

On Fridays my family and I usually spend part of the day out and about, enjoying the opportunity to get away from the house and be together. This is especially important to us since we are home all the time. I work from home, and Barbara stays home and cares for Elliot. The time we spend out on Friday as a family is great bonding time and it fights off any traces of cabin fever.

One of the most effective things about it, though, is the motivation. I get a direct correlation between working hard and putting my hours in, and getting an extra day to relax. I know that at any moment my 4 day week could become a 5 day week if I don't focus on my work and get my time in. This personal reward makes a great motivation to be consistent and not give in to laziness or procrastination. For this reason, I think it’s important for me that the day off be at the end of the week. I’m working towards a reward each week, not struggling to catch up.

The downside is that it has made it more difficult to spend time with friends during the week. I’m always working on weeknights, so we’ve had to turn down invitations to get together on several occasions. To overcome this I’m starting to plan for one free night each week, often working 3 hours during two other nights to compensate.

I definitely feel that this new schedule has made me more productive and consistent, and I’d certainly recommend it to those who have the opportunity. I’m definitely glad to be away from the rigid corporate world and free to pursue the style of work that fits me best!

uWSGI and Django Secure Requests

Fri, 29 Oct 2010 00:27:09 -0500

A colleague and I ran into a frustrating situation this evening when transitioning a site from Apache and mod_wsgi to uWSGI. Django's request.is_secure() wasn't working correctly. After much Googling and shouting at the SSH console, we finally discovered the UWSGI_SCHEME uWSGI parameter.

To fix the issue, we edited /etc/nginx/uwsgi_params (which originally comes from nginx/uwsgi_params in the uWSGI tarball) and added an additional param at the bottom:

uwsgi_param  UWSGI_SCHEME   $scheme;

This prompts uWSGI to set wsgi.url_scheme to the appropriate value, which Django then detects in the wsgi handler.

I'd really love to see this in the documentation on the uWSGI site, or (even better) included by default in the uwsgi_params in the tarball. In the meantime, though, it's a relatively quick and easy fix. Enjoy!

Postgres 9 Streaming Replication and Django-Balancer

Wed, 20 Oct 2010 23:26:03 -0500

Over the past couple of weeks I've had the opportunity to dig in to Postgres 9's streaming replication. What I found was a relatively easy way to configure a system for basic replication with very impressive speed. Postgres's streaming replication is an enhancement to the log-shipping warm standby that was available in previous versions. Instead of replaying write-ahead log (WAL) files as they become available on the slave's filesystem, the slave nodes connect directly to the master and stream the log as it generated. This results in extremely fast replication, typically under a second.

It can't match the cascading features of Slony, but if you're looking for a simple structure with a master database feeding one or more read-only slaves then Postgres 9 is a great choice.

Here's how I got everything working on Ubuntu 10.04.

UPDATE: I've changed the NFS details below to squash the user and group IDs and set the anonymous access user and group to the slave server's user and group IDs. This resolves issues where the postgres use has a different ID on the slave server than the master server, so files received from master are owned by the wrong user.

Postgres 9

Postgres 9 isn't included in the Lucid repos (or Maverick, for that matter), so you'll need to use a PPA. I used Martin Pitt's.

$ sudo apt-get install python-software-properties
$ sudo add-apt-repository ppa:pitti/postgresql
$ sudo apt-get update
$ sudo apt-get install postgresql postgresql-server-dev-9.0 postgresql-contrib-9.0

If you're using GeoDjango, you'll need to re-compile the requirements from source after 9.0 is installed. Make sure to use Postgis 1.5.2, which is the first release to support Postgres 9.

Because the default encoding is ASCII, you'll need to drop the cluster, and re-initialize it as UTF8.

$ sudo -u postgres pg_dropcluster --stop 9.0 main
$ sudo -u postgres pg_createcluster --start -e UTF-8 9.0 main

At this point, you can restore any database backups needed. If you'd like, you can begin accepting live queries - the base backup needed below can run while the database is still available.

NFS Sharing

I'm using log-shipping and streaming replication together, just in case something goes wrong with the direct connection. I use NFS to share a folder on the slave node and mount it on the master, so that the master can send WAL files to the slave.

On the Slave Server

$ sudo apt-get install nfs-kernel-server nfs-common portmap

Note: If you're on Rackspace, you'll have trouble with nfs-kernel-server. See my post here for details on how to work around it.

$ sudo dpkg-reconfigure portmap

Select 'no'.

$ sudo restart portmap
$ sudo emacs /etc/exports

Find out the user and group IDs for postgres. You'll use them in the next step.

$ id -u postgres
104
$ id -g postgres
108

Add a line like this:

/path/for/backups   10.0.1.101/32(rw,async,no_subtree_check,all_squash,anonuid=104,anongid=108)

Replace the CIDR notation with your own. Use a CIDR calculator if needed. Replace the anonuid and anongid with the appropriate values for your server.

Then, restart the server and export your shares. Make sure the path exists on the filesystem before doing so, and adjust the permissions as needed so that the path can be written to.

$ sudo /etc/init.d/nfs-kernel-server restart
$ sudo exportfs -a

On the Master Server

$ sudo apt-get install nfs-common portmap
$ sudo emacs /etc/fstab

Add a line like this:

slavehostname:/path/for/backups /path/for/mount nfs rsize=8192,wsize=8192,timeo=14,intr 0 0

Then mount it:

$ sudo mount /path/for/mount

Test the NFS connection by touching a new file on the primary machine, and making sure it can be seen on the standby machine.

Replication

On the Master Server

Edit postgresql.conf to enable WAL archiving:

wal_level = hot_standby

archive_mode = on

archive_command = 'cp -i %p /path/for/mount/%f </dev/null'

max_wal_senders = 1

Adjust max_wal_senders to the number of slave servers you plan on using.

Create a superuser for streaming replication:

CREATE USER myuser WITH SUPERUSER ENCRYPTED PASSWORD 'mypassword';

Modify pg_hba.conf to allow the replication user to connect:

host    replication     myuser     10.0.1.102/32             md5

You'll need to restart Postgres at this point.

Base Backup

Next, a base backup needs to be created. This must be a filesystem-level backup, not a logical backup like the pg_dump command creates. On the master server, run the pg_start_backup command:

SELECT pg_start_backup('base_backup');

Then, tar and bzip the data directory:

$ cd /var/lib/postgresql/9.0/
$ sudo tar -cjf ~/postgres-data.tar.bz2 main

This will take awhile. Once it's done, run the pg_stop_backup command:

SELECT pg_stop_backup();

Then, copy the archive to the standby server. Make sure postgresql is not running on the standby, and unzip the archive:

$ cd /var/lib/postgresql/9.0/

# Make SURE you're on the standby server here
$ sudo rm -rf main
$ sudo tar -xjf ~/postgres-data.tar.bz2
$ sudo chown -R postgres:postgres main

On the Slave Server

Edit postgresql.conf to enable hot standby:

hot_standby = on

Create a recovery.conf in /var/lib/postgresql/9.0/main/ similar to this::

standby_mode = 'on'
primary_conninfo = 'host=10.0.1.101 port=5432 user=myuser password=mypassword'
restore_command = 'cp /path/for/backups/%f %p'
trigger_file = '/path/for/backups/trigger_file'
archive_cleanup_command = 'pg_archivecleanup /path/for/backups %r'

The postgresql-contrib-9.0 package doesn't link pg_archivecleanup into the path. You can do so manually:

$ sudo ln -s /usr/lib/postgresql/9.0/bin/pg_archivecleanup /usr/bin/

Smoke Test

To make sure everything is working properly, first check and make sure wal sender and receiver processes are live on the master and slave servers. On the primary, run ps -ef | grep "wal sender" and make sure you see a wal sender process. On the standby run ps -ef | grep "wal receiver".

Next, create a test table on the primary:

CREATE TABLE test (test varchar(30));
INSERT INTO test VALUES ('Testing 1 2 3');

On the standby, query the table:

SELECT * FROM test;

You should see the following output:

     test      
---------------
 Testing 1 2 3
(1 row)

You can then delete your test table:

DROP TABLE test;

Load Balancing

For load balancing, I decided to leverage the excellent multidb capability introduced in Django 1.2. To do this, I created the django-balancer project. It builds on the example of database routers in the Django docs, a post on Eli Bendersky's blog, and Jeff Balogh's django-multidb-router project.

For my project, I decided on the WeightedMasterSlaveRouter. I want to include my master database in the read pool, but to a lesser extent than my slave database

I borrowed master-pinning from Jeff Balogh's project for django-balancer, but I ended up not needing it in my project. In my tests, I never had an issue with the databases being out of sync. The replication was really that fast! Even in situations where a user just created an object and was being rerouted to a rendered view of that object, the replication was speedy enough to keep up and prevent a 404.

Installation

Install with pip:

$ pip install django-balancer

Then, add the desired router to your DATABASE_ROUTERS setting. Also add any settings required by that router.

DATABASE_ROUTERS = ['balancer.routers.WeightedMasterSlaveRouter']
DATABASE_POOL = {
    'default': 1,
    'slave1': 2,
}
MASTER_DATABASE = 'default'

Make sure to configure your DATABASES setting to add the new slave database.

And with that, you should be all set!

Create Your Own

You can also use the classes and mixins inside the django-balancer project to create your own routers. If you come up with one that you'd like to share, fork me on Github. I'd be glad to pull it into the project.

Django Admin Customization Examples

Mon, 04 Oct 2010 22:50:03 -0500

I've worked on a couple of projects recently that required some customization of the Django admin. One of the things that I love about Django's built-in admin is that it is created with a very extensible class-based structure. Here are some of the ways I was able to customize the admin to add functionality.

Adding Buttons to the Change View

One of the major features needed for a project was the ability to manage a workflow for a model. The model progressed through different status stages, and the admin team needed to be able to make decisions based on that current status. To handle this, I added some new buttons to the model's change view, and used my ModelAdmin and ModelForm subclasses to move the model around in the workflow accordingly.

Customizing the Template

In order to differentiate the workflow management options from the standard Admin delete & save buttons, I added a row of buttons directly above the standard delete & save row. I created a file in my templates directory with the path admin/myapp/mymodel/change_form.html that looked something like this:

{% extends "admin/change_form.html" %}

{% block after_related_objects %}
  {% if original.pk %}
    <div class="submit-row">
      <p class="deletelink-box">Manage this request</p>
      {% if original.requested %}
        <input type="submit" value="Approve" name="approve">
        <input type="submit" value="Deny" name="deny">
      {% endif %}
      {% if original.submitted %}
        <input type="submit" value="Accept" name="accept">
        <input type="submit" value="Return" name="return">
      {% endif %}
      {% if original.submitted or original.assigned %}
        <input type="submit" value="Cancel" name="cancel">
      {% endif %}
      {% if original.accepted %}
        <input type="submit" value="Complete" name="complete">
      {% endif %}
      {% if original.cancelled %}
        <input type="submit" value="Reopen" name="reopen">
      {% endif %}
    </div>
  {% endif %}
{% endblock %}

The original context variable refers to the object being created or changed. I checked for a primary key because I only wanted the extra buttons to show up on edits. The .requested, .submitted, etc. attributes are properties on my model that check a status field.

Handling the Extra Submit Options

To handle these extra submit options, I created a ModelForm subclass. On my ModelAdmin, I set form = MyAdminForm to point to my new ModelForm. On the form, I implemented the clean method to look for the workflow actions:

def clean(self, *args, **kwargs):
    cleaned_data = super(MyAdminForm, self).clean(*args, **kwargs)

    if self.data.has_key('approve'):
        # Handle the approval logic here
        # ...
        # If there's a problem with the action, you can raise a
        # forms.ValidationError("Your message here.")
    elif self.data.has_key('deny'):
        # Handle the denial logic here
        # ...

    # Etc...

    return cleaned_data

In order to give the user effective feedback that their action was success, I wanted to customize the response message given to them after they click the custom button. To do that, I went back to my ModelAdmin subclass and overrode the response_change method:

def response_change(self, request, obj, *args, **kwargs):
    if request.POST.has_key("approve"):
        msg = 'The %(name)s "%(obj)s" was approved.'  % {
            'name': obj._meta.verbose_name, 'obj': obj
        }
        self.message_user(request, msg)
        return HttpResponseRedirect("../")
    if request.POST.has_key("deny"):
        msg = 'The %(name)s "%(obj)s" was denied.' % {
            'name': obj._meta.verbose_name, 'obj': obj
        }
        self.message_user(request, msg)
        return HttpResponseRedirect("../")

    # Etc...

    if request.POST.has_key("reopen"):
        msg = 'The %(name)s "%(obj)s" was reopened. ' % {
            'name: obj._meta.verbose_name, 'obj': obj
        }
        msg += 'You may manage it below.'
        self.message_user(request, msg)
        if request.REQUEST.has_key('_popup'):
            return HttpResponseRedirect(request.path + "?_popup=1")
        else:
            return HttpResponseRedirect(request.path)

    return super(MyModelAdmin, self).response_change(
        request, obj, *args, **kwargs
    )

Adding Modules to the App Index

In order to create a dashboard for the workflow-based app, I wanted to add some modules to the app index that would act as a queue for items in a certain status. I decided to handle this through template tags:

@register.inclusion_tag('admin/myapp/_recent_requests.html')
def recent_requests():
    return {'requests':
            MyModel.pending.order_by('-requested')}

I added an admin/myapp/app_index.html template that looked like this:

{% extends "admin/app_index.html" %}

{% load myapp_admin_tags %}

{% block sidebar %}
  <div id="content-related">
    <div class="module">
      <h2>Shortcuts</h2>
      <ul>
        <li><a href="{% url admin:myapp_mymodel_changelist %}?ot=desc&is_closed__exact=0&o=3">
          Open requests
        </a></li>
        <!-- Etc... -->
      </ul>
      {% recent_submissions %}
    </div>
  </div>
  <div id="content-secondary">
    {% recent_requests %}
  </div>
{% endblock %}

Creating a Site-Specific Admin

On one project with several sub-sites, I needed to create a site-specific admin so that administrators of one site could use the Django admin without interfering with content from the other sites. To do this, I created a secondary admin site.

Addind an Additional Admin Site

I wanted users of the site-specific admin to be a member of a specific group. With this in mind, I created a subclass of AdminSite with an overridden has_permission method:

from django.contrib.admin.sites import AdminSite

class LocalAdminSite(AdminSite):
    def has_permission(self, request):
        """
        Checks to make sure that the user is authenticated and is either a
        staff member or a member of the 'Content editor' group.
        """
        from django.contrib.auth.models import Group
        ce_group = Group.objects.get(name='Content editor')
        return request.user.is_authenticated() and (
            request.user.is_staff or ce_group in request.user.groups.all()
        )

local_admin = LocalAdminSite(name='local')

I created an app within my project called admin and added the above code to admin/sites.py. Then, to use my site, I could simply do this:

from myproject.admin.sites import local_admin

Custom Autodiscovery

I wanted to be able to add a local_admin.py file in each app that should be available in the site-specific admin. To do this, I took the admin.autodiscover() method and slightly modified it:

def autodiscover_module(module_name):
    """
    Auto-discover the specified module in INSTALLED_APPS and fail silently
    when not present. This forces an import on them to register anything they
    may want.
    """

    import copy
    from django.conf import settings
    from django.contrib.admin.sites import site
    from django.utils.importlib import import_module
    from django.utils.module_loading import module_has_submodule

    for app in settings.INSTALLED_APPS:
        mod = import_module(app)
        # Attempt to import the app's admin module.
        try:
            before_import_registry = copy.copy(site._registry)
            import_module('%s.%s' % (app, module_name))
        except:
            # Reset the model registry to the state before the last import as
            # this import will have to reoccur on the next request and this
            # could raise NotRegistered and AlreadyRegistered exceptions
            # (see #8245).
            site._registry = before_import_registry

            # Decide whether to bubble up this error. If the app just
            # doesn't have an admin module, we can ignore the error
            # attempting to import it, otherwise we want it to bubble up.
            if module_has_submodule(mod, module_name):
                raise

That way I can use autodiscover_module('local_admin') to automatically register all of my local admin modules. I stuck it in urls.py - not the most semantically correct location, but it's convenient.

Excluding the Sites Attribute

To make things nice and DRY, I wanted to use the existing admin.py definitions, but limit them to the current site and strip the sites attribute from them. To do this, I got a little crazy and wrote a function to automatically exclude them. I am absolutely certain that this could be refactored to be more elegant, I just haven't had the time to do it yet.

def exclude_sites(admin):
    """
    Remove occurrences of site and sites from admin fields.

    Returns: fields, fieldsets, list_display, list_filter
    """
    exclude = ('sites', 'site')
    new_fields = []
    new_fieldsets = []
    new_list_display = []
    new_list_filter = []

    # Take sites out of the fields
    if super(admin.__class__, admin).fields:
        new_fields = list(super(admin.__class__, admin).fields)
        for exclusion in exclude:
            if exclusion in new_fields:
                new_fields.remove(exclusion)

    # Take sites out of the fieldsets
    if super(admin.__class__, admin).fieldsets:
        for fieldset in super(admin.__class__, admin).fieldsets:
            title = fieldset[0]
            fields = list(fieldset[1]['fields'])
            for exclusion in exclude:
                if exclusion in fields:
                    fields.remove(exclusion)
            new_fieldsets.append((title, {'fields': tuple(fields)}))

    # Take sites out of the list_display
    if super(admin.__class__, admin).list_display:
        new_list_display = list(super(admin.__class__, admin).list_display)
        for exclusion in exclude:
            if exclusion in new_list_display:
                new_list_display.remove(exclusion)

    # Take sites out of the list_filter
    if super(admin.__class__, admin).list_filter:
        new_list_filter = list(super(admin.__class__, admin).list_filter)
        for exclusion in exclude:
            if exclusion in new_list_filter:
                new_list_filter.remove(exclusion)

    return tuple(new_fields), tuple(new_fieldsets), tuple(new_list_display), tuple(new_list_filter)

To make use of this, I created a base class that I could then subclass in all of my local_admin.py modules. I also overrode the queryset in this base class to use the local manager I assigned to each model.

class LocalModelAdmin(admin.ModelAdmin):

    def __init__(self, *args, **kwargs):
        """
        Runs the fieldsets, list_displays, and list_filters through the
        exclude_sites function, which returns the attributes without the
        'site' or 'sites' field.
        """
        super(LocalModelAdmin, self).__init__(*args, **kwargs)
        self.fields, self.fieldsets, self.list_display, self.list_filter = \
            exclude_sites(self)

    def queryset(self, request):
        """
        Returns a queryset with results filtered by the current site. Requires
        a manager called 'local' to be assigned to the model that does the
        filtering.
        """
        qs = self.model.local.get_query_set()
        ordering = self.ordering or ()
        if ordering:
            qs = qs.order_by(*ordering)
        return qs

Then, in my local_admin.py files I used multiple inheritance to inherit functionality both from this base class and the original admin.py module definition.

from myproject.admin.sites import local_admin
from myproject.myapp.admin import ItemAdmin

class LocalItemAdmin(LocalModelAdmin, ItemAdmin):
    pass

local_admin.register(Item, LocalItemAdmin)

Adjusting the Queryset Used in Related Fields

Related fields still show results from other sites, however. To fix this, I extended the formfield_for_dbfield method on my LocalModelAdmin subclass.

class LocalItemAdmin(LocalModelAdmin, ItemAdmin):
    def formfield_for_dbfield(self, db_field, **kwargs):
        if db_field.name == 'my_field':
            kwargs['queryset'] = MyModel.local.all()
        return super(LocalItemAdmin, self).formfield_for_dbfield(
            db_field, **kwargs
        )

local_admin.register(Item, LocalItemAdmin)

Class-based Views with Django-Baseviews

Tue, 28 Sep 2010 23:56:48 -0500

I've always loved the concept of class-based views because views often become repetitive. There are a lot of common operations in a view that work best when they are defined once and reused. Class inheritance, in my opinion, is the best way to keep views DRY and allow you to focus on what makes each view different.

At first, I used to break out pieces of functionality into their own functions in the views.py file. This worked okay, but I had to pass a lot of data around in the arguments and I had to do a lot of imports if I had functions from other app's views that I wanted to use in this app. This quickly became tedious.

After seeing django-haystack use class-based views and really loving the way I could customize their behavior for my own project, I decided to start building some base views of my own to build on. While working on new Pegasus News apps, I started building up a set of class-based views that helped me stop repeating everything that the views had in common. Once the base views were in place, the process of creating new views for my apps because a little quicker and more organized.

I've received permission to open-source the app, so it's now out on Github. I've already received a few contributions (from Justin Quick, Jannis Leidel, and Chuck Harmston - thanks guys!) and I'd be happy to receive more. I've been using the project in production on Pegasus News since June, and it has worked well so far.

How I'm using django-baseviews

For the apps I'm using baseviews with on Pegasus, I start out with a BasicView for the index. Then, for any views that deal with a specific model instance I create a base class for the app that takes the slug or id from the url pattern, grabs the object, and adds it to the class instance.

class ItemView(BasicView):

    def __init__(self, request, slug):
        self.item = get_object_or_404(Item, slug=slug)
        super(ItemView, self).__init__(request)

I then subclass the item-specific view as the base for any views that deal with a specific item. This gives me the self.item attribute on my subclasses without having to repeat the get_object_or_404 call for every view. This is very a basic example, and it's just the start of what you can do.

For a real world example - in one of my apps, I had a view that simply displayed all instances of a model. Later on after the app was launched, I found out that I needed to create a page that showed all the instances of that model within a specific category. All I had to do was move the queryset creation out into its own method, create a quick subclass of my original view, and override the queryset method on the subclass to filter by the requested category.

Plans for continued development

After watching Ben Firshman's Alternative Views talk from Djangocon US, I've learned about the concept of mixins - using Python's multi-inheritance to add small pieces of functionality. For example, I could create a mixin that renders everything to JSON, or one to render in XML, or one for YAML. Then, when subclassing FormView I could add the JSON mixin to the front of the new class's bases::

class ItemFormView(RenderJson, FormView):
    ...

Assuming that all RenderJson does is override the render method, this would allow the RenderJson class to handle rendering and FormView to handle all the other methods.

I'd also like to take a look at Jacobian's class-based generic views and do something similar for baseviews.

I'm going to keep building up baseviews as I use it for other projects, and I encourage you to contribute any view classes to baseviews that you think would help speed up development for others. Once I get enough view classes, I'll start to split views.py up into different files based on the type of views contained.

The image at the top of this post is from Natalie Dee, a funny and exceedingly irreverent web comic.

Django on uWSGI and Nginx

Tue, 14 Sep 2010 00:19:31 -0500

I recently moved Pegasus News from Perlbal, Lighttpd, and Apache to Nginx and uWSGI. We balance the traffic between 3 physical servers, and the systems were struggling under the load even after weeks of Apache conf tweaking. We began having issues with excessively slow page loads, request timeouts, and intermittent errors with OGR transformations.

I decided to move us to a lighter application server so that we could get the most out of our system resources, and after a lot of research and testing I chose uWSGI. While I was at it, I decided to replace Perlbal and Lighttpd with Nginx because of its great configuration syntax and excellent performance. I also upgraded us to Ubuntu 10.04 and Postgres 8.4.

The result was a resounding success! Memory usage and CPU load on each of the web nodes dropped dramatically. Swap usage dropped to almost nothing. The overall responsiveness of the site improved noticeably, and the timeout errors and OGR failures disappeared entirely.

If you'd like to give this stack a try, read on for an overview of the setup on Ubuntu 10.04. I'm using the standard Ubuntu source package for Nginx, but modifying it slightly and them installing it with dpkg-buildpackage.

This post is just about the setup relevant to Nginx and uWSGI. If you need a more complete server setup guide, try my Provisioning a new server post.

uWSGI

Before we build Nginx, uWSGI needs to be compiled so that its module can be included in the Nginx build.

$ sudo apt-get install build-essential python-dev libxml2-dev
$ wget http://projects.unbit.it/downloads/uwsgi-0.9.5.4.tar.gz
$ tar -xzf uwsgi-0.9.5.4.tar.gz
$ cd uwsgi-0.9.5.4
$ make -f Makefile.Py26

Copy the executable to the local sbin directory.

$ sudo cp uwsgi /usr/local/sbin

Also, copy the default uwsgi settings to the /etc/nginx directory.

$ sudo mkdir /etc/nginx
$ sudo cp nginx/uwsgi_params /etc/nginx
$ cd ..

Nginx

We need to slightly modify the nginx package from Ubuntu to add uWSGI (and, optionally, SSL).

$ sudo apt-get install libssl-dev
$ sudo apt-get build-dep nginx
$ apt-get source nginx
$ cd nginx-0.7*
$ emacs debian/rules

Add the following lines to the end of the configure options. Make sure to include backslashes so that all the options are interpreted as being on one line. If you don't need SSL, ignore that line.

    --with-http_ssl_module \
    --add-module=$(CURDIR)/../uwsgi-0.9.5.4/nginx

Build, install, and hold the packages.

$ dpkg-buildpackage
$ cd ..
$ sudo dpkg -i nginx*.deb
$ echo "nginx hold" | sudo dpkg --set-selections
$ echo "nginx-dbg hold" | sudo dpkg --set-selections

Starting with nginx-0.8.41, you can add something like --http-uwsgi-temp-path=/var/lib/nginx/uwsgi to the debian/rules so that the temp files are kept in the right place. Until then (probably a later version of Ubuntu), you'll need to create a /usr/local/nginx/uwsgi_temp folder to use for the temp files.

$ sudo mkdir -p /usr/local/nginx/uwsgi_temp

Supervisor

To manage the uWSGI processes, I use Supervisor. In Ubuntu 10.04, you can simply install it with apt-get.

$ sudo apt-get install supervisor

Configuration

Supervisor

To configure uWSGI, I use command-line options in my Supervisor config. The example below is similar to what I use in production for Pegasus, but you'll want to take a look at the uWSGI docs and tweak for your situation.

[program:myapp]
command=/usr/local/sbin/uwsgi
  --home /home/myuser/.virtualenvs/myapp/
  --module myapp.deploy.wsgi
  --socket 10.1.2.3:10000
  --pythonpath /sites/myapp.com/code/myapp
  --processes 5
  --master
  --harakiri 120
  --max-requests 5000
directory=/sites/myapp.com/code/myapp
environment=DJANGO_SETTINGS_MODULE='myapp.settings'
user=www-data
autostart=true
autorestart=true
stdout_logfile=/sites/myapp.com/logs/uwsgi.log
redirect_stderr=true
stopsignal=QUIT

The module I specify in the --module option simply contains:

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

The IP address in the --socket option is the server's IP address on the interface reserved for local network traffic. On Rackspace, eth1 is your local interface. Use the ifconfig command to find your IP address on the local interface. If you're just using uWSGI on localhost, you can use something like --socket /sites/myapp.com/var/run/myapp.sock instead to avoid the overhead of the full TCP stack.

My value for --harakiri is rather high, and my value for --max-requests is rather low. You may not even need either of these options, but I'm using them to solve some problems specific to Pegasus.

Nginx

For the site-specific Nginx config, I'm using something like this:

upstream myapp {
    server 10.1.2.3:10000;
    server 10.1.2.4:10000;
    server 10.1.2.5:10000;
}

server {
    listen 80;
    listen 443;
    server_name myapp.com www.myapp.com;

    access_log /sites/myapp.com/logs/nginx-access.log;
    error_log /sites/myapp.com/logs/nginx-error.log;

    root /sites/myapp.com/public;

    ssl_certificate /sites/myapp.com/ssl/myapp.crt;
    ssl_certificate_key /sites/myapp.com/ssl/myapp.key;

    location / {
        # This checks for a file called simply "downtime" in the public
        # directory, and puts up the downtime page if it exists.
        if (-f /sites/myapp.com/public/downtime) {
            return 503;
        }

        uwsgi_pass myapp;
        include uwsgi_params;
    }

    location /media {
        # This makes static media available at the /media/ url.  The
        # media will continue to be available during site downtime,
        # allowing you to use styles and images in your maintenance page.
        alias /sites/myapp.com/public/media;
    }

    error_page 502 503 504 @maintenance;
    location @maintenance {
        # In this example, there's a directory in the site media files
        # called "downtime" that contains a "maintenance.html" file and
        # any styles and images needed for the maintenance page.
        root /sites/myapp.com/public/media/downtime;
        rewrite ^(.*)$ /maintenance.html break;    
    }
}

If you're just using uWSGI on localhost, then skip the upstream section and use something like uwsgi_pass unix:///sites/myapp.com/var/run/myapp.sock; in the root location definition instead.

Firing it up

Restart Supervisor with sudo /etc/init.d/supervisor restart, and it should reload the config and bring the uWSGI processes up. Restart Nginx with sudo /etc/init.d/nginx restart, and you should now be able to reach your site. If not, take a look at the logs and start troubleshooting.

Setting up a template_postgis on Lucid

Mon, 19 Jul 2010 15:59:18 -0500

I wasn't able to find instructions in the Django docs for setting up a template_postgis database with postgis-1.5 and Postgres 8.4 on Ubuntu Lucid (10.04). Below is what worked for me.

GeoDjango installed via Ubuntu 10.04 packages

#!/usr/bin/env bash
POSTGIS_SQL_PATH=`pg_config --sharedir`/contrib
createdb -E UTF8 template_postgis # Create the template spatial database.
createlang -d template_postgis plpgsql # Adding PLPGSQL language support.
psql -d postgres -c "UPDATE pg_database SET datistemplate='true' WHERE datname='template_postgis';"
psql -d template_postgis -f $POSTGIS_SQL_PATH/postgis.sql # Loading the PostGIS SQL routines
psql -d template_postgis -f $POSTGIS_SQL_PATH/spatial_ref_sys.sql
psql -d template_postgis -c "GRANT ALL ON geometry_columns TO PUBLIC;" # Enabling users to alter spatial tables.
psql -d template_postgis -c "GRANT ALL ON spatial_ref_sys TO PUBLIC;"

GeoDjango built from source

Use the above script, but change POSTGRIS_SQL_PATH to:

POSTGIS_SQL_PATH=`pg_config --sharedir`/contrib/postgis-1.5

Make sure to run sudo ldconfig after compiling everything, or else postgis won't be able to find geos. You'll see errors like this:

psql:/usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql:59: ERROR:  could not load library "/usr/lib/postgresql/8.4/lib/postgis-1.5.so": libgeos_c.so.1: cannot open shared object file: No such file or directory

Bitten by Python's Pass-By-Reference

Wed, 14 Jul 2010 17:19:41 -0500

I just got bitten by an interesting bug related to Python's pass-by-reference feature. I had a class method (my_method) with a keyword argument (my_kwarg) that defaulted to {}. I thought that each time my_method was called, if my_kwarg wasn't passed then it would default to an empty dict.

In reality, when the class was read by Python it evaluated the {} and made the default value a reference to the dict in memory. So each time my_method was called without passing my_kwarg, it was defaulting to that memory reference which contained data from a previous use. Here's an example class:

class MyClass(object):
    def my_method(self, my_kwarg={}):
        return my_kwarg

And here's an iPython shell session illustrating what I mean:

In [2]: first_class = MyClass()

In [3]: kwarg = first_class.my_method()

In [4]: kwarg
Out[4]: {}

In [5]: kwarg.update({'foo': 'bar'})

In [6]: kwarg
Out[6]: {'foo': 'bar'}

In [7]: kwarg = first_class.my_method()

In [8]: kwarg
Out[8]: {'foo': 'bar'}

In [9]: second_class = MyClass()

In [10]: other_kwarg = second_class.my_method()

In [11]: other_kwarg
Out[11]: {'foo': 'bar'}

The fact that the kwarg wasn't defaulting to a fresh dict each time led to a lot of confusion before I realized what was happening! There are a couple of ways to solve the issue, but the easiest is probably this:

class MyClass(object):
    def my_method(self, my_kwarg=None):
        my_kwarg = my_kwarg or {}
        return my_kwarg

Provisioning a new Ubuntu server for Django

Fri, 25 Jun 2010 02:29:58 -0500

I've been a long-time satisfied user of Webfaction, but recently I've had a strong urge to move to VPS hosting so that I can have greater control over the environment. After some research, I went with Rackspace Cloud because of the incredibly cheap low-end options. My site doesn't use a huge amount of bandwidth, so Rackspace looks to be the most feature-packed and still cost-effective option.

A friend of mine, Kevin Whitaker, recently posted a great article about getting up and running with Django in a server environment for testing or production. He used Ubuntu, Postgres, Nginx, and FastCGI to make up his stack. I've never set up Nginx before, so his post was a great help in getting Nginx configured. My stack is slightly different, however, since I prefer to use Gunicorn instead of FastCGI and I use supervisord to manage my processes. I also use virtualenv to manage dependencies like Django itself and psycopg2.

I'm including my install notes below so that I can provide a complete picture of one way of provisioning a new server from build to deploy. My site is relatively low-traffic, so I'm not going to go into postgres tweaking, connection pooling, caching, etc. I'll save those topics for another post.

These notes were made on a Rackspace Cloud instance, but they should apply to a variety of other VPS hosts with only minor modifications. If you've got tips on things that need to be changed for other providers, let me know and I'll edit the post. I want this to stay up to date for awhile.

After the build is complete

Configuration

My first step was to log into my new instance using SSH. You should get an email from Rackspace with the IP address and root password.

$ ssh root@123.45.67.890

root@123.45.67.890's password:

Welcome to Ubuntu!

I then installed my preferred editor - obviously an optional step.

$ apt-get install emacs23-nox
$ export EDITOR=emacs

Then I used visudo to set up sudo access the way I like it.

$ visudo

Under this line:

Defaults        env_reset

I added the following lines to preserve my editor and Django environment settings:

Defaults        env_keep += "EDITOR VISUAL"
Defaults        env_keep += "DJANGO_SETTINGS_MODULE PYTHONPATH"

Then I changed this line:

%sudo   ALL=(ALL) ALL

To:

%admin  ALL=(ALL) ALL

This allows users with the group admin to use sudo for all commands.

Then I added my first user and group. I typically prefer to use a separate user account for each developer or administrator that will be accessing the box, and add them to appropriate groups for access.

$ adduser myuser
$ addgroup admin
$ adduser myuser admin

Then I switched to my new user, and added my public key so that I can SSH without requiring a password. If you're not familiar with this, the guide at Github is quick and to the point.

$ su - myuser
$ mkdir .ssh
$ cd .ssh
$ wget http://mydomain.com/path/to/my/pubkey/id_rsa.pub
$ mv id_rsa.pub authorized_keys

Then, I logged out and made sure I can SSH in using the new user with no problems. Once that's done, I locked down the root account for security (probably because I'm a long time Ubuntu user and it just feels wrong any other way).

$ sudo passwd -l root

I also make my editor choice permanent.

$ sudo update-alternatives --config editor

Nginx

With that taken care of, I was ready to install Nginx.

$ sudo apt-get install nginx
$ sudo /etc/init.d/nginx start
$ sudo emacs /etc/nginx/nginx.conf

I used this Slicehost article to learn about the basic nginx conf options.

Postgres

Then I installed postgres. I first changed the password on the postgres user for security.

$ sudo apt-get install postgresql
$ sudo passwd postgres
$ sudo -u postgres psql

postgres=# \password postgres
postgres=# \q

Then I set up a new user to use for my site.

$ sudo -u postgres createuser myproject
$ sudo -u postgres psql

postgres=# \password myproject
postgres=# \q

I use a Unix domain socket for my postgres connection, so I have to edit pg_hba.conf to allow md5 login from domain sockets.

$ sudo emacs /etc/postgresql/8.4/main/pg_hba.conf

I change the lines that say:

# "local" is for Unix domain socket connections only                                                
local   all         all                               ident

To:

# "local" is for Unix domain socket connections only                                                
local   all         all                               md5

Then:

$ sudo /etc/init.d/postgresql-8.4 restart

(Substitute your favorite VCS here)

Now for my VCS. I use the Git PPA.

$ sudo apt-get install python-software-properties
$ sudo add-apt-repository ppa:git-core/ppa
$ sudo apt-get update
$ sudo apt-get install git-core

Setting up a Django site

Setting up the structure

First, I pulled over my git repositories. I use the /code directory for hosting the master copies of my repositories to use as a hub. I keep the live checkouts in /sites/mydomain.com/code/.

$ sudo mkdir /code
$ sudo chown myuser:admin /code
$ cd /code
$ git clone --mirror olddomain.com:/code/repository_name.git

$ sudo mkdir /sites
$ sudo chown myuser:admin /sites
$ cd /sites
$ mkdir -p mydomain.com/{code,public,logs,backup}
$ cd mydomain.com
$ sudo chown :www-data logs public
$ sudo chmod g+w logs public
$ git clone /code/repository_name.git code/

Initialized empty Git repository in /sites/mydomain.com/code/.git/

Media

I set up the media and made sure www-data could access it and write to the root directory.

$ cd public
$ ln -s ../code/projectname/media
$ sudo chown :www-data media
$ sudo chmod g+w media

Then I synced the media from my old server, so that user-created media wouldn't be lost.

$ rsync -avz --progress old.domain.com:~/path/to/media/ media/

Restoring the database

The next step was backing up and restoring the database.

$ ssh old.domain.com
$ pg_dump dbname | bzip2 > dbname.`date +%Y%m%dT%H%M%S`.sql.bz2
$ exit
$ ssh new.domain.com
$ cd /sites/mydomain.com/backup/
$ scp old.domain.com:~/dnmame.<timestamp>.sql.bz2 ./
$ sudo su postgres
$ createdb dbname
$ bzcat dnmame.<timestamp>.sql.bz2 | psql dbname
$ psql dbname

postgres=# GRANT ALL ON DATABASE dbname TO myproject;
postgres=# \q

$ exit

Installing dependencies

I then created my virtualenv and installed my dependencies.

$ sudo apt-get install build-essential python-dev libpq-dev
$ sudo apt-get install python-setuptools
$ sudo easy_install -U pip
$ sudo pip install virtualenv
$ sudo pip install virtualenvwrapper
$ mkdir /sites/.virtualenvs
$ emacs ~/.bashrc

At the bottom of .bashrc:

export WORKON_HOME=/sites/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh

Then, back in the shell:

$ . ~/.bashrc
$ mkvirtualenv projectname
$ workon projectname
$ pip install -r /sites/mydomain.com/code/requirements.txt
$ deactivate

PIL is a tough one. I usually just install it globally. Since I install my virtualenv's with --no-site-packages, I need to symlink PIL into the site-packages.

$ sudo apt-get install python-imaging
$ workon projectname
$ cdsitepackages
$ ln -s /usr/lib/python2.6/dist-packages/PIL
$ ln -s /usr/lib/python2.6/dist-packages/PIL.pth

Then, I edited my project's settings.py to reflect the new environment setup.

Finally, I added ntpdate to keep my server-s clock in sync.

$ sudo dpkg-reconfigure tzdata
$ sudo apt-get install ntpdate
$ sudo crontab -e

30 23 * * * /usr/sbin/ntpdate ntp.ubuntu.com > /dev/null

Nginx

I added an nginx.conf to my source control.

$ emacs /sites/mydomain.com/code/deploy/nginx.conf

The Nginx conf:

server {
  listen 80;
  server_name www.mydomain.com;
  rewrite ^/(.*) http://mydomain.com/$1 permanent;
}

server {
  listen 80;
  server_name mydomain.com;

  access_log /sites/mydomain.com/logs/access.log;
  error_log /sites/mydomain.com/logs/error.log;

  location /media {
    root /sites/mydomain.com/public;
  }

  location / {
    proxy_pass http://127.0.0.1:29000;
  }
}

I then symlinked it into /etc/nginx/sites-available/ and sites-enabled/.

$ sudo ln -s /sites/mydomain.com/code/deploy/nginx.conf /etc/nginx/sites-available/mydomain.com
$ sudo ln -s /etc/nginx/sites-available/mydomain.com /etc/nginx/sites-enabled/mydomain.com
$ sudo /etc/init.d/nginx restart

Gunicorn

Gunicorn should be installed in your virtualenv as part of your requirements.txt. If you're not using virtualenv and pip, or zc.buildout, then you should definitely read up on them. They are a vital part of any serious Django stack.

$ emacs /sites/mydomain.com/code/deploy/gunicorn.conf.py

I already had a simple Gunicorn conf in my source control, which I simply modified for the new environment.

bind = "127.0.0.1:29000"
logfile = "/sites/mydomain.com/logs/gunicorn.log"
workers = 3

Supervisord

$ sudo pip install supervisor
$ sudo emacs /etc/supervisord.conf

Here's a basic config file:

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)

[supervisord]
logfile=/var/log/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10          ; (num of main logfile rotation backups;default 10)
loglevel=info               ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false              ; (start in foreground if true;default false)
minfds=1024                 ; (min. avail startup file descriptors;default 1024)
minprocs=200                ; (min. avail process descriptors;default 200)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

[program:myproject]
command=/sites/.virtualenvs/myproject/bin/gunicorn_django -c deploy/gunicorn.conf.py
directory=/sites/mydomain.com/code
user=www-data
autostart=true
autorestart=true
stdout_logfile=/sites/mydomain.com/logs/supervisord.log
redirect_stderr=true

Then I created an init.d script:

$ sudo emacs /etc/init.d/supervisord

Here's what I used:

# Supervisord auto-start
#
# description: Auto-starts supervisord
# processname: supervisord
# pidfile: /var/run/supervisord.pid

SUPERVISORD=/usr/local/bin/supervisord
SUPERVISORCTL=/usr/local/bin/supervisorctl

case $1 in
start)
        echo -n "Starting supervisord: "
        $SUPERVISORD
        echo
        ;;
stop)
        echo -n "Stopping supervisord: "
        $SUPERVISORCTL shutdown
        echo
        ;;
restart)
        echo -n "Stopping supervisord: "
        $SUPERVISORCTL shutdown
        echo
        echo -n "Starting supervisord: "
        $SUPERVISORD
        echo
        ;;
esac

Then I finished up the init.d script.

$ sudo chmod +x /etc/init.d/supervisord
$ sudo update-rc.d supervisord defaults
$ sudo /etc/init.d/supervisord start

If all goes well, you can check the status on your site.

$ sudo supervisorctl status

myproject                    RUNNING    pid 11616, uptime 0:00:03

You can also grep for the gunicorn processes.

$ ps -ef | grep gunicorn

Testing

To test this new setup, you can add domain name overrides to the /etc/hosts file on your local machine.

# Do this on your local machine - not your server
$ sudo emacs /etc/hosts

123.45.67.890 mydomain.com

Now you can go to your browser and access mydomain.com. You should see your site.

To restart your site, use:

$ sudo supervisorctl restart myproject

Make sure to change your /etc/hosts back!

Using virtualenvwrapper to start processes and swap config files

Tue, 22 Jun 2010 22:45:31 -0500

I do my Django development work locally on OS X, so I have several different daemons installed on my machine. I used to keep Postgres, MySQL, lighttpd, memcached, and more set up to autostart and run continuously, but I didn't like the burden on performance (real or imagined). Also, I switch between projects frequently and I often need to switch config files based on the project I'm about to work on.

My solution is to use the end-user customization hooks that Doug Hellmann's excellent virtualenvwrapper provides. If you're not familiar with virtualenvwrapper, it makes it easy to manage your virtualenvs in one place and switch between them quickly and conveniently. The project provides several scripts that are meant to be added to by the end-user to customize behavior.

Configuration

For my configuration files, I create multiple versions with the changes needed for each project, and I then create a symlink to the version I want at the moment. For example, with lighttpd I'll create an /etc/lighttpd/project-a.conf and an /etc/lighttpd/project-b.conf, and then create a symlink called /etc/lighttpd/lighttpd.conf that points to the one I need.

In OS X, I use launchd .plist files that have the OnDemand option set to True. I've posted the .plists I'm using currently to Github. Put these in your /Library/LaunchDaemon/ folder and then load them using the command:

sudo launchctl load /Library/LaunchDaemon/[plist_filename]

This registers the processes with OS X's launchd system. Afterwards, you can start and stop them using the commands:

sudo launchctl start [label]
sudo launchctl stop [label]

Note that, when starting and stopping, you don't have to use the path to the .plist file. Just use the daemon's label, which is typically the filename without the path or the .plist part.

Env-specific virtualenvwrapper scripts

I use the project-specific postactivate and postdeactivate scripts to swap in the right config and start the daemons that I need for a particular project. You'll find these in the $WORKON_HOME/[project_name]/bin directory. If your virtualenv is a little old, those scripts may not already be present. As long as you're up-to-date with a recent version of virtualenvwrapper, you can simply add those scripts to your virtualenv's bin directory yourself.

For $WORKON_HOME/project-a/bin/postactivate, I use a script similar to this:

#!/bin/bash
# This hook is run after this virtualenv is activated.

# Postgres
sudo launchctl start org.postgresql.postmaster

# Memcached
sudo launchctl start com.danga.memcached

# Lighttpd
sudo mv /etc/lighttpd/lighttpd.conf /etc/lighttpd/lighttpd.conf.bak
sudo ln -s /etc/lighttpd/project-a.conf /etc/lighttpd/lighttpd.conf
sudo launchctl start net.lighttpd

cd ~/code/webapps/project-a-dir/project-a

I like having the cd command there so that I can immediately run manage.py commands.

For $WORKON_HOME/project-a/bin/postdeactivate, I use:

#!/bin/bash
# This hook is run after this virtualenv is deactivated.

# Postgres
sudo launchctl stop org.postgresql.postmaster

# Memcached
sudo launchctl stop com.danga.memcached

# Lighttpd
sudo launchctl stop net.lighttpd

So now, when I run workon project-a, postgres, memcached, and lighttpd all come to life. When I run workon project-b, the postdeactivate script for project-a runs, and then the postactivate script for project-b runs. This effectively switches my lighty conf and restarts the process so it's immediately ready for me. When I deactivate, the processes shut down and my computer can take advantage of the free memory.

General virtualenvwrapper scripts

Virtualenvwrapper also provides global hooks directly in $WORKON_HOME. Brian Rosner suggested making the config switching generic so you wouldn't have to create scripts for each environment. I haven't tried this yet, but it definitely wouldn't be hard to do. The environment variable $VIRTUAL_ENV is available within postactivate scripts, so $WORKON_HOME/postactivate could include a block that looks something like this:

if [ -f /etc/lighttpd/$VIRTUAL_ENV.conf ]; then
    sudo ln -s /etc/lighttpd/$VIRTUAL_ENV.conf /etc/lighttpd/lighttpd.conf
fi

All you would have to do is name your config files after your environments, and they would automatically be symlinked into place when you activate the environment.