Albertoconnor . ca Blog

Viewport: Making webpages look great on iOS devices

Albert O'Connor — Thu, 19 Jan 2012 08:52:07 -0500

<meta name="viewport" content="width=device-width, initial-scale=1.0">

This is the default viewport meta tag found in the very useful html5 boilerplate.

If you are like me, and don't initially understand exactly what it does, this default will get you into trouble.

The main issue is your website will look great on your computer but when you look at it on an iPhone or iPad it will look small, squished even depending on your CSS. If you happen be making a website which is pixel perfect for the iPhone resolution, then everything will look good, but most websites I develop are meant to be viewed on computer browser and on mobile devices, so knowing what the viewport meta tag does helps you look great everywhere.

What is the viewport

The viewport is the space your webpage will rendered into. Think of it like the size of the browser window. When you zoom, you are zooming into the viewport, but the size of the viewport doesn't change. You can zoom out of the viewport either, though if the page is wider than the viewport you can scroll to see more.

The above viewport meta tag says make the viewport the same size as the device's screen width and a 1:1 scale, pixel perfect. In the case of an iPhone that means 320px wide. If your CSS is designed for 800px of minimum width your page is guaranteed to look bad.

You can specify any default width in pixels, and default scale or let the scale be determined automatically.

For more detail you can consult the in-depth Apple documentation on the viewport meta tag.

Looking good everywhere

If you have designed your website to be rendered at a minimum width of say 1000 pixels, you can make it look great on a computer and on a mobile device with the following meta tag:

<meta name="viewport" content="width=1000">

This will make the viewport, the space the website renders into, 1000 pixels wide, and default the scaling to show the entire width.

By default if no viewport meta tag is provided a default viewport width 985 pixel is used, which will actually works well for most sites. If you want your site to look really good and you have an optimal width of 600 pixels, explicitly setting the viewport width to 600 pixels ("width=600") will make it look just as you intend. If your sites optimal width is smaller than 985 pixel it is good to be explicit, otherwise your pages might appear kind of far away and the user will have to manually zoom in.

Good Night Patel

Albert O'Connor — Sun, 04 Dec 2011 12:21:59 -0500

Today is the day I decommission the first server I created for this venture.

Over a year ago I started investigating how modern web hosting works and discovered the awesome price points you can get for small servers designed to serve small amounts of traffic. Not only were these cloud based virtual machines inexpensive, they weren't tied to hardware allowing the providers to provide excellent up time.

I named my first server Patel, and launched him on Rackspace's Cloud. He used to serve this very site until a few months ago. Overall I was happy with Rackspace. They provided a very simplistic introduction to cloud computing which readied me for my next infrastructure evolution.

Amazon Web Service's free tier did its job on me, opening the door to exploring what Amazon had to offer. Everything was much more complicated, but also far more powerful and more secure. The key power I appreciate the most is the API controlled firewall which you can adjust through the web interface for AWS. In general I also felt like my instances were better protected by default. It is in Amazon's interest to protect every instance on their cloud including mine.

I have moved up from having one server on AWS for myself and select clients to planning a scalable infrastructure both in terms of resource allocation and security. The core idea is to heavily secure and monitor key front end instances while the app server and database servers are kept inaccessible from the outside world. This means more resources can be provisioned without needing to increase the amount of security information to review.

The end of Patel is the end of an era for me, but what comes next is even more exciting. Contact me if you want help getting into this new way of creating awesome websites and web services.

Hosting Django under different locations with Nginx and gunicorn

Albert O'Connor — Thu, 15 Sep 2011 20:40:44 -0400

Do you know how Django uses SCRIPT_NAME?

It's not often I host different instances of Django under different locations on the same domain. One legacy install has been setup that way for over a year with Apache2 and mod_wsgi (with Nginx infront). Almost all of my new deployments use virtual hosts with different subdomains for each service.

If you are using virtual hosts with Nginx it looks like this:

server {
    listen   80;
    server_name  service.albertoconnor.ca;

    # other settings

    location / {
        try_files $uri/index.html $uri.html $uri @cluster;
    }

    location @cluster {
        proxy_pass http://service.albertoconnor.ca;
    }
}

Assuming you have defined an upstream server cluster called "service.albertoconnor.ca".

If you are doing something by location on the same domain it might look something more like this:

server {
    listen   80;
    server_name  _;

    # other settings...

    location /a/ {
        proxy_pass http://127.0.0.1:8001/;
    }

    location / {
        proxy_pass http://127.0.0.1:8080/;
        proxy_redirect off;
    }
}

The idea is that something else, let's say Apache2, is running on 127.0.0.1:8080, and Django being served by gunicorn is running on 127.0.0.1:8001. So under /a/ the pages will be generated by Django; anywhere else Apache2 takes over.

If you try this though, things will go horribly wrong. Besides redirects not working because you need some proxy_redirect mojo, your generated links in your templates won't work. They will generate absolute urls like "/foo/bar" instead of "/a/foo/bar". But really how should {% url %} know any better.

Then I got that feeling. You know when you figure out why something doesn't work, but because of that reason you can't explain why other things ever worked in the first place. I have been successfully serving different Django services based on location with Apache2 and mod_wsgi for over a year. How, when I generate urls, could it know to prepend the location?

The answer is that mod_wsgi WSGIScriptAlias command was helping me. Django will actually do the magic prepend for url generation if the SCRIPT_NAME variable is set in the wsgi environment. mod_python also does this by the django.root PythonOption. The SCRIPT_NAME variable is mentioned in the gunicorn FAQ very briefly and without context.

From the FAQ and other hints around the internet I gathered that I really just need to set the header SCRIPT_NAME HTTP header myself and things will start working. Since the SCRIPT_NAME is also taken off of incoming urls I had to alter my proxy_pass a bit. Here is what I ended up with:

    # ...
    location /a/ {
        proxy_pass http://127.0.0.1:8001/a/;
        proxy_redirect http://127.0.0.1:8001/a/ http://$host/a/;
        proxy_set_header SCRIPT_NAME /a;
    }

    location / {
        proxy_pass http://127.0.0.1:8080/;
        proxy_redirect off;
    }
    # ...

I also needed a slightly different proxy_redirect directive than the default. This may not be ideal so suggestions are welcome. I will apply them to this post. You can comment or contact me though this website.

Update Oct 26th, 2011 When using CAS with services I discovered the redirection url has the internal host IP rather than the proxy host name. One way to fix it is to use the X_Forwarded_Host header. In the /a/ block add:

        proxy_set_header X-Forwarded-Host your.host.name.com;

Internally Django has a HttpRequest.get_host method that if the setting USE_X_FORWARDED_HOST is True will use the value of the HTTP-X-FORWARDED-HOST header as the current request's host.

USE_X_FORWARDED_HOST = True

NOTE: The setting is new as of Django 1.3.1 and it is worth reading security advisory, see the section headed "Host header cache poisoning". The up shot of which is in this case seems to be to filter on your specific host and not "server _;" as I did in my example above to avoid a security hole.

jQuery fallback for HTML5 placeholder attribute

Albert O'Connor — Sun, 21 Aug 2011 15:56:14 -0400

If you google placeholder and jQuery you quickly find some handy code for a javascript fallback for the HTML5 placeholder functionality:

https://gist.github.com/379601

If you want to read more check out the blog post.

As written this will always override the built in browser behaviour, and it isn't as good as the built in behaviour. We only want this code to be registered if the current browser doesn't natively support placeholder, fortunately detection is easy thanks to Dive Into HTML5

http://diveintohtml5.org/everything.html#placeholder

This leads to this combination of code:

if  (!('placeholder' in document.createElement('input')) || !('placeholder' in document.createElement('textarea'))) {
    $(document).ready( function() {
        $('[placeholder]').focus(function() {
          var input = $(this);
          if (input.val() == input.attr('placeholder')) {
            input.val('');
            input.removeClass('placeholder');
          }
        }).blur(function() {
          var input = $(this);
          if (input.val() == '' || input.val() == input.attr('placeholder')) {
            input.addClass('placeholder');
            input.val(input.attr('placeholder'));
          }
        }).blur().parents('form').submit(function() {
          $(this).find('[placeholder]').each(function() {
            var input = $(this);
            if (input.val() == input.attr('placeholder')) {
              input.val('');
            }
          })
        });
    });
}

The code makes the somewhat broad assumption that no browser supports placeholder for an input but not for a textarea, or if it did, the behaviour shouldn't be overridden. Part of the appeal for HTML5 for me is the ability to provide great user experience for those on modern browser and good experiences on those forced not to upgrade.

Wilderness Labs

Albert O'Connor — Tue, 28 Jun 2011 23:10:52 -0400

In December 2010, inspired by several interesting events I had heard about, I started a unique project with a few of my good friends. The idea is to find out what happens when you go away for the weekend with some talented people to a beautiful cottage with the intention to create something.

We decided to call it Wilderness Labs.

Inspiration came from the likes of /dev/fort and the 7 Cubed project. Since there aren't as many rent-able forts in Southern Ontario, cottages seemed like the logical replacement.

Some 8 months on we have already run our first event and are planning to go back on the weekend of Nov 11th to 13th. We are excited to share the experience with some new people, so if it sounds like the kind of thing you would be interested in please let us know on our get involved page.

If the first event is any indication we will be doing this for a long time. Going out to a cottage, enjoying a late night camp fire and being able to walk down to the beach made for a weekend which was both memorable and productive.

Best Practices: Off-Site Backups

Albert O'Connor — Tue, 31 May 2011 19:52:39 -0400

Having a backup strategy is key. What works for me won't necessary work for you. I am going to outline my method for my own reference, to get feedback, and in the hope it might help someone else out there.

I am using Amazon S3 as my back-end, taking advantage of 10 GB for a year for free. I am abusing Fabric as my glue. Fabric is meant for remote execution but I am using its local feature as a kind of Pythonic shell script.

The first major issue is security. S3 is private by default but you are implicitly trusting Amazon. The only solution is to encrypt your data, which is easier said then done. The core of this post is documenting my method for encryption.

Encrypting Your Bits

I use OpenSSL to do the encryption instead of GPG or PGP because it is available on my Mac and widely on Linux without apt-getting.

Conceptually a backup is like a message sent from present day me to future me using asymmetric, or public/private key, encryption. I want to encrypt the backup with my public key and decrypt it later with my private key.

The trouble is that with OpenSSL asymmetric encryption can only be used for small files. The solution is to generate a unique symmetric key for every backup and encrypt that small file with my public key. I save the symmetrically encrypted backup and the asymmetrically encrypted symmetric key together and send them both to S3. Without your private key the two files get you nothing since they are both encrypted.

The first thing you have to once and only once is generate the public/private keys for encrypting the symmetric encryption keys (passwords):

def create_keys():
    """
    Only do this once, if you overwrite your keys you won't be able to decrypt your backups!
    """
    key_file_name = os.path.join(root, "keys/backup.pem")
    local("openssl genrsa -des3 -out %s 1024" % key_file_name) # Generate encrypted private key
    local("openssl rsa -in %s -pubout > %s" % (key_file_name, key_file_name + ".pub")) #Output the public part

Here is the meat of a backup:

def backup_to_s3():
    today = datetime.date.today()
    id = today.strftime("%d-%m-%Y")
    
    # Tar everything you want to backup.  Assume a list of directories in backup_dirs.
    backup_file_name = os.path.join(relative_store, "backup-%s.tar.gz" % id)
    local("tar -zcf %s %s" % (backup_file_name," ".join(backup_dirs)))
    
    # Generate a 64 character symmetric key
    key = os.urandom(64)
    key_file_name = os.path.join(relative_store, "backup-%s.key" % id)
    f = open(os.path.join(local_store, "backup-%s.key" % id), "w")
    f.write(key)
    f.close()
    
    # Encrypt the backup with the symmetric key
    local("openssl enc -e -aes128 -pass file:%s < %s > %s" % (key_file_name,
                                                              backup_file_name,
                                                              backup_file_name + ".enc"))
    
    # Encrypt the symmetric key with the public key generated before
    public_key_name = os.path.join(root, "keys/backup.pem.pub")
    local("openssl rsautl -encrypt -inkey %s -pubin -in %s -out %s" % (public_key_name,
                                                                       key_file_name,
                                                                       key_file_name + ".pubenc"))
    
    # Securely remove the unencrypted symmetric key file
    local("srm %s" % key_file_name)
    
    # Tar the encrypted backup and encrypted key together
    unfied_backup_file_name = os.path.join(relative_store, "unified_backup-%s.tar.gz" % id)
    local("tar -zcf %s %s %s" % (unfied_backup_file_name,
                                 backup_file_name + ".enc",
                                 key_file_name + ".pubenc"))
    
    # Push the result to s3
    local("s3put -a %s -s %s -b amjoconn-backups -p %s %s" % (ACCESS_KEY,
                                                             SECRET_KEY,
                                                             local_store,
                                                             os.path.abspath(unfied_backup_file_name)))

This script has several global configuration variables which aren't addressed. It also won't clean up your local store except to delete the unencrypted key.

Recovery

Like an claiming an insurance policy, recovery is the most important part of any backup system. Like insurance people tend not to talk too much about how it will work nor do they test it often. Given the complexity of the backup process, I figured outlining the recovery process even if the automation ultimately fails me would be handy for reference.

def recover_backup(id):
    # Try to get the file
    conn = boto.connect_s3(ACCESS_KEY, SECRET_KEY)
    bucket = conn.get_bucket("amjoconn-backups")
    key = bucket.get_key("unified_backup-%s.tar.gz" % id)
    if key is None:
        print "Unable to find backup with id %s" % id
        return
    recover_name = os.path.join(recovery_location, "recover-%s.tar.gz" % id)
    print "Writing", "unified_backup-%s.tar.gz" % id, "to", recover_name
    print "This might take a bit of time..."
    key.get_contents_to_filename(recover_name)
    local("tar -xzf %s" % recover_name)
    
    # Decrypt the symmetric key file with the private key.
    key_file_name = os.path.join(relative_store, "backup-%s.key" % id)
    private_key_name = os.path.join(root, "keys/backup.pem")
    local("openssl rsautl -decrypt -inkey %s -in %s -out %s" % (private_key_name,
                                                                key_file_name + ".pubenc",
                                                                key_file_name))
    
    # Decrypt the backup with the recently decrypted symmetric key.
    backup_file_name = os.path.join(relative_store, "backup-%s.tar.gz" % id)
    local("openssl enc -d -aes128 -pass file:%s < %s > %s" % (key_file_name,
                                                              backup_file_name + ".enc",
                                                              backup_file_name))
    
    # Untar the data here.  Depending how you used tar, there will be some interesting directories created.
    local("tar -zxf %s" % backup_file_name)

Note, this doesn't clean up the decrypted symmetric key.

Wrap Up

There it is. How I am keeping my bits safe. Such a problem is never really solved though. I hope to be able to update this post over time as I discover better ways to backup because this solution has much room for improvement.

Best Practices: Backups

Albert O'Connor — Sat, 02 Apr 2011 18:23:04 -0400

My last post was on the importance of password management. Another critical best practice when you deal with other peoples data is backups.

Since the beginning of the digital age retaining data for both the short and long term has become an open problem or sorts. Fortunately fragile bits are easy to copy but every type of media has a limited lifetime, especially hard disks. Coming up with the right balance to ensure retention is tricky. Much like claim is the most important part of insurance, recovery is the most important aspect of any backup system and testing it can be tricky.

I will outline my approach to backing up data, but the real key is to have a strategy and to think about the problem. What works for me won't necessarily work for you. Don't be reliant a old beige computer in an office somewhere which you hope won't die on you.

Time Machine and Time Capsule

If you are running Mac OS X definitely take advantage of Time Machine. It can do automated hourly backups with a well tested recovery mechanism. A feature was added under System Preferences, Time Machine, Options... to enable or disable doing backups on battery power. I find disabling help keep performance up. My backups may not be hourly but when they happen they don't drag my system down.

Apple's Time Capsule sounds like a destination for your backups, but in my experience it is some what questionable. Apple has had some quality issues with Time Capsules. My first Time Capsule was newer than the generation which has officially be designated as flawed, but it still died after about 18 months. It died in such a way that the hard disk seemed functional, but I wasn't allowed to remove it and get warranty replace of the Time Capsule unless I paid an Apple certified tech to do it. The removal wasn't covered under warranty. For a data backup device this doesn't seem like a sensible policy.

There are likely better options which allow the removal of your disks if the device itself fails. I am currently still using my replacement Time Capsule, but when it dies in 18 months I will definitely look into replacing it with something else. Likely some kind of network hard disk since Gigabit Ethernet can move bytes faster than USB 2.0 which is important when moving around large 100 GB backup files when dealing with failure.

Two Separate Devices and Off-site Backups

The lesson I have learned is data really isn't backed up unless it is on two separate devices, not just two separate drives. The more critical data is the more places it should be stored. Often the most critical things are also things which need to be kept secure. Having many copies and security seem contradictory, but with a bit of encryption built on the passwords system I outline last post, both can be achieved.

The best backup strategies include off-site backups. The risk to your data are beyond just device failure. There is also external issues such as fire or other disaster. With data hosting services like Amazon's S3, Rackspace's Cloudfiles and others, you can take advantage of off-site backups while hooking into major players redundant storage infrastructure at a great price.

Unfortunately universal tools to make this easy and accessible to everyone aren't quite mature yet, but make no mistake they are on their way.

In my next post I will outline how I backup my most critical information to Amazon S3 in a secure way using Python, Fabric and OpenSSL.

Best Practices: Passwords

Albert O'Connor — Fri, 03 Dec 2010 08:39:33 -0500

When I decided to start working for myself, I felt I needed to establish best practices as soon as possible, while I still had time. My clients would be trusting me to with their information and access to their systems and I wanted to make sure I was being responsible.

Password management is one area I targeted. Even if you aren't in business for yourself, being secure online basically requires a good password management scheme.

Existing Options

There are a lot of solutions and partial solutions out there. For example browsers will remember arbitrary passwords for you, but that is only most convenient when you are using that browser. Many of my passwords are for ssh accounts and I planned on using multiple users on my laptop.

Other 3rd party solutions exists like Splash Data and 1Password. My problem with their approach was not being clear how to backup and recover the data cross computers. As I will write about soon, backups is one of the other best practices I focused on.

Rolling Your Own

I talked to my resident cryptography expert Douglas Stebila about how he was using a simple system which seem to fit my requirements. It was easy to use, backup, and completely transparent.

This approach is best for those who understand the command line, shell scripts and grep and who are using Unix compatible systems like Linux or Mac OS X. Cygwin would also work.

The core idea is to store all your password in an OpenSSL encrypted file. You decrypt the file and pass it through grep to look up passwords. I use a format of "description: password", but everything is customizable. I implemented a series of shell scripts and one Python script to make it all work.

First lets look at the passwords script.

#!/bin/bash
openssl enc -d -aes128 < /somewhere/accessible/passwords.aes128 | grep -i $1

# Usage: passwords blah
# password: <Enter master password>
# Finds all descripts or passwords with "blah" in them

I make sure that the passwords script is always on the path and the passwords.aes128 file is accessible. It is important you make sure your master password is a strong password since it protects everything.

The master password is really just the password you encrypt the passwords.aes128 file with, so when you are editing it and re-encrypting it, make sure you don't mistype it or you could loose your password archive. Again a backup strategy is important.

I manage the file with a set of scripts. First there is passwords-decompress and passwords-edit:

#!/bin/bash
openssl enc -d -aes128 < /somewhere/accessible/passwords.aes128 >/somewhere/accessible/passwords.aes128.decrypted

#!/bin/bash
vi /somewhere/accessible/passwords.aes128.decrypted

Most of these scripts are just so I need to remember paths and arguments. I want adding passwords to be as easy as possible. To generate a new password I use something like the following Python script.


import random

chars = "1234567890qwertyuiopasdfghjklzxcvbnmQWERTYUIOPASDFGHJKLZXCVBNM"

if __name__ == "__main__":
    print "Password Helper. Albert O'Connor. 2010"
        
    desc = raw_input("Enter password description: ")
    
    password = ""
    for i in range(8):
        password += random.choice(chars)
    
    print "%s: %s" % (desc, password)

The last line this script prints out is what I copy and paste into the passwords file. If you want to be more secure about it you can use Python Cryptography's Random. Once I have edited the decrypted passwords my satisfaction I run passwords-compress.

#!/bin/bash
openssl enc -e -aes128 < /somewhere/accessible/passwords.aes128.decrypted > /somewhere/accessible/passwords.aes128

This is where you have to be careful, since consistently mistyping your master password can lead to inaccessible data. I always test out the new password I added with passwords before backing up with passwords-backup.

#!/bin/bash
cp /somewhere/accessible/passwords.aes128 /somewhere/accessible/passwords.aes128.backup
scp /somewhere/accessible/passwords.aes128 user@server.org:~/backup/

This makes a local and off site backup of the encrypted file. Finally I clean up with passwords-cleanup:

#!/bin/bash
srm  /somewhere/accessible/passwords.aes128.decrypted

Because leaving decrypted versions of your passwords file around totally defeats the purpose. Now whenever you need a password, open a prompt, type passwords blah, enter your master password, and you will get all your passwords for blah. Close the terminal when you are done though!

No matter what, if you will be freelancing in a digital world, keeping track of passwords is something you have to do. This method is simple, the encryption is secure enough for the NSA, and highly usable. If it doesn't suit your taste employ another one, just make sure you are using one.

Autocomplete for Foreign Keys in the Django Admin Explained

Albert O'Connor — Fri, 03 Sep 2010 17:26:15 -0400

When the select drop-down for foreign keys in the Django Admin gets big it can become a usability issue. One solution is to create a custom UI for the the model, but stretching the Django Admin to the limit is one of the fun parts of working with Django.

Prior Art

The top hit on Google is django-autocomplete. This project is the basis of my approach and therefore much thanks goes to the author. There hasn't been an update since early 2009 and I wasn't able to get it working without a lot of changes. The changes evolved into what I consider a better approach.

Another interesting project is django-ajax-selects which is very fancy, complicated looking and actively maintained. It allows for very pretty autocompletes. My solution does it without any AJAX so it is a bit easier to use and makes for an easier case study. An AJAX based solution solves another critical problem: the drop down being so large it takes a long time for the admin page to render. Raw id fields also help to solve this problem albeit in a less elegant way.

There are a few other projects, some using YUI, some focusing on many to many fields. The solution presented here is jQuery based and for foreign keys in the admin only. This post attempts to explain the solution end to end and is based on Django 1.2.x.

Progressive Enhancement

Progressive enhancement is one of the key benefits of my approach. Instead of having a hidden input element to hold the real id of the foreign object, I actually keep the select drop-down and merely hide it and show the autocomplete field if Javascript is enabled.

Usage

Once complete, using it will look something like this inside an admin.py file:

admin.site.register(Foreign, AutocompleteTargetModelAdmin)
    
class MainAdmin(AutocompleteModelAdmin):
    #...
    autocomplete_fields = ('foreign',)
    #...

admin.site.register(Main, MainAdmin)

Where the Main model has a ForeignKey, foreign, to the model named Foreign.

What You Will Need

In addition to Django you will need:

Getting The Autocomplete Data

Normally the hard part is providing an AJAX response from some url, for which you have to define a view which provides the possible values. Requiring the client to do this every time is undesirable. There is a simpler way. Autocomplete can also work on static arrays, and since the drop-down is static it should work just as well. Actually, the select drop-down we are keeping hidden is a static list of possible values! We will use the select to generate a static array of values to populate the autocomplete.

The New Hard Part

The only really hard part left is keeping the "+" adding of items in the Django Admin working. This will force us to do one zany thing: require that our foreign model's admin class is a subclass of a special class we provide, the so called AutocompleteTargetAdminModel. More on this shortly.

Let's Do It

Now that you have some idea of what is coming let's get started. Define a separate app called something clever like "autocomplete". It will contain an admin.py file (abuse of naming: we will be subclassing classes in django.contrib.admin rather than defining new admin classes like in a regular app admin.py file) and a widget.py file where a custom widget is defined.

Starting with widget.py. The ForeignKeyAutocompleteInput is based the ForeignKeySearchInput in the source of django-autocomplete. The major difference is I hide the original select widget with Javascript instead of using a hidden widget.

from django.conf import settings
from django import forms
from django.db import models

from django.utils.safestring import mark_safe

class ForeignKeyAutocompleteInput(forms.widgets.Select):
    class Media:
        css = {
                'all': ('%s/css/jquery.autocomplete.css' % settings.MEDIA_URL,)
        }
        js = (
                '%s/js/jquery.js' % settings.MEDIA_URL,
                '%s/js/jquery.autocomplete.js' % settings.MEDIA_URL,
                '%s/js/autocomplete.popup.js ' % settings.MEDIA_URL
        )

    def text_field_value(self, value):
        key = self.rel.get_related_field().name
        obj = self.rel.to._default_manager.get(**{key: value})
        
        return unicode(obj)

    def __init__(self, rel, attrs=None):
        """
        rel - the relation for the foreign key.
        """
        self.rel = rel
        super(ForeignKeyAutocompleteInput, self).__init__(attrs)

    def render(self, name, value, attrs=None):
        if attrs is None:
            attrs = {}
        rendered = super(ForeignKeyAutocompleteInput, self).render(name, value, attrs)
        if value:
            text_field_value = self.text_field_value(value)
        else:
            text_field_value = u''
        return rendered + mark_safe(u'''
#
# Insert javascript wrapped in HTML as listed below here
#
                ''') % {
                        'MEDIA_URL': settings.MEDIA_URL,
                        'model_name': self.rel.to._meta.module_name,
                        'app_label': self.rel.to._meta.app_label,
                        'text_field_value': text_field_value,
                        'name': name,
                        'value': value,
                }

Let us focus on the key parts one at a time. First we define an internal class called Media which defines the external resources required to render.

class Media:
    css = {
            'all': ('%scss/jquery.autocomplete.css' % settings.MEDIA_URL,)
    }
    js = (
            '%s/js/jquery.js' % settings.MEDIA_URL,
            '%s/js/jquery.autocomplete.js' % settings.MEDIA_URL,
            '%s/js/autocomplete.popup.js ' % settings.MEDIA_URL
    )

We require jQuery and jquery.autocomplete which provides the jquery.autocomplete.css. We will need to write a bit of Javascript which I called autocomplete.popup.js to support adding new values.

Next we write some boiler plate the widget class needs to work.

def text_field_value(self, value):
    key = self.rel.get_related_field().name
    obj = self.rel.to._default_manager.get(**{key: value})
    
    return unicode(obj)

def __init__(self, rel, attrs=None):
    """
    rel - the relation for the foreign key.
    """
    self.rel = rel
    super(ForeignKeyAutocompleteInput, self).__init__(attrs)

The interesting bit is render.

def render(self, name, value, attrs=None):
    if attrs is None:
        attrs = {}
    rendered = super(ForeignKeyAutocompleteInput, self).render(name, value, attrs)
    if value:
        text_field_value = self.text_field_value(value)
    else:
        text_field_value = u''
    return rendered + mark_safe(u'''

This is followed by a huge Javascript filled string which is the real meat of what we are doing here. We have rendered the select normally and in the very last line we append a bunch of additional HTML which includes the autocomplete widget and the Javascript to create the array of possible values and pass it into an .autocomplete function.


<input type="text" id="lookup_%(name)s" value="%(text_field_value)s" size="40" style="display: none;"/>
<script type="text/javascript">

$(document).ready(function(){
    // Javascript is required to show the autocomplete field and hide the select field.
    $("#id_%(name)s").hide();
    $("#lookup_%(name)s").show();

    function liFormat_%(name)s (row, i, num) {
            var result = row[0] ;
            return result;
    }
    
    var %(name)s_data = Array();
    var %(name)s_id_map = {};
    
    function load_autocomplete_data_from_select() {
        %(name)s_data = Array();
        %(name)s_id_map = {};
        $("#id_%(name)s option").each(function(d) {
            %(name)s_data.push($(this).html());
            %(name)s_id_map[$(this).html()] = $(this).val();
        })
        
        $("#lookup_%(name)s").autocomplete(%(name)s_data, {
            delay:10,
            minChars:1,
            matchSubset:1,
            autoFill:false,
            matchContains:1,
            cacheLength:10,
            selectFirst:true,
            formatItem:liFormat_%(name)s,
            maxItemsToShow:10
        }); 
    }
    
    load_autocomplete_data_from_select(); // Inital load
    
    // Changing the autocomplete field needs to change the hidden select field
    $("#lookup_%(name)s").change(function() {
        new_value = %(name)s_id_map[$(this).val()];
        if (new_value == undefined) {
            new_value = ""
        }
        $("#id_%(name)s").val(new_value); 
    })
    
    // It is possible to "change" the autocomplete text field and have the change
    // event not happen.  This double checks right before we submit.
    $("form").submit(function() {
        $("#lookup_%(name)s").change(); // Just to make sure
    })
    
    // When the add feature is used, it only knows how to change the select field
    // so the auto complete field needs to be updated too.
    $("#id_%(name)s").change(function () {
        $("#lookup_%(name)s").val($(this).find("option:selected").html());
        load_autocomplete_data_from_select(); // Could be a new value from an add
    })    
});
</script>

We pass some values into the Javascript block allowing us to do some % substitution. For example the name of the field will be unique on the page so we can use for ids.

) % {
        'MEDIA_URL': settings.MEDIA_URL,
        'model_name': self.rel.to._meta.module_name,
        'app_label': self.rel.to._meta.app_label,
        'text_field_value': text_field_value,
        'name': name,
        'value': value,
}

Admin Integration

The widget is handy, and could probably be used alone pretty effectively. But to support adding new values, and to make it much easier to use, some deep admin integration is handy. This approach might not be extremely future-proof, but I have tried to cut down on the zany-ness.

admin.py is also based on django-autocomplete. The big change is splitting up the AdminModel subclasses into a target and main variants.

Let's look at it in four parts. First import a bunch of stuff. We are going deep into Django so we will need a lot of utils functions.

from django.conf import settings
from django.contrib import admin
from django.db import models

from django.utils.safestring import mark_safe
from django.utils.translation import ugettext as _
from django.utils.html import escape
from django.utils.encoding import force_unicode

from django.contrib.auth.models import Message
from django.http import HttpResponse, HttpResponseNotFound, HttpResponseRedirect

from widgets import ForeignKeyAutocompleteInput

Moving quickly on we have the part which makes

autocomplete_fields = ('foreign',)

work.

class AutocompleteModelAdmin(admin.ModelAdmin):
    def formfield_for_dbfield(self, db_field, **kwargs):
        # For ForeignKey use a special Autocomplete widget.
        if isinstance(db_field, models.ForeignKey) and hasattr(self, "autocomplete_fields") and db_field.name in self.autocomplete_fields:
            kwargs['widget'] = ForeignKeyAutocompleteInput(db_field.rel)
            # extra HTML to the end of the rendered output.
            if 'request' in kwargs.keys():
                kwargs.pop('request')
                                                        
            formfield = db_field.formfield(**kwargs)
            # Don't wrap raw_id fields. Their add function is in the popup window.
            if not db_field.name in self.raw_id_fields:
                # formfield can be None if it came from a OneToOneField with
                # parent_link=True
                if formfield is not None:
                    formfield.widget = AutocompleteWidgetWrapper(formfield.widget, db_field.rel, self.admin_site)
            return formfield

        return super(AutocompleteModelAdmin, self).formfield_for_dbfield(db_field, **kwargs)

One class overriding one function. If the field is a ForeignKey and is named in autocomplete_fields, then we use our widget from above, and do a bit of funky wrapping to make the "+" appear. More on the AutocompleteWidgetWrapper later.

I have separated out the next class. We use it on models we are targeting with an autocomplete field like this.

admin.site.register(Foreign, AutocompleteTargetModelAdmin)

The class looks like this:

class AutocompleteTargetModelAdmin(admin.ModelAdmin):        

    def response_add(self, request, obj, post_url_continue='../%s/'):
        """
        Determines the HttpResponse for the add_view stage.
        """
        opts = obj._meta
        pk_value = obj._get_pk_val()
        
        msg = _('The %(name)s "%(obj)s" was added successfully.') % {'name': force_unicode(opts.verbose_name), 'obj': force_unicode(obj)}
        # Here, we distinguish between different save types by checking for
        # the presence of keys in request.POST.
        if request.POST.has_key("_continue"):
                self.message_user(request, msg + ' ' + _("You may edit it again below."))
                if request.POST.has_key("_popup"):
                        post_url_continue += "?_popup=%s" % request.POST.get('_popup')
                return HttpResponseRedirect(post_url_continue % pk_value)
        
        if request.POST.has_key("_popup"):
                #htturn response to Autocomplete PopUp
                if request.POST.has_key("_popup"):
                        return HttpResponse('' % (escape(pk_value), escape(obj)))
                                        
        elif request.POST.has_key("_addanother"):
                self.message_user(request, msg + ' ' + (_("You may add another %s below.") % force_unicode(opts.verbose_name)))
                return HttpResponseRedirect(request.path)
        else:
                self.message_user(request, msg)

                # Figure out where to redirect. If the user has change permission,
                # redirect to the change-list page for this object. Otherwise,
                # redirect to the admin index.
                if self.has_change_permission(request, None):
                        post_url = '../'
                else:
                        post_url = '../../../'
                return HttpResponseRedirect(post_url)

One class, one complete method copied except for one key line:

return HttpResponse('' % (escape(pk_value), escape(obj)))

Here "opener" is the window a user spawns by hitting the green "+". This change in the response lets us call our slightly different popup handling code which enables our autocomplete field to update with the new value after the hidden select drop-down is updated by the default Django Admin code.

There might be a better way, but I haven't been able to find it yet. Suggestions are welcome. The key problem is, though the select will change by default, it won't produce a change event, so there is no way to update the autocomplete field.

Finally the wrapper, which is all thanks to the author of django-autocomplete.

class AutocompleteWidgetWrapper(admin.widgets.RelatedFieldWidgetWrapper):
        def render(self, name, value, *args, **kwargs):
                rel_to = self.rel.to
                related_url = '../../../%s/%s/' % (rel_to._meta.app_label, rel_to._meta.object_name.lower())
                self.widget.choices = self.choices
                output = [self.widget.render(name, value, *args, **kwargs)]
                if rel_to in self.admin_site._registry: # If the related object has an admin interface:
                        # TODO: "id_" is hard-coded here. This should instead use the correct
                        # API to determine the ID dynamically.
                        output.append(u' ' % \
                                (related_url, name))
                        output.append(u'' % (settings.ADMIN_MEDIA_PREFIX, _('Add Another')))
                return mark_safe(u''.join(output))

That is basically it except for a bit of Javascript that's needed to stitch together the popup logic which I placed in a file called autocomplete.popup.js

function showAutocompletePopup(triggeringLink) {
    var name = triggeringLink.id.replace(/^add_/, '');
    name = id_to_windowname(name);
    href = triggeringLink.href
    if (href.indexOf('?') == -1) {
        href += '?_popup=2';
    } else {
        href  += '&_popup=2';
    }
    var win = window.open(href, name, 'height=500,width=800,resizable=yes,scrollbars=yes');
    win.focus();
    return false;
}

function dismissAutocompletePopup(win, newId, newRepr) {
    newId = html_unescape(newId);
    newRepr = html_unescape(newRepr);
    var name = windowname_to_id(win.name);
    dismissAddAnotherPopup(win, newId, newRepr);
    $("#" + name).change();
}

Not super simple, but an effective way to get a friendlier admin user interface experience.

If you find this useful please let me know. If there is enough interest I will look into cleaning out the cruft and making an installable package out of this code.

Showing Current Values for FileFields in Django

Albert O'Connor — Mon, 09 Aug 2010 20:47:00 -0400

I am working on a dynamic form which contained multiple FileFields which could have initial values. The FileInput widget doesn't show initial values since that could expose too much about the servers underlying file system. Despite this I wanted to show the file name as user feedback.

If the form was static, I could pass current file names in through the context and just be very explicit with my template code. Since the form was dynamic the template code would have to iterate though the fields, and there is no easy way to look up unrelated data at the same time.

To learn some nifty tricks on how to create dynamic forms check out So you want a dynamic form on the b-list blog.

Meanwhile I need a solution to pass in the initial file name data with the form. The built in initial dictionary didn't really work, and the field objects initial data member was empty. I had to look outside the form. Beyond the form to the truth: there is not form!

There Is No Form

A pearl of wisdom in my grasp, the form object did not matter, only the field objects were required to render a form. (Assuming you don't need form-wide errors ...)

In the view I passed in 2-uples of the initial file names of the files and the field objects. I did not need to pass the form object into the template at all:

def view(request):
   posting = get_object_or_404(Posting, pk=id) # Get the thing the dynamic form depends on
   file_store = get_object_or_404(FileStore, pk=id) # Something like this

   if request.method == "POST":      
     file_form = make_file_application_form(posting)(request.POST, request.FILES)

     file_fields_and_names = [(file_form[key], getattr(file_store, key)) for key in file_form.fields]
     file_fields_and_names = [(field, file_name_from_file_obj(file)) for field,file in file_fields_and_names]

     if file_form.is_valid():
       file_data = file_form.cleaned_data
       for key in file_data:
          if file_data[key]:
            setattr(file_store, key, file_data[key])
       file_store.save()
       return redirect("view") 
     else:
       file_form = make_file_application_form(posting)()

       file_fields_and_names = [(file_form[key], getattr(file_store, key)) for key in file_form.fields]
       file_fields_and_names = [(field, file_name_from_file_obj(file)) for field,file in file_fields_and_names]

   return render_to_response(template,
                              dict(file_fields_and_names=file_fields_and_names, ...),
                              context_instance=RequestContext(request))

So if you need to, you can get creative with how you pass your fields into your template.