Josh Bohde blog

Don't Design the API to the Client

2012-09-17T09:17:35Z

A problem I often see with first production versions of HTTP APIs is they are tightly coupled to the client code. This is especially easy to do in web applications where both the client and server source are contained in the same repository.

A few signs in development you should watch out for:

Display changes on the client cause a change in the API
API resources that are only a combination of other resources
Client views and API resources have a one-to-one relationship

These hint that the API may be too specialized to the client. This make other clients harder to develop, and makes changing the current client difficult. Instead, let client development push the API to develop flexibility, instead of assumptions on how it will be used.

Document Summarization using TextRank

2012-09-02T00:55:35Z

For a gift recommendation side-project of mine, I wanted to do some automatic summarization for products. A fairly easy way to do this is TextRank, based upon PageRank. In this example, the vertices of the graph are sentences, and the edge weights between sentences are how similar the sentences are.

To go from an string of text to a list of scored sentences based upon how much they represent the overall text, we need to go through several steps:

Tokenize the text into sentences
Tokenize each sentence into a collection of words
Convert the sentences into graphs
Score the sentences via pagerank

Sentence Splitting

To get started, we'll need to split the document into sentences. For this, we'll use nltk's included Punkt module, and the opening paragraph of Sir Authur Conan Doyle's A Scandal In Bohemia.

>>> from nltk.tokenize.punkt import PunktSentenceTokenizer
 
>>> document = """To Sherlock Holmes she is always the woman. I have
seldom heard him mention her under any other name. In his eyes she
eclipses and predominates the whole of her sex. It was not that he
felt any emotion akin to love for Irene Adler. All emotions, and that
one particularly, were abhorrent to his cold, precise but admirably
balanced mind. He was, I take it, the most perfect reasoning and
observing machine that the world has seen, but as a lover he would
have placed himself in a false position. He never spoke of the softer
passions, save with a gibe and a sneer. They were admirable things for
the observer-excellent for drawing the veil from men’s motives and
actions. But for the trained reasoner to admit such intrusions into
his own delicate and finely adjusted temperament was to introduce a
distracting factor which might throw a doubt upon all his mental
results. Grit in a sensitive instrument, or a crack in one of his own
high-power lenses, would not be more disturbing than a strong emotion
in a nature such as his. And yet there was but one woman to him, and
that woman was the late Irene Adler, of dubious and questionable
memory.
"""
>>> document = ' '.join(document.strip().split('\n'))
 
>>> sentence_tokenizer = PunktSentenceTokenizer()
>>> sentences = sentence_tokenizer.tokenize(document)

Bag of Words

Before we can do any analysis, we need to convert the document into a bag-of-words, which is an unordered collection of word counts.

For the first sentence, we should have something like the following:

{'always': 1,
 'holmes': 1,
 'is': 1,
 'she': 1,
 'sherlock': 1,
 'the': 1,
 'to': 1,
 'woman': 1}

A simple implementation for converting a sentence to a bag-of-words can be done via the base Python datastructures:

from collections import Counter
 
def bag_of_words(sentence):
    return Counter(word.lower().strip('.,') for word in sentence.split(' '))

An alternative way to a bag-of-words is in a vector, where each column corresponds to a word. Instead of the previous dictionary, we'd have something like the following:

[1, 1, 1, 1, 1, 1, 1, 1]

The package scikit-learn provides text feature extraction, allowing us to build SciPy matrices out of a collections of texts.

>>> from sklearn.feature_extraction.text import CountVectorizer
 
>>> c = CountVectorizer()
>>> bow_array = c.fit_transform([sentences[0]])
>>> bow_array.toarray()
[[1 1 1 1 1 1 1 1]]

The resulting array is a bit different, since hyphens are not included in words, and the stopword "a" is excluded.

We can apply this to the entire collection, and get back a matrix.

>>> from sklearn.feature_extraction.text import CountVectorizer
 
>>> c = CountVectorizer()
>>> bow_matrix = c.fit_transform(sentences)
>>> bow_matrix
<11x127 sparse matrix of type '<type 'numpy.float64'>'
   with 183 stored elements in COOrdinate format>

Converting to a Graph

Now we have a matrix where the rows are sentences and the columns are words. We need to transform this into a graph relating the sentences to each other. To do this, we'll first normalize our matrix using Scikit-learn's TfidfTransformer.

>>> from sklearn.feature_extraction.text import TfidfTransformer
>>> normalized_matrix = TfidfTransformer().fit_transform(bow_matrix)

This will re-weight each word is based upon its tf-idf, which will dimish the effect of words common to each sentence.

>>> similarity_graph = normalized_matrix * normalize_matrix.T
>>> similarity_graph.toarray()
array([[ 1.        ,  0.        ,  0.13737879,  0.04767903,  0.04305016,
         0.04345599,  0.03330044,  0.05261648,  0.07798958,  0.        ,
         0.20047419],
       [ 0.        ,  1.        ,  0.0842143 ,  0.07819597,  0.        ,
         0.05171612,  0.        ,  0.        ,  0.        ,  0.        ,
         0.05807146],
       [ 0.13737879,  0.0842143 ,  1.        ,  0.        ,  0.07004069,
         0.09648614,  0.1069042 ,  0.06701793,  0.09437203,  0.20474295,
         0.1197599 ],
       [ 0.04767903,  0.07819597,  0.        ,  1.        ,  0.07558987,
         0.18678911,  0.05853972,  0.09249592,  0.10892262,  0.09110741,
         0.24159019],
       [ 0.04305016,  0.        ,  0.07004069,  0.07558987,  1.        ,
         0.07055583,  0.02370685,  0.07272032,  0.17253418,  0.08262451,
         0.17789849],
       [ 0.04345599,  0.05171612,  0.09648614,  0.18678911,  0.07055583,
         1.        ,  0.12952649,  0.06859301,  0.06837492,  0.13015945,
         0.15423071],
       [ 0.03330044,  0.        ,  0.1069042 ,  0.05853972,  0.02370685,
         0.12952649,  1.        ,  0.06307559,  0.03194234,  0.02852116,
         0.11271501],
       [ 0.05261648,  0.        ,  0.06701793,  0.09249592,  0.07272032,
         0.06859301,  0.06307559,  1.        ,  0.09411725,  0.        ,
         0.07702234],
       [ 0.07798958,  0.        ,  0.09437203,  0.10892262,  0.17253418,
         0.06837492,  0.03194234,  0.09411725,  1.        ,  0.12388421,
         0.14327969],
       [ 0.        ,  0.        ,  0.20474295,  0.09110741,  0.08262451,
         0.13015945,  0.02852116,  0.        ,  0.12388421,  1.        ,
         0.04706138],
       [ 0.20047419,  0.05807146,  0.1197599 ,  0.24159019,  0.17789849,
         0.15423071,  0.11271501,  0.07702234,  0.14327969,  0.04706138,
         1.        ]])

This is a mirrored matrix, where both the rows and columns correspond to sentences, and the elements describe how similar the two sentences are. Scores of 1 mean that the sentences are exactly the same, while scores of 0 mean the sentences have no overlap.

Pagerank

With a graph of sentences, we can use pagerank to score them. To do this, we'll use the pagerank function from NetworkX.

>>> import networkx as nx
>>> nx_graph = nx.from_scipy_sparse_matrix(similarity_graph)
>>> scores = nx.pagerank(nx_graph)
>>> scores
{0: 0.08671319149370108,
 1: 0.08316523582097997,
 2: 0.09513943890606882,
 3: 0.09440660636561657,
 4: 0.08968582008434481,
 5: 0.0949290548524502,
 6: 0.08616430411108938,
 7: 0.086273427821944,
 8: 0.09263320827229513,
 9: 0.08808392628885832,
 10: 0.10280578598265173}

This gives us a mapping of sentence indices to scores. We can use associate these back to our original sentences and sort them.

>>> ranked = sorted(((scores[i],s) for i,s in enumerate(sentences)),
                reverse=True)
>>> ranked[0][1]
'And yet there was but one woman to him, and that woman was the late '
'Irene Adler, of dubious and questionable memory.'

We can combine all of this into one function as follows:

import networkx as nx
import numpy as np
 
from nltk.tokenize.punkt import PunktSentenceTokenizer
from sklearn.feature_extraction.text import TfidfTransformer, CountVectorizer
 
def textrank(document):
    sentence_tokenizer = PunktSentenceTokenizer()
    sentences = sentence_tokenizer.tokenize(document)
 
    bow_matrix = CountVectorizer().fit_transform(sentences)
    normalized = TfidfTransformer().fit_transform(bow_matrix)
 
    similarity_graph = normalized * normalized.T
 
    nx_graph = nx.from_scipy_sparse_matrix(similarity_graph)
    scores = nx.pagerank(nx_graph)
    return sorted(((scores[i],s) for i,s in enumerate(sentences)),
                  reverse=True)

Using this technique, we have a simple way of choosing relevant sentences from a text. This model can also be adapted to find keywords that are important in a text, or used as part of more sophisticated scoring methods.

Programming With Only Functions

2012-02-23T21:55:41Z

Python's lambda keyword does not get the love it deserves. It can only execute expressions, instead of statements. With this restriction, we can still make something interesting.

A Tiny Computation System

The lambda calculus is a system of computation based on mathematical functions. It is the foundation for functional languages. This contrasts to the Turing machine, which is the foundation for imperative languages. It turns out that, in Python, all we need to implement the lambda calculus is the lambda keyword.

The identity functions would be:

identity = lambda x: x

Encoding

identity isn't very interesting, so let's do some calculations. Since we have only functions, we need some way to encode numbers as functions. Let's start with zero.

zero = lambda f: identity

zero takes a function f, and returns the identity function. For clarity, this expands to:

zero = lambda f: lambda x: x

After zero, we have one:

one = lambda f: lambda x: f(x)

And then, two, three, and four:

two = lambda f: lambda x: f(f(x))
three = lambda f: lambda x: f(f(f(x)))
four = lambda f: lambda x: f(f(f(f(x))))

Do you see the pattern yet? For a given number n, we can encode it as

n = lambda f: lambda x: f(f(f(....f(x)...)))

Such that there are n fs. This is method of encoding numbers is known as Church encoding.

Decoding

In order to turn an encoded number into a Python number, we need to decode it using a function of the form:

decode = lambda num: num(f)(x)

We know that decode(zero) should be 0 and that zero ignores f. Therefore x should be 0. We also know that f(0) should be 1 and f(f(0)) should be 2, and so on. Therefore f should be lambda x: x + 1. Substituting this, we get our definition of decode:

decode = lambda n: n(lambda x: x + 1)(0)

We can check this real quick:

assert decode(zero) == 0
assert decode(one) == 1

Successors

It would be rather tedious if we had to type out every definition of every number we needed. Is there a way we can get the successor of a number from just the definition of the number? Something like the following:

one = succ(zero)

We know a number definition requires two lambdas, so we'll need three in order to accept the original number. Therefore our definition will look like this:

succ = lambda n: lambda f: lambda x: ????

We know from the definition of decode that we can get the value of a number n by calling it it like n(f)(x). In order to get the successor of n, we'll need one more f. Therefore we have:

succ = lambda n: lambda f: lambda x: f(n(f)(x))

For easy testing, let's declare the following helper function:

equals = lambda x, y: decode(x) == decode(y)
assert equals(succ(zero), one)

Now we can easily define more numbers:

two = succ(one)
three = succ(two)
four = succ(three)
five = succ(four)
six = succ(five)

Addition

Now let us define a function plus that implements addition. An example usage:

assert equals(plus(two)(three), five)

If we think about it, succ could have been defined as plus(one). Is there a way we can include a one in the definition fo succ?

succ = lambda n: lambda f: lambda x: one(f)(n(f)(x))

For plus, we'd like to be able to specify numbers besides one, so we'll add another lambda:

plus = lambda m: lambda n: lambda f: lambda x: m(f)(n(f)(x))

Multiplication

When defining plus, we saw that if we have a a number m, m(f)(x) will add the decoded m to x. We also know that one(f)(x) adds 1 to x. Therefore, m(one(f))(x) should be equivalent to m(f)(x). We can use this to define multiplication:

mult = lambda m: lambda n: lambda f: lambda x: m(n(f))(x)
assert equals(mult(two)(three), six)

Exponents

We've started to combine these base functions we have in interesting ways. Since we know that m(n(f)) in the defintion of mult repeats the addition of n a total of m times, and that plus(n) is the addition of n, we can rewrite mult to use fewer lambdas:

mult = lambda m: lambda n: m(plus(n))(zero)

Since using multiplication to define exponents is a similar to using addition to define multiplication, we should be able to adapt this definition of mult for exp:

exp = lambda m: lambda n: n(mult(m))(one)
equals(exp(two)(two), four)

More Encodings

We aren't limited to just natural numbers when using the lambda calculus. We can encode boolean logic and lists, allowing us to write programs with control flow and data structures.

Combinators in Python

2012-02-20T18:19:35Z

Inspired by the Coffeescript library Katy, I've been exploring using combinators with Python. The result can be found in my Functional Python repository.

In my previous post on Function Python, I was attempting to translate the following imperative code into different styles.

def imperative_style(xs):
    results = []
    for x in xs:
        if x >= 7: 
            break
        if x < 2:
            result = 4 * x
            results.append(result)
    return results
 
assert imperative_style(range(10)) == [0, 4]

By using the linked Combinators class, we can translate that to the following:

from itertools import takewhile
 
def fluent_combinator_style(xs):
    return bw(xs).chain()\
                 .R(takewhile, lambda x: x < 7)\
                 .R(filter, lambda x: x < 2)\
                 .R(map, lambda x: 4 * x)\
                 .value()
    
assert fluent_combinator_style(range(10)) == [0, 4]

The combinator methods work by altering the form of function calls. The R method transforms bw(wrapped_value).R(takewhile, lambda x: x < 7) into the call takewhile(lambda x: x < 7, wrapped_value).

What I find interesting about this style is that it allows the end user of a class to add something that looks like a method to an object.

As I experimented using this class wrapping iterables, it began to remind me of Underscore.js. This resulted in a tiny (69 line) iterable wrapper It. Using It, we can redefine the above like so:

from itertools import takewhile
 
def it_style(xs):
    return It(xs).chain()\
                 .R(takewhile, lambda x: x < 7)\
                 .filter(lambda x: x < 2)\
                 .map(lambda x: 4 * x)\
                 .value()
 
assert it_style(range(10)) == [0, 4]

Notice how It doesn't support takewhile out of the box, but the user can easily add it to the method chain. It provides flexibility on the level of the object instance, allowing the user to add functionality in a lightweight manner.

Simple REST URIs

2012-02-11T00:55:35Z

When talking about REST APIs, one of the frequent questions pertains to the benefits of using an Accept header, rather than file extensions, to do content negotiation. Often, the benefits of the Accept header don’t seem to outweigh the familiarity of file extensions.

An Example Application

Let's say I have a small command line application that prints out issue information from an issue tracker corresponding to a local feature branch in Git. Usage may look like so:

$ git-issue-branch some-issue
Found a bug

I'm having a problem with this.

I could store the information in SQLite with a schema like this:

CREATE_TABLE issue_branches (
   id integer primary key,
   branch text,
   issue_uri text       
);

The script to do this would have very simple logic: lookup the issue_uri from the db, do an HTTP GET, parse the response, and print the title and body of the issue.

Transitioning the application

Let's say that this application uses an XML API, but I'd rather use JSON. The incidental complexity associated with this change depends on how the server handles representations. If the API honors Accept headers, this is easy: the client changes a header.

If the API doesn't honor these headers and instead relies on file extensions, things become more complex: the client developer needs to parse the URIs, replace the old extension with the new extension, test the URIs to make sure they are well-formed and functional, and perform data migration.

Client Simplicity

Allowing file extensions to determine the content type is fine, and it can be helpful in environments that don't allow setting headers. By allowing only file extensions to determine the content type, though, the API designer forces a more complex implementation on the client developer.

As API designers, we often focus on designing for a specific client, when we need to focus more on allowing the client developer decide implementation details.

Functional Python

2012-02-06T14:19:35Z

It seems to me that Python as a language has a weird relationship to functional programming. Things such as the itertools module and the builtin map, filter, and reduce rely heavily a functional style, yet these approaches are often not idiomatic.

As I explore other languages, I wonder what would Python written in a functional style would look like. An evening spent hacking on it has produces some interesting results. These techniques probably shouldn't be seriously used, but this code is just for fun.

Let's examine a short imperative script

def imperative_style(xs):
    results = []
    for x in xs:
        if x >= 7: 
            break
        if x < 2:
            result = 4 * x
            results.append(result)
    return results
 
assert imperative_style(range(10)) == [0, 4]

Whenever I see code like the above imperative_style, I mentally file it as point of possible complexity and bugs, especially as requirements change and more logic is tacked on.

As a comparison, let's look at a simple functional version. First, we're going to need takewhile from itertools, which will allow us to build something like the break statement

After that, we have the function definition, utilizing two builtins, map and filter, as well as takewhile to break the problem down into logically independent parts. Notice that the conditional in the takewhile is inverted.

from itertools import takewhile
 
def functional_style(xs):
    return map(lambda x: 4 * x,
               filter(lambda x: x < 2,
                      takewhile(lambda x: x < 7, xs)))
 
assert functional_style(range(10)) == [0, 4]

There are less moving parts here, but it seems too much like a Christmas tree for me. Can we make it flatter?

For this, we're going to need more tools for working with functions. First is compose, which let's us feed the result of one function as the argument of another. I'm always surprised that Python doesn't have this builtin. In more functional languages this is a basic feature, with Haskell making it one character (.)

def compose_two(g, f):
    """Function composition for two functions, e.g. compose_two(f, g)(x) == f(g(x))"""
    return lambda *args, **kwargs: g(f(*args, **kwargs))
 
assert compose_two(lambda x: x * 2,
                   lambda y: y + 4)(1) == 10
 
def compose(*funcs):
    """Compose an arbitrary number of functions left-to-right passed as args"""
    return reduce(compose_two, funcs)

With compose, we need one more function, partial, which can be used to provide only some of a function's arguments.

from functools import partial
composition_style = compose(
    partial(map, lambda x: 4 * x),
    partial(filter, lambda x: x < 2),
    partial(takewhile, lambda x: x < 7))
 
assert composition_style(range(10)) == [0, 4]

There's a quite a bit of boilerplate in this definition. Can we abstract out a reusable pattern?

from itertools import starmap
def composed_partials(*partial_funcs):
    return compose(*starmap(partial, partial_funcs))
 
composed_partials_style = composed_partials(
    (map, lambda x: 4 * x),
    (filter, lambda x: x < 2),
    (takewhile, lambda x: x < 7))
 
assert composed_partials_style(range(10)) == [0, 4]

This is less noisy, but it's a bit difficult to read the logic in reverse order. Can we change that?

def pipe(*partial_funcs):
    return composed_partials(*reversed(partial_funcs))
 
pipe_style = pipe(
    (takewhile, lambda x: x < 7),
    (filter, lambda x: x < 2),
    (map, lambda x: 4 * x))
 
assert pipe_style(range(10)) == [0, 4]

This definition is more dataflow oriented, much like using pipes in a shell, or Clojure's -> macro. If you look at it just right, this definition looks a bit like a lisp with currying.

This may be too far, but looking at the definition for composed_partials and pipe, they follow a similar structure. Can we extract that out?

def transform_args(func, transformer):
    return lambda *args: func(*transformer(args))
 
composed_partials = transform_args(compose, partial(starmap, partial))
pipe = transform_args(composed_partials, reversed)
 
assert composed_partials_style(range(10)) == [0, 4]
assert pipe_style(range(10)) == [0, 4]

With that, our final complete version is the following:

from itertools import takewhile, starmap
from functools import partial
 
def compose_two(g, f):
    """Function composition for two functions, e.g. compose_two(f, g)(x) == f(g(x))"""
    return lambda *args, **kwargs: g(f(*args, **kwargs))
 
def compose(*funcs):
    """Compose an arbitrary number of functions left-to-right passed as args"""
    return reduce(compose_two, funcs)
 
def transform_args(func, transformer):
    return lambda *args: func(*transformer(args))
 
composed_partials = transform_args(compose, partial(starmap, partial))
pipe = transform_args(composed_partials, reversed)
 
pipe_style = pipe(
    (takewhile, lambda x: x < 7),
    (filter, lambda x: x < 2),
    (map, lambda x: 4 * x))

It's a bit longer than the original version, but provides small reusable parts that can be quickly assembled into larger programs.

Using Pushstate with Backbone.js and Django

2011-12-30T20:54:35Z

After working on a few projects with Django and Backbone, I really wanted to replace my hash-driven apps with pushstate. Django-pjax is a library aimed at making this easier for more traditional projects, but would require me to duplicate my display logic server side, without using my current mustache templates. After a few iterations, I've settled on an approach that is DRY enough for my taste. You can view the work in my django-backbone-example project.

One way to keep the project keeps templates DRY is through a template tag that renders mustache templates inline. Combined with a client side template engine, templates can be shared between the client and server. Example usage:

<div class="tweet">
  {%mustache "path/to/my/tweetTemplate" tweet%}
</div>

By using Tastypie Resources in the view, the same data preparation can be used for both methods. The view from the example project is as follows:

class DetailView(TemplateView):
    template_name = 'index.html'
 
    def get_detail(self, pk):
        tr = v1.canonical_resource_for('tweet')
 
        try:
            tweet = tr.cached_obj_get(pk=pk)
        except Tweet.DoesNotExist:
            raise Http404
 
        bundle = tr.full_dehydrate(tr.build_bundle(obj=tweet))
        data = bundle.data
        return data
 
    def get_context_data(self, **kwargs):
        base = super(DetailView, self).get_context_data(**kwargs)
        base['data'] = self.get_detail(base['params']['pk'])
        return base

When adapting a client that already supports the hash-driven routing, it should just be a matter of altering the call to Backbone.history.start() to

Backbone.history.start({
    silent: true,
    pushState: true
});

Client Performance Monitoring with Boomerang and CouchDB

2011-06-04T17:20:16Z

A project that that interested me awhile ago, but I only recently got around to trying was Boomerang from Yahoo! It's a Javascript library that calculates the round trip time to load a page from the user's browser, using either cookies or the WebTiming API if it's available.

This project is exciting since it measures the time from when the user navigates away from the page to when the page has loaded, which could be affected by any number of things, such as latency, download speeds, or uncached static files. While there are best practices and tools to help improve client performance (YSlow, Chrome's built-in performance tools, DOM Monster), they aren't able to provide recommendations based upon real users' performance.

This data is can serve as an constant benchmark, which combined with the VCS data, can allow you to detect changes that result in performance regressions. This also opens up the possiblity of doing split tests on website performance.

Here are a few quick test ideas:

Testing of competing CDN solutions with actual users.
Script loaders versus traditional script tags.
Testing bundle combinations.

Round trip time isn't all that you can measure with Boomerang. It comes with bandwidth, latency, and DNS latency plugins, as well as instructions on how to make more granular timers.

Getting started with Boomerang is pretty easy. For this blog, I wrote a quick node.js app that translated the GET params Boomerang sends into a JSON document and uploads it to CouchDB. The code for this app is available on Github. I went with CouchDB because of the flexibility from it being schemaless, combined with near realtime reports from the incremental map reduce views.

First I grabbed a recent version of Boomerang from the repo, including it in the head. Then, I added the following configuration:

BOOMR.addVar('revision', '{{REVISION}}');
BOOMR.init({
  beacon_url: '{{ANALYTICS_SERVER}}',
  site_domain: "{{site.full_url}}", 
  BW: { enabled: false  }
  });

Here I'm doing a standard Boomerang setup, disabling the bandwidth test because it requires a bit more work, and adding REVISION, a custom Hyde variable, allowing me to keep track of the current git commit.

Here's an example CouchDB view I'm using to monitor this blog.

Map:

function(doc) {
    var timestamp = new Date(Date.parse(doc.timestamp)),
        time_done = parseInt(doc.t_done);
 
    if(timestamp && time_done){
        emit([timestamp.getFullYear(), timestamp.getMonth(), timestamp.getDate()],
             time_done);
    }
}

Reduce: _stats

And some example stats:

{"rows":[
    {"key":[2011,4,30],"value":{"sum":68474,"count":24,"min":146,"max":18990,"sumsqr":520080894}}
]}

With this combination of tools, it's easy to generate quick reports and tests. I'm looking forward to using Boomerang for upcoming projects.

Async Workers in Django with Celery

2011-02-10T22:09:41Z

Celery, the distributed task queue framework, recently introduced asynchronous workers in the 2.2 release. I found that when I tried to use the eventlet worker using Celery's Django integration, it started to hang after making some database calls, at least when using the psycopg2 database backend.

I found that I could get around this by applying eventlet's monkey patch before Django starts loading. I did this by writing a separate manage.py, like the following.

Now I'm able to run ./async_manage.py celeryd -P eventlet -c 100, and not have it hang on database calls.

I've not tried this method with gevent, but a similar approach should work.

Backbone.js and Django

2010-11-25T20:54:35Z

NOTE: Since this post, both Backbone.js and Tastypie have gone through a lot of changes. As of December 30th, 2011, the reference repo on Github has been updated to the latest versions, and changed to use backbone-tastypie, which provides an nice compatibility layer, and stays updated better than the examples in this post.

I've been using backbone.js recently, mostly with node.js, or with the localstore bundled with the example Todo application for quick prototypes. However, I wanted to integrate it within some existing Django applications, but had some trouble.

Since Backbone assumes a REST architecture, I needed some way to expose my Django models. For this, I went with Tastypie, which is the easiest way I've found to write a RESTful API for your app.

I've put together a sample application using Backbone and Tastypie, viewable on Github, though I'll do a quick walkthrough of the code.

Here's the models.py, which defines an overly simple tweet.

from django.db import models
 
class Tweet(models.Model):
    username = models.CharField(max_length=25)
    message = models.CharField(max_length=140)
    timestamp = models.DateTimeField(auto_now_add=True)

Using Tastypie for our API, we setup a resource

from tastypie.resources import ModelResource
from tastypie.authorization import Authorization
from tweets.models import Tweet
 
class TweetResource(ModelResource):
    class Meta:
        queryset = Tweet.objects.all()
        authorization = Authorization()

Then we setup some urlpatterns to access our data.

from django.conf.urls.defaults import *
from tweets.api import TweetResource
 
tweet_resource = TweetResource()
 
urlpatterns = patterns('',
    (r'^$',
     'django.views.generic.simple.direct_to_template',
     {'template':'index.html'}),
    (r'^api/', include(tweet_resource.urls)),
)

From here, you should be able to use curl to test out the api, like so:

$ curl -H 'Accept: application/json' http://localhost:8000/api/tweet/
{"meta": {"limit": 20, "next": null, "offset": 0, "previous": null, "total_count": 0}, "objects": []}

In the above urls, there was a direct to template view, which will be our simple app. This will load Backbone and dependencies, ICanHaz.js for templating, and define a template for displaying tweets, and the basic structure of our page.

{%load verbatim%}
<!doctype html>
<html lang="en">
  <head>
    <title>Django + Backbone.js</title>
 
    <link rel="stylesheet" 
          href="http://yui.yahooapis.com/3.2.0/build/cssreset/reset-min.css">
    <link rel="stylesheet" href="{{MEDIA_URL}}css/style.css">
 
    <script src="//ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.js"></script>
    <script src="http://cachedcommons.org/cache/underscore/1.1.0/javascripts/underscore-min.js">
    </script>
    <script src="{{MEDIA_URL}}js/ICanHaz.min.js"></script>
    <script src="{{MEDIA_URL}}js/backbone-min.js"></script>
    <script>
      TWEET_API = "{%url api_dispatch_list resource_name="tweet"%}";
    </script>
    <script src="{{MEDIA_URL}}js/app.js"></script>
 
    {%verbatim%}
    <script type="text/html" id="tweetTemplate">
      <a class="username">{{username}}</a>
      <p class="message">
        {{message}}
      </p>
      <abbr class="timestamp">{{timestamp}}</abbr>
    </script>
    {%endverbatim%}
  </head>
 
  <body>
    <div id="app">
      <div id="input">
        <strong>What's going on?</strong>
        <textarea id="message"></textarea>
        <button class="tweet">Tweet</button>
      </div>
      <h2>Timeline</h2>
      <ul id="tweets">
      </ul>
    </div>
  </body>
</html>

Notice there is a nonstandard tag, {%verbatim%}, which is used to escape the Mustache templates. This tag is by ericflo, and can be found on Github.

Finally, we need to setup our application code in Javascript. First we'll change Backbone's sync function, to do a GET upon receiving a HTTP CREATED. This requires 2 requests to do a create, so you may want to use some other method in production.

var oldSync = Backbone.sync;
 
Backbone.sync = function(method, model, success, error){
    var newSuccess = function(resp, status, xhr){
        if(xhr.statusText === "CREATED"){
            var location = xhr.getResponseHeader('Location');
            return $.ajax({
                       url: location,
                       success: success
                   });
        }
        return success(resp);
    };
    return oldSync(method, model, newSuccess, error);
};

Next, we'll define our Tweet model. Since we have so little logic in this application, we'll just define the url attribute so we can talk to the server. Tastypie will give the individual tweets a resource_uri attribute, but if it hasn't been persisted to the server, it should default to the collections url. We do this because a POST to the collection should add produce a new tweet. Tastypie also namespaces the resulting JSON, so we'll need to define a parse function on the collection that will return the objects attribute in the data returned from the server.

window.Tweet = Backbone.Model.extend({
  url: function(){
     return this.get('resource_uri') || this.collection.url;
  }
});
 
window.Tweets = Backbone.Collection.extend({
  url: TWEET_API,
  parse: function(data){
      return data.objects;
  }                                             
});

Next are the views that define our application, the first is responsible for displaying the individual tweets on the page.

window.TweetView = Backbone.View.extend({
  tagName: 'li',
  className: 'tweet',
 
  render: function(){
      $(this.el).html(ich.tweetTemplate(this.model.toJSON()));
      return this;
  }                                        
});

Last, we have a top level view that handles the user interaction. It is responsible for creating the collection of tweets, loading and display them. It also listens for a click event on the button that says "Tweet", and will create a new tweet in the collection.

window.App = Backbone.View.extend({
  el: $('#app'),
 
  events: {
      'click .tweet': 'createTweet'
  },
 
  initialize: function(){
      _.bindAll(this, 'addOne', 'addAll', 'render');
      this.tweets = new Tweets();
      this.tweets.bind('add', this.addOne);
      this.tweets.bind('refresh', this.addAll);
      this.tweets.bind('all', this.render);
      this.tweets.fetch();
  },
 
  addAll: function(){
      this.tweets.each(this.addOne);
  },
 
  addOne: function(tweet){
      var view = new TweetView({model:tweet});
      this.$('#tweets').append(view.render().el);
  },
 
  createTweet: function(){
      var tweet = this.$('#message').val();
      if(tweet){
          this.tweets.create({
                                 message: tweet,
                                 username: "Test User"
                             });
          this.$('#message').val('');
      }
  }                                  
});
 
window.app = new App();

And with that we've got a (admittedly very) simple Twitter clone working with our Django model. Hopefully this helps you in creating applications with Django and Backbone.

Using Thread Locals in Django

2010-09-26T23:25:36Z

The Django docs are pretty clear on threadlocals. One good use I can think of for threadlocals is for Basecamp style subdomains.

This seems like a great way to use Django's sites framework, except it requires the SITE_ID to be set in the settings. You can use a middleware that uses threadlocals to set this per request, such as the one provided by django-multisite. This is mostly a drop-in solution, but requires you to think of where your querysets are being built.

Let's take the following example models:

from django.db import models
from django.contrib.sites.models import Site
from django.contrib.sites.managers import CurrentSiteManager
 
class Author(models.Model):
    name = models.CharField(max_length=200)
    active = models.BooleanField()
    site = models.ForeignKey(Site)  
    on_site = CurrentSiteManager()

With the following form:

from django import forms
from models import Author
 
class ExampleForm(forms.Form):
    author = forms.ModelChoiceField(queryset=Author.on_site.filter(active=True))

Unfortunately, this may not work. The queryset will be built using the SITE_ID that is in effect when the form class is declared, resulting in the same SITE_ID being used process-wide.

You can enforce the SITE_ID to be evaluated in the proper context, assuming you aren't doing anything tricky, by changing the queryset upon the form initialization.

class ExampleForm(forms.Form):
    author = forms.ModelChoiceField(queryset=Author.objects)
 
    def __init__(self, *args, **kwargs):
        super(ExampleForm, self).__init__(*args, **kwargs)
        self.fields["author"].queryset = Author.on_site.filter(active=True)

This is just one example of how values in threadlocals may yield application errors when you aren't aware of the context the code is executed in.

Speeding Up Mingus

2010-03-18T03:32:29Z

Inspired by Cody Soyland's excellent post on caching in Django, I decided I was going to work on speeding up Mingus.

In the spirit of Mingus, I decided to use a reusable Django application to achieve this. My choice was StaticGenerator.

With the provided middleware, generating the static pages is simple. Cleaning this up, however, isn't as easy. Mingus leverages a very modular design for building pages, making it a great candidate for template tag and view level caching. Many individual parts are cached and reused for different pages, really decreasing the number of database queries.

My initial strategy was to clear the static cache of any changed pages using Django's Signals. It was simple, if a bit tedious, to set up. And it was incomplete.

Consider what happens when a user visits some page that caches the list of posts in memory, right before a new post is published. If the home page is visited before the cache expires, the new post will not show up in the list of posts, but will be shown in the body of the home page.

With direction from this Django Snippet, I added the ability to clear template caches on update, fixing this problem. Unfortunately, it currently requires hardcoding the template fragment names, making the cache invalidation logic dependent on the templates. I want to get rid of this, but haven't thought of an elegant way to solve this.

As I was writing this post, I stumbled upon another problem. If I saved a post as a draft, then visited it, StaticGenerator would allow anyone to see it. Luckily, Jon Smelquist on Github had a commit that added the ability to generate static pages only for anonymous users. With that and a few more checks, my own fork of StaticGenerator addresses this problem.

This is far from an optimal solution, however. One part of the Mingus home page I haven't been able to solve is the right hand popular links feed. Using full page caching without some sort of time-based clearing, these links are going to be out of sync from page-to-page. I've chosen to drop that module for now. Also, the Requests app doesn't receive data on cached pages. I don't see this as too much of a loss, as Google Analytics reports more information for those pages.

For the curious, here's a quick benchmark of the home page. I'm making no claims about this data, only publishing it.

$ ab -n 10000 -c 100 http://joshbohde.com/
... 
Document Path:          /
Document Length:        4975 bytes
 
Concurrency Level:      100
Time taken for tests:   1.097 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      51994488 bytes
HTML transferred:       49869400 bytes
Requests per second:    9118.79 [#/sec] (mean)
Time per request:       10.966 [ms] (mean)
Time per request:       0.110 [ms] (mean, across all concurrent requests)
Transfer rate:          46301.44 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    5   0.5      5       7
Processing:     3    6   0.9      6      10
Waiting:        2    5   1.0      5       8
Total:          8   11   0.8     11      13
 
Percentage of the requests served within a certain time (ms)
  50%     11
  66%     11
  75%     12
  80%     12
  90%     12
  95%     12
  98%     12
  99%     13
 100%     13 (longest request)

If you're interested you can view this code here. As always, I encourage forking and bug reports.

Herbie Hancock and the Headhunters - Watermelon Man

2010-03-12T21:17:35Z

Some days you just need some funk.