http://simonwillison.net/2010/Apr/25/redis/

Deep link to post

And the biggest performance boost of all: caching 404s and sending Cache-Control headers to the CDN on 404. Upwards of 66% of our server time is spent on serving 404s from spiders crawling invalid urls and from urls that exist out in the wild from 6-10 years ago. [Edit: We dropped our outgoing bandwidth by about 66% and our load average on our web server cluster by about 50% after implementing that change]

Amazing result for such a simple change. Bang for your buck optimizations are always the best ones.

This document details the internals of the Redis Virtual Memory subsystem. The intended audience is not the final user but programmers willing to understand or modify the Virtual Memory implementation.

Keys vs Values: what is swapped out?

The goal of the VM subsystem is to free memory transferring Redis Objects from memory to disk. This is a very generic command, but specifically, Redis transfers only objects associated with values. In order to understand better this concept we’ll show, using the DEBUG command, how a key holding a value looks from the point of view of the Redis internals.

Read more

We will repeat the same exercise from the first article, creating a username password store, using hashes.

Basic hash overview

Without going into the technical details we can simplify the concept of redis’s as a way to store fields in a redis key. In Pythonic terms we can make a redis key into a basic Python dictionary.

[key] : {‘field’ -> ‘value’, ‘field’ -> ‘value’, ‘field’ -> ‘value’}

Basic hash usage

>>> r.hset("user:adam", "fullname", "Adam Smith")
1
>>> r.hset("user:adam", "password", "thisisapassword")
1
>>> r.hkeys("user:adam")
['fullname', 'password']
>>> r.hvals("user:adam")
['Adam Smith', 'thisisapassword']
>>> r.hgetall("user:adam")
{'fullname': 'Adam Smith', 'password': 'thisisapassword'}

We will break it down line by line here

1. First we are going to make the redis key “user:adam” a hash and set the field “fullname” in that has to the value “Adam Smith” with the redis command hset (Hash Set)
2. Then we’ll do the same thing for the field “password” and set that field in the hash to it’s appropriate value.
3. With the hkeys command we can see all of the keys in the hash set on that redis key
4. hvals returns all of the values in the redis key
5. More useful, is the hgetall command. This will return a Python dictionary of all of the fields set in the key

As you can see, this is an excellent way to store information about an object without “faking” a relation like in our previous tutorial.

As I mentioned in the conclusion of the last article, if we want to change how we store information in redis all we should have to do is to change is change the inner workings of the functions add_user, authenticate_user, delete_user and the rest of our fictitious application should operate without any changes.

Creating a new user

r = redis.Redis("localhost")
from hashlib import md5

def add_user(username, fullname, password):
    if r.sadd("users", username):
        #r.set("user:%s:fullname" % username, fullname)
        #r.set("user:%s:password" % username, md5(password).hexdigest() )
        r.hset("user:%s" % username, "fullname", fullname)
        r.hset("user:%s" % username, "password", md5(password).hexdigest())
        return True
    else:
        return False

I left the original code in the function commented out so we can see the differences in the two methods here. In our original method we used the name space of redis itself to store the reference. In the updated fashion we are using redis’s hash data type to store the related fields.

>>> add_user("adam", "Adam Smith", "wealthofnations")
True
>>> add_user("adam", "Adam Smith", "wealthofnations")
False

This function operates in the same fashion as our old version did but, is using a different data structure in the backend.

Logging a user in

Now we’ll refactor our old authentication code to work with the new backend.

def authenticate_user(username, password):
    #if username in r.smembers("users"):
    if r.sismember("users", username"):
        passhash = md5(password).hexdigest()
        #if passhash == r.get("user:%s:password" % username):
        if passhash == r.hget("user:%s" % username, "password"):
            return True
        else:
            return False
    else:
        return False

Just as in the add_user function, we just need to make a small change. Now instead of fetching the related redis key we simply fetch the field from the hash of “user:*username*”.

>>> authenticate_user("adam", "wealthofnations")
True
>>> authenticate_user("adam", "bad_password")
False

We can see the function again performs just like the old function but, it pulls the password from the hash object instead of the related key

Deleting a user

Deleting a user using the hash store is much easier than using the related key store, since everything is just stored in one key instead of being spread out across multiple keys.

def delete_user(username):
    #if username in r.smembers("users"):
    if r.sismembers("users", username):
        r.srem("users", username)
        r.delete("user:%s" % username)
        #r.delete("user:%s:fullname" % username)
        #r.delete("user:%s:password" % username)
        return True
    else:
        return False
<pre>

Now instead of having to delete the 2 related keys, we just need to delete the one key storing the hash of all of the user data

We hinted at this style of structure in the first article when we got to talking about sets with our reddit like example of story voting.

r_server.sadd("story:5419:upvotes", "userid:9102")

Here we have a set in a redis called “story:5419:upvotes” and every element stored in the set is called “userid:xxxx”. This is an example of a basic relation.

Let’s make this example simpler and go with the concept of username / password storage for a website. Instead of using MySQL to store this information we will use redis.

Relations in a NoSQL world

We have to cheat in redis’s flat name space to make relations in our data. Redis isn’t going to be aware of these relations and unlike RDBMS (like MySQL), redis does nothing to help us out. No index’s, no nifty SQL syntax with WHERE or JOIN to do the work for us. We have to handle all of the relational logic in application code, which in turn means you (the developers) have to do extra documentation explaining just how everything fits together in redis or you are going to lose your data. The benefit you though is raw speed and flexibility during development. Don’t like the constraints one data model is giving you? Start over and just recode; no need to alter a DB schema or change a DB server; redis doesn’t care.

Our requirements

• Users have a username
• Usernames have a password in a one-to-one fashion
• We must be able to create a user and assign a password
• Users must be able to login (check username against password)
• We must be able to delete a username and their associated password

Designing our redis namespace

This the format of the description of the redis namespace we’ll be using
[key] – {datatype} – (example values)

We also will use the notation *variable* to denote that in a key name is a variable relating to something else.

[users] – {set} – (“adam”, “bob”, “carol”)
[user:*username*:fullname] – {string} – (“Adam Smith”, “Bob Barker”, “Carol Burnett”)
[user:*username*:password] – {string} – (md5 hash password, no example)

That’s it. As long as we stick with this design pattern here we can keep track of all of the users in our system and handle logins.

Creating a new user

r = redis.Redis("localhost")
from hashlib import md5

def add_user(username, fullname, password):
    if r.sadd("users", username):
        r.set("user:%s:fullname" % username, fullname)
        r.set("user:%s:password" % username, md5(password).hexdigest() )
        return True
    else:
        return False

We’ve created a function called add_user where we are taking in a username, the full name of the user, and password and will return True if we successfully create a new user and False if we don’t. Remember the redis command SADD (Set Add) returns False if the object already exists in the set, so if a user with that username already exists, we can’t create a new user so we return False; otherwise we add it to the set and then we set the associated keys “fullname” and “password” to the appropriate vaues. Hashing the password with a md5 hash since we never store passwords anywhere in plain text.

Let’s take a look at what this looks like being called.

>>> add_user("bob", "Bob Barker", "priceisright")
True
>>> add_user("bob", "Bob Barker", "priceisright")
False

The first time we try to add the username “bob” we succeed but, the second time it fails since there is already a user with that username. Let’s look at the data stored in redis.

>>> r.smembers("users")
set(['bob'])
>>> r.get("user:bob:fullname")
'Bob Barker'
>>> r.get("user:bob:password")
'543f24fdc95f4e3f52fc7a4f2166167e'

We see our key “user” that has a set stored with only one element, our username “bob”. And now we have our related keys “user:*username*:fullname” and “user:*username*:password” which in practice are “user:bob:fullname” and “user:bob:password”. As long as we stick to the naming convention we should always know where everything is!

Let’s go ahead and create a few more users just for good measure

>>> add_user("adam", "Adam Smith", "wealthofnations")
True
>>> add_user("carol", "Carol Burnett", "eartug")
True
>>> r.smembers("users")
set(['carol', 'bob', 'adam'])

Logging a user in

Now that we have created our relational data store in redis and have a few users in there, lets try to log one in.

def authenticate_user(username, password):
    #if username in r.smembers("users"):
    if r.sismember("user", username):
        passhash = md5(password).hexdigest()
        if passhash == r.get("user:%s:password" % username):
            return True
        else:
            return False
    else:
        return False

First thing we do is check to see if the username is in the set of all of the elements in the key “users” using the method sismember (set – is a member). If the username isn’t there we are going to return False since, that user doesn’t even exist. If the user does exist, I take a hash of their submitted password just like we did when we added the user, then we fetch that password from the related key and see if they are the same, if they are, we return True!

Let’s take a look at this in action

>>> authenticate_user("ghost", "idontexist!")
False
>>> authenticate_user("adam", "keynes")
False
>>> authenticate_user("adam", "wealthofnations")
True
>>> authenticate_user("bob", "priceisright")
True

So we can see here, a user that doesn’t exist is not authenticated. A user that does exist but, has the wrong password is denied. Given the right username and the right password a user gets in!. And just to show we can handle multiple users like a proper relational system should “bob” authenticates properly as well

Deleting a user

Eventually we are going to need to delete a user from our system and that is as easy everything else we have done as long as we stick to the same namespace we laid out before

def delete_user(username):
    if username in r.smembers("users"):
        r.srem("users", username)
        r.delete("user:%s:fullname" % username)
        r.delete("user:%s:password" % username)
        return True
    else:
        return False

Again the first thing we do is check that the user exists first, since if they don’t no need to do everything else. After we know the user exists, we remove them from the set of “users” then remove their related fullname and password keys. Let’s check this out in action

>>> authenticate_user("adam", "wealthofnations")
True
>>> delete_user("adam")
True
>>> authenticate_user("adam", "wealthofnations")
False
>>> delete_user("adam")
False
>>> r.smembers("users")
set(['bob', 'carol'])

We can authenticate the user “adam” properly. Then we delete him, and now he can’t authenticate any more. If we try to delete him again we see that we can’t since he doesn’t exist.

Wrapping Up

We can make very complex relational data structures in redis as long as we document and properly use the name space. Keep everything as simple and as straight forward as possible and document everything throughly. Also make sure all your actions on redis can be keep as modular as possible by wrapping them in functions or classes, just like in this example. Then if for some reason you need to alter your “schema” you only have one place to change it in your application.

Footnote:

This is an example and is not a secure method for handling user submitted data and does no sanitization or sanity checks on the information. I am aware of this but, this post was not designed to be a comprehensive analysis of AAA methods just a quick example of how to do basic relations in a key-value store.

A very simplified explanation of redis is that it is an in memory key-value store like memcached but, it is persistent on disk, unlike memcached which is volatile. Along with being disk-persistent redis also supports some basic data structures like lists, sets, ordered sets, hashes, and of course basic key-value string storage like memcached. Redis, with disk-persistence and basic data structures, remains blazing fast with published benchmarks of 110,000 SETs per second, about 81,000 GETs per second.

This post is the start of a series of articles on redis for Python programmers. A prerequisite for this is going to be some basic Python knowledge, which if you haven’t used before I highly recommend the free web book Dive Into Python. This is going to be a simple overview of the basic data types and usage for Python programmers and we will slowly progress into more complex usages, so if you haven’t done anything with redis before this is a perfect start.

Installing Redis

Redis is extremely easy to install on POSIX systems (Linux, OSX, Cygwin for Windows). Simply download either a release version or checkout the trunk version of redis from the project home page. Unpackage, make, and run. That’s it!

gettingstarted $ wget http://redis.googlecode.com/files/redis-1.2.5.tar.gz
gettingstarted $ gunzip redis-1.2.5.tar.gz
gettingstarted $ tar -xvf redis-1.2.5.tar
gettingstarted $ cd redis-1.2.5
redis-1.2.5 $ make
redis-1.2.5 $ ./redis-server
21 Mar 17:37:11 * Warning: no config file specified, using the default config. In order to specify a config file use 'redis-server /path/to/redis.conf'
21 Mar 17:37:11 - Server started, Redis version 1.2.5
21 Mar 17:37:11 - The server is now ready to accept connections on port 6379
21 Mar 17:37:11 . 0 clients connected (0 slaves), 832496 bytes in use, 0 shared objects

6 commands and redis is running and ready for a connection. Obviously this isn’t best practice for running redis in production but, it works well for just getting it up and running for some experimentation. Let’s leave that window open and open another for a Python console.

In every example below we only are covering the basic operations of each type. If you want to learn about all of the possible operations for redis read up on the redis Command Reference. This has all of the commands available to you.

Redis and Python

First thing we need is the redis-py module for Python. You can install this manually or just use easy_install.

easy_install redis

Everything has a place: keys

Redis is a key-value store just like a Python dictionary. So everything that goes into redis no matter what it is, has a key (or location, or address, whatever makes sense to you). You can’t store something without a place for it to go

Now let’s open up a Python console and start to play with the basic data structures

Data structure: String

The most basic data structure in redis is the string. Simply, you have a key and it stores a string.

>>> import redis
>>> r_server = redis.Redis("localhost")
>>> r_server.set("name", "DeGizmo")
True
>>> r_server.get("name")
'DeGizmo'

So, line by line let’s walk through what we’ve done here.

1. import the redis module
2. create an object called “r_server” of redis.Redis type with the parameter “localhost” which is the network location of our running redis server
3. call the function “set” on the “r_server” object with the parameters “name” and “DeGizmo”. The first parameter “name” is the key and “DeGizmo” is the value. So in redis we created a key-value pair of “name” -> “DeGizmo”
4. call the function “get” which will retrieve the value from the key which is passed as the parameter. We are getting the key “name” and redis returns the value “DeGizmo”

That’s it. If you go and stop the redis server and restart it and run r_server.get(“name”) command again, it will return “DeGizmo” again due to redis’s persistent store.

Data structure: Integer

The second data type available is an integer.

>>> r_server.set("hit_counter", 1)
True
>>> r_server.incr("hit_counter")
2
>>> r_server.get("hit_counter")
'2'
>>> r_server.decr("hit_counter")
1
>>> r_server.get("hit_counter")
'1'

Again using the “r_server” Redis object we run the following commands

1. We set the redis key “hit_counter” to a value of 1 just like in the example above
2. Then we run the command “incr” on the key “hit_counter”. This will increment (incr) the key by 1
3. We get the value for the key “hit_counter” to verify that it was been incremented
4. Then we run the command “decr” on the key “hit_counter”. This will decrement (decr) the key by 1
5. We get the value for the key “hit_counter” to verify that it was been decremented

Simple huh? Well remember that redis is a persistent store that can be accessed by multiple machines (or programs) simultaneously. If you have a horizontally scaled web farm with many different web servers running on separate machines, your web app can all call on and reference the same redis instance and all operate on the “hit_counter” key at the same time and increment it just like how you would use a MySQL database. You can kind of think of redis like a “shared” storage across different applications.

Data structure: Lists

Very similar to the built in Python list type, the redis list type has a few basic methods that combined can quite powerful. We are only covering a tiny portion of the commands available, you can find all the commands in the redis Command Reference.

>>> r_server.rpush("members", "Adam")
True
>>> r_server.rpush("members", "Bob")
True
>>> r_server.rpush("members", "Carol")
True
>>> r_server.lrange("members", 0, -1)
['Adam', 'Bob', 'Carol']
>>> r_server.llen("members")
3
>>> r_server.lindex("members", 1)
'Bob'

We don’t actually have to create or declare anything when using redis, we just use it and redis handles everything else for us. So we just go ahead and tell redis at the key “members” we are going to add (rpush, right push) the value “Adam”. Redis creates the list for us at key “members” and adds the value “Adam” to the list.

1. With the r_server object again we call the method “rpush” which will add the value “Adam” to the newly created list “members”
2. We add “Bob” to the same list
3. Finally we’ll add “Carol”
4. With the lrange method we are asking redis to return all the objects in “members”. lrange takes 3 arguments: key, start index in list, end index in list. We are getting the objects from the key “members” starting at index 0 and ending at -1 (which is technically the -1, or last index in the list, which will return everything)
5. The llen method asks redis to return the length of the list at the key “members” which now has 3 objects
6. lindex method tells redis that we want the object from the key “members” at the index position of 1 (remember lists start at index 0), so redis returns “Bob”

We’ve got some elements in the list at the key “members”; now lets get remove some elements.

>>> r_server.rpop("members")
'Carol'
>>> r_server.lrange("members", 0, -1)
['Adam', 'Bob']
>>> r_server.lpop("members")
'Adam'
>>> r_server.lrange("members", 0, -1)
['Bob']

1. With the method rpop (right pop) we remove the element in the list on the right side (tail), which is “Carol”
2. Now when we ask for the list “members” from redis again (from the start of 0, and the end of -1 which returns everything) we see our list now doesn’t have “Carol”
3. We now lpop (left pop) an element from the list “members”. This will remove the far left element from the list, “Adam”
4. Now the entire list only contains “Bob”

Data structure: Set

Again, sets perform identically to the built in Python set type. Simply, sets are lists but, can only have unique values. In the above example if we added the value Adam (r_server.lpush(“members”, “Adam”) ) 20 times our list would contain 20 values all containing the value “Adam”. In a set, all elements are unique.

>>> r_server.delete("members")
True
>>> r_server.sadd("members", "Adam")
True
>>> r_server.sadd("members", "Bob")
True
>>> r_server.sadd("members", "Carol")
True
>>> r_server.sadd("members", "Adam")
False
>>> r_server.smembers("members")
set(['Bob', 'Adam', 'Carol'])

1. First off we delete the old list in the key “members” so we can use it as a set
2. Then we sadd (set add) the value “Adam” to the key “members”
3. Do the same for the value “Bob”
4. Do the same for the value “Carol”
5. Now we try to add the value “Adam” to the key “members” again but, this time it returns “False” since we are working on a set, and a set only has unique values. There already is a value “Adam” present in this set
6. The method smembers returns all the members of the set in the Python type set

An example of a use of sets in a web application would be for “upvotes” on a reddit, or hacker news type website. We want to keep track of who up votes a story but, you should only be able to up vote a story once.

>>> r_server.sadd("story:5419:upvotes", "userid:9102")
True
>>> r_server.sadd("story:5419:upvotes", "userid:12981")
True
>>> r_server.sadd("story:5419:upvotes", "userid:1233")
True
>>> r_server.sadd("story:5419:upvotes", "userid:9102")
False
>>> r_server.scard("story:5419:upvotes")
3
>>> r_server.smembers("story:5419:upvotes")
set(['userid:12981', 'userid:1233', 'userid:9102'])

I added a little twist in here with the name of the key: “story:5419:upvotes” but, it’s easy to explain. Redis is “flat” with it’s keyspace. So if we have many different stories we use a fixed key naming convention for easy reference in redis. For this example our key is broken down like this: “object type” : “id” : “attribute”. So, we have an object of type “story” with an ID of “5419″ with an attribute “upvotes’. Redis has no idea what any of this means it just knows the key is “story:5419:upvotes” but, it doesn’t matter, we know what it means and we can divide up our objects into this name space to make it easier to work with and keep from “losing” things. The value being added to the key is divided up in the same way. “userid” is the type and “9102″ is the value or the ID for that user voting on the story.

1. Just like before we are adding the value “userid:9102″ to the key “story:5419:upvotes”
2. Now we are adding the value “userid:12981″
3. Finally adding the valud “userid:1233″
4. Now, the user with the ID 9102 tries to up vote the story with the ID 5419 again, and redis returns False since that user has already up votes this story before and you can’t up vote a story twice!
5. The method “scard” is asking redis for the cardinality of the set at key “story:5419:upvotes” or how many things are in this set, and redis returns 3.
6. Finally we return the list of userid’s that we have stored in the set.

Data structure: Ordered sets

The last data structure we are going to talk about today is an ordered (or sorted) set. This operates just like a set but, has an extra attribute when we add something to a set called a “score”. This score determines the order of the elements in the set. We will stick with the concept for this final example

>>> r_server.zadd("stories:frontpage", "storyid:3123", 34)
True
>>> r_server.zadd("stories:frontpage", "storyid:9001", 3)
True
>>> r_server.zadd("stories:frontpage", "storyid:2134", 127)
True
>>> r_server.zadd("stories:frontpage", "storyid:2134", 127)
False
>>> r_server.zrange("stories:frontpage", 0, -1, withscores=True)
[('storyid:9001', 3.0), ('storyid:3123', 34.0), ('storyid:2134', 127.0)]
>>> frontpage = r_server.zrange("stories:frontpage", 0, -1, withscores=True)
>>> frontpage.reverse()
>>> frontpage
[('storyid:2134', 127.0), ('storyid:3123', 34.0), ('storyid:9001', 3.0)]

Quick namespace explanation like before. For the key we are going to be referring to “stories:frontpage” which is going to be a set of stories slated for the front page of our website. We are storing in that key the value of “storyid:3123″ which is the ID of some story on the site and then a score, which in our case is going to be the number of votes on a story.

1. First we add the value “storyid:3123″ to “stories:frontpage”, and “storyid:3123″ in our example is going to have 34 votes.
2. Then add “storyid:9001″ with 3 votes
3. Then add “storyid:2134″ with 127 votes
4. We are going to try to add “story:2134″ to the set again but, we can’t since it already exists.
5. Now we are going to ask redis for all the elements in “stories:frontpage” from index 0 to index -1 (the end of the list) with all associated scores (withscores=True)
6. We’ve got the scores but, they are in ascending order, we want them in descending order for our website, so we are going to store the results in the variable “frontpage”
7. Then reverse it (which is an in place operation in Python)
8. Now print out the front page!

In conclusion let’s do a quick example of a “view” in an application in which a user will vote of a story using redis as a storage engine

#given variables
#r_server   = our redis server
#user_id    = the user who voted on the story
#story_id   = the story which the user voted on
if r_server.sadd("story:%s" % story_id, "userid:%s" % user_id):
    r_server.zincrby("stories:frontpage", "storyid:%s" % story_id, 1)

2 lines of code? This is might compact but, once we unravel it we can see how it makes sense and how powerful redis can be. Let’s start with the if statement.

if r_server.sadd("story:%s" % story_id, "userid:%s" % user_id):

We know the command “sadd” already. This will add an element to a set at a key. The key in this case is

"story:%s" % story_id

If story_id is 3211, then the resulting string will be “story:3211″. This is the key in redis which contains the list of users that has voted on the story.

The value to be inserted at this key is

"userid:%s" % user_id

Just like with story, if the user_id is 9481 then the string to be inserted into the set at “story:3211″ will be “user_id:9481″

Now the redis command “sadd” will return False if that element is already present in the set. So if a user has already voted on this story before we don’t execute the statement under the if. But, if it is added, then we have a brand new vote and we have to increment the votes for the front page.

r_server.zincrby("stories:frontpage", "storyid:%s" % story_id, 1)

We have an ordered set at the key “stories:frontpage” and we are going to increment the element “storyid:%s” % story_id (“story:3211″) by 1.

And now we’re done! We’ve made sure the user hasn’t voted on this story before and then we’ve incremented the number of votes for this story on the front page!

This section will be a bit short because we won’t really be using templates, which we’ll cover in the next section. We will just be looking at how the urls.py file and the views.py file interact and how we can use what we learned in our ORM tutorial not in the shell but, in our application.

So remember our urls.py file from the admin interface tutorial?

from django.conf.urls.defaults import *

from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
    (r'^admin/', include(admin.site.urls)),
)

It’s pretty sparse and doesn’t do much. Let’s make a home page

from django.conf.urls.defaults import *
from django.contrib import admin
admin.autodiscover()

from main.views import *

urlpatterns = patterns('',
    (r'^admin/',            include(admin.site.urls)),
    (r'^$',                 home),
)

We’ve added two lines here. The first “from main.views import *” is importing all the functions that are in the views file for the main application in our blog project (right now there are no applications). The second line I’ve added, “(r’^$’, home),”, is a tuple that I’ve added to the urlpatterns variable. This variable is where Django looks to find all the URL’s our project uses. The first part of the tuple is a regex expression for basically, nothing. It’s what happens when you just hit the domain of our site (like simpledjango.com). The second part of the tuple is the function “home”. If you goto your development server at http://127.0.0.1:8000you’ll see a Name Error. This function isn’t defined so lets go to the views.py in the main application (blog/main/views.py) and add the function home.

# Create your views here.
def home(request):
    pass

This is what my views.py file looks like now. When we refresh our development server you’ll see a new error.

ValueError at /
The view blog.main.views.home didn’t return an HttpResponse object.

Django requires response object to be returned from every view. Let’s import the HttpResponse object and have our function return it.

# Create your views here.
from django.http import HttpResponse
def home(request):
    return HttpResponse()

If we refresh our page again we’ll see the error is gone and we now have a blank page. Let’s type something in the HttpResponse object and see what happens

# Create your views here.
from django.http import HttpResponse
def home(request):
    return HttpResponse("testing, testing 1 2 3 ")

Refresh the page and we’ll see “testing, testing 1 2 3″ on our page. Anything that is passed into the HttpResponse object instance will essentially be passed along to our browser.

Let’s get some data out of our database and pass it along. Let’s try to display all the categories that we have

# Create your views here.
from django.http import HttpResponse
from main.models import Category
def home(request):
    cats = Category.objects.all()
    return HttpResponse(cats)

The first difference you’ll see is the “from main.models import Category”. Here we are importing the model Category that we created in our models tutorial so we can use it in our view here. Then just like in the DB-API tutorial we are storing all the categories in a variable called cats then passing that into the HttpResponse object instantiation. When we refresh our home page we’ll see all of our categories listed out, although in a quite ugly fashion.

So as you can see getting data from our database back to the web browser is really just the same thing as using it via the DB-API in the shell. I said this would be a short tutorial since the focus was really just bringing together the concepts of using the database via the ORM and taking it out of the shell into our web application.

The next tutorial will talk about how we can really use the information in the DB on web pages using templates. (Coming Soon)

Download at this step: GitHub

$ python manage.py shell
Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>

Now this might look like a normal Python shell because, well it is. We’ve just loaded up some stuff in the background like path information to make our lives a bit easier. Now we have two models that we created in the application “main”. Let’s load those up in the shell.

>>> from main.models import Category
>>> from main.models import BlogPost

Before we get too deep here, I want to mention that everything that we do in the shell we can do in our application. It’s the same thing. So any manipulation we make here we can use the exact same code anywhere in our application. The shell is a great way to test out ideas and algorithms before we put that code into our application. We have our two models loaded up lets play with the built in “objects” Manager.

>>> BlogPost.objects.all()
[<BlogPost: Our First Post>]
>>> Category.objects.all()
[<Category: Books>]

For both of our models we are calling the function all() in the object manager. This will return a special Django object called a QuerySet. These are very powerful objects which we’ll see in a second. Let’s get all of the posts for a specific category.

>>> Category.objects.all()
[<Category: Books>]
>>> cat = Category.objects.all()[0]
>>> cat
<Category: Books>

Now even though objects.all() returns an object called a QuerySet it also works just like a standard array. So I’ve just taken the first element of the array “[0]” and assigned it to cat. So now we have a variable “cat” which contains one of our Categories called “Books”. We can see that the category has a Name which is called “Books”. Remember when we defined our __unicode__ object back in the create our models part. This is why python is calling our object . If we hadn’t defined the __unicode__ function python would print out our object like this:<Category: Category object>

Now that we have a specific category lets find all the blog posts associated with that category. I’m going to show you two different ways of doing this just show you two different new things.

>>> BlogPost.objects.filter(Category=cat)
[<BlogPost: Our First Post>]

>>> all_blog_posts = BlogPost.objects.all()
>>> all_blog_posts
[<BlogPost: Our First Post>]
>>> all_blog_posts.filter(Category=cat)
[<BlogPost: Our First Post>]

In the first line of code we are taking the same BlogPost.objects but, instead of calling all() we are calling filter. Filter is a function where we can compare any field from our model declaration in models.py and find only things that match that comparison. So here we are finding all the BlogPost that have a Category=cat (which is Category: Books).

The second example we are taking all the objects from BlogPost and putting them into all_blog_posts. Now this is a QuerySet object we mentioned before. It has the same methods as all of our model’s. So I can simple just filter that QuerySet for all of the posts that are in the Category : Books. The result of BOTH of those examples is still a QuerySet so I can filter them again, and again. There are a lot of functions in QuerySet’s that are very cool. We only touch on a very of them here. For a more in depth look goto our Quick Reference: QuerySets

So we’ve seen the basic ways to retrieve objects. Let’s create a new BlogPost from the shell.

>>> from django.contrib.auth.models import User
>>> User.objects.all()
[<User: DeGizmo>]
>>> degizmo = User.objects.all()[0]

>>> new_post = BlogPost()
>>> new_post.Title = "blog post from the shell"
>>> new_post.Category = cat
>>> new_post.Slug = "this-slug-shell"
>>> new_post.PostText = "this is the text of the blog post!"
>>> new_post.Author = degizmo
>>> new_post.save()
>>> BlogPost.objects.all()
[<BlogPost: Our First Post>, <BlogPost: blog post from the shell>]

The first thing we did was import the model “User” which we used before when we declared our model. Then I selected myself (the only user) and assigned it to the variable “adam”.

Now I created a new BlogPost object called new_post and went through each field in the model and assigned it a value. Then I called the function .save() which saves the new object to our database. Which you can see when we look at all of the objects stored in BlogPost. Go check the admin interface and you’ll see our new BlogPost right there along side of the rest of them.

Let’s say we want to change the title of that blog post for some reason (maybe we don’t like the capitalization).

>>> BlogPost.objects.all()
[<BlogPost: Our First Post>, <BlogPost: blog post from the shell>]
>>> BlogPost.objects.get(Title="blog post from the shell")
<BlogPost: blog post from the shell>

>>> edit = BlogPost.objects.get(Title="blog post from the shell")
>>> edit.Title = "Blog Post From The Shell!"
>>> edit.save()
>>> BlogPost.objects.all()
[<BlogPost: Our First Post>, <BlogPost: Blog Post From The Shell!>]

So once again we look at all of our BlogPost objects. This time I use a different function called get. This function doesn’t return a QuerySet but, just a single object from the database. It works the same way as filter, you tell it which field you want to compare in the database and it will return the right object. I just used the post title we want to change. Once I have stored the right post in the variable “edit” I edit the field “Title” then save it to back to the database with the function call save()

Maybe we don’t like that post at all anymore so let’s delete it.

>>> BlogPost.objects.all()
[<BlogPost: Our First Post>, <BlogPost: Blog Post From The Shell!>]
>>> edit = BlogPost.objects.get(Title="Blog Post From The Shell!")
>>> edit
<BlogPost: Blog Post From The Shell!>
>>> edit.delete()
>>> BlogPost.objects.all()
[<BlogPost: Our First Post>]

I got the BlogPost I wanted the same way as before using the “get” function, stored it in the variable edit. Then just called the function delete()

From the console we’ve returned all the categories, blog posts, posts just for a category, a single blog post, edited a blog post and deleted a blog post. Without writing a single line of SQL. Odds are you’ll almost never have to write a single line of SQL while working in Django unless you have some really complex models.

Now that we know how to get data out of the database let’s write our first view

This is where we will start working with the models.py file in the “main” application directory (blog/main/). The models.py file is a definition file where we tell Django what all of our database models are. Let’s start with some categories for our blog entry.

from django.db import models
from django.contrib import admin

class Category(models.Model):
    Name    =   models.CharField(max_length=255)
    def __unicode__(self):
        return self.Name
admin.site.register(Category)

Let’s walk through this real quick since this was a lot of code real quickly. The first two lines we are first importing Django’s base class “models”. This is what we’ll subclass all of our models off of. The next line imports the Django admin object which we will use to include our model in Django’s admin interface.

Next we define our model “Category” subclasses it from the Django base class models.Model. Then we define a field in our model called “Name”. This is the name of a category and we assign name an instance of another subclass of a base Django class called “CharField” this is a simple multi-character field with a maximum length of 255 characters. You can find out more about the types of fields in our Quick Reference: Models page.

Next we define a function called __unicode__ in our new Category class. This is a special function which tell’s Django and Python when we want to “print” an instance of this class call this function. This function will return the value “Name” from the object so that a Category with the name “Books” will return “Books” when it is printed out.

The next line “admin.site.register(Category)” will tell Django to add our new model Category to the admin site.

Let’s go ahead and see what this little snippet of code does before moving on. From the console let’s first sync our database again

$ python manage.py syncdb
Creating table main_category

Here we see that Django has created a table in our database called main_category. “main” is the name of our application and “category” is the name of our class. Let’s go ahead and run the development server again and just keep it running in the background this time. It will reload automatically when files in your project and application are changed so I won’t remind you anymore to run the development server past this point in the tutorial

$ python manage.py runserver
Validating models...
0 errors found

Django version 1.1 alpha 1, using settings 'blog.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Now let’s go back to the admin interface at http://127.0.0.1:8000/admin/ and see our new model in the admin interface.

We can see our new model “Categorys” in the application “Main” here. Let’s click on “Categorys” (we’ll fix that in a minute) to see how cool the admin interface is.

Let’s click “Add category” in the top right there and create a new category


Now click on our newly created Category “Books” and check out what the Django Admin interface has done for us.

All we’ve done is defined our model and Django has given us a nice CRUD (Create, Edit, Update, Delete) interface for all of data. The admin interface is actually very powerful tool just by itself and anyone who’s developed web applications knows that doing something like for any project takes a huge amount of time. The admin interface is a massive time saver for any project.

Let’s go ahead and fix that “Categorys” typo before we define another model. Let’s look at our models.py file again and add a “Meta” option to our Category class.

class Category(models.Model):
    Name    =   models.CharField(max_length=255)
    def __unicode__(self):
        return self.Name
    class Meta:
        verbose_name_plural = "Categories"
admin.site.register(Category)

If the change isn’t that self-explanatory we’ve set the value of verbose_name_plural to the correct plural spelling for “Category”. Django will automatically pluralize models but, sometimes it’s not perfect. It already does so much for us let’s not complain about the small things.

Now let’s define our second model. This is for our blog posts.

from django.contrib.auth.models import User
class BlogPost(models.Model):
	Title		= models.CharField(max_length=255)
	Category	= models.ForeignKey(Category)
	Slug		= models.SlugField()
	Author		= models.ForeignKey(User)
	PostText	= models.TextField()
	PublishDate	= models.DateTimeField(auto_now_add=True)
	def __unicode__(self):
	    return self.Title
admin.site.register(BlogPost)

So we should be able to look at this code and begin to figure out what some of it does. The “Title” field is almost identical to our Category -> Name field but, the ForeignKey field is new and worth an explanation. Here we are defining the relationship between a BlogPost and a Category by saying that every BlogPost belongs in some Category. Then we have “SlugField” which is new too. A slug is the URL of a blog post, for example the slug of this blog post is “blog/2010/03/tutorial-create-our-models/”.

Next we have another ForeignKey but, it’s something we haven’t declared. I skipped over the first line in our code up there at first to explain it a bit more in detail here. We imported “from django.contrib.auth.models import User”. This is the “User” class that Django uses. Everyone on our website: admins, authors, commenter, even anonymous users are “Users”. In this line we are saying that every BlogPost has a single author “User”.

Hopefully PostText is self-explanatory so we’ll skip to PublishDate. This is a DateTimeField which not surprisingly stores a Date and a Time. The snippet in the paren’s “auto_now_add=True” means that when the model (the BlogPost) is created the date and time are automatically added instead of being defined by someone.

Let’s resync our database and go back to the admin screen and see what we have now

We see our model “Blog posts” included in our admin. Let’s go into that and write a blog post.

You can see how Django is handling the relationships you’ve defined in your models so far with the “Categories” and “Author” drop downs. Pretty cool huh? This is the real power of the admin interface in Django. It provides a ready made very powerful interface for your database out of the box with very little configuration needed. This admin interface is actually much more powerful that what we’ve covered here.

Go ahead and create another Category and one or two Blog Posts just to get some data into the database for the next part of the tutorial. There we’ll learn about the DB-API and the ORM in Django.

Download at this step: GitHub

Let’s open up the urls.py file in the root project (blog/) directory. I’ve removed the some of the commented out lines below for brevity

from django.conf.urls.defaults import *

# Uncomment the next two lines to enable the admin:
# from django.contrib import admin
# admin.autodiscover()

urlpatterns = patterns('',

    # Uncomment the next line to enable the admin:
    # (r'^admin/', include(admin.site.urls)),
)<

Now just like the comments say we want to uncomment out the “from django.contrib…”, “admin.autodiscover()”, and the “(r^admin/…)” lines in urls.py so it looks like this

from django.conf.urls.defaults import *
from django.contrib import admin

admin.autodiscover()

urlpatterns = patterns('',
 (r'^admin/', include(admin.site.urls)),
)

Now, what does all of this do? The first two lines (the imports) are bringing in the needed modules and the admin. namespace for us. The urlspatterns line is something we’ll be coming back to. A simple explination of it is that this is where all the URL’s of our project will end up living. Right now there is one URL “admin/” that is pointing to another URL file “admin.site.urls”. This is the base URL’s of the Django admin site.

After we’ve saved our changes let’s run the development server again

$ python manage.py runserver
Validating models...
0 errors found

Django version 1.1 alpha 1, using settings 'blog.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

This time let’s goto the URL http://127.0.0.1:8000/admin/

Well, now we’re getting somewhere. Log in with the superuser account that you created in step 3

Go ahead poke around for a few minutes. Check out the users section. Create a new user! Create a group, check out the permissions. This all came from adding one line of code to your INSTALLED_APPS. Pretty cool huh? You smiling yet?

This admin screen isn’t just for Django created stuff like users and groups. This is where your data will eventually live too. So let’s go ahead to the next step and create some models for our blog application.

Download at this step: GitHub