django

Django 2.0 Window expressions tutorial

Django 2.0 was released recently and among the most exciting things for me is support for Window expressions, which allows adding an OVER clause to querysets. We will use Window expressions to analyze the commits data to the Django repo.

So what is an over clause?

An over clause is of this format

SELECT depname, empno, salary,
  avg(salary)
    OVER (PARTITION BY depname)
FROM empsalary;

Compare this to a similar GROUP BY statement

SELECT depname, avg(salary)
FROM empsalary
GROUP BY depname;

The difference is a GROUP BY has as many rows as grouping elements, here number of depname. An over clause adds the the aggregated result to each row of the select.

Postgres documentation says, "A window function performs a calculation across a set of table rows that are somehow related to the current row. This is comparable to the type of calculation that can be done with an aggregate function. But unlike regular aggregate functions, use of a window function does not cause rows to become grouped into a single output row — the rows retain their separate identities. Behind the scenes, the window function is able to access more than just the current row of the query result." This is true for all other DB implementation as well.

What are real world uses of over expressions?

We will use the Django ORM with the Window expression to to some analysis on the most prolific committers to Django. To do this we will export the commiter names and time of commit to a csv.

git log  --no-merges --date=iso --pretty=format:'%h|%an|%aI' > commits.iso.csv

This is not ranking of Django developers, just of their number of commits, which allows us an interestig dataset. I am grateful to everyone who has contributed to Django - they have made my life immesureably better.

With some light data wrangling using Pandas, we transform this to a per author, per year data and import to Postgres. Our table structure looks like this.

experiments=# \d commits_by_year;
   Table "public.commits_by_year"
    Column     |  Type   | Modifiers
---------------+---------+-----------
 id            | bigint  |
 author        | text    |
 commit_year   | integer |
 commits_count | integer |

We define a model to interact with this table.

from django.db import models


class Committer(models.Model):
    author = models.CharField(max_length=100)
    commit_year = models.PositiveIntegerField()
    commits_count = models.PositiveIntegerField()

    class Meta:
        db_table = 'commits_by_year'

Lets quickly test if our data is imported. You can get a csv from here, and import to Postgres to follow along.

In [2]: Committer.objects.all().count()
Out[2]: 2318

Let us setup our environment and get the imports we need.

## Some ORM imports which we are going to need

from django.db.models import Avg, F, Window
from django.db.models.functions import  Rank, DenseRank, CumeDist
from django_commits.models import Committer

# We will use pandas to display the queryset in tanular format
import pandas
pandas.options.display.max_rows=20

# An utility function to display querysets
def as_table(values_queryset):
    return pandas.DataFrame(list(values_queryset))

Lets quickly look at the data we have.

as_table(Committer.objects.all().values(
  "author", "commit_year", "commits_count"
))

	author	commit_year	commits_count
0	Tim Graham	2017	373
1	Sergey Fedoseev	2017	158
2	Mariusz Felisiak	2017	113
3	Claude Paroz	2017	102
4	Mads Jensen	2017	55
5	Simon Charette	2017	40
6	Jon Dufresne	2017	33
7	Anton Samarchyan	2017	27
8	François Freitag	2017	17
9	Srinivas Reddy Thatiparthy	2017	14
...	...	...	...
2308	Malcolm Tredinnick	2006	175
2309	Georg Bauer	2006	90
2310	Russell Keith-Magee	2006	86
2311	Jacob Kaplan-Moss	2006	83
2312	Luke Plant	2006	20
2313	Wilson Miner	2006	12
2314	Adrian Holovaty	2005	1015
2315	Jacob Kaplan-Moss	2005	130
2316	Georg Bauer	2005	112
2317	Wilson Miner	2005	20

2318 rows × 3 columns

We will now use the Window expression to get the contributors ranked by number of commits, within each year. We will go over the code in detail, but lets look at the queryset and results.

# Find out who have been the most prolific contributors
# in the years 2010-2017

dense_rank_by_year = Window(
    expression=DenseRank(),
    partition_by=F("commit_year"),
    order_by=F("commits_count").desc()
)

commiters_with_rank = Committer.objects.filter(
        commit_year__gte=2010, commits_count__gte=10
    ).annotate(
        the_rank=dense_rank_by_year
    ).order_by(
        "-commit_year", "the_rank"
    ).values(
        "author", "commit_year", "commits_count", "the_rank"
    )
as_table(commiters_with_rank)

	author	commit_year	commits_count	the_rank
0	Tim Graham	2017	373	1
1	Sergey Fedoseev	2017	158	2
2	Mariusz Felisiak	2017	113	3
3	Claude Paroz	2017	102	4
4	Mads Jensen	2017	55	5
5	Simon Charette	2017	40	6
6	Jon Dufresne	2017	33	7
7	Anton Samarchyan	2017	27	8
8	François Freitag	2017	17	9
9	Srinivas Reddy Thatiparthy	2017	14	10
...	...	...	...	...
171	Joseph Kocherhans	2010	53	11
172	Ramiro Morales	2010	53	11
173	Jacob Kaplan-Moss	2010	42	12
174	Chris Beaven	2010	29	13
175	Malcolm Tredinnick	2010	26	14
176	Honza Král	2010	20	15
177	Carl Meyer	2010	17	16
178	Ian Kelly	2010	17	16
179	Simon Meers	2010	11	17
180	Gary Wilson Jr	2010	10	18

181 rows × 4 columns

Lets look a the the ORM code in more detail here.

# We are creating the Window function part of our SQL query here
dense_rank_by_year = Window(
    # We want to get the Rank with no gaps
    expression=DenseRank(),
    # We want to partition the queryset on commit_year
    # Each distinct commit_year is a different partition
    partition_by=F("commit_year"),
    # This decides the ordering within each partition
    order_by=F("commits_count").desc()
)


commiters_with_rank = Committer.objects.filter(
        commit_year__gte=2010, commits_count__gte=10
    # Standard filter oprtation, limit rows to 2010-2017
    ).annotate(
    # For each commiter, we are annotating its rank
        the_rank=dense_rank_by_year
    ).order_by(
        "-commit_year", "the_rank"
    ).values(
        "author", "commit_year", "commits_count", "the_rank"
    )
as_table(commiters_with_rank)

Now lets try getting the average commits per commiter for each year along with the other data.

avg_commits_per_year = Window(
    # We want the average of commits per committer, with each partition
    expression=Avg("commits_count"),
    # Each individual year is a partition.
    partition_by=F("commit_year")
)

commiters_with_yearly_average = Committer.objects.filter().annotate(
      avg_commit_per_year=avg_commits_per_year
    ).values(
        "author", "commit_year", "commits_count", "avg_commit_per_year"
    )
# We could have done further operation with avg_commit_per_year
# Eg: F(commits_count) - F(avg_commit_per_year),
# would tell us committers who commit more than average
as_table(commiters_with_yearly_average)

This gives us

	author	avg_commit_per_year	commit_year	commits_count
0	Wilson Miner	319.250000	2005	20
1	Adrian Holovaty	319.250000	2005	1015
2	Jacob Kaplan-Moss	319.250000	2005	130
3	Georg Bauer	319.250000	2005	112
4	Russell Keith-Magee	188.571429	2006	86
5	Jacob Kaplan-Moss	188.571429	2006	83
6	Luke Plant	188.571429	2006	20
7	Wilson Miner	188.571429	2006	12
8	Adrian Holovaty	188.571429	2006	854
9	Malcolm Tredinnick	188.571429	2006	175
...	...	...	...	...
2308	Adam Johnson	4.916084	2017	13
2309	Tom	4.916084	2017	13
2310	Srinivas Reddy Thatiparthy	4.916084	2017	14
2311	François Freitag	4.916084	2017	17
2312	Anton Samarchyan	4.916084	2017	27
2313	Jon Dufresne	4.916084	2017	33
2314	Simon Charette	4.916084	2017	40
2315	Mads Jensen	4.916084	2017	55
2316	Claude Paroz	4.916084	2017	102
2317	Mariusz Felisiak	4.916084	2017	113

2318 rows × 4 columns

You could try other Window functions such as CumeDist, Rank or Ntile.

from django.db.models.functions import CumeDist
cumedist_by_year = Window(
    expression=CumeDist(),
    partition_by=F("commit_year"),
    order_by=F("commits_count").desc()
)

commiters_with_rank = Committer.objects.filter(
        commit_year__gte=2010, commits_count__gte=10
    ).annotate(
        cumedist_by_year=cumedist_by_year
    ).order_by(
        "-commit_year", "the_rank"
    ).values(
        "author", "commit_year", "commits_count", "cumedist_by_year"
    )
as_table(commiters_with_rank)

Until now, we have partitioned on commit_year. We can partition on other fields too. We will partition on author to find out how their contributions have changed over the years using the Lag window expression.

from django.db.models.functions import Lag
from django.db.models import Value
commits_in_previous_year = Window(
    expression=Lag("commits_count", default=Value(0)),
    partition_by=F("author"),
    order_by=F("commit_year").asc(),
)

commiters_with_pervious_year_commit = Committer.objects.filter(
        commit_year__gte=2010, commits_count__gte=10
    ).annotate(
        commits_in_previous_year=commits_in_previous_year
    ).order_by(
        "author", "-commit_year"
    ).values(
        "author", "commit_year", "commits_count", "commits_in_previous_year"
    )
as_table(commiters_with_pervious_year_commit)

	author	commit_year	commits_count	commits_in_previous_year
0	Adam Chainz	2016	42	12
1	Adam Chainz	2015	12	0
2	Adam Johnson	2017	13	0
3	Adrian Holovaty	2012	40	98
4	Adrian Holovaty	2011	98	72
5	Adrian Holovaty	2010	72	0
6	Akshesh	2016	31	0
7	Alasdair Nicol	2016	13	19
8	Alasdair Nicol	2015	19	17
9	Alasdair Nicol	2013	17	0
...	...	...	...	...
171	Timo Graham	2012	13	70
172	Timo Graham	2011	70	60
173	Timo Graham	2010	60	0
174	Tom	2017	13	0
175	Unai Zalakain	2013	17	0
176	Vajrasky Kok	2013	14	0
177	areski	2014	15	0
178	eltronix	2016	10	0
179	wrwrwr	2014	21	0
180	Łukasz Langa	2013	15	0

181 rows × 4 columns

I hope this tutorial has been helpful in understanding the window expressions. While still not as felxible as SqlAlchemy, Django ORM has become extremely powerful with recent Django releases. Stay tuned for more advanced ORM tutorials.

Configure Django to log exceptions in production

Django default logging behaviour for unhandled exceptions is:

With DEBUG=True (Development)

Log the exception on console/stream.
Show the exception on page, i.e in http response.

With DEBUG=False (Production)

Do not log the exception on console/stream.
Do not show the exception in response.
Send an email to admin if email settings are configured correctly.

Usually not logging the exception on console isn't a problem since an exception email is sent to you which can help you know the source of exception. But this assumes that email settings are configured correctly else you will not receive the exception email.

You might not have email settings configured correctly and don't want to get into that right away. You might instead want to log the exception on console even with DEBUG=False. This post would help you in such scenario.

Default logging configuration

Django's default logging setting is:

DEFAULT_LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse',
        },
        'require_debug_true': {
            '()': 'django.utils.log.RequireDebugTrue',
        },
    },
    'formatters': {
        'django.server': {
            '()': 'django.utils.log.ServerFormatter',
            'format': '[%(server_time)s] %(message)s',
        }
    },
    'handlers': {
        'console': {
            'level': 'INFO',
            'filters': ['require_debug_true'],
            'class': 'logging.StreamHandler',
        },
        'django.server': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'django.server',
        },
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django': {
            'handlers': ['console', 'mail_admins'],
            'level': 'INFO',
        },
        'django.server': {
            'handlers': ['django.server'],
            'level': 'INFO',
            'propagate': False,
        },
    }
}

Without any explicit settings.LOGGING configured in settings.py, this is the default logging configuration Django works with. You can ignore django.server part.

Any unhandled Django exception is handled in function handle_uncaught_exception. The relevant code is on Github

Error is logged using logger.error in this function. This logger is an instance of django.request. Since logger django is a parent of django.request, so log records are propogated to logger django.

As you can see from DEFAULT_LOGGING, logger django has two handlers.

console
mail_admins

As you can see from DEFAULT_LOGGING, handler console has a filter called require_debug_true because of which this handler doesn't handle log records in production, i.e when DEBUG=False.

Logging to console in production

So you can create a new handler which directs ERROR log records to Stream when DEBUG=False.

This handler would look like

'console_debug_false': {
    'level': 'ERROR',
    'filters': ['require_debug_false'],
    'class': 'logging.StreamHandler',
}

And you can ask logger django to use this handler by adding this handler in loggers['django']['handlers'].

Your final settings.LOGGING would look like following:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse',
        },
        'require_debug_true': {
            '()': 'django.utils.log.RequireDebugTrue',
        },
    },
    'formatters': {
        'django.server': {
            '()': 'django.utils.log.ServerFormatter',
            'format': '[%(server_time)s] %(message)s',
        }
    },
    'handlers': {
        'console': {
            'level': 'INFO',
            'filters': ['require_debug_true'],
            'class': 'logging.StreamHandler',
        },
        'console_debug_false': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'logging.StreamHandler',
        },
        'django.server': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'django.server',
        },
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django': {
            'handlers': ['console', 'console_debug_false', 'mail_admins'],
            'level': 'INFO',
        },
        'django.server': {
            'handlers': ['django.server'],
            'level': 'INFO',
            'propagate': False,
        }
    }
}

If you don't want emails to be sent to admins, in case email settings aren't configured correctly, then you should remove mail_admins from loggers['django']['handlers'].

Template fragment caching gotchas

Variables in cached template fragment

Assuming this is in template.

{% cache 300 nums %}
{% for i in nums %}
    <p>i</p>
{% endfor %}
{% endcache %}

And assuming we send {'nums': range(100)} from context, then 0 to 99 will be sent in the response.

Now suppose we change context to {'nums': range(1000)}, still for next 5 minutes i.e until the cache expires, 0 to 99 will be sent in the response. 0 to 999 will not be sent in the response.

To fix this, we should use the variable too with the {% cache %} tag. So correct code would be

{% cache 300 nums_cache nums %}
{% for i in nums %}
    <p>i</p>
{% endfor %}
{% endcache %}

After this whenever context nums changes, cache would be reevaluated.

Boolean variable in cached template fragment

Assuming template contains

{% cache 300 hello %}
{% if hello %}
  <p>Hello</p>
{% endif %}
{% endcache %}

and assuming {'hello': True} is sent in context. Then Hello will be sent in response.

Now even when we send {'hello': False} in context, "Hello" will still be sent in response because it's already cached.

To fix this.

{% cache 300 hello_cache hello %}
{% if hello %}
  <p>Hello</p>
{% endif %}
{% endcache %}

request.user in cached template fragment

{% cache 300 username_cache %}
<p>{{request.user.username}}</p>
{% endcache %}

When current user logs out and a new user logs in, still the username of old user is shown on the web page because template fragment is already cached and would not be reevaluated.

Fix it like

{% cache 300 username_cache request.user.username %}
<p>{{request.user.username}}</p>
{% endcache %}

Gotcha while using base template

Assuming we are using template inheritance, and our base template i.e base.html looks like:

{% load cache %}
<html>
    <head>
    </head>
    <body>
    {% cache 300 base_body %}
    {% block body %}
    {% endblock %}
    {% endcache %}
    </body>
</html>

And the child template, say test.html looks like

{% extends "base.html" %}
{% load cache %}
{% block body %}
  {% cache 300 username request.user.username %}
    <p>{{request.user.username}}</p>
  {% endcache %}
{% endblock %}

Assuming our view uses test.html. Suppose cache is empty now, and you are logged in as user1 and then you make the request to a url which uses test.html. So response will contain "user1".

Now if you logout and login as user2 and make the request, still the response will be "user1" instead of "user2".

The reason for this is, as per Django docs.

The extends tag is the key here. It tells the template engine that this template “extends” another template. When the template system evaluates this template, first it locates the parent – in this case, “base.html”.

But in our case, the parent template itself has some cached content, which Django will use. Django will not even bother to look at the {% block body %} of child template if it finds cached content base_body for the parent template.

The fix for this is to differentiate the cache of different users even in the base template. So we could change the base template to look like.

{% load cache %}
<html>
    <head>
    </head>
    <body>
    {% cache 300 base_body request.user.username %}
    {% block body %}
    {% endblock %}
    {% endcache %}
    </body>
</html>

After this, if user1 is logged in the "user1" is sent in response. If user2 is logged in then "user2" is sent in response.

Gotcha with {% include %} tag.

It is similar to the gotcha of template inheritance discussed in last section.

Assuming the view uses test.html, which contains.

{% cache 60 body_cache %}
{% include "body.html" %}
{% endcache %}

And body.html contains

<p>Hello {{request.user.username}}</p>

So when first user, say user1 logs in, cache is evaluated and Hello user1 is sent in response. When user2 logs in, still Hello user2 will be sent in response.

To fix this, test.html should also use all the variables on which included template depends.

So code in test.html should change to

{% cache 60 body_cache request.user %}
{% include "body.html" %}
{% endcache %}

If included template, i.e body.html depended on any other variable, that other variable too should be used with {% cache %} tag.

Profiling Django Middlewares

I assume you have a basic understanding of Profiling, what it means and why we use it.

Why this post

Recently I had to profile a Django application which wasn't performing as fast as it should. This application had several custom middlewares too. So it was possible that custom middlewares were the cause of slow performance.

There are some existing Django libraries to profile Django code. eg: Django Debug Toolbar, django-silk , django_cprofile etc. Most of them can profile view code well but they can't profile other middlewares.

I wanted a way to profile middlewares too.

Problem with Django Debug Toolbar

I assume you understand middlewares and how the order in which middlewares are defined matter. If you want to get more idea about middlewares, this post might help.

Django debug toolbar is probably designed for profiling the views. It uses process_view() and returns an HttpResponse instace from process_view(). process_request() of all middlewares run before any middleware's process_view(). So using Django debug toolbar, it's not possible to profile what's going on inside process_request() of different middlewares.

And since process_view() of debug toolbar returns HttpResponse, process_view() of other middlewares is bypassed and so we can't profile process_view() of other middlewares.

So I guess it is not possible to profile middleware code using Django debug toolbar.

django-silk

Django silk seemed better at profiling middlewares too. It looks promising and I will play more with it.

But Django silk also tracks queries executed, inserts the results in db etc. In case you only wanted to know the time it takes to execute different functions and wanted to find out the most time consuming functions, you might not want the overhead of django silk.

Writing our own middleware

We want to write a simple middleware that just tells the most expensive functions/methods and time it took to execute those functions. We don't want to capture sql queries or anything fancy.

We will use standard Python provided cProfile to achieve our goal. This official doc can help you get familiar with cProfile in 10 mins.

Add the following in any app's middleware.py. Supposing you have an app called books and you add this in books/middleware.py

import cProfile, pstats, StringIO

class ProfilerMiddleware(object):
    def process_request(self, request):
        pr = cProfile.Profile()
        pr.enable()
        request._pr = pr

    def process_response(self, request, response):
        request._pr.disable()
        s = StringIO.StringIO()
        sortby = 'cumulative'
        # Sort the output by cumulative time it took in fuctions/methods.
        ps = pstats.Stats(request._pr, stream=s).sort_stats(sortby)
        # Print only 10 most time consuming functions
        ps.print_stats(10)
        print s.getvalue()
        return response

And add books.middleware.ProfileMiddleware at top of your MIDDLEWARE_CLASSES.

MIDDLEWARE_CLASSES = (
    'books.middleware.ProfilerMiddleware',
    # Assuming you have some custom middlewares here, even they will be profiled
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    # This middleware will be profiled too.
    # 'books.middleware.SomeCustomMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

Try any url and you should see the profiler output on the runserver console.

Explanation

We put our middleware at top of MIDDLEWARE_CLASSES.
So this middleware's process_request() will be executed before any other middleware's process_request(). Also it will be executed before any other middleware's any other function like process_view() etc.
We enable profiling in process_request() so everything hereafter will be profiled. So process_request() and process_view() of any other middleware will be profiled.
We disable profiling in process_response() of our middleware. process_response() of this middleware will run at last, i.e after process_response() of all other middlewares have run.
This way process_response() of all other middlewares get profiled too.

Understanding Django Middlewares

I assume you have read official Django docs on middleware. I will elaborate on things mentioned in the documentation but I assume you are familiar with basics of middleware.

In this post we will discuss the following.

What is a middleware
When to use middleware
Things to remember when writing middleware
Writing some middlewares to understand how order of middleware matters

What is a middleware

Middlewares are hooks to modify Django request or response object. Putting the definition of middleware from Django docs.

Middleware is a framework of hooks into Django’s request/response processing. It’s a light, low-level “plugin” system for globally altering Django’s input or output.

When to use middleware

You can use middleware if you want to modify the request i.e HttpRequest object which is sent to the view. Or you might want to modify the HttpResponse object returned from the view. Both these can be achieved by using a middleware.

You might want to perform an operation before the view executes. In such case you would use a middleware.

Django provides some default middleware. eg: AuthenticationMiddleware

Very often you would have used request.user inside the view. Django wants user attribute to be set on request before any view executes. Django takes a middleware approach to accomplish this. So Django provides an AuthenticationMiddleware which can modify the request object.

And Django modifies the request object like:

https://github.com/django/django/blob/master/django/contrib/auth/middleware.py#L22

Similarly you might have an application which works with users of different timezones. You want to use the user's timezone while showing any page to the user. You want access to user's timezone in all the views. It makes sense to add it in session in such case. So you can add a middleware like this:

class TimezoneMiddleware(object):
    def process_request(self, request):
        # Assuming user has a OneToOneField to a model called Profile
        # And Profile stores the timezone of the User.
        request.session['timezone'] = request.user.profile.timezone

TimezoneMiddleware is dependent on request.user. And request.user is populated in AuthenticationMiddleware. So TimezoneMiddleware written by us must come after Django provided AuthenticationMiddleware in the tuple settings.MIDDLEWARE_CLASSES.

We will get more idea about order of middlewares in coming examples.

Things to remember when using middleware

Order of middlewares is important.
A middleware only need to extend from class object.
A middleware is free to implement some of the methods and not implement other methods.
A middleware may implement process_request but may not implement process_response and process_view. Infact it is very common and lot of Django provided middlewares do it.
A middleware may implement process_response but not implement process_request.

AuthenticationMiddleware only implements process_request and doesn't implement process_response. You can check it here

GZipMiddleware only implements process_response and doesn't implement process_request or process_view. You can see it here

Writing some middlewares

Make sure you have a Django project with a url and a view, and that you are able to access that view. Since we will try several things with request.user, make sure authentication is properly set for you and that request.user prints the right thing in this view.

Create a file middleware.py in any of your app.

I have an app called books and so I am writing this in books/middleware.py

class BookMiddleware(object):
    def process_request(self, request):
        print "Middleware executed"

Add this middleware in MIDDLEWARE_CLASSES

MIDDLEWARE_CLASSES = (
    'books.middleware.BookMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

Make request to any url. This should get printed on runserver console

Middleware executed

Modify BookMiddleware.process_request so it looks like

class BookMiddleware(object):
    def process_request(self, request):
        print "Middleware executed"
        print request.user

Make request to a url again. This will raise an error.

'WSGIRequest' object has no attribute 'user'

This happened because attribute user hasn't been set on request yet.

Now change the order of middlewares so BookMiddleware comes after AuthenticationMiddleware

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'books.middleware.BookMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

Make request to any url. This should get printed on runserver console

Middleware executed
<username>

This tells that process_request is executed on the middlewares in the order in which they are listed in settings.MIDDLEWARE_CLASSES

You can verify it further. Add another middleware in your middleware.py

class AnotherMiddleware(object):
    def process_request(self, request):
        print "Another middleware executed"

Add this middleware in MIDDLEWARE_CLASSES too.

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'books.middleware.BookMiddleware',
    'books.middleware.AnotherMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

Now output would be:

Middleware executed
<username>
Another middleware executed

How returning HttpResponse from process_request changes things

Modify BookMiddleware so it looks like

class BookMiddleware(object):
    def process_request(self, request):
        print "Middleware executed"
        print request.user
        return HttpResponse("some response")

Try any url now and your output would be:

Middleware executed
<username>

You will notice two things:

Your view will no more be executed and no matter which url you try, you will see "some response".
AnotherMiddleware.process_request will not be executed anymore.

So if a Middleware's process_request() returns a HttpResponse object then process_request of any subsequent middlewares is bypassed. Also view execution is bypassed. You would rarely do this or require this in your projects.

Comment "return HttpResponse("some response")" so process_request of both middlewares keep executing.

Working with process_response

Add method process_response to both the middlewares

class AnotherMiddleware(object):
    def process_request(self, request):
        print "Another middleware executed"

    def process_response(self, request, response):
        print "AnotherMiddleware process_response executed"
        return response

class BookMiddleware(object):
    def process_request(self, request):
        print "Middleware executed"
        print request.user
        return HttpResponse("some response")
        #self._start = time.time()

    def process_response(self, request, response):
        print "BookMiddleware process_response executed"
        return response

Try some url. Output would be

Middleware executed
<username>
Another middleware executed
AnotherMiddleware process_response executed
BookMiddleware process_response executed

AnotherMiddleware.process_response() is executed before BookMiddleware.process_response() while AnotherMiddleware.process_request() executes after BookMiddleware.process_request(). So process_response() follows the reverse of what happens with process_request. process_response() is executed for last middleware then second last middleware and so on till the first middleware.

process_view

Django applies middleware's process_view() in the order it’s defined in MIDDLEWARE_CLASSES, top-down. This is similar to the order followed for process_request().

Also if any process_view() returns an HttpResponse object, then subsequent process_view() calls are bypassed and not executed.

Check our next post to see a practical use of middleware.

Tastypie with ForeignKey

Tastypie with ForeignKeys

This is a followup post on Getting started with tastypie. We will use the same project setup as used in the last post.

This post will cover:

Fetch ForeignKey data in GET calls
Create an object with ForeignKeys using POST calls

Setup the application

Let's add the capability to categorise the expenses

Add a model called ExpenseCategory

class ExpenseCategory(models.Model):
    name = models.CharField(max_length=100)
    description = models.TextField()

Add a FK from Expense to ExpenseCategory

class Expense(models.Model):
    description = models.CharField(max_length=100)
    amount = models.IntegerField()
    user = models.ForeignKey(User, null=True)
    category = models.ForeignKey(ExpenseCategory, null=True)

There already exists some Expense in db without an associated category, so make ExpenseCategory as nullable.

Create and apply migrations

python manage.py makemigrations
python manage.py migrate

Let's create an expensecategory from shell and associate it with an expense of user Sheryl.

u = User.objects.get(username='sheryl')
ec = ExpenseCategory.objects.create(name='Misc', description='Miscellaneous expenses')
e = Expense.objects.create(description='Went to Stockholm', amount='5000', user=u, category=ec)

Get FK fields in response too.

We want category in Expense GET endpoint too.

Our first approach would be adding 'category' to ExpenseCategory.Meta.fields. Try it

fields = ['description', 'amount', 'category']

Try the expense GET endpoint for Sheryl

http://localhost:8000/api/expense/?username=sheryl&api_key=1a23&format=json

Still don't see category in response. We need something more than this.

Adding fields.ForeignKey on ExpenseResource

There is no easy way to achieve this without adding a resource for ExpenseCategory.

We need to create an ExpenseCategoryResource similar to ExpenseResource

Add ExpenseCategoryResource to expenses/api.py

class ExpenseCategoryResource(ModelResource):
    class Meta:
        queryset = ExpenseCategory.objects.all()
        resource_name = 'expensecategory'

Add proper url pattern for ExpenseCategoryResource in expenses/api.py

expense_category_resource = ExpenseCategoryResource()

urlpatterns = patterns('',
    url(r'^admin/', include(admin.site.urls)),
    url(r'^api/', include(expense_resource.urls)),
    url(r'^api/', include(expense_category_resource.urls)),
)

Verify things are properly setup for ExpenseCategoryResource by accessing

http://localhost:8000/api/expensecategory/?format=json

Add the following to ExpenseCategory

category = fields.ForeignKey(ExpenseCategoryResource, attribute='category', null=True)

Try

http://localhost:8000/api/expense/?username=sheryl&api_key=1a23&format=json

After this you'll be able to see category in response

This will return resource_uri of ExpenseCategory by default

Using full=True

Probably you want to see the name and description of category in the response

Make the following modification

category = fields.ForeignKey(ExpenseCategoryResource, attribute='category', null=True, full=True)

Try the GET endpoint again

http://localhost:8000/api/expense/?username=sheryl&api_key=1a23&format=json

POST data with FK

There are several ways in which we can set category on expense while making POST call to create expenses.

Post with resource_uri of FK

We already have one ExpenseCategory in the db and the resource_uri for that expensecategory is '/api/expensecategory/1/'

We want to create an expense and set the category as our earlier created expensecategory.

post_data = {'description': 'Bought a phone for testing', 'amount': 2200, 'category': '/api/expensecategory/1/'}
post_url = 'http://localhost:8000/api/expense/?username=sheryl&api_key=1a23'
r = requests.post(post_url, data=json.dumps(post_data), headers=headers)

Posting entire data of FK

You find that the expense you want to create doesn't fit in any of the existing categories. You want to create a new expensecategory while making POST data to expense endpoint.

So we want to creating ExpenseCategory and Expense together.

You need to post the following data in such case.

post_data = {'description': 'Went to paris to attend a conference', 'amount': 9000, 'category': {'name': 'Travel', 'description': 'Expenses incurred on travelling'}}

No category exists for Travel yet.

Check the count of ExpenseCategory currently so that later you can verify that a new ExpenseCategory is created.

ExpenseCategory.objects.count()
1   #output

POST the data to expense endpoint

r = requests.post(post_url, data=json.dumps(post_data), headers=headers)
print r.status_code    #401

Why you got 401

Even though you tried creating an Expense on expense post endpoint, tastypie internally tries creating an expensecategory because of structure of post_data. But tastypie finds that ExpenseCategoryResource doesn't have authorization to allow POST yet.

So we need to add proper authorization to ExpenseCategory before this post call can succeed.

Add the following to ExpenseCategoryResource.Meta

authorization = Authorization()

POSTing again

Try the post call again.

r = requests.post(post_url, data=json.dumps(post_data), headers=headers)

This would have worked well and a new ExpenseCategory should have been created.

ExpenseCategory.objects.count()
2    #output

Also the new expense would have got associated with the newly created ExpenseCategory.

POST with id of FK

post_data = {'description': 'Bought second Disworld book', 'amount': 399, 'category': {'pk': 1}}

r = requests.post(post_url, json.dumps(post_data), headers=headers)

Optimizing FK field calculation

If you have full=True on FK resource then a database call will happen for each FK of each row.

eg:

category = fields.ForeignKey(ExpenseCategoryResource, attribute='category', null=True, full=True)

Here if you are hitting expense GET list, and suppose you are getting 20 expenses.

For each expense, it's category needs to be fetched from db and the categorie's full representation needs to be calculated. So 20 extra db calls will happen.

Fixing it. Set ExpenseResource.Meta.queryset to

queryset = Expense.objects.all().select_related("category")

Now the extra 20 calls will not be made.

Getting category_id without ExpenseCategoryResource

Suppose you are only interested in getting category_id in GET calls and don't care about name and description of category.

Comment out everything about ExpenseCategoryResource and relation to ExpenseCategoryResource from ExpenseResource

And add the following line to ExpenseResource

category_id = fields.IntegerField(attribute='category_id', null=True)

After this you will find category_id in GET responses.

Handling POST

Similarly you want to be able to POST without worrying about ExpenseCategoryResource.

You want to send the category_id in post data and using that id, expense should be associated with correct expensecategory.

data = {'description': 'Bought iphone', 'amount': 6500, 'category_id': 1}
r = requests.post("http://localhost:8000/api/expense/?username=sheryl&api_key=1a23", data=json.dumps(data), headers=headers)

Gotchas

We have a FK to ExpenseCategory from Expense and it is nullable. Similarly, we have an en FK from ExpenseResource to ExpenseCategoryResource which is nullable.

Mark ExpenseCategoryResource non-null for now, i.e remove null=True from it.

So it looks like

category = fields.ForeignKey(ExpenseCategoryResource, attribute='category', full=True)

Check the count of expenses in the db.

In [20]: Expense.objects.count()
Out[20]: 23

Make a POST request to expense endpoint but don't pass any associated expensecategory.

post_url = 'http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23'
post_data = {'description': 'Bought second Disworld book', 'amount': 399}
r = requests.post(post_url, json.dumps(post_data), headers=headers)
print r.status_code
400 #output

Status code 400 means that request wasn't successul and so expense shouldn't have been created.

But actually it was created even though tastypie returned us a 400 status code. You can check it using count.

In [22]: Expense.objects.count()
Out[22]: 24

This happened because at the model/db level ExpenseCategory allows null. So save() on Expense was successful. But at api/tastypie level, ExpenseCategoryResource is not nullable, so tastypie raised an error.

Getting started with Django tastypie

Django tastypie is a library to write RESTful apis in Django.

Why use REST

You have a database backed web application. This application tracks expenses. The application allows the capability to enter your expenses, view all your expenses, delete an expense etc. Essentially this application provides CRUD functionality. Django application has access to database credentials, but they are never seen by the users of the web application. Django application decides what to show to which user. Django application ensures that a particular user only sees the expenses entered by him and not somebody else's expenses.

Now you want to provide a mobile application (Android or iOS) corresponding to this web application. Android application should allow the user to view his expenses, create an expense as well as any other CRUD functionality. But database credentials could not be put in Android code as it is not too hard to decompile an apk and get the db credentials. And we never want a user to get the db credentials else he will be in a position to see everyone's expenses and the entire database. So there has to be another way to allow mobile applications to get things from the database. This is where REST comes into picture.

With REST, we have three components. A database, a Django application and a mobile application. Mobile application never accesses the database directly. It makes a REST api call to Django application. Mobile application also sends a api_key specific to the mobile user. Based on api_key, Django application determines what data to make visible to this particular api_key owner and sends the corresponding data in response.

Resource

REST stands for Representational State Transfer. It is a standard for transferring the state of a Resource, from web to mobile.

What do I mean by state of a Resource?

An expense could be a resource. A Person could be a resource. A blog post could be a resource. Basically any object or instance your program deals with could be a resource. And a resource's state is maintained in it's attributes. eg: You could have a model called Expense. The state of a expense instance is represented by its attributes.

Any REST library should be able to create and return a representation of such resource, which simply stated means that REST library should be able to tell us the attributes and their values for different model instances. And tastypie is adept at doing this.

Setting up the application

I am using Django 1.7. Some things might be different for you if you are using different version of Django.

As with all projects, I want to keep things in a virtual environment

$ mkvirtualenv tastier
$ workon tastier

Install Django

$ pip install Django

Start a Django project

(tastier) $ django-admin.py startproject tastier

(tastier) $ cd tastier/

Start an app

(tastier) $ python manage.py startapp expenses

Add this app to INSTALLED_APPS

Run migration

(tastier)~ $ python manage.py migrate

Runserver

(tastier)~ $ python manage.py runserver

Check that your are able to access http://localhost:8000/admin/login/

I have pushed the code for this project to Github. You will be able to checkout at different commits in the project to see specific things.

Getting started

Install django-tastypie.

(tastier) $ pip install django-tastypie

Create a file called expenses/api.py where you will keep all the tastypie related things.

Suppose your program deals with a resource called Expense. Let's create a model Expense in expenses/models.py

class Expense(models.Model):
    description = models.CharField(max_length=100)
    amount = models.IntegerField()

Run migrations

python manage.py makemigrations
python manage.py migrate

We will later add a ForeignKey(User) to Expense to associate an expense with User. Don't worry about it for now, we will come back to it.

Let's add few Expense instances in the database.

Expense.objects.create(description='Ate pizza', amount=100)
Expense.objects.create(description='Went to Cinema', amount=200)

Handling GET

You want the ability to get the representation of all expenses in your program at url "http://localhost:8000/api/expenses/".

To deal with a resource, tastypie requires a class which overrides ModelResource. Let's call our class ExpenseResource. Add following to expenses/api.py

from tastypie.resources import ModelResource

from .models import Expense

class ExpenseResource(ModelResource):

    class Meta:
        queryset = Expense.objects.all()
        resource_name = 'expense'

And you need to add the following to tastier/urls.py

from expenses.api import ExpenseResource

expense_resource = ExpenseResource()

urlpatterns = patterns('',
    url(r'^admin/', include(admin.site.urls)),
    url(r'^api/', include(expense_resource.urls)),
)

GET all expenses

After this you should be able to hit

http://localhost:8000/api/expense/?format=json

and you will see all the expenses from database in the response.

The response would be:

{"meta": {"limit": 20, "next": null, "offset": 0, "previous": null, "total_count": 2}, "objects": [{"amount": 100, "description": "Ate pizza", "id": 1, "resource_uri": "/api/expense/1/"}, {"amount": 200, "description": "Went to Cinema", "id": 2, "resource_uri": "/api/expense/2/"}]}

You will find the representation of expense instances in objects key of response.

Get a particular expense

You can get the representation of expense with id 1 at

http://localhost:8000/api/expense/1/?format=json

See how you are able to hit these two urls without adding them in urlpatterns. These urlpatterns are added by tastypie internally.

How these endpoints help and ties with mobile application example?

If the mobile app wants to show all the expenses it could use the url http://localhost:8000/api/expense/?format=json, get the response, parse the response and show the result on app.

Right now every user will see all the expenses. As we move forward we will see how only a user's expenses will be returned when a REST call is made from his/her mobile device.

Serialization

You must have realized that REST returns you serialized data. You might be wondering why use django-tastypie to achieve it, and not just use json.dumps. You can undoubtedly use json.dumps and not use django-tastypie to provide REST endpoints. But django-tastypie allows the ability to do many more things very easily as you will soon agree. Just hang on.

Changing Meta.resource_name

You can change ExpenseResource.Meta.resource_name from expense to expenditure.

class ExpenseResource(ModelResource):

    class Meta:
        queryset = Expense.objects.all()
        resource_name = 'expenditure'

And then the old urls will stop working. Your new GET urls in that case will be

http://localhost:8000/api/expenditure/?format=json
http://localhost:8000/api/expenditure/1/?format=json

Changing the resource_name changes the urls tastypie makes available to you.

Now change the resource_name back to expense.

We have our first commit at this point. You can checkout to this commit to see the code till this point.

git checkout b6a9c6

Meta.fields

Suppose you only want description in expense representation, but don't want amount. So you can add a fields attribute on ExpenseResource.Meta

class Meta:
    queryset = Expense.objects.all()
    resource_name = 'expenditure'
    fields = ['description']

Try

http://localhost:8000/api/expense/?format=json

So if you don't have fields attribute on Meta, all the attributes of Model will be sent in response. If you have fields, only attributes listed in fields will be sent in response.

Let's add amount also to fields. Though this gives us the same behaviour as not having ExpenseResource.Meta.fields at all.

class Meta:
    queryset = Expense.objects.all()
    resource_name = 'expenditure'
    fields = ['description', 'amount']

We have our second commit at this point. You can checkout till this point by doing:

git checkout 61194c

Filtering

Suppose you only want the Expenses where amount exceeds 150.

If we had to do this with Django model we would say:

Expense.objects.filter(amount__gt=150)

amount__gt is the key thing here. This could be appended to our url pattern to get the expenses where amount exceeds 150.

This could be achieved at url

http://localhost:8000/api/expense/?amount__gt=150&format=json

Try this. You will get an error because we haven't asked tastypie to allow filtering yet.

Add filtering attribute to ExpenseResource.Meta

class Meta:
    queryset = Expense.objects.all()
    resource_name = 'expense'
    fields = ['description', 'amount']
    filtering = {
        'amount': ['gt']
    }

You should be able to use

http://localhost:8000/api/expense/?amount__gt=150&format=json

This will only return the expenses where amount exceeds 150.

Now we want to get all the expenses on Pizza. We could get pizza expenses in following way from shell.

Expense.objects.filter(description__icontains='pizza')

So to achieve this thing in api, we need to make following changes to ExpenseResource.Meta.filtering:

class Meta:
    queryset = Expense.objects.all()
    resource_name = 'expense'
    fields = ['description', 'amount']
    filtering = {
        'amount': ['gt'],
        'description': ['icontains']
    }

And then following url would give us the pizza expenses

http://localhost:8000/api/expense/?description__icontains=pizza&format=json

With GET endpoints we were able to do the Read operations. With POST we will be able to do Create operations, as we will see in next section.

Handling POST

It's hard to do POST from the browser. So we will use requests library to achieve this.

Check expense count before doing POST.

>>> Expense.objects.count()
2

Tastypie by default doesn't authorize a person to do POST request. The default authorization class is ReadOnlyAuthorization which allows GET calls but doesn't allow POST calls. So you will have to disallow authorization checks for the time being. Add the following to ExpenseResource.Meta

authorization = Authorization()

You'll need to import Authorization class for it.

from tastypie.authorization import Authorization

After this, ExpenseResource would look like:

class ExpenseResource(ModelResource):

    class Meta:
        queryset = Expense.objects.all()
        resource_name = 'expense'
        fields = ['description', 'amount']
        filtering = {
            'amount': ['gt'],
            'description': ['icontains']
        }
        authorization = Authorization()

Don't get into detail of Authorization for now, I will come back to it.

Let's make a POST request to our rest endpoint which will create an Expense object in the database.

post_url = 'http://localhost:8000/api/expense/'
post_data = {'description': 'Bought first Disworld book', 'amount': 399}
headers = {'Content-type': 'application/json'}
import requests
import json
r = requests.post(post_url, json.dumps(post_data), headers=headers)
>>> print r.status_code
201

status_code 201 means that your Expense object was properly created. You can also verify it by checking that Expense count increased by 1.

>>> Expense.objects.count()
3

If you hit the GET endpoint from your browser, you will see this new Expense object too in the response. Try

http://localhost:8000/api/expense/?format=json

We have third commit at this point.

git checkout 749cf3

Explanation of POST

You need to POST at the same url where you get all the expenses. Compare the two urls.
One way of posting is to POST json encoded data. So we used json.dumps
If you are sending json encoded data, you need to send appopriate Content-type header too.

How this ties in with mobile

Android or iOS has a way to make POST request at a given url with headers. So you tell mobile app about the endpoint where they need to post and the data to post. They will call this rest endpoint, and the posted data will be handled by Django tastypie and proper row will be created in the database table.

Adding authentication

Currently GET and POST endpoints respond to every request. So even users who aren't registered with the site will be able to see the expenses. Our first step is ensuring that only registered users are able to use the GET endpoints.

Api tokens and sessions

In a web application, a user logs in once and then she is able to make any number of web requests without being asked to login every time. eg: User logs in once and then can see her expense list page. After first request she can refresh the page, and can still get response without being asked for her login credentials again. This works because Django uses sessions and cookies to store user state. So browser sends a cookie to Django everytime the user makes a request, and Django app can associate the cookie with a user and shows the data for this particular user.

With mobile apps, there is no concept of sessions, unless the mobile is working with a WebView. The session corresponding thing in a mobile app is Api key. So an api key is associated with a user. Every REST call should include this api key, and then tastypie can use this key to verify whether a logged in user is making the request.

Creating user and api token

Let's create an user in our system and a corresponding api token for her.

On a shell

u = User.objects.create_user(username='sheryl', password='abc', email='sheryl@abc.com')

Tastypie provides a model called ApiKey which allows storing tokens for users. Let's create a token for Sheryl.

from tastypie.models import ApiKey
ApiKey.objects.create(key='1a23', user=u)

We are setting the api token for sheryl as '1a23'

You need to ensure tastypie is in INSTALLED_APPS and you have migrated before you could create ApiKey instance.

The default authentication class provided by tastypie is Authentication which allows anyone to make GET requests. We need to set ExpenseResource.Meta.authentication to ensure that only users who provide valid api key are able to get response from GET endpoints.

Add the following on ExpensesResource.Meta.

authentication = ApiKeyAuthentication()

You need to import ApiKeyAuthentication.

from tastypie.authentication import ApiKeyAuthentication

Try the GET endpoint to get the list of expenses

http://localhost:8000/api/expense/?format=json

You will not see anything in response. If you see your runserver terminal, you'll notice that status code 401 is raised.

Api key should be sent in the request to get proper response.

Try the following url

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

With this Sheryl will be able to get proper api response.

Try sending wrong api_key for sheryl and you will not see proper response.

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a2

With these we ensure that only registered users of the system with proper api key will be able to make GET requests.

Fourth commit at this point

git checkout 48725f

How this ties in with mobile app

When user installs the app, he logs in using his username and password for first time. These credentials are sent to Django server using a REST call. Django server returns the api key corresponding to this user to the mobile app. Mobile app stores this api token on mobile end and then uses this token for every subsequent REST call. User doesn't have to provide the credentials anymore.

Making unauthenticated POST requests

Unauthenticated POST requests will not work anymore

Try creating an Expense without passing any api key.

post_data = {'description': 'Bought Two scoops of Django', 'amount': 399}
headers = {'Content-type': 'application/json'}
r = requests.post("http://localhost:8000/api/expense/", data=json.dumps(post_data), headers=headers)
print r.status_code       #This will give 401

Check that Expense count isn't increased

Making authenticated POST requests

You only need to change the url to include username and api_key in the url. This will make the request authenticated.

r = requests.post("http://localhost:8000/api/expense/?username=sheryl&api_key=1a23", data=json.dumps(post_data), headers=headers)

This should have worked and Expense count should have increased.

Try with wrong api_key and it will fail.

Getting only User's expense

Till now we aren't associating Expense to User. Let's add a ForeignKey to User from Expense.

Expense model becomes:

from django.db import models
from django.contrib.auth.models import User


class Expense(models.Model):
    description = models.CharField(max_length=100)
    amount = models.IntegerField()
    user = models.ForeignKey(User, null=True)

Since we already have some Expenses in db which aren't associated with a User, so we kept User as a nullable field.

Make and run migrations

python manage.py makemigrations
python manage.py migrate

Right now our authorization class is set to Authorization. With this every user is authorized to see every expense. We will have to add a custom authorization class to enforce that users see only their expenses.

Add the following to expenses/api.py

class ExpenseAuthorization(Authorization):

    def read_list(self, object_list, bundle):
        return object_list.filter(user=bundle.request.user)

And change authorization on ExpenseResource.Meta so it becomes:

class ExpenseResource(ModelResource):

    class Meta:
        queryset = Expense.objects.all()
        resource_name = 'expense'
        fields = ['description', 'amount']
        filtering = {
            'amount': ['gt'],
            'description': ['icontains']
        }
        authorization = ExpenseAuthorization()
        authentication = ApiKeyAuthentication()

Explanation of ExpenseAuthorization

When GET endpoint is called for expense list, object_list is created which gives all the expenses.
After this, authorization is checked where further filtering could be done.
In case of GET on list endpoint, authorization class' read_list() method is called. object_list is passed to read_list.
In tastypie there is a variable called bundle. And bundle has access to request using bundle.request
When authentication is used properly, bundle.request.user is populated with correct user.

Try expense list endpoint for Sheryl

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

You will not get any expense after adding ExpenseAuthorization

{"meta": {"limit": 20, "next": null, "offset": 0, "previous": null, "total_count": 0}, "objects": []}

This happenned because at this point no expense is associated with Sheryl.

Create an expense for Sheryl and try the GET endpoint

On the shell

u = User.objects.get(username='sheryl')
Expense.objects.create(description='Paid for the servers', amount=1000, user=u)
Expense.objects.create(description='Paid for CI server', amount=500, user=u)

Try expense list endpoint for Sheryl again

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

You should be able to see all of Sheryl's expenses.

Fifth commit here.

git checkout 26f7c1

How mobile app will use it.

When Sheryl installs the app, she will be asked to login for the first time. There will be a REST endpoint which takes the username and password for a user and if the credentials are right, returns the api key for the user. Sheryl's api key will be returned to the mobile app which will store it in local storage. And when Sheryl wants to see her expenses, this REST call will be made to Django server.

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

This will only return Sheryl's expenses.

POST and create Sheryl's expense

Till now if a POST request is made, even if with Sheryl's api key, expense is created in db but is not associated with Sheryl.

We want to add functionality where if POST request is made from Sheryl's device then expense is associated with Sheryl. If POST request is made from Mark's device then expense should be associated with Mark.

Tastypie provides several hookpoints. We will use one such hookpoint. ModelResource provides a method called hydrate which we need to override. Add the following method to ExpenseResource.

def hydrate(self, bundle):
    bundle.obj.user = bundle.request.user
    return bundle

This method is called during POST/PUT calls.
bundle.obj is an Expense instance about to be saved in the database.
So we set user on bundle.obj by reading it from bundle.request. We have already discussed how bundle.request is populated during authentication flow.

Make a POST request now with Sheryl's api_key.

post_data = {'description': 'Paid for iDoneThis', 'amount': 700}
r = requests.post("http://localhost:8000/api/expense/?username=sheryl&api_key=1a23", data=json.dumps(post_data), headers=headers)

Verify that the latest expense instance gets associated with Sheryl. You can also verify it by seeing that this object gets returned in GET expense list endpoint.

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

Sixth commit at this point

git checkout 17b932

Try on your own

Create one more user in database, from shell.
Create api key for this user.
POST to REST endpoint with this new user's api_key and username and verify that the expense gets associated with this new user.
Check GET expense list for this new user and verify that only expense created for this user is in the response.

Now is a good time to dig deeper into django-tastypie and understand about following:

Dehydrate cycle. It is used during GET calls.
Hydrate cycle. It is used during POST/PUT calls. Once you read about hydrate cycle, you will understand when method hydrate() is called.
More about authorization and different methods available on Authorization which could be overridden by you.

Want more?

I am still trying few things with tastypie. Hereafter I will not have much explanation, but I will point to the commit where I attain certain functionality change.

Authorization on detail endpoint.

Expense with id 1 is not associated with any user. But Sheryl is still able to see it at:

http://localhost:8000/api/expense/1/?format=json&username=sheryl&api_key=1a23

She shouldn't be able to see it as it is not her expense.

So add the following to ExpenseAuthorization

def read_detail(self, object_list, bundle):
    obj = object_list[0]
    return obj.user == bundle.request.user

After this Sheryl will not be able to see detail endpoint of any expense which doesn't belong to her. Try it

http://localhost:8000/api/expense/1/?format=json&username=sheryl&api_key=1a23

Commit id for this:

e650f3
git show e650f3

PUT endpoint

Expense with id 5 belongs to Sheryl. She wants to update this expense, she essentially want to change the description.

Current thing is:

http://localhost:8000/api/expense/5/?format=json&username=sheryl&api_key=1a23

Make PUT request

put_url = "http://localhost:8000/api/expense/5/?username=sheryl&api_key=1a23"
put_data = {'description': 'Paid for Travis'}
headers = {'Content-type': 'application/json'}
r = requests.put(put_url, data=json.dumps(put_data), headers=headers)

Description of Expense 5 is updated as you can verify by trying the detail endpoint again.

http://localhost:8000/api/expense/5/?format=json&username=sheryl&api_key=1a23

Notice that amount remains unchanged. So PUT changes whatever data you provide in the api call and lets everything else remain as it is.

DELETE endpoint

First check all of Sheryl's expenses

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

Sheryl wants to delete her expense with id 5. After this is done she will have one less expense in db.

delete_url = "http://localhost:8000/api/expense/5/?username=sheryl&api_key=1a23"
r = requests.delete(delete_url)

Verify that this expense got deleted.

So we were able to do Create, Read, Update and Delete with REST api calls.

Restrict POST request to certain users only

Suppose we want the users to be able to create expenses from web end but don't want to allow creating expense from mobile using the api. Yeah, weird requiremtn.

Also we don't want to disallow POST for all users. We still want Sheryl to be able to POST.

To try this we first need a new user with api key in our system. Create it from Django shell.

u = User.objects.create_user(username='mark', password='def', email='mark@abc.com')
ApiKey.objects.create(key='2b34', user=u)

Restricting POST for everyone except Sheryl

Add following method to ExpenseAuthorization

def create_detail(self, object_list, bundle):
    user = bundle.request.user
    # Return True if current user is Sheryl else return False
    return user.username == "sheryl"

Try making POST request as Mark and see you will not be able to do it. If you want you can see the expense count at this point.

post_data = {'description': 'Petty expense', 'amount': 3000}
r = requests.post("http://localhost:8000/api/expense/?username=mark&api_key=2b34", data=json.dumps(post_data), headers=headers)
print r.status_code #Should have got 401

Also you can check the expense count again to verify that expense isn't created.

Status code 401 tells that you aren't authorized to do this operation.

Verify that Sheryl is still able to create expense

Try posting the same post_data as Sheryl

r = requests.post("http://localhost:8000/api/expense/?username=sheryl&api_key=1a23", data=json.dumps(post_data), headers=headers)
print r.status_code

Status code must be 201 in this case which means expense is created. You will be able to see this expense at Sheryl's GET expense list endpoint.

Verify that Mark is still able to do GET

Mark or any other user should still be able to make GET request even if he isn't able to make POST request.

http://localhost:8000/api/expense/?username=mark&api_key=2b34&format=json

Since Mark doesn't have any expense in db, so no object is there is objects key of response. Try creating an expense for this user from shell and then try the GET endpoint again.

Explicitly defining fields and customising them

ModelResource.Meta.fields can be dealt with in a similar way to ModelForm.Meta.fields. If you add the field to ModelResource.Meta.fields then it gets sane default behavriour. But it can be customised by adding the field explicitly on the ModelResource.

Let's try customising description field.

Add it explicitly on ExpenseResource

description = fields.CharField()

You will need to import the following for it to work.

from tastypie import fields

Try the GET endpoint

http://localhost:8000/api/expense/?username=sheryl&api_key=1a23&format=json

You will see description as null for every expense. Because we committed a mistake.

While explicitly defining a field, we also need to tell the expense attribute that needs to be used for this particular field.

So we need to say

description = fields.CharField(attribute='description')

Now GET endpoint will work as it used to work earlier.

But we did not achieve anything by explicitly adding the field. So what's the point of explicitly adding it.

Suppose you want the description in the detail endpoint but don't want it on list endpoint. This can be achieved in following way

description = fields.CharField(attribute='description', use_in='detail')

Try the list and detail endpoint now and notice the difference.

http://localhost:8000/api/expense/?username=sheryl&api_key=1a23&format=json
http://localhost:8000/api/expense/8/?username=sheryl&api_key=1a23&format=json

use_in is documented here

Importing your old comments to Disqus site

In one of my latest blogpost on disqus I covered topics on integrating Disqus to the website and disqus SSO. In this post, I will let you know how to migrate the older comments to Disqus.

If you sneak peek in to the alluring features of disqus you may make your mind to migrate your custom commenting system on your blog to use disqus commenting system. The threaded comments and replies, powerful moderation and admin tools, RSS options and many more features come in as battaries included with Disqus which makes the commenting more interactive and easy to deal with.

Let us kick start the process. Till date importing the old comments directly from the blogger and wordpress to disqus is feasible. This can be achieved by using tools and plugins that are already existing in them and its pretty straight forward. In this post our prime concern will be laid on custom XML import format. If you are using neither blogger, nor wordpress the custom XML import format which is based on the WXR (WordPress eXtended RSS) schema comes to the rescue. Disqus also supports MovableType, and IntenseDebate but preferably WXR is of more concern.

I consider a case in a Django project, which does adopt a custom commenting system in which the comments are stored in a postgresql database. This process of migrating the comments to disqus comprises of two steps in this case.

Creating a WXR schema of the existing comments
Importing the WXR file to Disqus site.

1. Creating a WXR schema of the existing comments

WXR format is based on the Rss specification. This format is normally used for importing or exporting data from wordpress servers. Now the aim is to create a WXR file from the existing comments in our storage. For that we are going to write a python script. Disqus has prescribed a format of the file that we are planning to import to Disqus. The format looks in the following way.

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
  xmlns:content="http://purl.org/rss/1.0/modules/content/"
  xmlns:dsq="http://www.disqus.com/"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:wp="http://wordpress.org/export/1.0/"
>
  <channel>
    <item>
      <!-- title of article -->
      <title>Foo bar</title>
      <!-- absolute URI to article -->
      <link>http://foo.com/example</link>
      <!-- body of the page or post; use cdata; html allowed (though will be formatted to DISQUS specs) -->
      <content:encoded><![CDATA[Hello world]]></content:encoded>
      <!-- value used within disqus_identifier; usually internal identifier of article -->
      <dsq:thread_identifier>disqus_identifier</dsq:thread_identifier>
      <!-- creation date of thread (article), in GMT. Must be YYYY-MM-DD HH:MM:SS 24-hour format. -->
      <wp:post_date_gmt>2010-09-20 09:13:44</wp:post_date_gmt>
      <!-- open/closed values are acceptable -->
      <wp:comment_status>open</wp:comment_status>
      <wp:comment>
        <!-- sso only; see docs -->
        <dsq:remote>
          <!-- unique internal identifier; username, user id, etc. -->
          <dsq:id>user id</dsq:id>
          <!-- avatar -->
          <dsq:avatar>http://url.to/avatar.png</dsq:avatar>
        </dsq:remote>
        <!-- internal id of comment -->
        <wp:comment_id>65</wp:comment_id>
        <!-- author display name -->
        <wp:comment_author>Foo Bar</wp:comment_author>
        <!-- author email address -->
        <wp:comment_author_email>foo@bar.com</wp:comment_author_email>
        <!-- author url, optional -->
        <wp:comment_author_url>http://www.foo.bar/</wp:comment_author_url>
        <!-- author ip address -->
        <wp:comment_author_IP>93.48.67.119</wp:comment_author_IP>
        <!-- comment datetime, in GMT. Must be YYYY-MM-DD HH:MM:SS 24-hour format. -->
        <wp:comment_date_gmt>2010-09-20 13:19:10</wp:comment_date_gmt>
        <!-- comment body; use cdata; html allowed (though will be formatted to DISQUS specs) -->
        <wp:comment_content><![CDATA[Hello world]]></wp:comment_content>
        <!-- is this comment approved? 0/1 -->
        <wp:comment_approved>1</wp:comment_approved>
        <!-- parent id (match up with wp:comment_id) -->
        <wp:comment_parent>0</wp:comment_parent>
      </wp:comment>
    </item>
  </channel>
</rss>

Things to keep in mind:

As recommended by the Disqus we need some points to keep in mind while creating a WXR file of the existing comments.

Only use the "dsq:remote" tag (and its children) if you're using Single Sign-On; otherwise remove it.
If you're manually filling in make sure it's unique to each user or else all comments will display the same author name.
Each "item" tag can parent an infinite number of "wp:comment" tags.
There is a minimum character requirement for each comment message, which is 3 characters.

Every item is supposed to be a comment on the site. As I am not using the SSO in the site, we can remove the . The link specifies the site url to the specific post. In my case I happened to find some encoding issues with the which does take the whole body of the post so I preferred removing it from the WXR file and it worked fine.

The script appears to be in the below manner. I accessed my comment model to get the comment data. And then I created the xml file in the format prescribed by Disqus. I skipped some tags which ought to mess up encryption issues. But beware, try to create as the disqus prescribes for the easy flow. The script looks like the following way.

from mysite.models import Comment
from django.contrib.sites.models import Site

site = Site.objects.all()[0]

comments = Comment.objects.all()
xml_data_header = """<?xml version="1.0" encoding="UTF-8"?>
                <rss version="2.0"
                  xmlns:content="http://purl.org/rss/1.0/modules/content/"
                  xmlns:dsq="http://www.disqus.com/"
                  xmlns:dc="http://purl.org/dc/elements/1.1/"
                  xmlns:wp="http://wordpress.org/export/1.0/"
                >
                      <channel>
                  """
xml_data_footer = """
                      </channel>
                    </rss>
                  """
file_dis = open('disqus.xml','wb')
file_dis.write(xml_data_header)
for comment in comments:
    xml_data_all = ''
    xml_data = """
                <item>
                  <title>{0}</title>
                  <link>http://{1}</link>
                  <content:encoded><![CDATA[{2}]]></content:encoded>
                  <dsq:thread_identifier>{3}</dsq:thread_identifier>
                  <wp:post_date_gmt>{4}</wp:post_date_gmt>
                  <wp:comment_status>open</wp:comment_status>
                  <wp:comment>
                    <wp:comment_id>{5}</wp:comment_id>
                    <wp:comment_author>{6}</wp:comment_author>
                    <wp:comment_author_email>{7}</wp:comment_author_email>
                    <wp:comment_author_url>{8}</wp:comment_author_url>
                    <wp:comment_author_IP>{9}</wp:comment_author_IP>
                    <wp:comment_date_gmt>{10}</wp:comment_date_gmt>
                    <wp:comment_content><![CDATA[{11}]]></wp:comment_content>
                    <wp:comment_approved>1</wp:comment_approved>
                  </wp:comment>
                </item>

                """.format(comment.comment_for.title.encode('utf-8'), site.domain+comment.comment_for.get_absolute_url().encode('utf-8'), comment.comment_for.text.raw.encode('utf-8'), comment.comment_for.pk, comment.comment_for.created_on, comment.pk, \
                    comment.user_name.encode('utf-8'), comment.email_id.encode('utf-8'), comment.user_ip, comment.created_on, comment.text.encode('utf-8'), \
                    )
    # xml_data_all += xml_data
    file_dis.write(xml_data)

file_dis.write(xml_data_footer)
file_dis.close()

Once you run the above script the resultant will be a "disqus.xml" file. An example output of the file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
            <rss version="2.0"
              xmlns:content="http://purl.org/rss/1.0/modules/content/"
              xmlns:dsq="http://www.disqus.com/"
              xmlns:dc="http://purl.org/dc/elements/1.1/"
              xmlns:wp="http://wordpress.org/export/1.0/"
            >
                  <channel>                  
                    <item>
                      <title>Google Appengine - First Impressions</title>
                      <link>http://agiliq.com/blog/2008/04/google-appengine-first-impressions/</link>
                      <content:encoded><![CDATA[245]]></content:encoded>
                      <dsq:thread_identifier>245</dsq:thread_identifier>
                      <wp:post_date_gmt>2008-04-09 04:02:57</wp:post_date_gmt>
                      <wp:comment_status>open</wp:comment_status>
                      <wp:comment>
                        <wp:comment_id>482</wp:comment_id>
                        <wp:comment_author>Ivan</wp:comment_author>
                        <wp:comment_author_email>suchy.ivan@gmail.com</wp:comment_author_email>
                        <wp:comment_author_url></wp:comment_author_url>
                        <wp:comment_author_IP>None</wp:comment_author_IP>
                        <wp:comment_date_gmt>2008-04-09 08:48:53</wp:comment_date_gmt>
                        <wp:comment_content><![CDATA[<p>On dev server data is stored in /tmp folder</p>]]></wp:comment_content>
                        <wp:comment_approved>1</wp:comment_approved>
                      </wp:comment>
                    </item>

                     </channel>
            </rss>

Once we are done with creating the WXR schema we should import it to the disqus site.

2.Importing the WXR file to Disqus site.

To attain the second part you need to have a disqus account and a disqus site available for you. You can get to know more about creating a disqus site in the linked blogpost

Once you are done with the creation of disqus site go to the settings which can be accessible on admin panel. There appears the settings page, click on the discussion on the right top panel. Click "import" tab and there appears the different types of import formats that disqus supports. Select generic(WXR) and click on browse. Upload the disqus.xml file and once the process get completed it shows up if there are any errors in the uploaded file. If there are any errors, the process of importing comments will be kept on hold. If the comments are successfully imported it takes some time to get those comments reflected in your site.

TIP: The best practice would be to try importing in to test disqus site and once it works perfectly, move towards importing them to the main site.

Building a RESTful API with Django-rest-framework

API's turned to be the heart of every application in our time. With the rise of social media, API's are being developed at a faster pace and gaining a lot of attention. Gone are the days where RPC architectures like CORBA and XML-RPC are used to enable the exchange of information and REST has taken its place making its mark in getting the things pretty straight forward.

We use APIs in our everyday life. Take an instance booking a ticket for a movie through some website. Here is the process that takes place in the background. When dealing with the payment for the ticket, the website connects to the bank and sends your credit card details to the remote application and it gets verified. And once, the payment is confirmed by the remote application, it sends a confirmation response to the movie booking website to issue the tickets. The payment stuff turned possible here as the bank facilitates an API through which a website can connect to it and deal with the payments which resulted us an effective and seamless transaction.

In this post we will get to know how to build a Restful API for an application using a Django-rest-framework. Before dipping ourselves in the API building stuff, let us acquire a picture on REST.

REST

In simple words REST is a web architecture which lives on top of HTTP. It's a way in which we should use HTTP where it acts as a set of guiding principles behind the given application. It allows the clients and servers to communicate with each other in all the possible ways like getting the resource, posting the resource, deleting etc. REST is a resource based on which we use an HTTP verb to dictate what operation we want to do on a URI which can be GET, POST and so on. In REST we represent the resource data in JSON and XML format. An API is called a Restful API if it adheres to all the interaction constraints of REST.

Django-rest-framework

Django-rest-framework makes the process of building web API's simple and flexible. With its batteries included it won't be a tedious task to create an API. Before creating our own API let us look into some vital parts of the REST framework The important parts of Django-rest-framework are:

Serialization and Deserialization: The first part in the process of building an API is to provide a way to serialize and deserialize the instances into representations. Serialization is the process of making a streamable representation of the data which will help in the data transfer over the network. Deserialization is its reverse process. In our process of building an API we render data into JSON format. To achieve this Django-rest-framework provides JSONRenderer and JSONParser. JSONRenderer renders the request data into JSON, using utf-8 encoding and JSONParser parses the JSON request content.

Without dipping inside the technicalities let us see how it takes place.

Serializing data with JSONRenderer:

A JSONRenderer converts the request data into JSON using utf-8 encoding. Normally we use this when we want our data to be streamable.

>>> data
{'cal_id': u'2', 'username': u'tester'}
>>> content = JSONRenderer().render(data)
>>> content
'{"cal_id": "2", "username": "tester"}'

Deserializing data with JSONParser:

A JSONParser parses the JSON request content. Firstly, we parse a stream into Python native datatypes with BytesIO and then we will restore those native datatypes into to a fully populated object instance afterwards. We use JSONParser to desrialize the data.

>>> content
'{"cal_id": "2", "username": "tester"}'
>>> stream = BytesIO(content)
>>> data = JSONParser().parse(stream)
>>> data
{u'username': u'tester', u'cal_id': u'2'}

Requests and Responses

Requests and responses are the essential parts of Django-rest-framework which provides flexibility while parsing.

Request objects:

The request object in REST framework has more abilities. The attribute request.DATA in Request object is similar to Request.POST added with the capabilities of handling arbitrary data which works for 'POST', 'PUT' and 'PATCH' methods. And also we can use the different attributes like request.FILES, request.QUERY_PARAMS, request.parsers, request.stream which will help in dealing with the request a hassle free task.

Response objects:

The Response object helps to render the correct content type as requested by the client unlike the normal HttpResponse.

Syntax: Response(data, status=None, template_name=None, headers=None, content_type=None)

Status Codes: Instead of using the normal HTTP status codes, we will make use of explicit identifiers for every status code to make the process of reading the status code much simple. Here are some status codes that we represent in a REST framework we use normally.

HTTP_200_OK
HTTP_201_CREATED
HTTP_400_BAD_REQUEST
HTTP_401_UNAUTHORIZED
HTTP_403_FORBIDDEN

Views: APIView is a view provided by Django-rest-framework which subclasses the Django's view class. If we use this view the requests passed to the handler methods will no more Django's HttpRequest instances and they will be REST framework's Request instances. And while dealing with the responses they work to set the correct renderer on the response. The process of authentication and the permissions stuff also will be dealt with it.

Creating an API with Django-rest-framework

Using all the above stuff lets us build a simple API which gives the details of a user. The procedure goes the following way:

We will be using Django's built in user in this example. We will create a serializers.py file where we create serializers which are similar to Django forms. Just like model forms we have got model serializers here, which will stop replication of code. And we create a view which lists all the users, creates a new user, retrieve a user and update a user.

Here we go with the installation with virtual environment activated:
```
pip install djangorestframework
```

Create a Django project and an app. I created it with a name rest_example and restapp

django-admin.py startproject rest_example
cd rest_example
python manage.py startapp restapp

Add rest_framework and the app name to the installed apps.

INSTALLED_APPS = (
    ...
    'rest_framework',
    'restapp')

Run syncdb command
```
python manage.py syncdb
```

Create a restapp/serializers.py which should look like the below way.

from django.contrib.auth.models import User

from rest_framework import serializers

class UserSerializer(serializers.ModelSerializer):
    class Meta:
        model = User
        fields = ('id', 'username', 'first_name', 'last_name', 'email')

Create a restapp/views.py file for listing all users, creating a new user and so on.

from django.contrib.auth.models import User
from django.http import Http404

from restapp.serializers import UserSerializer
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status

class UserList(APIView):
"""
List all users, or create a new user.
"""
    def get(self, request, format=None):
        users = User.objects.all()
        serializer = UserSerializer(users, many=True)
        return Response(serializer.data)

    def post(self, request, format=None):
        serializer = UserSerializer(data=request.DATA)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data, status=status.HTTP_201_CREATED)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

    def delete(self, request, pk, format=None):
        user = self.get_object(pk)
        user.delete()
        return Response(status=status.HTTP_204_NO_CONTENT)

class UserDetail(APIView):
    """
    Retrieve, update or delete a user instance.
    """
    def get_object(self, pk):
        try:
            return User.objects.get(pk=pk)
        except User.DoesNotExist:
            raise Http404

    def get(self, request, pk, format=None):
        user = self.get_object(pk)
        user = UserSerializer(user)
        return Response(user.data)

    def put(self, request, pk, format=None):
        user = self.get_object(pk)
        serializer = UserSerializer(user, data=request.DATA)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

    def delete(self, request, pk, format=None):
        user = self.get_object(pk)
        user.delete()
        return Response(status=status.HTTP_204_NO_CONTENT)

Update the root urls.py

from django.conf.urls import patterns, include, url
from django.contrib import admin

from restapp import views

admin.autodiscover()

    urlpatterns = patterns('',
    url(r'^admin/', include(admin.site.urls)),
    url(r'^users/', views.UserList.as_view()),
    url(r'^users/(?P<pk>[0-9]+)/$', views.UserDetail.as_view()),)

Voila!! we are done creating our API.

Let us test our API now.

python manage.py runserver

Now go to your browser and try localhost:8000/users/1/ or use curl on your command line

curl http://127.0.0.1:8000/users/1/

You can get the user details what you have filled while creating the super user. which would look like this.

{"id": 1, "username": "restapp", "first_name": "", "last_name": "", "email": "rakesh@agiliq.com"}

Now we can try posting data:

curl -X POST http://127.0.0.1:8000/users/ -d '{"username":"rakhi", "email":"rakhi@agiliq.com"}' -H "Content-Type: application/json"

And it creates a new user with the username "rakhi" and email "rakhi@agiliq.com"

    {"id": 6, "username": "rakhi", "first_name": "", "last_name": "", "email": "rakhi@gmail.com"}

You can check out the example application on github

Disqus and Disqus SSO

Outsourcing the comments and discussions part to a third party widget always helps us to save a lot of effort. When you feel that if it is okay to share your thoughts and discussions with someone out there who serves the capabilities of social integration and spam detection won't it be a cake walk?

Disqus is such a kind of blog comment hosting service which is platform-agnostic and URL-agnostic. It is built with javascript and backend technology, Python using Django. Let us get to know how to add this to our Django app.

Adding Disqus widget to a site is pretty straight forward and effortless. Here are the steps to follow:

Create an account on Disqus and fill the details at https://disqus.com/admin/create/
Once you finish registration, access the universal code
In the next following page, get the script and use it wherever we want to display comments
Update the settings file with DISQUS_SHORTNAME

The first step includes signing up in to Disqus and getting an account. Once we are done with it, we will be heading to a home page where we can find "Add disqus to your site" on top left. If we click it, we will be redirected to site profile page where we should fill the details like the site name and unique Disqus URL which we call it as "disqus_shortname". Once the registration is completed it will show up the general settings page where we can select the platform i.e univeral code, wordpress, blogger, Tumblr, square space and so on. As we are planning to add Disqus to Django app we should select Universal code. The Universal code is a script which looks like this:

<div id="disqus_thread"></div>
<script type="text/javascript">
    /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
    var disqus_shortname = 'try1'; // required: replace example with your forum shortname

    /* * * DON'T EDIT BELOW THIS LINE * * */
    (function() {
        var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
        dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
        (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
    })();
</script>
<noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>

The script can be used in the template wherever we want our comments section to show up. That delivers us the comments section in our app with one step remaining. Access the "disqus_shortname" i.e. the unique Disqus URL that we specified in the process of registration and add it in the settings file of the Django app.

eg. DISQUS_WEBSITE_SHORTNAME = "mydjangoapp"

Disqus SSO

Commenting on a thread requires a person to log in with their social accounts. It would be an extra step for the people to log in again for commenting after logging in to the app. To abate this fatigue of logging in again we can use the "single sign on".

As of now SSO is available to Disqus as a free add on. To add to our app we should request the support team of Disqus through a contact form. Once we send a request to support team it takes like 24 hours to get the access to SSO console.

Steps to register Disqus application

Register the application at https://disqus.com/api/applications/register/
Once the registration is completed the secret keys and the public keys can be accessed at https://disqus.com/api/applications/
Add the keys to settings file as DISQUS_PUBLIC_KEY and DISQUS_SECRET_KEY
To avail console, SSO and stuff we can go to https://disqus.com/api/applications/
Pass existing user data

The first part to start using it is creating a Remote Domain which can be used as a reference between Disqus and the user accounts. We can do it right away at https://disqus.com/api/sso/. Add the public key and secret key of the application in the settings. We can access the keys at https://disqus.com/api/applications/ under the app name. Use HMAC-SHA1 to pass the user details to the Disqus to authenticate. The message that we pass should contain id, username and email. We can generate the HMAC-SHA1 the following way:

HMAC->SHA1(secret_key, message + ' ' + timestamp)

Once the message is generated the following script need to be used in the page where we want the Disqus to show up.

var disqus_config = function () 
{
    // The generated payload which authenticates users with Disqus
    this.page.remote_auth_s3 = '<message> <hmac> <timestamp>';
    this.page.api_key = 'public_api_key';
}

In Django we can accomplish this by registering a simple tag with function to generate the HMAC-SHA1 and using that tag in the template where we are displaying the Disqus comments. The flow moves on the following way:

Firstly, we should access the value of the DISQUS_SECRET key from the settings. In the next step we should create a json packet of the data attributes, i.e 'id', 'username', 'email'. Then we should encode the json packet to base64 and we should generate a time stamp for signing the message. To pass the secret key and the user details we should generate the HMAC signature. Thus acquired HMAC signature should be returned as a script tag to insert the SSO message. At the end the function appears as in the following code. Check out the Disqus SSO Example

The template tag should be used in the place where we want the Disqus comments to appear.

And that ends up the process of adding SSO to Disqus.

Travis and coveralls for private repo

Before we begin, i recommend you to read this first Continous integration with travis and coveralls.io for Django apps.

Here is how .travis.yml example file looks like:

language: python
python:
  - 2.7
install:
  - pip install -r requirements.txt
  - pip install coveralls
script:
  coverage run manage.py test
after_success:
  coveralls

Setting up coveralls for private repositories requires you to add just one more file .coveralls.yml.

1) Create a .coveralls.yml and make sure it resides in your project's root directory.

2) Add the following to this file:

service_name: travis-pro
repo_token: ****

service_name is to specify where Coveralls should look to find additional information about your builds.

You can get the repo_token from your repository's page on Coveralls, if you have the admin privileges. This is to tell which project on Coveralls your project maps to.

Make sure your repo_token remains secret and do not add this to your public repository.

3) Add the file, commit it and make a git push.

4) If everything is OK you should see some thing like the below in your travis build:

Submitting coverage to coveralls.io...
Coverage submitted!
Job #22.1
https://coveralls.io/jobs/54864565

Thats it now get a coverage badge from coveralls and add this badge in your repo's README.md.

Passing parameters to Django admin action

This post will show how to pass parameters to custom Django admin actions.

This post assumes that you have an idea of Django actions.

Django provides a default admin action which is 'Delete selected rows'. Changelist page for all the models registered with admin has this action available. This action allows to select multiple rows and delete them in a single POST request.

Problem statement

We want to allow following Django action from admin.

There is a model called Instrument. It has a field price
Allow selecting multiple rows for this model and update the value of price for selected rows.
Price should not be hardcoded. Admin should be able to enter the price.

We should be able to achieve something like:

Solution

Setup the project first.

Setup project

Start a Django project inside virtual environment

/tmp $ mkvirtualenv custom-actions

(custom-actions)/tmp $ django-admin.py startproject actions
(custom-actions)/tmp/actions $ python manage.py startapp instruments

Writing the model in instruments/models.py

class Instrument(models.Model):
    name = models.CharField(max_length=100)
    price = models.IntegerField()

    def __unicode__(self):
        return self.name

Add instruments to INSTALLED_APPS

Run syncdb

(custom-actions)/tmp/actions $ python manage.py syncdb

from django.contrib import admin

from .models import Instrument

admin.site.register(Instrument)

You should be able to see Instrument changelist page at http://localhost:8000/admin/instruments/instrument/

Create few instrument instances. You can also use the fixture provided at the github repo accompanying this blog.

python manage.py loaddata instruments/fixtures/instruments.json

You should be able to see Action Delete selected instruments in dropdown of Instrument changelist page.

Writing our custom action.

We want to pass a parameter which will be used to set price on selected rows. This requires showing a Form field.

When we customize the admin , like adding list_display, list_filter, inlines etc, we write a class which extends from ModelAdmin. ModelAdmin has an attribute called action_form which determines the form that gets used for action selection dropdown.

See https://github.com/django/django/blob/1.6/django/contrib/admin/options.py#L396

action_form is django.contrib.admin.helpers.ActionForm.

Instead of using Django provided ActionForm, we will use a custom form. This form should extend from Django provided ActionForm because we want to retain the functionality provided by Django while adding an additional field of our own.

This form would be:

from django.contrib.admin.helpers import ActionForm

class UpdateActionForm(ActionForm):
    price = forms.IntegerField()

Django provides a default action 'Delete selected' and this same form would be used for the default action too. So, we should be keeping the price field as optional so 'Delete selected' functionality keeps working even if we don't input anything in price field.

And we should be setting this form as action_form attribute of our ModelAdmin subclass.

class InstrumentAdmin(admin.ModelAdmin):
    action_form = UpdateActionForm

And this InstrumentAdmin should be used while registering model Instrument with admin.

admin.site.register(Instrument, InstrumentAdmin)

Final content of instrument/admin.py becomes

from django.contrib import admin
from django.contrib.admin.helpers import ActionForm
from django import forms

from .models import Instrument

class UpdateActionForm(ActionForm):
    price = forms.IntegerField(required=False)

class InstrumentAdmin(admin.ModelAdmin):
    action_form = UpdateActionForm

admin.site.register(Instrument, InstrumentAdmin)

With this your action form would contain a Price field. Verify it on Instrument changelist page.

Confirm that you are still able to delete rows using 'Delete selected instruments' action without the Price fields interfering with the existing functionality.

If you have written Django actions earlier, writing the function to handle our action should be a no brainer.

def update_price(modeladmin, request, queryset):
    price = request.POST['price']
    price = int(price)
    queryset.update(price=price)
update_price.short_description = 'Update price of selected rows'

And add it as actions of InstrumentAdmin.

class InstrumentAdmin(admin.ModelAdmin):
    action_form = UpdateActionForm
    actions = [update_price]

With this you should get 'Update price of selected rows' in actions dropdown.

You might argue, why did we use request.POST to get the price and not use action_form.cleaned_data. Have some patience, I will answer that.

Now try selecting some rows and input a price in the form and verify if your code works properly. It must work.

We used request.POST and not action_form.cleaned_data because we don't have access to the instance of action_form, i.e instance of UpdateActionForm, in any way inside function update_price. If we try modeladmin.action_form, it will return us class UpdateActionForm and not instance of UpdateActionForm

We can make update_price better by showing a message after the instances' price has been updated.

def update_price(modeladmin, request, queryset):
    price = request.POST['price']
    price = int(price)
    queryset.update(price=price)
    modeladmin.message_user(request, ("Successfully updated price for %d rows") % (queryset.count(),), messages.SUCCESS)

For message to display, Django message framework should be enabled. It is enabled by default, so unless you have played with settings.py, message will just work.

Heroku Django S3 for serving Media files

Why should i store my media files on AWS S3?

Heroku dynos have limited life span, and when they die and get replaced, the files within them are lost. So you should set up Django media handler to put the files somewhere permanent.

Let's begin:

1) Install dependencies:

$ pip install django-storages boto

(Update your requirements.txt using pip freeze > requirements.txt)


2) Update INSTALLED_APPS in your settings.py:
INSTALLED_APPS += ('storages',)



3) Sign in to AWS Console, select S3 under Storage & Content Delivery and create a bucket.
4) Don't forget to generate AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
5) Add the below to your settings file:
AWS_QUERYSTRING_AUTH = False
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
AWS_STORAGE_BUCKET_NAME = os.environ['S3_BUCKET_NAME']
MEDIA_URL = 'http://%s.s3.amazonaws.com/your-folder/' % AWS_STORAGE_BUCKET_NAME
DEFAULT_FILE_STORAGE = "storages.backends.s3boto.S3BotoStorage"



Note: You should never expose your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY there is some reason why they are called SECRET.
6) Install AWS CLI on your machine.
7) Its time to move your files to AWS S3 bucket.
$ aws s3 cp myfolder s3://mybucket/myfolder --recursive



8) After the previous step you can use the following command to view the contents of your bucket.
$ aws s3 ls s3://mybucket



9) Grant public-read permission to everyone, add the below to your S3 bucket policy.
{
  "Version":"2008-10-17",
  "Statement":[{
    "Sid":"AllowPublicRead",
        "Effect":"Allow",
      "Principal": {
            "AWS": "*"
         },
      "Action":["s3:GetObject"],
      "Resource":["arn:aws:s3:::bucket/*"
      ]
    }
  ]
}



You can also grant full access to specific IP addresses check this Bucket Policies
Thats it now your django app should be able to serve your media files.If not please check your bucket policy, you can check the url link under S3 Management Console->Properties.
Any problem or feedback? feel free to leave your comments below.



Supervisor with Django and Gunicorn
Supervisor with Django: A starter guide
This post assumes that you have used gunicorn and know what it does. I will try everything inside a virtual environment and hope you do the same.
What is supervisor.
Supervisor is a monitoring tool that can monitor your processes. It can restart the process if the process dies or gets killed for some reason.
Use of supervisor: Why I started using it.
In production, I use gunicorn as web server. I started a gunicorn process as a daemon and logged out from the server. My site ran as expected for few days. All of a sudden, we started getting '502 Bad Gateway' and I had no idea why. I had to ssh to the server to find out what went wrong. After ps aux | grep gunicorn, I found out gunicorn wasn't running anymore. My gunicorn process died on its own, and I had no idea when and why. Had I used supervisor, supervisor would have been controlling the gunicorn process. It must have recieved a signal when gunicorn died and it would have created a new gunicorn process in such scenario. And my site would have kept running as expected.
Other scenario
We want to run a process which doesn't allow deamonizing. eg: I wanted to keep celery, a Python worker, running on the production server. I could not run it in the foreground because I had to logout from the server. I did not find an easy way to run celery as a daemon. Here too, supervisor came handy.
Project setup
I am setting up a new Django project so you will have proper idea of the path and how to use paths in supervisor configuration file.
We will use Django 1.6.
Create a new Django project:
(PythonEnv)/tmp $ django-admin.py startproject testproj
(PythonEnv)/tmp $ cd testproj/

Project structure looks like:
(PythonEnv)/tmp/testproj $ tree
manage.py
testproj
    __init__.py
    settings.py
    urls.py
    wsgi.py

Run syncdb and runserver. Hereafter, we will run all our commands from directory /tmp/testproj/ with PythonEnv activated.
python manage.py syncdb
python manage.py runserver

And then access the '/admin/'. You should be able to see django admin.
Let's use gunicorn instead of using runserver.
pip install gunicorn

gunicorn testproj.wsgi:application --bind 127.0.0.1:8000

Make sure you can still see /admin/. You will not be able to view static files now, because we aren't using Django development server anymore. Read this post if you want to know how to serve static files using nginx. Though you don't need to do this now, our motive is not to see static files.
Let's run gunicorn as a daemon. Also, we will tell gunicorn to use a file to keep track of process id, so that we can use that process id to kill gunicorn whenever we want.
gunicorn testproj.wsgi:application --bind 127.0.0.1:8000 --pid /tmp/gunicorn.pid --daemon

The problem with this approach is, gunicorn might die at any moment and you will not know about it immediately. Since we are working on dev environment, it is acceptable but gunicorn dying on a server is not acceptable. Your site will loose too many users/customers.
Kill this gunicorn process, we will let supervisor handle gunicorn hereafter.
(PythonEnv)/tmp/testproj $ cat /tmp/gunicorn.pid 
19363
(PythonEnv)/tmp/testproj $ kill -9 19363

Using supervisor
Let's install supervisor inside our virtual environment.
(PythonEnv)/tmp/testproj $ pip install supervisor

With this, you should have a echo_supervisord_conf command available. Get a stub supervisor file in your current directory using it.
echo_supervisord_conf > ./supervisord.conf

With this, your directory structure should look like:
db.sqlite3
manage.py
supervisord.conf
testproj
    __init__.py
    settings.py
    urls.py
    wsgi.py

You can remove most of the sections of this supervisor configuration file if you want. Initially we can only keep:
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10) 
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=true               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)

From supervisor docs:
Before supervisord will do anything useful for you, you’ll need to add at least one program section to its configuration. The program section will define a program that is run and managed when you invoke the supervisord command. To add a program, you’ll need to edit the supervisord.conf file.
So, we'll add a program section for gunicorn to supervisord.conf. Add it after the [supervisord] section.
[program:gunicorn]
command=/home/akshar/.virtualenvs/PythonEnv/bin/gunicorn testproj.wsgi:application --bind 127.0.0.1:8000 --pid /tmp/gunicorn.pid ;
directory=/tmp/testproj/ ;

You need to change command and directory depending on your environment. My gunicorn executable resides at /home/akshar/.virtualenvs/PythonEnv/bin/gunicorn. Find where your gunicorn executable is and use that in command section of program.
We passed testproject.wsgi:application as application object to gunicorn, so testproject need to be available on the PATH for this to work. And for something to be available on the path, you need to be in correct directory. That's why we need the directory setting inside program section.
Make sure neither a gunicorn nor a supervisor process exists till here:
(PythonEnv)/tmp/testproj $ ps aux | grep supervisord
(PythonEnv)/tmp/testproj $ ps aux | grep gunicorn

Run supervisord
(PythonEnv)/tmp/testproj $ supervisord

Now a supervisor process as well as a gunicorn process should be running. Even though you did not explicitly start a gunicorn process, supervisor started gunicorn for you. Verify that supervisor as well as gunicorn process exists. You need to check these on a different terminal, because supervisord must be running on your first terminal in the foreground, since it wasn't daemonized.
(PythonEnv)/tmp/testproj $ ps aux | grep supervisord
(PythonEnv)/tmp/testproj $ ps aux | grep gunicorn

Access http://localhost:8000/admin/ to make sure your site is running fine.
Kill supervisord process(Ctrl+c), we will deamonize it now. Change nodaemon=true in your supervisord.conf to nodaemon=false.
Run supervisord again
(PythonEnv)/tmp/testproj $ supervisord

Supervisor should no more be running in foreground. Verify that you have a gunicorn process running too, gunicorn was started by supervisor.
(PythonEnv)/tmp/testproj $ ps aux | grep gunicorn

Also, match the process id given by 'ps aux' with pid shown in /tmp/gunicorn.pid
Verify that /admin/ is still accessible.
Supervisor client
Supervisor also provides a client using which you can see currently managed process by this supervisor instance. Add the following sections between supervisord and program section.
[inet_http_server]
port=127.0.0.1:9001   ;

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=http://127.0.0.1:9001 ;

You need to restart supervisor to see these changes taking effect. Kill the existing supervisor process by checking its id from supervisor pid file. Notice that you have a pidfile option under your supervisord section, which tells the file which stores process id of supervisor.
(PythonEnv)/tmp/testproj $ cat /tmp/supervisord.pid 
9820
(PythonEnv)/tmp/testproj $ kill -9 9820

This killed the exising supervisor process.
Also, you need to kill the running gunicorn process. Else supervisor will fail to start another gunicorn process because the current gunicorn process has already occupied port 8000.
(PythonEnv)/tmp/testproj $ cat /tmp/gunicorn.pid 
9824
(PythonEnv)/tmp/testproj $ kill -9 9824

Run supervisord again 
(PythonEnv)/tmp/testproj $ supervisord

Again verify that gunicorn process was started by supervisor.
Start supervisor client by issuing following command:
(PythonEnv)/tmp/testproj $ supervisorctl

It should tell you that a process for gunicorn is being managed by supervisor. Output should be something like:
gunicorn                         RUNNING    pid 9950, uptime 0:01:01
supervisor>

Suppose you made some change in your Django code and you need to restart gunicorn for that change to take effect. You can easily do it from supervisor client.
supervisor> restart gunicorn

Output should be:
gunicorn: stopped
gunicorn: started

You can verify that the pid of gunicorn changes everytime you restart gunicorn from supervisorctl.
supervisorctl provides a host of other functionalities too. Use command help on supervisorctl to know more.
Surprise
You have a running gunicorn process. Kill it.
(PythonEnv)/tmp/testproj $ cat /tmp/gunicorn.pid 
10137
(PythonEnv)/tmp/testproj $ kill -9 10137

Now gunicorn should be killed and you should not be able to access /admin/. But you will be able to access it, try it. Surprised? Check the pid of gunicorn now.
(PythonEnv)/tmp/testproj $ cat /tmp/gunicorn.pid 
10301

So, somehow a new gunicorn process was created as soon as we killed gunicorn, and the process id for that gunicorn process was written in /tmp/gunicorn.pid.
Why did it happen.
When you killed gunicorn, supervisor came to know that gunicorn process died. And it's supervisor's duty to restart any process if it dies, if supervisor controls that process. Since gunicorn was being controlled by supervisor, supervisor made sure to restart gunicorn. So, be assured that if you are using supervisor and let it control gunicorn, then supervisor will make sure to always keep gunicorn running. And your site will never be down.
This is how we used supervisor on our development environment. You can similarly use supervisor on a production server.
It is worth mentioning that supervisor conf can have multiple program section. So, you can use single supervisor instance to control gunicorn, celery and any other process you want.


Continuous integration with travis and coveralls.io for Django apps
Maintaining a solid rapport with the ongoing software development process always turns out to be a walk on air.
Ensuring a software build integrity and quality in every single commit makes it much more exciting.
If the current software bulid is constantly available for testing, demo or release isn't it a developer's paradise on earth?
Giving a cold shoulder to "Integration hell" the 'Continuous integration' process stands out to deliver all the above assets.
What's Continuous integration?
Continuous integration is a development practice that requires a developer to integrate code into a shared repository in which isolated changes are immediately tested and reported on when they are added to a larger code base.
Continuous integration (CI) originated from within the extreme programming paradigm, but the principles can be applied to any iterative programming model, such as agile programming. As critics noted several potential drawbacks on extreme programming many organizations have adopted CI without all of the extreme programming concepts.
Principles of CI:

Maintain a single source repository
Automate the build
Make the build self-testing
Keep the build fast
Everyone can see the results of the latest build
Automate deployment

Keywords:

Build: All steps necessary to compile and create deliverables.
Commit: The operation follows the validation of updates to existing code.
Update: This operation allows the update from the repository of the configuration management tool.
Checkout: This is the operation to extract a version of a project under development from the repository of the configuration manager.

Spices for the recipe:

Use of a version control tool (Git, SVN, etc.)
An automated build and product release process.
Instrumentation of the build process to trigger unit and acceptance tests for evey change that is commited
A notifying process that notifies for every single test fail and alerts the team.

The approach:

The developer makes a commit to the repository
The integration server detects the commit, makes a checkout, launches operations and testing.
Eventually the repository may become so different from the developers baseline referred to as "Merge hell"
In case of a failure in the testing process a notification is generated to the team.
The developers may discard the changes or fix the bugs to make it progress successfully.

So this makes us fancy trying out continuous integration. Let's study a CI software now.
Travis CI
Travis CI is a distributed build system for the open source community. Its a CI service used to build and test projects hosted on Github.
Travis is configured by adding a file named .travis.yml which is a YAML(a human-readable data serialization format) text file.
It automatically detects when a commit has been made and pushed to a GitHub repository that is using Travis CI, and each time this happens, it will try to build the project and run tests. It also builds and runs pull requests and once it is completed it notifies the developer in the way it is configured.
Steps to use Travis CI:

Sign-in: To get started with Travis CI we should sign-in to our github account.
Activate Github webhook: Once the Signup process gets completed we need to enable the service hook in the github profile page.
Add .travis.yml: We should add the yml file to the project.

Writing .travis.yml file:
In order for Travis CI to build our project we need to tell the system a little bit about it. we will be needed to add a file named .travis.yml to the root of our repository.
The basic options in the .travis.yml should contain are language key which tells which language environment to select for our project and other options include the version of the language and scripts to run the tests, etc.
Example for travis.yml for python project:
language: python
python:
  - "2.7"

# command to install dependencies
install:
  - "pip install -r requirements.txt"

# command to run tests
script: python manage.py test

Explanation:
In the first line we have specified python as the language environment, the next specifies the version of python. Travis will then make sure that the project runs well with all the python versions. If we have any python dependencies Travis will test our code in a virtualenv, so all we need to write a requirments.txt and tell Travis to install it and the script command will run tests when excited.
The complete Lifecycle

before_install
install
after_install
before_script
script
after_script
after_success or after_failure

Configuring .travis.yml
Build only specific branches:
Per default travis will build once we push to a branch on Github. That behavior can be annoying if we are committing often to development branches. We can restrict which branches Travis is monitoring for changes in the .travis.yml. Either we can blacklist the branches or whitelist them:
.travis.yml
# Whitelisting example
branches:
  only:
    - master

# Blacklisting example
branches:
  except:
    - develop
    - feature

Getting notified with lint errors:
We can also get notified or make bulid fail if there are any lint errors in the commit. For this we can use the option before-script.
We can create a lint.sh file and we can specify it in the before-script option so that the bulid fails and notifies when lint error occurs.
.travis.yml
before_script:
    -./lint.sh

Using coveralls with travis CI:
Coveralls is web service to help us track our code coverage over time and ensure that all our new code is fully covered.
Steps to use coveralls:


First, log in via Github and add your repo on Coveralls website.


Add pip install coveralls to install section of .travis.yml


Make sure you run your tests with coverage during the build in script part.
script:
    coverage run --source=yourpackagename setup.py test


Execute run coveralls in after_success section.
after_success:
    coveralls


Example:
travis.yml
language: python
python:
  - 2.7
install:
  - pip install -r requirements.txt
  - pip install coveralls
script:
  coverage run --source=moscowdjango,meetup manage.py test
after_success:
  coveralls

The above code makes custom report for data generated by coverage.py package and sends it to json API of coveralls.io service. All python files in our coverage analysis are posted to this service along with coverage stats


Writing styled forms with absolutely minimal HTML (Part-2)
In the earlier post we have cruised through the fundamentals of using crispy forms for writing styled forms with minimal HTML.
At this time, let us dip into getting a clear understanding on how to customize the crispy forms.
Before diving into the crux of this post we will become aware with the following key words:

FormHelper
Layout

FormHelper

Django-crispy-forms  implements a class called FormHelper to define the form rendering behavior.
Helpers give us a way to control the form attributes and its layout.
They control global form rendering behavior.

Layout

It is an another powerful class of Django-crispy-forms which allow us to change the way the form fields are rendered.
This allows us to set the order of the fields, wrap them in divs or other structure, etc.
Custom output
Flexible and highly reususable

In the first place we will get to know how to render a model form and then we will be customizing the divs and fields in the template.
Handling ModelForm with a Django-crispy-form.
As we know ModelForms render Model fields as HTML, let us do this using the crispy-forms now.
Now lets render an Address form containing the fields like name, street, landmark and so on...
The basic form stays the same except that we will be appending a FormHelper class.
The snippet of forms.py:
class AddressModelForm(forms.ModelForm):
    def __init__(self, *args, **kwargs):
        super(AddressModelForm, self).__init__(*args, **kwargs)
        self.helper = FormHelper(self)

    class Meta:
        model = Address

In the above code when the AddressModelForm is called the crispy-forms builds a default layout using form.fields for us, so we don't have to manually list all the fields whatever number of fields it may contain.
Then creation of the views stays the normal way as we do for a normal model-form like this:
views.py:
    form = AddressModelForm()
In the template we just need to add the crispy form tag and that's it and our mission is accomplished.
address.html
{% load crispy_form_tags %}
{% crispy form %}

That gets us done with handling ModelForm with a Django-crispy-form.

Customizing the form
Now in the second phase we shall start customizing our form using the powerful classes of crispy-forms -- Layouts and FormHelper.
In the above example, we could just render the form fields, but we are couldn't able to assign any attributes to the form.
In the normal scenario we tend to use the form_action and form_method in the template's form attributes, but in the crispy-forms we can do it by specifying them in the form itself. For this we take the help of FormHelper.
Here is the above example rewritten to get the needed task done:
The snippet of forms.py:

class AddressModelForm(forms.ModelForm):
    def __init__(self, *args, **kwargs):
        super(AddressModelForm, self).__init__(*args, **kwargs)
        form_method = 'post'
        form_id = 'address-form-id'
        form_show_errors = True
        render_hidden_fields = False
        self.helper = FormHelper(self)

    class Meta:
        model = Address

Additionally, we have got the capability to add custom helper attributes and Bootstrap Helper attibutes:
Examples for Bootstrap helper attributes:
`help_text_inline`
`error_text_inline`
`label_class`
`field_class` and so on..

Reorder the form fields
Now it's time for learning the approach to render the form of a template in the customized way. For this we can use the Layout class.
With Layout class we can arrange the fields according to the requirement.

What if we want to reorder the fields? And what if I have 100 such fields?

Example: forms.py
class ExampleFieldForm(forms.Form):
    helper = FormHelper()
    helper.layout = Layout('name', 'email', 'password')

Power of Layouts:
The Power of Layouts becomes much more powerful with layout objects.
Every layout object has an associated template.
There are three types of layout objects.

Universal layout objects (Div, HTML, Field, Fieldset..)
Uniform layout objects (ButtonHolder, MultiField)
Bootstrap layout objects (FormActions, AppendedText, Alert...)

Using the layout objects will result in tailor made outputs in the relatively simplest way.
Example:
class RegistrationForm(forms.Form):
[...]
def __init__(self, *args, **kwargs):
    super(RegistrationForm, self).__init__(*args, **kwargs)
    self.helper = FormHelper()
    self.helper.layout = Layout(
        Fieldset(
            'Registration form',
            'name',
            'Father's name,
            'email',
           ),
        Div(
            'Address',
            'City',
            'pincode'
            css_id = 'coloured_fields'
            ),
        Button(
            Submit('submit', 'Submit'),
        HTML(" {% if success %} <p> Successfully submitted </p> {% endif %}")
        )
    )

Explaination:


The fieldset attribute renders the name, fathers's name, email fields wrapped in a fieldset and the first argument in the fieldset is rendered as a legend for the fieldset.


The Div attribute wraps the fields Address, city, pincode in a div.


Button takes name and value resulting a beautiful button.


With HTML attribute we can render HTML content and the context can be accessed from where the form is rendered




Django backward relationship lookup
I often limit the lookup to fields of the model and forget about backward relations. 
Consider the following relationship:
class Group(models.Model):
    name = models.CharField(max_length=100)

class Student(models.Model):
    name = models.CharField(max_length=100)
    group = models.ForeignKey(Group)

A group can have many students.
We want to get the groups based on certain conditions on model Student. An example is getting the groups which contain a student named 'stud1'. If you can get it using model Group and without using Student, you can skip this post.
In such a scenario, we tend to use Student model. It's more intuitive because the Foreign Key relationship exists from Student to Group.
Let's load the following data to test our queries:
group1 = Group.objects.create(name='Group 1')
group1 = Group.objects.create(name='Group 2')
Student.objects.create(name='stud1', group=group1)
Student.objects.create(name='stud2', group=group1)
Student.objects.create(name='stud3', group=group1)
Student.objects.create(name='stud1', group=group2)

Get the groups which contain student named stud1.
Intuitive way:
We first try to get all the students which satisfy the criteria. Query for that would be:
Student.objects.filter(name='stud1')

And then we try to get the groups of those students. If we use the Student model, we can't get a queryset of Group. So our approach would be to first get the ids of desired groups and then get a queryset of Group using those ids.
group_ids = Student.objects.filter(name='stud1').values_list('group', flat=True)
groups = Group.objects.filter(id__in=group_ids)

Less obvious way:
What if we could use the Group model directly?
Group.objects.filter(student__name='stud1')

This gives the exact same result as given by the Intuitive way.
So, even though there is no field called student on Group and we didn't specify any relationship to Student from Group, Django is smart enough to figure out the relationship for us.
Get the groups with name Group1 which contain students named stud1
Intuitive way:
group_ids = Student.objects.filter(name='stud1', group__name='Group1').values_list('group', flat=True)
groups = Group.objects.filter(id__in=group_ids)

Less obvious way:
groups = Group.objects.filter(name='Group1', student__name='stud1')

Gives the exact same result as given by preceding two queries.
I always missed the backward relationship while following Django tutorial or whenever I read the docs. I tried finding it while writing this post to see if the docs missed mentioning it. I was wrong yet again, there is a section which mentions it.
https://docs.djangoproject.com/en/dev/topics/db/queries/#lookups-that-span-relationships



Writing styled forms with absolutely minimal HTML (Part-1)
When controlling the form rendering, we all know that we can use the three pre-baked output format
of the forms namely: as_u, as_p and as_table.
For example: if we use: {{ registration_form.as_ul}}
The output template would be:
<li>
    <label for="id_uname">Userame:</label>
    <input type="text" name="username" id="id_uname">
<li>

<li>
    <label for="id_email">Email:</lable>
    <input type="text" name="email" id="id_email">
<li>

And so on...
Here are some questions that come into the picture:

What if we to want deal with divs?
What if we want to reorder the fields? and what if I have 100 such fields?
And what if I want to deal with model forms? How about field.errors and form.non_field_errors?


To answer all those significant questions Django-crispy-forms steps into the arena.
What are Django-crispy-forms?


It's an app that brings up the best way to get the DRY Django forms,
providing the tag and filter that lets us quickly render forms in a div format.
This also provides an enormous amount of capability to configure and control the rendered HTML.


As an added asset Django-crispy-forms support several frontend frameworks like Twitter Bootstrap(versions- 2 and 3),
Uniform and Foundation.


Using Django-crispy-forms is very simple and effortless.


Let's dive in to learn to use the Django-crispy-forms with Twitter Bootstrap in straightforward 5 steps:
(We assume that you have set up the minimal Django configuration before following the below steps)


Step1: Installing using pip.
pip install django-crispy-forms


Step-2: Add crispy_forms to the INSTALLED_APPS in settings.py.
INSTALLED_APPS = (
    ....
    'crispy_forms',
    )


Step-3: Set up a default template pack in the settings.py for the project (I assume Bootstrap3 here)
CRISPY_TEMPLATE_PACK = 'bootstrap3'


Step-4: Create your desired custom form and view.


Eg: forms.py
class SimpleForm(forms.Form):
    user_name = forms.CharField(label='Username', required=True)
    ..........

Eg: views.py
class Index1View(FormView):
    template_name = 'crispy_example/index.html'
    form_class = SimpleForm


Step-5: Loading crispy forms tag in the template

In the template, add the below template tags:
1. {% load crispy_forms_tags %}
2. {% crispy form %}

Add the first tag in the initial part to the template and the second tag in the place where you want to render the form.
That's all, the above steps results you an impressive bootstrap form.
You can have a look at the complete example that I have created in github over here: Link


Minimal Nginx and Gunicorn configuration for Django projects
We will see how to deploy a Django project with nginx and gunicorn.
It was easier for me to understand nginx and gunicorn on development machine and then move to a publicly accessible server. So, we will cover this post in a similar sequence:
On development machine

Use django's runserver to serve the site.
Stop runserver and start using gunicorn instead.
Add nginx to this configuration.

On server

Deploy the project on a publicly accessible server using same stack.



Use runserver
We will use a minimal project with two apps and few static files. You can view this project at github.
Project structure:
|-- manage.py
|-- other_app
|   |-- __init__.py
|   |-- models.py
|   |-- static
|   |   `-- other_app
|   |       `-- styles.css
|   |-- tests.py
|   |-- urls.py
|   `-- views.py
|-- project_static
|   `-- base.css
|-- requirements.txt
|-- some_app
|   |-- __init__.py
|   |-- models.py
|   |-- static
|   |   `-- some_app
|   |       `-- styles.css
|   |-- tests.py
|   |-- urls.py
|   `-- views.py
|-- templates
|   |-- other_app
|   |   `-- home.html
|   `-- some_app
|       `-- home.html
|-- test.db
`-- test_project
    |-- __init__.py
    |-- settings.py
    |-- test_settings.py
    |-- urls.py
    `-- wsgi.py



This project was started using django-admin.py startproject test_project, so this structure is inside folder test_project.
Cloning and trying it out would help you follow along, so go ahead and clone this project.
Our assumption is that we are working from the same level as manage.py.
Runserver and verify that you can access
http://localhost:8000/some_app/home
http://localhost:8000/other_app/home



First url should have a red background and the second one should have a blue background. 
This project uses some static resources, you should check our post on serving static file if you are confused about serving static files during development.
Stop runserver and start using Gunicorn
Stop the Django development server.
Make sure you have gunicorn installed, else install it with 
pip install gunicorn



Gunicorn is a WSGI compliant server and you need to pass your application object to it. You should read about Basics of WSGI if you are not comfortable with wsgi.
Django provides an application object. The default structure provided by django-admin startproject gives a wsgi.py file which contains the application. You need to pass this application to gunicorn.
Run gunicorn passing it the application:
gunicorn test_project.wsgi:application



Note: In the recent django versions the wsgi file extension has been changed it is a python module(ex: test_project_wsgi.py) and so you can serve it with just module name like below
gunicorn test_project:application



Refresh your page at http://localhost:8000/some_app/home. Your page should be served, however the red background must have vanished.
Red background vanished because the static files(stylesheets) are not being served anymore. They are not being served because we do not have any url pattern for /static/ in urls.py. Earlier they were served because we were using Django development server, i.e runserver, which does some magical things behind the scence to serve static files.
Add nginx to this configuration
Gunicorn is meant to serve dynamic content, it should not be used to serve static files. We will add nginx to serve static files.
We want to serve static files from port 8000 and so it is required that gunicorn listens on some different port. Stop gunicorn and run it on port 8001.
gunicorn test_project.wsgi:application --bind=127.0.0.1:8001



Now you will not be able to see your page at http://localhost:8000/some_app/home/. It will be available on port 8001 at http://localhost:8001/some_app/home.
Make sure you have nginx installed or install it with:
sudo apt-get install nginx



Nginx acts as a reverse proxy here. All our request will initially come to nginx. It's for nginx to decide what requests it wants to serve and what requests it wants to pass to some other server.
In our case, we want nginx to serve requests for static resources and to pass any other request to gunicorn.
We need to tell nginx the location of our static resources and for that we need to make sure all our static resources are at a single location. Run python manage.py collectstatic to collect all the static resources at location specified by STATIC_ROOT.
Set STATIC_ROOT in settings.py:
STATIC_ROOT = os.path.join(PROJECT_DIR, '../staticfiles/')



Run collectstatic:
python manage.py collectstatic



You should see that a directory staticfiles gets created at the same level as manage.py. All your static files must have got copied to this directory. Get the path of this directory, we need it in the alias directive of nginx cofiguration.
Create a file /etc/nginx/sites-enabled/example and add following content to it.
server {
    listen localhost:8000;

    location / {
        proxy_pass http://127.0.0.1:8001;
    }

    location /static/ {
        autoindex on;
        alias /home/akshar/staticvirt/test_project/staticfiles/;
    }
}



Your alias directive will differ from mine.
Make sure nginx is running:
sudo service nginx restart



Nginx is listening for requests now.
Refresh page at http://localhost:8000/some_app/home/, the page should be visible with red background which conforms that static files are being served properly.
Explanation:

listen directive tells that nginx is listening for any request that comes at localhost:8000.
There are two location directives.
Bottom location can overide top location. Bottom will have preference over top.
When a request starting with / comes, its being passed to port 8001.

eg: If request comes for http://localhost:8000/some_app/home, nginx tries to match it with one of the location defined in the configuration file. In this case, it matches with first location. Nginx sees that a proxy_pass is defined in this case so it passes this request to the proxy_pass which is http://127.0.0.1:8001. Gunicorn is listening at port 8001, so gunicorn will repond to this request.

When a request starting with /static/ comes, bottom location is used as it has preference over the top location.

When a request comes for http://localhost:8000/static/some_app/styles.css, Nginx looks into the directory pointed to by alias which is staticfiles. It tries to find some_app/styles.css inside this directory and if this file is available then serves the file.
Now we are comfortable with serving django sites with nginx and gunicorn.
Deploy on a publicly accessible server.
I will use domain pythoninternals.com for illustration. We need to do the following things on the server where A record of domain pythoninternals.com points.
We don't need any change for gunicorn and can run it in the same way:
gunicorn test_project.wsgi:application --bind=127.0.0.1:8001



Create file /etc/nginx/sites-enabled/example on the server and add content:
server {
  listen 80; 
  server_name pythoninternals.com;
  location / { 
    proxy_pass http://127.0.0.1:8001;
  }

  location /static/ {
    alias /home/ubuntu/staticvirt/test_project/staticfiles/;
  }
}



This file is almost similar to the nginx conf we had on development machine.
Make sure that you have collected the static files in directory staticfiles.
Some differences in this nginx conf and dev machine's nginx conf are:

It's listening on port 80. When a request is made for domain pythoninternal.com, it comes on the default port which is 80. So, nginx must be listening for any request on port 80.
server_name makes sure that this configuration file will only be used for pythoninternals.com.
There might be another domain called abcde.com being served from this same server, we don't want this configuration to be used for abcde.com. That's why we specify the server_name.

After this a request for pythoninternals.com/some_app/home would be served properly from this server.
Running gunicorn the way we did, will keep it in the foreground and we will have to stop gunicorn to exit from the server. So, we need to run it as a daemon.
Run gunicorn as daemon:
gunicorn test_project.wsgi:application --bind=127.0.0.1:8001 --daemon



With this, gunicorn runs as a background process and we can quit from the server without affecting gunicorn.
Till now, we have set various configurations for gunicorn, like --bind and --daemon, on the terminal. The suggested way to do it is using configuration file. You can read about it at another post we wrote and should move these configurations to a separate file.


Writing thread-safe django - get_or_create
In this blog post, we'll discuss thread-safety, why it's important and how to
write thread-safe django code, especially for bulk operations like management
commands. We'll take a simple example - get or create.
Thread-safety:
Thread-safety means that our code can be run in multiple threads and behave as
expected. The reason that code can be unsafe with regard to threads is because
we'll be manipulating shared memory (e.g. database) from the threads and there's
a chance of a race-condition which will produce unexpected results.
To avoid this, we have the option of using read-write locks, transactions etc.
We'll look at some simple examples and try to understand these options.
The usual way:
Let's consider a management command that syncs data from another source (e.g. API,
remote database etc.. The correct way to do this would be to use the built-in
django utility - get_or_create:
Update: Updated the command to run each arg in a thread
class MyThread(Thread):

    def __init__(self, my_id):
        super(MyThread, self).__init__(name=my_id)
        self.my_id = my_id

    def run(self):
        instance, created = MyModel.objects.get_or_create(my_id=my_id)
        print '%s %s' % (instance.id, created)
        instance.delete()
        return


class Command(BaseCommand):
    args = '<my_id my_id ...>'
    help = 'Get or create instace of mymodel with my_id'

    def handle(self, *args, **options):
        for my_id in args:
            thread = MyThread(my_id=my_id)
            thread.start()

In this command, we'll be using the command line arg my_id get a MyModel
instance if it exists, else, we create one.
Note: we'll be discarding the object for simplicity.
Now, this management command (let's call it sync_myapp) is thread-safe because
we're just using the built-in get_or_create to do our database call.
To test this, run the command with same arg repeated multiple times:
$> python manage.py sync_myapp 1 1 1 1 1

This will create five threads which will simultaneously try to get or create an instance
with my_id = 1
$> 1 True
$> 1 False
$> 1 False
$> 1 False

Note that all the threads are successful, even though they were working on the
same database at the same time.
Now, even though get_or_create is the way to go, we might need to customize a
few things which are outside the scope of get_or_create. For example, let's
say we need to do something special just before creating a new instance.
The problem:
Let's assume our code was:
def run(self):
    created = False
    try:
        instance = MyModel.objects.get(my_id=my_id)
    except MyModel.DoesNotExist:
        something_special()
        instance = MyModel.objects.create(my_id=my_id)
        created = True
    instance.delete()
    return

If we try to test it with the above command, you'll probably get:
Exception in thread 1:
Traceback (most recent call last):
....
IntegrityError: duplicate key value violates unique constraint "myapp_mymodel_my_id_key"
DETAIL:  Key (my_id)=(1) already exists.

This indicates that the try except block isn't thread-safe.
Let's try to fix this problem.
Update: I've removed the section dealing with locks since it's only useful when
dealing with shared memory in python processes, it's not applicable to
databases.
Using database transactions:
We use django's built-in transactions module as follows:
Update: We only need to wrap the create command in a transaction
def run(self):
    created = False
    try:
        instance = MyModel.objects.get(my_id=self.my_id)
    except MyModel.DoesNotExist:
            try:
                something_special()
                with transaction.commit_on_success():
                    instance = MyModel.objects.create(my_id=self.my_id)
                created = True
            except IntegrityError:
                instance = MyModel.objects.get(my_id=self.my_id)
    print '%s %s' % (instance.id, created)
    instance.delete()
    return

Now, the transaction will be committed only if there's no error within the
context block, so we can be sure that only one thread gets the go-ahead for
create call.
In addition to the context manager, django also has options for using savepoints,
manually commits, rollbacks etc.
https://docs.djangoproject.com/en/1.5/topics/db/transactions
Caveats:
If you're using MySQL, refer to this open issue on problem with get_or_create:
https://code.djangoproject.com/ticket/13906
Conclusion:
Using database transactions, we can avoid data integrity issues and write
thread-safe code which can be run in parallel without any
issues.


Deploying django using docker
In this blog post we'll take a look at Docker and
discuss ways to deploy django using docker.
About Docker:
Docker is a tool that helps you manage lxc containers and images. You can
create an image, launch container from the existing image, attach to a running
container, and generally play around with the containers.
The most important benefit of using docker and containers in general is that
you have a clean, hygeinic and portable runtime enivorment for your app. This
means you don't have to worry about missing dependecies, packages and other
pain points during subsequent deployments. You can take a snapshot of a running
container and restore it again when required. Plus, each app runs in it's own
isolated container so you can have various versions of libraries and other
dependecies for each app without worrying about it's effect on other apps.
Coming from the python world where virtualenv is ubiqitous, it's a huge relief
to me that there's such a better and cleaner solution.

Note: Docker is still under heavy development. It should not yet be used in
 production. Check the repo for recent progress.

To know more about docker and install it, go to the "Getting Started" guide.
Here I'll discuss two ways of deploying django using docker:
Vanilla deployment:
Nothing fancy here, we just instantiate a container image and open up an ssh
port and a web port, then we can use our usual fabfile script to deploy it to
the container. Here I'll use the container image "dhrp/sshd" which I found by
searching the container index.
> docker pull dhrp/sshd



This will fetch the container image dhrp/sshd
Now let's run the container -d for daemonize, -p 22 -p 8000 for telling docker
to open port 22 and 8000, dhrp/sshd is the image we want to use, and /usr/sbin/sshd -D
is the command we want to run inside the container - here we want to run the sshd daemon
> docker run -d -p 22 -p 8000 dhrp/sshd /usr/sbin/sshd -D
>> c4cee8e86fa0



The above command will give an hash by which you can refer to the running
container instance in the future.
Now you can query for the NATed ports associated with our newly launched
container using:
> docker port c4cee8e86fa0 22
>> 49185

> docker port c4cee8e86fa0 8000
>> 49186



or you can also use the generic docker ps command for a human readable and
prettier output:
> docker ps



Now, we can ssh into our container through the NATed port - the password for
this container is screencast:
> ssh root@localhost -p 49185
>> root@c4cee8e86fa0:/#



Note that the password - screencast was set by the container image author.
For better security, you should login, change the password and commit your
changes immediately:
> docker commit c4cee8e86fa0 dhrp/sshd



Now, you can use this container just like any other server. You'll just need to
make sure to proxy the NATed web port through nginx. You can update your
fabfile to point to the container by updating the ssh port number and deploy as
usual.
Heroku-like deployment:
A few days back I came across dokku and
decided to give it a try by deploying a small django app. You'll need to clone
the dokku repository and use a buildpack like
heroku-buildpack-django as described
in the dokku docs. The
django buildpack I found worked quite well, except that it requires python-dev to be installed.
Luckily, it's quite easy to modify the buildstep script to handle this. Once I had this set-up,
I just had to create a tarball of my code and build it:
cat django-pastebin.tar | ./buildstep django-pastebin



This will build a container from scratch and install all dependecies required
for the buildpack and prepare the container for deployment.
Here are the repos I've used for this example:

https://github.com/tuxcanfly/buildstep
https://github.com/tuxcanfly/heroku-buildpack-django
https://github.com/agiliq/django-pastebin

Verify that it works:
> docker run -p 8000 django-pastebin /start web
>> 2013-06-14 13:31:22 [9] [INFO] Starting gunicorn 0.17.4
>> 2013-06-14 13:31:22 [9] [INFO] Listening at: http://127.0.0.1:8000 (9)
>> 2013-06-14 13:31:22 [9] [INFO] Using worker: sync
>> 2013-06-14 13:31:22 [14] [INFO] Booting worker with pid: 14



Now, you can daemonize it and let it run in the background:
> docker run -d -p 8000 django-pastebin /start web
>> 4dc243483b56



Conclusion:
I've found docker and containers very useful in managing multiple apps on a
single server even if it's a little rough around the edges. Tools like
container index are a great way of sharing
development enviroments and avoid getting bogged down by installation
processes. I'm eagerly looking forward to future versions and upcoming features
in docker.


Common testing scenarios for Django app.
People are often confused regarding what tests to write. Let's look into some scenarios for which tests can be written.
Setting up the project
We start a Django project inside a virtual environment. In this post we would be using django 1.4.
(dj)~/play/dj$ django-admin.py startproject testcases



Let's start an app named blog.
(dj)~/play/dj/testcases$ python manage.py startapp blog



We will have the following model in blog/models.py:
class BlogEntry(models.Model):
    title = models.CharField(max_length=100)
    text = models.TextField()
    is_published = models.BooleanField(default=True)
    user = models.ForeignKey(User)



We will do test driven development which requires:

Thinking about our assumption.
Writing the test to satisfy that assumption.
Run the test and it will fail since we won't have view written till this point.
Adding the view.
Run the test and fixing the view or anything else till our test passes.

If I could not explain the project structure properly, you can find the complete project here.
First test
We want a page which shows all blog entries at url /blog/entries/.
We need following line in urls i.e testcases/urls.py
url(r'^blog/', include('blog.urls')),



blog/urls.py
url(r'^entries/$', 'blog.views.entries', name='entries'),



Let's add a test which satisfies our assumption.
Every app we create gets a tests.py where we can put our tests. You can remove the simple addition test generated by default by Django.
blog/tests.py
from django.test import TestCase
from django.test.client import Client

class BlogEntriesTest(TestCase):

    def setUp(self):
        self.c = Client()

    def test_entries_access(self):
        response = self.c.get('/blog/entries/')
        self.assertEqual(response.status_code, 200)



Explanation

We need a test class which is BlogEntriesTest. Any test class we write must subclass TestCase which is defined in django.test.
Actual tests go in methods defined on the test class. So, our test goes in test_entries_access. Every test method name must start with test for it to be found by the django test runner.
Before every test, setUp method is run. So anything which is common on all the test methods can go in setUp.
We created a Client object in setUp. This object is needed to make GET or POST request. Client object simulates request to a url similar to a browser can. Since it is created in setUp, it will be available in all the test methods.
From test_entries_access, we make a GET request to the url which we have defined. We capture the response in variable response.
We are asserting that status_code of response must be 200 since we assumed this url to return a 200.

Running the test
Tests use a database. Since sqlite is faster than mysql or postgresql, we would use sqlite as the test database. You can continue using your any other database for development or production. We are only interested in running the tests using sqlite.
testcases/test_settings.py
from settings import *
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'abc'
     }
}



Running the test. 
python manage.py test blog --settings=testcases.test_settings



blog says that only run the tests for app blog. Not providing this would run the tests for all the apps defined in INSTALLED_APPS which would take quite some time. Make sure that you have added blog to INSTALLED_APPS.
--settings flag tells django test runner to use the specified settings file.
And our test fails.
ERROR: test_entries_access (blog.tests.BlogEntriesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
......
......
ViewDoesNotExist: Could not import blog.views.entries. View does not exist in module blog.views.

----------------------------------------------------------------------
Ran 1 test in 0.019s

FAILED (errors=1)



Because we have not written the view yet. We didn't have to go to browser to check if this url works, our test does it for us. 
We should test every url we want available in our project. So, that if some url breaks our test would tell that to us immediately.
Adding the view
from blog.models import BlogEntry
from django.shortcuts import render

def entries(request):
    entries = BlogEntry.objects.all()
    return render("blog/entries.html", {'entries': entries})



Make sure to add the template blog/entries.html in your template directory.
Run the test again.
python manage.py test blog --settings=testcases.test_settings

Ran 1 test in 0.019s

OK
Destroying test database for alias 'default'...



Minor modification to test
Since we should not hardcode the urls, we will use reverse.
def test_entries_access(self):
    response = self.c.get(reverse('entries'))
    self.assertEqual(response.status_code, 200)



Run the test again to make sure our change has not broken anything.
Test that context is populated in template
We want to make sure that our expectation regarding number of visible blog entries on the page matches the result. Say we have two entries in the database then both of them should be shown on the page as per our view definition.
A blog must be associated with a user. So, we will create an instance of user in setUp because we would need this user in other tests as well.
def setUp(self):
    self.c = Client()
    self.user = User.objects.create_user(username="test", email="test@test.com", password="test")



Testing the context
def test_entries_template_context(self):
    #create few blog entries
    BlogEntry.objects.create(title='Test', text='Test', user=self.user)
    BlogEntry.objects.create(title='Test', text='Test', user=self.user)

    response = self.c.get(reverse('entries'))
    #assert that context contains as many entries as you expect
    self.assertEqual(len(response.context['entries']), 2)



response contains an attribute context which is a dictionary containing the context sent to template.
Assertion

We created two blogs and asserted that context contains both of them.

This test is not much useful now. We will see how response.context becomes useful when we write a custom manager.
Only logged in user must access create entry page.
def test_entry_create(self):
    response = self.c.get(reverse('entry_create'))
    self.assertEqual(response.status_code, 302)

    self.c.login(username='test', password='test')
    response = self.c.get(reverse('entry_create'))
    self.assertEqual(response.status_code, 200)



Run the test. We know it will fail because we have not written the view or the url yet. 
python manage.py test blog --settings=testcases.test_settings

.....
ERROR: test_entry_create (blog.tests.BlogEntriesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
....
NoReverseMatch: Reverse for 'entry_create' with arguments '()' and keyword arguments '{}' not found.
----------------------------------------------------------------------
Ran 3 tests in 0.051s

FAILED (errors=1)



Explanation:

We expect a url with name entry_create to be available.
A non-logged user should not be able to access this url and should be redirected to login urll. So, we assert that status code be 302.
There is a login method defined on Client. So, we can call self.c.login().
login() takes a username and password. You should provide credentials of some user already present in db. Remember we created an user in setUp. We pass the same credentials here.
After this client i.e self.c starts behaving like a logged in user.
Now client should be able to access create page and so we assert the status code as 200.

Let's start fixing this test.
Need to add following in blog/urls.py
url(r'^entry/create/$', 'blog.views.entry_create', name='entry_create'),



Adding a modelform. This will be used in create view.
class BlogEntryForm(ModelForm):                                                                          
    class Meta:
        model = BlogEntry



Adding the view
def entry_create(request):
    form = BlogEntryForm()
    if request.method == "POST":
        form = BlogEntryForm(request.POST)
        if form.is_valid():
            return HttpResponseRedirect(reverse('entries'))
            form.save()
    return render(request, "blog/entry_create.html", {'form': form})



Make sure to create the template. And then run the test. 
You will still see an error.
File "/home/akshar/play/dj/testcases/blog/tests.py", line 30, in test_entry_create
    self.assertEqual(response.status_code, 302)
AssertionError: 200 != 302



Because we missed adding login_required to the view and even anonymous users are able to access this url. And hence they are getting a 200 instead of 302.
Let's fix it by adding login_required decorator to entry_create view.
Run the test again and it should pass now.
Test invalid form
So, we wrote entry_create with assumption that it will handle POST requests.
We want to make sure that this view doesn't allow invalid POST and raises an exception in that case.
def test_invalid_entry_create(self):
    self.c.login(username='test', password='test')
    data = {'text': 'Test text'}
    response = self.c.post(reverse('entry_create'), data)
    self.assertEqual(response.status_code, 200)
    self.assertFormError(response, "form", "title", "This field is required.")



Assertions:

Since we posted an invalid form, we expect to remain on the same page. So asserted for status code of 200.
We expect an error to be present on the title field.

Test valid form
def test_valid_entry_create(self):
    self.c.login(username='test', password='test')
    data = {'text': 'Test text', 'title': 'Test title'}
    data['user'] = self.user.id
    self.assertEqual(BlogEntry.objects.count(), 0)
    response = self.c.post(reverse('entry_create'), data)
    self.assertEqual(response.status_code, 302)
    self.assertEqual(BlogEntry.objects.count(), 1)



Assertions:

Before posting we assert that there is no BlogEntry in the db.
After posting we check that the user is redirected and so asserted for status code of 302
We make sure that a BlogEntry is created in the database on post by checking that count of BlogEntry has been increased to 1.

Test custom manager methods
Suppose you find yourself writing the same filter multiple times for getting the blog entries which have is_published as True. In that case you would write a custom manager.
We will add the custom manager in models.py
class PublishedBlogManager(models.Manager):
    def get_query_set(self, *args, **kwargs):
        return super(PublishedBlogManager, self).get_query_set(*args, **kwargs).filter(is_published=True)



Also we need to add this manager on BlogEntry. So, don't forget to add next two lines to BlogEntry
objects = models.Manager()
published = PublishedBlogManager()



Let's write a test now:
def test_entry_custom_managers(self):
    BlogEntry.objects.create(title='Test', text='Test', user=self.user, is_published=False)
    BlogEntry.objects.create(title='Test', text='Test', user=self.user)
    self.assertEqual(BlogEntry.objects.count(), 2)
    self.assertEqual(BlogEntry.published.count(), 1)



Assertions:

We created two entries. One with is_published as False, and another with True.
objects i.e default manager returns all the entries.
published i.e custom manager returns only entries which have is_published=True.

Using custom manager in test_entries_template_context:
Say now we decide that all entries should not be shown on list entries page. Only published entries should be shown.
Remember test_entries_template_context. We only created two blog entries in that test. Edit that test and create one more entry with is_published=False
def test_entries_template_context(self):
    #create few blog entries
    BlogEntry.objects.create(title='Test', text='Test', user=self.user)
    BlogEntry.objects.create(title='Test', text='Test', user=self.user)
    BlogEntry.objects.create(title='Test', text='Test', user=self.user, is_published=False)

    response = self.c.get(reverse('entries'))
    #assert that context contains only published entries
    self.assertEqual(len(response.context['entries']), 2)



We created three entries. Only two of them are published.
Assertion:

Entries page should only show 2 entries.

Run the test and it will fail.
======================================================================
FAIL: test_entries_template_context (blog.tests.BlogEntriesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/akshar/play/dj/testcases/blog/tests.py", line 29, in test_entries_template_context
    self.assertEqual(len(response.context['entries']), 2)
AssertionError: 3 != 2

----------------------------------------------------------------------



Happened because we forgot to change our view. Our view still says BlogEntry.objects.all(). We should change it to BlogEntry.published.all(). Make this change and the test will pass.
def entries(request):
    entries = BlogEntry.published.all()
    return render_to_response("blog/entries.html", {'entries': entries})



So, response.context helped us find that we do not show more entries than we should.
Test for pagination
Suppose we expect maximum ten entries to be available on each page. There are 15 entries in the db, so first page should show 10 entries and the second page should show 5. If user tries to access the third page a 404 page should be shown.
Our decided url pattern for getting entries on a particular page:
url(r'^entries/(?P<page>\d+)/$', 'blog.views.entries_page', name='entries_page')



Writing the test:
def test_entries_page(self):
    for i in range(15):
        BlogEntry.objects.create(title="title", text="text", user=self.user, is_published=True)

    #access first page
    response = self.c.get(reverse("entries_page", args=[1,]))
    self.assertEqual(response.status_code, 200)
    self.assertEqual(len(response.context['entries']), 10)

    #access second page
    response = self.c.get(reverse("entries_page", args=[2,]))
    self.assertEqual(response.status_code, 200)
    self.assertEqual(len(response.context['entries']), 5)



Run the test and it will fail. Let's add view to make it pass.
def entries_page(request, page):
    page = int(page)
    entries = BlogEntry.published.all()
    paginator = Paginator(entries, 10) #10 entries per page
    page_ = paginator.page(page)
    object_list = page_.object_list
    return render(request, "blog/entries_page.html", {"entries": object_list})



Test should pass now provided you have added the template.
Let's try to access third page in the test. We need to add following in test_entries_page for that.
    #access third page
    response = self.c.get(reverse("entries_page", args=[3,]))
    self.assertEqual(response.status_code, 404)



Running the test raises an error.
....
raise EmptyPage('That page contains no results')
EmptyPage: That page contains no results



We find that there is a bug in our view and any page which doesn't contain entries is not being handled as we want. Let's change our view:
def entries_page(request, page):
    page = int(page)
    entries = BlogEntry.published.all()
    paginator = Paginator(entries, 10) #10 entries per page
    if page > paginator.num_pages:
        raise Http404()
    page_ = paginator.page(page)
    object_list = page_.object_list
    return render(request, "blog/entries_page.html", {"entries": object_list})



Run the test again.
If you have a 404 template defined then your test will pass. In this project we do not have a 404 template and so we get another exception
ERROR: test_entries_page (blog.tests.BlogEntriesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
....
    raise TemplateDoesNotExist(name)
TemplateDoesNotExist: 404.html



So, let's add a 404 template. Test passes once we do it.
So, this test also helped us find a missing 404 template.
Test for static files
You can look at our blog on static file if you have some confusion regarding how static files are served in Django.
We will create a static directory in our app blog and will put an image in this directory. Let's say this image is default.jpg
Let's add a test to make sure this image is found by django static file handlers. This test makes us confident that we can expect this image to be served at {{STATIC_URL}}default.jpg
from django.contrib.staticfiles import finders
from django.contrib.staticfiles.storage import staticfiles_storage

def test_images(self):
    abs_path = finders.find('default.jpg')
    self.assertTrue(staticfiles_storage.exists(abs_path))



Run the test and it should pass.
You can view the complete project here.


Logging in Django
This is a talk I gave at Agiliq on 21-March-2013 as a part of the Hyderabad Python Meetup Group




Serving static files in Django
Serving static files, especially during development, is frequently a pain point. In this post, we will discuss the various settings and directory structure and how they interact with each other. Let's start with development when you have DEBUG = True.
We will create a Django project from scratch so that we are fully aware about what file in what directory we are talking about. So, this blog post will be a little longer than it should be. We will try these things with Django 1.4, but these will work on Django 1.3 as well. I have not tested it with Django 1.2, so not sure about 1.2.
Setting up the project.
If you think you don't need this section, then skip to the next section titled Serving static files. Just make sure to see the directory structure at the end of this section to get an idea of it, so that you will be comfortable in the next section.
We will try everything inside a virtual environment. We name our virtual environment as staticvirt. So the command we need is virtualenv staticvirt.
~$ virtualenv staticvirt



Next we need to start a Django project inside this virtualenv. Make sure you are inside the virtualenv and have activated it. Also make sure that you install Django inside your virtual environment as we don't want to pollute your system wide site packages.
~$ cd staticvirt/
~/staticvirt$ source bin/activate
(staticvirt)~/staticvirt$ pip install django==1.4



Start a Django project.
django-admin.py startproject test_project



Change to directory containing the project.
cd test_project/



Let's see the directory structure at this point.
(staticvirt)~/staticvirt/test_project$ tree
.
|-- manage.py
`-- test_project
    |-- __init__.py
    |-- settings.py
    |-- urls.py
    `-- wsgi.py

1 directory, 5 files



Check the contents of test_project/settings.py. Search for all the lines which contain the substring static. I will list all those lines below.
STATIC_ROOT = ''

STATIC_URL = '/static/'

STATICFILES_DIRS = (
)

STATICFILES_FINDERS = ( 
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
#    'django.contrib.staticfiles.finders.DefaultStorageFinder',
)

INSTALLED_APPS = ( 
    ....
    ....
    'django.contrib.staticfiles',
    ....
)



Whatever we saw till here is the default setting provided to us by Django. We have not made any modifications.
Let's create an app in which we will have a template and then we will write a static file, which will be a stylesheet, and will use that stylesheet in this template.
python manage.py startapp some_app



Add some_app to INSTALLED_APPS in test_project/settings.py.
We need a urls.py file for some_app where we will define the urls available on some_app. Project's urls.py should include the urls.py of some_app. So, we add the following line to test_project/urls.py.
url(r'^some_app/', include('some_app.urls'))



We have following line in urls.py of some_app i.e in some_app/urls.py.
url(r'^home$', direct_to_template, {"template": "some_app/home.html"})



Create a directory called templates and add it to TEMPLATE_DIRS. I created this templates directory on the same level as manage.py.
Adding templates directory to TEMPLATE_DIRS require following changes for me. It will be same if you use the same directory structure as I am using.
PROJECT_DIR = os.path.dirname(__file__)

TEMPLATE_DIRS = ( 
    os.path.join(PROJECT_DIR, '../templates'),
)



We need to create home.html for app some_app and it needs to go into templates. So, we create a template file templates/some_app/home.html. We have the following content in the html file.
<html>
    <body>
        <h1>This is home for some_app</h1>
    </body>
</html>



Let's check our project directory structure, so as to remove any confusion.
~/staticvirt/test_project$ tree -I *.pyc
.
|-- manage.py
|-- some_app
|   |-- __init__.py
|   |-- models.py
|   |-- tests.py
|   |-- urls.py
|   `-- views.py
|-- templates
|   `-- some_app
|       `-- home.html
`-- test_project
    |-- __init__.py
    |-- settings.py
    |-- urls.py
    `-- wsgi.py

4 directories, 11 files



We did not want .pyc files to be shown and so used a switch with tree to ignore .pyc files.
Start the server. Make sure you have added your database settings.
(staticvirt)~/staticvirt/test_project$ python manage.py runserver



Navigate to http://127.0.0.1:8000/some_app/home. Hereafter, we will call this page as home page of some_app. You should see the html we just wrote.

Serving static files.
Let's edit home.html of some_app and add a stylesheet to it. There doesn't exist any stylesheet yet, we will create it after editing html.
<html>
    <head>
        <link href="{{STATIC_URL}}styles.css" rel="stylesheet" type="text/css">
    </head>
    <body>
        <h1>This is home for some_app</h1>
    </body>
</html>



Refresh home page of some_app. You will not see any changes yet because we have not created the stylesheet file.
Also navigate to http://127.0.0.1:8000/static/styles.css, you will see a 404 page.
Static files should be created inside the static/ subdirectory of apps
Let's create the stylesheet file. Since we want to use this stylesheet in the template of some_app, we will create it inside the static/ subdirectory of some_app.
So, let's create a file some_app/static/styles.css. Add following contents to this file.
body
{
    background-color: red;
}



Refresh home page of some_app. You should see that the background of this page becomes red. Also, refresh page at http://127.0.0.1:8000/static/styles.css. You will see that it's no more a 404 page and you will see the contents of your stylesheet at this url. If you can't see these changes, make sure that some_app is added to INSTALLED_APPS and restart the server.
Points to be noted

We did not make any changes to any of the static settings provided by Django to us. We left the static settings as they were in default settings.py provided by Django.
You don't need any change in your urls.py for serving your static files in development. You don't need to add staticfiles_urlpatterns(). Many a times, I got confused with this.
You do not need python manage.py collectstatic for serving static files in development.



How it works internally

First, check all the static settings we have in settings.py
They are STATIC_URL, STATIC_ROOT, STATICFILES_FINDERS, STATICFILES_DIRS.
Also, we have 'django.contrib.staticfiles' in INSTALLED_APPS.
For now, forget about STATIC_ROOT and STATICFILES_DIRS. Even if you comment it or delete it from settings, your project will continue to work as it is working currently.
We need 'django.contrib.staticfiles' in our INSTALLED_APPS if we want Django's default server to serve static files.
By Django's default server, we mean python manage.py runserver, that's provided with Django.

Django's default server will serve the static files at STATIC_URL. Notice that STATIC_URL is set to '/static/'. 
That's why we have our static file, ie stylesheet, getting served at http://127.0.0.1:8000/static/styles.css. If you try to get it at http://127.0.0.1:8000/static_changed/styles.css, you will not be able to, you will get a 404 instead.
If you want to get the stylesheet at http://127.0.0.1:8000/static_changed/styles.css, make STATIC_URL='/static_changed/'. Go ahead make this change and check. With this change, static files will get served at /static_changed/.
This change was only to illustrate what is the purpose of STATIC_URL. Change it back to its default, i.e STATIC_URL='/static/'. 


Next question is how does Django know from where to read the static files or where to find the static files. That's where STATICFILES_FINDERS comes into picture.
Check that we have two entries in STATICFILES_FINDERS, i.e 'django.contrib.staticfiles.finders.FileSystemFinder' and 'django.contrib.staticfiles.finders.AppDirectoriesFinder'. You can ignore FileSystemFinder for now, if you wish you can go ahead and comment the FileSystemFinder line.
Appdirectoriesfinder tells Django to look into static/ subdirectory of each app in INSTALLED_APPS while looking for static files. Remember, we have styles.css under static/ subdirectory of some_app. That's why Django was able to find it and it got served properly by Django server. 
If you change the name of this subdirectory from 'static/' to something else, your static resources will not be served. Go ahead, try it.
Try commenting Appdirectoriesfinder line and your css file at http://127.0.0.1:8000/static/styles.css will not be served anymore. Ok, uncomment it back after trying this out.


Now, we are fine with what STATIC_URL and STATICFILES_FINDERS does. We still do not need STATIC_ROOT and STATICFILES_DIRS.
To understand few other things about serving static files, we need one more app.
Let's create it.
python manage.py startapp other_app



Edit project urls.py to include the url for other_app. Now project's urls.py i.e test_project/urls.py contains two lines.
url(r'^some_app/', include('some_app.urls')),
url(r'^other_app/', include('other_app.urls')),



We need following line in other_app's urls.py i.e in other_app/urls.py
url(r'^home$', direct_to_template, {"template": "other_app/home.html"})



Let's create other_app/home.html inside templates/ directory.
<html>
    <body>
        <h1>This is home for other_app</h1>
    </body>
</html>



Check the directory structure at this point.
~/staticvirt/test_project$ tree -I *.pyc
.
|-- manage.py
|-- other_app
|   |-- __init__.py
|   |-- models.py
|   |-- tests.py
|   |-- urls.py
|   `-- views.py
|-- some_app
|   |-- __init__.py
|   |-- models.py
|   |-- static
|   |   `-- styles.css
|   |-- tests.py
|   |-- urls.py
|   `-- views.py
|-- templates
|   |-- other_app
|   |   `-- home.html
|   `-- some_app
|       `-- home.html
`-- test_project
    |-- __init__.py
    |-- settings.py
    |-- urls.py
    `-- wsgi.py



Add other_app to INSTALLED_APPS.
Now check the url http://127.0.0.1:8000/other_app/home.
Let's add a stylesheet to home of other_app. Let's assume we want to make background color of other_app's home as blue. For this we create the stylesheet other_app/static/other_style.css
body
{
    background-color: blue;
}



Now add this stylesheet to other_app's home template. So, we edit templates/other_app/home.html and change its content to
<html>
    <head>
        <link href="{{STATIC_URL}}other_style.css" rel="stylesheet" type="text/css">
    </head>
    <body>
        <h1>This is home for other_app</h1>
    </body>
</html>



Refresh your page at http://127.0.0.1:8000/other_app/home. You will see a blue background on this page. You might have to restart your server to see this change. Also, you will be able to see the stylesheet for this page at http://127.0.0.1:8000/static/other_style.css.
Also, verify that home of some_app still has a red background at http://127.0.0.1:8000/some_app/home.
What happened here.
When we made a request for /static/other_style.css, Django knows that STATIC_URL is set to '/static/' which matches the first fragment of the url we provide and hence infers that we want it to serve a static file. So, it starts looking into static/ subdirectory of all the apps. It looks into 'static/' subdirectory of all apps because STATICFILES_FINDERS contains 'django.contrib.staticfiles.finders.AppDirectoriesFinder'. Since it finds a file called other_style.css in the static/ subdirectory of other_app, it serves it.
However, there is one problem with this approach. You must have noticed that we named the stylesheet of other_app as other_style.css. What if we wanted to name this stylesheet as styles.css. Let's try doing it.
mv other_app/static/other_style.css other_app/static/styles.css



Once we make this change, we will also have to change the html of homepage of other_app to include this file. So, edit templates/other_app/home.html and make this change. We have to do this because we renamed other_style.css to styles.css. Html of other_app changes to:
<html>
    <head>
        <link href="{{STATIC_URL}}styles.css" rel="stylesheet" type="text/css">
    </head>
    <body>
        <h1>This is home for other_app</h1>
    </body>
</html>



Now see both the html pages, we have created.
http://127.0.0.1:8000/some_app/home
http://127.0.0.1:8000/other_app/home
You will find that the background of both pages is shown as red now. It depends on what order you have the apps listed in INSTALLED_APPS. If some_app appears before other_app in INSTALLED_APPS, both the pages will have red background. If other_app appears before some_app, both the pages will have blue background. In my INSTALLED_APPS, some_app appears before other_app, so both pages have red background for me.
Why this happened:
Both these pages try to access a static file named styles.css. Django tries to find this file in the static/ subdirectory of each app listed in INSTALLED_APPS. Once it finds it in static/ subdirectory of some_app, it serves it and does not try to find it in static/ subdirectory of other_app. Since, static/styles.css for some_app says that the background should be red, so the background is shown red for both of these pages. 
How do we prevent it:
So, we want to name stylesheet in both apps as styles.css. For this, we need one extra level of directory inside static/ subdirectory of each app. Let's do the following
mkdir some_app/static/some_app
mv some_app/static/styles.css some_app/static/some_app

mkdir other_app/static/other_app
mv other_app/static/styles.css  other_app/static/other_app/



So, we created a directory inside static/ subdirectory of each app. The name of this directory is the same as the app name. Then we moved the static resource i.e stylesheet inside the new directory.
We will also have to change the path to stylesheet in the templates.
Edit templates/some_app/home.html and change the path of stylesheet. So, the new content becomes:
<html>
    <head>
        <link href="{{STATIC_URL}}some_app/styles.css" rel="stylesheet" type="text/css">
    </head>
    <body>
        <h1>This is home for some_app</h1>
    </body>
</html>



Make similar change to templates/other_app/home.html, so that new content becomes:
<html>
    <head>
        <link href="{{STATIC_URL}}other_app/styles.css" rel="stylesheet" type="text/css">
    </head>
    <body>
        <h1>This is home for other_app</h1>
    </body>
</html>



Now see both the html pages again
http://127.0.0.1:8000/some_app/home
http://127.0.0.1:8000/other_app/home
You should find red background for some_app and blue background for other_app. 
What happened here

Template of some_app references http://127.0.0.1:8000/static/some_app/styles.css.
Django sees that this url starts with /static/, which is the STATIC_URL and infers that it needs to serve the static file some_app/styles.css
It starts looking for file some_app/styles.css in static/ subdirectory of apps.
It finds this file in static/ of some_app and serves it.
Template of other_app references http://127.0.0.1:8000/static/other_app/styles.css.
Django starts looking for file other_app/styles.css in static/ subdirectory of apps.
It finds it in static/ of other_app and serves it.

Hope STATIC_URL, STATICFILES_FINDERS and how static files are served makes more sense now.
About STATICFILES_DIRS
Till now our assumption was that we need separate styling on templates of some_app and other_app. So, we wrote different stylesheet for them.
Assume we want some style which should be common on all the pages across the project and they are not specific to a particular app. In that case, we do not keep such stylesheet in the static/ subdirectory of any particular app. Instead we create a directory at the same level as manage.py and put project wide static resources in that directory.
Let's see it in action.
Create a directory called project_static at the same level as manage.py.
mkdir project_static



Let's create a file named base.css inside it.
touch project_static/base.css



Edit this file project_static/base.css and add the following content to it.
h1
{
    font-style: italic;
}



So, we want to make h1 as italic across entire project.
Django is still not aware about this file and will not serve it. For making Django aware about this file we need to add the directory containing this file to STATICFILES_DIRS. So edit test_project/settings.py and add the required directory to STATICFILES_DIRS.
STATICFILES_DIRS = ( 
    os.path.join(PROJECT_DIR, '../project_static'),
)



Try to access url 127.0.0.1:8000/static/base.css. You should see the contents of the stylesheet you just wrote. Make sure that you have 'django.contrib.staticfiles.finders.FileSystemFinder' in STATICFILES_FINDERS, otherwise this url will give you a 404 page.
What happened here:

Django server sees that a request comes for a static resource, since it is for a url that starts with '/static/'.
It tries to find the static file i.e base.css in any of the directory defined in STATICFILES_DIRS.
Since we have only one directory defined in STATICFILES_DIRS, which is project_static, Django server tries to find this file inside this directory. It finds the file it is searching for in this directory and serves it.
Had it not found this file in any of the directory defined in STATICFILES_DIRS, it would have tried to find the file in static/ subdirectory of each of the apps defined in INSTALLED_APPS.
Notice that we still didn't need to add staticfiles_urlpatterns().

For using this file in templates, we need to reference this stylesheet. Add the following line to both of the templates we defined.
<link href="{{STATIC_URL}}base.css" rel="stylesheet" type="text/css">



Refresh the urls for both the pages. You will see that h1 appears in italic on these pages.
Let's check the final directory structure which will help in case you have some issues.
(staticvirt)~/staticvirt/test_project$ tree -I *.pyc
.
|-- manage.py
|-- other_app
|   |-- __init__.py
|   |-- models.py
|   |-- static
|   |   `-- other_app
|   |       `-- styles.css
|   |-- tests.py
|   |-- urls.py
|   `-- views.py
|-- project_static
|   `-- base.css
|-- some_app
|   |-- __init__.py
|   |-- models.py
|   |-- static
|   |   `-- some_app
|   |       `-- styles.css
|   |-- tests.py
|   |-- urls.py
|   `-- views.py
|-- templates
|   |-- other_app
|   |   `-- home.html
|   `-- some_app
|       `-- home.html
`-- test_project
    |-- __init__.py
    |-- settings.py
    |-- urls.py
    `-- wsgi.py

11 directories, 20 files



About STATIC_ROOT

You should never require STATIC_ROOT in development if you are using Django's runserver.
Once you go to production, you can use it on the server. Django provides a management command called collectstatic, which will collect all the static resources, i.e the resources found in STATICFILES_DIRS and the resources found in static/ subdirectory of apps, into a single location defined by STATIC_ROOT.
STATIC_ROOT only comes into picture if you want to use collectstatic management command provided by Django.

Let's try it out.
Create a directory named static_resources.
mkdir static_resources



Edit test_project/settings.py and add the following line:
STATIC_ROOT = os.path.join(PROJECT_DIR, '../static_resources')



Now run the following command:
python manage.py collectstatic



You will be asked for confirmation. Type 'yes', and you will see all your static resources are collected into the directory defined by STATIC_ROOT.
And on your production server you can configure your server to serve all the request for static files from the directory defined by STATIC_ROOT.
Again, this STATIC_ROOT part was only for illustration purpose. You should not need it during development.


Two Scoops of Django: Review

Two scoops of Django is the new book on Django best practices by Daniel Greenfeld and Audrey Roy. I just finished reading it and found it extremely useful.
The book is a collection of tips, their justification and code organized in logical areas. I have been using Django since 2008, but I still found a few tips which were new to me and many which served as good reminder. At about 200 pages, its comprehensive but not overwhelming. If you are an advanced Djangonaut, you can read it in a weekend (And you probably are using all its recommendations anyway). If you are just getting started, putting all the tips to use will keep you busy for month.
A random sampling of tips (The book has more than 50 of them.)

Use pip, virtualenv and virtualenvwrapper
Don't keep your virtualenv inside the project folder
Apps should be small and do one thing well
Use relative modules, prefer from .models import foo
All settings file should be versioned - local_settings.py is an anti-pattern
Every settings module should have a corresponding requirements.txt
Don't hardcode your paths
SingleTable model inheritance is good, multi table model inheritance is bad.
Create custom managers, but don't reset  default manager.
If you use CBV, ensure logic doesn't go to urls
Use Meta.fields and Never Use Meta.excludes

I especially like that the book mentions alternate ways of doing things. Eg. it recommends CBV as the new and shiny, but advises that FBV are just as reasonable choice.
If you use Django, I highly recommend this book. At just $12, it would be hard to go wrong.


Easy client side form validations for Django: Django Parsley
Parsleyjs is a JavaScript library to do client side data validations.
It does this in a non-intrusive way via adding a data-* attributes to form fields. Check it out, it's really slick.
Django forms do a great job of server side data validations. By integrating Parsley with Django form, you get good client side data validations as well. Get the app here.
Some design considerations
When I started writing this app I wanted to make it really easy to use. My first thought was to make this a filter or a template tag which would have allowed you to do:
{{ form|parsleyfy }}



However, I use Django-crispy-forms in all my projects. If parsley was a filter, it would have made piping to |crispy impossible as the render step would already have been completed. Hence parsleyfy is class decorator. Django-crispy-forms and family will play well with parsleyfy.
Here is the readme.

What is it?
Parsleyjs is a JavaScript library to do client side data validations.
It does this in a non-intrusive way via adding a data-* attributes to form fields.
When you define a Django form, you get server side validations for free using
the form field attributes. Django-parsley adds these validations to client side, by tagging your form with data-* attributes.
Installation
Add parsley to your INSTALLED_APPS.
Usage
parsley provides a single class decorator called parsleyfy. Decorate your Form with parsleyfy to get the validations.
Eg.
from parsley.decorators import parsleyfy

@parsleyfy
class FieldTypeForm(forms.Form):
    name = forms.CharField(min_length=3, max_length=30)
    url = forms.URLField()
    url2 = forms.URLField(required=False)
    email = forms.EmailField()
    email2 = forms.EmailField(required=False)
    age = forms.IntegerField()
    income = forms.DecimalField()



Your rendered form's HTML will look like this
<p><label for="id_name">Name:</label> <input data-required="true" data-minlength="3" maxlength="30" type="text" data-maxlength="30" id="id_name" name="name" /></p>
<p><label for="id_url">Url:</label> <input type="text" data-required="true" data-type="url" name="url" id="id_url" /></p>
<p><label for="id_url2">Url2:</label> <input type="text" data-type="url" name="url2" id="id_url2" /></p>
<p><label for="id_email">Email:</label> <input type="text" data-required="true" data-type="email" name="email" id="id_email" /></p>
<p><label for="id_email2">Email2:</label> <input type="text" data-type="email" name="email2" id="id_email2" /></p>
<p><label for="id_age">Age:</label> <input type="text" data-required="true" data-type="digits" name="age" id="id_age" /></p>
<p><label for="id_income">Income:</label> <input type="text" data-required="true" data-type="number" name="income" id="id_income" /></p>



Note the data-* attributes.
You could also do
FieldTypeForm = parsleyfy(FieldTypeForm)
Which is the same thing.
Put this form inside a
<form data-validate="parsley">
    {{ form.as_p }}
</form>



Include the parsleyjs and you are good to go.
License
3 Clause BSD.
Bug report and Help
For bug reports open a github ticket. Patches gratefully accepted. Need help? Contact us here


Not exactly, not exactly tim the enchanter
I've been putting off writing about my experience with django form wizard
for some time now. I came across this blog post by Kenneth Love
which finally compelled me to write this down.
About Django Form Wizard:
Form Wizard can be handy when you have a huge form, and you want to make it less
intimidating to the user by splitting it into multiple steps.
If you have glued multiple forms together with sessions or whatnot and it
turned out to be a mess, you now what I'm talking about :)
Think surveys, checkouts, lengthy registrations etc.
The class WizardView implements most of functionality while subclasses
SessionWizardView, CookieWizardView etc. specify which backend to use.
Typically, the view takes a list of forms and presents them one by one.
After each step, the form is validated and the raw form data is saved in the backend
(session or cookie for POST data and file_storage for files). After the last step,
each form is filled with raw data from the backend and validated again.
The list of fully-filled up form instances is then passed to the done handler
where the forms can be processed.
I guess the key bit of info here is: Nothing is written to the database until all
the steps are completed and the forms are saved manually.
I know that feel:
I'm not a fan of putting more stuff than necessary in urls either, and I think
this can be easily remedied the way Kenneth wants it.
For example, to specify forms_list and url_name in the views instead of urls one could do:
from holygrail.forms import NameForm, QuestForm, ColorForm

class MerlinWizard(NamedUrlSessionWizardView):

    ...
    ...

    @classonlymethod
    def as_view(cls, *args, **kwargs):
        kwargs.update({
            'form_list': [
                NameForm,
                QuestForm,
                ColorForm,
            ],
            'url_name': 'merlin_wizard'
        })
        return super(MerlinWizard, cls).as_view(*args, **kwargs)

    ...
    ...



and voilà, you no longer need to pass any args to MerlinWizard
ModelForms/Formsets:
I think modelforms/formsets are perfectly fine for usage in the form wizard,
although, you'll need to dive into a few details:
Save a modelform:
We can use the process_step method here. This methods takes the form instance
and returns the data to be saved in the storage for the current step. By default,
it returns the raw form data.
def process_step(self, form):
    """
    you can save the form here,
    since it's guaranteed to be valid
    """
    if self.steps.current == '2':
        instance = form.save()
    return super(MerlinWizard, self).process_step(form)



If you need to pass around the instance to another step, you can add it to the step's storage data e.g:
Let's say we need to save a model in step 2 and create a formset based on its instance
in step 3:
def process_step(self, form):
    """
    you can save the current form here,
    since it's guaranteed to be valid
    """
    step_data = super(MerlinWizard, self).process_step(form).copy()
    if self.steps.current == '2':
        instance = form.save()
        step_data['instance_pk'] = instance.pk
    return step_data



then you can retrieve the instance_pk like this:
def get_form_kwargs(self, step=None):
    """
    in step 3, use the instance from step 2
    as a kwarg to the formset
    """
    kwargs = super(MerlinWizard, self).get_form_kwargs(step=step)
    if step == '3':
        step_data = self.storage.get_step_data('2')
        instance_pk = step_data['instance_pk']
        instance = MyModel.objects.get(pk=instance_pk)
        kwargs.update({
            'instance': instance
        })
    return kwargs



Defaults:
I agree with Kenneth that storages should just work™ like they work with forms/modelforms.
But, it's handy when you want to keep the wizard media away for the rest of the user media.
One workaround is to just set the file storage to the default storage engine:
from django.core.files.storage import default_storage

...
...

class MerlinWizard(NamedUrlSessionWizardView):

    file_storage = default_storage
    ...
    ...



Are we done, yet?:
You can handle all the forms you want to save to the database in done method.
AFAIK, there's no need to do:
del self.request.session["wizard_key_here"]



The wizard cleans up both when it's started and when it's done.
Also, relying on request.session breaks the storage abstraction,
so you're better off using self.storage if your really want to mess with the storage.
This way, you can switch to a different wizard backend without worrying too much.
Conclusion:
Having spent a good deal of time with the form wizard over the past few weeks, I feel that
form wizard does a very good job at handling the use case it was designed for.
Even though the documentation is not perfect, the implementation is clear, concise and flexible
enough to handle a variety of scenarios.


The missing documentation for django.utils.datastructures

Note
django.utils.datastructures is intentionally not documented by the django
core devs because it is an internal API and is liable to change without any
notice. This file is not governed by django's lenient backwards-compatible
policy. You have been sufficiently warned!

With the note out of the way, let's look at the interesting datastructures in
this file. You may ask why we should learn about those when we shouldn't be
using them? Reading code is the best way of learning and this file has some
beautiful code.
MergeDict is the first of the datastructures in the file. It provides a
dictionary like interface but can look up from multiple dictionaries provided
during the initialization.
Here's an example:
>>> md = MergeDict({"foo": "bar", "moo": "cow"}, {"abc": "def"})
>>> md["foo"]
'bar'
>>> md["abc"]
'def'
>>> md.get("abc")
'def'
>>> md["xyz"]
KeyError:
>>> md.items()
[('foo', 'bar'), ('moo', 'cow'), ('abc', 'def')]
>>> md.keys()
['foo', 'moo', 'abc']
>>> md.values()
['bar', 'cow', 'def']

The MergeDict is used within django in attaching values with a form widget and in
request.REQUEST.
The built-in dictionary does not maintain the order of the items but the
SortedDict is a subclass of the built-in dictionary that maintains the keys
in exactly the same order they were inserted.
Here's an example:
>>> dd = {"foo": "bar", "moo": "cow", "abc": "def"}
{"abc": "def", "foo": "bar", "moo": "cow"}
>>> sd = SortedDict((("foo", "bar"), ("moo", "cow"), ("abc", "def")))
{"foo": "bar", "moo": "cow", "abc": "def"}
>>> dd["xyz"] = "pqr"
>>> dd
{'abc': 'def', 'foo': 'bar', 'moo': 'cow', 'xyz': 'pqr'}
>>> dd["lmn"] = "ghi"
>>> dd
{'abc': 'def', 'foo': 'bar', 'lmn': 'ghi', 'moo': 'cow', 'xyz': 'pqr'}
>>> sd["xyz"] = "pqr"
>>> sd
{'foo': 'bar', 'moo': 'cow', 'abc': 'def', 'xyz': 'pqr'}
>>> sd["lmn"] = "ghi"
{'foo': 'bar', 'moo': 'cow', 'abc': 'def', 'xyz': 'pqr', 'lmn': 'ghi'}

The SortedDict is fairly widely used inside of django generally to build a
hierarchy (like models and it's parents), maintaining the order of the form fields
while iterating etc.
In python 2.7, a new datastructure was introduced that mimics the SortedDict in
the collections module and is called OrderedDict.
MultiValueDict is a dictionary subclass that can handle multiple values
assigned to a key.
Here's an example:
>>> dd = {"abc": "def", "foo": ["bar1", "bar2"]}
>>> dd["foo"]
['bar1', 'bar2']
>>> mvd = MultiValueDict({"abc": "def", "foo": ["bar1", "bar2"]})
>>> mvd["foo"]
'bar2'
>>> mvd.getlist("foo")
['bar1', 'bar2']
>>> mvd.getlist('blah')
[]
>>> mvd.getlist('abc')
'def'
>>> mvd.setlist('xyz', ['pqr', 'ghi'])
>>> mvd
<MultiValueDict: {'xyz': ['pqr', 'ghi'], 'abc': 'def', 'foo': ['foo1', 'foo2']}>
>>> mvd.appendlist('xyz', 'ijk')
>>> mvd
<MultiValueDict: {'xyz': ['pqr', 'ghi', 'ijk'], 'abc': 'def', 'foo': ['foo1', 'foo2']}>
>>> mvd.update({'xyz': 'lmn'})
>>> mvd
<MultiValueDict: {'xyz': ['pqr', 'ghi', 'ijk', 'lmn'], 'abc': 'def', 'foo': ['foo1', 'foo2']}>

The MultiValueDict is used in binding data to request.POST, the files to request.FILES
and in the get parameter parsing.
The ImmutableList is an immutable datastructure that raises errors when it is
attempted to be mutated.
Here's an example:
>>> il = ImmutableList(['foo', 'bar', 'abc'])
>>> il += 'lmn'
AttributeError: ImmutableList object is immutable.
>>> il = ImmutableList(['foo', 'bar', 'abc'], warning='Custom warning')
>>> il[1] = 123
AttributeError: Custom warning

The ImmutableList is used in request.upload_handlers to prevent
modification after the request.POST or request.FILES have been accessed.
The DictWrapper is a subclass of the built-in dictionary that prefixes the keys.
It takes a dictionary, a function and a prefix as arguments. If a specific key lookup
begins with the prefix then the value is passed through the function before it is
returned.
Here's an example:
>>> dw = DictWrapper({'foo': 'bar', 'moo': 'cow'}, lambda x: x, 'abc_')
>>> dw['foo']
'bar'
>>> dw['abc_foo']
'bar'
>>> dw['xyz_foo']
KeyError: 'xyz_foo'
>>> def post_process_value(value):
...     return "The value is " + value
>>> dw = DictWrapper({'foo': 'bar', 'moo': 'cow'}, post_process_value, 'abc_')
>>> dw['foo']
'bar'
>>> dw['abc_foo']
'The value is bar'

The DictWrapper is used in quoting names for SQL queries with the key prefix.
Hope you enjoyed learning about these hidden gems and how django works under the hood but
take the note on the top into consideration.


Rails for Django developers
My presentation at Pycon India 2012
 






Download it from here


Dropbox file upload handler for django
Dropbox announced new pro plans last week and some accounts have had their
storage size doubled. Wouldn't it be wonderful if we could upload all our
files to dropbox from our django webapp?
In this post, I write a custom file upload handler that will upload files
from our application to dropbox.
Let us see how to use the custom file upload handler.
Install the Dropbox Python SDK before you setup your django app to handle
the file uploads.
In your settings.py, add the following attributes (with the values filled):
DROPBOX_APP_KEY = ""
DROPBOX_APP_SECRET_KEY = ""
DROPBOX_APP_ACCESS_TOKEN = ""
DROPBOX_APP_ACCESS_TOKEN_SECRET = ""

# Optional values below

# The folder where you want the files uploaded.
# Example: /Public or /
DROPBOX_FILE_UPLOAD_FOLDER = ""
# The value below may be either 'app_folder' or 'dropbox'
DROPBOX_ACCESS_TYPE = ""

The DROPBOX_APP_KEY and DROPBOX_APP_SECRET_KEY are provided to you when you
create a new dropbox app. Fetching the access token and access token secret
is outside the scope of this blog post but you can follow the Getting Started
Guide until the Get an access token section and then paste the access token
key and secret in the DROPBOX_APP_ACCESS_TOKEN and DROPBOX_APP_ACCESS_TOKEN_SECRET
attributes respectively.
Add the DropboxFileUploadHandler to any app (in my case testapp) and reference
it in the FILE_UPLOAD_HANDLERS in settings.py:
FILE_UPLOAD_HANDLERS = (
    "testapp.dropbox_upload_handler.DropboxFileUploadHandler",
)

That's it and you are done!

Note
Since dropbox doesn't support chunked uploads, the file is first uploaded
to the temporary file upload directory on the server and then onto dropbox.

Here's how you would handle the file post upload in your view:
def file_upload_handler_view(request):
    if request.method == "POST":
        file_uploaded = request.FILES["name_of_file_input"]
        print file_uploaded.read()
        # Helpful attribute to get dropbox file metadata
        # like path on the server, size, thumbnail etc
        file_uploaded.dropbox_metadata

The DropboxFile returned is an instance of httplib.HTTPResponse and so all
file like methods are not defined but some basic methods like read are supported.
There is an attribute called dropbox_metadata on the uploaded file that holds
the dropbox metadata.


Deploying Django apps on Heroku
Deploying Django apps on Heroku:
Read this first: http://devcenter.heroku.com/articles/django.
This is a great article by the Heroku. I am just filling in some more details and making this step-by-step.

Get your Django project code.

Create a virtualenv with a no-site-packages command ::
virtualenv vent --no-site-packages


Install Django, psycopg2 (postgres connector), gunicorn and any other required Django libraries.

Confirm that you have all the required libraries and you can run your code locally using manage.py runserver.

Create a requirement.txt by using 
    ::
pip freeze > requirements.txt





Make sure you have a requirements.txt at the root of your repo. Heroku uses this to identify that the app is a Python app.


Create a Procfile. Put this entry: ::
web: python mysite/manage.py run_gunicorn -b "0.0.0.0:$PORT" -w 3shabda


This is what your directory structure will look like
::
[root-of-heroku-folder]
    requirements.txt
    Procfile
    [Django-project-folder]
        __init__.py
        manage.py
        settings.py
        [app1]
        [app2]





(If you aren't tracking your files with git yet.) Track your files with git. ::
git init
git add .
git commit -am "First commit"


Make sure you have the heroku toolbelt installed. If not go to http://toolbelt.herokuapp.com/ and install.


11.You should have these commands available now: ::
heroku
foreman





Authenticate to heroku with 
::
heroku auth:login





Run this command. 
    ::
heroku create --stack cedar





This will create a new Heroku app and create a new remote in your git repo.


Push your code to heroku. 
    ::
git push heroku master





Your app should be working on Heroku now. heroku open will show your site.




Deploy Django App in 5 Easy Steps
So you just bought a new VPS, have installed Ubuntu and want to deploy
your django app, GREAT!! We shall get your app, up and running in 5 easy steps, using best(arguably) of tools available. The post is targeted to audience who are new to deployment arena, but assumes you are comfortable with developing basic django apps. We shall be using gunicorn as our server and nginx nginx as our reverse proxy and static hanlder. Here we go:
1. Login and OS Updation:
$ ssh root@{your_ip}

# apt-get update

# apt-get upgrade

# apt-get dist-upgrade

# dpkg-reconfigure tzdata #choose your time zone



2. Setup Users and Permissions:
# useradd agiliq

# mkdir /home/agiliq

# chown agiliq:agiliq /home/agiliq

# passwd agiliq  #choose a password

# chsh agiliq -s /bin/bash  #choose a shell

# visudo #lets give sudo access to the user agiliq

    root     ALL=(ALL) ALL 
    agiliq   ALL=(ALL) ALL

# su agiliq # switch from root to agiliq



3. Install necessary stuff and create a dev environment:
$ sudo apt-get install python-pip git mysql-client mysql-server

$ python-mysqldb nginx emacs(you might choose vim or stick to default vi/nano)

$ pip install virtualenv #please refer to documentation of virtualenv

$ virtualenv projhq # creates a project directory,

$ cd projhq

$ projhq source /bin/activate

(projhq)$ pip install django gunicorn



4. Deploy some code for test
you may clone this polls application for the sake of testing
    $ cd ~/projhq #your project directory
    (projhq)$ git clone git://github.com/bitaVersion/Hello-Django.git



5. Necessary Configurations
i) nginx
    $ cd /etc/nginx/sites-enabled/
    $ touch nginx.conf



and inside the file place the following snippet, make sure to change your 
servername, username and static files directory. 
server {
        listen 80;
        server_name “your server name”;
        access_log /home/username/access.log;
        error_log /home/username/error.log;
        location /static {
            root /home/path_to_proj_directory_containing static_files;
        }

        location / {
            proxy_pass http://127.0.0.1:8000;
        }
}



ii)gunicorn configuration
cd into project directory/app 
touch gunicorn_cfg.py 
place the following snippet:
bind = "127.0.0.1:8000"
logfile = "/var/log/gunicorn.log"
workers = 3
pid = projhq/pid/gunicorn.pid



iii)server restart script
make a directory called scripts parallel to project root
and place the following snippet in some file(i name start.sh):
PROJDIR="/path/to/proj/" #directory which contains settings.py
PIDFILE="$PROJDIR/pid/gunicorn.pid" #create a directory called pid inside projdir
cd $PROJDIR
source /path/to/bin/activate
if [ -f $PIDFILE ]; then
    kill `cat -- $PIDFILE`
    rm -f -- $PIDFILE
fi
gunicorn_django -c path_to_gunicorn_file/gunicorn_cfg.py



Few checks if you have missed them:
create mysql db:
   create a database with the name matching your settings.py



Start nginx:
  sudo service nginx restart



Start server:
go to scripts directory: 
chmod +x start.sh(only for the first time)
bash start.sh



The above post was a general introduction of how you can begin using lightweight yet scalable tools, in the next post we shall be talking about tools as fabric, supervisor and Monit, which makes things far easier and more secure. Experienced admins can point out flaws in the approach and suggest improvements.


Dynamically attaching SITE_ID to Django Caching
It would be useful and convenient, if you have an automatic way to add the SITE_ID, especially, when you have multiple sites running on the same deployment. Django provides a cache prefix function KEY_FUNCTION in settings which can be used to achieve this.
Just follow the following steps, and your cache, automatically prepends SITE_ID to the cache key, making it unique across multiple sites.


Put the following into the settings file.
     CACHES = {

     'default': {

      'BACKEND': 'django.core.cache.backends.db.DatabaseCache',

       'LOCATION': 'cache_table',

       KEY_FUNCTION = ‘projectname.appname.modulename.functionname’,

                }

         }





Write a function to get current site id, say, get_current_site(), which returns current SITE_ID.


Add a function like below, as functionname at the same path as specified in KEY_FUNCTION.
from django.utils.encoding import smart_str

def prefix_site_id(key, key_prefix, version):

   site_id = get_current_site()

   return ':'.join([str(site_id), str(version), smart_str(key)])





That’s it. You have successfully added an automatic, dynamic, function based cache prefix for django.


Deploying Django apps on Heroku
Read this first: http://devcenter.heroku.com/articles/django.
This is a great article by the Heroku. I am just filling in some more details and making this step-by-step.

Get your Django project code.

Create a virtualenv with a no-site-packages command
virtualenv vent --no-site-packages


Install Django, psycopg2 (postgres connector), gunicorn and any other required Django libraries.

Confirm that you have all the required libraries and you can run your code locally using manage.py runserver.


Create a requirement.txt by using
pip freeze > requirements.txt




Make sure you have a requirements.txt at the root of your repo. Heroku uses this to identify that the app is a Python app.

Create a Procfile. Put this entry:
web: python mysite/manage.py run_gunicorn -b "0.0.0.0:$PORT" -w 3shabda


This is what your directory structure will look like

[root-of-heroku-folder]
    requirements.txt
    Procfile
    [Django-project-folder]
        __init__.py
        manage.py
        settings.py
        [app1]
        [app2]



(If you aren't tracking your files with git yet.) Track your files with git.
git init
git add .
git commit -am "First commit"


Make sure you have the heroku toolbelt installed. If not go to http://toolbelt.herokuapp.com/ and install.


11.You should have these commands available now:
heroku
foreman


Authenticate to heroku with
heroku auth:login


Run this command.
heroku create --stack cedar



This will create a new Heroku app and create a new remote in your git repo.

Push your code to heroku.
git push heroku master


Your app should be working on Heroku now. heroku open will show your site.




How to use pep8.py to write better Django code


Here is another or screencast I created for my "Getting Started with Django" series. Like this? Email us on what would you like to see.
(Watch in fullscreen.)


Screencast: Django Tutorial Part 1
Django Screencast Tutorial 1 from Shabda Raaj on Vimeo.


I am creating screencasts for Django tutorial and other Django videos. Here is part 1. Liked this? Leave me a comment or email me and tell me what would you like me to create screencasts for.



(Watch the screencast in full screen mode.)



How and why to use pyflakes to write better Python


Here is another of the screencasts I created for "Getting started with Python" series. Liked them? Let me know what else would you like me to create screencasts about?


Getting started with South for Django DB migrations
South is a migration tool used with Django.There will be times when you would be adding fields to a model or changing the type of field (eg: Field was an IntegerField and you want to change it to FloatField). In such cases syncdb doesn't help and South comes to your rescue.
There were times when i tried "python manage.py migrate southapp", got some error and then tried "python manage.py migrate southapp 0001 --fake". In some cases, that worked. When it did not work, i tried something else and so on. There were confusions regarding what --fake does and why it does so. In this post, i intend to remove(lessen) that confusion.
We will be seeing some scenarios which will help us understand south better. At various points, we will intentionally make some mistakes and will rectify them later. Also, we keep looking at our database schema as we go. 
Scenario 1: Using south on a brand new app.
Let's create an app named "farzi_app".
python manage.py startapp farzi_app



In settings.py, add 'south' to INSTALLED_APPS and run syncdb.
python manage.py syncdb



Running syncdb creates a table named 'south_migrationhistory' in your database. 
This table 'south_migrationhistory' keeps a track of all the migrations which have been applied. So, this is the table used by south to determine which migrations have been applied and which need to be applied, when you run "python manage.py migrate app_name". More on this table later. You can see this table in your database. See it.
python manage.py dbshell
mysql> show tables;



See the entries in table south_migrationhistory.
select * from south_migrationhistory;



You would see an Empty set at this point.
The command used to perform migrations is:
python manage.py migrate



Run this command.
Since at this point, none of our apps are being managed by south, so performing migration do not do anything.
Let's make a model in our app 'farzi_app'. Let's name the model Employee. This model looks like:
class Employee(models.Model):
        name = models.CharField(max_length=100)



Let's set "farzi_app" to be managed by South(south works on an app level and not on a model level). Had you not wanted to manage this app(farzi_app) by South, you would have run syncdb and a table named "farzi_app_employee" would be created in your database.
But since we want to use south on a brand new app we won't be using syncdb, we need to write a migration which will create the corresponding table in database. 
Before writing the migration, add "farzi_app" to INSTALLED_APPS in settings.py .Writing our migration:
python manage.py schemamigration farzi_app --initial



We used --initial because this is the first migration we wrote. Since, we don't have any migration written for "farzi_app", we need to use --initial. Other option which can be used is --auto. But, --auto needs a previous migration. In our next migration, we will use --auto and see what it does. So, the output for our previous command is :
 + Added model farzi_app.Employee
Created 0001_initial.py. You can now apply this migration with: ./manage.py migrate farzi_app



Now, check your database. Still the table "farzi_app_employee" is not created in your database. Also, you can't find any row in table south_migrationhistory. Let's confirm this:
python manage.py dbshell
show tables; (You can't see farzi_app_employee)
select * from south_migrationhistory; (This gives Empty Set)



You need to apply the migration you wrote. So, command you use is:
python manage.py migrate farzi_app



What happens when you run this command? 
South checks all the migration written for "farzi_app" in the directory "farzi_app/migrations". Then it checks the table "south_migrationhistory" to see what migrations have already been applied. If a migration has already been applied(if its applied,it is entered in south_migrationhistory), south will not try to apply that migration. For migrations which have not been applied till now(not entered in south_migrationhistory), south will try to apply those migrations and try making the corresponding change in the database. 
In present scenario, south could not find any entry in 'south_migrationhistory' table, so it assumes no migration has been applied till now and tries to apply whatever migrations are there in "farzi_app/migrations". Currently, we have only one migration named "0001_initial.py" in this directory. So, south will try to apply this migration. In this situation, migration would be successful and hence our command "python manage.py migrate farzi_app" is successful. 
Now, check the tables in database and entries in south_migrationhistory.
python manage.py dbshell
show tables; (You can see farzi_app_employee)
select * from south_migrationhistory; (This contains one row now. You can easily identify that this table keeps track of what migrations have been applied.)



Let's change our model now. Add a new field named 'salary' to model 'Employee'. So, our model looks like.
class Employee(models.Model):
        name = models.CharField(max_length=100)
        salary = models.IntegerField(null=True)  
    """Yeah i know salary should not be left as null. But if we don't use it, we will have to make some one-off value and all 
        which we don't want to see right now."""



Making this corresponding change to database would have been difficult with syncdb, but not with south. We need to write one more migration. This time we will use --auto(remember last time we used --initial). Let's write the migration.
python manage.py schemamigration farzi_app --auto

Output:
 + Added field salary on farzi_app.Employee
Created 0002_auto__add_field_employee_salary.py. You can now apply this migration with: ./manage.py migrate farzi_app



What happens when we run this command? 
South sees our current code, sees the last migration which we wrote, which in current situation is 0001_initial.py, figures out what has changed since our last migration and then writes the information about the changes in our new migration(0002_auto__add_field_employee_salary.py). 
Just creating this migration would not make the corresponding change in our database i.e at this point column 'salary' is not added to table 'farzi_app_employee'. For that, we need to run this migration.Let's do that:
python manage.py migrate farzi_app

Output:
Running migrations for farzi_app:
- Migrating forwards to 0002_auto__add_field_employee_salary.
> farzi_app:0002_auto__add_field_employee_salary
- Loading initial data for farzi_app.
No fixtures found.



What happened when we ran "python manage.py migrate farzi_app"?
Similar to what we saw earlier. South checks all the migrations in farzi_app/migrations/ . It sees two migrations namely '0001_initial.py' and '0002_auto__add_field_employee_salary.py'. It then checks 'south_migrationhistory' table to see what migrations have been applied. It could find one row(this row was inserted by south when we last ran 'python manage.py migrate farzi_app'). So, south figured out that '0001_initial.py' is applied since this entry is there in table 'south_migrationhistory'. Also, south figured out that '0002_auto__add_field_employee_salary.py' is not applied since this entry was not in table 'south_migrationhistory'. So, it tries applying this migration i.e 0002_auto__add_field_employee_salary.py'. In this case, this try is successful and corresponding change is made in table farzi_app_employee. Also, an entry would be now made in 'south_mihgrationhistory' that tells that '0002_auto__add_field_employee_salary.py' is applied. So, next time somebody runs migrate, this migration '0002_auto__add_field_employee_salary.py' will not be applied. Check your database now.
python manage.py dbshell
desc farzi_app_employee; (you can see column salary).
select * from south_migrationhistory;



south_migrationhistory has two rows now, which indicates both the migrations we wrote till now has been applied. Next time user runs migrate, south will not try to apply these migrations.
Understanding --fake:
Let's get some errors intentionally which will help understand --fake better. Delete the second row in "south_migrationhistory" table. We will see an actual use of --fake in the next scenario, but i am doing this just to illustrate the use of --fake.
Deleting the second migration from south_migrationhistory:
delete from south_migrationhistory where migration='0002_auto__add_field_employee_salary';



Now, only one row remains in 'south_migrationhistory'.
Let's change our Employee model a bit and write a migration. Add one more field to 'Employee', so it looks like:
class Employee(models.Model):
        name = models.CharField(max_length=100)
        salary = models.IntegerField(null=True)
        age = models.IntegerField(null=True)



Writing the migration:
python manage.py schemamigration farzi_app --auto



This creates a migration named '0003_auto__add_field_employee_age.py' in directory farzi_app/migrations. Hereafter we will be referring to this migration as 0003 and will refer to the other two migrations as 0002 and 0001. To add the column 'age' to table 'farzi_app_employee', you need to run this migration i.e 0003.
Let's try running the migration.
python manage.py migrate farzi_app

Output:
Running migrations for farzi_app:
- Migrating forwards to 0003_auto__add_field_employee_age.
> farzi_app:0002_auto__add_field_employee_salary
! Error found during real run of migration! Aborting.

! Since you have a database that does not support running
! schema-altering statements in transactions, we have had 
! to leave it in an interim state between migrations.

   ! You *might* be able to recover with:   = ALTER TABLE `farzi_app_employee` DROP COLUMN `salary` CASCADE; []

! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS.
! NOTE: The error which caused the migration to fail is further up.
   Traceback (most recent call last):
 File "manage.py", line 14, in <module>
   execute_manager(settings)
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 438, in execute_manager
   utility.execute()
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 379, in execute
   self.fetch_command(subcommand).run_from_argv(self.argv)
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 191, in run_from_argv
   self.execute(*args, **options.__dict__)
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 220, in execute
   output = self.handle(*args, **options)
 File "/usr/local/lib/python2.7/dist-packages/south/management/commands/migrate.py", line 105, in handle
   ignore_ghosts = ignore_ghosts,
 File "/usr/local/lib/python2.7/dist-packages/south/migration/__init__.py", line 191, in migrate_app
   success = migrator.migrate_many(target, workplan, database)
 File "/usr/local/lib/python2.7/dist-packages/south/migration/migrators.py", line 221, in migrate_many
   result = migrator.__class__.migrate_many(migrator, target, migrations, database)
 File "/usr/local/lib/python2.7/dist-packages/south/migration/migrators.py", line 292, in migrate_many
   result = self.migrate(migration, database)
 File "/usr/local/lib/python2.7/dist-packages/south/migration/migrators.py", line 125, in migrate
   result = self.run(migration)
 File "/usr/local/lib/python2.7/dist-packages/south/migration/migrators.py", line 99, in run
   return self.run_migration(migration)
 File "/usr/local/lib/python2.7/dist-packages/south/migration/migrators.py", line 81, in run_migration
   migration_function()
 File "/usr/local/lib/python2.7/dist-packages/south/migration/migrators.py", line 57, in <lambda>
   return (lambda: direction(orm))
 File "/home/akshar/Posts/learnsouth/southlearn/farzi_app/migrations/0002_auto__add_field_employee_salary.py", line 12, in forwards
   db.add_column('farzi_app_employee', 'salary', self.gf('django.db.models.fields.IntegerField')(null=True), keep_default=False)
 File "/usr/local/lib/python2.7/dist-packages/south/db/generic.py", line 282, in add_column
   self.execute(sql)
 File "/usr/local/lib/python2.7/dist-packages/south/db/generic.py", line 150, in execute
   cursor.execute(sql, params)
 File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 34, in execute
   return self.cursor.execute(sql, params)
 File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 86, in execute
   return self.cursor.execute(query, args)
 File "/usr/lib/pymodules/python2.7/MySQLdb/cursors.py", line 166, in execute
   self.errorhandler(self, exc, value)
 File "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 35, in defaulterrorhandler
   raise errorclass, errorvalue
   _mysql_exceptions.OperationalError: (1060, "Duplicate column name 'salary'")



OOPS, you got an error and migration did not run properly. Why this happened?
South saw the migrations directory and saw three migrations there namely 0001, 0002 and 0003. Then it checked 'south_migrationhistory' table and found out only one row in the table which corresponds to the migration 0001 (remember , we deleted 0002). So, south feels that migration 0002 and 0003 needs to be applied. Remember, 0002 was written to add the column 'salary' to table 'farzi_app_employee'. So while running 0002, south tries adding column 'salary' while actually, that column exists in the database (this was added when we ran migrate for the second time). So, this causes the error. This is evident from the last line of the error we got, which says "_mysql_exceptions.OperationalError: (1060, "Duplicate column name 'salary'")".
So, we need to FAKE the migration which is causing this error. What we want to do is, we somehow lead south to beleive that migration 0002 has been applied and south should not try to apply this migration next time we run migrate. So, our command will be:
python manage.py migrate farzi_app 0002 --fake



So what this command did was, it created an entry into 'south_migrationhistory' corresponding to migration 0002. So, south now beleives that 0002 has been applied. But this command did not try to make any changes to the database. So, we did not get any error. Check 'south_migrationhistory', you would again see two rows there each corresponding to migration 0001 and 0002.
Now, run the migration to add the column 'age', which has to be done by migration 0003.
python manage.py migrate farzi_app



This should run without giving any error now. What we talked about --fake was just to see what --fake does, we will see a practical use of --fake in our next scenario, which is eventually my next post.


Behind the Scenes: Request to Response
In the previous installment of "Behind the Scenes", we saw how the control flows from Form to File Storage. Today, we are going to see how the application reacts from request to response.
In this post, we are going to assume that we are using django's inbuilt runserver. The flow doesn't change much for other WSGI servers available.
When you invoke the runserver management command, the command line options are validated and an instance of WSGIServer is created and passed the WSGIRequestHandler, which is used to create the request object (WSGIRequest). After the request object is created and the request started signal is fired, the response is fetched through the WSGIRequestHandler.get_response(request).
In the get_response method of the request handler, first the urlconf location (by default the urls.py) is setup based on the settings.ROOT_URLCONF. Then a RegexURLResolver compiles the regular expressions in the urlconf file. Next, the request middlewares are called in the order specified in the settings.MIDDLEWARE_CLASSES followed by the view middlewares after matching the view (callback) function against the compiled regular expressions from the urlconf. Then the view (callback) is invoked and verified that it does not return None before calling the response middlewares.
You can see the pictorial representation of the flow below:



Haml for Django developers
Haml is taking the Ruby(and Rails) world by storm. Its not used as heavily by Python(and Django) developers as the Python solutions aren't as mature. I finally gave Haml a try and was pleasantly surprised how easy it was.
The most mature Python implementation of Haml is hamlpy, which converts the hamlpy code to Django templates.
Others are shpaml and GHRML
Lets look at some templates from Django tutorial
{% if latest_poll_list %}
    <ul>
    {% for poll in latest_poll_list %}
        <li><a href="/polls/{{ poll.id }}/">{{ poll.question }}</a></li>
    {% endfor %}
    </ul>
{% endif %}



Here is this template converted to Haml
-if latest_poll_list
 %ul 
  -for poll in latest_poll_list 
    %li
        %a{'href':'/polls/{{ poll.id }}'}= poll.question



You can see that this lost a lot of noise from the Django template. Being a Django programmer,
significant whitespace is very pleasing.
Here is another one from the same tutorial.
<h1>{{ poll.question }}</h1>
<ul>
{% for choice in poll.choice_set.all %}
    <li>{{ choice.choice }}</li>
{% endfor %}
</ul>



And converted to hamlpy
%h1
    = poll.question
%ul
    -for choice in poll.choice_set.all
        %li
            = choice.choice



Again, I lose a lot of noise from the templates, and the signifiant whitespace improve the chance that the out put would be valid html.
Here are a few more templates.
In particular see this which is haml for this Pinax file

Resource

Haml
Hamlpy



Behind the Scenes: From HTML Form to Storage
In this post, we are going to see what happens behind the scenes when a file is uploaded to a django powered web application.


An HTML form with a file input (atleast one) and encoding set to multipart/form-data is submitted. The MultiPartParser parses the POST request and returns a tuple of the POST and FILES data (request.POST, request.FILES). The MultiPartParser processes the uploaded data using the File Upload Handlers objects (through the new_file, receive_data_chunk and upload_complete methods). The request.FILES values are a sequence of instances of UploadedFile.
In the django form, we pass the request.FILES MultiValueDict. These UploadedFile instances are validated by the full_clean method on the Form. The full_clean method in turn calls the _clean_fields method which calls the clean method on the forms.FileField and checks if the data is empty.
After the form is successfully validated, we might assign and save the cleaned_data of the form to a model instance (or by saving the ModelForm). When the save method on the model instance is called, the save_base method calls a pre_save method on each field of the model. This pre_save method returns the value of the file instance bound to that FileField and calls it's save method. This save method (on the models.FileField) in turn calls the save (Storage) method on the Storage which is either picked up from the arguments passed to the FileField (FileField(storage=...)) or the DEFAULT_FILE_STORAGE settings attribute.
This is the flow from the HTML Form all the way upto the File Storage. Hope you liked it.


Real time applications with Django, XMPP and StropheJS

TL;DR:

django-pubsub allows you to create Twitter like real-time updates for your models with Admin like ease.
Demo at http://chat.agiliq.com/pubsub/
Code at https://github.com/agiliq/django-pubsub



Introduction:
PubSub is a XMPP extension which allows publishing and subscribing to events.
This is useful when you instantly want to notify many clients about something
interesting happening on your server.
Quoting the authors of PubSub specs:
The protocol enables XMPP entities to create nodes (topics) at a
pubsub service and publish information at those nodes; an event
notification (with or without payload) is then broadcasted to all
entities that have subscribed to the node. Pubsub therefore
adheres to the classic Observer design pattern and can serve as
the foundation for a wide variety of applications, including news
feeds, content syndication, rich presence, geolocation, workflow
systems, network management systems, and any other application
that requires event notifications.

To understand this better, think of newspapers as publishers and the
people who subscribe to the newspapers as subscribers. Getting your
daily newspaper is similar to checking your RSS feeds because you
only get information at regular intervals. (Of course you could keep
buying newspapers every 'x' hours, i.e. keep refreshing your RSS
reader so often). This is as inefficient as it sounds. For
instance, you never know if a newspaper actually has new stuff, you
need to buy and read it before you realize there's nothing new. And
there will be always be a delay (unless you are really lucky) between
the time something happens and you read it in the your 'n'th
newspaper of the day.
PubSub solves this problem by having a system where subscribers subscribe to
nodes and the publisher notifies its subscribers when something happens. Nodes
are like TV channels, so you only subscribe to the channels you are interested
in.


How it can be useful to you:
In addition to the points mentioned in the quote, pubSub can be used
in your project to send instant notifications, maintain the same
state across multiple sessions, or monitor live activity.


Introducing django-pubsub:
django-pubsub allows you write PubSub enabled applications easily.
This demo using django-pubsub shows how to use the module. The stack used
in the demo is:

fcgi
nginx
ejabberd with mod_http_bind and mod_pubsub

You can register a model with the pubsub app and all registered models
will automatically send out notification (with the payload) on save
to the default node.
Then you can use the PubSub client provided in media to subscribe and
receive events from the registered models.
The default payload includes an XML serialized instance of the model.
You can use jquery to extract required fields from the payload.
You can also directly use pubsub.publish to publish any item to any
node. An example is provided in the demo included with the app.


How it works:
Since XMPP and HTTP speak different languages, you need some kind of a
bridge to connect these two. This can be achieved using BOSH.
The BOSH bridge is provided by ejabberd's mod_http_bind
module. (You can also use Punjab for this purpose)
If you are using mod_http_bind, you will also need to setup
appropriate url forwarding using nginx because mod_http_bind listens
on the port 5280 and will not be able to communicate with the client
on port 8080.
Strophe will use the BOSH url to talk to ejabberd. Our PubSub
client which is written using Strophe will then be able to subscribe
and receive notifications from the server.
On the server side, the pubsub app uses xmpppy to send out the correct Iq
stanzas on save or publish. The client will receive these stanzas
on subscribing to the appropriate node.


Conclusion:
If your app needs real-time updates, push notifications, and scalable
infrastructure, you can make use of XMPP and Strophejs.


References:
ejabberd's blog post on related topic:
http://www.process-one.net/en/blogs/article/introducing_the_xmpp_application_server
a similar project using comet, orbit and twisted instead of xmpp:
http://www.clemesha.org/blog/realtime-web-apps-python-django-orbited-twisted



Seven reasons why you should switch to Vim
So you want a better IDE for developing django, huh? Why not give good old vim a try?
Use pathogen to maintain your vim plugins (and sanity).
Using this, you can clone the repositories listed here to .vim/bundle/ and start using them
right away.
Also, consider adding your .vimrc and .vim to a repository. Include .vimrc inside .vim and
symlink .vim/.vimrc to ~/.vimrc to version control your .vimrc.
My vim files can be found here. Also, here's an imgur album demonstrating
these plugins in action.
1. Syntax highlighting for django templates
Starting from vim 7.1, syntax highlight for django templates works out
of the box. (Check your vim version using vim --version or :version within vim)
If you are on an older version, use this plugin
2. Django snippets with snipmate
SnipMate provides commonly used "snippets". This eliminates a lot of
boiler plate code genreally required to start a django project from scratch.
Django specific snippets can be found at robhudson's repository
Its very easy to write custom snippets in case you need them.
3. On-the-fly error checking with pyflakes
This one is not django specific, but it can save a lot of time spent debugging typos or silly mistakes.
PyFlakes detects errors in your python code and highlights the 
offending lines so you can easily rectify them.
4. Git integration with fugitive
Fugitive is a git plugin for vim.
This one integrates seamlessly with my workflow. You can add/reset files, commit them, view diff, blame, logs all 
within vim!
5. Auto-completion using pysmell
Pysmell is great for autocompleting your django project code.
6. Browse code with taglist
Taglist allows you to view your code structure and jump between
classes/functions.
7. Write faster html using sparkup
Sparkup allows you write zen code
Other tips:


set makeprg to python manage.py syncdb. Run :make from your vim to start syncdb


use nerdtree/lusty explorer for file
management


use vimango for navigating your django project




Importing wordpress
We now have a way to import wordpress to blogango, and all our old posts are on this blog.
Thanks for reading.


Getting trending Github projects via YQL
Github.com/explore allows you to see the trending Github topics. They use repopular.com for this,
which use twitter retweets to find out the popular Github repos.
Since neither Repopular, nor Github have a RSS of these trending repos, I wanted to get a list of these. Here is how easy it is with 
YQL.


How we do it


Go to YQL console. Give the SQL query to get the data from the webpage. 


where url="repopular.com" and css="div.pad a" is the magic which select the webpage, and also what DOM elemenst we are interested in.


We get this data in JSON format which is munged to get the list of links.


This list of links is passed via is_github_project which gets me just the Github projects.


And we are done.




PS. YQL is amazing.


Django is not flexible
Django is not flexible at all because you can not do simple
things like. 




Using various authentication mechanisms.

You can authenticate via Username,
emails,
Facebook, Twitter or any combination of these.


Using different database backends.

Use MySQL, PostgreSQL,
Non-relational databases,
more
or a combination.


Use different mail sending strategies.

Send mail via smtp,
queue to database,
log to files
and more.


Different file storage method.

Store locally, FTP,
Amazon S3
or write your own.


Store messages in Sessions, cookies or others.


Cache things in Files,
DB,
Memcache
or just fake it.


Save sessions in files, DB, Memcache or cookies.


Use Jinja,
Mako or Gensi as templating library.



One of the most lingering criticisms of Django has been
that it is not flexible. People have criticised its design
as too-tightly coupled, hard to extend, and inflexible.
As a result, people have recommended to roll their own
using SQLchemy, Jinja or web.py.


I think this view is unfounded and wrong. You would be hard
pressed to get the choices and flexibility which Django offers.
Eg, while development I run with Sqlite, Dummy caching, and emails
on console, while on production I switch to PostgreSQL, Memcache, and
queued emails.


Many apps use this backend pattern themselves.
Django registeration,
Django-celery
and our own merchant.


I have always been annoyed at Django is inflexible meme, and the recent 
Charles Leifer's post
put various Django plug points to make writing this too post easy to not do it.




Rails and Django commands : comparison  and conversion
The most commonly used Rails commands and their Django equivalents
Rails                   | Django
rails console            | manage.py shell                 

rails server             | manage.py runserver                

rake                     | None                            

rails generate           | None                            

rails dbconsole          | manage.py dbshell               

rails app_name           | django-admin.py startproject/manage.py startapp          

rake db:create           | manage.py syncdb    


The salient points to note are,

Django has all commands via manage.py, Rails has it broken into rails and rake.
Overall there are more Rails+Rake commands available than Django commands
There is no one to one mapping between Rails and Django commands.
Eg. There are no equivalent to rake doc:* or rake notes in Django.

Similarly there is no equivalent to manage.py createsuperuser in rails.
References
http://docs.djangoproject.com/en/dev/ref/django-admin/
http://guides.rails.info/command_line.html
http://github.com/uswaretech/Acts-as-Django/blob/master/commands.markdown


Doing things with Django models - aka - Django models tutorial
Django abstracts most of the actions you would be doing with the Database.
What it doesn't abstracts, and doesn't try to abstract is the Database modelling part. This is a quick tutorial
describing to how model your data in Django models.py, and how to access and modify them.
Consider a hypothetical HR department, which wants you to build an application to track and manage their processes.
They have employees who work for a department, contractors who work for multiple department. Let's see how you
you would do that in Django.
from django.db import models

class Employee(models.Model):
    name = models.CharField(max_length = 100)
    department = models.ForeignKey("Department")

class Department(models.Model):
    name = models.CharField(max_length = 100)

class EmployeeHistory(models.Model):
    employee = models.OneToOneField(Employee)
    date_joined = models.DateField()
    marital_status = models.BooleanField()

class Contactor(models.Model):
    name = models.CharField(max_length = 100)
    departments = models.ManyToManyField(Department)



Let's see the type of relationship we created here.
An Employee has a Many-to-one relationship with Department, (i.e. One department will have many
Employee, but one employee will have a single departments.)
So Employee has a field department = models.ForeignKey("Department"). See that the ForeignKey field
was added on the class which has in many. In database, this creates a FK from in Employee table which refernces
Departments.
A Contractor has many-to-many relationships with Department, so it has a ManyToMany field. This field can be created
on either classes. At a database level this creates a new table which has a FK to both the tables.
A Employee has one-to-one relationship with EmployeeHistory. The thing to note here is that whatever we could do with
OneToOneField, we can do by including the fields in the Other Class directly. However when there are fields which are only rarely
needed with a given model, it is useful to separate them via a one-to-one field. The one to one field can be on either class.
Accessing objects.
Getting a specific employee.
Employee.objects.get(pk = someval)
Employee.objects.get(name= someval)



Given an department get all employees.
department.employee_set.all()



The department gets an attribute name <FKClass>_set.
Given an employee, get all colleagues.
employee.department.employee_set.objects.all()



To get siblings, get parents, and get all children.
Given an department, get number of employees.
department.employee_set.all().count()



Ok so the HR department bosses are happy with what they see, and ask you to track this data,
who is the manager of each employee, who is the HOD of each department and when did each employee took leave.
from django.db import models

class Employee(models.Model):
    name = models.CharField(max_length = 100)
    manager = models.ForeignKey("self", blank = True, null = True)
    department = models.ForeignKey("Department")

class Department(models.Model):
    hod = models.ForeignKey("Employee")
    name = models.CharField(max_length = 100)

class EmployeeHistory(models.Model):
    employee = models.OneToOneField(Employee)
    date_joined = models.DateField()
    marital_status = models.BooleanField()

class Contactors(models.Model):
    name = models.CharField(max_length = 100)
    departments = models.ManyToManyField(Department)

class EmployeeLeave(models.Model):
    leave_taken = models.DateField()
    employee = models.ForeignKey(Employee)



So we now have new fields hod = models.OneToOneField("Employee") in department and
manager = models.ForeignKey("self", blank = True, null = True) and a new model EmployeeLeave. Let su see
the new realtions,
As one Department will have one Employee as hod, and Employee can head at max one Department, we have a one to one relationship

As many Employee will report to another Employee, so we have a FK on Employee, referencing self.
As Employee will take many EmployeeLeave, EmployeeLeave has a FK to Epployee
Let us see some queries.
Get all leaves taken by an employee this year
import datetime
today = datetime.date.today()
start_of_year = datetime.datetime(today.year, 1, 1)
leaves_taken = employee.employeeleave_set.filter(leave_taken__gt=start_of_year, leave_taken__lt = today)



Get a list of all HODs.
#As there are going to be as many Employees as departments
departments = Department.objects.all()
employees = [department.hod for department in departments]



Get a list of all departments a contractor works for.
contractor.department_set.all()



Get a list of all contractor who work for a department.
department.contractor_set.all()



Note that each side of a many to many relationship get a Manager.
List of all managers in a given department.
Todo#(Can't think of any one query way to do this. If you know, let me know :) )



List of all leaves taken in a given department.
EmployeeLeave.objects.filter(employee__department = department)



None that we used double underscores __ to do a filtering on a field across Entities.
List of all employees which joined a given department this year.
Employee.objects.filter(department = department, employment_history__date_joined__gte=start_of_year, )



Note that we used a double underscore __ twice, first to go across entities, and then to define the type of filter(__gte).
Also we specified two filter conditions, so they were ANDed together.
List all employees which either report too a given employee, or joined before him.
Employee.objects.filter(Q(manager = employee)|Q(employee_history__date_joined__lt = employee.employee_history.date_joined))



Note the new construct Q, they are used to specify complex boolean operations. Here we used the | (or operator) to specify or condition.

Do you subscribe to our rss feed? The new feed has full text feed and many other goodness. So make sure you are subscribed to the new feed with extra fortified vitamins. Subscribe now.


Wordpress and Django: best buddies
Summary:
How to integrate a non Django database system in your Django code, using Wordpress
as example. The completed code is available at github or you can see some screnshots

Though there are quite a few good Django blog applications, our blog is based on
Wordpress. A number of plugin's
make moving to a Django based app a bad decision
for us, and not in the spirit of "best tools for the job".
We moved the other way, and decided to use Django to admin the
Wordpress database. The completed code is available on Github
It is not too hard, with the the builtin Django commands. Django provides the
inspectdb command which allows you to build your models.py from an existing
non Django database.
Here we will see the steps followed for Wordpress, but it would be about the same for all
systems.
Take a back up of wordpress
mysqldump -u wordpress_user -p --database wordpress_database > data.sql



Create a new project, and set its settings to use the Wordpress database.
    DATABASE_ENGINE = 'mysql'           # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
    DATABASE_NAME = ''             # Or path to database file if using sqlite3.
    DATABASE_USER = ''             # Not used with sqlite3.
    DATABASE_PASSWORD = ''         # Not used with sqlite3.
    DATABASE_HOST = ''             # Set to empty string for localhost. Not used with sqlite3.
    DATABASE_PORT = ''             # Set to empty string for default. Not used with sqlite3.



Get the initial models.py
./manage.py inspectdb > models.py



This will create all the database tables in a form Django can understand. Here is what this command creates for a
my Wordpress installation with the YARPP plugin. http://gist.github.com/278962
Create a new app and put this models.py there.
With this done, you can treat the non Django part as a
standalone application. Since Wordpress appends all its table with wp_ prefix,
we name this applications wp to maintain table name compatibility with Django naming
conventions.
You will notice that all models have the db_table populated, so we can rename tables, without changes to the database.
Differences between Wordpress and Django style naming.
At this point you will notice some differences in how Django names things (in a
best practice sort of way), and how Wordpress does it.
a. Django table and model class name are (generally) singular. eg class Post(models.Models) leads to table app_post.
Wordpress tables are (most of them) named plural eg wp_posts.
b. Django attributes are generally named without the table name part. Eg
class Comment(models.Model):
    author_name = models.TextField()
    content = models.TextField()



Wordpress is explicit here and includes the table prefix with attributes.
mysql> desc wp_comments;
+----------------------+---------------------+------+-----+---------------------+----------------+
| Field                | Type                | Null | Key | Default             | Extra          |
+----------------------+---------------------+------+-----+---------------------+----------------+
| comment_ID           | bigint(20) unsigned | NO   | PRI | NULL                | auto_increment | 
| comment_post_ID      | bigint(20) unsigned | NO   | MUL | 0                   |                | 
| comment_author       | tinytext            | NO   |     | NULL                |                | 
| comment_author_email | varchar(100)        | NO   |     |                     |                |

.....



I believe this is due to the way you would generally be using the code. In Django you would do
comment.author where being explicit doesn't add any value, while in Wordpress, you would use,
select comment_author, post_title ... from wp_comment, wp_post ... where join, where being explicit
is useful.
You can decouple the Django and database names by using the db_table and db_column attributes.
We choose to rename the Class names to match Django conventions while we let the column names remain the same.
Add Admin and other Django niceties.
Wordpress doesn't (seem to) have foreign key constraints setup correctly, and
uses  bigint(20) unsigned without foreign key constraints to refer to referred entities.
This means Django creates all ForeignKeys as IntegerFields.
Modify them to use ForeignKey instead. Also add __unicode__, to your classes.
Add an admin.py to register all your classes. 
And you are done! Now you can access, and work with your Wordpress data inside Django
and Django admin.

There are a few more things which will allow a easier Wordpress setup.
Create template tags to show the latest posts and comments.
@register.inclusion_tag("wp/recent_posts.html")
def show_posts(num_comments):
    return {"posts": Post.objects.filter(post_type="post", post_status="publish").order_by("-post_date")[:num_comments]}



So you can see that there is nothing Wordpress specific we need too do here.
Create a better admin.
Add ModelAdmin to generally used models.



Allows accessing attributes via the Django style names.
If you override __getattr__, you can access
the attributes via other names. Eg in the current setup you need to do comment.comment_content, comment.comment_author etc,
while we would like to do comment.content  and comment.author as a shortcut.
class WordPressModel(object):    
    def __getattr__(self, v):
        if v in self.__dict__:
            return self.__dict__[v]
        else:
            new_v = "%s_%s" % (self.__class__.__name__.lower(),  v)
            if new_v in self.__dict__:
                return self.__dict__[new_v]
            else:
                raise AttributeError



It is highly debatable whether this is a good idea :), but it is too convenient right now not to test this method out.

Here are some screenshots.






Do you subscribe to our feed? We recently made a full text feed available, so if you are using the old feed, you should change it. Subscribe now.


Doing things with Django forms
Forms are one of the best features of Django. (After models, admin, url routing etc :) ). Here is a quick tutorial describing how to do things with Django forms.

Basic form

Prob. You want to show a form, validate it and display it.
Ans. Create a simple form. 
class UserForm(forms.Form):
    username = forms.CharField()
    joined_on = forms.DateField()



This wil take care that the form is displayed with two text field,
and a value for them are filled in, and the second field has correct formatting for a date.
2.
Prob. Create a form which has values populated depending upon a runtime value,
eg you might want to show only the values corresponding to the current subdomain.
Ans. Create a form with an custom __init__.
class UserForm(forms.Form):
    username = forms.CharField()
    plan = forms.ModelChoiceField(queryset = Plan.objects.none())

    def __init__(self, subdomain, *args, **kwargs):
        self.default_username = default_username
        super(UserForm, self).__init__(*args, **kwargs)
        self.fields['plan'].queryset = Plan.objects.filter(subdomain = subdomain)



Here in the __init__ we are overriding the default queryset on field plan. Any of the attributes can similarly be overridden.
However the self.fields is populated only after super(UserForm, self).__init__(*args, **kwargs) is called.
3.
Prob. The same form is used in multiple views, and handles the cleaned_data similarly.
Ans. Create a form with a custom .save()
class UserForm(forms.Form):
    username = forms.CharField()

    def save(self):
        data = self.cleaned_data
        user = User.objects.create(username = data['username'])
        #create a profile
        UserProfile.objects.create(user = user, ...some more data...)



You could have called this method anything, but this is generally called save, to maintain similarity with ModelForm
4.
Prob. You need to create a form with fields which have custom validations.
Ans. Create a form with custom clean_fieldname
class UserForm(forms.Form):
    username = forms.CharField()

    def clean_username(self):
        data = self.cleaned_data
        try:
            User.objects.get(username = data['username'])
        except User.DoesNotExist:
            return data['username']
        raise forms.ValidationError('This username is already taken.')



Here we can validate that the usernames are not repeated.
5.
Prob. You want to create a field which has cross field validations.
Ans. Create a field with a .clean
class UserForm(forms.Form):
    username = forms.CharField()

    password1 = forms.PasswordField()
    password2 = forms.PasswordField()

    def clean(self):
        data = self.cleaned_data
        if "password1" in data and "password2" in data and data["password1"] != data["password2"]:
            raise forms.ValudationError("Passwords must be same")



6.
Problem.
You need a form the fields of which depends on some value in the database.
Eg. The field to be shown are customisable per user.
Create a form dynamically
def get_user_form_for_user(user):
        class UserForm(forms.Form):
            username = forms.CharField()
            fields = user.get_profile().all_field()
            #Use field to find what to show.



This post provides much more details
7.
Prob. You need to create a Html form which writes to multiple Django models.
Ans. Django forms are not a one to one mapping to Html forms. 
#in forms.py
class UserForm(forms.ModelForm):
    class Meta:
        model = User
        fields = ["username", "email"]

class UserProfileForm(forms.ModelForm):
    class Meta:
        model = UserProfile

#in views.py
def add_user(request):
    ...
    if request.method == "POST":
        uform = UserForm(data = request.POST)
        pform = UserProfileForm(data = request.POST)
        if uform.is_valid() and pform.is_valid():
            user = uform.save()
            profile = pform.save(commit = False)
            profile.user = user
            profile.save()
            ....
    ...

#in template
<form method="post">
    {{ uform.as_p }}
    {{ pform.as_p }}
    <input type="submit" ...>
</form>



8.
Prob. You want to use multiple forms of same type on one page.
Ans.
a. If you want a datagrid style ui, use formset.
from django.forms.formsets import formset_factory
forms = formset_factory(UserForm, extra = 4)
#



Formsets are described much more comprehensively here
b. If you do not need a datagrid style ui, use prefix argument to forms.
Eg. you have a survey app, and you want a page with all questions from that survey displayed.

#IN views.py
def survey(request, survey_slug)
    ...
    questions = survey.questions.all()
    question_forms = []
    for question in questions:
        qform = QuestionForm(question=question, prefix = question.slug)
        question_forms.append(qform)
        ...
    if request.method == "POST":
        for question in questions:
            qform = QuestionForm(question=question, prefix = question.slug, data = request.POST)
        #Validate and do save action
        ...
    ...




Do you subscribe to our feed? We recently made a full text feed available, so if you are using the old feed, you should change it. Subscribe now.


django-forum
twitter ready version:
We have released a Django forum application, with some cool features not in any other Django based forum. You can get it here or see it in action. 
blog version
There are quite a few Django based forum applications, so why another? Its a bit of a rhetorical question,  as the answer is "none of them met my needs exactly, so we created my own", as you might expect.
Without further ado here is a list of features. It is available on uswaretech.com/forum/.

Based on, inspired by and ripped from PunBB. (Thanks!)
PunBB like 3 phased hierarchy [Forum]->Subforum->Topic->Post
Tightly integrated with Socialauth. (Screenshots)
Gravatar support
Moderation features (Close, stikify, make announcement etc.)
Ajax based posting

We are starting our forums based on this app, so we plan to be supporting and developing this for foreseeable future. Fork this on Github or check this out now.

Screenshots
Main page
 
 

Post page
 
 

Login Page
 
 


We build amazing web apps. Contact us


Using bpython shell with django (and some Ipython features you should know)
What is bpython?

bpython is a fancy interface to the Python interpreter for Unix-like operating system.

says the bpython home page. It provides syntax highlighting, auto completion, auto-indentation and such stuff.
Unlike iPython, which implements then entire shell functions and emulates the standard python shell, and adds enhancements, bpython just adds features on top of the existing python shell functionality, using the curses module. 


The "killer feature" of bpython is, as you can see from the image above, the IDE like prompting of the function parameters and the doc string of the function dynamically. I have always thought, what IntellijIDEA is to Java, IPython is to Python*. But bpython makes it more so, in terms of the code completion, which adds a lot of value to a ramping up developer.
The only Python IDE that provides Source Assistant and Go-To-Source functionality conveniently, is the commercial one, Wing IDE Professional. Even with that, since all the defined models are replaced (by using meta-classes) in the runtime, that source assistant is not perfect. Newer versions of Wing seems to claim to obtain data dynamically at the runtime, but its not always comfortable to write code, keeping the IDE in a breakpoint. But hey, bpython provides all these for free!
But I already use Ipython
You do? So do I. In fact, I am a Power Ipython user, I use all kinds of %magic functions: %whos, %logstart, %bookmark, %ed  and sometimes use Ipython as an alternative even to bash: %cd, %ls, %ll. And even for doc testing: %doctest_mode, copy-pasting %hist -n or running some code module in the interpreter namespace: %run -i and searching code %psearch. You can also give any arbitrary bash(or your default shell) command by escaping it with !. Oh, ?, ?? and / are of course the starters.
In fact did you know, you could get to ipython shell within any arbitrary place within your django code? Just use the following:
from IPython.Shell import IPShellEmbed
ipython = IPShellEmbed()



and then call ipython() anywhere within your view, model, forms and you will be dropped to the shell. Its like ipdb.set_trace() only better (unless you want to step through, that is).
The Python readline bug
The Python version 2.6.4 (the version that is shipped with Ubuntu 9.10), introduced a readline bug that adds spaces after tab completion. This bug also affects the Ipython, as it uses the same readline module. If you spend a lot of time on the shell (if you use python, you must, right?), it is very annoying to backspace after each tab, all the time.
Although the bug got fixed pretty soon, it hasn't yet made it to any release that ubuntu updates to. There are ways to workaround this problem, by fixing it at the python level and at the Ipython level, many of them are discussed on the corresponding Ipython bug 
Using bpython with django
Now that you want to use bpython with django, either because you like the auto completion, or because you find the read line bug annoying (and don't want/care-enough to patch your python locally), or you just want to try it, how to do it?
django manage.py shell unfortunately, drops only into the ipython shell (if you have ipython installed), and there is no straight forward way to get it to drop to bpython.
But there is still a way to use bpython with django. Just modify your ~/.bashrc to define a python startup environment variable
export PYTHONSTARTUP=~/.pythonrc



And within it, setup the django environment, that is, do it here manually the thing that python manage.py shell would do for you:
#.pythonrc
try:
    from django.core.management import setup_environ
    import settings
    setup_environ(settings)
    print 'imported django settings'
except:
    pass



This way, bpython or even just python on the command line, should import the django environment for you.
Importing all models automatically into the shell
But then, if you are possibly already used to python manage.py shell_plus (If you are not, you should be.) that is defined by django_command_extensions.
So while we are at setting up the django environment, lets just also import all the models into the shell namespace, so that it is convenient to work. Following is some ugly quick hack to get it done.


This can of course be improved upon. If you do it, just leave it in the comments! In fact it would be good if it also includes from startup.py import * in a try catch, so that, you can put some extra code into startup.py. Saving into startup.py from within a bpython shell is just a Ctrl+s anyway. That way each time you get to the shell, you can have the same expected environment; and it is quite easy to change that file.
*I know, I know, IPython doesn't refactor code, nor build, nor does a million things that Intellij does, but Python is not Java and basically both intend to enhance developer productivity. And I spend a lot of time on the IPython shell and find it to be a tool just like Intellij is a tool, for Java.


Python metaclasses and how Django uses them
Foss.in is without doubt India's largest FOSS technology conference. Lakshman gave a talk today on "Python metaclasses and how Django uses them". Here are the slides from that talk.
Doing magic with python metaclassesView more documents from Usware Technologies.


[Edit]
Some reactions,

http://twitter.com/jaideep2588/status/6295483833

http://twitter.com/kunalbharati/status/6296572939
And the talk images, http://twitpic.com/rxhn7



You should follow us on twitter and Subscribe to our blog


Django quiz
A quick django quiz. Answers available tomorrow. Get it as a text file (django-quiz) or on google docs or read below.
### Easy

1. You have a class defined as

    class Post(models.Model):
        name = models.CharField(max_length=100)
        is_active = models.BooleanField(default=False)

You create multiple objects of this type. If you do
Post.objects.get(is_active=False),

what exceptions is raised?

a. MultipleObjectsReturned
b. ManyObjectsReturned
c. SyntaxError
d. MultipleModelReturned
e. ManyModelReturned

2. Where is the function render_to_response defined?

a. django.views
b. django.shortcuts
c. django.templates
d. django.contrib.templates
e. django.contrib.shortcuts

3. What is the default name for the table created for model named Post
in application blog

a. post_blog
b. blog_post
c. postblog
d. blogpost
e. Postblog

4. How do you send a 302 redirect using django?

a. return HttpRedirectResponse()
b. return HttpResponseRedirect()
c. return HttpRedirectResponse(permanent=True)
d. return HttpResponseRedirect(permanent=True)
e. return HttpRedirectResponse

5. In django.contrib.auth, Users passwords are kept in what form?
a. Plain text
b. Hashed
c. Encrypted
d. Hashed then encrypted
3. Encrypted then hashed

6. Which of the following is correct way to find out if a request uses
HTTP POST method?

a. request.is_post() == True
b. request.METHOD == 'post'
c. request.METHOD == 'POST'
d. request.method == 'post'
e. request.method == 'POST'

7. Generic views have access to request context.

a. True
b. False
c. Default is True but can be set to False by GENERIC_REQUEST_CONTEXT in settings.
d. Default is False but can be set to True by GENERIC_REQUEST_CONTEXT in settings.
e. Can not be determined without extra information.

8. The current released version of Django is

a. 0.96
b. 1.0
c. 1.1
d. 1.2
e. Vyper logix 2.0

9. Which of these is a correct context_processor?

a.

def featured_posts(request):
    return Post.objects.filter(is_featured=True)

b.

def featured_posts(request):
    return {'featured_posts': Post.objects.filter(is_featured=True)}

c.

def featured_posts(request, response):
    return Post.objects.filter(is_featured=True)

d.

def featured_posts(request, response):
    return {'featured_posts': Post.objects.filter(is_featured=True)}

e.     
def featured_posts(request, response):
    response.write('featured_posts'= Post.objects.filter(is_featured=True))

10. Which of the following is a valid Middleware?

a.

class LatestPostMiddleware:
    def process_request(request):
        request.latest_post = Post.objects.latest()

b.

class LatestPostMiddleware(django.middlewares.BaseMiddleWare):
    def process_request(request):
        request.latest_post = Post.objects.latest()

c.

class LatestPostMiddleware(django.middlewares.BaseMiddleWare):
    def process_request(request):
        return {'latest_post': Post.objects.latest()}

d.

class LatestPostMiddleware():
    def process_request(request):
        return {'latest_post': Post.objects.latest()}

e.

class LatestPostMiddleware():
    def process_request(request):
        request.write('latest_post'= Post.objects.latest())

#Moderate

11. In the model below which line can be removed for the class to work

class Post(models.Model):
    name = models.CharField(max_length = 100)#Line 1
    desc = models.TextField()#Line 2
    post = models.ForeignKey(Blog)#Line 3
    blog = models.ForeignKey(Blog)#Line 4

a. Line 1
b. Line 2
c. Line 3
d. Line 4
e. Either of line 3 or 4

12. Who can access the admin site in Django?

a. user with is_super_user True
b. user with is_staff True
c. user with is_admin True
d. Either of a or b
e. Either of a, b, c

13. Which of the following code is closest to login_required decorator in Django?

a.

def login_required2(request):
    if request.user.is_authenticated():
        return True
    else:
        return False

b.

def login_required2(request, view_func):
    def check_login(request):
        if request.is_authenticated() and request.has_permissions(request.REQUIRED_PERMISSIONS):
            return view_function(request)
        else:
            return HttpResponseRedirect('/login/')
    return check_login(view_func)

c.

def login_required2(request, view_func):
    def check_login(request):
        if request.is_authenticated():
            return view_function(request)
        else:
            return HttpResponseRedirect('/login/')
    return check_login(view_func)

d.

def login_required2(view_func):
    def new_func(request):
        if request.user.is_authenticated():
            return view_func(request)
        else:
            return HttpResponseRedirect('/login/')
    return new_func

e.

def login_required2(view_func):
    def new_func(request):
        if request.user.is_authenticated() and request.has_permissions(request.REQUIRED_PERMISSIONS):
            return view_func(request)
        else:
            return HttpResponseRedirect('/login/')
    return new_func

14. Which of the following is a valid method to model Many to many relation ship in Django?

a.

class Foo(models.Model):
    bar = models.ManyToManyField(Bar)

class Bar(models.Model):
    foo = models.ManyToManyField(Foo)

b.

class Foo(models.Model):
    bar = models.ForeignKey(Bar)

class Bar(models.Model):
    foo = models.ForeignKey(Foo)

c.

class Foo(models.Model):
    bar = models.ManyToManyField(Bar)

class Bar(models.Model):
    pass

d. both B and C

e. All a, b, c

15. Which of the following is not included by default in TEMPLATE_CONTEXT_PROCESSORS

a. django.core.context_processors.auth
b. django.core.context_processors.debug
c. django.core.context_processors.i18n
d. django.core.context_processors.media
e. django.core.context_processors.request

16. Which of these is the currect way to validate uniqueness of a field named "slug" in a form
subclassing django.form.Forms.

a.

def clean(self):
    try:
        Post.objects.get(slug = self.cleaned_data['slug'])
        raise forms.Error(ValidationError, 'This slug already exists')
    except Post.DoesNotExist:
        return self.cleaned_data

b.

def clean(self):
    try:
        Post.objects.get(slug = self.cleaned_data['slug'])
        raise ValidationError('This slug already exists')
    except Post.DoesNotExist:
        return self.cleaned_data

c.

def clean_slug(self):
    try:
        Post.objects.get(slug = self.cleaned_data['slug'])
        raise forms.Error(ValidationError, 'This slug already exists')
    except Post.DoesNotExist:
        return self.cleaned_data

d.

def clean_slug(self):
    try:
        Post.objects.get(slug = self.cleaned_data['slug'])
        raise ValidationError('This slug already exists')
    except Post.DoesNotExist:
        return self.cleaned_data

e.

17. To add custom commands to your project setup, you need to,

1. Add management/command/commandname.py to your django app.
2. Add management/command/commandname.py to your django project.
3. Add command/commandname.py to your django app.
4. Add command/commandname.py to your django project.
5. Add management/command/manage.py to yout django project.

18. You want to enable caching for Django pages for AnonymousUser, which of these
is a valid way to do this.

a.

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.middleware.cache.CacheMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
)

b.

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.cache.CacheMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
)

c.

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.cache.CacheMiddleware',
)

d.

MIDDLEWARE_CLASSES = (
    'django.middleware.cache.CacheMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
)

e.

MIDDLEWARE_CLASSES = (
    'django.middleware.cache.CacheMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
)

19.

Consider the urlconf given below

    from django.conf.urls.defaults import *

    urlpatterns = patterns('news.views',
        (r'^articles/2003/$', 'year_archive_2003'),
        (r'^articles/(\d{4})/', 'year_archive'),
        (r'^articles/2003/(\d{2})/$', 'month_archive_2003'),
        (r'^articles/(\d{4})/(\d{2})/', 'month_archive'),
        (r'^articles/(\d{4})/10/', 'month_archive_october'),
    )
What view is called when accessing /articles/2003/10/

a. year_archive_2003
b. year_archive
c. month_archive_2003
d. month_archive
e. month_archive_october

20.

Consider the urlpatterns.

    urlpatterns = patterns('blogapp.views',
        (r'^list/(?P<year>\d{4})/(?P<month>[a-z]{3})/$','archive_month'),
    )

Which if the following is a valid view function which will be called on accesing
/list/2009/jun/

a. def archive_month(request):
        ...

b. def archive_month(request, *args):

c. def archive_month(request, **kwargs):

d. def  archive_month(request, year, month,):

e. Both c and d

# Hard

None yet




Follow us on twitter or subscribe to our blog


Django for a Rails Developer
This is not yet another Django vs Rails blog post. It is a compilation of notes I made working with Django after having worked on Rails for years.
In this post I want to give a brief introduction to Django project layout from a Rails developer point of view, on what is there, what is not there and where to look for things. It should help a rails developer working on django be able to find the necessary files and underatnd the layout of the project files.
Once you have both the frameworks installed on your system you can create the projects using the commands
# creating a rails project
rails rails_project
# creating a Django project
django-admin.py startproject django_project



Lets look at the files and structure created by the respective frameworks
#rails_project
README
Rakefile
app/
    controllers/
        application_controller.rb
    helpers/
        application_helper.rb
    models/
    views/
        layouts/
config/
    boot.rb
    database.yml
    environment.rb
    environments/
        development.rb
        production.rb
        test.rb
    initializers/
        backtrace_silencers.rb
        inflections.rb
        mime_types.rb
        new_rails_defaults.rb
        session_store.rb
    locales/
        en.yml
    routes.rb
db/
    seeds.rb
doc/
    README_FOR_APP
lib/
    tasks/
log/
    development.log
    production.log
    server.log
    test.log
public/
    404.html
    422.html
    500.html
    favicon.ico
    images/
        rails.png
    index.html
    javascripts/
        application.js
        controls.js
        dragdrop.js
        effects.js
        prototype.js
    robots.txt
    stylesheets/
script/
    about
    console
    dbconsole
    destroy
    generate
    performance/
        benchmarker
        profiler
    plugin
    runner
    server
test/
    fixtures/
    functional/
    integration/
    performance/
        browsing_test.rb
    test_helper.rb
    unit/
tmp/
    cache/
    pids/
    sessions/
    sockets/
vendor/
    plugins/



that is huge....
lets look at the django project files
# django_project
__init__.py
manage.py
settings.py
urls.py



far lesser when compared to the rails project.
In fact a rails project comes with everything a web application needs. When I say everything I mean everything..... base application, routing, database configuration, development, test and production environment specific configurations and their respective log files,  javascript, test, some standard html files and some helpful scripts for developing.
Why then does a Django project have so less number of files? It has got to do with the Django's philosophy and the concept of applications. The django project is not complete without the application, so lets create a application inside the project and have a look at the structure
django-admin.py startapp app

# django_project
__init__.py
app/
    __init__.py
    models.py
    tests.py
    views.py
manage.py
settings.py
urls.py



even after including this, the number of files is still less than the rails project.
Lets break it down and relate both the frameworks. 

    
        
            
            Rails
            Django
        
    
    
        
            Configuration Files
            
                config/database.yml for database settings
                config/environments/
                development.rb for development specific settings
                production.rb for production specific settings
                test.rb for test specific settings
            
            
                settings.py, one file for everything, database configuration and any other configuaration or settings will be in this file
            
        
        
            URLs
            
                config/routes.rb
            
            
                urls.py
            
        
        
            Schema/Models
            
                db/schema.rb for the ruby version of db schema
                app/models/* for the models
            
            
                complete schema is not stored any where
                Under every application in your project, models.py file will contain the table specific schema stored as django models
            
        
        
            Management Commands
            
                ./script/server,  start server
                ./script/console, ruby console
                ./script/dbconsole, database console
                rake db:migrate, run database migrations
            
            
                manage.py is the file for all your tasks
                ./manage.py runserver, start server
                ./manage.py shell, python console
                ./manage.py dbshell, database console
                ./manage.py syncdb
            
        
        
            Application Code
            
                app/controllers/* will contain the application logic
            
            
                views.py file under each application folder is the place to write to your application logic, file can be named with any name, views.py is the general convention
            
        
        
            Application templates
            
                app/views//* is the place for the templates
            
            
                 default: looks inside 'templates' directory, under the application directory
                 looks in the directory specified by 'TEMPLATE_DIRS' in settings.py 
            
        




and lets have a look at the other things in rails_project
    

        Logging, unders the logs directory 
        Some default html files for some standard http errors, under the public directory 
        Rails has very good support for testing, for that bunch of files under tests 
        Vendor/Plugin, place for some third party plugins/applications. 
    

Missing pieces in Django (for a rails developer)
    

        Scaffold magic
        Generate commands
        Migrations
    

and what is Django is providing by default? Sorry no extra files in the project; but you will get an authentication system and Django's killer feature, admin, just by modifying your 'INSTALLED_APPS' in your settings. It is similar to the plugins feature in Rails, Django's philosophy of resusable apps helps you in getting any particular functionality into your project.
Following is a list of what I like in Django (and associated apps):
Admin (almost everybodys favourite)
Generic Views, helps from writing a lot of repetetive code, James Bennett's blog on generic views
Django DB API & QuerySets, (chaining and the filters that are missing in ActiveRecord)
Forms and above all, my favorite yet: ModelForm.
You can get the migrations feature in Django using South. Being an external app, it a bit of pain in setting it up to work with South. Other than that it is more like rails migrations
django-command-extensions
Search the Django way

Following are the questions that I keep getting, 
 
When a project will almost have a application why creating project & app has to be two different steps?
Why not create a 'templates' directoy and a 'base.html' either in project's directory or in the apps's directory, because creating the same templates directory and same base.html for every project is not DRY :) ?
 Why serving static files in development has to be a additional setup, as no developer wants to setup a server for serving static files, I am aware of 'django.static.serve' but still that is an additional setup, why not create a sample media directory and a url for the same in urls.py ?

let me know via comments, if you have any answers 
program used for listing the files in a directory
import os
import sys

try:
    directory = sys.argv[1]
except IndexError:
    directory = os.path.dirname(os.path.abspath(__file__))

def r_list_dir(directory, indent=0):
    dir_files = sorted([os.path.join(directory, file_name) for file_name in  os.listdir(directory)])
    for item in dir_files:
        if os.path.isdir(item):
            print " " * indent + os.path.split(item)[1] + '/'
            r_list_dir(item, indent+4)
        else:
            print " " * indent + os.path.split(item)[1]

r_list_dir(directory)



Writing your own template loaders
Django has three builtin template loaders which are used to get the templates for rendering.
TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.load_template_source',
    'django.template.loaders.app_directories.load_template_source',
#     'django.template.loaders.eggs.load_template_source',
)



Writing your template loader is a awfuly easy. It is a callable which

Returns a tuple of (openfile, filename) if it can find the template.
Raise TemplateDoesNotExist if the templates cannot be found.

The simplest template loader can be
from django.template import TemplateDoesNotExist
def load_template_source(template_name, template_dirs=None):
        try:
            return open(template_name).read(), template_name
        except IOError:
            raise TemplateDoesNotExist, template_name



if you put this to your template loaders directory,
TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.load_template_source',
    'django.template.loaders.app_directories.load_template_source',
    'dgrid.load_template_source'
#     'django.template.loaders.eggs.load_template_source',



You can do access a template using an absolute path.
In [6]: get_template('/home/shabda/abc.txt')
Out[6]: <django.template.Template object at 0x91c860c>

In [7]: temp = get_template('/home/shabda/abc.txt')

In [8]: temp
Out[8]: <django.template.Template object at 0x9358a0c>

In [9]: temp.render
Out[9]: <bound method Template.render of <django.template.Template object at 0x9358a0c>>




This is the first in the series of short and sweet Django posts we are going to do. You are interested, right. Do follow us on twitter or subscribe to our feed.

We build Amazing We Apps. Talk to us or email us at sales@uswaretech.com .


Django gotchas
This is announcement about our new work,  Django Gotchas, a teeny tiny ebook about commonly occurring gotchas with Django. Here is the readme copied from the project.

Django-gotchas is a collections of gotchas which happen commonly when you are working with Django.
They are some errors which I have made commonly or seen others do, these are not the errors which
happen because they are hard to reason about, these are those errors which hapen when you close your
eyes for a moment when coding.

This is still very much a work in progress, released in the spirit of release early, release often. Click here to get it, or fork it on Bitbucket
In Other news

We have slightly updated the Django design patterns, though not the public website.
cyrildoussin has done a lot of changes to Socialauth and made it much better.  You can get it here . Thank! We will be merging this soon.



Django-SocialAuth - Login via twitter, facebook, openid, yahoo, google using a single app.
TL;DR version: Here is an app to allow logging in via twitter, facebook, openid, yahoo, google, which should work transparently with Django authentication system. (@login_required, User and other infrastructure work as expected.) Demo and Code.Longer version follow:

We are releasing our new app. Django-Socialauth. This app makes it awfully easy,
to allow users to login your site using Yahoo/Google/Twitter/Facebook/Openid. A demo is available here.
This is released under an Attribution Assurance License. A copy of the same is
provided included with the code.
After installing this app, you can use @login_required on any view and users identified
via any means can access protected content.

We provide services to integrate and implement this, for a low price of USD 1600.
Please contact us at licenses@uswaretech.com to discuss your exact needs.

The README is copied here for convenience.
What it does.
Allow logging in via various providers.
Logging In
This is a application to enable authentication via various third party sites.
In particular it allows logging in via

Twitter
Gmail
Facebook
Yahoo(Essentially openid)
OpenId

Libs you need to install

python-openid (easy_install)
python-yadis (easy_install)
python-oauth(easy_install)

The API Keys are available from

http://www.facebook.com/developers/createapp.php
https://developer.yahoo.com/dashboard/createKey.html
https://www.google.com/accounts/ManageDomains
http://twitter.com/oauth_clients

How it works.
Openid: Users need to provide their openid providers. Talk to the providers and
login.

Yahoo: Yahoo is an openid provider. Talk to Yahoo endpoints. (Endpoint: http://yahoo.com)

Google: Google is a provider. Talk to them. (Endpoint: https://www.google.com/accounts/o8/id)

Facebook: Facebook connect provides authentication framework.

Twitter: We use Twitter Oauth for authentication. In theory, Oauth shouldn't be
used for authentication. (It is an autorisation framework, not an authentication one),
In practice it works pretty well. Once you have an access_token, and a name, essentially
authenticated.


References

Demo of app
Code for app
http://openid.net/developers/
http://developer.yahoo.com/openid/
http://code.google.com/apis/accounts/docs/OpenID.html
http://apiwiki.twitter.com/OAuth-FAQ
http://developers.facebook.com/connect.php

Below the hoods

For all providers(except Facebook) there are two urls and views. (start and done)
Start sets up the required tokens, and redirects and hands off to the correct
provider.
Provider handles authentication on their ends, and hands off to Us, providing
authorization tokens.
In done, we check if the user with these details already exists, if yes, we
log them in. Otherwise we create a new user, and log them in.

For all of these, we use standard django authenication system, with custom
auth_backends, hence all existing views, and decorators as login_required
will work as expected.
Urls
/login/ Login page. Has all the login options  
/openid_login/ AND /openid_login/done/  
/yahoo_login/ AND /yahoo_login/done/  
/gmail_login/ AND /gmail_login/done/  
/twitter_login/ AND /twitter_login/done/  
/facebook_login/done/ We dont have a start url here, as the starting tokens are  
set in a popup, as per FB Connect recommendations.



Implementation

Install required libraries.
Get tokens and populate in localsettings.py
Set the token callback urls correctly at Twitter and Facebook.
Add the OpenId middleware. Set the Authentication backends. (Set in localsettings.example.py)



A response to Dropping Django
Brandon Bloom yesterday wrote an interesting post titled
dropping Django. Despite a lot of hand waving(We needed a pragmatic template language to replace Django's idealistic one.),
it raises some valid questions, so here is a solution to some of them.
No support for hierarchical url creation.
The simplest representation of nested urls I can think of is a nested tuple. Lets represent,
the urls for a simple app by,
>>> tree_urls = ('', 'list',
...     ('edit/', 'edit', ('auto/', 'edit_auto')),
...     ('^add/', 'add'),
...     ('delete/', 'delete', ('hard/', 'delete_hard'))
...     )



Guess what, urls.py is just a python module which excepts a variable names urlpatterns.
Which means it is very easy to write a function which converts this nested structure to flat, structure.
Here is a quick attempt at that,
def merge(url):
    full_url=[]
    for i, el in enumerate(url):
        if i%2==0:    
            full_url.append(el)
    full_url = ''.join(full_url)
    return full_url

def combineflatten(seq):
    items= tuple(item for item in seq if not isinstance(item, tuple))
    yield items
    for item in seq:
            if isinstance(item, tuple):
                for yielded in combineflatten(item):
                    yield items+yielded

def generate_flat_urls(tree_urls):
    """
    >>> tree_urls = ('', 'list',
    ...     ('edit/', 'edit', ('auto/', 'edit_auto')),
    ...     ('^add/', 'add'),
    ...     ('delete/', 'delete', ('delete/', 'delete_hard'))
    ...     )

    >>> generate_flat_urls(tree_urls)
    [('^$', 'list'), ('^edit/$', 'edit'), ('^edit/auto/$', 'edit_auto'), ('^^add/$', 'add'), ('^delete/$', 'delete'), ('^delete/delete/$', 'delete_hard')]
    """
    return [('^%s$'%merge(el), el[-1]) for el in tuple(combineflatten(tree_urls))]



With this you can use hierarchical urls in urls.py as,
#All of urls.py
tree_urls = ('', 'list',
    ('edit/', 'edit', ('auto/', 'edit_auto')),
    ('^add/', 'add'),
    ('delete/', 'delete', ('delete/', 'delete_hard'))
    )

flat_urls = generate_flat_urls(tree_urls)

urlpatterns = patterns('app.views',
**flat_urls
)



No support for per user, per resource authorisation.
If you want to do this in a almost no-touching-the-existing-code way, replace all your render_to_response with,
def render_with_auth_check(request, payload, request_context, *args, **kwrags):
    for el in payload.itervalues():
        try:
            has_auth = el.check_auth(request.user)
            if not has_auth:
                raise AuthFailException
        except ValueError:
            pass #Not all objects have check_auth
    return render_to_response(request, payload, request_context, *args, **kwrags)



And enable this middleware,
class AuthFailHandlerMiddleware:
    def process_exception(self, request, exception):
        if type(exception) == AuthFailException:
            return render_to_response('accounts/login/', {}, RequestContext(request))



This assumes that all resources which are authorisation protected have a .check_auth,
but I cant see any way round that in any other way as well.
A different, and more robust way would be to write custom managers for all resources, which need authorization.
class ResourceAuthManager(models.Manager):
    def get(self, user, *args, **kwargs):
        res = super(ResourceAuthManager, self).get(*args, **kwargs)
        try:
            has_auth = res.check_auth(request.user)
            if not has_auth:
                raise AuthFailException
        except ValueError:
            pass #Not all objects have check_auth

    ...



The Django templating language is too constrained.
Despite believing that Django templating language hits the sweet spot between, power
and convenience to people who would be using it, I never understood this argument,
If you are using sqlalchemy with Django you lose the admin, what do you lose if you use Jinja?
In particular what do you lose by replacing render_to_response with this,
def render_jinja_to_response(template_name, payload):
    #This should probably go in settings.py
    from jinja2 import Environment
    env = Environment(loader=PackageLoader('app', 'templates'))

    template = env.get_template(template_name)
    response = template.render(**payload)
    return HttpResponse(response)



Extending authentication - In particular enable loggin in with email
Django authenticates against a wide range of backends,
including Email, LDAP, Twitter and Facebook. While it is true that email backend doesn't work
with Admin login, Admin is meant for use by staff and superusers, so why cant we provide
them usernames?
There are a few other questions raised, some of which I agree with("Sadly, it[the admin app]
struggles a little bit with nullable fields and is tricky to customize."), and some which I dont,
("I will never write CSS by hand again." - You shouldn't be, someone else on your team should be doing that. )
But this line is interesting Personally, I've developed a moderate fear of the word "framework".
This is arguable, but the way all frameworks essentially help you is by providing
Dependency Injection, (the Hollywood principle - Dont call me, I will call you).
Django provides that in a lot of ways. (You write a middleware, and django calls
it at appropriate places - Dont call me, I will call you), but still leaves something to be desired.
My ideal framework would be one where models are POPO and views are functions which return strings
. (Plain old python objects -like POJO). JPA does this correctly,
but then JPA has the typing information, so maybe in python the only way to provide the required typing
information is name=models.CharField(...). But the point being, we need better Dependency Inversion,
not less of it. We have been dow that path earlier, and it is a much harder way to build complex systems.

Resources

This code on github
Discussion about converting the nested tuple to a flat one


My apologies if this post was aggressive, argumentative or disrespectful to the
original author. We generally try to stay away from controversial posts on this blog. But
I believe that this post raised some valid technical questions, and the intent of this
post was to answer them. For Django to develop the best way, we need more of these kind
of "What is wrong with Django" posts, not less, so my thanks to the original author. :)

Do you want to build Amazing Web Apps? We can help.


Django aggregation tutorial
One of the new and most awaited features with Django 1.1 was aggregation. As usual,
Django comes with a very comprehensive documentation for this. Here, I have tried to
put this in how-to form.
Jump to howtos or Get source on Github.
Essentially, aggregations are nothing but a way to perform an operation on group of rows. In databases,
they are represented by operators as sum, avg etc.
To do these operations Django added two new methods to querysets.

aggregate
annotate

When you are have a queryset you can do two operations on it,

Operate over the rowset to get a single value from it. (Such as sum of all salaries in the rowset)
Operate over the rowset to get a value for each row in the rowset via some related table.

The thing to notice is that option 1, will create one row from rowset, while option 2 will
not change the number of rows in the rowset. If you are into analogies, you can think that
option 1 is like a reduce and option 2 is like a map.
In sql terms, aggregate is a operation(SUM, AVG, MIN, MAX), without a group by,
while annotate is a operation with a group by on rowset_table.id. (Unless explicitly overriden).

Ok enough talk, on to some actual work. Here is a fictional models.py representing
a HRMS application. We will use this to see how to use aggreagtion to solve
some common problems.
from django.db import models

class Department(models.Model):
    dept_name = models.CharField(max_length = 100)
    established_on = models.DateField()

    def __unicode__(self):
        return self.dept_name

class Level(models.Model):
    level_name = models.CharField(max_length = 100)
    pay_min = models.PositiveIntegerField()
    pay_max = models.PositiveIntegerField()

    def __unicode__(self):
        return self.level_name

class Employee(models.Model):
    emp_name = models.CharField(max_length = 100)
    department = models.ForeignKey(Department)
    level = models.ForeignKey(Level)
    reports_to = models.ForeignKey('self', null=True, blank=True)

    pay = models.PositiveIntegerField()
    joined_on = models.DateField()

class Leave(models.Model):
    employee = models.ForeignKey(Employee)
    leave_day = models.DateField()

"""
#Populate DB, so we can do some meaningful queries.
#Create Dept, Levels manually.
#Get the names file from http://dl.getdropbox.com/u/271935/djaggregations/names.pickle
#Or the whole sqlite database from http://dl.getdropbox.com/u/271935/djaggregations/bata.db
import random
from datetime import timedelta, date
import pickle
names = pickle.load(file('/home/shabda/names.pickle'))
for i in range(1000):
    emp = Employee()
    emp.name = random.choice(names)
    emp.department = random.choice(list(Department.objects.all()))
    emp.level = random.choice(Level.objects.all())
    try: emp.reports_to = random.choice(list(Employee.objects.filter(department=emp.department)))
    except:pass
    emp.pay = random.randint(emp.level.pay_min, emp.level.pay_max)
    emp.joined_on = emp.department.established_on + timedelta(days = random.randint(0, 200))
    emp.save()
"""

"""
employees = list(Employees.objects.all())
for i in range(100):
    employee = random.choice(employees)
    leave = Leave(employee = employee)
    leave.leave_day = date.today() - timedelta(days = random.randint(0, 365))
    leave.save()

"""



Find the total number of employees.
In sql you might want to do something like,
select count(id) from hrms_employee
Which becomes,
Employee.objects.all().aggregate(total=Count('id'))
If fact doing a connection.queries.pop() shows the exact query.
SELECT COUNT("hrms_employee"."id") AS "total" FROM "hrms_employee"
But wait, we have a builtin method already for that, Employee.objects.all().count(), so lets try something else.
Find the total pay of employees.
The CEO wants to find out what is the total salary expediture, this also converts
the queryset to a single value, so we want to .aggregate here.
Employee.objects.all().aggregate(total_payment=Sum('pay'))
Gives you the total amount you are paying to your employees.
Find the total number of employees, per department.
Here we want a value per row in queryset, so we need to use aggregate here. Also,
there would be one aggregated value per dpeartment, so we need to annotate Department
queryset.
Department.objects.all().annotate(Count('employee'))
If you are only interested in name of department and employee count for it, you can do,
Department.objects.values('dept_name').annotate(Count('employee'))
The sql is
SELECT "hrms_department"."dept_name", COUNT("hrms_employee"."id") AS "employee__count" FROM "hrms_department" LEFT OUTER JOIN "hrms_employee" ON ("hrms_department"."id" = "hrms_employee"."department_id") GROUP BY "hrms_department"."dept_name"



Find the total number of employees, for a specific department.
Here you could use either of .annotate or .aggregate,
Department.objects.filter(dept_name='Sales').values('dept_name').annotate(Count('employee'))
Department.objects.filter(dept_name='Sales').aggregate(Count('employee'))



If you see the SQLs, you will see that .annotate did a group by, while the .aggregate
did not, but as there was only one row, group by had no effect.
Find the total number of employees, per department, per level
This time, we can not annotate either Department model, or the Level model, as we
need to group by both department and level. So we will annotate on Employee
Employee.objects.values('department__dept_name', 'level__level_name').annotate(Count('id'))



This leads to the sql,
SELECT "hrms_department"."dept_name", "hrms_level"."level_name", COUNT("hrms_employee"."id") AS "id__count" FROM "hrms_employee" INNER JOIN "hrms_department" ON ("hrms_employee"."department_id" = "hrms_department"."id") INNER JOIN "hrms_level" ON ("hrms_employee"."level_id" = "hrms_level"."id") GROUP BY "hrms_department"."dept_name", "hrms_level"."level_name



Which combination of Employee and Deparments employes the most people
We can order on the annotated fields, so the last query is modified to,
Employee.objects.values('department__dept_name', 'level__level_name').annotate(employee_count = Count('id')).order_by('-employee_count')[:1]



Which employee name is the most common.
We can want to group by emp_name, so emp_name is added to values. After that we order on the annotated field
and get the first element, to get the most common name.
Employee.objects.values('emp_name').annotate(name_count=Count('id')).order_by('-name_count')[:1]

This was a overview of how django annotations work. These remove a whole class of queries for which
you had to use custom sql queries in the past.

Resources

Source on Github
sqlite file for this model to test
Aggregation on Django docs


Want to build a Django app? Talk to us


Aren't django signals a little like comefrom?

In computer programming, COMEFROM (or COME FROM) is an obscure control flow
structure used in some programming languages, primarily as a joke.


I never liked using signals. Recently I was asked that, and has no answer, but a little thinking led me to this question. Aren't signals a little like COMEFROM. If yes, aren't they as bad as COMEFROM? If you do not know what a COMEFROM is, [wikipedia to the rescue](http://en.wikipedia.org/wiki/COMEFROM)
   
Some hypothetical code using COMEFROM, again from wikipedia,

    from goto import comefrom, label
    comefrom .repeat
    name = raw_input('what is your name? ')
    if name:
        print "Hello",name
        label .repeat
    print "Goodbye!"
    
    
And Some actual Django code using signals

    class Post(models.Model):
        name = models.CharField(...)
        ...
        num_comments = models.PositiveIntegerField(default = 0)
        
    class Comment(models.Model):
        ...
        post = models.ForeignKey(Post)
        
    def handle_num_comments(sender, **kwargs):
        instance = kwargs['instance']
        instance.post.num_comments+=1
        instance.post.save()
        
    from django.db.signals import post_save
    
    post_save.connect(handle_num_comments, sender=Comment)
    
And the same code using COMEFROM

    class Post(models.Model):
        name = models.CharField(...)
        ...
        num_comments = models.PositiveIntegerField(default = 0)
        
    class Comment(models.Model):
        ...
        post = models.ForeignKey(Post)
        
        def save(self, *args, **kwargs):
            super(Comment, self).save(*args, **kwargs)
            instance = self
            LABEL .post_save
        
    def handle_num_comments(sender, **kwargs):
        instance = kwargs['instance']
        COMEFROM .post_save
        instance.post.num_comments+=1
        instance.post.save()
        
So isn't the signals code a little like COMEFROM, and why is it superior to COMEFROM?

--------------
* [There is a library to add goto and comefrom to Python](http://entrian.com/goto/)
* [Discussion about COMEFROM](http://www.c2.com/cgi/wiki?ComeFrom)

-------------
You should follow me on [twitter here](http://twitter.com/uswaretech).




Django design patterns
This is announcement about our new work, Django design patterns, a ebook about, well, Django design patterns. (Well imagine that!). Here is the readme copied from there.

[Edit] Syntax highlighting and indentation preservation were totally brroken. Fixed now.

Django design patterns is a book about commonly occuring patterns in Django. Not
patterns in Gof sense, but patterns in the sense, we work this way, and it works
for us. For us it is a ~pattern of work~ sense.
At this point this is just a lot of brain dump from us.
The latest sources are always available from
http://github.com/uswaretech/django-design-patterns/tree/master
and latest html from http://djangodesignpatterns.uswaretech.net/

Please leave errata, feedback, critique as comments here.

This is still very much a work in progress, released in the spirit of release early, release often. Click here to get it, or fork it on Github


Remote debugging - debugging pesky server only bugs
Here is a quick tip. (Obvious if you work with Django for any length of time, but I have seen a few people who are not aware)
You can put debug trace import pdb; pdb.set_trace() in your code, and put it on the server. When you access this view from your local browser, the debug is still hit and you have a debug shell on your server where you can step through. (Obviously this works only if you ran the code via manage.py runserver. But manage.py runserver start the server to listen only on local connections. If you want to access remotely you need to run as,
python manage.py runserver 0.0.0.0:8000



Edit: As SmileyChris commented, a faster way is,
python manage.py runserver 0:8000



The 0.0.0.0 implies that remote connections are possible.
For me, this has been a lifesaver against those pesky bugs which show themselves only on the server, but not on the local machine.


Django Request Response processing
Have you wondered the steps a users request goes through before being responded to by Django? The answer lies in reading django.core.handlers.base and django.core.handlers.wsgi. Here is a diagram explaining what happens. (Click to enlarge.)

The steps are.
(With Apache/Mod_wsgi, similar steps for other setup.)

User's request comes to Apache etc.
Apache sends request to django.core.handlers.wsgi via mod_wsgi.
A list of request and response middleware callables is created.
Request middleware is applied. If it sends a response, it is returned to the user.
urlresolvers.resolve finds the view funcion to use.
View middleware is applied. If response comes, it is sent back to the user.
View function is called. It talks to models to do business logic, and renders the templates.
The response middleware is applied, and response is sent back to the users.

This misses a lot of important steps (Exception middleware, request_context populating, ...) but is a basic high level overview. 
Resources

django/core/handlers/base.py
django/core/handlers/wsgi.py
.dia and other files for the image


Do you twitter? Do you Github? Find us there.


Better Python package management using source and version control systems
Thanks to awesome django community, there is plenty of open source django code around.
These packages get updated quite often and if you use it often like we do, you'd have possibly realized the need to manage these packages better.
Thankfully, all python ever needs is the source, and all you need to do is to place the source in the python path. 
Most projects use Distributed Version Control Systems like Mercurial and Git, and they come locally with the entire history of the source which provides a lot of control to use any version of the code. For code that we use often, like django packages, using the source from a version control system seems to be the best way to manage.
In order to install django-registration, a popular django app by James Bennett, you need to:
Clone the repository:
$ hg clone https://becomingguru@bitbucket.org/ubernostrum/django-registration/ django_registration



Create a symbolic link in the python path:
$ sudo ln -fs ~/django_registration/registration /usr/lib/python2.6/dist-packages/django_registration



Thats it. django_registration is now on the python path, courtesy of symbollic link, that links the django-registration source to python path, in this case the dist-packages/ folder; thus it will be possible to import this app within the python environment.
You now have the latest version of the source in the folder ~/django_registration, which you can check, and modify. With an editor/IDE that has go to source option, you can browse the source by using a simple short cut.(which may not have been simple if the code were is some egg file in a less well known folder)
Because of cloning of the mercurial repository, you now have access to all the revisions of the application. So you can easily update, or change to other versions.
To update to a newer version when one exists:
$ hg pull -u



To move to any particular revision:
$ hg update -r200



To check the tags:
$ hg tags

tip                              221:b360801eae96
v0.7                             205:d073602dc103
v0.6                             154:e263c551ef7b
v0.5                             139:dc2bf754aa94
v0.4                             110:4d2a725d8c18
v0.3                              75:f41e8a239016
v0.2                              58:e5110bb8d48a
v0.1                              42:d28e5a770ff8



If you want to update to a recent release, say 0.7,
 $  hg update v0.7
7 files updated, 0 files merged, 0 files removed, 0 files unresolved



The same is also applicable while dealing with SVN repositories; alebit some of the revision changes go thro the network to the central server.
In order to install and work with django trunk, check out the trunk and add a symlink, as before
$ svn co http://code.djangoproject.com/svn/django/trunk/ django-trunk
$ cd django-trunk/django
$ sudo ln -fs . /usr/lib/python2.6/dist-packages/django-extensions



You will now be able to access the latest version of django:
$ python
>>> import django
>>> django.get_version()
u'1.1 beta 1 SVN-11092'



To use any other version of django you can do some SVN manipulation, as following:
#To update to a latest version
svn up
At revision 11092.

#To update to a particular revision
svn update -r11090
At revision 11090.

#To update to a last committed version
svn up -r'COMMITTED'
At revision 11079.

#Change to a tagged version - to release 1.02
svn switch http://code.djangoproject.com/svn/django/tags/releases/1.0.2/

#Change to any branch
svn switch http://code.djangoproject.com/browser/django/branches/soc2009



svn switch commands talk to the central servers and can be time consuming.
Possible shortcomings of using source and symbollic links:
Setuptools distribution adds to the python standard library Distutils module, by adding a lot of functionality, among others that stores the package meta data like the version, location of download. If you perform
easy_install django



after installing it by source as above, easy_install again downloads the latest released version of django 1.02, listed in the Python Package Index, as it hasn't been informed of django installation. 
If you are not going to use easy_install at all, this isn't a problem; but it is good to also update setup tools that a version of this application exists, if you at all need. You can then use the pkg_resources module installed with setup tools distibution, to query to find all the installed applications, from python.
If the application contains a setup.py file, you could manually inform setup tools to update using the following command
python setup.py develop



The workflow
To install packages, in a way with high control following needs to be done:
* Checkout the repository
* Add a symbollic link to the repository
* Update setup tools of this installation.



Enter PIP
PIP is a package management tool that does the workflow we need, well. It automatically checks out from the repos and runs python manage.py develop and updates few other resources, so that python package management tools in the system are well aware of the installed packages.
If you are checking out the source and creating symbolic links, it is kind of awkward to have to create symbolic links all the time; Pip automates that. Pip also updates the setuptools with the metadata of the application, so that next time if you ask the same application to be installed, via easy_install or pip, it doesn't download and install again.
So the following command:
pip install -e hg+http://bitbucket.org/ubernostrum/django-registration/#egg=django_registration




Clones the mercurial repository django-registration locally, into Env/src/django_registration
Creates the required symbollic links in the python path, in this case in the site-packages folder
Updates the local package management tools by setup tools of the new package.

-e option tells pip to install the package as editable, that is to leave the repository undeleted. The hg+ and egg=django_registration indicate the repository and the local folder to install to.
Pip works well with git, SVN, Hg an Bzr. Provided you inform the pip of the repository and the folder name, pip completes the work-flow.
Since the source is present in the Env/src folder,  along with the SVN, Git or Hg data, one can modify the versions using version control systems, as illustrated in some of the commands above
Multiple environments
Most often the requirement will be just to update the package into the latest version, for which simple revision control update commands seem good enough. Its good to also mess around with the code changing into different versions and trying different things.
But if you seem to be changing of the versions too often, for compatibility between different environments, you might want to using virtualenv, that isolates a python working environment. virtualenvwrapper adds a bash wrapper to use virtualenv conviniently.

Do you also want your app development managed well? Look us up


Develop Twitter API application in django and deploy on Google App Engine
Twitter's robust REST API enables building of good applications that leverage its increasingly large and real-time data.


Google's cloud system App Engine not only provides a scalable platform for deploying applications, but also proves to be economically viable because of its pay per resource usage model and a large free quota.
Little wonder then, there are increasingly large number of twitter apps being deployed on App Engine.
In this post, I am going to examine how to create a simple application based on twitter's REST API and deploy it on Google App Engine. A deployed version can be found on Twitter-Follow. The specification is simple. It finds if a twitter user is following another twitter user, given their user names.

The application is developed using django and deployed on Appengine using the app engine patch project.
The code is open sourced with GPL v3 and can be checked out from Google Code.
Lets get started building this application.
Create an application
Create a new application from App Engine Dashboard by registering an unique appspot.com sub-domain. This will be your application's unique identifier. You may have to register first.
Install App Engine SDK
Download and install the App Engine SDK. There is no installation for linux users, just place the folder in /usr/local/google_appengine
Download app-engine-patch
Download the sample project. It comes bundled with all the essential django 1.0 files that are required.
Since we do not need the sample application, you may remove the unnecessary sample application folders and correspondingly modify settings.py, but for this demo app, it doesn't matter even if you dont.
Modify app.yaml:
application: twitter-follow
version: 1
runtime: python
api_version: 1

default_expiration: '3650d'

handlers:
- url: /media
static_dir: media

- url: /.*
script: common/appenginepatch/main.py



The only thing you need to modify in this file is the unique application name- for which you registered, and the static files' folder.
As we are using the patch, the handler script to run should always point to main.py of the appenginepatch folder. Leave it as it is.
The /media maps to the static directory ./media, in which you place the static JS and CSS files.
Modify urls.py
urlpatterns = auth_patterns + patterns('',
    (r'^$', 'views.disp'),



Pretty much straight forward, we handle only 1 page and the request is directed to a specified view function ie disp in views.py file which we are going to create.
There are some more imports in the file than django urls.py for the patch to work. Leave them as it is.
What about models.py?
For this application, we do not need any database. We just query twitter as and when user requests.
What about session persistence? Oh, we achieve that using hidden input fields in the form, as we will see in the views.
Hence we don't even need to create an application, just a view and a form in the root directory
Write forms.py
from django import forms

class input_form(forms.Form):
    follower = forms.CharField(max_length=50, required=True, error_messages={'required':'Follower Name is required'})
    following = forms.CharField(max_length=50, required=True, error_messages={'required':'Following Name is required'})



We need a simple form that we can create in our function, so that it is displayed in the template, by the framework.
After all, the django forms framework is one big advantage of using django over the App Engine's builtin webapp framework, even for simple applications. With forms now done, we proceed to the views.
Write views.py
is_follows:
def is_follows(follower, following):

    url = 'http://twitter.com/friendships/exists.json?user_a=%s&user_b=%s' %(
    follower, following)

    password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
    top_level_url = "http://twitter.com/"
    password_mgr.add_password(None, top_level_url, settings.TWNAME, settings.TWPASSWORD)
    handler = urllib2.HTTPBasicAuthHandler(password_mgr)
    opener = urllib2.build_opener(urllib2.HTTPHandler, handler)
    urllib2.install_opener(opener)

    try:
    return simplejson.load(urllib2.urlopen(url))
    except IOError, e:
    #print "Something wrong. This shouldnt happen"
    print e,"Protected user's details cant be found "
    return False



This function queries twitter and returns whether a follower is following following. Lot of boilerplate code, a standard way of adding authentication headers to all requests to the specified domain. Otherwise, the function just examines json returned by twitter and returns corresponding value.
disp and helper functions:
def format_is_follows(follower, following):
    fol = 'YES' if is_follows(follower,following) else 'NO'
    new_entry = (fol,follower,following)
    return "%s:%s:%s" %new_entry

def get_entry_tuples(str1="YES:str1:str2;NO:str3:str4"):
    list1 = str1.split(';')
    list2 = [tuple(i.split(":")) for i in list1]
    return list2

def disp(req):
    if req.method == 'POST':
    form = forms.input_form(req.POST)

    if form.is_valid():
        new_str = format_is_follows(form.cleaned_data['follower'],
                    form.cleaned_data['following'])
        old_str = req.POST['old']

        form_data = "%s;%s" %(new_str,old_str)
        entries = get_entry_tuples(form_data)[:-1]

        payload = {'entries':entries,
               'form':form,
               'disp':True,
               'form_data':form_data}
    else:
        payload = {'form':form,
               'disp':True}
    else:
    form = forms.input_form()
    payload =  {'form':form,'disp':False}
    return render_to_response('page3.html',payload,)



disp is the function called by the framework each time an HTTP request arises. Validates the form data and passes contextual variables for displaying. Some additional code, for examining the previously queried values from the hidden field in the form.
format_is_follows returns the follower following data in the format required by the template we are going to be using- a tuple of 3 values and get_entry_tuples returns the data
that we are going to store in the hidden form. They are Separated into multiple functions here, just for clarity.
template.html
<html><body>

    {% for fol in entries %} 
        {% ifequal fol.0 "YES" %}
            Yes. <a href="http://twitter.com/{{fol.1}}">@{{fol.1}}</a> is following <a href="http://twitter.com/{{fol.2}}">@{{fol.2}}</a>  {% else %} 
            No. <a href="http://twitter.com/{{fol.1}}">@{{fol.1}}</a> is not following <a href="http://twitter.com/{{fol.2}}">@{{fol.2}}</a>
        {% endifequal %}
    {% endfor %}

<form action="." method="POST">
    {{ form.following.errors }} {{ form.follower.errors }}
    Is {{ form.follower }} following {{ form.following }} 
    <input type="hidden" name="old" value="{{ form_data }}">
    <input type="submit" align="center" value="Find out!" /> 
</form>

</body></html>



It contains, the result display part and the form. The form also has a hidden input field. Thanks to django form framework not adding the <form> tags and the submit button. This provides us the flexibility to be able to add the hidden field with the content to be displayed in the next post request.
Run the server
With all the necessary files ready, you are now ready to run the server. In the command prompt enter
python manage.py runserver
Yes, you run the django manage.py. The patch application has placed hooks around and starts the App Engine server. Verify the running application from localhost.
If you happen to have the latest version of Python, sorry. Use the python2.5 instead.
Upload the Application
Thats it. If you have got it working on the localhost, now throw the code at google's scalable and no maintenance cloud and it starts working.
python manage.py update
You don't bother about deployment, server maintenance or log files. Rather, check out the awesome application admin dashboard and if you run out of the allocated free quota, buy some more resource time. Its costs $0.10/CPU hour and one cent for 100 emails sent!
A lot is happening in the field of App Engine support for twitter API. There is even a new App Engine module for twitter oAuth authentication, out of the box, as there is one for django.
App Engine is a powerful platform. django is a high productive framework. Combining them is very powerful and there is also an ongoing project for enhancing the the patch project and making it totally compatible with django.
Resources:

Jesaja Everling Tutorial
Twitter API Documentation
App Engine Tutorial on Webapp Framework
App Engine Patch - Getting Started
App Engine Patch - Documentation


Want to develop an application based on twitter API? We can help.


An Interview with Adrian Holovaty - Creator of Django
Adrian Holovaty is the co-creator of Django, author of the Django book and is currently the BDFL of Django along with Jacob Kaplan-Moss. He studied journalism at University of Missouriâ€“Columbia and has worked with many news sites including Lawrence Journal World and Washington Post. He currently works at EveryBlock, a startup he founded.

Shabda: Would you tell a little about yourself? You majored in Journalism, how did you move to Programming?
Adrian: Sure! I have a dual background in journalism and computer science. I
intended to get a degree in journalism and a minor in computer
science, but things got a little off track. I heard from a
professional colleague that a certain Web site would probably hire me
if I graduated a semester early, so I hurriedly dropped my CS minor
and got special permission to graduate early -- but once the time came
to ask for that job, the company was in the middle of a hiring freeze
and didn't have a position for me. So, in hindsight, I should've stuck
with the minor, but things ended up working out OK.
Because journalism and computer science don't normally go together,
I've had some success in this silly little niche of employing Web
development in news organizations -- "journalism via computer
programming." Professionally, I've always worked as a Web developer at
news organizations, up until my current gig, which is to run
EveryBlock. But in its own way, it's a news
organization as well.
Aside from all of that, I'm quite into music, particularly gypsy jazz,
which is the music of Django Reinhardt (hence the name of the Web
framework). I just recently attended a week-long gypsy jazz guitar
camp in Massachusetts and had a tremendous time
staying up until early hours of the morning jamming with great
musicians from various parts of the world. If money were no object, I
would love to play music full time. In the meantime, I'll settle for
posting YouTube videos of my guitar playing, for an audience that seems to be
comprised mostly of 16-year-old boys who constantly pester me for
transcriptions of my arrangements.
I live with my wife in the beautiful city of Chicago.
Shabda: What are your current roles and responsibilities at Django?
Adrian: Well, I co-created Django back in 2003 with my friend Simon Willison
while we were working together. Now, along with Jacob Kaplan-Moss, I'm
Benevolent Dictator For Life of the project -- a title we shamelessly
ripped off from Guido/Python. This means that I have a hand in all
sorts of things, from high-level API design to checking in patches and
fixing bugs. I've touched probably every bit of the framework over the
years, including implementing the initial ORM (back when it was a code
generator!), pair-programming the original Django template system and
URLconf/view framework with Simon and building the original admin
application with design help from Wilson Miner.
One of the things I enjoy the most is writing documentation -- both
because I like technical writing and because I cannot stand bad
open-source documentation.
Shabda: Can you describe the philosophy behind Django. What are its overarching goals?
Adrian: A couple of years ago, I wrote the "Design philosophies" document,
which sums up the main points nicely.
Generally, the goal is to make Web development fast, fun and easy for
the developer, while keeping performance as fast as possible and code
as easy to understand as possible.
Shabda: How did the move at WorldOnline from PHP to Python happen? Why did you start from scratch (which finally lead to creation of Django), instead of using something like Zope?
Adrian: Simon and I had been big fans of Mark Pilgrim (and still are!), and
we'd read his online book "Dive Into Python." This was around the same
time that the PHP "infrastructure" I'd developed had begun to feel
really, really crufty, so...one day we just decided to start using
Python! It's quite nice working at a small organization with a very
loose management structure; our boss, Rob Curley, was cool enough to
let the developers themselves decide which technologies to use, as
long as the work got done. "I don't care how the sausage is made," he
always used to say.
We looked at the existing Python Web frameworks/libraries, but none of
them felt 100% right to us. Simon and I are both quite into best
practices, such as clean URLs, proper use of GET vs. POST, etc., so we
were very picky in our analysis of those existing tools.
Shabda: Can you tell us about Everyblock? Why did you choose to create your own mapping engine instead of using something like Google Maps? How hard has it been creating a new mapping engine?
Adrian: EveryBlock is the project I lead for my day job. It's funded by a
two-year grant from the Knight Foundation, and the goal is to
experiment with block-specific news -- that is, the site lets you
enter an address (currently only in Chicago, New York City and San
Francisco) and view recent news within a very tight radius of that
address. I'm pretty confident it's the most granular attempt at local
news ever attempted on the Web. Nobody else is crazy enough to do it,
I guess!
We do a TON of work collecting local information, normalizing it and
pulling it together for people in one place. A large part of what we
publish is government data such as crime reports, restaurant
inspections, building permits, business licenses...and even local
movie filmings. Much of this stuff either is buried in "deep Web"
government databases or has never before been available online. A
second part of what we do is detecting addresses and locations in news
articles and blog entries. Plus, we pull in various other geographic
Web stuff like Flickr photos and Yelp reviews.
All in all, our goal is to show you recent, geographically relevant
stuff that you might not have heard about. Say the local newspaper
wrote something about your neighborhood on page B-23 -- would you
really have noticed that article? Say there was an aggravated assault
on your block -- would you really have remembered to check your police
department's crime-reports Web site on a daily basis? That's the basic
philosophy.
Regarding our custom mapping engine...Let's face it: Google Maps is so
passe. As one of the original Google Maps mashup guys (I developed
chicagocrime.org by reverse-engineering Google's JavaScript, before
the API was released), I have all the respect in the world for what
Google has done to invigorate the world of Web maps. But it's time to
take the next step. The Google Maps API doesn't give you any control
over the colors of the map tiles, or change the fonts in street
labels, or disable building footprints, or hide one-way street markers
or subway stations or bus stops or any of the other stuff that's
essentially hard-coded in the map. So, at EveryBlock, we rolled our
own maps so we could have much, much more fine-grained control over
all of these things. Not to mention it's a great way to differentiate
ourselves from the 2.5 million boring, same-old-yellow-blue-orange
Google Maps mashups out there.
Paul Smith from the EveryBlock team has written an article at A List
Apart with more of the technical specifics.
Shabda: As per the Knight grant rules, you would be releasing the code for Everyblock next year. Would you not be giving away your secret sauce? How do you plan to maintain competitive advantage once that code is freely
 avaliable under a permissive license?
Adrian: Our competitive advantage is that we're an incredible team, and I'm
sure we'll come up with a way to feed our families.
Shabda: What problems would you like Django to tackle after 1.0? What big features would you be most interested in having in Django, after it hits 1.0?
Adrian: Generally I'd like to add higher and higher levels of abstraction to
the framework. The Django admin application is a good example of that
-- it's not just an abstraction of an HTTP request; it's an
abstraction of an entire Web application! We should do more of those.
Shabda: What is one thing about Django which you absolutely love, and one thing which you think Django should do differently?
Adrian: I love the way URLconfs work -- like a table of contents for your Web
app. I also love template inheritance. I don't love the fact that
we're generally slow in keeping up with tickets and feature requests.
Shabda: A lot of people these days have to evaluate between Django and ROR. How can they make this decision? When should Adrian Holovaty use ROR? When should DHH use Django?
Adrian: My answer is simple: Try out both frameworks and see which one you like better.
Shabda: Does Django need to market itself differently? What can Django community do for this?
Adrian: I'm comfortable with the amount of attention (or lack thereof,
depending on your perspective) that our project gets. I'm comfortable
with the size of the community. I'm comfortable with the fact that the
right people have found out about it through word of mouth, books,
blogs, Google App Engine or wherever else. If things continue at their
current rate, we'll continue to do just fine, as a healthy open-source
project.
The one thing I'd ask the community to do is to continue staying
civil, polite and approachable.
Shabda: What does the phrase 'journalism via computer programming' mean? How can these two divergent fields be tied together?
Adrian: "Journalism via computer programming," in my opinion, is when a
journalist writes code to tell a story. Instead of talking on the
phone with sources to gather facts, this could involve screen-scraping
Web sites to gather raw data. Instead of writing a newspaper article,
this could involve building a database-driven Web site.
Journalism has several subdisciplines -- photography, information
graphics, video. I advocate that computer programming should be
another one of those subdisciplines. Just as a newspaper employs
photographers and graphic artists, it should employ programmers who
help gather information and tell stories with it.
Shabda: How can traditional journalists do more 'journalism via computer programming'? How can programmers do more 'journalism via computer
programming'? Is it easier for Programmers to move to this field, or for
Journalists?
Adrian: I believe it's easier for programmers to become journalists than it is
for journalists to become programmers, but both sides need to gain an
appreciation for the other in order for this sort of thing to happen
more often. Fortunately, some news organizations are starting to hire
developers with this in mind, and some geeks are realizing journalism
is a great, (mostly) pure field that lets you improve the world
through information.
One concrete thing programmers can do is to look for jobs at news
organizations, which desperately need technical talent. Developers,
you will be loved, you will be treated like geniuses, and your
non-techie coworkers will be very easily impressed!
Another route programmers can take is to get training in journalism --
in fact, Northwestern University's journalism school is giving out
full-tuition scholarships for programmers who want to learn
journalism.
Shabda: Thanks Adrian.

This was Adrian's interview, the last in django-interviews series. But we have a lot of interesting things coming up, so stay tuned.


An Interview with Jacob Kaplan-Moss - Creator of Django

Jacob Kaplan-Moss is the co-creator of Django along with Adrian Holovaty, as well as the author of the Django Book. He has been involved with Django since before it was called Django. He is currently employed at Whiskey Media where his job is hacking at Django. He blogs on Jacobian.org. He graciously agreed to be interviewed at the 42topics blog.

Shabda: Would you tell a little about yourself? How did you get started with Django? What other software/applications have you worked with. (Both OSS and otherwise)?
Jacob: So a bit about me: I grew up in Silicon Valley, so like many other geeks I got started with computers really early; I was programming professionally before I graduated high school. I didn't start out doing web development; in college I worked on video surveillance systems for airports, harbors, marinas, and highways.   That's where I found Python: I rewrote a Java-based camera controller in Python and haven't looked back since.
I started doing web development pretty seriously when I moved to New York and took a job for a design firm there. The job was pretty terrible, and the technologies were worse: the good sites were PHP, and there was a bunch of WebLogic crap that was absolute hell to maintain.
So in 2004 I saw the job opening that Simon and Adrian posted on their blogs  -- and jumped on it. Web development in Python seemed like a dream job, and it really was.
So that's how I ended up in Kansas working for the local paper. At that point I suppose I was using Django, though we didn't have a name for it yet: we just called it "The CMS".
I'd been getting more into Open Source all along, though I'd barely contributed (I think I got a single line into Python at one point years ago -- that was my largest contribution).
Shabda: What areas of Django have you worked most in? What are you current areas of focus?
Jacob: Well, at this point I've touched pretty much every part of Django at some point weather in the form of improvements, refactoring, or just small patches and bug fixes. I'm far from an expert in any particular area, though, so I'd say my main role is more holistic: I'm most concerned with making sure that Django "feels" correct and that APIs and conventions match across the framework.
Right now my current focus is documentation: I've been working on improving the structure and organization of the documentation so that it's easier to find what you're looking for. We've got something like 40,000 lines of documentation so organization and metadata is critical.
Now that I'm lucky enough to get to work on Django at work, I'll probably be able to take on some more big tasks like that in the future.
Shabda: One of the most loved things about Django is its comprehensive documentation. What is the motivation behind refactoring this? What is going to be the new organization? How is this progressing?
Jacob: Thanks -- I've always thought that Django's documentation set us apart from most Open Source projects, and I've been really proud of what we've done there. However, over the past year or so as the size of the documentation (and Django itself) grew I think we've slipped from "outstanding" to merely "above average." To us (the core maintainers and I) that's not acceptable.
The best place to read up on the project and goals is in a post I made to Django-dev a couple of months ago. In a nutshell, I'm breaking up the documentation into smaller, more manageable chunks -- the current DB API is almost sixty printed pages! I'm also separating more high-level how-to's and topical guides from the detailed API references that get in the way when you're just learning.
There's some tool improvements going on under the hood that'll make the documentation easier to write, edit, maintain, and contribute to; hopefully that'll help decrease the number of undocumented features.
The work's pretty close to done, actually. I was trying to finish before I went on vacation a month ago, but didn't quite get there. So that means I need to roll in a month's worth of documentation improvements that happened to the current docs while I was gone, but that shouldn't take much time. You can expect to see the new docs rolled out online very soon.
Shabda: What does Whiskey Media, the startup you are currently associated with do? Is your role there developing Django, or are you associated with other day to day activities as well?
Jacob: I think I'll have to be a bit coy and not say very much about Whiskey Media; we'd rather let our actions speak for themselves than try to build some sort of artificial hype. I'll just say that we're trying to solve some of the big problems in web publishing; you can probably see why Django's the best technical bet for a company trying to be the on the cutting edge of content publishing.
Most of my time at Whiskey is devoted to working on Django -- coding, but also community management, evangelism, and organization. I also have some internal responsibilities, of course, but those are more nebulous: mostly I just help out wherever I'm needed.
Shabda: As you said "we'd rather let our actions speak for themselves than try to build some sort of artificial hype". Do you see Django taking a potential hit in marketing due to similar belief of Django community? Does Django need to market itself differently than it has been doing till now? What can Django community do for this?
Jacob: Huh, that's a tricky one. Well, let me break down the question a bit and look at a few different angles.
So first, I've always been a bit uncomfortable with the idea -- pervasive in Open Source -- that this is some sort of popularity contest. People are always comparing traffic figures, or numbers of job postings, or numbers of Google results, or whatever. I think the idea's supposed to be that if Project Foo has more users than Project Bar that Foo somehow "wins". But I don't see it that way at all.
I mean, I moved from New York City to Lawrence, KS. New York City has a population of, what, 8 million or so? Lawrence has a population of around 80,000. Does that mean New York City is "better"? To most of those 8 million, sure... but to me? Hell no.
As in most areas, there's no accounting for taste: one of my good friends here in Lawrence is a die-hard Perl fan, and though it makes him a bit twisted it doesn't mean that I'm somehow "better" than him -- we just have different needs and different ways of thinking.
All that said, though, there is undeniably a value in popularity.
Only a tiny number of people who use a given piece of Open Source software become involved in the community, and only a tiny fraction of those contribute back to the project, and only a tiny fraction of those contributors become long-term committers. So more users translates directly to more contributors, and more contributors brings more "value" to the project.
(By "value" I'm referring to code, good ideas, etc.)
So as you can probably tell this is something we struggle with. On one hand I think Django's great, and it's in the project's best interest to persuade others to give it a shot. But on the other hand there's a huge amount of bullshitting that goes on where software is concerned, and I try my best to not add to the pile.
It's hard to tell when you've crossed the line from evangelism to hype, and hype can be toxic. Witness the recent backlash against Ruby on Rails: do you think they'd get such violent vitriol if they'd been more modest in their promotion?
Shabda: Talking of ROR, a lot of people these days have to evaluate between Django and ROR. What questions should they ask themselves to answer this question? (Well apart from "do you know Python better or Ruby better?"). To make this more interesting when would YOU choose ROR over Django, if you knew both Ruby and Python equally well?
Jacob: So first let me back up a step -- I'll answer directly in a moment -- and make a quick point about web development in general. The past couple-three years have seen a radical improvement in the quality of tools available to web developers. It really wasn't that long ago that CGI represented the state-of-the-art in web development.
Today, though, there's really some fantastic tools available. Rails, Django, Symfony, Pylons, TurboGears, Seaside... any of these tools represent a major improvement over the CGI/PHP model of web development. If you're still writing web sites the way you did five years ago, you're missing out.
When it comes to which of these tools to choose, of course, we're back to that question of taste. Each tool comes from a different world, and has a different "attitude" towards web development. The cool part, though, is that most new web development tools emphasize a quick start as a key feature: you could probably evaluate a dozen web development frameworks in just a couple of days. So my best advice is to to try a few and see what "clicks".
It's probably important to also pay attention to the language that the framework uses. One of the real pleasures of writing Django apps is that you get to take advantage of the awesome Python community. I've got a set of libraries in Python that I can't imagine developing web apps without; many of those libraries don't have analogues in any other environment.
So if I knew Python and Ruby equally well -- I don't -- I'd still probably lean towards Python, and towards Django. However, there's one place I can think of where Rails is far superior: Rails runs on the JVM. This is a big deal: there's any number of large corporate environments where the JVM is the only game in town. And obviously if I had to choose between Java and Ruby I'd choose Ruby!
I'll mention, though, that Jython (Python on the JVM) is improving by leaps and bounds, and that getting Django working perfectly on the JVM is one of the Google Summer of Code projects the Jython team is sponsoring.
Shabda: Coming back to the marketing question.. For most of non hackers choosing the framework to use is a big question, and the decision they will make depend on various non-tech factors, such as availability of capable skilled people. You mentioned that you would choose Django over Rails due to the awesome Python libraries. For the long term survival and growth of Django, do you not think that an early capture of mindshare in developer community is important? For example I have pitched Django to a fair number of people, and I always have to start with "Django, what?" as compared to "Yeah, we are evaluating Rails for our requirements."
Jacob: Sure, there's definitely a value of having Django be familiar to your friendly local Pointy Haired Boss, so in that context a certain amount of advertising is important. There's nothing worse than being forced by management to use the wrong tool for the job. Keep in mind, though, that this cuts both ways: I'd feel pretty unhappy if there was a team using Django because someone read about it in CTO Monthly. I'd agree that we could be doing more in terms of "brand awareness" or something, but all in all I'm pretty happy with the size and quality of Django's community.
Shabda: There is not much information about the early days of Django, so a little about that. How did the move at WorldOnline from PHP to Python happen? Why did you create a new framework, instead of reusing something like Zope?
Jacob: I wasn't yet at World Online when we moved from PHP to Python, but from what I understand it was a pretty typical change. Adrian and Simon got fed up with the pain and suffering wrought by PHP, and wanted something cleaner and -- most importantly -- something that would be easy to maintain. Python really shines here.
The main reason we ended up building our own framework was that we didn't know we were building a framework. We just wanted to "build cool shit" and, over time, we built tools to help us do that. It wasn't until we started showing it off that we realized we had something that could be used by other people.
This, by the way, is one of the reasons I think Django turned out so great. If you sit down one day and say, "I'm going to develop a framework!" you're almost certainly going to become an Architecture Astronaut, and if you ever actually finish the thing'll be so over designed nobody will want to use it. If, on the other hand, you simply try to solve real-world problems in a clean, obvious way, you'll eventually end up with a great tool.
Look at Rails, TurboGears, even PHP; they all started as simple libraries written by frustrated programmers just trying to get the job done.
Shabda: What are the overarching goals of Django? Again to make this more interesting, Here is a quote from you. "My work as a core developer of Django focuses on giving anyone -- even (especially) non-programmers -- the tools to create dynamic, content-driven websites." Should not that be the job of something like Wordpress, while Django should aim to give "programmers, but not non-programmers -- the tools to create dynamic, content-driven websites."
Jacob: Wordpress is great if you want to publish a blog, but what if you want a website to track a book collection, or sell tickets to concerts, or organize a local farmer's market? There's a great deal that Wordpress (and other single-purpose tools) can do, but the amazing thing about the web is just how wide-open the possibilities are. There's a whole world of possibilities out there, and the end-goal is to help anyone self-publish anything they can dream up.
One of the most fascinating aspects of the history of communication is how intertwined literacy is with social controls. For most of the history of humankind, literacy was strictly available to the elite. The Web created the greatest democratization of publishing ability in history, and has almost immediately turned into a battleground between traditional, centralized publishing and decentralized democratic publishing.
As a programmer, we don't personally play much of a role in this sea change, but I do see the lowering of the barrier to self-publishing as something we ought to continuously think about.
Shabda: If Django's goal is "and the end-goal is to help anyone self-publish ", does not that mean Django is trying to fill the niche of an Extensible CMS like Drupal, as compared to filling the gap left by PHP? Are non programmers really using Django, or are they using apps built with Django?
Jacob: No, you're right that Django's lower-level than something like Drupal. Django's not trying to be a CMS but to be a tool you could use to build your own CMS. It's easier to design your own content models than to shoehorn your publishing into a CMS limited by the ideas of its developers.
There are indeed quite a few non-programmers using Django, though in the long run I think the interesting trend is that more and more computer users are learning a little bit of programming -- enough to develop a site with Django, say.
Shabda: (Since I ask this to everyone!) What is one thing about Django which you absolutely love, and one thing which you think Django should do differently?
Jacob: I only get to choose one thing I love?
I think my favorite bit of Django is the URLconf system. I love that Django forces me to think about URL design as part of my application instead of some byproduct; I've always hated web tools that try to someone pretend that you're writing a desktop app. I'll admit to obsession over my URL design from time-to-time, but I really enjoy clean, semantic URLs.
As for something we should do differently: the assumption that you'll only use a single database is a bad one, and needs to be fixed. Unfortunately, it's an assumption that we made really early on, which means that fixing it is going to be tricky. There's some smart people working on it right now so I've got high hopes, but I wish we'd not made that assumption to begin with.
Shabda: Django started as a in house project and was later open sourced. At your
current startup your major job is with Django itself. The Django book which
you wrote was released under a permissive license. How difficult is it to
convince people outside the hacker culture of the business value of open
source? What are the best ways to do this? How difficult was doing this at
World online?
Jacob:
These days, thankfully, the business value of Open Source is pretty
well-established. There really hasn't been a lot of "convincing"
necessary. For example, Apress didn't just agree to release the Django
Book under a permissive license: they actively encouraged it. You'd
have to ask them if it was "worth it" in terms of sales, but I'm sure
that being the first Google hit for "django book" didn't hurt!
Releasing Django at World Online is actually an interesting story. We
decided at the 2005 PyCon that we should Open Source some of our
software, and started building a business case for doing so. We
prepared a series of arguments -- open source will increase our
visibility and lead to easier sales and easier hiring, open source
will improve the quality of our software, etc. -- and took them to our
management. All of those things turnned out to be true, by the way,
but o our surprise, the argument that was the most effective was
actually a "moral" one. We talked about how Open Source had helped our
business (Apache, Linux, Python, etc.) and argued that it was time to
"give back" to the community. The World Company has always been a
company that's tried to be a conscientious part of our local community
here in Lawrence, and they really jumped at the chance to participate
in the global Open Source community.
I like to tell this story, by the way, because it really makes me
hopeful that this "hacker culture" is in fact compatible with business
culture. Fact is that most businesses are in fact concerned with
doing the "right thing" -- they just often don't know what that is
when it comes to technology. Of course the business case for Open
Source needs to be there, too -- and it is -- but I think there are a
lot of companies that'll jump on the chance to "give back."
Shabda: Before we close would you like to share any tips with us?
Jacob: If you don't read the Django community aggregator you really should: there are some incredibly smart people blogging about Django and you'll learn something new from all of them.

This was the interview of Jacob Kaplan-Moss. We have a few more Django interviews coming, before we close this series of Django interviews, so stay tuned.


An interview with Michael Trier

Michael Trier is  a long time Django user and evangelist. He has worked with a number of technologies including Rails and .net. His insights on marketing Django to traditionally Enterprisy areas were extremely informative. He produces TWiD, along with Brian Rosner which is great to keep abreast of the latest happenings in the Django community. He graciously agreed to be interviewed by the 42topics blog.

Shabda:  Would you tell a little about yourself, how did you get started with Django, what other projects have you used or are associated with?
Michael: Well, I've been programming ever since I can remember, probably around 11 years old.  I grew up in Silicon Valley, and that whole story is a pretty interesting one.  I did the usual thing of starting out with languages like ASM, C, C++ , Pascal.  Moved on to things like Delphi, VB, and most recently I spent quite a bit of time with Ruby, Rails, and within the past year and a half dabbling with Django.
I came to Django for a particular reason.  I was focussed on building a high content push type of site and it just seemed like Django was a much better fit for that than Rails.  Obviously I could have done it with either language, but I believe in using the right tool for the job.
Shabda: You are using Django for your next venture. What are the specific areas where using Django been a better way to develop for you, compared to any other choice you might have made, say Rails?
Michael: I hate to bring up the scaling issue, but that was certainly something I was seeing as a problem with Rails.  The type of site I'm building I hope to see some pretty high traffic (don't we all).  So that was one thing.  The other thing was that it seems like if you're doing a large content driven site, Django just makes that very easy.  Additionally, as most people say, one big win for Django is the built in Admin.  It has allowed me to focus on the front end of the site, while giving the other people involved a way to immediately start working on the content part.  Right away this helps us see where we may need to enhance functionality or perhaps we've built in too much flexibility.
That kind of feedback comes back to us quickly and so that is invaluable.
Shabda: How do you compare Rails to Django? In what areas is Django better than Rails (Apart from scaling/efficiency)? What does Django still need to learn from Rails?
Michael: That's a good question and not something I've given a lot of thought to, but off the top of my head I would say that with something like the Admin, Django definitely wins over Rails.  Now within the Rails community there are a lot of interesting third-party plugins that have attempted to mimic the Django admin, but up until now those that I've looked at have fallen short.  It's a huge effort as we've seen with the amount of work being put into the NewForms-Admin rewrite.
I also think the middleware stuff in Django is very nicely done, although it could use a bit more options in terms of request / response ordering of the middleware items.  I also think Django templates are just perfect.  I really can't see how to enhance on those at all.
As far as Rails is concerned, there's a lot of nice features in rails that Django could learn from, and most of it just has to do with the maturity of the two projects.
Rails caching takes caching one step further.  Rails has model level validations which are very nice and quite frankly the right way to do it, in my opinion.
ActiveRecord, Rails' ORM, has things like aggregation support, and supports a lot of flexibility in how you are able to filter the dependency relationships between your models with things like has_many :through (basically intermediate models).
I also like the before_filter and after_filter type of stuff in rails.  It makes it real easy to invalidate cache or do other things in an event driven way.
Oh one thing back on the Django side, I think NewForms are done really well.  It's an elegant solution.  And it is interesting because in the .net world I'm starting to see a copy of that through things like dynamic data controls.
Shabda: One specific area which I believe Django can learn from Rails is marketing. What would you say to this? With 1.0 release coming soon, how can Django start to market itself better.
Michael: I would agree with you on this.  I think the impending 1.0 release will definitely do a lot on its own to market the product.  There are also at least 3 more books coming out in the next several months.  The Google App Engine announcement that heavily featured the name of Django has also helped to bring it focus.
But there's another element that I don't know if you can fabricate and that is that DHH is a charismatic individual.
In terms of real things that can be done are obviously things like more screencasts just featuring the product.  It's interesting to note that consistently the number 1 screencast on iShowU is the Django one that features a simple tutorial on Django.  Although a lot of people have no interest in screencasts, there are a lot of people that do.  We all learn in different ways.
I also think that on the djangoproject.com weblog we can do a much better job of regular blogging on what is going on within the community.
There are times when you look at the weblog and it's 2 or 3 months old.  From the face of the website you would think that nothing is going.  The last release is over a year old.  Meanwhile those of us that are in the community every day, we see that tons of stuff is happening.  This needs to be communicated better.
Shabda: Talking of GAE, what effect do you see of it on Django? How can Django make the GAE-Django integration painless, or even if this should be done, and what efforts the Django community should expend on this instead of say focusing the efforts on 1.0 release?
Michael: I think GAE helps Django, but only a little bit.  It helps it from the standpoint of name recognition and will cause a lot of people to say to themselves, "what's this Django thing all about."  I think GAE really helps Python in a real way.
In terms of making the integration painless, that's going to be a very difficult task.  I have a taste of that with the django-sqlalchemy project I'm working on, and I'm just mapping a relational model to another relational model.  With GAE it's quite different.  That said, Google folks are working on on a project to do "some" mapping of Django to GAE.  I think over time it will expand in its focus.  I think if they are willing they are the best people to do the work.  I'd rather see Django focus on getting to that 1.0 point.
As far as GAE itself is concerned, I'm kind of on the fence on the real value there.  I know that people like Jaiku are porting their stuff over to it, and Kevin Rose said that it would have been a great platform for something like Digg. Personally, I'm not really convinced of that, but it's still early so we'll all have to wait and see what comes of it.
Shabda: Your TWiD has been a great help for people to keep track of all the happenings in the community. Would you like to share some interesting tidbits you have learned with TWiD, and point to the more interesting ones?

Michael: Thank you, I'm glad that people find it helpful.  I think one of the interesting things is the amount of work that goes into it.   A lot of people are surprised because neither Brian nor I are very professional, so when we're on the mic it sounds like we're just sitting there talking about whatever.  The reality is that it takes quite a bit of work in finding the stuff we want to discuss, attempting to understand enough about the topics to at least sound somewhat intelligent on them, and then there is the recording and post-production.
As far as actual topics the two recent episodes on Internationalisation have been a great learning experience for me personally.  Going into it, I really didn't even know enough about it to know what questions to ask. Thankfully Malcolm Tredinnick put the whole thing together.  He's been a huge help to the show.
Another show that was fascinating for me was the one on GeoDjango with Matthew Wensing.  It's a fascinating topic and I hope to put together more shows on that subject.
I think a lot of people like the interviews, and we do too, but we also don't want to make the show just interviews every week.  So we try to split it up.
Shabda: Would you tell a little about the venture you are working on?
Michael: Sure.  The project is a hyper-local media site.  We're working on  providing an online place for a mix of general media type of content (stories, events, etc...) with citizen journalism types of offerings.  A lot of people are working in this space, trying to figure out how you bring the big media type of stuff down to a local level.
We then want to mix some of that with interesting datasets, a la EveryBlock, as well as provide some level of social interaction.  The cool thing is that once we release we're going to make the entire thing available on a New BSD license.
Frankly, initially it's not going to be that interesting.  A lot of what we'll provide in the way of Classifieds, Marketplace, Aggregators, stories, etc.. will be similar to the types of offerings that you see in things like Ellington or just about any newspaper's online presence these days.  Down the road I hope that we can expand it into something quite useful.
The framework we'll be releasing is called ArEyah.  So expect to see something from me on that in the next several months.  Timeframes are tough to nail down at this point because this is a side project in addition to some of the other things I'm involved in.
Shabda: For many people, such as me, the question is what business advantage remains for you after you are releasing all your secret sauce under an open license. For example even EveryBlock will have to release all their source under an open license, after the end of the Knight Grant expiry period. What would you say to this? What do you hope to achieve with releasing the project under a BSD license?
Michael: That's really a good question.  For me it was two-fold.  First I wanted to take the "intellectual value" of it off the table.  In other words, I did not want to be in a position with my partners where I had an extremely unfair advantage.  Secondly I really think this is something that could benefit communities everywhere; of course that remains to be seen.  If that is the case, then I think a lot of individuals would get involved and make it a better product for the benefit of all.
As far as competitive advantage, I really see that being in the execution.  In comes down to how well we serve our community.  If we are not doing a good job of it then someone else should be able to come along and beat us out there.  There were certainly be some things that are specific to the community that we're targeting that don't have a place in the general framework.  Those things will be our own and they will be highly tailored for our use, in order to serve a specific need we have here.
Shabda: We see a lot of comparison going on between Django and ROR, or Django and Turbogears. But we do not see enough comparison between Django and other traditionally 'Entrprisy' frameworks. Say Java based frameworks like Struts+Hibername or Asp.net. As you have worked with .net, how would you compare Django with these frameworks. What can Django do to make itself popular in the areas dominated by these frameworks?
Michael: The differences have less to do with feature set comparisons.  A lot of what I do every day for corporate clients could be done much more quickly and actually often more robustly with something like Django.  So it's not a thing where you can say feature x is in .net but not in Django.  That said like any framework / language there are edge case types of things where you might say "Java is the right tool for this job."
What it really comes down to is the corporate culture.  .NET and Java own those corporate cultures.  There are real reasons why it makes good business sense to build your corporate infrastructure on Microsoft products.  You can pick up the phone and get three .NET developers tomorrow.  Regardless of whether your company is based in Louisville, Kentucky or White Plains, New York.
In the case with Microsoft, much more than in the Java world, they provide a singular full-stack solution.  No one, in the executive sense, needs to make any more decisions about which reporting tool, which database backend, or which IDE they are going to decide to use.  Often that alone is motivation enough.
Shabda: Your last post on your blog was about the benefits of DVCS as compared to Centralized systems. What are the compelling benefits of DVCS over, say, SVN. In particular how can moving to a DVCS help Django?
Michael: Well I'm somewhat new to the DVCS world, as are a lot of people, so I'm no expert on the subject.  To me though the benefits of a DVCS are at more of a personal level.  I have seen tremendous personal benefit in being able to commit while sitting in a coffee shop, or being able to branch code locally and then merge that back in.
As far as benefit to a centralized project like Django, I guess it remains to be seen.  I've been watching the shift of Rails to Git with great interest.
So I guess in summary, I just don't know enough at this point to say that it would be beneficial. The shift is probably more psychological. I think, for a lot of projects, it would be just another way to have a centralized repository.
Shabda: You and Brian Rosner are working on django-sqlalchemy. How would a sqlalchemy based ORM be better than the current implementation? What is the status of this project?
Michael: Yes, Brian and I are probably the primary committers, but we have several other individuals that have pitched in on the project.  I like to think about the benefits in two separate areas.  First, there's the approach where you just plug django-sqlalchemy in and continue to use Django with its filter syntax, etc...
In that case the benefits you gain are not benefits at the ORM level (because you're still using Django's syntax, which doesn't support things like aggregation).  The benefits instead are things provided as a result of having SQLAlchemy as the backend.  So this would be things like multi-database backends, sharding, additional database support like DB2 or Firebird.
The second approach is where you actually need things like aggregation or more complex queries, without resorting to raw sql.  In that case we expose SQLAclhemy's ORM right on your models.  So the full power of SQLAlchemey gets exposed and available for you to use whenever you need.
So I could technically have 90% of my ORM code just using Django's syntax but then realize that I need to do something a little outside of its capabilities.  In that situation I might chose to just use the exposed properties to get what I need.
Finally there's one more thing related to this that we still do not have clear at this point, but will come down the road, and that's actually adding different functionality at the model level.  This might be things like Intermediate Model support being made available through django-sqlalchemy.
As far as status of the project, it has been moving along very well.  We have a few more filters to implement and a handful of management commands, plus lots more testing.  One of the things I've done in the past week is to code up a test application using django-sqlalchemy to see in a "real-world" sense where some of the problems are.  That has been really helpful.
Shabda: So if the extra features added by SQLachemy is not needed, then this aims to be backwards compatible with current Django syntax?
Michael: Yes, our aim is that you could plug it in and run all your stuff just as is.  In other words with django-sqlalchemy as your backend db (that's how it gets exposed) we should be able to pass all of Django's test.  Once we're able to do that we tag it 1.0 and get some people hammering on it.
We do have one big hurdle that I have not discussed.  Currently we are using multiple inheritance to modify the Django classes.  That doesn't work for the contrib apps or third-party apps because that would require a change to their code base.
We took this approach originally just because we wanted to focus on the mapping issues first and prove the concept.
The eventual plan is to use some class replacement techniques to inject our stuff right into the Django models at evaluation time.  That will make it work across the board.
Shabda: Before we leave. Would you like to share a tip, or hard to find information about Django with our readers?
Michael: Not really a code tip, because we do those each week on TWiD, but more of an approach tip.  The generic views stuff is extremely powerful, and quite often I see a lot of new users doing stuff in views that could easily be done with a generic view, or with a wrapped generic view.  James Bennett has a great post on this, and I suggest everyone check it out.  I think often, especially for people that are new to the community, the power of it is overlooked.
Finally one more thing, spend some time in IRC. IRC is a great way to get an education in Django very quickly.  Reading the questions and the responses has been invaluable to me in learning how to use Django more effectively. It's a great community.
Shabda: Thanks a ton for this great interview. It was extremely  informative and interesting.
Michael: Thank you.

This was the interview of Michael Trier. This week I am not going to any more interview, but stay tuned for next week, we have even more Django interviews coming.
And of course, the 42topics is live now. And we have a Django section. (How 42topics works?)
So join now, and lets get rolling.


Popularizing Django -- Or Reusable apps considered harmful.
For all its technical merits, Django is still a very niche technology. It is my belief that the thing which is holding Django back the most, is due to one of its strengths.
Making reusable apps is easy and simple in Django. In Django this is the correct way to do things. You take a few apps, mix them together in your project, and deploy to start your site.
Compare the installation steps of Wordpress and an imaginary blog software better than Wordpress called Djangopress.
Wordpress

FTP wordpress to webserver.
Point browser to site.com/blog
Next-Next-Next done.

Djangopress

Svn checkout Djangopress
Svn checkout django-registration
Svn checkout other Django apps Djangopress depends on. Maybe django-mptt, django-threadedcomments or a few others.
Edit your settings.py to add all these apps to INSTALLED_APPS.
Add database settings, and other changes if needed.
Telnet to your server and do syncdb
Create templates. Done.

This does not take into account the extra hoops Apache makes  you jump through, compared to using a PHP app.
How I got started with web programming.
I wanted to run a forum. PhpBB was free, and seemed most widely used. Installed it, and wanted to tinker with it, so learnt Php. If there was a different forum software, which was technically superior, but which asked me to write templates for it before I could start a forum, guess which one I would have chosen?
So how to popularize Django.
In my interview of James Bennett, I asked what is Django's killer app. And he said there need not be a Killer app for Django, reusable apps will do. I guess I will have to disagree. Even internet needed a killer app to get breakthrough popularity. Let's see what a Killer app gives you.

It fills a big niche, so people are forced to learn your language/framework.
It forces the Hosting company to support your language/framework.
If a large number of places use it, it gives your framework name recognition.

So to popularize Django, I propose setting up DjangoPackagedApps.com to distribute packaged Django apps, to complement reusable Django apps. A packaged Django app, must have these properties.
1. All dependencies must be included.
2. Beautiful templates must be included out of the box.
3. Users must not need to modify anything in settings.py apart from the database settings.
And installing the PackagedApp must be no more than the number of steps needed in Wordpress.

Svn checkout/FTP DjangoPackagedApp
Only thing to edit in settings.py is database settings.
Do syncdb. done.


Do yo use Django? Do you program? Find things which YOU will love reading at 42topics.com.


An interview with Russell Keith-Magee - Django core contributor

Russell Keith-Magee is a longtime core contributor to Django. He has worked extensively with the Django testing and serialization components.  He is
currently working on django-evolution, the ability to do schema evolution from Django, an often requested feature, and is mentoring a GSOC proposal to add
aggregate queries support to Django ORM. He is currently a Senior R&D Software Engineer at plugger.com.au. He graciously agreed to an email interview with the 42topics blog.

Shabda: Would you tell a little about yourself? How did you get started with
Django? What areas of Django have you contributed to?
Russell: I'm a thirty-one year old, father of one, living in the most isolated
capital city in the world - Perth, Western Australia.
Strangely, for all my involvement in Django, my background has almost
nothing to do with the web. I studied Physics as an undergraduate, and
studied neural networks for my PhD. My first job was with a startup in
the defence industry developing simulation frameworks. Over time,
mostly as a result of my association with Django, I've become more
involved in building web services.
Shabda: How did you get started with
Django? What areas of Django have you contributed to?
Russell: I've always had a bunch of little pet projects that I've fiddled with
in my spare time. Over time, these projects have migrated over a range
of languages until I found (and fell in love with) Python. Around the
time of Web 1.0, I started looking at various web frameworks (PHP,
Zope, etc), but I never really managed to get mental traction - they
were either so complex that it was hard to work out where to start, or
they provided a bunch of tools but didn't really make it obvious how
the pieces all fitted together. Plus, the documentation for these
frameworks was up to the common open source standard - an even mix of
the awful, the incomprehensible and the non-existent.
Around mid 2005, the buzz about Ruby on Rails started to grow, so I
took a look. I could see that it had promise - it was an interesting
new way of looking at solving web problems - but again it suffered
from the documentation problem. One day, I found a blog entry
referring to Django, and finally all the pieces fell into place. The
documentation was excellent, the simple stuff was painfully simple,
and there was a clear separation of concerns - there is the URL module
which routes requests, an ORM to access the database, etc.
At the time, the pet project I was using to test new frameworks was a
time/task tracking system. After working through the tutorials, I
tried implementing a task tracker, and quickly realized that Django
didn't have support for aggregate functions. I made a few suggestions,
which were universally rejected (in retrospect, for very good
reasons), so I went digging in the code to try and come up with some
better ideas.
In the process, I ended up spending a lot of time in the query system,
which put me in a position to submit patches for a few small bugs in
queries. Once I had found my feet, I tackled some larger enhancements.
One of my first big contributions was modifying the filter syntax so
that queries could span reverse relations; that is, if Book has a
foreign key on Author, I modified the syntax to allow Author queries
to ask about their related books. Of course, between the Magic Removal
changes and Malcolm's Queryset refactor, most of this code has now
been thrown to the wind, but at the time it was a big improvement.
Adrian noticed the contributions I was making, and offered to give me
commit access and make me a core developer.
Not long after I joined as a core developer, the Magic Removal branch
started, which consumed pretty much all my efforts for about six
months. This involved more changes to the query system, but also with
the way models were defined and instantiated. Following that, I
developed the test framework, which also involved working with the
serialization framework. I did some work on newforms-admin in the
early days, including developing the new Media definition framework
and a refactor of form input fields for files.
I have also spent some time on side projects that aren't directly
targeting the Django trunk, but might be integrated one day. The most
notable of these is Django Evolution - an implementation of schema
evolution for Django.
I've been a little bit quiet over the last few months; I've just
changed jobs, which is consuming a lot of my time, and I've had a few
bouts of illness which have slowed my progress. I have also been
helping out James Bennett by acting as Technical Reviewer for his
upcoming book "Practical Django Projects"; there won't be much to show
for this until the book is published, but when it comes out, you're
all in for a treat - it's going to be a great resource.
I'm hoping to get back into the swing of things in the next few
months. One of the first new tasks will be a very old issue - I'm
mentoring a student (Nicolas Lara) in a GSOC project to implement
aggregates in the query system. So - after almost 3 years, my original
itch may actually get scratched.
Shabda: Have you worked with
other competing frameworks, say Ruby On Rails, Turbogears etc?
Russell:
I've looked briefly at a lot of other web frameworks. In the early
days I looked at Zope and PHP while gaining a foothold in the web
world. Since joining the Django project, I occasionally look at what
the other frameworks - in particular, Ruby on Rails and TurboGears -
are doing. It's occasionally helpful to see how other people are (or
aren't) solving various problems, borrowing the good ideas where
possible.
Shabda: What other
software/applications have you worked with. (Both OSS and otherwise)?
Russell: A little bit of everything. I used Linux as a personal computing
platform back in the glory days of Slackware and Yggdrasil. I was
compiling KDE and GNOME before they were even v1.0, and I've toyed
around with writing for both frameworks. I've used Windows in a
commercial environment, and I've gotten to know the eccentricities of
every version of Microsoft Visual Studio from v6 to 2008 in far too
much detail. I've written Java, both server side and client side.
I've spent a lot of time with a simulation protocol called HLA. One of the more notable products of that work was
securing some funding and commercial support for an open source
implementation of the HLA protocol.
I'm a huge fan of Linux, especially on the server side. However, I
attribute most of my recent productivity to being a Mac Switcher. I
found that once I made the switch, I spent a lot less time messing
around with compiling the latest, slightly less buggy version of some
esoteric utility, and a lot more time building useful code. It's
possibly a little 'post hoc ergo propter hoc', but my Mac switch
correlates almost exactly with being accepted as a Django core
developer. Regardless or the causation, give me a MacBook Pro, a copy
of TextMate and a good net connection and I'll be a very happy man for
a long time.
Shabda: You have worked with a number of web frameworks. What made you choose
Django over them? What are the strong points of Django compared to say ROR?
Compared to Turbogears? What can Django learn from these other frameworks?
Russell: I haven't worked extensively with any other web framework. I've had
the occasional poke around - mostly to see how the other guys do
things. To my eyes, at a technical level the various modern web
frameworks have a lot more in common than they have differences. All
of this generation of web frameworks are about providing a clear layer
between the database and the user, proving a simple mechanism for
mapping web requests to functions that serve those requests, and
providing a clean way to render content for the responses.
Sure, there differences in language of implementation. There are some
philosophical differences like whether templates should allow
executable code. There are also some feature differences. However,
these are all minor in the grand scheme of things. The philosophical
differences haven't stopped anyone from building web applications in
any of the frameworks. Given enough time and resources, any feature
could be implemented in any framework. Language choice is also largely
irrelevant. I've used a bit of Ruby, and while it has some interesting
features as a language, the difference between Python and Ruby is a
lot less than the difference between, say, Python and C++.
What this leaves is the stuff outside the technical realm - resources,
documentation, and the community. This is the reason I choice Django
in the first place, and I still think that they are areas of strength
- I certainly don't regret my original decision.
I remember when I first found the Django website - I very quickly got
the impression that it was built by a group of people that really
cared about what they were building, and how it was perceived. The
website and other resources weren't just rapid constructions or
reflections of this week's design fad - they were designed with care.
The same went for the documentation - it wasn't just a bunch of
documents autogenerated from source code comments or a sprawling wiki
- they were custom written by someone who knew how to write, and more
importantly, knew how to edit. As a result, Django's tutorials and
documentation made getting started an almost painless process - both
as a user, and as a developer. The attitude of the community is also
important; Adrian and Jacob have established a very civil community
and the community as a whole tries to keep it that way.
I think the biggest lessons Django can learn from other frameworks is
in how to market itself. Because we're still pre-v1.0, we haven't
really been aggressive about marketing ourselves. Some of the other
frameworks have been selling themselves for a long time, so we are in
a position to learn from their experience.
Shabda: One of the most requested features with Django is schema-evolution, which
you are working on. What are the design goals for django-evolution? How
production ready is this?
Russell: At the moment, Django Evolution handles the simple cases
(add/delete/rename simple field) fairly well. There are some bugs in
the more complex cases, particularly those involving foreign keys and
many-to-many fields. It tends to work quite well for SQLite and
Postgres; MySQL tends to be more problematic. At this time, I'd
classify it as "production ready for capable developer". As long as
your schema evolution needs aren't too advanced and you're willing to
do a bit of hand holding and carefully check the SQL that is produced,
you can use Django Evolution right now. If you're not as confident
with raw SQL, you might want to wait a while.
Obviously, the long term goal is to make it "production ready for
everyone", and support every possible change that you can make to a
Django model. Completely aside from fixing bugs and supporting other
databases, one of the big open issues is populating data into new
columns. At present, we have some simple mechanisms to put in default
data, but these are somewhat limited. Longer term, we're aiming to be
able to populate a new column with data from existing columns. This
will require providing a subquery as initial data, and my hope is that
the new Queryset Refactor features will make this sort of thing much
easier.
Shabda: I asked this to Malcolm as well, but he was unwilling to make a
prediction. So let me ask this slightly differently. What problems would you
like Django to tackle after 1.0? What big features would you be most
interested in having in Django, after it hits 1.0?
Russell: My efforts in prognostication have a nasty habit of going badly
wrong... but here goes:
One of the recent shifts in in the Django project as a whole has been
the move to allow user extension as much as possible. The ability to
have user-defined field types, database backends, and management
commands are all examples of the push to move Django development into
the community as much as possible. I would be very happy to see the
majority of Django v2.0 features be a merge of established
community-based projects into the trunk, rather than projects tightly
integrated with Django internals.
That said, some changes can't be done as extensions. One such
reoccurring feature request is for multi-database support. It isn't
something I have a huge need for myself, but it is a reoccurring
request on the Django users mailing list, so it would be good to be
able to support it.
Support for aggregation functions (sum, average, etc) in the ORM is a
feature I have a use for, and I'd like to see - especially since the
lack of aggregation functions is the reason I got involved with Django
in the first place. I'm currently mentoring a GSOC project to
implement this capability, but I expect that this won't be complete
until post v1.0.
I would also expect that the resources around the community will get
some attention. There is plenty of room for new tutorials. There are
also a few contrib services that need much better documentation (in
some cases, any documentation). Given that v1.0 is a point of API
stability, it also makes a good time to add lots of new tutorials and
other documentation. Formalization and documentation of some of
Django's more notable internals (like the _meta model class and the
various extension mechanisms) would also be nice to have.
Shabda:  You have worked in many different areas of Django. What suggestion would
you give to somebody who is just looking to start contributing to Django?
Are there some areas, where the learning curve is flatter, and people can
make meaningful contribution more easily?
Russell: My suggestion would be "don't be timid - jump in". Even the most
complex parts of Django - like the query system - are relatively
straightforward once you've spent a bit of time walking through them.
If someone is looking to contribute to Django, my suggestion would be
to do the same thing I did when I was starting out - pick a Django
component with a bunch of bugs, pick one, write a test case for the
bug (if there isn't one already), then walk through the code until you
can fix it. Rinse and repeat, but stay in the same component.
Eventually, you'll know that component as well as anyone in the world,
and you'll be able to work on more complex bugs or contribute a
feature. Occasionally, tracking down one bug will lead you into
another component; use that as an opportunity to learn how another
part of Django works.
The other thing to keep in mind is that working on Django means more
than just code - if you want your contributions to be taken seriously,
you need to get in the habit of writing rigorous tests and
comprehensive documentation. Don't just fix the bug and then complain
that we don't commit your tickets - you need to do the whole job.
Shabda: A lot of Django people come from diverse backgrounds, You have studied
Physics, Adrian studied journalism, Malcolm mathematics and James
philosophy. Is there something about with Django which attracts people from
diverse backgrounds (keep simple things simple ...)? Has a lack of people
with CS degree been a problem sometime, as in "We wish there was a CS guy to
discuss this problem with"?
Russell: My undergraduate degree is in Physics, but my honours and PhD program
were both in pure CS. Plus, for all their lack of CS degrees, Adrian,
Malcolm and James have certainly proven they know how to cut a line of
code, and there are plenty of people in the Django community with pure
CS backgrounds that contribute ideas and code. I don't think Django is
suffering from a lack of CS experience.
However, I do think that having a multidisciplinary background does
predispose you to a certain way of looking at the world. It has been
my experience that some people with a narrow IT/CS focus lose sight of
the fact that at the end of the day, a computer is a tool, most people
don't like them very much, and only put up with them to the extent
that they help to make their lives better. I can appreciate the
elegance of a neat algorithm as much as the next CS major, but at the
back of my mind I always have the question "How can I use this to
solve a real world problem?"
One of the reasons that I suspect Django has been a success is that it
has never lost sight of the fact that it exists to solve problems.
Internally, it does a lot of smart things, but outwardly it tries to
be as simple as possible. You download the code, and it works - you
don't need to resolve dozens of dependencies. The development server
just works. There is a simple and easily understood pipeline from
request to rendering. Django identifies that building web applications
is a problem that people need to solve, and tries to provide the
simplest possible path to let that happen.
Shabda: You have worked with a defense startup, and defense industry is presumed
to be very conservative. What can Django do to market itself to market
itself to people outside web 2.0 niches, such as in large corporations or
defense related software?
Russell: There is a running joke in the Australian military that the Army's
idea of the perfect technology advancement would be a self-sharpening
grease pencil. It's not quite that bad, but the military is very
conservative. This is understandable, because lives are potentially on
the line if a technology fails. However, it does breed some
interesting limitations on new technology: in the circles I was
working, IE6 on Windows 2000 was considered to be a very modern web
platform.
The way to market to people outside of Web 2.0 niches is to stop
trying to sell Web 2.0 as if it was an answer to anything. Web 2.0 is
a technology. Web 2.0 may give you a competitive advantage. However,
your customer doesn't give a pair of fetid dingo's kidneys about your
competitive advantage. The way to sell your Web 2.0 idea is to forget
all about how cool the technology is, or how great it is that you can
do this as a RESTful web service. Instead, focus on how your solution
solves a specific problem of your customer in a way that your customer
is comfortable with.
A person in industry has a problem. They need a solution. They don't
particularly care if the solution is implemented using an alien slave
hiding in a cardboard box connected to the server using electric
spaghetti - as long as the solution is cheap, reliable, solves their
problem, and doesn't require them to fundamentally change the way they
do business. If you can convince your customer that your product meets
these criteria, you'll have a long and profitable career in the
military-industrial complex - regardless of whether you use Web 2.0 or
not.
Shabda: What is one thing about
Django which you absolutely love, and one thing which you think Django
should do differently?
Russell: On a technical level: I think the decision to keep logic out of
templates is 100% correct. Keeping logic out of templates forces the
separation between content and presentation, and simplifies the
interface for the design team, who often will not be gung-ho
programmers. I think this is one of the best decisions that Django has
made - I know there are many in the world that claim that Django's
templates are it's biggest weakness, but I simply don't agree. I take
some solace from the fact that Google has tacitly agreed with this
position by providing Django's templating engine inside their
App Engine.
One area that I think Django could do better is in error handling -
especially in the way that exceptions get caught and rethrown. There
are a number of ways that import errors or dangling named URL
references can be misreported in stack traces or 500 pages; given that
both of these are relatively common errors, it would be nice to
provide better handling.
On a community level: I love the general disposition of the Django
community. It's polite and professional, and appreciates the value of
good design.
If I had to pick a flaw, it's probably the way that the core
developers (myself included) sometimes let the community down in the
way we communicate overall project direction. When plans are made
(however informally), they aren't always communicated effectively to
the community. It also isn't always clear (beyond the simple fact that
we're busy) why specific tickets with bug fixes don't get committed to
trunk in a timely fashion.
Even I get bitten by this sometimes - being based in Perth, Western
Australia, the cost of travel prevents me from attending PyCon or the
Sprints in person. As a result, I don't get to participate in the
in-person discussions - I sometimes have to read between the lines of
blog posts and mailing list messages to work out what the plans are,
sometimes confirming details using our secret core developer BatPhone.
This is certainly an area where we could do better. Constructive
suggestions are always welcome.
Shabda: Frameworks like Turbogears and Zope take a "best of breed" approach, and
tie together a lot of existing components to build the full framework. In
comparison, Django takes an integrated approach, and most of Django's
components are Django specific. For this reason, many people have accused
Django of having a NIH.
syndrome.
What are the pros and cons of both these approaches? Why did Django rebuild
most of its components?
Russell: Saying that Django rebuilt most of its components is one way of
looking at it. The other way to look at it is to say that Django
identified a core set of functions that need to be tightly integrated,
and need to meet some specific design goals, and built tools that met
those requirements. In some cases, it's not even fair to say that
components were rebuilt. Remember, Django went public in mid 2005, but
existed as an in-house project at the Journal-World for some time
before that. For some of these components, Django's implementation
predates the 'existing component'.
Those that throw the NIH argument at Django are also ignoring two
important points. Firstly, Django already uses a number of prebuilt
components from elsewhere - the signal dispatcher, the JSON
serializer, and any number of other components are taken from existing
projects. We also have dependencies on external software - most
notably in database backends, but also in serialization and markup
engines.
Secondly, there is the presumption that plumbing together external
projects requires no effort. This is patently false - you only have to
look at the places that Django does rely on external projects to see
why. The MySQLdb 1.2.2 prerelease bindings had all sorts of problems
that require special handling; similar workarounds are required for
unicode handling in recent versions of Markdown. Each version of these
external components can potentially require different handling,
sometimes in subtly incompatible ways. Once you introduce multiple
interacting components, you introduce the problem of incompatibility
between versions of those tools. Integration like this isn't easy.
The Django approach guarantees that everything works out of the box,
and there is only one core configuration to document and test. It
reduces the barrier to entry: making the first act of a new user a
decision about which slightly different template engine will best meet
their needs isn't the best way to make a good impression. It also
focuses the efforts of the community in testing, documentation,
tutorials and advice. If improvements are required, the entire
community is in a position to provide an opinion, not just the subset
of the community that works with that component option. In a
Balkanized world, there will always be a tendency for the question
"How do I do X using tool Y" to be answered "don't use tool Y, use Q
instead".
Building the components from scratch also guarantees that all the
components fit - and I don't just mean that they satisfy their mutual
interfaces. There is also the issue that the core components should
all 'taste' the same - that they follow similar design philosophies,
apply similar assumptions, and so on. This sort of consistency is an
invaluable resource, as it forms a kind of implicit documentation:
once you have learned one part of the system, you automatically know
how other parts of the system work. You don't need to relearn the
rules for each component.
One notable (and frequently made) argument against this approach is
that you lose flexibility. While this is theoretically true, I don't
think it matters much in practice. For all the supposed inadequacies
of the Django template engine, I challenge anyone to present a problem
that can only be solved by using a different templating engine. On top
of that - there's absolutely nothing preventing you from using your
own template engine, or your own database backend, or any other
component if you don't like Django's offerings. If you don't believe
me, go look at Google App Engine.
Shabda: What are the focus areas Django testing framework? What utilities does
this add too unittest or doctests? How does this fit in with browser based
tools like Selenium?
Russell: Like most things in Django, the test framework aims to be as simple as
possible, but no simpler. It provides an environment in which tests
can be run, it provides the facilities for finding and invoking test
cases, and provides a few helpful web-specific assertions that can be
used during testing. Base Python unittests and doctests get a dummy
database to work with, plus a dummy mail server; the Django TestCase
extends unittest.TestCase to provide fixture loading and a dummy web
client that can be used to programatically poke views.
So far, the Django testing tools have concentrated on testing Django
views as if they were ordinary Python methods - after all, that's what
they are. Selenium takes things another step, allowing for tests of
how a web browser will behave when given the output of a view. I tend
to think that testing the view itself is a more rigorous test, as it
allows for direct inspection of view output. However, Selenium does
have a place, especially when dealing with Javascript or other dynamic
on-page interactions. Adding support for Selenium to Django is on the
to-do list - it just requires someone to do the work.
Shabda: Thanks for the interview Russell.

This was the interview of Russell Keith-Magee, a longtime Django core contributor based in Perth, Australia. Next we interview Michael Trier, a longtime Django user and evangelist.
Michael's This Week in Django has been a great resource for the Django community, and we discuss Django, and how best to market Django. So stay tuned.
Oh and yes, 42topics.com is live now! (How are we different?) And we have a Django section. So regsiter let's get rolling!


Interview with James Bennett - Django release manager

James Bennett is the release manager of Django, and a long time contributor. He works on Ellington, a CMS designed for news organizations. His book, Practical Django Projects, is being published by Apress, and is scheduled to hit bookshelves in June 2008. He graciously agreed to be interviewed at the 42topics.com blog. His blog, The B-List, can be found here.
Shabda: Would you tell something about yourself, how did you get started with Django, and what other OSS projects are you involved with?
James: I got into Django fairly soon after the initial public release; I'd been doing PHP and Perl work (mostly Textpattern on the PHP side, Scoop on the Perl side), and I was working on teaching myself Ruby and Rails because it looked interesting. But I'd always liked Python; it was just that there weren't a whole lot of good options for Python web development at that point. You could do Zope, or you could do Twisted, but they both had pretty steep learning curves when compared to the type of work I was doing, so it just wasn't worth it.
Django changed that; I think I did the tutorial the day it was released, and I just fell in love.
These days most of my open-source time is devoted to Django, and to various Django-based apps I've written and released. Back in the day I used to do the occasional bit of PHP hacking, and I wrote some plugins for blogging engines, but none of that stuff's maintained anymore.
Shabda: What are your responsibilities as the release manager of Django? Who are the other core contributors of Django, and what are their current areas of focus?
James: My responsibility is largely bureaucracy. Within the project, I keep an eye on the various branches and their progress, try to stay on top of active/problem areas in the ticket tracker, and maintain a list of things that have to happen before 1.0.
Outside the project itself, I also stay in touch with people who are distributing Django -- Linux distros which package it, etc. -- and watch their Django-related bug reports. When there's a security fix, I also send out an advance notice to them that we're going to roll a release, and do the initial disclosure so they have some lead time to respond to that.
Outside of that, I also contribute to the documentation, as well as the occasional code patch, and I maintain the 0.91-bugfixes branch, which provides legacy support and security updates for some of the really ancient Django installs out there (our policy is to support the current release and two prior releases with security updates, which means 0.91, 0.95 and 0.96 right now).
The rest of the core team is just people everybody knows if they watch the developers' list or the Trac timeline: Adrian and Jacob are the lead guys, obviously. Then there's Malcolm, who you interviewed the other day, and who's carved out a pretty big niche for himself: he did the bulk of the Unicode work and he's doing queryset-refactor, he does a lot of maintenance on the i18n system, and he somehow still finds time to have a day job. I don't know how he does that.
Then there's Russell Keith-Magee, (russelm in Trac), who's contributed all sorts of useful stuff. He's the go-to guy for testing and serialization, and just an all-around brilliant guy who fixes things left and right.
Gary Wilson and Joseph Kocherhans are also both big names you'll recognize from commit messages; Joseph used to work at the next desk over at World Online, but now he's up in Chicago with Adrian, working on EveryBlock. He's done a lot of the grunt work on newforms and on laying groundwork for newforms-admin.
Ian Kelly and Matt Boersma keep our Oracle support working, and do an amazing job of helping to track down and solve obscure problems with that.
And rounding out the core commit bits are Simon Willison and Luke Plant, who aren't as active these days but can still be seen popping up every so often.
Out on the branches, we've had a lot of work done on newforms-admin lately by Brian Rosner, who's been stepping up and helping to really get that sorted out, and there's been a lot of unsung UI work by Christian Metts, who's a designer at World Online but also a not-bad JavaScript programmer and can flaunt some Python when he feels like it.
And then there's the "gis" branch, GeoDjango, which is adding support for GIS -- spatial querying -- to the Django ORM.
Justin Bronn and Jeremy Dunck really kick-started that, Justin gave a nice demo of some of their work at PyCon this year.
And recently we've also been giving commit access to some of the translators, so they don't have to go through as much bureaucracy to submit updated translation files; that's sped some things up on the i18n front, because we're blessed with a large number of people who are willing to pitch in and do that work.
Oh, and I can't forget Wilson Miner; he doesn't flaunt it that often, but he's the guy who originally designed the Django admin, and he's been known to make the occasional tweak to fix CSS stuff.
(hope I didn't forget anybody there)
Shabda: Would GeoDjango be sometime merged with the trunk, or is it forever going to be a parallel branch to trunk?
James: The goal is that the GIS branch will merge, sometime after queryset-refactor lands, and it's going to provide an application -- django.contrib.gis -- that you can use to enable spatial queries for your models. There's a pretty good writeup on the wiki page of how that works, and they've been tracking queryset-refactor because it helps make some of the custom query construction easier.
I don't know for certain if it'll hit trunk before Django 1.0, but it will not be a branch forever; they've put in a ton of work on that, and I'm looking forward to getting to use it :)
Shabda: Once Django hits 1.0, would 0.91 be end of lifed, but support for .96 and .95 continue?
James: Like I said, the policy is we provide security fixes for the current release plus the previous two, so 0.91 would sort of fall off there after Django 1.0. But there are a lot of legacy installs out there that are perfectly happy for now, and I wouldn't be surprised to see people unofficially continuing to submit patches and keep that maintained for as long as there are people who want to use 0.91.
Shabda: A little about you. You have majored in philosophy. I can think of one other, Paul Graham. Would you say, what you learnt from philosophy help with programming?
James: Well, I wouldn't say there's anything specific necessarily. But I think there's a big place for people with liberal-arts backgrounds to come to programming, and I think philosophy's a good path to do that.
If you look at a typical philosophy program, you're doing a lot of logic, a lot of critical analysis, a lot of abstract reasoning.
You have to get comfortable sooner or later with all sorts of formalisms that don't necessarily have any practical meaning, and that's very similar in a lot of ways to programming :)
And when you get right down to it, as programmers, about 90% of what we're paid to do is think: our job is to take a problem, analyze it, break it down into pieces and solve them.
And that's not terribly different from what you spend four years doing in a philosophy program.
I've actually joked about that a bit with some of my former professors, that I still get to argue as much as when I was doing philosophy, but the programming pays a lot better.
I do think, though, that there's a big need for that sort of thing; we don't really teach critical thinking anymore, and while it's a vital skill to have no matter what you do for a living, it's absolutely crucial to programming. So if you can get a good liberal-arts background where you've been taught how to look at things and pick them apart and analyze them, you can definitely do well as a programmer. Though it'd also be a good idea to take at least a few elective math courses...
Shabda: You have been working on Ellington, how is Ellington CMS different from, say, a customized version of Drupal?
James: Well, they have some things in common: both are modular CMS-style products, both are meant to be extensible.
But Ellington is really targeted from the ground up at news operations. There's all sorts of specialized stuff in there that's really optimized for the way a newsroom works, most of it culled from our experience as a newspaper, and of working with other papers.
So where with Drupal you'd really have to do a lot of customization because nobody's really done this kind of "news all the way through" version of Drupal, with Ellington it's there out of the box, and you hit the ground running.
I think we also have a very unique position because we are a news company, we've got a bunch of newspapers, some periodicals, TV news, etc.
And we work with those folks every day, we see how they do stuff, we hear about it when they run into problems, and so we're in a good spot to see just what a newsroom staff really needs out of their online platform.
Shabda: Wordpress is PHP's killer app, arguably Basecamp is ROR's. What would you say Django's killer app is?
James: Honestly I don't know right now; I think the thing that Django wins on is that there doesn't have to be the One Big App that everybody uses, instead there's this huge blooming ecosystem of applications.
In a way, it's like asking what the "killer library" of the Python stdlib is; the killer feature is that you've got all that stuff available.
Though there are definitely some cool Django apps out there right now, and a lot more on the way. I think there's a different mentality, though, because in general Python people seem to keep their heads down and just get stuff done. So it may be you don't hear about some project until maybe they decide to do an open-space talk at PyCon or OSCON, and then you just get blown away.
There was a guy at PyCon who came up and showed me an app he's been developing, and I won't spoil it and give away what it is, but it made my jaw drop.
He'd taken something that's a really common software niche that's been dominated by these abominable products because it's not really a sexy thing to be doing, and just absolutely nailed it. Guy's probably gonna make millions.
But I don't know if there is a general-purpose "killer app" for Django right now, and I'm not really sure I want there to be one; people can see that sort of thing and think "oh, that's all it does". They look at, say, Ellington, and think "oh, this is only good for a newspaper-style CMS", or they see Rails and Basecamp and think "oh, this is no good for me, I'm not doing a Web 2.0 thing". So in a way I'm kind of glad we don't have a "killer app" hanging over us and pigeonholing Django.
Though I should definitely give a shout out to Review Board; of the public Django apps I've seen, it's probably the coolest, and again takes something that's not usually sexy in terms of software and really nails it.
Shabda: As you said, "Python people seem to keep their heads down and just get stuff done", do you think Django needs to a better job marketing itself. For example, I have pitched Django to a fair number of people, and I always have to start with with "Django what?", as compared to say ROR, or PHP which seems to have a good brand recall.
James: Well, to some extent Django hasn't done a whole lot of explicit marketing because we're not at 1.0 yet, and I expect that after 1.0 it'll both be a lot easier to do marketing and that there will be more of it going on.
But at the same time, Django's doing pretty well as it is; Google's doing their App Engine stuff with Django bundled, there are startups using it to build the next big thing, and it's even been quietly sneaking its way into some huge corporations.
Plus it seems like every time I turn around there's another book coming out.
I saw one yesterday at the bookstore downtown; I hadn't heard about it until that moment, but I think that makes five or six books that'll be out by the end of this year.
So I think that'll help. Digital Web magazine did a feature article on Django just recently, and I did an article for Sitepoint a while back as well.
I like to view this as the phase where we build up momentum until eventually Django is an unstoppable juggernaut and everybody's listening to gypsy jazz music ;).
Shabda: Everyone is pretty excited about your coming book. When can we have it in book stores? Would you give a brief overviews of what's in it?

James: The book will, I think (and hope) be shipping around the end of June.
It's very much a hands-on introduction to Django: walking through building three applications, picking up progressively more advanced bits of Django as you go, and seeing some best practices in action.
So you start out with just simple stuff, using the contrib apps and learning the basics of getting Django running. Then you do some simple customizations of admin templates, then start building some models and views, then on into building full-on applications.
There's not room to cover every single thing you can do with Django, but I think it handles a pretty good spectrum of techniques from basics up to some advanced things that let you poke around and really get a feel for how stuff works.
And of course, you get periodic bouts of me up on my soapbox yelling at people about how to write reusable applications, because that's what I do.
Shabda: You talk a little about App Engine in Batteries sold separately. What effect do you see of App Engine on python hosting ecosystem, on Python web applications, and on Django?
James: I honestly don't know what to expect from App Engine. It's a very different kind of thing from what anybody outside Google is used to, and it'll probably be a while before it's really shaken out and we get an idea of the impact it'll have. I fully expect that Google's going to start supporting other languages, so there won't be this effect where you'll always have to use Python to use App Engine. And the way they've sandboxed it is going to make it feel weird to Python people, I think. And that's on top of getting used to the fact that you're not using an RDBMS; I've been watching the blog flamewars about that, and that seems to be the big takeaway.
If I had to make a prediction now, I'd say that App Engine will bring some people to Python, but probably not in hordes, and that its big long-term effect is going to be to point out to a lot of folks that they're not really using the "R" in "RDBMS", and so maybe it's OK to think about their applications in a different way. And that's not Python-specific at all.
Shabda: What were the focus areas of the newforms-admin branch. How far have they been achieved. About when would the newforms-admin branch be merged? How backwards incompatible is this going to be?
James: Well, there are a couple goals running parallel.
The first thing, obviously, is that with the oldforms package deprecated we've got to get stuff migrated to newforms, and the admin's got to make that jump.
And because the admin does a lot of tricky and complex stuff with forms, it's also been a fertile proving ground for designing some advanced newforms features that'll find their way back into trunk and make it easier to do that tricky stuff on your own.
Another big thing is cleaning out the coupling issue where you declare a class inside a model to activate the admin interface; there's no reason for that to happen, so instead there's a class you'll subclass and override stuff on to customize the behavior, and then you'll register a model to have the admin interface you've set up that way.
Finally, there's been a lot of stuff abstracted in newforms-admin; while I don't think it's really an official design goal, there's been a lot of opportunity to provide hooks where people can go in and customize stuff. I looked at the code a couple months ago, and it was about 95% of the way to being able to run completely without django.contrib.auth, for example, because there are enough hooks where you can override something and replace defaults to make it use something else.
And that's a huge deal, because it means you could do stuff like run the admin on HTTP auth, or run it in a corporate environment against an LDAP database, and you barely have to touch anything; you can already do an auth backend that handles most of the work, then you tweak a few things for the admin and you're good to go with no hacking directly on Django code.
Plus, a lot of people will be happy to see that the methods you can go in and override all get the HttpRequest object as an argument, so even though it's bad workflow a lot of the time you'll be able to have the admin do stuff like automatically fill in a foreign key with the current user.
And along the way there've been huge numbers of bug fixes; there's a lot of stuff that's always just been really fragile in the admin, like edit_inline, and newforms-admin is a good opportunity to just yank out all the things that made them troublesome and do it right.
Shabda: Time for a meme. What is the one think you absolutely love about Django, and one thing which Django should have done differently?
James: I love contrib.localflavor, and I want people to go look at it and adore it and use it. There's so much useful stuff in there, you can validate checksums on Scandinavian social-security numbers, you can get lists of all the state in Brazil, it's just this great gigantic resource for localizing your apps, and it kicks so much ass.
Something to do differently... I'd probably rewrite the tutorial to not do the apps-inside-projects thing. That trips a lot of people up because they come away thinking they have to do it that way, and so they really miss out on the benefits of being able to write an app once and just use it over and over and over.
Shabda: Before we leave, would you like to give a quick tip, or hard to find information about Django to our readers?
James: The best tip I can give is not to be afraid to look at code. We do our best to document all the useful stuff, but that's a pretty huge area to cover, so there are always going to be neat little things hidden away that you either won't see unless you read the code, or won't see until some date in the far future when somebody gets time to document it.
Plus, if you find something cool you can write up a little tutorial and post it, and then people will start reading your blog. That's pretty much what happened to me, where my blog was for a long time just me writing up these little articles to remind myself of stuff I'd learned.
And especially if you're learning Python as you go, reading a significant piece of code like Django can really help to improve your understanding of the language. There's really nothing better than that for assimilating the way a language works, and I think Django's around the right size -- it's small enough you can carry a solid understanding of it around in your head, but it's big enough that there's lots of stuff, and lots of different kinds of stuff, that you can look at and learn from.
Shabda: Thanks a ton for this interview, and for sharing these useful information.
James: Sure.

This was James Bennett's interview. I plan to be interviewing a few more leaders from Django community, so if you would like me to ask any question, write them in the comments.


An interview with Malcolm Tredinnick - Django core contributor
Malcolm Tredinnick is a core contributor to Django, and was the driving force behind the Queryset-refactor branch of Django, which adds important capabilities such as model inheritance. He has a long association with OSS, and contributed significantly to GNOME and Django. He graciously agreed to be interviewed at 42topics blog. Malcolm's blog, Defying Classification, can be read here.
Shabda: Would you tell a few things about yourself, how did you get involved with OSS and Gnome, and with Django?
Malcolm: Here are some recollections I've written about: 1 and 2
Short version; started using Linux in university as a poor undergrad; kept using it since then (started back in 1993).
Started using GNOME in 1999 (after trying a very early version of Qt and what became KDE), started contributing to GNOME about 12 months later (mid-2000).
Started using Django in October 2005, I guess (a few months after it was open sourced) and haven't stopped. Started contributing patches more or less immediately and was given commit privileges about March of 2006, from memory.
Shabda: So what is one thing in Django which you absolutely love, and one thing which you think Django should have done differently?
Malcolm: I guess Django's separation of responsibilities is something that has always been attractive to me. It's very easy to keep the cross-application business logic and the persistent storage logic and the state control and the visual presentation separate.
I guess the piece that probably routinely annoys me the most is a slight inconsistency in the template tags: we often mix up how to use a literal string in template tags (sometimes it's in quotes, sometimes not), which makes it very hard to later make that tag also take a template variable.
As somebody who handles bug reports a lot, something like that ends up taking up a lot of time in my life, which could probably be better spent elsewhere. Still, it's almost impossible to change now, so that's the way life goes.
Shabda: Queryset-refactor is almost done, what would were the major overarching goals for this branch?
Malcolm: A number of things...
1. Clean things up internally, so that future extensions are easier. A lot of the existing (trunk) query construction grew from something much smaller. It all mostly worked, but it was getting pretty hard to manage. Some bugs were almost impossible to fix in the trunk form too, which was the real motivation.
2. Organise things so that I could add model inheritance, which was the main thing that started me looking deeply at all that. This has been accomplished.
3. Make it easier to add backends like Oracle and MS SQL Server that don't support, say, limit and offset in SQL. That meant providing a more object-based query construction so that they can tweak things before creating the SQL string.
In the process, we've made it a lot easier for people to extend this for backends and to add functionality to existing backends.
For example, adding new lookup types is now possible (not particularly easy, but possible). And the geo-django branch can do their stuff without having to patch the query construction code any longer.
(Justin Bronn, the geo-django maintainer has been tracking qs-rf very closely, so we know that geo-django works nicely with qs-rf).
I guess they're the main points. The first one -- a better/different internal organisation -- was the main one and was really the bulk of the work. Getting everything to mostly work was relatively easy. Getting it all to work perfectly took a lot longer. :-)
Shabda: So would this make, say, writing a backend for non relational DBs like Bigtable easier. Michael Trier was working on a Sqlachemy based ORM for Django, how would this be affected by qs-rf merge?
Malcolm: Well, that's really two questions. So, one at a time...
Whether it would make a non-relation backend easier,... yes. The QuerySet class passes all the work of talking to the backend, whatever it is, off to a class called Query. Writing a different sort of backend would require writing a new Query class that knew how to take the function calls QuerySet makes and turn them into the right return types.
Mostly, QuerySet does not poke at the internals of Query: it uses "public" methods only. This is deliberate. It means that writing something like a Query class for a different sort of backend (RDF store, or LDAP or whatever) might very well be possible.
As for something like SQLAlchemy, I gather it's made things easier. I know Michael and Brian have been writing their code against the qs-rf code and it seems to be working.
However, I haven't looked at their code, so I'm not sure if they're using the QuerySet/Query split very much or not. I gather they're doing it slightly differently from how I might have done based on somethings they mentioned when I was talking to them yesterday -- apparently a recent qs-rf commit "broke" their approach to something and that was a little surprising to me. But it wasn't a showstopper apparently, so I'm not too worried.
Shabda: As you said, a commit sort of "broke" their approach. How backwards comaptible is qs-rf. Is it mostly backwards compatible, but some corner cases will break, or would any thing which worked previously, but does not now should be reported as a bug?
Malcolm: It's almost 100% backwards compatible. The thing I "broke" (and I'm using quotes, because it wasn't really a breakage, just a change) was a very internal thing. Obviously if you're writing something like a new storage backend, you need to dive into the internal API of the ORM a bit and that's what the django-sqlalchemy stuff is doing...
For normal user code, there should be very few changes required. There's a list on the wiki page for the branch but they are mostly things that won't affect normal code.
Only if you are doing something very "tricky" or slightly unsupported will changes be required.
Of course, some people will see slightly different results in complex querysets because we have fixed a lot of bugs (over 60 that were reported to Trac plus some that never got that far) and some of those fixes change the results that were returned.
So some people may be relying on currently incorrect behavior, but I doubt that's going to be too troublesome.
Also, I've tried to make the error reporting a lot clearer so that when somebody does try to do something undefined, it should hopefully give an error message (e.g. introducing an infinite ordering loop in models)
Shabda: Speaking of Bigtable, Google's App Engine seemed to be sort of a letdown from the initial hype. What are your opinions about the effect of App Engine on Django?
Malcolm: Well, I think the letdown came from people letting their expectations get ahead of their brains a bit. Google didn't really seem to over-hype it or misrepresent it.
Obviously when Google releases something, everybody gets excited. But you have to at least take the time to look at what was released.
For a particular class of applications, I think App Engine is probably ideal. It gives cheap hosting, great reliability and access to some Google's storage stuff.
For example, if you have something that presents a read-only view onto a large bunch of data, App Engine would be very appropriate. Okay, you have to do data extraction from multiple models manually, since there's no joining, but that's not too hard to work around and people will write little helper functions for those cases.
Remember that a lot of very successful, very popular websites are basically read-only: a lot of newspaper sites, things like EveryBlock (and, formerly, Chicago Crime) -- all those sorts of sites. Even news.google.com.
There's a great deal of information that wants to be "presented" to people. Not everything requires form entry and comments and the like and even those are possible on App Engine if you need them.
So I'm kind of glad the initial reaction phase has settled down a bit and people are starting to think about what is possible, rather than focusing on what isn't possible. Some interesting things will come out of App Engine, I suspect. We just don't know what they are yet because people are still writing them.
As far as how it affects Django, that's probably both good and bad.
If people view App Engine as being representative of everything Django, it's not going to be too handy. However, I doubt that will be the prevailing opinion. People realise that Google have used portions of Django to implement portions of the App Engine experience. And those pieces of Django are quite handy. The URL dispatching, the general data flow, the templates -- all will be nice for people to use.
It's also getting a few people interested in the internals of Django as they try to work out whether they can move the extra bits that are 'missing' from App Engine into the fold.
In addition, we've also already received some direct positive benefits: some of the Google team, particularly Guido van Rossum, have been filing a few bugs against Django as they did the initial App Engine development. So there are a few bug fixes in things like newforms and the templating component that are a direct result of App Engine development inside Google.
Shabda: There is app-engine-django, which is trying to bridge the impedance mismatch between Django ORM, and App Engine ORM. Is there any effort to bridge this from Django side by making Django ORM play nice with Appengine. If such a request comes in what would Django's response be?
Malcolm: shrug. I have no idea.
I think this is an ideal opportunity for some people to write stuff like django-appengine and work out what might be needed and come up with a concrete list of things that might be needed. Predicting the future is hard enough to be mostly futile and it's not really worth speculating.
Right now, the core Django developers are fully focused on getting Django 1.0 released. That's where our time is going. How other people spend their time is up to them and it's definitely not for me to judge whether it's worthwhile or not.
People need to experiment. It's the way new software products are developed. So anybody who's interested and working on this should be encouraged. That's how open source development has always worked and I think it works quite well.
I guess I could hazard a guess that any intrusive changes just to support some App Engine style thing probably aren't interesting prior to Django 1.0, since they're not necessary to get that out of the door. But that's as much of a prediction as I'm willing to make and anybody who actually attaches any value to that prediction is probably being foolish. I'm wrong more often than I'm right. :)
Shabda: Well this gets asked a lot on the mailing lists, qs-rf is almost done, Newforms admin should be done soon, so when can we expect Django 1.0, and what new goodies will this bring?
Malcolm: heh
Well, the real answer is "as soon as possible".
Notice that this is a big step up from "when it's ready", which is the normal answer here. Hopefully people appreciate that the maintainers want to get 1.0 out the door as much as anybody else. It's just that release 1.0 when it's not at a point we're happy with won't be useful in the future.
More concretely, we have a few major things to do between now and 1.0 and they're all getting pretty close.
Then there's a period of stabilizing and random bug squashing and triage to make sure we haven't missed anything big.
The idea (and driving goal) is that whatever we release as 1.0 should remain backwards-compatible on an API level until, say, 2.0. So we need to get all the big infrastructure changes in before 1.0.
That isn't the same as saying 1.0 will be bug free or any pipe dream like that, because it won't be. But fixing bugs is an ongoing job and as long as we can feel comfortable that we can fix existing bugs without having to make major breakages, we can feel happy about 1.0.
Shabda: This has happened to me a few times, I am trying to pitch Django to a client, and then I mention Django has not hit 1.0 and we would be developing their site against a subversion checkout, and boom, I am against a brick wall. I am guessing this happens to a lot of people. Any suggestions on overcoming client's reluctance on this point.
Malcolm: Well, this is the hard problem, of course. There are at least three issues here.
Firstly, it's well known that in a lot of corporate situations, they only want to go with "released" versions, whatever that means. The fact that released versions are often a disaster (e.g. Microsoft Vista) seems to be forgotten, but that's the way things go. However, it shouldn't be the ultimate driving force in this equation. It's one consideration only and having the software released faster just to meet some immovable, often-times unsupportable policy isn't the answer.
The second aspect is to realise that there are a lot of successful sites built on Django 0.96 and even Django 0.91. So it's not impossible to stick with the last release.
Now, of course ,that tends to show up the problem with the first point, since 0.96 has some bugs that have subsequently been fixed, so sticking to a release handicaps you there.
However, if we did more frequent releases, that adds a lot of burden to the release process. For example, distributions (Debian, Suse, Fedora, etc), now have a bigger problem in choosing the right version to ship since they also want some stability. Security releases either become harder (we have to patch more versions) or else we have to "end of life" earlier releases more frequently, which harms those people using those releases (again, it reduces stability for the userbase).
The third point is somewhat related to that: in order to make sure that a release can be relied upon, we sometimes need to make sure that we do all the necessary preliminary work so that we don't immediately break any code that relies on this new release.
That is exactly the situation we're in now. We release 0.96 as a sort of "stability point" for people to rely on. Almost immediately, we then started to introduce a bunch of necessary changes to things that will require code to be ported when people upgrade. When this round of churning has finished, we'll make the next release. That happens to be 1.0 in this case.
If we released, say 0.97 at some point in the past few months, people would have a similar but slightly different sort of marketing problem: now they have 0.96 code that has to be ported if they want to move to 0.97. But it's also just "more work" on some level, since they'll have to do even more porting when 1.0 comes out. So we aren't helping developers by providing them with a lot of stability.
It's fairly well understood by most experienced developers that this is a tough path to walk. With open source software it's even harder, since everybody is a volunteer and we have a largely unknown userbase to satisfy.
Sometimes that difference is forgotten by people who argue that "no company ever works like this". A corporation can set release deadlines and in exchange for everybody having to work to those deadlines, those people get paid. It's called a salary. Open Source software doesn't work like that.
I'm sure that once Django reaches 1.0 or perhaps a little later, we'll slip into time-based releases so that things go a bit more smoothly. At the moment that simply isn't practical because of all the changes we need to make.
Every project goes through that phase. Before Linux had time-based releases, they had to get an amount of features in and a certain stability level reached. Ditto for GNOME (which only started doing time-based releases at 2.0 and there was a long gap between 1.4 and 2.0 precisely because we needed to be able to guarantee that at 2.0). Ditto for KDE and Ubuntu and ...
So, yeah ,it's slightly tough times for everybody at the moment. Partly Django is a victim of its own success here: everybody wants to use it and we're trying to keep up.
Hopefully people realise that ultimately that's a good thing. It's an investment in the future in the sense that we'll still be here in a year, two years, five years... because of this great use.
Shabda: What are the future plans for Django. Django has most of the things which I want in a framework, (import soul), and we would hate to have Django suffer from featuritis. After 1.0 what are the areas Django wants to tackle?
Malcolm: Again, this is asking for a predication and I don't do those.
Partly because I don't know and am not really in a position to say, in any case. (I'm just a contributor). Partly because having too many goals too far ahead of time is possibly going to restrict people.
Django runs on its contributors. A lot of the ideas that are implemented are initially suggested and often developed in quite some detail by people other than the core developers. It might not always seem that way, because you sometimes have to see the common thread in a bunch of requests before you notice the lurking feature requirement ,but it's true.
So what happens post 1.0 is going to depend a lot on how people use Django, on what people offer in the way of code and, particularly, what good ideas are suggested.
Who could have predicted Google AppEngine when Django 0.96 was released? What other things like that will appear as a result of Django 1.0? Who knows?
Of course, there are some things that we could probably say will be worked on (successfully or not is another question entirely). Multi-database support seems to be popular and one day somebody might do it; maybe there'll be more front-end developments -- e.g. somebody like you who keeps wanting more Ajax support will actually develop something that provides this support everybody seems to want but hasn't ever specified. :-)
We can sort of see a few things coming up again and again, so I guess they'll be worked on. But it's really going to be up to the people who write the code, who come up with the ideas, who write the websites that use Django, who try to teach Django to others. And right now, I have no idea what direction that will go in.
Shabda: Before we close, would you like to share a handy tip which you use a lot, but does not get used so much otherwise?
Malcolm: People possibly overlook both the {% debug %} template tag (best to use it as <pre>{% debug %}</pre> in a template) and the debug context processor django.core.context_processors.media.
Both of those are very useful for trying to see what's going on when you're passing information between a view function and a template.
Shabda: Thanks Malcolm. It was great talking to you.

I plan to interview Leaders from Django community, in coming few weeks, so if you would like me to ask any specific questions, put them in comments, and I will ask them when I interview.


Five Things I Hate About Django.
The five things I hate about * meme seems have died down, and memes, should not be allowed to die.
Of course I love Django, and have bet very heavily on it. But we do not know a topic, until we know it warts, so here you go. The listing is in no particular order, so sorry no numbering.
Ajax with Django is hard:
Most of the Django Community has decided that bundling Javascript helpers with a python framework is bad idea. Though I can understand the reasoning, (argued here and rebuttal), that Javascript is so basic that you can not be expected to not know it, I can not agree with it. SQL is as basic as Javascript, and yet we have ORM for abstracting away the common and the tedious.
Of Course, with simplejson, and a good Javascript library, you can build Ajax apps fast and with only a minimal amout of fuss. And yet switching between Python and Javascript, twice every hour is a huge time drain. Eg. I put commas after the last element in Python arrays, with JS this would work in FF, but fail with weird errors in IE.
Lack of identity map:
If you get the same row from the DB twice using Model.objects.get, you will get two different objects. Apart from the performance problems of two DB queries, when only one should have done, when you update one of them, the other does not get updated, and you will have interesting things happening in your application. And if you update both of them, you might write two inconsistent changes to the DB.
Look at this code for example.
See this code
In [2]: from django.contrib.auth.models import User
In [3]: usr1 = User.objects.create_user('ram', 'demo@demo.com', 'demo')
In [4]: usr2 = User.objects.get(username='ram')
In [5]: usr3 = User.objects.get(username='ram')
In [6]: user2 == user3
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
...
In [7]: usr2 == usr3
Out[7]: True
In [8]: usr3.username = 'not_ram'
In [9]: usr3.save()
In [10]: usr2.username
Out[10]: u'ram'
In [11]: us3.username
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
...
In [12]: usr3.username
Out[12]: 'not_ram'
In [13]: usr2 == usr3
Out[13]: True


Whether Sessions are browser length/persistent are set sitewide:
You can set whether you want sessions to be browser length/persistent using SESSION_EXPIRE_AT_BROWSER_CLOSE in settings.py. But you can not set them per user, without mucking with django internal. This might seem a minor annoyance, yet this is something which you need to do for every app, as the remember me, function will not work without this.
Newforms is very limited:
Let us say you want the Form to contain a varible number of fields. How can you define the NewForms class to do your biddings.
from django import newforms as forms
class MyForm(forms.Form):
    foo = froms.CharField()
    bar = froms.CharField()


This can only create a form with a fixed number of fields. While there are ways to generate forms with variable number of fields, (generate the Form class programatically), they are not easy or well documented. (Remind me to write such tutorial sometime.)
Bonus question: How can you generate a form with same form elements multiple (and variable number) times, ala what happens with edit_inline?
Settings mixes application configuration which should be public and passwords, which should be private:
If I am distributing an app MIDDLEWARE_CLASSES is something which I would assume users would not (generally) modify. Similarly, in most of the cases, INSTALLED_APPS, would also be something which users would not change, (unless you are distributing standalone_apps). This means, I want to source control settings.py. But settings.py also contain my DB setiings, and SECRET_KEY, which means, I cannot source control settings.py.
And while we are at it, can we refactor settings.py, so it works without
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'


Bonus:
Two things which used to bug me but no more.
1. You cannot extend Models - Well now you can if you use queryset-refactor, or soon can if you are on trunc.
2. Url configuration using regexes. - Now they have two problems. joke notwithstanding, mapping URLs to views is one problem where regexes fit the problem beautifully. With less that 50 lines of code, you can manage a large number of views, and Url patterns.
Now stay tuned for Five things I love about Django


New tutorial - Building a search engine with Appengine and Yahoo
I wrote a new tutorial on building a search engine using Appengine, and Yahoo Search API here. This uses pure Appengine API, and not Django, and is a tutorial on how to use Appengine without Django.


Two Django+Appengine Tutorials
I have posted two Tutorials for Using Django with Appengine.

For people who do not know Django
For people who already know Django  see what we build in this tutorial

And here a few good links about the topic.

James Bennet tells exactly why Appengine and Django are not so good together.
Ian Bicking has an interesting take on how Appengine can change the economics of Python hosting.
The guys at Joyent reading my mind on why I or you can not deploy any production site on Appengine. (Hint. you mean I can never move away, without writing half my code?)



Using Appengine with Django, why it is pretty much unusable
We are hard at work building 42topics.com, and are looking at the best places to deploy this. So when I heard about Appengine, with out of box Django support(gasp!) I was delighted. After using it for a day, and posting a tutorial, I am so disappointed.
Peeves in no particular order.

Appengine is a very constrained environment, so out goes any chance to run background jobs.
The ORM-API is very similar to Django, but yet the Django API is much better. modelobj.filter('attr =', value1).filter('attr2 =', value2) or modelobj.filter(attr = value1, attr2 = value2). Putting entity level operations on modelobj.manager is much cleaner that making them classmethods, as argued here.
Very half baked documentation. Releasing without sufficient testing on windows. If you follow the Getting started guide, and are on windows, I hope you like debugging regexes
NO JOINS? Ok so I can use obj.master to simlute this. Umm can I get GROUP BY? What about UNION? Admittedly Bigtable is not relational, but are you telling me Google built all their search without a way to simulate GROUP BY.
PDB does not work on dev server. With Django's dev server, I put breakpoints with PDB all the time, and it works perfectly. With Appengine devserver, pdb will give you BDBQuit exceptions. I hope you enjoy debugging by reading logfiles.
Modules used by django are empty.
You can not start a dbshell on the dev server. (And django-admin framework does not work). When I am coding, I write the data insertion views first, and then the other views. Whether the Insert views are working or not can be checked immediately, from the db shell, or from admin. Here until I write the other views, I can not check on my create view.

The other flaws I can live with, but a dbshell not working and pdb breakpoints raising exceptions make this unusable for me. I guess I will stick with Django on a normal web host, and look at EC2, when we really need to scale.


Google Appengine - First Impressions
Google launched their EC2 competitor, Appengine yesterday, and all hell broke loose. And in about 24 hours, the 10,000 accounts were used up. Currently it is tied to only working with python, and Django 0.96.1 works out of the box.
<!- - more - ->
The Good

Python powered. Django works out of the box.
No sysadmining chores.
Promise of infinite scalability with no configuration. (Ah!)
Free for now.

The Bad

Python powered, if you want to use ruby/java/php, sorry you are out of luck.
Your code is tied to Google. You might be able to reuse most of your code, but the DB/ORM sepcific code, ah you are out of luck. And if you are building database backed websites, well most of your most complex code talk to ORM.
Too magical. Explicit is better than implicit. On the dev server I do not know where my data is stored. If I change the data model, the changed models are available immediately. But well, how do you do it?
Free for now, and no way to pay for when your usage out grows the free quota limits.

	Rails	Django
Configuration Files	config/database.yml for database settings config/environments/ development.rb for development specific settings production.rb for production specific settings test.rb for test specific settings	settings.py, one file for everything, database configuration and any other configuaration or settings will be in this file
URLs	config/routes.rb	urls.py
Schema/Models	db/schema.rb for the ruby version of db schema app/models/* for the models	complete schema is not stored any where Under every application in your project, models.py file will contain the table specific schema stored as django models
Management Commands	./script/server, start server ./script/console, ruby console ./script/dbconsole, database console rake db:migrate, run database migrations	manage.py is the file for all your tasks ./manage.py runserver, start server ./manage.py shell, python console ./manage.py dbshell, database console ./manage.py syncdb
Application Code	app/controllers/* will contain the application logic	views.py file under each application folder is the place to write to your application logic, file can be named with any name, views.py is the general convention
Application templates	app/views//* is the place for the templates	default: looks inside 'templates' directory, under the application directory looks in the directory specified by 'TEMPLATE_DIRS' in settings.py