Blog Roll

ESI using varnish and django

2010-01-03T21:43:00.000-04:00

In my previous post [2] I wrote that Varnish [1] has much more to offer than upstream cache. I have decided to explore its ESI supports [3].

ESI : Edge Side Includes (ESI) is an XML-based markup language that provides a means to assemble resources in HTTP clients. Unlike other in-markup languages, ESI is designed to leverage client tools like caches to improve end-user perceived performance, reduce processing overhead on the origin server, and enhanced availability. ESI allows for dynamic content assembly at the edge of the network.

On of the most common difficulties that leads you to not cache a page for a logged in user is that you want to display some custom information for that user. For example you want to be able to display:

* " welcome joes, [ link to his profile ] "

However the rest of the page will be common for all the users. The diagram below explains the composition of the page :

The yellow part of the page is common for all the users where the green part of the page should be customized for every user.

This is a very common pattern, you can also have a header, footer and a navigation block that don't change very often and the rest of the page which is more dynamic like: recent activity, last articles, ... So the idea here is to use varnish to assemble information coming from different urls and having a different lifetime in cache for each item.

So in our example we will cache the "dyn_page" (yellow in the diagram) for five minutes and we will never cache the user info. I am going to start by dumping the code for this toy app and then explain it as we progress.
Here it is the code of the views.py:


import time
from django.views.generic.simple import direct_to_template
from django.views.decorators.cache import cache_page, never_cache
@never_cache
def user_info(request):
    return direct_to_template(request, "esi_app/user_info.html")
@cache_page(60*5)
def view_dyn_page(request):
    #Simulate the fact that this page take a lot of time to be built
    time.sleep(2)
    return direct_to_template(request, "esi_app/dyn_page.html")
@never_cache
def view_page(request):
    USE_ESI = True
    if not USE_ESI:
        #Simulate the fact that the dynamic part of the page
        # take a lot of time to be built
        time.sleep(2)
    return direct_to_template(request,
                              template="esi_app/page.html",
                              extra_context={'USE_ESI':USE_ESI,})

The views.py contains 3 views one that can display the information for each individual block (yellow and green in the diagram above) and one that can displays the complete page. Note the "USE_ESI" variable that we will utilize in our template. I have added a sleep of 2 seconds in the code to simulate an operation which is taking a lot of time thus the caching strategies make more sense and my ab test later or will be more meaningful.

Here it is the code of of the urls.py:


from django.conf.urls.defaults import *
urlpatterns = patterns('esi_app.views',
    url(r'^page/', 'view_page', name='view_page'),
    url(r'^dyn_page/', 'view_dyn_page', name='viewd_dyn_page'),
    url(r'^user_info/', 'user_info', name='user_info'),
)

Here it is the code for the base.html:



<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"

        "http://www.w3.org/TR/html4/loose.dtd">

<html>

<head>

    <title>{% block title %} {%endblock%}</title>

    <style type="text/css">

        #user_info {

          background:lightgreen;

        }

        #content {

            background:lightyellow;

        }

    </style>

</head>

<body>

    <div id="user_info">

        {% block user_info %} {%endblock%}

    </div>

    <div id="content">

        {% block content %} {%endblock%}

    </div>

    

</body>

</html>

Here it is the code for the page.html:


{% extends "esi_app/base.html" %}

{% block user_info %}

    {% if USE_ESI %}

        <esi:include src="http://192.168.1.18:6081/esi/user_info/"/>

    {% else %}

        {% include "esi_app/user_info.html" %}

    {% endif %}

{%endblock%}

{% block content %}

    {%if USE_ESI %}

        <esi:include src="http://192.168.1.18:6081/esi/dyn_page/"/>

    {% else %}

        {% include "esi_app/dyn_page.html" %}

    {% endif %}

{%endblock%}

This template use the variables "USE_ESI" to decide whether the page will be built using an ESI server or not. This allows a graceful degradation and will help you to debug your page. In a real life situation this variable might come from a django's context processor. The idea here is that "/esi/page/" is built using "/esi/dyn_page/" and "/esi/user_info/"

{% load webdesign %}

<h1>{% lorem 2 w random %}</h1>

<p>{% lorem 2 p random %}</p>

Here it is the code for the user_info.html:

{% if user.is_authenticated %}

    Welcome, {{ user }} -- <a href="/admin/logout">Logout</a>

{% else %}

    <a href="/admin/login">Login</a>

{% endif %}

Then you will need to configure varnish to make it do the ESI transformation on the page with the url equal to "/esi/page/".
Here it is the varnish configuration for the /etc/varnish/default.vcl:


backend default {
.host = "127.0.0.1";
.port = "8080";
}
sub vcl_recv {
     unset req.http.Accept-Encoding;
     #unset req.http.Vary;
}
sub vcl_fetch {
     if (req.url == "/esi/page/") {
         esi;
     }
}

The code below is rather self explanatory it tells Varnish to do ESI substitution on the page located at "/esi/page/". has in my previous post Cherokee is located on the port 8080 and Varnish on the port 6081. The trickiest part there is the vcl_recv, in this section varnish explicitly prevents the backend from gzipping the content.

All the machinery is in place now so you can use your favorite browser to visualize the result :
  * go to http://192.168.1.18:6081/esi/page/ to view the page generated by varnish
  * go to http://192.168.1.18:8080/esi/page/ to view the page returned by Cherokee

Curl is another tools that is useful when playing cached page :
  * curl [url] wil display in the console the html source code
  * curl -I [url] will show the document info in a console.
  * curl [url] -H "Accept-g: gzip,deflate"

Here it is the page rendered by cherokee :

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"

        "http://www.w3.org/TR/html4/loose.dtd">

<html>

<head>

    <title> </title>

    <style type="text/css">

        #user_info {

          background:lightgreen;

        }

        #content {

            background:lightyellow;

        }

    </style>

</head>

<body>

    <div id="user_info">

        <esi:include src="http://192.168.1.18:6081/esi/user_info/"/>

    </div>

    <div id="content">

        <esi:include src="http://192.168.1.18:6081/esi/dyn_page/"/>

    </div>

</body>

</html>

Here it is the page after the substitution done by Varnish :


<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"

        "http://www.w3.org/TR/html4/loose.dtd">

<html>

<head>

    <title> </title>

    <style type="text/css">

        #user_info {

          background:lightgreen;

        }

        #content {

            background:lightyellow;

        }

    </style>

</head>

<body>

    <div id="user_info">

    <a href="/admin/login">Login</a>

    </div>

    <div id="content">

<h1>ipsa sed</h1>

<p><p>Nemo perferendis delectus pariatur aliquid repellendus repellat explicabo facilis, molestiae veritatis odit, accusantium repellat culpa ab laboriosam, iste laborum amet et harum, iusto illum ipsa a quos necessitatibus voluptatem consectetur cumque? Doloremque atque delectus ipsa ad veniam incidunt cum exercitationem voluptates labore, sapiente ducimus deserunt expedita aperiam temporibus omnis magnam qui architecto, pariatur voluptates nesciunt nam ab dolore omnis, quo voluptatem nihil accusamus aperiam excepturi exercitationem? Consectetur mollitia neque quod quas.</p>



<p>Tempore amet voluptate ipsum suscipit placeat exercitationem labore nam voluptas, debitis esse dignissimos, fugiat illum asperiores suscipit deleniti maiores consequuntur, doloribus architecto repellendus dicta nemo corporis explicabo? A fuga ex, voluptates quam dignissimos aspernatur, reprehenderit accusantium id magni ut debitis adipisci esse voluptas tempora, quas doloribus blanditiis voluptatum veniam nam magni et adipisci fuga pariatur provident? Aut ipsum quam quia earum quod cum sapiente officia inventore delectus, expedita ex quia ipsam consectetur exercitationem ut ad sunt illum minus voluptatum, accusantium maxime facere eos numquam explicabo, rerum ab dolorem repellendus, praesentium debitis tempora aut facere sapiente odit veniam quae?</p></p>

    </div>

</body>

</html>

Some quick and dirty ab test will show you the interest of this Technic. Varnish is very fast at assembling the content coming from different sources. Several months ago Adrian Holovaty has written an article about an alternate approach to this class of problem.

I would be glad to hear from you what other varnish tricks can be used on top of a django web application.

[1] http://varnish.projects.linpro.no/
[2]http://yml-blog.blogspot.com/2010/01/response-time-optimisation-with-varnish.html
[3] http://varnish.projects.linpro.no/wiki/ESIfeatures
[4] http://www.w3.org/TR/esi-lang
[5] http://www.holovaty.com/writing/django-two-phased-rendering/

Response time optimisation with Varnish

2010-01-02T08:56:00.000-04:00

This blog post shows you how to optimize the tools chain on your server to improve its performance by an order of magnetude with out changing a single line in your django project. In order to do so I will use again django-cms [1] as guinea pig because there is a fair amount for processing to display a page but it is still easy to install. Note: django-cms example has the cache middleware activated by default.

Then I will run ab testing on a particular page and compare the results. These tests are being performed on my laptop hp dv6-1030. The important information is not the figures but by them self but rather the variation of the response time.

Before starting my test I have moved django-cms to be mounted under "/". In order to do this you will need to change the configuration into the file called example_uwsgi.py.


import os
import django.core.handlers.wsgi

# Set the django settings and define the wsgi app
os.environ['DJANGO_SETTINGS_MODULE'] = 'example.settings'
application = django.core.handlers.wsgi.WSGIHandler()

# Mount the application to the url
applications = {'/':application, }

Then you need to change the rule behavior in cherokee admin to reflect this change. Cheorkee admin makes this task a breeze.

Before diving head first into the the meat of this article here it is a diagram of the architecture that we are going to work with :

The goal of this article is to show you the incredible boost that varnish can give to certain type of web application.

varnish [2] is a state-of-the-art, high-performance HTTP accelerator. It uses the advanced features in Linux 2.6, FreeBSD 6/7 and Solaris 10 to achieve its high performance.
Some of the features include:

VCL - a very flexible configuration language
Load balancing with health checking of backends
Partial support for ESI
URL rewriting
Graceful handling of "dead" backends
...

The bottom line is that just by installing it and using it with a vanilla configuration, on ubuntu, will increase the responsiveness of your site by an order of magnitude that is hard to believe we are talking here of an improvement factor ranging from 50 to 600 times.

The first thing that you would like to do is to install varnish [2]. On ubuntu varnish is very easy to install/configure since there is a package that exists. Once this operation is executed you will need to define the backend, this varnish jargon means that you need to tell varnish where Cherokee is located.


backend default {
.host = "127.0.0.1";
.port = "8080";
}

Here it is some ab tests that I have done to illustrate this article, 8080 and 6081 are respectively the port for Cherokee and Varnish.

Cherokee

ab -n 100 -c 1 http://192.168.1.18:8080/

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient).....done

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /
Document Length: 3440 bytes

Concurrency Level: 1
Time taken for tests: 15.285 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 364300 bytes
HTML transferred: 344000 bytes
Requests per second: 6.54 [#/sec] (mean)
Time per request: 152.851 [ms] (mean)
Time per request: 152.851 [ms] (mean, across all concurrent requests)
Transfer rate: 23.28 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 2.4 0 24
Processing: 133 153 17.3 149 235
Waiting: 133 153 17.3 149 235
Total: 134 153 17.2 149 235

Percentage of the requests served within a certain time (ms)
50% 149
66% 158
75% 164
80% 166
90% 172
95% 175
98% 230
99% 235
100% 235 (longest request)

ab -n 100 -c 50 http://192.168.1.18:8080/

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient).....done

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /
Document Length: 3440 bytes

Concurrency Level: 50
Time taken for tests: 8.202 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 364300 bytes
HTML transferred: 344000 bytes
Requests per second: 12.19 [#/sec] (mean)
Time per request: 4101.021 [ms] (mean)
Time per request: 82.020 [ms] (mean, across all concurrent requests)
Transfer rate: 43.37 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.1 1 3
Processing: 740 3283 1158.0 3906 4438
Waiting: 740 3283 1158.0 3906 4438
Total: 743 3284 1157.1 3906 4438

Percentage of the requests served within a certain time (ms)
50% 3906
66% 4048
75% 4112
80% 4182
90% 4285
95% 4341
98% 4359
99% 4438
100% 4438 (longest request)

ab -n 100 -c 100 http://192.168.1.18:8080/

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient).....done

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /
Document Length: 3440 bytes

Concurrency Level: 100
Time taken for tests: 8.236 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 364300 bytes
HTML transferred: 344000 bytes
Requests per second: 12.14 [#/sec] (mean)
Time per request: 8235.626 [ms] (mean)
Time per request: 82.356 [ms] (mean, across all concurrent requests)
Transfer rate: 43.20 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 3 4 0.8 4 5
Processing: 699 4533 2337.0 4650 8228
Waiting: 699 4533 2337.0 4650 8228
Total: 704 4537 2336.2 4654 8230

Percentage of the requests served within a certain time (ms)
50% 4654
66% 5884
75% 6717
80% 7017
90% 7749
95% 8115
98% 8221
99% 8230
100% 8230 (longest request)

Varnish

ab -n 100 -c 1 http://192.168.1.18:6081/

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient).....done

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 6081

Document Path: /
Document Length: 3440 bytes

Concurrency Level: 1
Time taken for tests: 0.030 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 374800 bytes
HTML transferred: 344000 bytes
Requests per second: 3320.49 [#/sec] (mean)
Time per request: 0.301 [ms] (mean)
Time per request: 0.301 [ms] (mean, across all concurrent requests)
Transfer rate: 12153.53 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 0 0.9 0 8
Waiting: 0 0 0.9 0 8
Total: 0 0 0.9 0 8

Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 4
99% 8
100% 8 (longest request)

ab -n 100 -c 50 http://192.168.1.18:6081/

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient).....done

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 6081

Document Path: /
Document Length: 3440 bytes

Concurrency Level: 50
Time taken for tests: 0.012 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 378548 bytes
HTML transferred: 347440 bytes
Requests per second: 8522.97 [#/sec] (mean)
Time per request: 5.866 [ms] (mean)
Time per request: 0.117 [ms] (mean, across all concurrent requests)
Transfer rate: 31507.35 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.2 0 3
Processing: 0 3 2.0 3 8
Waiting: 0 3 2.0 2 8
Total: 0 4 2.8 4 11

Percentage of the requests served within a certain time (ms)
50% 4
66% 5
75% 6
80% 7
90% 9
95% 10
98% 11
99% 11
100% 11 (longest request)

ab -n 100 -c 100 http://192.168.1.18:6081/

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient).....done

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 6081

Document Path: /
Document Length: 3440 bytes

Concurrency Level: 100
Time taken for tests: 0.013 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 374800 bytes
HTML transferred: 344000 bytes
Requests per second: 7662.25 [#/sec] (mean)
Time per request: 13.051 [ms] (mean)
Time per request: 0.131 [ms] (mean, across all concurrent requests)
Transfer rate: 28045.03 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 2 3 0.6 3 4
Processing: 7 7 0.7 7 9
Waiting: 5 6 0.6 6 7
Total: 9 10 1.2 10 13

Percentage of the requests served within a certain time (ms)
50% 10
66% 10
75% 10
80% 11
90% 12
95% 13
98% 13
99% 13
100% 13 (longest request)

Conclusion

Installing varnish in front of your web server is propably this first step you should take in the end less journey of optimising your web application. It is interesting to note that in addition of dramatically improving the response time Varnish will also reduce the load on your application server stack [ uWSGI + django +db].

This blog post barely scratches the surface of how django can take advantage of of caches, django gives you the possibility to cache information at different stages during the request/response cycle. You can cache the output of specific views, you can cache only the pieces that are difficult to produce, you can cache a portion of template, or you can cache your entire site. Django also works well with "upstream" caches, such as varnish and browser-based caches. These are the types of caches that you don't directly control but to which you can provide hints (via HTTP headers) about which parts of your site should be cached, and how. If you want more information about this you can read the django's cache documentation.

Varnish is also a beast by itself, you can fine tuned it to suit your particular situation and you can used it to do much more in your infrastructure than just upstream cache of your dynamic web site.

[1] http://yml-blog.blogspot.com/2009/12/flup-vs-uwsgi-with-cherokee.html
[2] http://varnish.projects.linpro.no/
[3] http://docs.djangoproject.com/en/1.1/topics/cache/#topics-cache

flup vs uWSGI with cherokee

2009-12-29T12:54:00.001-04:00

Cherokee is one of this many web servers that supports both deployment strategies. Since I have recently blogs about uwsgi

Several people asked me how do they compare performance wise. Up to now my answer was I don't know and I don't really care because this is not the most critical aspect in my opinion.

However since this question keep comming I have decided to give it a more accurate answer. This blog post takes you on how to setup cherokee with both alternatives and compare their performance. The guinea pig application I have chosen to do this comparison is django-cms because I think it represents fairly well the overhead introduced by the dynamic generation of the page.

Note : in this example django-cms will not be mounted under "/" this will cause all sort of issues if you try to do this but none of them impact the particular page that we try to access.

One of the nice Cherokee's feature is its documentation, here it is the 2 pages related to this blog post :
* flup [1]
* uswgi [2]

In this blog post I will assume that django-cms is installed, its example project properly configured and that you have added a page to test test against it. I will aslo assume that cherokee, flup and uwsgi are setup.
During this test I have used uwsgi changeset: 145:db356717823c.

Cherokee provides 2 wizard that enables you to get running very fast. First thing first you will need to start cherokee admin :

sudo cherokee-admin -u

media admin
Add a rule to serve the "/media/admin" in my case these files are in "/opt/www/django-cms_tutorial/ve/lib/python2.6/site-packages/django/contrib/admin/media"

media cms
Add a rule to serve the "/media/cms" in my case these files are in "/opt/www/django-cms_tutorial/ve/src/django-cms/cms/media/cms"

Flup

In this case you will need to use the wizard called "Plateforms > django". This wizard takes the web directory and the project directory. In our case :

web directory : /flup
project directory : /opt/www/django-cms_tutorial/ve/src/django-cms/example

Then we need to modify the interpreter command from the "django 1" source because we used virtualenv instead of the global python.
So we need to change it from :

python /opt/www/django-cms_tutorial/ve/src/django-cms/example/manage.py runfcgi protocol=scgi host=127.0.0.1 port=37134

to:

/opt/www/django-cms_tutorial/ve/src/django-cms/example/start_fcgi.sh

Then you need to create this file called start_fcgi.sh


example$ cat > start_fcgi.sh << EOF
> source /opt/www/django-cms_tutorial/ve/bin/activate;
> python /opt/www/django-cms_tutorial/ve/src/django-cms/example/manage.py runfcgi protocol=scgi host=127.0.0.1 port=37134
> EOF

chmod +x start_fcgi.sh

Restarts cherokee and direct your browser to "/flup/".

uWSGI

In order to setup the uwsgi server we are going to use the cherokee's wizard and then modify the result to adapt it to our particular use case.

uwsgi wizard takes only one argument the path to the uwsgi configuration file :

/opt/www/django-cms_tutorial/ve/src/django-cms/example/example_uwsgi.py

We are going to edit the interpreter command to adapt it a bit to our use case :


/usr/local/bin/uwsgi -s 127.0.0.1:42597 -t 10 -M -p 1 -C -w example.example_uwsgi -H /opt/www/django-cms_tutorial/ve/src/django-cms

to :


/usr/local/bin/uwsgi -s 127.0.0.1:46075 -t 10 -M -p 10 -C -w example.example_uwsgi -H /opt/www/django-cms_tutorial/ve/

The file called example_uwsgi looks like this :

import os
import django.core.handlers.wsgi
# Set the django settings and define the wsgi app
os.environ['DJANGO_SETTINGS_MODULE'] = 'example.settings'
application = django.core.handlers.wsgi.WSGIHandler()
# Mount the application to the url
applications = {'/uwsgi':application, }

Comparison: uwsgi vs flup

flup

Resource usage for concurency equal to 50 :

(ve)yml@yml-laptop:django-cms_tutorial$ ab -n 1000 -c 50 http://192.168.1.18:8080/flup/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /flup/
Document Length: 3485 bytes

Concurrency Level: 50
Time taken for tests: 82.385 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 3688000 bytes
HTML transferred: 3485000 bytes
Requests per second: 12.14 [#/sec] (mean)
Time per request: 4119.256 [ms] (mean)
Time per request: 82.385 [ms] (mean, across all concurrent requests)
Transfer rate: 43.72 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 3
Processing: 750 4042 1424.0 3791 11665
Waiting: 750 4041 1424.0 3791 11665
Total: 753 4042 1424.2 3791 11666

Percentage of the requests served within a certain time (ms)
50% 3791
66% 4098
75% 4350
80% 4509
90% 4972
95% 5566
98% 10388
99% 10974
100% 11666 (longest request)

(ve)yml@yml-laptop:django-cms_tutorial$ ab -n 1000 -c 100 http://192.168.1.18:8080/flup/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /flup/
Document Length: 3485 bytes

Concurrency Level: 100
Time taken for tests: 82.614 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 3688000 bytes
HTML transferred: 3485000 bytes
Requests per second: 12.10 [#/sec] (mean)
Time per request: 8261.424 [ms] (mean)
Time per request: 82.614 [ms] (mean, across all concurrent requests)
Transfer rate: 43.59 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.8 0 3
Processing: 1209 7920 1519.3 7694 14118
Waiting: 1209 7920 1518.1 7694 14118
Total: 1213 7920 1519.5 7694 14120

Percentage of the requests served within a certain time (ms)
50% 7694
66% 8161
75% 8405
80% 8574
90% 9215
95% 10456
98% 13048
99% 13983
100% 14120 (longest request)

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /flup/
Document Length: 3485 bytes

Concurrency Level: 100
Time taken for tests: 81.737 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 3688000 bytes
HTML transferred: 3485000 bytes
Requests per second: 12.23 [#/sec] (mean)
Time per request: 8173.722 [ms] (mean)
Time per request: 81.737 [ms] (mean, across all concurrent requests)
Transfer rate: 44.06 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.2 0 5
Processing: 830 7843 1455.0 7574 14385
Waiting: 830 7843 1455.0 7574 14385
Total: 835 7843 1455.3 7574 14388

Percentage of the requests served within a certain time (ms)
50% 7574
66% 7912
75% 8137
80% 8335
90% 8893
95% 10624
98% 13435
99% 13670
100% 14388 (longest request)

Resource usage for concurency equal to 100 :

uwsgi

uWSGI resource usage for concurrency equal to 50 :

(ve)yml@yml-laptop:django-cms_tutorial$ ab -n 1000 -c 50 http://192.168.1.18:8080/uwsgi/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /uwsgi/
Document Length: 3494 bytes

Concurrency Level: 50
Time taken for tests: 84.013 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 3697000 bytes
HTML transferred: 3494000 bytes
Requests per second: 11.90 [#/sec] (mean)
Time per request: 4200.629 [ms] (mean)
Time per request: 84.013 [ms] (mean, across all concurrent requests)
Transfer rate: 42.97 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.3 0 2
Processing: 682 4115 472.7 4189 4707
Waiting: 682 4115 472.7 4189 4707
Total: 684 4115 472.5 4189 4707

Percentage of the requests served within a certain time (ms)
50% 4189
66% 4246
75% 4286
80% 4316
90% 4378
95% 4440
98% 4515
99% 4544
100% 4707 (longest request)
(ve)yml@yml-laptop:django-cms_tutorial$ ab -n 1000 -c 100 http://192.168.1.18:8080/uwsgi/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
apr_socket_recv: Connection reset by peer (104)
Total of 839 requests completed

This reveals the conservative nature of uwsgi. I had a discussion about this with Roberto De Ioris. He mades a detailled answer detailled answer explaining the situation. So the bottom line is that we will need to modify our configuration and increase the socket timeout "-z" and the socket listen queue "-l"

/usr/local/bin/uwsgi -i -l 120 -z 60 -p 10 -M -s 127.0.0.1:46075 -w example.example_uwsgi -H /opt/www/django-cms_tutorial/ve/

(ve)yml@yml-laptop:django-cms_tutorial$ ab -n 1000 -c 100 http://192.168.1.18:8080/uwsgi/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /uwsgi/
Document Length: 3494 bytes

Concurrency Level: 100
Time taken for tests: 86.246 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 3697000 bytes
HTML transferred: 3494000 bytes
Requests per second: 11.59 [#/sec] (mean)
Time per request: 8624.594 [ms] (mean)
Time per request: 86.246 [ms] (mean, across all concurrent requests)
Transfer rate: 41.86 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.7 0 7
Processing: 762 8248 1460.7 8509 9706
Waiting: 762 8248 1460.7 8509 9706
Total: 766 8248 1459.5 8509 9706

Percentage of the requests served within a certain time (ms)
50% 8509
66% 8669
75% 8852
80% 8998
90% 9182
95% 9360
98% 9474
99% 9566
100% 9706 (longest request)

uWSGI resource usage for concurrency equal to 100 :

Conclusion

flup is slightly faster than uWSGI at this point but this has to be put into perspective and you need to take into consideration the features that come with uWSGI, an exhaustive list can be found here. uWSGI is unable to complete the ab test with the following argument -n 1000 -c 100 with its default settings you will need to adjust the timeout socket and socket listen queue. However it is interesting to note that the memory footprint for uwsgi is lower by an order of magnitude and that my laptop remains responsive during the ab test using uWSGI where it was almost taken down by same test with flup.

[1] http://www.cherokee-project.com/doc/cookbook_django.html
[2] http://www.cherokee-project.com/doc/cookbook_uwsgi.html
[3] http://projects.unbit.it/uwsgi/

Add bpython to django's shell management command

2009-12-28T22:30:00.001-04:00

Like many of us I am spending a lot of time in the REPL loop to develop django code and this bug in readline cost me an extra backspace each time I hit "tab" in ipython.

This blog post capture my attention so I have decided to give bpython a try and so far I have been impressed by what I have seen. Here it is a management command that adds the support for bpython to django's shell management command.

import os
from django.core.management.base import NoArgsCommand
from optparse import make_option
from IPython.Shell import IPShell

def start_plain_shell():
    import code
    # Set up a dictionary to serve as the environment for the shell, so
    # that tab completion works on objects that are imported at runtime.
    # See ticket 5082.
    imported_objects = {}
    try: # Try activating rlcompleter, because it's handy.
        import readline
    except ImportError:
        pass
    else:
        # We don't have to wrap the following import in a 'try', because
        # we already know 'readline' was imported successfully.
        import rlcompleter
        readline.set_completer(rlcompleter.Completer(imported_objects).complete)
        readline.parse_and_bind("tab:complete")

    # We want to honor both $PYTHONSTARTUP and .pythonrc.py, so follow system
    # conventions and get $PYTHONSTARTUP first then import user.
    if not use_plain: 
        pythonrc = os.environ.get("PYTHONSTARTUP") 
        if pythonrc and os.path.isfile(pythonrc): 
            try: 
                execfile(pythonrc) 
            except NameError: 
                pass
        # This will import .pythonrc.py as a side-effect
        import user
    code.interact(local=imported_objects)

def start_ipython_shell():
    import IPython
    # Explicitly pass an empty list as arguments, because otherwise IPython
    # would use sys.argv from this script.
    shell = IPython.Shell.IPShell(argv=[])
    shell.mainloop()
    
def start_bpython_shell():
    from bpython import cli
    cli.main(args=[])
    

class Command(NoArgsCommand):
    option_list = NoArgsCommand.option_list + (
        make_option('--plain', action='store_true', dest='plain',
            help='Tells Django to use plain Python, not IPython.'),
        make_option('--ipython', action='store_true', dest='ipython',
            help='Tells Django to use ipython.'),
        make_option('--bpython', action='store_true', dest='bpython',
            help='Tells Django to use bpython.'),
    )
    help = "Runs a Python interactive interpreter. Tries to use IPython, if it's available."

    requires_model_validation = False

    def handle_noargs(self, **options):
        # XXX: (Temporary) workaround for ticket #1796: force early loading of all
        # models from installed apps.
        from django.db.models.loading import get_models
        loaded_models = get_models()

        use_plain = options.get('plain', False)
        use_ipython = options.get('ipython', False)
        use_bpython = options.get('bpython', False)
        
        try:
            if use_plain:
                # Don't bother loading IPython, because the user wants plain Python.
                raise ImportError
            elif use_ipython:
                start_ipython_shell()
            elif use_bpython:
                start_bpython_shell()
            else:
                # backward compatible behavior.
                start_ipython_shell()

        except ImportError:
            # fallback to plain shell if we encounter an ImportError
            start_plain_shell()

Is there an exiting debugger that use bpython ? I am looking for something equivalent to ipdb but for bpython. I am looking for something that could replace :


import ipdb; ipdb.set_trace()

Paginated feed for cmsplugin_feed

2009-12-14T18:00:00.000-04:00

In my previous post [1] I wrote about my experience of writing a new plugins for django-cms [2]. This plugin gives you the capability to add a feed to a django-cms' Page.

I ended my post by asking about the best practice used to paginate the content of a plugin. Since I didn't get flooded by the answers I assumed that this shouldn't be different than doing it on a django app. Here it is what the fine django documentation says about this topic.

So I have decided to implement this approach, the key point here is to understand that django-cms' plugin has access to the context :

class FeedPlugin(CMSPluginBase):
    [...]
    def render(self, context, instance, placeholder):
        feed = get_cached_feed(instance)
        if instance.paginate_by:
            is_paginated =True
            request = context['request']
        [...]

This capability lets your plugin react to the GET and POST parameters. Once I have understood this the only thing that I had to do is to use the django's Paginator on the list of feed entries.

As you can see in the code below there is nothing specific to django-cms there.


            is_paginated =True
            request = context['request']
            feed_page_param = "feed_%s_page" %str(instance.id)

            feed_paginator = Paginator(feed["entries"], instance.paginate_by) 
            # Make sure page request is an int. If not, deliver first page.
            try:
                page = int(request.GET.get(feed_page_param, '1'))
            except ValueError:
                page = 1
            # If page request (9999) is out of range, deliver last page of results.
            try:
                entries = feed_paginator.page(page)
            except (EmptyPage, InvalidPage):
                entries = feed_paginator.page(paginator.num_pages)

The complete implementation of this feature is available in the bitbucket repository of cmsplugin_feed.

I would be glad to hear from you if this implementation could be enhanced.

[1] http://yml-blog.blogspot.com/2009/12/feed-extension-for-django-cms.html
[2] http://www.django-cms.org/
[3] http://docs.djangoproject.com/en/dev/topics/pagination/
[4] http://bitbucket.org/yml/cmsplugin-feed/changeset/7d0644c7668f/

feed extension for django-cms

2009-12-12T16:33:00.001-04:00

I have been lately looking at the open source CMS alternatives based on django. After some times investigating I have decided to give django-cms [1] a try.

django-cms provides an API to write plugin it is well documented [2] and you can also look the implementation of plugins listed there [3]. I have decided to experiment with it by writing a plugin that displays the content of a feed into a page. The source code is available on my bitbucket account in a project called cmsplugin-feed. The development of this plugin has been relatively straight forward and once you get to know the conventions defined to implement an extension. You feel like you are writing python code for a django application. The CMS part does not stand on your way at least it didn't in my experimentation.

Here it is 2 screenshots showing this plugin in action :
In order to add a feed you need give a name and an URL to your feed

Then the plugin displays the field on a page.

I would be interested to read from someone the "best practice" to paginate the content of a plugin. Ideally I would like to add a parameter for every feed indicating how many items I want per page. Then I would like to paginate the feed in the placeholder of every page and this will allow the user to navigate between the pages.

[1] http://www.django-cms.org
[2] http://www.django-cms.org/en/documentation/2.0/custom_plugins/
[3] http://www.django-cms.org/en/extensions/
[4] http://bitbucket.org/yml/cmsplugin-feed/

uWSGI reaches the 0.9.3 Milestone

2009-11-30T10:30:00.000-04:00

uWSGI project [1] reaches the 0.9.3 milestone. You can review the complete announcement [2] on the mailing list and download the code here [3]

This new release brings some new exiting features :

- Nginx 0.7.x module
- configuration via python module
- support (non-standard) for Python 3.x
- Twisted client resource adapter
- graceful restart of worker processes and hot-plug substitution/upgrade
of the uWSGI server
- shared memory area to share data between workers/processes
- Tomcat handler
- support for virtualenv

In this post [4] I have covered the usage of both the "configuration via a python module" and the "virtualenv support".

uWSGI allows you to run your favorite wsgi application on top of prefered web server (apache, cherokee or nginx).

[1] http://projects.unbit.it/uwsgi/
[2] http://lists.unbit.it/pipermail/uwsgi/2009-November/000020.html
[3] http://projects.unbit.it/downloads/uwsgi-0.9.3.tar.gz
[4] http://yml-blog.blogspot.com/2009/11/setting-up-django-project-with-cherokee.html

Continous sphinx build

2009-11-17T20:50:00.000-04:00

I have always found the cycle : Edit the documentation source => build the sphinx based documentation [1] => open a browser pointing the updated documentation and re iterate until your are satisfied a bit painful.

Today I have finally found an automated version of this workflow that
gave me one of this nice WAHOUUUU feeling that happens after you remove this little stone from your shoe.

The idea is to continuously build the documentation using watch [2]


watch -n 2 make htlm

This will update the generated documentation every 2 seconds. This is nice in itself and it leads me to discover that Epiphany automatically reload the page when it changes.

[1] http://sphinx.pocoo.org/
[2] http://linux.die.net/man/1/watch
[3] http://projects.gnome.org/epiphany/

Setting up a django project with Cherokee, uWSGI and vitualenv (continued)

2009-11-12T22:35:00.001-04:00

In my previous post [1] I have started to talk about this stack but I have avoided to mention that the django project was indeed encapsulated into a virtualenv. This brought a couple small quirks and questions to the surface. So I spent some times investigating and exchanging with the community behind uWSGI and cherokee in order to solve them. The enhancements I discuss here require that you use the tip of uWSGI [2] because some of them are new capability added in the past days or small improvements to my recipe of using these products.

Before I start to dive into the enhancements let me sum up what we had to do in order to serve our WSGI project, in our case a django project.

1> I had to add 2 configurations files : uwsgi.conf, django_wsgi.py
The first one is an XML file where you define the PYTHONPATH and the mount point for your application, the second one is a python module where you define your WSGI configuration.

2> I had to use the cherokee's wizard to create the appropriate configuration to serve my django app using uWSGI. This step add an "Information Sources" with the following settings in the interpreter section :


/usr/local/bin/uwsgi -s /tmp/cherokee-source1.sock -C -x /opt/webapps/example-ve/src/project/uwsgi.conf

The first thing that can be improved in this recipe is that you can reduce the configuration to only one python module. This allows you to avoid the uwsgi.conf file. uWSGI comes with an option that allows you to directly specify a python module.


-w     name of wsgi config module (no ROCK_SOLID)

This would give us the following command :


uwsgi -s /tmp/cherokee-source1.sock -C -w project.django_wsgi

The django_wsgi file is very simple and looks like this :


import os
import django.core.handlers.wsgi

# Set the django settings and define the wsgi app
os.environ['DJANGO_SETTINGS_MODULE'] = 'project.settings'
application = django.core.handlers.wsgi.WSGIHandler()

# Mount the application to the url
applications = {'/':'application', }

This is all good and simple but it only works if "project" is in the PYHTONPATH. I handle all my projects in a dedicated virtualenv. uWSGI handle this gracefully now, there is a new option that allows you to define your Python Home.


-H     set python home/virtualenv

Here it the improved settings for the interpreter section of the information sources :


uwsgi -s /tmp/cherokee-source1.sock -C -w project.django_wsgi -H /opt/webapps/example-ve

I would be glad to hear from you and very happy if you give this setup a spin on your django project and let me know how it performs compare to your actual setup.

[1] http://yml-blog.blogspot.com/2009/11/how-to-set-up-cherokee-and-uwsgi.html
[2] http://projects.unbit.it/uwsgi/changeset/87%3Ae78ef65c833e

How to set up cherokee and uWSGI

2009-11-03T21:11:00.003-04:00

A couple of post ago I have "blogged" about cherokee which is a light weight web server, fast and extremely simple to configure thanks to its great admin interface.

The most common configuration to use cherokee with django is described here and to put is in a nutshell it was based on flup using SCGI or FASTCGI protocol.

However a couple of weeks ago a new contender pop up on my radar. It is called uWSGI and it might become in the near future very popular to power django application. You can find more information about it on on the pages below :
* example [2]
* Documentation [3]

From an architecture stand point it might provide us python web developer an interesting piece of software that could operate with virtually any webserver. There is already an apache module and cherokee handler. The documentation mentions that the core developers have started to develop an nginx adaptor.

Software prerequisites

cherokee 0.99.26
uWSGI (changeset: 85:61dcfd718023)

I have installed cherokee using the PPA on launchpad [1] and uWSGI directly from the mercurial repository.


hg clone http://projects.unbit.it/hg/uwsgi

Installation

Installation of both product is relatively straight forward and well documented. I will not detail it more here but rather point you to cherokee [4] and uWSGI website [5].
Here it is how you would compile uWSGI on Ubuntu 9.04


make -f Makefile.Linux.Py26

Note: The compilation will create a binairy called uwsgi26. It is important to add it into your PATH under the name of uwsgi. If you don't do this cherokee's wizard will complain about not being able to find uWSGI server. I have created a symbolic link but you could also copy it into /usr/local/bin.

Configuration

The amazing admin interface of cherokee make it dead simple once you know what need to be done. In the latest version of cherokee a wizzard has been added to make the configuration even simpler.

First thing first you need to create 2 configurations file into your django project directory called respectively : django_wsgi.py and uwsgi.conf

------ django_wsgi.py ------------------


  import os import django.core.handlers.wsgi  
  os.environ['DJANGO_SETTINGS_MODULE'] = 'project.settings' 
  application = django.core.handlers.wsgi.WSGIHandler()

------ uwsgi.conf --------------------


<uwsgi>

    <pythonpath>/home/yml/workdir/webdev/fabric_factory/20091023/fabric_factory/ve/lib/python2.6/site-packages/</pythonpath>

    <pythonpath>/home/yml/workdir/webdev/fabric_factory/20091023/fabric_factory/src/</pythonpath>

    <app mountpoint="/">

        <script>project.django_wsgi</script>

    </app>

</uwsgi>

Once this is done you can launch cherokee admin : 'cherokee-admin -u' and point your browser to this URL :
* http://127.0.0.1:9090/

Once in cherokee you can use the uWSGI wizard to create automatically your information sources and add a new rule. In order to do so you need to go into the virtual server section on the behavior tab [6]. Below the table there is an hyperlink called wizards. It will take you to a list of wizards, click on uWSGI.

Cherokee asks for the path to the uwsgi.conf file that we have created earlier. The last bit you might want to do is to add a rule to serve the admin media.

Troubleshooting

Here it is some basic technics to understand what is going on and solve your issues. Look at cherokee's log files :


sudo tail -f /var/log/cherokee/cherokee.error
and
sudo tail -f /var/log/cherokee/cherokee.access

Look into the section called "information sources" in cherokee-admin. Explore the information sources created by the uWSGI wizard. Copy the text in the interpreter field, in my case it is :


/usr/local/bin/uwsgi -s /tmp/cherokee-source1.sock -C -x /[Path to my django project]/uwsgi.conf

Then you can paste it in a terminal this will give you a way to see what it is going on.

This new combination is still very young but it looks promising to me. I am sure that if we get enough people to hammer on it can quickly becomes popular.

[1] https://launchpad.net/~cherokee-webserver/+archive/ppa
[2] http://projects.unbit.it/uwsgi/wiki/Example
[3] http://projects.unbit.it/uwsgi/wiki/Doc
[4] http://www.cherokee-project.com/doc/basics_installation.html
[5] http://projects.unbit.it/uwsgi/wiki/Install
[6] http://127.0.0.1:9090/vserver/10

django-piston authentication against django auth -- part 4

2009-10-22T20:12:00.001-04:00

The previous post ends with one known "Open Issue". The authentication for the ajax call was sitll hard coded [1]. In this post I am going to show you how you can extend django-piston to authenticate your users against django.contrib.auth. This might be important if your web app already take advantage of this module to manage and authenticate users.

A bit of reading of the source code and the documentation of django-piston [2] leads me to understand that django-piston has been designed from the ground to enable you to easily write your own authentication handler. Out of the box it comes with "HttpBasicAuthentication" and OAuthAuthentication. "HttpBasicAuthentication" is a very good example how to implement a DjangoAuthentication.

An authentication handler is a class, which must have 2 methods: is_authenticated, challenge


class DjangoAuthentication(object):
    """
    Django authentication. 
    """
    def __init__(self, login_url=None, redirect_field_name=REDIRECT_FIELD_NAME):
        if not login_url:
            login_url = settings.LOGIN_URL
        self.login_url = login_url
        self.redirect_field_name = redirect_field_name
        self.request = None
    
    def is_authenticated(self, request):
        """
        This method call the `is_authenticated` method of django
        User in django.contrib.auth.models.
        
        `is_authenticated`: Will be called when checking for
        authentication. It returns True if the user is authenticated
        False otherwise.
        """
        self.request = request
        return request.user.is_authenticated()
        
    def challenge(self):
        """
        `challenge`: In cases where `is_authenticated` returns
        False, the result of this method will be returned.
        This will usually be a `HttpResponse` object with
        some kind of challenge headers and 401 code on it.
        """
        path = urlquote(self.request.get_full_path())
        tup = self.login_url, self.redirect_field_name, path 
        return HttpResponseRedirect('%s?%s=%s' %tup)

The implementation is relatively simple, is_authenticated uses the method with the same name available from a django User and the challenge redirect the user to the login page.

This post will end the series on django-piston it was for me a pretext to get familliar with this great app that make creating a RESTFUL api for django easy. You can find all the modifications I did into my fork on bitbucket [3]

[1] http://bitbucket.org/yml/django-piston/src/a8bcb7f9756e/examples/blogserver/templates/edit_ajaxy_post.html#cl-6
[2] http://bitbucket.org/jespern/django-piston/wiki/Documentation#authentication
[3] http://bitbucket.org/yml/django-piston/

django-piston form validation -- part 3

2009-10-18T23:50:00.003-04:00

In my previous post titled "Exploration of django-piston -- part 2" [1] I end the post by 2 open issues. This post will propose a solution to the first one:

The validation errors are returned as a pseudo xml string.


Bad Request <ul class="errorlist"><li>content<ul class="errorlist"><li>This field is required.</li></ul></li><li>title<ul class="errorlist"><li>This field is required.</li></ul></li></ul>

To put it in other words I would prefer if django-piston return the validation errors of the form in the format specified in the request.

JSON


$ curl -u testuser:foobar -X POST -d content="This post is created using the api" http://127.0.0.1:8000/api/posts/?format=json
Bad Request {"title": ["This field is required."]}

yaml


$ curl -u testuser:foobar -X POST -d content="This post is created using the api" http://127.0.0.1:8000/api/posts/?format=yaml
Bad Request title: [!!python/unicode 'This field is required.']

xml


$ curl -u testuser:foobar -X POST -d content="This post is created using the api" http://127.0.0.1:8000/api/posts/?format=xml
Bad Request <ul class="errorlist"><li>title<ul class="errorlist"><li>This field is required.</li></ul></li></ul>

After a while reading the source of django-piston and poking around I have implemented this feature in my branch.

The biggest hurdle was that for some reasons "form.errors" cannot be serialized directly.


simplejson.dumps(form.errors)
*** TypeError:  is not JSON serializable

In order to work around this issue you need to force python to evaluate this proxy object.


dict((key, [unicode(v) for v in values]) for key,values in form.errors.items())

The last bit is to change the JS in order to adapt it to the fact that now the validation errors are returned as a json string.


    response = response.substring(12, response.length);
    errors = eval("("+response+")");

    $.each($(':input'), function(index, value) {
        field =$(value).attr('id');
        field = field.substring(3, field.length);
        field_errors = errors[field];
        if (field_errors) {
            ul = "<ul class=\"errorlist\">\n";
            $.each(field_errors, function(index, value){
              ul += "<li>"+value+"</li>\n";
            });
            ul += "</ul>\n";
            $(value).parent().prepend(ul);
        };
    });

The complete code related to the implementation of this feature is available in my branch [3].

Open Issue

Authentication is still hardcoded into the html pages. What is the best way to keep HttpBasicAuthentication to the external api that could easily be used by curl and add single sign on with django admin interface ?

The example is still very incomplete. It is missing among other things the capability to update/delete a blog post

I would be glad to read from you on how to improve this example or to get pointer of reusable application that have been built on top of django-piston.

[1] http://yml-blog.blogspot.com/2009/10/exploration-of-django-piston-part-2.html
[2] http://bitbucket.org/yml/django-piston/changeset/a8bcb7f9756e/
[3] http://bitbucket.org/yml/django-piston/

exploration of django-piston -- part 2

2009-10-14T20:55:00.002-04:00

Djangocon 09 was really food for thought and this post follow my previous post about django-piston. I found the talk from Avi Bryant about his experience building Trendly [1].

My take away of this talk was that they have a single HTML page with a lot of JS. The JS is the used to get the JSON data the JSON from the server. Unfortunately I am unable to find a link to a better description of its talk.

My last post hopefully convince you that it is easy relatively easy to create a web api to your django application [2]. This Post aims to show you how you can create a page to list and create a blogpost.

List and create a blogpost

The idea here is to reuse the web api that we have created in the previous article to build a web page. One of the interesting feature of django-piston is the decorator @validate(BlogpostForm, 'PUT'). It enables you to validate the data send to your server and it takes as input a django form. A longuer description of this feature is available here [4].

The view cannot be simpler [5] :


def create_ajaxy_post(request):
    form = BlogpostForm()
    return direct_to_template(request,
                              template='edit_ajaxy_post.html',
                              extra_context={'form':form} )

The template is using jQuery [6] which ease the ajax call and the DOM manipulation. The template [7] has a very simple structure :


{% block content %}

<h1>Ajaxy</h1>

<div id="content"></div>

<h2>Ajaxy form</h1>

<form method="POST" action="{% url posts %}" id="post_form">

    <div id="post_form_error"></div>

    {{ form.as_p }}

    <input type='submit' name='create' onclick="return send_form();" value='Create'></input>

</form>

{% endblock %}

In order to give life to this page you need some JS which will get from the JSON string of the blogposts and POST the DATA to create a new blogpost and add it to the list.

If you try to POST an empty blogpost the data will be validated on the server and the validation errors will be displayed to the user :


* content
  o This field is required.
* title
  o This field is required.

Open Issue

The validation error are returned as a pseudo xml string :


Bad Request <ul class="errorlist"><li>content<ul class="errorlist"><li>This field is required.</li></ul></li><li>title<ul class="errorlist"><li>This field is required.</li></ul></li></ul>

It will be much more convenient to get a json dict with the error. Any one has an idea on how to get this ?

Authentication is still hardcoded into the html pages. What is the best way to keep HttpBasicAuthentication to the external api that could easily be used by curl and add single sign on wiht the admin ?

The example is still very incomplete. It is missing among other things the capability to update/delete a blog post

I would be glad to read from you on how to improve this example or to get pointer of reusable application that have been built on top of django-piston.

[1] http://trendly.com/
[2] http://bitbucket.org/yml/django-piston/src/tip/examples/blogserver/README.txt
[3] http://bitbucket.org/yml/django-piston/src/72c72e0b4b7e/examples/blogserver/templates/edit_ajaxy_post.html
[4] http://bitbucket.org/jespern/django-piston/wiki/Documentation#form-validation
[5] http://bitbucket.org/yml/django-piston/src/tip/examples/blogserver/blog/views.py
[6] http://jquery.com/
[7] http://bitbucket.org/yml/django-piston/src/72c72e0b4b7e/examples/blogserver/templates/edit_ajaxy_post.html

Add a web api to your app with django-piston

2009-09-21T17:04:00.002-04:00

More and more often I found myself in a situation where I would like to add a web api to my django applications. I have recently tried to used django-piston http://bitbucket.org/jespern/django-piston for this and I found the learning curve a bit steep. It is not particularly hard but it requires you to understand few things before being able to enjoy it.

This post should help you to understand how to create the handlers to read, create, update, delete an object and see how you can call this web api from the command line using curl. I have forked django-piston to extend the example "blogserver". I would recommend you to also read the README inside the example

You can grab the code like this :



hg clone http://bitbucket.org/yml/django-piston/

curl is a command line tool to transfer data from or to a server, using one of the supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE). The command is designed to work without user interaction.

Here it is the 4 things that I found a bit hard to understand.

1. @require_extended

This decorator is handy if you want to restrict the access to your handler to only the request that have have one of the header listed below :
• application/json
• application/x-yaml
• text/xml
• application/python-pickle

The direct effect on the curl command line is that you will need to add a -H 'Content-Type:', this will give you something like :



$ curl -u testuser:foobar -H 'Content-Type:application/json' http://127.0.0.1:8000/api/posts/?format=json

If the header is omitted django-piston will return a 'Bad Request'.

2. Request Method

The handler for your resource can be composed of the following method : read, create, update, delete that are respectively mapped to the following request method : GET, POST, PUT, DELETE. "-X " is used to specify the method you want to use. Here it is an example that execute the update method of the blogserver :



$ curl -u testuser:foobar -H 'Content-Type:application/json' -X PUT -d '{"content": "Update test", "title": "Update test"}' http://127.0.0.1:8000/api/post/1/

3. Passing data to your handler

In this example I am going to demonstrate how to pass JSON string with curl. In order to do this you should use the -d followed by the JSON string :



$ curl -u testuser:foobar -H 'Content-Type:application/json' -X PUT -d '{"content": "Update test", "title": "Update test"}' http://127.0.0.1:8000/api/post/1/

django-piston will automatically turn this string into a python dictionary that is ready to be used by your handler. You will find this data in 'request.data' the raw string is available in "request.raw_post_data"

4. How to pass parameter to your handler

It took me a while to understand that the same handler can be mounted to several urls this will allow you to pass additional parameters like the primary key ID or the slug :



....

blogposts = Resource(handler=BlogpostHandler, authentication=auth)

urlpatterns = patterns('',

url(r'^posts/$', blogposts),

url(r'^post/(?P.+)/$', blogposts),

.....

In the example above "blockposts" is mapped to 2 different urls, the second one will be particularly handy to read, update, delete a particular post. Additional parameters from the URL will be passed to the method : read, create, update, delete. I encourage you to check out the code from bitbucket to see how you could take advantage of this technique.

This post barely scratch the surface of how to use django-piston, to put it in a nutshell this reusable application makes creating a web api for your own project simple. It avoids you to write a lot of boilerplate code because it abstracts all the machinery and let you focus on important things.

I would be glad to hear from you how you use it and what is your favorite trick.

Fabric factory

2009-09-13T22:04:00.001-04:00

This is a project I have been working on recently after I spent a day to look at the existing solution to run periodically a test suite. Most of the project I look at were either difficult to setup or require to learn yet another specific domain specific language or had dependency on a larger software stack.

As a reaction to this situation I have decided to see if I could write something simple that achieves gracefully this task. I also try to make it as easy to setup as possible.

I have decided to use cpython as platform, django as web framework for the server and Fabric as library to automate the task execution.

The result of this mix can be found on bitbucket in a project called Fabric Factory. This will eventually become a complete Job Server that could be used to distribute any kind of task scripted in Fabric.

Installation

This assumes that python is installed on your computer and that you have an internet conection.

You can download the code using mercurial:
* hg clone http://bitbucket.org/yml/fabric_factory/
A fabfile will help you to quickly setup your environment.
* fab quickstart

Note : In order to run the command above you will need the latest version of Fabric the following command will take care of this:



pip install -e git://github.com/bitprophet/fabric.git#egg=Fabric

Usage

"quickstart" has created a virtualenv which must be actived before you continue.



. ve/bin/activate

Once the virtualenv is activated you can go inside "src/project". This is a django project so from there you can do several things :

* create an sqlite db : python manage.py syncdb
* run the server : python manage.py runserver
* run the test suite : python manage.py test

The main app of this django project called fabric factory is called "factory".

Once the server is started and that you have created some "Build" in django's admin interface you can open a new terminal and run the client side of the project:



cd src/worker

python run_worker.py --daemon=start

python run_worker.py --daemon=stop

Use case

Now that you have understood the layout of the project. Let us see how we can achieve something useful with it.

We are going to create a Build that will :
* download the Fabric Factory
* setup the environement
* run the test suite
* Report the result

1> Direct your browser to that url http://127.0.0.1:8000/admin/ and key in the username/password you have chosen for your administrator.
2> Add the fabfile recipe store in docs http://127.0.0.1:8000/admin/factory/fabfilerecipe/add/ and call it "fabric factory use case"
3> Replace example.com by 127.0.0.1:8000 in sites : http://127.0.0.1:8000/admin/sites/site/1/
4> Create a Build that will download setup and run the test here : http://127.0.0.1:8000/admin/factory/build/add/

The fabfile recipe that we have downloaded earlier contains a task called : 'download_setup_and_test' This task as been writen to do what we want.

We are now going to configure the client to run this task. However before doing this let us see how the server publish the tasks that need to be executed. Point your browser to this url : http://127.0.0.1:8000/factory/build/oldest_not_executed/

5> Open a new terminal and move into the the worker directory then you can start the worker in daemon mode :



. ve/bin/activate

cd src/worker

python run_worker.py --daemon=start

If you want to look at what is happening in the background you can watch at the log file in realtime



tail -f worker.log

6> You can see the status of the build here : http://127.0.0.1:8000/admin/factory/build/ Keep in mind this task is pretty long to run because we are downloading all the dependancies (django, fabric). It took me almost 5 minutes to execute this task and to see the result in the admin.

Conclusion

This project which is still very immature seems to prove that this stack is well suited to build this kind of tool. I would be glad to hear your experience about this kind of tool. Please do not hesitate to copy, fork, contribute to this project to make sure that soon we have a simple easy to setup yet flexible tool to distribute tasks.

sphinx autodoc and django app

2009-06-22T19:28:00.000-04:00

Today I have been getting my foot wet with autodoc extension from sphinx. Here it is what the documentation of sphinx say about it :

"""
Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license.
"""

I have to say that before today my experience with sphinx as been excellent. It is relatively straightforward to get started once you pass the first little annoyances that come with the fact that you are learning a new tool.

The module I have been using to conduct this experiment is django-geotagging this reusable app enables you to geotag any object in the database. If you want to know more about this you can read the sphinx based documentation. This is the goal of the documentation, isn't it ? :-)
If adding a manually written documentation is very easy and well documented. The modification required to move from a very primitive documentation written using some rst file to sphinx is shown here. Most of it is automatically generated by : sphinx-quickstart.

Taking it a step further have been more difficult than anticipated and this for several reasons : lack of example, and a bug
Today I wanted to add to the existing documentation in django-geotagging and API section. In order to do so I have spoted in the sphinx documentation an extension called autodoc. It seems to be exactly what I need, here it is an extract from its doc :

"""

This extension can import the modules you are documenting, and pull in documentation from docstrings in a semi-automatic way.

"""

The first modification I add to do in sphinx's conf.py was to setup the settings in my environment. This can be done by adding the following 3 lines to conf.py

"""
from geotagging_demo_project import settings
from django.core.management import setup_environ
setup_environ(settings)
"""

Once this is done you should just be able to create a file that will be used as placeholder to describe the documentation you want to extract. Let us take an example now, since I want to describe the API of "models.py" I am going to create a file called "model.rst". In this file I need to add the following lines :

"""
:mod:`models` -- geotag models
==========================================

.. automodule:: geotags.models
:members:
:show-inheritance:

.. autoclass:: Point
"""

I will let you read the documentation for each of this directive to understand what they are doing. This is were the bug come into play because once you have done this you should be able to enjoy the automatically extrated documentation the next time you build it. Instead of this I have observed this bug. To put it in a nutshell sphinx is complaining about the line 27 of
"/usr/lib/python2.6/django/contrib/gis/db/models/proxy.py"

I am not really sure where the bug is however it seems that doing a small modification there enable me to build the documentation.

I hope that this feedback about my experience will help you to get started with using sphinx and its autodoc extension. If you have an opinion about the django ticket #11353 I am very interested to hear it.

Serving Django via CherryPy behind cherokee

2009-02-27T13:31:00.001-04:00

Almost one year ago "Peter Baumgartner" wrote about Serving Django via cherrypy I am going to explain in this post how to take this approach one step further and to add load and balancing of the requests to several instances of cherrypy. This sounds like a lot of "*.conf" editing, isn't it ? In fact the nice thing about this approach is that the only file you will have to edit is your settings.py to add one line. To do so I am going to use cherokee mainly because it has a user friendly interface called cherokee-admin which provides a very easy way to configure your server. For this article I have used cherokee Version 0.98.1.

I will assume in this article that you have a working django project in a virtualenv. First you will need to install cherrypy and django-cpserver. You have several way to do this the easier is probably to use pip

pip install cherrypy
pip install -e git://github.com/lincolnloop/django-cpserver.git#egg=django-cpserver

Then you need to edit you settings.py to add "django_cpserver" in the list of your INSTALLED_APPS. This will give you a convenient django management command to start cherrypy server.
./manage.py runcpserver port=8089

Believe it or not this was the hardest part of the recipe from now to the end we will use a nice web interface. In order to launch cherokee-admin on ubuntu I use the following command :

sudo cherokee-admin

We need to define 2 remote sources in the admin interface :

127.0.0.1:8088 and 127.0.0.1:8088 are the addresses on which cherokee can contact the cherrypy instances. Several interested things to note here, the adresses can be spread on several computer and several ports.

Then we need to define a new target "/django" (alias) that will load and balance the requests to cherrypy instances.

Then for this target we need to set the handler to "HTTP reverse proxy".

It is time to use the remote sources we have defined earlier.

The last bit is to rewrite the url before passing it to the cherrypy instances

This is the end of the recipe you can now save the modification and restart the cherokee. I would be glad to read from you the enhancements that could be added to this recipe.

django full text search with solango

2009-02-21T13:49:00.000-04:00

2 months ago I wrote a post titled "django fulltext search part-1" this post was explaining how to take advantage of djangosearch to interface between django and solr.

The big advantage of djangosearch is the fact that it comes with a plugable backend architecture. This can be a strength since in theory it enables you to abstract the details of the the full text engine you are using however in practice I ended up writing a patch to bypath the abstraction layer because it was preventing me of doing the query I wanted. So to make a long story short djangosearch was not working out of the box for my needs.

However 2 week ago Sean Creeley released solango and this significantly changed landscape of full text search in the django eco-system. I will not go in the details about solango in this post. It comes with some management commands that are so convenient that I still wonder why I haven't thought at implementing them on top of djangosearch. I will copy below a short extract form the documentation which is excellent :

./manage.py solr --help
#solango schema options
--fields              Prints out the fields the schema.xml will create
--flush               Will remove the data directory from Solr.
--reindex             Will reindex Solr from the registry.
--schema              Will create the schema.xml in SOLR_SCHEMA_PATH or in the --path.
--start               Start solr running java -jar start.jar
--path=SCHEMA_PATH    Tells Solango where to create config file

In the rest of the post I am going to assume that you have installed and configured solango. When I have done this I have not seen any major obstacle. Once again Sean has done an excellent job at documenting this project. one thing that annoys me while implementing solango in one of the project I am working on is the fact that you cannot restrict your search in what is often call an "advanced search". The good news is that solango has been recently improved to be easily extended. The only thing you will have to do is to defined a django form to represent your advanced search and to add an url that use it. This can literally be done in less than 30 lines of code including the comments and the imports.

Let us start by the writing the forms.py :


from django import forms
from solango.solr import get_model_from_key
import solango

def model_choices():
    """
    Return a list of tuple with all the models that have been indexed.
    This tuple is used in the AdvancedSearchForm to selects the models
    you want to search your term in.
    """
    models = [(model_key, get_model_from_key(model_key)._meta.verbose_name_plural)
             for model_key in solango.registry.keys()]
    return models

class AdvancedSearchForm(forms.Form):
    """
    Form that represents an advanced search
    """
    q = forms.CharField(required=False)
    model = forms.MultipleChoiceField(choices=model_choices(), required=False,
        widget=forms.CheckboxSelectMultiple)
    def clean_q(self):
        q = self.cleaned_data.get("q")
        if q == '':
            raise forms.ValidationError("You cannot query for an empty string")
        return q

Nothing really complex there we have defined a form with a charfield named "q" that will be used to enter your search terms and a set of check boxes displaying all the models indexed. The user will be able to select one or several models to restrict its query.
Now that the form is defined we are going to use it. In order to do so you need to add the following url somewhere in your project.


from django.conf.urls.defaults import *

from solango.views import select
from populous.search.forms import AdvancedSearchForm

urlpatterns = patterns('',
            url(r'^advanced/$', select,
                {
                    "form_class":AdvancedSearchForm,
                    "template_name":None,
                },name="search-advanced"),
)

You will recognize there the same pattern used in the generic views. You can customize the solango's view, called "select", by passing a "form_class" and a "template_name". In this example I have not over loaded the template_name thus django will used the solango default template.

I would be glad to read from you the customized form you have built on top of solango. Please do not hesitate to post them as comment.

Updated the 22th Feb 2009 : Correct the name of Sean Creeley, sorry for that.

Enable distutils for a django reusable app

2009-01-31T21:41:00.000-04:00

As you might already be aware Pinax move to distutils, this change has been described by James Tauber in this post. Today I have decided to make my feet wet with this approach for a django reusable app. I have been guided by jezdez on #pinax.
Before starting I would recommend you to read this page, yes I know, it is a bit long but very interesting. after this reading you will be well prepared to start to work on you django reusable app.

I have used one of my project django-geotagging to experiment with this approach. The project can be found on launchpad there. The good news about this approach is that all the major rcs are supported : SVN, BZR, HG, ... If your favorite versionning system is missing there is a good chance that I just forget to mention it. This list is not exhaustive.

The core of this approach is a file called setup.py that need to be paced at the root of your repository, most of its argument are self explanatory.

This file enable you to setup a complete env in 4 steps :

Create a virtual env : virtualenv geotagging_env
Move into that env : cd geotagging_env
Activate the env : . bin/activate
Install pip : easy_install pip
Use pip to grab django-geotagging : pip install -e bzr+http://bazaar.launchpad.net/~yml/django-geotagging/geotags/#egg=django-geotags

The coolness of this increase with the number of reusable apps you have to install in order to build your web project.

I would be glad to learn from you what kind of cool things can be done once this infrastructure is in place.

How to use the same widget as GeoDjango

2009-01-19T20:09:00.000-04:00

At the end of this post you will be able to use the same widget than the automatically generated admin interface in geodjango. I spent quite sometimes today to rediscover how to do this. Yes, rediscover because I have already written about this a couple of month ago. My first post on that topic can be read there. Happily "jbronn" on #geodjango gave the solution to me.


# Getting an instance so we can generate the map widget; also
# getting the geometry field for the model.
admin_instance = PointAdmin(Point, admin.site)
point_field = Point._meta.get_field('point')

# Generating the widget.
PointWidget = admin_instance.get_map_widget(point_field)

In fact all the complication at the moment there is no static widget that widget you could use in your own form. You have to build them dynamically.

I am now going to break down the 3 lines of code.
PointAdmin is the ModelAdmin class which is a representation of a model in the admin interface. Here it is an example :


from django.contrib.gis import admin
from geotagging.models import Point

class PointAdmin(admin.GeoModelAdmin):
list_filter = ('content_type','point' )
list_display = ('object', 'point', 'content_type', 'object_id')

Point in the model we are working on so Point._meta.get_field('point') is accessing the field called point of the Point mode. The code below should help you to understand :


class Point(models.Model):
  """
  """
  point = models.PointField(verbose_name=_("point"),srid=4326)
  content_type   = models.ForeignKey(ContentType,
          related_name="content_type_set_for_%(class)s")
  object_id      = models.CharField(_('object ID'),max_length=50)
  object = generic.GenericForeignKey(ct_field="content_type", fk_field="object_id")
  objects = models.GeoManager()

  def __unicode__(self):
      return 'Point for %s' % self.object

The last line is actually where the geodjango specificity is :
* PointWidget = admin_instance.get_map_widget(point_field)
get_map_widget is defined here

Now that we have a PointWidget we can use it in our form. Here is is a small example :


class PointForm(forms.ModelForm):
    point = forms.CharField(widget=PointWidget())
    class Meta:
        model = Point
        exclude = ("content_type","object_id")
    class Media:
        js = ("http://openlayers.org/api/2.6/OpenLayers.js",)

Now you can use geodjango super widgets in your forms.

Social Product Development Network

2008-12-13T21:10:00.000-04:00

I am a bit overwhelmed by the number of excelent PLM related blog posts. Today I am going to try to answer the question asked by Oleg : How to implement Social Networking for PLM ?

PLM has always been the concept of managing the information produce by a variety of Computer Aided Authoring tools. In a PLM system everything is organized around the BOM, so all this information is linked to Part and the Parts are organized in a tree.
PLM vendor love to claim that they are also capturing the communication thanks to their workflow system.
But the reality is that these workflows capture at best 2 or 3 very well formalized process : RFQ, ECN, ...

But everybody in the companies knows that the most important discussion about the product developement are done else where : by emails, over phone, meetings... This is even more true with the concept of global engineering team located in several countries. Some companies have a very aggressive objective in term offshore engineering and design, I know a couple of companies that are targeting 50 % of these activities in low cost countries.

I think that time has come to invent yet another acronym Social Product Development Network -- SPDN. This new marketing beast will offer :

an internal blog to every employee with the possibility to publish article on internet.
a wiki to capture the user generated documentation around the Product
An issue tracker to follow the issue related to the product development
A twitter clone to exchange short messages
gallery to share pictures of the product prototypes, tests , ...
BOM management
PDM
Workflow to control the 2 or 3 key processes

You are going to tell me that this is impossible because PLM vendor does not have the resources/skills to implement all this software stack. You are right but the good news is that you don't have to do it. All this and much more is already available out there in many flavors. For example I have been recently contributing and using such platform recently. It is called Pinax a video presenting the solution is available there. The only pieces that you will have to worry about are the core of the PLM (BOM, PDM, Workflow) and for each of them you will have only to focuse on implementing the vision on an leveraging a modern infrastructure.
No you don't have to develop yet another ORM it is there. No you don't have to invent yet another template language, or an url dispatcher, a cache stragetgy. You use all your ressources to focus on the implementation of the PLM vision rather than the technical details. This technology power web site that receive million hits a day. So I assume it will scale the PLM needs.

But all this requires a complete shift from selling software licences to sell a vision.

Conclusion :
If you really have a vision for PLM sell it with the service to implement it rather than the licences.

Django fulltext search -- part 1

2008-12-12T15:40:00.000-04:00

I have been recently working on adding full text search to a django based web application since this is not yet particularly well documented. Here it is my little stone to the amazing pyramid already in place waiting for you to use it. In order to add this "must have" feature to your great web application you will need 3 components :

Starting by the latest the first challenge is to get it installed, I haven't had luck with the usual "apt-get install solr". It must be me but after an installation I was only getting a blank page instead of solr admin interface. On #solr channel someone advise me to just grab the latest release tar ball and to run with it. Then I have performed the tutorial this help me a lot to get started with solr. Believe it or not when you are there you have done the hardest part of it.

Now you need to go inside the django project you want to work on and follow the following recipe :

download and put into your PYHTONPATH djangosearch and pysolr
Add djangosearch inside the INSTALLED_APPS tupple
add the following settings :

SEARCH_ENGINE = "solr" # solr, lucene, xapian, estraier
SEARCH_RESULTS_PER_PAGE = 10
SOLR_URL= "http://localhost:8983/solr"

At this point you are almost done the last operations you will need to do is to define inside models.py the fields you want to index with solr. In order to do this djangosearch come with a class called : ModelIndex
For example if you want to index : tease, and story fields of your Story model you will do something like this :


from djangosearch import ModelIndex
class Story(models.Model):
  ...
 tease = models.TextField(_('tease'), blank=True)
 story = models.TextField(_('story'), blank=True)
  ...
  index = ModelIndex(fields=["tease","story"])

Now you need to tell solr about this 2 new fields this is in fact easier that it seems to be, the theory is explained there. In practice what you need to do open this file (apache-solr-1.3.0/example/solr/conf/schema.xml) with your editor of choice and add this 2 lines inside the fields section:

You will now need to restart solr in order to take this modification into account.

The meal is now ready you can go and taste it.

Open a browser go to the admin interface and add a new story or update an existing one.
Open a django shell and enjoy your new full text search


from dajngosearch import search
search_results = search("foo")

I hope this will be useful to someone and will avoid you the pain I had to get all these pieces dancing together.
I would be glad to read from you what can be improve in this raw recipe and also what you are doing with your search and search_results.

Blindness of closed source software company

2008-12-08T08:55:00.000-04:00

Since yesterday I am using this blog in a slightly different way than usual. For some reasons it seems that something is happening in PLM realm.

This post is again a reaction to the post from Oleg Shilovitsky's. The post was titled :

Who will play role of Google Maps in PLM?

I think this is an excelent question. I am taking this opportunity to add some information on how this could happen. I have been working closely with both concept, recently I have been working hard to developed a Geographic Information System (GIS) platform to manage Geo localized content. I will make a post related to this platform as soon as it is ready for prime time.
Let us get back to the question, just a bit of history before google map GIS was complex, expensive, not web based software, closed source, understood and used by a very small subset of the population. Do you see any resemblance with PLM ? :-)
Google maps brings it to the mass.

I agree with you that none of the application or technology below deliver something close to what google maps bring.

Enterprise Mashups (application area emerged from ability to mix data coming from different sources and present it to user.
Microsoft Excel Services (high acceptance of Excel user interface make it usable everywhere)
Dassault Systemes 3DLive (very promising lifelike user experience comes last year in new DS products).

My explanation to this is that google maps was just the tip of the iceberg the cherry on the cake. The nice UI that you present to your user. But the truth is that in fact there is now a complete open source stacked available to build geo enabled mashup:

This is just to named few but in reality there is plenty of component available with a vibrant community around them.
So if PLM want to get close to that ubiquity PLM vendor will need to open the doors and the windows in order to get some fresh air, idea in. This put them in front of 2 challenges :
- Ability to deliver code able to compete with other open source software. In my eyes that means improve their quality and documentation
- Build a business model to keep up this effort and make benefits from it.

Afresco manage to do it in a different realm I am sure someone will do it the only questions left are who and when.

What if SMB as a vision for PLM and PLM vendors don't undertand it.

2008-12-07T19:02:00.000-04:00

This post has been started as a comment of a post from the virtualducthman but since it start to be too long I have decided to write my own post. This is not a rant but rather the impression I had after I moved from a famous PLM Vendor to tier one automotive supplier.

As always extremely well written post without any flaw in the reasoning if you buy in the initial hypothesis. However I disagree with the statement that you use as foundation : Software vendor are proposing excellent product with a good ROI and SMB customer don't understand it because they do not have a vision.

In fact as PLM manager for a tier one supplier in automotive I have seen quite the opposite. I have seen all the big player coming and trying to approach me with solutions that have all the exact same problem :

No interoperability between authoring tools in the Engineering department and Plant
No backward or forward compatibility between version with the tools that they are supposed to integrate with.
All these tools share an interesting feature, no open standard to exchange data (xml, json, rss, atom, ...)
Impossible to mash up the information you put in their repository with other source of information. Ok, I agree impossible is a bit to strong you can always develop this feature and add it to your application. At least by designed these applications are making such thing hard.

What if rather than saying that SMB do not have a vision we try to find a different explanation to your observation. I tend to believe that they have an extremely well defined vision. SMB know that if they want to be in the business in five years from know they need to be agile in order to transform the way they do things to adapt themselves to the new constraints of the OEM. It is interesting to note that nobody is blaming Ford, GM, ... of not being able to see that they have good chance to go bankrupt in some month from now. It is interesting that many people blame now these companies of not being able re-invent/adapt their products to their market. when all of them where using PLM systems and had huge PLM projects on going. May be this can be an interesting post to develop this idea. Why PLM system only help you to develop faster/cheaper product that you are used to produce ? or Why PLM system can hurt your creativity in introducing innovative to your customer ?

In order to develop this agility SMB need to put very high in the list of capabilities for their information system the following features : Agility, re-configuration, continuous evolution / transformation, openness, ease of integration with unknown system, overall strategy of the PLM vendor.

That put us in front of a slightly different picture, isn't it? I am not going to develop in that post why and how each of this term could help to contribute to convince the SMB to change their mind on the value proposition your are doing but I am convince that the impact will be significant. Also I hope that you understand that now known on the solution on the market give them such freedom.

As an example, Alfresco has proven that this approach can work, I understand that this is not in the PLM realm but in content management and web content management but this area shared a couple of similarity with PLM. The market was dominated by closed source vendors providing the same kind of solution for the last 15 years, with no major evolution nor integration of the new tools introduced by the development of internet : wiki, Blog, chat, tag, fulltext search, low cost, commodity software, ... Of course each PLM product have changed several times their names but unfortunately the code, concepts, people behind them are the same it was just a marketing transformation.

Where is the fresh air, idea, innovation ... May be this industry need to show some code rather than powerpoint. It would be nice also if they change their approach in trying to force their solution rather than listening to their customer SMB is not OEM. They do not have will/money to spend in top/down project they need to be able to scale their solution with a bottom up approach. They need to be able to scale from 1 project to a department then a division and at the end to the whole company.

Any way Mr virtualduchman thanks for writing your excellent posts. This post shouldn't be read as PLM do not bring value to SMB but rather like PLM bring value but this can be highly increased by listening to their needs and feedbacks.

How to explore dynamically your django code

2008-11-08T13:22:00.000-04:00

I have been using the methods presented by Simon Willison on this post.
For quite sometimes now and I always felt that it would be nice to have the tab completion when using the debugger inside my views or elsewhere in my django code.

Here it is the magical line:
import ipdb; ipdb.set_trace()

Note : That "pdb" as been replace by "ipdb" which is the IPython debugger.

You then get all the nice stuff from IPython (tab completion, nice tracebacks) right in pdb. In order to install it you can for example do : easy_python ipdb