Out of curiosity, I wanted to see how other frameworks perform in this benchmark.

Frameworks of this benchmark:

• bottle
• CherryPy
• Django
• Falcon
• Flask
• Pecan: This was already part of Falcon's default test. I had never heard of it.
• web.py
• Werkzeug: More like a utility library.
• wheezy.web

Iteration 10

In this test, benchmark is iterated 10 times. The result:

1. wheezy.web........39,681 req/sec or  25 μs/req (28x)
2. Falcon............23,981 req/sec or  42 μs/req (17x)
3. Bottle.............9,479 req/sec or 106 μs/req (7x)
4. webpy..............6,246 req/sec or 160 μs/req (4x)
5. Werkzeug...........5,949 req/sec or 168 μs/req (4x)
6. Pyramid............3,573 req/sec or 280 μs/req (3x)
7. Flask..............3,079 req/sec or 325 μs/req (2x)
8. Pecan..............2,640 req/sec or 379 μs/req (2x)
9. django.............1,770 req/sec or 565 μs/req (1x)
10. CherryPy..........1,410 req/sec or 709 μs/req (1x)

wheezy.web looks like the winner.

Iteration 100

In this test, benchmark is iterated 100 times. wheezy.web is the winner of this iteration as well.

1. wheezy.web........49,022 req/sec or  20 μs/req (35x)
2. Falcon............29,378 req/sec or  34 μs/req (21x)
3. Bottle............11,685 req/sec or  86 μs/req (8x)
4. webpy..............6,763 req/sec or 148 μs/req (5x)
5. Werkzeug...........6,533 req/sec or 153 μs/req (5x)
6. pyramid............3,932 req/sec or 254 μs/req (3x)
7. Flask..............3,334 req/sec or 300 μs/req (2x)
8. Pecan..............2,813 req/sec or 355 μs/req (2x)
9. django.............1,867 req/sec or 536 μs/req (1x)
10. CherryPy..........1,405 req/sec or 712 μs/req (1x)

Iteration 1,000

Benchmark is iterated 1,000 times. wheezy.web is the winner of this iteration.

1. wheezy.web........51,343 req/sec or  19 μs/req (37x)
2. Falcon............29,742 req/sec or  34 μs/req (21x)
3. Bottle............12,030 req/sec or  83 μs/req (9x)
4. webpy..............6,795 req/sec or 147 μs/req (5x)
5. Werkzeug...........6,759 req/sec or 148 μs/req (5x)
6. pyramid............4,061 req/sec or 246 μs/req (3x)
7. Flask..............3,231 req/sec or 310 μs/req (2x)
8. Pecan..............2,839 req/sec or 352 μs/req (2x)
9. django.............1,875 req/sec or 533 μs/req (1x)
10. CherryPy..........1,386 req/sec or 722 μs/req (1x)

Iteration 10,000

Benchmark is iterated 10,000 times. wheezy.web is the winner of this iteration.

1. wheezy.web........52,259 req/sec or  19 μs/req (45x)
2. Falcon............30,345 req/sec or  33 μs/req (26x)
3. Bottle............12,013 req/sec or  83 μs/req (10x)
4. webpy..............6,831 req/sec or 146 μs/req (6x)
5. Werkzeug...........6,723 req/sec or 149 μs/req (6x)
6. pyramid............4,061 req/sec or 246 μs/req (3x)
7. Flask..............3,263 req/sec or 306 μs/req (3x)
8. Pecan..............2,825 req/sec or 354 μs/req (2x)
9. django.............1,862 req/sec or 537 μs/req (2x)
10. CherryPy..........1,173 req/sec or 852 μs/req (1x)

Iteration 50,000

Benchmark is iterated 50,000 times. wheezy.web is the winner of this iteration.

1. wheezy.web........52,245 req/sec or  19 μs/req (48x)
2. Falcon............30,195 req/sec or  33 μs/req (28x)
3. Bottle............11,977 req/sec or  83 μs/req (11x)
4. webpy..............6,950 req/sec or 144 μs/req (6x)
5. Werkzeug...........6,837 req/sec or 146 μs/req (6x)
6. pyramid............4,035 req/sec or 248 μs/req (4x)
7. Flask..............3,300 req/sec or 303 μs/req (3x)
8. Pecan..............2,881 req/sec or 347 μs/req (3x)
9. django.............1,882 req/sec or 531 μs/req (2x)
10. CherryPy..........1,090 req/sec or 917 μs/req (1x)

I've run Falcon's benchmark including my favorite web frameworks (especially wheezy.web) just to see how they perform. In overall, I'm quite surprised of Flask's performance. I knew wheezy.web was going to perform very well so I wasn't really surprised. Andriy's own benchmark shows similar results.

Here's benchmark as a chart.

This is a small proof-of-concept project that mixes Flask, web.py and Apache Shiro. I have tested on Jython 2.7a2.

Apache Shiro is a Java security framework that performs authentication, authorization, cryptography, and session management. You may also be interested in reading this article as well.

This is a demonstration of Flask and web.py with WTForms running on Jython and uses Apache Shiro.

Source code is on GitHub if you're interested.

Related Article:

• Storing Passwords: Password Hashing

• To hide differences between different versions of ZeroMQ.
• To provide a space for development of more sophisticated API semantics.
• To wrap ZeroMQ core API in semantics that are natural and lead to shorter and more readable applications.

CZMQ provides two different structures out of the box:

• zlist: A Singly linked list.
• zhash: A Hash table.

I have implemented a new structure: ztree.

ztree is a generic binary search tree implementation. Sample usage:

int main (int argc, char const *argv[]) {

  zctx_t* context = zctx_new();
  void* socket = zsocket_new(context, ZMQ_REP);
  zsocket_bind(socket, "tcp://*:5050");

  printf("Starting server...\n");

  ztree_t* tree = ztree_new((ztree_compare_fn) compare);
  ztree_insert(tree, "hello");
  ztree_insert(tree, "world");
  ztree_insert(tree, "bonjour");
  char* s = ztree_find_max(tree);
  for(;;) {

    char* msg = zstr_recv(socket);

    char* mss = ztree_find_rec(tree, msg);
    printf("Received: %s\n", msg);

    if(mss) {
      printf("Found: %s\n", mss);
      zstr_send(socket, s);
    } else {
      printf("Not Found: %s\n", mss);
      zstr_send(socket, "Not Found");
    }

    free(msg);
  }

  zsocket_destroy(context, socket);
  zctx_destroy(&context);

  return 0;
}

Source code is on github if anyone is interested.

I didn't do much customizations. It's a straightforward Hadoop that comes out of the box. I had my map code and reducer code and ran the code to find out the most important words of some documents I had on my computer since I don't have access to a corpus right now.

Things to Note

Hadoop sorts using Quicksort by default. If you want to change it to something else (Heapsort or Mergesort), you need to edit mapred-site.xml. For example;

<property>
    <name>map.sort.class</name>
    <value>org.apache.hadoop.util.HeapSort</value>
    <description>Use heap sort class for sorting keys.</description>
</property>

Note:

• If you want to use Mergesort be careful: If the length of data is less than 7, then it will be sorted with Insertion sort. See line 40 in org.apache.hadoop.util.Mergesort.java.
• If the depth of recursion goes too deep, then Quicksort will switch to Heapsort. See line 35 and line 76 in org.apache.hadoop.util.Quicksort.java.

Steps

1. Let's have a clean system.

   hadoop namenode -format

2. Once the system has been formatted then we could start the nodes.

   ./start-all.sh

3. Let's create a new folder in HDFS.

   hadoop dfs -mkdir input

4. Then run your map code and reduce code.

When everything goes well, I could go to http://localhost:50030 and see how the jobs perform.

Everything went better than expected

Let's have an error on reduce code (Yes, I made reduce code fail on purpose to take the screenshot). In that case, the graph looks like

Houston, we have a problem

You can do anything with Hadoop. Anything at all. The only limit is yourself.

Why would you want to detect the important words?

The answer is quite easy. Let's say you want to build keywords of a document to describe it. There are more complex ways of analysing the most important words but the simplest approach would be calculating the Tf.Idf scores of a document.

What is Tf.Idf?

• Tf: Term Frequency
• Idf: Inverse Document Frequency

Tf (Term Frequency)

Let's start with finding the Tf of a given document. The formula is simple:

Tf(d, t)

• d: Document
• t: Number of times term t appears in document d.

Term Frequency

Here's an example on how to construct Tf table. The sentences are from Friedrich Nietzsche's Ecce Homo.

• document1: how one becomes what one is.
• document2: the remotest idea of what one is.
• document3: one becomes what one is.

The problem with term frequency is most frequent words don't tell much about the document and in a real-world example the most common words you're going to get will be; and, the, of, a, to....

Most common words are not the most important words of a document.

In a real-world situation you'd most likely want to split the words with a smart REGEX and by filtering out with a stop words list. Anyway, so if we want to calculate Tf scores then simplest approach would be;

List<String> stop_words = new ArrayList<String>();

static {
    stop_words.add("a");
    stop_words.add("an");
    stop_words.add("the");
}

List<String> build_words(String document) {
    Matcher matcher = Pattern.compile("[a-z]{2,}").matcher(document);
    List<String> list = new ArrayList<String>();
    while(matcher.find()) list.add(matcher.group());
    return list;
}

Map<String, Int> tf(String document) {
    Map<String, Int> map = new HashMap<String, Int>();
    String[] list = document.split(" ");
    for (String s: list) {
        Int counter = map.get(s);
        if (counter == null) map.put(s, new Int());
        else counter.incr();
    }
    return map;
}

Df (Document Frequency)

The formula is:

Df(c, t)

• c: Documents (corpus) of a given dataset.
• t: Number of times term t appears in corpus c.

Inverse Document Frequency Formula:

1 / Df(c, t)

Document Frequency

Document frequency is the total number of documents that word w appears in.

• document1: how one becomes what one is.
• document2: the remotest idea of what one is.
• document3: one becomes what one is.

When you calculate Idf values of words then you start getting somewhere but it's still not good enough.

Map<String, Int> df(List<String> documents) {
    List<String> list = new ArrayList<String>();
    for (String s: documents) {
        Set<String> set = new HashSet<String>();
        set.addAll(Arrays.asList(s.split(" ")));
        for (String d: set) list.add(d);
    }
    return counter(list);
}

Map<String, Double> idf(List<String> documents, Map<String, Int> map) {
    Map<String, Double> map2 = new HashMap<String, Double>();
    for (String s: documents) {
        String[] split = s.split(" ");
        for(String d: split) {
            if (map.get(d) != null) {
                double val = 1.0 / (map.get(d).getCounter() + 1e-100);
                map2.put(d, val);
            }
        }
    }
    return map2;
}

Please note that, these codes shouldn't be used in real-world. I'm just trying to make things easier and simple.

Tf.Idf

Now, we know how to calculate Tf and Idf, as it can be easily guessed Tf.Idf is the combined version of Tf and Idf.

The formula is:

Tf.Idf(c, d, t) = Tf(d, t) / Df(c, t)

Le Problem

The problem with the simplest Tf.Idf approach is it gives very high score for very rare words so we need to "tweak" the formula. The simplest way would be adding log.

The updated formula:

Tf.Idf(c, d, t) = Tf(d, t) log (N / Df(c, t))

• N: The number of total documents.

Le Problem

We still have a problem with the tweaked formula. The longer documents will have higher Tf.Idf scores than the shorter documents. Solution would be dividing the Tf.Idf score by the document length.

The updated formula:

Tf.Idf(c, d, t) = Tf.Idf(c, d, t) / |dN|

• dN: Document length.

So, finally the most important words:

def most_important_words(documents, length=5):
  mapp = tfidf(documents)
  mapp = collections.OrderedDict(sorted(mapp.items()), key=lambda x: x[1], reverse=True)
  return mapp.items()[:length]

I've used stop words to get more accurate results. The stop words I've used are from google.appengine.ext.search package.

All words are important but some words are more important than the others.

The major features of web.py is; simplicity, minimalism, doesn't get in your way and plays well with other libraries.

Why web.py?

If you're after:

• simplicity
• freedom
• don't want to shoot yourself in the foot
• writing clean code
• minimalism
• a solid web framework

then web.py is the web framework you're looking for.

A quote from web.py's mailing list that explains web.py quite well:

I don't want to be forced to abstract my database into objects and compartmentalize my code into MVC. I don't wanna have four technologies (html/css/js/php) intertwined all in one file. I don't want to compile all my code every time I change one line just to test it. Some of my objects actually have sub-classes six levels deep. Web.py empowered me to start working immediately.

Getting Started

Skeleton! Your web application needs a skeleton. I had created a sample skeleton for web.py and put it on GitHub. You could have a look at it if you wish.

Here's my web.py skeleton and what each element does.

• doc: Here goes the project's documentation files.
• licenses: That's where I put the licenses (GPL, MIT, BSD etc.) of the project and each library I use for the application.
• requirements: Required 3rd party libraries for the project.
• sh: Bash script files of the project.
• www: The project itself.
  • app: Contains the application modules.
    • controllers: The project itself is based on MVC architecture so this module contains the handler modules of controllers package.
    • tools: Tools that is used for the project.
    • views: Template files.
    • models: Database models of the application. It's 120% chance that these are SQLAlchemy files.
    • bridge: Sometimes you need to communicate with a server that's written in another language. That's where I put all those "communication" files.
  • lib: These are libraries I develop for the project but aren't really tools. Most likely these are written in C (Python modules though). You know, sometimes you need to "put the pedal to the metal". The difference between a lib and a tools is; the modules in libs could be used in another projects whereas tools are dependent to project itself and can't be used elsewhere.
  • log: Application's log files. I log everything. EVERYTHING.
  • public: These folder contains the minimized, compiled CSS, Javascript, CoffeeScript files and images so the files in this folder are production ready and can't be used in development.
    • css: Production ready CSS files which are minimized and compiled (from LESS).
    • js: Generated Javascript files from CoffeeScript and minimized and compiled using Google Closure.
    • img: Minimized images files. These are compressed using unusual techniques.
  • static: Contains the development CSS, CoffeeScript, Javascript, and images files of the project.
    • css: LESS/CSS files for development. For this project we didn't use LESS though.
    • cs: CoffeeScript files.
    • js: Javascript files in development mode.
    • img: Images of the application.
  • test: Test files of the application. These are, as you could guess easily, automated by using Selenium and nose.
  • tmp: Garbage files basically.
  • main.py: The only files that's executed directly by the server (WSGI, FastCGI or whatever it's).
  • main_development.py: This is the main executed file in development mode.
  • settings.py: Global constants and settings of the application. This is somehow similar to Django's settings file.
  • urls.py: Contains the URLs of the application.

High Level Diagram

Here's the high level diagram of this skeleton. I think it's self explanatory. The skeleton is based on MVC (model-view-controller) architecture.

Package diagram - Architecture is based on MVC.

High Level Package Diagram

High Level Diagram

web.py features

Database

web.py has very interesting features. It comes with a db package which lets you use various of different databases. However, it's not an ORM. It's somehow similar to sqlite3 package of Python but is capable of connecting to multiple databases (This feature was missing in Django for a quite long time but it was a feature of web.py from day 1). This is a compelling reason for people who are good at SQL and don't like to use an ORM.

As I said, the main thing about web.py is simplicity. If you're doing something simple this package is very suitable. However, I'm a big fan of SQLAlchemy and the last thing I did was not really a simple application so I didn't use web.db.

The best thing about this web.py module is you could just take it and use it independently. Here, I've created a small Flask application which uses web.py's database module web.db.

from web import database
from flask import Flask, render_template, abort, flash, redirect, url_for, request

DATABASE = '/tmp/flaskr.db'
SECRET_KEY = 'what'
DEBUG = True

app = Flask(__name__)
app.config.from_object(__name__)

conn = database(dbn='sqlite', db=DATABASE)

@app.route('/')
def index():
    entries = conn.select('entries',
                          what='id, title, text',
                          limit=20,
                          order='id DESC')
    return render_template('index.html', entries=entries)

@app.route('/add', methods=['GET', 'POST'])
def add():
    if request.method == 'POST':
        conn.insert('entries',
                    title=request.form['title'],
                    text=request.form['text'])
        flash('a new entry')
        return redirect(url_for('index'))
    else:
        return render_template('add.html')

@app.route('/entry/<int:entry_id>')
def show_entry(entry_id):
    entry = conn.select('entries',
                        what='id, title, text',
                        where='id = $id',
                        vars={'id': entry_id},
                        limit=1)
    try:
        return render_template('entry.html', entry=entry[0])
    except IndexError:
        return abort(404)

def a():
    q = """
create table entries(
id integer primary key autoincrement,
title string not null,
text string not null
);
"""

if __name__ == '__main__':
    app.run(debug=DEBUG)

web.py is so flexible and has flexible modules that you could wipe out whatever you want from it and use it with another web framework. The above code is proof of that. Entire application is on GitHub.

Forms

web.py comes with a forms package which let's you create forms and validators. Ironically, it doesn't have built-in protection against CSRF. You want to create a login form? You could use forms package to create yours. Here's a simple example;

class IFormLogin(object):
  @staticmethod
  def render_login_form(token):
    login_form = Form(
      form.Textbox('username', form.Validator('Please write your username',
                                              lambda
                                                  x: USERNAME_MAX_LENGTH >
                                                     len(x)) >
                                                     USERNAME_MIN_LENGTH),
                   description='Username:',
                   id='username',
                   class_='span4'),
      form.Password('password', form.Validator('Please write your password',
                                               lambda
                                                   x: PASSWORD_MAX_LENGTH
                                                      > len(x)) >
                                                      PASSWORD_MIN_LENGTH),
                    description='Password:',
                    id='password',
                    class_='span4'),
                form.Hidden('token', id='token', value=token),
      validators=[form.Validator('Username or password is wrong',
                                 Member.exists)]
    )
    return login_form

Related Posts

1. A Python Development Story: Part 1 Jenkins-CI meets web.py and PyCharm
2. A Python Development Story: Part 2

What we did

I can't talk much about it much but the big picture was to analyze and visualize data sets. It wasn't a some sort of a blog engine that 99.99% of Django applications are about (That's the part I salute Cal Henderson).

Tools

It was a Python application with the following technologies;

• web.py
• SQLAlchemy
• Mako templates
• CoffeeScript
• and some C sauce

Framework Choice

We couldn't choose Django, web2py or any other macro-framework since the application we built wouldn't be suitable for them. It wasn't a web application, it was simply a web service to analyze and visualize data sets. For example;

• Django's ORM would literally cry if we ever wanted to use it and its forms modules would be useless for us.
• I've never used web2py so I didn't want to touch something I don't know and since it's not really small and I was afraid learning curve might be too long.

By looking at these options we had 3 alternatives on the table;

• web.py: The ultimate web framework that used to power reddit.
• Flask: New Python micro framework that relies on Werkzeug.
• bottle: The web framework that's only 1 file.
• brubeck: The most interesting framework I've seen for a quite long time.

Why not Flask?

If you bother to read the rest of the article, you'll notice that I mention "strict regex rules" somewhere.

What I want: Regex based URLs

Flask provides helpers such as int/float but they're useless in our application. We don't have integers in URLs nor names. We have UUIDs and bunch of other things.

The Problem With Flask

Maybe it's my ignorance but after searching the documents and the web, it's come to my attention that this is what I need to do if I want REGEX URLs in Flask:

First I need to create a Converter class:

from werkzeug.routing import BaseConverter

class Converter(BaseConverter):
  def __init__(self, mapper, *args):
    super(Converter, self).__init__(mapper)
    self.regex = args[0]

I've no clue why I needed this in the first place. I know Flask relies on Werkzeug but at the end of the day if I wanted to do something with Flask, I had to import a 3rd party library. This doesn't look very right to me and as far as I know other frameworks I've coded such as Django, Brubeck and Google App Engine's webapp don't have this kind of thing. After defining this converter class, this is what I need to do;

app.mapper.converters['regex'] = Converter

After this, I could write REGEX URLs such as;

app.add_url_rule(r'/u/[a-z]{5,10}', view_func=IndexHandler.as_view('index'))

Maybe I looked at the wrong documentation but I really shouldn't needed to do all of these. In my point of view, this is a horror story. If you look at Django, web.py, Google App Engine's webapp or Brubeck you just write your REGEX URLs.

Mako Support

There's an extension for Flask to support Mako but I do remember I had run into some problems with it (and I ended up writing my own mako integration) and a quick search on StackOverFlow showed me that I wasn't the only one who had problems.

I didn't look at the rest of features Flask offers after these.

If I look at the bright side; Flask has an amazing documentation and is downloadable. Downloadable documentation is useful if you code in some place that you don't have Internet access. I wish more libraries would do that.

However, I had noticed Flask is good enough for creating small applications but I wouldn't touch it for creating complex applications.

Why not bottle?

I wanted class based controllers and as far as I know bottle doesn't provide such functionality. If you do bottle, please forgive my ignorance!

Why not Brubeck?

Brubeck relies on mongrel2 and some other cool technologies however we had to use Apache (we didn't have any alternatives and it wasn't our call to make a decision).

Why web.py?

web.py gives you freedom and does what you say. The thing I like about it is its flexibility thus it was very suitable for us. However, it wasn't all that good. I don't like a thing in web.py: URL definitions.

URL definitions in web.py are defined as in string pairs. For example;

urls = (r'/hello', Hello, r'/world', World)

I think this isn't very right. In my point of view, this looks more beautiful;

urls = [(r'/hello', Hello), (r'/world', World)]

Apparently I'm not the only one who complains about this.

However, web.py met all the mandatory conditions we looked for so it was suitable for us. If you're interested in more about why web.py, you may want to read the follow up article.

The Big 3: Development, Source Control, Bug Tracking

The Cycle

The cycle of an application is;

• Develop
• Deploy
• Maintain

Development

Development is the most important part since it's a continuos process. You need to select the right tools for the job. Some rules we followed:

• If a code can't pass PEP8*, I don't want to see it in codebase. I don't care if it works.
• If PyLint scores the code less than 8/10, I don't want to see it. I don't care if it works.
• If there's no comment, that code has no place in codebase.
• If you introduce a new method and you didn't write the necessary test for it, I don't want to see it.
• Don't forget the clone detection and code execution order analysis.

*: We used a slightly modified PEP8.

IDE Choice

I'm an emacs user and I do development in emacs when it comes to Python, C, C++, Cilk++, Erlang etc. I use an IDE (IntelliJ IDEA) only for Java development.

However, I had always wanted to try PyCharm and for this project I used PyCharm.

PyCharm

A Python IDE from JetBrains. It's just great. I loved it even more when it did coverage automatically. My only criticise about PyCharm is themes. There should be a way to import TextMate themes (if there's a way, I don't know).

It's not only a Python IDE though. It does lint checking, auto completion for Javascript, CoffeeScript and CSS. It supports Mako templates out of the box and provides GUI for version control systems and connects to database too.

And the price is bargain if you consider what it can do. There are some criticism about PyCharm being slow but I didn't have any problems.

The only problem I saw was it takes a little bit more time to load the project if you have lots of files and maybe it's my ignorance but I couldn't see a way to disable loading the files on some folders. It'd be nice if there was a setting to ignore loading the files on startup on some certain folders. This is a serious issue if you have more than 280,000 log files in your log folder. In this case PyCharm never loads the project and you give up and close it.

PyCharm does lint checking and error checking for Javascript and CoffeeScript files as well.

PyCharm - CoffeeScript Error Checking

Another cool feature of PyCharm is it allows you to see the auto-generated Javascript code of your CoffeeScript side by side.

CoffeScript code on the left and auto-generated Javascript code on the right

Talk to me about Testing

Automated Tests

Yes, all the cool kids use Selenium which is good enough and integrates with nose well. It automates all the tests. Takes screenshots when necessary. Taking screenshot is quite useful I think since it lets you see what a user experience. The visible error message are very useful too.

For example consider the following piece of Selenium test code;

1. test_get_something: Goes to the URL and finds the text named My View and clicks on it, then browser waits 10 seconds (since the other page is loading), then after 10 seconds find a text named view.
2. test_screenshot: saves the screenshot of the given URL.

def test_get_something(self):
  self.browser.get(self.url)
  text = self.browser.find_element_by_link_text('My View')
  text.click()
  WebDriverWait(self.browser, 10).until(lambda x: x.find_element_by_id('view'))
  find_text = self.browser.find_element_by_id('view')
  self.assertEqual(find_text, None)

def test_screenshot(self):
  self.browser.get(self.url + '/go_where')
  time.sleep(5)
  self.browser.save_screenshot('test.jpg')
  self.browser.close()

Stress Testing

It's all about testing the limits of your application. ab is your friend. I was quite happy when I found out our web.py application was capable of handling 40,000 requests and 165 multiple connections without any caching or anything else.

A sample ab command would be;

ab -c 180 -n 40000 http://192.168.100.124/

Performance

Performance measurement of a web application has 2 parts:

1. Client Side
2. Server Side

Client Side Performance

This could be measured by YSlow, an extension for Firebug. Google has a tool for performance measurement but it's only for 32 bit architecture.

Server Side Performance

As I said ab is your friend. It does the job.

Penetrate Testing

There are a lot of tools you could use for penetrate testing. For example; XSSer for XSS testing. The main thing we (we means I by the way since I was the only developer) had no choice but to use offline testing tools. Online testing tools are useless for various of reasons;

• 99.99% times your development servers don't have Internet connection and the only way to connect them is via VPS.
• I'd like to run the test on development code not on production code.
• Let's say I found a security vulnerability on production code. I can't fix it immediately. I know, I know, real men do make changes on production code using nano but I'm a coward.

Online testing tools were useless in our case since our application servers didn't have Internet connection and we could only connect them via VPS.

Source Control

This part is important. You need to select something that won't screw you back or won't corrupt your files. Our choice was git whereas the other team (there were two teams on the project which designed different parts of the project) decided to go with subversion.

Before Committing

Before committing files to repository, the following steps were done:

• Do lint analysis
• Optimize imports
• PEP8 and PyLint analysis

Bug Tracking

There are a lot of bug tracking software out there but the teams went with the following:

• YouTrack: Issue tracker and bug tracker from JetBrains. It's awesome-o.
• Trac: It's good enough and was the choice of other team.

Security

If you don't use a framework, or if you use a minimalist framework it's your responsibility to secure your application. Even if you use a framework such as Rails or Django, it's still your responsibility to secure your application. I'm not much a security guy but I took care of every vulnerability I know and took necessary steps to store the passwords. No, passwords are not stored in plain text nor are they hashed with SHA1 without any salt key (I'm looking at you LinkedIn).

• XSS: web.py provides a method to protect against XSS. However, I used markupsafe library.
• SQL-Injection: Since we used SQLAlchemy this wasn't really an issue.
• CSRF: web.py doesn't do any protection against CSRF and as far as I know there's not a 3rd party library so I had to code my own protection. The key point of CSRF is to generate a key that's hard to guess for an attacker so I generated random SHA256 keys for CSRF tokens.
• Cookie Poisoning: How I handled this is easy. You need to send the cookie to user for obvious reasons but the cookie the application sends to user is a SHA256 digest which stores the necessary information. There's an encrypt key, validation key and secret key which are used to generate the cookie and these keys are stored only on server side and the cookie is stored in database too. I don't want to go into too much detail how these different keys are used.

Later on you need to take care of active network attackers and bunch of other things.

Application Specific Security

This part is about form validation and storing the passwords. Since I can't go into too much detail but we had lots of forms which needed their own validation and protection layers. This involved writing strict regex rules and strictly checking every single input.

Storing Passwords

User passwords are hashed N times where N is different for each user by using a global salt key and a salt key which is different for each user.

Deployment (aka Production vs. Development)

The development code and the production code are different than each other. You need to compile, minimize the static files such as Javascript, CSS and image files. Even the Python files are different on production and on development mode.

For this reason, I use my own tools. I'm not cool enough to use fabric (no, there's no sarcasm or arrogance when I say it).

Compile Javascript

There are lots of tools for this job. In our case the Javascript files were auto-generated from CoffeeScript.

• Google Closure Compiler: My personal favorite.
• Yahoo Compressor: Never used it to compress Javascript. I have no clue.
• Dojo Compressor: Never used it. I have no clue.
• JSMin: Never used it. I have no clue.

JS Lint Checking

PyCharm provides lint checking out of the box.

PyCharm - Javascript Lint Checking

Minify CSS

• Yahoo Compressor: Good enough. Does the job.
• Less: LESS provides a minimizer as well, you could also use Yahoo Compressor with it though.
• Icey's Compressor: This is an online tool so there's no way to use it offline. That's the disadvantage I noticed first. Never really used it.

Gzip All The Things!

/usr/bin/gzip -cn9 main.js > small.js.gz

Sorcery?

Team Issues

Every person has different point of views and skills and abilities. The most common issue is communication issues.

Communication Issues

This part is huge. What I understand is different than what you understand. People have different levels of skills and abilities. If another team member doesn't have the knowledge and technical background as you then the entire team is in trouble.

Another part is the language. English is the second or the third language for some people. Even if the person is the native speaker of English, if he doesn't have the technical background, level of knowledge differences end up with miscommunication problems.

Communication

Telephone, SMS, Skype, meetings and email are the tools you could use for communication. You could also "shout at each other" as a communication tool. It works but we didn't use it, well we rarely used it, maybe only once.

The Thing That Should Not Be

Software methodologies. Waterfall, agile, scrum, XP, you name it. I could write something about them but legendary Zed Shaw said it very well.

Lessons Learnt

1. Hubris will cause you problems.
2. Never say "what can go wrong" and take frequent backup of files and make sure you commit everything. EVERYTHING.
3. Make sure you commit and backup everything (I know I said this above). Seeing node.right can't be null: ReferenceException in the last minute isn't very pleasant.
4. C can play with Python quite well.
5. Even if it's a web application there's nothing wrong with "putting the pedal to the metal". C is your friend where suits.
6. Asynchronous all the things!
7. web.py is solid and capable of creating complex web services.

Related Posts

1. A Python Development Story: Part 1 Jenkins-CI meets web.py and PyCharm
2. A Python Development Story: Part 3 - web.py Skeleton

You could grab the theme from here.

Jenkins-CI

We use the following tools with Jenkins-CI:

• git: Our vcs choice
• nose: Unit test library.
• sloccount: Calculates the lines of code.
• PyLint: Static analyzer
• clonedigger: Detects duplicate code.
• coverage: Measures code coverage by monitoring the program, traces hooks, notes which parts have been executed, which parts could have been executed but wasn't.
• pep8: Coding style.

The following plugins are necessary:

• git
• Sloccount
• Python
• Violations
• Warnings
• xUnit

All these plugins could be installed from plugin manager.

** There's also ShiningPanda plugin which adds some useful builders such as virtualenv but we've not tried it.

Jenkins takes care of everything. It clones your git repository and does the necessary steps.

Shell Build Steps

Shell build steps are straightforward. For example;

for filename in `find . -name *.py | egrep -v '^./modules/'`; do pep8 --ignore=E111 $filename; done || :

PyCharm

PyCharm is a Python IDE from JetBrains. In addition to plain Python, it can syntax highlight and check for errors for Cython, CoffeeScript, Mako/Jinja2 templates as well. You could connect to database and work on database tables too!

It supports git, virtualenv, pip (and probably lots of other things I've not noticed yet).

For example; Cython highlighting and error and spelling checking:

Cython highlighting and error and spelling checking

and when you connect to your database in PyCharm;

Pretty cool, huh?

The rest is web.py, SQLAlchemy, Mako, gevent and ZeroMQ.

Related Posts

1. A Python Development Story: Part 2
2. A Python Development Story: Part 3 - web.py Skeleton

Patricia Trie was discovered by Donald Morrison in 1968. It takes log N bit for comparison of each search even though insert/delete operations run on O(n).

Nodes in Patricia Trie contain a field which tracks the bit positions. We use a dummy node and initialize its bit to -1. Root of the trie is root.left.

template<typename K, typename T>
class INode {
public:
  virtual ~INode() {}
  virtual unsigned hash(K key, unsigned bit) = 0;
};

template<typename T>
class StringKeyNode: public INode<string, T> {

private:
  static const int SIZE = sizeof(string) << 2;

public:

  string key;
  T value;
  int bit;
  StringKeyNode<T>* left;
  StringKeyNode<T>* right;

  StringKeyNode(string key, T value, int bit) {
    this->key = key;
    this->value = value;
    this->bit = bit;
  }

  ~StringKeyNode() {
    this->left = NULL;
    this->right = NULL;
    this->bit = -1;
    this->value = NULL;
  }

  virtual unsigned hash(string key, unsigned i) {
    if(key.empty()) return 0;
    unsigned index = static_cast<unsigned>(i / SIZE);
    if(index >= key.length()) return 0;
    unsigned val = (1 << SIZE - 1) >> (i & SIZE - 1);
    return key.at(index) & val;
  }
};

Insertion

Inserting a new node takes O(n) time.

For example;

    -- NULL : initialization
    -- insert:  ababbb -> key
                123456 -> positions
    -- insert:  ababba:
                search: ababb**b** == ababba
                diff. at position 6;

         [6]
        /   \
       a     b
      /       \
    ababba   ababbb

Inserting a new node begins with searching. search method returns a unique key. Once we detect the leftmost position of the key we're inserting and the search key, we call the insert_inner method which recursively walks the tree and inserts the node where node is pointing at.

StringKeyNode<T>* insert_inner(StringKeyNode<T>* node,
                                  K key,
                                  T value,
                                  int i,
                                  StringKeyNode<T>* root) {
    if(node->bit >= i || node->bit <= root->bit) {
      StringKeyNode<T>* temp = new StringKeyNode<T>(key, value, i);
      const bool FLAG = node->hash(key, temp->bit) != 0;
      temp->left = FLAG ? node : temp;
      temp->right = FLAG ? temp : node;
      return temp;
      delete temp;
    }
    if(node->hash(key, node->bit) == 0) {
      node->left = cilk_spawn insert_inner(node->left, key, value, i, node);
    } else {
      node->right = insert_inner(node->right, key, value, i, node);
    }
    cilk_sync;
    return node;
}

void insert(string key, T value) {
    StringKeyNode<T>* node;
    node = cilk_spawn search_inner(this->root->left, key, -1);
    cilk_sync;
    if(node != NULL) {
      string temp = node->key;
      if(temp.compare(key) != 0) {
        int i = 0;
        while(node->hash(key, i) == node->hash(temp, i)) ++i;
        this->root->left = cilk_spawn insert_inner(this->root->left,
                                                    key,
                                                    value,
                                                    i,
                                                    this->root);
        cilk_sync;
      } else {
        node->value = value;
      }
    }
  }

Searching

Important part of searching is the search_inner method which walks the trie recursively and finds a unique key which can have the record of node's key by using the bits of the trie.

StringKeyNode<T>* search_inner(StringKeyNode<T>* node, K key, int i) {
    if(node->bit <= i) return node;
    if(node->hash(key, node->bit) == 0) {
      return search_inner(node->left, key, node->bit);
    } else {
      return search_inner(node->right, key, node->bit);
    }
}

T search(string key) {
    StringKeyNode<T>* node;
    node = cilk_spawn search_inner(this->root->left, key, -1);
    cilk_sync;
    if(node == NULL || node->key != key) return NULL;
    return node->value;
}

Inorder

Putting the nodes in order is no different than an ordinary tree. The only difference is we check the bits of the nodes to detect if node's bit is smaller than parent's bit.

string inorder_inner(StringKeyNode<T>* node, int i) {
    if(node == this->root) return "";
    if(node->bit <= i) return node->value;
    return inorder_inner(node->left, node->bit) + inorder_inner(node->right,
                                                                  node->bit);
}

string inorder() {
    return inorder_inner(this->root->left, -1);
}

Testing

Here's a small code to insert some values to test the trie.

vector<string> generate_names(string s) {
  // generating some strings to put into trie.
  vector<char>v(s.begin(), s.end());
  vector<string> e;
  e.push_back(s);
  while(next_permutation(v.begin(), v.end())) {
    stringstream ss;
    for(unsigned i = 0; i < v.size(); i++) ss << v[i];
    e.push_back(ss.str());
  }
  return e;
}

int factorial(int n) {
  int res = 1;
  for(int i = 1; i <= n; i++) res *= i;
  return res;
}

Benchmarking

To see how our trie performs in various of scenarios and different workers, we could use cilkview for analyzing.

int cilk_main() {

  cilk::cilkview cv;

  string generator = "abc";
  const int LENGTH = factorial(generator.length());
  vector<string> names = generate_names(generator);

  cv.start();

  for(int i = 0; i < LENGTH; i++)
    tree->insert(names[i], SomeValue(names[i], names[i]));

  cv.stop();

  cout << "tree filled in " << cv.accumulated_milliseconds() / 1000.f << " seconds.\n";
  cv.reset();

}

Fun Stuff

Since we've constructed our trie and it works like a champ so now we could play with it. Let's rotate the trie (for absolutely no particular reason)

void rotate_inner(StringKeyNode<T>* node, int bit) {
    if(node != NULL && node->bit > bit) {
      cilk_spawn rotate_inner(node->left, node->bit);
      rotate_inner(node->right, node->bit);
      cilk_sync;
      swap(node->left, node->right);
    }
}

void rotate() {
    cilk_spawn rotate_inner(this->root->left, -1);
    cilk_sync;
}

Source Code

You could get the source code on GitHub. Feel free to fork and improve or fix it. Released under "do whatever you want to do with it" license.

References:

1. NIST