Is numpy really that much faster ?

2017-04-02T00:00:00+00:00

numpy has been the rage for quite sometime within the python community and I have yet to find a nice write up that really compares the performance of using numpy vs regular python lists to get specific tasks done so I decided on writing up a quick and dirty comparison.

First lets simply compare the amount of memory required to store 1 million integers in a python list vs 1 million integers in a numpy array:

from memory_profiler import profile

@profile
def main():
    data = [0 for _ in range(0, 1000000)]

if __name__ == '__main__':
    main()

and in numpy:

from memory_profiler import profile

import numpy as np

@profile
def main():
    data = np.zeros(1000000)

if __name__ == '__main__':
    main()

We used the neat little library memory_profiler to take care of profiling memory usage for the main() function and in this one little test we can already see the tremendous benefits of using numpy over traditional python lists:

> python 1_million_python_list.py
Filename: 1_million_python_list.py

Line #    Mem usage    Increment   Line Contents
================================================
     3     13.1 MiB      0.0 MiB   @profile
     4                             def main():
     5     21.0 MiB      7.9 MiB       data = [ 0 for _ in range(0, 1000000)]


2017-04-02 15:25:09 $?=0 pwd=/home/rlgomes/workspace/python3/numpy venv=env duration=65.931s

> python 1_million_numpy_array.py
Filename: 1_million_numpy_array.py

Line #    Mem usage    Increment   Line Contents
================================================
     5     26.1 MiB      0.0 MiB   @profile
     6                             def main():
     7     26.4 MiB      0.3 MiB       data = np.zeros(1000000)


2017-04-02 15:23:57 $?=0 pwd=/home/rlgomes/workspace/python3/numpy venv=env duration=.290s

Which in terms of time to execute numpy is 227x faster and in terms of memory usage it is 26x more efficient.

Now what if we assume the list already exists in memory and we don’t care about the time it took to load it or even how much space it occupies what happens when we try to calculate statistics over the million integers:

import timeit
import random

import numpy as np

python_list = [random.random() for _ in range(0, 1000000)]
numpy_array = np.array(python_list)

elapsed = timeit.timeit('sum(python_list)', number=100, globals=locals())
print('%.5fs for python sum(list)' % elapsed)

elapsed = timeit.timeit('np.sum(numpy_array)', number=100, globals=locals())
print('%.5fs for python numpy.sum(array)' % elapsed)

elapsed = timeit.timeit('max(python_list)', number=100, globals=locals())
print('%.5fs for python max(list)' % elapsed)

elapsed = timeit.timeit('np.amax(numpy_array)', number=100, globals=locals())
print('%.5fs for python numpy.amax(array)' % elapsed)

The above gives the following output on my machine:

> python million_integer_stats.py
0.56150s for python sum(list)
0.06869s for python numpy.sum(array)
2.05134s for python max(list)
0.06859s for python numpy.amax(array)

Which puts numpy at 8x faster at calculating the sum of a million integers and at 29.9x faster at calculating the maximum value.

Any other comparisons will surely continue to show just how much more efficient numpy is at statistical analysis and there are a ton of more functionality the library has to offer from operating on matrices to fitting polynomials to existing data.

The main thing to takeaway from this post is to use numpy when you are doing any kind of math over hundreds of thousands of numbers as it will perform much better and remove the need for coding up your own statistical functions.

Human friendly context aware duration parsing library

2017-03-04T00:00:00+00:00

Recently needed the ability to parse durations from human readable strings that were also context aware. The context being the date to start your duration calculation from so that if you started on January 1st 2017 and wanted 2 months you’d get exactly 31 (number of days in January 2017) + 28 (number of days in February). Then if I gave it the context of April 1st I’d get 61 days since there were two months with 31 days each.

I tried to find an existing library with no luck so I wrote delta to take care of the job and hopefully someone else would find it of use. You can get your hands on delta easily through pypi like so:

pip install delta

Once installed you can use it like so:

import delta

from datetime import datetime

print(delta.parse('1 year 2 months and 3 days'))
print(delta.parse('2 months and 3.5 weeks', datetime(2017, 3, 4)))

You can see that delta allows you to easily include a context or not and when you don’t supply the context it will assume the current date. Another thing you may have noticed is you can get quite expressive with the duration expressions being able to do all of the following:

1 year 2 months and 3 weeks
2 months, 3 weeks and 12 days
1y 2m 3w 4d
3.5 years and 2.7 days

delta will handle all of those without any issues.

If you find delta useful then head over to the github project and open any issues or contribute a PR for any additional features you’d like.

Shell tips and tricks

2016-12-03T00:00:00+00:00

Its been a while since I’ve written a blog post and I thought I’d dedicate this one to some of the tricks I’ve used when working with my shell in my terminal that make me more productive on a daily basis.

To start I like to use zsh because it has a lot of features that few other shells have such as:

autocomplete commands such as kill with the list of currently running processes and it does this for quite a few other commands
shared history (among different zsh sessions)
hook functions (we’ll talk about these below)
many more features…

There are a ton of them but the ones that I find the most useful are the hook functions which you can read more about here. I find myself using the hook functions to make my life easier every day when working. The first very simple set of things I do with the hooks is to use the chpwd hook to automatically change the title of my terminal to match the name of the directory I’m currently in. In my .zshrc I have:

set_window_title() { 
    echo -ne "\033]0;"$PWD"\007"
}

# set the title on the first pass through here
set_window_title

# update the console title on directory change
chpwd() {
    set_window_title
}

The previous will make it so that your title matches the current working directory at all times but it won’t handle well the situation where the current working directory name is super long. So I’ve devised a slightly different solution where I’ll truncate the longer title and put a few dots and show the last part of the path which is more important to the user, like so:

pad() {
    if [ ${#1} -lt $2 ]
    then
        pad "$1 " "$2"
    else
        echo "$1" fi
}

ltrunc() {
    FILL="..."
    if [ "$3" != "" ]
    then
        FILL="$3"
    fi
    print -P "%$2<$FILL<$1"
}

lpad_title(){
    VAR=`ltrunc "$1" 32`
    pad "$VAR" 32
}

set_window_title() { 
    export PREFIX=""
    if [ "$VIRTUAL_ENV" != "" ] 
    then
        VENV=$(basename $VIRTUAL_ENV) 
        export PREFIX="($VENV)"
    fi

    STRPATH=`lpad_title "$PWD"`
    echo -ne "\033]0;"${PREFIX}${STRPATH}"\007"
}

Its definitely more complex but makes for better looking titles when the directo name is longer than 32 characters.

The next trick I like to use is to wrap “long” running commands so I can get desktop notifications when the’ve completed. This can be done on any shell really since I just uses aliases to achieve the desired effect. This is an example implementation:

NOTIFY_CMDS=(grunt npm bower git make ant python)

run_cmd() {
    $@
    CMD=`echo $@ | tr -d '\r'`
    RETURN =$?
    if [ $RETURN == 0 ]
    then
        notify-send -t=5000 "Shell" "[$CMD], finished"
    else
        notify-send -t=5000 "Shell" "[$CMD], finished with failure"
    fi
    return $RETURN
}

for CMD in "${NOTIFY_CMDS[@]}"
do
    alias $CMD="run_cmd $CMD"
done

The above makes it so that the commands in the variable NOTIFY_CMDS will be aliased to run through the run_cmd function that can then check the return code and in my case use libnotify through notify-send to show the notifications on my desktop for those commands and the status of how they exited.

So why do I find the notification mechanism above so useful ? Well I like to start those long running tasks and then move onto something else while things are building, compiling or copying and not have to constantly come back to the window to see if the command has completed. With this notification I know I’ll get a little popup that I can look at quickly and know the result right then and there without having to switch back to another window.

Pseudo machine learning

2016-10-09T00:00:00+00:00

Been reading up on a lot of machine learning python tool kits and how exactly one can apply data science to everyday scenarios. While doing so I’ve been wondering how much of these machine learning algorithms can be written in a less mathematical way and instead by analyzing the problem at hand and attempting to devise an algorithm to do some data classification but just devising a data structure (model) that can capture from a given data sample what makes something fall into category A or B.

For this specific experiment I’m going to use the data set from Amazon fine food reviews from Kaggle which has about 500K reviews from amazon including the review score given by the person who wrote the review. What I’ll attempt to do is to create a model that can take lets say 250K worth of reviews and figure out what set of words (ngrams) make for a high score of 5 vs a low score of 1 and then apply this to all other 250K reviews and see how close the model comes to predicting the reviews score based solely on the words used. The idea here is that we’d end up with a model that basically understands sentiment of what the person was writing and if they actually liked or disliked the thing they were reviewing.

Lets get started by downloading the data from the Kaggle source above and for this example we’re going to use the sqlite database provided. I created a quick virtualenv like so:

virtualenv env -p python3

Then proceeded to write a simple script called analysis.py, like so:

import sqlite3

def main():
    conn = sqlite3.connect('amazon-fine-foods/database.sqlite')
    cursor = conn.cursor()

    results = cursor.execute('select score, summary, text from reviews limit 5')

    for result in results:
        print(result)

if __name__ == '__main__':
    main()

Which produces the output

(5, 'Good Quality Dog Food', 'I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than  most.')
(1, 'Not as Advertised', 'Product arrived labeled as Jumbo Salted Peanuts...the peanuts were actually small sized unsalted. Not sure if this was an error or if the vendor intended to represent the product as "Jumbo".')
(4, '"Delight" says it all', 'This is a confection that has been around a few centuries.  It is a light, pillowy citrus gelatin with nuts - in this case Filberts. And it is cut into tiny squares and then liberally coated with powdered sugar.  And it is a tiny mouthful of heaven.  Not too chewy, and very flavorful.  I highly recommend this yummy treat.  If you are familiar with the story of C.S. Lewis\' "The Lion, The Witch, and The Wardrobe" - this is the treat that seduces Edmund into selling out his Brother and Sisters to the Witch.')
(2, 'Cough Medicine', 'If you are looking for the secret ingredient in Robitussin I believe I have found it.  I got this in addition to the Root Beer Extract I ordered (which was good) and made some cherry soda.  The flavor is very medicinal.')
(5, 'Great taffy', 'Great taffy at a great price.  There was a wide assortment of yummy taffy.  Delivery was very quick.  If your a taffy lover, this is a deal.')

Now the way I’d break this down is that for each review body I’d assume there is a language used that highly influences the score given to the review. Things such as stating I disliked the way or I truly enjoyed would be indicative of a positive or negative review and the sum of all of these kind of statements would have a high correlation to the score given by the user. That means we want to create a structure that can capture the average score associated with the use of certain groupings of words. Those groupings are known as n-grams and you can read more on that here and we can easily break up those reviews into n-grams like so:

def ngramize(text, minimum=2, maximum=6):
    words = text.split()
    result = []

    for ngrams in range(minimum, maximum):
        tuples = zip(*[list(words[index:]) for index in range(ngrams)])
        result += [ ' '.join(ngram) for ngram in tuples]

    return result

The above simply breaks up the text provided into words and then iterates through the resulting elements in order constructing n-grams of length minimum to maximum along the way.

Now that we have our n-grams I’ve simply decided I only care about n-grams of length 2 to 6 for the time being and we’ll want to now use the existing score to calculate the average score associated with each n-gram’s usage through out the various reviews we use as our training data. Here’s the quick and dirty approach I came up with:

import json
import sqlite3

def ngramize(text, minimum=2, maximum=6):
    words = text.split()
    result = []

    for ngrams in range(minimum, maximum):
        tuples = zip(*[list(words[index:]) for index in range(ngrams)])
        result += [ ' '.join(ngram) for ngram in tuples]

    return result

def main():
    conn = sqlite3.connect('amazon-fine-foods/database.sqlite')
    cursor = conn.cursor()

    results = cursor.execute('select score, summary, text from reviews limit 5')

    # this will store each ngram the average score associated with said ngram's
    # usage
    ngram_scores = {}
    total = 0

    for score, summary, review in results:
        total += 1
        ngrams = ngramize(review)

        for ngram in ngrams:
            if ngram not in ngram_scores:
                ngram_scores[ngram] = {
                    'score': score,
                    'appearances': 1
                }

            else:
                ngram_scores[ngram]['score'] += score
                ngram_scores[ngram]['appearances'] += 1

    for ngram in ngram_scores.keys():
        ngram_scores[ngram] = ngram_scores[ngram]['score'] / ngram_scores[ngram]['appearances']

    print(json.dumps(ngram_scores, indent=2))

if __name__ == "__main__":
    main()

Running the above produces a lengthy output of n-grams to average score values and for just 5 reviews has over 900 n-grams associated. We’ve started seeing a few silly things in the output such as n-grams across sentences which obviously don’t make sense:

  "good product.I love these chips": 5.0,

as well as HTML tags in the middle of the n-grams constructed:

  "it's ready.<br /><br />Tastes": 5.0,

So lets use Beautiful Soup to extract the HTML tags out of our way and instead of calculating n-grams for the whole text lets do so at the sentence level by splitting our text on the final period of a sentence. Here’s how things look now in terms of the python solution:

from bs4 import BeautifulSoup

import json
import sqlite3

def ngramize(text, minimum=2, maximum=6):

    # remove HTML tags
    soup = BeautifulSoup(text, 'html.parser')
    text = soup.get_text()

    result = []
    # process sentence by sentence
    for sentence in text.split('.'):
        words = sentence.split()

        for ngrams in range(minimum, maximum):
            tuples = zip(*[list(words[index:]) for index in range(ngrams)])
            result += [ ' '.join(ngram) for ngram in tuples]

    return result

def main():
    conn = sqlite3.connect('amazon-fine-foods/database.sqlite')
    cursor = conn.cursor()

    results = cursor.execute('select score, summary, text from reviews limit 5')

    # this will store each ngram the average score associated with said ngram's
    # usage
    ngram_scores = {}
    total = 0

    for score, summary, review in results:
        total += 1
        ngrams = ngramize(review)

        for ngram in ngrams:
            if ngram not in ngram_scores:
                ngram_scores[ngram] = {
                    'score': score,
                    'appearances': 1
                }

            else:
                ngram_scores[ngram]['score'] += score
                ngram_scores[ngram]['appearances'] += 1

    for ngram in ngram_scores.keys():
        ngram_scores[ngram] = ngram_scores[ngram]['score'] / ngram_scores[ngram]['appearances']

    print(json.dumps(ngram_scores, indent=2))
    print(len(ngram_scores))

if __name__ == "__main__":
    main()

Running the analysis script against the first 100 reviews takes 1.3s and for 1000 takes 4s. I’m sure we could spend a ton of time optimizing things but right now I’ll take correctness over speed and will dive back into any performance gains later.

Now that we have a mapping of n-grams to an average score associated with said n-gram we can start to verify how well this model works. The first idea that occurred to me is to see exactly how close the model predicts the scores for the exact same reviews it was using for training. I simply calculated the score by averaging the value of all of the n-grams found within a review and spitting out the number side by side with the actual score. The below output is from using the first 1000 reviews as training data and simply applying the model to the first 20 reviews:

predicted: 4.87, actual: 5.00
predicted: 1.21, actual: 1.00
predicted: 4.01, actual: 4.00
predicted: 2.48, actual: 2.00
predicted: 4.92, actual: 5.00
predicted: 4.00, actual: 4.00
predicted: 4.89, actual: 5.00
predicted: 4.83, actual: 5.00
predicted: 4.94, actual: 5.00
predicted: 4.91, actual: 5.00
predicted: 4.88, actual: 5.00
predicted: 4.89, actual: 5.00
predicted: 1.50, actual: 1.00
predicted: 4.07, actual: 4.00
predicted: 4.90, actual: 5.00
predicted: 4.78, actual: 5.00
predicted: 2.41, actual: 2.00
predicted: 4.84, actual: 5.00
predicted: 4.92, actual: 5.00
predicted: 4.90, actual: 5.00

So far this is very promising but lets not forget this is the model running against the exact same data it was trained on.

Now we’ve reached the point where we want to simply train with the first 10K of reviews and then run the model against the next 10K of reviews and calculate the root mean square error (RMSE) of our prediction.

The new python script looks like so:

from bs4 import BeautifulSoup
from numpy import mean, sqrt, square

import json
import sqlite3

def ngramize(text, minimum=2, maximum=6):

    # remove HTML tags
    soup = BeautifulSoup(text, 'html.parser')
    text = soup.get_text()

    result = []
    # process sentence by sentence
    for sentence in text.split('.'):
        words = sentence.split()

        for ngrams in range(minimum, maximum):
            tuples = zip(*[list(words[index:]) for index in range(ngrams)])
            result += [ ' '.join(ngram) for ngram in tuples]

    return result

def main():
    conn = sqlite3.connect('amazon-fine-foods/database.sqlite')
    cursor = conn.cursor()

    results = cursor.execute('select score, summary, text from reviews limit 20000')

    # this will store each ngram the average score associated with said ngram's
    # usage
    ngram_scores = {}
    total = 0

    for score, summary, review in results:
        total += 1

        if total > 10000:
            break

        ngrams = ngramize(review)

        for ngram in ngrams:
            if ngram not in ngram_scores:
                ngram_scores[ngram] = {
                    'score': score,
                    'appearances': 1
                }

            else:
                ngram_scores[ngram]['score'] += score
                ngram_scores[ngram]['appearances'] += 1

    for ngram in ngram_scores.keys():
        ngram_scores[ngram] = ngram_scores[ngram]['score'] / ngram_scores[ngram]['appearances']

    # lets use the n-gram scores to calculate the predicted score for an
    # existing review to see just how close we can get

    errors = []

    for score, summary, review in results:
        ngrams = ngramize(review)
        predicted_score = 0.0
        ngrams_found = 0

        for ngram in ngrams:
            if ngram in ngram_scores.keys():
                predicted_score += ngram_scores[ngram]
                ngrams_found += 1


        predicted_score = predicted_score / ngrams_found
        #print('actual: %2.2f prediction: %2.2f' % (score, predicted_score))

        errors.append(score - predicted_score)

    print('RMSE: %2.2f' % sqrt(mean(square(errors))))

if __name__ == "__main__":
    main()

The output from this script gave us the following:

RMSE: 1.16

Which means on average we’re off by 1.16 with our prediction of the review score as provided by the person who wrote the review. Now that isn’t horrible on a scale of 1-5 we’re only off by 23% with a super simple model we wrote in less than an hour. There a few things we can try to make this work better and one of them is to use the notion of stop words to avoid creating n-grams composed solely of stop words since those are “fluff” in the spoken language. I won’t go into those just yet as I want to see how this experiment does as we train on a bigger chunk of the data set. So I trained the model against the first 100K of reviews and then predicted the following 100K and we got:

RMSE: 1.04

Which means that with a larger training set we’re seeing the model behave much better in terms of predictions. Another thing to note is while the script runs its building an enormous hash table for all of n-grams found and it can grow to several gigabytes of space. So for my last experiment I’ll try to train with the first 250K worth of reviews and see how the RMSE looks for predicting the remaining 250K of reviews. The process hit 3.7GB of usage and after a little over 5 minutes we got got the following:

RMSE: 1.04

Which means we’d be able to predict the overall we’re able to predict a review score with an error of about 21% and we used a method that was very easy to explain and follow along without any unnecessary complexities of complicated machine learning algorithms.

Another idea I had was to simply train the model with the first 100K of reviews and then see if I wrote my own negative/positive review and see if the score I would get would reflect what I had written. Therefore providing me with a pretty good sentiment analysis tool. So I restructured the existing script so I could train separately from predicting and also be able to save the model after training so I could then use to analyze text and be provided with a value from 1 (negative sentiment) to 5 (positive sentiment). This is what the tool looks like now:

#!/usr/bin/env python3

import os
import pickle
import sqlite3

import click
import requests

from bs4 import BeautifulSoup
from numpy import mean, sqrt, square


def ngramize(text, minimum=2, maximum=6):

    # remove HTML tags
    soup = BeautifulSoup(text, 'html.parser')
    text = soup.get_text()

    result = []
    # process sentence by sentence
    for sentence in text.split('.'):
        words = sentence.split()

        for ngrams in range(minimum, maximum):
            tuples = zip(*[words[index:] for index in range(ngrams)])
            result += [' '.join(ngram) for ngram in tuples]

    return result

@click.group()
def cli():
    pass

@cli.command()
@click.option('--limit', '-l',
              type=int,
              help='how many entries from the amazon fine foods sqlite db to ' +
                   'train with')
@click.option('--amazon-reviews-path',
              default='amazon-fine-foods',
              help='the path to the directory containing the amazon fine ' +
                   'foods review database.sqlite file')
@click.option('--output-model',
              default='model.pickle',
              help='output filename to save the trained model')
def train(limit, amazon_reviews_path, output_model):
    conn = sqlite3.connect(os.path.join(amazon_reviews_path, 'database.sqlite'))
    cursor = conn.cursor()

    results = cursor.execute('select score, text from reviews limit %d' % limit)

    # this will store each ngram the average score associated with said ngram's
    # usage
    ngram_scores = {}
    total = 0

    for score, review in results:
        total += 1

        ngrams = ngramize(review)

        for ngram in ngrams:

            if ngram not in ngram_scores:
                ngram_scores[ngram] = {
                    'score': score,
                    'appearances': 1
                }

            else:
                ngram_scores[ngram]['score'] += score
                ngram_scores[ngram]['appearances'] += 1

    for ngram in ngram_scores.keys():
        ngram_scores[ngram] = (ngram_scores[ngram]['score'] /
                               ngram_scores[ngram]['appearances'])

    with open(output_model, 'wb') as output:
        pickle.dump(ngram_scores, output, pickle.HIGHEST_PROTOCOL)

@cli.command()
@click.option('--skip', '-s',
              default=1000,
              type=int,
              help='how many entries to skip ahead, ie more than the ones ' +
              'you trained against')
@click.option('--limit', '-l',
              default=1000,
              type=int,
              help='how many entries the --skip value to predict scores for')
@click.option('--amazon-reviews-path',
              default='amazon-fine-foods',
              help='the path to the directory containing the amazon fine ' +
                   'foods review database.sqlite file')
@click.option('--input-model',
              default='model.pickle',
              help='input filename of the previously trained model')
def predict(skip, limit, amazon_reviews_path, input_model):

    with open(input_model, 'rb') as input:
        ngram_scores = pickle.load(input)

    conn = sqlite3.connect(os.path.join(amazon_reviews_path, 'database.sqlite'))
    cursor = conn.cursor()

    results = cursor.execute('select score, text from reviews limit %d' %
                             (skip + limit))

    # lets use the n-gram scores to calculate the predicted score for an
    # existing review to see just how close we can get
    errors = []
    total = 0
    for score, review in results:
        total += 1

        if total < skip:
            continue

        ngrams = ngramize(review)
        predicted_score = 0.0
        ngrams_found = 0

        for ngram in ngrams:

            if ngram in ngram_scores.keys():
                predicted_score += ngram_scores[ngram]
                ngrams_found += 1

        if ngrams_found != 0:
            predicted_score = predicted_score / ngrams_found

        #print('actual: %2.2f prediction: %2.2f' % (score, predicted_score))
        errors.append(score - predicted_score)

    print('RMSE: %2.2f' % sqrt(mean(square(errors))))

@cli.command()
@click.argument('text')
@click.option('--input-model',
              default='model.pickle',
              help='input filename of the previously trained model')
def analyze(text, input_model):

    with open(input_model, 'rb') as input:
        ngram_scores = pickle.load(input)

    ngrams = ngramize(text)
    ngrams_found = 0
    score = 0

    for ngram in ngrams:

        if ngram in ngram_scores.keys():
            score += ngram_scores[ngram]
            ngrams_found += 1

    if ngrams_found != 0:
        score = score / ngrams_found

    print('score: %s' % score)


if __name__ == "__main__":
    cli()

The script grew pretty significantly in size but now we can actually train the model and subsequently test it against different ranges in the review database as well as against a string of text. So lets first train against the first 100K reviews:

./senti.py -l 100000

Now we can run against the next 1000 reviews in the database and see how high or low the RMSE is:

> ./senti.py predict -l 100000 -s 1000
RMSE: 0.75

Nothing new about these last few uses but now comes the more interesting part in which we see how well the model can be used to analyze a new arbitrary piece of text in terms of a positive or negative sentiment:

> ./senti.py analyze 'I like this scripting tool'
score: 4.236562683156721

Now what if express that we really like this scripting tool:

> ./senti.py analyze 'I really like this scripting tool'
score: 4.246059117643124

The score was slightly higher and what if we love this scripting tool:

> ./senti.py analyze 'I love this scripting tool'
score: 4.637583998372766

How well does negative sentiment detection work:

> ./senti.py analyze 'I do not like this scripting tool'
score: 3.639240038501364

That wasn’t as low as we’d like but lets see if expressing a strong dislike results in a lower value:

> ./senti.py analyze 'I really do not like this scripting tool'
score: 3.8001524620308818

Lower but no where near where I’d want it to be so I just started expressing in a harsher manner how negatively I felt about this tool:

> ./senti.py analyze 'I hate this scripting tool'
score: 2.586253369272237

I’m now starting to see that as well as this tool has been working its having a hard time when predicting scores for negative sentiments. Experimenting a bit I’m seeing that smaller text is behaving badly and my theory is that small n-grams such as “I really” “this scripting” and others that don’t express actual sentiment on their own would have a negative effect on the prediction since the sentence “I really hate” vs “I really like” would be skewed by the usage of “I really” in the various reviews. So I decided to only train the model on n-grams with a minimum length of 3 and maximum length of 6 and the resulting model is behaving much better:

> ./senti.py analyze 'I like this'
score: 4.549878345498784
> ./senti.py analyze 'I love this'
score: 4.7824878387769285
> ./senti.py analyze 'I really love this'
score: 4.54860095976375
> ./senti.py analyze 'I do not like this'
score: 3.118788898794749
> ./senti.py analyze 'I hate this'
score: 1.0

Which is behaving quite a bit better than before and now I’m curious if the model does a better job of predicting the scores for the next 100K of reviews:

> ./senti.py predict -l 100000 -s 100000
RMSE: 1.01

So we actually improved even if just a tiny bit on the already existing RMSE of 1.04.

This was simply an experiment to see how well this whole idea would work and I’m surprised I was able to get any usefulness out of something I wrote up in a few hours while documenting and experimenting with the approach as I went along. If you attempt to use the code here make sure to install the following requirements:

beautifulsoup4==4.5.1
click==6.6
numpy==1.11.2
requests==2.11.1

Which you can do by issuing pip install xyz=1.2.3 for each of those lines and the above script should just work granted you’ve downloaded the Amazon fine food reviews data set.

iPython Notebooks Rock!

2014-07-06T00:00:00+00:00

Decided to write a quick and short post on using ipython notebooks as I came across this just today and found it to be extremely useful. If you already used ipython then I think ipython notebooks will be super quick to pick up. The idea behind the ipython notebooks is that they’re an ipython session in the browser that you can save and share with others. Then within that session you can easily edit the existing session so you can undo parts of it that you didn’t want to share and the final product is a clean session of python code that you were attempting to show someone else how something works.

The nice thing about ipython notebook is that if you’re already using ipython then you already have it installed and ready to go:

> ipython notebook
...

The previous command will open your browser on an ipython session and you can start writing into the Web UI the same expressions you’d do in ipython on the command line like so:

import random
data = [ random.randint(1,1000) for _ in range(0, 10) ]
print data

Once you’ve filled in a slot on the screen you can hit Ctrl+Enter or go to the top and press the play button to have your code intepreted and the result rendered in the ipytho notebook. For the above you’d have something like so:

Now you can save the above session and share it with a colleague with ease. Just click on the File -> Download as and pick your option. You can start to see just how useful this is when you share your ipynb file with another pythonista and actually share a piece of executable code that can be used to iterate on an idea within the ipython shell.

For a more interesting example make sure to pip install the vincent library and then have a look at the vincent session from within your own ipython notebook session by starting that session in a directory that contains the ipynb file. If the previous loading worked correctly you’ll be looking at a similar session to the following one:

Pythonbrew - the best way to install and manage python on your system

2013-06-20T00:00:00+00:00

This will be a short post but I just wanted to make sure to write up a simple post on how exactly pythonbrew works and how much time and effort it can save you when managing different python installs on any given OS.

So pythonbrew is quite simple to install, you can start by having a look at the github project here, but installing it is as easy as:

curl -kL http://xrl.us/pythonbrewinstall | bash

and then adding the following to your bashrc:

[[ -s $HOME/.pythonbrew/etc/bashrc ]] && source $HOME/.pythonbrew/etc/bashrc

With this you now have pythonbrew ready to go and you can check what are the currently available version of python you can install with:

>pythonbrew list -k # Pythons Python-1.5.2 Python-1.6.1 Python-2.0.1 Python-2.1.3 Python-2.2.3 Python-2.3.7 Python-2.4.6 Python-2.5.6 Python-2.6.8 Python-2.7.3 Python-3.0.1 Python-3.1.4 Python-3.2.3 Python-3.3.0

Now isn’t that impressive that you can actually install a python version from 1.5.2 to the very latest bleeding 3.3.0 version ? So installing any of those versions is as easy as:

> pythonbrew install Python-2.4.6 Downloading Python-2.4.6.tgz as /home/rlgomes/.pythonbrew/dists/Python-2.4.6.tgz ######################################################################## 100.0% Extracting Python-2.4.6.tgz into /home/rlgomes/.pythonbrew/build/Python-2.4.6 This could take a while. You can run the following command on another shell to track the status: tail -f "/home/rlgomes/.pythonbrew/log/build.log" Patching Python-2.4.6 Installing Python-2.4.6 into /home/rlgomes/.pythonbrew/pythons/Python-2.4.6 pythonbrew list Downloading distribute_setup.py as /home/rlgomes/.pythonbrew/dists/distribute_setup.py ######################################################################## 100.0% Installing distribute into /home/rlgomes/.pythonbrew/pythons/Python-2.4.6 Installing pip into /home/rlgomes/.pythonbrew/pythons/Python-2.4.6 Installed Python-2.4.6 successfully. Run the following command to switch to Python-2.4.6. pythonbrew switch 2.4.6

Checking what versions are currently available is also very easy:

> pythonbrew list # pythonbrew pythons Python-2.4.6 Python-2.7.2 Python-2.7.3 (\*) Python-3.2 Python-3.3.0

As you can see the above is stating that version 2.7.3 is currently in use and for you to switch the current terminal session over to another version you can do this like so:

> pythonbrew use 2.4.6 Using `Python-2.4.6

By this point you realize this is just to darn easy and makes using multiple python versions on the same host quite easy. The other nice thing is that pythonbrew already has virtual environments built in and you can see the rest of those commands right in the README provided in github.

You should really start using pythonbrew from now on as it makes installing and changing python versions quite easy.

From Prototype To Production

2012-10-21T00:00:00+00:00

START 12:01 October 14th, 2012

In the sequence of my previous post I’d like to give a more real world example of how a modern software engineer should be able to write and test code using the vast amount of tools/frameworks available to get the job done in timely fashion while producing high quality code.

To start of course we’ll need some sort of a project to build while writing about how to solve the issues we come across. So for this writing I will attempt to create a key/value store in Python that can be used to mock out a real key/value store such as Redis, LevelDB or Amazon’s Dynamo. The current requirements will be:

Support 3 simple API calls:
- GET key
- SET key value
- DEL key
The protocol used to communicate should be human readable and really efficient, just like Redis’s communication protocol is.

As part of this writing I will track in real time how many hours I’m spending on this project while writing the post so that at the end you can get an idea of how little overhead writing tests and documentation while developing really has.

STOP: 12:11 on October 14th, 2012

START 14:57 on October 14th, 2012

So the first thing we’ll have to do is to define the exact protocol we’re going to use in a way that can be easily consumed by others who will attempt to talk the same protocol or create clients to talk this custom protocol. We mentioned earlier we’d be using a protocol similar to what Redis uses. You can read up on Redis’s communication protocol here and we’ll be greatly simplifying this protocol for this writing like so:

\*[number of arguments] CR LF
[command name] CR LF
[argument 1] CR LF
[argument 2] CR LF

Which means that the first thing sent is the indication of how many arguments will follow separted by a carriage return and line feed character (\r\n). Then each of the arguments a single line termianted by \r\n.

So sending a SET request for the key A to set it to the value 100 would look like so on the wire:

\*3\r\nSET\r\nA\r\n100\r\n

Replies will also be very similar to the way Redis deals with this type of thing and we’ll basically start a response with a + on success followed by a single line response, or we’ll start with - if there was an error followed by a single line with the error message.

We now have to decide what we’ll use to build the protocol server on and currently one of the most flexible and best performant ones in the Python world is Twisted which can be used to easily create your own custom protocol or better yet used to easily build your own HTTP, FTP, etc server in minutes. I had to brush up on my knowledge of Twisted and how to create my own custom protocol and after reading through the documentation for some 15 minutes I found that what I wanted to use was the LineReceiver implementation to build my protocol on a per line reading of any connection. The first example that you may be able to put together using the LineReceiver class may look like this:

from twisted.internet.protocol import Factory
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor

class Answer(LineReceiver):

    answers = {'How are you?': 'Fine', None : "I don't know what you mean"}

    def lineReceived(self, line):
        if self.answers.has_key(line):
            self.sendLine(self.answers[line])
        else:
            self.sendLine(self.answers[None])

class AnswerFactory(Factory):

    def __init__(self):
        pass

    def buildProtocol(self, addr):
        return Answer()


reactor.listenTCP(9999, AnswerFactory())
reactor.run()

This of course is just an example of how you can use twisted to make a line reading protocol handler. Now lets actually use this to read our new custom protocol which is a multi-line protocol that needs to reconstruct each command from the multiple lines that it is broken up into on the wire.

Now even before we start writing the actual server handler code lets first write up a few very simple unit test that we can use to verify that we have a working set and get commands:

import socket
import unittest

class ProtocolTest(unittest.TestCase):

    def setUp(self):
        self._connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self._connection.connect(('localhost', 9999))

    def tearDown(self):
        self._connection.close()

    def _send_cmd(self, cmd, *args):
        cmd_string = '*%d\r\n' % (len(args) + 1)
        cmd_string += '%s\r\n' % cmd

        for arg in args:
            arg = str(arg)
            cmd_string += '%s\r\n' % arg

        self._connection.sendall(cmd_string)

        return self._read_response()

    def _read_response(self):
        status = self._connection.recv(1)

        response = ''
        read = self._connection.recv(1)
        while read != '\n':
            response += read
            read = self._connection.recv(1)

        if status == '+':
            # successful response with a single line response
           return response
        else:
            # unsucceful response with error message
            raise Exception(response)

    def test_basic_set_and_get(self):
        self._send_cmd('SET', 'A', 100)
        resp = self._send_cmd('GET', 'A')
        assert resp == '100', 'expected 100 got %s' % resp
        
if __name__ == '__main__':
    unittest.main()

There is quite a lot of test code displayed there but that’s because we had to create those helper methods for sending commands and receiving responses. The actually test itself is just 3 lines to send a SET command verify with a GET command that the current server recorded our 100 value correctly.

We’re now back to the server code because we now need to restructure our CommandReader so that it can actually read each command line by line and translate that into the right underlying set/get command. In a first approach at writing our CommandReader what we need to do is to make this LineReceiver act as state machine that transitions between commands in a very well defined manner. Every line that starts with an asterisk is expected to be a new command that is consumed till all arguments are read and the command is dispatched and the response is sent back to the client awaiting a response.

A first approach may look like so:

class CommandReader(LineReceiver):
    """
    state machine comand reader 
    """

    def __init__(self):
        self.start_command = False
        self.reading_argument_size = False
        self.expected_arguments = -1
        self.args = []
       
    def lineReceived(self, line):

        if line.startswith('*'):
            # new command starting
            if self.start_command:
                self.sendLine('-unexpected start of new command\r')
            self.start_command = True
            self.expected_arguments = int(line[1:])
            self.args = []

        if self.expected_arguments == 0:
            # command complete lets dispatch and return OK
            self.sendLine('+OK')
            self.start_command = False

        if self.start_command:
            # we're still reading the arguments
            line = line.strip()
            self.args.append(line)
            self.expected_arguments -= 1

Of course if we run our test against this server it thinks it has stored data and will fail to retrieve the desired data because we’re always responding with ‘OK’ as you can see here:

> python tests/protocol.py F ====================================================================== FAIL: test_basic_set_and_get (__main__.ProtocolTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/protocol.py", line 45, in test_basic_set_and_get assert resp == '100', 'expected 100 got %s' % resp AssertionError: expected 100 got OK ---------------------------------------------------------------------- Ran 1 test in 0.002s FAILED (failures=1)

Lets take our current implementation and make the CommandReceiver smart enough to look up the correctly handler based on the command name supplied and return the response that the handling function returns.

After some debugging and restructuring the code a bit as I made changes and re-ran the tests I realized that the checking for a complete command should always be done after processing each line. Then I also figured that socket.sendLine already adds the newline character at the end of each response. Once I fixed the code up like so:

class CommandReader(LineReceiver):
    """
    state machine comand reader 
    """

    def __init__(self, cmdhandler):
        self._cmdhandler = cmdhandler
        self._cmd_names = dir(cmdhandler)

        self.start_command = False
        self.reading_argument_size = False
        self.expected_arguments = -1
        self.args = []
       
    def lineReceived(self, line):
        if line.startswith('*'):
            # new command starting
            if self.start_command:
                self.sendLine('-unexpected start of new command\r')

            self.start_command = True
            self.expected_arguments = int(line[1:])
            self.args = []
            return 
        
        if self.start_command:
            # we're still reading the arguments
            line = line.strip()
            self.args.append(line)
            self.expected_arguments -= 1

        if self.expected_arguments == 0:
            # lookup the method handler
            command = self.args[0]

            # remove the command name from the arguments
            self.args = self.args[1:]

            if command in self._cmd_names:
                func = getattr(self._cmdhandler, command)
                result = func(*self.args)
                self.sendLine('+%s' % result)
            else:
                self.sendLine('-unknown command %s' % command)

            self.start_command = False

Our unit test now passes successfully, like this:

> python tests/protocol.py . ---------------------------------------------------------------------- Ran 1 test in 0.002s OK

Now I have a working prototype that can actually do set and get requests and save that information into memory at runtime. At this point its 16:30 on October the 14th, 2012 and with writing the unit test and writing the code I’ve spent just a little over an hour and a half to have a working prototype that could be used by a dependent service to start integrating against.

What we’ll focus on next is using tools such as pylint to identify problems in the code as well as using setuptools to create a setup script that can be used by anyone to easily install this service and start it for others to integrate with while features are being added to the base source.

STOP 16:32 on October 14th, 2012

START 18:59 on October 14th, 2012

So in order to share our code with others we’ll have to create a setup script that can be used to easily install and startup the service as well as being able to upgrade your current installation as further updates are made to the code base. For this specific code being written we’ll create a simple setup.py file like so:

from setuptools import setup, find_packages

setup (
    name='kvs',
    version='0.0.1',
    author='Rodney Gomes',
    author_email='rodneygomes@gmail.com',
    url='',
    test_suite="tests",
    keywords = ['keyvalue', 'storage'],
    py_modules = [],
    scripts = ['kvsserver.py'],

    license='Apache 2.0 License',
    description='simple key value store',
    long_description=open('README.md').read(),
    packages = find_packages(exclude='tests'),
    install_requires = [ 
                        'twisted',
                       ],

    entry_points = {
        'console_scripts' : [
            'kvs_start = kvsserver:main',
        ],                                                                      
    },                
)

With that we can now check this code in and anyone who wants to run your service can easily do the following on the command line with Python 2.7 installed:

> git clone https://github.com/rlgomes/kvs.git > git checkout v0.0.1 > python setup.py install running install ... Finished processing dependencies for kvs==0.0.1

Now your service can easily be started by using the ‘kvs_start’ script that should now be in your path.

Before we proceed to start adding more features and tests to our code lets introduce the notion of a code style checker and static code analyzer called pylint and how we can use it to make sure our code is clean and lean and little less prone to errors. Using pylint is super easy as you can install the python package with a quick ‘pip install pylint’ and then you can call it on any code base like so:

> pylint kvsserver.py No config file found, using default configuration \************* Module kvsserver C: 1,0: Missing docstring W: 7,0:CommandReader: Method 'rawDataReceived' is abstract in class 'LineReceiver' but is not overridden C: 21,4:CommandReader.lineReceived: Invalid name "lineReceived" for type method (should match [a-z_][a-z0-9_]{2,30}$) C: 54,0:CommandReaderFactory: Missing docstring C: 59,4:CommandReaderFactory.buildProtocol: Invalid name "buildProtocol" for type method (should match [a-z_][a-z0-9_]{2,30}$) C: 62,0:CommandHandler: Missing docstring C: 67,4:CommandHandler.set: Missing docstring C: 71,4:CommandHandler.get: Missing docstring C: 76,0:main: Missing docstring E: 78,4:main: Module 'twisted.internet.reactor' has no 'listenTCP' member E: 79,4:main: Module 'twisted.internet.reactor' has no 'run' member ... other output ommited ... Your code has been rated at 6.27/10

There is quite a bit of output that pylint supplies the most important parts are shown above. We can quickly see that we’re missing quite a few docstrings and then there are a few things we can ignore such as the missing members that is obviously just pylint not finding the imports correctly. As with any tool pylint is intended to point you in the direction of a problem and you ultimately have to make the decision to fix something or leave it be and use for example in this case a docstring to tell pylint to be quiet. The score given to your code is an interesting way of showing developers if their code is up to par with how Python code should be written and maintained.

Lets add those docstrings and silence the missing member functions that we know are in fact there. With a subsequent rerun of pylint I now have a score of 7.45 which is a pretty decent score. Something like pylint should be used in order to make sure that the code quality doesn’t degrade drastically with time and that certain levels of code quality and proper code writing are maintained across the team.

We’re now at a stage where others are already able to install and use our code and need to continue adding features to our existing service while making sure that with each checkin we don’t break any of the older functionality and yet are able to quickly introduce new features or bug fixes.

STOP 20:01 on October 14th, 2012

START 20:22 on October 20th, 2012

At this point I decided to restructure the code a bit by creating a kvs package in which the CommandHandler logic into its own module. That way I can continue development on the way we’re storing/retrieving data without having to muck around in the kvsserver module. While doing this I also created a few more test to verify the set, get and new del operation all work correctly.

We now have the full API available with a few additional unit tests that verify the various use cases for set/get and delete operations. I also spent sometime and created a very simple set of performance tests to have an idea of how well this whole thing performs. To create the basic performance tests I first created a BaseTest test case to build that had the earlier used send command to be use to easily send and receive data from the server and then I built the following very simplistic performance test:

ITERATIONS=20000

class PerformanceTest(BaseTest):

    def test_1st_set_small_key_performance(self):
        """
        """
        start = time.time()
        for index in range(0, ITERATIONS):
            self.send('SET', 'key-%d' % index, 'tiny little value')
        elapsed = time.time() - start
        print('SET %f ops/sec' % (ITERATIONS/elapsed))

    def test_2nd_get_small_key_performance(self):
        """
        """
        start = time.time()
        for index in range(0, ITERATIONS):
            self.send('GET', 'key-%d' % index)
        elapsed = time.time() - start
        print('GET %f ops/sec' % (ITERATIONS/elapsed))

    def test_3rd_del_small_key_performance(self):
        """
        """
        start = time.time()
        for index in range(0, ITERATIONS):
            self.send('DEl', 'key-%d' % index)
        elapsed = time.time() - start
        print('DEL %f ops/sec' % (ITERATIONS/elapsed))

The performance numbers were above my expectations, as I was expecting a couple of thousand operations per second but got:

SET 7829.786726 ops/sec GET 6993.390858 ops/sec DEL 7976.453482 ops/sec

I was satisfied with the single threaded performance of the kvs store at this point and want on making the code easier to read & write and so I spent a little time restructuring the CommandReader class to be a bit smarter in terms of how we basically parse each command by switching the lineReceived method implementation at run time. I also fixed up the base test class to be more specific on the SET/GET & DEL methods being used to talk to the server. Here’s what the new CommandReader looks like:

class CommandReader(LineReceiver):
    """
    state machine comand reader 
    """

    def __init__(self, cmdhandler):
        self._cmdhandler = cmdhandler
        self._cmd_names = dir(cmdhandler)

        self.start_command = False
        self.reading_argument_size = False
        self.expected_arguments = -1
        self.args = []

        self.lineReceived = self._start_command

    def _start_command(self, line):
        assert line.startswith('*'), 'got %s' % line

        # new command starting
        if self.start_command:
            self.sendLine('-unexpected start of new command\r')

        self.expected_arguments = int(line[1:])
        self.args = []

        self.lineReceived = self._read_command
        self.start_command = True

    def _read_command(self, line):
        line = line.strip()
        self.command = line
        self.expected_arguments -= 1
        self.lineReceived = self._read_arguments

    def _read_arguments(self, line):
        # remove the command name from the arguments
        line = line.strip()
        self.args.append(line)
        self.expected_arguments -= 1

        if self.expected_arguments == 0:
            # lookup the method handler

            if self.command in self._cmd_names:
                func = getattr(self._cmdhandler, self.command)
                try :
                    result = func(*self.args)
                    self.sendLine('+%s' % result)
                except Exception as e:
                    self.sendLine('-%s' % e)
            else:
                self.sendLine('-unknown command %s' % self.command)

            self.start_command = False
            self.lineReceived = self._start_command

The nice thing at this point is that I’m constantly able to change code quite drastically without having to worry if I broke something because the unit tests are able to give me some confidence in the changes I’m making.

STOP 21.15 on October 21st, 2012

START 15:39 on October 21st, 2012

At this point I’d like to take sometime to analyze how much test code I’ve written vs how much real product code I’ve written. I’ll do this in the simplest way possible by just comparing line count:

> wc -l tests/*.py 55 tests/base.py 0 tests/__init__.py 41 tests/performance.py 45 tests/protocol.py 141 total > wc -l kvs/*.py 41 kvs/cmdhandler.py 0 kvs/__init__.py 91 kvs/kvsserver.py 132 total

So right now we actually have a few more lines of test code than we have of actual product code. The thing to realize though is that as we add more API calls to service, the amount of test code won’t grow by as much as it has till this point. Lets really show how this works by adding a few new APIs:

SHUTDOWN
RESET
KEYS key_reg_ex

The SHUTDOWN command is used to basically shutdown the server and the RESET command is used to reset the store back to empty. The KEYS command is a little trickier since it involves returning all of the keys that match the regular expression specified. This will force us to introduce a new return type to the protocol. What we had in terms of protocol return specification till now was:

+ means the operation succeeded and is followed by OK or the value of what you wanted to return
- is used before the error message of an operation that failed
* is used before starting a multi-value response in which the integer after the * is the number of lines to read

With the multi-line response we can now implement the KEYS correctly. Having implemented all of those features we now have a little more product code lines than tests and have a pretty well working key/value store that is being used by others while we make changes and easily reverify our code as we make those changes. here is the lines of test code vs lines of product code comparison:

> wc -l tests/*.py 65 tests/base.py 0 tests/__init__.py 41 tests/performance.py 67 tests/protocol.py 173 total > wc -l kvs/*.py 70 kvs/cmdhandler.py 0 kvs/__init__.py 112 kvs/kvsserver.py 182 total

Now I personally don’t care if I have more product code than test code because to me test code is valuable code that allows me as a developer to actually write code that can be used by others and guarantees my code at least does what I was originally intending.

STOP 16:46 on October 21st, 2012

The interesting thing that I’d like to analyze now is roughly how much time was spent on this little project till now and of that time how much time was spent writing tests vs writing product code.

So the start and stop times tally looks like so:

START	STOP	Description	Duration (minutes)
12:01	12:11	Initial Project Specification	10
14:57	16:32	Prototype Development	95
18:59	20:01	Making beta Version Available	62
20:22	21:15	Restructuring ∧ Performance Testing	53
15:39	16:46	Refactoring Code ∧ Adding new APIs	53

So just after just 4h and 33m of working on this project we have a working service that can be used by others and we’re able to easily and quickly extend this service while making sure to test existing and new features before each and every checkin.

Now there are a few things I should have also worked on but just didn’t feel it would have fitted into the length of the post I was working on writing. The few things I would have focused on next would have included:

Making sure to document the protocol specification with the code in a format that would allow others to easily and quickly write their own clients. This would also make updating the API documentation easier since it resides with the code that implements the API.
Adding more tests that would verify the limits of the API usage such as the max key and value lengths. Not forgetting to test all of the negative scenarios of using the protocol such as invalid integer values, invalid operation names, etc.

I hope that after reading this post you’ll see that you can easily apply the same ideas to any of your projects and allow yourself to be a more efficient engineer and also allow you to produce better code.

The code written during this writing can be cloned from here

The Origin of The Modern Software Engineer

2012-10-13T00:00:00+00:00

In software development these days, what most call “software engineers”, could be better defined as code monkeys. I myself was quite a good code monkey back in the day and can’t blame most for finding themselves in the same situation. What I hope to achieve with this writing is to give you an impression of how we’ve evolved as software engineers over the last 15 years and where this evolution may lead us to. I hope to also introduce a possible future approach on how software development should be done and just maybe save some fellow engineers hours or days of torturous bug hunting.

The Stone Age Engineer

One of my first jobs never had a QA team or quality engineers to work on qualifying and verifying products before we put them out the door. The developers (or stone age engineers) were responsible for taking the features requested by their bosses and basically developing code to fulfill those requirements and testing that said code on the devices at hand. The person writing code that had no formal training in quality assurance or even understood how to make things more testable, was expected to write code and make sure it was ready for production. They were simply expected to produce good quality software in a timely fashion. In this scenario, the daily routine of the stone age engineer consisted of:

writing new code that had to be tested manually by said developer or another developer with which this developer was partnering with.
working on any bugs reported by customers on the code that had been released into the customers hands.
attempting to verify that the current code being released has the new features working correctly as well as all older features working as previously manually tested.

I found myself many times questioning how well I could test an application like that by hand and how consistent I was at even reproducing the same steps from previous “test runs” so that I could at least catch previous issues as well. This is of course what we call today automation and its a word that quite a few people use but very few actually practice it correctly. Time for automation was never scheduled and I had no other choice then to get my code written and tested by hand in order to make deadlines. The unfortunate side effect of such a development process is that days or weeks later we had yet again produced bug 3214 for the 5th time in 3 months because we had no way to easily verify that we were at least not breaking any existing features that were working for customers.

This forces developers to context switch a lot between features they’re currently writing and ones that they wrote quite a while ago. Since the code that was written a while ago never received the required “quality” attention that it should have, making changes in that code can be difficult and bug ridden. This results in very long nights of coding to fix production issues and also makes your code very unmaintainable as most developers never attempt to re-factor code given that they don’t have tests that would allow them to have confidence in making any major changes to the code base.

The Medieval Engineer

I’d consider the past 5 years to be the medieval times of development where developers are now familiar with the idea of writing unit tests and defending themselves from late night coding battles by having the right “armor” at hand when they’re developing code. This new confidence in making changes is still a bit mis-guided as some of you may already know because there are a few things that are not being tested by unit tests that developers will certainly be prone to breaking that need to be fixed after pushes to the integration environment.

During my medieval engineer phase I was introduced to the notion of having one team that develops code while another team that is entirely responsible for qualifying and verifying said code. This didn’t mean that writing unit tests was no longer required, because of course nothing can replace good unit tests. This really meant that there was now a team dedicated to doing all of the other phases of testing which meant the integration testing across services as well as performance testing of the services.

This of course works well when your QA team is a team of engineers that actually understand what they’re role is in the software development process. I say this because in my experience most quality engineers don’t understand that they’re not in a position of blocking developers but instead helping said developers find their problems earlier and being able to reverify quickly that changes have not broken any existing features.

So in the medieval times of development we have developers who write code and unit test it and then ship the code to the quality assurance engineers who may simply do manual black box testing which has to be rerun on each new code push from developers. Once this code has reached some level of quality that the testers feel is sufficient to be pushed to production or made part of an upcoming release, then the QA team gives the thumbs up. Of course the notion of automation was well understood by me at this point and manual testing was avoided with the right tools and frameworks for automating testing so that on each new push of code we would push a button and wait for the results. With this the testers can actually spend time writing new tests for the new features and helping developers diagnose/debug existing bugs in the code.

The Renaissance Engineer

We live in what I like to call the age of the renaissance engineering, because we have acquired a lot of knowledge on how to do testing well and we’ve also learned processes for tracking code development in a way that allows individual developers to feel like they are making a difference on the team while management can easily know when to expect things to ship. You may be aware of some of these processes which include the likes of Agile, Scrum, Waterfall, etc. I’m not a fan of taking one of these methods and following it to the tee since I believe that the process by which many engineers can work together to produce software really depends greatly on the quality of engineers on that team.

The availability of frameworks and tools that can empower the developer to get his/her job done are everywhere these days. From tools that can be used to analyze your code for bad practices or memory leaks to frameworks that can be used to easily graph out the performance behavior of your REST API. We have also started understanding the notion of testing early using continuous integration environments to find problems as soon as possible so they can be fixed sooner than later.

The renaissance engineer is capable of writing code and tests at the same time without feeling that they’re “wasting” time. They use well known tools all the time and experiment with new tools when they hear about such. Some of them even prefer to write their tests before they write their code (i.e. test driven development) and find that their tests to be more valuable than they code they write. This of course is only the case for a few Leonardo Da Vinci type’s who have actually learned how to make themselves better engineers throughout their career and will continue to do so moving forward.

For the majority of software development shops though the renaissance engineer usually finds him/her self in teams with various other developers at different stages of their evolution and sometimes find that the testing responsibility is totally in the hands of a QA team that may or may not be doing the right thing. This really irks the renaissance engineer who would like to see the QA team do things better and at the same times wants to make sure that the development team isn’t a blocker for the QA team in terms of giving them the required testing hooks and or fixing blocking problems first. The renaissance engineer living in today’s day and age, finds him/her self quite frustrated with most of the processes being used and how difficult it is to get others to see that if everyone follows a similar process of writing/testing and pushing code then the whole system works better as a whole.

The Modern Age Engineer

If we put ourselves in the shoes of a software engineer from the future, looking back at the various eras of the software engineering. We are be able to evaluate from all of the failures and success, what approaches really worked in the software development process. The question is now how we each interpret these results and what we feel would actually put us on a faster pace of evolving towards becoming a modern age engineer sooner than later in our careers. The following is my interpretation of how what I believe would be the future of software engineering.

In a future with a modern age engineers one would not have any separation between developers and testers. There wouldn’t even be such a distinction because all members of the team are software engineers capable of writing and understanding code. There would be a single software engineering team responsible for producing any given product. This team consists of engineers who write code to implement features in the product as they also supply the tests that are to be used to verify that the code does in fact behave as expected. The team consists of experts in their own areas of interest, each having expertise in areas such as build & release processes, automation, distributed systems, etc. Of course engineers would be allowed to do cross team work if they had frameworks/tools or expertise that are of use to other teams in the company can benefit from. The notion of someone writing exclusively writing tests would be dead and instead engineers would be responsible for writing their own tests as well as writing integration tests with other engineers that work on dependent services or APIs.

There following would hold true for the whole organization:

Continuous integration environments are setup and used by everyone to build, deploy and test their code on a regular basis and everyone understands how their code is deployed as well as how their tests are executed against their code.
Test are written before the code and the code is written to make those tests pass as they are the contract or specification by which the code should behave.
These same tests should really serve as documentation on how the code is to be used by others. If written using documentation macros, decorators or annotations they can easily generate documentation from the tests themselves.
Everyone’s uses the same build/deploy and execution scripts, so that building and deploying someone’s code is as easy as building and deploying your own code.
The team uses all of the available tools to make finding problems in code such as findbugs, valgrind and coverity a regular and routine practice. All while using other tools to recommend better practices of writing code in certain languages such as pylint, javascript lint, etc so that the code is kept clean of easy to detect problems.
Code style checkers are enforced at check-in time to verify that everyone follows the same conventions of either tabs/spaces or whatever magical concoction that makes code easier to read and maintain by all.
Reviewing code is just a natural part of the process of getting your changes into the trunk because you don’t want to end up having everyone frown at you when that change breaks something just because you couldn’t wait another hour for someone to review your changes before you check them in.
Individual unit tests can be marked as performance enabled tests, that can then be used by a performance framework to stress tests each unit of available functionality and come up with performance data within a few hours of code landing in the trunk.
The frameworks used are all well understood and maintained in languages that most of the team members want to use for writing code and test in. The frameworks are part of the code that is written or maintained in order to help drive the development process to get things ready for pushing to production quicker. These same tools and frameworks are built/deployed and executed just like the rest of the code base and are treated the same way production quality code is treated.
Everyone feels ownership of the whole code base from the production code, build and release scripts and the tests that make it possible for their production code to be stable and ready for deployment.

Yes most of the previous statements may seem like a Utopian view of the software development process but I see it as a very possible reality which doesn’t get in the way of software development but does in fact make software development a well known scientifical process.

Conclusion

I started this writing with exposing how I feel we’ve evolved in terms of software engineering and software development processes and hoped to reach a phase of our evolution which is true to most in this day and age. I believe we could all do much better at being better software engineers than we currently are and that I myself, am still learning new approaches to software development every day. I didn’t spend time focused on the exact tools and frameworks to use, since I believe these types of things change too often and I really focused on identifying the boundaries created between a traditional software development process.

The fact that we so carefully distinguish the developer role and quality engineer role and how much of a waste of time and money that distinction is to companies and the whole development process is one of the main points of my writing. The real focus should be on how to make all of your engineering power work together as a cohesive software development group with everyone understanding how things are built/deployed and executed and all engineers feeling like they can move around and help others while still being able to count on others for help when necessary.

I do hope that after reading this writing you should at least feel that there is one thing you can work on immediately to make yourself a better software engineer and possibly another 2-3 things that you can set as short term goals to get started on at your current position.

I also hope that you realize that this writing is in no way a precise account of the situation at all companies but certainly reflects my own experiences and the experiences of a few other colleagues in the business and hopefully can help others feel they’re not alone or even generate a lot of interesting conversations.

Speeding up Python with Numba

2012-08-26T00:00:00+00:00

Just a quick post on a sweet dynamic compiler for Python which is numpy aware and does JIT compiling to LLVM bit-code to speed up your Python loops by quite a few tremendous orders of magnitude.

Its still quite beta software and is under heavy development but its extremely promising where this project is going. Take for example the following simple code that just sums a large matrix of numbers like so:

def matrix_sum(arr):                                                            
    M, N = arr.shape                                                            
    result = 0.0                                                                
    for i in range(M):                                                          
        for j in range(N):                                                      
            result += arr[i,j]                                                  
    return result

Of course the example is using numpy arrays and in this case we’d generate the array by using the numpy module like so:

random_matrix = numpy.random.randn(5000, 5000)

That generates a random matrix of 5000x5000 elements and now we can use that to measure how long our function takes to calculate the sum of all of the elements in the matrix:

start = time.time()                                                             
matrix_sum(D)
print('time to use matrix_sum %s' % (time.time()-start))

The time it takes to run this little sum of elements is:

time to use matrix_sum 18.0450868607

That’s 18 seconds which of course illustrates how bad Python is at handling loop with even a simple data type such a float.

Now to use numba it involves decorating the method we want to have numba replace with LLVM bit-code and you can do so currently like so:

from numba import double
from numba.decorators import jit

@jit(arg_types=[double[:,:]])(matrix_sum)
def matrix_sum(arr):                                                            
    M, N = arr.shape                                                            
    result = 0.0                                                                
    for i in range(M):                                                          
        for j in range(N):                                                      
            result += arr[i,j]                                                  
    return result

So very simply we tell numba where to apply its magic and our argument and return types. The developers on numba are already looking into how to use introspection to figure out the argument and return types on their own and you’d just have to place the @jit decorator on the method you wanted to apply the numba JIT’ing on.

With that very small addition our same loop now executes in:

time to use jit_matrix_sum 0.0553958415985

That is an increase in performance of 325x without having to make the code harder to read or use some elaborately hard to implement and compreenhend algorithm. For kicks I increased the array size by 9x and made it a 15000x15000 array and still didn’t need more than half a second to calculate the sum of this new array with the JIT’ed method:

time to use jit_matrix_sum 0.494293928146

To understand how this is possible you have to realize what numba is doing is converting your array oriented program (ie for loop) and using LLVM to execute this as quickly as possible on your hardware which has various elements that are actually designed better for array oriented programs and this is what numba is taking advantage of.

I just wanted to note this library for future reference as I believe they’re on the right path and the ideas in the numba library could be integrated into Python core and allow for these type of optimizations to be done on all code paths, boosting the awesomeness of Python.

Using SST and Freshen

2012-05-27T00:00:00+00:00

I work in the quality assurance field and I’ve built and used many tools in my career to accomplish my every day tasks. I recently had to do some web testing (ie Selenium appropriate) and had a few requirements a long the lines of not just having a development language for testing but a testing language that could be used by non coder friendly individuals (ie black box testers or even designers as they put together mocks for UI’s).

For the web testing aspect I got familiar with SST and since its written in Python and couldn’t be easier to setup and get started I found this to be a great way to driven Selenium tests. You should check the site out and you’ll see how quick and easy you can start writing tests and have a full blown acceptance suite up and running within an afternoon.

Now the bigger challenge was what language I would use to allow non coder types to write tests quick and effectively and yet be able to automate some of their work so that their tests could be integrated into the acceptance testing suite and catch issues before shipping and without having tons of black box testing done. I started digging around and the first big obvious choice is Cucumber but I found that its written in Ruby and wanting to stick to Python and its familiar tool chain I searched around for something that would give me the same language and be written in Python. What I found was Freshen, a very well done clone of the Ruby Cucumber language which had even a few special additions to the Cucumber language which made it even more interesting to use for this type of testing.

Freshen is very easy to extend and what I wanted to do was to have Freshen give you a nice and natural way of writing tests that would drive the SST framework so that you could easily test a web site by people who are not strong coders. The first aspect to figure out is that given Cucumber tests look like so:

Scenario: Divide regular numbers
  Given I have entered 3 into the calculator
  And I have entered 2 into the calculator
  When I press divide
  Then the result should be 1.5 on the screen

We need to figure out exactly the wording that we wanted to use when doing certain actions through SST that would allow you as the test writer to really trigger the right events and validate the right things have happened and while doing this you do want to “hide” certain aspects of the HTML content such as the exact path to certain elements on the page (maintaining XPath or css selector expression is hard enough for developers). So lets pick the main actions from SST that we’d like to expose in the Freshen language:

go_to - open a specific page in the currently configured browser.
go_back - hit the back button on the browser.
click_elemnt - to click on the various interactable elements on the page.
assert_title - validate the title of the page is correct.
assert_link - validate there is a link to another page on the page.
assert_text - validate that certain textual elements are on the page.

Let’s keep the list to just those for now and as for what we’ll test I shall pick my blog site which I’d like to make sure is working correctly at any given moment and can be correctly navigated by any person using it. We could start defining the language for our tests like so:

Feature: Personal Blog

  Scenario: Visit the main page
   Given I am at http://localhost:4000
   Then I should see the title Rodney's Corner
    And I should see the link Blog, Archive, About
    And I should see the headers Blog, Recent Posts, Coding

That was a first analysis and from that we can see that we need a few different ways of validating content on the page. There’s still an issue with the way we get to the blog by referring to the url directly, I’d really prefer if we could refer to it by a name (ie alias) that could be maintained in the Python configuration file and easily modified to point to different testing environments. To be able to run the Freshen tests you’ll need to first make yourself a virtualenv (or install on your base sytem if you prefer) and install the following:

pip install -U sst
pip install ‘git+git://github.com/rlisagor/freshen.git#egg=freshen’

Once you have that installed we can now start defining the steps Python module required to start executing our Freshen tests against the blog. We’ll start by defining the Given statement and the first Then which are the simplest to begin with:

from sst.actions import *
from freshen import *

@Given('I am at (.*)')
def i_am_at(url):
    go_to(url)

@Then('I should see the title (.*)')
def should_see_the_title(title):
    assert_title(title)

The previous steps implementation allows us to execute a reduced version of the previous navigation tests, like so:

Feature: Personal Blog

  Scenario: Visit the main page
   Given I am at http://localhost:4000
   Then I should see the title Rodney's Corner

And to run this you can do so using nosetests command line tool like so:

> nosetests --with-freshen -v Personal Blog: Visit the main page ... ok ---------------------------------------------------------------------- Ran 1 test in 2.894s OK

If you’d like to get the output from SST just make sure to pass the argument –nocapture and you’ll get the stdout output. Lets look into making the current steps a bit smarter and easier to maintain in the long run. So the first thing is how to replace the exact url with a location alias instead. Any easy way of doing this could be like so:

from sst.actions import *
from freshen import *

import os

test_env = 'test'
if 'ENV' in os.environ.keys():
    test_env = os.environ['ENV']

if test_env == 'test':
    url_alias = {
                 'main blog site' : 'http://localhost:4000',
                }
elif test_env == 'prod':
    url_alias = {
                 'main blog site' : 'http://rlgomes.github.com',
                }
else:
    raise Exception("Unknown environment %s" % test_env)

@Given('I am at (.*)')
def i_am_at(url):
    if url in url_alias:
        url = url_alias[url]

    go_to(url)

@Then('I should see the title (.*)')
def should_see_the_title(title):
    assert_title(title)

I decided to also incorporate the notion of test and prod environment configuration which you can easily drive from the command line by exporting the ENV variable. We now need to implement the other steps that are required to verify links and headers on the page. So here’s a complete solution that handles the lists of links and headers:

from sst.actions import *
from freshen import *

import os

@After
def teardown(scenario_ctx):
    close_window()
    stop()

test_env = 'test'
if 'ENV' in os.environ.keys():
    test_env = os.environ['ENV']

if test_env == 'test':
    url_alias = {
                 'main blog site' : 'http://localhost:4000',
                }
elif test_env == 'prod':
    url_alias = {
                 'main blog site' : 'http://rlgomes.github.com',
                }
else:
    raise Exception("Unknown environment %s" % test_env)


def find_element(search_string):
    """
    Attempt to find an element by using the specified search string to find the
    element in the following order of searching:

        1. by id
        2. by css_class
        3. by text
        4. by text_regex

    """
    if exists_element(id=search_string):
        return get_element(id=search_string)

    if exists_element(css_class=search_string):
        return get_element(css_class=search_string)

    if exists_element(text=search_string):
        return get_element(text=search_string)

    if exists_element(text_regex=search_string):
        return get_element(text_regex=search_string)

    raise Exception("Can't find the element %s" % search_string)

@Given('I am at (.*)')
def i_am_at(url):
    if url in url_alias:
        url = url_alias[url]

    go_to(url)

@Then('I should see the title (.*)')
def should_see_the_title(title):
    assert_title(title)

@NamedTransform('{list}', '([\w\, ]+)', '([\w\, ]+)')
def transform_user_list(list):
    return [ name.strip() for name in list.split(',') ]

@Then('I should see the links? {list}')
def should_see_the_link(links):
    for link in links:
        element = find_element(link)
        assert_link(element)

@Then('I should see the headers? {list}')
def should_see_the_headers(headers):
    for header in headers:
        aux = 'text()="%s"' % header
        get_element_by_xpath('//h1[%s] | //h2[%s] | //h3[%s] | //h4[%s]' % \
                             (aux, aux, aux, aux))

I added a teardown method to correctly close the browser and stop it between each scenario, so we have a nice clean state when we start the next scenario. I also created the find_element function that can do a pretty good job at trying to find your element by trying a few different methods. You could easily define your own find_element with different rules on how you look for elements based on the names that are passed. Now, in order for us to write some more useful tests we actually need to be able to click on those links and move back and forward through the browsing experience. We can do this by using the SST commands: go_back and click_element and here’s how we’d integrate those actions into the available Freshen steps:

from sst.actions import *
from freshen import *

...

@When('I click back')
def click_back():
    go_back()

@When('I click on (.*)')
def click_on(element):
    element = find_element(element)
    click_element(element)

We can now write tests like so:

Feature: Personal Blog

  Scenario: Visit the main page
   Given I am at main blog site
    Then I should see the title Rodney's Corner
     And I should see the links Blog, Archive, About
     And I should see the headers Blog, Recent Posts, Coding

  Scenario Outline: Navigate from the main page
   Given I am at main blog site
    Then I should see the title Rodney's Corner
   When I click on <link>
    Then I should see the links <links_to_validate>
   When I click back
    Then I should see the title Rodney's Corner

Examples:
  |  link   |  links_to_validate   |
  |  Blog   | Blog, Archive, About |
  | Archive | Blog, Archive, About |
  |  About  | Blog, Archive, About |

I really like how you can define a Scenario Outline and then add entries to an ASCII table which drives that test with additional data points. With this I can now write quite an extensive suite of tests for my blog site that would at least validate that I can move between the various links on my site without any navigation issues and that the content on the site is showing the right headers and links in each of the various pages available.

There’s also a command line tool called freshen-list installed along with your Freshen install which allows you to basically list the available commands for your Freshen tests from any sub directory with Freshen steps defined. You can call it like so:

> freshen-list tests GIVEN tests/steps.py I am at (.*) WHEN tests/steps.py I click back I click on (.*) THEN tests/steps.py I should see the headers? ([\w\, ]+) I should see the links? ([\w\, ]+) I should see the title (.*)

Its very simple and could use a little tweaking to make it more useful but it gives you an immediate sense of what commands are available for writing tests. The last thing I’ll say about Freshen is that it gives you the ability to hide a lot of the complexity of how you interact with a given system and allows you to express those actions in simple English which can be maintained by someone who doesn’t have good coding skills. Another thing about this is that you can always change the driving language underneath (i.e. switch to ruby cucumber) without having to change your tests.

I hope this write up has helped you get acquainted with Freshen as well as learning a bit about SST which I feel is really great for web UI testing that hides a lot of the complexities of working with Selenium.

Prolog and Graphs

2012-05-22T00:00:00+00:00

There are a few things we’ve shown that Prolog can do better than other languages and now we’re going to show you a data structure that can be very easily represented in Prolog and for which you can very easily define traversal methods that do things that in other languages would take hundreds of lines of code and a lot of testing.

So lets say we wanted to represent the following graph:

%  a --- b --- e
%   \         /
%    \       /
%     c --- d --- f
%            \
%             \
%              g

We could represent the above using the following facts in Prolog:

path(a,b).
path(b,e).
path(a,c).
path(c,d).
path(e,d).
path(d,f).
path(d,g).

Now all we need is to define a simple predicate that can calculate paths by composition of existing paths. For example if X is connected to Z and Z is connected to Y then there is a path between X and Y, which is Prolog is very similar to the sentence we just wrote:

path(X,Y) :-path(X,Z), path(Z,Y).

With that predicate we can now ask Prolog a few different questions:

| ?- [graphs_v1]. ... yes | ?- path(a,b). true ? yes | ?- path(a,e). true ? yes | ?- path(a,x). Fatal Error: local stack overflow (size: 8192 Kb, environment variable used: LOCALSZ)

Oh shoot, seems when we ask for a path that can’t be solved that we run out of stack space. This is because we don’t have a termination rule and Prolog keeps instantiating variables for Z that it can then match on a longer sequence of path facts until it runs out of stack space. The easiest way to solve this is to make sure that when we start on a path to look for Z that connects X and Y that we simply validate that both X and Y are valid atoms and not variables. Which would look like so:

path(a,b).
path(b,e).
path(a,c).
path(c,d).
path(e,d).
path(d,f).
path(d,g).

path(X,Y) :- atom(X), atom(Y), path(X,Z), path(Z,Y).

This now works as desired but presents a small problem which is that we can’t query Prolog for all of the nodes we can reach from a, like so:

| ?- findall(X, path(a,X), List). List = [b,c] yes | ?-

We’d really like the answer to be [b,c,d,e,f,g] and for that we’re going to have to find a way to change our solution so that we don’t stack overflow and still be able to calculate all of the reachable nodes for a given node. Now part of the problem is that we don’t have a predicate for validating a path with a different name from the fact that represents the edge between two nodes. So we really need to fix this by naming these two things differently, like so:

edge(a,b).
edge(b,e).
edge(a,c).
edge(c,d).
edge(e,d).
edge(d,f).
edge(d,g).

Now the path method can be defined as finding an edge between X and Y directly or finding an edge between X and Z and a path between Z and Y, something like the following:

path(X,Y) :- edge(X,Y).
path(X,Y) :- edge(X,Z), path(Z,Y).

The nice thing is that now our function behaves beautifully and even avoids overflowing the stack, but results in a few duplicate entries when requesting all the paths from a to all other nodes:

| ?- findall(X, path(a,X), List). List = [b,c,e,d,f,g,d,f,g] yes

Removing the duplicates is quite easy with the setof/2 predicate which does the same as the findall predicate but without duplicates in the resulting set, like so:

| ?- setof(X, path(a,X), List). List = [b,c,d,e,f,g] yes

Everything so far is working well because the paths are unidirectional and we don’t have any cycles that would give us troubles. Lets just create a simple edge from g back to d. Now when we run a simple path(g,f) you’ll notice it doesn’t run out of solutions because it keeps identifying a new path between g and d that involves going through the g or d again. To fix this issue we’ll have to use a visited list and keep track of visited nodes, like so:

edge(a,b).
edge(b,e).
edge(a,c).
edge(c,d).
edge(e,d).
edge(d,f).
edge(d,g).
edge(g,d).

path(X, Y) :- path(X,Y,[]).

path(X, Y, _) :- edge(X,Y).
path(X, Y, V) :- \+ member(X, V), edge(X, Z), path(Z, Y, [X|V]).

The previous solution now handles cycles just fine and can still be used with the findall and setof predicates. Lets complicate the graph by adding weights for each of the connections and then having Prolog calculate which is the quickest path between two nodes. So lets show you how to represent the weights in the graph quite easily and then also show you how to find a path through two nodes and calculate the weight of traveling that path as well as the path traveled. Lets start with representing weights in the edges between the nodes, like so:

%
%  a -1- b -2- e
%   \         /
%   3        3
%    \       /
%     c -2- d -2- f
%            \
%            3
%             \
%              g
%

edge(a,b,1).
edge(b,e,2).
edge(a,c,3).
edge(c,d,2).
edge(e,d,3).
edge(d,f,2).
edge(d,g,3).

I’ve also represented our test graph using an ASCII format that can be easily understood with the weights of traveling those paths. If we now want to calculate all of the possible paths and then evaluate which is the best path then we need to think of solving two problems separately. First is how do we find all the paths between node X and node Y and the second is how do we pick the minimal path. The first question we’ll solve using the predicate findapath which can be expressed as such:

findapath between X and Y has weight W if there is an edge between X and Y of weight W.
else findapath between X and Y of weight W is true if we can find a path between X and Z of weight W1 and there is a findapath between Z and Y of weight W2 where W is W1 + W2.

Note the above is missing a check that we haven’t already visited the X while doing subsequent matches on the 2nd rule of the findapath predicate in order to avoid running forever on a cycle.

With the above rules our findapath predicate may look like so:

findapath(X, Y, W, [X,Y], _) :- edge(X, Y, W).
findapath(X, Y, W, [X|P], V) :- \+ member(X, V),
                                 edge(X, Z, W1),
                                 findapath(Z, Y, W2, P, [X|V]),
                                 W is W1 + W2.

The above predicate is a bit more complicated since we’re calculating the weight and also tracking the exact path we followed till we find the route all the way to the Y node. With the above predicate you can query with lets say the path between a and g and if you ask Prolog to give you other solutions with the ; character you can get the following:

| ?- findapath(a,g,Weight,Path,[]). Path = [a,b,e,d,g] Weight = 9 ? ; Path = [a,c,d,g] Weight = 8 ? ; no

So Prolog is capable of finding the two paths that lead from a to g and to also tell us there is no other possible paths aside from the two already calculated.

We now need to write a function that has a similar design pattern to that of the already used findall and setof predicates. In which, we attempt to find all the solutions for a given Goal but in the process we also pick from each subsequent solution the best one and save that to return at the end. The design pattern used for writing a findall is something similar to the following:

findall(X, Goal, Xlist) :- call( Goal),
                           assertz(queue(X)),
                           fail.

findall(_, _, XList) :- assertz(queue(bottom)),
                        collect(Xlist).

collect(L) :- retract(queue(bottom)), !, L = [].
collect(L) :- retract(queue(X)), !, L = [X|Rest], collect(Rest).

This a slightly modified version from what you may find online and I’ve avoided using the ; operator which is an else operator which condenses the writing of multiple rules but makes it much harder to read. So the findall function above can be read:

For the goal Goal we call the Goal first and then that instantiates the term X with its value which we then assertz into the Prolog database and then we fail so that Prolog will back track and find other solutions for our Goal (which in turn get asserted into the Prolog database).
Once we’ve finished satisfying all of the possible solutions for our Goal we’ll assert one last element into the Prolog database that will allow us to collect the various results with the collect predicate.

The collect predicate is also quite involved but you just need to understand something very simple about how it works: It retracts from the Prolog database in order from the very first assertz to the last fact that we asserted called queue(bottom) at which point its done collecting all of the results required to form the resulting list.

What we’re trying to write is a similar predicate but we want to on every newly found solution compare to the last best solution and immediately decide which one to keep in the Prolog database so that we only keep track of 1 solution at any given time during the predicates solution (very memory efficient solution). Given the definition of the findall predicate it isn’t that hard to get to a solution like so for our findminpath predicate:

:-dynamic(solution/2).
findminpath(X, Y, W, P) :- \+ solution(_, _),
                           findapath(X, Y, W1, P1, []),
                           assertz(solution(W1, P1)),
                           !,
                           findminpath(X,Y,W,P).

findminpath(X, Y, _, _) :- findapath(X, Y, W1, P1, []),
                           solution(W2, P2),
                           W1 < W2,
                           retract(solution(W2, P2)),
                           asserta(solution(W1, P1)),
                           fail.

findminpath(_, _, W, P) :- solution(W,P), retract(solution(W,P)).

This is without a doubt one of the most complex predicates we’ve written to date so don’t be too worried if you don’t get it at first glance. So the first rule is there to just populate the Prolog database with the first solution and then proceed with the underlying multiple solution gathering and comparison in order to always leave the best solution in the Prolog database which the last rule is going to query and retract and return to the user requesting the solution.

You can have written this with two rules and in the current 2nd rule you’d create a dummy solution like assertz(solution(100000,[])) which would be immediately lose to the first found path and return nothing but that would be a lame solution prone to issues such as returning an empty path when there is no path between two specified nodes or having issues when you’re sum of an existing path is more than the magical 100000 that you thought was big enough weight that you would never exceed.

You can now use your newly created findminpath to easily calculate the best path between a and g and you’d get the expected result of:

| ?- findminpath(a,g,W,P). P = [a,c,d,g] W = 8

This post has covered quite a few things but I hope the one thing you can take away from this post is that Prolog can be extremely easy to express certain data structures and to also write up certain functions used in every day coding requirements that would otherwise take days of careful writing and testing to come up with.

An interesting thing I came across that was written using Prolog is an article on how IBM used Prolog to do natural language processing for their Jeopardy winning computer program called Watson, you can read more here.

Taming Prolog

2012-05-20T00:00:00+00:00

In this post I’d really like to go over how Prolog executes your predicates and how you can control how this execution is handled. Lets start with another predicate that we can trace its flow and see how Prolog executes a predicate with multiple ramifications. Lets pick a simple predicate that can validate if given Term1,Term2 and Term3 if Term3 is the minimum of the two.

minimum(X, Y, X):- X =< Y.
minimum(X, Y, Y):- X > Y.

Now when we trace the execution of lets say minimum(1,2,X) we are expecting the answer that X=1 and then we’d be done, but here is what we get when tracing and allowing the system to give us all the available options:

| ?- minimum(1,2,X). 1 1 Call: minimum(1,2,_16) ? 2 2 Call: 1=<2 ? 2 2 Exit: 1=<2 ? 1 1 Exit: minimum(1,2,1) ? X = 1 ? ; 1 1 Redo: minimum(1,2,1) ? 2 2 Call: 1>2 ? 2 2 Fail: 1>2 ? 1 1 Fail: minimum(1,2,_16) ? no

We know for a fact that as soon as you validate that 1 is the minimum of those terms that there is no reason to backtrack and try the other rule since it would logically be false and you could no longer find any solutions that would be valid. In Prolog you are able to tell the engine to not backtrack any longer by using a cut which is the operator ! and for the previous predicate it would be used like so:

minimum(X, Y, X):- X =< Y, !.
minimum(X, Y, Y):- X > Y.

Now if you trace this you’ll see that as soon as Prolog reaches the cut it will stop searching for other solutions, like so:

| ?- minimum(1,2,X). 1 1 Call: minimum(1,2,_17) ? 2 2 Call: 1=<2 ? 2 2 Exit: 1=<2 ? 1 1 Exit: minimum(1,2,1) ? X = 1 yes

This may not seem like a big deal right now but as we show more complicated predicates you’ll find that this could meant the difference between executing a rule in linear time vs exponential time due to the fact that Prolog attempts to exhaust all possible solutions for a given predicate.

You’ll find that cut can be heard to understand but as you trace your code and see situations where Prolog is wasting time by doing backtracking and attempting to match on other rules that would never work you’ll realize that a simple placement of the appropriate cut can speed up the execution of your Prolog predicates.

Lets try to cover another interesting feature in Prolog which is the ability to add new predicates to your running database or remove existing ones. Which means you can now make your predicates dynamic and really get some interesting things done in Prolog. The 2 predicates used are asserta/1 and retrace/1 which respectively add and remove the specified term from the Prolog database. Lets look at how you’d define the Fibonacci function in Prolog:

fib(0,0).
fib(1,1).
fib(N,F) :- N1 is N-1,
            N2 is N1-1,
            fib(N1,F1),
            fib(N2,F2),
            F is F1 + F2.

Now to execute the above with large numbers you may need to tweak your global and local stack to something higher. The simplest way is by setting the environment variables: LOCALSZ and GLOBALSZ to something higher like so:

export LOCALSZ=131072 export GLOBALSZ=131072

With the above settings if you start executing our Fibonacci example with numbers from 1 to 25 you’ll notice that the execution time starts to increase quite quickly and also the stack size would need to be further made larger to be able to calculate the Fibonacci of 100. Well the good thing is that we just learned about asserta/1 and can basically use the Prolog database to memoize previous results. This is what the new solution could look like and this solution can now calculate Fibonacci of 100 without any issues:

:-dynamic(fib/2).
fib(0,0).
fib(1,1).
fib(N,F) :- N1 is N-1,
            N2 is N1-1,
            fib(N1,F1),
            fib(N2,F2),
            F is F1 + F2,
            asserta(fib(N,F)).

The dynamic predicate call is required in order to tell Prolog that the fib/2 predicate can be dynamically modified. The other small change was to basically memoize the fib(N,F) so it wouldn’t be recalculated on every subsequent call.

There’s a lot you can do with cut and being able to assert and retract from the Prolog database and I believe you should go and experiment with the newly found features of Prolog so you can become learn better how they work.

Learning more Prolog

2012-05-17T00:00:00+00:00

In this post we’ll cover the built-in functions that Prolog has that are useful and also get into more advanced topics such as using assert and retract methods and talk a bit about how Prolog executes the code you write and how to trace through this execution to better understand how Prolog works.

Lets start by listing a few of the most important built-in functions that you’ll find yourself using a regular basis:

Type Testing

var/1 - succeeds if term is currently not instantiated.
nonvar/1 - succeeds if term is instantiated.
atom/1 - succeds if term is an atom.
integer/1, float/1, etc.

Term Unification

=/2 (unification) - tests if Term1 and Term2 can be unified, example: A is 2 + 1, A = 3. % will succeed.
=/2 (not unifiable) - tests that Term1 and Term2 can not be unified, example: 2 + 1 = 3. % will fail to unify.

Term Comparison

==/2 (equals) - straight up equality of the terms being compared.
==/2 (not equals) - straight up inequiltiy of the terms being compared.
@</2 (lest than), etc - the various other comparison operators.

Lists

append/3 - when it succeeds you’ll have a appended the first two terms to each other and the resulting list will be the contained in the third term.
member/2 - succeeds if term1 is found within the list that term2 is referencing.
delete/3, permutation/2, sublist/2

Above we’ve also introduced the predicate notation used to indicate how many arguments the predicate has which is /n where n is the number terms that the predicate requires.

You can always look up more predicates that are available to default installations on your own time now the interesting thing is going to be showing how powerful some of these harmless predicates really are. Lets start with the append/3 predicate and actually first show how this predicate is implemented:

append([],Ys,Ys).
append([X|Xs],Ys,[X|Zs]) :- append(Xs,Ys,Zs).

The above is already built in so dont’ try loading a file with that otherwise you’ll be greated with a message stating “ native code procedure append/3 cannot be redefined (ignored)”. But lets use the predicate to show how easy it is to append lists to eachother, like the following shows:

That works as expected and gives us the response we’re expecting. Here is where we can show one of the most powerful features of Prolog and that is the ability to do inference of your predicates in order to logically fulfill them and do things that in other languages would require quite a lot of code writing.

Before we show how inference works lets instead do a trace of the previous usage of append so we can better understand how inference works. So to trace a Prolog execution in GNU Prolog once the interpreter is up hit Ctrl+C and pick t to enable trace. Then type the predicate as before and now you’ll be stepping through each step in the Prolog engine on the screen, like so:

| ?- Prolog interruption (h for help) ? t The debugger will first creep -- showing everything (trace) | ?- append([1,2,3],[4,5,6],L). 1 1 Call: append([1,2,3],[4,5,6],_29) ? 1 1 Exit: append([1,2,3],[4,5,6],[1,2,3,4,5,6]) ? L = [1,2,3,4,5,6] yes {trace} | ?-

Humm that was useless because the native method doesn’t trace through the Prolog implementation the same way. For this example load the previous append definition with a different name such as append1 and then you should be able to get a similar trace to the following:

| ?- Prolog interruption (h for help) ? t The debugger will first creep -- showing everything (trace) | ?- append1([1,2,3],[4,5,6],L). 1 1 Call: append1([1,2,3],[4,5,6],_29) ? 2 2 Call: append1([2,3],[4,5,6],_62) ? 3 3 Call: append1([3],[4,5,6],_89) ? 4 4 Call: append1([],[4,5,6],_116) ? 4 4 Exit: append1([],[4,5,6],[4,5,6]) ? 3 3 Exit: append1([3],[4,5,6],[3,4,5,6]) ? 2 2 Exit: append1([2,3],[4,5,6],[2,3,4,5,6]) ? 1 1 Exit: append1([1,2,3],[4,5,6],[1,2,3,4,5,6]) ? L = [1,2,3,4,5,6] yes {trace}

Now the trace can be a bit hard to understand at first but as you trace through more and more predicate execution you’ll get the hang of how things work. So for our append implementation we defined append in a special way because we were taking advantage of tail recursion in Prolog and we defined the append predicate like so:

If you have an empty list and a list Ys then the result of appending them is Ys.
If you have a list with head X and tail Xs and another list Ys, then the resulting list is going to consist of putting X at the head of the list with Zs being the concatenation of Xs and Ys.

Because of that special definition the Prolog engine will create your resulting list as it exits from each of the calls and now while its proceeding to match the termination predicate at append1([],[4,5,6],_116). So knowing this and looking at the trace you now know that the Prolog enginew as trying to reduce your initial request of append1([1,2,3],[4,5,6],L) into one that terminated with the append1([],[4,5,6],_XX) and then worked its way backwards to create the resulting appended list.

The really interesting thing about this backward inference and the ability to deduce which elements were moved around based on how you defined a predicate allows Prolog to do even more interesting things such as:

| ?- append(A,[4,5],[1,2,4,5]). A = [1,2] ? yes

We just used our append method to infer what list appended to [4,5] gives us the list [1,2,4,5]. This may not seem ground breaking but think about how hard this would be to implement in another language and you’ll soon see the power of inference. If you want to see a quick way that inference can be used, lets take the problem:

given a list of elements enumerate all of the possible combinations of 2 lists that can be append to create the resulting list.

Getting Prolog to do this for us is as simple as:

| ?- append(A,B,[1,2,3,4]). A = [] B = [1,2,3,4] ? ; A = [1] B = [2,3,4] ? ; A = [1,2] B = [3,4] ? ; A = [1,2,3] B = [4] ? ; A = [1,2,3,4] B = [] ? ; no

Instead of having the interpret iterate the possible solutions lets introduce the usage of the findall predicate which can be used to create a List of solutions. The predicate itself is defined as findall(Object,Goal,List) where the Object is the elements from the Goal you wish to put in the List for each solution found. So lets just show how to use findall to get all the solutions in a nice neat little list.

| ?- findall((A,B),append(A,B,[1,2,3,4]),List). List = [([],[1,2,3,4]),([1],[2,3,4]),([1,2],[3,4]),([1,2,3],[4]),([1,2,3,4],[])] yes

While on the subject what if you wanted to not include any solutions with empty lists as part of the concatenation ? Here’s one possible solution:

| ?- findall((A,B),(append(A,B,[1,2,3,4]), A \= [], B \= []),List). List = [([1],[2,3,4]),([1,2],[3,4]),([1,2,3],[4])] yes

A simple non unification test for A and B to the empty list term and we’ve fixed our problem.

We’ve covered quite a few things in this post and I am probably running through a lot of things without explaining too many details and showing more implementation and usage scenarios. I really don’t want to spend too much time writing up theory and explanations you can find online and instead would like to focus on how to use the various features to complete tasks that would be very difficult in other languages.

The next post will cover more details of tracing and I will try to also introduce the notion of “cutting”, which involves controlling how Prolog does backtracking through your predicates.

Learning Prolog basics

2012-05-16T00:00:00+00:00

I wanted to review my Prolog skills an at the same time write up a quick set of posts on how to use Prolog to get certain tasks done in a more efficient manner. This first post is about going over how you write a basic Prolog program and how to wrap your mind around this “different” programming paradigm.

Prolog is a logic programming language based that uses facts and rules to evaluate if what you are trying to compute is true or false logically. These facts and rules are loaded into what is usually called the Prolog database and then you can query them in order to get the answers to the questions that you want to solve. The most basic thing you can define in Prolog is a fact and a fact has the form:

fact(atom).

A fact can be just a simple name followed by the open and closing parenthesis or it can include arguments which are actually called atoms. To understand better an atom is a general purpose name used to identify elements and is represented by a lower case sequence of characters. For example lets say we wanted to expression that rodney is a human being. The fact for such a thing in Prolog could be written like so:

human(rodney).

The above is also equivalente to “human(rodney) :- true.”. Now, lets fire up the gnu prolog interpreter and load the file that contains the above fact, like so:

> prolog GNU Prolog 1.3.0 By Daniel Diaz Copyright (C) 1999-2007 Daniel Diaz | ?- [basics]. compiling /home/rlgomes/workspace/prolog/prolog_basics/basics.pl for byte code... /home/rlgomes/workspace/prolog/prolog_basics/basics.pl compiled, 5 lines read - 413 bytes written, 4 ms (4 ms) yes | ?-

Loading the file is done with the usage of the square brackets surrounding the name of the file that has a .pl extension. Once loaded you can query the Prolog engine by writing a rule and verify if it matches something in the Prolog database, like so:

| ?- human(rick). no

Here you can see for the first time how the Prolog interpreter responds with no in order to tell you that it does not know if ‘rick’ is human. So we can also query the engine for truthful facts like so:

| ?- human(rodney). yes

Facts are interesting and the basis of everything that is known to be true within a Prolog engine but the really interesting part is when you start writing rules. A rule is a very similar predicate construction that uses variables to match other values in order to evaluate to a truthful statement. So for example lets define a rule that says that all humans are mortals, like so:

mortal(X) :- human(X).

The above rule is read “X is mortal if X is human” and is a very simple rule that you can use just as before to validate that “mortal(rodney).” and you’ll get the response yes. We can make a more interesting program with the few things we’ve learned so far and lets just jump into the program below:

% samantha is alice's mother
mother(samantha, alice).
mother(samantha, bob).

% joe is alice's father
father(joe, alice).
father(joe, bob).
father(joseph, joe).

% X is sibling of Y if father X has father Z and Y has father Z
siblings(X, Y) :- father(Z, X), father(Z, Y).

grandparent(X, Y) :- father(X, Z), father(Z, Y).
grandparent(X, Y) :- mother(X, Z), father(Z, Y).
grandparent(X, Y) :- father(X, Z), mother(Z, Y).
grandparent(X, Y) :- mother(X, Z), mother(Z, Y).

This short program can now easily validate very easily if people are siblings or if someone is someone’s grandparent. You can see how to create multiple rules to validate the same general predicate but defining the various variations that would make that predicate true. Lets see how quickly we can verify if the various people are related.

| ?- siblings(alice, joe). no | ?- siblings(alice, bob). true ? yes

You may have noticed that when verifying that Alice and Bob were siblings you got a slightly different prompt. This new prompt identifies that there is one way to verify that Alice and Bob are siblings and if you hit just enter you don’t have any interest in additional solutions but if you hit ‘;’ followed by enter the Prolog engine will try to verify alternate ways of evaluating that Alice and Bob are siblings. We’ll go into more details on the alternate routes to verify the same predicate in future posts.

Lets make a quick introduction into lists and how Prolog represents and uses them and then we’ll leave until the next post to get into more complex parts of Prolog. Now lists are represented in a very easy to understand manner like so:

A=[1,2,3].

The above will evaluate on the interpreter to truth but doesn’t serve any purpose in a Prolog file. We can now talk about how to use lists within Prolog predicates and basically take a list apart. When specifying a list in a new rule we can separate the current head of the list for the rest of the list like so:

some_rule_over_lists([H | T]) :- true.

Once again the above doesn’t serve any purpose but just introduces you to the notion of processing a list with a Prolog rule. Now if we wanted to write a function to calculate the length of a list we can’t really return a value so what needs to be done is you need your “length” function to actually have a second argument which would house the result. So we’d define length like so:

len([], 0).
len([_|T], N) :- len(T,NT), N is NT + 1.

This last implementation has quite a few new things in it so lets start by explaining that when defining any rules that operate over lists you usually have the predicate that handles the empty list [] and the predicate that handles a list with a head and tail, like so [H|T]. This is a very common pattern for recursive functions over lists and is used all the time when handling lists in Prolog. We are also introducing how to do arithmetic in Prolog using the is statement to attribute to N the value of calculating the length of the tail T plus 1. I also decided to introduce the anonymous variable _ because you will get a warning about H not being used whenever you have things that are matched but not used to calculate anything of importance.

You can use the previously defined len function like so:

> prolog GNU Prolog 1.3.0 By Daniel Diaz Copyright (C) 1999-2007 Daniel Diaz | ?- consult(basics). compiling /home/rlgomes/workspace/prolog/prolog_basics/basics.pl for byte code... /home/rlgomes/workspace/prolog/prolog_basics/basics.pl:7: warning: singleton variables [H,T] for blah/1 /home/rlgomes/workspace/prolog/prolog_basics/basics.pl compiled, 10 lines read - 1033 bytes written, 10 ms (4 ms) yes | ?- len([a,b,c,d,e,f,g,h,i,j], L). L = 10 yes | ?-

I would advise you to go and play with the interpreter and defining other Prolog functions such as:

sum function that can sum up the elements in a list of numbers, with the syntax sum([1,2,3,4,5], S) and returns the sum in the variable S.
replace function which given a list of elements replaces a specific element in the list with another element specified and as before the result should be the last argument in your function. Start with the definition for your function like so replace(List, Element, Replacement, NewList) and you should be able to write that up with a little effort.

In the following post I will be looking at the built in functions and how to assert and retract facts from the Prolog database.

Now powered by bootstrap

2012-05-16T00:00:00+00:00

Finally decided the blog required a more professional and better look & feel to it and at the same time wanted to learn a bit more about this Bootstrap framework that some engineers at Twitter had released.

I started by simply having a look at the available examples in the source code and the basic templates and was quickly able to get a grasp on how to use this along with Jekyll to make my site look way more professional.

Having spent a total of about 8 hours on the task I am completely satisfied with the new look and will make further changes in the weeks to come as I find small visual issues that I haven’t already fixed. To get an idea here is a screenshot of what the site use to look like before using bootstrap:

And now powered by Bootstrap:

Tuning the Java Garbage Collector

2011-12-30T00:00:00+00:00

Recently, I’ve found a few situations where when writing a prototype for multi-node storage software that I want to be able to guarantee the stability of my performance numbers as the number of concurrent requests grows and the JVM is under high stress where the GC kicks in more often and has the unfortunate effect of adding higher latency to your requests.

Recently, I wanted to learn a bit more about the Java GC and how an applications performance can be “fixed” by tweaking the garbage collector being used. We’ll start by creating an application that will force the JVM to garbage collect more often: Call a function in a loop and this function will create a large object and calculate something complex (log(n)) and then throw away the object.

We will always run the application with the JVM option: “-Xmx64M” which basically means we can’t use more than 64MB during our test runs and this allows us to force the GC to kick-in more often. Our slow/bad application can be as simple as:

/**
 * Call a function in a loop and this function will create a large object
 * and calculate something complex (log(n)) and then throw away the object.
 */
public class BadApp1 {

    /**
     * Generate a large array of integers and then calculate which is the
     * maximum value and returns that releasing the large object previously
     * created.
     */
    public static int maximum_random() {
        int[] random_integers = new int[10*1024*1024];
        int maximum = random_integers[0];

        for (int value : random_integers) {
            if ( maximum < value ) maximum = value;
        }

        return maximum;
    }

    public static void main(String[] args) {

        long durations = 0;
        long duration = 0;
        long start, stop;
        for(int i = 0; i < 100; i++) {
            start = System.currentTimeMillis();
            maximum_random();
            stop = System.currentTimeMillis();
            duration = (stop-start);
            durations += duration;
            System.out.println("calculation took " + duration + "ms");
        }

        System.out.println("average duration was " + durations/100 + "ms");
    }
}

When we start analyzing how much time it takes to run that program and in each iteration print out how long we spent calculating the maximum value along with running with the following options:

java -Xmx64M -XX:+PrintGCApplicationStoppedTime BadApp1

The -XX:+PrintGCAppliactionStoppedTime tells the JVM to print out exactly how much time is the GC taking and not allowing the application to do its work. Here is a quick output of what I’m currently getting on my system:

>java -Xmx64M -XX:+PrintGCApplicationStoppedTime BadApp1 ... Total time for which application threads were stopped: 0.0117340 seconds calculation took 68ms Total time for which application threads were stopped: 0.0116940 seconds calculation took 69ms average duration was 69ms

Now we can see that we’re spending a good 0.012s in garbage collection code on every single iteration, which is basically 12ms of time in each of those 69ms of average time spent calling that “bad” function.

Before we start trying out different garbage collectors and tweaking their settings you should read some of the following links first:

http://www.oracle.com/technetwork/systems/index-156457.html
http://www.petefreitag.com/articles/gctuning/

if you want a better explanation of how the various garbage collectors work.

So getting back to our situation we’re suffering a 12ms overhead from the GC in every execution of ours and would like to reduce that by tuning the GC. The first idea might be to try and use the Parallel Copy Collector and see if this reduces the over head slightly:

> java -Xmx64M -XX:+UseParNewGC -XX:+PrintGCApplicationStoppedTime BadApp1 ... Total time for which application threads were stopped: 0.0110810 seconds calculation took 64ms Total time for which application threads were stopped: 0.0075140 seconds calculation took 62ms average duration was 65ms

A very minimal improvement which isn’t totally new since the only thing being done by this garbage collector is to have a thread per core when collecting the young generation garbage. Since my machine is a dual core machine and there really is only 1 object to garbage collect in my code then I don’t expect almost any improvement.

For the particular application that we wrote could benefit from a garbage collector that has to deal with a large young generation and as the documentation states:

“Use the throughput collector when you want to improve the performance of your application with larger numbers of processors. In the default collector garbage collection is done by one thread, and therefore garbage collection adds to the serial execution time of the application. The throughput collector uses multiple threads to execute a minor collection and so reduces the serial execution time of the application.”

So lets see what the benefit is like:

>java -Xmx64M -XX:+UseParallelGC -XX:+PrintGCApplicationStoppedTime BadApp1 ... Total time for which application threads were stopped: 0.0042150 seconds calculation took 46ms Total time for which application threads were stopped: 0.0053080 seconds calculation took 45ms average duration was 48ms

Now we’re 30% faster just by making a better choice in the GC being used. We’ve gotten this “additional boost in performance” by now spending just 4ms per GC.

In general you don’t go about fiddling with the garbage collector unless you’ve noticed your application isn’t behaving as you would expect it to and that you’re unable to guarantee stable performance behavior and after some investigation have found that the GC is in fact to be “blamed”. There are plenty of other situations in which tuning the GC can lead to a better performing Java application but I just wanted to show that even a badly written application can be “fixed” by a small change to the GC that is being used.

Adding comments to your blog with disqus

2011-12-25T00:00:00+00:00

This is just a quick write up on how easy it is to add comments to your blog or site with Disqus. You’ll need to create an account at disqus. Once that is done you can go to your dashboard and add a new site for the specific site you want to track comments on. The last step is to simple follow the instructions at Install section within your new sites admin page.

In my case all I had to add was:

<div id="disqus_thread"></div> <script type="text/javascript"> /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ // required: replace example with your forum shortname var disqus_shortname = 'example'; /* * * DON'T EDIT BELOW THIS LINE * * */ (function() { var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js'; (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); })(); </script> <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript" >comments powered by Disqus.</a></noscript> <a href="http://disqus.com" class="dsq-brlink">blog comments powered by <span class="logo-disqus">Disqus</span></a>

Once I replaced the ‘example’ disqus shortname with the shortname of my site everything worked perfectly. The only thing you’ll find is that if you have a static blog with jekyll as I do you won’t be able to see the comments on the static site when you startup jekyll locally with the “jekyll –server” command.

Memoize decorator for python

2011-12-21T00:00:00+00:00

I ran into a situation when writing some code recently where I had to add caching to an existing function in order to speed up the execution of my function when it was called multiple times with the same arguments. I know from experience that this involves a technique called memoization in which you basically cache the previous result and when the same arguments are passed to your function you pull from the cache instead of re-executed the same code that takes a while and would render the exact same result.

Adding the memoization/caching feature to your function isn’t hard but immediately I found it made the whole function look ugly and cluttered, and I was not satisfied with that. Luckily I was using python which has a feature called decorators which allow you to easily add more “functionality” to an existing function without cluttering its “code space”.

As any good engineer the first thing to do was to look around and see if someone had already created a memoize decorator for python and I quickly found there was a decent example in the python documentation but it lacked quite a few features including the ability to handle functions with lists or other objects as arguments. So I set out to create my own memoize module that could be easily used in other projects without having to clutter my code with “memoization” code.

I built the memoize module that is now available here and the README included has enough information on how to use this in your own project.

Optimizing Haskell Programs

2011-11-14T00:00:00+00:00

Before we start optimizing the wc command line tool we wrote lets first find a simple way to compare this implementation with the wc command tool available on my linux (written in C). I created a file with 197,000 dummy lines (with each line just over 80 characters long) and measured how long it takes to count the number of lines, with each tool:

time bash -c "cat test | wc -l" 197000 bash -c "cat test | wc -l" 0.00s user 0.02s system 65% cpu 0.043 total time bash -c "cat test | ./wc.hs -l" 197000 bash -c "cat test | ./wc.hs -l" 1.46s user 0.07s system 95% cpu 1.590 total

So the current Haskell implementation is 37x slower than the native C version. The first thing to note is how running the haskell program without compiling is not efficient at all. So lets put together a simple make file and use ghci to compile the .hs file to a native executable. Here’s a possible make file:

init:
    mkdir -p build
    cp *.hs build

wc: wc.hs init
    cd build ; ghc --make wc.hs -o wc

clean:
    rm -fr build *.o *.hi

So after a simple compilation the results we’re now getting are:

time bash -c "cat test | build/wc -l" 197000 bash -c "cat test | build/wc -l" 0.96s user 0.04s system 95% cpu 1.049 total

Which puts at at 24x slower which is already some progress with absolutely no code changes. Now the ghc compiler also allows you to use a few optimizing flags that can help make the output code quicker. Lets use the basic -O2 optimization and see how much we can gain. We’ll actually add the compilation directive to the .hs file with the following line:

#! /usr/bin/env runhaskell
{-# LANGUAGE DeriveDataTypeable #-}
{-# OPTIONS_GHC -O2 #-}
..._removed other code from here

Now after recompiling, here is the current performance of our tool:

time bash -c "cat test | build/wc -l" 197000 bash -c "cat test | build/wc -l" 0.96s user 0.04s system 95% cpu 1.049 total

Not much of a boost really and its not a surprise since our program isn’t very complicated we can’t expect the compiler to be able to save time that easily. There are a few things to look at before we start profiling and here’s a list of the usual suspects when it comes to bad performance in Haskell:

String is painfully slow and is known for being 20x slower than a similar C implementation. The fix is to use the ByteString type which is known to only be 2x slower than a similar C implementation.
read and show are known to also perform badly and you should use the same functions that manipulate the ByteString datatype.

So lets start by importing the required library to use the ByteString datatype and using all of the functions that handle ByteString. Be aware that it can be a bit of hassle to get your functions working with the ByteString data type and quite a bit of work to get all the types lined up just write, but here’s what the wc command implementation looks like that now uses ByteStrings:

#! /usr/bin/env runhaskell
{-# LANGUAGE DeriveDataTypeable #-}
{-# OPTIONS_GHC -O2 #-}

module Main (main) where

import System....
import Control.Arrow
import qualified Data.ByteString.Char8 as C

data WC = WC {chars :: Bool, lines_ :: Bool, words_ :: Bool}
    deriving (Show, Data, Typeable)

wc = WC { chars = def &= name "m" &= help "print the byte counts",
          lines_ = def &= help "print the character counts",
          words_ = def &= help "print the word counts" }
        &= help ("Print newline, word, and byte counts for each FILE, " ++
                 "and a total line if more than one FILE is specified." ++
                 " With no FILE, or when FILE is -, read standard " ++
                 "input.  A word is a non-zero-length sequence of " ++
                 "characters delimited by white space.")
        &= summary "wc v0.0.1, (C) Rodney Gomes"

countwords = C.pack . show . length . C.words
countlines = C.pack . show . length . C.lines
countchars = C.pack . show . C.length

space = C.pack " "
flat (a,(b,c)) = C.concat( a : space : b : space : c : [] )

addnl x = C.concat( x : (C.pack "\n") : [])
optionHandler WC{chars=True} = addnl . countchars
optionHandler WC{lines_=True} = addnl . countlines
optionHandler WC{words_=True} = addnl . countwords
optionHandler _ = addnl . flat . ( countlines &&& countwords &&& countchars )

main = cmdArgs wc >>= C.interact . optionHandler

Just a few tweaks really in terms of the functions being used and how to handle the new ByteString datatype. I also realized that the tool wasn’t outputting a necessary newline at the end of the output so I added that. After this small change surprisingly enough we’re now really close to the C implementation speed:

time bash -c "cat test | build/wc -l" 197000 bash -c "cat test | build/wc -l" 0.02s user 0.04s system 77% cpu 0.073 total

Now we’re still 69% slower than the C implementation which means we have room for improvement, but the interesting part is we’re actually 72% faster than the C implementation when we have to calculate the number of lines, words and characters at the same time:

time bash -c "cat test | wc" 197000 3743000 19306000 bash -c "cat test | wc" 0.57s user 0.02s system 95% cpu 0.613 total time bash -c "cat test | build/wc" 197000 3743000 19306000 bash -c "cat test | build/wc" 0.30s user 0.04s system 94% cpu 0.357 total

Now we’ve reached the point where if we want to get our counting of lines to perform as well as the C implementation we’re going to have to profile our Haskell program. To profile we need to compile with a few additional flags:

wc: wc.hs init
    cd build ; ghc -prof -auto-all -rtsopts --make wc.hs -o wc

We added the -prof -auto-all to build with profiling enabled, the -auto-all generates cost centres for all top level functions, you can read more about that in the Haskell documentation on profiling. When you try to run the make wc again if you get something like so:

wc.hs:7:8: Could not find module `System....': Perhaps you haven't installed the profiling libraries for package `cmdargs-0.9'? Use -v to see a list of the files searched for.

Just run the cabal install command like so for each package:

cabal install --reinstall -p cmdargs

That will reinstall the package and make sure to compile the required profiling information. You can now run the same command like so:

wc -l +RTS -p -RTS

You’l now have a nice wc.prof file to look at which contains information like this in it (slightly reformatted to fit):

Mon Nov 14 19:05 2011 Time and Allocation Profiling Report (Final) wc +RTS -p -RTS -l total time = 0.04 secs (2 ticks @ 20 ms) total alloc = 13,593,584 bytes (excludes profiling overheads) COST CENTRE MODULE %time %alloc main Main 50.0 1.3 countlines Main 50.0 98.5 individual inherited COST CENTRE MODULE no. entries %time %alloc %time %alloc MAIN MAIN 1 0 0.0 0.0 100.0 100.0 main Main 358 3 50.0 1.3 100.0 99.9 optionHandler Main 361 1 0.0 0.0 50.0 98.5 addnl Main 363 1 0.0 0.0 0.0 0.0 countlines Main 362 2 50.0 98.5 50.0 98.5 wc Main 360 0 0.0 0.0 0.0 0.0 CAF Main 352 33 0.0 0.0 0.0 0.0 addnl Main 364 0 0.0 0.0 0.0 0.0 wc Main 359 1 0.0 0.0 0.0 0.0 CAF Data.Typeable 350 5 0.0 0.0 0.0 0.0 CAF GHC.Show 348 1 0.0 0.0 0.0 0.0 CAF Data.HashTable 290 3 0.0 0.0 0.0 0.0 CAF GHC.IO.Handle.FD 288 3 0.0 0.0 0.0 0.0 CAF GHC.IO.FD 272 4 0.0 0.0 0.0 0.0 CAF GHC.IO.Handle.Internals 252 1 0.0 0.0 0.0 0.0 CAF GHC.IO.Encoding.Iconv 246 2 0.0 0.0 0.0 0.0 CAF GHC.Conc.Signal 243 1 0.0 0.0 0.0 0.0 CAF Data.Data 227 3 0.0 0.0 0.0 0.0 CAF System.....Implicit.Global 226 3 0.0 0.0 0.0 0.0 CAF System.....Implicit.Reader 223 1 0.0 0.0 0.0 0.0 CAF Data.Generics.Any.Prelude 222 2 0.0 0.0 0.0 0.0 CAF System.....Explicit 209 6 0.0 0.0 0.0 0.0 CAF System.....Explicit.Type 202 1 0.0 0.0 0.0 0.0 CAF System.....Explicit.Help 189 1 0.0 0.0 0.0 0.0 CAF System.....Implicit.Ann 187 4 0.0 0.0 0.0 0.0 CAF System.....Annotate 186 1 0.0 0.0 0.0 0.0 CAF Data.ByteString.Char8 184 1 0.0 0.0 0.0 0.0

With the above we can now see that we’re spending 50% of our time in countlines and the other 50% in the main function which is most likely in the interact function reading the input and 50% of the time parsing and counting in the countlines. Since we don’t have access to the interact function what we can do is create cost centres for the countlines function and see if it identifies something we weren’t expect. So we’ll add this to our source:

countwords = C.pack . show . length . C.words
countlines = {-# SCC "C.pack" #-} C.pack . {-# SCC "show" #-} show . \
             {-# SCC "length" #-} length . {-# SCC "C.lines" #-} C.lines
countchars = C.pack . show . C.length

So when we recompile and run with the above cost centres we can then get more detail from profiling:

COST CENTRE MODULE no. entries %time %alloc %time %alloc$ $ MAIN MAIN 1 0 75.0 0.5 100.0 100.0$ C.pack Main 358 1 0.0 0.0 25.0 99.4$ show Main 359 1 0.0 0.0 25.0 99.4$ length Main 360 1 0.0 0.0 25.0 99.4$ C.lines Main 361 1 25.0 99.4 25.0 99.4$

We can see that we’re spending about 25% of our time each of the places we set those cost centres. The thing to do now is try to understand why it takes 25% of the time to do the pack call or the show call when they’re just converting an Int to String and then to a ByteString. I just wanted to show how to profile an existing tool and that Haskell can in fact be as quick as C without having to any major changes and using the right types.

So we’ve pretty much optimized our wc implementation and the only thing left to do is a detailed comparison of the performance of our implementation vs the C implementation:

Counting lines in a file with 500,000 lines where all lines have 80 columns:

Ref Implementation: 0.084s
Our Implementation: 0.149s

Counting words in a file with 500,000 lines where all lines have 80 columns:

Ref Implementation: 1.243s
Our Implementation: 0.742s

Counting chars in a file with 500,000 lines where all lines have 80 columns:

Ref Implementation: 1.224s
Our Implementation: 0.116s

Writing wc command line tool in Haskell

2011-11-13T00:00:00+00:00

Now after having gotten a pretty good overview of how to use Haskell’s most important features we can now use this newly learned skills to write a clone of the ‘wc’ command line tool completely in Haskell and compare this to what the current code for the wc command line tool is written in.

The ‘wc’ command has the following help menu on my linux box:

$ wc --help Usage: wc [OPTION]... [FILE]... or: wc [OPTION]... --files0-from=F Print newline, word, and byte counts for each FILE, and a total line if more than one FILE is specified. With no FILE, or when FILE is -, read standard input. A word is a non-zero-length sequence of characters delimited by white space. -c, --bytes print the byte counts -m, --chars print the character counts -l, --lines print the newline counts --files0-from=F read input from the files specified by NUL-terminated names in file F; If F is - then read names from standard input -L, --max-line-length print the length of the longest line -w, --words print the word counts --help display this help and exit --version output version information and exit Report wc bugs to bug-coreutils@gnu.org GNU coreutils home page: http://www.gnu.org/software/coreutils/ General help using GNU software: http://www.gnu.org/gethelp/ For complete documentation, run: info coreutils 'wc invocation'

So we’re going to try and implement the counting of characters, lines and words and also make sure that we can handle piping of a file to the stdin with our newly created command line tool.

First thing that is required to write a command line tool is a command line parsing library. I choose CmdArgs since I found it worked really well and allowed you to easily define the help menu and summary quite nicely. You can read up on this library here.

The CmdArgs library is really great at expressing what arguments are available and what each of them do. You have to firstly define the datatype that identifies the various arguments and their types, like so:

data WC = WC {chars :: Bool, lines_ :: Bool, words_ :: Bool}
    deriving (Show, Data, Typeable)$

So the above just defines that we have 3 flags that can be used and each of them are either present or not (ie Bool type). The underscore following the name is used whenever you have a namespace collision such as the fact that lines and words are both functions that already existing in Haskell. CmdArgs will automatically strip the underscore when parsing the command line.

We now have to instantiate our WC data type and fill in the required information on what each option does as well as give some additional information on what the tool does and how to use it. Here is how this is done in Haskell when using CmdArgs:

wc = WC { chars = def &= name "m" &= help "print the byte counts",
          lines_ = def &= help "print the character counts",
          words_ = def &= help "print the word counts" }
        &= help ("Print newline, word, and byte counts for each FILE, " ++
                 "and a total line if more than one FILE is specified." ++
                 " With no FILE, or when FILE is -, read standard " ++
                 "input.  A word is a non-zero-length sequence of " ++
                 "characters delimited by white space.")
        &= summary "wc v0.0.1, (C) Rodney Gomes"

The above is using the &= operator is used to annotation the chars type with additional information such as the name of the option and the help to show when displaying this option. The whole WC datatype is annotated with the help and summary which are then used to generate the help menu like so:

wc v0.0.1, (C) Rodney Gomes wc [OPTIONS] Print newline, word, and byte counts for each FILE, and a total line if more than one FILE is specified. With no FILE, or when FILE is -, read standard input. A word is a non-zero-length sequence of characters delimited by white space. Common flags: -m --chars print the byte counts -l --lines print the character counts -w --words print the word counts -? --help Display help message -V --version Print version information

There you have your argument parsing and menu printing all in less than 15 lines of Haskell. The next bit we’re going to focus on is how do we actually count lines, words and characters using Haskell. Instead of writing functions that can calculate the number of lines/words in a String we can easily look through the Prelude module and find that there are two functions that do the trick: lines and words and using the already familiar function composition we can write:

countlines = length . lines
countwords = length . words

Now creating a main function in Haskell requires you understand a few things about the way IO is handled. In Haskell as mentioned early in my posts is a purely functional language and for things such as IO which are basically side effects within your function Haskell. This side-effect is handled by expressing side effects as a Monad. Now I won’t go into all of the details of a Monad in this post and I suggest you read up on Monads with these few links:

Yet Another Monad Tutorial is a great source of informatoin but its very detailed and runs over the course of a few posts
Monads for functional programming the original Monad paper that expresses how to use Monads within a functional language so you can do impure actions within a pure function.

The quick and dirty introduction to Monads is tha they’re used to do a few things that we take for granted in other imperative languages, such as: exceptions, state, output. In the case of the main function has an IO () which just means that this function generates output and returns the unit type () which I usually view as void in Haskell. Lets look at a simple program that prints ‘Hello World’:

main = putStrLn "Hello World"

The function putStrLn of course has a type of

putStrLn :: String -> IO ()

The other thing to introduce is the do notation which is heavily used along side Monads because it better expresses the imperative sense of the Monadic actions. Its not the only way to express monadic actions but seems to be the easiest to start with for developers coming from the imperative world. I would read up on the other available options when trying to chain monadic actions because you’ll find that Haskell has very elegant ways of handling this. The notation itself allows you to execute separate statements that are not necessarily used when generating the output of your function. For example:

main = do
         putStrLn "Input name:"
         name <- getLine
         putStrLn ("Hi there " ++ name)

There you can see some Haskell code that looks extremely like an imperative program. Now you don’t have to write things like that and can in fact be more functional when writing code in Haskell and do the same like so:

main = putStrLn "Input name:" >> getLine >>= putStrLn . (++) "Hello there "

There are a few new functions being used here which you may already know if read the tutorials on Monads I previously mentioned but they’re not hard to understand the first is » which has the signature:

(>>) :: (Monad m) => m a -> m b -> m b

What it does is simply accept two monadic actions and only return the second, basically ignore the return from the first function. This was necessary since we wanted to print the “Input name:” string but didn’t care for its return. The »= function on the other hand is the function composition function for Monadic functions. Its signature is more familiar:

(>>=) :: (Monad m) => m a -> (a -> m b) -> m b

You can easily see what the function does and its purpose with in Haskell. Then the only other magic done in that one line was to infix the ++ operator and use it to concatenate the return of the getLine with the “Hello there “ string. I really prefer not using the do notation when possible just because its not as functionally elegant as other available options. I think its a matter of taste and you’ll find what makes more sense to use in different situations.

So the only thing missing is the ability to handle the input from the standard input from our program in our program. In the Prelude there is a function that allows us to handle the stdin and output directly to the stdout. This is the interact function:

interact :: (String -> String) -> IO ()

This function will take your String->String function and do the required IO () output. The input to your function is the whole of the standard input and what you’ll be returning is the whole of what you want to be printed on the screen (ie standard output). So you can basically apply the countlines function like so:

module Main (main) where

countlines = show . length . lines
main = interact countlines

I just introduced the module declaration line to also show how to correctly define your module and tell the ghci compiler which function is your main entry point. We had to use the show function to convert the number calculated by our countlines function back into a String. With the above you should be able to run commands such as:

$ cat test.hs | runhaskell test.hs 5

With all that we’ve gone over at this point you should be able to write up the wc command line tool and it may look something like this:

#! /usr/bin/env runhaskell
{-# LANGUAGE DeriveDataTypeable #-}

module Main (main) where

import System.Console.CmdArgs
import Control.Arrow

data WC = WC {chars :: Bool, lines_ :: Bool, words_ :: Bool}
    deriving (Show, Data, Typeable)

wc = WC { chars = def &= name "m" &= help "print the byte counts",
          lines_ = def &= help "print the character counts",
          words_ = def &= help "print the word counts" }
        &= help ("Print newline, word, and byte counts for each FILE, " ++
                 "and a total line if more than one FILE is specified." ++
                 " With no FILE, or when FILE is -, read standard " ++
                 "input.  A word is a non-zero-length sequence of " ++
                 "characters delimited by white space.")
        &= summary "wc v0.0.1, (C) Rodney Gomes"

countwords = show . length . words
countlines = show . length . lines
countchars = show . length

flat (a,(b,c)) = " " ++ a ++ " " ++  b ++ " " ++ c

optionHandler WC{chars=True} = countchars
optionHandler WC{lines_=True} = countlines
optionHandler WC{words_=True} = countwords
optionHandler _ = flat . ( countlines &&& countwords &&& countchars )

main = cmdArgs wc >>= interact . optionHandler

That is a our implementation of the wc command line tool (minus the counting of bytes and some of the extended options). There are a few more things introduced in this implementation that weren’t covered before such as:

&&& operator, also called the ‘fanout’ operator which has the following signature (&&&) :: a b c -> a b c’ -> a b (c, c’) is basically used to apply the two functions to the same argument and return the result which consists of the tuples of the results of those two functions. We then created a flat function to flatten out the result of applying using the &&& operator.
records pattern matching - we had introduced the records notation which allows you to basically allows you to define the chars,lines_ and words_ functions which can be used against the WC datatype. Now when pattern matching you can use the same function to match the exact element you’re looking for and that’s what we’ve done above in the optionHandler function.

The most astounding thing about the above piece of code is how many lines we’ve actually had to write. The above code is exactly 34 lines of code (in its current incarnation) and the source for the current C implementation of the wc command line tool in the source of my current Ubuntu Oneiric installation is over 700 lines of code.

Of course lines of code is not a true way to compare software quality between different languages, but it is a measure of complexity. The biggest difference here is how easy it is to read this code vs reading the same program written in C. Just looking at the code above you can see how easy it is to add more functionality and also how easy it is to read the program.

In the next post we’re going to actually analyze the performance of the wc command we wrote and see how close we can get to the performance of the C implementation of the wc command.

Advanced Typeclasses

2011-11-08T00:00:00+00:00

Lets go back a bit to the typeclass topic and analyze and understand well a few important typeclasses that allow you to do a few intestine things. We’ll start with the Functor typeclass, which has the following definition:

class Functor f where
    fmap :: (a -> b) -> f a -> f b

We’ve introduced a new notation in our signature with the f a, in this simple signature we’re using a generic type constructor reference instead of a concrete constructor such as Maybe or from our previous post Tree. So when we create an instance of a Functor f we basically letting any function that wants to use the Functor how to apply and reconstruct our type. We can see how to easily implement a Functor instance for lists, like so:

instance Functor [] where
    fmap = map

Lets define the instance of Functor for the Tree data type we defined in the previous post:

instance Functor Tree a where
    fmap _ Leaf = Leaf
    fmap f (Node a l r) = Node (f a) (fmap f l) (fmap f r)

With the above we can now easily apply higher order functions to our Tree instances like so:

Main> fmap (+2) sampleTree 3 4 . . 5 6 . . .

This is where I believe you should really start to see the beauty in the way that Haskell works with your data types and being able to easily modify, transform your types. There are a few other class types to look at which include Foldable, Traversable, etc.

We’re now going to have a look at the Show and Read class types. We’ve already seen the Show class type but we want to show the complete function here and understand a bit better how this works with the Read class type. So here’s the full definition of the Show class type:

class  Show a  where
    showsPrec        :: Int -> a -> ShowS
    show       :: a -> String
    showList         :: [a] -> ShowS

-- Mimimal complete definition:
-- show or showsPrec
    showsPrec _ x s   = show x ++ s

    show x        = showsPrec 0 x ""

    showList []       = showString "[]"
    showList (x:xs)   = showChar '[' . shows x . showl xs
                        where showl []     = showChar ']'
                              showl (x:xs) = showChar ',' . shows x .
                                             showl xs

As we did before we only have to implement the show function (don’t want to get into the showsPrec function at this stage). We’ll create a new binary tree data type and define a very simple Show instance for this, here’s what we’re looking at:

data BTree a = Leaf a | Branch (BTree a) (BTree a)

showBTree :: (Show a) => BTree a -> String
showBTree (Leaf a) = show a
showBTree (Branch l r) = "(" ++ showBTree l ++ "|" ++ showBTree r ++ ")"

instance Show a => Show (BTree a) where
    show a = showBTree a

sampleTree = Branch (Branch (Leaf 3) (Leaf 2)) (Leaf 1)

So when we do a show of the sampleTree we get the following output:

((3|2)|1)

Now we have to read the Read instance so that we can easily parse the expressions of a BTree back into a BTree type that we can then process with the various functions that’ll we’ll eventually write. The Read class iself has the following definition:

class  Read a  where
    readsPrec        :: Int -> ReadS a
    readList         :: ReadS [a]

-- Minimal complete definition:
-- readsPrec
    readList         = readParen False (\r -> [pr | ("[",s)  <- lex r,
                                                    pr       <- readl s])
                       where readl  s = [([],t)   | ("]",t)  <- lex s] ++
                                        [(x:xs,u) | (x,t)    <- reads s,
                                                    (xs,u)   <- readl' t]
                             readl' s = [([],t)   | ("]",t)  <- lex s] ++
                                        [(x:xs,v) | (",",t)  <- lex s,
                                                    (x,u)    <- reads t,
                                                    (xs,v)   <- readl' u]

As before we only need to implement the readsPrec function to have a working Read implementation. So lets also have a look at the ReadS type:

type ReadS a = String -> [(a, String)]

This type defines what a parse does which is to return for parts of the original string with the accompanying converted type. It also allows you to parse whatever is part of your Read implementation and return the rest of the string that wasn’t parsed.

Before we implement the Read instance for BTrees lets talk about list comprehension and how to use it. List comprenhension is another feature in Haskell that is implemented quite elegantly. List comprenhension has the following syntax:

[ expr | qualifier0 , ... , qualifierN ]

In which the expr defines the way the elements are composed within the list being generated and the qualifiers identify which elements are to be in the resulting list. So lets for example construct a list of all of the odd numbers from 1 to 100:

odd100 = [ i | i <- [1..100], odd(i) ]

Now here’s how we could start putting together out readTree function:

readsTree :: (Read a) => ReadS (BTree a)
readsTree ('(':s) = [ (Branch l r, u) | (l, '|':t) <- readsTree s,
                                        (r, ')':u) <- readsTree t ]
readsTree s = [(Leaf x, t)  | (x,t) <- reads s]

instance Read a => Read (BTree a) where
    readsPrec _ s = readsBTree s

Now the above function can be a bit to take in all at once but if you have a closer look its really not doing much more than stating that if a string starts with the ’(‘ symbol then a Branch l r can be parsed from this string in which the l comes from readSTree s and the returned string must start with the ’|’ and then continues with t which would be parsed to return the r side of the Branch and would leave you with at least the ’)’ symbol followed by the rest of the string (which may be empty). The last pattern of the function assumes everything else would be a Leaf x where is comes from reads of s.

This can now be easily be used to parse string back into the BTree type, like so:

Main> (reads "(1|(2|3))blah")::[(BTree Int,String)] [((1|(2|3)),"blah")]

We do have to tell Haskell what the type of the return is because otherwise Haskell can’t deduce if its a binary tree of integers or a binary tree of strings.

By now I would hope you’d be able to write Read and Show instances for any of your own new datatypes and be able to have Haskell really work for you.

Higher order functions in Haskell

2011-11-01T00:00:00+00:00

From Haskell’s documentation: “A higher-order function is a function that takes other functions as arguments or returns a function as result.”. With this in mind lets start by writing a few functions over lists:

addone :: [a] -> [a]
addone [] = []
addone (x:xs) = (x+1) : addone(xs)

divlist :: (Fractional a) => [a] -> a -> [a]
divlist [] _ = []
divlist (x:xs) n = (x / n) : divlist xs n

We could add a few others but the idea here is that we have to write a new function everytime we want to traverse a list of elements and apply a function to these elements while recreating the list in the same order. What we realy need is a function with the following signature:

applyf :: (a -> b) -> [a] -> [b]

You can see that the function takes a function that transforms a to b’s and a list of a’s to create a list of b’s. We can even define the applyf function quite easily as:

applyf :: (a -> b) -> [a] -> [b]
applyf f [] = []
applyf f (x:xs) = (f x) : applyf f xs

There actually already exists an applyf function called map in the Haskell Prelude library. Lets show how to quickly redefine the addone and divlist function using the map function:

addone = map (+1)
divlist n = map (/n)

You can quickly see how to use map to basically apply any function to a list of items and get back the list of the results. We can now start to introduce a few other high order functions that are in the Prelude library:

filter - the filter function takes a boolean function as an argument and removes allelements from the input list that do not satisfy the boolean function. This is very useful when you want to quickly filter out elements based on a simple boolean function such as filtering a list of tuples in which the second element is the age and we the list to contain only the teenagers:

filterTeens = filter (\x -> snd(x) > 12 && snd(x) < 20)

foldl - reduces a list of elements down to a simple object by applying a function you supplied and a starting value. You can use this to quickly sum up a list of values or even concatenate a list of strings, here are a few examples:

sumlist = foldl (+) 0
concatlist = foldl (++) ""

foldr - same as foldl but starts the “folding” from the end of the list which can have a completely different meaning depending on the folding operation being used.

There are many other higher order functions that are extremely useful when wanting to apply transformations to lists. These functions can also be generalized to other structures such as the Tree data that we introduced in the previous post. We’ll leave the creation of a map, filter for Tree as an exercise for those who are interested in testing out their skills.

Custom data types in Haskell

2011-10-31T00:00:00+00:00

Hopefully after the first post in this series you can quickly write up simple functions over existing Haskell types and also understand basic polymorphism as well as function composition. We’ll now start digging into creating custom types and how to handle these in our functions. Lets start by creating a Tree type:

data Tree a = Leaf | Node a (Tree a) (Tree b)

When defining elements of the type above you can do so in the following manner:

sampleTree = Node 2 (Node 1 Leaf Leaf) (Node 3 Leaf)

Which represents the Tree:

      2
     / \
    /   \
   1     3
  / \   / \
 *   * *   *

We now want to write a function that can calculate the maximum depth of a Tree and when we write this function we must not forget to handle all of the cases that compose the data type Tree. Which means handling the Leaf and handling the Node x y z, like so:

maxdepth :: Tree a -> Int
maxdepth Leaf = 0
maxdepth (Node _ l r) = 1 + max (maxdepth l) (maxdepth r)

As you can see pattern matching on your custom data type is extremely easy to read and is just like doing so with any built in type. We introduced the use of the _ variable which is how you handle variables when you don’t care to use them in your functions calculations. We’re using the max function from Prelude to handle calculating the maximum of those two possible choices in a Tree.

Lets look at how to print an existing tree with in Haskell. So firstly if you try to just show the current sampleTree you’ll find that ghci shell will actually complain it can’t show this new element:

No instance for (Show (Tree Integer)) arising from a use of `print' Possible fix: add an instance declaration for (Show (Tree Integer)) In a stmt of an interactive GHCi command: print it

The above error is basically trying to tell you that Haskell doesn’t know how to “show” the data type you just created. One easy thing to do is to just let Haskell derive a basic representation for your type by adding the following to the declaration of the type:

data Tree a = Leaf | Node a (Tree a) (Tree b)
    deriving (Show)

With that you can now get a simple representation of your data like so:

Main> sampleTree Node 1 (Node 2 Leaf Leaf) (Node 3 (Node 4 Leaf Leaf) Leaf)

But you can also define exactly how you’d like to represent your trees. This is done by defining an instance of the class Show for your datatype and then defining the function show for your type, something like so:

padding :: (Num a) => a -> String
padding 0 = ""
padding n = " " ++ padding(n-1)

showTree :: (Show a, Num b) => Tree a -> b -> String
showTree Leaf n = (padding n) ++ "."
showTree (Node a l r ) n = let showl = showTree l (n+4) in
                           let showr = showTree r (n+4) in
                           let showc = (padding n) ++ (show a) in
                           showc ++ "\n" ++ showl ++ "\n" ++ showr

instance (Show a) => Show (Tree a) where
    show a = showTree a 0

I’ve introduced here the concept of typeclasses, they are heavily used in conjunction with polymorphism to better describe the type of elements that can be used with the current function. Typeclasses seem like class definitions but they’re much more powerful in the sense that you are defining an abstract operation that needs to be defined per type that wants to be usable by certain functions. In the code above you’ll notice how the padding function is expressing that it can accept any a as long as its an implementation of the typeclass Num.

Basically you’re telling Haskell that to show a Tree a type you want Haskell to represent it in the manner a specific manner by giving Haskell the definition of the show function you’d rather use. Now when we try to reference our sampleTree you’ll get a more readable representation like so:

Main> sampleTree 1 2 . . 3 4 . . .

If we wanted we could even extend this show implementation to draw a few ASCII lines and make the tree a bit easier to read. We’ll leave that as an exercise for the reader.

Lets find something more interesting function to write and we’ll start by writing up a insert function takes a Tree and an element and inserts the Tree with the new element inserted while at least making sure that the nodes are in order in the tree so that if we print the tree in order it will print the elements in order

insert :: (Ord a) => a -> Tree a -> Tree a
insert a Leaf = Node a Leaf Leaf
insert a (Node b l r) = if a < b
                            then Node b (insert a l) r
                            else Node b l (insert a r)

Again we used the typeclass Ord which is define as:

class  (Eq a) => Ord a  where
   compare              :: a -> a -> Ordering
   (<), (<=), (>=), (>) :: a -> a -> Bool
   max, min             :: a -> a -> a

        -- Minimal complete definition:
        --      (<=) or compare
        -- Using compare can be more efficient for complex types.
    compare x y
         | x == y    =  EQ
         | x <= y    =  LT
         | otherwise =  GT

    x <= y           =  compare x y /= GT
    x <  y           =  compare x y == LT
    x >= y           =  compare x y /= LT
    x >  y           =  compare x y == GT

-- note that (min x y, max x y) = (x,y) or (y,x)
    max x y
         | x >= y    =  x
         | otherwise =  y
    min x y
         | x <  y    =  x
         | otherwise =  y

Basically the definition tells you that defining the compare function is enough to for the other functions be inferred from.

One last thing about defining data types is the ability to create synonyms for existing types. This is done using the type keyword and allows you to make your functions more readable. Here are a few examples:

type String = [Char]

type Name = String
data Address = None | Addr String
type Person = (Name,Address)

type StringList = [String]

Haskell Basics

2011-10-30T00:00:00+00:00

I will start my learning of Haskell by first going over how to write a simple program in Haskell and have it execute within the Haskell interpreter. Lets start with writing a simple “Hello World” application in Haskell:

#!/usr/bin/env runhaskell
module Main(main) where
main = putStrLn "Hello, World!"

That is a pretty small program even when you compare to writing hello world in an imperative language such as C. I added the shebang to make sure you’d be able to easily run this on a unix system. To run this you’ll need to install GHC compiler and interpreter which are supported on all major OS’s. For more information on getting GHC on your system have a look at: http://www.haskell.org/ghc/ .

Lets get into what makes a piece of code in Haskell. So unlike most mainstream languages that are imperative or object-oriented, functional programming languages do not have the notion of a sequence of commands/instructions to execute. A functional programming language consists of a collection of functions that do a single task and return a result without any side-effects. I’m bringing up the topic of side-effects so we can understand early on that you can’t do any of the following in a function (by default):

throw an exception, raise an error
read/write to any file (including stdin/stdout)
change global state (global variables)

Side-effects are not “easily” represented as a pure function in a functional programming language.

Lets write our first function that takes a number and gives this number plus one:

plus1 a = a + 1

When writing your functions make sure to always write the signature of the function. Writing the signature helps you understand what you’re trying to write as well verifies that you haven’t created any scenario in which your function returns an unexpected result:

plus1 :: Int -> Int
plus1 a = a + 1

As you look at this function you’ll find the notation is very similar to when you were writing mathematical expressions in school. Of course now that we have this function lets load it into the ghci shell and try to apply it to a few different values:

> ghci GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help Loading package ghc-prim ... linking ... done. Loading package integer-gmp ... linking ... done. Loading package base ... linking ... done. Prelude> :l test.hs [1 of 1] Compiling Main ( test.hs, interpreted ) Ok, modules loaded: Main. *Main> plus1 0 1 *Main> plus1 2 3 *Main> plus1 (plus1 2) 4 *Main>

Lets jump into deal with lists of data and how to write a few useful functions to deal with lists. The first thing to realize is that a list in Haskell is very simply represented as:

[1,2,3,4]

A string is a list of Char and therefore has the type [Char] and can be represented as:

['a','b','c']

which is in fact the string “abc”. Lists can also be expressed using the operator (:) which takes an element and adds it to the another list so the above list can also be presented as:

1:(2:(3:(4:[])))

With this operator we can also introduce pattern matching and how to write functions that can handle lists. Lets start by writing our very own length function that can calculate the length of a list. The first thing to write is the type of this function:

len :: [Int] -> Int

The above signature is already length takes a list of Ints and returns an Int. We can easily take this a step further and make this function polymorphic which allows it to be applied to a list of any type. The signature would look like so:

len :: [a] -> Int

The type a now represents any type that can be put in a list and with this we can write our length function, like this:

len :: [a] -> Int
len [] = 0
len (x:xs) = len xs + 1

This definition reads very simply: the length of an empty list is 0 and the length of a list with an element x and a tail xs is equal to the length of the tail plus one. We’ve introduced here how to do pattern matching on lists and also how polymorphism works when you want to create methods that can be used against various data types that share a common structure. The len function shown can be used against a list of integers as well as a string which is a list of Chars. Here’s an example:

... Prelude> :l test.hs [1 of 1] Compiling Main ( test.hs, interpreted ) Ok, modules loaded: Main. *Main> len [3,2,1,3,4,5,5] 7 *Main> len "Hello, World!" 13

Polymorphism is one of the other things I find that Haskell does really well when compared to other languages. All other languages refer to polymorphism as templates and generics and are in no way as elegant or simple to understand as polymorphism is in Haskell.

Most of the operations on lists you’ll ever need are already implemented in the Prelude library. Just search for “haskell prelude” and you’ll find all of the available functions, but here are a few of the everyday useful ones:

head - returns the head of a list
last - returns the last element of a list
tail - returns the same list without the first element
init - returns the same list without the last element

With these functions we can now start to talk about function composition. Which from algebra class when you had function f and function g and wanted to apply it to a single argument you’d write something like so:

(f o g) x

In haskell there is the composition function which can take two functions as its arguments and compose them together. This is how the definition of such a function might look:

(.) :: (b -> c) -> (a -> b) -> (a -> c)
f . g = \x -> f(g(x))

We’ve suddenly introduced 2 other concepts with this definition:

infix functions - By default functions are defined in a prefix manner which means the function name appears before the arguments. When you want to declare functions that will appears in the middle of their arguments such as the +,/,- operators then you need to declare them as you see above wrapped with brackets and then you can define them with the function name in the middle of the arguments. You can also turn any regular prefix function into an infix function by just putting single quotes around it like so: div is now an infix version of the prefix div function.
lambda expression - A lambda expression allows you to define an in place function in a more mathematical friendly format. You basically describe for each x what the function will do. So for example:

\x -> x + 2

declares a function that for each x passed as an argument, the function will returns x plus 2. Lambda expressions are great for saving space and not having to declare additional named functions when you just need a function to get the job done there and then.

I’ll end this post with a few examples of how function composition works and how it can be useful to write functions composed of other well known functions. To start lets take a few of the useful prelude list functions we shown above and try to write a few other useful list functions by using composition:

lets write a function that can give you the penultimate element of a list. We can simply call it penult
lets write a function that can give you the same list without the first and last element and we’ll call it middle

So for the penult function we can compose the functions init and last, the first will give us the list without the last element and the last call will give us the last element of that list which is the penultimate element, like so:

penult :: [a] -> a
penult = last . init

and the middle function is the composition of the init and tail functions, like this:

midddle :: [a] -> [a]
middle = tail . init

By now you can see that the Haskell language is extremely powerful and allows you to take existing functions and put them together in a very simple way that allows you to create new and useful functions quickly and cleanly.

In my next post I’ll start to dive into custom data types and how to pattern match those when writing your own functions and hopefully get into representing Trees, Graphs and other data types and how writing functions for those in a functional language is extremely clean and simple.

Back to Haskell

2011-10-29T00:00:00+00:00

So I recently decided to get back on the functional programming band wagon and get re-acquainted with haskell. Haskell is one of very few purely functional languages that really expresses programming in a very beautiful mathematical manner. I learnt Haskell early in university and was always fascinated by how simple and yet powerful the language was.

Of course, shortly after leaving school I found that functional programming languages were in no way mainstream or being used by software companies at large. This was of course in 2003 when I finished my masters and things have changed quite a bit in the past 8 years. In today’s programming scene there are a few functional programming languages that are being used quite extensively. Erlang and Scala seem to have found mainstream adoption in a few places and even my beloved Haskell has some usage behind the curtains at Facebook and Google.

Now I haven’t touched Haskell in ages but I thought I’d try and relearn Haskell and be able to write a few simple tools in Haskell. I will be documenting my progress in the form of a few blog posts and I hope to spike the interest of others in the language or at least make it easier for myself in the future when I want to review the language quickly.

Python performance module

2011-06-03T00:00:00+00:00

I just created a new Python performance module to easily give you performance information about important functions with in your existing Python code without having to litter your code with test code and checks here and there. The new library is called pyperf and it uses Python decorators to mark the methods you want to track performance statistics on.

I wrote this because I needed to do some performance measurement on another project that I’m going to put on my github account that is an old school project on l-systems and generating/rendering those l-systems into nice 3D images that look like organic objects (i.e. trees, bushes, shrubs in a game). I’ll write up another post on that library a little later for now I wanted to release this module in case anyone else finds it useful and wants to help extend and make it useful for themselves and others.

The pyperf library is very simple to use and you can start measuring the performance of your critical methods quickly. The first thing to do is decorate the functions you want with the pyperf.measure decorator. By default the pyperf library is not enabled and to enable it you have two choices:

Set the pyperf.PYPERF to True and this will make it so that the pyperf library is monitoring the function that have been previously decorated.
Sending the signal SIGUSR1 to the Python application you have decorated with the pyperf.measure decorat will enable/disable the pyperf library which will print the current state to the logs so you know if its turned on or off.

The performance report will be printed at the end of your codes execution when your Python application exits. If you’d like to get a report at runtime then send the SIGUSR2 signal to the Python application and the pyperf library will print the current report information.

There are a few configuration options that you have that can be set by changing the global variables:

pyperf.PYPERF_TRACKARGUMENTS - when set to True the pyperf report will track the function calls by the arguments to the same function and separating the results by the arguments.

pyperf.PYPERF_TRACKCALLER - when set to True the pyperf report will track the function calls by the caller to the function you decorated.

There are still a few features I’d like to implement which I’ve added to the github issues of this repository and I think that configuring the library could be done in a nicer way but for now this is a first good step in the direction of creating a nice and clean performance library for any Python code.

Android Development and Ant

2011-03-09T00:00:00+00:00

When thinking about automating the building/testing of an Android project I ran across the necessity to do so from the command line and not using Eclipse. After some searching I found that its quite easy to get this done with the tools that Android has already available which can generate a simple build.xml file to be use for building/installing/packaging an Android project.

To build your Ant build file you can use simply the following from the root of your project:

android update project -p .

Now if you need to link additional Android projects to this one you can do so with the following command:

android update project -p . -l path/to/other/android/project

To see the available targets just issue the ususal: ant -p and you’ll get all of the available targets:

Main targets:

 clean      Removes output files created by other targets.
 compile    Compiles project's .java files into .class files
 debug      Builds the application and signs it with a debug key.
 install    Installs/reinstalls the debug package onto a running emulator or
            device. If the application was previously installed, the signatures
            must match.
 release    Builds the application. The generated apk file must be signed before                             it is published.
 uninstall  Uninstalls the application from a running emulator or device.
Default target: help

Fixing Eclipse Content Assist

2011-03-06T00:00:00+00:00

If you use eclipse like I do for developing anything form Python scripts to Android projects you may have noticed that recently your content assist feature is either really slow or literally freezes up eclipse. Well you’re in luck because the reason its most likely doing this is because of the ADT Plugin that you’re using to do Android development.

So it seems there are a few bugs open on the subject but put simply the content assist tries to look for the Android source code under each of the Android folders at sdk_location/platforms/android-x (where x is the android level). When the content assist can’t find the sources folder then it just goes nuts and wastes alot of cpu cycles, the easy fix is to create a simple sources folder under each of those android-x folders and your content assist will be back to behaving as usual.

The longer and less easy fix is to download the source for each of those builds and place it there, but I really don’t think the majority of developers care for all of the source code.

New look and feel for DTF documentation

2011-01-28T00:00:00+00:00

Spent sometime cleaning up the way I generate DTF documentation so that it outputs the HTML elements with the required class and id attributes. With this I am now able to correctly style the DTF generated documentation to make it a little easier on the eyes. As you can see from a few comparative screenshots below there are a few good improvements and I’ll definitely be working on cleaning this up further but I’m pretty happy with the current results.

old index page

new index page

old tag documentation page

new tag documentation page

old feature documentation page

new feature documentation page

Python decorators

2011-01-22T00:00:00+00:00

I recently discovered decorators in Python and have been pretty impressed with how they work. Basically it allows you to change the behavior of a function or method without having to alter the code that represents that function. Instead with a decorator you basically label/annotate the method with a keyword that basically identifies which decorator to apply to this function.

Lets look at the simple example of you having creating a bunch of functions that do different tasks and then when you’re running in DEBUG mode you’d like to be able to get the amount of time spent on each of these calls. Lets first piece together a simple Python module with a few functions that do some simple tasks in it, for this example I’ve decided to create a simple logger module:

def i(msg):
    print("I: %s" % msg)

def e(msg):
    print("E: %s" % msg)

def d(msg):
    print("D: %s" % msg)

def w(msg):
    print("W: %s" % msg)

There are four simple methods that log info, error, debug and warning messages and now we can design our perf module that will be used to decorate these methods so we can do things such as counting the occurrences of method calls or even just calculate the average time spent on these calls at runtime. Lets start by putting together the decorator that can calculate how much time we spent on each individual call.

So a decorator is nothing more than a function that accepts another function as an argument. The very basic decorator definition looks like so:

import time

def track(f):
    def new_f(*args, **kwargs):
        t = time.time()*1000
        ret = f(*args, **kwargs)
        dur = (time.time() * 1000) - t
        name = f.__name__
        print("%s executed in %dms" % (name, dur))
        return ret

    return new_f

This is a decorator defined as a function you basically have to return a function as part of the contract of defining a decorator this way. In our case we create a simple function that wraps the existing one and return that. In this new function we’re just making the same call as your code intended but measuring the time spent. The args are the arguments passed to the original function and the kwargs are the keyword arguments passed to the original function.

Now Lets put together a silly test that simply logs a bunch of lines with our test logger. Here is this test:

import logger

logger.i("just a message")
logger.e("oh crap!")
logger.d("debug some stuff")
logger.w("you should have a look a this")

We’re ready to go back to the original module and add the @track decorator to each of our calls. Our module will look like so:

from perf import track

@track
def i(msg):
    print("I: %s" % msg)

@track
def e(msg):
    print("E: %s" % msg)

@track
def d(msg):
    print("D: %s" % msg)

@track
def w(msg):
    print("W: %s" % msg)

When we execute our same test module we’ll now see the following in the logs

> python test.py
I: just a message
i executed in 0ms
E: oh crap!
e executed in 0ms
D: debug some stuff
d executed in 0ms
W: you should have a look a this
w executed in 0ms

Of course the amount of time spent is less than 0ms and that’s not a surprise but what you can see here is that we now have the ability to track the performance of these calls by simply marking them with the decorator keyword.

Lets take sometime to make that decorator extra smart and have it turned off/on based on a global DEBUG flag. The end result should look like so:

import time

global DEBUG
DEBUG = False

def track(f):

    global DEBUG

    if ( DEBUG ):
        def new_f(*args, **kwargs):
            t = time.time()*1000
            ret = f(*args, **kwargs)
            dur = (time.time() * 1000) - t
            name = f.__name__
            print("%s executed in %dms" % (name, dur))
            return ret

        return new_f
    else:
        return f

This actually shows that you can make a decorator that adds no overhead when it is turned off because it just returns the same function that was already in place before the decorator had been used. Basically we can create a set of performance measuring decorators that have a negligible impact on our code when it running in production.

Of course there are more things that can be done with Python decorators but I just wanted to write up the starting point for future reference and hopefully inspire others to come up with some brilliant uses of Python decorators.

I created a small Eclipse project while writing this entry and you can get it from here if you’d like.

Timetracker with toggl support

2011-01-21T00:00:00+00:00

The timetracker tool is now using the online time tracking service toggl as the default tracker and will automatically create new projects by the name of the activity you’ve given your current task and file all of those tasks under the workspace timetracker. You can change the workspace name in your configuration file.

Once you’ve started tracking your tasks you’ll be able to open up toggl and see some pretty awesome pie charts of where you’re spending your time.

Of course it will be nicer to make more interesting charts/graphs once toggl matures a bit further but for now at least you can use timetracker to really figure out where you spend most of your time when you’re on your laptop.

Working on timetracker

2011-01-17T00:00:00+00:00

Been working on a time tracking utility for a while now and made timetracker to basically track what I do on a daily basis on my gnome desktop. From being able to tell when I’m chatting with a friend to on the web doing social interactions on facebook or twitter. The application works very well for me and has been built so you could easily extend it to monitor your Mac or Windows desktop by creating your own windowmanager instance.

The reporting output has also changed significantly from starting with a default of using the hamster time tracking utility to moving to an online toggl service which can handle the amount of data generated on a daily basis by my usage patterns. The hamster applet wasn’t designed to handle a few thousand task changes per day and was really having issues keeping up. I filed a bug here in the hopes that gets fixed in the future. Meanwhile the default tracker is the toggl tracker.

toggl offers a free account for tracking your activities and seems to have a simple and clean interface. I still need to figure out exactly how to inject the tasks into each project/workspace in a way that makes toggl’s stats and graphs interesting to the end user.

If you’re interested in trying out timetracker, then start by reading the README at the base of the project and it will let you know how to install on linux system. For those of you interested in using this on another OS please contact me so we can work out how to achieve this.

My First Post

2011-01-17T00:00:00+00:00

Just got the blog up and running with Jekyll and thought I’d post a quick note to see how things look. This is nothing fancy and just makes tracking my time and interest in projects very easy since adding a new post is creating a simple text file and checking it in.

Hopefully I will be able to write more on the things I’m currently working on in terms of technology as well as some of my hobbies which include music and piano playing. With time I’ll make things look a little nicer but I really want something simple and easy on the eyes.

Continuous Integration Environment for DTF

2011-01-17T00:00:00+00:00

I’ve been working on building a CI environment for DTF for a while now and I’m at the stage where the CI environment I put together work quite well for me. I am using Amazon’s EC2 to host my CI server which has a few simple scripts that when the EC2 instance is started will do the following:

checkout my DTF source tree
build DTF
generate the DTF Documentation (JavaDoc documentation that generates documentation for all tags in DTF) to here
run the JUnit tests & DTF unit test suite and post the result to here
run the DTF performance verification tests and post the results to here
shutdown the EC2 instance

Takes about 20 minutes to run through everything and the instance itself stays up and running for an additional 10 minutes (in case I need to ssh into the instance for any maintenance).

There are quite a few tools/technologies that were used to achieve the above and I’ll now try to give a very simple explanation of each one used:

I choose to use Amazon’s EC2 because it was really easy to get an instance up and running on EC2 and my only issue was how do I get that instance to start and stop on demand. After some investigation I wrote up a simple tool that would generate the HTTP POST/GET calls that I could use to start and stop my instances. This tool is available here and can be used by anyone else wanting to remotely administer their EC2 instances.
The result tracking of the JUnit tests and DTF’s unit test suite are all done by outputting results into a JUnit XML format and then using the JUnit report task in Ant to generate the simple HTML reports.
The graphs generated from executing DTF’s performance verification tests are generated using Google’s Charting API. After poking around at the API for a while I basically made the DTF performance tests record their performance results by appending to existing event files and then outputting at the end of the PVT execution all of the performance results gathered to a file that could be executed to download all of the chart PNG files and then push those to github to be shown in a wiki page.

There are still a few things that need some cleaning up in terms of polish and also I would like to include the DVT (Deployment Verification Tests) to this automated testing but for now I am quite happy with how easily I can validate my changes to DTF do not break anything in an unpredictable manner.

Introducing DTF

2011-01-17T00:00:00+00:00

DTF is my testing framework that has been in development for 4+ years and has survived my transition from Sun Microsystems to Yahoo. It was open sourced since May 2010. Now that I have my blog up and running I thought I’d make one of my first posts be an introduction to DTF and what it can do.

To start DTF stands for Distributed Testing Framework and it is written in 100% Java so that it can be easily executed on any environment where Java is supported. The tests in DTF are written in XML and unlike some of the XML languages you’ve seen its sort of a mash-up of Ant + Jelly + Some additional features you’ve never seen in XML.

The biggest advantage in DTF vs any other Java based distributed testing tool is that you can write a test that interacts with N different machines and the test explicitly describes all of the interactions of the components in your system in one easy to read XML. Most other framework rely on multiple configuration files and even different files to describe the behavior of each peace. Aside from that there is also the feature to deploy your setup to your host machines from a single machine using an XML format to describe your setup.

With all of this the best thing to do is to start by reading the documentation written for those interested in DTF and to start really digging in to what DTF can do for you. DTF’s documentation can be found here: https://github.com/rlgomes/dtf/wiki