After writing about Munin last week and mentioning some of its uses outside of system performance tracking I decided to write a collection of plugins for tracking website popularity. The full code is available on GitHub.

The plugins include:

• Alexa traffic rank
• Number of pages in the Google index
• Technorati authority
• Feedburner RSS subscribers
• Twitter followers

Munin will automatically handle the generation of daily, weekly, monthly and yearly graphs. All of the plugins support graphing of multiple sites/accounts, so you can show several sites on the same graph, as is the case with the Google index graph above, or you can decide to show a single site/account per graph. Some example graphs are shown below:

On a Debian-based system installing Munin is as simple as running the following command, and then going to http://your-server/munin/ in your browser:

sudo aptitude install munin

Munin comes with lots of monitoring plugins by default, including those for MySQL, PostgresSQL, Apache, Tomcat, Squid, and for things such a CPU and memory usage, load average, network traffic, and many more. You can also find lots of user submitted plugins on sites like Munin Exchange.

Munin doesn’t have to be used solely for monitoring server performance though. Being so easy to extend Munin is also a great tool for tracking non-server performance related trends over time. In just a few lines of code you could write plugins to track the following stats about your website:

• Number of User signups
• Google PageRank
• Pages in Google’s index
• Number of backlinks
• Number of twitter mentions
• Alexa traffic rank

The number of pages in Google’s index is actually a plugin I’ve written. Simple put the following code in your /etc/munin/plugins directory to see it in action:

#!/bin/sh
# Munin Plugin to display the number of pages in the
# google index for all of the given websites
# Ben Dowling - www.coderholic.com

# Change this to whatever sites you're interested in
websites="www.yahoo.com www.google.com www.twitter.com"

if [ "$1" = "autoconf" ]; then
        echo yes
        exit 0
fi

if [ "$1" = "config" ]; then

        echo 'graph_title Number of Pages in Google Index'
        echo 'graph_args --base 1000 -l 0 '
        echo 'graph_vlabel number of pages'
        echo 'graph_category google'
        echo 'graph_info This graph shows the number of pages in the Google index for a given website.'

        i=0
        for site in $websites
        do
                name="site_${i}"
                echo "${name}.label ${site}"
                echo "${name}.draw LINE2"
                echo "${name}.info The number of pages in the google index."
                i=$((i+1))
        done
        exit 0
fi

i=0
for site in $websites
do
        name="site_${i}"
		value=$(wget -q --user-agent=Firefox -O - "http://www.google.com/search?q=site:${site}" | grep -E "of about [0-9,]+" -o | grep -E "[0-9,]+" -o | sed "s/,//g")
        echo "${name}.value ${value}"

        i=$((i+1))
done

For more details about Munin see their homepage, which also includes detailed documentation on writing your own plugins.

Let me know if you can think of any more Munin plugins that could be interesting, or if you’ve used any yourself!

Lots of comments were left pointing out some sites which weren’t included in my list, so I’m following up here with a list of 10 more top programming puzzle websites:

1. Code Chef

Code Chef has lots of practice puzzles, and monthly competitions with cash prizes. The site officially supports over 35 programming languages!

2. SPOJ

The Sphere Online Judge contains 1871 different programming problems. More points are awarded for better performing solutions, which can be submitted in a range of languages.

3. Code Golf

The aim with code golf is to submit a solution using the fewest characters possible.Solutions can be submitted in Perl, Python PHP or Ruby.

4. Uva Online Judge

Over 2600 great programming puzzles, and also regular contests. Submissions in C, C++, Java or Pascal are automatically checked for you.

5. Timus Online Judge

An online competition site that automatically checks your submissions. Supports Java, C#, Pascal, C and C++.

6. Google Code Jam

The code jam is a programming contest from Google. The top 25 contestants get to travel to Google’s HQ in California. Entries are accepting in any programming language.

7. USA Computing Olympiad

Programming puzzles designed to provide “pre-college students with opportunities to sharpen their computer programming skills”. The puzzles are still interesting and fun even if you’ve got a CS degree!

8. Informatics Olympiad

A British version of the computing olympiad. Again aimed at school and college students, but fun and interesting for everyone.

9. Programming Challenges in C, C++ and C#

About.com’s C/C++/C# section regularly posts interesting programming puzzles. Successful solutions get acknowledged on the site once the deadline has passed.

10. Java Bat

A site dedicated to practical Java programming problems. You can type your code directly into the website, and it’ll tell you if you’ve solve the problem correctly or not.

bmd /~: ./dbSize.sh coderholic root ********
wp_comments Data: 6.60MB Indexes: .15MB Total: 6.75MB
wp_links Data: 0MB Indexes: 0MB Total: 0MB
wp_options Data: 1.57MB Indexes: .01MB Total: 1.58MB
wp_postmeta Data: .01MB Indexes: .01MB Total: .02MB
wp_posts Data: .57MB Indexes: .02MB Total: .60MB
wp_term_relationships Data: 0MB Indexes: .01MB Total: .02MB
wp_term_taxonomy Data: 0MB Indexes: 0MB Total: 0MB
wp_terms Data: 0MB Indexes: 0MB Total: 0MB
wp_tla_data Data: 0MB Indexes: 0MB Total: 0MB
wp_tla_rss_map Data: 0MB Indexes: 0MB Total: 0MB
wp_usermeta Data: 0MB Indexes: 0MB Total: 0MB
wp_users Data: 0MB Indexes: 0MB Total: 0MB
*** 12 Tables | Data: 8.78MB Indexes: .25MB Total: 9.03MB ***

With a few small modifications I’m sure it’d be possible to get the script working PostgreSQL or other RDBMSes. The full code is below:

#!/bin/bash
# Calculate the storage space used up by all tables in a given MySQL database
# Ben Dowling - www.coderholic.com
database=$1
username=$2
password=$3

if [ ${#database} -eq 0 ]
then
	echo "Usage: $0 <database> [username [password]]"
	exit
fi

if [ "$password" ]
then
   password="-p$password"
fi

mysql="mysql -u $username $password $database"

$mysql -se "USE $database";

tables=$($mysql -se "SHOW TABLES")

totalData=0
totalIndex=0
totalTables=0

for table in $tables
do
   output=$($mysql -se "SHOW TABLE STATUS LIKE \"$table\"\G")
   data=$(echo "$output" | grep Data_length | awk -F': ' '{print $2}')
   dataMegs=$(echo "scale=2;$data/1048576" | bc)
   index=$(echo "$output" | grep Index_length | awk -F': ' '{print $2}')
   indexMegs=$(echo "scale=2;$index/1048576" | bc)
   total=$(($index+$data))
   totalMegs=$(echo "scale=2;$total/1048576" | bc)

   echo "$table Data: ${dataMegs}MB Indexes: ${indexMegs}MB Total: ${totalMegs}MB"

   totalData=$(($totalData+$data))
   totalIndex=$(($totalIndex+$index))
   totalTables=$(($totalTables+1))
done

dataMegs=$(echo "scale=2;$totalData/1048576" | bc)
indexMegs=$(echo "scale=2;$totalIndex/1048576" | bc)
total=$(($totalIndex+$totalData))
totalMegs=$(echo "scale=2;$total/1048576" | bc)

echo "*** $totalTables Tables | Data: ${dataMegs}MB Indexes: ${indexMegs}MB Total: ${totalMegs}MB ***"

id, name, date
0, name, 2009-01-01
1, another name, 2009-02-01

You’d end up with something like:

["id", "name", "date"],
["0", "name", "2009-01-01"],
["1", "another name", "2009-02-01"]

In some situations it is nice to have a dictionary of keys and values though, so that instead of a simple list of columns we end up with:

{"id": "0", "name": "name", "date": "2009-01-01"},
{"id": "1", "name": "another name", "date": "2009-02-01"}

This would allow us to refer to fields by name rather than position in the list. Do you really want to remember that date is in position 2? And what happens if the input data changes, and a new column is added between name and date? If we’re referring to columns by position then we’ll have to change our existing code, but by referring to it by name we won’t have to change anything.

It turns out this is pretty easy to achieve, in only a few lines of python:

import csv
data = csv.reader(open('data.csv'))
# Read the column names from the first line of the file
fields = data.next()
for row in data:
        # Zip together the field names and values
	items = zip(fields, row)
	item = {}
        # Add the value to our dictionary
	for (name, value) in items:
		item[name] = value.strip()

The csv module allows you to specify a delimiter, so if your data separated you just need to make a single change:

data = csv.reader(open('data.tsv'), delimiter='\t')

Update

Thanks to several people for mentioning csv.DictReader, which does exactly what I’ve mentioned here. Having a look at the code it does something very similar, but also takes into account rows of different length, ignores empty columns, and uses the method Tim mentioned in the comments for creating the dictionary:

    # From csv.py
    def next(self):
        if self.line_num == 0:
            # Used only for its side effect.
            self.fieldnames
        row = self.reader.next()
        self.line_num = self.reader.line_num

        # unlike the basic reader, we prefer not to return blanks,
        # because we will typically wind up with a dict full of None
        # values
        while row == []:
            row = self.reader.next()
        d = dict(zip(self.fieldnames, row))
        lf = len(self.fieldnames)
        lr = len(row)
        if lf < lr:
            d[self.restkey] = row[lf:]
        elif lf > lr:
            for key in self.fieldnames[lr:]:
                d[key] = self.restval
        return d

Writing Hadoop applications

Hadoop takes care of distributing your application over several nodes, but the rest is down to you. You can write you application in Java, or as a “streaming” application in any language. Streaming apps simply reads data from stdin and write their output to stdout. Writing a Java application seems to give you much more control over various options, and presumably improved performance too, but I opted for streaming for simplicity.The following post gives a good example of a streaming program for Hadoop: Writing an Hadoop MapReduce program in Python.

One of a best things I’ve found with streaming jobs is you can test them on a small amount of data on the command line, with no need for Hadoop:


cat input-data | ./mapper | sort | ./reducer


Running Hadoop

Rather than installing a local copy of Hadoop I used virtual machine from Yahoo! which comes with Hadoop pre-installed. The virtual machine is available from their Hadoop tutorial page, which includes full details of how to run the VM and start runninng your hadoop application on it. The VM even includes an Eclipse plugin for Hadoop, which is mainly focused on writing Java application but it still useful if you’re writing streming apps becuase it provides you with job progress details and easy access to the Hadoop file system on the VM.

Multiple Nodes

The Yahoo VM is a great way to test your Hadoop apps locally, but doing distributed computing on a single node kind of misses the point. If you want to do any serious processing with hadoop you’ll need lots of machines. Amazon’s Elastic Complete Cloud (EC2) is perfect for this.

Amazon actually provide an Elastic Map Reduce service which sits on top of EC2 and manages a cluster of EC2 instances for you automatically while running your Hadoop application. The article Finding Similar Items with Amazon Elastic MapReduce, Python, and Hadoop Streaming contains details of how to use the service with a real-world example.

An alternative option to running Hadoop on EC2 is to use the Cloudera distribution. Their guide gives full details on how to set this up, and also links to client side scripts which make it extremely simple to to bring up a cluster and start running your Hadoop job. Using the Cloudera distribution has the benefit of being cheaper than using the Elastic Cloud Compute service, because you only pay for your EC2 usage. Another advantage is that you have more control over your EC2 instances, so you can install any packages that you wish, so for example you could book up your instances with subversion or git installed and automatically run a script to checkout the latest version of your code and data and run your Hadoop program!

Alternatives to Java and streaming apps

There are several projects that are built on top of Hadoop that allow you to access or process your data in different ways. Hive allows you to write SQL like queries on your data, which are automatically converted to MapReduce tasks. Dumbo allows you to write Hadoop applications in Python at a lower level than you can with streaming apps.

For more information check out the main Hadoop page, which is packed full of documentation.

One of the best sample applications I’ve found is WhereAreMyFriends, which uses the phone’s location awareness and Google Maps integration to display your location and that of everyone in your phone book.

Getting WhereAreMyFriends up and running the with 1.5 simulator is a little tricky, but this guide runs through the required steps.

Now that I’ve finally started writing some code for the Android I’ll be writing some posts on the subject. If anybody has any suggestions on any specifics they’d like me to write about let me know!

watch ls -l

The watch program clears the screen and displays the output of “ls -l” every 2 seconds.

Sometimes I’ll want to monitor a command that only outputs a single line. If I wanted to see the total number of files in a directory rather than the files themselves I could use the command “ls -l | wc -l”. The fact that watch clears the whole screen can be a little annoying here though, because the command is only outputting a single line. That is why I came up with the following small bash script, linewatch.

Linewatch repeatedly calls any arguments passed to it every 2 seconds (in the same way watch does), but only clears a single line rather than the whole screen. Here is the code:

#!/bin/bash
clearline="\b\033[2K\r"
command=$@

while true
do
    eval "$command"
    sleep 2
    echo -n -e "$clearline"
done

And here is an example of how to call it:

$ ./linewatch "ls -l | wc -l"
24

The number of files in the current directory (24 in the example) will keep update every 2 seconds. Just hit Ctrl-C when you want to quit,

It wouldn’t take much to get it working with other version control systems such as Git or Bazaar, or to do some nice formatting of the output instead of outputting the raw svn log as-is. Let me know if you find it useful!

#!/bin/bash
# Shell script to email the latest changes in an SVN
# repsitory to a specified email address.
# Ben Dowling - wwww.coderholic.com

svnUrl="http://anonsvn.wireshark.org/wireshark/trunk/"
lastRevisionFile="./.last-revision"
mailto="ben@coderholic.com"

function getCurrentRevision {
  # Get the current SVN revision, eg. "r4670"
  currentRevision=$(svn log "$svnUrl" -r HEAD 2>/dev/null | head -n2 | grep -v -- "-------" | awk '{ print $1 }')
  # Strip off the 'r'
  currentRevision="${currentRevision:1}"
  echo "$currentRevision"
}

currentRevision=$(getCurrentRevision)

# If we've run this program before then we've stored the SVN revision at the time
if [ -f "$lastRevisionFile" ]
then
  lastRevision=$(cat "$lastRevisionFile")
  #  Check what the current revision is, and exit if there
  # haven't been any changes since we last checked
  if [ $currentRevision -lt $lastRevision ]
  then
      echo "No changes since last check"
      exit
  fi
else
  # We haven't run this program before, so set the last revision to the current revision - 10
  lastRevision=$(echo "$currentRevision - 10" | bc)
fi

# Mail the SVN changes
svn log "$svnUrl" -r "HEAD:${lastRevision}" | mail -s "SVN changes for $svnUrl" $mailto

# Store the current revision + 1 as the last revision
revision=$(echo "$currentRevision + 1" | bc)
echo "$revision" > "$lastRevisionFile"