Craig Balfour's Blog

ScaleConf 2013

2013-05-14T11:07:00.002+02:00

I attended ScaleConf a few weeks back.

It was a really great conference - perhaps the best I have ever attended.

Kirstenbosch Gardens is a fantastic venue. I took every opportunity between talks and after lunch to walk around the gardens and just absorb the peacefulness of the place.
The speakers in general have interesting talks. And it is always interesting to hear from international speakers about the scaling issues they face.
And the food - both snacks and lunch at Moyo - was fantastic.

Here are a few of themes I took away with me:

Scale horizontally, not vertically.
Do this by decoupling your service into smaller independent systems and use message queuing (RabbitMQ and friends) to get data to appropriate components.
Use NOSQL/memcache/key-value stores where possible since relational DBs don't scale well.
And measure everything so you know when stuff is broken.

I'd highly recommend attending next year if you get the opportunity.

Using MySQL as handle.net database back-end

2013-05-14T10:43:00.002+02:00

I've recently been working on getting a CNRI handle (aka persistent identifier) server up and running. We wanted to use MySQL as our database back-end. Since the documentation didn't cover how to do this in sufficient detail so here is what I did.

PS: We are using a pretty old version of the handle.net software - v 6.2. Hopefully this configuration still applies to the newer versions.

Background

By default the handle.net software stores its data in a Java jdb database. In my opinion this is less than ideal since there are no tools to manipulate or even view your data. You can, I guess, write your own java code to do this, but this is a pain.

Everybody knows MySQL so we wanted to use it as our back-end. The handle server does support SQL but the document only provides a config example for postgres. (Strangely, it does provide a table layout for MySQL) Google also didn't provide any useful answers.

So, in case it is useful to anybody else, here is my configuration.

Setup MySQL

Install the MySQL server package. Use apt-get or yum depending on your linux distribution.

Create a database. I decided to call mine handle.

 mysqladmin -u root -p create handle

Create a MySQL user and give it access to the database.

 grant all on handle.* to handle@'localhost' identified by 'supersecretpassword';

Setup the table structure as per the MySQL example in the documentation.

 create table nas (  
   na varchar(255) not null,  
   PRIMARY KEY(na)  
 ) ;  
 create table handles (  
   handle varchar(255) not null,  
   idx int4 not null,  
   type blob,  
   data blob,  
   ttl_type int2,  
   ttl int4,  
   timestamp int4,  
   refs blob,  
   admin_read bool,  
   admin_write bool,  
   pub_read bool,  
   pub_write bool,  
   PRIMARY KEY(handle, idx)  
 );

Download MySQL's JDBC jconnector and put it somewhere useful. I put it in /opt/mysql-connector-java-5.1.25. Uncompress the archive and you should end up with a sub-directory all sorts of stuff but most importantly a jar file - in my case, mysql-connector-java-5.1.25-bin.jar.

Configure Handle Software

I assume that you already have it up and running using the Java jdb database. If not follow the documentation and get this working first.

I'll just describe what I did to change from the local jdb database to MySQL.

Edit your startup script to add the jconnector JAR to your classpath.

 #!/bin/bash  
 java_path=/opt/jdk1.7.0_21/bin  
 $java_path/java -cp "/hs/bin/handle.jar:/opt/mysql-connector-java-5.1.25/mysql-connector-java-5.1.25-bin.jar" net.handle.server.Main /hs/svr_1

Notice the colon (:) between the two paths which make up your classpath (cp). If you are running this on Windows you need to use a semi-colon (;) delimiter.

Update your config.dct. Add the following config snippet to the existing "server_config" section.

 "storage_type" = "sql"  
 sql_settings = {  
  sql_url = "jdbc:mysql://localhost:3306/handle"  
  sql_driver = "com.mysql.jdbc.Driver"  
  sql_login = "handle"  
  sql_passwd = "supersecretpassword"  
  sql_read_only = "no"}  
  }

The things to note - which differ from the Postgres example in the docs - is the sql_url and sql_driver.

Fire up your handle server and make sure it starts properly. If you get a MySQL error the server either cannot find the jar file (check your classpath) or something in your config.dct is wrong.

Fire up the handle admin tool and use it to re-home your server by clicking Server Admin | Home Naming Authority. This will put your naming authority into the nas table.

And you're done!

Of course, your handle table is now completely empty and you will need to get your digital repository to re-populate it. (Unless of course you are up to writing your own java code to read the existing data our of the jdb and putting it into the MySQL handle table).

In our case - where we use Digitool - it was just a matter of running a Publish Persistent Identifier job and all of our handles got republished into the MySQL tables.

Normalizing a MAC address string

2008-10-10T11:51:00.003+02:00

Over the last few days, I have been spending some time working on my python - reading the sections of Diving into Python that I have never got around to and refactoring parts of some of my python scripts to make better use of the features of language and, ultimately, to make them more robust (i.e. usable by people other than me).

The script I have started with is a simple one for registering hosts for DHCP access. Basically, it takes two command line arguments - a fully qualified hostname and a MAC address - and then does some validation, checks that neither address is already in use, normalizes the output to the correct format, constructs a properly formatted host stanza and appends it to the end of our ISC DHCP servers dhcpd.conf configuration file.

I have made improvements to various parts of the code but the changes I am most conflicted about are those I have made to the MAC address normalization function which works reliably and therefore probably isn't a good candidate for refactoring but (to me anyway) looks inelegant - something which I think matters in an elegant language like python.

The normalization function takes as inpute a MAC address in one of three (string) formats - unix (00:11:22:33:aa:55), Windows(00-11-22-33-AA-55) and Cisco (0011.2233.aa55) - and the same MAC address in the unix format.

Since I am trying to move towards a more test-driven development approach, I started out by writing a very basic unit test to make sure my new function is at least as reliable as the old function. Here is the code on the unit test (test.py):


#!/usr/bin/env python
import unittest
import sys
sys.path.append(".")
import oldmac as mac
#import newmac as mac

class Test(unittest.TestCase):
 """ Unit test/s for MAC address normalization function """
 normalize_values = (
  ('00:11:22:33:aa:55', '00:11:22:33:aa:55'),
  ('0011.2233.aa55', '00:11:22:33:aa:55'),
  ('00-11-22-33-aa-55', '00:11:22:33:aa:55'),
  ('00:11:22:33:AA:55', '00:11:22:33:aa:55')
 )

 def testMacNormalize(self):
  """ Normalize MAC addresses to lowercase unix format """
  for addr, expected in self.normalize_values:
   result = mac.normalize(addr)
   self.assertEqual(expected, result)

if __name__ == "__main__":
 unittest.main()

Here is my orginal normalization function (oldmac.py):


#!/usr/bin/env python
""" Old MAC normalization function """
import re

def normalize(m):
 """ Normalize a MAC address to lower case unix style """
 m = re.sub("[.:-]", "", m)
 m = m.lower()
 n =  "%s:%s:%s:%s:%s:%s" % (m[0:2], m[2:4], m[4:6], m[6:8], m[8:10], m[10:])
      return n

There are two things I don't like about this code:

Use of a regular expression for something as simple as eliminating the delimiters from the string. There must be a simpler way to do this.
And the bit I find inelegant - the construction of the normalised string, n. It looks ugly :)

First off, here is the results of running my unit tests against this old function:


$ ./test.py -v
Normalize MAC addresses to lowercase unix format ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK

So, I set out to rewrite the normalize function more elegantly. Here is the result (newmac.py):


#!/usr/bin/env python
""" New MAC normalization function """

def normalize(addr):

 # Determine which delimiter style out input is using
 if "." in addr:
  delimiter = "."
 elif ":" in addr:
  delimiter = ":"
 elif "-" in addr:
  delimiter = "-"

 # Eliminate the delimiter
 m = addr.replace(delimiter, "")

 m = m.lower()

 # Normalize
 n= ":".join(["%s%s" % (m[i], m[i+1]) for i in range(0,12,2)])

 return n

The differences between this version and old version are:

I replaced the regular expression with a simple string.replace. Why use a (something-big) when a (something-small) will do.
I replaced the normalization expression with does the same thing but using a list comprehension.

And my unit test runs against this version:


$./test.py -v
Normalize MAC addresses to lowercase unix format ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK

So, what do you think? An improvement or not?

Have I over-engineered the normalization code by replacing something static, simple and fast with something more complex, less readable and probably slower?

Is there a better way of normalizing a MAC address string?

Note: I have just presented one aspect of the refactoring on this script - the MAC address normalization. This is the part I am most conflicted about because the old code worked fine but the new code looks "cooler" :) . The other improvements such as the more robust validation I have left out (perhaps for another blog post).

A few days of relative fame

2008-08-28T10:31:00.004+02:00

Here's what happens when somebody posts a few things you've written about / in python to reddit.com:

A more efficient method of sorting a list of IP addresses in Python

2008-08-25T13:26:00.005+02:00

As ben, okplus and tom pointed out in comments to my Sorting a list of IP addresses in Python blog post, my sort function is very inefficient. As implemented, each IP address gets converted to decimal a number of times which is unnecessary and wasteful. Thinking about the bigger picture - using the function in an application - it doesn't make sense to convert the IP address to decimal in the sort function.

Since my goal was to extend the work of the iplib module, the solution, I think, should involve using this module more efficiently, if possible. The iplib module represents an IPv4 address using its IPv4Address class. When initializing an instance of this class, the address gets converted to decimal and stored in the _ip_dec attribute, accessible by calling its get_dec() method. So by converting the raw IP addresses to IPv4Address instances outside the sort function (which I would probably be doing anyway) and using the get_dec() method in the function should improve things.

So, here is what I hope is a more efficient sorting function:



#!/usr/bin/python

import iplib

ips = []

for ip in ["192.168.100.56", "192.168.0.3", "192.0.0.192", "8.0.0.255"]:
       ips.append(iplib.IPv4Address(ip))

def ip_compare(x, y):
       return cmp(x.get_dec(), y.get_dec())

ips.sort(ip_compare)

print [ip.address for ip in ips]

Of course, comments are welcome.

Thanks to ben, okplus and tom for their helpful comments to my previous attempt.

Playing with Google AppEngine

2008-08-22T17:19:00.008+02:00

I've just spent the last two days playing around with Google's AppEngine, and I like it.

When my language of choice was perl, I wrote plain ole CGI web applications (if you can call them that) using perl's CGI module.

In the python world though, web frameworks and WSGI seem to be the way to do things, so I've known for a while that I needed to learn one of the many frameworks out there. I spent about a week getting my head around TurboGears and the concepts of ORM a year or so ago using the TurboGears book. After a week I has finished the book and has the sample app working but had no more time left to spend on the project. Obviously, not using it, I've long since forgotten all of the TurboGears specific stuff. I had through to given Django my next try but AppEngine caught my eye in the meantime.

AppEngine has a nice Getting Started Guide which I used to get a simple little application up and running on the local development web server included in the SDK in an hour or two.

Learning from the mistakes of the past, I decided on a project - porting a perl CGI script I wrote a few years ago (a University of Cape Town map using the Google Maps API and some local data I had gathered) to AppEngine. The result is basic but its a start and its here. Hopefully, over time I'll develop it into something useful - maybe add some additional information such as a picture or two, a brief description, a list of departments, etc to each building.

The only problem I has was with bulkload_client.py, a tool for reading bulk data from a CSV file and importing it into the application. We are behind a proxy server, and bulkload_client.py doesn't support proxying (even though application publishing tool does). Since everything is written in python adding (or should I rather say hacking in) proxy support was not a big problem. Just edit google/appengine/tools/bulkload_client.py in the SDK and change line 136 from:

connection = httplib.HTTPConnection(host_port)

connection = httplib.HTTPConnection("127.0.0.1", 3128)

where "127.0.0.1" is the IP address of your proxy server and 3128 is the proxy port, and change line 138 from:

connection.request('POST', uri, body, headers)

to:

connection.request('POST', 'http://%s%s' % (host_port, uri), body, headers)

and it should use your proxy server to upload your data to your application. When I get a chance I will try add proxy support using getopt and produce a proper patch - maybe :)

Of course, this is only needed when you have deployed your app to the Google AppEngine infrastructure and you want to deploy your data, not when you're using the development server included in the SDK.

And that's all there is too it. AppEngine is still in beta and is limited to 10 applications per Google Account (and you can't currently delete your applications once you've deployed them - though this feature is apparently coming) but when it goes live it's going to be big. Hopefully they still allow small applications to use the infrastructure for free.

Next stop, Django.

Sorting a list of IP addresses in Python

2008-08-01T13:11:00.006+02:00

As I work a lot with network data, one of my favourite python modules is iplib. It takes care of quite a few of things I want to do with IP addresses but lacks a lot of functionality of perl's Net::Netmask which I relied on extensively when perl was my favourite language.

One of the iplib missing features is a method for sorting a list of IP addresses, or at the very least, a method for comparing two addresses.

Luckily this is easy enough to implement yourself in python using a customised sort function. See the Sorting Mini-HOW TO for a well written document on sorting in python.

Here is my attempt at a custom function for sorting IP addresses.


import iplib

ips = ["192.168.100.56", "192.168.0.3", "192.0.0.192", "8.0.0.255"]

def ip_compare(x, y):
    """ Compare two IP addresses. """
    # Convert IP addresses to decimal for easy comparison
    dec_x = int(iplib.convert(x, "dec"))
    dec_y = int(iplib.convert(y, "dec"))
    if dec_x > dec_y:
            return 1
    elif dec_x == dec_y:
            return 0
    else:
            return -1


ips.sort(ip_compare) # Sort in place

print ips
['8.0.0.255', '192.0.0.192', '192.168.0.3', '192.168.100.56']

Sorted.

Update: A few people left comments which helped me realise how inefficient the above approach is. Take a look at the improved version.

LEG download stats

2008-07-31T17:09:00.006+02:00

Over the last few days I've been playing around with analyzing and graphing the proftpd and Apache logs from University of Cape Town Linux Enthusiasts Group mirror server for a report I am writing.

The data is incomplete but what we have looks quite impressive - and this is only data to UCT IP addresses, not any off-campus data.

We have a lot more log information for ftp - incomplete but going back as far as 2000.

In the past, ftp was the way things were done. We installed linux using a network install over ftp, and we used ftp to fetch updates.

Times are changing. These days everything is http. Instead of doing a network install (does Ubuntu even support a network install?) we download the ISO (via http!), burn it and install that way. And, of course, we do updates over http as well. It is a pity we don't have more data on http and it would have been nice to see where the change happened.

I wrote the scripts to analyse the logs using python. I got to use pyparsing to parse up the Apache log file and the graphs were produced using pycha.

More pyparsing and DHCP hosts

2008-06-06T12:08:00.003+02:00

Since I wrote my original pyparsing post a few days ago, I've done some more work on refining my ISC dhcpd.conf host parsing example program. I also received some useful comments and suggests from Paul McGuire, the author the pyparsing module (thanks, Paul!), which I have also tried to incorporate.

It's it's currently just a useless toy program but it is starting to look quite pretty.


#!/usr/bin/python

from pyparsing import *

# An few host entries from dhcpd.conf
sample_data = """

# A host with dynamic DNS attributes
host a.foo.bar {
   ddns-hostname a;
   ddns-domainname "foo.bar";
   hardware ethernet 00:11:22:33:44:55;
   fixed-address 192.168.100.10, 192.168.200.50;
}

# A simple multi-line host
host b.foo.bar {
   hardware ethernet 00:0f:12:34:56:78;
   fixed-address 192.168.100.20;
}

# A simple single-line host
host c.foo.bar { hardware ethernet 00:0e:12:34:50:70; fixed-address 192.168.100.40; }
"""

digits = "0123456789"
colon,semi,period,comma,lbrace,rbrace,quote = map(Literal,':;.,{}"')
number = Word(digits)
hexint = Word(hexnums,exact=2)
dnschars = Word(alphanums + '-') # characters permissible in DNS names

mac = Combine(hexint + (":" + hexint) * 5)("mac_address")
ip = Combine(number + period + number + period + number + period + number)
ips = delimitedList(ip)("ip_addresses")
hardware_ethernet  = Literal('hardware') + Literal('ethernet') + mac + semi
hostname = dnschars
domainname = dnschars + OneOrMore("." + dnschars)
fqdn = Combine(hostname + period + domainname)("fqdn")

fixed_address  = Literal('fixed-address') + ips + semi
ddns_hostname = Literal('ddns-hostname') + hostname + semi
ddns_domainname = Literal('ddns-domainname') + quote + domainname + quote + semi

# Put the grammar together to define a host declaration
host = Literal('host') + fqdn + lbrace + Optional(ddns_hostname) + Optional(ddns_domainname) + Optional(hardware_ethernet) + Optional(fixed_address) + rbrace

results = host.scanString(sample_data)

for result in results:
 print result[0].fqdn, result[0].mac_address , result[0].ip_addresses

file.close()

The output is still the same as that of the original program.

If you want it to read your sample data from a file, just change the following line:


results = host.scanString(sample_data)

... to:


filename = "/etc/dhcp3/dhcpd.conf"
file = open(filename, "rb")
results = host.scanString("".join(file.readlines()))

Recursive Descent Parsers and pyparsing

2008-06-03T09:59:00.010+02:00

Yesterday while browsing the table of contents of the May 2008 issue of Python Magazine I came across a reference to the pyparsing module - a python module for writing recursive descent parsers using familiar python grammar.

O'Reilly's Python DevCenter has an excellent introduction to using this module entitled Building Recursive Descent Parsers with Python. Well worth a read.

It just so happens that I have a number of projects which are stalled because writing code to parse complexly structured data is not my strong point. I enjoy parsing up text line by line as much as the next guy but this recursive stuff I find tedious.

The ISC DHCP configuration file is, in my opinion, a good example of parsing complexity. It's configuration directives can contain many optional directives, can be nested, and can be all on a single line or broken up move multiple lines. Writing the parser using pyparsing makes this much simpler.

Here is a simple example of using pyparsing to parse a few host definitions which while simple is quite flexible.


#!/usr/bin/python
from pyparsing import *

# Define the grammar we will use
digits = "0123456789"
colon = Literal(':')
semi = Literal(';')
period = Literal('.')
comma = Literal(',')
lparen = Literal('{')
rparen = Literal('}')
number = Word(digits)
hexint = Word(hexnums,exact=2)
text = Word(alphas)

# Define host configuration specific grammar
host_keyword = Literal('host')
hardware_keyword = Literal('hardware')
ethernet_keyword = Literal('ethernet')
address_keyword = Literal('fixed-address')
mac = Combine(hexint + colon + hexint + colon + hexint + colon + hexint + colon + hexint + colon + hexint).setResultsName("mac_address")
ip = Combine(number + period + number + period + number + period + number)
ips = delimitedList(ip).setResultsName("ip_addresses")
hostname = Combine(text + period + text + period + text).setResultsName("hostname")
ethernet_statement = hardware_keyword + ethernet_keyword + mac + semi

ipaddress_statement = address_keyword + ips + semi
x = host_keyword + hostname + lparen + Optional(ethernet_statement) + Optional(ipaddress_statement) + rparen

# Here is some sample data for us to parse

host_declaration = """
host a.foo.bar {
    hardware ethernet 00:11:22:33:44:55;
    fixed-address 192.168.100.10, 192.168.200.50;
}

host b.foo.bar {
    hardware ethernet 00:0f:12:34:56:78;
    fixed-address 192.168.100.20;
}

host c.foo.bar { hardware ethernet 00:0e:12:34:50:70; fixed-address 192.168.100.40; }
"""

# Do the parsing

results = x.scanString(host_declaration)

# Print out the stuff we're interested

for result in results:
    print result[0].hostname, result[0].mac_address , result[0].ip_addresses

The output looks like this:


a.foo.bar 00:11:22:33:44:55 ['192.168.100.10', '192.168.200.50']
b.foo.bar 00:0f:12:34:56:78 ['192.168.100.20']
c.foo.bar 00:0e:12:34:50:70 ['192.168.100.40']

Ubuntu Hardy and OpenLDAP 2.4

2008-05-23T14:17:00.005+02:00

By the looks of it Ubuntu took a giant leap of faith with OpenLDAP in Hardy Heron! They upgraded from using a reasonably stable OpenLDAP 2.3.35 in Gutsy to 2.4.7 with lots of replication problems in Hardy.

Why did they do it?! According to OpenLDAP.org the 2.3 branch is the stable release branch and 2.4 is the general release branch. OpenLDAP 2.4 seems currently be plagued by replication stability issues. Surely it makes sense to run the stable branch?!

Last night I made the mistake of upgrading our syncrepl master LDAP server from Gutsy to Hardy and now it segfaults regularly. The slave has been running Hardy for some time now without any problems, so I mistakenly thought it was safe.

I first thought the problems were Berkeley DB back-end data corruption issues - of which we have had many in the past - but repairing the databases, and even completely rebuilding them from LDIF backup has made no significant different to the (in)stability.

The question now is: Do I upgrade to 2.4.9 (from source since there doesn't seem to be any debs around yet) which by the looks of the release announcement has plenty of replication fixes, or should I try and go back to 2.3.35 from Gutsy? Decisions, decisions!

Update: I've decided to go back to OpenLDAP 2.3.35 from Gutsy. By the looks of this blog post, 2.4.9's replication is looking much better and perhaps ready for the big-time. When I get a chance, I'll give it a go in the lab this time :).

exiqsumm: Getting to know your (Exim) mail queues

2008-05-23T12:42:00.004+02:00

Running exiqsumm a few times a day via cron is an excellent way to get a sense of what is normal for your mail queues and will help you pick up all sorts of problems.

For example:

Lots of mail waiting to be delivered to your local domain may indicate a local delivery problem like a broken local delivery agent (perhaps you just did a quick distribution update / upgrade and something important broke?), your mail volume has filled up, or some important, high volume user has filled up his mailbox and now his mail is just sitting on the queue?
Lots of mail waiting to be delivered to domain example.com? Perhaps example.com's mail server is down, or a network problem has made them unreachable?
Lots of mail waiting to go out to lots of different, perhaps suspiciously named, domains could indicate that your mail server is being used to relay spam.

Here's how you use it:

/usr/local/sbin/exim -bp | /usr/local/sbin/exiqsumm

Exiqsumm will give you a nice tabular summary of what is sitting in your mail queue on a per domain basis.

Count Volume Oldest Newest Domain
----- ------ ------ ------ ------

1 3276 20h 20h 0575.info
1 2048 47h 47h 1st4diaries.com
1 3891 33h 33h able-courier-inc.com
1 3481 41h 41h adelphia.com
1 3481 47h 47h alliance-lawyers.com
1 3993 18h 18h alvar.com
1 2662 37h 37h astound.net
1 2252 26h 26h bcinternet.net
1 4812 16h 16h belmarparkdental.com
5 8192 4d 3h blitzbeat.com
1 2150 47h 47h boardley.com
1 1843 47h 47h bochao.com.cn
1 2355 32h 32h brace-ingenierie.com
1 3379 13h 13h brkoday.com
1 3379 3h 3h broste.com
5 9932 5d 16h centrapoint.com
...
---------------------------------------------------------------
109 637KB 5d 2h TOTAL

Some nice advanced topic OpenLDAP articles

2008-05-22T17:20:00.003+02:00

I'm always on the lookout for information on advanced OpenLDAP topics, something I've found to be in short supply on the internet.

By advanced I mean documentation, tutorials and howto's that go beyond just setting up a simple standalone directory or an address book and gets into doing things with all of the fancy backends and overlays available in OpenLDAP these days.

I recently came across the Symas forums which have a few nice articles on advanced OpenLDAP topics:

Password policies (Managing Password Policies in the Directory) which describes how to add password policies such as expiry, aging, minimum length, history, etc which are features considered standard in Novell E-directory and Microsoft Active Directory accounts but are lacking in the basic OpenLDAP authentication directory configuration. Unfortunately, it falls short of giving any information on whether clients such as nss_ldap and pam_ldap support this, so I will need to test this in a lab sometime.
The Transluscent Overlay sounds really useful. One use I have for this feature is to tack the attributes I need for unix authentication (e.g. uid, gid, etc) to our organisations main directory (which I have no control over) so that I my users can use the same password to log into my servers as they use to log into the organisations systems.
There is also some stuff about replication, as well as the referential integrity and attribute uniqueness overlays.

A new blog is born

2008-05-22T15:01:00.002+02:00

I've decided to start a new blog.

I know, my family blog has been languishing badly but I've needed a place to post the details of my more technical exploits for a while now - even if only so I can refer back to the details again later - and that blog just hasn't seemed like the right place.

I've needed a new place and this is it. Let's see how this new blog works out.