Zen and the Art of Programming

Reasons to switch to DB2 9.7

2009-06-29T13:13:04Z

IBM recently put out a video with a few quotes from customers who’ve had a chance to try DB2 9.7 in production. At times the editing is a bit cheesy, but what our customers are saying is factual. DB2 9.7 is quite a breakthrough and it can save millions of dollars for large companies. If your needs are more modest, like a demanding Web application or a point of sale server, DB2 Express-C can still give you most of the same benefits at zero the cost. Wanna take it for a spin? You can download it here or sign up at RightScale.com to try it on the Cloud.

Social Bookmarks

Apple “Genius”

2009-06-26T21:47:21Z

Recently I sold my old, damaged MacBook Pro on eBay, and in doing so I claimed that there was a chance that it could be repaired (by Apple) for free. But how, you may be wondering, could I make such a bold claim? Was it all a strategy to over-sell my broken laptop? Not in the least. Back when I first found out about this Apple’s KB article, the contents of which appeared as though they would entitle me to a free repair, I headed straight to the Apple Store at the Fairview Mall in Toronto. I’d had a fairly positive customer care experience there just a week before, and I was rather optimistic that they’d repair it for me.

After I arrived at the Genius bar, I had to wait for quite a while before having the displeasure of dealing with an uncooperative “genius”, a young guy whose unfriendly attitude far outweighed any technical know-how he may have had. He immediately denied any knowledge of video issues on MacBook Pros from 2007 and only agreed to check my laptop after I’d showed him a printout of the knowledge base issue mentioned above. He essentially humored me in a rather reluctant way, and after a very short while told me that my laptop didn’t qualify for the free repair. Annoyed by this guy’s lack of care regarding my problem, I left the store.

Around the time when this situation arose, I had already visited the Genius Bar several times regarding various matters and was feeling a bit tired of dealing with a broken laptop. Ultimately I gave up on my old MacBook Pro (my first Mac ever) and when it was economically possible, I purchased a replacement laptop. Though my old MacBook Pro had cost me time and money, I felt that its tale was done and over with once I brought my new laptop home. That is until the day I decided that someone else could get more use out it than I was (as it was just sitting unused in my office), and that I could get a few bucks by selling it on Ebay. The old Mac’s auction wrapped up with a selling price of $578. It was bought by a fellow from Ontario, who was a very pleasant person to deal with.

If fact he was so kind that he sent me a follow-up regarding his own attempt to get the laptop fixed. Here’s what he wrote:

Just a quick update. The testing on the Macbook Pro was conducted this week, and found to conform to the warranty exception Apple identifies with the nVidia chip problem. A replacement is being installed now, but likely won’t be ready until next week. While you’re probably more content to wash your hands of the whole matter, I’d seriously consider a complaint against the apple store you took it to, and more specifically against the “genius” who served you, given he was clearly no genius at all, and likely cost you significantly in time and money as a result. Hope you’re having a good day.

I am genuinely happy for the guy. He took a risk by buying a broken laptop from me, trusted my story to be true (as it was!), and ended up scoring a working laptop that’s (now) worth far more than what he paid for it. His bet paid off and I wish him all the best.

The truth of the matter is that, as he says, I’m glad to “wash my hands of the whole matter”. At this point there is little Apple would do, even if I made a fuss about it (which I’m not going to, of course). Next time I walk by that particular Apple store, I may have a word with the manager (regarding the Genius I’d dealt with), but that’s about it.

The moral of the story is not that Apple’s customer care sucks or rocks. Apple Genii are just people, some are very good, others exceptionally bad, and most of them are somewhere in between. The take-home lesson for me here was that when dealing with Apple, if you get turned down by one store, you shouldn’t stop there! Take the time to visit a couple more and to persue the matter as far as you can.

Social Bookmarks

DB2 Express-C 9.7 and the Django adapter released

2009-06-19T21:00:05Z

This is a great day for those of us who love DB2, as DB2 Express-C 9.7 has just been released. As mentioned before, this is the best DB2 ever, and an extremely important release.

To learn more about what’s new in this release, please check out the recording of our latest webinar:

If you run Linux, Unix or Windows, download it while it’s hot.

DB2 9.7 on the Cloud

Another great aspect of this release is that for the first time ever, DB2 has been released both as a product and as a deployment on the Cloud. If you pop over to RightScale, you can get a trial account for free and should see DB2 Express-C 9.7 on both CentOS and Ubuntu within the partner catalog. RightScale has been an amazing partner and they really do wonders to simplify Cloud Computing. In ten minutes time you can be up and running on the Cloud, thanks to the templates provided.

DB2 support for Django

But the good times don’t stop there, we are also announcing the first official release of the Django adapter for DB2. It sounded crazy when I first proposed the idea within IBM back in 2006, but now it’s a reality.

You can download the .tar.gz archive from the Google Code homepage for the project, or simply by clicking here. This version fully supports the Django 1.0.2 API. For instructions on how to install it, please read the Getting started with the IBM DB Django adapter guide. The current version supports DB2 for Linux, Unix, Windows and MAC OS X, version 8.2 or higher (9.5 FP2 or higher for MAC OS X). In the future, IBM Cloudscape, Apache Derby, Informix (IDS) and both System i & z/OS will be supported.

ibm_db gem updated to 1.1

I’ll conclude this DB2-centric post with a smaller, but still interesting announcement. The ibm_db gem has been updated to version 1.1. This release includes support for ActiveRecord’s QueryCache mechanism, enhanced support for BigInt (and BigSerial), support for rename_column (requires DB2 9.7), parametrization of the timestamp datatype (requires DB2 9.7), and a few fixes and performance enhancements as well. It is recommended that you upgrade to this version.

Social Bookmarks

Selling my old MacBook Pro

2009-06-16T21:29:30Z

A few days ago I placed my old MacBook Pro up for auction on Ebay, with a starting price of $200 and no reserve. While the auction has generated interest, so far there has been only one bidder. The auction ends in a little over two days. If you are interested, please read the description carefully and then bid with confidence. Despite being damaged, a laptop like this is still worth a few hundreds dollars due the sheer amount of working parts in it (CPU, memory, hard-drive, LCD matte screen, new battery, MagSafe adapter, remote control which works with new models too, etc…). It could be used as an extra Mac in your arsenal if utilized through Remote Desktop or repaired. There is also a chance that it could be repaired for free by Apple (though of course I cannot guarantee this point).

Social Bookmarks

Discussing code reading and testing with Dave Thomas, Cory Foy and Tim Bray

2009-06-10T16:27:19Z

Pat Eyler just published an interview/discussion with Dave Thomas, Cory Foy, Tim Bray and I on the subject of code reading and testing legacy code. I think the resulting article is a very interesting read with a lot of insight (thanks to the other participants). It was a fun discussion which left me feeling honored to be among such distinguished people. If you’d like, you can read this interview on Pat’s blog.

Social Bookmarks

Do Androids Count Electric Sheep with DB2 or MySQL?

2009-06-05T18:04:51Z

Counting rows is an ubiquitous operation on the web, so much so that it’s often overused. Regardless of misuse, there is no denying that the performance of counting operations has an impact on most applications. In this post I’ll discuss my findings about the performance of DB2 9.5 and MySQL 5.1 regarding counting records.

For those of you who are not into science fiction, let me clarify that the odd title of this post is a tongue-in-cheek reference to the great novel, Do Androids Dream of Electric Sheep?.

I connected to the database, created the table, imported the data and benchmarked counting operations using ActiveRecord in a standalone script. Here is the code I used:

#!/usr/bin/env ruby
require "rubygems"
require "active_record"
require 'benchmark'

ActiveRecord::Base.establish_connection(
  :adapter  => :mysql,
  :username => "myuser",
  :password => "mypass",
  :database => "mydb")

ActiveRecord::Schema.define do
  create_table :people, :force => true do |t|
    t.string :name, :null => false
    t.string :fbid, :null => false
    t.string :gender
    t.string :profession
  end
end

class Person < ActiveRecord::Base
end

# This can be sped up by performing an import instead
Person.transaction do
  File.open("person.tsv").each_line do |line|
    line = line.split(/\t/)
    p = Person.new
    p.name = line[0]
    p.fbid = line[1]
    p.gender = line[6]
    p.profession = line[17]
    p.save!
  end
end

n = 100
Benchmark.bm(26) do |x|
  x.report("Count all:") { n.times { Person.count } }
  x.report("Count profession:") { n.times { Person.count(:profession) } }

  x.report("Count females:") do
    n.times { Person.count(:conditions => "gender = 'Female'") }
  end

  x.report("Count males w/ profession:") do
    n.times { Person.count(:profession, :conditions => "gender = 'Male'") }
  end
end

Please note that importing records in a huge transaction containing hundreds of thousands of INSERT operations is far from the most efficient way to import. Massive imports of data using the load/import facilities provided by each database is the way to go (also see the ar-extensions plugin). The lengthy import wasn’t benchmarked here though, so it isn’t determinant for this article.

people.tsv is a 92.7 MB tab separated values file that contains 875,857 records from the Freebase project (in my file I removed the header line, leaving only records).

For those who are not familiar with ActiveRecord, the queries executed behind the scenes are (in order):

SELECT count(*) AS count_all FROM people

SELECT count(people.profession) AS count_profession FROM people

SELECT count(*) AS count_all FROM people WHERE (gender = 'Female')

SELECT count(people.profession) AS count_profession FROM people WHERE (gender = 'Male')

While the table definition (for MySQL) is:

CREATE TABLE `people` (
	`id` int(11) DEFAULT NULL auto_increment PRIMARY KEY,
	`name` varchar(255) NOT NULL,
	`fbid` varchar(255) NOT NULL,
	`gender` varchar(255),
	`profession` varchar(255)
) ENGINE=InnoDB

As easily verified by enabling logging with:

ActiveRecord::Base.logger = Logger.new(STDOUT)

Without much further ado, here are the times I obtained on my last generation MacBook Pro 2.66 GHz with 4 GB DDR3 RAM, and 320 GB @ 7200 rpm hard disk, running Mac OS X Leopard:

MySQL:

  Count all:                  42.467522
  Count profession:           52.130935
  Count females:              54.575469
  Count males w/ profession:  64.046631

DB2:

  Count all:                  5.818485
  Count profession:           7.714391
  Count females:              8.556377
  Count males w/ profession:  9.656739

Or in graph form:

That’s an impressive difference. To be exact, in this example DB2 was between 6 and 7 times faster than MySQL. In the case of COUNT(*), DB2 counted almost a million records in 58 milliseconds, or in about the blink of an eye according to Wolfram Alpha.

For those who are skeptical, please note that DB2 was not manually fine-tuned in any way. The client codepage was set to 1252 to allow Greek letters, and the log size was increased to permit such a huge transaction during the import. That’s it, no optimizations were attempted. This is DB2 Express-C out of the box. It looks like smart androids count electric sheep with DB2 after all.

The advantages of DB2 over MySQL when dealing with a massive volume of traffic are well known (and not limited to performance either), but DB2 can dramatically improve performance even for your average web application. And DB2 9.7, which will be released this month, increases the performance and the ability to self-tune itself to the available resources and required workload even further. If you’d like to try DB2 Express-C for yourself, you can download it here. It doesn’t cost you a dime to obtain and can be used for development, testing and production absolutely free of charge.

Social Bookmarks

Of labels and limits

2009-05-27T17:26:53Z

In an attempt to satisfy our need for identity and belonging, we desperately try to wear as many labels as possible, and to a certain extent labels are a necessity. When people ask you what you do for a living, it’s far easier to reply “I’m a computer programmer” than to try and explain the plurality and complexity of the exact criteria of your job.

The problem with labels is that they can place you in a box, at times greatly limiting who and what you are. So while it’s okay to use labels to efficiently communicate with other people, it’s important not to fall into the trap of taking them too seriously, thus letting them become who you are - or are not.

It’s not the label per se, but rather our perception of what our identification with a given role implies. If I identify myself too strongly as a “rubyist” I may not be inclined to recognize the good that is found elsewhere in other programming languages, or worse still, reject such good in an attempt to defend the choice I opted to identify myself with. This inclination is the basis of many of the “religious wars” you see online.

I sometimes find myself in the odd predicament of limiting myself because of some label or assumption of what “a person like me” can and cannot do. In such instances though I’m reminded of a few stories about courageous individuals who went beyond labels, above the layer of conventionality, breaking what common sense would have considered a “difficult to challenge” limit. I’m reminded of blind people who took on photography and managed to be successful at it, or of a black kid of Kenyan origins who managed to become the President of the United States of America. But there is one story in particular that always gets me, it’s the story of Django Reinhardt, after whom the the popular Python framework was named.

Django was a Gypsy jazz guitarist who was severely injured in a fire when he was eighteen. As a result of this accident his right leg was paralyzed and the third and fourth fingers on his left hand were severely burned. Doctors recommended amputating his leg and were pretty darn sure that he would never play guitar again due to the extensive damage to his hand. Django refused the amputation though and left the hospital as soon as he could. Within a year he was able to walk again, albeit with the aid of a cane. Even more surprisingly, despite being “disabled” in his left hand, he persisted through the pain to practice his beloved instrument. He went on to reinvent the conventional approach to guitar playing by performing solos with the use of only two fingers, using his half-paralyzed fingers for chord work. Today Django is considered one of the most influential guitarists of the 20th century.

I’ve learned to consciously fight the urge to limit myself. Whatever labels you feel may be cutting your potential short or holding you back, I encourage you to break free and rise above them. Does doing so mean you’ll reinvent the way a musical instrument is played, reshape the course of history or become a hero in your field? Perhaps, but even if it doesn’t, your own life stands to become richer and freer because you decided not to live within the confines of a label.

Social Bookmarks

Announcing ThinkCode.TV’s new English newsletter

2009-05-25T13:10:50Z

Ten days ago I mentioned ThinkCode.TV, my startup on the side, the aim of which will be to produce high quality screencasts about programming, both in English and Italian. My two co-founders and I are relatively well known in Italy, so I was expecting the announcement to generate some buzz in my home country. What surprised me though was finding out that my informal pre-announcement generated quite a bit of interest in the English-speaking world as well.

In fact, over the past few days I have been contacted by several people who were curious about ThinkCode, some of whom asked me to send them an email when we release English content. Meanwhile, we found our second native English speaker (who’s a very solid programmer) who agreed to work with us to narrate and create original English videos.

It’s clear that things are moving fast and we realized that the presence of two distinct product lines, one in English and one in Italian, generally appeals to two different audiences. As such having a single (Italian) newsletter just won’t cut it. So today, we are pleased to announce our English newsletter, which I invite you to join if you want to stay in the loop. We haven’t decided yet if we are going the “private beta” route or not, but joining today will guarantee you early access if we do.

This way, English speakers can join our English newsletter (where only English content will be announced or discussed), and Italians on the other hand can continue to join our Italian newsletter (in which both Italian and English content will be announced in Italian; this to account for the fact that some Italians may be interested in purchasing products that are available in English only).

We’re really excited by positive responses we’ve already received and are really looking forward to providing you with amazing educational material. If you are interested in becoming an author, are a company who would like to strike up a partnership with us so that we can adapt and sell your videos in Italian, or simply would like to get in touch, please drop us a line at info@thinkcode.tv.

Social Bookmarks

TextMate bundle for DB2

2009-05-22T20:39:12Z

Before leaving on a recent business trip to Italy I begun working on a TextMate bundle for DB2. Here I’ll introduce it in the hope that it will interest some TextMate and DB2 users.

Installation

There are two simple prerequisites for using this bundle: 1) Install DB2 as a regular user (not root); 2) Source the db2profile by, for example, adding . ~/sqllib/db2profile to your shell profile (e.g., in ~/.profile). Both of these will ensure that your user is able to issue DB2 commands.

To install the bundle, you can run the following (assuming you have git installed):

sudo mkdir -p /Library/Application\ Support/TextMate/Bundles
cd /Library/Application\ Support/TextMate/Bundles
git clone git://github.com/acangiano/db2-textmate-bundle.git "DB2.tmbundle"

Alternatively, you can download this file, unzip it and then double click on the DB2.tmbundle that was extracted. Of course, the previous method has the advantage of being able to easily update the bundle through git pull.

If TextMate is running while you executed the previous step, you may want to also execute the following line:

osascript -e 'tell app "TextMate" to reload bundles'

This is equivalent to selecting Bundles ? Bundle Editor ? Reload Bundles from within TextMate.

Using the bundle

If you followed the instructions above, at this point you should see a DB2 menu under Bundles:

TextMate is a text editor, and as such it expects you to have an open document, in order to execute commands from this bundle. The good news is that an empty, untitled, open by default window will suffice.

As you can see from the image above, you can start the DB2 server, stop it, run an arbitrary query or DB2 command and open up the Information Center for DB2 9.5 in your browser. Below the separating line there are also two submenus, Database and Tables.

Let’s consider Database first:

Within the Database menu we can connect to a database (this step is required before being able to query a database), disconnect, create a database, drop it, and create the SAMPLE database that ships with DB2 so you can get some practice.

Likewise, while it’s at a very early stage of development, we have the Tables submenu:

So far we can list the tables within the database we are currently connected to, and drop an existing table.

You can use the Shift-Command-D combination to bring up a convenient contextual menu of the most common tasks, while you are within a document (unless you are editing an HTML file, in which case that combination brings up Safari):

The result of each command executed is flashed to the user in the form of an informative tooltip:

Conclusion

The aim of this bundle is to provide a more convenient way to work with DB2 without having to switch between your coding editor and the command line. It’s admittedly just a “seed”, a very early effort, but I sincerely hope that even at this stage it can be useful to some people. Being an open source project that’s released under the MIT license, you are free to contribute to it and more than welcome to fork it on Github as well.

Social Bookmarks

Memoization in Ruby and Python

2009-05-20T12:11:39Z

Wikipedia defines memoization as “an optimization technique used primarily to speed up computer programs by having function calls avoid repeating the calculation of results for previously-processed inputs.”. This typically means caching the returning value of a function in a dictionary of sorts using the parameters passed to the function as a key. This is done in order to reuse that returning value immediately without calculating it again, when the function is invoked with the same arguments. Even though we are trading space for time, it is often invaluable for speeding up certain recursive functions and when dealing with dynamic programming where intermediate calls are often repeated many times.

Using memoization in Ruby is very easy thanks to the memoize gem. The first step to getting started is therefore to install it:

$ sudo gem install memoize
Successfully installed memoize-1.2.3
1 gem installed
Installing ri documentation for memoize-1.2.3...
Installing RDoc documentation for memoize-1.2.3...

Now we can use the memoize method as illustrated in the example below:

require 'rubygems'
require 'memoize'
require 'benchmark'
include Memoize

def fib(n)
  return n if n < 2
  fib(n-1) + fib(n-2)
end

Benchmark.bm(15) do |b|
  b.report("Regular fib:") { fib(35) }
  b.report("Memoized fib:") { memoize(:fib); fib(35)}
end

In the first block we simply invoke fib(35), while in the second one we first invoke the method memoize(:fib) to memoize the method fib. Running this code on my machine prints the following:

                     user     system      total        real
Regular fib:    55.230000   0.160000  55.390000 ( 55.819205)
Memoized fib:    0.000000   0.000000   0.000000 (  0.001305)

We went from almost a minute of run time to an instantaneous execution. Optionally we could even pass a file location to the function memoize and this would use marshaling to dump and load the cached values on/from disk.

For Python we can write a simple decorator that behaves in a similar manner. In its simplest form it can be implemented as follows:

# memoize.py

def memoize(function):
    cache = {}
    def decorated_function(*args):
        try:
            return cache[args]
        except KeyError:
            val = function(*args)
            cache[args] = val
            return val
    return decorated_function

Or more efficiently:

# memoize.py

def memoize(function):
    cache = {}
    def decorated_function(*args):
        if args in cache:
            return cache[args]
        else:
            val = function(*args)
            cache[args] = val
            return val
    return decorated_function

When the memoized function has been invoked, we look in the cache to see if an entry for the given arguments already exist. If it does, we immediately return that value. If not, we call the function, cache the results and return its returning value.

Truth be told, the limit of this approach lies in the fact that since we are using a dictionary, only immutable objects can be used as keys. For example, we can use a tuple but are not allowed to have a list as a parameter. For the example within this article, this approach will suffice, but to take advantage of memoization when using arguments that are mutable, you may want to consider the approach described in this recipe.

We can now rewrite the Ruby example above in Python as follows:

import timeit
from memoize import memoize

def fib1(n):
    if n < 2:
        return n
    else:
        return fib1(n-1) + fib1(n-2)

@memoize
def fib2(n):
    if n < 2:
        return n
    else:
        return fib2(n-1) + fib2(n-2)	

t1 = timeit.Timer("fib1(35)", "from __main__ import fib1")
print t1.timeit(1)
t2 = timeit.Timer("fib2(35)", "from __main__ import fib2")
print t2.timeit(1)

Running this code on my machine prints the following:

9.32223105431
0.000314950942993

In Python 2.5’s case by employing memoization we went from more than nine seconds of run time to an instantaneous result.

Granted we don’t write Fibonacci applications for a living, but the benefits and principles behind these examples still stand and can be applied to everyday programming whenever the opportunity, and above all the need, arises.

Social Bookmarks