Zen and the Art of Programming

Heads up: IBM is looking for top notch student hackers

2010-03-05T18:19:09Z

As a thank you for following my blog, I’d like to introduce you to what I think is a great opportunity for the right students. My team is looking for two bright students for a 16 month, full-time internship opportunity with IBM.

Aside from being a bright and ambitious student, you should currently be working towards a Computer Science degree at any recognized University in the world (and have completed at least 2 years of your degree). You read that right, this opportunity is not limited to Canadian students. We are looking for the most talented students in the world, so as long as you’re ready to relocate to Toronto for 16 months, and are able to start the internship program in May 2010, we have the means to get you here. The location for this internship will be the Toronto Software Lab, in Markham, Ontario.

In particular, we are looking for students who are highly skilled in the area of web technologies and who are enthusiastic about cloud computing. This is NOT your run-of-the-mill, boring student job. You’ll be working with cutting edge technology on high visibility, fast paced projects that involve web technologies, cloud computing, Xp/Agile methodologies, and other related fields. You’ll be granted ample freedom to express yourself and your programming creativity. Should you be accepted for this well paid (in my opinion) internship program, you will report directly to my manager (a cool, down to earth guy) and I’ll be your mentor.

If you fit the criteria described above, please get in touch (cangiano at ca dot ibm dot com) and tell me a bit about yourself. You don’t need to attach your résumé at this stage, but if you drop me a line, I’ll be able to provide you with info on how to apply through the official IBM channels. We’ll take it from there.

Please note that I do NOT receive any form of compensation for referring you. I’m just hoping to provide a tiny subset of my readership with what could be a really great opportunity.

Bookmark and share:

DB2 support for Ruby/Rails turns 2.0

2010-01-21T21:03:24Z

The API development team just released a major version of the ibm_db gem. Detailed installation instructions are available on RubyForge (PDF). Among several improvements, there are three particularly newsworthy features:

Support for Ruby 1.9;
Support for mingw32, used by the latest version of the Ruby installer for Windows;;
Performance and security enhancements through “automatic” parameterized queries.

As we approach the release of Rails 3, supporting Ruby 1.9 is becoming more of a necessity. Likewise, the so called “One-Click installer” on Windows has been replaced by a current project that uses mingw32, which offers a much needed performance boost on Windows. Having a mingw32 compatible gem is starting to become a requirement for many of our Windows users.

Finally, DB2 is now the only database that supports prepared statements in ActiveRecord (without changing any of the application’s code). This has important performance benefits, as I explained in my article Improve the security and performance of DB2 Ruby on Rails applications using parameterized queries, which was published today by developerWorks.

Bookmark and share:

Grab DB2 Express-C 9.7.1 now and join the DB2Night Show

2009-12-03T18:42:25Z

IBM just released DB2 Express-C 9.7.1 (aka DB2 Express-C 9.7 FixPack 1). This version includes a long list of enhancements. Don’t forget to download and install it; as per its previous versions, it’s entirely free of charge.

You can grab it directly from here:

The DB2Night Show

While we are on the subject, I want to mention a great new independent initiative called the DB2Night Show. In the words of its organizers:

The DB2Night Show has the simple mission of building a fun, interactive community around DB2 LUW. Each episode will include fun stuff, education, tips, and special guests. Guests will range from industry experts to “Joe the DBA”.

The first six episodes have been wonderful, and the seventh one, which is about a comparison of DB2 Express-C with other free versions like SQL Server, as well as Cloud Computing, will air tomorrow.

You can reserve your free virtual seat for Friday, December 4, 2009 at 9am PST/ 10 am CST / 11 am EST, here. Even if you can’t make it, register anyway, so you’ll have access to replays. Every participant who attends will receive a $50 voucher to use for the Amazon EC2 cloud. Not only that, but one lucky, randomly selected attendee will also receive an Amazon.com gift certificate. With the holiday season coming up, that could come in handy.

Bookmark and share:

Setup Ruby Enterprise Edition, nginx and Passenger (aka mod_rails) on Ubuntu

2009-11-20T19:21:58Z

The following is a very short guide on setting up Ruby Enterprise Edition (REE), nginx and Passenger, for serving Ruby on Rails applications on Ubuntu. It also includes a few quick and easy optimization tips.

We start with setting up REE (x64), using the .deb file provided by Phusion:

wget http://rubyforge.org/frs/download.php/66163/ruby-enterprise_1.8.7-2009.10_amd64.deb
sudo dpkg -i ruby-enterprise_1.8.7-2009.10_amd64.deb
ruby -v

In output you should see “ruby 1.8.7 (2009-06-12 patchlevel 174)…” or similar. If this is the case, good; while you are there, update RubyGems and the installed gems:

sudo gem update --system
sudo gem update

Next, you’ll need to install nginx, which is a really fast web server. The Phusion team has made it very easy to install, but if you simply follow most instructions found elsewhere, you’ll get the following error:

checking for system md library ... not found
checking for system md5 library ... not found
checking for OpenSSL md5 crypto library ... not found

./configure: error: the HTTP cache module requires md5 functions
from OpenSSL library.  You can either disable the module by using
--without-http-cache option, or install the OpenSSL library in the
system,
or build the OpenSSL library statically from the source with nginx by
using
--with-http_ssl_module --with-openssl=
 options.

Instead, we are going to install libssl-dev first and then nginx and its Passenger module:

sudo aptitude install libssl-dev
sudo passenger-install-nginx-module

Follow the prompt and accept all the defaults (when prompted to chose between 1 and 2, pick 1).

Before I proceed with the configuration, I like to create an init script and have it boot at startup (the script itself is adapted from one provided by the excellent articles at slicehost.com):

sudo vim /etc/init.d/nginx

The content of which needs to be:

#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO

PATH=/opt/nginx/sbin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/opt/nginx/sbin/nginx
NAME=nginx
DESC=nginx

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
    . /etc/default/nginx
fi

set -e

. /lib/lsb/init-functions

case "$1" in
  start)
    echo -n "Starting $DESC: "
    start-stop-daemon --start --quiet --pidfile /opt/nginx/logs/$NAME.pid \
        --exec $DAEMON -- $DAEMON_OPTS || true
    echo "$NAME."
    ;;
  stop)
    echo -n "Stopping $DESC: "
    start-stop-daemon --stop --quiet --pidfile /opt/nginx/logs/$NAME.pid \
        --exec $DAEMON || true
    echo "$NAME."
    ;;
  restart|force-reload)
    echo -n "Restarting $DESC: "
    start-stop-daemon --stop --quiet --pidfile \
        /opt/nginx/logs/$NAME.pid --exec $DAEMON || true
    sleep 1
    start-stop-daemon --start --quiet --pidfile \
        /opt/nginx/logs/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS || true
    echo "$NAME."
    ;;
  reload)
      echo -n "Reloading $DESC configuration: "
      start-stop-daemon --stop --signal HUP --quiet --pidfile /opt/nginx/logs/$NAME.pid \
          --exec $DAEMON || true
      echo "$NAME."
      ;;
  status)
      status_of_proc -p /opt/nginx/logs/$NAME.pid "$DAEMON" nginx && exit 0 || exit $?
      ;;
  *)
    N=/etc/init.d/$NAME
    echo "Usage: $N {start|stop|restart|reload|force-reload|status}" >&2
    exit 1
    ;;
esac

exit 0

Change its permission and have it startup at boot:

sudo chmod +x /etc/init.d/nginx
sudo /usr/sbin/update-rc.d -f nginx defaults

From now on, you’ll be able to start, stop and restart nginx with it. Start the server as follows:

sudo /etc/init.d/nginx start

Heading over to your server IP with your browser, you should see “Welcome to nginx!”. If you do, great, we can move on with the configuration of nginx for your Rails app.

Edit nginx’ configuration file:

sudo vim /opt/nginx/conf/nginx.conf

Adding a server section within the http section, as follows:

    server {
        listen 80;
        server_name example.com;
        root /somewhere/my_rails_app/public;
        passenger_enabled on;
        rails_spawn_method smart;
    }

The server name can also be a subdomain if you wish (e.g., blog.example.com). It’s important that you point the root to your Rails’ app public directory.

The rails_spawn_method directive is very efficient, allowing Passenger to consume less memory per process and speed up the spawning process, whenever your Rails application is not affected by its limitations (for a discussion about this you can read the proper section in the official guide).

If you have lots of RAM (e.g., more than 512 MB) on your server, you may want to consider increasing you maximum pool size, with the directive passenger_max_pool_size from its default size of 6. Conversely, if you want to limit the number of processes running at any time and consume less memory on a small VPS (e.g., 128 to 256MB), you can decrease that number down to 2 (or something in that range). (Always test a bunch of configurations to find one that works for you). You can read more about this directive, in the official guide.

While you are modifying nginx’ configuration, you may also want to increase the worker processes (e.g., to 4, on a typical VPS) and add a few more tweaks (such as enabling gzip compression):

# ...
http {
    passenger_root /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.5;
    passenger_ruby /usr/local/bin/ruby;

    include       mime.types;
    default_type  application/octet-stream;

    access_log  logs/access.log;

    sendfile        on;
    keepalive_timeout  65;
    tcp_nodelay on;

    gzip on;
    gzip_comp_level 2;
    gzip_proxied any;   

    server {
    #...

When you are happy with the changes, save the file, and restart nginx:

sudo /etc/init.d/nginx restart

If you wish to restart Passenger in the future, without having to restart the whole web server, you can simply run the following command:

touch /somewhere/my_rails_app/tmp/restart.txt

Passenger also provides a few handy monitoring tools. Check them out:

sudo passenger-status
sudo passenger-memory-stats

That’s it, you are ready to go! I hope that you find these few notes useful.

Bookmark and share:

What is DB2 pureScale?

2009-10-21T14:38:20Z

There are two main types of scalability: vertical and horizontal. Vertical scalability consists of potentiating the hardware specs of a given server. This is typically done by increasing the number (and to a minor extent, speed) of the processors, adding more RAM, and so on. Commodity hardware tends to impose a strong limit on the resources that can be augmented. There is only so much that can be stuffed into your typical x64_86 Dell server, even replacing the entire machine with the top of the line model (having the same architecture). For this very reason, many customers opt to purchase more expensive hardware that has the advantage of facilitating vertical scalability, by allowing much more headroom for additional resources. This is why Unix-based solutions like IBM System p + AIX and Sun’s Solaris + SPARC are common choices in demanding environments.

For all their merits, even premium hardware solutions can only do so much in terms of vertical scalability. They may postpone the problem, by allowing a large number of processors and a huge amount of RAM to be installed, but they are still not limitless. And that’s when it becomes cheaper and much more feasible to attempt horizontal scalability, or the process of adding extra servers that share data and the overall system load.

Commercial databases, particularly DB2, excel at vertical scalability, which incidentally is one of the weakest aspects of open source RDBMS like MySQL. When it comes to horizontal scalability though, even commercial databases tend do struggle a little. Scalability is achieved, but a lot of work is required to configure complex setups involving masters, slaves, replication, failover servers, and so on. Until now.

IBM has just introduced a new technology for DB2 called pureScale, which accomplishes three incredible feats:

Horizontal scalability: Virtually unlimited capacity by simply adding new nodes to the cluster.
Application transparency: No code changes needed.
Continuos data availability: regardless of which server fails, the system will continue to operate albeit with a proportionally reduced throughput.

DB2 pureScale applies to DB2 running on Unix (IBM System p running AIX only, for the moment), the same principles and rock solid technology employed by DB2 for z/OS Data Sharing and SYSPLEX. This technology happens to be employed by some of the largest companies in the world and it’s considered the golden standard when it comes to easily create OLTP database clusters, providing virtually unlimited scalability capabilities.

You can read more about it here and here. If you are going to IOD this year, don’t forget to attend this overview by one the architects of DB2 pureScale. And even if you can’t head over to Las Vegas next week, you can still attend online our Chat with the Lab today at 11:30am EST. Database scalability is a hot topic and this is going to be one of the most popular chats with the lab ever, so register for free now.

Bookmark and share:

Add code highlighting to your Google Waves

2009-10-20T00:53:46Z

Google Wave is still rough around the edges, but it has a lot of potential in terms of becoming a great collaboration tool. As a developer, your first question will probably be: “How do I add code highlighting to my waves?”. The answer is straightforward, however not very easy to find if you google it. I hope this post will help fellow developers who are experimenting with Google Wave.

The following steps are required to obtain syntax highlighting for your code:

Create a new wave and add the Syntaxy robot to your wave. Use the wave address: kasyntaxy@appspot.com.
Reply to your first message or within it, thereby creating a reply (called “blip” in Google lingo).
Specify your code’s language, prefixing the name with a hash and exclamation mark, like #!python or #!ruby.

At this point, as you type the code in your blip it will be highlighted by the Syntaxy bot as shown in the picture below:

More advanced automatic syntax highlighting bots will probably appear as Google Wave progresses, but this one should do the trick for now. On a side note, if you copy and paste code from XCode, the code formatting will be kept in your waves and blips without the need for bots.

Bookmark and share:

Getting MacRuby’s compiler to work

2009-10-08T20:49:32Z

There is major news in Rubyland today. MacRuby’s team just released their fist beta of version 0.5 (an experimental, still incomplete version of Ruby), which brings JIT, removal of the dreaded GIL (Global Interpreter Lock), native threads, GCD (Grand Central Dispatch) for multicore computing, and a whole new set of features found in the release announcement to the table.

The most important new feature is the presence of a compiler. That’s right, thanks to this release, Ruby code can now become highly optimized executable code. How awesome is that? I can sense that you’re pumped by this news, so why not head over to MacRuby.com and download the installation file for yourself? After you’ve done that, the next thing you’re going to want to do is run a small test like the following:

$ macrubyc world_domination.rb -o world_domination
Can't locate program `llc'

Oh noes! llc is a tool that ships with the LLVM (upon which MacRuby is built), however it’s not included with MacRuby’s installer (it will be in the future). But fear not my friends, there is a solution:

$ svn co -r 82747 https://llvm.org/svn/llvm-project/llvm/trunk llvm-trunk
$ cd llvm-trunk
$ ./configure
$ UNIVERSAL=1 UNIVERSAL_ARCH="i386 x86_64" ENABLE_OPTIMIZED=1 make -j2
$ sudo env UNIVERSAL=1 UNIVERSAL_ARCH="i386 x86_64" ENABLE_OPTIMIZED=1 make install

If your machine does not have 2 cores, remove the -j2 option from the fourth line or adjust the number accordingly.

The compilation phase may take a couple of centuries, depending on your machine’s speed, but it should eventually build the LLVM. llc will be placed in your PATH, and you’ll finally be able to compile Ruby code and obtain an executable to help you carry out your world domination plans.

$ macrubyc world_domination.rb -o world_domination
$ ./world_domination
MUAHAHAHAHA!

Bookmark and share:

The CRTC tries to eliminate ISP competition in Canada

2009-09-24T16:18:21Z

The CRTC is at it again. This time they want to allow Canadian phone companies to cut off their wholesale offering, essentially killing ISP competition in Canada. Please express your concerns here, where you can quickly email Industry Minister Tony Clement, Prime Minister Harper, the Opposition Leader (Michael Ignatieff) and your local MP. Let them know that you care about this important issue.

For more details, I’m attaching an informative email I received from TekSavvy:

Hello,

We need your help — we need you to lend us your voice and state your opinion on a recent CRTC decision that could allow the incumbent phone companies (such as Bell and Telus) to discontinue offering services (Internet and phone) to wholesale providers which in effect would eliminate competition in Canada.

With no competition to help fuel the market; Internet innovation could come to stand still and prices for broadband access could skyrocket out of control. The incumbents would also be free to do with the lines as they please. This would open up future privacy invading Deep Packet Inspection (DPI) technologies and direct marketing/advertising based on *you* and *your families* internet activities and habits.

They could also use this same technology to turn the Internet into a pay-per-website service the same way that TV is a pay-per-channel service. Can you imagine a world where you have to pay to use the Google website to search for information? Then, if you find what you are looking for you may be restricted to the content because your ISP doesn’t have a “peering” agreement with said company.

Please help us stop this major potential catastrophe.

All concerned consumers please visit:
http://www.competitivebroadband.com/consumer/

All concerned businesses please visit:
http://www.competitivebroadband.com

Read the information presented and decide for yourself.

Once you’ve decided what you think is right, you can click the “Make a Difference” link, fill out the form with your details and send a pre-written message to Minister of Industry Tony Clement, Prime Minister Harper, the Opposition Leader, and your local MP or choose to write your own.

Together we can make a difference.
Thank you,
TekSavvy Solutions, Inc.

Bookmark and share:

Benchmarking Tornado vs. Twisted Web vs. Tornado on Twisted

2009-09-14T07:10:11Z

FriendFeed, which was recently acquired by Facebook, just released an interesting piece of open source software.

Tornado is an open source version of the scalable, non-blocking web server and tools that power FriendFeed. The FriendFeed application is written using a web framework that looks a bit like web.py or Google’s webapp, but with additional tools and optimizations to take advantage of the underlying non-blocking infrastructure.

The story so far

This release generated widespread interest among the Python and open source development communities. Rightfully so. There are many reasons to like Tornado. To begin with, it’s fast — and that’s fundamental for a web server. By using nginx as a load balancer and a static file server, and running a few Tornado instances (usually one per core available on the machine) it’s possible to handle thousands upon thousands of concurrent connections on relatively modest hardware; and this isn’t just theory. Tornado has already proven its worth in the field, by allowing FriendFeed to scale graciously.

Tornado is not only a fast web server, it acts as a very lightweight application framework as well. As such, it’s an appealing alternative to well established frameworks to the growing group of developers who’d like to develop “closer to the metal” and avoid the baggage associated with full-fledged web frameworks. The two things combined make Tornado ideal for developing “real time” web services and applications.

The feedback so far hasn’t been all positive though. Criticism of the project has mainly focused on the lack of test coverage and the fact that FriendFeed has opted not to contribute to, and improve on, the existing Twisted Web project (which has similar goals). To make things worse, there were a few nonchalant comments about it as well. Performance issues and lack of ease of use were the reported motivations for starting a new project from scratch.

Dustin Sallings started working on a hybrid solution (henceforth Tornado on Twisted) that would reportedly keep the good parts that Tornado introduced, while using Twisted as its core for networking and HTTP parsing.

At this point I became naturally curious about the speed of these three web servers. Is Tornado really faster than Twisted Web? And what about Tornado on Twisted, would it be faster or slower? Let’s find out.

Benchmark results

I ran a simple Hello World app for all three web servers. All the web servers were run in standalone mode without a load balancer. I stress tested the web servers with httperf using a progressively larger amount of concurrent requests. 100,000 requests were generated for each test. The web servers were run on a desktop machine with an Intel® Core™2 Quad Processor Q6600 (8M Cache, 2.40 GHz, 1066 MHz FSB) processor and 8GB of RAM. The operating system of choice was Ubuntu 9.04 (x86_64).

Without further ado, here are the results:

As you can see Tornado turned out to be faster than the rest of the Python web servers. Handling a peak of almost 3900 req/s with a single front-end and on commodity hardware is nothing to sneer at.

Twisted Web didn’t do too bad either (max. 2703.7 req/s), but the difference in performance is noticeable. Likewise, the performance of Tornado on Twisted was virtually identical to that of Twisted Web.

There you have it. I was curious about the possible outcome and now I know. Remember, this is a report on the numbers I got on my machine, not a research paper. But I hope that you find them interesting nevertheless.

Show me the code

Tornado:

import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import logging

from tornado.options import define, options

define("port", default=8888, help="run on the given port", type=int)

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world!")

def main():
    tornado.options.parse_command_line()
    application = tornado.web.Application([
        (r"/", MainHandler),
    ])
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

if __name__ == "__main__":
    main()

Twisted Web:

from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor
from twisted.web import server, resource

class Simple(resource.Resource):
    isLeaf = True
    def render_GET(self, request):
        return "Hello, world!"

site = server.Site(Simple())
reactor.listenTCP(8888, site)
reactor.run()

Tornado on Twisted:

from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor

import tornado.options
import tornado.twister
import tornado.web
import logging

from tornado.options import define, options

define("port", default=8888, help="run on the given port", type=int)

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world!")

def main():
    tornado.options.parse_command_line()
    application = tornado.web.Application([
        (r"/", MainHandler),
    ])

    site = tornado.twister.TornadoSite(application)
    reactor.listenTCP(options.port, site)

    reactor.run()

if __name__ == "__main__":
    main()

UPDATE (September 14, 2009):

The original version of this post included Unicorn as well. This wasn’t fair however, since it’s not an asynchronous web server.
EventMachine HTTP Server was added, but I have since decided to remove it as I prefer to let the article be a fair comparison between asynchronous Python web servers.
I initially used Apache Benchmark (ab). The results were misleading at best. I re-ran the tests with httperf and updated the results above.
Stock Tornado couldn’t be tested with httperf because their HTTP Server doesn’t implement getClientIP(). I had to manually modify a method to return the remote ip address. This may introduce a very minimal advantage for Tornado, but it should be negligible in this context.
I modified the examples for Twisted and Tornado on Twisted, to ensure that both took advantage of the epoll-based reactor.

Bookmark and share:

Installing and configuring DB2 Express-C 9.7 on Ubuntu 9.04 Server

2009-09-11T01:53:00Z

DB2 Express-C 9.7 can easily be installed on Ubuntu 9.04 Desktop edition by simply issuing sudo ./db2setup. This will open up a launchpad and you’ll be able to install the product through a wizard. But what about setting up DB2 Express-C 9.7 on Ubuntu 9.04 Server edition? When you are ssh-ing into your VPS or dedicated sever, there are no GUIs to help you out.

IBM provides all the information on how to proceed through a series of detailed documents, which explain why each command is required and what your options are. But if you are looking for a single document that is slightly more concise in nature, you can read this excellent step by step guide that was published today on the blog bits.of.info.

Bookmark and share: