Ken's Blog

Double buffering with HTML 5 Canvas

2012-12-22T01:38:00.002+00:00

A neat trick with the HTML 5 canvas element in JavaScript is to use a second canvas to draw your canvas of screen, and flip it to a main canvas (examples include JQuery):

/* Prepare your canvas */
mainCanvas = $('#myCanvas');
offscreenCanvas = document.createElement('canvas');

/* Make sure your offscreen size matches your on screen
* size */
offscreenCanvas.width = mainCanvas.width;
offscreenCanvas.height = mainCanvas.height;

/* Do some drawing on the offscreen canvas */
ctxOffscreen = offscreenCanvas.getContext('2d');
ctxOffscreen.strokeRect(0, 0, 10, 10);

/* Now flip the offscreen canvas to the onscreen
* one */
ctxMain = mainCanvas.getContext('2d');
ctxMain.drawImage(offscreenCanvas, 0, 0);

And then your done. Whats nice about this, is its a good technique for creating flicker-reduced animations so you can prepare your next screen and 'flip' it when ready. The actually copy is done in hardware in most browsers so its quite performant.

Whats really nice about this concept, is that one can use 'drawImage' to overlay multiple offscreen canvases onto a single canvas. Combined with the use of alphas (transparency) you can create layers to a scene at separate stages and combine them at 'paint' time to be displayed on the onscreen canvas. Quite cool really.

Another technique I've heard about but never really used is to prepare 2 canvases and change the visibility:

/* Lets presume you already have two canvases with
* absolute layouts in the same place */
firstCanvas = $('#firstCanvas');
firstCanvas.css('display', 'block');
secondCanvas = $('#secondCanvas');
secondCanvas.css('display', 'none'); // hidden like a ninja

/* Do some drawing on the 'hidden' canvas */
ctxSecond = firstCanvas.getContext('2d');
ctxSecond.strokeRect(0, 0, 10, 10);

/* Now flip the two canvases, display one and hide the
* other */
firstCanvas.css('display', 'none');
secondCanvas.css('display', 'block');

I can't speak for the performance gain of this second methodology, but I can see it might have its advantages. Most conversations around double buffering suggest a technique like this, so its probably worth investigating.

How I got involved with Configuration Management

2012-06-29T16:40:00.002+01:00

I sometimes speak to people publicly for my company, and I tell them I've been involved in Configuration Management for 13 years. Obviously Puppet has not been around for that long, and a lot of people ask me if it was cfEngine or something else I used, well - it was bespoke - and I want to talk a little bit about that.

In fact if someone was to look at my Linked in account, they would realise that the length of my administration experience is roughly that - 13 years. This is because there was never a time where I wasn't using configuration management of some kind.

When I was younger and much more foolish I started work at a company called Microplex. Microplex was a Sydney based ISP and I had managed to work my way into Engineering thanks to the kind promotion of an old friend and mentor Michael O'Reilly. At round that time Microplex was bought by Optus - one of the larger Telco's in Australia.

Michael O'Reilly used to partly own iiNet, one of the bigger ISP's in Australia today. When he started working in engineering at Microplex, he introduced a system for configuration management from the start - it was called 'conform' and it consisted of about 100 lines of Perl code.

My first experiences as an Engineer were what I would later consider to be some of the best in my life. The team was way ahead of their time, and in many cases way ahead of most technical teams even today (and this was 1999).

At the time I had no idea about how to administrate systems, but my senior colleagues were automation mad. In fact we were all called 'Perl Programmers' not administrators. This was DevOps long before the term was invented.

We had 'conform' doing our configuration management, which was awesome. In fact I think I was one of the earlier people to create a fully automated build, at the time it was the 'usrweb' system - you know - the place where you use to upload your personal 5 mb website for Optus/Microplex? That was rebuilt by me using conform - 100% automatically. In 1999 no less.

When I left that group to work in other teams I tried to follow their ways of working but always hit up against the manual error problem, being young and naive I couldn't understand how these people worked at all. Having grown up with pure CM deployment I was shocked to learn that the outside world had no idea nor concept of this. My attempts at introducing other tools always hit up against dusty engineers spouting 'we aren't developer' mantras which drove me mad.

In fact, there was a time where building systems was documented fully in large 200 page engineering manuals. Every configuration file, every Cisco router configuration. I was totally shocked by this way of working and only lasted a very short amount of time in these types of roles as my frustration with my colleagues drove me mad.

Later I tried reimplementing the things I had at my first job at other places. I tried coercing JASS and Jumpstart to do things, but the single build concept was useless in a world of changes. I tried early versions of cfEngine only to find myself doing more work then necessary and living in a sea of badly documented tooling.

I tried writing tools myself and tried forcing them on teams, only to find myself being the only one who maintained them as no one saw the benefit. Yep, my career was depressing on that level - as it seemed everyone was quite happy and job satisfied with making manual changes. Sad really knowing what I knew.

Later I came back to work nearby the team I originally worked with at Optus/Microplex (I moved a bit within Optus) and found they were still using the old tooling, 8 years later. And still succeeding with it. In fact the tooling was so ingrained into their deployment process it was mandatory to use it, and manual changes were not allowed unless under incident. Brilliant :-).

Later when I left Optus to come to England for fame and opportunity I found myself working for Alfresco. Alfresco at the time I had very little systems administration, as there infrastructure was small at that time. I immediately wanted to implement a tool, and learning the lesson from having written my own tools in the past I went looking for an Open Source tool.

And this is how I got into Puppet, I tried all the popular tools - and Puppet started working for me quickly and I understood it the best. The resource model and granular ordering was something I had petitioned for at Optus for conform, but the change was too big and hard to make. So seeing that a tool was already a generation or two ahead of our old thinking and got right into it, and couldn't stop.

I went on to automate almost 80% of what we did at Alfresco, and I believe they still use the tool.

I had a stint at another company, and did the same thing - automated the hell out of it.

Then James Turnbull from Puppetlabs reached out to me to work for them - I couldn't say no. Now I'm a developer I'm as happy as larry, as this is my favourite problem domain to work on. I'm also surrounded by extremely smart people who are passionate about this problem domain as well. What more could I want?

So that's it, I've always been into configuration management. There hasn't been a time really when I wasn't or wasn't trying to get it into a company (and being ignored mostly). So if I struggle to answer your questions when you ask me why Configuration Management - its because I don't really understand the world without it. Its like a world without a modern Linux operating system, or a world without Open Source ... I just can't comprehend the world without it.

And - when people say to me 'automation is too hard, it will take us years to automate' remember this: Microplex was doing it in 1999, without any tools on the market to help them besides Perl. If your not doing it yet, change your attitude fast, and step into it - ignorance or fear shouldn't be an excuse any more.

A Review of the book 'OpenNebula 3 Cloud Computing'

2012-06-29T15:28:00.002+01:00

Some time ago I was at a conference and listened to an introduction to OpenNebula 2 by Tino Vasquez from the OpenNebula team. I was fascinated by the simplicity of the product and went on to write a Puppet module for it.

Today PacktPub have asked me to write a review of an early beta of the most recent book on the platform 'OpenNebula 3 Cloud Computing' by Giovanni Toraldo and lacking anything better to do I thought I would give it a shot (just to be clear I'm doing this for the fun of it, no money has changed hands - I just like to write about things).

For those who aren't savvy with the technology OpenNebula is an Open Source cloud framework aimed at enterprises who wish to run private clouds, but it also provides hooks into public clouds as well so you get this 'single management' effect no matter which direction you wish to go.

The book begins with the normal introduction to the PaaS, SaaS & IaaS terminology for those who aren't in the know. Starting small is a good idea I believe, it really is a brave new world for some people.

It also provides a lot of terminology that applies to most clouds. When the cloud industry started people barely knew where to start. Its funny now we can effectively model how most clouds should look, and the intro does a good job at describing that model.

Building

3 hypervisors are covered in detail: KVM, Xen and VMWare ESXi. Which are the three bigger players in the market. Its important that all three are covered in a book such as this, as different companies have invested in different paths.

It talks about how Libvirt plays in with these technologies, Libvirt being integral to the integration within OpenNebula as it takes away some of the heavy lifting.

The book states at the beginning that its focused on using Ubuntu 10.04. While I think this is fine I can hear the audible whine of the corporates who have rested their laurels on Redhat. I guess most people need to be mindful that this is just the hypervisor not the OS that applications will sit on, however haters will hate and divisions are clear. In my opinion over the years I've started to see that a hybrid approach is probably better, but still this doesn't help with licensing this kind of environment when your guests are going to be Redhat for example.

Still, Debian and Ubuntu have put a lot of effort in making this platform work for them which is a testament to their great efforts lately in the server space.

The book does however talk about installation from source and pre-built packages, and with enough knowledge Redhat users can probably happily find packages or build them using FPM if need be.

Storage

The storage section is detailed, it goes through the basic variants using SCP transfers and of course NFS backed storage. Which alone are enough to get most people up and running.

It also spends time talking about some distributed storage systems: MooseFS and GlusterFS. Its a shame Ceph isn't ready for prime-time yet, as I would love to see this covered at some point as well (perhaps next edition). Conversely though it does step into how a transfer driver is written, which gives you the grounding for adapting to any storage mechanism if you chose.

Instances

The book walks you though how to launch the example ttylinux demo instance. This instance itself is more of a light example for users, not really a production instance

The real meat that I wanted to see however was how to setup your own instance, as this is what most people have trouble with. However the chapter was very light on this, just mentioning it needs to be done on a another PC and imported into the OpenNebula environment. I would have liked to have seen a discussion on this in greater detail and some points on how to do it in a few hypervisors with Linux and Windows. Oh well, you can't have everything.

The chapter however does talk about contexualisation of your VM quite a bit. This is where you individualise a machine from a stored image by passing scripts and data to the VM as it boots in the form of a mini ISO image. Philosophically this is the place where your own individual solutions come out, as this is more about how to modelling your environment - and everyones needs are different.

Web Interfaces

The section on setting up Sunstone wasn't my favourite, but should be good enough to get people up and running. The tool itself is a pinch to setup, being a simple Ruby Sinatra application. However I would have preferred documentation on how to make this scale with a proper Ruby application server such as Unicorn or Passenger, which are the more preferred technologies for production Ruby web applications.

They do have some information on how to configure Nginx however as a reverse-proxy, which is at least a start.

The instructions on how to use the interface was very detailed however and included a lot of screenshots which was nice.

Monitoring

The monitoring sections were very detailed and went into how to setup custom monitors for VM state which was interesting, especially for those who have custom requirements. It also describes how to setup scripted hooks for scenarios like VM failure, which is a real treat for those who are control freaks.

There was an entire section for setting up Ganglia for uptime monitoring and OpenNebula integration. Its nice to see this covered, as monitoring is pretty critical to a real solution. Even users using other monitoring tools can take something away from this section as it helps one understand the entry points for monitoring regardless of tool.

Public Cloud Integration

I must say the entire section on AWS integration was very detailed. I'm not an expert on this part of OpenNebula, and the integration with AWS is often glossed over. The author however has spent quite a bit of time explaining how to make this work with Linux and Windows instances.

In the next section of the book the author also covers the EC2 interface that OpenNebula emulates so you can use EC2 compatible API's which is a very nice feature.

Other Interfaces

The final chapter gets into the nuts and bolts of how to configure OpenNebula to expose an EC2 and OCCI interface to the world, so that it can be called by third party tools designed around these protocols. The EC2 feature is of course interesting, as this means you can expose your private and public clouds using the same EC2 interface.

It also talks about one of my more favourite topics: Zones and Virtual Data Centres. The concept of Zones here is to allow a single OpenNebula cloud serve multiple 'customers' if you like: be it separate IT groups or even separate commercial customer entities. I believe this to be an integral part of the 'server provider' aspect of cloud computing weither you are going to zone your various development groups or customers A, B and Z.

In this section we get some setup instructions on how to configure the oZones service using Apache (not Nginx this time) as a reverse-proxy and some nice detailed instructions on how to use the management GUI.

Summary

I think all in all the book is a good reference guide, and would be perfect for those stepping out into building an OpenNebula driven cloud for themselves. It reads well, and the layout is front-to-back so you can step through it in a lab.

In some parts the author could have extended into more detail on production setups (use of Passenger, Unicorn for example) and scaling the oned daemon, as I feel these are the questions people ultimately ask.

It drills into some detail and explanations, and generally doesn't focus on one type of solution which is nice to see. All in all a good read and I'd recommend it to anyone stepping into the world of cloud building with OpenNebula. Well done Giovanni :-).

If you are interested in obtaining this book, it can be purchased from the PacktPub website at: http://link.packtpub.com/NUSJO7

Torquebox: Making Message Processors behave Synchronously

2012-06-11T15:28:00.000+01:00

In Erlang using processes and in-built functions you are able to send message to processes anywhere in your cluster, and there is a pattern to extend this to wait on a returned message as well. This allows you to call out to a separate utility process to do some work and wait for its response.

This can be quite useful for example where you might want to move common processing off to a remote machine, (like image processing or something that takes time & cpu) where you don't want to consume the CPU on the current machine.

In erlang the code example is quite straight-forward for this (this is a simplified variation taken from: http://www.erlang.org/doc/getting_started/conc_prog.html):

-module(async).

-export([start/0, ping/1, pong/0]).

ping(Pong_PID) ->
    Pong_PID ! {ping, self()},
    receive
        pong ->
            io:format("Ping received pong~n", [])
    end.

pong() ->
    receive
        finished ->
            io:format("Pong finished~n", []);
        {ping, Ping_PID} ->
            io:format("Pong received ping~n", []),
            Ping_PID ! pong,
            pong()
    end.

start() ->
    Pong_PID = spawn(async, pong, []),
    spawn(async, ping, [Pong_PID]).

In this example in the 'start()' function we begin by spawning the receiver process 'pong'. This process is setup to receive messages and respond to them using the syntax:

Ping_PID ! pong

Thus we have our receiver/processor ready and prepared. The second step is that we spawn a second process that is responsible for sending the message, ie. the 'ping'. Now here is where it gets interesting.

The outbound message is sent with:

Pong_PID ! {ping, self()}

This means the outbound message is sending the symbol 'ping', and a reference to its own process. It then blocks indefinately, waiting for a response of 'pong':

receive
    pong ->
        io:format("Ping received pong~n", [])
end.

Quite simple really. The entire point here is that both of these processes can live on completely separate machines, the receiver or processor able to do work on its own, while the requester can simple wait for a response.

I saw this pattern recently echoed when I was looking at the Torquebox platform:

http://torquebox.org/documentation/2.0.3/messaging.html#synchronous-messaging

Here we have a very basic copy of the same concept:

queue = TorqueBox::Messaging::Queue.new('/queues/foo')
Thread.new {
  queue.receive_and_publish(:timeout => 5000) { |message| message.upcase }
}
message = queue.publish_and_receive "ping", :timeout => 5000

Which is interesting on its own, but it does require you to write your own Torquebox Service to listen for these requests.

I figured we could do better then that though. In Erlang it most certainly supports the concept of processor groups and supervisors and workers for scaling up a set of workers and I wanted to mimic this somehow.

So ... Torquebox already has its own concept of workers that can receive messages on queues and scale up as your needs scale - these are called Message Processors:

http://torquebox.org/documentation/2.0.3/messaging.html#messaging-consumers

Now the beauty of these Message Processors is that they are fairly much just Ruby objects that have little knowledge of the fact they are at the end of a queue. What I like about them is that they allow me to think using the Actor model, however they don't support the synchronous methodology out of the box as described above so I set out to mimic it myself.

First of all, synchronous messages contain some extra data when sending messages to designate the request and response. In particular the call 'publish_and_receive' includes the message parameter synchronous = 'true'.

So lets start out by making sure our message processor setup is correctly. In your torquebox.yml you add the following:

queues:
  /queues/ping_pong:
    durable: true

messaging:
  /queues/ping_pong:
    PingPong:
      selector: "synchronous = 'true'"
      concurrency: 5

Note the selector setting - this means it will only receive messages with this parameter setting in the JMS message. This is important and very relevant to ensure the receiver doesn't get all the response messages we're going to send back onto the queue later on. I've also set the concurrency to 5 just to show how you can control the number of workers/receivers.

Now for the receiver code (ping_pong.rb):

require 'rubygems'
require 'torquebox-messaging'

class PingPong < TorqueBox::Messaging::MessageProcessor
  def initialize(hash = {})
    @logger = TorqueBox::Logger.new
  end

  def on_message(body)

    # The synchronous methods always put the user message inside

    # a :message parameter, hence the need to descend here.
    message = body[:message][:command]

    # This should result in 'ping' being displayed in your logs

    @logger.info(message)

    sync_reply({'command' => 'pong'})
  end

  def sync_reply(reply)
    queue_name = message.jms_message.jms_destination.name
    jms_message_id = message.jms_message.jms_message_id

    queue = TorqueBox::Messaging::Queue.new(queue_name)
    queue.publish(reply, {

      :correlation_id => jms_message_id,

      :encoding => :json}

    )
  end
end

What's happening here, is that when the queue receives a message that matches the selector above - it will send the entire body of the message to the #on_message method. The body is already decoded using JSON.parse so we can just treat it like a hash. The main difference between this and normal Message Processor behaviour is that the method #publish_and_receive will always embed the users message inside the 'message' parameter, so we need to retrieve things from there instead.

At the end of processing (in this case we just log the received message) I've provided an extra method called 'sync_reply'. This method is responsible for creating the reply and adding the necessary 'correlation_id' setting to make sure the response is only caught by the original sender.

The important part here is that we must make sure:

We send the message back on the same queue as it was sent out on. We do this by unpacking the JMS message and pulling out the 'jms_destination'.
We make sure we add a :correlation_id field when sending the message back, this correlation_id must match the original sender's 'jms_message_id'. This is because the original sender is now waiting using a receive call with a selector designated to only match that correlation_id.

So now that you have a receiver built, if you want to send a message it's simple, you can use the built-in publish_and_receive method from somewhere else in your application:

queue = TorqueBox::Messaging::Queue.new("/queues/ping_pong")
response = queue.publish_and_receive(
  {:command => "ping"},
  {:encoding => :json, :timeout => 60000 }
)
puts response.inspect

And thus we have achieved synchronous messaging using Torquebox Message Processors.

Newbie programmer: Focusing on getting the job done

2012-06-09T02:53:00.001+01:00

So I've started work for the first time as a full time developer at a popular software company recently, and I've been jotting down some notes on my first impression on how to get stuff done and be as productive as possible. It might be totally wrong, its primarily from the perspective of a newbie - but might be an interesting read none-the-less.

If you have your own ideas of getting things done, and you have more experience then this spring chicken, feel free to comment.

Prototype first, optimize later

Premature optimisation is actually an anti-pattern according to Wikipedia, and I'm starting to realise why. With the amount of backwards and forwards one can do in code during a design phase - it makes sense to optimize later once the prototype and design has been agreed on.

I find though doing this in chunks once you feel you have a feature working enough is a good idea. Leaving optimisation late can make it feel like a less interesting task, so wrap it up in chunks.

Style

Get style issues out of the way early. Write using good style all the time, it can avoid a lot of unnecessary chatter with your colleagues. Programmers I know have a special aversion to bad style and its not worth the hassle I assure you. Getting to know their style and sticking to it saves time.

Task Tracking

Use a micro task tracker to keep track of tasks that don't really belong in a major task tracker. I use OmniFocus but any GTD style tracker is probably adequate.

Music

Get yourself some playlists for music - something that runs for many hours so you don't have to break concentration to switch tunes. I use Youtube for my music on my spare Ubuntu desktop here, but if the ads nag you, get something on ITunes or whatever is your favourite poison.

I also have different play-lists for different moods. I have slower music for concentration and thinking about problems, and faster music for just hammering things out.

Distractions in the Workspace

Sort out distractions early, don't let background noise linger - make it clear to people around you that you are in a sprint and you don't want to be disturbed.

I work from home a lot. My kids know now that I need my peace and quiet during 'work hours', and I try to make it up to them when I'm out of work mode.

Socialising Specifications and Design

Get your specifications out in the open earlier, if you are designing a website, protocol or otherwise - write the design early, and get the feedback loop happening earlier. This saves you having to rewrite a lot of code later on.

You might presume that the design is 'obvious' and follows standards. The truth is probably far from it. Once you put it on paper (well - google docs or whatever) you'll find your assumptions shattered. This is a good thing.

Fussing over old code

When looking at new features you want to add to existing code, you may find you are not happy with the existing code. Mostly, this is due to your initial understanding of the objectives of this code as written at the time - that will pass. But regardless, its not worth fretting or pondering why code was written in such a way. If you find yourself wanting to adjust the code - just do it and move on, but make sure you understand the overall objective before destroying someones dream.

You're probably wrong

When socialising your work amongst your colleagues, you may get opinions that don't make sense to you. Welcome to the real world. I've learnt that the best thing to do is to understand these needs and try to meet them if they are reasonable, chances are the person has a good perspective. I think as a programmer you will have to get used to outside influence - I think embracing with good debates and discussions is easier then rejecting it.

Also - be humble. Be prepared for being wrong. As a user you were always right ... as a programmer, you have to listen to your users.

Pretty Ruby Data

2012-02-28T22:37:00.000+00:00

Here is the output of some code I wrote to output hashes and arrays in a more colourful way:

The code is here if you're interested:

https://github.com/kbarber/pretty_data/blob/master/pretty_data.rb

Ruby YAML and sorting top-level keys

2012-02-10T16:57:00.000+00:00

I wanted to sort the top-level keys in a YAML hash in Ruby in a once-off way (that is, not sort all the time - just this once). I found two methods.

The first one is based on something I found on StackOverflow. So given a hash:

a = { 
  "a" => "b",
  "e" => "b",
  "c" => "b",
  "p" => "b",
  "z" => "b",
  "x" => "b",
  "d" => "b",
}

The function looks something like:

def sort_yaml1(obj)
  YAML::quick_emit(obj) do |out|
    out.map(obj.taguri, obj.to_yaml_style) do |map|
      obj.keys.sort.each do |k| 
        v = obj[k]
        map.add(k, v)
      end 
    end 
  end 
end

puts sort_yaml1(a)

The second methodology is very manual and not quite as elegant, but is probably less dependant on the YAML internals:

def sort_yaml2(obj)
  arr = obj.sort
  out = "---\n"
  arr.each do |element|
    entry = {element[0] => element[1]}
    out += entry.to_yaml.to_a[1..-1].join + "\n"
  end 
  out 
end

puts sort_yaml2(a)

Its a shame the SortedKeys option isn't supported out of the box in Ruby 1.8.7, but it seems the SortedKeys option isn't passed to the underlying syck driver at all.

A Basic Ceph Storage & KVM Virtualisation Tutorial

2012-02-05T21:54:00.001+00:00

So I had been meaning to give CEPH & KVM Virtualisation a whirl in the lab for quite some time now. Here I have provided for you all a set of command-by-command instructions I used for setting it up on a single host. The goal here is really just to get it to the 'working' stage for some basic functional experimentation.

Ideally you would want to set this up on multiple hosts with proper network separation so you can see how it performs with real-world attenuation.

Overview

For those who don't know it, CEPH is a distributed storage solution that allows you to scale horizontally with multiple machines/heads instead of the more traditional methodologies which use centralised heads with large amounts of storage attached to them.

The principle here is that you should be able to buy lots of inexpensive computers with a bunch of direct attached storage and just cluster them to achieve scalability. Also, without a central point of failure or performance bottleneck you should be able to scale beyond the limitations of our past storage architectures.

So CEPH like most distributed storage solutions, really has 3 main components:

Object Storage Device (ODS): this is where the blocks get stored. You would usually have lots of these.
Meta-Data Server (MDS): This where the metadata gets stored. You would have less of these. They are used for looking up where the blocks are stored and storing meta-data about files and blocks.
Monitor (MON): Cluster management, configuration and state. This component keeps track of the state of the clustering really.

Here is a basic diagram provided from the official site (so yes I stole it - I hope thats okay):

As you can see, ideally these components are meant to be ran on different sets of systems, with the OSD component being the most frequent. I'm just going to run them all on the same host for this demo, which is useful for a functional test, but not for a destructive or performance test.

By the way the OSD part can used different types of backends and filesystems for storage, but in this example I've chosen BTRFS.

So CEPH itself supports multiple different ways of mounting its storage, which makes it quite a flexible solution.

In this demo I'm going to concentrate only on the RBD and Ceph DFS mechanisms.

This installation was tested with:

ceph 0.4.0
qemu-kvm 1.0
libvirt 0.9.8
debian 7.0 (wheezy)

I'm using the bleeding edge versions of these components because CEPH is really in heavy development its better to see how the main-line of development works to get a clearer picture.

OS Preparation

It was tested with real hardware hosted by Hetzner in Germany. The box specs were roughly something like:

Quad-Core processor
16 GB of RAM
2 x 750 GB disks (no hardware raid)

To begin, I personally built a Debian 6.0 system (because thats all Hetzner offers you within its Robot tool) with a spare partition that I later use for the OSD/BTRFS volume. The layout was something like:

/dev/md0: /boot, 512MB
/dev/md1: LVM, 74 GB
/dev/md2: the rest of the disk

And in the LVM partition I defined the following logical volumes:

/: 50 GB (ext4)
swap: 20 GB

The device /dev/md2 I reserved for BTRFS. I believe a more optimal configuration is to not use an MD device but just use /dev/sda2 & /dev/sda3 and let BTRFS do the mirroring. I however have no data or performance statistics to prove this at the moment.

To get the system upgraded from Debian 6 to 7 is fairly straight-forward. First update the APT sources list.

/etc/apt/sources.list:

deb http://ftp.de.debian.org/debian/ wheezy main contrib non-free
deb http://ftp.de.debian.org/debian/ wheezy-proposed-updates main contrib non-free

Then run the following to then get the latest updates:

apt-get update
apt-get -y dist-upgrade

The kernel would have been upgraded so you should reboot at this point.

CEPH Installation

Install CEPH using the ceph package. This should pull in all the dependencies you need.

apt-get install ceph

Create some directories for the various CEPH components:

mkdir -p /srv/ceph/{osd,mon,mds}

I used a configuration file like this below. Obviously you will need to change the various parts to suit your environment. I've left out authentication in this demo for simplicity, although if you want to do real destructive and load-testing you should always include this.

/etc/ceph/ceph.conf:

[global]

log file = /var/log/ceph/$name.log

pid file = /var/run/ceph/$name.pid

[mon]

mon data = /srv/ceph/mon/$name

[mon.<your_short_hostname>]

host = <your_hostname>

mon addr = <your_ip_address>:6789

[mds]

[mds.<your_short_hostname>]

host = <your_hostname>

[osd]

osd data = /srv/ceph/osd/$name

osd journal = /srv/ceph/osd/$name/journal

osd journal size = 1000 ; journal size, in megabytes

[osd.0]

host = <your_hostname>

btrfs devs = /dev/md2

btrfs options = rw,noatime

Now for configuration, CEPH chooses to try and SSH into remote boxes and configure things. I believe this is nice for people who are just getting started, but I'm not sure if this is correct going forward if you already have your own Configuration Management tool like Puppet, Chef or CFEngine.

So to begin with, there is a command that will initialise your CEPH filesystems based on the configuration you have provided:

/sbin/mkcephfs -a -c /etc/ceph/ceph.conf --mkbtrfs --no-copy-conf

Starting the daemon was a little strange (see the -a switch?):

/etc/init.d/ceph -a start

So just to test its all working, lets mount the CEPH DFS volume onto the local system:

mount -t ceph <your_hostname>:/ /mnt

What you are looking at here is the CEPH object store mounted in /mnt. This is a shared object store - and you should be able to have multiple hosts mount it just like NFS. As mentioned before-hand however, this is not the only way of getting access to the CEPH storage cluster.

CEPH DFS & Directory Snapshots

So I just wanted to segway a little and talk about this neat feature. CEPH & the ceph based mount point above has the capability to do per-directory snapshots which could come in useful. The interface is quite simple as well.

making a snapshot:

mkdir /mnt/test

cd /mnt/test

touch a b c

mkdir .snap/my_snapshot

deleting a snapshot:

rmdir .snap/my_snapshot

finding a snapshot:

The .snap directory won't show up when you do a ls -la in the dir.

Simply assume its there and do something like:

ls -la .snap

... in the directory and the snapshots should show up under the names you created them with.

RADOS Block Device

So an alternative way of using your CEPH storage is by using RBD. The RBD interface gives you the capability to expose an object onto a remote system as a block device. Obviously this has the same caveats as any block device, so multiple hosts that mount the same device must ensure they use some sort of clustered file system such as OCFS2.

So first if its not already, load the rbd kernel module:

modprobe rbd

Using the 'rbd' command line tool, create an image (size is in megs):

rbd create mydisk --size 10000

You can list the current images if you want:

rbd list

Now to mount the actually device, you just have to tell the kernel first:

echo "<your_ip_address> name=admin rbd mydisk" > /sys/bus/rbd/add

It should create a device like /dev/rbd/rbd/mydisk. Lets now format it with a real filesystem and mount it:

mkfs -t ext4 /dev/rbd/rbd/mydisk

mkdir /srv/mydisk

mount -t ext4 /dev/rbd/rbd/mydisk /srv/mydisk

KVM/Qemu Support

QEMU (and libvirt for that matter) at some point merged in patched to allow you to specify an 'rbd' store as a backend to a QEMU virtual instance. I'm going to focus on using an Intel/KVM image for this tutorial.

So lets start by installing KVM & Qemu and the various other pieces we'll need:

apt-get install kvm libvirt-bin virtinst iptables-persistent

We want to probably create a pool for vm disks separate from the pre-existing ones. You can create as many of these as you need:

rados mkpool vm_disks

Now create a qemu image inside the pool. Notice we are just using 'qemu-image' to do this?

qemu-img create -f rbd rbd:vm_disks/box1_disk1 10G

Create yourself a bridge network by modifying the correct Debian configuration file.

/etc/network/interfaces:

auto virbr0

iface virbr0 inet static

bridge_ports none

address 192.168.128.1

netmask 255.255.255.0

network 192.168.128.0

broadcast 192.168.128.255

And now bring up the interface:

ifup --verbose virbr0

We'll need some firewall rules so that NAT works in this case. Obviously your network needs may vary here.

/etc/iptables/rules.v4:

*filter

:INPUT ACCEPT [0:0]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [0:0]

-A FORWARD -s 192.168.128.0/24 -m comment --comment "100 allow forwarding from internal" -j ACCEPT

-A FORWARD -d 192.168.128.0/24 -m comment --comment "100 allow forwarding to internal" -j ACCEPT

COMMIT

*nat

:PREROUTING ACCEPT [0:0]

:INPUT ACCEPT [0:0]

:OUTPUT ACCEPT [0:0]

:POSTROUTING ACCEPT [0:0]

-A POSTROUTING -s 192.168.128.0/24 -o eth0 -m comment --comment "500 outbound nat for internal" -j MASQUERADE

COMMIT

And restart iptables-persistent to load the rules:

service iptables-persistent restart

Turn on forwarding for IPv4:

echo 1 > /proc/sys/net/ipv4/ip_forward

Now that the network is done, we want to create a script to help us launch our VM instance.

First of all create a device definition file called disk.xml with the following contents. This allows us to work-around limitations in virt-install, as it doesn't yet support these extra options as command-line arguments.

/root/disk.xml:

</disk>

Now lets create our script.

/root/virt.sh:

#!/bin/bash

set -x

virt-install \

--name=box1 \

--ram=512 \

--vcpus=1 \

--location=http://ftp.de.debian.org/debian/dists/wheezy/main/installer-amd64/ \

--extra-args="console=ttyS0" \

--serial=pty \

--console=pty,target_type=serial \

--os-type=linux \

--os-variant=debiansqueeze \

--network=bridge=virbr0,model=virtio \

--graphics=none \

--virt-type=kvm \

--noautoconsole \

--nodisks

# This is because virt-install doesn't support passing rbd

# style disk settings yet.

# Attaching it quickly before system boot however seems to work

virsh attach-device box1 disk.xml --persistent

And finally we should be able to run it:

./virt.sh

Now attach to the console and go through the standard installation steps for the OS.

virsh console box1

Note: There is no DHCP or DNS server setup - for this test I just provided a static IP and used my own DNS servers.

As you go through the setup, the RBD disk we defined and created should be available like a normal disk as you would expect. After installation you shouldn't really notice any major functional difference.

Once installation is complete, you should be able to boot the system:

virsh start box 1

And then access the console:

# virsh console box1

Connected to domain box1

Escape character is ^]

Debian GNU/Linux 6.0 box1 ttyS0

box1 login:

And then your done.

Summary

So this is quite an interesting exercise and one worth doing, but the software is still very much early-release. They even admit this themselves.

I'm wary of performance and stability more then anything, something I can't test with just a single host - so if I ever get the time I'd really like to run this thing properly.

I had a brief look at the operations guide, and it seems the instructions for adding and removing a host to the OSD cluster looks not as automatic as I would like it. Ideally, you really want the kind of behaviour that ElasticSearch offers on this level so that adding and removing nodes is almost a brain-dead task. Having said that, adding a node seems easier then some of the storage systems/solutions I've seen about the place :-).

So regardless of my concerns - I think this kind of storage is definitely the future and I'm certainly cheering the CEPH team on for this one. The functionality was fun (and yes kind of exciting) to play with and I can see that the real-world possibilities of such a solution in the open-source arena are quite probable now.

Other things to try from here

Check out another alternative: Sheepdog, which also seems to be gaining ground but only on the QEMU storage front. Its a very specific solution as apposed to CEPH's generic storage solution.
Test CEPH integration with OpenNebula and OpenStack so you can see it within a full cloud provisioning case. This might require some custom scripts to support cloning RBD-stored base images etc. but should be interesting.
Test the S3 emulation provided by the RadosGW component.

References

Formal (yet incomplete) Documentation:

http://ceph.newdream.net/docs/latest/

Wiki:

http://ceph.newdream.net/wiki/Main_Page

Installation on Debian:

http://ceph.newdream.net/wiki/Installing_on_Debian

RBD:

http://ceph.newdream.net/wiki/Rbd

QEMU-RBD:

http://ceph.newdream.net/wiki/QEMU-RBD

Snapshots:

http://ceph.newdream.net/wiki/Snapshots

Social Aspects of Open Source Contribution

2012-01-23T19:06:00.001+00:00

Lately I've been helping out on a piece of software that has a reasonably wide circulation, and I must say the experience has been interesting. My role in that project is to not only think about wider issues and write code for them, but also to help out with community contribution.

While the technical side of this is interesting, a lot of time is spent on the social side - and I wanted to talk about my experiences, my deficiencies and various gripes from both contributors and "merge-masters" specifically around code contribution. I'm an absolutely crap social person (all my ex-girlfriends think I'm a bastard), and recent discussions with a friend/colleague made me think of writing this article and the struggles with being sociable in this arena in an effort to get myself and others thinking about it more.

Talking Terse

Although its hard not to, its unavoidable in this game that emotion becomes involved. I get a lot of commits that come through, and sometimes I'm not the best at appearing to be "nice". If you've seen me do this - its not even that I'm trying to be nasty - but my choice of language is short because I'm in a rush and I haven't taken the time to think about how people might interpret it.

I think its easy for an external committer to take personal pride in his submission - and so he should. My terseness or brevity isn't a reflection on this, in fact I think every community commit is a piece of awesomeness. But like everyone we have limited time and can't always display that emotion all the time. Like my wife says: "You are an insensitive guy after all".

When you have a good relationship with a committer that kind of social thing isn't necessary, but even then I find I need to sit up and just be a person again - not a code testing machine. I think the trick here is that we have to be "nice" no matter what. But more importantly, if you are nasty - say sorry :-).

Differences of Opinion

Sometimes you get into arguments as well differences of opinion and sometimes all out slinging matches (although I've had very little of these). I've found stepping back and trying to put myself in the submitters shoes is the best thing. I have once or twice sparked off emotionally and said 'no we'll never do that'. Only later to realise I was being an arse. The thing is, people are raising these issues because they feel its important to them - the fact that someone has submitted any amount of code shows it matters. So I try to always consider their perspective, but I'm not always successful.

Its easy to be myopic as well about an issue, and I've tried recently to purposely give up on arguments when I new the consequences of letting a change through aren't really going to affect the overall product anyway. I'm a terribly myopic individual at times, but dealing with other people in this case is starting to loosen that up.

Processes

My company has a semi-strict processes for code (although not as bad as others I've seen). Broken builds must be fixed quickly, and all commits if possible should have tests. All code must be peer reviewed, and must have a ticket. It can be painful - but in our defence we're asking these things because we have found over time this is the simplest path to quality. Yes - all processes can be argued and not every process fits every problem. But for us we find its the best path to success, and we (my org that is) generally all agree on this.

Not everyone gets this though, sometimes its because they are new. Sometimes they have their own process and this is just an annoying distraction. Sometimes they can't see the benefit for them in following the process.

So I try hard to help out. For example - writing tests for code is actually generally harder then writing the code itself, but it stops regression and increases the chances that the code was well thought through. Yes its a hassle, but we can't compromise since we want our product to be high quality and have as little technology debt going forward. I find showing people examples helps, so if I write tests I try to make them readable and reproducible.

For Git based workflow problems (which happen when committers are new) - I will actually spend more time helping them get through the process then if it took me to just write the code. A lot of people ask me why I bother. My attitude is that if I help people out, they contribute more. I want their contribution experience to be a nice one (despite sometimes me ruining it :-).

Some people bitch to me about the process. I hear them loud and clear, I hated the process myself when I first started. Now I need the process. I need to know when a ticket needs to be merged (so change the status). I need to have clear commit messages (so I can see what that change was all about after all). I need the ticket in the commit message (so I can go back and read the back-story behind the change). From my perspective all the various pending changes appear as a stream of tasks I must complete ... and without some semblance of order my brain explodes :-).

Having said that, I want the process to be easier for everyones sake. Always questioning the process if there is a perception it is wrong is the key to this I believe. As a contributor you should also be questioning but more importantly suggesting alternatives. Most projects aren't doing something because they are trying to cause pain, finding the reasoning behind a process and suggesting an alternative I'm sure for most project members would be more welcome then 'your process sucks'.

For the common good

Sometimes people want their change to go through, but don't want to think about how this affects the product as a whole. I always have to think about the product as a whole. Its my curse and well - my job.

I try as much as I can to get people to think about use-cases beyond their own. Sometimes this doesn't work and I probably come across as difficult. But while I'm being difficult to that person, its in reality being nice to everyone else trying to use it. Does that make sense? Its a bit existential I know :-).

I think the answer sometimes is to provide a plugin system where peoples ideas can be tested on an audience without making it into core. This is certainly a valuable test, as the numbers and the popularity of a solution then speak for themselves.

Anticipating problems

I've seen the effect of when a bad change comes through, then the follow up tickets where people bitch about it being broken. I hate that, so I'm pedantic at times. Why? Because I don't want those follow up tickets - not a single one. I want it right every single bloody time.

But this is not always realistic. Sometimes I have my doubts and let the fates decide, but when I know its a bad idea I dig my heals in. My biggest problem is I have trouble getting my point across as to why I think its a bad problem. Sometimes its just a feeling and hard to quantify. I think my biggest problem as a technical person is my ability to explain an issue via a text based format, and something I need to work on.

Great idea, bad timing

And finally there is the ticket that is awesome and cool, but we want to wait because it needs a big change somewhere else. I like these tickets, they confirm that our thinking is right. But as usual we need to consider the blocking change from all perspectives, not just from a 1-bug perspective. Its a pain, and its slow and excruciating and if I could I would love to work on big architecture changes all day long to make the minor functional changes as easy as pie.

But I think these things turn people off. Ultimately there is a perception that these big changes need some staff member or senior to execute it. This in part is true, its much easier to get what you want done when you work near the top of a project (either as part of the company sponsoring it or you are in the upper ranks). This most certainly is something I feel is a big problem, and needs more thought as to its resolution.

But the people are awesome

And this is so true. I've met many awesome individuals while doing this kind of work. Sometimes they are short-term relationships, and others I consider to be friends.

Whats awesome about this aspect of the community, is that your working with this people as if they were colleagues - but its all volunteer and without as much constraint. No bosses, no contracts, no business plan, no strict HR - just human beings getting along, writing code and working for the greater good. And this is what I think makes Open Source totally awesome :-).

Travis CI

2012-01-22T00:52:00.000+00:00

So previously I wrote about Heroku which is a cloud based PaaS service. On the same theme I want to try out another cloud service for development: Travis CI. This tool was suggested to me today by a community committer at my work.

Travis CI is a cloud based continuous integration tool that has deep integration with Github so fits perfectly with the recommended stack outlined in the Rails tutorial I mentioned in my last post.

So I wanted to try this tool out with some sample code, I decided to just try it on the tutorial code I had written last time: Jakovo.

So I followed the Getting Started instructions on the Travis website:

Logged into the website using Github
Authorised Travis-CI to access my public github projects
Went to my profile page and flicked the switch for Jakovo
Added a .travis.yml file to my project
Committed the git changes to my project, and push it to Github

At first it broke. The results are here (but here is a screenshot):

There needs to be a database created before the tests can succeed. So I jumped to the next part of the Travis CI documentation and read-on.

Turns out the solution is to just provide a setup step before you run the test, its recommended that if you have a lot of steps you write separate scripts for this - but I'm going to take the easy option and just add:

before_script: bundle exec rake db:migrate

To the definition file. Again, committing and pushing this to Github, I get back positive results for the next test. Fairly impressive really:

What impressed me about this, is just like Jenkins it displays the running results in real-time so you can see it do the build instead of waiting for a failure/success.

The Github integration is sublime and suits my preferred workflow perfectly. The ability for users to have their clones be tested before they submit a pull request to a main clone will surely be helpful as well (although I haven't looked into how to do this on a topic branch yet).

I guess the only gap I see is the ability to have code tested on multiple platforms, which in my business is important. However, for a lot of projects this won't matter.

The ability to choose different Ruby revisions, Gemfiles and Environments creates an interesting matrix capability which parallels the capability in Jenkins that I find quite useful.

All in all - this looks like a fascinating and useful tool.

Rails 3.1 and Heroku

2012-01-16T01:02:00.004+00:00

So I finally found some quiet moments to try out the latest Rails 3.1 incarnation. I had tried Rails 2 many years ago and found it quite enjoyable but figured I'd give Rails 3.x a try since I've been doing more development along these lines lately.

First of all I started with the Ruby on Rails Tutorial by Michael Hartl. This is a fine starting guide for most people and includes modern techniques including:

Using github to manage your code
Using Heroku to upload your code to a cloud host

I really wanted to try out the Heroku capabilities to see how far the state of the PaaS for Ruby had evolved as I had tried Google App Engine many years ago with both Java and Python and found it very inflexible.

So I created a little piece of code and called it Jakovo loosely based on the tutorial code just to try the Rails + Heroku combination. Its meant to be a task list, as I find task lists are always great for tutorials of this kind.

At the moment I've only gotten as far as creating a basic model with relationship between a user and task, and spent no time on making it look pretty. If I find more time I'll continue the work.

Extra Heroku Steps

When uploading to Heroku I found a couple of problems that I had to work-around that weren't clear from the tutorial, so I'll outline these first.

First of all, Heroku uses Postgresql as a database, so my code I added the following to Gemfile (for bundler) to handle this:

group :production do

gem "pg"

end

For asset support, I added therubyracer gem to my Gemfile. This uses the Chrome V8 javascript runtime.

group :assets do

# A javascript runtime used during asset compilation in heroku

gem 'therubyracer'

gem 'sass-rails', '~> 3.1.5'

gem 'coffee-rails', '~> 3.1.1'

gem 'uglifier', '>= 1.0.3'

end

Then I compiled the assets and checked them into git:

# bundle exec rake assets:precompile

# git add public/assets

# git commit -a

This is because you can't modify the filesystem remotely using the heroku interface.

To prepare the database I ran the following heroku command:

# heroku run bundle exec rake db:migrate

Which works the same as it does locally.

Deploying Using Heroku

The Heroku gem was a pleasure to use. After creating a Heroku account on heroku.com I added my keys:

# heroku keys:add

Then created an application:

# heroku create

This automatically added the necessary configuration to my local .git/config file that allowed me to deploy:

# git push heroku master

The deployment then walks through on the console all the steps it is performing before telling you it is done. I then navigated to the website:

# heroku open

Troubleshooting Heroku

To troubleshoot, Heroku includes some nice tools for this. Here are just a few I found handy during my attempts to dig into problems:

# heroku logs

This displays the remote logs in a nice colourful format.

# heroku pg:psql

Gives you a console to the remote Postgresql database, great for analysing your data.

# heroku apps:info

Shows detailed information on applications.

Time Capsule 4th Generation

2012-01-15T18:11:00.000+00:00

So recently I obtained a Time Capsule (4th generation) for work for backing up my Mac laptop. Its not a bad little device. This one has 2 TB of internal storage, plus USB support for plugging in additional storage.

While most of my work doesn't require backups as I use Git for all my development, I do have the odd file that I wouldn't want to lose - and in the case of a catastrophic failure or during a full restore case this becomes quite important.

Time Machine

Time machine is a lovely backup solution I must admit, more because it mostly follows the 'just works' principles I would expect from a Mac. Getting the Time Machine backups setup with my Time Capsule took very little effort and I've only had a couple of problems with it.

After setup for the initial backup, I did this over Ethernet so it would occur in reasonable time. Ongoing backups I do over wireless.

The Time Machine interface is not so bad, the idea is that on the right of the interface you get a timeline that you can roll back to. You are presented primarily with a Finder GUI so you can analyse files on a particular time line easily and restore them if necessary.

While I think this is nice I'd like to explore any command line options that are available as well, but this is more my requirement as I live in the CLI quite a lot.

Wireless Support

The unit also acts as an Airport wireless access point. I thought this would be useful due to its 802.11n support, plus when I'm travelling I can hook it into hotel room Ethernet connections and it would provide wireless when its not available - or even better wireless when the hotel wireless has a lot of contention.

From a range perspective, the range for my phone is actually worse compared to when I use my Thompson Router V2 (the thing that O2 gives you in the UK) although this could be an Android problem as I haven't had another phone to compare it with. For my laptop the range is reasonably acceptable. It supports WPAv2 Personal which is enough for my requirements and can act as a wireless extender, but only with supported routers or other Airport compatible access points.

Problem 1 - Sitting in Preparing backup ...

So I had an issue with the backups sitting in "Preparing backup ..." forever (well for a few days anyway). I cleared this issue first by switching off backups in Time Machine preferences:

Then clearing the time machine settings:

rm /Library/Preferences/com.apple.TimeMachine.plist

Opening up the Time Machine settings again I chose my disk and re-enabled backups again. This seemed to clear the issue without backup loss.

I've only had to do this once, but its fairly repeatable without a major problem (well at least none that I've identified yet).

Problem 2 - Ethernet Port not working

I plugged in another laptop to the Ethernet port of the Time Capsule and the port wouldn't work. I fixed this by rebooting the unit. Such a fundamental problem is pretty disappointing I must admit but it wasn't a complete blocker.

Things I'd like to see improved

The unit is fairly mac-centric. A web interface for configuration would be awesome, but this doesn't affect me much as I have a Mac to manage the unit.

There is no quota support for managing how much of the in-built disk you want to use for backups versus general storage. The work-around is to dedicate an external USB disk for your own storage requirements but this comes at an extra cost.

Also - there is no in-built encryption for the internal disk for the purpose of backups. Two work-arounds I have seen for this:

Use an external storage unit that is encrypted
Find a different encryption solution that serves itself up as a file, and back that up. Store your important data using this solution.

There are lots of discussions about this on the net, and people saying that this 'isn't an important feature' but I think this is from people who haven't had to live with strict corporate policies and higher encryption requirements for storage.

Summary

I think the solution isn't a bad one if you can afford it. It certainly was an easy solution without any major hiccups so totally recommend it for non-power users. I think an alternative would be to use just an Airport unit with external HDD via USB which would save some costs (and probably be more flexible) but as a complete solution its not that bad.

Using rvmrc files for project based gemsets

2012-01-01T21:45:00.001+00:00

So my good friend Adrien Thebo recommended I check out rvmrc files as a technique for project specific RVM settings. I wanted to see if I could produce gemsets specific to each project without the hassle of switching it myself all the time I change between projects.

So I started with a basic example. Lets start at the root of my dev area:

cd ~/Development

And I create a project called monstermash:

mkdir monstermash
cd monstermash

Now I need to create a .rvmrc file for this, but I specify a brand new gemset:

# rvm --rvmrc --create use ruby-1.8.7@monstermash
Using /Users/ken/.rvm/gems/ruby-1.8.7-p357 with gemset monstermash

Now there are two patterns already prepared for preserving what gem to use: bundler and gemset exports. Both reasonable choices but I'm going to use gemset export/import files for now since it needs very little setup. Bundler is a better choice probably when not everyone uses RVM.

So let me build up the monstermash gemset manually now by just installing particular gems:

gem install rspec --version 2.6.0
gem install json --version 1.6.2
gem install mocha --version 0.9.12
gem install jgrep --version 1.3.1

And then finally lets export them:

# rvm gemset export
Exporting current environments gemset to .gems

This saves the gemset in the local .gems file. Now I modify the created project .rvmrc in ~/Development/monstermash/.rvmrc and comment out the gemset area:

filename=".gems"
if [[ -s "$filename" ]]
then
rvm gemset import "$filename" | grep -v already | grep -v listed | grep -v complete | sed '/^$/d'
fi

Okay. To test, I cd back into my development area and specifically delete the gemset I just created:

# cd ~/Development

# rvm gemset delete monstermash
WARN: Are you SURE you wish to remove the entire gemset directory 'monstermash' (/Users/ken/.rvm/gems/ruby-1.8.7-p357@monstermash)?
(anything other than 'yes' will cancel) > yes

Now lets test if the rvmrc does the right thing by changing into the monstermash directory. As you can see I needed to 'trust' the file first, but this is only needed the first time you try to use it and every time it changes.

# cd monstermash
<snip>Big notice about trusting the file</snip>
Do you wish to trust this .rvmrc file? (/Users/ken/Development/monstermash/.rvmrc)
y[es], n[o], v[iew], c[ancel]> y
installing diff-lcs 1.1.3...
diff-lcs 1.1.3 installed.
installing jgrep 1.3.1...
jgrep 1.3.1 installed.
installing json 1.6.2...
json 1.6.2 installed.
installing rspec 2.6.0...
rspec 2.6.0 installed.
Using: /Users/ken/.rvm/gems/ruby-1.8.7-p357@monstermash

And there we go. Not only did it create the monstermash gemset, but it populated it from my projects .gems file:

# gem list

*** LOCAL GEMS ***

bundler (1.0.21)
diff-lcs (1.1.3)
jgrep (1.3.1)
json (1.6.4, 1.6.2)
rspec (2.6.0)
rspec-core (2.6.4)
rspec-expectations (2.6.0)
rspec-mocks (2.6.0)

Now whenever I cd into the directory, it does the right thing:

# cd ~/Development/monstermash
Using: /Users/ken/.rvm/gems/ruby-1.8.7-p357@monstermash

So in theory you could check-in the .rvmrc and .gems files into your VCS system so everyone could benefit from this. Of course, you'd want to make sure your project team was happy with this workflow as it only really benefits RVM uses.

Zsh prompt and rvm current

2011-12-31T17:56:00.003+00:00

So I finally got around to doing something with my boring prompt in zsh. I added this to my .zshrc:

autoload -U colors && colors
export RPROMPT="%{$fg[green]%}$(rvm current)%{$reset_color%}"

And then I added the export to my precmd so it runs it each time:

precmd () {
export RPROMPT="%{$fg[green]%}$(rvm current)%{$reset_color%}"
}

Now I get a little green prompt in the far right telling me what version of Ruby (and rvm gemset) I'm using.

I guess having the prompt change when I switch rvm itself is probably more efficient, I'll look into that another day.

Backwards Compatible rbconfig

2011-12-30T10:19:00.000+00:00

My friend James Turnbull came up with this little gem the other day:

CONFIG = Object.const_get(defined?(RbConfig) ? :RbConfig : :Config)::CONFIG

The idea being that since Config is now deprecated in Ruby 1.9.3 and yet RbConfig is not available pre-1.8.5. This work-around covers both angels.

Maybe its the old Perl hacker in me, but there is something about it I really like.

Esper and Structured Content Struggles

2011-12-28T18:04:00.002+00:00

Working with structured data in Esper means you need to deal with a varying amount of issues including special character types and nesting. As Esper is Java all this data must be represented as a java.util.Map type before representation which adds some special cases.

I've compiled a number of problems and solutions that I have experience directly. The code and data examples will be using JRuby since this is what I develop with. For more details getting this working see my article provided earlier.

Doing a select on a stream that contains special characters

Sometimes structured data can contain special characters that need escaping. This is especially a problem in the label or key part of the data. Lets take the character '@'. If you need to escape, you must use the back-tick character.

Input Data:

{"itemName"=>"test","price"=>100, "@foo" => { "bar" => "baz" }}

Select statement:

select `@foo`.bar from OrderEvent

This would return:

{"@foo.bar"=>"baz"}

Without the escape, you would get the error:

NativeException: com.espertech.esper.client.EPStatementSyntaxException: Incorrect syntax near '@' at line 1 column 7, please check the select clause [select @foo.bar from OrderEvent]

Returning a column with special characters

The converse of the last section is the returning of special characters in column names. Lets say we want to return the above example but with a column name of '@selected'. You'll need to escape it:

select `@foo`.bar as `@selected` from OrderEvent

The problem with this, is that now your column will still contain the back-ticks. I'm not sure if this is meant to be a bug or not, but I wrote a small function to clean this up:

def clean_escapes(old_hash)
new_hash = {}
old_hash.each do |k,v|
new = k.gsub('`','')
new_hash[new] = v
end
new_hash
end # def clean_escapes

This will strip out back-ticks aggressively, but you may find you want something more selective that only does this on the first and last characters.

Selecting a property that is dynamic

Dynamic properties are allowed in Esper, however they must be dealt with during a select properly or else you will receive an error if you hit a block of data that does not contain the property.

Lets say we have 3 pieces of input data:

{"itemName"=>"test","price"=>100, "@foo" => { "bar" => "baz" }}
{"itemName"=>"test","price"=>200}
{"itemName"=>"test","price"=>300}

You can see that the property '@foo' only exists in one case. This is a dynamic property.

So first of all declare the event type. For some reason, without defining @foo earlier, you will get an error (this only seems to happen at the top level with properties that require escaping):

order_event_type = {
"itemName" => "string",
"price" => "double",
"@foo" => {},
}
ep_config.addEventType("OrderEvent", order_event_type)

The select statement to grab @foo.bar whenever it exists would be:

select `@foo`.bar? from OrderEvent

This then returns:

{"@foo.bar?"=>"baz"}
{"@foo.bar?"=>nil}
{"@foo.bar?"=>nil}

As you can see, it returns only a match once.

Returning original and new data

Sometimes you want to preserve the original data as well as add new properties. Lets say you have the following set of data:

{"id" => 4, "itemName"=>"test", "price"=>100}

{"id" => 5, "itemName"=>"test", "price"=>100}

{"id" => 6, "itemName"=>"test", "price"=>300}

{"id" => 7, "itemName"=>"test", "price"=>300}

{"id" => 8, "itemName"=>"foo", "price"=>300}

{"id" => 9, "itemName"=>"test", "price"=>500}

And you want to provide a brand new property called 'doublePrice'. You normally would select the information this way:

select price * 2 as doublePrice from OrderEvent

Which produces the following output:

{"doublePrice"=>200}
{"doublePrice"=>200}
{"doublePrice"=>600}
{"doublePrice"=>600}
{"doublePrice"=>600}
{"doublePrice"=>1000}

Now if instead you want to return the original and new properties, the SQL query can be modified as such:

select *, price * 2 as doublePrice from OrderEvent

The problem however, is that your UpdateListener class now receives a new class type:

com.espertech.esper.event.WrapperEventBean

This wrapper contains separate underlying returned events for the origin and new properties. So using a small change to my handler code, I merge this whenever it occurs:

if event.class == com.espertech.esper.event.WrapperEventBean then

  c = event.getUnderlying.first.merge(event.getUnderlying.second) 

  puts c.inspect

else

  puts event.getUnderlying.inspect

end

Now I get combined output:

{"id"=>4, "itemName"=>"test", "price"=>100, "doublePrice"=>200}

{"id"=>5, "itemName"=>"test", "price"=>100, "doublePrice"=>200}

{"id"=>6, "itemName"=>"test", "price"=>300, "doublePrice"=>600}

{"id"=>7, "itemName"=>"test", "price"=>300, "doublePrice"=>600}

{"id"=>8, "itemName"=>"foo", "price"=>300, "doublePrice"=>600}

{"id"=>9, "itemName"=>"test", "price"=>500, "doublePrice"=>1000}

Now this won't work for nested data as Ruby's Hash merge will only work on the shallow level and override keys on the left hand side of the merge operator. If you want to merge structured data, you will need a far more intelligent way of merging data in that case.

Esper Aggregation Examples

2011-12-28T17:03:00.001+00:00

I've been playing with more Esper stuff. Here is some quick useful aggregation examples I've found. Again - if your looking for the context of these posts check my 'esper' label.

Aggregating Events into a Single Event

If you have multiple events, and wish to aggregate them into a single event, you can use aggregation functions with a window. The window provides a time frame in which to evaluate, as aggregation is only really that useful within a span of time, or span of events (there are however exceptions).

Given the following data:

{"itemName"=>"test","price"=>100}

{"itemName"=>"test","price"=>200}

{"itemName"=>"test","price"=>300}

We want to return every 5 seconds the sum of all these pieces. This can be done with a query using a time_batch window, and using the aggregate 'sum' function:

select sum(price) from OrderEvent.win:time_batch(5 sec)

Which will return the following if the input data is pushed into the stream in the first 5 seconds:

{"sum(price)"=>600}

You may notice however, that it will emit events every 5 seconds like this afterwards:

{"sum(price)"=>nil}

You can correct this by changing the select clause to match on the aggregate function. This isn't done with a 'where' clause but instead with aggregates you can use the 'having' clause:

select sum(price) from OrderEvent.win:time_batch(5 sec) having sum(price) >= 0

Now it will only emit an event if the sum of the price is equal to or over 0 and nil events will no longer be returned.

Other examples of aggregate functions are provided in the documentation.

Selecting Distinct Events from a Stream

Often you want to be able to pick out distinct events from a series of events defined within a window (either a time window or number of events usually).

Lets say you have a series of events:

{"id" => 4, "itemName"=>"test", "price"=>100}
{"id" => 5, "itemName"=>"test", "price"=>100}
{"id" => 6, "itemName"=>"test", "price"=>300}
{"id" => 7, "itemName"=>"test", "price"=>300}
{"id" => 8, "itemName"=>"foo", "price"=>300}
{"id" => 9, "itemName"=>"test", "price"=>500}

You can use the distinct keyword during the select to pluck out only the distinct events:

select distinct itemName, price from OrderEvent.win:time_batch(1 sec)

Which would return:

{"itemName"=>"test", "price"=>100}
{"itemName"=>"test", "price"=>300}
{"itemName"=>"foo", "price"=>300}
{"itemName"=>"test", "price"=>500}

OS package search facilities

2011-12-24T16:11:00.000+00:00

So I have compiled here a list here of searchable facilities for different OS and distributions. I often need to look this information up myself, thought it might be handy for others.

Meta Search Engines

Linux Packages Search: http://pkgs.org/

Debian Family

Debian: http://www.debian.org/distrib/packages

Ubuntu: http://packages.ubuntu.com/

Nexenta: http://archive.nexenta.org/

Redhat Family

Fedora/EPEL: https://admin.fedoraproject.org/updates/

SuSE

SLES: http://support.novell.com/products/server/supported_packages/
openSUSE Build Service: http://software.opensuse.org/search

BSD Family

OpenBSD Ports: http://openports.se/
Mac Ports: http://www.macports.org/ports.php
FreeBSD Ports: http://www.freebsd.org/ports/searching.html
NetBSD Ports: http://www.uk.netbsd.org/ports/

Solaris Family

Solaris - OpenCSW: http://www.opencsw.org/search/
Solaris - Blastwave: http://www.blastwave.org/jir/packages.fam
OpenIndiana: http://wiki.openindiana.org/oi/Package+Repositories

Others

Gentoo: http://packages.gentoo.org/
Arch Linux: http://www.archlinux.org/packages/

Finding the tag that contains a commit in git

2011-12-24T14:30:00.002+00:00

I was trying to figure out the exact version when RbConfig was introduced to Ruby 1.8.

So I checked out Ruby from http://github.com/ruby/ruby and then changed to the Ruby 1.8 branch:

% git checkout ruby_1_8

I found the commit in question with a combination of git log and git blame:

% git show c24a0d897e97c3bb8cc32effd6ca5e9bd9de58c1

Then I used git describe to found the containing tag:

% git describe --contains c24a0d897e97c3bb8cc32effd6ca5e9bd9de58c1
v1_8_5_preview1~259

Version 1.8.5. Too easy.

My Initial Assessment of Android 4.0

2011-12-23T11:36:00.000+00:00

I'm basing a lot of the comparative judgements from my memory of 2.3 since I don't have it around to compare with. Also most of these items are based on my usage of my phone - other people use their phone in different ways so I won't be surprised if opinions differ.

Here is what I like:

It feels faster. It doesn't get pauses when I scroll around. The initial splash screen for example doesn't differ and refreshes itself without my brain noticing any fault.
The calendar widget is resizable, scrollable and covers multiple days.
Widget resizing in general is great.
The alarm now has an adjustable snooze time.
Favourites bar at the bottom is quite nice. When I first saw this I was apprehensive but now I actually find it quite handy.
Talk seems to finally have support for switching between my 2 google apps accounts.
Adding a new app adds a shortcut, which is actually turning out to be a nice convenience for me.
Viewing all the apps is now much nicer as the screens a flippable, not a huge mess of scrolling.
GMail App is much nicer and feels easier to navigate. Switching between accounts is more accessible.
The data usage component in settings is quite handy. I had a third part app for this but now its integrated with extra features. Seeing usage per application is quite handy.

Things I haven't been so impressed with:

The voice integration doesn't work for me. The accuracy rate is far too low to be interesting, and its not obvious how to correct this. At least for now. This isn't a surprise really, no one has really gotten voice working that well - although I do remember Dragon Dictate getting it reasonably accurate at the cost of CPU and $$.
The bookmark widget is not so nice to look at and use and I can't refresh it manually I think.

Things I want to see:

Google Docs synchronisation native to the Google Docs app would be nice. Maybe there is a setting - I know there is a cache, but I want it to work offline.

I'm tempted to get a tab soon with Android, primarily because now I think they seemed to have solved the scheduling/performance issues I was seeing on the old Android software. I'll have to find one with Android 4.0 to give it a try again.

Esper and JRuby

2011-12-23T01:21:00.001+00:00

Based on some advice from RI Pienaar I started to experiment a little with using the Esper complex event processing (CEP) engine in JRuby.

So I've got some sample code which does a basic match if anyone is interested:

https://github.com/kbarber/esper_testing

It requires the Esper jars to be placed in lib/ to work and has only been tested with Ruby 1.6.5 & Esper 4.4.0.

In the basic example I have the script 'basic_match.rb' I ran a series of data (3 sample orders with different prices in this case):

{"itemName"=>"test","price"=>100}
{"itemName"=>"test","price"=>200}
{"itemName"=>"test","price"=>300}

So the first query I ran against it was a basic averaging EPL query:

select avg(price) from OrderEvent

This simply averages the price over time. The output is:

% ./basic_match.rb
Dec 23, 2011 10:49:21 AM com.espertech.esper.core.service.EPServiceProviderImpl doInitialize
INFO: Initializing engine URI 'default' version 4.4.0
matched:
- {"avg(price)"=>100.0}
matched:
- {"avg(price)"=>150.0}
matched:
- {"avg(price)"=>200.0}

The second query of interest was using a clause:

select * from OrderEvent where cast(price,double) > 150

Where I'm matching all data where the price is over 150. You'll notice I had to case the price to a double to make it work:

% ./basic_match.rb
Dec 23, 2011 10:51:27 AM com.espertech.esper.core.service.EPServiceProviderImpl doInitialize
INFO: Initializing engine URI 'default' version 4.4.0
unmatched:
- {"itemName"=>"test", "price"=>100}
matched:
- {"itemName"=>"test", "price"=>200}
matched:
- {"itemName"=>"test", "price"=>300}

As you can see here, I have handling for matched and unmatched data.

Breaking up git-pull

2011-12-22T11:30:00.004+00:00

Up until now, whenever I wanted to clean up mistakes in a branch on my local repo I had been executing the following:

% git checkout master
% git pull origin +master

Which has forced the update from my remote origin branch, wiping anything locally and reverting me back to where I wanted to be.

So since git-pull is really just a convenience wrapper around git fetch & merge, I was always curious about the real commands were behind the git pull, and finally figured it out:

% git fetch -u origin +master

The key pieces meaning:

+ - force a fetch regardless of whether the branch is a descendant of the current branch. This is required because your trying to rewrite history.
-u - update the current HEAD. Without this git will refuse to do the operation.