All About the Code

The Clash of the Classnames

2022-10-02T00:00:00+02:00

I recently came across a very sneaky bug in one of our projects using custom web components at work and I thought I’d document it for future google searches. The structure of the code is this: We have a shared base class called ReactiveElement which is, as the name implies, a class handling reacting to property and attribute changes, via several different callbacks abstracting some of the functionality of custom web components. The project in questions has a lot of components extending this class, specifying which properties and attributes it wants to have callbacks fired for. This is the very core of several projects and it works well.

We also have a class called VirtualList that does caching, scrolling and handling large lists of items with some optimizations. We both use this VirtualList stand-alone and we have some specific list classes extending VirtualList.

And update in the ReactiveElement class broke this but only in a very specific way and only for certain use cases, namely the components extending VirtualList. And the update was in ReactiveElements method static get observedAttributes() which the browser calls on a class when it is registered as a custom element via the customeElementRegistry.define() method, to avoid it running twice for the same class, due to some server rendering functionality. But I digress.

Time for some code:

class VirtualListElement extends ReactiveElement {
  static get observedAttributes() {
    return {
      ...super.reactiveAttributes,
      listItems,
    }
  }
  ...
}

class SpecificListElement extends VirtualListElement {
    static get observedAttributes() {
    return {
      ...super.reactiveAttributes,
      shared,
    }
  }
}

Expected behaviour is that the class SpecificListElement should have both the property sharedand listItems from VirtualListElement when it’s created. But it only ends up with the property listItems. What gives?

Turns out the fix in ReactiveElement static get observedAttributes() now includes this code

static get observedAttributes() {
    if (this._observedAttributes === undefined) {
    ...
    }
}

which means that if the method has been called once for a class and its static this._observedAttributes has been set, it won’t be set again. Remember this is a static method and this in this context is the class itself, not an instance. This fact, combined with that VirtualListElement was both used standalone and as superclass, creates trouble. Deep in the VirtualList code, was this code:

customElementRegistry.define('virtual-list', VirtualListElement);

So when that code was run first, the VirtualListElement static get observedAttributes() is called first, that class gets its attributes set first. When SpecificListElement comes along, the browser will call its observedAttributes method (in reality on the super class ReactiveElement), but as the _observedAttributes_ property doesn’t exists on the class itself, it will look up the inheritance chain, find it on VirtualListElement and then stop. The properties specified in SpecificListElement will be ignored.

The fix is quite simple. Instead of

customElementRegistry.define('virtual-list', VirtualListElement);

We do

customElementRegistry.define('virtual-list', class extends VirtualListElement {});

to register the element with a anonymous class expression instead of a named class. You can also create any other named class extending the class used both as stand-alone element and superclass if you wish. A simple fix for a very confusing bug.

The Effects of Automation

2019-08-02T00:00:00+02:00

I want to reflect on the effects that automation can have on your daily life as a developer, after I came to think of the shift I have been through the last couple of years when it comes to deployments and habits and automation surrounding deployments. I dawned on me the other day, that deployments are now completely a non-event for me, and has been so for quite some time. I just don’t think about it anymore. In stark contrast to the extremely tired meme #nodeployfriday circulating on twitter. How did that happen?

Some context: I work in a team developing and maintaining about 20ish web applications. The team is small and fairly autonomous, at least when it comes to technical matters. My efforts into automating deployments and builds started around 2015-2016 at my previous employer (very similar in team size and technical stack) and the basis was a kind of semi-automated deploys but with few safeguards and the feelings involved in deployments where mixed. Deployments were talked about a lot, coordinated with several people and was a medium sized deal. Not huge, but definitely not small either. And the rule was don’t deploy on fridays (which is huge culture smell, maybe worth a whole post in itself). And deploys had to monitored.

The end goal was of course to be able to cut down on lead times for putting code into production, with side goals of making it less of a pain to deploy, fewer failed deployments and fewer nervous faces when doing deployments.

Steps to safer deployments

There are several steps you can take to make your deployments more automated and safer, depending on the context you are in. Adding more unit and integration tests to the code, considering using feature toggles for new functionality, thinking carefully about the way the team communicates about the code (pair programming, code reviews, mob programming etc), using smoketests and possibly integration tests and performance tests before advancing the deploy from one stage to another and so on. Also, trying to deploy smaller chunks at a time contributes to safer deployments and can give you positive spiral effects. Deploying often (as you do with small changesets) is painful when deployment is a manual process. This leads to automation of the deployment process, which makes it easier to deploy, so you do it more often etc. etc.

In our case we made use of integrated healthchecks in the applications (checked by the orchestration platform) and smoketests built into the deployment pipelines. The effects are that 1) the new version does not start when healthcheck fails (first gate) and if smoketests fail (second gate) deployments halts and is rolled back. If this is enough for you, depends on the context and the apps involved. First we automated the build of the apps and not the deployment. The deployment was manual, with only healthchecks when doing deployment. This made deploys much less of a worry, but still very much a topic of conversation. It took quite some time before everybody was comfortable with deploying on fridays (rule now was that you could deploy anytime, but be prepared to clean up afterwards if something went wrong). Gradually, deployments were not talked about anymore, but the decision to deploy was still a conscious one. The habit settled nicely, but I felt that we where missing out on some benefits in terms of automation.

Going the Distance

This was the case when I switched employers and got the opportunity to start anew with more experience. My goal was to be able to deploy apps (at least some) directly to production on push to master. The apps in question here varied a lot in terms of importance, quality and scope. The discussion now became: “How much, and what, do we need to do in order to make this app deploy directly to production?”. That is profoundly different conversation to have, rather than “how often should we deploy” or “can we deploy on friday?”. A related question that can crop up in cases like this, is “how do we communicate changelogs to sales/marketing/users/whoever?”. This is not a topic you should skip! It’s important to get this right, but it’s also worth asking some questions about who uses the information in the changelogs and for what? But, automating the changelog is also possible via commits or other means.

So, automating deploys to production: how to achieve it? First try to determine if this is at all wanted in your context. Web development can more or less always achieve this, but there are a lot of other types of software where the work involved may not match the perceived benefit. This is an important question. But let’s say you want to go that way, here are some tips:

Take small steps
Make build and deploy errors discoverable (build status to dashboard, slack etc)
Make errors clear and understandable
make the log clear to read (see previous point)
Use version control for your build pipelines and infrastructure
add tests to important parts first
add health checks
add smoketests
make deployments visible in dashboard with application metrics

Start with auto-deploy to test environment, build out unit tests, integration tests and other tests as needed. Gain confidence in the application and deploymentes. Measure how often builds and deploys to that env fails, if possible display it on a graph that everybody can see. Start deploying automatically to staging environment og pre-production depending on what you have. Gain more confidence - rince and repeat. Gain confidence also in the build and deployment infrastructure itself. If it fails you, improve it.

This confidence builds slowly. Let it accrue over time, don’t force it. Especially let team members not interested or unexperienced in the tao of deploys take part, improve and use it. This is important, since when deploys are automatic, nobody (and everybody) will have to take action when something goes wrong.

As deployments are made increasinly more automatic, they just fade from your mind and if you’re not paying attention to this you might not realize it. The chatter about deploys disappear from the workplace and things just happen. The point of being able to see the deploys marked on metric graphs (see my post on deploy overlays) is important when deploys are automatic, to be able to catch when a deploy introduces some weird behaviour in an application.

For me, the end result of this is a more relaxed work situation where changes flow from development to production. No deployment anxiety, no periods where deployment isn’t allowed and we can spend our time focusing on other things rather than talk about deployments. I like it this way.

Latest is such a bad idea

2018-05-29T00:00:00+02:00

For the second time I have found myself pulling what little hair I have left out because of people falling for the trap that is using “latest” as version number in a dependency, on the web specifically. This is such a bad idea, but people keep falling into the pit of using it. I can understand that, because it is very alluring and simple. “Just use latest, and you’re always up-to-date. No need to keep track of releases and just code and be happy.” Understandable reasons, but the downsides are huge.

Most of these points refer to using an asset via an url in the markup of your site (maybe developed by another in-house team, or someone else entirely), but they apply to lesser extent to build-time dependencies too.

Low TTL

Using latest as the version number almost always require low cache times for the asset. After all, if you cache it for a long time, you would not get the benefits of latest and get the newest version. Low cache time hurts your google-fu and pagerank and contributes to making your site slower for users. Don’t do that.

Undetected dependency changes

This is the flipside of the “feature” of “latest” as it updates your dependencies without you knowing. Great feature, right? Yeah, when all goes well it is. When someone botches the build and pushes a bad version you get blindsided with a failing site, with no warning. This happened to us recently. The core team released a botched build and it hit our site 10 minutes before the whole team was leaving for summer party. 1 hour later an everyone would be well into their beers. All because a guy on the team thought using “latest” was the simplest way forward. It was for a time, but it caught up to us.

Don’t botch the build, you say? Well, good luck in never having human errors occur.

Which version are we actually running?

Which version is the latest? Was the last upgrade a patch, a minor or a major? Does the release warrant changes to our site because of non-backward compatible changes? How do you find out, without having some sort of relationship with the version number? Ye gods the trouble that can come from this.

Reproducible builds

When talking build-time dependencies, the concept of reproducible builds is important in devops, to be able to recreate a build later in time, from a git version. This is hard to get 100%, but simple steps will take you a long way towards it. This absolutely requires having specific version numbers in dependency management systems. Even version ranges, which Node users love, makes this near impossible. What version did “latest” resolve to at build time? How would you even begin to find out what the diff was between the two builds where “latest” resolved to two different versions? Using “latest” in e.g. node dependencies will create bugs for you, it’s just a matter of time and how bad the bug will be. This goes even for dev dependencies, even though the repercussions are smaller. Gemfile.lock and package-lock.json does take some steps to avoid falling into this trap, and when you commit those files to version control, you can reproduce the builds back in time. But even so, I would argue that discreet version numbers are easier to manage and above all read and understand. It is right there, without having to check other files.

Using dynamic versions like latest, in the early stages of development is easy and should be fine, but you run the risk of forgetting to fix them later.

The Kubernetes Wars: Day 34

2016-12-25T00:00:00+01:00

Memory? Who needs it?

As more java apps where being moved whe started out by duplicating the memory settings for the old app, e.g. -Xmx 1024mb on the command line for the JVM. And then setting the memory limit for the container to 2-300 MB more than that. This works fine. Sometimes. For a while.

But then someone notices a weird high restart count on one of the apps. Not so much that the alarms go off, but maybe a couple of times a day or a handful a week. So someone slaps some more memory on it, but then stops to think «This container eats 300 megs more than the Xmx setting of the JVM in the container?» That’s a little strange isn’t it? What is actually going on here?

If you ask linux what memory the process is using, it will answer in (usually) RSS (Resident Set Size), which for java roughly translates to Heap Size + Metaspace + offheap size. Here, Offheap size consists of thread stacks, direct buffers and mapped files (libraries and jars), plus the jvm code itself. MetaSpace contains class instances in the code that runs. This all adds up.

The big eater in the family is the threads which as a default has 1M stack size. Some of our apps has a lot to do and can get between 500-1000 threads per instance in peak traffic. That’s a gig of memory right there.

And the mapped libraries and jvm eats some more. So the answer is: measure it properly to find out just how much memory your app needs in a docker container to stay alive and thrive. It’s more than you think. A nice guide tomeasuring and understanding memory in java is http://trustmeiamadeveloper.com/2016/03/18/where-is-my-memory-java/.

CPU? Who needs it?

And then CPU came along. Based on the rough estimates of CPU requirements each app got a handlful and made do. This usually works. But now and then, somebody working on java apps said that their apps would just get killed a short while after startup, with no apparent reason. What’s happening here? After some digging, it turns out that the app was being killed by kubernetes itself, roughly 30s after starting, not by OOM but by not replying OK on liveness and readyness probes. The app in question is a low traffic app with a 2 CPU core limit. Shouldn’t that be enough when normal traffic barely uses half a core?

Well, these apps where running spring boot, spring frameworks less (but only a little) ugly cousin. Less ugly because it’s a modern framework, but still ugly because, well, it’s spring (YMMV). Turns out that all the housekeeping spring wants to do on startup takes a lot of time when all you give it is two cpu cores. The outcome is that it won’t start in time for the readyness/liveness probes complete in time and kubernetes determines this app is dead and finish it off. So there’s two fixes, increase the initialWait for the liveness/readyness checks or boost the cpu limit way higher. I like speedy deploys, so we use the latter strategy.

Measurements

We measure cpu and memory utilization (with the requirement and limit in the graph) per app via heapster/influxdb and display them in grafana alongside our other graphs. This is a great way to see how your app is doing and tuning the limits to suit the app. This is really hard without visualisation in my experience.

End log Operation k8s, day 34

Reading Files in Bash

2016-06-15T00:00:00+02:00

I came across a small but interesting difference in ways of reading files in bash, and specifically processing a file line by line.

I had a small function to read property files and splitting each line in a key and a value and suddenly a bug cropped up. This worked fine when reading existing property files. But when introducing a space right of the =, it failed. It made two “lines” out of “foo=bar zoot”, while one line out of “foo=bar”.

for line in $(< ${file}); do
  echo ${line}
done

It’s our old friend IFS (internal field separator) at work here, with its default value of “" in bash. So we unset it and bob's our uncle.

prev_ifs=${IFS}
IFS=''

for line in $(< ${file}); do
  echo ${line}
done

IFS=${prev_ifs}

(It’s nice to preserve the value of IFS and reset it after you’re done. If you’r in a function you can of course local it instead)

This code works fine without any IFS twiddling, though:

while read -r line; do
  echo ${line}
done < "$file"

There are manny more ways to read files in bash of course. The lesson is of course that I should have tested this better when I first wrote it :-)

The Kubernetes Wars: Day 7

2016-06-12T00:00:00+02:00

DNS mofo

And so one of our ops guys stumbled on the solution. The traffic was generating several thousand dns requests a second, for calling other services and sending metrics. DNS resolution in node is a bit naive, you could say and default it will look up the address in use every time it should be called, with no regard for TTL or caching. And in our kubernetes clusters we are running skydns for some nice features and since we also used an RC template not specifying a dns policy, we got ClusterFirst. This meant that the also under-cpu’ed skydns instances were getting hammered with every single one of those severalt thousand dns resolves every second, peaking on cpu and returning slower and slower responses. This made the node event loop wait increasingly longer for replies and in the end everything slowed to a crawl.

So the fix was to set dns policy to Default which means using the docker dns setup. In our setup that meant dnsmasq running locally on all nodes, which we in hindsight should have been running all along. You learn something every day, eh?

And with that, everyhing just cleared up and we could serve all the traffic (and some more, but that is a tale for another time) on the same number of instances in kubernetes.

End log Operation k8s, day 7

The Kubernetes Wars: Day 3

2016-06-04T00:00:00+02:00

Config

How to deal with configuration? Kubernetes likes app config in environment variables, not config files. This is easy in our node apps using convict, pretty easy in our ruby apps and ranging from relatively easy to bloody hard in our java apps. But how to get config into the replication controllers? We opted for using configmaps (a kubernetes object) to store the config, reference the variables from the rc files and maintain it in git controlled files. So when we want to change to app config, update the config files and run a script which updates the configmap and reloads all the pods for the app. Incidentally, the way we do that, is to delete them, and let kubernetes recreate them. Don’t do this if you run one cluster ;-) We should make the apps read the config automatically, but since none our apps do that, we needed a solution that works now.

This also means we can have separate config for different environments while the file definition of the RC remains the same. We started out with only the configuration variables external to the RC file, but soon realized we needed to externalize the version of the image, the replica number and the limits too.

This has worked really well so far.

The deploy script which either creates everything on the first deploy (from yaml files) or performs a rolling upgrade from one version to another. The deploy script also need to handle the configmap and substituting the values from external files.

#!/bin/bash

. /usr/local/amedia-tools/dev/dev_functions.sh
yaml_dir="/usr/local/k8s-files"
kubectl="/usr/local/bin/kubectl"
kube_opt="--kubeconfig=/etc/kubernetes/config"

if [ $# -lt 2 ]; then
    echo "usage: k8sdeploy APP ENV [VERSION]"
    echo "Deploy a new version of an app to the specified environment. In production, one cluster at a time."
  exit 1
fi

app=$1
env=$2

if [ ! -z $3 ]; then
   version=$3
fi

if [ ${env} == "prod" ]; then
    env="production"
fi

if [[ ${env} == snap* ]]; then
    namespace="${app}-${env}"
else
    namespace=${app}
fi

function deploy_from_file() {
    dc=$1

    line=$(${kubectl} ${kube_opt} get --cluster=${dc} --namespace=${namespace} rc $app 2>&1)
    if [[ $line != Error* ]]; then
        warnlog "RC for ${app} already exist, will not try to re-create from yaml files again"
        exit 1
    fi
    
    if [ -d $yaml_dir/$app ]; then
        ${kubectl} ${kube_opt} create namespace ${namespace} --cluster=${dc} 
        ${kubectl} ${kube_opt} create --namespace=${namespace} --cluster=${dc} -f ${configmap_file}        
        ${kubectl} ${kube_opt} create --cluster=${dc} --namespace=${namespace} -f $yaml_dir/$app/$app-svc.yaml

        #before we create the rc, expand variables from resources file
        #make a tmp file for the newly generated rc file
        rcfile=$(expand_placeholders_from_resources ${app}-rc.yaml ${yaml_dir}/${app} ${env}) 
        
        ${kubectl} ${kube_opt} create --cluster=${dc} --namespace=${namespace} -f ${rcfile}

        #if test or snapshot, scale down to 1 replica
        if [ ${env} != "production" ]; then
            ${kubectl} ${kube_opt} scale --cluster=${dc} --namespace=${namespace} --replicas=1 rc ${app}
        fi
    else
        errlog "Could not find yaml files for app $app in $yaml_dir/$app"
        exit 1
    fi
}

function rolling_update() {
    #check if the service is there
    local dc=$1
    local app=$2
    local env=$3

    line=$(${kubectl} ${kube_opt} get --cluster=${dc} --namespace=${namespace} rc $app 2>&1)
    if [[ $line == Error* ]]; then
        deploy_from_file $dc
    else
        #update configmap first
        out=$( ${kubectl} ${kube_opt} delete configmap config --namespace=${namespace} --cluster=${dc})
        out=$( ${kubectl} ${kube_opt} create --namespace=${namespace} --cluster=${dc} -f ${configmap_file})

        if [ $? != 0 ]; then
            errlog "Error occured when refreshing configmap for app. See error below"
            echo $out
            exit 1
        fi
        
        ${kubectl} ${kube_opt} rolling-update $app --namespace=${namespace} --update-period=1s --poll-interval=2s --timeout=2m --cluster=${dc} --image=dr.api.no/amedia/$app:$version 

        #if successful, patch resource file with new image version
        if [ $? == 0 ]; then
            image=$(${kubectl} ${kube_opt} get --namespace=${namespace} --cluster=${dc} rc ${app} -o yaml | grep "image:" | tr -d "\t " | cut -f2-3 -d:)
            #patch image version into resources file
            patch_file ${yaml_dir}/${app}/${env}.resources ${app} "image" ${image}
        else
            errlog "Rolling-update failed. Inspect the status of the app with k8sstatus ${app} ${env} and dig from there"
            exit 1
        fi
    fi
}

#update k8s-files repo
cd $yaml_dir
git pull

if [ $? != 0 ]; then
    errlog "Could not pull the k8s-files repo. Please check permissions on jump.api.no:/usr/local/k8s-files"
    exit 1
fi

cd - > /dev/null 2>&1

#generate configmap
configmap_file=$(generate_configmap_file ${yaml_dir} ${namespace} ${app} ${env}) 

if [ ! -z "$version" ]; then
    if [[ $env == prod* ]]; then
        for dc in osl2 osl3 ksd1; do
            rolling_update $dc $app $env
        done
    elif [ ${env} == "test" ]; then
        rolling_update $env $app $env
    else
        rolling_update snapshot $app $env
    fi
else
    #first deploy, we need the yaml
    if [[ $env == prod* ]]; then
        deploy_from_file osl2
        deploy_from_file osl3
        deploy_from_file ksd1
    elif [ ${env} == "test" ]; then
        deploy_from_file $env
    else
        deploy_from_file snapshot
    fi
fi

report_deploy $env $app $version

And then the script for just updating the config of an app, without deploying anything.

#!/bin/bash

. /usr/local/amedia-tools/dev/dev_functions.sh
yaml_dir="/usr/local/k8s-files"
kubectl=/usr/local/bin/kubectl
kube_opt="--kubeconfig=/etc/kubernetes/config"

if [ -z $1 ]; then
    echo "usage: k8sconfig APP ENV"
    echo "Update the config for an application. This script will pull k8s-files, convernt the env.properties file to a kubernetes configmap, and restart all pods to read the new config."
    exit 1
fi

app=$1
env=$2
port=$(port_for_app $app)

if [ $env == "prod" ]; then
    env="production"
fi

if [[ ${env} == snap* ]]; then
    namespace="${app}-${env}"
else
    namespace=${app}
fi


function create_config_from_etcd() {
    dc=$1
    env=$2

    out=$(${kubectl} ${kube_opt} get --namespace=${namespace} --cluster=${dc} configmap config 2>&1)

    if [[ ${out} != Error* ]]; then
        #config map is there, delete it and recreate
        ${kubectl} ${kube_opt} delete configmap config --namespace=${namespace} --cluster=${dc}
    fi

    ${kubectl} ${kube_opt} create --namespace=${namespace} --cluster=${dc} -f ${configmap_file}

    if [[ $? != 0 ]]; then
        errlog "Something bad happened and we could not create configmap. Aborting restart"
        exit 1
    fi
    
    #do a replace on the rc, to accomodate for new variables in the yaml
    # first, patch the rc file to the latest (running version)
    image=$(${kubectl} ${kube_opt} get --namespace=${namespace} --cluster=${dc} rc ${app} -o yaml | grep "image:" | tr -d "\t " | cut -f2-3 -d:)

    #patch image version into resources file
    patch_file ${yaml_dir}/${app}/${env}.resources ${app} "image" ${image}

    #make a tmp file for the newly generated rc file
    rcfile=$(expand_placeholders_from_resources ${app}-rc.yaml ${yaml_dir}/${app} ${env}) 

    ${kubectl} ${kube_opt} replace --namespace=${namespace} --cluster=${dc} -f ${rcfile}
    rm ${rcfile}
    
    #if test or snapshot, scale down to 1 replica
    if [ ${env} != "production" ]; then
        ${kubectl} ${kube_opt} scale --cluster=${dc} --namespace=${namespace} --replicas=1 rc ${app}
    fi
    
    # delete existing pods, to recreate and read new config
    had_num=$( ${kubectl} ${kube_opt} get pods --cluster=${dc} --namespace=${namespace} -o yaml | grep -i podip | cut -f2 -d: | wc -l)
    infolog "stopping running pod instance(s) in cluster=${dc} (${had_num} instance(s) found running)"
    ${kubectl} ${kube_opt} delete --namespace=${namespace} --cluster=${dc} --all pod

    echo -ne "[${BOLD}${INFO_STYLE}INFO${RESET}]\tChecking if they are back up (checking in intervals, max 150s): "
    i=1
    END=10
    are_pods_up ${app} ${port} ${had_num} ${dc} ${namespace}
    status=$?

    while [[ $i -le $END && ${status} -eq 0 ]]; do
        ((i = i + 1))
        if [ $i -gt 5 ]; then
            echo -n "."
            sleep 20s
        else
            echo -n "."
            sleep 10s
        fi
        are_pods_up ${app} ${port} ${had_num} ${dc} ${namespace}
        status=$?
    done

    # still on status 0 means end of loop but instance not up ag
    if [ $status -eq 0 ]; then
        errlog "Error: Pods seem unable to restart. Please check their status in ${dc} manually. We need ${had_num} running per server center. Found ${num} running instances"
        exit 1
    else 
        echo "Yay! Back up."
    fi
}

#we need the yaml
cd ${yaml_dir}
git pull 

if [ $? != 0 ]; then
    errlog "Could not pull the k8s-files repo. Please check permissions on jump.api.no:/usr/local/k8s-files"
    exit 1
fi

cd - > /dev/null 2>&1

configmap_file=$(generate_configmap_file ${yaml_dir} ${namespace} ${app} ${env}) 

if [[ $env == prod* ]]; then
    create_config_from_etcd osl2 ${env}
    create_config_from_etcd osl3 ${env}
    create_config_from_etcd ksd1 ${env}
elif [ ${env} == "test" ]; then
    create_config_from_etcd $env ${env}
else
    create_config_from_etcd snapshot ${env}
fi

rm ${configmap_file}

(Yeah, it could do with some refactoring)

Container Metrics

The question that popped up was: when do we know the cluster is running out of resources, and preferably before the deploy fails with events saying you’re shit out of memory or cpu? Container metrics to the rescue.

We have a existing metric system backed in graphite which has worked well for us. We have used the host name as the metric path to give us the oppurtunity to isolate metrics per host. When kubernetes manages the containers, the host on which it runs becomes ever changing. to solve this, we landed on using heapster for gathering container metrics, and then storing them in influxdb. The influx query language gives easy ways to abstract away over changing host names. I am seeing use of influxdb for app metrics in the future too, for the same reason.

Next time on the kubernetes wars: the curious case of the slow node apps.

End log Operation k8s, day 3

The Kubernetes Wars: Day 0

2016-06-02T00:00:00+02:00

So What is Kubernetes?

Kubernetes is container orchestration, deployment, management and scaling, simply put. And boy can it do a lot for you. Take the kubectl cli, with its 48 commands and a grand total of 22 shared command line options and about 10-12 options per command it’s a whopper of a system. So what to do?

Bash, of course.

Write bash (or any other scripting language) to abstract away the forest of options you need for every command, like cluster (one per data center, more on that later) and namespace (more on that too), object name to operate on and so on. You will need an abstraction. I also wanted to fit k8s deployment and management into existing scripts dealing with both straight docker and old school native servers so a separate set of k8s scripts was born. I opted for one per command instead of the k8s style with one big ass one with a million options, because that’s how I roll.

Namespaces, Clusters and stuff

Kubernetes docs says to be vary about multi location clusters so we opted for one cluster per data center, three in total. This gives a bit of overhead when it comes to deployment (we can’t deploy in one big batch, but need three deploys to update all pods) but at the same time we can take a whole data center offline and upgrade the cluster without affecting the other two, which is nice.

Another thing we discovered was that kubernetes by default make configuration and variables available to all containers in a namespace and everyone could see everything when all apps were deployed in the default namespace. So we namespace all our apps into a separate namespace named after the app. As longs as all commands are namespaced (hence wrapper scripts) there is not much hassle. For snapshot and test environments, we use one cluster and not three to keep things simpler.

Scripts, scripts and more scripts

We’ll cover all the scripts in later blog posts, but let’s start. I asked myself the question: what information about a running app do I need to get out of kubernetes most often? In our older stacks, we have hade a ping command which pinged the in-app ping endpoints and gave us a lot of information: where the app is running, what version it is running and which port. Could I get this from kubernetes?

The result was actually two scripts.

k8sstatus APP ENV gives you this:

 
osl2:
NAME        DESIRED   CURRENT   AGE       CONTAINER(S)   IMAGE(S)                           SELECTOR
manifesto   1         1         44d       manifesto      dr.api.no/amedia/manifesto:0.0.3   name=manifesto
NAME              READY     STATUS    RESTARTS   AGE       NODE
manifesto-lxf9x   1/1       Running   3          15d       docker006-osl2
-----------------------------------
osl3:
NAME        DESIRED   CURRENT   AGE       CONTAINER(S)   IMAGE(S)                           SELECTOR
manifesto   1         1         44d       manifesto      dr.api.no/amedia/manifesto:0.0.3   name=manifesto
NAME              READY     STATUS    RESTARTS   AGE       NODE
manifesto-uwu4l   1/1       Running   0          2d        docker009-osl3
-----------------------------------
ksd1:
NAME        DESIRED   CURRENT   AGE       CONTAINER(S)   IMAGE(S)                           SELECTOR
manifesto   1         1         44d       manifesto      dr.api.no/amedia/manifesto:0.0.3   name=manifesto
NAME              READY     STATUS    RESTARTS   AGE       NODE
manifesto-8q4f7   1/1       Running   0          16d       docker008-ksd1
-----------------------------------

!/bin/bash

kubectl="/usr/local/bin/kubectl"
kube_opt="--kubeconfig=/etc/kubernetes/config"

if [ -z $1 ]; then
    echo "usage: k8sstatus APP ENV"
    echo "Display status for an app, both Replication controller and pods"
    exit 1
fi

app=$1

if [ ! -z $2 ]; then
    env=$2
fi

function list_pods() {
    app=$1
    cluster=$2
    namespace=$3

    ${kubectl} ${kube_opt} get pods --cluster=${cluster} --namespace=${namespace} -o wide
    echo "-----------------------------------"
}

#default is prod
if [[ -z ${env} || ${env} == prod* ]]; then
    namespace=${app}
    echo "osl2: "
    ${kubectl} ${kube_opt} get rc --cluster=osl2 --namespace=${namespace} ${app} -o wide
    list_pods ${app} osl2 ${namespace}

    echo "osl3: "
    ${kubectl} ${kube_opt} get rc --cluster=osl3 --namespace=${namespace} ${app} -o wide
    list_pods ${app} osl3 ${namespace}

    echo "ksd1: "
    ${kubectl} ${kube_opt} get rc --cluster=ksd1 --namespace=${namespace} ${app} -o wide
    list_pods ${app} ksd1 ${namespace}
elif [ ${env} == "test" ]; then
    namespace=${app}
    ${kubectl} ${kube_opt} get rc --cluster=test --namespace=${namespace} ${app} -o wide
    list_pods ${app} test ${namespace}
else
    namespace="${app}-${env}"
    ${kubectl} ${kube_opt} get rc --cluster=snapshot --namespace=${namespace} ${app} -o wide
    list_pods ${app} snapshot ${namespace}
fi

Which tells us first the status of the Replication Controller for the app (and the version) and then iterates through all pods for this RC and shows where these are running and status for them. What this doesn’t tell us, is the actual reply from the ping endpoint in the app, and not which IP it is running on (not always needed in kubernetes, but nice to test against when debugging. Thus k8sping was born.

 
k8sping manifesto prod
http://10.30.6.7:9644/manifesto/apiadmin/ping OK 0.0.3 master-undefined
http://10.31.9.9:9644/manifesto/apiadmin/ping OK 0.0.3 master-undefined
http://10.29.8.13:9644/manifesto/apiadmin/ping OK 0.0.3 master-undefined

And whammo, port and ip for the individual pods, and the return value from the ping endpoint.

Source:

#!/bin/bash

kubectl="/usr/local/bin/kubectl"
kube_opt="--kubeconfig=/etc/kubernetes/config"

if [ -z $1 ]; then
    echo "usage: k8sping APP ENV"
    echo "Ping application ping endpoint the pods, all instances."
  exit 1
fi

app=$1
env=$2

port=$(port_for_app $app)

#echo "Deploying $app $version on jump.api.no"

function ping_app() {
    local dc=$1
    local app=$2

    IFS=$'\n'
    ips=($(${kubectl} ${kube_opt} get pods --cluster=${dc} --namespace=${namespace} -o yaml | grep -i podip | cut -f2 -d: | sed 's/\s//g'))
    for ip in "${ips[@]}"; do
        value=$(curl -s http://${ip}:${port}/${app}/apiadmin/ping)
        echo "http://${ip}:${port}/${app}/apiadmin/ping ${value}"
    done

}

if [[ ${env} == prod* ]]; then
    namespace=${app}
    for dc in osl2 osl3 ksd1; do
        ping_app ${dc} ${app}
    done
elif [ ${env} == "test" ]; then
    namespace=${app}
    ping_app ${env} ${app}
else
    namespace="${app}-${env}"
    ping_app snapshot ${app}
fi

Kubectl command output is a joy to parse. Everything (well, almost everything) makes sense.

That’s enough for one day, this is Gordon, signing off :-)

End log Operation k8s, day 0

Talk Friday: Microservice Architecture

2016-03-11T00:00:00+01:00

I quick post recommending a good talk on microservice architecture by Neal Ford from the JavaZone conference last year. He touches on the implications of a microservice architecture and the how and why of it. Lots of details and a broad coverage, delivered in usual neal Ford style. Worth a watch if you’re about to jump on the microservice bandwagon.

Why this talk now? Because I just saw it now ;-)

Building Microservice Architectures - Neal Ford from JavaZone on Vimeo.

Deploy Overlay in Metric Graphs

2015-09-10T00:00:00+02:00

I got a question about this setup on twitter and I thought I’d explain this is in a blog post instead of 140 char twitter messages (which is a pain).

We do quite a lot of metrics in our apps, which are nice for debugging and seeing what is going on. But one thing we have found very handy is to see when deploys to production happen (and they can happen up to 5-10 times a day in various apps, so without a graph it can be very hard to see which deploy made the response time shoot through the roof, or the rate of 500 errors skyrocket.

We use graphite to collect the metrics which are most often sent via the dropwizard metrics framework (which is excellent) and Grafana to present pretty graph dashboards to the user.

First step is to record the deploy. From the deploy scripts, each production deploy runs this little command:

DATE=$(date +%s)
echo -n "deploys.${NAME} 1 ${DATE}" | nc -w 1 -u $graphite-host 2003

Where NAME is the app name, the value sent is 1 and then the timestamp.

Second we use the annotations feature in grafana which makes it easy to toggle this on and off to draw vertical lines for all timestamps where the value is 1. In raw graphite, this is the drawAsInfinite() function. The nice thing grafana adds here, is mouse over boxes with more info. The result looks like this:

The mouse over is the killer feature and shows us that the app orwell was deployed at 2015-09-09 15:36. If this deploy had caused a spike in the graph, it would be immediately visible. This has saved us a lot of digging on more than one occasion not to mention alerted us to the fact that somebody (which shall remain nameless) introduced javascript which double the page rendering time of our front pages :-).

For more info about our metrics appproach, see my earlier post on the subject.

Cache invalidation: explained

2015-01-29T00:00:00+01:00

"There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors"

After seeing a talk on JavaZone 2014 which touched on cache handling in a very unfulfilling way, I thought I’d take a stab at one of these supposedly hard problems. And no, it will not be off-by-one errors.

There are many ways to do cache invalidation, and I am going to be talking about how we do it at Amedia, one of Norways largest online media houses. We use Varnish [3] for all our caching needs and the implementation is kind of tightly coupled to that. But the principles should be useful across other technologies. The central point is providing enough information in your cache objects, to be able to flush what you need, when you need it.

Setup

The server setup which acts as a background for all this, is a fairly complex one. Amedia runs the digital parts of 79 (give or take) small and large newspapers in Norway. This means 151 app servers (excluding the CMS servers, which are 246 instances alone) of which 39 are running Varnish instances with differing configurations. These are physical or virtual machines running multiple apps, spread across 10 different environments and 3 data centers. This system has roughly 6.5M page views daily (in prod) and the sustained throughput of the front varnishes during the day is about 45 Mbps of traffic (each) and combined bandwith usage is around 800Mb/s in the daytime.

Every piece of data, except app <-> database communication, runs over HTTP and through Varnish caches. There is caching in every step of the architecture and it follows that we need to finely tune the cache times, the cache headers and the tools to invalidate on a coarse or fine grained level.

Also, the main use of cache invalidation is journalist writing and updating articles (and other content created by internal users in a similar way) and we, devopsers and developers flushing cache more or less manually by deploying apps, fixing bugs and generally sorting through all the weirdness we can experience in our stack.

Cache headers in general

RFC 7234 [1] (the revised HTTP/1.1 spec, cache portion) mentions the normal cache headers, which can be useful to know about, even though all of them are not part of the cache invalidation scheme I will be discussing.

Age is often set by a cache to describe how old an object is. Apps can also set it, if needed.
Expires sets a human readable timestamp that specifies at which time in the future (hopefully) this object is to be considered no longer fresh and fetched again. In practise it is not needed in our setup as Age together with max-age and channel-maxage specifies what we need.
Max-age is how long, in seconds, should this object be cached by a browser/end user.
Channel-maxage is how long should a cache caches this object, in seconds (not in the RFC).
Cache-Control is the collecting header where you can specify multiple caching values in one header, and groups as we shall see.

See RFC 7234 for the whole truth on these headers. When one app uses several other apps under the hood, the lowest channel-maxage and max-age header from all the backends is used for the response. So a compound response is never older than the youngest “member” object.

Extensions to cache-channel

A draft by Mark Nottingham [2] back in 2007 introduced the concept of Cache channels which are specified as an extension to the cache-channel header (See RFC 7234). Extensions are a part of the HTTP/1.1 spec (revised) and nothing new, but the channel, channel-maxage and group extensions were introduced in this draft but seem to have been shelved after that. I can’t find a mention in any RFC after this. But Varnish can implement this easily through VCL and this is what we do. And it is very useful for cache invalidation.

Varnish Concepts

Some key varnish concepts you should be familiar with:

Purge: A purge removed an object (and its variants) from object memory immediately.
Ban: A ban will add the object to a list and that list will filter objects in the cache. On the ban list means not served, and thus fetched again from the backend. The ban list is also used by the ban lurker process walking the object space and evicting objects.
Softban: A soft ban will ban the object but put in a state of grace. This means it will only be evicted if it can be fetched again. If not, the old one will be served.
Grace mode: Enables grace time for objects softbanned. Grace time is of course configurable.
Saint mode: This mode lets you configure Varnish to not ask a backend for an object for a period of time, in case of errors or other unwanted replies. If all backends fails and saint kicks in, the existing object will be served according to grace config.

Headers

Under this regime HTTP headers becomes all important and not something you just throw around for good measure. They make or break the performance of the whole stack, and need to be kept a watchful eye on. The first thing to be checked when a new app approaches production, is that the cache headers make sense and follow protocol.

Implementation

We have enforced very strict rules in all our apps regarding cache headers and run them through filtering or a library removing headers we do not want. This is important! Rogue expires headers can wreak havoc on a caching solution such as this. The short version is this:

We do not use expires. Ever. Expires is time in human readable form, while ages are in seconds. The combination can be hard to debug to say the least.
Cache-control is used, exclusively. Channel-maxage communicates the TTL for this object to varnish. When Varnish receives a request, the age of the object is compared to the channel-maxage, and this determines wether a cached copy is returned, of a new one is fetched. Max-age is used to set a reasonable default for browsers (Varnish does not use this) to facilitate debugging.
Cache-control groups are added for all necessary keywords for invalidating the cache for a multitude of scenarios, see example below.

Varnish does not out of the box use channel-maxage so this is our implementation in VCL. If channel-maxage is specified, that overrides age/expires that may or may not be present in the object.

# handle channel-maxage in cache-channels, override ttl 
sub vcl_fetch {
    if (beresp.http.cache-control ~ "channel-maxage=[0-9]") {

      set beresp.http.x-channel-maxage = regsub(beresp.http.cache-control, ".*channel-maxage=([0-9]+).*", "\1");

      set beresp.ttl = std.duration(beresp.http.x-channel-maxage + "s", 3666s);

      # subtract age if it exists
      set req.http.x-beresp.ttl = beresp.ttl;
      set beresp.ttl = beresp.ttl - std.duration(beresp.http.age + "s" , 0s);

      # debug
      std.log("CC:beresp.ttl before: " + req.http.x-beresp.ttl + " beresp.http.age: " + beresp.http.age + " beresp.ttl after: " + beresp.ttl);
      std.log("CC:channel-maxage found in " + beresp.http.cache-control + ", duration: " + std.duration(beresp.http.x-channel-maxage + "s", 3666s) + "Age: " + beresp.http.age + ", beresp.ttl: " + beresp.ttl );

    }
}

Example

The app pollux generates complete web pages meant for the end user browser. The data comes from severals systems. One is the data backend, connected to the CMS for the relevant publication. One other is for ad information and a third is a component app, producing parts of the page (served through ESI) again calling other systems further down. These are HTTP requests done in the backend when serving up the page. All these responses have cache-control groups on them, relevant for the app serving them. These are then aggregated up the chain and gets added to the final response to Varnish. Varnish removes them on the way out to the browser, replacing them with “must-revalidate” so the browser always asks Varnish for a fresh copy. But these groups are stored with the object in Varnish and can be used to invalidate the object, on demand if the object should need to be before it expires. Allow me to illustrate:

HTTP/1.1 200 OK
Date: Sun, 05 Oct 2014 16:18:54 GMT
Cache-Control: must-revalidate, channel-maxage=216, group="/pub41", group="/relax-isdans", group="/ece_frontpage", group="/sec71", group="/dashboard", group="/sec25292", group="/art7620213", group="/art7619956", group="/art7620986", group="/art7498595", group="/art7620735", group="/art7619069", group="/art7619936", group="/art7620157", group="/art7619542", group="/art7618923", group="/art7617985", group="/art7617623", group="/art7617283", group="/art7617256", group="/art7617019", group="/art7613958", group="/art7612903", group="/art7615510", group="/art7615883", group="/art7615813", group="/art5520206", group="/art7622276", group="/art7622263", group="/art7622226", group="/art7622224", group="/art7622201", group="/art7622165", group="/art7622067", group="/art7621945", group="/art7621937", group="/art7621892", group="/art7621926", group="/art7621706", group="/art7621476", group="/art7618522", group="/art7621204", group="/art5520202", group="/art7621332", group="/art7621874", group="/art7620605", group="/art7619046", group="/art7620962", group="/art7620562", group="/art7620557", group="/art7620036", group="/art7620014", group="/art7436640", group="/art7621809", group="/art7619526", group="/art7622215", group="/art7622347", group="/art7367496", group="/art7621708", group="/art6456542", group="/art7621547", group="/art7621542", group="/art7619791", group="/art7617979", group="/art7621393", group="/art7619986", group="/art7620883", group="/art7622414", group="/art7621772", group="/art7619994", group="/art7619234", group="/art5520191", group="/art7617945", group="/art7618720", group="/art7621635", group="/art7617936", group="/art7618688", group="/art7620739", group="/art7620879", group="/art7620237", group="/art7620872"
X-Cache-Status: [ normal ;  ]
X-Trace-App: [ pollux ; $host ; Sun, 05 Oct 2014 16:18:54 GMT ]  [ acpcomposer ; $host ; Sun, 05 Oct 2014 12:10:14 GMT ]  [ acpece4 ; $host ; Sun, 05 Oct 2014 12:10:14 GMT ]  [ relax ; $host ; Sun Oct 05 14:10:12 CEST 2014 ]
Surrogate-Control: ESI/1.0
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked

We see the channel-maxage for this object is calculated by Varnish to be 216 seconds, and that is how long it will live in the cache if no purging occurs before that. We also include the must-revalidate keyword for responses meant for browsers, so they will ask varnish on each request. Response headers meant for other apps do not include this.

Here we see the groups for the article data, containing the groups for the publication, the app, the article id and referenced article ids. These will ensure that this page is purged automatically if a journalist edits one of these articles. If we, for some reason want to purge every article, section and front page for that publication, we purge “group=/pub41” and we’re done. Or we purge individual articles or sections, which of course is the common case, for CMS data.

One thing to be mindful of with this model is that if the Cache-Control header exceeds 2048 chars in length, you will run into all sorts of funky error statuses (like 413) from the (jetty) server apps involved, and possibly Varnish too. These can be confusing and hard to debug. So we have code in place to cut groups from the header, if the header grows too long. Different server stacks can use different, more or less arbitrary limits here.

As this applies to all apps, we could just as easily purge all objects using ad data, or using menu data. Or in fact almost everything in our caches in one go. Remember: with great power comes great responsibility.

Here’s the Varnish VCL code to allow PURGE requests to softban[5] objects from the cache.

sub vcl_recv {
  if (req.request == "PURGE") {
    if (!client.ip ~ purge) {
      error 405 "Not allowed.";
    }
    softban("obj.http.Cache-Control ~ group=" + {"""} + req.url + {"""});
    error 200 "Purged.";
  }
}

The implementation and use of this feature is in essence the varnish-cc daemon doing curl on the varnish servers with the HTTP method set to PURGE with the name of the group we want to purge in the path.

The whole chain from backend system registering that someone is editing an object, to the varnish cache being invalidated look like this:

The app itself will send a HTTP message to Atomizer [6] saying that a certain cache-control group should be invalidated. Atomizer (open sourced by us, under the Apache license) persists this in a MongoDB database. The atom feed that Atomizer produces is a 30 second rolling window of cache invalidation events, which atomizer-cc (a perl script, of all things) reads and sends PURGE requests to varnish instances. One varnish cc for each varnish instance is required in this setup. Varnish CC also holds some state internally to make sure that we don’t purge objects that just have been purged, via timestamps but it is quite simple (if you can call anything written in Perl simple, that is).

Some stats

On a normal day, there are about 1500 journalists writing and updating articles, and classified ads being created feeding this system with cache invalidation requests. Normally during the day we see about 50-150 invalidation requests per minute, with an average around 60 or so.

References:

A Usable and Robust Private Npm Repo

2014-10-15T00:00:00+02:00

Since we started using a node.js stack in Amedia we have gone through several generations of private npm repos. And since we finally have found a model that acutally works I thought I’d share it.

Our main reason for having a private npm repo is to be able to deploy and work without being dependant on the global repo. We follow this model for all our app stacks (using nexus for java, private gem server for ruby). We also have internal node modules shared between applications which we won’t publish to global repo because of, ehm, reasons. All repo managers go down, and you can bet they will go down when it least fits you. Been there done that. And the long haul network latency can be a pain too.

First version: replicating couchdb repo

The first repo we set up was following the “standard” guide of using couchdb and the couchapp that runs the global repo. And what a total disaster it was. The replication was growing in size more or less by the minute by node.js hackers reinventing every wheel they can lay there eyes on. Granted, starting out with too small disks was my bad, but worse was also that couch was unstable and the replication thread was spontanously dying on us and not tell a soul about it. Suddenly the repo was out ouf date or just down. Not much better than using the global repo, actually worse at times. Granted, this was on an earlier version of couchdb, but it caused a lot of pain.

Second version: reggie

Then we found reggie as a lighweight, filebased alternative. It used a separate client for publishing and version specs in package.json used a different syntax than normal, but it was stable and usable. Reggie would only hold our internal packages so we returned to using the global repo for public packages. But then npm inc. happened and with that a lot of changes came into npm. Reggie was abandoned by its authors and soon became incompatible with newer node/npm versions. So what to do now?

Three’s a charm

After reading about the setup at Finn but short on details I set out to stitch something together. By this I had testet using npm_lazy as a caching proxy for registry.npmjs.org, to avoid downtime and latency, and that worked well. To work around the (major) deficiency in npm, with only support for one repo, I used Kappa as the repo software. Kappe pointed to two backends, a npm repo couchapp running in a recent couchdb as the first (for writing and holding our internal packages) and npm_lazy as the second, acting as a caching proxy between us and registry.npmjs.org. A bonus is that we now use the exact same syntax as global packages and use npm for publishing.

And since this baby came into production, I can’t recall a single error or minute of downtime for it.

That makes me a happy camper.

The Metrics Approach

2013-10-20T00:00:00+02:00

Recently we started a new initiative at work, and as per usual in our company, things just start to trickle into production when somebody (usually I) get an idea. This time, it was metrics. I read some articles from Etsy and not to forget Coda Hale from Yammer (at the time, at least) (see http://pivotallabs.com/139-metrics-metrics-everywhere/)

The itch started with wanting to know more about what our apps are doing, what effekt our deployment has (at least when arrow start to point in the wrong direction) and generally just know more. Our stack is roughly divided in three non-equal parts: java, ruby and node.js and I wanted something that could handle metrics from all 3 easily.

As the metrics backend we chose graphite, written in python, which is fairly easy to set up, stable and performant for a large number of reporting apps. It supports sending data over both TCP and/or UDP and uses a simple, debuggable plaintext protocol format.

Getting your data into graphite

This is the easy part. You can send pretty much from anything. Command line netcat stuff, java libraries, ruby gems and node modules make sending metrics quite easy. Once they are in graphite, the harder work of figuring out what and how to display the information begins. Metrics not shown, are pretty much useless. Graphites web graphing capabilities are pretty good, but I ended up writing a small in-browser-app wrapping it and making it even easier to navigate graphs, create simple dashboards and set up new graphs when they appear. I still use the native graphite system for testing out new metrics and experimenting. And support for SVG graphs is pretty cool. Fullscreen graphs in a heartbeat.

For java apps, we chose metrics from none other than Coda Hale, which is a well written java library for reporting arbitraty metrics from any app and storing them in graphite (among other backend). Easy integration into an already shared lib made it very easy to report from multiple apps in a couple of days.

For now, we measure:

Exception rates
HTTP status rates for apps concerned with serving content
failed/sucessfull logins for user apps
Load time, render time, request times etc. for the front pages of our newspapers (via a private instance of webpagetest.org)
Production deploys for apps
Sonar violations and duplications for java apps.

And a key part here is that we graph both e.g. exceptions for an app with deployments for that app and relevant collaborating apps. This has already saved our bacon several times, when we see the graph go up just after a deploy, and dive in to fix it straight away. Especially for the front end performance graphs this is very nice (we really don’t want to increase our load time any more than necessary).

Here’s an example graph showing deploys (vertical yellow and brown lines) causing a spike in render times for our front page:

Display

Metrics need to be shown to be understood. We have several big screens around the office showing some of the most interesting graphs and this has led to discovering bad deploys pretty quickly. But we already see that there are too few screens to show everything we should show.

An interesting project I aim to take a look at is the Kale stack consisting of Skyline and Oculus (cool names!). This is a anomaly detection system (Skyline) and anomaly correlation system (Oculus) written specifically for handling a large number of metrics - large enough that you can’t watch them all. Etsy have a lot of cool stuff - and not just the merchandise.

Next Steps

We do not, as of yet, report any metrics from ruby or node apps, but we definitely plan to, using the relevant libraries in Ruby and Node. We mostly display technical metrics for the time being and the reasons for this are: We have other system reporting stuff like click-through tracking, user time, sessions behaviour etc. One idea is to incorporate this into the same graphs as, for instance, deployments to see if there is a pattern to user behaviour after deployments of new feature.

More technical metrics regarding A/B testing is also something we should look at in the near future.

Rapid Java Deployment

2012-11-15T00:00:00+01:00

At my job, Norway’s now second largest online (and paper) newspaper company, I have been doing some work to cut down on deployment and delivery times for our java applications, as well as simplifying the procedures involved. Continuous Delivery is a hot buzzword nowadays, along with DevOps and I think we are quite good at the latter, but we needed improvement on the former. Besides, continuous delivery is about continuous improvements to the pipeline too. And we had some stales processes and tools in place.

Background and setup

We have 4 different environments to which we deploy our software, snapshot, stage, test and production, and snapshot has 4 versions so each team can use one for their stuff, without being overwritten by other teams. Local developer setups exists also, but are not covered here. These are proper, rather beefy servers with quite a lot of apps running, plus two varnish instances.

We use puppet to ensure how the servers are set up, and to distribute changes to config and setup. Puppet is mainly the concern of the ops team.

The old script for deploying one app to, say, snapshot1, was a shell script that did the following:

Run a script on the ci server (CruiseControl) and copy all latest artifacts for all relevant apps (8-10 java/jetty apps of varying size)
Commit these in a subversion repository
Do some magic tagging in subversion
Run puppet-sync on the relevant servers, reading binary data from subversion.
Restart varnish caches for the environment in question.

Time spent: around 10 minutes, not counting the varnish warm-up on the first request. Also you had to wait for CruiseControl to build the app before it could be pushed. There could be a queue.

And most of the time, you were only deploying one or maybe two apps, but everything was done nevertheless. This was needed every time you wanted to deploy an app for someone else to see, a product owner for instance. The reason this had become so slow, was, as is the case in many organizations, that it is written for a simpler time, but then at one point, it stopped evolving along with the organisation and the app ecosystem. This is a grave mistake.. The number of apps to support grows, the puppet-syncing from subversion gets slower, the varnish refresh takes longer etc. It all adds up. And puppet isn’t very fast, and syncing large binary files is something it is particularly bad at.

CruiseControl was also slow at building several apps at once, and you often got some coffee time there too.

What was worse, was that when deploying to stage, test and prod, there was no shared script. There were a few lying around, but nothing solid. And you had to do parts of it manually. Run shell commands for tagging, finding the right servers (they change) and puppet-syncing. And for instance for production, you can’t do all 6 servers at once. You take one or two, and wait for the varnish ping to find them, let it warm up, and do the next - all manually and watching as servers came up and down. Manual procedures are a pain, and many developers were vary of this. Lots of help seeking and asking of questions. And mind you, this procedure is not done by sysadmins, but by the developers. This is a good thing, but the threshold for doing it, and doing it well, had to be lowered. Our goal is that all developers should be able to push to production, safely, fast and with confidence.

The rest of the blog post is about what we did to improve this.

The New Way

Goals were:

Faster CI pushing artifacts to repo manager.
Scripted one-click for all environments
Not do more than was required, i.e. only deploy and restart the apps specified.
Ditch puppet for syncing large binaries, use it for server config only.
Speed things up in general.

As a backdrop, we migrated to Jenkins with four build nodes for speeding up build times when there is a lot going on. Simultaneously, we migrated from subversion to git, and started using Java7 and maven3 (yes, I know, we’re slow on the uptake).

The main job here was to create a script that was to be run on the app servers themselves which would do:

Get a app name and a version on the command line
Fetch that app from a Nexus repo, other file repo or git repo
Optionally restart.

There are several benefits to this. One is that it is easy to script, another that it is very flexible and you can choose to run it via puppet, manually or via a script.

So we then made some big changes. All apps got snapshot builds (by maven) deployed to nexus, alongside release builds. Snapshots are pushed from jenkins CI, or manually. But we also added the option of building an app from source if you need to e.g. push a branch to a server for someone to test. The old deployment routine only pushed war files for the webapp project to the server, while now we build a zip file bundling the war file and the dependencies needed for jetty startup too. This will make future jetty upgrades much easier too.

The deploy scripts looks like this on the command line:

update-dev -s dev0 appname
# will update appname to the latest snapshot release on environment dev0

update-dev -r dev0 appname
# will update appname to the latest release version on environment dev0

update-dev -a -r dev0 
# will update all apps to the latest release version on environment dev0

# if you want a specific version, either release or snapshot, do
update-dev dev0 appname:1.45

# if you want to custom-build either do

update-dev dev0 appname:branch-foo:21451a
# to clone repo for appname, and build revision 21451a of branch 
# branch-name. This is then copied to git repo, and deployment 
# script fetches it from there. if revision is omitted, you get HEAD (no pun intended).

The main points here are:

Only update the apps you specify via the script running on the server.
Optionally update server config, via puppet.
Ban URLs in Varnish, don’t restart.

In addition to this, the script asks puppet which server is the current one for the environment, instead of hard coding this in the script. This means less maintenance burden when servers change. A central point is also to only restart apps you specify (instead of all apps) and flushing/banning Cache-Control groups in Varnish instead of restarting.

The work is distributed between the two scripts thusly:

Script run on dev machine checks for valid version
Does custom build if needed
Finds servers
Other misc. stuff like command line parsing
Calls the right script on the right server

Script on server will then

Fetch version
Unzip app bundle, move stuff into place
Tell puppet which version is now current for the environment in question
Optionally restart

And then dev script again takes over and flushes caches.

Updating one app to latest snapshot or release and banning app URLs in Varnish is now down to 10-12 seconds not counting savings in cache fill time for the first request. Several of these seconds are spent querying puppet for metadata. This will be a bit faster in production.

Further Work

Next up is using the same deployment strategy for all environments, but with a custom rollback feature for production and a wait strategy between servers. We thinking about extending this to not wait a certain amount of time, but rather monitor Varnish and move on once the backend is reported as up, and answering requests. Also, a wrapper script to be called from Jenkins for deploying to stage, run smoke tests, await result and deploy to prod if all is well is planned. And of course, no varnish flushing in production.

All in all, shaving of ~10 minutes of deploy time (this amounts to hours of wait/ineffective time on a busy day) is pretty significant. But the main win is less manual procedures in production, more power to the developers and faster delivery, when we need to.

Helpful stack traces? Not so Much

2012-10-10T00:00:00+02:00

I am dabbling around with Node.js at work these days and we are trying out Jake as a build script. Jake itself is not so bad and the syntax is pretty neat, with a very small DSL for writing build scripts in JavaScript. But, with some google-fu, it’s not that hard to find accounts of people not knowing what to make of a node stack trace. So, here’s what I got when running a build script:

nikopol: $ jake --trace build-extension
jake aborted.
Error: Process exited with error.
    at fail (~/nvm/v0.8.2/lib/node_modules/jake/lib/api.js:221:18)
    at utils.mixin.exec (~/nvm/v0.8.2/lib/node_modules/jake/lib/utils/index.js:64:9)
    at EventEmitter.emit (events.js:91:17)
    at ChildProcess.utils.mixin._run (~/nvm/v0.8.2/lib/node_modules/jake/lib/utils/index.js:186:20)
    at ChildProcess.EventEmitter.emit (events.js:91:17)
    at Process._handle.onexit (child_process.js:674:10)

Hookay, something is clearly wrong here but what is it? There’s some event emitting going on, and some mixins being visited. And an exec() there too. Well, if we inspect the build script there is exec’ing of some shell commands for catting some files together. But what is the error being thrown from the shell command? Turns out, the error is actually a missing directory. Jake and Node could really be more helpful by giving some hints of that. No mention of the underlying OS error, no mention of the exit status, nor the command throwing the error (the exec call can take an array of commands to exec) or anything. Score -1 for Jake when it comes to good error messages. I’ll see if I’ll try to fork it and submit a patch on the mixin._run method to spare some other soul some useless debugging.

The Tweet Storm

2012-10-08T00:00:00+02:00

A while back I experienced the full force of Twitter, in a nutshell. I am not a very active twitter user, but I follow som celebrity developers and once in a while I throw out a tweet in reply to one. This normally just leads to (at best) a few tweets back and forth and nothing more. But not this time. I responded to a discussion between Ted Neward and Scott Hanselman, both with a sizable number of followers. They were talking about some presentation from Microsoft on best practises. I spoke my mind on it:

"There are no best practices, there are only practices and their suitability depends on the context."

This lead to a retweet from first Ted Neward and then Scott Hanselman. And then things really took off. In the next 24 hours I got something like 50-60 retweets, just from being retweeted by some celebrities. Granted it’s not that many, but so much more than I have ever gotten before. That’s simply how Twitter work. Funny thing is that only one (1) spoke out in disagreement of the statement.

Context is King

The statement in itself is a post modern stance on applying practises in software development. Some, but not that many, argue that some practises are good regardless of the context you are applying them in. I disagree. In the past I have felt this too, but every single time someone or something has some time later disproved that, and I now try to think about the context before doing anything regarding process or practises. Universal truths are few and far between, especially in software development.

For instance, I have been, and still am, when the context fits, a strong proponent of some practises:

Writing tests first, and writing tests in general.
Don’t copy and paste code and keep duplication low.
Monitoring code quality (test coverage, coupling, cyclomatic complexity etc.)

I have then seen talks from e.g. Dan North about the work at DRW and Fred George about the work at Forward. For instance at Forward they have a micro service architecture, where a larger system is built of tens or hundreds of really small services or apps. Really, really small. So what happens to your feelings towards unit testing when the entire app is maybe 100-200 lines of code? And the maintainability of the code? Is it more efficient to replace (write a new one) that change it, if the change is more than a small fix? This is actually what they do at Forward: write little to no tests and replace apps rather than change them (for bigger values of “change”). Continuous integration? Don’t need it. Refactoring? Nope. Sprints? Nix. User stories? Nope. Developers and Customers talking to each other making the decisions. This is the single responsibility principle applied to the application level, not the class or module level.

Dan North has talked about how his attitude towards classic agile methods when he started at DRW. He has collected some new patterns (of effective software delivery, as he calls them) of how experienced programmers can deliver software much faster than the so-called “hyper productive teams” that Scrum is peddling. And this is done by breaking the rules of classic agile and using other practices instead. is reasoning is that class agile presupposes a lot of things, and when those things aren’t true, neither is the practise that it mandates. And so other practises emerge. Like using TDD as an exploration tool, starting with a private void testBlah() and just exploring the code with no intent of keeping the test. But if you decide to keep the test, clean it up, change the name and commit it. Likewise with a class. Just start with a class, mess around with it and later on decide if you want to keep it or not. Defer the commitment to the code and write tests for it later, and make it all TDD-shaped and nice. But not before you know you are going to keep it.

These are really interesting thoughts and shows that some practises, that seem very sane in some (or perhaps most) contexts, will disappear completely in another. And both DRW and Forward are hugely successful at what they do, too. Kent Beck has been touching on this in his talks on The Effects of Acceleration where he talks about the effects of faster delivery has on your practices. Well worth checking out.

References:

JavaZone Video Mashup

2012-10-02T00:00:00+02:00

Video.javazone.no was just sorta soft launched, a mashup site collecting the videos of talks from JavaZone from 2009 through 2012. As with any project where you don’t have the time you want, there are some things I am happy with, and some things i am, well not so happy about. But overall I am pretty happy.

The site collects a total of 417 talks from 290 speakers over the last four years, and 100 additional talks from the same speakers from other conferences and meetups. The main features are more metadata on the talks, display of session information (abstract, speaker bio etc.) alongside the vimeo video and metadata about the video.

Implementation

The main app is written in Play Framework 1.2.5 and using MongoDB in the backend. The reason for choosing MongoDB is primarily that it is easy to work with (low developer friction) and has enough querying capabilities for this project. And it’s not SQL. The data model also fits pretty well, with little coupling of different objects at the database level. A simple db reference handles it nicely. MongoDB has some, how should I put it, interesting views on data integrity and there are situations where you can lose data running on a single node. The data set for this app is all generated and updated while running, but can be recreated at any time. So losing data isn’t a big deal, and it doesn’t happen that often, either. So I decided MongoDB was a nice fit.

The reason for Play was mainly to test it out and see if it’s a good framework or not, and also to get away from some of the friction of using standard java server frameworks like Spring MVC (or indeed any server toolkit doing it the normal java way, which is tedious and needlessly complicated). Play lends itself quite heavily on the convention over configuration approach and removes a lot of clutter when setting up a simple controller that has responsbility over an URL, fetches some data, feeds that into a template and renders it. The fact that you default can’t run a single test but have to run all tests in a test class is a real PITA. Play has some other mild annoyances too, but generally I am pretty happy with it.

All in all, these things have been involved in the site:

Play Framework
MongoDB
Twitter Bootstrap for templating/design/responsive stuff.
FontAwesome for nice icons.
SASS for better stylesheet authoring.
Rake and git for vcs, packaging and deployment.
Heroku and MongoHQ cloud providers for running test sites.

Here’s a little screenshot for you:

I collaborated with bjartek on it for a while and as he is a scala/functional programming buff, we tried out a few different techniques. Some using Google Guava and some using Functional Java. This made the codebase a little messy in some parts. The java syntax doesn’t really support it well, either. Will try to clean up that soon. Anyway, if you try the site, I’d love some feedback on it. Just bear in mind that I am no designer :-)

Headless Tests with Buster.js and Phantom.js

2012-09-28T00:00:00+02:00

This setup was pieced together via several gists, tweets and articles so I thought it would be good thing to collect it all in a post. http://busterjs.org/docs/browser-testing/ says phantom.js support is “not yet landed in the beta”, but that is the config sugar for doing it through the buster.js config file. It works just fine nonetheless.

We have been moving tests for the frontend projects at work to buster.js instead of JsTestDriver. Mainly because Buster.js is faster and has a richer set of tools built in. JsTestDriver has some annoying bugs too and does not seem to be actively developed these days.

But, running JavaScript tests in a browser is fairly easy, both with Buster and JsTestDriver by capturing a browser and sending data back and forth. But when running fast unit tests and running tests in a CI server, headless tests are nice. Fast and with no need for setting up multiple browsers on the ci server running on a X display of some kind. Of course, this can be done, but it is a hassle. Now if you need browser testing in several browsers, I suggest using an external service for that. Anyway, for our tests, most of them are not threading the fine line between browser incompatibilities as we are developing a chrome browser extension. So we can run the tests in phantom.js normally and run them in the target browser periodically.

Enter Phantom.js

Phantom.js is a 100% headless (as of version 1.6) headless webkit browser engine. It can be used for all kinds of nifty things like taking screenshots of web pages (the whole page, and not just the viewport), scripting interaction with webpages (there’s an example of using Google Maps to find driving directions) and finding the load time of a web page, to name a few cool things.

Installing Buster.js and Phantom.js

Install Phantom.js by downloading it and unpacking the tar-file. If you’re on a mac you can do:

brew update && brew install phantomjs

Buster.js is a node module and can be installed by running the following command in a recent node version:

npm install -g buster

or checking out http://busterjs.org/docs/getting-started/ if you need more handholding. Depending on your platform you may need to adjust your path variable or symlink the buster executables into your $PATH

Running the Tests

So to run some buster tests in phantom I have made a small project with some tests

buster.js
bin/
  server.sh
  kill-server.sh
  phantom.js
test/
  math-test.js
lib/
  math.js

The lib/math.js file is a small object with a simple function, and test/math-test.js is a test case for that. Nothing fancy there, but I list them for completeness. These files should hold your production code and tests.

// lib/math.js

myapp = {};

myapp.Math = function() { };

myapp.Math.prototype.square = function(i) {
  return i*i;
};

// test/math-tests.js
buster.spec.expose(); // Make some functions global

describe("A math module", function () {
  this.foo = new myapp.Math();

  it("squares 1", function () {
    expect(this.foo.square(1)).toEqual(1);
  });
         
  it("it raises any number to its power", function () {
    expect(this.foo.square(2)).toEqual(4);
    expect(this.foo.square(3)).toEqual(9);
  });
           
});

The contents of buster.js (buster config file):

var config = module.exports;

config["My tests"] = {
    rootPath: "./",
    environment: "browser", // or "node"
    sources: [
        "lib/*.js"
    ],
    tests: [
        "test/*.js"
    ]
}

server.sh is a script for starting the buster server and the phantom.js instance, and run the phantom.js script on startup:

#!/bin/bash

buster-server & # fork to a subshell
sleep 2 # takes a while for buster server to start
phantomjs ./bin/phantom.js &

Then we have the phantom.js script for capturing the browser, in phantom. We also redirect any alerts to console.log() instead, since we won’t see them.

var system = require('system'),
    captureUrl = 'http://localhost:1111/capture';
if (system.args.length==2) {
    captureUrl = system.args[1];
}

phantom.silent = false;

var page = new WebPage();

page.open(captureUrl, function(status) {
  if(!phantom.silent) {
    //console.log(status);
    if (status !== 'success') {
      console.log('phantomjs failed to connect');
      phantom.exit(1);
    }

    page.onConsoleMessage = function (msg, line, id) {
      var fileName = id.split('/');
      // format the output message with filename, line number and message
      // weird gotcha: phantom only uses the first console.log argument it gets :(
      console.log(fileName[fileName.length-1]+', '+ line +': '+ msg);
    };

    page.onAlert = function(msg) {
      console.log(msg);
    };
  }
});

The last script, kill-server.sh is for running after the tests and shut everything down again.

#!/bin/bash
# just call with ./kill-server buster-server|phantom

function get_buster_server_pid(){
    echo `ps aux|grep buster-server|grep node|awk '{ print $2 }'`
}

function get_phantom_server_pid(){
    echo `ps aux|grep phantomjs|head -1|awk '{ print $2 }'`
}

case "$1" in
  "buster-server") server_pid=`get_buster_server_pid` ;;
  "phantom") server_pid=`get_phantom_server_pid` ;;
esac


if [ "$server_pid" != "" ] ; then
    kill $server_pid
    echo "$1 killed"
else
    echo "$1 not killed. Found pid was=$server_pid"
fi

Running all these together, after another, we start up a buster server on localhost:1111, start phantomjs and use that to capture the buster server, run the tests, and the kill the server and phantom afterwards.

{nikopol:buster: ->./bin/server.sh 
buster-server running on http://localhost:1111
{nikopol:buster: ->buster-test 
PhantomJS 1.6.1, OS X: ..                                                                               
1 test case, 2 tests, 3 assertions, 0 failures, 0 errors, 0 timeouts
Finished in 0.006s
{nikopol:buster: ->./bin/kill-server buster-server && ./bin/kill-server phantom
buster-server killed
phantom killed
{nikopol:buster: ->

If you’re running the tests locally, you can leave the server and phantom running. But we want to run them in the CI server, so we clean up after tests are run.

Running on Jenkins

We run the build of this project in Jenkins and want to display the test results there, along with csslint, jslint and other quality checks. Buster.js has a nifty command line switch for outputting junit compatible xml, which Jenkins reads out of the box. For instance this command, instead of plain buster-test in Jenkins, with config for the violations plugin pointing to the file, will graph the test results nicely.

buster-test --reporter xml > reports/test-report.xml

The scripts can also of course be configured to run through npm, if you want.

That’s it, happy phantom.js and buster.js hacking!

Blog Reboot

2012-09-26T00:00:00+02:00

Firstly, I apologize for resetting the RSS feed for my blog, but that was I am afraid, unavoidable :-)

I got really tired of (not) upgrading my Movable Type installation up to the point where it was becoming a blatant security risk, more or less. So, after working a lot with git lately, both at work migrating every project from Subversion over to git, and privately, I decided to switch the whole blog over to git and github. I am now using Jekyll and more precisely Jekyll Bootstrap which is a layer built on top of Jekyll, with some themes using Twitter Bootstrap for the design stuff. A small rakefile is used for deploying the generated site to my web hosting provider.

Jekyll is very easy to work with and generally I am pretty happy with it. The Hooligan theme is a bit heavy on the effects and I might tone it down when I get round to it. There are at present no comments on the blog, and I am quite frankly unsure if I need them. Perhaps some social media sharing buttons will work just as well. Anyhow, jekyll bootstrap has among others Disqus support built in, so I might take that for a spin later.

Lastly, now that mountain biking season is very much near the end, I might even find time to write a few more blog posts too. Updates has been slow over the summer. I have some interesting projects going on at work now, and there might be a blog post or three later this fall. Topics will be git, continuous delivery and perhaps some NoSQL/architecture stuff. We’ll see.

What I Create

2012-02-24T22:43:13+01:00

Along with a lot of other people I know and follow on Twitter one of my resolutions for this year (I’m normally not big on new years resolutions) was to consume less, twitter, blogs, articles and videos, and create more. I spend far too much time on twitter, following interesting links, and well, not so interesting but entertaining links and soon enough I am laughing at a cat with glasses farting unicorns off somewhere in the darkest corners of the Interntet. Enough of that!

I came across a nice quote from Why the lucky stiff in Giles Bowketts excellent blog article from a few days ago

- When you don’t create things, you become defined by your tastes rather than ability. Your tastes only narrow & exclude people. So create.

So in order to up my game a bit, I thought I’d blog about what I create and what I intend to create in 2012.

An ebook for my son

I described it in more detail in a blog post of it’s own. I am proud of this one, not so much because of the technical merit, but how well it worked. My son is reading like a champ, got praise from his teacher and is moving on the books and Donald Duck magazines. This is also an ongoing effort, as we’re only at book 3 now.
Conference tutorial

I have submitted a tutorial suggestion for the 2012 Roots conference on the topic of smoke testing web apps. As of writing the jury is still out on this one, but I hope it gets approved. I have done some work on smoke testing at work these last few weeks and it has already saved our bacon several times.
Javazone video site

I am also working on a video site collecting the vimeo videos of the last two years worth of presentation videos from JavaZone and mashing it up with some other data sources. It’s not quite ready for prime time url dropping just yet, but I’ll return to a separate blog post when that happens. I hope it’s received well.
Acting on suggestions

I have become more active in suggesting new things to try at work, for improving our workflow and following through, building scripts etc. The smoke testing effort is one example of this, but also trying out some suggestions for process changes. Next step here is to encourage my co-workers to do the same also. That is the harder part in this.

Non-tech

I count actually doing non-tech related stuff also a big part of this. Time is short when the agenda is hogged by kids, but there are two big things planned this spring. First is doing a little house refurnishing and actually building something from scratch, not just mounting Ikea furniture. The second is taking another step in performance on the mountain bike. More hours on the bike seat and working on improving the quality of the training as well. And we’ll see how that plays out in the end of May when the first race is on.

All in all I would definitely say that this plan has improved my focus and I also find it brings more creativity with it. I have never come up with this many ideas of things to try before and also having the energy to try them too, is just great.

I am a happy camper.

Getting My Son to Read With Technology

2012-01-18T20:35:46+01:00

I have the exact same challenge as Kenneth Kvalvik (article in Norwegian): A 6 year old boy who is starting to read but can’t find anything he likes to read. He hates home work assignments (and who can blame him, they’re boring). And as the son of the author of the article above, my son also loves Star Wars. He hasn’t seen any of the films but plays star wars games quite diligently on his Nintendo DS and plays with Star Wars lego a lot.

So why not give it a go?

The originator bought a star wars book and translated it and added images from the movies. I skipped that step and went straight to the movies. In part because after reading for only a couple of months (and having only covered about half the alphabet in detail) my son needs very simple text.

One tough question is: which order should I present the movies in? Original trilogy first or prequel first? I feel every fiber of my Star Wars fanboy body resisting, but I chose prequel trilogy first. Anything else would probably add to the confusion.

The Plan

My son also loves using the IPad, so I thought I’d take advantage of that and create an ebook app (or “app”) for that platform. I also needed to be able to add more pages quickly (if it was a hit). So I landed on using plain HTML5 and just creating a web page and use the “add to home screen” feature and make it look like an app.

I am using swipe.js which is very lightweight javascript library for just doing swipe gestures on the IPad. I checked out both Sencha Touch and JQTouch (which I have used before) but for this small project they were both overkill.

The layout of the app is a simple directory structure, one folder for a book, with one sub folder for a chapter with images and videos for that chapter. Each chapter is one html file, with divs that swipe.js is showing and hiding as you swipe. Each page has either an image or a video, and from one to three short sentences in uppercase (written in lowercase and text-transformed with css, so I can lowercase it again when he’s better at reading).

I went with using movie screen shots for some pages (and the parts where we need some censorship of gory details (not much gore in Star Wars but for a six year old it can be a bit scary)). I also added some small movie clips to some of the pages to keep his interest up and give him a little break in the reading. I researched som HTML5 video players, but found that they all, as per usual, sucked in their own unique way, and went with the safari default video player on the IPad. Which works wonderfully for my use.

Another takeaway is how much you can cut away from those movies (especially the prequel trilogy that, well, isn’t all that good) without losing the core of the history. The second movie with all that sticky love stuff between Anakin and Padme will be cut even more. 6yo boys are not that into romance.

Results

So how did this turn out? I created two chapters of the first book before starting a beta test on my first (and only (non-paying)) customer. He read everything in one sitting and immediately asked for more chapters. And that question was repeated every day over the following weeks where I hadn’t had the time to write more chapters. Never had such a pushy user before!

Technically I have some duplication in the code, specifically the chapter menu. The layout isn’t all that great either, but I plan to fix that soon. The customer was so demanding that I was forced to take on some technical debt to get the product out into the market :-) I have also planned to add slideshow variant to make images clickable and go full screen.

Now we’re at the end of the first book, and he is asking when the second books becomes available. So I would say in terms of getting a reluctant 6 year old to read, it’s a huge success. He want’s to read it again too. He comes for the action sequences in the video clips, but there are no protests when I say we should read too.

(Since these movies are all copyrighted, and in these dire #SOPA times, I can’t publish the finished results anywhere, I’m afraid. The text, in Norwegian, is in the code over at Github, but the images and videos are not. And won’t be.)

A #smidig Retrospective

2011-11-16T08:52:30+01:00

(This was written just after the Smidig conference but got lost in daily life and something else (probably very important!). Just posted it to get it out of the way)

I attended the 5th Norwegian agile conference Smidig in November, and as any good agilist, I thought I’d do a little retrospective for the conference. Often the good ideas flurry at the conference, but when reality sets in when you return to work, the good ideas have a tendency to move to the back of the consciousness and not get tried. So, in order to prevent that here’s a list of impressions and more importantly a list of things to try out at work.

1. Pull based Meetings

This is an idea I had before the conference, but it solidified during it. Instead of running regularly scheduled demos, plannings and backlog grooming meetings, make them happen on-demand. Demo when you have something to demo, estimate and ready stories when backlog is getting slim. And perhaps use shorter, more frequent meetings instead of longer, infrequent meetings. Long meetings suck.

2. Code Review as Step on The Board

We have an informal, voluntary code review step, but use is infrequent and varies a lot. After a lunch talk with my former colleague Rune Sundling, who work in kanban heavy shop, I thought it worth a try to create a code review step before “done”. Wonder how the team will react to the suggestion, though :-)

3. Defining requirements quantitatively

I didn’t catch Tom Gilbs talk at smidig, but I know his ideas on this subject. And I am in a position to try this out now. A lot of work will have to go into finding the right quantitative measures for the requirements, but I like the idea and I think it fits our domain and prioritization process well.

Btw, the talk videos are already available on the smidig vimeo account (first videos appeared the same day the talks were held!)

JavaZone, New Job and the Case of the Missing Blog Post

2011-08-30T21:32:31+02:00

I’m working hard these days putting the finishing touches on my presentation for this years JavaZone conference. It is a privilege to be able to talk there and I thank the organizer for giving me a full hour on the topic. I will be talking on the subject of deployment pipelines using Jenkins CI as the tool, and going through a lot of techniques and scenarios for evolving a pipeline to match your context. And reasons for creating a pipeline in the first place. I gave a lightning talk at the Roots conference in May on the same topic, but there was a lot more to be said than fitted in ten minutes. Hope to see a lot of people there!<div>
</div><div>And on the subject of continuous delivery: I wrote two blog posts on CD and deployment pipelines a while ago and promised a third one. The third one is coming, but I won’t make any promises as to when. I am starting a new job September 1st, at APressen Digitale Medier and I am guessing things will be hectic. Mountain biking season is soon ending and I guess I’ll get some more time when winter hits. I’ll also be spending the winter doing some more ruby stuff and doing HTML5 and other front end work on the munin project and the upcoming 2.0 release.</div>

Roots 2011 Braindump II

2011-05-24T12:37:48+02:00

Second day of Roots 2011 is over and here are my highlights:

The 7 Duties

Opening keynote of the day was Jurgen Appelo with a talk on the 7 duties of a software professional. Basically how and why you should work on improving yourself in every aspect: motivating yourself, direct yourself, connect yourself (to others), measure yourself and so on. A previously recorded version of this talk, with jokes about the Germans instead of the Swedes is on the internet. All in all a great talk and nice advice for moving forward instead of standing still in your life and career.

Graph Goodness

After that came Jim Webbers Neo4j hacking tutorial by going through (some of) the neo4j koans koans to get a feel for the neo4j API and how to program it. Didn’t have time to cover a whole lot, but I will try to go through the rest of Koans back home. I have used Neo4j a bit earlier, but I want to get the feel of it in a larger context and use the API more thoroughly.

Secrets of Agile Architecture

Dan North was on once again after lunch with a talk on Secrets of Agile Architecture. He was all over the map talking about topics such as treating organisations as systems and how changing the architecture changes the organisation, managing impact instead of obsessing about likelihood of options and choices. Conway’s law and Dan’s corollary to it, Real Options, PDCA, the role of the architect as a minstrel in modern software development and much much more. Dan is never short on ideas. Some cognitive overload happened there and I need some time to digest all the ideas. Nice talk.

HAML And SASS

Anders Norås then again took over with a tuturial on using HAML and SASS for writing web applications. Great intro and kudos for creating usb sticks with virtual image with pre-installed software. I have used these two technologies a bit before but I joined anyway to pick up some new tricks. Nothing revolutionary there, but a nice reminder that I use it to seldom in daily life. Need to prioritize my rails projects now that the talk for Roots is out of the way.

Roots 2011 braindump I

2011-05-23T13:29:58+02:00

I'm here at the Roots 2011 conference in Bergen and the program is packed with wonderful stuff. So I thought I'd venture into liveblogging/core dumping the interesting stuff I see.

A Way North!

Dan North kicked of the ball with the opening keynote titled Patterns of Effective Delivery and North is entertaining as always. He outlined several patterns he has observed which goes against a lot of conventional wisdom in the agile world. Like starting writing a test named "public void testBlah()" and just start coding. And then renaming it later on when you see if the code is worth keeping. Or starting a new piece of code by copying and pasting (Ginger Cake Pattern). And possibly refactor out common pieces much later, when you see where the code is taking you. All in all a great talk and I will dig deeper into Dan's findings. The keynote will be online in video form later if all goes well, too.

Jim Webber: Rest in Practice

Jim Webber, Chief scientist for Neo Technologies, makers of fine graph databases, was next with a 1 1/2 hour rant on SOA and REST. In short. SOA sucks, REST is good, which is hard to disagree with. And a lot of colourful English idioms was picked up during the talk as well. Great fun, but a bit lacking in nutritional content, perhaps.

Lightning Round!

Lots of small gems here, too long to go in depth on all of them. Johannes Brodwallstarted off with a talk on how to be an agile architect with some nice tips. Andreas Heim talked about virtualizing your development environment for a more rapid startup and better control over your infrastructure, using vagrant, chef/puppet/linger. After that Filip Van Laenen talked about outsourcing self-discipline to the computer and running stuff such as static code analysis metrics as a part of your development process. Interesting take! The first session was rounded of be total geekery on GPU-accelerating normal apps using NVidia CUDA processors.

Second Round

Second slot was the track on continuous delivery, started off by Stein Inge Morisbak on version control branching strategies for making rapid/continuous delivery possible. Followed by his colleague Ole-Christian Rynning on techniques for automation and delivery: promote binaries, create built-in health checks and let the team deploy. Then there was me talking about deployment pipelines in Jenkins (slides can be found on slideshare.net) and the session was rounded off by Mario Aparicio on facts and speculation about NoSQL. After doing quite a bit with of work with various NoSQL databases, I found that last one to be a bit shallow and lacking.

Third Round

Third round was a fantastic talk by @anoras (Anders Norås) on the notion that we are to busy sharpening the saw/following the next trend to get things done. And even getting things done is a trend. He rounded off by handing out certification documents certifying all in the audience as Bul Shi artists, an ancient Japanese technique. Great fun. Thommy Bommen followed up with a really nice talk on be a responsible person in life and in work, which was followed by a meta-talk on automating lightning talks and getting people to give them inside your own company. Another fine talk by Eirik Bjørsnøs. Last talk was Ellen D. Varsi on hitting the nail on the head and finding the right abstractions for software and getting it right.

Open Spaces

The open spaces didn't work as well as I have grown accustomed to at Smidig. I think a lot of people went home before them. So few ideas and small groups, but it turned out OK. First slot i used to hack on couchDB replication (dead simple and cool) and CouchApp with a few of the local bergen couch mafia. Good fun. Last slot was the question of are we doing to much automated testing through the GUI. It turned into testing strategies and at the end Dan North popped up and joined too.

And then there was beer and music...

Continuous Delivery II: Smoketests in Ruby and Rails

2011-04-04T19:33:03+02:00

This is part II in a blog mini series on building a build pipeline with Jenkins. The other parts, part I and part III could be worth a read too.

This part is about a implementing built-in smoke tests in a rails application. This Is by no means new, but I added a little something at the end which I thought was quite nifty.

Class definition

I created a small class to hold the actual smoke tests (pardon my ruby, I’m learning):

class Smoketest
  attr_accessor :description, :status, :message

  def initialize(descr)
    self.description = descr
  end

  def run
    yield
  end

  def self.OK
    "OK"
  end

  def self.FAIL
    "FAIL"
  end

end

These are called from the SmoketestController. I chose to just put all (well all two) smoke tests in the controller, but they could easily be distributed and put wherever you want. The controller methods looks like this:

require 'smoketest'

class SmoketestController < ApplicationController

  def index
    @tests = Array.new

    mongodb_smoke_test
    test_test

    respond_to do |format|
      format.html
      format.xml { render :action => 'index.xml.haml',
                          :layout => false }
    end
  end


  def test_test
    test2 = Smoketest.new("Test")
    test2.run {
      test2.status = Smoketest.OK
    }

    @tests &lt;&lt; test2
  end


  def mongodb_smoke_test
    test = Smoketest.new("MongoDB test connection")

    test.run {
      begin
        User.first
        test.status = Smoketest.OK
      rescue Exception => e
        test.status = Smoketest.FAIL
        test.message = e.message
      end
    }

    @tests &lt;&lt; test
  end

end

The “test” one is just for making it more than one, and the mongoDB test tries to do a query and gives it and ok status if that works, and a fail with stack trace if somethings amiss. In a bigger application with more external systems, the number of smoke tests would go up. Running this, gives this nice web page for visual inspection:

(The layout is a bit wide since I wrapped in the standard layout for the app.)

The Junit output

I wanted the jenkins build job to track these smoke tests, which are being run as a build step after deployment. The easiest way to do this I figured was to make it spit out Junit XML format and let Jenkins chew on that. So a created an XML template like so:

%testsuite{:name => "smoketests", :failures => @tests.select { |x| x.status == "FAIL" }.count, :tests => @tests.count, :skipped => 0}
  - @tests.each do |test|
    %testcase{:name => test.description}
      - if test.status != "OK"
        %failure{:message => test.message}

And when fetching the url http://server/smoketest.xml you get the following output:

<testsuite failures="0" name="smoketests" skipped="0" tests="2">
<testcase name="MongoDB test connection"></testcase>
<testcase name="Test"></testcase>
</testsuite>

(It handles failures too).

The following small shell script is used to check the status:

#!/bin/bash

if [ -z "$1" ]
then
    echo "usage: smoketest.sh &lt;URL&gt;"
    exit 1
fi

status=`curl --silent --head $1 | head -1 | cut -f 2 -d' '`

if [ $status != "200" ]
then
    echo "status was other than 200: was $status"
    exit 1
fi

if [ ! -e "reports" ]
then
  mkdir "reports"
fi

#put the xml version of the page into a file
curl --silent $1 &gt; reports/smoke.xml

The shell scripts stores the file which is then read by Jenkins. Works like a charm!

Stay tuned for the third article on build pipelines where I explore some future possibilities when using a build pipeline.

Continuous Delivery I: Pipeline Plugin and a Ruby Project

2011-04-04T17:57:44+02:00

I have been reading the excellent book Continuous Delivery by Jez Humble and Dave Farley and been dying to try out some of it in practice. Since work these days is filled with big not-so-easy-to-change java legacy systems, a small greenfield ruby project was the way to go.

This article ran long, so it is split into three parts:

part I (this part): Jenkins setup and project setup.
part II: Ruby Smoketest Implementation
part III: Further Build Steps

My aim was to deploy the Ruby on Rails project to heroku which is a nice and easy-to-use cloudy service for hosting ruby apps. Elegant and easy on both web interface and command-line tools.

The Project Setup

The project I just started is a Rails3 app using bundler and running in ruby 1.9.2 using rvm. This, it turns out, brings some special considerations to the mix. Just for kicks and because I start to dislike SQL databases more and more, especially for the cases where you don’t really need it, I use MongoDB and Mongoid as the ORM. MongoDB is supported by Heroku so all’s good there.

One gotcha it is easy to run into with Heroku and a rails app is that you should have your rails app be at the top level of the git repo. It is supported to run with a rails app in a sub directory, as I did, and you can configure it to work via the config.ru file, but bundler does not seem to run well on heroku when you do that. So keep the rails app on the top level if you can.

The projects uses rspec for unit tests and some rails-specific integration/functional tests, and cucumber for end-to-end acceptance style tests. More on that when it comes to the deployment pipeline and jenkins project setup.

Jenkins Setup

I belong to the school that dictates that the first thing you should do with a project is to deploy it and then setup CI for it and make that deploy for you. The rake plugin for Jenkins supports rvm so we can use rake tasks directly on the project configuration.

First of all, I needed cucumber and rspec to output junit format test results to make pretty graphs and record status in Jenkins. Cucumber has a handy --format junit, but rspec does not. Luckily the gem ci_reporter does the trick for us. But some customs rake tasks are in order. This is the file lib/tasks/jenkins.rake:

namespace :jenkins do
  def cucumber_report_path
    "reports/features/"
  end

  def rspec_report_path
    "reports/rspec/"
  end

  Cucumber::Rake::Task.new({'cucumber'  => ['db:test:prepare']}) do |t|
    t.cucumber_opts = %{--profile default  --format junit --out #{cucumber_report_path}}
  end

  task :cukes_report_setup do
    rm_rf cucumber_report_path
    mkdir_p cucumber_report_path
  end

  task :spec_report_setup do
    rm_rf rspec_report_path
    mkdir_p rspec_report_path
  end


  task :ci => [:report_setup, 'jenkins:setup:rspec', 'rake:spec', 'cucumber']

  task :unit => [:spec_report_setup, "jenkins:setup:rspec", 'rake:spec:lib']

  task :functional => [:cukes_report_setup,
                       :spec_report_setup,
                       "jenkins:setup:rspec",
                       'rake:spec',
                       'cucumber', ]

  namespace :setup do
    task :pre_ci do
      ENV["CI_REPORTS"] = rspec_report_path
      gem 'ci_reporter'
      require 'ci/reporter/rake/rspec'
    end
    task :rspec => [:pre_ci, "ci:setup:rspec"]
  end

end

It does some setup of of report output directories, some setup of ci_reporter in the :setup task and makes sure rspec tasks does some setup first. The tasks called from jenkins are jenkins:unit for the first build step and jenkins:functional for the second step. The jenkins:ci task is an all in one if that’s your bag. The main point is that the first step runs unit test, hence rake:spec:lib only, and the second step runs all tests and metrics. Ideally if the metric_fu plugin in Jenkins had supported bundler well, I would like to run some metrics in the first step and fail the build if values where below or above a limit. But that does not seem to fly easily.

I used a small shell script to deploy to Heroku and the test server instead of using the git publisher plugin. This is to make the deployment build step fail the build. The git publisher plugin is run as a post build action and this won’t fail the build. Likewise, the smoke test script needs to run as a build step since the recording of test results is done post build.

And here’s where the ideals of Continuous Delivery (CD) kicks in. The book advocates doing the build in several steps and the central point is fast feedback. The first step is compile, fast-running unit tests, metrics and building of artifacts (if you’re on a platform that does that). And if all that passes should you go on to step two using the same artifacts created in step one, possibly aided by a repository manager of some sort to take care of the files in between steps.

If the first step fails you get unit level and metric violations feedback. If the second fails, you get system level functionality feedback. If later steps involving e.g. performance testing and manual testing you get feedback on those. But you don’t go to that step with a bad build from step one.

The build pipeline plugin basically gives you a view and a(nother) way of specifying upstream/downstream projects. The view looks like this after it is configured according to this article with the four build steps in place.

The build pipeline plugin is in version 1.0.0 and has some major drawbacks. The main one being that it doesn’t stop the pipeline when one step fails. This is a bug and it has been reported on the issue tracker already. I would guess this is fixed in the next release. Another is that it does not seem to be easy to pass params from one build step to the next. There has been some mention of using the parameterized build plugin for that, but haven’t had a chance to test it yet and I don’t know how well the two plugins play well together.

The Build

So in my case, the first step runs unit tests (spec:lib) and the second runs all spec, cucumber features and metric_fu for metrics. The third step deploys to my local test server and runs an automated smoke test. See part II for details on the implementation of that. The output is in junit format, so Jenkins records it and outputs it along with the other tests. If that test brake, the build in step 3 brakes too. I would like to rollback to the previous version on build failure, but I’m not there yet. This article on using capistrano looks interesting in that regard

The fourth step deploys to heroku via a script and runs the same smoke test as the step before.

Benefits and Ending Thoughts

The biggest benefit of using a pipeline is more fine grained feedback on the build process. The compile step is fast and fails almost immediately. The separate steps for integration tests and deployment to test server and heroku gives more accurate feedback on those steps too. I mean, combining deployment to test and production in a single step would be nonsensical. And you can do the deployment to production (or test) a manual one, which the plugin supports. For instance only after manual testing in the test environment. Combined with Jenkins’ matrix style security model you can grant access to certain users for running that build step. There are a lot of possibilities and I will be exploring more of them in part three of the series. Part two is implementation of the smoke tests in ruby.

Rewriting Fixtures in Migrations?

2011-02-25T13:29:31+01:00

I had an idea in a twitter discussion with @olecr about automating different aspects of the development process, coupled with and in-house discussion at work on maintaining database test data for integration tests, which can be a real pain. How about hooking into migrations to rewrite/update fixture data when migrating database schema?

In ruby one approach to test data, is to use fixtures, often written in yaml, to define test data sets, which gets inserted into the test database before running tests. But you still have to evolve the fixtures when the schema evolves. And the more often it changes, the more of a maintenance nightmare you got on your hands. Other approaches can be to have object factories that create data in the database, or SQL script that gen run (which is what yaml fixtures are, only in a more sane and not-so-verbose format). But the maintenance is there no matter what the approach.

And then the idea struck me: if you use fixtures for inserting test data into the database before running tests (or between tests) and migrations to modify the schema of the database as you develop the application, couldn’t you create a clever hack that modifies the fixtures to match the migrations? All in one operation? And you can minimize the manual maintenance to the stuff that is really hard to automate and to sanity check to results.

Yaml is machine-readable format, so one approach could be to read the migrations, find the diff for the migration in question, read the yaml fixtures, modify them and write them out again. In ruby/rake this could be some custom rake tasks that squeezes in with the migrations or something to that effect.

I would think this could work for things like deleting columns from a table (delete that key in the yaml file), adding new columns (add columns in fixture with dummy value according to data type) and column and table renaming. Some user edits would be necessary nonetheless.

What do you think? Viable idea or just plain stupid?

The logical next step of rewriting the test code using these fixtures to match the new schema, well I leave that as an exercise to the reader… :-)

Certified Keyboard Driven Developer

2010-12-22T14:55:47+01:00

In the latest episode of The Agile Toolkit podcast in an interview with Dave Thomas about, well just about everything, they talk jokingly about the notion of a certification for keyboard shortcuts and I thought, wow, that is just a fantastic idea! This would be a really useful certification as opposed to the many completely or nearly completely useless certifications out there. None mentioned none forgotten.

Shortcuts makes you more productive all over the place. In your IDE, on the command line and all sorts of other tools. Not having to remove your hand from the keyboard in order to use the mouse saves you time and concentration. You start out by typing, move your one hand to use the mouse, possibly moving your eyes to see the mouse, then moving the mouse pointer to the menu item or button you want, clicking it and then moving your hand back to the keyboard. When the shortcut is second nature you type, enter shortcut and continue typing. No lateral movement of neither hand(s) nor eyes (except to see result of command, but that is true of not using the shortcut too).

The effect is more flow in your work. Most developer know when they are in the zone and want to stay there to keep the code flowing. Every little distraction should be avoided to maintain the state of flow. Agreed, using the mouse is not a big distraction from the flow, but everything helps, right?

If you get repetitive stress injuries from using the mouse, using the keyboard could help make them go away. Just make sure you get a good quality keyboard, adjust your desk and chair to the correct height and take frequent short brakes.

So How Should We Do This in Practice?

There is the trouble of all the different sets of keyboard shortcuts for different tools, of course. Mac users have an advantage here since the platform streamlines a bunch of shortcuts. Tools like IntelliJ IDEA and Resharper which lets you choose between different shortcut schemes (us Emacs users love the Emacs shortcut scheme, at least I do, but sadly it doesn’t quite fit in IntelliJ on the Mac) and you can use the same one in several tools.

So I guess we need different certifications for different tools, that would make sense. You could then embroider your CV with the title “Certified Mac Keyboard Developer”, Certified Microsoft 2010 Visual Studio Keyboard Core Foundation User” (the Microsoft marketing division can get a say in this, I don’t care), or Certificed Emacs Hacker.

And the actual test? Give the candidate a list of tasks to to (normal stuff every developer needs like opening files, navigating project structure, running unit test, build/package/deploy, the most common refactoring shortcuts, snippets, code formatting, line navigation and editing and other useful stuff. And just simply disable the mouse or remove it altogether. If they’re done within the time frame, they’re certified! I see a million dollars coming my way with this scheme.

Command Line Users Have the Edge

Users accustomed to using the command line for all sorts of tasks are used to remembering commands and shortcuts to greater extent than “normal” windows users (I use the command line in windows too, but not the crummy built-in cmd.exe) and could get certified more easily. The history navigation in the unix shell, combined with aliases and piping is awesome for productivity.

Training For the Exam

There are some techniques for training for this kind of certification. First there are cheat sheets that should decorate your desktop or cubicle wall and http://www.cheat-sheets.org/ has a huge collection. Some java IDEs have plugins that will remind you of the fact that a shortcut exists if you select a menu item with the mouse. Some can even be configured to disable the menu item if you won’t take the hint. Lastly just time yourself doing something you do every day and avoid the mouse altogether and see the effects for yourself.

Disabling the mouse for some period of time could also be tried. It hurts in the beginning, but you should quickly get over it. Jon Galloway also has some good suggestions in his blog post on mouseless computing.

I for one try to use the mouse as little a possible.

Log4net, Windows Event Log and IIS Applications

2010-12-17T07:56:37+01:00

I am doing a C# project at the moment where we are using log4net to log to the Windows event log. From the windows service this is straight forward but it turns out when you are trying to do this from an application running in IIS, a WCF service in our case, this is surprisingly hard. So I thought I’d document this one for later reference.

Those Pesky Permissions

Just duplicating the config for a normal windows application doing perfectly good event logging isn’t going to work as you will quickly discover. A good way to debug this is to set up a FileAppender to log to a temporary directory somewhere. Log lines ends up there when the config is read OK, but not in the event log. It’s a permission thing. The user running IIS probably has got privileges to log to the event log, but not create the event source initially. Nor should it, since running IIS as a privileged user is security disaster waiting to happen. Don’t do it.

The config used for logging in this example is this:

<log4net debug="false">
  <root>
    <priority value="INFO"/>
    <appender-ref ref="EventLogAppender"/>
  </root>
  <appender name="EventLogAppender" type="log4net.Appender.EventLogAppender">
    <applicationName value="My Application"/>
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline"/>
    </layout>
  </appender>

</log4net>

The Solution

Step one is to create the event source by hand as a normal user. You can do this via the following code in a C# app.

EventLog.CreateEventSource(source, "My Application")

Or do it from the command line using the eventcreate program

c:\> eventcreate /ID 1 /L APPLICATION /T INFORMATION /SO "My Application" /D "Dummy log message"

This gives you a “dummy log message” in the application log. And you should be good to go. But not so fast. Eventcreate requires an EventID between 1 and 1000. Default logging from log4net for some reason uses event id 0, which will give you an event log error message in the application log. It contains your log message but it looks like a mess. So how to persuade log4net to use event id 1, which we used when creating the event source?

There are two ways of doing this. One is to use the log4net extension log4net.Ext.EventID and configure it there. If that isn’t your cup of tea, you can quite easily, although a bit hackish, do it directly in your code before any log messages are sent with this piece of code:

log4net.ThreadContext.Properties["EventID"] = 1;

For instance in a constructor or other suitable initialization code.

Talking at Smidig2010

2010-10-29T15:18:24+02:00

I’m super excited that I got one (so far, they normally only accept one per person) of my abstracts for the Norwegian Agile conference Smidig 2010 in Oslo 16. and 17. of November. This is my first proper conference talk even though I have been giving talks in other context to large audiences in the past.

The talk title is hard to translate but the topic is on the agility of NoSQL databases in general and document databases in particular and I will be contrasting old school relational databases to new school document databases in areas such as management, testing, development friction etc.

And the conference in itself is super cool with three tracks of four 10-minute lightning talks before lunch and open spaces crowd sourced by the participants after lunch, both days. I attended previous two years and it’s a blast.

All Your Detection Belongs to Modernizr

2010-09-22T12:19:00+02:00

I’ll hopefully be doing some HTML5 work soon and I came across the Modernizr project, which is very cool for doing feature detection as opposed to browser detection. It does all the heavy (well, not so heavy but tedious) lifting when it comes to detecting if the browser visiting your site supports all the fancy schmancy new features from HTML5 and CSS3. Like websockets, CSS 2D and 3D transforms, web workers, canvas, offline storage, browser databases, video and audio elements and much more awesomeness. Check the link for the complete list. And it’s really easy to use too. The nice thing about it is that it does not accomplish this through browser detection, which is very unreliable, but rather through checking for the feature directly by trying to call API functions or creating stuff.

Let's detect!

Load it like this:

<script src="modernizr-1.5.min.js" type="text/javascript" charset="utf-8"></script>

and the next time you need to know it some feature is supported, check the automagically created Modernizr object and see if that property is true or false:

if (Modernizr.canvas) {
   var c = document.createElement('canvas');
}

As a nice bonus it assigns css classes to the html element so you can do different styling based on whether a feature is available or not.

/* In your CSS: */
.no-audio #audio {
   display: none; /* Don't show Audio options */
}
.audio #audio button {
   /* Style the Play and Pause buttons nicely */
}

Twitter Data and Neo4j-rest in Pure Javascript

2010-08-11T00:00:00+02:00

The Stage

Some free time prompted me to test out the REST interface for the most excellent graph database Neo4j. And to gives those rusty javascript skills a refresh, I’d thought I’d to it in pure javascript. To help see what is going on, I visualize the graph using JIT and use jQuery under the hood to do the rest calls and the DOM scripting more efficiently.

I start out simple by just fetching and using data accessible in the public twitter API. The model I choose was to query twitter for friends and followers of a particular user, store it in neo4j and plot it as a directed graph on a HTML5 canvas element using JIT. Next time around the graph can be fetched directly from neo4j without going to twitter. Everything is in javascript and the only html involved is a input field for the username to search for and a HTML5 canvas element for drawing the graph using JIT.

Implementation

Browser-side javascript can be hard to develop and debug. I chose JSpec as my testing framework for the day, which has a nice DSL for writing RSpec-like tests and has support for running in all browser through a command-line tool. It Also has rhino support.

JSpec tests are cool. JSpec comes bundled in a ruby gem which you install and after that you get a command line runner for firing browsers, or rhino, and looking up syntax and doing other neat stuff to your test suite. It looks like a good project with steady releases and progress. Here’s an example of how a test looks with the DSL in action:

describe 'twitter'

  before
    network = {}
    neo = {}
    stub(neo, 'node_exists').and_return( { status: true,
                                           "data": {self:"http://localhost:9999/node/1"} })

    stub(neo, 'get_node').and_return( {id: 123} )

    $ = { };
    stub($, 'ajax').and_return(true);
  end

  before_each
    stub(network, 'log').and_return( { } )
    twitter = Twitter({"network": network, "neo": neo})
    twitter.get_request = function(url, name, callback){
       callback.call(twitter, [{ id: 1}, { id: 2}]);
    };
    data = { }
  end

  describe 'lookup()'

    it 'should call callback on fetch for valid user'
      stub(neo, 'get_node').and_return( {id: 123} )

      twitter.lookup("knuthaug", function(in_data){
                   data = in_data;
               });
      data.id.should.eql 123
   end
end

Patterns

Javascript gives you plenty of ammunition to shoot yourself in the foot, and I decided to follow a pattern from Douglas Crockfords book “Javascript, The Good Parts” and implement my objects with what we can call the “that” pattern. Instead of defining the object like this:

var Node = function(spec){

  this.foo = function(){
           var status = "";
           that.bar(status);
      }
   };
}


//and call it like this
var myNode = new Node( {id: 1} );

we write it like this.

var Node = function(spec){

   var that = spec;

   that.foo = function(){
           var status = "";
           that.bar(status);
      }
   };

 return that;
}

//and call it like so:

var myNode = Node( {id: 1} );

This encapsulates all variables and functions into a variable which is returned to the caller and if we were to implement private functions on the object, something javascript normally doesn’t let you do, we define them on “this” instead of “that”. The spec argument to the constructor is just an easy way of passing in default values to the object being created, or giving it a prototype to inherit from. the “new” keyword normally does this for you and with this technique you have to do it yourself to get a prototype chain set up.

The point is that I’m not sure I’m fond of this technique since I find the code to get a bit muddled and the “that” keyword has a strange ring to it. Next time round I’ll probably stick to the normal way of setting up objects.

I ended up with and object encapsulating the graph, one for the node, one wrapping the twitter API and lastly one wrapping neo4j-rest for storing and retrieving data. The code, if you’re interested can be found on github (very unpolished and probably won’t ever be polished so take it with a grain of salt or two).

Gephi

Gephi is an open source tool for visualizing a graph using Open GL for fast and fancy rendering capabilities. A version released a short time ago (get it here) has support for connecting directly to a (sadly, not running) instance of Neo4J and import the whole graph.

For this little pet project the main focus for using Gephi was to test it out and see how it works, but the potential is huge for graphs with more data in them. Gephi can do all sorts of graph manipulation and analysis and showing the graph according to values stored in the nodes etc. In the twitter graph I stored the number of followers in each of my followers as a number and Gephi could easily be configured to draw the node size relative to that, color edges based on data and much more. And all in nice openGL smooth graphics performance.

All in all a great little adventure. And you should learn proper javascript :-)

Comparing MongoDB java frameworks

2010-07-28T11:13:34+02:00

I while ago I gave a talk at a Oslo JavaBin meeting on NoSQL and in the research for that I wounded up writing some sample code for three different frameworks for using MongoDB from Java. The code is available on GitHub. The three are the 10Gen mongo driver (and not really a framework), morphia and mungbean.

The Whole Dynamic Thing

MongoDB is a schema-free document database and does not impose any restriction on what you can store in a collection. There is nothing stopping you from puttting entirely different objects into the same collection and thus nothing stopping you from omitting fields in two different objects of the same kind either. So how does this play with Javas strong static typing compared to a dynamic language like Ruby?

Dude, Where’s My ORM?

Developers used to relational databases have been using ORMs for a long time to abstract away the dirty details of dealing with the database and separating the domain from the persistence. So, are any of the three frameworks an ORM for MongoDB? Well, no. But a more interesting question is: Do I need one? MongoDB is a very different beast than oracle/mysql/MSsql/PostgreSQL and different beasts need different handling. You need an abstraction model for MongoDB, not necessarily the ORM abstraction model.

MongoDB Java Driver

10Gen java driver basically gives you to options when storing data in MongoDB: Either subclass BasicDBObject which is the general database object, or implement the DBObject interface. Both approaches gives you the interface of Map from the standard java library and lets you put and get keys and values. The drawback is that there are a lot of methods in this interface and it will soon become tedious to implement them all for different domain objects. This it the DBObject interface methods:

    public Object put(String s, Object o) { }

    public void putAll(DBObject dbObject) { }

    public void putAll(Map map) { }

    public Object get(String s) { }

    public Map toMap() { }

    public Object removeField(String s) { }

    public boolean containsKey(String s) { }

    public boolean containsField(String s) { }

    public Set&lt;String&gt; keySet() { }

    public void markAsPartialObject() { }

    public boolean isPartialObject() {}

So I’d prefer to subclass BasicDBObject instead or wrap it.

So how does code using it look like? Assume the very minimal domain objects Person and Address

package no.kh.mongo.direct;
import com.mongodb.BasicDBObject;

public class Person extends BasicDBObject {

    public Person () { }

    public Person(String fullName, Address newAddress) {
        put("name", fullName);
        put("address", newAddress);

    }
    
}

package no.kh.mongo.direct;
import com.mongodb.BasicDBObject;

public class Address extends BasicDBObject {

   public Address() {
  }

  public Address(String streetName, String postalCode, String place, String country) {
    put("street", streetName);
    put("postalcode", postalCode);
    put("place", place);
    put("country", country);
  }

  public String place(){
    return (String) get("place");
  }
   
}

Using this class is done like this in the form of functional/integration tests, since they touch the database but written in Junit:

package no.kh.mongo.direct;

import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.Mongo;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.net.UnknownHostException;
import static junit.framework.Assert.assertEquals;
import static junit.framework.Assert.assertNotNull;
import static org.junit.Assert.assertNull;

public class PersonPersistance {

    DB testDb;
    DBCollection persons;

    @Before
    public void setUp() throws UnknownHostException {
       Mongo m = new Mongo( "127.0.0.1" , 27017 );

       testDb = m.getDB( "test" );
       persons = testDb.getCollection("persons");
       persons.setObjectClass(Person.class);
    }

    @Test
    public void insertPersonSavesPersonToDatabase () {

        Person test = new Person("Knut Haugen",
                                 new Address("Josefines gate", "0401",
                                             "Oslo", "Norge"));
        persons.insert(test);
        assertNotNull(test.get("_id"));

    }


    @Test
    public void personWithDocumentNotMatchingObject(){
      BasicDBObject tmp = new BasicDBObject();
      tmp.append("foo", "value");
      persons.insert(tmp);

      Person test2 = (Person) persons.findOne();
      assertEquals(test2.get("foo"), "value");
      assertNull(test2.get("name"));
    }

    @Test
    public void retrievePersonFromDatabase(){
        Person test = new Person("Knut Haugen",
                                  new Address("Josefines gate", "0401",
                                              "oslo", "Norge"));
        persons.insert(test);

        Person test2 = (Person) persons.findOne();
        assertEquals(test.get("name"), test2.get("name"));
        assertEquals( ((Address) test.get("address")).place(), "oslo");
    }

    @After
    public void tearDown(){
        persons.drop();
    }

}

Notice the call to setObjectClass() on the collection object to get “type-safe” operations even though you still have to cast the return value from findOne() to get you precious object back. Other than that it is pretty straight forward. Call insert() on the collection to insert an object, findOne() or any other query method to retrieve it. But to the bottom line is this driver really begs for some more abstraction when you’re beyond toy samples. The positive effect is more direct access to the data which is often the way it’s done in dynamic languages. But does that suit Java? I’m not sure but tend to think no. And how about those nulls? Well, if one object doesn’t store a value in a field, the field is null when returned from the database, as you can see from the test personWithDocumentNotMatchingObject.

Morphia

Morphia is a (according to the blurb) a light-weight type-safe mapper for MongoDB providing DAO<T, V> and Datastore abstractions. It takes the annotation approach. Here’s our Person and Address classes for Morphia:

package no.kh.mongo.morphia;
import com.google.code.morphia.annotations.Embedded;
import com.google.code.morphia.annotations.Entity;
import com.google.code.morphia.annotations.Id;

@Entity
public class Person {

    @Id
    private String id;
    private String name;

    @Embedded
    private Address address;

  public Person() {
        address = new Address("", "", "", "");
    }

    public Person(Address newAddress){
        address = newAddress;
    }

    public String getId() {
        return id;
    }


    public String getName() {
        return name;
    }

    public void setId(String newId) {
        id = newId;
    }

    public void setName(String newName) {
        name = newName;
    }

    public void setAddress(Address newAddress) {
        address = newAddress;
    }

}

package no.kh.mongo.morphia;
import com.google.code.morphia.annotations.Embedded;

@Embedded
public class Address {

  private String streetName;
  private String postalCode;
  private String place;
  private String country;

  public Address() {
  }

  public Address(String streetName, String postalCode, String place, String country) {
    this.streetName = streetName;
    this.postalCode = postalCode;
    this.place = place;
    this.country = country;
  }

}

Where Address is annotated as an embedded object. This is the same approach taken by the ruby Mongo_Mapper with its MongoMapper::Document and MongoMapper::EmbeddedDocument classes.

package no.kh.mongo.morphia;

import com.google.code.morphia.Morphia;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.Mongo;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.net.UnknownHostException;
import static junit.framework.Assert.assertNotNull;
import static junit.framework.Assert.assertNull;

public class PersistanceThroughMorphia {

    Morphia morph;
    Mongo mongo;
    DBCollection persons;

    @Before
    public void setUp() throws UnknownHostException {
       morph = new Morphia();
       mongo = new Mongo("127.0.0.1", 27017);
       // This is where we map Persons and addresses
       // But shouldn't the annotation be able to handle that?
       morph.map(Person.class).map(Address.class);
       DB testDb = mongo.getDB( "test" );
       persons = testDb.getCollection("persons");
    }

    @Test
    public void storePersonThroughMorphiaMapping () {

        Person test = new Person(new Address("Josefines gate", "0401", "Oslo", "Norge") );
        test.setName("Knut Haugen");

        persons.save(morph.toDBObject(test));
        Person test2 = morph.fromDBObject(Person.class, persons.findOne());
        assertNotNull(test2.getId());

    }


    @Test
    public void personMissingField () {

        Person test = new Person(new Address("Josefines gate", "0401", "Oslo", "Norge"));
 
        persons.save(morph.toDBObject(test));
        Person test2 = morph.fromDBObject(Person.class, persons.findOne());
        assertNull(test2.getName());

    }
    

    @After
    public void tearDown(){
        persons.drop();
    }

}

It seems like something out of the department of redundancy department that you annotate the document and the embedded document and then have to specify the relationship between them in a method call. My first reaction was that that would have been cleaner if the relationship could be specified in the annotation too. The calls to morph.toDBObject() and morph.fromDBObject() breaks up an otherwise elegant solution. It also introduces some more code and it basically wraps up a cast. That could have been a lot cleaner.

Mungbean

Mungbean is our last contestant and represents a third way of doing the mapping. It wraps up everything you need for accessing MongoDB with generic collection classes and introduces a DSL for querys and the like. There’s also a clojure version if that is more like your poison. The domain classes with mungbean:

package no.kh.mongo.mungbean;

public class Address {

  private String streetName;
  private String postalCode;
  private String place;
  private String country;

  public Address() {
  }

  public Address(String streetName, String postalCode, String place, String country) {
    this.streetName = streetName;
    this.postalCode = postalCode;
    this.place = place;
    this.country = country;
  }

  public String place(){
    return place;
  }

}

Nothing special here, no imports and no annotations. (Almost the) Same with Person except for the import and field of type ObjectID which handles the object ids generated by mongo on insert:

package no.kh.mongo.mungbean;
import mungbean.ObjectId;

public class Person {
  private String name;
  private ObjectId _id = new ObjectId();
  private String address;


  public Person(String name, String address){
    this.name = name;
    this.address = address;
  }

  public String getAddress(){
    return address;
  }

  public ObjectId getId(){
    return _id;
  }
}

Using it on the other hand, creates a very different looking code than the other two, thanks to the wrapper classes for mongodb connections, the generic collections and the query DSL:

package no.kh.mongo.mungbean;

import mungbean.DBCollection;
import mungbean.Database;
import mungbean.Mungbean;
import mungbean.Settings;
import mungbean.query.Query;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import static junit.framework.Assert.assertEquals;
import static junit.framework.Assert.assertNotNull;

public class mungbeanPersistence {

  private Database test;
  private Person testPerson;
  private DBCollection&lt;Person&gt; persons;


  @Before
  public void create() {
    test = new Mungbean(new Settings(), "localhost", 27017).openDatabase("test");
    testPerson = new Person("Knut Haugen", "josefines gate");
    persons = test.openCollection("persons", Person.class);

  }

  @Test
  public void storePerson(){
    persons.save(testPerson);
    assertEquals(persons.query(new Query().field("name").is("Knut Haugen")).size(), 1);

  }


  @Test
  public void personGetGeneratedIdAndAddress(){
    persons.save(testPerson);
    Person found = persons.query(new Query().field("name").is("Knut Haugen")).get(0);
    assertNotNull(found.getId());
    assertEquals(found.getAddress(), "josefines gate");

  }


  @After
  public void destroy() {
    test.dbAdmin().dropDatabase();
  }

}

The syntax is nice but perhaps a tad verbose for mye taste. I find the abstraction quite good, at least better than the other two. I also like the fact that there is almost no trace of the library in the domain classes and as such it is by far the best of the three.

The Verdict

It’s Mungbean, by a nose! Mainly because of the cleaner domain objects and the DSL. There is more code involved but I found it to be more elegant than the other approaches. I want to note that both morphia and mungbean are not immensely mature and done by any definition of the word and that has to come into consideration when using them. And it may be that a wordy statically typed language like Java has a bit of friction with a very dynamic database backend like MongoDB. I don’t know, but I’ll be looking into ruby drivers in the future and we’ll see.

I haven’t looked into the different code generation frameworks for mongo, namely Sculptor and GuicyData which take a different approach to accessing MongoDB. That’s for another time and another post.

A Book Overflow

2010-04-30T15:53:44+02:00

I am a habitual book buyer and I kinda like the paper kind which is evident if you see my office shelf or my living room. So once again I bought more books than I can read in the time I have left from living my normal life.

Growing software

Growing Object-oriented Software Guided by Tests By Steve Freeman and Nat Pryce. This is a book I’ve only heard positive things about and the TOC seem to deliver on that. It offers a lot of details on the how and more important the why of TDD/BDD and testing in relation to development. I am so looking forward to this one.

Being Productive

The Productive Programmer by Neal Ford. I’m a productivity junkie just like Neal Ford and some of the tricks (and the first part of the book is a long list of tools and tips) are already in my tool belt. But there’s always more to learn, all in the spirit of kaizen. He throws in some good advice on software design also, all in the spirit of being productive.

The Art of Agile

The Art of Agile Software Development by James Shore and Chromatic. Another one I have heard a lot about and all positive. By the first look it seems to cover the state of agile software development today and touches on some lean and TPS-inspired concepts too. And you can’t really navigate around that in today’s agile landscape.

Effective Java

Effective Java second edition by Joshua Block. I’ll be writing more Java in the coming months as I’ll be changing jobs so this is in order to brush up on the old java skills. Also a book that is widely regarded as a must.

The Legacy You Inherit

Working Effectively with Legacy Code by Michael Feathers. How do you test legacy apps? Highly recommended by a colleague who recently inherited a largish legacy codebase with only a tauntingly small amount of tests. What’s the definition of legacy code? Code without tests :-) Even if you write it today…

Cooking with Mongodb and Solr

2010-04-27T13:46:28+02:00

I’ve recently changed storage backend and search backend for a small web project and it has been a real blast. What follows is an overview of the reasons for the change, what the change actually was and the relative amount of joy involved.

The Old System

System was built using PHP/Apache2 and MySQL and it covers a very simple domain with only a single object (Person, sort of) and simple data records for several years.

No writing through the web interface, only search and lookup.
Batch updates with between 4 and 5 million records in each update, 4-5 years history, so total 16-20 million records
“Search” through very simple text indexes on the relevant columns in MySQL.
InnoDB backend
File is transformed to LOAD DATA INFILE format and fed into MySQL with manual delete of the set for that year beforehand.

The Pain Points

Batch update with 4 million rows averages (on prod machine: dual core 3GHz 4GB Ram, roughly 1GB set aside for MySQL) taking 4-5 hours hours with index updates being the main culprit. This could be done as a check-the-record-and-update-if-changed but that would also require a lot of queries and updates to the database.
Queries with wildcards are dead slow when hitting outside the query cache.
Not really advanced search as such.

The Plan

Replacing MySQL with MongoDB as there is no actual relations needed and everything fits in one collection of documents
Replacing MySQL indexes with Apache Solr for consolidating search across several other systems. And speed.
Use the PECL extensions for both MongoDB and Solr.

MongoDB is a document database storing documents in binary json form, written in C++.

Solr is built on top of the java version of Lucene and does indexing over HTTP and runs happily in tomcat, jetty or most other servlet engines.

The Implementation

Names of domain objects are changed to protect the guilty - and the domain.

Both Solr and MongoDB are fast and easy to work with. There is very little in your way when it comes to just doing what you want and solving the problems in a straight-forward manner. Some examples:

The MongoDB “upsert” feature saves you some round-trips to the database. Normally if you want to update an existing record if it’s there or inserting it if it’s not you need to query first and then insert a new one when not found. If you just want to update/add data to part of the object, it complicates the matter further. With mongodb you can call update with a special parameter in the data array and the rest is handled server-side.
Solr has the default behavior of updating instead of complaining when you send a document with a primary key field that already exists in the index.
Solr does everything over HTTP and you get easy-to-read xml message back as responses. This is also handy when you need to debug what data is sent over the wire.

I created a very thin layer between mongodb and the domain, with an insert() method (which as we will see, also handles updates) that take a DataRecord (read from the file) as an argument.

public function insert(DataRecord $record) {
       $this->collection->update(array('id' => $record->id() ), 
           array('$set' => array(
               'list.' . $record->year() => $record->getDetails()->toArray())), 
           array('upsert' => true));
}

This will insert a document in the collection if it’s not there. When it is there, it will add an element to the (nested) ‘list’ element with the value of $record->year() as key. The value will be the value of $record->getDetails(). The toArray() call is there because the mongo driver expects arrays to store. The super cool part is that if the key exists, it will just be updated with the data from the details object. Read more on the details of the MongoDB update options.

For indexing the document in Solr, I added a similarly thin wrapper for the SolrClient object with an index() method. This method takes a SolrInputDocument as an argument. I chose to delegate to the domain object to decide what should be indexed and thus create the index document object but the responsibilities could easily have switched around. The finer point is that when indexing you have to read the complete object from the database in order to get all data. The DataRecord that was read from file and stored with upsert may just have been part of the picture. Reading back the updated object incurs a performance penalty that wasn’t present in the old system. It was also a consequence of structuring the data as a collection of person objects in Mongodb, rather than a long list of records in the old version. This maps better to the domain.

public function index(SolrInputDocument $document) {
        $response = $this->solr->addDocument($document);

        if($this->pendingDocuments++ == $this->commitInterval) {
            $this->commit();
            $this->pendingDocuments = 0;
        }

        return $response->getResponse();

    }

Commit on every Solr document makes indexing very slow. Small tests indicated 3 minutes for indexing 5000 documents with commit on every submit and 15 seconds with one commit every 2000 document (and at the end of course). The code above commits every $commitInterval(10000 default) to speed things up a bit. Note also that the commit() and optimize() calls for Solr may time out as they can take a long time to finish. Solr does not time out but rather the java application server you’re running times out. When this happens an exception is thrown in the php driver which has to be caught.

The Results

Platform is Ubuntu 9.10 server edition 64 bit and all timings from the shell are done with time on linux. MySQL times are the times reported from MySQL itself.

Time for batch insert/update

Commit interval for solr: 10000
update-logging for Solr turned off (default is very verbose)
nssize=1024 for Mongodb

System	Operation	Time
MySQL	initial import	16m 5.7s
MySQL	update (delete+insert)	3h 24m 56s (delete) + 40m 56s insert
Mongodb (no indexing)	initial import	12m 16s real, 10m 41s user
Mongodb+solr	initial import	78m 39s real, 25m 28s user.
Mongodb (no indexing)	update	13m 22s real, 11m 8s user.
Mongo+solr	update	69m 8s real, 19m 25 user

Space usage

No pre-allocation was done for MongoDB so it created the data files as needed. This means that the last was created at 2GB and very well may be almost empty. Mongo creates files in a doubling fashion from 64 MB to 2G like this: 64, 128, 256, 512, 1GB, 2GB.

System	Index	Data
MySQL initial import	717 MB	516 MB
MySQL initial import + 1 additional dataset	1.4 GB	1 GB
Mongo+Solr initial import	744 MB	1294 MB (3GB of datafiles)
Mongo+Solr initial + 1 additional dataset.	1538 MB	4348 MB

CPU usage

When importing to MySQL it more or less maxes on CPU for the entire import. When doing the import with a php script feeding data to mongodb and solr, the component using the most cpu is the php script splitting the file, creating objects and calling the mongodb and solr APIs.This takes up 35-40% CPU and around 10 MB of ram. Mongodb is using around 10% cpu (with 1.1GB of ram) and solr (tomcat, that is) is spending 30% and around 300MB of ram. Disks on these machines are virtualized, through Vmware, 15K SAS disks on an IBM S3200 storage array with dedicated GB LAN between blade center and storage. Disk IO seems to be the bottleneck here.

Space usage

Solr and mongodb seem to be a bit more sloppy with their space usage than MySQL but I guess this is the price to pay for some of the other benefits you get. See mongo faq on data files for info on how to see real space usage for databases and not just file sizes. In return for more storage space spent, you get much better search capabilities (and faster) and faster (although small improvement) query times against database.

Naming Classes and Interfaces

2010-04-16T08:45:58+02:00

The naming of interfaces and corresponding implementation classes in languages that use them (of which Java and C# is the most used I guess) is a subject of sometimes heated debate. A growing distaste of the "best practice" (at least on Microsofts part) of using a capital "I" on interfaces prompted this rundown and a round of pro et contra. My ultimate goal is readability of code but a slight predicament emerges when considering a team: if the majority of team members are used to one particular convention is that, in that particular context, more readable for that team than another approach, all other things being equal? Even though the name of the interface reads more like natural English? The jury's still out on this one.

The different patterns I’ve found:

Interfaces are prefixed with a capital I, where IReportGenerator is the interface and e.g. ReportGenerator is the implementation. Some argue for making the interface name read "I generate reports" and keep the I prefix, thus: IGenerateReports instead.
Ditching the capital I and appending "imp" or "impl" to the implementation. Like so: UserManager and UserManagerImpl. Some use uppercase or lowercase "c" as prefix for concrete classes too but I find that horribly ugly, to be frank.
Naming an interface for it's role and naming the implementing class(es) for what distinguishes it from (possibly) other implementations. E.g. UserManager for the interface and DatabaseUserManager or LDAPUserManager for the concrete implementations.
Naming abstract class with the word "Abstract" as a prefix to make a distinction for abstract and concrete classes. Possibly combined with other conventions for interfaces as well. I would argue the context often reveals an abstract class to the reader.

C# vs Java

In C# the extension of an abstract base class and the implementation of an interface uses the same syntax and which one it is isn't always easy to discern. A naming convention makes this a bit easier:

public class DatabaseUserManager : UserManager {

}

//versus

public class DatabaseUserManager : IUserManager {

}

In java the syntax is different:

public class LDAPUserManager implements UserManager {

}

//versus

public class LDAPUserManager extends UserManager {

}

The need for a prefix for readability is less needed in Java, compared to C#.

Good names trumps conventions?

Clean Code by Robert C. Martin argues for good names: good names for variables, good names for methods and good names for interfaces and classes. The name should reveal intention. Choosing good names takes time but saves more than it takes. (Clean Code, page 18). The book also mentions not using the "I" prefix and prefers to encode the implementation, if at all.

The question is: if the names are as good as they can be, do we need a prefix or suffix or other kinds of encoding to indicate the type? In my mind we don't. I find that the I hurts readability and the "impl" certainly doesn't look good to me. So how to choose good names?

Classes should have noun names or noun phrases, e.g. Customer, Policy or StreetAddress.
Use name from the problem domain. E.g. If you're dealing with a customer or client object. In a medical system, should it be called Patient? Or if it's dealing with social benefits maybe BenefitsReceiver is a better name? The domain influences the choice of names. In stock-broking perhaps the name FuturesHolder is name candidate for a special kind of customer? (note: I'm not very familiar with either of these domains so the examples may be somewhat off. You get the point: know your domain and find good names from it.).
Use names from the solution domain. If you use e.g. the Visitor pattern, make it a part of the name(s) so other programmers see it InsuranceCustomerVisitor is better than CustomerTraverser

So unless some team coding standard absolutely makes me (and that is, I'm afraid, likely) I will not in the future prefix my interfaces with I and not adorn my implementations with c, impl or any other stuff. Just meaningful, names from the domain. Easier to write and easier to read.

Lastly, I posted this question on twitter and @borud made the comment: "the only useful information conveyed by an I-prefix to interface types is information about the author of the code." That's a suitable end note I think :-)

Analysis of the NoSQL Landscape

2010-03-17T08:01:47+01:00

This is an overview of the current state of the NoSQL landscape. It’s getting large and somewhat unwieldy and there may be projects which have landed in the wrong category here. I have included object databases in the mix too. Seriously folks, some of you need to pick more google friendly project names. Here are the types and the players in each category. Background data is available in this Google docs spreadsheet.

Key/value stores

key/value in-memory: memcached, Repcached, Oracle Coherence, Infinispan, Websphere eXtreme scale, JBoss cache, Velocity, Terracotta Ehcache
Regular key/value stores: Keyspace, Amazon SimpleDB, Flare, Schema-free, RAMCloud, Twisted Storage (TSnoSQL), Redis, Tokyo Cabinet, Lightcloud, NMDB, Lux IO, Memcachedb, Actord, BerkeleyDB, Scalaris, GT.M, Mnesia, HamsterDB, Chordless
Eventually Consistent key/value stores: Amazon Dynamo, Voldemort, Dynomite, SubRecord, Mo8onDb, Dovetaildb, KAI

Column-oriented stores

Google BigTable, HBase, Cassandra, HyperTable, OpenNeptune, KDI, QBase

Document Databases

CouchDB, MongoDB, Apache JackRabbit, ThruDB, CloudKit, Perservere, Lotus Domino, Riak, Terrastore

Object Databases

ZODB, db40, Versant, Gemstone/s, Progress Objectstore

Graph Databases

Neo4j, VertexDB, Infogrid, Sones, Filament, Allegrograph, HyperGraphDB

Projects by Type

If we graph all the projects by type we get this view:

There are more key/value stores than the other types combined. Why is this? Are key/value stores that much easier to implement? I would at least guess that the first area where we see projects being abandoned and convergence of projects is this one. The important thing is the features users want, not the project themselves. There must be a lot of overlap here and a lot of projects that are slightly different and almost identical. On the other hand a lot of knowledge of these kinds of system is spread around and there is a good chance of innovation. The combination of the best technical features and API features will hopefully bubble to top and stay on.

License Breakdown

If we graph the projects in the list above by license chosen we get the following:

This shows a clear dominance for open source licenses over commercial ones. Some product have chosen a dual licensing model (neo4j and BerkelyDB). Quite a few are unknown which really means they are unable to communicate their license in a understandable manner or the project wasn’t really found on the web at all (see point about google friendly names).

Language Breakdown

Graphing the projects by implementation language we get the following:

Java takes the lead by with C and C++ following close behind. But is the prevalence of Java a result of the amount of Java knowledge spread around and the big Java usage in Open source, or is Java more suited than other languages to implement these kinds of systems? Interesting to note the number of Erlang implementations and also the fact that quite a few of the projects have implementations in more than one language. The ones with more than one implementation are mostly commercial ones.

Some ending questions:

Have we reach the maximum of projects that are sustainable now or will the ecosystem continue to grow even more?
Will more of them go commerical? Or will more choose the model with support as the income, like 10Gen has with MongoDB?
How does one choose the right one to use for a given project? This is an increasingly hard problem, at least for key/value stores.

References:

NoSQL presentation by Steve Yen
http://nosql-database.org/
http://nosql.mypopescu.com/
http://www.dbms2.com/2010/03/14/nosql-taxonomy/
http://blog.nahurst.com/visual-guide-to-nosql-systems
All the various project pages and product websites.

A Brief History of NoSQL

2010-03-16T14:45:16+01:00

NoSQL is getting a lot of traction and hype these days but in reality it’s not that new of a thing. I thought I’d trace the roots of NoSQL and see what I’d find. The name “NoSQL” was in fact first used by Carlo Strozzi in 1998 as the name of file-based database he was developing. Ironically it’s relational database just one without a SQL interface. As such it is not actually a part of the whole NoSQL movement we see today. The term re-surfaced in 2009 when Eric Evans used it to name the current surge in non-relational databases. It seems like the name has stuck for better or for worse. Note that not all projects are included in this post. See the post on analyzing the NoSQL landscape for a more complete listing.

1960s

MultiValue (aka PICK) databases are developed at TRW in 1965.
According to comment from Scott Jones M[umps] is developed at Mass General Hospital in 1966. It is a programming language that incorporates a hierarchical database with B+ tree storage.
IBM IMS, a hierarchical database, is developed with Rockwell and Caterpillar for the Apollo space program in 1966.

##1970s

InterSystems develops the ISM product family succeeded by the Open M product, all M[umps] implementations. See comment from Scott Jones below.
M[umps] is approved as a ANSI standard language in 1977.
in 1979 Ken Thompson creates DBM which is released by AT&T. At it’s core it is a file-based hash.

1980’s

Several successors to DBM spring into life.

TDBM supporting atomic transactions
NDBM was the Berkeley version of DBM supporting having multiple databases open at the same time.
SDBM - another clone of DBM mainly for licensing reasons.
GT.M is the first version of a key-value store with focus on high performance transaction processing. It is open sourced in 2000.
BerkeleyDB is created at Berkeley in the transition from 4.3BSD to 4.4BSD. Sleepycat software is started as a company in 1996 when Netscape needed new features for BerkeleyDB. Later acquired by Oracle which still sell and maintain BerkeleyDB.
Lotus Notes or rather the server part, Lotus Domino, which really is a document database has it’s initial release in 1989, now sold by IBM. It has evolved a lot from the early versions and is now a full office and collaboration suite.

1990’s

GDBM is the Gnu project clone of DBM
Mnesia is developed by Ericsson as a soft real-time database to be used in telecom. It is relational in nature but does not use SQL as query language but rather Erlang itself.
InterSystems Caché launched in 1997 and is a hybrid so-called post-relational database. It has object interfaces, SQL, PICK/MultiValue and direct manipulation of data structures. It is a M[umps] implementation. See Scott Jones comment below for more on the history of InterSystems
Metakit is started in 1997 and is probably the first document oriented database. Supports smaller datasets than the ones in vogue nowadays.

2000-2005

This is were the NoSQL train really picks up some momentum and a lot is starting to happen.

Graph database Neo4j is started in 2000.
db4o an object database for java and .net is started in 2000
QDBM is a re-implementation of DBM with better performance by Mikio Hirabayashi.
Memcached is started in 2003 by Danga to power Livejournal. Memcached isn’t really a database since it’s memory-only but there is soon a version with file storage called memcachedb.
Infogrid graph database is started as closed source in 2005, open sourced in 2008
CouchDB is started in 2005 and provides a document database inspired by Lotus Notes. The project moves to the Apache Foundation in 2008.
Google BigTable is started in 2004 and the research paper is released in 2006.

##2006-2010

JackRabbit is started in 2006 as an implementation of JSR 170 and 283.
Tokyo Cabinet is a successor to QDBM by (Mikio Hirabayashi) started in 2006
The research paper on Amazon Dynamo is released in 2007.
The document database MongoDB is started in 2007 as a part of a open source cloud computing stack and first standalone release in 2009.
Facebooks open sources the Cassandra project in 2008
Project Voldemort is a replicated database with no single point-of-failure. Started in 2008.
Dynomite is a Dynamo clone written in Erlang.
Terrastore is a scalable elastic document store started in 2009
Redis is persistent key-value store started in 2009
Riak Another dynamo-inspired database started in 2009.
HBase is a BigTable clone for the Hadoop project while Hypertable is another BigTable type database also from 2009.
Vertexdb another graph database is started in 2009
Eric Evans of Rackspace, a committer on the Cassandra project, introduces the term “NoSQL” often used in the sense of “Not only SQL” to describe the surge of new projects and products.

(Some of these dates need to be taken with a small pinch of salt as finding out exactly when the projects started can be a bit difficult. Also not all projects started in last few years have been included)

In 2009 and 2010 we also saw the coming of NoSQL conferences like NoSQL live in Boston in 2010, the upcoming NoSQL eu in London in April 2010. Last year we also saw the NoSQL east conference in Atlanta.

The Employee View of Agile Interviews

2010-02-07T23:43:23+01:00

I found two interesting blog posts via Twitter, Just Interview new Developers and http://www.davenicolette.net/agile/index.blog/1947137/the-new-interview/ both covering a new and perhaps more agile (hey! buzzword!) way of doing interviews. The key point is to audition the candidate rather than to just ask questions. Doing actual pair programming and seeing how people how actually work to solve a concrete programming tasks is much more valuable than to give them a random task, ask if they remember some obscure part of a spec everybody looks up anyway or not least, see what certifications they have. Would you like to work for a company that put more faith in your certifications than your current abilities?

Both posts cover this from the viewpoint of the potential employer but I can see just as much value for the employee. When interviewing for a new job you rarely get an accurate view of how the company really is through the interview. The reality really dawns on you a couple of months into the job. My (somewhat limited) experience with interviews is that they are very rarely used to dig deep enough, both from the standpoint of the employer and the employee. But we should! Hiring the wrong people is expensive for the company and choosing to work for the wrong company is just waste of time for an employee. Or worse.

###The Questions to Ask

If I where to look for a new job I would milk the interview for all it’s worth and use it to see what kind of company I’m dealing with.

If they’re not doing pair programming in the interview, will they do it in real life?
How serious are they about their agility?
Do they check what I remember or do they make an effort to find out how I work?
Are they interested in the buzzwords I know or the process I use to develop software and solve problems?
Are they interested in finding out about the real me?
If at all possible: pair program with the people I’m going to be working with

I would perhaps even go as far as asking them why they’re are hiring they way they are, regardless of how, and use the answer to decipher their way of thinking about people and the craft of software development. I mean: would you choose to work for a company which isn’t really interested in finding out the real you but rather just hire you?

Some other questions I would consider asking to find out more about the company:

What kind of computer hardware and software are developers using (this can seem like useless geek obsessions, but it tells me something about how they value the productivity of their developers. Best answer: you get to choose :-))
How does the company (top to bottom) ensure continuous improvement?
How does the company measure the productivity of their developers? (trick question).
How far up the management chain does the agile principles really go? Think Toyota Production System-like environment.
For a consulting firm: how much/often do you spend time inside the mother ship opposed to on-site at clients?

The Time it Takes

Doing interviews in this fashion takes time and I have heard stories (at Hashrocket I think) of week long pair programming sessions with the entire team and one “no” from one team member is all it takes to turn down the candidate. This probably works better in the US with their system of two weeks notice, but not so well in Norway with (normally) three months mutual notice. But for sure: one hour of questions doesn’t tell you enough. Practice trumps talking, talking trumps reading - to paraphrase the kanban saying.

Learning Ruby With Ruby Koans

2010-01-12T15:58:55+01:00

I’m learning Ruby these days and after reading the excellent book The Ruby Programming Language I found myself in need of actually writing and reading some code. I tried starting a pet project in Ruby but the, uhm, distractions of two young children made it difficult to get that off the ground and I also quickly realized that I needed some more training before I could get productive on that project.

So enter Ruby Koans written by Jim Weirich and Joe O’Brien of EdgeCase. This little git repo guides (and forces you) through a series of test files with (at first) failing unit tests where you read code and make failing tests pass one after another. It starts out easy with variables and true false and the likes and end up with open classes, message passing and modules.

If you tolerate some zen puns and need a quick intro to ruby and testing in ruby, it’s worth a look. The beauty for me was the ease of doing one pomodoro of Ruby Koans each morning and the mental context needed to do that was very small.

Systems Thinking applied to software development

2010-01-05T22:29:50+01:00

I have watched some talks by John Seddon on Systems Thinking and where he feels Lean has failed. Seddon’s focus is service organizations in public and private sector like health-care and customer support and sales for various products. But how would systems thinking apply to software development? Have we got it all wrong? I’m going to try and map some of the ideas of Systems Thinking over to existing software development methodologies and practices and see where we end up.

Systems Thinking in a nutshell

Systems Thinking is about seeing the organization as a system and studying it as a system from the view of the customer. Some core ideas summarized:

The only plan for changing an organization is to get knowledge
Study demand going into the system
Differ between failure demand (bug reports, wrong feature and other sources of rework) and value demand (new features/contracts/projects).
Study the variability and predictability of the failure demand. Only what is predictable is preventable. Take steps to prevent failure demand
Peoples behavior is a product of the system in which they work. If you want to change the people, change the system
Train people to handle the incoming requests and let them pull help on the things they don’t know how to solve.
Don’t standardize the work
Give the workers the means to control the work and the power to change it.
Measure the actual value delivered to customers.
The best way to learn counter-intuitive truths is to see and experience them for yourself

The goals are of course to deliver maximum value (the right features at the right time to solve the right problems) to customers in the shortest time frame possible. Sounds familiar? Read on for more in-depth discussion on each of the items in the list.

Get Knowledge

This very much applies when you’re going to change a team or an organization, but how about the isolated act of writing software? To achieve a successful software project we have to know what the customer wants and needs and more importantly deliver that. So study the clients needs and the client’s process and find out what they actually need. For, as many has experienced, what they say they need and what they actually need isn’t necessarily the same thing. And maybe Seddon’s idea of following the work is applicable to us as well? Actually manually following the work process one is trying the improve or eliminate with a software system. I don’t think we as an industry do this well enough. And we must be prepared for the possibility that the answer might be “we don’t need the software system after all”. Scary for consultants I know, but what is more important: solving the problems of the client/company/society/world or making money?

This also applies when trying to improve a software development process. Seddon advocates actually seeing and experiencing the flow of work in a process instead of merely performing a value stream mapping as kanban/lean advocates sometimes mention. Actually walk with the work item as it passes through the various stages of development and into production. And better yet, if the managers involved need some hard lessons - bring him/her along to see for themselves where the pain points lie. How can we change a development process if we don’t know how it actually works? So 1) start with studying the process you’re trying to change, follow the flow of work and then see what tools can solve the problems.

Study Demand

I see failure demand as more or less all the stuff developers hate and frustrates them: bug reports, wrong features (not what I really wanted etc. ), bad performance, instability etc. All sources of re-work. To avoid re-work makes programmers happier and more productive as well as improve the steady flow of new features through the process. Value demand on the other hand are all forms of new work: feature requests, new contracts and other forms of new work. What is more fun: fixing your colleagues bugs ore creating new features? We all know the answer to that one, don’t we?

And most important of all: identify the preventable forms of failure demand (is some classes of bug reports preventable by using a different testing strategy or development methodology? How can we change the architecture to avoid instability etc). Choose the right tools to prevent as much as possible of preventable failure demand and the value demand will increase by itself. Seddon’s point is that it is futile to try to maximize the value demand without dealing with the failure demand. You’re merely sub-optimizing. I’m not aware of any methodology that has a focus on this but I would like to hear from you if know.

Change The System

In Deming’s words, if you work on the people you work on the 10%, and if you work on the system, you work on the 90% as people’s behavior is a product of the system. In my mind this shows how important the right development methodology is for the productivity of the team and the people in it. But the methodology must suit the software your are developing, the risks involved and to a certain degree the people involved. Because programmers can be a tricky bunch :-) Systems Thinking also advocates putting the workers in charge of changing the work. This maps directly to the idea of self-organizing teams discussed in most agile methodologies like Scrum, XP and Kanban. I think this is an important idea and strict top-down management of development teams should be a thing of the past. And whatever methodology you pick as a starting point for a project, never stop adapting it and improving it. Borrow and steal ideas, cheat or whatever improves the value you deliver.

Training the Worker

Seddon’s arguments come from the front-office workers in service organizations, but it maps well to software development teams in my view. You need developers trained to handle the kind of work that the projects deals with, but not everyone needs to handle all aspects of it. This closely relates to the idea of cross-functional teams, collective ownership of the code and mentoring. Encourage developers to take on work items they don’t necessarily have the knowledge to solve, but institute a system of asking for, and getting help on it. The best way to learn is to work together with someone who knows how to solve the problem. Pair-programming is great in this regard and I would say the fastest way to spread knowledge in a team and educate newcomers. Collective ownership of the code is a bonus.

Important in this regard is also the culture on the team. This is not something one can simply declare. This is culture that has to be created and nurtured from the ground up.

The idea of software craftsmanship also comes to mind here, with established master-apprentice models in place for training new developers. I lighter version of this is a mentor/coach model working in the team with the new member and bringing that person up to speed. This transfers not only raw skill in the technology at hand but also the team culture and their way of doing things. A culture of learning is vital to a successful team in the long run and especially a team where people are replaced with some frequency.

Another point is that programmer fresh out of college or university learns much faster by working with a mentor on a real-world project that working on made-up pet projects in the safe confines of a company. Because they learn the technical skills but also (and more importantly) the soft skills required when working as a programmer like talking to customers, discussing software design, formulating requirements (hey, sometimes clients can’t do this) and just behaving around a lot of different kind of people.

The concept of kaizen (continuous improvement) is also a part of this. Create a culture where people and process is constantly scrutinized and improved. As managers in Toyota (TPS) have experienced: if a process isn’t improving, it is degrading. And this in my view is true for technical skills and soft skills in people too.

Don’t Standardize the Work

The closest analogy that comes to mind is coding standards, testing standards and the like and this is a tricky one. I don’t readily think this should be done with without good reasons. Some have argued that the need for coding standards, both for syntax and design, is greater when using dynamic languages like Ruby, Perl or Python, than with statically typed languages like C# or Java. This is primarily because the tools enforce much less and the need for discipline is greater. The need also increase with the size of the team.

I’m not ready to agree fully with this, but I can see a valid point. It also is a factor of the skill of the team members and what they perceive as beautiful/clean code and what is dirty. One developer’s diamond can be another’s turd - simple as that. Another point is this: a team left to self-organize and dictate their own work and not doing a good job at maintaining a clean code-base that supports re-factoring and addition of new features need to be shown the path by someone who knows how. And notice the “shown the path” part, not “showed forward on the path”.

I also think some fairly loose rules regarding “how we do it here” is worth having regardless of the technology involved. But the rules should come from the team, not from above. Company-wide standards can be very counter-productive especially when different technologies are involved in the projects.

Measure the Value Delivered

Kai and Tom Gilb have in particular argued that agile methodologies aren’t focused on delivering value and measuring that value and I agree. Their Evo method for measuring business valyue address this. How can you know if you are actually delivering value to the customer when you’re not measuring it? Evo has a lot of tools for this and I think it can be useful in many situations. A real-world case presented by Kai Gilb at a recent Oslo XP meetup meeting concluded with more focused development efforts and more precise features delivered for the same amount of money compared to the clients earlier experience.

The concrete task of measuring the quantifiable value you are delivering is a good idea and it’s the only metric actually being concerned about. Another important point from Seddon from service organizations is that you get the behavior you measure i.e. if you measure code coverage of the test suite, people will cheat to get a high coverage. And this does not mean good tests. This goes for all historical metrics that have been tried and failed in software development: lines-of-code produces, number of (unit) tests, work items etc.

And as long as we’re on the topic: what is the actual value of estimation? There are some values in experiencing how long a task takes and you often can get a head-start on designing and seeing potential future problems when talking and thinking about a feature. But this a product of talking and thinking about a feature, not estimating it! In many organizations management want estimates. But aren’t they really after a commitment on what will be done at what time? Kanban practitioners often see this and use empirical evidence to determine how long something will take rather than predict it up front. A technique like a cumulative flow diagram show you the time a work item take to clear the work process and if you have a fairly low variation in the size of the work items (you can control this) you use this to predict future deliveries.

Summary

I think the current software methodologies have something the benefit from the ideas of Systems thinking, but as Seddon himself argues against Lean we can’t just take the ideas and run with them as they are primarily for a different domain (Lean for production and Systems thinking for public sector service organizations) but the we must adapt them to our field. As Seddon also advices when considering a tool:

Who invented the tool?
What problem was he trying to solve?
Do I have that problem?

Great advice in all walks of life.

Linkdropping on NoSQL, Lean and Systems Thinking

2010-01-05T14:03:02+01:00

I thought I would round up a collection of links that interested me the past few weeks on various topics. First off: NoSQL databases:

An almost impossibly detailed run-down of NoSQL databases: http://www.vineetgupta.com/2010/01/nosql-databases-part-1-landscape.html. Part 2 should be good too.
Slides from a talk by Emil Eifrem of Neo4j on the different types of NoSQL databases: http://www.slideshare.net/emileifrem/nosql-east-a-nosql-overview-and-the-benefits-of-graph-databases
Update Emil Eifrem mentions in his blog post in the comment another very nice overview of the NoSQL ecosystem by Jonathan Ellis of Rackspace.

Then over to Lean and Kanban. Henrik Kniberg and Mattias Skarin just published a new book on InfoQ titled “Kanban and Scrum - making the most of both”. If you’re not familiar with Henrik Kniberg’s work, I also suggest “Scrum and XP from the trenches”. Erling Wegger Linde’s “A Kanban brown bag recipe” is also worth a read.

And in the spirit of Lean: There is a video of talk by John Seddon of Vanguard titled “Cultural Change is Free”. Mainly about systems thinking in the public sector, but private sector aren’t infallible either. Seddon often criticize Lean for being wrong in many places, but I often feel he is criticizing a wrongful implementation of lean ideas, much the same as scrum is often criticized for the misgivings of wrongful implementations. Or rather he is criticizing the tool focus of a lot of lean consultants not the lean principles themselves. And he stresses the differences between the Toyota Productions system and other kinds of organizations. You could also check out is talk “Re-thinking Lean service” on InfoQ which deals with the same topic in a slightly different packaging.

Posting to Movable Type From Emacs

2009-12-29T16:55:56+01:00

I stumbled across weblogger.el which is an xml-rpc interface for posting to several blog engines, including Movable Type which I use. And I thought: Hey, why use that somewhate crappy online editor for writing blog posts when one can use a fairly OK operating system to do the job (aka Emacs).

Setup

Download weblogger.el from the above link and also the requisite xml-rpc and load them in .emacs in the usual manner.

Run weblogger-setup-weblog and fill in the correct values for configuring the blogging engine. For Movable Type the URL endpoint is http://your.blog.url/mt/mt-xmlrpc.cgi. The user name is your usual username, but the password is the special API password which can be found and set on the bottom of the user page in the admin interface. After that it’s just C-c C-n for a new entry. The setup will fetch all existing entries automatically so you can cycle through and edit too.

Caveats

It seems the current version of weblogger.el doesn’t handle drafts properly against MT 4. The first version saved as a draft is published instantly :-/. Later saves keep the draft status. Also, the familiar C-c C-s for saving the buffer actually publishes. So you’re going live pretty soon.

A MacOS X Keyboard Layout for Programming

2009-12-11T20:47:08+01:00

I have previously created my own keyboard layout for windows and Linux in order to have a layout where the often-used keys in programming are more accessible than in the standard Norwegian layout. And when I recently got a (shiny) MacBook pro, I wanted to have the same keyboard layout for the Mac also. Consistency is key across machines, so the layout is the same as in the original progn keyboard layout (most braces and slashes available on home row or the row above). Downside is if decide to update it, I have to update three copies. Oh well…

In the windows and Linux I ended up mapping caps-lock to alt-gr and binding a lot of new key combinations using that modifier. I chose a similar although slightly different model for the Mac. I choose to use the caps-lock key as another alt key (set in the keyboard system preference) and then map keys to alt+for easier access. This means I loose the keys already bound on the alt key, but I keep the original keyboard layout easily accessible, and switching is a breeze. On the positive side, I don't mess with key bindings in either the Terminal app, Aquamacs (when fn is set to be meta) nor IntelliJ IDEA. Here's the finished new bindings:
Home row is the most important one, with often used characters, and the braces on homerow+1 on the right hand. Pointy brackets and pipe on the left hand.

Recipe

Here’s how I created the new keyboard layout for mac running 10.6 Snow Leopard (I think it’ll work on Leopard (10.5) as well, but haven’t tested it). Finding an existing layout in the .keylayout XML format was quite difficult but I found an extended Norwegian layout which had some often used keys already mapped nicely, like norwegian quation marksand some others. So I used that as a base.

Download Ukulele from the web page and install it.
Either start from scratch defining keys in Ukelele or start modifying an existing keyboard layout.
Open the character viewer (System preferences -> Language & Text -> Input Sources and select Character and keyboard viewer
Drag characters from the character viewer while holding down the desired modifier key.
Save under a suitable name. Be creative :-).
Copy to “/Library/Keyboard Layouts” for access for all users or “~/Library/Keyboard Layouts” for just your user.
log out and log in, and you should be able to choose the new keyboard layout from System Preferences.

I called my keyboard layout progn, like the others I have created and you can download it if you want to use it for further tweaking. I made it available under the same creative commons license as this blog, so remixing is allowed provided you make the re-mixed versions available to the world wide intertubes.

Doing do-release-upgrade on an offline ubuntu mirror

2009-12-03T13:42:13+01:00

I have an offline mirror of the ubuntu package repositories for use on a network not connected to the internet. This works like a charm for updating packages on individual machines, but when it comes to doing a ‘do-release-upgrade’ to the next release, like these days I’m trying to upgrade to 9.10 Karmic Koala, we need some tricks to make this work. Firstly, sync your mirror so all the new packages of the new release are locally available. What you probably find is that the upgrade manager/do-release-upgrade software can’t seem to find a new available automagically. So how do we accomplish this?

<ol><li>update-manager and do-release-upgrade reads the file /etc/update-manager/meta-release to find the location of the meta-release file. This points to the internet location changelogs.ubuntu.com normally. And if you just mirror the package repos, the meta-release file isn’t included. So we need to fetch it first: ‘wget http://changelogs.ubuntu.com/meta-release’.
</li><li>Store it, for instance on the root of the internal mirror or some other convenient location, and put the url to it in the “URL” value in the /etc/update-manager/meta-release. If you’re upgrading to a LTS release, fetch the meta-release-lts file too and repeat the process.</li><li>Edit the meta-release file you just downloaded and substitute the external mirror address with the url for the internal mirror so all package locations match up. For me this was replacing ‘archive.ubuntu.com’ with ‘explorer/mirror’ since the internal mirror is available at http://explorer/mirror/ubuntu/. Make sure the file is readable via http (or file permissions if using file access to repo).
</li><li>Run update-manager or do-release-upgrade and the upgrade should work as you were using an internet mirror.
</li></ol>

Hacking on munin 1.4

2009-11-27T20:17:46+01:00

I have been doing the ol’ open source thing lately and helping out some old colleagues from Redpill-Linpro hacking on the much-awaited 1.4 version of munin (out today) which has tons of new features and not to mention tons of new plug-ins for just about all kinds of hardware and software out there. The changelog is longer than most I’ve seen in a while, and that’s definitely a good thing. I’ve been doing mostly html+css work on 1.4 but I am planning to dig deeper into the core in the future and help out with some much needed re-factoring of the Perl code. And hopefully contribute some tests as well. It’s good to write Perl again.

Planned plugins

I thought I work on some new plugins as well when I get spare time, namely

SNMP plugin for Synology NAS
SNMP plugin for Qnap NAS
SNMP plugin for IBM Bladecenter H (if it has some good data available)
SNMP plugin for IBM Storage DS3000 series

And I also going to see if the NetApp at work spits out something more useful not covered in the existing netapp plugins.

If you need monitoring and graphing of key metrics from your servers, network hardware, storage hardware and software servers (jmx plug-ins new in 1.4 too), you should give munin a spin.

Creating Craftsmen And a Craftsmanship company

2009-10-25T18:04:28+01:00

How does one go about creating software craftsmen in a company and more how does one create/transform a company to make this possible? These are the questions and thoughts I found myself with after attending an open space session at Smidig 2009 (the Norwegian Agile conf).

Perhaps the most known craftsmanship consulting firms are Obtiva and 8th Light, both located in Illinois USA (one on Chicago and the other in Round Lake). They are, as far as I know, created specifically to a have a master and apprentice model and takes this to extremes. Their craftsman exchange is a fine example of this, where senior consultants got to piggyback the other firm's people on real world projects for a week, just to learn from them on how they build software and how they run their company. They also hire programmers with the title of apprentice and one account of being an apprentice at 8th Light is found in this blog. Not bad to commit JRuby code while formally being and apprentice. I especially noticed the part about being specific (coding) challenges and the getting feedback from not just one but all masters in the firm. I suspect the amount of condensed learning is formidable. But does this approach work when you trying to transform a "normal" consultancy firm into something more craftsmanship friendly?

Let's examine some other tools to use, more tuned in to (Norwegian) consulting firms and companies that do not have this in place now, but rather have their hands full with competing and hanging on to clients. What should one do?

Organic interest groups
Mike Cohn explained in Oslo XP meetup talk recently Google's model of special groups within the company forming horizontally across projects. A talk on the same subject at JavaZone 2009 in Oslo discussed the same idea employed in KnowIT ObjectNet, a consulting firm in Oslo, Norway. They even have a cooperation with the Simula Research center on finding out what actually works best.The important bits are:

Voluntary membership based on interest (Google, KnowIt)
Each person can join many groups (KnowIt)
Organic growth bottom-up is preferred, but it should be allowed to transform into more formal groups over time. (Cohn)
Given a mandate to change practices (google, KnowIt)
Communicate the results

This is a key issue. People need to have a forum to evolve their skills outside of the current customer project, which in many cases isn't well suited for that, deadlines and all. Learning a new language, which in my mind brings with it a lot of benefits especially when learning one from another paradigm than one is used to, is best done outside the creation of enterprise code under time pressure. Besides, one needs room to experiment and fail. Demanding some kind of output from the work is important to keep it in check and make the group prioritize and work toward a goal, not to mention to disseminate the knowledge gained to non-members and even the development community at large.

Reports, internal or external talks or patterns & practises changes are some types of outputs. In the concrete case of KnowIt, the groups were allowed to bring in outsiders which in one case created a cooperation with the Norwegian Scala user group ScalaBin. A good example of an activity that is of value to the entire community, not just the company itself. And I would argue that a good software development community is an important factor in creating good companies and good developers.

This should work well for consultancies in my opinion. People need some time off the project to keep in touch with the company and what better way than to spend it doing research on a hot topic that may bring in future clients? KnowIt also found that you need some hot topics to keep the juices flowing in the developers, but the company could put more regular topics to good use to. And it all depends on the people you have on board.

Coaching
A new programmer fresh out of college, needs guidance to avoid stepping into balls of mud and re-inventing the wrong wheels. How do we accomplish this? Several thoughts emerged on the open space at smidig 2009. Pair programming is an excellent way of transfering knowledge from one head to another. There are a number of factors that need to be in place for it to function properly, but done right can produce great results. For a consultancy it requires that a customer allows junior programmers to be put on the project team and the (at least perceived overhead) of mentoring them.

Concrete feedback and guidance while working on real-world problems was mentioned by several people and the rationale was that learning isn't as rapid or valuable in constructed problems. You need wet feet to really appreciate the boots you're getting. Code reviews can also be used to make novice programmers see the errors of their ways and seeing how it should have been done.

Quality
Never compromise on quality. If a customer ask you to deliver the same features, only a little bit faster, say no. Plain old no. If you're doing it faster, you're cutting corners. You may not think you are, but you are. This requires backbone from both developers in the trenches and managers negotiating contracts.

Don't do overtime (at least not as a rule). You need sustainable pace. Burned-out developers make poor craftsmen and they need a life outside work. Besides, who has the energy to learn Clojure or contribute on open source projects if they spend all their time and energy on work? You need time off. Use it.

Learning and teaching
At the core of it all is learning. Becoming a craftsman requires lifelong learning.The principle of Kaizen (from Lean) embodies what I think is the right path: small incremental learning each and every day, improving a small part of the workplace, your skills or your team. Big steps are much harder to do and get right.

Some other points regarding learning: some for the developer and some are for the company.

Realise how little you know and be open to learn from everyone you meet.

Always challenge your beliefs.

Always buy the books you want (and read them)

Keep up with the trends

Seek new challenges

For the company:

Allow consultants the time to develop

Nurture a culture of learning and improve the company in small increments, not just the people. Kaizen all the way.

Allow for mistakes and but ensure learning from them, both individually and as a group.

And remember: you can't expect to change other people, you can only change yourself.

A craftsman shares his knowledge with others and tries to make the community around him grow with him. Give talks at conferences, user groups or inside you company. As a company, encourage developers to give talks, give them time and resources to do so. Follow up on it. Spread the word. Nothing forces you to know your stuff quite like giving a talk on it to an audience. Scary but rewarding.

Tools
A craftsman needs to know the tools of the trade and to use the right tool for the right job. This is mainly a skill one gets through experience. But the right kind of experience - a varied one. This is the consultants world and consultants are in better shape than most in-house developers when it comes to this. But learning a new paradigm or language will take you far in my opinion. Just the act of learning a totally different language, even though you just use if for toy projects, refreshes your perspectives on your "day-job" language. Especially if your job language is C# or java and you learn a dynamic language or a functional one. The perspective you attack problems with in your job language will change and change for the better.

Knowing your IDE or other developer tools is essential, as is knowing and tweaking you platform. This means also that companies must let developers have full control over their development machine. The opposite actually still occurs. And not to mention: the developers need the sexiest hardware money can buy and plenty of screen real-estate.

People skills
A craftsman need people skills. She needs to understand the customer and make the customer understand her. She needs to be a team player and know when to shut up and when to speak up. She must be responsible, honest and brave. How to achieve all this? Can introvert geeks manage these things? Well, some people will have to work more on it than others, but I feel the environment we operate in and the feedback we get, can develop many of the skills. Keywords are giving developers responsibility to make technical decisions, to work independently, embrace, encourage and reward team efforts, not individual efforts. Challenge their fears and push the limits of their comfort zone a bit, within reason. Careful not to push to hard :-)

And besides: The primary output of a software project is people, not customer value and software (paraphrasing Ward Cunningham). You are the sum of the work laid down. As an ending note: the book

Apprenticeship PatternsGuidance for the Aspiring Software Craftsman by Dave Hoover and Adewale Oshineye is out on O'Reilly now and discuss a lot of this in depth (although I haven't read it yet).

This blog post is getting to long as it is. Feel free to continue in the comments if you have something to add.

A SyntaxHighlighter brush for Lisp

2009-10-14T21:13:31+02:00

I use SyntaxHighlighter for highlighting code snippets on this blog and I found out that it didn’t come with a brush for the 2.0 version highlighting lisp code. So I wrote my own. I found a SyntaxHighlighter 1.5 brush through google and I converted that to a 2.0 brush and it’s now in a state that sort of works. At least it highlights some code :-) I used that as an excuse to try out qunit as well and write some unit/integration tests for the code. Qunit is cool. There’s not much code in the brush itself as it leans heavily on the core SyntaxHighlighter stuff. So the tests are testing the framework also. But hey, I’m not picky.

It’s all to be found on github:knuthaug/sh-lisp.Patches/pulls via github is more than welcome. I plan to dig out my copy of “Ansi common Lisp” and go through the lists of functions, keywords and macros a bit more in detail. Some more tests are in order to. One question which comes to mind is if it’s best to do one that does common lisp and emacs-lisp, or one brush for each dialect?

Here’s the code for brush:

SyntaxHighlighter.brushes.Lisp = function(){

    var funcs     = 'lambda list progn mapcar car cdr reverse member append format';
    var keywords  = 'let while unless cond if eq t nil defvar dotimes setf listp numberp not equal';
    var macros    = 'loop when dolist dotimes defun';
    var operators = '> < + - = * / %';

    this.regexList = [
     { regex: SyntaxHighlighter.regexLib.doubleQuotedString, css: 'string' },
     { regex: new RegExp('&\\w+;', 'g'), css: 'plain' },
     { regex: new RegExp(';.*', 'g'), css: 'comments' },
     { regex: new RegExp("'(\\w|-)+", 'g'), css: 'variable' },
     { regex: new RegExp(this.getKeywords(keywords), 'gm'), css: 'keyword' },
     { regex: new RegExp(this.getKeywords(macros), 'gm'), css: 'keyword' },
     { regex: new RegExp(this.getKeywords(funcs), 'gm'), css: 'functions' },
    ];
    
}

SyntaxHighlighter.brushes.Lisp.prototype = new SyntaxHighlighter.Highlighter();
SyntaxHighlighter.brushes.Lisp.aliases   = ['lisp'];

Definition of an Agile Leader

2009-09-30T20:59:54+02:00

Patrick Wilson Welsh blogged a while back about the role of the agile leader and I think he is pretty bang on. He outlines four qualities or tools an agile leader needs to be successful in an environment where the old maxims of project management, or management in general, or even technical leadership doesn’t cut it anymore. The tools are:
<ol><li>Continous team building</li><li>Continous planning</li><li>Continous unblocking
</li><li>Continous improvement (which in the lean tradition is also known as Kaizen).</li></ol>The importance of each one varies with the environment in which a team operates, but all are necessary. And one of the key things a lot of organisations doesn’t seem to understand is that building a team isn’t about creating a team that works well with a benevolent dictator keeping a watchful eye on them at all times, but rather creating teams that to almost no degree needs a “boss” at all. This is a team where continous self-improvement for the individuals and the team as a whole isn’t something mysterious regarding japanese logographics, but rather routine. The thing a team really needs help with is unblocking. This must be done by someone who shouldn’t write code at the same time. I’ve tried that model and it sucks royally.

But how do we create or foster leaders like this? There are several things we need. We need whole organisations that are willing and easy to change. No small feat. Is this best achieved from the bottom up or top down? The lean/toyota philosophy dictates bottom up or rather at all levels in the organisation and that all levels are of equal importance. A common trait in companies and organisations is that rigidity comes with size. And a rigid already established company is even harder to change that creating lean smaller companies.

A team with such a leader wouldn’t function very well in an old-school environment, and they must go hand in hand. Educational institutions must somehow adress these challenges in the future as they’re at least not in Norway even adressing agile methodologies properly. They’re way behind the curve as it is. And methodologies aren’t enough as Abby Fichtner wrote in a blog post which was a precursor to Patrick’s. Some of these values or tools mentioned by Abby and Patrick are mentioned in several methodologies but perhaps not the continous part in all settings. And do you really need a full blown methodology that dishes out all this and gives you a certification for just showing up at a course? I don’t think so.

A screenscraping filesystem for accessing media archive

2009-09-18T10:48:07+02:00

This is probably directly interesting to Norwegians since the content in question is in Norwegian, but the concept is so very cool. In Norway we have NRK the state broadcast corporation which has a very contemporary view on sharing its on content to the public. It is after all paid for by the people through taxes. And they have a rich library of media streams available on the net of mostly all television productions they created themselves.

I came across this little blogpost (in Norwegian) about Erlend Klakegg Bergheim’s effort to write a fuse filesystem which scrapes the archive web pages and creates a file system which you can mount as a regular file system on linux. Then just point your asx-capable media player (like vlc, mplayer, totem or kaffeine) to the “file” and play away. It is simply genius and ingeniously simple and makes browsing the content a breeze.

It’s written in python and chimes in at a 250 lines of code in total. The source is available at github:klakegg/nrkfs

The shunting yard algorithm kata

2009-09-15T10:21:26+02:00

After attending a code dojo randori with the Oslo XP meetup where we worked our way about half-way through the kata for the Shunting yard algorithm, I decided to have a go at finishing it completely in Java (which I am quite rusty at) and the trying it in Ruby (at which I am a total newbie) and examine the differences. Especially Javas somewhat sketchy regexp support was something that annoyed me, and Ruby excels at that part.

The Shunting yad algorithm is an algorithm for transforming infix notation to reverse polish notation, often used for arithmetic expressions and takes operator precedence and associativity into account.

My book list

2009-09-15T10:08:32+02:00

This is a list of all the books I’ve read that are in some way related to programming and software development, along with some thoughts on them.

Languages

The Ruby Programming Language by David Flanagan and Yukihiro Matsumoto. Really a comprehensive guide to Ruby 1.8 and Ruby 1.9, but not a "how-to program" type of book. Simply wonderful information density and one of the clearest programming books I've read in a long time.
Programming C# 3.0
C# 3.0 Cookbook
C# 3.0 Design patterns by Judith Bishop. A nice overview of common design patterns and how best to implement them in C#.
Programming Perl. A must when beginning to program Perl. Nice reference and enough depth to keep you going for a long time.
Practical mod_perl
Perl Best Practices by Damian Conway
Higher-order Perl by Mark Jason Dominus. This is a gem of a book for every Perl developer that wishes to expand her repertoire of tricks, but not suitable for beginners. It tackles topics such as memoization, currying, recursion and iterators, recursive-descent parsers and other neat stuff for transforming code using code.
Perl Cookbook. A collection of recipes for solving small and large problems in idiomatic Perl.
Perl Testing by Ian langworth and Chromatic. Nice rundown of Perl testing covering both Test::Class and Test::More. It's in the developer notebooks series so it's really hands on all the way through.
Javascript: The Definite Guide.
Javascript: the good parts by Douglas Crockford. What is says on the cover. Really.
Ansi Common Lisp by Paul Graham. I would say this book can be compared to Kernigan & Ritchie's The C Programming Language. More parenthesis, though.
XSLT 2.0 by Michael Kay. Handy reference for those times when there's no way to escape XSLT.

Methodology

Agile Software Development with Scrum By Ken Schwaber
Agile Estimating and planning by Mike Cohn. A very nice book covering a lot of nooks and crannies regarding estimating and planning agile software development. I would say a must for all people involved in agile projects.
Agile Software development by Alistair Cockburn. One of the best discussions on agile software development as a field, without stranding on specific methodologies. A very good read, especially the parts on communication and individuals.
Test-driven Development by Dave Astels
The Pragmatic Programmer by Dave Thomas and Andy Hunt
Continuous Integration by Paul M. Duvall. Didn't tell me much I didn't already know, but gives a nice overview of continous integration for newbies in the field.
Pragmatic Project Automation by Mike Clark. Quite basic rundown of the task of automating all (as many as possible) tasks in building a software project. Beginner-level.
Xunit Test Patterns by Gerard Meszaros. High-level discussion of test patterns for development. Testing in a TDD/BDD sense.
Junit Recipes by J.B. Rainsberger
Clean Code by Robert C. Martin.
Scrumban by Corey Ladas.
Code Complete 2nd edition by Steve McConnel. A bible on software development in many ways. Packed with useful advice even though a lot is common knowledge if you been around for a few years.
Scrum and Kanban - making the most of both by Henrik Kniberg and Mattias Skarin
Kanban - Successful Evolutionary Change for Technology Organizations by David Anderson
Extreme Programming explained by Kent Beck
Agile Software Development, Principles, Patterns, and Practices

Design and architecture

Domain-driven Design by Eric Evans
Refactoring by Martin Fowler
Code Complete 2nd ed. by Steve McConnell

Machine learning

Data mining by Ian H. Witten and Eibe Frank
Data Mining - Concepts and Techniques by Jiawei Han and Micheline Kamber
Pattern Classification by Duda, Hart and Stork
Pattern Recognition and Machine Learning by Christopher M. Bishop
Fundamentals of Neural Networks by Laurene Fausett
Neural Networks for Pattern Recognition by Christopher M. Bishop
Fuzzy Modeling and Genetic Algorithms for Data Mining and Exploration by Earl Coz

Other

Applied Cryptography by Bruce Schneier. If you don't know who Bruce Schneier is, take a look at .

Wish list/future reading

Leading Lean Software Development: Results Are not the Point by Mary and Tom Poppendieck.
Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman By Dave Hoover, Adewale Oshineye
The Art of Agile Development by James Shore and Chromatic
Working Effectively With Legacy Code by Michael Feathers.
Software Craftsmanship by Pete McBreen

About the Author

2009-09-15T10:07:46+02:00

I am Knut Haugen [Knu:t Hæugen], a software developer working in Oslo, Norway. I am programming on and off in Perl, Ruby, C#, PHP and Java and I have even had my hands on Lisp. I prefer to use a Linux environment whenever I can, but I’m forced to work on windows now and again. Ooh, and I recently got a mac (shiny!) and I love it. Other interests in the field of software development includes methodologies like almost everything that falls within the agile sphere, and scrum and lean/kanban in particular. I’m also partial to TDD/BDD/DDD techniques but I still consider myself in the learning stage at this.

I got started with computers one summer vacation when I was around 14 or so, when my uncle (who got started with computers back in the 70s) had me tag along when he programmed a horse-betting game on a casio calculator in BASIC. And then I generally didn’t catch on besides playing Doom and 3D Wolfenstein on my dads 386. I was more into photography and sports and worked as a freelance photographer from the age of 15 to 20. I applied for photojournalist school, but didn’t get in. My second choice was computer science and I ended up there. I bought my first own computer before starting the course and the night before I moved my uncle (yes the same uncle) came by with a stack of cds and said “install these first then windows”). The cd’s where FreeBSD 2.2.6 and I can’t think of a deeper pool to jump into when fairly new to computers.

My first programing language was VB5 as that was the language the course “introduction to programming” was taught in. I luckily moved on to Java (eh, not so lucky after all, but better than VB) and then Perl while in school and got stuck mostly doing Perl for the first seven years of my career as a consultant. Nowadays it’s mostly Ruby, C# and PHP and I can see a java project on the horizon. If I can choose freely I would prefer working in Perl (which I still know best) or Ruby. In fact any dynamic language is better than a statically typed one.

Some more links:
http://twitter.com/knuthaug
http://github.com/knuthaug
http://no.linkedin.com/in/knuthaugen
http://www.flickr.com/photos/megafon/

My developer toolchain for Ruby

2009-08-20T15:49:15+02:00

I recently learned Ruby and I thought I’d lay out the different tools I use when programming Ruby just for self documentation purposes. Most weight is put on the IDE and the TDD tools that I use.

IDE

IDE is a big deal for many people and I have tried two, Eclipse and Emacs. Since I normally only program Ruby on Linux, these are my two alternatives. In Eclipse I’ve tried the ruby plugin from Aptana and the standard one in the “Programming languages” section of the eclipse update manager. and I can’t say I prefer one over the other. Since Ruby is a dynamic language you won’t get the benefit of refactoring and completion like you would in Java, and this has lead me to ditch Eclipse for Ruby in favor of Emacs. What I miss in Emacs is integrated docs and integrated test runner. It’s possible in emacs as well but takes some hacking. We’ll see if I get around to that. I’ve been using Emacs for 12 years or so, and the first tool you pick always stick. This is my current ruby setup for Emacs:

(autoload 'yaml-mode "yaml-mode" "" t)
(autoload 'ruby-mode "ruby-mode" "" t)
(autoload 'rhtml-mode "rhtml-mode" "" t)

(require 'flymake-ruby)
(add-hook 'ruby-mode-hook 'flymake-ruby-load)

(add-hook 'ruby-mode-hook
          (lambda()
            (add-hook 'local-write-file-hooks
                      '(lambda()
                         (save-excursion
                           (untabify (point-min) (point-max))
                           (delete-trailing-whitespace)
                           )))
            (set (make-local-variable 'indent-tabs-mode) 'nil)
            (set (make-local-variable 'tab-width) 2)
            ;;(imenu-add-to-menubar "IMENU")
            (local-unset-ket "TAB")
            ;;(define-key ruby-mode-map "TAB" 'yas/expand)
            (define-key ruby-mode-map "\C-m" 'newline-and-indent)
            (require 'ruby-electric)
            (ruby-electric-mode t)
            ))

;(add-hook 'ruby-mode-hook
;          '(lambda ()
;             (make-variable-buffer-local 'yas/trigger-key)
;             (setq yas/trigger-key [tab])))

(provide 'ruby)

I use flymake to do interactive compile as I type but I suspect I’ll be turning this off when Ruby gets more into my fingers.

I want to get irb up and running from within Emacs but haven’t gotten that far yet. In addition to the ruby specific stuff I use ECB, auto-complete and yasnippet for extra support. In fact my complete emacs setup is available at the emacs github repository. I’m also thinking about getting some elisp together to be able to integrate the testrunner into a shell window in emacs. The shell part isn’t hard but it takes some configuring to get the window electric and behaving the way I want.

TDD/BDD tools

I develop in a TDD/BDD style and most of the tools are geared towards that. Here’s a list:

The ruby metric_fu for all your analysis and metric needs. It wraps a lot of other rubygems that are handy. It covers code coverage with rcov, cyclomatic complexity with Saikuro, Flay for duplication detection, Flog for misc. pain points, reek for code smells and more. Nice graphs and all.
Cucumber for BDD style outside-in testing. Does a great job both for web applications and non-web applications.
Test::Unit for low-level unit test for driving out the details of classes.
RSpec for the times when the Test::Unit assertions feels awkward.
Github for storing everything. Git is the coolest version control system since, well, since ever.

Generally things move so damn fast in the ruby space and it’s hard to keep up when it’s not your main activity.

What's next

I haven’t tried Ruby-on-Rails yet and the next webapp idea I get, I’m seriously considering doing that. Yeah, I know. I’m slow.

All your APIs are belong to us

2009-08-18T10:19:29+02:00

I saw the link to gotapi.com link referenced from stackoverflow question 63918 and if you’re like me and program in multiple languages and multiple frameworks it is the closest thing I’ve found to a one stop website for API references. It acts as a wrapper around API docs and specs from all over the web and covers things like Java API, JQuery (albeit an older version), Python, PHP, GWT, Ruby (and Rails), HTML, CSS XSL/XPATH/XSD, Perl and much more. And to top it off, you can cherrypick which ones to search in at start.gotapi.com. This replaces a lot of bookmarks for me.

Luke - the Lucene index viewer

2009-08-17T10:20:25+02:00

I’m working on a web application at the moment which uses Lucene as a search engine and the <a href=”http://framework.zend.com/manual/en/zend.search.lucene.html”>Zend Lucene PHP port</a>. And when debugging the indexing of documents I came across <a href=”http://www.getopt.org/luke/”>Luke</a> a browser for the Lucene index file format. It’s a nifty tool and really the swiss army chainsaw you need when you can’t really search for what you thought you added or thought you added something that isn’t really searchable at all. It is written in Java and uses the java Lucene libraries but reads the Zend Lucene index file without problems (They’re compatible after all). It lets you view fields in the index and all sorts of information about them and browse all documents stored in the index and of course search it to actually see what you get. A very cool tool when it comes to debugging Lucene.

On Ruby's elegance and Perl's lack of it

2009-07-29T10:15:47+02:00

I’ve been programming Perl for a number of years and I like the language a lot. It’s my weapon of choice when tackling some random programming task or just spiking something quickly. One thing that has bothered me over the years and surprised me to an extent is Perl’s rather lousy reputation. It’s criticized for being just line-noise, hard to read and generally frowned upon by people outside the Perl community. But Perl is a very flexible and powerful language. True you can write incomprehensible Perl, just as you can write gibberish in most any other language, but you can certainly write good, clean OO code. You just have to know what you are doing (or use Moose which is quite superb at covering up the transgressions of the Perl OO system).

I recently picked up a copy of The Ruby Programming Language and the more I read the more it dawned on me just how much Ruby has gotten from Perl. This is no secret as the wikipedia page on Ruby spells out quite clearly. Yet Ruby is considered clean, elegant and straight-forward and has a much better reputation than Perl. Why is this? I decided to dig a little deeper. One point is that in Perl your effort, at least when new to Perl, must be on keeping the code from going all over the place and in Ruby you have to really try if you’re gonna write sloppy style. Wherein lies the actual differences?

The regex syntax

The regular expression syntax is quite similar in Ruby and Perl in that it’s expressed directly with // and not requiring you to instantiate an Regex object first like in C# or Java. Perl:

print "match" if  $string =~ /\w+?/;

Ruby:

puts "match" if string =~ /\w+?/

But here’s where Ruby fanatical devotion to the Pope, er, objects comes into play. In Perl you can choose to compile the regex once to avoid multiple regex engine compilations in a loop and store it in a variable. But all access to groups is done via $1, $2 etc. . In Ruby this creates a new object of the Regexp class hich you can either assign to a variable to use it multiple times or pass around or call methods on. Ruby:

regex = /\w+?/
puts "match" if regex.match("foo")

# or:

/\w+?/.match "foo"

and the match function returns MatchData object(s) which can be used for further digging. An you’ve got all sorts of cool functions like union which returns a Regexp object which will match all the patterns given as arguments. Doing that in Perl can be very messy. All in all Ruby regexp handling can look like Perl’s but goes much further and gives you a lot more flexibility to do what you want in a clean and readable manner.

Variables and nested data structures

First there are to important distinctions: perl use special symbols on variables to indicate variable type, $ for scalars, @ for arrays and % for hashes. Ruby uses (fewer) symbols for indicating variable scope, namely $ for global variables, @ for instance variables, and @@ for class variables. For local variables you do without a sigil. But the more interesting point here is the syntax for defining nested hash tables and arrays in combination with these.

defining an hash of arrays in Perl:

my %hash = ( 'key' => [ '5', '3', '1' ] );

#or as a reference directly

my $hash =  { 'key' => [ '5', '3', '1' ] };

and in Ruby

hash =  { 'key' => [ '5', '3', '1'] }

#or with symbols as key

hash = { :key => [ '5', '3', '1'] }

Quite similar. But when we want to access the inner hash and use that one directly, for instance in a sort, look what happens when we have to de-reference the array ref inside the hash: Perl:

my %hash = ( 'key' => [ '5', '3', '1' ] );
@sorted = sort @{ $hash{'key'} };

#or with custom block for sorting reverse

@sorted = sort { $b <=> $a } @{ $hash{'key'} };

Ruby:

hash = { :key => [ '5', '3', '1'] }
sorted = hash[:key].sort

#or with custom block for sorting revers
sorted = hash[:key].sort { |a, b| b <=> a }

Much less noise in the ruby version and I can certainly see that the Perl version, which wouldn’t be regarded as line noise at all to seasoned Perl programmer, could look intimidating to a java programmer. But the ruby way is more elegant. And it’s all about perception.

Block syntax

While we’re at it, let’s look at block syntax. This is not so much a noise issue as a power issue which also contributes to cleanliness in the code. Ruby’s block syntax with parameters as we saw in the previous example can be used for all sorts of cool things. A block is also an object in Ruby, which it isn’t in Perl.

hash = { :b => "foo", :a => "bar" }

hash.each { | key, value | puts "#{key}: #{value}" }

# or a multiline do..end block

hash.each do | key, value |
  puts "#{key}: #{value}"
end

Let’s take an example comparing map() from Perl and Array.map in Ruby (including quoting array values to clean things up a bit in both Perl and Ruby).

@array = qw(a b c d e f);

#transform array to ASCII value for the elements
@chars = map { ord } @array;

array = %w{a b c d e f}
chars = array.map { |x| x[0] }

In this example the Perl version looks cleaner to me, but it will fail spectacularly when you need more than one parameter into the block. Ruby handles any number of block arguments, while Perl only has the one value from the data structure you’re mapping. This makes the Hash.each_pair (with a block) very useful in Ruby while you cant really do the same without either using a foreach on the keys or values or using map() on the keys and accessing values in the block. If you need both the key and the value inside the block that is. More code and less elegance.

Closing points

Ruby's do..end syntax generally cleans up the language a bit, since this is in use for modules, classes, methods and blocks. Perl has only {} and this combined with other brackets and special characters can induce noise.

Combination of block or loop syntax with pattern matching can give a noisy appearance.

@array = ( [1, 2, 3], [4, 5, 6], [7, 8, 9]);
map { push @joined, sort { $b <=> $a } @{ $_ } }  @array;

and in ruby:

array = [ [1, 2, 3], [4, 5, 6], [7, 8, 9] ]
joined = Array.new
array.each { |x| joined.push(x.sort { |y,z| z <=> y }) }

Still not all that bad to a Perl hacker, but still.

Ruby OO strictness saves you a lot of typing if you're comparing to old school Perl OO. Moose takes much of that away nowadays.
Ruby's policy of optional parenthesis both on most method calls and if-statements removes some line-noise factor but can be less readable to the untrained eye. But you get used to it pretty quickly.

Keyboard layout and remapping for fun and profit

2009-07-22T10:18:27+02:00

After having lived with some annoying CAPS-LOCK mishaps and a general feeling of pain in the lower forearm, I decided to remap my keyboard layout for a smoother programming experience. Since I’m Norwegian and use a Norwegian keyboard layout, I have to resort to AltGr+0 to produce a }. Brackets are equally far away. Parenthesis are shift+8 and shift+9 on the number row and all this means moving the hand away from the home row a lot. I wanted to be able to program and maintain my hands in a more or less touch typing position.The goal was improved speed at programming and less discomfort. The fact that I daily program on both windows and linux ( both native and via the cygwin X server) made things a bit more complex than anticipated, as we will see. This article tackles re-mapping the caps-lock key in windows and creating a new keyboard layout in both windows and linux (Xorg that is). Info about re-mapping the caps-lock key under linux is just a google search away.

The following setup was used in this little adventure:

Windows XP running Cygwin.
Ubuntu 9.04 (as X client only)
Xorg server 1.5.7 with Xkb packages running in cygwin.

Caps lock remapping

The caps lock key is irritating and I’ve never used as far as I can tell. But I haven’t gotten around to doing anything about it. Now’s the time. First I thought about remapping it to control and binding a lot the characters I want more accessible to ctrl+ bindings. This messes with application shortcuts a lot so I went for altGr instead. I used a small program named KeyTweak to perform the binding. The result of running that program is that i new value is written to the registry. It looks like this:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]

"Scancode Map"=hex:00,00,00,00,00,00,00,00,02,00,00,00,38,e0,3a,00,00,00,00,00

Either apply the file or do it from within KeyTweak and the caps lock key is now an AltGr key. A reboot is probably in order to put it into effect. It is windows, after all.

Keyboard layout for windows

I used a Microsoft tool called Microsoft Keyboard Layout Creator which is a nice tool for creating a new layout from scratch or basing it on an existing layout. It has some limitations however in that you can’t remap things like caps lock and the windows key. But on the other hand it generates nice install packages for your keyboard layout. I decided on the following keyboard shortcuts (remember caps lock is now AltGr so it is easily available for the left pinky finger):

	AltGr+a = !
	AltGr+s = #
	AltGr+d = @
	AltGr+f = $
	AltGr+g = (
	AltGr+h = )
	AltGr+j = /
	AltGr+k = \
	AltGr+l = ~
	AltGr+ø = =
	AltGr+æ = +
	AltGr+u = {
	AltGr+i = [
	AltGr+o = ]
	AltGr+p = }

Keyboard layout for linux/ Xorg

When applying the custom layout for windows this worked well for native windows apps, but when logging into the linux servers and X-forwarding apps to the cygwin X server, it naturally didn’t work. A similar custom keyboard layout for the X server was needed.

I found a great detailed guide on creating keyboad layouts for X11/Xorg with XKB. There are some steps involved:

Copy the layout you want to base you new layout on to a new file in /usr/share/X11/xkb/symbols.
Tweak it to your liking.
Add it to the list of supported keyboard layouts in /usr/share/X11/xkb/rules/xfree86.lst
Restart your X server and use setxkbmap to set it, or add it in xorg.conf

The interesting parts of the symbols file looks like this:

key { [	u,	U,	braceleft,		uparrow ] };
key { [	i,	I, 	bracketleft,	idotless ] };
key { [	o,	O,	bracketright,	Ooblique ] };

This snippet maps e.g. AltGr+u to braceleft which displays a nice left curly brace. The format is broken down like this:

The key name AD07 is the 7th (07) alphanumeric (A) key from the left of the fourth row (D) from the bottom. Bottom row is A, second to bottom B and so on.
First column is character displayed when the key is pressed normally
Second column is character displayed when the key is pressed with shift down
Third column is character displayed when AltGr is down
Fourth column is character displayed when both AltGr and shift is down

Key names can be found in /usr/X11/include/X11/keysymdef.h

The keyboard layout is exactly the same as for windows and I just based it on the standard Norwegian layout and called it progn (pun intented). The entire progn keyboard symbols file is available for download. So now I have remapped the keyboard to my liking in both Linux and windows. I’ll probably tweak the keymappings a bit after some use, but generally I’m very happy with them.
References:

Videos from Lean & Kanban 2009

2009-07-02T10:17:34+02:00

A nearly complete set of videos of the talks at the Lean & Kanban 2009 conference in Miami is available at http://www.sep.com/lk2009 with talks from among others Corey Ladas on scrumban, Alan Shalloway on going beyond Toyota, David Anderson on Kanban, Karl Scotland on Kanban flow and cadence and Amit Rethore on why agile isn’t enough. All familiar names from the kanbandev Yahoo group which I started to follow recently.

Now I just hope I find enough time to watch some of them.

All About the Code

The Clash of the Classnames

The Effects of Automation

Steps to safer deployments

Going the Distance

Latest is such a bad idea

Low TTL

Undetected dependency changes

Which version are we actually running?

Reproducible builds

The Kubernetes Wars: Day 34

Other post in the series

Memory? Who needs it?

CPU? Who needs it?

Measurements

Reading Files in Bash

The Kubernetes Wars: Day 7

Other post in the series

DNS mofo

The Kubernetes Wars: Day 3

Other post in the series

Config

Container Metrics

The Kubernetes Wars: Day 0

Other post in the series

So What is Kubernetes?

Namespaces, Clusters and stuff

Scripts, scripts and more scripts

Talk Friday: Microservice Architecture

Deploy Overlay in Metric Graphs

Cache invalidation: explained

Setup

Cache headers in general

Extensions to cache-channel

Varnish Concepts

Headers

Implementation

Example

Some stats

A Usable and Robust Private Npm Repo

First version: replicating couchdb repo

Second version: reggie

Three’s a charm

The Metrics Approach

Getting your data into graphite

Display

Next Steps

Rapid Java Deployment

Background and setup

The New Way

Further Work

Helpful stack traces? Not so Much

The Tweet Storm

Context is King

JavaZone Video Mashup

Implementation

Headless Tests with Buster.js and Phantom.js

Enter Phantom.js

Installing Buster.js and Phantom.js

Running the Tests

Running on Jenkins

Blog Reboot

What I Create

Non-tech

Getting My Son to Read With Technology

The Plan

Results

A #smidig Retrospective

1. Pull based Meetings

2. Code Review as Step on The Board

3. Defining requirements quantitatively

JavaZone, New Job and the Case of the Missing Blog Post

Roots 2011 Braindump II

The 7 Duties

Graph Goodness

Secrets of Agile Architecture

HAML And SASS

Roots 2011 braindump I

A Way North!

Jim Webber: Rest in Practice