Graham Brooks

Working in Interesting Times: Generative AI as Disruptive Tech in Software Development

2023-07-03T00:00:00-06:00

As someone delving into the world of software development, the journey seemed smooth and set. With incremental releases, continuous integration, and agile methodologies, it appeared the path was that of constant and iterative improvement. But just as I started getting comfortable, a seemingly disruptive technology, Generative AI, came into the scene. This turned the world of development upside down, thus making life fascinatingly intriguing.

This blog post was almost entirely generated with the help of AI Assistant based on the following prompt to explore the possible.

A blog article in asciidoc format that talks about working in interesting times. Just when I thought that software development was set on a path of continuous improvement a disruptive technology of generative ai comes along to throw a spanner in the works and makes life really interesting

— Graham Brooks

Understanding Generative AI’s Disruption

Generative AI applications - like the OpenAI model, language translation algorithms, and automated music generators - are radically changing the tech landscape.

It’s no longer about improving existing code but generating entirely new code blocks potentially surpassing human-developed ones!

The idea might seem inherently unsettling, yet it’s the reality of our fascinating era of tech disruption.

// An example of a typical AI-generated Hello, World! program
public class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello, World!");
    }
}

The generative AI models are capable of creating such simple as well as complex programs, making it an exciting time for software developers.

The Challenge and the Excitement

It’s almost surreal - the algorithm capable of independently crafting the code - has put software developers on their toes. On the one hand, there’s the more significant challenge of mastering this new technology. On the other, there’s an unquenchable excitement to leverage this for developing highly sophisticated and efficient applications.

The Future with Generative AI

Will Generative AI replace developers? The answer is an emphatic no. Instead, it paves the way for a symbiotic existence. It allows developers to transition from coders to supervisors, overseeing and directing the AI in creating complex applications, thereby embracing an exciting future stacked with unimaginable possibilities.

Finally, it is indeed an interesting time to be a part of the software development skyline. The days might be challenging, the path might appear herculean, but there’s an undeniable allure to being a part of this revolutionary phase. Buckle up, for we’ve boarded a journey on the path of an enthralling tech adventure.

About the Author

AI Assistant is a software development assistant, offering insights into the fields of AI, machine learning, and software development. It helps developers navigate through the interesting yet challenging realm of disruptive technologies.

Iteration Zero - the first iteration

2022-11-25T00:00:00-07:00

There has been quite a lot of commentary on various social media channels about agile ceremonies and the traps that they often pose to enterprises. if not taken in the context of the value that they provide.

This is not a n agile problem. Agile principles are not agile ceremonies. Becoming or being agile is very different from doing agile.

Iteration Zero or sprint zero could be considered to be an agile ceremony fraught with the dangers we have seen with other activities that have been brought under the agile umbrella. Just like the other things in this umbrella, like stand-ups, retrospectives, and so on, Iteration Zero is a tool that can be used by a team to delivery value to customers more quickly and with better safety. It is not to applied blindly. There are likely cases and projects where it does not make sense but in today’s web based world it should be more of a decision not to use it.

Now that we have all the warnings out of the way lets explore the benefits of Iteration Zero.

What is Iteration Zero?

Iteration Zero is the first iteration of an agile project. Why Zero? The name comes from the common programming language practice of starting an index into an array or vector from zero instead of one - the cause of many off by one errors :) It is a non-functional iteration or sprint. Its goal is to set up everything we need to deliver working software into our production environment to be available to our customers. It is a basic test that we can operate software in that environment, know that it works and that our initial architectural assumptions hold true.

The most basic application can be built, tested and deployed
It is observable through logs and monitors
Any required analysis tools for security scanning, code defects and coverage are all operational and providing feedback.

Why should this work be done before our first development story?

Why spend time at the start of a project?

In the excitement to start development on a new project or product it is tempting to dive into building and exploring the features of that product. After all that’s the whole purpose of the work. So why spend time on infrastructure. Surely we can take care of that later when we know what the product is going to be doing and we know more about how it works?

The last mile problem

The last mile problem is that last step from build to deploy. After weeks or months of development and testing we finally have our nice new product ready to put in front of the customer (ignoring here that we should have been getting feedback from the customer all along). Everyone is excited and ready to launch!

It is at this point that reality hits. While we have been building and developing the application we have also been baking in assumptions and ignoring many of the concers that are key to running an application. Security, logging, tracing monitoring are all inconvenient requirements that all to often get ignored along the way …. until we hit the production deployment wall.

It is not uncommon for some fundamental requirement of deployment or operations to have been missed. Key architectural decisions become a challenge when the target environment just does not match what we have built into the application.

Sometimes addressing these concerns can take as long as the product development to resolve.

Even if we have built similar applications in the past and are using the same application stack setting up the path to production at the start of the project within the first iteration is a valuable proof and allows us to move forward with confidence.

How to make Iteration Zero work?

There are two outcomes

The path to prod is built triggered either 'on commit' for trunk based development or 'on merge' for feature branch development.
Monitoring and logging are in place and operational.

Both the path to prod and monitoring are maintained throughout the development.

We probably have an idea of what our architecture is likely to be. Let’s prove it. What are the distinguishing factors we are looking for in the stack and how do we test it to prove our hypothesis?

Tech Stack: Have we used the tech stack before? If not then lets stand it up and run our tests to make sure that it behaves the way that we need it to behave - Tests and more tests
Code management: Is the code for the new project to be managed as part of an existing monorepo or will it be a new repo? Will the infrastructure code be part of the smae repository or managed separately - GitOps style.
Build: Setup or configure builds. If we are using something new in the tech stack then make sure that and tools used during the build work the way we expect with the new tech stack. More testing.

Builds are taceable from commit and in reverse from deployment to commit so we know what changed to casue a problem as the development proceeds.

Test

Testing is built into the process and executed for every build

Unit testing
Functional testing
Benchmarking
Security scanning
Code coverage
Static analysis
Resilience testing

Even with a few lines of 'Hello world' code we can use that code to test our build and release pipelines.

Deploy: is the deployment repeatable. Does it match our availability targets minimizing any customer disruption. Can we scale to meet changes in demand (up and down).
Monitor: If something goes wrong how do we know? How do we know that it is fixed? How long does it take to recover. Longer term how to we maintain data integrity. How do we restore data in the event of data corruption?

Wrapping up

This may sound like a lot of work but remember this work will need to be done at some point in the project anyway. What we are doing is front loading the project with this work to reduce project risk, avoiding the last mile problem.

If you have ever been caught out by the last mile problem then you know how painful it can be. For your next project take the Iteratin Zero approach and see difference in launching your project.

If you are just starting out or have been lucky in avoiding the last mile problem then Iteration Zero may not make sense for you … yet. Remember iteration zero should be short and adds value throughout the project by increasing team confidence and ultimately reducing the project risk.

Kilpse Blogging

2021-09-11T00:00:00-06:00

Interactive web applications are always more appealing than static sites. This blog explores using the klipse plugin to add interactive code snippets to this blog.

Very often seeing the output of running code is key. I was really excited when http://lighttable.com became available . It seems to be a very quiet project these days but the output and value of variables while working on the code is really exciting - especially when writing unit tests. The RED/GREEN refactor loop running as you type is a real boost

Adding a code block to Jekyll blog using asciidoc

Asciidoc source code block

[code.language-klipse-go] (1)
---- (2)
import "fmt"

func main() {
fmt.Println("Hello all!")
}
---- (3)

1	Declare a code block with a language-klipse-go CSS class
2	Start of the code block
3	Terminate the code block

When run through asciidoctor it renders like this and is editable!!

import "fmt"

func main() {
  fmt.Println("Hello all!")
}

To activate the code to be interactive and run in the browser in the plugin we need to include the plugin CSS and JavaScript Asciidoc allows us to use a passthrough block that we add to the article to include the bits we need.

++++
<link rel="stylesheet" type="text/css" href="https://storage.googleapis.com/app.klipse.tech/css/codemirror.css">

<script>
window.klipse_settings = {
    selector_golang: '.language-klipse-go',
};
</script>
<script src="https://storage.googleapis.com/app.klipse.tech/plugin_prod/js/klipse_plugin.min.js"></script>
++++

A more elaborate example

A more elaborate example importing a library.

At the time of writing imports are not supported beyond the standard libraries. If you don’t see an error then it means that imports are now supported and please drop me a line.

import (
	"fmt"
	"github.com/grahambrooks/fingerprint/fingerprinter"
)

func main() {
    options := fingerprinter.Options{
        GuaranteeThreshold: 4,
        NoiseThreshold:     4,
    }

    fingerprint := fingerprinter.TextFingerprint("Some text as a source for the fingerprint", options)

    fmt.Println(fingerprint)
}

User-defined build settings for Bazel builds

2021-08-30T00:00:00-06:00

For the last couple of years I have been building command line applications in Go using Bazel. As part of the development workflow it’s important to run the application as it is expected to be run by other people - the customers. That means installing the CLI on to the path. Bazel works by building everything in a sandbox directory. This is great for keeping the source tree clean but makes installing localy slightly harder. One option would be to install the application after it has been built and published but that does not help out with experimentation - trying features and approaches that could end up as dead ends.

In this blog we explore using a custom command line arguments and some Starlark rules to install the cli as part of the build.

Official bazel/starlark documentation: https://docs.bazel.build/versions/main/skylark/config.html. You may notice the name skylark being used in directory names and some older documentation. Starlark is the official name. In the link above the url uses skylark but the documentation is all about starlark.

Problem

Lets say we want to be able to install a binary created by bazel into a local directory. Bazel uses a sandbox for all builds and provides a symlinks inside the bazel workspace for the sandbox. One option would be to just write a small shell script to copy the built application from the sandbox onto our local path.

If we chose this option then this would be a very short article. We would also run into some potential problems.

We would have to run the script manually and keep track of when the binary is built
The sandbox directory can change over time breaking the script

It would be convenient if we could define a rule to copy the output of a build to a convenient location and even better if we could provide that path on the command line. This avoids coupling into the sandbox and only copies the application if the app changes.

First lets work out what it takes to accept command line arguments in our build

Command line arguments

Native settings - settings that are defined within the bazel application by the bazel authors

Users can query the values (in build scripts) and change those values by passing the values as arguments.

User-defined settings are defined within your own workspace. The settings are defined in BUILD.bazel files and the supporting rules are defined in Bazel (.bzl) files. These setting’s rules are marked as flags and the flag values passed to the build on the bazel command line to customize the build.

Defining a user-defined build setting that can be passed through the bazel command line fits the bill. Using this approach we can use bazel run to install and specify where we would like the application to be installed.

TL;DR the installation command

bazel run //app/hello:install --//:local-install-path=/foo

Let’s first break down what’s happening and then walk through how this works with rule definitions.

bazel (1)
        run (2)
        //app/hello:install (3)
        --//:local-install-path=/foo (4)

1	The usual bazel command line application
2	run maps this to a run rule (the default) we could change this by defining a build or test rul but run seems to fit the semantics - run install
3	The application //app/hello to be installed with the :install target
4	Set the installation path to /foo on the workstation. `--` specifies that this is a command line flag `//` specifies that we are defining the value with an absolute path within the workspace adn :local-install-path references the required variable in the top level build file in the workspace. This syntax is not as clean as the usual *NIX style of --local-install-path but it is more flexible. We can define command line arguments at any point in the workspace tree and reference them uniquely even if they have the same name. It also avoids conflicts with native command line arguments.

The application build file

For this example we are building an application in Go. The app itself is trivial - prints 'Hello' to the console, but it is sufficient for this example. Bazel uses the Go toolchain to compile and link the hello application - all in the sandbox.

The directory structure is significant. Bazel uses the directory structure to infer the build file to use based on location.

bazel build //app/hello expects a BUILD or BUILD.bazel file in the app/hello directory that contains the WORKSPACE.

app/hello/BUILD.bazel

load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library") (1)
load("//:rules/defs.bzl", "local_deploy") (2)

go_library( (3)
    name = "hello_lib",
    srcs = ["hello.go"],
    importpath = "github.com/grahambrooks/building-with-bazel/hello/app/hello",
    visibility = ["//visibility:private"],
)

go_binary( (4)
    name = "hello", (5)
    embed = [":hello_lib"],
    visibility = ["//visibility:public"],
)

local_deploy( (6)
    name = "install",  (7)
    srcs = [":hello"],  (8)
)

1	Imports the rules for building go applications - library and binary
2	Imports our local build defintion for local deploy
3	go libarary rule to build the hello.go file
4	Matching binary rule that packages the library as a binary `hello`
5	Name for the binary. Must match the local deploy src argument value defined in local_deploy() call..
6	Call the `local_deploy` rule defined in //:rules/defs.bzl
7	the build target name to install the application `bazel run //app/hello:install --//:local-install-path=/foo` in our example. The choice of name is up to you as long as the characters you choose are acceptable to bazel for rule naming the hello app
8	The application(s) to install. `:hello` matches the go binary rule name in this build file.

We are not going to cover building the go app but instead focus on installing locally and being able to change the installation directory.

Defining the local_deploy rule.

The local_deploy rule is defined in a .bzl file and imported into the BUILD file using load(). Rules are typically defined in two parts the exported rule declaration and the implementation as a bazel function.

The rule is defined using the rule() method.

rules/defs.bzl

local_deploy = rule( (1)
    executable = True, (2)
    implementation = _local_deploy_impl, (3)
    attrs = {
        "srcs": attr.label_list(allow_files = True), (4)
        "_install_path": attr.label(default = ":local-install-path") (5)
    },
)

1	sets local_deploy to be the result of defining a rule. The result of calling rule is a function type that encapsulates the fields defined in the call.
2	Defines this as an executable rule that can be called with `bazel run`
3	Required implementation for the rule <see below>
4	Defines a srcs attribute for the rule that in our case accepts the generated binary files.
5	The installation path attribute as a label referencing a yet to be defined argument defined at the root level of the workspace.

Implementing local_deploy

rules/defs.bzl

def _local_deploy_impl(ctx): (1)
    target = ctx.attr._install_path[InstallDirectoryProvider].path  (2)

    shell_commands = ""
    for s in ctx.files.srcs:
        shell_commands += "echo Installing %s to %s\n" % (s.short_path, target)
        shell_commands += "cp %s %s\n" % (s.short_path, target) (3)

    ctx.actions.write(
        output = ctx.outputs.executable,  (4)
        is_executable = True,  (5)
        content = shell_commands,  (6)
    )
    runfiles = ctx.runfiles(files = ctx.files.srcs)  (7)
    return DefaultInfo(  (8)
        executable = ctx.outputs.executable,  (9)
        runfiles = runfiles,  (10)
    )

1	All rule implementation functions take a single argument ctx
2	Reads the target installation directory from the context using the defined provider [_defining_the_provider]
3	Build a string representing the contents of a shell script that bazel will pass to the shell for execution.
4	Write the output to be executed.
5	Mark it as executable.
6	Set the content of the action to be our generated shell script text.
7	Make sure that the source binary files are available to the shell when the action in run.
8	Return a provider to bazel which is called to complete the action
9	include the executable to be run
10	and the runfiles to execute agains.

We now have everything except the target path. For this we are going to define a provider.

Defining the provider

Out of the box bazel supports boolean and string build settings. Other types can be defined with custom user defined functions (out of scope for this article)..

There are several functions that return providers. The DefaultInfo provider in local_deploy is a provider returned from a rule. For our purposes we define a provider type using the provider method. See Providers.

InstallDirectoryProvider = provider(fields = ['path']) (1)

1	Create a new provider constructor that builds structures (objects) with a single field `path` e.g. `InstallDirectoryProvider(path="/usr/bin")` returns an object with a `path` field set to the string value "/usr/bin".

We need to define a rule that represents the command line flag. The rule needs an implementation and a build_setting parameter. build_setting is a config type. For a command line flag that accepts a path we use a config.string and set the flag parameter to true so the value can be set via the command line.

Rule Declaration

install_path = rule(
    implementation = _install_path_impl, (1)
    build_setting = config.string(flag = True) (2)
)

1	The implementation of the rule
2	Mark this rule as a build setting rule that can be defined/overridden on the command line.

Rule Implementation

def _install_path_impl(ctx):
    return InstallDirectoryProvider(path = ctx.build_setting_value) (1)

1	For our use-case the rule takes the given value and wraps it by calling the provider to return an object. The `build_setting_value` is automatically defined because we defined the `build_setting` parameter when we defined the `install_path` rule

Defining the command line flag in the top level BUILD file with public visibility means that we can use the flag from multiple build rules for components in the repository.

BUILD.bazel

load("//:rules/defs.bzl", "install_path")

install_path(
    name = "local-install-path", (1)
    build_setting_default = "/usr/local/bin", (2)
    visibility = ["//visibility:public"], (3)
)

1	The name value defines the name of the command line flag
2	Set the default argument value to `/usr/local/bin`
3	Default visibility (`//visibility:private`) would mean the value would only be accessible within the top level build. By setting public visibility rules can access the value anywhere in the build.

Running the app and install

bazel run //app/hello (1)
bazel run //app/hello:install (2)
bazel run //app/hello:install --//:local-install-path=foo (3)
bazel run //app/hello:install --//:local-install-path=/foo (4)

1	Run the application - in the sandbox
2	Run the install. Installing the built application into the default location.
3	Install the application into a local path in the sandbox - fail
4	Install the application into `/foo`

Cleaning up the command line

--//:local-install-path is a pretty inconvenient and not what most applications use. Bazel allows us to define aliases for command line arguments, and it is convenient to define these in a .bazelrc file like so:

.bazelrc

run --flag_alias=local_install_path=//:local-install-path  (1)

1	Defines an alias when running `bazel run` so we can use local_deploy_path. Note the underscores '_' in the alias.

bazel run //app/hello:install --local_install_path=/foo

Much more idiomatic for command line use. Aliasing also means that command line arguments can be defined anywhere in the workspace tree and then aliased for convenience.

We don’t have to stick to a single alias.

.bazelrc

run --flag_alias=install=//:local-install-path

bazel run //app/hello:install --install=/usr/local/bin

Found this article interesting? Found a bug or want more info? Drop me an email or comment below.

Text fingerprints

2021-06-06T00:00:00-06:00

I have always be fascinated with how things work. In my early years my focus was on the things around me. Bicycles, Motorcycles and then cars. When I was first introduced to a computer I was hooked and had to know how things work.

That was a long time ago but there are still times when I discover something and just need to know how it works. Typically this means working it out by building something.

Software development at scale presents its own unique challenges that mean manual inspection or review are just not viable. Pattern recognition for the humans is what we do. Machines tend to be a little more pedantic in comparing two things.

The source code for this article can be found here https://github.com/grahambrooks/fingerprint and the original article here http://igm.univ-mlv.fr/~mac/ENS/DOC/sigmod03-1.pdf

Using k-grams and winnowing (selecting a single entry from the k-gram) hides details that then allows us to compute similarity. if we don’t remove something from the original text or stream of data then we are comparing and not identifying similarity.

A worked example

The following gist scans the filesystem for readme files, computes a fingerprint for the contents of each file and then looks for pairs of files that have similar fingerprints.

For files with a similarity score above 0.75 the gist uses a diff library to highlight the differences between the files that are shown as similar.

package main

import (
	"fmt"
	"github.com/grahambrooks/fingerprint/fingerprinter"
	"github.com/grahambrooks/fingerprint/similarity"
	"github.com/sergi/go-diff/diffmatchpatch"
	"io/fs"
	"io/ioutil"
	"os"
	"path/filepath"
	"strings"
)

type FileFingerprint struct {
	path        string
	fingerprint fingerprinter.Fingerprint
}

type Clusters []FileFingerprint

func main() {
	options := fingerprinter.Options{
		GuaranteeThreshold: 4,
		NoiseThreshold:     4,
	}

	var matchingFiles Clusters

	scanRoot := os.Args[1]
	fmt.Printf("Scanning %s\n", scanRoot)
	filepath.WalkDir(scanRoot, func(path string, d fs.DirEntry, err error) error {
		if err != nil {
			fmt.Println(err)
			return nil
		}
		switch {
		case d.IsDir():
			switch d.Name() {
			case "node_modules":
				return filepath.SkipDir
			}
			return nil
		case strings.EqualFold(d.Name(), "readme.md"):
			fallthrough
		case strings.EqualFold(d.Name(), "readme.adoc"):
			content, err := ioutil.ReadFile(path)
			if err != nil {
				return err
			}
			fingerprint := fingerprinter.TextFingerprint(string(content), options)
			matchingFiles = append(matchingFiles, FileFingerprint{path: path, fingerprint: fingerprint})
			return nil
		default:
			return nil
		}
	})

	for i, f1 := range matchingFiles {
		for j, f2 := range matchingFiles {
			if i != j {
				s := similarity.Compare(f1.fingerprint, f2.fingerprint)
				switch {
				case s == 1.0:
					fmt.Printf(`
near identical
    %s
    %s
`, f1.path, f2.path)
				case s > 0.75:
					fmt.Printf(`
similar
    %s
    %s
`, f1.path, f2.path)
					dmp := diffmatchpatch.New()

					content1, err := ioutil.ReadFile(f1.path)
					if err != nil {
						return
					}

					content2, err := ioutil.ReadFile(f2.path)
					if err != nil {
						return
					}

					diffs := dmp.DiffMain(string(content1), string(content2), true)

					fmt.Println("================================================================================")
					fmt.Println(dmp.DiffPrettyText(diffs))
					fmt.Println("================================================================================")
				}
			}
		}
	}
}

This example is not very practical. At the moment the fingerprint is represented as an array of structs. Finding pairs that means O(n²) solution that is just way too slow for anything over a small number of files.

The fingerprinting library has only been tested on small samples and a very small set of options. Tuning and testing recommended!

Installing built binaries locally with Bazel

2021-05-31T00:00:00-06:00

I quite often use Bazel to build Go command line applications. One of the annoying aspects of the bazel workflow is that the binary is built in a sandbox - a completely separate directory structure. Although we can run the binary using bazel the current working directory of the running app is in the sandbox making it harder to use the app in your current directory.

Adding a command line flag to change the current working directory is another way to run your app through bazel in the right directory but this means changing the application to work around the issue.

Bazel builds and runs everything in a sandbox that is connected to the project workspace with a symlink (on linux and mac). This approach makes it a lot easier to keep the source tree clean of build artifacts but makes it hader to just run the built binary.

In go for example go build main.go generates an executable binary main in the current directory making it really easy to add this to source control, or some other inadvertent action. By working in a sandbox this is avoided. It does make running that generated binary a lot harder.

One solution to this problem is to have Bazel install the built binary onto the local path as an additional target for the command line binary BUILD.bzl script.

The local deploy rule takes the generated binary in the sandbox and copies it to a convenient location on your local path e.g. /usr/local/bin

Source: https://github.com/grahambrooks/snippets/tree/main/bazel

Example BUILD.bzl

In this Go example build file as well as building a go library and binary the script also defines an 'instal' target that copies the binary to an installation directory.

load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
load("//rules:local-deploy.bzl", "local_deploy")    (1)
go_library(
    name = "golang_lib",
    srcs = ["main.go"],
    importpath = "github.com/grahambrooks/building-with-bazel/book/examples/golang",
    visibility = ["//visibility:private"],
)

go_binary(
    name = "golang",
    embed = [":golang_lib"],
    visibility = ["//visibility:public"],
)

local_deploy(
    name = "install", (2)
    srcs = [":golang"], (3)
)

1	Include the local deploy rule definition function into our build script
2	Call the rule something unique
3	Reference the built binary that is to be deployed to the target folder.

Defining a Bazel Rule

Bazel works with rules that are defined by calling the rule() method. The 'local_deploy' rule definition links the implementation method _local_deploy_impl and the expected input sources and target with a default value and type.

local_deploy = rule(
    executable = True, (1)
    implementation = _local_deploy_impl, (2)
    attrs = {
        "srcs": attr.label_list(allow_files = True), (3)
        "target": attr.string(default = "/usr/local/bin", doc = "Deployment target directory"), (4)
    },
)

1	Mark to be run using `bazel run [label]`. This attribute means that the rule implementation generates an executable that is run by bazel
2	Reference to the implementation function (see below)
3	The list of files to be deployed - typically the binaries generated by another rule
4	Overridable parameter that is built into the executable. The directory for the deployment

Local Deploy Implementation method

The bazel implementation function creates an executable script that is then run by bazel to do the actual installation. This is the heart of the implementation. The rule is reusable and parameterized in each BUILD.bzl file that uses it. The implementation method is run by bazel for each use and ends up generating a shell script that is run by bazel when the binary is (re)built.

def _local_deploy_impl(ctx):
    target = ctx.attr.target (1)
    shell_commands = ""

    for s in ctx.files.srcs:
        shell_commands += "echo Copying %s to %s\n" % (s.short_path, target)
        shell_commands += "sudo cp %s %s\n" % (s.short_path, target) (2)

    ctx.actions.write(  (3)
        output = ctx.outputs.executable,
        is_executable = True,
        content = shell_commands,
    )
    runfiles = ctx.runfiles(files = ctx.files.srcs) (4)
    return DefaultInfo( (5)
        executable = ctx.outputs.executable,
        runfiles = runfiles,
    )

1	Sets the rule to be executable.
2	The shell command that actually does the work copying the built binary into the given target
3	Generates a script that is run to install the built binary.
4	Sets the run files for the given input
5	The result of the rule - a scrpt that is run by bazel to install the binary.

Running the installation

Once the local_deploy target is added to the BUILD.bzl file we can call the installation script by adding the :install target.

> bazel run //golang:install

For some systems we have to use sudo to write to /usr/local/bin so you will be prompted for your password. Once installed and assuming that /usr/local/bin is on your path

> golang

runs the built application

Summary

Bazel is slightly different than most build systems that generate libraries and binaries into the local filesystem with the source. If we build something in the local filesystem we typically wrap the build in something like a makefile that then copies the generated files to install them locally. We could copy the files from the symlinked sandbox but this creates some pretty tight coupling between the bazel owned sandbox. A makefile for example would need to know the bazel sandbox directory structure which is based on the operating system architecture.

Having Bazel do the installation work is a lot simpler and avoids this coupling.

Go and Bazel to build containers

2020-04-05T00:00:00-06:00

Yesterday I spent some time experimenting with building containerized Go applications. Writing a small restful web service in go is really quite straightforward and there are many better examples out there on the web.

What was interesting for me was seeing how Bazel supports a typical workflow building and running containerized web services. In this case using a small Go program.

Bazel is still relatively new but gaining in popularity. Over the last 6 months or so stability and functionality has really improved. Bazel really shines when combined with a mono-repo approach. For this example the bazel configuration feels more than required to achieve the result.

Components used in Building the Go service container

Pre-requisites

Install Docker (required for both local and container builds) Docker
Bazel (required for local build) Bazel

Up and Running

> docker images (1)
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

> bazel run //api:api-container --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 (2)
INFO: Analyzed target //api:api-container (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //api:api-container up-to-date:
  bazel-bin/api/api-container-layer.tar
INFO: Elapsed time: 0.605s, Critical Path: 0.01s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
932da5156413: Loading layer [==================================================>]  3.062MB/3.062MB
dffd9992ca39: Loading layer [==================================================>]  15.44MB/15.44MB
43babe50bd4f: Loading layer [==================================================>]  6.246MB/6.246MB
84ff92691f90: Loading layer [==================================================>]  10.24kB/10.24kB
Loaded image ID: sha256:525fa22e2c6beccba45e2a0dbc7370d7d808342785d6d8b825096ce0a338279f
Tagging 525fa22e2c6beccba45e2a0dbc7370d7d808342785d6d8b825096ce0a338279f as bazel/api:api-container

> docker images (3)
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
bazel/api           api-container       525fa22e2c6b        50 years ago        23.1MB

> docker run -p 8080:8080 bazel/api:api-container (4)

1	Current list of installed images - none
2	Run bazel to create and tag the container. Not the target flag - because the api will be running in a linux container we need to build a binary that will work in the environement
3	The container image is now available in my local docker install.
4	Fire up the application container and expose port 8080

Testing the container

> curl http://localhost:8080/hello/Everyone
{"Message":"Hello","Name":"Everyone"}

This short blog records my experiments in building a container using Bazel containing a Go application.

The API

is a simple JSON service that accepts /hello/:name and returns a json response with a message and the name given in the request URL:

The full program

package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
)

func main() {
	router := mux.NewRouter().StrictSlash(true)
	router.HandleFunc("/hello/{name}", helloHandler).Methods("GET")

	log.Fatal(http.ListenAndServe(":8080", router))
}

type helloResponse struct {
	Message string
	Name string
}

func helloHandler(writer http.ResponseWriter, request *http.Request) {
	writer.Header().Set("Content-Type", "application/json")
	vars := mux.Vars(request)
	r := helloResponse{
		Message: "Hello",
		Name:    vars["name"],
	}

	encoder := json.NewEncoder(writer)
	_ = encoder.Encode(r)
}

Expected JSON response

{
  "mmessage": "Hello",
  "name": ":name"
}

Bazel

Bazel is a little more difficult to explain.

Project layout

.
├── BUILD.bazel (2)
├── Dockerfile (6)
├── WORKSPACE (1)
├── api
│   ├── BUILD.bazel (3)
│   └── main.go (4)
├── go.mod (5)
└── go.sum

1	The project workspace. Loads the Bazel Skylark components and go dependencies for the project
2	Helper build for Gazelle dependency management
3	The API build script including targets for the API, container image and container
4	API service source
5	Go module dependencies. If the go.mod file is changed you can update the bazel dependencies with `bazel run //:gazelle — update-repos -from_file=go.mod`
6	Containerized Bazel build

It is both a build system and an environment for plugins supporting application build.

Bazel build rules for go

https://github.com/bazelbuild/rules_go

Gazelle is used to generate Bazel dependencies from the go.mod file. These dependencies appear in the bazel WORKSPACE file in the root of the project and then referenced in the BUILD.bazel files for each module.

Building a container in a container

Building software in a container has several advantages. Docker container images are just files and those files can be created without docker. The Bazel docker components don’t need docker installed to build an image. Using the same command line

Dockerfile

FROM l.gcr.io/google/bazel:latest

WORKDIR /build
COPY . /build

RUN bazel run //api:api-container

Running the containerized build

> docker build . (1)
Sending build context to Docker daemon  107.5kB
Step 1/4 : FROM l.gcr.io/google/bazel:latest
 ---> dc530fa1c5ce
Step 2/4 : WORKDIR /build
 ---> Using cache
 ---> 9e3158821eed
Step 3/4 : COPY . /build
 ---> 3d309ccc2b35
Step 4/4 : RUN bazel run //api:api-container (2)
 ---> Running in 7d632d5bb256
Extracting Bazel installation...
Starting local Bazel server and connecting to it...

... omnitted logs ...

[0 / 35] [Prepa] Creating source manifest for @io_bazel_rules_docker//container/go/cmd/create_image_config:create_image_config [for host]
[36 / 45] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 4s processwrapper-sandbox
[36 / 45] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 14s processwrapper-sandbox
Target //api:api-container up-to-date:
  bazel-bin/api/api-container-layer.tar (3)
INFO: Elapsed time: 64.211s, Critical Path: 19.27s
INFO: 27 processes: 27 processwrapper-sandbox.
INFO: Build completed successfully, 45 total actions
INFO: Running command line: bazel-bin/api/api-container.executable
INFO: Build completed successfully, 45 total actions
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (4)
The command '/bin/sh -c bazel run //api:api-container --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64' returned a non-zero code: 1

1	Running docker instead of bazel to build the project
2	Build and runtimes match so we don’t have to specify the platform for the bazel build
3	The container file is created but because we are running in a container without docker the
4	image is not loaded or tagged

The bazel docker rules provide ways to upload the built image - something to look into next …

Wrapping up

There’s a lot more to experiment with and this article only scratches the surface.

Building container images without having to install Docker simplifies the build process
Running the build in a container minimizes local development configuration management and keeps project isolated. This is particularly important when projects depend on different runtimes.

Technical Health over Technical Debt

2018-12-27T00:00:00-07:00

Ward Cunningham is credited with the first use of the team Complexity as Debt or Technical Debt (YouTube video) as a metaphor poor software implementation. For a software application or system technical debt refers to the internal code quality. We have accrued Technical Debt when the implementation does not match current requirements or when implementation shortcuts lead to complex and coupled code. Over time people have added more and more items to the general bucket of technical debt.

Term Technical Debt has become very popular. Technical Debt has become the universal metaphor for characterizing refactoring work. Work that is technical in nature and not linked to features and functions. As a useful metaphor 'Technical Debt' has stood the test of time and is now used as an argument for driving investments in internal quality changes.

Martin Fowler describes technical debt and provides a categorization model in his article Technical Debt Quadrants. The article focuses on debt sources. There are many views and perspectives on technical debt. Bob Martin on A Mess is not a Technical Debt. Out in the wild Technical Debt is used for almost anything not related to feature function delivery.

In my own work, I have considered how categorization of and code volatility analysis can be used to guide when and how to address technical debt Metrics Based Refactoring

When Ward came up with the Technical Debt analogy software development and online systems were very different to today. The ecosystem that ran the software was much more static. Servers were bought and manually maintained. A lot of software running on desktops would be released once or twice a year. Patches and small updates were infrequent.

Today the number of software updates each day, week or month is mind-boggling compared to a decade or two ago. When there are hundreds of releases a day planned technical debt does not work. Planning technical debt work is not an option in today’s software development world.

Technical Health

A few years ago I heard the term Technical Health as an alternative term to Technical Debt. At first, I thought it was only a more positive view of the same concept.

Recently I have come to think that the technical health metaphor has a better fit than technical debt.

First, let’s look at some of the challenges with using Technical Debt as a metaphor:

The word debt is powerfully linked to financial management. Most consumers manage credit card debt. This direct connection between technical and personal financial debt. This direct connection is what makes the metaphor so powerful. As Ward points out building up a little debt can be a very useful thing in meeting objectives.

A challenge I see with this financial connection is paying off the debt. Technical debt is different. It often takes a long time to address even a small amount of debt built up in the code base.
We think of taking on debt as a deliberate act. Borrowing money today to take advantage of a situation or meet an obligation. Martin Fowler’s quadrant model only deliberate and prudent borrowing, meets this classification. Most technical debt does not come from deliberate and prudent borrowing. Today’s software exists in a toxic ever-changing environment. Software systems are more of a liability than an asset because it requires frequent updates and maintenance to survive. Tightly-coupled complex software is harder to change and update.

Maintaining working software in production environments is no longer an occasional activity. We have to think about maintenance all the time. Putting it off until later gets out of control to fast. 'Technical Health' implies continuous activity, making it more relevant to today’s software development.
Getting fit and healthy takes both effort and time. If you have an unhealthy codebase it is going to take a while to tackle the challenges. If the code base is unhealthy it is unlikely that you can clear up all the problems. Adding unit tests to an existing code base improves maintainability for example. Adding unit tests to an existing codebase provides a regression test suite. Changing legacy code is hard. It is much harder to change the legacy code so it compares with code developed through Test Driven Development. Sure you have better test coverage but this is more from a regression perspective.

Many organizations managing technical debt feel that an occasional investment fixes things. The expectation then is that it’s ok to run up debt between these investments. One-off or occasional investments encourages this behavior. 'If you have the cash you can solve debt at any time'. It’s a one-time thing and once paid up the code base is debt free, for now. Unlike financial debt, it is very difficult to recognize a zero balance. The investment runs out before all the problems solved. This background debt keeps building up.
Maintaining health is a cultural, habit-driven change. You can skip exercise for a while but it is then harder to get back into and will be harder when you return. Diet and exercise are an ongoing investment in time energy and discipline. The same is true for a healthy codebase. Time takes its toll. The software is more of a liability.
Libraries and operating systems need upgrading to handle security issues. Sitting on the shelf software deployability degrades.
Becoming healthy (fit) and maintaining it is hard (or at least it is for me).

Humans choose immediate gains over greater future gains. Making investments today that only provide benefit later is much much harder. That benefit may not be for ourselves it may be for some future team or others.

Refactoring is often done at the end of a development cycle. Cleaning up the code that now meets functional requirements. The benefits are found in the future. It is easy to say done when the code works - even if it is not clean or healthy.

We know that the security issues are a possible future problem but getting the next release often pushes such concerns down the road. Until they become a real problem and then we have the equivalent of a heart attack in software and have to drop everything to solve the problem.

If only we had spent just a little bit of time getting better at being healthy we would not have to contend with these crises.

The key difference between these two metaphors (IMO) is that Technical Debt focuses on explanation, Technical Health focuses on the ongoing change of habits. The debt metaphor is often interpreted as a single one-time transaction to pay down the debt. In the worst case, this might be the rewrite. Technical Health drives behavioral changes. Changes in day-to-day activities for code improvement.

Thinking about technical health promotes the changes necessary for changing habits. Making those changes leads to cumulative benefits over time. Making maintaining a healthy codebase a daily habit puts health generating work into a day of feature development work. They are not separated.

Ward Cunningham rightly points out that taking on technical debt to meet an urgent objective is often the right thing to do. Martin Fowler might place this in the Deliberate and Prudent quadrant. It is planned and understood to meet a goal. Teams taking on this kind of debt know that they will address the source code problems they introduce to meet the goal. This fits in well with the health metaphor. This might be considered as taking a break from the gym to finish up some urgent work. As a one of it does not affect long-term code health. It becomes a problem when skipping the gym becomes the norm because urgent delivery is every day.

Talking about technical health is positive. Talking about technical debt is all about a negative past. Developing a healthy code base is also positive but eliminating technical debt negative.

People are generally motivated to make things better. Cleaning up debt feels like a chore.

Focusing on the positive drives toward a better future. It’s about providing code goodness to your future self or to future colleagues.

Talking about technical health, what it means and how to achieve it is a better organizational goal than managing debt.

Of Languages And Containers

2018-12-09T00:00:00-07:00

At the start of my career I worked with fairly dumb terminals and mainframes. Programs were natively compiled but resources were limited and shared with the other users of the system. Shared computing power meant that we favored languages that were efficient typically compiling to binaries.

As personal computers made their appearance developers and users had much more computing power at their disposal. That power was reflected in the servers that powered the early internet. With the increase in computing power both memory and CPU meant that less efficient but more effective runtimes could be supported. Java, .NET, Ruby all benefited from this additional power.

Software development is a continuously evolving landscape of capabilities and I think we are a new juncture.

Light weight containers and the rate of change with cloud providers the drivers for efficiency are becoming more and more important.

Cloud vendors and their customers are realizing that compute density, efficiency and startup speed are becoming more and more important. AWS lambda support for Rust and C++ is an interesting move. With first class support for efficient natively compiled applications the vendor wins by reducing waste in spinning up new instances and cloud customers can maintain a lower cost for runtimes that may balance out the higher cost of development.

Developer Workflow

It has always been important for developers to run and evaluate their work quickly. As capabilities move from application/service code into infrastructure it becomes more difficult to run a complete stack locally. As cloud providers develop more and more compelling cloud capabilities the ability to run a complete application on a development machine becomes harder. Maintaining a local development environment with the complexity of a cloud is both costly, complex to maintain and unlikely to be completely 'production like'.

As an alternative to 'runs on my laptop' there is a growing trend towards it runs on my environment - a cloud based extension to the developer workstation. In this model code is written locally, built but then deployed to an environment for evaluation. It’s unsurprising that the lines between local IDE development and cloud runtimes are blurring with inbuilt support for remote deployment and debugging.

Dilemma

Software developers are now faced with a dilemma. Continue to develop test frameworks and stubs that emulate the runtime environment of their applications or embrace the rapidly evolving environment provided by the cloud vendors for development. Command line tools and integration with IDEs improves the impediments to using cloud services as an extension to the developer workstation becomes ever more tempting. Developing test rigs that run on the workstation only become available later. As the capabilities develop trying to mimic an entire cloud providers services becomes more expensive.

It has been a long time since I last developed 'on a mainframe' as development environment. How well does the modern day cloud environment match up to developer expectations for local development and test?

Orchestration And Choreography

2018-04-29T00:00:00-06:00

Orchestration and Choreography are often discussed as design options for event driven systems. Some use the term interchangeably while others feel that the two are very distinct. I am in the later group in thinking that the two are very distinct and lead to very different implementations and outcomes.

When thinking about these two patterns I find it helpful to relate their behaviors to the most popular implementations - music and dance.

In orchestral music the conductor is a key component to a successful production. Each person in the orchestra seeks guidance from this central figure. The conductor responds to actions by indicating shifts of behavior and actions from other sections.

In choreography the choreographer’s role is most dominant before the performance and does not take an onstage role during the performance. Dancers perform based on pre-production direction and practice but respond dynamically during the performance based on the actions take by the other dancers.

Relating this to event driven systems

In event driven systems we might consider a Process Manager as a conductor orchestrating the event process flow in production. For Event Choreography systems there is no role beyond the services making up the production performance.

Let’s consider two systems with identical functionality. One build with an orchestrating Process Manager and one based on choreography. Often in software complexity is discovered over time - versioning, additional requirements, changed requirements, system upgrade.

Change

By change I mean change that affects other components or services. Registration for example.

Let’s assume that we have two services. One to handle registrations and the other to handle accounts. Let’s also assume that we have a service that handles sending emails. New requirements mean that we need to send a welcome message when the someone registers and a new account has been opened for them. In Orchestrated systems the Process Manager would be updated to handle the RegistrationCompleted message. The Process Manager would call the email service after creating the account. In choreographed systems the email service would be updated to listen to the RegistrationCompleted message and send the email. The changes are very similar in both cases but where those changes are made are very different. Our 'placement of function' changes radically.

Services oriented architectures allow for service changes. Let’s say we are adding an SMS or messaging service that allows users to receive SMS instead of emails for some more real-time activities like registration.

In orchestrated systems the process manager might be make this decision based on the preferences captured during registration. In choreographed systems both the email and SMS services require updating. The email service now needs to account for user preferences. In the case where a user opts for SMS the email service can ignore the event and 'assume' that there is another service that picks up the required communication. In our case this would be the SMS service.

Service Orchestration handles sequential processes well. Taking action when an event is received and then calling other services to complete the process. Service Choreography does not really support these sequential event/action sequences across multiple services. Each service needs to be independent and events at a level that give enough information that recipients and process without reference to others. I am assuming that services don’t call other services here.

Service Choreography requires more careful design because each component in the systems needs to be aware of the events around it an those events need to be at an appropriate point and level within the overall business process. Events need to contain sufficient information for imagined recipients to take action without collecting their own view of state. Service Orchestration is much simpler to think about. A process manger handles actions based on events, can handle failures, availability, reporting, workflow and a myriad of other responsibilities. Unfortunately over time without constraints it is very easy for the Process Manager to take on many system attributes and capabilities and can grow out of control.

Wrap-up

Choosing the right design pattern for event driven systems is hard. Moving from a choreographed to orchestrated is much simpler than orchestrated to choreographed so the decision needs to be made earlier in the development life-cycle.

I have a natural bias to the lower coupling and higher availability provided by choreographed event driven systems and accept the additional design effort in crafting services with sufficient scope to make choreography practical.

Quantifying Commit Risk

2018-02-17T00:00:00-07:00

Risk is typically expressed as a function of likelihood and severity or impact. Probability is an important factor but this article is about understanding the potential impact of change on a production software environment.

All changes carry an element of risk. Key to managing that risk is understanding and quantifying the risks inherent in a change. Not all changes present the same level of risk. A single character change in an HTML file or template is much lower risk than the same level of change in a configuration setting that affects the behavior of the entire application or service. Let assume that each cause a problem. In the first case a error affects a single view. In the later the impact could be the entire application crashes if the configuration setting is unexpected in production.

Just categorizing by the type of file is insufficient. If that one small HTML change makes the login page break then it could have at least the same impact at the configuration file change.

By taking deliberate steps to quantify risks we can add additional measures to the development process in a more focused way.

Risks can be reduced. Building automated tests around the changed behavior reduces the risk that the change will not behave as expected. Adding inspection processed (code reviews) or pair programming help reduce the risks of unexpected behavior but still there is an element of risk.

In general

Small changes are lower risk than large
Tested changes are lower risk than untested changes
Changes with more dependencies are higher risk than those with fewer

Continuous Delivery (CD) and Test Driven Development (TDD) are techniques that significantly reduce the first two issues. Having rapid recovery practices can, to some degree, reduce the impact of the third.

In a typical Continuous Delivery build and release pipeline each change is assessed through automated testing. Each change is added to a strong known foundation because the pervious version is of known quality: passing tests. With continuous deployment each incremental change is deployed into production reducing deployment risks.

Are we there yet?

For organizations working towards continuous delivery and deployment the risk of problems occurring in production is typically much higher during the transition because the testing just isn’t there yet.

Existing software was not developed with Continuous Delivery in mind. Automated test coverage might be low and provide little confidence. Releases often contain a large number of changes that have been tested as a complete set. As delivery cycles are reduced more attention needs to be applied to the release. Given limited testing capacity risk evaluations provide a valuable tool for focus efforts.

Yesterday’s weather

Some code can be particular difficult to change without causing problems. Some code are bug magnets. In some cases is may not be worth going through a significant refactoring effort to make the code more maintainable. If a particular file or module has caused the last 9 out of 10 problems then statistically another change is 90% likely to cause a problem. We should be using bug data to predict the likelihood of a future problem and pay particular attention to testing that area.

Quantifying risk input sources

Assessment of impact by affected source type (HTML/JS/Java/SQL/Config)
Assessment of dependencies - how many other areas depend on the change area
Defect resolution data for areas that have proven problematic in the past.

Implementing a risk assessment

Lets first assume that the potential impact in a production change is the sum of the potential impacts of the files changed as part of the change. To quantify the risk we first need to know what files were changed and then an assessment of the potential impact of those file changes so we can sum them up to an overall risk assessment.

One of the most complex parts to implimenting a risk assessment tool is knowing what changes to include.

We all know that software systems need to be versioned but that versioning needs to be traceable to the original version control revision and branch. Only then do we know the files changed between that revision and the revision last deployed in the environment.

Environment changes often involve multiple components. Decomposing systems into smaller parts using techniques and designs like microservices help to contain impacts to a single source repository. This significantly improves our ability to assess deployment risk.

Monolith or microservice from a risk point of view the problem is the same. Compile a list of files that have changed since the version deployed in that environment. Assign a risk value based on the file type or role and then sum up all those changes.

Reverse tracing changes

Some teams use the revision ID as the build number or least significant part of the version number. In git that means using the commit sha (sort or long), but what about components or services built from multiple repositories? If a deployment involves multiple dependent system changes it is common to define the deployment using a release manifest. A list of components and their versions. Armed with the list and versions that can be traced back to version control we can identify all the files that have changed.

Another approach is to add the versions into the asset itself either as a file containing all the contributing revision numbers and repositories or some form of meta data in properties. Lets assume that a deployable component is built from multiple source code repositories. The dependencies could be binary as libraries or through source as sub-modules.

Yet another approach is to add the risk assessment to the component. This last one seems attractive but relies on the production (last known good) version to remain stable. If an interim release is made then the assessment is out of date (although arguably it is airs on the side of pessimism).

Once the two revisions are know a simple diff generates a list of changes. For git adding the --name-only flag produces a list of files that have changed between the two revisions.

From the list we can apply recognition patterns for files that are considered higher risks. The patterns can be generated by agreement within the team on areas that are particularly sensitive to change or from past experiences and bug reports.

The path to a million (loc)

2017-07-07T00:00:00-06:00

Over the last month or so I have been writing some data collection and simple analysis routines for the things we (developers) use every day. The ultimate goal is to have a system that can collect, interpret and recommend actions based on data from source code, version control, build and other systems related to software development. The project is kuona.

GitHub has 1000s of project repositories publicly available. This article is about one of them elasticsearch

kuona is using elasticsearch for data storage so it made sense for elasticsearch to be the an initial candidate.

GitHub already provides a contributor graph page which is a useful comparison for the first chart generated - Number of commits per day over the lifetime of the project.

Nothing really surprising here and a reasonable correlation to the github version.

The next chart is a little more relevant to the title. Lines of code for each commit on the master branch.

The vertical purple lines indicate days when there were no commits. What’s interesting and requires more input is that there are some pretty significant code change (reduction points)

Given the scale of the chart these code drops are pretty significant. Probably due to refactoring.

Elasticsearch has been in very active development starting in 2013 - number of commits per day increased by between 2x and 5x the previous years. There are some significant spikes of commits (148 Aug 27 2014). The code growth reflects the commit rate and started accelerating at the end of 2014.

Just looking at these charts makes me want to look at other areas.

Is code duplication at a healthy level. Code is being added faster than it is being removed.
Is the volume of code making it harder to resolve issues?
What is the ratio of test code to production code?
How has build time changed over time?
How does elasticsearch compare against similarly sized code bases?

Next steps

Currently focusing on gathering the same data from many more (1000s) of code bases that are publicly available. Once gathered we will need to work out how to compare codebases are there correlations between size/maturity/comments to popularity/adoption.

After gathering lots of code the next focus is on depth - number of modules, ratios of test to production code over time. External dependencies. Issues/defects etc.

Builder anti-patterns

2017-06-19T00:00:00-06:00

In Object Oriented languages and particularly in Java the Builder pattern is very popular at the moment. Originally designed to handle object construction challenges (Wikipedia uses the term telescoping constructor anti-pattern) there seems to be a trend to implement a builder as the default pattern for objects that have construction variations.

Typical builder use:

        Car car = new CarBuilder().withWheels(WheelType.CHROME).color(CarColor.RED).build();

So far so good but this does not really show the benefits of the pattern. I could quite easily construct a car with two constructor parameters.

        Car car = new Car(WheelType.CHROME, CarColor.RED);

Adding more attributes to Cars starts to show how the builder can help manage more parameters during construction.

        Car car = new CarBuilder()
                .withWheels(WheelType.CHROME)
                .color(CarColor.RED)
                .withSunroof()
                .withTowbar()
                .withMirrors(MirrorType.ELECTRIC)
                .build();

As we add methods to the builder we also need to update the construction build method

        Car car = new Car();
        car.setWheelType(wheelType);
        car.setTowbar(towbar);
        car.setSunroof(sunroof);
        car.setMirrors(mirrorType);
        return car;

Validation

We want to make sure that we have valid objects during but more importantly after construction. A builder is mutable the thing we build is mutable during construction but could be immutable after construction by making all the setters protects/private to the builder - a nice benefit.

Which object should be responsible for validating that the supplied parameters make sense. When we building an object through its constructor we expect a valid object or an exception during construction.

The same should be true upon completion of the build() method but it is not clear if the builder is responsible for validating that the object is valid or if the object we are building validates its state.

Making the object we are building (e.g. Car) responsible for validation aligns nicely with constructor construction but conflicts with the concept of a builder putting together valid objects.

Anti-pattern: Validation Envy.

Spreading validation logic across multiple classes makes it harder to understand the rules.

Pick one class to understand what is valid and what is not. If you build mutable objects then they will need to validate changes

Nested structures

Cars need engines and engines are complex in their own right. We could add the engine parameters to the CarBuilder class.

        builder.pattern.Car car = new CarBuilder2()
                .withWheels(WheelType.CHROME)
                .color(CarColor.RED)
                .withSunroof()
                .withTowbar()
                .withMirrors(MirrorType.ELECTRIC)
                .withEngineType(EngineType.PETROLIUM_SPIRIT)
                .engineCylinders(16)
                .build();

If we map the builder construction over to the built objects then we end up with a single Car class. From an object oriented design perspective this is a poor model. A better model would be to have an engine class for engine related data and behavior.

The builder could be enhanced to create a Car object that contains an Engine. Or we could create a builder for engines.

        builder.pattern.Car car = new CarBuilder3()
                .withWheels(WheelType.CHROME)
                .color(CarColor.RED)
                .withSunroof()
                .withTowbar()
                .withMirrors(MirrorType.ELECTRIC)
                .withEngine()
                .withEngineType(EngineType.PETROLIUM_SPIRIT)
                .engineCylinders(16)
                .buildEngine()
                .build();

Builders are linear constructs relying on method chaining. Breaking up the method calls onto separate lines improves readability but the sequence of calls is still linear. Our car structure is hierarchical - Cars have Engines.

The builder structure with a nested build does not match our target structure - there is disconnect between the models.

We could break out a separate builder and pass the result into the parent builder. This helps with structure but if there are inter-dependencies between the classes we are back to a linear call sequence.

Anti-pattern: Structure Misalignment

Make sure that the builder structure and the target structure are in alignment. A linear builder for a hierarchical structure creates a tension between the two models. The builder layout is linear but the resulting built object is not.

Alternatives

The builder pattern helps build sparsely populated objects that need to be initialized in many different ways. Outside of this context there are other patterns than should be considered.

Composition

Create objects that represent logical field groups within the parent. Compose the parent object through construction. Variation can be handled within each composite.

Hash-map initialization

In many languages passing a hash-map of key-value pairs that mirrors the target object graph is a simpler construction method. It can be sparse to match the problem and focuses validation into the constructed object. (Building hash-maps in Java is still a problem).

Wrap-up

Match the design to the problem. The builder work well for linear construction but breaks down for complex, deep object graphs.

Introduce when constructor complexity becomes unmanageable - can’t compose parameters into smaller object parameters. Rule of thumb 7 parameters for constructors.

Use strongly typed parameters over primitives. Varargs for lists.

Commit History Metrics

2017-06-15T00:00:00-06:00

For the last few years I have been very interested in what we can learn about our software development processes by looking at the data generated by the tools that we use to manage and deploy our work. This work started the Kuona project (Kuona is the shona word "To see.." http://kuona.io. All the code is open source on github https://github.com/kuona

Initial work focused on creating a dashboard from gathered metrics. Building full stack meant great flexibility but a lot of work and finding time for that work challenging with busy schedules. Over the past couple of weeks I have been working on feeding data from the collectors into elasticsearch and then using kibana to visualise the collected data.

Cutting to the chase heres and example chart:

I am not close enough to the project to draw significant insight from this chart but as an outside observer I can see that the project has been continually active since 2013 in this repository.

Here’s another metric drawn from Spring Boot history:

Putting these metrics together in Kibana is really straightforward. But how do we get the data there?

Setting up the Git Kuona Collectors

Kuona and the collectors are still in development so you will need to build/run from source.

Pre-requisites

Make sure you have an up to date Java Development Kit JDK.
Install Leiningen https://leiningen.org/#install
Clone the git collector locally git clone git@github.com:kuona/git-collector.git

Before running you will need to write a configuration file for the collector to use.

properties.json

{"workspace": "<workspace directory>",
  "git": [{ "url": "git@github.com:spring-projects/spring-framework.git" },
                { "url": "git@github.com:spring-projects/spring-boot.git" }]
}

Change <workspace directory> to some temporary space. On first run the git collector clones the repositories into the workspace. Subsequent runs update the repository before scanning history.

The collector only supports history on a single branch (master). Still need to work out how to handle multiple branches well.

Install ElasticSearch. If you are using a mac brew install elasticseach works nicely
Install Kibana. Again homebrew has a recipe brew install kibana
Make sure Elastic search is running http://localhost:9200
Make sure that Kibana is running http://localhost:5601

Elasticsearch does not have any of or data so we will need to postpone Kibana configuration until we have at least one record.

Running the collector:

In the git-collector directory

Create your properties file (see above)

lein run

Sit back and relax - collecting revision history takes a while.

Once the initial clone is complete records will appear in Elasticsearch so we can configure Kibana to use the generated data:

Once set up the index should look something like this:

It would be great to get feedback on this and the other tools being developed for Kuona. Currently working on a collector for Jenkins https://github.com/kuona/jenkins-collector that works like the git collector but for Jenkins build jobs.

Check out the project https://github.com/kuona. Join the team http://kuona.io https://groups.google.com/forum/#!forum/kuona-dev or check out the slack channel https://kuonadev.slack.com/

Defending the Domain Model

2017-06-06T00:00:00-06:00

Creating useful software models that represent the buisness they serve is hard. Creating a single model for a business at a point in time might be possible but as the business changes models need to be continually updated. Because these models are hard to produce and require a lot of analysis keeping them up to date becomes an impossibility. One model just can’t keep up with business reality.

Domain Driven Design (DDD) offers an alternate approach. By decomposing systems around bounded contexts and domains the models we produce are not only simpler because the capture less of the overall business domain but they are also optimized for a particular context. Each domain maintains its own models. Changes to a domain should have little or no impact on another domain.

It is very rare for any system to operate in isolation. Systems need to work with other systems using different models and technologies to deliver functionality to client applications and users. Email is ubiquitous, almost any consumer system probably needs to send emails from time to time.

Users of an online sales system need to maintain their contact details. Lets assume that each time a change is made a confirmation email is sent to the customer. A typical use-case. Email systems typically use a templating system. To send an email the Sales system needs to send meta-data to choose the right template and a key/value pair appropriate to the template. It is unlikely that any business oriented domain model would have data as key/value pairs.

Architecturally we need something that manages the exchange of information between the domain model and the other systems or domains they need to interact with. This something is typically referred to as the anti-corruption layer. The layer might be implemented as a facade or adapter to the other systems. I prefer the gateway name and pattern.

So at a high level we have a layer (gateway) that maintains a clean separation between our domains and services and capabilities that they require. What does this mean at lower levels of interactions?

Identities

Most systems use unique identities for entities that they manage. Traditionally these identities are database generated to guarantee uniqueness, and are numeric. Each system likes to maintain its own identities and the related entity lifecyle. Domain Driven models are the same. Entities, Aggregates and Aggregate roots use unique identities so that data can be referenced and found.

Often these identities are stored within the data model as external references. Adding these to the domain model corrupts the original model. If a domain has many integrations then multiple external identities need to be stored as references.

Assuming that these identities represent the 'same' objects a responsibility of the gateway should be to maintain a mapping of identities. Domain services use their own identities to initiate requests to external systems. Inbound data with external identities are mapped by the gateway.

Lets assume that the sales system needs to update the CRM system.

Figure 1. Identity Mapping

Maintaining a mapping requires persistence. This means that implementing a gateway is more complex than a simple facade.

Identities are typically unique only with the system that generated them. To understand an identity we need to know the system that generated it (realm) and its type (customer, order etc). Relational and graph databases make good stores for mapping identities.

The use of statistically unique values has changed the way we generate identities. In the past monotonically increasing index keys generated by a database were the norm. Computing a unique identity within the application has a number of advantages.

In our example lets assume that we need CRM changes to be reflected within the domain model. A new customer entry in the CRM system means that we need to create a new entity in the domain. The gateway maintains the mapping so needs to understand the relationship between the CRM identity and the new domain entity. This could be done with a callback or some other hook into the domains methods. Another option is for the gateway to assign the identity (uuid) and provide that as a request to the domain: 'New Customer to be known by <uuid>'. Traditional systems would still require correlating responses with requests or to receive the identity as a return value.

Data Conversion

Precision, type, composition/decomposition are conversion activities. Systems have developed interesting formats. Key-Value, 'compound values. Older systems assume timezones for time and data. More recent domain implementations typically use typed fields. When ingesting two fields for date (DD/MM/YY, MM/DD/YY or YYYY/MM/DD) and time (HH:MM:SS) the gateway can enrich the data based on it’s knowledge of the system supplying the data 'Tue Jun 6 18:18:12 MDT 2017'.

Data Mapping

Data mapping applies a look-up transformations to data values. Enum values and other defined lists are unlikely to match between systems. The Gateway provides a bi-directional transformation between systems.

Wrapping up

Preserving domain integrity reduces the domain code making the domain more maintainable. Business logic changes centered in the domain code is isolated from integration concerns. External systems are easier to replace by writing a new adapter for the new system. The gateway becomes a 'choke point' for information flow that can be monitored/controlled.

Many other daily maintenance activities like API changes are isolated to the gateway. Interaction are explicit and easier to understand/maintain and update.

Creating gateways that isolate integration concerns maintains a clear set of responsibilities and helps defend domain models.

Testing Logging

2017-05-26T00:00:00-06:00

As a developer I recognise the importance of generating logs for the systems that I help design and develop. At the outset often everyone recognizes the importance of logging but somehow it’s very difficult to engage stake-holders in requirements.

Building an application or system is a learning process that involves feedback. Agile software development practices highlight the importance of these feedback cycles with showcases of the developing application. Quite often these ceremonies highlight the functionality delivered and leave little room for conversations around cross functional requirements. For most development projects this is quite reaonable - after all the functionality is what is going to generate revenue. Logging and other cross functional requirements are considered a cost of doing business. This is true for logging as much as other cross cutting concerns.

Despite this lack of interest during development, development teams include logging for their own benefit - pre-production evaluation and a future insurance when things go wrong.

Logging often becomes a topic as systems are deployed into production and receiving customer traffic. Even with continuous deployment into production environments it’s not until customers are actively using the application that stakeholders become very interested in logging.

Not testing logs: The impact of not testing logging was brought home to me on a project recently. We were using the ELK stack. Our services produced JSON formatted structured logs that were then shipped to ElasticSearch. ElasticSearch has a great capabiltiy of generating its own schema (mapping in ElasticSearch terminology). This is a great feature, until you dump an object graph into your log message and end up with MBytes mapping file and things start to really slow down.

Assuming everyone recognizes the importance of logging why is it typically not rigorously tested beyond visual inspection prior to release? With release and functional development time pressures and that visual inspection is relatively inexpensive arguing for more comprehensive testing is hard. We need to make automated testing of logging easy. IMO this means tools that are easy to introduce and less expensive to maintain than the occasional visual inspection.

The available techniques are not that well known and without a simple way of testing logging other aspects that are easier to test get tested.

Logging is complex

Logging is a deceptively complex topic. Over logging leads to performance impact and storage issues. Storage is particularly relevant in regulated environments logs often need to be retained for extended periods. What needs to be logged and at what level is often a contentious topic. Partly because these rules or guidelines try to apply to all parts of a system.

HTTP response codes are a really good example of where context matters.

Codes starting with 4 are considered client errors (e.g. 404 Not found). The client is asking for something that the server knows nothing about. Maybe its a race condition or some coding issue in the client application. Because the code is an 'error' it is often logged as such within the server. IMO this should be logged as at WARN level - it becomes a server issue if a lot of 404 errors are returns not if there is only one or two.

For the client receiving a 4xx response the situation is very different. Typically this would be considered an ERROR.

These are examples of general logging rules that development teams often agree on. They affect aspects of the system. The HTTP example above would be handled in the client HTTP libraries and in the server HTTP request handling code. The controller in MVC.

To be useful the log would need to contain the request, context and the response. At this point we need to consider how to test these capabilities that apply to a layer within our architecture.

Some data should not be logged. Credit card details and other personal data items either need to be excluded from the log or masked so they are of little or no use without additional context.

These tests are general and typically apply to a set of components - e.g. Web services.

Testing these general requirements is difficult because they apply based on code responsibilities. One of the most common service patterns is Model View Controller (MVC). In MVC general informational logging is a responsibility of the controller. It handled the request, initiates changes to the model and delegates responses to the view. Errors and exceptions are often handled either by the controller or the framework being used.

A testing requirement might be

All requests are logged at INFO level and contain the supplied request parameters and response code.

To satisfy this requirements the test code needs to recognize calls to controllers, capture the generated log messages and compare the log content with the parameters of the particular request.

Testing Cross cutting requirements

Testing that cross cutting requirements are met is a tougher challenge than testing specific scenarios. Requirements are often given like this:

all requests are logged with their parameters and reponse codes.
Exceptions are logged at level x
All log messages are in JSON line format

Some environments may go further

All method entry and exist points are logged with parameters
Sensistive personal information is masked in logs.

Verifying these requirements automatically requires that first of all we have a very high level of test coverage to make sure that all the interesting methods that produce logs. We need to recognize when an interesting method has executed, capture the logs generated during that execution and check them against the agreed rules.

Most of the projects that I have worked on over the last few years have used a web framework. These frameworks provide some really useful abstractions over the complex world of HTTP request/response processing. Each framework presents its own challenges to identifying the methods that generate logs.

At least we can check the log formats…

Testing specific log requirements is a little simpler.

Testing Specifics

There are times when we want to write an automated test to verify that a particular method or class generates a log entry given a specific set of inputs. In most cases these are related to failure scenarios.

We could consider testing the generated log files themselves. After each test run example the expanding logs for format and structure.

Logging frameworks are very flexible, allowing both static (configuration file) and runtime customization. Appenders and formatters can be configured, loaded and unloaded at runtime. Tests can take advantage of this flexibility and runtime configuration to verify that logging is being generated as expected.

We need both specific tests for a function or class e.g. Given this invalid value a log containing the value is output at a particular log level. We also need more cross cutting tests e.g. All logging contains a component name and are JSON formatted.

The example test below is taken from a github project https://github.com/grahambrooks/testing-logging put together to show the basics of wiring up a logging appender in Log4j2./

  @BeforeAll
  public static void addAppender() {
    ConfigurationBuilder<BuiltConfiguration> builder = ConfigurationBuilderFactory.newConfigurationBuilder();

    builder.setConfigurationName("LogTestingBuilder");

    AppenderComponentBuilder appenderBuilder = builder.newAppender("TestLogging", "TestingLoggingAppender");

    builder.add(appenderBuilder);

    builder.add(builder.newRootLogger(Level.ALL).add(builder.newAppenderRef("TestLogging")));
    LoggerContext context = Configurator.initialize(builder.build());
    context.start();
  }

  @Test
  @DisplayName("Some long name that could be the method name.")
  public void testLogging() {
    assertLogs("Global logger", () -> AClassThatLogs.functionLoggingWithRootLogger());
    assertLogs(INFO, "Global logger", () -> AClassThatLogs.functionLoggingWithRootLogger());
  }

  @Test
  public void testDefaultLoggerCapture() {
    assertLogs(INFO, "Default logger info message", () -> AClassThatLogs.functionThatUsesDefaultLogger());
  }

  @Test
  public void testNamedLoggerCapture() {
    assertLogs(TRACE, "Named logger info message", () -> AClassThatLogs.functionThatUsesNamedLogger());
  }

  @Test
  public void testMultipleLogsDuringASingleCall() {

    List<LogEvent> logEvents = assertLogs(() -> AClassThatLogs.multipleLogMessages());

    assertEquals("Message 1", logEvents.get(0).getMessage().getFormattedMessage());
    assertEquals("Message 2", logEvents.get(1).getMessage().getFormattedMessage());
    assertEquals("Message 3", logEvents.get(2).getMessage().getFormattedMessage());
    assertEquals("Message 4", logEvents.get(3).getMessage().getFormattedMessage());
    assertEquals("Message 5", logEvents.get(4).getMessage().getFormattedMessage());
  }

Wrapping Up

More investment in tools to make testing logging easier for teams. At the moment the effort required in early investment during a project is not seen as an urgent need. So each project follows the same pattern leaving logging testing to be a very late if at all activity.

It would be great to see framework developers provide easier ways to verify cross cutting logging requirements.

In the interim testing specific logging actions is possible with a little work with your favorite logging and unit testing frameworks. Hopefully we will see more support for this style of testing in the future.

Templated styles

2017-05-20T00:00:00-06:00

While developing web applications I have often needed to change the style of one or more elements based on the state of the data model used by the application. I have an inherent aversion to conditionals in views so went looking for other options. I came up with templated styles that works with both server and client side templating systems.

The code examples use Angular and can be found here

The example displays a traffic light which has 4 states

stop
prepare-to-go
go
prepare-to-stop

Some traffic light systems jump from stop to go and don’t use the prepare-to-go state. Adding this state for the example shows how differences between the view and data model could be handled.

There are 3 lights red, amber, green. Each light is either on or off based on the status but there are really three states - one for each light.

e.g. for Red

The status of the red light when the red light is lit
The status of the red light when the amber light is list.
The status of the red light when the green light is list.

This model was used in coming up with the styles to be applied.

Conditional View approach

Probably the most obvious approach is to use a conditional in the template based on the state:

          <div ng-switch on="TrafficLights.status">
            <div class="animate-switch" ng-switch-when="prepare-to-go|prepare-to-stop" ng-switch-when-separator="|">
              <div class="circle traffic-light-red-amber"></div>
              <div class="circle traffic-light-amber-amber"></div>
              <div class="circle traffic-light-green-amber"></div>
            </div>
            <div class="animate-switch" ng-switch-when="stop">
              <div class="circle traffic-light-red-red"></div>
              <div class="circle traffic-light-amber-red"></div>
              <div class="circle traffic-light-green-red"></div>
            </div>
            <div class="animate-switch" ng-switch-when="go">
              <div class="circle traffic-light-red-green"></div>
              <div class="circle traffic-light-amber-green"></div>
              <div class="circle traffic-light-green-green"></div>
            </div>
            <div class="animate-switch" ng-switch-default>
              <p>Unknown state<p>
            </div>
          </div>

There is no change to the model. The view interprets the state and represents that state as an appropriate view of the traffic light.

The code is pretty verbose. Adding traffic light states means changes to the view.

Templated Styles approach

This approach removes all the conditional code from the view and uses a new color view model attribute to control the style applied to each light.

Templated style traffic light

          <div>
            <div class="circle traffic-light-red-{{TrafficLights.color}}"></div>
            <div class="circle traffic-light-amber-{{TrafficLights.color}}"></div>
            <div class="circle traffic-light-green-{{TrafficLights.color}}"></div>
          </div>

Using template substitution for the element class removes the need for a conditional. But we have to update model to include a color attribute derived from the status.

First we need to change the controller attribute into a property so when the value changes we can recognize the change.

Property definition

    Object.defineProperty(trafficLight, 'status', {
      get: function () {
        return _status;
      },         
      set: function (value) {
        _status = value;
        updateColor(); (1)
      }
    });

1	Called each time the status is changed

The 'updateColor' function looks very similar to the template conditional

    function updateColor() {
      console.log("Update color called");
      switch (_status) {
      case "go":
        trafficLight.color = "green";
        break;
        
      case "stop":
        trafficLight.color = "red";
        break;
        
      default:
        trafficLight.color = "amber";
        break;
      }
    }

A key difference is that JavaScript is much easier to unit test.

The CSS

.circle {
   width: 50px;
   height: 50px;
   border-radius: 50%;
   display: inline-block;
   margin-right: 20px;
}

.traffic-light-red-red {
   background-color: #a02128;
}

.traffic-light-red-amber {
   background-color: gray;
}

.traffic-light-red-green {
   background-color: gray;
}

.traffic-light-amber-red {
   background-color: gray;
}

.traffic-light-amber-amber {
   background-color: #F7BA0B;
}

.traffic-light-amber-green {
   background-color: gray;
}

.traffic-light-green-red {
   background-color: gray;
}

.traffic-light-green-amber {
   background-color: gray;
}

.traffic-light-green-green {
   background-color: #317f43;
}

Wrapping up: By creating a view model with a view attribute for the color the view becomes simpler - zero code

Being able to test views is key differentiator. Unit tests are far faster than functional tests so feedback is faster. The view is logic free.

Each combination can be styled differently so the view model is more explicit.

As I mentioned at the beginning this technique can be applied with any server or client side templating systems that allow attributes to be templated.

Emacs es-mode - REPL experiences for ElasticSearch development

2016-08-09T00:00:00-06:00

I don’t usually write reviews but I was so deleted by es-mode for emacs that I felt compelled to share. If you have worked with REST APIs and ElasticSearch in particular you probably have had similar experiences of using something to develop queries. Coding in one of the client libraries does not provide the interactive experience you need to develop quickly and it often becomes a frustrating exercise. es-mode takes away a lot of that pain.

Create a file for your queries and experiments, enter the HTTP method and url on one line and add the Json body if needed on the line below.Two keystrokes C-c C-c and the query is executed; the results shown in another emacs buffer. Its just like having a REPL for ElasticSearch. The Elastic Search Command Center command is just icing on the cake - real time view into your ES cluster.

Check it out on GitHub

Dependency Inversion - Applying the principle

2016-07-11T00:00:00-06:00

The Dependency Inversion Principle was named in 1994 but quite often it is overshadowed by Dependency Injection. They share a number of similarities and outcomes but while Dependency Injection is about providing dependencies based on context Dependency Inversion is about layering. Dependency Inversion is very useful refactoring interaces defined with getters (attribute accessor methods) into something more extensible. Here’s how:

Intent

High-level modules should not depend on low-level modules. Both should depend on abstractions.
Abstractions should not depend on details. Details should depend on abstractions.

An abstraction != interface but an interface is an abstraction.

Typical dependency. The architecturally higher level component depends on the lower level implementation.

Inverted dependency. The higher level component depends on abstractons. The lower level component depends on the higher level’s implementation.

into

Job Scheduler example

Job scheduling is an example of a fairly common orchestraton problem. We have an application that wants to provide the execution ruls (when) but not what has to happen.

In this example we have a simple interface declaration Job that defines a couple of properties for the name and schedule that the Job requires. A concrete realization of that class SimpleJob and then the calling application.

There are a few problems with this design. A job can only be run on a single schedule. The calling application interprets property values and the two classes share knowledge through the Schedule enum, limiting the options available.

As the interface evolves additional properties will be needed. Interdependence between properties will increase the complexity of the calling code, increasing the risks of complex defects.

public interface Job {
    String getName();
    Schedule getSchedule();
    void run(PrintStream output);
}

public class SimpleJob implements Job {

    public String getName() {
        return "Simple";
    }

    public Schedule getSchedule() {
        return Schedule.DAILY;
    }

    public void run(PrintStream output) {
        output.println("Daily activity");
    }
}

public class Application {
    private final Map<String, Job> hourlyJobs = new HashMap<String, Job>();
    private final Map<String, Job> dailyJobs = new HashMap<String, Job>();

    public static void main(String[] args) {
        new Application().run();
    }

    private void run() {
        final Job job = new SimpleJob();

        switch (job.getSchedule()) {
            case HOURLY:
                hourlyJobs.put(job.getName(), job);
                break;
            case DAILY:
                dailyJobs.put(job.getName(), job);
                break;
        }
    }
}

An interface that defines getters is a smell. Interfaces should define behaviour not access to attributes. To every rule there are exceptions but it stands true in this case:

Testing is pretty difficult. All the logic for registration and execution is handled within the parent. SimpleJob has no registration logic and at most the property accesses can be tested.

Improving the dependencies. In this example the calling code and called code share an interface declaration Registrar. The Job interface is much simpler, defining just two methods. One to register the job within the context, and the second to execute the job.

The calling application code is also simpler, removing the need for an enum entirely by implementing the Registrar interface.

The previous limitations are removed. Jobs can register themselves to multiple schedules without changing the Registrar interface.

public interface Job {
    void register(Registrar registrar);
    void run(PrintStream output);
}

public interface Registrar {
    void daily(String name, Job job);

    void hourly(String name, Job job);
}

public class SimpleJob implements Job {
    public void register(Registrar registrar) {
        registrar.daily("Simple", this);
        registrar.hourly("Simple", this);
    }

    public void run(PrintStream output) {
        output.println("Daily activity");
    }
}

public class Application implements Registrar {
    private final Map<String, Job> hourlyJobs = new HashMap<String, Job>();
    private final Map<String, Job> dailyJobs = new HashMap<String, Job>();

    public static void main(String[] args) {
        new Application().run();
    }

    private void run() {
        final Job job = new SimpleJob();
        job.register(this);
    }

    public void daily(String name, Job job) {
        dailyJobs.put(name, job);
    }

    public void hourly(String name, Job job) {
        hourlyJobs.put(name, job);
    }
}

Testing is also improved. The registrar can be stubbed or mocked to test the registration process.

    @Test
    public void registersSelfHourlyAndDailyAsSimple() {
        Registrar registrar = mock(Registrar.class);

        SimpleJob job = new SimpleJob();

        job.register(registrar);

        verify(registrar).daily("Simple", job);
        verify(registrar).hourly("Simple", job);
    }

Summary

Getter interfaces should be avoided. Any interface that interrogates objects to decide what to do breaks the encapsulaton of the class.

A -depends on→ B into B -dependes on→ A. This typically shows up as list of getters in B that are used by A.

Useful Links

The Value in value types

2015-01-25T00:00:00-07:00

In domain driven design (DDD) we talk a lot about entities; objects that have identity and lifetimes that typically outlive their values. e.g. An account lives longer than the email address, name, postal address etc. Value types might be mentioned in discussion but then quickly fade. This is a shame because they are the fundamental building blocks, after all they hold the application data. One reason for this lack of attention is that we have some pretty powerful substitutes already supported by our implementation languages - int, double, string. But is that really good enough?

Consider:

...
Email e = new Email("Graham", "Brooks", "graham@grahambrooks.com");

e.sendTo(...)
...

which uses the Email constructor:

class Email {

    public Email(String firstName, String secondName, String emailAddress) { ... }
    ...
}

Tests are the only thing that are going to stop things like:

...
Email e = new Email("graham@grahambrooks.com", "Graham", "Brooks");

e.sendTo(...)
...

making it into production but it does add to developer workload - because every time I need to call a method with the String, String, String parameters I need to check and double check. I want my language and tools to work for me not against me.

If we take moment and create some pretty simple types then (UML diagrams using PlantUML):

class Email {

    public Email(Name userName, EmailAddress emailAddress) { ... }
    ...
}

And the calling code:

...
Email e = new Email(new Name("Graham", "Brooks"), new EmailAddress("graham@grahambrooks.com"));

e.sendTo(...)
...

This last code snippet looks like a lot more work than before both for me as a developer, the compiler and the runtime environment. The problem is really with th example. With a well constructed set of domain objects the Name and EmailAddress objects will be fields in some entity - possibly Account. The compiler might be doing a little more work but that work is in my interests - checking the types.

Email address format verification can now be put where it should be - in the EmailAddress class.

You can probably guess that Value types should be:

Small
Immutable

Values should be constructed at the boundaries of the system. Strings → EmailAddresses. Within the domain methods manipulate the value objects - not the data stored in them …

Build Analytics - better development understanding

2015-01-19T00:00:00-07:00

Working with small teams is a lot of fun and I find it fairly easy to keep track of what is happening with version control and build systems. Errors and failures don’t come up that often and when they do they can quite often get solved there and then. On larger projects or working in a large organisation it’s impossible to keep track of everything. There are too many moving parts and changes. Incidents are more frequent and their impact much larger. A broken build or build system can affect 10s or 100s of people. For these larger development projects I find I have to collect and chart data, looking for tends and anomalies and then delve deeper into the data if and systems if there are problems.

I am facinated with how looking at data in a particular way can provide a better understanding of how things are working and what is bing done by developers and teams. GitHub’s activity heat map is a great example of how a simple visual can help find patterns really quickly. Or in this case how random my code submission have been.

When I first started building web sites it was quite difficult to know if and how the site was being used. Analytics programs that collect, analyse and chart data have become an essential part of of web development. Big data, despite the hype is spawning new toolchains and capabilities that were just unavailable to many teams.

Working with larger organizations on larger programs of work provides additional challenges. It may seem obvious but the shear volume of information generated by the systems that support teams in large organizations is daunting. It has taken a while but version control is pretty much universally part of every developers daily life. I am a firm believer in keeping everything in version control. For a small team working on a single piece of software keeping an eye on what is happening to the source is pretty simple - daily or weekly review and conversations. As things scale up though reviewing 100s of commits would become a full time job or part time for a larger group that then needs to be aligned.

Version control is just one source of data. We have ticketing systems for bugs, requirements management systems, build systems, release systems and a plethora of other data sources. Tracing changes though these disparate systems is hard. Spotting problems is even harder.

We need a way to make this data manageable

Applying big data techniques to development

“The greatest value of a picture is when it forces us to notice what we never expected to see.”

John Tukey, 1977

Big data systems and the analysis systems that they support have proven very effective in understanding product effectiveness. It is perhaps time to use these tools and techniques to understand the factors that improve software delivery in a quantifiable way.

There has already been some groundbreaking work in this area. The article ‘ Jonathan Andrew Wolter interpreting commit log visualisations’ talks about his work with Miško Hevery showing how analyzing commit logs that are generated as part of development can be used to indicate how a team is adopting new practices.

Products like sonarqube focus on analyzing source code and provide valuable insight into the current health of a code-base. Taking this further by combining this work with other analysis should provide a more complete picture of the development organization and where things can be improved.

Effectiveness

How good is your build system. Is something like this profile of build times good or bad?

Out of context I have no idea if these numbers are good or bad. But over time and combined with other data we could discern trends and compare different points in time.

Other things that might provide more context for the project

Build time counts -
Average mean min/max feedback clycle time per commit
Commit heat map - when are commits made
Commit frequency on active projects
Test to Production commit ratios and trends
Build times and trends

Hidden commits

Combining commit information from source control and build information from CI systems means that we can measure the latency or cycle time between a commit and the build that used that commit. We can also see how build times might be masking commits that happen during the build:

Long delays between builds caused by too much demand or build scheduling means that it is more likely the build will contain multiple commits. Perhaps not a huge problem when everything stays green but if a build fails it is more difficult to find the commit that caused the failure.

The same is true if build times are longer than your regular team commit cycle (time between commits). The first CI stage should be fast to avoid this situation but with many builds it is too expensive to track build times manually. Processing the logs to gather this data regularly and track trends is relatively straight forward and more cost effective.

Another visualisation example is Mark Crossfield’s visualisation of a GO configuration file as a graph realized in D3 GO Delivery Pipeline Visualisation. Again something that would be difficult to do by hand but the patterns presented might reflect improvements over time that lead to a way of spotting problems earlier.

These examples are fairly straightforward but also of limited value. Focusing in on one or two metrics can be dangerous and create anti behavior patterns. By combining measurements from a number of dimensions the expectation is that we can see more macro trends within an organization. For changes we can pose hypotheses about what we expect to happen and then monitor or look for the effects that we expect as well as things that we did not expect.

With so many tools and analyses that have been built to understand how complex systems behave in production maybe we can apply the same tools or techniques to the inner workings of the development organization’s systems.

Summary

The examples above are relatively simple but every journey needs to start with a first step. The code ratios can be generated by parsing data from a single source - version control. Hidden commits are likely to involve two systems - CI and version control.

Collecting, analyzing and visualizing development organization data is still in its infantcy but changing quickly. Having the collection ability means that we can pose hypothoses about the effect of changes on the development eco-system and then monitor indicators to see if the changes are having the expected results. To date this has meant tracking single metrics, probably manually and just tracking the data that caused unexpected side effects.

By tracking many data points we should be able to evolve better models and maybe detect trends and predict future events.

I started the Kuona project to test the viability of these approaches. If you are interested in trying things out for yourself and perhaps getting involved in the project check out the GitHub projects.

Service IoC containers

2014-04-07T00:00:00-06:00

How can we get the benefits of IoC containers at a higher level - the services that we deploy into environments? Typical service tiers are implemented with a fronting load balancer that allocates servers to satisfy requests. Applications or services that need these services are give domain or IP information of the load balancers and the load balancers are given details of the servers running instances of the services that they need. Essentially each system with dependencies news an instance of its dependencies. Inverting this dependency requires a container but it leads to some interesting advantages.

Before digging into the detail lets define a few terms. Paul Hammant has an article differentiating components and services. For our purposes a service exposes an interface exposed over a medium like a network and the exchange of information involves converting the data to facilitate the exchange. Most of the services I write these days are RESTful in nature.

Most of the commercial projects I am involved in use multiple environments. An environment is typically discrete and has its own network (sub-net). This might be local to my development environment, a shared QA environment and perhaps a pre-production preview. The discrete nature of these environments is an important consideration when using these discovery mechanisms.

I am using the term IoC (Inversion of Control) liberally here.

Discovery

Applications build containers from specifications that are either defined in code or some external configuration. For services running within a networked environment the container is external to a component or application. Registration service components like Apache Zookeeper can be told about services. Zookeeper was originally an Hadoop sub-project but is now a top level Apache project. Zookeeper has a number of use-cases including discovery. Applications can ask about the services that are available and instances that provide those services. Operating in this way the directory service acts like a container.

But before the container can be used it needs to be found.

Discovery by convention

Jenkins and Hudson provides an interesting method of discovery through multi-casting.

UDP multi-cast/broadcast

Jenkins/Hudson listen on a UDP port (33848). Clients can broadcast a packet (e.g. targeting 255.255.255.255) or multi-cast packet (targeting 239.77.124.213). When a packet is received an XML response is sent to the UDP port of the sender. In this way Jenkins and Hudson servers running on the local network can be discovered.

The client and server share information about the protocol (XML format and ports) for discovery of the available instances. Once discovered further communication is directed to the running instance(s).

DNS multi-cast

DNS muli-cast achieves the same result based on the shared knowledge that Jenkins/Hudson advertise themselves as “_hudson._tcp.local”. The version, url and port information are advertises in the DNS record of the server.

More details of these approaches can be found here.

Using the container

For our situation we need to discover a registration. Lets assume Zookeeper is the registrar and that is does not advertise itself on the local network (I could not find such a feature in the docs). It would be relatively straightforward to add a responder that worked in a similar way to Jenkins/Hudson to the running Zookeeper or Zookeeper cluster that would allow clients to discover the registrar and then make use of it.

Service registration

Each time a new service instance starts up on the network it first identifies the local registrar (e.g. zookeeper). Once identified it queries for the services it needs (perhaps looping until they are all available). Once received the service can continue initializing; connecting to the services it needs to perform its functions (e.g. database and service endpoints). When it has complete initialising itself it then registers itself to start receiving traffic.

There are some interesting benefits to this approach. Services typically take time before they are ready to reliably service requests. Perhaps caches need to be primed or configuration received and processed. By delaying service registration until after the service is ready we avoid overwhelming the service instance by sending traffic to it too early.

Scaling in this environment is also simpler. More service instances can be started without additional configuration in the environment. Each new instance discovers the environment dynamically and adjusts its settings accordingly.

Monitoring systems can ‘hook’ into the registry to discover the available services and how each instance can be monitored.

Zookeeper polls services to make sure that they are still available. If a service fails to respond or is causing errors it is removed from the registry.

Taking this a step further monitoring can fire up new instances dynamically to compensate for individual failures or increasing demand.

Auto-scaling and resilience

Netflix Chaos Monkey actively seeks out auto scaling groups and terminates instances. This approach encourages the system to be self-healing. New instances come into being to compensate for the loss. By shifting to discovery and a service container we could encourage services to self-expire. A service starts and decided how long it should live based on some configuration of fixed value (perhaps randomly within a lifetime range to avoid every service expiring at the same time. When a service instance reaches the end of its lifetime is initiates the creation of its replacement. Some time later the replacement advertises that it is able to take on the parents role allowing the parent to expire.

A true Phoenix server?

Container-less web apps

2014-04-07T00:00:00-06:00

In his keynote ‘This is water’ Neal Ford talks about “Yesterday’s best practices become tomorrow’s anti-patterns”. This seems to be the case for application servers like WebSphere, WebLogic and JBoss.

When these containers started to become popular (typically in large enterprises) servers took a long time to commission and configure. Containers were a way of deploying applications more quickly and with greater safety. But time change and today’s patterns and teams favour lighter weight, nimble solutions. Our ability to manage servers and networks through code is one of the key drivers for this change.

2011 seems to have been a tipping point with articles advocating change Stop Wasting Money On WebLogic, WebSphere, And JBoss Application Servers and data indicating that the change was already underway The Death of WebSphere and WebLogic App Servers? New Infographic shows the Rise of OSS Java.

Drivers for change

I think that there are a number of drivers moving IT organisations to use lighter weight frameworks and tools. Particularly open source frameworks and tools. Some of these drivers reasons are technology based but others are more subtle. 15+ years ago we were still trying to understand how to build web based applications. Modern web applications have very little in common with applications built 10 years ago.

Lower Cost

This sticker price is most obvious here. Driven by licensing costs organizations are constrained in where and how they can make use of commercial containers. This often leads to severe constraints on non-production environments. Where this was not a problem the effort involved in setting up a pre-production environment was very high. Given the cost teams were encourage to share with other teams leading to conflicts and ‘unusual’ errors caused by conflicting requirements.

Productive

We Cannot Measure Productivity but as developers we know when things are easier and more fun. Lighter weight tools like Jetty and Netty make programming more enjoyable. Instead of an environment controlling our applications our applications control the environment.

Firing up a Jetty server is pretty simple

public class SimplestServer
{
    public static void main(String[] args) throws Exception
    {
        Server server = new Server(8080);
        server.start();
        server.join();
    }
}

This is a particularly important element of control. The application controls its HTTP(S) interface not the interface controlling the application. I am a big fan of this approach and Jetty has proven itself on a lot of projects.

Even if we cannot measure productivity my experience is that using lighter weight open source tools feel more productive. And deployments are definitely faster and more predictable.

Testable

I practice Test Driven Development. Testing code that runs in a container is hard. Very hard. Test coverage is quite often much lower than other environments and it is difficult to get a higher coverage figure without resorting to integrated system tests that involve containers.

Jetty for example makes it possible to test controllers in unit tests without running up a full server. For automated tests that involve network activity servers can easily be started on separate threads.

Deterministic test behavior is critical. Timing loops that wait for external systems to be ‘ready’ make it difficult to be reliably deterministic. Calling a method to start a server is about as deterministic as it gets.

Configuration

Application containers abstracted environmental differences behind logical consistent interfaces. Applications would ask for resources using a logical reference and if the container was configured correctly it would receive an implementation to use.

I remember spending a lot of time writing application container configuration scripts to ensure that each environment (e.g. dev and qa) were consistently configured. Those scripts were driven from version controlled property files.

Container-less deployments typically take those same property files and deploy them with the application to the same effect.

State Replication

Being able to scale a web application or service is a key cross functional requirement. A single server just can’t deliver the levels of service and reliability organizations require.

Many applications are statefull. Each request is dependent on the completion of some previous event. Application containers provide state replication between containers on behalf of applications. This means that state does not have to reside in a database or some other external store. This is a very complex problem to solve and it not having to worry application developers with this issue is important.

Current patterns push state out of the web and application tiers down into persistent storage. Being stateless means that we don’t have to worry about servers talking to one another - there are limits to how many servers can be accommodated in a distributed state model.

Monitoring

Servers and applications need to be monitored to make sure that they are healthy and delivering the expected levels of service. Having a consistent interface to query makes operational controls easier. Application servers naturally provide consistency that are agnostic to the applications they run.

The abstractions provided by the container are similar to monitoring facades used for container-less applications. The monitoring systems need to be able to use a consistent interface. It now become the application’s responsibility to provide that interface.

Fortunately monitoring has shifted from proprietary interfaces to more open standards (typically HTTP based). Lightweight frameworks like Dropwizard generate warnings if monitoring interfaces are not implemented. The framework provides components that make implementing these monitoring interfaces pretty easy to implement.

Alternatives

So what are the alternatives to running applications in an application container? Here are some popular and robust options that immediately come to mind. There are many more:

The container-less options are very compelling. Key for any technology in my opinion is the community driving its adoption. Commercial or open source a strong community is a good indicator that developers enjoy working with the technology and are keen for others to share their enjoyment.

Team Distribution Models and Conways Law

2014-03-31T00:00:00-06:00

If Conway’s law applies the how we set up and organise teams should either be aligned to the architecture of the applicaiton or system being developed or application/system architecture will shift to match the team structure.

Conways law

organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations

Conway, Melvin E. (April 1968), “How do Committees Invent?”, Datamation 14 (5): 28–31, retrieved 2009-04-05

Conway’s law is an adage named after computer programmer Melvin Conway, who introduced the idea in 1968; it was first dubbed Conway’s law by participants at the 1968 National Symposium on Modular Programming.[1] It states that organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations

How can team members be separated and how does this affect daily working life?

Degrees of separation

Distance: Two people do not need to be far apart before the separation challenges our usual informal communication channels. Separate rooms or buildings within the same location can produce similar symptoms of disconnect to much greater distances.
Overlap: Time zones and different working arrangements mean that overlapping work time can be reduced. In extreme cases there may be no overlapping 'normal' work time. These factors affect the way the individuals interact. Individual motivation to communicate and coordinate are the key to compensating for the differences between co-located and non co-located working. Tools can improve the communication but they cannot compensate for the lack of motivation.

Assumptions

There is a basic assumption that everyone is motivated to deliver the project responsibilities.

Risks

Team members become disengaged
Effort is wasted because key information has not been shared sufficiently.

Co-location

Colocation is the default arrangement for agile teams. All the project participants occupy the same space and typically the same table. This ‘Same room’ option delivers high levels of collaboration and very low ceremony. Communication is very easy and it is easy to self organise around solving problems.

Sometimes it is difficult to occupy the same space. Often product owners and other stakeholders have other responsibilities which make it difficult to occupy the same space as the delivery team or their work might require higher levels of privacy. Often the entire team is in the same building. Meetings and discussions within the team are still very high bandwidth but meetings and discussions with the stakeholders out of the team room requires more work and coordination.

Multi-location

In this model teams and perhaps team rooms are set up in multiple locations. Each team has a full compliment of skills (except for stakeholders like the product owner).

Bridgehead

Logically the bridghead model is very similar to the same building model. In many cases it operates in the same way. Business and business requirements are communicated from the product owner and other stakeholders via an intermediary (typically a business analyst) to the delivery team. This is a popular off-shoring model. People from the offshore development center travel to the buisiness owner’s office and work remotely from the delivery team.

People may work on rotation, traveling to the product owner for some months and then ‘rotating’ back to work with the team while another team member joins the product owner in their location.

In this model the internal design of the system is still (predominantly) done in a single location. In this case at the offshore development center. Often there is a very strong sense of ownership for the internal architecture and quality of the system. The offshore team has very little insight into how the sysem is being used. The product feedback cycle is much more difficult and providing a high fitness for purpose is difficult without constant attention to detail and high bandwidth communication and frequent rotation.

Scattered

This model is interesting from a number of perspectives and is the complete antitheses of the same-room co-located team. And yet open-source project teams are often cited as an example of where this model has been proven to be effective.

Conway’s law might indicate that in a fully scattered or distributed arrangement with all communication lines being electronic that the developing system would be made up of many systems perhaps even one per team member or developer. A counter view might be that because no two team members have a preferential status with respect to communication that the team is actually most reflects the co-located team but with less efficient group communications.

Temporal distribution

Working in a team separated into different time-zones adds an additional complication. One of the key elements to effective communication is availability. With other team members spread around the globe finding group collaboration times becomes more difficult and topics of conversation have to be stored up until it is convenient to share in conversation. Asynchronous/written communication is a common fallback but the depth of exchange is limited and requires more effort.

lazybuilder v0.3

2014-02-23T00:00:00-07:00

The idea for lazybuilder came up during a recent coding dojo doing code kata. We were using Vim and a separate terminal to run the tests. Switching back and forth between two windows when trying to move quickly seemed like a waste of time.

There are several tools in this space but most of them work by polling for changes or use linux events that are not available on a mac. lazy builder uses CoreServices.

I found during testing with a rails app that some events are received after the build has completed. To avoid looping I added a 100ms delay after a build. This is a bit of a hack and I am working on trying to recognize changes that are due to build activity as an alternative.

lazybuilder

with no parameters watches the current directory and runs make.

lazybuilder . rake

watches the current directory and runs rake

If you want to run more complex build commands enclose them in quotes:

lazybuilder . "raake test"

Roadmap

Improve cool down period
Avoid triggering build on temp file changes generated by emacs
Consider committing changes to version control (particularly git) if the build command is successful

If you get to try out lazy builder I would really appreciate feedback, bug reports and pull requests.

Installation

brew update && \
brew tap grahambrooks/lazybuilder && \
brew install lazybuilder

From Source

Installs to /usr/local/bin

git clone git@github.com:grahambrooks/lazybuilder.git
cd lazybuilder
make install

CODE keyboard

2014-01-28T00:00:00-07:00

After a long wait CODE keyboards are available again and I received mine last night. I have to confess that I am a bit of a keyboard junkie. I am reluctant to count them all but I am edging into double digits :(.

The keyboard arrived nicely packaged inside a larger box with lots of packing material to avoid damage in transit. The box containing the keyboard itself is minimal but nicely designed.

Having a removable cable did not seem significant but the velcro strap makes carrying the cable a nicer experience and I don’t end up wrapping the cable around the keyboard and messing up the keys.

Using the dip switches to configure the keyboard makes it easier to use with multiple machines. I don’t need to go in and configure my preferred settings each time I connect to a different machine. One tip though - only move the switches when not connected.

It is going to take a while to get used to the O rings on the green switches. I have been using a Das Keyboard for a while and may miss the full travel and almost zero dampening that it offers - even if it is more annoying to everyone around me who has to hear me hammering on the keys.

The illuminated keys are a very nice feature - not only for programming in the dark but just in general. I don’t look down that often but when I do the keys are clear and brightly marked.

I am really impressed by the build quality. The keyboard is solidly built and has a nice ‘brushed’ feel that is a joy to use.

Cling A C++ REPL?!

2013-12-19T00:00:00-07:00

Browsing some blogs recently I came across some interesting C++11 lambda code and thought it would be cool to try it out for myself. The idea of spinning up a project or even putting a new test in an existing project did not feel right. Wouldn’t a C++ REPL be useful for just these occasions?

A couple of web searches later turned up Cling which seemed to be just the thing. There was no distribution for OS X so my first thought was to build from source. This is a little more complex than I thought it should be. You need to set up a viable llvm development environment and run some patches to get cling to compile. I ran into a number of problems doing this both on my Mac and on an Ubuntu VM. (compilation, linking and version incompatibilities between the repositories). Really wished for some CI goodness at that point.

I ended up falling back on VMs and the Ubuntu binary distributions. If you are using Vagrant and VirtualBox I found that the 386 based Ubuntu 12.04 image would not run the binaries but the AMD one worked just fine.

Once I had the binaries working on the virtual machine I copied them over to more usual locations +/usr/local+ for more general use.

Cling uses clang under the hood and has the same command line arguments. I like to use C++11 at the moment and this can be specified in the usual way:

vagrant@vagrant-ubuntu-precise-64:~$ cling --std=c++11

****************** CLING ******************
* Type C++ code and press enter to run it *
*             Type .q to exit             *
*******************************************
[cling]$ 

Once you have the cling command shell running you have some additional commands available:

[cling]$ .help
Cling meta commands usage
Syntax: .Command [arg0 arg1 ... argN]

.q				           - Exit the program
.L <filename>	           - Load file or library
.(x|X) <filename>[args]	   - Same as .L and runs a function with signature
 	                             ret_type filename(args)
.I [path]			       - Shows the include path. If a path is given - 
	                         adds the path to the include paths
.@ 				           - Cancels and ignores the multiline input
.rawInput [0|1]			   - Toggle wrapping and printing the execution
			                 results of the input
.dynamicExtensions [0|1]   - Toggles the use of the dynamic scopes and the
                             late binding
.printAST [0|1]			   - Toggles the printing of input's corresponding
 	                             AST nodes
.help                      - Shows this information
[cling]$ 

At this point you can start writing code. Leaving the ‘;’ off the end allows cling to display information about the line that has been executed.

[cling]$ auto x = 10;
[cling]$ auto y = 11
(int) 11
[cling]$ x*y
(int) 110
[cling]$ auto foo = [](int x) { return x * 2; }
(class <lambda at input_line_8:2:13>) @0x7fb61fde7028
[cling]$ foo(100)
(int) 200
[cling]$ 

Boost program_options

2013-12-06T00:00:00-07:00

I have been spending time recently writing command line apps in C++11. Each time I wanted a way of handling command line arguments flexibly. I chose to use the boost::program_options library. The documentation is pretty good but there are some assumptions (aliased namespace) and the example code is broken up with paragraphs of text explaining what the code does.

When I start to explore an API to a library I start by writing tests. After working with the program_options library on a few small applications the same pattern came up each time so I thought I would share the results as a complete app, although admittedly the app does nothing other than accept command line parameters.

The full source code can be found at http://github.com/grahambrooks/boost-program-options

The application is also linked up to a continuous integration server that triggers a build and test on each commit.

To keep the dependencies as low as possible the tests are written using the boost unit testing framework.

The tests are somewhat artificial in that they test the command line options results in an application state and not an observable behavior but are perfectly valid test cases.

Basic use case

 app -v --version -username [value] file file

The basic use case offers two variations of switch (-s –switch) a numeric argument option and a list of files.

== Test: No parameter behavior

So the first test verifies that we can handle (detect) that no command line parameters were supplied.

#include <boost/test/unit_test.hpp>

#include "command_line_argument_parser.hpp"

BOOST_AUTO_TEST_CASE(recognises_no_supplied_arguments) {
  command_line_argument_parser parser;
  const char *argv[] = {"app"};
  auto args = parser.parse(sizeof(argv)/sizeof(*argv), argv);
  
  BOOST_CHECK(args.no_arguments());
}

Test: Invalid argument

Not everyone is going to get the parameters right so we need to know when something goes wrong.

#include <boost/test/unit_test.hpp>

#include "command_line_argument_parser.hpp"

BOOST_AUTO_TEST_CASE(raise_exception_on_invalid_argument) {
  command_line_argument_parser parser;
  const char *argv[] = {"app", "--silly-argument"};

  BOOST_CHECK_EXCEPTION(parser.parse(sizeof(argv)/sizeof(*argv), argv),
			std::logic_error,
			[] (std::logic_error const& e) -> bool {return strcmp(e.what(), "unrecognised option '--silly-argument'") == 0;});
}

Test: Accepting a switch

Accept switch arguments for binary options

#include <boost/test/unit_test.hpp>

#include "command_line_argument_parser.hpp"

BOOST_AUTO_TEST_CASE(recognises_a_switch_argument) {
  command_line_argument_parser parser;
  const char *argv[] = {"app", "--verbose"};
  auto args = parser.parse(sizeof(argv)/sizeof(*argv), argv);
  
  BOOST_CHECK(args.verbose());
}

Test: Accepting a value argument

Accept values on the command line

#include <boost/test/unit_test.hpp>

#include "command_line_argument_parser.hpp"

BOOST_AUTO_TEST_CASE(recognises_username_argument) {
  command_line_argument_parser parser;
  const char *argv[] = {"app", "--username","graham"};
  auto args = parser.parse(sizeof(argv)/sizeof(*argv), argv);
  
  BOOST_CHECK_EQUAL("graham", args.username());
}

Test: Filename parameter list

Accept a list of filenames positionally at the end of the command line

#include <boost/test/unit_test.hpp>

#include "command_line_argument_parser.hpp"

BOOST_AUTO_TEST_CASE(captures_filenames_from_the_commandline) {
  command_line_argument_parser parser;
  const char *argv[] = {"app", "a", "b", "c"};
  auto args = parser.parse(sizeof(argv)/sizeof(*argv), argv);
  
  auto filename_arguments =  args.filenames();

  BOOST_CHECK_EQUAL(3, filename_arguments.size());
  BOOST_CHECK_EQUAL("a", filename_arguments[0]);
  BOOST_CHECK_EQUAL("b", filename_arguments[1]);
  BOOST_CHECK_EQUAL("c", filename_arguments[2]);
}

Test: Main method to run the tests

#define BOOST_TEST_MAIN
#ifndef MAKEFILE_BUILD
// Work around for xcode's default to always link against dynamic libraries
// and the need to static linking for distribution.
#define BOOST_TEST_DYN_LINK
#endif
#define BOOST_TEST_MODULE DupsTests

#include <boost/test/unit_test.hpp>

Application bootstrap

Bootstrapping the app. Main methods need to be as small as possible and responsible for wiring up the application and passing command line parameters into the application.

#include "application.hpp"

int main(int argc, const char *argv[]) {
  application app;

  return app.run(argc, argv);
}

Main application class

For anything more than this trivial application the application class would contain the main application algorithm or defer processing to a collection of other classes.

#pragma once

#include "command_line_argument_parser.hpp"

class application {
public:
  int run(int argc, const char* argv[]) {
    command_line_argument_parser parser;

    parser.parse(argc, argv);

    return 0;
  }
};

The command line processing classes

Class responsible for capturing runtime arguments.

#pragma once

#include <boost/program_options.hpp>
#include <vector>

using namespace std;

class arguments {
  constexpr static auto username_option_name = "username";
  constexpr static auto username_option = "username,u";
  constexpr static auto verbose_option_name = "verbose";
  constexpr static auto verbose_option = "verbose,v";
  constexpr static auto files_option_name = "input-files";
  
  boost::program_options::variables_map variables;
  friend class command_line_argument_parser;
public:
  arguments(boost::program_options::variables_map variables) : variables(variables) {}

  bool no_arguments() {
    return variables.size() == 0;
  }

  bool verbose() {
    return variables.count(verbose_option_name) > 0;
  }

  string username() {
    return (variables.count(username_option_name) > 0) ? variables[username_option_name].as<string>() : "";
  }
  
  const vector<string>& filenames() {
    return variables[files_option_name].as<vector<string>>();
  }
};

class command_line_argument_parser {
    boost::program_options::options_description desc;
  public:
    command_line_argument_parser() {
      desc.add_options()
	(arguments::verbose_option, "Typical long and short switch argument")
        (arguments::username_option, boost::program_options::value<std::string>(), "username to use")
	(arguments::files_option_name,  boost::program_options::value<vector<string>>(), "input file");
    }

  arguments parse(int argc, const char *argv[]) {
    boost::program_options::variables_map variables;
    
    boost::program_options::positional_options_description p;
    p.add(arguments::files_option_name, -1);

    boost::program_options::store(boost::program_options::command_line_parser(argc, argv).options(desc).positional(p).run(), variables);
    
    boost::program_options::notify(variables);

    return arguments(variables);
  }
};

Java build tools 2013

2013-11-15T00:00:00-07:00

I am about to embark on a new Java project so I thought I would take a look at the current state of build tools for Java projects. First I thought I would draw up some assessment criteria to use as a loose guide. This is what I came up with:

Fast

I want a build tool that spends as much time as possible building my application and not setting itself up to perform the task.

Manages Dependencies

Gone are the days when I can or want to manage my dependencies manually. Current Java applications want to make use of frameworks and those frameworks want to build upon other frameworks. To take advantage of these I need something that can take care of finding and delivering project dependencies.

Quick to set up

I don’t want to spend a long time learning a new tool. I am willing to invest in learning when I have an unusual problem to solve in the build system but out of the box I want to be up and running quickly.

Encourages good practices

I work with others in team to deliver projects. A tool that encourages and rewards good project source and binary management is important. A clean well supported syntax for any build scripts is important. Unnecessary syntax is annoying and distracting when it comes to build system creation and management.

Good Tool Support

I use an IDE for Java development (IntelliJ Idea) and others might use Eclipse. I want a tool that either understands my IDE or is understood by my IDE. Given that I am likely to manage my dependencies in the build file (I prefer command line builds) then a build system that can update my IDE is better than an IDE that knows how to interpret my build file.

Flexible

There are always fringe cases where I might need to introduce some other types of code e.g. ANTLR grammar files that compile into Java source. I need to be able to introduce pre-compilation and post compilation steps but I am happy that I may need to do a little work here.

Support or planned support for other JVM based languages would be a bonus.

Extensible

No two projects are the same and people that write build tools cannot predict all of their users requirements so I want a build too that I can extend for my projects.

Good Community Support

For open source tooling (preferred) an active community is essential for help - particularly if the documentation is not particularly extensive.

Support Continuous Integration

All my projects run with continuous integration (CI). I run my own CI server for my personal projects. CI systems ‘know’ how to interface with some build apps to monitor progress and capture output.

Test and compilation reports need to be captured for reporting and metrics.

Candidates

The candidate list came from tools that I have used on projects in the past and some basic web searching - I am sure it is not extensive!

I have been spiking a number of web projects using the dropwizard framework. It has some really nice features and made a good candidate for the evaluation (at least from a Hello World type app perspective).

Ant & Ivy

I have been using Ant for a very long time. Surprisingly Ivy has not come up on projects in the past but I have heard some very good things about its performance in finding and acquiring dependencies.

In the past I have added the Ant distribution to my source repository to minimize the setup work for developers. This has worked very well in the past and I thought I would use this technique this time around. It could be me but trying to set up Ivy integration with Ant in a single directory took longer than I was willing to wait. There was little documentation on how to use Ant and Ivy with Dropwizard and from past experience even a simple application with few dependencies results in 100s of lines of XML. After an initial attempt to get things working I decided to see if there were better options later with the option to coming back to an old friend if I needed to.

Maven

Maven ‘invented’ the de-facto standard of dependency management for Java projects that is now making itself known in other spheres. It is also the standard in many corporate environments. I will admit that I am a ‘light’ maven user and not been forced or motivated to drill below the other interface.

Maven has a particular view of the builds and their stages that is relevant to a lot of projects but I also find that on most of these projects there are also some exceptions that ‘break the model’.

Maven is similar to Ant in that the build files are XML based. I apologize but I just cant get over the angle bracket tax. I will use it if it is the project norm but the signal to noise ratio is just not strong enough.

Raven & Rake

Some years ago I worked with some developers who were passionate about building Java projects using Ruby and Rake. Since then there have been a number of projects in this space (Raven being one of them). Looking at the source repository Raven has not been updated since 2007. Rake does not understand Java dependency management norms and would require a lot of libraries or custom code to get working effectively. While poking around in this space I came across http://buildr.apache.org[Apache Buildr].

Gradle

I have heard some good things about using Groovy and in particular Gradle for Java builds. I installed the distribution and wrote a 3 line ‘Hello World’ build script that took 1.3 seconds to load and run

just far too long. So I gave up pretty quickly.

Buildr

Apache Buidlr is another build tools built in Ruby and based on Rake. But and this is a big but the libraries and build DSL work with the common build conventions for build projects. After a bit of messing around with transitive dependencies I ended up with a very simple 19 line build file for my Dropwizard project.

Downloaded all the dropwizard dependencies in about 20 seconds (there are a lot of dependencies).

Compiling a few Java classes is not quite as fast as I would like but below 2 seconds.

Simple build file

repositories.remote << 'http://repo1.maven.org/maven2'

desc 'My project build'
define 'my-java-project' do
  project.version = 1.0
  project.group = 'My group name'
  manifest['Copyright'] = 'Graham Brooks 2013'
  compile.options.target = '1.8'

  desc 'The API component'
  define 'my-api' do
    compile.with transitive('com.yammer.dropwizard:dropwizard-core:jar:0.6.2')
    package :jar
    run.using :main => %w(com.HelloWorldService server hello-world.yml)
  end

Buildr infers a lot from the project structure. This can be overridden (flexible) but if you use a Maven file structure you don’t need to write much of a build script.

The run line took a while to work out how to add command line parameters.

run.using :main => %w(com.HelloWorldService server hello-world.yml)

Combining the parameters with the classname to run makes sense after you see it but usually the two are separated out into different parameters.

Buildr has been under development since 2007 so it has some pedigree. It also supports ant tasks if you use the JRuby variant.

So if you are looking around for an alternative or starting a new project I think Buildr is definitely worth evaluating.

ci console 0.0.2 released

2013-10-12T00:00:00-06:00

I like to commit code frequently when working on a project. I also like to use the command line for building, testing and committing code. So I thought it would be nice to have a way to check the build status from the command line. I have been wanting to write something in C++11 and a small lightweight command line tool seemed like a good opportunity.

My efforts can be found here https://github.com/grahambrooks/ci/

I have been playing around with a few different report formats and version 0.0.2 seems to be working the way I like to work.

With no options the tool loads configuration from a Json formatted configuration file somewhere on the current path and reports the number of builds in each category (successful, failed, building, fixing). Adding the –verbose option lists all the builds from the server.

Status

Error handling is basically non-existent
Not enough tests - currently the code feels and looks like a prototype
Needs input from others

You can download a DMG of the compiled binary or build from source https://github.com/grahambrooks/ci

If you decide to build from source I suggest homebrew for boost. Make sure you install boost with the –with-c++11. You will also need the latest (5.0) XCode for Clang

Let me know what you think and of course send me your pull feature requests

Zero Down-time - relational databases

2013-08-29T00:00:00-06:00

Continuous Deployment is the act of automatically deploying an application every time the build is successful. I am currently working with a development team that is working towards continuous deployment as part of their continuous delivery adoption plans. The system involves several internal web services which seemed like a good place to start working on not only automating the deployments but maintaining a very high degree of up-time during those deployments. Automating deployments involves both development and techops groups so I thought I would search for some worked examples that would help illustrate the techniques and steps required. I found several blogs and articles talking about different approaches but no worked examples.

This article steps through the process but also includes a complete worked example.

For the impatient the source code and rake script for this example can be found at https://github.com/grahambrooks/zero-down

If you have worked with Rails or Play framework then you will know about database migrations (evolutions in Play). Each migration changes the database schema including the migration of data. These systems are pretty powerful and have significantly improved the reliability and speed with which web applications can be updated.

But there is a catch. The upgrade process assumes that there is a service outage or that there is a data gap between the old system and the new.

Service outages are the traditional approach where the old system is turned off or made inaccessible by clients while the database and applications are updated. Typically these updates are done when the system is lightly loaded to minimize business impact.

To avoid this some implementation use blue/green deployment slices but this can lead to data discrepancies between versions that need to be resolved after the upgrade. Blue/green updates have the advantage that active users are minimally disrupted (if at all). For 24/7 operations this approach is very attractive.

Updates that minimize or eliminate service disruption should be the norm so the techniques covered here focus on maintaining service.

Martin Fowler has a good introduction to the blue green release process on his site Martin Fowler Blue Green Deployment

Blue/Green releases

Initial Green state

Starting out we have the green system clients call a web service through a load balancer. The web service is backed by a relational database. A fairly typical setup.

Next we add the new ‘blue’ system which is a complete stack of system components including the app server and database.

Adding new blue slice

The switch over is quite simple. At a convenient point in-bound traffic is switched from the green system to the new blue version. For the blue system to be brought up and running a copy of the green database is taken and then upgraded using the database migrations for the new blue version. Any transactions processed by the green system while the blue system is being upgrade and made ready for the cut over need to be re-played to avoid data loss. Until the data is moved user data sent to the green system will not be available in the new blue system which can easily lead to discrepancies that are very difficult to unwind and resolve in transactional systems.

Switching green for blue

Finally the green system is removed.

Final blue state

This option creates a new database and service instance using the new release and then migrates callers from the old version to the new. Typically this is done by updating load balancer configuration.

This option requires that requests to either version are processed successfully into the database otherwise transitions to the original system will not be available after the transition.

If workable this option is lower risk. On upgrade failure traffic immediately falls back to the original system until a fixed version is available.

Backward and Forward compatible database

This approach has a few more steps and needs more code that is later thrown away but it solves the key problem of data integrity during and after the upgrade.

Assuming we start in the same state with an active green system the first step is to apply database updates. These updates need to support both the current system and the new version. Quite a few database changes can support both versions but simple restructuring the database schema are problematic and need additional measures.

Adding blue system and blue/green database

Now when we switch over there is no data discrepancy. Database updates apply to both systems and data integrity can be maintained.

Switching over

Once traffic is being served from the new blue system any code in the database to support the old green system can be removed.

Final blue state

Worked Example

This example is not a full blown app but contains all the key elements for zero down-time updates.

UML of the initial ‘green’ database schema

UML of the transitional or interim ‘blue-green’ database schema

UML of the final ‘blue’ database schema

Back to front updates

This approach updates the database first and then each of the application servers. For this approach to work the database has to be compatible with both the current version and the new version of the service.

Naturally this approach has more risk. If the database update fails or is not compatible the release is blocked.

Tech

Persistence - MySQL
Application - Ruby Sinatra app using active record
Load Balancer - HAProxy

Development tools

Rake for automating the deployment tasks
dbdeploy for database migrations

The app is extremely simple. It serves a single index page showing a list of users. Version 1 shows a single address stored in user table. Version 2 shows each user with zero or more addresses.

Update basic flow

Initial State

Server version 1 is running on port 8001
HAProxy is configured to accept in-bound connections on port 8000 and passes them on to the application on port 8001
Database contains a single table

CREATE TABLE users (
        id int(11) NOT NULL AUTO_INCREMENT,
	username VARCHAR(20) NOT NULL,
	address_line_1 VARCHAR(100),
      	address_line_2 VARCHAR(100),
	PRIMARY KEY (id)
);

Target State

Server version 2 running on port 8002 (in real life this is likely to be a different server)
HAProxy accepting in-bound connections on port 8000 and forwarding request to version 2 on 8002
Database contains two tables (users and addresses)

Upgrade steps

Create the new table (addresses)
Setup triggers to create, update and delete address records when a user record is updated
Copy the address data from the users table to the addresses table
Run Server V2
Update the load balancer configuration (HAProxy) to point to Server V2
Shutdown Server V1
Remove the database triggers
Remove the unused columns from the users table

Setting up Version 1

Create the Database

CREATE DATABASE rolling;

Set up the required table for dbdeploy

CREATE TABLE changelog (
  change_number BIGINT NOT NULL,
  complete_dt TIMESTAMP NOT NULL,
  applied_by VARCHAR(100) NOT NULL,
  description VARCHAR(500) NOT NULL
);

ALTER TABLE changelog ADD CONSTRAINT Pkchangelog PRIMARY KEY (change_number)
;

Run the first migration

CREATE TABLE users (
        id int(11) NOT NULL AUTO_INCREMENT,
	username VARCHAR(20) NOT NULL,
	address_line_1 VARCHAR(100),
      	address_line_2 VARCHAR(100),
	PRIMARY KEY (id)
);

Start the server

ruby server/v1/server.rb -p 8001

Start HAProxy with Version 1 configuration

haproxy -f cfg/v1/haproxy.cfg -D -p $(<haproxy-private.pid) -st $(<haproxy-private.pid)

At this point the service is available at: http://localhost:8000 and the server endpoint at http://localhost:8001

Setting up to install version 2

First we need to set up the new table structure and the mechanism for maintaining backward and forward compatibility in the database. The approach shown here uses triggers and an update script to get things started.

DROP TABLE IF EXISTS addresses;
//
CREATE TABLE addresses (
        id int(11) NOT NULL AUTO_INCREMENT,
	user_id INT NOT NULL,
	address_line_1 VARCHAR(100),
      	address_line_2 VARCHAR(100),
	PRIMARY KEY (id),

	FOREIGN KEY (user_id) REFERENCES users(id)
);
//
DROP TRIGGER IF EXISTS create_addresses_trigger;
//
CREATE TRIGGER create_addresses_trigger AFTER INSERT ON users FOR EACH ROW
       INSERT INTO addresses (user_id, address_line_1, address_line_2) VALUES(NEW.id, NEW.address_line_1, NEW.address_line_2);
//
DROP TRIGGER iF EXISTS update_addresses_trigger;
//
CREATE TRIGGER update_addresses_trigger AFTER UPDATE ON users FOR EACH ROW
BEGIN
       UPDATE addresses SET address_line_1 = NEW.address_line_1, address_line_2 = NEW.address_line_2 WHERE user_id = NEW.id;

       INSERT INTO addresses (user_id, address_line_1, address_line_2)
       SELECT u.id, u.address_line_1, u.address_line_2
       FROM user u
       LEFT JOIN addresses ua ON ua.user_id = u.id
       WHERE ua.user_id IS NULL;
END
//

CREATE TRIGGER user_delete_addresses_trigger BEFORE DELETE ON users FOR EACH ROW
       DELETE FROM addresses WHERE user_id = OLD.id;
//
INSERT INTO addresses (user_id, address_line_1, address_line_2)
       SELECT u.id, u.address_line_1, u.address_line_2
       FROM users u
       LEFT JOIN addresses ua ON ua.user_id = u.id
       WHERE ua.user_id IS NULL
;

Now we can start version 2

ruby server/v2/server.rb -p 8002

The server is now up and running and available at http://localhost:8002 for testing and warm-up

Now we can reconfigure the load balancer

haproxy -f cfg/v2/haproxy.cfg -p $(<haproxy-private.pid) -st $(<haproxy-private.pid)

HAProxy completes transactions to server 1 but new requests go to server version 2. When there is no traffic to Server version 1 that server can be shut down and the old database columns in the user table finally removed.

ALTER TABLE users
      DROP COLUMN address_line_1,
      DROP COLUMN address_line_2
;

Summary

Achieving a zero down-time upgrade is definitely achievable but with everything there is a cost. The steps outlined in this article still assumed a level of manual intervention running each script in sequence and following checks that the previous operation was successful. A next step would be to integrate some production level post deployment automated smoke tests to fully automate the process. Only then can it become part of an automated delivery pipeline.

Useful articles

http://sysadvent.blogspot.com/2012/12/day-3-zero-downtime-mysql-schema-changes.html

word-finder

2013-08-13T00:00:00-06:00

Following up on the next_permutation from a couple of weeks ago word finder uses next_permutation to find words in character sequences. Uses the aspell C api to look up each permutation. Searching for 3,4,5 and 6 character words in a 6 character sequence takes about 200ms.

Using C++11 meant that I could use a nice lambda expression to process each permutation by passing the function object to the permute() function.

#pragma once

#include "word_accumulator.h"
#include "spell_checker.hpp"

void permute(vector<char> chars, std::function<void (const string&)> f) {
    do {
        string perm;
        for (auto x : chars)
            perm += x;

        f(perm);
    } while (next_permutation(chars.begin(), chars.end()));
}

class word_finder {
    const char *seed;
public:
    word_finder(const char *seed) : seed(seed) {
    }

    template<typename T> vector<T> sorted_vector_of(const T *seed) {
        vector<T> y(seed, seed + strlen(seed));
        sort(y.begin(), y.end());
        return y;
    }

    vector<string> find() {
        spell_checker checker;

        accumulator<string> accumulator;

        permute(sorted_vector_of(seed), [&](const string& w) {

            for (int i = 3; i <= w.size(); i++) {
                for (int j = 0; j <= w.size() - i; j++) {
                    string to_check(&w.c_str()[j], i);

                    accumulator.append_if(to_check, [&](const string& t) {
                        return checker.is_correct(t);
                    });
                }
            }
        });

        return accumulator.sort_by([](const string& a, const string& b) {
            return b.length() == a.length() ? a < b : b.length() > a.length();
        });
    }
};

What's in a name?

2013-08-10T00:00:00-06:00

Graham Brooks: Architect, Software Engineer

Over the years I have been described as a programmer, analyst programmer, software developer, software engineer and many other derivation of software X professional. For most of the time it has not seemed to be important. Recently though I have started to wonder how these names might affect the way software professionals see their role and perhaps how they view themselves.

In certain countries and regions the term engineer has more special definition. Wikipedia emphasizes the method or approach to developing software. In Europe the term infers that someone has undergone formal training and attained a status through examination or peer election to a learned society.

But why should we care?

In 1999 Andy Hunt and Dave Thomas wrote The Pragmatic Programmer: From Journeyman to Master. Their first tip was Care About Your Craft. If like me you care about the craft/practice/discipline of building software and software applications then you probably feel that naming things is very important. Without a clear name and common understanding of what a name means it is very difficult to communicate effectively.

I also take a sense of pride in describing and talking about what I do. That pride comes from the quality of the work that I produce and the study and knowledge that I have gained from building software.

If I am going to use a name to describe what I do it needs to be reasonably well understood, match at least part of my self image.

Searching for definitions brought up a number of suggestions mostly around these themes

Programmer or ‘coder’
- Has less that 5 years experience
- Works from specifications
- Code Monkey
Software Developer
- Gather requirements
- Designs applications
- Researches technologies
Software Engineer
- Designs and implements components, frameworks for developers and programmers to use.

So many people seem to infers a level of experience to each of these terms with Programmer and I think there is a level of experience but also levels of self awareness or interest in particular ares of software.

I don’t see the term programmer being used much these days in job adverts. The most popular term seems to be Software Developer for developing applications. All these terms are fine but while I really enjoy building applications my passion is for building better tools and applying new techniques and languages to the problem. This feels more like engineering to me. So after a an absence of some 15 years Software Engineer is coming back to my vocabulary. No one term can capture everything about a person and does not translate directly to my role at any point in time but it is a starting point.

next_permutation

2013-08-03T00:00:00-06:00

Working on a small program recently I found a quirk in next_permutation. The prorgam read a sequence of characters from the command line and then tried to find words by testing each permuation. For a sequence of 3 characters there are 6 permutations.

Most of the online examples look something like this:

    vector<int> set {'a', 'b', 'c'};
    
    do {
        for (auto x : set)
            cout << (char)x;
        cout << endl;
    } while (next_permutation(set.begin(), set.end()));

Which produces:

abc
acb
bac
bca
cab
cba

No surprises but when I cam to process input like ‘bac’ the number of permutations found were far too small.

bac
bca
cab
qcba

The problem is that the next_permutation modifies or mutates the vector so it needs to have a state to know when it has completed. That state is the vector itself. next_permutation returns false when all the elements are in increasing order.

To work with arbitrary input the sequence first needs to be sorted.

	set = {'b', 'a', 'c'};
    
    sort(set.begin(), set.end());
    
    do {
        for (auto x : set)
            cout << (char)x;
        cout << endl;
    } while (next_permutation(set.begin(), set.end()));

which now produces the right result

abc
acb
bac
bca
cab
cba

Emacs Sharing

2013-07-25T00:00:00-06:00

I like Emacs. Some people like VI(M) but emacs has been in my toolbox for a long time and I feel very at home working with it.

Updating the editor has become easier over the years and installing an update is a simple matter of drag and on my Mac. But I use several computers and keeping each in sync with my current favorite settings was becoming a problem.

I recently came up with the idea of using Dropbox to share a single configuration file set for all the computers I use.

My current setup has all my configuration in an emacs Dropbox folder named emacs. The Dropbox client runs in the background syncing file changes. First I moved my local personal settings into the shared folder

ls -la ~/Dropbox/emacs
total 8
drwxr-xr-x   6 graham  staff   204 Jul 24 15:24 .
drwx------@ 79 graham  staff  2686 Jul 25 19:38 ..
-rw-r--r--   1 graham  staff  2974 Jul 23 23:11 .emacs
drwxr-xr-x  22 graham  staff   748 Jul 24 15:24 .emacs.d
drwxr-xr-x   9 graham  staff   306 Jul 24 09:51 .xemacs

Next I created symbolic links from my home folder to these new files.

➜~  ls -la
...
lrwxr-xr-x    1 graham  staff      34 Jul 24 21:30 .emacs -> /Users/graham/Dropbox/emacs/.emacs
lrwxr-xr-x    1 graham  staff      36 Jul 24 21:31 .emacs.d -> /Users/graham/Dropbox/emacs/.emacs.d
-rwlrwxr-xr-x 1 graham  staff      35 Jul 24 21:30 .xemacs -> /Users/graham/Dropbox/emacs/.xemacs

Once this change has been made on each computer that I use or on a new machine I get a consistent emacs experience. When I get bored with a particular UI theme and make a change that change gets replicated so I have the same settings everywhere.

If

2013-02-14T00:00:00-07:00

Quotes: Things that inspire and be remembered

“If you can dream - and not make dreams your master; If you can think - and not make thoughts your aim; If you can meet with Triumph and Disaster And treat those two impostors just the same;”</blockquote>

Kipling

All truth

2013-02-14T00:00:00-07:00

“All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident.”

Arthur Schopenhauer, German philosopher (1788 – 1860)

I can't believe I missed this!

2013-02-06T00:00:00-07:00

Just over 20 years ago Jack W. Reeves wrote an article in the C++ Journal entitled “What is Software Design?” and I missed it. Not only that but no one thought to point out that I was missing a very important article. An article that challenged and changed/clarified my mental model of software design and construction - 20 years after it was published. A copy of the original article can be found here. I guess it is better late than never and it demonstrates that some things stay relevant and important. Sometimes they remain controversial.

Traditional Thinking

Traditional thinking places writing source code or ‘coding’ firmly in the construction phase. Architecture and design being another activity quite often divorced from the actual code and coding activity. Quite often done around whiteboards or visual design tools. I remember long review sessions to make sure that code actually matched the original design. This view has lead to writing code to being viewed as a commodity which I think very often results in sub-optimal solutions (I am being kind here).

In his article Jack Reeves considers construction to be confined to ‘compilation’ and coding to be design. Even with long running builds in this model the act of construction is short compared to design activities. It also (in my view) more closely matches what building/engineering/crafting software is all about. It also highlights why a lot of analogies to other human activities like house building break down pretty quickly. If building a house from the design was cheap and near instantaneous I wonder what sort of world we might live in.

Dynamic interpreted languages

It is interesting to extrapolate this view to interpreted languages like Ruby and JavaScript. For interpreted languages there is no explicit construction/compilation step. Interpreting and compiling happen during execution typically shortening the feedback cycle significantly - albeit with some impact to runtime performance.

Design is hard

Design is hard, iterative, incremental and demands feedback. Writing code is hard, iterative, incremental and demands feedback. Design is a creative process that is informed both by external forces but also the design being created. Writing code is an act of design not an act of construction - that’s the compiler’s job.

DRYing out code

2012-12-31T00:00:00-07:00

Removing duplicate code is a great way to improve the internal quality of your application code. Duplications mean that you have more code than you should and are often the source of more subtle bugs of the “I’ve already fixed that ..” variety.

While working on a scriptable command line tool for Java (see Automated refactoring for library updates) I discovered a neat way of identifying duplications at the method and block level.

The tool is written in Scala (still learning so improvement suggestions welcome) and uses ANTLR4 to generate an Abstract Syntax Tree (AST) of Java source supplied to the parser. The AST and the Java grammar I am using make it really easy to traverse the tree after processing and identify blocks [cci lang=”scala”]{ code }[/cci]. The lexer ignores white space and comments so the AST just contains tokens of the blocks.

Buckets

Once the source code has been parsed and blocks identified they can be placed into buckets. A bucket is a hash-map of lists. The key for the hashmap must be unique to the significant elements of the block. For this I used a hash from the token text of all terminal elements within the block. The list contains the block represented by the hash. In this way lists with more than one element identify block duplicates.

private def bucketBlocks(models: List[CodeModel]): BucketSet[BlockDeclaration] = {
    val blockBuckets = new BucketSet[BlockDeclaration]
    models foreach (model => {
      model.blocks foreach (block => blockBuckets.add(block.signature, block))
    })
    blockBuckets
  }

The block.signature is a computed hash of all the token in the block. A BlockDeclaration holds a reference to the root node in the AST for the block.

Iterating through all the blocks and added them to the buckets means that duplicate blocks are buckets with more than one block:

class BucketSet[T] {

  val buckets = new mutable.HashMap[Signature, Bucket[T]]

  def add(signature: Signature, item: T) {
    if (buckets.contains(signature)) {
      buckets(signature).add(item)
    } else {
      buckets(signature) = new Bucket(item)
    }
  }

  def duplicates(): List[Bucket[T]] = buckets.values.filter(bucket => bucket.hasDuplicates).toList

  def getDuplicateCount = duplicates().size

  def getDuplicates = duplicates()

  def eachDuplicate(function: (Bucket[T]) => Any) {
    buckets.values foreach (bucket => {
      if (bucket.hasDuplicates) {
        function(bucket)
      }
    })
  }

  private def hasDuplicates(items: List[T]): Boolean = {
    items.size > 1
  }
}

What’s interesting about this approach is that the identified duplicates are naturally low hanging fruit for extracting common methods.

Integrating GPUs in Application Development - From Concept to Deployment

2012-12-24T00:00:00-07:00

This post is a little overdue :(

In June I presented at QCon NYC on using GPUs. It was a great chance to catch up on all the changes to C and C++ that I have missed out on in recent years. You can catch up on the presentation over on InfoQ

The source code used in the presentation is on github https://github.com/grahambrooks/qcon-ctod

Automated refactoring for library updates

2012-07-23T00:00:00-06:00

After watching Clang MapReduce – Automatic C++ Refactoring at Google Scale I was struck with the idea that this could help with the upgrade problem. Almost every application uses libraries. Those libraries need to be updated from time but each time they are updated all the code using those libraries also needs to be updated. For development teams finding time to upgrade to the latest libraries against competing functional updates is challenging. What if as part of the release a set of refactoring commands or programs accompanied the libraries. These refactoring scripts would automatically update the consuming application code to use the new libraries saving time and money.

Google uses the Clang compiler to generate and store abstract syntax tree (AST) information about the build. Google build all their applications from source everytime so the data about a particular version of the source code is known and all the binary dependencies are up to date. This AST data is then processed via map-reduce to refactor the code-base.

Chandler Carruth talks about using semantic predicates to identify source to be updated from the AST data. Similar to modern testing and mocking frameworks the semantic predicates are used to match source code elements to be updated. Refactoring functions are then applied using Clag’s source rewriting system. Chandler mentioned that Google are looking to open source this capability. When writing this I could find no reference to the open source version. Hopefully it will be released into the wild soon.

So if we have a system that can programatically refactor code then the refactoring program could be shipped with particular version of a library to upgrade the client code. Upgrading from version 1 to version 3 would roll up changes from the intervening versions. Code can now be considered data and updated in a similar way we update databases with Active Record migrations or dbDeploy DDL

This capability could be integrated into a Continuous Integration systems and in particular Continuous Delivery pipelines. Large enterprise development teams can keep up with their colleagues changing the libraries that they depend on. The reduction in technical debt in such environments could be huge. Of course its not just about the semantics. High levels of automated test coverage would also be required.

Metrics based Refactoring for cleaner code

2012-01-02T00:00:00-07:00

Refactoring is a key practice to improved code hygiene. Making refactoring part of your next project is one thing but if you have just joined a team or project with a significant amount of debt how do you work on making things better? Over the last few months I have been assessing a number of code-bases and speaking about technical debt management. While preparing for these engagements I realized that combining two code and project metrics could be used to help focus efforts on code that would deliver the most benefit. Toxicity is a combined measurement of static code analysis metrics. Volatility a measure of changes made to files within a code-base over time. By combining these two measures we can create a source file scatter chart correlating toxicity against volatility.

Toxicity Charts have proved very useful in quantifying the amount of technical debt in a code-base. The magnitude of the debt is quantified by comparing a value against an arbitrary threshold. The toxicity is expressed as a score against the threshold.

Thresholds are not quite arbitrary. They are derived from peer code reviews and other observations on how readable a code-base is. A fuller description of these thresholds can be found in Erik Dörnenburg’s article here

The volatility chart is derived from the version control system - typically from a log of activity on the trunk branch. If a team is working of multiple branches then the activity from all of the branches should be included. We are trying to count the number of changes made to each source file over a reasonable period.

Choosing the right time period is key. We are trying to identify files that require frequent changes. If the chosen period is too short then it is going to skewed by the current work. Too long and it could be skewed by some historical instability. A period of 3-6 months should be reasonable time period.

Volatile and Toxic - Refactor Now!

So things that are both volatile and toxic should be our primary focus. The code is in flux. People are working with toxic code on a regular basis. Improvements here will deliver an immediate benefit to the team.

Stable but Toxic - Refactor Later

Toxic but stable code is not causing any immediate problems. If there are identified defects but we are not working on them then they are not causing us any pain (other than knowing there is a big ball of mud waiting the cause problems). It would also seem likely that code in this category will over time either be eroded during refactoring the volatile and toxic code or move into another category over time.

Volatile but Clean

Highly volatile clean code is likely to be caused by unstable requirements. The code is being maintained in a good state but changes are being requested in a small area of the code-base indicating that things have not settled down. It might also be that the sample period for volatility is too small.

Stable and Clean

Obviously this is the idea state. Changes are spread through the system with no clear hot spots. The code is well factored with low toxicity allowing changes to be made more easily. Over time this should be where the majority of you code lives.

Oh My ZSH shell productivity

2011-12-27T00:00:00-07:00

oh-my-zsh is framework for managing zsh configuration. The default configuration adds some interesting enhancements.

The following shows all the Java files in the current and sub directories.

ls **/*.java

Ryan Bates has a nice Railscast episode covering oh-my-zsh

I have been using oh-my-zsh for the last couple of weeks. The GIT enhancement showing the current git branch in the prompt is a nice useful and elegant enhancement.

AgileDC - Introduction to Continuous Delivery

2011-10-27T00:00:00-06:00

Yesterday I presented a talk entitled Introduction to Continuous Delivery at AgileDC. The audience was great and the room packed which is always a recipe for success. I really enjoyed talking about Continuous Delivery and there were some really interesting questions.

As promised I have uploaded the slides from the presentation to broken link

As I mentioned yesterday Jez Humble was kind enough to donate his slides from which I borrowed shamelessly.

Agile DC

2011-10-25T00:00:00-06:00

I am really syched about talking at AgileDC tomorrow. The topic of my talk is an Introduction to Continuous Delivery

Encapsulating Databases

2011-08-11T00:00:00-06:00

Small systems grow with success. As these systems grow they often take on more and more functionality either directly into the main system component or into sub-systems. As the systems grow in complexity and responsibility their database requirements grow at a similar rate become more and more complex.

In these complex systems each component needs to not only have access to data but also to transitive data generated by other components. This need often leads to the database becoming the inter-component communication channel.

This design means that the components are coupled through one or more databases. In normal operation this design is simple and fairly effective. Upgrading areas of the system however becomes more and more difficult whenever database changes are required. Things become even more complex if application logic in embodied in the database in some way (Stored Procedures, Triggers etc).

So let’s consider a simple scenario where two components A and B both require the same data from a table T.

Simple Table dependency

Now lets suppose a change to component A requires some change to the table T. Before we can deploy the update B also needs to be updated to use updated table. Quite often development is not isolated to a single component. Business need to move quickly to take advantage of a shifting market. These changes are unlikely to be isolated to to a single component but changes are often constrained by priority and resources. This added logistical complexity has to be managed by the development team either by extending time-lines or complicating the way the team use version control etc.

Upgrades requiring database changes

Wrapping the database within an encapsulating layer of software that provided application level interfaces to the components we essentially break the direct implementation dependency between the components and the table implementation.

Encapsulating the database - single adaptor

Encapsulating database - multiple adaptor instances

In this configuration if component a A change requires a database change the encapsulation layer can be updated to support the new acomponent A functionality whilst supporting the existing functionality of component B. We need to regression test both components with the new encapsulation layer but in deployment we only need to deploy the new component A and the updated encapsulation layer.

As the system grows further we can break up the database into smaller more manageable chunks with logical groupings within one or more encapsulation layering components.

Over the years I have seen lots of systems develop their integration patterns around the database. This pattern, while initially successful leads to greater and greater challenges for the development organisation. These challenges surface as elaborate proceedures, complex deployment policies and large comlicated deployments.

Insulating against failure using Caching Reverse Proxies

2011-07-07T00:00:00-06:00

Reverse proxies have been around for a very long time and depending on your application either interesting additions or a key element to your architecture.

Despite their long history I was recently reminded of some interesting applications of Caching reverse proxies that are worth revisiting.

First lets look at a fairly typical configuration where a caching reverse proxy is fronting some sort of web application. The web application exposes its functionality on one port (say 8080) and the proxy connects to this port for in-bound HTTP requests (say on port 80). The application responds with the data matching the request and if cacheable the proxy stores a copy of that content in its cache. There are lots of other paths - typically around failure and different responses but essentially the proxy acts as a pipe between the Internet and the application.

Basic Caching Reverse Proxy configuration

Lets assume that the application makes use of some internal services on the corporate network. Typically these might be HTTP based web services. This sort of arrangement is fairly common and quite often the design stops here.

Basic Caching Reverse Proxy with internal service

Since we are using HTTP to communicate between the web application and some internal web service we can add a reverse proxy between the application and the service. In doing so we have added a layer of insulation. Lets assume that the proxy is installed and configured on the web application side. It has been configured to honor cache headers and to serve stale content should something go wrong.

This insulating capability can then be leveraged to to provide a degrading service during updates and system failures.

Caching Reverse Proxy used to insulate from service failures.

Should the service be unavailable for any reason content that is already in the cache can be used. Updates and other non-cached operations would fail as usual but for a large group of web applications this may be acceptable and presenting the user with information about the problem and that it is being worked on may be all that is required.

Just like insulation the proxy does not block all errors propagating from the service, but in the right circumstances it can lessen the blow of something going wrong.

Attributes of this design

These are just some of the attributes that I think apply to this design.

Multiple requests resulting in the same cacheable service request (by URL) results in a single service call.
Simpler application design. The application need not worry about caching service call results.
Resilient to intermittent back-end service failure
Improved performance
Reduced service call bandwidth requirements
Lower load on service calls.
More complex configuration and deployment process
Slightly increased latency for in-bound and out-bound requests.

asciidoc experiments

2011-06-26T00:00:00-06:00

I have always been interested in text processing systems. This is probably rooted in the time that I discovered computing and programming. At that time the state of the art was ROFF (T)ROFF and a whole family of plain text processing engines that produced nicely formatted output.

With the advent of word processing applications the emphasis moved away from plain test precessing systems to more sophisticated systems involving opaque binary formats. More recently the use of binary formats has fallen out of favor and many word processing systems now provide a level of interoperability with other systems though more open document formats.

A notable exception has been in the evolving popularity of Wiki - online collaborative web sites where the content is entered in plain text with formatting instructions included in the text. Over time these formats have evolved into a sophisticated mark up language in their own right.

Plain text mark-up makes it very easy to compare two versions or variants of a document. At GitHub the Gollum system is entirely based on controlling content using Git.

One of the things I find challenging about word processing systems is their lack of features that allow for collaborative writing. Many documents these days are produced by a team of people and in these situations editing a document means someone driving the process, distributing updated version of the document and incorporating changes into a master copy. Often these documents are distributed by email and the filename used to distinguish versions. This approach means that a lot of time is spent updating and merging.

Git Scribe takes a different view. It uses asciidoc formatted text in one or more source files and produces documents in various formats using different back-end processors. In this approach the document source (plain) text can be managed using version control more easily. Changes can be merged together easily. What you lack in fine grained formatting control you gain in improved collaborative document development and more optimal work-flow.

Open Source project durations

2011-04-28T00:00:00-06:00

During a recent discussion about open source development we wondered how long these projects lasted. In particular if there was a rapid drop off in activity.

One of the great things about recent open source code repositories is that they often provide APIs allowing this sort of analysis. The chart below was generated from a sample of 2360 public GitHub repositories. I took the started pushed-at and created-at dates to calculate how many days a repository had been worked on. I then grouped them together into 30 day increments and charted the count in each 30 day period.

This has probably been done before but I find the results interesting.

Stabilizing Velocity

2011-03-26T00:00:00-06:00

I came across this post by Michael Norton and thought I would reference it here: Stabilising Velocity Michael makes some keen observations on both causes and effects of unstable velocity.

Predictable velocity is a key agile planning metric. There is an implicit assumption in agile planning that this iterations velocity will be similar to the last iteration. Without this predicability it is very difficult, if not impossible, to provide and sort of forecast.

Trying to identify the root causes of unstable velocity is difficult. Quite often there are layer upon layer of symptoms masking or hiding the real cause. I delving into root causes I find these techniques useful

Value Stream Analysis – helps identify waste: time waiting for some dependency.
The 5 Why's – for finding the root cause
Story dependency analysis
Cause and effect diagramming </ul>

Rake db:migrate MySQL gem dependency

2011-01-24T00:00:00-07:00

If you have already added gem 'mysql2' to your Gemfile but get a message saying that it is missing when you try to migrate

Make sure that you have installed libmysql-ruby

sudo apt-get install libmysql-ruby

This one caught me out on the weekend and it took a while to work out what was going on. I am guessing that not having libmysql-ruby installed caused the gem to incorrectly advertise that it was not available. I did not get any errors when installing the mysql2 gem which is strange.

Create MySQL database with Capistrano

2011-01-24T00:00:00-07:00

I was playing about with Capistrano over the weekend. I wanted to automate the deployment of a Rails application to my server. The server was (I thought) just about ready to accept the app but I did not want to go through another manual deployment. I thought I would take the opportunity to script the deployment. The first task I set myself was to create a production MySQL database. Searching for how to do this threw up lots of interesting information about building and deploying database.yml files, but not much about configuring MySQL. The first set of tasks I came up with were:

Does the database exist?
Create database if missing
Grant permissions

Does the database exist?

  def database_exits?
    exists = false

    run "mysql --user=#{db_admin_user} --password=#{db_admin_password} --execute=\"show databases;\"" do |channel, stream, data|
      exists = exists || data.include?(db_name)
    end

    exists
  end

Not particularly pretty and would welcome improvements but functional.

Create database

def create_database
    create_sql = <<-SQL
      CREATE DATABASE #{db_name};
    SQL

    run "mysql --user=#{db_admin_user} --password=#{db_admin_password} --execute=\"#{create_sql}\""
  end

This snippet could be a task but I have not yet done that refactoring - the original was parameterised taking in all the required values as parameters.

Grant Permissions

 def setup_database_permissions
    grant_sql = <<-SQL
      GRANT ALL PRIVILEGES ON #{db_name}.* TO #{db_user}@localhost IDENTIFIED BY '#{db_password}';
    SQL

    run "mysql --user=#{db_admin_user} --password=#{db_admin_password} --execute=\"#{grant_sql}\""
  end

Again this step can now be converted into a task.

At the moment the values are coded in the deployment script deploy.rb - something that I definitely want to change. The code should be checked into source control but this would release the usernames and passwords necessary to do deployments. My current thoughts are to extract the runtime properties to a small yaml file and read that in, unless anyone has a smarter idea?

Five Ws sitting on a bench

2011-01-22T00:00:00-07:00

Over the past year I have been involved in preparing quite a few documents, proposals, presentations etc. I have also been asked to review quite a few of these documents. As an author or reviewer I like test what I am working on against 5 Ws and one H to see if the important aspects have been covered. The Wikipedia article provides a good background and history, including the 1902 reference to Rudyard Kipling Just So Stories.

It’s not that every document or presentation needs to answer each of these questions but that it should be a conscious choice not to include something.

A proposal/presentation view of the 5 Ws + 1 H – who is using the system and/or who is the proposal for

Who
- Who is the document for
- Who is going to use the system
What
- What is to be achieved
- What is the system required to do
When
- When is the solution required
Where
- Where is the system to be used or deployed
Why
- Why is the new system required
How
- How is the system to be developed

The new Mac App Store

2011-01-07T00:00:00-07:00

When Apple announced the new App Store for OS X I have to say I was a little underwhelmed at the prospect and did not think the new store would hold much interest. After all I have been pretty happy with finding and installing apps. I could see the point for the iPhone and iPad but for my laptop?

So with today’s updates came the new App Store application and out of general interest in what has been updated and installed I decided to take the app for spin.

And now I think I see where it all fits in. The interface is clean, crisp and easy to use, but most importantly applications are easy to find and look tempting. So tempting I decided to purchase a couple of likely looking apps that I might find useful. After entering Apple ID details the installation begins with a really nice animation of the application icon landing in the dock.

By providing a seamless link between developer and consumer I think the new App Store is going to be a big hit, especially with solo developers and startups. I suspect we are going to see a massive explosion in applications available for the mac.

Pulse Camp 1.0 and RHoK

2011-01-07T00:00:00-07:00

One of the great things about working at a consultancy is that I get to meet a wide range of interesting clients with interesting problems that need solving. One of the most interesting for me was Pulse Camp 1.0, a 3 day event to start defining a system framework that can help NGOs and other groups make better informed decisions based on real-time data.

The UN Global Pulse initiative is in the early stages of development but the excitement, participation and drive I experienced at Pulse Camp 1.0 fills me with hope that this ambitious program of work will be successful.

Pulse Camp 1.0 closed with speeches from the UN Secretary-General Ban Ki Moon and the supporters of Random Hacks of Kindness. The speeches marked the end of the Pulse Camp and the start of the RHoK. RHoK is an open global event that matches people with ideas that can improve our world with people who can help create the software make those changes.

Details of the Pulse Camp 1.0 can be found at http://www.unglobalpulse.org/tags/pulse-camp-10 and ongoing work is being captured here http://sites.google.com/site/unglobalpulse/home

Progressive Enhancement

2010-12-24T00:00:00-07:00

Progressive Enhancement is a web development technique or pattern. The basic premise is that a web site should be accessible to all users and then to overlay additional functionality based on the client’s capability.

Web technologies and good web development practices over the last few years had encouraged this approach. Popular web sites delivery semantic HTML to allow screen readers and other client devices to understand the content of being delivered. Advances in both browser capabilities and CSS have encouraged this and the use of inline styles has reduced over time. The advent of JavaScript libraries that hide browser compatibility problems have made the development of progressively enhanced content easier to achieve.

Progressive enhancement can be used in a number of ways. Here are a few examples:

Segmentation by freshness
Included function

Segmentation by Freshness

The term Segmentation by freshness was coined by Martin Fowler to describe how progressive enhancement can be used to partition content by how fresh or how frequently it is updated.

Included Function

Many web sites tailor their functionality based on how well you are known to the site. An anonymous user can perhaps browse content but not contribute to the site. To contribute content you need to log in perhaps using a username and password.

To provide this feature pages might include a link to the login form <a href='login'>Login</a>. In the basic implementation this is a complete in terms of functionality. The user clicks on the link, is presented with the login form, which they fill in and submit. If the credentials are correct they are redirected back to the page containing the link - but are now authenticated and have access to additional features.

Although functional just providing a link to login seems a little light. Most web sites designers would prefer to include the login feature directly in the page and not require users to navigate to another page.

Including the login form in every page is both time consuming and leads to maintainance problems later when perhaps an additional ‘remember me’ feature is required. One option is to do some sort of server side include where the contents of the form is included by a server side statement and the page is aggregated before the content is returned to the client.

An alternative approach is to use JavaScript in the client browser that replaces the link to the login page with the contents of the page returned by the link’s href attribute.

So far these two approaches appear the same, but what if the request for the login form checks to see if the user is already authenticated? In this case the request might return a different result ‘Welcome back Graham’ or some other message. For the server-side include this would mean that the page contents depends on the user requesting the page and is therefore not cacheable. For the JavaScript version the main page is cacheable and only the small login form or message is specific to the users requesting the content.

On the server side things are simpler too. All included content is delivered as web content over HTTP. For each piece of included content an additional request is required but this request does not slow down the rendering of the main page because it is loaded later (although the delay may be imperceptible to the user).

Emacs Package manager

2010-12-24T00:00:00-07:00

I have a long history with text editors and I have always been facinated with the different flavours of interfaces and capabilities. After working my way through ED and then VI Emacs was a bit of a revalation. I found myself idle for a few minutes this morning and thought I would do some editor maintenance and install haml-mode into my copy of Emacs. While surfing around I came across the Emacs Lisp Package Archive which made the installation a breeze and I then found myself installing all sorts of other additions that might be useful in the future.

ELPA installation instructions (copied from the ELPA site)

(let ((buffer (url-retrieve-synchronously
	       "http://tromey.com/elpa/package-install.el")))
  (save-excursion
    (set-buffer buffer)
    (goto-char (point-min))
    (re-search-forward "^$" nil 'move)
    (eval-region (point) (point-max))
    (kill-buffer (current-buffer))))

Paste the code into a scratch buffer, and then execute the code in the buffer. Once complete run the package-list-packages command. In the packages buffer ‘i’ marks an entry for installation and ‘x’ installs the marked packages.

Haml

2010-12-19T00:00:00-07:00

Haml is a markup language for generating xml and other markup - most popularly HTML. Over the past few weeks I have been writing a few Ruby on Rails applications and chose Haml as the templating language. For someone who has traditionally avoided positional languages this was a strange choice. Having written a few simple applications I find that the writing HTML in Haml is both straightforward and intuitive. Good HTML is naturally hierarchical and having this structure both encouraged and enforced in Haml feels right.

Generating

#some-id.someclass

is one of the great things about haml

Produces

<div id='some-id' class='someclass'></div>

There are definitely some quirks. Sometimes I have to switch styles or div tags and in some cases declare them explicitly with a %div. These cases are generally when generating html in iterators and not a big deal. %script tags, in line with the rest of Haml, can’t have additional indentation - which encourages me to take this content and push it into included JavaScript files, something that I think is good practice anyway.

Watching the white horses on the Hudson ...

2010-12-12T00:00:00-07:00

Watching the white horses on the Hudson today and wondering why I am flying on a day like today.

A good day of discussions in Chicago tod...

2010-11-16T00:00:00-07:00

A good day of discussions in Chicago today. Although I should say yesterday given the time.

Test Driven Talks

2010-11-07T00:00:00-06:00

Taking a leaf out of the XP book and in particular test driven development I have had some successes in using this idea when doing a presentation. I have dubbed the idea ‘Test driven talks’. The basic idea is to quickly canvas the audience for things that they would like covered during the presentation. Once these have been captured on a whiteboard or flip-chart.

The next step is to filter the topics in the same way as an agile team accepts stories into an iteration it is likely that I wont be able to cover all the expectations from the audience. The nice thing about this step is that I can manage people’s expectations. If I accept a question or topic then people should expect me to satisfy them. If I reject the topic then people know that I won’t be able to satisfy their need.

If the talk is going to be repeated then each time it is done involving the ‘Test Driven Talk’ approach allows me to tune the talk to satisfy a greater proportion of the audience. It also provides inspiration for other related talks – based on audience need.

I am not sure how this approach will work for larger audiences. And I am not sure what the break point in audience size is but hopefully over time I will be able to refine the technique and gather more data about how effective it is. Given that I am not a known or popular speaker finding speaking opportunities is likely to be a challenge but maybe in the space of a year I might find enough events to gather enough data points to prove or disprove the effectiveness of the technique.

If anyone else is trying this sort of thing I would love to hear about your experiences

Testing - what to test when

2010-10-20T00:00:00-06:00

After tweeting that I wanted to have suggestions for blog entries I got a single reply asking for my thoughts on DAO and unit testing – essentially asking should DAO be unit tested.

The problem with me answering this question is not that I don’t have an opinion but as the question was posed I was reading Steve Freeman, Nat Pryce on just that topic in their new book ‘Growing Object-Oriented Software, Guided by Tests’.

I have taken the diagramming style of the book for my own effort:

I think that the emphasis should be on keeping the ring around the core application as thin as possible. To definitely test persistence but to not make it part of the initial unit testing cycle. Unit tests need to provide rapid feedback and compared to inline code execution setting up and interfacing to a database is typically orders of magnitude greater.

Technology Synergy

2010-10-20T00:00:00-06:00

I started writing this post quite some time ago but never got around to wrapping up the loose ends. The article is really a summary of what I learnt during a pretty intense media web site development project. Since pictures equate to 1000s of words here is my effort to express how technologies can work symbiotically to delivery value that is more than the sum of their parts.

But since systems are built from parts: here are the parts that we used.

Extended SiteMesh

SiteMesh is a pretty amazing technology for creating web sites mashups – or combining HTML documents on the server. We found that by partitioning the content and the parts of the server that delivered that content before combining the individual HTML documents together using SiteMesh to be extremely effective. The server code for each HTML document need only handle the a well defined area but the site could be easily reconfigured to aggregate the content for new purposes.

In memory tests

Think super fast builds, no deployment (in fact no need for a web server), pure Java acceptance tests (i.e. refactorable) and the 80/20 rule on what is good enough to give you confidence that the system works.

Post/Redirect/Get pattern

We built into the application framework that enforced the PRG pattern. POST request handlers could only return a redirect. Any database access performed during a GET request worked with a read only connection or a connection with an transaction rollback making GET requests idempotent.

Simple transaction boundaries

Transaction boundaries have been enforced so that developers do not need to worry about them at all in production code. GET requests run in a transaction that will always be rolled back, POST requests will always commit if successful and always roll back if an exception is thrown. Some serious work went into taming Hibernate so that it did not auto commit, had no mutable static state and was completely encapsulated.

AHAH (Asynchronous HTML and HTTP)

This enables much simpler JavaScript to be written (think JavaScript that never needs to replicate the domain model or business rules on the client) and allows for complete reuse of server side logic. This combined with behaviour CSS bindings ( http://www.bennolan.com/behaviour/ ) leads to NO in-line JavaScript nastiness and more semantic html.

No Session State just persistent documents

The application servers had NO session state, all state changes to documents are persisted which leads to a number of advantages: Users never loose data they have filled in, marketing can see exactly how far a user got before bailing out of a work flow. Users can return to incomplete work and complete the tasks. Users can fill in data in any order they want and only when they are ready do we action their request. Domain objects are only updated once the document has been validated.

Progressive Enhancement and Accessibility

All stories are played vanilla html version first and a second story for JavaScript enhancements. This leads to cleaner simpler semantic html and also allows feedback from the first story to be tacked onto the second story.

NO Logic in the View

We are using StringTemplate to enforce that NO logic can be written in the view, plus it’s super fast, has no “or loops” (think ruby each blocks).

On my current project we are using a number of complementary technologies and techniques. The overall effect is (IMHO) greater than the sum benefits of each design component and I thought I would try and capture this idea in a diagram.

RESTful

Unique URLs for each resource is a simple technique but is a liberating experience. Combine this with persistent documents that maintain workflow state and Post/Redirect/Get and you have a pattern that handles situations very cleanly, making many design decisions for you.

Post/Redirect/Get pattern

We have built a simple interface contract that enforces that all POSTs return a redirect so that the Back button will always work. This combined with the transaction boundaries ensures that all GET requests are idempotent.

Simple transaction boundaries

Extended SiteMesh

AHAH (Asynchronous HTML and HTTP)

This enables much simpler JavaScript to be written (think JavaScript that never needs to replicate the domain model or business rules on the client) and allows for complete reuse of server side logic. This combined with behaviour CSS bindings (http://www.bennolan.com/behaviour/) leads to NO in-line JavaScript nastiness and more semantic html.

No Session State just persistent documents

We have NO session state maintained within the application. All application state is maintained in the database. Partially completed workflow steps are persisted to the database as documents representing the partially complete workflow. On completion these documents are validated and converted into domain objects.

In memory tests

Progressive Enhancement and Accessibility

NO Logic in the View

We are using StringTemplate to enforce that NO logic can be written in the view, plus it’s super fast, has no “for loops” (think ruby each blocks).

Fighting layout bugs

2010-10-06T00:00:00-06:00

Finding problems in the layout of web pages is just plain hard. But fighting layout bugs seems to hold a glimmer of light for web developers. Definitely something to check out. Michael Tamm presenting his brainchild is available http://www.infoq.com/presentations/Fighting-Layout-Bugs

Eben Bayer: Are mushrooms the new plastic?

2010-10-06T00:00:00-06:00

Fascinating TED video on using mushrooms to create materials with plastic characteristics. It looks like Nature got there first...

</param></param></param> </embed>

Dallas geek night

2010-08-18T00:00:00-06:00

Last week I happened to be in Dallas for a client and was invited along to the Dallas Geeknight held at the ThoughtWorks offices there.

The Dallas Geeknight is ‘oldschool’ which is not surprising since Paul Hammant informed me he formed the original Geeknight in the UK. The primary driver is producing code and committing that code to a repository.

At the start of the evening people offered up their current open source projects elevator pitch style. I pitched two projects bvira and Intercept. There was interest in both but as most people have their own web frameworks the consensus was to work on Intercept.

After some initial hiccups around the IDE files for the project we were up and running. For most of the evening we worked as a group of 3 or 4 on a single laptop. The discussion and engagement was great and I am deeply thankful for the interest in my pet project. A lot of my work at the moment is around talking, writing and generally I don’t spend enough time working with code to satisfy the geek side of my nature. It was a truly refreshing experience to spend a few hours having fantastic technical conversations — at the end of the evening I could not believe how great I felt about what had been achieved.

A big thank you to everyone who was there but especially those who gave their time to work on Intercept.

Dallas Geeknight is definitely an event I will be attending again when I am next in Dallas.

Using a local Wiki for personal knowledge management

2010-08-03T00:00:00-06:00

For sometime I have been wondering how best to manage all the information that collects in my laptop. I don’t even have to try to collect data it just arrives in an irregular stream of emails, instant messenger and other symptoms of this connected world. And then there are the other things I want to collect, data from websites, blog articles, eBooks, PDFs and other documents. Added to this meeting notes, observations, ideas for new projects.

All of this information is actually quite difficult to manage. For some time I have been organising things on the filesystem but this approach only captures data in a very structured way. It is not easy to cross link data from one point in the directory structure to another (I could use symbolic links but that would quickly breakdown with such a high maintenance load.

Last night I remembered that someone had suggested using a person wiki to help manage all this data so I thought I would give it a go. I wanted the system to run on my laptop. I am not always connected to the internet and a local installation helps when I am far away from wireless and 3G. This approach also minimises security concerns in case any sensitive data ends up on the site. I tried a couple of wiki solutions and decided on MoinMoin. Written in python this wiki server uses the filesystem to hold its pages - which helps with recovery if nasty things happen.

After some happy hours organising my data I have to attest that using a wiki for this is a liberating experience. I have not deleted the original files yet but being able to simply link information together, including journal entries, meeting notes to people, technologies. I have not settled on a structure yet. I am letting the data that I collect over the coming weeks guide its organisation before taking a careful look at how it could be organised. At the top level I have Work, Profession, Library and Incubator. The last is a bucket for any ideas that I come up with. If the ideas collect data then they might be worth pursuing - only time will tell. The Library is where I put all my PDF format eBooks which I can then cross link from pages related to their content.

If you have not tried it yet I recommend installing a local wiki to see if works for you.

Update:

For some time I have been collecting thoughts in an electronic journal using one file per day and naming the file based on the YYYY-MM-DD format. The following bit of ruby then generates a simple reverse chronological listing of those journal entries as a page called ‘Journal’. The regex is particularly lazy and could do with a cleanup but the script is functional for my immediate needs.

For this code to work you need to enable xmlrpc which appears to be off after a clean install. The simplest but least secure is to enable this in the wikiconfig.py file by adding the line

actions_excluded = []

require 'xmlrpc/client'

server = XMLRPC::Client.new('localhost', '/?action=xmlrpc2', 8080) 
journal_page = ""

result = server.call("getAllPages")
result.sort.reverse.each do |page_name|
  if page_name =~ /....-..-../
    journal_page << " * \[\[#{page_name}\]\]\n"
  end
end

puts journal_page  

server.call("putPage", "Journal", journal_page)

Agile Software Development and Golf

2010-06-30T00:00:00-06:00

Faced with the question ‘What do you do?’ from family and friends is more challenging than it should be. Most other professions have been around longer than a generation but writing software applications is different. Especially for my parents and so this is for the vast majority of people who don’t do what I do but would like some sort of idea of what I do.

But first a disclaimer – I don’t really know that much about golf. I have played a few rounds in the dim and distant past and watched a bit on the TV. My apologies in advance to golfers everywhere for the mistakes I am about to make.

Teeing off

Every project needs to start somewhere. If a round of golf were a project then it would start at the first Tee. Arriving with golf bag with an arsenal of clubs ready to begin. Even before arriving at the first tee there has been a bit of planning, appropriate clothing for the weather, umbrella in case it changes on the way, score card and pencil in hand it is time to start the project/game.

If the fairway does not stretch out in front of you then this is probably pitch and put and you are overdressed and equipped. Take a couple of clubs from the bag and put the rest back in the car. Pitch and put might be an appropriate metaphor for the support and maintenance phase of a software project but that is not the game for today.

So the first step is to work out how to approach the first hole. This is similar to planning the first release. If the hole is close by you might be able to reach it in one shot. Typically though the pin is some way off in the distance, perhaps behind some trees or lake and although you might have a general idea of where it is, trying to get there in a single stroke is very risky and could land you in a whole lot of trouble. The sensible approach is to work out how much of the course you do know about (the bit in front of you) and try to get the ball a far down the fairway as you can without hitting an unknown obstacle. Each shot can be compared to an agile software development iteration which takes on enough work to move the project forward and deliver value without putting the overall project at risk.

The drop shot

What makes golf and software development so interesting is all the factors that can affect the game/process. Weather, tools, mood, other people. Sometimes even the best players end up a long way from where they intended to be and have to take a drop shot to continue the game.

Sometimes development iterations don’t go as expected. There are so many variables that predicting the outcome of an iteration let a lone a full product release is not reliable.

Optimising build times

2010-06-28T00:00:00-06:00

I just came across this post and want to remember it so posing here. http://doublebuffered.com/2009/02/11/optimizing-build-times-for-large-c-projects/. I am most interested in the uplift in compilation time based on unused #includes and build reductions for SSDs.

New BTree algorithms for virtual memory systems

2010-06-20T00:00:00-06:00

This is a very interesting article http://queue.acm.org/detail.cfm?id=1814327 which demonstrates that we should not take things for granted - including algorithms that have been around for a long time.

Blogging from emacs

2010-05-05T00:00:00-06:00

After installing WebloggerMode some time ago I had not gotten around to trying it out so I thought I would take it for a spin with this entry.

Nothing earth shattering but the interface is clean and I have a personal commitment to reaquaint myself with the power of emacs.

Installation and setup is straightforward although I would like a little more 1,2,3 style instructions. But then there are only 3 basic instructions before this point:

. Install . Configure . Create first entry

As an update the post is submitted as draft. I then pulled the entry back down with weblogger-fetch-etries which then provides a number of additional fields.

Performance testing web applications

2010-04-25T00:00:00-06:00

When performance testing a web application (as in raw operations per second) I have seem many people try to benchmark their new system against the agreed performance level right off the bat. The problem with this approach is that most applications need to be tuned to get the most out of them. Optimistically firing off 100s of requests will most likely cause the server to choke and if you are unlucky die in a gibbering heap.

A more successful approach is to establish the maximum load that the application can cope with and then looking for performance bottlenecks. When operating at peak load any measurements are more likely to indicate true areas for improvement. Overloading the server is more likely to show hot areas around resource allocation where each thread is fighting for survival. By resolving issues around the first constraint (memory, threads, database connections, cpu) and then moving on to the next is most likely to yield rapid results.

On a fairly recent project the performance environment was not available until late in the project cycle. I was pretty sure that we would see some horrific results from the first performance run although I was confident that the architecture would support the anticipated load we had not run in a production like environment with production volumes of data. Sure enough the first few runs were far from satisfactory but over the following few days transaction times were down by just over two orders of magnitude. When an application first experiences heavy load it is important to maintain perspective and work through the problem. It is also helpful for this first run to be private to the development team so they can tackle the problem without attracting too much attention – ideally performance testing would be happening throughout the build. Most of the time ideally does not happen that often.

ReleaseIt partial review

2010-04-10T00:00:00-06:00

I have been catching up on my reading recently and one of the books I have been trying looking forward to reading is ReleaseIt. I had heard very good things about the book and to my delight they are true. If you are a developer building almost any sort of application but in particular working on java based web applications you need to read this book! I wish it had been available years ago and that I had read it.

On the last couple if java web projects I have worked on we implemented a form of system health check which allowed us to interrogate the running application and retrieve data about it’s health. Implementing this sort of feature started (for me) as a way of knowing the properties a particular server was using. Production environments are particularly opaque for most development teams for good reasons but this makes problem diagnosis very difficult, especially when there are problems just following a major update and the pressure is on to find a solution. Tension is high and although everyone wants the problem found, everyone is also just as keen for the problem not to be theirs.

Having a page at lists the properties and their values (except passwords) available internally to the development team can really speed up problem diagnosis.

The problems we were seeing which led to the development of these status pages matched the recommendations of the book. As did an embarrassing number of other incidents. Reading the book will help me not experience the other situations that i have not encountered yet!

As a bonus it’s a really easy read. Highly recommended.

7 days with the iPad

2010-04-10T00:00:00-06:00

So it has been 7 days since buying an iPad. Has it been the life changing event promised by all the pre-launch advertising?

In my case I don’t think my life has changed but it has been enhanced by having one. In a similar way that the iPhone made many on the go tasks easier and in a lot of cases fun the iPad has made other tasks more engaging and fun. I wanted an eReader that would allow me to read technical books while on the go and was willing to pay for that facility. The iPad delivers. Reading is a pleasure and since I already had a nice collection of books from the pragmatic programmers who offer multiple formats for each of their books my bookshelf is nicely full. For other publications in PDF Calibri does a reasonable conversion.

Other things that I really appreciate:

bulk email operations - but i am looking forward to the unified inbox and multiple exchange account.
watching TED videos through iTunes
reading tweets. The UI is just made for scanning and absorbing lists of data.

And just generally how the device has made a place for itself in my life. An idea pops into my head and I can capture it or dig deeper without bringing out the laptop.

Scraping Railscasts

2010-04-07T00:00:00-06:00

I had a few moments today to get reacquainted with Ruby and Rails programming. It has been a while but I have found time to watch the occasional Railscast. I want to watch these videos while on the move but unfortunately the iTunes podcasts are not compatible with the iPad/iPhone. Downloading each one is tedious so I came up with the following:

require 'open-uri'
require 'hpricot'

(1..6).each do |page|

  doc = Hpricot(open("http://railscasts.com/episodes?page=#{page}"))

  (doc/"a").each do |l|
    href = l[:href]
    if href =~ /.*ipod_videos.*/ then
      file_name = File.basename(href)

      unless File.exist?(file_name) then
        puts "Downloading #{file_name}"

        File.open(File.basename(href), "w") do |file|
          file.write(open(href).read)
        end
      end
    end
  end
end

iPad and keyboards

2010-04-06T00:00:00-06:00

There are some quirks to the way the apple bluetooth keyboard works with the iPad. Most of the keys work as you might expect and the pairing system works just fine. There is no real way to navigate around applications and the system that I have found using the a keyboard though so I find myself switching back and forth between screen and keyboard.

The dock is quite vertical which can make the screen position a little off if your are tall and working on a normal desk and chair.

This setup is not what the designers were originally thinking when they created the device but for typing blogs, email etc with my large hands using a keyboard is a lot quicker and easier.

iPad first day on the road

2010-04-05T00:00:00-06:00

Today I took my new iPad for a day at work:

A colleague had a 3G card for his laptop and shared the connection. Strangely this caused the iPad to pause quite often as if it were trying to connect to something on the net. The connection seemed to drop quite frequently and I was asked to reconnect quite often.

In a meeting I took notes using Pages. On the whole that was successful but downlighting from the ceiling was a little distracting. I was using the iPad with the apple case which inclines the iPad at an angle that makes the keyboard easier to use.

I still can’t touch type but I am getting closer.

iPad first impressions

2010-04-04T00:00:00-06:00

There has been a lot of hype around the iPad and I have to confess that after watching the original advertisement I was hooked. But was all the hype justified?

On first use if you are an iPhone user there are few surprises. The larger touch screen is truly superb as an interface and the apps load quickly. I was disappointed that the iBook application was not installed by default and it took me a while to realize I had to download it from the app store.

Generally the zoom feature for iPhone apps works well but I am sure that iPad versions will be out for the few that are not available In that form yet.

Pages and keynote especially work well on the iPad. But apple were not kidding when they said that email, calendar and addressbook were fantastic. For me having all my calendar entries laid out on a touch screen makes a lot of sense. As well as being great to look at.

The keyboard will take a little getting used to and I am not sure if I will be able to touch type on it. My hands are just too big.

One disappointment is the small docking station is not compatible with the protective sleeve. I don’t want to keep taking the iPad out of the sleeve so I suspect that the dock will find little use for me unless I pair it with my wireless keyboard.

Overall I am really impresses there are some UI querks that take a little getting used to but the form factor and UI in general more than make up for it.

Tomorrow will be the first day of iPad in a meeting - makes going to meetings much more fun :)

Post Redirect Get web interaction pattern

2010-03-27T00:00:00-06:00

Post/Redirect/Get or Redirect after Post is an HTTP interaction pattern that can be used when developing web applications. I have been mentioning it quite a few times in my consulting work and thought I would take a stab a at diving a little deeper in the pattern and its benefits.

Post-Redirect-Get pattern

The sequence diagram above shows how the pattern works to create a resource. First the client requests a form that is used to create the resource. The server obliges returning a HTTP 200 status code and the mark up containing the form.

<form method="POST"> <input name="somefield" type="text" />
    <input name="submit" type="submit" />
</form>

For this example the user enters data into all the required fields and submits the form.

The server receives the requests processes the data, realising the data in some backing store and returns a 3xx status code (redirect) to the client with the url for the object created in the store.

The client honours the redirect and makes a request for the URL returned in the redirect message. The server returns markup representing the new resource.

On the server side

Model View Presenter pattern (MVP) is derived from the Model View Controller software pattern that has proven to be very effective for UI base applications.

Model

represents the collection of objects containing the data and operations of the system. The model may be a Domain Model or not.

View

The view is responsible for rendering data to the user through the UI.

Presenter

The presenter is responsible for handling GET HTTP requests, retrieving the model from store and possibly manipulating the data to be available for the view.

Command

The command is a responsible for handling POST HTTP requests. Data presented to the server as part of the request is processed by the command to manipulate the model. Commands return a redirect URL that results in a subsequent GET request handled by a presenter. The diagram below illustrates the relationships between Model, View, Presenter and Command.

Statically-typed language implementations of these components revolve revolve around classes derived from ‘model’, ‘view’, ‘presenter’ and ‘command’. Frameworks based on dynamic languages might take a different approach with presenter and commands being implemented as methods on a single class.

Putting things together

The sequence diagram below is my attempt at illustrating the sequence of operations and client/server interactions that are present when combining PRG with MVP(C).

The sequence represents the following sequence of user interactions

Visit the site (home page)
Click on the /someform link
Enter invalid data and click the submit button
Review errors, correct the data and submit
View the result of the operation

Each post request results in a write to the database and is followed by a GET request to retrieve data that was updated as a result of the POST. This may appear to be a significant overhead but it does ensure that no state is maintained in the server which simplifies load balancer configuration and the ability to upgrade servers with minimal disruption to service. Unless large amounts of data has been stored in cookies then the redirect-get is not expensive in terms of bandwidth but latency can become an issue.

Breaking up interactions simplifies the server code, each operation is represented by a method or class. These request handlers only need to handle the single request which avoids more complex logic to understand the client state. In PRG the data is represented in the URL and the HTTP verb (with some user related data in cookies perhaps).

A comment from a colleague:

The big advantage of PRG for me is that it opens the possibility of being able to return to partially completed workflows, and being able to return to long-running server tasks by bookmarking (or refresh buttons or back buttons in browsers) without worrying about duplicate form posts. – Tom Czarniecki http://watchitlater.com/blog/>

The MVP pattern does not separate request handling based on the HTTP verb. This separation helps encapsulate the difference in handling a GET and POST request.

Collaborating distributed teams

2010-03-23T00:00:00-06:00

Internet Relay Chat (IRC) has been around for a very long time (1988) and Instant Messaging (IM) for even longer – although IM did not become really popular until the mid 90’s with the emergence of GUI clients. There are many similarities between the two systems but for this article I am concentrating on the differences, specifically around their use as a team communication medium.

IM is optimised for point to point communication between two people. There are options for group communication but this is more of an exception where one person takes on the role of organiser and brings other people into the conversation. IRC on the other hand uses the channel concept, where a channel is synonymous with a topic of conversation. The channel exists independent of the participants in the conversation. Neither system offers a view of the entire conversation, although a lot of clients log conversations.

For teams working on a software project effective communication is a key ingredient for success. Ideally every member of the team would be as close to one another as possible to foster a sense of team and effect the least expensive way of communicating - face to face speech.

For teams that are more disparate the cost and effort in maintaining a effective communications is much higher. The separation of team members need not be that great to effectively sever the communication paths available to a co-located team. In these instances electronic communication methods can help.

When working with a distributed team though Instant Messaging although useful is not as useful as IRC - the workhorse of open source development teams. Where IM provides efficient point to point communication between individuals, IRC helps create a community and allows topics to be discussed in an open forum. IM suffers from the same problem as e-mail – the person you are trying to talk to has to be there to respond to the message. IRC on the other hand tends to be topic based. You may know who is most likely to respond to a question or comment but it could be that someone else has more recent or better information to share. When something is unknown or opinion is being sought of the group IRC wins again.

Most organisations don’t want their project information leaking outside the organisation but since IRC has been around the Internet for a very long time there are plenty of open source options to choose from when setting up a server. It is often common to have automated (ro)bots listening to the IRC chatter, collecting conversation and contributing things like build status or even version control commits.

Communicating via IRC has a very different feel to IM. Because the information shared is generally available to everyone on the team (and perhaps beyond) the conversations tend to be inclusive rather than exclusive. Questions are directed at the how/how/where/when rather than the who.

One problem is that if you cannot be connected to the stream of information flowing from IRC then it is sometimes difficult to understand what is happening when you join part way through a team conversation. Bots can help capture the stream and persist it for review and possible search for those times when you know a topic has been discussed in the past but you can’t remember when or what the conclusions were.

If you have not tried IRC for distributed team communication and are having long email converstations, give IRC a go.

It always takes a second

2010-03-18T00:00:00-06:00

Why do application developers think that adding more functionality is always a good thing for the user? When I first started working on GUI applications (Windows 1.0) we worked very hard to get sub-second response times. Now in theory applications running on multi-core 2+GHz processors should out strip 286 CPUs running in the MHz range. What seems to have append instead is that applications have maintained the same sort of response times (in general) taking 1 or more seconds to perform a task.

In most cases I am happy to wait for a second. Typically I am switching tasks and short waits give me a chance to think ahead a little bit so I can get going quickly.

When I am working on a single task that involves multiple applications then waiting – even for a second just feels too long. Sure the applications have a lot more features but for most of my development work I want speed. Waiting around for an application to start is a complete waste of my time (most of the time).

So the next time you are working on developing application keep response times down — And I mean way way down in the 100ms range. And if you are waiting on potentially slow resources (e.g. network) then for goodness sake spin off a thread and don’t keep me waiting around for the network to time out – I have enough grey hair as it is and I don’t want to grow old waiting for the connection to time out.

Inclined Keyboard on MBP

2010-03-06T00:00:00-07:00

As anyone who knows me will say I have a thing about keyboards and have a few in my current collection.

On a whim today I bought the apple wireless keyboard thinking that I would pair it with my laptop for when I am at home working at my desk. I already have the wired version and have been very happy with the feel of the keyboard. Unfortunately it occupies a USB slot and when I have both my external drive and SLR connected I have to keep swapping peripherals.

So when I got home with my shiny new toy I found the desk rather crowded. Now I know I could have cleared things up but decided to go ahead and pair the new keyboard while placing it on my lap. By happenstance I got up and put the keyboard on top of my MBP keyboard – now nothing very special about that but I realised that I now had an inclined keyboard covering the original and still giving me access to the trackpad.

The arrangement works pretty well. Everything is where it should be and the keyboard does not obscure the screen from a normal viewing position. The rubber on the bottom of the keyboard makes if feel pretty secure – well at least it has not shot across the room so far.

Time to take it on the road…

Face to Face pairing

2010-02-28T00:00:00-07:00

Since joining ThoughtWorks in 2004 I have enjoyed Pair Programming and my programming skills have improved significantly. Working on a problem with someone else full time is one of those practices that is difficult to convince people off until they have actually tried it. Laurie Williams at NCSU has done some pretty interesting research into the effectiveness of of pair programming, which is well worth a read.

There are various different styles of pair programming WikiPedia talks about remote and ping-pong paring where people sit side by side using the same screen, keyboard and mouse. Work is typically split along test/application boundaries where one pair writes a test and the other then implements the code to support the test. Control switches back and forth between each of the pairs. The are many variations on this including the use of two keyboards and mice. Pairing in this way with a laptop is often quite difficult unless a separate keyboard and mouse are used.

Yesterday

Yesterday and colleague and I needed to work on a document. Nothing particularly unusual about that but we were also bot h in the same office, which for me is unusual at the moment. We both had laptops but the office did not really lend itself to side by side pairing. Laptops don’t really lend themselves to effective pairing. The screen and keyboard layout encourage the user to sit directly in front of the screen to work. For the pair not actively typing that means looking at the screen from the side – and almost over their colleagues shoulder – which is far from ideal.

Thinking about this problem I remembered reading an article about face to face or screen pairing. Screen sharing software is available for most operating systems these days and for OS X users the facility is natively available in the Sharing System preferences.

Setup was a snip we positioned ourselves on a section of the desk that stuck out into the room with the laptops placed back to back. Each of us had an unencumbered view of the shared screen and working on a local network meant that there was imperceptible latency between one person typing a character and the other seeing the result.

What went well

Communication was easy and unencumbered. I found talking face to face to be a lot easier easier than turning to talk to someone next to me and eye contact was easy to establish.

Switching control back and forth did not require moving keyboard our mouse making the transition more immediate and helped improve the flow of work. Each of us worked with the hardware we were familiar with and although we were both using MacBook Pros there are nuances in keyboard feel and trackpad set up that might take someone a few moments to get used when they are driving.

Often when I am pairing I find the need to draw diagrams or make notes to follow up on later. Sitting side by side it is easier to doodle a quick diagram to elaborate and idea with person sitting next to you. Working face to face one or both would need to move to a more convenient location.

Face to face pairing might be a very similar experience if applied to remote pairing. I could imagine setting up a couple of monitors one on top of the other at each location. The upper monitor showing the remote pair. In this configuration it is then easy to quickly context switch to talk about the problem at hand. I suspect that looking up is a lot less work than looking to one side or switching applications.

Server Rebuild - Part II

2010-02-25T00:00:00-07:00

So after having the server nicely set up and ready for configuration I could not resist just ‘getting on with it’. I had grand plans to plan it out and record everything (good practice) but each step seemed so simple it seemed simplest just to get stuck in.

And for the most part it was simple. Running Unbuntu makes installing software and tools a breeze and after a few hours of intermittent changes everything was up and running from the old server.

A couple of things that caught me out

. I did not check my own passwords so I wasted time trying to track down a problem with wordpress talking to the back end database when in-fact I had used the wrong password (one digit out). I think this is a case where unpronounceable passwords would have made me use copy and paste and saved some time.

. It seems fairly obvious in retrospect but the web user (www-data) needs to have write access to the wordpress folder to do updates - including configuration changes and the fantastic auto-update feature for wordpress and its plugins.

Once the basic server was set up and serving content I ran some manual tests by adding the domains to my hosts file to check that everything was working fine before updating the DNS to point to the new server.

For the other parts the build server is up and running for some of my projects - I will get around to the others the next time I do some work on them. The build publication is a little clunky at the moment and could so with a dynamic web site to handle things more elegantly.

I keep the source to my web sites in subversion so I also set up some publication build tasks to take care of updates. So I can now make changes locally on my laptop and then commit those changes to the repository and the build server will take care of the rest.

All in all quite a pleasurable experience - things just worked as expected. Now the difficult bit - making sure that I have everything off the old server before shutting it down and deleting the image.

Server rebuild – part 1

2010-02-18T00:00:00-07:00

I have been running my own server for some time now but my requirements have changed and the current underlying VM architecture does not conveniently support what I want to do.

The current linux server is running under OpenVZ which is not that conducive to running Java Applications. So after spending quite a few hours struggling with that thorny problem I have decided to stay with my current hosting provider (VPSlink) but switch to Xen.

The changes in what I need to use the server for are around open source. The current server is fine for serving up my homepage and a few wordpress blogs but I would like to run a Continuous Integration server that independently builds open-source projects that I am involved with.

After a 10 minute web search I concluded that hardware requirements for CI servers don’t appear to be one of the first things you need to know - so will have to do some trial and error.

Preparation Tasks:

Decide on a reasonable starting plan. Memory seems to be the key price influencer here. The current server is running 256MB and big hungry java applications like GB. The cost of upgrading is more money and a quick reboot so decided to start with 512MB and take it from there.
Install Ruby, and run a hello world application
Install Java, and run a hello world application
Validate configuration by installing Ruby and Java. My open-source projects tend to be either Ruby or Java so need to make sure that I can deploy them for testing.
Install Git - some of my projects are hosted on github these days and I have had problems getting validation working from servers without a tty port.
Validate resource assumptions by installing a CI server.

Results:

Second decision point: what to call the server. Not that it matters much since I am probably the only one who will see this information but I still like to choose a nice name.

In 20 minutes I had ruby, java, git, and hudson installed. Completely painless.

I ran a quick bit of ruby from the command line, used hudson to test that the Java runtime was installed properly and checked out a project from github to make sure that git was behaving itself.

Now for the more difficult bit. Transferring all the other data.

Look after your tools

2010-02-18T00:00:00-07:00

Look after your tools and they will look after you!

I was very lucky to grow up in the country with a father who enjoyed teaching me practical skills like carpentry, motor mechanics and just about every other skill needed to keep the house, car and garden up to scratch. One of the things that was drummed into me at this early age was to look after the tools (at that time I had to borrow my father’s tools).

Planes should always be placed on their side to protect the blade. Chisels kept sharp and covered ready for the next time they are needed.

This principle has stayed with me and affects the way I look after and maintain my computer equipment. My laptop is kept clean and free of dust and grit. I have had it for just under 3 years. It has two minor scratches on the case and wear to the keyboard (a sure sign that I use it too much). I cycle the battery about once a month. And generally keep an eye out for performance changes and other signs of problems. I always slide it carefully into its protective sleeve inside the padded bag.

Maybe I am a little too careful but my computer equipment works and I have a rather low failure rate.

Investing in good tools

Lets consider one of the most common tools: a claw hammer. A basic hammer can set you back a few pounds or dollars. This hammer will do what is asked of it driving nails into and pulling nails from all wood and other materials.

With careful care this hammer will last a long time. But the next time you are in a hardware store try out a professional hammer. The difference in feel, balance and performance is quite astounding. Designed for high use the user is cushioned from the repeated impact by careful design and choice of materials.

Oil to avoid rust.

I find that more and more people are taking technology tools for granted and after a short time look old, battered and to be honest to my eyes ‘sad’.

Someone spent a lot of time designing the tools we use in IT today. Not caring for them shows that we have no respect for the efforts they have gone to. We may not like the design but there is a definite feeling of disrespect to mistreat the result of their design.

The more robust the item the more abuse it seems to get

Just because a hard disk can sustain a fall of two meters does not mean it will not break if you throw it around all the time!

Heed the old adage: Looking after your tools and they will look after you.

Don't use application Zoom

2010-02-18T00:00:00-07:00

A plaintive call to not over use the lovely zoom features in Excel and Word. The zoom parameters are stored in the document so when you receive them it is very unlikely that your zoom setting will match my zoom requirements.

If I change the zoom setting then the document has been changed.

If you store your documents in version control like me then playing about with the zoom settings means a lot more binary style checkins too.

Age discrimination in technology

2010-02-18T00:00:00-07:00

This article post ‘The painful truth about age discrimination in tech’ left me a little confused. Am I still considered a developer or have I given up the mantle and considered a statistic.

Do I cut code? - yes but not as much as I would like and it is not the thing I do every day.

Is there age discrimination? If there is it not that explicit or something I have taken an offence at.

Do I agree that it is difficult to stay technical after 20+ years in the industry? Absolutely! By taking on other roles related to software development I have had the opportunity to have a much greater effect on a project.

Do I miss cutting code from dawn to dusk? Again Absolutely! I now do most of my coding from dusk until dawn.

Is this a problem? And if it is do I or anyone else have an answer?

The rise and fall of waterfall - on you tube

2010-02-09T00:00:00-07:00

If you have been in software development for a while this might make you chuckle :)

The Rise And Fall Of Waterfall

</param></param></param></embed>

Switching back and forth - Intellij IDEA and Eclipse

2010-02-05T00:00:00-07:00

I have to confess that I am an IntelliJ Idea fan. I will confess that the key bindings take a little getting used to - especially on a Mac but I find I am most productive using it.

I do like to keep up with what is happening with the Eclipse project and thought I would pick p the latest Eclipse 3.6M5 (Helios) release. Not really intending to do more than load up a couple of projects and play around for a while.

On one of those projects I had just completed splitting the project into two modules (core and server). During the process I had ended up with some duplicate classes. I thought that I had cleaned all of that up so I was a little surprised to see that Eclipse did not compile my project. Eclipse and IntelliJ handle projects rather differently and for this particular project Eclipse thought that the two source files should be compiled together - not really a problem because the build script handled the partitioning into separate core and server jars. But because Eclipse worked in this way it helped me find two duplicate files that should only have existed in the core jar. There were a number of other small things that Eclipse thought were important and IntelliJ didn’t (probably default setting difference) but switching IDEs helped me find the major duplication errors and made me think about the other potential issues.

So it perhaps pays to switch your IDE from time to time. Even if it is only to make sure that your project is available to all.

XP and the cost of change

2010-02-01T00:00:00-07:00

I have found myself drawing the classic cost of change graph a few times recently so thought I would blog about it. The graph was popular a few year ago in explaining the differences between and eXtream Programming (XP) team cost and a waterfall team cost.

The important thing to remember in interpreting the graph is that the lifetime of a software system is typically far longer than the initial development. Most systems spend much longer in ‘maintenance’ than they ever did in ‘development’ so anything that can be done to make maintenance (ongoing development) more effective has to make sense.

The XP premies is that by augmenting the codebase with automated tests changes are easier to make later on. Developers can regression test the code to make sure that they have not broken anything and that their new tests also pass. This requires the same if not more discipline than the original development.

It is also important that refactoring be part of this ‘maintenance development’ without it the system will degrade, making it more difficult to effect changes. A couple of years ago I was fortunate to work with a team developing a web application. Nothing special perhaps but the combination of background and skills for this project turned out to rather special. After the first release which was made under extreme time pressures new features were needed and added.

The key difference was the aggressive and determined pursuit of simple elegant code present in all of the team. At around the second release it was felt that ‘the code just wrote itself’. The code seemed to support the addition of new features and productivity improved dramatically.

Some time later it was decided that ongoing development be moved offshore.

As a lead developer this can be a troubling time but as part of the handover myself and another senior developer were asked to take the project offshore to ensure a smooth transition.

And this is where things got interesting. The team were delighted with the code and were keen to get started. Some scheduling conflicts meant that they could do a development iteration (1 week) and work on some of the backlog before envisioning the next major release.

That week was great fun with many whiteboard discussions about the principles underlying the architecture and the design.

After the development iteration the team felt confident in moving forward. That confidence proved to be true. The rate of changed increased and the client was delighted with the results.

If you are reading this and were part of the onshore or offshore team then thank you for such a fun time and wonderful code.

P.S. And yes I remember how hard the project was too.

Screen casting

2010-01-31T00:00:00-07:00

There have been a number of threads recently talking about screen/podcasting and the software required to do a decent job so I thought I would blog about the setup I am using.

After trying a few applications I eventually settled on IShowU HD. It handles screen capture well and supports a wide range of formats.

I tried a couple of microphones (including the built in mic on my MBP) before settling on the Samson C10U. The mic works well but I have not found it to be that sensitive so have to position it between the laptop and me which sometimes makes it difficult to type. Perhaps I will have to invest in a boom to bring the mike over the top - although that may make it more difficult to see the screen.

I also use headphones so I can hear the audio as I speak but there is a slight delay as the sound is processed which can be a little distracting at times.

I find that it is difficult to maintain momentum for a long time so I capture a scene in iShowU and then stitch them together in iMovie before burning a completed screen-cast.

Once complete I have tried using HeyWatch! to convert the MP4 into flash but there is quite a degradation in quality during the conversion. This would probably not be a problem for normal video but the text of the screen-cast becomes almost unreadable. This may be because of the settings I have used so far so I will probably give that another go when I get around to recording the next episode.

Wordpress theme update failure

2010-01-30T00:00:00-07:00

If like me you installed WordPress and added some themes before understanding that WordPress has a great way of adding themes through the UI then make sure that you have changed the ownership of the folders containing those themes.

Otherwise the autoupdate fails and if you update the current theme end up with a maintenance screen. Getting rid of the maintenance screen is pretty easy (delete the .maintenance file) in the blog folder but it still takes a few minutes to solve the problem.

Web application testing - episode 1

2010-01-20T00:00:00-07:00

This is my first screencast so all and any comments welcome!!

This screencast is about adding an acceptance test to validate the title element in the returned HTML from the application.

[flowplayer src=http://www.grahambrooks.com/video/testing-web-apps-episode-1.mp4 width=550 height=413 ]

The companion code for this episode can be downloaded from http://www.github.com/wookie870/bvira

Subway reading

2009-12-13T00:00:00-07:00

Over the last month I have been reading Prmodoro Technique Illustrated a fascinating book detailing a technique to apply real focus to problem solving and work in general. One problem I have with almost any kind of reading is finding the time to actually read. But this book is nicely split up into bite-sized chapters which can be read on short journeys. I have taken to downloading books to Stanza an iPhone application which accepts epub format documents. The transfer is a little fiddly but once a URL to the book download is entered Stanza takes care of the download.

This reading method is working well for me at the moment. I just can’t travel with a large collection of books weighing me down. But I always have my phone with me for a little ad-hoc reading.

Cross post: A Brief And Incomplete History Of Build Pipelines

2009-12-13T00:00:00-07:00

Sam Newman has just published a Brief And Incomplete History Of Build Pipelines which makes interesting reading.

It will be interesting to see where the concepts take us in the next few years.

Testing abstrations

2009-11-18T00:00:00-07:00

These ideas were presented as at ThoughtWorks Agile Southeast conference in Atlanta. The idea for these diagrams came about after the first time I spoke about web test driven development at Agile East in Philadelphia and New York. I am striving to show how introducing a level of abstraction affects development costs.

Web applications should testing at various levels, from unit tests through acceptance, workflow and performance (to name just a few categories). For the purposes of this article Acceptance and Workflow tests are defined as follows:

Acceptance Tests - validate the content of the returned document from the application (typically) HTML. Using the test form:

Given system in a particular state

When a request for a particular resource is made

Then the returned document contains these

Workflow Tests simulate how a user is expected to interact with the system to perform the anticipated browser actions.

Both Acceptance and Workflow tests could be executed through the browser but the Acceptance tests could be run against the application directly saving the expense of passing data through the browser. Having the Acceptance tests make direct calls to the application being significantly faster means that the feedback cycle is much shorter. Typically there are many more Acceptance style tests than Workflow tests so keeping the Acceptance tests fast is a significant benefit.

Having chosen to take advantage of this performance increase it is likely that the code to exercise the application will be different and use different technologies.

However, there are significant advantages in maintaining a common abstraction layer available to both test types.

The chart below expresses the cost of changing an application correlated against the number of tests supporting development. The red line represents the cost of change with a minimal or no abstraction. The blue shows change costs if an abstraction layer is introduced early in the project.

These charts are not supported by project data but illustrate experiences on projects. The next chart illustrates that the red (no abstraction) curve could present a different profile.

What I am pretty sure I have seen on a project is a team stall unable to proceed because the developing new tests to support new functionality has hit the exponential part of the curve. Having hit this wall there are several options but adding a common abstraction layer to support both Acceptance and Workflow style tests is effective at unblocking the team. This is illustrated by the chart below.

You might have noticed the small brown dot over the intersections of the two lines. If it were possible to identify this point either based on the axes on these charts or some other dimensions then a team could start with a simple ad-hoc approach and then adopt the time saving abstractions as they approach the point where the curves intersect. At the moment not knowing how soon a team would hit this point it would seem prudent to take the cost earlier than later when the cost of changing all the tests is significant.

How the Pendulum Swings

2009-11-05T00:00:00-07:00

A couple of decades ago I spent quite a lot of time working on my car. To be honest it was near the end of its life when I bought it at auction but I had only saved/borrowed enough for a car of that age and at the time I thought I knew a lot more about maintenance than I really did but what I did not know I now had my own car to learn from. As you might expect it was not too long before disaster struck and driving home one day the engine died and resisted my attempts to revive it. Time to to get stuck into car maintenance 101.

Once I had the engine in pieces I discovered that every piston ring had broken causing a complete lack of compression. I had however been very lucky and the pieces had caused minimal scoring of the combustion chamber. So all I really needed was a new set of rings. Since I had taken the engine apart I thought it would be a prime opportunity to update the main bearings and extend the life of the engine. With that in mind I made out a list and headed off to a specialist to obtain the parts.

If you have never been into such a place it is very different from the comparatively glitzy motor factors of today. Anyway after I listed out everything I thought I needed the guy behind the counter disappeared into the back room collecting all the parts I had asked for. Some time later he came back with a collection of grubby boxes and started entering the part numbers into a terminal. In those days most stock managing systems were centralised using remote terminals for access of serial lines. The screens 80x25 character displays.

What sticks in my memory was the speed with which the part numbers were entered. The guy’s fingers fair flew over the keys and he hardly looked at the screen. He had become so used to entering the stock numbers that he did not need to reference the screen while entering the details.

At the end of the process he hit a function key and without stopping started to put the parts in a bag.

This sort of interface was common. Graphical user interface had not made it out of the research labs into the mainstream. Operators were required to learn the systems rather than the systems be tailored to them.

When graphical user interfaces became more popular we designed interfaces that were easier to learn and use. Mice became the ubiquitous way of selecting items from list and moving the input focus from one field to the next.

My recent experiences at the motor factors are quite different from the earlier stories but one theme that I have seen time and again is that the data entry tasks take a lot longer than I remember. The introduction of the mouse has reduced the barrier to entry for most systems and allowed people to become functional at interacting with a system through this common interface but the cost of switching back and forth between the mouse and keyboard has an ongoing cost for every interaction.

The guys over at JetBrains have just launched a very interesting but tracking system. The fact that they have developed a bug tracker is not the interesting point for me. The interesting bit that caused me to remember the guy entering all the serial numbers is the interface that they have developed.

On the face of it the interface is a sophisticated web application but they have instilled a sophisticated text based user interface inside the graphical one. They have put together a domain specific language (DSL) that gives the user extensive control over the data model behind the interface.

People using issue management typically spend a lot of time interacting with the system and are willing to invest time in learning how to be more effective with the tool. Introducing a text based UI seems like a very valid approach in this case. The user is given a now familiar graphical user interface to start working with the tool and then a more expressive and powerful interface as they become more familiar and want to do more.

WebTDD at Agile East

2009-10-31T00:00:00-06:00

Just a quick note to say thank you to everyone who attended my talk on Test Driven Web Development at Agile East this week.

Prag Pro Wri Mo

2009-10-31T00:00:00-06:00

I am going to take part in Prag Pro Wri Mo to find out if I have the inclination and capacity to write or if I am one of those people who would have liked to have written a book.

This initiative seems like a really fun and challenging idea and I am looking forward to getting started.

Agile East WebTDD

2009-10-26T00:00:00-06:00

On Thursday this week I will presenting a talk entitled ‘WebTDD’ at the ThoughtWorks Agile East conference in Conshohocken PA. It is the first outing for this talk and I have been working on the supporting software for some time.

I am really looking forward to this first speaking opportunity in the US this year.

If you are reading this and attending I look forward to meeting you there.

Wednesday evening in Seattle

2009-10-08T00:00:00-06:00

This photo was taken from the hotel bar on the top floor of the Hyatt Regency. We were lucky to arrive just as the sun was setting. If you look closely at the horizon you can just make out the mountain profile. Some are already topped with snow.

Just for a moment looking at the sunset makes all the travel worthwhile.

Its Tuesday in Seattle

2009-10-07T00:00:00-06:00

Not that I have seen anything of the city but I now find myself on the west cost after arriving on the east cost just over a week ago.

The Hyatt Regency</a> here is a pretty nice hotel though and I look forward to a nice nights sleep.

The big apple

2009-10-02T00:00:00-06:00

October finds me in New York and for the last couple of days I have been walking to and from work (about 10 blocks).
One thing that has struck me is that finding your way around the city is a piece of cake in comparison to say London. In NY the grid system not only makes is easy to find a place but also to know how far away you are.

Another excellent talk on Ted

2009-10-02T00:00:00-06:00

</param></param></param> </embed>

Adium 1.4ß

2009-08-24T00:00:00-06:00

I have just discovered that Adium - my favourite IM client for the mac has a new ß version available and now supports IRC so one less application to run every day.

Ultimate agility?

2009-07-19T00:00:00-06:00

The truly amazing Dan Osman

This has got to be one of the ultimate demonstrations of agility, confidence and poise.... </param> </param> </param> </embed>

Shenandoah '06

2009-07-19T00:00:00-06:00

I found this whilst going through some old(ish) photos taken in Shenandoah national park in 2006.

It brought a lot of memories back about that trip - which was just a weekend away but a little adventurous - the first time I had seen a Bear in the wild and probably the hottest, dampest walk I have every done.

Relaxation

2009-07-19T00:00:00-06:00

Another photo from the trip to Shenandoah. Yours truly taking time out…

I remember there was a cool breeze drifting up the river - it was a warm day

Java Properties handling - 1

2009-07-18T00:00:00-06:00

I have been on a number of Java projects recently and one of the things I end up brewing for each of these projects is some enhanced management of runtime properties.

Having said there are problems with using .properties files they are well understood and supported by all the major Java tools and IDEs. There would have to be a very strong reason to use another format (and no I don’t mean XML).

The basic problem is that the Java API supports property loading but no semantic validation of the supplied property values. In addition it is quite common to want to vary the properties from one environment to another (otherwise the need for properties would not exist). When trying to diagnose a problem is it very useful to know where the value of a property came from. In fact to start with I would like to know that the value of the runtime properties is in the first place.

During the course of the last couple of Java web projects I have come up with the following:

The runtime properties are available through a simple web page - typically secured behind a masked internal URL.</li>
Sensitive property values such as password are obscured.</li>
Along with the property name and value some notion (typically a filename) of where the property value came from.</li>
An assessment of the property value.</li>

Requirements

I came up with the following basic requirements:

I want to declare the property requirements within the application source.</li>
I want to be able to specify if the property is required</li>
I want to know at runtime where a property came from</li>
I want to fail fast if a required property is missing or malformed.</li>

public class TestRuntimePropeties {
    @RuntimeProperty(name = "some.property.name", defaultValue = "foo.bar")
    public String somePropertyName;

	@RuntimeProperty(name = "required.property.name", required = true)
    public String requiredProperty;

	@RuntimeProperty(name = "missing.property.name")
	public String missingProperty = "missing";

    public String nonPropertyField = "noproperty";
}