codeartisan

NoSQL East 2009 Redux

2009-10-31T21:06:00.005-04:00

I just attended NoSQL East down it Atlanta over the last two days. This was a fantastic conference, well-organized, with not only great content and speakers, but also a very well-educated audience. There was a telling moment in the first non-keynote talk where the speaker asked the audience "How many people have read the Dynamo paper?" and easily 95% of the audience put their hands up.

I'd divide the major focus areas into the following groups:

key-value stores (Riak, Voldemort, Cassandra)

massively parallel data mining (Pig, Cascading)

column-oriented datastores (Cassandra, HBase)

document collection databases (CouchDB, MongoDB)

stuff I couldn't make up my mind about (Neo4J, Redis, Tin)

As one of the speakers put it, which one you need depends on the shape of your data; these are all isomorphic in one way or another to each other, just as Turing complete languages are all essentially complete. But you still want to choose the right tool for the job. One nice touch was that a number of the systems were introduced not by one of the project committers, but rather by people who were using them to get stuff done. It was a very practical focus, much appreciated.

There were a couple of interesting tidbits from the conference, including a rollicking, confusing talk by John "My first network address was '12'" Day about network architecture and how TCP/IP had really gotten it all wrong. This guy was about 3 planes of existence above what I could follow at 9:15am--I had a distinct feeling he knew exactly what he was talking about, and was probably right, but there was no way he was making sense even to a very technical audience. Then there was the poor guy from Microsoft Research who had done some interesting distributed IDE work...for the .NET framework. In a sea of Mac and Linux laptops across the audience, he had a bunch of people who really couldn't make practical use of what he'd done.

Key-Value Stores

I've got to give a pretty big nod to Riak here for a straight-up key-value store; Justin Sheehy basically gave a talk about decentralized, scalable, lights-out system design that might well have been a good keynote, and it was clear that Riak was designed with high priority on those aspects. Major advantages over Voldemort in my mind are: ability to add/subtract nodes from running clusters, tunable (per-bucket, per-request!) quorum parameters (N,R,W), plus support for pluggable consistent hashing modules, partition distribution modules, and conflict resolution modules. For most of my use cases, with N=3, I'd probably do R=1,W=3 to optimize for read latency, as most of our writes would either be a user clicking "save" or would be the result of an asynchronous background process. On the other hand, if I wanted to build something like SQS on top of this (which I conjecture is possible), I'd probably do W=1,R=3 to optimize for the "post a message" latency.

While I was there, I had a great talk with them where we went over how there were going to expose the conflict resolution out up to the client via their HTTP/JSON interface. There's no out-of-the-box support for multiple datacenter awareness, although it seemed possible to add it via the pluggable partition distribution module, although their inter-node communication leverages Erlang's native networking, meaning some kind of VPN/IPSec tunnel would be the main way to make this work across sites. I'm pretty sure no matter what you would want to use in this space you'll probably have to end up using a VPN over WAN to give the appearance of a seamless LAN cluster (although the Cassandra guys piped up in the conference IRC channel that Cassandra has multi-site support already).

I can't tell whether being written in Erlang is a plus or a minus. On the one hand, it won't plug in well to our JMX-based monitoring frameworks, and I'm pretty sure very few folks within our organization have ever built or deployed an Erlang system. On the other hand, Erlang is the perfect choice for highly concurrent, fault-tolerant programming, and would probably be right up the alley of several of our programmers (I'm proud to say that while we are primarily a Java shop, we have a lot of other language proficiencies in-house spanning Ruby, Python, Clojure, and Scheme. Based on my brief escapade with it last night, I'm confident that (1) we have a number of folks who could pick it up easily and (2) who would jump at the chance). This is probably a wash.

Very interesting in my mind is that some of the key-value stores have brought over some features from other design spaces; Riak allows some basic link structure in their values, including a version of targeted MapReduce that can follow links, which starts to make it feel like a graph-oriented database like Neo4j. Similarly, Cassandra has support for column-oriented indices like HBase does. It's clear there's probably a project out there to scratch your particular itch.

Massive-scale data mining

We saw a couple of talks from folks who were using Hadoop for analyzing very large data sets. But furthermore, they were using frameworks like Pig and Cascading on top of Hadoop to do a lot of ad-hoc queries. Twitter in fact uses Pig to do all of their interesting web analytics, and they've taken all their analytics BAs who are used to doing ad-hoc SQL queries and trained them up with little problem in Pig. This is probably somewhere on our horizon, although there are larger cats to skin at the moment.

Column-oriented datastores

Cassandra and HBase were the major players here. Digg is moving all of their backend storage over to Cassandra after some successful pilots, and we saw a great talk by Mark Gunnels about how he is using HBase because it makes it "simple to get sh*t done". Apparently recent releases of HBase have made strides in fault tolerance (there is now an election algorithm for some of the special name nodes) and latency (apparently performance optimization was a major focus of the 0.20 release). There's an interesting article that describes some wide-area replication schemes that are available with HBase that sound intriguing (although once you are beyond two datacenters, I am convinced that you are better off with the VPN LAN solution if you want to have any hope of achieving eventual consistency).

Document-oriented databases

While the column-oriented datastores are good at organizing semi-structured data, the document-oriented guys are really all about organizing largely unstructured documents, and focus on doing some wild ad-hoc MapReduce queries. I still need to be convinced about the scaling, replication, and geographic distribution capabilities in this space, so it may be a while before we dip our toes in here.

Other

There was a very interesting talk about Tin ("the database so tiny they had to shorten its name"). This basically seemed to be a very clever approach for delivering stock data that leveraged a basic filesystem with range queries and rewriting rules via Sinatra. Turns out web servers serving static files goes pretty fast, if that's a suitable representation for your stuff (seems like his primary use case is for delivering read-only out to clients, where the data gets updated asynchronously in the background by the system). We actually have some datasets that are like that, so this is intriguing!

There was a talk from Neo4j, a graph-oriented database, which I just can't wrap my head around yet. I think I need to read up on some of the background research papers in this area. Certainly, the notion of a datastore based around the notion of link relations would likely be easy to expose as a REST HTTP interface, which is attractive. Our particular domain model can actually be nicely normalized, however (we are currently running off a traditional RDBMS after all), so I'm not sure we need the full semantic flexibility this offers.

There was also a talk from Redis; this is primarily an in-memory storage system with blazing-fast speeds, and the ability to write out to disk is really an afterthought. During the talk he showed a screen snapshot of a "top" running on a cloud-hosted virtual node with 60GB of memory. I cannot make up my mind whether this is ultimately just a badass memcached, or if he's on the cusp of something: if you have enough memory to hold your whole dataset, and you are sufficiently fault-tolerant, why bother writing to disk at all? Especially if you can easily get a periodic snapshot out to disk in the background that can be properly backed up for disaster recovery.

Conclusion

As folks pointed out, "NoSQL" might better be written "NOSQL" to mean "Not Only SQL". I didn't sense a lot of MySQL hatred in here; quite the contrary, many people were very complementary that there were certain things it does *really* well, and that the MySQL hammer was something that was staying firmly in their toolboxes. However, it is clear that there is a maturing community of very practically-minded folks that are looking for a new set of tools to drive their particular screws. Although to be sure (and this is something that I think some of the conference tweets corroborated), this also implies we all have a screw loose or two....

Object Calls == Message Passing in Erlang

2009-10-30T11:53:00.004-04:00

I started playing around with Erlang last night as a result of learning about Basho's key-value store Riak at NoSQL East yesterday (more specifically, it was due to Justin Sheehy's talk, and two realizations: (1) this guys *gets* a lot of important *operational* design choices in this space, and (2) he decided to build his system in Erlang).

So I decided to read the online Erlang "course" and started working on the exercises. One of them was:

Write a function which starts N processes in a ring, and sends a message M times around all the processes in the ring. After the messages have been sent the processes should terminate gracefully.

And so, summoning vaguely-remembered lectures by Bob Harper in "Fundamentals of Computer Science II" at CMU on doing object-oriented programming in Scheme, and remembering that the original Smalltalk OOP guys always said "send an object a message" rather than "invoke a method on an object", I set to work. [Editor's note: please feel free to post comments showing me better ways, I have known Erlang for all of about 12 hours at this point!]

Let's get the declarations out of the way. I need to define the entry point function which creates the ring of N nodes and sends the message around it M times, and I know I'm going to need a function representing a node in the ring, since I'm going to have to spawn processes for them.


-module(ring).
-export([ring_msg/3, ring_node/1]).

Ok, what job does a node in the ring have? Well, most of the time, when it receives a message, it just needs to pass it on to the next guy. So my node process is going to need to know about its next neighbor. Now in Erlang, what I would normally think of as an object can be modelled as a recursive function that passes its current state back into itself as an argument, and processes "method calls" by receiving messages. Interestingly, not all method calls actually have to send something back to the caller!


ring_node(Next) ->
  receive
    { pass, M, Msg } ->
      % Note that we got this message. 
      io:format("Node ~w~n",[Msg]),
      % Pass the message on around the ring.
      Next ! { pass, M, Msg },
      % If the count was down to zero, I can
      % exit, otherwise, I loop and wait for
      % the next incoming message.
      if M == 0 -> ok;
         true -> ring_node(Next)
      end
  end.

Ok, seems pretty straightforward. But if I had a ring of these set up, a message would just keep running around the ring. At least one node needs to be special, so that it can decrement the count M as the message comes through. It's pretty similar to the ring_node above, but is a little different.


init_node(Next) -> receive
    % message has been all the way around
    % the last time, so I can quit
    { pass, 0, _ } -> ok;
    % otherwise, log the message and pass
    % it on, decrementing the count
    { pass, M, Msg } ->
      io:format("Node ~w~n",[Msg]),
      Next ! { pass, M-1, Msg },
      init_node(Next)
  end.

Now an interesting thing here is that the init_node and the ring_node can both handle the "pass" message, and that when they send the message on, they don't actually care what their "Next" process is. It's like both of these "objects" implement the following interface:


public interface MessagePasser {
  void pass(int count, Object msg);
}

Ok, so now if we can create a ring with 1 init_node and (N-1) ring_nodes, we're all set if we inject the initial Msg into the init_node. So let's think about constructing a ring of nodes; if we have a node handy, we can pass that in as the initial argument (think "constructor") to a ring_node process to use as its Next node, then we just count down:


ring(Last, 0) -> Last;
ring(Last, N) -> 
  RN = spawn(ring, ring_node, [Last]),
  ring(RN, N-1).

Hmm, that's close, but that's a linked-list of nodes, not a ring. But we can't pass a node in as a constructor argument to the first node we create, because we don't have any yet! So it seems like we'll need to construct a linked-list of nodes, and then "close the loop" by stitching the front and the back together. Our init_node is already a special node, so maybe we can extend it this way:


init_node(Next) -> receive
    % acknowledge the request, update state
    { setNext, N, From } -> From ! ok, init_node(N);
...

In other words, the init_node can get a special message telling it to "update" its Next state. In some sense, we've just done this:


public interface InitNode extends MessagePasser {
  void setNext(MessagePasser N);
}

We want to acknowledge the request so our ring construction knows when that message has been processed -- we don't want to hand the ring back until it's all stitched together, and we can't guarantee ordering of message delivery unless we specifically wait for a response. So here's the full ring construction:


ring(N) when is_integer(N) ->
  % just pass in a placeholder for Next
  RN0 = spawn(ring, init_node, [nil]),
  ring(RN0, RN0, N-1);
% finished stitching, can return our
% init node to the caller
ring(Init) -> receive ok -> Init end.
ring(Init, Last, 0) -> Init ! { setNext, Last, self()}, ring(Init);
ring(Init, Last, N) ->
  RN = spawn(ring, ring_node, [Last]),
  ring(Init, RN, N-1).

Finally, the thing we're trying to do (including optimizing the degenerate cases):


ring_msg(0, _, _) -> ok;
ring_msg(_, 0, _) -> ok;
ring_msg(N, M, Msg) ->
  Init = ring(N), Init ! { pass, M, Msg }, ok.

Actually runs, too! It's pretty neat to see polymorphism via being able to accept the same message, and I've always loved the pattern matching in ML (both SML and OCaml variants!). Some pretty serious systems programs are getting written in this language; it's clear that the process spawning methodology lends itself well to a SEDA-style approach which is great for graceful degradation of service, and the fully-functional style (no mutation) means that you have no locks, no shared state, and hence safe concurrency (as long as you can model what you're doing properly).

Agile Architecture Anti-Patterns

2009-08-26T21:54:00.004-04:00

I have been the solution architect for an enterprise-spanning project which is now getting geared up for its next phase. This project touches more than 10 different development teams across at least three company divisions and involves at least one external ISV. So, you know, a pretty small project.

As I reflect on the process of hashing out all the technical details, I realized I got stretched pretty thin. I really didn't have time to write much down in any formal fashion this time around, due to the time pressure on the project and the number of other things I was also trying to juggle. I have instead been spending a lot of time drawing stuff on whiteboards for various technical audiences. The things I did have time to write up were only the most general concepts and patterns - i.e. the really high-level architecture.

At least part of the reason I didn't have time to write anything else down was that I was spending all my time in technical design discussions with all the various groups, hashing out nitty-gritty integration details and then bringing full integration designs back to the development groups to run with. The last time through, this worked out well, because the developers in my home organization are used to agile development and being responsible for their own technical design, so they were able to take my whiteboard sketches, continue to consult with me, and produce solid, working code. It also worked out ok because the last phase was actually the first phase of this project, and I had enough lead time to work out all the details ahead of time.

This time around, the time pressure is pretty high, plus the work-ahead time I would have had in a traditional "up-front" architecture process was actually consumed by helping to get the first phase out the door. So I find I actually don't have enough time to do the same detailed design that I did the first time around before the development teams are scheduled to start. This would seem to be a major conundrum.

Antipattern 1: waterfall, up-front architecture doesn't pipeline well if the architecture phase extends concurrently all the way through the development phase.

Corollary: trying to waterfall development and QA doesn't pipeline well either for exactly the same reason (because developers have to keep working their code to fix bugs).

The other anti-pattern that I was running into was with development teams in some of the other divisions; I would work with the enterprise architects to hash out a pretty detailed design, but when we brought that to the development teams, there was a ton of convincing, resistance, and redesign that ended up happening. Now, I'm not sure exactly which anti-pattern we were running into, but it was one of the following:

Antipattern 2: up-front architecture doesn't work if the architects aren't familiar enough with the details of the systems/teams they are designing for, because the designs won't "fit right".

Antipattern 3: up-front architecture doesn't work if the architects haven't been able to build up sufficient technical cred to sell their designs to the developers.

For sure both of those applied to me with respect to the developers and technical leads of the groups in the other divisions.

I've been thinking a lot recently about the right way to approach this, and really enjoyed this essay on Agile Enterprise Architecture by Scott W. Adler. He proposes a much more lightweight, hands-on approach to architecture which can best be described as guiding the agile development teams to develop the right architecture themselves.

Fred Brooks writes in The Mythical Man-Month that a systems' architecture must proceed from one or a small number of minds in order to be coherent (I can't find a copy right now, so that's paraphrased). At the same time, agile development techniques like Extreme Programming (XP) or Scrum suggest that the development teams ought to be responsible for evolving their architecture organically. I'm starting to think that both of these are true, and that they are not mutually exclusive, and it is related to another realization I've had (but that I have a hard time remembering):

A solution architect is responsible for seeing that the end-to-end solution hangs together technically. It is not necessary for the architect to produce the whole end-to-end design himself to achieve this.

Going forward, I think I'm going to take the following approach: figure out broadly which development teams are going to need to interact and what their coarse responsibilities are, then immediately get them involved on working out the solution. In true agile fashion, let them work out the details, and just play Product Owner to set high-level (technical) requirements, being on-hand to participate in the design discussions. I think this leverages best the available technical design talent in the development teams, while making sure that someone is tracking it all and fitting it into a coherent mental model to make Mr. Brooks happy.

RSA Public Key Cryptography in Java

2009-05-15T16:18:00.006-04:00

Public key cryptography is a well-known concept, but for some reason the JCE (Java Cryptography Extensions) documentation doesn't at all make it clear how to interoperate with common public key formats such as those produced by openssl. If you try to do a search on the web for how to make RSA public key cryptography work in Java, you quickly find a lot of people asking questions and not a lot of people answering them. In this post, I'm going to try to lay out very clearly how I got this working.

Just to set expectations, this is not a tutorial about how to use the cryptography APIs themselves in javax.crypto (look at the JCE tutorials from Sun for this); nor is this a primer about how public key cryptography works. This article is really about how to manage the keys with off-the-shelf utilities available to your friendly, neighborhood sysadmin and still make use of them from Java programs. Really, this boils down to "how do I get these darn keys loaded into a Java program where they can be used?" This is the article I wish I had when I started trying to muck around with this stuff....

Managing the keys

Openssl. This is the de-facto tool sysadmins use for managing public/private keys, X.509 certificates, etc. This is what we want to create/manage our keys with, so that they can be stored in formats that are common across most Un*x systems and utilities (like, say, C programs using the openssl library...). Java has this notion of its own keystore, and Sun will give you the keytool command with Java, but that doesn't do you much good outside of Java world.

Creating the keypair. We are going to create a keypair, saving it in openssl's preferred PEM format. PEM formats are ASCII and hence easy to email around as needed. However, we will need to save the keys in the binary DER format so Java can read them. Without further ado, here is the magical incantation for creating the keys we'll use:


# generate a 2048-bit RSA private key
$ openssl genrsa -out private_key.pem 2048

# convert private Key to PKCS#8 format (so Java can read it)
$ openssl pkcs8 -topk8 -inform PEM -outform DER -in private_key.pem \
    -out private_key.der -nocrypt

# output public key portion in DER format (so Java can read it)
$ openssl rsa -in private_key.pem -pubout -outform DER -out public_key.der

You keep private_key.pem around for reference, but you hand the DER versions to your Java programs.

Loading the keys into Java

Really, this boils down to knowing what type of KeySpec to use when reading in the keys. To read in the private key:


import java.io.*;
import java.security.*;
import java.security.spec.*;

public class PrivateKeyReader {

  public static PrivateKey get(String filename)
    throws Exception {
    
    File f = new File(filename);
    FileInputStream fis = new FileInputStream(f);
    DataInputStream dis = new DataInputStream(fis);
    byte[] keyBytes = new byte[(int)f.length()];
    dis.readFully(keyBytes);
    dis.close();

    PKCS8EncodedKeySpec spec =
      new PKCS8EncodedKeySpec(keyBytes);
    KeyFactory kf = KeyFactory.getInstance("RSA");
    return kf.generatePrivate(spec);
  }
}

And now, to read in the public key:


import java.io.*;
import java.security.*;
import java.security.spec.*;

public class PublicKeyReader {

  public static PublicKey get(String filename)
    throws Exception {
    
    File f = new File(filename);
    FileInputStream fis = new FileInputStream(f);
    DataInputStream dis = new DataInputStream(fis);
    byte[] keyBytes = new byte[(int)f.length()];
    dis.readFully(keyBytes);
    dis.close();

    X509EncodedKeySpec spec =
      new X509EncodedKeySpec(keyBytes);
    KeyFactory kf = KeyFactory.getInstance("RSA");
    return kf.generatePublic(spec);
  }
}

That's about it. The hard part was figuring out a compatible set of:

openssl DER output options (particularly the PKCS#8 encoding)

which type of KeySpec Java needed to use (strangely enough, the public key needs the "X509" keyspec, even though you would normally handle X.509 certificates with
the openssl x509 command, not the openssl rsa command. Real intuitive.)

From here, signing and verifying work as described in the JCE documentation; the only other thing you need to know is that you can use the "SHA1withRSA" algorithm when you get your java.security.Signature instance for signing/verifying, and that you want the "RSA" algorithm when you get your javax.crypto.Cipher instance for encrypting/decrypting.

Many happy security returns to you.

Downloading your Blogger archives

2009-02-10T18:26:00.005-05:00

A friend was looking for a way to grab an archive of his Blogger posts into a CSV file he could do text mining on (and presumably, for a low-fi backup mechanism).

I wrote this Python script for him, enjoy.


#!/usr/bin/env python
#
# Copyright (C) 2009 by Jon Moore
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import csv
import urllib2
import unicodedata
import xml.etree.ElementTree as etree

blog_feed = 'http://codeartisan.blogspot.com/feeds/posts/default'
output = 'posts.csv'

ATOM_NS = 'http://www.w3.org/2005/Atom'

def norm(s):
    if not s: return None
    return s.encode('ascii','ignore')

def main():
    f = open(output, 'wb')
    csv_wr = csv.writer(f)
    url = blog_feed + '?max-results=100'
    csv_wr.writerow(['id','published','updated','permalink','title','content'])
    while url:
        print "fetching", url
        feed = etree.fromstring(urllib2.urlopen(url).read())
        for entry in feed.findall("{%s}entry" % ATOM_NS):
            id = entry.find("{%s}id" % ATOM_NS).text
            published = entry.find("{%s}published" % ATOM_NS).text
            updated = entry.find("{%s}updated" % ATOM_NS).text
            title = norm(entry.find("{%s}title" % ATOM_NS).text)
            content = norm(entry.find("{%s}content" % ATOM_NS).text)
            perm_url = ''
            for link in entry.findall("{%s}link" % ATOM_NS):
                if (link.get('rel') == 'alternate'
                    and link.get('type') == 'text/html'):
                    perm_url = link.get('href')
                    break
            csv_wr.writerow([id,published,updated,perm_url,title,content])
            print "wrote",id
        url = None
        for link in feed.findall("{%s}link" % ATOM_NS):
            if link.get('rel') == 'next':
                url = link.get('href')
                break
    f.close()

if __name__ == "__main__":
    main()

Business Cases and Cloud Computing

2009-01-24T13:52:00.005-05:00

I just read a very interesting article by Gregory Ness on seekingalpha.com that talks about some of the technology trends behind cloud computing. One key quote:

Automation and control has been both a key driver and a barrier for the adoption of new technology as well as an enterprise’s ability to monetize past investments. Increasingly complex networks are requiring escalating rates of manual intervention. This dynamic will have more impact on IT spending over the next five years than the global recession, because automation is often the best answer to the productivity and expense challenge.

One other cited link is to an IDC study that includes the following graph:

Note that staffing accounts for 60% of the cost of maintaining a server over its lifetime. Cloud infrastructure services like Amazon EC2 would really only save an enterprise data center on hardware setup / software install costs, which are probably, in terms of staffing, a small amount of staff time for a given server. Actually administering the server once it is running is really the bulk of the cost, and that won't go away on EC2 -- you'll still need operations staff to provision/image cloud infrastructure. EC2 makes sense if the economies of scale of AWS are such that they can achieve a lower operational cost for that other 40% than you can, or if there is a business / time-to-market value proposition that makes sense in being able to provision hardware on EC2 more rapidly than we can acquire and install hardware yourself.

Given the huge economy of scale that the large cloud providers have--tens of thousands of servers, it is going to be hard to get your costs for that 40% lower than what they can achieve with their existing infrastructure automation and ability to purchase hardware in bulk, especially for a startup company whose hardware needs are initially modest. Let's guess that there's a 33% markup on cost for EC2, so when you are getting charged $0.10 per CPU hour, it's really only costing them $0.075. Let's assume a 75% experience curve on infrastructure (meaning, once you have doubled the number of servers you have deployed, the last server costs only 75% of what the halfway point was).

By one estimate, Amazon has 30,000 servers. Now let's work backward (1/0.75 = 1.33): at 15,000 servers, their cost was $0.075 * 1.33 = $0.9975. At 7500 servers, their marginal cost was $0.9975 * 1.33 = $0.13. In other words, you'd have to be planning to deploy 15,000 servers in order to have a hope of getting your marginal cost under what they'll charge you retail.

(I think this is actually a conservative estimate: the experience/learning curve for infrastructure deployment is probably steeper than 75% due to existing hierarchical deployment patterns and a product (provisioned servers) that lends itself well to automation. Also, due to the high barrier to entry for cloud computing in terms of number of servers you need to be competitive, they can probably get away with charging an even higher markup).

One corollary of this is that if you are currently running a data center with far fewer servers (i.e. the hardware is a sunk cost), you might actually be better off turning your data center off and leasing from Amazon. Now of course, there are some things (customer credit card data, extremely sensitive business information) that you just wouldn't be willing to host somewhere outside your own data center. But that's probably a very specific set of data--host that stuff and lease the rest in the cloud, particularly if you can get adequate SLAs from your cloud vendor.

So that deals with the 40% of the TCO for a server that isn't staffing. How do you cut costs on the other 60%?

You won't really be able to make a dent in that 60% until you get not just to fully automated infrastructure provisioning, but until you get to fully automated software deployment and provisioning. This is not possible until you get to standardized computing platforms with specific functionality that are scale-on-demand, like Akamai NetStorage, Amazon S3/EBS/SQS/SimpleDB, and Google AppEngine. These are known as "Platform-as-a-Service" (PaaS) offerings.

There's a similar experience curve argument here: you could spend internal development time here to set up some kind of application deployment framework, but you'd essentially have to be willing to build and deploy within orders of magnitude the number of different apps as the Google App Engine team in order to get your costs under what Google will charge you. Unless you are in the business of directly competing with them in the PaaS market, you might as well buy from them and focus your energy on providing your unique business value, not software or hardware infrastructure. [Editor's note: this was something Matt Stevens said to me a while ago, and it wasn't until I went through the mental exercise of writing this article that I actually got it].

Yesterday I implemented (not prototyped) a service in Google App Engine in about 6 hours that would cost around $400 per month (according to their recent pricing announcements) if projected usage were more than double what it is now. I estimate this would require at least 10 database servers just to host the data in a scalable, performant fashion, nevermind the REST data interface (webnodes) sitting in front of it. On Amazon EC2, that'd be $720 per month on your small instances (assuming those were even beefy enough), and per the experience curve argument above, it's probably way more than that in our data center. And that's not counting any of the reliability/load balancing infrastructure.

So my open question is: how, as a software developer, can you justify not building your app in one of these cloud frameworks?

Websites are also RESTFul Web Services

2009-01-12T18:53:00.033-05:00

I have been reading the Richardson and Ruby book RESTful Web Services and recently had an epiphany: if you design a RESTful web site it is also a RESTful web API. In this post I'll show exactly how that works and how you can use this to rapidly build a prototype of a modern web application.

First of all, let's start with a very simple application concept and build it from the ground up. Let's consider a simple application that lets you keep a list of favorite items. The resources we'll want to model are:

a favorite item

a list of a user's favorite items

We'll assign the following URLs:

/favorites/1234 for favorite item with primary key 1234

/favorites is the list of everyone's favorite items

/favorites/-/{owner}joe@example.com (or its URL-encoded equivalent) is the list of items belonging to our friend Joe (this is a URL format inspired by Google's GData protocol).

Now, we'll support the following HTTP methods:

You can GET and DELETE an individual item (we could allow PUT if we wanted to allow editing, but we'll keep the example simple)

You can create a new favorite item with a POST to /favorites.

You can GET the list of a user's favorites.

Ok. Now we want to rapidly prototype this so we know if we have the resources modelled correctly. Fire up your favorite web application framework (Ruby on Rails, Django, Spring, etc.) and map those URLs to controllers. Now most of these frameworks let you fill in implementations for the various HTTP methods. We'll make a minor simplification and allow "overloaded POST" where we allow passing a URL parameter to POST to specify PUT and DELETE (e.g. "_method=DELETE"). We can implement the proper HTTP method but we'll allow you to use POST to do it too; browsers and some Javascript HTTP implementations can only do GET and POST.

Ok, now a funny thing happens: it you render an HTML response for everything, you can start playing with your API in your browser! In particular, when we render the list of items, we will naturally put the text of those items on the page, but we can also throw the following HTML snippet at the top of the page:


<p>Add a new favorite:</p>
<form action="/items" method="post">
  <input type="text" name="itemname"/>
  <input type="submit" value="Add"/>
</form>

We can also add the following form after each item's text:


<!-- use a specific item's URL for the action -->
<form action="/items/1234" method="post"/>
 <input type="hidden" name="_method" value="DELETE"/>
 <input type="submit" value="Delete"/>
</form>

which gives us this ugly beastie:

A Few of My Favorite Things:

Now, one key item is that when you render the result page for adding an item, you can send a 201 (Created) response and say something like "item added", throwing in a link back to the list page. The whole HTML response might be nothing more than:


<html>
  <head></head>
  <body>
  <p>I created your item <a href="/items/2345">here</a>.</p>
  <p>All your items are <a href="/items/-/{owner}joe@example.com">here</a>.</p>
  </body>
</html>

We similarly want to render a confirmation page after a DELETE. This makes for an awkward user experience "add, ok, add, ok,..." but you'll notice that the back and forward buttons on your browser actually work without having to rePOST any exchanges.

[Side note: you could, instead of returning a success page, return a 302 that lands you back on the list page, which maybe gets you closer to what you wanted from a user experience, but this is precisely what will break your browser's back button and make you rePOST.]

Now you also have the interesting property that all the links on your site are safe (without side effects) GETs, and all the buttons are potentially destructive (write operations of one sort or another). I say only "potentially" because you might have a search form with action="get" to do a query, and not all of your POST/PUT/DELETEs will actually change anything.

At any rate, at this point, you have a functionally working website that someone could use, if somewhat awkwardly. Plus, if you have my frontend aesthetic design sensibilities, your users will have the pleasure of suppressing a gag reflex while using your site.

So let's spruce this up a little bit. Now, on the HTML page for the list of favorites, we can apply some Javascript. At a first blush, we can hide the delete buttons until a mouseover, which cleans things up somewhat. But the real magic happens when we attach an AJAX event to the delete buttons. Now the script can actually do the very same POST that the form would have done, and then check the HTTP status code, removing the item text from the DOM on success.

Suddenly, the user never leaves that list page, and we haven't had to change any of the rest of the API -- just the HTML representation of that list page. The AJAX call doesn't care if it gets HTML back (in this case), it just cares about the response code. Now we have the nice AJAXy experience we would expect, but oddly enough you still have a website that will work for people with Javascript disabled.

The last step towards finishing out your API is probably simply to make structured versions of your representations available (e.g. JSON or XML formats like Atom) with an optional parameter like "?format=json". Now all of your client-side functions can call URLs with the appropriate format on them and get well-structured data, and everyone else gets HTML.

Well, I guess that's the second to last step. You probably actually want to apply some graphic design and CSS to your site too...

Simple Backups for your Mac

2008-09-19T22:24:00.013-04:00

You are probably well aware of the need for offsite backups; as a technology professional this is one of the first arrangements I look into for any permanent storage of business information. When I started two years ago for an internal "startup" for a large company, one of the first things we did was set up an SVN repository and then work out an arrangement with an offsite data storage provider. However, the cobbler's children have no shoes: I've never set up a proper backup scheme for my own data at home, and it's about time to take care of business.

Fortunately, now that we've moved to using Macs at home and with the advent of cheap UN*X hosting providers, it's about time I stopped putting this off. The scheme here is pretty simple: get a hosted Linux server from someone like 1and1.com, dreamhost.com, or rackspace.com where the storage is backed up and they take care of security updates for the OS. Then set up a pretty simple combination of the UN*X utilities rsync, ssh, cron, and bash scripts to get secure nightly backups going. Just to make it more fun, I'm going to challenge myself to have this all working in under an hour! I'll keep notes as I'm doing it as to how long it takes, not counting the writeup before or afterward.

I decided to register a domain name with a hosting provider, since it was included. My basic requirements were:

SSH access

rsync installed

enough storage for my data

Be sure to acquire the following information from your hosting provider:

username/password with SSH access (preferably root, if you want to use the server for other purposes, but this is not necessary)

IP address

SSH host key of the server

I ended up registering a new domain with at dreamhost.com at $9.95/month. As it happened, DreamHost was running a promotion with unlimited disk space and bandwidth for the lifetime of my account. Score! I did have to email tech support to get the ssh host key. If you find yourself in a similar position, you can ask for the output of:


$ ssh-keygen -l -f ssh_host_rsa_key.pub
2048 0e:c2:f6:f4:d9:86:9d:4b:c4:3d:77:e7:a4:bb:59:14 ssh_host_rsa_key.pub

Ok, great! Now you have a destination for your offsite storage. Next step is to make sure we can securely log in over the network (we'll use ssh for this). On the Mac you want to back up, open up a Terminal window, and ssh into your server using your username and the server's hostname, as in the following. N.B. Do not finish connecting if the ssh server host key you got from your hosting provider does not match the key you see when you try this!


macbook:~ jonm$ ssh jonm@backup.dreamhost.com
The authenticity of host 'backup.dreamhost.com (67.205.39.2)' can't be established.
RSA key fingerprint is 0e:c2:f6:f4:d9:86:9d:4b:c4:3d:77:e7:a4:bb:59:14.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'backup.dreamhost.com,67.205.39.2' (RSA) to the list of known hosts.
jonm@backup.dreamhost.com's password:
[backup]$

Ok, so far so good. Now we need to make sure we can do it without needing a password; this is where user ssh keys come into play. First, let's create an ssh key to use for backups. We'll want to do this as the root user on our Mac, so that when we run the backup script out of cron, we won't run into permissions problems. You can use the "sudo" command to become root on your Mac:


macbook:~ jonm$ sudo su -

WARNING: Improper use of the sudo command could lead to data loss
or the deletion of important system files. Please double-check your
typing when using sudo. Type "man sudo" for more information.

To proceed, enter your password, or type Ctrl-C to abort.

Password: <enter jonm's password on my mac>
macbook:~ root#

Now we need to create an SSH public/private key pair; this is a similar concept to PGP email encryption/signing; you can read a really interesting description of the chronology behind public key cryptography in the book Crypto by Steven Levy. We'll keep the private key locally on our Mac, and take a copy of the public key and copy it securely up to our backup server; then ssh will use the private key when we connect, allowing the backup server to verify using the public key that we are who we say we are, without having to send a password. Nice.

Specifically, we will want to do the following (still as root):


macbook:~ root# mkdir .ssh
macbook:~ root# chmod 700 .ssh
macbook:~ root# ls -ld .ssh
drwx------  2 root  wheel  68 Sep 19 22:01 .ssh
macbook:~ root# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/var/root/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/root/.ssh/id_dsa.
Your public key has been saved in /var/root/.ssh/id_dsa.pub.
The key fingerprint is:
fd:47:1d:a6:ac:d0:7d:fb:a5:17:cf:e2:8a:93:a5:30 root@jon-moores-macbook.local
macbook:~ root#

Use an empty passphrase (i.e. just hit return when prompted for the passphrase), as this will allow the ssh program to load the key without interaction from you. Also note, however, that anyone who gets root access to your Mac will be able to ssh into your backup server at will. Given that our backup server contains a copy of what this
would-be hacker would be able to see on the actual Mac anyway, I don't really see this being a big risk....

Now, we need to copy the public key over to the backup server:


macbook:~ root# scp .ssh/id_dsa.pub jonm@backup.dreamhost.com:
jonm@backup.dreamhost.com's password:
id_dsa.pub                             100%  619     0.6KB/s   00:00
macbook:~ root#

You'll have to verify the server SSH key one more time, because now you are connecting from root rather than from your normal user account. Now we'll tell the backup host to accept a login from this key pair:


[backup]$ mkdir -p .ssh
[backup]$ chmod 700 .ssh
[backup]$ cat id_dsa.pub > ~/.ssh/authorized_keys
[backup]$ chmod 600 ~/.ssh/authorized_keys
[backup]$ exit

Now, we should be able to log in without a password from our Mac:


macbook:~ root# ssh jonm@backup.dreamhost.com
[backup]$

Sweet. Now we create a directory where our mirrored filesystems will live:


[backup]$ mkdir mac-backups
[backup]$ chmod 700 mac-backups
[backup]$ exit
macbook:~ root#

The utility we'll use to do the mirroring is the rsync utility, which can be invoked to run securely over ssh. This actually makes a nice backup utility for regular use, as the rsync protocol is actually pretty smart about being able to find just the small subsets of data that changed since the last sync; after the first big sync, for most
personal file use, there won't be much work to do every night.

For now, let's set up a test directory on our local Mac.


macbook:~ root# mkdir /tmp/back-me-up
macbook:~ root# echo "data" > /tmp/back-me-up/afile.txt

Now, to make the magic happen, we do this:


macbook:~ root# rsync -avz -e ssh /tmp/back-me-up jonm@backup.dreamhost.com:mac-backups
building file list ... done
back-me-up/
back-me-up/afile.txt

sent 116 bytes  received 40 bytes  62.40 bytes/sec
total size is 5  speedup is 0.03
macbook:~ root#

Now we can keep a window open on our backups host, and we should see everything show up there:


[backup]$ ls -lR mac-backups
mac-backups:
total 4
drwxr-xr-x 2 jonm pg1807352 4096 2008-09-19 19:11 back-me-up/

mac-backups/back-me-up:
total 4
-rw-r--r-- 1 jonm pg1807352 5 2008-09-19 19:11 afile.txt
[backup]$

Just for fun, run the same rsync command above and see that nothing happens if there have been no changes (or rather, just that a very small amount of data gets exchanged to verify no changes).

Let's just make sure changes show up:


macbook:~ root# echo "changed-data" > /tmp/back-me-up/afile.txt
macbook:~ root# rsync -avz -e ssh /tmp/back-me-up jonm@backup.dreamhost.com:mac-backups

(other window)


[backup]$ cat mac-backups/back-me-up/afile.txt
changed-data
[backup]$

Ok, looking good. Next step is to identify all the directories you want to back up; let's keep a list of them in a config file on our mac:


macbook:~ root# mkdir -p /usr/local/etc
macbook:~ root# cat - > /usr/local/etc/backups.conf
/Users/jonm/Documents
/tmp/back-me-up
macbook:~ root#

Note that it is important *not* to have trailing slashes on these directory names, as this changes rsync's behavior slightly in a way that you will probably find annoying (it won't copy the directory name over, just the contents).

Ok, now the next step is to set up a script that can sync each of the directories:


macbook:~ root# mkdir -p /usr/local/bin
macbook:~ root# touch /usr/local/bin/do-backups
macbook:~ root# chmod 700 /usr/local/bin/do-backups
macbook:~ root# cat - > /usr/local/bin/do-backups
#!/bin/sh
for dir in `cat /usr/local/etc/backups.conf`; do
  rsync -avz -e ssh $dir jonm@backups.dreamhost.com:mac-backups
done
macbook:~ root#

Now we run it once by hand to make sure it works:


macbook:~ root# /usr/local/bin/do-backups

Finally, we install this in root's crontab as follows:


macbook:~ root# crontab -l > /tmp/root.cron
macbook:~ root# cat - >> /tmp/root.cron
# take a backup every day at 3am
0 3 * * * /usr/local/bin/do-backups >/dev/null
macbook:~ root# crontab /tmp/root.cron
macbook:~ root# rm /tmp/root.cron

Nice and simple. Now the backups are off and running every night without your intervention.

If you ever need to restore from the backup, you can always reverse the rsync process like this:


macbook:~ root# rsync -avz -e ssh jonm@backup.dreamhost.com:mac-backups/back-me-up /tmp

for each of the directories you have backed up over there.

Enjoy, and sleep well tonight....

P.S. Total elapsed time for the exercise was 2 hours from the time I placed the hosting order to the time the crontab was installed, but I took a one hour break in the middle for dessert and bedtime with the kids. So I'll claim this really did only take one hour of "CPU time" for me.

Measure your improvements

2008-09-02T15:50:00.009-04:00

Metrics are an important part of any development group's toolset. If we want to continually improve our ability to develop software (through a lean engineering kaizen approach, or simply as a learning organization), then we need to have a way to figure out:

what parts of our process need improvement?
when we make a change, did it help or hurt?

This is where process metrics come into play. I'll start with my definition of a metric, which is a numerical measurement of something. If you can count it, it can be a metric. So "number of outstanding bugs" is a metric, but "software quality" is not. The term "quantitative metric" is redundant, and "qualitative metric" is an oxymoron.

There are generally two types of metrics we can capture:

causal metrics: these are metrics that have a direct business impact: for example, ROI for a feature, unique monthly visitors, click-through ad rate, etc.
symptomatic metrics: these are metrics that do not directly affect ROI (although we might believe they do) but are downstream indicators for one or more causal metrics. The number of outstanding bugs in a product, the number of bugs caught in a certain phase of development, percentage of code covered by unit tests, etc. are all symptomatic metrics.

My general observation, based on reading articles around the use of metrics for improving your processes, is that a lot of metrics-based improvement projects fail to distinguish between these two types of metrics. Partly, I think this is because while the causal metrics properly align your improvement efforts with your business's interests, they are also harder to define and measure. By contrast, a lot of symptomatic metrics are easy to find and measure, but their relationship to the business may be less clear.

For example, consider the balance between software quality and time to market. You can take a longer time when developing a feature or product to reduce the number of bugs that show up at deployment, or you can ship a feature more quickly, knowing that there may be both known and unknown bugs present. In this case, you can measure both number of known bugs at deployment time, and you can measure overall time-to-production for a feature (time from feature conception to deployment).

Now, if you can decrease time-to-production without increasing the bugginess of your code, that's a win. Similarly, if you can reduce bugginess without lengthening your production time, that's a win. However, both of these things will probably require some effort to implement. Another interesting possibility would be to simply make an adjustment of where you sit on this balance. For example, simply spend more time looking for and fixing bugs in your QA phase, to tradeoff fewer bugs for a slower time-to-market. Or vice versa to get to market more quickly, possibly with more bugs. Both of these adjustments are probably relatively painless to implement, in that no one has to change what they do, just how long they do it for.

So the question is, which one of these things ought we to do? My argument is that bugginess and time-to-production, both being symtomatic metrics, don't give us the answer directly. It all depends on our product environment. For example, when producing software to run medical equipment, a company reputation for quality might be more important than shipping new features quickly; or, in a highly innovative internet space, time to market might be king in terms of how much market share you can capture.

It's management's job to both help implement win-win changes as well as to set the "slider" of the quality/time tradeoff at the right spot. The trick, of course, is that it might be hard to measure this directly; there are several symptomatic things we could measure, including:

time spent in development
time spent in QA
time spent in deployment
number of outstanding bugs
profit for the product at a given point in time

Now really, the profit over some window of time (e.g. for a website, revenue vs. development spend for a given month) is the thing we want to optimize. The interesting idea here is for management to be able to run a series of experiments: if I increase/decrease QA or development time, how does it affect ROI for my product? How does the relative bugginess of a release affect its profitability? For certain "sliders" in the business, it is relatively simple to take a series of measurements to find a current "sweet spot".

An interesting idea here is that sometimes we work through things backwards. For example, we try to estimate "how long will it take to fully regression test a release", or "how long will it take to code up a feature", rather than "how buggy will the release be if we test it for X amount of time", or "how much of this functionality can you develop in X amount of time." In other words, rather than deriving the time-to-market from a set of estimates for all the steps, instead set the time to market by timeboxing those steps, and see what the outcome is. This is a powerful notion of metrics-based management that is hinted at (in the notion of Scrum timeboxed iterations) but which I have not seen explicitly suggested anywhere[1]. (Please post all the references to things that I've missed in the comments section--I'm sure there are plenty).

At the end of the day, however, it is hard to optimize things we can't measure. I think important metrics to gather are:

the levers we have available to manipulate our process (e.g. timeboxing)
causal metrics that affect our business (ROI, product profit)

We need to be aware which metrics are causal, and which are merely symptomatic, so that we are measuring things that directly affect the business somehow. This approach permits empirical management--adjust something you can control, see how it affects your causal metrics, rinse, repeat.

[1] Scrum timeboxes an entire iteration, but does not timebox an individual feature, so a team may be able to spend all their time on one feature, or spread their effort across many features. The closest thing I've seen here is the notion of the "Small" in INVEST user stories, where stories are limited to a certain amount of complexity. However, the story points in this case are still estimates of the work involved, rather than timeboxes around how much time to spend implementing a feature; the "small" requirement is really to permit more accurate estimation rather than to timebox the amount of effort (although it does secondarily have this effect, I've not seen this stressed in articles about this).

Cracking down on technical debt

2008-08-03T12:57:00.003-04:00

"Simplicity is the ultimate sophistication." --Leonardo da Vinci.

"Everything should be made as simple as possible, but no simpler." --Albert Einstein

"A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away."
--Antoine de Saint-Exupery

I've written before about the notion of technical debt. In this post, I want to discuss a few specific sources of technical debt that are easy to accrue, particularly in an agile, iteration-based setting.

Incomplete technology transitions

These can arise when a technical decision gets made to transition from one technology/architecture/design to another, and the transition happens incrementally. What can end up happening is that an agile team, say one operating under the Scrum framework, does not complete its incremental transition during the current sprint. Now, although the code is in a working state, there is a good chunk of technical debt arising from having code operating under two separate systems. This transition debt is problematic for a few reasons:

First, this can complicate debugging efforts -- when there is a problem with the system, someone has to determine under which scheme the code in question was written. Typically this can mean looking in two different source code hierarchies, or looking through two separate sets of configuration. The system is, as a result, more complicated than it needs to be.

Secondly, this can be an attractor for additional debt; if the old system is still around, and a developer is more familiar with the old system than the new, there is a very strong temptation to make changes/additions in the old system. This work simply adds to the outstanding transition work, and despite the developer's familiarity, is likely to be implemented in a more difficult or less efficient way (assuming, of course, there were valid technical reasons for making the transition in the first place).

Finally, this can cause extra work to happen during feature development that touches/interacts with the subsystem in transition, because either the cooperating subsystems have to special case two different interaction styles, or an adaptation layer has to be built to handle both subsystems and abstract their existence away from clients' concerns. Either way, you are writing more code that you would have if you had completed the transition and you just had one implementation of the subsystem.

Teams working on an iteration-based methodology need to do several things to avoid the pitfalls from transition debt:

when a technical decision for a transition has been made, it must be communicated clearly to the whole development team, including the reasons for the transition. This can help prevent the unintentional accrual of additional transition debt.
plan for more refactoring time when signing up for work, to leave time to complete transitions before an iteration ends.
communicate the existence of the transition debt to the Product Owner at the review, so that completing the transition can be scheduled as a backlog item. Furthermore, stress the priority of this carryover work to ensure that the transition debt exists for the shortest amount of time possible.

Obsolete/extraneous configuration

We'll call this type of technical debt configuration debt. There are a couple of sources of this type of debt:

transitional runtime configuration that still exists after the transition. For example, when a data partner was making an id space transition to extend the length of their ids, we had a flag to govern whether to use old or new ids with that partner, so that we could decouple our code releases from the partner's transitions. Over a year later, the flag still exists, but it is always set to use "new" ids, so there is certainly unneeded code to handle this.
exposing properties that would only change with a code drop as runtime configuration. In this case, the values of the properties would really only change if we rolled a new code release, so they could just as easily be compile-time constants that would not require the scaffolding to make them runtime properties, no matter how simple that scaffolding might be.

Unnecessary code hurts you in several ways:

it took someone time to write it in the first place
you have to compile it or run its unit tests over and over again while you're developing (death of a thousand cuts)!
people need to keep it in their mental model of the system instead of leaving room for parts of the system that actually do something useful

The easiest way to prevent the accrual of configuration debt is to review any new runtime configuration parameters at the end of each sprint (which you probably have to do anyway so your operations folk know how to properly configure the new system). Then, where possible:

turn as many runtime parameters into compile-time constants as possible
ask under what conditions the parameters will no longer be needed (for example, for configuration that assists with an external transition, add an item to the product backlog to clean up the codebase after the transition is successful)

Obsolete/insufficient architecture and design

Architectural debt is probably the most nefarious, because this is debt that doesn't actually get created when the code is first written. Instead, this is usually caused by external factors such as:

business environment changes and expected traffic is significantly different than originally anticipated. This can leave you either with an overly complex, over-engineered system, or with a too-simple system that can't easily scale.
product direction changes, and the architecture is not flexible along the new axis of change, so that new development is overly difficult.
expected performance of a new provisional architecture is invalidated by experience

Basically, as soon as you realize you need to change your architecture, you have magically "created" technical debt out of all the code that depended on the first architecture. In reality, this debt is probably unavoidable, and what you've really done is convert your inability to perfectly predict the future into a set of work that incorporates new knowledge about the problem domain.

This can also be hard to identify by scrutinizing the code, but there are some external symptoms of it:

difficulty meeting desired performance or scalability targets, especially when concentrated in a certain feature subsystem
adding new instances of a certain class of feature does not get easier over time
lots of bugs being generated by a specific subsystem
increased time-to-market for new features
accleration of bug creation rates
accrual of standard operational processes that require manual intervention/support

So when you have some or (gulp) all of those symptoms, you probably have architectural debt lurking in your system. Once you have identified it and have a new target architecture, a lot of this will get converted into transitional debt while you are making the changes.

Technical debt vs. technical investment

I want to be careful here to distinguish between two sorts of non-functional requirements that might show up on a "tech backlog":

technical debt: this is current brokenness or unneeded complexity in the system that is actively slowing down the business of turning product backlog into working software for your customer.
technical investment: these are things that are not necessarily broken per se, but which could speed things up for someone. A good example of this would be automating a manual process.

Technical investments can probably be put off while you have existing technical debt, although it can sometimes be hard to distinguish between the two. Clear technical debt should probably be prioritized at the top of a product backlog, unless there are really high ROI items that might trump it. In general, getting rid of technical debt will increase the ROI of everything else on the backlog, simply by decreasing the "I" part. It can also make estimation more accurate by reducing the complexity of the system to which new functionality will be added.

Whose responsibility is technical debt?

Generally, as the folks with the technical ability to recognize it, it is the development team's responsibility to try to avoid accruing technical debt while producing product. Failing that, it is their responsibility to recognize/document existing debt and to advocate for its removal. However, note that there are often symptoms of technical debt, such as those I've listed above for architectural debt, that can be recognized by non-technical folks too.

On the flip side, business folks / product owners need to be able to trade off short term wins that accrue technical debt vs. taking longer to produce a product with less debt. Communication with the tech team is of vital importance here; undoubtedly there will be times when a short-term win will be important (especially with a first-to-market situation), but it needs to be accompanied by a plan to eliminate the accrued debt. i.e. Treat your technical debt like credit card debt that should be paid down ASAP, and not as a long-term mortgage.

The interest on your technical debt is probably not tax-deductible.

The Crucible

2008-04-18T19:11:00.004-04:00

We've recently re-adjusted our sprint length to be 3 weeks, down from 4. The reasoning behind this was to allow us to align with sprints on other products, to give us a uniformity of scheduling and to permit inter-product developer swaps from sprint to sprint, if needed.

One of the side effects is that our planning process now takes up proportionately more time of the sprint and so the actual work time of the sprint is quite compressed. Couple this with a handful of developers being out for personal reasons (wedding, health issues, impending birth of a child) and suddenly this sprint had a lot of high priority sprint backlog and not a lot of available story point capacity on the teams.

But then something magical happened--what I'll call the "crucible moment." The status of the sprint was that there were a bunch of high priority, smaller maintenance tasks (cleanup of existing features, releasing the previous sprints' work to production, etc.) and then one big new feature. As the teams were working their way down the list of user stories and filling up their story points, everyone quickly realized that that big new feature might not fit into the sprint.

With the purifying fire of timeboxing (forgive my overly dramatic metaphors here), the sprint teams and product teams immediately began self-organizing and negotiating. No one wanted to have a sprint without a killer feature, and so the horse trading began. Some of the higher priority stories were off the "tech backlog" -- non-functional investments in build infrastructure, etc. Developers began identifying tech backlog items that could stand to wait. Product owners began reconsidering some of the higher priority smaller stuff, conceding that some of them might not be so important after all. Different development teams tried to juggle stories between themselves so that the large story could get onto one team's sprint backlog (we prefer not to split stories across teams, to minimize cross-team dependencies). Some stories were scoped down to take less time.

And then, when it was all done, we discovered there was enough room for two pretty big features.

This was scrum at its best--focusing on the art of the possible to squeeze as much as possible into a fixed amount of time. No time for extraneous explorations, just a focus on extracting the maximum product value out of the available resources, with a brutal willingness to critically re-examine priorities. What's more, it was carried out in a context of teamwork between the product owners and the teams; everyone was trying to figure out how to get those features in.

So, a great day for our product and for our organization. We might actually be getting the hang of this scrum thing....

Sprint Review

2008-03-26T23:14:00.003-04:00

Just wanted to follow up on my previous post about how we were going to get back to scrum basics this sprint. We had our sprint review and sprint retrospectives today, and the sprint was viewed as a great success.

Here are some things that I think contributed to our success:

Better estimation: We used past task estimation history plus monte-carlo simulation, as I described earlier, to sign up for a set of achievable tasks. The end result was that we showed up at the review with 90-95% of the committed work done. The parts that didn't get done were due to external blockers we couldn't do anything about (e.g. external partner didn't have a data feed ready) or were due to too many round-trips through the design-IA-product-eng collaboration cycle (essentially, deviating from the plan we set forth at the beginning of the sprint) that took time away from other things.

Commitment: Towards the end of the set of product backlog our team was planning, we hit a fairly large task that was hard to estimate out. However, we had two team members who were very lightly booked, and we simply asked them if they were willing to commit to finishing the task somehow by the end of the sprint. They signed up, and they got it done. There was really no talk of punting things off the sprint, although there were a couple of rescopings and/or a change of plan due to time constraints that happened to get things done by the review. In general, I think these changes actually just meant the abstract functionality in the user story got built with fewer resources -- i.e. we completed them more efficiently thanks to the pressure of timeboxing.

Empowerment: Thanks to being committed to their tasks, my teammates basically kept themselves unblocked most of the sprint by engaging with other departments (design, product, IA, ops, QA) early on to make sure they could get what they needed. Most of the time things stayed unblocked by getting the right folks together in one room. One of my teammates mentioned that one of the things I was best able to do was to simply identify which people needed to be in the room! Then he went off and arranged the meetings and got things done himself.

Shippable product: We planned explicit tasks for each feature to test/verify it, to show QA how it worked, and to demo it and get explicit signoff from the product owner. Our product owner gave me the feedback that this resulted in most of the user stories we called "done" being potentially shippable.

Self-organization: The team dynamically swapped tasks with one another to load balance effectively during the sprint. There was such appreciation for this help that people started buying each other donuts as thank-yous, and I think we ended up with around 4 dozen donuts being exchanged by the end of the sprint. Bad for the waistline, but good for team morale!

On another note, this was a re-entry for me into the dual role of scrum master and team member (in that I signed up for development sprint tasks as well as being scrum master). For other developers who find themselves being called on to carry the scrum master mantle, here are some tips to help you survive:

Only book yourself half-time. The rest of the time will be eaten up by scrum mastering, and if you don't scale back on your coding commitments, you're just going to end up staying up late working all sprint.
Tell people how to get unblocked rather than trying to unblock them yourself. This seems pretty simple, but is a big time saver. For one thing, it's probably quicker to describe what to do (e.g. just set up a meeting with X, Y, and Z) than it is for you to send out the meeting invite yourself. Plus, you remove yourself as a bottleneck -- so your teammate doesn't have to wait for you to send the invite. Furthermore, after you do this a few times, your teammates will pick it up and will be able to just solve more of their problems themselves without asking you for help.
Get good reports. I invested 5-6 hours towards the beginning of the sprint to get some automated reports set up to get info out of our ticketing system. I really ended up just using two reports:
1. Storyboard: for each story, identify who has tickets against it, and identify whether each ticket is "Not Started", "In Progress", or "Finished". This provides a quick way to scan the status of each story and reminds you to ask questions like: what do we need to do to close out that story again? why are we working on lower priority things instead of the higher priority ones? etc.
2. Burndown count: for each story, add up time remaining on each ticket, and then provide a total for the amount of hours left for the sprint.
So what I did was: right before scrum, I pulled up the Storyboard report to check status so I could ask followup questions during scrum. Right after scrum, I hit the burndown count and then actually plotted by hand on a big piece of flipchart paper the burndown graph. That was pretty much the extent of the reporting I needed.

So, all in all, things turned out pretty well. I'm excited to see the teams back in a good sprint cycle groove and turning things out (our review meeting lasted almost 5 hours, mainly because so much had gotten done across the multiple teams working on the product that it took a while to go over it all). We'll be mostly keeping things the same going into the next sprint, so I'll be curious to see if the teams are able to accelerate now that they're used to working this way.

REST: Unix programming for the Web

2008-03-16T22:12:00.006-04:00

I've been giving some thought to REST-style architectures recently, and recently re-read some of The Art of Unix Programming by Eric S. Raymond. ESR notes that some of the characteristics of Unix-style programming include:

Do one thing, and do it well (attributed to Doug McIlroy)

Everything is a file.

Comprise complex systems by connecting smaller, simpler programs (e.g. Unix pipes).

Unix-style systems have had undeniable success and remarkable stickiness for a technology; I have an old copy of my father's System V manual from when he worked at Bell Labs (yeah, they actually printed out the man pages and bound them!), and I pretty much recognize everything in there. Sure, there are many new commands available, new kernels, new distributions, etc., but they are all very recognizably Unixy.

I've been thinking about how this philosophy applies to the Web 2.0 world. I think this list turns into:

Do one thing, and do it well.

Everything is a RESTful service.

Comprise complex systems by interconnecting smaller, simpler services.

For one thing, "do one thing, and do it well" isn't limited to Unix, this is a key part of abstracting and decomposing a technical problem. But we see it everywhere: I read my mail on Gmail, keep my to-do lists on Remember The Milk, store photos on Flickr, keep my browser bookmarks on del.icio.us, etc. All these sites adhere to this principle.

But taking things down a layer, what does this mean to someone architecting or implementing a Web 2.0 service? For one thing, I think this means that it would make sense to break your overall service down into individual microservices that are individually maintained and deployed; i.e. break things down to the smallest level where they still make coherent sense.

As for "everything is a REST service", the everything-is-a-file abstraction worked for Unix because there was a small set of common operations that applied to files (open, close, dup, read, write). Sounds a lot like doing REST over HTTP, with HEAD, GET, PUT, POST, DELETE.

Combining various small REST microservices into larger services is already being done (this is, after all, basically what iGoogle is) on a one-off basis. The main question is: what is the Web 2.0 equivalent of the pipe? Namely, is there an easily understood abstraction for composing webservices? Sounds like a topic that might be rife for some kind of logical calculus (like the relational calculus for databases or the pi-calculus for concurrent processes), e.g. the REST calculus. If there were a couple of easily understand and specifiable combinators, these could be pretty easily built into some language-specific libraries for use in quickly building some new macro-services.

I might try to interest some of my old colleagues from my programming language research days to see if they have anything to say about the matter....

Comments definitely welcome here. Is this a new idea? Are others espousing this? Is this even a good idea?

Return on INVESTment

2008-02-28T23:53:00.003-05:00

At our last pre-planning meeting, we made a point of putting all the user stories into INVEST format, and I was pleased that there was a general consensus that this worked well.

I think this took quite a while, partially because we were all getting used to evaluating the statement of the stories critically, but I think this was worth it. We had product, IA, and engineering folks suggesting wordings for stories, and at least some of our stories came in in a canonical format:

"As <who>, I want <feature>, so that <value>."

Having the full INVEST filter did help us with a few things:

Independent: We did not have a lot of stories that were dependent, but we did end up with a very few that were cross-team. Since this was just a handful of the stories, we figured it would be ok to leave the story as is (since we couldn't quickly come up with a way to rewrite it) and track the dependencies through our scrum-of-scrums.
Negotiable: I think we generally nailed this one. We did not look at any IA wireframes or Design mockups during pre-planning, and were able to identify the high-level goals here. This has already become useful in planning, where my team has suggested a new / faster approach to at least one story which was not as originally conceived, but our Product Owner agreed that we still hit the abstract requirement.
Valuable: There were a few stories that were original cast as abstract technical requirements (e.g. make the middleware support foo), but we refactored them in a way that expressed value to the end user (which, incidentally, will make it more obvious how to test).
Estimable: We've already agreed not to do a small handful of stories because we realized there were too many unknowns; for one of these, there was a middleware task that depended on IA we hadn't seen and on data we didn't have in the DB yet. We actually spent a lot of time trying to talk about how to tackle this before we realized we couldn't estimate it. Since it was not critical for implementation this sprint, we agreed with the product owner to turn this into a story where we would do a feasibility study and rough design by the end of the sprint instead, which was something we could commit to. (The value here is to the product owner, who will then be able to write an Estimable story about the feature!)
Small: We did a good job here of refactoring large stories into multiple pieces and then figuring out how to get Value out of each piece. We only had a handful of medium to large stories.
Testable: We're asking each team during planning to make sure they hash out "how to demo" for the feature with the Product Owner, so we should know if we've gotten there by the end of sprint planning.

With the up-front work on the stories we did, our team has found it easier to negotiate a specific solution and in some cases to actually plan without the product owner, which is actually convenient (he's doing multiple duty so he's having to bounce back and forth between multiple sprint planning sessions!).

INVESTing in user stories

2008-02-24T16:21:00.004-05:00

In my previous post I made reference to the INVEST acronym for evaluating user stories:

(I)ndependent
(N)egotiable
(V)aluable
(E)stimable
(S)mall
(T)estable

I'd like to spend a little bit of time talking about each of these characteristics, and motivating why each is important through some anecdotes about what happens when each characteristic is not attained.

Independent: The idea here is that stories are free of dependencies from one another. (A good test would be to ask if you could implement them in any order, rather than just in the priority order generated by the Product Owner). We want this for multiple reasons: first, on a multi-team project, it allows stories to be load-balanced across teams more easily, since in theory any one story could get moved around, rather than having to move an entire batch of them. Secondly, and on a related note, it means that when sprint teams sign up for work, they can draw the commit line anywhere. Thirdly, it helps avoid cross-team dependencies (although a scrum-of-scrums and specific attention to dependent situations can handle it, it is still much easier to handle all your dependencies intra-team).

For a specific example, consider if you have teams that are layered horizontally by functional layer (e.g. a database team, and a webapp team). If you have two stories which are the "halves" of building some functionality, and assign one to each team, you take on the following risks:

teams must coordinate closely around this feature (extra communication/tracking overhead)
if one team finishes but the other doesn't, you may have extra "clean up" work at the end of the sprint to make the software still work, and you may have had a team do work that ultimately had no product impact that sprint (e.g. middleware can handle some new data and display it, but data didn't actually apppear in the DB, so no actual behavioral difference)

We'll see that some of the other INVEST characteristics actually help track this down as well.

Negotiable. To me, this means that the requirements given are as abstract as possible, so that the actual details of what is going to get built are determined by the team and Product Owner during sprint planning and modified as needed over the course of the sprint. For example, "user can change timezone with one click" vs. "user timezone is shown in a 100x25 dropdown box in the header". (Of course, if the latter language actually represented a contractual obligation, then that might be as abstract as you can make it, but you get the idea...).

This is primarily important for two reasons: to allow the team to exercise maximal creativity and to allow the team to adjust for the unpredictable events that will happen mid-sprint. In the former case, the dropdown box might be tacked on to the story as a starting point or suggestion, but the team may come up with an approach that is easier to implement or which actually satisfies the Product Owner more. In the latter case, it leaves some room to negotiate if the team has under-estimated and is behind mid-sprint, and still be able to "finish" the user story.

The danger of not being able to negotiate the functionality is really twofold: first, not completing the full set of work in a sprint eventually crushes team morale; not being able to complete an assignment is a big downer, especially when you have worked hard and because, due to the inherent complexity of software development, things just took longer than you thought. Eventually you get to the point where your team will just shrug, and say, oh well, I guess we won't finish that stuff. I have seen this happen first-hand, and it is demotivating.

Secondly, there is danger to the product roadmap. By not having requirements couched abstractly, you run the risk of slipping features or having unfinished, "carryover" work, which adds up and pushes the product roadmap back. By giving the team the freedom to brainstorm a way to satisfy the requirement in less time, you are not letting them off the hook -- you are using the deadline of the end of the sprint as pressure to encourage the team to develop the functionality as efficiently as possible. The resulting functionality may not be as complex or deep as originally hoped for or conceived, but if the spirit of the story is met, then you have some aspect of it ready to go out as shippable product.

Valuable. Each user story should provide value somehow. The original articles I read about this amended this to "provide value to the end user / customer." While I think noting the value explicitly can help product owners check off that the story will help their Key Performance Indicators (KPI), I think the more important thing this brings is that the team is aware of why this story is important. This can help constrain the solution space for Negotiation in an important way. Finally, this is just a cross-check for Indepencence; if this story is completed, and no other, does it generate value, or do we also need another story to be finished in order to get the value?

On a side note, the phrase "to the end user" is an interesting one. We keep a "tech backlog" of infrastructure/refactoring ideas around, and prior to each sprint, we evaluate which ones are critical to current development, and ask for those to be prioritized in with the current product backlog. Generally, we have been very judicious about this, maybe in a nod to providing direct benefit to an end user -- we usually wait until an infrastructure adjustment is needed or desirable for implementing new functionality before taking it on. This suggests that other infrastructure work get carried out during lab days, to scratch those developers' itches. I'm still up in the air over how strict we should be about "to the end user", but I do lean towards requiring that most of the time.

Estimable. If you can't put an estimate on it, either the requirements are too vague (see Testable), or the technical solution is unknown (e.g. "I don't even know if this is feasible!"). In the latter case, a suggested tactic would be to alter the user story into a feasibility study / research effort for this sprint, which could be easily timeboxed; the original story could then be revisited in a later sprint, when there will be less cloudiness around it, and it can be properly estimated.

Where this can bite you is in signing up for work you don't know you can finish. This sets up for an expectation mismatch with your Product Owner, and also prevents you from making efficient use of your time; you run the risk of attempting to estimate it, and either grossly underestimating it, putting all the lower-priority sprint backlog at risk, of grossly overestimating it and not signing up for enough work, or of padding the estimate to try to account for the risk and then spending more time on it than the feature is really worth.

Small. We want things that will take no more than a full sprint to do, and hopefully less. While the full scope of a feature vision may require more than one sprint, you want to refactor it somehow so that you get something out of even the very first sprint (see Valuable); then you can be sure that you reap some result from your effort in shippable product, as opposed to doing some partial work, and then having the remaining work deprioritized for several months before you can extract value.

Finally, a purely pragmatic reason for keeping the stories small is to reduce the risk of gross under-estimation. I've personally been way off on a single big story (as everyone is from time to time), and when it's been a big one, I've simultaneously trashed my personal life for a month of overtime and sleep deprivation, while requiring a bunch of load-balancing and rejuggling across multiple sprint teams due to the fact that the rest of the sprint backlog I signed up for was now at risk. So don't do that -- take smaller bites, just like Mom used to say.

Testable. This primarily serves three purposes: ensuring that Product Owner expectations are in-tune with what the team thinks it is delivering (see also Negotiable), giving the team a way to know when it can stop working on the story, and giving the testers a starting point for writing their test cases. Beware of non-quantifiable adjectives like "good" or "acceptable" in your user story descriptions. For a while, when we were doing sprint planning on spreadsheets, we had a "How to Demo" column--this worked great while we did it, but we never had enough discipline to follow through here and continue doing it. This is one of the things I'm hoping to bring back during my Scrum revival next sprint.

There's nothing worse than showing up to a sprint review and having your Product Owner say, "but that's not at all what I asked for, or that's not what I meant." Big morale crusher for everyone involved (team and stakeholders).

Finally, this gives the Scrum master a hook to save the team from perfectionism or unbounded creativity. For example, if you've gotten the feature to the point where it satisfies the acceptance tests and has been built up to your standards of quality, just stop working on it, and start working on the next user story. This is your old friend, the Pareto principle, at work -- would you rather spend a day mining the long tail of a functionally complete feature, or would you rather spend it getting the up-front meat of a new feature? The other place this helps is when you finish a feature, and you and the Product Owner are looking at it, and you now see something totally awesome that is now possible -- stop, ship the feature you have, and queue the good idea up as a user story for the next sprint so it can be properly prioritized with everything else. Again, this is about efficient use of the time in the sprint.

I'm anticipating having this be a little painful as we work through this together with the product team the first time; we've all signed off on this in principle, but we've never actually attempted to make each story adhere to INVEST. I suspect, like all similar things, it will be a bit robotic for the first few stories, but we'll quickly get the hang of it and be able to move on during pre-planning. I'll let you know how it goes.

Back to Basics

2008-02-23T12:59:00.003-05:00

I've been re-reading "Agile Software Development with Scrum" to see if it has any insight for some of our current product struggles. Fortunately, I think it does, and I've been getting fired up for a great Scrum revival. [Revival tent picture courtesy of PinkMoose on Flickr, used under a Creative Commons Attribution license.] I'm going to be reprising my role as Scrum Master for my team next sprint as well, so hopefully I'll be able to transmit some of these values.

Looking forward to getting back to some Scrum fundamentals, including:

sprint goal: let's set an overarching sprint goal we can be working towards
INVEST user stories: in particular, the "N" -- negotiable -- means we have maximum flexibility to react with agility during the sprint
commitment: the team signs up for a set of work, and commits to completing it by the end of the sprint. No carryover. If burndown shows we are behind, we need to collaborate with the Product Owner to revise functionality / approach so that we have an achievable set of work. Also, we need to have the team take responsibility for completing the work -- no more reporting blockers and then giving up without trying something else.
product increment: end of the sprint, we have a potentially shippable product. No half-baked / half-finished pages on the site.

I'm pretty excited for the upcoming sprint. We've got a lot of talented folks, we just need to really get them motivated and self-organized, and then awesome stuff will erupt.

REST-based service design, v2

2008-02-13T21:21:00.003-05:00

I want to revisit the basic "favorite food" service from last post, in light of some further discussion I've had with colleagues.

http://host/path/{userid}/favorites

GET: returns a list of the user's favorite foods, in some representation. Returns 404 NOT FOUND if the user has never specified favorites.
PUT: sets the list of the user's favorite foods to be the value sent, creating it if necessary.
DELETE: removes the user's favorite list. This is different than doing a PUT with an empty list.
POST: returns 405 METHOD NOT ALLOWED.

http://host/path/{userid}/favorites/{food}

GET: returns 200 OK if the food is a favorite of the user. Returns 404 NOT FOUND if the food is not a favorite of the user.
PUT: makes the food one of the user's favorites
DELETE: makes the food *not* a favorite
POST: returns 405 METHOD NOT ALLOWED

So, PUT can create a resource if it doesn't exist, and DELETE of a URI means the next GET of it (in the absence of any other operations) will be a 404 NOT FOUND.

One note is that as a client of this API, I want and can make use of the most atomic operation available. For example, to make "spaghetti" a favorite food, I could either:

GET http://host/path/{userid}/favorites
PUT http://host/path/{userid}/favorites (new list with spaghetti added in)

or I could just:

PUT http://host/path/{userid}/favorites/spaghetti

Note that in the first case, I might have an atomicity issue in the presence of concurrent access, so I might need to build in some sort of optimistic locking protocol, where the representation of the favorites list has a version number on it that is checked by the PUT operation. However, if I just use the second method, I don't have this issue, because the server handles all my concurrency/transactional stuff for me.

REST-based service design

2008-02-12T22:30:00.004-05:00

Just wanted to walk through a quick REST-style API design exercise. Let's suppose I want a service that lets folks maintain a list of favorite foods. Just as a quick strawman design, let's walk through some URIs and supported operations:

http://host:port/path/{userid}/favorites: GET on this URI returns a list of the current user's favorites. List might be empty. PUT on this URI overwrites an existing list of favorites. DELETE on this URI sets the list of favorites to the empty list.
http://host:port/path/{userid}/favorites/{food}: GET on this URI returns something (really, we just want a 200 status code) if the food is a favorite, or returns 404 if it is not. DELETE on this URI un-favorites the food for the user. PUT on this URI makes the food a favorite.

So, questions for the audience:

Does this look reasonable?
Does the use of PUT look right, or would you use POST here?
How about the use of DELETE to set the full list to the empty list? Does it make sense to DELETE a URI and then be able to GET it and have something be there?

Code Artisan's New Year's Resolutions

2007-12-30T11:05:00.001-05:00

Well, it's a new year, so time to sit down and draw up a list of development resolutions. Without further ado:

1. I will create unit tests for all my classes. There's no excuse not to have at least some kind of unit test for each class that gets created; this at the very least provides a stratum on which other people can flesh out the tests to cover things that come up. This may not be full-blown TDD, but we could at least enforce this with some kind of unit test coverage tool like Cobertura.

2. When debugging a defect, the first step will be to write an automated test case that captures it. Nothing is more annoying than a bug that can't be reproduced. If you can actually get to the point where you can reliably trigger the bug, you're probably most of the way towards finding the problem. Furthermore, in the process, you'll have created an easy-to-run way to know when the bug is fixed. Since the test is automated, there's a good chance that test will get re-run a whole bunch more times in the future of the codebase, making sure that once a bug gets squashed, it stays squashed.

3. I will write Javadocs for all my public methods, except where they can be inherited from the Javadocs in an interface. This is just common courtesy to myself and the other developers. I'm cutting myself a break here, since one could argue that you really ought to document the protected ones too, but we'll start small. This will also encourage definining interfaces and coding to them, as it saves a lot more effort if you can just Javadoc up the interface itself! For bonus points this year, write or find a tool to automatically check this.

4. I will have explicit up-front design sessions before writing any code. I really don't know why we don't do this more than we do, other than we forget to do it. There are all kinds of good reasons to do this, like the fact that we always come up with better designs in a small group than we do by ourselves, or that this will help all of us become better designers by being present while good designs are being born. Or that a good design is usually easier to implement, and with all the hard work I've already set out for myself, I don't mind finding resolutions that save me some effort.

5. I will relentlessly learn to estimate better by considering my past estimating history when providing a new estimate. First step here is to actually look at my past history, so I know what it is! (Fortunately, we are already collecting the data for this...) Then possibly use techniques like those I laid out in a previous post about improving estimates.

Well, that's it. Those are some pretty specific resolutions, which, I'm given to understand, makes it more likely that they'll be adhered to. What are your coding resolutions for the new year?

Who designed this stuff, anyway?

2007-12-27T21:36:00.001-05:00

Many development organizations have an "architect" role, meant to provide technical direction over an entire engineering process [As far as I can tell, "architect" really seems to be shorthand for "someone with deep and broad technical understanding and experience"]. How does this mesh with scrum-based, grass-roots development by empowered teams? Or perhaps, the real question is: if I have folks that would traditionally fill an architect role, what's the best way to utilize them in a scrum setting?

Three areas in which these folks tend to contribute are (although this list is certainly not exhaustive):

R&D: advanced prototyping/proof-of-concept work
Due Diligence: evaluation of emerging technology
Design: participating in system and/or software architecture and design

Let's take each of these areas in turn.

R&D.

Here the notion is to invest some amount of exploratory effort to remove technical risk from a product or feature--to the point where the feasibility of an approach can be established and an associated rough LOE (level of effort) can be given for a real implementation using this approach. We typically hand this type of task off to an architect, hoping his/her broad technical background can help provide hints/leads to navigating the solution space, and thus reduce maximum uncertainty with a minimum of effort (bang for the buck!).

Depending upon the nature of the problem at hand, and/or the structure of the available scrum teams, there are two readily apparent approaches:

run an R&D scrum team; this makes sense when there are sufficient architects around to form a team of them, and when the explorations are not directly tied to an existing product (e.g. technical feasibility for a new product)
embed your architects on the scrum teams themselves: this makes sense when your architects are more scarce (because it will be beneficial to have them on the teams for the other reasons we'll mention below), but also when the explorations are directly related to an existing product. In this case, the rest of the scrum team they are on will benefit directly from (and help participate in) the exploration--this learning process will make a later production implementation easier, and will furthermore help teach the other team members how to do this type of work. This is vitally important if we are short on architects and good ones are hard to hire--then we need to view these other folks not as "non-architects" but rather as "architects-to-be." This model is also easy to follow if you are already using the idea of prototyping sprints.

Technical Due Diligence.

Here, again, we are counting on our architect's experience to enable a thorough and accurate assessment of some technology. If there is already an "architecture scrum" running as described above, this might be a natural place to fit these efforts. However, in the absence of a dedicated group, it may fall on an architect that is embedded on a scrum team (ideally one working on a product that might be impacted by the technology in question), which would also be perfectly fine.

Design.

Architects' experience helps most during design exercises--knowing what kind of pitfalls to look for, what kind of generality to build in, and what kind of approaches/patterns are available and relevant. Design is also an area where collaboration is both possible and usually helpful (at least up to the amount of people that can fit comfortably around a whiteboard!). So, no doubt--we want our architects participating here in order to get good designs up front and hopefully minimize refactoring later.

Again, I am going to argue for the embedded approach here, as opposed to running an architecture scrum that spits out designs. I think the benefits here are overwhelming:

by generating designs as they are needed for feature development, effort will not be spent building in generality that may never be exercised
by having the scrum team participate in the design, those familiar with the details of the existing system will have input, ensuring the design is complete
by having the team participate, they will learn to become better designers themselves (again, a key benefit if there are not enough architects to go around--and there probably won't ever be, by the way--as the architecture will have to live and grow regardless of whether an architect is available to design it all).

Archi-who?

Well, I seem to have convinced myself, anyway, that the right thing to do with your architects is to just put them on your scrum teams and let them have at it (Ken Schwaber would be so proud!). In other words, don't treat them any differently than you would any other engineer in your development organization (indeed, in our current organization, we have "architects" that do production development, and "engineers" that do software architecture, so we're probably not far from this anyway). The main benefits of this approach are:

relevance - by staying grounded in the day-to-day realities of creating products, our architects will be focused on real needs
integration - by having architecture activities happen in the context of normal scrums, those activities will have a high likelihood of impacting production code
collaboration - as a team member, there are no artificial barriers to communication, no "us" and "them" mentality between architecture and engineering
resourcing - architects can leverage their teammates to provide extra horsepower towards architecture tasks
learning - by having scrum teams responsible for architecture tasks, experienced team members will teach less experienced teammates how to accomplish architecture-related tasks

What's the APR on your technical debt?

2007-12-18T18:57:00.001-05:00

Today we're going to discuss technical debt--what it is, what its impact is, and what to do about it.

One of the key tenets of scrum and other agile development methodologies is the idea of producing production-ready increments of work at the end of each iteration. Ken Schwaber calls this "sashimi" in the scrum framework--a slice of product. You may have also heard scrum practioners talk about the definition of "done", i.e. what steps have to be completed before we can consider a user story satisfactorily completed, which may include steps like:

code written!
unit tests written and passing, with a certain degree of code coverage
automated acceptance tests written and passing
code refactored
code documented/commented (e.g. javadoc)
product owner signoff
documentation updated (e.g. UML diagrams)
etc.

In general, the notion is that the definition of "done" should capture the degree of code quality the team and product owner mutually agree to.

Now, anyone who has tried to do this has probably run into this (as we have): a sprint gets started, and at some point through the sprint, you realize that you're not going to be able to finish all the user stories you signed up for by the end of the sprint. Or at least not get them all the way "done". So, you yield to the temptation and cut corners, just doing the implementation without all the rest of the supporting infrastructure, so that at the sprint review you can call this done. Oooh, it's so tempting! C'mon, the product owner isn't going to look at your CruiseControl outputs....

But now, what you've done is that you've incurred technical debt. All that code doesn't have automated test coverage or adequate documentation, and probably has more complexity than it needs because you didn't get a chance to clean it up and refactor it. Now it's going to be harder for someone else to work on it (including you, two months down the line!), because it's inadequately documented and complex, and it's even hard to refactor it without introducing bugs because you don't have the automated tests to help you know if you covered everything! Thus, future work on this codebase gets bogged down a little bit by all this cruft.

Now, technical debt works just like credit card debt: if you keep racking it up, it really starts to kill you. Because you are now working slower, you again find yourself behind the 8-ball of not being able to finish everything by the end of the sprint, and you rack up even more technical debt. Our product owner recently commented to me that he felt like we were getting less done than we used to at the beginning of the project with fewer people, and I'm starting to think it's all this accrued technical debt.

Unfortunately, getting out of technical debt is pretty similar to credit card debt, and is also probably psychologically just as hard. The first step is cut up the credit cards -- refuse to accrue any more technical debt from this time forward, and instead only demo things at the review that are really fully "done", even if this means you don't get to everything you thought you would. This takes a certain degree of courage, and is probably something to do right after a major release, rather than just in front of one....

Then, start paying down the debt. There are a number of ways of measuring this debt, many of them easily automated with Maven, such as:

Cobertura - unit test code coverage measurement
PMD - static analysis to identify unclean code patterns, including code complexity and design issues
Javadoc output - how many of your packages are missing documentation? how many classes don't have class documentation, and how many of the public/protected methods are missing javadocs? Do you have any javadoc warnings?

So the point is, you can measure all of these things, and come up with some numerical measure (pick some formula you like) of your technical debt. Furthermore, if you are tracking the user story velocities of your teams, you can actually start correlating your technical debt measurements at the beginning of a sprint with the story point velocity output by that sprint. In fact, you may even be able to figure out exactly how much drag your technical debt is putting on development, which means, based on your average cost for developer time, you can actually put a dollar amount on it. You can actually tell how much interest you're paying on your technical debt in real dollars.

Then see how the conversation with your product owner goes when you ask for some technical debt paydown to get prioritized on the product backlog....

Scrum thoughts: Improving individual estimation

2007-12-05T23:34:00.001-05:00

In my last post, I described that our team routinely overestimates the amount of work it can finish in a sprint. The amount of this overestimation is quite high at times - sometimes we only finish two-thirds of what we initially plan.

So how can we get our estimation better? I think there are two sources of estimation error: one is how many hours per day you actually have to apply to sprint tasks; the other is how many hours it will take you to actually finish something.

In his article on Evidence-Based Scheduling, Joel Spolsky describes how we can compare the history of a developer's estimates vs. the actual time required to get an error factor in their estimates. I think this does a good job of addressing the second type of error, but you need to account for both if you are going to get an accurate estimate of how much you can do in a sprint.

For example, as a team lead, I often get interrupted while coding when someone asks me a technical question, or to participate in a discussion on the dev mailing list, etc. I usually "leave the clock running" for these things, so I think evidence-based scheduling would cover this kind of inaccuracy in my estimation. However, I also get scheduled to attend extra meetings which don't go towards finishing sprint tasks, and I don't leave the clock running for that. I also sometimes forget to log all my time spent against tickets, even though I update the remaining time on the tickets to make the burndown accurate. So that's another source of inaccuracy.

Fortunately, I think there's a pretty simple solution that covers all these things. Let's assume we recorded in the past sprint (or past few sprints) how many working days someone was going to be available. Then we simply take the time logged against tickets that were originally planned for in the sprint and divide that by the number of days worked to get a per-day burndown figure. If someone works nights and weekends, they'll just have a higher per-day burndown rate. If someone routinely has other duties that keep them off sprint tasks (e.g. doubling as a scrum master), their per-day burndown will be lower. If someone forgets to book time, their per-day burndown will be lower. If they routinely forget to plan for required elements of a task or if the product owner injected some new requirements mid-sprint, the per-day burndown goes down.

Now, once you know that, you can use the evidence-based scheduling trick from Joel's article to use the actual task estimation history (here's where we compare original estimates vs. actual estimates for finished tasks) to put an actual confidence level on how likely that developer will have a set of estimated tasks done by a certain time.

So, during planning, you let someone estimate hours as if they would get to spend uninterrupted time on it, and then you can use the EBS plus per-day burndown to monitor whether something will likely get done by the end of the sprint.

To make it concrete, let's say we want to put a 95% confidence level on being able to finish all sprint tasks. So, for each new ticket I write, we go through the following exercise:

100 times, do:

for each ticket so far, select a random estimate ratio from my estimation history, multiply by the original estimate to get a likely actual time
add up total likely time, divide by per-day burndown figure to get a likely number of days
if likely days <= days available this sprint, trial is a success

If you have at least 95 successes, you may take on that task for that sprint. Otherwise, you're not allowed to commit to it.

Scrum thoughts: Fixed feature set vs. fixed time

2007-12-05T22:51:00.001-05:00

I think we're going to enter into an interesting discussion with our product owners where they would like to propose moving away from a fixed-length (in time) iteration model to a fixed feature set model. The claim is that it is too hard to maintain a release roadmap when the set of features that comes out of a sprint is fluid.

[Sidebar: our sprint teams routinely overestimate how much we can accomplish, so items are always falling off the end of our sprints. We've mitigated this somewhat by at least marking the bottom part of our sprint backlog as "low confidence items", but the truth is we rarely get to those items, i.e., we never totally finish our sprint backlog.]

So I think there's a valid complaint here, however, I think it is misplaced. We also never actually get far enough ahead on story point estimation with the product team to have more than a sprint's worth of rough LOEs on things. So I'm not sure that any roadmap that would get produced would have any grounding in actual engineering estimates.

So now that I think about it, this is probably more of a version numbering problem more than anything else. We increment our version numbers with every sprint, and furthermore release the output of every sprint to production. So I think the issue is that, for internal consumption's sake, our product owner can't say "the following features will be in 1.7, these other ones in 1.8, etc."

It sounds like what we want to do is to choose a set of features for a release, and then simply run iterations until either that feature set is complete, or the product owner decides that the output of some iteration is good enough to release. Naturally, of course, now that I've written this out, this is the by-the-book definition of how to do release management with scrum. Not quite sure how we got away from doing things that way.

I'm pretty convinced we should stick with the fixed-length iteration method, mainly for the reprioritization effect it has at planning. For every sprint where we had carryover, big chunks of that carryover were routinely deferred down the priority list from sprint to sprint, suggesting that we actually produced more product value than we would have if we had just finished each sprint straight out.

Scrum thoughts: Staggering sprint teams

2007-10-16T21:26:00.001-04:00

When you have two or more sprint teams dedicated to a product, I think there's an opportunity to stagger their efforts in the following way. Let's track a user story through two sprints. In the first sprint, the prototyping sprint, design, UX, and development cooperate to put together a working proposal for a user story. At the sprint review, if possible, the whole user story will be complete, where design, UX, and implementation are all consistent.

This is very important, because one of weaknesses of the traditional waterfall method is that design, UX, and implementation must be consistent before the software is finished. Obviously, the implementation will be influenced by the design and wireframes, the wireframes will be influenced by what is visually possible on the page, the design will be influenced by the functionality required by the wireframes, the design will be influenced by what is hard or easy to realize in HTML/CSS, the wireframes and functionality will be influenced by what is feasible to build in the backend (whether from an LOE standpoint or from a performance point of view). The waterfall method doesn't allow for the full richness of this interaction to help arrive at the "optimal" way to solve a user story from all points of view.

However, if we are considering features where we want more lead time than a month, then it may not be possible to get all the way to a finished feature in one sprint. Instead, it may be enough to get to a consistent prototype, where there are designs, wireframes, and mostly-working code that can be shown at the review. All of the creative output may not be integrated yet, but it should be close enough to know that it can be gotten to a consistent state.

This is where the next sprint comes into play: the finishing sprint, where the mockups are brought to production by the same team. Naturally, they will take product owner feedback from the review into consideration during the finishing sprint.

In the simplest case, let's say we have two scrum teams. We can stagger their sprints such that when one team is in the prototyping sprint, the other is in the finishing sprint; this way we're still able to release new functionality to production after every sprint while still enabling cross-functional development of larger features.

However, we can actually take things a step further: if, during pre-planning, we can roughly separate the stories into "small" (can be taken from start to finish in one sprint) and "large" stories, then we can actually mix things out in interesting ways, like:

	Sprint 1	Sprint 2	Sprint 3	Sprint 4
Team 1	SF	SF	SF	SF
Team 2	LP	LF	LP	LF
Team 3	SF	LP	LF	FP
Team 4	SF/LP	SF/LF	SF/LP	SF/LF

Here:

SF : finishing sprint for small user stories
LP : prototyping sprint for large user stories
LF : finishing sprint for large user stories

So here we have Team 1 kept full with small user stories every sprint. Teams 2 and 3 are kept full with large user stories, but staggered (note how Team 3 starts with a small finishing sprint so they have something to do during Sprint 1!). Then Team 4 operates under a model where some of their user stories are small, and some are large.

I think the only operational trickiness here, from a software development standpoint, is that any "LP" work needs to happen in a branch and not the main development trunk, because it will not be production-ready by the end of the sprint. When the product owners finally sign off on an LP prototype as being ready (it's always possible that the product owners will not like--or will require significant enough changes to--a user story solution, thus requiring a follow-on LP sprint), then the first task of the LF sprint would be to port the branch back into the trunk and continue on.

Scrum thoughts: scrums-of-scrums and daily scrums

2007-10-16T20:48:00.002-04:00

After planning, we dive right in to having daily scrums. We've gotten these to be pretty efficient (today we had a 10-person scrum finish in around 8 minutes). One key thing is starting on time--we charge people $1 if they are late to the scrum, and no one (not even the scrum master) is exempt. When we get enough saved up, we make a donation to charity (there was a thought that this should go to buy donuts for the team, but that felt a little like rewarding ourselves for being late). Our scrums are mainly a forum for people to schedule ad-hoc meetings with the people they are blocked on ("right after scrum in my office" being the most common meeting that gets scheduled). But running individual scrums is pretty well-documented and understood in the literature, I think.

As I mentioned in my post on sprint planning, we track individual burndowns on a daily basis -- how many hours does each person have left against each user story. At our scrum-of-scrums, which happens twice a week, the user story statuses for each team are put together to get an overall sprint status for the product. If you'll recall, we kept track of the global priorities of the user stories in our pre-planning session, so we can put this together in global priority order.

Then based on the time remaining in the sprint, we can again reassess confidence levels for user stories:

high confidence (green) : requires less than 50% of each person's remaining time
medium confidence (yellow) : requires less than 80% of each person's remaining time
low confidence (red) : requires more than 80% of someone's remaining time (possibly even getting into the "punt" range -- someone doesn't have enough time to finish their part this sprint, because they are overbooked)

The scrum-of-scrums is then a rebalancing effort. If one team falls behind, then you start to see a "striping" effect, where some of their user stories start falling in confidence, even though the surrounding user stories from the other teams stay the same. If you actually color these reports, it becomes pretty visually obvious. The rebalancing is all about trying to swap resources around such that the following principle holds:

No user story should have a lower confidence level than a user story with lower priority.

Ways that we rebalance include swapping tasks from one team to another (this is easier when the teams are vertical striped across architectural levels, as its likely that another team will still be able to do the task in question; however, a similar effect can sometimes be achieved by moving functionality from one architectural level to another (e.g. computing something in the middleware vs. the database)), or by rescoping user stories to make them easier.

Unfortunately, the way we are running sprints now ties the teams down a bit much to always make this flexibility possible. For example, a user story might say "put a Flash widget on the page that does X", and if we only have one Flash guy, we're pretty much stuck. If instead the user story said something more like: "we want a user to be able to see or do A,B,C on the page", then we might have some alternatives if the Flash guy gets overbooked.