codeartisan

Managing Software Development Flow

2010-11-27T09:17:00.009-05:00

This is the third post in a series about software development flow, which I'm describing as the conversion of customer requests (both for new features as well as bug reports) into working software. In the first post on this topic, we talked about how a software development organization can be viewed as a request processing engine, and how queuing theory (and Little's Law in particular) can be applied to optimize overall throughput and time-to-market (also called "cycle time"). In the second article, we revisited these same concepts with a more intuitive explanation and started to identify the management tradeoffs that come into play. This article will focus mostly on this final area: what metrics are important for management to understand, and what are some mechanisms/levers they can apply to try to optimize throughput?

I am being intentionally vague about particular processes here, since these principles can be applied regardless of particular process. I will also not talk about particular functional specialities like UX, Development, QA, or Ops; whether you have a staged waterfall process or a fully cross-functional agile team, the underlying theory still applies. Now, the adjustments we talk about here will have the greatest effect when applied to the greatest organizational scope (for example, including everything from customer request intake all the way to delivering working software), but Little's Law says they can also be applied to individual subsystems (for example, perhaps just Dev/QA/Ops taken together, or even just Ops) as well. Of course, the more you focus on individual parts of the system, the more likely you are to locally optimize, perhaps to the detriment of the system as a whole.

Managing Queuing Delay

As we've seen previously, at least some of the time a customer request is moving through our organization, it isn't being actively worked on; this time is known as queuing delay. There are different potential causes of queuing delay, including:

batching: if we group individual, independent features together into batches (like sprints or releases), then some of the time an individual feature will either be waiting for its turn to get worked on or it will be waiting for the other features in the batch to get finished
multitasking: if people can have more than one task assigned, they can still only work on one thing at a time, so their other tasks will be in a wait state
backlogs: these are explicit queues where features wait their turn for implementation
etc.

The simplest way to observe queuing delay is to measure it directly: what percentage of my in-flight items don't have someone actively working on them? If your process is visualized, perhaps with a kanban board, and you use avatars for your people to show what they are working on, then this is no harder than counting how many in-flight items don't have an avatar on them.

[ Side note: if you have also measured your overall delivery throughput X, Little's Law says:

NQ / N = XRQ / XR = RQ/R

In other words, your queuing delay RQ is the same percentage of your overall cycle time R as the number of queued items NQ is to the overall number of in-flight items N. So you can actually measure your queuing delay pretty easily this way.]

The primary mechanism, then, for reducing queuing delay is to reduce the number of in-flight items allowed in the system. One simple mechanism for managing this is to adopt a "one-in-one-out" mechanism that admits new feature requests only when a previous feature has been delivered; this puts a cap on the number of in-flight items N. We can then periodically (perhaps once a week, or once an iteration) reduce N by taking a slot out: in essence, when we finish one feature request, we don't admit a new one, thus reducing the number of overall requests in-flight.

Undoubtedly there will come a time when a high priority request shows up, and there would be too much opportunity cost to waiting for something else to finish up so it can be inserted. One possibility here is to flag this request as an emergency, perhaps by attaching a red flag to it on a kanban board to note its priority, and temporarily allow that new request in, with the notion that we will not admit a new feature request once the emergency feature finishes.

Managing Failure Demand

Recall that failure demand consists of requests we have to deal with because we didn't deliver something quite right the first time--think production outages or user bug reports. Failure demand can be quite expensive: according to one estimate, fixing a bug in production can be more than 15 times as expensive than correcting it during development. In other words, having work show up as failure demand is probably the most expensive possible way to get that work done. Cost aside, however, any amount of failure demand that shows up detracts from our ability to service value demand--the new features our customers want.

From a monitoring and metrics perspective, we simply compute the percentage of all in-flight requests that are failure demand. Now the question is how to manage that percentage downward so that we aren't paying as much of a failure demand tax on new development.

To get rid of existing failure demand, we need to address the root causes for these issues. Ideally, we would want to do a root cause analysis and fix for every incident (this is the long-term cheapest way to deal with the problems), but for organizations already experiencing high failure demand, this might temporarily drag new feature development to a halt. An alternative is to have a single "root cause fix" token that circulates through the organization: if it is not being used, then the next incident to arrive gets the token assigned. We do a root cause analysis and fix for that issue, then when we've finished that, the token frees up and we look for the next issue to fix. This approach caps the labor investment in root cause analysis fixing, and will, probabilistically, end up fixing the most common issues first. Over time, this will gradually wear away at the causes of existing failure demand. It's worth noting that you may not have to go to the uber root cause to have a positive effect--just fixing the issue in a way that makes it less likely to occur again will ultimately reduce failure demand.

However, we haven't addressed the upstream sources of failure demand yet; if we chip away at existing failure demand but continue to pile more on via new feature development, we'll ultimately lose ground. The primary cause of new failure demand is trying to hit an aggressive deadline with a fixed scope--something has to give here, and what usually gives is quality. There may well be reasons that this is the right tradeoff to make; perhaps there are marketing campaigns scheduled to start or contractual obligations that must be met (we'll save a discussion for how those dates got planned for another time). At any rate, management needs to understand the tradeoffs that are being made, and needs to be given the readouts to responsibly govern the process. "Percent failure demand" turns out to be a pretty simple and informative metric.

Managing Cycle Time

Draining queuing delay and tackling failure demand are pretty much no-brainers: they are easy to track, and there are easy-to-understand ways to reduce both. However, once we've gotten all the gains we can out of those two prongs of attack, all that is left is trying to further reduce cycle time (and hence raise throughput) via process change. This is much harder--there are no silver bullets here. Although there are any number of folks who will claim to "know" the process changes that are needed here, ranging from Agile consultants, to other managers, to the folks working on the software itself, the reality is that these ideas aren't really guaranteed solutions. They are, however, a really good source of process experiments to run.

Measuring cycle time is important, because thanks to queuing theory and Little's Law, it directly corresponds to throughput in a system with a fixed set of work in-flight. Furthermore, it is very easy to measure average cycle time; the data can be collected by hand and run through a spreadsheet with little real effort. This makes it an ideal metric for evaluating a process change experiment:

if cycle time decreases, keep the process change as an improvement
if cycle time increases, revert back to the old process and try something different
if cycle time is not affected, you might as well keep the change but still look for improvement somewhere else

Keeping "no-effect" process changes in place sets the stage for a culture of continual process improvement; it encourages experimentation if nothing else (and the cycle time measurements have indicated it hasn't hurt). Now, regardless of the experiment, it's important to set a timebox around the experiment so that we can evaluate it: "let's try it this way for a month and see what happens". New processes take time to sink in, so it's important not to run experiments that are too short--we want to give the new process a chance to shake out and see what it can really do. It's also worth noting here that managers should expect some of the experiments to "fail" with increased cycle time or to have no appreciable effect. This is unfortunately the nature of the scientific method--if we could be prescient we'd just jump straight to the optimized process--but this is a tried and true method for learning.

Now, process change requires effort to roll out, so a good question to ask here is how to find the time/people to carry this out. There's a related performance tuning concept here known as the Theory of Constraints, which I'll just paraphrase as "there's always a bottleneck somewhere." If we keep reducing work in-flight, and we have the end-to-end process visualized somewhere, we should be able to see where the bottleneck in the process is. The Theory of Constraints also says that you don't need to take on any more work than the bottleneck can process, which means, depending on your process and organizational structure, that we may find that we can apply folks both "upstream" and "downstream" of the bottleneck to a process change experiment without actually decreasing overall throughput. Furthermore, by identifying the bottleneck, we have a good starting point for selecting an experiment to run: let's try something that will alleviate the bottleneck (or, as the Theory of Constraints says, just move it elsewhere).

Conclusion

In this article, we've seen that managers really only need a few easy-to-collect metrics on an end-to-end software delivery flow to enable them to optimize throughput:

total number of items in-flight
number of "idle" in-flight items (not actively being worked)
number of in-flight items that are failure demand
end-to-end average cycle time

We've also identified several mechanisms, ranging from reducing work-in-progress to root cause fixes of failure demand, that can enable managers to perform optimizations on their process at a pace that suits the business. This is the classic empirical process control ("inspect and adapt") model that has been demonstrated to work effectively time and again in many settings, from the shop floor of Toyota factories to the team rooms of agile development organizations.

Intuitions about Software Development Flow

2010-11-25T06:37:00.011-05:00

In a previous post, I described the underlying theory behind optimizing the throughput of a software development organization, which consists of a three-pronged attack:

remove queuing delay by limiting the number of features in-flight
remove failure demand by building in quality up front and fixing root causes of problems
reduce average cycle time by experimenting with process improvements

In this article, I'd like to provide an alternative visualization to help motivate these changes. Let's start with some idealized flow, where we have sufficient throughput to deal with all of our incoming customer requests. Or, if we prefer, the rate at which our business stakeholders inject requests for new features is matched to the rate at which we can deliver them.

Queuing Delay

Now let's add some queuing delay, in the form of some extra water sitting in the sink:

If we leave the faucet of customer requests running at the same rate that the development organization can "drain" them out into working software, we can understand that the level of water in the sink will stay constant. Compared to our original diagram, features are still getting shipped at the same rate they were before; the only difference is that now for any particular feature, it takes longer to get out the other side, because it has to spend some time sitting around in the pool of queuing delay.

Getting rid of queuing delay is as simple as turning the faucet down slightly so that the pool can start draining; once we've drained all the queuing delay out, we can turn the faucet back up again, with no net change other than improved time-to-market (cycle time). There's a management investment tradeoff here; the more we turn the faucet down, the faster the pool drains and the sooner we can turn the faucet back up to full speed at a faster cycle time. On the other hand, that requires (temporarily) slowing down feature development to let currently in-flight items "drain" a bit. Fortunately, this is something that can be done completely flexibly as business situations dictate--simply turn the knob on the faucet as desired, and adjust it as many times as needed.

Failure Demand

We can model failure demand as a tube that siphons some of the organization's throughput off and runs it back into the sink in the form of bug reports and production incidents:

Our sink intuition tells us that we'll have to turn the faucet down--even if only slightly--if we don't want queuing delay to start backing up in the system (otherwise we're adding new requests plus the bug fixing to the sink at a rate faster than the drain will accommodate). Now, every time we ship new features that have bugs or aren't robust to failure conditions (particularly common when rushing to hit a deadline), it's like making the failure demand siphon wider; ultimately we're stealing from our future throughput. When we fix the root cause of an issue, it's like making the failure demand siphon narrower, and we not only get happier customers, but we reclaim some of our overall throughput.

Again, there are management tradeoffs to be made here: fixing the root cause of an issue may take longer than just triaging it, but it is ultimately an investment in higher throughput. Similarly, rushing not-quite-solid software out the door is ultimately borrowing against future throughput. However, it's not hard to see that if we never invest in paying down the failure demand, eventually it will consume all of our throughput and severely reduce our ability to ship new features. This is why it is important for management to have a clear view of failure demand in comparison to overall throughput so that these tradeoffs can be managed responsibly.

Process Change

The final thing we can do is to improve our process, which is roughly like taking all the metal of the drain pipe (corresponding loosely to the people in our organization) and reconfiguring it into a shorter, fatter pipe:

This shows the intuition that if we focus on cycle time (length of the pipe) for our process change experiments, it will essentially free up people (metal) to work on more things (pipe width) at a time, thus improving throughput. There is likewise a management tradeoff to make here: process change takes time and investment, and we'll need to back off feature development for a while to enable that. On the other hand, there's simply no way to improve throughput without changing your process somehow; underinvestment here compared to our competitors means eventually we'll get left in the dust, just as surely as failing to invest cash financially will eventually lead to an erosion of purchasing power due to inflation.

Summary

Hopefully, we've given some intuitive descriptions of the ways to improve time-to-market and throughput for a software development organization to complement the theory presented in the first post on this topic. We've also touched on some of the management tradeoffs these changes entail and some of the information management will need to guide things responsibly.

Credits: Sink diagrams are available under a Creative Commons Attribution-ShareAlike 2.0 Generic license and were created using photos by tudor and doortoriver.

How to Go Faster

2010-11-19T21:11:00.014-05:00

Ok, I'm going to tell you how to make your software development organization go faster. I'm going to tell you how to get more done without adding people while improving your time to market and increasing your quality. And I'm going to back it all up with queuing theory. [ By actually explaining the relevant concepts of queuing theory, not just by ending sentences with "...which is obvious from queuing theory", which is usually a good bluff in a technical argument being had over beers. Generally a slam dunk in mixed technical/non-technical company. But I digress. ]

An Important Perspective

It's worth saying that this article assumes you've figured out how to deliver software incrementally somehow, even if that's just by doing Scrumfall. The point is that you are familiar with breaking your overall feature set down into discretely deliverable minimum marketable features (MMFs), user stories, epics, tasks, and the like. If you have any customers, you are probably also familiar with production incidents and bugs, which are also discrete chunks of work to do. Now, here's the important perspective:

Your software development organization is a request processing system.

In this case, the requests come from customers or their proxies (product managers, etc.), and the organization processes the request by delivering the requested change as working software. This could end with a deployment to a live website, publishing an update to an app store, or just plain cutting a release and posting it somewhere for your customers to download and use. At any rate, the requests come into your organization, the software gets delivered, and then the request is essentially forgotten (closed out). Now, looking at your organization this way is important, because it means you can understand your capacity for delivery in terms borrowed from tuning other request processing systems (like websites, for example) for performance and scale. Most importantly, though, is that this mysterious branch of mathematics called queuing theory applies to your organization (just as it applies to any request processing system).

A Little Light Queuing Theory

One of the basic principles in queuing theory is Little's Law, which says:

N=XR

where N is the average number of requests currently being processed by the system, X is the transaction rate (requests processed per unit time), and R is the average response time (how long it takes to process one request). In a software development setting, R is sometimes called cycle time.

To put this in more familiar terms, suppose we have a walk-in bank with a number of tellers on staff. If customers arrive at an average rate of one person per minute (X) and it takes a teller an average of 2 minutes to serve a customer (R) then Little's Law says, on average, that we'll have XR = 1(2) = 2 tellers busy (N) on average at any given point in time. We can similarly flip this around: if we have 3 tellers on staff, what's the maximum average customer arrival rate we can handle?

X = N/R = 3/2 = 1.5 customers per minute

Ok, the last thing we need to talk about is: what happens if we suddenly get a rush of customers coming in? Anyone who has entered a Starbucks or visited Disneyland knows the answer to this: a line forms. (The time a customer spends waiting in line is known as "queuing delay" if you want to get theoretical about it.) Let's go back to our bank. Suppose we just have 5 people suddenly walk in all at once, in addition to our regular arrival of one person per minute. What happens? Well, we get a line that is 5 people long. But if we only have 2 tellers on staff, then people come off the line at exactly the same rate that new people are entering from the back, which means: the line never goes away and always stays 5 people long.

What does this look like from the customers' point of view? Well, we know they'll spend 2 minutes with the teller once they get up to the front of the line, and we know that it will take 5 minutes to get to the front of the line, so my average response time is:

R = RV + RQ = 2 + 5 = 7

where RV is the "value added time" where the request (customer) is actually getting worked on/for, and RQ is the amount of time spent waiting in line (queuing delay). Now we can see that on average, we'll have:

N = XR = X(RV + RQ) = 1(2 + 5) = 7

people in the bank on average. Two people at the tellers, and five people waiting in line. We all know how frustrating an experience that is from the customer's point of view. Now, let me summarize this section (if you didn't follow all the math, don't worry, the important thing is that you understand these implications):

If you try to put more requests into a system than it can handle, lines start forming somewhere in the system.
If the request rate never falls below the system's max capacity, the lines never go away.
Time spent waiting in a line doesn't really serve much useful purpose from the customer's point of view.

Software development as customer request processing

If your experience is anything like mine, there is an infinite supply of things the business stakeholders would like the software to do, which means the transaction rate X can be as high as we actually have capacity for. This means one of the primary goals of the organization is figuring out how to get X as high as possible so we can ship more stuff. At the same time, we're also concerned with getting R as low as possible, since this represents our time-to-market and can be a major competitive advantage. If we can ship a feature in a week but it takes our competitors a month to get features through their system, who's more reactive? Every time the competition throws up a compelling feature, we can match them in a week. Every time we ship a compelling feature, it takes them a month to catch up. Who's going to win that battle?

Now, one of the tricky things here is that software development is often far more complicated than our example bank with tellers, since we tend to staff folks with different skillsets. If I have a team of one graphic designer, three developers, a tester, and a sysadmin, it's really hard to predict how long it will take that team to ship a feature, because they will have to collaborate. If I want to hire someone to help them, is it better to hire another tester or another designer? Probably I can't tell a priori, because it depends on the nature of the features being worked on, and it's really hard to measure things like "this user story was 10% design, 25% development, 50% testing, and 15% operations." Nonetheless, we can look at this from another point of view, which is that I have a fixed number of people in the organization, and each person can only be working on one thing at a time (just as a teller can only actively serve one person at a time), and they are probably (hopefully) collaborating on them.

This means the maximum number of things you can realistically be actively working on is less than the number of people in the organization.

If we have more things in flight than that, we know at least some of the time those things are going to be sitting around waiting for someone to work on them (queuing delay). Perhaps they are sitting on a product backlog. Perhaps they are simply marked "Not Started" on a sprint taskboard. Perhaps they are marked "Done" on a sprint taskboard but they have to wait for a release to be rolled at the end of the sprint to move onwards towards production or QA. As we saw above, this queuing delay doesn't increase throughput, it just hurts our time-to-market. Why would we want that?

First optimization: get rid of queuing delay

Ok, as we saw above, we know that the total response time R consists of two parts; actual value-adding work (RV) and queuing delay (RQ). Typically, it's really hard and time consuming to try to measure these two pieces separately without having lots of annoying people running around with stopwatches and taking furious notes. Fortunately, we don't have to resort to that. It is really easy to measure R overall for a feature/story: mark down when the request came in (e.g. got added to a backlog) and then mark down when it shipped. Simple.

Now, let's think back to our bank example where we had a line of people. Most software development organizations have too much in flight, and they have lines all over the place inside, many of which aren't even readily apparent because that's just "the way we do things around here." Lines are bad. Now, we know the only way to drain these queues is if the incoming feature request rate is less than the rate at which we ship them. Sometimes we can try hiring more "tellers", but in a recession that's not always an option. Instead, for many organizations, the best option is admission control, which is to say that we don't take on a new request until we've shipped one out the other side. You can think of this as having a certain number of feature delivery "slots" available, and you can't start something new until you've freed up a slot. This at least prevents you from having your lines get any bigger.

In order to drain the lines out of the system, the easiest thing to do is to periodically retire a slot after it ships. In other words, don't let something new in just that once. This will reduce the overall number of things in flight, and since presumably everyone is still working hard, what we've just gotten rid of must be queuing delay. Magic! So we can just keep doing this and draining queuing delay out of the system, improving our time to market all the time, without necessarily having to change anything else about the way we do things. When do we stop? We stop once we have people standing around not doing anything. At that point, all the queuing delay is out of the system (for now), and we know that we're at a level where all of our "tellers" are busy. To summarize:

We can remove queuing delay from our delivery process simply by limiting and reducing the amount of work in-flight; this improves time-to-market without having to change anything else.
We can keep doing this until people run out of things to work on; at that point we've squeezed all the queuing delay out.

Second optimization: reduce failure demand

The next thing to realize is that the N things we have in flight actually come in two flavors: value demand and failure demand. In our case, value demand consists of requests that create value for the customer: i.e. new and enhanced features. Failure demand, on the other hand, consists of requests that come from not doing something right previously. These are primarily things like website outages (production incidents), bug reports from users, or even support calls from users asking if you've fixed the problem they previously reported. If you have someone collecting these, then these are requests that your organization as a whole has to deal with. On the other hand, for each request of failure demand, someone is busy triaging/fixing it when then could be creating new value. In other words:

N = NV + NF

where NV is value demand and NF is failure demand. Or, if we look at things this way:

X = N/R = (NV + NF)/R = NV/R + NF/R

we can see that the failure demand is stealing a portion (NF/R) of our organization's throughput! This is, incidentally, why spending extra energy on quality up front results in lower overall costs (as Toyota showed); failure demand essentially requires rework.

This means that another way to improve overall throughput of the organization is to reduce failure demand, reclaiming that portion of your throughput that's getting siphoned off. One way to do this involves figuring out how to "build quality in" on new development, but since software development is a creative process (different every time for every feature), it's not possible to actually completely prevent bugs. That said, there are many techniques like test-driven development and user experience testing that can help improve quality. The other way to reduce failure demand involves vigorously fixing root causes of failure as we experience them. In other words, when we fix a problem for a customer, we should fix it in a way that prevents that type of problem from ever occurring again, for any customer. This keeps overall failure demand down by preventing certain classes of it, thereby reserving that precious organizational throughput for delivering new value. To summarize this section:

Improve value delivery capacity by reducing failure demand (production incidents and bug reports).
The cheapest way to reduce failure demand is by building in quality up-front.
When serving a failure demand request, we can reduce overall failure demand by also fixing the root cause of the problem.

Final optimization: cycle time reduction

Ok, now we've gotten to the point where RQ = 0 (or near zero), so R = RV. Now at this point, let's look back at Little's Law:

N = XR

We've already established via draining out our queuing delay in the first phase what our target N is (number of requests in-flight). But we still want to ship more with the same number of people; we want X to go up. But recall that:

X = N/R

If our N is fixed due to the number of people we have on staff, then the only way to increase throughput is to reduce R. Now is where we start to look at process changes and automation. How do we make it so that it takes people less time to handle a request? Focusing on this improves not only time to market but also overall throughput. And furthermore, if we are measuring R over time, we have an easy way to do this: change the process in a way you think will help, and then measure if R went down or not. If it didn't help, try something else. If it made things worse, go back to the old way. Rinse, repeat. The things to try are going to be different for every organization, and one of the best sources of ideas will be the folks actually doing the work. But this doesn't require any kind of high-tech tracking software -- post-it notes on walls with the start and end dates written on them are more than sufficient to measure R and carry these experiments out.

As failure demand and queuing delay are squeezed out of the system, the only way to improve throughput is by reducing response time.
Response time can only be reduced by process changes.
By measuring response time, we have a convenient experimental lab to understand if process changes help or not.

Say, haven't I heard this all before?

Well, yes. You may have heard pieces of this from all sorts of places. The feature "slots" we were talking about before as a means to limit "work-in-progress" (WIP), and are often called kanban. The notion of continually adapting your process to improve it is a tenet of Scrum. Test-driven development and pair programming are methods from Extreme Programming (XP) of building in quality up front. Failure demand is sometimes called out as a form of technical debt, and the list goes on and on.

Hopefully what I've done here, though, without putting a name on any kind of methodology, is explain why all these things are good ideas (or are good ideas to try). Ultimately, practices won't help unless they do one of three things:

drive out queuing delay (RQ);
reduce value-adding response time (RV); OR
reduce failure demand (NF/R)

In general, the easiest way to do these for an organization is:

reduce the number of things in-flight
aggressively beat back failure demand by fixing root causes and building in quality up-front
measure response (cycle) time and improve via process experimentation

Fortunately, all of those things are very, very easy to measure. If you can mark a request as either value or failure demand, if you can count the number of things in-flight, and if you can measure the time between starting something and shipping it, that's all you need.

Update: See the next post on this topic for a more intuitive motivation of the theory presented in this article.

Tales of Test-Driven Development

2010-10-29T21:11:00.002-04:00

Inspired by a talk about Clojure given by Rich Hickey at Philly Emerging Tech earlier this year, I've been toying with building a Java library of pure (immutable) data structures, starting with the Map implementation based on Phil Bagwell's Hash Tries. Yes, I know I could probably just figure out how to use them straight out of Clojure by interoperating, but that would deprive me of an interesting coding exercise.

At any rate, I hadn't really gotten around to doing this in any great detail yet, and as it turns out, I'm glad I didn't. I was fortunate to be able to attend a refactoring and test-driven development (TDD) class taught by Bob Martin this week, and one of the code examples we ran through was the "Bowling Game" of writing an algorithm to score ten frames of bowling. Prior to developing this with a TDD approach, we identified a pretty simple object-oriented design including things like Games, Frames, Rolls, TenthFrames, etc. Yet when we actually got down to it, it turned out we just needed a pretty simple algorithm built into the single Game class--ultimately a much simpler design.

Rewinding to the pure hashmaps: I had previously been thinking about how to decompose the internal datastructure for the hash tries into classes like InternalNodes and LeafNodes that would implement a common TrieNode interface, and then mark all of those things as package private so I could hide all the messy implementation details from the client behind a PureHashMap facade. Hoo boy.

Instead, over the course of a very few hours today, I instead took the following approach: first, I used TDD to develop a brain-dead simple implementation of a PureHashMap using real HashMaps as the backing store, but cloning them on modification. Didn't even make an attempt at the Hash Trie implementation. Wouldn't work well at all on large data sets, but it did allow me to develop a set of unit tests that documented the required functional behavior of a PureHashMap, and it didn't really take long at all.

Next, I did something crazy: I completely threw away the simple implementation of PureHashMap, breaking all the tests. Then I started hacking the hash trie implementation in there until I could get all the tests to pass one by one. Now, this was ugly, cut-n-pasted, massively high cyclomatic complexity code--I shudder to think of it. But it really didn't take too long to get that working either, with the tests as a guide. Finally, as any TDD afficionado would know, once I had all my tests working again, I was able to easily but mercilessly refactor until it was all cleaned up.

The end result was something far better than I could have imagined: a relatively understandable PureHashMap hash trie implementation in a single class file with 100% unit test coverage, built in less than an afternoon. That's powerful stuff. Thanks, Uncle Bob.

The Power of Visualizing Iterative Waterfall

2010-09-26T07:52:00.005-04:00

We're going through a process mapping exercise at work just to try to understand how we get things done. Now, we are running what I would describe as "scrumfall"; doing Scrum for development but having that sit inside a traditional waterfall process. The waterfall is run iteratively and pipelined, although the degree of true pipeline isn't what most people think it is, due to developers having to support upstream and downstream activities like backlog grooming and addressing bugs in QA. I thought I would work through the exercise to try to define the value stream that our features actually experience.

Recorded	user story appears on a backlog
Defined	user story has acceptance criteria and estimate
Prioritized	user story has been assigned a priority/rank
Committed	user story has been pulled into a sprint
Coded	user story has been marked 'complete' in the sprint
Accepted	user story has been shown and accepted in a sprint review
Released	user story has been included in a versioned release
Tested	enclosing release has achieved an acceptable quality level
Approved	enclosing release has been approved for launch (go/no-go)
Deployed	enclosing release has been deployed to production

Several of Scrum's standard meetings (plus some other common ones) show up here: backlog grooming moves stories from "recorded" to "defined", sprint planning moves stories from "prioritized" to "committed", daily scrum moves stories from "committed" to "coded", and the sprint review moves stories from "coded" to "accepted".

Just having laid it out and thinking about it, some observations:

we ask our product owners and other stakeholders to sign off on a particular user story twice, once at the sprint review, and once at the go/no-go meeting.
user stories that may well be production-ready upon reaching "coded" get batched and bound to surrounding stories and thus become deployment dependent on them afterwards, even though they may not be functionally dependent on them
interestingly, even though good user stories are supposed to be independent of one another (the "I" in INVEST), we nonetheless batch them together into sprints and treat them as a unit
we don't have a good way to understand what happens to stories that don't get completed in a sprint, or bugs that are deemed non-launch blockers, or production incidents

Another thing as we think about batch size is whether a two week sprint iteration is actually tied to any relevant process capability metrics. For example, interesting metrics to consider here are some things that are per-batch (sprint or release); these are things that take roughly the same amount of time whether they contain one story or one hundred:

how long does it take us to produce a release?
how long does it take us to deploy a release?
how long does it take us to run a full regression test?
what is the lead time for scheduling a meeting with all the necessary folks in it?

And then there are some activities that are dependent on the complexity of a particular story:

how long it takes to define acceptance criteria for the story
how long it takes to code the story
how long it takes to define and update test cases for the story
how long it takes to discuss the story and determine if it was acceptably implemented

Batching makes sense if the organization's overall throughput bottleneck is on a batch-size-independent step, in which case, sizing the batch so that it runs in cadence with the cycle time of the bottleneck will maximize throughput. To make that more concrete, let's say we only have certain deployment windows available and can only do a deployment once a week; if this is the slowest part of our process, then we should take batches of work in a way so that upstream steps produce a deployable release once a week. Or, if the slowest part is running a full regression of manual tests over three days, then again, we should take batches that can be finished in three days. Perhaps the product owner is only available once a month to carry out sprint planning or sprint review; then we should batch at a month.

It might seem weird that the calendar of your product manager might be the bottleneck in your software development process, or that it makes sense to roll a release of completed work every three days, but that's queuing theory for you. Optimizing an overall system's throughput means organizing the work according to the current bottleneck's constraints (even if that means non-bottleneck parts might not be locally optimized) and/or moving the constraint elsewhere in the system (Theory of Constraints).

Interestingly, putting the entire workflow up on a kanban board would make a lot of this very obvious, even if all we did was put up WIP limits corresponding to obvious limitations (I can only deploy one release at a time, and I can only test as many releases as I have QA environments, etc.). The great thing about kanban-style development is that you don't have to change your process to start using it; you just model your current one, visualize it, and then watch what happens. You probably have all the information needed to track the metrics that matter, although you may have to start writing down times when various emails pass through your system (like whether the release went out, or whether the new build got deployed to QA, etc.).

However, to me, the most powerful reason to start visualizing the flow is that it shows you exactly what parts of your process you should change, and when. There's nothing like being able to show a product manager that their availability is driving overall throughput to encourage spending more time with the team. Or being able to show a development manager that the amount of time being spent doing bugfixing rework in QA is the bottleneck--encouraging practices like TDD. In other words, being able to make an empirical case for the potential use of Agile practices that aren't currently in place, and then being able to show that they worked. This is a good way to bring about an Agile evolution grounded in facts relevant to the current organization and not just based on opinion or philosophy.

Testable System Architecture

2010-08-31T08:15:00.004-04:00

At work we were having a discussion about how we wanted to do SSL termination for a particular web service. We had narrowed the possibilities down to doing hardware SSL termination in our load balancer or doing software SSL termination in an Apache layer sitting in front of our web apps.

During the course of the conversation, we talked about factors like performance (would there be a noticeable effect on latency), capacity (were we already CPU bound on the servers that would run the Apaches), maintainability (is it easier to update configs on a single load balancer or to script config changes across a cluster with 40+ servers), cost (how much does the SSL card cost), and scalability (will we be able to expand the solution out to higher traffic levels easily).

I think this was a pretty typical example of taking a reasoned approach to system design and trying to cover all the potential points of view. However, it ended up that we left a big one off: testability.

The business rules about which URLs need to be SSL terminated and which ones don't (or shouldn't) need to be encoded somewhere, and we'd already ruled out doing the SSL termination in the application itself for other reasons, so that means they'd be encoded in either a load balancer config or an Apache config. Which one of these is easier to get under automated test on a developer workstation? For an agile shop where quality and time-to-market are of primary importance, this is a question we can't forget to ask when designing our system architecture.

RESTful Refactor: Combine Resources

2010-08-13T22:14:00.007-04:00

I've been spending a lot of time thinking about RESTful web services, particularly hypermedia APIs, and I've started to discover several design patterns as I've begun to play around with these in code. Today, I want to talk about the granularity of resources, which is roughly "how much stuff shows up at a single resource". Generally speaking, RESTful architectures work better with coarser-grained resources, i.e., transferring more stuff in one response, and I'll walk through an example of that in this article.

Now, in my previous article, I suggested taking each domain object (or collection of domain objects) and making it a resource with an assigned URL. While following this path (along with the other guidelines mentioned) does gets you to a RESTful architecture, it may not always be an optimal one, and you may want to refactor your API to improve it.

Let's take, for example, the canonical and oversimplified "list of favorite things" web service. There are potentially two resource types:

a favorite thing (/favorites/{id})
a list of favorite things (/favorites)

All well and good, and I can model all sorts of actions here:

adding a new favorite: POST to /favorites
removing a favorite: DELETE to the specific /favorites/{id}
editing a favorite: PUT to the specific /favorites/{id}
getting the full list: GET to /favorites

Fully RESTful, great. However, let's think about cache semantics, particularly the cache semantics we should assign to the GET to /favorites. This is probably the most common request we'd have to serve, and in fact it ought to be quite cacheable, as in practice (as with a lot of user-maintained preferences or data) there are going to be lots of read accesses between writes.

There's a problem here, though: some of the actions that would cause an update to the list don't operate on the list's URL (namely, editing a single entry or deleting an entry). This means an intermediary HTTP cache won't invalidate the cache entry for the list when those updates happen. If we want a subsequent fetch of the list by a user to reflect an immediate update, we either have to put 'Cache-Control: max-age=0' on the list and require validation on each access, or we need the client to remember to send 'Cache-Control: no-cache' when fetching a list after an update.

Putting 'Cache-Control: max-age=0' on the list resource really seems a shame; most RESTful APIs are set up to cross WAN links, and so you may be paying most of the latency of a full fetch that returned a 200 OK even if you are getting a 304 Not Modified response, especially if you have fine-grained resources that don't have a lot of data (and a textual list of 10 or so favorite items isn't a lot of data!).

Requiring the client to send 'Cache-Control: no-cache' is also problematic: the cache semantics of the resources are really supposed to be the server's concern, yet we are relying on the client to understand something extra about the relationship between various resources and their caching semantics. This is a road that leads to tight coupling between client and server, thus throwing away one of the really useful properties of a REST architecture: allowing the server and client to evolve largely independently.

Instead, let me offer the following rule of thumb: if a change to one resource should cause a cache invalidation of another resource, maybe they shouldn't be separate resources. I'll call this a "RESTful refactoring": Combining Resources.

In our case, I would suggest that we only need one resource:

the list of favorites

We can still model all of our actions:

adding a new favorite: PUT to /favorites a list containing the new item
removing a favorite: PUT to /favorites a new list with the offending item removed
editing a favorite: PUT to /favorites a list containing an updated item
getting the full list: GET to /favorites

But now, I can put a much longer cache timeout on the /favorites resource, because if a client does something to change its state, it will do a PUT to /favorites, invalidating its own cache (assuming the client has its own non-shared/private cache). If the resource represents a user-specific list, then I can probably set the cache timeout considering:

how long am I willing to wait for another user to see the results of this user's updates?
if the same user accesses the resource from a different computer, how long am I willing to allow those two views to stay out of sync? (bearing in mind that the user can usually, and pretty intuitively, hit refresh on a browser page that looks out of date)?

Probably these values are a lot larger than the zero seconds we were using via 'Cache-Control: max-age=0'. When you can figure out how to assign longer expiration times to your responses, you can get a much bigger win for performance and scale. While revalidating a cached response is probably faster than fetching the resource anew, not having to send a request at all to the origin is waaaaaaay better.

The extreme case, here, of course, would be a web service where a user could just get all their "stuff" in one big blob with one request (as we modelled above). There are many domains where this is quite possible, and when you factor in gzip encoding, you can start to contemplate pushing around quite verbose documents, which can be a big win assuming your server can render the response reasonably quickly enough.

Thoughts on Hypermedia APIs

2010-08-11T21:34:00.010-04:00

The REST architectural style is defined in Roy Fielding's thesis, primarily chapter 5, where the style is described as a set of architectural constraints. A quick summary of these constraints is:

client-server

The system is divided into client and server portions.

stateless

Each request from client to server must contain all of the information necessary to understand the request.

cache

Response data is implicitly or explicitly marked as cacheable or non-cacheable.

uniform interface

All interactions through the system happen via a standard, common interface. This is achieved by adhering to four sub-constraints:

identification of resources: Domain objects are assigned resource identifiers (e.g. URIs)
manipulation via representations: Actions occur by exchanging representations of current or intended resource state.
self-descriptive messages: Messages include control data (e.g. cache-related), resource metadata (e.g. alternates), and representation metadata (e.g. media type) in addition to a representation itself.
hypermedia as the engine of application state: Clients move from one state to the next by selecting and following state transitions described in the current set of representations.

layered system

Components can only "see" the component with which they are directly interacting.

code-on-demand (optional)

Clients can by dynamically extended by downloading and running code.

Achieving a RESTful architecture with XHTML

Mike Amundsen proposed using XHTML as a media-type of choice for web APIs rather than the ubiquitous Atom (or other application-specific XML) or JSON representations commonly seen. By using XHTML profiles, we are able to define the semantics of the data contained within a particular document, as well as the semantics of contained link relations and form types.

Now, let's throw a few simple rules into the system:

all domain objects (including collections of domain objects) are resources and get assigned a URL
beyond an HTTP GET to the API's "home page", a client simply follows standard XHMTL semantics from returned documents; namely, doing a GET to follow a link, and constructing a GET or POST request by filling out and submitting a form.
retrieval (read) of resource state should be accomplished by GET, and modification of resource state should happen with POST (via a form).

Interestingly, this means that in addition to programmatic clients being able to parse XHTML (as a subset of XML) and apply standard XHTML semantics for interactions, it is possible for a human to use a browser to interact with the resources (or, as my colleague Karl Martino put it, "you can surf an API!").

Evaluation

So how well does this match up against the REST constraints? By leveraging HTTP directly as an application protocol, we can get a lot of constraints for free, namely: client-server, statelessness, caching, layered system, and self-descriptive messages.

Now, we also get a uniform interface, because all of our domain objects are modelled as resources with identifiers, reads are accomplished by retrieving XHTML documents as representations, and writes are accomplished by sending form-encoded inputs as representations. Finally, because a client accomplishes its goals by "clicking links and submitting forms", the hypermedia features of XHTML let us model the available state transitions to the client, who can then select what to do next and know how to follow one of the available transitions. Also, because an update to a resource is modelled as a PUT to the same URL we would use to GET its state, this plays nicely and naturally with standard HTTP/1.1 cache semantics (invalidation on write-through).

Finally, we're not using code-on-demand, in our case, although we could include Javascript with our XHTML representations to provide additional functionality for that human "surfing" our API, even if a programmatic client would ignore the Javascript. However, code-on-demand is listed as an optional constraint anyway.

Coming soon...

This is an intentionally high-level post that I'm intending will be the first in a series of posts that go over specific examples and examine some practical considerations and implementation patterns that are useful. Hopefully, we'll also be able to illustrate some of the architectural strengths and weaknesses that the REST architectural style is purported to have. Stay tuned!

Why Lab Week is So...Awesome

2010-03-26T20:12:00.004-04:00

At work, once a quarter, we have "lab week": folks are allowed to form groups and work on self-directed projects. We usually finish up at the end of the week with a "science fair" where folks set up posters and demos of what they've worked on for the week (we even had cookies and lemonade at today's science fair!). I am always amazed at the amount of innovation and progress that comes out of these weeks; in some ways it outshines what our organization manages to do over the rest of the quarter. In this post I'd like to reflect a bit on what makes lab week so awesome.

Merit-based Project Ideas

Where we work, you have to actually recruit for lab week. This means you typically have to pitch your idea if you have one to enough people to get them to join your team. We actually set up special lunchtime meetings just for "Pitch Day". Ultimately this means that the projects that get worked on are the most innovative--because those are the most exciting ones and the easiest to recruit for. This is wisdom of the crowds at its finest; rather than having a select and small group of management identify a roadmap, it's a free-for-all where anyone can submit an idea and the best ones attract teams and get worked on. [Ed: this is not to say that management isn't needed to guide the execution and selection of ideas, just that they need not be the only source for idea generation]

Self-Selecting Teams

Again, due to the recruiting nature of lab week, groups are self-forming. As I think about it, it's amazing how well the group dynamics and team composition work out. I just re-read the section of Malcolm Gladwell's Tipping Point where he talks about Dunbar's number of 150: in a group smaller than that, you know your relationship to everyone as well as everyone's relationship to each other. In practice, this means when you are recruiting for lab week, you are consciously and unconsciously choosing folks that will bring the needed skills and experience to your project in a way that's compatible with the rest of the group.

When I look at it this way, it's not surprising that lab week teams gel very quickly and immediately start working together well. When you have responsibility for deciding who you work with, you end up wanting to work with your team. The group dynamics just sort themselves out effortlessly. Apparently Gore (makers of Gore-Tex) organize their high-tech development in this fashion.

Ownership and Buy-in

With a self-selected team and a self-selected project, folks on a lab week team are implicitly engaged in what they do, because it is their work. They own it, front to back, and pour their effort into it. You can see the pride in the demos, in the cute homemade posters (it does bear a striking resemblence to a stereotypical grade school science fair!).

On the well-known Gallup Q12 survey for how engaged your employees are, lab week covers: knowing what's expected of you (set by you!), having the needed raw supplies (often because the work is chosen with an eye to the possible), having an opportunity to do what you do best (self-selected projects), having your opinions count (recruiting, self-organizing teams), having dedicated teammates (self-selecting teams with ownership), having opportunities to learn (the whole point of lab week). That's six of the twelve right there. No wonder people willingly put in overtime on their lab week projects.

Timeboxed Exploration

Lab week lasts exactly a week. You don't have time to fully productionize what you do, and you have to focus. Ultimately this forces you to separate out all the chaff and focus on the real core of your idea and just deliver that, because that's all the time you have. This is the XP notion of Design Spikes, but in reality it's the Pareto Principle in full effect: do the 20% of the work that has the 80% impact.

Looked at it in another way, because the work effort is limited to a week, it is a way for the company to do rapid exploration with minimal risk or expense. I'd wager the company gets way more value in terms of idea creation during lab week than they miss out on from not doing "normal" work. It's clear the executives agree, because after each lab week, they agree to have the next one!

Thinking Outside the Box

Anything goes during lab week; this is a chance for folks to play with new technologies (it seems there is never a shortage of new technologies or of engineers who want to tinker with them) or practices. Our group used CRC Design Cards and did full-on TDD for our project, a first for many of us, and while I think we built a pretty cool project, the main benefit (echoed by group members) was what we learned about development this week. We ran a Cobertura report at the end of lab week and discovered we had 91% branch coverage in our code and an average cyclomatic complexity of 3 (without measuring during the week or even shooting for particular measures here). On a lab week project, no less. Wild.

In many cases, I think the science fair shouldn't be "What we built during lab week" so much as a presentation of "What we learned during lab week", which I suspect is actually the majority of the value offered to the company and the participants.

Permission to Experiment

Finally, it is clear that lab week is a no-pressure situation. There are no contracts to fulfill, no launch deadlines to meet (other than the science fair, I guess!), and you don't have to get approval from any more people than it takes to form a team. Almost anything goes (I have heard that underwater basketweaving is off limits, but not much else). If it doesn't work out, no problem, you go back to your "day job" next week, until the next lab week rolls around. There is nothing like this kind of no-risk environment (even the name "lab week" is suggestive) to foster creativity.

In Summary

Someone remarked at how much positive energy permeated the science fair. It was fun, and there were a lot of really cool ideas. People get into lab week where I work, and it makes the experience...awesome.

Agile Architecture Kanban

2010-03-19T22:42:00.005-04:00

We've recently spun up a new software architecture group at work, and at least some of what the architects are expected to do is provide "consulting" services: providing feedback on technical designs and approaches, doing technical research, providing technical opinions to product managers, etc. Since many of these are similarly sized, and "cycle time" for getting a response to our clients is an important metric, we opted to manage this work using a kanban system.

After a month-long iteration, we stopped to take a look at some of the data we had collected. We were able to produce a statistical process control chart, indicating our cycle time in business days (measuring the time between when a customer asked for something to be added to our consulting backlog and the time when we finished it), something like this one:

This shows our average cycle time was around 6 days, and that our process was under statistical control; all samples were less than the upper control limit (red line) at 11 days (3 standard deviations above the average). This means that we had a relatively predictable process. Now, at the same time, we were able to produce a cumulative flow diagram, like this one:

which showed the number of consulting "stories" in each state of the workflow. One of the things we were able to derive is the average arrival rate for the stories, by finding the slope of the line between the starting and ending points on the "ready" line. We were also able to find our average throughput by finding the similar slope between the starting and ending points of the "done" line. What we found (and which you can see on the graph), was that the request rate was higher than our throughput (by about 0.2 stories per day), which resulted in a slowly but persistently growing backlog. Now, we happened to measure our average cycle time about halfway through the month, and found that it was 4.5 instead of 6 back then. In the ten business days between measurements, our average cycle time went up by around the amount our backlog length grew, as predicted by the difference between our customers' request rate and our service rate.

It would appear even architects are subject to queuing theory.

Going forward, in order to remain responsive to our clients (many of our engineering teams run two week sprints, so we wanted to shoot for an average cycle time of 3 days), we realized we were going to have to limit the size of our backlog. In other words, we were going to have to essentially issue a 503 (Temporarily Unavailable) response to some of our clients and simply not take their request onto our backlog and ask them to come back later, so as to remain responsive to our other customers. Just like we'd do in a web application server that was overloaded. Perhaps we'll even develop a cute picture of a flying acquatic mammal to try to soften the "not yets" we'll have to start handing out.

NoSQL East 2009 Redux

2009-10-31T21:06:00.006-04:00

I just attended NoSQL East down it Atlanta over the last two days. This was a fantastic conference, well-organized, with not only great content and speakers, but also a very well-educated audience. There was a telling moment in the first non-keynote talk where the speaker asked the audience "How many people have read the Dynamo paper?" and easily 95% of the audience put their hands up.

I'd divide the major focus areas into the following groups:

key-value stores (Riak, Voldemort, Cassandra)
massively parallel data mining (Pig, Cascading)
column-oriented datastores (Cassandra, HBase)
document collection databases (CouchDB, MongoDB)
stuff I couldn't make up my mind about (Neo4J, Redis, Tin)

As one of the speakers put it, which one you need depends on the shape of your data; these are all isomorphic in one way or another to each other, just as Turing complete languages are all essentially complete. But you still want to choose the right tool for the job. One nice touch was that a number of the systems were introduced not by one of the project committers, but rather by people who were using them to get stuff done. It was a very practical focus, much appreciated.

There were a couple of interesting tidbits from the conference, including a rollicking, confusing talk by John "My first network address was '12'" Day about network architecture and how TCP/IP had really gotten it all wrong. This guy was about 3 planes of existence above what I could follow at 9:15am--I had a distinct feeling he knew exactly what he was talking about, and was probably right, but there was no way he was making sense even to a very technical audience. Then there was the poor guy from Microsoft Research who had done some interesting distributed IDE work...for the .NET framework. In a sea of Mac and Linux laptops across the audience, he had a bunch of people who really couldn't make practical use of what he'd done.

Key-Value Stores

I've got to give a pretty big nod to Riak here for a straight-up key-value store; Justin Sheehy basically gave a talk about decentralized, scalable, lights-out system design that might well have been a good keynote, and it was clear that Riak was designed with high priority on those aspects. Major advantages over Voldemort in my mind are: ability to add/subtract nodes from running clusters, tunable (per-bucket, per-request!) quorum parameters (N,R,W), plus support for pluggable consistent hashing modules, partition distribution modules, and conflict resolution modules. For most of my use cases, with N=3, I'd probably do R=1,W=3 to optimize for read latency, as most of our writes would either be a user clicking "save" or would be the result of an asynchronous background process. On the other hand, if I wanted to build something like SQS on top of this (which I conjecture is possible), I'd probably do W=1,R=3 to optimize for the "post a message" latency.

While I was there, I had a great talk with them where we went over how there were going to expose the conflict resolution out up to the client via their HTTP/JSON interface. There's no out-of-the-box support for multiple datacenter awareness, although it seemed possible to add it via the pluggable partition distribution module, although their inter-node communication leverages Erlang's native networking, meaning some kind of VPN/IPSec tunnel would be the main way to make this work across sites. I'm pretty sure no matter what you would want to use in this space you'll probably have to end up using a VPN over WAN to give the appearance of a seamless LAN cluster (although the Cassandra guys piped up in the conference IRC channel that Cassandra has multi-site support already).

I can't tell whether being written in Erlang is a plus or a minus. On the one hand, it won't plug in well to our JMX-based monitoring frameworks, and I'm pretty sure very few folks within our organization have ever built or deployed an Erlang system. On the other hand, Erlang is the perfect choice for highly concurrent, fault-tolerant programming, and would probably be right up the alley of several of our programmers (I'm proud to say that while we are primarily a Java shop, we have a lot of other language proficiencies in-house spanning Ruby, Python, Clojure, and Scheme. Based on my brief escapade with it last night, I'm confident that (1) we have a number of folks who could pick it up easily and (2) who would jump at the chance). This is probably a wash.

Very interesting in my mind is that some of the key-value stores have brought over some features from other design spaces; Riak allows some basic link structure in their values, including a version of targeted MapReduce that can follow links, which starts to make it feel like a graph-oriented database like Neo4j. Similarly, Cassandra has support for column-oriented indices like HBase does. It's clear there's probably a project out there to scratch your particular itch.

Massive-scale data mining

We saw a couple of talks from folks who were using Hadoop for analyzing very large data sets. But furthermore, they were using frameworks like Pig and Cascading on top of Hadoop to do a lot of ad-hoc queries. Twitter in fact uses Pig to do all of their interesting web analytics, and they've taken all their analytics BAs who are used to doing ad-hoc SQL queries and trained them up with little problem in Pig. This is probably somewhere on our horizon, although there are larger cats to skin at the moment.

Column-oriented datastores

Cassandra and HBase were the major players here. Digg is moving all of their backend storage over to Cassandra after some successful pilots, and we saw a great talk by Mark Gunnels about how he is using HBase because it makes it "simple to get sh*t done". Apparently recent releases of HBase have made strides in fault tolerance (there is now an election algorithm for some of the special name nodes) and latency (apparently performance optimization was a major focus of the 0.20 release). There's an interesting article that describes some wide-area replication schemes that are available with HBase that sound intriguing (although once you are beyond two datacenters, I am convinced that you are better off with the VPN LAN solution if you want to have any hope of achieving eventual consistency).

Document-oriented databases

While the column-oriented datastores are good at organizing semi-structured data, the document-oriented guys are really all about organizing largely unstructured documents, and focus on doing some wild ad-hoc MapReduce queries. I still need to be convinced about the scaling, replication, and geographic distribution capabilities in this space, so it may be a while before we dip our toes in here.

Other

There was a very interesting talk about Tin ("the database so tiny they had to shorten its name"). This basically seemed to be a very clever approach for delivering stock data that leveraged a basic filesystem with range queries and rewriting rules via Sinatra. Turns out web servers serving static files goes pretty fast, if that's a suitable representation for your stuff (seems like his primary use case is for delivering read-only out to clients, where the data gets updated asynchronously in the background by the system). We actually have some datasets that are like that, so this is intriguing!

There was a talk from Neo4j, a graph-oriented database, which I just can't wrap my head around yet. I think I need to read up on some of the background research papers in this area. Certainly, the notion of a datastore based around the notion of link relations would likely be easy to expose as a REST HTTP interface, which is attractive. Our particular domain model can actually be nicely normalized, however (we are currently running off a traditional RDBMS after all), so I'm not sure we need the full semantic flexibility this offers.

There was also a talk from Redis; this is primarily an in-memory storage system with blazing-fast speeds, and the ability to write out to disk is really an afterthought. During the talk he showed a screen snapshot of a "top" running on a cloud-hosted virtual node with 60GB of memory. I cannot make up my mind whether this is ultimately just a badass memcached, or if he's on the cusp of something: if you have enough memory to hold your whole dataset, and you are sufficiently fault-tolerant, why bother writing to disk at all? Especially if you can easily get a periodic snapshot out to disk in the background that can be properly backed up for disaster recovery.

Conclusion

As folks pointed out, "NoSQL" might better be written "NOSQL" to mean "Not Only SQL". I didn't sense a lot of MySQL hatred in here; quite the contrary, many people were very complementary that there were certain things it does *really* well, and that the MySQL hammer was something that was staying firmly in their toolboxes. However, it is clear that there is a maturing community of very practically-minded folks that are looking for a new set of tools to drive their particular screws. Although to be sure (and this is something that I think some of the conference tweets corroborated), this also implies we all have a screw loose or two....

Object Calls == Message Passing in Erlang

2009-10-30T11:53:00.005-04:00

I started playing around with Erlang last night as a result of learning about Basho's key-value store Riak at NoSQL East yesterday (more specifically, it was due to Justin Sheehy's talk, and two realizations: (1) this guys *gets* a lot of important *operational* design choices in this space, and (2) he decided to build his system in Erlang).

So I decided to read the online Erlang "course" and started working on the exercises. One of them was:

Write a function which starts N processes in a ring, and sends a message M times around all the processes in the ring. After the messages have been sent the processes should terminate gracefully.

And so, summoning vaguely-remembered lectures by Bob Harper in "Fundamentals of Computer Science II" at CMU on doing object-oriented programming in Scheme, and remembering that the original Smalltalk OOP guys always said "send an object a message" rather than "invoke a method on an object", I set to work. [Editor's note: please feel free to post comments showing me better ways, I have known Erlang for all of about 12 hours at this point!]

Let's get the declarations out of the way. I need to define the entry point function which creates the ring of N nodes and sends the message around it M times, and I know I'm going to need a function representing a node in the ring, since I'm going to have to spawn processes for them.

-module(ring).
-export([ring_msg/3, ring_node/1]).

Ok, what job does a node in the ring have? Well, most of the time, when it receives a message, it just needs to pass it on to the next guy. So my node process is going to need to know about its next neighbor. Now in Erlang, what I would normally think of as an object can be modelled as a recursive function that passes its current state back into itself as an argument, and processes "method calls" by receiving messages. Interestingly, not all method calls actually have to send something back to the caller!

ring_node(Next) ->
  receive
    { pass, M, Msg } ->
      % Note that we got this message. 
      io:format("Node ~w~n",[Msg]),
      % Pass the message on around the ring.
      Next ! { pass, M, Msg },
      % If the count was down to zero, I can
      % exit, otherwise, I loop and wait for
      % the next incoming message.
      if M == 0 -> ok;
         true -> ring_node(Next)
      end
  end.

Ok, seems pretty straightforward. But if I had a ring of these set up, a message would just keep running around the ring. At least one node needs to be special, so that it can decrement the count M as the message comes through. It's pretty similar to the ring_node above, but is a little different.

init_node(Next) -> receive
    % message has been all the way around
    % the last time, so I can quit
    { pass, 0, _ } -> ok;
    % otherwise, log the message and pass
    % it on, decrementing the count
    { pass, M, Msg } ->
      io:format("Node ~w~n",[Msg]),
      Next ! { pass, M-1, Msg },
      init_node(Next)
  end.

Now an interesting thing here is that the init_node and the ring_node can both handle the "pass" message, and that when they send the message on, they don't actually care what their "Next" process is. It's like both of these "objects" implement the following interface:

public interface MessagePasser {
  void pass(int count, Object msg);
}

Ok, so now if we can create a ring with 1 init_node and (N-1) ring_nodes, we're all set if we inject the initial Msg into the init_node. So let's think about constructing a ring of nodes; if we have a node handy, we can pass that in as the initial argument (think "constructor") to a ring_node process to use as its Next node, then we just count down:

ring(Last, 0) -> Last;
ring(Last, N) -> 
  RN = spawn(ring, ring_node, [Last]),
  ring(RN, N-1).

Hmm, that's close, but that's a linked-list of nodes, not a ring. But we can't pass a node in as a constructor argument to the first node we create, because we don't have any yet! So it seems like we'll need to construct a linked-list of nodes, and then "close the loop" by stitching the front and the back together. Our init_node is already a special node, so maybe we can extend it this way:

init_node(Next) -> receive
    % acknowledge the request, update state
    { setNext, N, From } -> From ! ok, init_node(N);
...

In other words, the init_node can get a special message telling it to "update" its Next state. In some sense, we've just done this:

public interface InitNode extends MessagePasser {
  void setNext(MessagePasser N);
}

We want to acknowledge the request so our ring construction knows when that message has been processed -- we don't want to hand the ring back until it's all stitched together, and we can't guarantee ordering of message delivery unless we specifically wait for a response. So here's the full ring construction:

ring(N) when is_integer(N) ->
  % just pass in a placeholder for Next
  RN0 = spawn(ring, init_node, [nil]),
  ring(RN0, RN0, N-1);
% finished stitching, can return our
% init node to the caller
ring(Init) -> receive ok -> Init end.
ring(Init, Last, 0) -> Init ! { setNext, Last, self()}, ring(Init);
ring(Init, Last, N) ->
  RN = spawn(ring, ring_node, [Last]),
  ring(Init, RN, N-1).

Finally, the thing we're trying to do (including optimizing the degenerate cases):

ring_msg(0, _, _) -> ok;
ring_msg(_, 0, _) -> ok;
ring_msg(N, M, Msg) ->
  Init = ring(N), Init ! { pass, M, Msg }, ok.

Actually runs, too! It's pretty neat to see polymorphism via being able to accept the same message, and I've always loved the pattern matching in ML (both SML and OCaml variants!). Some pretty serious systems programs are getting written in this language; it's clear that the process spawning methodology lends itself well to a SEDA-style approach which is great for graceful degradation of service, and the fully-functional style (no mutation) means that you have no locks, no shared state, and hence safe concurrency (as long as you can model what you're doing properly).

Agile Architecture Anti-Patterns

2009-08-26T21:54:00.005-04:00

I have been the solution architect for an enterprise-spanning project which is now getting geared up for its next phase. This project touches more than 10 different development teams across at least three company divisions and involves at least one external ISV. So, you know, a pretty small project.

As I reflect on the process of hashing out all the technical details, I realized I got stretched pretty thin. I really didn't have time to write much down in any formal fashion this time around, due to the time pressure on the project and the number of other things I was also trying to juggle. I have instead been spending a lot of time drawing stuff on whiteboards for various technical audiences. The things I did have time to write up were only the most general concepts and patterns - i.e. the really high-level architecture.

At least part of the reason I didn't have time to write anything else down was that I was spending all my time in technical design discussions with all the various groups, hashing out nitty-gritty integration details and then bringing full integration designs back to the development groups to run with. The last time through, this worked out well, because the developers in my home organization are used to agile development and being responsible for their own technical design, so they were able to take my whiteboard sketches, continue to consult with me, and produce solid, working code. It also worked out ok because the last phase was actually the first phase of this project, and I had enough lead time to work out all the details ahead of time.

This time around, the time pressure is pretty high, plus the work-ahead time I would have had in a traditional "up-front" architecture process was actually consumed by helping to get the first phase out the door. So I find I actually don't have enough time to do the same detailed design that I did the first time around before the development teams are scheduled to start. This would seem to be a major conundrum.

Antipattern 1: waterfall, up-front architecture doesn't pipeline well if the architecture phase extends concurrently all the way through the development phase.

Corollary: trying to waterfall development and QA doesn't pipeline well either for exactly the same reason (because developers have to keep working their code to fix bugs).

The other anti-pattern that I was running into was with development teams in some of the other divisions; I would work with the enterprise architects to hash out a pretty detailed design, but when we brought that to the development teams, there was a ton of convincing, resistance, and redesign that ended up happening. Now, I'm not sure exactly which anti-pattern we were running into, but it was one of the following:

Antipattern 2: up-front architecture doesn't work if the architects aren't familiar enough with the details of the systems/teams they are designing for, because the designs won't "fit right".

Antipattern 3: up-front architecture doesn't work if the architects haven't been able to build up sufficient technical cred to sell their designs to the developers.

For sure both of those applied to me with respect to the developers and technical leads of the groups in the other divisions.

I've been thinking a lot recently about the right way to approach this, and really enjoyed this essay on Agile Enterprise Architecture by Scott W. Adler. He proposes a much more lightweight, hands-on approach to architecture which can best be described as guiding the agile development teams to develop the right architecture themselves.

Fred Brooks writes in The Mythical Man-Month that a systems' architecture must proceed from one or a small number of minds in order to be coherent (I can't find a copy right now, so that's paraphrased). At the same time, agile development techniques like Extreme Programming (XP) or Scrum suggest that the development teams ought to be responsible for evolving their architecture organically. I'm starting to think that both of these are true, and that they are not mutually exclusive, and it is related to another realization I've had (but that I have a hard time remembering):

A solution architect is responsible for seeing that the end-to-end solution hangs together technically. It is not necessary for the architect to produce the whole end-to-end design himself to achieve this.

Going forward, I think I'm going to take the following approach: figure out broadly which development teams are going to need to interact and what their coarse responsibilities are, then immediately get them involved on working out the solution. In true agile fashion, let them work out the details, and just play Product Owner to set high-level (technical) requirements, being on-hand to participate in the design discussions. I think this leverages best the available technical design talent in the development teams, while making sure that someone is tracking it all and fitting it into a coherent mental model to make Mr. Brooks happy.

RSA Public Key Cryptography in Java

2009-05-15T16:18:00.007-04:00

Public key cryptography is a well-known concept, but for some reason the JCE (Java Cryptography Extensions) documentation doesn't at all make it clear how to interoperate with common public key formats such as those produced by openssl. If you try to do a search on the web for how to make RSA public key cryptography work in Java, you quickly find a lot of people asking questions and not a lot of people answering them. In this post, I'm going to try to lay out very clearly how I got this working.

Just to set expectations, this is not a tutorial about how to use the cryptography APIs themselves in javax.crypto (look at the JCE tutorials from Sun for this); nor is this a primer about how public key cryptography works. This article is really about how to manage the keys with off-the-shelf utilities available to your friendly, neighborhood sysadmin and still make use of them from Java programs. Really, this boils down to "how do I get these darn keys loaded into a Java program where they can be used?" This is the article I wish I had when I started trying to muck around with this stuff....

Managing the keys

Openssl. This is the de-facto tool sysadmins use for managing public/private keys, X.509 certificates, etc. This is what we want to create/manage our keys with, so that they can be stored in formats that are common across most Un*x systems and utilities (like, say, C programs using the openssl library...). Java has this notion of its own keystore, and Sun will give you the keytool command with Java, but that doesn't do you much good outside of Java world.

Creating the keypair. We are going to create a keypair, saving it in openssl's preferred PEM format. PEM formats are ASCII and hence easy to email around as needed. However, we will need to save the keys in the binary DER format so Java can read them. Without further ado, here is the magical incantation for creating the keys we'll use:

# generate a 2048-bit RSA private key
$ openssl genrsa -out private_key.pem 2048

# convert private Key to PKCS#8 format (so Java can read it)
$ openssl pkcs8 -topk8 -inform PEM -outform DER -in private_key.pem \
    -out private_key.der -nocrypt

# output public key portion in DER format (so Java can read it)
$ openssl rsa -in private_key.pem -pubout -outform DER -out public_key.der

You keep private_key.pem around for reference, but you hand the DER versions to your Java programs.

Loading the keys into Java

Really, this boils down to knowing what type of KeySpec to use when reading in the keys. To read in the private key:

import java.io.*;
import java.security.*;
import java.security.spec.*;

public class PrivateKeyReader {

  public static PrivateKey get(String filename)
    throws Exception {
    
    File f = new File(filename);
    FileInputStream fis = new FileInputStream(f);
    DataInputStream dis = new DataInputStream(fis);
    byte[] keyBytes = new byte[(int)f.length()];
    dis.readFully(keyBytes);
    dis.close();

    PKCS8EncodedKeySpec spec =
      new PKCS8EncodedKeySpec(keyBytes);
    KeyFactory kf = KeyFactory.getInstance("RSA");
    return kf.generatePrivate(spec);
  }
}

And now, to read in the public key:

import java.io.*;
import java.security.*;
import java.security.spec.*;

public class PublicKeyReader {

  public static PublicKey get(String filename)
    throws Exception {
    
    File f = new File(filename);
    FileInputStream fis = new FileInputStream(f);
    DataInputStream dis = new DataInputStream(fis);
    byte[] keyBytes = new byte[(int)f.length()];
    dis.readFully(keyBytes);
    dis.close();

    X509EncodedKeySpec spec =
      new X509EncodedKeySpec(keyBytes);
    KeyFactory kf = KeyFactory.getInstance("RSA");
    return kf.generatePublic(spec);
  }
}

That's about it. The hard part was figuring out a compatible set of:

openssl DER output options (particularly the PKCS#8 encoding)
which type of KeySpec Java needed to use (strangely enough, the public key needs the "X509" keyspec, even though you would normally handle X.509 certificates with the openssl x509 command, not the openssl rsa command. Real intuitive.)

From here, signing and verifying work as described in the JCE documentation; the only other thing you need to know is that you can use the "SHA1withRSA" algorithm when you get your java.security.Signature instance for signing/verifying, and that you want the "RSA" algorithm when you get your javax.crypto.Cipher instance for encrypting/decrypting.

Many happy security returns to you.

Downloading your Blogger archives

2009-02-10T18:26:00.005-05:00

A friend was looking for a way to grab an archive of his Blogger posts into a CSV file he could do text mining on (and presumably, for a low-fi backup mechanism). I wrote this Python script for him, enjoy.

#!/usr/bin/env python
#
# Copyright (C) 2009 by Jon Moore
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import csv
import urllib2
import unicodedata
import xml.etree.ElementTree as etree

blog_feed = 'http://codeartisan.blogspot.com/feeds/posts/default'
output = 'posts.csv'

ATOM_NS = 'http://www.w3.org/2005/Atom'

def norm(s):
    if not s: return None
    return s.encode('ascii','ignore')

def main():
    f = open(output, 'wb')
    csv_wr = csv.writer(f)
    url = blog_feed + '?max-results=100'
    csv_wr.writerow(['id','published','updated','permalink','title','content'])
    while url:
        print "fetching", url
        feed = etree.fromstring(urllib2.urlopen(url).read())
        for entry in feed.findall("{%s}entry" % ATOM_NS):
            id = entry.find("{%s}id" % ATOM_NS).text
            published = entry.find("{%s}published" % ATOM_NS).text
            updated = entry.find("{%s}updated" % ATOM_NS).text
            title = norm(entry.find("{%s}title" % ATOM_NS).text)
            content = norm(entry.find("{%s}content" % ATOM_NS).text)
            perm_url = ''
            for link in entry.findall("{%s}link" % ATOM_NS):
                if (link.get('rel') == 'alternate'
                    and link.get('type') == 'text/html'):
                    perm_url = link.get('href')
                    break
            csv_wr.writerow([id,published,updated,perm_url,title,content])
            print "wrote",id
        url = None
        for link in feed.findall("{%s}link" % ATOM_NS):
            if link.get('rel') == 'next':
                url = link.get('href')
                break
    f.close()

if __name__ == "__main__":
    main()

Business Cases and Cloud Computing

2009-01-24T13:52:00.006-05:00

I just read a very interesting article by Gregory Ness on seekingalpha.com that talks about some of the technology trends behind cloud computing. One key quote:

Automation and control has been both a key driver and a barrier for the adoption of new technology as well as an enterprise’s ability to monetize past investments. Increasingly complex networks are requiring escalating rates of manual intervention. This dynamic will have more impact on IT spending over the next five years than the global recession, because automation is often the best answer to the productivity and expense challenge.

One other cited link is to an IDC study that includes the following graph:

Note that staffing accounts for 60% of the cost of maintaining a server over its lifetime. Cloud infrastructure services like Amazon EC2 would really only save an enterprise data center on hardware setup / software install costs, which are probably, in terms of staffing, a small amount of staff time for a given server. Actually administering the server once it is running is really the bulk of the cost, and that won't go away on EC2 -- you'll still need operations staff to provision/image cloud infrastructure. EC2 makes sense if the economies of scale of AWS are such that they can achieve a lower operational cost for that other 40% than you can, or if there is a business / time-to-market value proposition that makes sense in being able to provision hardware on EC2 more rapidly than we can acquire and install hardware yourself.

Given the huge economy of scale that the large cloud providers have--tens of thousands of servers, it is going to be hard to get your costs for that 40% lower than what they can achieve with their existing infrastructure automation and ability to purchase hardware in bulk, especially for a startup company whose hardware needs are initially modest. Let's guess that there's a 33% markup on cost for EC2, so when you are getting charged $0.10 per CPU hour, it's really only costing them $0.075. Let's assume a 75% experience curve on infrastructure (meaning, once you have doubled the number of servers you have deployed, the last server costs only 75% of what the halfway point was).

By one estimate, Amazon has 30,000 servers. Now let's work backward (1/0.75 = 1.33): at 15,000 servers, their cost was $0.075 * 1.33 = $0.9975. At 7500 servers, their marginal cost was $0.9975 * 1.33 = $0.13. In other words, you'd have to be planning to deploy 15,000 servers in order to have a hope of getting your marginal cost under what they'll charge you retail.

(I think this is actually a conservative estimate: the experience/learning curve for infrastructure deployment is probably steeper than 75% due to existing hierarchical deployment patterns and a product (provisioned servers) that lends itself well to automation. Also, due to the high barrier to entry for cloud computing in terms of number of servers you need to be competitive, they can probably get away with charging an even higher markup).

One corollary of this is that if you are currently running a data center with far fewer servers (i.e. the hardware is a sunk cost), you might actually be better off turning your data center off and leasing from Amazon. Now of course, there are some things (customer credit card data, extremely sensitive business information) that you just wouldn't be willing to host somewhere outside your own data center. But that's probably a very specific set of data--host that stuff and lease the rest in the cloud, particularly if you can get adequate SLAs from your cloud vendor.

So that deals with the 40% of the TCO for a server that isn't staffing. How do you cut costs on the other 60%?

You won't really be able to make a dent in that 60% until you get not just to fully automated infrastructure provisioning, but until you get to fully automated software deployment and provisioning. This is not possible until you get to standardized computing platforms with specific functionality that are scale-on-demand, like Akamai NetStorage, Amazon S3/EBS/SQS/SimpleDB, and Google AppEngine. These are known as "Platform-as-a-Service" (PaaS) offerings.

There's a similar experience curve argument here: you could spend internal development time here to set up some kind of application deployment framework, but you'd essentially have to be willing to build and deploy within orders of magnitude the number of different apps as the Google App Engine team in order to get your costs under what Google will charge you. Unless you are in the business of directly competing with them in the PaaS market, you might as well buy from them and focus your energy on providing your unique business value, not software or hardware infrastructure. [Editor's note: this was something Matt Stevens said to me a while ago, and it wasn't until I went through the mental exercise of writing this article that I actually got it].

Yesterday I implemented (not prototyped) a service in Google App Engine in about 6 hours that would cost around $400 per month (according to their recent pricing announcements) if projected usage were more than double what it is now. I estimate this would require at least 10 database servers just to host the data in a scalable, performant fashion, nevermind the REST data interface (webnodes) sitting in front of it. On Amazon EC2, that'd be $720 per month on your small instances (assuming those were even beefy enough), and per the experience curve argument above, it's probably way more than that in our data center. And that's not counting any of the reliability/load balancing infrastructure.

So my open question is: how, as a software developer, can you justify not building your app in one of these cloud frameworks?

Websites are also RESTFul Web Services

2009-01-12T18:53:00.037-05:00

I have been reading the Richardson and Ruby book RESTful Web Services and recently had an epiphany: if you design a RESTful web site it is also a RESTful web API. In this post I'll show exactly how that works and how you can use this to rapidly build a prototype of a modern web application.

First of all, let's start with a very simple application concept and build it from the ground up. Let's consider a simple application that lets you keep a list of favorite items. The resources we'll want to model are:

a favorite item
a list of a user's favorite items

We'll assign the following URLs:

/favorites/1234 for favorite item with primary key 1234
/favorites is the list of everyone's favorite items
/favorites/-/{owner}joe@example.com (or its URL-encoded equivalent) is the list of items belonging to our friend Joe (this is a URL format inspired by Google's GData protocol).

Now, we'll support the following HTTP methods:

You can GET and DELETE an individual item (we could allow PUT if we wanted to allow editing, but we'll keep the example simple)
You can create a new favorite item with a POST to /favorites.
You can GET the list of a user's favorites.

Ok. Now we want to rapidly prototype this so we know if we have the resources modelled correctly. Fire up your favorite web application framework (Ruby on Rails, Django, Spring, etc.) and map those URLs to controllers. Now most of these frameworks let you fill in implementations for the various HTTP methods. We'll make a minor simplification and allow "overloaded POST" where we allow passing a URL parameter to POST to specify PUT and DELETE (e.g. "_method=DELETE"). We can implement the proper HTTP method but we'll allow you to use POST to do it too; browsers and some Javascript HTTP implementations can only do GET and POST.

Ok, now a funny thing happens: it you render an HTML response for everything, you can start playing with your API in your browser! In particular, when we render the list of items, we will naturally put the text of those items on the page, but we can also throw the following HTML snippet at the top of the page:

<p>Add a new favorite:</p>
<form action="/items" method="post">
  <input type="text" name="itemname"/>
  <input type="submit" value="Add"/>
</form>

We can also add the following form after each item's text:

<!-- use a specific item's URL for the action -->
<form action="/items/1234" method="post"/>
 <input type="hidden" name="_method" value="DELETE"/>
 <input type="submit" value="Delete"/>
</form>

which gives us this ugly beastie:

A Few of My Favorite Things:

Now, one key item is that when you render the result page for adding an item, you can send a 201 (Created) response and say something like "item added", throwing in a link back to the list page. The whole HTML response might be nothing more than:

<html>
  <head></head>
  <body>
  <p>I created your item <a href="/items/2345">here</a>.</p>
  <p>All your items are <a href="/items/-/{owner}joe@example.com">here</a>.</p>
  </body>
</html>

We similarly want to render a confirmation page after a DELETE. This makes for an awkward user experience "add, ok, add, ok,..." but you'll notice that the back and forward buttons on your browser actually work without having to rePOST any exchanges.

[Side note: you could, instead of returning a success page, return a 302 that lands you back on the list page, which maybe gets you closer to what you wanted from a user experience, but this is precisely what will break your browser's back button and make you rePOST.]

Now you also have the interesting property that all the links on your site are safe (without side effects) GETs, and all the buttons are potentially destructive (write operations of one sort or another). I say only "potentially" because you might have a search form with action="get" to do a query, and not all of your POST/PUT/DELETEs will actually change anything.

At any rate, at this point, you have a functionally working website that someone could use, if somewhat awkwardly. Plus, if you have my frontend aesthetic design sensibilities, your users will have the pleasure of suppressing a gag reflex while using your site.

So let's spruce this up a little bit. Now, on the HTML page for the list of favorites, we can apply some Javascript. At a first blush, we can hide the delete buttons until a mouseover, which cleans things up somewhat. But the real magic happens when we attach an AJAX event to the delete buttons. Now the script can actually do the very same POST that the form would have done, and then check the HTTP status code, removing the item text from the DOM on success.

Suddenly, the user never leaves that list page, and we haven't had to change any of the rest of the API -- just the HTML representation of that list page. The AJAX call doesn't care if it gets HTML back (in this case), it just cares about the response code. Now we have the nice AJAXy experience we would expect, but oddly enough you still have a website that will work for people with Javascript disabled.

The last step towards finishing out your API is probably simply to make structured versions of your representations available (e.g. JSON or XML formats like Atom) with an optional parameter like "?format=json". Now all of your client-side functions can call URLs with the appropriate format on them and get well-structured data, and everyone else gets HTML.

Well, I guess that's the second to last step. You probably actually want to apply some graphic design and CSS to your site too...

Simple Backups for your Mac

2008-09-19T22:24:00.014-04:00

You are probably well aware of the need for offsite backups; as a technology professional this is one of the first arrangements I look into for any permanent storage of business information. When I started two years ago for an internal "startup" for a large company, one of the first things we did was set up an SVN repository and then work out an arrangement with an offsite data storage provider. However, the cobbler's children have no shoes: I've never set up a proper backup scheme for my own data at home, and it's about time to take care of business.

Fortunately, now that we've moved to using Macs at home and with the advent of cheap UN*X hosting providers, it's about time I stopped putting this off. The scheme here is pretty simple: get a hosted Linux server from someone like 1and1.com, dreamhost.com, or rackspace.com where the storage is backed up and they take care of security updates for the OS. Then set up a pretty simple combination of the UN*X utilities rsync, ssh, cron, and bash scripts to get secure nightly backups going. Just to make it more fun, I'm going to challenge myself to have this all working in under an hour! I'll keep notes as I'm doing it as to how long it takes, not counting the writeup before or afterward.

I decided to register a domain name with a hosting provider, since it was included. My basic requirements were:

SSH access
rsync installed
enough storage for my data

Be sure to acquire the following information from your hosting provider:

username/password with SSH access (preferably root, if you want to use the server for other purposes, but this is not necessary)
IP address
SSH host key of the server

I ended up registering a new domain with at dreamhost.com at $9.95/month. As it happened, DreamHost was running a promotion with unlimited disk space and bandwidth for the lifetime of my account. Score! I did have to email tech support to get the ssh host key. If you find yourself in a similar position, you can ask for the output of:

$ ssh-keygen -l -f ssh_host_rsa_key.pub
2048 0e:c2:f6:f4:d9:86:9d:4b:c4:3d:77:e7:a4:bb:59:14 ssh_host_rsa_key.pub

Ok, great! Now you have a destination for your offsite storage. Next step is to make sure we can securely log in over the network (we'll use ssh for this). On the Mac you want to back up, open up a Terminal window, and ssh into your server using your username and the server's hostname, as in the following. N.B. Do not finish connecting if the ssh server host key you got from your hosting provider does not match the key you see when you try this!

macbook:~ jonm$ ssh jonm@backup.dreamhost.com
The authenticity of host 'backup.dreamhost.com (67.205.39.2)' can't be established.
RSA key fingerprint is 0e:c2:f6:f4:d9:86:9d:4b:c4:3d:77:e7:a4:bb:59:14.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'backup.dreamhost.com,67.205.39.2' (RSA) to the list of known hosts.
jonm@backup.dreamhost.com's password:
[backup]$

Ok, so far so good. Now we need to make sure we can do it without needing a password; this is where user ssh keys come into play. First, let's create an ssh key to use for backups. We'll want to do this as the root user on our Mac, so that when we run the backup script out of cron, we won't run into permissions problems. You can use the "sudo" command to become root on your Mac:

macbook:~ jonm$ sudo su -

WARNING: Improper use of the sudo command could lead to data loss
or the deletion of important system files. Please double-check your
typing when using sudo. Type "man sudo" for more information.

To proceed, enter your password, or type Ctrl-C to abort.

Password: <enter jonm's password on my mac>
macbook:~ root#

Now we need to create an SSH public/private key pair; this is a similar concept to PGP email encryption/signing; you can read a really interesting description of the chronology behind public key cryptography in the book Crypto by Steven Levy. We'll keep the private key locally on our Mac, and take a copy of the public key and copy it securely up to our backup server; then ssh will use the private key when we connect, allowing the backup server to verify using the public key that we are who we say we are, without having to send a password. Nice.

Specifically, we will want to do the following (still as root):

macbook:~ root# mkdir .ssh
macbook:~ root# chmod 700 .ssh
macbook:~ root# ls -ld .ssh
drwx------  2 root  wheel  68 Sep 19 22:01 .ssh
macbook:~ root# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/var/root/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/root/.ssh/id_dsa.
Your public key has been saved in /var/root/.ssh/id_dsa.pub.
The key fingerprint is:
fd:47:1d:a6:ac:d0:7d:fb:a5:17:cf:e2:8a:93:a5:30 root@jon-moores-macbook.local
macbook:~ root#

Use an empty passphrase (i.e. just hit return when prompted for the passphrase), as this will allow the ssh program to load the key without interaction from you. Also note, however, that anyone who gets root access to your Mac will be able to ssh into your backup server at will. Given that our backup server contains a copy of what this would-be hacker would be able to see on the actual Mac anyway, I don't really see this being a big risk....

Now, we need to copy the public key over to the backup server:

macbook:~ root# scp .ssh/id_dsa.pub jonm@backup.dreamhost.com:
jonm@backup.dreamhost.com's password:
id_dsa.pub                             100%  619     0.6KB/s   00:00
macbook:~ root#

You'll have to verify the server SSH key one more time, because now you are connecting from root rather than from your normal user account. Now we'll tell the backup host to accept a login from this key pair:

[backup]$ mkdir -p .ssh
[backup]$ chmod 700 .ssh
[backup]$ cat id_dsa.pub > ~/.ssh/authorized_keys
[backup]$ chmod 600 ~/.ssh/authorized_keys
[backup]$ exit

Now, we should be able to log in without a password from our Mac:

macbook:~ root# ssh jonm@backup.dreamhost.com
[backup]$

Sweet. Now we create a directory where our mirrored filesystems will live:

[backup]$ mkdir mac-backups
[backup]$ chmod 700 mac-backups
[backup]$ exit
macbook:~ root#

The utility we'll use to do the mirroring is the rsync utility, which can be invoked to run securely over ssh. This actually makes a nice backup utility for regular use, as the rsync protocol is actually pretty smart about being able to find just the small subsets of data that changed since the last sync; after the first big sync, for most personal file use, there won't be much work to do every night.

For now, let's set up a test directory on our local Mac.

macbook:~ root# mkdir /tmp/back-me-up
macbook:~ root# echo "data" > /tmp/back-me-up/afile.txt

Now, to make the magic happen, we do this:

macbook:~ root# rsync -avz -e ssh /tmp/back-me-up jonm@backup.dreamhost.com:mac-backups
building file list ... done
back-me-up/
back-me-up/afile.txt

sent 116 bytes  received 40 bytes  62.40 bytes/sec
total size is 5  speedup is 0.03
macbook:~ root#

Now we can keep a window open on our backups host, and we should see everything show up there:

[backup]$ ls -lR mac-backups
mac-backups:
total 4
drwxr-xr-x 2 jonm pg1807352 4096 2008-09-19 19:11 back-me-up/

mac-backups/back-me-up:
total 4
-rw-r--r-- 1 jonm pg1807352 5 2008-09-19 19:11 afile.txt
[backup]$

Just for fun, run the same rsync command above and see that nothing happens if there have been no changes (or rather, just that a very small amount of data gets exchanged to verify no changes).

Let's just make sure changes show up:

macbook:~ root# echo "changed-data" > /tmp/back-me-up/afile.txt
macbook:~ root# rsync -avz -e ssh /tmp/back-me-up jonm@backup.dreamhost.com:mac-backups

(other window)

[backup]$ cat mac-backups/back-me-up/afile.txt
changed-data
[backup]$

Ok, looking good. Next step is to identify all the directories you want to back up; let's keep a list of them in a config file on our mac:

macbook:~ root# mkdir -p /usr/local/etc
macbook:~ root# cat - > /usr/local/etc/backups.conf
/Users/jonm/Documents
/tmp/back-me-up
macbook:~ root#

Note that it is important *not* to have trailing slashes on these directory names, as this changes rsync's behavior slightly in a way that you will probably find annoying (it won't copy the directory name over, just the contents).

Ok, now the next step is to set up a script that can sync each of the directories:

macbook:~ root# mkdir -p /usr/local/bin
macbook:~ root# touch /usr/local/bin/do-backups
macbook:~ root# chmod 700 /usr/local/bin/do-backups
macbook:~ root# cat - > /usr/local/bin/do-backups
#!/bin/sh
for dir in `cat /usr/local/etc/backups.conf`; do
  rsync -avz -e ssh $dir jonm@backups.dreamhost.com:mac-backups
done
macbook:~ root#

Now we run it once by hand to make sure it works:

macbook:~ root# /usr/local/bin/do-backups

Finally, we install this in root's crontab as follows:

macbook:~ root# crontab -l > /tmp/root.cron
macbook:~ root# cat - >> /tmp/root.cron
# take a backup every day at 3am
0 3 * * * /usr/local/bin/do-backups >/dev/null
macbook:~ root# crontab /tmp/root.cron
macbook:~ root# rm /tmp/root.cron

Nice and simple. Now the backups are off and running every night without your intervention.

If you ever need to restore from the backup, you can always reverse the rsync process like this:

macbook:~ root# rsync -avz -e ssh jonm@backup.dreamhost.com:mac-backups/back-me-up /tmp

for each of the directories you have backed up over there.

Enjoy, and sleep well tonight....

P.S. Total elapsed time for the exercise was 2 hours from the time I placed the hosting order to the time the crontab was installed, but I took a one hour break in the middle for dessert and bedtime with the kids. So I'll claim this really did only take one hour of "CPU time" for me.

Measure your improvements

2008-09-02T15:50:00.010-04:00

Metrics are an important part of any development group's toolset. If we want to continually improve our ability to develop software (through a lean engineering kaizen approach, or simply as a learning organization), then we need to have a way to figure out:

what parts of our process need improvement?
when we make a change, did it help or hurt?

This is where process metrics come into play. I'll start with my definition of a metric, which is a numerical measurement of something. If you can count it, it can be a metric. So "number of outstanding bugs" is a metric, but "software quality" is not. The term "quantitative metric" is redundant, and "qualitative metric" is an oxymoron.

There are generally two types of metrics we can capture:

causal metrics: these are metrics that have a direct business impact: for example, ROI for a feature, unique monthly visitors, click-through ad rate, etc.
symptomatic metrics: these are metrics that do not directly affect ROI (although we might believe they do) but are downstream indicators for one or more causal metrics. The number of outstanding bugs in a product, the number of bugs caught in a certain phase of development, percentage of code covered by unit tests, etc. are all symptomatic metrics.

My general observation, based on reading articles around the use of metrics for improving your processes, is that a lot of metrics-based improvement projects fail to distinguish between these two types of metrics. Partly, I think this is because while the causal metrics properly align your improvement efforts with your business's interests, they are also harder to define and measure. By contrast, a lot of symptomatic metrics are easy to find and measure, but their relationship to the business may be less clear.

For example, consider the balance between software quality and time to market. You can take a longer time when developing a feature or product to reduce the number of bugs that show up at deployment, or you can ship a feature more quickly, knowing that there may be both known and unknown bugs present. In this case, you can measure both number of known bugs at deployment time, and you can measure overall time-to-production for a feature (time from feature conception to deployment).

Now, if you can decrease time-to-production without increasing the bugginess of your code, that's a win. Similarly, if you can reduce bugginess without lengthening your production time, that's a win. However, both of these things will probably require some effort to implement. Another interesting possibility would be to simply make an adjustment of where you sit on this balance. For example, simply spend more time looking for and fixing bugs in your QA phase, to tradeoff fewer bugs for a slower time-to-market. Or vice versa to get to market more quickly, possibly with more bugs. Both of these adjustments are probably relatively painless to implement, in that no one has to change what they do, just how long they do it for.

So the question is, which one of these things ought we to do? My argument is that bugginess and time-to-production, both being symtomatic metrics, don't give us the answer directly. It all depends on our product environment. For example, when producing software to run medical equipment, a company reputation for quality might be more important than shipping new features quickly; or, in a highly innovative internet space, time to market might be king in terms of how much market share you can capture.

It's management's job to both help implement win-win changes as well as to set the "slider" of the quality/time tradeoff at the right spot. The trick, of course, is that it might be hard to measure this directly; there are several symptomatic things we could measure, including:

time spent in development
time spent in QA
time spent in deployment
number of outstanding bugs
profit for the product at a given point in time

Now really, the profit over some window of time (e.g. for a website, revenue vs. development spend for a given month) is the thing we want to optimize. The interesting idea here is for management to be able to run a series of experiments: if I increase/decrease QA or development time, how does it affect ROI for my product? How does the relative bugginess of a release affect its profitability? For certain "sliders" in the business, it is relatively simple to take a series of measurements to find a current "sweet spot".

An interesting idea here is that sometimes we work through things backwards. For example, we try to estimate "how long will it take to fully regression test a release", or "how long will it take to code up a feature", rather than "how buggy will the release be if we test it for X amount of time", or "how much of this functionality can you develop in X amount of time." In other words, rather than deriving the time-to-market from a set of estimates for all the steps, instead set the time to market by timeboxing those steps, and see what the outcome is. This is a powerful notion of metrics-based management that is hinted at (in the notion of Scrum timeboxed iterations) but which I have not seen explicitly suggested anywhere[1]. (Please post all the references to things that I've missed in the comments section--I'm sure there are plenty).

At the end of the day, however, it is hard to optimize things we can't measure. I think important metrics to gather are:

the levers we have available to manipulate our process (e.g. timeboxing)
causal metrics that affect our business (ROI, product profit)

We need to be aware which metrics are causal, and which are merely symptomatic, so that we are measuring things that directly affect the business somehow. This approach permits empirical management--adjust something you can control, see how it affects your causal metrics, rinse, repeat.

[1] Scrum timeboxes an entire iteration, but does not timebox an individual feature, so a team may be able to spend all their time on one feature, or spread their effort across many features. The closest thing I've seen here is the notion of the "Small" in INVEST user stories, where stories are limited to a certain amount of complexity. However, the story points in this case are still estimates of the work involved, rather than timeboxes around how much time to spend implementing a feature; the "small" requirement is really to permit more accurate estimation rather than to timebox the amount of effort (although it does secondarily have this effect, I've not seen this stressed in articles about this).

Cracking down on technical debt

2008-08-03T12:57:00.004-04:00

"Simplicity is the ultimate sophistication." --Leonardo da Vinci.

"Everything should be made as simple as possible, but no simpler." --Albert Einstein

"A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away." --Antoine de Saint-Exupery

I've written before about the notion of technical debt. In this post, I want to discuss a few specific sources of technical debt that are easy to accrue, particularly in an agile, iteration-based setting.

Incomplete technology transitions

These can arise when a technical decision gets made to transition from one technology/architecture/design to another, and the transition happens incrementally. What can end up happening is that an agile team, say one operating under the Scrum framework, does not complete its incremental transition during the current sprint. Now, although the code is in a working state, there is a good chunk of technical debt arising from having code operating under two separate systems. This transition debt is problematic for a few reasons:

First, this can complicate debugging efforts -- when there is a problem with the system, someone has to determine under which scheme the code in question was written. Typically this can mean looking in two different source code hierarchies, or looking through two separate sets of configuration. The system is, as a result, more complicated than it needs to be.

Secondly, this can be an attractor for additional debt; if the old system is still around, and a developer is more familiar with the old system than the new, there is a very strong temptation to make changes/additions in the old system. This work simply adds to the outstanding transition work, and despite the developer's familiarity, is likely to be implemented in a more difficult or less efficient way (assuming, of course, there were valid technical reasons for making the transition in the first place).

Finally, this can cause extra work to happen during feature development that touches/interacts with the subsystem in transition, because either the cooperating subsystems have to special case two different interaction styles, or an adaptation layer has to be built to handle both subsystems and abstract their existence away from clients' concerns. Either way, you are writing more code that you would have if you had completed the transition and you just had one implementation of the subsystem.

Teams working on an iteration-based methodology need to do several things to avoid the pitfalls from transition debt:

when a technical decision for a transition has been made, it must be communicated clearly to the whole development team, including the reasons for the transition. This can help prevent the unintentional accrual of additional transition debt.
plan for more refactoring time when signing up for work, to leave time to complete transitions before an iteration ends.
communicate the existence of the transition debt to the Product Owner at the review, so that completing the transition can be scheduled as a backlog item. Furthermore, stress the priority of this carryover work to ensure that the transition debt exists for the shortest amount of time possible.

Obsolete/extraneous configuration

We'll call this type of technical debt configuration debt. There are a couple of sources of this type of debt:

transitional runtime configuration that still exists after the transition. For example, when a data partner was making an id space transition to extend the length of their ids, we had a flag to govern whether to use old or new ids with that partner, so that we could decouple our code releases from the partner's transitions. Over a year later, the flag still exists, but it is always set to use "new" ids, so there is certainly unneeded code to handle this.
exposing properties that would only change with a code drop as runtime configuration. In this case, the values of the properties would really only change if we rolled a new code release, so they could just as easily be compile-time constants that would not require the scaffolding to make them runtime properties, no matter how simple that scaffolding might be.

Unnecessary code hurts you in several ways:

it took someone time to write it in the first place
you have to compile it or run its unit tests over and over again while you're developing (death of a thousand cuts)!
people need to keep it in their mental model of the system instead of leaving room for parts of the system that actually do something useful

The easiest way to prevent the accrual of configuration debt is to review any new runtime configuration parameters at the end of each sprint (which you probably have to do anyway so your operations folk know how to properly configure the new system). Then, where possible:

turn as many runtime parameters into compile-time constants as possible
ask under what conditions the parameters will no longer be needed (for example, for configuration that assists with an external transition, add an item to the product backlog to clean up the codebase after the transition is successful)

Obsolete/insufficient architecture and design

Architectural debt is probably the most nefarious, because this is debt that doesn't actually get created when the code is first written. Instead, this is usually caused by external factors such as:

business environment changes and expected traffic is significantly different than originally anticipated. This can leave you either with an overly complex, over-engineered system, or with a too-simple system that can't easily scale.
product direction changes, and the architecture is not flexible along the new axis of change, so that new development is overly difficult.
expected performance of a new provisional architecture is invalidated by experience

Basically, as soon as you realize you need to change your architecture, you have magically "created" technical debt out of all the code that depended on the first architecture. In reality, this debt is probably unavoidable, and what you've really done is convert your inability to perfectly predict the future into a set of work that incorporates new knowledge about the problem domain.

This can also be hard to identify by scrutinizing the code, but there are some external symptoms of it:

difficulty meeting desired performance or scalability targets, especially when concentrated in a certain feature subsystem
adding new instances of a certain class of feature does not get easier over time
lots of bugs being generated by a specific subsystem
increased time-to-market for new features
accleration of bug creation rates
accrual of standard operational processes that require manual intervention/support

So when you have some or (gulp) all of those symptoms, you probably have architectural debt lurking in your system. Once you have identified it and have a new target architecture, a lot of this will get converted into transitional debt while you are making the changes.

Technical debt vs. technical investment

I want to be careful here to distinguish between two sorts of non-functional requirements that might show up on a "tech backlog":

technical debt: this is current brokenness or unneeded complexity in the system that is actively slowing down the business of turning product backlog into working software for your customer.
technical investment: these are things that are not necessarily broken per se, but which could speed things up for someone. A good example of this would be automating a manual process.

Technical investments can probably be put off while you have existing technical debt, although it can sometimes be hard to distinguish between the two. Clear technical debt should probably be prioritized at the top of a product backlog, unless there are really high ROI items that might trump it. In general, getting rid of technical debt will increase the ROI of everything else on the backlog, simply by decreasing the "I" part. It can also make estimation more accurate by reducing the complexity of the system to which new functionality will be added.

Whose responsibility is technical debt?

Generally, as the folks with the technical ability to recognize it, it is the development team's responsibility to try to avoid accruing technical debt while producing product. Failing that, it is their responsibility to recognize/document existing debt and to advocate for its removal. However, note that there are often symptoms of technical debt, such as those I've listed above for architectural debt, that can be recognized by non-technical folks too.

On the flip side, business folks / product owners need to be able to trade off short term wins that accrue technical debt vs. taking longer to produce a product with less debt. Communication with the tech team is of vital importance here; undoubtedly there will be times when a short-term win will be important (especially with a first-to-market situation), but it needs to be accompanied by a plan to eliminate the accrued debt. i.e. Treat your technical debt like credit card debt that should be paid down ASAP, and not as a long-term mortgage.

The interest on your technical debt is probably not tax-deductible.

The Crucible

2008-04-18T19:11:00.005-04:00

We've recently re-adjusted our sprint length to be 3 weeks, down from 4. The reasoning behind this was to allow us to align with sprints on other products, to give us a uniformity of scheduling and to permit inter-product developer swaps from sprint to sprint, if needed.

One of the side effects is that our planning process now takes up proportionately more time of the sprint and so the actual work time of the sprint is quite compressed. Couple this with a handful of developers being out for personal reasons (wedding, health issues, impending birth of a child) and suddenly this sprint had a lot of high priority sprint backlog and not a lot of available story point capacity on the teams.

But then something magical happened--what I'll call the "crucible moment." The status of the sprint was that there were a bunch of high priority, smaller maintenance tasks (cleanup of existing features, releasing the previous sprints' work to production, etc.) and then one big new feature. As the teams were working their way down the list of user stories and filling up their story points, everyone quickly realized that that big new feature might not fit into the sprint.

With the purifying fire of timeboxing (forgive my overly dramatic metaphors here), the sprint teams and product teams immediately began self-organizing and negotiating. No one wanted to have a sprint without a killer feature, and so the horse trading began. Some of the higher priority stories were off the "tech backlog" -- non-functional investments in build infrastructure, etc. Developers began identifying tech backlog items that could stand to wait. Product owners began reconsidering some of the higher priority smaller stuff, conceding that some of them might not be so important after all. Different development teams tried to juggle stories between themselves so that the large story could get onto one team's sprint backlog (we prefer not to split stories across teams, to minimize cross-team dependencies). Some stories were scoped down to take less time.

And then, when it was all done, we discovered there was enough room for two pretty big features.

This was scrum at its best--focusing on the art of the possible to squeeze as much as possible into a fixed amount of time. No time for extraneous explorations, just a focus on extracting the maximum product value out of the available resources, with a brutal willingness to critically re-examine priorities. What's more, it was carried out in a context of teamwork between the product owners and the teams; everyone was trying to figure out how to get those features in.

So, a great day for our product and for our organization. We might actually be getting the hang of this scrum thing....

Sprint Review

2008-03-26T23:14:00.004-04:00

Just wanted to follow up on my previous post about how we were going to get back to scrum basics this sprint. We had our sprint review and sprint retrospectives today, and the sprint was viewed as a great success.

Here are some things that I think contributed to our success:

Better estimation: We used past task estimation history plus monte-carlo simulation, as I described earlier, to sign up for a set of achievable tasks. The end result was that we showed up at the review with 90-95% of the committed work done. The parts that didn't get done were due to external blockers we couldn't do anything about (e.g. external partner didn't have a data feed ready) or were due to too many round-trips through the design-IA-product-eng collaboration cycle (essentially, deviating from the plan we set forth at the beginning of the sprint) that took time away from other things.

Commitment: Towards the end of the set of product backlog our team was planning, we hit a fairly large task that was hard to estimate out. However, we had two team members who were very lightly booked, and we simply asked them if they were willing to commit to finishing the task somehow by the end of the sprint. They signed up, and they got it done. There was really no talk of punting things off the sprint, although there were a couple of rescopings and/or a change of plan due to time constraints that happened to get things done by the review. In general, I think these changes actually just meant the abstract functionality in the user story got built with fewer resources -- i.e. we completed them more efficiently thanks to the pressure of timeboxing.

Empowerment: Thanks to being committed to their tasks, my teammates basically kept themselves unblocked most of the sprint by engaging with other departments (design, product, IA, ops, QA) early on to make sure they could get what they needed. Most of the time things stayed unblocked by getting the right folks together in one room. One of my teammates mentioned that one of the things I was best able to do was to simply identify which people needed to be in the room! Then he went off and arranged the meetings and got things done himself.

Shippable product: We planned explicit tasks for each feature to test/verify it, to show QA how it worked, and to demo it and get explicit signoff from the product owner. Our product owner gave me the feedback that this resulted in most of the user stories we called "done" being potentially shippable.

Self-organization: The team dynamically swapped tasks with one another to load balance effectively during the sprint. There was such appreciation for this help that people started buying each other donuts as thank-yous, and I think we ended up with around 4 dozen donuts being exchanged by the end of the sprint. Bad for the waistline, but good for team morale!

On another note, this was a re-entry for me into the dual role of scrum master and team member (in that I signed up for development sprint tasks as well as being scrum master). For other developers who find themselves being called on to carry the scrum master mantle, here are some tips to help you survive:

Only book yourself half-time. The rest of the time will be eaten up by scrum mastering, and if you don't scale back on your coding commitments, you're just going to end up staying up late working all sprint.
Tell people how to get unblocked rather than trying to unblock them yourself. This seems pretty simple, but is a big time saver. For one thing, it's probably quicker to describe what to do (e.g. just set up a meeting with X, Y, and Z) than it is for you to send out the meeting invite yourself. Plus, you remove yourself as a bottleneck -- so your teammate doesn't have to wait for you to send the invite. Furthermore, after you do this a few times, your teammates will pick it up and will be able to just solve more of their problems themselves without asking you for help.
Get good reports. I invested 5-6 hours towards the beginning of the sprint to get some automated reports set up to get info out of our ticketing system. I really ended up just using two reports:
1. Storyboard: for each story, identify who has tickets against it, and identify whether each ticket is "Not Started", "In Progress", or "Finished". This provides a quick way to scan the status of each story and reminds you to ask questions like: what do we need to do to close out that story again? why are we working on lower priority things instead of the higher priority ones? etc.
2. Burndown count: for each story, add up time remaining on each ticket, and then provide a total for the amount of hours left for the sprint.
So what I did was: right before scrum, I pulled up the Storyboard report to check status so I could ask followup questions during scrum. Right after scrum, I hit the burndown count and then actually plotted by hand on a big piece of flipchart paper the burndown graph. That was pretty much the extent of the reporting I needed.

So, all in all, things turned out pretty well. I'm excited to see the teams back in a good sprint cycle groove and turning things out (our review meeting lasted almost 5 hours, mainly because so much had gotten done across the multiple teams working on the product that it took a while to go over it all). We'll be mostly keeping things the same going into the next sprint, so I'll be curious to see if the teams are able to accelerate now that they're used to working this way.

REST: Unix programming for the Web

2008-03-16T22:12:00.007-04:00

I've been giving some thought to REST-style architectures recently, and recently re-read some of The Art of Unix Programming by Eric S. Raymond. ESR notes that some of the characteristics of Unix-style programming include:

Do one thing, and do it well (attributed to Doug McIlroy)
Everything is a file.
Comprise complex systems by connecting smaller, simpler programs (e.g. Unix pipes).

Unix-style systems have had undeniable success and remarkable stickiness for a technology; I have an old copy of my father's System V manual from when he worked at Bell Labs (yeah, they actually printed out the man pages and bound them!), and I pretty much recognize everything in there. Sure, there are many new commands available, new kernels, new distributions, etc., but they are all very recognizably Unixy.

I've been thinking about how this philosophy applies to the Web 2.0 world. I think this list turns into:

Do one thing, and do it well.
Everything is a RESTful service.
Comprise complex systems by interconnecting smaller, simpler services.

For one thing, "do one thing, and do it well" isn't limited to Unix, this is a key part of abstracting and decomposing a technical problem. But we see it everywhere: I read my mail on Gmail, keep my to-do lists on Remember The Milk, store photos on Flickr, keep my browser bookmarks on del.icio.us, etc. All these sites adhere to this principle.

But taking things down a layer, what does this mean to someone architecting or implementing a Web 2.0 service? For one thing, I think this means that it would make sense to break your overall service down into individual microservices that are individually maintained and deployed; i.e. break things down to the smallest level where they still make coherent sense.

As for "everything is a REST service", the everything-is-a-file abstraction worked for Unix because there was a small set of common operations that applied to files (open, close, dup, read, write). Sounds a lot like doing REST over HTTP, with HEAD, GET, PUT, POST, DELETE.

Combining various small REST microservices into larger services is already being done (this is, after all, basically what iGoogle is) on a one-off basis. The main question is: what is the Web 2.0 equivalent of the pipe? Namely, is there an easily understood abstraction for composing webservices? Sounds like a topic that might be rife for some kind of logical calculus (like the relational calculus for databases or the pi-calculus for concurrent processes), e.g. the REST calculus. If there were a couple of easily understand and specifiable combinators, these could be pretty easily built into some language-specific libraries for use in quickly building some new macro-services.

I might try to interest some of my old colleagues from my programming language research days to see if they have anything to say about the matter....

Comments definitely welcome here. Is this a new idea? Are others espousing this? Is this even a good idea?

Return on INVESTment

2008-02-28T23:53:00.004-05:00

At our last pre-planning meeting, we made a point of putting all the user stories into INVEST format, and I was pleased that there was a general consensus that this worked well.

I think this took quite a while, partially because we were all getting used to evaluating the statement of the stories critically, but I think this was worth it. We had product, IA, and engineering folks suggesting wordings for stories, and at least some of our stories came in in a canonical format:

"As <who>, I want <feature>, so that <value>."

Having the full INVEST filter did help us with a few things:

Independent: We did not have a lot of stories that were dependent, but we did end up with a very few that were cross-team. Since this was just a handful of the stories, we figured it would be ok to leave the story as is (since we couldn't quickly come up with a way to rewrite it) and track the dependencies through our scrum-of-scrums.
Negotiable: I think we generally nailed this one. We did not look at any IA wireframes or Design mockups during pre-planning, and were able to identify the high-level goals here. This has already become useful in planning, where my team has suggested a new / faster approach to at least one story which was not as originally conceived, but our Product Owner agreed that we still hit the abstract requirement.
Valuable: There were a few stories that were original cast as abstract technical requirements (e.g. make the middleware support foo), but we refactored them in a way that expressed value to the end user (which, incidentally, will make it more obvious how to test).
Estimable: We've already agreed not to do a small handful of stories because we realized there were too many unknowns; for one of these, there was a middleware task that depended on IA we hadn't seen and on data we didn't have in the DB yet. We actually spent a lot of time trying to talk about how to tackle this before we realized we couldn't estimate it. Since it was not critical for implementation this sprint, we agreed with the product owner to turn this into a story where we would do a feasibility study and rough design by the end of the sprint instead, which was something we could commit to. (The value here is to the product owner, who will then be able to write an Estimable story about the feature!)
Small: We did a good job here of refactoring large stories into multiple pieces and then figuring out how to get Value out of each piece. We only had a handful of medium to large stories.
Testable: We're asking each team during planning to make sure they hash out "how to demo" for the feature with the Product Owner, so we should know if we've gotten there by the end of sprint planning.

With the up-front work on the stories we did, our team has found it easier to negotiate a specific solution and in some cases to actually plan without the product owner, which is actually convenient (he's doing multiple duty so he's having to bounce back and forth between multiple sprint planning sessions!).

INVESTing in user stories

2008-02-24T16:21:00.005-05:00

In my previous post I made reference to the INVEST acronym for evaluating user stories:

(I)ndependent
(N)egotiable
(V)aluable
(E)stimable
(S)mall
(T)estable

I'd like to spend a little bit of time talking about each of these characteristics, and motivating why each is important through some anecdotes about what happens when each characteristic is not attained.

Independent: The idea here is that stories are free of dependencies from one another. (A good test would be to ask if you could implement them in any order, rather than just in the priority order generated by the Product Owner). We want this for multiple reasons: first, on a multi-team project, it allows stories to be load-balanced across teams more easily, since in theory any one story could get moved around, rather than having to move an entire batch of them. Secondly, and on a related note, it means that when sprint teams sign up for work, they can draw the commit line anywhere. Thirdly, it helps avoid cross-team dependencies (although a scrum-of-scrums and specific attention to dependent situations can handle it, it is still much easier to handle all your dependencies intra-team).

For a specific example, consider if you have teams that are layered horizontally by functional layer (e.g. a database team, and a webapp team). If you have two stories which are the "halves" of building some functionality, and assign one to each team, you take on the following risks:

teams must coordinate closely around this feature (extra communication/tracking overhead)
if one team finishes but the other doesn't, you may have extra "clean up" work at the end of the sprint to make the software still work, and you may have had a team do work that ultimately had no product impact that sprint (e.g. middleware can handle some new data and display it, but data didn't actually apppear in the DB, so no actual behavioral difference)

We'll see that some of the other INVEST characteristics actually help track this down as well.

Negotiable. To me, this means that the requirements given are as abstract as possible, so that the actual details of what is going to get built are determined by the team and Product Owner during sprint planning and modified as needed over the course of the sprint. For example, "user can change timezone with one click" vs. "user timezone is shown in a 100x25 dropdown box in the header". (Of course, if the latter language actually represented a contractual obligation, then that might be as abstract as you can make it, but you get the idea...).

This is primarily important for two reasons: to allow the team to exercise maximal creativity and to allow the team to adjust for the unpredictable events that will happen mid-sprint. In the former case, the dropdown box might be tacked on to the story as a starting point or suggestion, but the team may come up with an approach that is easier to implement or which actually satisfies the Product Owner more. In the latter case, it leaves some room to negotiate if the team has under-estimated and is behind mid-sprint, and still be able to "finish" the user story.

The danger of not being able to negotiate the functionality is really twofold: first, not completing the full set of work in a sprint eventually crushes team morale; not being able to complete an assignment is a big downer, especially when you have worked hard and because, due to the inherent complexity of software development, things just took longer than you thought. Eventually you get to the point where your team will just shrug, and say, oh well, I guess we won't finish that stuff. I have seen this happen first-hand, and it is demotivating.

Secondly, there is danger to the product roadmap. By not having requirements couched abstractly, you run the risk of slipping features or having unfinished, "carryover" work, which adds up and pushes the product roadmap back. By giving the team the freedom to brainstorm a way to satisfy the requirement in less time, you are not letting them off the hook -- you are using the deadline of the end of the sprint as pressure to encourage the team to develop the functionality as efficiently as possible. The resulting functionality may not be as complex or deep as originally hoped for or conceived, but if the spirit of the story is met, then you have some aspect of it ready to go out as shippable product.

Valuable. Each user story should provide value somehow. The original articles I read about this amended this to "provide value to the end user / customer." While I think noting the value explicitly can help product owners check off that the story will help their Key Performance Indicators (KPI), I think the more important thing this brings is that the team is aware of why this story is important. This can help constrain the solution space for Negotiation in an important way. Finally, this is just a cross-check for Indepencence; if this story is completed, and no other, does it generate value, or do we also need another story to be finished in order to get the value?

On a side note, the phrase "to the end user" is an interesting one. We keep a "tech backlog" of infrastructure/refactoring ideas around, and prior to each sprint, we evaluate which ones are critical to current development, and ask for those to be prioritized in with the current product backlog. Generally, we have been very judicious about this, maybe in a nod to providing direct benefit to an end user -- we usually wait until an infrastructure adjustment is needed or desirable for implementing new functionality before taking it on. This suggests that other infrastructure work get carried out during lab days, to scratch those developers' itches. I'm still up in the air over how strict we should be about "to the end user", but I do lean towards requiring that most of the time.

Estimable. If you can't put an estimate on it, either the requirements are too vague (see Testable), or the technical solution is unknown (e.g. "I don't even know if this is feasible!"). In the latter case, a suggested tactic would be to alter the user story into a feasibility study / research effort for this sprint, which could be easily timeboxed; the original story could then be revisited in a later sprint, when there will be less cloudiness around it, and it can be properly estimated.

Where this can bite you is in signing up for work you don't know you can finish. This sets up for an expectation mismatch with your Product Owner, and also prevents you from making efficient use of your time; you run the risk of attempting to estimate it, and either grossly underestimating it, putting all the lower-priority sprint backlog at risk, of grossly overestimating it and not signing up for enough work, or of padding the estimate to try to account for the risk and then spending more time on it than the feature is really worth.

Small. We want things that will take no more than a full sprint to do, and hopefully less. While the full scope of a feature vision may require more than one sprint, you want to refactor it somehow so that you get something out of even the very first sprint (see Valuable); then you can be sure that you reap some result from your effort in shippable product, as opposed to doing some partial work, and then having the remaining work deprioritized for several months before you can extract value.

Finally, a purely pragmatic reason for keeping the stories small is to reduce the risk of gross under-estimation. I've personally been way off on a single big story (as everyone is from time to time), and when it's been a big one, I've simultaneously trashed my personal life for a month of overtime and sleep deprivation, while requiring a bunch of load-balancing and rejuggling across multiple sprint teams due to the fact that the rest of the sprint backlog I signed up for was now at risk. So don't do that -- take smaller bites, just like Mom used to say.

Testable. This primarily serves three purposes: ensuring that Product Owner expectations are in-tune with what the team thinks it is delivering (see also Negotiable), giving the team a way to know when it can stop working on the story, and giving the testers a starting point for writing their test cases. Beware of non-quantifiable adjectives like "good" or "acceptable" in your user story descriptions. For a while, when we were doing sprint planning on spreadsheets, we had a "How to Demo" column--this worked great while we did it, but we never had enough discipline to follow through here and continue doing it. This is one of the things I'm hoping to bring back during my Scrum revival next sprint.

There's nothing worse than showing up to a sprint review and having your Product Owner say, "but that's not at all what I asked for, or that's not what I meant." Big morale crusher for everyone involved (team and stakeholders).

Finally, this gives the Scrum master a hook to save the team from perfectionism or unbounded creativity. For example, if you've gotten the feature to the point where it satisfies the acceptance tests and has been built up to your standards of quality, just stop working on it, and start working on the next user story. This is your old friend, the Pareto principle, at work -- would you rather spend a day mining the long tail of a functionally complete feature, or would you rather spend it getting the up-front meat of a new feature? The other place this helps is when you finish a feature, and you and the Product Owner are looking at it, and you now see something totally awesome that is now possible -- stop, ship the feature you have, and queue the good idea up as a user story for the next sprint so it can be properly prioritized with everything else. Again, this is about efficient use of the time in the sprint.

I'm anticipating having this be a little painful as we work through this together with the product team the first time; we've all signed off on this in principle, but we've never actually attempted to make each story adhere to INVEST. I suspect, like all similar things, it will be a bit robotic for the first few stories, but we'll quickly get the hang of it and be able to move on during pre-planning. I'll let you know how it goes.