Continuous attention to technical excellence & good design increases agility

To discuss this topic, we’ve got to start with an explanation of what “technical excellence” means. Technical excellence means maximizing technical value, which is distinct from business value.

When I say “business value,” I mean things that fall squarely in the category of being immediately and directly valuable to the business. This includes:

• working features
• the ability to add new features quickly (including the occasional “I need it yesterday” level of quick)
• the ability to test new features shortly after they are produced
• daily updates on progress from engineers
• regular planning meetings between technical & business people

Technical value, the ying to business value’s yang, means things that are valuable to the engineers. This includes:

• clean & easy-to-understand code
• automatic tests
• clear direction
• good tools
• snacks (ok, it’s not directly *technical value*, but it does help)

By focusing on things that are valuable to us as engineers, we are able to quickly and effectively deliver business value to our clients.

Let’s take the first one as an example — readable code. As an engineer, I can tell you that once a project goes over a few thousand lines of code, there are sections of the code that I might not have to look at for weeks or even months. Without small, well-named functions, and good comments (those that explain the *why* and not *what*), I will have to spend time digging to figure out what I need to change to make the code deliver new business value. Think of these things as signposts that show the organization of the code, the same way that chapters and page numbers show the organization of a book. Finding things quickly is valuable, and if you disagree, then I dare you to not use Google for a month.

A metaphor that’s useful in understanding technical excellence is the idea of “technical debt”. This was proposed by Ward Cunningham to explain how technical excellence affects the ability to make forward progress.

Let’s take Project X. Project X has really well-written code. When a fast change (the “I need it yesterday” kind) comes in, it’s safe to incur a bit of technical debt. We’ll borrow from our good credit (good code) to hack a solution into place quickly; something only possible because we’re organized. Later, when we have the luxury of time, we’ll pay off the technical debt by fixing the hack and bringing it back to our usual standards.

The analogy to financial debt is simple: if you keep your credit score healthy, when time and money are important, you can borrow against your credit line. Also, if you have too much debt, you get trapped and spend most of your resources by paying off interest. Same thing goes for technical debt, where the team spends time working around bugs, and trying to figure out what code does, instead of making forward progress on new features.

That all being said, there are responsible ways of using debt, both financial and technical. Imagine a recent high school graduate who wants to go to college, but doesn’t have the cash up front. Is it a good idea to take out a student loan? Absolutely! Take the loan, incur the debt, then pay it off when the resources become available. In this case, the student is starting a new venture with future potential — they are a startup! Same goes for a startup company.

At the beginning of a project, in the prototyping phase, it’s ok to take on debt. In Cloudspace’s line of work, our clients are often startups. This means that at the beginning of a project, the important thing is for our clients to test out their idea, get feedback, learn what works, and make changes to their plan. In engineering parlance, this means that they need a prototype. It’s ok to incur technical debt at this stage, because the business value of being able to experiment and learn outweighs any amount of technical debt. Also, there’s so little code, it’s very easy to refactor. Recovering from debt is fast. This phase usually lasts for the first few months, until 10-20 people have been able to test the prototype. After that, the learning period starts ending, and like our high-school-now-college graduate, it’s time to pay the loan back.

This should give you a good idea of how technical excellence helps keep a project flexible. By having everything organized ahead of time, you can move quickly when it comes time. If you’ve got questions, drop me a line in the comments. I’d love to discuss them with you.

dependency = Gem::Dependency.new "authlogic", ">2"
specs = Gem.source_index.search dependency
puts "the gem is not installed" if specs.empty?

Thanks to rubygems-sing for the simple implementation.

Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.

At a previous employer, I remember a coworker of mine. Let’s call him M.

M was a good guy, capable, and hard-working. He always put in what was required, and often more. On one particular project, there was a deadline coming up, and his team wasn’t going to make it. So, he buckled down and put in over 100 hours that week to make sure that they got everything delivered. Then the deadline got pushed back, giving them a bit more time, but he had to put in another 100+ hour week. Then the deadline was pushed again, with the same result — another 100+ hour week. He didn’t even go home, he’d just go to sleep under his desk at night, then climb back into his chair and keep working (except for 2 showers each week). This kept happening for a number of weeks until he gave up and left the company.

This is a perfect case of unsustainablity. I’ve definitely watched the clock roll past 4 AM to find myself still hacking away at code, but that’s because of two things. One, I like hacking. Two, it’s not long-term. Everyone can redline themselves for a short period of time, as long as there are breaks (I’m a big fan of the strike team approach, myself). Paradoxically, time spent not working is almost as important as the time spent working. It’s the nature of humans.

Agile development acknowledges that longevity is good in software teams. Turnover within a team due to burnout is bad, not just because of the loss of key members, but also because there’s a decline in quality and performance leading up to the loss. At Cloudspace, we keep moving forward at a steady pace (see my post on velocity), which means we move forward at a predictable pace. We’ll work late if a last minute feature is needed for a launch. But these situations are the exceptions, and not the rule.

The same goes for the client role. It’s important that our clients be able to keep a reasonable pace too. Daily updates and weekly meetings help serve this purpose. This type of frequent communication between the stakeholders means there aren’t last minute surprises. Remember the lesson of Mr Aesop’s friends, the tortoise and the hare. Slow and steady wins the race.

Do you have any stories of how unsustainable engineering practices resulted in the failure of a project? Share them in the comments.

class Item
  # simple default scope to add an order_by to all queries
  def self.default_search_scope(&additional)
    search = Sunspot.new_search(Item)
    search.build(&additional)
    search.build do
      order_by :created_at, :desc
    end
    search.execute
    search.results
  end

  # pardon the contrived example
  def self.with_keywords(user, query)
    default_search_scope do
      keywords query
    end
  end
end

This way you can specify your search options in the Sunspot DSL from each method, while moving repetitive options to a single location. Clean, readable, and effective.

Great props to Mr Mat for his help

What I’m writing here is the single most important take-away from my Sun years, and it fits in a sentence: The community of developers whose work you see on the Web, who probably don’t know what ADO or UML or JPA even stand for, deploy better systems at less cost in less time at lower risk than we see in the Enterprise. This is true even when you factor in the greater flexibility and velocity of startups.

Tim Bray, Doing It Wrong

Working software is the primary measure of progress.

Let’s talk what this principle means in real projects, and then get a bit more general as to why it’s good.

How does this principle show up in a real Agile process? There’s a concept we call “velocity”. We assign velocity points for new functionality — not tickets completed! The danger of using “tickets completed” as a proxy for project value is that fixing a bug means closing a ticket. But, fixing a bug means the original ticket wasn’t done right, a.k.a: the software produced wasn’t “working software”. So, if someone spends a day setting up user avatar uploads, they’ve gotten 2 points of velocity. If someone spends a few hours fixing the login system because it was supposed to use emails instead of usernames, they get no velocity points.

Velocity is very useful from a management perspective, and it also enhances productivity. When you give a system of rules for scoring “points” to a group of people, I promise you that their level of performance will go up. Really, there’s only one problem with velocity: the illusion that you can compare velocity across projects and teams.

Every group estimates every project differently. When estimating the complexity of a change (in order to give the ticket the point value), some people naturally estimate high and some estimate low. The great thing about velocity is that it doesn’t matter if you’re high or low, just that you are consistent. If the project and the team are relatively consistent, they’ll produce consistent estimates. Consistency is good, because it makes it possible to plan programming endeavors.

So, don’t think of velocity as “the bigger the number, the more functionality being added!” It’s only valuable for comparing a team and a project against itself, to answer questions like:

• The developers are busting their butts. Are we adding as much functionality as we did a month ago?
• The developers are busting their butts. Are we spending all our time fixing bugs?
• Did adding a developer improve our productivity? Roughly how much?
• Did removing a developer decrease our productivity? Roughly how much?
• Did a new process or tool affect productivity?

One extra note: if the same team and project’s velocity is highly variable, or ever spikes/drops, it’s fair to ask why.

Why does this work?

This is a hugely important principle. There’s almost nothing more important than producing working software. Beautiful code that doesn’t work is like a band filled with technical experts that sound like crap. The problem tends to be people who absolutely love and are insanely great at doing the thing, are so focused on being in love and insanely great that they don’t focus on business value. Technical purists mixed with business makes for rocky waters, but you can sail between them.

On the flip side, the average person’s definition of “working software” usually just means “when I sat down for the demo and clicked the button, I saw the expected result”. That’s not working software. Other things that contribute to “working software” are:

• handling unusual but likely situations well (not rare and arcane situations — that’s premature optimization)
• maintainability for the developers (present and future)
• usability for the end user
• free of bugs

On the subject of working software: there are many times when someone is fully capable of producing working code, but doesn’t want to when the solution doesn’t feel beautiful. Every scientific and artistic field has certain definitions of beauty or purity — the times when it all comes together. No-hitters in baseball, hat tricks in hockey, drifting cars, and math equations that are simple but convey a fundamental law of the universe. Writing software is no different — there’s certain things that can make an algorithm or architecture simple and powerful. But idealism can kill. These things are best when they come about naturally. Sometimes it takes some extra work to make them, but don’t go too far out of your way. Nothing trumps working code. I used to be guilty of this, and I’ve gotten more done since getting past it than I ever did before. I know it’s a problem in our industry, and I want others to benefit from it.

Except for one minor flaw: TextMate saves files with extended file attributes – basically arbitrary metadata associated with an individual file. The vboxsf filesystem with which you must mount shared folders in the guest OS does not support extended attributes. This results in a basically corrupted file.

You can see if a file has these attributes by performing a long listing in a directory:

jlindsey-mbp:javascripts jlindsey$ ls -l
total 160
-rw-r--r--  1 jlindsey  staff  19145 Nov 24 10:53 jquery-ui.js
-rw-r--r--  1 jlindsey  staff  57254 Nov 24 10:53 jquery.js
-rw-r--r--@ 1 jlindsey  staff   1204 Nov 24 11:33 messages-table.js

Note the “@” symbol after the permissions for messages-table.js. This indicates that it has extended attributes associated with it:

jlindsey-mbp:javascripts jlindsey$ xattr -l messages-table.js
com.macromates.caret: {
    column = 31;
    line = 24;
}

Thankfully, the fix is simple. Simply close TextMate and run this command:

defaults write com.macromates.textmate OakDocumentDisableFSMetaData 1

If you have any files that are still corrupted, opening them and re-saving them in TextMate will not clear the attributes. To do this, you can either open them with an editor (such as vim) in the guest OS and save them again, or in OS X run this command:

xattr -d com.macromates.caret messages-table.js

Substituting of course your file name, which will delete the named attribute.

(Cross-posted from my personal blog for your enjoyment: TextMate extended attributes corrupting files)

There’s a parallel here. When doing a startup (or nearly any type of software development), code written sooner is worth more than code written later. Discount your code flows. The sooner the code is written:

• the sooner you can start validating ideas internally
• you can release to customers
• you can get feedback from customers and iterate on it

When it comes to building a new product, specs can be valuable, but even better is a prototype with working code. Spending a few minutes with working code will tell you more than four people in a meeting for hours speculating on the best way to do something. What can you do now that will provide returns as time passes?

Give me your thoughts in the comments below. What’s the most valuable thing you’ve learned by iterating early and often?

I think that Roozbeh has a bit too much time on his hands…