In his foreword to Working Effectively with Legacy Code by Michael C. Feathers, Robert C. Martin says:

Legacy code. The phrase strikes disgust in the hearts of programmers. It conjures images of slogging through a murky swamp of tangled undergrowth with leaches beneath and stinging flies above. It conjures odors of murk, slime, stagnancy, and offal. Although our first joy of programming may have been intense, the misery of dealing with legacy code is often sufficient to extinguish that flame.

Besides serving as inspiration for poetic descriptions of wetlands and humorous sequences of drawings, legacy code is also something you don’t want in your Emacs. But how can you prevent it from getting there? At what point does code turn from “perfectly good” or at least “I guess it’s OK” into “legacy crap”? We need to define what “legacy code” is.

Many people will characterize legacy code as simply old, but that’s not a very practical definition. Good code doesn’t turn into legacy code on a rainy Tuesday afternoon at 2:17pm. A better definition gives us some clues on how to prevent code from becoming legacy code. You can’t stop time.

Perhaps legacy code is a feeling. It’s the feeling that the code is too hard to approach, risky to change, and difficult to understand. This makes for a slightly better definition, because it suggests some things to avoid (don’t write crappy code, duh!). However, basing a definition like this on feelings is a bit… scary.

In the book, Feathers gives a very simple definition:

To me, legacy code is simply code without tests.

This is a pretty good definition. Tests make the code simpler to approach, less risky to modify, and easier to understand. Tests protect against the very things that make legacy code legacy.

With this definition, we arrive at a shocking realization: someone in your team could be writing legacy code right now.

Do you write legacy code?

Related posts:

1. Why Write Code When You Can Remove Some?

So you make an iPhone app, and it doesn’t enjoy the success you think it deserves. Or you make an app, and you create a technology framework on the side, and they ban your framework, and your app and framework don’t enjoy the success you think they deserve. Here’s why.

First, a detour to clarify what Apple makes money from. For perspective, let’s look at Microsoft first.

Microsoft is, and has pretty much always been, in the business of selling Windows. Microsoft wants PCs to be cheap, so they can sell more Windows. The whole PC and peripherals market is fiercely competed and the profit margins for hardware vendors are really low. That’s been working out nicely for Microsoft, since most PCs still ship with a version of Windows included, and Windows is making tons of money.

Microsoft is also in the business of selling Microsoft Office. In 2008, Microsoft made as much money with Office as it did with Windows. Kind of like the fax machine was (and still is in some parts of the world), Office is the standard method of information interchange in the business world. Just like the fax machine, you need Office because everyone else has it, too.

Apple, on the other hand, is in the business of selling hardware. Despite their recent high-profile focus in the iPhone OS software, Apple is very much a hardware company.

So let’s get this straight: Apple is not making a lot of money off the App Store. Apple is not even trying to make a lot of money off the App Store.

Apple wants apps to be a cheap commodity. When there are a lot of cool apps around, and they cost next to nothing, it really makes you want to go buy an iPhone, doesn’t it? Useful, cheap apps sell more iPhones. Microsoft is the opposite: cheap hardware sells more Windows.

In this light, I find it strange how Apple is conducting some of their business towards the community of independent software developers. The latest ban on using anything but Objective C, C, C++ for native apps, for example, strikes me as quite strange. By which I mean it strikes me as completely idiotic. Banning programmers’ favorite tools is stuff which makes talented developers call it quits and stop working on the iPhone platform. How can that possibly be in Apple’s business interests?

Many iPhone developers seem to be driven by the “publish an app and get rich quick” myth. However, the app market is fiercely competed. In order to make any significant money on the App Store, your app must be really good, one of the first of it’s kind (if not the first), and you must be lucky. This is App Store poker. It’s like trying to win the World Series of Poker by entering a satellite freeroll. It can happen, but I’m willing to be it won’t happen to you.

Is App Store poker enough to maintain a steady influx of apps to iPhone OS? Oh yes, it is. It is more than enough. Like with their hardware, Apple wants the apps to be about quality, not quantity. Robert Scoble says:

So, what does Apple need? Is it more apps? No way. At 130,000 apps Apple already has enough apps to keep a sizeable lead for years over its competitors like Google’s Android OS.

No, what Apple needs is better quality apps. So, does Apple care about templated apps or ones developed in Flash or some other cross-device language/system? No way.

Apple doesn’t want your app because your app sucks. Probably. Statistically, anyway. Most apps aren’t anything special, why should yours be any different?

No related posts.

Did you know that your regex matching library may suck? Most libraries do, in certain circumstances, suck exponentially. This came to me as a great suprise when, a long time ago in the olden days before Google, iPods, and salmon skin leather, I was researching regex matching engines hoping to find a good one for use in a real-time system.

I’ll illustrate with a dusty ten-year-old picture from my Master’s Thesis:

Horror! When I showed this to my colleagues, they didn’t believe it. Everyone just sort of assumed that the regex matchers they were using every day would, you know, work. The lesson here is that some regex matchers take potentially exponential time to find a match, while others always take linear time.

This picture may be over ten years old, but the world of regex matchers hasn’t really changed a whole lot. Virtually all libraries included in programming languages or their standard libraries use so called “backtracking” matchers and suffer from this performance issue. Python, Perl, PCRE, PHP, Java, and Ruby all come with a backtracking matcher. An infinite number of combinations of regexes and input strings exist which completely choke these matchers. For a detailed explanation, I recommend Regular Expression Matching Can Be Simple And Fast by Russ Cox, where he also has these better pictures (note difference in y axis scale in the two images):

Time to match a?nan against an

So when does this behavior matter? If you let untrusted users supply regexes, you may have a security issue in your hands, a denial-of-service the very least, if your library is bad. If you want to be sure your program won’t run out of memory or out of time, no matter what regex you throw at it, you cannot use the built-in libraries.

Luckily, there are some good efficient regex libraries which guarantee linear time matching and fixed memory usage:

• RE2 by Russ Cox, released 2010
• The Plan 9 regex matcher by Rob Pike, late 1980s, early 90s?
• TRE by me, released 2001

Others may exist, but these are in library form. RE2 is mostly PCRE compatible. TRE is POSIX compatible, and also supports approximate matching. The Plan 9 matcher has its own API.

So, there you have it. Parsing text is one of the things which is best done with a suitable theoretical background. You can’t parse HTML with regex, and you cannot efficiently parse a regular language with a backtracking matcher.

Related posts:

1. Approximate Regex Matching in Python
2. Overriding System Functions for Fun and Profit

There are few hard and fast rules in software development, but this one comes close: don’t copy and paste code. Keeping multiple copies around within the same code base is almost never a good idea.

Question is, do you have duplicate code in your code base? How much would you be willing to bet?

Recently I stumbled across a great tool called Simian. It scans through your code and reports any duplicate lines. It understands enough programming language syntax so that it can ignore comments, whitespace, curly braces, and variable names.

Simian hasn’t been updated for a couple of years now, but it works. It’s super easy to point it to your code and have it find duplicates. I encourage you to try it. Unfortunately, you may be in for an unpleasant surprise.

Friends don’t let friends copy & paste. That’s why you’ll want to set up this tool (or something similar ;) to run nightly against your code base to see if you’re getting better or worse. I’ve built a “total duplication” count and a “top 10 biggest chunks of duplicate code” reports, updated every night.

Related posts:

1. If Your Code Crashes in Production, Does it Make a Sound?
2. Get Rid of Source Code Templates

When I was a young student, I had a fashionable infatuation with martial arts.   One day at the dojo I was busy practicing a particular punch to the side of the neck.  The teacher watched my repeated fumbling for a while, and finally pulled me aside and gave me one of the best lessons I’ve ever received on how to become a better software developer.

Why learning only one thing is hard

“Listen, you won’t learn the punch right by practicing it in isolation.  You’re going through the motions but you don’t know what you’re doing.  You’re going to need some context.”

OK, I thought, that seems like a good idea.  I’m not getting anywhere anyway.

“Let’s make the punch a part of a story,” he said.

So he taught me a kata.  It incorporated the punch into a sequence of moves against an opponent.  Before learning the kata I did basically know how to execute the punch.  But the kata showed me why I was doing it and that made all the difference.

My teacher’s lesson applies to software development as well.  The part you’re working on is part of a bigger story.  You have to know that story.  It is much easier to do the right things if you teach yourself a little bit about the customers, the business, usability, technology, or anything which is not your specialty but is happening around you anyway.

Are you a fake specialist?

When you pick a specialty, strive for technical depth.  As Chad Fowler says in The Passionate Programmer:

Too many of us seem to believe that specializing in something simply means not knowing about other things.

This hits the nail on the head.  If  ”specialist” does not mean technical depth, then all that is left is an excuse to avoid learning anything else.

So, here is my advice. Be both.  Be a generalist so you can understand the whole story.  Be a specialist to do your part better than anyone else could.

No related posts.

A matrix organization allows you to build teams with a high degree of specialization and top-notch skills, plan your resourcing easily, and encourages sharing knowledge between product teams. A matrix organization is great. Except that it’s not, not for developing software.

It’s not great when coders don’t do any testing because it’s the QA department’s responsibility. It’s not great when it takes a week to go through the DBA team to have a column added to a database table. It’s not great when developers have never directly talked to an actual end-user.

Please take a number. We will be able to look at your request in three or four days.

This is why startups get things done quickly. They have a small number of people, and most of them can and will do everything. There is no DBA team and no red tape. You can just add that damn column when you feel like it.

If you have separate departments for customer support, QA, development, product management, and so on, you should seriously consider tearing down those silos and replace them with real teams.

Of course, there are limits to what responsibilities teams should have to be successful. You probably shouldn’t have to set up your own network infrastructure, negotiate the rent, or unblock the toilet. If you have someone to do these things for you, that’s great. If you don’t… well, that’s life. I would rather unblock a toilet myself than try to keep it in. You?

Related posts:

1. The Birth of the Grumpy Asshole Programmer
2. Do You Make These Mistakes When Recruiting Software Developers?

I have a confession to make. I occasionally fall prey to a most basic mistake: I think a tool will solve a problem before I even understand the problem.  Here’s what happened this time…

Like any company, we have an intranet. Ours is a motley collection of internal websites and tools: we have a wiki, a bug tracker, documents on file shares, and a number of other services of varying importance. Finding information is hard if you don’t know where to look. So I figured the best way to solve the problem is a Google Mini.

A Google Mini is your own private google rolled into a single rack unit. It crawls your network, indexes your content (including PDF and various Microsoft formats), and comes with the familiar google search interface. It sounded like the perfect thing to solve our little problem.

So I ordered one. The device arrived less than 24 hours after the order which was impressive given that it was shipped from the UK to Finland. Setting it up was easy and everything worked as advertised.

The obvious next step was to let the bot loose to crawl our network.  I could hardly wait to get all the information in our intranet at my fingertips.  What could go wrong?

Lesson 1: Some services choke when crawled.

It hadn’t occurred to me that our internal web servers had never been crawled before. Oops.

Some pages would take several seconds of CPU time to generate and googlebot was pinning the CPU to 100% on these servers.

Some servers generated never-ending lists of URLs, so googlebot was constantly finding “new” pages.  It quickly went over the limit of number of documents supported by the device.

Most of these problems were solved by blacklisting certain URL patterns in the Google Mini configuration.  Also, instead of a “crawl everything” approach, I switched to a “crawl only selected servers” approach.

Lesson 2: Some services cannot be crawled at all.

Most of our internal web services are not designed to be crawled at all. They may require logging in, for example, and Google Mini may have trouble handling that. None had a proper robots.txt and sitemap.  I’m still working on getting some of that stuff indexed at all.

Lesson 3: Indexing bad content won’t improve anything.

It’s like shining a bright light on a big pile of crap. It’s still a pile of crap, but you can see it more clearly.

Our intranet is hardly built using good SEO practices, so it’s natural that googlebot has trouble figuring out what pages and keywords are important.  But mostly the problem is just an abundance of obsolete and outdated content.  The solution will be to to get rid of most of it.

Don’t get me wrong: our Google Mini works very well, it’s already been quite useful, and I’m quite happy with it.  I can actually find stuff from our wiki now.  But it won’t turn a collection of obsolete Word documents from ‘02 into useful content.

No related posts.

Meet Frank. Frank works as a product manager at Faxes-R-Us. He’s finishing up a presentation on their latest fax machine model: FRU-1221i with integrated Twitter and Facebook support. This one will sell like hot cakes.

Frank’s presentation is due tomorrow, and he’s now trying to include pictures on some of the slides. At Faxes-R-Us, they have a company policy to build all presentations using SlidePointX. This is where you come in: SlidePointX is your product. As Frank drags a picture of the FRU-1221i into SlidePointX, the program promptly crashes. No worries, mutters Frank and tries again. Another crash. Again. Crash. Again. Crash. Frank starts to panic. Again. It worked! Phew.

What do you think will be Frank’s next step? Will he reproduce the bug, maybe inspect some crash dumps, and write a nice detailed bug report and send it to you? No, I didn’t think so either. All Frank wanted is to finish his presentation, so that’s what he’s going to concentrate on. Your product just crashed four times in production use, and you will never hear about it.

Enter automatic error reporting.

Automatic error reporting, a.k.a. crash reporting is an application or technology which captures software crash data and sends it to you – at the user’s consent, of course. The particulars vary between different solutions, but typically you get access to crash dumps or at least stack traces.

Automatic error reports are invariably much more detailed than anything a normal user could ever give you. There are few things in life more frustrating than trying to reproduce a bug based on inadequate information. A full crash dump is pure bliss compared to “crashes when dragging pictures, please fix”.

But that’s not where the real power of automatic error reporting lies. Here’s what Microsoft has to say about Windows Error Reporting (WER):

Broad-based trend analysis of error reporting data shows that across all the issues that exist on the affected Windows platforms and the number of incidents received:

• Fixing 20 percent of the top-reported bugs can solve 80 percent of customer issues.
• Addressing 1 percent of the bugs would address 50 percent of the customer issues.

Let me repeat that last part in case you missed it: Addressing 1 percent of the bugs would address 50 percent of the customer issues. You can’t do that if you don’t know what the top bugs are. Automatic error reporting provides you with invaluable intelligence of the stability of your product across different versions and over time.

WER is built into Windows. If you ship applications on Windows, you need to start looking at your WER reports. I insist. To get going, the only cost for you is to get a code-signing certificate from VeriSign and you’re all set for WER. Don’t just sit there, go do it now!

On other platforms (Mac, Linux, commercial Unix), the sad state of affairs seems to be that there are typically no built-in tools that software vendors can use. If you want automatic error reports, you may have to implement it yourself. If you know about non-Windows solutions in this area, please let me know in the comments.

One of your goals should obviously be that there are no crashes in the field. But just in case there are problems, wouldn’t you like to see the full gory details?

Related posts:

1. Platforms Come With a Culture
2. Metric of the Month: Duplicate Code
3. The Birth of the Grumpy Asshole Programmer

Every so often, I get drawn into a discussion about the merits of static typing (Java, C, C#, ML) versus dynamic typing (JavaScript, Python, Ruby, Scheme). Many programmers seem to be firmly in one camp or the other. But somewhere in between, there’s a third door. Let me explain…

I hate dynamic typing.

Dynamic typing leaves too much room to leave mistakes in your code. Stupid ones, too, the kind where you try to subtract apples from golf clubs. And the kinds where you sort of forgot to implement the “else” part.

These are typical mistakes which could be proved, by the compiler, to never possibly work. Only it’s not even trying, and it’s leaving it all up to me, a dodgy forgetful emotional interrupted tired inferior human, to stare at it long enough to assure myself that it will work correctly in all possible situations. It won’t, of course.

I hate static typing.

Static typing makes you think about types when you want to think about your program. Before you can run anything, the compiler insists that everything is perfect, but you’re only just starting to build the shape of your code. You probably don’t know what it’s going to look like when it’s finished. Perfect is for later.

I hate lack of type inference.

Spelling out the types for each variable and parameter and return type and field makes your programs ugly. Most of that type information is implicitly clear to you, so it makes no sense that you have to always explicitly mention every type in the actual code. The compiler and IDE should do some of that for you.

I hate type inference.

Full-blown type inference makes type errors insanely difficult to locate. The compiler figures out that the types in your code aren’t adding up, but it has no idea where the error actually is. The problem is that often you have no idea, either.

The point I want to make is that the choice between dynamic and static typing is a false dichotomy. You don’t need to choose between showering and brushing your teeth. You do both. In the same way, you shouldn’t have to choose between static typing or dynamic typing. You should be able to use the appropriate tools in the appropriate places at the appropriate time. The right kind of static typing is discretionary static typing.

Luckily, programming languages are already moving in this direction. Yesterday, Google published a new programming language called Go:

Go is an attempt to combine the ease of programming of an interpreted, dynamically typed language with the efficiency and safety of a statically typed, compiled language

That sounds like just what the doctor ordered. C# from Microsoft is also steadily gaining some type inference and dynamic typing features. Both C# and Go are actively trying to fix some of the biggest pain points in the working programmers’ lives.

I for one will be taking a good look of what’s behind the third door. Will you?

Related posts:

1. Copy-On-Write 101 – Part 3: The Sum of Its Parts
2. The Essence of Lambda
3. Programming Problems in Disguise

I decided to change the name of this blog to something more descriptive and less “meh”. This blog is about developing software and managing development teams. That is, about hacking, and about being a boss. Frequently at the same time.

It is likely that RSS subscribers will see old posts coming up as new. I apologize for the inconvenience.

And now, back to our regular program.

No related posts.

When I was studying Scheme on a programming course at HUT, I remember I was a bit baffled by first-class functions. My previous experiences were with C, Pascal, Basic, and other languges which didn’t have first-class functions. Data is data, code is code. You don’t go mixing those two. Why would you want to create new functions on the fly and pass them around as values? Crazy talk!

What really made me understand was the following homework exercise:

Define cons, car, and cdr without using any built-in data structures. Hint: Use first-class functions.

As you may know, cons is a function which takes two arguments and returns a single value which contains the two argument values inside. This single value is called a pair (or sometimes a cons cell). The car and cdr functions extract the left and right parts of a pair, respectively.

Without further ado, here’s the solution. I suppose most readers aren’t necessarily familiar with Scheme, so I’m including both JavaScript and Scheme code.

function cons(a, b) {
  return function(x) {
    if (x) {
      return a
    } else {
      return b
    }
  }
}
 
function car(p) {
  return p(true)
}
 
function cdr(p) {
  return p(false)
}

This really helped my understand what this lambda business is all about. It’s more than a function without a name. It’s less magical than generating code on the fly, which was what I first believed must be happening. A first-class function is a way to bind together some values (a and b) together with some code operating on those values (the if). The result is a function you can call like a regular function.

Sometimes, I’m a bottom-up kind of guy, so the final missing piece of understanding had to come from figuring out how lambda could be implemented by the compiler or interpreter. Here’s how:

function cons_f(a, b, x) {
  if (x) {
    return a
  } else {
    return b
  }
}
 
function cons(a, b) {
  return [cons_f, a, b]
}
 
function car(p) {
  return p[0](p[1], p[2], true)
}
 
function cdr(p) {
  return p[0](p[1], p[2], false)
}

So when the compiler encounters a function body which refers to variables outside the body, it kind of lifts the lambda into a top-level function. Any open variables are made into new explicit parameters for the function. The function value is basically a vector with a reference to this “normal” function, plus the values of the open variables at the time.

Technically this is not exactly how things happen, but hopefully you get the point. The point being that deep down, lambda is really not that much more special than allocating a vector.

As for the solution itself, we can actually do a bit better. We don’t really need to use a conditional:

function cons(a, b) {
  return function(f) {
    return f(a, b)
  }
}
 
function car(p) {
  return p(function(a, b) { return a })
}
 
function cdr(p) {
  return p(function(a, b) { return b })
}

We can go even more basic than this. The above code uses functions with two arguments. Let’s make it curried, and only use functions with one argument. It’ll change the API, though, so instead of cons(a, b) you call cons(a)(b).

var cons =
  function (a) {
    return function(b) {
      return function(f) {
        return f(a)(b) 
      }
    }
  }
 
function car(p) {
  return p(function(a) {
    return function(b) {
      return a }
  })
}
 
function cdr(p) {
  return p(function(a) {
    return function(b) {
      return b
    }
  })
}

This is arguably as basic as it gets. We’ve defined cons, car, and cdr using nothing but function definitions and function calls.

Hopefully this little exercise helped you as much as it helped me. They say you don’t really know a programming language until you’ve learned to hate it with a passion. I’d say you don’t really know a programming language until you’ve implemented a compiler for it.

Related posts:

1. Programming Problems in Disguise
2. Overriding System Functions for Fun and Profit
3. The Three Doors of Type Systems

A long time ago, I was actively playing a MUD. A MUD is like World of Warcraft, only on a much smaller scale, and there are no graphics. It’s all text based. To make your character do anything, you have to type commands at the prompt.

The thing with this MUD was that fights were not strictly turn based. If you were too slow to cast your next spell, your wizard would be standing there with a vacant look in his eyes while some troll is bashing him to bits with a morning star. The faster you type, the faster you cast the next spell.

After a few hundred hours of playing, typing no longer exists as a separate activity. It’s automatic. You think “fireball” and the words appear on screen. Technically, your fingers are still pressing keys, but for all practical purposes your mind is now directly linked with the game.

The same kind of mind-to-machine link forms after enough practice with your programming language and IDE, or shell. You don’t need a sci-fi socket on the back of the skull for that. All you need is enough practice.

You also need a good keyboard.

A good keyboard is one which you forget is there. A good keyboard doesn’t jam the Enter key when you press it on the upper right corner. A good keyboard has zero packet loss from your brain to the computer.

My new favorite keyboard is the Apple aluminium keyboard. The tactile feedback is just right. It’s small, so there’s more room for other stuff on my desk. It’s thin, so I don’t need a separate wrist rest at all. It looks beautiful, and is obviously the result of a great deal of design work. This keyboard even sounds good.

I would be very interested to hear what you have to say about keyboards. What is your favorite keyboard? Give your opinion in the comments section!

Related posts:

1. A Little Known Way to Learn Touch Typing

How well do you know the fundamental unit of information, communication, computation?

When interviewing coders, I often like to ask them to explain how they would implement a function which counts the number of bits set in an integer. This is often called the “population counting” problem. In Beautiful Code: Leading Programmers Explain How They Think, Henry S. Warren, Jr. gives a wonderful treatment of the population count problem. This C code implements a clever algorithm to compute number of set bits in one 32-bit word:

/* Counts 1-bits in a word. */
int pop(unsigned x) {
  x = x - ((x >> 1) & 0x55555555);
  x = (x & 0x33333333) + ((x >> 2) & 0x33333333);
  x = (x + (x >> 4)) & 0x0F0F0F0F;
  x = x + (x >> 8);
  x = x + (x >> 16);
  return x & 0x0000003F;
}

Can you figure out how it works?

For a really hard-core puzzle, try to figure out this piece of code from the Quake code. It computes the inverse square root of a floating point number:

float InvSqrt(float x){
   float xhalf = 0.5f * x;
   int i = *(int*)&x;
   i = 0x5f3759d5 - (i >> 1);
   x = *(float*)&i;
   x = x*(1.5f - xhalf*x*x);
   return x;
}

Wow. If you need to cheat (I did), the web is full of explanations. The one that really helped me understand was by Kalid Azad at BetterExplained.

I firmly believe that every programmer should try to build a good working understanding of the bit world. It’s not that you have to know that stuff to be able to write (most) programs. But look at it this way. That computer on your desk has a CPU in it. That CPU is executing billions of instructions per second to execute your code. Are you really not interested at all on what’s going on in there? Don’t you want to know why computers suck at math?

If the next coder you interview has trouble answering more than a couple of the following, that should be a no hire:
The next time you are interviewing a coder, here are some questions you could ask to probe their understanding of the binary system:

• What are the first ten powers of two?
• What is 0×7F in binary? In decimal?
• What does two’s complement mean?
• How would you extract the 8 low-order bits from a word?
• How would you count the number of bits in a word?
• What arithmetic operation does shifting right by one bit correspond to?

To me, total lack of understanding in this area is a big red flag and should cause a no hire. It means that you either haven’t been programming long enough to encounter these bit-level problems, or worse, you did encounter them but gave up for one reason or another.  It’s a real confidence booster to know that you can, if need be, go all the way down to the bit level to solve a problem or understand a bug.

It’s all ones and zeroes, right? The bits are your friends. How well do you know them?

No related posts.

Do you remember your last quarrel over whether the current architecture of your software is a good one? Were you defending or attacking?

I’m sure some parts of that discussion weren’t completely rational. Here’s a list of some of my favorite irrational arguments against fixing a broken architecture. Maybe the list helps you recognize them as irrational the next time you hear them or use them yourself.

1. It’ll be fine after these fixes

This one is popular among people who have never really written a lot of code, at least not of the high quality kind. They view software quality as something than can be added later. They think quality is something which can be poured in later, in controlled doses, carefully avoiding adding too much to avoid spending too much time on quality which is above the market expectation.

High software quality has to be built in right from the start. You can’t pour it in afterwards. If the design is poor, no amount of bug fixing will get you to rock solid. If you keep fixing it long enough, you might get to “works most of the time”, but you will never reach rock solid.

2. There is no problem

Flat-out denying that there is any problem is a powerful trick. Deny all evidence. If there is evidence, place the blame on third parties, the operating system, the users, whatever. If you have to, shoot the messenger.

The tendency to employ this technique seems to be stronger in people who have spent considerable amounts of time designing and creating the broken architecture. You probably define yourself through the work you have done, at least to some extent. What was the last thing you created which you are proud of? How hard would it be to admit that your creation is, in fact, a load of crap? I bet it would be easier for you to find fault somewhere else, or simply deny that there is anything wrong in the first place.

3. Developers always want to rewrite everything anyway

The power of this technique lies in the fact that there is a grain of truth in it. Many developers have this built-in tendency to build new stuff instead of patching up the old stuff. For me, I think it has something to do with how I was introduced to programming. I fell in love with creating new impressive things, and that’s what I want to do again and again.

However, that grain of truth is not a logical argument against fixing your architecture. If you know that the smelly guy down the hall always eats at Burger King and never at McDonalds, you still have no information about which is the better choice for lunch. All you know is that the smelly guy likes his whoppers.

4. It was built by our best engineers

The architecture was designed by your very best engineers, so it cannot possibly be a bad architecture, right? Besides, those engineers are too important to risk making them mad. If we can’t trust our best engineers, who can we trust?

Smart people do stupid things all the time. It’s often impossible to know beforehand how something turns out to work in real life. The smart thing to do is to adjust, refactor, redesign, and rewrite as necessary. It’s not very smart to keep going with the first thing that kinda worked.

5. This architecture will give us benefits in the future

This technique works best if you are vague. Things such as scalability and flexibility sound good, as do promises about how the architecture can be leveraged in the future to rapidly build new and innovative applications. Being very specific and concrete will probably reveal that the architecture is not, in fact, superior to the alternatives.

So, that was my list of favorite techniques to bury your head in the sand. When was the last time you used them yourself? What would you add to the list?

No related posts.

Porting software to another platform is tricky business. Take the first version of Safari for Windows as an example. It was not a great success, because Safari 3 on Windows did not quite fit in:

Other grumbles were more because Safari seems like a Mac-application, making it seem out of place on a Windows desktop.

Mac-users, firmly convinced of the superiority of the Mac platform may assume that a Mac-app running on Windows should “obviously” appear superior to Windows users. That’s not the case.

Even if you were to accept that the font rendering in Safari 3 is superior to Windows standard font rendering, the difference sticks out like a sore thumb. In Safari 4 for Windows Apple fixed most of the issues:

Safari now looks more like a standard Windows app. Previous versions stuck with the iTunes/OS X brushed metal interface, which stood out for several reasons: the buttons and scroll bars were standard OS X Aqua elements, rather than standard Windows items.

So, when porting to a new operating system simply getting it to run is nowhere near enough. Operating systems come with a culture, complete with their own installers, GUI style, jargon, and soft drink preferences. Your port has to match the culture.

Sometimes you need to port between platforms which aren’t operating systems. You might feel the need to support two or more different RDBMS’s, for example. SQL is a standard, right? What could go wrong? Jim Melton, the editor of the SQL standard for 20 years, said in SE Radio Episode 137: SQL with Jim Melton:

“It’s unclear exactly what the driving factors were for most of the organizations [for standardizing SQL]. I know for some of the vendors who eventually became dominant, ones like Oracle, IBM, and Sybase, the driving motivation was the ability to have the illusion of portability. Yes, I’m being honest here – so that you could sell your system to a customer and allow him to believe that there was some chance he would be able to port his programs, and that there wouldn’t be vendor lock-in.

Not all standards were born alike, huh?

So, all this is evidence that you shouldn’t take porting lightly. It’s easy to underestimate the cost or downplay the importance of porting the culture. And the technical side ain’t always a walk in the park, either.

After the initial port, you’ll need to develop, test, support, and maintain the software on two platforms instead of just the one. Introducing a second platform can effectively double the surface area for bugs to attach to. Also, once you’ve opened that door, why not add a third platform? A fourth? Are you sure it’s going to be worth it?

Related posts:

1. 6 Tips for Small Software Vendors to Understand Enterprise Customers
2. How to Distribute Commercial Python Applications

A colleague recently pointed me to the book Thinking Forth by Leo Brodie.

While it is heavily a book about Forth (the programming language), it’s even more a book about programming and problem solving in general. I don’t know Forth, and have no intention of learning it, but I found this book to be chock full of insight and humor which apply today just as well as they did when Thinking Forth was first published.

Among the fun parts are gems like this:

Among the pearls of wisdom, of which there are many, you can find this:

“You don’t understand a problem until you can simplify it.”

And this:

“Start simple. Get it running. Learn what you’re trying to do. Add complexity gradually, as needed to fit the requirements and constraints. Don’t be afraid to restart from scratch.”

Today, this would probably be called some sort of an agile method. I’m kind of thinking it’s just common sense.

Especially the first half of the book concentrates on universal concepts like analysis and planning, conceptual modeling, problem-solving techniques, and refactoring. This is how Brodie describes the book himself:

“We need a consistent and practical methodology for thinking about software problems. That is what I have tried to capture in this book. Thinking Forth is meant for anyone interested in writing software to solve problems. It focuses on design and implementation; deciding what you want to accomplish, designing the components of the system, and finally building the program.”

If you’re old-fashioned like me, you’ll want a dead-tree version.  The book is published under the Creative Commons license, so you can download the PDFs for free from the Thinking Forth Project website.

Thinking Forth was first published in 1984. It was around that time when I wrote my first computer program. This is certainly one of the books I wish I had read sooner.

No related posts.

This is a story, based on true life, on how you turn from a happy programmer into something sinister.

Stage 1: The happy creative programmer

You’re developing a new exciting product.  You’re the senior programmer in a small and talented team.  You’re responsible for a lot of the core functionalities - you get to write a lot of code and make a lot of the important design decisions. Things go swimmingly, and you’re having the time of your life.

Stage 2: The proud father

You and your team ship the first version of the product.  Despite the highly embarrassing bug in the installer, someone buys it.  You’re now spending most of your time on improvements, bug fixing, and polishing the existing features.  It’s not quite as fun as implementing all that new stuff, but still very rewarding.  It’s your baby, and you want to take care of it.

Stage 3: The product guru

You’ve ironed out the biggest kinks, and the product is selling well.  Your user base is growing, and you’re busy planning what cool features to put in the next big release. However, with the growing number of users, the support requests start rolling in. The support engineers don’t yet quite know the product, but you’re happy to help (your baby).  By now, there’s enough questions and inquiries coming your way to be a bit of a distraction, but you can still find stretches of a couple of hours or even a full afternoon here and there to work on the next version.

Stage 4: The grumpy asshole programmer

The next big version ships.  The customer base keeps growing still, and together with it, the volume of tricky problems grows fast.  The support staff is competent enough to take care of all the easy cases, so you’re left with the hairiest problems.  Since you’ve been so helpful in the past (your baby), many people have developed a habit of just coming to you when they encounter a problem they can’t immediately handle themselves.  You try to counter by producing documentation, and it helps a little.  Still, you spend an increasing fraction of afternoons with a queue outside your door.

Finally, you develop a fatalistic attitude about getting anything done.  You’re still the lead developer.  How in the hell are you supposed to get anything done when there’s someone constantly bothering you about how this-or-that works, requesting help in troubleshooting customer issues, and demanding estimates on how long it would take to add this-and-that feature?  Before, you got almost no email.  Now, you’re afraid to open your email.  In fact, you avoid opening your email before lunch just so you could get something done.  In response, people begin calling your mobile and checking by your office if you don’t answer their email in 10 minutes.  You stop even trying to code, because you would be interrupted anyway.

You stop being nice to people, hoping that they would in turn stop asking you things.  It doesn’t help, since you’re the resident expert on the product.  You throw empty Pepsi cans at people coming to your door, and fantasize about throwing full ones. You hide in meeting rooms and turn off your phone.  In the evenings, you fashion tiny phallic figurines out of Blu-Tack and stick them on product managers’ doors.   None of this helps much, but it kind of makes you feel better.

No matter what you try, you are trapped and there is no escape.  You have become the grumpy asshole programmer.

No related posts.

This is a breakdown of yearly expenses for one software developer (in USD), highly irresponsibly gathered from various sources on the internet.  For illustrative purposes only, parts may be missing, your mileage will vary, batteries not included.  But, I think it’s not that far from the truth.

So, what does this picture tell me?  Lots of things.

The opportunity to save on, say, office chairs is extremely limited.

You really really don’t want high turnover.  Any extra salaries you have to pay really hurts.

Spending an extra 20% on an extra nice office is worth it, if you get an extra 2% of productivity from your developers that way. That’s about 9 minutes per day.

You want to maximize the bang for the buck you get from the big fat red blob.  If you don’t give your software developers proper tools, such as two good monitors, plenty of RAM, and a proper chair, you are a raving lunatic.

What does the picture tell you?  What does it tell to your boss?  Your CFO?

Update: The references for this data:
PayScale – Software Developer Salary, Average Salaries
How much does an employee cost?
IFMA workplace study: Average space per employee
InformIT: The Cost of Managing Paper: A Great Incentive to Go Paperless!
2009 Health Care Cost Survey Reveals High-Performing Companies Gain Health Dividend
Herman Miller chairs prices (assumed to last 10 years)
Prices for monitors and RAM you can check e.g. at newegg. Assumed to last a year or two.

No related posts.

Selectively overriding functions in shared libraries is a little known but simple enough trick. You too can replace system functions with your own versions, or hook into them to add extra functionality.

What follows works as-is on most Linux distributions. For other Unix flavors you may need to tweak a thing or four, but the general principle is the same.

Dynamic linker basics

Executable programs almost always depend on a number of shared libraries. The exception is statically linked executables, but they are nowadays exceedingly rare. You can list shared library dependencies with the ldd command. For example, /bin/date depends on a number of libraries:

$ ldd /bin/date
        linux-vdso.so.1 =>  (0x00007fffd51fe000)
        librt.so.1 => /lib/librt.so.1 (0x00007f0dccd6b000)
        libc.so.6 => /lib/libc.so.6 (0x00007f0dcca09000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x00007f0dcc7ed000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0dccf74000)

When you execute a program, the dynamic linker looks at this list of libraries. It locates the libraries on the filesystem based on configuration files and environment variables, loads the libraries into memory, links the pieces together to make a working whole, and finally executes the program.

The dynamic linker on most modern Unix flavors has a feature where you can load additional libraries to programs and selectively override functions in other shared libraries. On Linux, this feature is available via the LD_PRELOAD environment variable.

How to override functions

Notice “libc.so.6″ on the ldd output? That’s the C library. It provides the functions in the standard C library, such as malloc(), printf(), and localtime().

To override a particular function, you simply build a shared library which exports that function. You can get a hold of the original definition of the function using dlsym. Here’s a minimal example:

datehack.c:

#define _GNU_SOURCE
#include <time.h>
#include <dlfcn.h>
#include <stdio.h>
 
struct tm *(*orig_localtime)(const time_t *timep);
 
struct tm *localtime(const time_t *timep)
{
  time_t t = *timep - 60 * 60 * 24;
  return orig_localtime(&t);
}
 
void
_init(void)
{
  printf("Loading hack.\n");
  orig_localtime = dlsym(RTLD_NEXT, "localtime");
}

Build this into a shared library:

gcc -Wall -fPIC -DPIC -c datehack.c
ld -shared -o datehack.so datehack.o -ldl

And we’re ready to roll:

$ date
Wed Sep 23 18:56:08 EEST 2009
$ LD_PRELOAD=./datehack.so date
Loading hack.
Tue Sep 22 18:56:11 EEST 2009
$

Hey presto! When datehack.so is loaded, we get yesterday’s time from localtime. As you can see, the _init() function is a bit special: it is called automatically when the shared library is loaded into the host process.

Some fun

In the previous post I already mentioned libtre. Here’s a trick you can do to introduce approximate matching capability to any (dynamically linked) binary which uses the POSIX regex API.

First, compile libtre with the system ABI compatibility enabled:

wget http://laurikari.net/tre/tre-0.8.0.tar.bz2
tar xjf tre-0.8.0.tar.bz2
cd tre-0.8.0
./configure --enable-system-abi
sudo make install

Then load it in your favorite program. I use “less” a lot so let’s use that as an example. Let’s run it on the TRE README file:

LD_PRELOAD=/usr/local/lib/libtre.so.5 less README

To do a regex search, enter / followed by your regex. Try this:

/\<(complier){~3}\>

The above regex uses libtre’s syntax for approximate matching to match “complier” within tree errors. The \< and \> match at the beginning and end of a word, so the regex won’t match partial words.

This search turns up matches for words like “compliant”, “complete”, “compiled”, and “compiler”.

Not just a toy

There are a bunch of tools which use the LD_PRELOAD trick for something useful. Perhaps the most common use is overriding malloc(), free(), and friends to detect memory leaks and such. One of the best such tools is Valgrind. Valgrind does a whole lot more than just memory leaks, and I highly recommend it especially to C programmers.

The socksify tool intercepts calls to the connect() function (among others), and reroutes TCP connections through a SOCKS proxy. That’s highly useful if your corporation is suffering from a highly paranoid IT department which allows connections only through SOCKS.

Fakeroot makes it look like you can access the filesystem as root without actually being root. This allows you to create tarballs and other packages with uid 0 files in them, without having to use root privileges.

So, there’s a lot you can do with this little trick. Your imagination is the limit. What do you want to override today?

Related posts:

1. The Essence of Lambda
2. Is Your Regex Matcher Up to Snuff?

We all love a little regex hacking now and then. I loved it enough to even write a regex matching library called libtre. The cool thing about this library is that it supports searching for approximate matches.

The approximate matching features of this library are being used for things like improving OCR results, generating “did you mean?” suggestions for users’ searches, and filtering spam. The library comes with a 2-clause BSD license so you can use it for pretty much whatever you want.

Installing the Python extension

The extension works with Python 2. Python 3 is not yet supported.

Windows

Install the prebuilt Python extension: tre-0.8.0.win32-py2.6.msi. The package works with Python 2.6 from python.org.

Linux, OS X, and other Unix

Although you can probably find prebuilt binaries for your system, the Python extension is usually not included. First, either install some of those binaries or compile from source:

wget http://laurikari.net/tre/tre-0.8.0.tar.bz2
tar xjvf tre-0.8.0.tar.bz2
cd tre-0.8.0
./configure
sudo make install

Then, build and install the Python extension:

cd python
umask 022
sudo python setup.py install

Taking it for a test drive

The first thing you can try is run the example program included with the extension. Here’s the code for example.py:

import tre
 
pt = tre.compile("Don(ald)?( Ervin)? Knuth", tre.EXTENDED)
data = """
In addition to fundamental contributions in several branches of
theoretical computer science, Donnald Erwin Kuth is the creator of
the TeX computer typesetting system, the related METAFONT font
definition language and rendering system, and the Computer Modern
family of typefaces.
"""
 
fz = tre.Fuzzyness(maxerr = 3)
print fz
m = pt.search(data, fz)
 
if m:
    print m.groups()
    print m[0]

When you run example.py the output should look like this:

tre.Fuzzyness(delcost=1,inscost=1,maxcost=2147483647,subcost=1, maxdel=2147483647,maxerr=3,maxins=2147483647,maxsub=2147483647)
((95, 113), (99, 108), (102, 108))
Donnald Erwin Kuth

So what does this program do? First we compile the regex Don(ald( Ervin)?)? Knuth. This simple regex matches any of the following three strings:

• Donald Ervin Knuth
• Donald Knuth
• Don Knuth

The interesting part is obviously this:

fz = tre.Fuzzyness(maxerr = 3)
m = pt.search(data, fz)

This searches for an approximate match to the regex in data, with at most three errors. The search finds “Donnald Erwin Kuth” which matches our regex within those three errors.

Some more details

The tre module contents are similar to the contents of the re module which comes with Python. Here are the key parts:

tre.compile(pattern[, flags])
Compiles a regex pattern into a regex object. The regex object can then be used to search for approximate matches using its search() method.

A warning about regex syntax: this library uses the POSIX regex syntax, which is not 1:1 compatible with the usual Python regex syntax.

tre.Fuzzyness([delcost=1, inscost=1, subcost=1, maxcost=<unlimited>, maxdel=<unlimited>, maxins=<unlimited>, maxsub=<unlimited>, maxerr=<unlimited>])
Creates a “fuzzyness” object, used with the search() method to restrict the number of errors allowed.

RegexObject.search(string, fuzzyness[, flags])
Searches for a match in the string with the given fuzzyness.

So, as usual with regex libraries, you first compile a regex and then use it to find matches. The fuzzyness object defines how many errors of which kinds the match is allowed to have. The different kinds of errors are:

• insertion: an extra character, like changing Donald to Donnald
• deletion: a missing character, like changing Knuth to Kuth
• substitution: a changed character, like changing Ervin to Erwin

The total number of allowed errors is controlled by the maxerr field. This is probably enough for most applications.

For more fine-grained control, the maximum number of errors can be controlled for each error type separately with maxins, maxdel, and maxsub. The allowed errors can also be limited using costs. The default cost of each error is one, but can be set to any value using inscost, delcost, and subcost. The maximum cost is set with maxcost.

There’s a lot more I could say, but I’ll save my breath for further posts. I’ll just close with a modification on the by now old regex matching adage first uttered in 1997:

Some people, when confronted with a problem, think “I know, I’ll use regular expressions with approximate matching.” Now they have, like, maybe three or so problems.

Related posts:

1. Is Your Regex Matcher Up to Snuff?
2. How to Distribute Commercial Python Applications
3. Overriding System Functions for Fun and Profit

Have you ever written a parser for a configuration file?  Or a data driven interpreter which walks through data, and dispatches appropriate actions?  I’ve written too many to count, and more than half of them were a waste of time. They were a waste of time because they were programming problems in disguise.

A classical disguised programming problem is makefiles. Traditional tools, such as make, treat building programs as a configuration problem.  For make, you create a configuration file to define what programs to build, what commands to run to generate each target, and so on.  For a simple project, makefiles work fine.  Sometimes, though, your build process is a bit more complicated.

Maybe you have to do a slightly different thing on different platforms.  Perhaps you’d like to avoid listing source code files, instead using a naming convention to indicate which files to build.  Or maybe you find that the makefiles have a lot of repetition which could be eliminated if you just could define some kind of an abstraction to reuse that piece of logic everywhere you need it. As a sidenote, there are some enlightened build tools which correctly treat building programs as a programming problem.  SCons is one of these tools, and I encourage you to check it out.

The appropriate solution to a programming problem is straight up code.  Instead of a configuration file parser or a data driven interpreter, write a library. Instead of a configuration file or input for your interpreter, you write normal code using your library.  You get to enjoy the power and flexibility of a real programming language, and you don’t need to implement any of it yourself.

Occasionally, you really do need to provide a closed interface with some kind of mini language, because that just the way it is.  Most regex APIs, for example, compile strings into regex objects.  But there is no reason why you cannot also provide an API to programmatically build regex objects without having to go through the stupid strings.

Here are some telltale signs to detect a disguised programming problem:

• You have to generate configuration files to work around limitations.
• The configuration file format has grown abstraction capabilities like functions, macros, and reusable variables.
• The configuration allows embedding code.
• In general, any features which smell like a programming language: loops, inheritance, namespaces, types, and so on.

So the next time you find yourself writing a configuration file parser or interpreter, note that you are writing that program in a perfectly good programming language.  Why not use it?

Related posts:

1. The Essence of Lambda
2. The Programming High
3. Overriding System Functions for Fun and Profit

They say you can’t control what you can’t measure. How else would you know if it’s getting better or worse, right? This is true, sort of, if you know what you’re doing.

You can measure the weather but you can’t control it. What you measure, and how, sets limits to how useful your measurements can be for making improvements.  In this post I’ll explain what is a useful metric, and show a way to maximize the useful impact of metrics in your organization.

How useful can you get?

First, a definition.  The complexity of a metric is the somewhat subjective complexity of the system being measured.  The lines of code produced per day is a simple metric.  The corporate operating profit is a complex metric.

The actionability is the ease at which you can choose clear actions to help improve the reading for a metric.  Yes, “actionability” is a word.  I found it on the interwebs.

Let’s draw this relationship into a chart:

When the metric is simple (lines of code) , then the actions to improve results for that metric are clear (learn to cut’n'paste, duh). When the metric is complex (operating profit), then the actions to improve results are not so clear. As the metrics get more complex, it quickly becomes harder to figure out what to do to improve.

In short, complex metrics suck if you want to use them to figure out what to do about the results.

There’s a similar relationship between alignment of metrics with high level goals:

This chart show us that  simple metrics suck because they’re not connected to your high level goals at all.

To be really useful, a metric should be both easy to act upon and aligned with the high level goals.  Putting this in the form of an equation, we get:

usefulness = actionability × alignment

Multiplying the two charts above together gives this, with staggering mathematical precision:

Not very encouraging, huh? At best, metrics are barely above useless. But it’s really true. There is no single metric which would be directly connected to your bottom line and at the same time be really easy to convert into simple steps to improve the results. There is no silver bullet.

This chart gives us another useful piece of information. The most useful metrics are the ones in the middle of the complexity scale.  The useful ones leave room for an inductive leap or two to both directions, so everyone gets to actually use their brain.  That can’t be bad.

This may look like science, but in reality this is only my opinion. Multiplying random functions together may make you twitch, and rightfully so. There is no body of research where I’m drawing this from, just my personal experience.

Using metrics in your R&D

My recommendation is something called metric of the month.  In metric of the month, you choose a problem area for the next month.  Then you try to find the best metric which is connected to that problem.  Run with the metric for one month, try to improve the reading as much as you can, and then throw the metric away.  Start over with a new metric for the next month.

This way you can maximize on the excitement of the fast initial progress with a new metric.  Our compilation warning chart is no longer useful to us, for example.  It has lost it’s mojo.  So, it makes sense to change focus as soon as that happens, and pick the low hanging fruit from a new area.  It keeps things improving, and helps keep things interesting.

Whatever you do, don’t tell developers to improve on a metric like the corporate operating profit, EBIT, the stock price, or anything like that.  You could just as well tell them to control the weather.

No related posts.

My post on recruitment mistakes was intended for managers trying to find software developers. This post is for those on the other side of the table: you, the developer, trying to land a job. Here’s how to make your job application stand out, and how to impress me in the subsequent interview.

This is written purely from my personal perspective. If you want me to hire you, this is the perfect guide for that. For other managers, your mileage will vary. Wildly.

Your application

The application is usually my first point of contact with you. Make it count. Here are the most important questions I want your application to answer:

Do you have the required skills?

It’s all about the skills.  You need to have them, and your application must make it clear what your skills are. I’m looking for real experience in any and all areas of producing software: programming, testing, design, specifications, version control, build systems, tech support, maintenance, optimizing, documentation, algorithms, project management, debugging, IDEs, tools.   The more you know the better.

Do you get things done?

While talking about software is fun, we need to actually build some every now and then. I need to know that you are someone who gets things done. So be specific about what you’ve done in the past. I don’t want to hear how “you played an important role in the development of the Fnorb Buzz product suite”. I want to know what you did exactly, in plain engineer speak.  Did you write code? Specs? Run tests? Manage projects? What tools did you use? What did you learn?

Are you passionate?

I’m looking for signs of passion. Passion for developing software, of course – while passion for martial arts or cultivating Spanish snails are nice, it’s not what I’m primarily looking for in a software developer. If you spend time with software and computers on your free time, please don’t keep it a secret. It’s always cool to read about how you set up off-site backups for your mom with some cron jobs, rsync, and shell scripts, or about that data center you built in your closet.

Are you a good communicator?

I don’t want to read a 15-page account on every single computer related thing you’ve ever done. Keep it short, sweet, and relevant. Try to put yourself in my shoes – what is it about yourself that I would find interesting? I need to know about your skills and what you can do. What compelling evidence do you have to showcase your awesomeness?

That’s all, really. If it seems that you’re not lacking in any of these departments, I’m not going to throw your application in the trashcan. Yet.

The phone screen

If your application looks interesting, I’m going to schedule a short phone interview with you. My main goal is to find out more about your communication skills, but now verbally instead of in written form.  I don’t have a particular method for doing the call.  I’ll just ask about things I’ve jotted down on the margin of your application while reading it.

The interview

If the phone interview went well, then we’ll finally get to meet face-to-face. In addition to you and me, there will be one or two other people present in the interview as well.

Talking about code

I’m going to make you do a coding exercise or two on paper or a whiteboard, or possibly a laptop.

If I already know for a fact that you can code, I may skip the coding exercise. In lack of other evidence, I’ll have to rely on the coding exercise. I’ll find out if you can program your way out of a wet paper bag. Then I’ll find out if you can work with code for real.

We talk about code, you write some, and I’ll find out what you know. I’ll see how you react when you run into problems, what you do when you get incomplete instructions, and in general how you go about solving problems.

Other questions

We won’t just talk about code, of course.  We’ll try to find out what kind of a person you are, so we can determine if you would fit into our team or not. Also, we’ll need to find out some basics about why you’re looking for a job, what kind of a salary are you hoping for, and so on.  You should want the job for the right reasons.

My advice for these questions is simply to be honest. Tell me the truth. Don’t tell me what you think I want to hear, because that’s probably not what I really want to hear.

Towards the end of the interview I’ll ask if you have any questions about anything. If you have no questions, I’m bound to think that you’re not very enthusiastic about getting the job. On the other hand, if your questions indicate that you’re already thinking about what it would be like to work for us, it’s a good sign. I wouldn’t like to hire you if you just need a job, any job.

So, now you know how I will deal with your application.  To make me hire you, all you need is the skills, the right kind of motivation, and a suitable personality.  Easy, right?

Related posts:

1. Do You Make These Mistakes When Recruiting Software Developers?
2. 6 Tips for Small Software Vendors to Understand Enterprise Customers

Is there always something on your to-do list? Is your project manager constantly barraging you with task after task after task? Is there an “innovation initiative” going on in your company, but somehow nobody seems to be doing all that innovating?

If this sounds like your company, I have good news for you. By the end of this post you’ll not only understand why your company isn’t very innovative – you will also know what you can do about it.

The #1 Innovation Killer

The one thing responsible for most innovation-stifling in companies today is probably optimizing human resources for maximum efficiency. You know, doing the biggest amount of work possible with the least amount of resources.

On the surface, it seems like a reasonable thing to strive for. Why hire five people to do customer support when four people can handle it, if they work hard? There’s nothing wrong with working hard, is there? Why should you be paid for slacking off?

Well, of course there is nothing wrong with working hard. But it’s kind of hard to have a broad mind and think out of the box and, you know, innovate, when there’s a manager breathing down your neck demanding for a completion date on that critical bug fix.

One of my old bosses had a fun way to put this. He said “we’re too busy pushing the bicycle to hop on and start riding“. That’s exactly what is going on in the 100% efficient organization. Pushing the bicycle is 100% efficient – all effort is spent in moving the bicycle. But it sure isn’t the most effective way to move a bicycle from A to B.

In Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency, Tom DeMarco puts it this way:

In our constant quest to make our organizations more efficient (reduction of overhead, standardization of processes, overworking management and resources), we have actually made them less effective. The solution lies in (re)introducing `slack’. Slack is the lubricant required to effect change, it is the degree of freedom that enables reinvention and true effectiveness.

Why You Cannot Innovate Under Pressure

Dan Pink recently gave a fantastic talk on the science of motivation. I invite you to watch this video now:

Did you watch it yet? Please do, it’s really worth it.

I’ll just wait here until you’re done…

OK, good.

The key point I want to make here is that If a task requires even rudimentary cognitive skill, a larger reward leads to poorer performance. This is a scientific fact, and can be extrapolated from trick questions such as the candle problem to “real work”. See Functional fixedness.

What Should You Do?

If you’re a manager and you can decide on how time is spent in your company, fantastic! The first thing to try is to have a FedEx day a few times a year. If you watched the video, you already know what it is: set aside a day for developers to work on whatever they want and then demonstrate results to the entire company.  Repeat often.

If you’re an executive manager, you may have the power go the Google way. Decide that your knowledge workers can spend one day a week on whatever they want to.

If you’re a developer, show this post to your manager. Hopefully you can cajole him into trying some of that FedEx. If not, do a FedEx day or three as a skunkworks project. Nobody needs to know, right? Until, of course, when you get some results worth showing.  As they say, sometimes it’s easier to ask forgiveness than it is to get permission.

So, pull your nose out of the task list, bug tracker, inbox, or calendar for a while.  It’s time to put the fun back into your work.

Related posts:

1. You Don’t Need a Deadline

Most of us in the software business are not in a position to release our source code to the public.  As much as I love free and open source software, I also understand it’s not part of all business models.  Python is enjoying commercial success, but perhaps less so in cases where end-users actually install the software.

Distributing applications written in Python can be tricky, especially if you don’t want to distribute your source code. I set out to find out how exactly do you distribute a Python program, sans source code, on various operating systems. It turns out there are quite a few alternatives to consider.

In each alternative, the first step is to compile your code to bytecode (.pyc or .pyo).  That’s what you’ll be shipping instead of the source code.

1. Using a preinstalled Python

Nowadays, many operating systems actually come with a sane Python interpreter preinstalled. Most Linux distributions ship with Python, and Mac OS X comes with Python, for example. Windows does not come with Python, nor do most commercial Unix systems.

Even if a preinstalled Python is available, it may not work for you.  Maybe it’s an older version, and you want to use the latest and greatest.  Another potential hurdle is that the Python version will be different on different operating systems. Mac OS X 10.5 (Leopard) comes with Python 2.5.1, Mac OS X 10.4 (Tiger) came with Python 2.3.5. Ubuntu 8.04 came with Python 2.5.2, Ubuntu 9.04 comes with Python 2.6.2… You get the point.

Sometimes modules are deprecated between versions and you’ll need to write stuff like this to suppress warnings:

try:
    import hashlib
    md5_new = hashlib.md5
except ImportError:
    import md5
    md5_new = md5.new

Varying Python versions may or may not be a problem for you application.  The example above is actually a fairly bening problem, and it’s easily solved.  If you ever need to support both Python 3 and Python 2, you will have bigger problems than just deprecated modules.

2. Freezing tools

Freezing tools compile Python programs to executables. If you’re distributing a small number of executables, a freeze tool might be just what you need.  There are quite a few of these around.  Disclaimer: I haven’t used most of these tools.  Caveat emptor.

py2exe

If you only care about Windows, py2exe fits the bill nicely.

py2exe is a Python Distutils extension which converts Python scripts into executable Windows programs, able to run without requiring a Python installation.

Applications such as BitTorrent and SpamBayes use py2exe – so it should be stable.  From what I’ve read, py2exe  works quite well.

py2app

py2app makes standard Mac OS X application bundles from Python scripts.  It’s “py2exe for Mac”.

cx_Freeze

cx_Freeze is similar to py2exe, but claims to be portable to both Windows and Unix systems.

bbfreeze

bbfreeze works on Windows and Unix, but not on OS X. Originally based on cx_Freeze.

Freeze

The Freeze tool ships with Python. It compiles Python programs to executables on Unix systems.

In my simple tests, it was able to handle a trivial hello world application pretty well, if we ignore the size of the resulting executable (several megabytes). Throwing anything more complex at it tended to fail miserably for various reasons. It seems to me that Freeze does not work very well out of the box.

PyInstaller

PyInstaller creates executables for Windows, Linux, and Irix.  Yeah, Irix.

3. Bundling CPython with your application

CPython is the default, most widely used implementation of Python.  It is written in C.  When people talk about the Python interpreter, they mean CPython unless they explicitly mention IronPython (for .NET and SilverLight), Jython (for JVM), Stackless Python (for concurrent programming), or something like that.

Sometimes the freezing tools aren’t a good choice.  They might lack platform coverage or could have other problems.  If you’re distributing a large collection of separate programs the overhead of bundling the full Python interpreter to each program will be prohibitive.  A good alternative is to bundle the Python interpreter with your application yourself.  You gotta love some DIY in the software business, right?

The first step is to bundle the CPython interpreter with your application. That’s fairly straightforward if you know what you’re doing. You simply include the Python interpreter with your code.

The second step is to cherry-pick what you need from the Python standard libraries. Tools like snakefood or modulegraph can help you do that. From what I can tell, these tools aren’t perfect, so be prepared to manually fix the results or patch the tools.

A third and optional step is to customize your Python distribution. For example, you might inline all bytecode directly into your executables instead of separate files. Then you can override the loading functions to pass the code directly from memory instead of loading from disk.

4. My recommendations

My first choice is to use the Python interpreter which comes with your target platform, if it has one.  Use distutils to install custom modules or extensions. If your application only supports Windows, use app2exe py2exe.

For complex scenarios, my recommendation is to go with bundling CPython yourself and steer clear of the freezers.  If a freezing tool works for you, awesome. You just saved yourself from a lot of trouble.  But if you need to cover a large number of platforms, or your application has special needs from the interpreter, then packaging CPython yourself seems to be the best way to go.

The added benefit of bundling CPython yourself is that you can customize the interpreter to your heart’s content.  Not that you’ll really need to, but it sounds like fun.  There’s nothing like making your own bubble wrap, complete with a custom hand-painted smiley face on each bubble.

Subscribe to Hacker Boss today!

Related posts:

1. Approximate Regex Matching in Python
2. Programming Problems in Disguise
3. The Shell is Like a Dishwasher

Some of the most satisfying commits I’ve ever done were the ones where I got to remove a lot of old cruft.

As your codebase grows older, it will accrue technical debt. It comes in many forms, but sometimes the most straightforward cure is to cut the bad parts out.

I’m not talking just about dead code. I’m talking about obsolete code. Obsolete code is code that was there for a reason, but the reason is no longer relevant. They are the appendixes of software – still there, still alive, but apparently not serving any useful function. Cut it out before it gets infected.

There are some simple tricks and practices to keep tabs on obsolete code. One of my favorites is the XXX comment:


// XXX (tag1, tag2): Description

You know, like

// XXX (obsolete): When 2.0 is released, this won't
// be needed anymore.

or

// XXX (bug): drunk, fix later


It’s easy, flexible, and greppable. If you have a lot of code taking care of backwards compatibility, you’ll want to spend extra attention on making it easy to remove when it becomes obsolete.

Sometimes, grepping won’t help and you have to work harder. To cut out the bad part, you need to shuffle things around, refactor, and build some new pieces. But the feeling of making it better is totally worth the trouble.

Don’t be afraid of deleting obsolete code. If you ever need it back (and I’ll bet you won’t), it’s going to be right there in your version control history, waiting for you to resurrect it. Don’t #if 0 or comment it out. Remove it. The best code is no code at all. The code is our enemy. Don’t treat the code as an asset. Treat it as a liability.

So… What is going to be your satisfying commit of the day?

Related posts:

1. Get Rid of Source Code Templates
2. Programming Problems in Disguise

I always thought of myself as a pretty hot-shot programmer.  I had been programming since I was a kid.  It wasn’t hard to find my first programming job.  Quickly, I recognized that I had much more programming experience than some of my older coworkers.  Comfortable with my skills, I took it easy…  I took it easy for a long time.

Then, I met a new guy, call him Aaron. Aaron was obviously much better than me in most areas. He was younger than me, too. He made me realize I hadn’t really improved much in the past years. I was an ad-hoc hacker, and a mediocre one at that.

This alerted me to consciously try to improve myself, and especially the quality of code I write.

Aaron led me to learn a lot of things. He taught me how most of the code I write will have to be maintained and extended for at least several years, so I should write the code with that in mind. I should write automatic tests for my code. Aaron was always talking about how I should never stop at the first working version, but refactor and refine until the code is elegant.  I discovered that the languages and tools I was using had a lot of room for improvement.

The most important thing I learned from Aaron was to never stop learning.

After a couple of years, Aaron left the company.  I felt empty.  The past years with him had lifted me to whole new levels of skill, and I realized I was now much better than the rest of the team. They were still writing bad code, and doing the same mistakes as before. I tried to teach them, but they had no interest to learn. In fact, they were annoyed that someone would be so arrogant to tell them what mistakes they were doing.

Luckily, I was able to change to a small motivated team. Everyone there was eager to learn more, and I loved it.

I’m glad I met Aaron. Without him, I’d probably still be working with the old gang, going nowhere, and thinking too much of myself.

This post is based on my answer to a Stack Overflow question.

Related posts:

1. The Programming High

Picture the last time you were hiring.  Did you have trouble finding the best programmers and technical talent?

Well, chances are you’re doing it wrong.

Here’s my top list of big mistakes I’ve seen or done myself, and some helpful pointers on how to avoid them.

Do you post boring job ads?

The world is full of ordinary and basic job ads.  The first step in the long and rocky road to a successful hire is to get people to make note of your ad.  Spend time crafting a killer job ad which captures developers’ attention.

Do you post ads in the wrong places?

You have to know your target audience.  Post your ad to the relevant places which great developers follow.  At the moment, one of these places might be  jobs.stackoverflow.com.  These things change quickly, so be sure to follow the trends even though you’re not looking to hire right now.

Do you fail to immediately answer all applications?

A long time ago, I applied for a programmer job at a well known Finnish software company.  I sent them my resume but heard nothing back for a long time.  Finally, after several weeks of submitting my application, I received a reply thanking me for my interest in the “storage attendant job”, and informed me that unfortunately at this time they cannot hire me.  By then I had already been hired somewhere else.

You should immediately answer all job applications to let the applicant know that their application was received.  Also inform the applicant on how your recruiting process works, and when should they expect to hear more from you.  Radio silence will never work to your advantage.

Do you skip phone interviews?

Phone interviews are a great way to quickly find out if the applicant is a good communicator.  Nothing is more frustrating than going through all the trouble of setting up a job interview, only to find out that you can’t quite grapple a thick Elbonian accent.

Do you hire people who “might be good”?

You should only hire people who are definitely good.  Only hire if you’re reasonably sure that the person would be a great fit in your team.  The cost for training in a developer are high.  If things don’t turn out well, you’ve lost a lot of time and money.  Not quite sure if the candidate would be great?  That’s a no hire.

Do you ignore the local universities?

This is a glaring problem at least in Finland – maybe things are better on other parts of the planet.  Universities are full of smart people.  At some point, these people start looking for their first jobs.  For the best programmers this could well be the only time they are on the job market.  For the rest of their careers they just kind of go work wherever they want.

You would be wise to build a relationship with the relevant fraternities, sororities, and student guilds.  By “relationship”, I mostly mean free beer in exchange of a little attention.  When the students start looking for that first job, your company could well be among the first that springs to mind.

Do you not do internships?

Summer internships are a fantastic way to get young and motivated people working for you, before they are forever gone from the job market.  It’s also less risky than flat-out hiring, both for the intern and your company.

Now, picture again the last time you were hiring.  What would you change if you could do it over again?

Related posts:

1. Inside Tips for Making Me Hire You
2. Building High Performance Software Teams

Building software for enterprise customers is a very different experience from building software for home users, for example.  If you’re working for a small software company, it can be sometimes difficult to understand how big corporations work.  Here are some tips based on my experience. Hopefully, these are helpful in figuring out how enterprise customers change some of the expectations for your software.

1. Enterprises are big on policy

There is a policy for everything.  Even though policies are usually defined in good faith, from the perspective of an individual employee policies are about compliance and not common sense.  So sometimes, a policy will be the only reason which prevents your software from being used, or prevents it from from being configured in the way which would make the most sense. Don’t fight the policy windmill. Work around it.

2. Enterprises have long deployment cycles

It takes ages for big companies to move to new releases.  As a case in point, think of Internet Explorer 6.  It’s an ancient browser but still widely used.  The last holdouts for IE6 tend to be corporations where IE6 is company policy.  The lesson for your software: you may need to support and interoperate with very old products and technologies.

3. Everything is standardized

Big corporations like to standardize their operations.  For software, this usually means that some part of the IT organization defines standard “builds” of operating systems. The build locks down the operating system, the OS version, applications, and their configurations. All new boxes are installed with a standard build. In order to be included in the standard build, your software must have no required setup steps after installation. Your configuration files, for example, should be designed so that the same exact configuration can be used on multiple hosts.

4. Prepare to integrate

The IT infrastructure in every big company will have a number of systems your software may need to integrate with.  Possible candidates for integration could be identity management systems (Active Domain, LDAP, NIS+) , single sign-on systems, content management systems, CRM, and so on.  You must design your software so you can integrate with everything and anything that makes sense for your application domain.  If your software requires user accounts to be created and maintained separately, don’t expect a bank to buy it.

5. Prepare to port

The largest companies have huge global IT environments.  They have acquired other companies.  Semi-independent subsidiaries have been running their IT operations in different ways for years.  This means that there will almost always be a number of various operating systems in use throughout the organization.  There’s Windows on the desktops and maybe on some servers as well.  There is usually a substantial number of servers running some flavor of UNIX: AIX, HP-UX, Linux, or Solaris.  In the banking world, there will be AS/400 or z/OS mainframes, or both.  Rarely, there are some Macs in desktop use as well.

Depending on the nature of your software, you may need to support a number of platforms.  If you’re lucky, it’s only going to be a couple of different UNIX.  In the worst case, you have to port your code to Windows, all the UNIX, plus the mainframe as well.

6. Prepare to release patches

Due to the large deployments, standardized builds, and dependencies to other software, you will be hard pressed to deliver fixes to problems by offering the fixes in the latest version.  Usually there are policies which allow for a lightweight validation process for patches but require a larger project for a new release in fear of regressions (and rightfully so). Prepare to deliver patches, or minimal changes in some other way, so your customers can get the fixes they need and only the fixes they need.

Related posts:

1. Inside Tips for Making Me Hire You
2. Platforms Come With a Culture

Looking back, I tend to do my most interesting programming at night. Writing the query optimizer for HiSQL kept me up a couple of nights. The first versions of libtre (a regex matching library with support for approximate matching) took a number of long evenings.   I wrote the first prototype for transparent tunneling of TCP connections over SSH for SSH Tectia ConnectSecure in the middle of the night.

There’s clearly a pattern here. Good code happens at night. Night time is the right time… for coding.  Sometimes, for me, at least.

The question is, why?  Am I so excited about the ideas, so beside myself with the anticipation of turning the idea into something real, that I cannot sleep?  Or is it that I simply need the peace and quiet in order to get into the zone?  Maybe when I’m in the zone I don’t want to stop, then loose track of time, and end up coding into the night?   All of the above?

Regardless of the reason, it does appear that I’m not alone with this.

This makes me wonder if software companies should rethink some of their policies regarding working times.  Wouldn’t it make sense to allow doing the work in flexible pieces? Maybe spend the morning until lunch at the office, and do the rest at home in the evening?

Furthermore, there’s evidence that a short nap can significantly boost productivity.   A 20-minute nap can make all the difference, apparently.  I certainly cannot get much thinking done in the early afternoon.  I tend to take care of some of my more rote, menial tasks during the groggy post-lunch time.  I’m a manager, so there’s plenty of that kind of work for me, alas.  But for a software developer, wouldn’t it make sense to take that 20-minute nap, and wake up fresh, instead of hours of half-witted monitor-staring?

No related posts.

The Rubber Duck method of debugging goes like this:

1. Beg, borrow, steal, buy, fabricate or otherwise obtain a rubber duck (bathtub variety).
2. Place rubber duck on desk and inform it you are just going to go over some code with it, if that’s all right.
3. Explain to the duck what you code is supposed to do, and then go into detail and explain things line by line.
4. At some point you will tell the duck what you are doing next and then realise that that is not in fact what you are actually doing. The duck will sit there serenely, happy in the knowledge that it has helped you on your way.

This works amazingly well. There’s a whole section in the book The Pragmatic Programmer about it. Also, The Practice of Programming describes the close relative, the Teddy Bear method of Filtering Bug Reports:

One university computer center kept a teddy bear near the help desk. Students with mysterious bugs were required to explain them to the bear before they could speak to a human counsellor.

You should definitely get these books if you’re interested in reading about insights similar to the rubber duck. These books are full of great stuff, but especially The Pragmatic Programmer does an awesome job of explaining things such as the broken window theory, defensive programming, and the DRY principle.

No related posts.

The primary purpose of the programming language is to make it easy for the programmer to communicate to other humans what the program does. Incidentally, the programs are also interpreted by the computer.

Jakob Nielsen, a famous usability guru, has defined ten general principles for user interface design. What follows are Nielsen’s principles applied to programming language design.

When reading this, bear in mind that a programming language is much more than just a language manual and a compiler. The whole development environment and ecosystem of libraries are as much part of the programming language as the syntax and semantics of the core language. Even the online community of programmers can be thought of part of the language. So when I’m talking about “programming languages”, I usually refer to the whole experience, not just the bare language.

These are ten general principles for programming language design. They are called “heuristics” because they are more in the nature of rules of thumb than specific usability guidelines. Some of these heuristics are clearly applicable to the design of the core language, some to the IDE, others perhaps to standard language libraries.

My apologies to Mr. Nielsen for messing with his stuff.

Visibility of system status

The system should always keep the programmer informed about what is going on, through appropriate feedback within reasonable time.

Match between system and the real world

The system should speak the programmers’ language, with words, phrases and concepts familiar to the programmer, rather than programming language theory-oriented terms. Follow real-world conventions, making the system behave in a natural and logical order.

Programmer control and freedom

Real-world system resources are often unavailable or fail. Program code using these will need an effortless “emergency exit” to leave the unwanted state without having to clutter the code with error handling.

Consistency and standards

Programmers should not have to wonder whether different syntaxes, contexts, or names mean the same thing. Follow platform conventions.

Error prevention

Even better than good error codes is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present programmers with information on the different ways their code may fail.

Recognition rather than recall

Minimize the programmer’s memory load by making objects, actions, and options visible. The programmer should not have to remember information from one part of the program to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

Flexibility and efficiency of use

Accelerators – unseen by the novice programmer – may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow programmers to tailor frequent actions.

Aesthetic and minimalist design

Functions should not require arguments which are irrelevant or rarely needed. Every extra unit of information in a parameter list competes with the relevant units of information and diminishes their relative visibility.

Help programmers recognize, diagnose, and recover from errors

Error and warning messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

Help and documentation

Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large.

Related posts:

1. Programming Problems in Disguise
2. The Three Doors of Type Systems
3. The Programming High

Read the intro, part 1, and part 2 first.

This is a description of a fictitious programming language. It is purposefully vague here and there. Nevertheless, I don’t think I’m exactly reaching for the moon here. Or perhaps I am, you tell me. Point is, this is just fantastic, wholesome, and thoroughly enjoyable academic masturbation. Shall we begin?

We need a name for the language, so let’s call it Shark.

Shark uses a purely copy-on-write data model. By this I mean that, as far as the programmer can tell, values are never modified. Once you create an integer, a string, a list, or a forest of suffix trees containing all the words in the Klingon language, you cannot modify it.

Shark is, of course, garbage collected. Shark is also strongly typed, supports both static and dynamic typing, and has type inference. You have your lambdas, partial evaluation, algebraic data types, pattern matching, a fantastic module system, and a clean and concise but not too unfamiliar syntax. As far non-existent programming languages go, Shark is the dog’s bollocks.

Shark has lightweight threads. Spawning new threads is almost free (only a handful of instructions on the CPU). Threads are automatically distributed to your CPUs and cores. Just to be absolutely crystal clear, Shark threads are not the same as operating system threads such as POSIX threads. Shark threads communicate exclusively through message passing. Each thread has a message queue from which it can read incoming messages when it wants to.

There are no locks, condition variables, and no modifiable shared resources. When threads share data they do it explicitly, in a controlled fashion, for selected data only. Shark comes with built-in support for software transactional memory. You can easily build databases from whatever data structures are the most natural for your application. You can kiss all that RDBMS crap goodbye. No more shoehorning your data into an awkward tables-and-columns format, no more marshalling and demarshalling your data to whatever data types the database happens to support, and no more SQL.

The natural model to maintain state in Shark programs is to have a single thread govern the evolution of that state. The thread can then pass around copies of the state, maybe receive operations from other threads to update the state, and so on. Typically the state takes the form of an STM database which can be updated transactionally from multiple threads concurrently.

Just one more thing. Shark STM databases can optionally be stored on disk. It’s so efficient that you can run tens of thousands of transactions per second and have them checkpointed to disk a dozen times per second. This gets you the Durability in ACID, the one piece that regular STM normally doesn’t provide. Even though the entire database must reside in main memory, you’ll need to have a pretty big database for that to become a problem. RAM is insanely cheap nowadays.

So, that’s my hallucinogen-induced (not really) vision of the perfect solution for concurrent programming and database management. Yeah, I know creating your own programming language is basically never the right thing to do. In my defense, at least I haven’t implement Shark. Not yet, at least…

Related posts:

1. I Have Seen the Future, and It is Copy-On-Write
2. Copy-On-Write 101 – Part 2: Putting It to Use
3. Copy-On-Write 101 – Part 1: What Is It?

In my last post I gave a quick introduction to copy-on-write data structures; make sure to read that first.  It all started with a vague assertion on how copy-on-write is supposed to be a great advance for concurrent programming.  I was met with a lot of blank stares, and possibly embarrassed silence.  So, instead of merely waving you off, dear reader, as ignorant, I’ll try to educate you on the marvels of copy-on-write (COW) data structures.

In this part I’ll show some areas where COW data structures have proven to be useful, and talk about some practical considerations when implementing COW data structures in real life.

Some applications

Undo

Often your programs allow the users to manipulate some data: text documents, configuration settings, diagrams, vegetarian pizza menus… any number of things.  If the manipulated data is stored in a COW data structure, implementing unlimited undo functionality becomes almost free.

All you need to do is put the different versions of the data on a list.  Structure sharing makes sure you don’t waste too much memory for the old versions.   Accessing the old versions takes no extra effort, they’re just right there.

Optimization

This appears to be what most people think copy-on-write is for.  It is correct that COW can be a useful optimization strategy, but it is by no means the only useful application.

Anyway, as an example, say you have a long string and you want to append something to the string.  In many programming languages strings cannot be modified like that efficiently; the space for the string has been preallocated and extending the string may require you to allocate a new larger memory area and move the data there.  A COW string typically allows for cutting and pasting pieces to anywhere in the string without having to copy everything.

Software transactional memory

This is a big one.  Software transactional memory is like your own private in-memory database.  If you haven’t heard of it before, I strongly suggest you to find out soon.  Software transactional memory provides everything an ACID database would without the durability part, since you’re dealing with data structures in RAM after all.  A big advantage is zero impedance mismatch, among other things.  COW data structures are the natural way to implement software transactional memory.

Practical considerations

Automatic memory management (e.g. garbage collection) is a must-have to conveniently deal with COW data structures.  In fact, I would make the same argument for almost any kind of programming, but it’s doubly true for COW.

Most of the data structures you’ll be dealing with are going to be some type of directed acyclic graphs built of nodes linked to other nodes, plus possibly some of the “real” data you are storing along with the graph structure nodes.  You need to allocate the nodes separately and, because this is COW, you’ll be allocating all the time.  Allocating cells from a garbage collected heap is typically extremely fast, so it’s the perfect way to manage memory in the COW world.

Constant factors

Often, you want to use a COW data structure because you need to keep some old versions around for a while.   But you don’t necessarily want to hang on to every version, so you may wonder if the performance hit of a COW data structure is getting too big.  In most cases, I wouldn’t worry about it very much.

To update one node in a mutable tree, you start from the root node, then make your way to the node you’re looking for. Once you’ve found that node, you just change it, in-place.  In a COW tree, you’ll need to trace back to to the root node and make copies of each node on the way back.

So the tree is traversed twice, once both ways, and that will take about twice the time, right?  It turns out that this is not the case on modern computers.  When it’s time to copy the path, the data you need is almost certainly already in the L1 cache so revisiting the nodes again takes almost no time at all.

Also, since we’re allocating memory for the new nodes from a garbage collected heap, allocating memory is really fast so that isn’t really going to make a big hit either.  For these reasons, the constant factor slowdown for a COW tree vs. a mutable tree can be quite small indeed.

Asymptotic complexity

It is a fact that there are problems which take linear time in an imperative programming language but suffer an unavoidable O(log n) slowdown in a purely functional (and strict) language.  These cases seem to be rare in practice, though.   There are algorithms even for efficient copy-on-write arrays.

Next: Copy-on-write 101 – Part 3: The sum of its parts — how COW makes concurrent programming possible without your brain turning into pineapple custard.

Related posts:

1. Copy-On-Write 101 – Part 1: What Is It?
2. I Have Seen the Future, and It is Copy-On-Write
3. Copy-On-Write 101 – Part 3: The Sum of Its Parts

My previous post was about how I think copy-on-write is the future of concurrent programming.  The next day, I was trying to explain the idea to a couple of colleagues but got just blank stares in return.  It struck me that most programmers don’t really know much about copy-on-write techniques, how (or even if) they work, and what kinds of problems they’re best suited for.  So, I’ve decided to write a small series of posts on this subject.   In this first post, I’ll explain the basic idea of copy-on-write data structures.

Traditional data structures are mutable.  Let’s look at a simple binary search tree, for example:

Each node in the tree consists of a value (the number) and zero, one, or two arrows pointing to other nodes.  Say you want to add a new node, a 34, in this tree.  That’s easy, we just draw the new node in the right place:

Note that the tree is now changed.  We had to update node 21 to point to node 34 in addition to 13.  The original tree for all practical purposes does not exist anymore.  Well, OK, it does on this page right here, but you get the point.  At any one time, there is exactly one version of the tree in existence in your program.

A copy-on-write data structure, on the other hand, is immutable.  Once you have gone and created a binary tree, you can never ever change it.  Adding an arrow from 21 to 34 is not allowed, no sir, over here in the wacky world of copy-on-write you have to create a modified copy instead of actually modifying anything.

If having to copy the entire tree seems a little bit ridiculous to you, that’s because it probably is.  Typically, two trees are considered equal if they represent the same structure.  Hence, it does not matter where the nodes of the tree are stored in memory, for example.  In fact, in many programming languages there is even no way to distinguish between two copies of otherwise identical pieces of data.  With this in mind, we can share structure instead of taking a full copy:

Here we have both the old tree (without the 34) and the new tree (including the 34).  An application will be looking at this data through a window that is the programming language, and all it sees are the two completely separate versions of the tree.  They happen to share some structure, but that arguably does not matter much to the application.

In general, if you’re updating one node, taking a modified copy of a tree will require copying only the path from the root node of the tree to the affected node.  And that’s how copy-on-write trees work: path copying.  Tah-dah!

It is not immediately clear if a structure-sharing copy-on-write counterpart can be created for all mutable data structures.  All tree-like structures can clearly be handled with the path copying trick.  Other structures, such as queues, seem harder, perhaps impossible at first glance.

Luckily, it turns out that smart people have already designed efficient copy-on-write versions of queues and many other important data structures; there’s a great book on the subject by Chris Okasaki called Purely Functional Data Structures.  It’s by far the best resource on copy-on-write data structures that I’ve encountered.

Note how I prefer to say “copy-on-write” instead of “purely functional”.  The term “functional” is just way too overloaded and carries a lot of negative connotations in the minds of working programmers who don’t much care for many of the things that functional programming is often associated with.

The use of copy-on-write data structures is not limited to functional programming, so let’s not marry the two by calling them with the same name.

Copy-on-write data structures are also sometimes called “persistent” data structures, because the old versions are preserved; they “persist” in memory.  In practice, this is easily confused with persistent storage (e.g. disk), so I avoid using that name as well.

Next: Copy-on-write 101 – Part 2: Putting it to use

Related posts:

1. Copy-On-Write 101 – Part 2: Putting It to Use
2. I Have Seen the Future, and It is Copy-On-Write
3. Copy-On-Write 101 – Part 3: The Sum of Its Parts

Copy-on-write is a similar advance for concurrent programming as garbage collection was for memory management.

I became a copy-on-write believer at around the turn of the millennium when I was working for the HiBase project as a research assistant.  Among other things, I implemented an in-memory SQL database server in Shines, our prototype programming language, which used a strictly copy-on-write memory model.  We employed an optimistic concurrency control algorithm to resolve possible conflicts for database transactions run in parallel.  The code was elegant, easy to understand, and ridiculously efficient.  There wasn’t a single lock or critical section in the entire system.  Just threads, message passing, and copy-on-write data structures.  Beautiful.

John McCarthy invented garbage collection for Lisp in 1959.  It took almost 40 years for garbage collection to finally hit the mainstream programming consciousness with Java.  I’m not exactly sure when the copy-on-write data model was invented, but based on the Wikipedia page on functional programming I’m going to place it somewhere in the 1970s.  It would be about time for copy-on-write to hit mainstream now, but I don’t see that happening.  What’s wrong?

The Erlang guys have touted the copy-on-write model to be the True Way to do concurrent programming for years.  But Erlang isn’t enjoying much success in the mainstream.  I suspect this is to do with the academic stigma that goes with functional programming and the unusual syntax inherited from Prolog.

Nowadays, copy-on-write techniques are growing increasingly popular also outside the confines of RAM.  In the past ten or so years, a number of file systems have popped up which use a copy-on-write data model to implement snapshotting, for example.  I find ZFS to be the most promising of these.  RAID-Z is just super cool.  But let’s get back to programming languages.

I believe the concurrent programming language for the mainstream will have the copy-on-write data model, threads with message passing, a familiar syntax, plus a big ecosystem of libraries and tools.  Most importantly, though, the language won’t called a “functional” language, even though it might in reality be awfully close to being one.  I wonder when this language will appear.  Or does it already exist?

Next: Copy-on-write 101 – Part 1: What is it?

Related posts:

1. Copy-On-Write 101 – Part 3: The Sum of Its Parts
2. Copy-On-Write 101 – Part 2: Putting It to Use
3. Copy-On-Write 101 – Part 1: What Is It?

Sometimes I’m just so glad I know shell.  I mean things like

find . -type f -print0 | xargs -0 -n 100 ls -1sk | sort -n | tail -10

For the uninitiated: the above command lists the ten largest files under the current working directory (including subdirectories).  How do the Windows power users do stuff like this?  PowerShell?  Python programs?  Can you even write one-liners in Python?

On a related note, it always pains me to watch people browse to a file in a file manager GUI.  You click through a bunch of folders and stare at each list of file names for a second before finding the right one, click on it, and by the time you’ve found your file the coffee’s gone cold.  With the shell, it’s “sTABnTABkTABsTABn” and you’re done in a fraction of the time.  Hot coffee.  Why can’t you do that in the file browser GUI?

The shell is like a dishwasher.  When I was a just poor undergrad CS student living off macaroni and ketchup “borrowed” from my roommate, I couldn’t afford a dishwasher.  There were more important things to spend money on, such as beer, science fiction, and macaroni.  At the time, washing dishes the old-fashioned way didn’t seem so bad.  In fact, I coped without a dishwasher for a long time after I would have been able to just buy one.  Now I wouldn’t dream of doing all my dishes manually.

No related posts.

From the Wikipedia article on unit testing:

“In computer programming, unit testing is a software verification and validation method where the programmer gains confidence that individual units of source code are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual program, function, procedure, etc., while in object-oriented programming, the smallest unit is a method, which may belong to a base/super class, abstract class or derived/child class.”

If you take this definition of unit testing literally you are going to be testing exactly the wrong things.

As an example, let’s assume we have a piece of code which is responsible for parsing some XML strings into data structures. This isn’t arbitrary XML, the strings are all of this form:

<user>
  <id>evalua</name>
  <name>Eva Lu Ator</name>
  <message>Anything in parentheses can be left out.</message>
</user>

The result from parsing should be a record in memory with the id, name, and message all parsed out nicely.  Let’s further assume that this code is implemented, foolishly, using regular expressions, because the person who implemented it thought that using an actual XML parser would be overkill for such a simple task. Sigh. At least the code is all in it’s own module with a clear signature:

structure Parser :> signature
   type result = {id : string, name : string, message: string}
   val parse : string -> result
end = struct
   <imagine a barrel full of hairy regex matching code here>
end

(That’s Standard ML, you insensitive clod.)

The final assumption is that the person who wrote this code, let’s call him Harry, had heard that unit testing is a good thing, and had decided to write some tests.  Our hapless Harry sought out the Wikipedia article to double check what unit tests actually are and proceeded to write tests for each and every function, plus lots of stubs so that he could successfully tests each function in perfect isolation.  The amount of test code Harry has written is about three times the amount of his hairy regex matching code.

Harry’s code works fine, until the first XML string with a multiline message comes up.  It wasn’t in the original requirements, so Harry didn’t originally add support for it because he had also read You Ain’t Gonna Need It.  Now he has to make a few adjustments to the code; maybe he splits one function into two and replaces another function with new code.  He makes little whimpering noises as he deletes the now defunct tests for the old code.  He curses under his breath when he writes the new test drivers and stubs for the new code.  But he manages to get it all done, and the code goes into production.

The holiday season arrives, and Harry flies off to a long-awaited vacation in Thailand.  While Harry is sipping piña coladas at the pool bar, his colleague, Ethan, gets an urgent bug to fix.  It turns out someone tried to put the ‘<’ and ‘>’ characters in the message, resulting in &lt; and &gt; in the XML, and Harry’s XML parser doesn’t handle them at all.  Ethan stares at Harry’s code for a while, deletes everything except the module signature, and writes some code to call a real XML parser and picks the needed parts straight from the DOM tree.

He writes a test program which parses example XML snippets and checks the results against known correct results.  He writes another test where the XML parser is a stub which fails in random ways (returns bad DOM trees, throws exceptions, etc.) to ensure his code can handle those situations.  Confetti starts streaming from the air conditioning vents, the birds outside burst into song, and a team of furiously cheerful tap-dancers dance along the corridors, joined by the office staff, all celebrating Ethan’s genius.

What did we learn from all this?  For one thing, unit tests won’t protect you against incompetent design. Second, blindly following rules will probably just get you in trouble. Most importantly, unit tests make the code rigid. It makes little sense to write heaps of testing code against interfaces which will be changed soon.  Harry spent a lot of time rewriting test code when he had to add features to his parser.  Even though I just said that you shouldn’t be just following rules, I will nevertheless introduce the golden rule of unit testing:

Test against interfaces which are not likely to change.

When the eventual refactoring is done, chances are that you’ll want to preserve some interface.  In Ethan’s case, he wasn’t so much refactoring but rewriting, but the module interface stayed the same.  Tests using only that module interface will help Ethan to test his new code.  Glass box tests designed for a particular implementation of the interface are not going to be helpful for Ethan.

If you’re writing software to go in a robotic spacecraft and you’re not exactly going to maintain and enhance that software after the launch because, to put it bluntly, you’d probably screw up and cause the $500m tin can to fly into the sun… In that kind of code everything can be unit tested, because nothing is going to be changed anyway, and it’s going to be worth it to gain a 0.1% improvement in the likelihood of catastrophic failure.  But in your regular software product which evolves all the time, along with the requirements, you’ll be wise to follow the golden rule and write tests only against those interfaces which don’t hopefully change all that often.

Oh, and please don’t ever parse XML using a heap of hairy regex code.

Related posts:

1. The Essence of Lambda
2. Programming Problems in Disguise

I’m constantly amazed at people who work with a computer every day, several hours per day, much of it typing in text, but don’t know how to type.  You’ve seen it: hands happily waving above the keyboard, twin index fingers scanning the rows of plastic keys, sometimes pecking at one; head bowed down, eyes fixed at the keyboard only to occasionally glance up at the screen to check the results.

Of course, I’m not the only one to have observed this.

I was an OK typist already as a kid.  I remember taking a typing class, in the ninth grade I think, and scoring a straight A without even trying (or a 10 actually, the Finnish school grade system goes from 4 to 10).  My friends also marveled at my typing speed.  Regardless of this, I had bad typing habits.  For one thing, I was looking at the keyboard a lot.   Then at high school I got a new friend who was, to my great frustration,  even faster than me.  He knew touch typing.  I didn’t even know what the little raised dots on F and J were for.

During the first year at the university I decided to finally do something about my typing.  Unlearning my old typing method turned out to be really hard.  I would always just fall back to my old ways, because that’s what I had been doing for ten years, and it was just so much easier.   At some point, it dawned on me that I would have to start from scratch: I would have to learn to type with a non-qwerty keyboard layout.  My old skills would be useless on the new layout, so I would be forced to relearn.

At the time, most of the writing I did was either in Finnish or program code, so Dvorak didn’t seem like an optimal choice. If I would have to learn a whole new layout, why not make something that would be perfect for me? So, I wrote a small program which captured all my key presses and kept it running for a week or so. Then I wrote a program which analyzed all the data, gathered the most common chords of two and three letters, and arranged they keys on the keyboard in an optimal way.

This is what my program made for me:

Changing the keyboard layout turned out to work wonderfully.  I couldn’t fall back to my old qwerty skills anymore, and started making real progress with touch typing.  Within a couple of weeks, I was up to my old typing speed.  After a couple of months I was already faster.

I still use this layout.

Nowadays I can easily reach 85-90 words per minute cold, typing in unfamiliar English text. In my first language, which is Finnish, I’m probably somewhat faster.  The fun part is that I have a keyboard layout nobody else in the known universe can use.  It’s kinda fun to let a coworker sit at my computer and watch them fumble.

How did you learn touch typing?  Or do you still look at your keyboard?

Related posts:

1. A Developer’s Most Important Interface
2. The Programming High

Vague ideas

In the beginning there are vague ideas, manifested as cubes of air floating around in the office, of what the finished product should do and what it should look like.

Renoir: La Vague

Actually, that’s not correct.  In fact, if you were to ask anyone who has been sitting in the meetings mulling over the project scope, they would probably tell you that they have a pretty good idea of what it’ll look like. Granted, some people will say that just to avoid having to go to another meeting mulling over the project scope. But many genuinely think they have a pretty good idea; some parts they may have even imagined in moderate detail, covering maybe 5% of the entire thing, and the rest they just blithely ignore.

Still, people think they have a pretty good idea, because the human mind is frighteningly good in filling in the gaps and creating this illusion that the gaps don’t even exist.  So, there’s not just one vague idea.  There are several vague ideas and they are all different.

Whence come deadlines?

Soon enough, it’s time to start sketching the project plan.  Based on the vague ideas someone comes up with guesstimates for how long it will take to turn each air cube into working code. In absence of detailed specs and explicit understanding, the guesstimates are of course wildly inaccurate.  And I don’t mean off by 10%, I mean crazy wrong, enough to make the corner of your left eye twitch embarrassingly at the project post-mortem.

Then there’s the meeting where the project is approved: the resources (manager-speak for developers) are granted, the scope agreed upon, and the deadline is set.  That’s the holy trinity of project parameters: resources, scope, and schedule.  And suddenly, they’re all more or less locked down.  The team, the features, the deadline.  One is based on vague ideas and another is derived from them.  Sounds good to me, now let’s get to work!

Pressure

Things appear to start off well.  Code is being written, features and tasks are being ticked off as complete, and the project seems to make good progress.

At some point, reality sinks in.  The requirements have become more clear, now that there are some builds that people can actually install and notice how it doesn’t do what they had imagined it would.  Features are added and removed.  GUIs are reworked.  Bug lists start to grow like the national debt.  The project manager grudgingly announces that the project is a bit late, but nothing we can’t handle, we’ll pick up the pace, start daily status update meetings, and work really hard.

The developers start to feel a little depressed.  The project is late, and they’re constantly being pressed to finish their tasks quicker and reworking parts they already though were fine.  They start to make bad engineering decisions because there’s “no time” to do things properly. The second half of the project is a mad coding frenzy where good design flies out of the window, sadly waving bye-bye, never to be seen again.

Go agile

If anything like the above happens in your organization, you should take a close look at your business model and decide if keeping made up schedules is really the most important thing.  How much will you lose if you admit that the original schedule estimates were just guesswork, and move the schedule up a few weeks or months?  How much will you lose if half of the time your developers are stressed, depressed, and doing a poor job just to meet the schedule?

Software projects don’t complete quicker because there is a tight deadline and a lot of pressure to finish on time.  They complete quicker because the developers feel productive, positive, and in control, they have a good understanding of the priorities, the specifications were up-to-date and descriptive, and the project is ahead of schedule.

So, the next time you notice you’re setting a project deadline several months in the future, based on mostly on air cubes, stop right there.  Instead, specify the first iteration well and set an ETA for that, at most one month in the future.  It’s much easier to specify and estimate in small pieces.  When the first iteration is done, rinse and repeat until it’s all done.   That’s what agile development is all about.

No related posts.