SKORKS

Learning A Software Development Lesson From A Children’s Poem

Alan Skorkin — Mon, 07 Jun 2010 11:33:12 +0000

As I was getting ready for work the other day, some children's program was on TV. Normally it's just background noise, but today something made me pay attention. Here is what I heard:

A wise old owl sat in an oak,
The more he heard, the less he spoke,
The less he spoke, the more he heard,
Why aren’t we all like that wise old bird.

I've recently been on a kick of extracting software development related lessons from everyday situations and events – that little poem immediately made me think of opinions. Software developers are an opinionated bunch, which is fair enough – comes with the territory. The trick with opinions though, is knowing when to shut up and listen to those of others. I could probably stand to follow that particular advice a lot more often :). If you can master that trick you're guaranteed to learn something. Even if all you learn is how to keep your opinion to yourself, once in a while, that's still a skill worth having.

Having said all of that, I do believe that you need to form an opinion about everything that happens, you don't have to defend your opinion to the death, but you do need to have one. There are important decisions to be made in software projects every day, regarding technology, process, requirements etc. You're not always going to be the most qualified person to make a decision, but if you let things slide without expressing your point of view – you deserve everything that happens. In that, software is like government. Having an opinion is the first step towards taking your destiny as a developer into your own hands (this applies equally to teams). And that's all I have to say about that for now.

Image by NuageDeNuit

Related posts:

When A ‘Mount Fuji’ Question Is Not Really A ‘Mount Fuji’ Question (Are You As Smart As You Think You Are)

Alan Skorkin — Wed, 02 Jun 2010 13:45:23 +0000

Many people hate 'Mount Fuji' style interview questions. Questions such as:

"How many barbers are there in your home town?"

"How would you move Mount Fuji?"

That last one being the most well known of these types of questions and the one from which the rest get their name. These questions were initially popularized by Microsoft, but many of the most well-known tech companies have used them since. They have recently fallen a bit out of favour, but you're still likely to come across some if you do few tech interviews here and there.

There is a good reason why many people don't like these types of questions. Most of the time, they don't really have a good answer. The question is designed to put a person under pressure (when they can't immediately see a way to tackle the problem) and see how they handle it, as well as see if they can come up with a creative solution on the spot. Some developers tend to enjoy that kind of challenge, others seem to loath it which is why opinions about these questions are highly polarised. While I personally don't think these questions are the be-all-end-all, I do believe they have their value, but that is not what I want to talk about.

You see the people who hate these questions, hate them with a passion. Whenever an interview contains a 'Mount Fuji' or two and people do badly, they know exactly where to cast the blame. "It was all because of the stupid Mount Fuji questions. I was never good at those, plus I wouldn't want to work for a company that would ask them anyway." That's the excuse you give to yourself. Makes you feel better about failing – which is actually not a bad rationalisation as far as rationalisations go. But, there is an issue with this thinking because sometimes, a question that sounds like a 'Mount Fuji' question isn't really one at all and with all the hate we direct towards them we might end up missing it completely. So, what excuse do we give ourselves then, hmmm :)?

What I want to do is present a couple of these 'Non-Mount Fuji' questions here. While they might seem like one of 'those' at first, they are actually very reasonable questions, relevant, with concrete solutions. In my opinion, very decent interview fodder as well, but you're likely to fail them just as badly as any 'Mount Fuji' problem if you haven't practiced, so let's have a look.

Here is the first:

There is a village full of people. One day the priest gathers all the villagers together and declares that their God has told him that some among the villagers are sinners. All sinners will be marked with a sign on their forehead so that all other people will be able to see if a person is a sinner or not, but noone will know if there is a mark on their own forehead. Furthermore, the mark will not be visible in the mirror and one person is forbidden from telling another if they are a sinner. The villagers must work out which among them are sinners at which point all sinners are to leave the village. The village has been given one week and if at the end of the week, there are still sinners present, the whole village will be destroyed.

The villagers go about their own business during the day, however the whole village gathers in the square at the end of every day so that everyone can see if there are still any sinners left among them. After the third such gathering all the sinners pack up their stuff and leave the village.

How many sinners were there? How did you arrive at that number?

This is a deductive reasoning question. We tend to use this kind of reasoning all the time when writing software, so the question is highly relevant to the software development profession. This question has many different forms and is pretty common, so you may have heard it before. If you have you should be able to answer it easily right :)?

Here is the second:

You are one of several recently arrested prisoners. The warden, a deranged computer scientist, makes the following announcement:

You may meet together today and plan a strategy, but after today you will be in isolated cells and have no communication with one another.

I have setup a "switch room" which contains a light switch, which is either on or off. The switch is not connected to anything.

Every now and then, I will select one prisoner at random to enter the "switch room". This prisoner may throw the switch (from on to off, or vice-versa), or may leave the switch unchanged. Nobody else will ever enter this room.

Each prisoner will visit the "switch room" arbitrarily often. More precisely, for any N, eventually each of you will visit the "switch room" at least N times.

At any time, any of you may declare: "we have all visited the 'switch room' at least once". If the claim is correct, I will set you free. If the claim is incorrect, I will feed all of you to the crocodiles. Choose wisely!

Devise a winning strategy when you know that the initial state of the switch is off.

Devise a winning strategy when you do not know whether the initial state of the switch is on or off.

This is a distributed coordination problem. The reasoning used to solve this one is relevant when it comes to both distributed computing and concurrency. Infact, I 'stole' this question from Chapter 1 of the "The Art of Multiprocessor Programming", which is a great book I've been browsing recently (it really is a very interesting book and I will likely be going through it in a lot more detail, don't worry though, you'll read about it right here when that happens :)). In my opinion this is an even better question than the last, it feels more 'computery' for whatever reason. More than that, once you've worked it out, the solution can pretty easily be expressed in code to validate your reasoning (which is exactly what I did). Infact, with a little bit of creative thinking, even the first question can be validated through some code. If you can express something in code it's a winner as far as I am concerned.

So What Do We Do With These?

What I want you to do is use this as an opportunity to practice. It doesn't really even matter if you're likely to ever get this kind of question in an interview; these are good questions to give the old programming brain a bit of a workout. Not only is this an opportunity to problem-solve, but it can also be an opportunity to write some quick, interesting code. After going through this exercise, if you ever do see these kinds of questions in an interview, you'll hopefully be able to distinguish them from the 'hated Mount Fuji' ones :).

So, get your thinking caps on and try to solve these questions – then post your solutions in the comments below. The first person to get everything right gets a prize, actually I lie, there is no prize, but apparently incentives don't work anyway, so it doesn't matter. What you will get is a decent mental workout and the satisfaction of solving an interesting problem. Don't worry if you can't get it though, you'll get better with practice as long as you give it a go. Anyway, I won't leave you in the lurch, I'll post my solutions to both of those questions in a few days, including the code I wrote to validate the second one (I might even write some code for the first one if I get time). If you're really brave, then also post how long it took you to work out the answer (it doesn't really matter, getting these right at all would be a very decent effort, but if you want to show everyone how much of a genius you really are, this is your chance :)). I am really looking forward to seeing what you come up with.

Images by Molas and "lapolab"

Related posts:

On Personal Skills And How Even Riding A Bike Is Not Like Riding A Bike

Alan Skorkin — Mon, 31 May 2010 13:19:37 +0000

It seems that in this industry we're operating under the false assumption that once we learn (or hear about) a skill, it is with us for good. You only have to look at a typical developer's resume for an example – it is truly buzzword central. I mean c'mon, just what exactly do you remember about XQuery or iBATIS at this point? I sometimes look at some of my own resumes I have used in the past and can't help being impressed – if only I were really that awesome :). If you're honest with yourself you can quickly classify all those skills into 3 categories:

Stuff you have used recently and know pretty well
Stuff you used to know pretty well, but haven't used for ages
Stuff you've had some exposure to, but don't really remember much about

Only the skills and tech that end up in the first group truly deserve a place in that buzzword list. The problem is that we don't tend to prune skills/tech off our resumes all that often. More stuff makes us sound more impressive – who wants to mess with that. This would actually have been ok if we made a conscious decision to include neglected and forgotten skills for marketing purposes, but the sad thing is the vast majority of developers/people truly believe that all those skills deserve a place.

The Power Of The Mind

When we want something to be true, we're really good at convincing ourselves that it actually is. Companies do it all the time, they want to be the market leader so they portray/see themselves as the market leader – the reality of the situation doesn't matter. The same is true for individuals. We have some minor exposure to a skill and we convince ourselves that we're better than a total novice. This may be true initially, but without some reinforcement, this quickly ceases to be the case. That however is irrelevant, we already believe we have some level of expertise and so the buzzword list grows. What's even more interesting are the tricks our mind plays on us when it comes to skills we actually did have some expertise in. You vividly remember when C, Smalltalk or XSLT was your bread and butter, it seems like only yesterday. But, it wasn't yesterday, it was years ago, sometimes many years – you would be surprised how far those skills can degrade with time. On some level we're aware of this, after all we're not stupid, but we tell ourselves that it doesn't matter, we're just a little rusty, we can pick it all up again in no time – after all, it's just like riding a bike. Well, let me tell you something about riding bikes.

Just Like Riding A Bike

I learned to ride a bike when I was very young and spent quite a large proportion of my childhood on one. It was simply the thing that we did during summer. I was pretty good (even if I do say so myself :)), that's what constant practice will do for you. Then, my family moved to Australia and I didn't ride for many years, until, during one fateful school camp, mountain biking was on the list of activities we had to participate in. I fully expected to ace that activity, my riding memories were still vivid in my mind. The reality was I sucked rather badly. Oh, I could ride of course, you never forget the absolute basics, but that was the only thing I really felt confident about. I could no longer judge properly how fast I could take the corners without falling over. I couldn't tell how fast I could safely go downhill, so I took it easy. Riding in a pack with other people seemed scary and unnatural and after only a short while my legs were aching and I was out of breath. Sure, the basics remained, but all the skills, which are built through continued practice and reinforcement, were no longer there. I was essentially reduced to a rank novice.

The Harsh Truth

Every skill is the same, without continued usage and practice they quickly begin to degrade. Your mind begins to page-out the knowledge and experience to make room for other information. Some goes into "long term storage", some is lost completely, but the net result is you degrade towards 'noobhood'. The longer you leave a skill untouched, the more you mind will page-out. It takes longer for skills you were more familiar with and some knowledge is reinforced in other ways so that you never really loose it; but the hard-earned expertise, the things that make you more than just mediocre can disappear with surprising speed. This is both sad and dangerous. It is sad because in software, we devote inordinate amounts of time and effort to acquire and master the skills we use in our work. When the pace of our industry forces us into using wholly different sets of skills/technologies all that time and effort begins to go to waste. Previously useful skills get relegated into disuse and only see the light of day in the resume buzzword section. This is especially sad considering that the "useless" old-school skills are often not quite so useless after all. Ruby is all the rage these days, but to make things go fast, what are all the native extensions written in – C, so what's old-school anymore? It is dangerous, because we absolutely refuse to acknowledge that we may not be as expert in some skills as we used to be. Sure we might loudly proclaim how we have forgotten everything about skill/technology X, but deep down we still believe that we can give these kids a run for their money when it comes to X (I am not precisely sure which kids and what money, but you get the picture :)). This is why we proudly retain X in the buzzword section, but if someone really needed that expertise in a hurry (like your employer for example), would you be able to deliver? I am certain you'll be able to get it eventually, but so could any other reasonably competent person, there is a difference between that and expert knowledge.

So What Are We Gonna Do About It

We can simply be scrupulously honest and remove all the skills and tech we're not sure about from the resume. But that actually helps noone, despite what I said above, you with your "rusty" skills, are still better than a complete beginner when it comes to those areas. So, by removing that stuff from your resume you're actually underselling yourself – it is wasteful and an easy way out. Rather than doing that, why not make sure your hard-earned skills and knowledge never get quite as rusty again.

As developers, we tend to do a lot of practice and self-study anyway, but most of us do it in a very diffuse and unfocused fashion. We learn about the latest and greatest things that strike our fancy and we practice by building stuff that catches our interest. All of that is fine as far as it goes, but we can do a lot better. Here is what I suggest. Engage a in a little self-examination by taking all your skills and sorting them into four buckets:

Core skills – these are the skills you consider primary to the direction you want your career to go. You want to make sure you become increasingly proficient with these and so should devote the most time to practicing them. There will only be a few of these at most (there is only so much time in a day).
Supporting skills – these skills support and underpin your core skills. You want to practice these to a lesser extent than the core skills, but you still want to see slow and steady growth. If chosen correctly many of your supporting skills will get a work-out as you practice your core skills, but you should still make sure you devote some time to them exclusively.
Peripheral skills – these are the skills that you consider valuable and worth maintaining. You're not looking to grow these skills, but you don't really want to lose them either. You want to devote enough time to these so that they don't fall into disuse.
Misc – this is the bucket for "the rest". The truth is you can't be an expert in everything, so this should contain the skills that you believe will add the least value to your development going forward.

You essentially end up devoting you practice and study time to your core, supporting and peripheral skills on a rotating basis, dividing your time between all of them based on their importance. It is difficult to sort your skills in this fashion (and even to work out what constitutes a skill, is building web apps a skill, or should it be specified into building Rails apps or alternatively generalised into being an HTTP protocol expert), so it might be useful to get some help, but it is an exercise that is eminently worth going through. What it gives you is a set of goals and a strategic framework for where you want to take your skills and your career. Rather than being slave to the latest trends when it comes to your practice, it will allow you to have some focus which can only make your self-study that much more productive (you'll be able to ignore the fads because you know what you need to focus on and work towards). The side benefit of this is that after a while, you will be a lot more confident about the buzzword section of you resume and be able to prune it without making it look sparse.

So, which skills should be your core and which should be supporting? This will be different for everyone, I've already told you what I think about fundamentals, so that is something to go on, but ultimately it is up to you to work out what knowledge and skills you want to devote the most time to depending on where you want to take your career. It is perhaps worth exploring this in a lot more detail (it is fine to have a strategic view, but it doesn't actually tell you what to do :)) and I might do this in a later post. Of course the other side of the coin is this, even if you do work out what skills you want to focus on, how do you actually go about practicing them, do you just write a lot of code? What if some of the skills I value are not coding related? That aspect of it is what I would call the tactical view of your growth as a developer – here it is all about deliberate practice and that is something I will definitely explore further in subsequent posts. Stay tuned.

Images by ItzaFineDay and Pete Prodoehl

Related posts:

How To Be A Real Elite Programmer And Make Sure Everybody Knows It

Alan Skorkin — Sat, 29 May 2010 15:09:01 +0000

Real elite programmers can't have distractions like kids and spouses in their life. Cut your family out of your life to maximize coding time. While you're there cut your friends out as well to maximize coding time even more.
Real elite programmers don't have hobbies that aren't coding. If you still engage in activities that don't involve a computer, this will have to change. If you find yourself burning out, just man-up and push through it.
Real elite programmers do their best work at night. Sleep is for wussies.
Real elite programmers hate their day job. Otherwise they would have nothing to complain about to other real elite programmers.
Real elite programmers eat pizza and drink Dr. Pepper. No other food is allowed, if you have fruit and vegetables in the house, they will have to go – stick to the diet of champions.
Real elite programmers don't read about programming, they DO the programming. Don't bother with books, just keep programming. Remember, you know best, those book-writing wimps are just a bunch of noobs, they should have been coding instead of writing, you're way better than that.
Real elite programmers are always right. This makes sense since everyone else is just a stupid noob.
Real elite programmers already know everything that is worth knowing. All the other knowledge is useless and only noobs would be interested in it.
Real elite programmers do not acknowledge anyone who is not a real elite programmer.
Real elite programmers know when to say "RTFM" in response to a question. This is a good answer to any question – noobs should know better than to ask stupid questions in the first place.
Real elite programmers are on social media. This is allowed, since it involves a computer. It is infact their duty since there are so many people on social media who need to be corrected about all things programming.
Real elite programmers complain loudly about the lack of in-depth programming content on social media. They are however too elite to provide this content.
Real elite programmers know when to say "TL;DR" in response to any in-depth programming content on social media.
Real elite programmers are particularly offended by minor inconsistencies in grammar, punctuation, colour schemes as well as any other subjective stylistic choices.
Real elite programmers never create any code for others to use. They have already told everyone they are awesome; there is no need to prove it.
Real elite programmers know how to put argumentative noobs in their place. Stupid noobs must be told they are stupid and ugly if they have the audacity to question elite wisdom.
Real elite programmers condescend to anyone who is not interested in whatever the real elite programmer is currently interested in.
Real elite programmers particularly despise non-programmers, they are worse than stupid noobs.
Real elite programmers use the most complicated and roundabout techniques to explain simple concepts. All must know how smart the real elite programmer is.

I do hope you realise I am being facetious here. However, there is a grain of truth in every joke. I have seen instances of this type of behaviour and have been guilty of engaging in some of it myself from time to time – you probably have as well. Just something to watch out for.

Image by Amarand Agasi

Related posts:

Ruby Procs And Lambdas (And The Difference Between Them)

Alan Skorkin — Thu, 27 May 2010 14:08:09 +0000

As you know, I am a big believer in knowing the basics. I find it is especially valuable to go back to the fundamentals of whatever skill you're trying to master periodically, as you gain more experience. Somehow you're always able to extract something valuable from the experience due to the extra insight you've acquired along the way. Recently I've been looking more closely at some Ruby concepts that I may have glossed over when I was learning the language initially, so I will likely be writing about this in some of my upcoming posts. I thought I would begin with one of the most powerful features of Ruby – procs (and lambdas :)).

What's So Good About Procs?

You know how everything in Ruby is an object, well, as it turns out that's not quite true. Ruby blocks are not objects! We can discuss the implications of that, but let's refrain, accept it as fact and move on. So, blocks are not objects, but you can turn them into objects without too much trouble. We do this by wrapping our block in an instance of the Proc class (more on this shortly). This is really great since it turns our block into a first class function, which in turn allows Ruby to support closures and once a language has closures, you can do all sorts of interesting things like making use of various functional concepts. That however is a story for another time, before we dive deep into those areas, let us really understand how Procs work.

Procs And Lambdas In Detail

There are three ways to create a Proc object in Ruby. Well, actually there are four ways, but the last one is implicit. Here they are:

Using Proc.new. This is the standard way to create any object, you simply need to pass in a block and you will get back a Proc object which will run the code in the block when you invoke its call method.
```
proc_object = Proc.new {puts "I am a proc object"}
proc_object.call
```
```
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
I am a proc object
```
Using the proc method in the Kernel module. Being in the Kernel module, this method is globally available. It is basically equivalent to Proc.new in Ruby 1.9, but not in Ruby 1.8. In Ruby 1.8 the proc method is equivalent to lambda. As you may have guessed there is a difference between procs and lambdas (see below), so you need to be aware of whether you're using Ruby 1.8 or 1.9. If you're using 1.9 there is a way to find out if you're dealing with a proc or a lambda. The Proc object lambda? method:
```
proc_object = proc {puts "Hello from inside the proc"}
proc_object.call
puts "Is proc_object a lambda - #{proc_object.lambda?}"
```
```
alan@alan-ubuntu-vm:~/tmp$ rvm use 1.9.1
Using ruby 1.9.1 p378
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
Hello from inside the proc
Is proc_object a lambda - false
```
If you're in 1.8 this method will not exist at all. By the way, did you notice how I had to switch Rubies and was able to do so painlessly through the awesome power of RVM, hope you're already using it. As far as I am concerned, if you need to switch between different versions of Ruby, don't bother with the proc method, just use Proc.new and you will never have an issue.
Using the Kernel lambda method. Once again, this is globally available being in the Kernel module, using this method will create a Proc object, but it will be a lambda as opposed to a proc (if you really need to know what the differences are at this point you can skip to the explanation :)).
```
proc_object = lambda {puts "Hello from inside the proc"}
proc_object.call
puts "Is proc_object a lambda - #{proc_object.lambda?}"
```
```
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
Hello from inside the proc
Is proc_object a lambda - true
```
The implicit way. When you write a method which will accept a block, there are two ways you can go about it. The first is to simply use the yield keyword inside your method. If there is a yield in a method, but there is no block you will get an error.
```
def my_method
  puts "hello method"
  yield
end
 
my_method {puts "hello block"}
my_method
```
```
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
hello method
hello block
hello method
a.rb:7:in `my_method': no block given (yield) (LocalJumpError)
    from a.rb:11:in `'
```
The other way is to use a special parameter in your method parameter list. This special parameter can have the same name but you will need to precede it with an ampersand. The block you pass into this method will then be associated with this parameter, you will infact no longer have a block – it will be a Proc object. You will simply need to call the Proc call method to execute the code in the block you passed in. The difference is, you can now return your block (or rather the Proc object) as the return value of the method, thereby getting your hands on a Proc object without having to use any of the above three methods explicitly.
```
def my_method(&my_block)
  puts "hello method"
  my_block.call
  my_block
end
 
block_var = my_method {puts "hello block"}
block_var.call
```
```
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
hello method
hello block
hello block
```

Those are all the different ways to get a Proc object in Ruby (either a proc or a lambda).

Fun And Interesting Trivia About Procs And Lambdas

If you have a method to which you pass a block, but instead of calling yield, you call Proc.new without giving it a block of its own, it will return a proc that represents the block you passed in to the method e.g.:
def some_method
  Proc.new
end
 
my_proc = some_method{puts "I am a happy proc"}
my_proc.call
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
I am a happy proc
In Ruby 1.9, there is actually another way to create lambdas. It isn't really a different way, it is simply some syntactic sugar, but it is crazy looking, so worth knowing :). The normal way to create a lambda is using the lambda keyword, and if you want your lambda to take parameters, you simply pass them in the normal block way e.g.:
normal_lambda = lambda {|param1, param2| puts "param1: #{param1} and param2: #{param2}"}
normal_lambda.call(10, 20)
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
param1: 10 and param2: 20
With the new syntax it is slightly different. We replace the lambda method with the -> method (as in minus followed by greater than sign). The parameters move out of the block, and move into some brackets right after the -> method (so they look like method parameters, but in reality are still block parameters). Everything else is the same, so our previous example would look like:
normal_lambda = ->(param1, param2) {puts "param1: #{param1} and param2: #{param2}"}
normal_lambda.call(10, 20)
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
param1: 10 and param2: 20
Everything still works fine (in Ruby 1.9, don't try this in 1.8), but if you're using Netbeans, this syntax is way too crazy for it and it will barf all over itself. This syntax is an interesting curiosity, but I recommend sticking to plain old lambda for your own sanity (especially if you're using Netbeans) and that of everyone else who has to read your code later.

Procs Are First Class (Functions That Is)

Once we have created our first class function (in Proc form), there are a few things we may want to do with it, such as calling it or comparing it to another Proc. As usual, Ruby makes all of this interesting. Let's look at Proc equality first. You can compare procs or lambdas to each other, using the == method (just like with any other object). Unlike many other objects though, defining exactly the same proc twice will not make them equal. For example:

string1 = "blah"
string2 = "blah"
 
puts "#{string1 == string2}"
 
proc1 = Proc.new{"blah"}
proc2 = Proc.new{"blah"}
 
puts "#{proc1 == proc2}"

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
String are equal? - true
Procs are equal? - false

There are only two ways that two procs or lambdas will be equal to each other. One way is to define two really simple procs that have the same object in them e.g.:

proc1 = Proc.new{string1}
proc2 = Proc.new{string1}
 
puts "Procs are equal? - #{proc1 == proc2}"

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
Procs are equal? - true

This is not really very useful. The other way that two procs will ever be equal is if they are clones of each other (one was created by using the clone or dup method on the other).

proc1 = Proc.new{|x|"blah1"*x}
proc2 = proc1.dup
 
puts "Procs are equal? - #{proc1 == proc2}"

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
Procs are equal? - true

Let us now look at executing procs. We have already seen the call method; this is the most common way to execute a proc or lambda.

puts Proc.new{|x|"blah1"*x}.call(2)

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
blah1blah1

But, there are actually two other ways. The first is to use the array syntax, this works in both Ruby 1.8 and 1.9:

puts Proc.new{|x|"blah1"*x}[2]

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
blah1blah1

The other is the dot syntax, which only works in 1.9.

puts Proc.new{|x|"blah1"*x}.(2)

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
blah1blah1

This syntax being 1.9 only will once again cause Netbeans to barf all over itself :). Both of those ways are nice and succinct, but I still recommend you use call; it is not that much longer and is much more readable, both for yourself and for others. The array syntax could be especially confusing, consider:

some_var[2,3]

That could refer to an array, a string, a proc – without context it's a bit too ambiguous for my tastes, stick to call.

Even More Trivia About Procs And Lambdas

You may not know this, but procs and lambdas have an arity method which allows you to find out how many arguments a Proc objects expects to receive (Method objects also have this method). For example:
my_proc = Proc.new{|x|"blah1"*x}
puts "I need #{my_proc.arity} arguments"
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
I need 1 arguments
Pretty simple – the fun bit starts when the block you use to create the proc expects an arbitrary number of arguments (or requires some, but sponges up the rest using a * prefixed final parameter) e.g.:
my_proc = Proc.new{|x, *rest|"blah1 #{x} - #{rest}"}
puts "I need #{my_proc.arity} arguments"
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
I need -2 arguments
What the hell? When you get a negative number for you arity, it means there is potentially an arbitrary number of arguments involved. In this case, all you can do is find out how many arguments are required (the rest will be optional). You do this by using the ~ operator on the return value of the arity method:
my_proc = Proc.new{|x, *rest|"blah1 #{x} - #{rest}"}
puts "I actually require #{~my_proc.arity} arguments, the rest are optional"
alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
I actually require 1 arguments, the rest are optional
I reckon that's interesting stuff to know, it won't save your life on a daily basis, but there will be that one time when knowing this will make you look like an absolute superstar :).

So What IS The Difference Between Procs And Lambdas?

It took a while, but we finally got here. Long story short, the real difference between procs and lambdas has everything to do with control flow keywords. I am talking about return, raise, break, redo, retry etc. – those control words. Let's say you have a return statement in a proc. When you call your proc, it will not only dump you out of it, but will also return from the enclosing method e.g.:

def my_method
  puts "before proc"
  my_proc = Proc.new do
    puts "inside proc"
    return
  end
  my_proc.call
  puts "after proc"
end
 
my_method

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
before proc
inside proc

The final puts in the method, was never executed, since when we called our proc, the return within it dumped us out of the method. If, however, we convert our proc to a lambda, we get the following:

def my_method
  puts "before proc"
  my_proc = lambda do
    puts "inside proc"
    return
  end
  my_proc.call
  puts "after proc"
end
 
my_method

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
before proc
inside proc
after proc

The return within the lambda only dumps us out of the lambda itself and the enclosing method continues executing. The way control flow keywords are treated within procs and lambdas is the main difference between them. Or rather, the way that the return keyword and the break keyword are treated is the difference since most of the others are treated in exactly the same way in both procs and lambdas. Let's have a look at break. If you have a break keyword within a lambda it is treated just like a return:

def my_method
  puts "before proc"
  my_proc = lambda do
    puts "inside proc"
    break
  end
  my_proc.call
  puts "after proc"
end
 
my_method

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
before proc
inside proc
after proc

However, if we change the lambda to a proc:

def my_method
  puts "before proc"
  my_proc = proc do
    puts "inside proc"
    break
  end
  my_proc.call
  puts "after proc"
end
 
my_method

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
before proc
inside proc
a.rb:64:in `block in my_method': break from proc-closure (LocalJumpError)
    from a.rb:66:in `call'
    from a.rb:66:in `my_method'
    from a.rb:70:in `'

We get a LocalJumpError, what happened? Well, the break keyword is used to break out of iteration, and we didn't have an iterator around our proc, so we couldn't break out of anything, so Ruby tries to punish us with an error. If we put an iterator around our proc, everything would be fine:

def my_method
  puts "before proc"
  my_proc = proc do
    puts "inside proc"
    break
  end
  [1,2,3].each {my_proc.call}
  puts "after proc"
end
 
my_method

alan@alan-ubuntu-vm:~/tmp$ ruby a.rb
before proc
inside proc
after proc

So, how come the lambda didn't have the same error, we didn't have an iterator there either. Well, this is the other difference between procs and lambdas. Having a break keyword within a lambda will cause us to break out of the lambda itself, but within a proc we get an error if there is no iterator around the proc and we break out of the iterator if we do have one.

There you go, the difference between Ruby procs and lambdas. There are other control flow keywords (next, redo, retry, raise), but they behave the same in both procs and lambdas, that behaviour, however, is quite interesting, I urge you to have a play with it. More interesting Ruby fundamentals soon.

Images by rofi and Richard Moross

Related posts:

Why I Love Reading Other People’s Code And You Should Too

Alan Skorkin — Wed, 19 May 2010 13:42:25 +0000

It occurs to me, that many programmers hate reading code – c'mon admit it. Just about everyone loves writing code – writing code is fun. Reading code, on the other hand, is hard work. Not only is it hard work, it is boring, cause let's face it, any code not written by you just sucks (oh we don't say it, but we're all thinking it). Even your own code begins to look increasingly sucky mere hours after you've finished writing it and the longer you leave it the suckier it seems. So, why should you waste hours looking at other people's crappy code when you can be spending this time writing awesome code of your own? Weren't we just over this; give it a couple of hours and then come back and see if your code still looks awesome. You're never going to become a master of your craft if you don't absorb the knowledge of the masters that came before you. One way to do that is to find a master in person and get them to teach you everything they know. Is this possible – certainly, is it probable – not so much, you'd have to be extremely lucky. You don't really need luck though, we're fortunate to be in a profession where the knowledge and skill of all the masters is right there for us to absorb, embedded in the code they have written. All you have to do is read it, sure it might take you a bit longer without someone sitting there explaining it to you, but it's eminently possible. To put it in perspective, try becoming a great carpenter just by looking at a bunch of well constructed furniture.

I love reading code, I have always intuitively felt that you get a lot from it, yes it can be annoying and boring, but the payoff is well worth the effort. Consider this, if you wanted to become a great writer, would you focus exclusively on writing? You could try it, but you're not going to get far. It is an accepted fact that most great writers are also voracious readers. Before you can hope to write anything decent you need to read other great writers, absorb different styles, see what others have tried before you and feed your creative self. Your knowledge will slowly grow and eventually, your own writing will begin to exhibit some maturity, you will develop a 'feel' for it. Coding is no different, why would you expect to write anything decent if you've never read any great code? The answer is you shouldn't. Reading great code is just as important for a programmer as reading great books is for a writer (I can't take credit for this thought, it belongs to Peter Norvig and he is awesome, so take heed).

Even if all of this is unconvincing, there is one fact which is undeniable. Being good at reading code is important to your survival as a professional developer. Any non-trivial project these days will be a team effort and so there will always be large chunks of code you had no hand in which you have to work with, modify and extend. And so, code reading will likely be the most used and most useful skill you can have; better bite the bullet and get good at it – fast.

How To Read Code Like … Some Kind Of Code-Reading Guy

I can't tell you how many times I've seen programmers scroll up and down through an unfamiliar piece of code, for minutes on end, with a sour expression on their face. They would later declare the code to be unreadable crap and why waste the time anyway; we can just work around the problem somehow. I am not sure what the expectation was here, absorb the meaning of the code by osmosis, or maybe intensely stare your way to enlightenment? You don't read code by just looking at it for ages, you want to understand it and make it your own. Here are some techniques I like to use, it is not an exhaustive list, but I have found these to be particularly helpful.

Try to build and run it. Often this is a simple one step process, like when you're looking at work code (as opposed to random code). However this is not always the case and you can learn a lot about the high level structure of the code from getting it to build and execute. And, on the subject of work code, you are intimately familiar with how to build your current project are you not? Builds are often complex, but there is a lot of understanding to be gleaned from knowing how the build does its thing to produce the executable bits.
Don't focus on the details just yet. The first thing you want to do is get a bit of a feel for the structure and style of the code you're reading. Start having a browse and try to figure out what the various bits of the code are trying to do. This will familiarise you with the high level structure of the whole codebase as well as give you some idea of the kind of code you're dealing with (well factored, spaghetti etc.). This is the time where you want to find the entry point (whatever it happens to be, main function, servlet, controller etc.) and see how the code branches out from there. Don't spend too long on this; it's a step you can come back to at any time as you gain more familiarity with the code.
Make sure you understand all the constructs. Unless you happen to be the premier expert on your programming language there are probably some things you didn't know it could do. As you're having a high level fly-through of the code, note down any constructs you may be unfamiliar with. If there are a lot of these, your next step is obvious. You're not going to get very far if you have no idea what the code is doing syntactically. Even if there are only a few constructs you're unfamiliar with, it is probably a good idea to go look them up. You're now discovering things about your language you didn't know before, I am happy to trade a few hours of code reading just for that.
Now that you've got a good idea about most of the constructs, it is time to do a couple of random deep-dives. Just like in step 2, start flying though the code, but this time, pick some random functions or classes and start looking through them line by line. This is where the hard work really begins, but also where you will start getting the major pay-offs. The idea here is to really get into the mindset (the groove) of the codebase you're looking at. Once again don't spend too much time on this, but do try and deeply absorb a few meaty chunks before moving on. This is another step to which you can come back again and again with a bit more context and get more out of it every time.
There were undoubtedly things in the previous step you were confused about, so this is the perfect time to go and read some tests. You will potentially have a lot less trouble following these and gain an understating of the code under test at the same time. I am constantly surprised when developers ignore a well-written and thorough test suite while trying to read and understand some code. Of course, sometimes there are no tests.
No tests you say, sounds like the perfect time to write some. There are many benefits here, you're aiding your own understanding, you're improving the codebase, you're writing code while reading it, which is the best of both worlds and gives you something to do with your hands. Even if there are tests already, you can always write some more for your own benefit. Testing code often requires thinking about it a little differently and concepts that were eluding you before can become clear.
Extract curious bits of code into standalone programs. I find this to be a fun exercise when reading code, even if just for a change of pace. Even if you don't understand the low level details of the code, you may have some idea of what the code is trying to do at a high level. Why not extract that particular bit of functionality into a separate program. It will make it easier to debug when you can execute a small chunk by itself, which – in turn – may allow you to take that extra step towards the understanding you've been looking for.
The code is dirty and smelly? Why not refactor it. I am not suggesting you rewrite the whole codebase, but refactoring even small portions of the code can really take your understanding to the next level. Start pulling out the functionality you do understand into self-contained functions. Before you know it, the original monster function is looking manageable and you can fit it in your head. Refactoring allows you to make the code your own without having to completely rewrite it. It helps to have good tests for this, but even if you don't have that, just test as you go and only pull out functionality you're sure of. Even if the tests seem totally inadequate – learn to trust your own skill as a developer, sometimes you just need to go for it (you can always revert if you have to).
If nothing seems to help, get yourself a code reading buddy. You’re probably not the only person out there who can benefit from reading this code, so go grab someone else and try reading it together. Don't get an expert though, they'll just explain it all to you at a high level and you will miss all the nuances that you can pick up by going though the code yourself. However, if nothing works, and you just don't get it, sometimes the best thing you can do is ask. Ask your co-workers or if you're reading open source code, try and find someone on the interwebs. But remember, this is the last step, not the first one.

If I was pressed for time and needed to understand some code reasonably quickly and could only pick one of the above steps, I would pick refactoring (step 8). You will not be able to get across quite as much, but the stuff you do get across you will know solid. Either way, the thing you need to keep in mind is this. If you're new to a significant codebase, you will never get across it instantly, or even quickly. It will take days, weeks and months of patient effort – just accept it. Even having an expert right there with you doesn't significantly cut the time down (this is what the last post in my teaching and learning series will be about). If however, you're patient and methodical about your code reading (and writing) you can eventually become intimately familiar with all aspects of the project and become the go-to man when it comes to the codebase. Alternatively you can avoid reading code and always be the guy looking for someone to explain something to you. I know which one I would rather be.

Seek Out Code Reading Opportunities – Don't Avoid Them

We love to write new code, it's seductive cause this time we're going to get it just right. Ok, maybe not this time, but next time for sure. The truth is, you're always evolving in your craft and you're never going to get it just right. There is always value in writing new code, you get to practice and hone your skills, but there is often just as much (if not more) value in reading and playing with code written by others. You not only get some valuable tech knowledge from it, but often domain knowledge as well (after all, the code is the ultimate form of documentation) which is often more valuable still.

Even code that is written in an esoteric fashion, without following any kind of convention, can be valuable. You know the code I am talking about, it almost looks obfuscated, but it wasn't intended to be (for some reason it is often Perl code :)). Whenever I see code like that, I think of it this way. Just imagine the kinds of stuff you will learn if you can only decipher this stuff. Yeah, it's a major pain, but admit it, there is that niggling secret desire to be able to write such naturally obfuscated code yourself (no use denying it, you know it's true). Well, if you invest some time into reading code like that, you're much more likely to eventually be able to write it – doesn't mean you will write it, but you want to be ABLE to. Lastly, attitude is always paramount. If you view code reading as a chore then it will be a chore and you will avoid it, but if you choose to see it as an opportunity – good things will happen.

Images by mafleen, Scott Ableman and LisaThumann

Related posts:

Closures – A Simple Explanation (Using Ruby)

Alan Skorkin — Mon, 17 May 2010 14:12:30 +0000

There are a great many decent developers out there who don't know what a closure is. I don't really have any concrete stats on this matter, it is simply an intuitive assessment based on experience. But, you know what – that's fair enough, considering that the most popular languages that are in use right now don't support closures (Java, C++). When the language you use day in day out doesn't support a concept, that concept is not going to be high on your agenda; infact you may not even be aware of it. And yes, I agree, good developers will know several (or perhaps many) different languages and there are plenty out there that do support closures – you're bound to stumble across one at some point. Which just goes to show that when most developers learn new programming languages, they go about it completely the wrong way and so when you hear someone say they know a dozen languages, they are likely overstating the situation by quite a margin. But, I'll leave that discussion for another time – today it is all about closures.

You probably came across closures when you were at uni, you just don't remember. Over the last 10 years or so, the curriculum of software related degrees has evolved to the point where concepts like closures are not emphasized, it is predominantly a functional concept and functional languages are out – Java is in. They were trying to make the degrees more industry-relevant and as a result there is a generation of programmers who have been professionally crippled (to some extent) through no fault of their own. You trust the education system to do right by you and there isn't much you can do when this trust is misplaced. I need go no farther than myself for an example. I did Java in the first year of my degree, C in the second. The first time we were exposed to functional programming, was during an AI subject where we had to do some Lisp along with having to learn a whole slew of other concepts. Needless to say, functional programming wasn't the emphasis in that subject; many people struggled and struggled badly. We certainly had some exposure to closures during that subject, but who had time to think about that, it was all most people could do to try and wrap their heads around a new style of programming when they had just spent two years getting used to a completely different one. By the time I found myself in industry, closures – along with most other functional concepts – were long forgotten. It took me years of self-study before I rediscovered this stuff – stuff that should have been the foundation of my education as a software developer. I once had a go at my CS degree for not teaching some things that would have made it more relevant. Were I to write a similar post these days, I would be a lot harsher – I still might.

The bitter irony is that while Java is still well and truly in, the functional style is coming back with a vengeance. Fully functional languages (Clojure) and functional hybrids (Scala) are in vogue and so the concepts that underpin them are once again relevant. But, the damage has already been done and will continue to be done – academia does not adjust itself in a hurry. You can't afford to be ignorant of functional concepts any more, even JavaScript – that Web 2.0 language that we all love to hate – is a functional hybrid. And so developers scramble to get their heads around these concepts and as they scramble they are presented with stuff like this:

"In computer science, a closure is a first-class function with free variables that are bound in the lexical environment."

What the hell does that even mean! Wikipedia does go on to explain it a little bit better, but without some functional context, it is still tough going. Here is another one you commonly hear:

"A closure is a function that is said to be "closed over" it’s free variables"

Really! A 'closure' closes over, why, that makes it all much clearer. Seriously, the next question is always bound to be a plea for clarification. An explanation like that is worse than no explanation at all, although it certainly does make someone sound smart. But, when you're trying to teach someone something it is not about self-aggrandisement – we can do better. Let's give it a go.

A Concept Explained Simply

A closure is basically a function/method that has the following two properties:

You can pass it around like an object (to be called later)
It remembers the values of all the variables that were in scope when the function was created. It is then able to access those variables when it is called even though they may no longer be in scope.

Let's fill in some more details. As you may have guessed, you don't get closures for free; they must be explicitly supported by the language. In order for the language to be able to support closures, it must support first-class functions. A first class function is a function that can be treated like an object in that you can store it in collections and pass it as a parameter to other functions. As I said, the ability to be passed around is the first property of a closure.

A normal function is defined in a particular scope (i.e. in a class) and can only be called within that scope. This function has access to all the variables in the scope that it is defined, like the parameters that are passed into it as well as class variables. A closure on the other hand may be defined in one scope and be called in a completely different scope (since we can pass it around before calling it). Because of this, when a closure is created, it retains the values of all the variables that were in scope when the closure was defined. Even if the variables are generally no longer in scope when the closure is called, within the closure they still are. In other words, the closure retains knowledge of its lexical environment at the time it was defined.

Hopefully that "lexical environment" sentence above is starting to make a little bit more sense now, but I am sure it would make a lot more sense if we had an example.

An Example In Ruby

No explanation is complete without some examples, that is usually what it takes to make things start to fall into place. We will use Ruby since it supports closures and I like it.

In Ruby, closures are supported through procs and lambdas. These constructs are very similar, but there are some subtle differences (I might do a post about that soon). Let's create a closure and see how it fulfils the two properties described above.

class SomeClass
  def initialize(value1)
    @value1 = value1
  end
 
  def value_printer(value2)
    lambda {puts "Value1: #{@value1}, Value2: #{value2}"}
  end
end
 
def caller(some_closure)
  some_closure.call
end
 
some_class = SomeClass.new(5)
printer = some_class.value_printer("some value")
 
caller(printer)

When we execute we get the following output:

alan@alan-ubuntu-vm:~/tmp$ ruby closures.rb
Value1: 5, Value2: some value

As you can see, the value_printer function creates a closure, using the lambda construct, and then returns it. We then assign our closure to a variable and pass that variable to another function, which then calls our closure. This satisfies the first property of a closure – we can pass it around. Notice also that when we called our closure, we printed out "5" and "some value". Even though both the @value1 and value2 variables were both well and truly out of scope in the rest of the program when we finally called the closure; inside the closure they were still in scope as it retained the state of all the variables that were in scope when it was defined. And so, our lambda satisfies the second property also which makes it a closure. I hope the example has made things a bit clearer.

Of course, we can look into closures a little bit more deeply. For example, how do they retain the values of the variables that were in scope when the closure was defined? This must be supported by the language and there are two ways to do that.

The closure will create a copy of all the variables that it needs when it is defined. The copies of the variables will therefore come along for the ride as the closure gets passed around.
The closure will actually extend the lifetime of all the variables that it needs. It will not copy them, but will retain a reference to them and the variables themselves will not be eligible for garbage collection (if the language has garbage collection) while the closure is around.

If the language supports the first way, then if we create two or more closures which access the same variables, each closure will have its own distinct copy of those variables when it is called. If a language supports the second way, then all closures will reference the same variables, i.e. they will in effect be dealing with exactly the same variable. This is how Ruby does things. Here is an example:

class SomeClass
  def initialize(value1)
    @value1 = value1
  end
 
  def value_incrementer
    lambda {@value1 += 1}
  end
 
  def value_printer
    lambda {puts "value1: #{@value1}"}
  end
end
 
some_class = SomeClass.new(2)
 
incrementer = some_class.value_incrementer
printer = some_class.value_printer
 
(1..3).each do
  incrementer.call
  printer.call
end

This produces the following output:

alan@alan-ubuntu-vm:~/tmp$ ruby closures.rb
value1: 3
value1: 4
value1: 5

We create two closures this time, one to increment a value and the other to print it out. When we then call both of the closures three times, we can see that both are operating on the same variable as the value is incremented on every iteration. If Ruby handled retaining variables for closures by copying them, we would have had the original value of 2 printed out on every iteration since our incrementer and printer closures would each have had a distinct copy of the @value1 variable.

Why Are Closures Useful

Now that we understand closures a bit better, what's the big deal about them anyway? Well, it depends on what kind of language you're using. In a functional language they are a very big deal. Functional languages are inherently stateless, but we can use closures to essentially store some state which will persist as long as our closure lives on (i.e. if the closure changes the value of a variable it will retain the new value the next time the closure is invoked). I hope it is reasonably self-evident how this can be useful. The existence of constructs such as closures along with several others, allow functional languages to be very terse in expressing logic which means you can do more with less code.

With non-functional languages things are a little murkier. There are better ways to represent state when it comes to imperative languages, so the only thing that makes closures compelling in that situation is the fact that you can use them to write terser code while still taking advantage of the imperative style. The existence of closures is partly the reason why people can often do more with less code in languages like Ruby (which support closures) than they can in languages like Java (which do not). If you have other things to add about the awesomeness of closures in any context, then do leave a comment, the more information we share the better off everyone will be.

Images by Tattooed JJ and ~Oryctes~ (Off)

Related posts:

Who Deserves The Credit For Software Craftsmanship and Great Design?

Alan Skorkin — Wed, 12 May 2010 14:27:13 +0000

How did people design great software before OO and how did we ever manage to run a successful project before Agile came along? Those are the questions young programmers undoubtedly ask themselves at some point in their career (I know I did, early on). You hear about and read horror story after horror story from the dark ages of software development (sometimes the dark ages are years ago, other times they include the project before this one) and then you hear about how things like OO or Agile are transforming (or have transformed) the industry. The truth is, projects didn't fail due to lack of Agile or OO (or any other 'revolutionary' practice) and there were plenty of successful projects even in the bad old days (it's just that you don't tend to hear about the good ones). Don't get me wrong, I believe in what Agile stands for and I haven't even seen non-OO software in my career :), but I don't subscribe to the buzzwords, I subscribe to what these concepts stand for, the core ideals if you like. Sometimes, you need to take a step back and consider that while the moniker may be shiny and new, the concepts it tries to package are often time honoured and fundamental (or as time honoured and fundamental as you can get considering the relative youth of our industry).

I was thinking about the Unix Philosophy the other day and it struck me that pretty much all of the ideas seem really familiar. So, I went and cracked open a copy of "The Art Of Unix Programming", just to make sure I wasn't getting myself mixed up. And what do you know, it was just as I remembered, but this time, I had a bit more experience to put it all in perspective for me. Here are a couple of quotes by Doug McIlroy:

"Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features."

Why, he is simply espousing making your software modular as well as giving a nod to separation of concerns. I dunno about you, but I keep something similar in mind when designing my classes, small, self-contained, I thought it was all OO principles :).

"Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them"

Release early, release often. Refactor and leave the code better than you found it.

"Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them"

Automate everything. Does it smell like agile in here or is it just me.

Let's keep going we don't even need to turn the page. Here is what Rob Pike had to say

"Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.
Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest."

Hmmm, avoid premature optimisation.

"Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures."

YAGNI. Even more agile wisdom. You have to remember that this stuff is from the 70s, years before OO got any traction and decades before the agile movement was even on the horizon. And these quotes are a distillation of thought, after these people had time to mull it over; they were actually doing this stuff years before.

Eric S. Raymond, the author of "The Art Of Unix Programming", summarised the Unix philosophy in the following 17 tenets.

1. Rule of Modularity: Write simple parts connected by clean interfaces.

2. Rule of Clarity: Clarity is better than cleverness.

3. Rule of Composition: Design programs to be connected to other programs.

4. Rule of Separation: Separate policy from mechanism; separate interfaces from engines.

5. Rule of Simplicity: Design for simplicity; add complexity only where you must.

6. Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.

7. Rule of Transparency: Design for visibility to make inspection and debugging easier.

8. Rule of Robustness: Robustness is the child of transparency and simplicity.

9. Rule of Representation: Fold knowledge into data so program logic can be stupid and robust.

10. Rule of Least Surprise: In interface design, always do the least surprising thing.

11. Rule of Silence: When a program has nothing surprising to say, it should say nothing.

12. Rule of Repair: When you must fail, fail noisily and as soon as possible.

13. Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.

14. Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.

15. Rule of Optimization: Prototype before polishing. Get it working before you optimize it.

16. Rule of Diversity: Distrust all claims for "one true way".

17. Rule of Extensibility: Design for the future, because it will be here sooner than you think.

Let’s pick out some choice tidbits. How about number 4, separate interface from implementation, I think we're all well drilled in that one these days :). I am really fond of number 8, this is the stuff that most agile developers live by, robust, simple and clear code (you can take robust to mean well tested if that is your preference). Number 15 is a good one, anyone do spikes when they code – I know I do? Number 16 is my favourite, I interpret it as "if something doesn't work, stop doing it and do something else", reminds me of Scrum (or as Conan the barbarian would say it "SCrom" :)). Much of it is about keeping it simple, i.e. the KISS principle, just like many of the concepts behind agile. But of course, just because we're keeping it simple, doesn't mean we are foolish about it (it's keep it simple stupid not keep it stupid stupid), which is where number 17 comes in.

So where am I going with all of this, did the dudes who wrote Unix invent agile (or at least the underlying ideals), or software craftsmanship, or great software design? It's unlikely, after all Don Knuth said:

"Premature optimization is the root of all evil."

He said this in 1974 (the link is pdf) – but was likely thinking it for years and there were probably people before him as well. It all had to start somewhere, of course, as I said; software is a relatively young industry. But, it doesn't really matter who the first person was that came up with these ideas, the fact that the ideas continue to figure prominently in newer concepts tells me that they are just basic common sense. When programming was born as a career, smart people came along, did the first project or two and quickly twigged on to what seemed to make sense and what didn't. All the nonsense, confusion and failure was mostly from a bunch of other people trying to fit software development into moulds it wasn't suited for – refusing to realise that this was an industry fundamentally unlike any that has come before and must have its own "rules" (rather than trying to inherit some from manufacturing or random other industries that "looked" similar). The saddest thing was that these people had power and could impose their will and make their ideas stick, which is why the whole industry went to hell for quite a while (even to this day), with signs of recovery only now beginning to emerge.

So Does Agile Get Credit For Anything?

Christianity and Islam are some of the most popular religions in the world. This should surprise you, since they are much younger than many other religions (especially Islam). Do you know why these religions are so popular? Because they were the first missionary religions, the first to actively seek out new converts for their faith. Most of the religions that came before, either did not actively welcome outsiders or were actively hostile to them. These religions are a testament to the fact that if you actively try to seek out converts and don't give up, even due to major setbacks, you will slowly gain more and more followers and will eventually spread your message far and wide.

The agile movement was the beginning of the "missionary work" for these common sense software development concepts that we have now all come to know and love. Up until then, there was no focus, no organisation, and even when there was organisation, there was no attempt to seek out converts (other developers) and actively spread the message about how software should be built. The agile movement changed all that, which just goes to show that developers are just people and are willing to buy into something that seems sensible and hits close to home. But, the message itself was never revolutionary, just like the religious messages were never revolutionary. It was simply a bunch of common sense concepts that many good developers have followed for years.

What I am trying to say (besides some curious factoids and a history lesson :)), is this. Don't faddishly follow "best practices", remember rule 16 – there is no "one true way". It doesn't matter what tech you're using, it doesn't matter what the configuration of your task board is. The key is to surround yourself with good people (this really is key and deserves a separate post) and to always remain aware of the situation – then figure out what makes sense and just do that (even if what makes sense is to follow best practice or to do the exact opposite :)). Good programmers have always known the score.

Image by RobertBasil

Related posts:

Software As A Destination vs Software As A Journey

Alan Skorkin — Mon, 10 May 2010 13:54:34 +0000

I am feeling a little existential and metaphorical today, so forgive me if this post has more than just a slight tinge of existential metaphor about it. I do hope it still makes some sense.

There are two fundamental ways of looking at software development. One is all about the final product and the direct benefits you can get from it. The other is all about the lessons you learn from the process of building the software. I call them software as a destination and software as a journey. Historically speaking and even into the present day, the majority of companies that build any kind of software are 'software as a destination' companies. They want to extract some direct financial benefit from the software they build (which is fair enough), until this benefit is realised, the process of building the software is just a resource (time, money) drain. On the other hand, most individuals who build software for those companies would instinctively like to be (and in many cases ARE) 'software as a journey' people. We want to enjoy our work, we want to learn as much as we can, we want to interact with those around us, we want to build code that we can be proud of and we want to get paid. All of this will happen throughout the project rather than just at the end, and so it is all about the journey, the final success of the product is almost icing on the cake. It seems that there is a fundamental disconnect between what a company wants to get out of building software and what the developers will try to extract from the experience.

This is why the software development industry is in a sort-of screwed-up state, or at least it has been for many years (things are getting better). Companies see the destination and want to arrive there as quickly as they can, the sooner they get there the sooner they see the benefits from all that money they spend and the sooner they can start getting their money back, either directly or indirectly. Anything that stands in the way of this goal must be swept aside and this includes the developers who want to enjoy the journey and 'smell the roses' on their way to the final product. Of course, the companies hold all the cards (they employ you, pay you – which indirectly, feeds and clothes' your children) and so can usually get their way if they really want to. Developers can try to make noise about the value of paying attention to the journey, but if the company is dead set on doing things their way, that's the way things will be done, developers can either fall into line or there will be other developers to replace them. However, just like a real, physical journey, if you have your eyes set firmly on the destination and are not looking where you're putting your feet, there is a good chance you will trip over some roots, or a snake will bite you, or you won't notice that cliff which is right between you and where you're going. This is how we get failed projects.

Once the project and the company are at the bottom of the cliff, you can kinda tell that everyone is in trouble. But it is difficult to figure out exactly how you ended up where you are, you had your eyes firmly on the destination and forgot to pay attention to what was happening around you, so it is hard to pinpoint the mistakes that landed you at the bottom of a cliff. Companies will do a post-mortem, come up with some vague reasons for why things went to hell, everyone feels better that they have an explanation (leaving everyone none the wiser in reality). And so next time they go out and do the same thing all over again. All the while, the answer is staring everyone right in the face. If only they had listened to the developers who told them to pay attention to the journey and not worry so much about the destination things would have been different. Even novice programmers soon learn, from personal experience, that when we slow down and pay attention to what we are doing right now, we get a lot more out of it, we learn more and the final product is improved as a result. There are only benefits, no downsides, alright, you lose some time in the short term, but gain it all back with interest in the long term. But, at the macro level where committees make decisions, this same lesson takes years and decades to permeate the hive-mind. Not to mention the fact that seeing only the destination is much easier than constantly paying attention, and so even the occasional developer gets seduced by the 'software as a destination' mentality (it's really more than just the occasional), which sets everyone back by introducing conflicting opinion and muddying the waters unnecessarily.

There was a bunch of people who were all 'software as a journey' people who basically got sick of all the back and forth. They recognised the fact that the journey is what really matters not just at the personal level, where each individual developer is concerned, but at the team, project and company level as well. They decided to give a united voice to the message that as long as you take care and extract the maximum benefit along the way, the destination will take care of itself – you will arrive somewhere you want to be. It may not be exactly the same place where you thought you would end up, but it will be a nice place nevertheless and best of all, you will have a very good idea of precisely how you got where you are, despite not having a narrow focus on the final objective from the start. More like-minded people joined the initial bunch and the movement got more organised. Some of these people started companies of their own; others brought their message to larger organisation. And now that the 'software as a journey' message had a united voice (albeit still a comparatively small one), the success of looking at software development this way could be demonstrated, but more importantly, disseminated to the industry at large.

It is a major undertaking for a company to develop any significant piece of software. Not just from a financial perspective, but as far as airing out the dirty baggage and discovering the process inefficiencies that every significant company will have. After all, in order to put something logically into code you need to figure out exactly what that something is and how it works. This is a golden opportunity for the company to streamline and improve how their business works, figure out if some processes are unnecessary. You're building software, but you're paying attention to what you're doing not just at the technical level but at the organisational level as well. As a result, not only do you get more robust software that better fits your needs, but you also improve the day to day operations of the company; all because you were paying attention to the journey rather than focusing on the destination.

But, it is so much easier to simply look towards the goals and on paper it seems like it SHOULD be more efficient. If you're in management you want to seem more efficient and focused rather than being wishy-washy and touchy-feely, preaching self-awareness and a less-rapid pace just isn't strategic enough. And so companies create artificial deadline pressures (like end of financial cycle of some sort), to make everything neater and make the people in charge look good and it feels like for every step the agile movement takes towards running software projects better, some company somewhere, takes two steps back.

Robert A Heinlein once said:

"The supreme irony of life is that hardly anyone gets out of it alive."

Infact, he had many other quotes which I think are quite relevant to the software profession, maybe I'll do a post about that at some point. To me this quote says that it's the moment that really matters, enjoy the life you have now rather than constantly looking forward to something better. I am not sure if there is any irony to be found in software, but if you direct all your focus towards your goal without paying due attention to the nitty gritty of what you're doing every day, you're likely to not get any useable software out of it. As long as you have a reasonable idea of where you want to end up, you just need to get the details right and the bigger picture will tend to sort itself out. On the other hand, you can have the clearest possible goal in mind, but if you let the details slide, bad things will almost certainly happen.

Images by Norma Desmond and alicepopkorn

Related posts:

Sort Files Like A Master With The Linux Sort Command (Bash)

Alan Skorkin — Sat, 08 May 2010 16:23:51 +0000

If you do your development work in Linux, there are certain commands that you owe it to yourself to master fully. There are a number of these with the main ones being grep, find and sort. Just about everyone has at least a passing familiarity with these commands, but with most people the knowledge is superficial, they don't even realise how powerful those commands can be. So, if you really put in the effort to master them, not only will you make your own life much easier, but you will also be able to impress all you friends with your elite Linux skills when you pair with them :). I will cover grep and find (as well as other valuable commands) in subsequent posts – here we will concentrate on sort.

Note: I am using bash, so your mileage might vary if you're using a different shell.

Sorting is a fundamental task when it comes to programming, if you have a decent knowledge of various sorting algorithms, their advantages and disadvantages, you will be a better software developer for it. However, often enough you just don't need to draw on this deeper knowledge. Whether you're answering an interview question about sorting or simply need to quickly sort some data in you day to day work – the Linux sort command is your friend.

The extent of most people's knowledge ends with:

sort some_file.txt

Which is fair enough, you rarely need to dig deeper, the default behaviour will usually do what you need and when it doesn't – we have Ruby or Perl, we can hack something together. Well I hope that we're somewhat more curious than your average developer :). As much is we like hacking things together, if a tool can already do all the work for us, we want to know about it. We can look at the man page for sort and discover all sorts of interesting bits, but even the man page is not really clear on the more advanced aspects of sort usage. It helps to have examples to truly grok the kinds of stuff you can do with sort, so let's have a look.

Sorting Basics

To start with we have the following file:

alan@alan-ubuntu-vm:~/tmp/sort$ cat letters.txt
b
D
c
A
C
B
d
a

We'll do the most basic sort first:

alan@alan-ubuntu-vm:~/tmp/sort$ sort letters.txt
a
A
b
B
c
C
d
D

Looks good, how about doing it in reverse:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -r letters.txt
D
d
C
c
B
b
A
a

Also easy, but what if we want to be case insensitive? Hang on a sec, according to the output it's already case insensitive. But the man page has an option for this:

-f, --ignore-case
              fold lower case to upper case characters

If sort is case insensitive by default, what is this option for. We'll that one is a bit of a gotcha, it looks like GNU sort is case insensitive by default, but the man page also contains the following:

*** WARNING *** The locale specified by the  environment  affects  sort
       order.  Set LC_ALL=C to get the traditional sort order that uses native
       byte values.

What this means that we need to set the LC_ALL environment variable to get the behaviour that we would expect from sort (i.e. capital letters before non-capitals). Let's try that:

alan@alan-ubuntu-vm:~/tmp/sort$ export LC_ALL=C
alan@alan-ubuntu-vm:~/tmp/sort$ sort letters.txt
A
B
C
D
a
b
c
d

That's better, and now our -f option is actually useful:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -f letters.txt
A
a
B
b
C
c
D
d

That looks ok, but something still seems a little funny, all the capitals appear before all the non-capitals every time. That's because the sort is not stable, but we can make it stable:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -f -s letters.txt
A
a
b
B
c
C
D
d

Now that's exactly what we wanted, it's case insensitive and stable, i.e. if the small letter appeared before the capital when unsorted (and the letters are the same), this order will be the same in the sorted list.

Ok, but what if we have numbers:

alan@alan-ubuntu-vm:~/tmp/sort$ cat numbers.txt
5
4
12
1
3
56

A normal sort is not what we want:

alan@alan-ubuntu-vm:~/tmp/sort$ sort numbers.txt
1
12
3
4
5
56

But we can fix that:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -n numbers.txt
1
3
4
5
12
56

And, if our lines happen to have some leading blanks:

alan@alan-ubuntu-vm:~/tmp/sort$ cat blank_letters.txt
b
D
   c
A
C
    B
d
a

We can easily ignore those and still sort correctly (using the -b flag):

alan@alan-ubuntu-vm:~/tmp/sort$ sort -f -s -b blank_letters.txt
A
a
b
    B
   c
C
D
d

Of course none of this actually writes the sorted output back to the file, we only get it on standard output. If we want to write it back to the file, we have to redirect the output to a new file and then replace the old file with the new file:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -f -s -b blank_letters2.txt > blank_letters2.sorted
alan@alan-ubuntu-vm:~/tmp/sort$ cat blank_letters2.sorted
A
a
b
    B
   c
C
D
d
alan@alan-ubuntu-vm:~/tmp/sort$ mv blank_letters2.sorted blank_letters2.txt
alan@alan-ubuntu-vm:~/tmp/sort$ cat blank_letters2.txt
A
a
b
    B
   c
C
D
d

As an alternative to redirection, sort also has the -o option:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -f -s -b blank_letters2.txt -o blank_letters2.sorted
alan@alan-ubuntu-vm:~/tmp/sort$ cat blank_letters2.sorted
A
a
b
    B
   c
C
D
d

Alright, this is all pretty standard stuff, let's see how we can do something a little bit more fancy.

Advanced Sort Usage

One of the most common use cases when it comes to sort is to pipe its output to uniq, in order to remove any duplicate lines, but this is often not necessary as sort has a uniq-type option built right in (-u):

alan@alan-ubuntu-vm:~/tmp/sort$ cat blank_letters.txt
b
D
c
A
C
B
d
a
alan@alan-ubuntu-vm:~/tmp/sort$ sort -f -s -u blank_letters.txt
A
b
c
D

It even took into account the fact that we wanted to be case insensitive. Compare that to a case-sensitive sort with a uniq option set:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -s -u blank_letters.txt
A
B
C
D
a
b
c
d

Nothing is removed as there are no duplicate lines.

But what if my lines are a little bit more complicated than just a single letter, what if there are multiple columns and I want to sort by one of those columns (not the first one). This is also possible:

alan@alan-ubuntu-vm:~/tmp/sort$ ls -al | sort -k5
total 36
-rw-r--r-- 1 alan alan   14 May  9 00:12 numbers.txt
-rw-r--r-- 1 alan alan   16 May  9 00:00 letters.txt
-rw-r--r-- 1 alan alan   16 May  9 00:40 blank_letters.txt
-rw-r--r-- 1 alan alan   20 May  9 00:33 blank_letters2.sorted
-rw-r--r-- 1 alan alan   20 May  9 00:33 blank_letters2.txt
-rw-r--r-- 1 alan alan   20 May  9 00:37 blank_letters3.txt
-rw-r--r-- 1 alan alan   84 May  8 23:15 file1.txt
drwxr-xr-x 3 alan alan 4096 May  8 23:13 ..
drwxr-xr-x 2 alan alan 4096 May  9 00:40 .

As you can see we sorted the output of ls by the size (the 5th column). This is what the -k option is for. Basically -k tells sort to start sorting at a particular column given a particular column separator. The column separator is, by default, any blank character. So in the above example, we told sort to sort by the 5th column given the fact that the column separator is blanks.

But we don't have to be restricted by the default separator, we can specify our own using the -t option. Let's sort the first 10 lines of my /etc/passwd file (promise you won't hack my machine since I am giving it away like that :)) by the 4th column – the group id. As you know the /etc/passwd file uses the : (colon) character as the separator (for more info on the format see here). Here is the output unsorted:

alan@alan-ubuntu-vm:~/tmp/sort$ cat /etc/passwd | head
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
lp:x:7:7:lp:/var/spool/lpd:/bin/sh
mail:x:8:8:mail:/var/mail:/bin/sh
news:x:9:9:news:/var/spool/news:/bin/sh

There is a group id of 65534 in there which should appear last. Let's sort it:

alan@alan-ubuntu-vm:~/tmp/sort$ cat /etc/passwd | head | sort -t: -k4 -n
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
lp:x:7:7:lp:/var/spool/lpd:/bin/sh
mail:x:8:8:mail:/var/mail:/bin/sh
news:x:9:9:news:/var/spool/news:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
games:x:5:60:games:/usr/games:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync

We had to do a numeric sort since we're dealing with numbers, and we specified : (colon) as the column separator. The output is sorted correctly with 65534 being on the last line. Pretty cool! But the fun doesn't stop there, we can sort by multiple columns, one after the other. Consider this list of IP addresses:

alan@alan-ubuntu-vm:~/tmp/sort$ cat ips.txt
192.168.0.25
127.0.0.12
192.168.0.1
127.0.0.3
127.0.0.6
192.168.0.5

Let's sort it by the first column, so that all the addresses starting with 127 go together, and then sort it by the 4th column, to make sure that the IPs are sorted by the last column within each range.

alan@alan-ubuntu-vm:~/tmp/sort$ cat ips.txt | sort -t. -k 2,2n -k 4,4n
127.0.0.3
127.0.0.6
127.0.0.12
192.168.0.1
192.168.0.5
192.168.0.25

We specified the dot as the separator. The -k 2,2n syntax has the following meaning. Do a sort by column (-k), start at the beginning of column 2 and go to the end of column 2 (2,2). The n on the end is to indicate that we want to do a numeric sort since we are dealing with numbers. That is some powerful stuff, wouldn't you agree?

Cool/Useful Stuff

There is still more we can do with the sort command. Have you ever wanted to randomize the lines in a file? It is not a common use case, but does come in handy once in a while (if only for testing purposes, sometimes). Well, the sort command has you covered here also with the -R option (that's capital R):

alan@alan-ubuntu-vm:~/tmp/sort$ cat numbers.txt
5
4
12
1
3
56
alan@alan-ubuntu-vm:~/tmp/sort$ cat numbers.txt | sort -R
5
4
1
3
12
56
alan@alan-ubuntu-vm:~/tmp/sort$ cat numbers.txt | sort -R
3
4
1
56
5
12

We get a different order every time, which is what we would expect from randomizing the lines.

If you give sort multiple files on the command line, it will combine the contents of all the files and sort it as a whole:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -n numbers.txt numbers2.txt
1
1
3
4
4
5
7
8
10
12
22
23
26
56
56
68

This is really handy, but sometimes, the files you have are already sorted, you just want to merge them. Sort provides the -m options just for this purpose. The output of using sort on two files will be exactly the same whether you use -m or not, but merging should be faster:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -n -m numbers1.sorted numbers2.sorted
1
1
3
4
4
5
7
8
10
12
22
23
26
56
56
68

Lastly, if you just want to check if a file is sorted or not, without actually performing the sort, you have the -c option:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -n -c numbers.txt
sort: numbers.txt:2: disorder: 4
alan@alan-ubuntu-vm:~/tmp/sort$ sort -n -c numbers1.sorted
alan@alan-ubuntu-vm:~/tmp/sort$

There you go, the total awesomeness of the sort command laid bare. If you know any other handy things you can do with sort, do leave a comment. And remember – only use your new-found sort powers for good instead of evil :).

Image by SewPixie (so far behind!)

Related posts: