I use Wordpress as my blogging platform, it's a great tool, has a heap of plugins so you can do much with very little effort, which suits me fine :). As a software development blog I often have code snippets that I need to post and I want these to look good (i.e. formatting, syntax highlighting). This is why I use the WP-Syntax Wordpress plugin. I am not going to go into why it's so great, if you're curious you can check it out for yourself. Suffice to say, it does the job fine except for one very annoying issue. Whenever you have any kind of special characters in your code (which you inevitably do e.g. <, >, & etc.), these always render as their escaped representations. For example if you wanted to write:

if (foo < 5 && bar > 6)

you would end up with:

if (foo < 5 && bar > 6)

Extremely annoying. This only happens if you use the WYSIWYG editor in Wordpress, which is why the plugin FAQ recommends switching off the visual editor and using the source code editor, but that eventually gets to be beyond annoying.

The other day my annoyance threshold reached it's limit and I decided to find a solution to this issue. I armed myself with an apple (for munching purposes :)) and set to googling. I eventually found a couple of sites which mentioned this issue e.g. this one and this one. Both of these tried to point me to the following link – http://blog.felho.hu/escaping-problem-with-wp-syntax-wordpress-plugin.html.

Unfortunately when I tried to go there I got:

Parse error: syntax error, unexpected $end in /home/felho/www/blog.felho.hu/wp-includes/default-filters.php  on line 213

What a massive fail! Most of the time you don't want to duplicate information that can be found in other places on the web (which is why we link), but sometimes the beauty of the internet is the fact that you can find the same info in many different places. In this case – the internet failed. The information was only to be found in one place (with everyone else linking there) and that place was dead.

Thankfully the magic of google cache came to the rescue and I was able to recover the original post. So, in the interest keeping the information alive on the web :), I'll re-post the solution here. Here is what you need to do to fix that special character escaping issue:

• Jump into your Wordpress configuration and deactivate the WP-syntax plugin for the moment
• Jump into your WP-syntax plugin folder, in my case it was the wp-content/plugins/wp-syntax folder under the root of your Wordpress install
• Crack open the wp-syntax.php file and find the wp_syntax_highlight function (in my case it was on line 94)
• Inside the function find the following line:

$geshi = new GeSHi($code, $language);

Replace this line with the following:

$geshi = new GeSHi(htmlspecialchars_decode($code), $language);

If you're running your Wordpress install on a server with a PHP version which is older than 5.1, you will not actually have the htmlspecialchars_decode() function. Of course I would be surprised if your PHP version was older than 5.1, but just in case it is, here is what you need to do. You will need to roll your own htmlspecialchars_decode() function, so put in the following code right before the line that you replaced above:

if (!function_exists("htmlspecialchars_decode")) {
    function htmlspecialchars_decode($string, $quote_style = ENT_COMPAT) {
        return strtr($string, array_flip(get_html_translation_table(HTML_SPECIALCHARS, $quote_style)));
    }
}

• Save and exit the file and then go an reactive your plugin in the Wordpress config.

This should solve your special character escaping issue – it solved mine.

Here is the funny thing, after I made sure everything was working I went and had a look at the code in wp-syntax.php again, and here is what I found right above the line that we replaced:

if ($escaped == "true") $code = htmlspecialchars_decode($code);

Now, I am no PHP expert, but it looks like we shouldn't have needed to do what we did, this line should have taken care of everything, but obviously $escaped isn't equal to "true" and so htmlspecialchars_decode() function never gets called (either that or it does get called, but shouldn't be). I didn't really want to experiment with my live blog, but if you have a extraneous Wordpress install with WP-syntax floating around somewhere feel free to poke around. It feels like this shouldn't be an issue. Anyways, do let me know if you find something out. In the meantime you have a work-around thanks to blog.felho.hu (which is still down :)) and google cache.   

Image by Dr Case

Related posts:

1. Installing And Using SQLite With Ruby On Windows
2. The Most Handy NetBeans Shortcuts For Ruby Development
3. Writing A More Ruby-ish Array Intersection Function And Sorting Structs

In my previous post on boolean search, we wrote an efficient list (array) intersection function. But, to be honest, I wasn’t very happy with it. Considering the fact that we were using Ruby, it just wasn’t very Ruby-like. So, this time I am going to try and show how that function can be re-written to not only be more Ruby-ish but to also be tighter and easier to understand. In addition I will look at a couple of interesting aspects of OpenStruct, which will lead nicely into my next post on skip pointers (I’ve spoken about covering skip pointers before).

The Ruby-ish Intersection Function

The last time we implemented an array intersection function it looked like this:

def intersect_lists(list1, list2)
  final_list = []
  current_list1_index = 0
  current_list2_index = 0
 
  while(current_list1_index < list1.length && 
                          current_list2_index < list2.length)
    if list1[current_list1_index] == list2[current_list2_index]
      final_list << list1[current_list1_index]
      current_list1_index += 1
      current_list2_index += 1
    elsif list1[current_list1_index] < list2[current_list2_index]
      current_list1_index += 1
    else
      current_list2_index += 1
    end
  end
  final_list
end

There is actually nothing wrong with the code itself. It is an efficient implementation. The reason it is efficient is due to the fact that we only need to walk through each of the lists once, in order to produce the intersected list. So if the sizes of our two lists are x and y, the intersection takes x+y operations, i.e. the complexity of the algorithm is O(x+y). If we were writing that function in C, everything would be fine at this point, but we’re using Ruby. What we really want our function to look like is something along the lines of:

def intersect_lists(list1, list2)
  list1.each do |item1|
    list2.each do |item2|
      #stuff
    end
  end
end

This would actually work, but we go back to the naive array intersection implementation where we scan through the whole of one list for each of the elements in the other. What we really want is to be able to exit the inner each iterator to allow the outer one to advance, but then restart where we left off the next time we enter the inner one. This way we can compare all the values in both arrays to produce the intersection while still scanning both lists only once. Fortunately with Ruby, we can simply open up the array class and create just such an iterator:

class Array
  def each_continued
    if !@current_index
      reset_current_index
    end
    while @current_index < self.size
      next_index = yield(self[@current_index],@current_index)
      if next_index
        @current_index = next_index
        break
      end
      @current_index += 1
    end
  end
 
  def reset_current_index
    @current_index = 0
  end
end

As you can see our new iterator uses a class level variable (@current_index) to keep track of where the iteration is at. We stop the iteration based on the value that is returned from the block that needs to be passed to this iterator. If the value returned from the block is nil, we continue to iterate. When the value returned is a number, we set it to be the index from which we will restart the iteration next time the method is called. This allows us to not only stop and restart the iteration where we left off, but also roll the iteration forward or back if necessary. This is actually important as we will see shortly. There is only one issue with this implementation, the fact that we need to use a class level variable. The problem is the fact that when one full iteration through the array completes, you need to remember to rewind (i.e. must call reset_current_index) – not great. The only way to overcome this would be keep track of the current position of the iteration externally and pass this value in as a parameter to the iterator, but we won’t worry about that for now, for our purposes, resetting the array is not such a big deal.

We can now re-implement our intersection function, it will look like this:

def rubyish_intersect_lists(list1, list2)
  list3 = []
  list1.each do |value1|
    list2.each_continued do |value2, index2|
      if value1 == value2
        list3 << value2
        index2 + 1
      elsif value1 < value2
        index2
      end
    end
  end
  list3
end

You’ll probably agree that’s much tighter and more Ruby-ish looking. There are two important things to note, in order to understand how it works.

1. When the values from both lists match, we copy the result into our intersection array, we then need to advance both lists. In order to do this, we get the block of the inner iterator to return the index of the next value in the list. This will cause the inner iterator to terminate, but will also prime it to restart from the index we return out of the block (see what I mean about the importance of giving our iterator the ability to restart the iteration from any index we want). This way we advance our inner iterator and naturally advance the outer one, since the inner one has exited.
2. When the value of the array in the inner loop is bigger than the outer one, we want to keep advancing the outer iterator, but keep the inner one where it is. We do this very easily, by making the block of the inner iterator return the current index of the list. This has the effect of exiting the inner iterator and priming it to restart in exactly the same place on the next iteration, and of course it naturally advances the outer iterator.

The only other scenario is when the value of the outer loop iterator is bigger then the inner one, in which case we want to advance the inner one. This will happen by default, and so needs no special code to handle it.

Sorting OpenStructs

As I said, this post actually a lead into another post, which will be about skip pointers. This in itself, is no big deal, we still want to intersect lists, the only issue is, they have to be lists of objects rather than lists of primitive integers. After all you actually need somewhere to put the skip pointers (if you don’t really get it, don’t worry, I’ll explain in the next post). When I was playing around with skip pointers initially I decided to use OpenStruct to wrap the integers (which are the values of the arrays we were going to intersect). In hindsight I should have created an object, but I was a bit too far along and didn’t want to go around changing a bunch of code. Here is the problem.

In order to test my intersection function, I generate arrays of integers, using the following function.

def create_raw_mock_postings_list(min_postings, max_postings, corpus_size)
  postings_list = []
  num_postings = min_postings + rand(max_postings)
  num_postings.times do
    postings_list << rand(corpus_size) + 1
  end
  postings_list.sort.uniq
end

This allows me to dummy up what a posting list for a term would look like (a particular number of document ids each of which is a random number between 1 and the size of the corpus). The important bits are in the end i.e. postings_list.sort.uniq. It is very important that the posting list is sorted and all values are distinct. This is fine when your value are integers, it all just works. It would also be fine if the value were objects, since all you would need to do would be to implement the spaceship operator (<=>, so that you can compare them to each other) for everything to once again work (which is why I should have used an object). I, of course, was using OpenStruct and I don’t know of any way to implement the comparison operator.

When it comes to sort, this is reasonably easy to overcome, since that function takes a block that it can use for comparison, but when it comes to uniq, we’re out of luck. I needed a function similar to the following:

def create_wrapped_mock_postings_list(min_postings, max_postings, corpus_size)
  postings_list = []
  num_postings = min_postings + rand(max_postings)
  num_postings.times do
    wrapped_id = OpenStruct.new
    wrapped_id.value = rand(corpus_size) + 1
    postings_list << wrapped_id
  end
  postings_list.sort{|x,y| x.value <=> y.value}.block_uniq{|item| item.value}
end

Ruby goodness came to the rescue, since we can, once again, open up Array and create our own uniq function. Like so:

class Array
  def block_uniq
    uniqness_hash = {}
    uniqed_array = []
    self.each do |value|
      if block_given?
        uniqness_value = yield(value)
      else
        uniqness_value = value
      end
      if !uniqness_hash.key?(uniqness_value)
        uniqed_array << value
        uniqness_hash[uniqness_value] = nil
      end
    end
    uniqed_array
  end
end

We use a hash, to make sure the items are unique (i.e. if it is already in the hash, discard it). If no block is given to this function it will just operate like a normal uniq function, but when the block is there, it will use the return value of the block rather than the array items themselves to do it’s job. Implementing this function allows our create_wrapped_mock_postings_list function to work as intended. One last thing to note is the fact that our new block_uniq function preserves the order of the elements of the array we are uniquing, always good to keep things neat.

Intersecting Structs

Now that we’re dealing with structs rather than integers, our previous intersection function is no longer correct (for exactly the same reason, why sort and uniq were such a pain, I really should have used an object instead of OpenStruct, it’s a good learning experience, but let that be a lesson to me for later).

We could just slightly alter our intersection function to work with our structs:

def rubyish_intersect_for_wrapped(list1, list2)
  list3 = []
  list1.each do |item1|
    list2.each_continued do |item2, index2|
      if item1.value == item2.value
        list3 << item2
        index2 + 1
      elsif item1.value < item2.value
        index2
      end
    end
  end
  list3
end

This is actually what I did, but is not necessarily a good solution, since we can augment the original function to work either with integers or with structs or anything else, in a similar way to how we implemented our block_uniq function. For example:

def rubyish_intersect_for_wrapped(list1, list2)
  list3 = []
  list1.each do |item1|
    list2.each_continued do |item2, index2|
      if yield(item1) == yield(item2)
        list3 << item2
        index2 + 1
      elsif yield(item1) < yield(item2)
        index2
      end
    end
  end
  list3
end

We can now call this function like so, when we want to work with structs:

rubyish_intersect_for_wrapped(list1, list2) {|item| item.value}

when we have just plain integers, we would do:

rubyish_intersect_for_wrapped(list1, list2) {|item| item}

Of course, it would be nicer if we had used the block_given? method so that we don’t have to supply any block at all in the second case, but I’ll leave that one as an exercise :).

We’re now ready to tackle skip pointers and see if can actually get better performance (than O(x+y)) from our intersection function. This is coming up very shortly. 

Image by Thomas Hawk

Related posts:

1. Using Ruby Blocks And Rolling Your Own Iterators
2. A Wealth Of Ruby Loops And Iterators
3. More Advanced Ruby Method Arguments – Hashes And Block Basics

Much of the agile process literature is very general. There is a lot of focus on practices and philosophy and empowering teams (nothing wrong with that by the way), but there is also often a lack of practical advice on how to solve the day-to-day problems you run into when working in an agile environment. It is difficult, as every team encounters different problems and even similar problems often have a different root cause depending on the environment. Regardless, I can see a need for a cookbook type of text, but in the meantime I'd like to weigh on an issue that I have seen quite often with agile teams, that of pair programming and the apparent reluctance of many developers to actually swap pairs regularly.

If this kind of thing is of interest to you then remember to subscribe to my RSS feed as I often write about this sort of stuff. 

A Novice Agile Team

I don't really want to spend too much time here as the issues in the case of novice agile teams are reasonably clear. It is likely a people or a team dynamics problem. I would look at one of two things:

1. Did you introduce all the agile practices at once (the big bang approach)? If so, you've likely found your problem. It is often best to introduce practices slowly to allow everyone to acclimatize. Otherwise people start to feel adrift and try to anchor themselves somehow. The easiest way to do that (anchor yourself) is to make sure you complete the stories you start and so people avoid swapping pairs. It may be worth ignoring the lack of pair swapping and giving the team some more time to get used to the new process (focus on pair swapping a little later).
2. If people seem comfortable with the practices then your problem is probably lack of buy-in. The team doesn't understand the benefits of pair swapping and sees no need for it. Once again, there is a certain level of discomfort involved with pair swapping. You're leaving a story in a half-finished state and developers don't like to do that. You're also potentially leaving someone you get along with really well (you current pair) for a person who is less familiar to you. Without buy-in there is nothing to counter-balance that discomfort and pair swapping suffers. Your path here is clear, you need to sell people on the benefits that pair swapping brings. Communicate with the team and explain, until you get to the point where people start to prompt each other to swap pairs. 

It is that simple when it comes to a novice agile team.

The Experienced Team

When it comes to an experienced agile team, the problems are not as easy to diagnose. The people are fully aware of the benefits and everyone is comfortable with the process and yet pair swapping is still an issue. It is likely not a people or a team dynamics problem, the culprit is elsewhere in your process. Consider the fact that your stories might be too big – it takes too long for a pair to finish a story.

In order to comfortably swap pairs a developer needs to reach a certain level of understanding regarding a story:

• have a clear idea of the business requirements (where the current story is concerned)
• be comfortable with how everything will be implemented from a technical perspective
• have a good idea of how everything will look in a finished state

To get to this level of understanding you need to be involved with a story until it is approximately half-finished (based on personal experience). Otherwise you feel like you don't really have a handle on the situation, haven't yet made enough of a contribution and so will be reluctant to move on to something else.

One of the benefits of pair swapping is to get more people across a particular story, so it is actually beneficial to swap half way through a story. Unfortunately, not all stories are the same size even when the estimated point value is the same, so all stories reach the half-way point at different times. This is why most teams try to swap pairs at regular time intervals (i.e. half way through the day, start of every day etc.). But, if your stories take two or more days to implement, there will be a lot of reluctance to swap half way through the day. You can certainly make the swap interval longer, but then the new person needs to make up so much ground as far as context is concerned that it is almost counter-productive, they become a passenger. Not to mention the fact that when a story is long and complex you want to see it through to the end to get that satisfied, "job-well-done" feeling.

You would be better off recalibrating the point values for your estimates so that a 1-pointer becomes a lot smaller (less complex and so takes less time to implement) and consequently larger point-value stories are also smaller. I would go as far as to say that the majority of the stories should be 1-pointers with a few 2-pointers, anything bigger than that should be broken up.

If all of this is unconvincing, here are some reasons why smaller stories are better stories.

1. They allow the team to check in more often (not as many conflicts)
2. Smaller stories are easier to wrap your head around
3. Go along way to preventing situations where you're half way through multiple stories at the end of an iteration (because when you underestimate a small 1-pointer, you're only a few hours off, when you underestimate a huge 3-pointer, you could be off by more than a day)

And of course, people will be a lot happier to swap pairs when the stories are smaller as it takes a lot less time to reach that level of understanding (that I talked about above), where a pair swap is no longer a pain.

The "Rules" For Breaking Up Functionality

I have found that people tend to write stories according to a set of pet rules, e.g.:

• each story must be a vertical
• each story must have tangible business value (must be tied directly to requirements)
• you must be able to express a story in the form – "As an X I want to be able to do Y"
• etc.

The thing to remember is that all those are guidelines. If sticking to those rules makes a story too big then break them. Don't worry too much about business value and it is ok not to make a story a complete vertical. When you break a story up, remember that stories don't have to be completely independent. Agile teams often try to maintain the illusion that each story is independent and the system is built up from a bunch of discrete verticals. This is never true, there are always dependencies, so if you need to, you can reorganise your task board to make these explicit, this way it becomes easier to break up stories, since you no longer have to try and make each story make sense in isolation.

The lesson here is this, if you see an issue in an experienced agile team (like the reluctance to pair swap), don't automatically assume it is a people problem or lack of understanding. Look to other parts of your process, there may be a connection you wouldn't expect. Do you think there are other reasons why people are often reluctant to pair swap? If so, please leave a comment, I'd love to hear them.

Image by mikebaird

Related posts:

1. Rules Of Standup – You Don’t Need To Justify Your Own Existence
2. Are You Actually A Post-Agilist?
3. How To Get The Most Out Of Your Design Sessions

I find that whenever I get a large data file from somewhere (i.e. extract some data from a database, crawl some sites and dump the data in a file) I always need to do just that little bit of extra processing before I can actually use it. This processing is always just non-trivial enough and I do it just uncommonly enough for me to always forget exactly how to go about it. Of course, this is to be expected, if you learn something and want it to stick you have to keep doing it. It's all part and parcel of how our brain works when it comes to learning new skills, but that doesn't make it any less annoying.

Back to our data file, for me I find that I almost always need to do 3 things (amongst others) before doing anything else with my file.

• delete the first line (especially when pulling data out of the database)
• delete the last line 
• remove all blank lines

Don't ask me why but for whatever reason, you always get an extraneous first line and unexpected blank lines (and less often an extraneous last line) no matter how you produce the file :).

Anyways, my tool of choice in the matter is bash - it is just too trivial to use anything else (plus I love the simplicity and power of the shell). So, to make sure I never forget again here is the easiest way of doing all the three things above using sed:

sed 1d input_file | sed '$d' | sed '/^$/d' > output_file

Update: As Evan pointed out in the comments, it would be more efficient to do the following:

sed -e 1d -e ‘$d’ -e ‘/^$/d’ input_file > output_file

This way the file doesn't have to go through multiple pipes.

Of course since we're using bash, there should be numerous ways of doing the above.

You can remove the first line using awk:

awk 'FNR>1'

but I don't know how to remove the last line using awk. Anyone?

You can use head or tail to get rid of the first and last line:

head --lines=-1 input_file | tail --lines=+2

but not to remove blank lines.

You can use grep to remove blank lines

grep -v "^$" input_file

but it would be silly to try and use it to remove the first and last line (possible though).

If you know of an easier way to do the above three things in a one-liner using bash – do share it.

What are some of the most common (but non-trivial enough) things that you find yourself doing with bash when it comes to pre-processing that large data file?

For more tips and opinions on software development, process and people subscribe to skorks.com today.

Image by rachel_thecat

Related posts:

1. Using Bash To Output To Screen And File At The Same Time
2. Bash Shell Awesomeness – Mass Killing Processes (On Ubuntu)
3. Bash Shortcuts For Maximum Productivity

Firstly, let me start by saying that I am not recommending you dress up in black pyjamas and go on a killing spree :). With that out of the way, this idea actually came to me when I was reading chapter 4 of Beautiful Teams (I've mentioned Beautiful Teams before). For whatever reason I must have had ninjas on my mind (as you do :)) and thought that it would be interesting to draw some parallels, since ninjas and managers have a lot more in common than the obvious potential scariness factor.

You see, ninjas are not all about killing. Yes, they may be assassins for hire, but they are, in essence, great enablers. They remove obstacles and through their actions make the impossible, probable. They remain unseen, in the shadows, but what they do has a tremendous impact. This is what every manager should be – an enabler. They should concentrate on removing obstacles to enable you to have the most impact as a professional. When there is an organisational obstacle, that you see no way to overcome, they should be able to use their skills and cunning to make the impossible happen. All this should be done without kicking up too much of a fuss. All you would need to do is watch, slightly bemused, as roadblocks, that were preventing you from doing what you needed to do, disappeared almost by magic. That is true management ninjutsu.

Let's speak plainly. You help your people, but you allow them to shine. You don't seek credit (how useful is an assassin that seeks fame?), you seek to achieve goals. You gain validation through the achievements of your team. What's the main theme here? It is the fact that as a manager, you're actually a servant – not a dictator. You serve your people and try to make their lives better, kinda like what a politician should be :). This in turn will allow your people to have a greater positive impact on the wider community (i.e. the company). Everyone benefits in the long run.

Enough Ninjas

The rest of what I have to say doesn't really fit my metaphor, so I am going to abandon it at this point (since I don't really want to engage in any kind of verbal contortionism :)). It is reasonably easy to distil the qualities that a good manager needs (even though everyone has different ideas here), even I have done it once before in a guest post. The difficult part is HOW do you come to embody those qualities, especially when some of them go directly against your personality?

It really comes down to only one factor, you need to genuinely care about the people who work for you (if you don't give a shit about people you're out of luck). And I don't mean you need to 'say' that you care in a group meeting, you actually have to do it, on an individual basis and then back it up when it counts (i.e. when the person really needs your help). And you have to keep showing that you care – for each person as an individual – this is how you build trust. It is difficult for most people to talk about "touchy-feely" stuff in a one-on-one setting, makes us feel uncomfortable. So, we overcome it by avoiding it. Not a good strategy, all it gets you is a generic corporate atmosphere, it does not engender any loyalty. If you can generate loyalty, through building trust and genuinely caring for your people, then when you ask me to put in some extra effort, I am likely to do it, because I know you care and I know you wouldn't ask frivolously.

A good manager is a good listener, more than that, they're good at reading between the lines. Remember what I said about removing roadblocks? Well, you're lucky if the issues are visible and clear-cut (e.g. "We need more machines"). Often the issues are subtle, two team-members don't get along, people don't feel empowered, personal problems are affecting work etc. You would be the luckiest manager in the world for someone to just come out and tell you that these problems exist. Often your people themselves aren't even aware of the issue, or even when they are they may not be comfortable confiding, or simply can't articulate it. But, through your individual interactions with the people you manage you should be able to deduce what the problems are and it is up to you to then do something about it (yeah you have to be almost a mind-reader – I didn't say I had quick and easy answers :)).

Lastly, a good manager, just like a good developer should know his/her business. It doesn't matter how you fell into your management role, it is up to you to become a professional in what is now your trade. It is more than just annoying when the only qualification a manager has is their age or being in the right place at the right time, it is dangerous. When you as a manager don't try to improve yourself constantly, it paints an unflattering picture, it undermines everything you try to do. Like a fat personal trainer – it is tantamount to hypocrisy.

For more tips and opinions on software development, process and people subscribe to skorks.com today.

Image by Narisa

Related posts:

1. How To Become A Spammer (as a programmer) Regardless Of People Following You On Twitter

From my last two posts about searching (i.e. basic indexing and boolean queries), you might get an impression that writing a search engine is a fairly straight forward process. Infact nothing could be further from the truth. To highlight this I am going to explore one of the issues that I have glossed over previously in a little bit more detail – that of linguistic preprocessing of documents before indexing occurs. This is a massive topic and I have no hope of covering all of it in any kind of detail in just one post, instead I will try and touch on some of the main concepts and issues involved, to give everyone an idea of the scope that even this tiny area of information retrieval can cover.

So, let's assume we have just fetched a document, what exactly do we need to do before we begin indexing?

Determining Document Encoding And Getting A Character Sequence

The first thing we need to be aware of is the fact that all we have is a byte stream. We don't necessarily know what kind of document we have just fetched. Of course if we are building a custom search system where we know the kinds of documents that we'll be dealing with then this is not the case. But, for a complex search engine that needs to deal with multiple document formats, before doing anything else, we need to determine the encoding of the document. We need to do this in order to correctly turn our byte stream into a character stream (some encoding systems use multiple bytes per character e.g. unicode). This may be simple for some documents as they will contain their encoding as metadata, otherwise it is somewhat more complex. I won't go into too much more detail, you can look here for a little bit more info about determining Unicode, do look around if you're interested and don't forget to let me know what you find :).

Once we have determined the encoding, our work is not yet over. Our ultimate aim is to get a character stream, but the document may be an archive (e.g. zip) which we may need to extract in order to work with it's contents. Alternatively a document may be a binary format (e.g. pdf, or MS Word document) where getting the character stream is quite a bit more involved. Even when this is not the case, most documents will contain spurious data which we don't want to deal with (e.g. html or xml tags) and will therefore need to strip from our document. Having done all of that we eventually end up with a character sequence that we can process further. It is also worth noting that, this part of the system will potentially need to be highly extensible in order to cater for different encodings and document formats (including future ones).

Find The Document Unit

Having extracted the relevant characters from the raw document, we now need to determine exactly what constitutes a document for the purposes of our index. In other words we need to find what's known as a document unit – i.e. the chunk of text that will be treated as a distinct document by our indexer. This is not as simple as it might appear. Consider – we have pulled-in a document which contains an e-mail conversation thread, do you treat each e-mail as a separate document, for indexing purposes, or do you just take the whole thread? What if some of the e-mails in the thread have attachments, do you ignore them, or treat each one as a separate document, or make it part of the e-mail that it was attached to? Taking this idea further, let's say you're indexing very large documents like books. Is the whole book one document? If someone does a multi-word query you don't necessarily want the book to come back as a result if one of the words appears at the start and the other at the end – they may be completely unrelated and would not satisfy the query. So, do you index the book with each chapter being a document, how about each paragraph? If your document units are too granular however you potentially miss out on concepts that ARE related and would thereby fail to return relevant results. As you can see it is a precision versus recall trade-off. Depending on your document collection (what you're trying to index), you may already have a clearly delineated document unit, or you may employ heuristic methods to work out the document units based on the content.

Tokenizing The Document

Once we know our document unit, it is time to tokenize. Once again this is deceptively complex. We want to split out document units into parts that will form the terms of our index. In the simplest case we want to split into words on whitespace and punctuation. But, this is only of limited utility. For example if we have words such as "are not" everything is fine, but what about "aren't". The semantic meaning is the same so we potentially want to get the same two words "are not" out of it, but if we simply split on punctuation and whitespace, we'll get "aren t" – not quite as useful. There are countless examples like that one. But wait, there is more, what about names and terms such as C++, where discarding the plus signs will alter the meaning significantly. Then there are special cases such as URLs, dates, IP addresses, etc. that only make sense if taken as a single token. And if that wasn't enough consider terms that are made up of multiple words which lose their meaning (or have different meanings) when split up, for example Baden-Baden, San Francisco, Sydney Opera House etc. There are many other similar considerations that we need to be aware of. The crucial thing to remember is this, whatever tokenizer we apply to our documents during indexing we also need to apply to our queries at search time, otherwise you will get at best unpredictable results – pretty obvious. As you can see, tokenization can be a complicated problem and if you're working with documents in multiple languages, it's an order of magnitude more complicated still, as each language presents it's own set of challenges (if you speak a second language you'll see what I mean, if you don't, consider the fact that some Asian languages are written without spaces :)).

Removing Stop Words (Or Not)

Once we have tokenized successfully, we are almost ready to index, but indexing some of the words adds very little or no value. These are normally called stop words (think words such as: a, an, the etc.) and are often discarded from the index entirely. To find the stop words we can sort our list of terms by frequency and pick the most frequent ones according to their lack of semantic value (this can be done by hand as stop lists are normally quite small). While stop words usually don't add much value and can be discarded safely, this is not always the case. For example, think of the phrase "to be or not to be" where every word is potentially a stop word, it is not the best outcome when the system returns no results for that kind of query. For this reason web search engines don't usually employ stop lists. If a search engine does decide to use a stop list, it must once again be applied not only to the documents during indexing, but also to the query during the search. 

Token Normalization

If you think that we are pretty much ready to start indexing, you would be wrong. The next thing we want to do with our tokens is to normalize them. This means we turn each token we have into its canonical form. For example, as far as we are concerned the terms: co-operation and cooperation are equivalent. So we can say that the canonical form of both of those terms is – cooperation. We want searches for either of the terms (co-operation or cooperation) to return results applicable to both. We can do this in a couple of ways:

• Equivalence classing – this is an implicit process that is based on a set of rules and heuristics. For example we can say that the canonical form of a word with a hyphen is the same word without one and we will therefore create an equivalence class between those words (e.g. co-operation, cooperation -> cooperation). These equivalence classes will be applied to the document terms during indexing as well as to the queries at runtime so that a query that contains one of the words in the equivalence class, will return documents that contains any of the words.
• Synonym lists – this is a more explicit and labour intensive process, but can give more control. Synonym lists are constructed by hand (e.g. think – chair, stool, seat, throne). These synonym lists can then be applied in one of two ways. At indexing time you can index the same document under all the terms that are synonyms of each other, this requires a bit more space, but at query time any of the synonym terms will match the document. You may also apply the synonym lists at runtime whereby you expand the query to contain more terms (i.e. all the synonyms) which means the query takes a bit longer to execute. How you decide to do it is up to you, it is a classic space/time trade-off.

There are other normalization steps that can be applied to our terms, such as case folding, whereby we lower-case all our terms. This will of course need to be applied to the query as well, but can come back to bite us, especially where proper names are concerned. For example Crunchy Nut Cornflakes is a breakfast serial, crunchy nut cornflakes are just three random words. Despite this, case folding is normally used by most search engines.

Stemming And Lemmatization

We've already done a lot of processing on our documents and terms, but there is still more to do. Different words often derive from the same root e.g. differ, different, difference, differing etc. It make sense that a query for one of these words will potentially be satisfied by a different form of it. We can therefore try to equate all of these words for indexing purposes. For this we can use either stemming or lemmatization (which are, I guess, types of normalization but deserve a section of their own in my opinion).

• Stemming basically concerns itself with chopping off suffixes using various heuristics to find the more of less correct base form of the word. There are several well known stemming algorithms. The Lovins algorithm is one of the earliest stemmers and the Porter stemmer is one of the best known. Stemming tends to increase recall while harming precision.
• Lemmatization is a more complex process using morphological analysis and vocabularies to arrive at the base form.

Neither stemming or lemmatization tend to have a lot of benefit when dealing with English language documents (which doesn't mean they are never used :)), however languages with richer morphologies will often benefit to a greater degree.

And there you have it. I hope this has given you an appreciation for how complex even a small area, like preprocessing, can be when it comes to search engines. Of course I am far from an expert, so if you have something to add or want to correct me :), then do feel free to leave a comment. I hope you've been enjoying my posts about searching, since there is more to come. In my previous post on boolean search I asked if anyone could think of a faster way to intersect lists, well in my next search-related post, I will attempt to do just that. Stay tuned.

Images by Shawn Econo and francescomucio

Related posts:

1. Search Fundamentals – Basic Indexing
2. Lets Roll Our Own Boolean Query Search Engine
3. Become A Better Developer By Indexing Your Brain

Ruby on Rails projects usually use MySQL or PostgreSQL for their database, but in the corporate world, Oracle is king. As much as you might like to have a Postgres backend, the powers-that-be have decreed and you must obey. Don't worry though, all is not lost, you don't have to slink back to Java, here is how you can get your Rails app working with an Oracle database. Note: All of this works with Rails 2.3.5, I haven't tried it with Rails 3, but feel free to give it a go and don't forget to let me know how it works out.

Getting Rails To Play With Oracle

Alright, first thing first, I hope you're using RVM, cause it's awesome and will make your life easier. With that out of the way, you need to install two gems, before trying to use Oracle with Rails.

1. ruby-oci8 gem – this is the ruby interface for oracle using the OCI8 API. You need to have a version of Oracle installed on your machine for this gem to work, otherwise the OCI8 library will not be available on your machine and this will make your life difficult (i.e. impossible :)), as you might imagine.
2. activerecord oracle enhanced adapter gem – this is an ActiveRecord adapter that has useful extra methods for working with new and legacy Oracle databases from Rails.

You can potentially vendor the oracle enhanced adapter gem within your rails app, but I wouldn't vendor the ruby-oci8 gem as it has native extensions. Normally that would be fine, but we're going to make a Java application out of this in the end (for deployment to Tomcat) and native extensions will once again make life very difficult (i.e. impossible :)).

So, to install these gems we do:

gem install ruby-oci8
gem install activerecord-oracle_enhanced-adapter

This will usually go smoothly, but I've had one situation where the native extensions failed to build properly when installing ruby-oci8, and I couldn't find anything that should have caused that. To fix the issue, I simply compiled it from source and then re-installed it as a gem again, at which point everything went smoothly. Something to keep in mind.

Rails Configuration

As you would expect, we now need to modify our database.yml to configure our Oracle database. It is reasonably straight forward, but there are a few things to be aware of. By default your database.yml file will have something like this:

development:
  adapter: mysql
  database: rails_development
  username: user
  password: pass

Of course there will also be similar entries for test and production.

Oracle has a slightly different model, we don't have to create different databases, we create different database users, so the username and password are used to distinguish between different Oracle dbs.

Our configuration will now look like this:

development:
  adapter: oracle_enhanced
  database: YOUR_ORACLE_SID
  host: localhost
  username: user
  password: pass
  port: 1522

The differences from the norm are as follows:

• you need to have an adapter entry and the value will always be oracle_enhanced
• the database entry should contain the SID of your oracle installation, this too will always remain the same
• as I mentioned the username and password are not just the credentials but will also identify the Oracle database (as per how Oracle works)
• if you changed the default port (the default is 1521) when installing Oracle you need to specify that too

You will once again need similar entries for test and production to maintain the standard Rails conventions.

Modifying Your Normal Workflow

Of course things are not quite as simple as that (they never are with Oracle :)). Firstly, some of the rake commands you're used to will not work with Oracle, such as:

rake db:create
rake db:drop

You will need to create your Oracle databases by hand, which in "Oracle world" means you will need to create some users. You have to log in to Oracle as an admin user, create another user and give them the right privileges. To log in to Oracle using sqlplus, you can do the following:

sqlplus admin_user/admin_password as sysdba

You can then execute a series of SQL commands similar to those below, in order to create a new user and give them the right privileges:

CREATE USER my_user IDENTIFIED BY my_password DEFAULT TABLESPACE some_tablespace TEMPORARY TABLESPACE temp QUOTA UNLIMITED ON some_tablespace;
CREATE ROLE my_role;
GRANT all privileges TO my_role;
GRANT my_role TO my_user;

The tablespace and quota stuff is optional (I think :)) so feel free to leave it out. If things are still not working after you've configured it all in Rails and tried it out, you can also try:

GRANT all privileges TO my_user;

This should fix most of the issues you might encounter. At this point all the migrate commands should work, such as:

rake db:migrate
rake db:migrate:down

These are the important ones as far as I am concerned, so I am not too worried about loosing some of the others.

One last thing to note, if you don't create a separate development and test user due to laziness or whatever (which is exactly what happened in my case :)), be aware that if you run tests through rake all your dev data will be blown away (Rails blows away the data in the test database every time).

But What If You Need To Deploy To Tomcat?

Once again the corporate overlords make their presence felt :) and you're only allowed to deploy to Tomcat. You can develop everything in JRuby and then roll a WAR file in the end, but why bother when you have Warbler.

Warbler is a gem that will make a WAR file out of your Rails app – pretty convenient. They way it does this is by bundling a copy of JRuby (i.e. you don't need to supply it) and then packaging all the resources in your Rails app into a WAR structure in a sensible fashion. To install Warbler all you need to do is:

gem install warbler

There are a few different things you can do with Warbler, but you really just want to create a WAR file, so all you need to do is go to the root of your Rails app and type:

warble

This will produce a WAR file for you which you can deploy directly to Tomcat – easy. There are however a few things to be aware of, especially in our case.

1. As I mentioned already, make sure you're not using any gems that have native extensions (e.g. hpricot). Warbler will try to bundle all the gems it needs and it will be pretty difficult (i.e. impossible :)) to do the compilation steps from within a container.
2. We're using Oracle, but we will not need the ruby-oci8 gem (which has native extensions), instead we can use the ojdbc14 JAR. All you need to do is place the JAR file in your Rails application's /lib folder and Warbler will take care of the rest.
3. Incidentally, if you need to include any other JAR files also put them in the /lib folder of your Rails app.

That's it, unless you were doing something really fancy, you should be able to produce a WAR file, drop it into Tomcat and have a working Java web application with no extra effort. The good thing about this is that you can develop your application in pure Ruby (i.e. you don't need JRuby), JRuby only comes into the picture when you need to deploy. I would recommend deploying as frequently as you can though, to make sure you don't run into any unexpected surprises.

If you think this unlikely scenario (using Ruby on Rails with Oracle for dev and deploying it all as a Java app to Tomcat) can never happen, think again. This is all based on "real world" events and no animals were harmed in the making of it :). There are a couple of other interesting things we did in the course of that project, such as serving user uploaded images from outside the Rails folder structure with the help of Passenger (during development). I am planning to do a quick post about that at a later date, so if you're interested – stay tuned.

Image by ellen reitman

Related posts:

1. Installing And Using SQLite With Ruby On Windows
2. Digging Into A Ruby Installation, Require vs Load And Other Stuff
3. Fetching RSS Feeds With Ruby From Behind A Proxy

Programmers love to work long hours! There I said it, c'mon admit it, your job/boss doesn't make you do it, we do it to ourselves. Alright, I'll concede, maybe not all programmers love long hours, but surely with the amount of overtime that is prevalent in this industry at least half of us must love it. Right?

I can hear the excuses already. "No, no that's not it, we just love working with cool tech and don't want to leave a problem unsolved. It is actually a good thing it's what makes us awesome!"

I say – you're not seeing the forest for the trees. Here is some perspective, you're not doing this for yourself, you're doing it for "the man". Admittedly he might be a nice man, but you don't owe him slavish commitment. Here is even more perspective, how often are you actually playing with interesting problems and cool tech and how many times are you churning out code desperately trying to get something delivered and meet some arbitrary deadline that someone has assigned to you? But hey, you're a business savvy developer, you're helping the company succeed, your manager has explained the financial situation to you – it has to be done, we're relying on you. Well, unless that same manager is right there with you, entertaining you with amusing anecdotes at 2 am, his words are worthless.

Let me tell you a story that a friend once told me. It is about a brilliant developer – lets call him John.

John was a superstar, a one in a million programmer. He had an uncanny ability to understand and write code and was 20 times more productive than anyone else. One day the company got a big contract that needed a fast turn-around. The client sent a massive spec document – to everyone's dismay. John came to the rescue, he took the spec home and noone heard from him for 3 days. When he came back to work, he looked like hell, but he had gone through the whole spec and had an outline of the solution already finished. Except for one bit which was impossible to implement, though the spec said otherwise – even the client didn't realise this, but John picked it up. Amazing!

When I first heard that story, I was pretty impressed, my first question was, "So, where is this guy now?". To which my friend replied – "He is dead, too much hard living!". Too much hard coding would be more like it. Kinda takes the wind out of that story a little bit – John was in his early 30s.

Programmers take a perverse pleasure from sharing death-march war stories. Even when we do it with disgust, it is a disgust tinged with pride – daring our peers to do "better". But it is a bit like those guys who wear their pants so low you can see ALL of their underwear or the people who take up smoking for the "trendy image". They and their friends think it's cool – everybody else thinks it's stupid.

Making A Bad Situation Worse

I can see the necessity of occasionally putting in some extra effort and burning the midnight oil at work for a day or two. But when "occasionally" turns to "often", when your boss stops thanking you profusely for your efforts and just treats it as norm, this is when we're all in trouble. It sets bad expectations, not just for you, for the whole industry. Humans are like dogs, we're eminently susceptible to positive and negative reinforcement. And this whole industry has been conditioned by years of death-marches to the point where it even rewards this behaviour. Every time we give-in to the long hours argument, we continue to negatively reinforce this trend.

It doesn't help that we're herd animals, you only need to get one person and everyone else wants to conform. Guilt comes into the equation – "we can't leave our mates by themselves to do the hard-yards, we gotta help them". The more people conform, the more pressure on the rest of the herd to do so until the whole team is chugging coke and eating pizza at midnight. But how do they suck even one person in, where is that famed programmer independence. We're happy to "stick it to the man" and do whatever we want in school, but as soon as we're in the workforce all bets are off. It is puzzling.

Interestingly, sometimes these gargantuan efforts aren't even tracked properly, as it would make the project look bad. So they "cook the books", as far as the client is concerned everybody is doing 40 hours a week (i.e. they get billed for 40 hours) and the project is coming in on schedule (maybe), never mind the other 40 hours that everyone on the team puts in. OK, maybe they'll track the real effort in a "second set of books". Accountants go to jail for these kinds of shenanigans, but our industry expects it – nay almost demands it.

The Sustainable Pace Effort

Most Agile processes talk about sustainable development pace. But, I've seen even self-confessed agile teams knuckle under and put in the hours, you know, for the greater good and all. They were still agile though, and don't you dare say otherwise.

When I think about this stuff I am always reminded of lawyers. You come in as a new lawyer and you put in massive amounts of effort and time, it is the accepted way to get ahead in that industry. No developer wants to be compared to lawyers, but often the situation is similar except you're not going to get ahead by doing a lot of overtime as a developer (unless you're working for a big 4 consulting company and then you might as well be a lawyer :)). So, lawyer vs programmer, which one is the chump?

Studies about productivity declines when working more than 40 hours a week surface with disturbing regularity. As a developer your creativity declines, you make more mistakes, you miss existing issue etc., to the point where you're doing more harm than good. Should I even mention the health concerns when you spend that much time engaged in the same activity (they even had rules about spending too much time at work in the Soviet Union, and those guys were all about putting in the time for the good of the people). What about diet, you can only survive on coke for so long – poor John couldn't even make it to 40.

Can you tell that I am against long hours and death marches yet :)? Maybe one of these days I'll tell you how I got my wake-up call, it is an interesting story. Herding cats is easy compared to getting developers to make a concerted effort in the same direction, it is something I both love and hate about our people (programmers) :). But I do wish that once in a while all the smart developers just took a stand to eliminate at least one of the truly crappy and counter-productive trends in our industry. As far as I am concerned, smart programmers don't like to work long hours and won't be pressured into it – there is more to life.

Image by Tattooed JJ

Related posts:

1. It’s Not About The Duct Tape It’s About The People (And More)
2. Are You The Best Developer In The World?
3. Does Software Development Have A Culture Of Rewarding Failure

I am going to give it away right here in the first sentence, it's the ability to ask for and take criticism. Yeah I know technically it's two skills. Before you switch off and browse away to look at funny cats with bad grammar, I don't mean the ability to say:

"Well, of course I can take criticism, I am hip, I am with-it."

It's like asking people if they are good in bed, it's not as if anyone is actually going to say that they are terrible. And yet most developers (most people) can't take criticism well and even fewer know how to ask for criticism and have it be received without uncomfortable silences.

Humans don't tend to be able to get a lot of insight into their own failings, so we need an outside opinion. Feel free to disagree with me here, but I'll tell you this. There are dozens of things I want to learn that I think will improve my skills and make me better at what I do, but it has always been external advice into how, where and what I should tackle that has given me the biggest boost in my growth as a developer. This advice was accidental every single time, just imagine if I could achieve a little bit of repeatability here. I wonder if this is the exact same thought that the dudes who were implementing waterfall for the first time had? Rigid processes aside, taking criticism and using it constructively is especially important for developers. It is basically a matter of survival within our field. There is so much to learn and new stuff comes along at a faster rate than you're able to "master" (when I say master, I mean become aware of) old stuff. We want our efforts to propel us forward in leaps rather than at the speed of a leisurely crawl (although there is something to be said for crawling as well, but I will get to this some other time).

So Why Are People Bad At Taking Criticism?

Well, it doesn't really help that most people are really crap at giving constructive criticism. There is just not a lot of criticism going around, how are you supposed to get good at it if you hardly ever get a chance to practice? Plus, our primal response to criticism is confrontational, either offensive or defensive (depending on personality), goes back to the cave days.

"I've survived right? So how dare you say I am doing something wrong!".

Double plus, developers are worse at this than most, we're ornery, we like to argue, we pick on minute details. By rejecting criticism we can immediately scratch too many itches, it's hard to ignore for the promise of some intangible benefits at a later date.  

What About Asking For Criticism?

Remember how I said that most people are crap at giving constructive criticism? Well, they know it. They also know you're bad at it. We're not particularly comfortable when we're asked to do stuff that we are no good at. So, they don't ask you and you don't ask them, everyone is happy (kind-of). It's a global conspiracy that everybody buys into – easier that way. Sort of similar to how everything is overpriced 10-fold in the enterprise world (including software), but it is easier to try and get a piece of the pie rather than call the whole industry on it. And when they try to make us criticise other people, like for those crazy annual reviews that most companies have (I've got thoughts about them); we do our best to get it over with as quickly as we can and try to use as many words as possible to say as few constructive things as we can get away with.

What To Do

Ha, I don't have all the answers. Just because I observe an issue and see the benefits if it were removed, doesn't mean I know how to remove it :). I am not particularly great at taking or giving criticism, I am a lot better than I used to be, but there is a long way to go. I've already given one tidbit of advice that can get you on your way – practice. The one surefire way to get better at something is to keep doing it. Find some people you trust, and go crazy on each other – hopefully you'll still be friends afterwards :).

Here is some more advice, when asking for criticism, it really helps if people don't know you're doing it. You need some serious communication wu-shu to pull this off, but I have seen people who were awesome at this. Unsurprisingly considering everything I said above, these people were also awesome at a lot of other things.

Last piece of advice – grow a thick skin without becoming deaf to what people are saying about you. It's a sad life when you go around taking everything personally, but at the same time you don't want to dismiss it. There is a saying in Russian "to coil on the moustache" (it seriously doesn't translate well, considering it's metaphoric nature), but it essentially means, you examine a piece of information for relevance and take it under advisement if you find it useful. So, ummm, do that.

Seriously, if people followed some of this advise, we could improve workplace atmosphere across the board. Then maybe people could relax a little bit more at work. Of course there are already places where if people were any more relaxed they'd need business hammocks. I think I am veering off-topic a little, having a short attention span sucks. Somebody really aught to pull me up on that.

Image by lism.

Related posts:

1. Are You The Best Developer In The World?
2. The Best Way To Interview A Developer
3. Become A Better Developer By Indexing Your Brain

I was reading Beautiful Teams the other day. In chapter 2, Scott Berkun talks about “Ugly Teams” and weighs in with an opinion on academic scores and how they tell you nothing about a persons aptitude (of course when I say “person”, in my mind that translates to – “developer” :)). I’ve heard similar opinions so many times now that I felt like I had to champion the other side of the argument. Here are some quotes:

“If we go beneath the superficial, perfect grades often mean the perfect following of someone else's rules”

or

“They (good grades) are not good indicators of passionate, free-thinking, risk-taking minds”

or

“The tragedy of a team of perfect people is that they will all be so desperate to maintain their sense of perfection, their 4.0 in life, that when faced with the pressure of an important project their selfish drives will tear the team apart”

or

“Beautiful people are afraid of scars: they don't have the imagination to see how beautiful scars can be”

Frankly I have to say – that is all so much bullshit. How is it that in the world of software, mediocre grades have come to be equated with creativity and flair and flashes of genius. “Oh, he is an academic underachiever you say – well that must mean he is a creative, entrepreneurial go-getter who thinks outside the box”. You see they just didn’t get the poor guy at university, it is so sad that his genius has been overlooked. What a load of rubbish, why is it that this side of the argument is never vociferously argued by people with perfect GPAs and a record of academic excellence? Sure, poor grades can be an indicator of a creative mind but they can also indicate laziness, inability to concentrate, lack of knowledge and skill, inability to complete tasks, poor interpersonal skills etc. Good grades can sometimes mean slavish obedience (rarely), but are more likely to indicate, superior knowledge, better study skills, adaptability, great teamwork skills, quicker mind etc. It is beyond me how high academics can possibly indicate lack of passion and creativity.

So you want to employ some people, you have some high academic achievers and you have some mediocre ones, who do you hire? Do consider the average Joe, those grade jockeys are just law-abiding drones, Joe is the radical who is going to revolutionise the industry, he just needs the right kind of motivation. What about motivation then? Lack of the right kind of motivation is often cited as the reason for average academic performance. If only our underachieving friends had the “right” kind of motivation, they would be brilliant. Well, it’s not as if the high achievers get their kicks out of boring lectures and worked examples. The difference is that the high achievers are able do well and motivate themselves despite the banality and minutia. So which skill is more valuable, ability to self-motivate no matter what, or passively waiting for the “right” type of motivation? Are you sure you have the “right” kind where you are?

Let’s get this one out of the way, cause it will inevitably come up. Yes, there are plenty of people in the world of software who had average (or worse) grades when they were studying, but have since “made it”. They gained kudos, money, the acceptance of their peers (either or all of those) there is no arguing this point. But that in itself is the problem, these academic underachievers who succeed have the spotlight cast upon them, because people didn’t expect it of them. They get hailed and written about – see, even the average Joe can make it. Thing is – they are the exception, not the rule. These people succeeded “despite the fact” and this is the very reason why they are interesting. People love an underdog story, makes us feel good, nobody wants to hear about the brilliant guy who lived up to his potential and made it big, as per expectations.

Rather than looking at high academic achievers as conformist drones, let me paint another picture for you. Those who get good grades are the ultimate survivors. They enter an environment, evaluate it critically to figure out what skills they need to succeed and thrive in said environment. They then put in the effort to gain and/or sharpen these skills. At this point they can apply their knowledge and skills with precision to get to the top of their environment and stay there. Excellent academic results are just a side-effect. And you know what, it’s not as if being a high achiever is black magic, all it takes is a little bit of work and practice (I am so going to cover that in a later post).

So yeah, good grades make better programmers, hell, good grades make better anything. And there is nothing that stops a high achiever from being a passionate, free-thinking, risk-taker. When faced with the pressure of an important project high achievers won’t get torn apart by their selfish desire for perfection, infact they will be better able to adapt and achieve superior results because that is what they do – evaluate their environment and figure out what is needed for them to succeed. Beautiful people (academically speaking) are not afraid of scars, but they do have the knowledge and skill to avoid the scars when they can and create superior scars when they have to. 

Image by GogDog

Related posts:

1. How To Become A Spammer (as a programmer) Regardless Of People Following You On Twitter
2. Building Software Development Expertise – Using The Dreyfus Model
3. How A Good Manager Is Like A Ninja