Robust Software

Ruby 2 - Module#prepend

2013-04-06T00:00:00+01:00

As you may know Ruby 2.0.0 has been released. Despite the major version it is mostly an incremental release. However, it does include a few breaking changes so a major version is warranted. However, whilst the usefulness of most of the new features was obvious to me, I couldn’t say the same of Module#prepend.

However, whilst listening to the Ruby Rogues podcast on Ruby 2 one of the Rogues (I think it was Josh Susser) referred to memoization as a case where it would make a difference. Because of that I thought I would implement memoization in 1.9.3 and then in 2.0.0 using Module#prepend and see what came out of it. The process led to a personal “ah-ha” moment and so I thought I’d share what I found.

Method dispatch

First, I think it’s worth a little diversion into how method dispatch works in Ruby to understand what Module#prepend allows you to do that you couldn’t do before.

Every object in Ruby has an ancestor chain, you can find out what this is at any point by calling .ancestors on a class:

class Example
end

Example.ancestors
# => [Example, Object, Kernel, BasicObject]

Whenever you call a method in Ruby, the ancestor chain is traversed looking for a matching method to invoke. It follows it from beginning to end until it finds a match.

Modules fit into this story as when you use Module#include they are added to the ancestor chain after the class they were included in:

module After
end

class Example
  include After
end

Example.ancestors
# => [Example, After, Object, Kernel, BasicObject]

What Module#prepend allows us to do is insert a module in-front of the class it was prepended to:

module Before
end

class Example
  prepend Before
end

Example.ancestors
# => [Before, Example, Object, Kernel, BasicObject]

On its own that’s interesting, but what is it good for?

Memoization example

Memoization is a technique whereby you cache the result of a, usually expensive, function so that the result can be returned for subsequent calls without having to calculate it again. Sometimes memoization is built into the implementation of a method, sometimes it is implemented using the interceptor pattern so neither the caller nor the implementation are aware that memoization is happening. We’ll be using the latter type of implementation within our example.

There are already several implementations of this technique in Ruby, including the Memoist gem that was extracted from ActiveSupport::Memoizable that used to be part of Rails. The problem with these gem-ed implementations for our uses is that they tend to be quite hard to read due to them being highly optimized and doing things such as guarding against overwriting methods unexpectedly.

I’ll be creating a simple implementation to demonstrate the concept of memoization and the effect Module#prepend has on an implementation. This code is not intended for production use!

To require memoization, we really need an expensive call to make. We’ll use the same class and method for both versions of Ruby and their memoization implementations:

class Universe

  def meaning_of_life
    sleep 1
    42
  end

end

Calling Universe#meaning_of_life once takes just over 1 second, and calling it 5 times takes just over 5 seconds. This is a prime example of a case where you might want to memoize the result.

For both implementations we will augment the Universe class by memoizing the #meaning_of_life function through the use of a Memoize module and its memoize method which will be implemented differently in each version of Ruby:

class Universe
  extend Memoize

  def meaning_of_life
    sleep 1
    42
  end

  memoize :meaning_of_life

end

This will mean that calling Universe#meaning_of_life once takes just over 1 second, and calling it 5 times will also take just over 1 second too due to the result of the first call being memoized.

Ruby 1.9

The way Memoize is implemented in Ruby 1.9 is to rename the true implementation of the method and replace it with a method that calls the true implementation for the first call, and then returns the result of that first call for all subsequent calls.

This looks something like:

module Memoize

  def memoize(method)
    # Work out what to rename the true implementation to
    unmemoized_name = :"__unmemoized_#{method}"
    # Create an alias to the true implementation for the
    # new name
    alias_method unmemoized_name, method

    # Overwrite the true implementation with a memoizing
    # version
    define_method method do
      # Ensure we have a place to store the result in
      # case we memoize multiple methods
      @__memoized_results ||= {}

      if @__memoized_results.include? method
        # If we've already calculated the result of this
        # function, return it
        @__memoized_results[method]
      else
        # Otherwise calculate the result by calling the
        # original implementation and store it for
        # future calls
        @__memoized_results[method] = send(unmemoized_name)
      end
    end
  end

end

If we change the Universe class’s implementation to look like it will after this bit of meta-programming, it would look like this:

class Universe

  def meaning_of_life
    @__memoized_results ||= {}

    if @__memoized_results.include? :meaning_of_life
      @__memoized_results[:meaning_of_life]
    else
      @__memoized_results[:meaning_of_life] = __unmemoized_meaning_of_life
    end
  end

  def __unmemoized_meaning_of_life
    sleep 1
    42
  end

end

Ruby 2.0

Whilst the same implementation used for 1.9 would work in 2.0, due to Module#prepend we have a second option involving anonymous modules.

Rather than renaming the method, we can create an anoymous module that has a method with the same name. This module can then be prepended to the class so that it comes before the real implementation in the ancestor chain. The method on the module can then call the true implementation of the method via super for the first call, and then return the result of that first call for all subsequent calls.

This looks something like this:

module Memoize

  def memoize(method)
    # Create an anonymous module
    memoizer = Module.new do

      # Define a method in the module with the same name
      define_method method do
        # Ensure we have a place to store the result in
        # case we memoize multiple methods
        @__memoized_results ||= {}

        if @__memoized_results.include? method
          # If we've already calculated the result of
          # this function, return it
          @__memoized_results[method]
        else
          # Otherwise calculate the result by calling
          # the original implementation and store it for
          # future calls
          @__memoized_results[method] = super()
        end
      end

    end

    # Prepend the anonymous module to the class so that
    # its method is called first
    prepend memoizer
  end

end

If we change the implementation of the Universe class to look like it will after this bit of meta-programming, it would look like this:

module AnonymousModule

  def meaning_of_life
    @__memoized_results ||= {}

    if @__memoized_results.include? :meaning_of_life
      @__memoized_results[:meaning_of_life]
    else
      @__memoized_results[:meaning_of_life] = super()
    end
  end

end

class Universe
  prepend AnonymousModule

  def meaning_of_life
    sleep 1
    42
  end

end

Comparison

In terms of lines of code, the two implementations are virtually the same. In fact, their structure is virtually identical (whitespace ignored and comments removed):

--- memo19.rb
+++ memo20.rb
@@ -1,8 +1,7 @@
 module Memoize

   def memoize(method)
-    unmemoized_name = :"__unmemoized_#{method}"
-    alias_method unmemoized_name, method
+    memoizer = Module.new do

     define_method method do
       @__memoized_results ||= {}
@@ -10,9 +9,13 @@
       if @__memoized_results.include? method
         @__memoized_results[method]
       else
-        @__memoized_results[method] = send(unmemoized_name)
+          @__memoized_results[method] = super()
       end
     end
+
+    end
+
+    prepend memoizer
   end

 end

I believe the choice between the two comes down to taste, and I prefer the Module#prepend method. The reason for this is that it feels like the Universe class has been tampered with less.

In the 1.9 implementation it has the method renamed and replaced, whereas in the 2.0 example it has an extra module added. Leveraging the ancestor chain also feels like a cleaner way of achieving the goal of intercepting a call to a method.

Conclusion

I feel Module#prepend is a good addition to the Ruby language. I don’t imagine I’ll use it regularly, nor can I think of previously unsolveable problems that it solves.

However, from working through this memoization example I can see that it does have its uses, and that perhaps we can now solve some problems in cleaner ways.

If you’ve got further thoughts on Module#prepend, or anything else Ruby, you can find me on Twitter or get in touch via email.

Logging

2012-12-30T00:00:00+00:00

If there was one thing that I learnt in 2012 that I would want to convey to all the developers I know, it would be this:

Logging is about so much more than failures

I don’t know if it was just my experience but little to no emphasis was put on logging aside from handling exceptions. In the .NET world this boiled down to adding ELMAH to your project and then forgetting about it, with Rails it meant having nothing but the logs that you got out of the box. I’ve found that if that’s all you have, you’re really missing out.

Benefits of good logging

Bug investigations

I think that every system I’ve ever worked on has been distributed, at least in some form. It may have been as simple as scheduled processes working alongside a web site, or it may have been a truly distributed, message-driven system. Distributed systems can be hard to monitor and bugs can be particularly tricky to track down. The complex interactions dependent upon the state of different components can be a nightmare to reproduce.

Good logging virtually solves this problem. Given sufficient detail in your logs, you no longer have to theorise or piece together how something happened. Your logs will lay it out for you, plain as day.

Here’s a contrived example to demonstrate where better logging can help you:

module Background

  def self.execute(user)
    if user.privileged? or transient_condition?(user)
      something_dangerous(user)
    end
  end

  def self.transient_condition?(user)
    user.first_name.start_with?(random_character)
  end

end

transient_condition? is the most contrived part but imagine you’ve received a bug report a day later and multiple things have changed on their account and the entire system that transient_condition? references since that time. You’ll end up with the same effect.

So, why did something_dangerous get executed for our friend Bob yesterday? Where do you start? Check the database for their current state? How do you know they haven’t updated their profile since? Perhaps you use event sourcing, how easy is it for you to roll back everything in the system that could have had an effect to that point in time? What was that point in time?

You can see where I’m going so I won’t labour the point. It probably will, however, be laborious to investigate this bug report. This is where I would have been a year ago, piecing together transient bits of information and responding with sentences containing “probably”.

Here’s how I would write that same code today:

module Background
  include Hatchet

  def self.execute(user)
    log.info "Entering execute(user:##{user})"

    do_something_dangerous = false

    if user.privileged?
      log.info "##{user} is privileged"
      do_something_dangerous = true
    elsif transient_condition?(user)
      log.info "##{user} satisfied transient condition"
      do_something_dangerous = true
    end

    if do_something_dangerous
      log.info "Doing something dangerous for ##{user}"
      something_dangerous(user)
    else
      log.info "Not doing something dangerous for ##{user}"
    end

    log.info "Exiting execute(user:##{user})"
  end

  def self.transient_condition?(user)
    log.info "Entering transient_condition?(user:##{user})"
    character = random_character
    log.info "Checking if first_name:##{user.first_name} starts with character:#{character}"
    result = user.first_name.start_with?(random_character)
    log.info "Exiting transient_condition?(user:##{user}) - return:#{result}"
    result
  end

end

Yes, this is much longer but it has the exact same logic. We’re capturing the context of each branch of logic so that we may be able to understand what happened after the fact. For me, this is the difference between clean code and production-ready code.

Now let’s go back to our bug report. Why did something_dangerous get executed for Bob yesterday? Let’s find out with a little grep-fu:

$ egrep -A 6 "Entering execute\(user:Bob" yesterday.log

13:53:01 INFO Background - Entering execute(user:Bob)
13:53:01 INFO Background - Entering transient_condition?(user:Bob)
13:53:01 INFO Background - Checking if first_name:Bob starts with character:B
13:53:01 INFO Background - Exiting transient_condition?(user:Bob) - return:true
13:53:01 INFO Background - Bob satisfied transient condition
13:53:01 INFO Background - Doing something dangerous for Bob
13:53:06 INFO Background - Exiting execute(user:Bob)

There’s our answer, Bob satisfied the transient condition. Now you can definitively respond to the bug report with the exact reason why something dangerous happened for Bob yesterday, regardless of the state of the system today. We can also see that doing something dangerous took around 5 seconds.

Which brings me on to a second benefit of logging.

Performance profiling

Particularly when working with bulk processes it can be extremely difficult to build up a test suite that truly reflects production. If you have a database involved you also have the effects of simultaneous queries being executed, the state of indexes, etc. The exact hardware can also have an effect. For all your efforts, sometimes there is no better place to profile code than in your production environment whilst working on real-world data.

However, you don’t want to attach a profiler to your production processes directly. That would mean too much access to a production system, it can often affect how the process executes, and if it’s a nightly task you may have to be up at 4am to watch it. None of those things are desirable.

If you invest in your logs you will not only get the information needed to troubleshoot things happening in your system, you’ll also get a good level of performance profiling too. Most logging frameworks can give you timestamps to millisecond precision, and in my experience that’s more than enough to narrow down which method or methods are taking the most time. This allows you to focus your efforts on where the problem lies in production, and avoids speculative optimisation based upon sterile tests.

Particularly with bulk processes I believe it is hard to log too much. They tend to be extremely reliant on state and can rarely be re-run after the fact, they often run overnight during your system’s quiet window, and they are often left untouched for months as their requirements tend to change much less frequently.

Good logs give you view into the behaviour of all your code. That insight really pays dividends for code that you rarely see and rarely touch.

Understanding unfamiliar code

Good logs give you insight into how a system works, beyond just looking at the code. It helps you understand how frequently a class is used, how frequently a method is called, how often the various branches in a method are hit, etc. Just looking at the code cannot give you that same insight.

Logs can be used to safely prune “probably dead” code or features. If you suspect a branch of logic can be removed, or that a feature is rarely used, you can simply add an additional log statement or raise the level of an existing one:

def perhaps_dead
  log.warn "DEPRECATED perhaps_dead called"
  ## does something
end

You can then review your logs a week or two later to see if any messages have been recorded. If there aren’t any messages you can be more confident in removing it. If there are several messages then you can keep it in place and remove the message without anyone ever being affected.

Good logs replace the need for a debugger, but better than that they provide a historic debugger. You can step through code executed days ago by following the messages that were logged as you look through the code, following the branches taken according to the next message logged. This is invaluable for answering questions like “how many people were affected by this bug?”

Additional context for errors

I’m pretty sure that everyone logs errors. However, they can often be pretty useless apart from telling you what line of code the failure came from. Think of the beloved null reference exception, if you have something like:

if left.id == right.id

If this throws a null reference exception which of left or right was null? Also, if this is 5 steps down a path of execution what are left and right in the first place?

Good logs give you the additional context of the state of the system before the error happened which can be vital in understanding why the error happened.

How to log well

Good logs are all about context. A good message answers the questions:

Where are you?
What state matters?
Why are you there?

The good news is that most decent loggers are class-based and so they deal with the majority of “where” on your behalf, you only have to add the method when it is useful. The difficult parts are the “what” and the “why”.

For the “what”, a trick I use when I am unsure is to imagine I have hit a breakpoint. I then imagine what variables I would be inspecting, are there particular properties of those variables that are interesting at that point in time? The things I would be inspecting with a debugger are the things I record in my log messages.

Much like a bug is a sign of a missing unit test, the need to use a debugger is the sign of a missing log message. Each time you reach for a debugger, ask yourself what log message would avoid you doing so and add it. If you aren’t sure, mindfully use the debugger and notice what you are looking at, then put the message in place.

You can’t attach a debugger to production, but you can read your logs. If the log message is useful to you now, it may well be useful to you in production some day.

Recording the “why” tends to be a bit more straight forward. You want every logical path in your code to result in a log statement, and that statement should record what was satisfied in order for you to hit that path. Our original example demonstrates this well.

I refer to this state of thinking as the maintainer’s mindset. Try and imagine yourself as the maintainer of the code, trying to resolve a production issue, and give yourself the information you would need to shorten the time to resolve it.

How to avoid being overwhelmed by messages

You are probably thinking you’ll start generating gigs and gigs of log files, and you may be right. However, remember that log messages can be written at various levels, and that you can configure your logger to discard particular messages based upon level or even the class they came from.

DEBUG, INFO, WARN, ERROR, and FATAL are pretty ubiquitous across all logging frameworks and they all convey a level of importance.

By default I have production loggers set up to record INFO or higher, meaning that DEBUG messages are discarded. This may make you wonder why you would leave them in, but if they were useful at one time they may be useful again. You can quickly and easily change your production configuration to record DEBUG messages, but you can’t quickly and easily add them across your code base.

Chosing the right level for a message

DEBUG

The finest level of detail. I rarely add a message at this level to begin with. I will usually start with the message at INFO and then downgrade it in a later release if it proves to have a low signal-to-noise level in production.

INFO

Something generally interesting but not something to be worried about. These should give enough information to understand what the system is doing at any point in time.

WARN

Not an error, but something worth highlighting to someone monitoring the logs. For example:

Handling a particularly rare edge case
Hitting deprecated code
Falling over to a secondary provider
Particularly slow response times

ERROR

Generally tied to an error though not always. For example, you might log an ERROR message when you encounter a state that can be handled gracefully but is not reasonable to expect. There was no error as such but the fact it happened deserves the additional attention a higher level demands.

If your logging framework allows, always pass through the error object if there is one. Logging frameworks can usually have their output configured to include the stack trace formatted well without you having to do it in your log message.

FATAL

Reserved for when the underlying process will be halted as a result. To that end I only use these on background processes as web servers should never die completely.

Frameworks

For .NET I’ve never seen a need to look beyond log4net, similarly for Java I’ve used log4j. For Ruby I couldn’t find a framework that did what I wanted so I created Hatchet.

Creating your own framework is rarely a good idea. In particular the parts that deal with writing messages are tough to get right and existing frameworks have been battle hardened. So while I had my reasons for creating Hatchet, the actual logging is performed by the logger in Ruby’s standard library.

Performance and configuration

It’s worth mentioning the performance hit of logging, and configuration come hand-in-hand with that. One of the main concerns I’ve heard for logging as I’ve described is “won’t it slow down the application?” The answer is obviously yes, you can’t do anything for free. However, the benefits I’ve outlined far outweigh the small cost of writing the logs if you get your configuration right.

The biggest mistake you can make in your log configuration is having everything written to a database. Doing so incurs the cost of an insert for every log message you write. In comparison, writing the message to a local disk is an order of magnitude or more faster. The performance penalty of going to the database can quickly add up. For example, I have switched some old SQL-backed logging code to log4net writing to local disk and saw a 10-20% performance increase from that change alone.

Costly calls should be reserved for infrequent events such as errors. Think of SaaS products such as Airbrake, they have the overhead of a network call but it is worth it to get notification of something as significant as an error. At Zopa we have errors sent to our HipChat channel as we feel the cost of the call is justified by the real-time notifications we receive for production issues.

Be aware that no message you log is free and configure the overhead incurred accordingly. 95% of your log messages will be WARN or lower, and that is the case that should be optimised.

Wrapping up

If in doubt log too much and log to a local disk. Having too much information about what your system is doing is significantly better than having too little. Logs can always be filtered, and message levels and log configurations can always be changed.

If you aren’t used to working with log files you will have to learn some new skills to query files rather than a database, but a decent log viewer like Ostros will make most things as easy if not easier than using SQL. Sometimes you can’t beat a bit of grep-fu, but those skills are useful any time you need to work with any kind of text file.

I hope I’ve conveyed the benefits of good logging and given some guidance on the practices to be able to produce good logs for your own code.

If you need any help getting started get in touch on Twitter (@gshutler) or via email.

Minimalism in an Age of Tremendous Hardware

2012-09-19T00:00:00+01:00

Usually – almost always – there’s a much simpler solution waiting to be
discovered, one that doesn’t involve all the architectural noise, convolutions
of the straightforward, and misguided emphasis on hooks and options for all
kinds of tangents which might be useful someday. Discovering that solution may
not be easy, but it is time well spent.
James Hague Minimalism in an Age of Tremendous Hardware

The Boy Scout Rule

2012-09-10T00:00:00+01:00

My friend Dom “Knave of Nottingham” Finn’s wrote a post To Fix Or Not To Fix. This is something I’ve thought about quite a lot as, lets be honest, everyone is maintaining a legacy codebase to some degree.

The Boy Scout (or Girl Guide) Rule is to leave the campsite tidier than you found it. This helps avoid making the problem any worse and the propagation of Broken Windows.

What motivates refactoring?

The question Dom raised is how much is too much and when should you refactor? Firstly, this is being done as part of a change with a business motivation isn’t it? If you are refactoring for the sake of refactoring, without some wider goal, you are wasting time that could be spent adding value. Refactoring should be in service of a change, not the change itself.

You revisit important sections of code frequently. If all you do is make a small improvement each time you alter a section of code, if that really is an important area of the code you’ll be back soon enough and those little improvements will compound over time. Also, each time you revisit the code you’ll know a bit more about the domain and be able to refactor more intelligently. Today’s obvious grand refactor could be tomorrow’s retarded idea. Delay major decisions as long as possible.

Everything you didn’t write is shit

The motivation for refactoring code comes from the fact you think it is shit. For some measure of shit. It might not be using the right pattern for your liking, it might be formatted wrong, it could be written in the wrong language.

That could could be yours that wrote a few months ago leaving a lot to be desired. There are many causes of this, you’ve improved as a programmer in that time, or you may understand the domain better with the experience gained.

As for other people’s code I always try, though it can be hard, to assume they had some deeper insight than I have right now, or at worst they just didn’t know any better than to create the travesty in front of me. The first thing to remember is that people generally do the best they can under whatever pressures they were had at the time.

Gain objectivity over the problem by removing the emotion from the situation. If you’re angry and start a code deleting frenzy you will cause more problems than you solve. Refactoring is a time for a cool, methodical head. Particularly if that code has little or no tests and you don’t fully understand what it is doing. Dump your emotions before you contemplate starting.

How much should you refactor?

As much as you need to get the job done in a sensible manner. Any refactoring you do needs to be kept within the context of the task that led you to that code. Don’t blow your allocated time because you’ve spent 3 hours of a 1 hour task tidying up because the code was “such a mess” without having made the change yet.

For me, there is an initial stage of refactoring where you move the code around, making it abide by coding standard so you can read it, commenting methods as you find out what they do. All this is to help you understand the code and identify where you need to make the change. This shouldn’t be more than 10% of the overall time allocated to the task.

Once you’ve identified where the change is required, consider reverting some or all of the changes you made in order to get there. Have you made the code better or just moved it around? If you’ve just moved it around perhaps revert it, it will make the intent of your real change clearer in the history and you will have changed less, reducing the possibility of introducing new bugs.

At this point you may think that your change will be easier to make if only you refactor this area first. This may be right, but always be sure to keep in mind the scope of the original task. If you have one day to do a task and refactoring for half a day will make it a 5 minute change that could be acceptable. But always keep in mind the risk you are adding by altering what is, I hope, code that works.

Don’t be afraid to throw changes away. Try that refactor and if you’re 30 minutes in and the rabbit hole just seems to keep getting deeper, stop. Get your bearings. Consider that going back to where you were and doing it without the major refactor could be the more pragmatic approach.

All this is made much easier if you commit often, with explanatory commit messages. Being able to step back 30 minutes to after the general tidy but before attempting the grand refactor gives you an easy way to back out.

I’m not you, I don’t know your code base

There are no hard and fast rules, every situation is different and you have to use your own judgement to decide the best course of action. All of this is just how I approach the problem, the steps I go through to try and avoid wasting time making changes that don’t really need to be made.

Every developer loves a good refactorbation session but we’re not paid to pleasure ourselves, we’re paid to use our skills to make our companies money.

Keep that in mind at all times and good luck out there.

Want to continue the conversation? You can get in touch with me by email at garry@robustsoftware.co.uk or on Twitter at @gshutler.

DDD 10 - 10 Practices

2012-09-08T00:00:00+01:00

Among some of the more interesting feedback were some requests for some links and an overview of the things mentioned in my talk.

A few people mentioned they felt misled by the title and synopsis of the talk. I read it back and I think it’s accurate, but then I would. If you are one of those people can you email me at garry@robustsoftware.co.uk with suggestions on how I should change it. It would be greatly appreciated.

Slides

I’ve put them up on Slideshare though without the context of the talk they might not be much use. I used the Bangers font from Google web fonts and the tech light palette from colourlovers.

Talking points

Standards

I made the point that standards are important for putting the focus on the logic of the code rather than its layout. However, not to worry too much about what the standards are, so long as they are consistent.

If I have to put standards into place I will generally point to the canonical standard for that language such as the C# Coding Conventions on MSDN though if there is a tool that can automate compliance such as StyleCop I will favour that as it settles all arguments.

Code reviews

I believe peer code review to be one of the most effective ways of sharing knowledge and improving the quality of a code base. Pairing comes close but I see that as more beneficial in that it gets two heads together to solve a problem. Code reviews have the benefit of the reviewer coming in cold with little context. This helps them identify inconsistencies a pair will also think of as obvious and flaws that a pair will have helped create.

I’ve used pieces of paper, email, and bug trackers for code reviews but I would recommend the ability to annotate the code inline that you get with Github or the open source Review Board. The additional context to the comments given make them much more effective and removes ambiguity over what is being discussed.

Business analysis

Ah, the difficult practice of listening and understand something completely foreign to you. The art of knowing what question to ask and when. The practice is made much more apparent when you are talking to someone non-technical but should extend to how you discuss topics with your technical peers.

I’ve never found a shortcut for this one and no resources are coming to mind. I think this is one of those skills that takes time, practice, and of course failure, to get better at. You can observe someone who is good at it and that will give you pointers but putting it in to practice is a different kettle of fish.

Learn constantly

No-one else cares about your career and improvement as much as you do. Often companies will support you in your efforts by buying books and so on but you need to be the one pushing for it.

It doesn’t take masses of effort, a couple of hours a week will get you through most technical books in a month or two. However, be sure to maximise your return on investment. By that I mean don’t read books where you’ll know 90% of the content. Pick a book on a new subject, particularly one that will give you tools you can use every day over one about development in the language you use each day.

Books that fall into this category for me:

Patterns of Enterprise Application Architecture
- a more practical, hands-on version of the Gang Of Four
Peopleware
- a great primer on managing people and teams
Driving Technical Change
- learn how to put your ideas into action more effectively

Learn new languages

If you want to invest the time in learning a new language, again maximise your return by learning a different style of language. You won’t learn much from Java that you don’t know from C#. Instead pick a different class of language: a dynamic one such as Python, Ruby, or Javascript, a functional one like F#, or Haskell, or something in between like Scala or Erlang.

You’ll find you’ll become a better better programmer in your original language as it will give you a different perspective on problems. Hell, you could hit a problem and say “Erlang is the perfect language to solve this” and end up writing an application in that language.

Testing

Testing is all about increasing confidence in a codebase. The difference between believing something works and knowing is massive. Automated testing is about reducing feedback loops from the point of code being written to knowing it works.

The smaller the feedback loop, the faster you can go as the easier problems become to solve. If you’ve changed one line of code since you last knew it worked you can be pretty sure that one line of code is where the problem is.

If it’s been two months since you threw that code over the wall to the test team you’re going to have a hard time knowing where the problem is and you won’t have the knowledge required at the very front of your mind any more.

Automation

Once you have automated tests it seems a shame to not have them run regularly one check-in. It then seems a shame to not automate the deployment of that code to a staging environment. Once you’re there it should be pretty easy to deploy to production so you may as well do that as well. This again reduces the feedback loop from developer machine to production, generally reducing the amount that is released at once helping your narrow down the problem space when issues arise.

The question isn’t “why should I automate that?” it’s “why shouldn’t I automate that?”.

Know when to move

Sometimes it’s not possible for you to achieve all that you want at your current company. It’s a shame but you need to recognise it and make a decision whether you need to move company to avoid the glass ceiling you’re under. However, have a conversation with your current company first to see if they can help you achieve what you want. It’s always worth asking.

Know what you want

If you decide to move companies, make sure you know what you are looking for. Interview your interviewers as to what they can offer you, the commitment of a new job is two-way after all. Push them on the points that really matter to you. If you’ve ever interviewed anyone before and found their CV is an exaggeration of the truth, don’t think a company’s job description is any different.

A new job is a commitment for a year or more of your life, do whatever is needed to find out whether you’re committing yourself to the right company.

New jobs are scary

I made the point that joining a new company is always a bit of a scary time. Will you fit in? Will you find out you’re a fraud? Was everything you were told in your interview true? Will I achieve what I want?

Rather than trying to ignore it, embrace the fear and channel it into positive action.

Leverage experience

Particularly when joining a new company leave your assumptions at the door. Just because something was accepted as the right way at your previous company or companies doesn’t mean it has or will be accepted here. Similarly what you think of as common knowledge may be completely foreign.

Introduce useful tools and practices. Send around links that “everyone must have read already”. At worst you’re going to end up with strong agreement, at best you can make a real positive difference to the company.

Prove yourself

The first thing you need to prove yourself to is yourself. That will give you self-confidence which will help you win everyone else over. Focus on yourself and everything else will sort itself out.

Be assertive

If you have an opinion and the forum is right for it, make it known. Just be careful to not go overboard and shout down any conflicting opinions. Remember the skill of listening.

Document everything

When developing code you have this mass of knowledge in your head about how the code you are writing works. Document that knowledge before you forget it. The next time you return to the code you’ll read that documentation and you’ll be able to load the knowledge required to make the change much quicker.

The only argument I hear against this is that stale documentation is worse than no documentation. That’s right but it’s the same as badly maintained tests being less than useful. Should I worry that test is failing or is it just out of date?

Add reviews of the documentation to your code reviews. If the code changed did the documentation change? Does the documentation still reflect the implementation? Again, the act of someone coming in cold to review the code helps here. If they read the documentation first it is much easier for them to pick out inconsistencies than the person who made the change.

Preconditions

Preconditions are the simplest form of code contracts, but to me the most effective. They can be easily implemented in any language and enforce the concept of fail-fast. The deeper you go into a code-base the more they help eliminate code as you can rely on assumptions that must have been verified further up the stack.

Without such enforcement you often end up with large amounts of “null checking” code at the very point you want the most clarity.

It also lends itself to more useful error messages. Your preconditions should always come with descriptive messages so that when they are triggered in development they can be resolved as quickly as possible.

Logging

Logging in its various forms is the only way you can peek into the guts of a production system. You can’t just attach a debugger or write a unit test on a live production system. All you have are your logs.

The habit use it to avoid using the debugger in development if at all possible. The urge to hit the debugger to check the state of a variable is a smell that I’m missing a log statement. It might be that it’s a debug one that will be disabled in production but if it’s there you can enable it if needed.

Also, as systems become more distributed, aggregating your logs through the use of tools such as Logstash is a great way to get a “whole world” view. If you aren’t logging then this isn’t possible.

Experiment

If you’re introducing or doing something for the third time, try something a little different. The first time would have been a learning experience, the second would have proved the first time wasn’t a fluke. The third time you should experiment a little to see if you can find a way that’s even better. At worst it will be no better and you can easily fall back to what you know, but you could discover something that’s massively better.

Stages of learning

I mentioned the Dreyfus model of skill acquisition and my preferred moniker of Shuhari.

Teaching other people is fun but you’ve got to let them stub their toes a little so they experience at least a little of the pain that make your best practices understandable. The journey is as important as the destination when learning new skills.

Goals

If there was one thing that I would highlight over all others that has made me the developer I am today it would be goals. Ones I’ve set myself, not those given to me in my annual company review that are forgotten until two weeks before the next one. Having these goals helped me focus my personal development and make the day-to-day decision that compound upon one another.

Wrapping up

Those were the points covered in my talk. There’s about 10 practices and 9 opinions in there.

A few people mentioned my nerves and yes, I am well aware of them but I thought I got them under control after the first 5 minutes or so. This was around my fifth talk ever and I reckon I doubled the number of people I’d ever spoke in front of in that one hour. If you thought I was nervous in that one you should have seen the other four. It’s a scary experience but it’s also something I want to be good at so I’m going to keep doing it.

All feedback gratefully received. You can get in touch with me by email at garry@robustsoftware.co.uk or on Twitter at @gshutler.

Versioning APIs Sucks

2012-07-13T00:00:00+01:00

Also, versioning APIs sucks. It’s not that it’s hard, it’s that once you publish
an API, you pretty much have to support it forever. This is especially true when
your API is being consumed through multiple layers. I can’t force game
developers to upgrade, because they can’t force their users to upgrade. The
lesson here is simple, if you don’t have to make something public, don’t!
Karl Seguin What I Learned Building Mogade

Software and Schrödinger's Cat

2012-02-05T00:00:00+00:00

An interesting article exploring the relationship between quantum physics and continuous delivery.

None of it is real until it is in the hands of actual users. I don’t mean
someone who will poke at it a bit or evaluate it. And I don’t mean a proxy who
will tell you if the users might like it. I mean someone who will use it for its
intended purpose as part of their normal routine. The experience those users
report is reality. Everything else is speculation.
Elisabeth Hendrickson What Software Has in Common With Schrödinger’s Cat

Gain Trust and Create Change - LDNUG - Followup

2012-02-05T00:00:00+00:00

Last Monday I gave my talk “Gain Trust and Create Change” for the first time at the London .NET user group. The guys at Skills Matter recorded the whole thing so you can watch it online if you missed it.

I am reasonably pleased with how the presentation went. Watching it back was an uncomfortable experience but is very useful to feed into the next time I give this talk. Speaking of which, if you run a user group and would like me to give this talk get in touch.

For those of you interested in buying the two books I mentioned during the talk, they are The Passionate Programmer and Driving Technical Change, both from The Pragmatic Programmers.

I’d like to thank everyone who turned up, Ian Cooper for giving me the opportunity and Skills Matter for hosting.

Heroku in Europe

2012-01-26T00:00:00+00:00

I tweeted yesterday (25th Jan 2012) to ask if anyone knew of a solution as convenient as Heroku but based in the UK or Europe.

The reason I asked was at Zopa we are thinking of migrating our front-end over to a Ruby stack and Heroku was the obvious option for hosting such a solution. However, as 95% or more of our traffic comes from the UK it doesn’t make complete sense to host our website outside of the UK or Europe at worst.

I got a lot of interest from Twitter as to whether I found anything but no suggestions of similar services based this side of the pond. At the same time I sent an email to Heroku’s support asking if they had any plans to give an UK or European hosting option.

Heroku got back to me yesterday evening and said they have no immediate plans for server resources outside the US but they are working on Safe Harbor certification if your concerns are of a more legal nature. They also confirmed that they are hosted within the US East region of AWS which is better than West for Europeans.

Their suggested strategy is to use a CDN alongside Heroku so that the majority of your assets will come from a local source and only your dynamic content will come from the US.

This probably rules out Heroku for us at Zopa as a CDN adds needless complexity for us given our traffic volumes and the fact that almost all of our traffic is from the same location. However, I thought I should knock a blog post together to share what I found.

DDD North - Introduction to Backbone.js

2011-10-09T00:00:00+01:00

Yesterday, I gave a presentation on Backbone.js at the inaugural DDD North. My thanks go out to Andrew Westgarth and his team for organising it, it was a great event.

I was a bit nervous as it was my first time speaking at a conference and I think it showed. I rattled through my presentation at break-neck speed, unfortunately finishing well under my allocated hour. Niall Merrigan gave me some tips for sorting that out so I’ll hopefully be more composed next time!

I’d like to thank everyone who showed up for my talk and look forward to receiving their feedback. The slides for my presentation are now up on Slideshare and the code I demonstrated is available on Github.

Finagle

2011-08-25T00:00:00+01:00

I came across Finagle a few weeks ago, it’s:

… a protocol-agnostic, asynchronous RPC system for the JVM that makes it easy to build robust clients and servers in Java, Scala, or any JVM-hosted language.

And they’ve now written a blog post announcing it to the world. I had a skim over it before but it looks like they’ve added to the documentation since then.

I’ve been looking for a framework for writing servers in Scala, the Play framework is on my radar for web applications, and this looks like a great contender.

LMAX Architecture - Martin Fowler

2011-07-13T00:00:00+01:00

I remember watching one of the original presentations on this architecture a few months ago (no idea why I didn’t link to it at the time) and Martin’s article puts it into a format that’s easier to consume at your leisure.

A really interesting approach!

The most important code isn't code

2011-06-08T00:00:00+01:00

I cannot stress how much I agree with this. I’ve come to exactly the same conclusions as Zach in the last year myself. I wish I had written this post myself!

Here are my highlights:

Documentation is the best way to communicate your thoughts to yourself.

Forcing myself first to consider the API, the interface, and the end result led to a clarity that inspired less code and a more impactful project.

It improves the clarity of my code. I get more done in less lines.

But documenting your code is cool as hell. More importantly, it makes me feel more confident about my own code.

Go read the whole thing now!

How to keep your software awesome

2011-06-02T00:00:00+01:00

There’s a whole school of thought that quantity of features is directly
proportional to what you can charge for software. While clearly this is true in
practice, that doesn’t mean that it’s not incredibly stupid. Every new feature
makes your software more complex to use.
Jesse Emery How to Keep Your Software Awesome

Tags are magic!

2011-02-09T00:00:00+00:00

The Guardian produced a interesting series of blog posts on how they use tags to categorise their content, display related articles and so forth:

Pragmatic web service design

2011-02-07T00:00:00+00:00

Web services are a crucial part of most solutions nowadays, I spend a significant portion of my time designing and writing them and I have read a lot about them to make them better, faster and more resilient each time. This is a summary of how I approach web service design and the things I bear in mind.

Protocols and content types

Unless you require extreme performance from your service then use the most compatible technologies available. Today that means HTTP, JSON and HTML forms. The lowest common denominator in any solution is usually Javascript in the browser. This shapes all your decisions about how to expose your service. HTTP, JSON and HTML forms are the easiest things to work with in Javascript and they are well supported in other languages. XML is an option but JSON is a more efficient transport medium and much easier to work with in Javascript.

Before you write a web service make sure to learn HTTP inside and out. It is a powerful protocol that solves many more problems than most people realise. I would recommend RESTful web services as a starting point, it demonstrates how to create a web service that is sympathetic to HTTP and there is a useful glossary in the back. This is not going to be another post about REST but if you know about it already there will be some familiar concepts.

Using the correct HTTP code is important, think of it as a well established domain-specific language for what the server thought of your request. It removes the need to duplicate something like a response code within the body of the response. A caveat on exploiting all that HTTP gives you is to avoid 300 errors as browsers will redirect the whole page even when it is a response to an AJAX request.

For sending and receiving data I recommend HTML form values in, JSON out. You may want to send requests in as JSON instead and that’s not a bad choice, I just find HTML forms easier. However, I would recommend using the same input and output content type for requests across your whole API whenever possible. It makes it easier to consume an API when you do not have to think what format a given method accepts and responds with.

Avoid supporting several content types, particularly for the first couple of releases. You will end up iterating over your service during initial development and having to maintain compatibility for several content types will be adding wasteful overhead at this point. Get it shipped with one pair of content types then look to fix your bugs, harden your API and learn from real usage patterns. Put this into the next version and repeat. Once things stabilise you can evaluate whether support for several content types is needed based upon real world requests without it being such a burden as your API goes through the churn of first contact with the real world.

Core design considerations

A lot of the considerations are identical to those of designing any API but there are some additional ones that are specific to web services as they involve transmission over a network.

Log everything

It should be obvious but you need to log every request that comes in. You may need to replicate an issue that is a result of several requests, without logging this will be difficult. You want to easily answer the question “what was the consumer doing at the time?”. Logs will also allow you to monitor usage, response times and other useful metrics. As your first attempt at an API is no more than an informed guess you need to be collecting metrics in order to make an educated decision about what to do next.

Comply with the expected behaviour of HTTP

There are several expected behaviours with HTTP such as GET not producing side effects and PUT and DELETE requests being idempotent. This all comes from knowing HTTP as previously encouraged. Being able to justify a design decision by referring to RFCs is awesome.

Less methods returning more data

The biggest bottleneck in communicating with a server is no longer the size of the data being communicated, it is in establishing connections. This is particularly true for internal services. The common bottleneck with web services is the number of concurrent requests they can serve, not the amount of data being transferred. You want to aim to have less methods but return more data from them, therefore reducing the number of requests consumers need to make. Having less methods also makes it easier for the consumer to make the right choice.

Imagine everything a consumer is likely to want to display as a result of making the request and give it to them. The only thing you need to be careful of is your internal implementation bleeding into your API. You do not want to expose yourself in public, especially early on. Following this advice makes it likely the consumer will get all the data they need in one request rather than making a separate request for each part. This reduces the surface area of your API which has several great benefits:

as a developer I have less methods to maintain
as a consumer I have a lower cognitive load, I might even be able to memorise your API
as an administrator less requests helps me with caching and usually means I can use less hardware, improving service and making scaling up cheaper

In terms of web service surface area and required requests, less is definitely more.

This may seem to contradict my recommendation of using JSON but choosing a more efficient protocol when there is no compelling reason not to is foolish.

Highlander principle

There should be one, and only one, way to do something. Not only does it save you effort, it makes things easier for the consumer. Sometimes this may mean that one action requires several requests but this is less confusing in the long run than creating a specific method for every action. It is generally acceptable that the solution is good enough, performing several simple steps is cognitively easier than wading through a sea of methods to find the one intended for your task.

Again I may seem to be contradicting myself as this goes againsts the idea of trying to reduce the number of connections required to do something and it does to some extent. However, when you consider most systems are at least 80% reads, that this situation usually applies to writes and that this should not be a regularly occurring problem if you have modelled the domain correctly then it should be a drop in the ocean of the general usage of the API.

Any method you add on a hunch as to what future use will be you will have to support for the lifetime of the API. It is better to wait until you have real statistics and use-cases to work from than to increase the surface area of your API speculatively. If an action requiring several requests becomes common practice then you can add a method to simplify it and you will be certain that it is adding value to your code base and consumers.

Give people URLs

Whenever possible provide URLs to the consumer, do not make them work them out. Every URL a consumer has to create is a support call waiting to happen. When someone has to create a URL it is likely your internals are bleeding into your API. Once someone else is creating a URL you can never change that URL and that restricts refactoring and scaling options. When you create URLs for your consumers you can rename them and point them towards different servers to name just two options that would be closed to you the second you are not in control of your URLs.

Version from the outset

There will be several versions of your web service. Think about identifying the schema of your data when it is returned, think about how you will host several versions of the API as you cannot just switch one on and the previous version off. It will happen and it will be difficult to lever in at a later date, make it a solved problem.

Example - Twitter’s feed

Here’s the response for a single tweet from my timeline:

[ { ...,
    "entities" : { "hashtags" : [  ],
        "urls" : [  ],
        "user_mentions" : [ { "id" : 1102,
              "id_str" : "1102",
              "indices" : [ 3,
                  10
                ],
              "name" : "David Ulevitch",
              "screen_name" : "davidu"
            } ]
      },
    "favorited" : false,
    "geo" : null,
    "id" : 34612066580955136,
    "id_str" : "34612066580955136",
    "in_reply_to_screen_name" : null,
    "in_reply_to_status_id" : null,
    "in_reply_to_status_id_str" : null,
    "in_reply_to_user_id" : null,
    "in_reply_to_user_id_str" : null,
    "place" : null,
    "retweet_count" : 10,
    "retweeted" : false,
    "retweeted_status" : {
        ...,
        "id" : 34583191385935872,
        "id_str" : "34583191385935872",
        ...,
        "retweet_count" : 10,
        ...,
        "text" : "The last 5% of a project is always the worst half of a project. :-)",
        "truncated" : false,
        "user" : { "contributors_enabled" : false,
            "created_at" : "Sun Jul 16 02:30:23 +0000 2006",
            "description" : "Positively disruptive.  Started OpenDNS, ...",
            ...,
            "id" : 1102,
            "id_str" : "1102",
            ...,
            "profile_background_color" : "9BE5E9",
            "profile_background_image_url" : "http://a3.twimg.com/profile...jpg",
            ...
          }
      },
    "source" : "web",
    "text" : "RT @davidu: The last 5% of a project is always the worst half of a project. :-)",
    "truncated" : false,
    "user" : { "contributors_enabled" : false,
        "created_at" : "Thu May 29 20:25:50 +0000 2008",
        "description" : "Freelance software developer fond of Linux, ...",
        ...
      }
  } ]

As you can see this contains everything about the tweet, including which tweet was being retweeted, who they were, what their profile preferences are, where their profile image is, everything you could want really. As a consumer I don’t need to make at least two additional request to retrieve the details for the involed users which would lead to a N+1 load on the server.

The only thing that sticks out to me as possibly being bad is the id values, that smells of internals details leaking out. Instead they might use user_url and instead give a URL for the user’s full profile and so forth. There also doesn’t appear to be any mention of a version for the message but they may handle versioning by using different URLs for different versions of the API.

Things to bear in mind during implementation

Do not reinvent the wheel

For example, methods to achieve caching already exist which utilise the caching mechanisms built into HTTP itself. Use these when possible rather than reimplementing it yourself. Squid and Varnish are two open source software based solutions that are easy to set up. Learn about the wide array of established HTTP headers available, it is rare that you need to create a custom header.

Comprehensive documentation

Yes, even for internal use. If your API is not documented it will be hard to use and under utilised. Without documentation you will spend a lot of time explaining how you go about using your API when you could tell a HTML file once and have that explain it to everyone.

Use examples of performing a common task with your API on top of an example for each method. Ideally the reader will be able to run these examples as they read. Ask a designer to throw a template together for you, it doesn’t need to be amazing but it will make reading your documentation a more pleasurable experience.

No breaking changes, ever

Once you’ve published it and it is exposed to the world, you cannot change anything, ever. Not even fix bugs as people will have written workarounds for them. You can add methods or return more data in response but you must never alter what has been published before. People will be relying on it and you will fuck them up.

TL;DR

Writing an API is very hard
Really learn HTTP
Remove choices and complexity by using less content types and exposing less methods
Reduce the quantity of requests by returning more data in responses
Give URLs to the consumer, don’t let them create them
Documentation, documentation, documentation

Cross-site XmlHttpRequest with CORS

2011-01-05T00:00:00+00:00

I was working on a personal project over the Christmas period and wanted to make cross-site requests from an AJAX client to an API.

As the client and the API were hosted on different domains they violated the same origin policy implemented by almost all browsers. Modern browsers allow cross-site AJAX requests if the API allows cross-origin resource sharing (CORS).

The resources I found at the time on this were a little bare on details but I managed to piece it together and get something working. I was planning on writing a blog post about how to do it but now I know all the terminology I came across this complete article that explains CORS very well, linking to several useful resources.

Why Rack should matter to .NET developers

2010-12-07T00:00:00+00:00

There’s been a lot of talk in the .NET community about Sinatra clones, namely Nancy and Nina in recent times but there are several others as well. Unfortunately, all these frameworks are missing the underlying reason that Sinatra is awesome and that is Rack. Sinatra packs a hell of a lot of functionality yet weighs in at 1198 lines of code including documentation. The main reason for that is the fact it is building on top of Rack which has allowed the authors to concentrate on creating a framework without having to worry about the details of dealing with server requests. So what is Rack, why you should care and why does it matter to you as a .NET developer.

What is Rack?

Rack is a layer of abstraction which sits between servers and frameworks. What it brings as a result is greater interoperability between HTTP server implementations and web frameworks. As the author of a HTTP server if you get your server to speak Rack you can now host applications written in several Ruby frameworks, Sinatra and Rails being two of the most well known. As a web framework author if you get your framework to speak Rack your application can now be hosted on several types of HTTP server.

This greatly reduces the barrier to adoption for both servers and frameworks. Rather than the previous method of having to write an adapter for each server or framework you wished to be compatible with; you are now able to write a single adapter for Rack and be hosted by several types of server or host multiple frameworks depending on which side of the boundary you are sitting.

Needless to say, the Ruby community has jumped all over this. It removes the duplication of effort that was previously required to get the same level of interoperability allowing the Ruby community to focus on innovation. For example, Sinatra did not exist before Rack. Rack didn’t make Sinatra possible but it made it much easier to create. Dealing with HTTP servers had been done for them. They could instead focus on turning requests into responses in an elegant manner.

Why should you care?

First, lets talk about testing. Rack is entirely Ruby based at its core. That means that when you are testing your application you can mimic a real HTTP request being invoked and see what response your application turns that into. Let me show you an example of that being done using RSpec against a simple Sinatra application:

get '/hello/:person' do |person|
  "Hello, #{person}"
end

We can test how that application will respond to a HTTP request using RSpec:

describe Server, 'making a hello request' do
  include Rack::Test::Methods

  let(:app) {subject}

  describe 'saying hello to world' do
    before do
      get '/hello/world'
    end

    it 'should respond "Hello, world"' do
      last_response.body.should == 'Hello, world'
    end
  end
end

I have commented the full source for these examples up on Github if you want further explanation of what is happening here.

We’ve just written an integration test for our application without having to deploy to a server or create our own abstraction of HTTP. This same type of test will work for any application written using a Rack-based framework. Just think of how painful the same thing is in .NET currently. Not only does Rack give us greater interoperability between servers and frameworks, it gives us a way to test them easily as well.

Rack also has the concept of middleware. These are modules built on top of Rack that can then be utilised by any application based upon Rack. This again reduces duplication of effort within the community. Things that can be implemented in the Rack layer, for example caching, are and that removes the need for functional clones for each Rack-based framework that wants the same functionality.

Why it matters to you as a .NET developer

If you’re thinking “well I don’t do integration tests, write HTTP servers or web frameworks, why should I care?” I’ll tell you why you should. For the frameworks you use, a .NET equivalent to Rack would eliminate a large amount of tedious and brittle code that adds no direct value to a framework. This would allow the developers of frameworks to focus their time on refinement giving you more powerful and stable tools to work with. You may not use it yourself but you will benefit from its existence within .NET.

A .NET equivalent to Rack would open the door for a whole new generation of web frameworks to be written. The barrier for writing a quality web framework would be lowered as the nuts and bolts of request and response handling would be a solved problem. We could see some real innovation in .NET rather than a mass of clones of frameworks from other languages.

The door would also be opened for existing servers, such as Apache, to be used for hosting .NET applications and enable entirely new servers to be created. IIS would have competitors. This alone makes it worthwhile. It would make the choice of .NET as a platform less tied, if at all, to choosing Microsoft as a vendor. Even more so than Mono does today, particularly in the web space.

There is a discussion going on right now about creating or choosing a single, common abstraction around HTTP, like Rack, for .NET. Get involved in this discussion and contribute some of your time. This is an opportunity to vastly improve the .NET ecosystem. We cannot afford to miss it.

Use integration tests when working in a new language

2010-11-19T00:00:00+00:00

This is in some respects a follow up to my previous post in that it has been triggered by our internal project in Ruby. When working in Ruby I consciously lean towards integration tests. Sinatra and Sequel make this really easy as using rack-test and running against an in-memory SQLite database it is easy to simulate a full conversation of HTTP requests and they run reasonably quickly too.

How does this relate to learning a new language? When writing anything in a new language you will make mistakes. You are likely to structure your application poorly. At a minimum you will find there is a much better way of achieving the same result. If you have based your work on unit tests you will have a straight-jacket tying your to your code making the effort to heavily refactor it much greater than it needs to be. Tests will need to be moved, rewritten, thrown away and renamed. However, when you use integration tests you are only verifying the behaviour of the whole system not its moving parts. This means you can replace the internals of your application entirely and your integration tests will not need to be changed and will ensure you have a system that still behaves the same as it did before.

Am I saying that you should never use unit tests in a new language? Of course not. However, you should chose the right level of test to match your level of understanding. Once your application’s structure has ossified it makes sense to exercise the complex parts directly through unit tests. The second you make that shift to using using tests you are setting your structure in stone so you want to make sure that does not happen before it has to.

Simple tools and fundamental principles

2010-11-18T00:00:00+00:00

We are doing an experiment in using Ruby for an internal project at work. I am the most experienced in Ruby so I’m leading the choices of gems and so forth and the team questioning those choices has lead to a bit of a personal epiphany.

I know that for a long while I have preferred simple tools and components. Simple tools tend to be more enabling as they give you control over precisely what is happening in your application whilst managing the boilerplate code for you. I’ve also focused my personal development and reading on fundamental principles to give me a solid grounding in concepts that can then be applied to any situation or language.

These opinions are being reflected in the tools I use for my Ruby development and therefore what I am encouraging the other developers at my company to use: Sinatra and Sequel. I enjoy the power you get from both to exploit all that is available from HTTP and SQL. I find it much easier to apply my knowledge of HTTP gained through reading about REST, though I doubt I’ll ever write a truly RESTful application. Things such as returning the correct HTTP codes and headers to enable client and proxy caching are ridiculously easy to do when compared to ASP.NET MVC. Being able to leverage my knowledge of SQL allows me to fine tune that vital query yet Sequel will still give me back easily consumable objects.

Choosing these simple tools for our task will help the team to learn Ruby and build an application, the real point of the task, rather than spending a large period of time learning how to use a more powerful framework such as Rails. The simpler tools should also let them exercise more of their existing knowledge, again allowing them to focus on learning the Ruby language rather than being distracted by Ruby frameworks.

As for me, I’m happy I’ve chosen this method of working and learning. I think it will stand me in good stead for working in any language I may want or need to learn in the future. It will be interesting to hear what the team thinks of this approach as we proceed.