Robust Software

What Software Has in Common with Schrödinger’s Cat

Sun, 05 Feb 2012 19:50:00 +0000

What Software Has in Common with Schrödinger’s Cat:

An interesting article exploring the relationship between quantum physics and continuous delivery.

None of it is real until it is in the hands of actual users. I don’t mean someone who will poke at it a bit or evaluate it. And I don’t mean a proxy who will tell you if the users might like it. I mean someone who will use it for its intended purpose as part of their normal routine. The experience those users report is reality. Everything else is speculation.

Gain Trust and Create Change - LDNUG - Followup

Sun, 05 Feb 2012 13:43:25 +0000

Last Monday I gave my talk “Gain Trust and Create Change” for the first time at the London .NET user group. The guys at Skills Matter recorded the whole thing so you can watch it online if you missed it.

I am reasonably pleased with how the presentation went. Watching it back was an uncomfortable experience but is very useful to feed into the next time I give this talk. Speaking of which, if you run a user group and would like me to give this talk get in touch.

For those of you interested in buying the two books I mentioned during the talk, they are The Passionate Programmer and Driving Technical Change, both from The Pragmatic Programmers.

I’d like to thank everyone who turned up, Ian Cooper for giving me the opportunity and Skills Matter for hosting.

Heroku in Europe

Thu, 26 Jan 2012 10:01:06 +0000

I tweeted yesterday (25th Jan 2012) to ask if anyone knew of a solution as convenient as Heroku but based in the UK or Europe.

The reason I asked was at Zopa we are thinking of migrating our front-end over to a Ruby stack and Heroku was the obvious option for hosting such a solution. However, as 95% or more of our traffic comes from the UK it doesn’t make complete sense to host our website outside of the UK or Europe at worst.

I got a lot of interest from Twitter as to whether I found anything but no suggestions of similar services based this side of the pond. At the same time I sent an email to Heroku’s support asking if they had any plans to give an UK or European hosting option.

Heroku got back to me yesterday evening and said they have no immediate plans for server resources outside the US but they are working on Safe Harbor certification if your concerns are of a more legal nature. They also confirmed that they are hosted within the US East region of AWS which is better than West for Europeans.

Their suggested strategy is to use a CDN alongside Heroku so that the majority of your assets will come from a local source and only your dynamic content will come from the US.

This probably rules out Heroku for us at Zopa as a CDN adds needless complexity for us given our traffic volumes and the fact that almost all of our traffic is from the same location. However, I thought I should knock a blog post together to share what I found.

Gain Trust and Create Change - LDNUG

Mon, 16 Jan 2012 14:12:27 +0000

I’m doing a talk at Skills Matter on 30th January titled “Gain Trust and Create Change” for the London .NET user group.

Registration has opened so reserve yourself a place.

If you run a user group and would like me to present this topic then drop me an email and I’ll try and arrange something.

DDD North - Introduction to Backbone.js

Sun, 09 Oct 2011 15:05:00 +0100

Yesterday, I gave a presentation on Backbone.js at the inaugural DDD North. My thanks go out to Andrew Westgarth and his team for organising it, it was a great event.

I was a bit nervous as it was my first time speaking at a conference and I think it showed. I rattled through my presentation at break-neck speed, unfortunately finishing well under my allocated hour. Niall Merrigan gave me some tips for sorting that out so I’ll hopefully be more composed next time!

I’d like to thank everyone who showed up for my talk and look forward to receiving their feedback. The slides for my presentation are now up on Slideshare and the code I demonstrated is available on Github.

Resources For, And How I Learnt Backbone.js

Wed, 14 Sep 2011 13:04:02 +0100

Resources For, And How I Learnt Backbone.js:

I was going to compile a similar list of resources for my upcoming talk on backbone.js at DDD North but Derick Bailey beat me to it. Check it out.

Finagle

Thu, 25 Aug 2011 09:44:59 +0100

Finagle:

I came across Finagle a few weeks ago, it’s:

… a protocol-agnostic, asynchronous RPC system for the JVM that makes it easy to build robust clients and servers in Java, Scala, or any JVM-hosted language.

And they’ve now written a blog post announcing it to the world. I had a skim over it before but it looks like they’ve added to the documentation since then.

I’ve been looking for a framework for writing servers in Scala, the Play framework is on my radar for web applications, and this looks like a great contender.

LMAX architecture - Martin Fowler

Wed, 13 Jul 2011 14:30:37 +0100

LMAX architecture - Martin Fowler:

I remember watching one of the original presentations on this architecture a few months ago (no idea why I didn’t link to it at the time) and Martin’s article puts it into a format that’s easier to consume at your leisure.

Really interesting approach!

The Most Important Code Isn't Code

Wed, 08 Jun 2011 13:24:44 +0100

The Most Important Code Isn't Code:

I cannot stress how much I agree with this. I’ve come to exactly the same conclusions as Zach in the last year myself. I wish I had written this post myself!

Here are my highlights:

Documentation is the best way to communicate your thoughts to yourself.

Forcing myself first to consider the API, the interface, and the end result led to a clarity that inspired less code and a more impactful project.

It improves the clarity of my code. I get more done in less lines.

But documenting your code is cool as hell. More importantly, it makes me feel more confident about my own code.

"There’s a whole school of thought that quantity of features is directly proportional to what you can..."

Thu, 02 Jun 2011 17:28:00 +0100

“There’s a whole school of thought that quantity of features is directly proportional to what you can charge for software. While clearly this is true in practice, that doesn’t mean that it’s not incredibly stupid. Every new feature makes your software more complex to use.”

- “How to keep your software awesome” by Jesse Emery

GiveCamp UK

Sat, 28 May 2011 09:38:00 +0100

This morning I signed up for GiveCamp UK. It is an event taking place this October in London where people involved in the software industry are donating a weekend of their free time to help charities with the IT needs.

What is GiveCamp? from givecamp on Vimeo.

If you can spare the time then sign up as a volunteer or if you work for a charity who could use some help submit a proposal. I have no doubt it is going to be a great weekend which will make a real difference to several charities.

Tags are magic! - Guardian

Wed, 09 Feb 2011 13:39:41 +0000

Tags are magic! - Guardian

Pragmatic web service design

Mon, 07 Feb 2011 14:28:28 +0000

Web services are a crucial part of most solutions nowadays, I spend a significant portion of my time designing and writing them and I have read a lot about them to make them better, faster and more resilient each time. This is a summary of how I approach web service design and the things I bear in mind.

Protocols and content types

Unless you require extreme performance from your service then use the most compatible technologies available. Today that means HTTP, JSON and HTML forms. The lowest common denominator in any solution is usually Javascript in the browser. This shapes all your decisions about how to expose your service. HTTP, JSON and HTML forms are the easiest things to work with in Javascript and they are well supported in other languages. XML is an option but JSON is a more efficient transport medium and much easier to work with in Javascript.

Before you write a web service make sure to learn HTTP inside and out. It is a powerful protocol that solves many more problems than most people realise. I would recommend RESTful web services as a starting point, it demonstrates how to create a web service that is sympathetic to HTTP and there is a useful glossary in the back. This is not going to be another post about REST but if you know about it already there will be some familiar concepts.

Using the correct HTTP code is important, think of it as a well established domain-specific language for what the server thought of your request. It removes the need to duplicate something like a response code within the body of the response. A caveat on exploiting all that HTTP gives you is to avoid 300 errors as browsers will redirect the whole page even when it is a response to an AJAX request.

For sending and receiving data I recommend HTML form values in, JSON out. You may want to send requests in as JSON instead and that’s not a bad choice, I just find HTML forms easier. However, I would recommend using the same input and output content type for requests across your whole API whenever possible. It makes it easier to consume an API when you do not have to think what format a given method accepts and responds with.

Avoid supporting several content types, particularly for the first couple of releases. You will end up iterating over your service during initial development and having to maintain compatibility for several content types will be adding wasteful overhead at this point. Get it shipped with one pair of content types then look to fix your bugs, harden your API and learn from real usage patterns. Put this into the next version and repeat. Once things stabilise you can evaluate whether support for several content types is needed based upon real world requests without it being such a burden as your API goes through the churn of first contact with the real world.

Core design considerations

A lot of the considerations are identical to those of designing any API but there are some additional ones that are specific to web services as they involve transmission over a network.

Log everything

It should be obvious but you need to log every request that comes in. You may need to replicate an issue that is a result of several requests, without logging this will be difficult. You want to easily answer the question “what was the consumer doing at the time?”. Logs will also allow you to monitor usage, response times and other useful metrics. As your first attempt at an API is no more than an informed guess you need to be collecting metrics in order to make an educated decision about what to do next.

Comply with the expected behaviour of HTTP

There are several expected behaviours with HTTP such as GET not producing side effects and PUT and DELETE requests being idempotent. This all comes from knowing HTTP as previously encouraged. Being able to justify a design decision by referring to RFCs is awesome.

Less methods returning more data

The biggest bottleneck in communicating with a server is no longer the size of the data being communicated, it is in establishing connections. This is particularly true for internal services. The common bottleneck with web services is the number of concurrent requests they can serve, not the amount of data being transferred. You want to aim to have less methods but return more data from them, therefore reducing the number of requests consumers need to make. Having less methods also makes it easier for the consumer to make the right choice.

Imagine everything a consumer is likely to want to display as a result of making the request and give it to them. The only thing you need to be careful of is your internal implementation bleeding into your API. You do not want to expose yourself in public, especially early on. Following this advice makes it likely the consumer will get all the data they need in one request rather than making a separate request for each part. This reduces the surface area of your API which has several great benefits:

as a developer I have less methods to maintain
as a consumer I have a lower cognitive load, I might even be able to memorise your API
as an administrator less requests helps me with caching and usually means I can use less hardware, improving service and making scaling up cheaper

In terms of web service surface area and required requests, less is definitely more.

This may seem to contradict my recommendation of using JSON but choosing a more efficient protocol when there is no compelling reason not to is foolish.

Highlander principle

There should be one, and only one, way to do something. Not only does it save you effort, it makes things easier for the consumer. Sometimes this may mean that one action requires several requests but this is less confusing in the long run than creating a specific method for every action. It is generally acceptable that the solution is good enough, performing several simple steps is cognitively easier than wading through a sea of methods to find the one intended for your task.

Again I may seem to be contradicting myself as this goes againsts the idea of trying to reduce the number of connections required to do something and it does to some extent. However, when you consider most systems are at least 80% reads, that this situation usually applies to writes and that this should not be a regularly occurring problem if you have modelled the domain correctly then it should be a drop in the ocean of the general usage of the API.

Any method you add on a hunch as to what future use will be you will have to support for the lifetime of the API. It is better to wait until you have real statistics and use-cases to work from than to increase the surface area of your API speculatively. If an action requiring several requests becomes common practice then you can add a method to simplify it and you will be certain that it is adding value to your code base and consumers.

Give people URLs

Whenever possible provide URLs to the consumer, do not make them work them out. Every URL a consumer has to create is a support call waiting to happen. When someone has to create a URL it is likely your internals are bleeding into your API. Once someone else is creating a URL you can never change that URL and that restricts refactoring and scaling options. When you create URLs for your consumers you can rename them and point them towards different servers to name just two options that would be closed to you the second you are not in control of your URLs.

Version from the outset

There will be several versions of your web service. Think about identifying the schema of your data when it is returned, think about how you will host several versions of the API as you cannot just switch one on and the previous version off. It will happen and it will be difficult to lever in at a later date, make it a solved problem.

Example - Twitter’s feed

Here’s the response for a single tweet from my timeline:

[ { ...,
    "entities" : { "hashtags" : [  ],
        "urls" : [  ],
        "user_mentions" : [ { "id" : 1102,
              "id_str" : "1102",
              "indices" : [ 3,
                  10
                ],
              "name" : "David Ulevitch",
              "screen_name" : "davidu"
            } ]
      },
    "favorited" : false,
    "geo" : null,
    "id" : 34612066580955136,
    "id_str" : "34612066580955136",
    "in_reply_to_screen_name" : null,
    "in_reply_to_status_id" : null,
    "in_reply_to_status_id_str" : null,
    "in_reply_to_user_id" : null,
    "in_reply_to_user_id_str" : null,
    "place" : null,
    "retweet_count" : 10,
    "retweeted" : false,
    "retweeted_status" : { 
        ...,
        "id" : 34583191385935872,
        "id_str" : "34583191385935872",
        ...,
        "retweet_count" : 10,
        ...,
        "text" : "The last 5% of a project is always the worst half of a project. :-)",
        "truncated" : false,
        "user" : { "contributors_enabled" : false,
            "created_at" : "Sun Jul 16 02:30:23 +0000 2006",
            "description" : "Positively disruptive.  Started OpenDNS, ...",
            ...,
            "id" : 1102,
            "id_str" : "1102",
            ...,
            "profile_background_color" : "9BE5E9",
            "profile_background_image_url" : "http://a3.twimg.com/profile...jpg",
            ...
          }
      },
    "source" : "web",
    "text" : "RT @davidu: The last 5% of a project is always the worst half of a project. :-)",
    "truncated" : false,
    "user" : { "contributors_enabled" : false,
        "created_at" : "Thu May 29 20:25:50 +0000 2008",
        "description" : "Freelance software developer fond of Linux, ...",
        ...
      }
  } ]

As you can see this contains everything about the tweet, including which tweet was being retweeted, who they were, what their profile preferences are, where their profile image is, everything you could want really. As a consumer I don’t need to make at least two additional request to retrieve the details for the involed users which would lead to a N+1 load on the server.

The only thing that sticks out to me as possibly being bad is the id values, that smells of internals details leaking out. Instead they might use user_url and instead give a URL for the user’s full profile and so forth. There also doesn’t appear to be any mention of a version for the message but they may handle versioning by using different URLs for different versions of the API.

Things to bear in mind during implementation

Do not reinvent the wheel

For example, methods to achieve caching already exist which utilise the caching mechanisms built into HTTP itself. Use these when possible rather than reimplementing it yourself. Squid and Varnish are two open source software based solutions that are easy to set up. Learn about the wide array of established HTTP headers available, it is rare that you need to create a custom header.

Comprehensive documentation

Yes, even for internal use. If your API is not documented it will be hard to use and under utilised. Without documentation you will spend a lot of time explaining how you go about using your API when you could tell a HTML file once and have that explain it to everyone.

Use examples of performing a common task with your API on top of an example for each method. Ideally the reader will be able to run these examples as they read. Ask a designer to throw a template together for you, it doesn’t need to be amazing but it will make reading your documentation a more pleasurable experience.

No breaking changes, ever

Once you’ve published it and it is exposed to the world, you cannot change anything, ever. Not even fix bugs as people will have written workarounds for them. You can add methods or return more data in response but you must never alter what has been published before. People will be relying on it and you will fuck them up.

TL;DR

Writing an API is very hard
Really learn HTTP
Remove choices and complexity by using less content types and exposing less methods
Reduce the quantity of requests by returning more data in responses
Give URLs to the consumer, don’t let them create them
Documentation, documentation, documentation

Nero - Me And You I love this track and I noticed they’ve...

Thu, 13 Jan 2011 09:22:14 +0000

Nero - Me And You

I love this track and I noticed they’ve made a Tron inspired video for it!

Cross-site XMLHttpRequest with CORS

Wed, 05 Jan 2011 12:50:06 +0000

Cross-site XMLHttpRequest with CORS:

I was working on a personal project over the Christmas period and wanted to make cross-site requests from an AJAX client to an API.

As the client and the API were hosted on different domains they violated the same origin policy implemented by almost all browsers. Modern browsers allow cross-site AJAX requests if the API allows cross-origin resource sharing (CORS).

The resources I found at the time on this were a little bare on details but I managed to piece it together and get something working. I was planning on writing a blog post about how to do it but now I know all the terminology I came across this complete article that explains CORS very well, linking to several useful resources.

Why Rack should matter to .NET developers

Tue, 07 Dec 2010 09:15:17 +0000

There’s been a lot of talk in the .NET community about Sinatra clones, namely Nancy and Nina in recent times but there are several others as well. Unfortunately, all these frameworks are missing the underlying reason that Sinatra is awesome and that is Rack. Sinatra packs a hell of a lot of functionality yet weighs in at 1198 lines of code including documentation. The main reason for that is the fact it is building on top of Rack which has allowed the authors to concentrate on creating a framework without having to worry about the details of dealing with server requests. So what is Rack, why you should care and why does it matter to you as a .NET developer.

What is Rack?

Rack is a layer of abstraction which sits between servers and frameworks. What it brings as a result is greater interoperability between HTTP server implementations and web frameworks. As the author of a HTTP server if you get your server to speak Rack you can now host applications written in several Ruby frameworks, Sinatra and Rails being two of the most well known. As a web framework author if you get your framework to speak Rack your application can now be hosted on several types of HTTP server.

This greatly reduces the barrier to adoption for both servers and frameworks. Rather than the previous method of having to write an adapter for each server or framework you wished to be compatible with; you are now able to write a single adapter for Rack and be hosted by several types of server or host multiple frameworks depending on which side of the boundary you are sitting.

Needless to say, the Ruby community has jumped all over this. It removes the duplication of effort that was previously required to get the same level of interoperability allowing the Ruby community to focus on innovation. For example, Sinatra did not exist before Rack. Rack didn’t make Sinatra possible but it made it much easier to create. Dealing with HTTP servers had been done for them. They could instead focus on turning requests into responses in an elegant manner.

Why should you care?

First, lets talk about testing. Rack is entirely Ruby based at its core. That means that when you are testing your application you can mimic a real HTTP request being invoked and see what response your application turns that into. Let me show you an example of that being done using RSpec against a simple Sinatra application:

get '/hello/:person' do |person|
  "Hello, #{person}"
end

We can test how that application will respond to a HTTP request using RSpec:

describe Server, 'making a hello request' do
  include Rack::Test::Methods
  
  let(:app) {subject}
  
  describe 'saying hello to world' do
    before do
      get '/hello/world'
    end
    
    it 'should respond "Hello, world"' do
      last_response.body.should == 'Hello, world'
    end
  end
end

I have commented the full source for these examples up on Github if you want further explanation of what is happening here.

We’ve just written an integration test for our application without having to deploy to a server or create our own abstraction of HTTP. This same type of test will work for any application written using a Rack-based framework. Just think of how painful the same thing is in .NET currently. Not only does Rack give us greater interoperability between servers and frameworks, it gives us a way to test them easily as well.

Rack also has the concept of middleware. These are modules built on top of Rack that can then be utilised by any application based upon Rack. This again reduces duplication of effort within the community. Things that can be implemented in the Rack layer, for example caching, are and that removes the need for functional clones for each Rack-based framework that wants the same functionality.

Why it matters to you as a .NET developer

If you’re thinking “well I don’t do integration tests, write HTTP servers or web frameworks, why should I care?” I’ll tell you why you should. For the frameworks you use, a .NET equivalent to Rack would eliminate a large amount of tedious and brittle code that adds no direct value to a framework. This would allow the developers of frameworks to focus their time on refinement giving you more powerful and stable tools to work with. You may not use it yourself but you will benefit from its existence within .NET.

A .NET equivalent to Rack would open the door for a whole new generation of web frameworks to be written. The barrier for writing a quality web framework would be lowered as the nuts and bolts of request and response handling would be a solved problem. We could see some real innovation in .NET rather than a mass of clones of frameworks from other languages.

The door would also be opened for existing servers, such as Apache, to be used for hosting .NET applications and enable entirely new servers to be created. IIS would have competitors. This alone makes it worthwhile. It would make the choice of .NET as a platform less tied, if at all, to choosing Microsoft as a vendor. Even more so than Mono does today, particularly in the web space.

There is a discussion going on right now about creating or choosing a single, common abstraction around HTTP, like Rack, for .NET. Get involved in this discussion and contribute some of your time. This is an opportunity to vastly improve the .NET ecosystem. We cannot afford to miss it.

Use integration tests when working in a new language

Fri, 19 Nov 2010 08:04:06 +0000

This is in some respects a follow up to my previous post in that it has been triggered by our internal project in Ruby. When working in Ruby I consciously lean towards integration tests. Sinatra and Sequel make this really as using rack-test and running against an in-memory SQLite database it is easy to simulate a full conversation of HTTP requests and they run reasonably quickly too.

How does this relate to learning a new language? When writing anything in a new language you will make mistakes. You are likely to structure your application poorly. At a minimum you will find there is a much better way of achieving the same result. If you have based your work on unit tests you will have a straight-jacket tying your to your code making the effort to heavily refactor it much greater than it needs to be. Tests will need to be moved, rewritten, thrown away and renamed. However, when you use integration tests you are only verifying the behaviour of the whole system not its moving parts. This means you can replace the internals of your application entirely and your integration tests will not need to be changed and will ensure you have a system that still behaves the same as it did before.

Am I saying that you should never use unit tests in a new language? Of course not. However, you should chose the right level of test to match your level of understanding. Once your application’s structure has ossified it makes sense to exercise the complex parts directly through unit tests. The second you make that shift to using using tests you are setting your structure in stone so you want to make sure that does not happen before it has to.

Simple tools and fundamental principles

Thu, 18 Nov 2010 13:00:49 +0000

We are doing an experiment in using Ruby for an internal project at work. I am the most experienced in Ruby so I’m leading the choices of gems and so forth and the team questioning those choices has lead to a bit of a personal epiphany.

I know that for a long while I have preferred simple tools and components. Simple tools tend to be more enabling as they give you control over precisely what is happening in your application whilst managing the boilerplate code for you. I’ve also focused my personal development and reading on fundamental principles to give me a solid grounding in concepts that can then be applied to any situation or language.

These opinions are being reflected in the tools I use for my Ruby development and therefore what I am encouraging the other developers at my company to use: Sinatra and Sequel. I enjoy the power you get from both to exploit all that is available from HTTP and SQL. I find it much easier to apply my knowledge of HTTP gained through reading about REST, though I doubt I’ll ever write a truly RESTful application. Things such as returning the correct HTTP codes and headers to enable client and proxy caching are ridiculously easy to do when compared to ASP.NET MVC. Being able to leverage my knowledge of SQL allows me to fine tune that vital query yet Sequel will still give me back easily consumable objects.

Choosing these simple tools for our task will help the team to learn Ruby and build an application, the real point of the task, rather than spending a large period of time learning how to use a more powerful framework such as Rails. The simpler tools should also let them exercise more of their existing knowledge, again allowing them to focus on learning the Ruby language rather than being distracted by Ruby frameworks.

As for me, I’m happy I’ve chosen this method of working and learning. I think it will stand me in good stead for working in any language I may want or need to learn in the future. It will be interesting to hear what the team thinks of this approach as we proceed.

The Long Beard's Revenge

Thu, 14 Oct 2010 10:15:00 +0100

The Long Beard's Revenge:

Very interesting article on a reduction in contribution to open source and the possibly underlying forces behind it.

10 Common Mistakes Made by API Providers

Mon, 11 Oct 2010 09:39:54 +0100

10 Common Mistakes Made by API Providers