Code rant

The Benefits of a Reverse Proxy

noreply@blogger.com (Mike Hadlow) — Fri, 03 May 2013 15:22:00 +0000

A typical ASP.NET public website hosted on IIS is usually configured in such a way that the server that IIS is installed on is visible to the public internet. HTTP requests from a browser or web service client are routed directly to IIS which also hosts the ASP.NET worker process. All the functionality needed to produce the web site is embodied in a single server. This includes caching, SSL termination, authentication, serving static files and compression. This approach is simple and straightforward for small sites, but is hard to scale, both in terms of performance, and in terms of managing the complexity of a large complex application. This is especially true if you have a distributed service oriented architecture with multiple HTTP endpoints that appear and disappear frequently.

A reverse proxy is server component that sits between the internet and your web servers. It accepts HTTP requests, provides various services, and forwards the requests to one or many servers.

Having a point at which you can inspect, transform and route HTTP requests before they reach your web servers provides a whole host of benefits. Here are some:

Load Balancing

This is the reverse proxy function that people are most familiar with. Here the proxy routes incoming HTTP requests to a number of identical web servers. This can work on a simple round-robin basis, or if you have statefull web servers (it’s better not to) there are session-aware load balancers available. It’s such a common function that load balancing reverse proxies are usually just referred to as ‘load balancers’. There are specialized load balancing products available, but many general purpose reverse proxies also provide load balancing functionality.

Security

A reverse proxy can hide the topology and characteristics of your back-end servers by removing the need for direct internet access to them. You can place your reverse proxy in an internet facing DMZ, but hide your web servers inside a non-public subnet.

Authentication

You can use your reverse proxy to provide a single point of authentication for all HTTP requests.

SSL Termination

Here the reverse proxy handles incoming HTTPS connections, decrypting the requests and passing unencrypted requests on to the web servers. This has several benefits:

Removes the need to install certificates on many back end web servers.
Provides a single point of configuration and management for SSL/TLS
Takes the processing load of encrypting/decrypting HTTPS traffic away from web servers.
Makes testing and intercepting HTTP requests to individual web servers easier.

Serving Static Content

Not strictly speaking ‘reverse proxying’ as such. Some reverse proxy servers can also act as web servers serving static content. The average web page can often consist of megabytes of static content such as images, CSS files and JavaScript files. By serving these separately you can take considerable load from back end web servers, leaving them free to render dynamic content.

Caching

The reverse proxy can also act as a cache. You can either have a dumb cache that simply expires after a set period, or better still a cache that respects Cache-Control and Expires headers. This can considerably reduce the load on the back-end servers.

Compression

In order to reduce the bandwidth needed for individual requests, the reverse proxy can decompress incoming requests and compress outgoing ones. This reduces the load on the back-end servers that would otherwise have to do the compression, and makes debugging requests to, and responses from, the back-end servers easier.

Centralised Logging and Auditing

Because all HTTP requests are routed through the reverse proxy, it makes an excellent point for logging and auditing.

URL Rewriting

Sometimes the URL scheme that a legacy application presents is not ideal for discovery or search engine optimisation. A reverse proxy can rewrite URLs before passing them on to your back-end servers. For example, a legacy ASP.NET application might have a URL for a product that looks like this:

http://www.myexampleshop.com/products.aspx?productid=1234

You can use a reverse proxy to present a search engine optimised URL instead:

http://www.myexampleshop.com/products/1234/lunar-module

Aggregating Multiple Websites Into the Same URL Space

In a distributed architecture it’s desirable to have different pieces of functionality served by isolated components. A reverse proxy can route different branches of a single URL address space to different internal web servers.

For example, say I’ve got three internal web servers:

http://products.internal.net/
http://orders.internal.net/
http://stock-control.internal.net/

I can route these from a single external domain using my reverse proxy:

http://www.example.com/products/    -> http://products.internal.net/
http://www.example.com/orders/    -> http://orders.internal.net/
http://www.example.com/stock/    -> http://stock-control.internal.net/

To an external customer it appears that they are simply navigating a single website, but internally the organisation is maintaining three entirely separate sites. This approach can work extremely well for web service APIs where the reverse proxy provides a consistent single public facade to an internal distributed component oriented architecture.

So …

So, a reverse proxy can off load much of the infrastructure concerns of a high-volume distributed web application.

We’re currently looking at Nginx for this role. Expect some practical Nginx related posts about how to do some of this stuff in the very near future.

Happy proxying!

Stop Your Console App The Nice Way

noreply@blogger.com (Mike Hadlow) — Mon, 22 Apr 2013 15:38:00 +0000

When you write a console application, do you simply put

Console.WriteLine("Hit <enter> to end");
Console.ReadLine();

at the end of Main and block on Console.ReadLine()?

It’s much nicer to to make your console application work with Windows command line conventions and exit when the user types Control-C.

The Console class has a static event CancelKeyPress that fires when Ctrl-C is pressed. You can create an AutoResetEvent that blocks until you call Set in the CancelKeyPress handler.

It’s also nice to stop the application from being arbitrarily killed at the point where Ctrl-C is pressed by setting the EventArgs.Cancel property to true. This gives you a chance to complete what you are doing and exit the application cleanly.

Here’s an example. My little console application kicks off a worker thread and then blocks waiting for Ctrl-C as described above. When Ctrl-C is pressed I send a signal to the worker thread telling it to finish what it’s doing and exit.

using System;
using System.Threading;

namespace Mike.Spikes.ConsoleShutdown
{
    class Program
    {
        private static bool cancel = false;

        static void Main(string[] args)
        {
            Console.WriteLine("Application has started. Ctrl-C to end");

            // do some cool stuff here
            var myThread = new Thread(Worker);
            myThread.Start();

            var autoResetEvent = new AutoResetEvent(false);
            Console.CancelKeyPress += (sender, eventArgs) =>
                {
                    // cancel the cancellation to allow the program to shutdown cleanly
                    eventArgs.Cancel = true;
                    autoResetEvent.Set();
                };

            // main blocks here waiting for ctrl-C
            autoResetEvent.WaitOne();
            cancel = true;
            Console.WriteLine("Now shutting down");
        }

        private static void Worker()
        {
            while (!cancel)
            {
                Console.WriteLine("Worker is working");
                Thread.Sleep(1000);
            }
            Console.WriteLine("Worker thread ending");
        }
    }
}

Much nicer I think.

Serializing Lua Coroutines With Pluto

noreply@blogger.com (Mike Hadlow) — Thu, 18 Apr 2013 15:11:00 +0000

In my last post I showed how it’s possible to use Lua as a distributed workflow scripting language because of its build in support for coroutines. But in order to create viable long-running workflows we have to have some way of saving the script’s state at the point that it pauses; we need a way to serialize the coroutine.

Enter Pluto:

“Pluto is a heavy duty persistence library for Lua. Pluto is a library which allows users to write arbitrarily large portions of the "Lua universe" into a flat file, and later read them back into the same or a different Lua universe.”

I downloaded the Win32 build of Tamed Pluto from here and placed it in a directory alongside my Pluto.dll. You have to tell Lua where to find C libraries by setting the Lua package.cpath:

using (var lua = new Lua())
{
    lua["package.cpath"] = @"D:\Source\Mike.DistributedLua\Lua\?.dll";

    ...
}

Lua can now find the Pluto library and we can import it into our script with:

require('pluto')

Here’s a script which defines a function foo which calls a function bar. Foo is used to create a coroutine. Inside bar the coroutine yields. The script uses Pluto to serialize the yielded coroutine and saves it to a file.

-- import pluto, print out the version number 
-- and set non-human binary serialization scheme.
require('pluto')
print('pluto version '..pluto.version())
pluto.human(false)

-- perms are items to be substituted at serialization
perms = { [coroutine.yield] = 1 }

-- the functions that we want to execute as a coroutine
function foo()
    local someMessage = 'And hello from a long dead variable!'
    local i = 4
    bar(someMessage)
    print(i)
end

function bar(msg)
    print('entered bar')
    -- bar runs to here then yields
    coroutine.yield()
    print(msg)
end

-- create and start the coroutine
co = coroutine.create(foo)
coroutine.resume(co)

-- the coroutine has now stopped at yield. so we can
-- persist its state with pluto
buf = pluto.persist(perms, co)

-- save the serialized state to a file
outfile = io.open(persistCRPath, 'wb')
outfile:write(buf)
outfile:close()

This next script loads the serialized coroutine from disk, deserializes it, and runs it from the point that it yielded:

-- import pluto, print out the version number 
-- and set non-human binary serialization scheme.
require('pluto')
print('pluto version '..pluto.version())
pluto.human(false)

-- perms are items to be substituted at serialization
-- (reverse the key/value pair that you used to serialize)
perms = { [1] = coroutine.yield }

-- get the serialized coroutine from disk
infile, err = io.open(persistCRPath, 'rb')
if infile == nil then
    error('While opening: ' .. (err or 'no error'))
end

buf, err = infile:read('*a')
if buf == nil then
    error('While reading: ' .. (err or 'no error'))
end

infile:close()

-- deserialize it
co = pluto.unpersist(perms, buf)

-- and run it
coroutine.resume(co)

When we run the scripts, the first prints out ‘entered bar’ and then yields:

LUA> pluto version Tamed Pluto 1.0
LUA> entered bar

The second script loads the paused ‘foo’ and ‘bar’ and continues from the yield:

LUA> pluto version Tamed Pluto 1.0
LUA> And hello from a long dead variable!
LUA> 4

Having the ability to run a script to the point at which it starts a long running call, serialize its state and store it somewhere, then pick up that serialized state and resume it at another place and time is a very powerful capability. It means we can write simple procedural scripts for our workflows, but have them execute over a distributed service oriented architecture.

The complete code for this experiment is on GitHub here.

Lua as a Distributed Workflow Scripting Language

noreply@blogger.com (Mike Hadlow) — Thu, 04 Apr 2013 15:39:00 +0000

I’ve been spending a lot of time recently thinking about ways of orchestrating long running workflows in a service oriented architecture. I was talking this over at last Tuesday’s Brighton ALT NET when Jay Kannan, who’s a game developer amongst other things, mentioned that Lua is a popular choice for scripting game platforms. Maybe I should check it out. So I did. And it turns out to be very interesting.

If you haven’t heard of Lua, it’s a “powerful, fast, lightweight, embeddable scripting language” originally conceived by a team at the Catholic University of Rio de Janeiro in Brazil. It’s the leading scripting language for game platforms and also pops up in other interesting locations including Photoshop and Wikipedia. It’s got a straightforward C API that makes it relatively simple to p-invoke from .NET, and indeed there’s a LuaInterface library that provides a managed API.

I got the source code from the Google code svn repository and built it in VS 2012, but there are NuGet packages available as well.

It turned out to be very simple to use Lua to script a distributed workflow. Lua has first class coroutines which means that you can pause and continue a Lua script at will. The LuaInterface library allows you inject C# functions and call them as Lua functions, so it’s simply a case of calling an asynchronous C# ‘begin’ function, suspending the script by yielding the coroutine, waiting for the asynchronous function to return, setting the return value, and starting up the script again.

Let me show you how.

First here’s a little Lua script:

a = 5
b = 6

print('doing remote add ...')

r1 = remoteAdd(a, b)

print('doing remote multiply ...')

r2 = remoteMultiply(r1, 4)

print('doing remote divide ...')

r3 = remoteDivide(r2, 2)

print(r3)

The three functions ‘remoteAdd’, ‘remoteMultiply’ and ‘remoteDivide’ are all asynchronous. Behind the scenes a message is sent via RabbitMQ to a remote OperationServer where the calculation is carried out and a message is returned.

The script runs in my LuaRuntime class. This creates and sets up the Lua environment that the script runs in:

public class LuaRuntime : IDisposable
{
    private readonly Lua lua = new Lua();
    private readonly Functions functions = new Functions();

    public LuaRuntime()
    {
        lua.RegisterFunction("print", functions, typeof(Functions).GetMethod("Print"));
        lua.RegisterFunction("startOperation", this, GetType().GetMethod("StartOperation"));

        lua.DoString(
@"
function remoteAdd(a, b) return remoteOperation(a, b, '+'); end
function remoteMultiply(a, b) return remoteOperation(a, b, '*'); end
function remoteDivide(a, b) return remoteOperation(a, b, '/'); end

function remoteOperation(a, b, op)
    startOperation(a, b, op)
    local cor = coroutine.running()
    coroutine.yield(cor)

    return LUA_RUNTIME_OPERATION_RESULT
end
");
    }

    public void StartOperation(int a, int b, string operation)
    {
        functions.RunOperation(a, b, operation, result =>
            {
                lua["LUA_RUNTIME_OPERATION_RESULT"] = result;
                lua.DoString("coroutine.resume(co)");
            });
    }

    public void Execute(string script)
    {
        const string coroutineWrapper =
@"co = coroutine.create(function() 
{0}
end)";
        lua.DoString(string.Format(coroutineWrapper, script));
        lua.DoString("coroutine.resume(co)");
    }

    public void Dispose()
    {
        lua.Dispose();
        functions.Dispose();
    }
}

When this class is instantiated it creates a new LuaInterface environment (the Lua class) and a new instance of a Functions class that I’ll explain below.

The constructor is where most of the interesting setup happens. First we register two C# functions that we want to call from inside Lua: ‘print’ which simply prints from the console, and ‘startOperation’ which starts an asynchronous math operation.

Next we define our three functions: ‘remoteAdd’, ‘remoteMultiply’ and ‘remoteDivide’ which all in turn invoke a common function ‘remoteOperation’. RemoteOperation calls the registered C# function ‘startOperation’ then yields the currently running coroutine. Effectively the script will stop here until it’s started again. After it starts, the result of the asynchronous operation is accessed from the LUA_RUNTIME_OPERATION_RESULT variable and returned to the caller.

The C# function StartOperation calls RunOperation on our Functions class which has an asynchronous callback. In the callback we set the result value in the Lua environment and execute ‘coroutine.resume’ which restarts the Lua script at the point where it yielded.

The Execute function actually runs the script. First it embeds it in a ‘coroutine.create’ call so that the entire script is created as a coroutine, then it simply starts the coroutine by calling ‘coroutine.resume’.

The Functions class is just a wrapper around a function that maintains an EasyNetQ connection to RabbitMQ and makes an EasyNetQ request to a remote server somewhere else on the network.

public class Functions : IDisposable
{
    private readonly IBus bus;

    public Functions()
    {
        bus = RabbitHutch.CreateBus("host=localhost");
    }

    public void Dispose()
    {
        bus.Dispose();
    }

    public void RunOperation(int a, int b, string operation, Action<int> resultCallback)
    {
        using (var channel = bus.OpenPublishChannel())
        {
            var request = new OperationRequest()
                {
                    A = a,
                    B = b,
                    Operation = operation
                };
            channel.Request<OperationRequest, OperationResponse>(request, response =>
                {
                    Console.WriteLine("Got response {0}", response.Result);
                    resultCallback(response.Result);
                });
        }
    }

    public void Print(string msg)
    {
        Console.WriteLine("LUA> {0}", msg);
    }
}

Here’s a sample run of the script:

DEBUG: Trying to connect
DEBUG: OnConnected event fired
INFO: Connected to RabbitMQ. Broker: 'localhost', Port: 5672, VHost: '/'
LUA> doing remote add ...
DEBUG: Declared Consumer. queue='easynetq.response.143441ff-3635-4d5d-8e42-6b379b3f8356', prefetchcount=50
DEBUG: Published to exchange: 'easy_net_q_rpc', routing key: 'Mike_DistributedLua_Messages_OperationRequest:Mike_DistributedLua_Messages', correlationId: '50560dd9-2be1-49a1-96f6-9c62641080ae'
DEBUG: Recieved 
    RoutingKey: 'easynetq.response.143441ff-3635-4d5d-8e42-6b379b3f8356'
    CorrelationId: '50560dd9-2be1-49a1-96f6-9c62641080ae'
    ConsumerTag: '101343d9-9497-4893-88e6-b89cc1de29a4'
Got response 11
LUA> doing remote multiply ...
DEBUG: Declared Consumer. queue='easynetq.response.f571f6d7-b963-4a88-bf62-f05785009e39', prefetchcount=50
DEBUG: Published to exchange: 'easy_net_q_rpc', routing key: 'Mike_DistributedLua_Messages_OperationRequest:Mike_DistributedLua_Messages', correlationId: '0ea7e1c3-6f12-4cb9-a861-2f5de8f2600d'
DEBUG: Model Shutdown for queue: 'easynetq.response.143441ff-3635-4d5d-8e42-6b379b3f8356'
DEBUG: Recieved 
    RoutingKey: 'easynetq.response.f571f6d7-b963-4a88-bf62-f05785009e39'
    CorrelationId: '0ea7e1c3-6f12-4cb9-a861-2f5de8f2600d'
    ConsumerTag: '2c35f24e-7745-4475-885a-d214a1446a70'
Got response 44
LUA> doing remote divide ...
DEBUG: Declared Consumer. queue='easynetq.response.060f7882-685c-4b00-a930-aa4f20f7c057', prefetchcount=50
DEBUG: Published to exchange: 'easy_net_q_rpc', routing key: 'Mike_DistributedLua_Messages_OperationRequest:Mike_DistributedLua_Messages', correlationId: 'ea9a90cc-cd7d-4f05-b171-c6849026ac4a'
DEBUG: Model Shutdown for queue: 'easynetq.response.f571f6d7-b963-4a88-bf62-f05785009e39'
DEBUG: Recieved 
    RoutingKey: 'easynetq.response.060f7882-685c-4b00-a930-aa4f20f7c057'
    CorrelationId: 'ea9a90cc-cd7d-4f05-b171-c6849026ac4a'
    ConsumerTag: '90e6b024-c5c4-440a-abdf-cb9a000c131c'
Got response 22
LUA> 22
DEBUG: Model Shutdown for queue: 'easynetq.response.060f7882-685c-4b00-a930-aa4f20f7c057'
Completed
DEBUG: Connection disposed

You can see the Lua print statements interleaved with EasyNetQ DEBUG statements showing the messages being published and consumed.

So there you go, a distributed workflow scripting engine in under 100 lines of code. All I’ve got to do now is serialize the Lua environment at each yield and then restart it again from its serialized state. This is possible according to a bit of googling yesterday afternoon. Watch this space.

You can find the code for all this on GitHub here:

https://github.com/mikehadlow/Mike.DistributedLua

Coders, Musicians and Cooks

noreply@blogger.com (Mike Hadlow) — Tue, 05 Mar 2013 08:52:00 +0000

In a light hearted Twitter exchange yesterday I asked why so many coders also played guitar. Mark Seemann suggested that there was also a high correlation with cooking.

How about a survey? Mark and I have over 4000 twitter followers between us so I was sure we could get a few of them to click a few radio buttons, with no promise of reward, but with the warm feeling that they were helping to progress the anthropological understanding of the coding tribe. So I created a survey and waited for the results to stream in. Of course, as Rob Pickering pointed out, this kind of self selecting survey is very unscientific, but fun nonetheless.

What have we learnt?

It is ridiculously easy to create an online survey using Google Drive. Just click ‘new form’, type in a few questions, and Bob’s yer uncle. It automatically builds a nice summary graphic on the fly as the survey progresses (see below).
Most of Mark and I’s followers are C# coders. 81%. That’s not really surprising.
More coders cook than play musical instruments: 85% verses 53%. That’s not really surprising either. I expect if you took a survey of the general population you’d find quite a high percentage who cook verses those who play.
Guitar is by far the most popular instrument among coders. 30% play guitar. This confirms my hunch, but doesn’t answer my question – why it’s such a high percentage.

Here is the Google Drive graphic of the results.

Fractured Product Syndrome

noreply@blogger.com (Mike Hadlow) — Wed, 27 Feb 2013 17:00:00 +0000

How do you sell your software product? Do you fork the source code of your previous customer, modify it a little and deploy it as its own instance with its own database? Maybe each customer even has their own server? Their own set of servers?

You are suffering from the Fractured Product Syndrome.

Let me tell you a story.

Colin and Alex have a little web development consultancy, called Coaltech (get it?) they are pretty handy with ASP.NET and SQL Server and already had a respectable list of satisfied clients when a local car hire company, Easy Wheels, asked them if they could build them a car booking system. It would keep a list of Easy Wheels’ cars and allow customers to view them and book them online.

Colin and Alex signed a contract with Easy Wheels and got busy. Their “just do it” attitude to getting features done meant that the code wasn’t particularly beautiful, but it did the job and after a couple of months work the system went live. Easy Wheels were very happy with their new system, so happy in fact that they started to tell other independent car rental companies about it. A few months later Colin and Alex got a call from Hire My Ride. They loved what Coaltech had done for Easy Wheels and wanted a similar system. Of course it would have to be redesigned to fit in with Hire My Ride’s branding, but the basic functionality was pretty much identical. Colin and Alex did the obvious thing, they took a copy of the Easy Wheels code and tweaked it for Hire My Ride. Of course they charged the same bespoke software price for Hire My Ride’s system.

Soon a third and fourth car hire company had asked Coaltech for their ‘system’. Each time Colin and Alex took a copy of the last customer’s code – it made sense to take the most recent code because they inevitably added a few refinements with each customer – and then they altered it to match the new customer’s requirements. Before long they had arranged a deal with another web agency to take over all their non-car-hire work and decided to concentrate full time on the ‘system’. It needed a name too, they couldn’t carry on calling it ‘Easy Wheels’ and they certainly couldn’t market it like that. Soon it became ‘Coaltech Cars’ with a new marketing website. They also found that they couldn’t keep up with demand with just the two of them doing customer implementations. Each new customer took around six weeks of development work with the inevitable to-and-fro of slightly different requirements and design changes. To help meet demand they started to hire developers. First Jez, then Dave, then Neville. They all modified the software in slightly different ways, but it didn’t seem to matter at the time.

Fast forward five years. Coaltech now has 50 employees and around 50 customers. It’s fair to say that they are the leading vendor of independent car rental management systems. The majority of their staff are developers, although they also have a small sales, HR and customer relationship management team. You would have thought that Colin and Alex would be happy to have turned their little web development company into such a success, but instead life just seemed to be one long headache. Although the company had grown, it always seemed hard to turn a profit. As it was customers often balked at the price of the system. The same bug would turn up again and again for different customers. There never seemed to be enough time to fix them all. Even though they might have delivered a new feature for one client, it always seemed to take just as long to implement it for another; sometimes longer depending on what code they’d been forked from. The small team that looked after the servers were in constant fire fighting mode and got very upset when anyone suggested ‘upgrading’ any of the clients - it always meant bugs, downtime and screaming customers. And then the government changed the Value Added Tax rate. Colin had to cancel his holiday and they lost two of their best developers after several weeks of late nights and no weekends while they updated and redeployed 50 separate web applications.

The end for Coaltech cars came slowly and painfully. Alex had the first hint of it when he was made aware of a little company called RentBoy. They had a little software-as-a-service product for small car hire companies. To use it you entered your credit card number, a logo, a few other details, and you were good to go. They weren’t any immediate competition for Coaltech Cars, having only a small subset of the features, but they soon captured the low end of the market, the one or two man band car companies that had never been able to afford Coaltech’s sign up or licensing fees.

But then the features started coming thick and fast and soon Coaltech found they were loosing out to RentBoy when bidding for clients. Colin found an article on the High Scalability blog about RentBoy’s architecture. They had a single scalable application that served all their customers, one place to fix bugs, one point of scalability and maintenance. They practiced continuous deployment allowing them to quickly build and release new features for all their customers. The company had four employees and already more customers than Coaltech. They charged about a tenth of Coaltech’s fees.

Coaltech’s new client revenues soon dried up. They’d always made a certain amount of money from sign-up fees. Too late they realised that they had to start shedding staff, always a painful thing for a small closely knit company. The debts mounted until the bank would no longer support them, and before long they had to declare bankruptcy. Luckily Colin and Alex managed to get jobs as project managers in enterprise development teams.

The moral of the story? Try to avoid Fractured Product Syndrome if you possibly can. Although simply forking the source code for each new customer appears by far the easiest thing to do at the start, it simply doesn’t scale. Start thinking about how to build multi-tenanted software-as-a-service long before you get to where Coaltech got to. Learn to say ‘no’ to customers if you have to. It’s far better to have a high number of low value customers than smaller numbers of higher value ones on a differentiated platform. It’s much easier for a low-value volume software provider to move into the high-value space than for a high-value provider to move down.

Learn to recognise Fractured Product Syndrome and address it before it gets serious!

EasyNetQ on .NET Rocks!

noreply@blogger.com (Mike Hadlow) — Thu, 21 Feb 2013 12:39:00 +0000

Last week I had the pleasure of being interviewed by Carl Franklin and Richard Campbell for a .NET Rocks episode on RabbitMQ and EasyNetQ. It was terrific fun and a real honour to be invited on the show. I’ve been listening to .NET Rocks since it started in 2002 so you can imagine how excited I was. Carl and Richard are seasoned pros when it comes to podcasting, and the awesome ninja editing skills they posses turned my rather hesitant and rambling answers into something that almost sounded coherent.

You can listen to the show on the link below:

http://www.dotnetrocks.com/default.aspx?ShowNum=848

Now Richard, about that Tesla …

EasyNetQ in Africa

noreply@blogger.com (Mike Hadlow) — Wed, 06 Feb 2013 10:53:00 +0000

Anthony Moloney got in touch with me recently. He’s using EasyNetQ, my simple .NET API for RabbitMQ, with his team in Kenya. Here’s an email he sent me:

Hi Mike,

Further to the brief twitter exchange today about using EasyNetQ on our Kenyan project. We started using EasyNetQ back in early November and I kept meaning to drop you a line to thank you for all your good work.

Virtual City are based in Nairobi and supply mobile solutions to the supply chain and agribusiness industry in Africa. African solutions for African problems. I got involved with them about 2 years ago to help them improve the quality of their products and I have been working on and off with them since then. Its been a bit of a journey we are getting there.

We have a number of client applications including android and wpf working in an online/offline mode over mobile networks. We need to process large amounts of incoming commands from these applications. These commands are also routed via the server to other client apps.

The application had originally used MVC and SQL server to synchronously process and store the commands but we were running into severe performance problems. We looked at various MQ solutions and decided to use RabbitMQ, WebApi & Mongo to improve processing throughput. While researching a .Net API for RabbitMQ I noticed that you had created the EasyNetQ API.

EasyNetQ greatly simplifies interacting with RabbitMQ and providing your needs are not too complicated you really don't need to know too much about the guts of RabbitMQ. We replaced the existing server setup in about a week. The use of RabbitMQ has greatly increased the scalability of the product and allows us to either scale up or scale out.

We are also using the EasyNetQ management API for monitoring queue activity on our customer services dashboard.

Kind Regards

Anthony Moloney

One of the great rewards of running an open source project is hearing about the fascinating ways that it’s used around the world. I really like that it’s an ‘African solution for African problems’ and built by a Kenyan development team. It’s also interesting that they’ve used OSS projects like RabbitMQ and Mongo alongside .NET. It reminds me of the Stack Overflow architecture, a .NET core surrounded by OSS infrastructure.

How to Write Scalable Services

noreply@blogger.com (Mike Hadlow) — Fri, 01 Feb 2013 13:02:00 +0000

I’ve spent the last five years implementing and thinking about service oriented architectures. One of the core benefits of a service oriented approach is the promise of greatly enhanced scalability and redundancy. But to realise these benefits we have to write our services to be ‘scalable’. What does this mean?

There are two fundamental ways we can scale software: 'Vertically' or 'horizontally'.

Vertical Scaling addresses the scalability of a single instance of the service. A simple way to scale most software is simply to run it on a more powerful machine; one with a faster processor or more memory. We can also look for performance improvements in the way we write the code itself. An excellent example of company using this approach is LMAX. However, there are many drawbacks to the vertical scaling approach. Firstly the costs are rarely linear; ever more powerful hardware tends to be exponentially more expensive and the costs (and constraints) of building sophisticated performance optimised software are also considerable. Indeed premature performance optimisation often leads to overly complex software that's hard to reason about and therefore more prone to defects and high maintenance costs. Most importantly, vertical scaling does not address redundantcy; vertically scaling an application just turns a small single point of failure into a large single point of failure.
Horizontal Scaling. Here we run multiple instances of the application rather than focussing on the performance of a single instance. This has the advantage of being linearly scalable; rather than buying a bigger, more expensive box, we just buy more copies of the same cheap box. With the right architectural design, this approach can scale massively. Indeed it's the approach taken by almost all of largest internet scale companies: Facebook, Google, Twitter etc.. Horizontal Scaling also introduces redundancy; the loss of a single node need not impact the system as a whole. For these reasons, horizontal scaling is the preferred approach to building scalable, redundant systems.

So, the fundamental approach to building scalable systems is to compose them of horizontally scaled services. In order to do this we need to follow a few basic principles:

Stateless. Any services that stores state across an interaction with another service is hard to scale. For example, a web service that stores in-memory session state between requests requires a sophisticated session-aware load balancer. A stateless service, by contrast, only requires simple round-robin load balancing. For a web application (or service) you should avoid using session state or any static or application level variables.
Coarse Grained API. To be stateless, a service should expose an API that exposes operations as a single interaction. A chatty API, where one sets up some data, asks for some transition, and then reads off some results, implies statefulness by its design. The service would need to identify a session and then maintain information about that session between successive calls. Instead a single call, or message, to the service should encapsulate all the information that the service requires to complete the operation.
Idempotent. Much scalable infrastructure is a trade-off between competing constraints. Delivery guarantees are one of these. For various reasons it's is far simpler to guarantee 'at least once' delivery than 'exactly once'. If you can make your software tolerant of multiple deliveries of the same message it will be easier to scale.
Embrace Failure. Arrays of services are redundant if the system as a whole can survive the loss of a single node. You should design your services and infrastructure to expect and survive failure. Consider implementing a Chaos Monkey that randomly kills processes. If you start by expecting your services to fail, you'll be prepared when they inevitably do.
Avoid instance specific configuration. A scalable service should be designed in such a way that it doesn't need to know about other instances of itself, or have to identify itself as a specific instance. I shouldn't need to have to configure one instance any differently than another. This would include communication mechanisms that require messages to be addressed to a specific instance of the service, or some non-convention based way that the service was required to identify itself. Instead we should rely on infrastructure (load-balancers, pub-sub messaging etc.) to manage the communication between arrays of services.
Simple automated deployment. Have a service that can scale is no advantage if we can't deploy it when we are close to capacity. A scalable system must have automated processes to deploy new instances of services as the need arises.
Monitoring. We need to know when services are close to capacity so that we can add additional service instances. Monitoring is usually an infrastructure concern; we should be monitoring CPU, network, and memory usage and have alerts in place to warn us when these pass certain trigger points. Sometimes it's worth introducing application specific alerts when some internal trigger is reached, such as the number of items in an in-memory queue, for example.
KISS - Keep It Small and Simple. This is good advice for any software project, but is especially pertinent to building scalable resilient systems. Large monolithic codebases are hard to reason about, hard to monitor, and hard to scale. Building your system out of many small pieces makes it easy to address those pieces independently. Design your system so that each service has only one purpose and is decoupled from the operations of other services. Have your services communicate using non-proprietary open standards to avoid vendor lock-in and allow for a heterogeneous platform. JSON over HTTP, for example, is an excellent choice for intra-service communication. Every platform has HTTP and JSON libraries and there is abundant off-the-shelf infrastructure (proxies, load-balancers, caches) that can be used to help your system scale.

This post just gives a few pointers to building scalable systems, for far more detailed examples and case studies I can't recommend the High Scalability Blog enough. The Dodgy Coder blog has a very nice summary of some of the High Scalability case studies here.

Visual Studio: Multiple Startup Projects

noreply@blogger.com (Mike Hadlow) — Thu, 31 Jan 2013 12:38:00 +0000

I’ve been using Visual Studio for over 10 years, but still keep on learning new tricks. That’s because I learn things very slowly dear reader! Today’s ‘wow, I didn’t know you could do that moment’, was finding out that it’s possible to launch multiple startup projects when one hits F5 – or Control-F5 in this case.

My scenario is that I’m writing a little spike for a distributed application. Each of my services is implemented as a console app. During development, I want to run integration tests where the services all talk to each other, so before running the tests I want to run all the services. Now I’m used to right-clicking on a project and choosing ‘Set as startup project’, but you can’t select multiple projects this way and it’s very tedious to launch multiple projects by going to each one in turn, setting it as the startup project, and then hitting ctrl-F5. What I didn’t know is that you can right click on the Solution node, select ‘Set Startup Projects’ and you get this dialogue:

You can then select multiple startup projects and choose any number of them to ‘Start without debugging’.

Now I can hit ctrl-F5 and all my little services start up. Wonderful.

Converting Between Unix Time And DateTime

noreply@blogger.com (Mike Hadlow) — Wed, 16 Jan 2013 15:31:00 +0000

Unix time is the time value used in Unix based operating systems and is often exposed by Unix based APIs. To convert it to, or from, a .NET System.DateTime simply calculate the number of seconds since the Unix Epoch, midnight on the 1st January 1970. I’ve created a little class you can use to do just that. Note that the Unix Epoch is UTC, so you should always convert your local time to UTC before doing the calculation.

public class UnixTime
{
    private static readonly DateTime epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);

    public static long FromDateTime(DateTime dateTime)
    {
        if (dateTime.Kind == DateTimeKind.Utc)
        {
            return (long)dateTime.Subtract(epoch).TotalSeconds;
        }
        throw new ArgumentException("Input DateTime must be UTC");
    }

    public static DateTime ToDateTime(long unixTime)
    {
        return epoch.AddSeconds(unixTime);
    }

    public static long Now
    {
        get
        {
            return FromDateTime(DateTime.UtcNow);
        }
    }
}

You can convert from Unix time to a UTC DateTime like this:

var calculatedCurrentTime = UnixTime.ToDateTime(currentUnixTime);

Convert to Unix time from a UTC DateTime like this:

var calculatedUnixTime = UnixTime.FromDateTime(myDateTimeValue);

And get the current time as a UTC time value like this:

Console.Out.WriteLine(UnixTime.Now);

As an interesting aside, the 32 bit signed integer used in older Unix systems will overflow at 14 minutes past 3 o’clock and 7 seconds on the morning of 19th January 2038 and interpret the date as 1901. I shall come out of retirement and spend happy well paid hours as a ‘unix time consultant’. 64 bit systems will overflow in the year 292,277,026,596.

A Geek Christmas Quiz–The Answers!

noreply@blogger.com (Mike Hadlow) — Mon, 24 Dec 2012 12:56:00 +0000

Thanks for everyone who had a go at my Geek Christmas Quiz. The response was fantastic with both Iain Holder and Rob Pickering sending me emails of their answers. I’m pretty sure neither of them Googled any of the questions, since their scores weren’t spectacular :)

So, now the post you’ve all been waiting for with such anticipation… the answers!

Computers

G.N.U stands for GNU is Not Unix. A recursive acronym, how geeky is that?
The A in ARM originally stood for ‘Acorn’ as in Acorn Risc Machine. Yes, I know it stands for ‘Advanced’ now, but the question said ‘originally’.
TCP stands for Transmission Control Protocol.
Paul Allen founded Microsoft with Bill Gates. I’ve just finished reading his memoirs ‘Ideas Man’. Hard work!
F2 (hex) is 15(F) * 16 + 2 = 242. 1111 0010 (binary)
Windows ME was based on the Balmer Peak theory of software development.
The first programmer was Ada Lovelace. Yes yes, I know that’s contentious, but I made up this quiz, so there!
UNIX time started in 1970 (1st January to be exact). I know this because I just had to write a System.DateTime to UNIX time converter.
SGI, the mostly defunct computer maker. You get a mark for Silicon Graphics International (or Inc).
Here’s monadic ‘Bind’ in C#: M<B> Bind<A,B>(M<A> a, Func<A, M<B>> func)

Name That Geek!

Bill Gates – Co-founder of Microsoft with Paul Allen.
Tim Berners-Lee – Creator of the World Wide Web.
Larry Ellison – Founder of Oracle. Lives in a Samurai House (how geeky is that?)
Linus Torvalds – Creator of Linux.
Alan Turing – Mathematician and computer scientist. Described the Turing Machine. Helped save the free world from the Nazis.
Steve Jobs – Founded Apple, NeXT and Pixar.
Natalie Portman – Actress and self confessed geek.

Science

The four ‘letters’ of DNA are C, G, T and A. If you know the actual names of the nucleotides (guanine, adenine, thymine, and cytosine), give yourself a bonus point – you really are a DNA geek!
The ‘c’ in E = mc² is a constant, the speed of light.
The next number in the Fibonacci sequence 1 1 2 3 5 8 is 13 (5 + 8).
C₈H₁₀N₄0₂ is the chemical formula for caffeine.
According to Wikipedia, Australopithecus, the early hominid, became extinct around 2 million years ago.
You would not find an electron in an atomic nucleus.
Nitrogen is the most common gas in the Earth’s atmosphere.
The formula for Ohm’s Law is I = V/R (current = voltage / resistance).
A piece of paper that, when folded in half, maintains the ratio between the length of its sides, has sides with a length ratio of 1.618, ‘the golden ratio’. Did you know that the ratio between successive Fibonacci sequence numbers also tends to the golden ratio? Maths is awesome!
The closest living land mammal to the cetaceans (including whales) is the Hippopotamus.

Space

The second (and third) stage of the Apollo Saturn V moon rocket was powered by five J2 rocket engines.
Saturn’s largest moon is Titan. Also the only moon in the solar system (other than our own) that a spaceship has landed on.
You would experience 1/6th of the Earth’s gravity on the moon. Or there about.
This question proved most contentious. The answer is false, there is nowhere in space that has no gravity. Astronauts are weightless because they are in free-fall. Gravity itself is a property of space.
A Geosynchronous spaceship has an orbital period of 24 hours. So it appears to be stationary to a ground observer.
The furthest planet from the sun is Neptune. Far fewer people know this, than know that Pluto used to be the furthest planet from the sun. Actually Pluto was only the furthest for part of it’s, irregular, orbit.
There are currently 6 people aboard the International Space Station.
According to Google (yes, I know) there are 13,000 earth satellites.
Prospero was the only satellite built and launched by the UK. It was launched by stealth after the programme had been cancelled, that’s the way we do things in the UK.
The second man on the moon was Buzz Aldrin. He’s never forgiven NASA.

Name That Spaceship!

In this round, give yourself a point if you can name the film or TV series the fictional spacecraft appeared in.

Red Dwarf. Sorry, you probably have to be British to get this one.
Space 1999. Sorry, you really have to be British and over 40 to get this one … or a major TV space geek.
Voyager. Difficult, interplanetary probes all look similar.
Apollo Lunar Excursion Module (LEM). You can have a point for ‘Lunar Module’, but no, you don’t get a point for ‘Apollo’. Call yourself a geek?
Skylab. The first US space station, made out of old Apollo parts. Not many people get this one. A read a whole book about it, that’s how much of a space geek I am.
Darth Vader’s TIE fighter. You can have a point for ‘TIE Fighter’. You can’t have a point for ‘Star Wars’. Yes yes, I know I’m contradicting myself, but, come on, every geek should know this.
Curiosity. No, no points for ‘Mars Rover’.
2001 A Space Odyssey. Even I don’t know what the ship is called.
Soyuz. It’s been used by the Russians to travel into space since 1966. 46 years! It’s almost as old as me. Odd, when space travel is so synonymous with high-technology, that much of the hardware is actually ancient.

Geek Culture

‘Spooky’ Mulder was the agent in the X-Files, played by actor David Duchovny.
Kiki is the trainee witch in ‘Kiki’s Delivery Service’, one of my favourite anime movies by the outstanding Studio Ghibli.
The actual quote: “Humans are a disease, a cancer of this planet.” by Agent Smith. You can have a point for Virus or Cancer too. Thanks Chris for the link and clarification.
Spiderman of course!
“It’s a Banana” Kryten, of Red Dwarf, learns to lie.
My wife, who is Japanese, translates ‘Otaku’ as ‘geek’. Literally it means ‘you’ and is used to describe someone with obsessive interests. An appropriate question for a geek quiz I think.
The name R2D2 apparently came about when Lucas heard someone ask for Reel 2 Dialog Track 2 in the abbreviated form ‘R-2-D-2’. Later it was said to stand for Second Generation Robotic Droid Series 2, you can have a point for either.
Clarke’s 3rd law states: “Any sufficiently advanced technology is indistinguishable from magic.”
African or European? From Monty Python’s Holy Grail.
Open the pod bay doors please HAL. 2001 A Space Odyssey. Or on acid here.

So there you are. I hope you enjoyed it, and maybe even learnt a little. I certainly did. I might even do it again next year.

A very Merry Christmas to you all!

A Geek Christmas Quiz

noreply@blogger.com (Mike Hadlow) — Fri, 21 Dec 2012 11:36:00 +0000

God rest ye merry gentlemen! Welcome to my 2012 Geek Christmas Quiz. Every Friday morning at 15below we have a ‘DevEd’ session. Usually this is a presentation about some interesting tech, or a new way we want to do something at the company, but today I thought I would try to gauge the true geekiness of our development team with a quiz. The winners, and therefore crowned top geeks, were Toby and Linda who got a total of 32 points. See if you can do better dear reader.

You get one point for each correct answer. The quiz is split into six sections: Computers, ‘Name That Geek’, Science, Space, ‘Name That Spaceship’, and Geek Culture.

Update: The answers are here.

Computers

What does G.N.U. stand for?
What did the A in ARM originally stand for?
What does TCP stand for?
Who founded Microsoft with Bill Gates?
What is F2 (hexadecimal) in decimal?
Which operating system's development was based on the 'Balmer Peak'?
Who was the first programmer?
What year does UNIX time start?
What did SGI stand for?
Write down the type signature of the Monadic Bind method.

Name that Geek

Science

What are the four letters of DNA?
What does the 'c' in E = mc2 stand for?
What is the next number in this sequence: 1 1 2 3 5 8 _ ?
What is C8 H10 N4 02 ?
When did Australopithecus become extinct? (in millions of years ago)
Which of the following would you not expect to find in an atomic nucleus (electron, neutron, proton)
What is the most common gas in the Earth's atmosphere?
Write the formula for Ohm's law.
If, after you fold a piece of paper in half, the ratio between its longest side and its shortest side is the same, what is that ratio?
What living land mammal is the closest evolutionary relative to Whales? (cetaceans)

Space

What rocket engine powered the 2nd stage of the Saturn V?
What is Saturn's largest moon?
What fraction of the Earth's gravity would you experience on the moon?
Astronauts are weightless in space because there is no gravity. true or false?
What is the orbital period of a geosynchronous satellite?
What is the furthest planet from the sun? (now that Pluto has been demoted)
How many people are currently living aboard the ISS?
To the nearest thousand, how many satellites are currently orbiting the earth?
What was the name of the only satellite launched by the UK?
Who was the second man on the moon?

Name that spaceship

Geek Culture

Who was Spooky Mulder?
Was Kiki a trainee witch or an evil princes?
"Humans are a _____" (Agent Smith)
Who is Peter Parker?
"It's a b... It's a b... It's a small, off-duty Czechoslovakian traffic warden!" What is it really?
What does 'Otaku' (Japanese) mean?
What does R2D2 stand for?
What is Clarke's 3rd law?
What is the air speed velocity of an unladen swallow?
Open ___ ___ ___ ____ please H.A.L

My Super Simple Node Twitter Re-Tweeter

noreply@blogger.com (Mike Hadlow) — Tue, 18 Dec 2012 13:18:00 +0000

I’ve been having a lot of fun writing a little ‘re-tweeter’ this morning. We basically want to monitor our user stream and then re-tweet any status with a particular hash tag. I thought this would be an excellent little project for node, and indeed it proved to be extremely easy to do. I used the node-twitter library which worked fine for what I wanted to do.

If you want to use this code, you’ll need to do the following:

First you’ll need to go to https://dev.twitter.com/apps and register a new app. You can then copy and paste your consumer key, consumer secret, access token and access token secret into the ‘xxx’ fields.

Next install node-twitter with npm:

npm install twitter

Then just run the code with node (I’m a poet and I didn’t know it):

node twitter-retweeter.js

Here’s the code in the twitter-retweeter.js file:

var util = require('util');
var twitter = require('twitter');

var twit = new twitter({
    consumer_key: 'xxx',
    consumer_secret: 'xxx',
    access_token_key: 'xxx',
    access_token_secret: 'xxx'
});

var hashtag = '#iloveprog'

function write(data) {
    if ( typeof data === 'string') {
        console.log(data);
    }
    else if (data.text && data.user && data.user.screen_name) {
        console.log(data.user.screen_name + ": " + data.text);
        testForHashtag(data);
    }
    else if (data.delete) {
        console.log('DELETE');
    }
    else if (data.message) {
        console.log('ERROR' + data.message);
    }
    else {
        console.log(util.inspect(data));
    }
}

function testForHashtag(data) {
    if(data.retweeted) return;
    if(data.text.indexOf(hashtag) != -1) {
        twit.retweetStatus(data.id_str, function(){
            console.log('retweet callback');
        });
    }
}

function reconnect() {
    setTimeout(startStreaming, 1000);
}

function startStreaming() {
    twit.stream('user', function(stream) {
        console.log('starting stream');
        stream.on('data', write);
        stream.on('end', reconnect)
    });
}

startStreaming();

console.log('lisening for tweets');

It’s all really straight forward. The startStreaming function kicks of the callback on the twitter user stream. Each time an event occurs it calls the write function which checks for the given hashtag and then retweets the status if there’s a match.

Lovely!

WebRequest Throws On 404 Status Code

noreply@blogger.com (Mike Hadlow) — Wed, 05 Dec 2012 15:38:00 +0000

WebRequest, or rather HttpWebRequest has the annoying behaviour or throwing a WebException when the server returns 404 ‘not found’ status, or in fact any unexpected status number. It would be much better if it didn’t do this and simply allowed the application to decide what it should do on different status codes. At the very least there should be some way of turning this behaviour on or off. In fact it would be nice if the whole WebRequest class wasn’t a monolithic black box, but a toolbox of components that allowed you to tailor an HTTP client to your requirements. I was a little surprised when I did some Googling earlier and couldn’t find a nice open source alternative to WebRequest; it’s the sort of thing that the community is usually quite good at coding around. Oh well, I’ll add that to my ever growing list of potential future GitHub projects (that will never happen).

My quick and dirty fix for this problem was an extension method that catches WebException, checks if the type of the exception is a protocol exception – this seems to be the status for status code exceptions, and then returns the response from the exception’s Response property. It’s horrible, but it seems to work:

public static HttpWebResponse GetHttpResponse(this HttpWebRequest request)
{
    HttpWebResponse response = null;

    try
    {
        response = (HttpWebResponse)request.GetResponse();
    }
    catch (WebException exception)
    {
        if (exception.Status == WebExceptionStatus.ProtocolError)
        {
            response = (HttpWebResponse) exception.Response;
        }
        else
        {
            throw;
        }
    }

    return response;
}

It conveniently returns an HttpWebResponse instead of a WebResponse.

You could use it like this …

var response = request.GetHttpResponse();
if (response.StatusCode != HttpStatusCode.NotFound)
{
    // handle appropriately
}

Of course, if you want to handle the response asynchronously, you’ll have to write an extension method for EndGetResponse as well.

… or, if you’re using .NET 4.5, you could use HttpClient. It wraps WebRequest internally, but it does return status codes rather than throwing it seems.

The Onion Of Compromise

noreply@blogger.com (Mike Hadlow) — Tue, 04 Dec 2012 15:42:00 +0000

I love little phrases that sum up large scale behaviours in software systems and the organisations that produce them. One of my favourite is “The Onion Of Compromise.” I first heard this gem from my excellent friend Iain Holder. Iain doesn’t claim to be the author, that honour goes to a mysterious third person named ‘Mike’.

Being a programmer is all about making decisions. Lots and lots of little decisions. In fact every line of code is a decision; a little cog in the wheel of a grander machine. The simple thing that separates a good programmer from a poor programmer is that they tend to make relatively more good decisions and less bad ones.

Incidentally, that’s why it’s a mistake to think that you can hire an experienced ‘chief architect’ who ‘designs’ your system, while rooms full of junior/cheap developers churn out the code - and expect anything other than a disaster to occur. The decisions are just too granular to be made by one person on a large project.

Good decisions are ones which aid the evolution and stability of an application. They are summed up by epithets that describe general good practice, such as ‘Don’t Repeat Yourself’, ‘Open Closed Principle’ and a hundred others. An experienced programmer will employ a range of these rules-of-thumb to ensure that they don’t get tangled up in needless complexity as their application grows. You can tell a project where good decisions have been made; it’s easy to add new features and there are few bugs.

A bad decision often doesn’t seem like a bad decision at first, merely a way of implementing a feature or fixing a bug with the least possible impact on the code. Often the bad decision will introduce a constraint on further evolution of the software or a special case given a particular combination of factors. If a bad decision isn’t rolled back it can quickly lead to further bad decisions as the programmer works around it. Soon layers of poor design wrap that initial poor decision. This is ‘The Onion of Compromise’. That initial first mistake (or compromise) leads to a cascade of poor choices. Another name for the layers of the onion is ‘Technical Debt’.

It’s easy to spot software that has suffered from The Onion of Compromise; it’s brittle, you change one thing and it breaks seemingly unrelated parts of the system; it seems to take ages to implement new features; and there’s a high bug count.

RabbitMQ On Windows With .NET, A Case Study

noreply@blogger.com (Mike Hadlow) — Mon, 26 Nov 2012 15:20:00 +0000

Any reader of this blog will know that my big project over the last year has been to create a simple .NET API for RabbitMQ called EasyNetQ. I’ve been working as a software architect at 15Below for the last year and a half. The prime motivation for writing EasyNetQ was so that our developers would have an easy API for working with RabbitMQ on .NET. I was very fortunate that, founder and technical director, John Clynes, supported my wish to build it as an open source library. I originally wrote this post for the VMWare blog as a case study of running RabbitMQ in a Microsoft environment.

15Below is based in Brighton on the south coast of England, famous for it’s Regency pavilion and Victorian pier. We provide messaging and integration services for the travel industry. Our clients include Ryanair, Qantas, JetBlue, Thomas Cook and around 30 other airline and rail customers. We send hundreds of millions of transactional notifications each year to our customer’s passengers.

RabbitMQ has helped us to significantly simplify and stabilise our software. It’s one of those black boxes that you install, configure, and then really don’t have to worry about. In over a year of production we’ve found it to be extremely stable and reliable.

Prior to introducing RabbitMQ our applications would use SQL Server as a queuing mechanism. Each task would be represented by a row in a workflow table. Each process in the workflow would poll the table looking for rows that matched its status, process the rows in in a batch, and then update the rows’ status field for the next process to pick up. Each step in the process would be hosted by an application service that implemented its own threading model, often using a different approach to all the other services. This created highly coupled software, with workflow steps and infrastructure concerns, such as threading and load balancing, mixed together with business logic. We also discovered that a relational database is not a natural fit for a queuing system. The contention on the workflow tables is high, with constant inserts, selects and updates causing locking issues. Deleting completed items is also problematic on highly indexed tables and we had considerable problems with continuously growing tables.

I wrote about the ‘Database As Queue Anti-Pattern’ in a previous post in more detail.

RabbitMQ provides a number of features that helped us overcome these problems. Firstly it is designed from the beginning as a high-performance messaging platform. It easily outperformed our SQL Server based solution with none of its locking or deletion problems. Rabbit’s event-oriented messaging model also takes away much of the need for complex multi-threaded batch processing code that was previously a cause of instability in our systems.

We first introduced RabbitMQ about 18 months ago as the core infrastructure behind our Flight Status product. We wanted a high performance messaging product with a proven track record that supported common messaging patterns, such as publish/subscribe and request/response. A further requirement was that it should provide automatic work distribution and load balancing.

The need to support messaging patterns ruled out simple store-and-forward queues such as MSMQ and ActiveMQ. We were very impressed by ZeroMQ, but decided that we really needed the centralised manageability of a broker based product. This left RabbitMQ. Although support for AMQP, an open messaging standard, wasn’t in our list of requirements, its implementation by RabbitMQ made us more confident that we were choosing a sustainable strategy.

We are very much a Microsoft shop, so we had some initial concerns about RabbitMQ’s performance and stability on Windows. We were reassured by reading some accounts of RabbitMQ’s and indeed Erlang’s use on Windows by organisations with some very impressive load requirements. Subsequent experience has borne these reports out, and we have found RabbitMQ on Server 2008 to be rock solid.

As a Microsoft shop, our development platform is .NET. Although VMWare provide an AMQP C# client, it is a low-level API, not suitable for use by more junior developers. For this reason we created our own high-level .NET API for RabbitMQ that provides simple single method access to common messaging patterns and does not require a deep knowledge of AMQP. This API is called EasyNetQ. We’ve open sourced it and, with over 3000 downloads, it is now the leading high-level API for RabbitMQ with .NET. You can find more information about it at EasyNetQ.com. We would recommend looking at it if you are a .NET shop using RabbitMQ.

15Below’s Flight-Status product provides real-time flight information to passengers and their family and friends. We interface with the airline’s real-time flight information stream generated from their operation systems and provide a platform that allows them to apply complex business logic to this stream. We render customer tailored output, and communicate with the airline’s customers via a range of channels, including email, SMS, voice and iPhone/Android push. RabbitMQ allows us to build each piece; the client for the fight information stream, the message renderer, the sink channels and the business logic; as separate components that communicate using structured messages. Our architecture looks something like this:

The green boxes are our core product systems, the blue boxes represent custom code that we write for each customer. A ‘customer saga’ is code that models a long-running business process and includes all the workflow logic for a particular customer’s flight information requirements. A ‘core product service’ is an independent service that implements a feature of our product. An example would be the service that takes flight information and combines it with a customer defined template to create an email to be sent to a passenger. Constructing services as independently deployable and runnable applications gives us great flexibility and scalability. If we need to scale up a particular component, we simply install more copies. RabbitMQ’s automatic work sharing feature means that we can do this without any reconfiguration of existing components. This architecture also makes it easy to test each application service in isolation since it’s simply a question of firing messages at the service and watching its response.

In conclusion, RabbitMQ has provided a rock solid piece of infrastructure with the features to allow us to significantly reduce the architectural complexity of our systems. We can now build software for our clients faster and more reliably. It scales to higher loads than our previous relational-database based systems and is more flexible in the face of changing customer requirements.

Using BlockingCollection To Communicate Between Threads

noreply@blogger.com (Mike Hadlow) — Fri, 23 Nov 2012 15:56:00 +0000

Consider these (somewhat) common programming challenges:

I’m using a third party library that is not thread safe, but I want my application to share work between multiple threads. How do I marshal calls between my multi-threaded code to the single threaded library?
I have a single source of events on a single thread, but I want to share the work between a pool of multiple threads?
I have multiple threads emitting events, but I want to consume them on a single thread?

One way of doing this would be to have some shared state, a field or a property on a static class, and wrap locks around it so that multiple threads can access it safely. This is a pretty common way of trying to skin this particular cat, but it’s shot through with traps for the unwary. Also, it can hurt performance because access to the shared resource is serialized, even though the things accessing it are running in parallel.

A better way is to use a BlockingCollection and have your threads communicate via message classes.

BlockingCollection is a class in the new System.Collections.Concurrent namespace that arrived with .NET 4.0. It contains a ConcurrentQueue, although you can swap this for a ConcurrentStack or a ConcurrentBag if you want. You push objects in at one end and sit in a loop consuming them from the other. The (multiple) producer(s) and (multiple) consumer(s) can be running on different threads without any locks. That’s OK because the Concurrent namespace collection classes are guaranteed to be thread safe. The ‘blocking’ part of the name is there because the consuming end blocks until an object is available. Justin Etheredge has an excellent post that looks at BlockingCollection in more detail here.

For an example, let’s implement a parallel pipeline. A ventilator produces tasks to be processed in parallel, a set of workers process the tasks on separate threads, and a sink collects the results back together again. It shows both one-to-many and many-to-one thread communication. I’ve stolen the idea and the diagram from the excellent ZeroMQ Guide:

First we’ll need a class that represents a piece of work, we’ll keep it super simple for this example:

public class WorkItem
{
    public string Text { get; set; }
}

We’ll need two BlockingCollections, one to take the tasks from the ventilator to the workers, and another to take the finished work from the workers to the sink:

var ventilatorQueue = new BlockingCollection<WorkItem>();
var sinkQueue = new BlockingCollection<WorkItem>();

Now let’s write our ventilator:

public static void StartVentilator(BlockingCollection<WorkItem> ventilatorQueue)
{
    Task.Factory.StartNew(() =>
    {
        for (int i = 0; i < 100; i++)
        {
            ventilatorQueue.Add(new WorkItem { Text = string.Format("Item {0}", i) });
        }
    }, TaskCreationOptions.LongRunning);
}

It just iterates 100 times creating work items and pushing them on the ventilatorQueue.

Here is a worker:

public static void StartWorker(int workerNumber,
    BlockingCollection<WorkItem> ventilatorQueue,
    BlockingCollection<WorkItem> sinkQueue)
{
    Task.Factory.StartNew(() =>
    {
        foreach (var workItem in ventilatorQueue.GetConsumingEnumerable())
        {
            // pretend to take some time to process
            Thread.Sleep(30);
            workItem.Text = workItem.Text + " processed by worker " + workerNumber;
            sinkQueue.Add(workItem);
        }
    }, TaskCreationOptions.LongRunning);
}

BlockingCollection provides a GetConsumingEnumerable method that yields each item in turn. It blocks if there are no items on the queue. Note that I’m not worrying about shutdown patterns in this code. In production code you’d need to worry about how to close down your worker threads.

Next let’s write our sink:

public static void StartSink(BlockingCollection<WorkItem> sinkQueue)
{
    Task.Factory.StartNew(() =>
    {
        foreach (var workItem in sinkQueue.GetConsumingEnumerable())
        {
            Console.WriteLine("Processed Messsage: {0}", workItem.Text);
        }
    }, TaskCreationOptions.LongRunning);
}

Once again, this sits in an infinite foreach loop consuming items from the sinkQueue.

Finally we need to wire up the pieces and kick it off:

StartSink(sinkQueue);

StartWorker(0, ventilatorQueue, sinkQueue);
StartWorker(1, ventilatorQueue, sinkQueue);
StartWorker(2, ventilatorQueue, sinkQueue);

StartVentilator(ventilatorQueue);

I’ve started the sink first, then the workers and finally the producer. It doesn’t overly matter what order they start in since the queues will store any tasks the ventilator creates before the workers and the sink start.

Running the code I get output something like this:

Processed Messsage: Item 1 processed by worker 1
Processed Messsage: Item 2 processed by worker 0
Processed Messsage: Item 0 processed by worker 2
Processed Messsage: Item 5 processed by worker 2
Processed Messsage: Item 3 processed by worker 1

....

Processed Messsage: Item 95 processed by worker 0
Processed Messsage: Item 98 processed by worker 0
Processed Messsage: Item 97 processed by worker 2
Processed Messsage: Item 96 processed by worker 1
Processed Messsage: Item 99 processed by worker 0

This pattern is a great way of decoupling the communication between a source and a sink, or a producer and a consumer. It also allows you to have multiple sources and multiple sinks, but primarily it’s a safe way for multiple threads to interact.

The complete example is here on GitHub.

A C# .NET Client Proxy For The RabbitMQ Management API

noreply@blogger.com (Mike Hadlow) — Thu, 15 Nov 2012 16:31:00 +0000

RabbitMQ comes with a very nice Management UI and a HTTP JSON API, that allows you to configure and monitor your RabbitMQ broker. From the website:

“The rabbitmq-management plugin provides an HTTP-based API for management and monitoring of your RabbitMQ server, along with a browser-based UI and a command line tool, rabbitmqadmin. Features include:

Declare, list and delete exchanges, queues, bindings, users, virtual hosts and permissions.

Monitor queue length, message rates globally and per channel, data rates per connection, etc.

Send and receive messages.

Monitor Erlang processes, file descriptors, memory use.

Export / import object definitions to JSON.

Force close connections, purge queues.”

Wouldn’t it be cool if you could do all these management tasks from your .NET code? Well now you can. I’ve just added a new project to EasyNetQ called EasyNetQ.Management.Client. This is a .NET client-side proxy for the HTTP-based API.

It’s on NuGet, so to install it, you simply run:

PM> Install-Package EasyNetQ.Management.Client

To give an overview of the sort of things you can do with EasyNetQ.Client.Management, have a look at this code. It first creates a new Virtual Host and a User, and gives the User permissions on the Virtual Host. Then it re-connects as the new user, creates an exchange and a queue, binds them, and publishes a message to the exchange. Finally it gets the first message from the queue and outputs it to the console.

var initial = new ManagementClient("http://localhost", "guest", "guest");

// first create a new virtual host
var vhost = initial.CreateVirtualHost("my_virtual_host");

// next create a user for that virutal host
var user = initial.CreateUser(new UserInfo("mike", "topSecret"));

// give the new user all permissions on the virtual host
initial.CreatePermission(new PermissionInfo(user, vhost));

// now log in again as the new user
var management = new ManagementClient("http://localhost", user.name, "topSecret");

// test that everything's OK
management.IsAlive(vhost);

// create an exchange
var exchange = management.CreateExchange(new ExchangeInfo("my_exchagne", "direct"), vhost);

// create a queue
var queue = management.CreateQueue(new QueueInfo("my_queue"), vhost);

// bind the exchange to the queue
management.CreateBinding(exchange, queue, new BindingInfo("my_routing_key"));

// publish a test message
management.Publish(exchange, new PublishInfo("my_routing_key", "Hello World!"));

// get any messages on the queue
var messages = management.GetMessagesFromQueue(queue, new GetMessagesCriteria(1, false));

foreach (var message in messages)
{
    Console.Out.WriteLine("message.payload = {0}", message.payload);
}

This library is also ideal for monitoring queue levels, channels and connections on your RabbitMQ broker. For example, this code prints out details of all the current connections to the RabbitMQ broker:

var connections = managementClient.GetConnections();

foreach (var connection in connections)
{
    Console.Out.WriteLine("connection.name = {0}", connection.name);
    Console.WriteLine("user:\t{0}", connection.client_properties.user);
    Console.WriteLine("application:\t{0}", connection.client_properties.application);
    Console.WriteLine("client_api:\t{0}", connection.client_properties.client_api);
    Console.WriteLine("application_location:\t{0}", connection.client_properties.application_location);
    Console.WriteLine("connected:\t{0}", connection.client_properties.connected);
    Console.WriteLine("easynetq_version:\t{0}", connection.client_properties.easynetq_version);
    Console.WriteLine("machine_name:\t{0}", connection.client_properties.machine_name);
}

On my machine, with one consumer running it outputs this:

connection.name = [::1]:64754 -> [::1]:5672
user:   guest
application:    EasyNetQ.Tests.Performance.Consumer.exe
client_api: EasyNetQ
application_location:   D:\Source\EasyNetQ\Source\EasyNetQ.Tests.Performance.Consumer\bin\Debug
connected:  14/11/2012 15:06:19
easynetq_version:   0.9.0.0
machine_name:   THOMAS

You can see the name of the application that’s making the connection, the machine it’s running on and even its location on disk. That’s rather nice. From this information it wouldn’t be too hard to auto-generate a complete system diagram of your distributed messaging application. Now there’s an idea :)

For more information, check out the documentation.

Nicer Client Properties For EasyNetQ

noreply@blogger.com (Mike Hadlow) — Mon, 12 Nov 2012 17:05:00 +0000

EasyNetQ is my lightweight easy-to-use .NET API for RabbitMQ.

Today I added a small but very nice feature, better client properties. Now when you look at connections created by EasyNetQ you can see the machine that connected, the application and the application’s location on disk. It also gives you the date and time that EasyNetQ first connected. Very useful for debugging.

Here’s an example. Check out the ‘Client Properties’ section.

EasyNetQ Publisher Confirms

noreply@blogger.com (Mike Hadlow) — Tue, 06 Nov 2012 14:47:00 +0000

EasyNetQ is my easy-to-use .NET API for RabbitMQ.

The default AMQP publish is not transactional and doesn't guarantee that your message will actually reach the broker. AMQP does specify a transactional publish, but with RabbitMQ it is extremely slow, around 200 slower than a non-transactional publish, and we haven't supported it via the EasyNetQ API. For high-performance guaranteed delivery it's recommended that you use 'Publisher Confirms'. Simply speaking, this an extension to AMQP that provides a call-back when your message has been successfully received by the broker.

What does 'successfully received' mean? It depends ...

A transient message is confirmed the moment it is enqueued.
A persistent message is confirmed as soon as it is persisted to disk, or when it is consumed on every queue.
An unroutable transient message is confirmed directly it is published.

For more information on publisher confirms, please read the announcement on the RabbitMQ blog.

To use publisher confirms, you must first create the publish channel with publisher confirms on:

var channel = bus.OpenPublishChannel(x => x.WithPublisherConfirms())

Next you must specify success and failure callbacks when you publish your message:

channel.Publish(message, x => 
    x.OnSuccess(() =>
    {
        // do success processing here
    })
    .OnFailure(() => 
    {
        // do failure processing here
    }));

Be careful not to dispose the publish channel before your call-backs have had a chance to execute.

Here's an example of a simple test. We're publishing 10,000 messages and then waiting for them all to be acknowledged before disposing the channel. There's a timeout, so if the batch takes longer than 10 seconds we abort with an exception.

const int batchSize = 10000;
var callbackCount = 0;
var stopwatch = new Stopwatch();
stopwatch.Start();

using (var channel = bus.OpenPublishChannel(x => x.WithPublisherConfirms()))
{
    for (int i = 0; i < batchSize; i++)
    {
        var message = new MyMessage {Text = string.Format("Hello Message {0}", i)};
        channel.Publish(message, x => 
            x.OnSuccess(() => {
                callbackCount++;
            })
            .OnFailure(() =>
            {
                callbackCount++;
            }));
    }

    // wait until all the publications have been acknowleged.
    while (callbackCount < batchSize)
    {
        if (stopwatch.Elapsed.Seconds > 10)
        {
            throw new ApplicationException("Aborted batch with timeout");
        }
        Thread.Sleep(10);
    }
}

A Functional IoC Container

noreply@blogger.com (Mike Hadlow) — Wed, 10 Oct 2012 15:24:00 +0000

Today I was idly thinking about an idea I had a couple of years ago for a functional IoC container. I’d had a go at implementing such a beast, but soon got bogged down in a tangled mess of spaghetti reflection code and gave it up as too much bother. But today it suddenly occurred to me that there was no need for any reflection voodoo; the C# type system is powerful enough to do all the work for us.

In object oriented programming languages we build programs from classes. Classes declare the contract(s) they support with interfaces and declare their dependencies with constructor arguments. We use an IoC container to wire instances of our classes together to make a running program.

Pure functional languages, like Haskell, don’t have any concept of class, instead they use currying and partial application to compose hierarchies of functions.

Here’s an example of a purely functional program written in C#.

public static class Module
{
    public static Data GetAndTransform(Func<Input,Data> dataAccsessor, Func<Data,Data> transformer, int id)
    {
        var input = new Input() {Id = id};
        var data = dataAccsessor(input);
        var transformed = transformer(data);
        return transformed;
    }

    public static Data DataAccsessor(Input input)
    {
        return new Data
        {
            Id = input.Id,
            Name = "Test"
        };
    }

    public static Data Transformer(Data original)
    {
        original.Name = original.Name + " transformed";
        return original;
    }
}

GetAndTransform simply takes an int id argument, does some work, and then returns some data. It needs a dataAccsessor and a transformer in order to do its job.

C# doesn’t support currying or partial application, so in order to run it we have to compose the program and execute it all in one step. For example:

var id = 10;
var data = Module.GetAndTransform(Module.DataAccsessor, Module.Transformer, id);

Console.Out.WriteLine("data.Id = {0}", data.Id);
Console.Out.WriteLine("data.Name = {0}", data.Name);

But what if we had a ‘currying container’, one that could compose the program in one step and then return a function for us to execute in another? Here is such a container at work with our program:

var registration = new Container()
    .Register<Func<Input, Data>, Func<Data, Data>, int, Data>(Module.GetAndTransform)
    .Register<Input,Data>(Module.DataAccsessor)
    .Register<Data,Data>(Module.Transformer);

var main = registration.Get<Func<int, Data>>();

var data = main(10);

Console.Out.WriteLine("data.Id = {0}", data.Id);
Console.Out.WriteLine("data.Name = {0}", data.Name);

In the first line, we create a new instance of our container. On the next three lines we register our functions. Unfortunately C#’s type inference isn’t powerful enough to let us do away with the tedious type annotations; we have to explicitly declare the argument and return types of each of our functions.

Once our functions are registered we can ask the container for a program (main) that takes an int and returns a Data instance. The container works out that it needs to curry GetAndTransform and then partially apply DataAccsessor and Transformer to it to produce the desired function.

We can then run our ‘main’ function which gives us our expected output:

data.Id = 10
data.Name = Test transformed

The container turns out to be very simple, just a dictionary that’s keyed by type and contains a collection of constructor functions that know how to build the target (key) type.

public interface IRegistration
{
    void Add(Type target, Func<object> constructor);
    T Get<T>();
}

public class Container : IRegistration
{
    private readonly Dictionary<Type, Func<object>> registrations = new Dictionary<Type, Func<object>>();

    public void Add(Type target,  Func<object> constructor)
    {
        registrations.Add(target, constructor);
    }

    public T Get<T>()
    {
        return (T)registrations[typeof (T)]();
    }
}

The magic sauce is in the Registration function overloads. If you take the standard functional idea that a function should only have one argument and one return type, you can take any input function, curry it, and then partially apply arguments until you are left with a Func<X,Y>. So you know what the ‘target’ type of each function should be, a function from the last argument to the return type. A Func<A,B,C,R> gets resolved to a Func<C,R>. There’s no need to explicitly register a target, it’s implicit from the type of the provided function:

public static class RegistrationExtensions
{
    public static IRegistration Register<A,R>(this IRegistration registration, Func<A, R> source)
    {
        var targetType = typeof (Func<A, R>);
        var curried = Functional.Curry(source);

        registration.Add(targetType, () => curried);

        return registration;
    }

    public static IRegistration Register<A,B,R>(this IRegistration registration, Func<A, B, R> source)
    {
        var targetType = typeof (Func<B, R>);
        var curried = Functional.Curry(source);

        registration.Add(targetType, () => curried(
            registration.Get<A>()
            ));

        return registration;
    }

    public static IRegistration Register<A, B, C, R>(this IRegistration registration, Func<A, B, C, R> source)
    {
        var targetType = typeof(Func<C, R>);
        var curried = Functional.Curry(source);

        registration.Add(targetType, () => curried(
            registration.Get<A>()
            )
            (
            registration.Get<B>()
            ));

        return registration;
    }
}

Each overload deals with an input function with a different number of arguments. My simple experiment only works with functions with up to three arguments (two dependencies and an input type), but it would be easy to extend for higher numbers. The Curry function is stolen from Oliver Sturm and looks like this:

public static class Functional
{
    public static Func<A, R> Curry<A, R>(Func<A, R> input)
    {
        return input;
    }

    public static Func<A, Func<B, R>> Curry<A, B, R>(Func<A, B, R> input)
    {
        return a => b => input(a, b);
    }

    public static Func<A, Func<B, Func<C,R>>> Curry<A, B, C, R>(Func<A, B, C, R> input)
    {
        return a => b => c => input(a, b, c);
    }
}

Rather nice, even if I say so myself.

Of course this little experiment has many limitations. For a start it only understands functions in terms of Func< … >, so you can’t have more than one function of each ‘type’. You couldn’t have two Func<int,int> for example, which might be somewhat limiting.

The code is on GitHub here if you want to have a play.

EasyNetQ Cluster Support

noreply@blogger.com (Mike Hadlow) — Wed, 03 Oct 2012 11:07:00 +0000

EasyNetQ, my super simple .NET API for RabbitMQ, now (from version 0.7.2.34) supports RabbitMQ clusters without any need to deploy a load balancer.

Simply list the nodes of the cluster in the connection string ...

var bus = RabbitHutch.CreateBus("host=ubuntu:5672,ubuntu:5673");

In this example I have set up a cluster on a single machine, 'ubuntu', with node 1 on port 5672 and node 2 on port 5673. When the CreateBus statement executes, EasyNetQ will attempt to connect to the first host listed (ubuntu:5672). If it fails to connect it will attempt to connect to the second host listed (ubuntu:5673). If neither node is available it will sit in a re-try loop attempting to connect to both servers every five seconds. It logs all this activity to the registered IEasyNetQLogger. You might see something like this if the first node was unavailable:

DEBUG: Trying to connect
ERROR: Failed to connect to Broker: 'ubuntu', Port: 5672 VHost: '/'. ExceptionMessage: 'None of the specified endpoints were reachable'
DEBUG: OnConnected event fired
INFO: Connected to RabbitMQ. Broker: 'ubuntu', Port: 5674, VHost: '/'

If the node that EasyNetQ is connected to fails, EasyNetQ will attempt to connect to the next listed node. Once connected, it will re-declare all the exchanges and queues and re-start all the consumers. Here's an example log record showing one node failing then EasyNetQ connecting to the other node and recreating the subscribers:

INFO: Disconnected from RabbitMQ Broker
DEBUG: Trying to connect
DEBUG: OnConnected event fired
DEBUG: Re-creating subscribers
INFO: Connected to RabbitMQ. Broker: 'ubuntu', Port: 5674, VHost: '/'

You get automatic fail-over out of the box. That’s pretty cool.

If you have multiple services using EasyNetQ to connect to a RabbitMQ cluster, they will all initially connect to the first listed node in their respective connection strings. For this reason the EasyNetQ cluster support is not really suitable for load balancing high throughput systems. I would recommend that you use a dedicated hardware or software load balancer instead, if that’s what you want.

Parsing a Connection String With Sprache

noreply@blogger.com (Mike Hadlow) — Fri, 28 Sep 2012 15:30:00 +0000

Sprache is a very cool lightweight parser library for C#. Today I was experimenting with parsing EasyNetQ connection strings, so I thought I’d have a go at getting Sprache to do it. An EasyNetQ connection string is a list of key-value pairs like this:

key1=value1;key2=value2;key3=value3

The motivation for looking at something more sophisticated than simply chopping strings based on delimiters, is that I’m thinking of having more complex values that would themselves need parsing. But that’s for the future, today I’m just going to parse a simple connection string where the values can be strings or numbers (ushort to be exact).

So, I want to parse a connection string that looks like this:

virtualHost=Copa;username=Copa;host=192.168.1.1;password=abc_xyz;port=12345;requestedHeartbeat=3

… into a strongly typed structure like this:

public class ConnectionConfiguration : IConnectionConfiguration
{
    public string Host { get; set; }
    public ushort Port { get; set; }
    public string VirtualHost { get; set; }
    public string UserName { get; set; }
    public string Password { get; set; }
    public ushort RequestedHeartbeat { get; set; }
}

I want it to be as easy as possible to add new connection string items.

First let’s define a name for a function that updates a ConnectionConfiguration. A uncommonly used version of the ‘using’ statement allows us to give a short name to a complex type:

using UpdateConfiguration = Func<ConnectionConfiguration, ConnectionConfiguration>;

Now lets define a little function that creates a Sprache parser for a key value pair. We supply the key and a parser for the value and get back a parser that can update the ConnectionConfiguration.

public static Parser<UpdateConfiguration> BuildKeyValueParser<T>(
    string keyName,
    Parser<T> valueParser,
    Expression<Func<ConnectionConfiguration, T>> getter)
{
    return
        from key in Parse.String(keyName).Token()
        from separator in Parse.Char('=')
        from value in valueParser
        select (Func<ConnectionConfiguration, ConnectionConfiguration>)(c =>
        {
            CreateSetter(getter)(c, value);
            return c;
        });
}

The CreateSetter is a little function that turns a property expression (like x => x.Name) into an Action<TTarget, TProperty>.

Next let’s define parsers for string and number values:

public static Parser<string> Text = Parse.CharExcept(';').Many().Text();
public static Parser<ushort> Number = Parse.Number.Select(ushort.Parse);

Now we can chain a series of BuildKeyValueParser invocations and Or them together so that we can parse any of our expected key-values:

public static Parser<UpdateConfiguration> Part = new List<Parser<UpdateConfiguration>>
{
    BuildKeyValueParser("host", Text, c => c.Host),
    BuildKeyValueParser("port", Number, c => c.Port),
    BuildKeyValueParser("virtualHost", Text, c => c.VirtualHost),
    BuildKeyValueParser("requestedHeartbeat", Number, c => c.RequestedHeartbeat),
    BuildKeyValueParser("username", Text, c => c.UserName),
    BuildKeyValueParser("password", Text, c => c.Password),
}.Aggregate((a, b) => a.Or(b));

Each invocation of BuildKeyValueParser defines an expected key-value pair of our connection string. We just give the key name, the parser that understands the value, and the property on ConnectionConfiguration that we want to update. In effect we’ve defined a little DSL for connection strings. If I want to add a new connection string value, I simply add a new property to ConnectionConfiguration and a single line to the above code.

Now lets define a parser for the entire string, by saying that we’ll parse any number of key-value parts:

public static Parser<IEnumerable<UpdateConfiguration>> ConnectionStringBuilder =
    from first in Part
    from rest in Parse.Char(';').Then(_ => Part).Many()
    select Cons(first, rest);

All we have to do now is parse the connection string and apply the chain of update functions to a ConnectionConfiguration instance:

public IConnectionConfiguration Parse(string connectionString)
{
    var updater = ConnectionStringGrammar.ConnectionStringBuilder.Parse(connectionString);
    return updater.Aggregate(new ConnectionConfiguration(), (current, updateFunction) => updateFunction(current));
}

We get lots of nice things out of the box with Sprache, one of the best is the excellent error messages:

Parsing failure: unexpected 'x'; expected host or port or virtualHost or requestedHeartbeat or username or password (Line 1, Column 1).

Sprache is really nice for this kind of task. I’d recommend checking it out.

Replacing EasyNetQ Components

noreply@blogger.com (Mike Hadlow) — Tue, 25 Sep 2012 12:50:00 +0000

EasyNetQ, my simple .NET API for RabbitMQ, is a library composed of small components. Until today, the code simply wired the components in a messy hard-coded routine. Now it has its own tiny internal IoC container. When you write:

var bus = RabbitHutch.CreateBus("host=localhost");

... the static method CreateBus registers the components with the container and then resolves the IBus instance. The really cool thing about this is that it allows you, the user, to replace any of the internal components, including IBus, with your own implementations. An overload of the CreateBus method provides the hook which gives you access to the component registration. The signature looks like this:

public static IBus CreateBus(string connectionString, Action<IServiceRegister> registerServices)

The IServiceRegister interface provides a single method:

public interface IServiceRegister
{
    IServiceRegister Register<TService>(Func<IServiceProvider, TService> serviceCreator) where TService : class;
}

So to register your own logger, based on IEasyNetQLogger, you'd write this code:

var logger = new MyLogger(); // MyLogger implements IEasyNetQLogger
var bus = RabbitHutch.CreateBus(connectionString, 
    serviceRegister => serviceRegister.Register(serviceProvider => logger));

The Register method's argument, Func<IServiceProvider, TService>, is a function that's run when CreateBus pulls together the components to make an IBus instance. IServiceProvider looks like this:

public interface IServiceProvider
{
    TService Resolve<TService>() where TService : class;
}

This allows you to access other services that EasyNetQ provides. If for example you wanted to replace the default serializer with your own implementation of ISerializer, and you wanted to construct it with a reference to the internal logger, you could do this:

var bus = RabbitHutch.CreateBus(connectionString, serviceRegister => serviceRegister.Register(
    serviceProvider => new MySerializer(serviceProvider.Resolve<IEasyNetQLogger>())));

There’s nothing to stop you registering your own interfaces with the container that you can then use with your implementations of EasyNetQ’s service interfaces.

To see the complete list of components that make up the IBus instance, and how they are assembled, take a look at the ComponentRegistration class.