Did somebody say code?

The past two posts began to explain how to build a service gateway using MassTransit. In this post, I’m going to share some of the initial code that makes up the gateway service. The gateway itself consists of two components. The first implements the communication to the external service with a set of messages that are only used internally by the service. The second is the saga that provides the interface to the service gateway.

Service Contract

The interface exposed to the application consists of two messages, the first for the command and the second for the response. The message contracts are defined using interfaces, allowing the class for the message to be an internal implementation detail.

The message contract representing a request for order details includes the CustomerId and the OrderId.

public interface RetrieveOrderDetails
{
	string OrderId { get; }
	string CustomerId { get; }
}

When the order details are received, the following message is published.

public interface OrderDetailsReceived
{
	string OrderId { get; }
	string CustomerId { get; }
	DateTime Created { get; }
	OrderStatus Status { get; }
}

public enum OrderStatus
{
	Unknown = 0,
	Submitted = 1,
	Accepted = 2,
	InProcess = 3,
	Complete = 4,
}

The CustomerId and OrderId are the same as the values passed in the request. Created is when the order was created, and Status is an enum representing the status of the order.

Notice that no internal values are included — no primary key from the order table and no primary key from the customer table. The request and response are correlated on identifiers that make sense in the application domain. While SQL purists will point out that numeric primary keys are quicker for retrieving rows in a database, they make for a very fragile interface with other components in the system. Reliance on a primary key outside of the context of the system storing the order details is a path to friction or outright failure.

Time To Make The Saga

At this point in the design of our service, the need for a saga to manage the request state is not entirely obvious. While the TDD purists might want to call YAGNI at this point, let me assure you that “it will all be… revealed!” So for now, let us take a look at the first pass of our saga definition.

public class OrderDetailsRequestSaga :
	SagaStateMachine<OrderDetailsRequestSaga>,
	ISaga
{
	static OrderDetailsRequestSaga()
	{
		Define(Saga);
	}

Our saga state machine uses a static initializer to define the states, events, and transitions of a saga. The previous code merely defines our class as a saga and calls our saga initialization method (shown below).

	private static void Saga()
	{
		Correlate(RequestReceived)
			.By((saga, message) => saga.CustomerId == message.CustomerId &&
			                       saga.OrderId == message.OrderId &&
			                       saga.CurrentState == WaitingForResponse);

		Correlate(ResponseReceived)
			.By((saga, message) => saga.CustomerId == message.CustomerId &&
			                       saga.OrderId == message.OrderId &&
			                       saga.CurrentState == WaitingForResponse);

Since our request criteria include our customer id and our order id, we use those to correlate the message to the saga. We also include the state of the saga to ensure that we do not match to a request that has already completed. We will look at some other ways we can enhance the performance of the service later on by using some additional states.

	public static State Initial { get; set; }
	public static State WaitingForResponse { get; set; }
	public static State Completed { get; set; }

The three states we have defined, including an initial state when a new saga instance is created, a waiting for response state one our request has been sent to the service, and a completed state once the response has been received and published.

	public static Event<RetrieveOrderDetails> RequestReceived { get; set; }
	public static Event<OrderDetailsResponse> ResponseReceived { get; set; }
	public static Event<OrderDetailsRequestFailed> RequestFailed { get; set; }

The three events that we have defined, including the message contract that maps to the event. The subscription logic for the saga will automatically map message handlers for these events that will invoke the actions depending upon the current state of the saga.

		Initially(
			When(RequestReceived)
				.Then((saga, request) =>
					{
						saga.OrderId = request.OrderId;
						saga.CustomerId = request.CustomerId;
					})
				.Publish((saga, request) => new SendOrderDetailsRequest
					{
						RequestId = saga.CorrelationId,
						CustomerId = saga.CustomerId,
						OrderId = saga.OrderId,
					})
				.TransitionTo(WaitingForResponse));

The first event handler, RequestReceived, is invoked when the saga is created in response to the RetrieveOrderDetails message. The handler copies the properties of the request, and then publishes the request message to the proxy that will call the external web service. After the message is published, the state of the saga transitions to the waiting for response state. When using transactional queues, the receipt of the message, creation of the saga in the database, sending of the command message to the proxy, and saving the saga are all part of a single distributed transaction. This ensures that everything completes as a single operation to ensure no requests are lost.

		During(WaitingForResponse,
			When(ResponseReceived)
				.Then((saga, response) =>
					{
						saga.OrderCreated = response.Created;
						saga.OrderStatus = response.Status;
					})
				.Publish((saga, request) => new OrderDetails
					{
						CustomerId = saga.CustomerId,
						OrderId = saga.OrderId,
						Created = saga.OrderCreated.Value,
						Status = saga.OrderStatus,
					})
				.TransitionTo(Completed));
	}

The second event handler, ResponseReceived, is invoked when the OrderDetailsResponse message is received. The results of the request are stored in the saga and a message is published containing the details of the order back to the original requestor. Another approach would be to capture the requestor address (via the ResponseAddress header from the original message) and then resolve that address using the endpoint factory to send the response directly to the requestor. I don’t really encourage this approach without having a truly unique identifier for each request.

	public OrderDetailsRequestSaga(Guid correlationId)
	{
		CorrelationId = correlationId;
	}

	protected OrderDetailsRequestSaga()
	{
	}

	public virtual string CustomerId { get; set; }
	public virtual string OrderId { get; set; }
	public virtual OrderStatus OrderStatus { get; set; }
	public virtual DateTime? OrderCreated { get; set; }

	public virtual Guid CorrelationId { get; set; }
	public virtual IServiceBus Bus { get; set; }
}

The rest of the saga class is shown above for completeness. The properties are part of the saga and get saved when the saga is persisted (using the NHibernate saga persister, or in the case of the sample the in-memory implementation). The constructor with the Guid is used to initialize the saga when a new one is created, the protected one is there for NHibernate to be able to persist the saga.

The Service Proxy

The saga uses the external service proxy to perform the actual work, which is shown in the proxy class below.

public class OrderDetailsWebServiceProxy :
	Consumes<SendOrderDetailsRequest>.All
{
	public void Consume(SendOrderDetailsRequest request)
	{
		// make the call to the service to get the order details here

		var details = new OrderDetailsResponse
			{
				OrderId = request.OrderId,
				CustomerId = request.CustomerId,
				Created = (-1).Days().FromUtcNow(),
				Status = OrderStatus.InProcess,
			};

		CurrentMessage.Respond(details, x => x.ExpiresAt(5.Minutes().FromNow()));
	}
}

The message handler uses the criteria from the SendOrderDetailsRequest message (which was published by the saga) to call the external service and retrieve the order details. The details are then returned to the saga in the form of an OrderDetailsResponse message which is internal to the service (and therefore not part of the interface assembly that is provided to applications that want to use the order details service).

Test That Thang

Now that our saga has been developed, we need to be able to test it. A unit test will be created that creates a testing instance of the service bus (see the sample for the implementation details) and verifies that the saga responds properly to the request. To request order details, a very simple client would subscribe to the response message and then publish the request.

const string orderId = "ABC123";
const string customerId = "12345";

LocalBus.Subscribe<OrderDetailsReceived>(message =>
	{
		response.Set(message);
	},
	x => x.OrderId == orderId && x.CustomerId == customerId);

RetrieveOrderDetails request = new RetrieveOrderDetailsRequest(customerId, orderId);
LocalBus.Publish(request, x => x.SendResponseTo(LocalBus.Endpoint));

The subscribe method used above specifies that when a message of type OrderDetailsReceived is received, if the contents of the message match the predicate specified (which in this case, is checking the OrderId and CustomerId contained in the message) then the statement specified should be called. Our example is from the integration test (built using NUnit) that verifies the service performs from end-to-end.

The syntax above is functional, and it will work, but it does not represent the most scalable approach. In the next post, I’ll start to explain how to build a much more scalable method of handling thousands of concurrent requests on a single machine using IIS.

In the meantime, the sample code is available in the MassTransit trunk as a standalone solution. You can find it in the trunk\src\Samples\ServiceGatewaySample folder. There are unit tests that verify the calling syntax shown above and a service that hosts the services (both the saga, and the proxy service).

Continued…

In part one, I discussed two exchange patterns that are often exposed as a web service. In this installment, I’m going to cover a more complex exchange pattern that including makes a request to an external system in response to a request on our web service.

One of the comments on the previous post raised the question of using messaging for queries. Udi, Ayende, and I agree that this so-called “Data-SOA” is a bad thing. The queries I am presenting in this article are made against an external service and not an internal database. In many cases, this service might have limited availability and limited throughput, making it important to isolate our service to increase availability.

Complex Request

In the simple request, order status is a small enough data point that caching it at the web service makes sense. The details of an order, however, are much more involved and it is not practical to keep the details cached in case they are requested (which in this case, is much less often that the order status). To support the order detail request, we will apply a pattern that separates the inbound request from the outbound request, ensures that requests are performed only once (minimizing the click, click, click refresh mentality of some users), and even retain requests if the outbound service is unavailable until the request expires or the outbound service becomes available.

The first thing we want to build is the service that will be responsible for calling the external web service. To start, we’ll define two messages representing the happy path of calling the service. The first message contains the criteria of the request (such as the order id and perhaps the customer id) and the second message contains the details of the order that were returned by the web service. In addition to the messages on the happy path, we may also define additional messages used to publish exception information related to the request.

Once the message contracts are defined to call the web service, we need to build a service that will handle the request messages. To do this, we’ll build a message consumer for the request message, host it inside Topshelf, and subscribe to a service bus bound to the input queue of the service.
The message handler will use the request message properties to prepare the request to the external web service and produce the response message once the request returns. In this example, the external web service only supports synchronous requests (in a later article I will try to cover calling remote services that support asynchronous requests).

At this point, we have a service that responds to a command message (the request) and produces a result depending upon the outcome of the request. What we need now is a way to coordinate the requests to the service to ensure that requests are not lost due to an unavailable service, errors are retried once a failed service becomes available, and duplicate requests are ignored.

To coordinate the service requests, a saga will be created to manage the state of each request. A new message will be created to initiate the saga containing the same data that is needed to produce the request message that is sent to our gateway service. And as with the service, messages will be created to return the results of the request to the consumer of the external service. The saga will use our state machine driven saga syntax, making the business logic understandable at a glance. The saga will also define the retry parameters that should be used to recover from service outages by specifying an exception policy. The messages that are used by the requester to interact with the saga will become the interface that is used by our system to make requests to the external service. This includes the web service we are in turn providing to our customers. The internal messages used to communicate between the saga and the gateway service are not intended for use outside of our gateway service component.

A nice side benefit of this architecture involves the actual call made by the gateway service. Since the interface is defined by the messages that are orchestrated by the saga, the backing implementation can be changed without impacting the consumers of the saga-based front end. This can be a huge benefit when it comes time to change service providers or bring an external service in-house either through acquisition or new product developments.

With the saga in place (and for this example, we will go ahead and host it in the same process that we are hosting the gateway service), we are now ready to build the web service request handler. Rather than go straight into that now, I think I’m going to stop here and save that for part three. After part three is finished, I’ll start posting some of the code for the examples and include it in the samples folder to make it easy to pull down and experiment with on your own machine.

Introduction

A common way of applying a messaging solution to an existing system is to find a tightly coupled service reference in the system and insulate the system from that dependency. For example, the system may utilize a web service via a call from the user interface, blocking the user interface until the web service request completes. Another situation might involve a web service provided by the system which in turn calls another web service to complete the request. While these two examples are common in systems today, it is important consider how the system is using a dependent service when determining how best to insulate the system from that service.

To help make that decision, allow me to share some common exchange patterns implemented using a web service. While most web services are modeled after the request/response pattern, there are actually several types of exchanges that can occur within a web service method.

Drop Box

If the web service is called by the client to provide information to the application, availability is likely the most important concern. Clients may have limited connectivity (such as a wireless client), limited resources (such as an embedded system that is unable to store data for future delivery), or a combination of these and other factors. To deal with these limitations, the web service should be designed for maximum availability to avoid failed requests due to an outage behind the web service boundary.

To support this high availability, the web service should count on the only thing that is available at the time web service method is invoked — the local machine. If the web service attempts to write the information to a database located on a server across the network that is not available, the information in the request may be lost. In this case, it would be a better choice to write a message that contains the request information to a local queue. By doing this, the request information is retained and can be processed separately from the web service request. This allows allows the web service method to return to the caller, keeping resources available for other clients to report.

Once the data from the request is safely stored in the queue, a separate service is built that consumes the messages from the queue and sends the information to the internal system, which in this case might be a database server. If the database server is unavailable, the message is left in the queue until the database is available. This new service can also be stopped and even updated without disrupting the web service from receiving requests.

Simple Request

If we look at another scenario, one where the request contains search criteria and the response includes the data specified matching the request. In order to break this down in different way, we will look at two different types of requests. First, we will define a request to retrieve the status of an order. Second, we will define a request to retrieve the details of an order.

When only the status of the order is requested, we are dealing with a relatively small amount of data per order — in this case, the order id and the status of the order. If this consisted of maybe 50 bytes of data per order, we could easily store the status of 100,000 orders in only 5 megabytes of RAM. Since users might check the status of an order often, at least once an hour, this could generate thousands of requests an hour. Since the status of an order may only change once or twice a day, the need to query the database each time the order status is checked can put an unnecessary strain on the database server. In addition, the order status request service is now tightly coupled to the database, making the availability of the service dependent upon the availability of the database. This dependency chain can get even longer as more complex systems are designed, so limiting the dependencies of a service is a key parameter in increasing availability.

To reduce database load, eliminate the dependency on the database, and increase the availability of our order status web service, we can design our service to subscribe to order status updates. When the status of an order is updated, the update will publish a message containing the new status. As the status of orders are updated, our service would update an in-memory cache containing the status of every order in the system (well, every is not necessarily every — it could just be the orders placed over the last week that have not yet been received by the customer). On startup, the web service would query the database for the status of all orders placed and use the results of that query to seed the cache. Once the cache is seeded, as new orders are added and order status updates occur, the cache would update dynamically in response to the update messages. Since the cache is local to the web service, requests need only check the status by querying the cache and immediately returning the status of the order to the caller. In the case of an status request for an order that does not exist in the cache, the service could queue a request to get the order status from the database which would then publish that orders status so that it could be returned. If the order is not found, the service could return an unknown order response to the caller with instructions to perhaps try their request again later.

Up Next

The two exchanges described above are relatively easy to implement using messaging (and likewise, using MassTransit). The next exchange pattern I’m going to cover is the more complex request where the dependent service must be called to complete the request. Due to that complexity, I’m going to wait until the next installment to describe that in greater detail. After that, I’ll start to share some code as we build a solution to these exchanges.

Anyone who has used a computer with any interest is familiar with system monitoring tools. Task Manager (or if you’re really cool Process Explorer) is the first place Windows users look when their system starts to crawl, Mac users turn to Activity Monitor, and I’m sure Linux users have some really obscure command-line tool as well. These coarse grained tools are usually enough for users, however, an operations team needs a higher degree of visibility into application — particularly if they are expected to determine how to tune the application for better performance.

For operations on Windows, Performance Monitor provides detailed information for running applications in real-time. On a web server, it is easy to find out how many threads your ASP.NET application is using, as well as how many requests are queued. That information can be correlated with processor utilization to help determine if the bottleneck is the CPU, the network, or possibly even the database server. When it comes to troubleshooting issues on a live system, more information is always helpful to determine the source of the problem.

To support this level of visibility in MassTransit, performance counter support has been added. Performance counters in .NET are part of the System.Diagnostics namespace. There are various counter types that can be defined, including counts, rates, and averages. When an application wants to output performance counters, it has to create a category and specify the counters that are included in the category. For instance:

ConsumerThreadCount = new RuntimePerformanceCounter("Consumer Threads",
	"The current number of threads processing messages.",
	PerformanceCounterType.NumberOfItems32);

ReceiveRate = new RuntimePerformanceCounter("Received/s",
	"The number of messages received per second",
	PerformanceCounterType.RateOfCountsPerSecond32);

These are two of the counters defined by the MassTransit category. The first is a count that is updated when the number of threads in use changes. The second is a rate which gets incremented once for every message received. The actual calculation and display of the rate is handled by the performance monitoring tools – the application does not need to calculate it.

ConsumerDuration = new RuntimePerformanceCounter("Average Consumer Duration",
	"The average time a consumer spends processing a message.",
	PerformanceCounterType.AverageCount64);

ConsumerDurationBase = new RuntimePerformanceCounter("Average Consumer Duration Base",
	"The average time a consumer spends processing a message.",
	PerformanceCounterType.AverageBase);

This counter is used to report the average consumer duration of a message. For an average, two counters are used. One is the base which is incremented for each occurrence and the counter is the actual count that is added. So for each message, the base is incremented once and the duration is incremented by the amount of time spent executing the consumer.

In adding performance counter support, I wanted to do it in a way that didn’t leak the details of updating performance information throughout the framework. It was at this point that I turned to the Magnum Pipeline. Using the pipeline to publish the metrics allowed me to isolate the actual performance counter interface to a single method in a single class for the service bus. So instead of passing interfaces around all the components that make up the bus, a single event aggregator is passed instead. When you start up the bus, the performance counter code subscribes to the events as shown:

_eventAggregatorScope.Subscribe<MessageReceived>(message =>
	{
		_counters.ReceiveCount.Increment();
		_counters.ReceiveRate.Increment();
		_counters.ReceiveDuration.IncrementBy((long) message.ReceiveDuration.TotalMilliseconds);
		_counters.ReceiveDurationBase.Increment();
		_counters.ConsumerDuration.IncrementBy((long) message.ConsumeDuration.TotalMilliseconds);
		_counters.ConsumerDurationBase.Increment();
	});

Now, when the bus receives a message, it sends the event to the event aggregator (an instance of the Magnum Pipeline).

var message = new MessageReceived
	{
		MessageType = messageType,
		ReceiveDuration = receiveTime,
		ConsumeDuration = consumeTime,
	};

	_eventAggregator.Send(message);

Since the Magnum Pipeline is publish/subscription, additional consumers could also opt-in to the MessageReceived event and perform other actions as well. I also plan to add counters per message type, allowing a finer grained view at message counts and consumer durations.

While the main story behind this post is the new counters available in MassTransit, my hope is that this brief introduction to performance counters was useful as well. You can learn more about performance counters from various articles that have been posted (such as a good one on CodeProject). You can check out the Magnum Pipeline in the Magnum project which is hosted at GoogleCode.

Here is my list of (mostly free) software to add to your Mac to take it up a notch:

• Skype
  The best way to make phone calls on your Mac
• Quicksilver
  Free fast application launcher and more
• HandBrake
  Don’t let those old DVDs go to waste, use HandBrake to rip them to your iTunes library (and play them on your TV or iPhone)
• Tweetie
  If Tweetie for the iPhone is good, Tweetie for the Mac is an even better Twitter client
• Evernote
  Keep track of everything and sync it automatically between Mac, Windows, iPhone, and web. Includes character recognition on images for searching text on pictures taken with your iPhone
• Transmission
  The best torrent client for the Mac
• Cyberduck
  FTP, WebDAV, iDisk, whatever this client is fast, free, and easy to use
• Colloquy
  IRC isn’t dead yet, and this free client works great on the Mac and iPhone
• Adium
  The best IM client on the Mac, works with just about anything that chats

And then there are the less often useful, but sometimes helpful ones:

• KisMAC
  Because sometimes you really, really need a hot spot
• MacTheRipper
  For those stubborn DVDs that won’t rip with HandBrake

Those are all the free ones, I’ll post another list of all the paid items that I find myself using nearly every day soon.

Nonetheless, a way to update a service that publishes messages without requiring every subscribing service to be updated at the same time was need.

Eliminating Impediments

Before we could implement interface subscriptions, there were a few things in the way that needed to be addressed, things that were not easy to implement.

First, we were still doing binary message serialization. While we had the ability to use the .NET XML Serializer, it tends to be slow and difficult to fit into the model we had built with MT. Back in May, XML became the default serialization format using an entirely new serializer built from scratch.

Second, we wanted to ensure that a publisher could publish a single message and have it delivered to all of the interested subscribers regardless of whether they had subscribed to the message class or one of the interfaces implemented by the class. In MassTransit, subscriptions are added by type a defined using a plain old CLR object (POCO). In the 0.6 release, we replaced the message dispatcher with a new type-based pipeline for both inbound and outbound messages. Starting with an object and working down the type structure of the message, messages are pushed through the pipeline to interested message sinks. In the case of the outbound pipeline, it makes it easy to push a class through that has interfaces, since the interfaces can be assigned from the message object. Another hurdle eliminated.

Implementing Interfaces

Once the hurdles were eliminated, it was actually very easy to add interface subscriptions. Since most of the internal bits had been reworked leveraging the power of expressions and generics, it was simply a matter of tweaking a few parts of the serializer and we were ready to rock and roll. Ensuring that message objects retain their type through the various pathways inside the system was also important, and resulted in fixing a couple of low hanging bugs related to message retry and fault publishing.

The one bit of code that needed to be built was a way to provide a backing class for an interface to store the property values. At first, I looked at using something like LinFu or DynamicProxy2 to create a proxy for the interface and intercept the property accessors, but this had a problem. I did not want property setters on the interface. At that point, I started looking at using the Emit classes, AutoMapper, the FastProperty expression-based accessors, and how Udi had dealt with it inside NServiceBus. What I ended up with was a very fast, cached object builder implementation that is integrated within the message deserializer. In the words of Cartman, “It’s pretty cool.”

There isn’t really a difference in the code between using classes and interfaces from either the producing or consuming end. While a producer will likely continue to publish a class, it just has to implement the message interface on that class, allowing the consumer to subscribe to the interface, breaking the dependency on the actual class published by the producer. The pipeline will then properly serialize out a message for that interface and send it directly to the consumer.

I’m pretty excited about this, and hope to update some of the pre-built services to use interfaces instead of classes in the near future. In the meantime, pull down the latest trunk and check it out.

When MassTransit was first started, MSMQ was the only transport we intended to support. In due time, however, it was determined that support for transports such as ActiveMQ and TIBCO was important. The ability to run on Linux and OS X under Mono (which does not support the System.Messaging namespace) as well as interoperability with Java systems using JMS (a specification for messaging, implemented by messaging systems like ActiveMQ and TIBCO) were the primary drivers of this decision. At the same time, insulating developers from the particulars of each transport was equally important.

To communicate with an endpoint, MassTransit uses the IEndpoint interface. The service bus would receive messages from an endpoint using this method:

IEnumerable< IMessageSelector > SelectiveReceive(TimeSpan timeout);

This involved making a call that returned an enumeration of message selectors, allowing the caller to step through the messages until an interesting message (in the case of the bus, a message with a subscribed consumer). The concerns of receiving a message were seemingly spread at random across three or four different classes (and yes, I wrote this crap). The reason for the complexity was solid though – I need the ability to selectively receive a message from a queue and skip over ones in which I have no interest.

The complexity of dealing with the yield return/break syntax of enumerators and managing scope is difficult. The programming semantics behind it are difficult to understand. I wanted something better. With all the time I’ve been spending since this was written dealing with nested closures, lambda functions, and continuations I realized there was a better way to reduce the complexity while at the same time improving extensibility.

The new signature for the receive method on an endpoint looks like this:

void Receive(Func< object, Action< object > > receiver, TimeSpan timeout);

With this new interface, the caller need only pass a method that accepts an object and returns a method that also accepts an object. The first method provides the caller an opportunity to inspect the message object to determine if the message will be consumed by the bus. If the bus is not interested, it can simply return null. If it is interested, it returns a method (either anonymous or a regular class method) that will consume the message. The endpoint will then call the returned method with the message once it has been received successfully. If the endpoint determines that the message is no longer available (if it were picked up by another process reading from the same queue for example), the returned method is not called.

The calling method looks something like this:

_endpoint.Receive(m => message => { doSomethingWith(message); });

This interface is far less complex to implement, and also made it easy to make a clean separation of what is an endpoint and what is a transport. Which leads me to…

Endpoint and Transport Split?

Sadly that reads like a Hollywood headline, but it is true. Endpoints now deal only with address resolution of sending and receiving messages and translating between the transport format and a message object (including de/serialization). New transport classes are now responsible for the actual communication with the various queue implementations supported by MassTransit.

For example, previously there was one class, MsmqEndpoint, that contained all the aspects of talking to MSMQ regardless of the type of queue (local non-transactional, local transactional, remote). Now beneath the endpoint itself, there are three MSMQ transports, one for each of these scenarios. Each of these transports cleanly deals with the particulars only, for example, the non-transactional transport has no transactional concerns in it at all.

Introducing ITransport

The new ITransport interface is narrow, dealing only with the simplest form of communication — streams. The send and receive methods from the endpoint are matched, but instead of dealing with objects, streams are used. Every transport should provide stream support at a minimum. The receive method of the transport looks like:

void Receive(Func< Stream, Action< Stream > > receiver, TimeSpan timeout);

While all transports implement streams, there is a benefit to communicating at a level above streams for certain types of endpoints. For example, when using MSMQ there are advantages to communicating directly with the Message object such as having access to the transport level message ID, the message label, and other interesting properties. To support this, the MsmqEndpoint only accepts an IMsmqTransport interface, which inherits from ITransport and adds:

void Receive(Func< Message, Action< Message > > receiver, TimeSpan timeout);

Other transports may benefit from a custom interface as well, but it is only implemented for MSMQ at this point. ActiveMQ, Loopback, and Multicast UDP all use the base stream interface.

Looking Forward

This rewrite was not purely for entertainment value (well, it was fun). Latency when sending a message from a machine to a remote queue is orders of magnitude slower than writing to a local queue. And in addition, local queues have the advantage of being local — which is important considering the first fallacy of distributed computing — the network is reliable (NOT!). To compensate for this, a more reliable method of sending messages to a remote queue is needed. By ensuring that messages sent/published by an application are durable regardless of network failure, developers can use this fire-and-forget approach to messaging that is key to building event driven applications.

To handle this, MassTransit now uses a store and forward transport for remote MSMQ queues. The store and forward transport will automatically create a local queue to cache the outbound messages destined for the remote queue. When a message is sent to the remote queue, the transport writes it to the local queue and returns to the caller. An asynchronous method then delivers the message in the background. The same transports that are used by the endpoint are reused by the store and forward transport, maintaining that high level of code reuse.

Note that on Windows Server 2003, I have observed that MSMQ will accept messages destined for a unreachable remote queue and attempt redelivery itself, but only for transactional queues (at least, that is what I have seen).

Wrapping Up

While it is always hopeful that changes like this will go by unnoticed, there is always the chance that there are some unintended consequences (read: bugs). Hopefully any of these will be weeded out quickly. In the meantime, I hope to start work on some availability features to support load balancing of command services.

Now while you could use MassTransit out of the box to handle this type of event aggregation, it is a bit heavy. The in-memory message transport serializes the message, which makes it impossible to pass a continuation or an object reference as part of an event. There is also a very service-oriented thread model where each consumer runs on a different thread making synchronization an important concern for unit testing. While it would work, it is not always the shiniest hammer in the toolbox for UI-based application.

To address this, one of the things I’ve been adding to Magnum over the past few weeks is a new version of the pipeline that handles message distribution in MassTransit. In this implementation, I wanted a way to implement the event aggregator pattern with the same flexibility that I get with MassTransit but designed for an in-process mode of execution. At the same time, I wanted to make sure that I could scale this solution via adapters to extend events to MassTransit for publishing out-of-process.

Note, I use the word event and message interchangeably in this post.

First, I wanted it to be able to handle any object without any constraints on the type. To this end, I came up with a very narrow API that only deals with the publishing of a message.

public interface Pipe
{
	void Send< T >(T message) where T : class;
}

The Send method is fairly obvious, it is used to send a message to any consumers that are subscribed to the message. With this implementation, consumers that are subscribed to any type to which the message can be assigned will also get the message. Consider the following class structure:

public class CustomerChanged
{
	public Customer Customer { get; set; }
}

public class CustomerRatingDowngraded :
	CustomerChanged
{
}

A consumer that subscribed to the CustomerChanged type would receive the message if a CustomerRatingDowngraded message was published. It also works for interfaces, as long as the message object being published supports the interface.

An obvious omission from this API is any method of subscribing consumers to the pipeline. To subscribe to the pipeline, an extension method on the Pipe interface creates a new subscription scope. A subscription scope, represented by the ISubscriptionScope interface, makes changes to the pipeline resulting in the creation of a new pipeline. A series of visitors are used to create a new version of the pipeline with the consumers added, along with another visitor to remove the consumers when they unsubscribe. ISubscriptionScope implements IDisposable so to unsubscribe your application can just dispose of the object.

It is interesting to note that much like the Expression class in .NET, pipelines are immutable. Since pipelines cannot be changed, the need to lock parts of the pipeline during message distribution is removed. By removing the need for locking to ensure safe operation in a concurrent environment, performance improves and blocking is eliminated. At the same time, consumers can subscribe and unsubscribe from the pipeline as needed without disrupting the system.

public void Start()
{
	// this creates an empty pipeline that accepts any object
	_eventAggregator=PipeSegment.Input(PipeSegment.End());

	_scope=_eventAggregator.NewSubscriptionScope();
	_scope.Subscribe< CustomerChanged >(message=>Trace.WriteLine("Customer changed: "+message.CustomerName));
}

In this example, pipe and scope would likely be member variables that would be released when the containing object is stopped or disposed. Multiple subscriptions can be added to a single scope, each one modifying the pipeline as it is added.

When I discuss event-based programming, I often mention the need for visualization tools in order to ensure the system is performing as expected. In the example above, I could use the TracePipeVisitor to verify that the consumer was indeed subscribed to the pipeline (by calling new TracePipeVisitor() .Trace(_eventAggregator) and viewing the results in the output window).

Input< Object >:
RecipientList< Object >:
     Filter< CustomerChanged >: Allow Magnum.Specs.Pipeline.Messages.CustomerChanged
     RecipientList< CustomerChanged >:
          MessageConsumer< CustomerChanged >:

As consumers are added, the pipeline is built up using a series of PipeSegment classes. The Input segment is the initial entry point to the pipeline and by having the responsibility is the only segment that actually changes in the pipeline. The RecipientList is a one-to-many switch that delivers incoming messages to each consumer. The Filter segment only passes a specific type through the filter, preventing unwanted messages from receiving the consumer. The MessageConsumer actually invokes the method that was subscribed to the message.

In the above example, the message consumer was accepted using the MessageConsumer delegate type, which is analogous to Action with T being the message type. Another way to subscribe is to implement the IConsume method as shown below.

public class ListViewController :
	IConsume< CustomerChanged >
{
	public ListViewController(ListView customerListView)
	{
		_customerListView = customerListView;
	}
	public void Consume(CustomerChanged message)
	{
		_customerListView.DoSomeUpdate(message.Customer);
	}
}

A class can implement the IConsume method to indicate that it is interested in messages of type T. In this case, the CustomerChanged message is of particular interest as it is used to update the user interface in response to a customer change event. The instance of the controller can be subscribed to the pipeline by calling the Subscribe method passing the object reference itself.

public void BootstrapUserInterfaceControllers()
{
	_customerListViewController = new ListViewController(customerListView);

	_scope=_eventAggregator.NewSubscriptionScope();
	_scope.Subscribe(_customerListViewController);
}

This is the first in a series of posts about the pipeline in Magnum. As I add the remaining functionality, including asynchronous message consumers, aggregate consumers, and automatic binding to the Magnum StateMachine (similar to how sagas are done using MassTransit), I’ll post about how they are used. I encourage you to take a look at the code and particularly the unit tests to see the different ways the pipeline can be used.

To start working towards this endeavor, I had to first get a working test environment. ActiveMQ can run on Windows, Linux, and OS X. Since Dru Sellers and I both develop on Macs using VMware Fusion to host various versions of Windows, I wanted to install and run ActiveMQ on the Mac host, making it available to any of the virtual machines. I had not really dealt with setting up services on OS X yet, but was happy to learn that it is a pretty slick process to get things installed and running. Hopefully this will help if you decide to do the same.

I should note that I am not an ActiveMQ installation/administration expert. I am configuring ActiveMQ for use in a development environment. If you are going to use ActiveMQ in production, make sure it is configured for proper production operation with the appropriate security, storage, etc. That being said, let’s get started.

Getting Started

You need to download ActiveMQ. I got the Linux version by typing in the URL manually to get it to download using Safari. The archive will be unpacked into a tar file automatically by Safari (if not, just double-click it), which you can then open the tar file into a folder by double-clicking it again. If you are a command-line wizard, you already know how to handle the tar.gz files so enjoy.

Move the unpacked folder (apache-activemq-5.2.0 in my case) to the /usr/local folder by opening Terminal and entering:

sudo mv apache-activemq-5.2.0 /usr/local/

While still in terminal, change to the ActiveMQ folder. We need to modify the configuration.

cd /usr/local/apache-activemq-5.2.0
mate conf/activemq.xml

If you aren’t using TextMate, well, do whatever you need to do to open that file. I removed a lot of unused things from the file, but your needs may vary. You can download my configuration file if you want to use what I am using. You will need to modify the IP addresses to match your environment. I originally tried to use just localhost, but had issues with it connecting from my Windows 7 VM. If this is just a fluke, I’ll update my file later with my new settings.

Installing the launch daemon into OS X

To run ActiveMQ as a service, you need to create a property list that describes the application. This is just an XML file, but we need to create it and put it into the /Library/LaunchDaemons folder and call it com.apache.activemq so we can identify it later. You can download my version of the file to save some typing if you prefer.

Some of these settings can be adjusted if you don’t want to keep ActiveMQ running all the time. KeepAlive will automatically restart the service if it stops for some reason (including manually stopping it) and you can set that to false if you want to control it manually.

After creating the file, we need to configure OS X so it knows about the new service. To do this, type the following:

sudo launchctl load /Library/LaunchDaemons/com.apache.activemq
sudo launchctl start com.apache.activemq

Once you have done this, you can verify that it is started by running the OS X Console application (find it in QuickSilver/Spotlight if you don’t know where it is). Look at the message logs and you can see the startup messages from the service:

So how do we know that we have a working installation? Well, there is an admin console that you can reach by navigating to http://localhost:8161/admin that will let you view the queues, topics, etc. that are running. You can also use the JMX tools to dig into the queues as well, including the ability to send messages to the queues directly from the Java console! To get the console started, you need to run jconsole from Terminal. Once it is started, you need to connect to the URL that is configured:

Once you are in the JConsole, you can view all the queues. It should look like this (well, assuming you’ve created some queues, which I’ve done here with the Starbucks sample from MassTransit).

You can see the default URL that was connected to in the title bar, along with the tree view of all the objects. The more interesting tab is the Operations tab, which lets you run commands against the queue. In fact, you can past some XML straight into an input box and click “sendTextMessage” and the message will be stuffed into the queue right there.

So now that we know ActiveMQ is running and happy, we can modify our application to use the ActiveMQ transport instead of the MSMQ transport by simply changing the URI for the endpoint. So instead of msmq://localhost/mt_subscriptions you would specify activemq://192.168.0.195:61616/mt_subscriptions (in my case, that is the IPv4 address of my host machine). As long as the transport is in the same folder and you’re using the StructureMap base registry without specifying a specific transport, it should connect up to the host and start working. The other containers will hopefully get this support soon, it was just easy to add with the Scan() feature of StructureMap’s registry DSL.

I hope to dig deeper into the ActiveMQ transport support in MassTransit, as well as start testing it while running under Mono on OS X over the next few weeks. I already have the Windows bits working, I just want to test more exception cases such as losing the connection to ActiveMQ, as well as other runtime issues to make the code more production ready. I also want to try sending messages to/from other languages, such as Ruby via STOMP, but my Ruby skills are not the greatest.

At the very least, I hope this article helps you get ActiveMQ installed and running on your Mac using OS X Leopard. If you do run into issues or have problems, be sure to visit the MassTransit mailing list and post your questions/issues there.

Microsoft Most Valuable Professionals (MVPs) are exceptional technical community leaders from around the world who are awarded for voluntarily sharing their high quality, real world expertise in offline and online technical communities. Microsoft MVPs are a highly select group of experts that represents the technical community’s best and brightest, and they share a deep commitment to community and a willingness to help others.

A community-based award like this would not be possible without the support of my peers, and I look forward to meeting up with some of those very peers at the next MVP Summit! In the meantime, congratulations to all the new and renewed MVPs!