My previous post on Domain Events left some questions about how aggregate roots should be created unanswered. It would actually be more accurate to say how aggregate roots should *not* be created. It turns out that this is one of the less intuitive parts of domain-driven design and has been the source of many arguments on the matter. Let’s start with the wrong way:

   1:  using (ISession s = sf.OpenSession())

   2:  using (ITransaction tx = s.BeginTransaction())

   3:  {

   4:      Customer c = new Customer();

   5:      c.Name = "udi dahan";

   6:   

   7:      s.Save(c);

   8:      tx.Commit();

   9:  }

I understand that the code above is representative of how much code is written when using an object-relational mapper. Many would consider this code to follow DDD principles - that Customer is an aggregate root. Unfortunately - that is not the case. The code above is missing the real aggregate root.

There’s also the inevitable question of validation - if the customer object isn’t willing to accept a name with a space in it, should we throw an exception? That would prevent an invalid entity from being saved, which is good. On the other hand, exceptions should be reserved for truly exceptional occurrences. But if we don’t use exceptions, using Domain Events instead, how do we prevent the invalid entity from being saved?

All of these issues are handled auto-magically once we have a true aggregate root.

Always Get An Entity

Let’s start with the technical guidance - always get an entity. At least one. Also, don’t add any objects to the session or unit of work explicitly - rather, have some other already persistent domain entity create the new entity and add it to a collection property.

Looking at the code above, we see that we’re not following the technical guidance.

But the question is, which entity could we possibly get from the database in this case? All we’re doing is adding a customer.

And that’s exactly where the technical guidance leads us to the business analysis that was missing in this scenario…

Business Analysis

Customers don’t just appear out of thin air.

Blindingly obvious - isn’t it.

So why would we technically model our system as if they did? My guess is that we never really thought about it - it wasn’t our job. So here’s the breaking news - if we want to successfully apply DDD we do need to think about it, it is our job.

Going back to the critical business question:

Where do customers come from?

In the real world, they stroll into the store. In our overused e-commerce example, they navigate to our website. New customers that haven’t used our site before don’t have any cookies or anything we can identify them with. They navigate around, browsing, maybe buying something in the end, maybe not.

Yet, the browsing process is interesting in its own right:

• Which products did they look at?
• Did they use the search feature?
• How long did they spend on each page?
• Did they scroll down to see the reviews?

If and when they do finally buy something, all that history is important and we’d like to maintain a connection to it.

Actually, even before they buy something, what they put in their cart is the interesting piece. The transition from cart to checkout is another interesting piece. Do they actually complete the checkout process, or do they abandon it midway through?

Add to that when we ask/force them to create a user/login in our system.

Are they actually a customer if they haven’t bought anything?

We’re beginning to get an inkling that almost every activity that results in the creation of an entity or storing of additional information can be traced to a transition from a previous business state.

In any transition, the previous state is the aggregate root.

In the beginning…

Let’s start at the very beginning then - someone came to our site. Either they navigated here from some other web page, they clicked on an email link someone sent them, or they typed in our URL. This can be designed as follows:

   1:  using (ISession s = sf.OpenSession())

   2:  using (ITransaction tx = s.BeginTransaction())

   3:  {

   4:     var referrer = s.Get<Referrer>(msg.URL);

   5:     referrer.BroughtVisitorWithIp(msg.IpAddress);

   6:   

   7:     tx.Commit();

   8:  }

   9:   

And our referrer code could look something like this:

   1:  public void BroughtVisitorWithIp(string ipAddress)

   2:  {

   3:     var visitor = new Visitor(ipAddress);

   4:     this.NewVisitors.Add(visitor);

   5:  }

   6:   

This follows the technical guidance we saw at the beginning.

It also allows us to track which referrer is bringing us which visitors, through tracking those visitors as they become shoppers (by putting stuff in their cart), finally seeing which become customers.

We can solve the situation of not having a referrer by implementing the null object pattern which is well supported by all the standard object-relational mappers these days.

How it works internally

When we call a method on a persistent entity retrieved by the object-relational mapper, and the entity modifies its state like when it adds a new entity to one of its collection properties, when the transaction commits, here’s what happens:

The mapper sees that the persistent entity is dirty, specifically, that its collection property was modified, and notices that there is an object in there that isn’t persistent. At that point, the mapper knows to persist the new entity without us ever having to explicitly tell it to do so. This is sometimes known as “persistence by reachability”.

Where validation happens

Let’s consider the relatively trivial rule that says that a user name can’t contain a space.

Also, keep in mind that a registered user is the result of a transition from a visitor.

Here’s *one* way of doing that:

   1:  public class Visitor

   2:  {

   3:     public void Register(string username, string password)

   4:     {

   5:        if (username.Contains(" "))

   6:        {

   7:           DomainEvents.Raise<UsernameCantContainSpace>();

   8:           return;

   9:        }

  10:   

  11:        var user = new User(username, password);

  12:        this.RegisteredUser = u;

  13:     }

  14:  }

  15:   

This actually isn’t representative of most of the rules that will be found in the domain model, but it illustrates a way of preventing an entity from being created without our service layer needing to know anything. All the service layer does is get the visitor object and call the Register method.

Validation of string lengths, data ranges, etc is not domain logic and is best handled elsewhere (and a topic for a different post). The same goes for uniqueness.

Summary

The most important thing to keep in mind is that if your service layer is newing up some entity and saving it - that entity isn’t an aggregate root *in that use case*. As we saw above, in the original creation of the Visitor entity by the Referrer, the visitor class wasn’t the aggregate root. Yet, in the user registration use case, the Visitor entity was the aggregate root.

Aggregate roots aren’t a structural property of the domain model.

And in any case, don’t go saving entities in your service layer - let the domain model manage its own state. The domain model doesn’t need any references to repositories, services, units of work, or anything else to manage its state.

If you do all this, you’ll also be able to harness the technique of fetching strategies to get the best performance out of your domain model by representing your use cases as interfaces on the domain model like IRegisterUsers (implemented by Visitor) and IBringVisitors (implemented by Referrer).

And spending some time on business analysis doesn’t hurt either - unless customers really do fall out of the sky in your world

Background

Before diving right into the code, I wanted to take a minute to recall how we got here.

It started by looking for how to create fully encapsulated domain models.

The main assertion being that you do *not* need to inject anything into your domain entities.

Not services. Not repositories. Nothing.

Just pure domain model goodness.

Make Roles Explicit

I’m going to take the advice I so often give. A domain event is a role, and thus should be represented explicitly:

   1:  public interface IDomainEvent {}

If this reminds you of the IMessage marker interface in nServiceBus, you’re beginning to see where this is going…

How to define domain events

A domain event is just a simple POCO that represents an interesting occurence in the domain. For example:

   1:  public class CustomerBecamePreferred : IDomainEvent 

   2:  {

   3:      public Customer Customer { get; set; }

   4:  }

For those of you concerned about the number of events you may have, and therefore are thinking about bunching up these events by namespaces or things like that, slow down. The number of domain events and their cohesion is directly related to that of the domain model.

If you feel the need to split your domain events up, there’s a good chance that you should be looking at splitting your domain model too. This is the bottom-up way of identifying bounded contexts.

How to raise domain events

In your domain entities, when a significant state change happens you’ll want to raise your domain events like this:

   1:  public class Customer

   2:  {

   3:      public void DoSomething()

   4:      {

   5:          DomainEvents.Raise(new CustomerBecamePreferred() { Customer = this });

   6:      }

   7:  }

We’ll look at the DomainEvents class in just a second, but I’m guessing that some of you are wondering “how did that entity get a reference to that?” The answer is that DomainEvents is a static class. “OMG, static?! But doesn’t that hurt testability?!” No, it doesn’t. Here, look:

Unit testing with domain events

One of the things we’d like to check when unit testing our domain entities is that the appropriate events are raised along with the corresponding state changes. Here’s an example:

   1:  public void DoSomethingShouldMakeCustomerPreferred()

   2:  {

   3:      var c = new Customer();

   4:      Customer preferred = null;

   5:   

   6:      DomainEvents.Register<CustomerBecamePreferred>(

   7:          p => preferred = p.Customer

   8:              );

   9:   

  10:      c.DoSomething();

  11:      Assert(preferred == c && c.IsPreferred);

  12:  }

As you can see, the static DomainEvents class is used in unit tests as well. Also notice that you don’t need to mock anything - pure testable bliss.

Who handles domain events

First of all, consider that when some service layer object calls the DoSomething method of the Customer class, it doesn’t necessarily know which, if any, domain events will be raised. All it wants to do is its regular schtick:

   1:  public void Handle(DoSomethingMessage msg)

   2:  {

   3:      using (ISession session = SessionFactory.OpenSession())

   4:      using (ITransaction tx = session.BeginTransaction())

   5:      {

   6:          var c = session.Get<Customer>(msg.CustomerId);

   7:          c.DoSomething();

   8:   

   9:          tx.Commit();

  10:      }

  11:  }

The above code complies with the Single Responsibility Principle, so the business requirement which states that when a customer becomes preferred, they should be sent an email belongs somewhere else.

Notice that the key word in the requirement - “when”.

Any time you see that word in relation to your domain, consider modeling it as a domain event.

So, here’s the handling code:

   1:  public class CustomerBecamePreferredHandler : Handles<CustomerBecamePreferred>

   2:  { 

   3:     public void Handle(CustomerBecamePreferred args)

   4:     {

   5:        // send email to args.Customer

   6:     }

   7:  } 

This code will run no matter which service layer object we came in through.

Here’s the interface it implements:

   1:  public interface Handles<T> where T : IDomainEvent

   2:  {

   3:      void Handle(T args); 

   4:  } 

Fairly simple.

Please be aware that the above code will be run on the same thread within the same transaction as the regular domain work so you should avoid performing any blocking activities, like using SMTP or web services. Instead, prefer using one-way messaging to communicate to something else which does those blocking activities.

Also, you can have multiple classes handling the same domain event. If you need to send email *and* call the CRM system *and* do something else, etc, you don’t need to change any code - just write a new handler. This keeps your system quite a bit more stable than if you had to mess with the original handler or, heaven forbid, service layer code.

Where domain event handlers go

These handler classes do not belong in the domain model.

Nor do they belong in the service layer.

Well, that’s not entirely accurate - you see, there’s no *the* service layer. There is the part that accepts messages from clients and calls methods on the domain model. And there is another, independent part that handles events from the domain. Both of these will probably make use of a message bus, but that implementation detail shouldn’t deter you from keeping each in their own package.

The infrastructure

I know you’ve been patient, reading through all my architectural blah-blah, so here it is:

   1:  public static class DomainEvents

   2:  { 

   3:      [ThreadStatic] //so that each thread has its own callbacks

   4:      private static List<Delegate> actions;

   5:   

   6:      public static IContainer Container { get; set; } //as before

   7:   

   8:      //Registers a callback for the given domain event

   9:      public static void Register<T>(Action<T> callback) where T : IDomainEvent

  10:      {

  11:         if (actions == null)

  12:            actions = new List<Delegate>();

  13:   

  14:         actions.Add(callback);

  15:     }

  16:   

  17:     //Clears callbacks passed to Register on the current thread

  18:     public static void ClearCallbacks ()

  19:     {

  20:         actions = null;

  21:     }

  22:   

  23:     //Raises the given domain event

  24:     public static void Raise<T>(T args) where T : IDomainEvent

  25:     {

  26:        if (Container != null)

  27:           foreach(var handler in Container.ResolveAll<Handles<T>>())

  28:              handler.Handle(args);

  29:   

  30:        if (actions != null)

  31:            foreach (var action in actions)

  32:                if (action is Action<T>)

  33:                    ((Action<T>)action)(args);

  34:     }

  35:  } 

Notice that while this class *can* use a container, the container isn’t needed for unit tests which use the Register method.

When used server side, please make sure that you add a call to ClearCallbacks in your infrastructure’s end of message processing section. In nServiceBus this is done with a message module like the one below:

   1:  public class DomainEventsCleaner : IMessageModule

   2:  { 

   3:      public void HandleBeginMessage() { }

   4:   

   5:      public void HandleEndMessage()

   6:      {

   7:          DomainEvents.ClearCallbacks();

   8:      }

   9:  }

The main reason for this cleanup is that someone just might want to use the Register API in their original service layer code rather than writing a separate domain event handler.

Summary

Like all good things in life, 3rd time’s the charm.

It took a couple of iterations, and the API did change quite a bit, but the overarching theme has remained the same - keep the domain model focused on domain concerns. While some might say that there’s only a slight technical difference between calling a service (IEmailService) and using an event to dispatch it elsewhere, I beg to differ.

These domain events are a part of the ubiquitous language and should be represented explicitly.

CustomerBecamePreferred is nothing at all like IEmailService.

In working with your domain experts or just going through a requirements document, pay less attention to the nouns and verbs that Object-Oriented Analysis & Design call attention to, and keep an eye out for the word “when”. It’s a critically important word that enables us to model important occurrences and state changes.

What do you think? Are you already using this approach? Have you already tried it and found it broken in some way? Do you have any suggestions on how to improve it?

Let me know - leave a comment below.

There’s this belief that if we just reused more code, everything would be better.

Some even go so far as saying that the whole point of object-orientation was reuse - it wasn’t, encapsulation was the big thing. After that component-orientation was the thing that was supposed to make reuse happen. Apparently that didn’t pan out so well either because here we are now pinning our reuseful hopes on service-orientation.

Entire books of patterns have been written on how to achieve reuse with the orientation of the day.
Services have been classified every which way in trying to achieve this, from entity services and activity services, through process services and orchestration services. Composing services has been touted as the key to reusing, and creating reusable services.

I might as well let you in on the dirty-little secret:

Reuse is a fallacy

Before running too far ahead, let’s go back to what the actual goal of reuse was: getting done faster.

That’s it.

It’s a fine goal to have.

And here’s how reuse fits in to the picture:

If we were to write all the code of a system, we’d write a certain amount of code.
If we could reuse some code from somewhere else that was written before, we could write less code.
The more code we can reuse, the less code we write.
The less code we write, the sooner we’ll be done!

However, the above logical progression is based on another couple of fallacies:

Fallacy: All code takes the same amount of time to write

Fallacy: Writing code is the primary activity in getting a system done

Anyone who’s actually written some code that’s gone into production knows this.

There’s the time it takes us to understand what the system should do.
Multiply that by the time it takes the users to understand what the system should do
Then there’s the integrating that code with all the other code, databases, configuration, web services, etc.
Debugging. Deploying. Debugging. Rebugging. Meetings. Etc.

Writing code is actually the least of our worries.
We actually spend less time writing code than…

Rebugging code

Also known as bug regressions.

This is where we fix one piece of code, and in the process break another piece of code.
It’s not like we do it on purpose. It’s all those dependencies between the various bits of code.
The more dependencies there are, the more likely something’s gonna break.
Especially when we have all sorts of hidden dependencies,
like when other code uses stuff we put in the database without asking us what it means,
or, heaven forbid, changing it without telling us.

These debugging/rebugging cycles can make stabilizing a system take a long time.

So, how does reuse help/hinder with that?

Here’s how:

Dependencies multiply by reuse

It’s to be expected. If you wrote the code all in one place, there are no dependencies. By reusing code, you’ve created a dependency. The more you reuse, the more dependencies you have. The more dependencies, the more rebugging.

Of course, we need to keep in mind the difference between…

Reuse & Use

Your code uses the runtime API (JDK, .NET BCL, etc).
Likewise other frameworks like (N)Hibernate, Spring, WCF, etc.

Reuse happens when you extend and override existing behaviors within other code.
This is most often done by inheritance in OO languages.

Interestingly enough, by the above generally accepted definition, most web services “reuse” is actually really use.

Let’s take a look at the characteristics of the code we’re using and reusing to see where we get the greatest value:

The value of (re)use

If we were to (re)use a piece of code in only one part of our system, it would be safe to say that we would get less value than if we could (re)use it in more places. For example, we could say that for many web applications, the web framework we use provides more value than a given encryption algorithm that we may use in only a few places.

So, what characterizes the code we use in many places?

Well, it’s very generic.

Actually, the more generic a piece of code, the less likely it is that we’ll be changing something in it when fixing a bug in the system.

That’s important.

However, when looking at the kind of code we reuse, and the reasons around it, we tend to see very non-generic code - something that deals with the domain-specific behaviors of the system. Thus, the likelihood of a bug fix needing to touch that code is higher than in the generic/use-not-reuse case, often much higher.

How it all fits together

Goal: Getting done faster
Via: Spending less time debugging/rebugging/stabilizing
Via: Having less dependencies reasonably requiring a bug fix to touch the dependent side
Via: Not reusing non-generic code

This doesn’t mean you shouldn’t use generic code / frameworks where applicable - absolutely, you should.
Just watch the number of kind of dependencies you introduce.

Back to services

So, if we follow the above advice with services, we wouldn’t want domain specific services reusing each other.
If we could get away with it, we probably wouldn’t even want them using each other either.

As use and reuse go down, we can see that service autonomy goes up. And vice-versa.
Luckily, we have service interaction mechanisms from Event-Driven Architecture that enable use without breaking autonomy.
Autonomy is actually very similar to the principle of encapsulation that drove object-orientation in the first place.
Interesting, isn’t it?

In summary

We all want to get done faster.

Way back when, someone told us reuse was the way to do that.

They were wrong.

Reuse may make sense in the most tightly coupled pieces of code you have, but not very much anywhere else.

When designing services in your SOA, stay away from reuse, and minimize use (with EDA patterns).

The next time someone pulls the “reuse excuse”, you’ll be ready.

Further Reading

• Additional logic required for service autonomy
• Self-contained events & SOA
• Autonomous Services and Enterprise Entity Aggregation [MS Architecture Journal]
• Does SOA mean the end of OO? [Podcast]

I’ll be discussing business service boundaries, publish/subscribe eventing, and business activity monitoring - with some time for questions at end.

Update: Recording is now online - here.

Hope to virtually see you there.

Here’s the details of the live meeting:

Start Time: Monday, June 01, 2009 07:00 PM GMT
End Time: Monday, June 01, 2009 08:30 PM GMT

Attendee URL: http://snipr.com/virtualaltnet (Live Meeting)

“A common complaint is that the first call on a client object takes some disproportionately large amount of time, usually ten seconds or more, while successive calls are instantaneous. There are many reasons why this might happen so there’s no generic resolution for this problem.” — Nicholas Allen

The thing is that there IS a generic solution to this problem.

It’s queued messaging.

The only thing is that you have to give up talking to your services as if they were regular objects - calling methods on them and expecting a response. In other words, designing a distributed systems isn’t like designing a regular OO system just with some WCF sprinkled on top.

Even when trying to do fire and forget messaging on top of WCF (void method calls with the OneWay attribute), the underlying channel can still block your thread, as Nick mentioned.

A queue isn’t an implementation detail.
It’s the primary architectural abstraction of a distributed system.

First of all, the conventional wisdom:

“If you don’t choose assembly to represent component, the natural artifact candidate is then namespace.”

There’s only one minor assumption here that deserves being dragged out into the light.

While Visual Studio creates an assembly from every project by default, you can take those assemblies and merge them together into a single assembly using this nice little utility from Microsoft. It is likely that each project would have its own namespace too, so we should still be aligned with the conventional wisdom.

In other words, we could choose a Visual Studio project to represent a logical component and still be in the same camp as Jeremy:

“I’m very firmly in the camp that says you should only split assemblies by deployment targets”

What everyone agrees about seems to be that coupling hurts, and should be managed.

Where does coupling come from? Well, from references between two pieces of code. If we were to represent our logical components as Visual Studio projects, we could easily see those references without the help of any 3rd party tools. The compiler would even yell at us if we were to (accidentally) create an evil circular reference.

While some might complain about the long compile time when we have many projects in a single solution, good componentization often doesn’t require us to put all projects in a single solution. In fact, each component could theoretically have its own solution - since it’s reasonable to assume we’d really only be working on one component at a time. In which case, compile time per developer task would be a non-issue.

Going through the whole code base is usually only needed when doing a full-system debug when trying to track down a problem. This wouldn’t need to be done against a solution with all projects. We’d do this using PDBs of the merged projects (as that’s what actually got delivered, and where the bug was found). After spelunking through those PDBs, we’d eventually find the problematic component (or 2, or 3, or …), and open up developer tasks for each component.

Regardless of if we’re putting out a patch for an existing customer or rolling these changes into a release with other tasks, all the logical components would be built into a physical system (merged as necessary) and the system would be put through QA.

In short, it looks like just a bit of unconventional wisdom gets us a nice balance.

The common e-commerce example

We accept orders, bill the customer, and then ship them the product.

Fairly straight-forward.

Since each part of that process can be quite complex, let’s have each step be handled by a service:

Sales, Billing, and Shipping. Each of these services will publish an event when it’s done its part. Sales will publish OrderAccepted containing all the order information - order Id, customer Id, products, quantities, etc. Billing will publish CustomerBilledForOrder containing the customer Id, order Id, etc. And Shipping will publish OrderShippedToCustomer with its data.

So far, so good. EDA and SOA seem to be providing us some value.

Where’s the saga?

Well, let’s consider the behavior of the Shipping service. It shouldn’t ship the order to the customer until it has received the CustomerBilledForOrder event as well as the OrderAccepted event. In other words, Shipping needs to hold on to the state that came in the first event until the second event comes in. And this is exactly what sagas are for.

Let’s take a look at the saga code that implements this. In order to simplify the sample a bit, I’ll be omitting the product quantities.

   1:      public class ShippingSaga : Saga<ShippingSagaData>,

   2:          ISagaStartedBy<OrderAccepted>,

   3:          ISagaStartedBy<CustomerBilledForOrder>

   4:      {

   5:          public void Handle(OrderAccepted message)

   6:          {

   7:              this.Data.ProductIdsInOrder = message.ProductIdsInOrder;

   8:          }

   9:   

  10:          public void Handle(CustomerBilledForOrder message)

  11:          {

  12:               this.Bus.Send<ShipOrderToCustomer>(

  13:                  (m =>

  14:                  {

  15:                      m.CustomerId = message.CustomerId;

  16:                      m.OrderId = message.OrderId;

  17:                      m.ProductIdsInOrder = this.Data.ProductIdsInOrder;

  18:                  }

  19:                  ));

  20:   

  21:              this.MarkAsComplete();

  22:          }

  23:   

  24:          public override void Timeout(object state)

  25:          {

  26:              

  27:          }

  28:      }

First of all, this looks fairly simple and straightforward, which is good.

It’s also wrong, which is not so good.

One problem we have here is that events may arrive out of order - first CustomerBilledForOrder, and only then OrderAccepted. What would happen in the above saga in that case? Well, we wouldn’t end up shipping the products to the customer, and customers tend not to like that (for some reason).

There’s also another problem here. See if you can spot it as I go through the explanation of ISagaStartedBy<T>.

Saga start up and correlation

The “ISagaStartedBy<T>” that is implemented for both messages indicates to the infrastructure (NServiceBus) that when a message of that type arrives, if an existing saga instance cannot be found, that a new instance should be started up. Makes sense, doesn’t it? For a given order, when the OrderAccepted event arrives first, Shipping doesn’t currently have any sagas handling it, so it starts up a new one. After that, when the CustomerBilledForOrder event arrives for that same order, the event should be handled by the saga instance that handled the first event - not by a new one.

I’ll repeat the important part: “the event should be handled by the saga instance that handled the first event”.

Since the only information we stored in the saga was the list of products, how would we be able to look up that saga instance when the next event came in containing an order Id, but no saga Id?

OK, so we need to store the order Id from the first event so that when the second event comes along we’ll be able to find the saga based on that order Id. Not too complicated, but something to keep in mind.

Let’s look at the updated code:

   1:      public class ShippingSaga : Saga<ShippingSagaData>,

   2:          ISagaStartedBy<OrderAccepted>,

   3:          ISagaStartedBy<CustomerBilledForOrder>

   4:      {

   5:          public void Handle(CustomerBilledForOrder message)

   6:          {

   7:              this.Data.CustomerHasBeenBilled = true;

   8:   

   9:              this.Data.CustomerId = message.CustomerId;

  10:              this.Data.OrderId = message.OrderId;

  11:   

  12:              this.CompleteIfPossible();

  13:          }

  14:   

  15:          public void Handle(OrderAccepted message)

  16:          {

  17:              this.Data.ProductIdsInOrder = message.ProductIdsInOrder;

  18:   

  19:              this.Data.CustomerId = message.CustomerId;

  20:              this.Data.OrderId = message.OrderId;

  21:   

  22:              this.CompleteIfPossible();

  23:          }

  24:   

  25:          private void CompleteIfPossible()

  26:          {

  27:              if (this.Data.ProductIdsInOrder != null && this.Data.CustomerHasBeenBilled)

  28:              {

  29:                  this.Bus.Send<ShipOrderToCustomer>(

  30:                     (m =>

  31:                     {

  32:                         m.CustomerId = this.Data.CustomerId;

  33:                         m.OrderId = this.Data.OrderId;

  34:                         m.ProductIdsInOrder = this.Data.ProductIdsInOrder;

  35:                     }

  36:                     ));

  37:                  this.MarkAsComplete();

  38:              }

  39:          }

  40:      }

And that brings us to…

Saga persistence

We already saw why Shipping needs to be able to look up its internal sagas using data from the events, but what that means is that simple blob-type persistence of those sagas is out. NServiceBus comes with an NHibernate-based saga persister for exactly this reason, though any persistence mechanism which allows you to query on something other than saga Id would work just as well.

Let’s take a quick look at the saga data that we’ll be storing and see how simple it is:

   1:      public class ShippingSagaData : ISagaEntity

   2:      {

   3:          public virtual Guid Id { get; set; }

   4:          public virtual string Originator { get; set; }

   5:          public virtual Guid OrderId { get; set; }

   6:          public virtual Guid CustomerId { get; set; }

   7:          public virtual List<Guid> ProductIdsInOrder { get; set; }

   8:          public virtual bool CustomerHasBeenBilled { get; set; }

   9:      }

You might have noticed the “Originator” property in there and wondered what it is for. First of all, the ISagaEntity interface requires the two properties Id and Originator. Originator is used to store the return address of the message that started the saga. Id is for what you think it’s for. In this saga, we don’t need to send any messages back to whoever started the saga, but in many others we do. In those cases, we’ll often be handling a message from some other endpoint when we want to possibly report some status back to the client that started the process. By storing that client’s address the first time, we can then “ReplyToOriginator” at any point in the process.

The manufacturing sample that comes with NServiceBus shows how this works.

Saga Lookup

Earlier, we saw the need to search for sagas based on order Id. The way to hook into the infrastructure and perform these lookups is by implementing “IFindSagas<T>.Using<M>” where T is the type of the saga data and M is the type of message. In our example, doing this using NHibernate would look like this:

   1:      public class ShippingSagaFinder : 

   2:          IFindSagas<ShippingSagaData>.Using<OrderAccepted>,

   3:          IFindSagas<ShippingSagaData>.Using<CustomerBilledForOrder>

   4:      {

   5:          public ShippingSagaData FindBy(CustomerBilledForOrder message)

   6:          {

   7:              return FindBy(message.OrderId)

   8:          }

   9:   

  10:          public ShippingSagaData FindBy(OrderAccepted message)

  11:          {

  12:              return FindBy(message.OrderId)

  13:          }

  14:   

  15:          private ShippingSagaData FindBy(Guid orderId)

  16:          {

  17:              return sessionFactory.GetCurrentSession().CreateCriteria(typeof(ShippingSagaData))

  18:                  .Add(Expression.Eq("OrderId", orderId))

  19:                  .UniqueResult<ShippingSagaData>();

  20:          }

  21:   

  22:          private ISessionFactory sessionFactory;

  23:   

  24:          public virtual ISessionFactory SessionFactory

  25:          {

  26:              get { return sessionFactory; }

  27:              set { sessionFactory = value; }

  28:          }

  29:      }

For a performance boost, we’d probably index our saga data by order Id.

On concurrency

Another important note is that for this saga, if both messages were handled in parallel on different machines, the saga could get stuck. The persistence mechanism here needs to prevent this. When using NHibernate over a database with the appropriate isolation level (Repeatable Read - the default in NServiceBus), this “just works”. If/When implementing your own saga persistence mechanism, it is important to understand the kind of concurrency your business logic can live with.

Take a look at Ayende’s example for mobile phone billing to get a feeling for what that’s like.

Summary

In almost any event-driven architecture, you’ll have services correlating multiple events in order to make decisions. The saga pattern is a great fit there, and not at all difficult to implement. You do need to take into account that events may arrive out of order and implement the saga logic accordingly, but it’s really not that big a deal. Do take the time to think through what data will need to be stored in order for the saga to be fault-tolerant, as well as a persistence mechanism that will allow you to look up that data based on event data.

If you feel like giving this approach a try, but don’t have an environment handy for this, download NServiceBus and take a look at the samples. It’s really quick and easy to get set up.

Very little has changed, so the version 1.9 story described here is still accurate.

Just last week one of my clients went live with a rollout to one of the world’s biggest names in the hospitality industry and things are looking good. Since stability is such a big deal to them (and many of my other clients), they’ve rolled out on nServiceBus 1.8 but now are ready to make the move to 1.9. Hopefully we’ll be able to get a case study out of them

For more information, go to the NServiceBus site.

I’ve been to too many clients where I’ve been brought in to help them with their problems around service versioning when the solution I propose is simply to have version N+1 of the system be backwards-compatible with version N. If two adjacent versions of a given system aren’t compatible with each other, it is practically impossible to solve versioning issues.

Here’s what happens when versions aren’t compatible:

Admins stop the system from accepting any new requests, and wait until all current requests are done processing. They take a backup/snapshot of all relevant parts of the system (like data in the DB). Then, bring down the system – all of it. Install the new version on all machines. Bring everything back up. Let the users back in.

If, heaven-forbid, problems were uncovered with the new version (since some problems only appear in production), the admins have to roll back to the previous version – once again bringing everything down.

This scenario is fairly catastrophic for any company that requires not-even high availability, but pretty continuous availability – like public facing web apps.

If adjacent versions were compatible with each other, we could upgrade the system piece-meal – machine by machine, where both the old and new versions will be running side by side, communicating with each other. While the system’s performance may be sub-optimal, it will continue to be available throughout upgrades as well as downgrades.

This isn’t trivial to do.

It impacts how you decide what is (and more importantly, what isn’t) nullable.

It may force you to spread certain changes to features across more versions (aka releases).

As such, you can expect this to affect how you do release and feature planning.

However, if you do not take these factors into account, it’s almost a certainty that your versioning problems will persist and no technology (new or old) will be able to solve them.

Coming next… Units of versioning – inside and outside a service.

My article on “optimizing a large-scale Software+Services application” has been published in the April edition of MSDN Magazine.

Here’s a short excerpt:

“We had to juggle occasional connectivity, data synchronization, and publish/subscribe all at the same time. We learned that we couldn’t solve all problems either client-side or server-side, but rather that an integrated approach was needed since any changes on one side needed corresponding changes on the other side.”

Continue reading…

Those of you following me on twitter (here) have an inkling that it has to do with the connection between user interfaces and services - but that’ll have to wait. Don’t want to steal my own thunder.

A nice meaty blog post on the topic is coming.

Greg Young asserts:

“I believe that this shows there to be a rather negligible cost associated with the use of such a model. There is however a small cost, this cost however I believe only exists when one looks at the system in isolation.”

Ayende adds his perspective:

“The cost of messaging, and a very real one, comes when you need to understand the system. In a system where message exchange is the form of communication, it can be significantly harder to understand what is going on.”

Of course, both these intelligent fellows are right. The reason for the apparent disparity in viewpoints has to do with which part of the following graph you look at. Ayende zooms in on the left side:

As systems get larger, though, the only way to understand them is by working at higher levels of abstraction. That’s where messaging really shines, as the incremental complexity remains the same by maintaining the same modularity as before:

In Ayende’s post, he follows the design I described a while back on using messaging for user management and login for a high-scale web scenario. In his comments, he agrees with the above stating:

“I certainly think that a similar solution using RPC would be much more complex and likely more brittle.”

I feel quite conservative in saying the most enterprise solutions fall on the right side of the intersection in the graph.

That being said, don’t underestimate the learning curve developers go through with messaging. While the mechanics are similar, the mindset is very different. Think about it like this:

You’ve driven a car for years in the US. It’s practically second nature. Then you fly to the UK, rent a car, and all of a sudden, your brain is in meltdown. (or vice versa for those going from the UK to the US)

Summary

If you are going down the messaging route, please be aware that there are shades of gray there as well. You don’t have to implement your user management and login the way I outlined in my post if you don’t require such high levels of scalability, but even lower levels of scalability can benefit from messaging.

Just as there isn’t a single correct design for non-messaging solutions, the same is true for those using messaging. Finding the right balance is tricky, and critical.

When the code is simple in every part of the system, and the asynchronous interactions are what provide for the necessary complexity the problem domain requires, that’s when you know you’ve got it just right.

Definitely worth checking out.

Register here

What is it?

From the NServiceBus.com site:

“NServiceBus is a powerful, yet lightweight, open source messaging framework
for designing distributed .NET enterprise systems. Entirely pluggable yet simple to use, NServiceBus gives programmers a head-start on developing robust, scalable,
and maintainable service-layers and long-running business processes.”

One of the developers who downloaded nServiceBus, Jürgen, sent me this in an email:

“I took the samples, swapped in my own code, and had a machines subscribing, publishing, messaging, in like 15 minutes.

I’ve got to tell you - from reading the architecture stuff on your blog I always thought this stuff was going to be hard. I know you always say its supposed to be simple but I never really believed it. I mean, seriously, if I had to do this stuff with web services or WCF - well, I wouldn’t know where to begin.”

(His English totally surprised me - not what you’d expect from someone called Jürgen. Born in Sweden but grew up in the US.)

BTW, Sam’s showed where to begin with WCF: basic pub/sub without durability is 480 LOC.

What’s different - Containers

People who looked at earlier versions of nServiceBus will see many incremental improvements that smooth over previously rough parts of the framework.

Chris Patterson, co-founder of MassTransit mentioned one such area in his blog post:

“Both Dru and I needed a framework for asynchronous messaging to address some work-related application requirements. While MSMQ is provided out of the box [on Windows], it doesn’t directly encourage some good distributed application practices such a loose coupling. Our goal was to abstract the messaging aspects so the services could be built to deal with plain old objects (POCOs) instead of lower level transport messages.

Originally, we both looked at NServiceBus as a way to make this happen. I’ve followed Udi’s blog for a while and have really gained a lot of knowledge from his posts and presentations. However, our lack of experience in Spring.NET, along with a general lack of understanding of all the complexity of such a framework led us down the path of building our own framework.”

Version 1.9 takes a totally different approach (than v1.6/1.7 at which I believe Chris was looking back then) to dependency injection frameworks (like Spring.NET) and decreases their footprint. Developers don’t need to know anything about Spring, Castle, or any other container to get started, but always have the ability to configure it however they want and even swap in their container of choice.

What’s different - DLL Footprint

Another difference is the number of assemblies that come with nServiceBus. Nathan Stults had this to say about version 1.8:

“NServiceBus (for valid architectural reasons) is split up into eleven thousand, two hundred and nine different DLL’s. That may be an exaggeration, but many of the dll’s have only one or two code files in them, with only a few lines of code each.”

NServiceBus has always supported swapping out various technological implementations and will continue do so. For that reason, the development environment is organized into 79 projects whose dependencies are managed very strictly. As of version 1.9, most of these assemblies including many supporting libraries like Spring and Castle have been merged into NServiceBus.dll. There’s also NServiceBus.Testing.dll which supports fluent-unit-testing entire business processes.

There is one non-optional external dependency which hasn’t been merged and that is log4net - although if you configure in a Common.Logging provider to your own logging infrastructure, you can do without it as well. With log4net, the minimum deployment footprint is 2 assemblies. The other dependency which is optional is NHibernate. The reason for leaving these out is that many teams depend on specific versions of those assemblies.

In short - you reference ONE assembly. Just one.

What else?

There is quite a lot in there. Ayende’s put out several posts describing those features and similarities to the bus he’s working on:

• Managing Timeouts
• Long-running processes
• Load Balancing

Ayende’s description of the NServiceBus load-balancing capability was:

“The distributor section of NServiceBus is a thing of beauty.”

Performance

You may be surprised by the kind of performance we’re able to wring out of “basic” MSMQ. Keep in mind, though, that you can swap in your own transport - there are already some others out there on the Contrib site including ActiveMQ and shared memory.

The most thorough performance numbers I’ve seen on nServiceBus have been written up here:

“total throughput over 1 billion messages an hour. That was about 100 million per hour durable, 900 million per hour non-durable”

This was on an 3 blade centers (48 blades), 30 pizza boxes (1U), and 20 clusters, version 1.8.

On Gojko’s blog there is a video of talk on nServiceBus where Dave de Florinier mentioned throughputs of 600,000 messages per hour (no mention of supporting hardware) and I think it was version 1.7 or 1.8 of nServiceBus.

Recently, Raymond Lewallen posted his numbers for 1.9 to the discussion group:

“We load tested our full duplex and pub/sub scenarios the other day at about 500 messages per second with no hiccups at all.” (save yourself the math, its 1.8M/hr)

The really interesting numbers are coming in from one of my clients (on 1.9) where the focus is on business process (saga) throughput where they’re seeing millions completing each day on top of IO intensive technical processes. I’m excited.

Looking forward

After working with clients using nServiceBus over 4 years now (even before it was open source or had a real name), it just keeps getting better. It’s also really great to see more open source projects coming on the scene - MassTransit (now a year old), and Rhino Service Bus (young, but in production with NH-Prof).

The mutual stealing cross-pollination is increasing the pace all around.

I intend to take up TopShelf (which came out of MassTransit) as the generic service host. That combined with the robust distributor hosting model will make “grid-friendly” nServiceBus endpoints much easier. Ayende’s posts about automatically creating queues and other “first-time-developer” features have really worked to decrease the nServiceBus learning curve - and it shows.

Documentation is a notorious problem in open-source projects and nServiceBus hasn’t escaped it. API and internal documentation is only now getting close to 100% and the samples now give developers a good start on using it. Those developers looking at swapping out certain bits of functionality have a harder time, but that’s slowly improving as well. The discussion group is the best place to get those hard-to-find answers now with a community over 200 strong.

The difficulty developers have in adopting nServiceBus is giving up “request/response” thinking. This requires an adjustment to a system’s architecture and is what most of this blog has been about. Conversely, if you have been following this blog and this thinking resonates with you, you’ll find it very simple and straight-forward. The overview page on NServiceBus.com also gives a good description of messaging basics - stuff that I’ve kind of glossed over on this blog so far.

Go on then. Take it for a spin. Write a review. Sam Gentile had this to say about v1.8:

“The bottom line is: I like what I see. Although it’s a framework, not an ESB product like Neuron, it’s a powerful framework that takes the right approach on SOA and enforces a paradigm of reliable one-way, *non-blocking* calls.”

Can’t wait to hear the response to 1.9.

Factory methods on repositories

In the podcast, Rob describes an ecommerce application with orders, customers, products - all the stuff you’ve come to expect. In his pre-DDD design, the order class had multiple constructors representing different rules. Feeling the pain in testing and maintainability, Rob looked to use DDD principles. What he did to get rid of these constructors was to make use of the repository by creating methods for the various cases, like OrderRepository.CreateOrderForGovernmentCustomer(/*data*/); .

While this is better than the multiple constructors, it still has a way to go. Analyzing what’s going on here we understand that the way the order is created is dependent upon who the user is. The rules dictating terms of payment are probably different for government customers. Not only that but we know that the order created needs to be connected to that user.

Aggregate Roots & Polymorphism

For all these reasons, it looks like user, or customer, is our aggregate root. Thus, rather than our service layer calling the above method on the repository, we first get the user object by id, then create the order like so:

IUser u = session.Get<IUser>(IdOfUserLoggedIn);
u.CreateOrder(/* data */);

This way, our service layer doesn’t need to perform all sorts of business logic (if the user is a gov’t user, do this, a corporate customer, do that, etc). All of that gets encapsulated by the domain. By leveraging some polymorphism, the session will return an instance of the correct class when we ask it for a user by id. Thus, logic relating to how gov’t users create orders is encapsulated in the GovernmentUser class.

I’ve found that this pattern of having polymorphic aggregate roots is very useful and broadly applicable.

Bounded Contexts

Further into the podcast, Rob talks about how separating his system into 2 bounded contexts simplified the code greatly. The understanding that accepting an order and fulfilling an order require different logic, different data, and thus, different domain model objects is DDD at its finest.

However, when talking about the total cost of the order, it wasn’t clear what was responsible for that. From a pure programming perspective, we might think that Total was simply a property on Order or, at most, a GetTotal() method. Yet by looking at what is involved in calculating the total of an order, a different picture emerges:

The total cost of an order obviously includes all relevant taxes. We need to take into account state and federal taxes, tax-free items at each level, etc. There’s a fair amount of logic here. Once we start to take into account promotions like “buy one, get one free”, things get even more involved. There are also cases where refunds are applied within the same order that impact the total and tax (no tax on refunds). Finally, when we include shipping charges, tax on shipping, and other rules between all of the above, it’s clear that if we put all of this in a single method, we’ve got ourselves a big bloated sack of … well, you get the picture.

Separating all of this logic into different bounded contexts makes sense.

That is, until you think about how you’re going to take these and stitch out of them a single result shown to the user. This is the advanced side of DDD and ties into SOA, so I’ll leave it for a different post.

In Closing

Working with DDD provides a great deal of value by tying our code much more closely to business concepts and encapsulating business rules in the domain model.

Starting down the DDD path is intuitive and the code that results (like u.CreateOrder) is very understandable. Yet, as more DDD principles like Bounded Contexts are put to use, developers often find themselves in unfamiliar, less-intuitive waters. This is to be expected.

Some developers (and vendors) look at DDD as nothing more than the domain model and repository patterns. The truth is that they’re just the beginning.

I hope that this post has given those of you just starting down the DDD path some feeling for how deep the rabbit hole goes, and I assure you that there are patterns in place to answer all your questions. While those beginning DDD often say that it gives names to things they’ve always been doing, or always wanted to do, I can assure you that the further down you go, that is less and less the case.

Be ready to have some basic architectural assumptions shaken

Value Objects Don’t Count

While the canonical example presented is Customer -> Address, and has a good treatment here for nHibernate, it isn’t architecturally representative.

Addresses are value objects. What this means is that if we have to instance of the Address class, and they both have the same business data, they are semantically equivalent. Customers, on the other had, are not value objects - they’re entities. If we have two customers with the same business data (both of them called Bob Smith), that does not mean they are semantically equivalent - they are not the same person.

All Entities

Therefore, for our purposes here we’ll use something different. Say we have an entity called Job which is something that a company wants to hire for. It has a title, description, skill level, and a bunch of other data. Say we also have another entity called Job Board which is where the company posts jobs so that applicants can see them, like Monster.com. A job board has a name, description, web site, referral fee, and a bunch of other data.

A job can be posted to multiple job boards. And a job board can have multiple jobs posted. A regular many to many relationship. At this point, we’re not even going to complicate the association.

This is simply represented in the DB with an association table containing two columns for each of the entity tables’ ids.

In the domain model, developers can also represent this with the Job class containing a list of JobBoard instances, and the JobBoard class containing a list of jobs.

It’s intuitive. Simple. Easy to implement. And wrong.

In order to make intelligent DDD choices, we’re going to first take what may seem to be a tangential course, but I assure you that your aggregate roots depend on it.

Moving forward with our example

Let’s say the user picks a job, and then ticks off the job boards where they want the job posted, and clicks submit.

For simplicity’s sake, at this point, let’s ignore the communication with the actual job sites, assuming that if we can get the association into the DB, magic will happen later causing the job to appear on all the sites.

Our well-intentioned developer takes the job ID, and all the job board IDs, opens a transaction, gets the job object, gets the job board objects, adds all the job board objects to the job, and commits, as follows:

   1:          public void PostJobToBoards(Guid jobId, params Guid[] boardIds)

   2:          {

   3:              using (ISession s = this.SessionFactory.OpenSession())

   4:              using (ITransaction tx = s.BeginTransaction())

   5:              {

   6:                  var job = s.Get<Job>(jobId);

   7:                  var boards = new List<JobBoard>();

   8:   

   9:                  foreach(Guid id in boardIds)

  10:                      boards.Add(s.Get<JobBoard>(id));

  11:   

  12:                  job.PostTo(boards);

  13:   

  14:                  tx.Commit();

  15:              }

  16:          }

In this code, Job is our aggregate root. You can see that is the case since Job is the entry point that the service layer code uses to interact with the domain model. Soon we’ll see why this is wrong.

** Notice that in this service layer code, our well-intentioned developer is following the rule that while you can get as many objects as you like, you are only allowed one method call on one domain object. The code called in line 12 is what you’d pretty much expect:

   1:          public void PostTo(IList<JobBoard> boards)

   2:          {

   3:              foreach(JobBoard jb in boards)

   4:              {

   5:                  this.JobBoards.Add(jb);

   6:                  jb.Jobs.Add(this);

   7:              }

   8:          }

Only that as we were committing, someone deleted one of the job boards just then. Or that someone updated the job board causing a concurrency conflict. Or anything that would cause one single association to not be created.

That would cause the whole transaction to fail and all changes to roll back.

Rightly so, thinks our well-intentioned developer.

But users don’t think like well-intentioned developers.

Partial Failures

If I were to go to the grocery store with the list my wife gave me, finding that they’re out of hazelnuts (the last item on the list), would NOT buy all the other groceries and go home empty handed, what do you think would happen?

Right. That’s how users look at us developers. Before running off and writing a bunch of code, we need to understand the business semantics of users actions, including asking about partial failures.

The list isn’t a unit of work that needs to succeed or rollback atomically. It’s actually many units of work. I mean, I wouldn’t want my wife to send me to the store 10 times to buy 10 items, so the list is really just a kind of user shortcut. Therefore, in the job board scenario, each job to job board connection is its own transaction.

This is more common than you might think.

Once you go looking for cases where the domain is forgiving of partial failures, you may start seeing more and more of them.

Aggregate Roots

In the original transaction where we tried to connect many job boards to a single job, we saw that the single job is the aggregate root. However, once we have multiple transactions, each connecting one job and one job board, the job board is just as likely an aggregate root as the job.

We can do   jobBoard.Post(job);    or     job.PostTo(jobBoard);

But we need just a bit more analysis to come to the right decision.

While we could just leave the bi-directional/circular dependency between them, it would be preferable if we could make it uni-directional instead. To do that, we need to understand their relationship:

If there was no such thing as “job”, would there be meaning to “job board” ? Probably not.

If there was no such thing as “job board”, would there be meaning to “job” ? Probably. Yes. Our company can handle the hiring process of a job regardless of whether the candidate came in through Monster.com or not.

From this we understand that the uni-directional relationship can be modelled as one-to-many from job board to job. The Job class would no longer have a collection of Job Board objects. In fact, it could even be in an assembly separate from Job Board and not reference Job Board in any way. Job Board, on the other hand, would still have a collection of Job objects.

Going back to the code above we see that the right choice is   jobBoard.Post(job);   

Job Board is the aggregate root in this case. Also, the many-to-many mapping has now dissolved, leaving behind a single one-to-many mapping.

Let that sink in a second.

But Wait…

While the GUI showing which jobs are posted on a given job board are well served by the above decision (simply traversing the object graph from Job Board to its collection of Jobs), that’s not the whole story. Another GUI needs to show administrative users which Job Boards a given Job has been posted to. Since we no longer have the domain-level connection, we can’t traverse myJob.JobBoards.

Our only option is to perform a query. That’s not so bad, but not as pretty as object traversal.

The real benefit is in chopping apart the Gordian M-to-N mapping knot and getting a cleaner, more well factored domain model.

That gives us much greater leverage for bigger, system-level decomposition.

We’re now all set to move up to a pub/sub solution between these aggregate roots, effectively upgrading them to Bounded Contexts. From there, we can move to full-blown internet-scale caching with REST for extra scalability on showing a job board with all its jobs.

In Closing

We often look at many-to-many relationships just like any other relationship. And from a purely technical perspective, we’re not wrong. However, the business reality around these relationships is often very different - forgiving of partial failures, to the point of actually requiring them.

Since the business folks who provide us with requirements rarely think of failure scenarios, they don’t specify that “between these two entities here, I don’t want transactional atomicity” (rolling our technical eyes - the idiots [sarcasm, just to make sure you don't misread me]).

Yet, if we were to spell out what the system will do under failure conditions when transactionally atomic, those same business folks will be rolling our eyes back to us.

What I’ve found surprises some DDD practitioners is how critical this issue really is to arriving at the correct aggregate roots and bounded contexts.

It’s also simple, and practical, so you won’t be offending the YAGNI police.

Related Content

From CRUD to Domain-Driven Fluency

[Podcast] Domain Models, SOA, and The Single Version of the Truth

On Web 2.0 Mashups

One would think that sites like Weather.com and friends would be the first choice for implementing such a feature. Only thing is that they were strongly against being mashed-up Web 2.0 style on the client - they had enough scalability problems of their own. Interestingly enough (or not), these partners were quite happy to publish their weather data to us and let us handle the whole scalability issue.

Implementation 1.0

The current implementation was fairly straightforward - client issues a regular web service request to the GetWeather webmethod, the server uses the user’s IP address to find out their location, then use that location to find the weather for that location in the database, and return that to the user. Standard fare for most dynamic data and the way most everybody would tell you to do it.

Only thing is that it scales like a dog.

Add Some Caching

The first thing you do when you have scalability problems and the database is the bottleneck is to cache, well, that’s what everybody says (same everybody as above).

The thing is that holding all the weather of the entire globe in memory, well, takes a lot of memory. More than is reasonable. In which case, there’s a fairly decent chance that a given request can’t be served from the cache, resulting in a query to the database, an update to the cache, which bumps out something else, in short, not a very good hit rate.

Not much bang for the buck.

If you have a single datacenter, having a caching tier that stores this data is possible, but costly. If you want a highly available, business continuity supportable, multi-datacenter infrastructure, the costs add up quite a bit quicker - to the point of not being cost effective (”You need HOW much money for weather?! We’ve got dozens more features like that in the pipe!”)

What we can do is to tell the client we’re responding to that they can cache the result, but that isn’t close to being enough for us to scale.

Look at the Data, Leverage the Internet

When you find yourself in this sort of situation, there’s really only one thing to do:

Leverage the Internet

The internet is the biggest, most scalable data serving infrastructure that mankind was lucky enough to have happen to it. However, in order to leverage it - you need to understand your data and how your users use it, and finally align yourself with the way the internet works.

Let’s say we have 1,000 users in London. All of them are going to have the same weather. If all these users come to our site in the period of a few hours and ask for the weather, they all are going to get the exact same data. The thing is that the response semantics of the GetWeather webmethod must prevent intermediaries from caching so that users in Dublin and Glasgow don’t get London weather (although at times I bet they’d like to).

REST Helps You Leverage the Internet

Rather than thinking of getting the weather as an operation/webmethod, we can represent the various locations weather data as explicit web resources, each with its own URI. Thus, the weather in London would be http://weather.myclient.com/UK/London.

If we were able to make our clients in London perform an HTTP GET on http://weather.myclient.com/UK/London then we could return headers in the HTTP response telling the intermediaries that they can cache the response for an hour, or however long we want.

That way, after the first user in London gets the weather from our servers, all the other 999 users will be getting the same data served to them from one of the intermediaries. Instead of getting hammered by millions of requests a day, the internet would shoulder easily 90% of that load making it much easier to scale. Thanks Al.

This isn’t a “cheap trick”. While being straight forward for something like weather, understanding the nature of your data and intelligently mapping that to a URI space is critical to building a scalable system, and reaping the benefits of REST.

What’s left?

The only thing that’s left is to get the client to know which URI to call. A simple matter, really.

When the user logs in, we perform the IP to location lookup and then write a cookie to the client with their location (UK/London). That cookie then stays with the user saving us from having to perform that IP to location lookup all the time. On subsequent logins, if the cookie is already there, we don’t do the lookup.

BTW, we also show the user “you’re in London, aren’t you?” with the link allowing the user to change their location, which we then update the cookie with and change the URI we get the weather from.

In Closing

While web services are great for getting a system up and running quickly and interoperably, scalability often suffers. Not so much as to be in your face, but after you’ve gone quite a ways and invested a fair amount of development in it, you find it standing between you and the scalability you seek.

Moving to REST is not about turning on the “make it restful” switch in your technology stack (ASP.NET MVC and WCF, I’m talking to you). Just like with databases there is no “make it go fast” switch - you really do need to understand your data, the various users access patterns, and the volatility of the data so that you can map it to the “right” resources and URIs.

If you do walk the RESTful path, you’ll find that the scalability that was once so distant is now within your grasp.

For those of you who might be interested in hearing a bit more of my back-story, how I got into architecture, etc - you can find all that out in the current issue of the Microsoft Architecture Journal. Here’s a quick sneak-peek:

Q: It’s sometimes frustrating when you try to keep updated on latest technologies and trends but after investing the effort on experiencing yourself (coding, watching webcasts, testing demos, etc), there’s a lot more of new technologies in the pipeline (just look back to the last PDC). Do you have any recipe to efficiently deal with such avalanche?

A: There is indeed a continuous flood of information coming at developers today. Never has so much information been so readily available and accessible to so many. It turns out, though, that much of that information is about how something works, less about what problem it is intended to solve, and much less about which problems it isn’t suited for. As an architect looking at choosing the right tool for the right job, I focus a lot more on the last two. Often, I go digging to find out why a tool isn’t a good fit for a job. So, I guess my recipe is to mostly ignore the avalanche and tap my network to find out the stuff the avalanche won’t tell you.

Read on…

Here’s the info:

Scalability is the watch-word of enterprise development. With increasing numbers of users, performing more complex tasks involving more data, collaborating with each other inside the organization and across the internet, almost all enterprise systems are being distributed to more machines, and even to data-centres across the globe. With the increased size of these systems, more developers, testers, architects, and administrators are required to bring them to production and operate them.

This 3–day course will equip you with skills in service–oriented architecture & design, transaction management, reliability & robustness, high availability & capacity planning, as well as many other design principles and practices key to tackling large-scale enterprise projects.

On top of learning how to handle the technical complexities of building enterprise–class systems, you will see the use of Domain-Driven Design and other business analysis techniques needed to decompose large intricate business domains into smaller, more manageable sub–domains. You’ll also see how to balance data–consistency and data freshness against costs as well as how to work with stakeholders to uncover the cost-effective level of each non-functional requirement.

 

By the end of this course you will be able to:

1. Avoid common pitfalls in distributed systems
2. Use loosely coupled messaging communication
3. Identify and allocated business logic to services
4. Decompose services into layers, tiers, assemblies, and processes
5. Design for service management and monitoring in production environments

Click here for more information and registration.

The question of how web-based (or 3rd party) consumers can work with pub/sub based services comes up a lot.

Many developers are used to implementing web services exposing methods on them like GetAllCustomers.

When moving to pub/sub and other more loosely coupled messaging patterns, developers look to implement the same pattern, opting for something like duplex GetCustomersRequest and GetCustomersResponse. The reasoning is simple and straightforward - it is difficult to push data over the web to consumers.

However, there are still ways to disconnect the preparation of the data from its usage thus gaining many of the advantages of pub/sub.

By employing REST principles and modelling our customer list as an explicit resource, web-based consumers would simply perform regular HTTP GET operations on the URI to get the list of customers.

The resource itself could be a simple XML file - it wouldn’t need to be dynamic at all.

You can get all the scalability benefits of pub/sub for web based consumers. All you need is a bit of REST

The quantity of deep, insightful questions was impressive and I hope that everybody got their most pressing issues addressed. I know that by the time you got home you probably had a whole bunch more - that happens all the time. This SOA/EDA style often gives a whole new perspective on system design.

You can find the slides online here.

Anyway, please feel free to send me your questions and I’ll do my best to answer them here.

When Ohad gets the recording online, I’ll make sure to link to it for those of you who couldn’t make it.

Feedback from attendees

From Ohad’s blog: “The talk was awesome … Udi really made our second usergroup meeting a huge success !”

Rotem Bloom: “היה פשוט הרצאה פצצה מי שלא מגיע מפסיד”, translated, “The presentation was ‘da bomb’, if you weren’t there, you missed out”

Adlai Maschiach: “הרצאה נ-ה-ד-ר-ת !”, translated, “A W-O-N-D-E-R-F-U-L presentation!”

Yitzhak Gootvilig: “הרצאה מעולה. מומלץ למצוא את הדרך להאזין להקלטה “, translated, “Excellent presentation. I strongly suggest finding a way to listen to the recording”

Also, “passing reference data in a message makes the message-consuming systems dependent on the knowledge and availability of actual persistent data that is stored “somewhere”. This data must separately be accessed for the sake of understanding the event that is represented by the message.”

The discussion of self-contained events can be compared to integration databases vs application databases.

Centralized Integration - Pros & Cons

If everything in a system can access a central datastore, it is enough for one party to publish an event containing only the ID of an entity that that party previously entered/updated. Upon receiving that event, a subscriber would go to the central datastore and get the fields its interested in for that ID. The advantage of this approach is that the minimal amount of data necessary crosses the network, as subscribers only retrieve the fields that interest them. Martin Fowler describes the disadvantages as:

“An integration database needs a schema that takes all its client applications into account. The resulting schema is either more general, more complex or both. The database usually is controlled by a separate group to the applications and database changes are more complex because they have to be negotiated between the database group and the various applications.”

This is far from being aligned with the principle of autonomy so important to SOA. In that respect, the architectural principle of self-contained messages points us away from those problems and towards more autonomous services.

However, once we have these autonomous business services in place, we may find that we don’t need 100% fully self-contained messages anymore.

A Real-World Example

Let’s say we have 3 business services, Sales, Fulfillment, and Billing.

Sales publishes an OrderAccepted event when it accepts an order. That event contains all the order information.

Both Fulfillment and Billing are subscribed to this event, and thus receive it.

Fulfillment does not ship products to the customer until the customer has been billed, so it just stores the order information internally, and is done.

Billing starts the process of billing the customer for their order, possibly joining several orders into a single bill. After completing this process, it publishes a CustomerBilled event containing all billing information, as well as the IDs of the orders in that bill. It does not put all the order information in that event, as it is not the authoritative owner of that data.

When Fulfillment receives the CustomerBilled event, it uses the IDs of the orders contained in the event to find the order information it previously stored internally. It does not need to call the Sales service for this information or contact some central Master Data Management system. It uses the data it has, and goes about fulfilling the orders and shipping the products to the customer, finally publishing its own OrderShipped event.

Notice, as well, that in the original OrderAccepted event there were the IDs of products the customer ordered. These product IDs originated from another service, Merchandising, responsible for the product catalog. The same thing can be said for the customer ID originating from another service - Customer Care.

The Issue of Time

One could argue that since subscribers use previously cached data when processing new events, that data might not be up to date. Also, we may have race conditions between our services. In the above example, if Billing was extremely fast and more highly available than Fulfillment. Billing could have received the OrderAccepted event, processed it, and published the CustomerBilled event before Fulfillment had received the OrderAccepted event. In short, the CustomerBilled and OrderAccepted messages could be out of order in Fulfillment’s queue.

What would Fulfillment do when trying to process the CustomerBilled message when it doesn’t have the order information?

Well, it knows that the world is parallel and non-sequential, so it does NOT return/log an error, but rather puts that message in the back of the queue to be processed again later (or maybe in some other temporary holding area). This enables the OrderAccepted message to be processed before the CustomerBilled message is retried. When the retry occurs, well, everything’s OK – it’s worked itself out over time.

In the case where we retry again and again and things don’t work themselves out (maybe the OrderAccepted event was lost), we move that message off to a different queue for something else to resolve the conflict (maybe a person, maybe software). If/when the conflict is resolved (got the Sales system / messaging system to replay the OrderAccepted event), the conflict resolver returns the CustomerBilled message to the queue, and now everything works just fine.

As all of this is occurring, the only thing that’s visible to external parties is that it happens to be taking longer than usual for the OrderShipped event to be published. In other words, time is the only difference.

Summary

The problem of non-self-contained events is mitigated first and foremost by business services in SOA, and the apparent issue of time-synchronization by business logic inside these services.

Don’t be afraid to put IDs in your messages and events.

Do be afraid of using those IDs to access datastores shared by multiple “services”.

Using IDs to correlated current events to data from previous events is not only OK, it’s to be expected.

The architectural principle of fully self-contained messages steers us away from the problems of Integration Databases and towards Application Databases, autonomous services, and a better SOA implementation. From there, following the principle of autonomy from a business perspective, will lead us to services not publishing data in their messages that is owned by other services, taking us the next step of our journey to SOA.

Related Content

[Podcast] Message Ordering - Is it cost effective?

Don’t EDA between existing systems

[Podcast] Handling dependencies between subscribers in SOA

Durable Messaging

In order to prevent a message from being lost when it is sent from a publisher to a subscriber, the message is written to disk on the publisher side, and then forwarded to the subscriber, where it is also written to disk. This store-and-forward mechanism enables our systems to gracefully recover from either side being temporarily unavailable.

In my MSDN article on this topic, I outlined some problems with this approach. These problems are exacerbated for publishers. Imagine a publisher with 40 subscribers, publishing 10 messages a second, each containing 1MB of XML. If 10 of the subscribers are unavailable, that’s 100MB of data being written to the publisher’s disk every second, 6GB every minute. That’s liable to bring down a publisher before an administrator brews a cup of coffee.

Publishers have no choice but to throw away messages after a certain period of time.

Publisher Contracts

The whole issue of contracts and schema is considered one of the better understand parts of SOA. Unfortunately, the operational aspects of service contracts is hardly ever taken into account.

On top of the schema of the messages a service publishers, additional information is needed in the contract:

1. How big will this message be?
2. How often will it be published?
3. How long will this message be stored if a subscriber is unavailable?

This first two pieces of information are important for subscribers to do load and capacity planning. The last one is the most important as it dictates the required availability and fault-tolerance characteristic of subscribers.

For Example

In the canonical retail scenario, when our sales service accepts an order, it publishes an order accepted event. Other services subscribed to this event include shipping, billing, and business intelligence.

While shipping and billing are highly available and able to keep up with the rate at which orders are accepted, the business intelligence service is not. BI has two main parts to it - a nightly batch that does the number crunching, and a UI for reporting off of the results of that number crunching. Some even do the reporting in a semi-offline fashion, emailing reports back to the user when they’re ready.

Furthermore, nobody’s going to invest in servers for making BI highly available.

And wasn’t the whole point of this publish/subscribe messaging to keep our services autonomous? That not all services have to have the same level uptime?

Houston, do we have a problem.?

Data Freshness

There is a glimmer of light in all this doom and gloom.

Not all services have the same data freshness requirements.

The business intelligence service above doesn’t need to know about orders the second they’re accepted. A daily roll-up would be fine, and an hourly roll-up bring us that much closer to “real time business intelligence”.

So, while BI is ready to accept the sales message schema, it would like a slightly different contract around it - less messages per unit of time, more data in each message.

From the operational perspective of the sales service, it would be cost effective to have less “online” subscribers. It could even take things a few steps further. Instead of using the regular messaging backbone for transmitting these hourly messages, it could use FTP. The data could even be zipped to take up even less space. Since the total data size is less than the corresponding online stream, is stored on cheaper, large storage, and the number of subscribers for this zipped, hourly update is fairly small, these messages can be kept around far longer.

If you’ve heard about consumer-driven contracts, this is it.

Note that we’re still talking about the same logical message schema.

Summary

It’s not that lost notifications aren’t a problem.

It’s that they feed the design process in such a way that the resulting service ecosystem is set up in such a way that notifications won’t get lost. I know that that sounds kind of recursive, but that’s how it works. Either subscribers take care of their SLA allowing them to process the online stream of events, or they should subscribe to a different pipe (which will have different SLA requirements, but maybe they can deal with those).

It make sense to have multiple pipes for the same logical schema.

It’s practically a necessity to make pub/sub a feasible solution.

Related Content

MSDN article on messaging and lost messages

Durable messaging dilemmas

Additional logic required for service autonomy

More in depth example on events and pub/sub between services

Consumer-Driven Contracts

You can download the presentation as a PDF here.