Starting with CRUD

Let’s start with a traditional CRUD application and work our way out from there.

In these environments, we often see users asking us to build “excel-like” screens that allow them to view a set of data as well as sort, filter, and group that data along various axes. While we might not get this requirement right away, after some time users begin to ask us to allow them to “save” a certain “query” that they have set up, providing it some kind of name.

That, right there, is a task-based query and it is the beginning of deeper domain insight.

Pattern matching

Any time a user is repeatedly running the same query (this can be once a day or some other unit of time) there is some scenario that the business is trying to identify and is using that user as a pattern-matching engine to see if the data indicates that that scenario has occurred.

It’s quite common for us to get a requirement to add some field (often a boolean or enum) to an entity which defaults to some value and then see that same field used in filtering other queries. These measures are sometimes instituted as a temporary stop-gap while a larger feature is being implemented, though (as the saying goes) there is nothing more permanent than a temporary solution.

Where we developers go wrong

The thing is, many developers don’t notice these sorts of things happening because we don’t actually look at the kinds of queries users are running.

One excellent technique to better understand a domain is to sit down with your users while they’re working and ask them, “what made you run that query just now?”, “why that specific set of filters?”.

What I’ve noticed over the years is that our users find very creative ways to achieve their business objectives despite the limitations of the system that they’re working with. We developers ultimately see these as requirements, but they are better interpreted as workarounds.

I’ll talk some more about how a software development organization should deal with these workarounds in a future post, but I want to focus back in on the queries for now.

Oh, and don’t get me started on caching or NoSQL, not that I think that those tools don’t provide value – they do, but they’re only relevant once you know which business problem you’re solving and why.

Not all queries are created equal

Even before bringing up the questions I described in the previous section, any time you get query-centric requirements the first question to ask is “how often will the user be running this specific query?”.

If the answer is that the specific query will be run periodically (every day, week, etc), then drill deeper to see what pattern the user will be looking for in the data. If the person you’re talking to doesn’t know to answer that question, then go find someone who does. Every periodic query I’ve seen has some pattern behind it – and in my conversations with thousands of other developers over the years, I’ve seen that this is not just my personal experience.

But there is a case where a query does get run repeatedly without there being a pattern behind it.

I know this sounds like I’m contradicting myself, but the distinction is the word “specific” that I emphasized above.

There are certain users who behave very differently from other users – these users are often doing what I call research, i.e. the “I don’t know what I’m looking for but I’ll know it when I see it” people.

These researchers tend to repeatedly query the data in the system however they tend to run different queries all the time. This is the reason why traditional data warehouse type solutions don’t tend to work well for them. Data warehouses are optimized for running specific queries repeatedly.

Keeping the Single-Responsibility Principle in mind – we should not try to create a single query mechanism that will address these two very different and independently evolving needs.

And now on to Search

Search is a feature that is needed in many systems and whose complexity is greatly underestimated.

While the developer community has taken some decent strides in understanding that search needs to be treated differently from other queries, the common Lucene/Solr solutions that are applied are often overwhelmed by the size of the data set on which the business operates.

The problem is compounded by our user population being spoiled by Google – that simple little text box and voila, exactly what you’re looking for magically appears instantaneously. They don’t understand (or care) how much engineering effort went into making that “just work”.

Lucene and Solr work well when your data set isn’t too large, and then they become pretty useless as the quality of their results degrades. The thing is that many of us in IT tend to work on projects where we have an unrealistically small data set that we use to test the system and, at these volumes, it looks like our solutions work great. But if you have 20 million customers, do you think a full text search on “Smith” is going to find just the right one?

Larger data sets require a relevance engine – something that feeds off of what users do AFTER the query to influence the results of future queries. Did the user page to the next screen? That needs to be fed back in. Did they click on one of the results? That needs to be fed back in too. Did they go back to the search and do another similar search right after looking at a result – that should possibly undo the previous feedback.

And that’s just relevance for beginners.

You know what makes Google, you know, Google? It’s that they have this absolutely massive data set of what users do after the query that informs which results they return when. You probably don’t have that. That and search is/was their main business for many years – I’m betting that it’s not your main business.

You should discuss this with your stakeholders the next time they ask for search functionality in your system.

In closing

I know that the common CQRS talking points tell you to keep your queries simple, but that doesn’t mean that simple is easy.

It takes a fair bit of domain understanding to figure out what the queries in the system are supposed to be – what tasks users are trying to achieve through these queries. And even when you do reach this understanding, convincing various business stakeholders to change the design of the UI to reflect these insights is far from easy.

It often seems like the reasonable solution to give our users everything, to not limit them in any way, and then they’ll be able to do anything. What ends up happening is that our users end up drowning in a sea of data, unable to see the forest for the trees, ultimately resulting in the company not noticing important trends quickly enough (or at all) and therefore making poor business decisions.

Even if your company doesn’t believe itself to be in “Big Data” territory, I’d suggest talking with the people on the “front lines” just in case. Many of them will report feeling overwhelmed by the quantity of stuff (to use the correct scientific term) they need to deal with.

It’s not about Lucene, Solr, OData, SSRS, or any other technology.

It’s on you. Go get ‘em.

If you want to come please don’t wait to register.

We’re also going to be re-recording the course as the recording from Denver turned out pretty bad, only this time with a much more professional crew. This means that attendees will be getting the recording of the course which they themselves attended instead of that of a previous group – an added bonus.

Just as a reminder – if your company won’t pay for the course and you still want to come, there are significant discounts available. Just send me an email to training@udidahan.com.

Hope to see you there.

Register here

* PS – After this will be Stockholm and then London and that’ll be all folks!

Enjoy:

060518ud01.mp3

060808ud01.mp3

060831ud01.mp3

061017ud01.mp3

In any case, I’ll be giving the course 4 more times, and that’ll be it.

When I was in Denver last month, I had the SOA course professionally recorded so if you can’t make any of these dates, you’ll still be able to get the videos. If you bought the videos in the past, don’t worry, I’ll be getting you the new videos as soon as they’re available. The NServiceBus courses delivered by other trainers will continue to run the same as before.

This change is part of a larger change in priorities for me. There are some big things coming that I’m very excited about that require quite a bit more of my attention.

Anyway, hope to see you either in Oslo, New York, Stockholm, or London.

I think the reason for this interest is the rise in popularity of cloud platforms and queuing systems like RabbitMQ (which don’t support distributed transactions) and the difficulty of setting up distributed transactions even in on-premise.

Of course, there’s also the regular scalability hand-wringing going on even though most people wouldn’t bump up against those limits anyway.

In this post, I’ll talk about the nature of the problem, explain the pitfalls in some of the common solutions, but I’ll put off the description of how to provide consistency without distributed transactions to a future post as this one is already going to be quite long.

I’ll start with the basic fault-tolerance issues and then explain how things spiral out from there.

Starting with the basics

OK, so we have a queuing system in place that dispatches messages to our business logic which does some transactional work against a database.

Let’s say that we completed the transaction against our database but before we could acknowledge to the queue that the message was processed successfully, our machine crashed. What our machine comes up again, the queue will once again dispatch us the same message. Unless we have some logic to detect that we’ve already processed it (called “idempotence” in the REST community), we will end up processing it again.

In short, the problem is duplicates.

Attempted solutions to the duplicate problem

Most queuing systems don’t do anything about duplicates, actually giving it a proper architectural name: At-least-once message delivery, as opposed to the Once-and-only-once model that a queue that supports distributed transaction provides.

The solution often suggested is to have your logic check to see if it has already processed a message with that ID before – in essence storing the ID of each message processed for some period of time. Of course, there is some performance overhead with that, but it might be a small price to pay compared to dealing with it in the logic of every use case.

On the other hand, you’ll often have some messages (like Update commands) for which it looks like you can safely process them multiple times, in which case you might want not to pay the performance overhead there. The thing is, if your logic publishes an event in addition to the regular database work (something that is quite common) and you process the same message twice, you will probably end up publishing the event twice as well.

These duplicates are different in that here we have two distinct messages with different IDs that contain the same business data. This means that recipients of these messages will not be able to filter them out at an infrastructure level anymore.

NOTE: Deduplication abilities in queues

Although the Azure Service Bus doesn’t support distributed transactions meaning you still have the issue mentioned above, Microsoft added the ability to detect and filter out duplicates based on message contents rather than just the ID. This helps quite a bit but it’s important to understand that that doesn’t cover everything for you. Let me explain:

More complex logic

In some of your most important use cases, you may have both entity updates as well as entity creation happening together in your domain model. You might be using some kind of event model (like I wrote about here) to percolate out the information that an entity was created in order to keep your service layer decoupled from the internals of the domain model.

In the callback code from these domain events, you will likely publish out an event on the queuing system containing information like the ID of the entities created as well as other business data. And there’s the rub.

You see, without distributed transactions, you can run into some problematic scenarios:

For example, if you don’t make sure that your event publishing calls to the queuing system include the same transaction object as the one you used when retrieving the original message from the queue, then those calls could “escape” before you know if the database transaction is going to succeed. Deadlocks always happen at the lousiest times. Anyway, if you’re using database generated IDs for your entities, then those IDs will get published out in events despite the database rolling back and your subscribers will now be making decisions on wrong data – not just eventually consistent data.

In this case, processing the message again doesn’t really solve the problem – it just means that you’ll be publishing events with different IDs, so an infrastructure like Azure Service Bus couldn’t really de-duplicate them.

On the other hand, if you do use the same transaction and combine in the infrastructural message ID based de-duplication described above (as identifying duplicate calls for complex business logic is damn hard), you’ll run into another problem.

Consider what would happen if your server crashes right after finishing its database work but before it completes the transaction against the queuing system. When going to retry the message, the infrastructure filtering thing would know not to call your business logic again and that message would be quietly swallowed. Unfortunately, the event publishing calls to the queuing system from the first time the message was processed were rolled back and since your business logic isn’t called again, the event publishing won’t happen again.

Oops.

In closing

I hope I’ve been able to clarify what kind of scenarios distributed transactions solve for you and some of the difficulties in solving them yourself.

Now, to be clear, you could solve these problems by going in-depth on each of your use cases, analyzing the consistency needs and structuring the code differently to address those needs. But give this another thought, if our consistency is dependent on calling otherwise independent APIs in exactly the right order, and that a change in this order would not cause any visible functional effects, what would happen when developers with less expertise maintain this code?

The folks in the event sourcing community have their solution to this which is based on writing their business logic differently. As the adoption of this pattern is still pretty limited (probably still in the Innovator section of the Technology Adoption Curve), it’ll be interesting to see how it holds up with larger teams in the mainstream.

Oh, and in case it wasn’t clear from before, the guys in the REST community haven’t even begun addressing this problem when it comes to server-to-server integration.

We’re working on a solution for this with NServiceBus that won’t require you to change how you write business logic. We’ve got one big release to do before we can roll this in, and that’s coming soon (with all sorts of cool things like support for ActiveMQ and queues in the database). The solution we’ve found is architecturally sound but you’ll have to wait for my next post to hear about it.

Stay tuned.

A short intro to SOA, done right

In the “vertical services” approach I espouse, you often see components from multiple services deployed to any given endpoint. While these services usually don’t need to communicate with each other at all, occasionally you’ll see them leaving “breadcrumbs” behind for each other – things like UserId or OrderId in the session. What’s especially important is that these IDs are accessible even before the entity is finalized – this enables each service to collect its own data without needing any other service to know about that data.

In an ecommerce environment, we would see one service owning money, another the product catalog (excluding prices, those would be owned by the previous service), another service owning the customers’ payment info (credit cards, etc), and yet another owning shipping addresses – all of these separate from the one that owns the shopping cart. Let’s call these Finance, Catalog, Payment, Shipping, and finally Shopping – just so that we have something to reference later.

The API

While we can do all sorts of cool browser composition with the UI in our own system, enabling each service to collect and display its own information, if we want to expose an API for clients to call, we wouldn’t want to force those clients to have to make a separate call to each service in order to make a purchase. Instead, we’d want something that looks like:

MakePurchase(Guid orderId, Dictionary cart, CreditCardInfo cc, Address shippingAddress)

In case you were wondering, the service which owns the definition of this API is different from all of the above services – it is a service that is primarily technical in nature and is responsible for things like integration and data transformations. I call this service IT/Ops.

Getting the data from the API to the services

So, we don’t want any of our business-centric services to know about anybody else’s data structures, so that leaves it to IT/Ops to pass the data to them. The thing is that we still want to do that in the most loosely-coupled way possible – with messaging being a good candidate for that.

So, what we’ll do is have IT/Ops send a message containing the data to the other services, but with a slight twist.

Here’s what the code would look like with NServiceBus:

Bus.SendLocal(new Order { Id = orderId, Cart = cart,
                          CreditCard = cc, ShippingAddress = shippingAddress });

Why the SendLocal?

So that the components from the other services can all run together with IT/Ops in the same process giving a nice tight deployment model.

Before we get to the services, let me show you the Order class – specifically, the interfaces it implements:

public class Order : ShoppingOrder, PaymentsOrder, ShippingOrder
{
    public Guid OrderId { get; set; }
    public Dictionary<Guid, int> Cart { get; set; }
    public CreditCardInfo CreditCard { get; set; }
    public Address ShippingAddress { get; set; }
}

public interface ShoppingOrder
{
    Guid OrderId { get; set; }
    Dictionary<Guid, int> Cart { get; set; }
}

public interface PaymentsOrder
{
    Guid OrderId { get; set; }
    CreditCardInfo CreditCard { get; set; }
}

public interface ShippingOrder
{
    Guid OrderId { get; set; }
    Address ShippingAddress { get; set; }
}

Each of the above interfaces represents the data that each service cares about. Therefore, each service will provide an assembly that handles that message/data and persists it to its database, like this:

public class ShoppingAPIHandler : IHandleMessages<ShoppingOrder>
{
    public void Handle(ShoppingOrder message)
    {
        //persist to shopping service db
    }
}

public class PaymentsAPIHandler : IHandleMessages<PaymentsOrder>
{
    public void Handle(PaymentsOrder message)
    {
        //persist to payment service db
    }
}

public class ShippingAPIHandler : IHandleMessages<ShippingOrder>
{
    public void Handle(ShippingOrder message)
    {
        //persist to shipping service db
    }
}

Since the Order object being sent is a polymorphic match for all of these interfaces, NServiceBus knows to invoke all of these handlers. By the way, if you care about the order of invocation, then you can control that as well (but I won’t get into that here).

Also, since all of these handler assemblies are deployed to the same endpoint, and the Order object is sent just once, this means that all the handlers will be invoked in a single transaction on a single thread – either all of them succeeding, or all failing. Since they’re all connected on the same Order Id, referential integrity can be preserved as well.

Wrapping up

When you are building a system on SOA principles, you’ll often find that you need a service like IT/Ops to handle data transformation and other broker-centric tasks. While much of SOA is based on the Bus Architectural Style – meaning primarily publish/subscribe interaction between services – that doesn’t mean that your business-centric services cannot have their components deployed in the same process.

I’d go so far as to say that if you aren’t deploying components from multiple services in process with each other at least some of the time then it’s quite likely that your service boundaries are probably incorrect.

Anyway, I hope you found this post interesting. Shout out to Slawek who gave me the idea for this post.

By the way, if you’d like to learn more about these kinds of patterns, the next batch of courses is open for registration – but the early bird prices are almost over, so you’d better hurry.

Register for   Denver CO, USA,     Bad Ems, Germany,     Perth, Australia.

JPA is the Java Persistence API – a kind of ORM, as you’d expect. Luckily, a lot of the web services stuff was already on the way out by the time that EclipseLink DBWS came out. DBWS allowed you to expose database artifacts as web services.

I mean, it’s not like we have any other interoperable ways of accessing data, right?

Anyway, like I said, that didn’t take off, but now they’re reinventing it – this time with REST!

In case you had any doubts, REST is pure awesomeness and adding it to anything else makes it awesome too. Lest anybody take this out of context (it’s happened before), I’m being sarcastic.

Here it is.

God knows they couldn’t let Microsoft totally dominate this area with OData coming out quite some time ago. In case you were wondering, OData was designed to provide standard CRUD access of a data source over HTTP.

Of course, none of these support any transactions so if you actually wanted to do some meaningful business logic on top of this CRUD, you wouldn’t have any consistency. And, let’s face it, if you’re not doing any meaningful business logic, just basic persistence, you just do it. That problem’s been solved a long time ago.

Can we please stop reinventing SQL already?

Spain

Marçal Serrate will be teaching the Enterprise Development with NServiceBus course in Spanish in:

• Barcelona on Nov 26 – register here
• Madrid on Dec 10 – register here

Marçal has been working with NServiceBus for a couple of years now building a very large system for an insurance company and knows all the ins and outs of taking NServiceBus live at scale – including all the infrastructure issues of big deployments.

We’re very happy to have him bringing NServiceBus to the Spanish-speaking community.

Other locations

Jimmy Bogard will be giving the next NServiceBus course in Austin in 2 weeks – Oct 23 and then heading over to London to teach it with our good friends from Skills Matter on Dec 4.

SOA – US, UK, AU, IL, DE

In addition to teaching my 5-day course in Miami on Nov 5, I’ll also be dropping by the Tallahassee Code Camp as a guest on the Dot Net Rocks Road Trip, so if you’re in the area, come by and say hello.

I’m also going to be teaching my course in Denver Colorado early next year, Jan 14 and it’ll be my first time there. I’m still looking for the right venue so if you have any recommendations, I’d love to hear them.

Of course, I will be back in London one more time this year, Dec 3 and won’t be forgetting my local peeps in Israel either – that’s on Dec 23.

Since I had such a wonderful time in Germany last time, I’ll also be back again next year on Jan 28 because who couldn’t appreciate a venue like the Hackers Hotel, not just for its name, but the chairs in the dining room are kind of like thrones – makes you feel like you’re getting the royal treatment.

Finally, for the guys down-under – this time I’m going to Perth. The past 3 years were always Sydney, Sydney, Sydney and I think that I should see more of the country. We’ve just about finalized the venue with Cliftons but figured that we might as well open up registration too – the course will be on Feb 11.

In closing

Yes, just like this past year, I’ll be travelling quite a bit – trying to get the good word out. I hope you’ll be able to make one of these dates – it’s always great meeting some of my long-time blog readers.

One time I gave the course this year, and there wasn’t a single person who’d read my blog there – and in a public course, no less! Not that I’m complaining, of course, because often times people who’ve already been on the course recommend it to their colleagues who might not be big blog readers, but it was kind of a surreal moment for me.

Anyway, that’s all the announcements for now.

If you need the full information on these courses, you can see it all up on my training page here.

I’ve heard from countless developers and architects over the years about their frustration in getting everybody else to go along with them. “Can’t they see how much better [new approach] is over what we’re doing now?!” they ask, believing that things could and actually would be evaluated on their merits, especially in a rational field like IT.

The usual explanation I give has a couple of parts.

Conway’s Law

In 1968 Melvin Conway penned what later became known as Conway’s Law which stated:

“organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.”

An important corollary of that law is that if you wish to have a significant impact on the design of a system, you would need to have a similarly significant impact on the communication structure of the organization making that system.

The main problem is that the people that tend to be pushing for DDD, IoC, CQRS, SOA, etc are usually not as strong when it comes to the soft skills that are so necessary for bringing about organizational change. The thing is that, at a minimum, these types of changes take 3 to 5 years so it really takes a long-term commitment, both from the individual and the organization.

On the rationality of people in IT

First of all, people are a whole lot less rational than they’d like to believe – or that they’d like other people to notice. In fact, people will go to great lengths to maintain the appearance of consistency and rationality, even at the cost of harm to themselves. How’s that for irrational?

Don’t take my word for it – there’s a great book on the topic: Predictably Irrational: The Hidden Forces That Shape Our Decisions. The somewhat scary thing about it is that not only are we irrational beings, but that that irrationality can be predicted and, yes, even manipulated.

Once you can understand that the people you’re trying to convince aren’t Vulcan, you have a much better chance of being effective. I’d say that, for myself, understanding my own modes of irrationality increased my effectiveness as well, and made me quite a bit happier in life too.

Why you need to bring in a consultant

This isn’t me hawking my wares – believe me, I’m busy enough as it is, but let me know when this starts to sound familiar to you.

There’s a problem in your organization – could be that you’re not delivering software fast enough, high enough quality, whatever. Suffice it to say that Management isn’t happy. You’ve been living this pain for a while and know exactly what the source of the problem is (more often than not, management has at least a hand if not a whole arm in it). You come up with some recommendations, bring them to the higher-ups, but ultimately are ignored, dismissed, or don’t even get into the room.

Some time later, management brings in a Consultant (that’s right, with a capital ‘C’) who is there to figure out what’s wrong and come up with recommendations. In some cases, especially in larger organizations, they bring in a whole bunch of them from a brand name like McKinsey or Ernst & Young.

If these guys are smart, they listen to you, ultimately presenting your analysis and recommendations to management. Of course, those higher-ups are in awe of how quickly these guys were able to understand the inner workings of their organization. That awe lends instant credibility to their recommendations which are then adopted and given powerful political backing.

And you’re sitting there thinking, “but… but… but that’s what I was saying!!”.

It’s not the message – it’s the messenger.

Let me put it another way, explained from the perspective of management – we’re having problems, you work here, ergo you’re part of the problem. Also, you don’t make that much money (compared to management), so how smart could you be? Those brand-name consultants, well, they cost a LOT, so they MUST be good (good enough to know not to work here too).

Therefore the more the consultant costs, the more likely management is to listen, which ultimately creates the conditions for success, which makes the change happen, which proves to management that they were right to bring in an expensive consultant. A vicious (or virtuous) cycle – depending on how you look at it.

Now, it doesn’t always work this way, but it does often enough to perpetuate management’s world view.

In Closing

I do hope your organization and its leaders aren’t trapped in this kind of dysfunction, but if they are, know that you’re not alone and that you can get help – either via consulting or in some books:

Some good books include Made to Stick: Why Some Ideas Survive and Others Die and the grandfather of the field: How to Win Friends & Influence People. There are countless others and there isn’t any right place to start – the most important thing to do is to start.

It’s been over 40 years since Melvin Conway’s observation and, as an industry, we’re still relearning these things – usually through the school of hard knocks. But there is an upside here – I’m pretty sure that, knowing these patterns, you could pick up on some signals during the interviewing process and find a company that’s outgrown many of these issues – one that would be able to have more meritocratic discussions on technical choices.

In the worst case, you could become a consultant and make a living off of all this irrationality

Here’s the punchline:

“By using messaging, queues, and publish/subscribe patterns, we were able to upgrade a core part of our system without touching any other parts – even those that it communicates with on an ongoing basis. During the short period of time that it was down, the queues took care of buffering the communication for that service without affecting anything else. The system as a whole was never really down.

This was a totally new experience for our company – an upgrade without downtime, even with the thousands of websites we’re running.”

Read the full story.

By the way, if your in the Netherlands next week, you can catch Dennis speaking about these topics at the Dutch .NET User Group. You can get all the info and register here.

It’s something I touched on in my presentation last week about Balancing Architecture & Agile at the Agile Practitioners group in Israel but didn’t have as much time to get into.

Agile

You see, at the beginning of these projects, often our users or customers don’t exactly know what they want. Even if you go through some process of doing mock-ups, sometimes users can’t really know what they want until they can interact with working software.

This is one of the advantages of an agile approach that focuses on getting working software in the hands of users very early. It allows us to mitigate the risk of us building the wrong system.

Pitfalls

The mistake here is believing that this needs to be done in a production-ready manner, using all of the super-scalability and highly reliable infrastructural elements in our target deployment environment.

While there is value in building real functionality on top of your production infrastructure to learn more about the challenges in doing that and in that process, refining the API of your platform, baking more capabilities into it, etc, the problem is that you are slowing down the important learning of what system needs to be built.

Decouple those two things.

Build one to throw away.

Sometimes, the way to be most effective is to be somewhat inefficient.

Once you’ve gotten your users to the point that they can point at a piece of working software and say, “Yes – build me that”, then you can go about building it the right way on top of your production platform.

Let me go a step further, just to be crystal clear.

Do not unit-test that code. Do not use TDD, DDD, CQRS, Event Sourcing, SOA, etc.

Just get it done.

Architecture

You also want to have a good understanding of the use cases in your system before you make too many big architectural decisions (where that might even be just 1 decision, but you only find out after the fact). Some use cases are architecturally significant – many aren’t. This falls under the “just enough architecture up-front”, but is really very dependent on the experience and skills of the people on your team so I really can’t give you anything more prescriptive.

Of course, there’s another risk around this approach.

Organizational Dysfunction

In some organizations, there isn’t enough of an understanding or appreciation of the difference between production-ready software and a prototype. I think we’ve all been through this at one time or another. If users see “the software” works, we might be forced to deploy the prototype – all our protests for naught.

If this is the environment you’re in, don’t worry about it.

Seriously – don’t.

Growing up

You can explain to a child that fire is hot, and that they will get burned if they stick their hand in it, but all of that is theoretical – it doesn’t really sink in, because they haven’t yet had a visceral experience of “hot”.

If the organization you’re in isn’t that mature yet, they’re going to have to get burned a couple of times until they learn. The only thing you can do as a responsible adult is to *try* to get them started with smaller burns so that they’ll learn the lessons with the minimal amount of pain.

Given that the people doing the learning are higher up the food-chain, you really don’t have much influence. Accept that and move on. If necessary, cover your assets – write an email containing the “meeting minutes” in which you describe the concerns that you raised and that the decision was made against your recommendations – but phrase it gently. Depending on the state of the economy and your network, it might be a good time to start job-hunting.

In closing

I said it before but it bears repeating:

Efficiency is secondary to effectiveness.

You want to make sure your headed in the right direction before putting the pedal to the metal.

Rework is to be accepted and, yes, even embraced.

There’s even a book about it.

Have fun.

“It’s the first project I’ve been able to use the principles from your course and the resulting codebase is so simple. I was able to add reporting module without changing a line of existing code just by subscribing to events. Awesome. Keeping everything explicit is the best was to keep it simple which i find is easier developing for mobile. I’m sure there is lots I could improve. I’d like to take course again one day.”

This right here is what I use as an answer when people ask me, “don’t you get tired of giving the same course over and over again?”

Quite simply – I don’t. There’s really nothing quite like having this kind of impact on people.

First big update released of our system based on @udidahan ADSD training, makes a maintainable system & site updates in <1sec instead of 24h

— Dennis van der Stelt (@dvdstelt) September 1, 2012

It’s not just about getting the system built quickly the first time, and it’s not just about having a maintainable code base longer term, equally important is the ability to quickly deploy major releases to the system without downtime – ‘cuz when you’re system is down, it’s not providing any business value (which is what being Agile is all about).

What do you say, Dennis? Tell us some more!

UPDATE

Dennis gives the full story here.

These full-day events will include topics like CQRS, REST, as well as async and functional programming.

Full information and registration is available here.

Full information here.

Here’s the full information.

CQRS

As his been said several times before by many others, it’s not the denormalized view model that defines CQRS.

One of the things that sometimes surprising people after going through my course is that in most cases you don’t need a denormalized view model, or at least, not the kind you think. Yes, that’s right: MOST cases.

But I don’t want to get too deep into the CQRS thing in this post – that can wait.

SOA

The big thing I’m against is raw business data being duplicated between services.

Data that can be expected to be accessible in multiple services includes things like identifiers, status information, and date-times. These date-times are used to anchor the status changes in time so that our system will behave correctly even if data/messages are processed out of order. Not all status information necessarily needs to be anchored in time explicitly – sometimes this can be implicit to the context of a given flow through the system.

For example, the Amazon.com checkout workflow.

In that flow, if you provide a shipping address that is in the US, you are presented with one set of options for shipping speed, whereas an international address will lead you to a different set of options.

Assuming that the address information of the customer and the shipping speed options are in different services, we need to propagate the status InternationalAddress(true/false) between these services in that same flow. In this case, there isn’t a need to explicitly anchor that status in time.

But what’s so bad about duplication of data between services?

The danger is that functionality ultimately follows raw business data.

You start with something small like having product prices in the catalog service, the order service, and the invoice service. Then, when you get requirements around supporting multiple currencies, you now need to implement that logic in multiple places, or create a shared library that all the services depend on.

These dependencies creep up on you slowly, tying your shoelaces together, gradually slowing down the pace of development, undermining the stability of your code pace where changes to one part of the system break other parts. It’s a slow death by a thousand cuts, and as a result nobody is exactly sure what big decision we made that caused everything to go so bad.

That’s the thing, it wasn’t viewed as a “big decision” but rather as just one “pragmatic choice” for that specific case. The first one excuses the second, which paves the way for third, and from that point on, it’s a “pattern” – how we do things around here; the proverbial slippery slope.

So what’s with the word “Replication” in the title of this post?

While data duplication between services is very dangerous, replication of business data WITHIN a service is perfectly alright.

Let’s get back into multi-site scenarios, like a retail chain that has a headquarters (HQ) and many stores. Prices are pushed out from the HQ and orders are pushed back from the stores according to some schedule.

We know that we can’t guarantee a perfect connection between all stores and the HQ at all times, therefore we copy the prices published from the HQ and store them locally in the store. Also, since we want to perform top-level analytics on the orders made at the various stores, that would be best done by having all of those orders copied locally at the HQ as well.

We should not view this movement of data from one physical location to another as duplication, but rather as replication done for performance reasons. If there were some magical always-on zero-latency network that existed, we wouldn’t need to do any of this replication.

And that’s just the thing – logical boundaries should not be impacted by these types of physical infrastructure choices (generally speaking). Since services are aligned with logical boundaries, we should expect to see them cross physical boundaries – this includes SYSTEM boundaries (since a system is really nothing more than a unit of deployment).

I know that you might be reading that and thinking “What!?” but there isn’t enough time to get into this in any more depth here. You can read some of my previous posts on the topic of SOA for more info here.

Cross-site integration without replication

There are some domains where sensitive data cannot be allowed to “rest” just anywhere. Let’s look at a healthcare environment where we’re integrating data from multiple hospitals and care providers. While all of these partners are interested in working together to make sure that patients get the best care, which means that they need to share their data with each other, they don’t want any of THEIR data to remain at any partner sites afterwards (and are quite adamant about this).

In these cases, the decision was made that performance is less important than data ownership. Personally, I don’t agree with this mindset. The fact that data is “at rest” in a location as opposed to “in flight” does not change ownership. It could be stored in an encrypted manner so that only a certain application could use it, resulting in the same overall effect, but this is an argument that I’ve never won.

People (as physical beings) put a great deal of emphasis on the physical locations of things. It’s understandable but quite counterproductive when dealing with the more abstract domain of software.

In closing

By virtue of the fact that we don’t duplicate raw business data between services, that means that the regular data structures inside a service already look very different from what they would have looked like in a traditional layered architecture with an ORM-persisted entity model.

In fact, you probably wouldn’t see very many relationships between entities at all.

Going beyond that, you probably wouldn’t see the same entities you had before. An Order wouldn’t exist the way you expect; addresses (billing and shipping) would be stored (indexed by OrderID) in one service whereas the shipping speed (also indexed by OrderId) would be in another, and the prices may well be in yet another.

It is in this manner that data does not end up being duplicated between services, but rather is composed by many services whether that is in the UI of one system, the print-outs down by a second system, or in the integration with 3rd parties done by a third system.

If performance needs to be improved, look at having these services replicate their data from one physical system to another – in-memory caching is one way of doing this, denormalized view models might be though of as another (until you realize there isn’t very much normalization within a service to begin with).

And a word from our sponsor

For those of you on “rewrite that big-ball-of-mud” projects looking to use these principles, I strongly suggest coming on one of my courses. The next one is in San Francisco and I’ve just opened up the registration for Miami.

For those of you on the other side of the Atlantic, the next courses will be in Stockholm in October and in London this December.

The schedule for next year is also coming together and it will include South Africa and Australia too.

Anyway, here’s what one attendee had to say after taking the course earlier this month:

I wanted to thank you for the excellent workshop in Toronto last week. I spent the better part of the weekend reflecting over what was presented, the insights we learned through the group exercises, and how my preconceptions of SOA have changed. By the end of the course, all the tidbits of (usually) rather ambiguous information that I’ve collected from various blogs, books, and other sources, finally coalesced into something more intelligible – one big A-HA moment if you will. Overall, I found the content of the workshop to be incredibly enlightening and it left me feeling invigorated and excited to learn more.
- Joel from Canada

Hope you’ll be able to make it.

If travel is out of the question for you, you can also look at get a recording of the course here.

One final thing

If your employer won’t foot the bill for these, please get in touch with me.
I wouldn’t want you not to be able to come just because you’re paying out of pocket.

There are very substantial discounts available.

Contact me.

Here’s the plan:

• Sep 25 in Berlin
• Sep 26 in Dresden
• Sep 27 in Leipzig

If you’re in and around one of those areas, drop by and support the community. If you have some friends or colleagues that are working on distributed systems, drag them along as well. They’ll thank you for it later.

This is something I talk about in the first day of my course when going through the fallacies of distributed computing – specifically fallacy #3 which talks about bandwidth.

What about bandwidth?

When it comes to network bandwidth in your datacenter, there’s a pretty good chance you’re still on gigabit ethernet. When most developers hear that prefix, “giga”, there is an instantaneous translation in their brain to “so much I don’t need to worry about it”.

The thing is that it’s GigaBIT, not GigaByte, so we’re talking about 128 MBps.

Also, keep in mind that hardly anybody programs at the level of ethernet, we’re several layers up the stack. You can expect roughly 40% the bandwidth of ethernet up in TCP land due to its collision detection, exponential backoff, etc. So that’s roughly 50MBps, not counting overhead for serialization (which can be very significant if it’s text-based like XML or JSON).

In practice, you might be getting something like 25MBps – definitely not so much that you don’t even need to think about it.

Everybody’s talking scalability in terms of number of servers and memory, storage, CPU per server – but what about the network? More importantly, what happens when (not if) you run out? Well, the latency of your calls increase – and that can be quite substantial.

Business Priority

And now we get to the crux of the matter.

Consider a “Customer” web service with a bunch of methods on it, including these two: GenerateTopCustomersByRegionReport and MarkCustomerAsFraud.

Now, your system is under significant load and there’s just enough bandwidth left for one call to make it across the network without timing out. Two users invoke the functionality above – one doing the report, the other doing the fraud. Should the fact that one user clicked the button a millisecond before the other mean that the MarkCustomerAsFraud should be delayed to the point of failure?

I’m fairly sure that if we asked a business stakeholder the answer would be a clear no.

While we could try asking the network engineers to give higher priority to that webmethod, let’s face it, that’s never going to happen. Since both methods are on the same web service, clients are bound to the address where that web service is hosted, regardless of which methods they call.

The problem with “the” network

If there’s one word I loathe in the English language, it’s the word “the”.

Such a small word, tacked on in front of so many other words that, without you even noticing it, traps your mind into thinking you can have only one of that thing.

The network.
The database.

But you CAN have more than one – it’s YOUR system. Design it however you like.

Most servers (if not all) can have more than one network card these days. And even if yours couldn’t – you can ask your network engineers to set up multiple virtual networks on top of the physical network and divide up the bandwidth between them.

Putting it together

The next step is to simply put the MarkAsFraud method on a different web service. This way, you can decide at deployment time which web service should be hosted over which network.

When your system is under load, you will then be guaranteed that even if there are a large number of low priority calls being invoked, they will not use up the network bandwidth reserved for your higher priority calls. You will likely still need to take care of processing and other IO concerns on your servers, but if worst comes to worst, you can partition your server farm as well.

While this may sound a bit CQRS-ish but it would be more accurate to say that CQRS is a more specific case of this pattern – that of partitioning the API according to business priority.

One of the interesting things about messaging is that we tend to forgo the traditional “service contracts” where many methods are put together on a single “service”. Instead, each message definition stands on its own and can be routed to any destination.

In summary

If you are still using WCF and web services, be aware that these apparently little things can have an impact on how your system behaves under load. Even if you do use MSMQ under WCF, the traditional service contract made of multiple methods will still govern your routing.

If you do go all the way with this pattern, you’ll see that each of your service contracts ends up with only one method on it. This might make you wonder what’s the point of the whole service contract thing in WCF – that would be a very good issue to resolve.

Remember the first rule of remote communication – Don’t.

Take this “Bare-knuckle SOA” thing that’s started making the rounds. Not to say that the approach doesn’t have its merits (regardless of the SOA thing), but it seems to be talking about little more than regular XML and HTTP communication. Personally, I think that the functional testing done over HTTP in that post would have been much better done directly against the API of the component, bypassing HTTP entirely.

Appending the word “service” to a concept doesn’t change any of its architectural properties.

EmailService, CurrencyService, GeoLocationService – all of them just components which have methods that can be invoked remotely. If anything, the design would probably have been better if all the XML and HTTP was removed – if the components were all hosted in the same process.

In fact, in a well designed Service-Oriented Architecture, we tend to see components from many services deployed in process with each other (as I showed in my recent post UI Composition for Correct Service Boundaries).

But I’ve got to admit – the “bare knuckle” prefix, that’s some good link-bait.

Werner posts:

As this is a support forum, the nature of the threads are almost always about problems; which is fine. But I just feel like posting something not related to problems

After spending 2 weeks with [version] 3.x I must say that I like it a lot, and have recommended that we make a global change to the new version (we’re using 1.9 and 2.0).

I like the support that I see in this forum. Both from other users and especially from the people now making a living developing and supporting NServiceBus. No one mentioned, no one forgotten; thanks to all and keep up the good work!

————–

The most incredible thing about NServiceBus is our community – over 1600 of the best and brightest of the .NET developer ecosystem – over 500 of which have gone to the 5-day Advanced Distributed Systems Design course. In addition to going about building the most reliable software their industries have ever seen, many are there to lend a helping hand to those just getting started.

Not surprisingly, we’ve seen more people downloading NServiceBus than actually in the community, at least a 2 to 1 ratio. In my work with clients, I often see somewhere in the area 5-10 people actively developing with NServiceBus in a company, but only one of them downloading new releases and bringing them into the company.

I’ve heard mention of this “power law” before, but I’ve got to tell you, when I started on this journey with NServiceBus 6 years ago, in my wildest dreams I wouldn’t have ever thought that tens of thousands of developers around the world would be using my baby.

Thank you all.

“I’m sorry, but with all due respect I must strongly disagree. You haven’t avoided any orchestration work at all, you’ve just moved it in to client side script!

How are you going to deal with the scenario that one of the service calls fails? Say a failed credit card payment, or no more rooms left? In more javascript??

I would much rather take the less brittle approach of introducing an orchestration service. Like it or not, however trivial it may be, there is a relationship between these services, if one call fails, they both fail. This should be reflected in the architecture, not hidden in javascript. With an orchestration service you also either get transactions for free provided by infrastructure, or alternatively if the underlying service doesnt support this, explicit and unit testable control over recovery.”

Since this is a common point of view, I thought I’d take the time to explain a bit more.

Let’s start at a fairly high level.

On failures

I’ve talked many times in the past about how to handle technical causes for failure like server crashes, database deadlocks, and even deserialization exceptions. Messaging and queuing solutions like NServiceBus can help overcome these issues such that things don’t actually fail – they just take a little longer to succeed.

On the logical side of things, the CQRS patterns I talk about describe an approach where aggressive client-side validation is done to prevent almost all logical causes for failure. The only thing that can’t be mitigated client-side are race conditions resulting in actions taken by other users at the same time.

In short, it really is uncommon for things to fail when being processed server-side.

Back to the specific example

The concerns raised in the comment specifically talked about a failed credit card payment or no rooms left in the hotel, so let’s start with the credit card thing:

In my last post I talked about collecting guest and credit card information from the user as a part of the “checkout” process when making a reservation for a hotel room. Just to be clear – there is a final “confirm your reservation” step that happens after all information has been collected.

What this means is that we aren’t actually charging the customer’s card when we collect that data, therefore there is no real issue with a failed credit card payment that needs to be handled by the client-side javascript. When the customer confirms their reservation, yes, there might be a failure when charging the card though there are only some specific types of rates for which the hotel charges your card when you make a reservation.

In general, failed credit card payments are handled pretty much the same way for all ecommerce – an email is sent to the customer asking for an alternative form of payment, also saying that their purchase won’t be processed until payment is made.

In any case, it is only after the reservation is placed that the responsible service would publish an event about that. The service which collected the credit card information would be subscribed to that event and initiate the charge of the card when that event arrives (or not, depending on the rate rules mentioned).

With regards to there not being any rooms left, well, first of all, there’s overbooking – hotels accept more reservations than rooms available because they know that customers sometimes need to cancel, and some just don’t show up. Secondly, there is a manual compensation process if more people show up than there are actual rooms to put them in. In some cases, a hotel will bump you up to a higher class of room (assuming there aren’t too many reservations for those), and in others they will call a “partner” hotel nearby and put you up there instead.

In summary

While arguments can be made that yes, these issues have been addressed in this specific example, there may be other domains where it is not possible to do these kinds of “tricks”. Although I do agree with that in theory, I’ve spent the better part of 5 years travelling around the world talking to hundreds of people in quite a few business domains, and every single time I’ve found it possible to apply these principles.

In short, the use of UI composition allows services to collect their own data, making it so anything outside that service doesn’t depend on those data structures which makes both development and versioning much easier. Technical failure conditions can be mitigated at infrastructure levels in most cases and other business logic concerns can be addressed asynchronously with respect to the data collection.

Give it a try.

Let’s take for example a screen from the Marriott.com online reservation system (below). This screen collects information about the guest staying at the hotel (name, phone number, address, etc) and credit card information.

While we might have wanted to keep guest information in a separate service from the credit card information (which may very well be the corporate card of someone responsible for travel), the above screen would seem to indicate that the data would be collected together, validated together, and would also have to be processed together.

The traditional way

In standard layered architectures you would have all the data submitted by the user passed in a single call from a controller to some “service layer” (possibly running on a different machine), which would then persist that data in one transaction.

Even if some attempt was made to separate things out, there likely would be some “orchestration service” that received the full set of data and it would make calls to the other “services”, passing in the specific data that each “service” is responsible for.

I am putting quotes around the word “service” to indicate that I don’t consider these proper services in the SOA sense (as they lack the necessary autonomy) – they are more like functions or procedures, whether or not they’re invoked XML over HTTP is besides the point.

What to do?

Like so many other things, the solution is simple but a bit counter-intuitive as it doesn’t follow the way most web development is done, i.e. one submit button => one call to the server.

Let’s say the “Red” service is responsible for guest information and the “Blue” service is responsible for credit card data. In this case, each service would have its own javascript come done with the page and that script would register itself for a callback on the click of the submit button. Each service would take the data the user entered into its part of the page and independently make a call to “the” server (could be to 2 separate servers) where the data is persisted (potentially to 2 different databases).

This raises other questions, of course.

Now that the data submitted is being processed in 2 transactions rather than just one, we may need to figure out how to correlate the data. In this specific case, it’s not such a big deal as there is no direct relationship between the guest and the credit card – both need to be independently correlated to some reservation ID.

That reservation ID would likely have been “created” on a button click on a previous screen by some other service. The reason why I put the word “created” in quotes is that this could be as simple as having the client generate a new GUID and put that in a cookie (which would cause the reservation ID to end up being submitted along with subsequent requests). Another alternative would be to put the reservation ID in the session.

It’s quite possible that the reservation ID would only be persisted much later in the service that owns it when the user actually confirms the reservation on the website.

In any case, what we can see is that each of the commands of our respective services can now be processed independently of the others in an entirely asynchronous fashion thus vastly improving the autonomy of our services.

Some words on CQRS

This style of UI composition where services leverage javascript code running in the browser isn’t technically difficult in the slightest. The rest of the implementation of each service – having a controller that takes that data and passes it on for persistence can be quite simple.

I’d say even more strongly, most of the time you shouldn’t need to use any fancy-dancy messaging to get that data persisted – that is, unless you’re still stuck with the big relational database behind 23 firewalls type data tier. Embrace NoSQL databases for the simplicity and scalability they provide – don’t try to re-invent that using messaging, CQRS, persistent view models, event-sourcing, and other crap.

There are other very valid business reasons to embrace CQRS, but they have nothing to do with persistence.

Also notice, this is all happening within a service boundary / bounded context.

In closing

If you aren’t leveraging these types of composite UI techniques, it’s quite likely that your service boundaries aren’t quite right. Do be aware of the UI design and use it to inform your choices around boundaries, but be aware of certain programming “best practices” that may lead you astray with your architecture.

Also, if you’re planning on coming to my course in Toronto to learn more about these topics, just wanted to let you know that there’s one week left for the early-bird discount.

Finally, it’s good I have a birthday that comes around once a year to remind me that my time here isn’t unlimited and that I had better get off my rear and do something meaningful with the time I do have. If you get value from these posts, leave a comment or send me a tweet to let me know – it does wonders for my motivation.

Thanks a bunch.

In June, Andreas Ohlund will be teaching the Enterprise Development with NServiceBus course in London (details here) and on Wednesday June 27 he’ll be giving a user group presentation – Death to the Batch Job (details here).

In the beginning of July I’ll be in London delivering my Advanced Distributed Systems Design course (details here) and on Monday evening (July 2) will be giving a talk about Command, Queries, and Consistency (details here).

Towards mid-July I’ll be giving the same course in Israel (details here) and giving my “Who needs a service bus anyway?” talk on Wednesday July 18th to the Israel .NET Developers User Group – details here.

Hope you can make it.

Although the prices has gone up a bit to meet the rates of my training partners, I’m including the video recording of the course in the price so that you’ll have a complete reference to go back to after the course is over.

Hope to see you there.

Register here